🤖 fix: stop heartbeats and background wakes from pausing goals - #3955
Conversation
Three mechanisms made scheduled heartbeats appear to pause active goals (diagnosed from live session data): 1. Queue race: a message typed while the goal-creating turn was still streaming dispatched right after the queued set_goal applied and auto-paused the brand-new goal. MessageQueue now stamps entries with enqueuedAtMs, and goal safety skips the pause when the goal was created after the message was typed. 2. Fragile kickoff window: chat-tail reconciliation paused active goals whose in-memory kickoff candidate was lost (restart, eviction) before the first continuation fired — the next getGoal (heartbeat/wake tool assembly runs one every turn) silently flipped them to paused. Goals with lastContinuationFiredAtMs == null are now exempt from the active→paused manual_user reconciliation, making the kickoff window durable and self-healing. 3. Accounting noise: recordStreamAccounting charged paused/complete goals for maintenance streams (heartbeats, background wake turns), bumping updatedAtMs so every heartbeat looked like it had just touched the paused goal. Only goal-driven origins (goal_continuation / goal_budget_limit) now charge non-active goals, mirroring attributeChildReport.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e67e217ff9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
…econciliation, budget_limited accounting) - P1: preserve the projected goal's createdAtMs through the mid-stream drain so interventions queued against the visible optimistic goal are not misread as pre-goal input. - P2: persist enqueuedAtMs on queue-dispatched user rows and scope the never-driven kickoff exemption to rows authored before the goal existed; post-goal rows pause on reconciliation even when the dispatch-time auto-pause was lost to a crash. - P2 (accounting): extend the maintenance-stream skip to budget_limited goals so background wakes cannot inflate the recorded overshoot.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02ab4595ee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…e maintenance cost previews - Rejected manual sends (pricing gate) now persist timestamp + enqueuedAtMs so chat-tail reconciliation can classify the row as pre-goal after a restart instead of pausing a never-driven goal. - previewStreamAccounting receives the stream origin kind and mirrors final accounting's budget_limited maintenance skip, so the Goal UI no longer shows climbing heartbeat/wake cost that snaps back at stream end.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4bd9a185f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex P2: createGoal() stamped createdAtMs before the kickoff-model validation and streaming re-check awaits, so a message queued during those awaits postdated the stamp yet predated goal visibility, and the pre-goal guard misread it as an intervention. Re-stamp fresh projected goals immediately before publishPendingGoalSnapshot.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e712711382
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
…off candidate on ack failure Codex P2 x2: (1) the round-3 stamp still preceded the async activity-snapshot read inside publishPendingGoalSnapshot, so a message queued during that read postdated the stamp while the renderer had not yet received onActivityChange — publish first, then stamp fresh goals. (2) reordering acknowledgeUser before the candidate clear meant an acknowledgment throw skipped the clear entirely, letting a stale kickoff candidate dispatch a continuation despite the user's persisted intervention — clear conservatively on failure, then rethrow.
|
@codex review |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 930f18846d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ure authoring time at request entry Codex P1: round 4 moved pendingGoalMutations.set after the publication await, so a user abort landing during publication deleted nothing and the setter then installed the mutation anyway — resurrecting a goal the abort discarded and silently applying it at the next stream end. Install before publish; guard the post-publication re-stamp on mutation identity so an interleaved abort (or competing setter) is never overwritten. Codex P2: lastAddedAtMs was sampled at enqueue, after WorkspaceService preflight awaits (pricing gate, settings persistence) — a goal becoming visible during those awaits postdated the user's authoring. Capture authoredAtMs at request entry and thread it through queueMessage / MessageQueue, the pricing-rejection delegation, and direct sends.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1340998494
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Security Review
Here are some automated security review suggestions for this pull request.
Reviewed commit: 1340998494
Only the user who started this review can view the report in Codex.
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
…ublication setter Codex P2: a stream ending during the setter's publication await let applyPendingAfterStreamEnd take the mutation synchronously (outside the goal file lock) and persist the pre-publication construction stamp — a message authored during the publication await was then misclassified as a post-goal intervention. The drain now flushes the lock and re-reads the mutation, draining the finalized publication stamp (or honoring a discard that landed in the same window).
…hed sends Codex security P2: overlapping sends can complete preflight out of authoring order, letting an older pre-goal message overwrite a later post-goal stop/correction's authoring time — the batch then satisfied the pre-goal guard and kept the goal running despite the intervention. Fold each add's authoring time in via max(); seed entry creation with 0 so an authoredAtMs captured before slow preflight is never swallowed by the creation wall clock.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9b94d898d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…al-safety comparisons Codex P2: chat.jsonl rows are unchecked JSON, so a malformed enqueuedAtMs (negative/NaN/string) could beat a valid row timestamp in the nullish selection and misclassify a genuine post-goal intervention as pre-goal, leaving a never-driven goal running after restart. Only finite non-negative numbers participate; invalid values fall back to the validated row timestamp.
|
@codex review |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04e46a95a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…n consent The pre-goal timestamp bypass (enqueuedAtMs <= createdAtMs) let a model publish a goal AFTER a user's queued stop/correction and shield its autonomous continuations from the any-manual-turn-pauses boundary. - GoalRecordV1.lastUserActivationAtMs: stamped only by explicit user activations (direct create, Resume, board promote); model set_goal, auto-promotion, and accounting re-arms never stamp, so their goals fail closed (queued manual messages always pause them). - Dispatch-time hook and durable-tail kickoff exemption now require the consent stamp to postdate the message's authoring. - Durable tail additionally keeps never-driven goals active when the manual row was PROCESSED (completed assistant row follows it): the initiating prompt whose turn produced the goal is not an unprocessed intervention, preserving restart/eviction crash-safety for model-created goals born from an ordinary prompt. Codex security P2 (PRRT_kwDOPxxmWM6cSGrq)
…fetime to session busy claim - explicitPauseGenerations bumps inside writeGoal (commit point) so admission probes read stale during the publication window before the finalization hold arms (PRRT_kwDOPxxmWM6cSREI) - resumeStream holds the session-invisible reservation until session admission settles (PRRT_kwDOPxxmWM6cSREO) - direct sends release the reservation via onTurnAdmissionCommitted at the synchronous PREPARING claim instead of the service handoff (PRRT_kwDOPxxmWM6cSRkH)
|
@codex review |
|
@codex security review |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
💡 Codex Security Review
Here are some automated security review suggestions for this pull request.
Reviewed commit: e0626ebb4d
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
…robes recordUserStoppedStream leaves an active goal's status and identity unchanged, so pause/terminal/identity generations stay fresh across a Stop — a recovered goal-scoped follow-up admitted before the Stop could start an exec turn after it. Codex security P2 (PRRT_kwDOPxxmWM6cSx0M)
|
@codex review |
|
@codex security review |
There was a problem hiding this comment.
💡 Codex Security Review
Here are some automated security review suggestions for this pull request.
Reviewed commit: a9622bd72a
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9622bd72a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… consent ordering; non-synthetic followers - pendingStopAcknowledgmentCounts: a Stop in flight (generation bumped, acknowledgment not yet durable) refuses redispatch admissions and marks captured probes stale (PRRT_kwDOPxxmWM6cS7qG) - legacy compaction follow-ups carrying goalKind without goalId are conservatively discarded — no unscoped synthetic redispatch against a replaced goal (PRRT_kwDOPxxmWM6cS8Bq) - activation consent requires strict ordering: same-millisecond equality fails closed to pause (PRRT_kwDOPxxmWM6cS8Bu) - processed-row rule rejects synthetic assistant followers (e.g. goal-cleared summaries) as proof of settlement (PRRT_kwDOPxxmWM6cS8Bx)
|
@codex review |
|
@codex security review |
…ollback - goalIdentityGenerations now bumps on same-ID objective revisions (editInPlace renames), so captured continuations embedding the old objective go stale when the user redirects the goal (PRRT_kwDOPxxmWM6cS8B1) - promoteNextUpcomingUnlocked re-samples the stop veto after its board/goal/snapshot writes and rolls the promotion transaction back (restore board + prior record + re-publish) when a Stop landed inside them (PRRT_kwDOPxxmWM6cS8B4)
|
@codex review |
|
@codex security review |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e59e31241
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…pendent of kickoff window - terminalStatusGenerations now also bumps when a record leaves a terminal status (budget raise re-arms budget_limited→active), so captured budget wrap-up admissions go stale instead of charging a stopping turn after reactivation (PRRT_kwDOPxxmWM6cTN_o) - the durable-tail consent arm applies independently of the never-driven kickoff guard, preserving an explicit Resume for goals that already fired continuations (PRRT_kwDOPxxmWM6cTN_r)
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
1 participant
Summary
Scheduled heartbeats (and background wake turns) could leave active goals paused — or make already-paused goals look like they were still being touched. This PR fixes the three mechanisms behind the symptom: a queue race that auto-paused a freshly created goal, a fragile in-memory kickoff window that let any maintenance turn's
getGoal()flip a never-driven goal to paused, and stream accounting that charged paused/complete goals for maintenance streams.Background
Diagnosed from live session data (user report: "heartbeats are pausing goals"). In one session, a model-created goal was auto-paused ~500ms after creation by a user message that had been queued while the goal-creating turn was still streaming; heartbeat turns then ran for hours charging the paused goal ($7.11 / 8 turns,
updatedAtMsbumped every turn) without ever driving it. In another, an active goal whose kickoff continuation was blocked byactive_descendant_taskssurvived only via the in-memory kickoff candidate — any candidate loss (restart, eviction) let the nextgetGoal()(tool assembly runs one on every heartbeat/wake turn) silently pause it via chat-tail reconciliation.Implementation
Queue-race guard —
MessageQueuestamps each entry withlastAddedAtMsanddequeueNext()exposes it asenqueuedAtMs.applyManualUserMessageGoalSafetyskips the auto-pause (and candidate clear) whengoal.createdAtMs >= enqueuedAtMs: the user cannot have been intervening against a goal that did not exist when they typed. Messages typed after the goal exists still pause it; explicit Pause is unaffected.Durable kickoff window —
applyChatTailGoalModeno longer reconciles an active goal to paused off a manual-user chat tail whenlastContinuationFiredAtMs == null. A never-driven goal has nogoal_continuationrow yet by construction, so its tail always ends at a pre-goal manual row; pausing on that is a false positive. This makes the kickoff window durable across restarts and candidate eviction, and self-healing: the next stream end re-arms the continuation instead of stranding the goal.Maintenance-stream accounting skip —
recordStreamAccountingnow skips paused/complete goals for all origins exceptgoal_continuation/goal_budget_limit(previously onlyuserwas skipped). Heartbeats resolve touserorigin and wake turns toother; neither should charge turns/cost or bumpupdatedAtMson a goal that is not running. MirrorsattributeChildReport's existing paused/complete skip.Validation
"other"-origin accounting.src/node/servicessweep: the only failures (BackupRepoCache, TaskService recovery, agent_skill_delete) reproduce identically on a clean-HEAD probe worktree — pre-existing host issues, not regressions.Risks
Goal lifecycle logic; moderate regression surface:
enqueuedAtMsonly being present on queue-dispatched sends; direct (idle) sends never carry it and keep today's behavior.AgentSession, so this is bounded to cost accounting, not control flow.Generated with
xum• Model:anthropic:claude-fable-5• Thinking:xhigh• Cost:$49.89