fix(hooks): account an ingest commit apart from its own drain - #1792
Merged
Merged
Conversation
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65dcaa666a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+311
to
+315
| tracedecay_code_index_runtime::code_index_scheduler::query_runtime::mount_core_query_authority_on_project_open( | ||
| schedulers, | ||
| &worktree_root, | ||
| &scope, | ||
| &cursor_keys, |
There was a problem hiding this comment.
When the primary checkout or another published branch already has a query authority, this call constructs and mounts a fresh Arc<QueryAuthorityV1> rather than sharing the incumbent authority. mount_query_authority_from_project_peer rejects multiple same-project candidates unless they are Arc::ptr_eq, so after one branch is published, any later branch whose best-effort direct mount misses can no longer use the documented peer fallback and exact search returns authority_unavailable. Reuse a canonical project authority instead of creating a per-worktree instance.
AGENTS.md reference: AGENTS.md:L132-L134
Useful? React with 👍 / 👎.
Comment on lines
+208
to
+209
| if progress.resumed && progress.frames_persisted == 0 { | ||
| self.replayed = self.replayed.saturating_add(1); |
There was a problem hiding this comment.
When a previously scanned Cursor JSONL file receives a malformed or unsupported record, admission resumes from its cursor, consumes and covers the new frame, and reports zero persisted frames. This condition nevertheless marks the source as replayed, causing the hook response to claim exact_duplicate even though the new input was rejected rather than already durable. Only treat the source as replayed when no new frames or bytes were processed, or carry the refused/skipped disposition into the verdict.
AGENTS.md reference: AGENTS.md:L9-L12
Useful? React with 👍 / 👎.
6 tasks
`packaged_host_ingest_delivers_a_registered_advisory_cycle` failed roughly every other run on the 4 vCPU arm runner with `admission: accepted_for_replay, completed: true, messages_upserted: 0`: a terminal, non-retryable status that neither proves a commit nor invites a retry. Reproduced locally at ~30% under `taskset -c 0-3`. Root cause: `authority_changed` for the Cursor and Codex hook routes was derived only from the projections the pass drained itself. The projection queue is per scope and consumed on projection, and the daemon's project catch-up sweep (`ingest::project_provider::run_cursor`) drains the whole scope's queue, not just the rows it admitted. On a slow runner the scheduler tick lands between the ingest's admit and its own drain, so the ingest finds an empty queue and reports zero, discarding the two observations it had just persisted. Daemon debug evidence from a failing run: `transcript_admission_batch phase=complete total_frames=2 bytes_consumed=175 source_deferred=false` followed by `messages_upserted: 0`. Admission is the durable commit; projection is downstream materialization. Carry `observations_committed` (frames persisted) from the Cursor and Codex admit loops through to the capture outcome, count it as an authority change, and report it in the ingest payload so a zero-projection pass is readable without guessing which drainer won. A second, distinct state was equally vacuous: a pass that re-scans a source already at its stored cursor, whose rows a peer drained earlier, persists nothing and finds nothing. That is `exact_duplicate`, but Cursor had no way to say so: `CursorProjectionDrainStats::into_transcript_stats` dropped `exact_duplicates`, and `exact_duplicate` was derived only from Claude observation stats, which are always `None` for Cursor. Both gaps are closed; `JsonlObservationAdmissionProgress::resumed` supplies the evidence that a source was already admitted rather than empty, so a first-ever scan is never reported as a duplicate. A genuinely deferred pass still reports that honestly: `source_deferred` and the backpressure mapping are untouched. The acceptance assertion now accepts `committed` or `exact_duplicate` and still rejects `accepted_for_replay`. Both prove the transcript is durable; the retry path (a first pass deferred by an incomplete projection rebuild, then a re-scan of the exhausted source) legitimately terminates in the latter. Verification: `taskset -c 0-3` acceptance run 12/12 pass (was 2/6 and 0/1 before the fix, same binary shape); three new deterministic tests in `runtime::hosts::cursor::tests` cover the peer-drain steal, the replay, and the empty-source negative; `tracedecay-sessions --lib` 569/569, `tracedecay-mcp --lib` 367/367, `tracedecay-host-admission` 1/1. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ScriptedAlchemy
changed the title
ScriptedAlchemy
force-pushed
the
fix/master-ci-green-2
branch
from
September 18, 2026 23:47
65dcaa6 to
db2ee84
Compare
dashboard_progress_does_not_wait_for_the_scheduler_mutex asserted Fresh and ignored_dependency_waits_for_global_admission_before_publication_gate asserted an idle global admission, and both sampled a background pass instead of the quiet worktree they set up. Master run 35402148043 failed both on try 1 (the dashboard read Some(Verifying), the admission read 0 permits); under `taskset -c 0,1` they reproduced 3/25 and 4/25. Both read the same state through different windows: the freshness ladder's `refresh_in_flight` is the pass counter *or* the pending-arrival slot, and the admission permit is held for a pass's whole source reconcile. Settling the mount-era chain the seat now leaves behind (6318c18, 1828d6e) is not enough, because an empty arrival slot is not an empty queue: `note_wake` posts a `Notify` permit for an arrival a running pass then claims, and `note_worker_continuation` replenishes that permit whenever it cannot claim the slot. Either leaves a banked permit behind a settled owner, and the worker's next `notified()` spends it on a no-op pass. A dump at the dashboard failure showed exactly that: two receipts (Mount/Published, BusyFollowUp/Noop), `pending=Some(0)`, and a third pass already in progress. That replenish is the liveness net for a pending arrival whose permit this pass consumed, so it stays. The tests drain the banked permit instead: release the admission, give the worker's claim its turn, and settle, until a release leaves the permit free. The dashboard test also drops to a single background permit, since the default bound is the host core count and one held permit parks nothing there; it then holds that permit across the sample. Both pass 40/40 under `taskset -c 0,1`, and noop_reconcile_tests, unchanged_background_freshness_probe_posts_no_overflow_wake, distinct_stores_reconcile_in_parallel_under_bounded_admission and registry_clone_freshness_reports_coverage_and_update_accounting pass 6/6. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`legacy_generation_restore_does_not_materialize_its_evidence_segment` failed run 35407045249 at 1.288x of the evidence segment, inside the band 30c0a5d683 measured as clean on four earlier runs. The guard was measuring the wrong thing. Peak RSS cannot see this regression. VmHWM only moves for an allocation that does not fit in the heap the restored generation already made resident, and the restore itself grows that heap by 6.6 MiB before the first measured probe. Injecting a clone of every evidence page into a buffer held for the whole decode - the retention this test exists to refuse - moves the shipped guard by 0.302x and it passes. What VmHWM does register is additive noise: when the allocator trims between probes the next one faults fresh pages, which on a loaded 4 vCPU runner costs more than the 1.7 MiB segment under test. That, not a regression, is what failed the run. Assert instead on what the restore states about itself. Every segment read it issues is already visible to the caller through the read callback, and a restore that materializes the segment has to ask for it: the whole segment in one read, or a range that grows with it. The paged form of the identical generation is the reference, so the pre-paging form must ask for the same count of equally bounded ranges into a buffer bounded the same way. That is exact, catches the shipped regression (a whole-segment read) on the first probe in 0.47s, needs no memory reading, and holds on every platform rather than returning early off Linux. Peak RSS stays as a loose ceiling on the rest of the restore, and is made noise-tolerant without loosening the bound: each form is probed over four alternating rounds and the smallest growth is taken as its cost, because the noise is additive while a materializing restore pays on every round. The bound stays at one evidence segment. Verified on this lane: 10 consecutive runs pinned to 4 CPUs against a concurrent workspace `cargo check` on the same cores, all passing at 0.000-0.031x of the segment (was 0.12-1.288x); full code_index_suite 165/165 and the crate's 256 lib tests; clippy -p tracedecay-code-index --all-targets -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`background_refresh_and_reopen_report_only_servable_generations` was
FLKY-FL 2/2 on the 4 vCPU Linux root-transport runner: TRY 1 failed after
63s inside `ProductionProjectCompositionHarnessV1::open`, TRY 2 passed in
24s as the last test on the box. The failure is the harness's own publish
gate, not any assertion:
production-composition code index did not publish ... after 20000 ms;
composition gate capacity=1, admitted=1, waiting=0
with `progress.phase: Ready, completed_files: 98/98, committed_pages: 290,
committed_payload_bytes: 31005972, completed_lexical_units: 56970602` and
`clone_index: Stale { completed_source_pages: 0, total_source_pages: 290 }`.
The reopen was still working through the refresh batch when its 20s budget
expired.
Root cause is fixture weight, not a regression. The batch exists only to
keep one background refresh observable across a few status polls, but it
stayed installed for the rest of the journey, so both reopens re-indexed
12,288 symbols inside a fixed 20s gate the test neither controls nor
asserts on. dccdb15cdd already cut the batch 768 -> 96 files for exactly
this reason; 96 is still ~8x more than the observation needs and the cost
is paid three times.
Two changes, both to the fixture and none to an assertion or a budget:
retire the batch in the offline commit before the first reopen, so its
weight lands on the refresh it exists for; and drop each file from 128
symbols to 16, which buys no fewer `partial_refresh_in_progress` samples at
the 25ms poll interval.
Attribution: PR #1792's `51402cdf8d` (the published branch worktree's query
authority mount) is not the cause. Reverting it locally and re-running under
the same contention failed 2/2 (79.8s, 88.5s), byte-identical gate message.
Verification under `taskset -c 0-3`, contention emulated with four spinners
pinned to the same cores (base: FAIL at 66.6s, matching CI's 63.3s):
loaded 5/5 pass (34.0-47.1s), idle 5/5 pass (7.9-9.2s, was 36s). The
`wait_for_background_refresh` window is still observed on every idle run,
which is where it is tightest. Clippy clean on `tracedecay --features
tracedecay/test-transport --test transport_acceptance_suite`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`admitted_project_id` in the transport restart journey asserted success on its first `storage_status` right after `init`. The daemon can still be mounting the project's query authority then, and says so with a pre-admission `application.surface.unavailable` whose own retry directive is `after_delay` / `retry_after_millis`. Honour that directive the way a production client would instead of reading it as a verdict. `read_only_project_binding_refuses_before_scheduler_mutation` waited 5 s for the complete generation to seat; on a loaded 4 vCPU runner that background work has taken longer (CI: TRY 1 timed out at 5.2 s, TRY 2 seated in 0.6 s, and nextest's `flaky-result = "fail"` counts that as red). The bound is now 60 s; the assertion is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`completed_session_import_immediately_searches_canonical_message` timed out on its own 30 s deadline in the master root-transport partition (run 35402148043, both nextest tries). Reproduced under `taskset -c 0-3` with four competing CPU hogs: 42-45 s, deterministic. Unloaded it took 23 s against a 30 s wall, so the partition's own parallelism is all it takes to cross it. `ProjectProviderRun::run_codex` ended its pass at the first rollout that consumed bytes whenever another discovered path followed it, and marked the discovery frontier uncommittable on the way out. So a pass admitted exactly one rollout however much of its byte budget was left, the frontier never advanced, and the next pass rediscovered and re-read every rollout it had already exhausted. For the fixture's 33 rollouts that is 33 passes and 561 source admissions for 99 messages: the debug log shows 554 `transcript_admission_batch` events, n(n+1)/2. It is quadratic in the rollouts a project has, so a real project pays far worse than this test. The profile-scope loop in `ingest/user.rs` has always walked every discovered path until its byte budget runs out, over the same admission call for the same provider. The bound that owns a resumable cursor is `MAX_CAPTURE_WINDOW`: it stops inside one rollout and reports `source_deferred`, and the next pass resumes at the stored byte offset. Yielding per finished rollout has no cursor behind it, which is why it re-reads instead of resuming. Break only on `source_deferred`, exactly what `frontier_committable` already tests, and let `remaining` bound the pass as it does for the profile scope. The import now takes 5.3 s of ingest instead of 20 s, and the guarantees are unchanged: a partially read source still ends the pass and still blocks the frontier, cancellation and the byte budget still break at the loop head, and coverage is still Partial until a pass finishes every discovered path. `project_provider_yields_between_dated_rollouts_and_converges_without_loss` pinned the one-rollout-per-pass step with `max_new_bytes: u64::MAX`, so it asserted a bound the byte budget was never allowed to express. It now sizes its newest rollout past one capture window and asserts the yield that has a cursor: pass 0 stops inside that rollout, pass 1 resumes it and finishes the two behind it. Its convergence, no-loss and no-duplication assertions are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With the ingest pass no longer quadratic, the same import test failed on a new terminal instead: `termination: "failed"`, `failure_codes: ["session_history_not_current"]`, every stat zero, in 5.3 s. `await_import_history` is stage one of a two-stage import gate. It waits for both historical ingest workers through `wake_history_and_wait_until_idle`, and then also required both projection serving states to be `Current` and both projection stores to hold no pending rows. It does not wait for any of that. Stage two, `await_import_projection`, does: it wakes the projection workers, waits for them, and re-checks those same four conditions plus backlog and availability. But it only runs when stage one reported coverage complete with no failure codes, so the moment stage one read a projection that had not drained yet, the stage that exists to wait for it was skipped and the import reported a history failure for a history that was current. A slow ingest hid this: the projection always drained inside the 20 s stage one spent waiting on history, so the extra conditions were incidentally true by the time they were read. Making the ingest fast is what exposed the ordering. Check what the stage's name and its `session_history_not_current` failure code claim, and leave projection currency to the stage that waits for it. Stage two's conditions are a strict superset, so nothing stops being verified; a projection that genuinely never converges still terminates the import, through `session_temporal_projection_not_current`. Verification: the previously failing `completed_session_import_immediately_searches_canonical_message` passes 6/6 under `taskset -c 0-3`, three of them against four competing CPU hogs, in 14-16 s against its 30 s deadline (was 42-45 s and failing). Its whole module 6/6, `tracedecay-sessions --lib` 569/569, `tracedecay-session-runtime --lib` 113/113, `session_suite` 448/448, `transcript_ingest_suite` 190/190. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`concurrent_query_admissions_claim_one_pending_wake_before_worker_coalescing` failed in CI with zero of eight admissions accepted. The owner's worker releases the background admission halfway through a pass and drops its `reconcile_pass` guard before the branches that call `note_worker_continuation`, so a settled-looking owner can still stamp `BusyFollowUp` into the coalesced pending-wake slot. The test's raw `acquire_owned` returned at that mid-pass release, and the tail then refilled the slot the test had just emptied, so every concurrent `PendingWakeClaimV1::claim` saw it occupied and declined. Instrumenting the slot under contention reproduced it and named the stamper: `trigger=5` (`BusyFollowUp`) with the pass still in flight and all eight verdicts `Unavailable`. Take the admission through `quiesced_background_reconcile_admission` so no new pass can start, then clear the slot until it survives a quiet window, which is the proof `wait_for_settled_owner` cannot give while the guard drops early. The two query-claim-gate tests get the same fence: there a tail's stamp declines the request before the gate and hangs `wait_for_query_claim`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Audit of the reconcile module for the same pattern: a single-permit registry that holds the background admission, clears the coalesced pending-wake slot, then asserts the slot stayed empty. Each is exposed to the same pass tail, which drops its `reconcile_pass` guard before its `note_worker_continuation` branches and can stamp `BusyFollowUp` after the clear. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keeps #1793's history-worker continuation (a progressing window no longer pays the no-progress backoff) and takes this branch's versions of the two reconcile tests it also touched: the lane's fencing burns the banked wake permit and waits for the pass tail, which is the hole Codex's review flagged on #1793's variant. Its unused helper is dropped with it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
After a physical restart the authority record is published before the daemon has opened the project, and the first HTTP observation in that window is a typed pre-admission `unavailable` with `retry: after_delay` (master run 35411306897, TRY 1 of `reset_required_survives_http_mcp_and_rust_sdk_across_restart`). Honour the directive as a production client would and assert on the first admitted reply. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`dashboard_progress_does_not_wait_for_the_scheduler_mutex` still read `Verifying` on CI run 35412193695 after the settle: `refresh_in_flight` also reads the pending-wake slot, and the settled owner's pass tail can stamp `BusyFollowUp` into it after every settle check. With the single admission held that tail is finite, so clear the slot until it stays empty, the way the query-claim tests already do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`coalesced_publication_failure_preserves_the_scheduler_error_family`
induces an owner publication failure by writing `{` over
`active-code-generation-v1.json`, then asserts the coalesced follower
sees the owner's publication error family. Master run 35411306897 failed
it on try 1 in 0.64 s: the owner published successfully and returned an
outcome instead of failing closed. CI's nextest profile has
`flaky-result = "fail"`, so the try-2 pass still reported red.
Every production writer of that pointer reads it, edits it in memory and
renames a temporary over it while holding the exclusive generation-store
lock. The test wrote the file with no lock at all, so its corruption
raced an in-flight read-modify-write whose rename restored a valid
pointer; the owner's `validate_serving_is_active` then read an intact
generation and published.
An ordered trace of failing runs named the racer. The last event before
the corruption is always a text-artifact mutation about to rename its
edited pointer, and the pointer the owner reads afterwards is that same
valid pointer - one run even observed the restore from the test thread,
`on_disk_len=Ok(1753)` immediately after `hold.release()`. That mutation
runs in the background pass tail, after the pass released the background
admission permit this owner then took ("release the background admission
permit before HeadOpening / graph work", registry/mount.rs), so neither
the held admission nor the held scheduler mutex proves the store is
quiet - the same pass-tail exposure fenced in 656b532 and b5a784c.
Corrupt under the store lock instead. Being granted it proves no writer
is mid-transaction, and a writer that starts after it is released reads
the corruption under the lock and refuses rather than overwriting it.
The assertions are unchanged: the owner and the follower must both fail
with `Production(Publication(_))`.
Under `taskset -c 0,1` the target test reproduced 3/48, 5/48 and 7/60
before with 6 concurrent copies, 0/40 serially. After: 0/40 serially,
0/60 across 10 rounds of 6 concurrent copies, and the whole
`code_index_ignored_dependencies_test::` module passes 21/21.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`spawn_tracedecay_daemon_process` refused to start when the profile's endpoint was still connectable, asserting instantaneously. Stopping a daemon is asynchronous with respect to its endpoint: `TestChildProcess` kills and reaps the PID it spawned, but the kernel keeps the listening socket alive while any duplicate of that descriptor survives, including one a subprocess inherited across `fork` and still holds because it has not reached its own `exec` yet. A reaped PID is therefore not proof that the endpoint stopped accepting. `init_project_fixture` journeys walk straight into that tail: spawn a daemon, run `tracedecay init`, drop the daemon, spawn another one. On a loaded runner the teardown tail outlives the reap and the second spawn reported an ordinary teardown as a live daemon, which is how `branch_search_serves_a_committed_generation_behind_dirty_worktree_state` and its `branch_add_...` sibling went red. Wait a bounded time for the endpoint to stop accepting instead. A predecessor that keeps accepting past the timeout still fails with the same refusal, so a genuine daemon leak is still reported. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`preserved_profile_lcm_discovery_converges_without_blocking_retrieval` failed both nextest tries of run 35414007809 at ~21 s, on `12-hour direct-user lcm_grep must return parsed session identities` with `status: "stale", count: 0`. Reproduced 4/8 under `taskset -c 0-3` against four CPU hogs; every local failure carried that same typed `stale`, never a wrong-window hit and never a thin corpus. The journey asserts that retrieval stays admitted while convergence runs, so a window the projection has not caught up to is answered with typed staleness rather than blocked. That makes any single read racy against background ingest: `wait_for_preserved_discovery` had already been served non-empty hits for this exact query and window, and the projection fell behind again while the test walked the assertions between that gate and the final grep. Reverting `48cc9d5b59` (bound a project ingest pass by bytes) locally takes the failure to 0/8 and the run from 21-46 s to 99-129 s: the quadratic re-read it removed was pacing the test, so discovery converged only once ingest was nearly done and no lag window was left to observe. It is not a regression, it is the same journey run against an ingest that is no longer quadratic. Reverting `ba881a17a5` (5/8) or `7f76433808` (5/8) changes nothing; this journey never drives the import gate. `wait_for_pre_window_search` already looped on exactly this typed staleness, for one of the three window reads. Generalize it into `converged_window_read` and route all three through it, bounded by the same `CONVERGENCE_WAIT` the discovery gate uses. The elapsed time it returns is the served call alone, so `SEARCH_BUDGET` still measures one answer rather than the wait in front of it, and every existing window, budget and payload assertion is unchanged. Verification: 20/20 under `taskset -c 0-3` with four CPU hogs (was 4/8 failing), the module once, `cargo clippy -p tracedecay --all-targets --locked -- -D warnings` and `cargo fmt --all -- --check` clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`production_codex_hook_ingest_survives_message_search_reopen` failed TRY 1
of run 35414007809 at 6 s with
transcript ingest failed: authority_write_failed: submit observation
batch: ... Invalid parameter name: observation repository provenance
collision: retained provenance is not replayable
`Invalid parameter name` is not a binding bug: `repository::support::invalid`
builds `rusqlite::Error::InvalidParameterName` for every domain refusal this
crate raises through a `rusqlite::Result`, so the text after it is the whole
error. The refusal is `verify_observation_authority`'s, and it is a replay
verification: a write whose observation id already exists must read back the
rows the first apply wrote.
Root cause: the MCP hook route (`admit_codex_project_rollouts`) and the
daemon's project catch-up sweep (`ingest::project_provider::run_codex`) both
reach `try_admit_codex_jsonl_observations` for the same rollout under the same
scope, with no mutual exclusion, and both read the source cursor before they
write. Interleaved, the loser reads a cursor the winner has not published yet,
re-reads frames the winner has already committed, and re-submits the same
observation ids with its own independently captured repository provenance.
`repository_replay_anchor` normalizes only a later local capture clock, so a
second capture that differs in any other way is refused, and the host reports a
retryable infrastructure failure for what is a duplicate.
Hold one gate across the cursor read and the admit for one (scope, rollout)
pair. The loser then reads the advanced cursor, persists nothing, and returns
the `resumed` replay. The gate is per process; cross-process writers still meet
at the store transaction, which is what the replay verification is for.
That replay had no way to say it was durable. db2ee84 closed exactly this
for Cursor ("`CursorProjectionDrainStats::into_transcript_stats` dropped
`exact_duplicates`") but the Codex half was never landed:
`CodexJsonlAdmissionProgress` dropped `resumed` from the inner
`JsonlObservationAdmissionProgress`, so `capture_codex_project` could not set
`exact_duplicate` and every already-admitted rollout reported
`accepted_for_replay` -- db2ee84's own "terminal, non-retryable status that
neither proves a commit nor invites a retry". Carry `resumed` through and tally
it the way Cursor does: a pass is a duplicate only when it scanned at least one
rollout and every one of them resumed with nothing new to persist.
Evidence: instrumented at 6 concurrent copies under `taskset -c 0,1`, 12/12
runs reported `admission: accepted_for_replay, messages_upserted: 0` -- the
catch-up won every time and the hook never described the rollout it had just
proved durable. The test now rejects `accepted_for_replay` and accepts
`committed` or `exact_duplicate`, the same bar db2ee84 set for the Cursor
acceptance test; 72 post-fix runs, none reported the vacuous status.
Not fixed here, and not this race: the first `tracedecay_message_search` after
the ingest can still answer `outcome: complete_zero` with
`freshness.state: partial, generation_lag: 1` from a store that already holds
the rows. The project store's search generation is published only by the
background session-temporal refresh tick; no hook ingest route wakes it, and
`tracedecay_session_refresh_begin` converges the profile scope, not the project
store the search reads. Measured: converges within 500 ms of the failing read
in every instrumented case.
Verification, 6-way concurrent under `taskset -c 0,1` on a host shared with
other build lanes, so the rates drift with load and only their shape is
meaningful: baseline 22/30 failures; reverting 48cc9d5 17/30, ba881a1
22/30, 7f76433 13/30, so none of the three opens the race and all three
widen a pre-existing window; with this change 15/30 and later 30/42 at a higher
host load, which is the same rate, not an improvement. What did change is
categorical: every remaining failure is the search-generation lag above, and
across 114 post-fix runs none reported an admission status other than
`committed`/`exact_duplicate` and none hit a provenance collision.
`session_search_test::` module 6/6, `tracedecay-mcp --lib` 367/367,
`tracedecay-sessions --lib` 568/569 (the one failure,
`codex_session_meta_prefix_is_decoded_once_across_consumers`, fails identically
on the pristine tree and passes in isolation).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`cargo clippy -p tracedecay --all-targets --features tracedecay/test-transport -- -D warnings` fails on `unnecessary_sort_by` in `pr_context_facts`, which blocks clippy for the whole `mcp_suite` target under the feature lens the suite's own tests need. CI's workspace lens does not build that target, so the lint has been latent since c8fbc38. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`read_only_project_binding_refuses_before_scheduler_mutation` polled `latest_complete_fresh` every 25 ms while waiting for `Fresh`. That read leaves a coalesced wake behind whenever it finds the worker holding the scheduler with an expired proof, so the poll re-armed a no-op pass faster than the ladder could settle: CI run 35419627712 sat in `Verifying` for the full minute and passed in 0.3 s on the retry. Poll the side-effect-free dashboard projection and read the generation once the ladder is `Fresh`. 60/60 as six concurrent copies on two cores. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
advisory_runtime_acceptance::packaged_host_ingest_delivers_a_registered_advisory_cyclereportedaccepted_for_replayon roughly every other CI run (failed both tries on master run 35360666279 and on 35397498419). Reproduced locally at ~30% undertaskset -c 0-3with daemon debug logging: the ingest pass persisted its two observations, then the daemon's project catch-up sweep drained the scope-wide projection queue during itsscheduler_tickbefore the ingest's own drain ran, so the ingest found nothing to project, reportedmessages_upserted: 0, andauthority_changed(derived only from what the pass itself drained) was false — a real commit rendered as a vacuousaccepted_for_replay.Fix: admission is the commit.
observations_committedis carried from the Cursor and Codex admit loops intoTranscriptCaptureOutcomeandauthority_changed, and an already-consumed re-scan (rows drained by a peer earlier) now reports the typedexact_duplicate(Cursor previously could never say so becauseinto_transcript_statsdroppedexact_duplicates). Deferred passes are untouched. Three deterministic tests inruntime::hosts::cursor::testspin the peer-drain race; the replay test fails without the fix. Acceptance test: 12/12 undertaskset -c 0-3(2/6 before). The acceptance assertion acceptscommittedorexact_duplicateand still rejectsaccepted_for_replay; on failure it now dumps the daemon log.Validated: root-journeys green on fix/master-ci-green run 35405494516 (tip e9afba0, which carried this commit).
The branch-worktree query-authority mount that was previously bundled here was split out: on the same run it correlated with new mount/publish failures (
mounted_fan_out_recovers_then_synthesizes_and_hands_off,production_lsp_negotiates_and_projects_canonical_context,background_refresh_and_reopen_report_only_servable_generations) and is being reworked separately.🤖 Generated with Claude Code
Update (run 35407045249 triage, tip ba881a1): CI's nextest
ciprofile hasflaky-result = "fail", so every fail-then-pass counts as red. Seven more commits, each attributed from that run's logs:13a36df9fetest(code-index):dashboard_progress_does_not_wait_for_the_scheduler_mutexandignored_dependency_waits_for_global_admission_before_publication_gatesampled a no-op reconcile pass spent by a bankedNotifypermit that a settled owner can still hold; new helper burns it first. 0/40 undertaskset -c 0,1each (was 3/25 and 4/25).3e821288bbtest(code-index):legacy_generation_restore_does_not_materialize_its_evidence_segmentproved "does not materialize" with a peak-RSS delta that cannot see the regression (the author's own negative control passes at 0.30x); now asserts the codec's read shape, RSS kept as a loose ceiling. 10/10 under load, injection of a whole-segment read fails in 0.5 s.bf04c93ae7test(transport):background_refresh_and_reopen_report_only_servable_generationspaid its refresh batch at every reopen inside the harness's fixed 20 s publish gate; retire the batch before the reopens and shrink it. 5/5 loaded at 34-47 s (was failing at 66 s), idle 8-9 s. The parked branch-authority mount51402cdf8dwas exonerated for all three journeys it was blamed for.63f23d163dtest(daemon):partial_effect_survives_http_mcp_and_rust_sdk_across_restartasserted success on its firststorage_statusafterinit; the daemon can still be mounting the query authority and says so with a retryable pre-admission problem (retry: after_delay,retry_after_millis: 250). Honour it.read_only_project_binding_refuses_before_scheduler_mutation's 5 s seat wait raised to 60 s.48cc9d5b59fix(sessions): a project ingest pass ended at the first rollout that consumed bytes, never committing the frontier, so every pass re-read every exhausted rollout: n(n+1)/2 admissions, 554 for 33 rollouts. Bound the pass by bytes as the profile-scope loop already does;completed_session_import_immediately_searches_canonical_messageingest 20 s → 5 s.ba881a17a5fix(session-sync): with the ingest fast, the import's history stage reportedsession_history_not_currentfor a current history whose projection had not drained, skipping the projection stage that waits for exactly that. Stage one now gates on history alone; stage two's checks are a superset. Import test 12/12 under load, 14-16 s against a 30 s wall (was 42-45 s, failing).656b5328e7,b5a784cf32test(code-index):concurrent_query_admissions_claim_one_pending_wake_before_worker_coalescing(batch run 35407620833) raced the pass tail: the worker releases the background admission mid-pass and drops its guard beforenote_worker_continuationre-stamps the slot the test just cleared. New helper clears the slot until it survives a quiet window with the single admission held; applied to the five tests whose assertion a foreign stamp inverts. 360/360 under 6-way contention (was 6/48).93afc16effmerge of master after fix: clear remaining master CI test failures #1793: keeps its history-worker continuation and this branch's fencing of the two reconcile tests it also touched (Codex's review on fix: clear remaining master CI test failures #1793 flagged the hole the lane's version closes); its unused helper dropped.e2c8b592cdtest(transport):reset_required_survives_http_mcp_and_rust_sdk_across_restart(master run 35411306897) asserted on the first HTTP reply after a physical restart, which can be the retryable pre-admissionunavailablepublished before the project is open; assert on the first admitted reply.36b0b780c9test(code-index): the dashboard test readVerifyingonce more on run 35412193695; the pending-wake slot is also part ofrefresh_in_flight, so it is now emptied until quiet under the held admission. 60/60 as six concurrent copies on two cores.0d328fa0a9test(code-index):coalesced_publication_failure_preserves_the_scheduler_error_family(master run 35411306897) corrupted the active pointer with no lock while a background text-artifact attach was mid read-modify-write under the store lock; the attach's rename restored a valid pointer. The test now takes the generation-store lock for the corruption. 6-15% failure under 6-way contention became 0/60.d2b57e4ad7test(daemon): the harness asserted instantly that a stopped daemon's socket no longer accepts; the kernel keeps an AF_UNIX listener alive while any forked child still holds the inherited fd. Bounded wait, same refusal message; coversbranch_search_serves_a_committed_generation_behind_dirty_worktree_stateand its sibling.b725b7a0c6test(lcm):preserved_profile_lcm_discovery_converges_without_blocking_retrieval(run 35414007809, both tries) read one window with a single shot while the journey's own contract answers a lagging window with typedstale; all three window reads now loop on that typed staleness as one of them already did. The faster ingest from48cc9d5b59widened the legitimate lag window; reverting the other two candidates changed nothing.d8aaa0fa68fix(hooks):production_codex_hook_ingest_survives_message_search_reopen(run 35414007809) hit a real race: the MCP hook route and the daemon's project catch-up both admit the same Codex rollout with no mutual exclusion, and the loser re-submits committed observation ids with its own provenance, which the store refuses as a provenance collision surfaced as an infrastructure error. Per-(scope, rollout) gate across cursor read and admit;resumednow flows through the Codex progress so the loser reportsexact_duplicateexactly as the Cursor route does sincedb2ee84366. A second defect surfaced while hardening it (a partial generation answeringcomplete_zero) is being fixed separately.e8fe3561edstyle: pre-existingunnecessary_sort_bylint ingraph_analysis_test.rsthat blocked clippy under the test-transport lens.