v1.87.5.0 perf: remove idle waits from tests and CI planning

[garrytan]

Remove completed-work waits from test fixtures and keep paid-lane coordination independent of the execution image. Draft pending correction of the UI-scope eval. The GitHub free suite now passes; earlier local failures and remaining verification limits are recorded below rather than hidden.

Test execution

• Wake plan-count observers on output/exit, settle new output bursts for 250ms before reading split redraws, and preserve the 2s silent fallback, real CLI's 8s startup grace, input debounces, and permission guards. Only three synthetic CLI suites opt into explicit readiness markers.
• Clear losing PTY/Node deadlines, recognize already-exited daemon fixtures, and await watchdog shutdown with a bounded completion signal. Clean up cookie-picker fixture state between suites.
• Preserve the CSO Git-pointer race's rejection assertion while injecting the swap at the first bounded read. Add regressions for output/exit/close, missing readiness, timer cancellation, continuous redraws, and partial terminal writes.

CI coordination

Gate and periodic planners now run on pinned Bun without an eval container or dependency installation, in parallel with image lookup. Executors still require both successful prerequisites and retain their images, dependency restore, and builds. Reports also run dependency-free and retain fail-closed reconciliation. Fork/Dependabot restrictions, read-only planning/report permissions, full-history diff selection, and pinned actions are preserved.

The new tests compare dependency-free manifests and exercise successful, missing, failed, and corrupt reports. The first GitHub run completed the planner in 19 seconds, compared with total planner durations of 59/82/81 seconds in runs 35055737527, 35021760588, and 35019393408. Planning started while image lookup was still running. This is one observed planner result; end-to-end workflow savings are not yet measured.

Measured file-level performance

Same Linux machine, Bun 1.4.0. Final values include output-burst settling. Polyfill/daemon values are three-run medians on both sides; PTY baselines are single observed runs and final values are three-run medians. These are individual-file timings, not whole-suite wall-time savings.

Test file	Before	Final
browse/test/bun-polyfill.test.ts	17.30s	1.29s
design/test/daemon-discovery.test.ts	17.78s	11.99s
test/plan-count-file-permission.test.ts	97.05s	30.92s
test/plan-count-checkbox.test.ts	40.9s	15.46s
test/plan-count-completion.test.ts	36.6s	13.91s

All 41 selected PTY tests passed on each of three repetitions; the permission suite retains all 115 assertions.

Verification and blockers

• GitHub CI finished: the free suite passed all six shards on the first attempt (852-file census, approximately 234s), without invoking its configured flaky retry. Windows free tests, Windows setup, macOS gate, and CSO platform/integration checks also passed. Only the UI-scope eval and its dependent reconciliation check failed.
• The failed UI-scope eval captures a scope-selection menu whose branch-diff option says “no UI scope”; its broad phrase regex treats that option as a terminal early exit. The captured session also reports no plan target supplied. Fixing this requires reliable fixture delivery and proof of a real design-review question, not a weaker assertion. The report correctly accounts for all six slices and propagates this one failed eval.
• Final helper/consumer checks: 442 passed / 3,024 assertions across 27 files. The split-redraw regression plus unchanged exact-input integration suite: 12 passed / 662 assertions. Release/generated-skill checks: 376 passed.
• Actionlint and git diff --check pass. Complete scanner reports were compared directly after the repository wrapper truncated its JSON; only generic duplicate-mock-setup advisories were added. In-host review and an independent Claude task review, including the burst-settling follow-up, found no remaining introduced defects.
• Earlier local strict run: 14,632 passed, 55 skipped, 7 failed across 852 files, six workers, 252.5s. Used the pinned published CI image in a network-isolated container, with working Chromium sandboxing and the Docker CLI/Python venv prerequisites supplied locally. This is not a comparable before/after whole-suite benchmark. No paid evals were run locally.
• Three cold-setup migration failures also reproduce on untouched base a6b3a57: the docs resolver imports SNAPSHOT_FLAGS from snapshot.ts, which imports diff before dependencies are installed.
• Two URL-helper failures and one artifacts-init protocol failure accompany the existing shell-pipeline race. The untouched base's URL helper exited 141 with empty output in 17/300 control invocations.
• One additional multiline safety-hook test failed in the full run. It did not reproduce in 150 isolated baseline hook invocations; its cause is not yet established. This is included in the seven failures even though the runner's per-file epilogue omitted the multiline test name.

No assertions, safety checks, or failing tests were weakened or skipped to obtain these results.

[trunk-io]

Merging to main in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[capy-ai]

CI update from run 35247857229:

• Planner job: 19 seconds, versus 59/82/81 seconds in the three recorded historical runs. It started while image lookup was still running, emitted its manifest successfully, and did not initialize an eval container. This is one observed planner result, not a whole-workflow speedup measurement.
• Eval slice 6 failed in the unchanged test/skill-e2e-plan-design-with-ui.test.ts. The captured screen is a scope-selection question, not a completed no-UI exit. Its option describing the current branch says “likely little or no UI scope,” which matches the test's broad NO_UI_PHRASE regex and triggers the failure. The screen also says no plan target was supplied; this test sends its fixture as a delayed follow-up after the slash command rather than preloading it. A valid fix must establish fixture delivery and wait for a real design-review question, not merely relax the phrase assertion.
• This test does not use the new output-driven runPlanSkillCounting loop; its captured evidence is taken before close(). Other checks remain in progress. The PR stays draft pending the outstanding scope decision and full verification.

[github-actions]

E2E Evals: ❌ FAIL

3/3 tests passed | $.82 total cost | reconcile exit: 1

Shard	Result	Status	Cost
e2e-design/skill-e2e-design	3/3	✅	$0.82

Fail-closed reconciliation

[test:paid] report: 6/6 slices, 12 planned shards, tier=gate
  slice 1  passed              0s  test/llm-judge-recommendation.test.ts
  slice 1  passed             33s  test/skill-e2e-plan-devex-finding-floor.test.ts
  slice 2  passed            171s  test/skill-e2e-design.test.ts
  slice 2  passed            157s  test/skill-e2e-plan-devex-plan-mode.test.ts
  slice 3  passed             65s  test/skill-e2e-office-hours-auto-mode.test.ts
  slice 3  passed            192s  test/skill-e2e-plan-mode-no-op.test.ts
  slice 4  passed             41s  test/skill-e2e-plan-ceo-finding-floor.test.ts
  slice 4  passed              0s  test/skill-llm-eval-spec.test.ts
  slice 5  passed             91s  test/skill-e2e-plan-ceo-plan-mode.test.ts
  slice 5  passed              0s  test/skill-llm-eval.test.ts
  slice 6  failed             58s  test/skill-e2e-plan-design-with-ui.test.ts
  slice 6  passed              0s  test/skill-routing-e2e.test.ts

---

Sliced lane: diff-selected gate census via scripts/test-paid-shards.ts (planner → 6 executors → fail-closed report)