Skip to content

Fix flaky CI tests: live-clock polling, GitManager timeouts, web test budget#3778

Draft
t3dotgg wants to merge 3 commits into
mainfrom
cursor/fix-flaky-tests-316d
Draft

Fix flaky CI tests: live-clock polling, GitManager timeouts, web test budget#3778
t3dotgg wants to merge 3 commits into
mainfrom
cursor/fix-flaky-tests-316d

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

Audited the last ~50 failed CI runs and fixed the recurring cross-branch flakes (as opposed to genuine PR-branch failures):

  • CursorAdapter.test.ts — "cancels pending ACP approvals…" (7+ CI failures across unrelated branches). The test polled the mock agent's request log with bare Effect.yieldNow loops, which yield zero wall-clock time under it.effect's TestClock. The mock agent is a separate OS process, so its session/cancel log append routinely landed after the ~40 microtask hops and the assertion read stale data. A standalone repro of the old helper missed a write arriving just 5ms late, 3/3 times.
  • ProviderRegistry.test.ts — "re-probes when settings change the codex binaryPath" (documented known flake). Same mechanism: the re-probe spawns a real child process whose ENOENT arrives on the live event loop, but the poll loop only did TestClock.adjust + yieldNow.
  • GitManager.test.ts — cross-repo PR tests timing out at 12s (documented known flake). Six tests carried 12s/20s per-test timeouts left over from when the suite default was Vitest's 5s. The server suite now sets a 120s budget precisely because these git-subprocess-heavy tests crawl on loaded CI hosts — the stale overrides only shrank the budget. Removed them.
  • MessagesTimeline.test.tsx — "anchors a sent attachment message…" timed out at exactly the web suite's 15s budget under a full-monorepo parallel test run. Raised apps/web test/hook timeouts to match the root config's 60s.

Fix

New shared helper apps/server/src/provider/testUtils/pollUntil.ts: polls an effect until a predicate matches, sleeping on the live clock via TestClock.withLive between attempts (so child processes and libuv make progress while the test stays on the virtual clock), and dies loudly with the last polled value on timeout instead of silently returning stale data. Both flaky server test files now use it for every cross-process wait. readJsonLines also tolerates a partially written trailing line while the child is mid-append.

Updated .cursor/rules/cursor-cloud.mdc known-flaky notes to describe the fixes and point future polling waits at pollUntil.

Testing

  • vp run typecheck
  • vp check --fix (formatter-clean, 0 errors)
  • vp run test in apps/server — full suite, 160 files / 1397 tests passed
  • vp test src/components/chat/MessagesTimeline.test.tsx in apps/web
  • ✅ Stress test: previously flaky CursorAdapter cancel test ran 8×, and ProviderRegistry re-probe test 5×, under 6 concurrent CPU busy-loops on a 4-core VM — all passed
  • ✅ Standalone repro confirmed the old waitForJsonLogMatch misses a 5ms-late cross-process write (3/3 runs), validating the root cause
Open in Web Open in Cursor 

Note

Fix flaky CI tests by replacing manual polling loops with live-clock pollUntil utility

  • Adds pollUntil, a reusable test utility that polls an Effect on the live clock (default 10s timeout, 25ms interval) until a predicate matches, using TestClock.withLive to avoid interference with it.effect's test clock.
  • Updates CursorAdapter.test.ts and ProviderRegistry.test.ts to replace manual Effect.yieldNow loops with pollUntil, fixing races on out-of-process child process and file I/O timing.
  • Fixes partial JSON line errors in readJsonLines by silently skipping lines that fail to parse, handling mid-write appends from separate processes.
  • Removes explicit per-test timeout overrides in GitManager.test.ts, deferring to the framework default.
  • Increases Vitest hookTimeout and testTimeout from 15s to 60s in apps/web/vite.config.ts to align with root config budget under CI contention.
📊 Macroscope summarized c660f28. 3 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

cursoragent and others added 3 commits July 7, 2026 21:04
CursorAdapter.test.ts polled its mock-agent request log with bare
Effect.yieldNow loops, and ProviderRegistry.test.ts polled real
child-process probe results with TestClock.adjust + yieldNow. Neither
gives the child process or libuv any wall-clock time, so on loaded CI
hosts the session/cancel log line or the re-probe spawn landed after
the ~40 microtask hops and the tests failed on stale data.

Add a shared pollUntil test helper that sleeps on the live clock via
TestClock.withLive between attempts and dies loudly with the last
polled value on timeout, and use it for all cross-process waits in
both test files. Also make readJsonLines skip partially written
trailing lines while a child process is still appending.

Co-authored-by: Theo Browne <t3dotgg@users.noreply.github.com>
Six cross-repo PR tests carried 12s/20s per-test timeouts from when
the suite default was Vitest's 5s. The server suite now sets a 120s
budget precisely because these git-subprocess-heavy tests can crawl on
loaded CI hosts, so the leftover overrides only shrank the budget and
made the tests time out spuriously. Remove them so the tests inherit
the configured default.

Co-authored-by: Theo Browne <t3dotgg@users.noreply.github.com>
The MessagesTimeline attachment-anchor test hit the 15s web budget in
CI while every package's suite competed for the same cores; match the
root config's 60s budget. Update the cloud rules to describe the fixed
flakes and point future polling waits at pollUntil.

Co-authored-by: Theo Browne <t3dotgg@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 224a320d-61de-40fc-9990-bdd5c0d8f58e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch cursor/fix-flaky-tests-316d

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants