Fix flaky CI tests: live-clock polling, GitManager timeouts, web test budget#3778
Draft
t3dotgg wants to merge 3 commits into
Draft
Fix flaky CI tests: live-clock polling, GitManager timeouts, web test budget#3778t3dotgg wants to merge 3 commits into
t3dotgg wants to merge 3 commits into
Conversation
CursorAdapter.test.ts polled its mock-agent request log with bare Effect.yieldNow loops, and ProviderRegistry.test.ts polled real child-process probe results with TestClock.adjust + yieldNow. Neither gives the child process or libuv any wall-clock time, so on loaded CI hosts the session/cancel log line or the re-probe spawn landed after the ~40 microtask hops and the tests failed on stale data. Add a shared pollUntil test helper that sleeps on the live clock via TestClock.withLive between attempts and dies loudly with the last polled value on timeout, and use it for all cross-process waits in both test files. Also make readJsonLines skip partially written trailing lines while a child process is still appending. Co-authored-by: Theo Browne <t3dotgg@users.noreply.github.com>
Six cross-repo PR tests carried 12s/20s per-test timeouts from when the suite default was Vitest's 5s. The server suite now sets a 120s budget precisely because these git-subprocess-heavy tests can crawl on loaded CI hosts, so the leftover overrides only shrank the budget and made the tests time out spuriously. Remove them so the tests inherit the configured default. Co-authored-by: Theo Browne <t3dotgg@users.noreply.github.com>
The MessagesTimeline attachment-anchor test hit the 15s web budget in CI while every package's suite competed for the same cores; match the root config's 60s budget. Update the cloud rules to describe the fixed flakes and point future polling waits at pollUntil. Co-authored-by: Theo Browne <t3dotgg@users.noreply.github.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Audited the last ~50 failed CI runs and fixed the recurring cross-branch flakes (as opposed to genuine PR-branch failures):
CursorAdapter.test.ts— "cancels pending ACP approvals…" (7+ CI failures across unrelated branches). The test polled the mock agent's request log with bareEffect.yieldNowloops, which yield zero wall-clock time underit.effect'sTestClock. The mock agent is a separate OS process, so itssession/cancellog append routinely landed after the ~40 microtask hops and the assertion read stale data. A standalone repro of the old helper missed a write arriving just 5ms late, 3/3 times.ProviderRegistry.test.ts— "re-probes when settings change the codex binaryPath" (documented known flake). Same mechanism: the re-probe spawns a real child process whose ENOENT arrives on the live event loop, but the poll loop only didTestClock.adjust+yieldNow.GitManager.test.ts— cross-repo PR tests timing out at 12s (documented known flake). Six tests carried 12s/20s per-test timeouts left over from when the suite default was Vitest's 5s. The server suite now sets a 120s budget precisely because these git-subprocess-heavy tests crawl on loaded CI hosts — the stale overrides only shrank the budget. Removed them.MessagesTimeline.test.tsx— "anchors a sent attachment message…" timed out at exactly the web suite's 15s budget under a full-monorepo parallel test run. Raisedapps/webtest/hook timeouts to match the root config's 60s.Fix
New shared helper
apps/server/src/provider/testUtils/pollUntil.ts: polls an effect until a predicate matches, sleeping on the live clock viaTestClock.withLivebetween attempts (so child processes and libuv make progress while the test stays on the virtual clock), and dies loudly with the last polled value on timeout instead of silently returning stale data. Both flaky server test files now use it for every cross-process wait.readJsonLinesalso tolerates a partially written trailing line while the child is mid-append.Updated
.cursor/rules/cursor-cloud.mdcknown-flaky notes to describe the fixes and point future polling waits atpollUntil.Testing
vp run typecheckvp check --fix(formatter-clean, 0 errors)vp run testinapps/server— full suite, 160 files / 1397 tests passedvp test src/components/chat/MessagesTimeline.test.tsxinapps/webCursorAdaptercancel test ran 8×, andProviderRegistryre-probe test 5×, under 6 concurrent CPU busy-loops on a 4-core VM — all passedwaitForJsonLogMatchmisses a 5ms-late cross-process write (3/3 runs), validating the root causeNote
Fix flaky CI tests by replacing manual polling loops with live-clock
pollUntilutilitypollUntil, a reusable test utility that polls an Effect on the live clock (default 10s timeout, 25ms interval) until a predicate matches, usingTestClock.withLiveto avoid interference withit.effect's test clock.CursorAdapter.test.tsandProviderRegistry.test.tsto replace manualEffect.yieldNowloops withpollUntil, fixing races on out-of-process child process and file I/O timing.readJsonLinesby silently skipping lines that fail to parse, handling mid-write appends from separate processes.GitManager.test.ts, deferring to the framework default.hookTimeoutandtestTimeoutfrom 15s to 60s inapps/web/vite.config.tsto align with root config budget under CI contention.📊 Macroscope summarized c660f28. 3 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.