Description
TanStack AI version
main (8731d75)
Framework/Library version
CI only (.github/workflows/pr.yml, Coverage job)
Describe the bug and the steps to reproduce it
The PR Coverage job fails at random on PRs that change @tanstack/ai. No coverage metric drops. One package's test:coverage task fails, and it is a different package each time. A re-run passes.
Seen so far:
| Run | PR | Failed task |
|---|---|---|
| 40271697755 | #1894 | @tanstack/ai-harness:test:coverage (passed on re-run) |
| 40270604983 | #1887 | @tanstack/ai-harness:test:coverage |
| 40277024581 | #1894 | @tanstack/ai-react:test:coverage |
The same tests pass locally:
packages/ai-harness:vitest run, 2469 passed.packages/ai-react:vitest runandnx run @tanstack/ai-react:test:coverage --skip-nx-cache, 251 passed.
Likely cause: a change to @tanstack/ai makes the job measure about 76 packages, twice (PR head and merge-base). nx run-many runs them in parallel on one runner. Tests that depend on timing then time out under the load.
Second problem: the job log does not show the output of the failed task. pr.yml:76 and pr.yml:105 run nx run-many --outputStyle=static, and the log only has the Nx summary line ✖ @tanstack/ai-harness:test:coverage. So nobody can name the flaky test from CI.
Fix
- Make the Coverage job print the full output of a failed task, so the failing test name and error are in the job log.
- Find the timing-sensitive tests in
ai-harnessandai-react. To do this, run their coverage under CPU load locally, or use the log from step 1. Then fix them: remove real-time waits, use fake timers, or wait for events instead of a fixed time. - If step 2 does not remove all failures, cap the parallel tasks for the coverage step (
--parallel=N). Do not add retries that hide failures.
Your Minimal, Reproducible Example - (Sandbox Highly Recommended)
CI runs listed above. No local reproduction yet.
Do you intend to try to help solve this bug with your own PR?
Yes, I am also opening a PR to fix this issue.
Activity
test(ai-react, ai-harness): remove the two timing races behind the Co…