[Tests] Fix flaky outdated rate-limit cache test - #8705
Merged
Merged
Conversation
The test advanced the clock relative to the real system time while the rate-limit occurrences were recorded against it, so the occurrences could land exactly on the inclusive window start and stay throttled. Freeze the clock and advance strictly past the window instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Suleimanlatrsh
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY are these changes introduced?
@shopify/cli-kit src/private/node/conf-store.test.ts > runWithRateLimit > runs the task as usual when the cache is populated but outdatedfails intermittently on the Main tests workflow:Representative job: Node 26.1.0 in ubuntu-latest (2026-09-29,
stable/4.8). The same run's other 561 test files passed, and the immediately preceding run on the same branch passed, so the failure is intermittent rather than a real regression. The test block is byte-identical onmainandstable/4.8, so the defect is present onmain.The test recorded rate-limit occurrences against the live clock, then advanced time with
vi.setSystemTime(vi.getRealSystemTime() + 1000)— a 1000 ms jump measured from the real clock rather than from the timestamps just written.runWithRateLimitkeeps occurrences withoccurrence >= windowStart, an inclusive boundary. When under 1 ms of real time elapsed during setup, the recorded occurrences landed exactly onwindowStart, survived the sweep, and the task stayed throttled — returningfalsewhere the test expectstrue.This is the same class of flake that #7290 fixed for the two sibling tests in this
describeblock; this third test was left on the real clock.WHAT is this pull request doing?
Freezes the clock with
vi.useFakeTimers()so the occurrences and the subsequent advance share one time base, and advances withvi.advanceTimersByTime(timeIntervalToMilliseconds(timeout) + 1)so the jump is derived from the window under test and lands strictly past its inclusive start.Coverage is unchanged — the test still exercises the real
runWithRateLimitagainst aLocalStoragein a real temporary directory and still asserts both the return value and that the task ran. The existingafterEachalready restores real timers, matching the two sibling tests.I verified the fix is load-bearing rather than merely passing: with the clock pinned so zero real milliseconds elapse during setup, the original form reproduces the CI assertion exactly (
got=false), while the new form returnstrue.Validation on Linux / Node 22.20.1: the
runWithRateLimitblock and the full 32-test file pass, five consecutive--sequence.shuffleruns of the file pass, and a shuffled run alongside the neighbouring clock- and store-dependent suites (sleep-with-backoff,local-storage,fs) passes at 84 passed / 1 skipped.eslintandtsc --noEmitare clean. The originally failing platform was ubuntu Node 26.1.0; I confirmed the cause and the fix locally but not on that exact matrix entry, so final confirmation comes from CI.The other failures in the seven-day review window (2026-09-23 to 2026-09-30) are not addressed here. Eight of the ten failed jobs were version-command subprocess timeouts, which are already covered by open PR #8669, and one was an
app-doctornpm-audit timeout onstable/4.8.How to test your changes?
CI
Checklist
patchfor bug fixes ·minorfor new features ·majorfor breaking changes) and added a changeset withpnpm changeset add