Skip to content

[Tests] Fix flaky outdated rate-limit cache test - #8705

Merged
Suleimanlatrsh merged 1 commit into
mainfrom
tests-maintenance-36650442458
Sep 30, 2026
Merged

Suleimanlatrsh merged 1 commit into
mainfrom
tests-maintenance-36650442458

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

@shopify/cli-kit src/private/node/conf-store.test.ts > runWithRateLimit > runs the task as usual when the cache is populated but outdated fails intermittently on the Main tests workflow:

AssertionError: expected false to be true // Object.is equality
 ❯ src/private/node/conf-store.test.ts:581:19

Representative job: Node 26.1.0 in ubuntu-latest (2026-09-29, stable/4.8). The same run's other 561 test files passed, and the immediately preceding run on the same branch passed, so the failure is intermittent rather than a real regression. The test block is byte-identical on main and stable/4.8, so the defect is present on main.

The test recorded rate-limit occurrences against the live clock, then advanced time with vi.setSystemTime(vi.getRealSystemTime() + 1000) — a 1000 ms jump measured from the real clock rather than from the timestamps just written. runWithRateLimit keeps occurrences with occurrence >= windowStart, an inclusive boundary. When under 1 ms of real time elapsed during setup, the recorded occurrences landed exactly on windowStart, survived the sweep, and the task stayed throttled — returning false where the test expects true.

This is the same class of flake that #7290 fixed for the two sibling tests in this describe block; this third test was left on the real clock.

WHAT is this pull request doing?

Freezes the clock with vi.useFakeTimers() so the occurrences and the subsequent advance share one time base, and advances with vi.advanceTimersByTime(timeIntervalToMilliseconds(timeout) + 1) so the jump is derived from the window under test and lands strictly past its inclusive start.

Coverage is unchanged — the test still exercises the real runWithRateLimit against a LocalStorage in a real temporary directory and still asserts both the return value and that the task ran. The existing afterEach already restores real timers, matching the two sibling tests.

I verified the fix is load-bearing rather than merely passing: with the clock pinned so zero real milliseconds elapse during setup, the original form reproduces the CI assertion exactly (got=false), while the new form returns true.

Validation on Linux / Node 22.20.1: the runWithRateLimit block and the full 32-test file pass, five consecutive --sequence.shuffle runs of the file pass, and a shuffled run alongside the neighbouring clock- and store-dependent suites (sleep-with-backoff, local-storage, fs) passes at 84 passed / 1 skipped. eslint and tsc --noEmit are clean. The originally failing platform was ubuntu Node 26.1.0; I confirmed the cause and the fix locally but not on that exact matrix entry, so final confirmation comes from CI.

The other failures in the seven-day review window (2026-09-23 to 2026-09-30) are not addressed here. Eight of the ten failed jobs were version-command subprocess timeouts, which are already covered by open PR #8669, and one was an app-doctor npm-audit timeout on stable/4.8.

How to test your changes?

CI

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

The test advanced the clock relative to the real system time while the
rate-limit occurrences were recorded against it, so the occurrences could
land exactly on the inclusive window start and stay throttled.

Freeze the clock and advance strictly past the window instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added the no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. label Sep 30, 2026
@Suleimanlatrsh
Suleimanlatrsh marked this pull request as ready for review September 30, 2026 18:30
@Suleimanlatrsh
Suleimanlatrsh requested a review from a team as a code owner September 30, 2026 18:30
@Suleimanlatrsh
Suleimanlatrsh added this pull request to the merge queue Sep 30, 2026
Merged via the queue into main with commit 8e2a7f2 Sep 30, 2026
32 of 55 checks passed
@Suleimanlatrsh
Suleimanlatrsh deleted the tests-maintenance-36650442458 branch September 30, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant