Skip to content

Assert the warm path does no I/O, not that it takes under 5ms - #14

Merged
cport1 merged 1 commit into
mainfrom
fix/flaky-warm-path-perf-test
Jul 30, 2026
Merged

Assert the warm path does no I/O, not that it takes under 5ms#14
cport1 merged 1 commit into
mainfrom
fix/flaky-warm-path-perf-test

Conversation

@cport1

@cport1 cport1 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Blocking #13, and unrelated to it.

What failed

● AgentVerifier — Web Bot Auth › does not fetch the network on the warm path and verifies in <5ms p95
    expect(received).toBeLessThan(expected)
    Expected: < 5
    Received:   5.920060000000376

The test never checked its own headline claim

The name says "does not fetch the network on the warm path and verifies in <5ms p95". The body asserted only the second half — it never checked whether anything was fetched.

What it did assert was a wall-clock p95 over 200 Ed25519 verifications on whatever hardware happened to run it. A shared CI runner measures its neighbours as much as the code, so this was going to fail on someone eventually. "Re-run it" isn't a fix.

What it asserts now

That a warm verify touches no network — the thing that makes signature verification usable on a request path, and unlike a duration it's exact. fetchImpl is injectable, so the calls are simply counted:

await verifier.verify(req);
expect(fetches).toBe(1);        // one cold fetch populates the directory cache
// ...50 warm verifies...
expect(fetches).toBe(cold);     // and they add none

Timing is not asserted at all. A number that fails for reasons unrelated to the code is worse than no number, because the failure teaches you to ignore it.

Testing

turbo test --force (cache disabled) — 169 tests green across the monorepo.

…r 5ms

The test named "does not fetch the network on the warm path and verifies in
<5ms p95" asserted only the second half. It never checked whether anything was
fetched — the claim in its own name, and the one that matters.

What it did assert was a wall-clock p95 over 200 Ed25519 verifications, on
whatever hardware happened to run it. That failed CI at 5.92ms against a bound
of 5. Nothing was wrong: a shared runner measures its neighbours as much as the
code under test, so the assertion was going to fail on someone eventually, and
"re-run it" is not a fix.

The invariant worth protecting is that a warm verify touches no network. That is
what makes signature verification usable on a request path, and unlike a
duration it is exact — fetchImpl is injectable, so the calls can simply be
counted: one cold fetch to populate the directory cache, then fifty verifies
that add none.

Timing is not asserted at all now. A number that fails for reasons unrelated to
the code is worse than no number, because the failure teaches you to ignore it.
@cport1
cport1 merged commit 9f76c07 into main Jul 30, 2026
2 checks passed
@cport1
cport1 deleted the fix/flaky-warm-path-perf-test branch July 30, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant