Skip to content

fix: land the three root-caused flaky-test fixes (identity, parking, relay) - #1160

Open
Aditya Shantanu (aditya-shantanu) wants to merge 4 commits into
agent-substrate:mainfrom
aditya-shantanu:fix/flaky-tests-omnibus
Open

fix: land the three root-caused flaky-test fixes (identity, parking, relay)#1160
Aditya Shantanu (aditya-shantanu) wants to merge 4 commits into
agent-substrate:mainfrom
aditya-shantanu:fix/flaky-tests-omnibus

Conversation

@aditya-shantanu

@aditya-shantanu Aditya Shantanu (aditya-shantanu) commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Fixes #675, fixes #1100, fixes #1146; addresses the CI flake in #1106.

Why this PR

Flaky tests are the single biggest drag on this repo's velocity right now: the three flakes fixed here account for the majority of red CI runs over the last 7 days (identity: 35 failures, parking: 32, relay: 9 — from the flake dashboard's cross-PR analysis of ~630 runs). Every red run costs a contributor a rebase-and-rerun cycle and costs reviewers signal. This PR consolidates the three root-caused, in-flight fixes into one change to get CI green now and unblock the community — the goal is velocity, not authorship.

Credit where it's due

All three fixes were root-caused and written by others; this PR adopts them onto latest main with their tests, unchanged in substance. Each commit carries a Co-authored-by trailer:

Commit Original PR Author Root cause
e2e: give each probe fixture its own worker pool #1147 NekoPunch (@orangeCatDeveloper) identity/egressmitm/imagevolume suites share one workload: probe pool label; cross-suite selection under concurrent suite processes dials workers that are not there
atenet: never cancel an in-flight resume at the park budget #991 Omer Yahud (@omeryahud) the park budget doubled as the ResumeActor RPC deadline; a mid-restore cancel strands a RESUMING actor on a live worker
atunnel: close the relay's both ends before returning #1101 NekoPunch (@orangeCatDeveloper) the relay closed both ends from a context.AfterFunc goroutine the test never waits for

Steven Jin (@Stevenjin8)'s #1107 correctly diagnosed the ateom readiness race in #1106; the control-plane readiness gap it targets remains real and open — this PR only removes the e2e-fixture contention that makes it fire constantly in CI.

If maintainers prefer to land the original PRs individually instead, closing this one is completely fine — the point is that the fixes land somewhere, soon.

Evidence the flakes are actually fixed

TestRelayIngressCancellationClosesBothSides (unit, -race):

  • Unpatched main, -count=3000: 83 failures (2.8%) — matches the 2.9% observed across 308 CI runs this week
  • This branch, -count=10000: 0 failures

TestRequestParking (park-budget cancellation):

TestActorIdentity_AfterRestore_IsOwnID_NotGolden (probe pool isolation):

  • Not reproducible outside CI (needs concurrent suite processes on a contended kind node), so verified statically: ${FIXTURE_SUFFIX} is always -<suite> (internal/e2e/sandbox.go:189,201 — never empty), probe-sized already uses its own label, and no other manifest or selector references workload: probe. e2e: give each probe fixture its own worker pool #1147's CI data shows all three failure signatures (missing ateom.sock, runsc restore killed, router 502/503) trace to cross-suite pool sharing; per-suite labels make the selector suite-local by construction
  • The definitive check is this PR's own CI plus the flake dashboard's 7-day window after merge — I will report the post-merge rates on Race condition in Ateom startup and TestActorIdentity_AfterRestore_IsOwnID_NotGolden Test Flake #1106

Also run: go build ./..., go vet and the full -race suites of both touched packages — all green.

What this PR deliberately does NOT fix

TestActorEgressHTTPS (#1050, 4.6% this week, below the 5% flake threshold) has no root-caused fix yet — the 503 upstream connect error path needs investigation in a live cluster. #1103 (NekoPunch (@orangeCatDeveloper)) tightens the related TestActorArbitraryPortAccess assertion so those 503s stop passing silently; it should land after #1050's cause is fixed, or it converts hidden flakiness into visible red.

Update (post-CI investigation)

The first e2e runs failed on TestRequestParking/ParkThenServed (micro-VM lane). Investigation showed this is the pre-existing dominant mode of #675 — identical failures in main-era runs 32305728993 / 32397291519 / 32487958077 — not a regression: on micro-VM, SuspendActor returns before the snapshot upload completes, so the worker legitimately isn't free within the 5s park budget and the router's 503 is correct behavior. #991 fixes the other (mid-restore cancellation/stranding) mode. Commit d637690 makes the subtest retry while the worker is still freeing; a stranded worker still fails every attempt, so the regression stays pinned.

Additional validation:

  • CI e2e-test now passes both lanes (run 32754691017)
  • Local kind cluster built from this branch: parking suite 10/10 consecutive passes; identity + egressmitm + imagevolume run concurrently (the exact contention behind the identity flake) × 3 iterations — 9/9 suite passes

Adopted from agent-substrate#1147 by @orangeCatDeveloper to unblock CI velocity.
Fixes the TestActorIdentity_AfterRestore_IsOwnID_NotGolden flake
(26 of the last 80 failed runs): identity, egressmitm and imagevolume
suites shared one workload:probe pool label, so selection crossed
suite boundaries and a saturated pool dialed workers that were not
there.

Co-authored-by: NekoPunch <engineer.jyao@gmail.com>
Adopted from agent-substrate#991 by @omeryahud to unblock CI velocity. Fixes the
TestRequestParking flake (agent-substrate#675): the park budget doubled as the
ResumeActor RPC deadline, cancelling restores mid-flight and
stranding RESUMING actors on live workers.

Co-authored-by: Omer Yahud <oyahud@nvidia.com>
Adopted from agent-substrate#1101 by @orangeCatDeveloper to unblock CI velocity.
Fixes TestRelayIngressCancellationClosesBothSides (agent-substrate#1100): the close
ran on a context.AfterFunc goroutine the test never waited for.

Co-authored-by: NekoPunch <engineer.jyao@gmail.com>
SuspendActor returns before the suspend completes, and on the micro-VM
class the snapshot upload routinely outlives the 5s park budget under
CI contention — the router's budget-exhausted 503 is then correct
behavior, not a failure. Retry the request (bounded at 3 attempts):
each attempt parks anew and the completed suspend lets one resume the
actor. A stranded worker (agent-substrate#675's root cause) fails all attempts, so
the regression stays pinned.

This was the dominant historical failure mode of the agent-substrate#675 flake
(identical parking_test.go:113 503s in pre-PR runs 32305728993,
32397291519, 32487958077) — distinct from the mid-restore
cancellation that agent-substrate#991 fixes.
@aditya-shantanu

Copy link
Copy Markdown
Collaborator Author

The e2e failure here is worth a close look — it is not a regression from this PR, and it exposed that #991 fixes only part of the #675 flake:

What failed: TestRequestParking/ParkThenServed (micro-VM lane): parking_test.go:113: 503 "no free workers available", want 200, with BudgetExhaustion failing as a cascade of the shared subtest state.

Why it's pre-existing: the identical failure (same line, same 503 body, same lane) appears in main-era runs 32305728993, 32397291519 and 32487958077 — this is the dominant mode of the 13.5% micro-VM flake. Mechanism: SuspendActor(A) returns before the suspend completes, and on micro-VM the guest snapshot upload routinely outlives the 5s park budget under CI contention. The router's budget-exhausted 503 is then correct product behavior; the test's assumption that the worker always frees within one budget is the bug. #991 fixes the other (mid-restore cancellation → stranded worker) mode.

Fix pushed (d637690): ParkThenServed now retries the parked request (bounded at 3 attempts) while the worker is still freeing — each attempt parks anew, and a genuinely stranded worker (#675's root cause) still fails every attempt plus the follow-up assertions, so the regression stays pinned.

Validation: 10/10 consecutive suite passes against a local kind cluster built from this branch (gVisor lane; micro-VM needs KVM, so this PR's CI is the venue for that lane). The relay fix separately holds at 0 failures over 10,000 -race iterations vs 83/3,000 on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant