fix: land the three root-caused flaky-test fixes (identity, parking, relay) - #1160
Conversation
Adopted from agent-substrate#1147 by @orangeCatDeveloper to unblock CI velocity. Fixes the TestActorIdentity_AfterRestore_IsOwnID_NotGolden flake (26 of the last 80 failed runs): identity, egressmitm and imagevolume suites shared one workload:probe pool label, so selection crossed suite boundaries and a saturated pool dialed workers that were not there. Co-authored-by: NekoPunch <engineer.jyao@gmail.com>
Adopted from agent-substrate#991 by @omeryahud to unblock CI velocity. Fixes the TestRequestParking flake (agent-substrate#675): the park budget doubled as the ResumeActor RPC deadline, cancelling restores mid-flight and stranding RESUMING actors on live workers. Co-authored-by: Omer Yahud <oyahud@nvidia.com>
Adopted from agent-substrate#1101 by @orangeCatDeveloper to unblock CI velocity. Fixes TestRelayIngressCancellationClosesBothSides (agent-substrate#1100): the close ran on a context.AfterFunc goroutine the test never waited for. Co-authored-by: NekoPunch <engineer.jyao@gmail.com>
3aa4233 to
8a84e26
Compare
SuspendActor returns before the suspend completes, and on the micro-VM class the snapshot upload routinely outlives the 5s park budget under CI contention — the router's budget-exhausted 503 is then correct behavior, not a failure. Retry the request (bounded at 3 attempts): each attempt parks anew and the completed suspend lets one resume the actor. A stranded worker (agent-substrate#675's root cause) fails all attempts, so the regression stays pinned. This was the dominant historical failure mode of the agent-substrate#675 flake (identical parking_test.go:113 503s in pre-PR runs 32305728993, 32397291519, 32487958077) — distinct from the mid-restore cancellation that agent-substrate#991 fixes.
|
The e2e failure here is worth a close look — it is not a regression from this PR, and it exposed that #991 fixes only part of the #675 flake: What failed: Why it's pre-existing: the identical failure (same line, same 503 body, same lane) appears in main-era runs 32305728993, 32397291519 and 32487958077 — this is the dominant mode of the 13.5% micro-VM flake. Mechanism: Fix pushed (d637690): Validation: 10/10 consecutive suite passes against a local kind cluster built from this branch (gVisor lane; micro-VM needs KVM, so this PR's CI is the venue for that lane). The relay fix separately holds at 0 failures over 10,000 |
Fixes #675, fixes #1100, fixes #1146; addresses the CI flake in #1106.
Why this PR
Flaky tests are the single biggest drag on this repo's velocity right now: the three flakes fixed here account for the majority of red CI runs over the last 7 days (identity: 35 failures, parking: 32, relay: 9 — from the flake dashboard's cross-PR analysis of ~630 runs). Every red run costs a contributor a rebase-and-rerun cycle and costs reviewers signal. This PR consolidates the three root-caused, in-flight fixes into one change to get CI green now and unblock the community — the goal is velocity, not authorship.
Credit where it's due
All three fixes were root-caused and written by others; this PR adopts them onto latest main with their tests, unchanged in substance. Each commit carries a
Co-authored-bytrailer:workload: probepool label; cross-suite selection under concurrent suite processes dials workers that are not therecontext.AfterFuncgoroutine the test never waits forSteven Jin (@Stevenjin8)'s #1107 correctly diagnosed the ateom readiness race in #1106; the control-plane readiness gap it targets remains real and open — this PR only removes the e2e-fixture contention that makes it fire constantly in CI.
If maintainers prefer to land the original PRs individually instead, closing this one is completely fine — the point is that the fixes land somewhere, soon.
Evidence the flakes are actually fixed
TestRelayIngressCancellationClosesBothSides (unit,
-race):-count=3000: 83 failures (2.8%) — matches the 2.9% observed across 308 CI runs this week-count=10000: 0 failuresTestRequestParking (park-budget cancellation):
InFlightAttemptRunsToCompletionandLateRetryableErrorIsBudgetExhaustionunit tests (from atenet: never cancel an in-flight resume at the park budget #991) encode the exact failure mode from E2E test flaky: TestRequestParking #675 and pass undergo test -race -count=100 ./cmd/atenet/internal/router/ingress/TestActorIdentity_AfterRestore_IsOwnID_NotGolden (probe pool isolation):
${FIXTURE_SUFFIX}is always-<suite>(internal/e2e/sandbox.go:189,201 — never empty),probe-sizedalready uses its own label, and no other manifest or selector referencesworkload: probe. e2e: give each probe fixture its own worker pool #1147's CI data shows all three failure signatures (missingateom.sock,runsc restorekilled, router 502/503) trace to cross-suite pool sharing; per-suite labels make the selector suite-local by constructionAlso run:
go build ./...,go vetand the full-racesuites of both touched packages — all green.What this PR deliberately does NOT fix
TestActorEgressHTTPS(#1050, 4.6% this week, below the 5% flake threshold) has no root-caused fix yet — the 503upstream connect errorpath needs investigation in a live cluster. #1103 (NekoPunch (@orangeCatDeveloper)) tightens the relatedTestActorArbitraryPortAccessassertion so those 503s stop passing silently; it should land after #1050's cause is fixed, or it converts hidden flakiness into visible red.Update (post-CI investigation)
The first e2e runs failed on
TestRequestParking/ParkThenServed(micro-VM lane). Investigation showed this is the pre-existing dominant mode of #675 — identical failures in main-era runs 32305728993 / 32397291519 / 32487958077 — not a regression: on micro-VM,SuspendActorreturns before the snapshot upload completes, so the worker legitimately isn't free within the 5s park budget and the router's 503 is correct behavior. #991 fixes the other (mid-restore cancellation/stranding) mode. Commit d637690 makes the subtest retry while the worker is still freeing; a stranded worker still fails every attempt, so the regression stays pinned.Additional validation: