Skip to content

CI: balance the two slowest fixture parts and keep repair-flake evidence - #759

Merged
rmcdaniel merged 7 commits into
mainfrom
rust-325-ci-balance-parts
Oct 25, 2026
Merged

rmcdaniel merged 7 commits into
mainfrom
rust-325-ci-balance-parts

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

CI: balance the two slowest Shared Server fixtures parts, and keep repair-flake evidence.

Why. After #757, the slowest Shared Server fixtures part is mysql-execution (existing-php-mysql-matrix, native-tests): 17–18 min across three runs (40162397220, 40169120328, 40173583253). Of that, 13 min is the native execution suite at two test threads (job 121097162404, 06:38:04 → 06:51:03). The same suite takes 13 min in total on mysql-8.0 and MariaDB, and 12–13 min on PostgreSQL. Every other part takes 10–13 min, except sdk-directions on the same label (16 min).

What changes.

  • That label gets one more matrix combination, native-tests-b, on its own runner. native-tests keeps the MySQL database suite and TLS checks, and runs the first half of the execution suite. native-tests-b runs the second half.
  • The halves come from inside the container command: cargo test --test execution -- --list, sorted bytewise, dealt round-robin with awk. Every listed test runs in exactly one half, and a new test joins one automatically. An empty selection fails the part.
  • Every other label deals one shard, the whole suite, exactly as before.
  • The selection is inline, so the Docker isolation policy still finds no repository script inside a container command. That policy refused my first version, which called a helper script.
  • The cost is one more standard runner per run.

Expected effect of this first commit. That part drops from 17–18 min to about 11–12 min.

Checks.

  • ServerParityPartsTest.mjs (run in PHPUnit Feature) now checks:
    • the extra combination repeats the label's image and memory, and only that label shards;
    • the bootstrap command's exact shape;
    • by running the workflow's own awk program, that the two halves of a 489-name list are disjoint and complete and differ in size by at most one, and that one shard is the whole list.
  • Against the current suite locally, both halves selected 242 of 484 tests and each half passed. The second half's exact container command ran end to end.
  • actionlint, the Docker isolation policy and the comparator node tests (2,204 of 2,204) pass.

Second commit: flake diagnostics (#754). When a case fails, its excerpt now also lists every stored workflow task it kept: identity, type, status, attempt and repair counters, and clock fields. No payloads, errors or lease owners. The two newest flakes were reference repair passes acting mid-phase. One was an embedded record whose excerpt held only its error line, because embedded mode makes no wire requests. The next such failure will name the repaired task and when it was dispatched. ServerParityFailureExcerptTest.mjs adds a case checking that the extra fields stay out.

Third commit: the namespace-signal SDK profile gets its own native node.

  • sdk-directions (16 min on the existing PHP MySQL matrix) was the next slowest part. Its signal lane held all three signal profiles, because they share the server-parity-sdk-signals task queue: 162 + 365 + 213 = 740 s, against about 545 s for each other lane.
  • Each sdk-directions part now starts a second native node with its own database (MySQL and PostgreSQL native_ref_b, SQLite rust-b.sqlite) and passes both URLs to sdk-direction-lanes.sh.
  • Namespace-signal, timers and activity directions run on node b. The other two signal profiles stay together on node a with queries, and the updates lane is unchanged.
  • By measured durations the lanes are 577 s, 611 s and 640 s, so the slowest lane drops from 740 s to about 640 s.
  • ServerParitySdkDirectionLanesTest.mjs now checks that every lane names its server and each profile runs on its declared server. A task queue may be shared across lanes only between different servers. The script refuses to run without the second URL.
  • The existing cleanup removes native-b; SQLite's cleanup now also removes rust-b.

Together: the two slowest parts were 17–18 and 16 min. They should land near the others (10–13 min), shortening the fixtures critical path by about 4–5 minutes.

Measured: I'll post this PR's own part timings after its run.

🤖 Generated with Claude Code

rmcdaniel and others added 4 commits October 25, 2026 07:56
The existing-php-mysql-matrix native-tests part was the slowest fixture
part, at 17 to 18 minutes; 13 of them were the execution suite at two test
threads. A second part, native-tests-b, now runs half of that suite on its
own runner. The bootstrap lists the suite, sorts the names bytewise and
deals them round-robin, so every listed test runs in exactly one half and
a new test joins one automatically. An empty selection fails the part. The
other labels deal one shard, the whole suite, in native-tests as before.

The selection is inline in the container command, so the Docker isolation
policy still sees no repository script inside it. The parts test runs that
exact awk program to check the halves are disjoint and complete.

Checked against the current suite: each half selected 242 of 484 tests,
and the second half's exact command passed all 242.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two recent single-job flakes (#754) were reference repair passes acting
during a fixture phase: an embedded record ended with repair_count 1 on
one of its tasks, and a published PHP record's heartbeat named a recreated
task. The embedded record makes no wire requests, so its excerpt held only
the error line, and the record itself is not retained.

The excerpt now also lists every stored workflow task a failed case kept,
by identity, type, status, attempt and repair counters and clock fields.
Payloads, errors and lease owners are not read, so the next occurrence
names the repaired task and when it was dispatched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three signal SDK profiles share a task queue, so they had to run in one
lane against the shared native node: 162 + 365 + 213 s on the existing PHP
MySQL matrix, against about 545 s for each other lane, making the
sdk-directions part the slowest after native-tests (16 min).

Each sdk-directions part now starts a second native node with its own
database (MySQL native_ref_b, PostgreSQL native_ref_b, SQLite rust-b.sqlite)
and passes both URLs to the lanes script. The namespace-signal, timer and
activity profiles run on node b. The two signal profiles that remain on
node a share one lane, and the updates lane is unchanged. By the measured
durations the slowest lane drops from 740 s to about 640 s.

The lanes test now requires every lane to name its server, checks each
profile runs on its declared server, allows a shared task queue across
lanes only between different servers, and refuses a run without the
second URL. Both nodes are removed by each job's existing cleanup; SQLite's
cleanup now also removes rust-b.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lance-parts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

rmcdaniel commented Oct 25, 2026 •

Copy link
Copy Markdown
Member Author

Bounded Cloud wrapper review: exact head 54fee73e3051f642a548758fc5134dc833dd1028, compared with base 1bca67a6000bd5a3623bd0819b2e3f82db25225c. No additional finding in the requested workflow/resource/diagnostic slice.

  • Checked the extra MySQL matrix combination, two-shard selection and one-shard defaults. Executed the actual shell-quoted sed selection against synthetic list output, and the source AWK expression against 2, 17 and 491 synthetic names: halves are disjoint and complete; count=1 preserves every name. Multiple full-name filters match libtest's documented --exact contract. Native database suites and TLS checks remain in the original first part; no new claim about native-suite runtime or CI timing.
  • Checked separate native_ref_b bootstrap for MySQL/PostgreSQL, separate SQLite rust-b.sqlite, both bounded readiness loops and all three calls supplying the second URL. The nine direction profiles remain present exactly once; the shared-queue namespace-signal profile moves to the separate server. Existing aggregate failure reporting and profile cleanup remain, with explicit cleanup/log collection for SQLite's additional node.
  • Reviewed the diagnostic whitelist: task identity/type/status, counters and clocks are selected; payload, error and lease-owner fields are not selected, and table contents are not recursively traversed as request records. The existing output-count bounds remain.

This was source review plus synthetic shell/partition checks only. No native/shared/frozen implementation inspection or owner fixture execution. Full CI, real database/native qualification, measured queue/runtime improvement and merge remain with the Rust Server owner.

Follow-up at exact dc1872d31ad0d9fff45cadc5ee24633021470d42: reviewed the sole delta from the accepted 54fee73e3. The three readiness loops now enumerate the same fixed container URLs, preserve the 2-second connect/5-second request bounds and 30 attempts, and derive distinct evidence names. MySQL/PostgreSQL node names are preserved; SQLite uses the distinct sdk-rust / sdk-rust-b names within retained parity evidence. No resource, database, profile, partition or cleanup change. No additional wrapper finding or Cloud hold; the original review limits remain.

rmcdaniel and others added 3 commits October 25, 2026 10:55
SonarCloud's clear-text protocol rule (githubactions:S5332) flags curl
URLs whose host is a shell expansion, as on the three per-node readiness
loops. Loop over the fixed container URLs instead, as main's other probes
do, and derive the evidence name from the URL. The probe test accepts a
"$url/api/ready" target and still requires bounded timeouts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In #740's final fixtures run (40183862103) mysql-8.0 native-tests took
20.7 min against 21.0 for existing-php-mysql-matrix, almost all of it the
bootstrap step's native build and execution tests. Halving only the
existing matrix would leave mysql-8.0 as the run's longest part. Give
mysql-8.0 the same two shards and its own native-tests-b combination; the
parts test checks both labels' pairs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An added matrix combination's default job name lists every value it sets,
so the native-tests-b jobs showed the database image digest. Name the jobs
explicitly; the other combinations' names are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

CI receipt — exact head 30c0a5f39783560fdb5e40448ee8589ce3426da3

All checks green on base 1bca67a60 (current main): PHPUnit Feature, Shared Server fixtures (run 40188641265, 48 jobs), SonarQube (gate passed, 0 open issues), Public Boundary.

Changes since the first head (each pushed while the previous run was still going, so each cancelled it):

  • dc1872d31: SonarCloud's githubactions:S5332 flagged the three new per-node readiness loops, whose curl host was a shell expansion. They now loop over the fixed container URLs, as main's other probes do. Cloud's wrapper review (6396738066) covers this delta.
  • 79ffae7a5: mysql-8.0's execution tests are split in two as well. Run 40186725548 at dc1872d31 showed the existing matrix at 12.8 + 7.8 min, leaving mysql-8.0 native-tests (17.0 min) the longest part.
  • 30c0a5f39: the MySQL jobs are named mysql-execution (label, part), because an added combination's default name listed the image digest.

Timing, against #740's final fixtures run (40183862103):

before after
Run wall time 25.9 min 21.6 min
Longest part 21.0 (existing matrix native-tests) 15.7 (existing matrix sdk-directions)
existing-php-mysql-matrix native-tests 21.0 13.3 + 8.6 (-b)
mysql-8.0 native-tests 20.7 12.9 + 9.9 (-b)
sdk-directions, mysql-8.0 / postgres-17 / existing pgsql / mariadb / SQLite 17.1 / 17.0 / 15.8 / 15.3 / 14.0 15.3 / 14.4 / 11.0 / 11.8 / 8.9
Jobs / runner minutes 46 / 489 48 / 480

The two added jobs cost no extra runner minutes. Each part also builds the native binary, and the second SDK node's lanes recovered that cost. One SDK part varied upward: existing-matrix sdk-directions went 13.4 → 15.7, within the run-to-run spread this job has shown (12.8 on dc1872d31). The next longest parts are now the sdk-directions and mariadb/pgsql native-tests parts at 14–15 min, close enough that further splitting has diminishing returns.

The only failure in between, on dc1872d31, was the published PHP 2.11.1 SQLite reference losing a validation lease write to "database is locked". Root cause is on #754 (6397042102). A separate CI PR retries only that reference record.

@rmcdaniel
rmcdaniel merged commit 323ef9b into main Oct 25, 2026
62 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-ci-balance-parts branch October 25, 2026 11:46
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…lite-reference-retry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Reviewed the remaining matrix-only delta from dc1872d31 through exact 30c0a5f39783560fdb5e40448ee8589ce3426da3.

mysql-8.0 now uses the same two execution-test halves as existing-php-mysql-matrix, preserving its pinned image, memory setting, native/TLS work in the first half and the other six parts. The explicit job name only improves identification of the added combinations. Other database labels and fixture/profile commands are unchanged in this delta.

Independently ran the existing ServerParityPartsTest.mjs from an exact two-file Git snapshot in a read-only, network-disabled UID1000 Node22 container: 6 tests passed, including both MySQL label pairs and the existing complete/disjoint shell selection check. This qualifies the workflow wrapper only; no native/shared/frozen implementation or owner fixture was inspected or executed, and no new CI timing or full-suite success is claimed. No Cloud review hold on this delta. The owning agent retains full CI, timing and merge acceptance.

rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…-pip

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…dec-one-job

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
) into rust-325-nexus-operations

Native now serves 140 route methods: main's 138 plus the two
caller-indexed Nexus operation reads. The comparator node test list keeps
both sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
) into rust-325-workflow-stream-writes

Native now serves 140 route methods: main's 138 plus durable workflow
stream appends and closes. The comparator node test list keeps both
sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…ust-325-workflow-import

The comparator node test list keeps both sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…ust-325-record-local-activity

The comparator node test list keeps both sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…ust-325-cancellation-scopes

The comparator node test list keeps both sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant