Skip to content

Rust Server train: MySQL statement closes (#714), update validation, policy-map JSON, worker history presentation - #723

Closed
rmcdaniel wants to merge 23 commits into
mainfrom
rust-325-train-oct24
Closed

rmcdaniel wants to merge 23 commits into
mainfrom
rust-325-train-oct24

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

One train for four reviewed Rust Server #325 PRs on current main 960aa3051 (after #707), so CI runs once instead of four times while runners are saturated:

Merge order: main → #717 → #712 → #719 → #710. Only .github/workflows/server-parity.yml conflicted: #712 and main (#707) each added a contract test to the same node --test list. The resolution keeps both, so the list holds the union: ServerParityOperatorDashboardTypesContractTest.mjs, ServerParityClusterInfoContractTest.mjs and ServerParityUpdateValidationContractTest.mjs.

Local on this head: cargo check --all-targets, and SQLite cargo test --all-targets passes (lib 257, execution 456, plus the other targets).

When this train merges, the four constituent PRs close as merged through it, and #718 is rebased onto main. #708, #713, #718, #720, #722 and #673 stay separate under their own review gates.

…s, like Server 2.11.1

Worker polls and history pages add workflow_sequence to a consumed signal's
SignalReceived from its record, as WorkflowTaskPoller does; the run history
route still shows the event as recorded. Task snapshots keep compatibility,
the dispatch and claim stamps and errors when they are set, as
WorkflowHistoryEvent::taskSnapshot does.
An update whose durable contract declares a validator now waits for a
validator-capable worker to approve it, as published PHP does: admission
refusals in PHP's order, validation tasks keyed by run, update and request id,
the dedicated and multiplexed worker polls, fenced approve and reject routes,
lease expiry and re-claim, and 504 timeouts with their reasons. An accepted
update's receipt no longer carries accepted: true, as PHP's result mapper
drops it. The update-validation fixture records 49 steps against published
PHP; native matches all of them.
The six admission, durable-limit and external-payload policy maps are
read like PHP's EnvAuditor::jsonObject: unset, or raw text that decodes
to a JSON object. Malformed JSON, null, lists and scalars now stop
startup instead of silently becoming an empty policy. A set DW_ name
wins over its legacy name even when invalid, and the refusal names the
variable without its value. The external payload quota reads its
overrides with the same decoder.
COM_STMT_CLOSE has no reply. The published sqlx-mysql flushed it on its
own write after a cache eviction or an uncached execute, and flushed it
again before draining results. The next command then waited behind
Nagle's algorithm for the server's delayed acknowledgement, about 40 ms.
Across the 30-48 statements of a transition that held the global lock
for 0.3-1 s under load, and a due timer behind it fired more than the
1 s bound late (#714).

The vendored driver now queues each close with sequence 0 and leaves a
buffer of replyless closes unflushed until the next command, so both
go out in one write. A proxy-recorded qualification test fails with the
published driver and passes with the change on MySQL 8.0 and MariaDB.
Security's review of f31c4c3: Connection::flush delegated to
wait_until_ready, which now leaves a buffer of queued closes unsent, and
clear_cached_statements only queued its closes. Both report success with
the bytes still buffered.

Explicit flushes and cache clears now send every buffered byte before
returning (MySqlStream::flush_queued); only the path before the next
command keeps closes to share that command's write. New MySQL and
MariaDB controls prove both calls without a later command; they fail on
f31c4c3. The close test now describes what it observes: TCP read
chunks at a proxy, not write calls.
Workflow 2.10.2's applySignal takes a workflow sequence only from the
signal's own wait, so a declared signal applied while a condition wait
is open keeps no sequence; only an undeclared signal takes the condition
wait's. Native recorded the condition wait's sequence for both, which
became visible once worker polls showed the record: the published-SDK
signal and query cohorts require worker history to equal the run's
history prefix (all nine SDK pairings on every database). A new test
covers the declared case and fails without the change.
… rust-325-train-oct24

# Conflicts:
#	.github/workflows/server-parity.yml
rmcdaniel and others added 2 commits October 24, 2026 04:26
…orkflows

A newer push to the same pull request now cancels the run in progress for
Shared Server fixtures (13 jobs, about 415 job-minutes) and PHPUnit
Feature, as Sonar already does. The organization's 20 shared runners no
longer spend hours on replaced heads. Runs on main and manual dispatches
always finish.
The probe waits through the 10-second validation timeout, a 6-second lease
expiry, a 2-second completion wait and several short polls before PHP does
any work. On the published PHP SQLite runtime it exceeded the recorder's
generic 45-second probe limit while making its last calls (train #723,
job 120776618912: spawnSync php ETIMEDOUT). Expected results are
unchanged.
@rmcdaniel

rmcdaniel commented Oct 24, 2026 •

Copy link
Copy Markdown
Member Author

Head d1372fb80. The first run at 069270b4e failed execution-and-codec step 10 with spawnSync php ETIMEDOUT (job 120776618912). It was the published-PHP SQLite target running the new update-validation probe (#712). The probe hit the recorder's generic 45-second probe limit while making its last calls: the PHP server access log shows the final update-validation-tasks/poll and history reads in the same second as the timeout. The probe waits through the 10-second validation timeout, a 6-second lease expiry, a 2-second completion wait and several short polls before PHP does any work, so 45 seconds left no margin on SQLite. This head gives that fixture's probe a 90-second limit; expected results are unchanged. The first run's six remaining jobs are cancelled, and this head gets one fresh run. Before it was cancelled, that run had passed five codec cells (MySQL 8.0, MySQL 8.4, MariaDB 10.11, PostgreSQL 17, SQLite). The PostgreSQL 16 cell was cancelled before it finished.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Head 38b826773cfd9ab34a55ea28612ce8f9eb819a56 adds #724 to the train: the superseded-run cancellation for the fixture and PHPUnit workflows, with no product code. It is tested in this run instead of a separate full fixtures run. The train is now: main 960aa3051 + #717 + #712 + #719 + #710 + the update-validation probe limit + #724. The d1372fb80 run is cancelled; one fresh run continues on this head.

…perseded PHPUnit runs cancel

The update-validation probe's waiting child requests now retry PHP's
documented backend_lock_pressure refusal (nothing committed) with the same
rule as every other probe request. seconds stays the answered attempt's
own wait; retries are reported on stderr.

PHPUnit feature suite and Feature source qualification used a job-level
always(), which GitHub never cancels, so #724's superseded-run cancellation
left the old run holding the concurrency group. !cancelled() still runs
both jobs when a needed job failed, so the required checks stay fail-closed.
@rmcdaniel

Copy link
Copy Markdown
Member Author

New head 180d4e3caba23788fca760eb34c49a99ad51d7d2: two additions to the train.

1. Update-validation fixture, PHP reference on SQLite (the busy_approve_late failure). This comes from PHP 2.11.1 under SQLite write contention, not from native. With CI's PHP maintenance loop running, PHP's own writes fail with SQLSTATE[HY000]: General error: 5 database is locked: the validation timeout (markTimedOut) and the expired-lease reclaim (attempt_count = 2). The task stays leased, so the late approval answers update_validation_lease_expired and the old-attempt approval sees attempt 1. Local repro against published PHP 2.11.1 with CI's maintenance loop and resource limits:

  • maintenance running: 10 of 15 runs fail (busy_approve_late or approve_old_attempt). The database rows confirm the busy task is still leased with no timed_out_at.
  • maintenance stopped: 5 of 5 pass.

The checker reports the first differing step in key order, so a failed run shows only that first step.

This head adds one bounded change: the probe's waiting child requests now retry PHP's typed backend_lock_pressure refusal (retryable: true, nothing committed) with the same lockPressureOperationRetryDelay rule as every other probe request. That covers the earlier worker_lost 503. It does not cover the failures above: PHP's dedicated validation poll lets the lock error escape without the typed poll refusal. Expected results are unchanged. Remaining decision: record this fixture's PHP leg with the SQLite maintenance loop paused, as #722 does for the patch cohort, or accept it as a PHP-reference flake. A PHP Server report for the untyped poll refusal goes to the PHP owner.

2. PHPUnit superseded-run cancellation (fixes #724's intent). PHPUnit feature suite and Feature source qualification had a job-level if: ${{ always() }}. GitHub never cancels such jobs, so a newer push waited behind the old run instead of replacing it (seen on #725). Both now use if: ${{ !cancelled() }}. They still run when a needed job failed, so the required checks report that failure rather than being skipped. The contract test needles are updated. Locally: PhpunitFeatureWorkflowContractTest and OpenApiDocumentEvolutionTest pass 17/17, and the Docker workflow isolation policy passes.

The Security R2 handoff for #723 waits for terminal checks on this head.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security current-head check at 180d4e3: R2 acceptance remains withheld.

The required PHPUnit workflow is red for a source-level policy incompatibility, not an external runner failure. Run 40067888485 shows the central action-policy validator rejecting phpunit-feature.yml because the required check condition no longer contains always(). The protected current validator explicitly requires always() in that gate condition so the required check reports a failed preflight. The feature job did run and fail closed after the preflight failure, but the static central contract still correctly blocks the changed source.

A narrow candidate is a condition which retains always() while explicitly refusing cancellation, such as always() && !cancelled(), followed by the actual superseded-run cancellation counterfactual and unchanged failed-preflight reporting. Coordinate that contract with the central policy owner rather than weakening or bypassing the validator. The cleanup error for the absent scripts/ci/cleanup-database-tls.sh after checkout was skipped is secondary failure-path noise, not the root gate.

Separately, run 40063949579 / job 120786013151 remains a failed superseded original. Its single update-validation difference is recorded in private D500 with the public job link; the current full Shared Server matrix is still in progress. No exact-head Security merge authorization yet.

…or its PHP leg

The SQLite job's main corpus now excludes update-validation.json. A separate
script records that fixture for php, embedded and rust and compares the three.
Its published PHP record runs with PHP's maintenance loop stopped: under that
loop's SQLite writer contention, PHP 2.11.1 loses its own validation-timeout
and lease-reclaim writes to "database is locked" and its validation poll
answers without the typed lock-pressure refusal (10 of 15 local runs failed
with the loop running, 5 of 5 and 3 of 3 passed with it stopped). Expected
results are unchanged. MySQL and PostgreSQL keep the fixture in their main
corpus with their loops running.

The runner gains --exclude-fixture, which must name a selected fixture and
cannot be combined with --fixture. The maintenance window helper is #722's,
moved to a shared path. Tests cover the selection rules, the workflow wiring
and every restore path of the helper.

PHPUnit Feature returns to main's job conditions and drops its concurrency
group. The central action policy requires always() on the PHPUnit feature
suite gate, and GitHub never cancels an always() job, so cancel-in-progress
would only queue a newer run behind it. Shared Server fixtures keeps #724's
cancellation.
@rmcdaniel

Copy link
Copy Markdown
Member Author

New head 1fd9e549a897b71e8cc7711db0c08e884bc33bf6. The 180d4e3ca run failed Central action policy preflight: the central policy requires always() on the PHPUnit feature suite gate (required check must run after a failed central action policy preflight). The PHPUnit failures that followed came only from that gate. This head makes two changes.

1. PHPUnit Feature returns to main's job conditions and has no concurrency group. GitHub never cancels an always() job. With cancel-in-progress, a newer push would just wait behind the old suite. Shared Server fixtures keeps #724's cancellation, because none of its jobs uses a job-level always(). Superseded-run cancellation for PHPUnit waits on a central-policy decision. Locally, the central validator at .github origin/main 64fd4a9c passes on this head.

2. Decision on the update-validation fixture: record its PHP leg on SQLite with PHP's maintenance loop stopped. Accepting it as a flake would leave a required check that fails about two runs in three.

  • The SQLite job's main php, embedded and rust records, and their comparison, use the new --exclude-fixture tests/Fixtures/ServerParity/update-validation.json.
  • scripts/conformance/server-parity/update-validation-sqlite.sh records that fixture for all three targets and compares them. Only its PHP record runs inside run_without_maintenance, which is Server parity: SQLite published-SDK patch record without PHP's maintenance loop (#721) #722's helper moved to scripts/conformance/server-parity/php-maintenance-window.sh. The script's status is checked with the step's other deferred statuses.
  • Expected results are unchanged. MySQL and PostgreSQL keep the fixture in their main corpus with their loops running.
  • The runner's --exclude-fixture must name a selected fixture, and it cannot be combined with --fixture.

Tests.

  • New: ServerParityFixtureSelectionTest.mjs (selection rules plus workflow wiring) and PhpMaintenanceWindowTest.mjs (every restore path). Together with the visibility and update-validation contract tests, 122 of 122 pass locally.
  • The wiring test fails if any main record drops the exclusion, or if the separate script is removed (counterfactuals run).
  • PhpunitFeatureWorkflowContractTest passes 6 of 6.
  • Local end-to-end against published PHP 2.11.1 with CI's maintenance loop: PHP plus embedded plus their comparison pass 3 of 3. Each run's schedule is before: running, during php record: exited, after php record: running for 1s x3, and no "database is locked" lines appear in the maintenance log.

The PHP 2.11.1 poll defect (no typed lock refusal on update-validation-tasks/poll, lost markTimedOut and reclaim writes) has gone to the PHP owner. The Security R2 handoff waits for terminal checks on this head.

…ase probes

The published PHP SQLite reference shares its file with a one-second
maintenance loop. #723's run 40069368271 recorded the Prometheus
fixture's activity claim as 503 (then 422 for the completions that had
no task). An activity poll under lock pressure answers the documented
503 backend_lock_pressure refusal, which hands out no task and tells the
worker to retry.

The system-metrics and operator-dashboard probes already retry exactly
that refusal through lock-pressure-middleware.php. The Prometheus and
activity lease renewal probes build their own Guzzle clients for raw
worker polls and did not. They now push the same middleware. Expected
results are unchanged: the claim must still end at 200, and every other
status or body still reaches the contract.

ServerParityRunnerTest now requires the middleware in every corpus probe
that polls over its own HTTP client; it fails against the previous
Prometheus probe.
@rmcdaniel

rmcdaniel commented Oct 24, 2026 •

Copy link
Copy Markdown
Member Author

Classification of the execution-and-codec failure at head 1fd9e549a (run 40069368271, job 120802438457).

What failed. Step 10, "Execute and compare Rust, published PHP and embedded fixtures". One fixture differed: prometheus-metrics, "scenario requests". claim activity answered 503 where the fixture expects 200, so complete activity and complete run then answered 422, because there was no task to complete. Every later step in the job passed, including the query-authority cohort.

Which runtime. The published PHP 2.11.1 target on SQLite, not native:

  • The step records PHP first, and that command has no error guard. Its failure ended the step at once, at 06:40:29.
  • PHP's access log runs right up to that second. The native record and the comparison never started.
  • Around the same time, the PHP maintenance container (repair-pass and schedule:evaluate every second) logged a failed TaskRepair update inside a transaction, with SQLSTATE[HY000]: General e…. The bounded log tail cuts the message off there. SQLite reports a write lock as General error: 5 database is locked, but the full text is not in the retained evidence.

Cause. In published 2.11.1, an activity poll that meets lock pressure answers the documented 503 backend_lock_pressure refusal (bootstrap/app.php maps activity-tasks/poll to BackendLockPressure::workerPollResponse). It hands out no task and tells the worker to retry. The 503's body itself is not in the retained evidence: the bounded access log starts after it, and the observation is not uploaded. So "typed refusal" is inferred from the source and the timing, not read from the response. The system-metrics and operator-dashboard probes already retry exactly that refusal through lock-pressure-middleware.php. The Prometheus probe, and the activity lease renewal probe, build their own Guzzle clients for raw worker polls and did not. This is a harness defect. PHP's answer follows its contract, so there is nothing to report to the PHP owner. The PHP write-path defect in the same area, which does lose writes, is #726.

Correction (commit 9d2ab685f, pushed once this run's storage cells finish):

  • Both probes push the same middleware as their siblings. Expected results are unchanged: the claim must still end at 200, and any other status or body still reaches the contract. The middleware retries only the exact typed shape (503, reason: backend_lock_pressure, task: null), so if this 503 was anything else, the new head fails the same way and the inference above is wrong.
  • ServerParityRunnerTest now requires the middleware in every corpus probe that polls over its own HTTP client. It fails against the previous Prometheus probe. The 270 runner tests pass.
  • Local check: published 2.11.1 on SQLite with the maintenance loop running, both fixtures, PHP leg 6/6 pass. The maintenance log showed 9 lock errors, but no poll was refused during these runs, so they show no regression without exercising the retry itself.

Residual, not changed here. Four other corpus probes build their own clients without the middleware: search-attribute, storage-test, schedule-policies and service-catalog. They make control-plane writes or reads, not worker polls, and none has failed this way. They are left as they are rather than widened without evidence.

Sonar reported 11 style findings in the two new scripts. Conditions use
[[ ]], the state reader ends with an explicit return, and positional
parameters are read into named locals. Behavior is unchanged: a missing
COMMAND still fails under set -u, and each assertion still fails the
caller under set -e. PhpMaintenanceWindowTest passes 5/5, and one local
end-to-end record stopped and restored the real maintenance container.
Train #723's MySQL 8.0 pressure profile failed 'one real SDK budget
expires while pressure stays held' on the published PHP target, and the
log could not say whether shutdown took under 8 s or over 10 s. The
assertion now prints the measured time and backend. The window itself is
unchanged.
@rmcdaniel

Copy link
Copy Markdown
Member Author

New head ef5a6be70, three commits on 1fd9e549a. main has not moved (960aa3051).

  • 9d2ab685f: the Prometheus and activity lease renewal probes retry the documented 503 backend_lock_pressure poll refusal, like their sibling probes. ServerParityRunnerTest now requires that of every corpus probe that polls. Classification: comment 6381159759.
  • 9bb9cb978: Sonar's 11 shell findings in the two new SQLite update-validation scripts, fixed with no change in behavior. PhpMaintenanceWindowTest passes 5/5, and one local end-to-end record stopped and restored the real maintenance container.
  • ef5a6be70: the SDK pressure assertion prints the measured shutdown time. This is diagnostics only.

Terminal evidence at 1fd9e549a (run 40069368271): 11 of 13 jobs passed. That includes all six codec-deadline-health cells, so the MySQL-family stall #717 removes did not appear. Two jobs failed:

Sonar decision. rust/src/runtime/updates.rs admit_update has cognitive complexity 83 after #712's validation path. It is accepted for this train rather than restructured mid-review. The follow-up is to extract the validation wait into its own function in a later train, with behavior unchanged.

A fresh full run is now running on ef5a6be70. The Security R2 handoff follows when it is terminal.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security pre-terminal R2 source review at exact head ef5a6be704c1029408a3bbe6cd6c36d5eeeaee1b: do not merge this head. The new multiplexed update-validation poll advertises an idempotency contract that native does not implement.

rust/src/runtime/update_validation.rs::capability() advertises poll_request_id_binding: normalized_task_kind_set and poll_request_task_kinds_conflict. Published PHP proves that contract in WorkflowUpdateValidationTaskTest::test_poll_request_id_is_fenced_to_the_normalized_task_kind_set: reordered retries replay the same leased task and a different kind set is refused.

At this head, poll_task_kinds() and claim_update_validation() never read, bind, or persist poll_request_id. Only the ordinary workflow-task Store::claim() path consults dw_poll_receipts; that table is also foreign-keyed only to workflow_tasks, so it cannot replay a validation-task delivery. Consequences:

  • a lost multiplexed validation-poll response retried with the same request ID can lease a different validation or workflow task while the first lease remains active;
  • the same request ID can be reused with a different task_kinds set without the advertised typed conflict;
  • the shared fixture currently checks the capability bytes and ordinary ordering, but has no retry/conflict counterfactual, so its pass does not cover this contract.

There is a related atomicity gap: native reads first_task_kind, claims, and turns the cursor in separate globally serialized transactions. Two concurrent polls can read the same first kind before either cursor turn. PHP's WorkflowTaskKindSelector keeps cursor choice, claim, and turn in one transaction.

Required correction: bind a non-empty poll request ID to the normalized effective task-kind set and replay the same still-valid response for both workflow and validation deliveries; refuse changed sets with the documented fields/reason. Keep kind selection, at-most-one claim, and cursor turn in one durable transition. Add focused native and shared PHP/native fixture counterfactuals for reordered retry, changed-set conflict, validation-task replay, workflow-task replay, expired cached delivery, and concurrent multiplexed claims across the supported databases. Do not resolve this only by retaining the manifest claim without behavior; if parity is deliberately deferred, remove the unsupported advertised contract and document the compatibility decision explicitly.

The already reviewed MySQL close lifecycle, JSON-object fail-closed parsing, and worker-history presentation remain source-acceptable. The exact-head terminal matrix is still useful evidence for the unaffected surfaces, but it cannot authorize this R2 merge until the poll contract is corrected and the changed range is requalified.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Superseded by #729. Security's pre-terminal review (comment 6381682935) blocked this head on #712's multiplexed update-validation poll contract. #729 carries the four constituents Security found source-acceptable (#717, #719, #710, #724) plus the probe retry and the pressure diagnostic. #712 stays open and gets the poll-request-id correction as its own train, together with this branch's update-validation harness commits. This PR's running matrix is cancelled to free runners for #729.

@rmcdaniel rmcdaniel closed this Oct 24, 2026
rmcdaniel added a commit that referenced this pull request Oct 24, 2026
The probe waits through the 10-second validation timeout, a 6-second lease
expiry, a 2-second completion wait and several short polls before PHP does
any work. On the published PHP SQLite runtime it exceeded the recorder's
generic 45-second probe limit while making its last calls (train #723,
job 120776618912: spawnSync php ETIMEDOUT). Expected results are
unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native timer exceeds the 1s lateness bound under MariaDB qualification load

1 participant