Rust Server train: MySQL statement closes (#714), policy-map JSON, worker history presentation

.github/workflows/server-parity.yml

@@ -114,6 +114,13 @@ on:
       - rust/**
   workflow_dispatch:
 
+# A newer push to the same pull request supersedes the run in progress, so a
+# replaced head does not keep holding the organization's shared runners.
+# Manual dispatches always finish.
+concurrency:
+  group: shared-server-fixtures-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 permissions:
   contents: read
 

docs/rust-server-port-checkpoints/2026-10-23-mysql-statement-close.md

@@ -0,0 +1,71 @@
+# October 23, MySQL statement closes and late timers
+
+- **Found by.** Server #714. Twice on October 23, the hosted MariaDB 10.11
+  `codec-deadline-health` cell recorded a native timer more than the 1 s
+  maximum late. In one, `TimerFired` came 1.098 s after `fire_at`. MySQL 8.0
+  and 8.4 cells fail the same way, less often.
+- **Reproduction.** A disposable 4-vCPU VM ran the unchanged cell on main
+  `f67f2816e`, with temporary timing in a local-only build:
+  - MariaDB failed 2 of the first 8 cells, with timers 1.228 s and 1.213 s
+    late;
+  - PostgreSQL 17 and SQLite passed all 8 cells, at most 171 ms late.
+- **Attribution.** Every late timer waited behind the global transition lock:
+  - In the worst MariaDB cell, the tick's activity-deadline sweep waited
+    543 ms behind a workflow completion that held the lock 690 ms. The sweep
+    then held it 407 ms itself, and the timer fired 1.228 s late.
+  - Ordinary transitions held the lock 300 ms to 1 s. With every statement
+    logged by MariaDB, those transactions spent a median of 5 ms in the
+    database (p90 9 ms) across their 30 to 48 statements.
+  - Native was not CPU-starved. Wake-up probes on an async task and an OS
+    thread rarely saw 50 ms delays, and giving native 2 CPUs left the holds
+    unchanged.
+  - Timestamps on each statement inside the transactions showed gaps of
+    40 to 100 ms spread across ordinary statements, many near 58 ms. That is
+    Nagle's algorithm waiting for a delayed acknowledgement.
+- **Cause.** `COM_STMT_CLOSE` has no reply. The published `sqlx-mysql` 0.9.0
+  flushes it in its own write: when its 100-entry statement cache evicts a
+  statement, and after executing a statement it does not cache. Before the
+  next command it also flushes anything still buffered. The next command is
+  then a second small write while the close is still unacknowledged, so the
+  kernel holds it about 40 ms. SQLx sets no `TCP_NODELAY`; PHP's mysqlnd does.
+  PostgreSQL's driver sends its closes together with the next request.
+- **Change.** In the vendored driver, `MySqlStream::queue_packet` buffers each
+  statement close with sequence number 0. On the path before the next command,
+  `wait_until_ready` no longer flushes a buffer that holds only replyless
+  closes, so they leave in the same write as that command. `Connection::flush`
+  and `clear_cached_statements` still send queued closes before returning;
+  Security's review of the first revision found that they did not. Packets,
+  order and server behavior are unchanged. The patch record is
+  `rust/vendor/sqlx-mysql/DURABLE-PATCH.md`.
+- **Tests.** A proxy records the TCP read chunks it receives from the client.
+  TCP is a byte stream, so a chunk boundary shows bytes that arrived
+  separately.
+  - `qualification_statement_close_leaves_with_the_next_command` uses a
+    one-entry statement cache and one uncached statement, so both close paths
+    are exercised. With the fix it passes on MySQL 8.0.46 and MariaDB 10.11 in
+    0.01 s. With the published driver it fails on both: every close arrives
+    as a chunk of its own, and the same 14 statements take 0.6 s.
+  - `qualification_explicit_flush_sends_queued_statement_closes`: a queued
+    close stays buffered until `flush`, which sends it with no later command
+    and leaves `should_flush` false.
+  - `qualification_clearing_the_statement_cache_closes_them_now`:
+    `clear_cached_statements` sends all three closes before it returns. On
+    MySQL 8.0 with the performance schema on, the server's statements for the
+    connection drop from three to none.
+  - Both lifecycle tests fail with the first revision (`f31c4c361`) on MySQL
+    8.0 and MariaDB, and pass with this one. The existing cancelled-boundary
+    qualification still passes.
+- **Hosted-cell A/B on the VM.** Interleaved cells with the spec and limits
+  unchanged. Both builds were instrumented: one without the driver change, one
+  with its first revision. That revision already buffered the closes, but
+  `wait_until_ready` still flushed them before draining.
+
+  | | Cells passed | Timer lateness | Holds of 300 ms or more |
+  | --- | --- | --- | --- |
+  | MariaDB without the fix | 3 of 6 | 190 to 1045 ms | 54 |
+  | MariaDB with the fix | 6 of 6 | 49 to 289 ms | 0 |
+  | MySQL 8.0 and 8.4 without the fix | 5 of 8 | 138 to 1099 ms | 77 |
+  | MySQL 8.0 and 8.4 with the fix | 8 of 8 | 116 to 299 ms | 3 |
+
+  Clean release builds of main and of this branch are compared across all five
+  families in the PR's evidence comment.

docs/rust-server-port-checkpoints/2026-10-23-policy-map-json.md

@@ -0,0 +1,33 @@
+# October 23, policy maps must be JSON objects
+
+- **Found by.** PHP Server #705 (merged to main at `f67f2816e`, not yet in a
+  published image). Published PHP turned malformed admission-policy JSON
+  into an empty policy, so a configured cap could disappear without notice.
+  Native had the same defect.
+  - `DW_TASK_QUEUE_ADMISSION_OVERRIDES` and
+    `DW_NAMESPACE_ADMISSION_OVERRIDES` accepted malformed JSON.
+  - All five maps native refuses accepted `null`, `[]`, booleans and
+    numbers.
+  - The external-payload namespace overrides, which native honors, became
+    an empty policy when malformed.
+- **Change.** The six maps PHP now reads with `EnvAuditor::jsonObject` are
+  read by the same rule in `config_audit::json_object`:
+  - the value is either unset or raw text that decodes to a JSON object;
+  - Laravel's `env()` conversions do not apply, so `null`, `(null)`,
+    `empty`, quoted text and an empty value are refused;
+  - a set `DW_*` name wins over its legacy name even when the `DW_*` value
+    is invalid;
+  - the startup refusal names the variable it read, never the value.
+
+  Native still refuses a non-empty object in the five maps it has no policy
+  for. It still honors `DW_EXTERNAL_PAYLOAD_NAMESPACE_OVERRIDES`, and the
+  external payload quota now reads that value with the same decoder.
+- **Reference.** This follows unreleased PHP source, not the published
+  2.11.1 image. For any valid configuration the two behave the same. The
+  only values refused are ones published PHP silently discarded and the
+  next PHP image refuses at boot.
+- **Test.** `policy_maps_must_be_json_objects_like_php_705` covers all six
+  maps under both names, 17 invalid values, padded whitespace around `{}`,
+  DW-over-legacy precedence in both directions, and that the refusal never
+  repeats the value. The shipped Compose and Helm values (`{}` for the
+  durable maps) still start.

docs/rust-server-port-checkpoints/2026-10-23-worker-history-presentation.md

@@ -0,0 +1,48 @@
+# October 23, what a worker reads in history
+
+- **Found by.** Comparing the message-stream fixture's worker polls against
+  published Server 2.11.1.
+- **The gaps.**
+  - PHP's poll and history-page responses show a consumed signal's
+    `workflow_sequence` on its `SignalReceived` event, read from the signal's
+    record (`WorkflowTaskPoller::historyEventsWithSignalArgumentEnvelopes`).
+    Native showed the event as recorded.
+  - PHP's task snapshots in history keep every field that is set
+    (`WorkflowHistoryEvent::taskSnapshot`). Native's left out
+    `compatibility`, `last_dispatch_attempt_at`, `last_dispatched_at`,
+    `last_dispatch_error`, `last_claim_failed_at` and `last_claim_error`.
+- **What native does now.**
+  - Worker polls and history pages add `workflow_sequence` to a consumed
+    signal's `SignalReceived`, when the payload does not have one. The
+    run's own history route still shows the events as recorded, as PHP's
+    does.
+  - Task snapshots keep those fields when they are set.
+- **Which signals carry a sequence.** PHP's `applySignal` takes a sequence
+  only from the signal's own wait. A signal the run's contract declares is
+  applied without one even while a condition wait is open, and
+  `markConditionWaitSignalConsumed` then finds it applied, so its record keeps
+  no sequence. Only an undeclared signal takes the condition wait's sequence.
+  Native had recorded the condition wait's sequence for declared signals too.
+  That difference was invisible until worker polls showed the record, and the
+  published-SDK signal and query cohorts then caught it: their worker history
+  must equal the run's history prefix. Native now records a sequence only from
+  the signal's own wait, or from a condition wait for an undeclared signal.
+- **How it was checked.** A test runs on SQLite, PostgreSQL, MySQL and
+  MariaDB:
+  - a condition wait consumes an undeclared signal;
+  - the next poll and history page show that wait's sequence, and the
+    run's history does not;
+  - the completing task's snapshot has its dispatch stamps.
+
+  A second test applies a declared signal under a condition wait twice. The
+  worker's history shows no sequence on either `SignalReceived`, and it fails
+  without the record change.
+- **Not changed yet.** Some events PHP records with a task snapshot have
+  none in native, such as `ConditionWaitOpened` and `SignalWaitOpened`.
+  That audit is a #325 follow-up.
+- **Known difference now visible in snapshots.** PHP stamps a dispatch only
+  on tasks a control-plane call dispatches, such as a signal's task. Tasks a
+  workflow task completion creates, such as the next message task, carry
+  none. Native stamps every new task, so those snapshots show stamps where
+  PHP's do not. This is the cause of the known `tasks_dispatched_last_minute`
+  difference too; fixing it is a #325 follow-up.