Native timer exceeds the 1s lateness bound under MariaDB qualification load #714

Activity

1. rmcdaniel commented on Oct 23, 2026

   

   rmcdaniel

   on Oct 23, 2026

   MemberAuthor

   More actions

   Cause found; fix in #717. The 1.000 s bound is unchanged.

   Reproduction (disposable 4-vCPU VM, unchanged codec-deadline-health cell on main f67f2816e):

   • MariaDB 10.11 failed 2 of 8 cells (timers 1.228 s and 1.213 s late).
   • MySQL 8.0 and 8.4 failed 3 of 8.
   • PostgreSQL 17 and SQLite passed 8 of 8, at most 171 ms late.

   Attribution (temporary local-only instrumentation, never pushed):

   1. Timer path. Every late timer waited behind the global transition lock. In the 1.228 s case, the timer came due just after the scheduler tick began. The tick's activity-deadline sweep waited 543 ms behind a workflow completion holding the lock 690 ms, then held it 407 ms itself. The timer then waited 145 ms more.
   2. Database. Not slow. MariaDB's full statement log shows those transactions spent a median of 5 ms in the database (p90 9 ms) over 30 to 48 statements, while holding the lock 0.3 to 1 s. Only the lock acquisition statement itself was ever slow.
   3. Executor and CPU. Not starved. Async-task and OS-thread wake-up probes rarely saw a 50 ms delay. Giving native 2 CPUs instead of 1 left the holds unchanged, and MariaDB's own CPU averaged about 30%, like PostgreSQL's.
   4. Measurement boundary. Not the cause. The same per-statement stall appears in hold times measured inside native.
   5. The stall. Timestamps on each statement inside the transactions showed 40 to 100 ms gaps spread over ordinary statements, many near 58 ms. That is Nagle's algorithm waiting for a delayed ACK. sqlx-mysql 0.9.0 sends replyless COM_STMT_CLOSE packets in their own writes (on cache eviction, after an uncached execute, and in a flush before draining results), so the next statement waits about 40 ms. SQLx sets no TCP_NODELAY; PHP's mysqlnd does. PostgreSQL's driver batches its closes, which is why only the MySQL family fails.

   Fix (#717, vendored driver only): queue statement closes so they leave in the same write as the next command. A proxy-recorded regression test fails with the published driver and passes with the fix on MySQL 8.0 and MariaDB.

   Before and after. Clean release builds, same cell, interleaved:

   	main	#717
   MariaDB 10.11	2/4 passed, worst 1738 ms	4/4, worst 55 ms
   MySQL 8.0	3/4, worst 307 ms	4/4, worst 235 ms
   MySQL 8.4	1/4, worst 1672 ms	4/4, worst 292 ms
   PostgreSQL 17	4/4	4/4
   SQLite	4/4	4/4

   Full table and local test receipts: #717 (comment). The hosted MariaDB/MySQL cells rerun on #717's own CI.

   Cleanup. VM dw-rs325-20261023T212416 ran 21:24–22:51 UTC (about $0.10). Instance and temporary SSH key were deleted and verified absent. No other instances were touched. The local instrumentation branch was never pushed and is being deleted.
2. rmcdaniel

   added a commit that references this issue on Oct 24, 2026

   Merge pull request #729 from durable-workflow/rust-325-train-oct24-core

   ...

   6121960
3. rmcdaniel

   closed this as completedin #717on Oct 24, 2026