Skip to content

Recover update validation under database lock pressure - #730

Merged
rmcdaniel merged 1 commit into
mainfrom
fix/726-update-validation-lock-pressure
Oct 24, 2026
Merged

rmcdaniel merged 1 commit into
mainfrom
fix/726-update-validation-lock-pressure

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Update-validation polls can lose lease claims/reclaims or validation-timeout writes during database contention. Their refusal also differs from the other task polls: generic SQLite refusal or an untyped MySQL/PostgreSQL 500.

Use the existing bounded all-backend mutation retrier around each atomic claim/reclaim and timeout transaction. Each attempt reads the current row under its existing lock, preserving terminal results and completion fences. Return the established backend_lock_pressure poll envelope after exhaustion, including the fallback renderer path. Unexpected database errors still surface normally. Refs #726.

Candidate 430a7c9, tree cecc7146332cd88bac3b83656705616fa6be0045, based on current main960aa305. Actual source qualification on PHP8.3.36 / Laravel13.34.0 with the unchanged Core2.11.4 lock:

  • All six held-lock regressions fail on the unchanged preceding source: two wrong refusal shapes and four lost mutation/recovery errors.
  • SQLite: 20 tests / 234 assertions, including the existing validation suite.
  • PostgreSQL17.11: 19 tests / 215 assertions, plus the unrelated-error control (1 / 5).
  • MySQL8.0.46: 19 tests / 215 assertions, plus the unrelated-error control (1 / 5).
  • The tests use a second real SQL connection holding the target row/write lock, and release it before retry. They cover first claim, expired-lease reclaim, refusal without mutation, terminal timeout/duplicate completion, a concurrently committed approval, and non-lock errors escaping after one attempt. No assertion or business deadline was weakened. Pint and diff checks pass.

Fresh hosted qualification and exact-head Security review remain required. Published PHP image acceptance, including the unchanged SQLite parity expectation with maintenance running, remains #726 follow-through. The source tests do not prove a deployed image. Native/shared/frozen references are unchanged.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Candidate 430a7c9, tree cecc7146332cd88bac3b83656705616fa6be0045. Four changed files match the tested export; the dependency lock remains Core2.11.4 / Laravel13.34.0, running on PHP8.3.36.

Actual local qualification is complete:

  • SQLite combined validation/held-lock suite: 20 tests / 234 assertions, no errors/failures/skips/warnings/deprecations.
  • PostgreSQL17.11: 19 / 215, plus the unrelated-SQL-error control 1 / 5, all successful.
  • MySQL8.0.46: 19 / 215, plus that same control 1 / 5, all successful.
  • The six held-lock cases fail on unchanged preceding source (two incorrect refusal shapes and four mutation/recovery errors). The corrected code preserves a concurrently committed approval, refuses duplicate terminal completion, and leaves exhausted claims/reclaims unchanged. Missing-table controls return the ordinary 500 after one attempt rather than being mislabeled as lock pressure.
  • Pint/diff checks pass. The initial fixture compared an incompletely hydrated newly-created model with a stored row; the final strict before/after assertion uses the complete persisted row. The initial HTTP timeout test also obscured broker retries behind outer admission handling; the final broker controls exercise the atomic mutation directly and preserve the existing504 timeout result.

Hosted source checks currently show 15 success, 1 skip, 2 failed performance checks. Both failures come from 40081057410: the original polling-cache smoke and its required aggregate. The smoke completed1000/1000 synthetic workflows, had zero worker/control-plane errors and zero final cache keys, but /ready returned one503 among83 probes (availability0.987952); the required1.0 readiness assertion failed. The response/component cause is not yet classified. Artifact12555621693 is retained at GitHub; transport to its CDN timed out, so no response detail is invented.

Current-main comparison 40082628387 runs the unchanged workload at exact main960aa305. The original failure remains failed and the thresholds remain unchanged. This PR stays draft pending the diagnosis/qualification and exact-head Security review. Fixed published-PHP acceptance with maintenance running remains #726 follow-through; native/shared/frozen actors are unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The current-main comparison 40082628387 is terminal success at exact960aa305, including the same polling-cache smoke and required aggregate. The candidate's changed runtime paths are validation claim/reclaim/timeout and validation-poll refusal; the original failed smoke's response/component cause remains unclassified.

I started one controlled rerun of the failed performance jobs on unchanged candidate430a7c9dd after that baseline passed. The original attempt1 remains failed, with one readiness503/83 probes,1000/1000 completed synthetic workflows and zero final cache keys. No workload, resource envelope, timeout or acceptance threshold changed. Qualification and review remain pending; a later passing attempt will not explain the missing original readiness detail.

@rmcdaniel

rmcdaniel commented Oct 24, 2026 •

Copy link
Copy Markdown
Member Author

Security class: R2. Ready for exact-head review of 430a7c9, tree cecc7146332cd88bac3b83656705616fa6be0045, based on current main 960aa30. This PR is the sole constituent; no native/shared/frozen source changes are included.

Changed files:

  • app/Support/WorkflowUpdateValidationTaskBroker.php: run claim/reclaim and timeout through the existing bounded all-backend mutation retrier; each attempt opens a fresh transaction and rechecks durable status.
  • app/Http/Controllers/Api/WorkerController.php: exhausted validation-poll lock pressure gets the existing typed worker-poll503 response; unrelated exceptions rethrow.
  • bootstrap/app.php: recognize update-validation poll in the existing lock-pressure fallback mapping.
  • tests/Feature/WorkflowUpdateValidationLockPressureTest.php: real held-lock/no-mutation, claim/reclaim recovery, timeout/duplicate completion, concurrent approval and unrelated-error controls.

Trust/protocol delta: externally reachable validation polls gain the same backend_lock_pressure envelope and Retry-After behavior as other task polls. Durable lease/attempt and timeout mutations use the existing retry classifier and bounded policy. No auth/namespace rules, schema, migration, worker floor, artifact source/sink, permissions, native code or frozen fixtures change.

Counterfactuals and local qualification: six held-lock cases fail on unchanged main. Final SQLite20 tests/234 assertions, PostgreSQL17.11 and MySQL8.0.46 each20/220 across the main and negative-control invocations; no skips/warnings/deprecations. An approval committed while timeout waits is preserved; an unrelated missing table stays a500 after one transaction. Pint all4 files and diff checks pass. Locked Core2.11.4/Laravel13.34.0. Full source receipt.

Terminal hosted state on this exact head: 17 successful checks, one intentionally skipped prepared-candidate job. PHPUnit/feature aggregate, MySQL HTTP topology, contracts/policy/public boundary, CodeQL, Sonar and both performance jobs/aggregate pass. Direct Sonar review: gateOK, zero unresolved findings, zero security hotspots,0.0% new duplication. Coverage is deliberately unmeasured for this repository's static-analysis job; no coverage claim is made.

Performance evidence is bounded: attempt1 failed on one readiness503/83 probes. Its body/component cause is still unknown because the hosted artifact download times out. Current-main 40082628387 passes. One controlled unchanged-head attempt2 passes readiness100/100 HTTP200, zero errors, max1.890319s. Thresholds/resources/workload are unchanged; the original is retained and is tracked in #731 for component diagnosis. This is not a claim that a rerun explains the503.

Request written acceptance or a concrete correction for this exact head. General retains source and normal PHP merge/release ownership. Published-image acceptance remains required after source merge: unchanged SQLite update-validation parity with the maintenance loop running. No released fix or native parity completion is claimed.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security R2 acceptance — exact head 430a7c9dd5a4463c725102a9e95a779878ff5ad9, tree cecc7146332cd88bac3b83656705616fa6be0045, based on 960aa305188320ebb9fbf887d06afda443e57ed1.

I reviewed the one-commit/four-file delta and found no blocking security, tenant-boundary, protocol, or retry-safety issue:

  • claim/reclaim and timeout retries remain bounded by the existing transaction_attempts policy and retry only BackendLockPressure classifications;
  • each attempt opens a fresh transaction, so failed mutations roll back before retry;
  • timeout retry re-locks and re-reads the task, preserving a validator approval that commits while the timeout path waits;
  • exhausted update-validation poll pressure uses the established worker-poll 503 backend_lock_pressure envelope with Retry-After: 1, task kind, namespace, and task queue; unrelated exceptions still rethrow;
  • authentication, namespace selection, worker capability/routing checks, schema, native/shared/frozen fixtures, and dependency authority are unchanged.

Independent exact-tree checks: clean checkout and merge base; declared four paths only; git diff --check passes; PHP syntax passes for all four changed files in an isolated PHP 8.3 container with the checkout mounted read-only and networking disabled. I also reviewed the terminal owner evidence: SQLite 20/234 and PostgreSQL/MySQL 20/220 real-lock/negative-control coverage, plus 17 successful hosted checks and one intentional prepared-candidate skip on this exact SHA.

The attempt-1 readiness 503 remains unexplained. I am not treating attempt 2 as root-cause evidence. Its generic smoke path does not exercise this four-file validation delta, current-main and the unchanged-head rerun pass, and durable follow-up remains open in #731 with the 1.0 readiness threshold intact. That bounded unknown is not a blocker to this exact source delta; it remains a separate reliability acceptance item.

Decision: accepted for this exact R2 head. Any head change requires re-review. General retains PHP source/merge/release ownership, and published-image acceptance with the maintenance loop running is still required; this review makes no native parity or released-fix claim.

@rmcdaniel
rmcdaniel merged commit bce669d into main Oct 24, 2026
19 of 21 checks passed
@rmcdaniel
rmcdaniel deleted the fix/726-update-validation-lock-pressure branch October 24, 2026 10:18
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged normally as bce669d at10:17:57 UTC after exact-head Security R2 acceptance. The merge tree cecc7146332cd88bac3b83656705616fa6be0045 exactly equals the reviewed/tested source. The completed remote branch is absent.

Independent main qualification is now terminal success: feature 40085355406, performance 40085355386, HTTP topology 40085355418, public boundary40085355450, CodeQL/push40085355144 and Sonar40085355448. Direct Sonar analysis 1c0d716c-7ebd-4190-ae26-6f77f48572df names this exact merge; gateOK and zero new unresolved findings since the merge. Existing historical debt is separate.

#726 remains open for published PHP acceptance with the SQLite maintenance loop running. #733 independently adopts the published Core2.13 readers and prepares a fresh Server patch version; the existing2.11.2 tag is unchanged. Original readiness diagnosis stays open in #731 and worker-registration replacement fencing stays separate in #732. No released fix or native parity completion is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant