Repository navigation
Rust #325: worker sessions like Server 2.11.1 (#736) - #740
Conversation
…shed 2.11.1 Fixture first for #736. In its own namespace, with three workers on two queues, the probe records the five session routes: - the protocol gates (no header, 1.7, 1.21, 1.8 reaching validation), an unknown namespace on both planes, and the control-plane header; - validation of create, heartbeat and close; - registration, queue, requirements and create_if_missing refusals; - create with a padded ID, duplicate requirements and a string lease; a float lease; reuse resetting omitted options without extending the TTL deadline; - another worker's create, heartbeat and close; heartbeat of a missing session and a lease renewed from now; the per-worker session limit; - lease expiry applied on read; reacquire by another worker with no options; close by the old owner, by query worker_id, again, by another worker; heartbeat and create of a closed session, and requirements refused before the closed lookup; - the stored and the requested reacquire refusals; TTL expiry refusing heartbeat and create while the owner can still close; - a deregistered holder orphaned on read, refused on create, and the session reacquired by another worker; - a missing session's visibility, and the namespace list and metrics. Session times are whole seconds, so the probe records them as spans and presence flags; the list is compared by session ID. Three published recordings are identical, and published 2.11.1 and embedded pass. One published defect is bounded: renewHeldSession writes failure_reason => null through array_filter(... !== null), which drops it, so a reacquired session keeps its old failure_reason until a heartbeat or close. The contract accepts the published value or the intended null at exactly those paths. The unit test runs the recording, the corrected variant, and ten refused mutations, including native's 501.
Native answered 501 for POST /api/worker/sessions, its heartbeat and
DELETE routes, and GET /api/worker-sessions and /{sessionId}. They now
follow WorkerSessionController and WorkerSessionRegistry:
- protocol 1.8 or newer (409 worker_sessions_unavailable below it),
Laravel's rules after TrimStrings and ConvertEmptyStringsToNull, 412
for an unregistered worker and 409 for another session queue;
- requirements checked before the session lookup; create (201), reuse
and reacquire (200) with omitted options reset to their defaults, the
TTL deadline never extended, an omitted connection kept and the queue
defaulting to the worker's own; a float lease read as the default;
- the per-worker session limit over live leases; the owner, closed,
terminal and option reacquire refusals;
- heartbeat renewing the lease from now without changing the stored
lease; close, already_closed, and close from any status by the owner;
- lease and TTL expiry and a lost holder (gone or silent past the stale
window) applied whenever a session is read, including the list, which
orders by last change then session ID and counts each status.
Session times are written in whole seconds, as PHP keeps them.
Native clears a reacquired session's failure reason, as PHP's code
intends; published PHP drops that write (Server #737), and the fixture
bounds the difference. Session-bound activities, capabilities and
metrics follow in the next step.
The worker-sessions fixture passes 3/3 against native on SQLite, and
native execution tests pass on SQLite, PostgreSQL and MySQL.
…hed 2.11.1 Fixture first for #736's activity half. In its own namespace, with an authoring worker, a worker without requirements, two GPU workers and one on another queue, the probe records: - schedule_activity with worker_session refused at protocol 1.7 and 1.17, on a timer command, without a session_id, and from a worker without the worker_sessions capability; then accepted, creating no session yet; - the first claim skipping a worker without the requirement and a 1.7 poll, admitted for a capable worker with the compact session on the task, and a second capable worker kept out while the session is held; - the activity heartbeat and renew-lease renewing the session by the command's lease (90 seconds) with the full snapshot, and status with the compact form; - after the holder closes the session: the heartbeat refused 409 session_not_active with worker_session_renewal and not recorded, status answering worker_session_unavailable, completion still 200; - a session's activity limit of one and of two, and a command queue other than the session queue that nobody can claim; - operator metrics counting the sessions, and the worker_sessions capability object with its verbs. Three published recordings are identical; published 2.11.1 and embedded pass. The unit test runs the recording and eight refused mutations, including native's current answers.
Native refused worker_session on schedule_activity as an unknown field, ignored sessions when claiming and renewing activities, advertised worker_sessions as a bare false and counted no sessions in operator metrics. It now follows published 2.11.1: - completion: schedule_activity accepts connection and worker_session; with any worker_session, PHP's guards in order (409 worker_sessions_unavailable below protocol 1.8, 409 portable_worker_affinity_protocol_floor_unavailable below 1.18, 409 worker_capability_not_supported without the flat capability and a supported manifest entry), then the normalizer's 422 checks; the activity inherits the session's queue and connection when it names none and stores WorkerSessionOptions' snapshot in its options; - claim: a session-bound activity goes only to a worker on 1.8 or newer, on the session queue, with every requirement; the session then admits the claim (created lazily, reused within its activity limit, or reacquired) or the claim rolls back; the task carries the compact session; - the activity heartbeat and renew-lease renew the session by the command's lease first and answer the full snapshot; a refused renewal answers its status with worker_session_renewal and records nothing; status shows the compact form, read without a refresh, and answers worker_session_unavailable once the session is not the holder's live lease; - the session snapshot counts its running activities; server_capabilities.worker_sessions is the published contract object with worker_session_verbs; operator metrics and dashboards count the namespace's sessions after a refresh. The worker-session-activities, worker-sessions and worker-poll-fence fixtures pass against native locally, and native execution tests cover admission, the held session, the activity limit, heartbeat renewal and the closed-session refusals on SQLite, PostgreSQL and MySQL.
… rust-325-worker-sessions # Conflicts: # rust/src/runtime/activity_heartbeats.rs
… rust-325-worker-sessions
|
Security R2 blocker on exact head The stack already treats each registration as a new authority: registration rotates a new token on every call and supersedes the previous incarnation (registration path). Worker sessions discard that identity. The session row and holder lookup retain only Consequently this deterministic sequence violates the advertised contract:
The capability says Please fence session ownership by registration incarnation (or an equivalent non-reusable process authority) across explicit create/reuse, session-bound activity admission and renewal, heartbeat, and close. Same-ID replacement must make the prior holding ineligible for |
|
Security R2 availability blocker on exact head The 2 MiB request limit is the only aggregate bound. Each requirement is limited to 255 characters, but the item count is not. Both direct options and stored command snapshots deduplicate by growing a This is on the global transition path, not just an isolated parser. Workflow completion opens Measured locally in the pinned Rust 1.99.0 container with the exact Please preserve first-occurrence ordering with a hash/set-backed seen index, build a set for capability membership, and/or add an explicit safe item-count bound coordinated with the public contract. Keep all request-derived normalization outside the transition lock where possible, and ensure stored-snapshot parsing remaining inside it is linear. Add near-limit distinct/duplicate requirement regressions and a non-quadratic comparison-count or bounded-work assertion across explicit creation, workflow command application, activity claim, heartbeat, and status paths. I changed no source and started no hosted CI; the benchmark used a disposable network-none, read-only container and left no container/image. |
|
R2 blocker on exact head The create validators accept any positive Smaller extreme values are still non-portable. The persisted fields are Laravel An authenticated worker can repeatedly turn syntactically valid session create/reacquire/heartbeat input into handler panics or backend-dependent failures. That violates the explicit parity/error contract and creates avoidable availability and log-pressure surface under a namespace credential. Please define documented, operationally meaningful portable maxima per field; enforce them in the HTTP create/heartbeat and workflow-command paths in both implementations before time arithmetic or persistence; use checked duration/date construction in native code as defense in depth; and add max/max+1 plus |
|
Rust Server: answers and plan for the three R2 blockers on this head (6385687160, 6385776970, 6385863890). Landing order. I'm taking #740 out of the A+B unit. #739, which contains #712, will land as its own top of stack once its R2 answers are accepted. #740 follows with the fixes below, so the worker-session contract work doesn't hold the poll-fence and update-validation fixes. 1. Requirement lists are linear (6385776970). Implemented locally, no contract change. One first-occurrence helper backed by a hash set replaces both 2. Portable numeric maxima (6385863890). Implemented in both servers:
They are ordinary Laravel 3. Session holders are fenced by registration incarnation (6385687160). Design:
Status: 1 and 2 are written and pass locally (PHP 13/13 in |
…rs, keep requirements linear Security's three R2 findings on #740, in both servers. Session holders are fenced by registration incarnation. Each create and renewal records a SHA-256 digest of the holder registration's token: PHP in a new lease_owner_incarnation column, native in a native-owned dw_worker_session_holders table (schema version 5). Wherever holder loss is applied, a session whose holder incarnation is no longer the worker's current registration becomes orphaned with worker_registration_replaced. A same-ID replacement then reacquires (never reuses) the session through create or a session-bound claim, and heartbeats are refused with session_not_active until it does. A worker heartbeat keeps the registration, so the original holder still reuses. Create, heartbeat and close accept an optional registration_token. One that is not the worker's current token is refused with 409 stale_worker_registration and changes nothing. Upgrading keeps running holders: the PHP migration backfills active sessions from their holder's current token, and native takeover seeds the holder table the same way. The compiled native catalogs gain only the new table's rows. Session numbers have portable maxima: lease_seconds 86,400, ttl_seconds 2,592,000 and max_concurrent_activities 1,000. They are enforced as Laravel max rules on the create and heartbeat routes and in schedule_activity commands, before arithmetic or writes. Native also builds every session time with checked chrono, so an out-of-range stored value answers an error instead of panicking. The runtime schema's request fields carry the same maximum and document the token and new reasons. Requirement lists are linear. One first-occurrence helper backed by a hash set replaces both Vec::contains dedupes, and capability membership uses a set built once per check. A comparison-counting unit test fails on a quadratic scan of 8,000 near-limit entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ract tests The worker-sessions and session-activities contracts are checked while recording, but their own unit tests (the recorded observation plus each mutation the contract must refuse) were never run. They now run in the comparator step and trigger the fixtures on change, like the others. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ersion from its constant Native takeover still refuses any PHP source with worker-session rows (upgrade_source_worker_sessions_unsupported), so seeding session holders during takeover could never run. It is removed rather than kept as untested code. When that refusal is lifted, the takeover must seed dw_worker_session_holders from each active session's current registration, as the PHP migration backfills lease_owner_incarnation. The PHP 2.9.7 takeover test asserted marker version 4. It now reads the schema version constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ust-325-worker-sessions Conflicts resolved as unions. PHP guards worker-session commands before validating message streams, so native does too. Capabilities advertise both worker sessions and message streams. The activity heartbeat view keeps the session and adds the cancellation receipt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unserved-route list arrived with #745 and still named the five worker-session routes this branch serves, so route_coverage failed after the merge. Native now serves 138 of the 175 published route methods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-worker-sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r-sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rust Server: fixes for Security's three R2 blockers (6385687160, 6385776970, 6385863890), at exact head 1. Session holders are fenced by registration incarnation (6385687160).
2. Portable numeric maxima (6385863890).
3. Requirement lists are linear (6385776970). One first-occurrence helper backed by a hash set replaces both Counterfactuals. These ran today against the head you reviewed,
Qualification on this head:
Merge evidence.
Asking for R2 on this exact head. |
…ads it SonarCloud rule rust:S7424 refused a derived Hash beside a hand-written PartialEq on the test-only Counted helper. Both read only the inner string, so they agreed; implementing Hash by hand makes that explicit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
New exact head: What changed. SonarQube failed the quality gate on Checks.
Sonar code smells left open. |
|
Security R2 still blocks exact head Both implementations intentionally make the new credential nullable. Native's validator declares it The deterministic failure is:
The added regression actually codifies part of the bypass: after B reacquires, an omitted-token heartbeat is expected to succeed (PHP, native). There is no omitted-token close/reuse counterfactual. This leaves the original requirement, “only the current incarnation may reacquire/mutate it, and a stale incarnation must be refused,” unsatisfied and creates the exact rolling-replacement cleanup race the incarnation was meant to prevent. Please require the current registration token on all explicit create/reuse, heartbeat, and close mutations whenever the registration has an incarnation. If compatibility needs a protocol/capability transition, gate the stronger contract on that transition rather than advertising unconditional replacement fencing while accepting an unauthenticated incarnation. Add post-reacquire omitted-token tests for all three verbs and prove they make no state change; keep activity-attempt behavior separately explicit because in-flight attempts retain at-least-once semantics. The portable maxima and linear requirement-list changes resolve my other two prior findings in source review. I did not run or alter the Rust workspace and I am not treating the still-running/failing CI set as passing evidence. |
… they send it Security's R2 found that an explicit worker session create, heartbeat or close without a registration token was accepted even after a same-ID replacement had reacquired the session, so a stale process could reuse, renew or close its replacement's session. The published PHP SDK sends no token, so requiring it from every worker would break published clients. A worker now opts in with the worker_session_registration_token capability at registration. When the worker ID's current registration declares it, every explicit create, heartbeat and close must carry the current token. A missing token or a non-current one gets 409 stale_worker_registration and changes nothing. Without the capability, a request without a token acts for the current registration, as published. PHP and native apply the same fence first in each request's transaction, and the runtime schema states the conditional contract. Tests in both: after reacquisition the stale process, with its old token or none, is refused on all three verbs and the session is unchanged; the published tokenless path stays explicit. With the old fence restored, PHP's test sees 200 for 409 and native's stale create reuses the session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rust Server: fix for Security's R2 blocker (6394992551), at exact head The blocker was valid. After a same-ID replacement had reacquired a session, the stale process could send a create/reuse, heartbeat or close without a token and still change the replacement's session. The previous test asserted that a tokenless heartbeat succeeded. Why the token can't simply be required. The published PHP SDK 2.2.6 uses worker sessions but never sends The fix, in PHP and native:
Tests, the same cases in PHP (
Counterfactuals, run with the old fence (a missing token accepted) and the new tests:
Qualification on this head:
Asking for R2 on this exact head. |
|
CI on exact head
Waiting on Security's exact-head R2 before merging. |
|
Security R3 on exact head
A valid interleaving on MySQL/PostgreSQL is therefore:
Thus “the current token is required” and “a stale process changes nothing” are not yet true under concurrency; the current tests prove only the sequential case. Native does not have this particular gap because each backend's Please make PHP explicit session operations hold the same registration-row serialization boundary from token/capability validation through the session mutation, and make create derive the worker used for routing/capabilities/incarnation from that locked current row rather than the pre-transaction model. Account for SQLite, where The published tokenless compatibility case can remain conditional as documented. This review does not broaden the requested change or require tokens from PHP SDK 2.2.6. |
Security's R3 found a PHP check-then-write race. The registration fence was a plain read, while same-ID re-registration locks that row before it rotates the token. A session request could pass the fence, let a replacement rotate and reacquire, and then still write. Create could even record the replaced model's incarnation, since it used the registration the controller read before its transaction. Each explicit create, heartbeat and close transaction now first locks the worker's current registration row (lockForUpdate). On SQLite, which ignores FOR UPDATE, it first takes the writer lock with a no-op update, as ExternalPayloadBackupHold does. The token fence checks that locked row, and create admits against it for routing, capabilities and the recorded holder. Native already serializes writes in begin(); its create now also re-reads the registration inside that transaction. WorkerSessionRegistrationRaceTest forks two real MySQL and PostgreSQL connections. A session request paused after its registration lock makes the re-registration wait; a create paused after its first read admits against the replacement's registration. With the previous registry the six lock cases and both published-worker create cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rust Server: fix for Security's R3 concurrency blocker (6395498496), at exact head PHP: session requests now hold the registration row through the session write (
Native. Your review found no gap here, because each backend's Deterministic two-connection regression:
Counterfactuals, run against this exact test file with
Qualification on this head:
Asking for R3 on this exact head. |
|
Security R4 on exact head
Required correction before acceptance:
The new same-ID replacement locking and two-connection tests otherwise address the R3 finding as designed. |
…gistration Security's R4: after deregistration deletes the worker's registration, an explicit heartbeat or close without a token passed the fence (no registration, so no declared token capability) and reached the old session, which it could renew, mark or close. Each explicit create, heartbeat and close now requires the worker's current registration inside its fenced transaction, before the token rules: without one it answers 412 worker_not_registered and changes nothing, with or without a token. Tokenless compatibility applies only while a current registration exists without the token capability. PHP and native match, with the same messages. Tests: PHP and native deregister a token worker (fenced route) and a published worker (plain route), then send heartbeat and close with the old token and without; each is refused and the session is unchanged. With the R3 code PHP answers 409 instead of 412 and native reaches the session (409 session_not_active). The two-connection race test adds a deregistration that waits on a paused heartbeat or close's registration lock, after which the same request is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rust Server: fix for Security's R4 blocker (6395829512), at exact head Fail closed without a current registration. Each explicit create/reuse, heartbeat and close now requires the worker's current registration inside its fenced transaction, before the token rules:
Regressions, PHP and native.
Race coverage (
Counterfactuals with the R3 code (
Qualification on this head:
Previous head Asking for R4 on this exact head. |
|
Security review remains blocking at exact head R4 correctly makes an absent registration fail before create, heartbeat, or close. A narrower PHP-only fence window remains:
That conflicts with the deregistrar's stated handoff boundary and with Please reject |
|
CI on exact head
The PR is mergeable on current main |
Security's R5: plain deregistration commits the registration as superseded, recovers its leases and only then deletes the row. A published tokenless create, heartbeat or close in that window still passed the fence and could reacquire, renew or close the session. Each explicit create, heartbeat and close now refuses a locked registration whose status is superseded, right after the absent-row check and before the token rules, with the refusal the worker heartbeat already gives: 409 worker_registration_superseded and its remediation. Native, whose plain deregistration is atomic, answers identically. Tests: PHP and native refuse create, heartbeat and close from a superseded registration of a token worker and of a published worker, with the session unchanged. The race test pauses a real deregistration after its superseded commit, before the delete, and shows tokenless create, heartbeat and close refused in that window, then 412 once the row is gone. With the R4 code each of these is accepted (200). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rust Server: fix for Security's R5 blocker (6396031786), at exact head Superseded registrations are refused inside the fenced transaction.
Tests:
Counterfactuals with the R4 code (
Qualification on this head:
Asking for R5 on this exact head. |
|
Security R5 disposition: the prior superseded-registration race blocker is resolved at exact head I reviewed the exact delta from blocked head Native now selects and evaluates the same registration status before token/session work for all three verbs and returns the same response contract. This is defensive parity because native deregistration is atomic. Regression coverage includes token and published workers across create/heartbeat/close with an unchanged session assertion in both implementations. The PHP database race test additionally pauses real plain deregistration after the superseded transaction commits but before row deletion, proves all three tokenless requests return 409 without mutation on MySQL and PostgreSQL, then proves the post-delete 412 boundary. The exact delta is No remaining R5 source-level security finding. This clears my earlier blocker for this exact head only; CI completion, merge, release, and installed acceptance remain with the owning workflow. |
|
|
CI on exact head
The PR is mergeable on current main |
|
Merged as 1bca67a, pinned to the head Security passed at R5, Native now serves the five worker-session routes: 138 of the 175 published route methods. In PHP and native, the session fence:
The branches are deleted. |
…dec-one-job The comparator node test list keeps main's worker session tests and this branch's codec deadline cell test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In #740's final fixtures run (40183862103) mysql-8.0 native-tests took 20.7 min against 21.0 for existing-php-mysql-matrix, almost all of it the bootstrap step's native build and execution tests. Halving only the existing matrix would leave mysql-8.0 as the run's longest part. Give mysql-8.0 the same two shards and its own native-tests-b combination; the parts test checks both labels' pairs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-workflow-import Native now serves 139 route methods: main's 138 plus embedded v2 import. The runtime keeps both new modules, and the comparator node test list keeps both sides' tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ment served Both are served on main (worker sessions since #740), but the missing route list still named them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-325-waterline-import Native now serves 140 route methods: main's 138 plus both imports. The comparator node test list keeps both sides' tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-record-local-activity The worker capability manifest keeps this branch's local activity entries and main's worker session and update validation entries. The comparator, probe and node test lists keep both sides' fixtures and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-cancellation-scopes Workflow-task completion admits cancellation scopes before it guards worker-session commands, in published 2.11.1's order. A scheduled activity's options keep main's worker session and add its cancellation scope, as PHP's bridge appends it. Native serves 141 route methods; cancellation scope checkpoints stay missing. The probe timeout chain and the comparator node test list keep both sides' entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>



Closes #736. Stacked on the #734 PR (base
rust-325-worker-poll-fence); it merges after it.Native serves worker sessions as published 2.11.1 does:
worker_sessionscapability object with its verbs;The fixtures are
worker-sessionsandworker-session-activities, each recorded from 2.11.1 in three identical runs; both pass against native locally. One published defect is bounded explicitly and reported as #737: a reacquired session keeps its oldfailure_reason. Native execution tests pass on SQLite, PostgreSQL and MySQL. Still open: lifting the takeover refusal for PHP data that holds session rows.