Repository navigation
Rust Server train: command batches, named namespaces, completion checks, external inputs, operator surface, leases, deployments (#325) - #696
Merged
Conversation
GET|POST /api/system/metrics reports the namespace's workflow task failures, server-wide projection drift, and the namespace's request admission, durable-state and external payload usage, each with PHP's bounded label policy. Native enforces none of those quotas, so limits are null and rejection counters zero, as on an unconfigured PHP server. The new system-metrics fixture fails a workflow task in a named namespace, which native refused. That failure, and its retry, are scoped to the task's namespace, so named namespaces now accept it.
…d claim The published PHP SQLite reference shares its database with a one-second maintenance loop. A zero-wait poll could return no task, so the scenario recorded no claim and the failure request answered 422. The probe now long-polls for five seconds and retries the documented 503 backend_lock_pressure refusal, as the other probes do.
GET /api/system/prometheus-metrics reports the namespace's workflow, activity and task-queue series with latency histograms, task state and worker capacity, each family capped at 100 series with its suppression disclosed, as PrometheusMetricsSummary does. Latency buckets compare durations with numeric bounds on every backend. Published PHP on SQLite counts every duration in every bucket (server#683); the fixture records that reference defect without adopting it. As in PHP, an activity task scheduled by a workflow task is no longer stamped dispatched: PHP's completion publishes only timer tasks.
GET /api/system/external-payload-cleanup reports the namespace's cleanup metrics. POST .../pass renews expired completion-upload allowances whose activity can still complete and drops the rest, then reclaims the namespace's expired, unclaimed references under their object locks: a shared object keeps its file, a sole reference deletes it. Each pass is recorded in the namespace's totals; a blocked reference answers 207. Native deletes through the local driver only, honours a backup hold that PHP left active, and treats a missing hold row as no hold.
Locally, published PHP on SQLite refused a namespace delete with its retryable 503 backend_lock_pressure in two of four parity runs, and a storage diagnostic with a retryable 503 in two. The namespace-delete and payload-cleanup probes now retry lock pressure through the shared middleware, and the storage diagnostic retries a retryable 503 up to ten times.
POST /api/system/repair/pass recovers overdue ready tasks and expired leases, creates lost workflow tasks and deadline tasks, and enforces expired activity deadlines, with PHP's request rules, scope-fair selection and loop throttle. Recoveries native does not perform yet (an expired running activity lease, a missing task for a run waiting on an activity, timer, update, signal or child) are reported as candidate failures with 207. Repair status now shares the pass's candidate query, with PHP 2.11's availability rule and microsecond cutoff.
…d PHP 2.11 Published PHP claims only ready activity tasks; its repair loop settles an expired activity lease by closing the attempt as expired and either scheduling a retry (retry_reason lease_expired) or recording a terminal ActivityFailed with ActivityLeaseExpiredException. Native re-claimed the expired task as the next attempt, skipping the retry policy and its backoff. Native's claim queries now take only ready activity tasks, and its scheduler and the repair pass settle expired activity leases as PHP's enforcer does. The native-only restart probe expected re-claim of an activity without retries; it now declares two attempts and expects the lease-expired retry, matching PHP.
Both deregistration routes in published PHP revoke each activity lease the departing worker still holds (ActivityTaskLeaseRecovery::recoverForStoppedWorker): the attempt closes as expired and the activity's retry policy records ActivityRetryScheduled or ActivityFailed with failure_reason worker_deregistered and lease_revoked_at. Native released only workflow-task leases, leaving activity leases until they expired. The lease enforcer now has a transactional core shared by expiry and revocation, and both native deregistration routes call it.
…blished PHP 2.11 Native now honors DW_V2_LIMIT_PENDING_SIGNALS, _PENDING_UPDATES and _PAYLOAD_SIZE_BYTES (and their WORKFLOW_V2_LIMIT_* names) with the memo limit's casts; zero or less disables a check. Like StructuralLimits: - an admissible signal past the received-signal count or the payload size is rejected durably with structural_limit_exceeded, a null outcome and no history; an oversized one keeps no arguments; - an update past the accepted-update count or the payload size is a durable rejection with the limit's message and no arguments, and its answer (and a retried request's) carries the limit fields; - an activity result over the payload size fails the activity without a retry, with PHP's structural-limit exception and failure category. The 64 KiB update request budget is removed: PHP rejects oversized updates durably through the payload limit instead. Operator metrics now report the configured ceilings and memo limit.
…request id The structural-limit comparison showed three older update differences. PHP's update answer names the requester recorded on the command and carries rejected_at, and UpdateAccepted records request_id only when the request named one.
POST /api/activities starts an activity as a top-level job inside a host
run of type dw.standalone_activity, as StandaloneActivityStartService does:
StartAccepted, WorkflowStarted and ActivityScheduled, a pending activity
with a retry snapshot, and a dispatched activity task; no workflow task.
A closed id starts the next run; a running one, a workflow's id or another
namespace's id is refused with PHP's answers.
The activity's terminal outcome closes the host run instead of resuming
workflow code: a result completes it with the result as output, a failure
fails it with the activity's failure, and a timeout fails it as timed_out
(failed for a lost lease).
GET /api/activities/{id} describes the current host run with its attempts
from the run's activity view; GET /api/activities lists host runs newest
first by status bucket, with offset page tokens. Without input an activity
has no arguments, and its claim omits them.
The run list now reports a run's own business key before its instance's.
…ages load The PHP SDK reads the service catalog only when cluster info advertises durable-workflow.v2.service-catalog-read.contract version 1. Native served the eight catalog reads but did not advertise them, so Waterline 2.12.3 in service mode showed its Services pages as unavailable. Cluster info now carries Server 2.11.1's contract unchanged. Against the published Waterline image, every read route and the query, signal, repair and cancel actions now answer native as they answer published PHP.
POST /api/workflows/{id}/archive and /runs/{run}/archive follow
WorkflowStub::attemptArchive: an open run is refused durably with
rejected_run_not_closed; a closed run records an archive command and
ArchiveRequested, and its first archive also sets the run's archive
columns and records WorkflowArchived; a repeat answers archive_not_needed.
The reason and request_id are validated with Laravel's rules and
messages. A dw 2.3.0 sweep of 60 commands now differs from published
PHP only in redrive and the task-queue build-id lifecycle.
… that build, like published PHP 2.11
The build-ids test started a workflow on a queue whose only worker is build nv1 and expected its pending task in an unversioned row. Published 2.11.1 pins that start to nv1, and native now does too, so the pending task belongs to nv1.
Serve the task-queue build-id routes and /api/deployments over the PHP rollout rows. A promoted build pins new starts, a draining build's workers are refused polls, and a queue whose live workers all drain refuses starts. The build-ids snapshot reads the rollouts, and takeover no longer refuses a database that has them.
…refusals like PHP 2.11
… like PHP 2.11; route updates only to workers that declare them A signal keeps the runtime reference; an update or query carries the referenced bytes inline; each claims the object. Refused references get PHP's typed runtime-external-payload-error body with the route envelope, and checks follow incomingEnvelope's order. An input object naming a reference stays an object so a malformed envelope is refused. Native keeps delivering an accepted update only to a worker that advertises workflow_updates and declares it; PHP's delivery to any worker of the type is recorded as Server #690.
Native now runs PHP's completion stages: Laravel's TrimStrings (Str::trim set) and ConvertEmptyStringsToNull, WorkflowTaskCompletionValidation's transport rules in 25-command chunks with Laravel's ordering and names, digit-string integer normalization, validateWorkflowTaskCommandScopes with the activity and child timeout envelopes, payload codec canonicalization, and the normalizer's field scopes, timer, wait and version marker messages. An out-of-range marker version is the bridge's 409 invalid_commands. Bare payload strings are Avro blobs and plain continue-as-new arguments are no arguments, as PHP's normalizer reads them. Eleven older test expectations that published 2.11.1 contradicts were corrected against live probes (listed in the checkpoint).
…ike published PHP 2.11 PHP's executor leaves every run a completed task did not close waiting, even when the task recorded only version markers. Native left such a run pending.
PHP's normalizer reads a fixed set of fields per command type and ignores the rest unless a scope check refuses them, and ignores unknown retry policy keys. Native now drops a field before its own checks only when PHP neither reads nor scope-checks it for that command type: unknown fields, null options, queue on a timer, priority on an activity. Fields PHP reads stay, so native still refuses a feature it cannot run (worker_session) instead of silently ignoring it.
…e route check becomes a route table, and Laravel's rule loop gets helpers
…inal command anywhere, children alongside other work, update commands in any task, continue-as-new over open work and from a child
…workflow task completions like PHP 2.11; refuse two terminal commands with PHP's 409
… cancellation request in named namespaces like published PHP 2.11
…te upsert like published PHP 2.11
…and on activity results; report the heartbeat's run and task status
…r workflow type, like published PHP 2.11
This was referenced Oct 23, 2026
Compiling the library tests with two jobs now peaks near 4.2 GB. Under the 4 GB container limit, rustc was OOM-killed in execution-and-codec's codec step (exit 101), which stopped the later steps. A local run with the CI image and limits reproduces the kill at 4 GB and passes at 6 GB. The two database jobs compile the same library tests under the same limit, so they get the same headroom. GitHub's runners have 16 GB.
Member
Author
|
What failed
This is mostly a pre-existing flake. Five of the last ~35 completed runs before this PR had one failing codec cell, all on MySQL-family databases. Four of the five were this same busy-CPU check; the fifth was an operation deadline overrun. A/B on one 4-vCPU VM. Main
Decision. I'll merge on the required checks and the Rust suites. The cleanup-lateness regression and the pre-existing stall are both tracked for the next Rust Server PR. Every failure in both runs and on the VM is a timing or load check. A cell stops at its first failed check, so I can only say that the event and outcome checks it reached passed. |
MariaDB types COALESCE(MAX(unsigned int),0) as DECIMAL in a prepared statement, which the integer decoder refuses, so GET /api/system/metrics answered 503 on MariaDB 10.11. Read the plain MAX as an optional integer and default to 0, as schedule history already does.
Published PHP 2.11.1 stamps ActivityStarted a few milliseconds after the attempt start its deadline runs from. Its timeout enforcer can retry in that gap, which failed main's SQLite parity job once. The contract now requires the retry at or after the attempt start plus the timeout, keeps the event within a second of that start, and names the transport and margin when it fails. Decision recorded in ACTIVITY-RETRY-RECEIPTS.md.
Member
Author
|
New head MariaDB Main is red after #675 (run 39961767541, execution-and-codec). The contract now measures from the attempt start carried in the event snapshot ( codec-deadline-health. It failed on both MySQL-family cells again. Same classification as comment 6368856135; it is not a required check. |
Member
Author
|
New head On
Everything else passed too, except the classified MySQL-family |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rust Server (#325) train. Each change below was compared live with published Server 2.11.1 (
ghcr.io/durable-workflow/server@sha256:bf8bed13…). Each has an integration test that passes on SQLite, PostgreSQL and MySQL. Each has a checkpoint underdocs/rust-server-port-checkpoints/. The full native suite passes locally on SQLite, PostgreSQL and MySQL (449 tests each) at the same Rust tree.Operator surface
GET|POST /api/system/metrics, served with thesystem-metricsfixture (supersedes Serve the namespace system metrics like published PHP 2.11 #684).GET /api/system/prometheus-metrics, served with theprometheus-metricsfixture (supersedes Serve the Prometheus metrics summary like published PHP 2.11 #685).payload-cleanupfixture (supersedes Serve external payload cleanup and its pass like published PHP 2.11 #686).repair-passfixture (supersedes Serve the task repair pass like published PHP 2.11 #687).Leases and limits
worker_deregistered.DW_V2_LIMIT_*). Native's old 64-update lifetime cap and 64 KiB update budget are removed; PHP has neither.principalandrejected_at. A nullrequest_idis omitted.Run operations
POST/GET /api/activitiesandGET /api/activities/{id}.Routing
Runs are pinned to their start cohort's build, and polls give a pinned task only to that build's workers. Unpinned activities go to any worker; that contract was agreed with General on Workflow #908.
Build-id rollouts: promote, drain, resume and rollback on the task-queue routes and
/api/deployments, with:task_queue_drainingstart refusals;Takeover now accepts rollout rows.
An accepted update goes only to a worker that advertises
workflow_updatesand declares the update. Published PHP 2.11.1 gives it to any worker of the type, so a worker without the handler can fail it mid-rollout (Server #690). Main's PHP now routes the same way (Route accepted updates to workers with the matching handler #691). A new test,update_routing, covers the native rule.GET /api/task-queues/{queue}/priority-fairness.Cluster info advertises the service catalog read contract, so Waterline 2.12.3's Services pages load.
Request and response contracts
Workflow task completions whose command payloads are not complete Avro values are refused with PHP's field-scoped 422, before anything is recorded.
Workflow task completions run PHP's completion checks, before ownership as in PHP:
Malformed commands get PHP's Laravel field errors instead of native reasons. A run that a completed task leaves open is
waiting, even when the task recorded only markers. Digit strings, bare payload strings and plain continue-as-new arguments are accepted as PHP accepts them. A command field PHP neither reads nor scope-checks for that command type is ignored, as PHP ignores it; native still refuses a field PHP reads for a feature native cannot run, such asworker_session. Eleven older native test expectations that the published image contradicts were corrected against it.Unknown routes and wrong methods get PHP's 404 and 405 bodies, using an embedded copy of PHP's route table. Routes PHP serves and native does not yet still answer 501.
An empty request body is read as empty input, as PHP reads it.
Admission refusals on start, signal, update and query name the workflow, as PHP's do.
An activity heartbeat on a closed run gets PHP's 200 with
cancel_requested: true; native used to answer 409.Workflow task command batches are applied as PHP's
parseCommandsapplies them:invalid_commands;complete_updateandfail_updatefollow PHP'svalidateUpdateCommands, so an update completes alongside other commands and an update task may leave its update for a later task;workflow_typeandqueueapply to the new run.Native refused each of these before. Successful completions now carry PHP's
created_task_ids,protocol_versionandserver_capabilities. Workflow task failures and heartbeats, and activity results and failures, also carry the worker envelope. A workflow task heartbeat also reports the run's and the task's status.Named namespaces now serve every workflow command, plus update, repair, redrive and cooperative cancellation requests, as PHP does. Native used to refuse them with
development_namespace_command_unqualifiedordevelopment_namespace_operation_unqualified. Update admission and cancellation requests no longer look the instance up in the default namespace.An
upsert_search_attributescommand needs the worker'styped_search_attributescapability even withoutattribute_types, as PHP requires.Signal, update and query input accept a runtime external payload reference, as start input does. A signal keeps the reference; an update or query carries the bytes inline. Refused references get PHP's typed
runtime-external-payload-errorbody, which SDKs read to decide whether to retry.CLI and Waterline
dw2.3.0 CLI matches PHP's exit code on 58 of its 60 sweep commands. The other two are not gaps: a lock refusal on the SQLite reference, and a query timing race.Sonar maintainability findings (General's request on #325)
'staticlifetimes inbackend.rs, the wildcard import inquery_poll_requests.rsand a closure inactivity_metadata.rsare fixed.CI
execution-and-codec. The three test-compile steps now get 6 GB. A local run with the CI image and limits reproduced the kill at 4 GB and passed at 6 GB.Not in this train: #673 (under Security review).