perf(core): publish fan-out step messages before their creates commit; resilient step dispatch on by default by pranaygp · Pull Request #4102 · vercel/workflow

pranaygp · 2026-09-11T08:33:04Z

Motivation

Today a clean fan-out folds its eager step_created writes into createBatch chunks and publishes each chunk's queue messages only after that chunk commits. Resilient step dispatch (WORKFLOW_RESILIENT_STEP_DISPATCH=1, off by default since #3519, introduced in #3365) publishes in parallel with the create and carries the input on the message, but it was mutually exclusive with the batched fold (batchFanoutEligible required !resilientDispatchEligible), and its consumer recovery cost three round trips (bare start, 404, step_created write, bare start again).

Measured on 32-branch fan-outs: queue messages left 100-400 ms after the suspension began because they waited on chunk commits (155-182 ms each), and the first queued step started ~440-650 ms after run_created.

This PR composes the two so the fold publishes first, and turns resilient dispatch on by default.

Design

Producer (suspension-handler.ts). The !resilientDispatchEligible term is gone from batchFanoutEligible. When both the fold and resilient dispatch are eligible, each folded kind: 'step' entry whose dehydrated input is a Uint8Array of at most MAX_RESILIENT_STEP_INPUT_BYTES is published immediately by its prep op, once ensureRunReady() has settled (turbo still gates on run_started), with stepInput: { input }, the same stepDispatchIdempotencyKey / traceCarrier / requestedAt shape as before. Those sends run concurrently with the createBatch POSTs rather than after a chunk commits. Entries are flagged earlyPublished, and publishChunkSteps skips them; steps whose input is too large or non-binary keep publish-after-chunk-commit. Early publishes are recorded in queuedStepCorrelationIds up front and joined by the flush's trailing promise (so they ride deferredBatchWork, which the caller joins before ack; without the opt-in, the flush awaits them at return). The single-entry path joins them too via settlePhase. A publish failure stays fatal for the pass. The pair rows (inline-created/inline-started) are untouched. Outside the fold (hooks/attrs in the suspension, no createBatch, pre-slot-identity run) the per-step create ∥ publish branch applies unchanged, and it remains the only place resilientDispatchRecovered increments (in the fold a chunk failure is fatal as before, so there is no "recovered" outcome to count).

Consumer (runtime.ts). When a stepInput-carrying message hits step-missing on the bare step_started, the executor is re-run once with lazyStepInput: stepInput.input (and suppressOptimisticStart: true, so the claim is awaited and a 409 is seen before any body runs). Every World turns that into an atomic create + start with a synthetic step_created (#2478). If the lazy start returns skipped (409), the step now exists but the same 409 covers two winners, so the consumer reads the step entity (world.steps.get, resolveData: 'none', on this rare path only) and arbitrates: a pending step means the producer's create landed in between and one more bare start runs it; a running or terminal step means a peer delivery on another instance won the lazy create-claim and is executing the body, so this delivery acks as the loser without executing (runStepSingleFlight only serializes one process). RunExpiredError → gone is preserved. The deployment-affinity requeue of a queued step forwards the incoming stepInput, so a publish-first delivery misrouted before its create commits can still materialize the step on the pinned deployment. The eager redelivery pre-ensure (metadata.attempt > 1) is removed: the in-band path covers a redelivered dispatch whose step is missing at the same round-trip count the eager step_created write did, and a redelivery whose step exists (crash mid-body, throttle, the common case) no longer pays a conditional write that only ever came back 409. That also removes the last viaStepDispatch sender; nothing sends it on a step_started (the server rejects it on any event type other than step_created).

quickjs entrypoint. dispatchPendingOps has no createBatch fold (it never calls createBatch), so its existing per-step create ∥ publish branch is already publish-in-parallel; there is nothing to compose. The default flip turns that branch on for it too.

Default (constants.ts). isResilientStepDispatchEnabled() defaults on; WORKFLOW_RESILIENT_STEP_DISPATCH=0 or false (case-insensitive) disables it, mirroring isBatchTransitionsEnabled. This also means world-local and world-postgres (no createBatch) now publish overflow steps in parallel with their creates by default.

Safety analysis

The hazard that justified "off" was a World returning 412 for a stale step_created while the payload-carrying message was already out. Since [core] Drop pre-slot event ID support and preconditionGuard capability #3519 no World in this repository returns 412, the Vercel backend skips its precondition check entirely for slot-identity runs, and a duplicate-create 409 leaves the message deduped against the winner's dispatch on the shared idempotency key. The window no longer exists on slot-identity runs. The kill switch stays for operators running a World that does refuse.
A lazy step_started on an existing step is a 409 → skipped on every World, deliberately (it is the inline path's exactly-one-owner gate). So the lazy start is only ever the recovery for a step-missing bare start, never the first attempt, and retries/redeliveries of a materialized step keep the bare start.
Publishes are initiated early, not ordered before the creates. Each early publish runs concurrently with the createBatch commits and is joined alongside them (the flush's trailing work); both must succeed before the delivery acks. A durable step_created therefore does NOT by itself prove its message was published: a slow or failed send can leave a committed create whose message only goes out on the next replay's re-dispatch (deduped by the idempotency key). The unconditional re-enqueue of pending steps stays load-bearing. Removing it would need either a real publish-before-create ordering (the create waits for the send's acknowledgment, giving back the latency this PR removes) or a durable publish marker the replay can consult; neither is in this PR.
A delivery that beats a fold whose chunk later fails: the consumer materializes the step from the message, the orchestrator redelivers, and the fold's re-create is a tolerated 409. Same convergence as the per-step branch's transient case.
Turbo: early publishes still gate on ensureRunReady().
Composition with Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue #3457 (run identity on step-dispatch messages). The early publish-first sends stamp runContext exactly like every other producer, so a publish-first delivery takes the fetch-free prologue: with the resilient recovery in-band, a first delivery reads and writes NOTHING before its step_started claim. Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue #3457's eager redelivery re-ensure (ensureStepFromMessage) is gone on both prologues; the in-band lazy start covers a redelivered dispatch whose step is missing at the same round-trip count. The deployment-affinity re-route preserves both stepInput and runContext; the lazy-start recovery and the post-409 step read take deploymentId/specVersion from runContext when present and from the legacy runs.get otherwise.

Out of scope

Lazy queued creation without any step_created (dropping the create entirely for queued steps).
Dropping the unconditional re-enqueue (see the safety analysis: this PR does not establish the ordering that would justify it).
CreateEventParams.viaStepDispatch stays in @workflow/world (advisory, Worlds may ignore it); its doc comment still describes the old opt-in default and can be refreshed in a world-side follow-up.

Tests

packages/core/src/runtime/suspension-handler.test.ts:

(a) publish-first: all 33 foldable messages are out, each with stepInput, while all three gated createBatch calls (the pairs-only chunk from perf(core): commit pre-claimed inline pairs in their own batch chunk #4098, then 32 + 1 plain) are still pending; the message bytes are the create's bytes; no second send after the chunks commit.
(b) oversize input (random 200 KiB) falls back to publish-after-commit without stepInput, still reported in queuedStepCorrelationIds.
(c) WORKFLOW_RESILIENT_STEP_DISPATCH=0 keeps today's per-chunk ordering (the former ordering test, now under the kill switch, asserting no stepInput).
(d) queuedStepCorrelationIds covers every early publish at return time, before the deferred work settles.
Early-publish failure surfaces through deferredBatchWork (opt-in) or rejects the handler (no opt-in); a lone eager step publishes ahead of its single-path create and the send is joined.
Every early send carries runContext (deploymentId, specVersion, startedAt, rootRunId) and no persisted event data does; Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue #3457's queueBatch-shaped assertion of the same now runs under WORKFLOW_RESILIENT_STEP_DISPATCH=0, where publish-after-create is the only path that batches sends.
Composition with perf(core): commit pre-claimed inline pairs in their own batch chunk #4098's commitSingle: 2 inline + 1 eager, the lone plain entry beside the pair chunk publishes first with stepInput, its guarded events.create (not a one-row createBatch) is joined by deferredBatchWork along with the early send, and the single path never publishes it a second time. The same shape under WORKFLOW_RESILIENT_STEP_DISPATCH=0 keeps perf(core): commit pre-claimed inline pairs in their own batch chunk #4098's publish-after-create assertion.
Default-on and 0/false/FALSE kill-switch cases in the resilient describe.

packages/core/src/runtime.test.ts (consumer):

(e) step-missing → one lazy step_started carrying the input, no step_created write, no viaStepDispatch on any event, no ownership stamp, body runs once (world-vercel and world-local error shapes).
(f) lazy start skipped (409) → the arbitrating step read: pending → one more bare start runs the body once; running / completed / failed → acked as the loser with nothing further written and the body never run; the read never happens when the lazy start wins or the bare start succeeds.
Deployment-affinity re-route of a queued step carries the incoming stepInput and runContext on the re-enqueued message (alongside the existing fields, still without runInput); Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue #3457's parameterized re-route test covers both prologues.
With runContext: zero reads before the step (no runs.get), and the in-band recovery is the one lazy start (bare start, lazy start, body), no step_created write; Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue #3457's fan-out last-completer tests (exactly one lazy runs.get before the inline replay) pass unchanged.
(g) redelivery (attempt > 1) recovers in-band with no eager step_created; existing-step deliveries on attempts 1 and 2 write nothing extra; legacy no-stepInput paths unchanged.

cd packages/core && FORCE_COLOR=0 pnpm test (after rebasing onto main at 03455a2, #3457): 114 files passed, 1 skipped; 2471 tests passed, 3 expected fail, 1 skipped. pnpm typecheck for core is clean. No e2e run.

Docs Preview

Preview deployment: workflow-docs (behind deployment protection; requires Vercel team access).

Page	v5
`configuration/runtime-tuning` (`WORKFLOW_RESILIENT_STEP_DISPATCH` section)	https://workflow-docs-git-pgp-publish-first-fanout.vercel.sh/docs/configuration/runtime-tuning#workflow_resilient_step_dispatch
`changelog/batched-event-writes` (runtime integration section)	https://workflow-docs-git-pgp-publish-first-fanout.vercel.sh/docs/changelog/batched-event-writes#the-runtime-integration-suspension-fan-out-fold

No v4 page carries this text, so nothing was mirrored.

🤖 Generated with Claude Code

changeset-bot · 2026-09-11T08:33:09Z

🦋 Changeset detected

Latest commit: e408ace

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@workflow/core	Minor
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Minor
@workflow/world-testing	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-09-11T08:33:21Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
example-nextjs-workflow-turbopack	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
example-nextjs-workflow-webpack	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
example-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-astro-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-express-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-fastify-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-hono-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-nestjs-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-nitro-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-nuxt-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-python-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-sveltekit-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-tanstack-start-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workbench-vite-workflow	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workflow-docs	Building	Preview, v0	Sep 12, 2026 1:12am UTC
workflow-swc-playground	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workflow-tarballs	Ready	Preview, v0	Sep 12, 2026 1:12am UTC
workflow-web	Ready	Preview, v0	Sep 12, 2026 1:12am UTC

github-actions · 2026-09-11T08:33:21Z

📊 Workflow Benchmarks

commit e408ace · Sat, 12 Sep 2026 01:38:02 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	517 (-63%) 💚	1767 🔴 (+11%)	1798 🔴 (+11%)	1830 🔴 (+5.8%)	30
TTFS	stream	220 (+18%) 🔻	1752 🔴 (+16%) 🔻	1801 🔴 (+16%) 🔻	1880 🔴 (+19%) 🔻	30
TTFS	hook + stream	1725 (+32%) 🔻	2144 🔴 (+19%) 🔻	2170 🔴 (+16%) 🔻	2375 🔴 (+14%)	30
Fan-out TTFS	Promise.all(100 steps)	664 (+53%) 🔻	1243 (+90%) 🔻	1286 (+66%) 🔻	2787 (+50%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2159 (+10%)	7757 (+229%) 🔻	9698 (+310%) 🔻	135205 (+1608%) 🔻	10
STSO	1020 steps (inline)	110 (-8.3%)	148 (+4.2%)	168 (+5.0%)	270 (+2.3%)	1019
WO	1020 steps	149859 (+3.4%)	149859 (+3.4%)	149859 (+3.4%)	149859 (+3.4%)	1
CRTT	first chunk (pooled)	74 (-1.3%)	105 (-24%) 💚	143 (-15%)	325 (+8.0%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	99.5 (-22%)	274 (+13%)	484 (+20%)	805 (-8%)	214 (-17%)	10
size sweep (100/s, 160B-12KB)	95 (+2%)	197 (-29%)	336 (-1%)	503 (+5%)	141 (-42%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	104 (-17%)	188 (-27%)	238 (-86%)	698 (-79%)	256 (-40%)	3
replay eve-gpt-5.6-sol-2000t (1x)	213 (+76%)	180 (-17%)	343 (±0%)	4400 (+586%)	4112 (+667%)	2
replay eve-gpt-5.6-sol-2000t (2x)	94 (-13%)	226 (-66%)	328 (-72%)	979 (-40%)	485 (-19%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 144758ms → this run 149679ms (Δ +4921ms, +3%)

100-150 ms  █████████████████████┃██  main 856  this 779   -77
150-200 ms  ████░┃                    main 144  this 204   +60
200-250 ms  ┃                         main   8  this  20   +12
250-300 ms  ┃                         main   3  this   9    +6
300-350 ms  ┃                         main   3  this   1    -2
350-400 ms  ┃                         main   2  this   3    +1
400-450 ms  ┃                         main   2  this   1    -1
450-500 ms  ┃                         main   1  this   0    -1
500-550 ms  ┃                         main   0  this   1    +1
550-600 ms  ┃                         main   0  this   1    +1

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant  RTT 1ms→5s+             avg         p50         p90           p99     n
control  ······▁█▄▁···   192.8 (-2%)   156 (-8%)  484 (+20%)     805 (-8%)  3000
sweep    ······▁█▂▁···    161 (-19%)  146 (-22%)   336 (-1%)     503 (+5%)  3000
gw 1x    ·····▁▂█▂▁···  150.3 (-46%)  132 (-21%)  238 (-86%)    698 (-79%)  5295
eve 1x   ·····▁▃█▁▁▁▁·    357 (+88%)  124 (-22%)   343 (±0%)  4400 (+586%)  5186
eve 2x   ·····▁▂█▃▁▁··  186.3 (-51%)  153 (-47%)  328 (-72%)    979 (-40%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  ▁▄█▇▅▃▂▁▄▃  149–260ms
sweep    ▇▅██▆▃▃▂▁▃  129–190ms
gw 1x    █▆▁▂▃▂▁▁▂▃  139–180ms
eve 1x   ▁▁▁▁▁▁▁▁▁█  122–2349ms
eve 2x   ▂▂▂▁▁▃▄█▃▅  131–319ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  ▂▇▁▂█▃▂  160–163ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▁▇█▆▂▄▂▁▅▃  49–77ms
sweep    ▇▆█▄▆▅▄▁▃▄  49–75ms
gw 1x    █▄▂▁▆█▅▃▆▂  39–51ms
eve 1x   ▁▁▁▁▁▁▁▁▁█  23–172ms
eve 2x   █▃▁▁▁▄▃▄▇▅  24–32ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

github-actions · 2026-09-11T08:33:24Z

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

abortTimeoutWorkflow: timeout cancels long-running step (python · vercel-prod / vercel / node / production)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

cold-start-warmup · suite warmup (tanstack-start) · at 01:13:30Z · abandoned wrun_01M29JSHZ3FQA5TAZN0FFQ72ZY
run-pickup-stall · sleepWinsRaceWorkflow (vite) · at 01:14:34Z · abandoned wrun_01M29JVZ79HERM5PFBAT03G26X
run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 01:20:51Z · abandoned wrun_01M29K7F7QGGDPXTS7NTZ9GVEB

E2E Test Summary

Summary

	Passed	Skipped	Total
✅ ▲ Vercel Production	3662	685	4347
✅ 💻 Local Development	3998	510	4508
✅ 📦 Local Production	3998	510	4508
✅ 🐘 Local Postgres	3998	510	4508
✅ 🪟 Windows	320	2	322
✅ 🌐 Cross-language Conformance	68	74	142
✅ vercel-http-transport	823	143	966
✅ vercel-multi-region	27	0	27
✅ vercel-ws-transport	557	87	644
Total	17451	2521	19972

Details by Category

✅ ▲ Vercel Production

App	Passed	Skipped
✅ astro-node	133	28
✅ astro-quickjs	133	28
✅ example-node	133	28
✅ example-quickjs	133	28
✅ express-node	133	28
✅ express-quickjs	133	28
✅ fastify-node	133	28
✅ fastify-quickjs	133	28
✅ hono-node	133	28
✅ hono-quickjs	133	28
✅ nest-node	133	28
✅ nest-quickjs	133	28
✅ nextjs-turbopack-node	158	3
✅ nextjs-turbopack-quickjs	158	3
✅ nextjs-webpack-node	158	3
✅ nextjs-webpack-quickjs	158	3
✅ nitro-node	133	28
✅ nitro-quickjs	133	28
✅ nuxt-node	133	28
✅ nuxt-quickjs	133	28
✅ python-node	66	95
✅ sveltekit-node	152	9
✅ sveltekit-quickjs	152	9
✅ tanstack-start-node	133	28
✅ tanstack-start-quickjs	133	28
✅ vite-node	133	28
✅ vite-quickjs	133	28

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable-node	134	27
✅ astro-stable-quickjs	134	27
✅ express-stable-node	134	27
✅ express-stable-quickjs	134	27
✅ fastify-stable-node	134	27
✅ fastify-stable-quickjs	134	27
✅ hono-stable-node	134	27
✅ hono-stable-quickjs	134	27
✅ nest-stable-node	134	27
✅ nest-stable-quickjs	134	27
✅ nextjs-turbopack-canary-node	160	1
✅ nextjs-turbopack-canary-quickjs	160	1
✅ nextjs-turbopack-stable-node	160	1
✅ nextjs-turbopack-stable-quickjs	160	1
✅ nextjs-webpack-canary-node	160	1
✅ nextjs-webpack-canary-quickjs	160	1
✅ nextjs-webpack-stable-node	160	1
✅ nextjs-webpack-stable-quickjs	160	1
✅ nitro-stable-node	134	27
✅ nitro-stable-quickjs	134	27
✅ nuxt-stable-node	134	27
✅ nuxt-stable-quickjs	134	27
✅ sveltekit-stable-node	153	8
✅ sveltekit-stable-quickjs	153	8
✅ tanstack-start-node	134	27
✅ tanstack-start-quickjs	134	27
✅ vite-stable-node	134	27
✅ vite-stable-quickjs	134	27

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable-node	134	27
✅ astro-stable-quickjs	134	27
✅ express-stable-node	134	27
✅ express-stable-quickjs	134	27
✅ fastify-stable-node	134	27
✅ fastify-stable-quickjs	134	27
✅ hono-stable-node	134	27
✅ hono-stable-quickjs	134	27
✅ nest-stable-node	134	27
✅ nest-stable-quickjs	134	27
✅ nextjs-turbopack-canary-node	160	1
✅ nextjs-turbopack-canary-quickjs	160	1
✅ nextjs-turbopack-stable-node	160	1
✅ nextjs-turbopack-stable-quickjs	160	1
✅ nextjs-webpack-canary-node	160	1
✅ nextjs-webpack-canary-quickjs	160	1
✅ nextjs-webpack-stable-node	160	1
✅ nextjs-webpack-stable-quickjs	160	1
✅ nitro-stable-node	134	27
✅ nitro-stable-quickjs	134	27
✅ nuxt-stable-node	134	27
✅ nuxt-stable-quickjs	134	27
✅ sveltekit-stable-node	153	8
✅ sveltekit-stable-quickjs	153	8
✅ tanstack-start-node	134	27
✅ tanstack-start-quickjs	134	27
✅ vite-stable-node	134	27
✅ vite-stable-quickjs	134	27

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable-node	134	27
✅ astro-stable-quickjs	134	27
✅ express-stable-node	134	27
✅ express-stable-quickjs	134	27
✅ fastify-stable-node	134	27
✅ fastify-stable-quickjs	134	27
✅ hono-stable-node	134	27
✅ hono-stable-quickjs	134	27
✅ nest-stable-node	134	27
✅ nest-stable-quickjs	134	27
✅ nextjs-turbopack-canary-node	160	1
✅ nextjs-turbopack-canary-quickjs	160	1
✅ nextjs-turbopack-stable-node	160	1
✅ nextjs-turbopack-stable-quickjs	160	1
✅ nextjs-webpack-canary-node	160	1
✅ nextjs-webpack-canary-quickjs	160	1
✅ nextjs-webpack-stable-node	160	1
✅ nextjs-webpack-stable-quickjs	160	1
✅ nitro-stable-node	134	27
✅ nitro-stable-quickjs	134	27
✅ nuxt-stable-node	134	27
✅ nuxt-stable-quickjs	134	27
✅ sveltekit-stable-node	153	8
✅ sveltekit-stable-quickjs	153	8
✅ tanstack-start-node	134	27
✅ tanstack-start-quickjs	134	27
✅ vite-stable-node	134	27
✅ vite-stable-quickjs	134	27

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	1
✅ nextjs-turbopack-quickjs	160	0	1

✅ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
✅ python	68	0	74

✅ vercel-http-transport

App	Passed	Skipped
✅ example	133	28
✅ express	133	28
✅ hono	133	28
✅ nextjs-turbopack	158	3
✅ nitro	133	28
✅ vite	133	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Skipped
✅ example	133	28
✅ express	133	28
✅ nextjs-turbopack	158	3
✅ vite	133	28

📋 View full workflow run

github-actions · 2026-09-11T08:37:04Z

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay
✅ `smoke-no-steps`	completed	3	0ms	ok
✅ `smoke-one-step`	completed	6	0ms	ok
✅ `hook-at-step-started`	completed	12	0ms	ok
✅ `hook-at-step-completed`	completed	12	0ms	ok
✅ `hook-at-hook-created`	completed	12	0ms	ok
✅ `deadline-hook-wins`	completed	7	1.0h	ok
✅ `deadline-expires`	completed	7	1.0h	ok
✅ `long-sleep`	completed	11	30.0d	ok
✅ `hook-never-arrives`	stalled	3	0ms	skipped
✅ `step-retries-twice`	completed	10	2.0s	ok
✅ `parallel-steps`	completed	9	0ms	ok
✅ `hook-on-execution-state`	completed	12	0ms	ok
✅ `peek-hook-before-branch`	completed	12	0ms	ok
✅ `peek-hook-after-branch`	completed	12	0ms	ok
✅ `peek-hook-at-registration`	completed	12	0ms	ok
✅ `race-hook-before-probe`	completed	12	0ms	ok
✅ `race-hook-after-probe`	completed	12	0ms	ok
✅ `race-duplicate-delivery`	completed	13	0ms	ok
✅ `attr-hook-before-step`	completed	11	0ms	ok
✅ `attr-hook-after-step`	completed	11	0ms	ok
✅ `attr-from-step-body`	completed	13	0ms	ok
✅ `fork-hook-after-timeout`	completed	14	1.0m	ok
✅ `fork-hook-before-timeout`	completed	14	1.0m	ok
✅ `count-hook-after-timeout`	completed	17	1.0m	ok
✅ `count-hook-before-timeout`	completed	20	1.0m	ok
✅ `stale-read-step-count-fork`	completed	20	1.0m	ok
✅ `stale-read-equal-step-counts`	completed	14	1.0m	ok
✅ `step-vs-step-fork`	completed	12	0ms	ok
✅ `step-vs-step-fork-fenced`	completed	12	0ms	ok
✅ `fence-catches-benign-direction`	completed	12	5ms	ok
✅ `in-flight-before-decision`	completed	17	1.0m	ok
❌ `in-flight-before-decision-counted`	completed	17	1.0m	ok
✅ `in-flight-after-decision`	completed	19	2.0m	ok
✅ `stale-read-step-count-fork-fenced`	completed	20	1.0m	ok
✅ `fork-hook-wins`	completed	13	1.0m	ok
✅ `fork-timeout-wins`	completed	13	1.0m	ok
✅ `unclaimed-payload-under-fork`	completed	17	1.0m	ok
✅ `claimed-payload-under-fork`	completed	17	1.0m	ok
✅ `writers-independent-step-bodies`	completed	12	0ms	ok
✅ `writers-scripted-tempo`	completed	12	0ms	ok
✅ `cancel-mid-step`	cancelled	7	0ms	skipped

Full trace: world-sim.txt

github-actions · 2026-09-11T08:38:01Z

Framework	Flow route	Step reg.	Framework output
hono	250.7 KiB (±0)	93.0 KiB (±0)	1.89 MiB (+1.4 KiB)
nextjs-turbopack	257.2 KiB (±0)	426 B (±0)	897.9 KiB (+239 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

e408ace · run

Copilot

🟡 Changes recommended

Critical runtime recovery issues and a moderate QuickJS coverage gap remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR reduces fan-out latency by publishing eligible step messages before batched creates commit and enables resilient dispatch by default.

Changes:

Adds publish-first fan-out dispatch with deferred failure handling.
Adds lazy consumer-side step recovery.
Updates defaults, tests, documentation, and release metadata.

File summaries

File	Summary
`packages/core/src/runtime/suspension-handler.ts`	Implements publish-first fan-out dispatch.
`packages/core/src/runtime/suspension-handler.test.ts`	Covers batching, ordering, payload limits, defaults, and failures.
`packages/core/src/runtime/constants.ts`	Enables resilient dispatch by default; moderate QuickJS coverage concern remains.
`packages/core/src/runtime.ts`	Adds lazy recovery; critical requeue payload and duplicate-consumer recovery issues remain.
`packages/core/src/runtime.test.ts`	Tests consumer recovery, conflicts, redelivery, and legacy messages.
`docs/content/docs/v5/configuration/runtime-tuning.mdx`	Documents runtime dispatch behavior; preview link is missing.
`docs/content/docs/v5/changelog/batched-event-writes.mdx`	Documents publish-first batching; preview link is missing.
`.changeset/publish-first-fanout.md`	Adds release metadata.

Review details

Suppressed comments (3)

docs/content/docs/v5/changelog/batched-event-writes.mdx:65

The PR description still contains DOCS_PREVIEW_PLACEHOLDER, but these docs changes require a Docs Preview section with direct workflow-docs preview links for each changed page. Replace the placeholder with the actual preview links (including the runtime-tuning anchor and this batched-event-writes page) before merging.

**On by default.** The suspension handler folds a **clean fan-out** (the suspension's eager `step_created` and `wait_created` writes) into `createBatch` calls of at most 32 events (mirroring the server's transaction budgets). Chunks of a larger fan-out commit **concurrently**: slot assignment is the World's, so parallel chunks race for slot ranges exactly like the pre-fold path's parallel single writes did, and per-entity conditions, not commit order, carry correctness. The fold only engages when the World implements `createBatch`, the run is on slot identity, and the suspension carries no attribute writes and no hook writes; everything else keeps the single-event path byte-for-byte. [Resilient step dispatch](/docs/configuration/runtime-tuning#workflow_resilient_step_dispatch) composes with the fold: a folded step whose serialized input fits on the queue message is published, carrying that input, before its create commits.

docs/content/docs/v5/configuration/runtime-tuning.mdx:95

The PR description still contains DOCS_PREVIEW_PLACEHOLDER; replace it with a direct workflow-docs preview link for this changed page, including the #workflow_resilient_step_dispatch anchor, before merging.

### `WORKFLOW_RESILIENT_STEP_DISPATCH`

- Default: enabled
- When a suspension hands newly created steps to the queue, the runtime publishes each step's execution message without waiting for its `step_created` event write to commit, instead of sequencing the two. The message carries the serialized step input (`stepInput`), so a delivery that arrives before the write lands, or after a transient write failure (429 / 5xx / transport), still executes the step: the consumer's bare `step_started` finds no step, and it sends the start once more as a lazy `step_started` carrying that input, which every World turns into an atomic create-and-start (one round trip, the same write the inline path uses). If that lazy start loses to the producer's create landing in between (a `409`), one more bare start runs the step. A lazy `step_started` on a step that already exists is always a `409`, so the lazy start is only ever the recovery for a step-missing bare start, never the first attempt, and retries and redeliveries of a materialized step keep the bare start. This mirrors resilient start (`runInput`) and the legacy lazy hook resume's `hookInput` (which current producers no longer send; see [durable hook resume](/docs/changelog/lazy-hook-resume)).

packages/core/src/runtime/constants.ts:273

This default flip also activates the existing QuickJS create/publish race, but the focused tests cover the node:vm suspension path and the consumer, not dispatchPendingOps with a stepInput-carrying message. A regression in QuickJS could therefore omit the payload or restore create-before-publish while the new tests remain green. Add a QuickJS dispatch test covering the parallel publish and missing-step recovery path.

export function isResilientStepDispatchEnabled(): boolean {
  const raw = process.env.WORKFLOW_RESILIENT_STEP_DISPATCH;
  if (raw === undefined || raw === '') return true;
  return !(raw === '0' || raw.toLowerCase() === 'false');

Files reviewed: 8/8 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

pranaygp · 2026-09-11T20:46:51Z

+                      // message carries `stepInput`, the executor is re-run
+                      // ONCE with that input as a lazy `step_started`, which
+                      // every World turns into an atomic create + start


Fixed in 728daa7. The queued-step re-route now forwards the incoming stepInput on the re-enqueued message (alongside stepId/stepName/timing, still without runInput), so a publish-first delivery misrouted before its create commits can materialize the step on the pinned deployment. There is no runContext field on the payload, so stepInput is the only payload-bearing field that needed carrying. Covered by the new test "re-routes a publish-first queued step execution with its stepInput intact" in runtime.test.ts.

pranaygp · 2026-09-11T20:46:53Z

+                              // the producer's step_created landed between the
+                              // failed bare start and this write. The step now
+                              // exists, so the bare start it wanted all along
+                              // runs it.
                              return await executeQueuedStep();


Good catch, fixed in 728daa7. After a skipped lazy start the consumer now reads the step entity (world.steps.get, resolveData: 'none', only on this rare 409 path) and arbitrates by status: pending means the producer's step_created won (created, never started) and the bare start runs it; running or a terminal status means a peer delivery on another instance won the lazy create-claim and is executing the body, so this delivery acks as the loser and returns skipped without executing, the same way an in-process single-flight loser does. Tests cover both branches (pending → bare start, body once; running/completed/failed → two starts written, body never run, acked) and that the read is skipped when the lazy start wins or the bare start succeeds.

github-actions · 2026-09-11T08:44:42Z

Event Log Race Repro

vercel clean, 26 runs
local clean, 26 runs
postgres clean, 26 runs

Run History

Run	Lane	Total	Complete	Stuck
09-11 21:08	vercel	26	26	0
	local	26	26	0
	postgres	26	26	0
09-11 22:38	vercel	26	25	1
	local	26	26	0
	postgres	26	26	0
09-11 23:08 #2	vercel	24	21	3
	local	26	26	0
	postgres	26	26	0
09-11 23:52	vercel	26	26	0
	local	26	26	0
	postgres	26	26	0
09-12 01:19	vercel	26	26	0
	local	26	26	0
	postgres	26	26	0

Config

vercel: 26 runs / step-storm 6, hook-storm 6, blocked-branch 6, wake-loop 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / heartbeat 4000ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms

local, postgres: 26 runs / step-storm 6, hook-storm 6, blocked-branch 6, wake-loop 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / heartbeat 4000ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms

…; resilient step dispatch on by default Compose resilient step dispatch with the batched fan-out fold instead of making them mutually exclusive. Inside the fold, each queued step's message is published, carrying `stepInput`, as soon as its input is dehydrated and `run_started` has settled, concurrently with the `createBatch` commits; steps whose input cannot ride the message keep publish-after-chunk-commit. Early publishes ride the fold's trailing work (joined before ack) and are reported in `queuedStepCorrelationIds`. The queued-step consumer recovers a step-missing bare `step_started` with one lazy `step_started` carrying the message's input (atomic create + start on every World) instead of a `step_created` write plus a second bare start; a lazy start that loses to the producer's create (409) falls back to one more bare start. The eager redelivery pre-ensure is removed, as is the last `viaStepDispatch` sender. `isResilientStepDispatchEnabled()` now defaults on; `WORKFLOW_RESILIENT_STEP_DISPATCH=0`/`false` disables it. Since #3519 no World in this repository returns 412 for a stale write, and the Vercel backend skips its precondition check for slot-identity runs, so the window that justified "off" no longer exists there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ost lazy claim by step status; soften the publish-first ordering claim Review follow-ups on #4102: - The deployment-affinity requeue of a queued step execution now forwards the incoming `stepInput`, so a publish-first delivery misrouted before its step_created commits can still materialize the step on the pinned deployment. - After a lazy recovery start is skipped (409), read the step entity and bare-start only a `pending` step (the producer's create won). A `running` or terminal step means a peer delivery on another instance won the lazy create-claim and is executing the body; this delivery acks as the loser without executing, since runStepSingleFlight only serializes one process. - Reword the happens-before claim in the fold comment, the docs, and the changelog: publishes are initiated early and joined alongside the creates, both must succeed before ack, and a durable step_created does not by itself prove its message was published. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

pranaygp · 2026-09-11T23:28:38Z

Hold for review: possible regression on 49db6b0. The event-log-race-repro harness was 78/78 clean on the Vercel lane for the two earlier heads, but on this head it reports 1 stuck (run 34653938857 attempt 1) and then 3 stuck of 24 (attempt 2: step-storm ×1, blocked-branch ×2), and the Benchmark (vercel, nextjs-turbopack) lane timed out on 2 of 10 fan-out iterations waiting for benchFanOutStepsWorkflow to finish (e.g. wrun_41M29BCAQM1GZYYDKJA45CT665). The local and postgres lanes stay clean. Investigating the stuck runs' event logs now and running a control harness on main (https://github.com/vercel/workflow/actions/runs/34658163095). Do not merge until this is explained.

…y instead of acknowledging it Publish-first delivers a fan-out step's message while the producer's step_created is still committing. The Vercel World's conditional bare start then misses the entity, and the re-read that phrases its rejection already sees the committed `pending` row, so the consumer receives a 409 saying "Cannot start workflow step … with status 'pending'. Operation requires status 'pending' or 'running'.". The executor mapped every 409 to `skipped` and the delivery was acknowledged, stranding the step: every later re-publish of it is deduplicated against the acknowledged message. This is the `stuck` blocked-branch run in #4102's race harness (4 of 50 Vercel-lane runs across two harness runs) and the pending-forever timedNoopStep steps that failed the Benchmark (vercel, nextjs-turbopack) lane on the same head. A `skipped` start is no longer taken as "already started" by itself. The consumer reads the step and lets the entity decide: `running` or terminal acknowledges the delivery as the loser (the only reading the executor assumed), a missing step is materialized from the message's input (the existing step-missing recovery), and a `pending` step gets its start retried once. A step still `pending` after that fails the delivery so the queue redelivers it rather than acknowledging a step nobody started. The lost-lazy-claim arbitration shares the same path, so its follow-up bare start is covered too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

pranaygp · 2026-09-11T23:41:52Z

Race harness `stuck` runs: root cause and fix (`17f81a3`)

The 4 stuck blocked-branch runs on the Vercel lane (1 in the first harness run on 49db6b0, 3 in the rerun) and the pending-forever timedNoopStep steps that failed Benchmark (vercel, nextjs-turbopack) on the same head share one signature, and it is a real regression introduced by this PR's publish-first default, not by the #3457 reconciliation (the same log line already occurred on the pre-#3457 head 728daa7, preview 4ftxiim82, at 20:49 UTC; it never occurred on any main deployment).

Per run (wrun_41M299H9ZS0GPXG7XP0GTW063X / step_01M299H9ZS5XQ22M7J8K129TEM finalizeStep, wrun_41M29AXPWK0GXJCEFJCH51XE0Z / …N7R5 recoverStep, wrun_41M29B8J4F0GG30W9S6KENJB4S / …WRR0 settleStep, wrun_41M29B8J4G0GNXB2T8FDSXK5JF / …G3BV finalizeStep): exactly one queued step has step_created and no step_started; siblings from the same batch started 100 ms to 1.4 s later. The consumer invocation for that step (79 ms after the create's occurredAt in the first run) logged one write, POST createEvent -> 409, then

Step in terminal state, skipping { error: "Cannot start workflow step step_… with status 'pending'. Operation requires status 'pending' or 'running'." }

then Background step done but other steps pending, returning and HTTP 200. Not the lazy-start path: the first bare step_started got that 409, the SDK maps every 409 to skipped, and skipped acknowledges the delivery. Every later re-publish of the step is deduplicated against that acknowledged message (24 h idempotency window), so the step stays pending until the harness cancels at 240 s.

Mechanism. Publish-first delivers the message while the producer's step_created batch is still committing. The backend's conditional bare-start patch misses the entity, and the re-read that phrases the rejection already sees the committed pending row, so it answers with a self-contradictory 409 (a pending step that "requires pending").

Fix (this branch, 17f81a3). A skipped start is no longer trusted by itself. The consumer reads the step and lets the entity decide: running / terminal acknowledges as the loser (the only reading the executor assumed), a missing step is materialized from the message's input (the existing recovery), and a pending step gets its start retried once. Still pending after that fails the delivery so the queue redelivers it instead of acknowledging a step nobody started. The lost-lazy-claim arbitration shares the path, so its follow-up bare start is covered too. Unit tests in runtime.test.ts cover all four arbitration outcomes plus the redelivery case; full core suite green.

A server-side hardening (retry the start patch in throwStepNotStartable when the re-read satisfies the start guard, instead of throwing a 409 the client cannot distinguish) will go in a separate workflow-server PR. Re-running the race harness and the benchmark lane on this head is the verification.

pranaygp · 2026-09-11T23:46:41Z

Addendum to the root-cause comment above, for the hold thread:

Stuck-run evidence: harness run 34653938857 (attempt 1: 1 stuck; attempt 2: 3 stuck). Example run wrun_41M299H9ZS0GPXG7XP0GTW063X, step step_01M299H9ZS5XQ22M7J8K129TEM (finalizeStep): step_created, never step_started. Its consumer logged one write, POST createEvent -> 409, then Step in terminal state, skipping { error: "Cannot start workflow step step_01M299H9ZS5XQ22M7J8K129TEM with status 'pending'. Operation requires status 'pending' or 'running'." } and returned 200. The same line appears 14 times across this PR's preview deployments (incl. the pre-Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue #3457 head 728daa7) and zero times on any main deployment since 2026-09-09, so it is the publish-first default, not the Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue #3457 reconciliation.
Fix: 17f81a3. Invariant now in a code comment above the arbitration helper: a queued delivery must never acknowledge skipped while the step reads pending. One helper arbitrates both the bare start and the post-lazy-claim start. Tests: pending → retry once → completes; running/completed/failed → ack without executing; missing → materialize from the message; pending after the retry → delivery rejects (both entry points); no entity read when the first bare start succeeds or the lazy start wins.
Reachable on main today under WORKFLOW_RESILIENT_STEP_DISPATCH=1: with resilient dispatch on, the fold publishes before the create commits on main too, and main's consumer maps every 409 on the bare start to skipped the same way. Only the default (off) keeps it latent there.
The harness re-triggered on the push (34659002004, in progress).
Server-side hardening (retry the start patch in throwStepNotStartable when the re-read satisfies the start guard) is going up as a separate workflow-server PR; link to follow.

pranaygp · 2026-09-12T00:05:13Z

Server half of the fix is up: vercel/workflow-server#959 (retry the start patch when the read-back satisfies the start guard, bounded; honest 409 when spent; requiresState can no longer be thrown with the current state among the required ones; and the retried start is renumbered above the step_created that beat it, since the spec-7 sequencer otherwise handed the retry its old block and inverted the two events).

pranaygp · 2026-09-12T00:09:20Z

Hold lifted. On 17f81a3 the event-log-race-repro harness is 26/26 clean on the Vercel lane (run 34659002004; local and postgres 26/26), Performance Benchmarks passed (run 34659001999), and tests.yml is green including E2E Required Check. Deployment logs show the create-vs-start race still firing (~34 times during the harness) with every retry winning and zero exhausted redeliveries, so the arbitration is load-bearing. Server-side hardening for the contradictory 409 is in vercel/workflow-server#959. Ready for review.

…s step_created Under publish-first fan-out a queued step's bare step_started can take a counter-sequenced position below the step_created it belongs to (workflow-server specs/StepStartRaceFirstAttempt.cfg). Cover that the consumer, the ordered walk, the suspension-handler creation filter, and the ownership scans all match by correlation id rather than pair order: the step resolves, nothing is re-created or re-dispatched, and the step-name fence still applies to the early start. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

pranaygp · 2026-09-12T01:08:43Z

Ordering quirk this PR opens (documented, tolerated, tracked server-side).

With publish-first, a queued step's message can be consumed before the batch holding its step_created commits. On spec-7 (counter-sequenced) logs the consumer's bare step_started reserves its position before the create's block, so if the create lands in between, the accepted start commits below the create: step_started(X), step_created(X), step_completed(X), dense, same correlation id. Spec 6's slot fence makes this uncommittable; spec 7 has no equivalent for the first attempt (the retry path in workflow-server#959 does re-sequence).

The SDK tolerates it because the consumer matches by correlation id, not pair order: the step_started branch in step.ts records the stamp/timestamp without requiring hasCreatedEvent, the later step_created sets it, the suspension handler's stepsNeedingCreation filter then skips the step, and the raw-event scans (pendingStepIds, hasPendingStepOwnedByMessage) are set-based / latest-wins. QuickJS marks the pending op created on either event. e408ace pins this in packages/core/src/step-started-before-created.test.ts (result resolves, nothing re-created or re-dispatched, stamped start still derives ownership, step-name fence still applies to the early start).

Server-side fix tracked in vercel/workflow-server#960; model of the gap in workflow-server#959's specs/StepStartRaceFirstAttempt.cfg.

Copilot AI lite review requested due to automatic review settings September 11, 2026 08:33

pranaygp requested a review from a team as a code owner September 11, 2026 08:33

Copilot started reviewing on behalf of pranaygp September 11, 2026 08:33 View session

vercel Bot deployed to Preview – workbench-python-workflow September 11, 2026 08:33 View deployment

pranaygp added the event-log-race-repro Run the event log race reproduction job label Sep 11, 2026

vercel Bot deployed to Preview – workflow-web September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-express-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-astro-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workflow-tarballs September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – example-workflow September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack September 11, 2026 08:34 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack September 11, 2026 08:35 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow September 11, 2026 08:35 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow September 11, 2026 08:35 View deployment

vercel Bot deployed to Preview – workflow-docs September 11, 2026 08:36 View deployment

vercel Bot deployed to Preview – workflow-swc-playground September 11, 2026 08:36 View deployment

Copilot AI reviewed Sep 11, 2026

View reviewed changes

vercel Bot deployed to Preview – workbench-astro-workflow September 11, 2026 20:58 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow September 11, 2026 20:58 View deployment

vercel Bot deployed to Preview – workbench-express-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – example-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workflow-tarballs September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack September 11, 2026 20:59 View deployment

vercel Bot deployed to Preview – workflow-docs September 11, 2026 21:00 View deployment

vercel Bot deployed to Preview – workflow-swc-playground September 11, 2026 21:01 View deployment

pranaygp and others added 2 commits September 11, 2026 15:23

pranaygp force-pushed the pgp/publish-first-fanout branch from db26166 to 49db6b0 Compare September 11, 2026 22:25

vercel Bot deployed to Preview – workbench-python-workflow September 11, 2026 22:25 View deployment

vercel Bot deployed to Preview – workflow-web September 11, 2026 22:26 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow September 11, 2026 22:26 View deployment

vercel Bot deployed to Preview – workbench-express-workflow September 11, 2026 22:27 View deployment

Uh oh!

Conversation

pranaygp commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Design

Safety analysis

Out of scope

Tests

Docs Preview

Uh oh!

changeset-bot Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

⚠️ Flaky E2E Tests (passed on retry)

🛠 Infra Events (absorbed by the harness)

E2E Test Summary

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sim World

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (3)

Uh oh!

pranaygp Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

pranaygp Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Event Log Race Repro

Run History

Uh oh!

pranaygp commented Sep 11, 2026

Uh oh!

pranaygp commented Sep 11, 2026

Race harness stuck runs: root cause and fix (17f81a3)

Uh oh!

pranaygp commented Sep 11, 2026

Uh oh!

pranaygp commented Sep 12, 2026

Uh oh!

pranaygp commented Sep 12, 2026

Uh oh!

pranaygp commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pranaygp commented Sep 11, 2026 •

edited

Loading

changeset-bot Bot commented Sep 11, 2026 •

edited

Loading

vercel Bot commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Sep 11, 2026 •

edited

Loading

Race harness `stuck` runs: root cause and fix (`17f81a3`)