Repository navigation
[Bug] Nondeterminism replaying local activity markers recorded after a FORCE_CLOSE_COMMAND workflow task failure (1.34.0) #1976
Description
What are you really trying to do?
A workflow fans out ~900 regular activities with asyncio.gather. A workflow outbound interceptor records progress with a local activity: at most one is in flight, and when it resolves a done callback starts the next one with the changes accumulated in the meantime. (We have replaced it with a regular activity as a workaround.)
Describe the bug
While results streamed in, workflow tasks took 2.5-6 s and the server failed them 12 times with WORKFLOW_TASK_FAILED_CAUSE_FORCE_CLOSE_COMMAND (51-139 activity completions arrived during each task). After one of those, the next workflow task (started 08:52:05.021Z, completed 08:52:08.319Z) recorded, in this order:
- local activity marker seq 1296 (
activation_index1,original_schedule_time08:52:04.597Z,complete_time08:52:00.700Z) - 422 x ScheduleActivityTask (seq 1297-1718)
- local activity marker seq 1719 (
activation_index1,original_schedule_time08:52:04.637Z,complete_time08:52:02.229Z) - 49 x ScheduleActivityTask
- local activity markers seq 1769 (
activation_index2) and 1770 (activation_index3)
The complete_time of 1296 and 1719 is before that task started, so both ran during the force-closed attempt (08:52:00.674Z-08:52:04.941Z) and their results were handed to the next attempt.
Any later full replay of this history fails with:
[TMPRL1100] Nondeterminism error: Missing associated machine for LocalActivity(1719)
Replaying the downloaded history with Replayer, the same workflow code and temporalio 1.34.0 fails the same way, so the SDK cannot replay a history it wrote. On the live worker the workflow task kept failing until we terminated the workflow (114 attempts over 16 hours).
Our reading (may be wrong): in the original activation 1, the 1296 result was delivered after the 422 activity resolutions, so the next local activity got seq 1719. activation_index keeps 1296 in activation 1 but not its position among the other resolutions in that activation, so on replay the next local activity gets a different seq.
Minimal Reproduction
We don't have a minimal reproduction yet. The history contains customer data, so we can't attach it, but we can share sanitized event and marker metadata or run a patched build against it.
Shape: asyncio.gather over many activities, plus an outbound interceptor that chains one local activity at a time from done callbacks, plus enough concurrent completions to trigger FORCE_CLOSE_COMMAND (default history.maximumBufferedEventsBatch = 100).
Environment/Versions
- temporalio (Python) 1.34.0, which includes "Record order LAs are resolved in WFTs" (Record which activations LAs are resolved in sdk-rust#1616); Python 3.12; Linux container
- Temporal server 1.31.1, default dynamic config
- Related: [Bug] Local activity resolutions regrouped on replay since 1.32.0, delivering the wrong payload #1881 (closed), [Bug] non determinism when using
asyncio.gatherwith local activities #1578 (open)
Additional context
Workaround: send the recording call as a regular activity instead of a local activity.
Activity
mjameswh commentedon Oct 16, 2026
Thanks for reporting. A history file (even sanitized) and/or repro code would definitely help investigation. If you are a Temporal Cloud customer, I'd also recommend opening a support ticket on our portal, and provide information through that channel.
Also, can please answer these questions?
tconley1428 commentedon Oct 22, 2026
Feel free to reopen if more information becomes available.
lucafranco1994pot-cloud commentedon Nov 5, 2026
repro_1976_frozen.zip
Hi, I was able to reproduce this failure locally with a fully synthetic workload (no customer data). The attached zip contains the reproducer scripts, the histories referenced below and SHA-256 checksums.
Setup: Temporal CLI 1.10.0 (dev server, Server 1.32.2, default history.maximumBufferedEventsBatch=100), Python 3.13. Each run used exactly one SDK version from start to finish.
Workload (repro_peritem.py): 900 parallel chains of 2 regular activities each (1800 total). A workflow interceptor schedules one local activity per completed regular activity, strictly serially: the done callback of the running local activity starts the next one.
Live runs:
Cross-version replay of the same frozen histories:
For these two failing histories, upgrading the replayer from 1.34.0 to 1.35.0 does not resolve the replay failure. In contrast, the tested passing history generated under 1.35.0 replays successfully with either version.
This suggests that the observed difference involves how the histories are produced during execution, rather than a replay-only fix in 1.35.0. However, these tests do not distinguish between a history-generation issue, a replay limitation shared by both versions, or an interaction between the two. The root cause remains unverified.
Limits: small number of runs; I have not identified the change between the two versions; the controls "no interceptor" and "regular instead of local logging activities" never reached force close, so they are not valid controls yet. This is a synthetic reproduction of the reported pattern, not a reproduction of the original history.
Happy to run additional variants if that helps.
lucafranco1994pot-cloud commentedon Nov 5, 2026
Follow-up with captured worker logs. The attached repro_1976_frozen_v2.zip contains the synthetic reproducer, worker logs, histories and checksums.
I re-ran the same synthetic workload with Core log output captured. Each run is a separate worker process running exactly one workflow, and the error lines carry the same run_id as the exported history.
The log line matches the task-replacement scenario documented in temporalio/sdk-rust#1640 ("Buffer a new workflow task while the run already has one"). That PR buffers an incoming workflow task while Core still holds the current task, including while waiting for a local activity. The Python SDK 1.35.0 release notes list this fix (Core commit 212e8150).
This makes #1640 my leading hypothesis for the synthetic reproduction. It is not a confirmed diagnosis of the original production case in this issue: #1640 documents a different downstream nondeterminism error, and my comparison changes the full SDK version rather than isolating that fix (1.35.0 also contains other Core changes, e.g. #1620). I have not tested a 1.35.0-based build with only the #1640 buffering change reverted.
Separately, the two failing 1.34.0 histories from my previous comment also failed replay under 1.35.0. I keep that result distinct from the successful replay of histories newly generated under 1.35.0.
Could you help confirm whether this reproducer exercises the #1640 path and whether that task replacement explains the LocalActivity marker replay failure?
repro_1976_frozen_v2.zip
kkdy21 commentedon Nov 5, 2026
Sorry for the late reply, and thanks @lucafranco1994pot-cloud for the reproducer.
Answers to @mjameswh's questions first:
temporalio>=1.29,<2), so that code only ever ran on 1.32.0 through 1.34.0, and we did not notice this error before 2026-10-15. So we can't say it worked on an older SDK; we never ran it on one.I ran Luca's
repro_peritem.py(case A, n=900) against a local dev server (CLI 1.8.1, Server 1.31.2), two runs per SDK version. For each history I counted local activity markers whosecomplete_timeis earlier than the start of the workflow task that recorded them. Every run had FORCE_CLOSE_COMMAND failures (6 to 14 per run).These markers look the same as the ones in our production history. Markers 1296 and 1719 had a
complete_timebefore the recording WFT started, and anoriginal_schedule_timeafter theircomplete_time. In other words, local activities that ran during one workflow task were recorded by the next one.That fits temporalio/sdk-rust#1640. temporalio/sdk-rust#1442 (first shipped in 1.32.0) made a run that waits on a local activity look idle to Core, so a new workflow task arriving at that point replaced the outstanding one. #1640 (shipped in 1.35.0) buffers it instead. In our case the replacing task is the server's retry of the force-closed workflow task, not a query.
My reading:
If it helps, I can open a PR with a regression test for this path (a force-closed workflow task while a local activity is running), in whichever repo you prefer, sdk-python or sdk-rust.