Repository navigation
Stop activities when a heartbeat says the attempt can't continue #186
Description
Problem
When a Python activity's attempt can no longer finish, the activity keeps running. This happens when its start-to-close deadline passes, its attempt is closed, or its run closes. Its slot stays taken until it returns, and then its result is rejected.
- In each of these cases, the heartbeat response says
can_continue: falsewithcancel_requested: false, with reasons such asactivity_deadline_elapsed,attempt_closedandrun_closed. - Python's default heartbeat path raises only on
cancel_requested. Remote activities use that path unless cooperative cancellation is enabled. - The PHP SDK stops on
can_continue: false, and so does Python's own cooperative path.
So a Python activity that heartbeats faithfully gets no signal that its work is now wasted. The timeouts guide says a start-to-close timeout closes the attempt, but it doesn't say the worker's code keeps running.
Source-verified, not reproduced.
Acceptance criteria
- On the default path,
heartbeat()andheartbeat_sync()raise when the response hascan_continue: false, matching PHP. The exception carries the reason, so an activity can tell a timeout or closed run apart from cancellation. - The worker frees the slot once the activity exits and submits no result for the closed attempt.
- Tests cover an elapsed start-to-close deadline, a closed attempt and a closed run, and confirm that
cancel_requestedstill raises. - The Python guide and the timeouts page state that the activity stops at its next heartbeat after its attempt closes, and that code which never heartbeats runs until it returns.
Activity
rmcdaniel commentedon Oct 21, 2026
Accepted product correction: stop a refused heartbeat
This is an activity-authority bug, not a documentation-only gap. The default context currently checks cancellation but ignores an explicit
can_continue=false. The correction must preserve the original refusal reason, stop async and synchronous activity code at its next heartbeat, release its physical slot after exit, and send no completion or failure for the closed attempt. Cancellation remains distinct.Tests-first
68f6ed1adds thirteen controls for elapsed start-to-close, closed attempt and closed run on both heartbeat APIs and default worker paths, plus publication fencing when user code catches the heartbeat exception. Production is byte-identical tod2939d4efor the counterfactual: all thirteen actually FAIL in 0.77 seconds. That parent contains the independently qualified #185 maintenance-stop fix.Functional correction
a8659b70888e7780e4d2aaa04ca7b1506d1f52b8/ tree0724221f81a5a2639fae0a09e1b611f5c9141143preserves the original refusal inActivityLeaseAuthorityLost.reason, separately from cancellation. The default worker returnsclaim_abortedand emits neither result nor failure, including after user code suppresses the refusal and returns or raises an ordinary/non-retryable error. The same original Client authority is invalidated by task/attempt/generation/owner binding; a different original is not invalidated. Two real Client/mock-HTTP controls prove transport publication is fenced before an outgoing completion request.Final normal-install Python 3.10.22 acceptance: 3,120 tests PASS / three skips / 105 integration deselections, 164.06 seconds; 46 focused heartbeat/context/lease-worker tests PASS, overlapping the full total; Ruff passes and mypy passes all 36 source files. The earlier source component also passed 3,118 tests; a test-only lint failure was corrected explicitly. The final result includes the additional original-client authority/isolation controls. No failed original is called a pass.
Current reviewed head is
0c215c9a8ec56946930a511af1b518bffc9710c4/ tree87eaf99724600168ad3190bf7ec8f45fa606fc95. It addresses the two actionable Sonar locations by separating ordinary exception disposition from callback dispatch and creating the selected coroutine outside the exception guard. Original storage-admission propagation, failure metadata/logging and heartbeat-authority order are preserved. Actual focused worker/storage/context/heartbeat qualification on this head passes 387 tests / two existing skips, plus Ruff and mypy on 36 source files. The 3,120-test full result belongs to functional parenta8659b70, not a whole-head claim for this cleanup. Fresh hosted supported-version/integration/Sonar gates qualify the current head independently; no threshold/profile change or analyzer-fixed claim before readback.Before advancement, the original 3.12, corpus, package, portal/boundary and advisory cells had passed; 3.10/3.11/lint remained queued with no admitted assertions. Their original partial results remain visible. No running supported-version test was cancelled to obtain a pass.
README, the SDK reference, changelog and method docs explain the refusal and its physical-stop limits. No dependency, package version, Server/native/shared/schema or wire change. The branch incorporates merged #185 via
fe40c647, not a borrowed old runtime result. Fresh normal CI/Sonar, compatible publication, actual installed refusal/slot acceptance and the eight-locale website timeout explanation remain required before closing #186. Workflow #878's larger automatic-maintenance acceptance stays open independently. No provider operation or paid resource.rmcdaniel commentedon Oct 21, 2026
Source correction delivered through Python #187, merged at
d6da755e80357bf2bebf460c925ff71f3433e683on October 21 at 10:28:34 UTC. Its whole tree exactly matches qualified candidate87eaf99724600168ad3190bf7ec8f45fa606fc95.All supported Python 3.10/3.11/3.12, actual source-Server integration, lint/package/corpus/portal/boundary and target qualification gates pass. Exact-head Sonar passes with both actionable reports individually CLOSED/FIXED and zero OPEN/CONFIRMED. No threshold change or suppression. The completed remote head is deleted and independently returns 404.
The default heartbeat now preserves the original authority-loss reason and fences both results and failures, including when user code catches the refusal. Cancellation remains distinct. The thirteen actual parent-source failures remain recorded; the focused and full local results retain their exact source attribution in the PR.
Fresh main CI
39584856529, portal39584856514, Sonar39584856467and boundary39584856460remain independent and pending. Compatible publication, actual installed refusal/physical-slot acceptance and the eight-locale website follow-through remain required before closing #186. Workflow #878 remains open for the larger automatic-maintenance and published six-minute/worker-loss acceptance.rmcdaniel commentedon Oct 21, 2026
Fresh-main SDK source
d6da755e80357bf2bebf460c925ff71f3433e683remains unqualified for publication. Its normal integration job119190338104in run39584856529exceeded the unchanged 25-minute job limit (11:39:27–12:04:43 UTC), during Start Server stack, before endpoint provenance and test execution. The job annotation confirms the timeout and missing scenario artifacts. Supported Python unit cells, lint, package and corpus pass independently.No failing integration assertion or runtime cause is established. The job-log download also timed out; that network retrieval failure is not a product diagnosis. The failed/cancelled original is retained. I will inspect the available startup evidence and qualify the actual SDK/Server tuple before publication; no timeout increase, hidden pass or automatic repeated retry is applied.