Repository navigation
Conversation
ready_ids listed the active class twice when it also sat in ready_order. Child failure/cancellation policies had no writer, so they are removed and their persisted keys are dropped on load and on the queue-read projection: rows written before this change stay readable while their children wait. ContinuationStatus now uses one "canceled" spelling and the never-called runtime.replace_stream is deleted.
Pfannkuchensack
left a comment
There was a problem hiding this comment.
Reviewed at d24e13c (parent 5b776e3). No blockers. The load-compatibility fix covers both readers, the ready_ids fix is correct for every arrangement, and the removals are safe:
- No writer ever set a non-default policy (both
create_dependency/ChildExecutionCapabilitycall sites ingraph.pypass none), and nothing consumedparent_action/cancel_child_ids. Sibling cancellation stays with the queue lifecycle (cancel_workflow_call_children), so behavior under the old defaults is unchanged. ContinuationRecordandChildExecutionCapabilityare never persisted, so thecancelled→canceledrename needs no compatibility handling.- Not bumping
CURRENT_EXECUTION_STATE_VERSIONis right: old readers accept rows written here (the removed fields had defaults), and a bump would quarantine them on rollback. - The explicit
ValueErrorfor a non-mapping ledger is needed, because"garbage".items()would raiseAttributeError, which neitherexcept (TypeError, ValueError)catches.
A few low-severity follow-ups:
-
Dead aggregation branch (
child.py:618-643). Failed and canceled completions now return at 612-617, so whenlen(self.completions) == len(self.children)is reached, every completion iscompleted. Theany(... "failed")/any(... "canceled")scans can't run any more, and the deleted continue-policy test was their only coverage. Suggest collapsing the tail toself.transition_terminal("completed", None)with onereturn self._result(changed=True); lines 642 and 643 are currently the same return. -
Cancellation with pending siblings is untested. Both child tests use
fail_childonly. The single-child cancel test intest_execution_state_migration.pycan't tell "terminal immediately" from "terminal when all done". Parametrizingtest_failed_child_makes_the_dependency_terminal_before_siblings_finishoverfail_child/cancel_childwould pin the branch this PR made unconditional, and could absorbtest_failure_is_terminal_once_and_repeated_reports_change_nothing, which now overlaps it. -
Comment wording.
execution_state_migration.py:160-162says no engine branch honored the policies._record_completiondid honorcontinue, and_update_for_terminalhonoredcancel_siblings. The accurate rationale is that no writer set them and no caller consumed the update fields.scheduler.py:313-314says "usually also the first ready_order entry". The duplicate appeared wherever the active class sat inready_order. The new test only covers the active-first case; one case with activeyandready_order=("x", "y")would pin both the prepend and the dedup.
-
Test nits.
test_execution_state_migration.py:355:"failure_policy" not in dump_execution_state(...)is always true, because the field no longer exists.test_execution_engine_scheduler.py:88:scheduler.ready_ids == listedadds nothing. A consumingready_idsalready failslisted == tuple(popped).- The garbage-ledger case is now covered twice (
test_session_queue_dequeue.py:191andtest_session_queue_reads.py). Both run through the same shared helper, so one is enough. test_session_queue_reads.py:390:assert item.session.execution_child_dependenciescould beset(...) == {dependency_id}. Also, real legacy rows can only carry the defaultsfail_parent/cancel_parent, so those would be more representative thancontinuein both fixtures.
Optional, no change needed: the queue-read projection strips a ledger it never exposes (execution_child_dependencies is exclude=True and the projection skips model_post_init). Dropping the runtime ledgers from the projection payload would remove the second call site entirely. Legacy keys inside a nested waiting_workflow_call_child_session are also untested on both paths, but dump_execution_state pops that snapshot while the parent waits, so the case is defense-in-depth only.
Verified statically. Tests weren't re-run locally; CI is green on all platforms, including MySQL and MariaDB.
Summary
Child records in the execution engine carried failure/cancellation policy knobs that no code ever wrote, and
ready_idscould report the same node twice.ready_idscounted the active ready class twice when that class was also listed inready_order, so a queued node appeared twice. Before:ready_idsreturned("a2", "a2", "b1", "q1")whilepop_next()drained("a2", "b1", "q1"). Now each ready class is listed once, in pop order.failure_policy/cancellation_policyare removed: a failed or cancelled child always drives the parent's terminal aggregate, and nothing else could be configured.ContinuationStatususes onecanceledspelling, and the never-calledruntime.replace_streamis deleted.status="failed",Unable to load execution state: ... Extra inputs are not permitted). Now: the real item is returned.QA Instructions
test_ready_ids_lists_each_queued_node_once_in_pop_orderfails on the base withAt index 1 diff: 'a2' != 'b1'(left holds one extra'q1') and passes here. The same test also pins that readingready_idsdoes not consume the queue.dump_execution_statewithfailure_policyandcancellation_policyre-added to one child dependency record. On the base,get_queue_item_for_api()returned a quarantinedfaileditem with the twoextra_forbiddenerrors; heretest_a_row_written_with_child_policy_keys_still_readsasserts the item is readable,status == "in_progress", noerror_message, and the child ledger is populated.test_a_row_with_a_malformed_child_ledger_reads_as_unreadableandtest_fifo_quarantines_unreadable_snapshot_and_dequeues_later_work[garbage]. Both also pass on the base, so they are regression guards, not red-first evidence.pytest -n 4 tests/app/services/shared tests/app/services/session_queue tests/app/services/session_processor: 2165 passed, 23 skipped. Each changed test file also runs alone: scheduler 40, child 15, execution state migration 25, primitives 23, queue reads 33, queue dequeue 40 (2 deselected per-file marks).ruff checkandruff format --checkpass on all 12 changed paths. Limits: the local ruff is 0.11.13 while the pre-commit hook and CI pin 0.11.2, and the hooks could not run in this environment (uv tool/cache dirs read-only, PyPI unreachable), so both hook commands were run directly against the staged paths.scripts/family_e2e/manifest.example.yamlexists, no configured manifest, and no family graph invokesCallSavedWorkflow), mypy (make type-checkcovers thescripts/entrypoints, which do not import this module), and browser checks (no UI change).Review
extra="forbid"child records, so legacy key stripping now lives in one sharedstrip_legacy_child_policy_keysused by both readers, and it recurses into nested child sessions.extra="forbid"is unchanged, so any other unknown persisted key is still rejected. A non-mapping child ledger raisesValueErrorinside the shared helper before it is iterated, keeping it inside the existingexcept (TypeError, ValueError)quarantine contract on both paths.Compatibility / Rollout
failure_policy/cancellation_policyon child dependency records. Both the full rehydration path and the queue-read projection drop those two keys before validation, then validate normally.CURRENT_EXECUTION_STATE_VERSIONis still 2 and the writer never emits those keys, so no migration script or backfill is needed; already-quarantinedfailedrows are not un-quarantined, and retry remains the recovery path.origin/maintouches nothing underinvokeai/frontend/api, so no OpenAPI/type regeneration is required. No deployment ordering.replace_stream, orready_ids), so no documentation is included.Checklist
What's Newcopy (if doing a release after this PR)Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.