Skip to content

fix(workflows): drop unused child policies and read legacy child rows - #10242

Open
JPPhoto wants to merge 1 commit into
invoke-ai:mainfrom
JPPhoto:execution-engine-follow-ups
Open

JPPhoto wants to merge 1 commit into
invoke-ai:mainfrom
JPPhoto:execution-engine-follow-ups

Conversation

@JPPhoto

@JPPhoto JPPhoto commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Child records in the execution engine carried failure/cancellation policy knobs that no code ever wrote, and ready_ids could report the same node twice.

  • ready_ids counted the active ready class twice when that class was also listed in ready_order, so a queued node appeared twice. Before: ready_ids returned ("a2", "a2", "b1", "q1") while pop_next() drained ("a2", "b1", "q1"). Now each ready class is listed once, in pop order.
  • Child failure_policy / cancellation_policy are removed: a failed or cancelled child always drives the parent's terminal aggregate, and nothing else could be configured.
  • ContinuationStatus uses one canceled spelling, and the never-called runtime.replace_stream is deleted.
  • Execution state written while those fields existed stays readable. Before: such a row read back as quarantined (status="failed", Unable to load execution state: ... Extra inputs are not permitted). Now: the real item is returned.

QA Instructions

  • Red-first for the scheduler fix: test_ready_ids_lists_each_queued_node_once_in_pop_order fails on the base with At index 1 diff: 'a2' != 'b1' (left holds one extra 'q1') and passes here. The same test also pins that reading ready_ids does not consume the queue.
  • Red-first for load compatibility: a queue row built from dump_execution_state with failure_policy and cancellation_policy re-added to one child dependency record. On the base, get_queue_item_for_api() returned a quarantined failed item with the two extra_forbidden errors; here test_a_row_written_with_child_policy_keys_still_reads asserts the item is readable, status == "in_progress", no error_message, and the child ledger is populated.
  • Malformed child ledgers (non-mapping) stay quarantined rather than raising out of the read path: test_a_row_with_a_malformed_child_ledger_reads_as_unreadable and test_fifo_quarantines_unreadable_snapshot_and_dequeues_later_work[garbage]. Both also pass on the base, so they are regression guards, not red-first evidence.
  • pytest -n 4 tests/app/services/shared tests/app/services/session_queue tests/app/services/session_processor: 2165 passed, 23 skipped. Each changed test file also runs alone: scheduler 40, child 15, execution state migration 25, primitives 23, queue reads 33, queue dequeue 40 (2 deselected per-file marks).
  • ruff check and ruff format --check pass on all 12 changed paths. Limits: the local ruff is 0.11.13 while the pre-commit hook and CI pin 0.11.2, and the hooks could not run in this environment (uv tool/cache dirs read-only, PyPI unreachable), so both hook commands were run directly against the staged paths.
  • Not run, and why: family E2E (only scripts/family_e2e/manifest.example.yaml exists, no configured manifest, and no family graph invokes CallSavedWorkflow), mypy (make type-check covers the scripts/ entrypoints, which do not import this module), and browser checks (no UI change).

Review

  • Full rehydration alone was not enough: the queue-read projection validates the same extra="forbid" child records, so legacy key stripping now lives in one shared strip_legacy_child_policy_keys used by both readers, and it recurses into nested child sessions.
  • extra="forbid" is unchanged, so any other unknown persisted key is still rejected. A non-mapping child ledger raises ValueError inside the shared helper before it is iterated, keeping it inside the existing except (TypeError, ValueError) quarantine contract on both paths.
  • Remaining limitations: ruff version drift against CI, and family E2E unavailable. Child-aggregation and quarantine message wording are deliberately unchanged.

Compatibility / Rollout

  • Persisted execution state: rows written before this change may carry failure_policy / cancellation_policy on child dependency records. Both the full rehydration path and the queue-read projection drop those two keys before validation, then validate normally. CURRENT_EXECUTION_STATE_VERSION is still 2 and the writer never emits those keys, so no migration script or backfill is needed; already-quarantined failed rows are not un-quarantined, and retry remains the recovery path.
  • No API or generated artifact changed: the diff against origin/main touches nothing under invokeai/frontend/api, so no OpenAPI/type regeneration is required. No deployment ordering.
  • No documented behavior changed (no docs reference the removed fields, replace_stream, or ready_ids), so no documentation is included.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Meaningful regression coverage added / updated where needed; obsolete tests/code removed
  • Persisted-state and API changes include required migrations / compatibility validation
  • Relevant performance/efficiency opportunities considered; material claims have evidence
  • Material review findings resolved and relevant checks rerun
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

ready_ids listed the active class twice when it also sat in ready_order.
Child failure/cancellation policies had no writer, so they are removed and
their persisted keys are dropped on load and on the queue-read projection:
rows written before this change stay readable while their children wait.
ContinuationStatus now uses one "canceled" spelling and the never-called
runtime.replace_stream is deleted.
@github-actions github-actions Bot added python PRs that change python files services PRs that change app services python-tests PRs that change python tests labels Nov 11, 2026
@JPPhoto JPPhoto added the 7.0.0 label Nov 11, 2026

@Pfannkuchensack Pfannkuchensack left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at d24e13c (parent 5b776e3). No blockers. The load-compatibility fix covers both readers, the ready_ids fix is correct for every arrangement, and the removals are safe:

  • No writer ever set a non-default policy (both create_dependency / ChildExecutionCapability call sites in graph.py pass none), and nothing consumed parent_action / cancel_child_ids. Sibling cancellation stays with the queue lifecycle (cancel_workflow_call_children), so behavior under the old defaults is unchanged.
  • ContinuationRecord and ChildExecutionCapability are never persisted, so the cancelled → canceled rename needs no compatibility handling.
  • Not bumping CURRENT_EXECUTION_STATE_VERSION is right: old readers accept rows written here (the removed fields had defaults), and a bump would quarantine them on rollback.
  • The explicit ValueError for a non-mapping ledger is needed, because "garbage".items() would raise AttributeError, which neither except (TypeError, ValueError) catches.

A few low-severity follow-ups:

  1. Dead aggregation branch (child.py:618-643). Failed and canceled completions now return at 612-617, so when len(self.completions) == len(self.children) is reached, every completion is completed. The any(... "failed") / any(... "canceled") scans can't run any more, and the deleted continue-policy test was their only coverage. Suggest collapsing the tail to self.transition_terminal("completed", None) with one return self._result(changed=True); lines 642 and 643 are currently the same return.

  2. Cancellation with pending siblings is untested. Both child tests use fail_child only. The single-child cancel test in test_execution_state_migration.py can't tell "terminal immediately" from "terminal when all done". Parametrizing test_failed_child_makes_the_dependency_terminal_before_siblings_finish over fail_child / cancel_child would pin the branch this PR made unconditional, and could absorb test_failure_is_terminal_once_and_repeated_reports_change_nothing, which now overlaps it.

  3. Comment wording.

    • execution_state_migration.py:160-162 says no engine branch honored the policies. _record_completion did honor continue, and _update_for_terminal honored cancel_siblings. The accurate rationale is that no writer set them and no caller consumed the update fields.
    • scheduler.py:313-314 says "usually also the first ready_order entry". The duplicate appeared wherever the active class sat in ready_order. The new test only covers the active-first case; one case with active y and ready_order=("x", "y") would pin both the prepend and the dedup.
  4. Test nits.

    • test_execution_state_migration.py:355: "failure_policy" not in dump_execution_state(...) is always true, because the field no longer exists.
    • test_execution_engine_scheduler.py:88: scheduler.ready_ids == listed adds nothing. A consuming ready_ids already fails listed == tuple(popped).
    • The garbage-ledger case is now covered twice (test_session_queue_dequeue.py:191 and test_session_queue_reads.py). Both run through the same shared helper, so one is enough.
    • test_session_queue_reads.py:390: assert item.session.execution_child_dependencies could be set(...) == {dependency_id}. Also, real legacy rows can only carry the defaults fail_parent / cancel_parent, so those would be more representative than continue in both fixtures.

Optional, no change needed: the queue-read projection strips a ledger it never exposes (execution_child_dependencies is exclude=True and the projection skips model_post_init). Dropping the runtime ledgers from the projection payload would remove the second call site entirely. Legacy keys inside a nested waiting_workflow_call_child_session are also untested on both paths, but dump_execution_state pops that snapshot while the parent waits, so the case is defense-in-depth only.

Verified statically. Tests weren't re-run locally; CI is green on all platforms, including MySQL and MariaDB.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

7.0.0 python PRs that change python files python-tests PRs that change python tests services PRs that change app services

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants