Skip to content

Fixes #10: Fix recovery from stale running jobs during migration cutover - #47

Merged
jnovinger merged 1 commit into
mainfrom
10-a-stale-running-job-blocks-migration-cutover-after-its-worker-stops
Oct 20, 2026
Merged

jnovinger merged 1 commit into
mainfrom
10-a-stale-running-job-blocks-migration-cutover-after-its-worker-stops

Conversation

@pheus

@pheus pheus commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes: #10

Preserves the execution state of captured running Jobs so deleting a confirmed stale Job does not cause the reference pass to replay it.

Adds regression tests and documents how to recover when a stale Job blocks migration cutover, with pointers from cutover messages to the migration guide.

@pheus
pheus marked this pull request as ready for review October 15, 2026 16:47
@pheus
pheus requested a review from jnovinger October 15, 2026 16:47
jnovinger
jnovinger previously approved these changes Oct 16, 2026

@jnovinger jnovinger left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ternary at cutover.py:497 also turns a deleted row on an executed entry into executed instead of vanished, but the new tests only drive the running case. Did you want a test pinning that one too?

@pheus

pheus commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks! The two _close() calls after deletion are intended to cover running → executed → executed. I’ll add an assertion after each call and reload the journal between retries to make that coverage explicit.

@pheus
pheus force-pushed the 10-a-stale-running-job-blocks-migration-cutover-after-its-worker-stops branch from e889447 to 558ce13 Compare October 20, 2026 15:41
Record captured running jobs before refusing cutover and preserve their
execution state after deletion so the reference pass does not replay
them.

Add regression tests and document stale-job recovery, with pointers from
cutover messages to the migration guide.

Fixes #10
@pheus
pheus force-pushed the 10-a-stale-running-job-blocks-migration-cutover-after-its-worker-stops branch from 558ce13 to 27673cd Compare October 20, 2026 18:12
@pheus
pheus requested a review from jnovinger October 20, 2026 18:14
@jnovinger
jnovinger merged commit 432f8f5 into main Oct 20, 2026
20 checks passed
@jnovinger
jnovinger deleted the 10-a-stale-running-job-blocks-migration-cutover-after-its-worker-stops branch October 20, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A stale Running Job blocks migration cutover after its worker stops

2 participants