You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A job outside a pipeline goes back to PENDING on retry, and nothing re-enqueues it (worker/lib/decorators/job_management.py). Cleanup (worker/jobs/system/cleanup.py) then counts it as stalled and uses up another retry, so max_retries=3 gives one real retry.
scripts/run_job.py enqueues a job before committing its row, so the worker can pick up a row that doesn't exist yet.
Cleanup holds a row lock across a Redis await, and checks QUEUED jobs with no grace period, which can enqueue a job twice.
Scope
Reset mapping_state on cancellation as well as on exceptions.
Re-enqueue a retried job that isn't in a pipeline.
Commit the job row before enqueueing in run_job.py.
Release the row lock before the Redis await in cleanup, and give QUEUED jobs a grace period before they count as stalled.
Acceptance criteria
A test cancels a mapping job on timeout and asserts mapping_state isn't left processing.
A test asserts a standalone job with max_retries=3 runs up to four times.
A test asserts run_job.py commits before it enqueues.
worker/pipeline_management.md and the comments in cleanup.py match the new behavior.
Problem
Job lifecycle gaps remain after the stuck-job work in #788.
mapping_statestaysprocessingforever after the mapping job is cancelled on timeout. The reset inworker/jobs/variant_processing/mapping.pyruns only on ordinary exceptions, and Redesign stuck-RUNNING-job detection: progress-heartbeat + abort-first recovery #788 fixes only the job-run row.worker/lib/decorators/job_management.py). Cleanup (worker/jobs/system/cleanup.py) then counts it as stalled and uses up another retry, somax_retries=3gives one real retry.scripts/run_job.pyenqueues a job before committing its row, so the worker can pick up a row that doesn't exist yet.Scope
mapping_stateon cancellation as well as on exceptions.run_job.py.Acceptance criteria
mapping_stateisn't leftprocessing.max_retries=3runs up to four times.run_job.pycommits before it enqueues.worker/pipeline_management.mdand the comments incleanup.pymatch the new behavior.