Skip to content

Stage 12 coordinator can outlive its timeout while waiting and leaves workers running #792

Open

Description

@anth-volk

Problem

The Stage 12 coordinator can be killed before it records a failure, and it leaves workers running after a comparison has failed.

  • Separate wait timeouts: the coordinator waits up to 3,000 s for the baseline worker, then up to a fresh 3,000 s for the reform worker. Its own Modal timeout is 4,500 s. When workers are queued, the two waits can exceed that, and Modal kills the coordinator before it saves the failure.
  • Siblings keep running: when one worker's wait fails or times out, the other worker keeps running. That can mean a 32 GB container for up to 50 minutes on a comparison that has already failed.

One production report from 6 to 13 October 2026 waited 3,000.0175 s for baseline and then stored TimeoutError. Its reform worker had already been interrupted after about 305 s.

Expected

  • Both waits share one deadline that leaves the coordinator time to save the outcome before its Modal timeout.
  • When one worker fails or times out, the coordinator cancels every unfinished worker and records the cancelled one as failed.

Observing baseline and reform concurrently is out of scope here; it is part of #712.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions