Skip to content

Add cooperative drain controls for ephemeral runners - #4708

Draft
ashi009 wants to merge 2 commits into
actions:mainfrom
ashi009:ashi009/ephemeral-idle-drain
Draft

Add cooperative drain controls for ephemeral runners#4708
ashi009 wants to merge 2 commits into
actions:mainfrom
ashi009:ashi009/ephemeral-idle-drain

Conversation

@ashi009

@ashi009 ashi009 commented Sep 12, 2026

Copy link
Copy Markdown

Drafted with Codex assistance; open as a draft for author review.

Summary

Adds cooperative drain for --ephemeral runners with two explicit, independent inputs:

  • run --drain-on-sigusr1: SIGUSR1 latches a drain request on Linux/macOS, without a control file.
  • run --drain-file PATH: watches a supervisor-owned file at a configurable location. No default path and no writes to the installation directory. The two inputs can be enabled together.

Refs #2190 (original graceful-shutdown request) and #2582 (SIGINT follow-up). Credits #4461 for the sentinel approach. Persistent-runner support and existing SIGINT/SIGTERM semantics are unchanged.

Implementation

  • DrainRequest owns a shared, thread-safe latch and optional signal registration. Observed requests survive file removal and listener-session restarts, but not process restarts.
  • Both message listeners check the latch between long polls, including internal retries after empty responses. The signal handler only latches state; it does not cancel HTTP requests or worker tokens.
  • An outstanding response carrying a job is acknowledged/acquired and processed normally. Once a job is received, cancellation-message polling and the existing one-job completion path continue; drain does not enter ShutdownAsync early and cancel the worker.
  • Idle exit retains registration/local credentials. Normal ephemeral job completion retains automatic deregistration. The supervisor owns the optional file, listener restart policy, and host shutdown.
  • Add CLI help, usage documentation, and tests for the shared latch, native SIGUSR1, explicit file selection, missing option values, ephemeral-only validation, and dispatch boundaries.

For read-only containers, use the signal input or a separately mounted runtime directory for the optional file. Send SIGUSR1 directly to the opted-in listener after it is ready, not to the shell/service wrapper. The existing wrapper stop path can force-kill after 30 seconds and is not the drain interface.

Verification

  • The original file-only implementation passed nine regression cases against upstream 759385a3510197a58b5c08dc1f373b74b9f4643b; the unmodified baseline passed five and timed out in four idle/pre-listen cases.
  • That implementation also passed live temporary-runner smoke tests on Linux: idle exit 0 after 52.1 seconds, and a real job completed after a drain request before normal listener exit.
  • For this updated implementation, a small macOS probe compiled the production DrainRequest source and verified actual SIGUSR1 delivery without a file, optional file selection, and latching after file removal.
  • ./dev.sh format passed. The updated full regression suite still needs CI; upstream workflow runs currently require maintainer approval. The earlier file-only results are not claimed as coverage of the new signal integration.

Idle latency includes outstanding long polls. Ambiguous delivery after transport failures, forced termination, and actual runner upgrades are outside the verification claims.

@ashi009 ashi009 changed the title Add cooperative idle drain for ephemeral runners Add cooperative drain controls for ephemeral runners Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants