feat(cli): task run and eval run always go through the configured [runner]

[AlexS-scale]

task run and eval run now always hand their runs to the configured [runner] and follow them. There is no --submit flag, so the same commands work whether the runner is LocalRunner (the default, in this process) or an external one such as a Temporal or LittleHorse plugin.

Builds on #255 (per-run overrides) and #256 (submit(instance_id=...) and KEEP_KEY), both on main now; this diff is the CLI change and the runner fixes it needed.

Every option keeps working, with the same default under any runner:

Option	How it reaches the runner
--gateway-env-id, --service-db-env-id, and the other per-run overrides	user_overrides in the run's metadata (#255)
--context-json + --start-step	The CLI stores the file's context as a new instance, within the task's run scope, and submits that instance_id with start_step; if that write fails, the command fails before anything is submitted. The runner continues it from the stored record (#256). The command's --start-step and --keep always win over values saved in the file.
--keep	metadata[KEEP_KEY]. The runner leaves a run that ended on its own up (#256). The CLI prints it, holds it until Ctrl-C and tears it down from the stored instance; if following the runs fails, it tears it down before reporting the error. A run Ctrl-C cancels is torn down by the runner.
--max-concurrency (eval)	The CLI keeps at most that many of its runs in flight. The runner's own capacity still applies.

Following a run. Progress comes from the run index and the task instance, the records every runner keeps.

• A step is shown as the run ran it, with the instance's seed substituted into its id and verifier_id.
• A single task run prints each step's output, replayed from that step's own journal entry.
• Parallel runs and eval run print each step and each verifier step's score.
• Exit codes are unchanged: 130/143 on Ctrl-C/SIGTERM, 1 if a run failed. A run context task run can't write to --output-dir also exits 1, after teardown and --keep.
• The CLI never calls runner.start(), and always awaits stop() unless a second Ctrl-C stops waiting.

Runner changes this needed:

• Instance ids. A run's instance id derives from its task id (new_instance_id, the convention create_instance already used), for LocalRunner.submit and the CLI's pre-store. An @local task's run, and ids derived from it such as collected artifacts, stay in the @local namespace. Explorer-submitted runs get such ids too; the explorer already URL-encodes instance ids.
• Run ownership. Each LocalRunner run records the process that owns it (host and pid), which marks it seen every OWNER_SEEN_SECONDS (30) while it runs. start(), and a started runner on each tick after it, fails only a run whose owner is gone: on this host, its pid has exited; on another, it hasn't seen the run within OWNER_LEASE_SECONDS (120), so a hub back under a new hostname still reconciles its old runs. A run that names no owner is an orphan too. Opening the explorer while a CLI run is going no longer marks that run FAILED. cancel() only cancels a run this runner is running; for any other it returns False and leaves the record to its owner, so the explorer can no longer mark a CLI's run CANCELED while it goes on (which, under --keep, left its sandbox up).
• Run index. The run index makes its indexes on first use, since a CLI-hosted LocalRunner is never started.

Behaviour change: concurrency is the runner's everywhere. Under the default LocalRunner that is [runner.config] workers (2), where the in-process CLI used to start every --k run at once.

Limitations:

• task run-batch, bundle agent-env run and the validators still run in-process.
• A pid the OS has reused looks alive, so a run that owner left stays in flight until that pid exits.
• When a run resumes at --start-step, its seeded earlier steps are echoed as completed.

Tests:

• The task run / eval run CLI tests go through LocalRunner.
• task_run_stored_test.py runs real stored tasks through the real stores, all through the actual CLI:
  • An @local task, fresh and continued from --context-json, writes no run data to the configured store.
  • Seeded step ids and verifier ids report correctly at k=1 and k=2.
  • An unexportable context fails the command, and a failed run's own failure is preserved.
• Two LocalRunners sharing a store: start() spares the other's live run and fails a dead process's. A cancel through the runner not running a run returns False and leaves it running, and the owner's cancel still stops and tears it down. Through the CLI, a --keep run another runner tried to cancel completes and is held and torn down with its sibling.
• Each of these fails on 52022bf and passes now.

Confidence Score: 5/5

The PR appears safe to merge; the remaining reported cleanup issue is addressed.

Summary

task run and eval run submit through the configured runner and follow stored progress.

• Task and eval commands submit runs to the configured runner and follow them.
• Task runs resume from stored context and can keep their sandboxes up.
• LocalRunner tracks run owners and only cancels runs it owns.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Submit runs] --> B[Poll each unfinished run]
  B --> C[Save progress and collect read errors]
  C --> D{Any read failed?}
  D -->|No| E{All runs finished?}
  E -->|No| B
  E -->|Yes| F[Await runner.stop]
  D -->|Yes| F
  F --> G[Export results or report error]
  G --> H[Hold or tear down finished kept runs]

Loading

_{Reviews (13) · Last reviewed commit: "fix(cli): a failed status read doesn't h..." · Reviewed by Greptile}