[yorzi]

What this does

Since #664 a run is claimed once, so a redelivered job never runs it twice. The other side of that, which #664 left out of scope: a worker that dies after its claim (a kill, an out-of-memory error, a deploy that does not wait for jobs) leaves the run running for good. The redelivery sees a running run and leaves it, and nothing else ever looks at it. The same happens after AgentResumeJob claims a paused run, and for a web process that dies inside Agent#test_execute. Nothing in the run says whether anyone is still working on it: started_at is a start, and append_event writes with update_column, so updated_at does not move while the run works.

Ending such a run is only safe by cancelling it. unless_cancelled discards a late result over a cancel, but any other ending is overwritten. A run failed by anything other than a cancel while its worker was only slow came back complete, with the late output and the other ending's error message side by side.

This follows two things already in the engine: the scoring lease from #614 (a token, an expiry, a fenced write, column detection for hosts that have not migrated) and InputRequestExpiryJob (expired when read, plus an optional recurring job). It is four commits:

1. Give each run segment a lease that its progress renews. Migration 033 adds execution_token and execution_expires_at to agent_runs, indexed with status. Every claim of a run for an execution segment writes a new token and an expiry config.run_lease_duration ahead (30 minutes by default). Those claims are claim_pending, claim_pause and the creation in test_execute. Each progress event (one per LLM or tool call) renews the expiry while the run is running under that token. Without the columns, or with the duration set to nil, no lease is taken.
2. Discard a result from a worker whose run left its lease. unless_cancelled takes the worker's token and yields only while the run still holds it. Every write that ends a segment releases the lease (result, pause, failure), so whatever ended the run first is kept, and a late result is a logged no-op. test_execute now records through it as the jobs do, which also stops it overwriting a cancel.
3. Fail a run as lost once its lease runs out. AgentRun#expire_if_lost! is one conditional update on id, status, token and expiry. It records failed with error_class ActionAgent::AgentRun::WorkerLost and a message that says the run was not executed again. The run's page and the run list expire what they reach. ActionAgent::AgentRunExpiryJob reaches the rest and is not scheduled by default. A run with no lease, such as one running when the migration is installed, is never expired.
4. Document run leases and lost workers. A section in docs/framework/dashboard.md, the upgrade note and a Fixed entry in the CHANGELOG, and the option in the generated initializer.

Nothing is executed again, by design. The worker may have died after the provider billed the call or after a tool acted, so the run's message and the docs say so, and running the prompt again is a new run made by a person.

Choices made here that are yours to change:

1. Terminal state. failed with error_class WorkerLost (what record_failure! already writes), so lists, filters and scorecards need no change. The alternative is a new interrupted status.
2. Trigger. Expire on read plus the unscheduled job, mirroring InputRequestExpiryJob. Without a scheduler, a lost run nobody opens stays running, as today.
3. Lease length. 30 minutes, as JUDGE_LEASE_DURATION, renewed only by progress events. A single provider call longer than that looks lost, and its result is then discarded. A heartbeat thread would avoid that but would also keep a wedged process alive. Raising the setting is the documented answer.

Tests, all in actionagent/test/run_lease_test.rb plus one in install_generator_test.rb, 32 in all:

• Lease: the execution job's claim, a resumed segment (new token), and test_execute each take a lease; two claims never share a token; an event renews; a renewal leaves a run that finished or changed token alone; no lease without a duration or without the columns; the migration installs under a custom prefix and reverses.
• Fence: a worker whose run was ended under it records neither its result nor its own failure, in the execution job, the resume job and test_execute. A cancel during test_execute stays. A worker whose run was taken under another token records nothing. A completed, paused or failed segment releases its lease. Plus one real-connection test: the worker held in its generation on its own connection while the run is failed as lost on another.
• Expiry: failed once with WorkerLost, its existing metadata kept, announced once, one log line. Not executed again. A live run, a renewed one or one re-taken since it was read is left running, and a live run costs no query. Only running runs expire, never one without a lease, and nothing without the columns. A cancel that lands first wins. The job fails every lost run and only those. The run's page and the run list show it failed. A late worker's result is discarded after expiry. A run that keeps reporting progress outlasts the lease. Plus a real-connection race of two sweeps (one transition, one broadcast).

Each part of each commit reverted in turn fails at least one test (7 parts in the lease commit, 8 in the fence, 13 in the expiry). Each commit passes its own tests when checked out alone.

Backends: SQLite on Rails 8.1.4 and 7.2.3.2. PostgreSQL 17 through a standalone Active Record probe of the same method bodies, since the dummy schema does not load on PostgreSQL (an index name over 63 characters). One claim in 50 races. The expiry writes the JSON metadata through update_all. Two sweeps make one transition in 50 races. A renewal racing a sweep was consistent in 50 races. A late worker is fenced whichever commits first. The engine suite on PostgreSQL itself is not run.

Not in this PR: stopping a generation early when its renewal fails (it would save spend, but changes no guarantee); a "run again" action; a task for runs that predate the migration (cancel them).

Type of change

• Bug fix
• New feature
• Breaking change
• Documentation
• Performance improvement

Scope check

• I read the contributing guide
• This fits activeagent's stated focus
• This isn't application-specific logic that belongs in user code
• This benefits most users, not just my use case

Related issue

None filed. Follow-up to #664, which names this gap as out of scope. Precedents: #614.

Quality check

• The project's hooks and linters pass (bin/rubocop on the changed files; the repo has no commit hooks installed)
• The new tests fail before this change and pass after it
• The full test suite passes: bin/test with the CI placeholder keys. Rails 7.2, database built as the CI job builds it (db:create db:migrate, so migration 033 applies from scratch): 4426 runs, 0 failures, 0 errors. Rails 8.1: 4422 runs, 0 failures, 10 errors, all CliContractTest (rack-session undefined method 'id' for Hash), which fail identically on an untouched main in this environment. Each commit also passes the lease, lifecycle, generator and input request tests on its own
• I updated documentation where needed (docs/framework/dashboard.md, CHANGELOG with the upgrade note, the generated initializer)
• I didn't edit generated files by hand (the dummy migration and two schema lines follow the scoring lease's commit; a fresh db:create db:migrate builds the same columns)

Ownership

• I understand every line of this change and can answer questions about it

API changes

• Breaking change
• New public methods or classes (AgentRun#expire_if_lost!, #renew_lease!, .expire_lost!, .lease_attributes, .execution_leases?, .lost, AgentRun::WorkerLost, AgentRunExpiryJob, config.run_lease_duration)
• Changed method signatures (AgentRun#unless_cancelled takes an optional token:; callers without it behave as before)
• No API changes

[TonsOfFun]

Approved — thanks! It conflicts with main now (#672/#673 just landed, CHANGELOG and agent_run.rb). Could you rebase? I'll merge once it's green.

[yorzi]

@TonsOfFun ready to go!