[earakely-scale]

Summary

An episode's played agent is now played through a stand-in: an A2A agent with no model, which the task's own deploy_agent or install_agent deploys in the played agent's place.

Before this PR, EpisodeSpec.from_task swapped the played agent's steps for stand-in steps that deployed nothing. So it refused every task that uses that agent's sandbox: install_agent, and load_artifact, collect_artifacts, run_code, verify_sandbox and add_skills aimed at the agent.

With a stand-in deployed there:

• those steps run unchanged;
• a turn gains a bash tool, which runs in the sandbox, as the agent's user, in the agent's folder;
• each turn's tool calls and reply are recorded as the agent's trajectory.

The caller's API keeps its shape: from_task → start() → next_turn() → Turn → end() → result().

How a turn works

prompt_agent ──A2A message──> stand-in (holds it as an open turn)
                                  ^    │ the episode long-polls "next"
                          reply   │    v
     caller <── Turn ── Episode ── turn.tools(): the env's tools, plus bash ──exec──> the stand-in runs it
       │                          │
       └── turn.end(reply) ───────┘  (with the turn's tool calls as its trajectory)
prompt_agent <── an ordinary A2A reply: it records the response and trajectory as for any agent

Where the stand-in runs

The task…	The stand-in…	bash runs…
uses only envs	is served in this process, on loopback	not offered
deploys the agent as a container (deploy_agent)	is the container, from its own image	in /app, as user agent
installs the agent on a VM's host (install_agent)	is installed where the agent it replaces says it works	in that folder, as that user

rules.works_in_sandbox decides which: install_agent, load_artifact, run_code or add_skills aimed at the played agent, verify_sandbox or collect_artifacts aimed at its sandbox, or skills on its deploy_agent.

Changes

agentenv-protocol

• a2a_agent/stand_in.py: StandInAgent.
  • Each A2A message is an open turn until the driver replies.
  • A sender's tasks/cancel withdraws the turn.
  • Skills are written under ~/skills/<name>/.
• urn:agentenv:driver/v1, in extensions.py:
  • claim: the first caller sets a token, which every later call carries in its body.
  • next: a long-poll for the open turns.
  • reply: a reply's text or error, its trajectory and its tool-call count.
  • exec and output: start a command, then long-poll for its result.
• Commands run as bash -c in the stand-in's folder, as its user, with a 3600 s timeout. They return stdout, then stderr, uncapped, as verifiers' bash tool does. On a timeout or cancellation, the command's whole process group is killed, so a background child holding its output can't keep it waiting.

Core

• a2a_agent/stand_in/: the stand-in's Dockerfile and server.
  • It's registered as @local/agentenv-stand-in, an image known by its build context alone. The context holds the installed agentenv_protocol package and its agent requirements, pinned to the versions installed here.
  • No Docker is needed to register it. Each deploy builds it where it runs: this machine's Docker, Modal, or the VM.
  • Its install/v1 installs it on a VM's host, with the host's own Python in a venv.
• A2AAgent: an agent whose metadata sets takes_credentials to False (TAKES_CREDENTIALS) gets no model key, endpoint or headers, from whatever set them, and none of the object store's shared credentials. The stand-in sets it, so nothing bash runs can read a credential. On a VM host, its install commands name none of the litellm_* params.
• install_agent: a per-step stand_in param.
  • It reads the agent it replaces from that agent's card: two new install/v1 params, workdir_host and user_host. It then installs the stand-in there as {workdir} and {user}.
  • It refuses a container-mode install, and a sandbox that runs on this machine, where the client's commands would run on this machine itself.
• The shipped Claude Code agent declares workdir_host="/app" and user_host="root".
• The Episode API:
  • _EpisodeAgentStep now wraps the played agent's real deploy_agent or install_agent, swapping the stand-in in through step_params. For an env-only task, it serves the stand-in in this process instead.
  • Its prompt_agent steps run for real.
  • _EpisodeTurnStep and Episode._play are gone.
  • The episode claims the stand-in and drains its turns.
  • A message maps to the first prompt_agent step of its conversation not yet played.
  • Each tool call is recorded in the trajectory, failed ones with their error.
  • Turn gains skills, and turn.tools() gains bash.
  • Skill is exported beside Turn.

Rules

• Lifted:
  • install_agent for the played agent;
  • load_artifact, collect_artifacts, run_code and verify_sandbox aimed at it;
  • add_skills, and skills on its deploy_agent;
  • rubrics of its trajectory.
• Still refused:
  • retry_config;
  • a user simulator, a trigger executor or a peer played from outside;
  • prompt parts, output_format, agentenv_tools and snapshot_config on its prompt;
  • snapshots and changelogs;
  • register_agent_triggers;
  • agent validators;
  • judging inside the played agent, which a stand-in can't do;
  • a container-mode install.

Notes for review

• Episodes now run with namespace routing on, as agent-env run does, so the @local stand-in resolves. Under routing the object store shares no credentials with any agent, so the other agents of an episode (a judge, say) get none either, and use grants as they do in bundle runs.
• Why the stand-in isn't an entry of the agents bundle:
  • It's machinery the Episode API owns, like the infra envs env/bootstrap.py builds, not an agent a user picks.
  • It has to speak exactly this core's driver/v1, which the installed protocol package guarantees and a PyPI floor wouldn't.
  • A shipped agent's image installs the released protocol, so the stand-in couldn't build until a release.
• The played agent now appears in deployed_agents. Served in this process, it has no sandbox, as a human agent has none.
• Deadlines:
  • A missed one now fails with prompt_agent's own error ("A2A task … did not complete within Ns").
  • turn_timeout is rounded up to a whole second, since agent-config's timeout_seconds is an integer.
  • prompt_agent polls every 2 s, so a turn's reply can take up to 2 s to reach it.
• A turn whose prompt_agent step has failed is closed at once, before the stand-in drops it a long-poll later, so it's never handed out once the run's record says failed. abandon() closes the turns of the run it cancels.
• Turn.end() still never raises. A reply that can't reach the stand-in is logged, and the run's own failure reaches the caller through next_turn() or result().
• The in-process stand-in is one uvicorn server per episode, on loopback in a daemon thread. Its long-polls are 1 s, so one in flight ends within the server's shutdown.

Verification

• Unit: 9,219 passed, 25 skipped, on macOS at the head; ruff clean. New tests cover:
  • the stand-in app;
  • the rules and the copy of the task;
  • turns, bash and the trajectory through the in-process stand-in;
  • install_agent's stand_in param;
  • the credentials opt-out;
  • failed calls in the trajectory, and a continuing conversation's turns.
  • the stand-in's registration.
• CodeBuild's unit run caught a race in a deadline test: a turn that had just been withdrawn could still be handed out. I reproduced it by slowing the episode's view of the stand-in and its teardown. With the fix, the episode suite passes with and without the slowdown.
• Integration, local Docker (a Linux EC2 box): the episode tests pass, among them a new one with the agent deployed as a container.
  • The stand-in image is built from its context.
  • bash runs as agent in /app, with no LITELLM_ variable in its environment.
  • verify_sandbox scores what it left.
  • Nothing is left running.
• Integration, Modal:
  • A new modal_vm test installs an agent on the VM's host and plays it through the stand-in installed in its place. The stand-in works as the declared user in the declared folder, verify_sandbox on the VM scores 1.0, and the VM is stopped afterwards. It ran after the rebase too.
  • The existing Modal episode test passes on the in-process stand-in.
• Regression on the same box, before the rebase (branch against main at 42b65fa, with the open-file limit raised, since the box's 1,024 fails main's unit suite too):

Suite	main	branch
unit	8,738 passed, 2 failed	8,776 passed (+38), the same 2 failed
int-test-fast	171 passed	172 passed (+1)
int-test-slow	74 passed, 490 skipped	74 passed, 491 skipped (+1, the Modal test, with no Modal keys on the box)

The 2 unit failures are environmental: the MCP CLI codegen tests run generated scripts on the box's system Python, which has no httpx.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge.

Summary

Episodes now play outside agents through stand-ins, keeping sandbox tools, skills, and recorded trajectories available.

• Episode callers play each real agent prompt through a Turn.
• Sandbox-dependent task steps run against the stand-in in the agent’s place.
• The stand-in holds each A2A message open for its driver.

Diagram

sequenceDiagram
    participant Task
    participant StandIn as Stand-in
    participant Episode
    participant Caller
    Task->>StandIn: Send prompt
    Episode->>StandIn: Read open turns
    Episode-->>Caller: Return Turn
    Caller->>StandIn: Run tools through Turn
    Caller->>Episode: End Turn
    Episode->>StandIn: Send reply and trajectory
    StandIn-->>Task: Complete prompt

Loading

_{Reviews (3) · Last reviewed commit: "test(stand-in): count a killed child as ..." · Reviewed by Greptile}