fix(docker): harness isolation via COPY/PRUNE + GRADE-OUTSIDE. Close the criteria/grader leak

[CarlesUIPath]

Summary

Under --driver docker, the agent-under-test could read its own grading material — success_criteria, the skills-repo graders (check_*.py), RESOLUTION.md, reference agents, the staged task.yaml, and the per-task-dir mount — and separately could overwrite the operator's ~/.uipath/.auth (it was rw-shared with the host). A nightly audit measured ~2.4% of replicates reading the suite instead of solving the task; an adversarial "check /work" prompt drives it to ~100%.

This closes the leak by absence, not permissions: the agent container never contains grading material, grading moves to the host after the container exits, and host state (auth, checkout) is never rw-shared with the agent.

How it works — before (main) vs. after (this PR)

Before — main: agent and grader share one container/filesystem, and the criteria are staged verbatim into /work/input.

%%{init: {'flowchart': {'useMaxWidth': false, 'padding': 20, 'nodeSpacing': 60, 'rankSpacing': 55}}}%%
flowchart TB
  s1["HOST · stage — task.yaml = FULL criteria; mount RAW skills repo (tests/, check_*.py, reference); ~/.uipath rw-shared"]:::host
  c1["CONTAINER · AGENT + GRADER — same filesystem<br/>agent turn, then grades in-container"]:::leak
  n1["During its turn the agent can read its own success_criteria,<br/>graders, and reference — and overwrite ~/.uipath/.auth"]:::note
  s1 -->|"docker run (TASK_DIR / SKILLS_REPO_PATH point at the above)"| c1
  c1 -.-> n1
  classDef host fill:#eef2ff,stroke:#5566aa,color:#
  classDef leak fill:#ffe0e0,stroke:#cc0000,color:#111;
  classDef note fill:#fff3cd,stroke:#cc9900,color:#

Loading

After — this PR: phase 2 is agent-only; grading is a separate phase 3 on the host.

%%{init: {'flowchart': {'useMaxWidth': false, 'padd'rankSpacing': 55}}}%%
flowchart TB
  s2["HOST · stage (agent-only) — task.yaml strippeared); sanitized skill bundle only — NO raw repo /tests/ / graders / reference / task-dir; ~/.uipath & ~/.claude are throwaway copies"]:::host
  a2["CONTAINER · AGENT ONLY<br/>sees: skill docs ( workspace<br/>cannot see: criteria, check_*.py,tests/, reference, host auth"]:::safe
  g2["HOST · GRADE-OUTSIDE (regrade_on_host)<br/>fua TASK_DIR — agent never shared theirfilesystem"]:::host
  s2 -->|"docker run"| a2
  a2 -->|"container exits → artifacts copied out"| g2
  classDef host fill:#eef2ff,stroke:#5566aa,color:#
  classDef safe fill:#e0ffe0,stroke:#0aa00a,color:#111;

Loading

In one line: split the fused container into ade-on-host (phase 3), and stop mounting the answerkey — or host auth — into the agent's box.

Leaks closed

#	Surface (what the agent could reach on main)	How it's closed
1	Staged criteria — /work/input/task.yaml (full success_criteria) + context.json.source_yaml	task.yaml criteria-stripped (agent_safe_dump), source_yaml nulled, no task_full.json staged; full criteria stay on the host for grading
2	Skills-repo graders — tests/, check_*.py, RESOLUTION.md, reference agents (whole repo mounted)	raw repo not mounted; agent gets only a sanitized :ro bundle (skills/.claude-plugin/commands/agents/hooks)
3	Per-task-dir mount (host_task_dir:host_task_dir)	not mounted; grading reads it on the host via TASK_DIR
4	Reference solution — reference.file / reference.directory	not mounted; template_sources/system_prompt_file auto-mount rejected if it overlaps the task dir
5	Signpost env — TASK_DIR / SKILLS_REPO_PATH	live on the host grader env only; not in the agent's env
6	~/.uipath/.auth (rw-shared → agent could overwrite operator credentials)	agent gets a throwaway copy; the host ~/.uipath (incl. .auth) is never mounted
7	~/.claude/jobs — operator session timelines / conversation copied into every container	excluded from the ~/.claude copy denylist
8	Baked image content — mocks/tooling encoding answers	authoring invariant + baked-image scan test (mounts can't cover this)

Verification

• Docker acceptance (CI-gated): make test-docker-detectors (criteria-absence + host-unchanged proxy + baked-image scan) and the -m live host-unchanged sensor — a real --driver docker run leaves the host byte + metadata identical (also proves ~/.uipath/checkout are untouched).
• Live leak-closure: the adversarial "read your grader" task can no longer obtain the grader sentinel, confirmed on gpt / claude / gemini / kimi.
• Backwards-compat (real runs, all four harnesses): llm_judge, skill_triggered, agent_judge, and simulation (3-turn dialog) all confirmed intact under grade-outside. Early stop is intentionally DISABLED under docker for now (see Limitations).
• make check / typecheck / lint green; full make test = pass modulo pre-existing/environmental failures.

⚠️ The -m live byte-identical check is Linux-authoritative (macOS Docker Desktop's bind-mount remap), but closure is by absence, so it holds on both.

Limitations & follow-ups

• Early stop is a no-op under docker — disabled for now. Criteria are stripped from the container, so the in-container EarlyStopWatcher can't arm. DockerRunner logs a warning; the verdict is unaffected (the host grades the full criteria, and a completed run gates strict-AND). The leak-free follow-up is a host-side watcher over the live event stream (the host already receives the tool-call stream + can signal the container via the heartbeat channel) — see docs/DOCKER_ISOLATION.md § Limitations.
• The durable version of the bundle boundary belongs in the skills repo (an installable bundle that excludes tests/ by construction) — cross-repo follow-up.
• The ~/.claude/~/.uipath copies use a denylist; a follow-up should flip them to an allowlist so new dirs default to excluded.

⚠️ Known blocker — needs decision: pre_run/post_run harness scripts under --driver docker

Context: PR #88 (docker harness isolation, "COPY/PRUNE + GRADE-OUTSIDE"). Kept as draft because of this blocker.

Summary

Grade-outside makes the docker driver leak-safe by no longer mounting the skills-repo tests/ tree into the agent container (that tree holds the graders/criteria — the leak we're closing). But many tasks' pre_run/post_run commands invoke helper scripts that live under tests/ (e.g. python3 $SKILLS_REPO_PATH/tests/tasks/.../seed.py, cleanup_*.py). Those commands currently run inside the container, so under docker they can no longer find their script and fail.

Grading itself is unaffected — the skills suite grades workspace-relative (no criterion references an absolute container path). This is purely a setup/teardown problem.

Impact

Reproducible counts (grep over $SKILLS_REPO_PATH/tests/tasks):

Metric	Count
Total task YAMLs	1147
Distinct tasks whose pre_run/post_run references $SKILLS_REPO_PATH (the affected set)	243 (~21% of the tree)
• with it in post_run	180
• with it in pre_run	123
Tasks running uv sync (must stay in-container)	7

SR="$SKILLS_REPO_PATH/tests/tasks"
{ grep -rl "post_run" "$SR"; grep -rl "pre_run" "$SR"; } | sort -u \
  | xargs grep -l "SKILLS_REPO_PATH" | sort -u | wc -l   # → 243
grep -rlE "uv sync" "$SR" | wc -l                         # → 7

243 is ~21% of the full tree; as a share of the actively-run cloud/e2e suites (uipath-platform, maestro-flow,
maestro-case, agents, ixp, admin, governance, data-fabric) it is a large fraction.

Category split (estimate — derived from reading each distinct script, not a mechanical count):

Phase	Kind	Failure mode under docker today
pre_run	cloud seed / workspace-fixture write	seed with fail_on_error=True (default) → loud task ERROR; fail_on_error: false → agent runs unseeded →
post_run	teardown (read agent output → delete cloud resources)	non-fatal → silently leaves cloud resources orphaned
pre_run	uv sync venv build (7 tasks)	genuinely must run in-container (venv paths aren't portable)

Does this happen without docker? No — tempdir is fine

It's docker-only, and the reason is where the harness runs, not concurrency:

• tempdir: no container. The sandbox is a plain host tempdir, and pre_run/post_run run on the host (inheriting the host env), so $SKILLS_REPO_PATH disk, and creds are present → works.
• docker: pre_run/post_run run inside the container, which grade-outside strips of the tests/ tree → scripts missing.

Both drivers run a single task's `pre_run → y; the difference is the *isolation boundary
*. So this isn't a new class of bug — docker regressed the location where the harness runs.

Root cause

pre_run/post_run are harness operations (setup/teardown), but they currently execute in the agent's container. Grade-outside correctly removes the `testsand the harness scripts vanish with it.

Proposed fix — "harness-outside"

Extend the grade-outside principle to the wh/post_runon the host**, where the full repo + creds live. This does **not** re-open the leak — the scripts never enter the container. In effect, it makes docker do whattempdir` already does.

• post_run → host-only. Teardown/inspepace + cloud is inherently a host concern. No new knob.
• pre_run → new per-command runs_in: host/agent is the opt-in for setup that must build the agent's runtime environment in-container (the 7 uv sync tasks; image-baked fixtures). A host pre_run that writes workspace files runs into a staginr workspace.
• Migration cost is small: with host as default, ~236 of the affected tasks need zero change; only the ~7 uv sync tasks add one line (runs_in: agent)ma + a ~7-line skills-repo edit.

[CarlesUIPath]

pre_run/post_run harness blocker — mostly resolved (960e9a5)

Follow-up on the pre/post-under-docker blocker (tasks whose pre_run/post_run invoke $SKILLS_REPO_PATH/tests/… helper scripts that grade-outside no longer mounts into the agent container — ~229 skills tasks).

Now fixed for the bulk of tasks: pre_run/post_run run host-side under --driver docker (the host has the full repo + creds; the scripts never enter the agent container, so no leak is reopened). pre_run runs before the container into a staging dir that is seeded — files and directory trees — into the agent workspace; post_run runs after the container over the copied-out workspace. Grading stays host-side. Verified end-to-end on Bedrock: host pre_run seed → agent consumes it → capture → host post_run reads the output → host grading (all green).

Still gated to --driver tempdir: the 6 uipath-agents/coded/ tasks whose pre_run runs uv sync + uip codedagent setup --force — an in-container venv build (non-portable absolute paths) plus live-tenant provisioning that can't run host-side. These hard-skip per-task under docker with a redirect message (the rest of a suite still runs). Closing this is the runs_in: agent follow-up (lets a marked pre_run command run its setup step inside the container).

Scope note: this PR stays draft until the runs_in follow-up (or a decision to ship as-is with those 6 on tempdir).

[CarlesUIPath]

Stage 3 — runs_in: host | agent on pre_run (732791d)

Completes harness-outside. A pre_run that must run inside the container (e.g. a uv sync venv build) marks runs_in: agent; the host runs the runs_in: host subset (default), the container runs the agent subset before the agent. post_run stays host-only; default host leaves existing tasks unchanged. Verified end-to-end on Bedrock (host step host-side, agent step in-container, venv built).

Requires task changes to run under docker: the ~6 uipath-agents/coded/ tasks (cp -r _fixtures + uv sync + uip codedagent setup) need their pre_run split in the skills repo — cp stays host, uv sync/uip codedagent setup gets runs_in: agent (a coupled cross-repo change). Additionally, running UiPath tasks under docker needs the uip CLI (@uipath/cli) available in the agent image — an image-provisioning step, separate from this PR.

[github-actions]

Claude finished @CarlesUIPath's task in 1m 31s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Review the full diff (git diff origin/main...HEAD)
• Analyze security improvements and leak closures
• Check cross-file consistency
• Review the pre_run/post_run blocker and proposed solution
• Identify what's missing
• Post comprehensive review

Starting review now...