fix(docker): harness isolation via uid-drop barrier — close the criteria/grader leak by CarlesUIPath · Pull Request #85 · UiPath/coder_eval

CarlesUIPath · 2026-08-05T14:42:43Z

Summary

Under --driver docker, the agent-under-test could read its own grading material —
success_criteria, the skills-repo graders (check_*.py), RESOLUTION.md, reference
agents, the staged task.yaml, and the per-task-dir mount. A colleague's nightly audit
measured ~2.4% of replicates reading the suite instead of solving the task (claude-code
highest); an adversarial "check /work" prompt drives it to ~100% across frontier and
open-weight models. Prior denylist mitigations (zip/base64/compile/delete) did not hold.

This installs an allowlist permission barrier: the container stays root, but the agent's
CLI subprocess is dropped to an unprivileged agent uid (2000). All grading material is
root-owned chmod 0700 → EACCES to the agent, while grading (SuccessChecker /
run_command / judges) runs as root. The agent can read only its skill-docs and write its
own workspace.

How it works

Agent-agnostic drop: a setpriv shim; claude via ClaudeAgentOptions.user, codex via
launch_args_override, antigravity via the localharness PATH-shadow. The orchestrator/
grading process never leaves root.
Criteria strip + root-only channel: the agent-readable task.yaml is criteria-stripped
(agent_safe_dump); the real criteria/reference ride root-only in task_full.json and are
merged back (raw-dict, before parsing) for grading.
Single choke point: container_perms.lock_harness_root_0700 / grant_agent_ownership,
guarded by lint rule CE033.

Leak surface	Closed by
Staged criteria (`/work/input`, `task_full.json`)	root-0700 lock + strip
Skills-repo graders (`check_*.py`, `RESOLUTION.md`, reference agents)	raw host mount forwarded + locked
Per-task-dir mount	root-0700 lock
Reference solution (`reference.file`/`directory`)	mount forwarded + locked
Signpost env (`TASK_DIR`, `SKILLS_REPO_PATH`)	live on the root grader env only
Baked image content	authoring invariant

📖 Setup & design reference: the agent-uid ↔ root-grader arrangement, how tasks are
staged/authored under it, the barrier surfaces, the Docker-Desktop caveat, and the nightly
rollout are documented in docs/DOCKER_ISOLATION.md (extended in this PR).

Verification

New Docker test coverage (permanent, CI-gated). Adds a root-in-container acceptance
suite (tests/test_docker_user_separation.py, run by the new docker-isolation CI job via
make test-docker-isolation): it forks a child dropped to the agent uid and asserts EACCES
on all six surfaces + a real dropped-CLI check (id -u == 2000, HOME writable, grader/
task_full.json unreadable) inside the freshly built image on native Linux — the
authoritative environment (see caveat).

Independently confirmed the jailbreak reproduces → is closed. Beyond the committed
suite, an out-of-tree reachability harness was run in two modes on the same image:
reproduce (barrier omitted) shows the six surfaces ARE readable to the dropped uid —
reproducing the jailbreak — and verify-fixed (barrier applied via the real primitives)
shows they become EACCES. The before/after pair proves the fix actually closes the leak
(and that the check isn't vacuous).

Local: gates green; the Docker acceptance suite passes; full make test = 4009 passed
with only pre-existing/environmental failures (live-API judge tests needing creds, local
TASK_DIR/task-tag pollution, a stale model-name pin, and report-command output strings
that live on main) — none touch this diff.

⚠️ The barrier is Linux-authoritative. On macOS/Windows Docker Desktop the
gRPC-FUSE/VirtioFS uid-remap defeats chmod 0700 on bind mounts, so local bind-mount
checks are unreliable — trust the Linux CI job. (Locked mounts are :rw so the in-container
chmod applies; the lock fails loud if it can't.)

Out of scope / follow-ups

tempdir driver + the Windows nightly slice (no container/uid boundary) — separate.

+            if errors is not None:
+                errors.append((path, exc))
+            else:
+                logger.debug("container_perms: could not chown/chmod %s: %s", path, exc)


…eria/grader leak Under --driver docker, run the agent-under-test's CLI subprocess as an unprivileged `agent` uid (2000) while the container stays root, so grading material (success_criteria, task_full.json, skills-repo graders check_*.py, RESOLUTION.md, reference agents, the per-task-dir mount, reference solutions) is root-owned mode-0700 and returns EACCES to the agent, while grading runs as root. Closes the leak where agents read their own answer key (measured ~2.4% of nightly replicates; ~100% under an adversarial "check /work" prompt). Covers claude-code / codex / antigravity: a setpriv drop shim, claude via ClaudeAgentOptions.user, codex via launch_args_override, antigravity via the localharness PATH-shadow. Criteria stripped from the agent-readable task.yaml (agent_safe_dump) and restored root-only from task_full.json before grading (raw-dict merge). Hardening found via multi-model review + real containerized runs: - forward raw plugin/skills-repo host mounts via context.json and lock them - mount locked staging dirs read-write so the in-container 0700 chmod applies (a :ro mount EROFS-no-op'd the lock silently); lock now fails LOUD - relocate HOME/CODEX_HOME to an agent-owned /home/agent under the drop - lock the reference-solution mount (reference.file / reference.directory) - reject agent_run_uid authored from YAML/CLI/variant (framework-set only) - CE033 lint guards the container_perms choke point, forbids a container --user, and flags a raw task.model_dump into the staged yaml Test coverage: `make test-docker-isolation` runs a root-in-container six-surface EACCES-as-uid-2000 proof + a real dropped-CLI acceptance inside the built image (new `docker-isolation` CI job). The barrier was independently confirmed with an out-of-tree reachability harness (reproduce vs verify-fixed) showing the jailbreak reproduces on the vulnerable image and is EACCES under the barrier. Barrier is Linux-authoritative (native overlayfs); macOS Docker Desktop's bind-mount uid-remap makes local bind-mount checks unreliable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

+        # the barrier sets, not mkdtemp's default.
+        anc = tmp_path
+        while anc != anc.parent and str(anc).startswith("/tmp"):
+            os.chmod(anc, 0o711)  # traverse-only (o+x); NOT world-readable


+            os.chmod(anc, 0o711)  # traverse-only (o+x); NOT world-readable
+            anc = anc.parent
+        for extra in (output_dir, output_dir / "artifacts"):
+            os.chmod(extra, 0o711)  # traverse-only (o+x); NOT world-readable


+        # exercise the LEAF (agent-owned ws) perms, not mkdtemp's default.
+        anc = tmp_path
+        while anc != anc.parent and str(anc).startswith("/tmp"):
+            os.chmod(anc, 0o711)  # traverse-only (o+x); NOT world-readable


+        # Make the temp ancestor chain traversable (production /work mounts are).
+        anc = tmp_path
+        while anc != anc.parent and str(anc).startswith("/tmp"):
+            os.chmod(anc, 0o711)  # traverse-only (o+x); NOT world-readable


CarlesUIPath · 2026-08-06T10:07:09Z

Closing this PR — the uid-permission barrier isn't viable across OSs: on native Linux the chmod 0700 on the raw host bind mounts corrupts the real skills checkout (owner locked out, git status breaks), and on Docker Desktop the uid-remap defeats the lock so it doesn't protect at all. It's also SDK-version-coupled and fragile.

Another PR will follow up (copy/prune + grade-outside): the agent gets only a sanitized :ro bundle, the raw repo is never mounted, and grading runs outside the agent's reach.

…e the criteria/grader leak Under --driver docker the agent could read its own grading material (success_criteria, skills-repo graders check_*.py, RESOLUTION.md, reference agents, the staged task.yaml, the per-task-dir mount). A nightly audit measured ~2.4% of replicates reading the suite instead of solving the task; an adversarial "check /work" prompt drives it to ~100%. Prior denylist mitigations did not hold, and a uid-permission barrier (PR #85, closed) corrupted the host checkout on Linux and was defeated by the macOS uid-remap. This closes the leak by ABSENCE, not permissions: - Agent container mounts ONLY a sanitized :ro skill bundle (project_plugin_for_agent: skills/.claude-plugin/commands/agents/hooks — no tests/, graders, or reference) + its throwaway workspace. The raw skills repo, reference, and task-dir are NOT mounted. - The staged task.yaml is criteria-stripped (agent_safe_dump: success_criteria=[], reference=None) and context.json.source_yaml is nulled; no task_full.json is staged. - Grading runs on the HOST after the container exits (regrade_on_host, the evaluate-only Orchestrator seam), against the full criteria the host holds — so criteria/graders and the agent never share a filesystem or a moment in time. Nothing chmods a host mount. Hardening found via multi-model review + real containerized runs (codex/claude/gemini/kimi): - regrade switches the sandbox driver off 'docker' (Sandbox.setup rejects driver=docker) — without it every docker task ERRORed. - regrade seeds the container's turns (existing_turns, deep-copied) so trajectory-based criteria (skill_triggered / command_executed / agent_judge / llm_judge transcript) grade against the REAL trajectory — without it skill_triggered reported ~0 activation. - _copy_claude_home ignores jobs/ (operator session/conversation no longer copied in). - ~/.uipath forwarded as a throwaway rw COPY (never the host original). - pre_run/post_run not re-run on the host re-grade; template_sources/system_prompt_file auto-mount rejected if it overlaps the host task dir; authored empty success_criteria rejected at load (container re-parse bypasses via allow_empty_criteria); re-grade fail-safe never leaves a false SUCCESS on disk. Early-stop is a documented no-op under docker (criteria are stripped from the container, so the in-container watcher cannot arm); DockerRunner warns, verdict is unaffected (host grades the full criteria). The leak-free follow-up is a host-side watcher over the live event stream — see docs/DOCKER_ISOLATION.md § Limitations. Verification: make test-docker-detectors (absence + host-unchanged proxy + baked-image scan + no-uid-machinery) and the -m live host-unchanged sensor (a real docker run leaves the host byte + metadata identical) — both wired into the docker-isolation CI job. Backwards-compat confirmed on real runs: llm_judge / skill_triggered / agent_judge / simulation / early-stop all intact under grade-outside across claude/codex/gemini/kimi. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-advanced-security AI found potential problems Aug 5, 2026

View reviewed changes

CarlesUIPath changed the title ~~feat(docker): harness isolation via uid-drop barrier — close the criteria/grader leak~~ fix(docker): harness isolation via uid-drop barrier — close the criteria/grader leak Aug 5, 2026

CarlesUIPath force-pushed the fix/docker-harness-isolation branch from 83aad3c to 61a157f Compare August 5, 2026 14:57

CarlesUIPath force-pushed the fix/docker-harness-isolation branch from 61a157f to e67c70b Compare August 5, 2026 15:05

github-advanced-security AI found potential problems Aug 5, 2026

View reviewed changes

CarlesUIPath closed this Aug 6, 2026

Navigation Menu

Uh oh!

fix(docker): harness isolation via uid-drop barrier — close the criteria/grader leak - #85

fix(docker): harness isolation via uid-drop barrier — close the criteria/grader leak#85
CarlesUIPath wants to merge 1 commit into
mainfrom
fix/docker-harness-isolation

CarlesUIPath commented Aug 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CarlesUIPath commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

CarlesUIPath commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Verification

Out of scope / follow-ups

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CarlesUIPath commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CarlesUIPath commented Aug 5, 2026 •

edited

Loading