Add independently graded browser-agent repair pilot

.github/workflows/ci.yml

@@ -40,9 +40,11 @@ jobs:
         run: |
           pnpm exec tsx --test scripts/cloud/*.test.ts
           pnpm exec tsc -p scripts/cloud/tsconfig.cloud.json
-          pnpm exec eslint scripts/cloud/*.ts --max-warnings=0
+          pnpm exec eslint scripts/cloud/*.ts scripts/cloud/repair-*.mjs scripts/fixtures/replay-process*.mjs --max-warnings=0
+          for file in scripts/cloud/repair-*.mjs; do node --check "$file"; done
           bash -n scripts/cloud/bootstrap.sh
           pnpm exec tsx scripts/cloud/run.ts --help
+          pnpm exec tsx scripts/cloud/repair-run.ts --help
 
       - run: pnpm exec nx run-many -t lint test build typecheck --exclude domscribe-test-fixtures
 

docs/cloud-agent-repair-pilot.md

@@ -0,0 +1,59 @@
+# Agent repair pilot
+
+This development experiment asks a coding agent to reproduce and fix a seeded cart bug. Two variants share an SKU; editing one incorrectly updates both. The baseline gets full Playwright MCP and source access. The Domscribe condition gets the same tools plus a persisted annotation and exact source/runtime queries.
+
+**Cloud qualification is pending provider setup.** The native Codex kit starts, but attaching either existing provider-secret reference returned `409 failedPrecondition` without details. Without a binding, a minimal model request returned `401 Unauthorized`. All accepted diagnostic sandboxes were confirmed deleted. These observations concern this account/run; they do not establish a general platform defect. No credential values were retrieved or copied.
+
+## Run locally
+
+Use Node 24, pnpm 9.12.0 and an authenticated Codex CLI. Each invocation builds Domscribe artifacts without the Nx cache in an isolated worktree at public commit `23063854647954f4001d05a21a928bdc12bdc6fc`, validates their provenance, and creates a separate app worktree for the agent. The primary checkout and its lockfile are preserved.
+
+```sh
+pnpm exec tsx scripts/cloud/repair-run.ts --local --condition=baseline
+pnpm exec tsx scripts/cloud/repair-run.ts --local --condition=domscribe
+```
+
+Local runs use the current CLI sign-in and automatic approval review with workspace write permissions. They install Playwright MCP 0.0.83 in the temporary experiment directory. The agent has eight minutes; setup/build commands have separate deadlines. These runs use your model account. An exact model ID may not be reported by the CLI; raw events and measured token usage are retained when available. No model-provider credential is copied into the app or a cloud sandbox.
+
+## Run in Docker Cloud
+
+After an existing OpenAI provider binding works, run one condition at a time:
+
+```sh
+pnpm exec tsx scripts/cloud/repair-run.ts \
+  --condition=domscribe \
+  --provider-secret=secrets/<existing-resource-UUID>
+```
+
+The argument is a resource reference, not a token. The [Docker SDK provider-credential guide](https://docs.docker.com/ai/sandboxes-api/cookbook/get-a-stored-secret-into-a-sandbox/) describes attaching such references. This runner does not create or rotate them. It signs in to Docker through interactive SDK OAuth, launches one medium Codex kit, runs the existing replay smoke followed by the repair task, collects bounded evidence, and confirms deletion of that sandbox UID. The original 25-minute execution and 30-minute expiration limits apply. Cloud bootstrap setup can continue after the sandbox reaches RUNNING; the pilot separately checks tool/server readiness.
+
+CI runs offline tests and CLI/type/lint/syntax checks. It does not allocate sandboxes or call a model. This is not a GitHub issue-to-PR Action.
+
+## Acceptance and evidence
+
+The fixture uses browser localStorage, not a backend. Both variants receive the same plain issue and browser-first instructions; the Domscribe variant also receives instructions to inspect its annotation and runtime. That additional work is part of the condition and prevents treating a one-pair tool-count comparison as an efficiency benchmark.
+
+The controller must first observe the intended corruption: quantities become `3 / 3 / 1`. The agent may change only `src/App.jsx`. Its patch must include staged and unstaged changes against the immutable seed commit; commits, extra files, symlink substitutions and empty patches fail qualification. The exported patch is applied to a fresh worktree with a fresh frozen npm install. The independent browser verifier then checks:
+
+- Saving and reloading changes only the selected line (`3 / 2 / 1`).
+- Reversing and filtering the list preserve the intended line identity.
+- Cancel leaves the original cart intact.
+- Production UI works without Domscribe instrumentation.
+- No application console, page or network errors occur.
+
+The Domscribe condition also needs a successful runtime query that resolves the selected `line-a`, not just a completed tool call or an ambiguous source lookup. An agent's self-report or exit zero alone cannot pass.
+
+The printed evidence directory contains `pilot-result.json`, `repair-report.json`, `repair.patch`, raw agent JSONL, before/after screenshots, the enriched annotation and build/server logs. Reports identify local versus Docker execution, source/seed/harness/artifact digests, actual tool versions, completed MCP calls and available measured usage. Required cloud transfers retain fixed names and existing size limits. Cancellation stops owned command groups and servers. The grader is kept outside the agent worktree; it is not a security boundary against an adversarial process with access to the whole VM.
+
+Earlier local development trials produced correct source-only fixes in both conditions. Independent review then found and fixed gaps in staged-patch capture, production-build cancellation and artifact provenance. Those earlier trials are diagnostic evidence. Final qualification uses the fresh source and exported-patch path above. See this thread's dated evidence for final results; do not describe local results as cloud verification or claim market demand, speed or cost advantages from one seeded task.
+
+## Local qualification, October 25, 2026
+
+Both final local trials passed after the review fixes. Each independently built all 15 package targets from clean pinned source, verified the same artifact digest, and applied the exported patch to a separate fresh app worktree before grading. Both returned the identical minimal SKU-to-line-ID fix and passed every business assertion with zero application errors. The Domscribe trial resolved the selected line's props and source, including recovering from an ambiguous source-only query with an exact instance.
+
+| Condition           | Independent result | Completed MCP calls | Exact runtime evidence     |
+| ------------------- | ------------------ | ------------------: | -------------------------- |
+| Full browser/source | Pass               |                   5 | Browser/source workflow    |
+| Same plus Domscribe | Pass               |                  15 | Resolved selected `line-a` |
+
+These counts are MCP calls, not all agent actions, and the enriched prompt requires extra Domscribe work. Strategies differed and the exact model ID was not reported. There is one final trial per condition. The result establishes local repair and evidence feasibility; it does not show a navigation, speed, cost or adoption advantage. Cloud qualification remains pending the provider binding. Actual versions: Node 24.22.1, Codex CLI 0.164.0, Playwright MCP 0.0.83, Chromium 156.0.8078.4, React 18.3.1 and Vite 7.3.7.

docs/docker-cloud-replay.md

@@ -60,4 +60,4 @@ bash -n scripts/cloud/bootstrap.sh
 
 The SDK is a private development dependency, pinned at 0.53.0. Its `undici` dependency is narrowly overridden to 6.29.1 to include the upstream security patch; it does not enter Domscribe's public package exports.
 
-The next milestone is an agent attempting a realistic, deliberately broken UI task in this environment, with evidence that its fix changes the expected behavior. GitHub issue ingestion, repository write credentials and opening fix PRs are separate work after that experiment.
+The next experiment is the [agent repair pilot](./cloud-agent-repair-pilot.md), with independent grading of the exported fix. Its local and cloud results are reported separately. GitHub issue ingestion, repository write credentials and opening fix PRs remain separate work.

docs/superpowers/plans/2026-10-25-cloud-agent-repair.md

@@ -0,0 +1,47 @@
+# Cloud Agent Repair Implementation Plan
+
+> **For agentic workers:** Use superpowers:executing-plans inline. Obtain one fresh independent whole-branch review. Steps use checkboxes.
+
+**Goal:** Run and independently grade a seeded UI repair in Docker Cloud using full browser tools, with and without Domscribe.
+
+**Architecture:** A small controlled cart fixture and held-out Playwright verifier run in isolated cloud Git worktrees. The existing SDK lifecycle runner gains narrowly tested kit/secret-reference/extra-artifact options. A pilot CLI builds the fixed workload, launches one condition at a time, invokes the stock coding agent, downloads evidence and checks the independent result.
+
+**Tech Stack:** Node24, pnpm9, @docker/sandboxes0.53.0, stock Codex kit, React18/Vite7 source artifacts, Playwright MCP0.0.83.
+
+**Spec:** docs/superpowers/specs/2026-10-25-cloud-agent-repair-design.md
+
+## Global constraints
+
+- Preserve primary checkouts. Reuse isolated codex/cloud-agent-repair from main2306385.
+- Existing OpenAI cloud secret resource reference only; no credential values or host auth/config.
+- One cloud sandbox at a time, bounded lifecycle and same-UID deletion; no GitHub issue-to-PR wrapper.
+- Full browser/source baseline; only Domscribe availability differs. Record actual model/version/usage, do not claim efficiency from one trial.
+- Real seeded bug must fail independently before agent and pass held-out browser assertions afterward. Source-only nonempty patches, no verifier/config/dependency edits.
+
+## Review focus
+
+- Missing provider/tool readiness must fail rather than launch repeated agents.
+- Agent exit0 or self-report cannot override a failed independent assertion.
+- Evidence from another source/condition or a modified verifier cannot qualify.
+- Secret references must never become token values, command-shell injection or arbitrary source URLs.
+- Cleanup must survive failed agents, evidence transfers and cancellation; ordinary replay defaults remain identical.
+
+## Task1: Controlled fixture and independent grader
+
+- [x] Write failing regression tests for grading invalid before/after state, empty/out-of-scope patches and condition/provenance mismatches.
+- [x] Implement a stable-ID cart fixture, common issue prompt and independent Playwright verification outside the agent worktree.
+- [x] Prove the intended defect fails, and a temporary control correction passes, without giving the control patch to agent sessions. Preserve RED/GREEN evidence.
+
+## Task2: Native agent execution and delivery
+
+- [x] Add RED lifecycle tests for codex kit + nonempty secret reference + bounded safe extra artifacts; implement smallest extension and rerun default17tests.
+- [x] Implement pilot bootstrap/CLI with exact public-source build, full browser MCP, enriched-only Domscribe, source-only patch evidence and held-out grader.
+- [ ] Run native provider readiness probe, then one trial per condition sequentially. Retain failed trials and same-UID cleanup outcomes.
+- [ ] Run affected configured checks, independent review, resolve material findings, document actual scope/results, commit/push focused PR and inspect CI.
+- [ ] Update continuity ledger and automation with merged PR79, pilot results/limits and next action. No inferred admin bypass for a new PR.
+
+## Review and qualification ledger
+
+Independent review found staged-patch omission (P1), missing production-build cancellation (P2), and unverified local artifact provenance (P2). Regression tests reproduced the gaps. Capture now uses the immutable app seed, includes staged/unstaged edits, rejects agent commits, and applies the exported artifact to a fresh verification worktree/install. Local source builds use a separate exact-commit worktree and validated artifact digest; source cleanup is exercised on failure. Production builds receive cancellation signals and subsequent launches check cancellation.
+
+Final local baseline and Domscribe conditions both passed the strengthened path, returning identical minimal patches and the same artifact/harness digests. Both had zero application errors and working uninstrumented production UI. Counts were 5 versus 15 completed MCP calls; no efficiency claim. Cloud provider attachment returned explicit409failedPrecondition; kit without binding reaches the model API but returns401. All accepted diagnostic VMs were deleted. Native cloud trial remains blocked pending account setup; no release/deploy or GitHub issue wrapper.

docs/superpowers/specs/2026-10-25-cloud-agent-repair-design.md

@@ -0,0 +1,13 @@
+# Docker Cloud agent repair pilot
+
+The exact-instance/replay foundation and SDK smoke runner are merged. The next question is whether an agent using those tools can reproduce and repair a UI bug inside Docker Cloud and return a patch whose behavior we can independently verify. The user said to continue and approved the next experiment. This is a development experiment, not a GitHub Action or a claim of developer demand.
+
+Use two fresh in-sandbox Codex sessions with the same seeded React cart task. The cart has two lines with the same SKU and different variants; editing one quantity erroneously updates both. Give both sessions the same plain issue and full Playwright browser tools plus source access. Enable Domscribe only in the enriched condition, with MCP source/runtime/annotation tools. Keep browser, source, dependencies, prompt, model defaults, time limits and independent acceptance criteria equivalent; record differences and actual versions. One trial per condition establishes feasibility only, never statistical speed/cost superiority.
+
+The task is a realistic seeded business defect, not an actual reported customer issue. Acceptance requires only the selected cart line to change across sorting/filtering and reload, cancellation to leave quantities unchanged, and the production build to remain functional without instrumentation. Browser localStorage is the fixture's persistence contract; there is no external backend. Before the agent, independently reproduce the exact wrong-row failure. Afterward verify the patch with held-out browser assertions; retain before/after state, screenshots, source hash, patch, agent events and usage when available. Do not treat exit zero, agent self-report or successful replay as acceptance. An empty patch, altered verifier/config/dependencies, missing reproduction, bad business state or unconfirmed cloud deletion fails qualification.
+
+Reuse the isolated Domscribe worktree on codex/cloud-agent-repair from fresh main23063854647954f4001d05a21a928bdc12bdc6fc. Build local artifacts from a pinned public commit. Use the official SDK's codex kit and an existing OpenAI cloud secret reference; never read or create credential values, upload host auth/config, expose browser ports publicly or supply GitHub write access. Docker OAuth remains SDK-managed. Each sandbox has the smoke runner's bounded deadlines, medium compute, automatic expiration and explicit same-UID deletion. No more than one owned sandbox at a time. Missing provider authentication is a specific blocker, not permission to copy host credentials.
+
+Implementation calls left to the builder: fixture layout, file organization, static port allocation within a fresh VM, orchestration of browser/server readiness and artifact names. Reuse existing lifecycle safeguards and add only the launch/evidence options required for this controlled workload, with regression tests; default replay behavior must remain unchanged. Pin Playwright MCP to the observed registry version and use full browser capabilities, not a custom constrained navigation harness. Keep the grader outside the agent worktree and inject/run final assertions after agent execution. This reduces accidental test tailoring; it is not a security boundary against an adversarial root process.
+
+No automatic GitHub issue ingestion, fix PR creation, publication, production deployment or broader agent platform. Report unsuccessful trials and uncertainty honestly. The first deliverable is a repeatable pilot and actual evidence, with independent branch review and CI before PR readiness.

scripts/cloud/repair-agent.test.ts

@@ -0,0 +1,73 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { parseAgentEvents } from './repair-agent.js';
+test('counts successful full browser and Domscribe tool calls, retaining measured usage only', () => {
+  const log = [
+    {
+      type: 'item.completed',
+      item: {
+        type: 'mcp_tool_call',
+        server: 'browser',
+        tool: 'browser_navigate',
+        status: 'completed',
+        result: { isError: false },
+      },
+    },
+    {
+      type: 'item.completed',
+      item: {
+        type: 'mcp_tool_call',
+        server: 'domscribe',
+        tool: 'domscribe.targets.query',
+        status: 'completed',
+        result: {},
+      },
+    },
+    {
+      type: 'turn.completed',
+      usage: { input_tokens: 100, output_tokens: 20, cached_input_tokens: 50 },
+    },
+  ]
+    .map((x) => JSON.stringify(x))
+    .join('\n');
+  const r = parseAgentEvents(log);
+  assert.deepEqual(r.tools, [
+    'browser.browser_navigate',
+    'domscribe.domscribe.targets.query',
+  ]);
+  assert.equal(r.usage.length, 1);
+});
+test('failed tool calls and a claimed browser action in final text are not browser-use evidence', () => {
+  const log = [
+    {
+      type: 'item.completed',
+      item: {
+        type: 'mcp_tool_call',
+        server: 'browser',
+        tool: 'browser_navigate',
+        status: 'failed',
+      },
+    },
+    {
+      type: 'item.completed',
+      item: { type: 'agent_message', text: 'I used browser.browser_navigate' },
+    },
+  ]
+    .map((x) => JSON.stringify(x))
+    .join('\n');
+  assert.deepEqual(parseAgentEvents(log).tools, []);
+});
+
+test('Codex-normalized MCP errors are not successful tool evidence', () => {
+  const event = {
+    type: 'item.completed',
+    item: {
+      type: 'mcp_tool_call',
+      server: 'browser',
+      tool: 'browser_navigate',
+      status: 'completed',
+      result: { is_error: true },
+    },
+  };
+  assert.deepEqual(parseAgentEvents(JSON.stringify(event)).tools, []);
+});

scripts/cloud/repair-agent.ts

@@ -0,0 +1,51 @@
+/** Extract completed MCP calls and recorded token usage from Codex JSONL. */
+export function parseAgentEvents(text: string): {
+  tools: string[];
+  usage: unknown[];
+  runtimeTargets: string[];
+} {
+  const tools: string[] = [],
+    usage: unknown[] = [];
+  const runtimeTargets = new Set<string>();
+  for (const line of text.split('\n')) {
+    let event;
+    try {
+      event = JSON.parse(line);
+    } catch {
+      continue;
+    }
+    if (event?.type === 'item.completed') {
+      const i = event.item;
+      if (
+        i?.type === 'mcp_tool_call' &&
+        i.status === 'completed' &&
+        i.result?.isError !== true &&
+        i.result?.is_error !== true &&
+        !i.error &&
+        typeof i.server === 'string' &&
+        typeof i.tool === 'string'
+      ) {
+        tools.push(i.server + '.' + i.tool);
+        const data =
+          i.result?.structured_content ?? i.result?.structuredContent;
+        if (
+          i.server === 'domscribe' &&
+          /targets[._]query|query[._]bySource/.test(i.tool) &&
+          data?.browserConnected === true &&
+          data.total === 1 &&
+          (data.status === 'resolved' || data.selectionStatus === 'resolved')
+        ) {
+          const id = (data.context ?? data.runtime)?.componentProps?.line?.id;
+          if (typeof id === 'string') runtimeTargets.add(id);
+        }
+      }
+    }
+    if (
+      event?.type === 'turn.completed' &&
+      typeof event.usage === 'object' &&
+      event.usage !== null
+    )
+      usage.push(event.usage);
+  }
+  return { tools, usage, runtimeTargets: [...runtimeTargets] };
+}