diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ae1da04..87ce616 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,9 +40,11 @@ jobs: run: | pnpm exec tsx --test scripts/cloud/*.test.ts pnpm exec tsc -p scripts/cloud/tsconfig.cloud.json - pnpm exec eslint scripts/cloud/*.ts --max-warnings=0 + pnpm exec eslint scripts/cloud/*.ts scripts/cloud/repair-*.mjs scripts/fixtures/replay-process*.mjs --max-warnings=0 + for file in scripts/cloud/repair-*.mjs; do node --check "$file"; done bash -n scripts/cloud/bootstrap.sh pnpm exec tsx scripts/cloud/run.ts --help + pnpm exec tsx scripts/cloud/repair-run.ts --help - run: pnpm exec nx run-many -t lint test build typecheck --exclude domscribe-test-fixtures diff --git a/docs/cloud-agent-repair-pilot.md b/docs/cloud-agent-repair-pilot.md new file mode 100644 index 0000000..001181d --- /dev/null +++ b/docs/cloud-agent-repair-pilot.md @@ -0,0 +1,59 @@ +# Agent repair pilot + +This development experiment asks a coding agent to reproduce and fix a seeded cart bug. Two variants share an SKU; editing one incorrectly updates both. The baseline gets full Playwright MCP and source access. The Domscribe condition gets the same tools plus a persisted annotation and exact source/runtime queries. + +**Cloud qualification is pending provider setup.** The native Codex kit starts, but attaching either existing provider-secret reference returned `409 failedPrecondition` without details. Without a binding, a minimal model request returned `401 Unauthorized`. All accepted diagnostic sandboxes were confirmed deleted. These observations concern this account/run; they do not establish a general platform defect. No credential values were retrieved or copied. + +## Run locally + +Use Node 24, pnpm 9.12.0 and an authenticated Codex CLI. Each invocation builds Domscribe artifacts without the Nx cache in an isolated worktree at public commit `23063854647954f4001d05a21a928bdc12bdc6fc`, validates their provenance, and creates a separate app worktree for the agent. The primary checkout and its lockfile are preserved. + +```sh +pnpm exec tsx scripts/cloud/repair-run.ts --local --condition=baseline +pnpm exec tsx scripts/cloud/repair-run.ts --local --condition=domscribe +``` + +Local runs use the current CLI sign-in and automatic approval review with workspace write permissions. They install Playwright MCP 0.0.83 in the temporary experiment directory. The agent has eight minutes; setup/build commands have separate deadlines. These runs use your model account. An exact model ID may not be reported by the CLI; raw events and measured token usage are retained when available. No model-provider credential is copied into the app or a cloud sandbox. + +## Run in Docker Cloud + +After an existing OpenAI provider binding works, run one condition at a time: + +```sh +pnpm exec tsx scripts/cloud/repair-run.ts \ + --condition=domscribe \ + --provider-secret=secrets/ +``` + +The argument is a resource reference, not a token. The [Docker SDK provider-credential guide](https://docs.docker.com/ai/sandboxes-api/cookbook/get-a-stored-secret-into-a-sandbox/) describes attaching such references. This runner does not create or rotate them. It signs in to Docker through interactive SDK OAuth, launches one medium Codex kit, runs the existing replay smoke followed by the repair task, collects bounded evidence, and confirms deletion of that sandbox UID. The original 25-minute execution and 30-minute expiration limits apply. Cloud bootstrap setup can continue after the sandbox reaches RUNNING; the pilot separately checks tool/server readiness. + +CI runs offline tests and CLI/type/lint/syntax checks. It does not allocate sandboxes or call a model. This is not a GitHub issue-to-PR Action. + +## Acceptance and evidence + +The fixture uses browser localStorage, not a backend. Both variants receive the same plain issue and browser-first instructions; the Domscribe variant also receives instructions to inspect its annotation and runtime. That additional work is part of the condition and prevents treating a one-pair tool-count comparison as an efficiency benchmark. + +The controller must first observe the intended corruption: quantities become `3 / 3 / 1`. The agent may change only `src/App.jsx`. Its patch must include staged and unstaged changes against the immutable seed commit; commits, extra files, symlink substitutions and empty patches fail qualification. The exported patch is applied to a fresh worktree with a fresh frozen npm install. The independent browser verifier then checks: + +- Saving and reloading changes only the selected line (`3 / 2 / 1`). +- Reversing and filtering the list preserve the intended line identity. +- Cancel leaves the original cart intact. +- Production UI works without Domscribe instrumentation. +- No application console, page or network errors occur. + +The Domscribe condition also needs a successful runtime query that resolves the selected `line-a`, not just a completed tool call or an ambiguous source lookup. An agent's self-report or exit zero alone cannot pass. + +The printed evidence directory contains `pilot-result.json`, `repair-report.json`, `repair.patch`, raw agent JSONL, before/after screenshots, the enriched annotation and build/server logs. Reports identify local versus Docker execution, source/seed/harness/artifact digests, actual tool versions, completed MCP calls and available measured usage. Required cloud transfers retain fixed names and existing size limits. Cancellation stops owned command groups and servers. The grader is kept outside the agent worktree; it is not a security boundary against an adversarial process with access to the whole VM. + +Earlier local development trials produced correct source-only fixes in both conditions. Independent review then found and fixed gaps in staged-patch capture, production-build cancellation and artifact provenance. Those earlier trials are diagnostic evidence. Final qualification uses the fresh source and exported-patch path above. See this thread's dated evidence for final results; do not describe local results as cloud verification or claim market demand, speed or cost advantages from one seeded task. + +## Local qualification, October 25, 2026 + +Both final local trials passed after the review fixes. Each independently built all 15 package targets from clean pinned source, verified the same artifact digest, and applied the exported patch to a separate fresh app worktree before grading. Both returned the identical minimal SKU-to-line-ID fix and passed every business assertion with zero application errors. The Domscribe trial resolved the selected line's props and source, including recovering from an ambiguous source-only query with an exact instance. + +| Condition | Independent result | Completed MCP calls | Exact runtime evidence | +| ------------------- | ------------------ | ------------------: | -------------------------- | +| Full browser/source | Pass | 5 | Browser/source workflow | +| Same plus Domscribe | Pass | 15 | Resolved selected `line-a` | + +These counts are MCP calls, not all agent actions, and the enriched prompt requires extra Domscribe work. Strategies differed and the exact model ID was not reported. There is one final trial per condition. The result establishes local repair and evidence feasibility; it does not show a navigation, speed, cost or adoption advantage. Cloud qualification remains pending the provider binding. Actual versions: Node 24.22.1, Codex CLI 0.164.0, Playwright MCP 0.0.83, Chromium 156.0.8078.4, React 18.3.1 and Vite 7.3.7. diff --git a/docs/docker-cloud-replay.md b/docs/docker-cloud-replay.md index 04d8cfd..b4dd5ea 100644 --- a/docs/docker-cloud-replay.md +++ b/docs/docker-cloud-replay.md @@ -60,4 +60,4 @@ bash -n scripts/cloud/bootstrap.sh The SDK is a private development dependency, pinned at 0.53.0. Its `undici` dependency is narrowly overridden to 6.29.1 to include the upstream security patch; it does not enter Domscribe's public package exports. -The next milestone is an agent attempting a realistic, deliberately broken UI task in this environment, with evidence that its fix changes the expected behavior. GitHub issue ingestion, repository write credentials and opening fix PRs are separate work after that experiment. +The next experiment is the [agent repair pilot](./cloud-agent-repair-pilot.md), with independent grading of the exported fix. Its local and cloud results are reported separately. GitHub issue ingestion, repository write credentials and opening fix PRs remain separate work. diff --git a/docs/superpowers/plans/2026-10-25-cloud-agent-repair.md b/docs/superpowers/plans/2026-10-25-cloud-agent-repair.md new file mode 100644 index 0000000..78b00f1 --- /dev/null +++ b/docs/superpowers/plans/2026-10-25-cloud-agent-repair.md @@ -0,0 +1,47 @@ +# Cloud Agent Repair Implementation Plan + +> **For agentic workers:** Use superpowers:executing-plans inline. Obtain one fresh independent whole-branch review. Steps use checkboxes. + +**Goal:** Run and independently grade a seeded UI repair in Docker Cloud using full browser tools, with and without Domscribe. + +**Architecture:** A small controlled cart fixture and held-out Playwright verifier run in isolated cloud Git worktrees. The existing SDK lifecycle runner gains narrowly tested kit/secret-reference/extra-artifact options. A pilot CLI builds the fixed workload, launches one condition at a time, invokes the stock coding agent, downloads evidence and checks the independent result. + +**Tech Stack:** Node24, pnpm9, @docker/sandboxes0.53.0, stock Codex kit, React18/Vite7 source artifacts, Playwright MCP0.0.83. + +**Spec:** docs/superpowers/specs/2026-10-25-cloud-agent-repair-design.md + +## Global constraints + +- Preserve primary checkouts. Reuse isolated codex/cloud-agent-repair from main2306385. +- Existing OpenAI cloud secret resource reference only; no credential values or host auth/config. +- One cloud sandbox at a time, bounded lifecycle and same-UID deletion; no GitHub issue-to-PR wrapper. +- Full browser/source baseline; only Domscribe availability differs. Record actual model/version/usage, do not claim efficiency from one trial. +- Real seeded bug must fail independently before agent and pass held-out browser assertions afterward. Source-only nonempty patches, no verifier/config/dependency edits. + +## Review focus + +- Missing provider/tool readiness must fail rather than launch repeated agents. +- Agent exit0 or self-report cannot override a failed independent assertion. +- Evidence from another source/condition or a modified verifier cannot qualify. +- Secret references must never become token values, command-shell injection or arbitrary source URLs. +- Cleanup must survive failed agents, evidence transfers and cancellation; ordinary replay defaults remain identical. + +## Task1: Controlled fixture and independent grader + +- [x] Write failing regression tests for grading invalid before/after state, empty/out-of-scope patches and condition/provenance mismatches. +- [x] Implement a stable-ID cart fixture, common issue prompt and independent Playwright verification outside the agent worktree. +- [x] Prove the intended defect fails, and a temporary control correction passes, without giving the control patch to agent sessions. Preserve RED/GREEN evidence. + +## Task2: Native agent execution and delivery + +- [x] Add RED lifecycle tests for codex kit + nonempty secret reference + bounded safe extra artifacts; implement smallest extension and rerun default17tests. +- [x] Implement pilot bootstrap/CLI with exact public-source build, full browser MCP, enriched-only Domscribe, source-only patch evidence and held-out grader. +- [ ] Run native provider readiness probe, then one trial per condition sequentially. Retain failed trials and same-UID cleanup outcomes. +- [ ] Run affected configured checks, independent review, resolve material findings, document actual scope/results, commit/push focused PR and inspect CI. +- [ ] Update continuity ledger and automation with merged PR79, pilot results/limits and next action. No inferred admin bypass for a new PR. + +## Review and qualification ledger + +Independent review found staged-patch omission (P1), missing production-build cancellation (P2), and unverified local artifact provenance (P2). Regression tests reproduced the gaps. Capture now uses the immutable app seed, includes staged/unstaged edits, rejects agent commits, and applies the exported artifact to a fresh verification worktree/install. Local source builds use a separate exact-commit worktree and validated artifact digest; source cleanup is exercised on failure. Production builds receive cancellation signals and subsequent launches check cancellation. + +Final local baseline and Domscribe conditions both passed the strengthened path, returning identical minimal patches and the same artifact/harness digests. Both had zero application errors and working uninstrumented production UI. Counts were 5 versus 15 completed MCP calls; no efficiency claim. Cloud provider attachment returned explicit409failedPrecondition; kit without binding reaches the model API but returns401. All accepted diagnostic VMs were deleted. Native cloud trial remains blocked pending account setup; no release/deploy or GitHub issue wrapper. diff --git a/docs/superpowers/specs/2026-10-25-cloud-agent-repair-design.md b/docs/superpowers/specs/2026-10-25-cloud-agent-repair-design.md new file mode 100644 index 0000000..01131ec --- /dev/null +++ b/docs/superpowers/specs/2026-10-25-cloud-agent-repair-design.md @@ -0,0 +1,13 @@ +# Docker Cloud agent repair pilot + +The exact-instance/replay foundation and SDK smoke runner are merged. The next question is whether an agent using those tools can reproduce and repair a UI bug inside Docker Cloud and return a patch whose behavior we can independently verify. The user said to continue and approved the next experiment. This is a development experiment, not a GitHub Action or a claim of developer demand. + +Use two fresh in-sandbox Codex sessions with the same seeded React cart task. The cart has two lines with the same SKU and different variants; editing one quantity erroneously updates both. Give both sessions the same plain issue and full Playwright browser tools plus source access. Enable Domscribe only in the enriched condition, with MCP source/runtime/annotation tools. Keep browser, source, dependencies, prompt, model defaults, time limits and independent acceptance criteria equivalent; record differences and actual versions. One trial per condition establishes feasibility only, never statistical speed/cost superiority. + +The task is a realistic seeded business defect, not an actual reported customer issue. Acceptance requires only the selected cart line to change across sorting/filtering and reload, cancellation to leave quantities unchanged, and the production build to remain functional without instrumentation. Browser localStorage is the fixture's persistence contract; there is no external backend. Before the agent, independently reproduce the exact wrong-row failure. Afterward verify the patch with held-out browser assertions; retain before/after state, screenshots, source hash, patch, agent events and usage when available. Do not treat exit zero, agent self-report or successful replay as acceptance. An empty patch, altered verifier/config/dependencies, missing reproduction, bad business state or unconfirmed cloud deletion fails qualification. + +Reuse the isolated Domscribe worktree on codex/cloud-agent-repair from fresh main23063854647954f4001d05a21a928bdc12bdc6fc. Build local artifacts from a pinned public commit. Use the official SDK's codex kit and an existing OpenAI cloud secret reference; never read or create credential values, upload host auth/config, expose browser ports publicly or supply GitHub write access. Docker OAuth remains SDK-managed. Each sandbox has the smoke runner's bounded deadlines, medium compute, automatic expiration and explicit same-UID deletion. No more than one owned sandbox at a time. Missing provider authentication is a specific blocker, not permission to copy host credentials. + +Implementation calls left to the builder: fixture layout, file organization, static port allocation within a fresh VM, orchestration of browser/server readiness and artifact names. Reuse existing lifecycle safeguards and add only the launch/evidence options required for this controlled workload, with regression tests; default replay behavior must remain unchanged. Pin Playwright MCP to the observed registry version and use full browser capabilities, not a custom constrained navigation harness. Keep the grader outside the agent worktree and inject/run final assertions after agent execution. This reduces accidental test tailoring; it is not a security boundary against an adversarial root process. + +No automatic GitHub issue ingestion, fix PR creation, publication, production deployment or broader agent platform. Report unsuccessful trials and uncertainty honestly. The first deliverable is a repeatable pilot and actual evidence, with independent branch review and CI before PR readiness. diff --git a/scripts/cloud/repair-agent.test.ts b/scripts/cloud/repair-agent.test.ts new file mode 100644 index 0000000..fd3a9d1 --- /dev/null +++ b/scripts/cloud/repair-agent.test.ts @@ -0,0 +1,73 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { parseAgentEvents } from './repair-agent.js'; +test('counts successful full browser and Domscribe tool calls, retaining measured usage only', () => { + const log = [ + { + type: 'item.completed', + item: { + type: 'mcp_tool_call', + server: 'browser', + tool: 'browser_navigate', + status: 'completed', + result: { isError: false }, + }, + }, + { + type: 'item.completed', + item: { + type: 'mcp_tool_call', + server: 'domscribe', + tool: 'domscribe.targets.query', + status: 'completed', + result: {}, + }, + }, + { + type: 'turn.completed', + usage: { input_tokens: 100, output_tokens: 20, cached_input_tokens: 50 }, + }, + ] + .map((x) => JSON.stringify(x)) + .join('\n'); + const r = parseAgentEvents(log); + assert.deepEqual(r.tools, [ + 'browser.browser_navigate', + 'domscribe.domscribe.targets.query', + ]); + assert.equal(r.usage.length, 1); +}); +test('failed tool calls and a claimed browser action in final text are not browser-use evidence', () => { + const log = [ + { + type: 'item.completed', + item: { + type: 'mcp_tool_call', + server: 'browser', + tool: 'browser_navigate', + status: 'failed', + }, + }, + { + type: 'item.completed', + item: { type: 'agent_message', text: 'I used browser.browser_navigate' }, + }, + ] + .map((x) => JSON.stringify(x)) + .join('\n'); + assert.deepEqual(parseAgentEvents(log).tools, []); +}); + +test('Codex-normalized MCP errors are not successful tool evidence', () => { + const event = { + type: 'item.completed', + item: { + type: 'mcp_tool_call', + server: 'browser', + tool: 'browser_navigate', + status: 'completed', + result: { is_error: true }, + }, + }; + assert.deepEqual(parseAgentEvents(JSON.stringify(event)).tools, []); +}); diff --git a/scripts/cloud/repair-agent.ts b/scripts/cloud/repair-agent.ts new file mode 100644 index 0000000..27a79ac --- /dev/null +++ b/scripts/cloud/repair-agent.ts @@ -0,0 +1,51 @@ +/** Extract completed MCP calls and recorded token usage from Codex JSONL. */ +export function parseAgentEvents(text: string): { + tools: string[]; + usage: unknown[]; + runtimeTargets: string[]; +} { + const tools: string[] = [], + usage: unknown[] = []; + const runtimeTargets = new Set(); + for (const line of text.split('\n')) { + let event; + try { + event = JSON.parse(line); + } catch { + continue; + } + if (event?.type === 'item.completed') { + const i = event.item; + if ( + i?.type === 'mcp_tool_call' && + i.status === 'completed' && + i.result?.isError !== true && + i.result?.is_error !== true && + !i.error && + typeof i.server === 'string' && + typeof i.tool === 'string' + ) { + tools.push(i.server + '.' + i.tool); + const data = + i.result?.structured_content ?? i.result?.structuredContent; + if ( + i.server === 'domscribe' && + /targets[._]query|query[._]bySource/.test(i.tool) && + data?.browserConnected === true && + data.total === 1 && + (data.status === 'resolved' || data.selectionStatus === 'resolved') + ) { + const id = (data.context ?? data.runtime)?.componentProps?.line?.id; + if (typeof id === 'string') runtimeTargets.add(id); + } + } + } + if ( + event?.type === 'turn.completed' && + typeof event.usage === 'object' && + event.usage !== null + ) + usage.push(event.usage); + } + return { tools, usage, runtimeTargets: [...runtimeTargets] }; +} diff --git a/scripts/cloud/repair-controller.mjs b/scripts/cloud/repair-controller.mjs new file mode 100644 index 0000000..b2ccde4 --- /dev/null +++ b/scripts/cloud/repair-controller.mjs @@ -0,0 +1,506 @@ +// Controlled pilot coordinator. Agent worktrees receive app/issue files only. +import fs from 'node:fs'; +import path from 'node:path'; +import { createHash } from 'node:crypto'; +import { createRequire } from 'node:module'; +import { pathToFileURL } from 'node:url'; +import { execFileSync } from 'node:child_process'; +import net from 'node:net'; +import { writeCartFixture, SOURCE_HASH, ISSUE } from './repair-fixture.mjs'; +import { verifyBefore, verifyAfter } from './repair-verify.mjs'; +import { gradeRepair } from './repair-grade.ts'; +import { parseAgentEvents } from './repair-agent.ts'; +import { collectRepairPatch } from './repair-patch.ts'; +import { readArtifactProvenance } from './repair-source.ts'; + +async function freePort() { + const server = net.createServer(); + await new Promise((r) => server.listen(0, '127.0.0.1', r)); + const port = server.address().port; + await new Promise((r) => server.close(r)); + return port; +} +async function ready(url, signal) { + const end = Date.now() + 30000; + while (Date.now() < end) { + signal?.throwIfAborted(); + try { + if ((await fetch(url, { signal: AbortSignal.timeout(1000) })).ok) return; + } catch { + /* Best-effort cleanup/readiness; the owned process may already be gone. */ + } + await new Promise((r) => setTimeout(r, 100)); + } + throw Error('Cart server readiness timed out'); +} +export async function runRepairExperiment({ + sourceRoot, + workRoot, + reportDir, + condition, + sourceCommit, + environment = 'docker-cloud', + browserTools, + signal, +}) { + if ( + !['baseline', 'domscribe'].includes(condition) || + !['local-control', 'docker-cloud'].includes(environment) || + !/^[a-f0-9]{40}$/.test(sourceCommit) + ) + throw Error('Invalid controlled experiment input'); + const modeArgs = + environment === 'local-control' + ? [ + '--approve-for-me', + '--ignore-user-config', + '--add-dir', + reportDir, + '-c', + 'sandbox_workspace_write.network_access=true', + ] + : ['--dangerously-bypass-approvals-and-sandbox']; + execFileSync('codex', ['exec', ...modeArgs, '--help'], { + stdio: 'pipe', + timeout: 10000, + }); + if (fs.existsSync(workRoot)) throw Error('Repair work root already exists'); + fs.mkdirSync(workRoot, { recursive: true, mode: 0o700 }); + fs.mkdirSync(reportDir, { recursive: true, mode: 0o700 }); + const artifactProvenance = readArtifactProvenance(sourceRoot, sourceCommit); + const { startOwnedProcess, stopOwnedProcess, runOwnedBuild } = await import( + environment === 'docker-cloud' + ? new URL('./repair-process.mjs', import.meta.url).href + : new URL('../fixtures/replay-process.mjs', import.meta.url).href + ); + const harnessHash = createHash('sha256'); + for (const name of [ + 'repair-controller.mjs', + 'repair-fixture.mjs', + 'repair-verify.mjs', + 'repair-grade.ts', + 'repair-agent.ts', + 'repair-patch.ts', + 'repair-source.ts', + ]) + harnessHash.update(fs.readFileSync(new URL('./' + name, import.meta.url))); + harnessHash.update( + fs.readFileSync( + environment === 'docker-cloud' + ? new URL('./repair-process.mjs', import.meta.url) + : new URL('../fixtures/replay-process.mjs', import.meta.url), + ), + ); + const fixtureRequire = createRequire( + path.join(sourceRoot, 'packages/domscribe-test-fixtures/package.json'), + ); + const { chromium } = fixtureRequire('playwright'); + const packDir = path.join(workRoot, 'packages'); + fs.mkdirSync(packDir); + const { replayProject } = await import( + pathToFileURL(path.join(sourceRoot, 'scripts/fixtures/replay-project.mjs')) + .href + ); + const project = replayProject('react', undefined, '7.3.7'); + const dependencies = project.dependencies; + for (const name of [...project.artifacts, 'mcp']) { + const artifactManifest = JSON.parse( + fs.readFileSync( + path.join( + sourceRoot, + 'dist/packages/domscribe-' + name, + 'package.json', + ), + 'utf8', + ), + ); + if ( + JSON.stringify(artifactManifest.dependencies ?? {}).includes('workspace:') + ) + throw Error('Build and sync Domscribe artifacts before pilot packaging'); + const pack = JSON.parse( + execFileSync( + 'npm', + [ + 'pack', + path.join(sourceRoot, 'dist/packages/domscribe-' + name), + '--ignore-scripts', + '--pack-destination', + packDir, + '--json', + ], + { encoding: 'utf8', timeout: 30000 }, + ), + ); + dependencies['@domscribe/' + name] = + 'file:' + path.join(packDir, pack[0].filename); + } + const base = path.join(workRoot, 'base'), + app = path.join(workRoot, 'app'); + writeCartFixture(base, { dependencies }); + execFileSync( + 'npm', + ['install', '--ignore-scripts', '--no-audit', '--no-fund'], + { cwd: base, stdio: 'pipe', timeout: 180000 }, + ); + for (const args of [ + ['init'], + ['add', '.'], + [ + '-c', + 'user.name=Pilot', + '-c', + 'user.email=pilot@example.invalid', + 'commit', + '-m', + 'Seed isolated cart task', + ], + ['worktree', 'add', '--detach', app, 'HEAD'], + ]) + execFileSync('git', args, { cwd: base, stdio: 'pipe' }); + execFileSync('npm', ['ci', '--ignore-scripts', '--no-audit', '--no-fund'], { + cwd: app, + stdio: 'pipe', + timeout: 180000, + }); + fs.rmSync(path.join(base, 'node_modules'), { recursive: true, force: true }); + const sourceHash = createHash('sha256') + .update(fs.readFileSync(path.join(app, 'src/App.jsx'))) + .digest('hex'); + if (sourceHash !== SOURCE_HASH) + throw Error('Seed source differs from protocol'); + let tools = browserTools; + if (!tools) { + tools = path.join(workRoot, 'tools'); + execFileSync( + 'npm', + [ + 'install', + '--prefix', + tools, + '--ignore-scripts', + '--no-audit', + '--no-fund', + '@playwright/mcp@0.0.83', + ], + { stdio: 'pipe', timeout: 180000 }, + ); + } + const appSeedCommit = execFileSync('git', ['rev-parse', 'HEAD'], { + cwd: app, + encoding: 'utf8', + }).trim(); + const req = createRequire(path.join(app, 'package.json')); + const report = { + schemaVersion: 1, + at: new Date().toISOString(), + condition, + sourceCommit, + initialSourceHash: sourceHash, + environment, + appSeedCommit, + artifactProvenance, + harnessDigest: harnessHash.digest('hex'), + status: 'running', + versions: { + node: process.version, + playwrightMcp: JSON.parse( + fs.readFileSync( + path.join(tools, 'node_modules/@playwright/mcp/package.json'), + 'utf8', + ), + ).version, + }, + }; + const save = () => + fs.writeFileSync( + path.join(reportDir, 'repair-report.json'), + JSON.stringify(report, null, 2), + ); + save(); + const serverEnv = { + ...process.env, + VITE_DOMSCRIBE: condition === 'domscribe' ? '1' : '0', + NODE_OPTIONS: '--max-old-space-size=4096', + }; + const logs = []; + let dev, prod, browser; + const verificationApp = path.join(workRoot, 'verification-app'); + const cancel = () => { + void stopOwnedProcess(dev); + void stopOwnedProcess(prod); + void browser?.close().catch(() => { + /* Cancellation may race an already-closed browser. */ + }); + }; + signal?.addEventListener('abort', cancel, { once: true }); + function start(args, name, env = serverEnv, cwd = app) { + const fd = fs.openSync(path.join(reportDir, name), 'w'); + logs.push(fd); + return startOwnedProcess( + process.execPath, + [path.join(cwd, 'node_modules/vite/bin/vite.js'), ...args], + { + cwd, + env, + stdio: ['ignore', fd, fd], + }, + ); + } + try { + signal?.throwIfAborted(); + const dp = await freePort(), + pp = await freePort(); + const url = 'http://127.0.0.1:' + dp; + dev = start(['--port', String(dp)], 'repair-server.log'); + await ready(url, signal); + browser = await chromium.launch({ + headless: true, + args: ['--disable-dev-shm-usage'], + }); + signal?.throwIfAborted(); + report.versions.chromium = browser.version(); + report.before = await verifyBefore(browser, url, reportDir); + save(); + if (condition === 'domscribe') { + const context = await browser.newContext(); + const page = await context.newPage(); + try { + await page.goto(url); + await page.waitForFunction(() => window.__runtime?.isReady()); + const target = await page + .getByRole('article', { name: 'Midnight travel mug', exact: true }) + .getByRole('button', { name: 'Edit quantity', exact: true }) + .evaluate((el) => window.__runtime.describeTarget(el)); + const port = await page.evaluate(() => window.__DOMSCRIBE_RELAY_PORT__); + const { RelayHttpClient } = await import( + pathToFileURL(req.resolve('@domscribe/relay/client')).href + ); + const relay = new RelayHttpClient('127.0.0.1', port); + const exact = await relay.queryTargets({ + instanceId: target.instanceId, + }); + if ( + exact.status !== 'resolved' || + exact.context.componentProps.line.id !== 'line-a' + ) + throw Error('Exact cart runtime does not match selected line'); + const annotation = await relay.createAnnotation({ + mode: 'element-click', + interaction: { + type: 'element-annotation', + selectedElement: { + tagName: 'button', + selector: target.selector, + dataDs: target.entryId, + target, + }, + }, + context: { + pageUrl: url, + pageTitle: await page.title(), + viewport: { width: 1280, height: 820 }, + userAgent: await page.evaluate(() => navigator.userAgent), + userMessage: ISSUE, + runtimeContext: exact.context, + }, + }); + fs.writeFileSync( + path.join(reportDir, 'repair-annotation.json'), + JSON.stringify(annotation, null, 2), + ); + } finally { + await context.close(); + } + } else + fs.writeFileSync( + path.join(reportDir, 'repair-annotation.json'), + JSON.stringify({ condition, annotation: null }), + ); + await browser.close(); + browser = undefined; + const browserArgs = [ + path.join(tools, 'node_modules/@playwright/mcp/cli.js'), + '--headless', + '--isolated', + '--no-sandbox', + '--executable-path', + chromium.executablePath(), + '--output-dir', + path.join(reportDir, 'browser'), + ]; + const agentArgs = [ + 'exec', + '--json', + '-C', + app, + '-c', + `mcp_servers.browser.command=${JSON.stringify(process.execPath)}`, + '-c', + `mcp_servers.browser.args=${JSON.stringify(browserArgs)}`, + ]; + if (environment === 'docker-cloud') + agentArgs.push('--dangerously-bypass-approvals-and-sandbox'); + else + agentArgs.push( + '--approve-for-me', + '--ignore-user-config', + '--add-dir', + reportDir, + '-c', + 'sandbox_workspace_write.network_access=true', + ); + if (condition === 'domscribe') { + const mcp = path.join( + app, + 'node_modules/@domscribe/mcp/bin/domscribe-mcp.js', + ); + const args = [ + '--input-type=module', + '-e', + `process.chdir(${JSON.stringify(app)});await import(${JSON.stringify(pathToFileURL(mcp).href)});`, + ]; + agentArgs.push( + '-c', + `mcp_servers.domscribe.command=${JSON.stringify(process.execPath)}`, + '-c', + `mcp_servers.domscribe.args=${JSON.stringify(args)}`, + ); + } + const prompt = + ISSUE + + `\n\nApp URL: ${url}. Use the browser MCP server to reproduce before editing and verify after. Browser localStorage may be reset between reproductions. Do not read credential files or environment variables. You are already in an isolated Git worktree. ` + + (condition === 'domscribe' + ? 'Domscribe MCP is also available. Navigate in the browser first, then inspect the queued annotation and use domscribe.targets.query or domscribe.query.bySource to connect the selected cart line to live props/source before fixing. ' + : '') + + 'The independent grader is not part of your worktree; do not search for it.'; + agentArgs.push(prompt); + report.versions.codex = execFileSync('codex', ['--version'], { + encoding: 'utf8', + timeout: 10000, + }).trim(); + let agentExit = 0; + try { + await runOwnedBuild('codex', agentArgs, { + cwd: app, + env: serverEnv, + logPath: path.join(reportDir, 'repair-agent.jsonl'), + timeoutMs: 8 * 60 * 1000, + signal, + label: 'Agent', + }); + } catch (error) { + agentExit = 1; + report.agentError = String(error); + } + report.agent = { + exitCode: agentExit, + incomplete: agentExit !== 0, + ...parseAgentEvents( + fs.readFileSync(path.join(reportDir, 'repair-agent.jsonl'), 'utf8'), + ), + }; + save(); + signal?.throwIfAborted(); + if (agentExit !== 0) + throw Error('Agent invocation failed; inspect repair-agent.jsonl'); + Object.assign(report, collectRepairPatch(app, appSeedCommit)); + fs.writeFileSync(path.join(reportDir, 'repair.patch'), report.patch); + await stopOwnedProcess(dev); + dev = undefined; + execFileSync( + 'git', + ['worktree', 'add', '--detach', verificationApp, appSeedCommit], + { cwd: base, stdio: 'pipe' }, + ); + execFileSync('git', ['apply', '-'], { + cwd: verificationApp, + input: report.patch, + stdio: ['pipe', 'pipe', 'pipe'], + }); + if ( + !fs + .readFileSync(path.join(app, 'src/App.jsx')) + .equals(fs.readFileSync(path.join(verificationApp, 'src/App.jsx'))) + ) + throw Error('Exported patch does not reconstruct the verified source'); + execFileSync('npm', ['ci', '--ignore-scripts', '--no-audit', '--no-fund'], { + cwd: verificationApp, + stdio: 'pipe', + timeout: 180000, + }); + report.verification = { + kind: 'exported-patch-fresh-worktree', + seedCommit: appSeedCommit, + patchSha256: createHash('sha256').update(report.patch).digest('hex'), + }; + const verificationCli = path.join( + verificationApp, + 'node_modules/vite/bin/vite.js', + ); + signal?.throwIfAborted(); + await runOwnedBuild(process.execPath, [verificationCli, 'build'], { + cwd: verificationApp, + env: { ...serverEnv, NODE_ENV: 'production' }, + logPath: path.join(reportDir, 'repair-production-build.log'), + timeoutMs: 120000, + signal, + }); + signal?.throwIfAborted(); + dev = start( + ['--port', String(dp)], + 'repair-verification-server.log', + serverEnv, + verificationApp, + ); + await ready(url, signal); + signal?.throwIfAborted(); + prod = start( + ['preview', '--host', '127.0.0.1', '--port', String(pp)], + 'repair-production-server.log', + { ...serverEnv, NODE_ENV: 'production' }, + verificationApp, + ); + await ready('http://127.0.0.1:' + pp, signal); + signal?.throwIfAborted(); + browser = await chromium.launch({ + headless: true, + args: ['--disable-dev-shm-usage'], + }); + signal?.throwIfAborted(); + report.after = await verifyAfter( + browser, + url, + 'http://127.0.0.1:' + pp, + reportDir, + ); + gradeRepair(report, condition, sourceCommit, SOURCE_HASH); + report.status = 'passed'; + save(); + return report; + } catch (error) { + report.status = 'failed'; + report.error = String(error); + save(); + throw error; + } finally { + signal?.removeEventListener('abort', cancel); + await browser?.close(); + await stopOwnedProcess(dev); + await stopOwnedProcess(prod); + for (const fd of logs) fs.closeSync(fd); + for (const workspace of [app, verificationApp]) { + const lock = path.join(workspace, '.domscribe/relay.lock'); + if (fs.existsSync(lock)) { + const r = JSON.parse(fs.readFileSync(lock, 'utf8')); + if (r.workspaceRoot === workspace) { + try { + process.kill(r.pid, 'SIGTERM'); + } catch { + /* Best-effort cleanup/readiness; the owned process may already be gone. */ + } + } + } + } + } +} diff --git a/scripts/cloud/repair-fixture.mjs b/scripts/cloud/repair-fixture.mjs new file mode 100644 index 0000000..21ee86a --- /dev/null +++ b/scripts/cloud/repair-fixture.mjs @@ -0,0 +1,90 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { createHash } from 'node:crypto'; + +export const CART_SOURCE = `import React, { useState } from 'react'; +import { createRoot } from 'react-dom/client'; +import './style.css'; +if (import.meta.env.DEV && import.meta.env.VITE_DOMSCRIBE === '1') import('@domscribe/runtime').then(({ RuntimeManager }) => window.__runtime = RuntimeManager.getInstance()); +const initial = [ + { id: 'line-a', sku: 'MUG-18', label: 'Midnight travel mug', variant: 'Midnight', quantity: 1, price: 18 }, + { id: 'line-b', sku: 'MUG-18', label: 'Ocean travel mug', variant: 'Ocean', quantity: 2, price: 18 }, + { id: 'line-c', sku: 'NOTE-12', label: 'Graphite notebook', variant: 'Graphite', quantity: 1, price: 12 }, +]; +function readCart() { try { return JSON.parse(localStorage.getItem('cart-lines')) || initial; } catch { return initial; } } +function CartLine({ line, onEdit }) { + return
+
{line.sku}

{line.label}

Variant: {line.variant}

+
Quantity: {line.quantity}

$ {line.price * line.quantity}

+
; +} +function QuantityEditor({ line, onSave, onCancel }) { + const [quantity, setQuantity] = useState(line.quantity); + return
+

{line.label}

+
+
; +} +function App() { + const [lines, setLines] = useState(readCart), [editing, setEditing] = useState(null); + const [reverse, setReverse] = useState(false), [filter, setFilter] = useState(''); + function save(selected, quantity) { + const next = lines.map(line => line.sku === selected.sku ? { ...line, quantity } : line); + localStorage.setItem('cart-lines', JSON.stringify(next)); setLines(next); setEditing(null); + } + const visible = [...lines].filter(line => line.label.toLowerCase().includes(filter.toLowerCase())).sort((a, b) => reverse ? b.label.localeCompare(a.label) : a.label.localeCompare(b.label)); + return

FIELD SUPPLY / CHECKOUT

Your cart

Review quantities before placing your order.

+ + {visible.map(line => )} + {editing && setEditing(null)} />} +
Subtotal: $ {lines.reduce((sum, line) => sum + line.price * line.quantity, 0)}

Quantities are saved in this browser.

+
; +} +createRoot(document.getElementById('root')).render(); +`; +export const SOURCE_HASH = createHash('sha256') + .update(CART_SOURCE) + .digest('hex'); +export const ISSUE = `Editing one cart line changes another line with the same SKU. + +Reproduce in the browser: open Midnight travel mug, change Quantity to 3 and save. Ocean travel mug also becomes 3 although it was 2. Reload preserves the wrong result. + +Expected: save changes only the selected line. Other quantities stay unchanged. This must still work after reversing or filtering the list, across reload, and when two variants share an SKU. Cancel must leave the cart unchanged. + +Reproduce using the provided full browser tools before editing. Inspect source and implement the smallest justified fix; verify it in the browser afterward. Only src/App.jsx may change. Do not alter configuration, dependencies, storage keys, test/grader files, or remove functionality. Do not read or modify files outside this isolated app worktree. Do not commit, push, publish, or deploy. Return a brief root-cause/fix/test summary with any limitations. This is a seeded development task, not an actual customer report.`; + +export function writeCartFixture(dir, manifest) { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'package.json'), + JSON.stringify( + { + ...manifest, + name: 'cart-repair-pilot', + private: true, + type: 'module', + scripts: { dev: 'vite', build: 'vite build', preview: 'vite preview' }, + }, + null, + 2, + ), + ); + fs.writeFileSync( + path.join(dir, '.gitignore'), + 'node_modules\ndist\n.domscribe\n', + ); + fs.writeFileSync( + path.join(dir, 'index.html'), + 'Field Supply cart
', + ); + fs.writeFileSync(path.join(dir, 'src/App.jsx'), CART_SOURCE); + fs.writeFileSync( + path.join(dir, 'src/style.css'), + '*{box-sizing:border-box}body{margin:0;background:#f4f2eb;color:#162c28;font:16px system-ui}main{max-width:880px;margin:44px auto;padding:0 24px}h1{font-size:44px;margin:8px 0}h2{font-size:21px;margin:8px 0}p{color:#536760}.category{font-size:12px;letter-spacing:.12em}nav,article,.actions{display:flex;justify-content:space-between;gap:24px}nav{align-items:end;margin:32px 0}article,section{background:white;border:1px solid #d1d8d0;padding:24px;margin:12px 0}article>div:last-child{text-align:right}label{display:grid;gap:8px}input,button{font:inherit;padding:9px 12px;border:1px solid #81978e}button{background:#164d40;color:white;cursor:pointer}button:disabled{opacity:.5}footer{padding:24px 0}.actions{margin-top:20px;justify-content:end}', + ); + fs.writeFileSync( + path.join(dir, 'vite.config.mjs'), + `import {defineConfig} from 'vite';import react from '@vitejs/plugin-react';import {domscribe} from '@domscribe/react/vite';export default defineConfig({plugins:[react(),...(process.env.VITE_DOMSCRIBE==='1'?[domscribe({overlay:true,relay:{port:0}})]:[])],server:{host:'127.0.0.1',strictPort:true}});`, + ); + fs.writeFileSync(path.join(dir, 'ISSUE.md'), ISSUE); +} diff --git a/scripts/cloud/repair-grade.test.ts b/scripts/cloud/repair-grade.test.ts new file mode 100644 index 0000000..acbb645 --- /dev/null +++ b/scripts/cloud/repair-grade.test.ts @@ -0,0 +1,133 @@ +import test from 'node:test'; +import { createHash } from 'node:crypto'; +import assert from 'node:assert/strict'; +import { gradeRepair } from './repair-grade.js'; +const commit = 'a'.repeat(40), + hash = 'b'.repeat(64); +const initial = { 'line-a': 1, 'line-b': 2, 'line-c': 1 }; +const edited = { 'line-a': 3, 'line-b': 2, 'line-c': 1 }; +function valid() { + const patch = + 'diff --git a/src/App.jsx b/src/App.jsx\n--- a/src/App.jsx\n+++ b/src/App.jsx\n@@ -1 +1 @@\n-bug\n+fix\n'; + return { + appSeedCommit: 'e'.repeat(40), + versions: { node: 'v24.22.1' }, + artifactProvenance: { + commit, + node: 'v24.22.1', + cleanSource: true, + distDigest: 'f'.repeat(64), + }, + verification: { + kind: 'exported-patch-fresh-worktree', + seedCommit: 'e'.repeat(40), + patchSha256: createHash('sha256').update(patch).digest('hex'), + }, + schemaVersion: 1, + condition: 'baseline', + sourceCommit: commit, + initialSourceHash: hash, + changedFiles: ['src/App.jsx'], + patch: + 'diff --git a/src/App.jsx b/src/App.jsx\n--- a/src/App.jsx\n+++ b/src/App.jsx\n@@ -1 +1 @@\n-bug\n+fix\n', + agent: { + exitCode: 0, + incomplete: false, + tools: ['browser.browser_navigate'], + runtimeTargets: [] as string[], + }, + before: { + pageErrors: [], + consoleErrors: [], + networkErrors: [], + observed: { 'line-a': 3, 'line-b': 3, 'line-c': 1 }, + }, + after: { + saved: edited, + reloaded: edited, + sorted: { 'line-a': 1, 'line-b': 4, 'line-c': 1 }, + filtered: { 'line-a': 1, 'line-b': 2, 'line-c': 2 }, + cancelled: initial, + production: edited, + productionInstrumented: false, + pageErrors: [], + consoleErrors: [], + networkErrors: [], + }, + }; +} +test('accepts independently verified targeted repair including persisted, sorted, filtered and production states', () => { + assert.doesNotThrow(() => gradeRepair(valid(), 'baseline', commit, hash)); +}); +test('rejects exit zero when the wrong sibling changes or reload loses the correction', () => { + for (const key of [ + 'saved', + 'reloaded', + 'sorted', + 'filtered', + 'cancelled', + 'production', + ] as const) { + const r = valid(); + r.after[key] = { 'line-a': 9, 'line-b': 9, 'line-c': 9 }; + assert.throws(() => gradeRepair(r, 'baseline', commit, hash), key); + } +}); +test('requires the intended before-defect, not a startup or dependency failure', () => { + const r = valid(); + r.before.observed = initial; + assert.throws(() => gradeRepair(r, 'baseline', commit, hash)); +}); +test('rejects unrelated, empty or untracked fix artifacts and stale provenance', () => { + const empty = valid(); + empty.patch = ''; + assert.throws(() => gradeRepair(empty, 'baseline', commit, hash)); + const outside = valid(); + outside.changedFiles.push('vite.config.mjs'); + assert.throws(() => gradeRepair(outside, 'baseline', commit, hash)); + const hidden = valid(); + hidden.patch += 'diff --git a/package.json b/package.json\n'; + assert.throws(() => gradeRepair(hidden, 'baseline', commit, hash)); + assert.throws(() => gradeRepair(valid(), 'domscribe', commit, hash)); + assert.throws(() => gradeRepair(valid(), 'baseline', 'c'.repeat(40), hash)); + assert.throws(() => gradeRepair(valid(), 'baseline', commit, 'd'.repeat(64))); +}); +test('rejects a successful agent report without real browser use, errors or production stripping', () => { + const noBrowser = valid(); + noBrowser.agent.tools = []; + assert.throws(() => gradeRepair(noBrowser, 'baseline', commit, hash)); + const incomplete = valid(); + incomplete.agent.incomplete = true; + assert.throws(() => gradeRepair(incomplete, 'baseline', commit, hash)); + const instrumented = valid(); + instrumented.after.productionInstrumented = true; + assert.throws(() => gradeRepair(instrumented, 'baseline', commit, hash)); + const errors = valid(); + (errors.after.pageErrors as unknown[]).push('crash'); + assert.throws(() => gradeRepair(errors, 'baseline', commit, hash)); +}); +test('enriched trial must actually use exact-target/source runtime tools', () => { + const r = valid(); + r.condition = 'domscribe'; + assert.throws(() => gradeRepair(r, 'domscribe', commit, hash)); + r.agent.tools.push('domscribe.domscribe.targets.query'); + r.agent.runtimeTargets.push('line-b'); + assert.throws(() => gradeRepair(r, 'domscribe', commit, hash)); + r.agent.runtimeTargets[0] = 'line-a'; + assert.doesNotThrow(() => gradeRepair(r, 'domscribe', commit, hash)); +}); + +test('a reproduced quantity error with application errors cannot qualify', () => { + const r = valid(); + (r.before.consoleErrors as unknown[]).push('crash'); + assert.throws(() => gradeRepair(r, 'baseline', commit, hash)); +}); + +test('requires verified artifacts and grading the exported patch from its immutable seed', () => { + const r = valid(); + r.verification.kind = 'live-agent-tree'; + assert.throws(() => gradeRepair(r, 'baseline', commit, hash)); + const stale = valid(); + stale.artifactProvenance.commit = 'c'.repeat(40); + assert.throws(() => gradeRepair(stale, 'baseline', commit, hash)); +}); diff --git a/scripts/cloud/repair-grade.ts b/scripts/cloud/repair-grade.ts new file mode 100644 index 0000000..73d4ba1 --- /dev/null +++ b/scripts/cloud/repair-grade.ts @@ -0,0 +1,108 @@ +import { createHash } from 'node:crypto'; +/** Grade controller-owned browser observations, never an agent's final claims. */ +export function gradeRepair( + input: unknown, + condition: string, + commit: string, + sourceHash: string, +): void { + function require(value: unknown, message: string): asserts value { + if (!value) throw new Error(message); + } + function object(value: unknown): Record { + require(typeof value === 'object' && + value !== null && + !Array.isArray(value), 'Missing repair evidence object'); + return value as Record; + } + function state(value: unknown, quantities: number[], name: string) { + const actual = object(value); + require(Object.keys(actual).sort().join(',') === + 'line-a,line-b,line-c', `Unexpected cart identity in ${name}`); + require(quantities.every( + (q, i) => actual[['line-a', 'line-b', 'line-c'][i]] === q, + ), `Incorrect cart quantities in ${name}`); + } + const r = object(input); + require(r.schemaVersion === 1 && + ['baseline', 'domscribe'].includes(condition) && + r.condition === condition, 'Wrong repair condition/schema'); + require(r.sourceCommit === commit && + r.initialSourceHash === + sourceHash, 'Repair provenance differs from requested source'); + require(Array.isArray(r.changedFiles) && + r.changedFiles.length === 1 && + r.changedFiles[0] === 'src/App.jsx', 'Only src/App.jsx may change'); + require(typeof r.patch === 'string' && + r.patch.length > 0 && + r.patch.length < 128 * 1024, 'Missing or oversized patch'); + const headers = r.patch + .split('\n') + .filter((line) => line.startsWith('diff --git ')); + require(headers.length === 1 && + headers[0] === + 'diff --git a/src/App.jsx b/src/App.jsx', 'Patch changes unexpected paths'); + require(r.patch + .split('\n') + .some( + (line) => line.startsWith('+') && !line.startsWith('+++'), + ), 'Patch has no added source'); + const provenance = object(r.artifactProvenance), + verification = object(r.verification); + require(provenance.commit === commit && + provenance.cleanSource === true && + typeof provenance.distDigest === 'string' && + /^[a-f0-9]{64}$/.test(provenance.distDigest) && + provenance.node === + object(r.versions) + .node, 'Artifacts lack verified pinned-source provenance'); + require(typeof r.appSeedCommit === 'string' && + /^[a-f0-9]{40}$/.test(r.appSeedCommit) && + verification.kind === 'exported-patch-fresh-worktree' && + verification.seedCommit === r.appSeedCommit && + verification.patchSha256 === + createHash('sha256') + .update(r.patch) + .digest( + 'hex', + ), 'Exported patch was not independently verified from its seed'); + const agent = object(r.agent); + require(agent.exitCode === 0 && + agent.incomplete === false, 'Agent did not complete successfully'); + require(Array.isArray(agent.tools) && + agent.tools.every( + (t) => typeof t === 'string', + ), 'Missing agent tool evidence'); + require(agent.tools.some((t) => + String(t).startsWith('browser.'), + ), 'Agent did not use the browser'); + const runtimeUsed = agent.tools.some((t) => + /^domscribe\..*(targets[._]query|query[._]bySource)/.test(String(t)), + ); + require(condition === 'domscribe' + ? runtimeUsed + : !agent.tools.some((t) => + String(t).startsWith('domscribe.'), + ), 'Agent tool use differs from condition'); + require(Array.isArray(agent.runtimeTargets) && + (condition === 'domscribe' + ? agent.runtimeTargets.includes('line-a') + : agent.runtimeTargets.length === + 0), 'Missing exact selected-line runtime evidence'); + const before = object(r.before); + for (const name of ['pageErrors', 'consoleErrors', 'networkErrors']) + require(Array.isArray(before[name]) && + before[name].length === 0, `Before reproduction reported ${name}`); + state(before.observed, [3, 3, 1], 'before reproduction'); + const after = object(r.after); + for (const name of ['saved', 'reloaded', 'production']) + state(after[name], [3, 2, 1], name); + state(after.sorted, [1, 4, 1], 'sorted'); + state(after.filtered, [1, 2, 2], 'filtered'); + state(after.cancelled, [1, 2, 1], 'cancelled'); + require(after.productionInstrumented === + false, 'Production instrumentation remains'); + for (const name of ['pageErrors', 'consoleErrors', 'networkErrors']) + require(Array.isArray(after[name]) && + after[name].length === 0, `Application reported ${name}`); +} diff --git a/scripts/cloud/repair-patch.test.ts b/scripts/cloud/repair-patch.test.ts new file mode 100644 index 0000000..d128421 --- /dev/null +++ b/scripts/cloud/repair-patch.test.ts @@ -0,0 +1,99 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { execFileSync } from 'node:child_process'; +import { collectRepairPatch } from './repair-patch.js'; +function fixture() { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'repair-patch-test-')); + const base = path.join(root, 'base'), + app = path.join(root, 'app'); + fs.mkdirSync(path.join(base, 'src'), { recursive: true }); + const git = (args: string[], cwd = base) => + execFileSync('git', args, { + cwd, + encoding: 'utf8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + git(['init']); + fs.writeFileSync( + path.join(base, 'src/App.jsx'), + `export const update=(all,selected,quantity)=>all.map(line=>line.sku===selected.sku?{...line,quantity}:line);\n`, + ); + git(['add', '.']); + git([ + '-c', + 'user.name=Test', + '-c', + 'user.email=test@example.invalid', + 'commit', + '-m', + 'seed', + ]); + const seed = git(['rev-parse', 'HEAD']).trim(); + git(['worktree', 'add', '--detach', app, seed]); + return { root, base, app, seed, git }; +} +test('returned artifact includes staged repair and unstaged edits and replays the fix from the seed', async () => { + const f = fixture(); + try { + const file = path.join(f.app, 'src/App.jsx'); + fs.writeFileSync( + file, + fs + .readFileSync(file, 'utf8') + .replace('line.sku===selected.sku', 'line.id===selected.id'), + ); + f.git(['add', 'src/App.jsx'], f.app); + fs.appendFileSync(file, '// explanation\n'); + const r = collectRepairPatch(f.app, f.seed); + const fresh = path.join(f.root, 'fresh'); + f.git(['worktree', 'add', '--detach', fresh, f.seed]); + execFileSync('git', ['apply', '-'], { cwd: fresh, input: r.patch }); + const module = await import( + 'data:text/javascript,' + + encodeURIComponent( + fs.readFileSync(path.join(fresh, 'src/App.jsx'), 'utf8'), + ) + ); + assert.deepEqual( + module + .update( + [ + { id: 'a', sku: 'm', quantity: 1 }, + { id: 'b', sku: 'm', quantity: 2 }, + ], + { id: 'a', sku: 'm' }, + 3, + ) + .map((x: { quantity: number }) => x.quantity), + [3, 2], + ); + assert.deepEqual(r.changedFiles, ['src/App.jsx']); + } finally { + fs.rmSync(f.root, { recursive: true, force: true }); + } +}); +test('committed or unrelated changes cannot be returned as a source-only repair', () => { + const f = fixture(); + try { + fs.appendFileSync(path.join(f.app, 'src/App.jsx'), '// changed\n'); + f.git(['add', '.'], f.app); + f.git( + [ + '-c', + 'user.name=Test', + '-c', + 'user.email=test@example.invalid', + 'commit', + '-m', + 'unexpected', + ], + f.app, + ); + assert.throws(() => collectRepairPatch(f.app, f.seed)); + } finally { + fs.rmSync(f.root, { recursive: true, force: true }); + } +}); diff --git a/scripts/cloud/repair-patch.ts b/scripts/cloud/repair-patch.ts new file mode 100644 index 0000000..e031a03 --- /dev/null +++ b/scripts/cloud/repair-patch.ts @@ -0,0 +1,31 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { execFileSync } from 'node:child_process'; +/** Capture the deliverable against the immutable seed, including staged edits. */ +export function collectRepairPatch( + app: string, + seed: string, +): { changedFiles: string[]; patch: string } { + if (!/^[a-f0-9]{40}$/.test(seed)) throw Error('Invalid app seed commit'); + const git = (args: string[]) => + execFileSync('git', args, { + cwd: app, + encoding: 'utf8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + if (git(['rev-parse', 'HEAD']).trim() !== seed) + throw Error('Agent changed the app seed commit'); + const file = fs.lstatSync(path.join(app, 'src/App.jsx')); + if (!file.isFile() || file.isSymbolicLink()) + throw Error('Repair source must remain a regular file'); + const names = [ + ...git(['diff', '--name-only', '-z', seed, '--']).split('\0'), + ...git(['ls-files', '--others', '--exclude-standard', '-z']).split('\0'), + ].filter(Boolean); + const changedFiles = [...new Set(names)].sort(); + if (changedFiles.length !== 1 || changedFiles[0] !== 'src/App.jsx') + throw Error('Only src/App.jsx may change'); + const patch = git(['diff', '--binary', seed, '--', 'src/App.jsx']); + if (!patch) throw Error('Empty repair patch'); + return { changedFiles, patch }; +} diff --git a/scripts/cloud/repair-run.ts b/scripts/cloud/repair-run.ts new file mode 100644 index 0000000..2fd7d40 --- /dev/null +++ b/scripts/cloud/repair-run.ts @@ -0,0 +1,213 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { parseArgs } from 'node:util'; +import { oauth, Sandboxes } from '@docker/sandboxes'; +import { runCloudReplay } from './runner.js'; +import { gradeRepair } from './repair-grade.js'; +import { withPinnedSource, writeArtifactProvenance } from './repair-source.js'; + +const { values } = parseArgs({ + options: { + help: { type: 'boolean' }, + local: { type: 'boolean' }, + condition: { type: 'string', default: 'baseline' }, + 'provider-secret': { type: 'string' }, + 'report-dir': { type: 'string' }, + }, +}); +if (values.help) { + console.log( + 'Usage: pnpm exec tsx scripts/cloud/repair-run.ts --condition=baseline|domscribe [--local] [--provider-secret=secrets/] [--report-dir=]\nCloud mode uses one Codex kit and an existing proxy-managed OpenAI secret reference; local mode uses the current Codex CLI sign-in. Reports distinguish both environments. No credentials are created or copied.', + ); + process.exit(0); +} +if (!['baseline', 'domscribe'].includes(values.condition)) + throw Error('Invalid condition'); +if ( + !values.local && + !/^secrets\/[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test( + values['provider-secret'] ?? '', + ) +) + throw Error( + 'Cloud mode requires an existing provider secret resource reference', + ); +for (const name of [ + 'SANDBOXES_API_URL', + 'DOCKER_OAUTH_ISSUER', + 'DOCKER_TOKEN_EXCHANGE_URL', + 'DOCKER_GOVERNANCE_URL', +]) + if (process.env[name] !== undefined) + throw Error(`Unset ${name} before authentication`); +const sourceRoot = fileURLToPath(new URL('../../', import.meta.url)), + here = fileURLToPath(new URL('./', import.meta.url)); +const commit = '23063854647954f4001d05a21a928bdc12bdc6fc'; +const reportDir = values['report-dir'] + ? path.resolve(values['report-dir']) + : fs.mkdtempSync(path.join(os.tmpdir(), 'domscribe-repair-evidence-')); +fs.mkdirSync(reportDir, { recursive: true, mode: 0o700 }); +if (fs.readdirSync(reportDir).length) + throw Error('Report directory must be empty'); +console.log(`Pilot evidence: ${reportDir}`); +const environment = values.local ? 'local-control' : 'docker-cloud'; +const cancellation = new AbortController(); +for (const s of ['SIGINT', 'SIGTERM'] as const) + process.once(s, () => cancellation.abort()); +let client: Sandboxes | undefined; +try { + if (values.local) { + const controller = await import( + new URL('./repair-controller.mjs', import.meta.url).href + ); + const parent = fs.mkdtempSync( + path.join(os.tmpdir(), 'domscribe-repair-local-'), + ); + const { runOwnedBuild } = await import( + new URL('../fixtures/replay-process.mjs', import.meta.url).href + ); + await withPinnedSource(sourceRoot, commit, async (pinned) => { + const env = { + ...process.env, + NX_DAEMON: 'false', + NODE_OPTIONS: '--max-old-space-size=4096', + }; + await runOwnedBuild('pnpm', ['install', '--frozen-lockfile'], { + cwd: pinned, + env, + logPath: path.join(reportDir, 'pinned-install.log'), + timeoutMs: 180000, + signal: cancellation.signal, + label: 'Pinned source install', + }); + await runOwnedBuild( + 'pnpm', + [ + 'exec', + 'nx', + 'run-many', + '-t', + 'build,sync-dist', + '--exclude', + 'domscribe-test-fixtures', + '--parallel=2', + '--skip-nx-cache', + ], + { + cwd: pinned, + env, + logPath: path.join(reportDir, 'pinned-build.log'), + timeoutMs: 300000, + signal: cancellation.signal, + label: 'Pinned artifact build', + }, + ); + writeArtifactProvenance(pinned, commit); + await controller.runRepairExperiment({ + sourceRoot: pinned, + workRoot: path.join(parent, 'work'), + reportDir, + condition: values.condition, + sourceCommit: commit, + environment, + signal: cancellation.signal, + }); + }); + } else { + const auth = oauth({ + onVerification({ verificationUriComplete, verificationUri, userCode }) { + console.error( + `Docker sign-in: ${verificationUriComplete ?? verificationUri}`, + ); + if (!verificationUriComplete) console.error(userCode); + }, + }); + await auth.getAccessToken({ + signal: AbortSignal.any([ + cancellation.signal, + AbortSignal.timeout(300000), + ]), + }); + client = new Sandboxes({ auth, maxRetries: 1 }); + const payload = [ + 'repair-controller.mjs', + 'repair-fixture.mjs', + 'repair-verify.mjs', + 'repair-grade.ts', + 'repair-agent.ts', + 'repair-patch.ts', + 'repair-source.ts', + 'repair-process.mjs', + ]; + let bootstrap = + fs.readFileSync(path.join(here, 'bootstrap.sh'), 'utf8') + + '\nmkdir -p /tmp/domscribe-repair-controller\n'; + for (const name of payload) + bootstrap += `printf '%s' '${fs.readFileSync(name === 'repair-process.mjs' ? path.join(sourceRoot, 'scripts/fixtures/replay-process.mjs') : path.join(here, name)).toString('base64')}' | base64 --decode > /tmp/domscribe-repair-controller/${name}\n`; + bootstrap += `printf '%s' '{"type":"module"}' > /tmp/domscribe-repair-controller/package.json\nnode --input-type=module <<'DOMSCRIBE_PILOT'\nimport {runRepairExperiment} from '/tmp/domscribe-repair-controller/repair-controller.mjs';\nimport {writeArtifactProvenance} from '/tmp/domscribe-repair-controller/repair-source.ts';\nwriteArtifactProvenance('/tmp/domscribe-cloud-smoke/subject','${commit}');\nawait runRepairExperiment({sourceRoot:'/tmp/domscribe-cloud-smoke/subject',workRoot:'/tmp/domscribe-repair-pilot',reportDir:'/tmp/domscribe-cloud-evidence',condition:${JSON.stringify(values.condition)},sourceCommit:'${commit}',environment:'docker-cloud'});\nDOMSCRIBE_PILOT\n`; + const report = await runCloudReplay(client, { + commit, + reportDir, + bootstrap, + kit: 'codex', + secretReferences: [values['provider-secret'] as string], + extraArtifacts: [ + 'repair-report.json', + 'repair.patch', + 'repair-agent.jsonl', + 'repair-before.png', + 'repair-after.png', + 'repair-annotation.json', + ], + signal: cancellation.signal, + onProgress: (p) => console.log('Cloud pilot: ' + p), + }); + if (report.status !== 'passed') + throw Error('Cloud workload failed; inspect cloud-result.json'); + } + const result = JSON.parse( + fs.readFileSync(path.join(reportDir, 'repair-report.json'), 'utf8'), + ); + const fixture = await import( + new URL('./repair-fixture.mjs', import.meta.url).href + ); + gradeRepair(result, values.condition, commit, fixture.SOURCE_HASH); + fs.writeFileSync( + path.join(reportDir, 'pilot-result.json'), + JSON.stringify( + { + status: 'passed', + environment, + condition: values.condition, + sourceCommit: commit, + at: new Date().toISOString(), + }, + null, + 2, + ), + ); + console.log(`Repair pilot passed (${environment}, ${values.condition})`); +} catch (error) { + const message = + error instanceof Error ? error.message : 'Unknown pilot failure'; + fs.writeFileSync( + path.join(reportDir, 'pilot-result.json'), + JSON.stringify( + { + status: 'failed', + environment, + condition: values.condition, + error: message, + at: new Date().toISOString(), + }, + null, + 2, + ), + ); + console.error(message); + process.exitCode = 1; +} finally { + await client?.close(); +} diff --git a/scripts/cloud/repair-source.test.ts b/scripts/cloud/repair-source.test.ts new file mode 100644 index 0000000..46d8942 --- /dev/null +++ b/scripts/cloud/repair-source.test.ts @@ -0,0 +1,102 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { execFileSync } from 'node:child_process'; +import { withPinnedSource } from './repair-source.js'; +test('pinned build excludes newer commits and dirty caller files and cleans after callback failure', async () => { + const parent = fs.mkdtempSync(path.join(os.tmpdir(), 'repair-source-test-')), + repo = path.join(parent, 'repo'); + fs.mkdirSync(repo); + const git = (args: string[]) => + execFileSync('git', args, { + cwd: repo, + encoding: 'utf8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + const commit = (message: string) => + git([ + '-c', + 'user.name=Test', + '-c', + 'user.email=test@example.invalid', + 'commit', + '-am', + message, + ]); + try { + git(['init']); + fs.writeFileSync(path.join(repo, 'source.txt'), 'pinned'); + git(['add', '.']); + commit('pinned'); + const pinned = git(['rev-parse', 'HEAD']).trim(); + fs.writeFileSync(path.join(repo, 'source.txt'), 'newer'); + commit('newer'); + fs.writeFileSync(path.join(repo, 'source.txt'), 'dirty caller'); + let checkout = ''; + await assert.rejects( + withPinnedSource(repo, pinned, async (root) => { + checkout = root; + assert.equal( + fs.readFileSync(path.join(root, 'source.txt'), 'utf8'), + 'pinned', + ); + throw Error('callback failure'); + }), + /callback failure/, + ); + assert(!fs.existsSync(checkout)); + assert.equal( + fs.readFileSync(path.join(repo, 'source.txt'), 'utf8'), + 'dirty caller', + ); + assert.equal( + (git(['worktree', 'list', '--porcelain']).match(/^worktree /gm) || []) + .length, + 1, + ); + } finally { + fs.rmSync(parent, { recursive: true, force: true }); + } +}); + +test('artifact provenance rejects dist changes after a verified clean-source build', async () => { + const parent = fs.mkdtempSync( + path.join(os.tmpdir(), 'repair-artifacts-test-'), + ), + repo = path.join(parent, 'repo'); + fs.mkdirSync(repo); + const git = (args: string[]) => + execFileSync('git', args, { + cwd: repo, + encoding: 'utf8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + try { + git(['init']); + fs.writeFileSync(path.join(repo, '.gitignore'), 'dist\n'); + git(['add', '.']); + git([ + '-c', + 'user.name=Test', + '-c', + 'user.email=test@example.invalid', + 'commit', + '-m', + 'seed', + ]); + const commit = git(['rev-parse', 'HEAD']).trim(); + fs.mkdirSync(path.join(repo, 'dist/packages/test'), { recursive: true }); + const output = path.join(repo, 'dist/packages/test/file.js'); + fs.writeFileSync(output, 'original'); + const { writeArtifactProvenance, readArtifactProvenance } = + await import('./repair-source.js'); + writeArtifactProvenance(repo, commit); + assert.equal(readArtifactProvenance(repo, commit).commit, commit); + fs.writeFileSync(output, 'stale or altered artifact'); + assert.throws(() => readArtifactProvenance(repo, commit)); + } finally { + fs.rmSync(parent, { recursive: true, force: true }); + } +}); diff --git a/scripts/cloud/repair-source.ts b/scripts/cloud/repair-source.ts new file mode 100644 index 0000000..bb61c71 --- /dev/null +++ b/scripts/cloud/repair-source.ts @@ -0,0 +1,147 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { execFileSync } from 'node:child_process'; +/** Use an exact source checkout without carrying caller changes or old dist files. */ +export async function withPinnedSource( + repo: string, + commit: string, + work: (root: string) => Promise, +): Promise { + if (!/^[a-f0-9]{40}$/.test(commit)) + throw Error('Pinned source requires an exact commit'); + const git = (args: string[]) => + execFileSync('git', args, { + cwd: repo, + encoding: 'utf8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + if (git(['rev-parse', commit + '^{commit}']).trim() !== commit) + throw Error('Pinned commit is unavailable'); + const parent = fs.mkdtempSync( + path.join( + path.dirname(fs.realpathSync(repo)), + '.domscribe-pilot-source-', + ), + ), + root = path.join(parent, 'source'); + let registered = false, + completed = false; + const cleanup = () => { + if (registered) git(['worktree', 'remove', '--', root]); + fs.rmdirSync(parent); + }; + try { + git(['worktree', 'add', '--detach', root, commit]); + registered = true; + if ( + execFileSync('git', ['rev-parse', 'HEAD'], { + cwd: root, + encoding: 'utf8', + }).trim() !== commit + ) + throw Error('Source checkout identity mismatch'); + const result = await work(root); + completed = true; + cleanup(); + return result; + } catch (error) { + if (!completed) { + try { + cleanup(); + } catch (cleanupError) { + throw new AggregateError( + [error, cleanupError], + `Pinned source cleanup needs attention: ${root}`, + { cause: cleanupError }, + ); + } + } + throw error; + } +} + +import { createHash } from 'node:crypto'; +interface ArtifactProvenance { + commit: string; + node: string; + distDigest: string; + cleanSource: true; + at: string; +} +function sourceIdentity(root: string, commit: string) { + const options = { + cwd: root, + encoding: 'utf8' as const, + stdio: ['pipe', 'pipe', 'pipe'] as ['pipe', 'pipe', 'pipe'], + }; + if (execFileSync('git', ['rev-parse', 'HEAD'], options).trim() !== commit) + throw Error('Artifact checkout differs from pinned commit'); + if ( + execFileSync( + 'git', + ['status', '--porcelain', '--untracked-files=all'], + options, + ).trim() + ) + throw Error('Artifact source checkout is not clean'); +} +function distDigest(root: string) { + const hash = createHash('sha256'), + base = path.join(root, 'dist/packages'); + let files = 0; + function walk(dir: string) { + for (const name of fs.readdirSync(dir).sort()) { + const file = path.join(dir, name), + stat = fs.lstatSync(file); + if (stat.isSymbolicLink()) + throw Error('Artifact symlinks are unsupported'); + if (stat.isDirectory()) walk(file); + else if (stat.isFile()) { + files++; + hash.update(path.relative(base, file)); + hash.update('\0'); + hash.update(String(stat.mode & 0o777)); + hash.update('\0'); + hash.update(fs.readFileSync(file)); + } + } + } + walk(base); + if (!files) throw Error('No built artifacts'); + return hash.digest('hex'); +} +export function writeArtifactProvenance( + root: string, + commit: string, +): ArtifactProvenance { + sourceIdentity(root, commit); + const record: ArtifactProvenance = { + commit, + node: process.version, + distDigest: distDigest(root), + cleanSource: true, + at: new Date().toISOString(), + }; + fs.writeFileSync( + path.join(root, 'dist/.domscribe-artifacts.json'), + JSON.stringify(record, null, 2), + ); + return record; +} +export function readArtifactProvenance( + root: string, + commit: string, +): ArtifactProvenance { + sourceIdentity(root, commit); + const record = JSON.parse( + fs.readFileSync(path.join(root, 'dist/.domscribe-artifacts.json'), 'utf8'), + ) as ArtifactProvenance; + if ( + record.commit !== commit || + record.node !== process.version || + record.cleanSource !== true || + record.distDigest !== distDigest(root) + ) + throw Error('Built artifacts lack matching pinned-source provenance'); + return record; +} diff --git a/scripts/cloud/repair-verify.mjs b/scripts/cloud/repair-verify.mjs new file mode 100644 index 0000000..d84244e --- /dev/null +++ b/scripts/cloud/repair-verify.mjs @@ -0,0 +1,161 @@ +// Controller-owned browser assertions. Do not copy into the agent app worktree. +import fs from 'node:fs'; +import path from 'node:path'; +const labels = { + 'line-a': 'Midnight travel mug', + 'line-b': 'Ocean travel mug', + 'line-c': 'Graphite notebook', +}; +async function quantities(page) { + const result = {}; + for (const [id, label] of Object.entries(labels)) + result[id] = Number( + await page + .getByRole('article', { name: label, exact: true }) + .locator('output') + .innerText(), + ); + return result; +} +async function edit(page, id, value, cancel = false) { + await page + .getByRole('article', { name: labels[id], exact: true }) + .getByRole('button', { name: 'Edit quantity', exact: true }) + .click(); + await page + .getByRole('spinbutton', { name: 'Quantity', exact: true }) + .fill(String(value)); + await page + .getByRole('button', { + name: cancel ? 'Cancel' : 'Save quantity', + exact: true, + }) + .click(); + await page.getByRole('dialog').waitFor({ state: 'hidden' }); +} +function watch(page, evidence) { + page.on('pageerror', (e) => evidence.pageErrors.push(e.message)); + page.on('console', (m) => { + if (m.type() === 'error') evidence.consoleErrors.push(m.text()); + }); + page.on('requestfailed', (r) => + evidence.networkErrors.push({ + url: r.url(), + error: r.failure()?.errorText, + }), + ); + page.on('response', (r) => { + if (r.status() >= 400) + evidence.networkErrors.push({ url: r.url(), status: r.status() }); + }); +} +async function withPage(browser, url, evidence, callback) { + const context = await browser.newContext({ + viewport: { width: 1280, height: 820 }, + }); + const page = await context.newPage(); + watch(page, evidence); + try { + await page.goto(url); + await page + .getByRole('heading', { name: 'Your cart', exact: true }) + .waitFor(); + return await callback(page); + } finally { + await context.close(); + } +} +function errors() { + return { pageErrors: [], consoleErrors: [], networkErrors: [] }; +} +export async function verifyBefore(browser, url, reportDir) { + fs.mkdirSync(reportDir, { recursive: true }); + const e = errors(); + await withPage(browser, url, e, async (page) => { + await edit(page, 'line-a', 3); + await page.reload(); + await page + .getByRole('heading', { name: 'Your cart', exact: true }) + .waitFor(); + e.observed = await quantities(page); + await page.screenshot({ + path: path.join(reportDir, 'repair-before.png'), + fullPage: true, + }); + }); + if (e.pageErrors.length || e.consoleErrors.length || e.networkErrors.length) + throw Error('Before reproduction has application errors'); + if ( + JSON.stringify(e.observed) !== + JSON.stringify({ 'line-a': 3, 'line-b': 3, 'line-c': 1 }) + ) + throw Error( + 'Expected seeded same-SKU sibling corruption was not reproduced', + ); + return e; +} +export async function verifyAfter(browser, devUrl, productionUrl, reportDir) { + const e = errors(); + await withPage(browser, devUrl, e, async (page) => { + await edit(page, 'line-a', 3); + e.saved = await quantities(page); + await page.reload(); + await page + .getByRole('heading', { name: 'Your cart', exact: true }) + .waitFor(); + e.reloaded = await quantities(page); + await page.screenshot({ + path: path.join(reportDir, 'repair-after.png'), + fullPage: true, + }); + }); + await withPage(browser, devUrl, e, async (page) => { + await page + .getByRole('button', { name: 'Reverse order', exact: true }) + .click(); + await edit(page, 'line-b', 4); + await page.reload(); + await page + .getByRole('heading', { name: 'Your cart', exact: true }) + .waitFor(); + e.sorted = await quantities(page); + }); + await withPage(browser, devUrl, e, async (page) => { + const filter = page.getByRole('textbox', { + name: 'Filter items', + exact: true, + }); + await filter.fill('Graphite'); + await edit(page, 'line-c', 2); + await filter.fill(''); + await page.reload(); + await page + .getByRole('heading', { name: 'Your cart', exact: true }) + .waitFor(); + e.filtered = await quantities(page); + }); + await withPage(browser, devUrl, e, async (page) => { + await edit(page, 'line-a', 7, true); + await page.reload(); + await page + .getByRole('heading', { name: 'Your cart', exact: true }) + .waitFor(); + e.cancelled = await quantities(page); + }); + await withPage(browser, productionUrl, e, async (page) => { + await edit(page, 'line-a', 3); + e.production = await quantities(page); + e.productionInstrumented = await page.evaluate(() => + Boolean( + window.__runtime || + window.__DOMSCRIBE_RELAY_PORT__ || + [...document.querySelectorAll('*')].some( + (el) => + el.tagName.toLowerCase().includes('domscribe') || + [...el.attributes].some((a) => a.name.startsWith('data-ds')), + ), + ), + ); + }); + return e; +} diff --git a/scripts/cloud/runner.test.ts b/scripts/cloud/runner.test.ts index bdc88f6..31f5bf2 100644 --- a/scripts/cloud/runner.test.ts +++ b/scripts/cloud/runner.test.ts @@ -53,7 +53,8 @@ interface FakeOptions { } function fake(options: FakeOptions = {}) { const calls: string[] = []; - let launchInput: KitLaunchOptions | undefined, + let launchKit: string | undefined, + launchInput: KitLaunchOptions | undefined, createdOptions: CallOptions | undefined, cleanupSignal: AbortSignal | undefined, executionSignal: AbortSignal | undefined; @@ -153,6 +154,7 @@ function fake(options: FakeOptions = {}) { kits: { async launch(_kit, input, call) { calls.push('launch'); + launchKit = _kit; launchInput = input; createdOptions = call; if (options.launchError) @@ -186,6 +188,9 @@ function fake(options: FakeOptions = {}) { client, calls, bytes, + get launchKit() { + return launchKit; + }, get launchInput() { return launchInput; }, @@ -360,3 +365,52 @@ test('cancellation during streaming detaches and deletes the accepted sandbox wi assert(controller.signal.aborted); assert(f.cleanupSignal && !f.cleanupSignal.aborted); }); + +test('controlled agent workload launches the selected kit with a secret reference and retains required extra evidence', async () => { + const f = fake(); + f.bytes.set( + 'repair-report.json', + Buffer.from('{"kind":"controller-evidence"}'), + ); + const r = await exercise(f, { + kit: 'codex', + secretReferences: ['secrets/00000000-0000-4000-8000-000000000000'], + extraArtifacts: ['repair-report.json'], + }); + assert.equal(r.status, 'passed'); + assert.equal(f.launchKit, 'codex'); + assert.deepEqual(f.launchInput?.storage, { + secrets: ['secrets/00000000-0000-4000-8000-000000000000'], + }); + assert(r.artifacts.includes('repair-report.json')); +}); +test('agent evidence cannot overwrite local journals or escape the evidence directory', async () => { + for (const extraArtifacts of [ + ['cloud-result.json'], + ['../escape'], + ['bootstrap.log'], + ]) { + const f = fake(); + const r = await exercise(f, { extraArtifacts }); + assert.equal(r.status, 'failed'); + assert(!f.calls.includes('launch')); + } +}); +test('missing requested repair evidence fails even when the ordinary replay smoke passes', async () => { + const f = fake(); + const r = await exercise(f, { extraArtifacts: ['repair-report.json'] }); + assert.equal(r.status, 'failed'); + assert.equal(r.cloud.deletion, 'confirmed'); +}); +test('agent launches require a provider resource reference, never a token value', async () => { + for (const secretReferences of [ + [], + ['sk-never-transmit-test-value'], + ['secrets/../../other'], + ]) { + const f = fake(); + const r = await exercise(f, { kit: 'codex', secretReferences }); + assert.equal(r.status, 'failed'); + assert(!f.calls.includes('launch')); + } +}); diff --git a/scripts/cloud/runner.ts b/scripts/cloud/runner.ts index 51dc098..c047e8e 100644 --- a/scripts/cloud/runner.ts +++ b/scripts/cloud/runner.ts @@ -99,6 +99,7 @@ export interface CloudRunReport { source: { repository: string; commit: string }; sdkVersion?: string; cloud: { + kit?: 'shell' | 'codex'; name?: string; uid?: string; resources: 'medium'; @@ -111,6 +112,9 @@ export interface CloudRunReport { browser?: Record; } export interface RunOptions { + kit?: 'shell' | 'codex'; + secretReferences?: readonly string[]; + extraArtifacts?: readonly string[]; commit: string; reportDir: string; bootstrap: string; @@ -254,6 +258,7 @@ export async function runCloudReplay( source: { repository: SOURCE_REPOSITORY, commit: options.commit }, sdkVersion: options.sdkVersion, cloud: { + kit: options.kit ?? 'shell', resources: 'medium', ttlMs: 30 * 60 * 1000, deletion: 'not_created', @@ -267,6 +272,9 @@ export async function runCloudReplay( phase = 'input', output: number | undefined, writable = false; + let artifacts: readonly string[] = ARTIFACTS; + const extraArtifacts = options.extraArtifacts ?? []; + const secretReferences = options.secretReferences ?? []; const timeout = options.timeoutMs ?? 25 * 60 * 1000, cleanupMs = options.cleanupMs ?? 60000; const save = () => { @@ -287,6 +295,45 @@ export async function runCloudReplay( options.onProgress?.(next); }; try { + ensure( + ['shell', 'codex'].includes(options.kit ?? 'shell'), + 'Unsupported controlled workload kit', + ); + ensure( + Array.isArray(secretReferences) && + secretReferences.every( + (ref) => + typeof ref === 'string' && + /^secrets\/[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test( + ref, + ), + ), + 'Provider references must be secret resource names', + ); + ensure( + (options.kit ?? 'shell') === 'codex' + ? secretReferences.length === 1 + : secretReferences.length === 0, + 'Codex requires one provider reference; shell smoke requires none', + ); + ensure( + Array.isArray(extraArtifacts) && + extraArtifacts.length <= 8 && + new Set(extraArtifacts).size === extraArtifacts.length && + extraArtifacts.every( + (name) => + typeof name === 'string' && + /^[a-zA-Z0-9][a-zA-Z0-9._-]{0,79}$/.test(name) && + ![ + ...ARTIFACTS, + 'cloud-result.json', + 'cloud-result.json.tmp', + 'bootstrap.log', + ].includes(name), + ), + 'Unsafe, duplicate or reserved extra artifact', + ); + artifacts = [...ARTIFACTS, ...extraArtifacts]; ensure( /^[a-f0-9]{40}$/.test(options.commit), 'Use an exact lowercase40-character source commit', @@ -311,12 +358,15 @@ export async function runCloudReplay( report.cloud.deletion = 'unknown_creation'; save(); owned = await client.kits.launch( - 'shell', + options.kit ?? 'shell', { displayName: `domscribe-replay-${report.runId}`, resources: 'medium', lifecycle: { timeoutMs: report.cloud.ttlMs, onTimeout: 'delete' }, labels: { domscribeRunId: report.runId }, + ...(secretReferences.length + ? { storage: { secrets: [...secretReferences] } } + : {}), }, { ...execution, @@ -415,7 +465,7 @@ export async function runCloudReplay( phase = 'artifacts'; const transfer = call(60000); let total = 0; - for (const name of ARTIFACTS) { + for (const name of artifacts) { try { const bytes = await running.files.read( `${REMOTE_REPORT}/${name}`, @@ -442,7 +492,7 @@ export async function runCloudReplay( } if (report.failures.length === 0) { try { - for (const name of REQUIRED_ARTIFACTS) + for (const name of [...REQUIRED_ARTIFACTS, ...extraArtifacts]) ensure( report.artifacts.includes(name), `Missing required artifact: ${name}`, diff --git a/scripts/fixtures/replay-process.mjs b/scripts/fixtures/replay-process.mjs index ed801c5..08547b9 100644 --- a/scripts/fixtures/replay-process.mjs +++ b/scripts/fixtures/replay-process.mjs @@ -9,7 +9,9 @@ export async function stopOwnedProcess(child, graceMs = 1000) { if (!child?.pid) return; try { process.kill(-child.pid, 'SIGTERM'); - } catch {} + } catch { + /* Best-effort cleanup/readiness; the owned process may already be gone. */ + } if (child.exitCode === null && child.signalCode === null) { await new Promise((resolve) => { const timer = setTimeout(resolve, graceMs); @@ -22,16 +24,27 @@ export async function stopOwnedProcess(child, graceMs = 1000) { // The leader may have exited before its workers. Always retire the group. try { process.kill(-child.pid, 'SIGKILL'); - } catch {} + } catch { + /* Best-effort cleanup/readiness; the owned process may already be gone. */ + } } export async function runOwnedBuild( command, args, - { cwd, env, logPath, timeoutMs = 300000, graceMs = 1000 }, + { + cwd, + env, + logPath, + timeoutMs = 300000, + graceMs = 1000, + signal, + label = 'Production build', + }, ) { + signal?.throwIfAborted(); const log = fs.openSync(logPath, 'w'); - let child, timer; + let child, timer, onAbort; try { child = startOwnedProcess(command, args, { cwd, @@ -39,21 +52,32 @@ export async function runOwnedBuild( stdio: ['ignore', log, log], }); await new Promise((resolve, reject) => { - let timedOut = false; - timer = setTimeout(async () => { - timedOut = true; - await stopOwnedProcess(child, graceMs); - reject(new Error(`Production build timed out after ${timeoutMs}ms`)); - }, timeoutMs); + let interrupted = false; + const interrupt = (message) => { + if (interrupted) return; + interrupted = true; + stopOwnedProcess(child, graceMs).then( + () => reject(new Error(message)), + reject, + ); + }; + timer = setTimeout( + () => interrupt(`${label} timed out after ${timeoutMs}ms`), + timeoutMs, + ); + onAbort = () => interrupt(`${label} cancelled`); + signal?.addEventListener('abort', onAbort, { once: true }); + if (signal?.aborted) onAbort(); child.once('error', reject); child.once('exit', (code, signal) => { - if (timedOut) return; + if (interrupted) return; if (code === 0) resolve(); - else reject(new Error(`Production build failed: ${signal ?? code}`)); + else reject(new Error(`${label} failed: ${signal ?? code}`)); }); }); } finally { clearTimeout(timer); + if (onAbort) signal?.removeEventListener('abort', onAbort); await stopOwnedProcess(child, graceMs); fs.closeSync(log); } diff --git a/scripts/fixtures/replay-process.test.mjs b/scripts/fixtures/replay-process.test.mjs index 1ed71cb..d21fa73 100644 --- a/scripts/fixtures/replay-process.test.mjs +++ b/scripts/fixtures/replay-process.test.mjs @@ -85,9 +85,67 @@ test( for (const pid of JSON.parse(fs.readFileSync(marker, 'utf8'))) { try { process.kill(pid, 'SIGKILL'); - } catch {} + } catch { + /* Best-effort cleanup/readiness; the owned process may already be gone. */ + } } fs.rmSync(cwd, { recursive: true, force: true }); } }, ); + +test( + 'cancellation stops a running command group before its ordinary deadline', + { timeout: 5000 }, + async () => { + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'replay-cancel-test-')); + const marker = path.join(cwd, 'pid'); + const controller = new AbortController(); + let pid; + try { + const pending = runOwnedBuild( + process.execPath, + [ + '-e', + `require('node:fs').writeFileSync(${JSON.stringify(marker)},String(process.pid));process.on('SIGTERM',()=>{});setInterval(()=>{},1000);`, + ], + { + cwd, + logPath: path.join(cwd, 'log'), + timeoutMs: 1000, + graceMs: 50, + signal: controller.signal, + }, + ); + while (!fs.existsSync(marker)) + await new Promise((r) => setTimeout(r, 10)); + pid = Number(fs.readFileSync(marker, 'utf8')); + const started = Date.now(); + controller.abort(); + await assert.rejects(pending, /cancelled/); + assert( + Date.now() - started < 750, + 'cancellation must not wait for the normal deadline', + ); + let alive = false; + try { + alive = !fs + .readFileSync(`/proc/${pid}/stat`, 'utf8') + .split(') ')[1] + .startsWith('Z'); + } catch (e) { + if (e.code !== 'ENOENT') throw e; + } + assert.equal(alive, false); + } finally { + if (pid) { + try { + process.kill(pid, 'SIGKILL'); + } catch { + /* Best-effort cleanup/readiness; the owned process may already be gone. */ + } + } + fs.rmSync(cwd, { recursive: true, force: true }); + } + }, +);