diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8d175ce..ae1da04 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -36,6 +36,14 @@ jobs: - name: Verify browser harness build cleanup run: node --test scripts/fixtures/replay-process.test.mjs + - name: Verify Docker cloud runner offline + run: | + pnpm exec tsx --test scripts/cloud/*.test.ts + pnpm exec tsc -p scripts/cloud/tsconfig.cloud.json + pnpm exec eslint scripts/cloud/*.ts --max-warnings=0 + bash -n scripts/cloud/bootstrap.sh + pnpm exec tsx scripts/cloud/run.ts --help + - run: pnpm exec nx run-many -t lint test build typecheck --exclude domscribe-test-fixtures - name: Update coverage badge diff --git a/README.md b/README.md index cc8dc5f..02f62a9 100644 --- a/README.md +++ b/README.md @@ -449,6 +449,9 @@ nx run-many -t build test lint typecheck Conventions are in `.claude/rules/`. PRs welcome. +For an isolated browser replay check through the Docker Node SDK, see the +[Docker Cloud replay smoke test](./docs/docker-cloud-replay.md). + --- ## License diff --git a/docs/docker-cloud-replay.md b/docs/docker-cloud-replay.md new file mode 100644 index 0000000..04d8cfd --- /dev/null +++ b/docs/docker-cloud-replay.md @@ -0,0 +1,63 @@ +# Docker Cloud replay smoke test + +This development runner uses the official [`@docker/sandboxes` Node SDK](https://docs.docker.com/ai/sandboxes-api/get-started/) to run Domscribe's existing browser verification in a disposable cloud sandbox. It checks that a saved annotation selects the intended customer after rows reorder in a fresh Chromium session, reads the correct source/props/state, and changes only that customer. It also checks stale and ambiguous targets, application errors, and production instrumentation removal. + +The workload builds a pinned public Domscribe commit and installs its tarballs into a new React 18.3.1/npm/Vite 7.3.7 consumer. This is source-artifact verification; the unreleased APIs are not in npm 0.6.1. Other framework/browser combinations are covered separately by the portable-replay CI matrix. A cloud smoke run does not establish that an agent can fix arbitrary issues. + +## Run + +Use Node 24 and the repository's pnpm 9.12.0. From an isolated worktree: + +```sh +pnpm install --frozen-lockfile +pnpm exec tsx scripts/cloud/run.ts --help +pnpm exec tsx scripts/cloud/run.ts +``` + +The default commit is the reviewed foundation at `92e939a643b7f882b5f997e3197413befcc93b41`. To verify another public commit containing the same harness: + +```sh +pnpm exec tsx scripts/cloud/run.ts \ + --commit= \ + --report-dir=/path/to/an/empty/evidence-directory +``` + +The default OAuth flow displays a Docker sign-in link. Authentication is handled by the SDK and held in memory for this process; Docker CLI login is separate. For unattended use, `--auth=pat` reads `DOCKER_ID` and `DOCKER_PAT` from the environment. Supply them through your secret store. Do not put credential values in command arguments, files, reports, or GitHub issue text. No model-provider credential is needed. The runner rejects SDK endpoint environment overrides before authentication. + +One `medium` shell sandbox is requested with automatic deletion after 30 minutes. Execution is limited to 25 minutes, artifact collection to another minute, and explicit cleanup to another minute. Running it uses your Docker account's cloud resources. There is no automatic cloud run on a PR: CI only runs offline lifecycle tests, type checking, lint and shell syntax validation. + +Inside the sandbox, the bootstrap creates an isolated worktree, installs dependencies with the frozen lockfile, builds/syncs artifacts without the Nx cache, and installs Chromium with its Linux dependencies. Node 24 is installed from the official checksum-verified distribution if needed. Host workspaces, credentials and secret files are not uploaded. The optional stored-secrets feature is omitted entirely when launching this workload. + +## Evidence and failure handling + +The CLI prints the local evidence directory. Keep these files together: + +- `cloud-result.json`: requested commit, SDK version, accepted sandbox identity, process exit, browser validation and cleanup outcome. +- `bootstrap.log`: bounded streamed setup/build/browser output. +- `provenance.json`, `result.json`, `trace.json`, `replayed.png`: executed source/runtime, browser assertions, persisted four-step trace and replay screenshot. +- Consumer install, dev-server and production-build logs; available failure screenshots. + +Success requires all eight named browser assertions, zero recorded application console/page/network errors, matching source/runtime provenance, the saved trace/screenshot, and confirmed deletion of the same sandbox UID. A zero process exit or closed connection alone cannot pass. The fixed evidence allowlist is limited to 4 MiB per file, 16 MiB total; streamed output is limited to 8 MiB. These limits fail the run rather than silently accepting incomplete evidence. + +Cleanup runs after readiness, execution, artifact or validation failure with its own deadline, including cancellation with Ctrl-C. A same-name replacement with a different UID is never deleted. Both primary and cleanup failures are retained. `unknown_creation` means the launch response was lost before identity was obtained; `unconfirmed` means an accepted sandbox was not confirmed deleted. Inspect that run in Docker before starting another; the runner does not claim that client shutdown deleted a sandbox. A rejected validation request is recorded as `not_created`. + +Reports and logs are private local evidence by default. Review them before sharing: the fixed public demo needs no credentials, but future application workloads may capture private UI data. This runner has no general-purpose secret scrubber and is not a boundary for arbitrary untrusted issue instructions. + +## Verified cloud scope + +On October 25, 2026, the actual SDK/service run completed in about two minutes on source `92e939a`: Node 24.22.1, npm 11.19.1, Chromium 156.0.8078.4, React 18.3.1 and Vite 7.3.7. All eight browser checks passed, all three application error collections were empty, evidence downloaded, and same-UID deletion was confirmed. This was one fresh controlled consumer, not an autonomous issue repair or a cloud framework matrix. + +## Offline validation + +These commands do not authenticate or allocate cloud resources: + +```sh +pnpm exec tsx --test scripts/cloud/*.test.ts +pnpm exec tsc -p scripts/cloud/tsconfig.cloud.json +pnpm exec eslint scripts/cloud/*.ts --max-warnings=0 +bash -n scripts/cloud/bootstrap.sh +``` + +The SDK is a private development dependency, pinned at 0.53.0. Its `undici` dependency is narrowly overridden to 6.29.1 to include the upstream security patch; it does not enter Domscribe's public package exports. + +The next milestone is an agent attempting a realistic, deliberately broken UI task in this environment, with evidence that its fix changes the expected behavior. GitHub issue ingestion, repository write credentials and opening fix PRs are separate work after that experiment. diff --git a/docs/superpowers/plans/2026-10-25-docker-cloud-replay.md b/docs/superpowers/plans/2026-10-25-docker-cloud-replay.md new file mode 100644 index 0000000..1355728 --- /dev/null +++ b/docs/superpowers/plans/2026-10-25-docker-cloud-replay.md @@ -0,0 +1,34 @@ +# Docker Cloud Replay Runner Plan + +> **For agentic workers:** Use superpowers:executing-plans inline and obtain independent whole-branch review. Steps use checkboxes. + +**Goal:** Execute the existing Domscribe browser replay verification through the Docker Cloud Sandboxes Node SDK, retaining evidence and cleaning up owned cloud resources reliably. + +**Architecture:** A TypeScript development runner uses `@docker/sandboxes` directly. It launches one shell kit, clones a pinned public Domscribe commit into an isolated Git worktree, builds local artifacts and runs the existing browser harness. It downloads an allowlisted, bounded evidence bundle and records resource identity, process exit, browser result and confirmed deletion separately. A small orchestration module accepts the SDK-shaped client for deterministic lifecycle failure tests; live verification exercises the actual SDK/service. + +**Tech Stack:** Node24, TypeScript/tsx, Docker SDK0.53.0, existing pnpm9/Playwright harness. + +**Spec / user direction:** The approved next milestone in this conversation: SDK-driven shell-kit cloud smoke test first; coding-agent and GitHub issue wrapper follow after this works. User explicitly said to keep going while away. Choose routine implementation details autonomously. + +## Constraints and review focus + +- Reuse isolated worktree on codex/docker-cloud-replay from fresh main92e939a. Preserve primary repositories and unrelated lockfiles; root dev dependency/lock update is intentional for the SDK. +- Public repository and exact commit only. No model-provider credentials or production deployment/publication. Do not retrieve credential values; use SDK-managed interactive authentication or existing explicit environment credentials without logging values. +- One owned sandbox at a time; bound compute/lifetime, launch/setup/run/download/cleanup deadlines and retries. Save identity immediately after accepted creation; never delete a replacement resource with the same name. +- On command, readiness, transfer, result-validation or cleanup failure, retain useful local evidence and return failure. Connection EOF or exit0 alone is not browser success. +- Download fixed evidence names under a fixed remote directory, reject invalid results/oversized files, avoid logging arbitrary secrets or copying repository credentials. +- Clean up after failures using an independent deadline; accurately record unconfirmed cleanup. Client.close is separate from sandbox deletion. Preserve original error when cleanup also fails. +- Local/CI unit, type and shell-script checks run without Docker credentials/cloud charges. Live smoke is explicit, uses a shell kit, and records actual tested scope. No full issue-to-PR Action implementation in this change. + +## Task 1: Runner and lifecycle regressions + +- [x] Add private SDK dev dependency and inspect installed public types/examples for exact APIs. +- [x] Write meaningful failing regressions: process failure/EOF, missing or failed browser result, cleanup on readiness/command/download failure, UID mismatch, cleanup failure preserving primary cause, bounded artifact transfer. Observe RED before implementing. +- [x] Implement minimal runner, fixed pinned bootstrap script, evidence validation and safe lifecycle cleanup; make tests pass and typecheck against installed SDK. + +## Task 2: Real verification and delivery + +- [x] Run configured affected checks, process regression tests, shell syntax and representative local command/packaging validation. +- [x] Attempt a fresh SDK cloud run using available approved authentication; exercise genuine Chromium/relay/selection/state/replay/production flow and retain artifacts. If account consent/login is unavailable while user is away, finish all independent work and report the exact blind spot without claiming cloud success. +- [ ] Obtain independent review, fix material findings with regressions, update docs, push a focused PR and inspect CI. Respect merge protections; earlier admin-bypass approvals were PR-specific. +- [ ] Update development-state.md and daily automation continuity with exact results, cloud cost/resource cleanup status and the next useful step. diff --git a/package.json b/package.json index b98d305..1250fbe 100644 --- a/package.json +++ b/package.json @@ -29,6 +29,7 @@ }, "private": true, "devDependencies": { + "@docker/sandboxes": "0.53.0", "@eslint/js": "^10.0.1", "@nx/devkit": "22.7.12", "@nx/eslint": "22.7.12", @@ -135,7 +136,8 @@ "nx>follow-redirects": "^1.16.1", "nx>brace-expansion": "^5.0.12", "nx>smol-toml": "^1.9.1", - "postcss-selector-parser@<7.1.6": "^7.1.7" + "postcss-selector-parser@<7.1.6": "^7.1.7", + "@docker/sandboxes@0.53.0>undici": "6.29.1" } } } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 61671eb..e9ab324 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -10,11 +10,15 @@ overrides: nx>brace-expansion: ^5.0.12 nx>smol-toml: ^1.9.1 postcss-selector-parser@<7.1.6: ^7.1.7 + '@docker/sandboxes@0.53.0>undici': 6.29.1 importers: .: devDependencies: + '@docker/sandboxes': + specifier: 0.53.0 + version: 0.53.0 '@eslint/js': specifier: ^10.0.1 version: 10.0.1(eslint@10.13.0(jiti@2.6.1)) @@ -71,7 +75,7 @@ importers: version: 2.6.1 jsdom: specifier: ~28.1.0 - version: 28.1.0 + version: 28.1.0(@noble/hashes@1.8.0) jsonc-eslint-parser: specifier: ^3.3.0 version: 3.3.0 @@ -104,7 +108,7 @@ importers: version: 4.5.4(@types/node@25.3.3)(rollup@4.65.0)(typescript@5.9.3)(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) vitest: specifier: 4.1.12 - version: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0)(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) + version: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0(@noble/hashes@1.8.0))(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) packages/domscribe-cli: dependencies: @@ -1072,6 +1076,9 @@ packages: resolution: {integrity: sha512-0GEz8qJWokkUVYQYRukmkSHIFNoBH8QTreeHrAFbvz1XFYc3Y+puhMs8PQ5x8GNugiC2l5jvDBXKLbBIByeSBw==} hasBin: true + '@bufbuild/protobuf@2.16.0': + resolution: {integrity: sha512-FWa0sPlqYGJgpTs6OxBRcL/AW4JT0OIO+W1ajerluoTVh8CV0B7XM1qsNCRgnkSCkRCgOICKHRQlY17jrsm0+Q==} + '@cacheable/memory@2.2.0': resolution: {integrity: sha512-CTLKqLItRCEixEAewD3/j9DB3/o96gpTPD4eJ1v+DGOlxZRZncRQkGYqqnAGCscYd6RNeXfGeiuCphsPtqyIfQ==} @@ -1126,6 +1133,10 @@ packages: resolution: {integrity: sha512-y20e+e6dFYkOUUJLVUZTsJRuTiXZaUQ32WD+R/ux/HBybbTx4ge7cNINcua0pU8+SNkKuRbOF12mBmzuzM8n5w==} engines: {node: '>= 14.17.0'} + '@docker/sandboxes@0.53.0': + resolution: {integrity: sha512-copLxQxHrUAmMdyolkGHr5t5nzOe4+fHdsizT4KpyjRp+pauiwzFlLYRQEqsvigte9kJiCNyIS7dqqxTm5x0Ng==} + engines: {node: '>=20'} + '@emnapi/core@1.11.2': resolution: {integrity: sha512-TC8MkTuZUtcTSiFeuC0ksCh9QIJ5+F21MvZ4Wn4ORfYaFJ/0dsiudv5tVkejgwZlwQ39jL9WWDe2lz8x0WglOA==} @@ -1689,6 +1700,10 @@ packages: cpu: [x64] os: [win32] + '@noble/hashes@1.8.0': + resolution: {integrity: sha512-jCs9ldd7NwzpgXDIf6P3+NrHh9/sD6CQdxHyjQI+h/6rDNo88ypBxxz45UDuZHz9r3tNz7N/VInSVoVdtXEI4A==} + engines: {node: ^14.21.3 || >=16} + '@nodelib/fs.scandir@2.1.5': resolution: {integrity: sha512-vq24Bq3ym5HEQm2NKCr3yXDwjc7vTsEThRDnkp2DK9p1uqLR+DHurm/NOTo0KG7HYHU7eppKZj3MyqYuMBf62g==} engines: {node: '>= 8'} @@ -1960,6 +1975,9 @@ packages: cpu: [x64] os: [win32] + '@paralleldrive/cuid2@2.3.1': + resolution: {integrity: sha512-XO7cAxhnTZl0Yggq6jOgjiOHhbgcO4NqFqwSmQpjK3b6TEE6Uj/jfSk6wzYyemh3+I0sHirKSetjQwn5cZktFw==} + '@phenomnomnominal/tsquery@6.2.0': resolution: {integrity: sha512-Vo9nkhfZxDB/sBiqIY3pjDC4mOSyure+AFlEW5hcy/tRE82MqCXjRN4InnVNMldinRt0dLYqg4HAU2XPq5e1LA==} peerDependencies: @@ -2290,6 +2308,9 @@ packages: '@types/estree@1.0.9': resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==} + '@types/formidable@3.4.6': + resolution: {integrity: sha512-LI4Hk+KNsM5q7br4oMVoaWeb+gUqJpz1N8+Y2Q6Cz9cVH33ybahRKUWaRmMboVlkwSbOUGgwc/pEkS7yMSzoWg==} + '@types/html-minifier-terser@6.1.0': resolution: {integrity: sha512-oh/6byDPnL1zeNXFrDXFLyZjkr1MsBG667IM792caf1L2UPOOMf65NFzjUH/ltyfwjAGfs1rsX1eftK0jC/KIg==} @@ -2755,6 +2776,9 @@ packages: resolution: {integrity: sha512-HGyxoOTYUyCM6stUe6EJgnd4EoewAI7zMdfqO+kGjnlZmBDz/cR5pf8r/cR4Wq60sL/p0IkcjUEEPwS3GFrIyw==} engines: {node: '>=8'} + asap@2.0.6: + resolution: {integrity: sha512-BSHWgDSAiKs50o2Re8ppvp3seVHXSRM44cdSsT9FfNEUUZLOGWVCsiWaRPWM1Znn+mqZ1OfVZ3z3DWEzSp7hRA==} + asn1@0.2.6: resolution: {integrity: sha512-ix/FxPn0MDjeyJ7i/yoHGFt/EX6LyNbxSEhPPXODPL+KB0VPk86UYfL0lMdy+KCnv+fmvIzySwaK5COwqVbWTQ==} @@ -3293,6 +3317,9 @@ packages: engines: {node: '>= 16.0.0'} hasBin: true + dezalgo@1.0.4: + resolution: {integrity: sha512-rXSP0bf+5n0Qonsb+SVVfNfIsimO4HEtmnIpPHY8Q1UCzKlQrDMfdobr8nJOOsRgWCyMRqeSBQzmWUMq7zvVig==} + didyoumean@1.2.2: resolution: {integrity: sha512-gxtyfqMg7GKyhQmb056K7M3xszy/myH8w+B4RT+QXBQsvAOdc3XymqDDPHx1BgPgsdAA5SIifona89YtRATDzw==} @@ -3680,6 +3707,10 @@ packages: resolution: {integrity: sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==} engines: {node: '>= 6'} + formidable@3.5.4: + resolution: {integrity: sha512-YikH+7CUTOtP44ZTnUhR7Ic2UASBPOqmaRkRKxRbywPTe5VxF7RRCck4af9wutiZ/QKM5nME9Bie2fFaPz5Gug==} + engines: {node: '>=14.0.0'} + forwarded@0.2.0: resolution: {integrity: sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==} engines: {node: '>= 0.6'} @@ -3977,6 +4008,10 @@ packages: resolution: {integrity: sha512-2HvIEKRoqS62guEC+qBjpvRubdX910WCMuJTZ+I9yvqKU2/12eSL549HMwtabb4oupdj2sMP50k+XJfB/8JE6w==} engines: {node: '>=8'} + is-network-error@1.3.2: + resolution: {integrity: sha512-PhBY86zaxNZUuWP6h13Vu5oFe0XY6/UlKzQnYFELzGVHygP3MxmvTfYSG7GN3aIab/iWudSMgjSnG9Dq+nHrgA==} + engines: {node: '>=16'} + is-number@7.0.0: resolution: {integrity: sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==} engines: {node: '>=0.12.0'} @@ -4567,9 +4602,15 @@ packages: resolution: {integrity: sha512-7x81NCL719oNbsq/3mh+hVrAWmFuEYUqrq/Iw3kUzH8ReypT9QQ0BLoJS7/G9k6N81XjW4qHWtjWwe/9eLy1EQ==} engines: {node: '>=12'} + openapi-fetch@0.17.0: + resolution: {integrity: sha512-PsbZR1wAPcG91eEthKhN+Zn92FMHxv+/faECIwjXdxfTODGSGegYv0sc1Olz+HYPvKOuoXfp+0pA2XVt2cI0Ig==} + openapi-types@12.1.3: resolution: {integrity: sha512-N4YtSYJqghVu4iek2ZUvcN/0aqH1kRDuNqzcycDxhOUpg7GdvLa2F3DgS6yBNhInhv2r/6I0Flkn7CqL8+nIcw==} + openapi-typescript-helpers@0.1.0: + resolution: {integrity: sha512-OKTGPthhivLw/fHz6c3OPtg72vi86qaMlqbJuVJ23qOvQ+53uw1n7HdmkJFibloF7QEjDrDkzJiOJuockM/ljw==} + opener@1.5.2: resolution: {integrity: sha512-ur5UIdyw5Y7yEj9wLzhqXiy6GZ3Mwx0yGI+5sMn2r0N0v3cKJvUmFH5yPP+WXh9e0xfyzyJX95D8l088DNFj7A==} hasBin: true @@ -4593,6 +4634,10 @@ packages: resolution: {integrity: sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw==} engines: {node: '>=10'} + p-retry@7.1.0: + resolution: {integrity: sha512-xL4PiFRQa/f9L9ZvR4/gUCRNus4N8YX80ku8kv9Jqz+ZokkiZLM0bcvX0gm1F3PDi9SPRsww1BDsTWgE6Y1GLQ==} + engines: {node: '>=20'} + package-manager-detector@1.9.0: resolution: {integrity: sha512-zAMfbta9nPA8AKkA4WdmNAT8R8s6Dk0/xwPsqNnRidMFcLo1YkLuPDTgs2f64XdzS8vbLNnoJ+uUIkgbz7oRHA==} @@ -5481,6 +5526,10 @@ packages: undici-types@8.9.0: resolution: {integrity: sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==} + undici@6.29.1: + resolution: {integrity: sha512-EvRlUt/QqAEvXdQeRMYUfwOQsfzpDztykrG2owGEE/rBOBoYv9uF39ZzsNDh4lTFHhTr2XOZjKkYgueD12ZmGA==} + engines: {node: '>=18.17'} + undici@7.30.1: resolution: {integrity: sha512-03KrMbjIFDWQR5RNPrSw3bd1r12JpyFQl9/yxjUbJtqkjmllqC5lFjZh6ujeuJCKr8EOQvRW872Oy/az+nLYxg==} engines: {node: '>=20.18.1'} @@ -6597,6 +6646,8 @@ snapshots: dependencies: css-tree: 3.2.1 + '@bufbuild/protobuf@2.16.0': {} + '@cacheable/memory@2.2.0': dependencies: '@cacheable/utils': 2.5.0 @@ -6665,6 +6716,21 @@ snapshots: tough-cookie: 5.1.2 tunnel-agent: 0.6.0 + '@docker/sandboxes@0.53.0': + dependencies: + '@bufbuild/protobuf': 2.16.0 + '@types/formidable': 3.4.6 + '@types/ws': 8.18.2 + formidable: 3.5.4 + jose: 6.2.13 + openapi-fetch: 0.17.0 + p-retry: 7.1.0 + undici: 6.29.1 + ws: 8.22.0 + transitivePeerDependencies: + - bufferutil + - utf-8-validate + '@emnapi/core@1.11.2': dependencies: '@emnapi/wasi-threads': 1.2.2 @@ -6817,7 +6883,9 @@ snapshots: '@eslint/core': 1.2.1 levn: 0.4.1 - '@exodus/bytes@1.16.0': {} + '@exodus/bytes@1.16.0(@noble/hashes@1.8.0)': + optionalDependencies: + '@noble/hashes': 1.8.0 '@fastify/ajv-compiler@4.0.7': dependencies: @@ -7134,6 +7202,8 @@ snapshots: '@next/swc-win32-x64-msvc@16.4.3': optional: true + '@noble/hashes@1.8.0': {} + '@nodelib/fs.scandir@2.1.5': dependencies: '@nodelib/fs.stat': 2.0.5 @@ -7320,7 +7390,7 @@ snapshots: tsconfig-paths: 4.2.0 tslib: 2.8.1 vite: 7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0) - vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0)(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) + vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0(@noble/hashes@1.8.0))(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) transitivePeerDependencies: - '@babel/traverse' - '@nx/eslint' @@ -7341,7 +7411,7 @@ snapshots: optionalDependencies: '@nx/eslint': 22.7.12(@babel/traverse@7.29.10)(@swc-node/register@1.11.1(@swc/core@1.15.8(@swc/helpers@0.5.19))(@swc/types@0.1.28)(typescript@5.9.3))(@swc/core@1.15.8(@swc/helpers@0.5.19))(@zkochan/js-yaml@0.0.7)(eslint@10.13.0(jiti@2.6.1))(nx@22.7.12(@swc-node/register@1.11.1(@swc/core@1.15.8(@swc/helpers@0.5.19))(@swc/types@0.1.28)(typescript@5.9.3))(@swc/core@1.15.8(@swc/helpers@0.5.19)))(verdaccio@6.11.1) vite: 7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0) - vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0)(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) + vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0(@noble/hashes@1.8.0))(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) transitivePeerDependencies: - '@babel/traverse' - '@swc-node/register' @@ -7447,6 +7517,10 @@ snapshots: '@oxc-resolver/binding-win32-x64-msvc@11.24.2': optional: true + '@paralleldrive/cuid2@2.3.1': + dependencies: + '@noble/hashes': 1.8.0 + '@phenomnomnominal/tsquery@6.2.0(typescript@5.9.3)': dependencies: '@types/esquery': 1.5.4 @@ -7724,6 +7798,10 @@ snapshots: '@types/estree@1.0.9': {} + '@types/formidable@3.4.6': + dependencies: + '@types/node': 25.3.3 + '@types/html-minifier-terser@6.1.0': {} '@types/http-cache-semantics@4.2.0': {} @@ -8068,7 +8146,7 @@ snapshots: obug: 2.2.1 std-env: 4.3.0 tinyrainbow: 3.2.0 - vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0)(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) + vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0(@noble/hashes@1.8.0))(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) '@vitest/expect@4.1.12': dependencies: @@ -8114,7 +8192,7 @@ snapshots: sirv: 3.0.2 tinyglobby: 0.2.17 tinyrainbow: 3.2.0 - vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0)(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) + vitest: 4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0(@noble/hashes@1.8.0))(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) '@vitest/utils@4.1.12': dependencies: @@ -8412,6 +8490,8 @@ snapshots: array-union@2.1.0: {} + asap@2.0.6: {} + asn1@0.2.6: dependencies: safer-buffer: 2.1.2 @@ -8902,10 +8982,10 @@ snapshots: dependencies: assert-plus: 1.0.0 - data-urls@7.0.0: + data-urls@7.0.0(@noble/hashes@1.8.0): dependencies: whatwg-mimetype: 5.0.0 - whatwg-url: 16.0.1 + whatwg-url: 16.0.1(@noble/hashes@1.8.0) transitivePeerDependencies: - '@noble/hashes' @@ -8956,6 +9036,11 @@ snapshots: dependencies: address: 2.0.3 + dezalgo@1.0.4: + dependencies: + asap: 2.0.6 + wrappy: 1.0.2 + didyoumean@1.2.2: {} diff@8.0.4: {} @@ -9467,6 +9552,12 @@ snapshots: hasown: 2.0.4 mime-types: 2.1.35 + formidable@3.5.4: + dependencies: + '@paralleldrive/cuid2': 2.3.1 + dezalgo: 1.0.4 + once: 1.4.0 + forwarded@0.2.0: {} fraction.js@5.3.4: {} @@ -9620,9 +9711,9 @@ snapshots: dependencies: whatwg-encoding: 2.0.0 - html-encoding-sniffer@6.0.0: + html-encoding-sniffer@6.0.0(@noble/hashes@1.8.0): dependencies: - '@exodus/bytes': 1.16.0 + '@exodus/bytes': 1.16.0(@noble/hashes@1.8.0) transitivePeerDependencies: - '@noble/hashes' @@ -9789,6 +9880,8 @@ snapshots: is-interactive@1.0.0: {} + is-network-error@1.3.2: {} + is-number@7.0.0: {} is-potential-custom-element-name@1.0.1: {} @@ -9850,16 +9943,16 @@ snapshots: jsbn@0.1.1: {} - jsdom@28.1.0: + jsdom@28.1.0(@noble/hashes@1.8.0): dependencies: '@acemir/cssom': 0.9.32 '@asamuzakjp/dom-selector': 6.8.1 '@bramus/specificity': 2.4.3 - '@exodus/bytes': 1.16.0 + '@exodus/bytes': 1.16.0(@noble/hashes@1.8.0) cssstyle: 6.2.0 - data-urls: 7.0.0 + data-urls: 7.0.0(@noble/hashes@1.8.0) decimal.js: 10.6.0 - html-encoding-sniffer: 6.0.0 + html-encoding-sniffer: 6.0.0(@noble/hashes@1.8.0) http-proxy-agent: 7.0.2 https-proxy-agent: 7.0.6 is-potential-custom-element-name: 1.0.1 @@ -9871,7 +9964,7 @@ snapshots: w3c-xmlserializer: 5.0.0 webidl-conversions: 8.0.1 whatwg-mimetype: 5.0.0 - whatwg-url: 16.0.1 + whatwg-url: 16.0.1(@noble/hashes@1.8.0) xml-name-validator: 5.0.0 transitivePeerDependencies: - '@noble/hashes' @@ -10411,8 +10504,14 @@ snapshots: is-docker: 2.2.1 is-wsl: 2.2.0 + openapi-fetch@0.17.0: + dependencies: + openapi-typescript-helpers: 0.1.0 + openapi-types@12.1.3: {} + openapi-typescript-helpers@0.1.0: {} + opener@1.5.2: {} optionator@0.9.4: @@ -10465,6 +10564,10 @@ snapshots: dependencies: p-limit: 3.1.0 + p-retry@7.1.0: + dependencies: + is-network-error: 1.3.2 + package-manager-detector@1.9.0: {} pako@0.2.9: {} @@ -11438,6 +11541,8 @@ snapshots: undici-types@8.9.0: {} + undici@6.29.1: {} + undici@7.30.1: {} unicode-canonical-property-names-ecmascript@2.0.1: {} @@ -11608,7 +11713,7 @@ snapshots: tsx: 4.23.15 yaml: 2.9.0 - vitest@4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0)(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)): + vitest@4.1.12(@types/node@25.3.3)(@vitest/coverage-v8@4.1.12)(@vitest/ui@4.1.12)(happy-dom@21.0.4)(jsdom@28.1.0(@noble/hashes@1.8.0))(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.12 '@vitest/mocker': 4.1.12(vite@7.3.7(@types/node@25.3.3)(jiti@2.6.1)(terser@5.51.2)(tsx@4.23.15)(yaml@2.9.0)) @@ -11635,7 +11740,7 @@ snapshots: '@vitest/coverage-v8': 4.1.12(vitest@4.1.12) '@vitest/ui': 4.1.12(vitest@4.1.12) happy-dom: 21.0.4 - jsdom: 28.1.0 + jsdom: 28.1.0(@noble/hashes@1.8.0) transitivePeerDependencies: - msw @@ -11725,9 +11830,9 @@ snapshots: whatwg-mimetype@5.0.0: {} - whatwg-url@16.0.1: + whatwg-url@16.0.1(@noble/hashes@1.8.0): dependencies: - '@exodus/bytes': 1.16.0 + '@exodus/bytes': 1.16.0(@noble/hashes@1.8.0) tr46: 6.0.0 webidl-conversions: 8.0.1 transitivePeerDependencies: diff --git a/scripts/cloud/bootstrap.sh b/scripts/cloud/bootstrap.sh new file mode 100644 index 0000000..086426c --- /dev/null +++ b/scripts/cloud/bootstrap.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Controlled public-source smoke workload. No credentials or host mounts. +set -euo pipefail +export CI=true NEXT_TELEMETRY_DISABLED=1 NUXT_TELEMETRY_DISABLED=1 NX_DAEMON=false +export NODE_OPTIONS=--max-old-space-size=4096 RAYON_NUM_THREADS=2 +commit=${1:?exact source commit required} +[[ "$commit" =~ ^[a-f0-9]{40}$ ]] || { echo 'Invalid source commit' >&2; exit 2; } +root=/tmp/domscribe-cloud-smoke +report=/tmp/domscribe-cloud-evidence +# A new VM is required; never overwrite an earlier workload. +[[ ! -e "$root" && ! -e "$report" ]] || { echo 'Smoke workspace already exists' >&2; exit 2; } +mkdir -p "$root/tools" "$report" +cd "$root/tools" +if ! command -v node >/dev/null || [[ "$(node -p 'process.versions.node.split(".")[0]')" != 24 ]]; then + case "$(uname -m)" in x86_64) arch=x64;; aarch64|arm64) arch=arm64;; *) echo 'Unsupported architecture' >&2; exit 2;; esac + archive="node-v24.22.1-linux-$arch.tar.xz" + curl --fail --location --retry 2 --connect-timeout 15 --max-time 120 --output "$archive" "https://nodejs.org/dist/v24.22.1/$archive" + curl --fail --location --retry 2 --connect-timeout 15 --max-time 30 --output SHASUMS256.txt https://nodejs.org/dist/v24.22.1/SHASUMS256.txt + sha256sum --check --ignore-missing SHASUMS256.txt + tar --extract --file "$archive" + export PATH="$root/tools/node-v24.22.1-linux-$arch/bin:$PATH" +fi +npm install --prefix "$root/tools" --ignore-scripts --no-audit --no-fund pnpm@9.12.0 +export PATH="$root/tools/node_modules/.bin:$PATH" +git clone --no-checkout --filter=blob:none https://github.com/patchorbit/domscribe.git "$root/base" +git -C "$root/base" fetch --depth=1 origin "$commit" +git -C "$root/base" worktree add --detach "$root/subject" "$commit" +cd "$root/subject" +[[ "$(git rev-parse HEAD)" == "$commit" ]] +node --input-type=module - "$report/provenance.json" "$commit" <<'JS' +import fs from 'node:fs';import os from 'node:os'; +fs.writeFileSync(process.argv[2],JSON.stringify({commit:process.argv[3],node:process.version,platform:os.platform(),architecture:os.arch()},null,2)); +JS +pnpm install --frozen-lockfile +pnpm exec nx run-many -t build,sync-dist --exclude domscribe-test-fixtures --parallel=2 --skip-nx-cache +pnpm --filter @domscribe/test-fixtures exec playwright install --with-deps chromium +node scripts/verify-instance-replay.mjs --framework=react --package-manager=npm --vite-version=7.3.7 --report-dir="$report" +# Keep installation/onboarding scope honest: the subject lockfile must stay intact. +git diff --exit-code -- package.json pnpm-lock.yaml pnpm-workspace.yaml diff --git a/scripts/cloud/package.json b/scripts/cloud/package.json new file mode 100644 index 0000000..fd73b1b --- /dev/null +++ b/scripts/cloud/package.json @@ -0,0 +1,5 @@ +{ + "name": "domscribe-cloud-replay-smoke", + "private": true, + "type": "module" +} diff --git a/scripts/cloud/run.ts b/scripts/cloud/run.ts new file mode 100644 index 0000000..c65395d --- /dev/null +++ b/scripts/cloud/run.ts @@ -0,0 +1,128 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { createRequire } from 'node:module'; +import { fileURLToPath } from 'node:url'; +import { parseArgs } from 'node:util'; +import { oauth, pat, Sandboxes } from '@docker/sandboxes'; +import { DEFAULT_COMMIT, runCloudReplay, type CloudClient } from './runner.js'; + +const { values } = parseArgs({ + options: { + help: { type: 'boolean' }, + auth: { type: 'string', default: 'oauth' }, + commit: { type: 'string', default: DEFAULT_COMMIT }, + 'report-dir': { type: 'string' }, + }, +}); +if (values.help) { + console.log( + 'Usage: pnpm exec tsx scripts/cloud/run.ts [--auth=oauth|pat] [--commit=<40-character SHA>] [--report-dir=]\nRuns one medium Docker shell sandbox for up to30minutes. OAuth is interactive; PAT reads DOCKER_ID and DOCKER_PAT from the environment. No model-provider credential is needed.', + ); + process.exit(0); +} +if (!['oauth', 'pat'].includes(values.auth)) + throw Error('auth must be oauth or pat'); +if (!/^[a-f0-9]{40}$/.test(values.commit)) + throw Error('commit must be an exact lowercase40-character SHA'); +for (const name of [ + 'SANDBOXES_API_URL', + 'DOCKER_OAUTH_ISSUER', + 'DOCKER_TOKEN_EXCHANGE_URL', + 'DOCKER_GOVERNANCE_URL', +]) + if (process.env[name] !== undefined) + throw Error(`Unset ${name} before this production-service smoke test`); +const reportDir = values['report-dir'] + ? path.resolve(values['report-dir']) + : fs.mkdtempSync(path.join(os.tmpdir(), 'domscribe-cloud-replay-')); +fs.mkdirSync(reportDir, { recursive: true, mode: 0o700 }); +if (fs.readdirSync(reportDir).length !== 0) + throw Error('report-dir must be empty'); +console.log(`Evidence: ${reportDir}`); +const cancellation = new AbortController(); +for (const signal of ['SIGINT', 'SIGTERM'] as const) + process.once(signal, () => + cancellation.abort(new Error('Cloud run cancelled')), + ); +let client: Sandboxes | undefined; +try { + const auth = + values.auth === 'pat' + ? (() => { + const username = process.env.DOCKER_ID, + personalAccessToken = process.env.DOCKER_PAT; + if (!username || !personalAccessToken) + throw Error( + 'Set DOCKER_ID and DOCKER_PAT through your environment/secret store', + ); + return pat({ username, personalAccessToken }); + })() + : oauth({ + onVerification({ + verificationUriComplete, + verificationUri, + userCode, + }) { + console.error( + `Docker sign-in: ${verificationUriComplete ?? verificationUri}`, + ); + if (!verificationUriComplete) + console.error(`Verification code: ${userCode}`); + }, + }); + // Sign in before the separately bounded allocation/execution budget starts. + await auth.getAccessToken({ + signal: AbortSignal.any([ + cancellation.signal, + AbortSignal.timeout(5 * 60 * 1000), + ]), + }); + client = new Sandboxes({ auth, maxRetries: 1 }); + // Structural compatibility is checked against the installed public SDK types. + const api: CloudClient = client; + const sdkVersion = createRequire(import.meta.url)( + '@docker/sandboxes/package.json', + ).version as string; + const report = await runCloudReplay(api, { + commit: values.commit, + reportDir, + bootstrap: fs.readFileSync( + fileURLToPath(new URL('./bootstrap.sh', import.meta.url)), + 'utf8', + ), + sdkVersion, + signal: cancellation.signal, + onProgress: (phase) => console.log(`Cloud smoke: ${phase}`), + }); + console.log( + `Cloud smoke ${report.status}; sandbox deletion: ${report.cloud.deletion}`, + ); + if (report.failures.length) + console.error(JSON.stringify(report.failures, null, 2)); + process.exitCode = report.status === 'passed' ? 0 : 1; +} catch (error) { + // Do not serialize SDK error payloads, tokens or authentication transport data. + const e = error as { name?: unknown; httpStatus?: unknown }; + fs.writeFileSync( + path.join(reportDir, 'authentication-result.json'), + JSON.stringify( + { + status: 'failed', + cloudAllocationAttempted: !!client, + kind: typeof e?.name === 'string' ? e.name : 'Error', + httpStatus: + typeof e?.httpStatus === 'number' ? e.httpStatus : undefined, + }, + null, + 2, + ), + { mode: 0o600 }, + ); + console.error( + 'Cloud smoke could not start or finish; inspect the retained result metadata.', + ); + process.exitCode = 1; +} finally { + await client?.close(); +} diff --git a/scripts/cloud/runner.test.ts b/scripts/cloud/runner.test.ts new file mode 100644 index 0000000..bdc88f6 --- /dev/null +++ b/scripts/cloud/runner.test.ts @@ -0,0 +1,362 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { + runCloudReplay, + validateBrowserEvidence, + type CloudClient, + type CloudSandbox, +} from './runner.js'; +import type { CallOptions, KitLaunchOptions } from '@docker/sandboxes'; +const commit = '92e939a643b7f882b5f997e3197413befcc93b41'; +const names = [ + 'source ambiguity refused; exact instance has Lyra props', + 'exact editor instance exposes committed typed and checked state', + 'unmounted editor handle is stale after saving', + 'persisted annotation replayed in fresh Chromium context with reordered rows', + 'ambiguous replay stops before any action', + 'no application page errors', + 'production DOM has no source or instance instrumentation', + 'production UI still updates only the selected customer without page errors', +]; +function result() { + return { + framework: 'react', + packageManager: 'npm', + installMode: 'local-tarballs', + localArtifactOverrides: false, + node: 'v24.22.1', + installedVite: '7.3.7', + frameworkVersion: '18.3.1', + consumer: '/tmp/consumer', + mappedSource: '/tmp/consumer/src/main.jsx', + checks: names.map((name) => ({ name, pass: true })), + pageErrors: [], + consoleErrors: [], + networkErrors: [], + }; +} +interface FakeOptions { + eof?: boolean; + exitCode?: number; + readyError?: boolean; + transferError?: boolean; + missingResult?: boolean; + rpcNotFound?: boolean; + cleanupError?: boolean; + launchError?: boolean; + rejected?: boolean; + replacement?: boolean; + cancelDuringStream?: AbortController; +} +function fake(options: FakeOptions = {}) { + const calls: string[] = []; + let launchInput: KitLaunchOptions | undefined, + createdOptions: CallOptions | undefined, + cleanupSignal: AbortSignal | undefined, + executionSignal: AbortSignal | undefined; + const bytes = new Map([ + ['result.json', Buffer.from(JSON.stringify(result()))], + [ + 'provenance.json', + Buffer.from(JSON.stringify({ commit, node: 'v24.22.1' })), + ], + [ + 'trace.json', + Buffer.from( + JSON.stringify({ + version: 1, + steps: Array.from({ length: 4 }, () => ({ action: 'click' })), + }), + ), + ], + ['replayed.png', Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])], + ...['production-build.log', 'install.log', 'server.log'].map( + (n) => [n, Buffer.from('test log\n')] as const, + ), + ]); + const connection = { + async *[Symbol.asyncIterator]() { + yield { type: 'chunk', data: Buffer.from('remote log\n') }; + if (options.cancelDuringStream) { + options.cancelDuringStream.abort(); + executionSignal?.throwIfAborted(); + } + if (!options.eof) + yield { type: 'exited', exitCode: options.exitCode ?? 0 }; + }, + async close() { + calls.push('connection.close'); + }, + }; + const sandbox: CloudSandbox = { + name: 'sandboxes/owned', + uid: 'original-uid', + async waitUntilRunning() { + calls.push('ready'); + if (options.readyError) throw new Error('readiness failed'); + return sandbox; + }, + files: { + async write() { + calls.push('write'); + }, + async read(p: string) { + calls.push('read:' + path.posix.basename(p)); + if (options.transferError && p.endsWith('trace.json')) + throw Error('transfer failed'); + const name = path.posix.basename(p); + if (options.missingResult && name === 'result.json') + throw Object.assign(Error('not found'), { httpStatus: 404 }); + const b = bytes.get(name); + if (!b) + throw Object.assign( + Error('not found'), + options.rpcNotFound + ? { code: 'notFound', httpStatus: 200 } + : { httpStatus: 404 }, + ); + return b; + }, + }, + processes: { + async start() { + calls.push('start'); + return { + name: 'sandboxes/owned/processes/task', + async connect(_input, o) { + executionSignal = o.signal; + calls.push('connect'); + return connection; + }, + }; + }, + }, + async waitUntilDeleted() { + calls.push('deleted'); + }, + async delete(_input, o) { + calls.push('delete'); + cleanupSignal = o.signal; + if (options.cleanupError) throw Error('cleanup failed'); + return { + ...sandbox, + async waitUntilDeleted() { + calls.push('deleted'); + }, + }; + }, + }; + const client: CloudClient = { + kits: { + async launch(_kit, input, call) { + calls.push('launch'); + launchInput = input; + createdOptions = call; + if (options.launchError) + throw Object.assign( + Error('uncertain create'), + options.rejected + ? { + name: 'ValidationError', + httpStatus: 400, + code: 'invalidArgument', + violations: [ + { field: 'labels', description: 'Unsupported field' }, + ], + } + : {}, + ); + return sandbox; + }, + }, + async get() { + calls.push('get'); + return options.replacement + ? { ...sandbox, uid: 'replacement-uid' } + : sandbox; + }, + async close() { + calls.push('client.close'); + }, + }; + return { + client, + calls, + bytes, + get launchInput() { + return launchInput; + }, + get createdOptions() { + return createdOptions; + }, + get cleanupSignal() { + return cleanupSignal; + }, + }; +} +async function exercise( + f: ReturnType, + options: Record = {}, +) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cloud-run-test-')); + try { + return await runCloudReplay(f.client, { + commit, + reportDir: dir, + bootstrap: 'echo smoke', + ...options, + }); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} +test('process exit0 plus validated browser evidence and confirmed cleanup produce success', async () => { + const f = fake(); + const report = await exercise(f); + assert.equal(report.status, 'passed'); + assert.equal(report.cloud.deletion, 'confirmed'); + assert.equal(report.process.exitCode, 0); + assert(f.calls.includes('deleted')); + assert.equal(f.calls.at(-1), 'client.close'); + assert(f.launchInput && f.createdOptions); + assert.equal(f.launchInput.resources, 'medium'); + assert.equal( + f.launchInput.storage, + undefined, + 'omit stored-secrets feature when no secrets are used', + ); + assert.equal(f.launchInput.lifecycle?.onTimeout, 'delete'); + assert.equal(f.createdOptions.idempotencyKey, report.runId); +}); +for (const [name, options] of [ + ['readiness', { readyError: true }], + ['nonzero process', { exitCode: 2 }], + ['stream EOF', { eof: true }], + ['artifact transfer', { transferError: true }], + ['missing browser result', { missingResult: true }], +] as const) { + test(`cleans accepted sandbox after ${name} failure`, async () => { + const f = fake(options); + const report = await exercise(f); + assert.equal(report.status, 'failed'); + assert.equal(report.cloud.deletion, 'confirmed'); + assert(f.calls.includes('deleted')); + assert.equal(f.calls.at(-1), 'client.close'); + assert(f.cleanupSignal && !f.cleanupSignal.aborted); + }); +} +test('refuses to delete a replacement with the same name', async () => { + const f = fake({ replacement: true }); + const report = await exercise(f); + assert.equal(report.status, 'failed'); + assert.equal(report.cloud.deletion, 'unconfirmed'); + assert(!f.calls.includes('delete')); + assert.equal(f.calls.at(-1), 'client.close'); +}); +test('keeps command failure and cleanup failure distinct', async () => { + const f = fake({ exitCode: 2, cleanupError: true }); + const report = await exercise(f); + assert.equal(report.status, 'failed'); + assert(report.failures.some((x) => x.phase === 'run')); + assert(report.failures.some((x) => x.phase === 'cleanup')); + assert.equal(report.cloud.deletion, 'unconfirmed'); +}); +test('uncertain creation is not retried or reported as deleted', async () => { + const f = fake({ launchError: true }); + const report = await exercise(f); + assert.equal(report.status, 'failed'); + assert.equal(report.cloud.deletion, 'unknown_creation'); + assert.equal(f.calls.filter((c) => c === 'launch').length, 1); + assert.equal(f.calls.at(-1), 'client.close'); +}); +test('invalid commit fails before cloud allocation', async () => { + const f = fake(); + const report = await exercise(f, { commit: 'main; echo injected' }); + assert.equal(report.status, 'failed'); + assert(!f.calls.includes('launch')); +}); +test('exit0 cannot mask a failed browser assertion, stale provenance or app console error', () => { + const provenance = { commit, node: 'v24.22.1' }; + validateBrowserEvidence(result(), commit, provenance); + const bad = result(); + bad.checks[0].pass = false; + assert.throws(() => validateBrowserEvidence(bad, commit, provenance)); + assert.throws(() => + validateBrowserEvidence(result(), commit, { + ...provenance, + commit: 'a'.repeat(40), + }), + ); + const errors = result(); + errors.consoleErrors = ['unexpected'] as never; + assert.throws(() => validateBrowserEvidence(errors, commit, provenance)); + const missing = result(); + missing.checks.pop(); + assert.throws(() => validateBrowserEvidence(missing, commit, provenance)); +}); + +test('recognizes SDK notFound stream errors for optional failure screenshots', async () => { + const f = fake({ rpcNotFound: true }); + assert.equal((await exercise(f)).status, 'passed'); +}); +test('rejects oversized artifacts and still confirms sandbox deletion', async () => { + const f = fake(); + f.bytes.set('replayed.png', new Uint8Array(5 * 1024 * 1024)); + const report = await exercise(f); + assert.equal(report.status, 'failed'); + assert.equal(report.cloud.deletion, 'confirmed'); +}); +test('pre-cancelled work never allocates a sandbox', async () => { + const f = fake(); + const controller = new AbortController(); + controller.abort(); + const report = await exercise(f, { signal: controller.signal }); + assert.equal(report.status, 'failed'); + assert(!f.calls.includes('launch')); +}); +test('refuses a nonempty report directory without overwriting prior evidence', async () => { + const f = fake(); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cloud-run-test-')); + const p = path.join(dir, 'cloud-result.json'); + fs.writeFileSync(p, 'prior evidence'); + try { + const report = await runCloudReplay(f.client, { + commit, + reportDir: dir, + bootstrap: 'echo smoke', + }); + assert.equal(report.status, 'failed'); + assert.equal(fs.readFileSync(p, 'utf8'), 'prior evidence'); + assert(!f.calls.includes('launch')); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +test('records server validation fields and distinguishes rejected from uncertain allocation', async () => { + const f = fake({ launchError: true, rejected: true }); + const report = await exercise(f); + assert.equal(report.cloud.deletion, 'not_created'); + assert.equal(report.failures[0].code, 'invalidArgument'); + assert.deepEqual(report.failures[0].violations, [ + { field: 'labels', description: 'Unsupported field' }, + ]); +}); + +// The actual SDK streaming transport propagates its supplied abort signal. +// Model that behavior after acceptance, rather than only testing pre-cancel. +test('cancellation during streaming detaches and deletes the accepted sandbox with a fresh signal', async () => { + const controller = new AbortController(); + const f = fake({ cancelDuringStream: controller }); + const report = await exercise(f, { signal: controller.signal }); + assert.equal(report.status, 'failed'); + assert.equal(report.process.exitedObserved, false); + assert.equal(report.cloud.deletion, 'confirmed'); + assert(f.calls.indexOf('connection.close') < f.calls.indexOf('delete')); + assert(f.calls.includes('deleted')); + assert(controller.signal.aborted); + assert(f.cleanupSignal && !f.cleanupSignal.aborted); +}); diff --git a/scripts/cloud/runner.ts b/scripts/cloud/runner.ts new file mode 100644 index 0000000..51dc098 --- /dev/null +++ b/scripts/cloud/runner.ts @@ -0,0 +1,524 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { randomUUID } from 'node:crypto'; +import type { CallOptions, KitLaunchOptions } from '@docker/sandboxes'; + +export const SOURCE_REPOSITORY = 'https://github.com/patchorbit/domscribe.git'; +export const DEFAULT_COMMIT = '92e939a643b7f882b5f997e3197413befcc93b41'; +export const REMOTE_REPORT = '/tmp/domscribe-cloud-evidence'; +const SCRIPT = '/tmp/domscribe-cloud-bootstrap.sh'; +const REQUIRED_CHECKS = [ + 'source ambiguity refused; exact instance has Lyra props', + 'exact editor instance exposes committed typed and checked state', + 'unmounted editor handle is stale after saving', + 'persisted annotation replayed in fresh Chromium context with reordered rows', + 'ambiguous replay stops before any action', + 'no application page errors', + 'production DOM has no source or instance instrumentation', + 'production UI still updates only the selected customer without page errors', +]; +const ARTIFACTS = [ + 'result.json', + 'provenance.json', + 'trace.json', + 'replayed.png', + 'production-build.log', + 'install.log', + 'server.log', + ...Array.from({ length: 4 }, (_, i) => `failure-${i}.png`), +]; +const REQUIRED_ARTIFACTS = ARTIFACTS.slice(0, 7); +const MAX_FILE = 4 * 1024 * 1024, + MAX_TOTAL = 16 * 1024 * 1024, + MAX_OUTPUT = 8 * 1024 * 1024; +export interface CloudEvent { + type: string; + data?: Uint8Array; + exitCode?: number; +} +export interface CloudConnection extends AsyncIterable { + close(): Promise; +} +export interface CloudSandbox { + name: string; + uid: string; + waitUntilRunning(options: CallOptions): Promise; + files: { + write( + p: string, + content: string, + input: undefined, + options: CallOptions, + ): Promise; + read( + p: string, + input: { maxBytes: number }, + options: CallOptions, + ): Promise; + }; + processes: { + start( + input: { args: string[]; label: string; session: string }, + options: CallOptions, + ): Promise<{ + name: string; + connect(input: undefined, options: CallOptions): Promise; + }>; + }; + delete( + input: { force: boolean }, + options: CallOptions, + ): Promise; + waitUntilDeleted(options: CallOptions): Promise; +} +export interface CloudClient { + kits: { + launch( + name: string, + input: KitLaunchOptions, + options: CallOptions, + ): Promise; + }; + get(name: string, options: CallOptions): Promise; + close(): Promise; +} +export interface Failure { + phase: string; + kind: string; + message?: string; + httpStatus?: number; + code?: string; + violations?: Array<{ field?: string; description?: string }>; +} +export interface CloudRunReport { + schemaVersion: 1; + runId: string; + startedAt: string; + finishedAt?: string; + status: 'running' | 'passed' | 'failed'; + source: { repository: string; commit: string }; + sdkVersion?: string; + cloud: { + name?: string; + uid?: string; + resources: 'medium'; + ttlMs: number; + deletion: 'not_created' | 'unknown_creation' | 'unconfirmed' | 'confirmed'; + }; + process: { name?: string; exitCode?: number; exitedObserved: boolean }; + artifacts: string[]; + failures: Failure[]; + browser?: Record; +} +export interface RunOptions { + commit: string; + reportDir: string; + bootstrap: string; + sdkVersion?: string; + runId?: string; + signal?: AbortSignal; + onProgress?: (phase: string) => void; + timeoutMs?: number; + cleanupMs?: number; +} +class SmokeError extends Error { + override name = 'SmokeError'; +} +function ensure(value: unknown, message: string): asserts value { + if (!value) throw new SmokeError(message); +} +function object(value: unknown): Record { + ensure( + typeof value === 'object' && value !== null && !Array.isArray(value), + 'Expected evidence object', + ); + return value as Record; +} +function failure(phase: string, error: unknown): Failure { + const e = error as { + name?: unknown; + httpStatus?: unknown; + code?: unknown; + violations?: Array<{ field?: unknown; description?: unknown }>; + }; + return { + phase, + kind: + typeof e?.name === 'string' && /^[A-Za-z0-9_]{1,64}$/.test(e.name) + ? e.name + : 'Error', + ...(error instanceof SmokeError ? { message: error.message } : {}), + ...(e?.name === 'ValidationError' && error instanceof Error + ? { + message: error.message + .slice(0, 500) + .replace(/https?:\/\/\S+/g, '[URL]') + .replace(/Bearer\s+\S+/gi, 'Bearer [redacted]'), + } + : {}), + ...(typeof e?.httpStatus === 'number' ? { httpStatus: e.httpStatus } : {}), + ...(typeof e?.code === 'string' && /^[a-zA-Z]{1,64}$/.test(e.code) + ? { code: e.code } + : {}), + ...(e?.name === 'ValidationError' && Array.isArray(e.violations) + ? { + violations: e.violations.slice(0, 10).map((v) => ({ + field: + typeof v.field === 'string' ? v.field.slice(0, 160) : undefined, + description: + typeof v.description === 'string' + ? v.description.slice(0, 300) + : undefined, + })), + } + : {}), + }; +} +function isMissing(error: unknown) { + const e = error as { httpStatus?: number; code?: string }; + return e?.httpStatus === 404 || e?.code === 'notFound'; +} +function call(timeoutMs: number, signal?: AbortSignal): CallOptions { + return { + timeoutMs, + signal: signal + ? AbortSignal.any([signal, AbortSignal.timeout(timeoutMs)]) + : AbortSignal.timeout(timeoutMs), + maxRetries: 1, + }; +} +export function validateBrowserEvidence( + input: unknown, + commit: string, + provenanceInput: unknown, +): void { + const result = object(input), + provenance = object(provenanceInput); + ensure( + provenance.commit === commit, + 'Executed commit differs from requested commit', + ); + ensure( + typeof result.node === 'string' && + /^v24\./.test(result.node) && + provenance.node === result.node, + 'Expected the same Node24 runtime for build and browser verification', + ); + ensure(!result.error, 'Browser harness reported an error'); + ensure( + result.framework === 'react' && + result.packageManager === 'npm' && + result.installedVite === '7.3.7' && + result.frameworkVersion === '18.3.1', + 'Unexpected consumer integration/version', + ); + ensure( + result.installMode === 'local-tarballs' && + result.localArtifactOverrides === false, + 'Unexpected installation scope', + ); + ensure( + typeof result.consumer === 'string' && + typeof result.mappedSource === 'string' && + result.mappedSource.startsWith(result.consumer + '/') && + result.mappedSource.endsWith('/src/main.jsx'), + 'Missing exact consumer source mapping', + ); + const checks = result.checks; + ensure(Array.isArray(checks), 'Missing browser checks'); + ensure( + checks.every((c) => object(c).pass === true), + 'A browser check failed', + ); + for (const name of REQUIRED_CHECKS) + ensure( + checks.filter((c) => object(c).name === name).length === 1, + `Missing or duplicate required browser check: ${name}`, + ); + for (const name of ['pageErrors', 'consoleErrors', 'networkErrors']) + ensure( + Array.isArray(result[name]) && result[name].length === 0, + `Browser reported ${name}`, + ); +} + +export async function runCloudReplay( + client: CloudClient, + options: RunOptions, +): Promise { + const report: CloudRunReport = { + schemaVersion: 1, + runId: options.runId ?? randomUUID(), + startedAt: new Date().toISOString(), + status: 'running', + source: { repository: SOURCE_REPOSITORY, commit: options.commit }, + sdkVersion: options.sdkVersion, + cloud: { + resources: 'medium', + ttlMs: 30 * 60 * 1000, + deletion: 'not_created', + }, + process: { exitedObserved: false }, + artifacts: [], + failures: [], + }; + let owned: CloudSandbox | undefined, + running: CloudSandbox | undefined, + phase = 'input', + output: number | undefined, + writable = false; + const timeout = options.timeoutMs ?? 25 * 60 * 1000, + cleanupMs = options.cleanupMs ?? 60000; + const save = () => { + if (!writable) return; + fs.writeFileSync( + path.join(options.reportDir, 'cloud-result.json.tmp'), + JSON.stringify(report, null, 2), + { mode: 0o600 }, + ); + fs.renameSync( + path.join(options.reportDir, 'cloud-result.json.tmp'), + path.join(options.reportDir, 'cloud-result.json'), + ); + }; + const stage = (next: string) => { + phase = next; + save(); + options.onProgress?.(next); + }; + try { + ensure( + /^[a-f0-9]{40}$/.test(options.commit), + 'Use an exact lowercase40-character source commit', + ); + ensure( + timeout > 0 && + timeout <= 25 * 60 * 1000 && + cleanupMs > 0 && + cleanupMs <= 60000, + 'Deadlines exceed bounded smoke limits', + ); + ensure(/^[a-zA-Z0-9-]{1,64}$/.test(report.runId), 'Invalid run identity'); + fs.mkdirSync(options.reportDir, { recursive: true, mode: 0o700 }); + ensure( + fs.readdirSync(options.reportDir).length === 0, + 'Report directory must be empty', + ); + writable = true; + options.signal?.throwIfAborted(); + const execution = call(timeout, options.signal); + stage('launch'); + report.cloud.deletion = 'unknown_creation'; + save(); + owned = await client.kits.launch( + 'shell', + { + displayName: `domscribe-replay-${report.runId}`, + resources: 'medium', + lifecycle: { timeoutMs: report.cloud.ttlMs, onTimeout: 'delete' }, + labels: { domscribeRunId: report.runId }, + }, + { + ...execution, + timeoutMs: Math.min(timeout, 180000), + idempotencyKey: report.runId, + }, + ); + report.cloud = { + ...report.cloud, + name: owned.name, + uid: owned.uid, + deletion: 'unconfirmed', + }; + save(); + ensure( + owned.name && owned.uid, + 'Accepted sandbox lacks stable identity; cannot safely delete by name', + ); + stage('ready'); + running = await owned.waitUntilRunning({ + ...execution, + timeoutMs: Math.min(timeout, 180000), + }); + ensure( + running.name === owned.name && running.uid === owned.uid, + 'Readiness returned a different sandbox identity', + ); + stage('upload'); + await running.files.write(SCRIPT, options.bootstrap, undefined, execution); + stage('run'); + const task = await running.processes.start( + { + args: ['bash', SCRIPT, options.commit], + label: 'Domscribe browser replay smoke', + session: report.runId, + }, + { ...execution, idempotencyKey: report.runId + '-process' }, + ); + report.process.name = task.name; + save(); + const connection = await task.connect(undefined, execution); + output = fs.openSync( + path.join(options.reportDir, 'bootstrap.log'), + 'wx', + 0o600, + ); + let outputBytes = 0; + let streamError: unknown; + try { + for await (const event of connection) { + if (event.type === 'chunk' && event.data) { + outputBytes += event.data.byteLength; + ensure(outputBytes <= MAX_OUTPUT, 'Remote output exceeded8MiB limit'); + fs.writeSync(output, event.data); + } + if (event.type === 'exited') { + report.process.exitedObserved = true; + report.process.exitCode = event.exitCode ?? 0; + break; + } + } + } catch (error) { + streamError = error; + } finally { + try { + await connection.close(); + } catch (error) { + report.failures.push(failure('disconnect', error)); + } + fs.closeSync(output); + output = undefined; + } + if (streamError) throw streamError; + ensure( + report.process.exitedObserved, + 'Process stream ended before an exit event', + ); + ensure( + report.process.exitCode === 0, + `Remote command exited with code ${report.process.exitCode}`, + ); + } catch (error) { + const detail = failure(phase, error); + if ( + phase === 'launch' && + !owned && + detail.kind === 'ValidationError' && + detail.httpStatus === 400 && + detail.code === 'invalidArgument' + ) + report.cloud.deletion = 'not_created'; + report.failures.push(detail); + } finally { + if (output !== undefined) fs.closeSync(output); + if (running) { + phase = 'artifacts'; + const transfer = call(60000); + let total = 0; + for (const name of ARTIFACTS) { + try { + const bytes = await running.files.read( + `${REMOTE_REPORT}/${name}`, + { maxBytes: MAX_FILE }, + transfer, + ); + ensure( + bytes instanceof Uint8Array, + 'Artifact transfer did not return bytes', + ); + ensure(bytes.length <= MAX_FILE, 'Artifact exceeded4MiB limit'); + total += bytes.length; + ensure(total <= MAX_TOTAL, 'Artifacts exceeded16MiB total limit'); + fs.writeFileSync(path.join(options.reportDir, name), bytes, { + mode: 0o600, + }); + report.artifacts.push(name); + } catch (error) { + if (!isMissing(error)) { + report.failures.push(failure(phase, error)); + if (transfer.signal?.aborted) break; + } + } + } + if (report.failures.length === 0) { + try { + for (const name of REQUIRED_ARTIFACTS) + ensure( + report.artifacts.includes(name), + `Missing required artifact: ${name}`, + ); + const browser = JSON.parse( + fs.readFileSync( + path.join(options.reportDir, 'result.json'), + 'utf8', + ), + ); + validateBrowserEvidence( + browser, + options.commit, + JSON.parse( + fs.readFileSync( + path.join(options.reportDir, 'provenance.json'), + 'utf8', + ), + ), + ); + const trace = JSON.parse( + fs.readFileSync(path.join(options.reportDir, 'trace.json'), 'utf8'), + ); + ensure( + trace.version === 1 && + Array.isArray(trace.steps) && + trace.steps.length === 4, + 'Missing persisted four-step replay trace', + ); + ensure( + fs + .readFileSync(path.join(options.reportDir, 'replayed.png')) + .subarray(0, 8) + .equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])), + 'Replay screenshot is not PNG', + ); + report.browser = browser; + } catch (error) { + report.failures.push(failure('validate', error)); + } + } + } + if (owned) { + try { + const cleanup = call(cleanupMs); + ensure(owned.uid, 'Cannot confirm ownership without UID'); + const latest = await client.get(owned.name, cleanup); + ensure( + latest.uid === owned.uid && latest.name === owned.name, + 'Sandbox identity changed; refusing to delete replacement', + ); + const deleting = await latest.delete({ force: true }, cleanup); + if (deleting) { + ensure( + deleting.uid === owned.uid, + 'Deletion returned a different resource', + ); + await deleting.waitUntilDeleted(cleanup); + } + report.cloud.deletion = 'confirmed'; + } catch (error) { + report.failures.push(failure('cleanup', error)); + } + } + try { + await client.close(); + } catch (error) { + report.failures.push(failure('client.close', error)); + } + report.status = + report.failures.length === 0 && report.cloud.deletion === 'confirmed' + ? 'passed' + : 'failed'; + report.finishedAt = new Date().toISOString(); + // If the local output path is unusable, expose that local failure to the caller. + if (fs.existsSync(options.reportDir)) save(); + } + return report; +} diff --git a/scripts/cloud/tsconfig.cloud.json b/scripts/cloud/tsconfig.cloud.json new file mode 100644 index 0000000..31c9750 --- /dev/null +++ b/scripts/cloud/tsconfig.cloud.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "strict": true, + "noEmit": true, + "skipLibCheck": true, + "types": ["node"] + }, + "include": ["*.ts"] +}