Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,14 @@ jobs:
- name: Verify browser harness build cleanup
run: node --test scripts/fixtures/replay-process.test.mjs

- name: Verify Docker cloud runner offline
run: |
pnpm exec tsx --test scripts/cloud/*.test.ts
pnpm exec tsc -p scripts/cloud/tsconfig.cloud.json
pnpm exec eslint scripts/cloud/*.ts --max-warnings=0
bash -n scripts/cloud/bootstrap.sh
pnpm exec tsx scripts/cloud/run.ts --help

- run: pnpm exec nx run-many -t lint test build typecheck --exclude domscribe-test-fixtures

- name: Update coverage badge
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,9 @@ nx run-many -t build test lint typecheck

Conventions are in `.claude/rules/`. PRs welcome.

For an isolated browser replay check through the Docker Node SDK, see the
[Docker Cloud replay smoke test](./docs/docker-cloud-replay.md).

---

## License
Expand Down
63 changes: 63 additions & 0 deletions docs/docker-cloud-replay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Docker Cloud replay smoke test

This development runner uses the official [`@docker/sandboxes` Node SDK](https://docs.docker.com/ai/sandboxes-api/get-started/) to run Domscribe's existing browser verification in a disposable cloud sandbox. It checks that a saved annotation selects the intended customer after rows reorder in a fresh Chromium session, reads the correct source/props/state, and changes only that customer. It also checks stale and ambiguous targets, application errors, and production instrumentation removal.

The workload builds a pinned public Domscribe commit and installs its tarballs into a new React 18.3.1/npm/Vite 7.3.7 consumer. This is source-artifact verification; the unreleased APIs are not in npm 0.6.1. Other framework/browser combinations are covered separately by the portable-replay CI matrix. A cloud smoke run does not establish that an agent can fix arbitrary issues.

## Run

Use Node 24 and the repository's pnpm 9.12.0. From an isolated worktree:

```sh
pnpm install --frozen-lockfile
pnpm exec tsx scripts/cloud/run.ts --help
pnpm exec tsx scripts/cloud/run.ts
```

The default commit is the reviewed foundation at `92e939a643b7f882b5f997e3197413befcc93b41`. To verify another public commit containing the same harness:

```sh
pnpm exec tsx scripts/cloud/run.ts \
--commit=<exact-40-character-lowercase-SHA> \
--report-dir=/path/to/an/empty/evidence-directory
```

The default OAuth flow displays a Docker sign-in link. Authentication is handled by the SDK and held in memory for this process; Docker CLI login is separate. For unattended use, `--auth=pat` reads `DOCKER_ID` and `DOCKER_PAT` from the environment. Supply them through your secret store. Do not put credential values in command arguments, files, reports, or GitHub issue text. No model-provider credential is needed. The runner rejects SDK endpoint environment overrides before authentication.

One `medium` shell sandbox is requested with automatic deletion after 30 minutes. Execution is limited to 25 minutes, artifact collection to another minute, and explicit cleanup to another minute. Running it uses your Docker account's cloud resources. There is no automatic cloud run on a PR: CI only runs offline lifecycle tests, type checking, lint and shell syntax validation.

Inside the sandbox, the bootstrap creates an isolated worktree, installs dependencies with the frozen lockfile, builds/syncs artifacts without the Nx cache, and installs Chromium with its Linux dependencies. Node 24 is installed from the official checksum-verified distribution if needed. Host workspaces, credentials and secret files are not uploaded. The optional stored-secrets feature is omitted entirely when launching this workload.

## Evidence and failure handling

The CLI prints the local evidence directory. Keep these files together:

- `cloud-result.json`: requested commit, SDK version, accepted sandbox identity, process exit, browser validation and cleanup outcome.
- `bootstrap.log`: bounded streamed setup/build/browser output.
- `provenance.json`, `result.json`, `trace.json`, `replayed.png`: executed source/runtime, browser assertions, persisted four-step trace and replay screenshot.
- Consumer install, dev-server and production-build logs; available failure screenshots.

Success requires all eight named browser assertions, zero recorded application console/page/network errors, matching source/runtime provenance, the saved trace/screenshot, and confirmed deletion of the same sandbox UID. A zero process exit or closed connection alone cannot pass. The fixed evidence allowlist is limited to 4 MiB per file, 16 MiB total; streamed output is limited to 8 MiB. These limits fail the run rather than silently accepting incomplete evidence.

Cleanup runs after readiness, execution, artifact or validation failure with its own deadline, including cancellation with Ctrl-C. A same-name replacement with a different UID is never deleted. Both primary and cleanup failures are retained. `unknown_creation` means the launch response was lost before identity was obtained; `unconfirmed` means an accepted sandbox was not confirmed deleted. Inspect that run in Docker before starting another; the runner does not claim that client shutdown deleted a sandbox. A rejected validation request is recorded as `not_created`.

Reports and logs are private local evidence by default. Review them before sharing: the fixed public demo needs no credentials, but future application workloads may capture private UI data. This runner has no general-purpose secret scrubber and is not a boundary for arbitrary untrusted issue instructions.

## Verified cloud scope

On October 25, 2026, the actual SDK/service run completed in about two minutes on source `92e939a`: Node 24.22.1, npm 11.19.1, Chromium 156.0.8078.4, React 18.3.1 and Vite 7.3.7. All eight browser checks passed, all three application error collections were empty, evidence downloaded, and same-UID deletion was confirmed. This was one fresh controlled consumer, not an autonomous issue repair or a cloud framework matrix.

## Offline validation

These commands do not authenticate or allocate cloud resources:

```sh
pnpm exec tsx --test scripts/cloud/*.test.ts
pnpm exec tsc -p scripts/cloud/tsconfig.cloud.json
pnpm exec eslint scripts/cloud/*.ts --max-warnings=0
bash -n scripts/cloud/bootstrap.sh
```

The SDK is a private development dependency, pinned at 0.53.0. Its `undici` dependency is narrowly overridden to 6.29.1 to include the upstream security patch; it does not enter Domscribe's public package exports.

The next milestone is an agent attempting a realistic, deliberately broken UI task in this environment, with evidence that its fix changes the expected behavior. GitHub issue ingestion, repository write credentials and opening fix PRs are separate work after that experiment.
34 changes: 34 additions & 0 deletions docs/superpowers/plans/2026-10-25-docker-cloud-replay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Docker Cloud Replay Runner Plan

> **For agentic workers:** Use superpowers:executing-plans inline and obtain independent whole-branch review. Steps use checkboxes.

**Goal:** Execute the existing Domscribe browser replay verification through the Docker Cloud Sandboxes Node SDK, retaining evidence and cleaning up owned cloud resources reliably.

**Architecture:** A TypeScript development runner uses `@docker/sandboxes` directly. It launches one shell kit, clones a pinned public Domscribe commit into an isolated Git worktree, builds local artifacts and runs the existing browser harness. It downloads an allowlisted, bounded evidence bundle and records resource identity, process exit, browser result and confirmed deletion separately. A small orchestration module accepts the SDK-shaped client for deterministic lifecycle failure tests; live verification exercises the actual SDK/service.

**Tech Stack:** Node24, TypeScript/tsx, Docker SDK0.53.0, existing pnpm9/Playwright harness.

**Spec / user direction:** The approved next milestone in this conversation: SDK-driven shell-kit cloud smoke test first; coding-agent and GitHub issue wrapper follow after this works. User explicitly said to keep going while away. Choose routine implementation details autonomously.

## Constraints and review focus

- Reuse isolated worktree on codex/docker-cloud-replay from fresh main92e939a. Preserve primary repositories and unrelated lockfiles; root dev dependency/lock update is intentional for the SDK.
- Public repository and exact commit only. No model-provider credentials or production deployment/publication. Do not retrieve credential values; use SDK-managed interactive authentication or existing explicit environment credentials without logging values.
- One owned sandbox at a time; bound compute/lifetime, launch/setup/run/download/cleanup deadlines and retries. Save identity immediately after accepted creation; never delete a replacement resource with the same name.
- On command, readiness, transfer, result-validation or cleanup failure, retain useful local evidence and return failure. Connection EOF or exit0 alone is not browser success.
- Download fixed evidence names under a fixed remote directory, reject invalid results/oversized files, avoid logging arbitrary secrets or copying repository credentials.
- Clean up after failures using an independent deadline; accurately record unconfirmed cleanup. Client.close is separate from sandbox deletion. Preserve original error when cleanup also fails.
- Local/CI unit, type and shell-script checks run without Docker credentials/cloud charges. Live smoke is explicit, uses a shell kit, and records actual tested scope. No full issue-to-PR Action implementation in this change.

## Task 1: Runner and lifecycle regressions

- [x] Add private SDK dev dependency and inspect installed public types/examples for exact APIs.
- [x] Write meaningful failing regressions: process failure/EOF, missing or failed browser result, cleanup on readiness/command/download failure, UID mismatch, cleanup failure preserving primary cause, bounded artifact transfer. Observe RED before implementing.
- [x] Implement minimal runner, fixed pinned bootstrap script, evidence validation and safe lifecycle cleanup; make tests pass and typecheck against installed SDK.

## Task 2: Real verification and delivery

- [x] Run configured affected checks, process regression tests, shell syntax and representative local command/packaging validation.
- [x] Attempt a fresh SDK cloud run using available approved authentication; exercise genuine Chromium/relay/selection/state/replay/production flow and retain artifacts. If account consent/login is unavailable while user is away, finish all independent work and report the exact blind spot without claiming cloud success.
- [ ] Obtain independent review, fix material findings with regressions, update docs, push a focused PR and inspect CI. Respect merge protections; earlier admin-bypass approvals were PR-specific.
- [ ] Update development-state.md and daily automation continuity with exact results, cloud cost/resource cleanup status and the next useful step.
4 changes: 3 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
},
"private": true,
"devDependencies": {
"@docker/sandboxes": "0.53.0",
"@eslint/js": "^10.0.1",
"@nx/devkit": "22.7.12",
"@nx/eslint": "22.7.12",
Expand Down Expand Up @@ -135,7 +136,8 @@
"nx>follow-redirects": "^1.16.1",
"nx>brace-expansion": "^5.0.12",
"nx>smol-toml": "^1.9.1",
"postcss-selector-parser@<7.1.6": "^7.1.7"
"postcss-selector-parser@<7.1.6": "^7.1.7",
"@docker/sandboxes@0.53.0>undici": "6.29.1"
}
}
}
Loading
Loading