Add action-plane modes (computer/browser/hybrid) and Anthropic native computer/browser tools#51
Merged
Conversation
added 4 commits
July 8, 2026 16:49
- CuaAgent.setMode / CuaAgentHarness.setMode switch action planes at runtime (rejected when it conflicts with a configured nativeTool) - /mode <os|dom|hybrid> TUI slash command with autocomplete - region schemas use bounded arrays instead of Type.Tuple: tuples emit draft-07 `items: [...]`, which Anthropic's draft 2020-12 validation rejects (found via live API) - anthropic-native-smoke example covering the mode / native-tool matrix
os → computer, dom → browser (hybrid unchanged). The mode names now pair directly with the native tools: computer_20260601 ⇢ mode computer, browser_20260701 ⇢ mode browser. Plane files, action-set constants, types, the page executor, CLI flag values, and the /mode command follow suit. Hybrid tool prefixes stay computer_* / page_*. Re-verified live: canonical browser mode and native browser_20260701 against a Kernel browser after the rename.
Hybrid mode now exposes computer_* and browser_* tools, matching the mode names and Anthropic's tool vocabulary. Browser mode still strips the prefix. The CDP executor is BrowserExecutor (translator/browser.ts). Live-verified hybrid mode post-rename: model mixed browser_navigate, browser_snapshot, browser_click (by ref), and computer_wait.
rgarcia
changed the title
rgarcia
marked this pull request as ready for review
July 8, 2026 18:50
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 5 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 5 issues found in the latest run.
- ✅ Fixed: CDP close leaves pending hangs
- The socket close path now shares a disconnect handler that clears state and rejects all pending CDP command promises with a closed-connection error.
- ✅ Fixed: setMode ignores active tool subset
- CuaAgentHarness.setMode now reuses requestedActiveToolNames just like setModel so caller-selected active tools are preserved during mode changes.
- ✅ Fixed: javascriptExec ignored for native browser
- resolveCuaRuntimeSpec now folds javascriptExec into browser_20260701 native specs by defaulting enable_javascript_exec to true when requested.
- ✅ Fixed: new_tab leaves prior tab active
- BrowserExecutor.newTab now updates activeTargetId to the newly created target so follow-up actions without tab_id run on that new tab.
- ✅ Fixed: Viewport crop ignores reversed coords
- BrowserExecutor.screenshot now normalizes region coordinates using min/abs bounds so reversed corners crop the intended viewport area.
Or push these changes by commenting:
@cursor push c061091489
Preview (c061091489)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -476,7 +476,7 @@
async setMode(mode: CuaMode): Promise<void> {
this.runtime.setMode(mode);
const tools = this.runtime.tools();
- await super.setTools(tools, tools.map((tool) => tool.name));
+ await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name));
}
/** The action plane(s) currently exposed to the model. */
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -103,8 +103,12 @@
async screenshot(region?: [number, number, number, number], tabId?: string): Promise<{ data: Buffer; mimeType: string }> {
const session = await this.session(tabId);
+ const x0 = region ? Math.min(region[0], region[2]) : 0;
+ const y0 = region ? Math.min(region[1], region[3]) : 0;
+ const width = region ? Math.max(1, Math.abs(region[2] - region[0])) : 0;
+ const height = region ? Math.max(1, Math.abs(region[3] - region[1])) : 0;
const clip = region
- ? { clip: { x: region[0], y: region[1], width: Math.max(1, region[2] - region[0]), height: Math.max(1, region[3] - region[1]), scale: 1 } }
+ ? { clip: { x: x0, y: y0, width, height, scale: 1 } }
: {};
const { data } = await this.cdp.send<{ data: string }>("Page.captureScreenshot", { format: "png", ...clip }, session);
return { data: Buffer.from(data, "base64"), mimeType: "image/png" };
@@ -305,6 +309,7 @@
private async newTab(): Promise<string> {
const targetId = await this.cdp.createTarget("about:blank");
+ this.activeTargetId = targetId;
return `Opened tab_id ${shortTabId(targetId)}.\n${await this.tabContext(targetId)}`;
}
diff --git a/packages/agent/src/translator/cdp.ts b/packages/agent/src/translator/cdp.ts
--- a/packages/agent/src/translator/cdp.ts
+++ b/packages/agent/src/translator/cdp.ts
@@ -74,12 +74,7 @@
close(): void {
this.socket?.close();
- this.socket = undefined;
- this.opening = undefined;
- this.sessionsByTarget.clear();
- const error = new Error("CDP connection closed");
- for (const pending of this.pending.values()) pending.reject(error);
- this.pending.clear();
+ this.handleDisconnect();
}
private connect(): Promise<WebSocket> {
@@ -95,9 +90,7 @@
reject(new Error(`CDP connection to ${this.wsUrl} failed`));
});
socket.addEventListener("close", () => {
- this.socket = undefined;
- this.opening = undefined;
- this.sessionsByTarget.clear();
+ this.handleDisconnect();
});
socket.addEventListener("message", (event) => {
this.handleMessage(typeof event.data === "string" ? event.data : String(event.data));
@@ -132,4 +125,13 @@
for (const listener of this.listeners) listener(event);
}
}
+
+ private handleDisconnect(): void {
+ this.socket = undefined;
+ this.opening = undefined;
+ this.sessionsByTarget.clear();
+ const error = new Error("CDP connection closed");
+ for (const pending of this.pending.values()) pending.reject(error);
+ this.pending.clear();
+ }
}
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -474,6 +474,20 @@
expect(harness.getActiveTools()).toEqual([]);
});
+ it("preserves active tool selection when setMode refreshes tools", async () => {
+ const harness = new CuaAgentHarness({
+ ...(await createHarnessServices()),
+ browser,
+ client,
+ model: "anthropic:claude-opus-4-5",
+ });
+
+ await harness.setActiveTools([]);
+ await harness.setMode("hybrid");
+
+ expect(harness.getActiveTools()).toEqual([]);
+ });
+
it("re-applies the requested active tool subset and persists it when setModel refreshes tools", async () => {
const { env, session } = await createHarnessServices();
const harness = new CuaAgentHarness({
diff --git a/packages/agent/test/browser-executor.test.ts b/packages/agent/test/browser-executor.test.ts
new file mode 100644
--- /dev/null
+++ b/packages/agent/test/browser-executor.test.ts
@@ -1,0 +1,52 @@
+import { describe, expect, it, vi } from "vitest";
+import { BrowserExecutor } from "../src/translator/browser";
+import type { CdpTargetInfo } from "../src/translator/cdp";
+
+function makeTarget(targetId: string, title: string, url: string): CdpTargetInfo {
+ return { targetId, type: "page", title, url };
+}
+
+class FakeCdpConnection {
+ targets: CdpTargetInfo[] = [makeTarget("target-1", "Tab 1", "https://one.example")];
+ pageTargets = vi.fn(async () => this.targets);
+ attachToTarget = vi.fn(async (targetId: string) => `session-${targetId}`);
+ createTarget = vi.fn(async (url: string) => {
+ const targetId = `target-${this.targets.length + 1}`;
+ this.targets.push(makeTarget(targetId, "New Tab", url));
+ return targetId;
+ });
+ send = vi.fn(async (method: string, params: Record<string, unknown>, _sessionId?: string) => {
+ if (method === "Page.captureScreenshot") {
+ return { data: Buffer.from("png").toString("base64"), ...params };
+ }
+ throw new Error(`unexpected CDP method ${method}`);
+ });
+}
+
+describe("BrowserExecutor", () => {
+ it("makes browser_new_tab the active target for follow-up actions", async () => {
+ const cdp = new FakeCdpConnection();
+ const executor = new BrowserExecutor(cdp as never);
+
+ await executor.execute({ type: "browser_new_tab" });
+ await executor.execute({ type: "browser_screenshot" });
+
+ expect(cdp.attachToTarget).toHaveBeenLastCalledWith("target-2");
+ });
+
+ it("normalizes screenshot regions when coordinates are reversed", async () => {
+ const cdp = new FakeCdpConnection();
+ const executor = new BrowserExecutor(cdp as never);
+
+ await executor.screenshot([80, 60, 20, 10]);
+
+ expect(cdp.send).toHaveBeenCalledWith(
+ "Page.captureScreenshot",
+ {
+ format: "png",
+ clip: { x: 20, y: 10, width: 60, height: 50, scale: 1 },
+ },
+ "session-target-1",
+ );
+ });
+});
diff --git a/packages/agent/test/cdp.test.ts b/packages/agent/test/cdp.test.ts
new file mode 100644
--- /dev/null
+++ b/packages/agent/test/cdp.test.ts
@@ -1,0 +1,61 @@
+import { describe, expect, it, vi } from "vitest";
+import { CdpConnection } from "../src/translator/cdp";
+
+type Listener = (event?: unknown) => void;
+
+class FakeWebSocket {
+ static CONNECTING = 0;
+ static OPEN = 1;
+ static CLOSING = 2;
+ static CLOSED = 3;
+ static instances: FakeWebSocket[] = [];
+
+ readonly sent: string[] = [];
+ readyState = FakeWebSocket.CONNECTING;
+ private readonly listeners = new Map<string, Listener[]>();
+
+ constructor(_url: string) {
+ FakeWebSocket.instances.push(this);
+ queueMicrotask(() => {
+ this.readyState = FakeWebSocket.OPEN;
+ this.emit("open");
+ });
+ }
+
+ addEventListener(type: string, listener: Listener): void {
+ const listeners = this.listeners.get(type) ?? [];
+ listeners.push(listener);
+ this.listeners.set(type, listeners);
+ }
+
+ send(data: string): void {
+ this.sent.push(data);
+ }
+
+ close(): void {
+ this.readyState = FakeWebSocket.CLOSED;
+ this.emit("close");
+ }
+
+ private emit(type: string): void {
+ for (const listener of this.listeners.get(type) ?? []) listener();
+ }
+}
+
+describe("CdpConnection", () => {
+ it("rejects pending commands when the socket closes unexpectedly", async () => {
+ FakeWebSocket.instances = [];
+ vi.stubGlobal("WebSocket", FakeWebSocket as unknown as typeof WebSocket);
+ try {
+ const cdp = new CdpConnection("ws://example.test/devtools");
+ const pending = cdp.send("Runtime.evaluate", { expression: "1 + 1" });
+ await vi.waitFor(() => expect(FakeWebSocket.instances[0]?.sent).toHaveLength(1));
+
+ FakeWebSocket.instances[0]!.close();
+
+ await expect(pending).rejects.toThrow("CDP connection closed");
+ } finally {
+ vi.unstubAllGlobals();
+ }
+ });
+});
diff --git a/packages/ai/src/runtime-spec.ts b/packages/ai/src/runtime-spec.ts
--- a/packages/ai/src/runtime-spec.ts
+++ b/packages/ai/src/runtime-spec.ts
@@ -51,7 +51,13 @@
const mode = options.mode ?? (options.nativeTool ? modeForNativeTool(options.nativeTool) : "computer");
if (options.nativeTool) {
- const nativeTool = resolveNativeTool(options.nativeTool, model, mode);
+ const nativeToolSpec =
+ options.javascriptExec &&
+ options.nativeTool.type === "browser_20260701" &&
+ options.nativeTool.enable_javascript_exec === undefined
+ ? { ...options.nativeTool, enable_javascript_exec: true }
+ : options.nativeTool;
+ const nativeTool = resolveNativeTool(nativeToolSpec, model, mode);
const nativeModel: Model<Api> = { ...model, api: nativeApiForToolType(nativeTool.spec.type) as Model<Api>["api"] };
const executors = nativeToolExecutors(nativeTool);
return {
diff --git a/packages/ai/test/native-tools.test.ts b/packages/ai/test/native-tools.test.ts
--- a/packages/ai/test/native-tools.test.ts
+++ b/packages/ai/test/native-tools.test.ts
@@ -56,6 +56,18 @@
expect(spec.toolDefinitions.map((tool) => tool.name)).toEqual(["browser"]);
});
+ it("threads javascriptExec into browser_20260701 declarations", async () => {
+ const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", {
+ nativeTool: { type: "browser_20260701" },
+ javascriptExec: true,
+ });
+ const payload = {
+ tools: [{ name: "browser", description: "placeholder", input_schema: {} }],
+ };
+ const next = (await spec.onPayload?.(payload, spec.model)) as { tools: Array<Record<string, unknown>> };
+ expect(next.tools[0]).toEqual({ type: "browser_20260701", name: "browser", enable_javascript_exec: true });
+ });
+
it("swaps the placeholder tool for the native declaration in the payload", async () => {
const spec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "computer_20260601", enable_zoom: true } });
const payload = {You can send follow-ups to the cloud agent here.
The early-access PDF documented computer_20260601 with beta header computer-use-2026-06-01, but the API ships computer_20260701 behind computer-use-2026-07-01 (the 0601 tag is not accepted under any header). Live-verified end-to-end against a Kernel browser.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.
There are 9 total unresolved issues (including 5 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for all 4 issues found in the latest run.
- ✅ Fixed: Mode switch leaks CDP sockets
- Mode/model switches now dispose the previous translator/browser executor before creating a replacement, closing any open CDP socket.
- ✅ Fixed: Setup failure leaks browser
- setupHarnessRuntime now parses mode/native/thinking before provisioning and closes the provisioned browser handle on any setup failure path.
- ✅ Fixed: Invalid mode exits one
- Invalid mode/native/thinking values now throw a CLI usage error type that print and interactive entrypoints map to exit code 2.
- ✅ Fixed: Browser mode mixed screenshot frames
- Initial prompt screenshots now use browser-viewport capture in browser mode (and OS capture otherwise), matching post-action screenshot framing.
Or push these changes by commenting:
@cursor push af8bbd4d8d
Preview (af8bbd4d8d)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -175,7 +175,7 @@
setMode(mode: CuaMode): void {
this.runtimeSpec = this.resolveSpec(this.runtimeSpec.model, mode);
this.currentMode = mode;
- this.translator = this.createTranslator();
+ this.replaceTranslator();
}
get systemPrompt(): string {
@@ -184,7 +184,7 @@
setModel(model: CuaRuntimeInput): void {
this.runtimeSpec = this.resolveSpec(model);
- this.translator = this.createTranslator();
+ this.replaceTranslator();
}
tools(): AgentTool[] {
@@ -230,6 +230,11 @@
screenshot: this.runtimeSpec.screenshot,
});
}
+
+ private replaceTranslator(): void {
+ this.translator.close();
+ this.translator = this.createTranslator();
+ }
}
/** Default stream path: the shared CUA `Models` collection. */
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -54,6 +54,10 @@
constructor(private readonly cdp: CdpConnection) {}
+ close(): void {
+ this.cdp.close();
+ }
+
async execute(action: CuaBrowserAction): Promise<BatchReadResult[]> {
switch (action.type) {
case "browser_snapshot":
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -65,6 +65,11 @@
return this.browserExecutor;
}
+ close(): void {
+ this.browserExecutor?.close();
+ this.browserExecutor = undefined;
+ }
+
async screenshotRaw(): Promise<Buffer> {
return (await this.screenshot()).data;
}
diff --git a/packages/cli/src/action/harness-runner.ts b/packages/cli/src/action/harness-runner.ts
--- a/packages/cli/src/action/harness-runner.ts
+++ b/packages/cli/src/action/harness-runner.ts
@@ -2,7 +2,7 @@
import type { AssistantMessage, ImageContent } from "@onkernel/cua-ai";
import { writeFile } from "node:fs/promises";
import { stderr, stdout } from "node:process";
-import { captureScreenshot, type CuaBrowserHandle } from "../harness-browser";
+import { captureModeScreenshot, captureScreenshot, type CuaBrowserHandle } from "../harness-browser";
import { type ActionRequest, buildPrompt, DEFAULT_MAX_TURNS } from "./prompts";
import { type ActionEventInfo, type ActionResult, exitCodeFor, formatCompact, parseResult } from "./result";
@@ -142,7 +142,11 @@
if (opts.skipInitialScreenshot) return undefined;
const hasPriorTurn = await sessionHasPriorTurn(opts.session);
if (hasPriorTurn) return undefined;
- const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id);
+ const png = await captureModeScreenshot(
+ opts.browserHandle.client,
+ opts.browserHandle.browser,
+ opts.harness.getMode(),
+ );
if (!png) return undefined;
return [{ type: "image", data: png.toString("base64"), mimeType: "image/png" }];
}
diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts
--- a/packages/cli/src/cli-harness.ts
+++ b/packages/cli/src/cli-harness.ts
@@ -202,6 +202,12 @@
modelRef: CuaModelRef;
}
+class CliUsageError extends Error {}
+
+export function isCliUsageError(err: unknown): err is CliUsageError {
+ return err instanceof CliUsageError;
+}
+
function requireKernelApiKey(): { apiKey: string; baseUrl?: string } {
const apiKey = process.env.KERNEL_API_KEY?.trim();
if (!apiKey) throw new Error("missing Kernel API key (set KERNEL_API_KEY)");
@@ -376,66 +382,76 @@
extraPaths: flags.skillPaths,
disabled: flags.noSkills,
});
+ const mode = parseMode(flags.mode);
+ const nativeTool = parseNativeTool(flags.nativeTool, flags.jsExec);
+ const thinkingLevel = mapThinkingLevel(flags.thinking);
const provisioned = await provisionForFlags(flags, auth);
- const repo = createSessionRepo(flags.sessionDir);
+ try {
+ const repo = createSessionRepo(flags.sessionDir);
+ const skipDisk = opts.skipDiskSession === true && !hasExplicitSessionFlag(flags);
+ const resolved = skipDisk ? undefined : await resolveSession(repo, cwd, flags, provisioned.named);
- const skipDisk = opts.skipDiskSession === true && !hasExplicitSessionFlag(flags);
- const resolved = skipDisk ? undefined : await resolveSession(repo, cwd, flags, provisioned.named);
+ let inMemorySession: Session | undefined;
+ if (!resolved) {
+ const memRepo = new InMemorySessionRepo();
+ inMemorySession = await memRepo.create();
+ }
- let inMemorySession: Session | undefined;
- if (!resolved) {
- const memRepo = new InMemorySessionRepo();
- inMemorySession = await memRepo.create();
- }
+ const session = resolved?.session ?? inMemorySession!;
+ const { provider } = parseCuaModelRef(auth.modelRef);
- const session = resolved?.session ?? inMemorySession!;
- const { provider } = parseCuaModelRef(auth.modelRef);
+ if (resolved) {
+ await appendBrowserEntry(session, {
+ sessionId: provisioned.handle.browser.session_id,
+ liveUrl: provisioned.handle.browser.browser_live_view_url,
+ profileId: provisioned.handle.profileId,
+ createdAt: Date.now(),
+ });
+ if (provisioned.named) {
+ await recordTranscriptPath(provisioned.named.name, resolved.transcriptPath);
+ }
+ if (flags.verbose) {
+ stderr.write(`[cua] session=${resolved.transcriptPath}\n`);
+ if (resolved.resumed) stderr.write("[cua] resumed prior session into fresh browser\n");
+ }
+ }
- if (resolved) {
- await appendBrowserEntry(session, {
- sessionId: provisioned.handle.browser.session_id,
- liveUrl: provisioned.handle.browser.browser_live_view_url,
- profileId: provisioned.handle.profileId,
- createdAt: Date.now(),
+ const baseUrlOverride = providerBaseUrlOverride(provider);
+ const harness = buildCuaHarness({
+ cwd,
+ client: provisioned.handle.client,
+ browser: provisioned.handle.browser,
+ session,
+ model: auth.modelRef,
+ skills,
+ contextFiles,
+ thinkingLevel,
+ mode,
+ nativeTool,
+ javascriptExec: flags.jsExec,
+ playwright: flags.playwright,
+ modelBaseUrl: baseUrlOverride,
});
- if (provisioned.named) {
- await recordTranscriptPath(provisioned.named.name, resolved.transcriptPath);
+
+ return {
+ handle: provisioned.handle,
+ resolved,
+ session,
+ skills,
+ contextFiles,
+ harness,
+ provider,
+ modelRef: auth.modelRef,
+ };
+ } catch (err) {
+ try {
+ await provisioned.handle.close();
+ } catch {
+ // Preserve the original setup error.
}
- if (flags.verbose) {
- stderr.write(`[cua] session=${resolved.transcriptPath}\n`);
- if (resolved.resumed) stderr.write("[cua] resumed prior session into fresh browser\n");
- }
+ throw err;
}
-
- const thinkingLevel = mapThinkingLevel(flags.thinking);
- const baseUrlOverride = providerBaseUrlOverride(provider);
- const harness = buildCuaHarness({
- cwd,
- client: provisioned.handle.client,
- browser: provisioned.handle.browser,
- session,
- model: auth.modelRef,
- skills,
- contextFiles,
- thinkingLevel,
- mode: parseMode(flags.mode),
- nativeTool: parseNativeTool(flags.nativeTool, flags.jsExec),
- javascriptExec: flags.jsExec,
- playwright: flags.playwright,
- modelBaseUrl: baseUrlOverride,
- });
-
- return {
- handle: provisioned.handle,
- resolved,
- session,
- skills,
- contextFiles,
- harness,
- provider,
- modelRef: auth.modelRef,
- };
}
function hasExplicitSessionFlag(flags: HarnessCliFlags): boolean {
@@ -457,7 +473,7 @@
if (raw === undefined) return undefined;
const value = raw.trim().toLowerCase();
if (value === "computer" || value === "browser" || value === "hybrid") return value;
- throw new Error(`invalid --mode value "${raw}"; expected one of: computer | browser | hybrid`);
+ throw new CliUsageError(`invalid --mode value "${raw}"; expected one of: computer | browser | hybrid`);
}
function parseNativeTool(raw: string | undefined, jsExec: boolean | undefined): CuaNativeToolSpec | undefined {
@@ -467,7 +483,7 @@
// visual targeting; the executor implements the zoom crop locally.
if (value === "computer_20260701") return { type: "computer_20260701", enable_zoom: true };
if (value === "browser_20260701") return { type: "browser_20260701", ...(jsExec ? { enable_javascript_exec: true } : {}) };
- throw new Error(`invalid --native-tool value "${raw}"; expected one of: computer_20260701 | browser_20260701`);
+ throw new CliUsageError(`invalid --native-tool value "${raw}"; expected one of: computer_20260701 | browser_20260701`);
}
function mapThinkingLevel(raw: string | undefined): "off" | "minimal" | "low" | "medium" | "high" | "xhigh" {
@@ -488,7 +504,7 @@
case "":
return "low";
default:
- throw new Error(
+ throw new CliUsageError(
`invalid --thinking value "${raw}"; expected one of: off | minimal | low | medium | high | xhigh`,
);
}
@@ -528,8 +544,8 @@
flags: HarnessCliFlags,
): Promise<number> {
const runtime = await setupHarnessRuntime(flags);
- const { runInteractive } = await import("./tui/main");
try {
+ const { runInteractive } = await import("./tui/main");
return await runInteractive({
cwd: process.cwd(),
harness: runtime.harness,
diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts
--- a/packages/cli/src/cli.ts
+++ b/packages/cli/src/cli.ts
@@ -3,6 +3,7 @@
import { parseArgs } from "node:util";
import { type ActionType } from "./action/prompts";
import {
+ isCliUsageError,
runActionCommand,
runInteractiveCommand,
runModelsSubcommand as runModelsSubcommandHarness,
@@ -301,7 +302,7 @@
return await runPrintCommand(prompt, toHarnessFlags(flags));
} catch (err) {
stderr.write(`error: ${(err as Error).message}\n`);
- return 1;
+ return isCliUsageError(err) ? 2 : 1;
}
}
@@ -309,7 +310,7 @@
return await runInteractiveCommand(prompt, toHarnessFlags(flags));
} catch (err) {
stderr.write(`error: ${(err as Error).message}\n`);
- return 1;
+ return isCliUsageError(err) ? 2 : 1;
}
}
diff --git a/packages/cli/src/harness-browser.ts b/packages/cli/src/harness-browser.ts
--- a/packages/cli/src/harness-browser.ts
+++ b/packages/cli/src/harness-browser.ts
@@ -1,4 +1,4 @@
-import type { KernelBrowser } from "@onkernel/cua-agent";
+import { BrowserExecutor, CdpConnection, type KernelBrowser } from "@onkernel/cua-agent";
import Kernel, { NotFoundError } from "@onkernel/sdk";
/** Plain SDK-backed Kernel browser handle for the new harness wiring. */
@@ -100,3 +100,21 @@
return undefined;
}
}
+
+/** Capture first-prompt grounding using the active mode's screenshot frame. */
+export async function captureModeScreenshot(
+ client: Kernel,
+ browser: KernelBrowser,
+ mode: "computer" | "browser" | "hybrid",
+): Promise<Buffer | undefined> {
+ if (mode !== "browser") return captureScreenshot(client, browser.session_id);
+ if (!browser.cdp_ws_url) return captureScreenshot(client, browser.session_id);
+ const executor = new BrowserExecutor(new CdpConnection(browser.cdp_ws_url));
+ try {
+ return (await executor.screenshot()).data;
+ } catch {
+ return undefined;
+ } finally {
+ executor.close();
+ }
+}
diff --git a/packages/cli/src/print.ts b/packages/cli/src/print.ts
--- a/packages/cli/src/print.ts
+++ b/packages/cli/src/print.ts
@@ -1,7 +1,7 @@
import type { AgentHarnessEvent, CuaAgentHarness, Session, Skill } from "@onkernel/cua-agent";
import type { ImageContent } from "@onkernel/cua-ai";
import { stderr, stdout } from "node:process";
-import { captureScreenshot } from "./harness-browser";
+import { captureModeScreenshot } from "./harness-browser";
import type { CuaBrowserHandle } from "./harness-browser";
import { attachHarnessJsonlSink } from "./output/harness-jsonl";
import { parseSkillInvocation } from "./harness-skills";
@@ -96,7 +96,11 @@
if (opts.skipInitialScreenshot) return undefined;
const hasPriorTurn = await sessionHasPriorTurn(opts.session);
if (hasPriorTurn) return undefined;
- const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id);
+ const png = await captureModeScreenshot(
+ opts.browserHandle.client,
+ opts.browserHandle.browser,
+ opts.harness.getMode(),
+ );
if (!png) return undefined;
return [
{
diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts
--- a/packages/cli/src/tui/main.ts
+++ b/packages/cli/src/tui/main.ts
@@ -23,7 +23,7 @@
import { initTheme } from "@earendil-works/pi-coding-agent";
import { homedir } from "node:os";
import type { ImageContent, Model } from "@onkernel/cua-ai";
-import { captureScreenshot, type CuaBrowserHandle } from "../harness-browser";
+import { captureModeScreenshot, type CuaBrowserHandle } from "../harness-browser";
import { resolveCuaModelRef } from "../harness-models";
import type { ContextFile } from "../harness-skills";
import { openTuiDebugLog } from "./debug-log";
@@ -448,7 +448,11 @@
if (firstPromptSent) return undefined;
if (opts.skipInitialScreenshot) return undefined;
if (await sessionHasPriorTurn(opts.session)) return undefined;
- const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id);
+ const png = await captureModeScreenshot(
+ opts.browserHandle.client,
+ opts.browserHandle.browser,
+ opts.harness.getMode(),
+ );
if (!png) return undefined;
return [{ type: "image", data: png.toString("base64"), mimeType: "image/png" }];
}You can send follow-ups to the cloud agent here.
- CDP: reject pending commands when the socket closes unexpectedly - CuaRuntimeController disposes the previous translator (closing its CDP connection) on setModel/setMode - CuaAgentHarness.setMode keeps the requested activation state of tools that survive the mode switch instead of reactivating everything - resolveCuaRuntimeSpec folds javascriptExec into the browser_20260701 declaration (explicit enable_javascript_exec wins) - browser_new_tab makes the new tab active - browser_screenshot region clip normalizes reversed corners - CLI validates --mode/--native-tool at parse time (usage error, exit 2) and cleans up the provisioned browser when harness setup fails - browser mode skips the OS-display initial screenshot in the TUI and print paths so the viewport stays the only visual frame
Contributor
Author
bugbot triageAddressed all 9 findings in a6f06a4 — 8 fixed as reported, 1 fixed with a different remedy than suggested:
Full typecheck + suites green after the changes (142/40/37). |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: OS navigation leaves refs valid
- OS-plane goto/back/forward now flag pending navigation and invalidate browser snapshot refs immediately after the queued kernel batch flushes.
- ✅ Fixed: Zoom crop exceeds image bounds
- Zoom now reads screenshot metadata and clamps crop bounds to the image dimensions before calling sharp.extract().
- ✅ Fixed: Browser mode skill lacks screenshot
- First-turn /skill invocations now run through harness.prompt so any available initial images can be attached instead of calling harness.skill directly.
Or push these changes by commenting:
@cursor push bd5a1cbb76
Preview (bd5a1cbb76)
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -407,6 +407,17 @@
this.generations.set(targetId, this.generation(targetId) + 1);
}
+ /**
+ * OS-plane navigation (goto/back/forward) can change the active page
+ * without going through browser_navigate, so mark snapshot refs stale.
+ */
+ invalidateRefsForExternalNavigation(): void {
+ if (this.activeTargetId) {
+ this.invalidateRefs(this.activeTargetId);
+ }
+ this.refs.clear();
+ }
+
private async session(tabId?: string): Promise<string> {
return this.attach(await this.resolveTarget(tabId));
}
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -125,10 +125,16 @@
async executeBatch(actions: CuaAction[]): Promise<BatchExecutionResult> {
const result: BatchExecutionResult = { readResults: [] };
const pending: KernelBatchAction[] = [];
+ let pendingExternalNavigation = false;
const flush = async (): Promise<void> => {
if (pending.length === 0) return;
+ const invalidatesRefs = pendingExternalNavigation;
+ pendingExternalNavigation = false;
await this.runKernelBatch(pending.splice(0));
+ if (invalidatesRefs) {
+ this.browserExecutor?.invalidateRefsForExternalNavigation();
+ }
};
for (const action of actions) {
@@ -160,12 +166,15 @@
{ type: "type_text", type_text: { text: normalizeGotoUrl(action.url) ?? "" } },
keypress(["Enter"]),
);
+ pendingExternalNavigation = true;
break;
case "back":
pending.push(keypress(["Alt", "Left"]));
+ pendingExternalNavigation = true;
break;
case "forward":
pending.push(keypress(["Alt", "Right"]));
+ pendingExternalNavigation = true;
break;
default:
// Native computer mappings may omit click coordinates, meaning
@@ -192,10 +201,17 @@
async zoom(action: CuaActionZoom): Promise<{ data: Buffer; mimeType: string }> {
const screenshot = await this.screenshot();
const [x0, y0, x1, y1] = action.region;
- const left = Math.max(0, Math.trunc(Math.min(x0, x1)));
- const top = Math.max(0, Math.trunc(Math.min(y0, y1)));
- const width = Math.max(1, Math.trunc(Math.abs(x1 - x0)));
- const height = Math.max(1, Math.trunc(Math.abs(y1 - y0)));
+ const image = sharp(screenshot.data);
+ const metadata = await image.metadata();
+ const imageWidth = metadata.width ?? 0;
+ const imageHeight = metadata.height ?? 0;
+ if (imageWidth <= 0 || imageHeight <= 0) throw new Error("failed to read screenshot dimensions");
+ const left = clamp(Math.trunc(Math.min(x0, x1)), 0, imageWidth - 1);
+ const top = clamp(Math.trunc(Math.min(y0, y1)), 0, imageHeight - 1);
+ const right = clamp(Math.trunc(Math.max(x0, x1)), left + 1, imageWidth);
+ const bottom = clamp(Math.trunc(Math.max(y0, y1)), top + 1, imageHeight);
+ const width = right - left;
+ const height = bottom - top;
const data = await sharp(screenshot.data).extract({ left, top, width, height }).png().toBuffer();
return { data, mimeType: "image/png" };
}
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -26,6 +26,7 @@
function createFakeDom() {
const executed: CuaBrowserAction[] = [];
+ let invalidated = 0;
const dom = {
execute: async (action: CuaBrowserAction): Promise<BatchReadResult[]> => {
executed.push(action);
@@ -33,8 +34,11 @@
return [];
},
screenshot: async () => ({ data: Buffer.from("png"), mimeType: "image/png" }),
+ invalidateRefsForExternalNavigation: () => {
+ invalidated += 1;
+ },
} as unknown as BrowserExecutor;
- return { executed, dom };
+ return { executed, dom, invalidated: () => invalidated };
}
describe("InternalComputerTranslator DOM plane", () => {
@@ -59,6 +63,20 @@
const translator = new InternalComputerTranslator({ browser: { session_id: "b" } as KernelBrowser, client });
await expect(translator.executeBatch([{ type: "browser_text" }])).rejects.toThrow(/cdp_ws_url/);
});
+
+ it("invalidates browser refs after OS-plane navigation before the next DOM action", async () => {
+ const { client } = createClient();
+ const { dom, invalidated } = createFakeDom();
+ const translator = new InternalComputerTranslator({ browser, client, createBrowserExecutor: () => dom });
+
+ await translator.executeBatch([
+ { type: "browser_text" },
+ { type: "goto", url: "https://example.com" },
+ { type: "browser_text" },
+ ]);
+
+ expect(invalidated()).toBe(1);
+ });
});
describe("InternalComputerTranslator OS additions", () => {
@@ -73,6 +91,17 @@
expect(metadata.height).toBe(30);
});
+ it("clamps zoom crops that extend past the screenshot bounds", async () => {
+ const { client } = createClient();
+ const translator = new InternalComputerTranslator({ browser, client });
+ const result = await translator.executeBatch([{ type: "zoom", region: [90, 70, 150, 140] }]);
+ const read = result.readResults[0]!;
+ if (read.type !== "screenshot") throw new Error("expected screenshot read result");
+ const metadata = await sharp(read.data).metadata();
+ expect(metadata.width).toBe(10);
+ expect(metadata.height).toBe(10);
+ });
+
it("passes num_clicks through and resolves missing click coordinates from the cursor", async () => {
const { batches, client } = createClient();
const translator = new InternalComputerTranslator({ browser, client });
diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts
--- a/packages/cli/src/tui/main.ts
+++ b/packages/cli/src/tui/main.ts
@@ -325,13 +325,13 @@
messages.addNotice(`invoking /skill:${skill.name}`);
requestRender("skill_invocation");
const skillRemainder = parsed.remainder || undefined;
+ const firstSkillTurn = !firstPromptSent;
const skillImages = await maybeInitialScreenshot(opts, firstPromptSent);
firstPromptSent = true;
- if (skillImages) {
- // `harness.skill` has no images option; fall back to `prompt`
- // with the formatted skill invocation so the first turn sees
- // the browser screenshot.
- await opts.harness.prompt(formatSkillInvocation(skill, skillRemainder), { images: skillImages });
+ if (firstSkillTurn || skillImages) {
+ // `harness.skill` has no images option; keep first-turn skills on
+ // `prompt` so they can carry attached grounding images.
+ await opts.harness.prompt(formatSkillInvocation(skill, skillRemainder), skillImages ? { images: skillImages } : undefined);
} else {
await opts.harness.skill(skill.name, skillRemainder);
}You can send follow-ups to the cloud agent here.
Construct from the cdp_ws_url instead of an injected connection, so close() disposes a resource the executor actually owns. Drops the now-redundant createBrowserExecutor helper; the translator's factory option remains the test seam.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 5 total unresolved issues (including 3 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Browser mode skips first-turn image
- First-turn screenshot capture now attempts a browser-viewport CDP screenshot in browser mode and falls back to OS capture, so print/TUI first turns still attach grounding images.
- ✅ Fixed: CDP close races connection open
- CdpConnection now tracks opening sockets/rejecters with an epoch guard so close() cancels in-flight opens and ignores stale open/close events that previously revived torn-down connections.
Or push these changes by commenting:
@cursor push e4c23366ce
Preview (e4c23366ce)
diff --git a/packages/agent/src/translator/cdp.ts b/packages/agent/src/translator/cdp.ts
--- a/packages/agent/src/translator/cdp.ts
+++ b/packages/agent/src/translator/cdp.ts
@@ -31,6 +31,9 @@
export class CdpConnection {
private socket?: WebSocket;
private opening?: Promise<WebSocket>;
+ private openingSocket?: WebSocket;
+ private openingReject?: (error: Error) => void;
+ private closeEpoch = 0;
private nextId = 1;
private readonly pending = new Map<number, PendingCommand>();
private readonly listeners = new Set<CdpEventListener>();
@@ -73,9 +76,17 @@
}
close(): void {
- this.socket?.close();
+ this.closeEpoch += 1;
+ const socket = this.socket;
+ const openingSocket = this.openingSocket;
+ const openingReject = this.openingReject;
this.socket = undefined;
this.opening = undefined;
+ this.openingSocket = undefined;
+ this.openingReject = undefined;
+ openingReject?.(new Error("CDP connection closed"));
+ openingSocket?.close();
+ if (socket && socket !== openingSocket) socket.close();
this.sessionsByTarget.clear();
this.rejectPending(new Error("CDP connection closed"));
}
@@ -87,19 +98,41 @@
private connect(): Promise<WebSocket> {
if (this.socket && this.socket.readyState === WebSocket.OPEN) return Promise.resolve(this.socket);
- this.opening ??= new Promise<WebSocket>((resolve, reject) => {
+ if (this.opening) return this.opening;
+ const epoch = this.closeEpoch;
+ let openingPromise!: Promise<WebSocket>;
+ openingPromise = new Promise<WebSocket>((resolve, reject) => {
const socket = new WebSocket(this.wsUrl);
+ this.openingSocket = socket;
+ this.openingReject = reject;
socket.addEventListener("open", () => {
+ if (epoch !== this.closeEpoch || this.openingSocket !== socket) {
+ socket.close();
+ reject(new Error("CDP connection closed"));
+ return;
+ }
this.socket = socket;
+ if (this.opening === openingPromise) this.opening = undefined;
+ this.openingSocket = undefined;
+ this.openingReject = undefined;
resolve(socket);
});
socket.addEventListener("error", () => {
- this.opening = undefined;
+ if (this.opening === openingPromise) this.opening = undefined;
+ if (this.openingSocket === socket) this.openingSocket = undefined;
+ if (this.openingReject === reject) this.openingReject = undefined;
reject(new Error(`CDP connection to ${this.wsUrl} failed`));
});
socket.addEventListener("close", () => {
- this.socket = undefined;
- this.opening = undefined;
+ const isActive = this.socket === socket;
+ const isOpening = this.openingSocket === socket;
+ if (!isActive && !isOpening) return;
+ if (isActive) this.socket = undefined;
+ if (isOpening) {
+ this.openingSocket = undefined;
+ if (this.opening === openingPromise) this.opening = undefined;
+ if (this.openingReject === reject) this.openingReject = undefined;
+ }
this.sessionsByTarget.clear();
this.rejectPending(new Error("CDP connection closed"));
});
@@ -107,7 +140,8 @@
this.handleMessage(typeof event.data === "string" ? event.data : String(event.data));
});
});
- return this.opening;
+ this.opening = openingPromise;
+ return openingPromise;
}
private handleMessage(data: string): void {
diff --git a/packages/agent/test/cdp.test.ts b/packages/agent/test/cdp.test.ts
new file mode 100644
--- /dev/null
+++ b/packages/agent/test/cdp.test.ts
@@ -1,0 +1,100 @@
+import { afterEach, describe, expect, it } from "vitest";
+import { CdpConnection } from "../src/translator/cdp";
+
+type FakeListener = (event: { data?: string }) => void;
+
+class FakeWebSocket {
+ static readonly CONNECTING = 0;
+ static readonly OPEN = 1;
+ static readonly CLOSING = 2;
+ static readonly CLOSED = 3;
+ static instances: FakeWebSocket[] = [];
+
+ readonly url: string;
+ readyState = FakeWebSocket.CONNECTING;
+ readonly sent: string[] = [];
+ private readonly listeners = new Map<string, Set<FakeListener>>();
+
+ constructor(url: string) {
+ this.url = url;
+ FakeWebSocket.instances.push(this);
+ }
+
+ addEventListener(type: string, listener: FakeListener): void {
+ let set = this.listeners.get(type);
+ if (!set) {
+ set = new Set<FakeListener>();
+ this.listeners.set(type, set);
+ }
+ set.add(listener);
+ }
+
+ removeEventListener(type: string, listener: FakeListener): void {
+ this.listeners.get(type)?.delete(listener);
+ }
+
+ send(data: string): void {
+ this.sent.push(data);
+ }
+
+ close(): void {
+ if (this.readyState === FakeWebSocket.CLOSED) return;
+ this.readyState = FakeWebSocket.CLOSED;
+ this.emit("close", {});
+ }
+
+ emitOpen(): void {
+ this.readyState = FakeWebSocket.OPEN;
+ this.emit("open", {});
+ }
+
+ emitError(): void {
+ this.emit("error", {});
+ }
+
+ emitMessage(payload: unknown): void {
+ const data = typeof payload === "string" ? payload : JSON.stringify(payload);
+ this.emit("message", { data });
+ }
+
+ private emit(type: string, event: { data?: string }): void {
+ for (const listener of this.listeners.get(type) ?? []) listener(event);
+ }
+
+ static reset(): void {
+ FakeWebSocket.instances = [];
+ }
+}
+
+const originalWebSocket = globalThis.WebSocket;
+
+describe("CdpConnection", () => {
+ afterEach(() => {
+ globalThis.WebSocket = originalWebSocket;
+ FakeWebSocket.reset();
+ });
+
+ it("does not revive a stale socket after close() during connect", async () => {
+ globalThis.WebSocket = FakeWebSocket as unknown as typeof WebSocket;
+ const cdp = new CdpConnection("wss://example.test/cdp");
+
+ const first = cdp.send("Target.getTargets");
+ const stale = FakeWebSocket.instances[0];
+ expect(stale).toBeDefined();
+
+ cdp.close();
+ await expect(first).rejects.toThrow("CDP connection closed");
+
+ stale!.emitOpen();
+ const second = cdp.send("Target.getTargets");
+ expect(FakeWebSocket.instances).toHaveLength(2);
+ const fresh = FakeWebSocket.instances[1]!;
+
+ fresh.emitOpen();
+ await Promise.resolve();
+ const sent = JSON.parse(fresh.sent[0]!) as { id: number };
+ fresh.emitMessage({ id: sent.id, result: { targetInfos: [] } });
+
+ await expect(second).resolves.toEqual({ targetInfos: [] });
+ });
+});
diff --git a/packages/cli/src/harness-browser.ts b/packages/cli/src/harness-browser.ts
--- a/packages/cli/src/harness-browser.ts
+++ b/packages/cli/src/harness-browser.ts
@@ -100,3 +100,107 @@
return undefined;
}
}
+
+/**
+ * Best-effort first-turn screenshot capture:
+ * - browser mode: viewport screenshot over CDP (fallback: OS screenshot)
+ * - other modes: OS screenshot
+ */
+export async function captureInitialScreenshot(
+ client: Kernel,
+ browser: KernelBrowser,
+ mode: string,
+): Promise<Buffer | undefined> {
+ if (mode === "browser") {
+ const cdpWsUrl = browser.cdp_ws_url;
+ if (cdpWsUrl) {
+ const viewport = await captureViewportScreenshot(cdpWsUrl);
+ if (viewport) return viewport;
+ }
+ }
+ return captureScreenshot(client, browser.session_id);
+}
+
+async function captureViewportScreenshot(cdpWsUrl: string): Promise<Buffer | undefined> {
+ return new Promise<Buffer | undefined>((resolve) => {
+ const socket = new WebSocket(cdpWsUrl);
+ const timeout = setTimeout(() => {
+ cleanup();
+ resolve(undefined);
+ }, 5_000);
+ let settled = false;
+ const REQUEST_LIST_TARGETS = 1;
+ const REQUEST_ATTACH = 2;
+ const REQUEST_CAPTURE = 3;
+
+ const cleanup = (): void => {
+ if (settled) return;
+ settled = true;
+ clearTimeout(timeout);
+ try {
+ socket.close();
+ } catch {
+ // ignore best-effort close errors
+ }
+ };
+
+ socket.addEventListener("open", () => {
+ socket.send(JSON.stringify({ id: REQUEST_LIST_TARGETS, method: "Target.getTargets", params: {} }));
+ });
+ socket.addEventListener("error", () => {
+ cleanup();
+ resolve(undefined);
+ });
+ socket.addEventListener("close", () => {
+ cleanup();
+ resolve(undefined);
+ });
+ socket.addEventListener("message", (event) => {
+ let message: {
+ id?: number;
+ result?: {
+ targetInfos?: Array<{ targetId?: string; type?: string }>;
+ sessionId?: string;
+ data?: string;
+ };
+ };
+ try {
+ message = JSON.parse(typeof event.data === "string" ? event.data : String(event.data));
+ } catch {
+ return;
+ }
+ if (message.id === REQUEST_LIST_TARGETS) {
+ const targetId = message.result?.targetInfos?.find((target) => target.type === "page")?.targetId;
+ if (!targetId) {
+ cleanup();
+ resolve(undefined);
+ return;
+ }
+ socket.send(JSON.stringify({ id: REQUEST_ATTACH, method: "Target.attachToTarget", params: { targetId, flatten: true } }));
+ return;
+ }
+ if (message.id === REQUEST_ATTACH) {
+ const sessionId = message.result?.sessionId;
+ if (!sessionId) {
+ cleanup();
+ resolve(undefined);
+ return;
+ }
+ socket.send(
+ JSON.stringify({
+ id: REQUEST_CAPTURE,
+ method: "Page.captureScreenshot",
+ params: { format: "png" },
+ sessionId,
+ }),
+ );
+ return;
+ }
+ if (message.id === REQUEST_CAPTURE) {
+ const data = message.result?.data;
+ cleanup();
+ resolve(typeof data === "string" ? Buffer.from(data, "base64") : undefined);
+ }
+ });
+ });
+}
diff --git a/packages/cli/src/print.ts b/packages/cli/src/print.ts
--- a/packages/cli/src/print.ts
+++ b/packages/cli/src/print.ts
@@ -1,7 +1,7 @@
import type { AgentHarnessEvent, CuaAgentHarness, Session, Skill } from "@onkernel/cua-agent";
import type { ImageContent } from "@onkernel/cua-ai";
import { stderr, stdout } from "node:process";
-import { captureScreenshot } from "./harness-browser";
+import { captureInitialScreenshot } from "./harness-browser";
import type { CuaBrowserHandle } from "./harness-browser";
import { attachHarnessJsonlSink } from "./output/harness-jsonl";
import { parseSkillInvocation } from "./harness-skills";
@@ -94,12 +94,9 @@
async function maybeInitialScreenshot(opts: RunPrintOptions): Promise<ImageContent[] | undefined> {
if (opts.skipInitialScreenshot) return undefined;
- // Browser mode's only frame is the viewport; skip the OS-display capture
- // rather than mix coordinate frames on the first turn.
- if (opts.harness.getMode() === "browser") return undefined;
const hasPriorTurn = await sessionHasPriorTurn(opts.session);
if (hasPriorTurn) return undefined;
- const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id);
+ const png = await captureInitialScreenshot(opts.browserHandle.client, opts.browserHandle.browser, opts.harness.getMode());
if (!png) return undefined;
return [
{
diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts
--- a/packages/cli/src/tui/main.ts
+++ b/packages/cli/src/tui/main.ts
@@ -23,7 +23,7 @@
import { initTheme } from "@earendil-works/pi-coding-agent";
import { homedir } from "node:os";
import type { ImageContent, Model } from "@onkernel/cua-ai";
-import { captureScreenshot, type CuaBrowserHandle } from "../harness-browser";
+import { captureInitialScreenshot, type CuaBrowserHandle } from "../harness-browser";
import { resolveCuaModelRef } from "../harness-models";
import type { ContextFile } from "../harness-skills";
import { openTuiDebugLog } from "./debug-log";
@@ -447,11 +447,8 @@
): Promise<ImageContent[] | undefined> {
if (firstPromptSent) return undefined;
if (opts.skipInitialScreenshot) return undefined;
- // Browser mode's only frame is the viewport; skip the OS-display capture
- // rather than mix coordinate frames on the first turn.
- if (opts.harness.getMode() === "browser") return undefined;
if (await sessionHasPriorTurn(opts.session)) return undefined;
- const png = await captureScreenshot(opts.browserHandle.client, opts.browserHandle.browser.session_id);
+ const png = await captureInitialScreenshot(opts.browserHandle.client, opts.browserHandle.browser, opts.harness.getMode());
if (!png) return undefined;
return [{ type: "image", data: png.toString("base64"), mimeType: "image/png" }];
}
diff --git a/packages/cli/test/print.test.ts b/packages/cli/test/print.test.ts
--- a/packages/cli/test/print.test.ts
+++ b/packages/cli/test/print.test.ts
@@ -48,6 +48,27 @@
expect(exitCode).toBe(1);
});
+ it("attaches a first-turn image in browser mode", async () => {
+ fixture = await buildTestHarness({
+ turns: [
+ {
+ steps: [{ type: "text", text: "ok" }],
+ },
+ ],
+ });
+ await fixture.harness.setMode("browser");
+ // Keep the test offline: force first-turn capture to use the SDK fallback.
+ (fixture.kernel.browser as { cdp_ws_url?: string }).cdp_ws_url = undefined;
+ const lines: string[] = [];
+ const exitCode = await runPrintIntoBuffer(fixture, "look", lines);
+ expect(exitCode).toBe(0);
+ const entries = await fixture.session.getBranch();
+ const firstUser = entries.find((entry) => entry.type === "message" && entry.message.role === "user");
+ expect(firstUser).toBeDefined();
+ const content = (firstUser as { message: { content: Array<{ type: string }> } }).message.content;
+ expect(content.some((block) => block.type === "image")).toBe(true);
+ });
+
it("emits tool_call and tool_result envelopes for tool turns in jsonl mode", async () => {
fixture = await buildTestHarness({
turns: [You can send follow-ups to the cloud agent here.
…ndling - Prune stale ref entries when a target's generation bumps, and drop all refs/generations for a target when its session detaches - Enable Page events per attached session and invalidate refs on main-frame Page.frameNavigated, suppressing the bump for our own navigate() so it is counted once - Indent snapshot lines by rendered depth so skipped wrappers neither indent children nor consume the depth budget - Replace the dead textarea AX role with treeitem in interactive roles - Auto-dismiss native JavaScript dialogs and surface the dialog message as an extra read result on the next action - Accept an injected CdpConnection in BrowserExecutor for tests
- Accept alert/beforeunload dialogs instead of dismissing everything, so pages with unload guards can be left; word dialog notes per type - Consume the self-navigation flag on Page.navigatedWithinDocument so a same-document navigation can't suppress the next real invalidation - Render checked/pressed/expanded=false and heading level states - Mint refs for named content roles (heading, listitem, navigation, ...) and iframe nodes so they can be targeted by scroll_to and ref-scoped snapshots; stitch IframePresentational frames too - Merge consecutive StaticText siblings into one snapshot line - Heal duplicate refs by nth when the role+name cohort size is unchanged, instead of only healing singletons - Dispatch mouse input for OOPIF refs on the page session (box-model quads are main-viewport coordinates) - Bound ref growth: stop minting past the snapshot char limit, evict the oldest refs per target, and skip generation entries for frames that were never referenced - Run the cursor-hint DOM.describeNode calls in parallel and release scan objects via an object group - Note in browser_find's description that it searches the main frame
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 5 total unresolved issues (including 3 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: setModel restores stale active tools
setModenow persists the mode-filtered active tool list intorequestedActiveToolNames, so latersetModelcalls cannot reapply stale tool names.
- ✅ Fixed: Navigation helper wrong screenshot frame
- The navigation executor now receives the current mode and captures viewport screenshots in browser mode instead of always using OS display captures.
Or push these changes by commenting:
@cursor push 310b1abbd9
Preview (310b1abbd9)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -500,6 +500,7 @@
? tools.map((tool) => tool.name).filter((name) => !previousNames.has(name) || requested.includes(name))
: tools.map((tool) => tool.name);
await super.setTools(tools, active);
+ this.requestedActiveToolNames = active;
}
/** The action plane(s) currently exposed to the model. */
diff --git a/packages/agent/src/tools.ts b/packages/agent/src/tools.ts
--- a/packages/agent/src/tools.ts
+++ b/packages/agent/src/tools.ts
@@ -115,7 +115,7 @@
description: definition.description,
parameters: definition.parameters,
async execute(_toolCallId: string, params: unknown): Promise<AgentToolResult<NavigationDetails>> {
- return executeNavigationTool(translator, asNavigationInput(params));
+ return executeNavigationTool(translator, asNavigationInput(params), mode);
},
};
return tool;
@@ -191,7 +191,11 @@
return { content, details: { statusText: "Actions executed successfully.", readResults } };
}
-async function executeNavigationTool(translator: InternalComputerTranslator, params: CuaNavigationInput): Promise<AgentToolResult<NavigationDetails>> {
+async function executeNavigationTool(
+ translator: InternalComputerTranslator,
+ params: CuaNavigationInput,
+ mode: CuaMode = "computer",
+): Promise<AgentToolResult<NavigationDetails>> {
const action = params.action;
try {
let statusText = `${action} executed successfully.`;
@@ -204,7 +208,7 @@
} else {
await translator.executeBatch([{ type: action }]);
}
- const screenshot = await translator.screenshot();
+ const screenshot = mode === "browser" ? await translator.browser().screenshot() : await translator.screenshot();
return {
content: [
{ type: "text", text: statusText },You can send follow-ups to the cloud agent here.
- Remove the javascriptExec option: browser_evaluate is part of the default browser/hybrid action sets, and the native browser tool declaration gets enable_javascript_exec unless the spec sets it explicitly. Drop the CLI --js-exec flag. Opt out by passing an explicit actions list or native spec. - Remove the computerUseExtra option: the computer_use_extra navigation helper is always registered (deduped by name against caller executors). - Delete CUA_DEFAULT_BROWSER_ACTION_TYPES; the default browser set is now just CUA_BROWSER_ACTION_TYPES. - Rename the remaining OS/DOM-era identifiers, comments, and test titles to the computer/browser plane vocabulary (COMPUTER_ACTION_TYPE_SET, BROWSER_ACTION_TYPE_SET, createFakeBrowserExecutor, ...).
- computer_use_extra captures the browser viewport in browser mode instead of the OS display, matching post-action grounding captures - setMode updates requestedActiveToolNames so a later setModel re-applies the current mode's active subset rather than pre-switch names
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
There are 6 total unresolved issues (including 3 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Harness setMode desyncs on failure
- CuaAgentHarness.setMode now rolls the runtime mode back to the previous mode if super.setTools throws, keeping runtime and exposed tools consistent.
- ✅ Fixed: OOPIF subframe navigation skips invalidation
- Page.frameNavigated now invalidates child frame ids for OOPIF sessions when parentId is present so subframe navigations correctly expire refs.
- ✅ Fixed: Find skips iframe elements
- browser_find now searches both the main AX tree and stitched iframe AX subtrees and mints refs with the correct frame context.
Or push these changes by commenting:
@cursor push a667c3025f
Preview (a667c3025f)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -470,6 +470,7 @@
* plane conflicts with the requested mode.
*/
async setMode(mode: CuaMode): Promise<void> {
+ const previousMode = this.runtime.mode;
const previousNames = new Set(this.getTools().map((tool) => tool.name));
this.runtime.setMode(mode);
const tools = this.runtime.tools();
@@ -479,7 +480,12 @@
const active = requested
? tools.map((tool) => tool.name).filter((name) => !previousNames.has(name) || requested.includes(name))
: tools.map((tool) => tool.name);
- await super.setTools(tools, active);
+ try {
+ await super.setTools(tools, active);
+ } catch (error) {
+ this.runtime.setMode(previousMode);
+ throw error;
+ }
// The requested subset now reflects this mode's toolset; without this a
// later setModel would restore the pre-switch names.
if (requested) this.requestedActiveToolNames = active;
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -141,6 +141,10 @@
const targetId = this.targetsBySession.get(event.sessionId);
if (!targetId) return;
if (this.frameTargets.has(targetId)) {
+ if (frame.parentId) {
+ if (frame.id) this.invalidateFrame(frame.id);
+ return;
+ }
if (frame.id === targetId) this.invalidateFrame(targetId);
return;
}
@@ -455,15 +459,7 @@
const targetId = await this.resolveTarget(action.tab_id);
const session = await this.attach(targetId);
const { nodes } = await this.cdp.send<{ nodes: AXNode[] }>("Accessibility.getFullAXTree", {}, session);
- const queryTokens = tokenize(action.query);
- const scored = nodes
- .filter((node) => !node.ignored && node.backendDOMNodeId !== undefined && (node.name?.value || INTERACTIVE_ROLES.has(node.role?.value ?? "")))
- .map((node) => ({ node, score: overlapScore(queryTokens, tokenize(`${node.role?.value ?? ""} ${node.name?.value ?? ""}`)) }))
- .filter((entry) => entry.score > 0)
- .sort((a, b) => b.score - a.score)
- .slice(0, FIND_MATCH_LIMIT);
- if (scored.length === 0) return `No elements matched ${JSON.stringify(action.query)}. Try snapshot for the full tree.`;
- const ctx: RenderContext = {
+ const mainCtx: RenderContext = {
targetId,
frameKey: targetId,
sessionId: session,
@@ -471,8 +467,27 @@
interactiveOnly: false,
nthIndex: buildNthIndex(nodes),
};
+ const stitches = await this.stitchFrames(nodes, targetId, session, false);
+ const candidates: Array<{ node: AXNode; ctx: RenderContext }> = nodes.map((node) => ({ node, ctx: mainCtx }));
+ for (const stitch of stitches.values()) {
+ for (const node of stitch.byId.values()) {
+ candidates.push({ node, ctx: stitch.ctx });
+ }
+ }
+ const queryTokens = tokenize(action.query);
+ const scored = candidates
+ .filter(({ node }) => !node.ignored && node.backendDOMNodeId !== undefined && (node.name?.value || INTERACTIVE_ROLES.has(node.role?.value ?? "")))
+ .map(({ node, ctx }) => ({
+ node,
+ ctx,
+ score: overlapScore(queryTokens, tokenize(`${node.role?.value ?? ""} ${node.name?.value ?? ""}`)),
+ }))
+ .filter((entry) => entry.score > 0)
+ .sort((a, b) => b.score - a.score)
+ .slice(0, FIND_MATCH_LIMIT);
+ if (scored.length === 0) return `No elements matched ${JSON.stringify(action.query)}. Try snapshot for the full tree.`;
const text = scored
- .map(({ node }) => {
+ .map(({ node, ctx }) => {
const role = node.role?.value ?? "node";
const name = node.name?.value ? ` ${JSON.stringify(node.name.value)}` : "";
return `${role}${name} [${this.mintRef(node, ctx)}]`;
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -1,4 +1,4 @@
-import { describe, expect, it } from "vitest";
+import { describe, expect, it, vi } from "vitest";
import { createAssistantMessageEventStream, type AssistantMessage } from "@earendil-works/pi-ai";
import { resolveCuaRuntimeSpec } from "@onkernel/cua-ai";
import type Kernel from "@onkernel/sdk";
@@ -442,6 +442,25 @@
expect(names).toContain("browser_snapshot");
});
+ it("rolls back runtime mode when setMode fails to apply tools", async () => {
+ const harness = new CuaAgentHarness({
+ ...(await createHarnessServices()),
+ browser,
+ client,
+ model: "anthropic:claude-opus-4-5",
+ });
+ const initialTools = harness.getTools().map((tool) => tool.name);
+ const initialMode = harness.getMode();
+ const setToolsSpy = vi.spyOn(AgentHarness.prototype, "setTools").mockRejectedValueOnce(new Error("boom"));
+ try {
+ await expect(harness.setMode("browser")).rejects.toThrow("boom");
+ expect(harness.getMode()).toBe(initialMode);
+ expect(harness.getTools().map((tool) => tool.name)).toEqual(initialTools);
+ } finally {
+ setToolsSpy.mockRestore();
+ }
+ });
+
it("setMode keeps the requested activation state of surviving tools", async () => {
const harness = new CuaAgentHarness({
...(await createHarnessServices()),
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -239,6 +239,13 @@
return read.text;
}
+async function findText(executor: BrowserExecutor, query: string, action: Record<string, unknown> = {}): Promise<string> {
+ const results = await executor.execute({ type: "browser_find", query, ...action } as CuaBrowserAction);
+ const read = results[0]!;
+ if (read.type !== "browser_text") throw new Error("expected browser_text read result");
+ return read.text;
+}
+
const BUTTON_TREE = [
ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }),
ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 42, parentId: "1" }),
@@ -672,6 +679,49 @@
const text = await snapshotText(executor);
expect(text).toContain('button "Pay" [e');
});
+
+ it("invalidates refs belonging to OOPIF subframes when they navigate", async () => {
+ const { cdp, emit } = setupOopif();
+ const executor = new BrowserExecutor(cdp);
+ await snapshotText(executor);
+ const refs = refsOf(executor) as Map<
+ string,
+ {
+ backendNodeId: number;
+ targetId: string;
+ frameId: string;
+ sessionId: string;
+ generation: number;
+ role: string;
+ name: string;
+ nth: number;
+ cohort: number;
+ }
+ >;
+ refs.set("nested", {
+ backendNodeId: 70,
+ targetId: "TARGET-1",
+ frameId: "FRAME-INNER",
+ sessionId: "session-oop",
+ generation: 0,
+ role: "button",
+ name: "Inner",
+ nth: 0,
+ cohort: 1,
+ });
+
+ emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-INNER", parentId: "FRAME-OOP" } }, sessionId: "session-oop" });
+ await expect(executor.execute({ type: "browser_click", ref: "nested" } as CuaBrowserAction)).rejects.toThrow(/stale/);
+ });
+
+ it("includes OOPIF elements in browser_find results", async () => {
+ const { cdp } = setupOopif();
+ const executor = new BrowserExecutor(cdp);
+
+ const text = await findText(executor, "pay");
+
+ expect(text).toContain('button "Pay" [e');
+ });
});
describe("navigation tool grounding frame", () => {You can send follow-ups to the cloud agent here.
…e rollback - A navigation observed in an OOPIF target's session invalidates that target's refs even when it's a same-process subframe (their AX nodes are inlined in the frame target's tree, so its refs share the target's key) - browser_find searches stitched iframe trees too, matching what browser_snapshot renders; found refs resolve through their frame session - CuaAgentHarness.setMode restores the previous runtime mode when super.setTools fails, keeping the runtime in step with the exposed tools
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
There are 6 total unresolved issues (including 3 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Same-document nav leaves stale refs
- BrowserExecutor now tracks each target’s main-frame id and only clears pending self-navigation on matching same-document events, so subsequent real navigations always invalidate refs.
- ✅ Fixed: Repeated setMode wipes CDP state
- CuaRuntimeController.setMode now returns early when the requested mode is already active, preventing unnecessary translator disposal and CDP state resets.
- ✅ Fixed: Browser mode OS navigation mismatch
- Navigation tool actions goto/back/forward are now executed as browser_navigate actions in browser mode so navigation and screenshots stay on the same CDP plane.
Or push these changes by commenting:
@cursor push 272c26f8cb
Preview (272c26f8cb)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -167,6 +167,7 @@
}
setMode(mode: CuaMode): void {
+ if (mode === this.runtimeSpec.mode) return;
this.runtimeSpec = this.resolveSpec(this.runtimeSpec.model, mode);
this.currentMode = mode;
this.replaceTranslator();
diff --git a/packages/agent/src/tools.ts b/packages/agent/src/tools.ts
--- a/packages/agent/src/tools.ts
+++ b/packages/agent/src/tools.ts
@@ -202,6 +202,8 @@
if (action === "url") {
url = await translator.currentUrl();
statusText = `Current URL: ${url}`;
+ } else if (mode === "browser" && (action === "goto" || action === "back" || action === "forward")) {
+ await translator.executeBatch([{ type: "browser_navigate", url: action === "goto" ? (params.url ?? "") : action }]);
} else if (action === "goto") {
await translator.executeBatch([{ type: "goto", url: params.url ?? "" }]);
} else {
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -120,6 +120,7 @@
private readonly frameTargets = new Set<string>();
private readonly lastSnapshots = new Map<string, { key: string; shape: string }>();
private readonly selfNavigations = new Set<string>();
+ private readonly mainFrameIdsByTarget = new Map<string, string>();
private readonly dialogNotes: string[] = [];
private refCounter = 0;
private activeTargetId?: string;
@@ -151,6 +152,7 @@
if (frame.id) this.invalidateFrame(frame.id);
return;
}
+ if (frame.id) this.mainFrameIdsByTarget.set(targetId, frame.id);
if (!this.selfNavigations.delete(targetId)) this.invalidateRefs(targetId);
return;
}
@@ -160,7 +162,12 @@
if (!event.sessionId) return;
const targetId = this.targetsBySession.get(event.sessionId);
const { frameId } = event.params as { frameId?: string };
- if (targetId && frameId === targetId) this.selfNavigations.delete(targetId);
+ if (!targetId || !frameId) return;
+ const mainFrameId = this.mainFrameIdsByTarget.get(targetId);
+ const isMainFrame = mainFrameId ? frameId === mainFrameId : this.selfNavigations.has(targetId);
+ if (!isMainFrame) return;
+ if (!mainFrameId) this.mainFrameIdsByTarget.set(targetId, frameId);
+ this.selfNavigations.delete(targetId);
return;
}
case "Target.attachedToTarget": {
@@ -784,6 +791,7 @@
private dropTarget(targetId: string): void {
this.generations.delete(targetId);
this.selfNavigations.delete(targetId);
+ this.mainFrameIdsByTarget.delete(targetId);
this.lastSnapshots.delete(targetId);
this.frameSessions.delete(targetId);
this.frameTargets.delete(targetId);
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -1,4 +1,4 @@
-import { describe, expect, it } from "vitest";
+import { describe, expect, it, vi } from "vitest";
import { createAssistantMessageEventStream, type AssistantMessage } from "@earendil-works/pi-ai";
import { resolveCuaRuntimeSpec } from "@onkernel/cua-ai";
import type Kernel from "@onkernel/sdk";
@@ -13,6 +13,7 @@
type KernelBrowser,
type StreamFn,
} from "../src/index";
+import { InternalComputerTranslator } from "../src/translator/translator";
const browser = { session_id: "browser_123" } as KernelBrowser;
const client = {} as Kernel;
@@ -198,6 +199,23 @@
expect(agent.state.systemPrompt).toBe(resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "browser" }).defaultSystemPrompt);
});
+ it("does not recreate the translator on no-op setMode calls", () => {
+ const disposeSpy = vi.spyOn(InternalComputerTranslator.prototype, "dispose");
+ try {
+ const agent = new CuaAgent({
+ browser,
+ client,
+ initialState: {
+ model: "anthropic:claude-opus-4-5",
+ },
+ });
+ agent.setMode(agent.getMode());
+ expect(disposeSpy).not.toHaveBeenCalled();
+ } finally {
+ disposeSpy.mockRestore();
+ }
+ });
+
it("rejects setMode conflicting with a configured native tool", () => {
const agent = new CuaAgent({
browser,
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -272,11 +272,12 @@
it("does not let a same-document navigation suppress the next real navigation's invalidation", async () => {
const { cdp, emit } = createFakeCdp(BUTTON_TREE);
const executor = new BrowserExecutor(cdp);
+ emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-1" } }, sessionId: "session-1" });
await executor.execute({ type: "browser_navigate", url: "https://a.test/#section" } as CuaBrowserAction);
- emit({ method: "Page.navigatedWithinDocument", params: { frameId: "TARGET-1", url: "https://a.test/#section" }, sessionId: "session-1" });
+ emit({ method: "Page.navigatedWithinDocument", params: { frameId: "FRAME-1", url: "https://a.test/#section" }, sessionId: "session-1" });
await snapshotText(executor);
- emit({ method: "Page.frameNavigated", params: { frame: { id: "TARGET-1" } }, sessionId: "session-1" });
+ emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-1" } }, sessionId: "session-1" });
await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/);
});
@@ -699,22 +700,59 @@
describe("navigation tool grounding frame", () => {
const navTool = (mode: "computer" | "browser") => {
- const { client } = createClient();
+ const { client, batches } = createClient();
const { executor } = createFakeBrowserExecutor();
const translator = new InternalComputerTranslator({ browser, client, mode, createBrowserExecutor: () => executor });
- return buildCuaComputerTools({ toolExecutors: [], mode }, translator).find((tool) => tool.name === "computer_use_extra")!;
+ const tool = buildCuaComputerTools({ toolExecutors: [], mode }, translator).find((candidate) => candidate.name === "computer_use_extra")!;
+ return { tool, batches };
};
it("captures the viewport in browser mode and the OS display otherwise", async () => {
const viewportData = Buffer.from("png").toString("base64");
- const browserResult = await navTool("browser").execute("call_1", { action: "back" });
+ const browserResult = await navTool("browser").tool.execute("call_1", { action: "back" });
const viewportImage = browserResult.content.find((block) => block.type === "image");
expect(viewportImage).toMatchObject({ type: "image", data: viewportData });
- const computerResult = await navTool("computer").execute("call_2", { action: "back" });
+ const computerResult = await navTool("computer").tool.execute("call_2", { action: "back" });
const osImage = computerResult.content.find((block) => block.type === "image");
expect(osImage?.type).toBe("image");
expect((osImage as { data: string }).data).not.toBe(viewportData);
});
+
+ it("routes browser-mode navigation actions through CDP instead of OS keyboard shortcuts", async () => {
+ const browserSide = createFakeBrowserExecutor();
+ const browserClient = createClient();
+ const browserTranslator = new InternalComputerTranslator({
+ browser,
+ client: browserClient.client,
+ mode: "browser",
+ createBrowserExecutor: () => browserSide.executor,
+ });
+ const browserTool = buildCuaComputerTools({ toolExecutors: [], mode: "browser" }, browserTranslator).find((tool) => tool.name === "computer_use_extra")!;
+
+ await browserTool.execute("call_1", { action: "back" });
+ await browserTool.execute("call_2", { action: "goto", url: "https://example.com" });
+
+ expect(browserSide.executed).toEqual([
+ { type: "browser_navigate", url: "back" },
+ { type: "browser_navigate", url: "https://example.com" },
+ ]);
+ expect(browserClient.batches).toEqual([]);
+
+ const computerClient = createClient();
+ const computerSide = createFakeBrowserExecutor();
+ const computerTranslator = new InternalComputerTranslator({
+ browser,
+ client: computerClient.client,
+ mode: "computer",
+ createBrowserExecutor: () => computerSide.executor,
+ });
+ const computerTool = buildCuaComputerTools({ toolExecutors: [], mode: "computer" }, computerTranslator).find((tool) => tool.name === "computer_use_extra")!;
+ await computerTool.execute("call_3", { action: "back" });
+ expect(computerClient.batches).toEqual([
+ [{ type: "press_key", press_key: { keys: ["Left"], hold_keys: ["Alt_L"] } }],
+ ]);
+ expect(computerSide.executed).toEqual([]);
+ });
});You can send follow-ups to the cloud agent here.
- computer_use_extra's goto/back/forward/url use the browser plane in browser mode (Page.navigate / history / target url) instead of OS keyboard shortcuts, so navigation invalidates refs and matches the viewport grounding frame; adds BrowserExecutor.currentUrl() - setMode with the current mode is a no-op at the agent, harness, and runtime-controller level, so a repeated /mode selection no longer disposes the translator and its CDP-backed refs and tab state
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 6 total unresolved issues (including 4 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Harness setModel lacks failure rollback
- Wrapped the tool refresh in CuaAgentHarness.setModel with rollback to the previous runtime model when setTools fails so runtime and exposed tools stay aligned.
- ✅ Fixed: Hybrid extra tool OS navigation
- Updated executeNavigationTool to treat hybrid like browser mode for goto/back/forward/url and screenshot grounding so navigation stays on the browser/CDP plane.
Or push these changes by commenting:
@cursor push c3a4b4b6b4
Preview (c3a4b4b6b4)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -457,9 +457,16 @@
* concrete model selected by `@onkernel/cua-ai`.
*/
override async setModel(model: CuaRuntimeInput): Promise<void> {
+ const previousModel = this.runtime.model;
this.runtime.setModel(model);
const tools = this.runtime.tools();
- await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name));
+ try {
+ await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name));
+ } catch (err) {
+ // Keep the runtime in step with the currently exposed tools if tool refresh fails.
+ this.runtime.setModel(previousModel);
+ throw err;
+ }
await super.setModel(this.runtime.model);
}
diff --git a/packages/agent/src/tools.ts b/packages/agent/src/tools.ts
--- a/packages/agent/src/tools.ts
+++ b/packages/agent/src/tools.ts
@@ -196,25 +196,26 @@
mode: CuaMode,
): Promise<AgentToolResult<NavigationDetails>> {
const action = params.action;
+ const useBrowserPlane = mode !== "computer";
try {
let statusText = `${action} executed successfully.`;
let url: string | undefined;
- // In browser mode navigation stays on the browser plane (CDP) so it
- // invalidates refs and matches the viewport grounding frame; the OS
- // keyboard-shortcut path would navigate outside the plane the model sees.
+ // In browser/hybrid mode navigation stays on the browser plane (CDP) so it
+ // invalidates refs and matches tab state; the OS keyboard-shortcut path
+ // would navigate outside the browser-plane context.
if (action === "url") {
- url = mode === "browser" ? await translator.browser().currentUrl() : await translator.currentUrl();
+ url = useBrowserPlane ? await translator.browser().currentUrl() : await translator.currentUrl();
statusText = `Current URL: ${url}`;
- } else if (mode === "browser") {
+ } else if (useBrowserPlane) {
await translator.executeBatch([{ type: "browser_navigate", url: action === "goto" ? (params.url ?? "") : action }]);
} else if (action === "goto") {
await translator.executeBatch([{ type: "goto", url: params.url ?? "" }]);
} else {
await translator.executeBatch([{ type: action }]);
}
- // Same grounding frame as post-action captures: the browser viewport in
- // browser mode, the OS display otherwise.
- const screenshot = mode === "browser" ? await translator.browser().screenshot() : await translator.screenshot();
+ // Same grounding frame as post-action captures: browser viewport when
+ // navigation is on the browser plane, OS display otherwise.
+ const screenshot = useBrowserPlane ? await translator.browser().screenshot() : await translator.screenshot();
return {
content: [
{ type: "text", text: statusText },You can send follow-ups to the cloud agent here.
- computer_use_extra routes navigation through the browser plane whenever it is exposed (browser and hybrid modes), so refs invalidate and URL reads are tab-aware; only computer mode keeps the OS shortcut path - setModel restores the previous runtime model when super.setTools fails, mirroring setMode's rollback
open, url, press, and screenshot no longer build the LLM harness or require a model API key; new snapshot, find, text, fill, tabs, and click <x> <y> subcommands run the same way. find/fill share a new structured BrowserExecutor.findCandidates; fill resolves a lexical query to a unique fillable element in-process (refs never cross invocations). click/type-by-description, observe, and do stay model-mediated. Dead open/press/url prompts removed; help text and the cua-cli skill doc updated to the shipped surface.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 6 total unresolved issues (including 4 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Native mapper ignores feature flags
- Native executor mapping now threads resolved native spec flags into the mapper and rejects disabled
zoom/javascript_execactions instead of translating them.
- Native executor mapping now threads resolved native spec flags into the mapper and rejects disabled
- ✅ Fixed: Batch URL uses OS plane
InternalComputerTranslator.executeBatchnow routes canonicalurl/goto/back/forwardthrough the browser executor in browser and hybrid modes while keeping computer mode behavior unchanged.
Or push these changes by commenting:
@cursor push 5cc3cbcdb9
Preview (5cc3cbcdb9)
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -44,6 +44,7 @@
export class InternalComputerTranslator {
private readonly sessionId: string;
private readonly client: Kernel;
+ private readonly mode: CuaMode;
private readonly coordinateSystem: ComputerToolCoordinateSystem;
private readonly screenshotSpec?: CuaScreenshotSpec;
private readonly viewport: { width: number; height: number };
@@ -55,11 +56,12 @@
constructor(opts: InternalComputerTranslatorOptions) {
this.sessionId = opts.browser.session_id;
this.client = opts.client;
+ this.mode = opts.mode ?? "computer";
this.coordinateSystem = opts.coordinateSystem ?? { type: "pixel" };
this.screenshotSpec = opts.screenshot;
this.viewport = opts.browser.viewport ?? { width: 1920, height: 1080 };
this.cdpWsUrl = opts.browser.cdp_ws_url;
- this.browserExecutorOptions = { cursorHints: opts.cursorHints === true && opts.mode === "browser" };
+ this.browserExecutorOptions = { cursorHints: opts.cursorHints === true && this.mode === "browser" };
this.browserExecutorFactory = opts.createBrowserExecutor ?? ((cdpWsUrl, options) => new BrowserExecutor(cdpWsUrl, options));
}
@@ -155,23 +157,34 @@
break;
case "url":
await flush();
- result.readResults.push({ type: "url", url: await this.currentUrl() });
+ result.readResults.push({ type: "url", url: this.mode === "computer" ? await this.currentUrl() : await this.browser().currentUrl() });
break;
case "cursor_position":
await flush();
result.readResults.push({ type: "cursor_position", ...(await this.currentMousePosition()) });
break;
case "goto":
- pending.push(
- keypress(["Control", "l"]),
- { type: "type_text", type_text: { text: normalizeGotoUrl(action.url) ?? "" } },
- keypress(["Enter"]),
- );
+ if (this.mode !== "computer") {
+ await flush();
+ result.readResults.push(...(await this.browser().execute({ type: "browser_navigate", url: action.url })));
+ break;
+ }
+ pending.push(keypress(["Control", "l"]), { type: "type_text", type_text: { text: normalizeGotoUrl(action.url) ?? "" } }, keypress(["Enter"]));
break;
case "back":
+ if (this.mode !== "computer") {
+ await flush();
+ result.readResults.push(...(await this.browser().execute({ type: "browser_navigate", url: "back" })));
+ break;
+ }
pending.push(keypress(["Alt", "Left"]));
break;
case "forward":
+ if (this.mode !== "computer") {
+ await flush();
+ result.readResults.push(...(await this.browser().execute({ type: "browser_navigate", url: "forward" })));
+ break;
+ }
pending.push(keypress(["Alt", "Right"]));
break;
default:
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -28,15 +28,20 @@
function createFakeBrowserExecutor() {
const executed: CuaBrowserAction[] = [];
+ let currentUrlCalls = 0;
const executor = {
execute: async (action: CuaBrowserAction): Promise<BatchReadResult[]> => {
executed.push(action);
if (action.type === "browser_text") return [{ type: "browser_text", label: "text", text: "hello" }];
return [];
},
+ currentUrl: async () => {
+ currentUrlCalls += 1;
+ return "https://tab.test/";
+ },
screenshot: async () => ({ data: Buffer.from("png"), mimeType: "image/png" }),
} as unknown as BrowserExecutor;
- return { executed, executor };
+ return { executed, executor, currentUrlCalls: () => currentUrlCalls };
}
describe("InternalComputerTranslator browser plane", () => {
@@ -61,6 +66,29 @@
const translator = new InternalComputerTranslator({ browser: { session_id: "b" } as KernelBrowser, client });
await expect(translator.executeBatch([{ type: "browser_text" }])).rejects.toThrow(/cdp_ws_url/);
});
+
+ it("routes canonical navigation and url reads through the browser plane in browser and hybrid modes", async () => {
+ for (const mode of ["browser", "hybrid"] as const) {
+ const { batches, client } = createClient();
+ const { executed, executor, currentUrlCalls } = createFakeBrowserExecutor();
+ const translator = new InternalComputerTranslator({ browser, client, mode, createBrowserExecutor: () => executor });
+ const result = await translator.executeBatch([
+ { type: "goto", url: "example.com" },
+ { type: "back" },
+ { type: "forward" },
+ { type: "url" },
+ ]);
+
+ expect(executed).toEqual([
+ { type: "browser_navigate", url: "example.com" },
+ { type: "browser_navigate", url: "back" },
+ { type: "browser_navigate", url: "forward" },
+ ]);
+ expect(batches).toEqual([]);
+ expect(currentUrlCalls()).toBe(1);
+ expect(result.readResults).toContainEqual({ type: "url", url: "https://tab.test/" });
+ }
+ });
});
describe("InternalComputerTranslator computer additions", () => {
diff --git a/packages/ai/src/providers/anthropic/native.ts b/packages/ai/src/providers/anthropic/native.ts
--- a/packages/ai/src/providers/anthropic/native.ts
+++ b/packages/ai/src/providers/anthropic/native.ts
@@ -44,10 +44,14 @@
description: `Anthropic native ${resolved.spec.type} tool.`,
parameters: NativeActionSchema,
};
- const toActions =
- resolved.spec.type === "computer_20260701"
- ? (args: unknown) => mapNativeComputerInput(asNativeInput(args))
- : (args: unknown) => mapNativeBrowserInput(asNativeInput(args));
+ let toActions: CuaToolExecutorSpec["toActions"];
+ if (resolved.spec.type === "computer_20260701") {
+ const enableZoom = resolved.spec.enable_zoom === true;
+ toActions = (args: unknown) => mapNativeComputerInput(asNativeInput(args), { enableZoom });
+ } else {
+ const enableJavascriptExec = resolved.spec.enable_javascript_exec === true;
+ toActions = (args: unknown) => mapNativeBrowserInput(asNativeInput(args), { enableJavascriptExec });
+ }
return [{ definition, toActions }];
}
@@ -83,8 +87,16 @@
const MAX_KEY_REPEAT = 100;
+interface NativeComputerMapOptions {
+ enableZoom?: boolean;
+}
+
+interface NativeBrowserMapOptions {
+ enableJavascriptExec?: boolean;
+}
+
/** Map one `computer_20260701` tool input onto canonical computer-plane actions. */
-export function mapNativeComputerInput(input: NativeInput): CuaAction[] {
+export function mapNativeComputerInput(input: NativeInput, options: NativeComputerMapOptions = { enableZoom: true }): CuaAction[] {
switch (input.action) {
case "screenshot":
return [{ type: "screenshot" }];
@@ -126,6 +138,7 @@
case "cursor_position":
return [{ type: "cursor_position" }];
case "zoom":
+ if (!options.enableZoom) throw new Error(`unsupported computer_20260701 action "${input.action}"`);
return [{ type: "zoom", region: region(input.region) }];
default:
throw new Error(`unsupported computer_20260701 action "${input.action}"`);
@@ -133,7 +146,7 @@
}
/** Map one `browser_20260701` tool input onto canonical browser-plane actions. */
-export function mapNativeBrowserInput(input: NativeInput): CuaAction[] {
+export function mapNativeBrowserInput(input: NativeInput, options: NativeBrowserMapOptions = { enableJavascriptExec: true }): CuaAction[] {
const tab = tabId(input);
switch (input.action) {
case "navigate":
@@ -195,6 +208,7 @@
case "wait":
return [{ type: "wait", ms: durationSeconds(input) * 1000 }];
case "javascript_exec":
+ if (!options.enableJavascriptExec) throw new Error(`unsupported browser_20260701 action "${input.action}"`);
return [{ type: "browser_evaluate", code: text(input), ...tab }];
default:
throw new Error(`unsupported browser_20260701 action "${input.action}"`);
diff --git a/packages/ai/test/native-tools.test.ts b/packages/ai/test/native-tools.test.ts
--- a/packages/ai/test/native-tools.test.ts
+++ b/packages/ai/test/native-tools.test.ts
@@ -168,6 +168,20 @@
expect(actions).toEqual([{ type: "browser_click", ref: "e3" }]);
});
+ it("respects native feature flags when mapping tool calls", () => {
+ const computerSpec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { nativeTool: { type: "computer_20260701" } });
+ expect(() => computerSpec.toolExecutors[0]!.toActions({ action: "zoom", region: [0, 0, 10, 10] })).toThrow(
+ /unsupported computer_20260701 action/,
+ );
+
+ const browserSpec = resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", {
+ nativeTool: { type: "browser_20260701", enable_javascript_exec: false },
+ });
+ expect(() => browserSpec.toolExecutors[0]!.toActions({ action: "javascript_exec", text: "document.title" })).toThrow(
+ /unsupported browser_20260701 action/,
+ );
+ });
+
it("exports the anthropic namespace surface", () => {
expect(anthropic.mapNativeComputerInput).toBeTypeOf("function");
expect(anthropic.ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API).toBe(ANTHROPIC_NATIVE_COMPUTER_MESSAGES_API);You can send follow-ups to the cloud agent here.
- Apply the fillable-role filter inside findCandidates before the match limit so fill cannot miss a form field crowded out by non-fillable matches; findCandidates takes an optional role set - Map checkbox/radio fill values through an explicit true/false vocabulary instead of Boolean(string), rejecting anything else - Reject extra positionals on url/text/tabs and --filter outside snapshot before provisioning - Keep screenshot --out - stdout as pure PNG bytes; no status line - Guard translator.dispose() in teardown so handle.close() always runs - Route subcommand dispatch through deterministicActionFor and test it - Cover screenshot --out - and capture failure in the CLI suite - Align SKILL.md with the shipped surface (find/fill/screenshot rows, liveness message, transcript sentence)
Resolve prompts.ts/result.ts in favor of the deterministic url path: urlPrompt/extractFirstUrl and the action-result url extraction test target the removed model-mediated url subcommand.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 8 total unresolved issues (including 6 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: setModel leaves tools model split
- CuaAgentHarness.setModel now wraps both setTools and super.setModel in one rollback path that restores the prior runtime model and tool set if either step fails.
- ✅ Fixed: Closed tab leaves stale active target
- BrowserExecutor now marks when the active tab detaches and resolveTarget throws until a new tab is explicitly selected, preventing silent fallback to another tab.
Or push these changes by commenting:
@cursor push af0ca24383
Preview (af0ca24383)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -458,16 +458,24 @@
*/
override async setModel(model: CuaRuntimeInput): Promise<void> {
const previousModel = this.runtime.model;
+ const previousTools = this.getTools();
+ const previousActiveToolNames = this.requestedActiveToolNames ?? previousTools.map((tool) => tool.name);
this.runtime.setModel(model);
const tools = this.runtime.tools();
+ const nextActiveToolNames = this.requestedActiveToolNames ?? tools.map((tool) => tool.name);
try {
- await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name));
+ await super.setTools(tools, nextActiveToolNames);
+ await super.setModel(this.runtime.model);
} catch (err) {
- // Keep the runtime in step with the exposed tools when the switch fails.
+ // Keep runtime and exposed tools in step if either step fails.
this.runtime.setModel(previousModel);
+ try {
+ await super.setTools(previousTools, previousActiveToolNames);
+ } catch {
+ // Preserve the original setModel error if rollback tooling also fails.
+ }
throw err;
}
- await super.setModel(this.runtime.model);
}
override async setActiveTools(toolNames: string[]): Promise<void> {
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -130,6 +130,7 @@
private readonly dialogNotes: string[] = [];
private refCounter = 0;
private activeTargetId?: string;
+ private activeTargetClosed = false;
private readonly cursorHints: boolean;
private readonly cdp: CdpConnection;
@@ -651,6 +652,7 @@
private async newTab(): Promise<string> {
const targetId = await this.cdp.createTarget("about:blank");
this.activeTargetId = targetId;
+ this.activeTargetClosed = false;
return `Opened tab_id ${shortTabId(targetId)}.\n${await this.tabContext(targetId)}`;
}
@@ -806,6 +808,10 @@
}
private dropTarget(targetId: string): void {
+ if (this.activeTargetId === targetId) {
+ this.activeTargetId = undefined;
+ this.activeTargetClosed = true;
+ }
this.generations.delete(targetId);
this.selfNavigations.delete(targetId);
this.lastSnapshots.delete(targetId);
@@ -856,11 +862,15 @@
const match = targets.find((target) => shortTabId(target.targetId) === tabId || target.targetId === tabId);
if (!match) throw new Error(`unknown tab_id "${tabId}". Call list_tabs for current tabs.`);
this.activeTargetId = match.targetId;
+ this.activeTargetClosed = false;
return match.targetId;
}
if (this.activeTargetId && targets.some((target) => target.targetId === this.activeTargetId)) {
return this.activeTargetId;
}
+ if (this.activeTargetClosed) {
+ throw new Error("the active tab was closed. Call list_tabs and pass tab_id to choose a remaining tab.");
+ }
this.activeTargetId = targets[0]!.targetId;
return this.activeTargetId;
}
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -1,4 +1,4 @@
-import { describe, expect, it } from "vitest";
+import { describe, expect, it, vi } from "vitest";
import { createAssistantMessageEventStream, type AssistantMessage } from "@earendil-works/pi-ai";
import { resolveCuaRuntimeSpec } from "@onkernel/cua-ai";
import type Kernel from "@onkernel/sdk";
@@ -425,6 +425,26 @@
expect(harness.getTools()).toHaveLength(runtime.toolExecutors.length + 1);
});
+ it("rolls back tools when the pi model update fails after tools are refreshed", async () => {
+ const harness = new CuaAgentHarness({
+ ...(await createHarnessServices()),
+ browser,
+ client,
+ model: "anthropic:claude-opus-4-7",
+ });
+ const previousModelId = harness.getModel().id;
+ const previousToolNames = harness.getTools().map((tool) => tool.name);
+ const setModelSpy = vi.spyOn(AgentHarness.prototype, "setModel").mockRejectedValueOnce(new Error("model write failed"));
+ try {
+ await expect(harness.setModel("openai:gpt-5.5")).rejects.toThrow("model write failed");
+ } finally {
+ setModelSpy.mockRestore();
+ }
+
+ expect(harness.getModel().id).toBe(previousModelId);
+ expect(harness.getTools().map((tool) => tool.name)).toEqual(previousToolNames);
+ });
+
it("switches action planes through setMode", async () => {
const harness = new CuaAgentHarness({
...(await createHarnessServices()),
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -105,6 +105,7 @@
const sent: SentCommand[] = [];
const listeners: Array<(event: FakeCdpEvent) => void> = [];
let nodes = initialNodes as Array<{ backendDOMNodeId?: number }>;
+ let targets = [{ targetId: "TARGET-1", type: "page", title: "Page", url: "https://a.test/" }];
let cursorBackendIds: number[] = [];
const sessionTrees = new Map<string, Array<{ backendDOMNodeId?: number }>>();
const frameTrees = new Map<string, Array<{ backendDOMNodeId?: number }>>();
@@ -168,7 +169,7 @@
return {};
}
},
- pageTargets: async () => [{ targetId: "TARGET-1", type: "page", title: "Page", url: "https://a.test/" }],
+ pageTargets: async () => targets,
attachToTarget: async () => "session-1",
createTarget: async () => "TARGET-2",
close: () => {},
@@ -176,6 +177,9 @@
const setNodes = (next: unknown[]) => {
nodes = next as Array<{ backendDOMNodeId?: number }>;
};
+ const setPageTargets = (next: Array<{ targetId: string; type: string; title: string; url: string }>) => {
+ targets = next;
+ };
const setCursorBackendIds = (ids: number[]) => {
cursorBackendIds = ids;
};
@@ -195,6 +199,7 @@
sent,
emit,
setNodes,
+ setPageTargets,
setCursorBackendIds,
setSessionTree,
setFrameTree,
@@ -280,6 +285,24 @@
await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/);
});
+ it("errors on implicit actions after the active tab closes", async () => {
+ const { cdp, emit, setPageTargets } = createFakeCdp(BUTTON_TREE);
+ setPageTargets([
+ { targetId: "TARGET-1", type: "page", title: "A", url: "https://a.test/" },
+ { targetId: "TARGET-2", type: "page", title: "B", url: "https://b.test/" },
+ ]);
+ const executor = new BrowserExecutor(cdp);
+ await executor.execute({ type: "browser_text", tab_id: "TARGET-1" } as CuaBrowserAction);
+
+ setPageTargets([{ targetId: "TARGET-2", type: "page", title: "B", url: "https://b.test/" }]);
+ emit({ method: "Target.detachedFromTarget", params: { sessionId: "session-1" } });
+
+ await expect(executor.execute({ type: "browser_text" } as CuaBrowserAction)).rejects.toThrow(/active tab was closed/);
+ await expect(executor.execute({ type: "browser_text", tab_id: "TARGET-2" } as CuaBrowserAction)).resolves.toEqual([
+ { type: "browser_text", label: "text", text: "hello" },
+ ]);
+ });
+
it("does not double-bump the generation for its own navigate", async () => {
const { cdp, emit } = createFakeCdp(BUTTON_TREE);
const executor = new BrowserExecutor(cdp);You can send follow-ups to the cloud agent here.
snapshot/find refs now survive to later cua invocations: the executor's ref table (entries, generations, ref counter, active tab) serializes to <name>.refs.json alongside the named-session metadata and rehydrates on the next deterministic command, so cua -s x snapshot then cua -s x click e12 works across processes. New click <ref> and fill <ref> forms target refs directly over CDP; a stale ref exits 1 with the re-snapshot hint. Imported refs rebind their CDP session lazily (session ids are process-local), backend node ids stay valid for the life of the document, and the existing generation/self-heal machinery covers pages that changed between invocations. Refs files are removed on session stop.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 9 total unresolved issues (including 7 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Deterministic CLI skips mode validation
- Deterministic commands now call shared runtime flag parsing before provisioning, so invalid
--mode/--native-toolvalues throw usage errors instead of being ignored.
- Deterministic commands now call shared runtime flag parsing before provisioning, so invalid
- ✅ Fixed: Named session ignores runtime mode
- Interactive runs now persist the harness’s final mode back to named-session metadata on shutdown, ensuring later
-s <name>invocations reuse the updated runtime mode.
- Interactive runs now persist the harness’s final mode back to named-session metadata on shutdown, ensuring later
Or push these changes by commenting:
@cursor push 01f07ba1fa
Preview (01f07ba1fa)
diff --git a/packages/cli/src/cli-executor.ts b/packages/cli/src/cli-executor.ts
--- a/packages/cli/src/cli-executor.ts
+++ b/packages/cli/src/cli-executor.ts
@@ -3,7 +3,7 @@
import { stderr, stdout } from "node:process";
import { emitCompact, type RunActionResult } from "./action/harness-runner";
import { exitCodeFor, type ActionResult, type DeterministicActionType } from "./action/result";
-import { provisionForFlags, requireKernelApiKey, type HarnessCliFlags } from "./cli-harness";
+import { parseRuntimeFlags, provisionForFlags, requireKernelApiKey, type HarnessCliFlags } from "./cli-harness";
import { readNamedSessionRefs, writeNamedSessionRefs } from "./harness-named-sessions";
import { captureScreenshot, type CuaBrowserHandle } from "./harness-browser";
@@ -171,6 +171,7 @@
flags: HarnessCliFlags,
): Promise<number> {
const req = parseDeterministicArgs(action, rest, flags);
+ parseRuntimeFlags(flags);
const { apiKey, baseUrl } = requireKernelApiKey();
const provisioned = await provisionForFlags(flags, { kernelApiKey: apiKey, kernelBaseUrl: baseUrl });
const name = flags.namedSession;
diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts
--- a/packages/cli/src/cli-harness.ts
+++ b/packages/cli/src/cli-harness.ts
@@ -357,6 +357,8 @@
harness: ReturnType<typeof buildCuaHarness>;
provider: string;
modelRef: CuaModelRef;
+ mode: CuaMode | undefined;
+ nativeTool: CuaNativeToolSpec | undefined;
}
export interface SetupHarnessRuntimeOptions {
@@ -399,8 +401,7 @@
// Validate mode/native-tool flags before provisioning so a bad combination
// never leaves an orphaned browser behind.
- const mode = parseMode(flags.mode);
- const nativeTool = parseNativeTool(flags.nativeTool);
+ const { mode, nativeTool } = parseRuntimeFlags(flags);
resolveCuaRuntimeSpec(auth.modelRef, { mode, nativeTool });
const provisioned = await provisionForFlags(flags, auth);
@@ -453,8 +454,8 @@
await recordTranscriptPath(provisioned.named.name, resolved.transcriptPath);
await recordSessionModel(provisioned.named.name, {
model: auth.modelRef,
- mode: flags.mode,
- native_tool: flags.nativeTool,
+ mode,
+ native_tool: nativeTool?.type,
});
}
if (flags.verbose) {
@@ -489,6 +490,8 @@
harness,
provider,
modelRef: auth.modelRef,
+ mode,
+ nativeTool,
};
}
@@ -507,6 +510,16 @@
return value && value.length > 0 ? value : undefined;
}
+export function parseRuntimeFlags(flags: Pick<HarnessCliFlags, "mode" | "nativeTool">): {
+ mode: CuaMode | undefined;
+ nativeTool: CuaNativeToolSpec | undefined;
+} {
+ return {
+ mode: parseMode(flags.mode),
+ nativeTool: parseNativeTool(flags.nativeTool),
+ };
+}
+
function parseMode(raw: string | undefined): CuaMode | undefined {
if (raw === undefined) return undefined;
const value = raw.trim().toLowerCase();
@@ -601,6 +614,17 @@
skipInitialScreenshot: runtime.resolved?.resumed === true,
});
} finally {
+ if (flags.namedSession) {
+ try {
+ await recordSessionModel(flags.namedSession, {
+ model: runtime.modelRef,
+ mode: runtime.harness.getMode(),
+ native_tool: runtime.nativeTool?.type,
+ });
+ } catch (err) {
+ stderr.write(`[cua] cleanup warning: ${(err as Error).message}\n`);
+ }
+ }
try {
await runtime.handle.close();
} catch (err) {
diff --git a/packages/cli/test/cli-executor.test.ts b/packages/cli/test/cli-executor.test.ts
--- a/packages/cli/test/cli-executor.test.ts
+++ b/packages/cli/test/cli-executor.test.ts
@@ -227,6 +227,15 @@
// throw "missing Kernel API key" instead of the usage error.
await expect(runDeterministicCommand("open", [], baseFlags())).rejects.toThrow("usage: cua open");
});
+
+ it("runDeterministicCommand rejects invalid mode/native-tool values before provisioning", async () => {
+ await expect(runDeterministicCommand("url", [], baseFlags({ mode: "not-a-mode" }))).rejects.toThrow(
+ 'invalid --mode value "not-a-mode"; expected one of: computer | browser | hybrid',
+ );
+ await expect(runDeterministicCommand("url", [], baseFlags({ nativeTool: "not-a-tool" }))).rejects.toThrow(
+ 'invalid --native-tool value "not-a-tool"; expected one of: computer_20260701 | browser_20260701',
+ );
+ });
});
describe("runDeterministicOnHandle", () => {You can send follow-ups to the cloud agent here.
…del switches - Deterministic subcommands reject invalid --mode/--native-tool values with the same usage error as harness-backed entry points, instead of silently ignoring them - TUI /mode and /model switches patch the named session's metadata so a later cua -s <name> restores the selected plane and model instead of the startup values
A thrown Page.navigate / Page.navigateToHistoryEntry left the flag set, so the next main-frame frameNavigated consumed it and skipped ref invalidation. Both commands now run through a selfNavigate helper that disarms on rejection.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
There are 8 total unresolved issues (including 7 from previous reviews).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Browser multi-click uses one CDP pair
- Browser multi-click now dispatches one mousePressed/mouseReleased pair per click with incrementing clickCount, so double/triple clicks are emitted correctly.
Or push these changes by commenting:
@cursor push c1859263e2
Preview (c1859263e2)
diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -555,18 +555,20 @@
const point = await this.resolvePoint(action, targetId, session);
const modifiers = modifierBits(action.modifiers);
const button = action.button ?? "left";
- const clickCount = action.num_clicks ?? 1;
+ const clickCount = Math.max(1, Math.trunc(action.num_clicks ?? 1));
await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: point.x, y: point.y, modifiers }, point.session);
- await this.cdp.send(
- "Input.dispatchMouseEvent",
- { type: "mousePressed", x: point.x, y: point.y, button, clickCount, modifiers },
- point.session,
- );
- await this.cdp.send(
- "Input.dispatchMouseEvent",
- { type: "mouseReleased", x: point.x, y: point.y, button, clickCount, modifiers },
- point.session,
- );
+ for (let currentClick = 1; currentClick <= clickCount; currentClick += 1) {
+ await this.cdp.send(
+ "Input.dispatchMouseEvent",
+ { type: "mousePressed", x: point.x, y: point.y, button, clickCount: currentClick, modifiers },
+ point.session,
+ );
+ await this.cdp.send(
+ "Input.dispatchMouseEvent",
+ { type: "mouseReleased", x: point.x, y: point.y, button, clickCount: currentClick, modifiers },
+ point.session,
+ );
+ }
}
private async hover(action: CuaActionBrowserHover): Promise<void> {
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -251,6 +251,27 @@
];
describe("BrowserExecutor ref lifecycle", () => {
+ it("dispatches multi-clicks as repeated press/release pairs", async () => {
+ const { cdp, sent } = createFakeCdp(BUTTON_TREE);
+ const executor = new BrowserExecutor(cdp);
+ await snapshotText(executor);
+
+ await executor.execute({ type: "browser_click", ref: "e1", num_clicks: 3 } as CuaBrowserAction);
+
+ const clickEvents = sent
+ .filter((command) => command.method === "Input.dispatchMouseEvent")
+ .map((command) => ({ type: command.params.type, clickCount: command.params.clickCount }));
+ expect(clickEvents).toEqual([
+ { type: "mouseMoved", clickCount: undefined },
+ { type: "mousePressed", clickCount: 1 },
+ { type: "mouseReleased", clickCount: 1 },
+ { type: "mousePressed", clickCount: 2 },
+ { type: "mouseReleased", clickCount: 2 },
+ { type: "mousePressed", clickCount: 3 },
+ { type: "mouseReleased", clickCount: 3 },
+ ]);
+ });
+
it("prunes stale refs when a navigation bumps the generation", async () => {
const { cdp } = createFakeCdp(BUTTON_TREE);
const executor = new BrowserExecutor(cdp);You can send follow-ups to the cloud agent here.
… drift - Skip .refs.json sidecars and entries missing required fields in listNamedSessions so `cua session list` no longer crashes once a named session has persisted element refs - Focus the target element in browser_fill so a following `press Return` lands in the filled field and submits the form - Trim injected-script stacks from browser_fill errors to a single line - Correct SKILL.md: ref self-heal scope (navigation always invalidates), transcript JSONL record shape, session show/stop exit codes, and the models provider list (google/tzafon)
Browser-mode snapshots are the model's only page view, and cursor:pointer hints surface clickable elements with no ARIA role (the common div-soup case). The scan is already noise-bounded (skips native interactive subtrees, dedupes inherited cursors, caps at 100) and browser mode runs page-world JS by default anyway (browser_evaluate, fill), so the opt-in gate bought nothing. cursorHints: false opts out; hybrid/computer modes still never scan.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
There are 5 total unresolved issues (including 2 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Harness rollback keeps stale tool executors
- On setModel/setMode failure the harness now restores tools from the rolled-back runtime so executors rebind to the new rollback translator.
- ✅ Fixed: Mode switch not rolled back
- The /mode handler now tracks the previous mode and reverts the harness mode when persistence fails after a successful switch.
- ✅ Fixed: CDP send leaves pending hang
- CdpConnection.send now catches synchronous socket.send failures, removes the pending entry, and rejects the command promise immediately.
Or push these changes by commenting:
@cursor push 68c4b354e3
Preview (68c4b354e3)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -449,6 +449,11 @@
});
}
+ private async restoreToolsAfterRollback(activeToolNames: string[]): Promise<void> {
+ const rollbackTools = this.runtime.tools();
+ await super.setTools(rollbackTools, activeToolNames);
+ }
+
/**
* Mirror pi `AgentHarness.setModel()` while accepting CUA model refs.
*
@@ -458,6 +463,7 @@
*/
override async setModel(model: CuaRuntimeInput): Promise<void> {
const previousModel = this.runtime.model;
+ const previousActiveToolNames = this.getActiveTools().map((tool) => tool.name);
this.runtime.setModel(model);
const tools = this.runtime.tools();
try {
@@ -465,6 +471,7 @@
} catch (err) {
// Keep the runtime in step with the exposed tools when the switch fails.
this.runtime.setModel(previousModel);
+ await this.restoreToolsAfterRollback(previousActiveToolNames);
throw err;
}
await super.setModel(this.runtime.model);
@@ -483,6 +490,7 @@
async setMode(mode: CuaMode): Promise<void> {
if (mode === this.runtime.mode) return;
const previousMode = this.runtime.mode;
+ const previousActiveToolNames = this.getActiveTools().map((tool) => tool.name);
const previousNames = new Set(this.getTools().map((tool) => tool.name));
this.runtime.setMode(mode);
const tools = this.runtime.tools();
@@ -497,6 +505,7 @@
} catch (err) {
// Keep the runtime in step with the exposed tools when the switch fails.
this.runtime.setMode(previousMode);
+ await this.restoreToolsAfterRollback(previousActiveToolNames);
throw err;
}
// The requested subset now reflects this mode's toolset; without this a
diff --git a/packages/agent/src/translator/cdp.ts b/packages/agent/src/translator/cdp.ts
--- a/packages/agent/src/translator/cdp.ts
+++ b/packages/agent/src/translator/cdp.ts
@@ -48,7 +48,12 @@
const message = JSON.stringify({ id, method, params, ...(sessionId ? { sessionId } : {}) });
return new Promise<T>((resolve, reject) => {
this.pending.set(id, { resolve: resolve as (result: unknown) => void, reject });
- socket.send(message);
+ try {
+ socket.send(message);
+ } catch (error) {
+ this.pending.delete(id);
+ reject(error instanceof Error ? error : new Error(String(error)));
+ }
});
}
diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts
--- a/packages/cli/src/tui/main.ts
+++ b/packages/cli/src/tui/main.ts
@@ -504,11 +504,21 @@
messages.addError("usage: /mode <computer|browser|hybrid>");
return;
}
+ const previousMode = opts.harness.getMode();
+ let switched = false;
try {
await opts.harness.setMode(value);
+ switched = true;
if (opts.namedSession) await updateNamedSessionRuntime(opts.namedSession, { mode: value });
messages.addNotice(`mode → ${value}`);
} catch (err) {
+ if (switched) {
+ try {
+ await opts.harness.setMode(previousMode);
+ } catch (rollbackErr) {
+ messages.addError(`failed to rollback mode to ${previousMode}: ${(rollbackErr as Error).message}`);
+ }
+ }
messages.addError((err as Error).message);
}
}You can send follow-ups to the cloud agent here.
Snapshots always mark cursor:pointer elements with no interactive ARIA role as clickable hints; the cursorHints option is gone. The scan is noise-bounded and runs in the same CDP plane snapshots already use, so there was no configuration worth exposing. Also correct the 0.5.0 changelog entries: javascriptExec, cursorHints, and CUA_DEFAULT_BROWSER_ACTION_TYPES never shipped in a release, so their removal is not breaking; only the computerUseExtra removal is.
- Mode/model switches are two-phase: the outgoing translator stays alive until the new toolset is installed, and a failed switch restores the exact runtime the still-exposed tools are bound to (previously rollback left them wrapping a disposed translator) - CdpConnection.send rejects instead of hanging when the socket is no longer open (spec-silent discard) or send throws - browser_click dispatches one press/release cycle per click with an incrementing clickCount, matching native multi-click input - TUI /mode and /model treat a failed named-session persist as a warning on the successful switch, not as a failed switch
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Overlapping switches orphan translators
- I serialized CuaAgentHarness runtime-switch operations with a lock so setMode/setModel cannot overlap and each switch commits or rolls back its own translator lifecycle.
- ✅ Fixed: Native hold_key duration unit wrong
- I converted Anthropic native hold_key duration from seconds to milliseconds when mapping to canonical keypress, matching the wait and press_key duration units.
Or push these changes by commenting:
@cursor push 712542f00c
Preview (712542f00c)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -416,6 +416,7 @@
> extends AgentHarness<TSkill, TPromptTemplate, AgentTool> {
private readonly runtime: CuaRuntimeController;
private requestedActiveToolNames?: string[];
+ private pendingRuntimeSwitch: Promise<void> = Promise.resolve();
constructor(options: CuaAgentHarnessOptions<TSkill, TPromptTemplate>) {
const {
@@ -470,18 +471,20 @@
* concrete model selected by `@onkernel/cua-ai`.
*/
override async setModel(model: CuaRuntimeInput): Promise<void> {
- this.runtime.setModel(model);
- const tools = this.runtime.tools();
- try {
- await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name));
- } catch (err) {
- // The pre-switch tools stay exposed, so restore the runtime they are
- // bound to — including its still-live translator.
- this.runtime.rollbackSwitch();
- throw err;
- }
- this.runtime.commitSwitch();
- await super.setModel(this.runtime.model);
+ return this.withRuntimeSwitchLock(async () => {
+ this.runtime.setModel(model);
+ const tools = this.runtime.tools();
+ try {
+ await super.setTools(tools, this.requestedActiveToolNames ?? tools.map((tool) => tool.name));
+ } catch (err) {
+ // The pre-switch tools stay exposed, so restore the runtime they are
+ // bound to — including its still-live translator.
+ this.runtime.rollbackSwitch();
+ throw err;
+ }
+ this.runtime.commitSwitch();
+ await super.setModel(this.runtime.model);
+ });
}
override async setActiveTools(toolNames: string[]): Promise<void> {
@@ -495,34 +498,45 @@
* plane conflicts with the requested mode.
*/
async setMode(mode: CuaMode): Promise<void> {
- if (mode === this.runtime.mode) return;
- const previousNames = new Set(this.getTools().map((tool) => tool.name));
- this.runtime.setMode(mode);
- const tools = this.runtime.tools();
- // Tools that survive the mode switch (extraTools, shared names) keep
- // their requested activation state; names new in this mode activate.
- const requested = this.requestedActiveToolNames;
- const active = requested
- ? tools.map((tool) => tool.name).filter((name) => !previousNames.has(name) || requested.includes(name))
- : tools.map((tool) => tool.name);
- try {
- await super.setTools(tools, active);
- } catch (err) {
- // The pre-switch tools stay exposed, so restore the runtime they are
- // bound to — including its still-live translator.
- this.runtime.rollbackSwitch();
- throw err;
- }
- this.runtime.commitSwitch();
- // The requested subset now reflects this mode's toolset; without this a
- // later setModel would restore the pre-switch names.
- if (requested) this.requestedActiveToolNames = active;
+ return this.withRuntimeSwitchLock(async () => {
+ if (mode === this.runtime.mode) return;
+ const previousNames = new Set(this.getTools().map((tool) => tool.name));
+ this.runtime.setMode(mode);
+ const tools = this.runtime.tools();
+ // Tools that survive the mode switch (extraTools, shared names) keep
+ // their requested activation state; names new in this mode activate.
+ const requested = this.requestedActiveToolNames;
+ const active = requested
+ ? tools.map((tool) => tool.name).filter((name) => !previousNames.has(name) || requested.includes(name))
+ : tools.map((tool) => tool.name);
+ try {
+ await super.setTools(tools, active);
+ } catch (err) {
+ // The pre-switch tools stay exposed, so restore the runtime they are
+ // bound to — including its still-live translator.
+ this.runtime.rollbackSwitch();
+ throw err;
+ }
+ this.runtime.commitSwitch();
+ // The requested subset now reflects this mode's toolset; without this a
+ // later setModel would restore the pre-switch names.
+ if (requested) this.requestedActiveToolNames = active;
+ });
}
/** The action plane(s) currently exposed to the model. */
getMode(): CuaMode {
return this.runtime.mode;
}
+
+ private withRuntimeSwitchLock<T>(operation: () => Promise<T>): Promise<T> {
+ const queued = this.pendingRuntimeSwitch.then(operation, operation);
+ this.pendingRuntimeSwitch = queued.then(
+ () => undefined,
+ () => undefined,
+ );
+ return queued;
+ }
}
function composeOnPayload(first: AgentOptions["onPayload"], second: AgentOptions["onPayload"]): AgentOptions["onPayload"] {
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -1,4 +1,4 @@
-import { describe, expect, it } from "vitest";
+import { describe, expect, it, vi } from "vitest";
import { createAssistantMessageEventStream, type AssistantMessage } from "@earendil-works/pi-ai";
import { resolveCuaRuntimeSpec } from "@onkernel/cua-ai";
import type Kernel from "@onkernel/sdk";
@@ -7,6 +7,7 @@
AgentHarness,
CuaAgent,
CuaAgentHarness,
+ InternalComputerTranslator,
InMemorySessionRepo,
NodeExecutionEnv,
type AgentTool,
@@ -485,6 +486,46 @@
expect(active).not.toContain("custom");
});
+ it("serializes concurrent runtime switches so each stale translator is disposed", async () => {
+ const { env, session } = await createHarnessServices();
+ let writes = 0;
+ let releaseFirstWrite!: () => void;
+ const firstWriteBlocked = new Promise<void>((resolve) => {
+ releaseFirstWrite = resolve;
+ });
+ const gatedSession = new Proxy(session, {
+ get(target, prop, receiver) {
+ if (prop === "appendActiveToolsChange") {
+ const appendActiveToolsChange = Reflect.get(target, prop, receiver) as (...args: unknown[]) => Promise<unknown>;
+ return async (...args: unknown[]) => {
+ writes += 1;
+ if (writes === 1) await firstWriteBlocked;
+ return appendActiveToolsChange.apply(target, args);
+ };
+ }
+ return Reflect.get(target, prop, receiver);
+ },
+ });
+ const harness = new CuaAgentHarness({
+ env,
+ session: gatedSession,
+ browser,
+ client,
+ model: "anthropic:claude-opus-4-5",
+ });
+ const disposeSpy = vi.spyOn(InternalComputerTranslator.prototype, "dispose");
+ try {
+ const modeSwitch = harness.setMode("browser");
+ const modelSwitch = harness.setModel("anthropic:claude-opus-4-7");
+ releaseFirstWrite();
+ await Promise.all([modeSwitch, modelSwitch]);
+ expect(disposeSpy).toHaveBeenCalledTimes(2);
+ expect(harness.getMode()).toBe("browser");
+ } finally {
+ disposeSpy.mockRestore();
+ }
+ });
+
it("a failed mode switch keeps the pre-switch runtime and its live translator", async () => {
const { env, session } = await createHarnessServices();
let failWrites = false;
diff --git a/packages/ai/src/providers/anthropic/native.ts b/packages/ai/src/providers/anthropic/native.ts
--- a/packages/ai/src/providers/anthropic/native.ts
+++ b/packages/ai/src/providers/anthropic/native.ts
@@ -120,7 +120,7 @@
return Array.from({ length: repeat }, () => ({ type: "keypress" as const, keys: [text(input)] }));
}
case "hold_key":
- return [{ type: "keypress", keys: [text(input)], duration: durationSeconds(input) }];
+ return [{ type: "keypress", keys: [text(input)], duration: durationSeconds(input) * 1000 }];
case "wait":
return [{ type: "wait", ms: durationSeconds(input) * 1000 }];
case "cursor_position":
diff --git a/packages/ai/test/native-tools.test.ts b/packages/ai/test/native-tools.test.ts
--- a/packages/ai/test/native-tools.test.ts
+++ b/packages/ai/test/native-tools.test.ts
@@ -114,6 +114,9 @@
expect(mapNativeComputerInput({ action: "left_click_drag", start_coordinate: [1, 2], coordinate: [3, 4] })).toEqual([
{ type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] },
]);
+ expect(mapNativeComputerInput({ action: "hold_key", text: "Shift", duration: 2 })).toEqual([
+ { type: "keypress", keys: ["Shift"], duration: 2000 },
+ ]);
expect(mapNativeComputerInput({ action: "wait", duration: 2 })).toEqual([{ type: "wait", ms: 2000 }]);
expect(mapNativeComputerInput({ action: "zoom", region: [0, 0, 10, 10] })).toEqual([{ type: "zoom", region: [0, 0, 10, 10] }]);
});You can send follow-ups to the cloud agent here.
- beginSwitch disposes the superseded pending translator when a second mode/model switch starts before the first commits, instead of overwriting the rollback slot and orphaning the original - Native hold_key maps its seconds duration to the canonical keypress duration in milliseconds, matching how wait already converts
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Mode switch disposes active translator
- Translator disposal is now deferred until in-flight tool executions finish and harness model switches only finalize disposal after model update completion.
Or push these changes by commenting:
@cursor push 50e7567c51
Preview (50e7567c51)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -182,6 +182,8 @@
// alive until the new toolset is actually installed, because on failure
// the still-exposed pre-switch tools keep executing against it.
private previousRuntime?: { spec: CuaRuntimeSpec; translator: InternalComputerTranslator; mode?: CuaMode };
+ private readonly translatorInFlight = new Map<InternalComputerTranslator, number>();
+ private readonly translatorsPendingDispose = new Set<InternalComputerTranslator>();
private beginSwitch(spec: CuaRuntimeSpec): void {
if (this.previousRuntime) {
@@ -198,7 +200,9 @@
/** Dispose the pre-switch translator once the new toolset is installed. */
commitSwitch(): void {
- this.previousRuntime?.translator.dispose();
+ if (this.previousRuntime) {
+ this.disposeTranslator(this.previousRuntime.translator);
+ }
this.previousRuntime = undefined;
}
@@ -213,15 +217,13 @@
}
tools(): AgentTool[] {
+ const translator = this.translator;
return [
- ...buildCuaComputerTools(
- {
- toolExecutors: this.runtimeSpec.toolExecutors,
- mode: this.runtimeSpec.mode,
- playwright: this.options.playwright,
- },
- this.translator,
- ),
+ ...buildCuaComputerTools({
+ toolExecutors: this.runtimeSpec.toolExecutors,
+ mode: this.runtimeSpec.mode,
+ playwright: this.options.playwright,
+ }, translator).map((tool) => this.withTranslatorLease(tool, translator)),
...(this.options.extraTools ?? []),
];
}
@@ -255,6 +257,43 @@
mode: this.runtimeSpec.mode,
});
}
+
+ private withTranslatorLease(tool: AgentTool, translator: InternalComputerTranslator): AgentTool {
+ return {
+ ...tool,
+ execute: async (toolCallId, params) => this.runWithTranslatorLease(translator, () => tool.execute(toolCallId, params)),
+ };
+ }
+
+ private async runWithTranslatorLease<TResult>(
+ translator: InternalComputerTranslator,
+ run: () => Promise<TResult>,
+ ): Promise<TResult> {
+ this.translatorInFlight.set(translator, (this.translatorInFlight.get(translator) ?? 0) + 1);
+ try {
+ return await run();
+ } finally {
+ const remaining = (this.translatorInFlight.get(translator) ?? 1) - 1;
+ if (remaining > 0) {
+ this.translatorInFlight.set(translator, remaining);
+ } else {
+ this.translatorInFlight.delete(translator);
+ if (this.translatorsPendingDispose.delete(translator)) {
+ translator.dispose();
+ }
+ }
+ }
+ }
+
+ private disposeTranslator(translator: InternalComputerTranslator): void {
+ if ((this.translatorInFlight.get(translator) ?? 0) > 0) {
+ this.translatorsPendingDispose.add(translator);
+ return;
+ }
+ this.translatorInFlight.delete(translator);
+ this.translatorsPendingDispose.delete(translator);
+ translator.dispose();
+ }
}
/** Default stream path: the shared CUA `Models` collection. */
@@ -487,8 +526,13 @@
this.runtime.rollbackSwitch();
throw err;
}
- this.runtime.commitSwitch();
- await super.setModel(this.runtime.model);
+ try {
+ await super.setModel(this.runtime.model);
+ } finally {
+ // setTools already switched the exposed runtime; always finalize the
+ // pending switch so future mode/model changes don't treat it as pending.
+ this.runtime.commitSwitch();
+ }
}
override async setActiveTools(toolNames: string[]): Promise<void> {
diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -198,6 +198,40 @@
expect(agent.state.systemPrompt).toBe(resolveCuaRuntimeSpec("anthropic:claude-opus-4-5", { mode: "browser" }).defaultSystemPrompt);
});
+ it("keeps the pre-switch translator alive while an old tool call is in flight", async () => {
+ const agent = new CuaAgent({
+ browser,
+ client,
+ initialState: {
+ model: "anthropic:claude-opus-4-5",
+ },
+ });
+ const runtime = (agent as unknown as { runtime: { translator: { dispose(): void; executeBatch(actions: unknown[]): Promise<unknown> } } })
+ .runtime;
+ const dispose = vi.spyOn(runtime.translator, "dispose");
+ let release!: () => void;
+ const blocked = new Promise<void>((resolve) => {
+ release = resolve;
+ });
+ vi.spyOn(runtime.translator, "executeBatch").mockImplementation(async () => {
+ await blocked;
+ return {
+ readResults: [{ type: "screenshot", data: tinyPng, mimeType: "image/png" }],
+ };
+ });
+ const click = agent.state.tools.find((tool) => tool.name === "click");
+ expect(click).toBeDefined();
+
+ const pending = click!.execute("tool-1", { x: 10, y: 20 });
+ await Promise.resolve();
+ agent.setMode("browser");
+ expect(dispose).not.toHaveBeenCalled();
+
+ release();
+ await pending;
+ expect(dispose).toHaveBeenCalledTimes(1);
+ });
+
it("rejects setMode conflicting with a configured native tool", () => {
const agent = new CuaAgent({
browser,You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit e8207b1. Configure here.
The translator only depends on the provider's coordinate system and screenshot transform, both mode-independent, so replacing it on setMode destroyed live CDP state, tabs, and element refs for no reason. Switches now rebuild the translator only when a model change alters that config (e.g. pixel vs normalized coordinates); mode switches keep it. Drop the translator's dead mode option.
rgarcia
commented
Jul 9, 2026
rgarcia
left a comment
rgarcia
left a comment
Contributor
Author
There was a problem hiding this comment.
thermonuclear review: not merge-ready. two correctness failures fall directly out of the current state model, and the 1.2k-line browser executor needs decomposition before this surface grows further. the tests pass, but they do not assert the key invariants these paths need to preserve.
| // the still-exposed pre-switch tools keep executing against it. | ||
| private previousRuntime?: { spec: CuaRuntimeSpec; translator: InternalComputerTranslator; mode?: CuaMode }; | ||
|
|
||
| private beginSwitch(spec: CuaRuntimeSpec): void { |
Contributor
Author
There was a problem hiding this comment.
[P1] this two-phase controller is not safe under overlapping public setMode/setModel calls. there is one shared previousRuntime slot, while each caller independently awaits super.setTools. in the existing overlapping-mode test, if the first browser switch is blocked and the hybrid switch finishes first, the first call later installs its browser tools after the runtime has committed hybrid. worse, those browser tools wrap the translator that the second switch already disposed. i verified this by adding expect(harness.getTools().map(t => t.name)).toContain("computer_click") to that test; it fails while getMode() still reports hybrid. serialize the entire async transition (resolve/build -> session write -> install -> commit), and represent the candidate runtime as an immutable bundle owned by that transition rather than mutating shared controller fields before the await. the test should assert mode, installed tool names, active tool names, model, and that an installed tool's executor is still live.
| this.refCounter = Math.max(this.refCounter, state.refCounter); | ||
| this.activeTargetId = state.activeTargetId ?? this.activeTargetId; | ||
| for (const [frameId, generation] of state.generations) this.generations.set(frameId, generation); | ||
| for (const [ref, entry] of state.refs) this.refs.set(ref, { ...entry, sessionId: "" }); |
Contributor
Author
There was a problem hiding this comment.
[P1] persisted refs are trusted without proving that they still belong to the same document. no executor is connected while CLI invocations are apart, so a navigation between invocations cannot bump these persisted generation counters. after import, an old backend id fails and healEntry can silently retarget the ref to a same-role/same-name/same-nth element on a completely different page. i reproduced this with e1 pointing to button Save/backend 42, then importing into a new document containing button Save/backend 99: click e1 succeeds instead of returning stale. persist a document identity (e.g. main-frame loader/document identity per target/frame) and validate it lazily against CDP before any imported ref resolves; invalidate on mismatch. url alone is not enough because a full reload can keep the same url. add a cross-document import test, not only the current same-tree import test.
| * proceed), confirm and prompt dialogs are dismissed. The dialog message | ||
| * is surfaced as an extra read result on the next executed action. | ||
| */ | ||
| export class BrowserExecutor { |
Contributor
Author
There was a problem hiding this comment.
[P2] this adds a 1,179-line executor in one shot. it owns target/session attachment, frame lifecycle, navigation invalidation, persistent ref identity and healing, AX-tree fetching/stitching/rendering, find scoring, input dispatch, tabs, dialogs, screenshotting, and key translation. those concerns already have distinct state boundaries, but they are coupled through one 48-method class, which is why fixes keep adding flags/sets and event special cases here. please decompose before merging: a target/session registry, a ref/document registry, an AX snapshot+find component, and a browser input/navigation driver, with BrowserExecutor reduced to dispatch/orchestration. this is not a cosmetic file split: the ref registry should own document identity and invalidation, and the target registry should be the sole owner of attach/detach/active-target state. that would remove several current cross-cutting invariants instead of documenting them inside the monolith.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a
CuaMode(computer|browser|hybrid) selecting which canonical action plane(s) the model sees, a CDP-driven browser action plane, and opt-in support for Anthropic's nativecomputer_20260701andbrowser_20260701tools. Mode names, tool prefixes, and Anthropic's native tool vocabulary all align:--mode computerpairs with the computer tool,--mode browserwith the browser tool.Action planes (cua-ai)
The canonical vocabulary is split in code by plane:
packages/ai/src/actions/computer.ts— OS-level input (the existing actions, plus a newzoomscreenshot-crop action). Coordinates are OS screenshot pixels.packages/ai/src/actions/browser.ts— newbrowser_*actions executed over CDP:browser_snapshot(a11y tree with[eN]element refs),browser_text,browser_find,browser_click,browser_fill,browser_scroll_to,browser_navigate, tabs, viewportbrowser_screenshot, and gatedbrowser_evaluate.Modes
computer(default)browserbrowser_stripped (snapshot,click, …)hybridcomputer_*+ ref-onlybrowser_*Hybrid dedup: navigation/tabs live on the browser plane, pointer/keyboard and the only screenshot on the computer plane, and browser tools are ref-only so exactly one coordinate frame is live. Tool descriptions carry the arbitration rules (OS input preferred for stealth; browser tools for reading/locating and hard-to-hit elements).
Modes can be switched at runtime:
CuaAgent.setMode()/CuaAgentHarness.setMode()(+getMode()), surfaced in the TUI as/mode <computer|browser|hybrid>with autocomplete. Switching to a mode that conflicts with a configured native tool throws and rolls back.Native tools
resolveCuaRuntimeSpec(model, { nativeTool })accepts an exportedCuaNativeToolSpec(AnthropicComputerNativeTool/AnthropicBrowserNativeTool). Validation pairs each native tool with its like-named mode (computer_20260701⇢computer,browser_20260701⇢browser; hybrid throws, mirroring the API's one-frame rule). Native runs route to CUA-owned api ids; the registered anthropic provider dispatches them to pi's builtinanthropic-messagestransport with the rightanthropic-betaheader, anonPayloadhook swaps the placeholder tool for the native declaration, and incomingtool_useinputs map onto the same canonical actions the mode uses — canonical vs native is purely a wire-format difference.Execution (cua-agent)
translator/cdp.ts: minimal raw-CDP websocket client (no Playwright, no new deps).translator/browser.ts:BrowserExecutor— a11y snapshots with snapshot-scoped refs (stale refs return a self-recovering error string), CDP input dispatch, navigation/history, tabs with Tab Context blocks,Runtime.evaluate.zoomcrops the OS screenshot locally via sharp; coordinates stay in the full-screenshot frame.CLI
--mode computer|browser|hybrid,--native-tool computer_20260701|browser_20260701,--js-exec, and the/modeslash command.Compatibility
computermode remains the default and its tool names/descriptions are unchanged. Two additive schema changes:clickgains optionalnum_clicks;mouse_down/mouse_upcoordinates become optional (omitted = current cursor position, needed by the native mapping).CUA_ACTION_TYPESkeeps its meaning (the default computer-mode set); the full vocabulary isCUA_COMPUTER_ACTION_TYPES/CUA_BROWSER_ACTION_TYPES.Testing
packages/ai141/141,packages/agent39 passed/12 skipped,packages/cli37 passed/5 skipped).claude-opus-4-8(see PR comment for the full matrix): all five configs pass — canonicalcomputer/browser/hybrid, nativebrowser_20260701, and nativecomputer_20260701. Hybrid correctly mixesbrowser_navigate/browser_snapshot/browser_click-by-ref withcomputer_wait; the native computer run drove the task with pure OS-level input (screenshot → click omnibox → type URL → enter). Note: the early-access PDF documentscomputer_20260601/computer-use-2026-06-01, but the API shipscomputer_20260701/computer-use-2026-07-01— the 0601 tag is not accepted under any header.Type.Tupleemits draft-07items: [...], which Anthropic's draft-2020-12 validation rejects — region params are bounded number arrays instead.packages/agent/examples/anthropic-native-smoke.ts(CONFIG=computer|browser|hybrid|native-computer|native-browser).Follow-ups
browser_snapshotoutput.browser_clickvia ref-resolved OS-level input (stealth-preserving) as an alternative to CDP dispatch.goto/back/urlthrough CDP instead of keyboard hacks.Note
High Risk
Large new CDP execution surface and runtime mode/native-tool switching affect how every agent turn drives the browser; plus a breaking constructor option removal (
computerUseExtra).Overview
0.5.0 introduces action planes selected by
CuaMode(computerdefault,browser,hybrid): OS-level computer tools vs newbrowser_*canonical actions (a11y snapshots with[eN]refs, find/fill/navigate/tabs, optionalbrowser_evaluate).@onkernel/cua-aisplits actions intocomputer.ts/browser.ts, resolves per-mode tool names/schemas/prompts, and supportsnativeTool(computer_20260701/browser_20260701) with Anthropic beta headers and native→canonical mapping.@onkernel/cua-agentadds raw-CDPCdpConnectionandBrowserExecutor(~1.2k lines: ref lifecycle, iframe/OOPIF stitching, dialog guard, ref export/import), routes browser actions throughInternalComputerTranslator, and addssetMode()/getMode()with two-phase runtime switches that preserve CDP state when possible. Breaking:computerUseExtrais removed;computer_use_extrais always registered and is mode-aware (CDP navigation + viewport grounding in browser/hybrid). Post-action screenshots use the viewport in browser mode;zoomcrops the OS display.Docs, changelogs, CLI/TUI flags (
--mode, native tool), exports, and broad unit tests accompany the bump (cua-agent/cua-ai0.5.0,cua-cli0.3.0).Reviewed by Cursor Bugbot for commit f02a6e6. Bugbot is set up for automated code reviews on this repo. Configure here.