fix(computer-use): tighten batch and click-to-focus edge cases from the staging audit by waleedlatif1 · Pull Request #10012 · simstudioai/sim

waleedlatif1 · 2026-11-13T21:23:30Z

Summary

Follow-ups from a deep audit of the computer-use changes on staging (#9994, #9995, #9997, #10004):

Batch: every step is validated before step 1 runs (key names, required element/text/value/action, button and clickCount), so a typo never leaves a half-applied batch; a refused first step is reported as "Batch step 1 failed: …" so it is not mistaken for an unsupported batch; the first-step check is completed == 0 && !inputDispatched; cancellation is checked before every later step; dropped dead parameter copying
Batch observation: a batch now observes the one window all of its steps target (any untargeted step keeps the focused-window default), like single element/coordinate actions
Click-to-focus: waits up to 250 ms for an Accessibility focus request before clicking; a click refused by the mouse guards sends nothing and falls through to the focus check; after a click the caret moves to the end so typing inserts there; activation_required no longer claims nothing was dispatched after a click
Accessibility reads: the per-attribute fallback runs only when an element cannot answer a batched read (unsupported / illegal / not implemented), not on timeouts, so a hung app is not asked once per attribute
Projection: a wrapper cycle can no longer make a node its own parent; legend describes label/value/placeholder and omitted parents accurately
Contract synced from mothership (activateFirst wording); README describes the structure check's limits and the new coverage; removed a dead test parameter
Companion: simstudioai/mothership#960

Type of Change

Bug fix

Testing

Native checks on macOS 26 (PASS), including new batch pre-validation cases
Live fixture: new case — a click-only editor that already has text inserts at the end (live run pending a free VM; will confirm before merge)
desktop computer-use + IPC (101), projection (15), mothership-chat suites; new tests (batch window, wrapper cycle) fail on the previous code

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…he staging audit Validate every batch step before the first runs and name a refused first step; observe the window a batch targets; wait 250 ms for Accessibility focus before clicking, treat a refused click as no click and put the caret at the end; never report nothing-dispatched after a click; skip per-attribute fallback reads on a hung app; guard a self-parent in the projection; fix legend and README wording. Syncs the contract's activateFirst wording.

vercel · 2026-11-13T21:23:35Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Nov 13, 2026 9:47pm UTC

github-actions · 2026-11-13T21:23:49Z

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

❌ simstudioai/mothership#960 — OPEN, not merged (targets staging) — fix(computer-use): say activateFirst runs before the sequence and narrow the bat

greptile-apps · 2026-11-13T21:26:59Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with no actionable issues found.

Summary

Tightens batch checks, follow-up observations, click-to-focus behavior, and the tree shown to the model.

Batches reject malformed steps before any app action runs.
Text entry waits for focus, then clicks editors and starts at the end.
Batch results show the window the steps targeted.
Accessibility reads avoid repeating requests after timeouts.
The model's app tree no longer shows a node as its own parent.
Computer Use test setup and generated wording are cleaned up.

_{Reviews (3) · Last reviewed commit: "fix(computer-use): send no text when the..." · Reviewed by Greptile}

cubic-dev-ai

All reported issues were addressed across 10 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

… and scroll targets up front, and keep the escaped-newline row cost

waleedlatif1 · 2026-11-13T21:37:36Z

@greptile

waleedlatif1 · 2026-11-13T21:37:38Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-11-13T21:37:42Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 10 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

… end after a focus click

waleedlatif1 · 2026-11-13T21:47:52Z

@greptile

waleedlatif1 · 2026-11-13T21:47:53Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-11-13T21:47:59Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 10 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Nov 13, 2026

greptile-apps Bot reviewed Nov 13, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/tools/client/computer-tool-result.ts Resolved

cubic-dev-ai Bot reviewed Nov 13, 2026

View reviewed changes

Comment thread apps/desktop/native/computer-use/ComputerUse.swift Resolved

Comment thread apps/desktop/native/computer-use/ComputerUse.swift Resolved

Comment thread apps/sim/lib/mothership/tools/client/computer-tool-result.ts Resolved

fix(computer-use): check Stop before every batch step, validate click…

8ec2773

… and scroll targets up front, and keep the escaped-newline row cost

vercel Bot temporarily deployed to Preview November 13, 2026 21:36 Inactive

cubic-dev-ai Bot reviewed Nov 13, 2026

View reviewed changes

Comment thread apps/desktop/native/computer-use/ComputerUse.swift Resolved

fix(computer-use): send no text when the caret cannot be moved to the…

b3da664

… end after a focus click

vercel Bot temporarily deployed to Preview November 13, 2026 21:47 Inactive

cubic-dev-ai Bot reviewed Nov 13, 2026

View reviewed changes

waleedlatif1 merged commit 06f077d into staging Nov 13, 2026
70 checks passed

waleedlatif1 deleted the fix/computer-use-batch-audit branch November 13, 2026 21:53

waleedlatif1 mentioned this pull request Nov 13, 2026

v0.9.57: cua improvements, Monaco CRM integration, Hex and Stripe validation, keyless subjects held until trust policies apply, AWS Marketplace replica entitlement check, Slack channel search #10017

Merged

Conversation

waleedlatif1 commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 13, 2026

⚠️ Cross-repo companion check

Uh oh!

greptile-apps Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Nov 13, 2026 •

edited

Loading

vercel Bot commented Nov 13, 2026 •

edited

Loading

greptile-apps Bot commented Nov 13, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading