Skip to content

Commit e09144a

Browse files
authored
improvement(computer-use): allow every app by default, protect sensitive apps, and make per-app approval optional (#9980)
* improvement(computer-use): allow every app by default, block password managers, make per-app approval optional, and trim the settings Turning Computer Use on and granting Accessibility and Screen Recording now covers every app except Sim, System Settings and password managers. Approving each app's first use becomes an opt-in setting, and the settings section keeps only the switch, the two macOS permissions and the optional approvals. * fix(computer-use): protect terminals, match protected bundle IDs case-insensitively, and remember unlisted app names * fix(computer-use): protect more terminals, test the protected-app guard, and keep the approval copy for older shells * fix(computer-use): describe ask mode in the settings line and list the protected apps in the helper README
1 parent d8d7c8c commit e09144a

14 files changed

Lines changed: 193 additions & 49 deletions

File tree

‎apps/desktop/README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -239,9 +239,12 @@ Mothership's `computer` tool is available on macOS 14+ once the user opts in und
239239
group with the **Computer Use** restriction. Older desktop builds omit this optional bridge and
240240
continue to work.
241241

242-
The user grants Accessibility and Screen Recording through macOS, then approves each app for the
243-
current task or persistently. Persistent app approvals are listed in Settings and can be revoked.
244-
Sign-out and deployment changes clear computer-use approval and opt-in. The conversation shows the
242+
The user grants Accessibility and Screen Recording through macOS; every app is then available except
243+
Sim, System Settings, password managers and terminals (shell access has its own Terminal control),
244+
which the helper refuses. With **Ask before using a new
245+
app** on, each app's first use is approved for the task or persistently, and persistent approvals are
246+
listed in Settings and can be revoked. Sign-out and deployment changes clear opt-in, the ask choice
247+
and approvals. The conversation shows the
245248
active app/action and a Stop button; **⌘⇧Esc** also stops active work when the shortcut is available.
246249
Stop cancels queued actions, in-flight authorization, approval prompts and the native helper.
247250
Already completed input cannot be undone.

‎apps/desktop/native/computer-use/ComputerUse.swift‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -142,10 +142,26 @@ func elementRect(_ element: AXUIElement) -> CGRect? {
142142
AXValueGetValue(unsafeBitCast(rawSize, to: AXValue.self), .cgSize, &size) else { return nil }
143143
return CGRect(origin: position, size: size)
144144
}
145+
/// Apps computer use never operates, lowercased: macOS security settings, password managers, and
146+
/// terminals, whose shell access has its own Terminal control.
147+
let PROTECTED_APPS: Set<String> = [
148+
"com.apple.systempreferences", "com.apple.keychainaccess", "com.apple.passwords",
149+
"com.1password.1password", "com.agilebits.onepassword7", "com.agilebits.onepassword-osx", "com.bitwarden.desktop",
150+
"org.keepassxc.keepassxc", "com.lastpass.lastpass", "in.sinew.enpass-desktop", "me.proton.pass.electron",
151+
"com.markmcguill.strongbox", "com.markmcguill.strongbox.mac",
152+
"com.apple.terminal", "com.googlecode.iterm2", "dev.warp.warp-stable", "com.mitchellh.ghostty",
153+
"net.kovidgoyal.kitty", "org.alacritty", "com.github.wez.wezterm", "co.zeit.hyper", "org.tabby",
154+
]
155+
/// Whether computer use refuses an app: Sim's own apps and `PROTECTED_APPS`. Bundle identifiers match
156+
/// case-insensitively in Launch Services.
157+
func isProtectedApp(_ bundle: String) -> Bool {
158+
let key = bundle.lowercased()
159+
return key == Bundle.main.bundleIdentifier?.lowercased() || key.hasPrefix("com.simstudio.") || key.hasPrefix("ai.sim.desktop") || PROTECTED_APPS.contains(key)
160+
}
145161
func appFor(_ bundleId: String?, launch: Bool = false) async throws -> NSRunningApplication {
146162
let bundle = try required(bundleId, "bundleId")
147-
guard bundle != Bundle.main.bundleIdentifier, !bundle.hasPrefix("com.simstudio."), !bundle.hasPrefix("ai.sim.desktop"), bundle != "com.apple.systempreferences" else {
148-
throw ComputerError("protected_app", "Computer use cannot control its own app or macOS security settings.")
163+
guard !isProtectedApp(bundle) else {
164+
throw ComputerError("protected_app", "Computer use does not operate Sim, System Settings, password managers, or terminals; use the terminal tool for shell work.")
149165
}
150166
var apps = NSRunningApplication.runningApplications(withBundleIdentifier: bundle).filter { !$0.isTerminated }
151167
if apps.isEmpty && launch {

‎apps/desktop/native/computer-use/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Persistent newline-delimited JSON over stdin/stdout: `{id,method,params}` return
1010

1111
The helper observes up to 2000 accessibility nodes (depth 64, queue 4000, traversal deadline 8 seconds, each AX call timeout 250ms). Selected-window and focused-element ancestry are prioritized. The menu bar and menus are read in place, so an open menu's items and their submenus come first; items of closed menus are read last, capped at 80 menu nodes. Electron accessibility is enabled using its documented `AXManualAccessibility` attribute. The helper observes 100 windows and 1000 apps. Requests are bounded to 128 KiB during ingestion. Text/value input is capped at 32000 UTF-16 units. State references are per-process, per-app, expire after 60 seconds, and are consumed by every action that reaches the app; a request that fails before sending anything reports `dispatchState: "not_started"` and keeps them valid. An Accessibility failure counts as sent unless Accessibility rejected the request before the app saw it (unsupported attribute or action, illegal argument). Restarts invalidate all references. Screen captures exclude window shadows, preserve the selected window’s coordinate geometry, and are capped to 1600 pixels on the longest side and 8 MiB encoded. Complete JSON replies are capped at 16 MiB: oversized observations omit screenshots first, then truncate accessibility nodes, while preserving action outcomes. Screenshots are optional and failure is reported in `screenshotError` without discarding accessibility state. macOS 26 uses the screenshot-specific capture API; the macOS 14/15 API path includes only the selected window in a display filter and crops its exact frame. That older path requires the window to fit within one display and reports `capture_geometry_unavailable` otherwise.
1212

13-
Coordinates in action inputs are window-local points; node frames are global screen points. Coordinate input without a `windowId` targets the window the snapshot selected, and an element target uses its own window; `press_key` with an `elementId` focuses that element first. Window bounds must still match the observed snapshot. App PID and launch time must match. Secure fields and descendants are unavailable as action targets; secure values and labels are suppressed. Sim's own apps and System Settings are protected targets. `get_app_state` can open an installed bundle ID without activating it. `list_apps` discovers installed apps under `/Applications`, `/System/Applications`, and `~/Applications` (bounded to 5000 filesystem entries and depth 3), merging running regular apps; installed apps omit `pid`.
13+
Coordinates in action inputs are window-local points; node frames are global screen points. Coordinate input without a `windowId` targets the window the snapshot selected, and an element target uses its own window; `press_key` with an `elementId` focuses that element first. Window bounds must still match the observed snapshot. App PID and launch time must match. Secure fields and descendants are unavailable as action targets; secure values and labels are suppressed. Sim's own apps, System Settings, password managers and terminals are protected targets (`PROTECTED_APPS`, matched case-insensitively); shell work belongs to the separate Terminal control. `get_app_state` can open an installed bundle ID without activating it. `list_apps` discovers installed apps under `/Applications`, `/System/Applications`, and `~/Applications` (bounded to 5000 filesystem entries and depth 3), merging running regular apps; installed apps omit `pid`.
1414

1515
AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and the app that receives the point; the topmost window-list entry is not the click target because Notification Center and the Dock keep full-screen click-through windows. A hit on a window's own background counts as that window. Background tabs and minimized windows share bounds with the window in front, so they take no coordinate input. An action that opens UI (AXPress, AXPick, AXShowMenu) and times out while the app shows a sheet, modal or menu (`cannotComplete`) counts as dispatched; any other action that times out fails as uncertain. A menu-item press that leaves the item's own menu open with the app highlighting that item (Calendar on macOS 26) is completed with Return. Sheets, including Go to Folder and out-of-process open and save panels whose contents report no window, count as the window they are attached to. Menu item labels carry their shortcuts (for example ⇧⌘N). Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
1616

‎apps/desktop/native/computer-use/tests/NativeTests.swift‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ import AppKit
2626
expectError("unsupported_key") { _ = try parseKey("F99") }
2727
expectError("invalid_arguments") { _ = try parseKey("Hyper+A") }
2828
expectError("invalid_arguments") { _ = try required(Optional<String>.none, "example") }
29+
for bundle in ["com.apple.systempreferences", "com.apple.Terminal", "COM.GOOGLECODE.ITERM2", "net.kovidgoyal.kitty", "com.1password.1password", "com.apple.Passwords", "com.bitwarden.desktop", "ai.sim.desktop", "com.simstudio.computer-use"] {
30+
precondition(isProtectedApp(bundle), bundle)
31+
}
32+
for bundle in ["com.apple.TextEdit", "com.apple.finder", "com.apple.Safari"] { precondition(!isProtectedApp(bundle), bundle) }
2933
var queue = AXTraversalQueue<String>(limit: 1000)
3034
queue.append(AXTraversalEntry(element: "menu", parent: nil, depth: 0, menu: true, priority: false))
3135
queue.append(AXTraversalEntry(element: "content", parent: nil, depth: 0, menu: false, priority: false))

‎apps/desktop/src/main/computer-use/service.test.ts‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,13 @@ afterEach(() => {
1212
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true })
1313
})
1414

15-
function setup(enabled = true) {
15+
/** `askForApps` defaults on so the approval tests exercise the prompt; the default device mode asks for nothing. */
16+
function setup(enabled = true, askForApps = true) {
1617
const root = mkdtempSync(join(tmpdir(), 'sim-computer-use-test-'))
1718
roots.push(root)
1819
const config = createConfigStore(join(root, 'settings.json'))
1920
config.set('computerUseEnabled', enabled)
21+
config.set('computerUseAskForApps', askForApps)
2022
let sequence = 0
2123
const request = vi.fn(async (method: string): Promise<ComputerUseResult> => {
2224
if (method === 'status' || method === 'request_permission')
@@ -412,6 +414,25 @@ describe('native computer use authority and lifecycle', () => {
412414
await expect(click('s1')).rejects.toThrow('stale')
413415
})
414416

417+
it('operates any unprotected app without asking or saving an approval by default', async () => {
418+
const { state, click, service, native, approveApp, onActivity } = setup(true, false)
419+
await state()
420+
await click('s1')
421+
expect(approveApp).not.toHaveBeenCalled()
422+
expect(service.listAppPermissions()).toEqual([])
423+
expect(onActivity).toHaveBeenCalledWith(expect.objectContaining({ appName: 'Fixture' }))
424+
expect(native.request.mock.calls.filter(([method]) => method === 'list_apps')).toHaveLength(1)
425+
})
426+
427+
it('asks for new apps only once the user turns asking on, and clears that choice on reset', async () => {
428+
const { state, service, approveApp } = setup(true, false)
429+
await expect(service.setAskForApps(true)).resolves.toMatchObject({ askForApps: true })
430+
approveApp.mockResolvedValueOnce('deny')
431+
await expect(state()).rejects.toThrow('denied')
432+
service.reset()
433+
expect((await service.getStatus()).askForApps).toBe(false)
434+
})
435+
415436
it('clears grants and opt-in on account reset', async () => {
416437
const { state, service, native } = setup()
417438
await state()

‎apps/desktop/src/main/computer-use/service.ts‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -51,13 +51,19 @@ export class ComputerUseService {
5151
private snapshots = new Map<string, SnapshotOwnership>()
5252
private activity: ComputerUseActivity | null = null
5353
private taskGrants = new Map<string, Map<string, string>>()
54+
/** App names from the helper's app list, so an unprompted action can name its app without listing apps again. */
55+
private appNames = new Map<string, string>()
5456

5557
constructor(private readonly deps: ComputerUseServiceDeps) {}
5658

5759
isEnabled(): boolean {
5860
return this.deps.supported && this.deps.config.get('computerUseEnabled') === true
5961
}
6062

63+
private asksForApps(): boolean {
64+
return this.deps.config.get('computerUseAskForApps') === true
65+
}
66+
6167
getActivity(): ComputerUseActivity | null {
6268
return this.activity
6369
}
@@ -68,6 +74,7 @@ export class ComputerUseService {
6874
enabled: this.isEnabled(),
6975
permissions: { accessibility: false, screenCapture: false },
7076
activeAction: this.activity,
77+
askForApps: this.asksForApps(),
7178
}
7279
if (!status.enabled) return status
7380
const native = await this.deps.native.request('status', {})
@@ -94,6 +101,16 @@ export class ComputerUseService {
94101
return this.getStatus()
95102
}
96103

104+
async setAskForApps(ask: boolean): Promise<ComputerUseStatus> {
105+
const previous = this.asksForApps()
106+
this.deps.config.set('computerUseAskForApps', ask)
107+
if (!this.deps.config.flush()) {
108+
this.deps.config.set('computerUseAskForApps', previous)
109+
throw new Error('Could not save Computer Use settings.')
110+
}
111+
return this.getStatus()
112+
}
113+
97114
async requestPermission(
98115
permission: 'accessibility' | 'screenCapture'
99116
): Promise<ComputerUseStatus> {
@@ -127,7 +144,9 @@ export class ComputerUseService {
127144

128145
reset(): void {
129146
this.cancel()
147+
this.appNames.clear()
130148
this.deps.config.set('computerUseEnabled', false)
149+
this.deps.config.set('computerUseAskForApps', false)
131150
this.deps.config.set('computerUseAllowedApps', [])
132151
if (!this.deps.config.flush())
133152
throw new Error('Could not clear saved Computer Use permissions.')
@@ -338,16 +357,14 @@ export class ComputerUseService {
338357
})
339358
}
340359

360+
/** Every unprotected app is allowed unless the user chose to approve each app's first use. */
341361
private async authorizeApp(bundleId: string, work: ToolWork): Promise<string> {
342362
const approved = this.listAppPermissions().find((app) => app.bundleId === bundleId)
343363
if (approved) return approved.displayName
344364
const taskName = this.taskGrants.get(work.scopeId)?.get(bundleId)
345365
if (taskName) return taskName
346-
const apps = await this.deps.native.request('list_apps', {})
347-
this.check(work)
348-
const app =
349-
apps.kind === 'apps' ? apps.apps.find((app) => app.bundleId === bundleId) : undefined
350-
const target = { bundleId, displayName: app?.name ?? bundleId }
366+
const target = { bundleId, displayName: await this.appName(bundleId, work) }
367+
if (!this.asksForApps()) return target.displayName
351368
const answer = await this.deps.approveApp(target, work.controller.signal)
352369
this.check(work)
353370
if (answer === 'deny')
@@ -371,4 +388,18 @@ export class ComputerUseService {
371388
}
372389
return target.displayName
373390
}
391+
392+
private async appName(bundleId: string, work: ToolWork): Promise<string> {
393+
const known = this.appNames.get(bundleId)
394+
if (known) return known
395+
const apps = await this.deps.native.request('list_apps', {})
396+
this.check(work)
397+
if (apps.kind === 'apps') {
398+
this.appNames.clear()
399+
for (const app of apps.apps) this.appNames.set(app.bundleId, app.name)
400+
}
401+
/** An app outside the scanned folders keeps its bundle ID, so it is not rescanned on every action. */
402+
if (!this.appNames.has(bundleId)) this.appNames.set(bundleId, bundleId)
403+
return this.appNames.get(bundleId) ?? bundleId
404+
}
374405
}

‎apps/desktop/src/main/config.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,8 @@ export interface DesktopSettings {
111111
preventSleepWhileRunning?: boolean
112112
/** Native app access is opt-in and cleared when the signed-in account changes. */
113113
computerUseEnabled?: boolean
114+
/** Ask before operating an app for the first time; by default every unprotected app is allowed. */
115+
computerUseAskForApps?: boolean
114116
computerUseAllowedApps?: Array<{ bundleId: string; displayName: string }>
115117
/** Device-wide browser page appearance; `app` follows Sim. */
116118
browserTheme?: 'app' | 'light' | 'dark'

‎apps/desktop/src/main/ipc.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -583,6 +583,7 @@ describe('registerIpcHandlers', () => {
583583
computerRoots.push(root)
584584
const config = createConfigStore(join(root, 'settings.json'))
585585
config.set('computerUseEnabled', true)
586+
config.set('computerUseAskForApps', true)
586587
const status: ComputerUseResult = {
587588
kind: 'status',
588589
platform: 'darwin',

‎apps/desktop/src/main/ipc.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -751,6 +751,18 @@ export function registerIpcHandlers(deps: IpcDeps): void {
751751
return deps.computerUse?.requestPermission(permission)
752752
},
753753
},
754+
'computer-use:set-ask-for-apps': {
755+
kind: 'invoke',
756+
gate: 'app-origin',
757+
requiresAccountData: true,
758+
needsUserActivation: true,
759+
denied: null,
760+
handler: (ask) => {
761+
if (typeof ask !== 'boolean')
762+
throw new Error('A boolean Computer Use preference is required.')
763+
return deps.computerUse?.setAskForApps(ask)
764+
},
765+
},
754766
'computer-use:list-permissions': {
755767
kind: 'invoke',
756768
gate: 'app-origin',

‎apps/desktop/src/preload/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@ const api: SimDesktopApi = {
131131
setEnabled: (enabled: boolean) => ipcRenderer.invoke('computer-use:set-enabled', enabled),
132132
requestPermission: (permission: 'accessibility' | 'screenCapture') =>
133133
ipcRenderer.invoke('computer-use:request-permission', permission),
134+
setAskForApps: (ask: boolean) => ipcRenderer.invoke('computer-use:set-ask-for-apps', ask),
134135
listAppPermissions: () => ipcRenderer.invoke('computer-use:list-permissions'),
135136
revokeApp: (bundleId: string) => ipcRenderer.invoke('computer-use:revoke-app', bundleId),
136137
executeTool: (toolCallId: string, params: ComputerUseInput) =>

0 commit comments

Comments
 (0)