Skip to content

Commit be9ea16

Browse files
committed
fix(desktop): support rich editors and verified Mac input batches
1 parent 3cca741 commit be9ea16

25 files changed

Lines changed: 1923 additions & 102 deletions

File tree

‎apps/desktop/native/computer-use/ComputerUse.swift‎

Lines changed: 263 additions & 45 deletions
Large diffs are not rendered by default.

‎apps/desktop/native/computer-use/README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,16 @@ Compile each architecture with `swiftc -parse-as-library -O -target arm64-apple-
66

77
## Protocol
88

9-
Persistent newline-delimited JSON over stdin/stdout: `{id,method,params}` returns `{id,result}` or `{id,error:{code,message}}`. The producer contract is `worker/packages/contracts/src/computer-use.ts` in mothership, copied into the desktop bridge by contract sync. `request_permission` is a local onboarding method, never a model tool; params are `{permission:"accessibility"|"screenCapture"}`. Status/preflight never triggers prompts. Errors go through the protocol, and stdout contains no logs.
9+
Persistent newline-delimited JSON over stdin/stdout: `{id,method,params}` returns `{id,result}` or `{id,error:{code,message,dispatchState?}}`. The producer contract is `worker/packages/contracts/src/computer-use.ts` in mothership, copied into the desktop bridge by contract sync. `request_permission` is a local onboarding method, never a model tool; params are `{permission:"accessibility"|"screenCapture"}`. Status/preflight never triggers prompts. Errors go through the protocol, and stdout contains no logs.
1010

11-
The helper observes up to 500 accessibility nodes (depth 15, queue 1000, traversal deadline 8 seconds, each AX call timeout 250ms), 100 windows and 1000 apps. Requests are bounded to 128 KiB during ingestion. Text/value input is capped at 32000 UTF-16 units. State references are per-process, per-app, expire after 60 seconds, and are consumed by every attempted action. Restarts invalidate all references. Screen captures are selected-window only and capped to 1600 pixels on the longest side and 8 MiB encoded. Screenshots are optional and failure is reported in `screenshotError` without discarding accessibility state.
11+
The helper observes up to 2000 accessibility nodes (depth 64, queue 4000, traversal deadline 8 seconds, each AX call timeout 250ms). Selected-window and focused-element ancestry are prioritized; menu traversal is deferred and capped at 80 nodes. Electron accessibility is enabled using its documented `AXManualAccessibility` attribute. The helper observes 100 windows and 1000 apps. Requests are bounded to 128 KiB during ingestion. Text/value input is capped at 32000 UTF-16 units. State references are per-process, per-app, expire after 60 seconds, and are consumed by every attempted action. Restarts invalidate all references. Screen captures exclude window shadows, preserve the selected window’s coordinate geometry, and are capped to 1600 pixels on the longest side and 8 MiB encoded. Screenshots are optional and failure is reported in `screenshotError` without discarding accessibility state. macOS 26 uses the screenshot-specific capture API; the macOS 14/15 API path includes only the selected window in a display filter and crops its exact frame. That older path requires the window to fit within one display and reports `capture_geometry_unavailable` otherwise.
1212

1313
Coordinates in action inputs are window-local points; node frames are global screen points. Window bounds must still match the observed snapshot. App PID and launch time must match. Secure fields and descendants are unavailable as action targets; secure values and labels are suppressed. Sim's own apps and System Settings are protected targets. `get_app_state` can open an installed bundle ID without activating it. `list_apps` discovers installed apps under `/Applications`, `/System/Applications`, and `~/Applications` (bounded to 5000 filesystem entries and depth 3), merging running regular apps; installed apps omit `pid`.
1414

1515
AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and topmost window at the point. Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
1616

17+
`input_sequence` executes 1–32 key/text steps against one observed editable element. It validates every step before dispatch and checks the same editor, window, and secure-input ancestry between steps. Enter can continue into another step when the same editor and window retain focus. Actual focus loss, Tab, or navigation shortcuts end the sequence before later input. Explicit `activateFirst: true` activates the target app inside the batch before input; it never enables automatic refocusing between steps. Partial execution reports completed steps and an error; callers must observe before retrying. Every mutation is followed by fresh state under the same authorization and queue entry; `observeAfter` optionally requests a screenshot. Observation failure is reported separately so the mutation is never replayed. The model receives a compact, bounded projection of the native snapshot; omitted nodes are reported explicitly.
18+
1719
## Cancellation
1820

1921
SIGTERM/SIGINT sets a cancellation flag. Traversal, typing and drag loops check it; mouse up is sent through a defer before returning from a cancelled drag. Keyboard down/up are paired in one function. The owner should close stdin and send SIGTERM, allowing a grace period before SIGKILL. Foreground drags also stop if the active app or window geometry changes; this does not detect every physical user input within the same app. Forced termination cannot guarantee a final release event; input routines deliberately avoid long waits with held buttons. Cancellation cannot undo an action already dispatched.
@@ -24,6 +26,10 @@ Compile pure native tests with `swiftc -parse-as-library -D COMPUTER_USE_TEST Co
2426

2527
`python3 tests/protocol.py <compiled-helper>` checks persistent framing, bounded-request recovery, Unicode decoding, no-prompt status, invalid methods/permissions, permission gates and clean EOF. `tests/Fixture.swift` compiles into a disposable AppKit app with a text field, secure field, increment button/counter, slider, scroll area and a visual-only marker. `python3 tests/live.py <helper>` requires both grants and verifies actual outcomes. `--allow-missing-screen` explicitly reports screenshots blocked rather than passing them; `--existing-fixture` leaves an existing fixture running for coordinated model trials. Give the fixture bundle ID `com.mothership.computer-use-fixture` when packaging it for integration tests.
2628

29+
`python3 tests/electron-live.py <helper> --baseline-helper <previous-helper>` creates an offline Electron fixture with a rich-text editor at AX depth 47, large sibling/menu lists, a submission counter and a colored geometry marker. The September 24 acceptance run found the composer (the earlier helper did not), submitted five separate numbers in one ten-step batch with explicit activation, stopped Enter/text after a real focus change and Tab/text before typing into another control, and matched all four screenshot marker edges within 1.60 pixels. Both screenshot API paths passed that pixel oracle on macOS 26; execution on macOS 14/15 remains unverified. No external messages are sent by this test.
30+
31+
`python3 tests/background-live.py <helper> --state <fixture-state.json> --report <report.json>` tests a running inactive fixture created by `electron-live.py --launch-only`. The initial foreground-only suite missed Electron reporting an editor as focused while withholding app-level keyboard focus in the background. The helper now distinguishes this proven zero-keyboard-input failure with `activation_required` and `dispatchState: "not_started"`; the regression checks unchanged content, then explicit activation and fresh references produce exactly one local submission. Accessibility value assignment returned success without changing this fixture in either focus mode, so fresh observations remain required to verify it.
32+
2733
Both arm64 and x86_64 compilation passed on September 24, 2026. Protocol tests passed. The strict live fixture suite passed with real Accessibility reads, secure-value suppression, selected-window PNG capture, AX button presses/value edits, Unicode typing, Cmd+A/Backspace, verified explicit activation, coordinate drag moving a slider, and scrolling changing visible content. Every action was checked using fresh state. A separate macOS Calculator smoke test pressed All Clear, 2, Add, 3 and Equals through Accessibility while Calculator remained in the background, verified result 5, and confirmed the foreground app was unchanged. These checks establish tested behavior, not Codex-level app compatibility. Foreground coordinate tests ran on a Mac with multiple displays. Lock-screen operation, arbitrary hidden/private windows, non-US shortcut layouts and broad third-party app compatibility remain unverified.
2834

2935
TCC attribution matters: a helper directly spawned by Codex was attributed to Codex in macOS logs; launching it through LaunchServices made it independently responsible. Ad-hoc signing generates a cdhash-only requirement, so rebuilding invalidates independent grants. Distribution and meaningful onboarding tests must use a stable Developer ID and the real parent app responsibility. Never weaken the designated requirement or edit TCC databases to bypass a grant.

‎apps/desktop/native/computer-use/tests/NativeTests.swift‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ import AppKit
88
}
99
static func main() throws {
1010
let frame = CGRect(x: -1920, y: 120, width: 800, height: 600)
11+
try requireObservedFrame(frame, current: frame)
12+
expectError("stale_window") { try requireObservedFrame(frame, current: frame.offsetBy(dx: 1, dy: 0)) }
13+
expectError("stale_window") { try requireObservedFrame(frame, current: CGRect(origin: frame.origin, size: CGSize(width: 801, height: 600))) }
14+
expectError("stale_window") { try requireObservedFrame(frame, current: CGRect(x: 0, y: 120, width: 800, height: 600)) }
1115
let interior = try mapPoint(frame: frame, x: 20, y: 30)
1216
precondition(interior == CGPoint(x: -1900, y: 150))
1317
let origin = try mapPoint(frame: frame, x: 0, y: 0)
@@ -22,10 +26,36 @@ import AppKit
2226
expectError("unsupported_key") { _ = try parseKey("F99") }
2327
expectError("invalid_arguments") { _ = try parseKey("Hyper+A") }
2428
expectError("invalid_arguments") { _ = try required(Optional<String>.none, "example") }
29+
var queue = AXTraversalQueue<String>(limit: 1000)
30+
queue.append(AXTraversalEntry(element: "menu", parent: nil, depth: 0, menu: true, priority: false))
31+
queue.append(AXTraversalEntry(element: "content", parent: nil, depth: 0, menu: false, priority: false))
32+
queue.append(AXTraversalEntry(element: "focused-editor", parent: "content", depth: 35, menu: false, priority: true))
33+
precondition(queue.next()?.element == "focused-editor")
34+
precondition(queue.next()?.element == "content")
35+
precondition(queue.next()?.element == "menu")
36+
precondition(queue.next() == nil)
37+
var bounded = AXTraversalQueue<Int>(limit: 3)
38+
for index in 0..<10 { bounded.append(AXTraversalEntry(element: index, parent: nil, depth: 0, menu: false, priority: false)) }
39+
precondition(bounded.truncated)
40+
precondition(bounded.next()?.element == 0)
41+
precondition(bounded.next()?.element == 1)
42+
precondition(bounded.next()?.element == 2)
43+
precondition(bounded.next() == nil)
44+
try validateInputSteps([InputStep(action: "press_key", text: nil, key: "Cmd+A"), InputStep(action: "type_text", text: "hello 🌍", key: nil)])
45+
try validateInputSteps((1...5).flatMap { [InputStep(action: "type_text", text: String($0), key: nil), InputStep(action: "press_key", text: nil, key: "Enter")] })
46+
expectError("invalid_arguments") { try validateInputSteps([]) }
47+
expectError("invalid_arguments") { try validateInputSteps(Array(repeating: InputStep(action: "type_text", text: "x", key: nil), count: 33)) }
48+
expectError("invalid_arguments") { try validateInputSteps([InputStep(action: "type_text", text: String(repeating: "🌍", count: 16001), key: nil)]) }
49+
expectError("invalid_arguments") { try validateInputSteps([InputStep(action: "click", text: nil, key: nil)]) }
50+
expectError("unsupported_key") { try validateInputSteps([InputStep(action: "press_key", text: nil, key: "F99")]) }
51+
let tabFocus = try keyMayChangeFocus("Tab"); precondition(tabFocus)
52+
let submitFocus = try keyMayChangeFocus("Enter"); precondition(!submitFocus)
53+
let searchFocus = try keyMayChangeFocus("Cmd+K"); precondition(searchFocus)
54+
let selectFocus = try keyMayChangeFocus("Cmd+A"); precondition(!selectFocus)
2555
cancellationRequested = 1
2656
expectError("cancelled") { try checkCancellation() }
2757
cancellationRequested = 0
2858
try checkCancellation()
29-
print("PASS: negative-display geometry, boundaries, nonfinite coordinates, shortcut modifiers, invalid keys, cancellation")
59+
print("PASS: negative-display geometry, boundaries, nonfinite coordinates, shortcut modifiers, invalid keys, cancellation, focused-editor priority, menu deferral, bounded traversal queue")
3060
}
3161
}
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
#!/usr/bin/env python3
2+
"""Offline Electron regression for background AX value editing and explicit focus recovery.
3+
4+
Start an inactive fixture with electron-live.py <helper> --launch-only --artifacts <new-dir>.
5+
Then run this script with <helper> --state <new-dir>/state.json --report <report.json>.
6+
The fixture must already be inactive; this test never focuses an unrelated app to arrange it.
7+
It submits exactly one synthetic local message, relative to the existing submission baseline.
8+
"""
9+
import argparse
10+
import importlib.util
11+
import json
12+
import pathlib
13+
import time
14+
import uuid
15+
16+
SOURCE = pathlib.Path(__file__).resolve().parent
17+
spec = importlib.util.spec_from_file_location('electron_live', SOURCE / 'electron-live.py')
18+
fixture_tools = importlib.util.module_from_spec(spec)
19+
spec.loader.exec_module(fixture_tools)
20+
21+
22+
def main():
23+
parser = argparse.ArgumentParser(description=__doc__)
24+
parser.add_argument('helper')
25+
parser.add_argument('--state', required=True, type=pathlib.Path,
26+
help='State file of an already running, inactive offline composer fixture')
27+
parser.add_argument('--report', required=True, type=pathlib.Path)
28+
args = parser.parse_args()
29+
helper = fixture_tools.Helper(args.helper)
30+
initial = fixture_tools.fixture_state(args.state)
31+
nonce = 'background-recovery-' + uuid.uuid4().hex[:12]
32+
report = {'passed': False, 'fixturePid': initial['pid'], 'checks': []}
33+
try:
34+
def active():
35+
apps = helper.call('list_apps')['apps']
36+
app = next(app for app in apps if app['bundleId'] == fixture_tools.BUNDLE)
37+
assert app['pid'] == initial['pid'], 'Fixture PID changed'
38+
return app['isActive']
39+
40+
def editor():
41+
state = helper.state()
42+
target = fixture_tools.composer(state)
43+
assert target and target.get('editable'), 'Missing writable rich editor'
44+
return state, target
45+
46+
assert not active(), 'Fixture must start in the background'
47+
snapshot, target = editor()
48+
reply = helper.reply('input_sequence', bundleId=fixture_tools.BUNDLE,
49+
snapshotId=snapshot['snapshotId'], elementId=target['elementId'],
50+
steps=[{'action': 'type_text', 'text': 'MUST NOT DISPATCH'},
51+
{'action': 'press_key', 'key': 'Enter'}])
52+
assert reply.get('error', {}).get('code') == 'activation_required', reply
53+
assert reply['error'].get('dispatchState') == 'not_started', reply
54+
unchanged = fixture_tools.fixture_state(args.state)
55+
assert all(unchanged[key] == initial[key]
56+
for key in ['composer', 'sink', 'submissions', 'history'])
57+
assert not active(), 'Background rejection activated the fixture'
58+
report['checks'].append('background keyboard input rejected before dispatch with activation_required')
59+
60+
snapshot, target = editor()
61+
helper.call('set_value', bundleId=fixture_tools.BUNDLE,
62+
snapshotId=snapshot['snapshotId'], elementId=target['elementId'], value=nonce)
63+
time.sleep(0.2)
64+
changed = fixture_tools.fixture_state(args.state)
65+
snapshot, target = editor()
66+
background_value_verified = target.get('value') == nonce and changed['composer'] == nonce
67+
assert changed['submissions'] == initial['submissions'] and not active()
68+
report['backgroundSetValueVerified'] = background_value_verified
69+
if background_value_verified:
70+
report['checks'].append('background set_value verified in AX and fixture DOM')
71+
helper.call('set_value', bundleId=fixture_tools.BUNDLE,
72+
snapshotId=snapshot['snapshotId'], elementId=target['elementId'], value=initial['composer'])
73+
fixture_tools.fixture_state(args.state, lambda state: state['composer'] == initial['composer'])
74+
else:
75+
assert changed['composer'] == initial['composer'] and target.get('value') != nonce
76+
report['checks'].append('background AX set_value success was an unverified no-op; no submission or activation')
77+
78+
helper.call('activate_app', bundleId=fixture_tools.BUNDLE)
79+
snapshot, target = editor()
80+
helper.call('set_value', bundleId=fixture_tools.BUNDLE,
81+
snapshotId=snapshot['snapshotId'], elementId=target['elementId'], value=nonce)
82+
time.sleep(0.2)
83+
snapshot, target = editor()
84+
foreground_state = fixture_tools.fixture_state(args.state)
85+
report['foregroundSetValueVerified'] = target.get('value') == nonce and foreground_state['composer'] == nonce
86+
assert foreground_state['submissions'] == initial['submissions']
87+
result = helper.call('input_sequence', bundleId=fixture_tools.BUNDLE,
88+
snapshotId=snapshot['snapshotId'], elementId=target['elementId'],
89+
steps=[{'action': 'press_key', 'key': 'Cmd+A'},
90+
{'action': 'type_text', 'text': nonce},
91+
{'action': 'press_key', 'key': 'Enter'}])
92+
assert result['sequence'] == {'completedSteps': 3, 'totalSteps': 3}, result
93+
final = fixture_tools.fixture_state(args.state,
94+
lambda state: state['submissions'] == initial['submissions'] + 1)
95+
assert final['history'] == initial['history'] + [nonce]
96+
assert final['composer'] == '' and final['sink'] == initial['sink']
97+
time.sleep(0.15)
98+
assert fixture_tools.fixture_state(args.state)['submissions'] == final['submissions']
99+
snapshot, _ = editor()
100+
assert any(node.get('value') == f"Submissions: {final['submissions']}"
101+
or node.get('label') == f"Submissions: {final['submissions']}"
102+
for node in snapshot['nodes'])
103+
report['checks'].append('explicit activation and fresh observation recovered with exactly one local submission')
104+
report.update(passed=True, nonce=nonce, submissionsBefore=initial['submissions'],
105+
submissionsAfter=final['submissions'])
106+
finally:
107+
helper.close()
108+
args.report.parent.mkdir(parents=True, exist_ok=True)
109+
args.report.write_text(json.dumps(report, indent=2))
110+
print(json.dumps(report, indent=2))
111+
112+
113+
if __name__ == '__main__':
114+
main()

0 commit comments

Comments
 (0)