You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 377e81b
Browse filesBrowse the repository at this point in the historyBrowse files
fix(computer-use): type after a focus click only once the caret is confirmed at the end (#10019)
* fix(computer-use): type after a focus click only once the caret is confirmed at the end
The click-to-focus fallback typed at the click point when the editor exposed no
settable selection range or no string value, or when focus settled only after its
poll. It now reads the editor's length (value or AXNumberOfCharacters), accepts an
empty editor or a caret already at the end, verifies a caret move by reading the
range back, and otherwise refuses with caret_unconfirmed before any keystroke. An
input_sequence that starts with Cmd+A skips the check, since it replaces the value.
* test(computer-use): cover the caret_unconfirmed refusal with an editor that ignores caret moves
throwComputerError("caret_unconfirmed","The editor was clicked into focus, but its caret could not be moved to the end, so no text was sent. Observe the app, then use set_value or move the caret with press_key.")
throwComputerError("caret_unconfirmed","The editor was clicked into focus, but its caret could not be confirmed at the end, so no text was sent. Observe the app, then use set_value, or start an input_sequence with Cmd+A.")
647
639
}
640
+
throwComputerError("caret_unconfirmed","The editor was clicked, but it did not take keyboard focus in time, so no text was sent. Observe the app before retrying.")
641
+
}
642
+
/// Whether the caret is at the end of `editor`'s text, moving it there when the editor allows: an empty
643
+
/// editor qualifies, and a move counts only once the selection range reads back at the end.
Copy file name to clipboardExpand all lines: apps/desktop/native/computer-use/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ Coordinates in action inputs are window-local points; node frames are global scr
14
14
15
15
AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and the app that receives the point; the topmost window-list entry is not the click target because Notification Center and the Dock keep full-screen click-through windows. A hit on a window's own background counts as that window. Background tabs and minimized windows share bounds with the window in front, so they take no coordinate input. An action that opens UI (AXPress, AXPick, AXShowMenu) and times out while the app shows a sheet, modal or menu (`cannotComplete`) counts as dispatched; any other action that times out fails as uncertain. A menu-item press that leaves the item's own menu open with the app highlighting that item (Calendar on macOS 26) is completed with Return. Sheets, including Go to Folder and out-of-process open and save panels whose contents report no window, count as the window they are attached to. Menu item labels carry their shortcuts (for example ⇧⌘N). Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
16
16
17
-
`type_text` and `input_sequence` target a text control (text fields and areas, terminals, combo boxes, date and time pickers, or anything with an editable text selection), or a container whose focused editor already holds keyboard focus, such as a web composer; keystrokes are sent only while focus is in a text input. When an app ignores the Accessibility focus request (SwiftUI sheets such as Clock's city search), a foreground app's editor is clicked at its center to take focus after 250 ms without it, only if that point hit-tests to the editor (or a non-secure text control inside it) within its observed window; a click refused by the mouse guards sends nothing, the caret is moved to the end so typing inserts there, and the usual focus checks then apply. `input_sequence` executes 1–32 key/text steps against one such element. It validates every step before dispatch and checks the same editor, window, and secure-input ancestry between steps. Enter can continue into another step when the same editor and window retain focus. Actual focus loss, Tab, or navigation shortcuts end the sequence before later input. Explicit `activateFirst: true` activates the target app inside the sequence before input; it never enables automatic refocusing between steps. Partial execution reports completed steps and an error; callers must observe before retrying. `batch` runs 2–8 click, type_text, set_value, press_key, scroll or perform_action steps planned from one snapshot, each through its own action's checks; it stops at the first step that fails and after any step that changes the app's window structure (on-screen windows, which include sheets, menus and popovers, or the focused window's direct children), reports progress the same way, and a first step that sent nothing is refused with `dispatchState: "not_started"`. Every step is validated before the first one runs. The structure check is a heuristic: it misses modals inside web content and UI that appears after the step's render wait, and stops conservatively on tooltips and autocomplete popovers. Every mutation is followed by fresh state under the same authorization and queue entry; `observeAfter` optionally requests a screenshot. Observation failure is reported separately so the mutation is never replayed. The model receives a compact, bounded projection of the native snapshot; omitted nodes are reported explicitly.
17
+
`type_text` and `input_sequence` target a text control (text fields and areas, terminals, combo boxes, date and time pickers, or anything with an editable text selection), or a container whose focused editor already holds keyboard focus, such as a web composer; keystrokes are sent only while focus is in a text input. When an app ignores the Accessibility focus request (SwiftUI sheets such as Clock's city search), a foreground app's editor is clicked at its center to take focus after 250 ms without it, only if that point hit-tests to the editor (or a non-secure text control inside it) within its observed window; a click refused by the mouse guards sends nothing, the caret is moved to the end and read back so typing inserts there (an empty editor needs no move, and an input_sequence starting with Cmd+A skips the check because it replaces the value); if the caret cannot be confirmed at the end, or focus does not settle, nothing is typed and the call fails with `caret_unconfirmed`. The usual focus checks then apply. `input_sequence` executes 1–32 key/text steps against one such element. It validates every step before dispatch and checks the same editor, window, and secure-input ancestry between steps. Enter can continue into another step when the same editor and window retain focus. Actual focus loss, Tab, or navigation shortcuts end the sequence before later input. Explicit `activateFirst: true` activates the target app inside the sequence before input; it never enables automatic refocusing between steps. Partial execution reports completed steps and an error; callers must observe before retrying. `batch` runs 2–8 click, type_text, set_value, press_key, scroll or perform_action steps planned from one snapshot, each through its own action's checks; it stops at the first step that fails and after any step that changes the app's window structure (on-screen windows, which include sheets, menus and popovers, or the focused window's direct children), reports progress the same way, and a first step that sent nothing is refused with `dispatchState: "not_started"`. Every step is validated before the first one runs. The structure check is a heuristic: it misses modals inside web content and UI that appears after the step's render wait, and stops conservatively on tooltips and autocomplete popovers. Every mutation is followed by fresh state under the same authorization and queue entry; `observeAfter` optionally requests a screenshot. Observation failure is reported separately so the mutation is never replayed. The model receives a compact, bounded projection of the native snapshot; omitted nodes are reported explicitly.
assertfind(state(), role='AXTextField', label='Fixture fixed-caret field').get('value') ==before_text, 'Text reached an editor whose caret was not confirmed'
195
+
# Select-all first makes the caret position moot: the click-focused editor's whole value is replaced.
assertfind(state(), role='AXTextField', label='Fixture click-focus field').get('value') =='replaced', 'Select-all sequence did not replace the click-focused editor'
186
202
# Move focus to the other editor, then put another app in front: the click-only editor cannot take focus.
assertfind(state(), role='AXTextField', label='Fixture click-focus field').get('value') =='clicked twice', 'A background app was clicked or typed into'
210
+
assertfind(state(), role='AXTextField', label='Fixture click-focus field').get('value') =='replaced', 'A background app was clicked or typed into'
195
211
call('activate_app', bundleId=bundle)
196
-
print('PASS: real fixture state, secure suppression, AX click/set/action, Unicode typing, keyboard chord, coordinate drag and scrolling, text refused by a slider, typing into a date picker, a window target typing into its focused editor, a title-bar click, a batch that applies, stops at a failing step, refuses before dispatch and stops when a sheet opens, click-to-focus for an editor that ignores Accessibility focus (and none in the background); every mutation verified in fresh state; screenshot='+ ('verified'ifpermissions['screenRecording'] else'BLOCKED'))
212
+
print('PASS: real fixture state, secure suppression, AX click/set/action, Unicode typing, keyboard chord, coordinate drag and scrolling, text refused by a slider, typing into a date picker, a window target typing into its focused editor, a title-bar click, a batch that applies, stops at a failing step, refuses before dispatch and stops when a sheet opens, click-to-focus for an editor that ignores Accessibility focus (inserting at the end, refusing an unconfirmed caret, and none in the background); every mutation verified in fresh state; screenshot='+ ('verified'ifpermissions['screenRecording'] else'BLOCKED'))
0 commit comments