Skip to content

switchify-operating-guide.md

Latest commit

 

History

History
250 lines (178 loc) · 8.71 KB

switchify-operating-guide.md

File metadata and controls

250 lines (178 loc) · 8.71 KB
Raw

Switchify Operating Guide

This guide is included in Codex QA prompts so the child agent can reason about Switchify as a switch-access system, not as raw Android input.

Core Mental Model

Switchify does not drive Android by raw taps or key events. The QA runner sends approved debug bridge actions to the Switchify accessibility service. The service turns those actions into Switchify switch actions, and the active access technique handles the result.

Default runner actions:

next
previous
select
capture
stop

Full-profile bridge actions:

home
back
recents
quick_settings
notifications
lock_screen
media_play_pause
pause
stop_scanning
change_scanning_direction
toggle_gesture_lock
toggle_gesture_lock_rearm
toggle_gesture_repeat
reload_settings

Normal manual scanning maps like this:

  • next -> SwitchAction.ACTION_MOVE_TO_NEXT_ITEM
  • previous -> SwitchAction.ACTION_MOVE_TO_PREVIOUS_ITEM
  • select -> SwitchAction.ACTION_SELECT

Those actions enter ScanningManager.performAction(...), then the active access technique, usually ScanTree.

Useful Android search paths:

  • SwitchAction.kt
  • ScanningManager.kt
  • ScanTree.kt
  • ScanTreeNavigator.kt

First Action In Manual Scan

In manual or directional scan mode, the first scan action may only initialize scanning and highlight the first item. This comes from ScanTree.checkManualScanSetup().

If the first next or select appears to do very little, do not immediately call it a failure. Capture again or keep scanning unless the service is visibly disabled or broken.

Initial setup calls highlightCurrent() and returns without selecting a target.

ScanTree Levels

ScanTree is hierarchical:

tree item / row
  group
    node

Rules:

  • A row or group highlight is not necessarily the final actionable target.
  • Selecting a row or group often drills into it.
  • Selecting a final node activates that item or opens the Switchify menu, depending on auto-select settings.
  • In row-column scan, repeated next moves at the current level until escape or end behavior appears.
  • In non-row-column scan, nodes are flattened and scanned sequentially.
  • In directional mode, movement is spatial and ScanTree escape is disabled.

Useful Android search paths:

  • ScanTree.kt
  • ScanTreeNavigator.kt
  • ScanTreeItem.kt
  • ScanTreeHighlighter.kt

Highlight Colors

Colors come from ScanColorManager.kt. Rendering comes from NodeScannerUI.kt.

Default scan color set:

  • Primary: #2563EB
  • Secondary: #F59E0B

Meaning:

  • Primary is used by NodeScannerUI.showRowBounds(...).
  • Secondary is used by NodeScannerUI.showItemBounds(...).
  • Row and group containers generally use primary.
  • Individual nodes generally use secondary.
  • Single-node rows collapse to node behavior and use secondary.

Other color sets:

  • High Contrast: primary #000000, secondary #FFFFFF
  • Deuteranopia-safe: primary #1E40AF, secondary #FCD34D
  • Tritanopia-safe: primary #DC2626, secondary #0E7490
  • Monochrome: primary #1F2937, secondary #6B7280

Decision rules:

  • Do not assume blue always means row and amber always means item unless the default color set is active.
  • Prefer the shape and extent of the highlight.
  • A large band or rectangle across multiple controls usually means row or group.
  • A tight box around one control usually means actionable node.
  • If color and apparent scan state disagree, record a weird finding instead of guessing confidently.

Escape Behavior

ScanTreeNavigator has an EscapeState:

  • None
  • Item
  • Group

Escape appears when scanning reaches the end of a node, group, or item and movement returns false.

Important behavior:

  • ScanTree.highlightEscape(...) highlights the item or group that would be escaped from.
  • Selecting while escape is active confirms escape.
  • Moving past escape denies or skips escape and continues scanning or cycles.
  • In directional mode, escape is disabled.

Decision rules:

  • If the highlight appears to return to a containing row or group after scanning inside it, this may be an escape step.
  • Selecting escape backs out one ScanTree level.
  • ScanTree escape is not Android Back.
  • Android back is a system action and should not be used to resolve ScanTree escape unless the goal explicitly needs Android Back.

Keyboard Escape

The keyboard has a special cycle-break escape path.

Useful Android search paths:

  • BaseNodeScanner.kt
  • KeyboardManager.kt
  • KeyboardSelectionPolicy.kt
  • KeyboardEscapePrompt.kt

Behavior:

  • When the keyboard is visible and not escaped, KeyboardManager.shouldEnableCycleBreak() enables a cycle break.
  • When a scan cycle completes, BaseNodeScanner.onScanTreeCycleBreakStarted() shows KeyboardEscapePrompt.
  • The prompt text is Press to escape keyboard.
  • Selecting the cycle break calls KeyboardManager.escapeKeyboard().
  • After escaping, the Switchify menu can show Scan Keyboard to return to keyboard scanning.
  • If direct keyboard selection is enabled and the keyboard is visible and not escaped, keyboard keys may be selected directly instead of opening the menu.

Decision rules:

  • If the keyboard is visible and the prompt says Press to escape keyboard, selecting it exits keyboard scanning.
  • After escaping, use Scan Keyboard only if the goal is to return to typing.
  • If trying to act outside the keyboard, escaping is correct.
  • If trying to type or select keys, do not escape unless the current keyboard scan path is blocked.

Switchify Menu Behavior

When auto-select is disabled, selecting a normal screen item usually opens the Switchify accessibility menu instead of immediately tapping the item.

Use next and previous to move the menu highlight. Use select to activate the highlighted menu item.

Common main menu items:

  • Back: Android Back.
  • Home: return to the launcher or home screen.
  • Tap: single tap at the selected cursor or item location. Use this to activate a highlighted app, button, media item, or list item after the menu opens.
  • Gestures: gesture options such as Tap Gestures, Swipe Gestures, Drag, Pinch Gestures, and Finger Mode.
  • Scroll: scroll direction options: Scroll Up, Scroll Down, Scroll Left, Scroll Right.
  • Favourite Apps: configured favourite apps.
  • Device: system actions such as Recent Apps, Notifications, Quick Settings, Lock Screen, Power Dialog, Screenshot, and Volume Control.
  • Settings: Switchify settings actions.
  • Media Control: Play/Pause, Previous Track, Next Track, and volume controls.
  • Pause Switchify: pauses Switchify. Avoid this unless the goal explicitly requires pausing.

Common submenu and navigation items:

  • Close menu: close the Switchify menu without activating a target.
  • Previous menu: go back to the previous Switchify menu.
  • Next page / Previous page: move between pages when a menu has more items than fit on screen.
  • Confirm: confirm and run the pending gesture.
  • Reselect: pick a new gesture endpoint or target.
  • Cancel: cancel the pending gesture or menu operation.
  • Scan Keyboard: return to keyboard scanning after escaping from keyboard.

Decision rules:

  • Do not treat the Switchify menu as a blocker. It is the expected way to act on a selected target when auto-select is disabled.
  • If the goal is to open or activate the currently highlighted launcher, app, content, or control item and the menu is open, navigate to and choose Tap.
  • Do not choose Home to activate an item.
  • Use Home only when the goal explicitly requires returning to launcher or home and the current screen is not already there.
  • Use Back only as Android Back, not as ScanTree escape and not as menu activation.

Common Decision Patterns

If the goal is to open an app and the launcher icon is highlighted:

  • If the menu opens, find Tap.
  • Do not select Home.

If the goal is to leave media playing:

  • media_play_pause can be used only when media is focused or active.
  • If unsure whether media is paused or playing, request capture once before deciding.

If foreground state changes but the screenshot looks stale:

  • Request capture once before declaring a bug.

If scan highlight does not move after next:

  • Record a weird finding.
  • Good suspected areas: ScanTree, ScanningManager, or the ADB bridge action path.

If the service or menu overlay disappears:

  • Record a weird finding.
  • Continue only if the goal is still visibly navigable.

Weird Finding Search Hints

Useful hints for androidRepoSearchHints:

ScanTree.performSelectionAction
ScanTree.stepScanningForward
ScanTreeNavigator.handleEscape
ScanTreeHighlighter.highlightCurrent
NodeScannerUI.showRowBounds
NodeScannerUI.showItemBounds
KeyboardManager.escapeKeyboard
KeyboardSelectionPolicy.shouldEnableCycleBreak
KeyboardEscapePrompt
BaseNodeScanner.onScanTreeCycleBreakStarted
MenuManager
MenuPage