Skip to content

fix(ios): open the sole-screen gesture route on a full-screen iPad app - #3918

Open
thymikee wants to merge 4 commits into
mainfrom
fix/ipad-deep-tree-interaction
Open

thymikee wants to merge 4 commits into
mainfrom
fix/ipad-deep-tree-interaction

Conversation

@thymikee

@thymikee thymikee commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Summary

Taps, long presses, scroll/swipe/gesture-viewport, and the back fallback fail on an iPad simulator when the app's accessibility tree is too deep for XCTest: the sole-screen synthesized-gesture route accepted only an iPhone, though iPad hosts run the same hostile trees. RunnerFullScreenEvidenceFromForeground now decides the proof in-process by comparing the AXRuntime primary app's frame against the system app's frame — the same rule the snapshot bridge refuses a windowed capture on (app-window-not-full-screen, ADR 0004); neither read walks the app's tree. The comparison, tolerance, foreground-element protocol, and primary-pid check are one AGENT_DEVICE_FULL_SCREEN_RULE block both sources carry, pinned byte-identical by test/repo-guards/full-screen-rule-parity.test.ts (the two builds' packaged layouts are disjoint, so a shared include cannot reach both; see the review reply). RunnerResolveSoleScreen keeps one owner of the rule, early-returns on the phone idiom before probing, and opens on iPad only on fills-screen; windowed/Stage Manager or unknown keeps requiring a window. The back fallback point derives from the screen's top safe-area inset via one formula for every route. The nightly workflow gains an iPad matrix leg inside the existing job (no parallel job).

Validation

Head f4944b5. Runner unit tests (unit-test variant, own derived path): 12/12 on Review 3918 iPhone, 12/12 on Review 3918 iPad; the iPad log shows AGENT_DEVICE_RUNNER_FULL_SCREEN_EVIDENCE result=fills-screen ... window=(0,0 834x1210) screen=(0,0 834x1210) and the SpringBoard negative result=not-primary-app. macOS host lane: the five decision-seam tests pass.

Live, Release fixture app (cached artifact + JS repack), Settings → Open heavy feed, iPad Pro 11":

command base head
press id="heavy-post-0-like" 16.7s, point.x != INFINITY 1s, Liked 0→1
press 561 520 24.8s, same 1s, Tapped
longpress 417 600 800 24.7s, same 1s, Last action: held post 1
scroll down --pixels 500 7.0s, no interaction frame 1s, Scrolled
swipe 417 900 417 400 3.3s, viewport unavailable 1s, Flung
back refused 1s, feed list

iPhone 17 Pro: back from the heavy feed works; the fallback logs AGENT_DEVICE_RUNNER_BACK_FALLBACK_TOP_INSET read=62.0 windows=1 (point y=84, inside the bar band 62-116; no clamp), and the session log has zero probe lines — the phone never probes. Windowed negative: the executable refusal (not-primary-app → no sole screen) is asserted in testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen on the iPad lane. pnpm check:xctest-selection, the parity guard, and the darwin -Werror bridge compile pass; check:affected --run --head f4944b5 receipt on the review thread. The nightly iPad leg is new — its first run is a CI obligation.

A deep app tree makes XCTest's windows query fail with kAXErrorIllegalArgument,
and the synthesized-gesture route that recovers from it accepted only an iPhone:
the runner refused to name a screen it could not prove the app filled, so every
tap, long press, scroll, swipe, and back fallback on a hostile-tree iPad app
failed. iPad hosts run the same hostile trees as iPhone hosts.

RunnerApplicationFillsScreen now answers that proof in-process: it compares the
AXRuntime primary app's frame against the system app's frame, the same
comparison the Simulator snapshot bridge refuses a windowed capture on
(app-window-not-full-screen, ADR 0004). Neither read walks the app's tree. The
sole-screen rule keeps one owner (RunnerResolveSoleScreen) and opens on an iPad
only while the evidence says fills-screen; a windowed or Stage Manager app, and
any unknown, keeps requiring a resolved window.

The back fallback point now derives from the screen's own top safe-area inset
instead of a frame-height fraction, which sat below the iPad's navigation bar.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-3918/

Built to branch gh-pages at 2026-10-24 07:58 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.58 MB 5.59 MB +13.8 kB
Package (unpacked) 5.58 MB 5.59 MB +13.8 kB
Package (download) 1.69 MB 1.69 MB +3.8 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 40.0 ms 38.6 ms -1.4 ms
CLI --help 82.5 ms 80.3 ms -2.2 ms

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for this. The full-screen iPad route looks right at b18479a, but one change in the resolved-window path needs a decision before merge. All 19 reported checks pass. The PR CI runs only the iPhone runner lanes, and the new ipad-sole-screen job is nightly, so it has not run yet.

The new if frame.minY > 1 { y = frame.minY + 22 } branch in topLeadingNavigationFallbackPoint changes the back point on the resolved-window route. That route runs whenever the tree is readable but no navigation-bar button is hittable. Before, every route used the single inset-based formula from #3894. Now a window whose frame starts below the screen top gets a point at minY + 22. On an iPadOS 26 Windowed Apps or Stage Manager window, that is likely the window-controls strip (close, minimize, tile), not the app's back slot. So back could close or minimize the window. This is outside the linked requirement, which covers the full-screen sole-screen route. The PR's windowed negative used the deep tree, so it never reached this branch, and I could not tell whether XCTest reports a non-zero window origin there at all. Can you drop the branch so one inset-based formula serves every route? If you keep it, please add a live run on an iPad in Windowed Apps mode with a readable-tree screen that has no hittable navigation-bar button. The run should show the resolved window frame, the tapped point, and a performed back that does not hit a window control.

The iPad full-screen, windowed deep-tree and iPhone results in the PR body match 29ecf0f9, which is identical to this head. They have no attached logs, so I could not check them.

Not blocking, and fine to take or leave: (1) in RunnerXCTestEventBridge.m, loggedFills is set on any logged result, so if the first probe in a process answers windowed, not-primary-app or unknown, later fills-screen results are never logged, and the nightly grep depends on that line; set it only in the fills-screen case. (2) The frame comparison, 0.5 tolerance, foreground-element protocol and primary-pid check at RunnerXCTestEventBridge.m:234 copy SnapshotBridgeRuntime.m, and nothing pins that they match; a header-only file that both targets include would keep one copy, with each caller keeping its own unknown policy, and if the bridge's clang argv and source list cannot reach it, a check that pins both copies would do. (3) The negative in RunnerTests+NavigationTests.swift:56 now passes through the sole screen on an iPad, so no test covers windows after a rejected query; drive it through a target the probe cannot prove, such as springboard, or rename it and drop the comment. (4) screenTopSafeAreaInset at RunnerTests+Navigation.swift:219 reads the scene in the runner process; if that read is 0, the point clamps to y=56, above the iPhone 17 Pro bar, and the iPad results cannot show this, so please quote the AGENT_DEVICE_RUNNER_BACK_FALLBACK_TOP_INSET read= line from the iPhone back run (it should read 59). (5) The old doc lines at RunnerTests+SynthesizedInteractionTests.swift:160 ("an iPad refuses...") now contradict the new paragraph below them; delete them. (6) In RunnerXCTestEventBridge.m:416, RunnerFullScreenEvidenceUnknown means both "could not prove it" and "iPhone, not probed"; an early return on the phone idiom before probing would keep one meaning.

Before merge, delete the frame.minY > 1 branch or add the windowed-iPad run above, and fix the loggedFills flag.

* origin/main:
  fix(android): bound the dumpsys reads behind dialog and foreground checks (#3936)
  feat(android): detect Fire OS devices as Android TV targets and document Fire TV support (#3923)
  docs(macos): document Electron/Chromium app automation through the accessibility tree (#3932)
  test(capture-kit): encode solid test PNGs unfiltered to stay inside the unit budget (#3933)
  fix: guard bounded lifecycle and claim recovery (#3911)
  fix(ios): clear a long text value wherever the caret lands (#3908)
  fix(snapshot): stop iOS 27 floating-bar containers from covering content (#3912)
  ci: cache and bound the Linux desktop dependency install (#3919)
…a, one rule block, one nightly job

- Drop the frame.minY > 1 branch so one inset-based formula serves every
  back route (a minY+22 point on a windowed iPad is the window-controls strip).
- Set the probe's loggedFills flag only in the fills-screen case so a first
  windowed answer cannot hide the line the nightly grep reads.
- Share the foreground-ownership and frame-fill rule between the runner
  bridge and the snapshot bridge as one byte-identical block, pinned equal by
  test/repo-guards/full-screen-rule-parity.test.ts; delete both copies.
- Early-return on the phone idiom so RunnerFullScreenEvidenceUnknown keeps
  one meaning.
- Reuse the nightly full-suite job for the iPad leg via its matrix instead of
  a parallel job; rename the back fixture test to what it now proves and
  delete the stale iPad-refuses doc lines.
… rule block

Fixes the lint failure at 17ae6b1 and cuts the pinned block to the rule its
two callers actually share, which is the size the review asked for.
@thymikee

Copy link
Copy Markdown
Member Author

All six points resolved at f4944b5. Point by point:

Blocking 1 — frame.minY > 1 branch: deleted. One inset-based formula (min(max(inset + 22, 56), 96)) now serves every route; the doc paragraph and the lowered-window unit case that described the branch are gone with it. No live windowed run was needed since the branch no longer exists.

Blocking 2 — loggedFills: fixed as asked. The flag is now set only in the fills-screen case (loggedFills = loggedFills || evidence == RunnerFullScreenEvidenceFillsScreen), so a first windowed/unknown answer can never spend it. The fills line logs once per process; refusals log every time, since a refusal is what routed a gesture away and is worth a line each time it happens.

Non-blocking 2 — one implementation, with the pinned-equal form. The frame comparison, the 0.5 tolerance, the foreground-element protocol, and the primary-pid check now live in one AGENT_DEVICE_FULL_SCREEN_RULE block that both sources carry; the runner's copies are deleted and both callers call the same AgentDeviceForegroundWindowFillsScreen / AgentDeviceForegroundOwnershipMatches. Each caller keeps its own unknown policy (RunnerFullScreenEvidenceFromForeground vs the bridge's keep-the-capture rule). I used your fallback — a check that pins both copies — because a header cannot reach both builds: the checkout's two targets build from disjoint source roots, and a packaged install moves them further apart — the runner packs to dist/apple/runner/** (package.json files ships dist only), the bridge stays at apple/snapshot-bridge/** and compiles from an exact-source-filename argv (buildSnapshotBridgeCompileArgv, SNAPSHOT_BRIDGE_{SOURCE,COMPILE}_FILENAMES). A relative #include "../snapshot-bridge/…" compiles in the checkout and breaks in the package; making the header shareable means teaching the packager to rewrite includes into two layouts plus the bridge fingerprint/asset lists — a larger diff than the 30-line rule itself. test/repo-guards/full-screen-rule-parity.test.ts pins the two blocks byte-identical (both hash equal; the guard fails with the edit instruction on drift). If you want the header anyway, say so and I'll do the packaging plumbing as its own change.

Non-blocking 3 — the negative test. Took your second option exactly: renamed to testInAppBackAfterARejectedQueryTapsThroughTheRouteThatOpens and dropped the comment — the name now says what it proves (the route that opens carries the tap), and the refusal half lives in testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen via SpringBoard. The nightly selection follows the rename; pnpm check:xctest-selection passes.

Non-blocking 4 — the iPhone TOP_INSET line, quoted from today's back run (iPhone 17 Pro, heavy feed, head f4944b5):

AGENT_DEVICE_RUNNER_BACK_FALLBACK_TOP_INSET read=62.0 windows=1

Reads 62, not 0, so the point lands at y = 62+22 = 84, inside the iPhone bar band (62-116) — no clamp to y=56, and no clamp was needed: the runner-process scene read answers on the phone. back returned from the deep-tree feed (Back, 1s) and the phone's runner log contains zero FULL_SCREEN_EVIDENCE lines — the idiom early-return (point 6) never probes on iPhone.

Non-blocking 5 — stale doc lines deleted ("an iPad refuses…" gone; the remaining paragraph stands alone).

Non-blocking 6 — one meaning for RunnerFullScreenEvidenceUnknown. RunnerResolveSoleScreen now early-returns on the phone idiom before the probe; the probe runs only off the phone and Unknown means only "could not prove". Live: zero probe lines in the iPhone session log above.

Size. Diff 591+/59− → 625+/79− across the review round's deletions and the new parity guard; the parts you asked to cut:

  • .github/workflows/xctest-nightly.yml: the parallel 89-line job is gone; the iPad leg is now two matrix entries + one step inside the existing job (66 lines total, of which ~40 existed before), so toolchain, runner build cache, and artifact steps are shared.
  • AgentDeviceRunnerApp.m: back to +11/−… (net +5, was +23): the navigation-bar fixture keeps its original one-liner, and only the plain button keeps the safe-area anchors the iPad actually needs; the justification comments shrank to what the code cannot show.
  • The duplicated bridge code is deleted, not wrapped: the runner's protocol/tolerance/comparison are gone, the bridge's inline comparison and pid check call the shared block, and the shared block itself is 30 lines. The size report's +14.8 kB is dominated by the nightly workflow and the block's comments in two shipped sources — the block's comments are now a fifth of what they were, and the fixture-host growth is gone.

Evidence at f4944b5 (owned sims Review 3918 iPhone / Review 3918 iPad, own derived paths and state dirs):

  • Runner unit tests (unit-test variant build into my own AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH): 12/12 pass on iPhone, 12/12 on iPad. The iPad log shows result=fills-screen targetPid=… primaryPid=… window=(0,0 834x1210) screen=(0,0 834x1210) and the SpringBoard negative result=not-primary-app.
  • macOS host lane (macOS-compiled bundle): the five decision-seam tests pass, including both evidence-seam tests that fail without the fills-screen proof.
  • Live iPad, Release fixture app (cached artifact, JS repacked; Settings → Open heavy feed): press id="heavy-post-0-like" 1s Liked 0→1, press 561 520 1s, longpress 417 600 800 1s canary Last action: held post 1, scroll down --pixels 500 1s, swipe 417 900 417 400 1s Flung, back 1s to the feed list. Session runner.log carries the fills-screen line.
  • Live iPhone: back from the heavy feed works; TOP_INSET quoted above; zero probe calls.
  • pnpm check:xctest-selection passes; the parity guard passes; the darwin -Werror bridge conformance compile (49 cases) passes with the shared block; pnpm check:affected --run --head f4944b5 receipt below.

Windowed negative: the executable negative (SpringBoard → not-primary-app refusal, asserted in testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen) passes on the iPad lane; the pre-PR windowed deep-tree live check remains covered by the same app-window-not-full-screen refusal the bridge enforces with the now-shared comparison.

@thymikee

Copy link
Copy Markdown
Member Author

Validation closed at f4944b5: pnpm check:affected --run --head f4944b5c9af7ef6fff1ac03f57e99312bb82caf2 passed all runnable checks with exactCommit: true, and every PR check is green at this head (17 pass, 0 fail). The parity guard, the -Werror bridge conformance compile, and check:xctest-selection are among them.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the update. The earlier findings are fixed, and one new problem remains in the iPad nightly leg at f4944b5.

The merge into the matrix dropped the iPad job's mkdir -p .tmp/xctest-nightly-ipad. The iPad step now pipes xcodebuild into tee .tmp/xctest-nightly/runner.log (workflow line). That directory is created only inside 'Run the full iOS runner XCTest suite', which is skipped on the iPad leg by if: matrix.leg.lane == 'nightly', and no action under .github/actions creates it either. So tee cannot open the file and exits 1, and under set -euo pipefail the step fails before the grep runs. The 'iPad sole-screen evidence' leg would then fail every night, whatever the runner code does. That leg is the only executable guard on the iPad half of the rule, and a real revert to idiom-only routing would look the same as this harness failure. The rule is that every step that writes under .tmp/xctest-nightly creates the directory in its own leg, so please add the mkdir -p at the start of the iPad step, or move the existing one into a step that both legs run. Then run the workflow once with workflow_dispatch, check that the iPad leg reaches the grep, and quote the matched fills-screen line.

Not blocking, and fine to take or leave: after the early return on the phone idiom, the only production call is RunnerSoleScreenRouteOpensForEvidence(NO, evidence) (RunnerXCTestEventBridge.m), so the isPhoneIdiom parameter is reached only from the unit test, and dropping it would leave one statement of the phone rule.

CI is green at f4944b5, with 19 checks and none failing. PR CI runs only the iPhone runner lanes, and the changed iPad leg runs only nightly or by workflow_dispatch, so no PR check exercises it yet. The finding above comes from reading the workflow, not from a failed run. I took the quoted iPad fills-screen line, the live iPad command table and the iPhone TOP_INSET read=62.0 line as reported, since no logs are attached. There are no conflicts. Before merge, the iPad leg needs its directory fix and one successful dispatch run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant