diff --git a/.github/skills/android-e2e-tester/SKILL.md b/.github/skills/android-e2e-tester/SKILL.md new file mode 100644 index 00000000..14612707 --- /dev/null +++ b/.github/skills/android-e2e-tester/SKILL.md @@ -0,0 +1,703 @@ +--- +name: android-e2e-tester +description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on a connected real device or an emulator: provision the device, install the app, drive the UI, verify via logcat, loop fix-and-retest, and — for ADO test cases — always write an HTML + Markdown report. Only use this skill when the user explicitly asks to run an end-to-end test — e.g. 'run the E2E test', 'test this feature end to end', 'verify the feature on a device/emulator', 'does this work on a device', 'run this ADO test case', or 'try the sign-in flow'. Do NOT auto-start it after a feature finishes development, when a coding-agent PR lands, or as an implicit step of any other workflow (including an orchestrated build→test flow) — a human has to ask for an E2E run." +--- + +# Android E2E Tester + +Take an in-development Android Auth feature and prove it works on a real device or an emulator: provision +the device, deploy the app, run the scenario (auto-handling every input the AI reasonably can), read the +logs to decide pass/fail, and loop fix-and-retest until it passes — or stop and ask when genuinely blocked. +For a test case driven from Azure DevOps, always finish by writing an HTML + Markdown test report. + +## When this runs + +- **Only when the user explicitly asks to run an E2E test** — e.g. "run the E2E test", "test this feature + end to end", "verify the feature on a device/emulator", or "run this ADO test case". This skill does **not** + auto-start after a feature finishes development, when a coding-agent PR lands, or as an implicit step of any + other workflow (including an orchestrated build→test flow) — a human has to ask for an E2E run. Once asked, + run Phase 1 to confirm *what* to test before touching the device. + +## Run folder + +Put all artifacts (log snapshots, screenshots, per-iteration notes) **outside the repo** so nothing is +committed: +``` +$env:USERPROFILE\android-e2e-runs\-\ +``` +Pass this as `-Out` to `authlogs.ps1`/`deviceui.ps1`. Reuse the resolved emulator `-Serial` on every +script call so commands target the right device. + +## Scripts + +All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` mentally via the +`.SYNOPSIS` in each file. They self-resolve the SDK; no hardcoded paths. + +| Script | Purpose | Key commands | +|---|---|---| +| `emulator.ps1` | Device pool: emulators **and** real devices | `ensure`, `status`, `list`, `pool`, `list-images`, `create`, `start` | +| `devicelease.ps1` | Lease a device so concurrent tests don't collide | `acquire`, `heartbeat`, `release`, `list`, `reap` | +| `appcontrol.ps1` | App build/install/state | `build`, `install`, `launch`, `clear`, `uninstall`, `is-installed`, `grant`, `list-apks` | +| `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text` (`-Clear`, `-CharByChar`, `-SecretRef`), **`flow`** (run a whole known screen sequence in ONE call), `unlock`, `key`, `finger`, `finger-status`, `finger-enroll`, `screenshot`, `current-app` | +| `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | +| `labapi.ps1` | Provision/reset LAB test accounts (EasyAuth via WAM SSO); fetch tenant passwords from Key Vault | `create-user`, `reset`, `enable-policy`, `disable-policy`, `delete-device`, `open`, `fetch-password` | +| `report.ps1` | Render the HTML + Markdown test report (mandatory for ADO test cases); `summary` = overall multi-case run report | `render` (a per-case run JSON), `summary` (a batch/run folder) | +| `secrets.ps1` | Encrypted (DPAPI) store so passwords/PINs never hit the chat | `set`, `set-device-pin`, `list`, `test`, `get-masked`, `remove`, `path` | + +## Execution model — run inside a sub-agent, and supervise it + +An E2E run is long and chatty (boot, install, dozens of UI dumps/taps, log scans). **Delegate the actual +device-driving to a sub-agent** so the parent's context stays clean and the parent supervises rather than +performs. + +1. **Launch one sub-agent to own the run, on the SAME model as the parent.** If the parent is running on + Claude Opus 4.8, request the sub-agent on Claude Opus 4.8 (match whatever model the parent is on). + Give it a **self-contained** prompt: the scenario + explicit success criterion (Phase 1 output), the + leased `$serial` (or tell it to lease one with **its own agent id** as owner), the app/module + + package, credential-handling rules (type on device, never print), the run-folder path, any + mocks/flights to apply, and an instruction to **finish with a `PASS` / `FAIL` / `BLOCKED` verdict plus + evidence** (success signal + correlation_id, or the exact blocker). +2. **The parent SUPERVISES on a wall clock — it does not merely block on a completion.** Start the + sub-agent in the background, then poll it in a loop with a **bounded** timeout + (`read_agent … wait=true timeout=180`) — **never** a single open-ended wait. **Do not end your turn, and + do not report to the user, until the sub-agent's verdict is in.** Returning early is failure mode #1 (the + sub-agent's result never reaches the user); its evil twin is failure mode #2: **blocking forever on a + wait that can only be woken by a *completion*.** A **hung** sub-agent never completes, so it **never + fires a completion notification** — and because it's frozen inside a blocking device call (classically a + wedged `adb install`) it **cannot fire its own per-point abort cap either**. If your only wake-up is "the + agent finished," a wedged lane hangs the whole turn indefinitely (this is real: two lanes wedged on + `adb install` once stalled a suite run for ~6 hours). So the parent owns the **wall-clock watchdog** in + step 3, and long device calls are bounded (`appcontrol.ps1 install -TimeoutSec `) so a wedge fails + fast instead of freezing the agent. +3. **Watchdog — detect a STALL on the wall clock, not via notifications:** + - **Track liveness yourself every poll.** The sub-agent emits a progress line + refreshes its lease + (`devicelease.ps1 heartbeat`) at each phase. Between bounded `read_agent` polls, tail its + `tc\progress.log` and note `tool_calls_completed`. **Growth in *either* = alive; no growth in + *both* across two consecutive bounded polls (~6–10 min of zero forward motion) = STALLED**, whether or + not a completion ever arrives. + - **Stalled → abort the lane, don't keep waiting.** Treat a stall as a dead lane: mark the case + **ABORTED** from its **partial** `run.json` (the sub-agent renders a skeleton early, so partial + evidence survives its death), then **recover the device before reuse** (`adb kill-server; + adb start-server`, re-`acquire` the lease under a fresh owner). Re-dispatch a fresh sub-agent from the + last known state, or move on — never sit on an unbounded wait. + - **Result in → stop.** If the verdict/evidence is already available, stop waiting / terminate the + sub-agent and move to reporting — don't let a finished run linger. + - **Cap restarts** (e.g. 2). If still unresolved, escalate to the user with the full evidence trail. +4. **Only the parent reports to the user.** Collect the sub-agent's verdict + artifact links and present + the Phase 7 report yourself. + +If the harness can't spawn a sub-agent (or you're already the deepest agent), run the phases inline — but +keep the same discipline: **don't declare done until the verdict is in, and don't loop forever on a hang.** +Bound any long blocking device call so a wedge can't freeze you past your own cap — install with a timeout +(`appcontrol.ps1 install -TimeoutSec `; on timeout it kills the `adb install` and throws, so recover with +`adb kill-server; adb start-server` rather than rebooting the device mid-install). + +## Running multiple test cases (batch) — split across devices, run in parallel + +When asked to run **more than one** test case (a suite, a list of test points, several ids), treat it as a +**batch**, not a serial slog: + +1. **Enumerate the work and the devices.** Resolve each case's scenario (Phase 1 per case) and **expand each + case into its test points** (Phase 1 → "Test points and configurations") so you know how many points each + case has and which build each uses (`Local\` for a `LocalFlights` config, `ECS\` otherwise). The **unit of + work is the whole case** — all of a case's test points run together and land in **one consolidated report** + (a case is one lane; its points run back-to-back within it). List every free device in the pool + (`emulator.ps1 pool` / `devicelease.ps1 list`) and count how many you can drive in parallel (respect the + lease pool cap, `-MaxPoolSize`). +2. **Fan out one sub-agent per test case, capped at the device count.** Give each case its own sub-agent + (same model as the parent) with a **self-contained** prompt: that case's scenario + success criterion, + the app/module + which APKs to install, credential rules (type on device, never print), its run-folder + subpath, and "finish with a PASS/FAIL/BLOCKED verdict + evidence". The sub-agent **runs every test point + for its case** (back-to-back on its leased device) and writes a **single consolidated report** for the + case. **Each sub-agent leases its OWN device** with its own agent id as owner + (`devicelease.ps1 acquire -Owner `) so no two lanes touch the same device. If there are **more + cases than devices**, each device is a **lane** that runs its queue of cases one after another; the parent + keeps every lane busy until the queue drains. +3. **One shared batch folder, one folder per case (all its test points inside).** Use a single + `…\android-e2e-runs\-\`. Give **each case** one folder `tc\` holding a single + case-level `run.json` and a single `TestReport.html/.md`. When a case has more than one test point, record + the points as a **`testPoints[]` array** in that one `run.json` (each entry carries its own + `ado.testPointId` + `configuration` + `buildSource`, steps, evidence, verdict) and put each point's + screenshots in a point-scoped subfolder `tc\\iter\` so paths stay relative. `report.ps1 + render` then produces **one report per case** with a section per test point + a single shared "Proposed + test steps". The suite summary still scans this folder and expands each case's points into their own rows + in the **Config** column. +4. **The parent runs a polling watchdog across ALL lanes — on a wall clock.** Poll each live lane with a + **bounded** `read_agent` (e.g. `wait=true timeout=180`), rotating through the lanes; between polls tail + each lane's `tc\progress.log` and note its `tool_calls_completed`. Apply the single-run watchdog per + lane: **growth in the progress log *or* `tool_calls_completed` = alive; no growth in both across two + consecutive polls (~6–10 min of no motion) = STALLED → mark that case ABORTED from its partial report, + recover the device (`adb kill-server; adb start-server`, re-lease under a fresh owner), and re-dispatch + the next pending case onto the freed lane.** The **parent** — not just the in-agent cap — enforces the + wall-clock caps below, because a sub-agent frozen in a blocking call (e.g. a wedged `adb install`) + **cannot fire its own cap**. + + > **⏱️ Two caps, and the second one is the one that gets forgotten.** + > - **Per test point: 30 min** (unless told otherwise) — one attempt at one point. + > - **Per CASE: 45 min TOTAL across *all* iterations and retries** — this is a *separate*, *cumulative* + > budget. A re-dispatch after a partial result does **not** reset the clock. Without it a case silently + > costs 60–100+ min: in one measured batch, three cases ran a second iteration and landed at 58, 65 and + > 67 minutes of work each while every *individual* iteration was honestly under 30. **Track case-start + > wall time in the parent and stop the case when it hits the total**, recording whatever verdict the + > partial evidence supports (PARTIAL/ABORTED) rather than buying another 30 minutes. + + **Never block the turn on a single + open-ended wait:** a hung sub-agent emits **no completion notification**, so a notification-only loop + hangs forever if every live lane wedges at once (this stalled a real suite run for ~6 hours). Keep the + sub-agents' installs bounded (`appcontrol.ps1 install -TimeoutSec `) so a wedge surfaces as a fast + failure the agent can record rather than a freeze. + + > **🚫 Never queue a `write_agent` follow-up behind a busy lane.** A `write_agent` message is only + > delivered once the target agent's **entire turn** completes. Sending a follow-up to a lane that still + > has a queue of cases means it sits undelivered until that whole queue drains — measured cost in one + > batch: a follow-up queued at 12:50 was not picked up until 14:20, **90 minutes of dead air**, and the + > batch lost **159 minutes total** this way. If you need a follow-up run, **dispatch it as a fresh agent + > on a free lane**; only use `write_agent` on a lane that is idle or about to be. + + Keep every lane busy until the queue drains, and + **don't end the turn until every case has a verdict** (including ABORTED for lanes you had to kill, and + an explicit "not reached" for cases the batch never got to). +5. **Aggregate into an overall report.** After all cases finish, render the **suite summary** over the batch + folder (Phase 7): `./scripts/report.ps1 summary -In -Title ""` → `SUMMARY.html` + + `SUMMARY.md` (per-case verdict table linked to each report, plus overall counts). Present the overall + verdict and the per-case breakdown — not just the last case. + +**Only one device available?** Run the cases **serially** in a single lane (still clean-state per case, still +a per-case report each, still a final suite summary). **Independence is mandatory:** every case starts from a +clean state (Phase 3) and must not rely on another case's leftover accounts/registrations. + +## Workflow + +Execute in order. Announce a one-line status at each phase. + +### Phase 1 — Determine what to test + +Synthesize a concrete E2E scenario from the session context, in priority order: + +1. **User-provided test steps** in the session (explicit steps or a scenario the user described). +2. **An ADO Test Case work item** — the primary home for feature-specific steps. Fetch and parse it (see + "Sourcing test-case-specific steps" below); the `test-planner` skill can author/export these. +3. **A known test-steps file** the session points to (an exported plan, a markdown checklist, or a path + the user names). +4. **The design spec** (`design-docs/`) — its acceptance criteria and flows. +5. **The implementation diff** — `git -C diff` across changed repos to see what actually + changed and which app/module it affects. + +Produce: the **feature summary**, the **target app/module** (see +[references/app-and-module-map.md](references/app-and-module-map.md)), and a **step-by-step scenario** +with an explicit, observable **success criterion** (e.g. "AcquireTokenSilent returns a token; logs show +`executed successfully` with a correlation_id; no crash"). + +**Check whether the case is already automated.** For **Authenticator** cases especially, the team maintains a +compiled UIAutomator suite annotated with the same ADO case ids — see +[references/existing-ui-automation.md](references/existing-ui-automation.md). If the case is in that table, +say so up front and read the corresponding Kotlin test for its selectors and step ordering (it's the app +team's own ground truth for what the flow really needs) even if you still drive the case manually here. + +**Ask the user if** the scenario is ambiguous, multiple flows could be meant, or you cannot tell what +"working" looks like. Do not guess a scenario when the intent is unclear. + +**Keep specifics out of the skill.** This skill is generic. Anything specific to one feature — which app +configuration / client id to select, which lab account, which broker to pair, the exact tap sequence, the +expected success markers — belongs in the **test case**, not here. Read those specifics from the sources +above at run time; never hardcode a particular feature's steps into the skill or its scripts. + +#### Sourcing test-case-specific steps + +- **From ADO** (preferred for anything repeatable): a **Test Case** work item stores its steps in the + `Microsoft.VSTS.TCM.Steps` field (HTML/XML — each `` has an action and an expected result). Fetch it: + ```powershell + az boards work-item show --id --org ` + --fields "System.Title,Microsoft.VSTS.TCM.Steps" --output json + ``` + Parse the steps into an ordered action/expected list, drive them in Phase 4, and use each step's + **expected result** as its success criterion. A linked **Shared Steps** work item is fetched the same + way. (The `test-planner` skill authors and pushes these test cases, so the loop is: + test-planner → ADO → this skill.) +- **From a known file**: if the session provides a test-steps file (an exported plan, a markdown + checklist, or a path the user names), read the steps from there and treat them like ADO steps. +- If neither exists and the scenario isn't otherwise clear, **ask the user** rather than inventing + feature-specific steps. + +#### Test points and configurations (which build to run) + +A single ADO **Test Case** can appear in a suite as **one or two *test points*, each carrying a +*configuration*** (visible as the "Configuration" column on the test point). In the Android **Broker** suite +the two configurations are `RC MSAL - RC Broker` and `RC MSAL - RC Broker (LocalFlights)`, and the +configuration decides **which staged build folder you install from**: + +> ⚠️ **Counter-intuitive mapping — get this right:** +> - configuration name **contains `LocalFlights`** → install from the **`Local\`** folder +> - **any other** configuration (plain, no `LocalFlights`) → install from the **`ECS\`** folder +> +> `LocalFlights` does **not** mean the ECS folder. See +> [references/app-and-module-map.md → ECS vs Local builds](references/app-and-module-map.md#ecs-vs-local-builds-test-point-configuration). + +**Run every test point of the case, once each** — a 2-point case runs **twice** (once from `Local\` for the +LocalFlights point, once from `ECS\` for the plain point); a 1-point case runs once. The **only** exceptions +are when the **case body itself** says it is ECS-only or Local-only — then run just that one build. + +**Enumerate a case's test points + configurations from the Test Plan** (this is the source of truth — a case +in isolation doesn't tell you its points). Verified recipe: +```powershell +$org='https://dev.azure.com/identitydivision'; $project='Engineering' +$plan=; $suite= +$tok = az account get-access-token --resource '499b84ac-1321-427f-aa17-267ca6975798' --query accessToken -o tsv +$h = @{ Authorization = "Bearer $tok" } +$u = "$org/$project/_apis/testplan/Plans/$plan/Suites/$suite/TestPoint?api-version=7.1-preview.2" +$pts = (Invoke-RestMethod -Headers $h -Uri $u -Method GET).value +$pts | ForEach-Object { + [pscustomobject]@{ + testPointId = $_.id + caseId = $_.testCaseReference.id + config = $_.configuration.name + build = if ($_.configuration.name -match 'LocalFlights') { 'Local' } else { 'ECS' } + } +} | Where-Object caseId -eq | Format-Table +``` +`499b84ac-1321-427f-aa17-267ca6975798` is the Azure DevOps AAD resource GUID (`az` is already signed in for +identitydivision). Each returned row is **one run** for this case: feed its `build` into Phase 3 (which folder +to install from) and record `configuration` + `buildSource` in that run's `run.json` (Phase 7) so the report +and suite summary show a **Config** column. When you're driving a **batch**, each `(case, test point)` pair is +its own lane/run (see "Running multiple test cases"). + +### Phase 2 — Provision the device (emulator or real device) + +The **device pool** is every usable target: emulators **and** any real device connected over adb. +`emulator.ps1 pool` lists them. Derive requirements from the feature (min API, Google Play services for +broker/push flows) — see "Emulator / real-device requirements per feature" in the app-and-module map — +then one command finds-or-creates, reuses-or-starts, and waits for boot: + +```powershell +./scripts/emulator.ps1 ensure -RequireGoogleApis -ApiLevel 30 -Wait +# prefer a connected real device if one meets the requirements: +./scripts/emulator.ps1 ensure -RequireGoogleApis -PreferPhysical -Wait +# pin an exact device (emulator serial or real-device serial): +./scripts/emulator.ps1 ensure -Serial 39121FDJH003ZK +# or target a specific AVD: ./scripts/emulator.ps1 ensure -Avd Pixel_7 -Wait +# stick to emulators only: add -NoPhysical +``` +A connected, booted real device that satisfies the requirements is a valid target (real devices are +never created/booted — only used if already connected). By default `ensure` reuses a running emulator +first, then a matching real device, then boots/creates an AVD; `-PreferPhysical` flips it to try real +devices first. Capture the printed `SERIAL=` and use it as `-Serial` everywhere. **When multiple tests +may run concurrently, lease the device first** (see Phase 2a) so two runs don't collide. If provisioning +fails, see [references/troubleshooting.md](references/troubleshooting.md) (missing images, hypervisor, +boot hang). + +> **Speed check (do this if the run feels slow).** Run `./scripts/emulator.ps1 resolve-sdk` — it prints the +> host GPU/perf profile. On a **Cloud PC / VM / RDP host there is no GPU**, so the emulator uses slow +> **software rendering** (SwiftShader) and its NAT makes Play Store downloads crawl — nothing makes an +> emulator fast there. **Prefer a connected physical device** (the skill auto-prefers one on a GPU-less +> host; force with `-PreferPhysical`). AVDs the skill creates already appear in **Android Studio's Device +> Manager** (shared `~/.android/avd`); to see one in Studio's **Running Devices**, start it from Studio and +> let the skill reuse it. Details: [references/emulator-performance.md](references/emulator-performance.md). + +### Phase 2a — Lease the device (avoid collisions with other tests) + +Multiple E2E tests can run at once (different agents/sessions). To stop two runs from driving the same +device, **lease it** before use and **release it** when done: + +```powershell +# Acquire (reaps abandoned leases, picks a free device, or boots one if the pool cap allows). +# Use YOUR agent/session id as the owner so a dead agent's lease can be reclaimed. +$serial = (./scripts/devicelease.ps1 acquire -Owner $AgentId -Feature ` + -RequireGoogleApis -ApiLevel 30 -Wait | + Select-String '^SERIAL=(.+)$').Matches[0].Groups[1].Value +``` +- **Owner = your agent/session id** (or set `$env:E2E_AGENT_ID`). Liveness is heartbeat-based: refresh + during the run so a long test isn't reclaimed out from under you: + `./scripts/devicelease.ps1 heartbeat -Owner $AgentId -Serial $serial` (call it at each phase). +- **Pool cap:** `-MaxPoolSize N` (default 4) limits concurrent devices. If the pool is full and every + device is held by a **live** owner, acquire fails with guidance (wait, raise the cap, or connect a + device). If a holder's agent/session is **dead** (heartbeat older than `-StaleMinutes`, default 30), + its device is reclaimed automatically on the next `acquire`/`reap`. +- `acquire` will **boot/create an emulator** to add a device when the pool has no free one and the cap + allows — so it subsumes Phase 2's `emulator.ps1 ensure` (pass the same requirement flags). Use + `emulator.ps1 ensure` directly only when you deliberately don't want leasing. +- Do the whole run against the leased `$serial` (pass it as `-Serial` everywhere). + +### Phase 3 — Deploy the app-under-test + +0. **Prep the device once (two settings, big payoff).** Before installing anything, disable the autofill + overlay and animations on the leased device — the first removes the #1 cause of dropped/misplaced + credential input (and lets you type in fast bulk mode instead of char-by-char), the second removes + transition latency and mid-flight taps: + ```powershell + adb -s shell settings put secure autofill_service null + adb -s shell settings put global window_animation_scale 0 + adb -s shell settings put global transition_animation_scale 0 + adb -s shell settings put global animator_duration_scale 0 + ``` + Both persist on the device. See + [references/run-speed.md](references/run-speed.md#device-prep-turn-off-animations-and-autofill). +1. Decide the test surface and whether a **broker** must also be installed (brokered flows need a + calling app + a broker) — see the app-and-module map's pairing rules. **If Authenticator is the app under + test, uninstall Teams and Company Portal first** so broker election is deterministic (unless the case + deliberately tests one of them as the broker). +2. Prefer an APK Android Studio already built (`appcontrol.ps1 list-apks`); otherwise build + (`appcontrol.ps1 build -Module <:module>`). Gradle builds need the repo's Maven creds — if they fail + with 401, that's an environment blocker for the user. **When the case is staged with both an `ECS\` and a + `Local\` folder, the folder to install from is dictated by the test point's configuration** — `Local\` + for a `LocalFlights` config, `ECS\` otherwise (Phase 1 → "Test points and configurations"; + [map](references/app-and-module-map.md#ecs-vs-local-builds-test-point-configuration)). Run the case once + per test point, each from its own folder. +3. **Start every test case from a clean state (unless told otherwise): uninstall, then freshly install all + test apps.** A `pm clear` alone is **not** a clean slate — it leaves AccountManager work-account entries + and broker/WPJ registration behind; only **uninstalling** the app removes those. So for each app the case + needs (app-under-test + any broker/companion), uninstall any existing copy first, then install the + provided/built APK: + ```powershell + ./scripts/appcontrol.ps1 uninstall -Package -Serial # ok if "not installed" + ./scripts/appcontrol.ps1 install -Module :msalTestApp -Serial # or -Apk for a provided APK + ./scripts/appcontrol.ps1 launch -Package -Serial + ``` + Verify the exact package/activity at runtime (the map shows how) — don't trust hardcoded IDs. Known + provided-APK filenames (e.g. `app-production-universal-release-signed` = the **Authenticator** under test; + `com.microsoft.windowsintune.companyportal-signed` = **Company Portal**) are in + [references/app-and-module-map.md](references/app-and-module-map.md#known-provided-apks). **Do not tear + down accounts/registrations at the *end* of a case** — leaving them is intended (the *next* run's + uninstall clears them, or a tester handles them); see Phase 7. + +### Phase 4 — Execute the scenario (auto-handle inputs) + +Clear the log buffer first, then drive the flow: + +```powershell +./scripts/authlogs.ps1 clear -Serial +``` + +Loop per screen using [references/ui-interaction.md](references/ui-interaction.md), following the +**fast path** (see [references/run-speed.md](references/run-speed.md)): `dump` **once** per screen and +compute every target from that one XML → act → verify by the **next** screen's anchor with +`tap-text -Then ""` (tap + wait in one call) or `wait-text`, **never a fixed `Start-Sleep`**. +Re-`dump` only when you must read genuinely new state; verify a navigation by its +anchor, not a reflexive re-dump after every tap. Save a `screenshot` only at **milestones** and only on +screens that actually render (skip FLAG_SECURE screens — they come back black; capture a `uiautomator dump` +as evidence instead). + +> **⚡ Drive a KNOWN sequence with `deviceui.ps1 flow`, not one call per screen.** Measured: a screen driven +> as its own tool call costs a **~25 s median** agent round-trip (p90 60 s) for **~2.5 s** of device work — so a +> deterministic ~12-screen preamble (app first-run gates, an add-account wizard, a settings path) burns ~5+ +> minutes to do ~30 seconds of work, and in one measured batch that same preamble was paid **9 times**. The +> overhead is **fixed per call, not proportional to how hard the decision is** — measured, easy deterministic +> steps cost 25.7 s and hard auth/error steps cost 27.5 s, i.e. the same. So it can't be fixed by "thinking +> less"; it's fixed by **taking fewer turns**, which pays off best precisely where the steps are most obvious: +> ```powershell +> ./scripts/deviceui.ps1 flow -Serial -Spec +> ./scripts/deviceui.ps1 flow -Serial -Text '[{"tap":"Allow","optional":true},{"tapRes":"...:id/accept"}]' +> ``` +> Steps support `wait`/`waitRes`/`tap`/`tapDesc`/`tapRes`/`input`/`secretRef`/`key`/`then`/`thenRes`/ +> `screenshot`, plus **`"optional": true`** for screens that only *sometimes* appear (a permission dialog +> already granted, a "Save password?" infobar) — those SKIP instead of failing. It stops at the first required +> failure (exit 5) and prints a per-step trace, so a failure is still precisely located. Reserve one-call-per- +> screen for the genuinely *decision-driven* parts of the test — which is the part the case is actually about. +> Ready-made Authenticator specs: +> [authenticator-app.md](references/authenticator-app.md#ready-made-flow-specs-copy-paste). + +> **Do only what the case asks.** Setup that the case doesn't need is pure cost — one measured run completed a +> full passwordless/device-registration setup and then logged *"backing out … (out of scope)"*, spending +> minutes on something it then discarded. Before any multi-step setup, check it's required by a step you must +> execute; if not, skip it. + +> **Trigger a time-limited challenge only when you're ready to consume it.** A number-match/push approval has +> a short TTL (~1–2 min). Don't fire the sign-in that raises it and *then* start figuring out how to read the +> notification — have the approval path resolved first, then trigger, then approve in one continuous +> sequence (ideally one `flow` call). Otherwise the request expires and the whole segment is repeated. + +**Auto-handle** everything the AI reasonably can: type lab test credentials, tap Next/Sign in/Accept/ +Allow/Yes, pick an account, grant permissions, simulate a fingerprint (`finger`), enter a TOTP if the +seed is known, set/enter a device PIN. **Do not** print or commit credentials — when the user gives you +a password or device PIN, take it via the encrypted store and reference it by name (see +[references/secrets-and-files.md](references/secrets-and-files.md)), never inline. + +**Typing into eSTS/WebView credential fields (hard-won).** The email/password pages are a WebView and +Chrome's autofill/passkey overlay can silently swallow a bulk `input text` (the value lands in the wrong +field or is dropped, producing "Enter a valid email"). **The durable fix is to disable the autofill service +in Phase 3 device prep** (`settings put secure autofill_service null`) — with the overlay gone, fast bulk +typing just works and you avoid the per-character cost and its dropped-character failure mode. Only fall +back to `-CharByChar` if verification shows a value still didn't land: +```powershell +./scripts/deviceui.ps1 input-text -Text $upn -Clear -Serial # bulk (fast; correct once autofill is off) +./scripts/deviceui.ps1 input-text -SecretRef labpw -Clear -Serial # password from the store (never echoed) +./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial # fallback if it didn't land +``` +`-Clear` empties the field first, `-CharByChar` defeats the overlay. For the **password**, prefer +`-SecretRef ` (resolves from the encrypted store and implies `-Secret`) so the value never lands +in a tool call or the transcript — see [references/secrets-and-files.md](references/secrets-and-files.md); +`-Text $pw -Secret` still works if you already hold the value out-of-band. For **shared lab-tenant +passwords**, don't ask the user to paste — pull it from Key Vault into the store first with +`./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw` (needs `az login` + +`TM-MSIDLABS-DevKV`), then reference `-SecretRef labpw`. Dismiss a passkey/"Save +password" bottom sheet with `key ESCAPE` before typing if one appears. +See [references/common-blockers.md](references/common-blockers.md). + +**Unlocking the lock screen on a physical device.** If the device sleeps/relocks mid-run and a step needs a +PIN, seed it once with `secrets.ps1 set-device-pin` (done by the human — it picks the device, or shows a +numbered menu when several are attached, and stores the PIN as `devicepin_`), then let the tool enter ++ **verify** it: `deviceui.ps1 unlock -Serial ` **auto-resolves** that per-device PIN (no `-SecretRef` +needed). It confirms the keyguard actually cleared and +**stops after 3 attempts** (`-MaxAttempts`, default 3, exits 3 if it gives up) so a wrong PIN never trips +Android's Gatekeeper lockout — do **not** loop it yourself. With several devices attached (even the same +model) each has a unique adb serial; because the PIN is stored per serial, `unlock -Serial ` always +uses the right device's PIN. A PIN only substitutes for biometric where a "Use PIN" path is offered; a +fingerprint-only step needs an emulator (`finger`) or a human. See +[references/secrets-and-files.md](references/secrets-and-files.md) and +[references/common-blockers.md](references/common-blockers.md). + +**Provision / repair the lab account with the LAB API** when the scenario needs a fresh account or the +account state is stuck (e.g. MFA already registered from a previous run, a CA policy blocking the step): +```powershell +./scripts/labapi.ps1 create-user -UserType GlobalMFA # temp user, auto-deletes in 60 min +./scripts/labapi.ps1 reset -Upn $upn -Operation mfa # clear stale MFA registration (temp users only) +./scripts/labapi.ps1 disable-policy -Upn $upn -Policy GlobalMFA # unblock a CA-gated segment +./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw # pull tenant pw from Key Vault (no paste) +``` +**Don't reset a password just because sign-in rejected it.** If the device shows *"Your account or password is +incorrect"* / *"password has expired"* / *"that Microsoft account doesn't exist"*, do **not** run +`reset -Operation password` (or any password change) unless the **test case explicitly tells you to**. A wrong +password almost always means you have the wrong *value* or the wrong *account*, not that the account needs +changing — so instead: re-pull the shared value with `fetch-password` (it may have rotated in Key Vault), confirm +the UPN/tenant matches what the case named, and check the account isn't `Locked_…` from a prior lockout. Resetting +is especially dangerous for **shared durable accounts** (durable, pre-created accounts, not temp `Locked_…` users) that other tests reuse — +changing their password breaks every other case. If the value is genuinely right and it still fails, mark the run +**BLOCKED** with the exact on-screen error rather than mutating the account. + +**Account policy — prefer fresh ID4SLAB2 temp users, one per case.** Provision a **new temp user for each +test case** (`create-user` makes an ID4SLab2 user that auto-deletes in ~60 min) instead of reusing one across +cases — **even when a test case names a fixed MSIDLAB4 account** (that lab is being deprecated): create the +matching `-UserType` instead (e.g. `MAMCA`, `MDMCA`, `Basic`, `GlobalMFA`). Reuse a specific named/durable +account only when the case genuinely requires that exact identity, or the user tells you to. + +**Freshness gate — poll ≤ 3 min, then recreate or reuse a < 30-min-old user.** After `create-user`, a brand-new +temp user can lag ESTS replication and show *"This username may be incorrect"* at sign-in. Don't keep waiting: +if the new user isn't **consistently** sign-in-able within **3 minutes** of polite polling, **create another** +temp user, **or reuse a previously created temp user still under ~30 minutes old** (inside its 60-min TTL and +already propagated — one that already signed in once is safest). Note the swap in the report; never "fix" the lag +with a password reset. See +[common-blockers.md → Fresh temp user not sign-in-able yet](references/common-blockers.md#fresh-temp-user-not-sign-in-able-yet-ests-propagation-lag). + +See [references/lab-api.md](references/lab-api.md) for all endpoints, usertypes, the account policy, and the auth workaround. + +**Set flags & mock what's missing (don't fake a pass).** If the scenario needs a feature flag on, or a +step depends on data/a dependency you can't produce naturally (a server API not deployed yet, a +collaborator app you can't drive), set the flag and mock the missing piece — including **temporary code +changes** that you revert afterward. If a middle piece genuinely can't be mocked, test the flow in +**segments**. See [references/mocking-flights-and-segments.md](references/mocking-flights-and-segments.md). + +**Do system-settings / on-device actions yourself before calling something blocked.** Some steps have no +adb command (advance the device clock, delete a user certificate, toggle a system setting, change the +language). Don't jump straight to BLOCKED — **open the relevant Settings screen and drive it with +`deviceui.ps1` (`dump` → `tap-text`/`tap-desc` → `input-text`) exactly like a human tester would.** E.g. to +advance the clock: Settings → *Date & time* (Samsung: *General management → Date and time*) → turn **off** +automatic date/time → set it manually. Only fall back to a blocker when the surface itself is genuinely +undrivable — a **secure/native system dialog** (Knox cert install, keyguard credential prompt, biometric +sensor) uiautomator can't read/act on, or an action that truly needs **root** (e.g. expiring an app's +*internal* monotonic cached-token timer on a non-rooted retail device — moving the wall clock won't affect +it). See [references/common-blockers.md](references/common-blockers.md#doing-it-yourself-in-system-settings). + +**Stop and ask the user** only for genuine blockers (see the consolidated list below). + +### Phase 5 — Verify from logs + +```powershell +./scripts/authlogs.ps1 scan -Package -Serial -Out \iter1\logcat.txt +``` +`scan` prints a heuristic verdict plus evidence (crash stacks, failure lines, success lines, AADSTS +codes, correlation IDs). **Reason over the evidence** — don't blindly trust PASS/FAIL. Confirm the +scenario's specific success criterion is met (a positive success signal, not just absence of errors), +per [references/log-signals.md](references/log-signals.md). For automation-test mode, also read the +`build/reports/androidTests/` results. + +Classify the outcome: +- **PASS** — success criterion met, no crash → go to Phase 7. +- **FAIL (real defect)** — crash in changed code, `AADSTS50011/700016`, regressed silent auth, broken + IPC → go to Phase 6. +- **Environment/harness problem** (missing image/creds, signature mismatch, stale snapshot, offline + adb) → fix the setup per troubleshooting, then re-run from Phase 2/3. Do **not** send these to the fix loop. +- **INCONCLUSIVE** — wrong package filter, action never executed, logging off → re-run the scenario + cleanly before deciding. + +### Phase 6 — Iterate: fix and retest + +For a real defect: + +1. **Root-cause** from the evidence: the failing signal, correlation_id, stack, and the code in the + diff it points to. State the hypothesis with the log line that supports it. +2. **Get it fixed:** + - If the code is in a checked-out sub-repo the agent can edit, **fix it directly** (Kotlin-first per + repo conventions), then rebuild. + - If the work is managed via a coding-agent PR, hand the diagnosis to that agent (e.g. `@copilot` + on the PR, or the `pbi-dispatcher` skill) with the exact evidence and root cause. +3. **Reinstall → clear → re-run the scenario → re-scan** (Phases 3–5). +4. Repeat, **max 3–5 iterations**. If still failing, **stop and escalate** to the user with the full + evidence trail (what was tried, current hypothesis, why it's stuck). Do not loop forever. + +Track iterations in the run folder (one subfolder per attempt) so regressions/progress are diffable. + +### Phase 7 — Report + +Summarize: scenario tested, target app/emulator, final verdict, iterations taken, key evidence +(success signal + correlation_id, or the blocker), and links to the run-folder artifacts (logs, +screenshots). If blocked, state exactly what you need from the user to proceed. + +**For an ADO Test Case, a written test report is MANDATORY — always generate it, on every outcome +(PASS / FAIL / BLOCKED / PARTIAL), not only on success.** Render both an HTML and a Markdown report into +the run folder with `report.ps1`: +```powershell +# Build a run JSON (verdict, device/app/account metadata, per-step action/expected/result, evidence, +# blockers, artifact paths — UPN only, never a password), then render: +./scripts/report.ps1 render -In \run.json # writes TestReport.html + TestReport.md next to it +``` +The step list should mirror the ADO test case's steps, and each step's **result** should be judged +against that step's **expected result**. Include the ADO ids (`testCaseId`/`planId`/`suiteId`) and a link +so the report ties back to the test plan. **When the run came from a specific test point, also record +`ado.testPointId`, `ado.configuration` (the config name), and `ado.buildSource` (`ECS` or `Local`)** so the +report header and the suite summary's **Config** column show which build this run exercised. + +**If a case has more than one test point, put them all in ONE report** — don't write a separate report per +point. Give the case one `run.json` with a **`testPoints[]` array**; each entry carries its own point-level +fields (`ado.testPointId`/`configuration`/`buildSource`, `device`, `app`, `account`, `steps`, `evidence`, +`blockers`, `verdict`) and references its screenshots by a point-scoped relative path (e.g. +`ecs/iter1/07_token.png`, `local/iter1/07_token.png`). `render` emits one section per test point and derives +the overall verdict from the points. + +**Every ADO case report MUST carry a "Proposed test steps" block — this is not optional.** You have just +walked the case step-by-step, so you are the best-placed reviewer it will get: record how the steps should be +reworded to be correct and unambiguous. It lives at **case level** and must be **generic across every test +point** (don't mention ECS/Local or a specific point); give each proposed step an optional `attachment` (a +screenshot path/URL) to fill the report's **Attachments** column. If the case truly reads fine, still say so +explicitly — `proposedScope: "No change needed"` with `proposedSteps: []`. **An empty block is a conclusion; +an absent block is an omission**, and `report.ps1 render` flags it as **⚠ MISSING**. (This applies to *every* +app — it was historically done for Broker cases but silently skipped on all 26 Authenticator ones.) See +[references/test-reporting.md](references/test-reporting.md) +for the JSON schema and a worked example. (For an automation-test run, also attach the +`build/reports/androidTests/` output.) Present the verdict and the report paths to the user; optionally +publish the outcome back to the ADO Test Run. + +**When you ran a batch (multiple ADO cases), also generate the overall run summary** — in addition to each +per-case report — so the user gets one roll-up verdict: +```powershell +./scripts/report.ps1 summary -In -Title "" # SUMMARY.html + SUMMARY.md: per-case table + counts +``` + +**Re-running a case? Update its existing report — and keep the old verdict as history.** When a case already +has a `tc\run.json` from an earlier run, write the new outcome into **that same file** rather than starting +a fresh batch folder, and append the previous outcome to a **`runHistory[]`** array (oldest first, last entry's +verdict = the current `verdict`). Put the re-run's screenshots in their own subfolder +(`tc\rerun-\iter\`) so the earlier evidence survives. The report then renders a *Run history* +table and the suite summary gains a **Was** column (`BLOCKED → PASS`), which is the only way a reader can tell +which re-run actually rescued which case. Omit `runHistory` for a case that has only ever run once. See +[references/test-reporting.md → Re-running a case](references/test-reporting.md#re-running-a-case--record-the-trajectory-dont-overwrite-it). + +**Do not tear down accounts or registrations after a case (unless told otherwise).** Leave temp users, +device/WPJ registrations, and installed apps as they are — the *next* run's clean-state step (Phase 3 +uninstall+reinstall) removes them, or a tester handles them manually. This keeps a failed/blocked case's +state available for inspection and avoids racing another lane (temp lab users self-destruct in ~60 min +anyway). The only required end-of-case teardown is **releasing the device lease** (below) and **reverting any +temporary code/flag/mocks** (Guardrails). + +**Release the device lease** so the next test can use it (do this even on failure/blocked): +```powershell +./scripts/devicelease.ps1 release -Owner $AgentId -Serial $serial +``` + +## When to ask the user + +Ask (don't guess) when: +- The **scenario/intent is unclear**, or several flows could be meant, or success can't be defined. +- A **blocker** needs a human — but only *after* you've tried the on-device/Settings path yourself (Phase 4): + real push-notification MFA on another device, SMS/phone OTP, a hardware key/NFC/QR/camera, CAPTCHA, a + **secure native system dialog** (Knox cert install, keyguard/biometric prompt), a credential the AI doesn't + have, or a tenant/CA policy it can't provision. +- The **implementation is incomplete** for the path under test (stubs, `TODO`, missing wiring, feature + flag off with no way to enable) — **first try** to set the flag (temp code change) and mock/segment + around a missing middle piece (see + [mocking-flights-and-segments.md](references/mocking-flights-and-segments.md)); report the gap and ask + only if the feature genuinely can't be exercised even in segments. +- **Environment setup** is missing and only the user can fix it (Maven creds/PAT, no sub-repo checkout, + no system image, no lab account). + +When you can complete a manual step *with* the user live (e.g. they approve a push), pause, let them do +it, then resume the automated flow. + +## References + +Load these as needed (don't preload all): + +| File | Read it when | +|---|---| +| [references/app-and-module-map.md](references/app-and-module-map.md) | Choosing/deploying the test app, package discovery, broker pairing, credentials, emulator requirements | +| [references/existing-ui-automation.md](references/existing-ui-automation.md) | **Check first for an Authenticator case** — the team's compiled UIAutomator suite already automates ~24 ADO cases (id→test table); where it lives, how it differs, which to use, and the selectors/settings we borrowed from it | +| [references/authenticator-app.md](references/authenticator-app.md) | Driving **Microsoft Authenticator** — the 4-screen first-run flow, adding an account from home, and the **same-device** AAD Work/School account-add + first-time MFA proof-up (number-match) that is often wrongly marked BLOCKED | +| [references/lab-api.md](references/lab-api.md) | Provisioning/resetting a lab test account, the account policy (prefer fresh ID4SLAB2 temp users over MSIDLAB4), LAB API endpoints/usertypes/policies, the EasyAuth auth workaround, fetching tenant passwords from Key Vault, `labapi.ps1` | +| [references/common-blockers.md](references/common-blockers.md) | Recurring hiccups & when to switch to an emulator (fingerprint/App-Lock, number-match MFA, session timeouts, FLAG_SECURE, autofill/passkey overlay, screenshot corruption); driving System Settings yourself before declaring a blocker; clean-state between runs | +| [references/test-reporting.md](references/test-reporting.md) | Writing the mandatory ADO test report — run-JSON schema, `report.ps1`, worked example | +| [references/run-speed.md](references/run-speed.md) | The run feels slow; understanding per-step latency sources and how to shorten them | +| [references/emulator-performance.md](references/emulator-performance.md) | The run is slow, you're on a Cloud PC/VM/RDP (software rendering), or you want the emulator to show in Android Studio's Device Manager / Running Devices | +| [references/log-signals.md](references/log-signals.md) | Interpreting logcat, success/failure patterns, AADSTS codes, per-flow pass criteria, eSTS correlation | +| [references/ui-interaction.md](references/ui-interaction.md) | Driving auth screens, AI-vs-human inputs, FLAG_SECURE gotcha, selector strategy | +| [references/mocking-flights-and-segments.md](references/mocking-flights-and-segments.md) | A flag must be set, a dependency/server data is unavailable, or the flow can't run fully E2E (mock it or test in segments) | +| [references/secrets-and-files.md](references/secrets-and-files.md) | The user must hand you a password / device PIN / keystore password, or an APK / test file — keep secrets out of the chat (`secrets.ps1`, `labapi.ps1 fetch-password`, `-SecretRef`, `unlock`) and use a file drop-folder | +| [references/troubleshooting.md](references/troubleshooting.md) | Emulator/build/install/uiautomator/broker failures; env-vs-defect triage | + +## Guardrails + +- **Never** print, log, or commit real or lab credentials, tokens, or secrets. Type them onto the device + only — use `input-text -SecretRef ` (or `-Text ... -Secret`) so the value never hits the + transcript. When the user needs to hand you a password or device PIN, take it via the encrypted store + (`secrets.ps1 set`) and reference it by name; for **shared lab-tenant passwords** pull them straight + from Key Vault with `labapi.ps1 fetch-password` (no paste needed); for APKs/files use a drop-folder path. + See [references/secrets-and-files.md](references/secrets-and-files.md). +- **Artifacts stay outside the repo** (the run folder). Never commit logs/screenshots/reports. +- **For an ADO test case, always generate the HTML + Markdown report** (`report.ps1`) on every outcome — + PASS, FAIL, BLOCKED, or PARTIAL. A run driven from a test case is not "done" until the report exists. +- **Every ADO case report must include "Proposed test steps"** (`proposedScope` + `proposedSteps`) — for + **every app**, not just Broker. If the case needs no change, state that explicitly + (`proposedScope: "No change needed"`, `proposedSteps: []`); never just leave it out. +- **Run every test point of a case, and match the build to its configuration.** A Broker-suite case may have + two test points; run each once. Install from **`Local\`** when the point's configuration name contains + `LocalFlights`, from **`ECS\`** otherwise (⚠️ not the other way round). Skip a point only if the case body + says it's ECS-only or Local-only. Record `configuration`/`buildSource` in each run's report. +- **A batch of ADO cases also needs the overall `report.ps1 summary`** over the batch folder, not just the + per-case reports. +- **Split a multi-case batch across available devices and run in parallel** (one sub-agent per case, each + leasing its own device); fall back to serial only when a single device is free. See "Running multiple test + cases". +- **Start each case clean; don't clean up after.** Unless told otherwise, uninstall+reinstall the test apps + at the **start** of every case (a `pm clear` doesn't remove work accounts/registrations), and **do not** + tear down accounts/registrations at the **end** — leave them for the next run or a tester. +- **Drive System Settings / on-device UI yourself before declaring a blocker** (advance the clock, delete a + cert, toggle a setting) — only block on a genuinely secure/native dialog or a true root requirement. +- **Prefer fresh ID4SLAB2 temp accounts, one per case** — use `create-user -UserType ` even when a + case names a deprecated MSIDLAB4 account, unless a specific durable account is required. +- **Don't reset/change a password on an auth failure unless the test case says so.** "Incorrect password" or + "password expired" on the sign-in screen means re-fetch the value (`fetch-password` — it may have rotated) and + re-check the UPN/tenant, **not** `reset -Operation password`. Never change the password of a shared durable + account (a durable, pre-created account, not a temp `Locked_…` user) — it breaks other tests; if the value is right and it still fails, + mark the run **BLOCKED** with the exact error. +- **Prefer an emulator when a step needs an injectable fingerprint/biometric** (App Lock, biometric-gated + number-match). `adb emu finger touch` works only on emulators; a physical device needs a human at the + sensor. See [references/common-blockers.md](references/common-blockers.md). +- **When delegating to a sub-agent, wait for its verdict** before reporting or ending the turn; restart it + if it hangs and the test isn't done, terminate it once the result is in. Never surface "done" without the + sub-agent's PASS/FAIL/BLOCKED evidence. +- **Lease the device before use and release it after** (even on failure) so concurrent runs don't collide. +- **Temp changes for mocks/flags stay uncommitted and get reverted** — never commit/push a flag flip or a + mock; leave a `TODO: REVERT` marker and confirm the tree is clean before finishing. +- **Environment problems are not code defects** — fix setup and re-run; only hand real defects to the fix loop. +- **Cap the loop** (3–5 iterations) and escalate with evidence rather than looping indefinitely. +- **Honour BOTH wall-clock caps: 30 min per test point AND 45 min per case across *all* iterations.** A + re-dispatch does not reset the case budget; stop and record a PARTIAL/ABORTED verdict instead of buying + another 30 minutes. +- **Batch deterministic screen sequences into one `deviceui.ps1 flow` call.** One tool call per screen costs a + ~25 s median round-trip for ~2.5 s of device work, and that overhead is **fixed** — measured, an obvious step + costs the same as a hard one (25.7 s vs 27.5 s). So a preamble with no decisions in it must not be driven + screen-by-screen. Keep per-screen driving for the decision-heavy part the case is actually testing. +- **Never queue a `write_agent` follow-up behind a lane that still has work** — it isn't delivered until that + lane's whole turn ends (measured: 90 min of dead air). Dispatch a fresh agent on a free lane instead. +- **Don't do setup the case doesn't require**, and **don't trigger a time-limited challenge (number-match / + push approval, ~1–2 min TTL) until the approval path is ready** — otherwise it expires and you repeat the + segment. +- **Require a positive success signal** matching the scenario before declaring PASS. +- Follow repo conventions when fixing code (Kotlin for new code, the `Logger` class, minimal changes). diff --git a/.github/skills/android-e2e-tester/references/app-and-module-map.md b/.github/skills/android-e2e-tester/references/app-and-module-map.md new file mode 100644 index 00000000..249701db --- /dev/null +++ b/.github/skills/android-e2e-tester/references/app-and-module-map.md @@ -0,0 +1,185 @@ +# App & Module Map — What to Build, Install, and Test + +Table of contents: +- [Pick the test surface from the feature](#pick-the-test-surface-from-the-feature) +- [Module → path → typical package](#module--path--typical-package) +- [Known provided APKs (staged test builds)](#known-provided-apks) +- [Discover the exact package & launch activity at runtime](#discover-the-exact-package--launch-activity-at-runtime) +- [Two E2E execution modes](#two-e2e-execution-modes) +- [Broker pairing rules](#broker-pairing-rules) +- [Test credentials (Lab API) and when they block](#test-credentials-lab-api-and-when-they-block) +- [Emulator / real-device requirements per feature](#emulator--real-device-requirements-per-feature) + +The repo root (`android-complete`) aggregates sub-repos cloned via `git droidSetup`: +`msal/`, `common/`, `broker/`, `adal/`, and optionally `authenticator/`. Gradle modules are +declared in `settings.gradle`. If a sub-repo folder is missing, the checkout hasn't been set up — +that is a blocker to raise with the user. + +## Pick the test surface from the feature + +Map the changed repo(s) (from `git diff` / the design) to the app you drive: + +| Feature lives in | App to install & drive | Also install (dependency) | +|---|---|---| +| **msal** (SDK) | `:msalTestApp` (interactive) or `:msalautomationapp` (automated) | A broker for brokered flows (see pairing) | +| **broker / broker4j** | `:brokerHost` test host + a broker (`:AADAuthenticator`) | `:msalTestApp` as the calling client | +| **common / common4j** (IPC, cache, telemetry) | Both sides: `:msalTestApp` **and** a broker | — | +| **adal** (legacy) | `:adalTestApp` | A broker for brokered ADAL flows | +| **authenticator** (opt-in) | `:MSAuthenticator` | — | + +Isolated broker testing without real Authenticator/Company Portal can use the **mock brokers**: +`:mockauthapp` (mock Authenticator), `:mockcp` (mock Company Portal), `:mockltw` (mock Link-to-Windows). + +## Module → path → typical package + +Package IDs below are **typical hints** — always verify at runtime (next section), because they +vary by build variant (e.g. `localDebug` often adds a `.local` suffix). + +| Gradle module | Project path | Typical applicationId | +|---|---|---| +| `:msalTestApp` | `msal/testapps/testapp` | `com.msft.identity.client.sample.local` | +| `:msalautomationapp` | `msal/msalautomationapp` | `com.microsoft.identity.client.msalautomationapp` | +| `:brokerHost` | `broker/userapp` | `com.microsoft.identity.testuserapp` | +| `:brokerautomationapp` | `broker/brokerautomationapp` | `com.microsoft.identity.client.brokerautomationapp` | +| `:AADAuthenticator` | `broker/AADAuthenticator` | `com.azure.authenticator` | +| `:adalTestApp` | `adal/userappwithbroker` | `com.microsoft.aad.adal.testapp` | +| `:mockcp` | `broker/mockbrokers/mockcp` | mock Company Portal | +| `:mockauthapp` | `broker/mockbrokers/mockauthapp` | mock Authenticator | +| `:MSAuthenticator` | `authenticator/PhoneFactor/app` | `com.azure.authenticator` | + +Real production brokers used in some E2E: Microsoft Authenticator `com.azure.authenticator`, +Intune Company Portal `com.microsoft.windowsintune.companyportal`, Link to Windows `com.microsoft.appmanager`. + +## Known provided APKs + + +When the user hands you a folder of APKs (commonly `…\Downloads\ECS` and `…\Downloads\Local`, or +`…\Downloads\APKs\…`), map them by **filename** so you install the right thing without re-deriving it each +run. `-signed` = a production-signed release build; `-dist-debug` / `-local-debug` = the test/automation +harness builds. Always confirm the real `applicationId` after install (next section) — filenames are a +convenience, not a contract. + +| Provided APK filename (with/without `.apk`) | App | Package (verify at runtime) | Role in a test | +|---|---|---|---| +| `app-production-universal-release-signed` | **Microsoft Authenticator (app under test)** | `com.azure.authenticator` | The build being tested; **universal** ABI — safe on emulators too. | +| `app-production-arm64-v8a-release-signed` | Microsoft Authenticator (arm64 only) | `com.azure.authenticator` | Same app, arm64-only — use on a **physical** arm64 device; can fail dexopt on an x86_64 emulator (prefer the universal one there). | +| `com.microsoft.windowsintune.companyportal-signed` | **Intune Company Portal** | `com.microsoft.windowsintune.companyportal` | Broker / MDM companion for WPJ / device-registration / MAM-CA flows. | +| `company-portal` | Intune Company Portal | `com.microsoft.windowsintune.companyportal` | Same app, alternate filename. | +| `teams` | Microsoft Teams | `com.microsoft.teams` | 1P OneAuth client (some brokered/SSO scenarios). | +| `msalTestApp-dist-debug` | MSAL test app | `com.msft.identity.client.sample(.local)` | Calling client for MSAL / brokered flows (Mode B). | +| `adalTestApp-dist-debug` | ADAL test app | `com.microsoft.aad.adal.testapp` | Calling client for legacy ADAL flows. | +| `brokerHost-local-debug` | Broker Host test app | `com.microsoft.identity.testuserapp` | Drives broker APIs directly (Get Accounts / device state). | + +> **Driving Microsoft Authenticator?** The 4-screen first-run flow, adding an account from the home screen, +> and the **same-device** AAD Work/School account-add + first-time MFA proof-up (number-match) are documented +> in [authenticator-app.md](authenticator-app.md) — read it before testing an Authenticator scenario so you +> don't re-hit the false "needs a second phone" blocker. + +## ECS vs Local builds (test-point configuration) + +When a case is staged with both an `ECS\` and a `Local\` folder, **which folder you install the app-under-test +from is decided by the ADO test point's *configuration*, not by a fixed default:** + +| Test-point configuration name | Build folder to install from | +|---|---| +| contains **`LocalFlights`** (e.g. `RC MSAL - RC Broker (LocalFlights)`) | **`Local\`** | +| anything else (e.g. plain `RC MSAL - RC Broker`) | **`ECS\`** | + +> ⚠️ **Counter-intuitive — read twice.** The `LocalFlights` configuration maps to the **`Local\`** folder, and +> the *plain* configuration maps to the **`ECS\`** folder. Do **not** assume "LocalFlights → ECS". A case with +> two test points is therefore installed **from `Local\` for the LocalFlights point and from `ECS\` for the +> plain point** — i.e. you run that case **twice**, once per folder. + +Install the **app-under-test** from the folder the point's configuration selects; pull any *other* required app +(a broker the case needs but didn't stage, Teams, Chrome) from that **same** folder if present, else the Play +Store. How to enumerate a case's test points and read each one's configuration is in the SKILL +([Phase 1 → Test points and configurations](../SKILL.md#test-points-and-configurations-which-build-to-run)). + +## Discover the exact package & launch activity at runtime + +Do not trust the hints above blindly. Resolve the real values: + +```powershell +# applicationId from the module's build.gradle (accounts for variant suffixes): +Select-String -Path msal/testapps/testapp/build.gradle -Pattern 'applicationId' + +# After install, confirm the installed package and find the LAUNCHER activity: +./scripts/appcontrol.ps1 is-installed -Package com.msft.identity.client.sample.local +adb shell cmd package resolve-activity --brief -c android.intent.category.LAUNCHER + +# Or read a built APK directly: +& "$env:ANDROID_HOME\build-tools\\aapt.exe" dump badging | Select-String 'package:|launchable-activity' +``` + +Then launch with the resolved values: +```powershell +./scripts/appcontrol.ps1 launch -Package -Activity +``` + +## Two E2E execution modes + +**Mode A — Automated instrumented tests (preferred when a matching test exists).** +The automation apps drive full sign-in flows and fetch lab accounts automatically via the Lab API. +Run the connected test task for the variant: +```powershell +./gradlew :msalautomationapp:connectedLocalDebugAndroidTest --tests "**" +./gradlew :brokerautomationapp:connectedLocalDebugAndroidTest --tests "**" +``` +Results land in `build/reports/androidTests/connected/` and `build/outputs/androidTest-results/`. +Read those reports plus `./scripts/authlogs.ps1 scan` to judge pass/fail. + +**Mode B — Interactive UI-driven (for new features with no automated test yet).** +Install the test app, launch it, and drive the UI with `deviceui.ps1` (tap the acquire-token +button, sign in, handle prompts), then verify with `authlogs.ps1 scan`. Use this when the feature +is brand-new or the scenario is exploratory. See [ui-interaction.md](ui-interaction.md). + +Choose Mode A if an automation test already covers the flow; otherwise Mode B. + +## Broker pairing rules + +Brokered auth requires **a calling app + a broker app**, both installed: + +- The broker must be a build the client trusts (signature/redirect-URI must match). Local `localDebug` + test apps are built to trust the `localDebug` broker signatures — keep both on the same variant. +- Signature mismatch on install shows `INSTALL_FAILED_UPDATE_INCOMPATIBLE` → uninstall the old one first + (`./scripts/appcontrol.ps1 uninstall -Package `). +- Only one active broker is used at a time. If both Authenticator and Company Portal are present, the + broker-selection logic picks one; for deterministic tests install a single broker (or a mock). +- **When Authenticator is the app under test, uninstall the competing brokers first** so the broker election + can't pick the wrong one: + ```powershell + ./scripts/appcontrol.ps1 uninstall -Package com.microsoft.teams -Serial + ./scripts/appcontrol.ps1 uninstall -Package com.microsoft.windowsintune.companyportal -Serial + ``` + Teams and Company Portal both register as brokers, so leaving either installed makes "which broker handled + this?" non-deterministic between runs. The Authenticator team's own automated suite does exactly this as a + fixed pre-test rule (see [existing-ui-automation.md](existing-ui-automation.md)). Obviously **skip this for + a case that deliberately tests Company Portal or Teams as the broker** (e.g. device registration via CP). +- After installing/replacing a broker, **clear the client app state** (`appcontrol.ps1 clear`) so it + re-discovers the broker cleanly. + +## Test credentials (Lab API) and when they block + +- Automated tests (Mode A) obtain lab accounts through `common/labapi` / the Lab API automatically — + no manual credential handling. +- Interactive tests (Mode B) need a test account. Preferred sources, in order: + 1. A lab account the user provides in the session. + 2. A credential the automation harness/Lab API exposes. +- **Provision or repair accounts on demand with `scripts/labapi.ps1`** (`create-user`, `reset`, + `enable-policy`/`disable-policy`, `delete-device`). Full endpoint list, usertypes/policies, the EasyAuth + auth workaround, and entitlements are in **[lab-api.md](lab-api.md)**. +- **Blocker to raise with the user** if none is available, or if the scenario needs a real MSA/consumer + account, a specific tenant/CA policy, or a federated account that the AI cannot provision (try + `labapi.ps1 disable-policy` first if it's a lab CA policy). +- **Never** hardcode, print, or commit real credentials. Type them into the device only (`input-text + -Secret`); never echo them into logs or the transcript. + +## Emulator / real-device requirements per feature + +Pass these to `emulator.ps1 ensure` (they gate both AVD selection and real-device eligibility): + +- **Broker / push / GMS-dependent flows** → `-RequireGoogleApis` (needs Google Play services for FCM, + account manager). Prefer a `google_apis` or `google_apis_playstore` image. +- **Min OS behavior** (e.g. scoped storage, notification permission on API 33+) → `-ApiLevel `. +- **Play Store app installs** (rare in tests) → `-RequirePlayStore`. +- Most auth E2E works on a standard `google_apis` image, API 30+. diff --git a/.github/skills/android-e2e-tester/references/authenticator-app.md b/.github/skills/android-e2e-tester/references/authenticator-app.md new file mode 100644 index 00000000..610b1566 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/authenticator-app.md @@ -0,0 +1,382 @@ +# Microsoft Authenticator — first-run flow & AAD account-add (proof-up / number-match) + +Microsoft Authenticator (`com.azure.authenticator`) is one of the most common apps-under-test in this +skill. Two of its flows trip up automation in ways that are **not** obvious from the screen, and both cost +real runs before they were understood: + +1. the **first-run flow** — a fresh install shows **4 gates before the home screen**; miss one and you + never reach the account list, or you accidentally start an account-add from the upsell; and +2. the **AAD Work/School account-add with first-time MFA proof-up (number-match)** — which is fully + automatable on **one device** but was repeatedly (and wrongly) marked BLOCKED as "needs a second phone". + +Read this before driving any Authenticator scenario. Match screens by their **on-screen text**, not a fixed +index — order varies slightly by OS/app version. See also +[common-blockers.md](common-blockers.md) (App Lock / number-match / blue-pixel tap / blank Chrome / FLAG_SECURE) +and [app-and-module-map.md](app-and-module-map.md#known-provided-apks) (APK filenames + package). + +- [First-run flow (fresh install → home)](#first-run-flow-fresh-install--home) +- [Home screen & adding an account](#home-screen--adding-an-account) +- [Ready-made flow specs (copy-paste)](#ready-made-flow-specs-copy-paste) +- [AAD Work/School account-add with proof-up (number-match) — SAME DEVICE](#aad-workschool-account-add-with-proof-up-number-match--same-device) +- [Enabling Passwordless sign-in (PSI) on an account](#enabling-passwordless-sign-in-psi-on-an-account) +- [App Lock auto-enables when a device PIN exists](#app-lock-auto-enables-when-a-device-pin-exists) +- [What went wrong before — and the fix](#what-went-wrong-before--and-the-fix) +- [Cross-references](#cross-references) + +## First-run flow (fresh install → home) + +A freshly installed Authenticator (Phase 3 uninstall+reinstall gives you exactly this) shows **4 screens +before the home screen**. Drive them in order — each is a `deviceui.ps1 dump` → `tap-text`: + +1. **Enable notifications** — the Android runtime-permission dialog ("Allow *Authenticator* to send you + notifications?") → tap **Allow**. **Do not deny/skip:** the proof-up's number-match arrives as a *push*, + so notifications must be on. +2. **Privacy agreement** → tap **Accept**. (No need to open the "Microsoft privacy statement" link.) +3. **"Help us improve Microsoft Authenticator"** (telemetry opt-in) → leave the checkbox **unchecked**, + tap **Continue**. +4. **"Peace of mind for your digital life"** (an upsell to add an account) → **do not** add an account + here; tap **Skip** in the **upper-right corner** → lands on the **home screen**. + +```powershell +# fast path: one dump per screen, tap by text, verify by the next screen's anchor +./scripts/deviceui.ps1 tap-text -Text "Allow" -Then "Accept" -Serial # notifications → privacy +./scripts/deviceui.ps1 tap-text -Text "Accept" -Then "Continue" -Serial # privacy → telemetry +./scripts/deviceui.ps1 tap-text -Text "Continue" -Then "Skip" -Serial # telemetry → upsell +./scripts/deviceui.ps1 tap-text -Text "Skip" -Serial # upsell → home +``` + +**Prefer the resource IDs when a text tap misses.** Text is locale- and version-fragile; these ids are the +ones the team's own UIAutomator suite drives, so they're the canonical, stable handles +([see the automated suite](existing-ui-automation.md)): + +| Screen | Resource ID | +|---|---| +| Notifications permission — negative button | `android:id/button2` | +| Privacy agreement — **Accept** | `com.azure.authenticator:id/privacy_consent_button` | +| "Help us improve" — **Continue** | `com.azure.authenticator:id/privacy_consent_continue_button` | +| "Peace of mind…" upsell — **Skip** | `com.azure.authenticator:id/frx_skip_button` | +| Home — **Add account** (zero accounts) | `com.azure.authenticator:id/zero_accounts_add_account_button` | +| Home — account list | `com.azure.authenticator:id/account_list` | +| Toolbar — overflow (⋮) | `com.azure.authenticator:id/menu_overflow` | +| Overflow → **Add account** | `com.azure.authenticator:id/menu_item_add_account` | +| Overflow → **Settings** | `com.azure.authenticator:id/menu_item_settings` | +| Overflow → **Check for notifications** | `com.azure.authenticator:id/menu_check_for_notifications` | +| Number-match — code input | `com.azure.authenticator:id/auth_enter_correct_num_text_input` | +| Number-match — confirm | `com.azure.authenticator:id/positiveButton` | + +The **"+" (add account)** button in the top-right is best matched by **content-description containing +"Add"** rather than text (it's an icon). The overflow *menu items* are a **Compose popup** whose resource +ids are unreliable once open — match those by text. + +Notes: +- The **notifications** dialog is an Android 13+ (API 33+) system prompt; on older OS it may appear as an + in-app prompt or not at all. If a screen from this list doesn't appear, just move to the next — that's why + you match on text, not position. +- **Screenshots: mostly fine — don't assume they're all black.** Verified on a physical Galaxy (Android 16): + the **account list / home screen renders normally** in `screencap`, and you *need* that because the + white-vs-gray row distinction below is **only** visible in pixels. Some inner screens are still + `FLAG_SECURE` and come back black — so **capture both**: a `screencap` *and* a `uiautomator dump` (XML) at + each milestone, and if the PNG is black fall back to the XML. Always **cross-check the screenshot against + the dump's `package=`** before trusting it (see the multi-display trap below). + See [common-blockers.md → FLAG_SECURE](common-blockers.md#flag_secure-black-screenshots). +- **Foldables/multi-display: never pass `-d 0` to `screencap`.** On a Galaxy Z Flip (two HWC displays) + `adb shell screencap -p -d 0` returned a **stale frame of a different app** (Chrome) while the device was + actually showing Authenticator. Plain `adb shell screencap -p` was correct. A screenshot that disagrees + with `dumpsys window | mCurrentFocus` is a **stale capture**, not a real screen — re-capture without `-d`. + +## Home screen & adding an account + +The **home screen** is the account list. When it's empty it shows a big **"Add account"** button; there is +also a **"+"** in the **upper-right corner** that does the same thing. **Always add accounts from home** via +**"+"** / **"Add account"** — never from the first-run "Peace of mind…" upsell. + +Add-account chooser sequence: +1. **"+"** / **"Add account"** → +2. **"What kind of account are you adding?"** → **Personal account** / **Work or school account** / + **Other (Google, Facebook, etc.)**. For AAD pick **Work or school account**; for MSA pick + **Personal account**. +3. For Work or school it then offers **Sign in** / **Scan a QR code** → pick **Sign in** (QR needs a second + screen — see the QR note in [common-blockers.md](common-blockers.md)). + +**If the device already has work accounts, the broker's account chooser comes first.** Tapping **Sign in** +can open `com.microsoft.identity.common…BrokerAuthorizationActivity` showing **"Pick an account"** with the +existing accounts listed. To add a *new* one, tap **"Use another account"** — otherwise you silently re-add +an account that's already there. + +## Ready-made flow specs (copy-paste) + +The sequences above are **deterministic** — they contain no decisions — so driving them one tool call per +screen wastes a ~25 s median agent round-trip on each for ~2.5 s of device work, and that overhead is the same +whether the step is obvious or hard +([why](run-speed.md#the-overhead-is-fixed-per-call--it-is-not-thinking-time)). Run each as **one** +`deviceui.ps1 flow` call instead. Steps that only *sometimes* appear are marked `"optional": true`, so the +same spec works on API 30 and API 33+, and on a device where a permission was already granted. + +**First run → home screen** (the whole 4-gate preamble, one call, ~15–40 s): +```powershell +./scripts/deviceui.ps1 flow -Serial -Text '[ + {"label":"notifications Allow","tap":"Allow","optional":true,"waitSec":8}, + {"label":"privacy Accept","tapRes":"com.azure.authenticator:id/privacy_consent_button","waitSec":30}, + {"label":"telemetry Continue","tapRes":"com.azure.authenticator:id/privacy_consent_continue_button","waitSec":20}, + {"label":"upsell Skip","tapRes":"com.azure.authenticator:id/frx_skip_button","waitSec":20}, + {"label":"home - account list","waitRes":"com.azure.authenticator:id/account_list","waitSec":20,"optional":true}, + {"label":"home - zero accounts","waitRes":"com.azure.authenticator:id/zero_accounts_add_account_button","waitSec":5,"optional":true}, + {"label":"evidence","screenshot":"01_home.png"} +]' +``` +Resource-ids are matched first because they're locale- and version-proof. **Check the trace**: if *both* the +`account_list` and `zero_accounts_add_account_button` steps SKIP, you did **not** reach home — dump the screen +and investigate rather than continuing. + +> **Verified on-device** (emulator API 30, ECS `app-production-universal-release-signed.apk`): this exact spec +> ran **6 ok · 1 skipped · 0 failed in 16.2 s** and landed on the empty account list. The skipped step was the +> notifications dialog — API 30 doesn't show the Android 13+ prompt, which is precisely why it's +> `"optional": true`. The same spec therefore works on both older and newer OS levels. + +**Home → eSTS sign-in page for a Work/School account** (add-account chooser, one call): +```powershell +./scripts/deviceui.ps1 flow -Serial -Text '[ + {"label":"tap + (add account)","tapDesc":"Add","optional":true}, + {"label":"or Add account button","tapRes":"com.azure.authenticator:id/zero_accounts_add_account_button","optional":true}, + {"label":"account type chooser","wait":"Work or school account","waitSec":20}, + {"label":"pick Work or school","tap":"Work or school account","waitSec":20}, + {"label":"Sign in","tap":"Sign in","optional":true,"waitSec":10}, + {"label":"broker chooser (only if work accounts exist)","tap":"Use another account","optional":true,"waitSec":6}, + {"label":"eSTS email page","waitSec":45,"wait":"Sign in"}, + {"label":"evidence","screenshot":"02_ests.png"} +]' +``` + +> **Never put a credential in a flow spec.** Use `"secretRef":""` (resolves from the encrypted store and +> is never echoed), or a separate `input-text -SecretRef ` call — never `"input":""`. +> See [secrets-and-files.md](secrets-and-files.md). + +Full step-key reference is in the `Invoke-UiFlow` doc-comment in `scripts/deviceui.ps1`: +`label, wait, waitRes, waitSec, optional, tap, tapDesc, tapRes, index, exact, input, secretRef, clear, +charByChar, key, then, thenRes, thenSec, screenshot, sleepMs`. + + +## Duplicate account rows — white = official, gray = ignore + +**After a Work/School account is added but *before* its setup is finished, the home list can show TWO rows +for the SAME UPN**: one with a **white** background and one with a **gray** background. This is **normal and +happens after a SINGLE tap of the pairing link** — it is *not* caused by tapping the link twice, and it is +not an error. + +- **The GRAY row is not the official record. Always act on the WHITE row** — open it, enable Passwordless + sign-in on it, read its OTP from it. +- **Acting on the gray row is what produces a bogus `Passkey — Unknown error`** when you try to enable + Passwordless sign-in. That error means "wrong row", not "the feature is broken". +- **The gray row disappears by itself** once the account is fully set up (verified: the list went 4 rows → + 3 rows the moment Passwordless sign-in finished on the white row). So don't try to delete it. + +**How to tell them apart — you cannot do it from the XML.** Both rows expose *identical* +`resource-id="com.azure.authenticator:id/account_chevron_right_icon"` nodes with the **same** +`content-desc="Open account details for "`; only `bounds` differ. The background colour is **not** in +the accessibility tree. Two reliable options, in order: + +1. **Sample the pixel (authoritative).** The list is not FLAG_SECURE, so screencap it and read the row + background just left of the chevron. Verified values on a 1080-wide device: **white/official = + `RGB(255,255,255)`**, **gray/ignore = `RGB(189,189,189)`**. + ```powershell + Add-Type -AssemblyName System.Drawing + $bmp = [System.Drawing.Bitmap]::FromFile("$run\shots\list.png") + # for each chevron node: bounds="[960,916][1032,988]" -> x = left-60, y = vertical centre + $p = $bmp.GetPixel(900, 952) + $isOfficial = ($p.R -ge 245 -and $p.G -ge 245 -and $p.B -ge 245) # white => official row + $bmp.Dispose() + ``` +2. **Fallback heuristic** (if the screenshot is unusable): when a UPN appears twice, the **upper/earlier** + occurrence is the white/official one and the gray duplicate sorts **last**. + +## AAD Work/School account-add with proof-up (number-match) — SAME DEVICE + +When you add a Work/School account whose tenant requires first-time MFA registration, Authenticator can't +finish the pairing in-app — it hands you to a browser to "prove up". **This entire flow runs on the one +device you're already on.** The browser is launched *by Authenticator*, and a deep-link bounces the +number-match back into the app. Drive it end-to-end: + +1. Home → **"+"** → **Work or school account** → **Sign in**. +2. On the Microsoft sign-in page enter **UPN** then **password** and sign in. (Type into these WebView fields + with `-Clear -CharByChar`; password via `-SecretRef` — see + [common-blockers.md → Chrome autofill overlay](common-blockers.md#chrome-autofill--passkey-overlay-steals-input).) +3. Authenticator shows **"Finish setting up on a web browser"** — *"To set up Microsoft Authenticator, + you'll need to go to aka.ms/mfasetup on a web browser."* with **Cancel** / **Open browser**. + → tap **Open browser**. **This is the normal path — NOT a "second device" blocker.** It opens the + proof-up in a browser on **this** device. +4. Browser: **"Let's keep your account secure"** → tap **Next**. +4b. Browser: **"Install Microsoft Authenticator"** may appear as an *extra* screen here (it isn't in every + tenant's flow) → tap **Next**. Don't treat it as a wrong turn. +5. Browser: **"Now pair Authenticator with your account"** — offers the hyperlink **"Pair your account to + the app by clicking this link."**, plus **"Show QR code"**, **"Other options"**, and a **Next** button. + → tap the **"Pair your account to the app by clicking this link."** hyperlink **exactly once**. **This is + the critical, easy-to-miss step — do NOT tap Next, and do NOT tap "Show QR code".** The link fires a + `msauth://` deep-link back into Authenticator that drives the number-match. (The hyperlink lives inside a + WebView and often is **not** a tappable node in `uiautomator dump`; if `tap-text` can't find it, use the + blue-pixel tap — see [common-blockers.md → Single-use pairing / setup links](common-blockers.md#single-use-pairing--setup-links).) +5b. **Confirm the tap by the app coming to the FOREGROUND, not by the page changing.** A success **toast** + fires and the Chrome page **deliberately stays on the same content** — an unchanged page is **not** a + missed tap. The authoritative signal is `dumpsys window | mCurrentFocus` flipping to + `com.azure.authenticator/...MainActivity`. **Never tap the link a second time** — it is single-use, and a + re-tap yields **"Unable to add the account"**, then a fallback **Add account** screen (pre-filled *Code* + + *URL* + **FINISH**) whose FINISH fails with **"QR code already used"**. Those two errors are *proof the + first tap worked* — treat them as confirmation, not as a new failure. +6. Browser: **"Let's try it out"** shows a **number** (e.g. `87`) — *"Enter the number shown in the app to + approve the sign-in request."* **Read this number** from the Chrome page (`deviceui.ps1 dump` or a + screenshot — Chrome is **not** FLAG_SECURE, so it captures fine). **Act promptly — the number times out.** + **The browser wizard may instead dead-end** on the pairing page with *"We're sorry, we ran into a problem. + Please choose 'Next' to try again."* **even though the app-side add already succeeded.** Don't fight it and + don't re-tap the link: **go check the app's account list first** — if the account is there (see the + white/gray rule above), the add is done and you can finish setup entirely in-app (next section). The + browser wizard is **not** required to enable Passwordless sign-in. +7. Authenticator fires a push **"Approve sign-in?"** for the account. **Give it ~10 s before you look** — + the FCM push lands about 2 s after the trigger as a **heads-up banner that covers Chrome's URL bar**, then + auto-collapses after ~5 s. Reading the screen too early gets you the banner instead of the page (or an + obscured number); reading it too late loses the banner. Then bring up the number-match screen by **any** + of these (whichever appears first): + - pull down the **notification shade**, find **"Approve sign-in?"**, and tap it; **or** + - **open the Authenticator app** — the number-match screen often pops up on its own; **or** + - **Check for notifications** from the overflow menu (⋮ `menu_overflow` → + `menu_check_for_notifications`) — a **deterministic** poll, and preferable to the gesture below; **or** + - if it still doesn't show, **pull-to-refresh** in the app to summon it. +8. Number-match screen **"Are you trying to sign in?"** (some versions title it **"Enter the number shown to + sign in"**). **Two variants exist — check which one you got:** + - **Number-entry variant** — an **"Enter number here"** field plus **YES** / **NO, IT'S NOT ME** / + **I CAN'T SEE THE NUMBER**: type the number from step 6, then tap **Yes**. + - **Simple approve variant** (verified on Android 16) — **no** number field, just + `positiveButton` **Yes** / `negativeButton` **No, not me** / `cancelButton` **Cancel**: just tap **Yes**. + Don't stall looking for a field that isn't there. + + *(If Authenticator's **App Lock** / a biometric gates this screen, satisfy it first — prefer an + **emulator** so you can inject a fingerprint; see + [common-blockers.md → fingerprint/App Lock](common-blockers.md#steps-that-need-a-fingerprint--biometric--app-lock).)* +9. Browser returns to **"Authenticator Added"** (green check) — *"This is now your default sign-in method."* + → tap **Done**. The browser is now signed in to the account. +10. Return to the **Authenticator app** → the account now appears in the home list. **That's the success + criterion** — verify it with **both** a `screencap` (the list renders; you need the pixels for the + white/gray check) and a `uiautomator dump` of the account list. + +Reference screenshots of a successful GlobalMFA add (human-driven) are archived at +`C:\Users\zhipanwang\Pictures\Screenshots\tc2579657` (the "Open browser" screen, the pairing screen, the +number `87`, the number-match dialog, and "Authenticator Added"). + + +## Enabling Passwordless sign-in (PSI) on an account + +Do this **on the white row** (see [duplicate rows](#duplicate-account-rows-white-vs-gray)). Verified +end-to-end on a physical Galaxy running Android 16: + +1. Home list → tap the **white** row's chevron → **account detail**. A healthy record shows + **Sign-in notifications**, a live **One-time password code**, and the buttons **Create a passkey** and + **Set up Passwordless sign-in requests**. +2. Tap **"Set up Passwordless sign-in requests"**. This starts a **re-auth**, which raises a number-match + challenge — the challenge number is displayed on the *re-auth page itself* (`Approve sign in request` + + the number). +3. A push **"Approve sign-in?"** lands in the notification shade for that UPN. Expand the shade + (`adb shell cmd statusbar expand-notifications`) and tap it. +4. **"Are you trying to sign in?"** → tap **Yes** (see the two variants in step 8 above). +5. **"Let's secure your account — Passwordless sign-in requests — Sign in without a password."** → tap + **Continue**. +6. A **biometric prompt** appears: *"Scan your fingerprint."* with a **"Use PIN"** fallback. On a **physical** + device you cannot inject a fingerprint — **tap "Use PIN"**. That opens the OEM credential screen + (Samsung: `com.samsung.android.biometrics.app.setting`, field + `…:id/lockPassword`, already focused) → type the stored device PIN with + `deviceui.ps1 input-text -SecretRef devicepin_` (never echoed) → tap **Continue**. + **This "Use PIN" fallback is why PSI does *not* require an emulator.** +7. **"Account added — You can now sign in or verify using the following methods: Passwordless sign-in + requests"** → tap **Done**. +8. Back on the home list the account now shows the **blue left-edge bar** (= PSI enabled), and any **gray + duplicate row for that UPN disappears**. + +**If you get `Passkey — Unknown error` here, you almost certainly opened the GRAY row.** Go back and repeat +on the white one. + +### ⚠️ Run PSI on a PHYSICAL device — it needs WPJ, which fails on the emulator + +The walkthrough above was verified on a **physical** Galaxy. On a **Google-APIs emulator (API 36)** the same +flow inserts an extra gate after step 5's **Continue**: **"Register your device"** → *"Help us keep your +device secure"* → **Register** → screen-lock passkey credential → and then dead-ends at +**"Device isn't registered"**. Reproduced on **2 independent end-to-end attempts** (an earlier expired-session +attempt instead surfaced *"Passkey — Unknown error"*, which is the number-match TTL, not this). + +Consequences to plan around: +- **PSI-enabled state is unreachable on an emulator**, so the account's gear menu never offers + **"Disable Passwordless Sign in"** (only **Remove account**) — any teardown step that depends on disabling + PSI first is unverifiable there. +- The failure is at **AAD device registration**, *not* at the PSI request/number-match flow — that half + succeeds cleanly (approval → *"Let's secure your account"* → Continue). Say so in the report. +- Budget **~12 minutes** for WPJ; if it fails twice, record the exact error as the blocker and move on. +- Registering a **shared** physical phone leaves durable state — that's a legitimate reason to mark the step + BLOCKED rather than proceed. Use a dedicated/disposable real device when PSI must actually be proven. + + +## App Lock auto-enables when a device PIN exists + +**If the device has a screen lock (PIN/pattern/biometric), Authenticator turns App Lock ON by itself** as +soon as an account lands — it shows an *"App Lock enabled"* popup (dismiss with **OK**). Nobody asked for it, +and it then gates **every** return to the app behind a biometric/PIN prompt. That is a silent killer for any +flow that switches away and back (browser → app → browser), which is exactly what proof-up, number-match and +PSI all do. + +**So unless the case is specifically testing App Lock, turn it off immediately after the account is added:** + +1. Dismiss the *"App Lock enabled"* popup → **OK**. +2. Overflow (⋮ `menu_overflow`) → **Settings** (`menu_item_settings`) → toggle **App Lock** off. +3. Return to the account list and continue the flow. + +This is what the team's own automated suite does as a fixed step in its account-add flow — it is not +optional hygiene, it is the difference between a flow that switches apps cleanly and one that stalls on a +biometric prompt you may not be able to satisfy on a physical device. Combine with the +[fingerprint/App Lock decision](common-blockers.md#steps-that-need-a-fingerprint--biometric--app-lock): +if a case *does* require App Lock, prefer an emulator so you can inject a fingerprint. + +Corollary: if you set a device PIN yourself (via `locksettings set-pin`, see +[common-blockers.md](common-blockers.md#steps-that-need-a-fingerprint--biometric--app-lock)), **expect App +Lock to switch on** the moment an account is added — plan the disable step into the flow rather than being +surprised by it mid-run. + + +## "Your sign-in information may have changed" can hijack the run + +Authenticator may raise a modal — *"Your sign-in information may have changed. You will need to log in again +to your account."* with **Cancel** / **Continue** — for **any** account it holds, including an **old, expired +temp user unrelated to the account you're setting up**. It steals the foreground and can intercept the +pairing deep-link, which then makes the *browser* wizard fail with "We're sorry, we ran into a problem". + +- **Read the dialog's account before acting.** Tapping **Continue** launches a full re-auth **for that other + account** (`BrokerAuthorizationActivity` pre-filled with *its* UPN) and derails your run. +- Unless that account is the one under test, tap **Cancel** (or back out) and carry on. +- Expect this whenever the device carries **temp lab users older than their ~60-min TTL** — their refresh + tokens are dead, so the app keeps asking. + +## What went wrong before — and the fix + +The batch that marked cases **2579657, 2723696, 2916481, 1579402, 2916570, 1579417** as BLOCKED with an +"AAD first-time MFA proof-up wall — same-device pairing infeasible" note was **wrong**. The flow above works +on a single device. Three mistakes, each with its fix: + +| Mistake | Symptom logged | Fix | +|---|---|---| +| Treated **"Finish setting up on a web browser / aka.ms/mfasetup"** as needing a *second device* | "same-device pairing infeasible" → BLOCKED | It's the normal path. Tap the app's **Open browser** button; the whole proof-up runs on this device. | +| Navigated to **aka.ms/mfasetup manually** in a separate Chrome tab | "aka.ms/mfasetup blank / BadRequest" | **Don't hand-type that URL.** Use the **Open browser** button the app provides so the deep-link handoff stays intact. If any Chrome page is blank, clear the FRE first — [common-blockers.md → Chrome First Run Experience](common-blockers.md#chrome-first-run-experience-swallows-the-auth-page-blank-webview). | +| On **"Now pair Authenticator with your account"** tapped **Next** (or **Show QR code**) | pairing "didn't return to the app" | Tap the **"Pair your account to the app by clicking this link."** hyperlink — that's the deep-link that drives the number-match. | +| Treated the **unchanged Chrome page** after tapping the pair link as a failed tap | re-tapped → "Unable to add the account" / "QR code already used" → concluded pairing was broken | The page is *supposed* to stay put; confirmation is a toast you'll miss. Verify by **Authenticator coming to the foreground**, then continue in the app. | +| Ran **"Set up Passwordless sign-in requests"** on the **gray** duplicate row | "Passkey — Unknown error. Please try again later." → PARTIAL | Use the **white** row (pixel `RGB(255,255,255)`). Verified: PSI succeeded immediately on the white row for the *same* account. [Duplicate rows](#duplicate-account-rows-white-vs-gray) | +| Kept retrying the **browser wizard** after it errored | stuck in a "We're sorry, we ran into a problem" loop | The app-side add had already succeeded. Check the account list; finish setup **in-app** — the wizard isn't required for PSI. | +| `pm clear`-ed **Chrome** to drop a previous account's SSO | re-triggered Chrome's First Run Experience + profile prompts, adding many extra steps | Use the sign-in page's own **"Use another account"** — no data clearing needed. | + +**Only** escalate an AAD proof-up as a genuine blocker when the number-match screen is gated behind a +biometric you can't inject (App Lock on a **physical** device — move that segment to an emulator), or the +tenant/CA genuinely can't be satisfied. "It sent me to a browser" is not a blocker. + +## Cross-references + +- **Number-match, App Lock, biometric gating** → [common-blockers.md](common-blockers.md#number-match-mfa) +- **Pairing hyperlink not in the accessibility tree (blue-pixel tap)** → + [common-blockers.md → Single-use pairing / setup links](common-blockers.md#single-use-pairing--setup-links) +- **Blank Chrome page on first navigation** → + [common-blockers.md → Chrome First Run Experience](common-blockers.md#chrome-first-run-experience-swallows-the-auth-page-blank-webview) +- **Authenticator screenshots are black (FLAG_SECURE) — use `uiautomator dump`** → + [common-blockers.md → FLAG_SECURE](common-blockers.md#flag_secure-black-screenshots) +- **APK filenames + package + module** → + [app-and-module-map.md](app-and-module-map.md#known-provided-apks) diff --git a/.github/skills/android-e2e-tester/references/common-blockers.md b/.github/skills/android-e2e-tester/references/common-blockers.md new file mode 100644 index 00000000..d5fc997e --- /dev/null +++ b/.github/skills/android-e2e-tester/references/common-blockers.md @@ -0,0 +1,579 @@ +# Common Scenarios, Hiccups & Blockers + +Recurring friction from real E2E runs, what causes it, and how to get past it — plus **when to switch +from a physical device to an emulator**. Read this before driving an auth flow so you don't lose time +rediscovering a known trap. See also [ui-interaction.md](ui-interaction.md) (screen-by-screen driving) +and [troubleshooting.md](troubleshooting.md) (env/build/install). + +Table of contents: +- [Decision: emulator vs physical device](#decision-emulator-vs-physical-device) +- [Steps that need a fingerprint / biometric / App Lock](#steps-that-need-a-fingerprint--biometric--app-lock) +- [Number-match MFA](#number-match-mfa) +- [WPJ device registration fails on the emulator (blocks Passwordless Sign-In)](#wpj-device-registration-fails-on-the-emulator-blocks-passwordless-sign-in) +- [Verification codes read from Gmail — the list preview goes stale](#verification-codes-read-from-gmail--the-list-preview-goes-stale) +- [A pre-granted runtime permission suppresses the in-app consent prompt](#a-pre-granted-runtime-permission-suppresses-the-in-app-consent-prompt) +- [Session timeouts & SSO resets mid-flow](#session-timeouts--sso-resets-mid-flow) +- [Chrome autofill / passkey overlay steals input](#chrome-autofill--passkey-overlay-steals-input) +- [Chrome First Run Experience swallows the auth page (blank WebView)](#chrome-first-run-experience-swallows-the-auth-page-blank-webview) +- [FLAG_SECURE black screenshots](#flag_secure-black-screenshots) +- [Screenshot corruption via redirection](#screenshot-corruption-via-redirection) +- [Single-use pairing / setup links](#single-use-pairing--setup-links) +- [Stale account state between runs](#stale-account-state-between-runs) +- [Fresh temp user not sign-in-able yet (ESTS propagation lag)](#fresh-temp-user-not-sign-in-able-yet-ests-propagation-lag) +- [Password rejected at sign-in — don't reset it](#password-rejected-at-sign-in--dont-reset-it) +- [Doing it yourself in System Settings](#doing-it-yourself-in-system-settings) +- [Genuine blockers (stop and ask)](#genuine-blockers-stop-and-ask) +- [Quick reference table](#quick-reference-table) + +## Decision: emulator vs physical device + +The skill can run on either, but some steps are **only automatable on an emulator** because they need an +input you can inject programmatically. Choose up front from the test case's steps: + +**Prefer an emulator when the scenario includes any of:** +- A **fingerprint / biometric** prompt you must satisfy (enroll + inject a touch). +- **App Lock** in Microsoft Authenticator (it re-prompts for device credential/biometric). +- A **biometric-gated number-match** approval. +- Anything that needs `adb emu ...` (finger, sensors, GSM, battery) — emulator-only console commands. + +**A physical device is fine (or better) when:** +- No biometric is required (pure password + KMSI + token acquisition). +- You're on a **GPU-less host** (Cloud PC / VM / RDP) where the emulator is painfully slow — a physical + device is much faster there (see [emulator-performance.md](emulator-performance.md)). +- The feature needs real hardware (real FCM push, real SIM) — though real push MFA is still a human step. +- The step needs **Workplace Join (WPJ) device registration** — it does **not** complete on the emulator, so + anything gated behind it (notably **Passwordless Sign-In**) can only be proven on a real device + ([below](#wpj-device-registration-fails-on-the-emulator-blocks-passwordless-sign-in)). + +> Rule of thumb: **biometric/App-Lock → emulator; everything else → whatever is fastest** (usually a +> connected physical device on a Cloud PC). If you start on a physical device and hit an unavoidable +> fingerprint gate, that's the signal to re-run the biometric segment on an emulator. + +> **Caveat — the emulator isn't always available.** On some GPU-less hosts the emulator **won't stay up**: +> the Bluetooth HAL crash-loops and takes `system_server` down repeatedly, so a heavy app (e.g. +> Authenticator) can't be driven even after it boots. See the boot recipe and the instability signal in +> [troubleshooting.md](troubleshooting.md#emulator-wont-start-or-boot). If you've exhausted that recipe and +> it still won't stabilize, then a **biometric-gated step becomes a genuine blocker on this host** — do the +> non-biometric parts on the physical device, mark the biometric step BLOCKED with evidence, and ask the +> user (don't burn the whole run fighting the emulator). Also make sure you install the **right ABI**: use +> the **universal** APK on emulators — a large `arm64-v8a`-only APK can crash install-time dexopt on an +> `x86_64` image (see [Install failures](troubleshooting.md#install-failures)). + +## Targeting the right device (multiple devices / same model) + +Every attached device gets a **unique adb serial**, independent of model — two identical phones still show up +as two different serials. List them with their model/transport so you can tell them apart: + +```powershell +adb devices -l +# emulator-5554 device product:sdk_gphone64_x86_64 model:sdk_gphone64_x86_64 transport_id:21 +# 39181FDH2000ABC device product:husky model:Pixel_8_Pro transport_id:9 +# 39181FDH2000XYZ device product:husky model:Pixel_8_Pro transport_id:12 <- same model, different serial +``` + +Then **always pass `-Serial `** to every `deviceui.ps1` call so the action lands on the intended +device. If you omit `-Serial` while more than one device is attached, adb **errors out** ("more than one +device/emulator") instead of guessing — a safe failure, not a wrong-device action, but it will stall the run. +When the human owns several devices, ask which serial to use (or which model), and store each device's PIN +with **`secrets.ps1 set-device-pin`** — it shows a numbered picker (serial + model, so you can tell same-model +units apart) and saves the PIN under that device's own name `devicepin_`. Then `unlock -Serial ` +**auto-resolves** the matching PIN, so it always targets the correct device without you re-typing a secret name. +Serials are stable for a physical device across reconnects; emulator serials (`emulator-55xx`) +depend on the console port and can change between boots, so re-check `adb devices -l` at the start of a run. + +## Steps that need a fingerprint / biometric / App Lock + +`adb emu finger touch ` injects a fingerprint **only on an emulator**. A physical device has no adb +path to press the sensor — a human must do it. Microsoft Authenticator makes this bite because it +**enables App Lock by default** right after you register an account, so the *next* action (re-opening the +app, approving a browser number-match) is gated behind the device PIN/biometric. + +Options, best first: +1. **Run on an emulator.** Enroll once, then inject touches: + ```powershell + ./scripts/deviceui.ps1 finger-enroll -Serial # sets a PIN + enrolls (idempotent) + ./scripts/deviceui.ps1 finger -Text 1 -Serial # inject a touch when the prompt appears + ./scripts/deviceui.ps1 finger-status -Serial # verify one is enrolled + ``` +2. **Turn App Lock off** so no biometric is required: Authenticator → Settings → toggle **App Lock** off + (drive it with `tap-text`), then proceed with password-only steps. Do this early if the scenario + doesn't specifically test App Lock. +3. **Fall back to a device PIN.** Many biometric prompts have a "Use PIN" path, so a known device PIN often + substitutes. Which route you take depends on whether the device **already has a screen lock**: + + **(a) No lock set → provision your own PIN with `locksettings`** (works on emulators *and* physical + devices; this is how the team's UIAutomator suite does it, and it's far more reliable than driving the + Settings UI). Full lifecycle — set at case start, clear at case end: + ```powershell + adb -s shell locksettings set-pin 123456 # only succeeds if NO credential exists yet + adb -s shell locksettings verify --old 123456 # confirm it took + adb -s shell locksettings clear --old 123456 # remove it when the case is done + ``` + ⚠️ `set-pin` **fails if a credential already exists** — so clear defensively first (`clear --old `) + to recover from a crashed/retried run that left one behind, and always clear at the end so the next case + starts from a known state. Note that setting a PIN makes **Authenticator auto-enable App Lock** — see + [authenticator-app.md](authenticator-app.md#app-lock-auto-enables-when-a-device-pin-exists). + + **(b) The device already has the user's lock → don't touch it.** Seed the PIN once into the encrypted + store, then let the tool type + **verify** it — it checks the keyguard actually cleared and **stops after + 3 tries** (`-MaxAttempts`, default 3) so a wrong PIN can't drive a physical device into an escalating + lockout: + ```powershell + ./scripts/secrets.ps1 set-device-pin # picks the device, you paste the PIN (DPAPI-encrypted) + ./scripts/deviceui.ps1 unlock -Serial # auto-uses devicepin_; verified; gives up after 3 + ``` + On an **emulator**, satisfy biometric prompts with `adb -s emu finger touch 1` once a PIN exists. +4. **Probing an unknown PIN on a physical device — don't brute-force.** `adb shell locksettings verify --old ` + tests a guess **non-destructively** (it doesn't change anything), but Android's **Gatekeeper throttles + after ~5 wrong tries** and too many failures can lock the user out of their own device. Try at most a + couple of obvious values, and if they miss, **stop and ask the user for the PIN** — never loop guesses on + hardware you don't own. +5. **Physical device + human.** If the scenario *must* run on hardware, pause and ask the user to press the + sensor, then resume — see [Genuine blockers](#genuine-blockers-stop-and-ask). + +## Number-match MFA + +Modern MFA shows a **number in the browser** that you must select/enter in Authenticator. It's +automatable **only if** getting into Authenticator isn't biometric-gated: +- If App Lock is on → satisfy the biometric first (see above) → then read the number from the browser + (`deviceui.ps1 dump` on the Chrome page — it's **not** FLAG_SECURE) and tap the matching tile in + Authenticator. +- If the number appears as a **push you must approve on a *separate* physical phone** whose Authenticator + is *already registered* and you don't control (no TOTP seed), it's a genuine blocker — ask the user. + +> **Not a blocker: adding a work account to *this* device's Authenticator (first-time proof-up).** When you +> add an AAD Work/School account and the tenant requires first-time MFA registration, Authenticator sends +> you to a browser ("Finish setting up on a web browser / aka.ms/mfasetup") and the number-match bounces +> back to the app **on the same device** — this is fully automatable and must **not** be marked BLOCKED as +> "needs a second phone". The critical, easy-to-miss step is tapping the **"Pair your account to the app by +> clicking this link."** hyperlink (not **Next**, not **Show QR code**) on the *"Now pair Authenticator with +> your account"* screen. Full step-by-step, plus the exact mistakes that caused false BLOCKEDs before, are in +> [authenticator-app.md](authenticator-app.md#aad-workschool-account-add-with-proof-up-number-match--same-device). + +### The number-match session expires in ~1–2 minutes — collapse the steps around it + +The pending number-match session is **short-lived (~1–2 min)**. Any detour you take between the browser +showing the number and you approving it in the app can kill it — the app then shows nothing to approve, or +the browser errors out ("we're having trouble verifying", *"Passkey — Unknown error"*, "couldn't add +account"). Real cases where this bit: +- Toggling **airplane mode** on/off to prove offline behaviour burned the session mid-toggle. +- Chasing a missing push through Settings/notification-shade UI took longer than the TTL. + +**Fix: make the whole detour one scripted shell call** so the wall-clock cost is seconds, not minutes. E.g. +offline-then-refresh in a single ~21 s call: +```powershell +adb -s $s shell cmd connectivity airplane-mode enable +adb -s $s shell uiautomator dump /sdcard/off.xml # evidence: nothing surfaced offline +adb -s $s shell cmd connectivity airplane-mode disable +Start-Sleep -Seconds 6 # let Wi-Fi reassociate +# overflow ⋮ → Check for notifications, then dump +``` +Batching it this way reproduced the re-surface **3×** in a row on a case that had previously been ABORTED. +If the session *does* expire, don't fight it — re-trigger from the browser to get a **fresh** number rather +than retrying the dead one. + +## WPJ device registration fails on the emulator (blocks Passwordless Sign-In) + +**Passwordless Sign-In (PSI)** in Authenticator is gated behind **Workplace Join / device registration**: the +setup flow goes *Set up Passwordless sign-in requests* → number-match approval → *"Let's secure your account"* +→ **"Register your device"** → Register → screen-lock/passkey credential. + +On a **Google-APIs emulator (verified API 36)** the number-match and passkey portions succeed, but the AAD +device-registration step then fails with **"Device isn't registered"** — reproduced on 2 independent +end-to-end attempts. So: + +- **PSI-enabled state is unreachable on an emulator.** Anything downstream of it — most importantly + *"Disable Passwordless Sign in"* in the account's gear menu — can't be exercised there either (the menu + only offers **Remove account**). +- **Don't burn the run on it.** Give WPJ ~12 minutes; if it fails twice, record the exact error as the + blocker and move on. It is a genuine, reportable finding, not a harness bug to work around. +- **Use a real device you're willing to register** if the case genuinely requires PSI. Note that registering + a *shared* physical phone leaves durable state — decide deliberately (a disposable/dedicated device is the + right answer; declining to WPJ a shared phone is a legitimate reason to mark the step BLOCKED). +- The **failure is at device registration, not at the PSI request flow** — say so in the report, since the + approval half genuinely passed. + +## Verification codes read from Gmail — the list preview goes stale + +When a flow emails a one-time code (e.g. MSA email-2SV) and you read it from the Gmail app, **the message +list's preview text is cached and can lag by many minutes**. Reading the code from that preview gives you a +**previous** run's code, which the site rejects — and the rejection looks exactly like "wrong code typed", +so it's easy to keep retrying the same dead value. This cost ~11 minutes on one run: the same stale code was +typed **4×** before the real one was found. + +**Rule: never read a code from the message-list preview. Open the message and check its timestamp.** +1. Note the wall-clock time you requested the code. +2. Open Gmail, **tap into the message** (don't read the list row). +3. Confirm the message's own timestamp is **newer than** the request time. If it isn't, the mail hasn't + arrived yet — pull to refresh and wait, don't type what you can see. +4. Read the code from the opened message body. + +Same caution applies to any notification-shade preview of an incoming code. + +## A pre-granted runtime permission suppresses the in-app consent prompt + +If a test step's expected result **is the permission prompt itself** (e.g. an LBAC / location-based access +case whose assertion is *"Share your precise location with <tenant>"* appearing in-app), then +**pre-granting that permission during device prep destroys the thing you're trying to observe** — Android +never shows a consent dialog for an already-granted permission, and the step looks like a silent failure. + +- **Don't** `appcontrol.ps1 grant` (or `pm grant`) a permission the case expects the user to be *asked* for. + Pre-granting is only a time-saver for permissions that are incidental to the assertion. +- If you already granted it, **revoke and re-trigger**: + ```powershell + adb -s $s shell pm revoke android.permission.ACCESS_FINE_LOCATION + adb -s $s shell pm revoke android.permission.ACCESS_COARSE_LOCATION + ``` + then re-run the step — the prompt surfaces normally. +- Read the case's expected results **before** device prep and keep a short "do not pre-grant" list for that + case. + +## Session timeouts & SSO resets mid-flow + +Switching between the browser and Authenticator (or a slow manual segment) can **time out the eSTS web +session**. Symptoms: you return to the browser and it's back on the password page, or SSO silently +re-prompts. Mitigations: +- Keep the biometric/PIN setup done **before** you start the timed web segment so you don't stall on it. +- If you get bounced to sign-in, just re-drive the email/password screens (SSO often re-completes quickly). +- Don't leave the flow parked while doing long setup on the side — provision the account/emulator first, + then run the web segment in one go. +- **Adding a SECOND/THIRD account in the SAME tenant on the same device? Don't nuke Chrome — drop just the + web session.** When you tap the app's **Open browser** for the next account, Chrome may re-authenticate the + **previous** account from its cached web session, so the proof-up wizard shows the wrong identity. Escalate + in this order and stop at the first that works: + 1. **Try it as-is first.** The handoff carries a `login_hint` for the new UPN, so eSTS often lands on the + right account with no intervention. Only act if the page actually shows the wrong identity. + 2. **Switch identity on the page** — use the account picker's **"Use another account" / "Sign in with a + different account"**, or the avatar in the top-right → **Sign out**. Cheapest by far: a couple of taps, + no state lost. + 3. **Delete cookies only** — Chrome **⋮ → Settings → Privacy and security → Delete browsing data** → + tick **Cookies and site data** (time range *All time*) → **Delete data**. This drops every web session + while **keeping** the Chrome profile/FRE state. + 4. **Last resort: `pm clear com.android.chrome`** (`appcontrol.ps1 clear -Package com.android.chrome`). + ⚠️ This wipes the whole profile, so you pay the **entire First Run Experience again** (Use without an + account / **Stay signed out** → **No thanks**, plus profile sign-in prompts) — several extra screens and + a slower, more fragile run. Use it only when the lighter options genuinely fail. + +## Chrome autofill / passkey overlay steals input + +On eSTS **WebView** email/password fields, Chrome's autofill / **passkey** overlay can intercept a bulk +`input text` — the value lands in the wrong field or is dropped, and eSTS shows "Enter a valid email". + +**Fix it at the source: turn the autofill service OFF for the whole device.** The overlay is driven by the +system autofill provider (on a Google-services device that's +`com.google.android.gms/...AutofillService`), so disabling it removes the interception entirely instead of +fighting it per-keystroke: +```powershell +adb -s shell settings get secure autofill_service # inspect (e.g. com.google.android.gms/...) +adb -s shell settings put secure autofill_service null # disable — do this once at case start +``` +This is **device-persistent** and safe to leave on a test device; restore with +`settings put secure autofill_service com.google.android.gms/com.google.android.gms.autofill.service.AutofillService` +if a case genuinely needs autofill. Do it in Phase 2/3 device prep alongside +[animations-off](run-speed.md#device-prep-turn-off-animations-and-autofill). + +With autofill disabled, **bulk typing works** — which matters because `-CharByChar` costs one adb round-trip +per character (a 20-char password ≈ 20 × ~120 ms ≈ 2.4 s vs ~0.15 s bulk, and it's where dropped/reordered +characters come from). So: +```powershell +# 1. preferred — autofill disabled at case start, then bulk: +./scripts/deviceui.ps1 input-text -Text $upn -Clear -Serial +./scripts/deviceui.ps1 input-text -SecretRef labpw -Clear -Serial +# 2. fallback only if a value still doesn't land (OEM autofill, or the setting was re-enabled): +./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial +``` +`-Clear` empties the field, `-CharByChar` types one character at a time (defeats a still-present overlay), +`-SecretRef`/`-Secret` keeps the value out of the transcript. + +**Dismissing the prompts that still appear.** Chrome's own password manager is separate from the autofill +service and can't be disabled from the shell (`/data/data/com.android.chrome/` isn't shell-writable), so a +**"Save password?" InfoBar** may still show after sign-in: + +> ⚠️ **Never press BACK (or `key ESCAPE` at the page level) to dismiss "Save password?"** — it navigates the +> *page* backwards, e.g. from "Keep your account secure" back to the password page, silently undoing a step. + +Dismiss it on the InfoBar itself, in this order: tap **"No thanks"** → tap **"Never"** → tap the close (X) +button `com.android.chrome:id/infobar_close_button`. If none is present, **just proceed** — the InfoBar +auto-dismisses after a few seconds and page interactions (e.g. tapping "Next") still work underneath it. +A "use passkey" *bottom sheet* is different and can be dismissed with `key ESCAPE` before typing. + +**Verify by whether the page advances**, not by reading the field's `text` — a WebView often doesn't reflect +typed content back in the accessibility tree (the password field `i0118` is an exception and does show a +length). + +## Chrome First Run Experience swallows the auth page (blank WebView) + +**Symptom:** on a **fresh Chrome profile** — right after `pm clear com.android.chrome`, or the first-ever +Chrome launch on a new emulator — an auth handoff (an app's "Sign in" Custom Tab, or a direct +`login.microsoftonline.com` URL) shows a **blank page** and never returns to the app. It looks like a +broken WebView; it is **not**. + +**Cause:** Chrome's **First Run Experience** (the "Make Chrome your own / Sign in to get your bookmarks…" +promo + a sync/notifications prompt) intercepts the very first navigation on a fresh profile and **swallows +the auth URL**, so the `msauth://`/app-link handoff never fires. This repeatedly ABORTed emulator AAD +registration cases until the root cause was found — past the FRE, the same emulator renders the real MS +sign-in form (email field `i0116`) normally. + +**Fix — dismiss the FRE before any auth handoff** (at case start, and again after ANY Chrome clear): +```powershell +adb -s shell am start -a android.intent.action.VIEW -d 'https://login.microsoftonline.com' com.android.chrome +./scripts/deviceui.ps1 tap-text -Text "Use without an account" -Serial # main FRE button +./scripts/deviceui.ps1 tap-text -Text "No thanks" -Serial # sync/turn-on promo, if shown +# also dismiss any "Not now"/"Got it"/notification prompt; Chrome is ready when mCurrentFocus=ChromeTabbedActivity +``` +Then re-run the flow — the sign-in page loads. **If a test step clears the browser cache, re-dismiss the FRE +afterward** or the next auth page is blank again. This is most common on emulators (fresh profile) but the +same promo can appear on a freshly-provisioned physical device. + +**The Chrome identity prompt is a *separate* thing and survives a data clear.** "Sign in as … / Continue as … +/ **Stay signed out**" and the "**Chrome notifications**" dialog come from Chrome's *account-consistency* +system reading the **Android device accounts** — not from cookies — so `pm clear` does **not** prevent them, +and on a device with a personal Google account signed in they reappear every launch. Dismiss with the first +match of: `Stay signed out` → `No thanks` (also the answer to the *Chrome notifications* dialog) → +`Continue without` → `dismiss` → `Turn off` → the negative button +`com.android.chrome:id/negative_button`. On a clean CI device with no Google account this is a no-op. + +## FLAG_SECURE black screenshots + +eSTS login pages, some broker screens, and **Microsoft Authenticator** set `FLAG_SECURE`, so `screencap` +returns an all-black image and some nodes are hidden. Don't rely on screenshots to verify these: +- Verify state with `uiautomator dump` + XML parse instead (the account list, a specific `resource-id`, + the focused activity via `deviceui.ps1 current-app`). Save the **XML** as your evidence artifact. +- Chrome pages are **not** FLAG_SECURE and screenshot fine — capture those normally. +- Full detail: [ui-interaction.md](ui-interaction.md#the-flag_secure-gotcha). + +## Screenshot corruption via redirection + +`adb ... screencap -p > file.png` through PowerShell **corrupts the PNG** (newline translation on the +binary stream). Always capture on-device then pull — which is exactly what `deviceui.ps1 screenshot` does: +```powershell +adb shell screencap -p /sdcard/_sc.png ; adb pull /sdcard/_sc.png # never `screencap -p > file` +``` + +## Single-use pairing / setup links + +> Driving the **AAD account-add proof-up** end-to-end (where this pairing hyperlink comes from)? See the +> full same-device flow in +> [authenticator-app.md](authenticator-app.md#aad-workschool-account-add-with-proof-up-number-match--same-device). +> The pairing link is reached via Authenticator's own **Open browser** button — don't hand-navigate to +> `aka.ms/mfasetup`. + +MFA-setup "pair your account to the app" deep-links (e.g. from `aka.ms/mfasetup`) and QR codes are often +**single-use** — a second attempt shows "code already used" or a stale QR. If a pairing step fails on a +retry, **re-generate** the link/QR from the setup page rather than reusing the old one, or provision a +fresh user (`labapi.ps1 create-user`). + +**The same-device "Pair your account…click this link" hyperlink is consumed by the FIRST tap — never +double-tap it.** Tapping *"Pair your account to the app by clicking this link"* launches Authenticator in the +**background** and hands it the single-use token. **Confirmation arrives as a near-instant *toast*, and the +Chrome page content deliberately does NOT change** — it stays on "Now pair Authenticator with your account". +You will usually **not** catch that toast (it lives ~2–3.5 s and **never** appears in `uiautomator dump` — see +[ui-interaction.md → Transient toasts](ui-interaction.md#transient-toasts--you-will-not-find-them-in-a-dump)), +and **that is expected — an unchanged page is not a failed tap.** Do **not** tap it again: a second tap +re-opens the now-spent link, Authenticator shows **"QR code already used — You've already used this QR code to +add an account,"** and you can end up with a spurious extra registration for the account. + +**Rule:** tap the link **exactly once**, then verify by the **downstream** signal instead of the toast — check +that Authenticator came to the **foreground** (`deviceui.ps1 current-app`) / a push arrived / the number-match +screen is up. Continue in Authenticator (notification shade → *Approve sign-in?*, or open the app, or +pull-to-refresh); never re-tap in Chrome. If you genuinely need a fresh token (it timed out), regenerate it via +the browser wizard's **Back → Next** rather than re-tapping the old link. + +**What a second tap actually looks like (verified) — and why it is *confirmation*, not failure.** A re-tap +produces, in order: **"Unable to add the account"** (Cancel), then a fallback **Add account** screen with a +**pre-filled Code + URL and a FINISH button**, and tapping FINISH yields **"QR code already used."** If you see +that chain, the **first** tap already succeeded — go look at the account list rather than retrying. + +**⚠️ Correction — a duplicate row is NOT evidence of a double-tap.** Earlier guidance here blamed the extra +account row on tapping the link twice. That is **wrong**: a **single** tap reproducibly leaves **two rows for +the same UPN** (one white, one gray), and the gray one clears itself once setup finishes. See +[authenticator-app.md → Duplicate account rows](authenticator-app.md#duplicate-account-rows-white-vs-gray). + +**"Set up Passwordless sign-in requests" can fail with "Passkey — Unknown error. Please try again later."** +**Root cause found: you opened the GRAY duplicate row instead of the WHITE one.** The gray row is not the +official record, and PSI registration against it reproducibly errors. **Fix: go back to the home list, pick the +**white** row (verify by pixel — white `RGB(255,255,255)` vs gray `RGB(189,189,189)`), and run PSI there** — +this was verified to succeed on the same account that had just failed from the gray row. Full recipe: +[authenticator-app.md → Duplicate account rows](authenticator-app.md#duplicate-account-rows-white-vs-gray) +and [→ Enabling Passwordless sign-in](authenticator-app.md#enable-passwordless-sign-in-psi). Only after a +clean run **from the white row** should you record a real **FAIL**. + +**Finding a link that isn't in the accessibility tree.** The pairing hyperlink often lives **inside a +WebView** and does **not** appear as a tappable node in `uiautomator dump`, so `tap-text` can't find it. +It's rendered in the usual link **blue**, though, so detect it visually and tap by pixel: +1. `deviceui.ps1 screenshot` the Chrome page (Chrome is **not** FLAG_SECURE, so it captures fine). +2. Scan the PNG for link-blue pixels (Python + PIL) and tap the centroid of the topmost blue run. A blue-ish + test that works well: `b > 120 and (b - r) > 50 and (b - g) > 30 and r < 120`. +```powershell +python -c "from PIL import Image; im=Image.open(r'').convert('RGB'); w,h=im.size; px=im.load(); \ +ys=[(x,y) for y in range(0,h) for x in range(0,w) if (lambda r,g,b: b>120 and b-r>50 and b-g>30 and r<120)(*px[x,y])]; \ +print(min(ys,key=lambda p:p[1]) if ys else 'none')" +# then: deviceui.ps1 tap -X -Y (coords are in device pixels) +``` +Remember screenshot coords are **device pixels**; if you scaled the image, scale the tap back up. Once you +know the link's rough y-band on a given page you can re-tap it directly without re-scanning. + +## Stale account state between runs + +A lab account that already completed first-time MFA setup will **skip** the very registration step you +want to test, making the run look like it "passed" without exercising anything. Reset the state: +```powershell +./scripts/labapi.ps1 reset -Upn $upn -Operation mfa # clear MFA registration +./scripts/labapi.ps1 create-user -UserType GlobalMFA # or just get a brand-new temp user +``` +See [lab-api.md](lab-api.md). + +**Clean state comes from a fresh install, at the START of a case — not teardown at the end.** `pm clear` +does **not** remove work-account entries from AccountManager or an existing broker/WPJ registration; only +**uninstalling** the app does. So each case's clean slate = **uninstall → reinstall** the app-under-test (and +any broker it uses) as the first step. Do **not** try to delete accounts/registrations after a case — leave +them in place for the next run's uninstall (or a human) to clear. Combined with a **fresh temp account per +case**, this keeps runs independent without brittle post-run cleanup. Device-clock or policy state you changed +mid-run should still be restored so it doesn't leak into the next case. + +## Fresh temp user not sign-in-able yet (ESTS propagation lag) + +A brand-new temp user from `CreateTempUserID4SLab2` (or any just-created lab account) can take **several +minutes to become consistently sign-in-able**. During that window the sign-in page shows *"This username may +be incorrect. Make sure you typed it correctly."* and the error often **flaps** — it clears for one attempt, +then returns — because the account exists in Graph (you got an `objectId`) but ESTS replication across +front-end nodes lags behind. Observed lag has exceeded **14 minutes**. This is **not** a typo or a real +product failure, so per [Password rejected at sign-in](#password-rejected-at-sign-in--dont-reset-it) do **not** +reset or mutate anything. Options, in order: + +1. **Reuse an already-propagated temp user** you created earlier in the same run (they live ~60 min). For flows + where the account is incidental — e.g. a *browser* SSO test where only the MSAL APK folder differs between + test points — reusing one propagated user across both points is legitimate and does not weaken the + assertion. Note the reuse in the report. +2. **Pre-warm**: create the temp user **early** (at the start of the case, before installing/clean-stating apps) + so replication finishes while you do other setup. Better still, create the *next* case's user before you need it. +3. **Poll politely**: retry sign-in every ~60–90 s (re-type the UPN each time) rather than hammering in a tight + loop; tight retries just re-hit the same stale node. + +**Freshness gate — poll ≤ 3 min, then recreate or reuse a < 30-min-old user.** Don't wait indefinitely on a +stuck new account. If a freshly created temp user is not **consistently** sign-in-able within **3 minutes** of +polite polling, stop waiting on it and either **create another** temp user, **or reuse a previously created temp +user that is still very fresh — under ~30 minutes old** (comfortably inside its ~60-min TTL, and old enough to +have finished replicating; one that already signed in once is the safest reuse). Record the substitution — +which user you abandoned and which you used instead — in the report. This is a time cap on option 1–3 above, +not a replacement for them. + +If a case genuinely needs a *distinct fresh* account and none is sign-in-able even after recreating, record the +lag as a **PASS-with-note** (if you completed via reuse) or **BLOCKED** with the exact on-screen error and the +`objectId` as evidence — not a password reset. See [lab-api.md](lab-api.md). + +## Password rejected at sign-in — don't reset it + +When eSTS shows *"Your account or password is incorrect"*, *"Your password has expired"*, or *"That Microsoft +account doesn't exist"*, the reflex to "fix" it by resetting the password is almost always **wrong**. Resetting +mutates a shared lab account and can mask a real product bug. Treat a rejected password as a *value/identity* +problem first: + +1. **Re-fetch the value** — the shared tenant password may have rotated in Key Vault since you cached it: + `./scripts/labapi.ps1 fetch-password -TestTenant -IntoSecret `, then re-type with + `deviceui.ps1 input-text -SecretRef -Secret`. +2. **Check the identity** — confirm the UPN and tenant match exactly what the test case named (an easy slip is + using an `@msidlab4` account with the `id4slab2` password, or vice-versa). +3. **Check for a lockout** — a `Locked_…` prefix or repeated failures may mean a prior run locked the account; + provision a **fresh temp user** instead of hammering it. +4. **Re-type carefully** — special characters can drop in `-CharByChar` mode; verify the field length matches. + +Only run `labapi.ps1 reset -Operation password` when the **test case itself** exercises a password-change / +expiry flow. **Never** reset a **shared durable account** (a durable, pre-created account, not a temp +`Locked_…` user) — other cases reuse it. If the value is confirmed correct and sign-in still fails, mark the +run **BLOCKED** with the exact on-screen error rather than changing the account. See [lab-api.md](lab-api.md). + +## Doing it yourself in System Settings + +Some test steps have **no adb command** — advance the device clock, delete a user certificate, toggle a +system setting, switch language, add/remove an account. That does **not** make them blockers: a human tester +just opens **Settings** and taps through, and so should you. Drive the Settings app with `deviceui.ps1` the +same way as any other screen (`dump` → `tap-text`/`tap-desc` → `input-text`) **before** you ever mark a step +BLOCKED. + +Open Settings (or jump straight to a sub-screen via its intent action): +```powershell +adb -s shell am start -a android.settings.SETTINGS # top-level Settings +adb -s shell am start -a android.settings.DATE_SETTINGS # Date & time +adb -s shell am start -a android.settings.SECURITY_SETTINGS # Security (credential/cert mgmt is under here) +adb -s shell am start -a android.settings.LOCALE_SETTINGS # Language +# then: deviceui.ps1 dump / tap-text / tap-desc to drive the screen +``` + +**Worked example — advance the clock >1h** (a common token-expiry step). Moving the *device wall clock* needs +**no root** (only `adb shell date` does): +1. `am start -a android.settings.DATE_SETTINGS` (Samsung path: Settings → **General management → Date and time**). +2. `dump` to confirm state, then `tap-text "Automatic date and time"` to turn it **off**. +3. `tap-text "Set date"` / `tap-text "Set time"`, set a value >1h ahead, confirm with **Done/OK**. +4. Do the app action you were timing, then restore automatic time afterward if later steps need real time. + +**When it *is* still a blocker.** Only fall back to BLOCKED when the surface is genuinely undrivable: +- A **secure/native system dialog** uiautomator can't read or act on — Samsung **Knox** certificate install, + the **keyguard** credential prompt, a **biometric** sensor, a PIN pad on a FLAG_SECURE screen. +- An action that truly needs **root** — e.g. expiring an app's *internal* monotonic cached-token timer + (`SystemClock.elapsedRealtime`), which the wall clock doesn't affect; that belongs on a rooted device or + emulator, not a retail phone. (Advancing the *wall clock* above is **not** this case.) +- A step that would leave **partial/destructive state** on a device you don't own (e.g. a real MDM + enrollment) with no clean rollback — pause and ask. + +Capture the exact screen (XML dump / screenshot) as evidence and mark just that step BLOCKED, per +[Genuine blockers](#genuine-blockers-stop-and-ask). + +## Genuine blockers (stop and ask) + +These can't be produced by the AI — report them and ask the user (see SKILL "When to ask the user"): +- **Real push-notification MFA** approval on a *separate* physical device with no TOTP seed. +- **SMS / phone-call OTP** to a real number. +- **Hardware** FIDO2 key, NFC, real camera/QR the emulator can't provide. +- **CAPTCHA** / "prove you're human". +- A **fingerprint on a physical device** that the scenario insists must run on that hardware. +- A credential/tenant/CA policy the AI can't provision (try `labapi.ps1 disable-policy` first if it's a + lab CA policy). + +## Quick reference table + +| Scenario / step | Automatable? | Where | How / workaround | +|---|---|---|---| +| Fingerprint / biometric prompt | ✅ emulator · ❌ physical | **Emulator** | `finger-enroll` + `finger`; on physical, human presses sensor | +| Authenticator **App Lock** re-prompt | ✅ emulator · ⚠️ physical | **Emulator** | inject biometric, or turn App Lock **off**, or use device PIN | +| Number-match MFA (same device) | ✅ if not biometric-gated | Either | read number from Chrome dump → tap tile in Authenticator | +| Number-match **expired** during a detour (airplane toggle, hunting the push) | ✅ | Either | session TTL is **~1–2 min** — collapse the detour into **one scripted call** (~21 s); if it still expires, re-trigger for a **fresh** number, don't retry the dead one | +| **Passwordless Sign-In (PSI)** setup / *Disable Passwordless Sign in* | ⚠️ **physical only** | **Physical** | needs **WPJ device registration**, which fails on the emulator (**"Device isn't registered"**, reproduced 2×) — cap WPJ at ~12 min then record the blocker; note the failure is at *device registration*, not the PSI approval | +| Verification code read from **Gmail** rejected repeatedly | ✅ | Either | the **list preview is stale** — **open the message** and confirm its timestamp post-dates your request before typing the code | +| Step's expected result **is a permission prompt** (e.g. LBAC location consent) | ✅ | Either | **don't pre-grant** that permission in device prep — it suppresses the dialog; `pm revoke` + re-trigger if you already did | +| Confirming the **browser** finished signing in while the app is foregrounded | ⚠️ | Either | Chrome suspends the page poll in the background — single-device runs often can't observe it; assert app-side state instead and note the limitation | +| AAD **first-time proof-up** adding a work account (sends you to a browser) | ✅ **same device** | Either | **Not** a "second phone" blocker — tap the app's **Open browser**, then the **"Pair your account…link"** hyperlink; [authenticator-app.md](authenticator-app.md#aad-workschool-account-add-with-proof-up-number-match--same-device) | +| Real push MFA on another *already-registered* phone | ❌ | — | **Blocker** — ask the user | +| Authenticator first-run gates (fresh install) | ✅ | Either | 4 screens → Allow · Accept · Continue · **Skip** (upper-right); [authenticator-app.md](authenticator-app.md#first-run-flow-fresh-install--home) | +| eSTS password typing | ✅ | Either | disable autofill in device prep (`settings put secure autofill_service null`), then bulk `input-text -Clear -SecretRef `; `-CharByChar` only as fallback | +| Autofill/passkey overlay | ✅ | Either | **`settings put secure autofill_service null`** at case start (durable fix); `key ESCAPE` for a passkey bottom sheet; char-by-char as fallback | +| Chrome **"Save password?"** InfoBar after sign-in | ✅ | Either | tap "No thanks"/"Never"/`infobar_close_button` — **never BACK** (it navigates the page backwards); or ignore it, it auto-dismisses | +| Chrome **"Stay signed out" / "Chrome notifications"** prompt | ✅ | Either | comes from *device Google accounts*, so `pm clear` doesn't stop it — tap `Stay signed out` / `No thanks` / `negative_button` | +| Slow transitions / taps landing mid-animation | ✅ | Either | zero the three `*_animation_scale` settings in device prep — [run-speed.md](run-speed.md#device-prep-turn-off-animations-and-autofill) | +| Need a device PIN and **no lock is set** | ✅ | Either | `locksettings set-pin ` → `verify --old` → `clear --old` at the end (fails if a lock already exists) | +| Authenticator turned **App Lock on by itself** | ✅ | Either | happens automatically once a device PIN exists — dismiss the popup, then Settings → App Lock **off**; [authenticator-app.md](authenticator-app.md#app-lock-auto-enables-when-a-device-pin-exists) | +| Push notification not showing in the app | ✅ | Either | overflow ⋮ → **Check for notifications** (`menu_check_for_notifications`) — deterministic, better than pull-to-refresh | +| Case might already be automated | ✅ | Either | check the ADO-id → test table in [existing-ui-automation.md](existing-ui-automation.md) before driving it by hand | +| Auth page blank on fresh Chrome profile (post `pm clear` / new emulator) | ✅ | Either | **Chrome First Run Experience** — dismiss "Use without an account" + "No thanks" before the auth handoff | +| Verify state on FLAG_SECURE screen | ✅ | Either | `uiautomator dump` + XML (screenshot is black) | +| Session timed out mid-flow | ✅ | Either | re-drive sign-in; set up biometric/PIN before timed segment | +| Single-use pairing link reused | ✅ | Either | tap the same-device "click this link" hyperlink **exactly once**; a 2nd tap → "Unable to add the account" → prefilled Code/URL screen → **"QR code already used"** — that chain *proves the first tap worked*, so check the account list instead of retrying | +| Pairing link inside a WebView (not in uiautomator) | ✅ | Either | screenshot → PIL blue-pixel scan → `tap-xy -X -Y` | +| UPN typed into Chrome's **address bar**, not the login field | ✅ | Either | `url_bar` is an EditText too — tap the web field by coords, confirm focus is `i0116`/`i0118` (omnibox `focused=false`) before typing | +| 2nd account, **same tenant**, browser shows the wrong (previous) account | ✅ | Either | use the page's own **"Use another account"** first — don't `pm clear` Chrome (that re-triggers the FRE) | +| **Two rows for the same UPN** (one white, one gray) | ✅ | Either | **Normal after a single tap.** Gray = not official; act on the **white** row (pixel `RGB(255,255,255)` vs `RGB(189,189,189)`); gray self-clears after setup — [authenticator-app.md](authenticator-app.md#duplicate-account-rows-white-vs-gray) | +| "Set up Passwordless" → **"Passkey — Unknown error"** | ✅ | Either | **You opened the GRAY row.** Redo PSI from the **white** row — verified to succeed on the same account; real FAIL only if it fails from the white row | +| PSI setup hits **"Scan your fingerprint"** on a physical device | ✅ | Either | tap the **"Use PIN"** fallback → OEM `lockPassword` field → `input-text -SecretRef devicepin_` — **no emulator needed** | +| Browser wizard dead-ends: *"We're sorry, we ran into a problem… choose Next to try again"* | ✅ | Either | the **app-side add may already have succeeded** — check the account list; the wizard isn't required for PSI | +| Modal *"Your sign-in information may have changed"* steals focus | ✅ | Either | it may name a **different/expired** account — read the UPN, tap **Cancel** (Continue re-auths the wrong account) | +| `screencap` shows a stale/other app on a **foldable** | ✅ | Either | don't pass `-d 0`; use plain `screencap -p` and cross-check against `mCurrentFocus`/dump `package=` | +| Emulator won't stay up (BT-HAL / system_server loop) | ⚠️ host-dependent | — | try boot recipe; if unstable, biometric step is a **blocker** on this host | +| Wrong-ABI APK on emulator (arm64 on x86_64) | ✅ | Emulator | install the **universal** APK; keep arm64 APK for physical | +| Stale MFA already registered | ✅ | Either | `labapi.ps1 reset -Operation mfa` or new temp user | +| Fresh temp user "username may be incorrect" (just created) | ✅ | Either | ESTS propagation lag (can be >14 min) — **don't** reset; poll ≤3 min then **recreate** or **reuse a <30-min-old** temp user; pre-warm early | +| System-settings step (advance clock, delete cert, toggle) | ✅ drive Settings UI · ❌ only if secure/root | Either | `am start -a android.settings.*` → `dump`/`tap-text`; block only on Knox/keyguard/root | +| Clean slate for a new case | ✅ | Either | **uninstall+reinstall** app-under-test (`pm clear` keeps work accounts/registration); don't tear down after | +| SMS / phone / hardware-key / CAPTCHA | ❌ | — | **Blocker** — ask the user | +| Multiple devices / same model attached | ✅ | Either | unique adb serial each; always pass `-Serial`; `adb devices -l` to pick; `secrets.ps1 set-device-pin` to store per-device PIN | +| Unlock the lock screen with a PIN | ✅ | Either | `secrets.ps1 set-device-pin` once, then `unlock -Serial ` (auto-uses saved PIN) — verifies + **stops after 3** tries | +| "Password incorrect / expired" at sign-in | ✅ | Either | **don't** `reset -Operation password`; re-`fetch-password` (may have rotated), re-check UPN/tenant, use a fresh temp user; reset **only** if the case says so; never reset shared durable accounts | diff --git a/.github/skills/android-e2e-tester/references/emulator-performance.md b/.github/skills/android-e2e-tester/references/emulator-performance.md new file mode 100644 index 00000000..1583c347 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/emulator-performance.md @@ -0,0 +1,127 @@ +# Emulator Performance & Android Studio Integration + +Table of contents: +- [Why the emulator is slow (and the #1 fix)](#why-the-emulator-is-slow-and-the-1-fix) +- [Diagnose your host in one command](#diagnose-your-host-in-one-command) +- [Fast-path: use a physical device](#fast-path-use-a-physical-device) +- [Making the emulator as fast as it can be](#making-the-emulator-as-fast-as-it-can-be) +- [Slow Play Store / network downloads](#slow-play-store--network-downloads) +- [Android Studio: Device Manager & Running Devices](#android-studio-device-manager--running-devices) + +## Why the emulator is slow (and the #1 fix) + +The dominant factor is **GPU rendering mode**: + +- **With a real GPU** the emulator uses **host (hardware) rendering** — UI, `screencap`, and `uiautomator dump` are fast. +- **Without a usable GPU** — typically a **Cloud PC / VM / RDP session** (e.g. a `CPC-*` Windows 365 box, which reports only a "Microsoft Hyper-V Video" / "Remote Display" adapter) — the emulator falls back to **SwiftShader software rendering** (the CPU draws every frame). Everything graphical is slow, and there is **no flag that makes a GPU appear**; `-gpu host` can't help when the host has no GPU. + +CPU virtualization (WHPX/HAXM/AEHD) is separate and usually still works on a VM (nested virtualization), so the CPU isn't the bottleneck — the **GPU is**. + +**#1 fix: run the scenario on a physical device instead of the emulator.** The skill's device pool includes +connected real devices, and on a GPU-less host `emulator.ps1 ensure` / `devicelease.ps1 acquire` will +**auto-prefer a connected physical device** for you. A phone renders on its own GPU and downloads over real +Wi-Fi, so flows that crawl on a software-rendered emulator (installing Company Portal, driving WebViews) +run in seconds. + +## Diagnose your host in one command + +```powershell +./scripts/emulator.ps1 resolve-sdk +``` +It prints the host GPU/perf profile, e.g. on a Cloud PC: +``` +host GPU: NONE -> emulator uses SLOW software (SwiftShader) rendering +host type: VM (Virtual Machine), RDP session, Cloud PC +emu defaults: -gpu swiftshader_indirect -cores 6 -memory 6144 +TIP: use a physical device for a fast run (emulator.ps1 ensure -PreferPhysical). +``` +If `host GPU` says NONE, expect a slow emulator and prefer a real device. + +## Fast-path: use a physical device + +1. Connect the device over USB (enable **USB debugging**) or Wi-Fi (`adb connect :5555`), and accept the + RSA prompt. Confirm it's in the pool: `./scripts/emulator.ps1 pool`. +2. Acquire/target it — on a GPU-less host it's automatic, or force it explicitly: + ```powershell + ./scripts/devicelease.ps1 acquire -Owner $AgentId -PreferPhysical -Wait # lease a real device + # or, without leasing: + ./scripts/emulator.ps1 ensure -PreferPhysical -Wait + ``` +3. Drive everything against that `$serial` exactly as with an emulator. + +**Caveats for real devices:** brokered flows with a **production** broker (Company Portal / Authenticator) +won't release tokens to a **debug-signed** calling app (caller-trust) — that's a signing/environment limit, +not a defect (see [troubleshooting.md](troubleshooting.md#signing--redirect-uri-mismatch-aadsts50011)). +Biometric prompts need a **real enrolled fingerprint** (the emulator-only `emu finger` simulation doesn't +apply) — see [ui-interaction.md](ui-interaction.md). Never print credentials; type them on the device. + +## Making the emulator as fast as it can be + +When you must use the emulator (no device available), these help — but won't beat a GPU-less ceiling: + +- **More cores/RAM.** `emulator.ps1` now auto-sizes `-cores`/`-memory` from the host (e.g. 6 cores / 6144 MB + on a big Cloud PC, up from a stock AVD's 4/1536). Override with `-Cores` / `-Memory`. +- **Keep the warm snapshot.** Don't pass `-ColdBoot` unless a snapshot is corrupt — a warm boot from + snapshot is much faster than a cold boot. +- **Explicit GPU mode.** The script auto-picks `host` when a GPU exists, else `swiftshader_indirect`. + Force with `-Gpu host` on a machine that has a GPU but defaulted to software. +- **Headless where possible.** `-NoWindow` skips the Qt UI (uiautomator/screencap still work); saves some + overhead, especially over RDP. +- **Use x86_64 images on x86 hosts.** ARM images on an x86 host run under slow translation. (On a physical + ARM phone, native arm64 is fine — build the matching ABI.) +- **Install the matching APK ABI.** On an `x86_64` emulator, prefer a **universal** APK + (`app-production-universal-release-signed.apk`). A large **`arm64-v8a`-only** APK can pass the ABI check + (the image lists `x86_64,arm64-v8a`) yet **crash install-time dexopt** — a Watchdog kill / `Broken pipe` / + `Can't find service: package`. Keep the arm64 APK for **physical** devices. Details in + [troubleshooting.md](troubleshooting.md#install-failures). +- **Know when to abandon the emulator.** On some GPU-less hosts the emulator boots but stays **unstable** — + the Bluetooth HAL crash-loops (`hci_backend_aidl.cc:40 initializationComplete`) and repeatedly restarts + `system_server`, so a heavy app can't be driven. `-feature -Bluetooth` may not fix it. Once you've tried the + [cold-boot recipe](troubleshooting.md#emulator-wont-start-or-boot) and it still won't settle, **switch to a + physical device** rather than burning time — accept that a biometric-gated step then becomes a blocker on + that host (see [common-blockers.md](common-blockers.md#decision-emulator-vs-physical-device)). +- **Raise timeouts, don't fight slowness.** On a slow host, prefer longer `wait-text` / boot / install + timeouts and poll for completion (e.g. `is-installed` in a loop) instead of fixed sleeps, so a slow-but- + correct step isn't misread as a failure. + +## Slow Play Store / network downloads + +Large downloads (e.g. Company Portal ~30 MB) can take **minutes** on an emulator that finishes in **seconds** +on a phone. Causes and mitigations: + +- The emulator routes through a **user-mode NAT** with its own DNS; throughput is limited and unrelated to + your real link speed. `-netspeed full -netdelay none` (already set) removes artificial throttling but can't + make the NAT fast. +- Software rendering also slows the Play Store UI itself. +- **Mitigation:** poll for install completion by package presence rather than a fixed wait; mind any + **time-boxed** feature step (a slow download can outrun a TTL — re-trigger with the app already installed). + For download-heavy flows, **use a physical device** — this is the clearest win. + +## Android Studio: Device Manager & Running Devices + +Two different things, and the skill already lines up with the first: + +- **Device Manager (the AVD list).** The skill creates AVDs in the **default AVD home** (`~/.android/avd`, + unless you set `ANDROID_AVD_HOME`) — the **same** place Android Studio reads. So **AVDs the skill creates + already appear in Android Studio's Device Manager**, and you can start/stop/wipe/delete them there. (Verify + with `./scripts/emulator.ps1 list` vs. Device Manager — same names.) + +- **Running Devices (the embedded mirror window).** This tool window mirrors an emulator that Android Studio + **launched into it**. The skill starts the emulator as a **standalone process** (its own window / gRPC + endpoint), so it shows up in `adb devices` and Device Manager but **not automatically inside the Running + Devices window**. Two ways to get parity: + + 1. **Start from Studio, let the skill reuse it (recommended).** In Studio: *Settings → Tools → Emulator → + "Launch in the Running Devices tool window"* (enable), then start the AVD from **Device Manager** — it + opens inside **Running Devices**. Now run the skill **without** `-Avd`/`-Serial`; `ensure` **prefers an + already-running emulator**, so it drives the exact device you see mirrored in Studio. (Confirm the skill + picked it: its `SERIAL=` matches `adb devices`.) + 2. **Mirror the skill's emulator.** With that same setting enabled, recent Android Studio can also mirror an + externally-started emulator in Running Devices once it detects it (it may prompt). If it doesn't appear, + use option 1 — it's the reliable path. + + Either way the skill and Studio share one emulator, one AVD list, and one adb — no duplication. + +> Net: if you're on a Cloud PC/VM (no GPU), the fastest, least-friction setup is a **connected physical +> device** (the skill auto-prefers it). If you want the emulator visible in Studio's **Running Devices**, +> start it from Studio and let the skill reuse it. diff --git a/.github/skills/android-e2e-tester/references/existing-ui-automation.md b/.github/skills/android-e2e-tester/references/existing-ui-automation.md new file mode 100644 index 00000000..fb374262 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/existing-ui-automation.md @@ -0,0 +1,151 @@ +# The existing Authenticator UIAutomator suite — check it before driving a case by hand + +**There is already a compiled, unattended UIAutomator test suite for Microsoft Authenticator, and it is +annotated with the same ADO Test Case ids this skill runs manually.** Before you drive an Authenticator +case screen-by-screen, check the table below: if an automated test exists, running it (or at minimum reading +it) is faster, more repeatable, and tells you the exact selectors and ordering the app really needs. + +This skill and that suite are **complementary, not competing** — see +[Which one to use](#which-one-to-use). + +- [Where it lives](#where-it-lives) +- [ADO test case → automated test](#ado-test-case--automated-test) +- [How to run it](#how-to-run-it) +- [How it differs from this skill](#how-it-differs-from-this-skill) +- [Which one to use](#which-one-to-use) +- [What we borrowed from it](#what-we-borrowed-from-it) + +## Where it lives + +| | | +|---|---| +| Repo | `msazure` / project `One` / **`AD-MFA-phonefactor-phoneApp-android`** (ADO) | +| Path | `/PhoneFactor/uiautomator-tests` | +| Module type | `com.android.library` — a **standalone** test module; it does **not** bundle the app, it installs APKs onto the device | +| Tests | `…/src/androidTest/java/com/azure/authenticator/standalone/tests/` | +| How-to doc | IdentityWiki → `Teams/DevEx-Android/How-Tos/How-to-run-Authenticator-UI-Automation-tests-locally.md` | + +It is **not** part of `android-complete`, so it is not checked out by `git droidSetup` — read it via the ADO +API or clone that repo separately. + +## ADO test case → automated test + +Confirmed by the `Test Case ` annotation in each test class: + +| ADO case | Automated test | Flow | +|---|---|---| +| 1579381 | `MfaViaSignInTest`, `MfaViaSignInUpgradeTest` | Entra MFA registration via sign-in | +| 1579382 | `MfaViaLinkTest`, `MfaViaLinkUpgradeTest` | Entra MFA registration via link | +| 1579386 | `MfaRichContextTest` | MFA rich context | +| 1579390 | `PendingAuthForegroundTest` | Pending auth in foreground | +| 1579395 | `NgcViaSignInTest`, `NgcViaSignInUpgradeTest` | NGC (passwordless) via sign-in | +| 1579396 | `NgcViaMfaUpgradeTest` | NGC upgrade from MFA | +| 1579397 | `NgcCheckForAuthTest` | NGC check-for-auth | +| 1579400 | `NgcRichContextTest` | NGC rich context | +| 1579401 | `LbacInteractiveGpsTest` | LBAC, interactive GPS | +| 1579402 | `LbacBackgroundGpsTest` | LBAC, background GPS | +| 1579405 | `AccountMgmtPagesTest` | Account-management pages | +| 1579419 | `Add3rdPartyViaUrlTest` | Add 3rd-party (TOTP) account via URL | +| 1579422 | `SharedDeviceRegistrationTest` | Shared-device registration | +| 1579428 | `AppLockManualTest` | App Lock, manual enable | +| 1579429 | `AppLockAutoEnabledTest` | App Lock, auto-enabled | +| 1588699 | `DeviceRegistrationWithCpBrokerTest` | Device registration with Company Portal broker | +| 2547138 | `NgmsSanityCheckTest` | NGMS sanity | +| 2916347 | `PasskeyInAppRegistrationTest` | Passkey in-app registration | +| 2916524 | `PasskeyDeregisterTest` | Passkey deregistration | +| 3094596 | `PasskeyInAppAllPreConfiguredTest` | Passkey, all pre-configured | +| 3094640 | `PasskeySkipScreenLockTest` | Passkey, skip screen lock | +| 3094641 | `PsiSkipDeviceRegistrationTest` | PSI, skip device registration | +| 3094649 | `PasskeyFromL2Test` | Passkey from L2 | +| 3261599 | `PsiPushNotificationTest` | PSI push notification | + +> **Re-verify this table rather than trusting it.** It's a snapshot; grep the module for +> `Test Case` to regenerate. Presence in this table also doesn't guarantee the test currently passes. + +**Coverage is largely disjoint from what this skill runs.** In the 20-case Authenticator batch this skill +ran, only **three** cases had an automated counterpart — 1579397, 1579401, 1579402 — and *all three* were +cases the manual run failed to complete (one ABORTED, two PARTIAL), while none of the five clean manual +passes had one. That is the strongest argument for using both: the automated suite already owns several of +the flows that are hardest to drive interactively. + +## How to run it + +Per the wiki how-to (read it for the current details — this is the shape, not a substitute): + +1. **Prereqs:** a connected device (the suite targets **Android 36 / Pixel 9**), the APKs pushed to + `/sdcard/`, **no pre-existing screen lock** (the PIN rule sets its own and fails if one exists), and the + lab cert at `/data/local/tmp/LabAuth.pfx` with `labSecret` in your **global** `~/.gradle/gradle.properties`. +2. **APK names it expects:** `authenticator-rc.apk` (default under test), `authenticator-old.apk` (upgrade + tests), `authenticator-ngms-rc.apk`, `company-portal.apk`, `teams.apk`. +3. Run a single test from Android Studio, or via Gradle's `connectedAndroidTest` with a class filter. + +Note the module runs under **AndroidX Test Orchestrator** with `clearPackageData=true` (fresh state per +test) and `animationsDisabled = true`. + +**Its default JUnit rule chain** (worth knowing even if you never run it — it's a good checklist of what +"clean state" means for Authenticator): + +``` +Timeout (50 min) + → RetryRule (2 retries → up to 3 attempts) + → CopyApkRule # /sdcard/*.apk → /data/local/tmp/ (pm install CANNOT install from /sdcard) + → CleanBrokerAppsRule # uninstall Teams + Company Portal so Authenticator wins broker election + → FreshInstallRule # uninstall+install; also `wm size reset` / `wm density reset` + → DevicePinRule # locksettings set-pin, verify, clear afterwards + → GrantPermissionRule(POST_NOTIFICATIONS) +``` + +## How it differs from this skill + +| | UIAutomator suite | This skill (`android-e2e-tester`) | +|---|---|---| +| **What it is** | Compiled Kotlin JUnit4 + UIAutomator instrumentation | AI agent driving `adb` + `uiautomator dump` from PowerShell | +| **Test definition** | Hard-coded in a `.kt` test class | Read at run time from the **ADO Test Case** work item | +| **Adding a case** | Write + compile + review + merge Kotlin | Point the skill at a new ADO case id — no code | +| **Handles an unexpected screen** | Fails (the selector isn't there) | Reasons about the dump and adapts | +| **Non-determinism** | `RetryRule` — brute-force re-run | Diagnosis + targeted recovery | +| **Device** | Fixed target (Android 36 / Pixel 9), no pre-existing lock | Any leased emulator or physical device in the pool | +| **State reset** | JUnit rules + Test Orchestrator `clearPackageData` | Phase 3 uninstall→reinstall, leases, `pm clear` | +| **Accounts** | LAB API via a cert-authenticated Java client (`labSecret` + `LabAuth.pfx`) | LAB API via `labapi.ps1` (EasyAuth/WAM SSO) — **same lab, same ID4SLAB2 temp users, same usertypes** | +| **Output** | JUnit XML / `build/reports/androidTests/` | HTML + Markdown ADO test report (`report.ps1`) + suite summary | +| **Runs unattended in CI** | Yes (that's the point) | No — it's an agent-driven interactive harness | +| **Cost per run** | Cheap once written | Model tokens + wall-clock per run | +| **Cost to cover a new case** | High (engineering work) | Low (an ADO case + a prompt) | + +**The real connection point: they share our own libraries.** The suite depends on +`com.microsoft.identity:uiautomationutilities` (`com.microsoft.identity.client.ui.automation.utils.UiAutomatorUtils`) +and `com.microsoft.identity:lab-api-utilities` (`com.microsoft.identity.labapi.utilities.*`) — both published +out of **`microsoft-authentication-library-common-for-android`**, i.e. the `common/` repo this project owns. +So the Authenticator team's automation is built on our automation library and our lab plumbing; a selector or +lab helper improved in `common` benefits both sides. + +## Which one to use + +- **Case has an automated test (table above)** → prefer the automated test for regression/repeatability. + Use this skill when you need it run *right now* on a device you have, against a specific ECS/Local build, + or when the automated test is red and you need to see what the screen actually does. +- **Case is not automated** → this skill. That's most of the Broker suite and most of the Authenticator + cases outside the table. +- **Case is automated but keeps failing** → drive it with this skill to get the screen-by-screen evidence, + then feed the fix back into the Kotlin test. This is the highest-value pairing: the agent is good at + *diagnosing* a flaky selector, the compiled test is good at *never regressing* once fixed. +- **New flow that will be run repeatedly** → run it here first to establish the steps and the selectors, + then hand those to the Authenticator team to encode as a Kotlin test. + +## What we borrowed from it + +Already folded into this skill: + +| Borrowed | Where it landed | +|---|---| +| `settings put secure autofill_service null` — kill the overlay at the OS level instead of typing char-by-char | [common-blockers.md](common-blockers.md#chrome-autofill--passkey-overlay-steals-input), [run-speed.md](run-speed.md#device-prep-turn-off-animations-and-autofill) | +| `animationsDisabled` → the three `*_animation_scale 0` settings | [run-speed.md](run-speed.md#device-prep-turn-off-animations-and-autofill) | +| Canonical Authenticator resource IDs (FRX, account list, overflow, number-match) | [authenticator-app.md](authenticator-app.md#first-run-flow-fresh-install--home) | +| App Lock auto-enables when a device PIN exists → disable it right after the account is added | [authenticator-app.md](authenticator-app.md#app-lock-auto-enables-when-a-device-pin-exists) | +| `menu_check_for_notifications` as a deterministic alternative to pull-to-refresh | [authenticator-app.md](authenticator-app.md#aad-workschool-account-add-with-proof-up-number-match--same-device) | +| Heads-up push covers Chrome's URL bar for ~5 s → wait ~10 s before reading the number | [authenticator-app.md](authenticator-app.md#aad-workschool-account-add-with-proof-up-number-match--same-device) | +| Never press BACK on Chrome's "Save password?" InfoBar; use `infobar_close_button` | [common-blockers.md](common-blockers.md#chrome-autofill--passkey-overlay-steals-input) | +| Chrome identity prompt ("Stay signed out" / "Chrome notifications") survives `pm clear` | [common-blockers.md](common-blockers.md#chrome-first-run-experience-swallows-the-auth-page-blank-webview) | +| `locksettings set-pin` / `verify --old` / `clear --old` to provision a PIN without asking the user | [common-blockers.md](common-blockers.md#steps-that-need-a-fingerprint--biometric--app-lock) | +| `pm install` cannot install from `/sdcard/` — copy to `/data/local/tmp/` first | [troubleshooting.md](troubleshooting.md) | +| Uninstall Teams + Company Portal so Authenticator wins broker election | [app-and-module-map.md](app-and-module-map.md) | diff --git a/.github/skills/android-e2e-tester/references/lab-api.md b/.github/skills/android-e2e-tester/references/lab-api.md new file mode 100644 index 00000000..84fa6815 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/lab-api.md @@ -0,0 +1,228 @@ +# LAB API — Provisioning and Managing Test Accounts + +The **MSID LAB user-manager API** (`https://labusermanagerapi.azurewebsites.net`) creates and manages +lab test accounts for E2E runs. Drive it with `scripts/labapi.ps1`. The canonical, always-current list of +endpoints/parameters is the LAB **URL generator** web app: +. + +Table of contents: +- [Authentication (why a normal token fails)](#authentication-why-a-normal-token-fails) +- [Endpoints](#endpoints) +- [Account policy (prefer fresh ID4SLAB2 temp users)](#account-policy-prefer-fresh-id4slab2-temp-users) +- [`labapi.ps1` usage](#labapips1-usage) +- [Response shape (create-user)](#response-shape-create-user) +- [When to use which endpoint](#when-to-use-which-endpoint) +- [Entitlements](#entitlements) + +## Authentication (why a normal token fails) + +The API sits behind **Azure App Service Authentication (EasyAuth)**. A service token from +`az account get-access-token` is rejected with `consent_required` — EasyAuth wants an **interactive** +Entra session, not a raw bearer token. The LAB URL-generator page works because your browser is already +signed in; the endpoint validates against that existing session. + +`labapi.ps1` reproduces that: it opens the endpoint in **headless Microsoft Edge** +(`--headless=new --dump-dom`), which reuses your Entra **WAM SSO** session, then parses the JSON the +endpoint returns. It caches an isolated **per-process** Edge profile under `%TEMP%\labapi_edge_p` so the +first call may do a silent WAM handshake and later calls in the same run are fast, while two concurrent runs +never share one profile. If a call comes back needing sign-in, run it once +in a visible browser to seed the session: +```powershell +./scripts/labapi.ps1 open -Url "https://labusermanagerapi.azurewebsites.net/api/CreateTempUserID4SLab2?usertype=GlobalMFA" +``` + +> **Edge `--dump-dom` regression (Edge 150+).** Newer headless Edge can return **0 bytes** for +> `--dump-dom` (and even `--version` prints nothing), which silently broke provisioning. `labapi.ps1` +> now **auto-falls back to the DevTools protocol (CDP)**: it relaunches Edge with +> `--remote-debugging-port`, finds the real `http(s)` page target (skipping `edge://` dialogs), opens a +> `System.Net.WebSockets.ClientWebSocket`, and reads `document.body.innerText` via `Runtime.evaluate`. +> This needs no Node/npm and works on Windows PowerShell 5.1+ and PowerShell 7. You don't do anything — +> if `--dump-dom` is empty the script switches automatically. If you ever need to debug it, `-Raw` prints +> whatever raw text the (dump-dom **or** CDP) path captured. + +> This is a **harness auth workaround**, not a code path under test. It only works on a machine where the +> operator is interactively signed in to Edge with an entitled account. + +## Endpoints + +Base URL: `https://labusermanagerapi.azurewebsites.net/api/` + +| Endpoint | Params | What it does | +|---|---|---| +| `CreateTempUserID4SLab2` | `usertype` | Creates a **temp cloud user** in the ID4SLab2 lab. **Auto-deletes after ~60 min.** Returns UPN + metadata. | +| `ResetID4SLab2` | `upn`, `operation` | Resets `mfa` or `password` for a user. **Password reset = temp users only.** | +| `EnablePolicyID4SLab2` | `upn`, `policy` | Enables a CA/special policy for a locked user. | +| `DisablePolicyID4SLab2` | `upn`, `policy` | Disables a CA/special policy for a locked user. | +| `DeleteDeviceID4SLab2` | `upn`, `deviceid` | Removes a device registration from Entra ID. | +| `List of Test Accounts` | `team` | KeyVault **deep-link** to the pre-created test-account JSON secret (team: `Android`, `JS`, `iOS`, `OneAuth`). Opens in the Azure Portal — use `labapi.ps1 open`. | +| `Fetch Password for Tenant` | `testTenant` | KeyVault **deep-link** to a tenant's password secret (`ID4SLAB2`, `ID4SLAB1`, `ARLMSIDLAB1`, `MNCMSIDLAB1`, `MSIDLAB4`, `MSIDLAB3`, `MSIDLAB8`). Opens in the Azure Portal. | + +**`usertype` values:** `Basic`, `GlobalMFA`, `MAMCA`, `MDMCA`, `MFAONSPO`, `MFAONEXO`, `FIDOBasic`, +`FIDOMDM`, `AuthappLBAC`, `AuthappRichContext`. + +**`policy` values:** `GlobalMFA`, `MAMCA`, `MDMCA`, `MFAONSPO`, `MFAONEXO`, `AuthappLBAC`, +`AuthappRichContext`. + +**`operation` values:** `mfa`, `password`. + +The two KeyVault items return **portal deep-links to a secret**, not a JSON API — they need an interactive +browser (`labapi.ps1 open`), and reading the secret needs the DevKV entitlement below. + +## Account policy (prefer fresh ID4SLAB2 temp users) + +**Default to a brand-new ID4SLAB2 temp user for every case** — even when a test case names a specific +**MSIDLAB4** (or other legacy-lab) account. MSIDLAB4 is being deprecated, its durable accounts carry state +from prior runs, and reusing one account across cases is exactly what produces the "already registered, step +skipped" false pass. `create-user` (→ `CreateTempUserID4SLab2`) hands you a clean, isolated account that +self-deletes in ~60 min, so there is nothing to tear down. + +Map the **capability** the case wanted onto an ID4SLAB2 `usertype` rather than reusing the named account: + +| Case asks for… | Use instead | +|---|---| +| MSIDLAB4 MAM-CA account | `create-user -UserType MAMCA` | +| MSIDLAB4 MDM-CA account | `create-user -UserType MDMCA` | +| A plain/basic Entra user (e.g. old `…?usertype=Basic` endpoint) | `create-user -UserType Basic` | +| Global-MFA user | `create-user -UserType GlobalMFA` | +| FIDO / passkey user | `create-user -UserType FIDOBasic` / `FIDOMDM` | + +Provision it **just before** the run (the ~60-min TTL), and pair it with a fresh app install for a truly +independent case. Only fall back to a specific named durable account when the case genuinely needs *that +identity's* pre-provisioned data (a seeded mailbox, a long-lived group membership) a temp user can't have — +then read its password with `fetch-password`, and **don't** mutate or tear it down. + +**Freshness gate — poll ≤ 3 min, then recreate or reuse a < 30-min-old user.** A just-created temp user can lag +ESTS replication and show *"This username may be incorrect"* at sign-in. Don't wait indefinitely: if the new +user isn't **consistently** sign-in-able within **3 minutes** of polite polling, either **create another** temp +user, **or reuse a previously created temp user that is still under ~30 minutes old** (inside its 60-min TTL and +already propagated — one that already signed in once is safest). Note the swap in the report; never "fix" the lag +with a password reset. See +[common-blockers.md → Fresh temp user not sign-in-able yet](common-blockers.md#fresh-temp-user-not-sign-in-able-yet-ests-propagation-lag). + +## `labapi.ps1` usage + +```powershell +# Create a fresh temp user of a given type (prints UPN=... on success): +./scripts/labapi.ps1 create-user -UserType GlobalMFA + +# Clear a stale MFA registration so a first-time-setup flow can be re-run cleanly: +./scripts/labapi.ps1 reset -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Operation mfa + +# Reset the password of a temp user — ONLY when the test case explicitly asks for a password reset/change flow. +# A rejected password at sign-in is NOT a reason to do this (see "When to use which endpoint" below). +./scripts/labapi.ps1 reset -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Operation password + +# Temporarily disable a CA policy that blocks a segment you're not testing, then re-enable it after: +./scripts/labapi.ps1 disable-policy -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Policy GlobalMFA +./scripts/labapi.ps1 enable-policy -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Policy GlobalMFA + +# Remove a stale device registration at the START of a run if a leftover Entra device would interfere: +./scripts/labapi.ps1 delete-device -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -DeviceId + +# Open a KeyVault deep-link (test-account list / tenant password) in a visible browser: +./scripts/labapi.ps1 open -Url "https://labusermanagerapi.azurewebsites.net/api/WebApp" + +# Fetch a tenant's shared password straight from Key Vault into the local DPAPI store (no paste, no browser): +./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw +# ...then type it on-device without it ever appearing in chat: +./scripts/deviceui.ps1 input-text -SecretRef labpw -Secret + +# Debug: dump the raw DOM Edge returned instead of parsing JSON: +./scripts/labapi.ps1 create-user -UserType Basic -Raw +``` + +Flags: `-TimeoutSec ` (Edge virtual-time budget, default 30), `-Fresh` (throwaway Edge profile instead +of the cached one). + +## Response shape (create-user) + +`CreateTempUserID4SLab2` returns JSON like: +```json +{ + "title": "User Creation Successful", + "userType": "GlobalMFA", + "upn": "Locked_xxx@ID4SLab2.onmicrosoft.com", + "passwordUri": "https://ms.portal.azure.com/#@.../Microsoft_Azure_KeyVault/Secret/https://msidlabs.vault.azure.net/secrets/ID4SLAB2", + "credentialVaultKeyName": "https://msidlabs.vault.azure.net:443/secrets/ID4SLab2", + "tenantId": "", + "tenantName": "ID4SLab2.onmicrosoft.com", + "labName": "ID4SLab2", + "authority": "https://login.microsoftonline.com/", + "objectId": "", + "userObject": { "UserPrincipalName": "Locked_xxx@ID4SLab2.onmicrosoft.com", "...": "..." } +} +``` +`labapi.ps1 create-user` echoes the full JSON and a convenience `UPN=` line. The **password** for a +temp user is the shared lab password (kept in the KeyVault the `passwordUri` points to). You do **not** need +the user to paste it: pull it from Key Vault directly with `labapi.ps1 fetch-password` (see next section), +which caches it DPAPI-encrypted and types it via `deviceui.ps1 input-text -SecretRef -Secret`. +**Never print the password into the transcript.** + +## Fetch a tenant password from Key Vault directly + +The "Fetch Password for Tenant" generator link is just an Azure Portal deep-link to a secret in the +`msidlabs` vault (e.g. `https://msidlabs.vault.azure.net/secrets/ID4SLAB2`). If you are signed into the +**Azure CLI** (`az login`) with an account that holds **`TM-MSIDLABS-DevKV`** (or equivalent vault GET +access), you can read that secret straight from the vault — no browser, no manual paste: + +```powershell +# Cache the ID4SLab2 shared password into DPAPI secret 'labpw' (prints only a masked length): +./scripts/labapi.ps1 fetch-password -TestTenant ID4SLAB2 -IntoSecret labpw +# -> "Fetched password into DPAPI secret 'labpw' (20 chars). ..." + +# Or pass the exact secret URI create-user returned in credentialVaultKeyName / passwordUri: +./scripts/labapi.ps1 fetch-password -SecretId "https://msidlabs.vault.azure.net/secrets/ID4SLAB2" -IntoSecret labpw +``` + +`-TestTenant` accepts the KeyVault secret names: `ID4SLAB2`, `ID4SLAB1`, `ARLMSIDLAB1`, `MNCMSIDLAB1`, +`MSIDLAB4`, `MSIDLAB3`, `MSIDLAB8` (override the vault with `-Vault`, default `msidlabs`). The command: + +- reads the value with `az keyvault secret show --id --query value -o tsv` (no parentheses in the + JMESPath, so it survives the `az.cmd`/cmd.exe wrapper), +- writes it **only** to the local DPAPI store (`%USERPROFILE%\.android-e2e-secrets\.sec`), the + same on-disk format `secrets.ps1` uses, and +- prints **only** a masked confirmation (`... (NN chars).`) — the plaintext never touches the host, + pipeline, logs, or git. + +Then `deviceui.ps1 input-text -SecretRef labpw -Secret` resolves it in-process and types it on the device. + +If `fetch-password` errors with a vault/auth message, it's a **setup blocker**: run `az login` with an +entitled account (see Entitlements below), or fall back to `secrets.ps1 set -Name labpw` and paste once into +the hidden prompt. This uses **your** entitled CLI session — if the token expires or you sign out, access +stops. + +## When to use which endpoint + +- **Need a clean account** → `create-user`. Remember it self-destructs in ~60 min; provision it just + before the run, not at the start of a long setup. Prefer this over any named MSIDLAB4 account (see + [Account policy](#account-policy-prefer-fresh-id4slab2-temp-users)). +- **A first-time-registration flow already registered on a prior attempt** (so the app skips the very step + you want to test) → `reset -Operation mfa` to clear MFA, or provision a brand-new user. +- **Sign-in says the password is incorrect or expired** → do **NOT** `reset -Operation password`. That message + almost always means you're typing the wrong *value* or targeting the wrong *account*, not that the account + needs changing. First re-pull the shared value with `fetch-password` (it may have rotated in Key Vault) and + double-check the UPN/tenant the case named; also check the account isn't `Locked_…` from an earlier lockout. + Only run `reset -Operation password` when the **test case itself** describes a password-change/expiry flow. + **Never** reset the password of a **shared durable account** (a durable, pre-created account, not a temp + `Locked_…` user) — other cases reuse it and a reset breaks them. If the value is confirmed correct and it + still fails, mark the run **BLOCKED** with the exact on-screen error instead of mutating the account. +- **A CA policy blocks a segment you're not testing** (e.g. you want to test token acquisition but MFA + keeps interrupting) → `disable-policy`, run the segment, then `enable-policy` to restore state. +- **A device-registration test left a stale device** and the *next* run needs a clean slate → optionally + `delete-device` **at the start of the next run**. Per the clean-state policy you do **not** tear down + registrations at the end of a case — a fresh app install + fresh account already isolates the next run; + use `delete-device` only if a lingering Entra device object would actually interfere. +- **You need the exact UPNs of the durable, pre-created accounts** (not temp) → open the **List of Test + Accounts** KeyVault link for your team; get the tenant password via `fetch-password` (CLI, no paste) or + the **Fetch Password for Tenant** deep-link. + +## Entitlements + +Request/manage at : + +- **`TM-MSIDLabs-Ext`** — required for **all** LAB APIs. Needs both **RO** and **RW**. +- **`TM-MSIDLABS-DevKV`** — required to read the **Mobile Build Vault** (`msidlabs`): the KeyVault + deep-links *and* `labapi.ps1 fetch-password` (via `az`). Needs both **RO** and **RW**. + +If a call returns a sign-in/consent page instead of data, you're either not signed into Edge with an +entitled account or you're missing one of the above — that's a **user/setup blocker**, not a defect. diff --git a/.github/skills/android-e2e-tester/references/log-signals.md b/.github/skills/android-e2e-tester/references/log-signals.md new file mode 100644 index 00000000..d6a59ee2 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/log-signals.md @@ -0,0 +1,135 @@ +# Log Signals — Judging E2E Pass/Fail from logcat + +Table of contents: +- [How auth logging works](#how-auth-logging-works) +- [Capture discipline](#capture-discipline) +- [Success signals](#success-signals) +- [Failure signals](#failure-signals) +- [Operation flow markers](#operation-flow-markers) +- [Common AADSTS / error codes](#common-aadsts--error-codes) +- [Judging success per flow](#judging-success-per-flow) +- [Correlating with eSTS](#correlating-with-ests) + +`authlogs.ps1 scan` automates most of this. Use this doc to interpret the evidence it prints and +to reason beyond the heuristic verdict. + +## How auth logging works + +MSAL, Common, Broker, and ADAL all log through the shared **`Logger`** class (not `android.util.Log` +directly). Lines carry a component tag and, for most auth operations, a **correlation_id** (a GUID) +that ties a client request to the broker and to the eSTS token service. Verbose/PII logging is off by +default; tokens and secrets are redacted. Do not expect to see token values — presence of a +`correlation_id` plus a success marker is the signal, not the token itself. + +Typical tags to filter on: `MSAL`, `Broker`, `Common`, `ADAL`, `OneAuth`, plus operation classes like +`BrokerMsalController`, `CommandDispatcher`, `SilentTokenCommand`, `InteractiveTokenCommand`. + +## Capture discipline + +1. `authlogs.ps1 clear` **immediately before** running the scenario — otherwise stale lines pollute the verdict. +2. Run the scenario. +3. `authlogs.ps1 scan -Package ` — always pass `-Package` so pass/fail signals are + scoped to auth-relevant + app lines (reduces false positives from system noise). +4. Save every run's snapshot into the run folder so failures can be diffed across iterations. + +## Success signals + +| Pattern | Meaning | +|---|---| +| `executed successfully` | A broker/command operation completed | +| `AcquireToken...success` / `Token ... acquired` | Token obtained (silent or interactive) | +| `TokenResult ... SUCCESS` | Result object reports success | +| `Retrieved ... token from cache` | Silent cache hit | +| `Saved ... token` | Token cached after acquisition | +| `PRT is already registered` / `Loading Workplace Join entry` | Healthy device/PRT state | + +## Failure signals + +| Pattern | Meaning / likely cause | +|---|---| +| `FATAL EXCEPTION`, `E AndroidRuntime` | App crash — always a FAIL; capture the stack | +| `AADSTS\d+` | eSTS-side error (see codes below) | +| `error_code=` / `errorCode:` | Broker/MSAL error surfaced to caller | +| `No PRT present` | Missing Primary Refresh Token — silent auth will fail | +| `INTERACTION_REQUIRED` | Silent failed; interactive needed (may be expected) | +| `invalid_grant` | Token/refresh token rejected | +| `BrokerCommunicationException` | Client↔broker IPC broke (bind/permission/signature) | +| `NullPointerException`, `NoSuchMethodError`, `ClassNotFoundException` | Code/wiring bug in the change under test | +| `CertPathValidatorException` | TLS/cert issue (often network/proxy) | + +## Operation flow markers + +Trace which operations ran (helps localize where a flow broke): + +| Log marker | Operation | +|---|---| +| `GetDeviceModeMsalBrokerOperation` | Check if Shared Device Mode is enabled | +| `AcquireTokenSilentMsalBrokerOperation` | Silent token acquisition | +| `AcquireTokenInteractiveMsalBrokerOperation` | Interactive auth | +| `GetCurrentAccountMsalBrokerOperation` | Fetch signed-in account | +| `SignOutFromSharedDeviceMsalBrokerOperation` | SDM sign-out | +| `RemoveAccount` / `removeAccount` | Account removal | + +A healthy interactive sign-in usually shows: command dispatched → interactive operation → eSTS round +trip (correlation_id) → token saved → `executed successfully`. + +## Common AADSTS / error codes + +| Code | Meaning | Usually means | +|---|---|---| +| `AADSTS50011` | Redirect URI mismatch | App registration / signature / redirect config wrong | +| `AADSTS65001` | Consent required | Grant consent in the flow (AI can tap Accept) | +| `AADSTS50076` / `50079` | MFA required | Interaction/MFA step needed | +| `AADSTS50126` | Invalid username/password | Wrong test credential | +| `AADSTS700016` | App not found in tenant | Wrong client id / tenant | +| `AADSTS50058` | Silent sign-in failed, no session | Expected before an interactive sign-in | +| `AADSTS530021` | App not approved-client-app (CA) | Wrong/unapproved app config — see troubleshooting | + +`AADSTS50011`/`700016` typically indicate a **real defect** in the change under test → root-cause and +hand to the fix loop. `AADSTS65001`/`50076`/`50058` are often **flow steps**, not defects. +`AADSTS530021` is an **environment/config** block (the signed-in app config isn't CA-approved) that stops +the flow *before* the step under test — switch to an approved app config; do not treat it as a defect. + +## Broker telemetry keys worth grepping + +Brokered flows emit per-request telemetry that is often the clearest pass/fail signal — grep the snapshot +for these keys (values are logged as `Key: , Value: <...>`): + +| Key | Meaning | +|---|---| +| `broker_app_used` | `true` = the request was actually serviced through the broker; `false` = it wasn't (e.g. broker ineligible / not found) | +| `request_eligible_for_broker` | `true` = the request passed broker-eligibility and was routed to the broker | +| `is_successful` | Overall request success as the SDK reports it | +| `api_status_code` | e.g. `RequiredBrokerMissing`, `UserCanceled`, a success status | +| `auth_flow` | `Broker` vs non-broker path taken | + +For a brokered scenario, `broker_app_used=true` + `request_eligible_for_broker=true` is a strong PASS +signal that the broker path executed; `broker_app_used=false` with a `RequiredBrokerMissing` status +points at broker discovery/eligibility rather than the token exchange. + +## Judging success per flow + +- **Silent token (AcquireTokenSilent):** PASS = token retrieved from cache/refresh with a success + marker and no `INTERACTION_REQUIRED`/`No PRT`. If it returns `INTERACTION_REQUIRED` when a valid + account exists, that is a regression. +- **Interactive sign-in (AcquireToken):** PASS = the flow reaches the account/consent, completes, and + logs token saved + `executed successfully`. A lingering login page or an `AADSTS` error is a FAIL. +- **Sign-out / account removal:** PASS = removal operation runs and a subsequent `GetCurrentAccount` + shows no account. +- **Crash at any point → FAIL**, regardless of other signals. + +Do not declare PASS on the absence of errors alone — require a positive success signal that matches the +scenario. Absent both → verdict is INCONCLUSIVE; investigate (wrong package filter, logging off, or the +action never executed). + +## Correlating with eSTS + +When on-device logs are ambiguous, take the `correlation_id` from the snapshot and correlate with eSTS +telemetry using the **kusto-analyst** or **incident-investigator** skill. Basic query shape: + +```kql +AllPerRequestTable +| where env_time >= ago(1d) +| where CorrelationId == "" +| project env_time, CorrelationId, Call, Result, ErrorCode +``` diff --git a/.github/skills/android-e2e-tester/references/mocking-flights-and-segments.md b/.github/skills/android-e2e-tester/references/mocking-flights-and-segments.md new file mode 100644 index 00000000..7f3425f0 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/mocking-flights-and-segments.md @@ -0,0 +1,93 @@ +# Mocking, Flights & Segment Testing — Getting an E2E Run Unblocked + +Table of contents: +- [Principle: make it testable, don't fake a pass](#principle-make-it-testable-dont-fake-a-pass) +- [Temporary code changes — the revert discipline](#temporary-code-changes--the-revert-discipline) +- [Setting feature flags / flights](#setting-feature-flags--flights) +- [Mocking unavailable data or dependencies](#mocking-unavailable-data-or-dependencies) +- [Segment testing when the flow can't run end to end](#segment-testing-when-the-flow-cant-run-end-to-end) +- [Reporting mocked / segmented runs](#reporting-mocked--segmented-runs) + +When a run is blocked by something that isn't a defect in the code under test — a feature flag that's +off, a server API that isn't deployed yet, a piece in the middle that isn't implemented — **don't stop at +"blocked" and don't fake a pass.** Get as much of the feature under real test as you can by setting the +flag, mocking the missing input, or testing the flow in segments. Temporary code changes are allowed for +this, provided they're reverted and never committed. + +## Principle: make it testable, don't fake a pass + +- Prefer the **most real** option available: real data > a faithful mock at the network boundary > + a stub at the client boundary > a hardcoded value. The closer to real, the more the test proves. +- A mock must match the **actual contract** (field names, types, status codes) you're standing in for — + a mock that doesn't match the agreed server/API shape proves nothing. +- Never let a mock silently turn a red flow green. State clearly (in the report) which parts were real + and which were mocked, and what that means for confidence. + +## Temporary code changes — the revert discipline + +Making a temp change (flip a flag default, stub a response, inject a value) is fine **only** with this +discipline: + +1. Keep the change **in the working tree only** — never `git commit`/`git push` it, and never stage it + into a feature commit. +2. Leave a searchable marker at the edit site, e.g. `// TODO: REVERT — E2E mock, do not commit`. +3. **Revert after the run** — `git restore ` (or `git stash` the change while you commit real + work, then drop it). Verify with `git status` that the tree is clean of test-only edits before you + finish. +4. If the change is in a **library** the app consumes as a dependency, remember you must re-publish + + rebuild for it to take effect — see + [troubleshooting.md → Testing a local library change](troubleshooting.md#testing-a-local-library-change-publish-to-mavenlocal). + +## Setting feature flags / flights + +Get the flag into the state the scenario needs, cheapest option first: + +1. **A runtime override the app already exposes** — a test-app UI toggle, a config/spinner, `adb shell + setprop`, or a flights provider you can set at startup. Use this if it exists; no code change needed. +2. **Flip the flag's source default (temp).** If the app never installs a flights provider so every + lookup returns the coded default (a real gap we hit — see + [troubleshooting.md → Feature flags / flights](troubleshooting.md#feature-flags--flights-not-taking-effect)), + the only lever is the **default** in the library: `MyFlight("Key", true)`. Flip it, re-publish + + rebuild, run, then **revert**. Never commit the flip; the shipped default must stay as designed. +3. **Multiple flags:** set every flag the path reads, and confirm from logs that the flag-gated branch + actually executes (don't assume the flip took — verify the code path ran). + +## Mocking unavailable data or dependencies + +When a step needs data you can't produce naturally (a server response for an API that isn't deployed, a +service you can't trigger from the device), mock it at the **highest-fidelity** point you can: + +| Situation | Mock approach | +|---|---| +| Server API / endpoint not deployed yet | Point the client at a **local mock server** returning the agreed JSON, or use the repo's existing `MockWebServer`/interceptor test infra if present. Match the real contract exactly. | +| A single response **field** the server will add later | Inject it at the client's response-parsing boundary (temp code), using the agreed field name/shape, so the downstream path runs. | +| A collaborating app/broker you can't drive | Use a **mock broker** (`:mockcp`, `:mockauthapp`, `:mockltw` — see the app-and-module map) instead of the real one. | +| Data normally fetched at runtime (account, config) | Feed it via **adb intent extras** or a test hook the app exposes, or preload cache/prefs the app reads. | +| A push/callback you can't originate | Simulate the inbound intent/broadcast with `adb shell am start`/`am broadcast` carrying the expected payload. | + +Rules: the mock must be faithful to the real contract; gate it behind the feature flag or a debug hook +where possible; mark and revert it (above); and record in the report that the segment ran against a mock. + +## Segment testing when the flow can't run end to end + +If a piece in the middle is genuinely missing and **can't** be mocked faithfully, don't declare the whole +feature blocked — **test the segments that can run**, each with its own success criterion: + +1. **Segment before the gap:** drive the flow up to the boundary and assert the correct outputs at that + boundary (the right request was made, the right state/telemetry was produced, the parked/queued item + exists). That proves everything up to the gap. +2. **Segment after the gap:** feed the boundary inputs the missing piece *would* have produced (via a + mock/fixture) and assert the rest of the flow proceeds correctly. That proves everything after the gap. +3. Together the two segments cover the feature minus the exact missing piece; name the gap precisely and + what still needs an end-to-end pass once it's available. + +This is strictly better than "blocked": you localize exactly what's untested (the gap) and prove the rest. + +## Reporting mocked / segmented runs + +In the Phase 7 report, always state: +- **What was real vs mocked** (which flag was flipped, which data/dependency was mocked, at which boundary). +- **Which segments passed** and the success signal for each (with correlation_ids where relevant). +- **The remaining gap** — the exact piece that still needs a true end-to-end pass, and what unblocks it + (server API deployed, feature implemented, real credential/policy). +- **Confirmation that all temp changes were reverted** and the tree is clean. diff --git a/.github/skills/android-e2e-tester/references/run-speed.md b/.github/skills/android-e2e-tester/references/run-speed.md new file mode 100644 index 00000000..a78e39f8 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/run-speed.md @@ -0,0 +1,270 @@ +# Run-Speed Analysis — why steps are slow and how to shorten them + +An analysis of where wall-clock time goes in a UI-driven run, using the AAD MFA sign-in run +(`20260721_184540`) as the reference, plus concrete speed-ups. This is guidance, not a hard gate — but the +default driving pattern should follow the "fast path" below. + +Table of contents: +- [Measured timeline](#measured-timeline) +- [The dominant cost: agent round-trips, not device time](#the-dominant-cost-agent-round-trips-not-device-time) + - [The overhead is FIXED per call — it is not thinking time](#the-overhead-is-fixed-per-call--it-is-not-thinking-time) +- [Where the time actually goes](#where-the-time-actually-goes) +- [Root causes](#root-causes) +- [Speed-ups (what to do differently)](#speed-ups-what-to-do-differently) +- [Fast-path recipe](#fast-path-recipe) +- [Expected savings](#expected-savings) + +## Measured timeline + +Timestamps are screenshot capture times; the gap is the time that segment took. + +| Segment | From → To | Elapsed | What happened | +|---|---|---:|---| +| App first-run | — → 01_firstrun 18:48:16 | (setup) | launch + accept privacy | +| Add account → sign-in page | 01 → 02_chrome 18:53:30 | **5m14s** | navigate menus, load eSTS WebView, type UPN | +| Enter password → post-sign-in | 02 → 03_after_signin 18:57:35 | **4m05s** | type password (char-by-char + retries), eSTS round-trips | +| MFA wizard | 03 → 04_mfa_wizard 18:58:11 | 36s | "More info required" wizard | +| Pairing | 04 → 05_pairing 18:58:54 | 43s | pair account | +| MFA challenge | 05 → 07_mfa_challenge 19:03:33 | **4m39s** | switch to browser, re-auth, number-match appears | +| Number match | 07 → 08_number_match 19:04:55 | 1m22s | read number, attempt approval (blocked by App Lock) | + +**UI total ≈ 16m40s**, plus **≈3–5m** of setup (provision account, install/verify APK, device lease). +A human does the same flow in **~3–4 minutes**. The gap is almost entirely **harness overhead**, not the +device or the network. + +## The dominant cost: agent round-trips, not device time + +This was re-measured on a 6-case Authenticator batch (`authn-rerun-tier12-20260729_121549`) and the result +overturns the intuitive explanation. **Process startup is ~nothing. The agent's own think-time is everything.** + +Benchmarked on a live emulator — the *same* 4-screen sequence, run two ways: + +| | Shell time | +|---|---:| +| 4 separate `deviceui.ps1` calls | 10.8s | +| the identical steps as one `deviceui.ps1 flow` call | 10.5s | + +**A 3% difference.** Batching saves almost nothing *in shell time*, because the cost is the adb round-trips, +which you pay either way. But in a real run those 4 calls were **4 separate tool calls** — and a tool call is +a full agent turn. + +Measured properly on the densest available sample (`tc2579657-main-20260727_094717\shots`, 108 artifacts, one +written per device interaction, so consecutive file mtimes = consecutive round-trips): + +| | Seconds between consecutive device interactions | +|---|---:| +| min | 2.3 | +| p25 | 19.9 | +| **median** | **24.4** | +| p75 | 35.8 | +| p90 | 59.6 | +| max | 120.6 | + +Against a directly measured device cost of **2.53 s** for one `dump` (3-run mean, physical device), returning +**~13 k chars ≈ 3.3 k tokens** of UI XML: + +``` +one screen driven as its own tool call ≈ 25 s median (of which ~2.5 s is the device) +``` + +**~90% of per-screen wall clock is turn overhead, not the phone.** Note "overhead", not "deliberation" — see +the next section for why that distinction decides which fix works. + +### The overhead is FIXED per call — it is not thinking time + +The obvious objection is: *if the test steps are clear, the next action is obvious, so why does it take 25 s?* +The data says the objection is right about the decision and wrong about the cause. Splitting the same run's +gaps by how hard the decision actually was: + +| Step class | n | median | mean | +|---|---:|---:|---:| +| **Easy** — deterministic nav (privacy/telemetry/upsell/home/settings) | 16 | **25.7 s** | 37.2 s | +| **Hard** — eSTS, proof-up, pairing, number-match, error screens | 22 | **27.5 s** | 32.2 s | + +**Essentially identical — 1.8 s apart on a ~26 s baseline.** Tapping *Accept* on a privacy screen, which needs +zero deliberation, costs the same as diagnosing a proof-up failure. So the time is *not* the model working out +what to do; it is the fixed cost of a turn: emitting the call, running it, and reading a multi-thousand-token +UI dump back into a growing context. + +Two consequences, and they are the whole point: + +1. **"Think faster" is not available as a lever** — there is little thinking in the 25 s to remove. +2. **"Take fewer turns" is the only lever, and it works best exactly where the steps are most obvious.** A + deterministic preamble is *all* fixed cost and *no* decisions, so collapsing its N screens into one call + removes ~(N−1) × 25 s and loses nothing. This is why speed-up #1 is the flow runner, and why it should be + pointed at the boring parts of a run, not the interesting ones. + +### What this cost the batch + +Reconstructed from each case's `progress.log` (work time = spans with gaps ≤ 10 min): + +| Case | iters | work (min) | span (min) | idle gap (min) | +|---|---:|---:|---:|---:| +| 1579397 | 2 | 58 | 94 | **36** | +| 1579401 | 1 | 26 | 26 | 0 | +| 1579411 | 2 | 65 | 155 | **90** | +| 1579416 | 1 | 11 | 11 | 0 | +| 1579417 | 2 | 67 | 100 | **33** | +| 1579425 | 1 | 39 | 39 | 0 | + +Three findings, none of which is "the 30-minute cap was ignored" (it wasn't — every *iteration* came in at +11–39 min): + +1. **The cap was per-ITERATION, not per-CASE.** A second iteration silently reset the budget, so a case could + legitimately consume ~60+ min. Fixed by adding a per-case total budget — see + [SKILL.md](../SKILL.md) Phase 1/batch section. +2. **159 minutes of pure dead time** (36 + 90 + 33) with *no work happening at all*. Cause: a `write_agent` + follow-up is only delivered once the target agent's **entire turn** completes, so a follow-up queued behind + a lane that still had 4 cases to run sat idle for **90 minutes**. **Never queue a follow-up behind a busy + lane — dispatch it as a fresh agent on a free lane.** +3. **Inside a ~28-min iteration, ~20 min was fixed setup preamble and only ~5 min was the actual test.** For + 1579397: provision → clean install → 4 first-run gates → add account → eSTS sign-in → proof-up pairing → + App Lock disable ran 12:19→12:39; the assertions the case actually cares about ran 12:42→12:47. That same + ~20-min preamble was paid **~9 times** across the batch (6 cases + 3 re-iterations) ≈ **3 hours of setup to + run ~30 minutes of assertions** — and the preamble is 100% deterministic, contains zero decisions, and is + already fully written down in [authenticator-app.md](authenticator-app.md). Every round-trip in it is pure + waste. **This is what speed-up #1 exists to remove.** + +## Where the time actually goes + +The three fat segments (5m14s, 4m05s, 4m39s) share the same shape — they're dominated by the +**observe→act loop overhead**, not by anything the app is doing: + +1. **Per-action adb round-trips.** A single "tap the button labeled X" is really: `uiautomator dump` → + `exec-out cat` the XML → parse → compute center → `input tap x y` → **fixed `Start-Sleep`** → often a + re-`dump` to confirm. That's 4–6 adb invocations and a hard sleep for **one** logical action, and each + screen has several actions. +2. **Fixed sleeps instead of polling.** Waiting a flat `Start-Sleep -Seconds 3–5` after every tap "to be + safe" is the single biggest tax. Screens that were ready in 300 ms still cost the full sleep; multiply + by dozens of actions across a run. +3. **One tool call per micro-step.** Every `powershell` tool call is a fresh process — but the process + startup itself turned out to be **negligible** (measured: 3% on a 4-step sequence). What is *not* + negligible is that each call is a separate **agent turn**, costing a ~25 s median round-trip for ~2.5 s of + device work, and that cost is **fixed regardless of how obvious the step is** (25.7 s for easy nav vs + 27.5 s for hard auth screens). Issuing steps as separate calls is the single largest tax in the whole run; + see [The overhead is FIXED per call](#the-overhead-is-fixed-per-call--it-is-not-thinking-time). +4. **Char-by-char typing.** The autofill/passkey overlay forces one-character-at-a-time input at ~55–60 ms + per character; a 20-char password + a UPN is a couple of seconds *just typing*, before any verification. +5. **Screenshot capture + pull + view.** `screencap` on-device → `adb pull` → open the PNG is ~1–2 s each; + taking one after every step to "see what happened" adds up fast (and is wasted on FLAG_SECURE screens + that come back black). +6. **Retries when input didn't land.** WebView typing that silently dropped (autofill) triggered re-typing + and re-verification — the 4m05s password segment is mostly this. +7. **Genuinely slow bits (unavoidable-ish).** eSTS WebView first paint, the switch to the browser for the + challenge, and any re-auth after a session timeout are real seconds — but they're the minority. + +## Root causes + +- **Chatty, one-action-per-call driving** — not because of process startup (measured negligible) but because + every call is a separate agent turn at a fixed ~25 s, so a 12-screen deterministic sequence costs 12 of them + (~5 min) to do ~30 s of device work. +- **Pessimistic fixed sleeps** substituting for readiness signals. +- **Verify-by-re-dump / verify-by-screenshot** after every action instead of only at decision points. +- **Overlay-forced char-by-char** typing on every field, even when bulk would have worked. +- **No reuse** of the uiautomator dump: we re-dump for the next action instead of reusing the XML we just + fetched. + +## Speed-ups (what to do differently) + +Ordered by payoff: + +0. **Device prep — two settings, once per case, that + pay off on every single step.** Run these right after the device is leased (Phase 2/3), before installing: + ```powershell + # kill the autofill/passkey overlay that forces slow char-by-char typing (see #4 below) + adb -s shell settings put secure autofill_service null + # remove animation time from every screen transition (this is what the team's UIAutomator suite does) + adb -s shell settings put global window_animation_scale 0 + adb -s shell settings put global transition_animation_scale 0 + adb -s shell settings put global animator_duration_scale 0 + ``` + Animations at `1.0` add ~200–400 ms of *unobservable-but-real* transition time to every navigation, and — + worse — they make an anchor appear **before** it is stable, which is a common source of taps landing on a + mid-flight view. At `0` the next screen is present the moment the transition is issued, so anchor polling + returns on the first poll instead of the second or third. Both settings are device-persistent, so a + dedicated test device only needs this once; re-apply after any factory reset or `pm clear` of Settings. +1. **Batch a KNOWN sequence into one `flow` call — + the single biggest lever.** Any run of screens whose order you already know (app first-run gates, an + add-account wizard, a settings path) has **no decisions in it**, so driving it one tool call per screen + burns a ~25 s agent round-trip on each of them for ~2.5 s of device work — and since that overhead is + *fixed* rather than proportional to difficulty, a no-decision sequence is the case where batching wins + most and risks least. `deviceui.ps1 flow` executes the whole sequence in **one** call: + ```powershell + ./scripts/deviceui.ps1 flow -Serial -Spec + # or inline: + ./scripts/deviceui.ps1 flow -Serial -Text '[{"tap":"Allow","then":"privacy"},{"tap":"Accept"}]' + ``` + Each step is `{ label?, wait|tap|tapDesc|tapRes|input|secretRef|key, then?, waitSec?, optional?, exact?, + clear?, charByChar?, screenshot?, sleepMs? }`. It prints a per-step trace with timings and a + `N ok, N skipped, N failed` summary, stops at the first **required** failure (exit **5**, remaining steps + marked SKIPPED), and — critically — **`"optional": true` steps that don't appear are SKIPPED, not failed**, + which is what makes it safe for real sequences where a screen only *sometimes* shows (a Save-password + infobar, an "App Lock enabled" popup, a permission dialog that was pre-granted). + Ready-made specs for the Authenticator preamble are in + [authenticator-app.md](authenticator-app.md#ready-made-flow-specs-copy-paste). Turning a ~12-screen, + ~20-minute preamble into one call that finishes in **~40 s of device time** is where the hours are. + *(Corollary: this supersedes the old advice to "keep one long-lived shell" — the saving was never the shell, + it was the round-trip.)* +2. **Replace fixed sleeps with anchor polling — and fuse tap+wait.** `deviceui.ps1 wait-text -Text + ""` returns the instant the screen is ready instead of always waiting N seconds. + Better still, `tap-text "