Readiness wait: hosted evaluation and keep/drop decision #3077

Activity

1. thymikee

   added

   needs-triageNew or unreviewed issueNew or unreviewed issue

   on Sep 30, 2026
2. thymikee commented on Oct 9, 2026

   

   thymikee

   on Oct 9, 2026

   MemberAuthor

   More actions

   The soak plan can't run as written: replays-nightly.yml only runs the parser fuzz job. Device replays were moved to manual dispatch in replays-manual.yml by #1781 because they failed on every scheduled run. So the green nightlies from 10/01 to 10/08 say nothing about the readiness wait, and the keep/drop call has no evidence yet.

   Revised plan: dispatch replays-manual.yml five times with --debug on the iOS and Android simulator suites, post the engaged-wait counts and replay wall-time p50 against a baseline run with the wait disabled, then make the keep/drop call. Five runs with no engagement means drop; a measurable rescue rate with no p50 regression above 10% means keep. Don't move device replays back onto a schedule for this.
3. thymikee

   added

   ready-for-agent

   and removed

   needs-triageNew or unreviewed issueNew or unreviewed issue

   on Oct 9, 2026
4. thymikee commented on Oct 12, 2026

   

   thymikee

   on Oct 12, 2026

   Last edited by thymikee

   MemberAuthor

   More actions

   Shipping readiness correction after checking current GitHub review state: CI is green on all four existing shipping heads, including Coverage/Integration, but #3555 and #3560 are not merge-ready. The remaining findings are launch-only arguments reaching the intent resolver (#3555) and zero-gesture iOS scroll coverage (#3560). Both have prepared, unpublished fixes; the intent regression fails on the old production code and passes after the fix (54 focused tests), while the strengthened scroll fixture still needs exact-head live proof. A delegated coordinator owns cold review, serialized gates, stack propagation, publication of these existing-PR fixes and current-head CI/device verification. Previous head evidence stays attributed to its original SHA.

   #3554 has both inline threads fixed/resolved and is ready. #3562's deadline thread has now been resolved after source verification and the maintainer's explanation: a positive stopped observation remains valid even if the probe completes late. It depends on the corrected #3560 stack base. #3490 remains a temporary experiment draft that must not merge. #3493 and #3494 are already merged. No merge instruction or retirement publication budget exception has been received.

   Final iOS route evidence is now recorded at unchanged production head f22ddb765: diagnostic run 38365631102 checks out the exact retirement source. Public prepare and selector drag pass; all six ordinary cases pass first attempt, 47 acting requests, zero healing, copied corpus hashes equal parent D/f22. Two owned-child three-second samples and redacted registration/publication metadata survived. The samples belong to a successful run and do not establish the cause of earlier startup failures.

   The private diagnostic workflow had two confirmed measurement defects (observer process-query timeout, then truncated sampler reports/missing directory-owner capture), repaired on two subsequent workflow heads; all three original runs and artifact hashes remain retained. No original job rerun or production source/corpus change. Final full fixture fails later before interacting with a missing automation-press node in the app-switcher snapshot, outside the six ordinary cases. Both Android and iOS changed routes now have live evidence, but no whole-workflow-green claim. Session closes and daemon stop succeed; simulator shutdown and complete post-host lease release are unproven. The diagnostic branch is NEVERPR/NEVERMERGE. No additional uninstrumented retry or behavior workaround is justified.

   Publication is awaiting the requested one-time exception to docs/agents/pull-requests.md:33 (1,000 gross lines): removal is 2,223 gross / cumulative stacked diff 2,716. The retirement PR body is prepared with exact evidence and limitations; no removal PR is open, no retirement-head Coverage/Integration result is claimed, and #3077 remains open until landing.

   Prior validation and evidence history

   Validated DROP implementation is frozen and pushed at f22ddb765811c9558025d4f3c0b7be14472bfac7 on t3/drop-readiness-3077. Review the retirement diff. Fresh independent review is clear after removing the entire unused replay-clock construction chain and stale contracts. Exact-head pnpm check:affected --base 5d05dc21 --run passes all runnable gates: 647 related files / 4,926 tests. Feature regression proofs and 116 focused tests remain recorded below. Post-gesture, scroll, snapshot freshness and post-action observation owners are byte-identical to the parent; no temporary experiment machinery is included. Two user-doc pages and versioned scripting help now show explicit waits.

   The readiness option was in released tags v0.21.19–27, so the user-directed API removal is marked as a breaking change. Generic unknown-input filtering remains; no deprecated-option shim or fallback is added. The retirement itself is 55 files, +460/-1763 (2,223 gross), and the whole stacked diff is 2,716 gross against origin/main. This exceeds the repository’s 1,000-line PR budget; no oversized PR has been opened.

   Normal live run 38359418524 is dispatched on exact f22ddb765 for both Android and iOS. A delegated read-only audit owns source/corpus identity, changed-route reachability, authored outcomes, retained observation behavior and cleanup. This is final implementation validation, not another comparison cohort. The first retirement audit failed at provider capacity without an assessment. The replacement audit completed on the same run: Android full fixtures and six ordinary replays pass, 87 first-attempt actions, zero healing, corpus identity and retained scroll stabilization verified. iOS build/boot pass but public prepare preflight fails with COMMAND_FAILED / daemon_startup_failed before any replay; original daemon.log is empty. The same startup failure occurred on parent D, so it is a recurring finding under source/workflow investigation, not a blind-retry candidate. Exact retirement-head iOS route evidence remains missing. No jobs have been rerun. Independent source review found no justified behavior fix. Metadata was collected after retirement, so missing lock/info does not prove registration was never acquired; graceful retirement records SIGTERM, not execution of shutdown handlers. A diagnostic-only child task will collect pre-deadline process samples and state on the same runtime head in a separate private workflow branch; the retirement branch remains unchanged. Coverage/Integration on the retirement head will be needed after PR publication. #3077 remains open until the removal lands; the explicit DROP decision is settled.

   Decision: DROP using current evidence, explicitly chosen in the latest instruction. It supersedes the earlier repair/five-pair choice. Both current platforms remain 4/5 eligible, so the original five-valid-pair criterion is not met and is not represented as met. Historical failed engagements and pair4 navigation failures remain retained; no causal benefit is demonstrated. No additional experiment is authorized or needed for this explicit decision.

   Historical implementation progress (superseded by final head above)

   Removal is implemented and frozen at private decision head d7f72816c680d352fdf9eb8475f59aeedfd4563c, based on shipping #3562 5d05dc21. It removes the public/typed readiness input, timeout widening, descriptor/guarantee traits, replay default and target-gate/selector repeated polling, preserving ordinary explicit waits, post-gesture stabilization and scroll observation. Help and user docs recommend explicit waits. Current sparse/covered/missing-target failures and genuine native capture recovery remain. The final branch contains no experiment/diagnostic harness or observer attribution additions.

   Regression evidence: the new public-client transport test fails against unchanged pre-drop production code (1 failed) because the retired option is forwarded. Three replay no-default cases and first-miss target-gate case fail against that same base (4 failed / 32 skipped); all five fail for the intended behaviors. Fixed focused suite: 11 files / 116 tests pass. A stale policy-table expectation was corrected to no-poll. Exact-head affected gates and independent fresh review are underway before publication; PR publication must account for the repository’s gross diff budget. No readiness removal has merged yet.

   #3562 is ready for review at unchanged 5d05dc21eb6873d666326305a78b9ab1aff437c4. Root verified actual-head CI: sixteen selected checks pass, one skips, none pending/failing, including Coverage and Integration; fresh review has no findings. The exact-head Android live job passes full fixtures, three fixture replays and six ordinary cases (87 actions, first attempts, zero healing). Seven timed Settings relaunches complete close before launch and observable state; successful changed-route reachability is established. Raw native foreground/process admission records are absent, so admission is inferred from exact-head control flow and timing; no live exhaustion/refusal occurred. Typed refusal and no downstream launch are regression-test evidence. Source budgets/cadence unchanged. Cleanup/session finalization and emulator shutdown are observed, post-shutdown host lease inventory unobserved.

   The same manual workflow fails separately at iOS daemon startup preflight; later iOS stages are skipped. It adds no iOS semantic evidence and does not change the earlier pair4 navigation finding. No whole-workflow-green or native-persistence-cure claim. The four shipping repairs #3554/#3555/#3560/#3562 are ready, unmerged. #3077 remains open: both current comparison cohorts are 4/5 eligible, zero engagement/rescues, no qualified p50/final KEEP/DROP. The existing criterion remains in force pending the policy response; no further source/data/criterion exception is applied.

   The single allowed Android infrastructure retry is independently eligible: run 38341326113, Android attempt 2 / job 115120490158, unchanged frozen 8527d1472489dc1a3b6770e6d7be30365a333987, disabled default 0, debug, zero replay retries. All six authored cases pass, 87 dispatch pairs match independent JUnit/request records, zero healing. Native preparation takes 46,569 ms, with all 48 raw package/history pairs and mandatory recheck verified before measured begin. Wall/JUnit 162306/156756 ms; zero readiness observations, engagement or rescues. Attempt 1's interrupted pre-replay dump remains excluded and retained. Carried iOS success is its original execution, not a second run.

   Both platforms now have eligible pairs 1/2/3/5 (4/5). Android's common eligible enabled cohort has sixteen first-poll observations; iOS twelve. Neither has repeated engagement, successful waited dispatches, causal rescues or unattributed waits. Pair 4's real Android wrong-context/foreground and iOS post-click navigation failures remain excluded; all failed-arm observations remain retained. There is still no qualified five-pair p50 or final KEEP/DROP decision. No additional rerun or source change is justified by this successful infrastructure retry. #3562's separate final-head normal live audit remains pending, and the criterion question is unanswered.

   Independent Android close-fence repair is published as draft #3562, layer 4 of shipping stack #3556, at 5d05dc21eb6873d666326305a78b9ab1aff437c4. Three files, 20 production lines: foreground/process deadline exhaustion now refuses with typed COMMAND_FAILED / android-app-close-timeout, phase/package/budget details; relaunch cannot open or observe after refusal. Existing two-second budgets, cadence and stable-process checks remain. Valid regression proof: base 3 failed / 32 passed, fixed 35 passed. Fresh review, install/build, format, scoped Fallow and all runnable exact affected gates pass (355 related files / 2,379 tests). This enforces the existing restart promise, not a native persistence or later-navigation cure; no docs/corpus/experiment infrastructure changes.

   Final-head normal workflow 38344348831 is dispatched at exact 5d05dc21; delegated live audit and CI Coverage/Integration remain pending, so #3562 stays draft. Separately, one scoped Android infrastructure retry on disabled run 38341326113 / original job 115112741523 is accepted on unchanged 8527d147; a distinct delegated audit owns attempt 2 and preserves original failure. Carried iOS success is not a new execution. Neither run is a new common-source comparison cohort. The policy question is pending; the existing five-valid-pair condition remains in effect.

   Completed final-cohort audit: all 20 jobs and raw artifacts at 8527d147 are verified. Android eligible pairs 1/2/5 (3/5); iOS 1/2/3/5 (4/5). Each eligible enabled cohort has twelve first-poll observations, zero engagement, successful waited dispatches, causal rescues or unattributed waits. All retained enabled observations, including failed arms: Android nineteen and iOS fifteen first-poll, still zero engagement. Neither platform qualifies for five-pair p50 or the current DROP criterion; no final decision.

   Failure classification is independently reviewed. Android pair 3 disabled's pre-begin probe exits 255 in 118 ms with an abruptly truncated native dump and empty stderr; the host helper did not time out or apply a buffer cutoff. AOSP ADB/dumpsys behavior supports an interrupted shell transport (specific failing component unproven). One unchanged Android-job-only infrastructure retry is justified under repository policy; original attempt remains excluded and retained, and another occurrence becomes a finding. Android pair 4 enabled has actual wrong-context navigation failure. Its process-close fence runs to the two-second deadline and silently permits launch; both foreground and process fence owners have this false-success bug. An isolated fix is in progress; it does not claim to explain native persistence or all later navigation failures. iOS pair 4 enabled has a real post-General-click destination failure: readable original captures did not match, then deadline cancellation; matched disabled tap geometry/capture producer differs. No causal readiness rescue or justified iOS patch/retry is established.

   The five-valid-pair requirement versus a decision from current zero-engagement evidence is pending clarification. Until changed explicitly, the existing criterion applies. No new common-source cohort, assertion weakening or runtime budget change is made.

   Audit continuation: the first full-cohort delegated task failed with provider capacity before delivering its final assessment. Its partial snapshot is preserved: Android pairs 1/2/5 provisionally eligible, pair 3 disabled failed a native-preparation broadcasts probe before measured begin, and pair 4 enabled failed with the launcher foreground; iOS collection was incomplete in that snapshot. These are partial findings, not a completed cohort or qualified p50. A replacement delegated audit now owns the same ten existing runs and retained raw evidence. No device job is rerun, no source/corpus changes, and no keep/drop decision is applied.

   The full frozen-source cohort is dispatched on 8527d147: five matched pairs, both Android and iOS, with debug and zero retries. A new delegated read-only audit owns all ten runs, their raw evidence, eligibility, engagement/rescue counts and qualified p50. No overall decision yet.

   Pair	Enabled	Disabled
   1	38339651351	38339657843
   2	38341313293	38341317475
   3	38341321352	38341326113
   4	38341333045	38341336792
   5	38341342501	38341346760

   Sanity-pair audit completed on exact 8527d147: all four jobs pass 6/6 first attempts, zero healed steps, source/corpus/default-only variant identity and every dispatch independently match JUnit/request records. Android has 87 dispatches per arm; iOS 47. Enabled readiness: Android four first-poll / zero engaged, iOS three first-poll / zero engaged. Disabled has zero readiness observations. No successful waited dispatches, causal rescues or unattributed resolved waits.

   The primary auditor held Android disabled because its Display screenshot showed loading content instead of result rows. Root and a second independent reviewer reject that added eligibility requirement: 04-text-input-keyboard.ad is authored as text input, asserting native focus, exact Display readback, keyboard and back/final Search, all independently verified. It contains no result-row assertion. Both platforms are assessed on their authored query-state outcomes: Android disabled shows Display plus loading content; both iOS arms show focused General with No Results. These are limitations, not proof of completed indexing/results. No source/corpus/assertion or keep/drop threshold changes. The original audit hold and raw evidence remain preserved; earlier HOME/black, failed focus/value and incomplete-capture exclusions remain excluded. No rescue is inferred from the result-content difference.

   After this explicit scope adjudication, the sanity pair is eligible for the authored replay contract on both platforms. Android preparation 45454/24782 ms (41/32 raw package/history pairs, mandatory rechecks), and iOS preflight 79144/92567 ms precede measured begin. Wall/JUnit ms: Android enabled 159314/154309, disabled 104431/101459; iOS enabled 231366/218050, disabled 193069/173461. Recovered diagnostics and warnings remain included. These are one pair, not p50 or a decision. The remaining four pairs will use unchanged frozen 8527d147, debug, zero retries; no scheduling.

   Current checkpoint: the private common-source comparison candidate is frozen and pushed at 8527d1472489dc1a3b6770e6d7be30365a333987, based on shipping C 30942efb302587802dbb23e43fdc73a2f39b6b85. Fresh review caught and resolved a collector gap: omitted dispatch pairs must be detected against the independent JUnit replayed count, including non-acting commands. The regression fails before the fix; all six collector tests pass after it. Partial evidence remains recorded and ineligible. All runnable exact-head pnpm check:affected --base 30942efb302587802dbb23e43fdc73a2f39b6b85 --run gates pass, including 232 related files / 1,369 tests. Implementation precedes one final gate-registration commit. This private harness has no PR and will never be merged.

   Fresh sanity pair: enabled 38339651351 and disabled 38339657843, both listed at exact 8527d147, both Android and iOS, debug and zero retries. Disabled changes only default 2000 to 0; setup/preflight precede measurement. A delegated read-only audit owns raw admission, source/corpus identity, trace completeness, semantic outcomes, failure evidence, timing and readiness attribution. The completed audit and scoped adjudication are recorded above; the remaining pairs retain unchanged source/corpus. Historical data below retains its own attribution and is not pooled with this candidate. No overall KEEP/DROP decision is applied.

   The hosted keep/drop evaluation is still incomplete in draft PR #3490. The issue body now reflects the existing revised five-pair manual plan; parser-only nightlies and the 46 local runs do not count. No decision is established. All 37 reported PR checks completed with no failures on b8ca87bb1fbefca1384e3da76800365fde4b8c46, including Coverage and Integration Tests. This clears CI validation; the semantic hosted evidence and five-pair decision remain outstanding.

   Historical c728 checkpoint: iOS has five eligible pairs; Android remains incomplete. The one policy-allowed iOS startup retry, 38314030878 attempt 2 / job 115041623767, passed all six cases and 47 actions on original c7289c282a85f91e8e45e08be010fa2939bff75d, disabled budget 0, debug, zero suite retries. The native focused/editable/exact General query and screenshot are verified. Attempt 1's pre-replay daemon-startup failure stays excluded and visible; it is not erased or relabeled. The carried Android result retains its original timestamps and is not a second execution.

   Platform	Eligible pairs	Enabled / disabled wall p50 ms	JUnit medians ms	Engagement / rescues
   iOS	5/5	218800 / 183480 (+19.25%)	200307 / 154452	0 / 0
   Android	2/5	Unavailable	Unavailable	2 failed / 0

   The iOS enabled arms contain fifteen first-poll observations and no repeated-poll engagements. All slow successful runs and recovered errors remain in timing. This satisfies the original zero-engagement DROP criterion for iOS; no global change has been applied while Android and the failed-engagement decision remain unresolved.

   Android adjudication preserves the failed Notifications engagement: three polls, ending stalled after 4273 ms, with readable captures before cancellation and no dispatched click. The target was absent on Search. The disabled action succeeded on a correct root presentation; this is not a controlled causal regression or a rescue. First Search's focused wait separately stalled during capture/bootstrap; editor absence, provider death and IME causality remain unproven. Cases 03/05/06 individually false-pass on Search because their assertions are broad. The full failed arm remains ineligible. An identified Settings intent leaves package identity absent, causing command-scoped helper startup and skipped post-open observation; that ownership seam needs a separate measured repair if the strict experiment continues.

   The three preparation refusals have an evidenced setup-deadline cause. Two still had progressing exact BOOT delivery pending at 40 seconds. Another reached both disabled components, matching false-DONT_KILL callbacks and terminal Boot delivery at 39.985 seconds, leaving only 15 ms for its mandatory recheck. Parser, provider/image and user/UID checks were correct. Failed arms remain ineligible.

   A minimal private setup correction is now published at b94c49e9db53670c6651a95b03481797273bf8a4: comparisons allow a finite 90 seconds, diagnostics retain 40 seconds / 240 total / full 180 replay + 15 cleanup reserve. Native checks, mandatory recheck, per-probe 15-second bound, cadence and immediate first-satisfied return are unchanged. Measured runtime, corpus and workflow are byte-identical to c728. Prior valid rows keep original heads/budgets. Thirty-eight focused tests, fresh review, Fallow, repository-wide format and the exact-head affected gate passed before push.

   The setup sanity audit is complete. Original enabled 38320033165 failed SDK provisioning before replay and remains visible; GitHub rejected its scoped rerun without creating attempt 2. Its sole same-head enabled replacement 38320997495 / job 115048802275 passed bounded preparation after 47,395 ms, exercising the 40–90-second extension, then failed replay. All 51 raw admission pairs and mandatory recheck are verified; preparation is excluded from measurement. First Search's focused capture stalled before typing; Notifications failed after readable captures then cancellation; remaining reported passes are invalidated by Search or HOME screenshots. Readiness records one first-poll and one engaged/stalled observation, zero rescues. This is the second historical failed engagement; it remains recorded despite pair ineligibility. Provider death and IME causality are unproven.

   Disabled 38320036081 / job 115045667776 independently passes all six cases and 87 actions, actual Settings surfaces/movement/lifecycle and focused exact Display query. Wall/JUnit 158238/152528 ms, readiness zero. Preparation took 22412 ms with 25 verified raw pairs. Enabled wall/JUnit 151177/146126 ms are excluded. No new matched pair is eligible; Android remains 2/5. All failures, recovered captures and warnings remain retained. Supplemental iOS jobs do not replace the original five pairs.

   Further unchanged collection is stopped. Two separate small shipping repairs are in fresh review on current main: one-shell ownership for the ordinary manual Android workflow, then bounded ADB requested-intent package resolution so Settings identity reaches session ownership, launch observation and helper leases. Neither changes readiness budgets, adopts the foreground app, adds GUI warmup, or merges diagnostic infrastructure. The ordinary workflow timing repair is published as #3554 at f9260b1ee01bc0cda719662fe883f0f422affbdc, after fresh review and all runnable exact-head affected gates passed. The regression models the action’s injected serial and proves lost timing state (-90s/0s before, 10s/25s after). The runtime repair is published as draft #3555, stacked on #3554 in stack #3556, at a3aba45aabcb73296dc52a762e3d4c0eb7b645fc. Review of effective launch arguments is fixed through the existing contract/daemon/Android resolution path and a shared native intent builder. All runnable exact-head affected gates passed, including 442 related files and 3,122 tests; pre-fix regressions failed, 70 focused tests passed before the byte-identical rebase. Normal hosted Android job 115058651049 passed and its raw evidence audit is complete on that exact head: seven Settings opens retain com.android.settings and record postOpenObservation: observable; drag, all ten fixture scenarios/three nested replays, and six ordinary cases pass (83 actions, first attempts, zero healed steps). This proves source-correlated reachability of the changed ownership route. Literal native-query output, launchConfirmation, persisted Settings state and Settings-specific helper leases remain unobserved; do not claim those from the successful suite. Screenshots confirm Display query/results, Notifications detail, reopen and list movement; independent focus/caret/individual swipe rendering proof remains incomplete, so full semantic certification is held. Eight fixture visibility misses before explicit scroll discovery remain retained. Setup92s/full398s/JUnit73570ms are finite ordinary timing, not comparison rows. Session/emulator cleanup is observed; explicit daemon termination and Settings helper release are unobserved. Android artifact11780483897 SHA256 a6e157dca325bb0cb68fceec38d75796dd9dff55f53d028afc0c53163382db40.

   All reported #3555 head checks completed without failures, including Coverage and Integration. The supplemental iOS job is 5/6 with nine attempts: 03-scroll-discovery fails all three attempts after fixed scrolls overshoot Accessibility, and 01's five-second wait fails once then passes. These are preserved outside the original iOS five-pair set. No provider-death or IME-causality claim follows.

   The shipping stack is updated and published with three layers: #3554 at 22fb210659c4b0bd747a3c2aa71833e3e2a75795, #3555 at 14de349c6b83e725a613e6de5673fdc93d174a48, and new draft #3560 at 30942efb302587802dbb23e43fdc73a2f39b6b85, all in stack #3556. #3554 retains failed-full-suite durations/exit code and tests independent injected serials; its three focused cases pass, and a hardcoded-serial mutation fails two cases. Both review threads are resolved. #3555's eight runtime files are byte-identical to the a3aba hosted result above. #3560 changes only three fixture scripts: native focus/exact query guards on Android/iOS and selector-bounded iOS scrolling. All existing cases/assertions stay; no runtime readiness policy or default budget changes. These are manual-selected test material, not automatic PR regression coverage.

   Fresh reviews and all runnable exact-head affected gates pass on all three heads; runtime head14de includes 442 related files/3,122 passing tests. Earlier CI/live evidence retains its original head attribution. Final-head CI is complete without failures on all three PRs, including Coverage/Integration. Both review findings on #3554 are resolved, and all three shipping PRs are ready for review. None has been merged.

   The runtime-head Android job passes six cases first attempts, 83 actions, zero healed; seven Settings opens retain the correct package with observable capture. Wall/JUnit69457/69461ms, setup/full92/390s. The corpus-head Android job passes six cases first attempts,87actions,zerohealed; focused exactDisplay and real results verified, wall/JUnit71907/71911ms,setup/full87/377s.

   Corpus-head iOS passes the six-case corpus first attempts,47actions,zerohealed,wall/JUnit211761/211782ms. Focused exactGeneral query/caret is verified, but screenshot saysNoResults: noGeneral result-row claim. Both bounded-scroll visibility assertions pass with zero gestures because targets are already visible; no movement claim follows. Supplemental runtime-head iOS retains old03's three failed Accessibility attempts. All ordinary failures/recovery/cleanup evidence remains visible, and these runs add zero comparison votes.

   The whole corpus-head iOS job fails later at unchanged full fixture test/integration/ios-simulator-e2e/live-full-scenarios.ts:162. Independent source/image/response adjudication confirms a P2 fixture precondition: it requires automation-press in a reduced partial switcher card without establishing target-center geometry. Original16-node interactive capture and later27-node plain diagnostic are separate requests with equal shared geometry, not proof of a temporal race. The target is clipped before/in the card; healthy capture does not promise all app children. No production capture defect is proven. The owning repair must establish a deliberate fixture target within the reported viewport and retain regular/raw occlusion assertions; sleeps or arbitrary button-presence weakening are not justified. This separate full-fixture failure prevents a whole-workflow-green claim, but lies outside the six-case comparison corpus.

   Remaining #3077 work is the keep/drop decision. A new private comparison branch from repaired C is being prepared by a delegated implementation task, with the original exact read-only native preparation, debug/zero retries, source/budget/corpus manifests and action attribution. It omits all diagnostic trace machinery and will never be merged. Both platforms will be measured on the same frozen candidate: original iOS five pairs remain valid historical evidence at c728, not current-C timings, and old Android rows will not be pooled with the changed source. Root will independently review and gate the candidate before a sanity pair, then complete the remaining pairs only after eligibility is demonstrated. No criterion change or KEEP/DROP implementation is made. The local emulator had another live owner and was not taken or closed. Earlier measurements retain their original source attribution.

   PR #3490 stays unchanged at b8ca87bb1fbefca1384e3da76800365fde4b8c46 with all 37 reported checks complete and no failures. Both private branches are unmerged. No global KEEP/DROP implementation is established; the eventual change will omit temporary experiment/diagnostic infrastructure.

   Earlier rows remain excluded or held:

   • First enabled/disabled failed before Android replay because the emulator action gave each script line a separate shell.
   • Repaired enabled/disabled failed on iOS scroll overshoot and Android capture. Scoped corpus repairs preserved all six cases/assertions.
   • At 8a342c2, enabled/disabled reported 6/6 on both platforms, but the text-input outcomes were not proven. The later direct Android diagnostic proved provider availability, a focused editor, exact Display readback and cleanup; it was outside comparison timing and cannot vote.

   At 26c0c317, the strengthened enabled/disabled pair verifies source/budget/corpus identity:

   • Android: both 5/6, failing the post-click focused-editor wait. Enabled records four first-poll readiness observations; disabled zero. The wait's readable nodes were not retained, so the later HOME failure tree does not establish the original wait's surface. No acting-target failure/rescue is proven. Android assessment.
   • Enabled iOS: 6/6, including a uniquely focused editor and exact General value; query screenshot confirms it. Three first-poll observations, zero engagement/rescues.
   • Disabled iOS: 5/6; click succeeded, then its first AX capture was canceled after 4866 ms by the fixture's new five-second wait. Typed reason wait_capture_stalled; no readable observation or query steps completed. This is not a proven readiness rescue. Truncated timing cannot supply comparable p50.
   • All 19 PR checks passed. Normal Android validation also passed fixture setup/full E2E and reached ordinary replay --retries 2, all six cases on first attempts. This proves the normal shell path, not a five-pair result.

   The revised source is b8ca87bb1fbefca1384e3da76800365fde4b8c46. Build, format, independent review and exact affected gate passed. Its iOS editor/value waits now inherit the existing ten-second policy, keeping the semantic assertions. This addresses the demonstrated short fixture budget without claiming eventual capture success.

   The next Android-only diagnostic retains bounded native activity/crash/focus logs before replay, then Search exit-info and activity/window/IME state. The obsolete warm GUI probe has been removed; its earlier evidence remains available. A diagnostic-only marker forces validForDecision=false, even for a complete successful corpus. Ordinary comparisons remain unchanged when diagnostics are off.

   Native diagnostic run 38293406954 completed on exact b8ca87bb1fbefca1384e3da76800365fde4b8c46: Android 5/6; iOS deliberately skipped. Source, all six corpus hashes, debug and zero retries are verified. These findings cover the ADB helper backend only.

   This failure differs from the earlier pair: Search launched and the focused-editor assertion passed. At device time 18:55:14.076, Android killed Search provider PID 2747, its window died, and SearchActivity was removed. Exit-info records reason 15 (STATE CHANGE), subreason 0, description change com.google.android.settings.intelligence; a second provider process received the same reason at 18:58:09.522. Android documents this reason as app/component state changes, with a separate reason for package updates. Historical IME evidence shows focus returning to Settings before typing; transport acknowledgement did not prove the Display value, and exact readback failed.

   Native collection covered the transition and subsequent cases (~221 seconds, 191,700 bytes, exit 143, empty stderr); all post-measurement probes succeeded. Clock anchors allow ordering but not exact cross-clock latency. The actor/component remains unknown. Boot and screenshot overlap are hypotheses, not causal findings; no runtime timeout, fallback or warmup is justified.

   The previous caller-trace checkpoint on t3/readiness-native-actor-3077 was published at 1dc0ac3cb9e59a91f875cde50ceb8c4071674102. Caller-trace run 38300846242 completed on that exact head, enabled mode, Android diagnostics; iOS was skipped. It runs only case 04 through the public test command, zero retries, preserving all six corpus hashes and source manifests. Existing ss/pm framework tracing, before/after provider component state and device/host clock anchors are bounded and owned cleanup handles failures/interruption. The partial diagnostic is explicitly diagnosticOnly=true, validForDecision=false.

   Independent review findings were fixed: failed log collection cannot silently turn a successful diagnostic into a pass; trace/probe completeness and raw logger status are separate. The actual workflow's per-line shell invocation test is retained. Eleven focused tests passed. Disabling cleanup failed the tests, a failed logger reproduced the previous misleading success, and a broken multiline workflow failed all six shell cases; mutations were restored. The required affected gates passed at 19ae3545, 0b6c1f51 and final 1dc0ac3c (final command: pnpm check:affected --base 0b6c1f51d2fe7e56fa030b006ec46ad34a6d08a7 --run); final review has no remaining findings. The completed caller audit is summarized above; its raw caller/clock/state evidence remains available in the run artifacts.

   Draft PR #3490 remains unchanged at b8ca87bb1fbefca1384e3da76800365fde4b8c46. Its Coverage and Integration Tests are confirmed successful on that head. The temporary probe will not be merged.

   No further pairs launch until the owning failure is understood and both complete semantic corpora can be verified. The temporary harness remains unmerged and will be omitted from the eventual decision change. Five eligible pairs and the applied keep/drop decision remain outstanding.
5. thymikee

   changed the title [-]Readiness wait: nightly soak and keep-or-drop rule[/-] [+]Readiness wait: hosted evaluation and keep/drop decision[/+] on Oct 12, 2026