Repository navigation
Conversation
Size Report
Startup median (7 runs, lower is better):
|
|
The code looks sound at 2a72c7e, but the evidence is not there yet. The repaired first pair, enabled run 38248643131 and disabled run 38248646748, has to finish. Each run's artifacts need test/artifacts/readiness-experiment/{android,ios}-summary.json with pathProven=true, all six corpus cases matched, requestFiles non-empty and validForDecision=true. The enabled arm should show readiness.observations > 0 and the disabled arm should show 0. We also need one readiness_wait=normal dispatch of replays-manual.yml that still reaches 'pnpm gate replay-android --retries 2' through the new single-shell Android script. CI is pending. Both Smoke Tests jobs are queued or in progress, and nothing has failed yet. iOS Smoke samples replay-ios, which runs through the replay-port adapter changed here. The only change on that route is extra data on a debug-level diagnostic, so a failure there would be unlikely to come from this diff. I did not trace which diagnostics scope gets the dispatch-path interaction_target_readiness lines in a real daemon run. If they land in the outer test session, eventBelongsToCases drops them. The enabled arm's pathProven check would only catch that if gate events are also missing. I also did not run the focused tests or the mutation claims, and I did not check that 'test' accepts --serial and --debug through pnpm gate replay-android. A hosted run will show that. Not blocking, and you can take or leave these. The collector rebuilds per-action readiness from request-log diagnostics, with about 120 lines of session, path, step and timestamp matching, while the response already carries it via interaction-touch-response.ts:236 and replay_action_stop is written in session-replay-action-runtime.ts. The baselineSource refusals in readiness-experiment.test.ts are asserted with a bare .toThrow(), so any throw passes. Please assert the message for both the changed and the duplicated declaration. evidenceValidity at readiness-experiment.ts:303 requires suitePassed with --retries 0, so a disabled run that fails because the wait is absent gets dropped from the vote. Per-arm pass and fail counts would keep that signal visible. The single-shell change in android-manual.sh now carries BOOT_FINISHED_AT and ANDROID_SERIAL across lines, which is a second fix that no normal-mode dispatch exercises yet. The normal dispatch above would cover it. Could replay_action_stop gain an optional readiness field, taken from the gate wait or the dispatch response? The collector could then count that field and read JUnit and wall time. That would drop the diagnostic tagging, the request-log scan and the timestamp attribution, and the experiment would no longer need --debug. Could the A/B also run from the draft branch through workflow_dispatch on that ref, with results posted to #3077 and the harness left unmerged or deleted in the keep/drop PR? Then main would not carry a one-off 360-line script, a vitest include and a source-rewriting step. Before merge, the repaired first pair must finish with validForDecision=true on both platforms, and only then should the other four pairs launch. Please also decide whether per-action readiness moves onto replay_action_stop instead of the tagged debug diagnostic. |
|
The corpus repair at 8a342c2 is in, and the replay corpus now matches what the readiness wait measures. The readiness-wait decision evidence is still missing, and the new tests no longer load on current main. The earlier request for hosted evidence still stands. At 8a342c2 we need five paired hosted runs, one with readiness_wait=enabled and one with readiness_wait=disabled for each pair. Each run should upload test/artifacts/readiness-experiment/{android,ios}-summary.json with pathProven=true, all six corpus cases matched, requestFiles non-empty and validForDecision=true. The enabled arm must show readiness.observations > 0 and the disabled arm must show 0. Please also dispatch replays-manual.yml once with readiness_wait=normal. Its log should show Coverage fails in scripts/replays/android-manual.test.ts and scripts/replays/readiness-experiment.test.ts with "Cannot find module ../../src/tests/test-utils/tmp-dir.ts". Fallow reports the same two files as unresolved imports. Both are this PR's own new tests. Main moved that helper to @agent-device/test-kit in #3483 (packages/test-kit/src/tmp-dir.ts), after this branch's merge-base. The other Fallow findings (the test-kit clone groups and the session-test-harness unused exports) come from main, not from this diff. I did not re-run the collector tests or the mutants here, because the tests fail to load on merged CI, so the mutant claim is unverified. I also could not check without a live run that the Android Settings search control exposes text="Search Settings" and not content-desc on the hosted emulator image. Next, please rebase onto main and import mkdtempForTest from @agent-device/test-kit in both scripts/replays/*.test.ts, so Coverage and Fallow can load them. Then collect the five paired hosted runs at the repaired corpus. Those two steps are what must happen before merge. Not blocking, and fine to take or leave: in test/integration/replays/ios/simulator/03-scroll-discovery.ad:5, This review covers 8a342c2. The newer head 30953a7 adds an Android search diagnostic and keeps the old tmp-dir import, so the points above still apply; I will look at the new diagnostic separately. |
30953a7 to
26c0c31
Compare
|
Android sanity-pair validation at Both uploaded artifacts verify the exact head, all six matching corpus hashes, and the expected source variants: default 2000 ms / SHA
Both fail All four acting click requests completed in each mode. Enabled has four actual readiness events, each |
|
iOS readiness evidence checkpoint for head
|
|
Update at
I am retaining the tagged debug diagnostics for this temporary experiment. Successful dispatch responses expose readiness only for repeated polls, and the adapter only attaches its gate evidence to failures; response-only tracing would lose first-poll path proof and successful gate waits. The revised issue explicitly requires The disabled failure/pass signal remains visible in per-arm JUnit, raw traces and assessment comments; incomplete suites cannot provide comparable wall-time p50. A matched acting-target failure can still be reviewed as a counterfactual to an actual successful rescue. No rescue is established in this pair. The baseline refusal test-message strengthening is accepted and will accompany the next source revision. |
|
Normal Android workflow validation passed at The job ran the single-shell helper in The log explicitly reached |
|
Thanks for the update. At 26c0c31 the code looks fine and all 19 checks pass, but the readiness evidence #3077 asks for is still missing. The normal-mode single-shell dispatch ask is now fixed: run 38289828169 shows it works. What remains is the paired data. #3077 asks for five paired hosted runs, one with readiness_wait=enabled and one with readiness_wait=disabled per pair. Each arm must upload test/artifacts/readiness-experiment/{android,ios}-summary.json with pathProven=true, all six corpus hashes matched, requestFiles non-empty and validForDecision=true. The enabled arm must show readiness.observations > 0 and the disabled arm 0. Today Android fails 04-text-input-keyboard step 4 ( |
26c0c31 to
b8ca87b
Compare
|
Thanks for the update. The code looks fine at b8ca87b, and all 37 checks pass. The hosted readiness runs are manual dispatches, so they are not part of PR CI. I could not see a merge conflict. The earlier gaps are now covered by the new harness, but the decision evidence for #3077 is still missing.
I did not open the hosted runs or artifacts, so pair eligibility is taken from the PR body. I did not run the new shell and native tests or the mutants. I also could not judge whether the preparation step on c7289c2 is corpus-neutral between the two arms. Please confirm that when you post the pairs. Before merge, the five eligible pairs from one harness commit and the aggregate audit must be posted. |
|
1 participant
Summary
Prepare the hosted readiness-wait decision for #3077. Fourteen files add source/mode/corpus manifests, action attribution, one Android shell invocation, semantic Settings replay checks, and opt-in native diagnostics. Diagnostic-only runs cannot vote. All six cases and existing assertions remain; gross diff is 989 lines.
Intentionally draft. Omit the temporary harness from the eventual decision change.
Validation
Tested PR head:
b8ca87bb1fbefca1384e3da76800365fde4b8c46.pnpm check:affected --runpassed; 231 related test files passed.b94c49e9dpreserves measured source/corpus. Disabled passes; original enabled SDK provisioning fails. Its sole replacement passes bounded preparation, then fails semantically: no new eligible pair. Separate shipping stack #3554, #3555, #3560 is ready for review with final-head CI and changed-route live proof. Later unchanged iOS full-fixture geometry assertion fails separately. Private8527d147passed review/gates; fresh enabled/disabled sanity pair passes authored query-state outcomes after independent scope adjudication; loading/No Results limitations retained. The completed audit and one scoped infrastructure retry establish 4/5 eligible pairs per platform; pair4 navigation failures remain excluded. No final decision. Diagnostic branch unmerged; PR head unchanged.