Skip to content

No command falls back to the demo app on its own - #1764

Open
abrichr wants to merge 8 commits into
mainfrom
claude/generality-demo-fallbacks
Open

abrichr wants to merge 8 commits into
mainfrom
claude/generality-demo-fallbacks

Conversation

@abrichr

@abrichr abrichr commented Nov 3, 2026

Copy link
Copy Markdown
Member

Part of the engine generality work (audit: #1741, findings D10, D11 and D12). OpenAdapt's demo app (MockMed) had become the default target in three engine paths. Each path now names its target or refuses before acting.

Change

  • replay (D10). With no --url and no backend.url, replay used to serve MockMed silently. That included Desktop replays whose staged config had no URL. Now it needs one of --url, backend.url or --demo-app, and refuses otherwise. --drift requires --demo-app, and --demo-app is refused together with a URL or an attached browser. api.replay(..., demo_app=True) matches. The launcher help, quickstart docs, quickstart_lifecycle.py, the e2e tests, the bug template and Desktop's frozen smoke test now pass --demo-app. When the variant router's first-screen probe has no URL, it now captures nothing, which halts as first_screen_unavailable.
  • qualify accept break-it gate (D11). The gate used to run every workflow's proposed oracle against a live MockMed: its banner text, /api/encounter, and /api/db. Now it builds the lie from the workflow itself. The store is an in-process read that did not change. The screen shows the workflow's own success text, every typed value, and whatever a screen read looks for. The fixture name is now unchanged_store.
  • Execute (D12). The reference worker used to run the MockMed tutorial driver against any admission's target_url, resetting and reading that app as if it were MockMed. Now it refuses such an admission before running anything. A synthetic admission that names a target is refused too. Admissions with no target still run on the bundled demo app.

Shown on the motivating app and another

  • MockMed: the existing gate, replay and Execute tests pass. The banner-oracle test still catches the lie.
  • Another app: new gate tests use a Frappe-Lending-shaped workflow. Its record read refutes the unchanged store, and a screen read of its own success text is fooled and refused.

Adversarial review

An independent reviewer compared old and new gate behaviour effect kind by effect kind and found the gate no weaker. They also confirmed the stub matches what RestRecordVerifier calls, so reads stay reachable and verdicts can't silently go INDETERMINATE. All of the reviewer's should-fix findings are applied in the second commit: two tests that replayed with no target, --demo-app with an attached browser, and synthetic admissions that name a target.

Tests

Replay, variants, undo, privacy, API, Execute and qualification tests: 100 passed. These ran on the host's shared CPU-slot wrapper with -n 2. CI runs the full suite.

This touches qualification and runtime entry points, and it is well under 1,000 code lines.

replay with no --url (and no backend.url) served the bundled MockMed
app; it now needs --demo-app for that, and refuses otherwise. --drift
needs --demo-app. The launcher help, quickstart docs, the quickstart
lifecycle script, e2e tests and Desktop's frozen smoke test pass it.
The variant router's first-screen probe returns nothing without a URL.

qualify accept's break-it gate ran every workflow's proposed oracle
against MockMed (its banner text, /api/encounter, /api/db). It now
builds the lie from the workflow: an in-process store that did not
change, and a screen that shows the workflow's own success text, every
typed value, and whatever a screen read looks for.

The reference Execute worker ran the MockMed tutorial driver against
any admission's target_url (resetting and reading it as MockMed). It
now refuses such an admission before anything runs.
…argets

Adversarial review of the fallback change: two tests replayed without a
target (now --demo-app); --demo-app with an attached browser was
silently ignored (now refused); a synthetic Execute admission that names
a target_url was reported as verified without running (now refused
before anything runs); the bug template shows --demo-app.
@abrichr

abrichr commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review (separate AI reviewer, given the diff only). Verdict: no blocker.

Break-it gate (qualification_oracle_gate.py): not weaker. Checked by reading the old and new code effect kind by effect kind.

  • Any effect with a screen channel puts screen channels into the oracle set. Those are actor channels whenever the workflow writes, so the shared-channel halt reason is added whatever the screen verdict is. The new banner text can only change the reason text; it can't turn a halt into a pass.
  • The old MockMed store was empty before the optimistic POST and still empty after it. The new stub returns an empty store both times, so judge_records gets the same inputs. count_new_only, field_equals, record_written (count ≥1), expected_count=0, empty exact_new_set, absent_is_null, resolve failures, api_binding effects, and writes with no effects all give the same verdicts as before.
  • The stub matches what RestRecordVerifier calls (get(url, timeout=), status_code, json(); there is no url attribute, so the origin check is skipped). The read is reachable and store_unchanged=True. Nothing silently becomes INDETERMINATE.
  • No PHI goes into the dumped verdicts. They keep only verdict, substrate and reason.

Replay: the refusal is correct.

  • Desktop and non-web backends return earlier, through _replay_desktop.
  • The worklist pool has its own --url check.
  • CDP and page-URL runs are exempt.
  • run is unaffected, and cli_self_repair already requires --url.
  • A Desktop replay whose staged config has no backend.url is now refused instead of silently running MockMed, which is intended.

Variant probe: returning None becomes a first_screen_unavailable halt, which is fail-closed.

Execute: the only producer of target_url in the repo is the new test, so no legitimate caller breaks.

Should-fix items, all fixed in 4f3212d:

  1. tests/test_privacy.py and tests/test_api.py replayed without a target. Both now pass --demo-app / demo_app=True.
  2. --demo-app with an attached browser (CDP or page URL) was silently ignored. It is now refused.
  3. A synthetic=True admission that also names a target_url was reported VERIFIED without running anything. It is now refused before anything runs, with a new test.
  4. The bug-report template showed replay bundle with no target. It now shows --demo-app.

After the fixes, the targeted tests pass: 100 passed (replay, variants, undo, privacy, API, Execute, qualification).

Desktop's frozen engine pins a released Flow, which serves the demo app
whenever no URL is given and doesn't know --demo-app.
@abrichr
abrichr enabled auto-merge (squash) November 4, 2026 00:14
# Conflicts:
#	packages/flow/CHANGELOG.md
#	packages/flow/openadapt_flow/qualification_oracle_gate.py
# Conflicts:
#	packages/flow/CHANGELOG.md
@abrichr
abrichr disabled auto-merge November 4, 2026 19:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant