Repository navigation
Engine generality audit; move the benchmark harnesses out of the wheel - #1741
Merged
Merged
Conversation
This was referenced Nov 3, 2026
Audit every place the engine (packages/flow/openadapt_flow) names a specific application, bucketed as example, data, harness in the package, or real special-casing: docs/GENERALITY_AUDIT.md. Move openadapt_flow/benchmark and openadapt_flow/instrument to packages/flow/benchmark/harness (import benchmark.harness). The wheel packages only openadapt_flow, so it no longer ships them. The benchmark command loads the harness from a source checkout and says so from a wheel; instrument already failed to import from a wheel. Reword two comments that named an app on a generic branch.
abrichr
force-pushed
the
claude/generality-audit-83a5d8
branch
from
November 3, 2026 19:04
22de5f6 to
7d884c0
Compare
This was referenced Nov 3, 2026
Member
Author
|
Independent AI review (separate reviewer) Verdict: should-fix. No blocker found. The move is clean, and the wheel no longer ships the harness. A few stale references and one doc claim need fixing. Findings
What I checked
What I could not check
|
Lint and format the moved harness in CI (benchmark/harness); the old comment said it was still linted. Exclude the new reliability-corpus path from the wheel and sdist. The benchmark command appends the checkout to sys.path instead of putting it first. The import guard allows an allowed module only a lazy (in-function) harness import. The changelog lists the move under Changed. The audit names the guard PR for the CI check and corrects two counts and a line number.
Member
Author
|
Review addressed in the latest commit:
Left as is:
|
abrichr
enabled auto-merge (squash)
November 4, 2026 00:14
abrichr
added a commit
that referenced
this pull request
Nov 4, 2026
* Engine generality audit; move the benchmark harnesses out of the wheel Audit every place the engine (packages/flow/openadapt_flow) names a specific application, bucketed as example, data, harness in the package, or real special-casing: docs/GENERALITY_AUDIT.md. Move openadapt_flow/benchmark and openadapt_flow/instrument to packages/flow/benchmark/harness (import benchmark.harness). The wheel packages only openadapt_flow, so it no longer ships them. The benchmark command loads the harness from a source checkout and says so from a wheel; instrument already failed to import from a wheel. Reword two comments that named an app on a generic branch. * Generality audit: link CONTRIBUTING without the guard PR's anchor * Engine generality audit; move the benchmark harnesses out of the wheel Audit every place the engine (packages/flow/openadapt_flow) names a specific application, bucketed as example, data, harness in the package, or real special-casing: docs/GENERALITY_AUDIT.md. Move openadapt_flow/benchmark and openadapt_flow/instrument to packages/flow/benchmark/harness (import benchmark.harness). The wheel packages only openadapt_flow, so it no longer ships them. The benchmark command loads the harness from a source checkout and says so from a wheel; instrument already failed to import from a wheel. Reword two comments that named an app on a generic branch. * Generality audit: link CONTRIBUTING without the guard PR's anchor * Move test fixtures and demo harnesses out of the openadapt-flow wheel The wheel ships only the engine (see packages/flow/docs/GENERALITY_AUDIT.md). Moved out of openadapt_flow: - integrations/harness2 -> benchmark/harness/harness2 (the harness2 extra is dropped; install [browser] beside Harness² from a checkout) - mockloan/ and demo_driver.record_disbursement_demo -> benchmark/harness/mockloan/ (recorder in demo.py) - guard_demo.py -> benchmark/harness/guard_demo.py - agent/ledger_demo.py -> packages/agent/tests/ledger_demo.py - mockmed/teach_scope.py -> packages/flow/tests/_teach_scope.py - learning/synth_stream.py: StructuralDiffInducer -> learning/induce.py; the MockMed drift stream -> benchmark/harness/synth_stream.py reward/seed.py is split: the bundle helpers (write_bundle, calibrate_bundle, self_signed_certificate) are now reward/bundle.py. The MockMed seed stays, because serve-reward --seed-mockmed needs it from a wheel. * Point the new teach-authorization test at tests._teach_scope * Restore #1741's packaging excludes, lazy-import guard and sys.path.append lost in the main merge * fix(diff): never apply a column rule to a row whose columns have no cell MACs (#1793 item 1) (#1801) Over max_rows a table keeps cell MACs only for its reference columns, but each row's contents MAC covers every column. A uuid backfill beside a clinical edit in the same row then showed only the uuid, and the profile's uuid-backfill rule excused the whole row in the rollback diff, the verifier's diff_database and the verify-run frame. Each of the three now applies a profile rule only on a table where every column has a cell MAC (db_snapshot.every_cell_hashed). Adds no excuse; only removes one. * API span: re-read the checks after step writes; keep an earlier halt reason (#1788) * API span: read the checks again after step writes; keep an earlier halt's reason The checks before saving ran in prepare(), before the span's step writes went out in send(). The GUI checks its empty-result guard after those writes, right before the save. When the span has step writes, send() now reads the checks again after the last write and halts without sending the save if one no longer passes. The end-of-run read of a still-armed wire check replaced the last step's error and category. It now adds the late refusal to an existing error and keeps an existing failure_category. * Test B: assert the span's own message, and pin category retention on a governed halt * fix(rollback): an audit entry names exactly one snapshot table (#1793 item 2) (#1803) rollback_state._same_table matched an unqualified log entry to any schema.table with that name. A rollback snapshot keys the read's own schema's tables unqualified and every other schema's as schema.table, so that let a default-schema entry credit (attribute, others_touched, _actors) or account for (unaccounted_writes) another schema's row. It now matches exactly, case aside. Every use only gets stricter: an entry that names no compared table attributes nothing, and the rollback goes to a person. The verify-run frame keeps its own matcher: its config names tables schema-qualified. * ci: skip the required reporters on a green run; balance Flow's unit shards (#1800) * ci: skip the required reporters on a green run; balance Flow's unit shards On 2026-11-03 a ci run had five serial stages, and each waited 15 to 30 minutes for a runner. ci and the two legacy required names did no work but added two of those stages. They now need packages, launcher and smoke directly and run only to fail the check (on a failure, a cancel or a draft). On a green run they are skipped, which GitHub counts as passing. The inner packages-gate still runs. flow-unit dealt files round-robin, so one run's shards took 4.9 to 9.4 minutes, and -n auto started 2 xdist workers on the 4-vCPU runner. scripts/ci/flow_shards.py packs files by measured time, longest first. Pull requests run 3 shards instead of 4, and flow-unit and flow-critical run one worker per vCPU. Every file still runs exactly once. CONTRIBUTING asks for one push per batch of fixes, a merge of main only on conflict, and gh run rerun --failed instead of empty commits. * ci: flow_shards review fixes Parse gh run view --log lines and colour codes, refuse to write empty durations, print nothing for an empty shard, and require every other ci.yml job in the reporters' needs. * Verifier baselines: OpenAdapt's verifier against a UV-class judge and API read-back (#1794) * Verifier baselines: pre-register the protocol and pin the inputs, before any scored run * Verifier baselines: UV-class judge port, effect_e2e arms, scorer, and results so far * Verifier baselines: report, final scores, and per-run data moved to the private corpus * Verifier baselines: score our verifier from the corrected Medplum replay (debdfe3) * Rollback: check desktop start screens through the step resolver (#1699) (#1799) * rollback(desktop): check start screens through the step resolver The desktop fresh session checked the run's start screen with a guard predicate, which resolves anchors through the visual ladder only. After a run the start screen's fields hold data, so the recorded crop no longer matched (measured on a 2880x1800 Windows lab: the start check refused with nothing covering the app). The start checks now use the same resolve() ladder a step uses: on UI Automation or AT-SPI its structural rung finds the same control and refuses one that is covered or off screen; pixel-only backends keep the visual ladder. Guard predicates in general are unchanged (their receipts are re-verified offline from frames). Qualified cleanup steps still run first. * lab(rollback): host profiles, operator review of selection keys, UIA lookup timing The Windows lab harness reads its guest paths, Python and RDP credentials from the lab's host profile (winrm_lab, #1789), so one script runs on both labs. It records each UIA lookup's timing, and applies the operator's risk review (qualification.set_action_classification) only to text-selection keys with no effect, recording each reviewed step and the rule. * rollback(desktop): a structural start hit counts only when nothing covers the control Review fixes for #1799. UI Automation and AT-SPI find a control behind a dialog or another app, so a structural hit alone doesn't show the start screen. It now counts only when a screen-wide hit test at its point names the same control (WindowsBackend: ControlFromPoint, hit_test_sees_every_window), and on any other backend only when the pixels show it too. A start state that isn't an anchor keeps the halt cleanup's guard. Each check keeps its frame and verdict under rollback/.fresh-session-N. * Linux AT-SPI element identity and CI qualification; macOS qualified on calibre (#1781) * WIP: Linux AT-SPI element identity, macOS Qt AX fixes, real-app qualification harnesses * Linux: AT-SPI read-back, physical input under a window manager; macOS: Qt cells, row identity; harnesses * Review fixes: AX splice race and UTF-16, macOS read-back keeps cell text, census reads rows, no alert dialog role; docs * Linux AT-SPI qualification results (fixture; GTK3 chart: record, compile, governed run, faults, repair) * Docs and results: macOS calibre qualification, Linux CI qualification, capabilities * fix(flow): an ISO date typed into a native date input lands as that date (#1806) * fix(flow): an ISO date typed into a native date input lands as that date Chromium splits <input type=date> into locale-ordered segments, so a typed "1984-03-07" doesn't land: "-" doesn't move to the next segment. type_text and type_text_guarded now set an ISO date into a native date input with fill, which takes the ISO value whatever the locale and fires input and change. The guarded path fills the leased element itself, and a failed fill raises (ActionDeliveryUncertain) instead of falling back to typing. Other text and other fields are typed as before. Salvaged from #1548 (second installation, Phase A). * fix(flow): date fill only for a real ASCII ISO date into an editable date input, through frames and shadow roots Review findings on #1806: fill clears a date input before rejecting a malformed value, so 2001-02-30 and non-ASCII digits are typed as before and never reach fill; readonly and disabled date inputs are typed as before; the guarded check and fill run on one element handle with a 1 s timeout; the unguarded path finds the focused date input inside frames and open shadow roots, and checks and fills that same handle. * Empty-result guard: mine a dialog's statement in any language from agreeing demonstrations (#1770) * Empty-result guard: mine a dialog's statement in any language from agreeing demonstrations A dialog line outside the English vocabulary is kept as a candidate by the compile and mined by induction when every demonstration, each with a different search, shows the same line. Induction no longer copies trace 0's guard when the demonstrations disagree; it drops it and notes why. The opener and its Confirm click are no longer read as a two-row loop. Qualification needs a business-rule case for a guard mined this way. * Empty-result guard evidence: refuse on disagreement, scope the checks (review round 1) Induction refuses (ambiguous_guard) when the demonstrations disagree on a mined dialog check, on the straight path and in optional, divergent or loop columns. The search must differ in the value typed last before the dialog. A question is never a candidate line. Qualification finds the checks on induced program graphs and only on a confirmation dialog's Confirm click. The induce CLI passes bundle compile notes; recording notes are not duplicated. * Empty-result guard evidence: keep induction's notes, compare openers too (review round 2) admit writes induction's dialog-check decisions into the induced bundle's compile_notes.json. The off-path agreement check also compares the opener's exact postcondition. The docs say plainly that the qualification campaign cannot run on an induced bundle yet. * Empty-result guard evidence: every typed value must differ, not just the last (review round 3) A later field that differs (a comment) no longer passes for a different search: every value typed since the last save and before the opener must be a parameter that differs across the demonstrations. Regression test on real recordings with the same name and different comments. Notes use step ids, and a confirmation pair off the straight path with no check is noted. * Empty-result guard evidence: count every typed value, one off-path note (review round 4) Drop the last-save boundary: an in-form Next or Apply button is inferred irreversible and let a later field stand in for the search. Every typed or selected value before the opener must differ. The off-path note is emitted once per column with the copied step's ids, and is tested. A record chosen by a click is documented as not counted. * Search-dialog fixtures: record headless, explicitly * Verifier specificity: operation semantics as contract data, and the leaderboard re-scored (#1795) * Leaderboard re-score: the verifier's reads served from the frozen Medplum history The write leaderboard's verifier lost its read token after an hour, so 163 of 222 scored runs were judged on an HTTP 401. The two stacks' databases kept every version and audit event. This replays ensure_record unchanged against them, as the server stood at each run's baseline and check, and reproduces all 59 verdicts the live verifier did reach (outcome, reasons, evidence). * ensure_record: operation semantics as contract data, and a per-app semantics spec An operations: section says what one write may also do; every default keeps the old rule. correction.between: consistent accepts a record saved in steps when every version held only the task's values; cardinality.own_transient accepts the agent's own duplicate deleted again, read by no one else; bookkeeping excuses exact element paths on other records, with an optional now_checks temporal rule. Nothing on the written record or on any record that is the person is excused. openadapt.app-semantics/v1 holds these per app and version, pinned by each contract; Medplum's is the sample. The Medplum harness renews the verifier's read token before each read: set once, it expired after an hour and the leaderboard's verifier judged 163 of 222 runs on HTTP 401. * Leaderboard re-score: pre-registration, before any unjudged run is re-scored * Leaderboard re-score: results, three logged deviations, and correction mode either The verifier, able to read, on the same 222 frozen runs: specificity 46/72 and 57/68 with the contracts on main, 55/72 and 42/68 with the pre-registered operations rule, 64/72 and 57/68 with the exploratory between: either (never stricter than #1655); sensitivity 25/28 and 30/33, every miss the post-check write A10. The replay now stops at the harness's next step (deviation 1) and reads a retried run's last seed (deviation 2); the as-registered results are kept beside the corrected ones. * Review fixes: a referenced duplicate refuses; bookkeeping names only counters, times and one extension From the independent review of #1795: a cleaned-up duplicate that any other write in the window refers to refuses, and the transient rule needs the audit heartbeat. Bookkeeping may name only meta.versionId, meta.lastUpdated and extension[url=...], never the contract's own type, and only for the agent's own updates; now_checks compares per path and checks dates. The semantics pin names the spec's sha256 and a relative path. consistent refuses extra elements on open contracts. Every re-score verdict is unchanged (deviation 4). * Register the re-score's public artifacts in the inventory * docs: add AGENTS.md with rules for automated coding agents (#1817) * docs: add AGENTS.md with rules for automated coding agents * docs(AGENTS.md): one claim rule for every agent * docs(AGENTS.md): claim through the refs API, which refuses an existing branch * Serve the fault benches as OpenEnv and Prime Intellect RL environments (#1780) * feat(flow): serve the fault benches as OpenEnv and Prime Intellect RL environments One generic browser environment (openadapt_flow.rl_env) serves every app over the OpenEnv protocol. The reward is verify_run's verdict on the app's record, read through a channel the agent's browser cannot reach; each episode also returns an outcome vector (completion, harm, false refusal, steps). - MockMed: 11-task public sample with hidden faults. MockLoan and the held-out MockMed faults ship no public tasks; they load from a pack kept outside the package (OPENADAPT_ENV_EXTRA_TASKS), and every pack is hashed. - OpenEMR (GPL-3.0), Frappe Lending (GPL-3.0) and openIMIS (AGPL-3.0): manifests name upstream images by digest; no app code is copied. OpenEMR has a task pack awaiting a live run; Frappe has its oracle; openIMIS has no reset yet. - A failure report on an impossible task pays only after the browser sent the task's write, so not trying never earns reward. - environments/prime/openadapt_web_envs: verifiers v1 taskset over the same servers. * fix(flow): close the review's reward leaks in the RL environments - Public generator holds public cases only; held-out packs are built from a private cases file. The held-out instances in the previous commit are retired and re-authored privately. - Task ids are numbers and the observation carries no task id, so nothing the agent sees names the fault. - External apps (one shared database) serve one session at a time. - A failed reset leaves no live task or judge; a route-handler error can no longer hang a request; write detection compares trimmed values and numbers by value; the write_requests rule also covers extra packs; goto takes only paths; missing environment files raise TaskError. - run_eval wraps verifiers' eval: it never uploads traces without --allow-upload, and a test fails if our tooling calls eval directly. * docs(flow): validation report for the RL environments; refuse impossible wrong_record cases MockMed's public sample passes over OpenEnv and the Prime path with zero model calls; 8 parallel sessions all reach the maximum reward; reset leaves no state behind. The generator now refuses a wrong_record task on the patient that fault writes to, which validation caught in a held-out pack. * fix(flow): register the RL environment files; keep the openenv extra on Python 3.11+ The public artifact inventory now lists environments/openenv. On Python 3.10 the OpenEnv stack resolved to dnspython 2.8.0 (GHSA-45c3-73f7-pv4m); the extra now applies on 3.11+ only, which verifiers already requires. * fix(flow): end an RL episode whose step or reset gets stuck A step longer than 120 s (reset 180 s) now returns a final observation with reward 0 and verdict INDETERMINATE, drops the stuck browser, and lets the next reset start afresh, instead of leaving the client waiting. Seen once on a WSL host during held-out validation. * fix(flow): a stuck RL step that wakes later cannot touch the next episode On abandonment the watchdog now kills the stuck browser's process tree (the Playwright driver and the Chromium under it) and replaces an in-process app copy, so the next reset starts on a fresh store. A test lets the stuck step wake after the reset and try its write: the new episode's store stays empty and its scripted run still earns 1.0. * fix(flow): mypy in rl_env environment (judge close, elements annotation) * Rollback on durable runs, and after a crash (#1699) (#1772) * feat(rollback): roll back desktop and remote-display runs (#1699) A desktop or RDP run with rollback on no longer refuses. A GUI compensation starts from the run's start screen: the halt cleanup (only when qualified) and its start check on the run's own backend. A screen that doesn't come back refuses the compensation, and the run ends RECONCILIATION_REQUIRED. The whole-database reads before and after each compensation prove the undo, as on the web. Adds the Windows lab harness (native and RDP) for Patient Notes. * test(rollback): a compensation the app reports done but never wrote; lab undo restores the kept value * feat(rollback): roll back durable runs and continue after a crash (#1699) A durable run with rollback on no longer refuses. The read from before the run, its row-digest key, the audit log as it was and the run's parameters are sealed (AES-256-GCM, the deployment's key) in <run>/rollback. A durable pause waits for a person: resume it, or roll it back with resume --roll-back. The first attempt seals its read after the stop and journals each compensation it starts; a later attempt judges the run's changes by that read, stops if anything else changed since, and re-reads before treating any started compensation as done. Without a key a durable run with rollback is refused. * lab(rollback): drop screen checks that quote the case-dependent note; search without Enter * docs(rollback): durable runs and a rollback a crash interrupted * rollback(desktop): check the compensation's own start screen; a screen undo counts for undo-or-person only with a qualified cleanup; docs Review fixes for #1771: before a GUI compensation on a desktop or remote display, check the compensation's first screen as well as the run's (a persistent menu can satisfy the run's check while the wrong record is up). A GUI undo there counts as an undo for the undo-or-person rule only when the bundle's halt cleanup has steps and is qualified; otherwise each save goes to a person first. Fresh-session dirs can't collide with undo ids. ROLLBACK.md, CAPABILITIES.md and llms-full.txt describe the desktop path. * rollback(durable): bind sealed state to its run; never run a compensation twice; --roll-back keeps resume's guards Review fixes for #1772: - Every sealed file carries the arming's id; a file from another run or a missing journal is refused. Arming afresh clears an earlier stop read and journal. A durable pause binds the state to the manifest's run id. - The journal is sealed. A compensation it marks finished never runs again; one only started is skipped when a fresh read shows the change gone, and otherwise goes to a person. A rollback that ended keeps its verdict, drops its reads, and refuses --roll-back. - --roll-back refuses a changed bundle version, changed rollback settings (attribution, exclusive, allow_delete, scope, profile), a managed dispatch or qualification authority, and a second concurrent rollback (lock). - Only durable runs seal state. A resumed run that stops again prints the --roll-back command. * fix(rollback): CI on #1771: fresh_session takes the compensation (mypy); check screen undo only with rollback on * rollback(durable): bind the sealed state before an in-process rollback, so a crash there can be continued; platform-guard the Windows lock (mypy) * fix(rollback): roll-back hints use the spelling the person typed (CI on #1772) * Zero-touch setup: generated contracts admitted by a sandbox, no person answers (#1753) * Zero-touch setup: literature review, design, sandbox validator prototype (WIP) * Zero-touch: fault_gate class names, #1520 decisions, now_checks docs, capability row * Zero-touch gate: fail closed on unplanted faults, fault every column, require core classes, strict pins, production equals validated * Zero-touch: type-made plants for typed columns; strict_pins pins a one-demonstration constant; docs * Zero-touch: two-phase sandbox restore; a duplicate the store's unique key refuses is not applicable (local, for review) * Zero-touch: a duplicate is not applicable only when a declared unique index covering unchanged columns refuses it; recorded per kind * Zero-touch round 2: failed reads never count as stopped; found keeps only constants of strict roles; validated field sets only; key and unrelated-edit faults on creates * Zero-touch: plants avoid declared unique keys; failing cases log their reason * Zero-touch design: failed reads, forbidden duplicates, validated field sets, strict pins * Zero-touch round 3: fresh app-made unique values in duplicates; transactional restore stops the setup; structured read evidence; plain unique indexes only * Zero-touch round 4: task-given columns come from the generator's bindings, not value matches; stopped setup keeps old contracts aside and exits non-zero * Zero-touch: typing-only fixes for mypy (no Optional reassignment, one type per variable) * fix(flow): refuse volatile FHIR metadata before run 1 (#1819) * fix(flow): refuse volatile FHIR metadata before run 1 Signed-off-by: abrichr <richard.abrich@mldsai.com> * ci: refresh checks after marking PR ready Signed-off-by: abrichr <richard.abrich@mldsai.com> --------- Signed-off-by: abrichr <richard.abrich@mldsai.com> * desktop: prepare review card input before paint (#1825) * fix(desktop): prepare review card input before paint Signed-off-by: abrichr <richard.abrich@mldsai.com> * chore(desktop): prepare 0.25.1 patch release metadata Signed-off-by: abrichr <richard.abrich@mldsai.com> --------- Signed-off-by: abrichr <richard.abrich@mldsai.com> --------- Signed-off-by: abrichr <richard.abrich@mldsai.com>
abrichr
added a commit
that referenced
this pull request
Nov 4, 2026
* Engine generality audit; move the benchmark harnesses out of the wheel Audit every place the engine (packages/flow/openadapt_flow) names a specific application, bucketed as example, data, harness in the package, or real special-casing: docs/GENERALITY_AUDIT.md. Move openadapt_flow/benchmark and openadapt_flow/instrument to packages/flow/benchmark/harness (import benchmark.harness). The wheel packages only openadapt_flow, so it no longer ships them. The benchmark command loads the harness from a source checkout and says so from a wheel; instrument already failed to import from a wheel. Reword two comments that named an app on a generic branch. * Generality audit: link CONTRIBUTING without the guard PR's anchor * CI: fail when the engine names a specific application scripts/check_engine_generality.py scans openadapt_flow/ for the application names and app-specific identifiers in scripts/generality.toml. app_profiles/ is data and passes; OpenAdapt's demo modules may name the demo app; a comment line may give an example under '# example:'. Hits that predate the check are counted in generality_baseline.json, which only shrinks: a rise fails, and so does a fall the baseline doesn't record. Runs in the flow lint job. CONTRIBUTING documents the rule: app knowledge lives in profiles, contracts and fixtures. * Register the generality denylist and baseline as public artifacts * Engine generality audit; move the benchmark harnesses out of the wheel Audit every place the engine (packages/flow/openadapt_flow) names a specific application, bucketed as example, data, harness in the package, or real special-casing: docs/GENERALITY_AUDIT.md. Move openadapt_flow/benchmark and openadapt_flow/instrument to packages/flow/benchmark/harness (import benchmark.harness). The wheel packages only openadapt_flow, so it no longer ships them. The benchmark command loads the harness from a source checkout and says so from a wheel; instrument already failed to import from a wheel. Reword two comments that named an app on a generic branch. * Generality audit: link CONTRIBUTING without the guard PR's anchor * CONTRIBUTING: state the app-knowledge rule positively * Move test fixtures and demo harnesses out of the openadapt-flow wheel The wheel ships only the engine (see packages/flow/docs/GENERALITY_AUDIT.md). Moved out of openadapt_flow: - integrations/harness2 -> benchmark/harness/harness2 (the harness2 extra is dropped; install [browser] beside Harness² from a checkout) - mockloan/ and demo_driver.record_disbursement_demo -> benchmark/harness/mockloan/ (recorder in demo.py) - guard_demo.py -> benchmark/harness/guard_demo.py - agent/ledger_demo.py -> packages/agent/tests/ledger_demo.py - mockmed/teach_scope.py -> packages/flow/tests/_teach_scope.py - learning/synth_stream.py: StructuralDiffInducer -> learning/induce.py; the MockMed drift stream -> benchmark/harness/synth_stream.py reward/seed.py is split: the bundle helpers (write_bundle, calibrate_bundle, self_signed_certificate) are now reward/bundle.py. The MockMed seed stays, because serve-reward --seed-mockmed needs it from a wheel. * Guard: reword new app-name examples from main; lower the baseline to 365 * Point the new teach-authorization test at tests._teach_scope * Guard: take main's harness import order; reword a new MockMed example * Guard: count app names inside identifiers; restore main's lint, packaging and import-guard lines * Undo an unrelated reformat of scripts/ * Restore #1741's packaging excludes, lazy-import guard and sys.path.append lost in the main merge * Guard: ordinary words like text_edit and mock_media pass; baseline 425 * fix(flow): lower generality baseline after harness relocation Signed-off-by: abrichr <richard.abrich@mldsai.com> --------- Signed-off-by: abrichr <richard.abrich@mldsai.com>
abrichr
added a commit
that referenced
this pull request
Nov 4, 2026
…#1774) * Engine generality audit; move the benchmark harnesses out of the wheel Audit every place the engine (packages/flow/openadapt_flow) names a specific application, bucketed as example, data, harness in the package, or real special-casing: docs/GENERALITY_AUDIT.md. Move openadapt_flow/benchmark and openadapt_flow/instrument to packages/flow/benchmark/harness (import benchmark.harness). The wheel packages only openadapt_flow, so it no longer ships them. The benchmark command loads the harness from a source checkout and says so from a wheel; instrument already failed to import from a wheel. Reword two comments that named an app on a generic branch. * Generality audit: link CONTRIBUTING without the guard PR's anchor * CI: fail when the engine names a specific application scripts/check_engine_generality.py scans openadapt_flow/ for the application names and app-specific identifiers in scripts/generality.toml. app_profiles/ is data and passes; OpenAdapt's demo modules may name the demo app; a comment line may give an example under '# example:'. Hits that predate the check are counted in generality_baseline.json, which only shrinks: a rise fails, and so does a fall the baseline doesn't record. Runs in the flow lint job. CONTRIBUTING documents the rule: app knowledge lives in profiles, contracts and fixtures. * Register the generality denylist and baseline as public artifacts * Verifier: no app's REST paths or key field as engine defaults RestRecordVerifier, RestCompensator, fault_gate's memory copy, effects.kind rest and rest reward recipes defaulted to MockMed's /api/db, records and /api/encounter/{id}. The paths are now required; a rest config without records_path is refused when its verifier is built. Compile-time REST binding admission guessed MockMed's write path /api/encounter and skipped its server-assigned 'source' field. The AdmissionFixture now names the app's read path, write path, server-assigned fields and idempotency key field; with no captured api_write and no write path, a step gets no proposal. compile_recording takes admission_fixture instead of admission_fixture_url. Effect mining and scaffold-verifier mined an idempotency key from any record field named 'key' (MockMed's). They mine one only from the field the caller names. * Verifier: FHIR layout, the no-record value and server facts come from data Three engine defaults encoded one app: - FhirEffectVerifier, effects.kind fhir and fhir reward recipes defaulted to OpenEMR's note-as-Observation layout (resource Observation, note in valueString). They now require resource_type and field_paths; a config without them is refused when the verifier is built. - The frame and rollback attribution read an audit entry for record "0" as naming no record, which is how OpenEMR files bookkeeping inserts. An app profile now lists such values (no_record_values); the OpenEMR profile says "0" (version 2026-11-03.1). An empty field still names no record in any app. - ensure-contract generation translated the Medplum probe's own output keys. It reads only the generic fact names now; the translation moved beside the probe, and the committed teach fixtures hold generic facts. * Engine generality audit; move the benchmark harnesses out of the wheel Audit every place the engine (packages/flow/openadapt_flow) names a specific application, bucketed as example, data, harness in the package, or real special-casing: docs/GENERALITY_AUDIT.md. Move openadapt_flow/benchmark and openadapt_flow/instrument to packages/flow/benchmark/harness (import benchmark.harness). The wheel packages only openadapt_flow, so it no longer ships them. The benchmark command loads the harness from a source checkout and says so from a wheel; instrument already failed to import from a wheel. Reword two comments that named an app on a generic branch. * Generality audit: link CONTRIBUTING without the guard PR's anchor * Verifier: keep issued identities and check digests; --effects-records-path Adversarial review found that dropping the records_path and resource_type defaults changed the verifier identity of every existing config, and the fault-gate check digest of an explicit resource_type: Observation. Both now hash the old values (LEGACY_HASHED_DEFAULTS), for hashing only; a read without its own path or resource is still refused. run --effects-kind rest takes --effects-records-path, and the CLI tests that relied on the old default pass it. Test configs that read the demo app name its path. * CONTRIBUTING: state the app-knowledge rule positively * Move test fixtures and demo harnesses out of the openadapt-flow wheel The wheel ships only the engine (see packages/flow/docs/GENERALITY_AUDIT.md). Moved out of openadapt_flow: - integrations/harness2 -> benchmark/harness/harness2 (the harness2 extra is dropped; install [browser] beside Harness² from a checkout) - mockloan/ and demo_driver.record_disbursement_demo -> benchmark/harness/mockloan/ (recorder in demo.py) - guard_demo.py -> benchmark/harness/guard_demo.py - agent/ledger_demo.py -> packages/agent/tests/ledger_demo.py - mockmed/teach_scope.py -> packages/flow/tests/_teach_scope.py - learning/synth_stream.py: StructuralDiffInducer -> learning/induce.py; the MockMed drift stream -> benchmark/harness/synth_stream.py reward/seed.py is split: the bundle helpers (write_bundle, calibrate_bundle, self_signed_certificate) are now reward/bundle.py. The MockMed seed stays, because serve-reward --seed-mockmed needs it from a wheel. * Verifier: contract hashes keep their old values; name the key field Second review: Contract.digest, store and identity hashed the verifier dump directly, so every sql and fhir contract changed its hash (write ledger keys, reward certificates). They go through the same legacy shim now; a test pins main's values for sql, rest and fhir contracts. compile_recording and scaffold-verifier take idempotency_key_field, so a caller can name the app's at-most-once key field. Test verifier subclasses name the demo app's records path. * Mine the 'key' convention as the default idempotency field; give the new tests their record paths * Guard: reword new app-name examples from main; lower the baseline to 365 * Point the new teach-authorization test at tests._teach_scope * Guard: take main's harness import order; reword a new MockMed example * Guard: count app names inside identifiers; restore main's lint, packaging and import-guard lines * Undo an unrelated reformat of scripts/ * Restore #1741's packaging excludes, lazy-import guard and sys.path.append lost in the main merge * Guard: ordinary words like text_edit and mock_media pass; baseline 425 * fix(flow): lower generality baseline after harness relocation Signed-off-by: abrichr <richard.abrich@mldsai.com> * test(flow): declare read paths in synthetic verifier fixtures Signed-off-by: abrichr <richard.abrich@mldsai.com> --------- Signed-off-by: abrichr <richard.abrich@mldsai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of several pull requests that make the engine (
packages/flow/openadapt_flow/) free of application-specific code.Audit
packages/flow/docs/GENERALITY_AUDIT.mdlists every place the engine names an application (OpenEMR, Medplum, Frappe, Stedi, MockMed and others) or uses an identifier that only one application has. Each hit has a file and line, a bucket and an action:benchmark/The (d) findings table says which pull request fixes each one. Three are follow-up issues: thresholds measured on one app, a hand-written list of non-identifying words, and a vendor eligibility client.
Change
openadapt_flow/benchmark/andopenadapt_flow/instrument/topackages/flow/benchmark/harness/(importbenchmark.harness). The wheel packages onlyopenadapt_flow, so it no longer ships them. I built the wheel and checked it: 625 files, none underbenchmark/orinstrument/.openadapt-flow benchmarkloads the harness from a source checkout. Run from a wheel, it says that the harness is not shipped, the waysilent-savesalready does.instrumentimported the benchmark when it loaded, so it was already broken in a wheel.tests/test_runtime_never_imports_benchmark.pynow fails if abenchmarkdirectory comes back inside the package, or if product code importsbenchmark.*.No engine logic changes.
Tests
The 15 test files that import the harness: 381 passed.