Engine generality audit; move the benchmark harnesses out of the wheel

.github/workflows/packages.yml

@@ -347,7 +347,7 @@ jobs:
         # every public data/config asset listed in public-artifacts.json.
         run: python scripts/check_release_consistency.py
       - name: Ruff lint
-        run: ruff check openadapt_flow
+        run: ruff check openadapt_flow benchmark/harness
       - name: Install uv (for the oldest-Python syntax check)
         uses: astral-sh/setup-uv@18d369b23da25a81e9458561248e1aafff6a70fa # v10.3.1
       - name: Syntax at the oldest supported Python (3.10)
@@ -362,7 +362,7 @@ jobs:
           ruff check --isolated --select E9 --target-version py310 .
           uv run --no-project --python 3.10 python -m compileall -q -x '(^|/)(\.venv|node_modules|build|dist)/' .
       - name: Ruff format check
-        run: ruff format --check openadapt_flow tests
+        run: ruff format --check openadapt_flow tests benchmark/harness
       - name: Mypy
         run: mypy
       - name: Mypy (strict, safety path only)

packages/flow/CHANGELOG.md

@@ -17,6 +17,10 @@
 
 ## Unreleased
 
+### Changed
+
+- The wheel no longer ships the benchmark harnesses. `openadapt_flow.benchmark` and `openadapt_flow.instrument` moved to `packages/flow/benchmark/harness/` in the source tree (import `benchmark.harness`); `openadapt-flow benchmark` runs from a source checkout and says so from a wheel. The engine audit for application-specific code is in `docs/GENERALITY_AUDIT.md`. Code that imported `openadapt_flow.benchmark` or `openadapt_flow.instrument` from an installed wheel imports `benchmark.harness` from a source checkout instead.
+
 ### Features
 
 - `run --self-repair` (and repair by default) now works outside a browser. `repair.session` in `deploy.yaml` says how to start the app for each probe, fault probe and resumed run (an argv, no shell, kept running while the app is up; `fault_commands`, `close_command`, `ready_s`). The run's configured backend must bind to a window owned by that command's process group, so a session never binds to the halted session or a copy already open; Flow ends the group afterwards, and closes the halted session before each probe. This works where the bound window is on Flow's machine (macOS, and the local RDP or Citrix client window); Windows over its agent and FreeRDP are refused for now. Without `repair.session`, a desktop run refuses repair with that reason, as before.

packages/flow/benchmark/agent_arm_verticals/README.md

@@ -26,7 +26,7 @@ budget without completing the write in every trial.
 ## Method and public/private boundary
 
 The public, generic agent-baseline mechanism remains in
-`openadapt_flow/benchmark/agent_baseline.py`. It provides explicit paid-run
+`benchmark/harness/agent_baseline.py`. It provides explicit paid-run
 opt-in, action and cost caps, usage accounting, and an agent self-report that
 is never trusted for scoring. The public benchmark schemas and aggregate
 failure taxonomy also remain available for independently authored tasks.

packages/flow/benchmark/code_as_action/measure_openemr.py

@@ -184,7 +184,7 @@
 
 
 def spec_for(key: str) -> Any:
-    from openadapt_flow.benchmark.openemr_local import SyntheticPatientSpec
+    from benchmark.harness.openemr_local import SyntheticPatientSpec
 
     return SyntheticPatientSpec(**REFERRALS[key], **COMMON_FIELDS)
 
@@ -390,7 +390,7 @@ def main() -> int:
     keys = [k for k in args.referrals.split(",") if k]
 
     from benchmark.openemr_local import qualify_hooks
-    from openadapt_flow.benchmark.openemr_local import SyntheticPatientSpec
+    from benchmark.harness.openemr_local import SyntheticPatientSpec
 
     fixture = qualify_hooks.fixture()
     demo = _demo()

packages/flow/benchmark/dom/BENCHMARK.md

@@ -23,7 +23,7 @@ row's Open button — the literal reading — and **DOM (name-filtered)**
 clicks the Open button in the row named `Jane Sample` — the
 identity reading, hardcoding the demonstrated patient exactly as the
 compiled arm's recorded identity band does. Every selector choice is
-documented in `openadapt_flow/benchmark/dom_arm.py`.
+documented in `benchmark/harness/dom_arm.py`.
 
 ## Verdict
 
@@ -208,7 +208,7 @@ Unmeasured variants, for completeness:
 ## Reproduce
 
 ```
-.venv/bin/python -m openadapt_flow.benchmark.dom_arm --out benchmark/dom --n-per-perturbation 2
+.venv/bin/python -m benchmark.harness.dom_arm --out benchmark/dom --n-per-perturbation 2
 ```
 
 (`--n-per-perturbation 2` matches the committed results.) No API key

packages/flow/benchmark/effect_e2e/EFFECT_E2E.md

@@ -2,7 +2,7 @@
 
 Date: 2026-07-21. This is the genuinely independent, end-to-end measurement of the
 silent-wrong-effect (SWER) result. Unlike the definitional
-`openadapt_flow.benchmark.silent_wrong_action` (raw `requests.post`; effect
+`benchmark.harness.silent_wrong_action` (raw `requests.post`; effect
 verifier and ground truth read the SAME in-process object; the effect contract
 restates the ground-truth definition -- so its `0/90` is circular by
 construction), every write below goes through the ACTUAL governed replay path

packages/flow/benchmark/effect_e2e/record_service.py

@@ -3,7 +3,7 @@
 This is the system of record the end-to-end silent-wrong-effect harness
 (:mod:`benchmark.effect_e2e.run`) writes to THROUGH the governed replay path.
 It exists to break the circularity of the older definitional benchmark
-(``openadapt_flow.benchmark.silent_wrong_action``), whose effect verifier and
+(``benchmark.harness.silent_wrong_action``), whose effect verifier and
 ground-truth judge read the SAME in-process Python object and whose write
 bypassed the :class:`~openadapt_flow.runtime.replayer.Replayer`.
 

packages/flow/benchmark/effect_e2e/run.py

@@ -1,7 +1,7 @@
 """End-to-end silent-wrong-effect (SWER) harness through the REAL replayer.
 
 This is the genuinely independent, end-to-end version of the silent-wrong-action
-result. The older ``openadapt_flow.benchmark.silent_wrong_action`` is
+result. The older ``benchmark.harness.silent_wrong_action`` is
 *definitional*: it issues raw ``requests.post`` (bypassing the
 :class:`~openadapt_flow.runtime.replayer.Replayer`), and its effect verifier and
 ground-truth judge both read the SAME in-process ``FaultDB`` object while the
@@ -629,7 +629,7 @@ def render_markdown(results: dict[str, Any]) -> str:
 
 Date: {date}. This is the genuinely independent, end-to-end measurement of the
 silent-wrong-effect (SWER) result. Unlike the definitional
-`openadapt_flow.benchmark.silent_wrong_action` (raw `requests.post`; effect
+`benchmark.harness.silent_wrong_action` (raw `requests.post`; effect
 verifier and ground truth read the SAME in-process object; the effect contract
 restates the ground-truth definition -- so its `0/90` is circular by
 construction), every write below goes through the ACTUAL governed replay path
@@ -774,7 +774,7 @@ def render_markdown(results: dict[str, Any]) -> str:
 
 def render_chart(results: dict[str, Any], out_png: Path) -> Path:
     """Bar chart: silent-wrong-effect rate per arm (large -> small -> zero)."""
-    from openadapt_flow.benchmark.chart_fonts import configure_bundled_font
+    from benchmark.harness.chart_fonts import configure_bundled_font
 
     plt = configure_bundled_font()
     m = results["metrics"]["per_arm"]
@@ -827,7 +827,7 @@ def write_outputs(results: dict[str, Any], out_dir: Path) -> None:
     out_dir.mkdir(parents=True, exist_ok=True)
     (out_dir / "results.json").write_text(json.dumps(results, indent=2) + "\n")
     (out_dir / "EFFECT_E2E.md").write_text(render_markdown(results))
-    from openadapt_flow.benchmark.chart_fonts import safe_render
+    from benchmark.harness.chart_fonts import safe_render
 
     safe_render(render_chart, results, out_dir / "effect_e2e.png")
 

packages/flow/benchmark/effectbench/reference_fault_model.py

@@ -31,16 +31,16 @@
 How it re-expresses the study, faithfully and without hardcoding verdicts:
 
 - Each of the study's nine scenarios (``benchmark.fault_model.faults.FAULTS``)
-  becomes a :class:`~openadapt_flow.benchmark.effectbench.schema.TaskSpec` plus
+  becomes a :class:`~benchmark.harness.effectbench.schema.TaskSpec` plus
   the KNOWN post-action system-of-record state that fault produces (the same DB
   states the study's own taxonomy tests assert). The states live in
   :data:`SCENARIOS` — the record shapes are MockMed ``FaultDB`` rows.
 - The TRUE effect is read by the substrate-agnostic
-  :class:`~openadapt_flow.benchmark.effectbench.oracle.RecordSnapshotOracle`
+  :class:`~benchmark.harness.effectbench.oracle.RecordSnapshotOracle`
   running the SAME shared ``judge_records`` decision logic the runtime uses —
   once for the at-most-once/collateral ``record_written`` contract and once for
   the ``field_equals`` note read-back — reduced by
-  :func:`~openadapt_flow.benchmark.effectbench.oracle.combine_true_states`.
+  :func:`~benchmark.harness.effectbench.oracle.combine_true_states`.
 - Two arms are scored against the identical independent oracle: ``screen_only``
   (the weak banner oracle — reports success whenever the screen looks saved) and
   ``effect_verify`` (the agent halts unless the oracle CONFIRMS). Only the
@@ -56,7 +56,7 @@
 from typing import Optional
 
 from benchmark.fault_model import faults as F
-from openadapt_flow.benchmark.effectbench import (
+from benchmark.harness.effectbench import (
     AgentReport,
     DivergenceCategory,
     Effect,
@@ -72,7 +72,7 @@
     oracle_verdict_of,
     summarize,
 )
-from openadapt_flow.benchmark.effectbench.schema import OracleChannel
+from benchmark.harness.effectbench.schema import OracleChannel
 
 # The consequential-save contract the study verifies: exactly one Triage
 # encounter for the target patient (at-most-once + no collateral loss) whose

packages/flow/benchmark/effectbench/task_pack/README.md

@@ -68,7 +68,7 @@ python -m benchmark.effectbench.task_pack.pack
 
 # Live-run the MockMed anchor through the oracle (prints per-task classification)
 python -c "from benchmark.effectbench.task_pack.driver import run_mockmed_pack; \
-from openadapt_flow.benchmark.effectbench import summarize; \
+from benchmark.harness.effectbench import summarize; \
 eps=run_mockmed_pack(trials=3); \
 print('screen SWER', summarize(eps, arm='screen_only').swer.rate); \
 print('effect SWER', summarize(eps, arm='effect_verify').swer.rate)"

packages/flow/benchmark/effectbench/task_pack/__init__.py

@@ -6,9 +6,9 @@
 ``benchmark.environments`` registry (PR #173) indexes, authored against the
 frozen EffectBench schema/oracle/metrics contract (PR #178).
 
-Each task is a :class:`~openadapt_flow.benchmark.effectbench.TaskSpec` (goal =
+Each task is a :class:`~benchmark.harness.effectbench.TaskSpec` (goal =
 INTENT only, never a step list) plus an
-:class:`~openadapt_flow.benchmark.effectbench.OracleSpec` bound to the
+:class:`~benchmark.harness.effectbench.OracleSpec` bound to the
 environment's system-of-record READ recipe, DESIGNED so a green screen
 (task-success) diverges from the correct business effect. The oracle reads
 pre/post SoR state independently of the screen; the non-gameability attestations

packages/flow/benchmark/effectbench/task_pack/_authoring.py

@@ -3,10 +3,10 @@
 These build the two things every consequential-write task needs:
 
 - the DECLARED expected effect (a typed
-  :class:`~openadapt_flow.benchmark.effectbench.Effect`), parameterized by run
-  params via :class:`~openadapt_flow.benchmark.effectbench.ValueExpr` so the
+  :class:`~benchmark.harness.effectbench.Effect`), parameterized by run
+  params via :class:`~benchmark.harness.effectbench.ValueExpr` so the
   record CHECKED is the record this trial WROTE (trial-unique payload); and
-- the :class:`~openadapt_flow.benchmark.effectbench.OracleSpec` that declares
+- the :class:`~benchmark.harness.effectbench.OracleSpec` that declares
   the independent read channel and the non-gameability attestations the
   adversarial audit signs off on.
 
@@ -21,13 +21,13 @@
 
 from typing import Any, Optional
 
-from openadapt_flow.benchmark.effectbench import (
+from benchmark.harness.effectbench import (
     Effect,
     EffectKind,
     OracleSpec,
     ValueExpr,
 )
-from openadapt_flow.benchmark.effectbench.schema import OracleChannel
+from benchmark.harness.effectbench.schema import OracleChannel
 
 # Canonical run-param names a trial binds a trial-unique value to. The driver
 # derives each from the trial seed so the oracle checks THIS run's exact write.

packages/flow/benchmark/effectbench/task_pack/audit.py

@@ -41,7 +41,7 @@
     _note_effect,
     _record_effect,
 )
-from openadapt_flow.benchmark.effectbench import RestRecordVerifier, Verdict
+from benchmark.harness.effectbench import RestRecordVerifier, Verdict
 
 _HTTP_TIMEOUT_S = 5.0
 TRIAL_NOTE = "triage note trial-unique-AUDIT"

packages/flow/benchmark/effectbench/task_pack/driver.py

@@ -8,9 +8,9 @@
 mocked verdicts.
 
 It depends ONLY on the public EffectBench surface
-(:func:`~openadapt_flow.benchmark.effectbench.score_episode`,
-:class:`~openadapt_flow.benchmark.effectbench.RestRecordVerifier`,
-:class:`~openadapt_flow.benchmark.effectbench.AgentReport`) and the MockMed
+(:func:`~benchmark.harness.effectbench.score_episode`,
+:class:`~benchmark.harness.effectbench.RestRecordVerifier`,
+:class:`~benchmark.harness.effectbench.AgentReport`) and the MockMed
 fault server. It does not reimplement any real baseline "arm"; the two arms
 here are the two SCORING conditions the benchmark contrasts:
 
@@ -46,7 +46,7 @@
     MockMedDrive,
     MockMedTask,
 )
-from openadapt_flow.benchmark.effectbench import (
+from benchmark.harness.effectbench import (
     AgentReport,
     EpisodeRecord,
     RestRecordVerifier,

packages/flow/benchmark/effectbench/task_pack/frappe_tasks.py

@@ -17,7 +17,7 @@
 
 ``oracle.config["query"]`` + ``["query_params"]`` is what the multi-baseline
 runner materializes into a
-:class:`~openadapt_flow.benchmark.effectbench.SqlRecordVerifier` (read-only role,
+:class:`~benchmark.harness.effectbench.SqlRecordVerifier` (read-only role,
 DB-API-bound params). The loan-application docname has a space
 (`` `tabLoan Application` ``), so the table is back-tick quoted in the SELECT.
 """
@@ -33,15 +33,15 @@
     record_written_effect,
 )
 from benchmark.effectbench.task_pack.openemr_tasks import ContainerTask
-from openadapt_flow.benchmark.effectbench import (
+from benchmark.harness.effectbench import (
     DivergenceCategory as DC,
 )
-from openadapt_flow.benchmark.effectbench import (
+from benchmark.harness.effectbench import (
     Effect,
     Substrate,
     TaskSpec,
 )
-from openadapt_flow.benchmark.effectbench.schema import OracleChannel
+from benchmark.harness.effectbench.schema import OracleChannel
 
 ENVIRONMENT = "frappe_lending"
 

packages/flow/benchmark/effectbench/task_pack/mockmed_tasks.py

@@ -8,7 +8,7 @@
 path is exercised in CI (no mocked verdicts).
 
 Each entry is a :class:`MockMedTask`: an agent-agnostic
-:class:`~openadapt_flow.benchmark.effectbench.TaskSpec` (goal = intent only)
+:class:`~benchmark.harness.effectbench.TaskSpec` (goal = intent only)
 plus a :class:`MockMedDrive` — the environment recipe the live driver replays
 to MANIFEST the divergence (which ``?fault=`` mode to inject, the write the
 agent performs, decoy/concurrent rows to seed, and the documented banner
@@ -34,9 +34,9 @@
     param,
     record_written_effect,
 )
-from openadapt_flow.benchmark.effectbench import DivergenceCategory as DC
-from openadapt_flow.benchmark.effectbench import Effect, Substrate, TaskSpec
-from openadapt_flow.benchmark.effectbench.schema import OracleChannel
+from benchmark.harness.effectbench import DivergenceCategory as DC
+from benchmark.harness.effectbench import Effect, Substrate, TaskSpec
+from benchmark.harness.effectbench.schema import OracleChannel
 
 TARGET_PATIENT = "p1"
 TARGET_TYPE = "Triage"

packages/flow/benchmark/effectbench/task_pack/openemr_tasks.py

@@ -18,7 +18,7 @@
 
 The SQL each oracle runs (``oracle.config["query"]`` + ``["query_params"]``) is
 what the multi-baseline runner materializes into a
-:class:`~openadapt_flow.benchmark.effectbench.SqlRecordVerifier` (read-only role;
+:class:`~benchmark.harness.effectbench.SqlRecordVerifier` (read-only role;
 values bound through DB-API params, never interpolated). Column names become
 record keys, so the same typed ``Effect`` the MockMed anchor uses judges these
 rows unchanged.
@@ -36,15 +36,15 @@
     param,
     record_written_effect,
 )
-from openadapt_flow.benchmark.effectbench import (
+from benchmark.harness.effectbench import (
     DivergenceCategory as DC,
 )
-from openadapt_flow.benchmark.effectbench import (
+from benchmark.harness.effectbench import (
     Effect,
     Substrate,
     TaskSpec,
 )
-from openadapt_flow.benchmark.effectbench.schema import OracleChannel
+from benchmark.harness.effectbench.schema import OracleChannel
 
 ENVIRONMENT = "openemr_local"
 

packages/flow/benchmark/effectbench/task_pack/pack.py

@@ -4,7 +4,7 @@
 the container-gated OpenEMR and Frappe families). It:
 
 - assembles the unified list of :class:`PackEntry` (one per authored task),
-  each carrying its :class:`~openadapt_flow.benchmark.effectbench.TaskSpec`, the
+  each carrying its :class:`~benchmark.harness.effectbench.TaskSpec`, the
   pinned environment it targets, and whether it runs LIVE here or needs a
   container;
 - validates every task against the ``benchmark.environments`` registry (the
@@ -29,8 +29,8 @@
 from benchmark.effectbench.task_pack.frappe_tasks import FRAPPE_TASKS
 from benchmark.effectbench.task_pack.mockmed_tasks import MOCKMED_TASKS
 from benchmark.effectbench.task_pack.openemr_tasks import OPENEMR_TASKS
-from openadapt_flow.benchmark.effectbench import TaskSpec
-from openadapt_flow.benchmark.effectbench.schema import DivergenceCategory, Substrate
+from benchmark.harness.effectbench import TaskSpec
+from benchmark.harness.effectbench.schema import DivergenceCategory, Substrate
 
 MANIFEST_SCHEMA_VERSION = 1
 

packages/flow/benchmark/frappe_lending/README.md

@@ -30,7 +30,7 @@ trials per cell remains the publication bar. Raw per-run rows, environment
 fingerprints, application-specific adapter/oracle wiring, and the detailed cost
 ledger are retained in the private `OpenAdaptAI/openadapt-corpus` repository.
 The public generic cost-capped agent mechanism remains in
-`openadapt_flow/benchmark/agent_baseline.py`. See the
+`benchmark/harness/agent_baseline.py`. See the
 [aggregate report](../agent_arm_verticals/README.md).
 
 The accepted run used baseline SHA-256

packages/flow/openadapt_flow/benchmark/__init__.py → packages/flow/benchmark/harness/__init__.py

@@ -1,4 +1,8 @@
-"""Benchmark: compiled replay vs. a frontier computer-use agent.
+"""Benchmark harnesses: compiled replay vs. a frontier computer-use agent.
+
+This package lives in the source checkout, outside ``openadapt_flow``, so the
+wheel does not ship it. Import it as ``benchmark.harness`` from
+``packages/flow``.
 
 Submodules (import them directly; nothing heavy is imported here):
 

packages/flow/openadapt_flow/benchmark/desktop_benchmark.py → packages/flow/benchmark/harness/desktop_benchmark.py

@@ -745,7 +745,7 @@ def _identity_transfer_verdict(results: dict) -> str:
 
 def render_chart(results: dict, path: Path) -> None:
     """Stacked outcome bars per arm (success / wrong-action / halt)."""
-    from openadapt_flow.benchmark.chart_fonts import configure_bundled_font
+    from benchmark.harness.chart_fonts import configure_bundled_font
 
     plt = configure_bundled_font()
     arms = list(results["arms"])
@@ -776,7 +776,7 @@ def write_outputs(results: dict, out_dir: str | Path) -> None:
     out_dir.mkdir(parents=True, exist_ok=True)
     (out_dir / "results.json").write_text(json.dumps(results, indent=2) + "\n")
     (out_dir / "BENCHMARK.md").write_text(render_markdown(results))
-    from openadapt_flow.benchmark.chart_fonts import safe_render
+    from benchmark.harness.chart_fonts import safe_render
 
     safe_render(render_chart, results, out_dir / "outcomes.png")