Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
bc98a3a
Engine generality audit; move the benchmark harnesses out of the wheel
abrichr Nov 3, 2026
22de5f6
Generality audit: link CONTRIBUTING without the guard PR's anchor
abrichr Nov 3, 2026
d10bcc3
CI: fail when the engine names a specific application
abrichr Nov 3, 2026
83300a0
Register the generality denylist and baseline as public artifacts
abrichr Nov 3, 2026
764339b
Engine generality audit; move the benchmark harnesses out of the wheel
abrichr Nov 3, 2026
7d884c0
Generality audit: link CONTRIBUTING without the guard PR's anchor
abrichr Nov 3, 2026
c6dec0b
Merge the rebased audit branch; keep a new comment app-neutral
abrichr Nov 3, 2026
cc145da
CONTRIBUTING: state the app-knowledge rule positively
abrichr Nov 3, 2026
2bdcfa2
Move test fixtures and demo harnesses out of the openadapt-flow wheel
abrichr Nov 3, 2026
7083212
Merge branch 'claude/generality-audit-83a5d8' into claude/generality-…
abrichr Nov 3, 2026
ae25f67
Merge remote-tracking branch 'origin/main' into claude/generality-mov…
abrichr Nov 4, 2026
69bbcb2
Merge remote-tracking branch 'origin/main' into claude/engine-general…
abrichr Nov 4, 2026
3ccd99c
Guard: reword new app-name examples from main; lower the baseline to 365
abrichr Nov 4, 2026
1ee83f9
Merge remote-tracking branch 'origin/main' into claude/generality-mov…
abrichr Nov 4, 2026
6fa3e1e
Merge remote-tracking branch 'origin/main' into claude/engine-general…
abrichr Nov 4, 2026
e47cdf1
Point the new teach-authorization test at tests._teach_scope
abrichr Nov 4, 2026
10e3734
Guard: take main's harness import order; reword a new MockMed example
abrichr Nov 4, 2026
5e3742e
Guard: count app names inside identifiers; restore main's lint, packa…
abrichr Nov 4, 2026
8a01ffe
Undo an unrelated reformat of scripts/
abrichr Nov 4, 2026
23b46d6
Restore #1741's packaging excludes, lazy-import guard and sys.path.ap…
abrichr Nov 4, 2026
88eaf68
Guard: ordinary words like text_edit and mock_media pass; baseline 425
abrichr Nov 4, 2026
c0814f8
fix(flow): preserve installed environments through harness relocation
abrichr Nov 4, 2026
e7051eb
Merge remote-tracking branch 'origin/claude/engine-generality-guard' …
abrichr Nov 4, 2026
cdf9c99
fix(flow): lower generality baseline after harness relocation
abrichr Nov 4, 2026
f7a173b
Merge remote-tracking branch 'origin/main' into agent/97
abrichr Nov 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,10 @@ jobs:
run: python scripts/check_release_consistency.py
- name: Ruff lint
run: ruff check openadapt_flow benchmark/harness
- name: No app-specific code in the engine
# App knowledge lives in profiles, contracts and fixtures (CONTRIBUTING:
# "App knowledge lives in data"); the denylist is scripts/generality.toml.
run: python scripts/check_engine_generality.py
- name: Install uv (for the oldest-Python syntax check)
uses: astral-sh/setup-uv@18d369b23da25a81e9458561248e1aafff6a70fa # v10.3.1
- name: Syntax at the oldest supported Python (3.10)
Expand Down
23 changes: 23 additions & 0 deletions packages/flow/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Python 3.10–3.12 are supported and exercised in CI.
ruff check openadapt_flow # lint
ruff format --check openadapt_flow # format (drop --check to auto-apply)
mypy # type-check (config in pyproject.toml)
python scripts/check_engine_generality.py # no app names in the engine
pytest -q # tests
```

Expand Down Expand Up @@ -174,6 +175,28 @@ surface from permissively licensed package artifacts.
The release-consistency gate inspects the actual wheel and sdist. A source-tree
notice alone is not sufficient.

## App knowledge lives in data

The engine (`openadapt_flow/`) works the same way for every application. App
knowledge lives in data:

- its bookkeeping writes: an app profile in `openadapt_flow/app_profiles/`,
loaded by name;
- its endpoints, tables, keys and field paths: the generated contract or the
deployment config (`deploy.yaml`);
- tests and benchmarks for it: fixtures under `tests/` and harnesses under
`benchmark/`, outside the shipped package.

`scripts/check_engine_generality.py` runs in CI and fails when engine source
names an application or one of its identifiers from the denylist in
`scripts/generality.toml`. Add new applications and identifiers to that list
when you meet them. A comment that only gives an example may name an
application on a line marked `# example:`. Hits that predate the check are
counted in `scripts/generality_baseline.json`, which only shrinks: when you
remove one, lower it with `--update-baseline`. OpenAdapt's own demo apps
(MockMed) may be named only in the demo modules listed in the denylist. The
audit behind the rule is `docs/GENERALITY_AUDIT.md`.

## Safety-sensitive areas

The identity gate, the resolution ladder, and the postcondition/halt logic are
Expand Down
7 changes: 4 additions & 3 deletions packages/flow/docs/GENERALITY_AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,10 @@ Audited 2026-11-03 on `main` at `213f3d9fe`. Scope: everything under
The rule: the engine has no code written for one application. What one
application needs lives in data (`app_profiles/*.yaml`, generated contracts,
deployment configs) or in test fixtures and benchmark harnesses outside the
shipped package. The CI check that keeps it that way,
`scripts/check_engine_generality.py`, lands in the pull request that follows
this audit, together with the rule in [CONTRIBUTING.md](../CONTRIBUTING.md).
shipped package. A CI check (`scripts/check_engine_generality.py`) keeps it
that way; [CONTRIBUTING.md](../CONTRIBUTING.md) states the rule. It counts an
app name inside an identifier too (`_openemr_base`, `isOpenEMR`); its baseline
started at 425 hits and only shrinks.

## Method

Expand Down
6 changes: 3 additions & 3 deletions packages/flow/openadapt_flow/ir.py
Original file line number Diff line number Diff line change
Expand Up @@ -835,7 +835,7 @@ class WebFormWrite(BaseModel):

- ``ambient``: the page sent it on its own. It came from a script of
another document than the form, one that was loaded before the form
opened (for example OpenEMR's tab shell, which posts
opened (for example an app's tab shell that posts
``execute_background_services.php`` on a timer), and the form page's
source doesn't name it. No replaced step caused it, so the API step
neither sends nor waits for it; the live page goes on as it does under
Expand Down Expand Up @@ -886,7 +886,7 @@ class WebFormSpec(BaseModel):
#: Page script the browser runs on this save that the API step can't
#: evaluate by reading the form: the form's ``on*`` handlers, the ``on*``
#: handlers of the fields it sends, a page script listening for submit,
#: and validate.js rule kinds such as OpenEMR's ``pastDate``. Before each
#: and validate.js rule kinds such as a ``pastDate`` rule. Before each
#: save the runtime runs that script itself, in a scratch page of the live
#: session with every write blocked, on the run's values, and sends only
#: when the page would submit exactly the body the API step built. When
Expand All @@ -901,7 +901,7 @@ class WebFormSpec(BaseModel):
submitter: Optional[str] = None
submitter_requests: list[str] = Field(default_factory=list)
#: The control the person clicked to confirm the save when it isn't the
#: submitter (for example OpenEMR's ``#confirmCreate`` in the duplicate
#: submitter (for example a ``#confirmCreate`` button in a duplicate
#: check dialog). Running the page's script clicks it too, in whichever
#: frame has it, and the page must then submit the form.
confirm: Optional[str] = None
Expand Down
2 changes: 1 addition & 1 deletion packages/flow/openadapt_flow/repair/self_repair.py
Original file line number Diff line number Diff line change
Expand Up @@ -2937,7 +2937,7 @@ def __new__(cls, items: tuple, *, cover: dict, pinned: str) -> "_CoveredHalt":

#: How far the share of the band left uncovered may be from the share of the
#: recorded text read: the dialog's left edge must be where the read stops.
#: On the lab's Patient Notes notice the two are 0.79 and 0.81.
#: On one lab app's session notice the two were 0.79 and 0.81.
_CUT_TOLERANCE = 0.1


Expand Down
2 changes: 1 addition & 1 deletion packages/flow/openadapt_flow/report_redaction.py
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ def is_credential_name(name: str, *, header: bool = False) -> bool:

The name is split into parts on anything but letters and digits; a part
that is a token, CSRF/XSRF token or session id makes it one
(``csrf_token_form``, ``X-Frappe-CSRF-Token``, ``app_sessid``). For a
(``csrf_token_form``, ``X-CSRF-Token``, ``app_sessid``). For a
header (``header=True``) an ``authorization`` part does too
(``Espo-Authorization``); elsewhere that word names an approval, not a
credential. A parameter or header named so is redacted in every app and
Expand Down
2 changes: 1 addition & 1 deletion packages/flow/openadapt_flow/rl_env/environment.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ def _norm(value: str) -> str:
def _scalars(body: str) -> set[str]:
"""Every scalar value in a JSON or form body, as text.

A form field holding JSON (as Frappe sends ``doc=...``) is read through.
A form field holding JSON (such as ``doc=...``) is read through.
"""
from urllib.parse import parse_qsl

Expand Down
2 changes: 1 addition & 1 deletion packages/flow/openadapt_flow/rl_env/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ class AgentSpec(_Strict):
#: the oracle uses, reset routes, and anything else that would let the
#: agent read or rewrite the answer.
blocked_paths: list[str] = Field(default_factory=list)
#: The requests that carry the task's write (``"POST /api/encounter"``).
#: The requests that carry the task's write (``"POST /api/items"``).
#: An infeasible task pays a failure report only if the agent's browser
#: sent one whose body holds every value in the task's ``params``.
write_requests: list[str] = Field(default_factory=list)
Expand Down
6 changes: 3 additions & 3 deletions packages/flow/openadapt_flow/runtime/actuators/form_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
and submit a form. Most forms run script on the way: key and change
handlers on the fields, an ``onsubmit`` validator, a click handler on the
save button, a page listener for ``submit``, custom validate.js rules such
as OpenEMR's ``pastDate``. The API step can't read what that script does,
as a ``pastDate`` rule. The API step can't read what that script does,
so it runs it, before each save, the way the person's steps would:

1. open the form page in a scratch copy of the live session: a separate
Expand All @@ -14,8 +14,8 @@
2. while the page loads, let only same-origin GET requests through (no
other origin at all). Once it has loaded, block every request except the
exact scripts and stylesheets the page loaded, the reads the person's
click made in the recording (``submitter_requests``, such as OpenEMR's
duplicate search) and the static ``.js``/``.css`` files of a recorded
click made in the recording (``submitter_requests``, such as a
duplicate-record search) and the static ``.js``/``.css`` files of a recorded
read's own frame, so nothing the page does can write; a form submit or a
blocked channel in any frame counts as the page sending;
3. type the run's values into the fields the person typed or picked, with
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""A throwaway approver for tests, demos and evaluations.

A real teach authorization is signed by a person with their approver key
(:mod:`openadapt_flow.teach_authorization`). Tests, the MockMed demos and the
(:mod:`openadapt_flow.teach_authorization`). Tests, the demos and the
injection evaluation need one without a person at the keyboard, so
:class:`LocalAuthorizer` makes an Ed25519 approver key in memory, starts (or
joins) a real signed approver roster in a key directory, and signs statements
Expand Down
2 changes: 2 additions & 0 deletions packages/flow/public-artifacts.json
Original file line number Diff line number Diff line change
Expand Up @@ -615,6 +615,8 @@
"schemas/sanitization-ingest-v1.json",
"schemas/sanitized-artifact-approval-v1.json",
"schemas/sanitized-artifact-manifest-v1.json",
"scripts/generality.toml",
"scripts/generality_baseline.json",
"source-policy.public.json",
"tests/golden/tutorial_run.json"
]
Expand Down
189 changes: 189 additions & 0 deletions packages/flow/scripts/check_engine_generality.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
"""Fail when the engine names a specific application.

The engine (``openadapt_flow/``) has no code written for one application. What
one application needs lives in data: ``app_profiles/*.yaml``, generated
contracts, deployment configs, or test fixtures and benchmark harnesses outside
the package. ``generality.toml`` beside this script holds the denylist of
application names and app-specific identifiers, and the few modules allowed to
name OpenAdapt's own demo apps.

A comment line may name an application as an example when it carries an
explicit ``# example:`` marker. Every other hit fails, unless it is in the
baseline: the hits that were already there when the check went live
(``docs/GENERALITY_AUDIT.md``). The baseline only shrinks. A file whose count
for a term rises fails; a file whose count falls must have its baseline entry
lowered in the same change (``--update-baseline`` writes it).

Usage::

python scripts/check_engine_generality.py
python scripts/check_engine_generality.py --update-baseline
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from collections import Counter
from pathlib import Path

try:
import tomllib
except ModuleNotFoundError: # Python 3.10
import tomli as tomllib

HERE = Path(__file__).resolve().parent
FLOW = HERE.parent
ENGINE = FLOW / "openadapt_flow"
CONFIG = HERE / "generality.toml"
BASELINE = HERE / "generality_baseline.json"

#: Engine files that hold text: code and the data shipped with it.
SUFFIXES = {".py", ".js", ".html", ".css", ".json", ".yaml", ".yml", ".toml", ".txt"}

#: The marker opens the comment: ``# example: ...`` or ``#: example: ...``.
EXAMPLE_MARKER = re.compile(r"#:?\s*example:", re.IGNORECASE)

#: A lower-case letter or digit followed by a capital: where a camelCase word
#: splits, so ``isOpenEMR`` reads as ``is Open EMR`` for the name patterns.
_CAMEL_BOUNDARY = re.compile(r"(?<=[a-z0-9])(?=[A-Z])")


def load_config(path: Path = CONFIG) -> dict:
return tomllib.loads(path.read_text(encoding="utf-8"))


def _term_patterns(config: dict) -> dict[str, tuple[re.Pattern[str], bool]]:
"""``{term: (pattern, is_name)}``; names also match inside camelCase words."""
terms: dict[str, tuple[re.Pattern[str], bool]] = {}
for section in ("names", "identifiers"):
for term, pattern in config[section].items():
terms[term] = (re.compile(pattern, re.IGNORECASE), section == "names")
return terms


def _allowed(rel: str, term: str, config: dict) -> bool:
for prefix in config["allow"]["paths"]:
if rel == prefix or rel.startswith(prefix.rstrip("/") + "/"):
return True
for prefix, terms in config["allow"].get("terms_in", {}).items():
if (
rel == prefix or rel.startswith(prefix.rstrip("/") + "/")
) and term in terms:
return True
return False


def scan(
engine: Path = ENGINE, config: dict | None = None
) -> dict[str, dict[str, list[int]]]:
"""Return ``{relative path: {term: [line numbers]}}`` for every hit."""
config = config if config is not None else load_config()
terms = _term_patterns(config)
hits: dict[str, dict[str, list[int]]] = {}
for path in sorted(engine.rglob("*")):
if not path.is_file() or path.suffix not in SUFFIXES:
continue
if "__pycache__" in path.parts:
continue
rel = path.relative_to(engine).as_posix()
try:
text = path.read_text(encoding="utf-8")
except UnicodeDecodeError:
continue
for number, line in enumerate(text.splitlines(), start=1):
stripped = line.lstrip()
if EXAMPLE_MARKER.match(stripped):
continue
split = _CAMEL_BOUNDARY.sub(" ", line)
for term, (pattern, is_name) in terms.items():
found = pattern.search(line) or (is_name and pattern.search(split))
if found and not _allowed(rel, term, config):
hits.setdefault(rel, {}).setdefault(term, []).append(number)
return hits


def counts(hits: dict[str, dict[str, list[int]]]) -> dict[str, dict[str, int]]:
return {
rel: {term: len(lines) for term, lines in sorted(by_term.items())}
for rel, by_term in sorted(hits.items())
}


def compare(
hits: dict[str, dict[str, list[int]]], baseline: dict[str, dict[str, int]]
) -> tuple[list[str], list[str]]:
"""Return (new hits, stale baseline entries)."""
new: list[str] = []
stale: list[str] = []
seen = Counter()
for rel, by_term in hits.items():
for term, lines in by_term.items():
allowed = baseline.get(rel, {}).get(term, 0)
seen[(rel, term)] = len(lines)
if len(lines) > allowed:
where = ", ".join(str(n) for n in lines)
new.append(
f"openadapt_flow/{rel}: {term!r} on line(s) {where} "
f"({len(lines)} found, {allowed} in the baseline)"
)
for rel, by_term in baseline.items():
for term, allowed in by_term.items():
if seen[(rel, term)] < allowed:
stale.append(
f"openadapt_flow/{rel}: {term!r} baseline {allowed}, "
f"now {seen[(rel, term)]}"
)
return new, stale


def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
parser.add_argument(
"--update-baseline",
action="store_true",
help="Lower the baseline to the current hits (it never grows)",
)
args = parser.parse_args(argv)

hits = scan()
baseline = json.loads(BASELINE.read_text(encoding="utf-8"))
new, stale = compare(hits, baseline)

if args.update_baseline:
if new:
print("New hits cannot enter the baseline:", file=sys.stderr)
print("\n".join(f" {line}" for line in new), file=sys.stderr)
return 1
BASELINE.write_text(json.dumps(counts(hits), indent=2) + "\n", encoding="utf-8")
print(f"Baseline lowered: {BASELINE.relative_to(FLOW)}")
return 0

if new:
print(
"The engine names a specific application. Put what one app needs "
"in data (app_profiles/*.yaml, a generated contract, a deployment "
"config) or in a fixture outside openadapt_flow/; see "
"CONTRIBUTING.md, 'App knowledge lives in data'. A comment that "
"only gives an example may carry '# example:'.",
file=sys.stderr,
)
print("\n".join(f" {line}" for line in new), file=sys.stderr)
if stale:
print(
"The baseline is higher than the engine now; lower it with "
"`python scripts/check_engine_generality.py --update-baseline`:",
file=sys.stderr,
)
print("\n".join(f" {line}" for line in stale), file=sys.stderr)
if new or stale:
return 1
total = sum(n for by_term in baseline.values() for n in by_term.values())
print(f"Engine generality OK ({total} baseline hits left)")
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading