Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
bc98a3a
Engine generality audit; move the benchmark harnesses out of the wheel
abrichr Nov 3, 2026
22de5f6
Generality audit: link CONTRIBUTING without the guard PR's anchor
abrichr Nov 3, 2026
d10bcc3
CI: fail when the engine names a specific application
abrichr Nov 3, 2026
83300a0
Register the generality denylist and baseline as public artifacts
abrichr Nov 3, 2026
31e5a27
Verifier: no app's REST paths or key field as engine defaults
abrichr Nov 3, 2026
2c78f01
Verifier: FHIR layout, the no-record value and server facts come from…
abrichr Nov 3, 2026
764339b
Engine generality audit; move the benchmark harnesses out of the wheel
abrichr Nov 3, 2026
7d884c0
Generality audit: link CONTRIBUTING without the guard PR's anchor
abrichr Nov 3, 2026
c6dec0b
Merge the rebased audit branch; keep a new comment app-neutral
abrichr Nov 3, 2026
e97d0df
Verifier: keep issued identities and check digests; --effects-records…
abrichr Nov 3, 2026
cc145da
CONTRIBUTING: state the app-knowledge rule positively
abrichr Nov 3, 2026
2bdcfa2
Move test fixtures and demo harnesses out of the openadapt-flow wheel
abrichr Nov 3, 2026
7083212
Merge branch 'claude/generality-audit-83a5d8' into claude/generality-…
abrichr Nov 3, 2026
d85e82f
Verifier: contract hashes keep their old values; name the key field
abrichr Nov 3, 2026
0f03dd5
Merge remote-tracking branch 'origin/main' into claude/generality-res…
abrichr Nov 3, 2026
89fb0a6
Mine the 'key' convention as the default idempotency field; give the …
abrichr Nov 4, 2026
ce188ec
Merge remote-tracking branch 'origin/main' into claude/generality-res…
abrichr Nov 4, 2026
ae25f67
Merge remote-tracking branch 'origin/main' into claude/generality-mov…
abrichr Nov 4, 2026
69bbcb2
Merge remote-tracking branch 'origin/main' into claude/engine-general…
abrichr Nov 4, 2026
3ccd99c
Guard: reword new app-name examples from main; lower the baseline to 365
abrichr Nov 4, 2026
1ee83f9
Merge remote-tracking branch 'origin/main' into claude/generality-mov…
abrichr Nov 4, 2026
6fa3e1e
Merge remote-tracking branch 'origin/main' into claude/engine-general…
abrichr Nov 4, 2026
9f2a20e
Merge remote-tracking branch 'origin/main' into claude/generality-res…
abrichr Nov 4, 2026
e47cdf1
Point the new teach-authorization test at tests._teach_scope
abrichr Nov 4, 2026
10e3734
Guard: take main's harness import order; reword a new MockMed example
abrichr Nov 4, 2026
5e3742e
Guard: count app names inside identifiers; restore main's lint, packa…
abrichr Nov 4, 2026
8a01ffe
Undo an unrelated reformat of scripts/
abrichr Nov 4, 2026
23b46d6
Restore #1741's packaging excludes, lazy-import guard and sys.path.ap…
abrichr Nov 4, 2026
88eaf68
Guard: ordinary words like text_edit and mock_media pass; baseline 425
abrichr Nov 4, 2026
c0814f8
fix(flow): preserve installed environments through harness relocation
abrichr Nov 4, 2026
e7051eb
Merge remote-tracking branch 'origin/claude/engine-generality-guard' …
abrichr Nov 4, 2026
cdf9c99
fix(flow): lower generality baseline after harness relocation
abrichr Nov 4, 2026
f7a173b
Merge remote-tracking branch 'origin/main' into agent/97
abrichr Nov 4, 2026
4d917b6
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 4, 2026
83b0f0d
Merge commit '4d917b6e3' into agent/97
abrichr Nov 4, 2026
1304436
test(flow): declare read paths in synthetic verifier fixtures
abrichr Nov 4, 2026
7c9ab66
Merge remote-tracking branch 'origin/main' into agent/97
abrichr Nov 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/agent/tests/test_approval_flow.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ def _bridge(
"web",
"--effects-kind",
"rest",
"--effects-records-path",
"/api/db",
"--effects-base-url",
mockmed["base_url"],
),
Expand Down
4 changes: 3 additions & 1 deletion packages/desktop/tests/test_engine/test_check_setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,9 @@ def test_fixed_values_are_the_recorded_ones_and_follow_the_sealed_version(tmp_pa
with pytest.raises(CheckSetupError, match="Unknown value"):
answer_values(dispatcher, "wf1", ["member_id"], answered_by="person")

store.write_record_check_config("wf1", {"kind": "rest", "base_url": BASE})
store.write_record_check_config(
"wf1", {"kind": "rest", "records_path": "/api/db", "base_url": BASE}
)
saved = store.load("wf1")
saved["record_check_config"] = str(store.root / "wf1.record-check.yaml")
store.save("wf1", saved)
Expand Down
4 changes: 4 additions & 0 deletions packages/flow/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@

### Fixes

- A REST read names its own records path: `effects.kind: rest`, a reward recipe of kind `rest`, `RestRecordVerifier`, `RestCompensator` and the fault gate's in-memory copy require `records_path` (and `records_key`, `delete_path_template`) instead of defaulting to the demo app's `/api/db`; `run --effects-kind rest` takes `--effects-records-path`. Compile-time REST binding admission (`compile_recording(admit_api_bindings=True, admission_fixture=...)`) takes the app's read path, write path, server-assigned fields and idempotency key field from an `AdmissionFixture`, and proposes nothing when no write was captured and no write path is named. Effect mining and `scaffold-verifier` mine an idempotency key only from a field the caller names. Verifier identities and fault-gate check digests of existing configs are unchanged.

- A FHIR read names its resource and field map: `effects.kind: fhir`, a reward recipe of kind `fhir` and `FhirEffectVerifier` require `resource_type` and `field_paths` instead of defaulting to one app's note-as-Observation layout (`DEFAULT_OBSERVATION_PATHS` stays importable for callers whose app stores notes that way). An app profile's `no_record_values` lists the record-field values its audit entries use for "no record" (OpenEMR's profile: `"0"`; profile version `openemr-frame-profile/2026-11-03.1`, so a config that pins the old profile digest is refused until it pins the new one); the frame and rollback no longer treat `0` as "no record" in every app. `generate ensure-contract` reads only the generic server-fact names from `facts.json`; the Medplum probe's translation moved to `benchmark/fhir_medplum/readpath/facts.py`.

- A qualification case can run on `--backend windows`. `WindowsBackend` now answers the qualification environment observer from one `/context/identity` call: the foreground executable's name, the SHA-256 of that executable file, the live logon-session digest, and a digest of the machine id and exact OS build. Before, `qualify run-case` on Windows refused with "qualification backend cannot observe the complete environment". The digest is withheld when the file changed after the process started (an updater replacing a running program), and it describes the host executable for an app run by an interpreter, browser or shared runtime. Guarded input now also binds the digest and the environment, so a same-named program from another path cannot take the foreground between the check and the input. The agent's session digest now accepts a connected RDP or Citrix session, not only the console session, so the agent can run inside a remote user session. An agent without the new `context_environment_v1` fields still refuses ([#1696](https://github.com/OpenAdaptAI/OpenAdapt/issues/1696)).

- A covered-band dismiss step takes its dialog box from the window the backend proved covers the band, before any pixel search, so a native message box that the accessibility tree doesn't label as a dialog can be dismissed. Without a proved window the step is refused as before.
Expand Down
4 changes: 2 additions & 2 deletions packages/flow/benchmark/contract_autogen/blind.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,14 @@
import hashlib
import json
import os
import shutil
import sys
import time
from pathlib import Path
from typing import Any, Optional

from benchmark.fhir_medplum.fixture import FixtureError, MedplumFixture, _http
from benchmark.fhir_medplum.readpath import suite as runner
from benchmark.fhir_medplum.readpath.facts import generic_facts

HERE = Path(__file__).resolve().parent
PROBE = HERE.parent / "fhir_medplum" / "readpath" / "probe.json"
Expand Down Expand Up @@ -101,7 +101,7 @@ def cmd_teach(args: argparse.Namespace) -> int:
out / "search_parameters.json",
read_all(f"{fx.fhir_url}/SearchParameter?base={rtype}&_count=200", reader),
)
shutil.copyfile(PROBE, out / "facts.json")
_dump(out / "facts.json", generic_facts(json.loads(PROBE.read_text())))
_dump(
out / "teach.json",
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

import json
from pathlib import Path
from benchmark.fhir_medplum.readpath.facts import generic_facts

HERE = Path(__file__).resolve().parent
SRC = HERE.parent / "fhir_medplum" / "readpath"
Expand Down Expand Up @@ -123,7 +124,14 @@ def main() -> None:
(OUT / "search_parameters.json").write_text(
json.dumps(params_bundle, indent=1) + "\n"
)
(OUT / "facts.json").write_text((SRC / "probe.json").read_text())
(OUT / "facts.json").write_text(
json.dumps(
generic_facts(json.loads((SRC / "probe.json").read_text())),
indent=2,
sort_keys=True,
)
+ "\n"
)
teach = {
"resource_type": "Patient",
"deployment": {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,87 +1,8 @@
{
"audit_lag_s": {
"C": {
"max": 0.043,
"median": 0.036
},
"D": {
"max": 0.049,
"median": 0.036
},
"U": {
"max": 0.043,
"median": 0.031
}
},
"audit_names_the_user_profile": true,
"audit_seen_on_an_immediate_read": [
"C by bot",
"C by bot",
"D by bot",
"U by other"
],
"audit_writes_are_logged": false,
"bot_admin_endpoint": "HTTP 403",
"bot_is_project_admin": false,
"bot_is_super_admin": false,
"bundle_and_graphql_writes_logged": [
"C Observation by bot",
"C Observation by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot"
],
"date_header": true,
"delete_in_history": true,
"meta_author_returned": false,
"policy": "AccessPolicy/f348be14-df4d-47cc-a49b-1f45e08dd554",
"restricted_bot_audit_read_only": true,
"restricted_bot_audit_writes": {
"create": "HTTP 403",
"delete": "HTTP 403",
"update": "HTTP 403"
},
"restricted_bot_can_write": {
"Observation": "allowed",
"Patient": "allowed",
"Task": "allowed"
},
"sweep_sees_delete": false,
"type_history": "HTTP 404",
"unrestricted_bot_audit_writes": {
"create": null,
"delete": null,
"update": null
}
"agent_cannot_write_audit": true,
"audit_lag_max_s": 0.049,
"audit_logs_writes": true,
"audit_names_actor": true,
"history_lists_versions": true,
"meta_author_returned": false
}
Original file line number Diff line number Diff line change
@@ -1,87 +1,8 @@
{
"audit_lag_s": {
"C": {
"max": 0.043,
"median": 0.036
},
"D": {
"max": 0.049,
"median": 0.036
},
"U": {
"max": 0.043,
"median": 0.031
}
},
"audit_names_the_user_profile": true,
"audit_seen_on_an_immediate_read": [
"C by bot",
"C by bot",
"D by bot",
"U by other"
],
"audit_writes_are_logged": false,
"bot_admin_endpoint": "HTTP 403",
"bot_is_project_admin": false,
"bot_is_super_admin": false,
"bundle_and_graphql_writes_logged": [
"C Observation by bot",
"C Observation by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"C Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"D Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot",
"U Patient by bot"
],
"date_header": true,
"delete_in_history": true,
"meta_author_returned": false,
"policy": "AccessPolicy/f348be14-df4d-47cc-a49b-1f45e08dd554",
"restricted_bot_audit_read_only": true,
"restricted_bot_audit_writes": {
"create": "HTTP 403",
"delete": "HTTP 403",
"update": "HTTP 403"
},
"restricted_bot_can_write": {
"Observation": "allowed",
"Patient": "allowed",
"Task": "allowed"
},
"sweep_sees_delete": false,
"type_history": "HTTP 404",
"unrestricted_bot_audit_writes": {
"create": null,
"delete": null,
"update": null
}
"agent_cannot_write_audit": true,
"audit_lag_max_s": 0.049,
"audit_logs_writes": true,
"audit_names_actor": true,
"history_lists_versions": true,
"meta_author_returned": false
}
47 changes: 47 additions & 0 deletions packages/flow/benchmark/fhir_medplum/readpath/facts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
"""Translate this probe's output into the generic server facts.

``openadapt_flow.generate.ensure_contract`` reads only the generic names in
``FACTS`` from a teach directory's ``facts.json``; what each Medplum probe
measured, and under which name, stays here with the probe.

PYTHONPATH=. python -m benchmark.fhir_medplum.readpath.facts probe.json > facts.json
"""

from __future__ import annotations

import json
import sys
from typing import Any, Mapping

from openadapt_flow.generate.ensure_contract import FACTS


def generic_facts(probe: Mapping[str, Any]) -> dict[str, Any]:
"""The generic facts this probe's output supports, by their generic names."""

if all(k in probe for k in FACTS):
return {k: probe[k] for k in FACTS}
out: dict[str, Any] = {}
logged = probe.get("bundle_and_graphql_writes_logged")
if isinstance(logged, list):
out["audit_logs_writes"] = len(logged) > 0
for name, key in (
("audit_names_actor", "audit_names_the_user_profile"),
("agent_cannot_write_audit", "restricted_bot_audit_read_only"),
("history_lists_versions", "delete_in_history"),
("meta_author_returned", "meta_author_returned"),
):
if key in probe:
out[name] = probe[key]
lags = probe.get("audit_lag_s")
if isinstance(lags, Mapping) and lags:
out["audit_lag_max_s"] = max(
float(v.get("max", 0)) for v in lags.values() if isinstance(v, Mapping)
)
return out


if __name__ == "__main__":
with open(sys.argv[1], encoding="utf-8") as f:
json.dump(generic_facts(json.load(f)), sys.stdout, indent=2, sort_keys=True)
sys.stdout.write("\n")
Loading
Loading