Repository navigation
Generator: per-request bookkeeping, row-scoped excuses, missing field sets (first-contact retest) #1677
Description
Findings from a held-out first-contact test of the verifier at 76cd4fd, on apps it had never seen. The evidence stays in the private study repository.
- Per-request bookkeeping stops every correct write. On three of four apps, every API call by the agent's user writes a column or row: a user's last-login time, a token's last-used time, or a first-login property row. The generator never proposes it, so every correct write was refuted. Fix: during setup, watch the app across a few read-only API calls as the agent's user, and propose those columns as bookkeeping on that user's own row only.
- A column excused for the target is excused on every row (fail-open). An accepted bookkeeping answer that excuses an assignment column on the asset being written also excused that column on a look-alike asset. A late write that checked out the look-alike passed the recheck. Fix: excuse a column only on the rows the write declares.
- Field sets with no contract. One contract per field set leaves many correct tasks with no contract, so they are stopped at begin. Fix: propose contracts with optional fields, or ask for the missing demonstrations up front.
- Coincidental joins and pins. The generator matched on ids that happened to be equal in both demonstrations, and pinned values seen in only one. Setup agents had to fix these by hand. Fix: refuse a join or pin the demonstrations can't tell apart, and ask instead.
- Person decisions. Setup needed 9 to 15 person decisions per app (the bar is 3), although the generator asked only 2 questions. Grants, demonstration choices and hand fixes made up the rest.
Activity
abrichr commentedon Nov 4, 2026
Claimed by Codex
abrichr commentedon Nov 4, 2026
Audit at current main
9f026dad3:generate/verify_contract.py:1951and:2004;test_generate_verify_contract.py:569covers a late write to a look-alike row._field_setatverify_contract.py:3974; the regression coverage starts attest_generate_verify_contract.py:653.test_generate_coincidences*.pycoverage; generate: check a copy as a copy even when its values read as the row's own key #1839 also fixed copies that resemble the row's own key.generate/zero_touch.pynow answers generator questions and admits only sandbox-validated kinds. That addresses generator answers, but does not establish the total person-decision count for the original setup study.Correction to this SQL-only audit: the separate
learnworkflow already watches checked requests, records only field names, and proposes approved actor-row excuses (openadapt_flow/learn.py,docs/LEARNED_EXCUSES.md). I missed that path when describing read-call observation as a missing input. #1729 identifies two concrete gaps in it: bracketed names and query names arriving during a check. I am working on that bounded follow-up. The original end-to-end total person-decision count, including grants and hand fixes, has not been established by this code audit.I am releasing this umbrella claim while its remaining end-to-end acceptance conditions are checked. No implementation or test assertion changed, and no new test run is claimed here.