fix(evals): recognize abbreviated assurance check claims - #171
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Independent shipping verification PASS at e33a717 against 7aa6c94. The reviewed source SHA-256 remains 36a8ac2d780308dc5186bfff9fea55de9710d29f2ed65411d446013476477c36; tests and retained fixture are unchanged from the independently verified test-first commit. The previous 126 passing tests and 387 assertions apply without drift. Checkout and diff are clean. Actual retained answer and original input hashes remain unchanged. Full consumption, native conclusion/count/status, authority, source and disclosure checks remain intact. This verifier-only result is not qualification. No paid calls, source edits or remote writes occurred. |
Why
A live 9.6qualification rejected a truthful handoff that said "Assurance: completion supported, with all 4checks satisfied." The native close response recorded completion-supported and four satisfied checks. Adding only the adjective "assurance" made the unchanged full-input grader pass.
Scope
Make "assurance" optional before "checks" inside the two existing qualified-count branches of
assuranceValue. Preserve the existing typed conclusion and count, capture groups, full-clause anchoring, and native count/status comparison. Add an exact retained-answer fixture and positive/adversarial regression coverage. Tests land before the fix.Tradeoffs
Recognize the abbreviated noun only inside the existing assurance-value context. Standalone "all 4checks" stays outside this change. No general natural-language judge or new contextual state is introduced.
Blast Radius
This changes the delivery presentation grader. Runtime, guides, release thresholds and case catalog remain unchanged. The original stopped campaign stays officially 57/58 with 14 primary cells unexecuted. Original reports and outcomes are preserved; offline diagnostics do not qualify the release.
Verification