Skip to content

fix(evals): recognize compound delivery headings without weakening facts - #170

Merged
vriesd merged 3 commits into
mainfrom
fix/delivery-claim-headings
Oct 5, 2026
Merged

vriesd merged 3 commits into
mainfrom
fix/delivery-claim-headings

Conversation

@vriesd

@vriesd vriesd commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Why

The fresh 9.6 qualification stopped after the grader rejected an otherwise truthful completed handoff. Its bold heading combined "Completed and archived" with "1 of 1 features complete". Whole-claim matching extracted neither fact. The native workflow, current-source gate and independent review succeeded.

Scope

  • Recognize a scoped compound closure and numeric progress heading using the existing scalar parsers.
  • Reject invalid clauses, contradictions, negations and unsupported tails, even when valid canonical facts appear elsewhere.
  • Preserve historical scope, Goal content, registered command arguments, authority checks, disclosures and native evidence requirements.
  • Retain the unchanged measured answer as a regression fixture with coherent test evidence.

Tradeoffs

This is a bounded structural recognition fix. It does not introduce an LLM judge or accept arbitrary narrative formats. Separate field labels remain valid. Labelled compound forms are outside this repair.

Blast Radius

Evaluation tooling and tests only. Runtime, guides, package inputs, catalog, thresholds and original paid reports remain unchanged. The original campaign remains failed at 57/58, with fourteen primary cells unexecuted. The maintained runner does not support resuming or stitching this stopped campaign into qualification.

Verification

  • Parent red run reproduced two passing controls and three failures before the fix.
  • Focused and independent verification passed 98 tests with 291 assertions, including conflicting facts, omitted facts, historical headings and registered-command controls.
  • The actual unchanged retained native input reproduces its original two errors under the frozen grader. The fixed grader accepts it. A separate development diagnostic accepts all 58 retained inputs. This is not qualification and does not rewrite any original outcome.
  • Independent comment audit found no introduced comments, suppressions or dead wrappers.
  • Full clean-commit push preflight passed at 4225687. The suite passed 2,265 tests, skipped 20 optional live checks, and had no failures.
  • Packed artifact SHA256 is unchanged: aa0ad68901d6c352a7956c327062328059ff6d7bbd234d933d0bec04dda781f3, exactly matching the actual campaign artifact. Source and evaluator identities change; this byte match does not qualify the stopped report.
  • All thirteen gated replays match.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-05T19:18:30.794356Z 4225687 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vriesd

vriesd commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Independent verification PASS at 4225687. The reviewed parser is unchanged. Final fixture formatting preserves the complete parsed fixture and original retained answer. Fresh delivery checks pass 98 tests with 291 assertions. The checkout is clean. Negation, contradiction, historical, Goal and command boundaries remain covered. This verifier fix establishes no paid-run qualification. No model calls or remote writes occurred.

@vriesd
vriesd merged commit 7aa6c94 into main Oct 5, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants