Skip to content

fix(evals): compose factual delivery claims - #177

Merged
vriesd merged 6 commits into
mainfrom
fix/delivery-handoff-contract
Oct 7, 2026
Merged

vriesd merged 6 commits into
mainfrom
fix/delivery-handoff-contract

Conversation

@vriesd

@vriesd vriesd commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Why

Fresh qualification stopped on a truthful completed handoff. The grader conflated sentence layout with closure, progress and command-integrity facts.

What changed

  • Compose closure and progress records from bounded claim pieces instead of separate sentence patterns.
  • Parse integrity subjects independently while preserving accepted-pass and immutable-script evidence checks.
  • Check blocked-feature counts against current runs, separately from review finding severity.

Scope

Evaluator code and regression tests only. Runtime prompts, package behavior and qualification thresholds remain unchanged.

Tradeoffs

The grammar supports a bounded vocabulary. It cannot interpret arbitrary paraphrases. A native text rewrite was rejected because the host hook cannot identify only the terminal response.

Blast Radius

This changes how delivery assertions are graded. Unknown tails and contradictory claims still fail. Ordinary implementation prose remains unrelated. Contracted state denials remain critical assertions. Goal and registered command contents remain opaque.

Verification

  • Original retained input fails before the repair and passes unchanged afterward. All 336 gradable retained attempts pass offline. One historical host failure remains excluded. All seven original report hashes remain unchanged.
  • Red tests reproduce the original gaps and an unrelated-ratio regression. All 480 focused delivery tests pass after the repair.
  • Full contribution preflight passed 2,731 tests with 20 existing skips and zero failures. Independent exact-head review returned PASS+NOTES. This repair is not successful live qualification. The stopped campaign remains unqualified.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T12:36:56.125125Z 1fdf43e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vriesd

vriesd commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Independent source-author-distinct review returned PASS+NOTES for head 1fdf43e against base dc665c3. Stable patch ID is def39e441ab32f144287dc27198dc4ac8748d126.

The reviewer exercised the unchanged native failing input through the whole grader. Baseline produced three issues and this head produced zero. All 336 gradable inputs from seven retained campaigns pass offline. The historical host exclusion and all original report hashes remain unchanged. Independent negative controls preserve accepted-pass proof, raw gate identity, verifier immutability, authority, opaque scopes and latest blocked-feature semantics.

Parent full contribution preflight passed with 2,673 tests, 20 existing skips and zero failures. Independent focused checks passed 422 delivery tests and typecheck. Comment audit has zero findings.

The supported vocabulary remains bounded. This repair has not passed fresh live qualification and does not establish a success probability. The stopped campaign remains unqualified. Review used an independent inherited Sol role, without model-family diversity.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1fdf43e645

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread evals/delivery-presentation.ts Outdated
@vriesd

vriesd commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

The earlier 1fdf43e verdict was withdrawn after the ordinary-prose regression. This exact-head verdict supersedes it after independent renewal.

Independent source-author-distinct review returned PASS+NOTES for head a4a0b01 against base dc665c3. Stable patch ID is 3e0bc72004d66eb98d1dbef7a0d93f019cc0fbc5.

The reviewer exercised the unchanged native failing input through the whole grader. Baseline produced three issues and this head produced zero. All 336 gradable inputs from seven retained campaigns pass offline. The historical host exclusion and all original report hashes remain unchanged. Independent negative controls preserve accepted-pass proof, raw gate identity, verifier immutability, authority, opaque scopes and latest blocked-feature semantics.

Parent full contribution preflight passed with 2,731 tests, 20 existing skips and zero failures. Independent focused checks passed 480 delivery tests and typecheck. Comment audit has zero findings.

The supported vocabulary remains bounded. This repair has not passed fresh live qualification and does not establish a success probability. The stopped campaign remains unqualified. Review used an independent inherited Sol role, without model-family diversity.

@vriesd
vriesd merged commit 22ece33 into main Oct 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants