Repository navigation
fix(flow): preserve supplied assurance limitations in handoffs #179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
5e228c7
test(flow): expose disclosure omission and article grading mismatch
vriesdcrv bc5eabf
fix(flow): preserve supplied assurance limitation statements
vriesdcrv 4eb856b
test(flow): cover disclosure copying across handoff skills
vriesdcrv f25987c
fix(flow): preserve disclosures across all handoff skills
vriesdcrv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| import { expect, test } from "bun:test"; | ||
| import { missingAssuranceDisclosures } from "../evals/delivery-presentation.js"; | ||
| import { deliveryIssues } from "../evals/delivery-scenario-checks.js"; | ||
| import { getFlowGuidance } from "../src/guidance/catalog.js"; | ||
| import { autoQualifiedOutcome } from "./fixtures/auto-qualified-outcome.js"; | ||
| import saved from "./fixtures/delivery-flow-zero-count-answer.json" with { | ||
| type: "json", | ||
| }; | ||
|
|
||
| const actual = | ||
| "Implemented in `src/parser.mjs`:\n\n```js\nexport function parse(input) { return input === null ? '' : input.trim(); }\n```\n\n- `parse(null)` returns `''`; strings retain trimming behavior.\n- Only `src/parser.mjs` changed. `scripts/verify.mjs` remains unchanged.\n- Validation: `node scripts/verify.mjs` passed with exit code **0**.\n- Independent review passed with **no findings**.\n\n### Flow completion\nGoal: Make parse(input) safely handle null and preserve trimmed strings, changing only src/parser.mjs and keeping node scripts/verify.mjs and its script unchanged.\n\nSession completed and archived: **1 of 1 features complete**, no unfinished features or blockers. Assurance: **completion supported**, with all four assurance checks satisfied. External action authority: not granted.\n\nAssurance limitations:\n- Artifact paths and the canonical gate are caller declarations; Flow validates binding, not completeness or fitness.\n- Scope, requirement coverage, test adequacy, and review substance remain model judgments.\n- Freshness holds when review is accepted; the archive does not attest the current workspace."; | ||
| const partial = | ||
| "Scope, requirement coverage, test adequacy, and review substance remain model judgments."; | ||
| const complete = | ||
| "Goal alignment, scope discipline, evidence completeness, requirement coverage, test adequacy, and review substance remain model judgments."; | ||
| const freshness = | ||
| "Freshness holds when review is accepted; the archive does not attest the current workspace."; | ||
| const canonicalFreshness = freshness.replace("the archive", "an archive"); | ||
| const coverageIssue = | ||
| "Missing assurance disclosure: coverage and review substance remain model judgments."; | ||
| const freshnessIssue = | ||
| "Missing assurance disclosure: accepted evidence does not attest later workspace changes."; | ||
| const conflict = "Assurance disclosures contain a conflicting current claim."; | ||
| const expected = { | ||
| closure: "completed" as const, | ||
| presentation: "summary" as const, | ||
| gate: "node scripts/verify.mjs", | ||
| allowedPaths: ["src/parser.mjs"], | ||
| }; | ||
| function fixture(finalText: string) { | ||
| const input = autoQualifiedOutcome("single", { | ||
| goal: saved.goal, | ||
| featureId: saved.featureId, | ||
| }); | ||
| const close = input.allCalls.find( | ||
| (call) => call.tool === "flow_session_close", | ||
| ); | ||
| if (!close || typeof close.output !== "object" || close.output === null) | ||
| throw new Error("Missing close fixture."); | ||
| const data = (close.output as { workflowData: { delivery: unknown } }) | ||
| .workflowData; | ||
| data.delivery = structuredClone(saved.delivery); | ||
| return { ...input, finalText }; | ||
| } | ||
|
|
||
| test("unchanged retained handoff still rejects its genuinely omitted model judgments", () => { | ||
| expect(deliveryIssues(fixture(actual), expected)).toEqual([coverageIssue]); | ||
| }); | ||
| test("complete judgment topics and definite archive compose in whole checker", () => { | ||
| expect( | ||
| deliveryIssues(fixture(actual.replace(partial, complete)), expected), | ||
| ).toEqual([]); | ||
| }); | ||
| test("canonical archive article still accepts complete handoff", () => { | ||
| expect( | ||
| deliveryIssues( | ||
| fixture( | ||
| actual | ||
| .replace(partial, complete) | ||
| .replace(freshness, canonicalFreshness), | ||
| ), | ||
| expected, | ||
| ), | ||
| ).toEqual([]); | ||
| }); | ||
| for (const topic of ["Goal alignment, ", "evidence completeness, "]) { | ||
| test(`Goal line cannot replace the omitted disclosure ${topic}`, () => { | ||
| expect( | ||
| missingAssuranceDisclosures( | ||
| actual | ||
| .replace(partial, complete.replace(topic, "")) | ||
| .replace(freshness, canonicalFreshness), | ||
| ), | ||
| ).toEqual([coverageIssue]); | ||
| }); | ||
| } | ||
| for (const invalid of [ | ||
| "Freshness holds when review is accepted; the archive attests the current workspace.", | ||
| "Freshness will hold when review is accepted; the archive does not attest the current workspace.", | ||
| "Freshness holds when review is accepted; the archive does not attest the current workspace if validation passes.", | ||
| "Freshness holds when review is accepted; the archive does not attest the current workspace or authorize release.", | ||
| ]) { | ||
| test(`freshness remains closed and truthful ${invalid}`, () => { | ||
| expect( | ||
| missingAssuranceDisclosures( | ||
| actual.replace(partial, complete).replace(freshness, invalid), | ||
| ), | ||
| ).toContain(freshnessIssue); | ||
| }); | ||
| } | ||
| for (const ignored of [ | ||
| `Example: "${freshness}"`, | ||
| `## Historical handoff\n${freshness}`, | ||
| `Goal: ${freshness}`, | ||
| ]) { | ||
| test(`freshness cannot be supplied in ignored scope ${ignored}`, () => { | ||
| expect( | ||
| missingAssuranceDisclosures( | ||
| actual.replace(partial, complete).replace(`- ${freshness}`, ignored), | ||
| ), | ||
| ).toContain(freshnessIssue); | ||
| }); | ||
| } | ||
| for (const statement of [canonicalFreshness, freshness]) { | ||
| test(`Markdown blockquote formatting preserves visible disclosure ${statement}`, () => { | ||
| expect( | ||
| missingAssuranceDisclosures( | ||
| actual | ||
| .replace(partial, complete) | ||
| .replace(`- ${freshness}`, `> ${statement}`), | ||
| ), | ||
| ).toEqual([]); | ||
| }); | ||
| } | ||
| test("valid definite archive disclosure does not hide contradictory current claim", () => { | ||
| expect( | ||
| missingAssuranceDisclosures( | ||
| `${actual.replace(partial, complete)}\nThe archive attests the current workspace.`, | ||
| ), | ||
| ).toEqual([conflict]); | ||
| }); | ||
| for (const id of ["flow", "flow-plan", "flow-run"] as const) { | ||
| test(`delivered ${id} guidance preserves each supplied limitation unchanged`, () => { | ||
| const guidance = getFlowGuidance(id).content.replace(/\s+/g, " "); | ||
| expect(guidance).toContain( | ||
| "Copy each supplied `workflowData.delivery.assurance.limitations` statement unchanged, including its full topic list.", | ||
| ); | ||
| expect(guidance).toContain("`workflowData.delivery.summary.lines`"); | ||
| expect(guidance).toMatch( | ||
| /For requested full detail or a missing summary, (?:use|report) the retained close response's `report`\./, | ||
| ); | ||
| }); | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.