diff --git a/docs/architecture.md b/docs/architecture.md index 0e3b686..eb85bc0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -108,7 +108,7 @@ left in it (adopted once, so a resumed session keeps counting turns and nothing | Session start | `sessionStarted` | Queues retention reports for an earlier session's edits and aborted reports for a killed session's injections (its owner process is gone); schedules engine setup, delivery and leftover distillation; answers the one-off engine notice, the only text besides a strategy that reaches the model. | | Prompt | `promptSubmitted` | A correction drops the session's candidate, then the last turn's reuses are reported again under `:correction`. The turn counter moves on and survives the session's end, so a resumed session never repeats a task id. One strategy, or a Recipe expanded into one, is recalled within what is left of the hook's budget and ends with `Apply them where they fit.` | | Tool result | `toolFinished`, `beforeEdit`, `takeWaitingStrategy`, `recallWhenStuck` | Notes each call: shell commands with their exit code (a run still going is settled by a later note, or counts as not run), edited paths, each file's hash before its first edit. Where the strategy waits for a tool result, the turn's first one takes it and books it, only while that turn is open; a turn with no tool call never gets it, and it is neither reported nor kept out of later recalls. Where a tool result's context reaches the model (`toolResultContext`, or a `tool-result` host), a failed test, build, lint or type-check recalls again, once per distinct failure and twice a turn at most, with a query built from its output and the turn's edits; a strategy that lands within the tool result's budget is added to that result and booked. | -| Turn end | `turnEnded` | Ends as `answer`, `aborted`, `refusal` or `error`. Every injection of the turn is reported (`applied: used`, tier `L1` when a check ran; tier `none` with `reported_by: agent` when the agent only reported its checks). A turn that answered, whose last check passed and that edited files or ran three commands, becomes the session's candidate. | +| Turn end | `turnEnded` | Ends as `answer`, `aborted`, `refusal` or `error`. Every injection of the turn is reported (`applied: used`, tier `L1` when the last check has a known result; tier `none` with `reported_by: agent` when the agent only reported its checks). A turn that answered, whose last check passed and that edited files or ran three commands, becomes the session's candidate. | | Session end | `sessionEnded` | Only writes: a turn still open ends, what is unreported goes out as aborted, the candidate is closed for background distillation. | **State** lives in one directory per host, every file named `-`. Writes go through an exclusive lock with @@ -242,7 +242,7 @@ scripts/ pack.mjs, publish-tag.sh | Report | `task_id` | `evidence` | |---|---|---| - | Turn end | `::` | `tier` `L1` when a check ran, else `none`; `applied: used`; `check` `{kind, passed, fail_then_pass}`; `aborted`; `stage: immediate` | + | Turn end | `::` | `tier` `L1` when the last check has a known result, else `none`; `applied: used`; `check` `{kind, passed, fail_then_pass}`; `aborted`; `stage: immediate` | | Next prompt corrects or approves | `…:correction` | `tier: L3`, `applied`, `stage: correction` | | Next session finds the edits kept or reverted | `…:retention` | `tier: L2`, `applied`, `stage: retention` | @@ -250,6 +250,13 @@ scripts/ pack.mjs, publish-tag.sh followed it, and a reused turn is distilled like any other, the engine's intake dropping a Gene it already has. `fail_then_pass` compares checks with output trimming (`| tail -9`, `2>&1`) removed, so the same check piped differently is still the same check. The outcome follows the last check; with none it follows how the turn ended. + A finished check with no native exit code is unknown (`failed: null`) unless its output contains failure + evidence or a supported Node/TAP summary with a positive pass count and zero failed tests. An unknown last + result keeps the report at tier `none`, creates no candidate and claims no saving; it never falls back to an + earlier passing check. For Codex runs without an exit code, text-only Jest, Vitest, pytest, Go and Cargo + successes stay unknown unless they emit a supported summary. This can reduce L1 evidence and candidate + generation in those projects. Unknown checks stay labelled unknown in the distillation prompt and provide no + execution trace exit code. A native exit code of zero still passes without requiring output. A tier `none` report counts as exposure only on the Hub and is not written to the local ledger. An older Proxy drops the new fields and the plugin runs unchanged. - **Report outbox.** Reports are keyed by task id and asset, queued once and retried for 8 days, long enough for a diff --git a/hosts/codex/evidence.js b/hosts/codex/evidence.js index 413759b..ce63ce2 100644 --- a/hosts/codex/evidence.js +++ b/hosts/codex/evidence.js @@ -46,8 +46,9 @@ function headerOf(output) { /** * A Bash result as PostToolUse hands it over. Codex gives the bare output, without the exit code the model sees, so - * a check is judged by its output; an `Exit code:` or `Process exited with code` header, where one comes along, - * decides, and a timed-out run failed. A run still going at its wait limit has no verdict until its completion. + * a check needs explicit passing or failing evidence in its output; ordinary stdout leaves it unknown. An + * `Exit code:` or `Process exited with code` header, where one comes along, decides, and a timed-out run failed. + * A run still going at its wait limit has no verdict until its completion. */ function shellNoteOf(command, output, callId) { const ids = callId ? { callId } : {}; diff --git a/hosts/dsh/tools.js b/hosts/dsh/tools.js index f562702..e0dbcf4 100644 --- a/hosts/dsh/tools.js +++ b/hosts/dsh/tools.js @@ -22,7 +22,7 @@ const textOf = (result) => (result?.content ?? []) function exitInText(text) { const last = [...text.matchAll(EXIT_MARKER)].at(-1); if (last) return Number(last[1]); - return KILLED_MARKER.test(text) ? KILLED_EXIT : 0; + return KILLED_MARKER.test(text) ? KILLED_EXIT : null; } // A call dsh refused or aborted, a run still going in the background, or one stopped on request has no verdict, @@ -32,7 +32,7 @@ function exitOf(result) { const { value } = result; if (value?.kind === 'foreground') { if (value.aborted || value.stopped !== undefined) return undefined; - return Number.isInteger(value.exitCode) ? value.exitCode : KILLED_EXIT; + return Number.isInteger(value.exitCode) ? value.exitCode : (value.timedOut || value.signal ? KILLED_EXIT : null); } if (value?.kind) return undefined; return exitInText(typeof value === 'string' ? value : textOf(result)); diff --git a/hosts/workbuddy/transcript.js b/hosts/workbuddy/transcript.js index 95bc88f..2711462 100644 --- a/hosts/workbuddy/transcript.js +++ b/hosts/workbuddy/transcript.js @@ -1,7 +1,7 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap -import { isVerificationCommand, verificationFailed } from '../../lib/verification.js'; +import { isVerificationCommand, verificationFailureOf } from '../../lib/verification.js'; // FinalStop carries neither the answer nor the turn's tool calls, so a turn is read off the session transcript: one // JSON record per line, `function_call` / `function_call_result` pairs joined by `callId`, messages with a `role`. @@ -86,8 +86,7 @@ const shellOutputOf = (text) => { function shellFailed(result) { if (result.status !== 'completed') return true; const exit = EXIT_CODE.exec(result.text); - if (exit && Number(exit[1]) !== 0) return true; - return verificationFailed({ text: shellOutputOf(result.text) }); + return verificationFailureOf({ exit: exit ? Number(exit[1]) : null, text: shellOutputOf(result.text) }); } const isCheck = (call) => SHELL_TOOLS.has(call.name) && call.args.run_in_background !== true && isVerificationCommand(call.args.command); diff --git a/lib/checks.js b/lib/checks.js index 6662b67..58d9a95 100644 --- a/lib/checks.js +++ b/lib/checks.js @@ -18,13 +18,27 @@ export function checkIdentityOf(command) { return String(command).replace(OUTPUT_PIPE, '').replace(STREAM_REDIRECT, '').replace(/\s+/g, ' ').trim(); } +export function verdictOf(check) { + if (check?.failed === true) return 'failed'; + if (check?.failed === false) return 'passed'; + return 'unknown'; +} + +export function checkExitOf(check) { + const verdict = verdictOf(check); + if (verdict === 'unknown') return null; + return verdict === 'failed' ? 1 : 0; +} + function failedEarlierThenPassed(checks, last) { const identity = checkIdentityOf(last.command); - return !last.failed && checks.slice(0, -1).some(({ command, failed }) => failed && checkIdentityOf(command) === identity); + return checks.slice(0, -1).some((check) => verdictOf(check) === 'failed' && checkIdentityOf(check.command) === identity); } export function checkEvidenceOf(checks) { const last = checks?.at(-1); - if (!last) return undefined; - return { kind: checkKindOf(last.command), passed: !last.failed, fail_then_pass: failedEarlierThenPassed(checks, last) }; + const verdict = verdictOf(last); + if (verdict === 'unknown') return undefined; + const passed = verdict === 'passed'; + return { kind: checkKindOf(last.command), passed, fail_then_pass: passed && failedEarlierThenPassed(checks, last) }; } diff --git a/lib/distill.js b/lib/distill.js index 27a8ed1..3ddd91f 100644 --- a/lib/distill.js +++ b/lib/distill.js @@ -1,6 +1,7 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap +import { checkExitOf, verdictOf } from './checks.js'; import { containsSecret, withoutSecrets } from './redact.js'; export const DRAFTS_TTL_MS = 8 * 24 * 60 * 60 * 1000; @@ -23,7 +24,7 @@ const SCOPE_MAX_CHARS = 300; export function isDistillableTurn({ reason, agentId, isPersonTurn, changedLines, checks }) { if (reason !== 'answer' || agentId !== undefined || !isPersonTurn) return false; if (!(changedLines > 0) || !Array.isArray(checks) || checks.length === 0) return false; - return checks.at(-1).failed === false; + return verdictOf(checks.at(-1)) === 'passed'; } export function distillPromptOf(scope = '') { @@ -83,9 +84,9 @@ function lastRunOfEachCheck(checks) { const command = String(check.command ?? '').trim().slice(0, COMMAND_MAX_CHARS); if (!command) continue; byCommand.delete(command); - byCommand.set(command, check.failed ? 1 : 0); + byCommand.set(command, checkExitOf(check)); } - return [...byCommand].slice(-MAX_TRACE_ROWS).map(([command, exit]) => ({ command, exit })); + return [...byCommand].filter(([, exit]) => exit !== null).slice(-MAX_TRACE_ROWS).map(([command, exit]) => ({ command, exit })); } function relativeArtifacts(paths, projectDir) { diff --git a/lib/distill/candidate.js b/lib/distill/candidate.js index d18fb1c..c066d85 100644 --- a/lib/distill/candidate.js +++ b/lib/distill/candidate.js @@ -1,6 +1,7 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap +import { verdictOf } from '../checks.js'; import { withoutSecrets } from '../redact.js'; // Distillation runs once a session, at its end, outside the conversation: each turn that finished verified only @@ -26,8 +27,9 @@ export function skipReason(reason, record) { if (reason !== 'answer') return 'not_completed'; if (!record.answer) return 'no_answer'; const lastCheck = record.checks.at(-1); - if (!lastCheck) return 'unverified'; - if (lastCheck.failed) return 'failed_check'; + const verdict = verdictOf(lastCheck); + if (verdict === 'failed') return 'failed_check'; + if (verdict === 'unknown') return 'unverified'; if (record.edits.length === 0 && record.commands.length < MIN_COMMANDS_WITHOUT_EDITS) return 'too_little_work'; return null; } diff --git a/lib/distill/prompt.js b/lib/distill/prompt.js index 9fb1c8a..31e9086 100644 --- a/lib/distill/prompt.js +++ b/lib/distill/prompt.js @@ -1,6 +1,7 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap +import { checkExitOf, verdictOf } from '../checks.js'; import { withoutSecrets } from '../redact.js'; const LIST_FIELDS = ['strategy', 'validation', 'signals', 'preconditions']; @@ -35,7 +36,7 @@ export function distillPrompt(candidate, focus = '') { lines(candidate.paths, (file) => `- ${file}`), '', '## Checks', - lines(candidate.checks, (check) => `- ${check.failed ? 'FAILED' : 'passed'}: ${check.command}`), + lines(candidate.checks, (check) => `- ${verdictOf(check) === 'failed' ? 'FAILED' : verdictOf(check)}: ${check.command}`), '', '## Final answer', candidate.answer, @@ -81,7 +82,7 @@ export function distillBody({ platform, sessionId, candidate, args, usage }) { // The core rejects a Gene that repeats this answer's values (answer_leak). final_answer: candidate.answer, ...args, - execution: { trace: candidate.checks.map((check) => ({ command: check.command, exit: check.failed ? 1 : 0 })) }, + execution: { trace: candidate.checks.map((check) => ({ command: check.command, exit: checkExitOf(check) })).filter(({ exit }) => exit !== null) }, persist: true, publish: false, ...(usage ? { distill_usage: usage } : {}), diff --git a/lib/reuse.js b/lib/reuse.js index 24a0c70..57927ea 100644 --- a/lib/reuse.js +++ b/lib/reuse.js @@ -32,8 +32,8 @@ function immediateOutcomeOf(evidence, turnReason) { return outcomeOfReason(turnReason) ?? 'failed'; } -function checkPhrase(check) { - if (!check) return 'no check ran'; +function checkPhrase(check, lastCheck) { + if (!check) return lastCheck ? 'last check result unknown' : 'no check ran'; const result = check.passed ? 'passed' : 'failed'; return `last ${check.kind} ${result}${check.fail_then_pass ? ' after failing earlier' : ''}`; } @@ -48,7 +48,7 @@ export function immediateReportOf({ asset, sessionId, turn, turnReason, checks, assetId: asset.assetId, taskId: turnTaskId(sessionId, turn), outcome: immediateOutcomeOf(evidence, turnReason), - reason: `Injected by Evolver into turn ${turn}; applied: ${applied}; ${checkPhrase(evidence.check)}.`, + reason: `Injected by Evolver into turn ${turn}; applied: ${applied}; ${checkPhrase(evidence.check, checks?.at(-1))}.`, evidence, }); } diff --git a/lib/turn-summary.js b/lib/turn-summary.js index 354e040..e4e802c 100644 --- a/lib/turn-summary.js +++ b/lib/turn-summary.js @@ -1,6 +1,7 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap +import { verdictOf } from './checks.js'; import { referenceReuseSavings, usdOf } from './savings.js'; import { outcomeOfReason } from './turn-outcomes.js'; @@ -25,7 +26,9 @@ function freshTokens(usage) { function goalPart({ lastCheck, turnReason }) { if (!lastCheck) return outcomeOfReason(turnReason) === 'success' ? '✓ Goal reached' : null; const label = lastCheck.label || lastCheck.command; - return lastCheck.failed + const verdict = verdictOf(lastCheck); + if (verdict === 'unknown') return `? Goal unverified (last check result unknown: ${label})`; + return verdict === 'failed' ? `✗ Goal not reached (last check failed: ${label})` : `✓ Goal reached (last check passed: ${label})`; } @@ -35,7 +38,7 @@ function goalPart({ lastCheck, turnReason }) { // approach from scratch, which a single answered question never cost. function savingOf({ lastCheck, changedLines, usage }) { const used = freshTokens(usage); - if (!lastCheck || lastCheck.failed || used === null) return null; + if (verdictOf(lastCheck) !== 'passed' || used === null) return null; return Math.min(referenceReuseSavings(changedLines).tokens, used); } diff --git a/lib/turn/record.js b/lib/turn/record.js index 76cdc9c..bd69b76 100644 --- a/lib/turn/record.js +++ b/lib/turn/record.js @@ -1,7 +1,7 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap -import { isVerificationCommand, verificationFailed } from '../verification.js'; +import { isVerificationCommand, verificationFailureOf } from '../verification.js'; const CHECK_COMMAND_MAX_CHARS = 200; @@ -12,7 +12,8 @@ const CHECK_COMMAND_MAX_CHARS = 200; * { kind: 'tool', command?, paths? } * { kind: 'answer', text } the turn's final answer, for a host that tells it before the turn's end * An exit code decides a check when the host gives one; the output is read too, since a failing run piped - * through `tail` still exits zero. A command still running has no verdict: a later note with its `callId` + * through `tail` still exits zero. Without an exit code or a supported passing summary, `failed: null` keeps + * the check's result unknown. A command still running has no verdict: a later note with its `callId` * settles it, and one still pending when the turn ends counts as not run. `label` names a check to the person, and * `lines` counts what an edit changed and `identifiers` the names it introduced, where the host says. */ @@ -44,11 +45,10 @@ export function turnRecordOf(notes) { } function checkFailed(note) { - const isError = Number.isInteger(note.exit) && note.exit !== 0; - return verificationFailed({ isError, text: note.output ?? '' }); + return verificationFailureOf({ exit: note.exit, text: note.output }); } /** Whether a tool note is a finished test, build, lint or type-check that failed. */ export function isFailedCheck(note) { - return note?.kind === 'shell' && !note.running && isVerificationCommand(note.command) && checkFailed(note); + return note?.kind === 'shell' && !note.running && isVerificationCommand(note.command) && checkFailed(note) === true; } diff --git a/lib/verification.js b/lib/verification.js index 330d87f..ecd59cf 100644 --- a/lib/verification.js +++ b/lib/verification.js @@ -41,3 +41,17 @@ export function verificationFailed({ isError, text }) { const evidence = String(text ?? '').replace(ZERO_COUNT, ''); return detectSignals(evidence).some((signal) => FAILURE_SIGNALS.has(signal)); } + +// Codex and some older/text-only host results omit the process exit code. A complete Node/TAP summary is +// positive evidence; ordinary output or just the absence of failure words is not. Failure always wins. +const PASSED_COUNT = /^\s*(?:#|ℹ)\s+pass\s+[1-9]\d*\b/m; +const ZERO_FAILED_COUNT = /^\s*(?:#|ℹ)\s+fail\s+0\b/m; +const FAILED_COUNT = /^\s*(?:#|ℹ)\s+(?:fail|cancelled)\s+[1-9]\d*\b/m; + +/** A finished check's failed flag: true = failed, false = passed, null = unknown. */ +export function verificationFailureOf({ exit, text }) { + const output = String(text ?? ''); + if (verificationFailed({ isError: Number.isInteger(exit) && exit !== 0, text: output }) || FAILED_COUNT.test(output)) return true; + if (exit === 0 || (PASSED_COUNT.test(output) && ZERO_FAILED_COUNT.test(output))) return false; + return null; +} diff --git a/test/checks.test.js b/test/checks.test.js index 877e2a2..8df1c33 100644 --- a/test/checks.test.js +++ b/test/checks.test.js @@ -19,6 +19,12 @@ test('the last check decides, and a failure of that same command earlier marks a assert.deepEqual(checkEvidenceOf([{ command: 'npm test', failed: false }, { command: 'npm test', failed: true }]), { kind: 'test', passed: false, fail_then_pass: false }); }); +test('an unknown last result is no check evidence; an earlier unknown result is no failure', () => { + assert.equal(checkEvidenceOf([{ command: 'npm test', failed: false }, { command: 'npm test', failed: null }]), undefined); + assert.deepEqual(checkEvidenceOf([{ command: 'npm test', failed: null }, { command: 'npm test', failed: false }]), + { kind: 'test', passed: true, fail_then_pass: false }); +}); + test('the same check is recognised when only its output trimming differs', () => { const checks = [ { command: 'cd /tmp/lru-e2e && node --test 2>&1 | tail -12', failed: true }, diff --git a/test/distill-prompt.test.js b/test/distill-prompt.test.js new file mode 100644 index 0000000..3f87858 --- /dev/null +++ b/test/distill-prompt.test.js @@ -0,0 +1,15 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { distillBody, distillPrompt } from '../lib/distill/prompt.js'; + +test('an earlier unknown check stays unknown in the model record and supplies no fabricated exit code', () => { + const candidate = { + prompt: 'Fix the retries', commands: ['npm run lint', 'npm test'], paths: [], answer: 'Fixed.', + checks: [{ command: 'npm run lint', failed: null }, { command: 'npm test', failed: false }], + }; + assert.match(distillPrompt(candidate), /- unknown: npm run lint/); + assert.doesNotMatch(distillPrompt(candidate), /passed: npm run lint/); + const body = distillBody({ platform: 'codex', sessionId: 's1', candidate, args: {} }); + assert.deepEqual(body.execution.trace, [{ command: 'npm test', exit: 0 }]); +}); diff --git a/test/distill.test.js b/test/distill.test.js index e6c514d..dbd3e44 100644 --- a/test/distill.test.js +++ b/test/distill.test.js @@ -67,6 +67,15 @@ test('the request keeps each check once, as its last run, only project-relative assert.equal(body.platform, 'codex', 'the host is the caller\'s, never assumed'); }); +test('the last unknown run replaces an earlier pass without inventing a successful trace row', () => { + const { body } = distillRequestOf({ lesson: LESSON, + checks: [{ command: 'npm test', failed: false }, { command: 'npm test', failed: null }, { command: 'tsc', failed: false }], paths: [], + sessionId: 's1', platform: 'codex', + }); + assert.deepEqual(body.execution.trace, [{ command: 'tsc', exit: 0 }]); + assert.deepEqual(body.validation, ['tsc']); +}); + const RESULT = { ok: true, distill_id: 'abc123', diff --git a/test/hosts/antigravity/readers.test.js b/test/hosts/antigravity/readers.test.js index d164fc2..49c1d99 100644 --- a/test/hosts/antigravity/readers.test.js +++ b/test/hosts/antigravity/readers.test.js @@ -47,6 +47,7 @@ test('an exit code decides a check; without one an ERROR step fails it and the o const checksOf = (text) => transcriptRecordOf(stepsOf(text), { stepIndex: 0, createdAt: null }).checks; assert.deepEqual(checksOf(turn(call(1, 'npm test'), row({ step_index: 2, source: 'MODEL', type: 'RUN_COMMAND', status: 'ERROR', error: 'exit status 2' }))), [{ command: 'npm test', failed: true }]); assert.deepEqual(checksOf(turn(call(1, 'npm test'), row({ step_index: 2, source: 'MODEL', type: 'RUN_COMMAND', status: 'ERROR' }))), [{ command: 'npm test', failed: true }]); + assert.deepEqual(checksOf(turn(call(1, 'npm test'), row({ step_index: 2, source: 'MODEL', type: 'RUN_COMMAND', content: 'done' }))), [{ command: 'npm test', failed: null }]); assert.deepEqual(checksOf(turn(call(1, 'npm test | tail -3'), row({ step_index: 2, source: 'MODEL', type: 'RUN_COMMAND', content: 'The command exited with code 0.\nOutput:\nAssertionError: expected 1 to equal 2' }))), [{ command: 'npm test | tail -3', failed: true }]); assert.deepEqual(checksOf(turn(call(1, 'npm test'))), [], 'a call no result answered was not run'); }); diff --git a/test/hosts/codex/readers.test.js b/test/hosts/codex/readers.test.js index 6d84c4a..e72f057 100644 --- a/test/hosts/codex/readers.test.js +++ b/test/hosts/codex/readers.test.js @@ -5,7 +5,9 @@ import { test } from 'node:test'; import { patchedPaths } from '../../../hosts/codex/evidence.js'; import codex from '../../../hosts/codex/profile.js'; +import { createDriver } from '../../../lib/agent/driver.js'; import { turnRecordOf } from '../../../lib/turn/record.js'; +import { PROMPT, memoryPorts } from '../../app/memory-ports.js'; // Payloads Codex 0.164.0 sent to a plugin's hooks (paths renamed), and the tool responses of the standalone // plugin's tests, from Codex 0.144. @@ -24,6 +26,23 @@ const TURN = '01a17b33-17ae-7f93-a0ea-d7608791f4bd'; const ONE_SESSION = ['SessionStart.startup', 'UserPromptSubmit', 'PreToolUse.apply_patch', 'PostToolUse.apply_patch', 'PreToolUse.Bash', 'PostToolUse.Bash.passed', 'PostToolUse.Bash.exit3', 'Stop', 'SessionEnd']; +test('an exit-less result with no passing evidence cannot verify a turn, including after an earlier pass', async () => { + for (const earlierPass of [false, true]) { + const { ports, queued } = memoryPorts(); + const driver = createDriver({ profile: codex, ports, lifetime: 'ephemeral' }); + await driver.handle('UserPromptSubmit', { ...payload('UserPromptSubmit'), prompt: PROMPT }); + await driver.handle('PostToolUse', payload('PostToolUse.apply_patch')); + if (earlierPass) await driver.handle('PostToolUse', payload('PostToolUse.Bash.passed')); + await driver.handle('PostToolUse', { + ...payload('PostToolUse.Bash.passed'), tool_input: { command: 'npm test' }, tool_response: 'boom\n', tool_use_id: 'silent-exit', + }); + await driver.handle('Stop', payload('Stop')); + assert.equal((await ports.candidates.read())[SESSION], undefined, `earlier pass: ${earlierPass}`); + assert.equal(queued[0].report.evidence.tier, 'none'); + assert.equal(queued[0].report.evidence.check, undefined); + } +}); + test('every event names its session; the prompt, its tool calls and the turn end name the turn', () => { for (const name of ONE_SESSION) assert.equal(codex.payload.sessionId(payload(name)), SESSION, name); for (const name of ['UserPromptSubmit', 'PreToolUse.apply_patch', 'PostToolUse.apply_patch', 'PostToolUse.Bash.passed', 'Stop']) { @@ -62,7 +81,7 @@ test('a Bash result carries no exit code, so a check is judged by its output', ( assert.deepEqual({ ...passed, output: undefined }, { kind: 'shell', command: 'node --test', exit: null, output: undefined, callId: 'call_4' }); assert.deepEqual(checksOf(passed), [{ command: 'node --test', failed: false }]); assert.deepEqual(noteOf('PostToolUse.Bash.exit3'), { kind: 'shell', command: 'node -e "console.log(\'boom\'); process.exit(3)"', exit: null, output: 'boom\n', callId: 'call_6' }); - assert.deepEqual(checksOf(bash('boom\n')), [{ command: 'npm test', failed: false }], 'a check that fails silently reads as passed'); + assert.deepEqual(checksOf(bash('boom\n')), [{ command: 'npm test', failed: null }], 'bare stdout carries neither an exit code nor positive passing evidence'); assert.deepEqual(checksOf(bash(RESPONSES_0144['bash.patchAndFailingTests'])), [{ command: 'npm test', failed: true }]); }); diff --git a/test/hosts/cursor/profile.test.js b/test/hosts/cursor/profile.test.js index c97cc7e..e764a52 100644 --- a/test/hosts/cursor/profile.test.js +++ b/test/hosts/cursor/profile.test.js @@ -37,6 +37,8 @@ test('the chat model is kept as Cursor names it; an empty name is the default', test('a Shell result is a check by its exit code and output; writes, deletes and edits are noted by path', () => { assert.deepEqual(noteOf('postToolUse.shellPassed'), { kind: 'shell', command: 'npm test', exit: 0, output: 'ℹ tests 2\nℹ pass 2\nℹ fail 0\n' }); assert.deepEqual(noteOf('postToolUse.shellFailed'), { kind: 'shell', command: 'npm test', exit: 1, output: 'done' }); + const unknown = cursor.evidence.toolNote({ ...payload('postToolUse.shellPassed'), tool_output: JSON.stringify({ output: 'done' }) }); + assert.deepEqual(turnRecordOf([unknown]).checks, [{ command: 'npm test', failed: null }]); assert.deepEqual(noteOf('postToolUse.write'), { kind: 'tool', paths: ['/work/app/lib/redis.js'] }); assert.deepEqual(noteOf('postToolUse.delete'), { kind: 'edit', paths: ['/work/app/lib/old.js'] }); assert.deepEqual(noteOf('afterFileEdit'), { kind: 'edit', paths: ['/work/app/lib/redis.js'] }); diff --git a/test/hosts/dsh/readers.test.js b/test/hosts/dsh/readers.test.js index 5821b1a..a9a9c9c 100644 --- a/test/hosts/dsh/readers.test.js +++ b/test/hosts/dsh/readers.test.js @@ -6,6 +6,7 @@ import { answerTextOf, modelOf, personTextOf } from '../../../hosts/dsh/messages import profile from '../../../hosts/dsh/profile.js'; import { editedFilesOf, toolNoteOf } from '../../../hosts/dsh/tools.js'; import { turnEndReason } from '../../../hosts/dsh/turn-end.js'; +import { turnRecordOf } from '../../../lib/turn/record.js'; const DSH = JSON.parse(readFileSync(new URL('../../fixtures/dsh/payloads.json', import.meta.url), 'utf8')); const PROJECT = '/home/dev/project'; @@ -60,11 +61,15 @@ test('a command still running in the background, or one dsh refused, has no verd test('a run without dsh 0.2\'s structured value is read off the markers its text ends with', () => { const run = (text, extra = {}) => toolNoteOf({ name: 'bash', arguments: { command: 'npm test' }, result: { isError: false, content: [{ type: 'text', text }], ...extra } }); assert.equal(run('ℹ fail 1\n[exit code: 1]').exit, 1); - assert.equal(run('ℹ pass 4').exit, 0); + assert.equal(run('ℹ pass 4').exit, null, 'missing markers must not invent a native exit code'); + assert.deepEqual(turnRecordOf([run('ℹ pass 4\nℹ fail 0')]).checks, [{ command: 'npm test', failed: false }]); + assert.deepEqual(turnRecordOf([run('done')]).checks, [{ command: 'npm test', failed: null }]); assert.equal(run('partial\n[timed out after 60000ms]\n[killed by signal: SIGTERM]').exit, 124); assert.equal(run('done', { value: 'ok\n[Command finished with exit code 2]' }).exit, 2); const killed = { ...DSH.tools.bashExit3, result: { ...DSH.tools.bashExit3.result, value: { ...DSH.tools.bashExit3.result.value, exitCode: null, signal: 'SIGKILL' } } }; assert.equal(toolNoteOf(killed).exit, 124); + const unknown = { ...killed, result: { ...killed.result, value: { ...killed.result.value, signal: null } } }; + assert.equal(toolNoteOf(unknown).exit, null, 'a missing structured exit with no timeout or signal is unknown'); }); test('write and edit are edits; a read names the file it read', () => { diff --git a/test/hosts/grok/readers.test.js b/test/hosts/grok/readers.test.js index cb48850..5f15b45 100644 --- a/test/hosts/grok/readers.test.js +++ b/test/hosts/grok/readers.test.js @@ -39,6 +39,8 @@ test('the prompt and the project are read off the payload; a subagent\'s event i }); test('a shell result carries its exit code; failure then pass reads as a fixed check', () => { + const unknown = grok.evidence.toolNote({ ...payload('PostToolUse.run_terminal_command.passed'), toolInput: { command: 'npm test' }, toolResult: { type: 'Bash', output_for_prompt: 'done' } }); + assert.deepEqual(turnRecordOf([unknown]).checks, [{ command: 'npm test', failed: null }]); const failed = noteOf('PostToolUse.run_terminal_command.failed'); assert.deepEqual(failed, { kind: 'shell', command: 'sh check.sh', exit: 1, output: '1 failing: README not fixed\n', callId: 'call_3' }); assert.deepEqual(noteOf('PostToolUse.run_terminal_command.exit3'), { kind: 'shell', command: 'sleep 1; exit 3', exit: 3, output: '', callId: 'call_18' }); diff --git a/test/hosts/opencode/profile.test.js b/test/hosts/opencode/profile.test.js index 381d615..c61a144 100644 --- a/test/hosts/opencode/profile.test.js +++ b/test/hosts/opencode/profile.test.js @@ -27,6 +27,7 @@ test('only the person\'s own sessions count: a task tool\'s child and the distil test('a bash check passes or fails on its exit code, and on failure output a `| tail` hid from it', () => { assert.deepEqual(checksOf(bash.passed, bash.failed, bash.failedThroughTail).map((check) => check.failed), [false, true, true]); + assert.deepEqual(checksOf({ title: 'npm test', output: 'done', metadata: {} }), [{ command: 'npm test', failed: null }]); }); test('a check the person interrupted gives no verdict, while one its timeout killed failed', () => { diff --git a/test/hosts/qoder/readers.test.js b/test/hosts/qoder/readers.test.js index 2b73f1e..d30dfb8 100644 --- a/test/hosts/qoder/readers.test.js +++ b/test/hosts/qoder/readers.test.js @@ -47,6 +47,8 @@ test('a subagent\'s event and the task notification Qoder submits for a backgrou }); test('a finished Bash call carries its exit code and output; a failed one its exit code in the error text', () => { + const unknown = qoder.evidence.toolNote({ ...payload('PostToolUse.Bash.passed'), tool_input: { command: 'npm test' }, tool_response: { stdout: 'done' } }); + assert.deepEqual(turnRecordOf([unknown]).checks, [{ command: 'npm test', failed: null }]); assert.deepEqual(noteOf('PostToolUse.Bash.passed'), { kind: 'shell', command: 'sh check.sh 2>&1 | tail -5', exit: 0, output: '# tests 1\n# pass 1\n# fail 0' }); assert.deepEqual(noteOf('PostToolUseFailure.Bash.exit1'), { kind: 'shell', command: 'sh check.sh', exit: 1, output: 'not ok 1 - readme\n# tests 1\n# pass 0\n# fail 1' }); assert.deepEqual(noteOf('PostToolUseFailure.Bash.exit127'), { kind: 'shell', command: 'npm test', exit: 127, output: '(eval):1: command not found: npm' }); diff --git a/test/hosts/workbuddy/readers.test.js b/test/hosts/workbuddy/readers.test.js index 9bdf949..6c5aef7 100644 --- a/test/hosts/workbuddy/readers.test.js +++ b/test/hosts/workbuddy/readers.test.js @@ -89,6 +89,7 @@ test('a non-zero exit or failure output fails a check; an incomplete call failed assert.equal(checksOf(bashRecords('npm test | tail -3', shellText('npm test | tail -3', 'AssertionError: expected 1 to equal 2\n1 test failed', 0)))[0].failed, true); assert.equal(checksOf(bashRecords('npm test -- --grep "test failed"', shellText('npm test', 'ℹ pass 4\nℹ fail 0', 0)))[0].failed, false); assert.equal(checksOf(bashRecords('npm test', 'Interrupted', { status: 'incomplete' }))[0].failed, true); + assert.equal(checksOf(bashRecords('npm test', 'boom\n'))[0].failed, null, 'completed says the tool returned, not that its process exited zero'); assert.deepEqual(checksOf([bashRecords('npm test', '')[0]]), []); assert.equal(checksOf(bashRecords('npm test', shellText('npm test', 'ℹ fail 0', 0)).map((record) => ({ ...record, name: 'PowerShell' }))).length, 1); }); diff --git a/test/turn-summary.test.js b/test/turn-summary.test.js index db20e22..538e2c8 100644 --- a/test/turn-summary.test.js +++ b/test/turn-summary.test.js @@ -62,6 +62,13 @@ test('a turn whose usage is unknown counts no saving and states no usage', () => ); }); +test('an unknown last check claims neither a reached goal nor a saving', () => { + assert.equal( + turnSummaryOf({ reusedNames: ['Redis pool'], lastCheck: { command: 'npm test', failed: null }, turnReason: 'answer', changedLines: 30, usage: USAGE }), + '? Goal unverified (last check result unknown: npm test) · reused EvoMap strategy "Redis pool" · this turn used 6k fresh tokens', + ); +}); + test('the summary is one line, since the desktop notice renders no line breaks', () => { const summary = turnSummaryOf({ reusedNames: ['Redis pool'], lastCheck: { command: 'npm test', failed: false }, usage: USAGE }); assert.equal(summary.includes('\n'), false); diff --git a/test/verification.test.js b/test/verification.test.js index 0191b43..a06c60b 100644 --- a/test/verification.test.js +++ b/test/verification.test.js @@ -1,7 +1,7 @@ import assert from 'node:assert/strict'; import { test } from 'node:test'; -import { isVerificationCommand, verificationFailed } from '../lib/verification.js'; +import { isVerificationCommand, verificationFailed, verificationFailureOf } from '../lib/verification.js'; test('test, build, lint and type-check runs are verifications; other commands are not', () => { for (const command of ['npm test', 'pnpm run build', 'yarn lint', 'node --test test/*.test.js', 'npx vitest run', 'go test ./...', @@ -34,6 +34,29 @@ test('a passing summary that counts zero failures passes', () => { assert.equal(verificationFailed({ text: '✔ Validation passed' }), false); }); +test('missing exit status is unknown unless a supported summary proves the check passed or failed', () => { + for (const exit of [undefined, null, '0', NaN]) { + for (const text of ['', 'boom\n', 'done', 'ok', 'ℹ pass 1']) { + assert.equal(verificationFailureOf({ exit, text }), null, `${exit}/${text}`); + } + } + assert.equal(verificationFailureOf({ exit: 0, text: '' }), false); + assert.equal(verificationFailureOf({ exit: 3, text: 'done' }), true); + assert.equal(verificationFailureOf({ text: 'TAP version 13\n# pass 1\n# fail 0\n' }), false); + assert.equal(verificationFailureOf({ text: 'ℹ pass 3\nℹ fail 0' }), false); + assert.equal(verificationFailureOf({ text: 'ℹ pass 3\nℹ fail 1' }), true); + assert.equal(verificationFailureOf({ exit: 0, text: '# pass 1\n# fail 1' }), true); + assert.equal(verificationFailureOf({ text: 'TypeError: cannot read properties of undefined' }), true); +}); + +test('a passing summary with cancelled tests does not verify a completed run', () => { + for (const prefix of ['#', 'ℹ']) { + const counts = `${prefix} pass 1\n${prefix} fail 0\n${prefix} cancelled `; + assert.equal(verificationFailureOf({ text: `${counts}1` }), true, prefix); + assert.equal(verificationFailureOf({ text: `${counts}0` }), false, prefix); + } +}); + test('a heredoc body that names a check is data, not a check; the command lines still count', () => { const editsReadme = "python3 - <<'EOF'\ntext = 'run npm test, then make'\nprint(text)\nEOF"; assert.equal(isVerificationCommand(editsReadme), false);