fix: preserve unknown check verdicts across host evidence flows

docs/architecture.md

@@ -108,7 +108,7 @@ left in it (adopted once, so a resumed session keeps counting turns and nothing
 | Session start | `sessionStarted` | Queues retention reports for an earlier session's edits and aborted reports for a killed session's injections (its owner process is gone); schedules engine setup, delivery and leftover distillation; answers the one-off engine notice, the only text besides a strategy that reaches the model. |
 | Prompt | `promptSubmitted` | A correction drops the session's candidate, then the last turn's reuses are reported again under `:correction`. The turn counter moves on and survives the session's end, so a resumed session never repeats a task id. One strategy, or a Recipe expanded into one, is recalled within what is left of the hook's budget and ends with `Apply them where they fit.` |
 | Tool result | `toolFinished`, `beforeEdit`, `takeWaitingStrategy`, `recallWhenStuck` | Notes each call: shell commands with their exit code (a run still going is settled by a later note, or counts as not run), edited paths, each file's hash before its first edit. Where the strategy waits for a tool result, the turn's first one takes it and books it, only while that turn is open; a turn with no tool call never gets it, and it is neither reported nor kept out of later recalls. Where a tool result's context reaches the model (`toolResultContext`, or a `tool-result` host), a failed test, build, lint or type-check recalls again, once per distinct failure and twice a turn at most, with a query built from its output and the turn's edits; a strategy that lands within the tool result's budget is added to that result and booked. |
-| Turn end | `turnEnded` | Ends as `answer`, `aborted`, `refusal` or `error`. Every injection of the turn is reported (`applied: used`, tier `L1` when a check ran; tier `none` with `reported_by: agent` when the agent only reported its checks). A turn that answered, whose last check passed and that edited files or ran three commands, becomes the session's candidate. |
+| Turn end | `turnEnded` | Ends as `answer`, `aborted`, `refusal` or `error`. Every injection of the turn is reported (`applied: used`, tier `L1` when the last check has a known result; tier `none` with `reported_by: agent` when the agent only reported its checks). A turn that answered, whose last check passed and that edited files or ran three commands, becomes the session's candidate. |
 | Session end | `sessionEnded` | Only writes: a turn still open ends, what is unreported goes out as aborted, the candidate is closed for background distillation. |
 
 **State** lives in one directory per host, every file named `<host>-<kind>`. Writes go through an exclusive lock with
@@ -242,14 +242,21 @@ scripts/               pack.mjs, publish-tag.sh
 
   | Report | `task_id` | `evidence` |
   |---|---|---|
-  | Turn end | `<host>:<session>:<turn>` | `tier` `L1` when a check ran, else `none`; `applied: used`; `check` `{kind, passed, fail_then_pass}`; `aborted`; `stage: immediate` |
+  | Turn end | `<host>:<session>:<turn>` | `tier` `L1` when the last check has a known result, else `none`; `applied: used`; `check` `{kind, passed, fail_then_pass}`; `aborted`; `stage: immediate` |
   | Next prompt corrects or approves | `…:correction` | `tier: L3`, `applied`, `stage: correction` |
   | Next session finds the edits kept or reverted | `…:retention` | `tier: L2`, `applied`, `stage: retention` |
 
   An injected strategy counts as used (dsh #55): the plugin cannot tell from a turn's evidence whether the model
   followed it, and a reused turn is distilled like any other, the engine's intake dropping a Gene it already has.
   `fail_then_pass` compares checks with output trimming (`| tail -9`, `2>&1`) removed, so the same check piped
   differently is still the same check. The outcome follows the last check; with none it follows how the turn ended.
+  A finished check with no native exit code is unknown (`failed: null`) unless its output contains failure
+  evidence or a supported Node/TAP summary with a positive pass count and zero failed tests. An unknown last
+  result keeps the report at tier `none`, creates no candidate and claims no saving; it never falls back to an
+  earlier passing check. For Codex runs without an exit code, text-only Jest, Vitest, pytest, Go and Cargo
+  successes stay unknown unless they emit a supported summary. This can reduce L1 evidence and candidate
+  generation in those projects. Unknown checks stay labelled unknown in the distillation prompt and provide no
+  execution trace exit code. A native exit code of zero still passes without requiring output.
   A tier `none` report counts as exposure only on the Hub and is not written to the local ledger. An older Proxy
   drops the new fields and the plugin runs unchanged.
 - **Report outbox.** Reports are keyed by task id and asset, queued once and retried for 8 days, long enough for a

hosts/codex/evidence.js

@@ -46,8 +46,9 @@ function headerOf(output) {
 
 /**
  * A Bash result as PostToolUse hands it over. Codex gives the bare output, without the exit code the model sees, so
- * a check is judged by its output; an `Exit code:` or `Process exited with code` header, where one comes along,
- * decides, and a timed-out run failed. A run still going at its wait limit has no verdict until its completion.
+ * a check needs explicit passing or failing evidence in its output; ordinary stdout leaves it unknown. An
+ * `Exit code:` or `Process exited with code` header, where one comes along, decides, and a timed-out run failed.
+ * A run still going at its wait limit has no verdict until its completion.
  */
 function shellNoteOf(command, output, callId) {
   const ids = callId ? { callId } : {};

hosts/dsh/tools.js

@@ -22,7 +22,7 @@ const textOf = (result) => (result?.content ?? [])
 function exitInText(text) {
   const last = [...text.matchAll(EXIT_MARKER)].at(-1);
   if (last) return Number(last[1]);
-  return KILLED_MARKER.test(text) ? KILLED_EXIT : 0;
+  return KILLED_MARKER.test(text) ? KILLED_EXIT : null;
 }
 
 // A call dsh refused or aborted, a run still going in the background, or one stopped on request has no verdict,
@@ -32,7 +32,7 @@ function exitOf(result) {
   const { value } = result;
   if (value?.kind === 'foreground') {
     if (value.aborted || value.stopped !== undefined) return undefined;
-    return Number.isInteger(value.exitCode) ? value.exitCode : KILLED_EXIT;
+    return Number.isInteger(value.exitCode) ? value.exitCode : (value.timedOut || value.signal ? KILLED_EXIT : null);
   }
   if (value?.kind) return undefined;
   return exitInText(typeof value === 'string' ? value : textOf(result));

hosts/workbuddy/transcript.js

@@ -1,7 +1,7 @@
 // SPDX-License-Identifier: MIT
 // Copyright (c) 2026 EvoMap
 
-import { isVerificationCommand, verificationFailed } from '../../lib/verification.js';
+import { isVerificationCommand, verificationFailureOf } from '../../lib/verification.js';
 
 // FinalStop carries neither the answer nor the turn's tool calls, so a turn is read off the session transcript: one
 // JSON record per line, `function_call` / `function_call_result` pairs joined by `callId`, messages with a `role`.
@@ -86,8 +86,7 @@ const shellOutputOf = (text) => {
 function shellFailed(result) {
   if (result.status !== 'completed') return true;
   const exit = EXIT_CODE.exec(result.text);
-  if (exit && Number(exit[1]) !== 0) return true;
-  return verificationFailed({ text: shellOutputOf(result.text) });
+  return verificationFailureOf({ exit: exit ? Number(exit[1]) : null, text: shellOutputOf(result.text) });
 }
 
 const isCheck = (call) => SHELL_TOOLS.has(call.name) && call.args.run_in_background !== true && isVerificationCommand(call.args.command);

lib/checks.js

@@ -18,13 +18,27 @@ export function checkIdentityOf(command) {
   return String(command).replace(OUTPUT_PIPE, '').replace(STREAM_REDIRECT, '').replace(/\s+/g, ' ').trim();
 }
 
+export function verdictOf(check) {
+  if (check?.failed === true) return 'failed';
+  if (check?.failed === false) return 'passed';
+  return 'unknown';
+}
+
+export function checkExitOf(check) {
+  const verdict = verdictOf(check);
+  if (verdict === 'unknown') return null;
+  return verdict === 'failed' ? 1 : 0;
+}
+
 function failedEarlierThenPassed(checks, last) {
   const identity = checkIdentityOf(last.command);
-  return !last.failed && checks.slice(0, -1).some(({ command, failed }) => failed && checkIdentityOf(command) === identity);
+  return checks.slice(0, -1).some((check) => verdictOf(check) === 'failed' && checkIdentityOf(check.command) === identity);
 }
 
 export function checkEvidenceOf(checks) {
   const last = checks?.at(-1);
-  if (!last) return undefined;
-  return { kind: checkKindOf(last.command), passed: !last.failed, fail_then_pass: failedEarlierThenPassed(checks, last) };
+  const verdict = verdictOf(last);
+  if (verdict === 'unknown') return undefined;
+  const passed = verdict === 'passed';
+  return { kind: checkKindOf(last.command), passed, fail_then_pass: passed && failedEarlierThenPassed(checks, last) };
 }

lib/distill.js

@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: MIT
 // Copyright (c) 2026 EvoMap
 
+import { checkExitOf, verdictOf } from './checks.js';
 import { containsSecret, withoutSecrets } from './redact.js';
 
 export const DRAFTS_TTL_MS = 8 * 24 * 60 * 60 * 1000;
@@ -23,7 +24,7 @@ const SCOPE_MAX_CHARS = 300;
 export function isDistillableTurn({ reason, agentId, isPersonTurn, changedLines, checks }) {
   if (reason !== 'answer' || agentId !== undefined || !isPersonTurn) return false;
   if (!(changedLines > 0) || !Array.isArray(checks) || checks.length === 0) return false;
-  return checks.at(-1).failed === false;
+  return verdictOf(checks.at(-1)) === 'passed';
 }
 
 export function distillPromptOf(scope = '') {
@@ -83,9 +84,9 @@ function lastRunOfEachCheck(checks) {
     const command = String(check.command ?? '').trim().slice(0, COMMAND_MAX_CHARS);
     if (!command) continue;
     byCommand.delete(command);
-    byCommand.set(command, check.failed ? 1 : 0);
+    byCommand.set(command, checkExitOf(check));
   }
-  return [...byCommand].slice(-MAX_TRACE_ROWS).map(([command, exit]) => ({ command, exit }));
+  return [...byCommand].filter(([, exit]) => exit !== null).slice(-MAX_TRACE_ROWS).map(([command, exit]) => ({ command, exit }));
 }
 
 function relativeArtifacts(paths, projectDir) {

lib/distill/candidate.js

@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: MIT
 // Copyright (c) 2026 EvoMap
 
+import { verdictOf } from '../checks.js';
 import { withoutSecrets } from '../redact.js';
 
 // Distillation runs once a session, at its end, outside the conversation: each turn that finished verified only
@@ -26,8 +27,9 @@ export function skipReason(reason, record) {
   if (reason !== 'answer') return 'not_completed';
   if (!record.answer) return 'no_answer';
   const lastCheck = record.checks.at(-1);
-  if (!lastCheck) return 'unverified';
-  if (lastCheck.failed) return 'failed_check';
+  const verdict = verdictOf(lastCheck);
+  if (verdict === 'failed') return 'failed_check';
+  if (verdict === 'unknown') return 'unverified';
   if (record.edits.length === 0 && record.commands.length < MIN_COMMANDS_WITHOUT_EDITS) return 'too_little_work';
   return null;
 }

lib/distill/prompt.js

@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: MIT
 // Copyright (c) 2026 EvoMap
 
+import { checkExitOf, verdictOf } from '../checks.js';
 import { withoutSecrets } from '../redact.js';
 
 const LIST_FIELDS = ['strategy', 'validation', 'signals', 'preconditions'];
@@ -35,7 +36,7 @@ export function distillPrompt(candidate, focus = '') {
     lines(candidate.paths, (file) => `- ${file}`),
     '',
     '## Checks',
-    lines(candidate.checks, (check) => `- ${check.failed ? 'FAILED' : 'passed'}: ${check.command}`),
+    lines(candidate.checks, (check) => `- ${verdictOf(check) === 'failed' ? 'FAILED' : verdictOf(check)}: ${check.command}`),
     '',
     '## Final answer',
     candidate.answer,
@@ -81,7 +82,7 @@ export function distillBody({ platform, sessionId, candidate, args, usage }) {
     // The core rejects a Gene that repeats this answer's values (answer_leak).
     final_answer: candidate.answer,
     ...args,
-    execution: { trace: candidate.checks.map((check) => ({ command: check.command, exit: check.failed ? 1 : 0 })) },
+    execution: { trace: candidate.checks.map((check) => ({ command: check.command, exit: checkExitOf(check) })).filter(({ exit }) => exit !== null) },
     persist: true,
     publish: false,
     ...(usage ? { distill_usage: usage } : {}),

lib/reuse.js

@@ -32,8 +32,8 @@ function immediateOutcomeOf(evidence, turnReason) {
   return outcomeOfReason(turnReason) ?? 'failed';
 }
 
-function checkPhrase(check) {
-  if (!check) return 'no check ran';
+function checkPhrase(check, lastCheck) {
+  if (!check) return lastCheck ? 'last check result unknown' : 'no check ran';
   const result = check.passed ? 'passed' : 'failed';
   return `last ${check.kind} ${result}${check.fail_then_pass ? ' after failing earlier' : ''}`;
 }
@@ -48,7 +48,7 @@ export function immediateReportOf({ asset, sessionId, turn, turnReason, checks,
     assetId: asset.assetId,
     taskId: turnTaskId(sessionId, turn),
     outcome: immediateOutcomeOf(evidence, turnReason),
-    reason: `Injected by Evolver into turn ${turn}; applied: ${applied}; ${checkPhrase(evidence.check)}.`,
+    reason: `Injected by Evolver into turn ${turn}; applied: ${applied}; ${checkPhrase(evidence.check, checks?.at(-1))}.`,
     evidence,
   });
 }

lib/turn-summary.js

@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: MIT
 // Copyright (c) 2026 EvoMap
 
+import { verdictOf } from './checks.js';
 import { referenceReuseSavings, usdOf } from './savings.js';
 import { outcomeOfReason } from './turn-outcomes.js';
 
@@ -25,7 +26,9 @@ function freshTokens(usage) {
 function goalPart({ lastCheck, turnReason }) {
   if (!lastCheck) return outcomeOfReason(turnReason) === 'success' ? '✓ Goal reached' : null;
   const label = lastCheck.label || lastCheck.command;
-  return lastCheck.failed
+  const verdict = verdictOf(lastCheck);
+  if (verdict === 'unknown') return `? Goal unverified (last check result unknown: ${label})`;
+  return verdict === 'failed'
     ? `✗ Goal not reached (last check failed: ${label})`
     : `✓ Goal reached (last check passed: ${label})`;
 }
@@ -35,7 +38,7 @@ function goalPart({ lastCheck, turnReason }) {
 // approach from scratch, which a single answered question never cost.
 function savingOf({ lastCheck, changedLines, usage }) {
   const used = freshTokens(usage);
-  if (!lastCheck || lastCheck.failed || used === null) return null;
+  if (verdictOf(lastCheck) !== 'passed' || used === null) return null;
   return Math.min(referenceReuseSavings(changedLines).tokens, used);
 }
 

lib/turn/record.js

@@ -1,7 +1,7 @@
 // SPDX-License-Identifier: MIT
 // Copyright (c) 2026 EvoMap
 
-import { isVerificationCommand, verificationFailed } from '../verification.js';
+import { isVerificationCommand, verificationFailureOf } from '../verification.js';
 
 const CHECK_COMMAND_MAX_CHARS = 200;
 
@@ -12,7 +12,8 @@ const CHECK_COMMAND_MAX_CHARS = 200;
  *   { kind: 'tool', command?, paths? }
  *   { kind: 'answer', text }        the turn's final answer, for a host that tells it before the turn's end
  * An exit code decides a check when the host gives one; the output is read too, since a failing run piped
- * through `tail` still exits zero. A command still running has no verdict: a later note with its `callId`
+ * through `tail` still exits zero. Without an exit code or a supported passing summary, `failed: null` keeps
+ * the check's result unknown. A command still running has no verdict: a later note with its `callId`
  * settles it, and one still pending when the turn ends counts as not run. `label` names a check to the person, and
  * `lines` counts what an edit changed and `identifiers` the names it introduced, where the host says.
  */
@@ -44,11 +45,10 @@ export function turnRecordOf(notes) {
 }
 
 function checkFailed(note) {
-  const isError = Number.isInteger(note.exit) && note.exit !== 0;
-  return verificationFailed({ isError, text: note.output ?? '' });
+  return verificationFailureOf({ exit: note.exit, text: note.output });
 }
 
 /** Whether a tool note is a finished test, build, lint or type-check that failed. */
 export function isFailedCheck(note) {
-  return note?.kind === 'shell' && !note.running && isVerificationCommand(note.command) && checkFailed(note);
+  return note?.kind === 'shell' && !note.running && isVerificationCommand(note.command) && checkFailed(note) === true;
 }

lib/verification.js

@@ -41,3 +41,17 @@ export function verificationFailed({ isError, text }) {
   const evidence = String(text ?? '').replace(ZERO_COUNT, '');
   return detectSignals(evidence).some((signal) => FAILURE_SIGNALS.has(signal));
 }
+
+// Codex and some older/text-only host results omit the process exit code. A complete Node/TAP summary is
+// positive evidence; ordinary output or just the absence of failure words is not. Failure always wins.
+const PASSED_COUNT = /^\s*(?:#|ℹ)\s+pass\s+[1-9]\d*\b/m;
+const ZERO_FAILED_COUNT = /^\s*(?:#|ℹ)\s+fail\s+0\b/m;
+const FAILED_COUNT = /^\s*(?:#|ℹ)\s+(?:fail|cancelled)\s+[1-9]\d*\b/m;
+
+/** A finished check's failed flag: true = failed, false = passed, null = unknown. */
+export function verificationFailureOf({ exit, text }) {
+  const output = String(text ?? '');
+  if (verificationFailed({ isError: Number.isInteger(exit) && exit !== 0, text: output }) || FAILED_COUNT.test(output)) return true;
+  if (exit === 0 || (PASSED_COUNT.test(output) && ZERO_FAILED_COUNT.test(output))) return false;
+  return null;
+}

test/checks.test.js

@@ -19,6 +19,12 @@ test('the last check decides, and a failure of that same command earlier marks a
   assert.deepEqual(checkEvidenceOf([{ command: 'npm test', failed: false }, { command: 'npm test', failed: true }]), { kind: 'test', passed: false, fail_then_pass: false });
 });
 
+test('an unknown last result is no check evidence; an earlier unknown result is no failure', () => {
+  assert.equal(checkEvidenceOf([{ command: 'npm test', failed: false }, { command: 'npm test', failed: null }]), undefined);
+  assert.deepEqual(checkEvidenceOf([{ command: 'npm test', failed: null }, { command: 'npm test', failed: false }]),
+    { kind: 'test', passed: true, fail_then_pass: false });
+});
+
 test('the same check is recognised when only its output trimming differs', () => {
   const checks = [
     { command: 'cd /tmp/lru-e2e && node --test 2>&1 | tail -12', failed: true },

test/distill-prompt.test.js

@@ -0,0 +1,15 @@
+import assert from 'node:assert/strict';
+import { test } from 'node:test';
+
+import { distillBody, distillPrompt } from '../lib/distill/prompt.js';
+
+test('an earlier unknown check stays unknown in the model record and supplies no fabricated exit code', () => {
+  const candidate = {
+    prompt: 'Fix the retries', commands: ['npm run lint', 'npm test'], paths: [], answer: 'Fixed.',
+    checks: [{ command: 'npm run lint', failed: null }, { command: 'npm test', failed: false }],
+  };
+  assert.match(distillPrompt(candidate), /- unknown: npm run lint/);
+  assert.doesNotMatch(distillPrompt(candidate), /passed: npm run lint/);
+  const body = distillBody({ platform: 'codex', sessionId: 's1', candidate, args: {} });
+  assert.deepEqual(body.execution.trace, [{ command: 'npm test', exit: 0 }]);
+});

test/distill.test.js

@@ -67,6 +67,15 @@ test('the request keeps each check once, as its last run, only project-relative
   assert.equal(body.platform, 'codex', 'the host is the caller\'s, never assumed');
 });
 
+test('the last unknown run replaces an earlier pass without inventing a successful trace row', () => {
+  const { body } = distillRequestOf({ lesson: LESSON,
+    checks: [{ command: 'npm test', failed: false }, { command: 'npm test', failed: null }, { command: 'tsc', failed: false }], paths: [],
+    sessionId: 's1', platform: 'codex',
+  });
+  assert.deepEqual(body.execution.trace, [{ command: 'tsc', exit: 0 }]);
+  assert.deepEqual(body.validation, ['tsc']);
+});
+
 const RESULT = {
   ok: true,
   distill_id: 'abc123',