feat(bench): self-improving SWE harness — measured supervisor evolution + automated improvement loop#576
feat(bench): self-improving SWE harness — measured supervisor evolution + automated improvement loop#576drewstone wants to merge 61 commits into
Conversation
Root-fix a scoring bug proven live: on django-12419 @ glm-4.6 the agent made the EXACT gold fix but scored 0 because the harness extracted the patch by parsing the model reply for a fenced diff block, and the model wrote prose instead. Two changes, standard SWE-bench practice (SWE-agent/OpenHands read the diff from repo state and pre-stage the checkout): - boxSetup(task): harness clones the instance repo at base_commit into a fixed /work BEFORE the agent shot (same session) so the agent only edits. A stochastic model cannot be trusted to clone to an exact path (observed cannot-change-to-/work failures). Adapter-agnostic seam on BenchmarkAdapter. - boxExtract(): git diff of the agent edits in /work (test files excluded; the judge applies the gold test_patch itself), preferred over the event-stream parse which stays the fallback. Prompt updated: repo pre-staged, agent only edits. Typecheck clean; calibration still green. Live glm-5.2 verification blocked on transient sandbox box-provisioning failures (box unavailable before start on a green health) - re-run when the platform host-agent recovers.
…proposer The code-surface proposer fed the coding agent the ~400-char DISTILLED findings (generationFailureDistiller), not the raw traces. The meta-harness edge (yoonholee.com/meta-harness) is raw-trace filesystem context: the coding agent greps/cats the FULL run traces of failed candidates to diagnose, rather than reading a pre-summary. Add rawTraceDistiller() — an additive analyzeGeneration producer that, instead of summarizing, points the proposer at the prior generation's real run traces already durable on disk under runDir (per-cell spans.jsonl event logs + cached-result.json scores + artifacts) with a grep/cat-to-diagnose instruction. It emits AnalystFinding[] with ABSOLUTE paths (the harness runs from a worktree cwd) so it drops into the same analyzeGeneration slot and renders through the same agenticGenerator prompt path. The existing distiller stays the default. improve() gains a one-line enable: opts.rawTraceContext = true wires rawTraceDistiller() when the caller has not supplied their own analyzeGeneration (explicit analyzeGeneration still wins; null still disables). Self-test builds a real tmp runDir with fake candidate + trace files and asserts the findings reference the actual absolute trace-file paths + the grep/cat instruction, plus worst-candidate-first ordering and the clean-generation fallback. tsc clean, biome clean, 3/3 new tests + 7/7 existing improve tests pass.
# Conflicts: # bench/src/benchmarks/swe-bench.ts # bench/src/run-benchmarks.ts # docs/api/primitive-catalog.md # src/improvement/raw-trace-distiller.test.ts # src/improvement/raw-trace-distiller.ts
…point Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…f-k + self-repair on docstring examples) Selection + repair grounded only on visible docstring >>> examples (doctest, nonce-sentinel), hidden check() grading physically after all harness decisions. Paired bootstrap + exact sign test. Calibrated: 164/164 hidden self-check, 75/164 coverage, 2/75 false-fail (dataset bugs). First result: Llama-3-8B 44.3%->57.3% pass@1 (+13.0pp, CI [+7.8,+18.4], p=1.5e-5, n=164). humaneval.ts: HUMANEVAL_GZ local-cache support (GitHub raw 429s under reps).
…ructural-lever rig Model writes single-line asserts from the prompt alone, BEFORE any candidate exists; asserts run individually in the jail and add to the doctest signal (split counts kept for audit). Extends honest-oracle coverage from 46% of HumanEval toward 100%. Opt-in via TESTGEN=<count>; default off keeps prior runs reproducible.
…all-k-samples-fail bucket
…EN), hidden=remaining asserts Same architecture as hev-structural (nonce judges, phase separation, incremental persistence, bootstrap+sign-test). Calibration: 427/427 hidden self-check, 0/427 visible false-fail, 0 dropped. Official-first scoring: MBPP one-sentence specs make model-guessed asserts ~70% wrong on passing code — shown assert ranks first, guesses break ties (pilot: selection -6.7pp -> +16.7pp).
… only net-new seam; extend strategy family, verifier-environment, field routing
…select + guarded repair (measured +8..+21pp)
…ull result (hev-eval, hev-improve)
…lRollout dials under the held-out gate
…tool-loop conversation runBrainLoop returned on a no-tool-call turn WITHOUT pushing the final assistant message, so ToolLoopResult.messages violated its own contract (seed + every assistant/tool turn): a shot that answers without tools came back with a transcript missing its own answer. Depth continuation then criticized a solution absent from the model's history, and structuralRollout's fenced-code candidate extraction (defaultExtractCandidate's designed fallback) read an empty conversation on non-tool-calling models (Llama-3-8B-Instruct-Lite never tool-calls under tool_choice:auto — measured on Together). Proven by the runtime smoke (bench/src/smoke-structural-rollout.mts): candidates now flow through the strategy on a real non-tool-calling model.
…ory-less consults consultAnalyst rendered EVERY raw consult in analyst framing (instruction as system, 'TASK: ...TRAJECTORY:' as the user turn). With no trajectory — the pre-task case, i.e. structuralRollout's authored-check generation — that user turn is just the coding task, which a weak model answers directly, ignoring the system instruction: measured on Llama-3-8B-Instruct-Lite, authored-assert yield 6/18 reps (0 asserts on the rest) vs 18/18 with the instruction and task fused into one user message (the proven hev-structural rig shape). Trajectory-bearing consults keep the analyst framing unchanged. Runtime smoke after the fix: authored checks on 20/20 HumanEval tasks (was 14/20).
…Eval, hidden grading script-side The ship gate for the ported strategy: runAgentic + structuralRollout(default policy: k=5, repairs<=2, testgen=6) over createVerifierEnvironment with an INERT check, so no hidden-test signal reaches selection/repair; visible checks are the strategy's own model-authored asserts run by the shipped sandboxCheckRunner over a docker --network=none exec channel (the script's thin adapter). The task's own check() suite grades every locked candidate script-side (runChecker) after each rollout finishes. Receipts are hard-verified against the recorded outcomes per candidate. Llama-3-8B-Instruct-Lite (Together), first 20 tasks: blind mean-of-k 63.0% -> selected 80.0% -> final 85.0% (+22pp), authored checks 20/20, 4 tasks rescued (selection or guarded repair passes hidden where sample 0 fails), 0 crashes.
…g (arms A-D, hard/control sets) Implements supervisor-lab PREREG-supervisor-showdown.md: fixed 62-hard/20-control HumanEval sets, equal-compute worker loop (k=5, testgen=6, <=2 repairs), visible-only evidence rendering, no-code plan contract with leak stripping + planLeaked audit, Phase A/B separation with the rig-local nonce hidden judge, per-model token+cost rows.
…glm plan calls exceed 240s under sustained load
…ator Per PREREG-swe-frontier Stage 0: glm-5.2 authors a repro script from the issue text (+ up to 3 requested file reads); validity = nonzero exit on the unpatched tree (1 feedback retry), soundness = exit 0 after the gold patch applied host-side to a COPY of the tree, run in the same :ro-mount jail as the env's run tool. Gold is script-side only, enforced by a leak assert on every outbound message. PYTHONPATH=/testbed pins imports to the mounted tree (script-path sys.path would silently test the image's baked-in install). swe-bench-env: image resolution extracted to exported resolveImageForMetadata (no behavior change).
…e are not a script Observed live on astropy__astropy-13033: the model fenced its READ: lines, extractScript shipped them to the jail, and the NameError crash registered as a false 'bug detected'. A fenced block consisting solely of READ: lines now routes to the read round.
…t the image's built /testbed Stage 0 measured the mount substrate killing 6/23 instances (astropy x2, matplotlib, sklearn x2, pytest) with import errors a fresh un-built clone cannot avoid (compiled extensions / generated version files live only in the image's own /testbed). The same 6 scripts were all valid+sound when executed in-image with the gold applied in-container (writable layer, discarded by --rm; __GOLD_APPLIED__ sentinel separates apply-failure from still-failing). REPRO_EXEC=image makes that substrate a first-class mode.
…ttempts) The 56s total backoff lost 5/23 instances to a sustained zai 429 burst at the tail of a conc-3 run (0 model calls, whole instances recorded as infra-error). Rate-limit retries now climb 30s/60s/120s/240s while transient errors keep the short ladder.
…script extraction Observed live on pylint-dev__pylint-7080: a reply opening with two consecutive ```python lines made the non-greedy fence match capture the empty span between them, discarding a complete script as authoring-failed.
… the grading substrate - canary: with the gold patch applied to the tree under test, import <pkg> must succeed AND resolve INTO that tree (exit 3 = site-packages shadow); a canary failure marks the instance env-unresolvable in that mode before any model call is spent - CANARY_ONLY=1: substrate-trust sweep with zero model calls (image mode also skips the host clone) - mount substrate: gold applied host-side up front, so the canary observes the tree exactly as the soundness run mounts it - 429 ladder raised to 60s/120s/240s over 7 attempts (the overnight code-1305 storm zeroed 6/23 instances on the 30s ladder) Measured on the 23-instance Stage 0 backbone: image canary 23/23 pass, mount canary 17/23 (the 6 compiled-package instances fail, matching the prereg amendment's diagnosis).
…ction + guarded repair vs solo swe-structural.mts runs both PREREG-swe-frontier Stage-1 arms on the cached backbone: ARM=system = canary-asserted image substrate, Stage-0 repro reuse (re-verified per instance: validity without gold, soundness with gold in-container), k independent emit-patch attempts, in-image candidate scoring (git apply to /testbed + repro), argmax (repro-pass first, crash-lowest, first-index ties), <=2 strictly-improving repair rounds, then the official swebench judge serialized in Phase B after every arm decision locks. ARM=solo = one emit-patch attempt, the July-protocol reference. - swe-jail.ts: the calibrator's canary-verified jail/zai-ladder/canary primitives extracted verbatim (swe-repro-calibrate now imports them — one jail, no fork) + assertNoHiddenLeak, the transport-chokepoint judge-separation guard (system/user messages only: assistant/tool text is the model's own or reads of the visible tree). - swe-bench-env.ts: opt-in cloneCache — one GitHub clone per instance, local copies per open, so k-attempt sampling cannot die on a clone flake mid-run. Default off, baseline unchanged. - Rows are incremental (OUT.phaseA then OUT post-judge) and both phases resume by instance id.
…ralRollout dials on real HumanEval with the library's held-out gate The first live campaign on the merged machinery: the 'rollout-policy' improve() surface (deterministic neighbor proposer) drives a REAL evaluator — runAgentic + structuralRollout over an inert verifier surface, visible checks via sandboxCheckRunner on a semaphored docker --network=none channel, script-side hidden grading by the nonce-sentinel judge — with fixed disjoint slices (DEV = HumanEval [0,60), HELD-OUT = [60,120)) passed as explicit budget.holdoutScenarios so the library enforces disjointness and its own defaultProductionGate (paired bootstrap, ship iff CI.low > 0.05) makes the ship/hold call. analyzeGeneration: null keeps the improver's context to DEV composites only. Worker default is Qwen2.5-7B-Instruct-Turbo: the original Meta-Llama-3-8B-Instruct-Lite and every 8B Llama variant were retired from Together serverless (model_not_available, verified 2026-07).
…ow-up on a headroom config Identical improve() rollout-policy loop as live-improve-campaign.mts (HumanEval, DEV saturated at 90%, gate HELD), moved to sanitized MBPP where Qwen2.5-7B measures 76.7% at k5/r2/t6 vs an 85.9% pass@5 bound. Shown assert rides task.meta.visibleChecks into officialChecksFromMeta so the official check outranks authored guesses; test_imports prepended for both judges; DEV = usable index [0,150), HELD-OUT = [150,300); MAX_CALLS hard cap + fail-loud model preflight + recompute-from-raw holdout cross-check added. mbpp-structural.mts gains an entrypoint guard (corpus-replay precedent) and exports basePrompt so the campaign imports the loader without launching the benchmark; direct execution is unchanged.
…mpty-holdout crash
When the worker stops answering mid-run (Together HTTP 402 credit exceeded /
429 rate limit / outage), the runtime swallows exhausted-retry shots as null,
so cells silently degrade (calls/cell 15 -> 3) and the run dies hundreds of
cells later at the holdout with a cryptic 'no scorable cells' gate error.
Now: when a cell produces zero candidates (repairStop=no-candidates, every shot
null), probe the worker directly and abort loud with the real HTTP status
(e.g. 402 credit exceeded) so the operator sees the actual cause immediately.
A healthy probe means the null was a one-off and only that cell fails.
Motivated by a live run: the loop completed all 1350 DEV/gen cells and picked
{k9,r2,t6} (+5.3pp DEV over the 82.0% baseline) but exhausted the Together
credit balance at the holdout phase before the gate could render a verdict.
…river over SWE-bench Verified Contract: supervisor-lab/docs/design/stream-loop.md. Base refactor of the Stage-1 structural pilot (swe-structural.mts @ 393ee50) into a running stream: F (frozen-v0: canary, manifest-repro reuse else fresh authoring, k=4 repro-argmax, <=2 guarded repairs) and L (byte-identical + memory recall via promptAppendix over an initially-empty agent-knowledge store, templated outcome-anchored notes, failure-class tally), interleaved per instance, official judge serialized per instance after arm decisions lock. Hard driver requirements (each traces to a measured Stage-1 failure): - per-instance DEADLINE_MS race (29h hang) -> error rows w/ partial receipts, stream continues; deadline reaches into the zai retry ladder - per-candidate TURN_CAP at the transport chokepoint (160-call blowup) -> synthetic no-tool-call completion, candidate scored as-is, receipted - zai conc <=2 total, 429 ladder 60/120/240s, 480s client timeout - explicit image pull -> run -> delete rotation with a hard presence assert; SWEBENCH_CACHE_LEVEL env override (default env) because the judge's cache_level=env silently pruned the 23-image fingerprint cache Ledger: one JSONL row per (instance x arm) with all structural receipts + recall/write receipts + cumulative resolved/$ per arm; events.jsonl incl. the stubbed every-25 batch-look; append-only persisted plan (stable streamIndex across continuations); resume by instance id. Shared protocol lift: repro-authoring prompt/parsers move from the Stage-0 calibrator into swe-jail (byte-identical), zaiChatRaw gains an optional deadlineAt, IMPORT_NAME covers seaborn/xarray.
…125 (5 attempts, backoff) Two defects the stream's all-error run exposed: (1) the docker-failure path read err.stdout but docker writes exit-125/mount/OCI errors to STDERR, so every infra failure surfaced as a bare 'docker: ' with no cause; (2) a transient daemon window (125) fatal-errored the instance — and since erroring instances burn through instantly with no model calls, one bad window nuked all 23 in ~3 min. Now: stderr captured, and 125/daemon-unreachable/overlay/no-space retried with backoff before being surfaced as infra error.
…uced nothing, the case a plan helps most; hold only already-passing + repro-timeout
…ashboard from a stream dir's ledger/events (reproducible, re-runnable for live state)
…with labeled nodes (role/model/harness/transport), and correct the mental model (best-of-k + supervised-repair, NOT supervisor-spawns-workers)
…SWE Docker judge
Fills the EvalRunner hole (marginal-lift.ts): drive each pinned Verified
instance with the multi-turn atom (runAgentic + refine) under the profile's
prompt, capture the git diff, grade it with the swebench Docker judge held
outside the agent, and return { composite: mean(resolved), costUsd }.
The environment/tasks/judge are injected (bench owns them; importing bench
from src would invert the package dependency). The driven system prompt folds
profile.prompt.instructions in after systemPrompt — the surface applyArtifact
mutates for prompt artifacts — so a prompt candidate measurably changes the
worker instead of scoring a fake zero delta. All-judge-failures throws rather
than reporting a fabricated composite.
bench/src/self-improve-swe.mts wires it end-to-end: promptGenerator (one
deterministic candidate via the injected refine seam) -> runLifecycle ->
thresholdPromotionGate(0) -> composeProfile, with an SMOKE=1 import-only path.
…-loop tool builder Replace the thin builder/author prompts with one shared senior doctrine (src/improvement/optimizer-prompt.ts): diagnose the dominant failure mode, state a hypothesis with a predicted lift, decompose into checkable sub-goals, design to isolate the mechanism, generalize past the shown findings, preserve what works, verify for real, then reflect. Seeded from GEPA's REFLECTION_SYSTEM and the /evolve, /pursue, self-improving-loop skill docs; embedded by toolBuildPrompt / mcpBuildPrompt / defaultBuildPrompt and by authorStrategy's authoring instruction. Add driverLoopGenerator: the driver->worker CandidateGenerator on the shipped atom (runBrainLoop + ToolLoopChat, the same seam driverAgent runs on). A driver LLM authors each worker instruction, observes the session's diff / files / verifier output, rates it, and decides refine / re-scope / decompose - replacing agenticGenerator's canned EMPTY_TREE_NOTE/failureNote respawn. Workers stay runLocalHarness in the candidate worktree; agenticGenerator is kept intact as the offline path. worktreeBuildCandidate wires the driver loop as the tool/mcp build default whenever a driver brain is configured. Completion oracle stays code-owned: after the driver stops, ground truth (dirty tree + raw-trace evidence + verifier exit) decides applied, never the driver's prose. Scripted-brain unit tests prove instruction threading, refine on red verifier, fail-closed gating, and the session cap.
…P into the scorer Phase 3 of the self-improve loop: a worktree-BUILT MCP server (stdio, cwd on the host) was unreachable by every shipped backend — none spawned profile.mcp stdio servers same-host, so a built tool could never be scored live. - connectStdioMcp (runtime/stdio-mcp-client): the ONE persistent newline-JSON-RPC spawn+handshake, extracted from mcpServeVerifier's probe; the verifier is rebased on it so 'verified it serves' and 'served while scored' can never drift. Spawn faults stay McpSpawnFault (setup bug, thrown); crash/timeout stays a failed candidate with the stderr tail. - materializeLocalMcp: spawn every enabled stdio server in profile.mcp, namespace tools <server>__<tool> (provider-safe schemas via the shared sanitizeMcpToolSchema), fail-closed when a declared server cannot boot. - localSandboxClient + resolveSandboxClient backend:'local': the same-host pseudo-box — router-brain tool loop (runBrainLoop) with the profile's MCP tools live; profile arrives per-create on backend.profile; delete() kills the children. Event protocol matches inlineSandboxClient. - sweEvalRunner opts.materializeMcp: overlay the live MCP tools onto the driven surface (tools()/call()) for the eval's duration, close in finally. Default stays prompt-only. - bench self-improve-swe.mts SURFACE=mcp: buildableGenerator over worktreeBuildCandidate (driver-loop brain via routerBrain), ranked/gated by the SAME eval runner with materializeMcp on. Verified: tsc green (root+examples); vitest 1318 passed 0 failed; chain smoke mcpServeVerifier -> applyArtifact -> materializeLocalMcp round-trips a live cwd-bound tools/call.
…sers consume real runs
Phase 4 of the self-improve loop: kill the trace-ingestion rot between what
the loop RECORDS and what its analysts READ.
- campaign-otlp: the missing converter from the campaign trace writer's
per-cell spans.jsonl ({name, cellId, startMs, durationMs, ...attrs}) to the
OTLP-flat JSONL OtlpFileTraceStore/the trace-analyst registry parse, plus
campaignTraceResolver(runDir) — the resolveTraces implementation for
traceAnalystProposer/haloProposer: proposing generation g reads gen-(g-1)
(baseline for g=0) under the same runDir the loop records into. One trace
per cell keyed on the cell's on-disk path (the same cellId recurs across
baseline and every candidate); deterministic FNV folds to 32/16-hex ids.
- findings: isAnalystFinding + toAnalystFindings replace the blind
'as AnalystFinding[]' cast in improvement-driver — real findings pass by
reference, untyped seeds are lifted into makeFinding envelopes (claim = the
curator extraction order, original under metadata.raw), garbage dropped
without throwing. Build prompts can no longer render undefined claims.
- improve(): rawTraceDistiller is now the DEFAULT analyzeGeneration for
durable runs (real runDir — where the traces live); the in-memory digest
distiller stays for mem:// runs and now emits TYPED AnalystFindings too,
so exactly one findings shape crosses the wire. rawTraceContext becomes the
explicit override in either direction.
Proof: vitest drives the real published runCampaign recording -> resolver ->
OtlpFileTraceStore (2 traces indexed, tool names visible) ->
traceAnalystProposer returns a candidate, offline via its analyze/fetch seams.
Converter lives in this repo because agent-runtime-swe consumes the published
agent-eval (0.108.0); lifting it into agent-eval next release is the follow-up.
… via stdio MCP
Adds the 'memory' ArtifactKind + AgentMemorySpec payload. applyArtifact
mounts it twice: the typed spec on profile.metadata.memory (local
extension — the published AgentProfile has no memory field yet) and a
live stdio server entry on profile.mcp, so the same-host client
(materializeLocalMcp) boots the memory during a scored run and
sweEvalRunner({ materializeMcp: true }) measures memory-as-treatment
lift with zero scorer changes.
The memory MCP server (memory_search / memory_get, deterministic
lexical retrieval) serves on the extracted createStdioToolServer core
(committed here; the memory server is its first consumer), with a bin
entry (agent-runtime-memory-mcp) resolved per install: dist/mcp/
memory-bin.js under node when built, the TS source through tsx in
dev/test. Fail-closed: an empty memory never serves; a broken store
fails the materialization instead of faking the ablation.
Lesson ingestion: memoryArtifactFromLessons /
memoryArtifactFromCuratedSurface adapt agent-eval's
memoryCurationProposer block (markers duplicated by convention —
flagged) into memory artifacts; memoryGenerator is the native
CandidateGenerator (observed findings only — judge-derived rows are
firewalled; carried memory accumulates; no-change generations emit
nothing). AgentMemorySpec.logPath writes a JSONL row per memory_search
— the retrieval-log seam for agent-knowledge's RetrievalHoldout, which
stays cross-package (not a dependency of this repo).
Also fixes a stale improvement-driver test: its partial report-finding
fixture now conforms to the P4 toAnalystFindings pass-through contract
instead of asserting a pre-lift id.
…or the research optimizer
- 'connection' ArtifactKind: an ExternalMcpGrant (http url / stdio launch,
literal headers/env, secrets by provider KEY NAME, optional hub grant)
promotes onto profile.mcp[key] via connectionMcpServer and onto
profile.connections (artifact wins per connectionId)
- buildableGenerator: BuiltCandidate.remote branches the mcp emit off the
hardcoded stdio transport — the adopt-not-build path emits a
{ transport:'http', url, headers, env } server with adopt provenance
- KeyProvider (runtime/key-provider.ts): declarative secretEnv
(mcp[key].metadata.secretEnv, names only) resolved at materialize time;
envKeyProvider reads the dotenvx-loaded env; materializeLocalMcp injects
values into the spawned server child env only — fail-closed on a missing
provider/key, values never on the profile or in logs; sweEvalRunner
forwards keys so an adopted server scores with its credential live
- research seam: driverLoopGenerator gains an optional research{query} tool
+ researchDriverNote (adopt-before-build doctrine), offered only when
wired; mcpBuildPrompt instructs adopt-over-build; no live web backend
ships yet (the seam is worktreeBuildCandidate driver.research)
The e2e driver imports sweEvalRunner, which lives in ../src and resolves only through bench/tsconfig path aliases (tsx applies them at cwd=bench). Running tsx bench/src/... from the repo root resolved the published agent-runtime in bench/node_modules and crashed on the missing export. Add a bench package script (self-improve) so the invocation always runs at cwd=bench, and document the requirement in the driver header.
…ves prompts from them
…/repeat on the SWE judge New bench/src/find-lift-swe.mts closes the intelligence loop across rounds: each round scores the current profile on the held-out set (sweEvalRunner), autopsies the failing rows into ranked mechanism findings (autopsySweFailures), measures a finding-authored population (reflectiveSweRefine + authorDiverseSweSeeds through promptGenerator, gated by thresholdPromotionGate(0) on the Docker judge held outside the agent), then composes the round's promoted winners onto the profile for the next round (composeProfile). Converges early when a round promotes nothing and the findings are unchanged; else runs ROUNDS. Writes per-round JSON (composite, findings, per-candidate lift/promoted, composed instructions) and prints a final trajectory table. SMOKE=1 is import-only. Exports the shipped reflective-swe primitives (autopsySweFailures, reflectiveSweRefine, authorDiverseSweSeeds, + SweFailureRow/AutopsyOptions/ ReflectiveSweOptions/FailureMechanism/FAILURE_MECHANISMS) from the lifecycle barrel so the bench resolves them through the same path alias as sweEvalRunner.
…back The autopsy's LLM classifier is enrichment, not a hard dependency: a transient router 502/503 mid-run was throwing away a whole scored round. Catch the classification failure and fall back to an empty map — every failing row then resolves to its deterministic mechanism (empty diffs detected without the model) via normalizeMechanism/defaultEvidence, so the round still ranks failures and proposes candidates.
…to no-candidate The candidate author needs the model (a targeted prompt is not deterministic), so unlike the autopsy it cannot fall back. Wrap its structured call in a wall-clock retry budget (REFLECT_AUTHOR_RETRY_BUDGET_MS, default 240s) with jittered backoff; if the router is genuinely down past the budget, return [] — a legitimate no-proposal round that scores baseline and promotes nothing, never a crashed run that discards the round's rollouts.
…02s on schema) Diagnosed: the Tangle router returns 502 (CF gateway) on schema-enforced structured output for the reflection calls — haiku rejects json_schema outright, glm times out on author-sized json_schema requests. Both models return clean json_object. Switch autopsy + author to jsonMode with the exact shape pinned in the prompt and validated on parse (the defensive filters already tolerate a missing/extra field), so the autopsy->evolve calls actually get through.
A transient router 5xx makes the worker return 0 completions -> 0 tokens, which sweEvalRunner scores as 0/resolved. Without a guard, a blip silently produces a fake all-zero baseline and the whole round (autopsy + paid candidate arms) is measured against it - wasting money on corrupted science. Abort the round when >50% of baseline rollouts return 0 tokens; re-run when the provider recovers.
…t arm runLifecycle always re-scored its own baseline arm. A round driver that already scored the identical baseline profile (to read its failures for an autopsy) then paid for a second, redundant scoring of the same profile every round. Add an optional baselineResult to RunLifecycleOptions; when supplied, use it as the shared 'without' arm instead of re-running. The find-lift driver now passes its step-(a) baseline through — a full arm of compute saved per round.
…ot in-sample The driver autopsied AND scored candidates on the SAME task set, so a proposed instruction was tuned on the exact tasks it was measured on (train-on-test) — the reported lift was in-sample and optimistic. Split into TRAIN_IDS (the autopsy learns from + the search optimizes on) and a DISJOINT frozen HOLDOUT_IDS the autopsy never reads; certify the composed profile on the holdout each round. The reported lift is now (composed on holdout) - (bare baseline on holdout) = a real generalization number. Guards the holdout baseline against dead-worker blips too.
runAgentic throws 'all-children-down' when a transient router 5xx downs every shot for a task. evaluateTask did not catch it, so ONE task's blip crashed the entire eval (and the whole find-lift round) mid-flight. Catch it: score that task 0 with 0 tokens + carry the error, so the aggregate dead-worker guard decides on the batch (abort a mostly-dead round, tolerate a single blip) instead of an uncaught crash. Makes the eval robust to intermittent provider weather.
When the search promotes 0 candidates, the composed profile is byte-identical to the one that produced the holdout baseline. Re-scoring it and subtracting reports pure worker run-to-run VARIANCE as 'lift' — a false positive (a run just printed '+25pp (real, frozen)' with 0 promoted / 0 instructions changed: 12.5% vs 37.5% was the same bare profile scored twice). Only re-score + report a holdout lift when the profile actually changed; otherwise the lift is 0 by construction.
…ev/swe rigs - hev-improve: on failure the judge notes now carry the Docker checker's traceback/assertion tail (last 800 chars) plus the model's emitted code (first 700 chars) instead of the single word 'failed'; rawTraceContext stays off deliberately (the prompt-tier gepaProposer cannot run grep over trace paths — evidence arrives via notes/emitted instead) - hev-improve: GENERATIONS/POPULATION defaults 1/2 -> 6/4 so the default run exercises the Pareto/combine path (generations=1 never fired it) - swe-improve: judge notes widened from the 200-char head of the swebench report JSON to 1500 chars (the report is flat — no separate failure section to extract)
…at equal budget Fast, self-auditing research instrument: HumanEval, local subprocess grading (seconds/task, no Docker clone), equal-budget IMPROVER (blind self-refine) vs CAPABILITY (run_python tool) arms, forced final answer, per-unit resilience, paired McNemar, empty-final-code artifact detection. Finding (EXP-038): NULL at power — a code-execution tool does not beat blind self-refinement at equal budget on HumanEval (gpt-4o-mini n=492: 86.4% vs 88.0%, p=0.30); the headroom-conditional law stays untested at the large headroom where the SWE prior lives.
…rovement loop (round 4) (#575) * feat(bench): swe-arena typed replay of the solo-vs-supervisor head-to-head Milestone 1 of the swe-arena unification: load the committed head-to-head artifacts (12 paired SWE-bench Verified runs, glm-5.2 solo vs supervisor) and reproduce the reference analyze.py analysis exactly, pinned as tests. - types.ts: LedgerRow/RejudgeRow/RematchRow/ArmSpec/HoldoutRegistry mirroring the fixture schemas; SweInstance for the task-meta fields we consume. - reconcile.ts: verdict reconciliation as pure functions — personal re-judges (*-final/*-final2/my-reverify) override automated verdicts; gold rows that the judge cannot resolve mark the instance ungradeable and shrink the denominator (2931 + 2317 excluded -> solo 7/10 vs sup 4/10). - analyze.ts: discordant pairs, exact sign test via agent-eval mcnemar (p=0.25 on 3 solo-only wins), cost/wall rollups with both token views: brain-only 560,554 (0.83x, what analyze.py printed) and true spend 1,722,390 incl. 1,161,836 worker tokens (2.55x solo). - replay.mts: CLI whose reference section is byte-identical to analyze.py output (diff-verified), plus reconciled verdict, rounds progression (matplotlib 0-line unresolved at SUP -> resolved at SUP4), holdout registry. - replay.test.mts: 56 vitest tests pinning sha256 of every fixture and every oracle number above; bench-scoped vitest.config.ts (root config excludes bench/**, other bench tests are tsx scripts). Fixtures copied verbatim from the experiment scratchpad (secret-scanned); sup-journal-true.json is derived from the supervisor journals with the same extraction as analyze.py so the token rollup survives scratchpad deletion. * feat(bench): swe-arena M2 — typed execution path proven by 2-patch judge parity Replaces the bash head-to-head harness (setup-ws/calibrate/judge/solo/sup4/ probe-capacity/orchestrate) with typed modules on top of the M1 replay: - materialize.ts: workspace from instance images (docker create/cp → reset --hard base → work branch); image path stays default — clone drops the environment_setup_commit build state (proven live in evolution round 2) - calibrate.ts: dual gate — repro (base FAILS, gold PASSES; git apply → patch --fuzz=3 fallback) + official-judge gold gate via adapter.judge; experiment-valid only if both hold - serialized-judge.ts + judge-child.mts: one judge at a time (in-process + cross-process pid lock — the pre-flock race caused a proven false-negative on psf__requests-1766), enforced 1800s ceiling floor (700s caused 3 spurious timeouts on requests-2317), stale-container pre-clean inside the mutex, one retry on empty/parse-fail, resolved:null when inconclusive - arms.ts: solo (opencode + oc-jsonl usage parse) and supervisor (wraps the real loops driver via the now-tracked run-supervisor.mjs shim; env knobs incl LOOPS_BRAIN_RETRIES/DRIVER_DEADLINE_MS) arm execution; patch extraction with the vendored excludes; cost recovery = journal metered events + workers-ndjson cwd → opencode sqlite join (worker-tokens.json provenance, verified byte-exact on flask-5014: 46737); armProvenance records the live loops commit + env knobs - capacity.ts: EndpointCapacityGate (k-of-n, steadyM, wait ceiling) with BOTH probe paths — z.ai coding (worker) and router.tangle.tools (brain); supervisor arms gate on the router path (3 infra-null rounds came from probing z.ai only) - run-experiment.mts: sequential paired runner with ledger-skip resume, serialized judging, M1-schema LedgerRow append, 15s cooldown; --dry-run-parity replays extraction + judging from committed patches with zero model spend Parity gate (ran for real, docker only): pallets__flask-5014 solo → resolved=true (22s, attempts=1) and pydata__xarray-4687 sup → resolved=false (71s, attempts=1), matching the pinned M1 ledger/rejudge verdicts; changed-file sets preserved through re-extraction. Full 12-instance arm parity re-run is a TODO gated on operator approval (~$1 + 4h). tsc: 34 pre-existing errors before and after, 0 in src/swe-arena. vitest: 93 passed (56 M1 + 37 new) + 3-test docker parity suite (opt-in via SWE_ARENA_PARITY=1, verified green twice). * feat(bench): round-4 outer loop — improve() in the optimizer seat over the loops supervisor One runRound() = one improve(surface:'code') call on the loops repo: DIAGNOSE (blind multi-model analyst ensemble over prior-round failure artifacts, fused union + agreement-rank, minority findings kept as competing hypotheses, composed with rawTraceDistiller at the analyzeGeneration seam) -> PROPOSE (agenticGenerator constrained to the declared change-space: extensions/pi/** + src/{worker-evidence, best-effort,worker-clone}.ts, enforced in the shot verifier AND fail-closed pre-spend in the dispatch) -> EVALUATE (each candidate commit gets a detached loops eval worktree; supervisor arm + serialized official judge over the 3-instance improvement set; frozen round-3 arm params asserted immutable) -> ACCEPT/REJECT (protocol_v2 keep-if-better with the +20% cost guard; gate always holds — the pre-registered 6-instance holdout runs only on operator approval; every candidate persists as a staircase row in supervisor-lab .evolve/rounds). Calibration hardening measured live on the round-2 django SUP2 smoke: one transport retry (router 524 storm observed), temperature spread for same-model analysts (temp-0 duplicates were byte-identical via edge cache), 16k analyst token ceiling (glm-5.2 reasoning consumed a full 8k ceiling leaving empty content), z.ai endpoint fallback flag. Deps: pin agent-eval/agent-runtime to the local file: builds (0.121.0 / 0.94.13) — improve()'s code surface, promotionGate, dispatchTimeoutMs and rawTraceDistiller do not exist in the old published pins. The refresh drifts 4 pre-existing type errors in retired experiment rigs (34 -> 38 total, all in files already red); swe-arena stays at 0. Tests: +37 vitest units (fusion, analyst-output parsing, change-space enforcement incl. traversal/rename edge cases, staircase row schema, keep-if-better verdicts, frozen-arm assertion, prompt contract). * feat(swe-arena): metadata bootstrap, post-gate dispatch clock, repsPerInstance Rebuild after a host reboot wiped the experiment scratchpad (/tmp), and fix the pre-crash abort: - bootstrap-meta.mts: regenerate task-meta.json / problem statements / dual calibration straight from the SWE-bench Verified dataset via the bench adapter — the scratch task-meta.json hand-file dependency is gone for good. Honesty firewall preserved: --problems emits problem statements only; gold patches surface only on the mechanical calibration path. - fixtures/verify/: the 3 improvement-set verify scripts, re-authored from problem statements only and dual-calibrated (repro base-fail/gold-pass AND official-judge gold gate; all 3 experiment-valid). Committed as fixtures — the durable home — and defaultRound4Config now points at them. - Dispatch clock fix: the campaign clock raced the ENTIRE dispatch including capacity-gate waits, so a 58-min gate hold was billed to the 7200s cell clock and the astropy baseline cell aborted 'rejected-incomplete'. The cell work clock (runWithPostGateClock) now starts after the gates clear; the campaign clock is widened by the worst-case gate holds as a backstop. - repsPerInstance (round 4 default: 2): replicate cells per instance; an instance counts resolved only when ALL replicates resolve (single-rep scoring flips instance outcomes run-to-run). Per-rep run dirs, staircase rows carry rep, parent resolved-count read from the parent's own record. - analyzeGeneration skips seed artifact dirs that no longer exist (wiped scratch must not feed empty bundles to the diagnosis analysts). Tests: 143 passed (130 baseline + 13 new units), tsc adds 0 errors to swe-arena. * fix(swe-arena): purge gitignored proposer dirt before finalize + salvage gen-0 candidates The round-4 gen-1 run died at the substrate's finalize-time integrity check: the cand-1 proposer ran a real pnpm install inside its CodeSurface worktree (38k node_modules paths). Ignored paths pass the change-space check (git status honors .gitignore) but verifyCodeSurfaceWithGit rejects ANY extra path, ignored included. purgeIgnoredArtifacts (git clean -Xdff) now runs as step 1 of loopsCandidateVerifier — the last our-code touchpoint before finalize — preserving tracked edits and untracked deliverables. Arm evals already ran in detached eval worktrees; that seam was not the dirt source. Tests: purge semantics + tree-hash invariance of a finalized candidate worktree across a mocked evaluation. Salvage: both machine-proposed candidate diffs recovered from unreachable commits (pinned as refs/salvage/* in the loops repo) with provenance README. * fix(swe-arena): pin the gate baseline, double author-shot timeout, add launch guards Gen-1 close-out (r4-mroh3rkt) graded 'winner 0/3 vs baseline 0/3' while the reps-confirmed measured baseline was 1/3 — a bookkeeping fault, not variance: improve() resumes its campaign from runDir and replays cached baseline cells WITHOUT dispatching them, so the in-process RoundRecorder labeled the first dispatched CANDIDATE (b08d31c910) as the baseline and published its partial cells as the baseline record (loop-provenance baselineSearchComposite=0.5 proves the campaign layer kept the true baseline verdicts throughout). - pinnedBaseline config (iid -> reps-confirmed bool): the accept/reject gate and staircase parent counts compare against the PIN, never a per-run recomputation; contradicting run cells log a loud BASELINE-DRIFT warning with both values and the pin still wins. Malformed pins fail at t=0. - RoundRecorder identifies the baseline by candidateCommit == baseCommit instead of dispatch order, so a resume can never mislabel a candidate. - proposerTimeoutMs default 20 -> 40 min: the agenticGenerator per-shot clock ('author shot timed out') fired 3x under degraded capacity in gen-1. - Launch guards on the spend path: refuse without TANGLE_API_KEY+ZAI_API_KEY (names dotenvx), and a pid-file single-instance lock per outDir with a staleness check so two outer-loops can never race. - --write-config --out-name <dirname> so a new generation gets a fresh campaign home instead of resuming the completed one. * fix(outer-loop): stop ambient ANTHROPIC_API_KEY from hijacking claude author shots Three gen-2 launches died fast with 'agenticGenerator: author shot exited with code 1' and no stderr on disk. Reproduced standalone: under the run's dotenvx env (agent-state.env) the claude CLI prefers the injected ANTHROPIC_API_KEY over the claude.ai login and exits 1 — 'API Error: 400 You have reached your specified API usage limits' (key org capped until 2026-08-01). Same command with the var unset returns rc=0. - runHarness seam: strip ANTHROPIC_API_KEY/AUTH_TOKEN/BASE_URL from the shot subprocess env (claude harness only; rest of the run untouched). - onShotCompleted seam: persist every shot receipt + bounded stdout/stderr tails to <outDir>/proposer-shots/ so the next failure names its cause.
Resolution notes: - src/lifecycle: accept main's deletion (#564 unified activation); drop the branch's lifecycle extensions and the two superseded rigs that consumed them (bench/src/self-improve-swe.mts, find-lift-swe.mts) — swe-arena supersedes both. - src/improvement: main's evolved relands win as the base; re-add the branch's rollout-policy surface, optimizer-method prompt, typed distiller findings, durable-run raw-trace default, and the shared-client mcpServeVerifier. - src/runtime/stdio-mcp-client: port main's #508 async stdin-EPIPE fast-fail into the shared connection (regression caught by tests/mcp-serve-verifier). - bench: main's published-package layout kept (no bench lockfile — installs via the root pnpm workspace); add agent-knowledge dep; keep the branch's structural rigs; teach run-package-tests.mjs to partition node:test vs vitest. - docs/api regenerated via docs:api; examples/open-source-skill-lifecycle removed with the lifecycle module.
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 1c6212a6
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-20T23:22:48Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟠 Value Audit — better-approach-exists
| Verdict | better-approach-exists |
| Concerns | 8 (1 medium-concern, 3 low, 4 weak-concern) |
| Heuristic | 0.2s |
| Duplication | 0.0s |
| Interrogation | 515.5s (2 bridge agents) |
| Total | 515.7s |
💰 Value — better-approach-exists
A coherent, mostly in-grain change that closes the self-improvement loop end-to-end (typed SWE harness + improve() in the optimizer seat + the library surfaces it needs), but it ships one clear in-package duplication: createStdioToolServer is a verbatim copy of createMcpServer's JSON-RPC loop that c
- What it does: (1) Adds a 'rollout-policy' surface to improve() so the held-out gate can tune the existing StructuralRolloutPolicy dials {k, repairRounds, testgen} that src/runtime/structural-rollout.ts already reads (that file's own docstring at line 24 anticipated 'may later tune StructuralRolloutPolicy as an optimizable surface'). (2) Adds a driverLoopGenerator CandidateGenerator that puts an LLM driver on th
- Goals it achieves: Make the runtime's own inference dials (structural-rollout) and code reachable from the ONE held-out-gated improvement verb, so self-improvement measures real architectures (solo vs supervisor, cheap+verify vs frontier) instead of assuming them — and so a built MCP server, once verified, is served live during scoring rather than only probed. Once merged, improve() can tune k/repairRounds/testgen u
- Assessment: Mostly in the grain of the codebase. The new code correctly reuses the canonical seams: driverLoopGenerator rides runBrainLoop/ToolLoopChat (not a new loop); the stdio and HTTP MCP paths share sanitizeMcpToolSchema; mcp-serve-verifier was properly refactored to delegate to connectStdioMcp; the 'local' sandbox backend slots into the existing resolveSandboxClient switch alongside sandbox/bridge/rout
- Better / existing approach: Searched: (a) grepped canonical-api.md for the documented recommendation on rollout-policy — line 89 (unchanged by this PR) says 'Workflow and rollout-policy files are code/config changes; use the code surface plus agent-eval's parameterSweepProposer for JSON sweeps'; parameterSweepProposer is a real export (docs/api/primitive-catalog.md:1336, 'Config/parameter-level proposer for FAPO's middle esc
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound-with-nits
Coherent extension of the established improve()/selfImprove spine — rollout-policy surface wired end-to-end, local-MCP pseudo-box + memory server each have reachable consumer surfaces, and the swe-arena outer loop drives the existing RSI verb from a documented bench entrypoint; only minor doc/versio
- Integration: All new public surface routes through src/improvement/index.ts and src/runtime/index.ts. The 'rollout-policy' surface is wired into defaultGeneratorFor (improve.ts:233), materializeImprovementProfileCandidate (improve.ts:565), and consumed by bench/src/live-improve-campaign.mts:543 and live-improve-campaign-mbpp.mts:543. localSandboxClient is reached through resolveSandboxClient({backend:'local'})
- Fit with existing patterns: Matches every established pattern. improve() is the documented one-verb RSI facade (CLAUDE.md, docs/canonical-api.md:89); rollout-policy is added as a fourth built-in default generator in the same shape as prompt/skills/memory. localSandboxClient sits alongside inlineSandboxClient/inProcessSandboxClient as another pseudo-box adapter behind the same resolveSandboxClient switch (resolve-sandbox-clie
- Real-world viability: The outer-loop has the right hardening for realistic use: serialized-judge.ts provides a mutex + retry around the official SWE-bench evaluator, dispatchDeadlineMs/costGuardRatio bound per-cell spend, the promotion gate is operator-gated by design ('holdout-approval-required', outer-loop.mts:1328-1334), and the change-space verifier (changeSpaceViolations, outer-loop.mts:130) is enforced both in th
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
🔎 Heuristic Signals
🟡 Cruft: console debug added bench/scripts/run-package-tests.mjs
+console.log(
🟡 Cruft: todo added bench/src/swe-arena/parity.test.mts
- TODO(operator approval): the full 12-instance ARM parity re-run (typed path
🟡 Cruft: magic number added bench/src/supervisor-arena.mts
const inContainerSecs = Math.ceil(dockerTimeoutMs / 1000) + 2
💰 Value Audit
🟠 createStdioToolServer duplicates createMcpServer's JSON-RPC loop without delegating to it [duplication] ``
src/mcp/tool-server.ts:73-174 (createStdioToolServer) and src/mcp/server.ts:155-309 (the serve loop inside createMcpServer) are two independent implementations of the same newline-delimited JSON-RPC 2.0 stdio MCP server in the same package: identical PROTOCOL_VERSION ('2024-11-05'), identical handle/serve/stop structure, and byte-identical rpcResult/rpcError/writeResponse helpers (tool-server.ts:176-195 ≈ server.ts:311-330). The tool-server.ts docstring even says 'Extracted from createMcpServer
🟡 rolloutPolicyProposer hand-rolls the sweep the substrate's parameterSweepProposer already provides [better-architecture] ``
docs/canonical-api.md:89 (unchanged by this PR) recommends for rollout-policy: 'use the code surface plus agent-eval's parameterSweepProposer for JSON sweeps', and parameterSweepProposer is a shipped export (docs/api/primitive-catalog.md:1336). src/improvement/rollout-policy.ts:146-211 (enumerateNeighborPolicies + rolloutPolicyProposer) hand-rolls the equivalent bounded single-dial sweep for the three rollout dials. Most of rollout-policy.ts is needed regardless (serialization, normalize, profil
🟡 PR bundles many independently-shippable library surfaces with a large bench suite [proportion] ``
113 files / +23.5k/−5.4k. The library surfaces (rollout-policy, memory server, stdio MCP client, local sandbox client, driver-loop generator, campaign-otlp) are each independent enablers for different facets of the self-improvement loop, and the bench side adds a 43-file swe-arena harness plus david-goliath/supervisor-arena/swe-stream/live-campaigns rigs. Each library piece is coherent on its own and the bench rigs are separate experiments; a reviewer assessing any one piece must absorb all of t
🎯 Usefulness Audit
🟡 index.ts docstring overstates driverLoopGenerator wiring [ergonomics] ``
src/improvement/index.ts:11 and the driver-loop-generator.ts header describe driverLoopGenerator as 'default for tool/mcp', but defaultGeneratorFor (src/improvement/improve.ts:221-239) only returns defaults for prompt/skills/memory/rollout-policy. Tool/mcp/hooks/subagents still throw the ConfigError requiring an explicit opts.generator. The live behavior is fail-loud and safe; the comment overstates integration. Either wire it into defaultGeneratorFor or correct the docstring to 'pass via opts.g
🟡 bench pins agent-knowledge major 1 while runtime pins major 4 [robustness] ``
bench/package.json:45 adds '@tangle-network/agent-knowledge: ^1.11.1' (resolves to 1.12.1, which transitively pulls agent-eval@0.115.3 — pnpm-lock.yaml:723/1864), while the workspace runtime declares '^4.1.0' (resolves to 4.1.0, agent-eval@0.122.8). bench/src/swe-stream.mts:67 imports {applyKnowledgeWriteBlocks, buildKnowledgeIndex, initKnowledgeBase, searchKnowledge} directly from this package; under the bench's own declared range, the v1 API shape may not match those names. Likely needs '^4.x'
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
❌ Needs Work —
|
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
❌ 3 Blocking Findings — 1c6212a6
Full multi-shot audit completed 8/8 planned shots over 113 changed files. Global verifier still owns final merge decision.
Full immutable report for this review: trace
Summary comment for this run: full summary
tangletools · 2026-07-21T00:20:06Z · immutable trace
2 participants
Results first (what closed PR #544 lacked)
Every claim below was produced by the harness in this diff, graded by the official SWE-bench evaluator, and re-verified by hand.
improve()in the optimizer seat): the machine-authored candidate resolved 2/3 vs 1/3 for the pinned baseline on the round's improvement set. Each resolved claim was re-judged personally through the official evaluator and the winning patches contain 0 test-file hunks (no test-gaming). Holdout certification is still running and will be posted to the experiment log before any promotion claim.docs/results/glm52-solo-vs-supervisor-swe-paired.mdintangle-network/supervisor-lab(branchfeat/harness-aware-capabilities).What's in the diff
61 commits, 113 files, +23.5k/−5.4k against
main(merge oforigin/mainalready resolved in-branch;git merge-treeproves a clean merge).bench/src/swe-arena/— the reviewable core (43 files, ~7.6k lines incl. tests): the typed SWE experiment harness.outer-loop.mts):improve()drives the loops supervisor itself — pinned gate baseline, post-gate dispatch clock, author-shot guards.src/improvement/additions:rollout-policyimprove surface, senior scientific-method optimizer prompt (optimizer-prompt.ts), driver-loop generator, typedAnalystFindingdistiller envelopes, durable-run raw-trace default, campaign→OTLP trace export, findings helpers.src/runtime/stdio-mcp-client.ts: the shared same-host stdio MCP connection (verifier and live serving share one code path), including the async stdin-EPIPE fast-fail from fix(improvement): handle MCP verifier stdin pipe errors #508 ported into the shared client.bench/src/hev-improve.mts,mbpp-structural.mts,swe-structural.mts, …): the experiment-rig history that produced the structural-lever results feeding swe-arena's design.Merge-resolution notes (the branch predates several main-side relands of its own work):
src/lifecyclewas deleted on main (feat(improvement): unify measured candidate activation #564); this branch accepts that deletion and drops its lifecycle extensions plus the two superseded rigs that consumed them (self-improve-swe.mts,find-lift-swe.mts) — swe-arena supersedes both.benchkeeps main's published-package layout. There is intentionally nobench/pnpm-lock.yaml(main's convention): bench installs via the root pnpm workspace —pnpm installat the repo root is all a consumer runs.bench/scripts/run-package-tests.mjsnow partitions node:test files from vitest files (the swe-arena suite is vitest) instead of crashing vitest suites undernode --test.Verification (run on this exact merge)
tsc --noEmit0 errors, examples tsconfig 0 errors, biome clean, vitest 1633 passed / 2 skipped, 157/157 files,verify:package+docs:checkfreshness green (API docs regenerated).typecheck:public0 errors; full benchtschas 0 errors insrc/swe-arena(remaining errors live only in non-CI experiment-rig.mtsfiles that predate main's API changes);run-package-tests31/31 files (25 node:test + 6 vitest) + Pier bridge; packed-consumer verify green against local runtime.Review guide
Start and end with
bench/src/swe-arena/— that is the deliverable.types.ts→replay.mts→execution→outer-loop.mts, with the test files as the spec. The rest of the bench diff is experiment-rig history retained for provenance; it is not on any CI gate and does not ship in the public package surface (typecheck:publiccovers what ships).Merge strategy
Recommend squash-merge — collapses historical commit trailers per repo policy.
🤖 Generated with Claude Code