[SUPR1M3]

Closes #116

Summary

Adds full_rewrite_probability. When diff_based_evolution: true, each iteration asks for a full rewrite instead of SEARCH/REPLACE diffs with probability p. The default 0.0 keeps current behaviour.

diff_based_evolution: true
full_rewrite_probability: 0.2   # 20% of iterations ask for a full rewrite

evolution_trace:
  enabled: true
  include_failures: true        # optional: log failed iterations too

Changes

• Mode selection: the worker picks each iteration's mode with a generator seeded from (random_seed, iteration). Runs stay reproducible whichever worker handles an iteration, and the ensemble's model sampling (Seed ensemble model sampling per iteration in worker processes #559) is untouched.
• Mode drives prompt and parsing: the chosen mode decides both the prompt template and how the response is parsed.
• Mode is recorded: it is stored on the iteration result (failures included), in program metadata (metadata["evolution_mode"]), in the evolution trace, and as the template key passed to log_prompt.
• Validation: checked in Config.from_dict and in the controller, so configs built in code are covered too. The probability must be in [0, 1], it requires diff_based_evolution: true, and it can't be combined with prompt.programs_as_changes_description.
• evolution_trace.include_failures (off by default): iterations that fail before producing a child are written to <trace>_failures.jsonl with their error and mode. With include_prompts, the prompt and LLM response are included too. Worker errors raised after the LLM call now carry the prompt, response, parent and island, so parse failures can be inspected.

Tests

• tests/test_full_rewrite_probability.py: mode selection and reproducibility, prompt and parse routing, metadata and trace recording, config validation.
• tests/test_controller_trace_failures.py: runs the controller loop and checks that failed iterations land in the failures file.
• tests/test_evolution_trace.py: tracer support for include_failures.
• Updated test_diff_format_validation_worker.py and test_worker_config_serialization.py for the new field.

python -m unittest discover tests: 1035 tests, OK.

Experiment results

The script and config are on exp/full-rewrite-116, kept out of this PR.

Setup: qwen3:8b (16k context, thinking off) on Ollama, Kaggle 2×T4 · circle_packing (phase 1 config) and function_minimization · p ∈ {0, 0.2, 0.5} · 3 seeds · 50 iterations · 18 runs. The trace was complete: no mode mismatches and no untraced iterations.

Final best score (mean ± SD over 3 seeds)

p	circle_packing	function_minimization
0	0.546 ± 0.014	1.4947 ± 0.0082
0.2	0.631 ± 0.004	1.4974 ± 0.0025
0.5	0.642 ± 0.033	1.4992 ± 0.0004

These final scores overstate the effect:

• In 2 of the 3 p=0.2 circle_packing seeds, the jump to ~0.62 came from a lucky diff at iteration 1, not from a rewrite. Ollama's sampling isn't reproducible with concurrent requests on the GPU, even with byte-identical prompts, so 3 seeds are noisy.
• function_minimization is at its ceiling (~1.5), so differences there are within noise. AUC does rise with p (1.467 / 1.481 / 1.491), and the spread across seeds shrinks.

Per-attempt comparison (circle_packing, first improvement attempts on the initial program)

Mode	Median score	Reached ≥ 0.57	Scored 0
Full rewrite	0.52	11/23	4/23
Diff	0.34	5/61	21/61

Within mixed runs, a full rewrite produced a new best 8% of the time vs 3% for a diff (95% Wilson intervals [4%, 14%] and [1%, 7%]). On function_minimization the two rates were equal (8% vs 7%).

Failure modes

• Parsing: rewrites almost never fail to parse (1–4%). Diff parse failures run 3–15%, and most of them are one error, "Unmatched SEARCH/REPLACE delimiter outside a diff block" (48 occurrences).
• Evaluation: failure rates are similar for the two modes (7–17%).
• Open question: diffs in mixed runs failed evaluation more often than at p=0 (circle_packing: 24/177 vs 4/130). It isn't explained by whether the parent was a rewrite (diff→diff 13% vs rewrite→diff 11%), nor by parent score or prompt length.

Takeaway

Full rewrites help mostly by escaping a weak starting program. A nonzero p looks worthwhile and 0.2 is a reasonable value to suggest. This data can't tell 0.2 from 0.5. A follow-up on circle_packing with 5–6 seeds and p ∈ {0.1, 0.2, 0.5} would settle it. The default stays at 0.0 so existing behaviour doesn't change.

Related

Two bugs found while running the experiment are fixed in separate PRs: #582 (per-iteration LLM request seed, a follow-up to #554/#559) and #581 (children dropped from the trace when their parent left the archive). This PR doesn't depend on either one, and all three merge cleanly in any order.

---

Added for the v0.4.1 release

Main is merged in, and these are on top of the original commit.

Review fixes for include_failures

• Timed-out and crashed iterations now reach the failures file.
• Failed LLM calls now keep the prompt, parent and island, like later failures.
• failures_path now matches the file written when compress is on.
• Dotted trace names no longer collide: run.v2 and run.v3 both wrote to run_failures.jsonl.

Issues folded in

• [Feature request] track llm used to generate each program #381: each child stores the sampled model as metadata["llm_model"], also in the trace.
• HDF5 trace metadata loses list types and cannot distinguish JSON-looking strings #574: HDF5 metadata lists now round-trip, and strings that look like JSON stay strings. The exporter lists its JSON-encoded attributes in __json_encoded_keys__; files without that list load as before.

README: cut from 1,074 to about 200 lines, leading with research adoption and without emojis. Detailed material moved to docs/.

Version: bumped to 0.4.1.

Verification

• Unit tests: 1064 pass, 5 skipped. The skipped ones are the HDF5 tests, because CI doesn't install h5py; they pass locally with h5py installed.
• The new tests fail on the source without these fixes.
• End-to-end with a stub LLM through the real process pool (p=0.5, 20 iterations): every iteration was accounted for once, all 12 children were full rewrites and all 8 failures were diffs.

Closes #116
Closes #381
Closes #574

🤖 Generated with Claude Code