Could having a ratio between diff_based_evolution and full rewrite a good idea? #116

Activity

1. codelion commented on Jul 1, 2025

   

   codelion

   on Jul 1, 2025

   Member

   More actions

   Yes, that was then thought behind having both configs full_rerwite and diff_based_evolution to be enabled at the same time, but in experiments it did not work reliably so I removed it. We can add that in again but will need to validate with some examples to ensure it does help.

   👍React with 👍2Reacted by Brad and Hoang Mai
2. BradKML commented on Jul 2, 2025

   

   BradKML

   on Jul 2, 2025

   More actions

   If we hack the agent to write code in a way that is both robust and modular, maybe? Otherwise diff-based solution would likely work better since often times could have >200 lines
3. SUPR1M3

   added 2 commits that reference this issue on Nov 1, 2026

   chore: experiment script sharding, time budget and trace checks (algo…

   ...

   41e17f9

   feat: mix full rewrites into diff-based evolution (algorithmicsuperin…

   ...

   47b3159
4. SUPR1M3 commented on Nov 1, 2026

   

   SUPR1M3

   on Nov 1, 2026

   Contributor

   More actions

   Results: full_rewrite_probability experiment

   I implemented full_rewrite_probability in #580: with diff_based_evolution: true, each iteration asks for a full rewrite instead of SEARCH/REPLACE diffs with probability p. The default 0.0 keeps current behaviour. Each iteration's mode is stored in program metadata (evolution_mode) and in the evolution trace. Failed iterations can optionally be logged to <trace>_failures.jsonl (evolution_trace.include_failures).

   To check whether it helps, I ran a small grid. The script is on exp/full-rewrite-116.

   Setup: qwen3:8b (16k context, thinking off) on Ollama, Kaggle 2×T4 · circle_packing (phase 1 config) and function_minimization · p ∈ {0, 0.2, 0.5} · 3 seeds · 50 iterations · 18 runs. The trace was complete: no mode mismatches and no untraced iterations.

   Final best score (mean ± SD over 3 seeds)

   p	circle_packing	function_minimization
   0	0.546 ± 0.014	1.4947 ± 0.0082
   0.2	0.631 ± 0.004	1.4974 ± 0.0025
   0.5	0.642 ± 0.033	1.4992 ± 0.0004

   These final scores overstate the effect, for two reasons:

   • In 2 of the 3 p=0.2 circle_packing seeds, the jump to ~0.62 came from a lucky diff at iteration 1, not from a rewrite. The p=0 and p=0.2 runs got byte-identical prompts at iteration 1 and the same request seed, but Ollama still returned different responses, likely because of nondeterminism with concurrent requests on the GPU. In practice the LLM's sampling isn't reproducible, so 3 seeds are noisy.
   • function_minimization is at its ceiling (~1.5). Differences there are within noise, though AUC rises with p (1.467 / 1.481 / 1.491) and the spread across seeds shrinks.

   Per-attempt comparison (cleaner signal)

   On circle_packing, among the first improvement attempts on the initial program:

   Mode	Median score	Reached ≥ 0.57	Scored 0
   Full rewrite	0.52	11/23	4/23
   Diff	0.34	5/61	21/61

   Within mixed runs (same populations), a full rewrite produced a new best 8% of the time vs 3% for a diff (95% Wilson intervals [4%, 14%] and [1%, 7%]). On function_minimization, the two rates were equal (8% vs 7%).

   Failure modes

   • Rewrites almost never fail to parse (1–4%). Diff parse failures run 3–15%, and most of them are one error: "Unmatched SEARCH/REPLACE delimiter outside a diff block" (48 occurrences). Rewrites avoid that format entirely.
   • Evaluation-failure rates are similar for the two modes (7–17%).
   • Open question: diffs in mixed runs failed evaluation more often than at p=0 (circle_packing: 24/177 vs 4/130). Whether the parent was a rewrite doesn't explain it (diff→diff 13% vs rewrite→diff 11%), and neither do parent score or prompt length.

   Takeaway

   Full rewrites help mostly by escaping a weak starting program. A nonzero p looks worthwhile, and 0.2 is a reasonable value to suggest. This data can't tell 0.2 from 0.5. A follow-up with circle_packing only, 5–6 seeds, and p ∈ {0.1, 0.2, 0.5} would settle that. #580 keeps the default at 0.0 so existing behaviour doesn't change.

   While running this I found two separate bugs, each fixed in its own PR:

   • Identical prompts returned identical responses because every request carried the same seed. This is a follow-up to Ensemble model selection is identical across worker processes when random_seed is set #554/Seed ensemble model sampling per iteration in worker processes #559, which fixed the same problem for model selection, and it keeps the reproducibility from Support seed for reproducibility #97 (fix(llm): vary the request seed per iteration in workers #582).
   • Children were dropped from the trace when their parent had left the archive (fix(trace): trace children whose parent left the archive #581).

   All three PRs are independent and merge cleanly in any order.

   👍React with 👍1Reacted by Brad