Skip to content

Seed ensemble model sampling per iteration in worker processes - #559

Merged
codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
fixfon:fix/554-ensemble-seed-per-iteration
Oct 29, 2026
Merged

codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
fixfon:fix/554-ensemble-seed-per-iteration

Conversation

@fixfon

@fixfon fixfon commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Summary

When random_seed is set (it defaults to 42), every worker process samples the same sequence of ensemble models. The controller writes one derived llm_seed into every llm.models[*].random_seed, each worker builds its own LLMEnsemble in _lazy_init_worker_components, and LLMEnsemble.__init__ seeds its sampler once from models_cfg[0].random_seed. A pick therefore depends on how many picks that worker has already made, not on the iteration: workers running even and odd iterations replay identical picks, and a recycled worker (max_tasks_per_child) restarts the same stream.

This adds LLMEnsemble.reseed(iteration), which reseeds the sampler from the configured seed and the iteration number, and calls it at the start of _run_iteration_worker. A pick now depends on the iteration alone, so it is the same whichever worker runs it, and seeded runs stay reproducible. When no seed is configured nothing changes. The str seed goes through random.seed's SHA-512 path, so it does not depend on PYTHONHASHSEED. The evaluator ensemble only calls generate_all_with_context, which never samples, so it is left as is.

Tests

New tests/test_ensemble_worker_seed.py runs the real _serialize_config, _worker_init and _run_iteration_worker with the real LLMEnsemble, stubbing only OpenAILLM.generate_with_context to record the sampled model:

  • workers running different iterations do not replay the same picks
  • recycled workers (one iteration each) do not all pick the same model
  • the same iteration picks the same model on any worker, in any order
  • unseeded runs stay random

Three of the four fail without the fix. python -m unittest discover tests: 916 tests OK (2 skipped). black --check is clean.

Fixes #554

With random_seed set, every worker process built its LLMEnsemble from the same model seed, so each worker replayed the same sequence of model picks whatever iterations it ran, and a recycled worker restarted it. Reseed the worker ensemble at the start of each iteration from the configured seed and the iteration number, so a pick depends on the iteration alone. Unseeded runs are unchanged.

Fixes algorithmicsuperintelligence#554
@codelion
codelion merged commit e2bdddd into algorithmicsuperintelligence:main Oct 29, 2026
5 checks passed
SUPR1M3 added a commit to SUPR1M3/openevolve that referenced this pull request Nov 1, 2026
With random_seed set (the default is 42), every LLM request carried the
same seed. On providers that honour it (Ollama, vLLM, OpenAI best effort)
identical prompts, common early in a run when iterations share a parent,
returned identical responses, so those iterations produced duplicate
children. In a 10-iteration pilot on Ollama, up to four iterations of one
run returned byte-identical programs.

After reseed(iteration) (algorithmicsuperintelligence#559), the ensemble now sends a seed derived from
the sampled model's seed and the iteration. Runs stay reproducible, an
explicit seed kwarg still wins, and ensembles that are never reseeded
(the evaluator's LLM feedback) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SUPR1M3 added a commit to SUPR1M3/openevolve that referenced this pull request Nov 1, 2026
…telligence#116)

Add full_rewrite_probability: with diff_based_evolution on, each iteration
asks for a full rewrite instead of SEARCH/REPLACE diffs with this
probability. The default 0.0 keeps today's behaviour.

- The worker picks the mode per iteration from a generator seeded with
  (random_seed, iteration), so runs stay reproducible whichever worker runs
  an iteration, and the ensemble's model picks (algorithmicsuperintelligence#559) are untouched.
- The chosen mode drives both prompt building and response parsing, and is
  recorded on the result (failures included), in program metadata
  ("evolution_mode"), in the evolution trace, and as the template key
  passed to log_prompt.
- Config validation (from_dict and the controller, for configs built in
  code): the probability must be in [0, 1], needs diff_based_evolution, and
  cannot be combined with prompt.programs_as_changes_description.

Also add evolution_trace.include_failures (off by default): iterations that
fail before producing a child are written to <trace>_failures.jsonl with
their error and mode, plus the prompt and LLM response when include_prompts
is on. Worker errors after the LLM call now carry the prompt, response,
parent and island, so parse failures can be inspected.
codelion pushed a commit that referenced this pull request Nov 2, 2026
With random_seed set (the default is 42), every LLM request carried the
same seed. On providers that honour it (Ollama, vLLM, OpenAI best effort)
identical prompts, common early in a run when iterations share a parent,
returned identical responses, so those iterations produced duplicate
children. In a 10-iteration pilot on Ollama, up to four iterations of one
run returned byte-identical programs.

After reseed(iteration) (#559), the ensemble now sends a seed derived from
the sampled model's seed and the iteration. Runs stay reproducible, an
explicit seed kwarg still wins, and ensembles that are never reseeded
(the evaluator's LLM feedback) are unchanged.
codelion pushed a commit that referenced this pull request Nov 2, 2026
…v0.4.1) (#580)

* feat: mix full rewrites into diff-based evolution (#116)

Add full_rewrite_probability: with diff_based_evolution on, each iteration
asks for a full rewrite instead of SEARCH/REPLACE diffs with this
probability. The default 0.0 keeps today's behaviour.

- The worker picks the mode per iteration from a generator seeded with
  (random_seed, iteration), so runs stay reproducible whichever worker runs
  an iteration, and the ensemble's model picks (#559) are untouched.
- The chosen mode drives both prompt building and response parsing, and is
  recorded on the result (failures included), in program metadata
  ("evolution_mode"), in the evolution trace, and as the template key
  passed to log_prompt.
- Config validation (from_dict and the controller, for configs built in
  code): the probability must be in [0, 1], needs diff_based_evolution, and
  cannot be combined with prompt.programs_as_changes_description.

Also add evolution_trace.include_failures (off by default): iterations that
fail before producing a child are written to <trace>_failures.jsonl with
their error and mode, plus the prompt and LLM response when include_prompts
is on. Worker errors after the LLM call now carry the prompt, response,
parent and island, so parse failures can be inspected.

* Round-trip list metadata through HDF5 trace exports

The exporter JSON-encodes non-scalar metadata but the loader only decoded
strings starting with "{", so lists came back as strings. Decoding every
"["-prefixed string instead would corrupt real strings that look like JSON.

The exporter now records which attributes it encoded in
__json_encoded_keys__, and the loader decodes exactly those. Files without
the list load as before, since their values can't be told apart.

Closes #574

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Complete failure tracing and record each program's LLM model

Review fixes for evolution_trace.include_failures:
- Timed-out and crashed iterations now reach the failures file; they were
  the failures with the least other evidence and were missing from it.
- Failed LLM calls keep the prompt, parent and island like later failures.
- failures_path now matches the file written when compress is on (it
  lacked .gz). Dotted trace names no longer share one failures file:
  run.v2 and run.v3 both wrote to run_failures.jsonl.

Each child now stores the sampled model as metadata["llm_model"], also in
the trace, so ensemble models can be compared. token_usage only names the
model when the provider returns usage data.

Closes #381

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bump version to 0.4.1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Streamline the README and lead with research adoption

Cut the README from 1,074 to about 200 lines. It now opens with adoption
and a list of papers using OpenEvolve, then results, quick start and an
overview. Emojis are gone, which also fixes the Quick Start and
Installation anchors they were breaking, and the dead example output link
is removed. The quick start now clones the repo, since the examples are
not part of the pip package.

Detailed material moves to docs/: LLM providers, configuration, system
messages, showcase, how it works, FAQ and roadmap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensemble model selection is identical across worker processes when random_seed is set

2 participants