Repository navigation
Seed ensemble model sampling per iteration in worker processes - #559
Merged
codelion merged 1 commit intoOct 29, 2026
Conversation
With random_seed set, every worker process built its LLMEnsemble from the same model seed, so each worker replayed the same sequence of model picks whatever iterations it ran, and a recycled worker restarted it. Reseed the worker ensemble at the start of each iteration from the configured seed and the iteration number, so a pick depends on the iteration alone. Unseeded runs are unchanged. Fixes algorithmicsuperintelligence#554
SUPR1M3
added a commit
to SUPR1M3/openevolve
that referenced
this pull request
Nov 1, 2026
With random_seed set (the default is 42), every LLM request carried the same seed. On providers that honour it (Ollama, vLLM, OpenAI best effort) identical prompts, common early in a run when iterations share a parent, returned identical responses, so those iterations produced duplicate children. In a 10-iteration pilot on Ollama, up to four iterations of one run returned byte-identical programs. After reseed(iteration) (algorithmicsuperintelligence#559), the ensemble now sends a seed derived from the sampled model's seed and the iteration. Runs stay reproducible, an explicit seed kwarg still wins, and ensembles that are never reseeded (the evaluator's LLM feedback) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SUPR1M3
added a commit
to SUPR1M3/openevolve
that referenced
this pull request
Nov 1, 2026
…telligence#116) Add full_rewrite_probability: with diff_based_evolution on, each iteration asks for a full rewrite instead of SEARCH/REPLACE diffs with this probability. The default 0.0 keeps today's behaviour. - The worker picks the mode per iteration from a generator seeded with (random_seed, iteration), so runs stay reproducible whichever worker runs an iteration, and the ensemble's model picks (algorithmicsuperintelligence#559) are untouched. - The chosen mode drives both prompt building and response parsing, and is recorded on the result (failures included), in program metadata ("evolution_mode"), in the evolution trace, and as the template key passed to log_prompt. - Config validation (from_dict and the controller, for configs built in code): the probability must be in [0, 1], needs diff_based_evolution, and cannot be combined with prompt.programs_as_changes_description. Also add evolution_trace.include_failures (off by default): iterations that fail before producing a child are written to <trace>_failures.jsonl with their error and mode, plus the prompt and LLM response when include_prompts is on. Worker errors after the LLM call now carry the prompt, response, parent and island, so parse failures can be inspected.
codelion
pushed a commit
that referenced
this pull request
Nov 2, 2026
With random_seed set (the default is 42), every LLM request carried the same seed. On providers that honour it (Ollama, vLLM, OpenAI best effort) identical prompts, common early in a run when iterations share a parent, returned identical responses, so those iterations produced duplicate children. In a 10-iteration pilot on Ollama, up to four iterations of one run returned byte-identical programs. After reseed(iteration) (#559), the ensemble now sends a seed derived from the sampled model's seed and the iteration. Runs stay reproducible, an explicit seed kwarg still wins, and ensembles that are never reseeded (the evaluator's LLM feedback) are unchanged.
codelion
pushed a commit
that referenced
this pull request
Nov 2, 2026
…v0.4.1) (#580) * feat: mix full rewrites into diff-based evolution (#116) Add full_rewrite_probability: with diff_based_evolution on, each iteration asks for a full rewrite instead of SEARCH/REPLACE diffs with this probability. The default 0.0 keeps today's behaviour. - The worker picks the mode per iteration from a generator seeded with (random_seed, iteration), so runs stay reproducible whichever worker runs an iteration, and the ensemble's model picks (#559) are untouched. - The chosen mode drives both prompt building and response parsing, and is recorded on the result (failures included), in program metadata ("evolution_mode"), in the evolution trace, and as the template key passed to log_prompt. - Config validation (from_dict and the controller, for configs built in code): the probability must be in [0, 1], needs diff_based_evolution, and cannot be combined with prompt.programs_as_changes_description. Also add evolution_trace.include_failures (off by default): iterations that fail before producing a child are written to <trace>_failures.jsonl with their error and mode, plus the prompt and LLM response when include_prompts is on. Worker errors after the LLM call now carry the prompt, response, parent and island, so parse failures can be inspected. * Round-trip list metadata through HDF5 trace exports The exporter JSON-encodes non-scalar metadata but the loader only decoded strings starting with "{", so lists came back as strings. Decoding every "["-prefixed string instead would corrupt real strings that look like JSON. The exporter now records which attributes it encoded in __json_encoded_keys__, and the loader decodes exactly those. Files without the list load as before, since their values can't be told apart. Closes #574 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Complete failure tracing and record each program's LLM model Review fixes for evolution_trace.include_failures: - Timed-out and crashed iterations now reach the failures file; they were the failures with the least other evidence and were missing from it. - Failed LLM calls keep the prompt, parent and island like later failures. - failures_path now matches the file written when compress is on (it lacked .gz). Dotted trace names no longer share one failures file: run.v2 and run.v3 both wrote to run_failures.jsonl. Each child now stores the sampled model as metadata["llm_model"], also in the trace, so ensemble models can be compared. token_usage only names the model when the provider returns usage data. Closes #381 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bump version to 0.4.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Streamline the README and lead with research adoption Cut the README from 1,074 to about 200 lines. It now opens with adoption and a list of papers using OpenEvolve, then results, quick start and an overview. Emojis are gone, which also fixes the Quick Start and Installation anchors they were breaking, and the dead example output link is removed. The quick start now clones the repo, since the examples are not part of the pip package. Detailed material moves to docs/: LLM providers, configuration, system messages, showcase, how it works, FAQ and roadmap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
random_seedis set (it defaults to 42), every worker process samples the same sequence of ensemble models. The controller writes one derivedllm_seedinto everyllm.models[*].random_seed, each worker builds its ownLLMEnsemblein_lazy_init_worker_components, andLLMEnsemble.__init__seeds its sampler once frommodels_cfg[0].random_seed. A pick therefore depends on how many picks that worker has already made, not on the iteration: workers running even and odd iterations replay identical picks, and a recycled worker (max_tasks_per_child) restarts the same stream.This adds
LLMEnsemble.reseed(iteration), which reseeds the sampler from the configured seed and the iteration number, and calls it at the start of_run_iteration_worker. A pick now depends on the iteration alone, so it is the same whichever worker runs it, and seeded runs stay reproducible. When no seed is configured nothing changes. Thestrseed goes throughrandom.seed's SHA-512 path, so it does not depend onPYTHONHASHSEED. The evaluator ensemble only callsgenerate_all_with_context, which never samples, so it is left as is.Tests
New
tests/test_ensemble_worker_seed.pyruns the real_serialize_config,_worker_initand_run_iteration_workerwith the realLLMEnsemble, stubbing onlyOpenAILLM.generate_with_contextto record the sampled model:Three of the four fail without the fix.
python -m unittest discover tests: 916 tests OK (2 skipped).black --checkis clean.Fixes #554