Repository navigation
Plain simulate trajectory ids leave out model and run settings #35
Description
What
A plain usersim simulate row gets
trajectory_id = sha256(persona_uuid | probe_family | probe_variant | random_seed
| user model name | assistant model name | PROMPT_VERSION | locale)[:16]
(core/identity.py::trajectory_id, called from core/episode_input.py).
Nothing else about the setup goes into it:
- the models' request settings (temperature,
max_tokens, reasoning settings, timeout) and their
provider (endpoint,extra_body) - the other model roles (judge, summary, API response)
- run settings such as
max_turns - asset contents and
USERSIM_*variables
So two runs that differ only in those, and draw the same persona for the same probe with the same
seed, give that row the same trajectory_id.
Why it matters
- Resume. With the default
--skip-existing, a second run into the same--outresumes the
latest run and skips ids already saved there. A rerun at a different temperature that draws the
same people simulates them, then drops the rows as already saved. usersim selectover several runs keeps the first row pertrajectory_id
(selection/io.py), so selecting over both runs silently keeps one setup's rows.- It's inconsistent with
simulate --inputs(feat(cli): run stored episode rows with simulate --inputs #30), whose ids cover everything that can change
a conversation.
Possible directions
- Put a setup fingerprint into plain ids, as
--inputsdoes. Every plain id would change, so a
run made before the change would not be resumed by a later version. - Keep the ids, but record each run's setup and refuse to resume into a run with a different one.
Raised in the review of #30.
Activity
ting-hong-shieh commentedon Oct 12, 2026
I'd like to contribute a fix, once the compatibility direction is agreed.
Reading
identity.py,episode_input.py, and #30, I see two options:selectdeduplicating rows from different setups.I prefer the second option, reusing the resolved-setup fingerprint contract from #30 rather than maintaining a separate model/provider/assets hashing implementation. I would keep the identity used inside a probe separate from the saved result identity where necessary: #30 documents that changing the execution ID can change the tools a probe draws. A stored input ID would remain available for pairing.
Proposed first scope:
Planned tests cover same-setup resume, changed model/settings/assets, cross-run selection, execution-identity stability, and legacy metadata. These are planned checks; I have not run a reproduction locally.
Would this direction be acceptable, and should the contribution follow #30/#34 after they land or target a separate agreed interface? Is anyone already implementing #35?
AI assistance: Codex helped inspect the code and prepare this proposal.
isarasua commentedon Oct 12, 2026
Thanks for picking this up. Nobody is implementing it, and I'd welcome the fix. Which direction to
take is @3mei's call.
On timing: I'd build on #30 and #34 once they land rather than ahead of them.
setup_fingerprint(incli/_inputs.py), the contract you'd reuse._run_settings.json), which is where a plain run's setup couldlive.
Both are still in review, so their details may change. The fingerprint will also grow: whichever of
#30 and #31 lands second adds
replay_reasoningto it.