feat: opt-in replay of the assistant's own earlier reasoning

architecture/cli.md

@@ -49,6 +49,22 @@ top_p, max_tokens, and any model-family-specific `extra_body`. A TOML row
 overrides the catalogue per field, and the row is self-describing for
 anything it sets.
 
+`replay_reasoning = true` on the `assistant_model` row sends the assistant
+the reasoning of its own earlier messages, for a model trained to receive it.
+It becomes `ConversationSimulatorConfig.replay_assistant_reasoning`, because
+Data Designer's model configs take no extra fields. A setup that cannot
+replay fails at config construction:
+- the switch on another alias
+- `--no-store-reasoning`
+- a provider that is not OpenAI-compatible
+
+A model override that changes the assistant's model turns it off unless the
+override sets it.
+
+It only helps where the endpoint passes `reasoning_content` on earlier
+assistant messages through to the model; a server or chat template that
+drops earlier thinking makes it a no-op.
+
 ## Failure handling
 
 `usersim.cli._errors.ConfigError` is a `ValueError` subclass, never a

docs/engine/AUTHORING_A_PROBE.md

@@ -579,7 +579,7 @@ from usersim.engine.core.cooking_bank import (
 )
 from usersim.engine.core.outcomes import WarningKind
 from usersim.engine.core.probes import (
-    BankBackedProbe, BankVerbatimMixin, register_probe,
+    BankBackedProbe, BankVerbatimMixin, dispatch_episode, register_probe,
 )
 from usersim.engine.probes.cooking_advisor.prompts import (
     COOKING_GATE_PROMPT, COOKING_USER_PROMPTS,
@@ -671,7 +671,9 @@ async def simulate_cooking_advisor(
         cfg=cfg, provenance=provenance, profile=profile, data=data,
         outcome_builder=outcome_builder,
     )
-    return await probe.run_dispatch(models=models, data=data, cfg=cfg)
+    # dispatch_episode installs the run's per-episode settings (reasoning
+    # replay) around run_dispatch, as the generator does.
+    return await dispatch_episode(probe, models=models, data=data, cfg=cfg)
 ```
 
 ### File 6: built-in probe list

docs/engine/EXTERNAL_PROBE_RUNTIME.md

@@ -72,6 +72,7 @@ print(result.run_id, result.written, result.skipped)
   conversation:
   - each simulator model as Data Designer sends it, provider settings
     included
+  - whether the assistant's earlier reasoning is replayed
   - the rows' shared settings
   - the asset contents
   - the `USERSIM_*` variables
@@ -102,8 +103,11 @@ print(result.run_id, result.written, result.skipped)
   saved row's `input_id`.
 - **Settings come from the rows.** Each row runs with the settings it was
   stored with, so a sampling flag (`--max-turns`, `--probe-mix`, ...) set to
-  anything but its default is refused. The rows in one call must share their
-  settings apart from locale and asset folder.
+  anything but its default is refused. The one exception is replaying the
+  assistant's reasoning: like the assistant itself, it comes from this run's
+  models file (`replay_reasoning` on `assistant_model`), whatever the rows were
+  stored with. The rows in one call must share their settings apart from
+  locale, asset folder and that replay setting.
   `--assets-dir` replaces the asset folder the rows were stored with, and is
   required when that folder doesn't exist on this machine.
 - **Concurrent writers.** Writes into `--out` take a lock, so two calls can
@@ -150,6 +154,14 @@ exposing `agenerate_text_embeddings()`); without one, dense retrieval falls back
 to lexical search with a warning, so results can differ from a standalone run
 that has one configured.
 
+### Replaying the assistant's reasoning
+
+When the row's `usersim_config` sets `replay_assistant_reasoning` (from
+`replay_reasoning = true` on `assistant_model` in the models file), each
+assistant request's earlier assistant messages carry their
+`reasoning_content`. Forward them to the model as they are. No other role's
+messages ever carry the assistant's reasoning. Replay only helps where the endpoint passes `reasoning_content` on earlier assistant messages through to the model; a server or chat template that drops earlier thinking makes it a no-op.
+
 ## Driving the episode
 
 `advance()` is the whole loop. The first call takes no result; every later call

docs/engine/README.md

@@ -452,7 +452,8 @@ diagnostics in the simulator-health report quantify that for a given run, and
 | `persona_grounding_ratio` | float | `1.0` | Fraction of trajectories whose first user turn incorporates persona-specific details. Lower it for a generic-query control arm |
 | `context_compression` | bool | `True` | Summarize older assistant responses to bound context growth |
 | `compression_window` | int | `1` | Keep last N assistant responses verbatim before compressing |
-| `store_reasoning` | bool | `True` | Keep the assistant's thinking trace on the assistant message when the model emits one (`--no-store-reasoning` to turn off). Gated at capture, so a suppressed trace is never attached. Independent of whether the model *produces* a trace: that is a model-side setting (`chat_template_kwargs` in the models TOML, plus a server-side reasoning parser). Never affects what a model receives: traces are stored, never replayed |
+| `store_reasoning` | bool | `True` | Keep the assistant's thinking trace on the assistant message when the model emits one (`--no-store-reasoning` to turn off). Gated at capture, so a suppressed trace is never attached. Independent of whether the model *produces* a trace: that is a model-side setting (`chat_template_kwargs` in the models TOML, plus a server-side reasoning parser). A trace reaches a model again only through `replay_assistant_reasoning` |
+| `replay_assistant_reasoning` | bool | `False` | Send the assistant model the reasoning of its own earlier messages (`replay_reasoning = true` on `assistant_model` in the models TOML). Needs `store_reasoning`; the assistant's history is then not compressed. No other model receives the assistant's reasoning during a simulation |
 | `verbosity` | int | `1` | Logging detail: 0=quiet, 1=normal, 2=detailed |
 | `random_seed` | int? | `None` | Seed for reproducible tool sampling |
 
@@ -471,7 +472,7 @@ what each one carries:
 | `simulation_traces` | str? (JSON) | Per-turn / per-call telemetry (keyed `turn_idx` x `call_idx`) with `kind` discriminator: `user_query_gate`, `user_followup_gate`, `fourth_wall_prefilter`, `assistant_quality`, `early_stop`, `api_response_reroll`, `tool_call_verifier`, etc. The debugging surface: **not** the assistant-evaluation surface. |
 | `probe_family` | str | Probe module's `PROBE_FAMILY` constant (e.g., `"general_open_ended"`, `"sov_ai_facts"`). |
 | `probe_variant` | str | Probe-specific discriminator (first entry of `PROBE_VARIANTS`, or an explicit row value when provided). Part of `trajectory_id` and the evaluator's stratification key. |
-| `conversation_messages` | str (JSON) | Full message list. Each entry is `{role, content}`; assistant entries add `tool_calls` when the turn called a tool, `tool` entries add `tool_call_id`, and an assistant entry adds **`reasoning_content`** when that model call emitted a thinking trace **and `store_reasoning` is on** (the default; `--no-store-reasoning` suppresses it). The key is absent (not empty) whenever either condition fails; the run manifest's `simulation_config.store_reasoning` records which was requested. Traces are attributed per call, so a tool-calling turn that appends several assistant messages carries each trace on the message that produced it. Stored for analysis only: `core/llm.py::_dicts_to_chat_messages` hard-codes the assistant's `reasoning_content` to `None` on the way back into a model, so a later turn never sees an earlier turn's thinking. |
+| `conversation_messages` | str (JSON) | Full message list. Each entry is `{role, content}`; assistant entries add `tool_calls` when the turn called a tool, `tool` entries add `tool_call_id`, and an assistant entry adds **`reasoning_content`** when that model call emitted a thinking trace **and `store_reasoning` is on** (the default; `--no-store-reasoning` suppresses it). The key is absent (not empty) whenever either condition fails; the run manifest's `simulation_config.store_reasoning` records which was requested. Traces are attributed per call, so a tool-calling turn that appends several assistant messages carries each trace on the message that produced it. Stored for analysis: `core/llm.py::_dicts_to_chat_messages` drops it on the way back into a model, so a later turn never sees an earlier turn's thinking, unless the run sets `replay_assistant_reasoning`, which sends the assistant its own earlier traces. |
 | `conversation_metadata` | str (JSON) | Loose per-turn judge / event log surfaced to interactive previews and the deep-dive analysis script. |
 | `conversation_status` | bool | True if the trajectory ran to completion (no gate exhaustion / no infrastructure failure) AND any optional `should_succeed(state)` hook returned True. |
 | `behavioral_profile` | str (JSON) | Computed behavioral parameters |

docs/outputs.md

@@ -9,7 +9,7 @@ report files.
 `output/trajectories/run=<id>/locale=<X>/probe_family=<Y>/`. One row per
 simulated conversation, each carrying:
 
-- `conversation_messages` (JSON): the full multi-turn message list. Each entry is `{role, content}`, plus `tool_calls` / `tool_call_id` on tool-using turns and `reasoning_content` on an assistant turn whose model emitted a thinking trace, unless `--no-store-reasoning` was passed. The trace is stored for analysis only; it is never replayed into a later turn (see [`docs/engine/README.md`](engine/README.md)).
+- `conversation_messages` (JSON): the full multi-turn message list. Each entry is `{role, content}`, plus `tool_calls` / `tool_call_id` on tool-using turns and `reasoning_content` on an assistant turn whose model emitted a thinking trace, unless `--no-store-reasoning` was passed. The trace is stored for analysis. It reaches a model again only when the models file sets `replay_reasoning = true` on `assistant_model`, which sends the assistant the reasoning of its own earlier messages (see [`docs/engine/README.md`](engine/README.md)).
 - `simulation_outcome` (JSON) holds a row-level structured record: status, failure-class taxonomy, per-actor attribution, per-model token counts, wall-clock, provenance (code SHA + asset bank versions + prompt versions).
 - `simulation_traces` (JSON): per-turn / per-call telemetry (judge-gate ratings, fourth-wall pre-filter triggers, capitulation checks, tool-call verification, etc.).
 - A content-hashed `persona_uuid` (stable across runs) and a deterministic `trajectory_id` (hash of `persona + probe + resolved model identities + prompt version`). These are the simulator's idempotency keys.

notebooks/01_simulate.ipynb

@@ -82,6 +82,7 @@
     "from usersim.cli._models import (\n",
     "    REQUIRED_SIMULATOR_ALIASES,\n",
     "    apply_model_overrides,\n",
+    "    assistant_replays_reasoning,\n",
     "    bundled_models_path,\n",
     "    default_models_path,\n",
     "    describe_models_source,\n",
@@ -727,6 +728,8 @@
     "        context_compression=True,\n",
     "        compression_window=1,\n",
     "        random_seed=RANDOM_SEED,\n",
+    "        # As build_simulator_config_builder sets it for the run itself.\n",
+    "        replay_assistant_reasoning=assistant_replays_reasoning(MODELS, store_reasoning=True),\n",
     "    )\n",
     "    # Describe the RUN, not just this pass. On a resumed run LOCALES holds only the\n",
     "    # locales this pass filled in, so recording it verbatim would understate a run that\n",

src/usersim/cli/_inputs.py

@@ -60,14 +60,18 @@
 
 #: Settings that may differ between the rows of one invocation.
 _PER_ROW_SETTINGS = ("locale", "assets_dir")
+#: Settings the run's models file decides, whatever the rows were stored with:
+#: replaying the assistant's reasoning belongs to the assistant under test.
+_RUN_SETTINGS = ("replay_assistant_reasoning",)
 
 
 @dataclass(frozen=True)
 class EpisodeInputs:
     """Stored rows checked and ready to run."""
 
     rows: list[dict[str, Any]]
-    #: The ``usersim_config`` every row shares, apart from ``_PER_ROW_SETTINGS``.
+    #: The ``usersim_config`` every row shares, apart from ``_PER_ROW_SETTINGS``
+    #: and ``_RUN_SETTINGS``, which the run settles (see :func:`_run_config`).
     config: dict[str, Any]
     #: The asset root the rows run against.
     assets_dir: Path
@@ -127,8 +131,9 @@ def prepare_episode_inputs(
     """Check stored rows before any model call, and settle what they run against.
 
     Raises ``ValueError`` naming the row and the problem. The rows must share
-    one setup: their ``usersim_config`` may differ only in locale and asset
-    root, so that one run holds one setup.
+    one setup: their ``usersim_config`` may differ only in locale, asset
+    root and the settings the run's models file decides, so that one run
+    holds one setup.
     """
     from pydantic import ValidationError
 
@@ -185,7 +190,11 @@ def _shared_config(rows: list[dict[str, Any]]) -> dict[str, Any]:
     """The setup every row shares; ``ValueError`` naming the settings that differ."""
 
     def setup(row: dict[str, Any]) -> dict[str, Any]:
-        return {key: value for key, value in row["usersim_config"].items() if key not in _PER_ROW_SETTINGS}
+        return {
+            key: value
+            for key, value in row["usersim_config"].items()
+            if key not in _PER_ROW_SETTINGS and key not in _RUN_SETTINGS
+        }
 
     first = setup(rows[0])
     for row in rows[1:]:
@@ -363,7 +372,8 @@ def setup_fingerprint(models: Any, inputs: EpisodeInputs) -> tuple[str, dict[str
     - every simulator model alias as Data Designer sends it, provider settings
       included (``api_key``, a provider's name and ``max_parallel_requests``
       are left out: they don't change what a model says);
-    - the shared ``usersim_config`` as loaded, defaults applied;
+    - the shared ``usersim_config`` as loaded, defaults applied, with whether
+      the assistant's reasoning is replayed as the run's models file sets it;
     - the contents of every asset root, and every non-operational
       ``USERSIM_*`` variable;
     - every probe's ``PROMPT_VERSION``, every extension's package release, and
@@ -375,7 +385,7 @@ def setup_fingerprint(models: Any, inputs: EpisodeInputs) -> tuple[str, dict[str
     from usersim.engine.core.manifest import _own_version
     from usersim.engine.core.storage import RUN_KIND_STORED
 
-    simulation = ConversationSimulatorConfig.model_validate(dict(inputs.config)).model_dump(mode="json")
+    simulation = ConversationSimulatorConfig.model_validate(_run_config(inputs, models)).model_dump(mode="json")
     for key in _PER_ROW_SETTINGS:
         simulation.pop(key, None)
     aliases = [*MODEL_ALIASES, simulation.get("finance_embedding_model_alias") or ""]
@@ -487,12 +497,29 @@ def plan_inputs_run(inputs: EpisodeInputs, models: Any, out: str | Path, *, skip
     return InputsPlan(fingerprint, record, matching_run=matching, pending=pending, saved=saved)
 
 
-def _column_config(inputs: EpisodeInputs, locale: str):
+def _run_config(inputs: EpisodeInputs, models: Any) -> dict[str, Any]:
+    """The rows' shared ``usersim_config``, with ``_RUN_SETTINGS`` as this run sets them.
+
+    The assistant under test comes from the run's models file, so whether its
+    earlier reasoning is replayed does too, as in a plain run. Raises
+    ``ConfigError`` for a models file that cannot replay.
+    """
+    from usersim.cli._models import assistant_replays_reasoning
+    from usersim.engine.config import ConversationSimulatorConfig
+
+    store_reasoning = ConversationSimulatorConfig.model_validate(dict(inputs.config)).store_reasoning
+    return {
+        **inputs.config,
+        "replay_assistant_reasoning": assistant_replays_reasoning(models, store_reasoning=store_reasoning),
+    }
+
+
+def _column_config(inputs: EpisodeInputs, locale: str, models: Any):
     """The simulator column for ``locale``'s rows, on the run's asset root."""
     from usersim.engine.config import ConversationSimulatorConfig
 
     return ConversationSimulatorConfig.model_validate(
-        {**inputs.config, "locale": locale, "assets_dir": str(inputs.assets_dir)}
+        {**_run_config(inputs, models), "locale": locale, "assets_dir": str(inputs.assets_dir)}
     )
 
 
@@ -615,7 +642,7 @@ def simulate_inputs_to_dataset(
     for locale, group in by_locale.items():
         requested[locale] += len(group)
         data_designer, builder = build_inputs_config_builder(
-            models=models, config=_column_config(inputs, locale), rows=[row for row, _ in group.values()]
+            models=models, config=_column_config(inputs, locale, models), rows=[row for row, _ in group.values()]
         )
         # Data Designer names a batch's artifacts by the second it started, so
         # two invocations in one directory could share them; each batch gets
@@ -652,7 +679,7 @@ def simulate_inputs_to_dataset(
                 trajectory_root=out,
                 models_config=models,
                 models_config_path=Path(models_path) if models_path else None,
-                sim_config=_column_config(inputs, next(iter(requested))),
+                sim_config=_column_config(inputs, next(iter(requested)), models),
                 locales_requested=sorted(requested),
                 probe_mix={probe: count / len(stored) for probe, count in sorted(probes.items())},
                 panel_path=inputs.source,