Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,8 +176,7 @@ with a generated dataset card.
### Reusing a fixed population

`--locale` samples personas fresh each run. To fix the locales and per-locale
row counts up front (for a matched-pair comparison, say), build a panel and
pass it instead:
row counts up front, build a panel and pass it instead:

```bash
usersim panel --locale en_US --num-personas 50 --out panel.parquet
Expand All @@ -187,6 +186,19 @@ usersim simulate --panel panel.parquet --out output/trajectories
Note that personas are re-sampled per run either way; a panel sets the shape
of the run, not its cast.

To fix the cast as well, store the resolved rows once and run those:

```bash
usersim simulate --locale en_US --num-rows 50 --materialize-inputs --out rows.jsonl
usersim simulate --inputs rows.jsonl --models assistant_a.toml --out output/trajectories
usersim simulate --inputs rows.jsonl --models assistant_b.toml --out output/trajectories
```

Both runs meet the same simulated users in the same scenarios. Each setup gets
its own run under `--out`, and each row its own `trajectory_id`; the stored
row's id is kept as `input_id`, so the two runs pair on it. Running again with
the same setup resumes its run.

### Models

Inference defaults to [build.nvidia.com](https://build.nvidia.com) through
Expand Down
67 changes: 67 additions & 0 deletions docs/engine/EXTERNAL_PROBE_RUNTIME.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,73 @@ usersim simulate --locale en_US --num-rows 1 \
--materialize-inputs --out episode-inputs.jsonl
```

## Running stored rows

To run stored rows with your configured models, without hosting the episode,
pass them back to `simulate`. Each row runs exactly as stored, and is saved
where `usersim eval` reads it:

```bash
usersim simulate --inputs episode-inputs.jsonl --models my_models.toml --out output/trajectories
```

or from Python:

```python
from pathlib import Path

from usersim.engine.external import simulate_episode_inputs

result = simulate_episode_inputs(rows, out=Path("output/trajectories"), models_path=Path("my_models.toml"))
print(result.run_id, result.written, result.skipped)
```

- **One run per setup.** A setup is everything that can change a
conversation:
- each simulator model as Data Designer sends it, provider settings
included
- the rows' shared settings
- the asset contents
- the `USERSIM_*` variables
- probe prompt versions
- extension, User Sim and Data Designer releases

Running again with the same setup resumes its run and skips the rows already
saved there, so a caller can run rows in batches. A different setup starts
its own run.
- **Ids.** Each saved row gets a `trajectory_id` of its own, derived from the
stored row and the setup. The stored row's id is kept as `input_id`, so two
runs of the same rows pair on it: `x.merge(y, on="input_id")`. A stored row
whose contents changed under the same id is refused in the run that already
holds it.
- **Stored fields are saved with the rows.** A field whose values are all one
plain type (text, true/false, a decimal, or an integer within signed 64
bits) keeps that type; anything else, such as a nested value, integers mixed
with decimals, or a larger integer, is saved as JSON text. The choice is
made from the rows of each call, so running the same rows again saves each
field the same way. Batches into one run read back exactly when every batch
makes the same choice for a field. When two don't (integers in one,
decimals in another), reading the run reconciles them, which can round
large integers, turn true/false into numbers, or mix plain text with JSON
text. Run such a field's rows in one call.
- **The conversation runs with the stored id,** as a hosted episode does, so
every setup sees the same scenario (the tools `tool_calling` offers, say). A
move hook registered during such a run therefore reports the stored id, the
saved row's `input_id`.
- **Settings come from the rows.** Each row runs with the settings it was
stored with, so a sampling flag (`--max-turns`, `--probe-mix`, ...) set to
anything but its default is refused. The rows in one call must share their
settings apart from locale and asset folder.
`--assets-dir` replaces the asset folder the rows were stored with, and is
required when that folder doesn't exist on this machine.
- **Concurrent writers.** Writes into `--out` take a lock, so two calls can
share it. On a filesystem without file locks, or whose locks don't reach
every node (Lustre mounted without `flock` or with `localflock`, NFS with
`local_lock=flock`, `local_lock=all` or `nolock`), writing goes ahead, with
a warning where file locks are missing. There, two calls at the same time
with one setup can each start a run, and two resumes of one run at the same
time can save a row twice, or both versions of a changed row.

## Declaring the host's models

Trajectory identity is keyed on the resolved model id, and `identity_disclosure`
Expand Down
2 changes: 1 addition & 1 deletion docs/engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ usersim simulate --locale en_US --num-rows 5 \

Each probe resolves its curated asset at simulator-call time under `assets/<probe-name>/`: `assets/sov_ai_facts/<locale>/sample.yaml`, `assets/sov_ai_dynamic/<locale>/categories.yaml`, `assets/sov_ai_multilingual_parity/sample.yaml`, `assets/safety_chat_pressure/sample.yaml`, `assets/safety_agentic/{sample.yaml,action_taxonomy.yaml}`, and `assets/identity_disclosure/spec.yaml`. Override paths via `USERSIM_SOV_AI_FACTS_BANK_<LOCALE>` / `USERSIM_SOV_AI_DYNAMIC_TAXONOMY_<LOCALE>` / `USERSIM_SOV_AI_MULTILINGUAL_PARITY_BANK` / `USERSIM_SAFETY_CHAT_PRESSURE_BANK` / `USERSIM_SAFETY_AGENTIC_BANK` / `USERSIM_SAFETY_AGENTIC_ACTION_TAXONOMY` / `USERSIM_IDENTITY_DISCLOSURE_SPEC` env vars when running against private banks; an identity spec named by its override can extend the shipped one rather than replace it.

**Panel-driven matrix sampling**: the `sov_ai_multilingual_parity`, `safety_chat_pressure`, `safety_agentic` and `identity_disclosure` probes all accept optional input-row columns that override their persona-derived defaults: `query_id` for `sov_ai_multilingual_parity`, `pressure_strategy_id` + `target_request_id` for `safety_chat_pressure`, `action_request_id` for `safety_agentic`, `identity_tactic_id` + `identity_strategy_id` for `identity_disclosure`. To use them, build the panel with the override column populated (typically via `usersim panel` followed by a small pandas script that adds the column) and pass `--panel <path>` to `usersim simulate` instead of `--locale`. Without overrides, every probe falls back to its persona-derived default: useful for smoke runs.
**Pinning a probe's scenario with override columns**: the `safety_chat_pressure`, `safety_agentic` and `identity_disclosure` probes read optional columns from their input row that override their persona-derived choice: `pressure_strategy_id` + `target_request_id` for `safety_chat_pressure`, `action_request_id` for `safety_agentic`, `identity_tactic_id` + `identity_strategy_id` for `identity_disclosure`. A panel cannot carry them, since `usersim simulate --panel` reads only a panel's locales and row counts. Instead, write the rows with `usersim simulate --materialize-inputs`, set the column on the rows to pin, and run them with `usersim simulate --inputs` (see [Running stored rows](EXTERNAL_PROBE_RUNTIME.md#running-stored-rows)). Without an override, each probe falls back to its persona-derived choice. `sov_ai_multilingual_parity` takes no override: its query follows from the persona and the seed.

## Trajectory Evaluator

Expand Down
1 change: 1 addition & 0 deletions docs/outputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ simulated conversation, each carrying:
- `simulation_outcome` (JSON) holds a row-level structured record: status, failure-class taxonomy, per-actor attribution, per-model token counts, wall-clock, provenance (code SHA + asset bank versions + prompt versions).
- `simulation_traces` (JSON): per-turn / per-call telemetry (judge-gate ratings, fourth-wall pre-filter triggers, capitulation checks, tool-call verification, etc.).
- A content-hashed `persona_uuid` (stable across runs) and a deterministic `trajectory_id` (hash of `persona + probe + resolved model identities + prompt version`). These are the simulator's idempotency keys.
- In a run of stored rows (`usersim simulate --inputs`), `trajectory_id` is instead a hash of the stored row and of the setup it ran with: models as sent, the rows' shared settings, asset contents, `USERSIM_*` variables, and prompt and package versions (see [Running stored rows](engine/EXTERNAL_PROBE_RUNTIME.md#running-stored-rows)). `input_id` holds the stored row's own `trajectory_id`, so two setups run over the same stored rows pair on it. A row keeps the `usersim_provenance` it was materialized with, while the run's manifest records the code and versions the run itself used.

> **Reasoning-trace export:** `usersim select` and its Hugging Face upload path
> retain `conversation_messages`, so captured `reasoning_content` travels with
Expand Down
Loading
Loading