Simulate a statistically representative population of users, sampled from persona datasets (e.g., census-grounded), having multi-turn conversations with your model or agentic system. Every run produces auditable trajectories: evaluation signal, failure analysis, and curated training data, before you have production traffic to learn from.
14 probes 路 9 shipped locales across 7 countries (+ 22 preview India-language variants) 路 7M+ census-grounded personas 路 3,407 tools across 143 toolsets for agentic runs 路 12 evaluator scorers 路 3,200+ tests 路 offline usersim smoke in seconds
Most teams either hand-test their models and agents or wait for production traffic to surface real-user feedback. Both are biased toward the populations you already have, and neither catches the long tail of locales, demographics, or safety scenarios that matter most. For an agent the gap is wider still, because a failure often needs several turns and a tool call to appear at all.
Autonomous driving reached this conclusion years ago. Simulated miles, grounded in real driving distributions, reach the rare and dangerous scenarios far faster than waiting for a fleet to encounter them. Simulation and real data are not a binary choice, but grounded simulation lets you move at the speed of light (SOL) rather than the speed of traffic.
Prompting a model to "act as a user" does not get you there. It produces unrealistically articulate, demographically homogeneous users. NeMo User Sim instead samples from Nemotron-Personas: millions of synthetic individuals grounded in real census data, generated through probabilistic graphical models that enforce realistic correlations between attributes, and written natively in each region's language. Specifically, it samples an extended version of these datasets distributed through NGC, which adds the OCEAN personality traits, names and further fields that the behavioural features depend on.
The datasets are downloaded on first use rather than bundled, and carry their
own licence separate from this project's, including an AI ethics clause. See
docs/personas.md for how to get them and what the terms
allow. The reasoning behind the design, and how far the results carry, is in
docs/methodology.md.
What a single simulated trajectory looks like in practice:
A census-grounded persona, an OCEAN-derived interaction style, a probe-driven multi-turn conversation, and the trajectory metadata that downstream evaluation reads off, all from one row of a parquet dataset.
- Model and agent teams who need to know how a system performs across demographics, languages, and interaction styles rather than on a static benchmark. Runs produce population-stratified evaluations with claims grounded in census distributions.
- Sovereign-AI builders with no user base yet. You get an in-language evaluation surface from day one, drawn from your country's actual census distribution: USA, Japan, India, Singapore, Brazil, France and Korea ship today.
- Agent builders putting a tool-using system under multi-turn load. Probes execute tools against a mock environment that holds state across turns, so a run surfaces scope creep, unauthorised calls, prompt-injection susceptibility and missing consequence disclosure, not just the final answer. Every tool call and its result is recorded on the trajectory.
- Safety teams testing multi-turn capitulation under pressure, gradual disclosure of sensitive information, and reasoning faithfulness. The evaluator runs out of sim, so iterating on a judge or rubric never re-simulates.
Two outputs from one run. The same trajectories answer two questions. How is the system doing: population-stratified scores across demographics, languages and interaction styles. And what should it learn next: the trajectories that passed your quality gates, curated into a dataset with the demographic metadata still attached, so you can target the populations or probes where it is weakest.
Every trajectory is auditable. A run records the full message list, each tool call and its result, the per-turn judge decisions, the assets and prompt versions it drew on, and the resolved model identities. A number in a report can always be traced back to the conversation that produced it, which is what makes failure analysis possible rather than just scoring.
How it differs. Benchmarks like MMLU or tau-bench define the task first and bolt on users second; this inverts that: start with a census-grounded population, then derive tasks from persona attributes. The user is simulated with the same care as the system under test. The test surface grows through a probe registry rather than being frozen at release, and extensions plug in through entry points without forking the repository.
Python 3.10+, in a virtual environment.
python -m venv .venv && source .venv/bin/activate
pip install "usersim>=0.2.0rc1" # in a uv project: uv add "usersim>=0.2.0rc1"
# Install the NGC CLI into the active virtual environment (idempotent,
# sha256-verified). Required to download the Nemotron-Personas datasets.
usersim setup-ngc
# Keys for live runs and persona download.
export NVIDIA_API_KEY="your-key-from-build.nvidia.com" # inference
export NGC_CLI_API_KEY="your-key-from-ngc.nvidia.com" # persona download
# Verify the install offline: no LLM calls, no network, CI-ready.
usersim smoke
usersim simulate --dry-run --locale en_US --num-rows 2 --out output/trajectoriessmoke is the fastest way to confirm an install is sound: it resolves the
model catalogue, every probe's assets, and the plugin entry points without
making a single model call. --dry-run prints the plan for a run, also
without calling a model.
To work on User Sim itself, or to run the notebooks, clone the repository and
run uv sync; CONTRIBUTING.md
has the details.
These steps make live model calls, using the keys above.
# 1. Simulate. Pick a locale and a row count; the default probe mix is the
# two `general` probes plus `tool_calling`.
usersim simulate --locale en_US --num-rows 30 --out output/trajectories
# 2. Score the assistant under test.
usersim eval --trajectories output/trajectories \
--out output/evaluations --judges judge_model
# 3. Render the capability dashboard.
usersim report --trajectories output/trajectories \
--evaluations output/evaluations --out output/report/report prints the index.html path as a clickable file:// URL; --open
launches it, and --list-runs shows which runs are available.
Pick any registered probe with --probe-mix:
usersim simulate --locale ja_JP --num-rows 30 \
--probe-mix 'sov_ai_facts=1/3,sov_ai_dynamic=1/3,sov_ai_multilingual_parity=1/3' \
--out output/trajectoriesSimulation produces evaluation signal and training data from the same run.
select applies quality gates over the evaluator's scores and the
simulator's own health metrics, then writes the passing subset with a funnel
manifest recording why each row survived:
usersim select --trajectories output/trajectories \
--evaluations output/evaluations \
--profile strict --out output/curatedTwo profiles ship: generous and strict. Add --stratify-by to balance
across locale or demographic cohorts, --holdout-fraction to reserve an
evaluation split, and --hf-repo to push the result to the Hugging Face Hub
with a generated dataset card.
--locale samples personas fresh each run. To fix the locales and per-locale
row counts up front (for a matched-pair comparison, say), build a panel and
pass it instead:
usersim panel --locale en_US --num-personas 50 --out panel.parquet
usersim simulate --panel panel.parquet --out output/trajectoriesNote that personas are re-sampled per run either way; a panel sets the shape of the run, not its cast.
Inference defaults to build.nvidia.com through
Data Designer's nvidia provider. Six aliases configure a full run:
user_model, assistant_model and judge_model drive the simulation,
api_response_model and summary_model support it, and evaluator_model
scores the result. All six are catalogued in
src/usersim/cli/models_default.toml;
point --models at your own file to change providers, endpoints or sampling.
simulate, eval and gen-assets take --dry-run, which resolves the run's
full plan and exits before the first network call.
Using it
docs/probes.md: the 14 probes and which failure mode each catchesdocs/outputs.md: the parquet columns, evaluation cells and report artifacts a run producesdocs/locales.md: the 9 shipped locales, the 22 India preview variants, persona-language matchingdocs/notebooks.md: the interactive flowarchitecture/selection.md: curating runs into training datasetsdocs/methodology.md: why grounded simulation works, and where to trust it
Understanding it
architecture/overview.md: the pipeline, the subsystems, the repository layoutarchitecture/engine.md路probes.md路evaluator.md: the simulation substratearchitecture/reporting.md路selection.md: turning trajectories into signalarchitecture/cli.md路assets.md: the command surface and asset resolutionarchitecture/plugins.md: adding a probe, scorer or backend without forking
Extending it
docs/engine/AUTHORING_A_PROBE.md: end-to-end tutorial for a new probedocs/engine/README.md: the substrate's full API surfacetemplates/probe/: copy-and-edit starters, one per probe shapeCONTRIBUTING.md: setup, conventions, and how to sign off
Simulation is a measurement system that reaches populations and scenarios production traffic won't. It complements real-user data rather than replacing it, and it's strongest where you have ground truth to check against: tool calls that either match a schema or don't, a tool sequence that either achieved the expected state change or didn't, facts that are either right or wrong, red-flag symptoms that were either caught or missed. Those are also the checks that matter most for an agent, where the final message can look fine while the actions behind it were wrong.
Three things are worth knowing before you quote a number.
Behavioural fidelity is bounded by the model doing the simulating. Current models flatten personality differences relative to what a persona description implies, so interaction-style coverage is best read as breadth across styles rather than a calibrated measure of any one of them.
Judges scoring simulated users is circular for subjective dimensions. A model's opinion of another model's helpfulness is not an independent measurement. Lean on the deterministic scorers where ground truth exists, and reserve human review for taste.
Content review status travels with the numbers. Bank entries authored by a
model rather than reviewed by a domain expert carry placeholder: true, and
every trajectory that drew on one records a used_placeholder_* warning on
its outcome. That warning is written to the trajectory parquet, so you can
always trace which findings rest on reviewed content.
docs/methodology.md covers all three in depth.
NeMo User Sim runs on NeMo Data Designer,
which shares anonymous usage telemetry with NVIDIA: the names of the models used
and their input and output token counts, with no user or device information.
Telemetry is on by default. Disable it with NEMO_TELEMETRY_ENABLED=false.
When User Sim is imported it sets NEMO_SESSION_PREFIX=usersim-, unless the
variable is already set, so these events can be told apart from other Data
Designer usage. Data Designer reads the prefix once, when it is first imported,
so a program that imports Data Designer before User Sim should set the variable
itself. Data Designer's
Telemetry & privacy
section has the details, including how the terms of third-party endpoints such
as NVIDIA Build apply.
Apache-2.0. See LICENSE.