Skip to content

feat(eval): warn when a trajectory_id appears more than once in a run - #41

Merged
3mei merged 2 commits into
mainfrom
isarasua/feat/warn-repeated-trajectory-ids
Oct 14, 2026
Merged

3mei merged 2 commits into
mainfrom
isarasua/feat/warn-repeated-trajectory-ids

Conversation

@isarasua

Copy link
Copy Markdown
Contributor

What changed and why

One trajectory_id is one conversation, but nothing checked that a run holds it once. A writer that
adds a restarted conversation's row beside the earlier one gives it two rows, and usersim eval
scores both, so it counts twice. Now eval warns, naming the ids:

2 trajectory_id(s) appear more than once in this run: t000000000000001 (2 rows), ... Each copy is
evaluated as a row of its own, so such a conversation can count more than once; a restarted
conversation's row should replace the earlier one.
  • Over the whole run. The check reads every row's id before any sampling, so a pass with
    --num-records or --num-records-per-locale still sees a repeat outside its sample. It names up to
    five ids and counts the rest.
  • Rows are evaluated as they are. Only the warning is new.
  • Docs. docs/outputs.md states the convention: one id per conversation per run, derived from
    what the conversation stands for, not from the attempt. A restart replaces its row. It also says what eval does with a repeat.

Closes #40

How it was verified

  • The new tests in tests/test_cli_eval_sampling.py:
    • a repeated id is named, with its row count, for a full pass, --num-records 1 and
      --num-records-per-locale 1, and every row is still evaluated
    • a run without repeats gives no warning
  • Deliberate breaks: dropping the check from either branch of _choose_rows fails its cases.
  • The full suite passes on Python 3.12 and 3.10 (3,987 tests). make check-all, make check-license-headers and
    make check-doc-links pass.

🤖 Generated with Claude Code

One trajectory_id is one conversation, but nothing checked that a run
holds it once. A writer that adds a restarted conversation's row beside
the earlier one gives it two rows, and eval scores both, so it counts
twice.

`usersim eval` now reads every row's id before any sampling and logs one
warning naming the repeated ids, up to five, with their row counts.
Rows are still evaluated as they are. `docs/outputs.md` states the
convention: one id per conversation per run, derived from what the
conversation stands for, not from the attempt, with a restart replacing
its row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: isarasua <isarasua@nvidia.com>
@isarasua
isarasua requested a review from a team as a code owner October 13, 2026 21:03

@3mei 3mei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isarasua one doc change is needed

Comment thread docs/outputs.md Outdated
… setup

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei self-requested a review October 14, 2026 13:02

@3mei 3mei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging

@3mei
3mei merged commit 7c21812 into main Oct 14, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

usersim eval scores a repeated trajectory_id once per row, silently

2 participants