Repository navigation
Simulated user v2, optional CHECKER (official setup), grader 0.8.7 - #1
Merged
Merged
Conversation
… of more help - Before "i don't know" (and on a finished turn), ask about each action the user has a rule for with the grader's own re-read question. - Answer an offer of optional further help with "no thanks, that's all for now"; the grader checks a decline as it checks "i don't know". - Record the simulated user's version (2) with each run. - Grader 0.8.6: an end that keeps the length of the event it moves is sourced in code. - Preference facts for three tasks, found by experiments/gaps.py and reviewed by hand. - experiments/: replay harness over recorded conversations, gap discovery, live run script. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…r styles - Route a missed proposal to its approval rule only when the reader and the rules judge both read it as one: with the reader alone, independent models rejected about a third of the routings. - Grader: optional CHECKER role, a model other than the judge that settles re-checks of the simulated user's approval decisions when the reader is unsure. Unset, verdicts are unchanged. - Writer: a prompt that says what to keep and what not to add (52% -> 67% of replies reworded in testing, no rise in meaning changes), and an opt-in ASSISTANTEVAL_USER_STYLE of terse or distracted. - Experiments: style overlays, an independent audit of routed decisions, probing of full run records, live A/B summary. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- experiments/RESULTS.md: method, population replay, independent audits, live A/B, writer and style results, what did not work, comparability. - Go-ahead twins: drop the user goal sentence that contradicted their openings (documentation only; no code reads it). - Share the simulated user's earlier-messages helper. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Harbor smoke's scripted user reached the decline check, which the keyword judge did not know. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A live Hermes run ended "if you'd rather have the debrief on Monday, tell me and I'll ask Ben"; the reader could not tell whether it had finished, and the user said "i don't know". An unsure reading that is an offer now gets a decline, which the grader checks as it checks "i don't know". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A question with no rungs (whether a result holds despite the simulated user's misreadings) is a person's alone; the checker answered it too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…reement Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- contracts.Stop: a host's reason to end a run now (such as a lost account lease) is recorded with its code, like a stop signal. - run_task.stop_on_signals is public. - grader and report commands take the answers store's class, for a host whose saved answers need reading another way. - An installed runner never asks git for its revision: it would name the repository the package is installed in. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er 0.8.7) The causality question names the checker as its only rung: never the reader or the judge, whose readings it weighs. Without a checker it waits for a person as before. The official setup now sets a CHECKER (Gemini 3.1 Pro). On the 360-run sweep, runs waiting for a person fall from 21 to 3 (published simulated user) and 20 to 1 (version 2). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
that-guy-wade
changed the title
that-guy-wade
marked this pull request as ready for review
October 10, 2026 21:09
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| recheck = "reader" in q.rungs and "judge" not in q.rungs | ||
| rungs = {*q.rungs, *(("checker",) if recheck else ())} | ||
| for rung in ("reader", "judge", "checker"): | ||
| if rung not in rungs or (rung == "checker" and not self.checker): |
There was a problem hiding this comment.
When a run was settled by CHECKER, regrading with --offline skips its saved answers. Ladder(store) has no live checker, so this condition skips the rung before _answer can read the cache. The run can return to needs_human_annotation despite having all its answers saved. Allow saved checker answers without requiring a live checker.
Comment on lines
+114
to
+116
| if CURRENT not in system: | ||
| raise SystemExit("_voice's prompt changed: CURRENT no longer matches") | ||
| return super().__call__(system.replace(CURRENT, TIGHT), user) |
There was a problem hiding this comment.
The documented tight.py run command exits on its first writer call. TightWriter requires the old CURRENT text, but _voice now sends REWRITE, which already equals TIGHT. This prevents teammates from rerunning the comparison. Keep explicit old and new prompts in the experiment rather than requiring production to still use the old one.
Comment on lines
+246
to
+248
| checker = endpoint("CHECKER") # only code with a checker rung takes one | ||
| extra = {"checker": Cached(judge_for(checker), cache)} if checker else {} | ||
| ladder = Ladder(store, reader=reader, judge=rules, **extra) |
There was a problem hiding this comment.
With CHECKER set by the documented official setup, AE_CODE cannot replay the published checkout. This code passes checker= to the imported Ladder, but the base checkout's constructor has no such argument. The baseline comparison fails before measuring any decisions. Check whether that checkout supports the checker before passing it, and leave the baseline arm checker-free.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Experiments to make the simulated user less brittle and more realistic, and the changes that held up. Results and method are in
experiments/RESULTS.md. Decided: adopt simulated user version 2; the official setup includes a CHECKER (Gemini 3.1 Pro), which also answers whether a result holds despite the simulated user's misreadings (grader 0.8.7).Results
Replay of every decision in all 1,489 published runs (2,521 decisions), official judges:
Live A/B (model baseline, Claude Sonnet 5.5 and Gemini 3.1 Pro, 8 tasks × 2, 32 runs per arm): "i don't know" 8/42 → 2/32 replies; runs with a disputed reading 10 → 6; outcomes waiting for a person 9 → 6; replies sent reworded 55% → 75%;
flight-before-ben2 pass + 2 pending → 4 pass.Regrading those live records with the opt-in CHECKER (Gemini 3.1 Pro): runs waiting for a person 9 → 7 with the published code, and 6 → 3 with this branch. On the full published records the checker agreed with 31 of the 34 adjudications it could answer. (An earlier checker also answered the person-only causality question; fixed.)
Hermes Agent in Harbor with simulated user v2: 5 live trials ran cleanly;
flight-before-ben(the case that started this) passed twice with no human annotation and no CHECKER.Impact on scores: the five published model assistants on the 12 tasks where v2 could matter, 3× each (180 runs per arm): pass rate 64.8% → 65.0%, per-model differences within noise; "i don't know" 7/98 → 2/89 replies. On the 1,530 full published records, this grader reproduces all 918 trial 3–5 outcomes from the published answers.
Changes
assistanteval/simulated_user.py, recorded in each run):ASSISTANTEVAL_USER_STYLE=terse|distracted(wording only; tested against the "same meaning, adds nothing" check).CHECKERrole: a model other than the judge settles re-checks of the simulated user's approval decisions when the reader is unsure. Unset, verdicts are unchanged (parity suite).experiments/gaps.py, reviewed by hand; go-ahead twins' contradictory goal sentence removed.Comparability
New runs with simulated user version 2 are not directly comparable with published runs on tasks where the user said "i don't know". Grader 0.8.6 differs from 0.8.5 only for kept-length ends. CHECKER changes verdicts only when set.
Testing
Complexity
Package: +180 / −32 lines.
SimulatedUser.replyCC 18 → 23 (the decline and routing branches);Ladder.decide_one8 → 10 (checker rung); new helpers ≤ 6. Experiments add ~3,000 lines of scripts and write-up outside the package.Follow-ups (not in this PR)
gaps.pyplus human review as new assistants are run.Linear: ORO-2748
🤖 Generated with Claude Code
Fix the lost saved checker answers before merging; the experiment failures are lower-impact follow-ups.
Summary
This PR adds simulated user version 2, optional
CHECKERgrading, task preferences, and experiment scripts.Review scope: the previous-review commit was unavailable, so the full PR diff was used. No previous threads or numbered findings were supplied; GitHub authentication was unavailable to retrieve them. Checks were based on source reads, not runtime execution.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD Q[Grading question] --> H{Saved human answer?} H -->|Yes| D[Use answer] H -->|No| R{Allowed reader is sure?} R -->|Yes| D R -->|No| J{Allowed judge is sure?} J -->|Yes| D J -->|No| C{Allowed checker is sure?} C -->|Yes| D C -->|No| A[Ask a person]Reviews (3) · Last reviewed commit: "Official setup: the CHECKER also answers..." · Reviewed by Greptile