Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion evals/mcp-tool-use/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ pnpm mcp-tool-use --model claude-opus-5-5 --out runs/opus

- `ANTHROPIC_API_KEY` in the repo root `.env` pays for Claude models and the Haiku shell simulator.
- Codex models (`gpt-*` in `MODELS` in `run.ts`) go through a CLIProxyAPI server, set up as the `vibeproxy` model provider in `~/.codex/config.toml`. Keep the proxy under about 12 samples in flight across all runs.
- Flags: `--case <ids>`, `--cases <file.jsonl>`, `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (samples in flight, default 8), `--max-steps <n>`, `--effort <level>`, `--dry-run` (prints request bodies, no model calls), and `--context` (prints the size of T3's text and schemas, no key needed).
- Flags: `--case <ids>`, `--cases <file.ndjson>`, `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (samples in flight, default 8), `--max-steps <n>`, `--effort <level>`, `--dry-run` (prints request bodies, no model calls), and `--context` (prints the size of T3's text and schemas, no key needed).
- `--baseline <results.jsonl> --stop-on-regression` stops as soon as a case that passed every baseline sample fails. Use it to test one copy change against a baseline run.
- Cost: a full Sonnet run is about $9, and a full Opus run about $17. Codex models through the proxy cost only the shell simulator.

Expand Down
8 changes: 4 additions & 4 deletions evals/mcp-tool-use/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1678,7 +1678,7 @@ const main = Effect.gen(function* () {
}
const fs = yield* FileSystem.FileSystem;
const path = yield* Path.Path;
const readJsonl = <A, E>(file: string, decode: (line: string) => Effect.Effect<A, E>) =>
const readNdjson = <A, E>(file: string, decode: (line: string) => Effect.Effect<A, E>) =>
fs.readFileString(file).pipe(
Effect.map((content) => content.split("\n").filter((line) => line.trim() !== "")),
Effect.flatMap((lines) => Effect.forEach(lines, (line) => decode(line))),
Expand All @@ -1687,10 +1687,10 @@ const main = Effect.gen(function* () {
const caseFiles =
flags.cases ??
(yield* fs.readDirectory(casesDir))
.filter((name) => name.endsWith(".jsonl"))
.filter((name) => name.endsWith(".ndjson"))
.toSorted()
.map((name) => path.join(casesDir, name));
const cases = (yield* Effect.forEach(caseFiles, (file) => readJsonl(file, decodeCase))).flat();
const cases = (yield* Effect.forEach(caseFiles, (file) => readNdjson(file, decodeCase))).flat();
const wanted = flags.case?.flatMap((ids) => ids.split(","));
const codex = isCodex(flags.model);
const harness = codex ? "codex" : "claude";
Expand Down Expand Up @@ -1764,7 +1764,7 @@ const main = Effect.gen(function* () {
shellModel: flags["shell-model"],
};
const previous =
flags.baseline === undefined ? [] : yield* readJsonl(flags.baseline, decodeBaselineRow);
flags.baseline === undefined ? [] : yield* readNdjson(flags.baseline, decodeBaselineRow);
const baseline = new Map(groups(previous, (row) => row.id).map(([id, r]) => [id, passRate(r)]));
// Only a case that passed every baseline sample can stop a run; flaky ones are just reported.
const passedBefore = (id: string) => baseline.get(id) === 1;
Expand Down
Loading