Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ An eval runs a real model against T3 Code's real text or code and scores what th

Write or run an eval when a change affects what a model sees or does: agent instructions, MCP tool descriptions and schemas, tool results, generated titles, or the guidance an agent gets for a feature such as Device Manager.

| Eval | Measures |
| ------------------------------- | --------------------------------------------------------------------------- |
| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly in fake threads. |
| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
| [test-signal](test-signal/) | Whether a cheap model running the test-signal check flags low-signal tests. |
| Eval | Measures |
| ------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly, including device routing and recovery, in fake threads. |
| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
| [test-signal](test-signal/) | Whether a cheap model running the test-signal check flags low-signal tests. |

## Run an eval

Expand Down
83 changes: 81 additions & 2 deletions evals/mcp-tool-use/README.md

Large diffs are not rendered by default.

262 changes: 262 additions & 0 deletions evals/mcp-tool-use/agent-device-help.txt

Large diffs are not rendered by default.

16 changes: 16 additions & 0 deletions evals/mcp-tool-use/cases/device-manager.ndjson

Large diffs are not rendered by default.

1,088 changes: 1,088 additions & 0 deletions evals/mcp-tool-use/device-grading.ts

Large diffs are not rendered by default.

913 changes: 913 additions & 0 deletions evals/mcp-tool-use/devices.test.ts

Large diffs are not rendered by default.

952 changes: 952 additions & 0 deletions evals/mcp-tool-use/devices.ts

Large diffs are not rendered by default.

2,891 changes: 2,890 additions & 1 deletion evals/mcp-tool-use/run.test.ts

Large diffs are not rendered by default.

371 changes: 313 additions & 58 deletions evals/mcp-tool-use/run.ts

Large diffs are not rendered by default.

Loading