docs(factories): document benchmark suites and tasks as definition files

src/content/docs/factories/factory-as-code.mdx

@@ -2,13 +2,13 @@
 title: Factory definition syntax
 description: >-
   Look up every file and key in a factory definition: factory.yaml, agents,
-  automations, runners, scorers, and skills.
+  automations, runners, scorers, benchmarks, and skills.
 sidebar:
   label: "Definitions as code"
 ---
 import { VARS } from '@data/vars';
 
-Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition.
+Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, benchmarks, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition.
 
 Definition files are YAML and Markdown. Keys are case-sensitive.
 
@@ -49,6 +49,11 @@ runners/
 scorers/
   tests-run/
     scorer.md
+benchmarks/
+  review-quality/
+    suite.yaml
+    tasks/
+      flag-missing-tests.yaml
 skills/
   repository-conventions/
     SKILL.md
@@ -382,6 +387,70 @@ Required. The model that judges the runs.
 
 Optional. When `true`, failing scores can feed the factory's self-improvement flow, which proposes definition changes as pull requests. Defaults to `false`.
 
+## `benchmarks/<suite-slug>/suite.yaml`
+
+Optional. Each directory under `benchmarks/` defines a benchmark suite: fixed tasks that compare configurations of one agent. The directory segment is a stable filesystem slug — the required `name` field is the suite's identity, so renaming the suite doesn't move its files. See [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for how suites run and how to read the results.
+
+```yaml title="benchmarks/review-quality/suite.yaml"
+name: review-quality
+description: Fixed review tasks for comparing models and runners
+agent: reviewer
+tasks:
+  - flag-missing-tests
+```
+
+### `name`
+
+Required. The suite's display name. Must be unique across the factory.
+
+### `description`
+
+Optional. What the suite measures.
+
+### `agent`
+
+Required. The name of an agent defined under [`agents/`](#agentsnameagentmd). Every task in the suite dispatches as this agent.
+
+### `tasks`
+
+Optional. Ordered task slugs, each naming a file at `benchmarks/<suite-slug>/tasks/<slug>.yaml`. Every slug listed here must have a matching file, and every task file must appear here exactly once. A suite with no tasks can be saved, but it can't launch until at least one task exists.
+
+## `benchmarks/<suite-slug>/tasks/<task-slug>.yaml`
+
+One benchmark task, referenced from its suite's `tasks` list by slug — the file name without the extension.
+
+```yaml title="benchmarks/review-quality/tasks/flag-missing-tests.yaml"
+title: Flag the missing tests in a risky diff
+prompt: |
+  Review the latest payments-service diff. Identify the changed code paths
+  that lack test coverage and name the specific tests that are missing.
+successCriteria: |
+  The review names every untested changed code path and proposes a concrete
+  test for each one.
+startingRepoRefs:
+  - github.com:acme/payments-service@0123456789abcdef0123456789abcdef01234567
+```
+
+### `title`
+
+Required. The task's display name. It doesn't need to match the file name.
+
+### `prompt`
+
+Required. The input the agent receives on every trial.
+
+### `successCriteria`
+
+Required. What a passing result looks like. The built-in **Correctness** Scorer judges every trial against these criteria.
+
+### `sourceRunId`
+
+Optional. Provenance for a task created from a completed run. It may later point to a deleted run.
+
+### `startingRepoRefs`
+
+Optional. Repositories the task starts from, each pinned to an exact commit so every launch reproduces the same starting state. Each entry is the shorthand `<host>:<owner>/<repo>@<commit-sha>` (the host is `github.com` or `gitlab.com`) or an object with `codeForge` (`GITHUB` or `GITLAB`), `owner`, `repo`, and `ref`. The `ref` is always a full 40-character commit SHA, never a branch or tag. Entries aren't checked against the factory's `repositories` until a run launches.
+
 ## Skills
 
 A skill is a directory containing a `SKILL.md`, not a YAML key. Skills under `skills/` are available to every agent in the factory; skills under `agents/<name>/skills/` are available only to that agent. See [factory skills](/factories/factory-skills/) for when to add one, and [Skills](/agents/capabilities/skills/) for the file format.

src/content/docs/factories/measure-and-improve.mdx

@@ -71,7 +71,7 @@ A benchmark compares configurations of a single agent on the same fixed tasks, s
 * **Scorers** - Your classification Scorers, applied to every trial.
 * **Repetitions** - The number of trials per task and configuration.
 
-You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch.
+You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. For suites and tasks defined as files in a factory definition, see the [`benchmarks/` syntax](/factories/factory-as-code/#benchmarkssuite-slugsuiteyaml).
 
 Every benchmark also runs **Correctness**, a built-in Scorer that marks each trial as pass or fail against the task's success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don't include model usage, so the true cost is higher.