diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index 4ef52de2..b835bdc3 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -2,13 +2,13 @@ title: Factory definition syntax description: >- Look up every file and key in a factory definition: factory.yaml, agents, - automations, runners, scorers, and skills. + automations, runners, scorers, benchmarks, and skills. sidebar: label: "Definitions as code" --- import { VARS } from '@data/vars'; -Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition. +Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, benchmarks, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition. Definition files are YAML and Markdown. Keys are case-sensitive. @@ -49,6 +49,11 @@ runners/ scorers/ tests-run/ scorer.md +benchmarks/ + review-quality/ + suite.yaml + tasks/ + flag-missing-tests.yaml skills/ repository-conventions/ SKILL.md @@ -382,6 +387,70 @@ Required. The model that judges the runs. Optional. When `true`, failing scores can feed the factory's self-improvement flow, which proposes definition changes as pull requests. Defaults to `false`. +## `benchmarks//suite.yaml` + +Optional. Each directory under `benchmarks/` defines a benchmark suite: fixed tasks that compare configurations of one agent. The directory segment is a stable filesystem slug — the required `name` field is the suite's identity, so renaming the suite doesn't move its files. See [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for how suites run and how to read the results. + +```yaml title="benchmarks/review-quality/suite.yaml" +name: review-quality +description: Fixed review tasks for comparing models and runners +agent: reviewer +tasks: + - flag-missing-tests +``` + +### `name` + +Required. The suite's display name. Must be unique across the factory. + +### `description` + +Optional. What the suite measures. + +### `agent` + +Required. The name of an agent defined under [`agents/`](#agentsnameagentmd). Every task in the suite dispatches as this agent. + +### `tasks` + +Optional. Ordered task slugs, each naming a file at `benchmarks//tasks/.yaml`. Every slug listed here must have a matching file, and every task file must appear here exactly once. A suite with no tasks can be saved, but it can't launch until at least one task exists. + +## `benchmarks//tasks/.yaml` + +One benchmark task, referenced from its suite's `tasks` list by slug — the file name without the extension. + +```yaml title="benchmarks/review-quality/tasks/flag-missing-tests.yaml" +title: Flag the missing tests in a risky diff +prompt: | + Review the latest payments-service diff. Identify the changed code paths + that lack test coverage and name the specific tests that are missing. +successCriteria: | + The review names every untested changed code path and proposes a concrete + test for each one. +startingRepoRefs: + - github.com:acme/payments-service@0123456789abcdef0123456789abcdef01234567 +``` + +### `title` + +Required. The task's display name. It doesn't need to match the file name. + +### `prompt` + +Required. The input the agent receives on every trial. + +### `successCriteria` + +Required. What a passing result looks like. The built-in **Correctness** Scorer judges every trial against these criteria. + +### `sourceRunId` + +Optional. Provenance for a task created from a completed run. It may later point to a deleted run. + +### `startingRepoRefs` + +Optional. Repositories the task starts from, each pinned to an exact commit so every launch reproduces the same starting state. Each entry is the shorthand `:/@` (the host is `github.com` or `gitlab.com`) or an object with `codeForge` (`GITHUB` or `GITLAB`), `owner`, `repo`, and `ref`. The `ref` is always a full 40-character commit SHA, never a branch or tag. Entries aren't checked against the factory's `repositories` until a run launches. + ## Skills A skill is a directory containing a `SKILL.md`, not a YAML key. Skills under `skills/` are available to every agent in the factory; skills under `agents//skills/` are available only to that agent. See [factory skills](/factories/factory-skills/) for when to add one, and [Skills](/agents/capabilities/skills/) for the file format. diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index aec5a541..2e4647a9 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -71,7 +71,7 @@ A benchmark compares configurations of a single agent on the same fixed tasks, s * **Scorers** - Your classification Scorers, applied to every trial. * **Repetitions** - The number of trials per task and configuration. -You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. +You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. For suites and tasks defined as files in a factory definition, see the [`benchmarks/` syntax](/factories/factory-as-code/#benchmarkssuite-slugsuiteyaml). Every benchmark also runs **Correctness**, a built-in Scorer that marks each trial as pass or fail against the task's success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don't include model usage, so the true cost is higher.