From a061f46ebb6e74c9504f9850697cf2fa202cef53 Mon Sep 17 00:00:00 2001 From: Hong Yi Chen Date: Mon, 31 Aug 2026 11:56:41 -0700 Subject: [PATCH 1/2] docs(factories): document benchmark suites and tasks as definition files The benchmarks//suite.yaml and benchmarks//tasks/.yaml file kinds are published in the machine-readable schema bundle (app.warp.dev/api/v1/factory-files/schemas/v1alpha1) but were the only definition kinds missing from the syntax reference. Adds both sections with fields from the published schemas, the directory-structure entry, and a pointer from the benchmarks section of Measure and improve. Co-Authored-By: Warp --- .../docs/factories/factory-as-code.mdx | 73 ++++++++++++++++++- .../docs/factories/measure-and-improve.mdx | 2 +- 2 files changed, 72 insertions(+), 3 deletions(-) diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index 4ef52de2..2bc8bad2 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -2,13 +2,13 @@ title: Factory definition syntax description: >- Look up every file and key in a factory definition: factory.yaml, agents, - automations, runners, scorers, and skills. + automations, runners, scorers, benchmarks, and skills. sidebar: label: "Definitions as code" --- import { VARS } from '@data/vars'; -Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition. +Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, benchmarks, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition. Definition files are YAML and Markdown. Keys are case-sensitive. @@ -49,6 +49,11 @@ runners/ scorers/ tests-run/ scorer.md +benchmarks/ + implementation-quality/ + suite.yaml + tasks/ + fix-flaky-test.yaml skills/ repository-conventions/ SKILL.md @@ -382,6 +387,70 @@ Required. The model that judges the runs. Optional. When `true`, failing scores can feed the factory's self-improvement flow, which proposes definition changes as pull requests. Defaults to `false`. +## `benchmarks//suite.yaml` + +Optional. Each directory under `benchmarks/` defines a benchmark suite: fixed tasks that compare configurations of one agent. The directory segment is a stable filesystem slug — the required `name` field is the suite's identity, so renaming the suite doesn't move its files. See [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for how suites run and how to read the results. + +```yaml title="benchmarks/implementation-quality/suite.yaml" +name: implementation-quality +description: Fixed implementation tasks for comparing models and runners +agent: implementer +tasks: + - fix-flaky-test +``` + +### `name` + +Required. The suite's display name. Must be unique across the factory. + +### `description` + +Optional. What the suite measures. + +### `agent` + +Required. The name of an agent defined under [`agents/`](#agentsnameagentmd). Every task in the suite dispatches as this agent. + +### `tasks` + +Optional. Ordered task slugs, each naming a file at `benchmarks//tasks/.yaml`. Every slug listed here must have a matching file, and every task file must appear here exactly once. A suite with no tasks can be saved, but it can't launch until at least one task exists. + +## `benchmarks//tasks/.yaml` + +One benchmark task, referenced from its suite's `tasks` list by slug — the file name without the extension. + +```yaml title="benchmarks/implementation-quality/tasks/fix-flaky-test.yaml" +title: Fix the flaky payments integration test +prompt: | + The payments integration test fails intermittently on CI. Find the cause + and fix it without changing the test's assertions. +successCriteria: | + The flaky test is deterministic, the fix explains the root cause, and the + full test suite passes. +startingRepoRefs: + - github.com:acme/payments-service@0123456789abcdef0123456789abcdef01234567 +``` + +### `title` + +Required. The task's display name. It doesn't need to match the file name. + +### `prompt` + +Required. The input the agent receives on every trial. + +### `successCriteria` + +Required. What a passing result looks like. The built-in **Correctness** Scorer judges every trial against these criteria. + +### `sourceRunId` + +Optional. Provenance for a task created from a completed run. It may later point to a deleted run. + +### `startingRepoRefs` + +Optional. Repositories the task starts from, each pinned to an exact commit so every launch reproduces the same starting state. Each entry is the shorthand `:/@` (the host is `github.com` or `gitlab.com`) or an object with `codeForge` (`GITHUB` or `GITLAB`), `owner`, `repo`, and `ref`. The `ref` is always a full 40-character commit SHA, never a branch or tag. Entries aren't checked against the factory's `repositories` until a run launches. + ## Skills A skill is a directory containing a `SKILL.md`, not a YAML key. Skills under `skills/` are available to every agent in the factory; skills under `agents//skills/` are available only to that agent. See [factory skills](/factories/factory-skills/) for when to add one, and [Skills](/agents/capabilities/skills/) for the file format. diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index aec5a541..2e4647a9 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -71,7 +71,7 @@ A benchmark compares configurations of a single agent on the same fixed tasks, s * **Scorers** - Your classification Scorers, applied to every trial. * **Repetitions** - The number of trials per task and configuration. -You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. +You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. For suites and tasks defined as files in a factory definition, see the [`benchmarks/` syntax](/factories/factory-as-code/#benchmarkssuite-slugsuiteyaml). Every benchmark also runs **Correctness**, a built-in Scorer that marks each trial as pass or fail against the task's success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don't include model usage, so the true cost is higher. From 802ce93e0074cc5bbf9b0aab90852c5ae081a271 Mon Sep 17 00:00:00 2001 From: Hong Yi Chen Date: Mon, 31 Aug 2026 13:05:05 -0700 Subject: [PATCH 2/2] docs(factories): use a declared agent in the benchmark suite example Self-review fix: the suite example dispatched an 'implementer' agent that the page's directory tree and full example never declare, so copying it into the page's own factory would fail validation. Recast the example around the declared reviewer agent. Co-Authored-By: Warp --- .../docs/factories/factory-as-code.mdx | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index 2bc8bad2..b835bdc3 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -50,10 +50,10 @@ scorers/ tests-run/ scorer.md benchmarks/ - implementation-quality/ + review-quality/ suite.yaml tasks/ - fix-flaky-test.yaml + flag-missing-tests.yaml skills/ repository-conventions/ SKILL.md @@ -391,12 +391,12 @@ Optional. When `true`, failing scores can feed the factory's self-improvement fl Optional. Each directory under `benchmarks/` defines a benchmark suite: fixed tasks that compare configurations of one agent. The directory segment is a stable filesystem slug — the required `name` field is the suite's identity, so renaming the suite doesn't move its files. See [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for how suites run and how to read the results. -```yaml title="benchmarks/implementation-quality/suite.yaml" -name: implementation-quality -description: Fixed implementation tasks for comparing models and runners -agent: implementer +```yaml title="benchmarks/review-quality/suite.yaml" +name: review-quality +description: Fixed review tasks for comparing models and runners +agent: reviewer tasks: - - fix-flaky-test + - flag-missing-tests ``` ### `name` @@ -419,14 +419,14 @@ Optional. Ordered task slugs, each naming a file at `benchmarks//tas One benchmark task, referenced from its suite's `tasks` list by slug — the file name without the extension. -```yaml title="benchmarks/implementation-quality/tasks/fix-flaky-test.yaml" -title: Fix the flaky payments integration test +```yaml title="benchmarks/review-quality/tasks/flag-missing-tests.yaml" +title: Flag the missing tests in a risky diff prompt: | - The payments integration test fails intermittently on CI. Find the cause - and fix it without changing the test's assertions. + Review the latest payments-service diff. Identify the changed code paths + that lack test coverage and name the specific tests that are missing. successCriteria: | - The flaky test is deterministic, the fix explains the root cause, and the - full test suite passes. + The review names every untested changed code path and proposes a concrete + test for each one. startingRepoRefs: - github.com:acme/payments-service@0123456789abcdef0123456789abcdef01234567 ```