docs(factories): document benchmark suites and tasks as definition files

[hongyi-chen]

What this feature does

Benchmark suites compare configurations of one agent on the same fixed tasks. Suites and their tasks can be defined as version-controlled files — benchmarks/<suite-slug>/suite.yaml plus tasks/<task-slug>.yaml — like every other factory resource, and they were the only definition file kinds missing from the syntax reference. Shipped in v0.2026.08.26.17.59.stable_01 (2026-08-27).

Summary

Found during a factories docs-coverage audit against the published definition schemas. Every field, requirement, and constraint is sourced from the live schema bundle at app.warp.dev/api/v1/factory-files/schemas/v1alpha1 (benchmark_suite.schema.json, benchmark_suite_task.schema.json).

Changes

src/content/docs/factories/factory-as-code.mdx

• Added benchmarks/<suite-slug>/suite.yaml and benchmarks/<suite-slug>/tasks/<task-slug>.yaml sections with per-field reference entries, matching the existing per-kind structure
• Added the benchmarks/ tree to the directory structure listing and to the page intro and description

src/content/docs/factories/measure-and-improve.mdx

• One-line pointer from "Compare configurations with benchmarks" to the new benchmarks/ syntax sections

Content design plan

Reader and job: A team that manages its factory as definitions as code and wants benchmark suites reviewed, versioned, and rolled back like the rest of the factory.

Gap today: factory-as-code documents every file kind except benchmarks/, and Measure and improve covers only the dashboard flow — file-based suites were undiscoverable.

Change: Two per-kind reference sections (suite.yaml fields; task file fields including commit-pinned startingRepoRefs) plus the directory listing and a cross-link. Excludes benchmark concepts and result interpretation, which stay on Measure and improve.

Unverified claims

None — all fields, requirements, and constraints (including the save-vs-launch behavior for empty suites and the 40-character commit-SHA pinning rules) come from the published schema bundle's own descriptions.

Additional context

The same audit evaluated other factories docs candidates; the notable deferral, recorded per the docs-worthiness criteria:

• Custom webhook intake (provider: webhook trigger, FactoryWebhook API): deferred, not rejected. The trigger provider is in the published schema, but the management endpoints (/factory/webhooks/*: create, rotate, deliveries, dry-run) are marked x-internal and there is no dashboard surface, so no released end-to-end setup path exists to document. Re-surface when those endpoints reach developers/agent-api-openapi.yaml (via the standing spec-sync automation) or a dashboard webhooks surface ships.
• Also evaluated and not pursued: an examples gallery page, foreman/scorers/pricing split-out pages, and a factories FAQ — reasons in the linked plan.

---

• Warp conversation
• Implementation plan

Co-Authored-By: Warp agent@warp.dev

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview	Aug 31, 2026 8:08pm

[warp-for-oss]

@hongyi-chen

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[warp-for-oss]

Overview

This PR adds benchmark suite and benchmark task definition-file syntax to the factory-as-code reference, updates the page metadata and directory tree, and links the benchmark overview page to the new reference sections.

Concerns

No material concerns found. The changed docs are documentation-only, the internal anchors match existing generated-anchor conventions for path-like headings, and spec_context.md reports no approved or repository spec context for implementation drift validation.

Verdict

Found: 0 critical, 0 important, 0 suggestions

Approve

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[warp-factories]

Overview

Adds benchmarks/<suite-slug>/suite.yaml and benchmarks/<suite-slug>/tasks/<task-slug>.yaml reference sections to the factory-as-code page and cross-links them from Measure and improve. Verified every field/requirement against the published benchmark_suite.schema.json and benchmark_suite_task.schema.json — the docs match the schema exactly (required/optional fields, startingRepoRefs shorthand and object forms, 40-char SHA constraint, save-vs-launch behavior). New anchors resolve correctly (confirmed with the repo's link checker) and CI is green.

Verdict

Checks: build pass, tests n/a, CI green, visual proof n/a (reference-only text change)

Found: 0 critical, 0 important, 0 suggestions, 1 nit

Approve with nits.

[warp-factories]

🧹 [NIT] Placeholder mismatch: this sentence uses <slug> (.../tasks/<slug>.yaml) but the section heading two lines below uses <task-slug> for the same path. Suggest aligning to <task-slug> here for consistency.

Suggested change

	Optional. Ordered task slugs, each naming a file at `benchmarks/<suite-slug>/tasks/<slug>.yaml`. Every slug listed here must have a matching file, and every task file must appear here exactly once. A suite with no tasks can be saved, but it can't launch until at least one task exists.
	Optional. Ordered task slugs, each naming a file at `benchmarks/<suite-slug>/tasks/<task-slug>.yaml`. Every slug listed here must have a matching file, and every task file must appear here exactly once. A suite with no tasks can be saved, but it can't launch until at least one task exists.