From 85e49334d54b1637eab00aed328cd2682607f7af Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Tue, 7 Jul 2026 14:40:02 +0200 Subject: [PATCH] docs(evals): audit promptfoo schema parity --- docs/plans/promptfoo-schema-parity-audit.md | 114 ++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 docs/plans/promptfoo-schema-parity-audit.md diff --git a/docs/plans/promptfoo-schema-parity-audit.md b/docs/plans/promptfoo-schema-parity-audit.md new file mode 100644 index 000000000..d83a27cc0 --- /dev/null +++ b/docs/plans/promptfoo-schema-parity-audit.md @@ -0,0 +1,114 @@ +# Promptfoo Schema Parity Audit + +Date: 2026-07-07 + +AgentV source: `d89518e96d332248b3c670d28032ca15cca1ef75` +Promptfoo source: `origin/main` at `786b2bdfe0099c823f1cee8ac93de2965be69771` + +This audit compares AgentV authored eval YAML against Promptfoo's current config +surface. Sources inspected: + +- AgentV schema/parser/validator: + `packages/core/src/evaluation/validation/eval-file.schema.ts`, + `packages/core/src/evaluation/yaml-parser.ts`, + `packages/core/src/evaluation/validation/eval-validator.ts`, + `skills-data/agentv-eval-writer/references/eval.schema.json` +- Promptfoo schema/types: + `/home/entity/projects/promptfoo/promptfoo/src/types/index.ts`, + `/home/entity/projects/promptfoo/promptfoo/src/validators/providers.ts`, + `/home/entity/projects/promptfoo/promptfoo/src/util/config/load.ts`, + `/home/entity/projects/promptfoo/promptfoo/site/static/config-schema.json` +- Product boundary docs: + `AGENTS.md`, `STRATEGY.md`, `ROADMAP.md`, + `.agents/product-boundary.md`, `.agents/conventions.md`, + `CONCEPTS.md`, `apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx` + +No builds, tests, evals, or schema/runtime changes were run for this research task. + +## Bottom Line + +AgentV is already closest to Promptfoo where it matters most: `prompts x tests/vars x providers`, +assertion objects, provider `id`/`label`, `env`, `scenarios`, `default_test`, +`derived_metrics`, `output_path`, `nunjucks_filters`, and `evaluate_options`. + +The largest remaining divergence is not capability, but authoring surface area. +AgentV still accepts several non-Promptfoo top-level and nested fields. Some are +intentional AgentV extensions, but several should be removed, moved, or explicitly +classified before they become durable public API. + +Keep these as intentional AgentV exceptions: + +- `environment` recipes at suite/test scope, including `file://` recipes. +- `ref://` composition for AgentV config-managed references. +- AgentV built-in providers and provider runtime details, behind clear Promptfoo + export diagnostics when not directly executable by Promptfoo. +- AgentV run bundles, Dashboard, and result artifact fields. + +Everything else should either align with Promptfoo directly or have an explicit +decision explaining why the extra AgentV surface is worth keeping. + +## High-Confidence Follow-Ups + +| Bead | Recommendation | Why | +| --- | --- | --- | +| `av-ibec` | Remove/hard-deprecate top-level eval YAML `experiment`; use `tags.experiment` or CLI `--experiment`. | `CONCEPTS.md` and docs already say there is no top-level `experiment`, but schema/parser still accept it. Promptfoo has no equivalent. | +| `av-sdit` | Add missing Promptfoo provider shapes: provider maps and `providers[].inputs`; document/export-gate AgentV-only provider `runtime`/`environment`/`hooks`. | Promptfoo `ProvidersSchema` accepts maps and `ProviderOptions.inputs`; AgentV docs claim this provider-layer parity but schema does not fully implement it. | +| `av-9j60` | Enforce native AgentV `snake_case` eval wire keys and translate at the Promptfoo boundary. | Promptfoo's established fields are `defaultTest`, `evaluateOptions`, `outputPath`, `derivedMetrics`, `nunjucksFilters`, etc.; AgentV's settled wire-format convention keeps native authored YAML in `snake_case`. Import/export/transpile must map to Promptfoo `camelCase` where required, and native AgentV validation should guide users away from accidental Promptfoo casing. | +| `av-sve2` | Move eval-level `defaults` and top-level `assert` toward Promptfoo-compatible slots (`default_test`, assertion `provider`, test/default options). | Promptfoo has `defaultTest` and assertion/test provider overrides, but no top-level `defaults` or suite-level `assert`. Keep `defaults.grader` only if deliberately config-level AgentV policy. | +| `av-n7ic` | Align `extensions` with Promptfoo's `file://path:function_name` contract or rename/document AgentV lifecycle hooks as AgentV-only. | AgentV validates `file://path:beforeAll`/`beforeEach`/`afterEach`/`afterAll` plus `agentv:agent-rules`; Promptfoo accepts extension function references. | +| `av-8h4l` | Prune or classify AgentV-only run-policy fields such as top-level `timeout_seconds`, `tests[].run`, and `tests[].execution`. | Promptfoo uses `evaluateOptions`, `tests[].options.repeat`, and assertion/test thresholds. AgentV has several overlapping run-policy paths. | +| `av-eniz` | Converge eval metadata and tags on Promptfoo-compatible `metadata` and `tags: Record`. | AgentV exposes top-level `name`, `category`, `version`, `author`, `license`, `requires`, and allows tags as string/list or non-string record values. | +| `av-ij5o` | Clarify or deprecate AgentV-only authored assertion types, especially public `llm-grader`. | Conventions say public authored YAML should use `llm-rubric` for Promptfoo-compatible rubric checks, while schema still accepts `llm-grader`. | + +## Divergence Table + +| Field/path | AgentV behavior | Promptfoo equivalent | Recommendation | Sources | +| --- | --- | --- | --- | --- | +| `providers` | Accepts strings and objects with `id`, `label`, `config`, `runtime`, `prompts`, `transform`, `delay`, `env`, `environment`, `hooks`; rejects old `targets`. | `ProvidersSchema` accepts string, provider function, array items, provider maps, and `ProviderOptions` with `id`, `label`, `config`, `prompts`, `transform`, `delay`, `env`, `inputs`. | Modify. Add provider maps and `inputs`; keep `runtime`/provider-local overlays only as explicit AgentV extensions. | AgentV schema lines 663-700; Promptfoo `src/validators/providers.ts` lines 15-24, 80-91. | +| `targets` / `target` | Hard-rejected in authored eval YAML with migration to `providers`. Internal artifacts still use target vocabulary. | Promptfoo `UnifiedConfigSchema` still accepts `targets` as an alias and rewrites it to `providers`. | Keep AgentV rejection. This is a cleaner greenfield choice and already matches product vocabulary. | AgentV schema lines 984-995; Promptfoo `src/types/index.ts` lines 1318-1338; `CONCEPTS.md` lines 7-12. | +| Provider `label` vs `id` | `id` is backend/spec; `label` is stable AgentV identity. | Promptfoo provider options have both `id` and `label`; CLI matching checks id then label. | Keep. This is compatible and well documented. | `CONCEPTS.md` lines 7-15; Promptfoo load lines 134-180. | +| `defaults.provider` / `defaults.grader` | Eval YAML can select default candidate/grader providers. | No Promptfoo top-level `defaults`; Promptfoo uses `defaultTest.provider`, `tests[].provider(s)`, assertion `provider`, and CLI/runtime options. | Modify. Prefer Promptfoo-compatible slots in eval YAML; keep config-level `defaults.grader` only if intentional AgentV runner policy. | AgentV schema lines 882-891, 963-970; parser lines 1943-1965; Promptfoo test/assert lines 859-889, 709-735. | +| `graders` | Hard-rejected; graders are providers selected by defaults/options/assertion provider. | No separate Promptfoo `graders` list. | Keep rejection. | AgentV validator lines 291-344; `CONCEPTS.md` line 15. | +| `environment` | Inline or `file://` host/Docker testbed recipes at suite/test/provider scope. | No Promptfoo normal-eval equivalent. | Keep as AgentV extension. Export host/filesystem subset only; Docker should fail clearly until a faithful boundary exists. | AgentV schema lines 520-606, 1007; `CONCEPTS.md` lines 37-45; parity docs lines 81-82. | +| `ref://` / `refs` | `default_test` can resolve `ref://name` via `.agentv/config.yaml` refs. | Promptfoo supports file refs and JSON schema dereferencing, not AgentV `ref://`. | Keep as AgentV composition exception, but keep it field-local. | Parser lines 580-669; config loader refs search showed `ReferenceMap` in config-loader. | +| `experiment` | Top-level string accepted and normalized into run grouping. | No Promptfoo config field; Promptfoo has `tags` and `metadata`. | Remove-hard-deprecate. Use `tags.experiment` or CLI `--experiment`. | AgentV schema line 998; parser lines 1465-1467, 1911-1929; `CONCEPTS.md` lines 33-35; docs next experiments lines 207-210. | +| `tags` | Accepts array/list selection form or record with string/number/boolean values; map feeds run metadata. | `tags: Record`. | Modify. Keep map form as canonical; decide whether selection list belongs in eval YAML. Normalize values to strings if retained. | AgentV schema lines 917-920; parser lines 1967-1994; Promptfoo `src/types/index.ts` lines 1162-1165. | +| `metadata` and top-level metadata fields | Validator knows `metadata`; schema also exposes top-level `name`, `category`, `version`, `author`, `license`, `requires`. | Promptfoo has top-level `metadata` and `description`; no top-level `name/category/version/author/license/requires`. | Modify. Move nonessential fields under `metadata` or remove; keep artifact-derived category/name outside Promptfoo-compatible authoring if needed. | AgentV schema lines 929-942; Promptfoo `src/types/index.ts` lines 1249-1250. | +| `default_test` | Inline object or `file://`/`ref://`; inherits vars/provider/providers/prompts/assert/options/threshold/metadata. | `defaultTest`, inline object or `file://`; inherits TestCase fields. | Keep native `snake_case`; map to/from Promptfoo `defaultTest` in import/export/transpile. Semantics mostly align; `ref://` remains AgentV composition sugar. | AgentV schema lines 742-759, 1006; Promptfoo lines 1201-1209. | +| `evaluate_options` | `budget_usd`, `max_concurrency`, `cache`, `delay`, `generate_suggestions`, `repeat`, `timeout_ms`, `max_eval_time_ms`, `filter_range`. | `evaluateOptions` with `cache`, `delay`, `generateSuggestions`, `suggestionsCount`, `maxConcurrency`, `repeat`, `timeoutMs`, `maxEvalTimeMs`, `filterRange`, plus internal/runtime fields. | Keep native `snake_case`; map to/from Promptfoo `evaluateOptions` in import/export/transpile. Add missing `suggestions_count` only if AgentV supports it; avoid top-level duplicates. | AgentV schema lines 761-773; validator lines 1761-1827; Promptfoo lines 259-304. | +| Wire-format casing | Promptfoo config uses camelCase fields such as `defaultTest` and `evaluateOptions`. | AgentV YAML, JSONL, artifacts, and CLI JSON use `snake_case`; internal TypeScript uses `camelCase`. | Keep settled AgentV boundary. Native AgentV eval YAML remains `snake_case`; Promptfoo import/export/transpile must translate to Promptfoo `camelCase` where required. | `.agents/conventions.md`; Promptfoo lines 259-304, 1201-1247. | +| Top-level `timeout_seconds` | Accepted/documented as optional per-case timeout. | Promptfoo uses `evaluateOptions.timeoutMs` for per provider/test timeout and `maxEvalTimeMs` for whole eval. | Remove/modify. Prefer `evaluate_options.timeout_ms`; reserve CLI/operator timeout flags for process control. | AgentV schema line 1002; docs next eval-files lines 180-183; Promptfoo lines 284-295. | +| Top-level `threshold` | Suite quality gate. | Promptfoo has assertion/test/defaultTest thresholds, not top-level suite threshold in `TestSuiteConfigSchema`. | Keep only as explicit AgentV CI gate or move to a future gate policy. Do not describe it as Promptfoo-compatible. | AgentV schema line 1005; parser lines 1897-1929; Promptfoo test/assert threshold lines 720-724, 917-918. | +| Top-level `assert` | Suite assertions appended to each test unless skipped. | Promptfoo puts inherited assertions in `defaultTest.assert`. | Modify/remove. Prefer `default_test.assert`. | AgentV schema lines 1017-1019; parser lines 672-685; Promptfoo lines 1201-1209. | +| `extensions` | Array of `file://path:beforeAll` hook refs plus AgentV `agentv:agent-rules`. | Array of `file://path:function_name` extension refs. | Modify or reclassify. Do not claim Promptfoo extension compatibility until the same string shape is accepted or exported. | AgentV schema lines 427-479, 1013; Promptfoo lines 1246-1247 and generated docs. | +| `providerPromptMap` / `provider_prompt_map` | Rejected with migration to `providers`. | Promptfoo supports `providerPromptMap` in resolved suite and prompt loading. | Keep rejection if product prefers explicit composition; document as deliberate Promptfoo divergence. | AgentV schema lines 952-961; Promptfoo `src/types/index.ts` lines 797-803 and load usage around 1012-1014. | +| Direct `input`, `expected_output`, `provider_output` | Rejected in authored YAML; users put data in vars and assertions. | Promptfoo tests can use `providerOutput`; expected data normally lives in assertion values or vars. | Mostly keep, but document `providerOutput` rejection as AgentV replay/deterministic-provider policy. | AgentV schema lines 943-948, 796-825; Promptfoo test lines 875-879. | +| `tests[].id` | Optional stable test identity used for run artifacts and selection. | Promptfoo `TestCaseSchema` has no explicit `id` field; metadata can carry arbitrary data. | Keep as AgentV artifact identity extension, but export should map or preserve via metadata if Promptfoo cannot validate it. | AgentV schema line 790; Promptfoo test lines 859-923. | +| `tests[].run` / `tests[].execution` | Per-test run overrides and execution controls such as repeat, budget, timeout, skip defaults. | Promptfoo uses `tests[].options.repeat`, `threshold`, and defaultTest/assertion structure. | Modify/remove unless a specific AgentV-only runner policy is documented. | AgentV schema lines 709-740, 803-805; validator lines 1320-1385. | +| Conversation fields (`turns`, `conversation_id`, `depends_on`, `mode`, `aggregation`, `window_size`) | AgentV-specific multi-turn/conversation orchestration. | No direct Promptfoo normal-eval equivalent in inspected schema. | Needs decision. Keep only if this is core to AgentV agent-workflow evals; otherwise push to examples/plugins or map through Promptfoo-compatible vars/prompts. | AgentV schema lines 807-815; Promptfoo test lines 859-923. | +| Assertions | AgentV accepts implemented Promptfoo overlap plus AgentV types (`script`, `field-accuracy`, `token-usage`, `execution-metrics`) and still accepts `llm-grader`. Unsupported Promptfoo trace assertions hard-error. | Promptfoo supports many assertion types, including `llm-rubric`, `agent-rubric`, `javascript`, `python`, `webhook`, trajectory types, and many NLP/security-specific types. | Keep subset with hard errors for unsupported Promptfoo types, but deprecate or justify `llm-grader` as authored YAML. Document AgentV-only assertion types as extensions. | AgentV schema lines 190-356; types lines 179-214; validator lines 2685-2744; Promptfoo lines 595-735. | +| SDK authoring | `defineAssertion()`/`defineScriptGrader()` create AgentV assertion/grader extensions. | Promptfoo custom code assertions use `javascript`, `python`, `webhook`, provider packages, and extension mechanisms. | Keep as AgentV extension, but avoid leaking SDK terminology into Promptfoo-compatible YAML examples unless the example needs AgentV-only scoring. | Parity docs lines 76-77; AgentV registry/discovery references inspected by `rg`. | + +## Fields Already Correctly Removed + +These should stay removed/rejected rather than reintroduced for Promptfoo parity: + +- `target` / `targets` as authored AgentV YAML, despite Promptfoo's `targets` + alias in `UnifiedConfigSchema`. +- `graders` as a separate top-level entity. +- `repeat`, `runs`, `early_exit`, and top-level `budget_usd`. +- `imports` as a wrapper/import table. +- `policy` and top-level `execution`. +- `preprocessors` / `postprocess` in favor of `transform`. +- `providerPromptMap` unless the product deliberately wants Promptfoo's provider + to prompt-subset map. + +## Open Product Decisions + +1. Should `defaults.grader` exist in eval YAML, or only in project/provider + config where AgentV owns provider-catalog policy? +2. Are AgentV conversation orchestration fields core authoring primitives, or + should they be represented through Promptfoo-compatible prompts/tests/vars + plus AgentV run artifacts? +3. Should top-level suite `threshold` become an explicit AgentV gate policy + object later, or remain a minimal scalar CI gate?