Skip to content

feat(core): support promptfoo provider declaration forms#1729

Merged
christso merged 3 commits into
mainfrom
av-sdit-provider-schema
Jul 7, 2026
Merged

feat(core): support promptfoo provider declaration forms#1729
christso merged 3 commits into
mainfrom
av-sdit-provider-schema

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

AgentV provider declarations now accept the Promptfoo-compatible forms that were missing from the parser: provider spec strings, provider maps, and providers[].inputs. Object-form id remains the backend/spec and label remains the stable AgentV selection and result identity.

The Promptfoo export helper now preserves compatible provider maps and inputs, and it rejects AgentV-only provider-local environment, provider hooks, and non-host runtime settings with explicit diagnostics instead of silently dropping testbed behavior. runtime: host continues to lower as a no-op because Promptfoo runs host providers by default.

Docs and the eval-writer schema now describe the supported provider declaration layer and call out runtime, provider-local environment, and provider hooks as AgentV extensions.

Validation

  • bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/loaders/config-loader.test.ts packages/core/test/evaluation/providers/targets-file.test.ts
  • bun run validate:promptfoo-export
  • bun run validate:eval-schema
  • bun --filter @agentv/core typecheck
  • git diff --check

Live provider dogfood was not run because this changes parsing, schema, docs, and Promptfoo export diagnostics only; provider invocation/runtime behavior is unchanged.

Risk

Low. The new parser path is additive for Promptfoo-compatible provider shapes, and existing provider object/string declarations remain covered by focused tests.

Related: Bead av-sdit


Compound Engineering
GPT--5_Codex

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 7, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 9b5b81b
Status: ✅  Deploy successful!
Preview URL: https://427e693b.agentv.pages.dev
Branch Preview URL: https://av-sdit-provider-schema.agentv.pages.dev

View logs

@christso christso force-pushed the av-sdit-provider-schema branch from 194ddb2 to 59a76ff Compare July 7, 2026 14:05
@christso christso merged commit 0d7b3fe into main Jul 7, 2026
9 checks passed
@christso christso deleted the av-sdit-provider-schema branch July 7, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant