fix(eval): reject top-level experiment authoring#1728
Merged
Conversation
christso
force-pushed
the
av-ibec-remove-experiment
branch
from
cd19766 to
24e51d4
Compare
Deploying agentv with
|
| Latest commit: |
24e51d4
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://90198b48.agentv.pages.dev |
| Branch Preview URL: | https://av-ibec-remove-experiment.agentv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Authored eval YAML now hard-rejects top-level
experimentinstead of treating it as a run label. The supported authoring path istags.experiment, while CLI--experimentand runtime/result artifact experiment metadata stay intact for run-time grouping and Dashboard comparison.The schema, validator, YAML parser, config loader, and SDK authoring helper now produce migration guidance that points users to
tags.experimentor CLI--experiment. Public docs, AI-facing eval-writing guidance, migration notes, and the generated eval JSON schema were updated to stop teaching top-levelexperiment.Validation
bun --cwd packages/core test ./test/evaluation/validation/eval-file-schema.test.ts ./test/evaluation/validation/eval-validator.test.ts ./test/evaluation/eval-inline-experiment.test.ts ./test/evaluation/yaml-parser-tags-map.test.tsbun --cwd packages/sdk test ./test/eval-authoring.test.tsbun --cwd apps/cli test ./test/commands/eval/tag-filtering.test.ts ./test/commands/eval/pipeline/input.test.tsbun run validate:eval-schemabun run typecheckbun run lintgit diff --checkAfter fixing the CI lint formatting failure, I also reran:
bun --cwd packages/core test ./test/evaluation/validation/eval-file-schema.test.tsExtra check:
bun run validate:examplesstill fails onexamples/showcase/multi-model-benchmark/evals/benchmark.eval.yamlbecause it uses an object-shapedevaluate_options.repeat; that same file is identical onorigin/main, so I left it out of this scoped PR.Live provider dogfood was not run because this change stays at authored schema/parser/docs validation boundaries. It does not change eval execution, provider runtime, graders, or artifact writing; CLI
--experimentcoverage is exercised by the focused CLI tests above.Post-Deploy Monitoring & Validation
No additional operational monitoring required. This is a local validation and authoring-contract change with no service runtime path; CI should verify schema generation, typecheck, lint, and tests on the PR.
Related
Related: Bead
av-ibec