Repository navigation
Show a run's per-group pass gates in the dashboard - #703
Merged
Merged
Conversation
A catalog product's named criteria lists (criteria: { shared: [...] })
parse and round-trip in ActiveAgent::Evals::Catalog, but ScenarioProduct
had no column for them. A dashboard import dropped the lists, the export
left them out, the stored digest never matched the records' export, and
the import preview reported the product as changed on every re-import of
an unchanged file.
Store the lists on the product through a guarded upgrade migration, as
add_catalog_scenario_metadata does for scenario extensions, and emit
them from ScenarioProduct#to_document. Both import modes take the
document's lists, as they take a set's criteria, so the merge preview
stays accurate. Which lists a set's run scores on is left for later; a
set still runs on its own criteria.
Fixes activeagents#687
A catalog product's and set's pass blocks (min_pass_rate, threshold, gated) parse in ActiveAgent::Evals::Catalog, but neither ScenarioProduct nor ScenarioSet had a column for them. A dashboard import dropped the bars, the export left them out, the stored catalog's pass_thresholds fell back to the defaults (every group gated at 100%), and the import preview reported the product and its sets as changed on every re-import of an unchanged file. Store each block on its record through a guarded upgrade migration, as add_catalog_product_criteria does for a product's named lists, and emit them from to_document. Both import modes take the document's bars, as they take a set's criteria. Gating a dashboard run on them is left to the run itself; this change only keeps what the document declared. Fixes activeagents#700
A suite can declare pass bars per group (pass: { min_pass_rate: 1.0 }
on one group, gated: false on another), and ActiveAgent::Evals::Report
gates each group by them. The dashboard dropped them: the report import
ignored pass_thresholds, a run rebuilt from stored results had no bars,
and a run started on a catalog set never read the set's pass block. A
run whose critical group slipped to 50% read as passed whenever the
other groups lifted its whole pass rate.
Keep the bars with the run in scores["_pass_thresholds"], beside the
verdict and comparison a run already records there: the import
validates a report's pass_thresholds with PassThresholds and stores
them, and a dashboard run of a catalog set records the product's and
the set's bars as it starts, so every replay and the fan-out completion
see the same ones, and a later catalog change does not regrade the run.
EvaluationRun#to_report passes them back to Report.new, and the gates
are worked out from the stored results into scores["_gates"] rather
than taken from the report.
The run's JSON adds gates and gates_met (both nil without bars). The
run page shows a Pass rate by group table, and the run badge reads
"gate missed", toned as an error, when a gate was not met. A missed
gate stays a verdict on a complete run, not a failed one.
Closes activeagents#678
This was referenced Nov 7, 2026
TonsOfFun
added a commit
to pcbeingused333/activeagent
that referenced
this pull request
Nov 12, 2026
The CHANGELOG moved under this branch: activeagents#699 rewrote Unreleased and activeagents#701 to activeagents#703 added entries. This keeps main's CHANGELOG and puts the Responses streaming entry at the top of Unreleased → Fixed, tagged `(activeagent)` like its neighbours. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCEJHsJpjXpDa9Kuq9i4Yg
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack, part 3 of 4. Merge in this order; each PR targets
main.evals run --min-pass-ratehonours the suite's per-group gates #677)Until #701 and #702 merge, this PR also shows their commits. The change under review here is one commit: 3a8dcbb. Once the PRs below merge, GitHub shows only that commit.
What this does
#676 lets a suite gate each group by its own bar, and
Report#gates/#passed?answer for a report built withpass_thresholds:. As #678 describes, the dashboard dropped the bars: the import ignoredpass_thresholds, a report rebuilt from stored results had none, and a run started on a catalog set never read the set'spass. A run whose critical group slipped to 50% read as passed whenever the other groups lifted its whole pass rate.Store the bars with the run. The run keeps them in
scores["_pass_thresholds"], beside the_verdict,_comparisonand_selectiona run already records there, so no migration. The import validates a report'spass_thresholdswithPassThresholdsitself (unknown keys, bars outside 0..1, a non-booleangated, more groups than results are refused with the reason) and stores the normalized bars.EvaluationRun#to_reportpasses them back toReport.new, so the rebuilt report'sto_h["gates"]equals what was published, and the export carries them.The dashboard's own runs. A run of an evaluation a catalog set materialized reads the set's and the product's
pass(stored since #702), looked up by the evaluation's catalog reference under the owner's catalogs, and records them as the run starts. The sequential runner, every fan-out scorer and the fan-out completion read that one record, so a catalog edit mid-run or later does not regrade it. A host adapter's run is gated by the same record when its report carries no bars.One verdict, worked out once. At completion the gates are computed from the stored results into
scores["_gates"](Report#gates), never taken from a report'sgates. The run's JSON addsgatesandgates_met(bothnullwithout bars). A missed gate leaves the runcomplete: it is a verdict, not an execution failure.The run page. A Pass rate by group table under the run header (group, model, passed of total, required rate, and
met,not metorreported only), and the run badge, in the list and the header, readsgate missedtoned as an error when any gate was not met, its tooltip naming each one. The browser only displays what the server computed. The bundle is rebuilt.Two behavior changes to confirm, @TonsOfFun:
thresholdnow scores a dashboard run's answers; the runner passed a fixed 0.7 before. Only catalogs that declare one change. Keep it, or keep 0.7 and gate only?Out of scope, per the issue: editing the bars in the UI, and the CLI (#677, in #704).
Tests:
actionagent/test/evaluation_pass_gates_test.rb(new), failing before. Dashboard runs: a set at 100% fails its gate while the whole run clears the product's 60%, withstatus: complete; an ungated set is reported only; a catalogthresholdscores the answers, sequential and fanned out; a finished run keeps its gates after the catalog changes, and a new run takes the new bar; a fan-out run keeps the bar recorded at its start through a mid-run import; a host adapter's run is gated by the recorded bars; an evaluation pointing at another owner's set, or at a set whose key changed, is not gated by it; a recorded value that is not a bar reads as none; the run JSON carriesgatesandgates_met, and a run that reached no gate has not passed. Imports: a published report's gates come back from the stored run and its export, and an identical retry stores nothing new; the stored verdict is worked out from the results even when the report'sgatesclaim otherwise; malformed bars and too many groups are refused with the reason.actionagent/frontend/test/runGatesViews.test.mjs(new): the helpers read the server's verdict and drop malformed rows; gate sentences matchGate#to_s; the table states each status in words; the badge saysgate missedwith an error tone whatever the pass rate, and a run without gates keeps its badge.Each part reverted in turn fails a test (20 Ruby, 6 JavaScript).
Type of change
Scope check
Related issue
Closes #678
Quality check
bin/rubocopon the changed Ruby files)CliContractTest(rack-sessionundefined method 'id' for Hash), the same on an untouchedmainhere. Frontend:npm test790 pass,npm run buildmatches the committed bundledocs/framework/dashboard.mdScenario evaluations and catalogs,docs/framework/evaluations.md, a CHANGELOG Added entry)npm run buildoutput)Ownership
API changes
EvaluationRun#pass_thresholds,#gates,#gates_met; run JSONgates,gates_met)