Skip to content

Show a run's per-group pass gates in the dashboard - #703

Merged
TonsOfFun merged 3 commits into
activeagents:mainfrom
yorzi:fix/evaluation-pass-gates
Nov 12, 2026
Merged

TonsOfFun merged 3 commits into
activeagents:mainfrom
yorzi:fix/evaluation-pass-gates

Conversation

@yorzi

@yorzi yorzi commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack, part 3 of 4. Merge in this order; each PR targets main.

  1. Keep a catalog product's named criteria lists #701 Keep a catalog product's named criteria lists (Evals: the dashboard keeps a product's named criteria lists across import and export #687)
  2. Keep a catalog's pass bars through dashboard import #702 Keep a catalog's pass bars through dashboard import (Evals: the dashboard drops a catalog product's and set's pass bars #700)
  3. Show a run's per-group pass gates in the dashboard #703 Show a run's per-group pass gates in the dashboard (Evals: show a run's per-group pass gates in the dashboard #678) (this PR)
  4. Fail evals run on a missed per-group pass gate #704 Fail evals run on a missed per-group pass gate (Evals: evals run --min-pass-rate honours the suite's per-group gates #677)

Until #701 and #702 merge, this PR also shows their commits. The change under review here is one commit: 3a8dcbb. Once the PRs below merge, GitHub shows only that commit.

What this does

#676 lets a suite gate each group by its own bar, and Report#gates / #passed? answer for a report built with pass_thresholds:. As #678 describes, the dashboard dropped the bars: the import ignored pass_thresholds, a report rebuilt from stored results had none, and a run started on a catalog set never read the set's pass. A run whose critical group slipped to 50% read as passed whenever the other groups lifted its whole pass rate.

Store the bars with the run. The run keeps them in scores["_pass_thresholds"], beside the _verdict, _comparison and _selection a run already records there, so no migration. The import validates a report's pass_thresholds with PassThresholds itself (unknown keys, bars outside 0..1, a non-boolean gated, more groups than results are refused with the reason) and stores the normalized bars. EvaluationRun#to_report passes them back to Report.new, so the rebuilt report's to_h["gates"] equals what was published, and the export carries them.

The dashboard's own runs. A run of an evaluation a catalog set materialized reads the set's and the product's pass (stored since #702), looked up by the evaluation's catalog reference under the owner's catalogs, and records them as the run starts. The sequential runner, every fan-out scorer and the fan-out completion read that one record, so a catalog edit mid-run or later does not regrade it. A host adapter's run is gated by the same record when its report carries no bars.

One verdict, worked out once. At completion the gates are computed from the stored results into scores["_gates"] (Report#gates), never taken from a report's gates. The run's JSON adds gates and gates_met (both null without bars). A missed gate leaves the run complete: it is a verdict, not an execution failure.

The run page. A Pass rate by group table under the run header (group, model, passed of total, required rate, and met, not met or reported only), and the run badge, in the list and the header, reads gate missed toned as an error when any gate was not met, its tooltip naming each one. The browser only displays what the server computed. The bundle is rebuilt.

Two behavior changes to confirm, @TonsOfFun:

  1. A catalog's threshold now scores a dashboard run's answers; the runner passed a fixed 0.7 before. Only catalogs that declare one change. Keep it, or keep 0.7 and gate only?
  2. A rebuilt imported report now counts criterion passes at the report's own threshold rather than 0.7, so its numbers match the publisher's.

Out of scope, per the issue: editing the bars in the UI, and the CLI (#677, in #704).

Tests:

  • actionagent/test/evaluation_pass_gates_test.rb (new), failing before. Dashboard runs: a set at 100% fails its gate while the whole run clears the product's 60%, with status: complete; an ungated set is reported only; a catalog threshold scores the answers, sequential and fanned out; a finished run keeps its gates after the catalog changes, and a new run takes the new bar; a fan-out run keeps the bar recorded at its start through a mid-run import; a host adapter's run is gated by the recorded bars; an evaluation pointing at another owner's set, or at a set whose key changed, is not gated by it; a recorded value that is not a bar reads as none; the run JSON carries gates and gates_met, and a run that reached no gate has not passed. Imports: a published report's gates come back from the stored run and its export, and an identical retry stores nothing new; the stored verdict is worked out from the results even when the report's gates claim otherwise; malformed bars and too many groups are refused with the reason.
  • Guards: a run without bars, and a report without them, record none and serialize as before.
  • actionagent/frontend/test/runGatesViews.test.mjs (new): the helpers read the server's verdict and drop malformed rows; gate sentences match Gate#to_s; the table states each status in words; the badge says gate missed with an error tone whatever the pass rate, and a run without gates keeps its badge.

Each part reverted in turn fails a test (20 Ruby, 6 JavaScript).

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the contributing guide
  • This fits activeagent's stated focus
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my use case

Related issue

Closes #678

Quality check

  • The project's hooks and linters pass (bin/rubocop on the changed Ruby files)
  • The new tests fail before this change and pass after it
  • The full test suite passes: Rails 7.2, database built as CI builds it: 4597 runs, 0 failures, 0 errors. Rails 8.1: 4593 runs, 0 failures, 10 errors, all CliContractTest (rack-session undefined method 'id' for Hash), the same on an untouched main here. Frontend: npm test 790 pass, npm run build matches the committed bundle
  • I updated documentation where needed (docs/framework/dashboard.md Scenario evaluations and catalogs, docs/framework/evaluations.md, a CHANGELOG Added entry)
  • I didn't edit generated files by hand (the bundle is npm run build output)

Ownership

  • I understand every line of this change and can answer questions about it

API changes

  • Breaking change
  • New public methods or classes (EvaluationRun#pass_thresholds, #gates, #gates_met; run JSON gates, gates_met)
  • Changed method signatures
  • No API changes

yorzi added 3 commits November 7, 2026 09:40
A catalog product's named criteria lists (criteria: { shared: [...] })
parse and round-trip in ActiveAgent::Evals::Catalog, but ScenarioProduct
had no column for them. A dashboard import dropped the lists, the export
left them out, the stored digest never matched the records' export, and
the import preview reported the product as changed on every re-import of
an unchanged file.

Store the lists on the product through a guarded upgrade migration, as
add_catalog_scenario_metadata does for scenario extensions, and emit
them from ScenarioProduct#to_document. Both import modes take the
document's lists, as they take a set's criteria, so the merge preview
stays accurate. Which lists a set's run scores on is left for later; a
set still runs on its own criteria.

Fixes activeagents#687
A catalog product's and set's pass blocks (min_pass_rate, threshold,
gated) parse in ActiveAgent::Evals::Catalog, but neither ScenarioProduct
nor ScenarioSet had a column for them. A dashboard import dropped the
bars, the export left them out, the stored catalog's pass_thresholds
fell back to the defaults (every group gated at 100%), and the import
preview reported the product and its sets as changed on every re-import
of an unchanged file.

Store each block on its record through a guarded upgrade migration, as
add_catalog_product_criteria does for a product's named lists, and emit
them from to_document. Both import modes take the document's bars, as
they take a set's criteria. Gating a dashboard run on them is left to
the run itself; this change only keeps what the document declared.

Fixes activeagents#700
A suite can declare pass bars per group (pass: { min_pass_rate: 1.0 }
on one group, gated: false on another), and ActiveAgent::Evals::Report
gates each group by them. The dashboard dropped them: the report import
ignored pass_thresholds, a run rebuilt from stored results had no bars,
and a run started on a catalog set never read the set's pass block. A
run whose critical group slipped to 50% read as passed whenever the
other groups lifted its whole pass rate.

Keep the bars with the run in scores["_pass_thresholds"], beside the
verdict and comparison a run already records there: the import
validates a report's pass_thresholds with PassThresholds and stores
them, and a dashboard run of a catalog set records the product's and
the set's bars as it starts, so every replay and the fan-out completion
see the same ones, and a later catalog change does not regrade the run.
EvaluationRun#to_report passes them back to Report.new, and the gates
are worked out from the stored results into scores["_gates"] rather
than taken from the report.

The run's JSON adds gates and gates_met (both nil without bars). The
run page shows a Pass rate by group table, and the run badge reads
"gate missed", toned as an error, when a gate was not met. A missed
gate stays a verdict on a complete run, not a failed one.

Closes activeagents#678
@TonsOfFun
TonsOfFun merged commit 3c5ddbd into activeagents:main Nov 12, 2026
18 checks passed
TonsOfFun added a commit to pcbeingused333/activeagent that referenced this pull request Nov 12, 2026
The CHANGELOG moved under this branch: activeagents#699 rewrote Unreleased and activeagents#701 to
activeagents#703 added entries. This keeps main's CHANGELOG and puts the Responses
streaming entry at the top of Unreleased → Fixed, tagged `(activeagent)`
like its neighbours.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PCEJHsJpjXpDa9Kuq9i4Yg
@TonsOfFun TonsOfFun mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals: show a run's per-group pass gates in the dashboard

2 participants