Repository navigation
Fail evals run on a missed per-group pass gate - #704
Merged
Merged
Conversation
evals run --min-pass-rate gated a dashboard run on its whole pass rate
alone. A suite can hold one group to its own bar (pass: { min_pass_rate:
1.0 }) and the dashboard now works each group's gate out for the run,
but the CLI never read them: a run whose critical group slipped to 50%
passed CI whenever the other groups lifted the whole-run rate over the
threshold.
gate() now reads the finished run's gates and gates_met. A run the
dashboard says missed a gate fails with EXIT_GATE, with --wait alone or
with --min-pass-rate, which stays the whole-run bar, so the run must
clear both. Each missed gate prints as one line in Gate#to_s's words,
its text passed through the table cell filter, and --json adds "gates"
beside "pass_rate" and "verdict". A run without gates is gated and
printed exactly as before, and without --wait the command still returns
as soon as the run starts.
Fixes activeagents#677
This was referenced Nov 7, 2026
Contributor
Author
|
The one red check, |
activeagents#706 landed its CHANGELOG entry at the top of Unreleased → Fixed after this branch was cut. This keeps main's CHANGELOG and puts the missed-gate entry above it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCEJHsJpjXpDa9Kuq9i4Yg
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack, part 4 of 4. Merge in this order; each PR targets
main.evals run --min-pass-ratehonours the suite's per-group gates #677) (this PR)Until #701, #702 and #703 merge, this PR also shows their commits. The change under review here is one commit: abfa003. Once the PRs below merge, GitHub shows only that commit.
What this does
As #677 describes,
evals run --min-pass-rategated a dashboard run on its whole pass rate alone, so a run whose critical group slipped to 50% passed CI whenever the other groups lifted the whole-run rate over the bar. With #703 the dashboard works each group's gate out for a run and servesgatesandgates_meton it; this reads them.A finished run the dashboard says missed a gate now fails with
EXIT_GATE(4), with--waitalone or with--min-pass-rate, which stays the whole-run bar, so a run must clear both. The verdict is the dashboard'sgates_met; the CLI does not regrade rows, and a run that reached no gate has not passed, asReport#passed?says. Each missed gate prints as one line inGate#to_s's words:The gate text goes through the same filter as table cells, so a group name from the server cannot move the cursor or retitle the terminal.
--jsonaddsgatesbesidepass_rateandverdictand keeps the rest of the payload. A run without gates is gated and printed exactly as before,--jsonincluded (nogateskey), and without--waitor--min-pass-ratethe command still returns once the run starts, before any gate exists.docs/framework/cli.md"Gating CI on a pass rate" shows the example above.Tests:
test/cli/cli_test.rb, failing before:--waitfails an unmet group gate and names it, leaving an ungated group out;--min-pass-rateneeds the rate and every gate, and fails below the rate with every gate met; every gated group met passes whatever an ungated group scored; empty gates withgates_met: falsefail;--jsonadds the gates and keeps the payload; control characters from the server are replaced.--waitgates are not read; a gate row that is not an object is skipped.actionagent/test/cli_contract_test.rb: against the dummy app, a catalog set held to 100% runs through the real facade,evals run --waitexits 4 with the gate line, andevals runs --jsoncarries the samegates_met.Each part reverted in turn fails a test.
Type of change
Scope check
Related issue
Fixes #677
Quality check
bin/rubocopon the changed Ruby files)CliContractTest(rack-sessionundefined method 'id' for Hash), the same on an untouchedmainhere; the new contract test is the eleventhdocs/framework/cli.md,docs/framework/evaluations.md, a CHANGELOG Fixed entry)Ownership
API changes
--jsonoutput gains agateskey for runs that have gates)