Skip to content

Fail evals run on a missed per-group pass gate - #704

Merged
TonsOfFun merged 3 commits into
activeagents:mainfrom
yorzi:fix/cli-group-gates
Nov 12, 2026
Merged

TonsOfFun merged 3 commits into
activeagents:mainfrom
yorzi:fix/cli-group-gates

Conversation

@yorzi

@yorzi yorzi commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack, part 4 of 4. Merge in this order; each PR targets main.

  1. Keep a catalog product's named criteria lists #701 Keep a catalog product's named criteria lists (Evals: the dashboard keeps a product's named criteria lists across import and export #687)
  2. Keep a catalog's pass bars through dashboard import #702 Keep a catalog's pass bars through dashboard import (Evals: the dashboard drops a catalog product's and set's pass bars #700)
  3. Show a run's per-group pass gates in the dashboard #703 Show a run's per-group pass gates in the dashboard (Evals: show a run's per-group pass gates in the dashboard #678)
  4. Fail evals run on a missed per-group pass gate #704 Fail evals run on a missed per-group pass gate (Evals: evals run --min-pass-rate honours the suite's per-group gates #677) (this PR)

Until #701, #702 and #703 merge, this PR also shows their commits. The change under review here is one commit: abfa003. Once the PRs below merge, GitHub shows only that commit.

What this does

As #677 describes, evals run --min-pass-rate gated a dashboard run on its whole pass rate alone, so a run whose critical group slipped to 50% passed CI whenever the other groups lifted the whole-run rate over the bar. With #703 the dashboard works each group's gate out for a run and serves gates and gates_met on it; this reads them.

A finished run the dashboard says missed a gate now fails with EXIT_GATE (4), with --wait alone or with --min-pass-rate, which stays the whole-run bar, so a run must clear both. The verdict is the dashboard's gates_met; the CLI does not regrade rows, and a run that reached no gate has not passed, as Report#passed? says. Each missed gate prints as one line in Gate#to_s's words:

Run 41 complete: 9/10 (90.0%), threshold 80.0%.
Gates not met:
  Critical · gpt-5-mini: 1/2 (50%) passed, below 100%
FAIL

The gate text goes through the same filter as table cells, so a group name from the server cannot move the cursor or retitle the terminal. --json adds gates beside pass_rate and verdict and keeps the rest of the payload. A run without gates is gated and printed exactly as before, --json included (no gates key), and without --wait or --min-pass-rate the command still returns once the run starts, before any gate exists. docs/framework/cli.md "Gating CI on a pass rate" shows the example above.

Tests:

  • test/cli/cli_test.rb, failing before: --wait fails an unmet group gate and names it, leaving an ungated group out; --min-pass-rate needs the rate and every gate, and fails below the rate with every gate met; every gated group met passes whatever an ungated group scored; empty gates with gates_met: false fail; --json adds the gates and keeps the payload; control characters from the server are replaced.
  • Guards, passing before and after: a run without gates reads as before in text and JSON; a failed run fails whatever its gates; without --wait gates are not read; a gate row that is not an object is skipped.
  • actionagent/test/cli_contract_test.rb: against the dummy app, a catalog set held to 100% runs through the real facade, evals run --wait exits 4 with the gate line, and evals runs --json carries the same gates_met.

Each part reverted in turn fails a test.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the contributing guide
  • This fits activeagent's stated focus
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my use case

Related issue

Fixes #677

Quality check

  • The project's hooks and linters pass (bin/rubocop on the changed Ruby files)
  • The new tests fail before this change and pass after it
  • The full test suite passes: Rails 7.2, database built as CI builds it: 4609 runs, 0 failures, 0 errors, the new contract test included. Rails 8.1: 4605 runs, 0 failures, 11 errors, all CliContractTest (rack-session undefined method 'id' for Hash), the same on an untouched main here; the new contract test is the eleventh
  • I updated documentation where needed (docs/framework/cli.md, docs/framework/evaluations.md, a CHANGELOG Fixed entry)
  • I didn't edit generated files by hand

Ownership

  • I understand every line of this change and can answer questions about it

API changes

  • Breaking change
  • New public methods or classes
  • Changed method signatures
  • No API changes (the CLI's --json output gains a gates key for runs that have gates)

evals run --min-pass-rate gated a dashboard run on its whole pass rate
alone. A suite can hold one group to its own bar (pass: { min_pass_rate:
1.0 }) and the dashboard now works each group's gate out for the run,
but the CLI never read them: a run whose critical group slipped to 50%
passed CI whenever the other groups lifted the whole-run rate over the
threshold.

gate() now reads the finished run's gates and gates_met. A run the
dashboard says missed a gate fails with EXIT_GATE, with --wait alone or
with --min-pass-rate, which stays the whole-run bar, so the run must
clear both. Each missed gate prints as one line in Gate#to_s's words,
its text passed through the table cell filter, and --json adds "gates"
beside "pass_rate" and "verdict". A run without gates is gated and
printed exactly as before, and without --wait the command still returns
as soon as the run starts.

Fixes activeagents#677
@yorzi

yorzi commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

The one red check, test (3.2, gemfiles/rails7.gemfile), is RunCancelConcurrencyTest#test_a_cancel_racing_the_finishing_worker_leaves_one_consistent_outcome erroring with SQLite3::BusyException: database is locked (4609 runs, 0 failures, 1 error). This PR only changes the CLI's gate and its tests; the same test passes in the same job on #701, #702 and #703, and #699 is about this race test on Rails 7.2. I can't rerun the job from a fork; a rerun should clear it.

TonsOfFun and others added 2 commits November 12, 2026 10:30
activeagents#706 landed its CHANGELOG entry at the top of Unreleased → Fixed after this
branch was cut. This keeps main's CHANGELOG and puts the missed-gate entry
above it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PCEJHsJpjXpDa9Kuq9i4Yg
@TonsOfFun
TonsOfFun merged commit 6501f42 into activeagents:main Nov 12, 2026
13 of 18 checks passed
@TonsOfFun TonsOfFun mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals: evals run --min-pass-rate honours the suite's per-group gates

2 participants