Skip to content

Keep a catalog's pass bars through dashboard import - #702

Merged
TonsOfFun merged 2 commits into
activeagents:mainfrom
yorzi:fix/catalog-pass-bars
Nov 12, 2026
Merged

TonsOfFun merged 2 commits into
activeagents:mainfrom
yorzi:fix/catalog-pass-bars

Conversation

@yorzi

@yorzi yorzi commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack, part 2 of 4. Merge in this order; each PR targets main.

  1. Keep a catalog product's named criteria lists #701 Keep a catalog product's named criteria lists (Evals: the dashboard keeps a product's named criteria lists across import and export #687)
  2. Keep a catalog's pass bars through dashboard import #702 Keep a catalog's pass bars through dashboard import (Evals: the dashboard drops a catalog product's and set's pass bars #700) (this PR)
  3. Show a run's per-group pass gates in the dashboard #703 Show a run's per-group pass gates in the dashboard (Evals: show a run's per-group pass gates in the dashboard #678)
  4. Fail evals run on a missed per-group pass gate #704 Fail evals run on a missed per-group pass gate (Evals: evals run --min-pass-rate honours the suite's per-group gates #677)

Until #701 merges, this PR also shows that commit. The change under review here is one commit: 43c66da. Once the PRs below merge, GitHub shows only that commit.

What this does

#676 lets a catalog product and set carry a pass block (min_pass_rate, threshold, gated), which ActiveAgent::Evals::Catalog parses and Catalog#pass_thresholds reads. The dashboard kept neither: ScenarioProduct and ScenarioSet had no column, the import never wrote entry.pass, and to_document never emitted it. After an import the stored catalog's pass_thresholds read as the defaults (every group gated at 100%), the export left the bars out, the stored digest never matched the records' export, and the preview reported the product and its sets as update on every re-import of an unchanged file.

Each record now stores its block in a pass JSON column and emits it from to_document. The migration is add_catalog_pass_bars (035; 033 is #674's and 034 #701's), built like 034: guarded per table, safe under a custom table_name_prefix, jsonb on PostgreSQL and json elsewhere with no JSON default. Until it runs, imports keep working and leave the bars out. Both import modes take the document's bars, as they take a set's criteria.

This branch is stacked on #701, since both add a column to scenario_products and touch the same import lines and schema.rb; it carries that commit until it merges. Gating a dashboard run on the stored bars is #678 and comes next.

Tests:

  • actionagent/test/catalog_pass_bars_test.rb (new), failing before: the stored catalog's pass_thresholds equal the parsed document's for the product and for one set; the bars survive reload and export, including a set-narrowed export; an unchanged re-import previews no change; a changed or removed bar previews as update and is replaced; merge mode takes the document's bars, merges a set export back unchanged, and clears them when the document has none (what its preview says); a stored value that is not a mapping reads as none.
  • Guards, passing before and after: a catalog without bars imports, exports and previews as before; before the migration runs, an import and export work without the bars.
  • actionagent/test/scenario_catalogs_api_test.rb: the same round trip through the import, export and preview endpoints.
  • actionagent/test/install_generator_test.rb: emitted after catalog creation and again only when missing; keeps old rows under a custom prefix, reverses idempotently, skips a missing table (one of the two missing does not stop the other); each adapter's JSON type.

Each part reverted in turn fails a test: both import writes, both column guards, both exports, both read normalizations, the export's presence check, and merge-mode clearing on the product and the set.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the contributing guide
  • This fits activeagent's stated focus
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my use case

Related issue

Fixes #700

Quality check

  • The project's hooks and linters pass (bin/rubocop on the changed Ruby files)
  • The new tests fail before this change and pass after it
  • The full test suite passes: Rails 7.2 with the database built as CI builds it (applies 035): 4580 runs, 0 errors, 1 failure in LocalSandboxBackendTest (a boot-interrupt timing test, run while a second suite loaded the machine; green 3 of 3 alone). Rails 8.1: 4576 runs, 0 failures, 10 errors, all CliContractTest (rack-session undefined method 'id' for Hash), which fail the same way on an untouched main here
  • I updated documentation where needed (docs/framework/dashboard.md scenario catalogs, docs/framework/evaluations.md, a CHANGELOG Fixed entry)
  • I didn't edit generated files by hand (schema.rb matches what a fresh db:migrate writes)

Ownership

  • I understand every line of this change and can answer questions about it

API changes

  • Breaking change
  • New public methods or classes (ScenarioProduct#pass, ScenarioSet#pass)
  • Changed method signatures
  • No API changes

yorzi added 2 commits November 7, 2026 09:40
A catalog product's named criteria lists (criteria: { shared: [...] })
parse and round-trip in ActiveAgent::Evals::Catalog, but ScenarioProduct
had no column for them. A dashboard import dropped the lists, the export
left them out, the stored digest never matched the records' export, and
the import preview reported the product as changed on every re-import of
an unchanged file.

Store the lists on the product through a guarded upgrade migration, as
add_catalog_scenario_metadata does for scenario extensions, and emit
them from ScenarioProduct#to_document. Both import modes take the
document's lists, as they take a set's criteria, so the merge preview
stays accurate. Which lists a set's run scores on is left for later; a
set still runs on its own criteria.

Fixes activeagents#687
A catalog product's and set's pass blocks (min_pass_rate, threshold,
gated) parse in ActiveAgent::Evals::Catalog, but neither ScenarioProduct
nor ScenarioSet had a column for them. A dashboard import dropped the
bars, the export left them out, the stored catalog's pass_thresholds
fell back to the defaults (every group gated at 100%), and the import
preview reported the product and its sets as changed on every re-import
of an unchanged file.

Store each block on its record through a guarded upgrade migration, as
add_catalog_product_criteria does for a product's named lists, and emit
them from to_document. Both import modes take the document's bars, as
they take a set's criteria. Gating a dashboard run on them is left to
the run itself; this change only keeps what the document declared.

Fixes activeagents#700
@TonsOfFun
TonsOfFun merged commit adc71f5 into activeagents:main Nov 12, 2026
18 checks passed
@TonsOfFun TonsOfFun mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals: the dashboard drops a catalog product's and set's pass bars

2 participants