Skip to content

Keep a catalog product's named criteria lists - #701

Merged
TonsOfFun merged 1 commit into
activeagents:mainfrom
yorzi:fix/catalog-product-criteria
Nov 12, 2026
Merged

TonsOfFun merged 1 commit into
activeagents:mainfrom
yorzi:fix/catalog-product-criteria

Conversation

@yorzi

@yorzi yorzi commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack, part 1 of 4. Merge in this order; each PR targets main.

  1. Keep a catalog product's named criteria lists #701 Keep a catalog product's named criteria lists (Evals: the dashboard keeps a product's named criteria lists across import and export #687) (this PR)
  2. Keep a catalog's pass bars through dashboard import #702 Keep a catalog's pass bars through dashboard import (Evals: the dashboard drops a catalog product's and set's pass bars #700)
  3. Show a run's per-group pass gates in the dashboard #703 Show a run's per-group pass gates in the dashboard (Evals: show a run's per-group pass gates in the dashboard #678)
  4. Fail evals run on a missed per-group pass gate #704 Fail evals run on a missed per-group pass gate (Evals: evals run --min-pass-rate honours the suite's per-group gates #677)

This PR stands alone on main; the three after it build on it.

What this does

#686 gave a catalog product named criteria lists (criteria: { shared: [...], strict: [...] }), which ActiveAgent::Evals::Catalog parses, layers and writes back. The dashboard's ScenarioProduct had no column for them, so, as #687 describes, an import dropped the lists, ScenarioProduct#to_document left them out of every export, the stored digest never matched the records' export, and the import preview reported the product as update on every re-import of an unchanged file. docs/framework/evaluations.md described the gap.

The product now stores the mapping in a criteria JSON column and emits it from to_document, so import, export and preview agree. The migration follows add_catalog_scenario_metadata (031): a numbered template (034, since #674 holds 033), guarded by table and column so it is safe on fresh and existing installs and under a custom table_name_prefix, jsonb on PostgreSQL and json elsewhere with no JSON default. Until it runs, imports keep working and leave the lists out, as scenario metadata does.

Both import modes take the document's lists, the same way they take a set's criteria, a product's metadata and its name, so the merge preview still says what the import will do and a narrowed export (one set, one scenario) merges back unchanged. Layering the lists by name and criterion key in merge mode, as layered catalog documents do, would be a small follow-up if you prefer it; the merge preview would then need to compare layered results instead of documents.

Not covered, per the issue: which product lists a dashboard run scores on. A set still materializes with its own criteria only, so no evaluation result changes. The product summary JSON is unchanged too.

Tests:

  • actionagent/test/catalog_product_criteria_test.rb (new), failing before: two named lists survive reload and equal the parsed document and digest, including product- and set-narrowed exports; the stored lists reach suite_for(...).criteria(:shared, :strict) next to the set's own criteria; an unchanged re-import previews no change and adds no records; a changed or removed list previews as update and is replaced; merge mode takes the document's lists, merges a set export back with no change, and clears the lists when the document names none (what its preview reports); a bare product list is stored as default; a stored value that is not a mapping of lists reads as none.
  • Guards, passing before and after: a catalog without product criteria imports, exports and previews as before; before the migration runs, an import and export work without the lists.
  • actionagent/test/scenario_catalogs_api_test.rb, failing before: the same round trip through the import, export and preview endpoints.
  • actionagent/test/install_generator_test.rb: the migration is emitted after catalog creation and again only when missing, keeps old rows under a custom prefix, reverses idempotently, skips a missing table, and picks each adapter's JSON type.

Each part reverted in turn fails a test: the export, the import write, its column guard, the merge policy, and the three read normalizations.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the contributing guide
  • This fits activeagent's stated focus
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my use case

Related issue

Fixes #687

Quality check

  • The project's hooks and linters pass (bin/rubocop on the changed Ruby files; the repo has no commit hooks installed)
  • The new tests fail before this change and pass after it
  • The full test suite passes: bin/test with the CI placeholder keys. Rails 7.2, database built as the CI job builds it (db:create db:migrate, which applies 034): 4568 runs, 0 failures, 0 errors. Rails 8.1: 4564 runs, 0 failures, 10 errors, all CliContractTest (rack-session undefined method 'id' for Hash), which fail the same way on an untouched main in this environment
  • I updated documentation where needed (docs/framework/dashboard.md scenario catalogs, the gap sentence in docs/framework/evaluations.md, a CHANGELOG Fixed entry)
  • I didn't edit generated files by hand (test/dummy/db/schema.rb carries the column and version a fresh db:migrate writes)

Ownership

  • I understand every line of this change and can answer questions about it

API changes

  • Breaking change
  • New public methods or classes (ScenarioProduct#criteria)
  • Changed method signatures
  • No API changes

A catalog product's named criteria lists (criteria: { shared: [...] })
parse and round-trip in ActiveAgent::Evals::Catalog, but ScenarioProduct
had no column for them. A dashboard import dropped the lists, the export
left them out, the stored digest never matched the records' export, and
the import preview reported the product as changed on every re-import of
an unchanged file.

Store the lists on the product through a guarded upgrade migration, as
add_catalog_scenario_metadata does for scenario extensions, and emit
them from ScenarioProduct#to_document. Both import modes take the
document's lists, as they take a set's criteria, so the merge preview
stays accurate. Which lists a set's run scores on is left for later; a
set still runs on its own criteria.

Fixes activeagents#687
@TonsOfFun
TonsOfFun merged commit c0df4e8 into activeagents:main Nov 12, 2026
18 checks passed
TonsOfFun added a commit to pcbeingused333/activeagent that referenced this pull request Nov 12, 2026
The CHANGELOG moved under this branch: activeagents#699 rewrote Unreleased and activeagents#701 to
activeagents#703 added entries. This keeps main's CHANGELOG and puts the Responses
streaming entry at the top of Unreleased → Fixed, tagged `(activeagent)`
like its neighbours.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PCEJHsJpjXpDa9Kuq9i4Yg
@TonsOfFun TonsOfFun mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals: the dashboard keeps a product's named criteria lists across import and export

2 participants