Repository navigation
Keep a catalog product's named criteria lists - #701
Merged
Merged
Conversation
A catalog product's named criteria lists (criteria: { shared: [...] })
parse and round-trip in ActiveAgent::Evals::Catalog, but ScenarioProduct
had no column for them. A dashboard import dropped the lists, the export
left them out, the stored digest never matched the records' export, and
the import preview reported the product as changed on every re-import of
an unchanged file.
Store the lists on the product through a guarded upgrade migration, as
add_catalog_scenario_metadata does for scenario extensions, and emit
them from ScenarioProduct#to_document. Both import modes take the
document's lists, as they take a set's criteria, so the merge preview
stays accurate. Which lists a set's run scores on is left for later; a
set still runs on its own criteria.
Fixes activeagents#687
This was referenced Nov 7, 2026
TonsOfFun
added a commit
to pcbeingused333/activeagent
that referenced
this pull request
Nov 12, 2026
The CHANGELOG moved under this branch: activeagents#699 rewrote Unreleased and activeagents#701 to activeagents#703 added entries. This keeps main's CHANGELOG and puts the Responses streaming entry at the top of Unreleased → Fixed, tagged `(activeagent)` like its neighbours. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCEJHsJpjXpDa9Kuq9i4Yg
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack, part 1 of 4. Merge in this order; each PR targets
main.evals run --min-pass-ratehonours the suite's per-group gates #677)This PR stands alone on
main; the three after it build on it.What this does
#686 gave a catalog product named criteria lists (
criteria: { shared: [...], strict: [...] }), whichActiveAgent::Evals::Catalogparses, layers and writes back. The dashboard'sScenarioProducthad no column for them, so, as #687 describes, an import dropped the lists,ScenarioProduct#to_documentleft them out of every export, the stored digest never matched the records' export, and the import preview reported the product asupdateon every re-import of an unchanged file.docs/framework/evaluations.mddescribed the gap.The product now stores the mapping in a
criteriaJSON column and emits it fromto_document, so import, export and preview agree. The migration followsadd_catalog_scenario_metadata(031): a numbered template (034, since #674 holds033), guarded by table and column so it is safe on fresh and existing installs and under a customtable_name_prefix,jsonbon PostgreSQL andjsonelsewhere with no JSON default. Until it runs, imports keep working and leave the lists out, as scenario metadata does.Both import modes take the document's lists, the same way they take a set's
criteria, a product'smetadataand itsname, so the merge preview still says what the import will do and a narrowed export (one set, one scenario) merges back unchanged. Layering the lists by name and criterion key in merge mode, as layered catalog documents do, would be a small follow-up if you prefer it; the merge preview would then need to compare layered results instead of documents.Not covered, per the issue: which product lists a dashboard run scores on. A set still materializes with its own
criteriaonly, so no evaluation result changes. The product summary JSON is unchanged too.Tests:
actionagent/test/catalog_product_criteria_test.rb(new), failing before: two named lists survive reload and equal the parsed document and digest, including product- and set-narrowed exports; the stored lists reachsuite_for(...).criteria(:shared, :strict)next to the set's own criteria; an unchanged re-import previews no change and adds no records; a changed or removed list previews asupdateand is replaced; merge mode takes the document's lists, merges a set export back with no change, and clears the lists when the document names none (what its preview reports); a bare product list is stored asdefault; a stored value that is not a mapping of lists reads as none.actionagent/test/scenario_catalogs_api_test.rb, failing before: the same round trip through the import, export and preview endpoints.actionagent/test/install_generator_test.rb: the migration is emitted after catalog creation and again only when missing, keeps old rows under a custom prefix, reverses idempotently, skips a missing table, and picks each adapter's JSON type.Each part reverted in turn fails a test: the export, the import write, its column guard, the merge policy, and the three read normalizations.
Type of change
Scope check
Related issue
Fixes #687
Quality check
bin/rubocopon the changed Ruby files; the repo has no commit hooks installed)bin/testwith the CI placeholder keys. Rails 7.2, database built as the CI job builds it (db:create db:migrate, which applies034): 4568 runs, 0 failures, 0 errors. Rails 8.1: 4564 runs, 0 failures, 10 errors, allCliContractTest(rack-sessionundefined method 'id' for Hash), which fail the same way on an untouchedmainin this environmentdocs/framework/dashboard.mdscenario catalogs, the gap sentence indocs/framework/evaluations.md, a CHANGELOG Fixed entry)test/dummy/db/schema.rbcarries the column and version a freshdb:migratewrites)Ownership
API changes
ScenarioProduct#criteria)