Repository navigation
Keep a catalog's pass bars through dashboard import - #702
Merged
Merged
Conversation
A catalog product's named criteria lists (criteria: { shared: [...] })
parse and round-trip in ActiveAgent::Evals::Catalog, but ScenarioProduct
had no column for them. A dashboard import dropped the lists, the export
left them out, the stored digest never matched the records' export, and
the import preview reported the product as changed on every re-import of
an unchanged file.
Store the lists on the product through a guarded upgrade migration, as
add_catalog_scenario_metadata does for scenario extensions, and emit
them from ScenarioProduct#to_document. Both import modes take the
document's lists, as they take a set's criteria, so the merge preview
stays accurate. Which lists a set's run scores on is left for later; a
set still runs on its own criteria.
Fixes activeagents#687
A catalog product's and set's pass blocks (min_pass_rate, threshold, gated) parse in ActiveAgent::Evals::Catalog, but neither ScenarioProduct nor ScenarioSet had a column for them. A dashboard import dropped the bars, the export left them out, the stored catalog's pass_thresholds fell back to the defaults (every group gated at 100%), and the import preview reported the product and its sets as changed on every re-import of an unchanged file. Store each block on its record through a guarded upgrade migration, as add_catalog_product_criteria does for a product's named lists, and emit them from to_document. Both import modes take the document's bars, as they take a set's criteria. Gating a dashboard run on them is left to the run itself; this change only keeps what the document declared. Fixes activeagents#700
This was referenced Nov 7, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack, part 2 of 4. Merge in this order; each PR targets
main.evals run --min-pass-ratehonours the suite's per-group gates #677)Until #701 merges, this PR also shows that commit. The change under review here is one commit: 43c66da. Once the PRs below merge, GitHub shows only that commit.
What this does
#676 lets a catalog product and set carry a
passblock (min_pass_rate,threshold,gated), whichActiveAgent::Evals::Catalogparses andCatalog#pass_thresholdsreads. The dashboard kept neither:ScenarioProductandScenarioSethad no column, the import never wroteentry.pass, andto_documentnever emitted it. After an import the stored catalog'spass_thresholdsread as the defaults (every group gated at 100%), the export left the bars out, the stored digest never matched the records' export, and the preview reported the product and its sets asupdateon every re-import of an unchanged file.Each record now stores its block in a
passJSON column and emits it fromto_document. The migration isadd_catalog_pass_bars(035;033is #674's and034#701's), built like034: guarded per table, safe under a customtable_name_prefix,jsonbon PostgreSQL andjsonelsewhere with no JSON default. Until it runs, imports keep working and leave the bars out. Both import modes take the document's bars, as they take a set'scriteria.This branch is stacked on #701, since both add a column to
scenario_productsand touch the same import lines andschema.rb; it carries that commit until it merges. Gating a dashboard run on the stored bars is #678 and comes next.Tests:
actionagent/test/catalog_pass_bars_test.rb(new), failing before: the stored catalog'spass_thresholdsequal the parsed document's for the product and for one set; the bars survive reload and export, including a set-narrowed export; an unchanged re-import previews no change; a changed or removed bar previews asupdateand is replaced; merge mode takes the document's bars, merges a set export back unchanged, and clears them when the document has none (what its preview says); a stored value that is not a mapping reads as none.actionagent/test/scenario_catalogs_api_test.rb: the same round trip through the import, export and preview endpoints.actionagent/test/install_generator_test.rb: emitted after catalog creation and again only when missing; keeps old rows under a custom prefix, reverses idempotently, skips a missing table (one of the two missing does not stop the other); each adapter's JSON type.Each part reverted in turn fails a test: both import writes, both column guards, both exports, both read normalizations, the export's presence check, and merge-mode clearing on the product and the set.
Type of change
Scope check
Related issue
Fixes #700
Quality check
bin/rubocopon the changed Ruby files)035): 4580 runs, 0 errors, 1 failure inLocalSandboxBackendTest(a boot-interrupt timing test, run while a second suite loaded the machine; green 3 of 3 alone). Rails 8.1: 4576 runs, 0 failures, 10 errors, allCliContractTest(rack-sessionundefined method 'id' for Hash), which fail the same way on an untouchedmainheredocs/framework/dashboard.mdscenario catalogs,docs/framework/evaluations.md, a CHANGELOG Fixed entry)schema.rbmatches what a freshdb:migratewrites)Ownership
API changes
ScenarioProduct#pass,ScenarioSet#pass)