Show a run's per-group pass gates in the dashboard

CHANGELOG.md

@@ -9,6 +9,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **The dashboard shows a run's per-group pass gates** (`actionagent`). A
+  published report's `pass_thresholds` are kept with the run, so the
+  rebuilt report, the export and the run page gate the stored results as
+  the application's report did, and a run started from the dashboard on a
+  catalog set is held to the set's and product's `pass` bars, recorded as
+  the run starts so a later catalog change does not regrade it. The run
+  page adds a **Pass rate by group** table, and the run's badge reads
+  `gate missed` when a gate was not met even if the whole-run pass rate
+  looks fine. The run's JSON adds `gates` and `gates_met`; a run without
+  bars reads as before (#678).
 - **An empty agent list shows how agents arrive** (`actionagent`). With no
   agents, the Agents page offered only Browse Templates and Create From
   Scratch. It now shows three ways in, telemetry first:
@@ -23,6 +33,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **The dashboard keeps a catalog's pass bars** (`actionagent`). A
+  product's and a set's `pass` blocks (`min_pass_rate`, `threshold`,
+  `gated`) parsed in `ActiveAgent::Evals::Catalog`, but neither
+  `ScenarioProduct` nor `ScenarioSet` kept them: an import dropped the
+  bars, the export left them out, the stored catalog's `pass_thresholds`
+  fell back to the defaults (every group gated at 100%), and the import
+  preview reported the product and its sets as `update` on every re-import
+  of an unchanged file. The bars are now stored and exported as imported,
+  and both import modes take the document's. Existing installs add the
+  columns with `bin/rails generate action_agent:install --skip` and
+  `bin/rails db:migrate` (`add_catalog_pass_bars`); imports keep working,
+  without the bars, until it runs. Bars dropped by an earlier version come
+  back when the original document is imported again.
+- **The dashboard keeps a catalog product's named criteria lists**
+  (`actionagent`). A product's `criteria: { shared: [...], strict: [...] }`
+  parsed and round-tripped in `ActiveAgent::Evals::Catalog`, but
+  `ScenarioProduct` had nowhere to keep it: an import dropped the lists, the
+  export left them out, the stored digest never matched the records'
+  export, and the import preview reported the product as `update` on every
+  re-import of an unchanged file. The lists are now stored on the product
+  and exported as imported, so the documents and digests match and an
+  unchanged file previews as unchanged. Both import modes take the
+  document's lists, as they take a set's `criteria`. A set still runs on
+  its own `criteria`; which product lists a run uses is left for later.
+  Existing installs add the column with
+  `bin/rails generate action_agent:install --skip` and `bin/rails db:migrate`
+  (`add_catalog_product_criteria`); imports keep working, without the lists,
+  until it runs. Lists dropped by an earlier version come back when the
+  original document is imported again (#687).
 - **A self-hosted dashboard shows its own trace endpoint** (`actionagent`).
   The Organization page's Telemetry card showed the platform's `/v1/traces`.
   A self-hosted engine takes traces at `<mount>/api/traces`, and the card

actionagent/app/models/action_agent/evaluation_run.rb

@@ -87,6 +87,48 @@ def recorded_comparison
       comparison if comparison.is_a?(Hash) && comparison.present?
     end
 
+    # The bars this run was gated by (ActiveAgent::Evals::PassThresholds),
+    # as the run recorded them when it started or was imported, so a later
+    # change to the catalog does not regrade it; nil for a run whose suite
+    # declared none, one recorded before runs kept them, and a framework
+    # without per-group bars.
+    def pass_thresholds
+      self.class.pass_thresholds_from(scores&.dig("_pass_thresholds"))
+    end
+
+    # A recorded `pass_thresholds` mapping (Report#to_h's: "threshold",
+    # "min_pass_rate", "groups") as PassThresholds, or nil for anything
+    # that is not one.
+    def self.pass_thresholds_from(value)
+      return nil unless value.is_a?(Hash) && defined?(ActiveAgent::Evals::PassThresholds)
+
+      value = value.stringify_keys
+      groups = value["groups"].nil? ? {} : value["groups"]
+      return nil unless groups.is_a?(Hash)
+
+      ActiveAgent::Evals::PassThresholds.new(suite: value.slice(*ActiveAgent::Evals::PassThresholds::SUITE_KEYS), groups: groups)
+    rescue ArgumentError
+      nil
+    end
+
+    # The gates the run's results met or missed (Report#gates, each
+    # `{ "group", "name", "label", "passed", "total", "pass_rate",
+    # "min_pass_rate", "gated", "met" }`), as recorded when it completed;
+    # nil for a run without bars.
+    def gates
+      value = scores&.dig("_gates")
+      value.select { |gate| gate.is_a?(Hash) } if value.is_a?(Array)
+    end
+
+    # Whether every gate was met, as Report#passed? answers: false when no
+    # gate was reached; nil for a run without bars.
+    def gates_met
+      rows = gates
+      return nil if rows.nil?
+
+      rows.any? && rows.all? { |gate| gate["met"] == true }
+    end
+
     # How the report names the judge, the way the suite panel does: the
     # judge the recorded verdict names — unless that is only the pass-rate
     # ranking — else the evaluation's judge model. nil when neither is set,
@@ -307,6 +349,7 @@ def to_report(links: report_links, estimate: true)
       kwargs[:judge_usage] = judge_usage_run if judge_usage_run && self.class.report_accepts?(:judge_usage)
       kwargs[:release] = release_summary if release_summary && self.class.report_accepts?(:release)
       kwargs[:comparison] = recorded_comparison if recorded_comparison && self.class.report_accepts?(:comparison)
+      kwargs[:pass_thresholds] = pass_thresholds if pass_thresholds && self.class.report_accepts?(:pass_thresholds)
       self.class.report_class.new(**kwargs)
     end
 

actionagent/app/models/action_agent/scenario_product.rb

@@ -21,6 +21,28 @@ def metadata
       value.is_a?(Hash) ? value : {}
     end
 
+    # The product's named criteria lists, as the catalog's `criteria:` gives
+    # them: list name => criterion hashes. A stored value that is not such a
+    # mapping, and a list left with no criterion, read as no list.
+    # @return [Hash{String => Array<Hash>}]
+    def criteria
+      value = self[:criteria]
+      return {} unless value.is_a?(Hash)
+
+      value.each_with_object({}) do |(name, list), lists|
+        entries = Array(list).select { |criterion| criterion.is_a?(Hash) } if list.is_a?(Array)
+        lists[name.to_s] = entries if entries.present?
+      end
+    end
+
+    # The bars the catalog's `pass:` declares (ActiveAgent::Evals::PassThresholds);
+    # a stored value that is not a mapping reads as none.
+    # @return [Hash{String => Object}]
+    def pass
+      value = self[:pass]
+      value.is_a?(Hash) ? value : {}
+    end
+
     # The agent a set of this product runs against when the caller names
     # none: the product's agent, else the project's agent under test, else
     # the owner's agent named by the document.
@@ -36,6 +58,8 @@ def to_document
         "name" => name,
         "description" => description,
         "agent" => agent_name.presence || agent&.name,
+        "criteria" => criteria.presence,
+        "pass" => pass.presence,
         "sets" => sets.map(&:to_document)
       }.merge(metadata).compact
     end

actionagent/app/models/action_agent/scenario_set.rb

@@ -37,6 +37,14 @@ def metadata
       value.is_a?(Hash) ? value : {}
     end
 
+    # The bars the catalog's `pass:` declares (ActiveAgent::Evals::PassThresholds);
+    # a stored value that is not a mapping reads as none.
+    # @return [Hash{String => Object}]
+    def pass
+      value = self[:pass]
+      value.is_a?(Hash) ? value : {}
+    end
+
     # The evaluation's name: the catalog, product and set keys.
     def evaluation_name
       "#{catalog.key}/#{product.key}/#{key}".first(EvaluationReportImport::MAX_EVALUATION_NAME)
@@ -117,6 +125,7 @@ def to_document
         "description" => description,
         "judge" => judge.presence,
         "criteria" => criteria.presence,
+        "pass" => pass.presence,
         "scenarios" => scenarios.map(&:to_document)
       }.merge(metadata).compact
     end

actionagent/app/serializers/action_agent/evaluation_serializer.rb

@@ -81,7 +81,9 @@ def scores(run)
     # checkout sandbox the run replayed against, if any: its session id and
     # checkout, never its token. `agent_version` is the version of the
     # agent the run scored, and `version_state` whether that is the agent's
-    # current version ("current", "earlier" or "unrecorded").
+    # current version ("current", "earlier" or "unrecorded"). `gates` are the
+    # per-group pass gates the run was held to (EvaluationRun#gates) and
+    # `gates_met` whether it met every one; both nil for a run without bars.
     def run_summary(run, number: nil)
       {
         id: run.id,
@@ -95,7 +97,9 @@ def run_summary(run, number: nil)
         created_at: run.created_at.iso8601,
         sandbox: run.sandbox,
         agent_version: run.agent_version_summary,
-        version_state: run.evaluation.standing_info.version_state(run)
+        version_state: run.evaluation.standing_info.version_state(run),
+        gates: run.gates,
+        gates_met: run.gates_met
       }
     end
 

actionagent/app/services/action_agent/evaluation_report_import.rb

@@ -95,6 +95,10 @@ def initialize(denial)
     JUDGE_USAGE_KEYS = %w[calls input_tokens output_tokens cost model by_kind source].freeze
     JUDGE_USAGE_SOURCES = %w[reported traces meter].freeze
     MAX_JUDGE_KINDS = 10
+    # Groups a report's `pass_thresholds` may name; a group holds at least
+    # one result, so no report needs more.
+    MAX_PASS_GROUPS = MAX_RESULTS
+    PASS_THRESHOLD_KEYS = %w[threshold min_pass_rate groups].freeze
     SCOPE_PATTERN = %r{\A[\w .:/@-]{1,100}\z}
     STATUSES = %w[passed failed errored].freeze
     # The envelope statuses a delivery may name.
@@ -674,10 +678,21 @@ def recorded_scores
         "_verdict" => report["verdict"],
         "_selection" => selection,
         "_metadata" => metadata,
-        "_judge_usage_run" => bounded_judge_usage(report.dig("judge_usage", "run"))
+        "_judge_usage_run" => bounded_judge_usage(report.dig("judge_usage", "run")),
+        "_pass_thresholds" => recorded_pass_thresholds
       }.compact.merge("_judge_label" => judge_label)
     end
 
+    # The bars the report was gated by, as PassThresholds normalizes them
+    # (validate_pass_thresholds! has refused anything else), so the rebuilt
+    # report gates the stored results the way the application's did. nil
+    # for a report whose suite declared none.
+    def recorded_pass_thresholds
+      return nil if report["pass_thresholds"].nil?
+
+      EvaluationRun.pass_thresholds_from(report["pass_thresholds"])&.to_h
+    end
+
     # The run's scores in the shape the Evaluations view renders
     # (ScenarioEvaluationRunner#scores_for), summarized from the stored
     # results as the application reported them — no cost is estimated here,
@@ -689,10 +704,14 @@ def recorded_scores
     def summarized_scores(run)
       rebuilt = run.to_report(estimate: false)
       comparison = rebuilt.comparison if rebuilt.respond_to?(:comparison)
+      # The gates are worked out from the stored results as well: the
+      # report's own "gates" are not taken as the verdict.
+      gates = rebuilt.gates.map(&:to_h) if rebuilt.respond_to?(:pass_thresholds) && rebuilt.pass_thresholds
       rebuilt.criterion_scores.merge(
         "_models" => rebuilt.summary_by_model,
         "_recommendations" => rebuilt.recommendations,
-        "_comparison" => comparison
+        "_comparison" => comparison,
+        "_gates" => gates
       ).compact.merge(recorded_scores)
     end
 
@@ -725,6 +744,7 @@ def validate!
       judge_trace_ids!(metadata["judge_trace_ids"])
       string!(report["judge"], "report.judge", 200)
       validate_release!
+      validate_pass_thresholds!
       validate_judge_usage!(report.dig("judge_usage", "run"), "report.judge_usage.run") if report["judge_usage"].is_a?(Hash)
       optional_object!(report["judge_usage"], "report.judge_usage")
       object!(report["models"], "report.models")
@@ -856,6 +876,33 @@ def validate_release!
       string!(release["label"], "report.release.label", 200)
     end
 
+    # The bars the suite declared, as Report#to_h writes them: "threshold"
+    # and "min_pass_rate" from 0 to 1, and "groups", each group's
+    # "min_pass_rate" and "gated", by group key. Checked by PassThresholds
+    # itself, so the import refuses what the framework would.
+    def validate_pass_thresholds!
+      bars = report["pass_thresholds"]
+      return if bars.nil?
+
+      object!(bars, "report.pass_thresholds")
+      unknown = bars.keys - PASS_THRESHOLD_KEYS
+      raise Invalid, "report.pass_thresholds has unknown keys #{unknown.join(', ')}" if unknown.any?
+
+      groups = bars["groups"]
+      optional_object!(groups, "report.pass_thresholds.groups")
+      if groups && groups.size > MAX_PASS_GROUPS
+        raise Invalid, "report.pass_thresholds.groups must name at most #{MAX_PASS_GROUPS} groups"
+      end
+
+      (groups || {}).each_key { |key| string!(key, "report.pass_thresholds.groups key", 200) }
+      # A framework without per-group bars stores none, as before.
+      return unless defined?(ActiveAgent::Evals::PassThresholds)
+
+      ActiveAgent::Evals::PassThresholds.new(suite: bars.slice(*ActiveAgent::Evals::PassThresholds::SUITE_KEYS), groups: groups || {})
+    rescue ArgumentError => e
+      raise Invalid, "report.pass_thresholds: #{e.message}"
+    end
+
     # What the judge spent, as ActiveAgent::Evals::Report writes it per
     # result and for the run.
     def validate_judge_usage!(usage, name)

actionagent/app/services/action_agent/scenario_catalog_import.rb

@@ -181,6 +181,10 @@ def replace_products(parsed)
           metadata: entry.metadata, agent: resolve_agent(entry) || (merge? ? product.agent : nil),
           project: resolve_project(entry) || (merge? ? product.project : nil)
         )
+        # Until add_catalog_product_criteria runs, imports keep working without the lists.
+        product.criteria = entry.criteria if product.has_attribute?(:criteria)
+        # Until add_catalog_pass_bars runs, imports keep working without the bars.
+        product.pass = entry.pass if product.has_attribute?(:pass)
         product.position = position_for(product, entry.position, offset)
         product.save!
         replace_sets(product, entry)
@@ -197,6 +201,7 @@ def replace_sets(product, entry)
           name: set_entry.name, description: set_entry.description, judge: set_entry.judge,
           criteria: set_entry.criteria, metadata: set_entry.metadata
         )
+        set.pass = set_entry.pass if set.has_attribute?(:pass)
         set.position = position_for(set, set_entry.position, offset)
         set.save!
         replace_scenarios(set, set_entry)