anth-volk · Oct 1, 2026 · Oct 1, 2026 · Oct 1, 2026 · Oct 1, 2026 · Oct 1, 2026
diff --git a/changelog.d/3862.breaking.md b/changelog.d/3862.breaking.md
@@ -0,0 +1 @@
+Replace large SPM diagnostic provenance payloads and duplicate completed-result SPM configuration with strict compact v2 receipts for household, society-wide, report, and Stage 12 artifact results. Older cached or stored SPM output is recalculated instead of translated, and runtime version values are required.
diff --git a/docs/canonical-spm.md b/docs/canonical-spm.md
@@ -155,16 +155,44 @@ storage; no additional schema migration is required.
 selected policy. No independent simulation-level override is supported. Top-level
 `spm` on simulation POST/PATCH is rejected instead of silently ignored.
 
-Successful canonical calculations add `spm_config` and `spm_provenance` beside
-`result`. A receipt describes the measurement the simulation was constructed
-with, not a guarantee that every SPM-dependent variable produced a value: a
+Successful canonical calculations add only `spm_provenance` beside `result`.
+Its forecast, scenario, geography, county-vintage and as-of fields are the
+complete resolved calculation configuration; completed results do not repeat
+that configuration in a sibling `spm_config` field. A receipt describes the
+measurement the simulation was constructed with, not a guarantee that every
+SPM-dependent variable produced a value: a
 calculation that never chose a measurement still carries the inherited one's
 receipt beside its null cells. Read the values to learn which of them a
-measurement produced. Provenance comes from the actual simulation and includes artifact,
-scenario, years, geography, composition/storage methods and runtime versions.
-It remains in JSON form through stored replay and cache hits. A tax-only receipt
-may have empty `years` and `geographies` because no SPM measurement was requested.
-Clients saving simulation outputs must retain this full response envelope.
+measurement produced. Provenance comes from the actual simulation and uses the
+required compact `canonical-spm-provenance-v2` schema:
+
+```json
+{
+  "schema_version": "canonical-spm-provenance-v2",
+  "forecast_id": "canonical-spm-forecast",
+  "forecast_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+  "scenario": "baseline",
+  "geography_kind": "national",
+  "geography_id": null,
+  "county_vintage": "2020",
+  "as_of": null,
+  "years": ["2026"],
+  "runtime_versions": {
+    "policyengine": "6.2.1",
+    "policyengine-core": "3.32.10",
+    "policyengine-us": "2.2.1",
+    "spm-calculator": "1.0.0"
+  }
+}
+```
+
+All fields are required. The four runtime-version values are non-null strings.
+`years` is sorted and unique, and it is empty only when no SPM dependency ran.
+Large diagnostic, mapping, request, effective-setting and source
+objects from the country package are intentionally excluded from HTTP and
+cache payloads. The compact receipt remains in JSON form through stored replay
+and cache hits. Clients saving simulation outputs must retain this response
+envelope.
 
 `POST /us/simulation` takes `population_id`, `population_type` (`household` or
 `geography`) and integer `policy_id`. It creates a pending record (HTTP 201) or
@@ -181,12 +209,13 @@ country, certification, or whether the household has saved `spm`.
 `PATCH /us/simulation` identifies the record with body `id` and accepts `status`
 (`pending`, `complete` or `error`), `output` and `error_message`. At least one
 update field must be non-null, and `complete` requires non-null `output`. Store
-the full calculation envelope inside `output`, including its `spm_config` and
+the full calculation envelope inside `output`, including its
 `spm_provenance`. A JSON-encoded output string is also accepted. This endpoint
-stores the supplied output; it does not validate SPM receipt integrity. Null
-update fields are ignored. The legacy `api_version` input is ignored; writes
-record the installed country model version. Top-level `spm`, even null, returns
-HTTP 400 `SPM_SETTINGS_UNSUPPORTED` on POST and PATCH.
+validates the one compact SPM receipt strictly before storage. Completed output
+that includes the former sibling `spm_config`, an old receipt or a rich receipt
+is rejected rather than translated. Null update fields are ignored. The legacy `api_version` input
+is ignored; writes record the installed country model version. Top-level `spm`,
+even null, returns HTTP 400 `SPM_SETTINGS_UNSUPPORTED` on POST and PATCH.
 
 POST, PATCH and `GET /us/simulation/{id}` return the simulation record inside
 `result`. Non-string JSON `output` and `simulation_spec_json` are returned as
@@ -201,9 +230,10 @@ Flask and the native specification route.
 
 HTTP response cache identity includes the normalized selection and model/bundle
 versions, and validates certification before reading the cache. The stored
-calculated-household cache uses schema version 2, includes the selection in
-identity, and stores settings/provenance atomically with the result and its
-warnings. Missing or mismatched canonical receipts are cache misses.
+calculated-household cache uses schema version 4, includes the selection in
+identity, and stores provenance atomically with the result and its
+warnings. Missing, old, rich or mismatched canonical receipts are cache misses
+and are recalculated rather than translated.
 
 ## Errors
 
@@ -261,22 +291,34 @@ from `/versions`, including agreement with the bundle's independently pinned
 artifact hash. An old or uncertified worker returns `SPM_CONFIGURATION_UNAVAILABLE`.
 
 Normalized settings are sent to both annual and budget-window jobs and participate
-in economy cache identity. Results carry `spm_config` and `spm_provenance` with
-`baseline` and `reform` receipt lists (one per executed regional segment). Each
-budget-window annual row carries these fields; annual results expose them inside
-`result`, and budget-window results inside `result.annualImpacts`. Stored results require matching
-settings and valid receipts. Typed SPM input errors remain structured 400s
+in economy cache identity. Results carry only `spm_provenance` using
+`canonical-spm-comparison-v2`. `baseline` and `reform` each contain one compact
+`receipt` plus an `execution_count`; the receipt collapses identical child
+execution receipts without transporting a repeated list. Baseline and reform
+receipts must be identical, and their receipt fields are the sole resolved SPM
+configuration. Each budget-window annual row carries this field;
+annual results expose it inside `result`, and budget-window results inside
+`result.annualImpacts`. Stored results require matching settings and valid
+receipts. Economy cache identity includes the comparison schema version, so old
+entries are not selected and are recomputed rather than translated. Typed SPM input errors remain structured 400s
 through asynchronous submission and polling. Poll-time typed failures retain
 their code and message for the existing cache lifetime. Later reads replay that
 failure after API service restarts without polling or resubmitting the failed
 job. Canonical cache identity and runtime-bundle refresh rules still apply.
 
+Completed US economy report outputs pass the same compact comparison validation
+before `report_outputs` or `report_output_runs` persistence. Their report cache
+identity uses schema version 3. Reads of earlier report records resolve
+to a new pending record with no copied output, preventing another client from
+receiving a previously stored rich receipt. Household and non-US report output
+shapes retain their existing behavior.
+
 See the [canonical SPM worker PR](https://github.com/PolicyEngine/policyengine-sim-api/pull/677)
 for the implemented worker paths and remaining coordinated release gates.
 
 
 Partial selections preserve omitted fields in JSON; omissions inherit the certified
 bundle defaults. The selection schema deliberately supplies no wire defaults,
 so generated clients preserve that distinction. Resolved settings freeze all six
-fields. Responses may omit null geography_id/as_of values, but must include every
-non-null resolved field; receipt replay never substitutes current defaults.
+fields. Responses include all six fields, including explicit null values for
+`geography_id` or `as_of`; receipt replay never substitutes current defaults.
diff --git a/docs/migration/stage-12-worker-contract.md b/docs/migration/stage-12-worker-contract.md
@@ -171,6 +171,14 @@ SHA-256 content digests, schema versions, row-identity metadata, and bounded
 provenance. When an option such as SPM produces a detached calculation receipt,
 the simulation descriptor and Parquet metadata retain it so the coordinator can
 validate the receipt before reproducing the existing aggregate response.
+`SimulationArtifactDescriptor.calculation_provenance` is either null or the
+exact typed `canonical-spm-provenance-v2` receipt. The receipt's forecast,
+scenario, geography, county-vintage and as-of fields are the complete resolved
+configuration; the descriptor does not transport a duplicate `spm_config`.
+It rejects partial, rich, legacy and dual-field receipt shapes rather than
+preserving or normalizing them.
+`AggregateReportArtifactPayload.result` applies the same rule to comparison
+provenance and rejects a sibling `spm_config` before artifact storage.
 
 Only the Simulation Entrypoint, the versioned worker application, the bucket's
 object-lifecycle process, and explicitly authorized operators may access the

diff --git a/policyengine_api/constants.py b/policyengine_api/constants.py
@@ -176,8 +176,8 @@ def get_bundle_default_dataset_option(country_id: str) -> dict:
 ) or _resolve_distribution_version(_dist_versions, "policyengine-core", "policyengine")
 
 RUNTIME_CACHE_SCHEMA_VERSIONS = {
-    "economy_impact": 2,
-    "report_output": 1,
+    "economy_impact": 3,
+    "report_output": 3,
 }
 
 

diff --git a/policyengine_api/openapi_spec.yaml b/policyengine_api/openapi_spec.yaml
@@ -439,8 +439,6 @@ paths:
                     nullable: true
                   result:
                     type: object
-                  spm_config:
-                    $ref: '#/components/schemas/SPMSelection'
                   spm_provenance:
                     $ref: '#/components/schemas/SPMProvenance'
         404:
@@ -531,8 +529,6 @@ paths:
                     nullable: true
                   result:
                     type: object
-                  spm_config:
-                    $ref: '#/components/schemas/SPMSelection'
                   spm_provenance:
                     $ref: '#/components/schemas/SPMProvenance'
         500:
@@ -623,9 +619,9 @@ paths:
         non-null output. Null update fields are ignored, and omitted fields retain
         their values. The installed country model version and current run state
         are updated with the record. Preserve the full household calculation
-        envelope, including spm_config and spm_provenance, inside output. Its SPM
-        selection belongs to the linked household; this operation does not select
-        a measurement or validate stored receipt integrity. Top-level spm is
+        envelope, including spm_provenance, inside output. Its SPM selection is
+        derived from that receipt and belongs to the linked household; this
+        operation does not select a measurement. Top-level spm is
         rejected, including null, with HTTP 400 SPM_SETTINGS_UNSUPPORTED.
       parameters:
         - name: country_id
@@ -658,7 +654,7 @@ paths:
         Object and array outputs and simulation_spec_json use the legacy
         JSON-encoded string representation, or null when absent. Stored scalar
         strings are returned unchanged. Decode a saved household output envelope
-        to read its spm_config and spm_provenance. Run identifiers and
+        to read its spm_provenance. Run identifiers and
         specification metadata may be null on historical records.
       parameters:
         - name: country_id
@@ -760,7 +756,10 @@ paths:
     patch:
       summary: Update a report output
       operationId: update_report_output
-      description: Update a report output status, result, or error message.
+      description: >-
+        Update a report output status, result, or error message. Completed US
+        economy outputs require compact v2 comparison provenance. Each receipt
+        contains the complete resolved configuration; a sibling spm_config is rejected.
       parameters:
         - name: country_id
           in: path
@@ -880,7 +879,7 @@ paths:
         # Query parameters are generated from policyengine_api.query_parameters.
       responses:
         200:
-          description: Annual economic impact, progress, or persisted error state. Canonical completed results include the selected SPM configuration and actual baseline/reform receipts.
+          description: Annual economic impact, progress, or persisted error state. Canonical completed results include actual baseline/reform receipts containing the selected SPM configuration.
           content:
             application/json:
               schema:
@@ -896,10 +895,8 @@ paths:
                     type: object
                     nullable: true
                     properties:
-                      spm_config:
-                        $ref: '#/components/schemas/SPMSelection'
                       spm_provenance:
-                        $ref: '#/components/schemas/SPMWorkerProvenance'
+                        $ref: '#/components/schemas/SPMComparisonProvenance'
         400:
           $ref: '#/components/responses/SPMValidationError'
         404:
@@ -969,10 +966,8 @@ paths:
                           properties:
                             year:
                               type: string
-                            spm_config:
-                              $ref: '#/components/schemas/SPMSelection'
                             spm_provenance:
-                              $ref: '#/components/schemas/SPMWorkerProvenance'
+                              $ref: '#/components/schemas/SPMComparisonProvenance'
                   progress:
                     type: integer
                     nullable: true
@@ -1188,7 +1183,7 @@ components:
           description: >-
             Non-string JSON output is JSON-encoded, or null when absent. Stored
             scalar strings are returned unchanged. Decode a saved household
-            calculation envelope once to recover spm_config and spm_provenance
+            calculation envelope once to recover spm_provenance
             beside its result, not beside this record.
           x-household-envelope-schema:
             $ref: '#/components/schemas/StoredHouseholdOutputEnvelope'
@@ -1217,7 +1212,7 @@ components:
       type: object
       description: >-
         Household calculation envelope saved as simulation output. Keep result
-        and both SPM fields together. Legacy calculations omit SPM fields; this
+        and SPM provenance together. Legacy calculations omit SPM provenance; this
         descriptive schema does not add validation to the persistence endpoint.
       properties:
         status:
@@ -1228,8 +1223,6 @@ components:
         result:
           type: object
           nullable: true
-        spm_config:
-          $ref: '#/components/schemas/SPMSelection'
         spm_provenance:
           $ref: '#/components/schemas/SPMProvenance'
     SimulationError:
@@ -1270,25 +1263,16 @@ components:
           description: JSON Schema for SPMSelection, preserving omitted fields.
         defaults:
           $ref: '#/components/schemas/SPMSelection'
-    SPMWorkerProvenance:
-      type: object
-      additionalProperties: false
-      required: [baseline, reform]
-      properties:
-        baseline:
-          type: array
-          minItems: 1
-          items:
-            $ref: '#/components/schemas/SPMProvenance'
-        reform:
-          type: array
-          minItems: 1
-          items:
-            $ref: '#/components/schemas/SPMProvenance'
     SPMSelection:
       description: Generated from policyengine_api.spm.SPMSelection.
+    SPMRuntimeVersions:
+      description: Generated from policyengine_api.spm.SPMRuntimeVersions.
     SPMProvenance:
       description: Generated from policyengine_api.spm.SPMProvenance.
+    SPMExecutionProvenance:
+      description: Generated from policyengine_api.spm.SPMExecutionProvenance.
+    SPMComparisonProvenance:
+      description: Generated from policyengine_api.spm.SPMComparisonProvenance.
   responses:
     SPMValidationError:
       description: >-

diff --git a/policyengine_api/routes/household_routes.py b/policyengine_api/routes/household_routes.py
@@ -137,10 +137,8 @@ def _calculation_response(calculation) -> dict:
     result = dict(status="ok", message=None, result=calculation.household)
     if calculation.warnings:
         result["warnings"] = list(calculation.warnings)
-    for field in ("spm_config", "spm_provenance"):
-        value = getattr(calculation, field, None)
-        if value is not None:
-            result[field] = value
+    if calculation.spm_provenance is not None:
+        result["spm_provenance"] = calculation.spm_provenance
     return result
 
 
@@ -231,7 +229,7 @@ def _calculation_cache_key(*args, **kwargs):
     country_id = request.view_args["country_id"]
     return hash_object(
         {
-            "schema": 2,
+            "schema": 3,
             "path": request.full_path,
             "payload": request.get_json(),
             "spm": g.spm,
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Replace large SPM diagnostic provenance payloads and duplicate completed-result SPM configuration with strict compact v2 receipts for household, society-wide, report, and Stage 12 artifact results. Older cached or stored SPM output is recalculated instead of translated, and runtime version values are required.