Skip to content

Fix live staging simulation tests - #3872

Merged
anth-volk merged 1 commit into
masterfrom
fix/live-staging-test-contracts
Oct 8, 2026
Merged

anth-volk merged 1 commit into
masterfrom
fix/live-staging-test-contracts

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #3873

Repair the invalid live-test requests and incorrect credit assertion reported in deployment run 37820880125. The California and Scotland macro tests now identify each run through supported policy labels instead of the undeclared staging_probe query parameter. The California household test expects $388.21, independently reproduced with its exact 2025 household using the pinned PolicyEngine bundle 6.2.1 / US model 2.2.1.

Add local regression coverage that executes the live macro request construction against the canonical query parser and the household assertions against the real pinned model. Include response text in HTTP polling exceptions so future request-validation errors appear in CI logs.

Validation: all 14 GitHub checks passed, including the full API test job, cross-database integration, both container builds, lint, type checking, and migration checks. Locally, 102 focused tests passed; the four request/model regressions also passed after formatting. Repository formatting, changed-file lint, changelog, and whitespace checks passed.

The full live suite against the exact failed staging candidate finished with 9 passed, 1 failed. Both California cases passed. The corrected Scotland request now reaches the simulation worker and exposes a separate upstream failure: ValueError: Variable 'country' not found in household data. The pinned policyengine package's UK country-region filter reads a country input column; the loaded dataset stores region, and the UK model derives country from that variable. The candidate metadata explicitly advertises country/scotland, so the live test continues to exercise that supported selector. Worker correlation IDs: b69a1bd89eac44a68fbf9d4b53223d90 (targeted run) and 786c6cffb1a74330be6e9903e07476df (full run).

@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.56%. Comparing base (b678789) to head (e7a570c).

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #3872       +/-   ##
===========================================
+ Coverage   45.22%   87.56%   +42.33%     
===========================================
  Files         183      198       +15     
  Lines       11073    12211     +1138     
  Branches     1905     2159      +254     
===========================================
+ Hits         5008    10692     +5684     
+ Misses       5644      922     -4722     
- Partials      421      597      +176     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk
anth-volk marked this pull request as ready for review October 8, 2026 20:02
@anth-volk
anth-volk merged commit b62c10e into master Oct 8, 2026
14 checks passed
@anth-volk
anth-volk deleted the fix/live-staging-test-contracts branch October 8, 2026 20:05

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix live staging test request contracts and California credit assertion

1 participant