Skip to content

Add parallel live simulation coverage - #3820

Merged
anth-volk merged 4 commits into
masterfrom
feat/2400-parallel-live-simulations
Oct 7, 2026
Merged

anth-volk merged 4 commits into
masterfrom
feat/2400-parallel-live-simulations

Conversation

@anth-volk

@anth-volk anth-volk commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2400

Summary

  • keep candidate health, metadata, and the stateful v2 policy lifecycle checks in the serial staging integration job
  • run the non-stateful live test selection inside one GitHub Actions job using two pytest-xdist worker processes with dynamic per-test scheduling
  • limit concurrency explicitly with -n 2, use --dist load, and schedule one test at a time with --maxschedchunk=1
  • use the exact tagged candidate URL and collision-resistant probe identifiers containing the deployment tag, workflow run, run attempt, and a generated worker-session UUID
  • add fixed 2025 household assertions for federal EITC, federal CTC, CalEITC, and Universal Credit for a household in Scotland
  • add society-wide California CalEITC and Scotland Universal Credit reforms while retaining the existing Utah and budget-window coverage
  • remove the obsolete Enhanced CPS configuration requirement from the linked issue; society-wide requests use the currently certified default dataset
  • require both the serial staging checks and the pytest-xdist live-test job to pass before staging promotion

Validation

  • make format
  • Ruff checks on all changed Python files
  • pytest tests/unit/test_cloud_run_deploy_scripts.py -q (102 passed)
  • executed the live-test runner locally with pytest-xdist: 2 workers distributed all 11 tests; all skipped because no deployed API URL was supplied
  • validated .github/workflows/push.yml with actionlint
  • validated the live-test runner with bash -n
  • parsed all new JSON fixtures with jq
  • resolved both reform parameter paths against the pinned US and UK packages
  • calculated the expected household results locally with the pinned US and UK packages
  • ran the changelog-fragment and diff checks

The deployed live checks were not run locally because they require a newly deployed tagged staging candidate.

@codecov

codecov Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.56%. Comparing base (b9ad8db) to head (ab8a9c1).

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #3820       +/-   ##
===========================================
+ Coverage   45.22%   87.56%   +42.33%     
===========================================
  Files         183      198       +15     
  Lines       11073    12211     +1138     
  Branches     1905     2159      +254     
===========================================
+ Hits         5008    10692     +5684     
+ Misses       5644      922     -4722     
- Partials      421      597      +176     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk

Copy link
Copy Markdown
Collaborator Author

Deployment validation for this PR currently depends on resolving the pre-existing failure in tests/integration/test_live_phase10_staging.py::test_live_phase10_activation_failure_and_retry on master (failed workflow run). Anthony is working on that failure; this PR should remain blocked until the existing deployment sequence completes successfully.

@anth-volk
anth-volk force-pushed the feat/2400-parallel-live-simulations branch from 4b44816 to 37a8e95 Compare September 11, 2026 17:06
@anth-volk
anth-volk force-pushed the feat/2400-parallel-live-simulations branch from 37a8e95 to bcf81e1 Compare September 11, 2026 18:45
@anth-volk
anth-volk force-pushed the feat/2400-parallel-live-simulations branch from bcf81e1 to ab8a9c1 Compare October 7, 2026 11:39
@anth-volk
anth-volk marked this pull request as ready for review October 7, 2026 12:31
@anth-volk
anth-volk merged commit 93f389f into master Oct 7, 2026
14 checks passed
@anth-volk
anth-volk deleted the feat/2400-parallel-live-simulations branch October 7, 2026 12:31

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand live simulation integration coverage across US and UK

1 participant