Stage 12A: Sign-in identity and v1-anchored user associations

[anth-volk]

Fixes #3892

Summary

Stage 12A of the unified API migration, Public API side. It adds user identity and server-side user associations anchored to v1 artifacts, so app v3 saves every user's work through the Public API: signed in with Auth0 social sign-on, or signed out as the browser's unauthenticated user.

• Two user types: users.user_type is standard (signed in with Auth0) or unauthenticated (a browser's signed-out user, whose ID is a random UUID the browser made and every other user column null).
• Who a request is made as (router-level require_user, before any 422):
  • Authorization: Bearer …: Auth0 RS256 validation (libs/user_auth.py); 403 for machine clients, 503 when auth is unconfigured. A bad token is a 401 and never falls back to the header.
  • Otherwise PolicyEngine-Unauthenticated-User: <uuid> (lowercase, hyphenated): an unauthenticated principal. No Auth0 settings are needed. A UUID that names a standard user is a 401, so a signed-out browser can never act on an account.
  • Neither: 401 with WWW-Authenticate: Bearer.
• Provisioning strategies: StandardAccountProvisioning (the user_identities (issuer, subject) → users.id mapping, race-safe, provisioning on any request) and UnauthenticatedAccountProvisioning (a users row with the browser's UUID, created only by a write: reads by a browser that has saved nothing return empty lists or 404 and create nothing), chosen by principal type.
• Upgrade at sign-in: POST /v2/me/upgrade?country_id= takes both the bearer and the header. A first sign-in converts the unauthenticated user in place (adds the identity, sets standard and the country), so every saved row stays; an existing account merges its rows in and the unauthenticated user is deleted; otherwise nothing happens (none). It never touches a standard user named in the header, and is safe under concurrent requests. Returns {item: MeItem, outcome}.
• /v2/me: GET (MeItem now has user_type, and primary_country may be null), PATCH (403 for unauthenticated users), GET /v2/me/export (UK GDPR; user is null for a browser that has saved nothing) and DELETE /v2/me (an unauthenticated user's data is deleted without any Auth0 call). There is no import route: signed-out work is already in the database.
• Write limits: every /v2/me* and /v2/user-* write rejects a body over 256 KiB with 413, checked after authentication (FastAPI has already parsed the body by then). NaN and Infinity are rejected with 422. description and source_note are capped at 10,000 characters, and the report validation snapshot has list and string bounds.
• Associations:
  • /v2/user-policies and /v2/user-households are locked down. The owner comes from the token or the unauthenticated header, user_id is removed from bodies and queries, and another user's row returns 404.
  • New /v2/user-simulations and /v2/user-reports.
  • All four accept v1 references (legacy_*_id), each checked to exist in Cloud SQL through the v1 SQLAlchemy models. A missing one returns 422.
• Shared reports: a saved report's ID is its shareable link. GET /v2/shared-reports/{id}?country_id= reads any user's saved report with no credential, outside the authenticated router, and never provisions a user (404 when no report has that ID in that country).
  • It leaves out user_id, user_policy_id and baseline_user_policy_id, and names the reform the report was compared with instead (baseline_reform: {name}, null for current law or once that reform is deleted).
  • The owner routes stay owner-scoped, and there is no shared list or search.
• Schema: one autogenerated v2 revision, fbb7488ebe8a (parent 91af4fc8d8c1):
  • creates user_identities;
  • users gains user_type (default standard, with a CHECK) and primary_country becomes nullable, required for standard users by ck_users_standard_primary_country;
  • every association gets nullable v1 and v2 references with a CHECK;
  • the dormant user_simulation_associations and user_report_associations are reshaped, dropping their FKs to simulations / reports. Their old country and label columns are kept, nullable and unused, so the previous API revision can still load rows; a later revision drops them;
  • v3 report fields are added: user_policy_id (the saved reform the report was run from), baseline_user_policy_id (the saved reform it was compared with, when not current law; both are the same owner's rows, ON DELETE SET NULL, and the baseline's name is that reform's own, so no label is stored), source_note, provisions (each may carry optional growth, growth_start, growth_base, growth_round_to and growth_round settings), validation, and budget_window_years (the years of the budget window run with the report, from its year; NULL for none, otherwise 2 to the economy route's 75-year limit, checked by ck_user_report_associations_budget_window_years).
• Persistence: every SQL access goes through SQLAlchemy or SQLModel, and every schema change through Alembic autogeneration. There are no v1 schema changes.
• Qualification: a read-only SQLAlchemy command, policyengine_api.data.v2.user_association_migration_qualification. It is wired into seed-v2-database.yml before the upgrade and fails if the dormant tables hold rows.
• Rules registry (from master): the sign-in and account-deletion libraries join the v2 mypy rule, and the user-service SQL guard is registered as api.v2.user-services-no-sql with proof fixtures.
• Ruff and CI scripts (from master): this branch's code passes ruff groups 1 and 2, and its 19 model bases are listed in runtime-evaluated-base-classes, so their subclasses keep the imports Pydantic needs at runtime. The staging release-test step now runs .github/scripts/run_release_tests.sh, which also unsets the PUBLIC_API_AUTH_* settings, so the staging job's make test never sees them.
• Bundled versions: PolicyEngine bundle 6.4.1, matching the simulation API (the update from Update PolicyEngine bundle to 6.4.1 #3939, which closed unmerged): policyengine 6.4.1, policyengine-core 3.33.0, policyengine-us 2.2.1, policyengine-uk 2.130.1 (was 2.126.12), spm-calculator 1.0.0. Certified data: US populace-us-2024-spm-20260915, UK policyengine-uk-data-1.58.1 (was 1.58.0). UK 2.130.1 adds variables and parameters, so the installed-catalog test expects the new counts.
• Cloud Armor: a proposed per-IP rate limit for user writes, in preview mode under docs/migration/armor/proposed/, which also bounds how fast anyone can create unauthenticated users. An operator applies it.
• Logging: an unauthenticated user's ID is the browser's credential, so it is never logged; signed-out events log only user_type.
• Observability: saved-work requests are logged as Supabase traffic with no compute labels. Unexpected 500s and persistence 503s are logged with the error type and frame locations, never the message. Responses use lenient read shapes, so any stored row serializes.
• Consumers: app v3 is the only consumer (Stage 12A: Sign in to app v3 and save work to the account policyengine-app-v2#1252): it sends the unauthenticated header when signed out, calls the upgrade at sign-in, and opens report links through /v2/shared-reports. The legacy builder stays browser-only. /v2/user-households and /v2/user-simulations have no app consumer yet.
• Docs: docs/migration/stage-12a-user-login.md.

⚠️ Merge prerequisites

Remaining: none before merge. The Auth0 custom domain auth.policyengine.org is verified and serves issuer https://auth.policyengine.org/, which matches staging's PUBLIC_API_AUTH_ISSUER.

Done (2026-10-15):

1. GitHub variables.
   • staging: PUBLIC_API_AUTH_REQUIRED=1, PUBLIC_API_AUTH_ISSUER=https://auth.policyengine.org/ (final: identities are keyed by (issuer, subject), so it must not change after the first sign-in), PUBLIC_API_AUTH_AUDIENCE=https://api.policyengine.org/.
   • production: PUBLIC_API_AUTH_REQUIRED=0. Signed-out saving works without Auth0 settings; the production issuer and audience are set at rollout, once the custom domain is verified and staging has passed.
2. Auth0. The audience is the existing "PolicyEngine API" (https://api.policyengine.org/, RS256), with offline access now on. A staging-only CI application ("PolicyEngine API staging CI", password-realm grant) and its test user live in a dedicated policyengine-staging-ci database connection with signup disabled, enabled only for that application. A password grant through the tenant domain returned a token for this audience.
3. Live staging test, GitHub staging environment. Variables STAGING_AUTH0_TEST_DOMAIN=auth.policyengine.org, STAGING_AUTH0_TEST_AUDIENCE=https://api.policyengine.org/ and STAGING_AUTH0_TEST_REALM=policyengine-staging-ci; secrets STAGING_AUTH0_TEST_CLIENT_ID, STAGING_AUTH0_TEST_CLIENT_SECRET, STAGING_AUTH0_TEST_USERNAME and STAGING_AUTH0_TEST_PASSWORD.
4. Account deletion, production only. "PolicyEngine API account deletion (production)" has only delete:users on the Management API. Its secret is version 1 of policyengine-api-auth0-management-client-secret in Secret Manager, readable by the production runtime account policyengine-api-cr-runtime only. production has AUTH0_MANAGEMENT_DOMAIN=policyengine.uk.auth0.com, AUTH0_MANAGEMENT_CLIENT_ID and the secret AUTH0_MANAGEMENT_CLIENT_SECRET_RESOURCE.
   • DELETE /v2/me reads the secret and gets the Management token before deleting anything, so a credential failure returns 503 with nothing deleted. A 401 from Auth0 refreshes the token and retries once.
   • These settings are production-only. The staging job doesn't pass them, and deploy validation fails if they are set outside production, because the staging live test deletes its own account and would otherwise delete the CI user from Auth0.
5. Qualification, run read-only as the migration role:
   • staging (kvvwlyzqrfpvrxinmlwt): user_simulation_associations and user_report_associations empty, user_policies 3, user_household_associations 0; outcome: ok.
   • production (kvrifaviwhzjztcbrfpy): all four tables empty; outcome: ok.
   • The seed workflow runs the same check before every upgrade.

Behaviour changes

• The two existing user routes now require authentication. /v2/user-policies and /v2/user-households previously accepted a caller-supplied user_id with no ownership check. Nothing in production calls them; app-v2 never did.
• PATCH /v2/user-households no longer accepts household_id. Repointing goes through legacy_household_id.
• Rows owned by the Stage 10 mirror (those with a legacy_user_policy_mappings entry) return 409 on PATCH and DELETE.
• Saved reports can be read by anyone holding their ID (/v2/shared-reports): their name, source note, provisions, budget window, validation pin and compared reform's name, never their owner.
• Status codes:
  • Validation errors return 422, matching existing v2 behaviour.
  • A body that isn't valid JSON gets 422 before authentication.
  • DELETE /v2/me returns 502 if the data was deleted but the Auth0 deletion then failed. A retry is safe.
• MeItem.email stays null unless an Auth0 Action adds email / email_verified to access tokens.

Rollback

Roll forward whenever possible. The schema still serves the previous revision, but that revision serves /v2/user-policies and /v2/user-households without authentication. Rolling the application back would let anyone list, read, rename, delete or create users' saved items.

• Before any user-owned record exists, an application rollback is safe and needs no database downgrade. Signed-out saving needs no configuration, so treat user-owned records as existing from the moment this revision serves traffic in an environment.

• After that, apply the rollback guard first:

  1. Apply docs/migration/armor/proposed/20261013-pol-api-lb-stage-12a-rollback-guard.yaml, a Cloud Armor rule that denies /v2/me* and /v2/user-* with 403.
  2. Set the service's ingress to internal-and-cloud-load-balancing, because *.run.app URLs bypass Cloud Armor.
  3. Shift traffic back.

  Saved work is unavailable while the guard holds. Signed-out flows and v1 routes are unaffected.

• No other route in the previous revision exposes Stage 12A data.

• A database downgrade is lossy. It fails while legacy-only rows or any simulation or report associations exist.

Testing

• After merging master at e76065e8 (ruff groups 1 and 2, and the push.yml steps moved into scripts) and updating the bundle to 6.4.1: the make test suite (tests/to_refactor, tests/unit, tests/contract and the budget-window dedupe test) has 2769 passed, both before and after the bundle update. ruff format --check, ruff check with master's rule families, mypy (167 files), the quality guards (migration contracts, rules registry, CI scripts) and the rules tool's tests (74) pass. The migration contract docs are unchanged by the exporter.
• The cross-database integration job, run locally on 6.4.1 against disposable Postgres 17, MySQL 8.4 and Redis 7.2: all seven test steps pass (83 tests). This includes the installed-catalog test that failed Update PolicyEngine bundle to 6.4.1 #3939, the production-scale catalog publication qualification, and the identity persistence tests. alembic -c alembic-v2.ini check is clean at head fbb7488ebe8a.
• Against a disposable Postgres 16 after adding budget_window_years: the v2 Alembic lifecycle tests, then test_v2_user_identity_persistence.py at head (21 passed together, including a test that the CHECK rejects windows of 1 and 76 years). The persistence tests cover concurrent first signed-out writes making one user, concurrent upgrades converting once, two browsers upgrading at once (one converts, one merges), a merge moving all four association kinds with links intact, signed-out writes racing a conversion, and a standard user named in the header never being locked. The policy and household persistence tests passed earlier (11).
• Cross-database integration against disposable Postgres 17 and MySQL 8.4: 74 passed on the first version. After the review fixes, the v2 Alembic lifecycle tests (38 passed) and the policy, household and identity persistence tests (51 passed) were rerun. This covers:
  • the v1 and v2 Alembic lifecycles, including a test that dormant rows make the upgrade fail;
  • the Stage 10/11 persistence and mirror tests;
  • the new test_v2_user_identity_persistence.py: 8 concurrent first sign-ins create one user, account deletion cascades, and qualification is read-only;
  • a test that reflects the previous revision's schema, upgrades to head, and runs insert, read, update and delete through the previous revision's tables on all 38 tables it maps, plus a user deletion cascade.
• A 401 test covers all 25 /v2/me* and /v2/user-* operations, read from the built app's OpenAPI.
• After adding /v2/shared-reports: tests/unit/v2, tests/contract, and the migration artifact, flag, logging and ASGI tests have 971 passed. New tests cover the credential-free read with the private fields left out, the owner read still needing a credential, 404, query and ID validation, its route group, its OpenAPI operation (no security), and the service reading another user's report with its compared reform's name (null once that reform is deleted). mypy, ruff and the quality guards pass; the migration contract docs are regenerated.
• alembic -c alembic-v2.ini check is clean; a fresh upgrade, then downgrade -1 and re-upgrade, all succeed. The revision was regenerated with autogenerate under the same ID.
• ruff format --check, ruff check and mypy are clean.
• The live staging test (now also covering a signed-out write, a conversion and a merge) has not been run; it needs this PR deployed to staging and the custom domain verified.

openspec amendments (local-only specs, not in this PR)

• v2-user-policy-associations:
  • the owner comes from the token, and user_id is removed;
  • 401 / 403 / 404 / 503 semantics;
  • legacy_policy_id, a nullable policy_id with a CHECK, baseline, provenance;
  • repointing, and 409 on rows owned by the Stage 10 mirror;
  • the Stage 10 projection sets legacy_policy_id.
• v2-user-household-associations: legacy references, owner scoping, and household_id is no longer patchable.
• New capabilities:
  • user_identities provisioning;
  • v1-anchored simulation and report associations;
  • user types, the unauthenticated principal and the upgrade;
  • /v2/me with export and deletion order;
  • the contract guard that requires Authorization or the unauthenticated header on user-owned routes.

Related

• Simulation Entrypoint permissions: Stage 12A: Check Auth0 permissions on Simulation Entrypoint routes policyengine-sim-api#758
• app v3 sign-in and account-backed stores: Stage 12A: Sign in to app v3 and save work to the account policyengine-app-v2#1252

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 92.34098% with 177 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.15%. Comparing base (e93da19) to head (77dc183).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
policyengine_api/fastapi_routes/dependencies.py	43.33%	19 Missing and 15 partials ⚠️
policyengine_api/libs/auth0_management.py	88.28%	10 Missing and 5 partials ⚠️
policyengine_api/libs/user_auth.py	91.47%	6 Missing and 5 partials ⚠️
...ata/v2/user_association_migration_qualification.py	91.11%	5 Missing and 3 partials ⚠️
...icyengine_api/services/v2/user_reports/services.py	87.30%	6 Missing and 2 partials ⚠️
policyengine_api/services/v2/user_reports/types.py	94.40%	3 Missing and 5 partials ⚠️
...ngine_api/fastapi_routes/v2/user_reports/routes.py	89.47%	5 Missing and 1 partial ⚠️
...licyengine_api/fastapi_routes/v2/user_resources.py	91.04%	5 Missing and 1 partial ⚠️
...e_api/fastapi_routes/v2/user_simulations/routes.py	89.47%	5 Missing and 1 partial ⚠️
...es/v2/user_households/database_connectors/reads.py	58.33%	2 Missing and 3 partials ⚠️
... and 30 more

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3891       +/-   ##
===========================================
+ Coverage   45.25%   88.15%   +42.90%     
===========================================
  Files         183      255       +72     
  Lines       11085    14349     +3264     
  Branches     1907     2435      +528     
===========================================
+ Hits         5016    12649     +7633     
+ Misses       5647     1031     -4616     
- Partials      422      669      +247     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.