Skip to content

Carry industry and worked-last-year on every US person record (ASEC restore + ACS crosswalk) - #1039

Merged
MaxGhenis merged 13 commits into
mainfrom
work-experience-inputs-719-v2
Oct 24, 2026
Merged

MaxGhenis merged 13 commits into
mainfrom
work-experience-inputs-719-v2

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #719. Replaces #745 (closed: it predated the #720 restoration and the ACS release join, and could not rebase across 1,600 commits). Delivers #922 item 5 at the code level.

What

Every released US person record carries three new PolicyEngine-US inputs (engine side: policyengine-us#9311, in the locked 2.2.1), next to detailed_occupation_recode:

column meaning ASEC ACS 2024
detailed_industry_recode industry of the longest job last year, 23 groups WEIND INDP → ACS_INDP_TO_WEIND
major_industry_recode same job, 15 groups WEMIND WEIND_TO_WEMIND of the above
worked_last_year worked at any time last year WKSWORK > 0 WKWN > 0

The design reuses existing boundaries rather than adding a stage:

  • ASEC (and PUF-support clones): asec_census_person_columns (Pooled ASEC 2022/2023 vintages lack the NOW_* coverage recodes: reported Medicaid at interview thins to 24.6M vs ~58M survey under 65 on the certified artifact #720) now also restores WEIND/WEMIND for every vintage from its pinned Census person member by exact PERIDNUM join before pooling. No H5 vintage carries them, so they are recorded as beyond the 8/23 offline-fix receipt.
  • ACS: the release predictor join (acs_release_predictors, crosswalk v2, digest re-pinned) maps native INDP/WKWN onto the same three raw columns, exactly as it maps OCCP onto POCCU2.
  • Every row: the release-time org_wages carry emits the three columns beside occupation; source_stages.json declares them; they join the post-reference required release inputs (coverage manifest: 170 required).

Why WEIND/WEMIND and WKSWORK > 0: they describe the longest job last year, the same job as POCCU2 and the same year as the income; the survey-week A_DTIND/A_MJIND would describe the current job. In all three pinned members WEIND carries a worker code exactly where WKSWORK > 0, while WORKYN = 1 misses ~600 allocation rows per year.

The ACS crosswalk takes each civilian code's group from the pinned 2025 ASEC member (same 2022 Census code list as ACS 2024 INDP; INDUSTRY determines WEIND exactly there) and reviews the eight ACS-only codes (seven Armed Forces branches → 22, as CPS's single 9890; 9920 → did-not-work 23). Blank INDP mirrors the OCCP rule (0 below 16, age 15 kept out of universe; 23 from 16, only ESR 6 may be blank).

Invariants (stated and executed)

  1. Nesting: WEMIND == WEIND_TO_WEMIND[WEIND] — checked in the ASEC restoration, by construction in the ACS join, and in the org_wages carry and gate.
  2. Forward worker identity: WKSWORK > 0 ⇒ WEIND in 1–22 — all three sites, every spine.
  3. Converse (ASEC only): WEIND in 1–22 ⇒ WKSWORK > 0 — checked only in the restoration, where rows are known to be ASEC. Intended asymmetry: ACS INDP covers a job in the past five years, so the release carry does not check it.
  4. Round trip: the carry emits its inputs unchanged; worked_last_year == (WKSWORK > 0).
  5. ACS universe: below 16 all outputs 0; blank INDP from 16 → 23; observed INDP → table; WKSWORK = WKWN or 0.

test_us_work_experience_properties.py checks all five with Hypothesis (200 examples each) plus a differential test running the same generated rows through the ASEC restoration and the release carry, requiring each to accept exactly when its stated identities hold. The differential draws both independent codes and nested industry pairs with any weeks, so each worker check decides some cases; deleting any of the three checks fails it.

Real-data evidence (in docs/us-work-experience-columns.md)

  • ASEC pppub23/24/25.csv: nonzero INDUSTRY→WEIND/WEMIND functional (263/263/259 codes; INDUSTRY = 0 covers both WEIND 0 and 23), WEIND→WEMIND the same function in all three, zero universe violations; weighted nonzero-industry shares 0.822–0.824, worked 0.520–0.523.
  • All 3,422,888 ACS 2024 persons (the release-pinned archive) through the join: no refusal fires, every INDP code mapped, WKWN>0 iff WKL=1, the 264,806 people with an industry but no weeks equal the WKL=2 count exactly; weighted nonzero-industry 0.812, worked 0.538. Gate bands bracket both spines.

Pins moved (all explained)

ACS crosswalk digest (v2); frozen source_stages.json digest; spec-engine inventory (source-manifest, resource-semantics, full-checkpoint and tail digests; tail control fields 934 → 937 = the three tail-contract columns); field-usage counts (+11 configuration fields = 3 derive inputs + 3 outputs + 2 nonnegative + 3 tail-contract) and the affected pointer claims; coverage evidence JSON regenerated by its tool.

Repinner (tools/repin_us_spec_engine_contracts.py)

The documented re-pin sequence regenerates every F0 pin. Three bugs were fixed during review:

  • Calibration projection. The tool hashed the compiled bundle, whose loader writes 5.0 as 5. It now builds the contract exactly as the semantics-test fixture does (pin aa65317d…). Stripping this PR's three tail names reproduces main's 05cb099e… from the raw contract, but not from the bundle.

  • Count pins found by old value. Pins were located by their old value, taken from the evidence JSON or the module's own pins, so a merge that moved one copy hid a stale other while --check passed. That happened in a trial merge with Add 2023 county-equivalent geography to US household ladders #1376. Every field-usage and coverage-test count is now refreshed by its asserted expression or key:

    • totals, claim count, inventory counts and the breakdown dictionaries;
    • the "<N> unclaimed normative field" expectation.

    Planting stale values makes --check fail and the tool restore them.

Merge with #1376

Merged after #1376, #1377 and #1378. The combined ledger has 42,229 configuration fields (main +12: 11 here, 1 from #1376) and passes 41/41 inventory checks. The pool tool's runtime graph has 80 modules. The bundle spec hash is 53fb8de6….

Not in this PR

Released files carry the columns once the pooled base and multispine pool are rebuilt and a release runs (#922 sequencing step 1). Until then the release input-coverage gate is red on these three columns by design (notes in the manifest).

axiom: n/a: data pipeline only (no policy rule changed)

🤖 Generated with Claude Code

MaxGhenis and others added 9 commits September 26, 2026 07:38
…year

The #720 restoration now appends the ASEC longest-job industry recodes
WEIND (detailed, 0-23) and WEMIND (major, 0-15) to every vintage from its
pinned Census person member. No H5 vintage carries them, so they sit
outside the 2026-08-23 offline-fix receipt and are recorded as such. After
restoring, each vintage is checked against two identities that hold in
all three members: WEMIND is the major group of WEIND (the pinned
WEIND_TO_WEMIND function), and WEIND carries a worker code (1-22) exactly
where WKSWORK > 0.

The release-time org_wages carry reads them next to POCCU2 and emits
detailed_industry_recode, major_industry_recode and worked_last_year
(WKSWORK > 0), refusing missing or out-of-range codes and checking the
halves of the identities that hold on every spine. Gate bands follow the
measured shares.

Part of #719.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release predictor join now also materializes WEIND, WEMIND and
WKSWORK for ACS-spine people, so the org_wages carry gives every released
person record an industry and a worked-last-year indicator, the same way
ACS OCCP already reaches detailed_occupation_recode through POCCU2.

ACS_INDP_TO_WEIND (267 codes) reads every civilian code's group from the
pinned 2025 ASEC member, where INDUSTRY determines WEIND exactly and the
2022 Census code list matches ACS 2024 INDP. The eight ACS-only codes are
reviewed: the seven Armed Forces branch codes collapse to CPS 9890's group
22, and 9920 is the did-not-work group 23. Blank INDP mirrors the OCCP
treatment: 0 below age 16 (age 15 keeps the out-of-universe sentinel), 23
from age 16 where only ESR 6 may be blank. WEMIND follows WEIND by the
pinned nesting; WKSWORK is WKWN (weeks in the past 12 months) or 0.
Positive weeks without a worker industry code is refused.

Crosswalk version 2; digest re-pinned.

Part of #719.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…release

source_stages.json's org_wages stage names WEIND, WEMIND and WKSWORK as
derive_from_asec inputs and detailed_industry_recode,
major_industry_recode and worked_last_year as outputs (the recodes
nonnegative). The three join the post-reference required release inputs
with notes that today's published defaults lack them by design, and the
coverage manifest regenerates (170 required). The bundle regenerates
from the manifest authority (sources.yaml, calibration.yaml tail
contracts) with the frozen source_stages digest re-pinned.

Adds docs/us-work-experience-columns.md (sources, crosswalk, identities,
real-data evidence on the pinned ASEC members and all 3,422,888 ACS 2024
persons) and the changelog fragment.

Part of #719.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hypothesis property tests state and check the #719 invariants over
generated inputs: WEMIND nests WEIND, positive weeks carry a worker
industry code, the ASEC-only converse, the release carry's round trip,
and the ACS crosswalk's universe rules. A differential test runs the
same rows through the ASEC restoration and the release carry and
requires each to accept exactly when its stated identities hold (the
restoration is the stricter site by design).

The spec-engine inventory vector moves with the org_wages manifest
change: source-manifest, resource-semantics, full-checkpoint and tail
digests, and 937 tail control fields (+3 tail-contract columns).

Part of #719.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ields

+11 configuration/authored fields (3 derive inputs, 3 outputs, 2
nonnegative outputs, 3 tail-contract columns); the calibration (312 ->
315) and source_stages (1,712 -> 1,720) pointer claims and the pointer
inventory digest move with them. Coverage evidence regenerated by
tools/spec_engine_coverage.py: 42,191/42,191 fields, 41/41 inventory
checks.

Part of #719.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve the exact ASEC industry restore and shared ACS crosswalk across
current fiscal and native ACS paths. Require complete canonical job inputs
on preserved donors and final staging artifacts. Regenerate source bundles,
input coverage, and spec-engine attestations with reproducible tooling.

Retain main's stage/graph architecture, dependency locks, and test layout.
Brings in main through 1eb0b1b. The us-f0-coverage.json conflict takes
main's version. The L0 export fixtures gain the job fields the work-experience
columns need, and the import-graph pin moves to 77 modules.

Rebuilt from the cancelled lane's salvage (refs/subfleet-salvage/
work-experience-inputs-719-v2-20261024T011240Z-a1). The lane had resolved
this merge in a private git dir but was stopped before committing; the tree
is that working tree without the private git dir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…merge

The merge of main took main's sources.yaml, dropping this branch's
work-experience entries (WEIND, WEMIND, WKSWORK inputs; industry and
worked_last_year outputs; the stage note). Regenerating the bundle from
the constants restores them exactly, and the documented re-pin sequence
(tools/repin_us_spec_engine_contracts.py --legacy-only, the bundle
generator, the re-pin, tools/spec_engine_coverage.py) moves the field
counts to 42_228 configuration / 32_448 authored and re-pins the stage,
calibration, tail and checkpoint digests. Both --check modes pass and the
coverage tool reports 41/41 inventory checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repinner hashed the compiled bundle's calibration, but the loader
writes integral floats as integers (5.0 becomes 5), so its digest
(080ec479...) never matched the semantics test, which hashes the
constants builder's raw contract (aa65317d..., failing engine-free CI).
The repinner now builds the contract exactly as that fixture does, and a
new test holds its derivation to the committed pin. Stripping this
branch's three new tail names reproduces main's pin (05cb099e...) from
the raw contract, not from the bundle.

The work-experience differential test now also draws nested industry
pairs with any weeks, so the forward and converse worker checks decide
the outcome; removing either check in either site now fails it. The
docs and crosswalk comment say nonzero INDUSTRY determines WEIND: in the
pinned 2025 member INDUSTRY = 0 maps to both WEIND 0 and 23.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 24, 2026 06:11
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Round-2 delta review (GPT-6.1 Sol, Subfleet 20261024-015826-lwi-r2-1039): APPROVE at 558d5b6, no findings.

  • The repinner's fixture_calibration_contract() and the semantics-test fixture produce identical raw bytes, so the pin is aa65317d….
  • Stripping the three new tail names reproduces main's 05cb099e… from the raw contract and 40b952b7… from the loaded bundle.
  • Full --check passes across all 18 pin paths.
  • The new nested-pair property strategy kills all three worker-check mutants (minimal counterexample ([1], [1], [0])).
  • The docs now qualify nonzero INDUSTRY.

Merging on green CI; if #1376 lands first, this branch takes main and re-pins with the fixed tool.

MaxGhenis and others added 4 commits October 24, 2026 02:36
The repinner moved the field-usage tests' mode and generation-0 counts
by value transition, taking each old value from the committed coverage
JSON. A merge can break that: in a trial merge of #1376 into this branch,
git auto-merged #1376's compiler_semantic 27718 into the JSON while the
conflicted test kept 27_717. The tool then saw no transition, left the
stale literal, and --check passed while the test failed.

The five asserted count dictionaries (source, mode and generation-0
counts in the engine-free field-usage test; mode and generation-0 counts
in the coverage-tool test) are now rewritten by key from the ledger. The
keys must match the observed ledger exactly. Totals still move by value
transition from the module's own pins. On this branch --check is
unchanged. On the trial merge the tool now flags and fixes the stale
count, and both tests pass (21 passed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-3 review noted the scalar totals still moved by value transition
from the module's own pins, so a merge that had already moved
EXPECTED_CONFIGURATION_FIELD_COUNT could leave a stale 42_228 in a test
that --check would not see. The engine-free field-usage test and the
coverage-tool test are now refreshed only by asserted expression:

- `len(field_ledger.fields)` and the distinct-pointer count;
- the coverage report's field totals and claim count;
- the inventory item counts and the four asserted inventory counts;
- the keyed breakdowns.

Planting stale values in three of these makes --check fail and the tool
restore them. The value-transition helper, now unused, is removed with
its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1376 and this branch both moved the F0 spec pins and the pool tool's
import graph. The conflicts resolve as follows:

- agent-guide.md keeps both paragraphs, in the one running paragraph main
  uses.
- The pool build tool's runtime graph is 80 modules: this branch's two
  crosswalk modules plus #1376's county_vintage, puma_county_vintage and
  block_ladder_sources.
- Every other conflicted pin was regenerated by the documented sequence
  (repinner --legacy-only, bundle generator, repinner, coverage tool).
  The ledger is now 42,229 configuration fields, which is this branch's
  +11 plus #1376's +1, and coverage passes 41/41 inventory checks. The
  US bundle spec_sha256 is 53fb8de6.

The repinner's keyed rewrite caught #1376's compiler_semantic 27718,
which git had auto-merged into the coverage JSON but not into the
conflicted tests. A final --check passes. 885 targeted engine-free and
engine-contract tests pass, plus the pool tool's 7 constants-adapter
tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-4 review found that the identity test's `r"87 unclaimed normative
field"` regex has no refresh path. Every field has exactly one primary
claim, so dropping the selection claim leaves exactly its observed
expansion (87) unclaimed. The repinner now rewrites that one
expectation from the expansion it already computes. With a stale 86
planted, --check fails and the tool restores 87.

The main merge also dropped #1376's note that the puma_county_vintage
draw module joins the seed protocol's attested kernel modules; it is
restored above the seed hashes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Reviews since 558d5b6 (GPT-6.1 Sol via Subfleet), all APPROVE with no findings:

Merging when CI is green at 7ab3a77.

@MaxGhenis
MaxGhenis merged commit 395a70e into main Oct 24, 2026
10 checks passed
@MaxGhenis
MaxGhenis deleted the work-experience-inputs-719-v2 branch October 24, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface work-experience industry and worked-last-year person inputs (LWI MVP schema)

1 participant