Add 1940 US Census dataset preprocessing pipeline and schema to dpsynth/research/datasets/census/. - #119
Open
copybara-service[bot] wants to merge 1 commit into
Open
Add 1940 US Census dataset preprocessing pipeline and schema to dpsynth/research/datasets/census/.#119copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
copybara-service
Bot
force-pushed
the
cl/960942812
branch
2 times, most recently
from
August 7, 2026 17:16
f842f0a to
df42709
Compare
copybara-service
Bot
changed the title
copybara-service
Bot
force-pushed
the
cl/960942812
branch
2 times, most recently
from
August 7, 2026 20:58
ba15588 to
904c674
Compare
…th/research/datasets/census/. Includes the end-to-end data preparation and benchmark evaluation pipeline for the 1940 US Decennial Census dataset (~137 million records, 147 attributes): - domain.yaml & codebook.json: checked-in public schema and value label codebook. - column_types.py: authoritative 147-attribute layout, classifications, and ranges. - parse_sas.py: extracts categorical value labels from IPUMS SAS metadata. - build_domain.py: generates public, data-independent domain.yaml and codebook.json. - convert_to_parquet.py: fast vectorized streaming conversion to sharded Parquet. - shuffle_parquet.py: two-phase uniform random scatter/gather shuffler. - precompute_marginals.py: single-pass 2-way evaluation marginals via mbi JIT bincount (saving the full CliqueVector JAX pytree directly via mbi.save into marginals.npz). - README.md: documentation for data access pathways and benchmark usage. Uses etils.epath throughout for clean, unified filesystem access across local and cloud storage backends without Copybara strips. PiperOrigin-RevId: 960942812
copybara-service
Bot
force-pushed
the
cl/960942812
branch
from
August 7, 2026 21:14
904c674 to
595c96b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add 1940 US Census dataset preprocessing pipeline and schema to dpsynth/research/datasets/census/.
Includes the end-to-end data preparation and benchmark evaluation pipeline for the
1940 US Decennial Census dataset (~137 million records, 147 attributes):
(saving the full CliqueVector JAX pytree directly via mbi.save into marginals.npz).
Uses etils.epath throughout for clean, unified filesystem access across local and
cloud storage backends without Copybara strips.