Skip to content

testing - #2122

Open
niveditasing wants to merge 2 commits into
datacommonsorg:masterfrom
niveditasing:us_pep_sex_race_code_fix
Open

testing#2122
niveditasing wants to merge 2 commits into
datacommonsorg:masterfrom
niveditasing:us_pep_sex_race_code_fix

Conversation

@niveditasing

@niveditasing niveditasing commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

This PR includes

  1. Code fix:

    • Pandas Modernization: Replaced the deprecated delim_whitespace=True parameter with sep=r'\s+' across all year-specific modules (national, state, and county).
    • Robust Geo ID Formatting: Refactored county GEO ID processing in county_1990_2000.py to use a clean pandas chain: .astype(int).astype(str).str.zfill(5).
    • Safe Post-Processing: Added existence checks in postprocess.py to prevent FileNotFoundError when merging intermediate output files.
    • Safe Downloader Execution: Added a null check on _FILES_TO_DOWNLOAD in preprocess.py to prevent errors when there are no files to process.
  2. Added golden checks

  3. Testing results: https://storage.mtls.cloud.google.com/datcom-import-test/scripts/us_census/pep/us_pep_sexrace/USCensusPEP_By_Sex_Race/2026_07_31T01_39_54_500027_07_00/input0/validation/validation_output.csv

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the US Census PEP sex/race processing scripts by replacing the deprecated delim_whitespace=True parameter with sep=r'\s+' in pandas read operations, adding validation configurations, and introducing safety checks for missing files and download lists. Feedback on these changes suggests raising a FileNotFoundError instead of logging a warning when intermediate files are missing in postprocess.py to prevent silent data loss. Additionally, it is recommended to use .loc for column assignment in county_1990_2000.py to avoid potential SettingWithCopyWarning issues.

Comment thread scripts/us_census/pep/us_pep_sexrace/postprocess.py Outdated
Comment thread scripts/us_census/pep/us_pep_sexrace/postprocess.py Outdated
Comment thread scripts/us_census/pep/us_pep_sexrace/postprocess.py Outdated
Comment thread scripts/us_census/pep/us_pep_sexrace/postprocess.py Outdated
Comment thread scripts/us_census/pep/us_pep_sexrace/county/county_1990_2000.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant