Skip to content

PyArrow: Collect NaN value counts when writing Parquet files - #4131

Open
DG47 wants to merge 1 commit into
apache:mainfrom
DG47:fix/collect-nan-value-counts
Open

DG47 wants to merge 1 commit into
apache:mainfrom
DG47:fix/collect-nan-value-counts

Conversation

@DG47

@DG47 DG47 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Closes #4130

Rationale for this change

Parquet statistics do not track NaN values, so data files written by PyIceberg never carry nan_value_counts. The strict metrics evaluator reads a missing NaN count as "no NaN values" (as Java does, since Java's Parquet writers always record these counts). That means Table.delete("x < 5.0") drops a file containing [1.0, NaN] as a whole, so the NaN row is lost even though it does not match the filter.

This change counts NaN values of float and double columns in ParquetFormatWriter.write, including nested struct fields, list elements and map values. It skips columns whose metrics mode is none, matching how the Java writers populate nan_value_counts. The counts are passed to data_file_statistics_from_parquet_metadata through a new optional argument. Files without float or double columns do no extra work.

Files that are already written keep their missing counts. Files registered through add_files are unchanged, since that path only has the Parquet footer.

Are these changes tested?

Yes.

  • tests/io/test_format_writers.py: NaN counts for top-level float and double columns, a struct field, list elements and map values across multiple write() calls, zero counts for columns without NaN, and no count for a column with metrics mode none.
  • tests/catalog/test_catalog_behaviors.py: delete("value < 5.0") on a file with [1.0, NaN] keeps the NaN row (fails on main for the memory and sql catalogs).
  • tests/integration/test_writes/test_writes.py::test_merge_manifests: updated the expected nan_value_counts from [] to [(6, 0), (7, 0)] for the float and double columns.

make test and make lint pass locally.

Are there any user-facing changes?

Data files written by PyIceberg now include nan_value_counts for float and double columns. This fixes the delete above and lets scans prune on NaN counts.

Parquet statistics do not track NaN values, so data files written by
PyIceberg never carried nan_value_counts. The strict metrics evaluator
reads a missing NaN count as "no NaN values", so a delete such as
`value < 5.0` dropped whole files and lost NaN rows that do not match
the filter. Count NaN values of float and double columns while writing,
as the Java Parquet writers do.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data files written by PyIceberg have no nan_value_counts, so delete() drops NaN rows that do not match the filter

1 participant