Repository navigation
Conversation
Decline FLOAT and DOUBLE range comparisons and zero or NaN literals. Spark treats -0.0 as equal to 0.0 and sorts every NaN above all numbers, while iceberg-rust compares by IEEE 754 total order, and its row-group pruning trusts statistics that can omit NaN (apache/iceberg-rust#3494). Decline equality under NOT on nullable columns. Iceberg passes `<=>` as `=`, so the native reader dropped NULL rows that Spark keeps for `NOT (id <=> 1)`. On columns whose type changed in the schema history, push only equality outside NOT and null checks, since iceberg-rust compares older, narrower files incorrectly (apache/iceberg-rust#3122). An incomplete history treats every BIGINT, DOUBLE and DECIMAL column as promoted. Closes apache#7136. Closes apache#7137. Part of apache#7138.
github-actions
Bot
added
bug
Spark rewrites `b = false` on a nullable boolean as `b IS NOT NULL AND NOT b`, which Iceberg passes as `not_null(b) AND not(b = true)`. Declining equality under NOT on a nullable column left that residual unpushed, so CometFuzzIcebergSuite found no residual for its boolean column. A NULL value fails a NOT_NULL conjunct, and with it the whole AND, in Spark and in the native reader alike. Treat a column checked that way as non-nullable within the AND, so the null-safe concern no longer applies.
…cans Extract referencedColumn so the residual converter and notNullColumns share the bare-reference check. Share the residual predicate loop of the new native tests, and assert that `id != 1` and `b = false` are still pushed while `NOT (id <=> 1)` and `b IS NOT TRUE` are not. CometFuzzIcebergSuite samples its equality literal from the data. Skip zero as well as NaN, since floating-point comparisons with zero are no longer pushed. Note in the Iceberg guide that `!=` is not affected by the negated-equality rule.
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Each case the serde now declines matches a real gap between Spark and the pinned iceberg-rust, which also installs the residual as a Parquet RowFilter, so a looser residual is not safe here as it is for Iceberg Java. Three points on the tests, a reference and a helper.
| * unpushed. | ||
| */ | ||
| private def checkResidualPredicates(table: String, predicates: Seq[String]): Unit = | ||
| for (predicate <- predicates; prefix <- Seq("", "l IS NOT NULL AND ")) { |
There was a problem hiding this comment.
With the prefix, the residual never converts. The AND branch needs both sides, and the null check on the array column l is declined by isComplexType, so each prefixed query reruns the unprefixed one with nothing pushed. The docstring says as much. That doubles the query count of the three new tests without reaching the code this PR changes. The prefix starts to matter with partial conjuncts in #6847. Could the prefix and the l column move there, so this PR and its backports only run the cases they fix?
| // Spark sorts every NaN above all numbers. The native row filter sorts negative NaNs | ||
| // below them, and iceberg-rust prunes row groups on min/max statistics that can | ||
| // leave NaN out, or hold it as both when parquet-mr 1.9 or earlier starts a row | ||
| // group with NaN (apache/iceberg-rust#3494). |
There was a problem hiding this comment.
apache/iceberg-rust#3494 was closed on 2026-10-23 by apache/iceberg-rust#3502 ("treat NaN row group bounds as might-match for inequalities"), which is 54 commits past the pinned 1f3bc34. Could the comment name #3502, so the next pin bump knows which part of this exclusion it can revisit? The fix is for row-group bounds, so the page index side would still need checking then.
| def promotedColumns(table: Any, schema: Any): Set[String] = { | ||
| import scala.jdk.CollectionConverters._ | ||
| val findField = getMethod(schema.getClass, "findField", classOf[Int]) | ||
| def typeOf(field: Any): String = getMethod(field.getClass, "type").invoke(field).toString |
There was a problem hiding this comment.
fieldTypeOf at line 1416 already does this lookup, so this can be fieldTypeOf(field).toString, the way this method already reuses fieldIdOf.
Which issue does this PR close?
Closes #7136.
Closes #7137.
Part of #7138.
Rationale for this change
The native Iceberg reader uses residuals to prune reads and filter rows. Comparisons that differ from Spark can discard matching rows before the post-scan filter runs. These correctness fixes were split from #6847 so they can land and be backported independently.
What changes are included in this PR?
Leave unsafe residuals to the post-scan filter:
!=and IN predicates remain eligible.b = false.Unrecorded promotions and imported files remain a known limitation under #7138. The Iceberg guide documents the exclusions.
How are these changes tested?
Local serde, native-scan and fuzz suites:
The Spark 4.1 run preceded only the test-version correction below; the Spark 3.5 results are from the final commit.
The Spark 3.5 run exposed a fixture assumption: Iceberg 1.8 cleans expired partition specs but retains schemas. The schema-cleanup portion now runs on Iceberg 1.9+, where that cleanup is supported. The test failed before this correction and passed afterward; ordinary promoted-column checks still run on 1.8.
Iceberg 1.11 / Spark 4.1 upstream shard-2 passed locally: 1,794 passed, 528 skipped, no failures or errors. Other upstream Iceberg versions and Spark 3.4/4.0/4.2 runtime suites were not run locally.