Skip to content

fix: [branch-1.1] treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5, 4.1.4 and 4.2.1 #7265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
snmvaughan wants to merge 2 commits into
base: branch-1.1
Choose a base branch
from
+332 −50
Open

fix: [branch-1.1] treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5, 4.1.4 and 4.2.1 #7265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0f2847f
fix: treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5…
andygrove Oct 20, 2026
c3b827d
Merge branch-1.1 into backport-6989-branch-1.1
snmvaughan Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/source/user-guide/latest/compatibility/floating-point.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,3 +120,15 @@ any version other than 4.2.0 falls back even if it includes SPARK-54918. A vendo
reports 4.2.0 but includes SPARK-59602 still runs natively; set
`spark.comet.expression.ArrayDistinct.enabled=false` and
`spark.comet.expression.ArrayUnion.enabled=false` on such a build.

## `arrays_overlap`

`arrays_overlap` runs natively on every Spark version and follows that version's rule for `FLOAT`
and `DOUBLE` elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat `-0.0` and `0.0` as one
value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is
one value on every version, and nested arrays and structs compare their elements in Spark's order,
in which the two zeros are equal.

Comet chooses the rule by the Spark version number. On a vendor build that reports an older
version but includes SPARK-59602, Comet returns `false` for two flat arrays whose only common value
is a zero of opposite sign, where Spark returns `true`.
Loading
Loading