You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Comet Native Executors: range-partition sorts on float, nested and expression keys #7366
On a Comet Native Executor cluster (#7289), the scheduler range-partitions a range exchange by split points it samples, so a global sort runs in parallel. It does so only when every sort key is a column whose type Arrow orders as Spark does: integers, decimals, dates, timestamps, strings, binary and booleans. A sort on a float or double, on a nested type, or on an expression still runs in one task.
The range shuffle compares raw values. Comet's sort normalizes float keys first (NormalizeNaNAndZero, NormalizeNestedFloats), so a NaN with its sign bit set, which x86 produces for 0.0 / 0.0, would land in the lowest range but sort last.
Describe the potential solution
Route rows by the same normalized expressions Comet's sort uses, built with PhysicalPlanner::create_sort_expr as Comet's own range shuffle does, instead of by raw columns. That needs try_encode_expr and try_decode_expr for those expressions in the cluster's CometPhysicalCodec, and the sample has to be taken of the same expressions. It would also cover sort keys that are expressions rather than columns.
Additional context
The check is orders_like_spark in native/native-executors/src/assembly.rs. It needs confirming that Ballista's range shuffle writer accepts sort expressions that are not columns.
Activity