Skip to content

Comet Native Executors: range-partition sorts on float, nested and expression keys #7366

Open

Description

@andygrove

What is the problem the feature request solves?

On a Comet Native Executor cluster (#7289), the scheduler range-partitions a range exchange by split points it samples, so a global sort runs in parallel. It does so only when every sort key is a column whose type Arrow orders as Spark does: integers, decimals, dates, timestamps, strings, binary and booleans. A sort on a float or double, on a nested type, or on an expression still runs in one task.

The range shuffle compares raw values. Comet's sort normalizes float keys first (NormalizeNaNAndZero, NormalizeNestedFloats), so a NaN with its sign bit set, which x86 produces for 0.0 / 0.0, would land in the lowest range but sort last.

Describe the potential solution

Route rows by the same normalized expressions Comet's sort uses, built with PhysicalPlanner::create_sort_expr as Comet's own range shuffle does, instead of by raw columns. That needs try_encode_expr and try_decode_expr for those expressions in the cluster's CometPhysicalCodec, and the sample has to be taken of the same expressions. It would also cover sort keys that are expressions rather than columns.

Additional context

The check is orders_like_spark in native/native-executors/src/assembly.rs. It needs confirming that Ballista's range shuffle writer accepts sort expressions that are not columns.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions