Skip to content

Comet Native Executors: sample a range exchange without re-running the stages below it #7367

Closed as not planned

Description

@andygrove

What is the problem the feature request solves?

To range-partition an exchange, the Comet scheduler (#7289) runs a sampling job of its own: a SampleExec over the stage that feeds the exchange. That is a separate Ballista job, so it runs that stage and every stage below it, then the query runs them all again. Every further range exchange above them adds another run. Spark's RangePartitioner also re-runs the child, but it reuses the shuffle outputs below it.

The sampling jobs also run one after another while the query is being planned, even for range exchanges whose inputs don't depend on each other.

Describe the potential solution

Options, roughly from least to most work:

  • Sample range exchanges whose inputs are independent concurrently.
  • Sample from the materialized shuffle output below the exchange, so only the producing stage runs twice.
  • Find the split points at run time, as Ballista's adaptive planner does for window functions: RuntimeStatsExec keeps a quantile sketch and UnorderedRangeRepartitionExec cuts by it. That covers only the first sort key today, so it would need compound keys. The driver-side sampler, SampleExec and its codec would then go away.

Additional context

The sampling is range_exchange in native/native-executors/src/assembly.rs and collect in src/frontend.rs.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Activity

  1. andygrove commented on Nov 1, 2026

    @andygrove
    MemberAuthor

    No longer applies: Comet Native Executors no longer sample range exchanges. As of 8b5f114 in #7289, a stage that reads a range exchange sorts the producer's partitions as they are and merges them with a SortPreservingMergeExec, as DataFusion plans a global sort, so there's no sampling job to re-run the stages below the exchange.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions