You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Comet Native Executors: sample a range exchange without re-running the stages below it #7367
To range-partition an exchange, the Comet scheduler (#7289) runs a sampling job of its own: a SampleExec over the stage that feeds the exchange. That is a separate Ballista job, so it runs that stage and every stage below it, then the query runs them all again. Every further range exchange above them adds another run. Spark's RangePartitioner also re-runs the child, but it reuses the shuffle outputs below it.
The sampling jobs also run one after another while the query is being planned, even for range exchanges whose inputs don't depend on each other.
Describe the potential solution
Options, roughly from least to most work:
Sample range exchanges whose inputs are independent concurrently.
Sample from the materialized shuffle output below the exchange, so only the producing stage runs twice.
Find the split points at run time, as Ballista's adaptive planner does for window functions: RuntimeStatsExec keeps a quantile sketch and UnorderedRangeRepartitionExec cuts by it. That covers only the first sort key today, so it would need compound keys. The driver-side sampler, SampleExec and its codec would then go away.
Additional context
The sampling is range_exchange in native/native-executors/src/assembly.rs and collect in src/frontend.rs.
No longer applies: Comet Native Executors no longer sample range exchanges. As of 8b5f114 in #7289, a stage that reads a range exchange sorts the producer's partitions as they are and merges them with a SortPreservingMergeExec, as DataFusion plans a global sort, so there's no sampling job to re-run the stages below the exchange.
Activity
andygrove commentedon Nov 1, 2026
No longer applies: Comet Native Executors no longer sample range exchanges. As of 8b5f114 in #7289, a stage that reads a range exchange sorts the producer's partitions as they are and merges them with a SortPreservingMergeExec, as DataFusion plans a global sort, so there's no sampling job to re-run the stages below the exchange.