Comet's explode is slower than Spark when the element is a struct, worst on nested rows

[andygrove]

What is the problem the feature request solves?

CometExplodeExec is slower than Spark's GenerateExec whenever the exploded element is a struct, and the gap is largest on the deeply nested, wide row shapes that motivate a generator in the first place.

Measured with CometExplodeBenchmark (added in #5381) on an Apple M3 Max, JDK 17, local[1], Spark 4.1 / Scala 2.13, release native build. These are whole-query times including the Parquet scan and the counting aggregate, so they understate the operator's share (Best ms):

Case	Spark	Comet	Relative
explode array<struct<a bigint, b string>>[10]	61	86	0.7X
explode array<struct>[4] at depth 1	69	156	0.4X
explode array<struct>[4] at depth 8	84	167	0.5X
explode array<struct>[4] at depth 8, then its inner array	96	181	0.5X

For contrast, over array<bigint> at fan-out 10 Comet is 1.2X faster, and at fan-out 2 it is 1.5X faster. So this is not a general generator problem — it tracks the element type.

The nesting itself is cheap. Going from depth 1 to depth 8 costs Comet 11ms and Spark 15ms, and nested schema pruning narrows the scan to exactly the events path either way, so both depths read the same leaf payload. What costs is the element being a wide struct: the depth-8 element here is struct<platform string, entries array<struct<type string, ts bigint, page string, source string>>>, and the plain array<struct<bigint, string>> case shows the same direction at a smaller magnitude.

The nested cases are the most reproducible numbers in the whole benchmark — across five runs Comet's arm varied by a few ms while Spark's swung under machine contention.

Describe the potential solution

ExplodeExec is Comet's fork of DataFusion's UnnestExec, and it gathers the unnested column through a take-index buffer sized to the output. For a struct element that gather recurses into every field, which is where the cost concentrates.

#5667 is in flight and replaces the gather with a slice of the child when the take indices would be a contiguous run, which covers plain explode and both columns of posexplode. Its Rust-level benchmark shows explode_element_type/struct down 91%, so it should close a large part of this. This issue is for whatever remains after that lands, and for the cases #5667 deliberately does not reach:

• explode_outer / posexplode_outer still gather, because a NULL or empty row is padded with a NULL that no slice of the child contains.
• Carried passthrough columns are still a real gather.
• Whether a struct element benefits from slicing as much as a primitive one needs measuring rather than assuming, since ListArray::slice leaves values whole and a struct's field arrays are sliced independently.

Additional context

The benchmark's nested group deliberately counts the whole event struct rather than one of its fields, so nested schema pruning cannot narrow the exploded element and leave the case measuring a two-column gather.

A generator over a map cannot be measured against Comet at all yet — see #2837 — so the shape a real schema would use for this data (a map keyed by platform) is an array<struct<platform, entries>> in the benchmark.