Explore options for accelerating InMemoryTableScanExec #2391
Description
What is the problem the feature request solves?
In queries with InMemoryTableScanExec, we have to perform a CometColumnarToRowExec before this operator and then we perform a CometSparkRowToColumnar after the operator.
Here is an example plan from the test SparkToColumnar over InMemoryTableScanExec.
*(1) CometColumnarToRow
+- !CometHashAggregate [key#304L, count#371L], Final, [key#304L], [count(1)]
+- !CometHashAggregate [key#304L], Partial, [key#304L], [partial_count(1)]
+- CometSparkRowToColumnar
+- Scan In-memory table abc [key#304L]
+- InMemoryRelation [key#304L, value#305L, (key + 1)#308L], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(2) CometColumnarToRow
+- CometProject [key#6L, value#7L, (key + 1)#10L], [id#0L AS key#6L, (id#0L % 8) AS value#7L, (id#0L + 1) AS (key + 1)#10L]
+- CometSparkRowToColumnar
+- *(1) Range (0, 1000, step=1, splits=5)
Describe the potential solution
No response
Additional context
No response
Activity
andygrove commentedon Sep 14, 2025
Spark's InMemoryTableScanExec can support columnar data (in Spark format, not in Arrow format). I wonder if we can leverage this rather than replacing the operator.
[-]Implement CometInMemoryTableScanExec[/-][+]Explore options for accelerating InMemoryTableScanExec[/+]andygrove commentedon Sep 14, 2025
It seems possible to either use
InMemoryTableScanExecdirectly, or copy the code into a newCometInMemoryTableScanExecif needed, and then provide a customCachedBatchSerializerserializer inCachedRDDBuilderwithin theInMemoryRelation.pchintar commentedon Jun 4, 2026
Hi @andygrove I've just opened a PR titled Add Native Support for In-Memory Cache for this issue.
My implementation follows the
CachedBatchSerializerapproach discussed above. It adds a custom cache serializer together withCometInMemoryTableScanExec, allowing cached data to be written and read in a Comet-compatible format without any extra conversion steps. I've also added diverse tests to check my new configuration flag behavior as thoroughly as possible.So, could you kindly pls check it out?
docs: link tracking issues for operator acceleration gaps