Repository navigation
Plan and execute common RowFn batches in one dispatch - #10675
connortsui20 wants to merge 4 commits into
Conversation
Merging this PR will regress 8 benchmarks3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Keep the owned decoder as the required extension point and default borrowed decoding to it. Native primitive inputs share their canonical buffers directly. Retain decoded storage after the input handle is dropped. Extracted from #9979. Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Inspect borrowed inputs once for lengths, dtypes, constant classification, and strict validity. Pair collected inputs with a separate execution plan while retaining the existing dispatch and retry paths. Extracted from #9979. Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Let the packed Boolean collectors consume the batch validity when constructing their result. Keep custom output validation before masking and attach validity directly to other canonical outputs. Extracted from #9979. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
55d9af7 to
178dcd8
Compare
Finish eligible batches in the planning visit and retain the planned retry and valid-row paths for the remaining partially valid batches. Decode direct Boolean inputs through private borrowed tuple adapters and keep collector selection outside the row loop. Extracted from #9979. Enforce the Boolean validity-length invariant in release builds before unchecked output construction. Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
178dcd8 to
5cbf00a
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.984x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.922x ➖, 6↑ 2↓)
datafusion / parquet / ns (0.965x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (0.961x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.753x ➖, 10↑ 1↓)
|
Summary
Review deferred until #10672, #10673, and #10674 merge. This draft temporarily includes their commits. After they merge, replay only the fused-execution commit onto
developfor a focused review.Extracts fused execution from #9979. A private visitor plans a concrete signature and finishes batches that need no retry in the same dispatch. Remaining partially valid batches retain dense retry, valid-row execution, and filtered fallback.
Changes
Keeps borrowed tuple decoding, direct Boolean collection, constant masks, and collector call boundaries inside the framework. The supported RowFn authoring API is unchanged. The packed collector checks validity length in release builds before unchecked output construction.
Retains coverage for nullable execution and output validation. The dispatch tests now reach a real second dispatch through partially valid deferred execution, and new coverage checks single-dispatch execution and rejection of a mismatched validity length.
The Vortex test suite was not run locally. Focused native measurements of the combined migration were run separately during the compiler investigation.