Repository navigation
Attach RowFn validity during Boolean output construction - #10674
connortsui20 wants to merge 1 commit into
Conversation
Merging this PR will degrade performance by 11.12%3 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
5541df9 to
abc8689
Compare
14c6b46 to
263d116
Compare
Let the packed Boolean collectors consume the batch validity when constructing their result. Keep custom output validation before masking and attach validity directly to other canonical outputs. Extracted from #9979. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
263d116 to
cfbd513
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.990x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.951x ➖, 2↑ 1↓)
datafusion / parquet / ns (0.819x ➖, 7↑ 1↓)
duckdb / vortex-file-compressed / ns (0.939x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.880x ➖, 3↑ 0↓)
|
Summary
Extracts output finalization from #9979. Packed Boolean collectors can construct the result with its input validity attached, avoiding a second array construction.
Targets
developand can merge independently of the borrowed-decoding and batch-inspection changes.Changes
The validity handoff stays private to the executor. Other canonical outputs receive validity directly, while custom kernel outputs retain validation before masking can hide an invalid result. Existing allocator cases are retained, and new cases cover attached Boolean validity and canonical batch output.
The Vortex test suite was not run locally. Focused native measurements of the combined migration were run separately during the compiler investigation.