Aggregate sum of zero valid values is null

[mhk197]

SUM of zero valid values now yields null , as per SQL standard (SQL SUM, ISO 9075-2 §10.9). This replaces he previous "all-invalid sums to zero" contract. Overflow still yields null. This matches arrow-rs (sum returns None for empty/all-null), DuckDB (SumState.isset), and DataFusion (SumAccumulator { sum: Option }).

The blocker for doing this at the root was that a null partial already means overflow and must poison merges, while "empty" must be the merge identity — one nullable scalar can't encode both, and Sum partials cross persisted merge boundaries (zoned layout stats). So SumPartial now carries a seen flag (DuckDB's isset), and the partial dtype becomes {sum, seen}.

Stat::Sum retains original behavior because persisted stat merging requires identity-on-empty.

Side Effects:

• mean of all-null input is now null
• DataFusion ColumnStatistics::sum_value is only exported when null_count < row_count
• Also changed StructGetItemRule to handle struct partials. get_item on a nullable struct with NonNullable/AllValid validity previously returned the raw non-nullable field.

[codspeed-hq]

Merging this PR will not alter performance

⚡ 12 improved benchmarks
❌ 9 regressed benchmarks
✅ 1609 untouched benchmarks
⏩ 42 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	sum_f64_all_valid	38.7 µs	52.3 µs	-26.07%
❌	Simulation	sum_i32_nullable_all_valid	41.6 µs	54.9 µs	-24.13%
❌	Simulation	sum_i32_clustered_nulls	65 µs	80.8 µs	-19.52%
❌	Simulation	sum_f64_clustered_nulls	68.3 µs	83.1 µs	-17.79%
❌	Simulation	rebuild_naive	91.4 µs	109.2 µs	-16.31%
❌	Simulation	compact[(16384, 90)]	813.9 ns	930.6 ns	-12.54%
❌	Simulation	fsl_sum_small	57.1 µs	64.9 µs	-11.92%
❌	Simulation	chunked_varbinview_opt_canonical_into[(100, 100)]	305.4 µs	341 µs	-10.43%
❌	Simulation	compact[(4096, 90)]	779.7 ns	867.2 ns	-10.09%
⚡	Simulation	bitwise_not_vortex_buffer_mut[128]	273.6 ns	215.3 ns	+27.1%
⚡	Simulation	bitwise_not_vortex_buffer_mut[1024]	333.9 ns	275.6 ns	+21.17%
⚡	Simulation	chunked_varbinview_into_canonical[(1000, 10)]	205.6 µs	169.8 µs	+21.11%
⚡	Simulation	list_sum_large	135.7 ms	112.1 ms	+21.11%
⚡	Simulation	listview_sum_large	108.1 ms	89.3 ms	+21.08%
⚡	Simulation	bitwise_not_vortex_buffer_mut[2048]	427.8 ns	369.4 ns	+15.79%
⚡	Simulation	list_sum_nullable_elements_large	495.9 ms	428.4 ms	+15.75%
⚡	Simulation	encode_varbin[(1000, 32)]	165.4 µs	144.6 µs	+14.39%
⚡	Simulation	list_sum_nullable_elements_medium	4.9 ms	4.3 ms	+12.19%
⚡	Simulation	encode_varbin[(1000, 8)]	154.6 µs	139.8 µs	+10.57%
⚡	Simulation	encode_varbin[(1000, 4)]	153.1 µs	138.9 µs	+10.17%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/sum-empty-null (8e8fdc6) with develop (1bcaf3f)}

Footnotes

1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[gatesn]

I think... with the new Zoned layout, the aggregate partial has become a wire API.

That said, we haven't updated FileStats to use aggregate partials. And I don't believe zone maps use Sum?

A safer option would be to add a separate Sum function with this new behavior and different partial. @robert3005 any thoughts on a preferred direction?

Either way, we should separately make sure we have forward/backward compatibility tests for zoned statistics!

[robert3005]

we have SUM in zonemaps

[robert3005]

@mhk197 I think simpler would be to add another function. Maybe even rename them such that the new sql semantic sum is called sum and the other is called sum_<I have no idea what a good name is>