Remove stored Union Nullability

[connortsui20]

Summary

Tracking issue: #7882

A union has no independent parent validity. A row is null when its selected child is null, so the union's logical nullability is determined by its variants.

Storing the same information on DType::Union created two sources of truth and allowed the outer value to disagree with the variants. This change makes inconsistent union DTypes impossible.

The remaining question is how an operation such as mask should change a union's variants when it introduces nulls. That remains deferred and does not block the canonical sparse UnionArray.

The Arrow implementation notes and deferred operation scope are in this comment on #7882.

Changes

• change DType::Union to store only UnionVariants
• derive union nullability at runtime from the variant DTypes
• name the non-zero-cost scan derived_nullability
• leave with_nullability unchanged for unions instead of rewriting variant schemas
• restrict union least-supertype coercion to identical variants for now
• remove Union nullability from Serde, protobuf, and FlatBuffers

Supersedes #8713.

[connortsui20]

should be fine to break this since nobody is using this yet

[connortsui20]

So with_nullability now does nothing for Union. I think this is the correct semantics since it is similar for struct where it will not change child nullabilities?

[connortsui20]

Is there a way to override the flatc check here?

Edit: making changes

[connortsui20]

@robert3005 are you ok with this?

[robert3005]

we cannot do this

[connortsui20]

😢

[robert3005]

poke

[codspeed-hq]

Merging this PR will not alter performance

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 1627 untouched benchmarks
⏩ 42 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	chunked_varbinview_into_canonical[(1000, 10)]	170 µs	206 µs	-17.49%
❌	Simulation	compare_int_constant	268.4 µs	298.3 µs	-10%
⚡	Simulation	chunked_varbinview_opt_canonical_into[(100, 100)]	340.2 µs	305.9 µs	+11.22%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codex/remove-union-nullability (e6e8eeb) with develop (d96e4ad)}

Footnotes

1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩