listlayout variant: top-level chunked

[mhk197]

Where #8071 decomposes list columns at the leaf (chunked(list(...)) — elements grouped inside each row-chunk), this variant decomposes at the top-level dispatcher (list(chunked(offsets), chunked(elements))): the list column is split into independently-chunked elements / offsets / validity sub-columns, with global u64 offsets, bounded element reads, and pruning routed to the offset/validity children. Env-gated behind VORTEX_EXPERIMENTAL_LIST_LAYOUT (off by default), same as #8071.

Benchmark: this variant is ~7× slower

statpopgen sf100, DuckDB, -f vortex, 3 iterations — this branch vs #8071's leaf approach:

• geomean 7.26× slower; per-query 6–16× slower, worst on selective reads (LIMIT / point lookups: 11–16×).
• File size similar (1.5 GB here vs 1.8 GB leaf), so it's not a storage/IO-volume issue — it's read amplification.

Root cause

Decomposing elements into a first-class primitive column subjects it to the write pipeline's 1 MB-uncompressed coalescing. Genotype elements compress ~30×, so each 1 MB-uncompressed block becomes a ~33 KB physical segment; GT's ~400 M elements → ~3,000 tiny segments vs the leaf variant's few coarse (multi-MB) ones. One query issued 3,169 reads here vs 39 on the leaf variant (~80× amplification), each ~33 KB vs up to 33 MB.

The leaf topology (#8071) avoids this by construction: elements ride inside coarse row-chunks (one segment per chunk, sized by the 8192-row floor), so they're never re-coalesced as a standalone compressible column.

[codspeed-hq]

Merging this PR will not alter performance

⚡ 4 improved benchmarks
❌ 2 regressed benchmarks
✅ 1624 untouched benchmarks
⏩ 42 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	rebuild_naive	91.4 µs	109 µs	-16.17%
❌	Simulation	chunked_varbinview_opt_canonical_into[(100, 100)]	305.4 µs	340.4 µs	-10.27%
⚡	Simulation	bitwise_not_vortex_buffer_mut[128]	273.6 ns	215.3 ns	+27.1%
⚡	Simulation	chunked_varbinview_into_canonical[(1000, 10)]	205.6 µs	169.5 µs	+21.32%
⚡	Simulation	bitwise_not_vortex_buffer_mut[1024]	333.9 ns	275.6 ns	+21.17%
⚡	Simulation	bitwise_not_vortex_buffer_mut[2048]	427.8 ns	369.4 ns	+15.79%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/list-layout-refactor (a0f96a5) with develop (1bcaf3f)}

Footnotes

1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩