Describe the problem
Reported during review of #5262: reading two visible values out of a sliced array allocated approximately 8 MiB, because DataFusion normalizes the entire backing buffer before slicing. Slicing first reduced the largest allocation to 416 bytes.
This is a component allocation measurement, not an end-to-end query timing.
Describe the potential solution
Slice before normalizing so the work is proportional to the visible range rather than the backing buffer. A benchmark demonstrating the allocation should land with the fix so the improvement is measurable and protected.
Additional context
Reported by @sunchao on #5262. The other allocation observation from the same review, the per-file Iceberg schema helper that built a field-id map for every task even with no required fields, was fixed in #5262 itself: presence is now tested per required id, so an empty required set does no work.
Describe the problem
Reported during review of #5262: reading two visible values out of a sliced array allocated approximately 8 MiB, because DataFusion normalizes the entire backing buffer before slicing. Slicing first reduced the largest allocation to 416 bytes.
This is a component allocation measurement, not an end-to-end query timing.
Describe the potential solution
Slice before normalizing so the work is proportional to the visible range rather than the backing buffer. A benchmark demonstrating the allocation should land with the fix so the improvement is measurable and protected.
Additional context
Reported by @sunchao on #5262. The other allocation observation from the same review, the per-file Iceberg schema helper that built a field-id map for every task even with no required fields, was fixed in #5262 itself: presence is now tested per required id, so an empty required set does no work.