Skip to content

perf(arrow/array): concatenate list offsets directly - #1323

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-list-concat-direct
Open

fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-list-concat-direct

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Concatenate List and LargeList offsets directly into the output buffer.

Why

  • Avoid temporary fixed-width buffer wrappers for every input chunk.
  • Improve concatenation when arrays are split into many chunks.

Implementation

  • Added concatOffsetsData using the existing direct 32-bit and 64-bit offset handlers.
  • Updated only the List and LargeList paths. Map and Binary paths are unchanged.
  • Added a benchmark covering both list types and 64, 1,024, and 8,192 chunks.

Benchmark on an Apple M1 Pro with 65,536 values:

Case ns/op before -> after allocs/op before -> after
List, 64 chunks 86.5us -> 82.3us 211 -> 145
List, 1,024 chunks 274.1us -> 229.5us 3,309 -> 2,273
List, 8,192 chunks 1.512ms -> 1.220ms 28,627 -> 19,656
LargeList, 64 chunks 93.7us -> 87.5us 211 -> 146
LargeList, 1,024 chunks 262.6us -> 222.5us 3,297 -> 2,254
LargeList, 8,192 chunks 1.467ms -> 1.192ms 28,244 -> 19,217

Tests:

  • go test ./arrow/array -count=1
  • go vet ./arrow/array
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant