Skip to content

perf: cut flush encoding allocations and page-write copies - #1133

Open
luoluoyuyu wants to merge 1 commit into
apache:developfrom
luoluoyuyu:perf/flush-encode-gc-dev
Open

luoluoyuyu wants to merge 1 commit into
apache:developfrom
luoluoyuyu:perf/flush-encode-gc-dev

Conversation

@luoluoyuyu

@luoluoyuyu luoluoyuyu commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Reduce allocations, boxing and buffer copies in the page/chunk write path, without changing chunk size, page size or produced bytes.

  • Reuse the uncompressed page ByteBuffer and the compressed output byte[] per writer (PageBodyBuffer), and reuse the RLE bit-pack scratch buffer.
  • Pre-size the page writer buffers and grow the chunk page buffer once after the first page.
  • Reset chunk writer page-sizing state when a writer is reused, so a recycled writer produces the same page boundaries as a fresh one.
  • Remove the per-value lambda and Integer boxing from the dictionary encoder; encode int RLE values from a primitive buffer.
  • Write the first page statistics up front for multi-page chunks instead of copying the whole chunk buffer when the second page arrives; compact in place for single-page chunks.
  • Batch Gorilla/Chimp bit writes through a 64-bit accumulator, and account for buffered bytes when sizing pages so page splits stay identical.

The dictionary index intentionally keeps HashMap (without the per-value lambda): an open-addressing variant saved ~0.55 MiB per encode but made dictionary encoding ~16% slower in a same-branch A/B, so it was reverted.

Results

Local IoTDB flush benchmark (200k rows x 6 columns, deterministic allocation):

Metric Before After
Encode allocation (dense, 2 chunks) 37.9 MiB 8.3 MiB
Encode peak heap 32 MiB 4 MiB
End-to-end flush, 1.6M points 40.4 MiB, 54 ms 25.5 MiB, ~25 ms

Allocation numbers are deterministic; CPU/wall time varies by machine.

Compatibility

  • Chunk size, page size and output bytes are unchanged. Verified with SHA-256 comparisons against the base for TEXT, INT32, INT64, BOOLEAN, FLOAT/DOUBLE GORILLA and CHIMP, the page layout file, and reused vs. fresh chunk writers.
  • Existing tests pass (78 encoder/page-writer tests).

Test plan

  • ./mvnw -P with-java -pl java/tsfile -am -Dtest=GorillaDecoderV2Test,ChimpDecoderTest,GorillaDecoderV1Test,IntRleDecoderTest,LongRleDecoderTest,DictionaryDecoderTest,PageWriterTest,ValuePageWriterTest,TimePageWriterTest,PageBodyEncoderTest test
  • Byte-level hash comparison against base for TEXT / all types / page layout / writer reuse
  • spotless:apply

Reduce allocations, boxing and buffer copies in the page/chunk write path
without changing chunk size, page size or produced bytes.

- reuse page body ByteBuffer and compressed byte[] per writer, and reuse
  RLE bit-pack scratch buffers
- pre-size page writer buffers and grow the chunk page buffer once after
  the first page
- reset chunk writer page-sizing state when a writer is reused
- remove the per-value lambda and Integer boxing from the dictionary
  encoder; encode int RLE values from a primitive buffer
- write the first page statistics up front for multi-page chunks instead
  of copying the whole chunk buffer, and compact in place for single-page
  chunks
- batch Gorilla/Chimp bit writes through a 64-bit accumulator and account
  for buffered bytes when sizing pages
@luoluoyuyu
luoluoyuyu force-pushed the perf/flush-encode-gc-dev branch from 87f3dfb to 9363e26 Compare November 19, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant