Skip to content

[Pipe] Reduce allocations in query Field copying and Tablet batch encoding - #19282

Open
Caideyipi wants to merge 5 commits into
masterfrom
codex/pipe-tablet-pr-20261113
Open

Caideyipi wants to merge 5 commits into
masterfrom
codex/pipe-tablet-pr-20261113

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Pipe's TsFile query parser converts each primitive Field to a boxed object before storing it in a Tablet. Raw Tablet batches also encode their payloads through PublicBAOS and DataOutputStream. This change copies primitive fields directly into Tablet columns and writes both tree-model and merged table-model batches into exactly sized ByteBuffers.

The existing null/default conversions, DATE representation compatibility, OBJECT fallback, deletion handling, and request wire format are preserved. For example, if timestamp 0 is fully deleted and timestamps 1 and 2 contain (11, null) and (12, "value"), the output still contains those two rows with the same null bitmap, even though the deleted row's slot is reused. Aligned and non-aligned tree batches and merged table batches retain byte-for-byte equivalence with stream serialization, including NaNs and signed zero.

Related pull requests

Depends on apache/tsfile#1101, specifically commit eaaad099d6a29bff60cc374ad0ff30fad6b40e3d, for Tablet.serialize(ByteBuffer) and matching NaN encoding. The POM selects version 2.5.1-261118-SNAPSHOT. This dependency must be integrated and made available before this PR can merge; the local checks used artifacts built from the specified source revision.

Performance

The following local JMH measurements cover the query-parser, tree-model, and merged table-model changes described above. Both variants use the same TsFile revision above, so these measurements isolate the IoTDB changes and do not attribute the results to changes between TsFile versions. They were collected on November 13 on the earlier development snapshot, before transplanting the same production-file changes onto public master; timings were not rerun on the final publication base.

Operation / input Baseline ms/op Optimized ms/op Mean latency reduction Allocated bytes/op reduction
QueryParser, 64 INT64 columns 336.106 232.381 30.9% 13.1%
QueryParser, 256 INT64 columns 1,439.066 1,140.021 20.8% 12.9%
Tree-model batch, mostly INT64 1.435 0.417 71.0% 0.3%
Tree-model batch, mixed types with nulls 2.239 0.699 68.8% 0.4%
Merged table-model batch, mostly INT64 2.117 1.670 21.1% 0.1%
Merged table-model batch, mixed types with nulls 3.412 2.411 29.4% 0.1%
  • Windows x86_64, Intel Core i9-13900H, JDK 17.0.15, JMH 1.37; one benchmark thread, -Xms1g -Xmx1g -XX:ActiveProcessorCount=4.
  • Each variant used two independent JVM forks, each with three 1-second warmup iterations and five 1-second measurement iterations. Allocation is JMH GCProfiler's normalized allocated bytes per operation.
  • QueryParser read one aligned device with 16,384 rows and either 64 or 256 INT64 columns (1,048,576 or 4,194,304 points per operation). Input generation was excluded, reads used a warm OS cache, and returned values and timestamps were consumed.
  • Batch inputs used 64 columns and 1,024 rows per Tablet. Mostly-INT64 input had one STRING column and 63 INT64 columns. Mixed input covered BOOLEAN, INT32, INT64, FLOAT, DOUBLE, DATE, STRING, and BLOB, with about 14% nulls outside the first column. Tree batches used one Tablet; merged table batches used two. Timing included production batch construction, memory accounting, merge/copy where applicable, final request encoding, and batch reset. Stream-format byte equality was checked outside timed regions.

These are preliminary component measurements on a shared workstation. The JMH 99.9% timing confidence intervals overlap for the 256-column QueryParser and both merged table-model cases, so their percentage changes should not be treated as established, stable speedups. End-to-end sender/network/receiver Pipe throughput and distribution packaging were not tested. The AirGap V1/V2 ByteBuffer serializer was added later and was not covered by these measurements.

Validation

On public master base ea7b4ffc5d68021f18a18f627218ab625eee7f7a, the required 28-module dependency closure was rebuilt from clean source, and the focused reactor test run completed with 16 tests passed, zero failures, errors, or skips (Windows, JDK 17.0.15, Maven 3.9.12). This includes the five regression tests present in that run; the AirGap test added later is covered separately below. The deletion fixture uses the public branch's existing PatternTreeMap factory; its deletion records and assertions are unchanged.

On the final source snapshot, the focused clean DataNode reactor run for PipeTransferSerializationSizeTest passed 6 tests, zero failures, errors, or skips (Windows, JDK 17.0.15, Maven 3.8.8). It includes the AirGap V1/V2 mixed-type byte-compatibility regression test, comparing the new serializer byte-for-byte with the previous DataOutputStream format. Coverage includes null bitmaps, NaNs, signed zero, integer boundaries, binary values, both aligned flags, and a non-ASCII V2 database name.

The original implementation snapshot passed 19 selected local tests. All six tests added by this PR have been run locally. Verification temporarily disabled build-scan extensions; that configuration change is excluded from the submitted diff.

Checklist

  • Independently self-reviewed once with no actionable findings in the original implementation snapshot; publication preserves the reviewed production-file contents.
  • Added unit tests covering direct Field copy, deleted-row slot reuse, complete request byte compatibility, and mixed-type AirGap V1/V2 byte compatibility.
  • Measured local component performance with the limitations stated above.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Self-review summary (re-review)

  • Reviewed SHA: 6788ae035ba0d6e1902ad72866c6a346f0d53954. Found one architecture violation: PipeTabletUtils.putFieldValue switched on TSDataType outside TypeService.
  • Fixed by routing type-specific field writing through TypeServices.Pipe.PIPE_FIELD_TABLET_VALUE_WRITER_SERVICE, preserving the existing null-field, DATE, and binary behavior.
  • Verification: Spotless passed. Focused DataNode tests passed: 5 tests, 0 failures, 0 errors, 0 skipped. Tests used TsFile artifacts built from source commit a1d3547f525c9a58f4c59eebb1757930672401dd with 2.5.1-SNAPSHOT as the local override because 2.5.1-261118-SNAPSHOT was not installed locally; the PR retains the requested dependency version.
  • Final local SHA: 685d6fe012f97c731cebcd4958664f8704964101. Confirmed remote branch and PR head: the same SHA. PR remains open and non-draft.

@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.95918% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 49.25%. Comparing base (ea7b4ff) to head (0ed9517).
⚠️ Report is 22 commits behind head on master.

Files with missing lines Patch % Lines
...n/java/org/apache/iotdb/db/utils/TypeServices.java 94.44% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19282      +/-   ##
============================================
+ Coverage     49.20%   49.25%   +0.04%     
- Complexity      699      717      +18     
============================================
  Files          5593     5648      +55     
  Lines        414717   416676    +1959     
  Branches      54769    55021     +252     
============================================
+ Hits         204047   205213    +1166     
- Misses       210670   211463     +793     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi

Caideyipi commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Scoped Independent Re-review

Reviewed only commits 685d6fe012f97c731cebcd4958664f8704964101 and b9044591a9f05a4a0399c679ac077429fdee7945, with their necessary implementation and test context, at PR head b9044591a9f05a4a0399c679ac077429fdee7945 against base ea7b4ffc5d68021f18a18f627218ab625eee7f7a.

  • Findings: None in the requested scope; no fixes were needed.
  • Review verification: Recorded focused architecture-fix tests passed (5 tests, 0 failures, 0 errors, 0 skipped). The recorded AirGap serializer tests also passed (5 tests, 0 failures, 0 errors, 0 skipped). git diff --check passed. No new tests were run during the review itself.
  • Test-coverage follow-up, added after the reviewed revision: Added mixed-type AirGap V1/V2 byte-compatibility coverage. The focused clean DataNode reactor run passed 6 tests, 0 failures, 0 errors, and 0 skipped; the new test compares serialized bytes with the previous DataOutputStream format and covers null bitmaps, NaNs, signed zero, boundary integers, binary values, both alignment flags, and a non-ASCII V2 database name.
  • Performance: The AirGap ByteBuffer path was not separately benchmarked; the existing component JMH measurements do not cover this follow-up.
  • Reviewed SHA: b9044591a9f05a4a0399c679ac077429fdee7945. Final local SHA and confirmed remote branch/PR head: 0ed95179459e21265dd5d364ce3b5c67dcb442eb. The PR remains open and non-draft.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant