Repository navigation
Conversation
Collaborator
Author
Self-review summary (re-review)
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19282 +/- ##
============================================
+ Coverage 49.20% 49.25% +0.04%
- Complexity 699 717 +18
============================================
Files 5593 5648 +55
Lines 414717 416676 +1959
Branches 54769 55021 +252
============================================
+ Hits 204047 205213 +1166
- Misses 210670 211463 +793 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Collaborator
Author
Scoped Independent Re-reviewReviewed only commits
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Description
Pipe's TsFile query parser converts each primitive
Fieldto a boxed object before storing it in a Tablet. Raw Tablet batches also encode their payloads throughPublicBAOSandDataOutputStream. This change copies primitive fields directly into Tablet columns and writes both tree-model and merged table-model batches into exactly sized ByteBuffers.The existing null/default conversions, DATE representation compatibility, OBJECT fallback, deletion handling, and request wire format are preserved. For example, if timestamp 0 is fully deleted and timestamps 1 and 2 contain
(11, null)and(12, "value"), the output still contains those two rows with the same null bitmap, even though the deleted row's slot is reused. Aligned and non-aligned tree batches and merged table batches retain byte-for-byte equivalence with stream serialization, including NaNs and signed zero.Related pull requests
Depends on apache/tsfile#1101, specifically commit
eaaad099d6a29bff60cc374ad0ff30fad6b40e3d, forTablet.serialize(ByteBuffer)and matching NaN encoding. The POM selects version2.5.1-261118-SNAPSHOT. This dependency must be integrated and made available before this PR can merge; the local checks used artifacts built from the specified source revision.Performance
The following local JMH measurements cover the query-parser, tree-model, and merged table-model changes described above. Both variants use the same TsFile revision above, so these measurements isolate the IoTDB changes and do not attribute the results to changes between TsFile versions. They were collected on November 13 on the earlier development snapshot, before transplanting the same production-file changes onto public
master; timings were not rerun on the final publication base.-Xms1g -Xmx1g -XX:ActiveProcessorCount=4.These are preliminary component measurements on a shared workstation. The JMH 99.9% timing confidence intervals overlap for the 256-column QueryParser and both merged table-model cases, so their percentage changes should not be treated as established, stable speedups. End-to-end sender/network/receiver Pipe throughput and distribution packaging were not tested. The AirGap V1/V2 ByteBuffer serializer was added later and was not covered by these measurements.
Validation
On public
masterbaseea7b4ffc5d68021f18a18f627218ab625eee7f7a, the required 28-module dependency closure was rebuilt from clean source, and the focused reactor test run completed with 16 tests passed, zero failures, errors, or skips (Windows, JDK 17.0.15, Maven 3.9.12). This includes the five regression tests present in that run; the AirGap test added later is covered separately below. The deletion fixture uses the public branch's existing PatternTreeMap factory; its deletion records and assertions are unchanged.On the final source snapshot, the focused clean DataNode reactor run for
PipeTransferSerializationSizeTestpassed 6 tests, zero failures, errors, or skips (Windows, JDK 17.0.15, Maven 3.8.8). It includes the AirGap V1/V2 mixed-type byte-compatibility regression test, comparing the new serializer byte-for-byte with the previousDataOutputStreamformat. Coverage includes null bitmaps, NaNs, signed zero, integer boundaries, binary values, both aligned flags, and a non-ASCII V2 database name.The original implementation snapshot passed 19 selected local tests. All six tests added by this PR have been run locally. Verification temporarily disabled build-scan extensions; that configuration change is excluded from the submitted diff.
Checklist