Skip to content

Add buffered TsFile input#868

Merged
jt2594838 merged 2 commits into
developfrom
add_buffered_tsfile_reader_develop
Jul 20, 2026
Merged

Add buffered TsFile input#868
jt2594838 merged 2 commits into
developfrom
add_buffered_tsfile_reader_develop

Conversation

@jt2594838

Copy link
Copy Markdown
Contributor

Summary

  • Add BufferedTsFileInput with a configurable ByteBuffer read cache and an 8 KiB default.
  • Keep positional and sequential read semantics, InputStream wrapping, seeking, EOF handling, and close-state checks.
  • Add focused unit tests and an opt-in TsFileSequenceReader performance test.
  • Compare LocalTsFileInput with 4/8/16/32/64 KiB buffers across 100/1,000/10,000 points per chunk.

Performance

Each configuration reads 100 TsFiles and 1 billion points. The test rotates reader order per file to reduce page-cache ordering bias.

Points per chunk Local Buffered 8 KiB Speedup
100 106.003 s 15.036 s 7.050x
1,000 20.293 s 10.235 s 1.983x
10,000 10.389 s 9.148 s 1.136x

Testing

  • mvn test-compile surefire:test -P with-java -pl java/tsfile -am -Dtest=BufferedTsFileInputTest -Dsurefire.failIfNoSpecifiedTests=false
    • 4 tests passed
  • Full opt-in performance matrix
    • 1 test passed, 18 billion points decoded and validated
  • Checkstyle and Spotless passed

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.40659% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.53%. Comparing base (4668950) to head (dbd56e0).

Files with missing lines Patch % Lines
...apache/tsfile/read/reader/BufferedTsFileInput.java 96.55% 3 Missing ⚠️
.../main/java/org/apache/tsfile/utils/PublicBAOS.java 0.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #868      +/-   ##
===========================================
+ Coverage    60.47%   60.53%   +0.06%     
===========================================
  Files          740      741       +1     
  Lines        48617    48708      +91     
  Branches      7713     7726      +13     
===========================================
+ Hits         29400    29486      +86     
- Misses       17809    17814       +5     
  Partials      1408     1408              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jt2594838
jt2594838 merged commit b8825b1 into develop Jul 20, 2026
14 checks passed
@jt2594838
jt2594838 deleted the add_buffered_tsfile_reader_develop branch July 20, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants