Repository navigation
Conversation
A table scan that selects (nearly) all field columns of a device previously deserialized the device's TimeSeriesMetadata column by column, re-walking the device metadata index once per leaf node. Add a batch path that reads the whole device's metadata in a single pass via TsFileSequenceReader.readDeviceMetadata, populating the per-measurement cache and building the aligned metadata through the existing modification/loader logic. The batch path is taken when the selected field columns reach a configurable fraction (device_metadata_batch_scan_ratio, default 0.8) of the table's total field columns.
The batch path read a device's TimeseriesMetadata via TsFileSequenceReader.readDeviceMetadata, which deserializes only the header and statistics and leaves chunkMetadataList null. The downstream DiskAlignedChunkMetadataLoader dereferences that list in getCopiedChunkMetadataList, so a wide-table scan NPE'd at query time, and the null-chunk entries also polluted the shared per-measurement LRU cache (breaking later narrow queries on the same device). Switch to getDeviceTimeseriesMetadata, which deserializes chunk metadata too, and adapt the unit test to the list-based API, asserting that chunk metadata flows through to the result.
Remove the unreachable time-column-only branch in the batch loader (the batch gate already requires a non-empty selection), and validate device_metadata_batch_scan_ratio so a non-positive value falls back to the default instead of routing every table scan through the batch path. Document the valid range in the properties template.
Remove the ignoreNotExists gate so the batch path is also taken for files with a device-level time index, where previously only file-level-index files (or recovery) used it. When the batch read finds the device is missing from the file, replicate the per-column path: skip silently when ignoreNotExists is set, otherwise throw CorruptedTsFileException to surface the time-index/metadata inconsistency.
Remove the device_metadata_batch_scan_ratio config parameter and hardcode the threshold at 0.6 (previously configurable, default 0.8) so the batch path covers a wider set of scans without requiring configuration.
Serialize concurrent reads of the same device+file in getDeviceTimeSeriesMetadata (matching the per-column get) so threads don't each deserialize the whole device and race on the shared reader. Also attach the missing device id to the CorruptedTsFileException thrown when a device is absent from a device-level time index file.
Count every value column (FIELD + ATTRIBUTE) as the batch scan ratio denominator instead of only FIELD columns, matching the selected-column numerator. Cache empty (count == 0) series as the null placeholder in the batch read, matching the per-column get, so later per-column reads don't re-read them. Add a unit test for the absent-device (empty map) path.
Cover the loadAlignedTimeSeriesMetadataBatch empty-map branch: a device absent from a device-level time index file throws CorruptedTsFileException, while with a file-level time index (or during recovery) it is skipped.
Drive AlignedSeriesScanUtil with all value columns selected and a matching total field column count so the batch path is taken, and assert the aligned pages are still read correctly.
The batch read offers no benefit for a single-column device (one leaf either way) and it bypasses the per-column cache's debug logging, which IoTDBDebugQueryIT.tableTest relies on. Require at least two value columns before taking the batch path.
getDeviceTimeSeriesMetadata always deserialized the device's whole metadata index and then re-populated the cache, never reading it, so repeated wide-table scans were slower than the per-column path that it populates. Check the per-measurement cache first, read the device only when one of the requested measurements is missing, and cache just those measurements (plus the null placeholders for absent ones). Align the absent-device behavior with the per-column get(): skip the device when the file's bloom filter proves it is missing, otherwise raise the read IOException so the existing wrapper records the suspicious-disk metric. Require a null device offset for the batch path instead of relying on the zero sentinel used by external scans, and extract the aligned metadata assembly shared by both paths. Count only FIELD columns in the ratio denominator to match the smaller numerator, and extend the tests to cover the cache hit, the bloom-filter decisions, the gating boundary and the batch path actually being taken.
The batch metadata scan bypasses TimeSeriesMetadataCache.get(..., debug=true) and does not emit equivalent hit/miss diagnostics, so a DEBUG SELECT on a wide table lost its metadata logs. Make the batch gate take the debug and offset-addressed state explicitly so DEBUG queries keep the per-column path, and cover both conditions in the gating unit test.
getDeviceTimeSeriesMetadata read the whole device metadata index and only then discovered the device was absent, while the per-column get() can skip that read by probing the file's bloom filter first. Probe the cached time column entry before the read, keep the same missing-device skip/throw rules, and account the bloom filter cache/disk/time statistics like get().
JackieTien97
left a comment
There was a problem hiding this comment.
Thanks for the optimization. I checked out 3fba980 and ran the PR's tests locally (FileLoaderUtilsTest, TimeSeriesMetadataCacheTest, AlignedSeriesScanBatchTest); all pass. I also added a throwaway probe test (not pushed) that writes a real aligned device with 700 columns (three measurement leaf groups) and compares getDeviceTimeSeriesMetadata against the per-column get(): all 701 series (time column plus 700 fields) matched, including statistics and chunk-metadata counts. So I found no wrong-result issue. The comments below are mostly about performance, diagnostics and metrics.
Main points:
- [P1] The batch path only counts as a cache hit when every selected sensor is in the LRU. If any selected column is absent from a file and that absence can't be cached as a placeholder (
ignoreNotExists=trueormay_cache_nonexist_series=false), every query re-reads the whole device. The per-column path doesn't (TimeSeriesMetadataCacheL362). - [P1] The missing-device error message describes the opposite inconsistency (L491).
- [P2] Metadata IO size and load counts are no longer comparable with the per-column path (L483, L370).
- [P2] The I/O saving only exists when the selected columns span more than one measurement leaf group (more than 256 series by default), and the gate doesn't reflect that (
FileLoaderUtilsL447). Could the same batching be done inside the existingget()miss path instead of a parallel path (L342)? - The PR description is out of date. It still describes a configurable
device_metadata_batch_scan_ratio(default 0.8), anIoTDBConfigchange, anignoreNotExistsgate andreadDeviceMetadata. The code hard-codes 0.6 with no switch, has no such gate, and usesgetDeviceTimeseriesMetadata. Could you update it and add benchmark numbers for a wide device (e.g. 800 columns, cold and warm cache) compared with master? That would make it easier to judge the 0.6 threshold.
Reuse cached measurements and batch-read only the missing ones, probing the per-measurement bloom filter first so absent columns do not force a whole-device read; the all-or-nothing cache check is gone. Count load statistics per series so they stay comparable with the per-column path, and document the batch read's missing IO-size accounting. Report a dedicated error naming the device and the TsFile when the resource records a device that the file metadata lacks (en + zh), share the load statistics recording between both paths, and pass the external flag through the bloom filter lookup. Only batch when the table has more field columns than the measurement index fan-out, so small devices stay on the per-column path. Add a real 700-column batch-vs-per-column comparison, a 300-column end-to-end batch scan spanning two leaf groups, and gate boundary cases.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved batch-path issues can trigger unnecessary device reads and absent-device I/O errors, while regression tests leave sparse all-null rows and multi-leaf field values unverified.
5 open issues
Add sparse all-null-row coverage for batched scans · New Skip device reads when the requested time column is absent · New Avoid whole-device reads for small remaining cache misses · New Include cold cache probes in disk lookup timing · New Assert values across all leaf groups in the multi-leaf test · New
What changed in this PR
This PR reduces repeated metadata-index traversal in DataNode wide-table queries by loading a device’s metadata in one batch when most field columns are selected.
Changes:
- Adds cache- and bloom-filter-aware batch loading with per-series query statistics.
- Gates batching by table width and a 60% projection threshold, excluding DEBUG and offset-addressed scans.
- Shares aligned metadata assembly, adds localized missing-device errors, and introduces regression tests.
| File | Description |
|---|---|
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/series/AlignedSeriesScanBatchTest.java |
Adds a real-file wide-device batch scan regression. |
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCacheTest.java |
Tests batch loading, caching, missing devices, and metadata equivalence. |
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtilsTest.java |
Tests batch eligibility boundaries and exclusions. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCache.java |
Implements cache-aware device metadata loading and statistics accounting. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/parameter/SeriesScanOptions.java |
Carries the table’s total field-column count. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/DataNodeTableOperatorGenerator.java |
Supplies field counts for eligible table scans. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/ExternalTsFileSeriesScanUtil.java |
Keeps external offset-addressed scans on the per-column path. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtils.java |
Selects the batch path and shares aligned metadata assembly. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/AlignedSeriesScanUtil.java |
Passes the field count into metadata loading. |
iotdb-core/datanode/src/main/i18n/zh/org/apache/iotdb/db/i18n/StorageEngineMessages.java |
Adds the Chinese missing-device error message. |
iotdb-core/datanode/src/main/i18n/en/org/apache/iotdb/db/i18n/StorageEngineMessages.java |
Adds the English error naming the device and TsFile. |
Review effort: Max
Skip the device read when the time column is known absent, so a bloom filter false positive on a field cannot force an absent-device read. Fall back to per-column lookups for a small remainder (one or two measurements), which reads a single leaf group instead of the whole device, and count the cold-call cache probe and lock wait in the disk lookup time. Strengthen the wide-scan tests: assert timestamps and every field value, compare a sparse batch scan against the per-column path, cover an all-null projection under ignoreAllNullRows, and add cache regressions for the filtered time column and the single-miss fallback.
TsFileSequenceReader is an external apache/tsfile dependency and has no batch API that accepts a measurement set, so the batch path still reads the whole device and filters afterwards. Note the constraint and the existing mitigations (bloom filter, per-column fallback) at the call site; a truly selective reader needs an upstream change.
Load the remaining misses through the per-column get() when the cache already holds most of the request: it touches only their leaf groups instead of re-reading the whole device. Replace the GC-sensitive WeakHashMap monitor with fixed lock stripes, and make the returned map complete for allSensors on every early return. Give aggregation table scans a totalFieldColumnCount so they can use the batch path, and make the fan-out gate span two leaf groups (>=), fixing its off-by-one and comment. Tests: production ignoreAllNullRows=false coverage including all-null projections, batch negative-caching parity, copy-on-return, a three-miss eviction case, the multi-leaf premise, and per-chunk offset/statistics comparison.
When a cold batch call delegates the remaining measurements to the per-column get(), the cache probes and lock wait were assigned to cache time while the cache hit count was zero, so neither branch reported them. Attribute the phase to disk time when there are no cache hits and allow a disk time with a zero count to be recorded.
JackieTien97
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. Status of my earlier comments: 9 of 12 are fixed, including both P1s. Still open by design: the batch read's IO bytes are not reported, the 0.6 ratio has no off switch, and the selective reader is an upstream follow-up.
Since this PR is a performance change, I benchmarked it against its base. A JUnit harness (not pushed) drives FileLoaderUtils.loadAlignedTimeSeriesMetadata over real TsFiles written with TsFileWriter (12 files, aligned devices, 4 chunks per series). It counts the bytes and read calls that reach the file and runs each variant in its own JVM (surefire defaults with -Xmx1024m, which gives a ~61 MB TimeSeriesMetadataCache). I measured f8b9c83 against ca8074d. The only batch-path change since then, 469394f, is time accounting, so the numbers apply to 6ab4b8e. Loaded metadata was identical to base in every scenario (checksums), so there is no correctness issue here.
Two runs per variant; ms per query (one device across 12 files), cold / warm:
| Scenario | base | PR | PR + the two changes below |
|---|---|---|---|
| Dense, 700 fields, select all, cache fits | 5.1–5.3 / 1.7–1.8 | 8.5 / 1.5–2.7 | 5.5–5.9 / 1.5–1.7 |
| Dense, 700 fields, select first 60% | 3.4–3.6 / 1.0–1.1 | 5.3–5.4 / 1.0 | 4.0–4.4 / 1.1 |
| Repeated select-all, working set larger than the metadata cache (steady state) | 4.5–9.9 ms, 1.7–4.1 MB per query | 10.1–16.1 ms, 11–15 MB per query | 4.4–5.4 ms, 1.0–2.0 MB per query |
| Sparse, ~20 of 700 fields per file (the motivating case) | 4.8–4.9 / 4.7 | 2.8–2.9 / 2.1 | 1.9–2.0 / 2.1 |
| 100-field table (below the gate) | 0.9 / 0.27 | unchanged | unchanged |
| Cold load, 96 devices x 300 fields, 8 / 32 threads (wall ms) | 225–232 / 237–247 | 208–220 / 193–210 | 199–213 / 186–189 |
The sparse case gains about 2x as intended, and the lock stripes do not hurt concurrency. There are two regressions, though. Each has an inline comment with the change that removed it locally:
- [P1] Under metadata-cache pressure, the "cache holds most of the request, so delegate to per-column
get()" fallback multiplies metadata reads 3–8x. - [P1] For dense wide devices, a cold call costs 45–70% more time than base. It moves the same bytes in fewer reads, so the extra cost is CPU: every selected measurement goes through two cache passes and a BloomFilter probe before the device read.
Two smaller points:
- With a 60% prefix projection, the device-wide read still loads a leaf group that no selected column needs, so it stays 10–30% slower even after both changes. That cost remains until the selective reader lands.
- The description says
getDeviceTimeseriesMetadatacosts about 2x IO. In these runs the batch read moved the same bytes as the per-column path (2019.9 KB vs 2021.0 KB per query, in 48 vs 72 reads). Also, the gate in the code is>=, while the description says "more than".
I'm happy to share the harness if it helps.
| // (later misses in the same group hit the cache) instead of re-reading the whole device. | ||
| // Those lookups record their own statistics. | ||
| if (possiblyPresent.size() <= MAX_PER_COLUMN_FALLBACK | ||
| || (cacheHitCount > 0 && possiblyPresent.size() <= cacheHitCount)) { |
There was a problem hiding this comment.
[P1] This fallback multiplies metadata IO 3–8x under cache pressure
Setup: repeated SELECT * over 8 devices x 12 files x 701 series, which is more than the ~61 MB TimeSeriesMetadataCache holds. In that steady state, most calls arrive with more than half of their keys cached and take this branch. Per query, the PR read 11–15 MB in 393–514 reads (1,520–2,006 readTimeseriesMetadata calls per round), while base read 1.7–4.1 MB in 61–144 reads. That made the PR 10–16 ms per query against base's 4.5–9.9 ms.
Each get() here re-reads and re-inserts the whole leaf group of the missing measurement (up to 256 series, with chunk metadata). Under eviction pressure, those entries do not survive until the next missing measurement of the same group is looked up. Visiting the misses in sorted order did not help (13 MB per query).
Dropping only the majority clause brought it down to 1.0–2.0 MB and 4.3–5.4 ms per query, at or below base. That means keeping possiblyPresent.size() <= MAX_PER_COLUMN_FALLBACK and otherwise doing the single device read. Partial-eviction runs with a cache large enough stayed at base level. Could you remove the clause, or replace it with a group-aware load that reads each needed leaf group once and serves the misses from that read instead of from the cache?
| loadBloomFilter(filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, false); | ||
| List<String> possiblyPresent = new ArrayList<>(missing.size()); | ||
| for (String measurement : missing) { | ||
| if (bloomFilterMayContain(bloomFilter, device, measurement)) { |
There was a problem hiding this comment.
[P1] Cold calls on dense wide devices are 45–70% slower than base
With the cache cleared (and large enough for the working set), loading a 700-field device across 12 files took 8.5 ms per query against base's 5.1–5.3 ms. A 60% projection took 5.3–5.4 ms against 3.4–3.6 ms. The bytes were the same and the batch path made fewer reads (48 vs 72 per query), so the extra cost is CPU. On a cold call, every selected measurement goes through:
- two
collectCachedMetadatapasses; - this probe, which builds
device.toString() + "." + measurementand runs k hashes, 700 times; - the copies into the
possiblyPresentHashSet, the inner result map,putAllandputIfAbsent.
The per-column path probes the filter once per leaf group.
What removed the gap locally, when cacheHitCount == 0:
- under the lock, re-check only the time-column key, and fall back to the current path if it appeared;
- probe the BloomFilter for the time column only, then read the device once and fill
resultdirectly; - consult the filter only for selected measurements the read did not find, to decide whether to cache a placeholder (as
get()does).
With that change, dense select-all came out at 5.5–5.9 ms and the 60% projection at 4.0–4.4 ms. It also sped up the sparse case (1.9–2.0 ms cold vs 2.8–2.9 ms), because it drops the second cache pass and the per-measurement toString(). Building the device.toString() + "." prefix once per call would help the warm path too.
| * Fraction of a table's field columns that a scan must select for it to use the batch | ||
| * device-metadata read instead of the per-column path. | ||
| */ | ||
| private static final double DEVICE_METADATA_BATCH_SCAN_RATIO = 0.6; |
There was a problem hiding this comment.
[P2] 60% prefix projections still lose 10–30% cold, and there is no off switch
Even with both changes from the inline comments on TimeSeriesMetadataCache, a projection of the first 60% of the fields (s000–s429 of s000–s699) is 10–30% slower cold than base: 4.0–4.4 ms vs 3.4–3.6 ms. The selected columns fall into two of the device's three leaf groups, but the device-wide read loads all three: 2.0 MB vs 1.5 MB per query. That cost stays until the selective reader exists.
Since the batch path has no runtime switch, I'd lean toward a more conservative threshold, or a config flag, until benchmark numbers are in the description.
Replace the device-wide batch read and its 0.6 projection gate with one getTimeseriesMetadata(Set) entry point: resolve cached columns first, then read each measurement leaf group that holds a selected column once, bloom-probing only the group probe and answering every requested column of the group from that read. get() becomes a single-column wrapper over it. This removes the static ratio, the parallel batch branch, the totalFieldColumnCount planner plumbing and the now-unused i18n exception constant. The cache and wide-device scan tests are rewritten for the new semantics.
An aligned device always has its time column, so when the requested set contains it and the cache or the bloom filter proves it absent, no selected series of that device can be read from this file. The per-column loader returned null at that point; the set-based loader kept probing the remaining measurements, and a bloom-filter false positive on any of them sent the device metadata index lookup into the reader, which throws when the device is not in the file (e.g. a seq file that only holds a subset of the devices read by cross compaction). Probe the time column first and stop the load when it resolves to null. Found by ReadPointAlignedCrossCompactionTest.test21.
The requested measurements were only added to the pending set by the cache pass. The cache-disabled and external-TsFile branch skipped that pass without seeding the set, so it stayed empty, no leaf group was read and the call returned nothing: read_tsfile queries returned zero rows on every external TsFile. Seed pending from the requested set in that branch and add a regression test that loads an external file through getTimeseriesMetadata with the device metadata offset.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues alter per-series counters, allow repeated metadata reads and unnecessary sibling loading, and leave shared test state dependent on execution order.
4 open issues
2 resolved since last review
Review effort: Balanced
Count disk lookups per requested measurement, cache hits included in the same total, as the per-column path did, so EXPLAIN ANALYZE and the monitoring counters stay comparable; leaf-read behavior stays covered by the cache tests rather than by the counter. When the metadata cache is disabled, filter the group read by the requested measurements instead of allSensors, so a non-aligned get() does not deserialize and drop its siblings' chunk metadata. Restore the shared test context to SERIES_SCAN after the external-file test, and document that the once-per-group guarantee needs a bloom filter.
Keep only the helpers with several call sites or real logic: inline the time-column and bloom predicates, rebuild the probe order where it is used, and return the cache-hit count from loadLeafGroups instead of wrapping it in a one-field class.
Replace the per-probe leaf-group loop with one call to TsFileSequenceReader.readTimeseriesMetadata(device, offset, measurements, returnedMeasurements, ...), which resolves the pending measurements against the leaf groups that hold them, reads each visited group once, and also returns the group siblings a single-column call needs for cache warming. The cache keeps the cache pass, the bloom-filter device gate, the negative placeholder and the per-series counters; the probe loop, probeFound, the group filter and the per-column bloom probes are gone. The aligned loader now always requests the time column. This needs the tsfile change that adds the set-based overload; the tsfile dependency must be bumped to a version containing it before this branch can be pushed.
The set-based readTimeseriesMetadata overload is not in the dated tsfile snapshot the build used before. Point the dependency at the 2.5.1 snapshot, which will contain it once apache/tsfile#1123 is merged and published. The IoTDB change that uses the new API must not be pushed until that snapshot exists in the configured repositories.
The build must not resolve the shared 2.5.1-SNAPSHOT, whose content can change under it. Depend on the dated 2.5.1-261117-SNAPSHOT that carries the selective metadata reader, and publish that snapshot before pushing the branch so CI no longer resolves a shared moving artifact.
Cache the TimeSeriesMetadataCacheKey hash in the constructor instead of allocating the Objects.hash argument array on every lookup and write, and pass the pending set straight to the tsfile reader when the aligned batch path requests exactly the set it caches.
JackieTien97
left a comment
There was a problem hiding this comment.
The set-based reader is a nice redesign, and it resolves two of the three regressions from my previous review:
- Under cache pressure, steady-state metadata IO is now below base. Repeated select-all reads 1.0–1.3 MB per query against base's 1.7–2.0 MB, and takes 4.3–4.5 ms against 5.1–7.0 ms.
- A 60% prefix projection now reads the same bytes as base.
I reran the same harness on aa52c51 and on its merge base d5c1534. It uses real TsFiles (12 files, aligned and non-aligned devices, 4 chunks per series), counts bytes and reads at the file, and runs each benchmark in its own JVM (two runs per build, -Xmx1024m). Loaded metadata matched base in every scenario. The PR's tests plus ReadPointAlignedCrossCompactionTest (37 cases) pass locally.
ms per query (one device across 12 files), cold / warm:
| Scenario | base | PR |
|---|---|---|
| Dense, 700 fields, select all, cache fits | 5.7 / 1.9–2.2 | 6.6–7.5 / 1.9–2.0 |
| Dense, 700 fields, first 60% | 3.5–3.8 / 1.0–1.4 | 4.0–4.2 / 1.0–1.2 |
| Dense, 1 column | 1.0 | 0.5–0.6 |
| Re-query after evicting 1% / 30% | 5.5–5.9 / 6.9–7.0 | 3.0–3.3 / 5.2–6.0 |
| Repeated select-all, working set larger than the cache (steady state) | 5.1–7.0, 1.7–2.0 MB per query | 4.3–4.5, 1.0–1.3 MB per query |
| 10 random fields absent per file, warm | 2.1–2.5, 0 KB | 3.0–3.2, 1.9 MB / 46 reads per query |
| 10 fields added later (absent in every file), warm | 2.1–2.2, 0 KB | 2.4–2.5, 517 KB / 24 reads per query |
| Sparse, ~20 of 700 fields per file | 5.0–5.2 / 4.9–5.0, warm 0 KB | 2.1–2.3 / 2.2–2.3, warm 59 KB / 24 reads |
| 100-field table | 0.9–1.0 / 0.3 | 1.0 / 0.25 |
| Tree model: 300 non-aligned series, loaded one by one | 2.9–3.1 / 1.1 | 4.1–4.4 / 1.7–2.1 |
| Tree model: 5 of 300 non-aligned series | 0.53 / 0.01 | 0.29–0.40 / 0.03 |
| Cold load, 96 devices x 300 fields, 8 / 32 threads (wall ms) | 221–227 / 222–233 | 163–189 / 160–161 |
Still open (each has an inline comment):
- [P1] A selected column that is absent from a file is never cached as absent, so every warm query re-reads the leaf groups it maps into. Base does no metadata IO for such columns once warm. Adding columns to a wide table leaves every older file without them, so this will be common.
- [P1] The single-column
get()is 33–53% slower cold and 48–96% slower warm. This is the tree-model path (300 non-aligned series per device). Bytes are identical; the cost is per-call allocation, even on cache hits. - [P2] Dense cold loads are still 15–32% slower, and 7–18% for the 60% projection, in line with the numbers in the description.
- [P2] Offset-addressed (external) reads now consult the BloomFilter. The javadoc says this is skipped, and master did skip it.
Also, tsfile.version points at 2.5.1-261117-SNAPSHOT. That snapshot is built from apache/tsfile#1123, which is still open with no maintainer approval and was republished on 11-18. This PR can't merge until that change is reviewed, merged and released.
| } | ||
| if (timeseriesMetadata == null || timeseriesMetadata == NULL_EXISTS_CACHE_PLACE_HOLDER) { | ||
| } | ||
| fillMissingWithNull(requested, result); |
There was a problem hiding this comment.
[P1] Requested measurements that are absent from the file are never remembered, so warm queries keep re-reading their leaf groups
readAndMerge caches a placeholder only for series the reader returned with count == 0. A requested measurement that the file does not contain at all only gets null here; nothing goes into the LRU. On the next query it misses the cache again, which triggers readTimeseriesMetadata again, and the reader re-reads the leaf group the measurement maps into.
Master avoids this in two ways. The per-measurement BloomFilter check rejects such a column without IO, and on a false positive the old get() caches NULL_EXISTS_CACHE_PLACE_HOLDER. So once warm, master does no metadata IO for absent columns.
Measured warm, with a cache large enough for the working set (4 devices x 12 files x 700 fields):
- 10 random fields absent per file: PR 1.9 MB and 46 reads per query, 3.0–3.2 ms; base 0 KB and 2.1–2.5 ms (15 KB with a file-level time index).
- 10 fields added later, so absent in every file: PR 517 KB and 24 reads per query, which is the last group of every file, re-read each time; base 0 KB.
- Sparse (20 of 700 present): PR 59 KB and 24 reads on every query; base 0 KB. It is still about 2x faster on CPU.
Adding columns to a wide table leaves every older file without them, so this hits ordinary SELECT * queries.
Suggestion: after the read, cache NULL_EXISTS_CACHE_PLACE_HOLDER for every requested measurement that is still missing, when !ignoreNotExists && isMayCacheNonExistSeries() (the same rule as before). With ignoreNotExists no placeholders are allowed, so in calls that are already mostly cache hits, the remaining measurements could go through the BloomFilter before deciding to read.
| QueryContext queryContext, | ||
| long[] deviceMetadataIndexNodeOffset) | ||
| throws IOException { | ||
| return getTimeseriesMetadata( |
There was a problem hiding this comment.
[P1] Single-column get() is 33–53% slower cold and 48–96% slower warm (tree-model path)
Benchmark: 8 non-aligned devices x 300 series x 12 files, with each series loaded through FileLoaderUtils.loadTimeSeriesMetadata as tree-model scans do. Cold went from 2.9–3.1 to 4.1–4.4 ms per device, and warm from 1.1 to 1.7–2.1 ms. Bytes and reads are identical, so the extra time is per-call CPU.
Every call, including a cache hit, now:
- concatenates
device.toString() + "."; - allocates a
HashMap, aHashSet, a singleton set and a secondTimeSeriesMetadataCacheKey; - goes through
getTimeseriesMetadataand its timing block, then looks the result up in the map.
On a miss it also copies allSensors into returned.
Could get() keep a fast path: call lruCache.getIfPresent(key), return a copy (plus statistics) on a hit, and delegate to the set-based path only on a miss? Computing devicePrefix only when the BloomFilter is actually consulted would also help the set-based path.
| || resource.getTimeIndexType() == ITimeIndex.FILE_TIME_INDEX_TYPE; | ||
|
|
||
| Map<String, TimeseriesMetadata> metadata = | ||
| cache.getTimeseriesMetadata( |
There was a problem hiding this comment.
[P2] Dense cold loads are still 15–32% slower than base
For a 700-field device, select-all cold measured 6.6–7.5 ms against base's 5.7 ms, and the first-60% projection 4.0–4.2 ms against 3.5–3.8 ms. Bytes are equal, and this matches the 17–37% in the description.
The extra work appears to be per call, repeated for every device and file:
- building the
pendingandresultsets for 701 names; - the reader turning
measurementsinto aTreeSetand back into a list, i.e. sorting 701 strings; getChildIndexEntriesover that list.
The selected set is identical for every device and file of a scan. Sorting it once per scan and passing the sorted list through (which needs the tsfile API to accept a pre-sorted list) should remove most of the repeated cost. A profile would confirm which part dominates.
| BloomFilter bloomFilter = | ||
| loadBloomFilter( | ||
| filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, externalTsFile); | ||
| if (deviceAbsent(requested, result, bloomFilter, devicePrefix)) { |
There was a problem hiding this comment.
[P2] Offset-addressed (external) reads now depend on the file's BloomFilter
The javadoc of deviceMetadataIndexNodeOffset says BloomFilter filtering is skipped for external TsFiles, but this branch still calls loadBloomFilter and deviceAbsent. An external device can therefore be skipped as absent based on that file's BloomFilter.
Master skipped the BloomFilter whenever an offset was given (if (deviceMetadataIndexNodeOffset == null) in both branches of the old get()). Presence is already known in that case, because the offset comes from iterating the file's device index. If an external file's BloomFilter does not contain device.toString() + "." (for example, a file from another writer), read_tsfile would silently drop that device.
Could you skip the BloomFilter when deviceMetadataIndexNodeOffset != null, as before?
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
9 open issues
This setstsfile.versionto a SNAPSHOT, which is generally not merge-safe/reproducible (builds… · New The Javadoc says bloom filtering is skipped whendeviceMetadataIndexNodeOffsetis non-null, but… · NewloadBloomFilterTimeis always recorded as 0 because it’s never updated around… · NewloadBloomFilterTimeis always recorded as 0 because it’s never updated around… · NewloadBloomFilterTimeis always recorded as 0 because it’s never updated around… · NewloadBloomFilterTimeis always recorded as 0 because it’s never updated around… · New This test createstarget/external-cache-disabled.tsfilebut never deletes it. That can pollute… · New This test createstarget/external-cache-disabled.tsfilebut never deletes it. That can pollute… · New This busywhile (true)loop can spin indefinitely ifhasNextFile()keeps returning… · New
4 resolved since last review
Review effort: Lite
| <tsfile.version>2.5.1-261117-SNAPSHOT</tsfile.version> | ||
| <i18n.locale>en</i18n.locale> | ||
| <tsfile.locale.opt/> | ||
| </properties> |
| * Load statistics are counted per requested measurement. | ||
| * | ||
| * @param deviceMetadataIndexNodeOffset non-null for external tsfiles, which address the device | ||
| * metadata index by offset; bloom filtering is skipped in that case |
| long startTime = System.nanoTime(); | ||
| long cachePhaseEndTime = 0; | ||
| long loadBloomFilterTime = 0; | ||
| long cacheHitCount = 0; | ||
| long diskLookupCount = 0; |
| BloomFilter bloomFilter = | ||
| loadBloomFilter(filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, false); |
| BloomFilter bloomFilter = | ||
| loadBloomFilter( | ||
| filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, externalTsFile); |
| recordLoadStatistics( | ||
| queryContext, | ||
| cacheHitCount, | ||
| cacheLookupTime, | ||
| diskLookupCount, | ||
| diskLookupTime, | ||
| loadBloomFilterTime); |
| try (TsFileWriter writer = new TsFileWriter(file)) { | ||
| writer.registerAlignedTimeseries( | ||
| deviceID, | ||
| Arrays.asList( | ||
| new MeasurementSchema("s1", TSDataType.INT32), | ||
| new MeasurementSchema("s2", TSDataType.INT32))); | ||
| for (int time = 0; time < 3; time++) { | ||
| TSRecord record = new TSRecord(deviceID, time); | ||
| record.addPoint("s1", time); | ||
| record.addPoint("s2", time); | ||
| writer.writeRecord(record); | ||
| } | ||
| } | ||
| cache.clear(); | ||
| bloomFilterCache.clear(); | ||
| long[] offset; | ||
| try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath())) { | ||
| LazyTsFileDeviceIterator iterator = new LazyTsFileDeviceIterator(reader); | ||
| assertTrue(iterator.hasNext()); | ||
| iterator.next(); | ||
| offset = iterator.getCurrentDeviceMeasurementNodeOffset(); | ||
| } | ||
| // external tsfiles bypass the metadata cache: every requested measurement must still be read | ||
| EnvironmentUtils.TEST_QUERY_FI_CONTEXT.setQueryDataSourceType( | ||
| QueryDataSourceType.EXTERNAL_TSFILE_SCAN); | ||
| try { | ||
| Set<String> allSensors = new HashSet<>(Arrays.asList("", "s1", "s2")); | ||
| TsFileResource resource = new TsFileResource(file); | ||
| resource.setStatusForTest(TsFileResourceStatus.NORMAL); | ||
| Map<String, TimeseriesMetadata> result = | ||
| cache.getTimeseriesMetadata( | ||
| file.getAbsolutePath(), | ||
| resource.getTsFileID(), | ||
| deviceID, | ||
| allSensors, | ||
| allSensors, | ||
| false, | ||
| false, | ||
| EnvironmentUtils.TEST_QUERY_FI_CONTEXT, | ||
| offset); | ||
| assertEquals(allSensors.size(), result.size()); | ||
| for (String measurement : allSensors) { | ||
| assertNotNull(measurement, result.get(measurement)); | ||
| assertEquals(measurement, result.get(measurement).getMeasurementId()); | ||
| } | ||
| } finally { | ||
| EnvironmentUtils.TEST_QUERY_FI_CONTEXT.setQueryDataSourceType( | ||
| QueryDataSourceType.SERIES_SCAN); | ||
| cache.clear(); | ||
| bloomFilterCache.clear(); |
| EnvironmentUtils.TEST_QUERY_FI_CONTEXT.setQueryDataSourceType( | ||
| QueryDataSourceType.SERIES_SCAN); | ||
| cache.clear(); | ||
| bloomFilterCache.clear(); |
| while (true) { | ||
| Optional<Boolean> hasNextFile = seriesScanUtil.hasNextFile(); | ||
| if (hasNextFile.isPresent()) { | ||
| return hasNextFile.get(); | ||
| } | ||
| } |
| if (!allSensors.contains("")) { | ||
| // the aligned loader always needs the time column, even if the caller's set omits it | ||
| allSensors = new HashSet<>(allSensors); | ||
| allSensors.add(""); | ||
| } |
There was a problem hiding this comment.
allSensors will never contain "", so just allSensors.add("") is enough.
| queryContext.getQueryStatistics().getLoadBloomFilterActualIOSize()::addAndGet; | ||
| boolean cacheHit = true; | ||
| boolean externalTsFile = queryContext.isExternalTsFileScan(); | ||
| String devicePrefix = device.toString() + IoTDBConstant.PATH_SEPARATOR; |
There was a problem hiding this comment.
lazily init it while you really need this.
| cachePhaseEndTime = System.nanoTime(); | ||
| // serialize reads of the same device+file (same lock key as get) so threads don't each | ||
| // walk the device metadata index and race on the shared reader | ||
| synchronized (deviceLock(device.toString(), filePath)) { |
There was a problem hiding this comment.
create a local field to save device.toString() and reuse it.
| * absent, no selected series of the device can be read from this file. For a single requested | ||
| * series, the bloom filter can rule that series out directly. | ||
| */ | ||
| private static boolean deviceAbsent( |
There was a problem hiding this comment.
we need to use bloomfilter to judge whether each timesereis exists and directly update result if bloomfilter say one sensor doesn't exists.
| * Reads the pending measurements from the leaf groups that hold them. When the cache is enabled | ||
| * the group siblings are returned as well, so a later single-column get() finds them in the LRU. | ||
| */ | ||
| private void readAndMerge( |
There was a problem hiding this comment.
add java doc to explain what's the meaning of each parameter, especially for the three set
| if (requested.contains(measurement)) { | ||
| result.put(measurement, null); | ||
| } | ||
| cacheNonExistenceIfNeeded(tsFileID, device, measurement, cacheEnabled, ignoreNotExists); |
There was a problem hiding this comment.
count = 0 doesn't mean this sensor doesn't exist
| if (cacheEnabled) { | ||
| lruCache.put(new TimeSeriesMetadataCacheKey(tsFileID, device, measurement), metadata); | ||
| } | ||
| if (requested.contains(measurement)) { |
There was a problem hiding this comment.
no need to do this contains again here, all loaded entries should only contains what you really need.
| } | ||
| if (timeseriesMetadata == null || timeseriesMetadata == NULL_EXISTS_CACHE_PLACE_HOLDER) { | ||
| } | ||
| fillMissingWithNull(requested, result); |
There was a problem hiding this comment.
you can remove sensor from pending set in tsfile function call, after which, the remaining is what really not existing sensors.
There was a problem hiding this comment.
And for these really not existing sensors, besides putting them in result map, you should also put them back in cache
| private static void fillMissingWithNull( | ||
| Set<String> requested, Map<String, TimeseriesMetadata> result) { | ||
| for (String measurement : requested) { | ||
| result.putIfAbsent(measurement, null); | ||
| } | ||
| } |
| // a key is built for every lookup and every write: compute the hash once instead of | ||
| // allocating the Objects.hash argument array on each call | ||
| this.hash = Objects.hash(tsFileID, device, measurement); |
There was a problem hiding this comment.
actually you don't need to do so, jvm will store it for each object


Description
A device's
TimeseriesMetadatais stored in measurement leaf groups (up tomax_degree_of_index_nodeseries per group). Loading it column by column walks the device metadata index and deserializes the containing leaf group once per requested column, so a scan that selects almost all columns pays that traversal once per column, and every column absent from a file costs its own cache probe and BloomFilter check. In the motivating workload (one device with ~700 columns, ~20 written per day, 12 files) loading the metadata of one device across its files dominated the query.This PR resolves an entire projection against the TsFile in one set-based read: the selective reader lives in apache/tsfile#1123, and the DataNode metadata cache calls it once per (device, file).
What changed
tsfile (apache/tsfile#1123)
MetadataIndexNode.getChildIndexEntries(keys, exactSearch)searches the floor child entry of a sorted key set in one pass, choosing a monotonic binary search or a merge scan by cost and reusing onePairinstance per child. Duplicate keys and mixed device-ID implementations (a plain key against string-array children) are handled.TsFileSequenceReader.readTimeseriesMetadata(device, offset, measurements, returnedMeasurements, ignoreNotExistDevice, ioSizeRecorder)locates the device's measurement metadata index node (or reads the supplied offset for external TsFiles), groups the measurements by child, reads each visited internal node and each visited leaf group once, and decodes chunk metadata only for the measurements the call returns.returnedMeasurementsmay be a superset ofmeasurements, which lets a single-column call warm its group siblings.DataNode
TimeSeriesMetadataCache.getTimeseriesMetadatais now: per-measurement cache pass → BloomFilter device-absence check (no reader is opened for a device the BloomFilter rules out) → onereadTimeseriesMetadatacall under the per-(device, file) lock → cache and assemble the result.Compared with master,
FileLoaderUtilsno longer loops over the selected columns callingget()for each of them: it makes onegetTimeseriesMetadatacall per (device, file), so a cold query performs one cache pass, one BloomFilter device check and one batch read instead of a per-column cache probe, a per-column BloomFilter check and a per-miss lock and leaf-group read.get()remains the single-column entry point used by tree-model scans and keeps warming the series that share the leaf group. (Earlier revisions of this PR used a device-wide read gated by a 0.6 projection ratio; the selective reader replaces that design.) Per-series cache/disk counters stay comparable with the per-column path; disk lookups are counted before the read, and lock wait is no longer charged to cache time.tsfile.versionmoves to2.5.1-261117-SNAPSHOT. That snapshot is published to the Apache snapshots repository for this PR and must be replaced by a released TsFile version before merge.Example
Table with 700 field columns; each of 12 TsFiles holds 20 of them (rotating);
SELECT *for one device:Dense 700-column device, select all, cold cache: the leaf-group reads and bytes are unchanged (36 reads, ~2020 KB; the batch read fetches the device metadata index once instead of once per group, so it reads slightly fewer index bytes). Cold time goes from 12.2–12.4 ms to 14.3–16.7 ms: with every column present, the per-column path already served most columns from the group-cache hits produced by its first read, while the batch path additionally builds the request and result sets for the call. Warm queries are unchanged (~1.5 ms).
Benchmark
Same machine and the same TsFile snapshot for both variants, cold metadata cache unless noted; bytes and read counts come from
QueryStatistics.The sparse wide-table case that motivated the change improves 3.5–4.5x; the dense and prefix projections trade a small cold-path overhead for the same I/O and unchanged warm queries.
Tests
MetadataIndexNodeTestcovers both batch search strategies, duplicate keys, mixed device-ID types, keys before the first child and exact search;TimeSeriesMetadataReadTestcompares the batch read with per-measurement reads (statistics and chunk metadata), covers absent names and devices, the offset-addressed variant, sibling warming, an internal measurement node, and asserts that absent names add no reads.TimeSeriesMetadataCacheTest(cache hits, placeholders, BloomFilter exclusion, external TsFile),AlignedSeriesScanBatchTest(dense, sparse and all-null projections end to end), andReadPointAlignedCrossCompactionTestfor the device-absent path.🤖 Generated with Claude Code