Skip to content

Batch scan TimeSeriesMetadata for wide-table queries - #19198

Open
ColinLeeo wants to merge 28 commits into
apache:masterfrom
ColinLeeo:colin/batch-scan-query-a2e8cc
Open

ColinLeeo wants to merge 28 commits into
apache:masterfrom
ColinLeeo:colin/batch-scan-query-a2e8cc

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Description

A device's TimeseriesMetadata is stored in measurement leaf groups (up to max_degree_of_index_node series per group). Loading it column by column walks the device metadata index and deserializes the containing leaf group once per requested column, so a scan that selects almost all columns pays that traversal once per column, and every column absent from a file costs its own cache probe and BloomFilter check. In the motivating workload (one device with ~700 columns, ~20 written per day, 12 files) loading the metadata of one device across its files dominated the query.

This PR resolves an entire projection against the TsFile in one set-based read: the selective reader lives in apache/tsfile#1123, and the DataNode metadata cache calls it once per (device, file).

What changed

tsfile (apache/tsfile#1123)

  • MetadataIndexNode.getChildIndexEntries(keys, exactSearch) searches the floor child entry of a sorted key set in one pass, choosing a monotonic binary search or a merge scan by cost and reusing one Pair instance per child. Duplicate keys and mixed device-ID implementations (a plain key against string-array children) are handled.
  • TsFileSequenceReader.readTimeseriesMetadata(device, offset, measurements, returnedMeasurements, ignoreNotExistDevice, ioSizeRecorder) locates the device's measurement metadata index node (or reads the supplied offset for external TsFiles), groups the measurements by child, reads each visited internal node and each visited leaf group once, and decodes chunk metadata only for the measurements the call returns. returnedMeasurements may be a superset of measurements, which lets a single-column call warm its group siblings.

DataNode

TimeSeriesMetadataCache.getTimeseriesMetadata is now: per-measurement cache pass → BloomFilter device-absence check (no reader is opened for a device the BloomFilter rules out) → one readTimeseriesMetadata call under the per-(device, file) lock → cache and assemble the result.

Compared with master, FileLoaderUtils no longer loops over the selected columns calling get() for each of them: it makes one getTimeseriesMetadata call per (device, file), so a cold query performs one cache pass, one BloomFilter device check and one batch read instead of a per-column cache probe, a per-column BloomFilter check and a per-miss lock and leaf-group read. get() remains the single-column entry point used by tree-model scans and keeps warming the series that share the leaf group. (Earlier revisions of this PR used a device-wide read gated by a 0.6 projection ratio; the selective reader replaces that design.) Per-series cache/disk counters stay comparable with the per-column path; disk lookups are counted before the read, and lock wait is no longer charged to cache time.

tsfile.version moves to 2.5.1-261117-SNAPSHOT. That snapshot is published to the Apache snapshots repository for this PR and must be replaced by a released TsFile version before merge.

Example

Table with 700 field columns; each of 12 TsFiles holds 20 of them (rotating); SELECT * for one device:

  • Before: 701 per-column cache lookups per file, and every column absent from the file took the per-device lock, a second cache check and a BloomFilter probe — 8,412 lookups and about 8,200 lock/probe cycles per query, 7.7–7.8 ms.
  • After: one cache pass, one lock per (device, file) and one batch read per file (a single leaf group in this fixture), 1.7–2.3 ms cold and 1.2–1.5 ms warm.

Dense 700-column device, select all, cold cache: the leaf-group reads and bytes are unchanged (36 reads, ~2020 KB; the batch read fetches the device metadata index once instead of once per group, so it reads slightly fewer index bytes). Cold time goes from 12.2–12.4 ms to 14.3–16.7 ms: with every column present, the per-column path already served most columns from the group-cache hits produced by its first read, while the batch path additionally builds the request and result sets for the call. Warm queries are unchanged (~1.5 ms).

Benchmark

Same machine and the same TsFile snapshot for both variants, cold metadata cache unless noted; bytes and read counts come from QueryStatistics.

Scenario base (master) this PR
700 columns, select all, cold / warm 12.2–12.4 / 1.35 ms 14.3–16.7 / 1.6 ms
700 columns, first 60%, cold / warm 2.9–4.1 / 0.6 ms 4.9–6.4 / 0.6 ms
Sparse: 20 of 700 columns per file, cold / warm 7.7–7.8 / 5.0–5.3 ms 1.7–2.3 / 1.2–1.5 ms
Repeated select-all, working set larger than the 61 MB cache (steady state) 6.0–13.0 ms (median 4.7–5.3) 5.4–6.2 ms (median 3.7–4.2)
Bytes per query (select-all / 60% / sparse) 2021.0 / 1475.6 / 64.1 KB 2019.9 / 1475.1 / 59.2 KB

The sparse wide-table case that motivated the change improves 3.5–4.5x; the dense and prefix projections trade a small cold-path overhead for the same I/O and unchanged warm queries.

Tests

  • tsfile: MetadataIndexNodeTest covers both batch search strategies, duplicate keys, mixed device-ID types, keys before the first child and exact search; TimeSeriesMetadataReadTest compares the batch read with per-measurement reads (statistics and chunk metadata), covers absent names and devices, the offset-addressed variant, sibling warming, an internal measurement node, and asserts that absent names add no reads.
  • DataNode: TimeSeriesMetadataCacheTest (cache hits, placeholders, BloomFilter exclusion, external TsFile), AlignedSeriesScanBatchTest (dense, sparse and all-null projections end to end), and ReadPointAlignedCrossCompactionTest for the device-absent path.

🤖 Generated with Claude Code

A table scan that selects (nearly) all field columns of a device previously
deserialized the device's TimeSeriesMetadata column by column, re-walking the
device metadata index once per leaf node. Add a batch path that reads the whole
device's metadata in a single pass via TsFileSequenceReader.readDeviceMetadata,
populating the per-measurement cache and building the aligned metadata through
the existing modification/loader logic.

The batch path is taken when the selected field columns reach a configurable
fraction (device_metadata_batch_scan_ratio, default 0.8) of the table's total
field columns.
The batch path read a device's TimeseriesMetadata via
TsFileSequenceReader.readDeviceMetadata, which deserializes only the
header and statistics and leaves chunkMetadataList null. The downstream
DiskAlignedChunkMetadataLoader dereferences that list in
getCopiedChunkMetadataList, so a wide-table scan NPE'd at query time, and
the null-chunk entries also polluted the shared per-measurement LRU cache
(breaking later narrow queries on the same device).

Switch to getDeviceTimeseriesMetadata, which deserializes chunk metadata
too, and adapt the unit test to the list-based API, asserting that chunk
metadata flows through to the result.
Remove the unreachable time-column-only branch in the batch loader (the
batch gate already requires a non-empty selection), and validate
device_metadata_batch_scan_ratio so a non-positive value falls back to
the default instead of routing every table scan through the batch path.
Document the valid range in the properties template.
Remove the ignoreNotExists gate so the batch path is also taken for files
with a device-level time index, where previously only file-level-index
files (or recovery) used it. When the batch read finds the device is
missing from the file, replicate the per-column path: skip silently when
ignoreNotExists is set, otherwise throw CorruptedTsFileException to surface
the time-index/metadata inconsistency.
Remove the device_metadata_batch_scan_ratio config parameter and hardcode
the threshold at 0.6 (previously configurable, default 0.8) so the batch
path covers a wider set of scans without requiring configuration.
Serialize concurrent reads of the same device+file in
getDeviceTimeSeriesMetadata (matching the per-column get) so threads don't
each deserialize the whole device and race on the shared reader. Also attach
the missing device id to the CorruptedTsFileException thrown when a device is
absent from a device-level time index file.
Count every value column (FIELD + ATTRIBUTE) as the batch scan ratio
denominator instead of only FIELD columns, matching the selected-column
numerator. Cache empty (count == 0) series as the null placeholder in the
batch read, matching the per-column get, so later per-column reads don't
re-read them. Add a unit test for the absent-device (empty map) path.
Cover the loadAlignedTimeSeriesMetadataBatch empty-map branch: a device
absent from a device-level time index file throws CorruptedTsFileException,
while with a file-level time index (or during recovery) it is skipped.
Drive AlignedSeriesScanUtil with all value columns selected and a matching
total field column count so the batch path is taken, and assert the aligned
pages are still read correctly.
The batch read offers no benefit for a single-column device (one leaf
either way) and it bypasses the per-column cache's debug logging, which
IoTDBDebugQueryIT.tableTest relies on. Require at least two value columns
before taking the batch path.
getDeviceTimeSeriesMetadata always deserialized the device's whole
metadata index and then re-populated the cache, never reading it, so
repeated wide-table scans were slower than the per-column path that it
populates. Check the per-measurement cache first, read the device only
when one of the requested measurements is missing, and cache just those
measurements (plus the null placeholders for absent ones).

Align the absent-device behavior with the per-column get(): skip the
device when the file's bloom filter proves it is missing, otherwise
raise the read IOException so the existing wrapper records the
suspicious-disk metric. Require a null device offset for the batch path
instead of relying on the zero sentinel used by external scans, and
extract the aligned metadata assembly shared by both paths.

Count only FIELD columns in the ratio denominator to match the
smaller numerator, and extend the tests to cover the cache hit, the
bloom-filter decisions, the gating boundary and the batch path actually
being taken.
The batch metadata scan bypasses TimeSeriesMetadataCache.get(..., debug=true)
and does not emit equivalent hit/miss diagnostics, so a DEBUG SELECT on a
wide table lost its metadata logs. Make the batch gate take the debug and
offset-addressed state explicitly so DEBUG queries keep the per-column
path, and cover both conditions in the gating unit test.
getDeviceTimeSeriesMetadata read the whole device metadata index and only
then discovered the device was absent, while the per-column get() can skip
that read by probing the file's bloom filter first. Probe the cached time
column entry before the read, keep the same missing-device skip/throw
rules, and account the bloom filter cache/disk/time statistics like get().

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the optimization. I checked out 3fba980 and ran the PR's tests locally (FileLoaderUtilsTest, TimeSeriesMetadataCacheTest, AlignedSeriesScanBatchTest); all pass. I also added a throwaway probe test (not pushed) that writes a real aligned device with 700 columns (three measurement leaf groups) and compares getDeviceTimeSeriesMetadata against the per-column get(): all 701 series (time column plus 700 fields) matched, including statistics and chunk-metadata counts. So I found no wrong-result issue. The comments below are mostly about performance, diagnostics and metrics.

Main points:

  1. [P1] The batch path only counts as a cache hit when every selected sensor is in the LRU. If any selected column is absent from a file and that absence can't be cached as a placeholder (ignoreNotExists=true or may_cache_nonexist_series=false), every query re-reads the whole device. The per-column path doesn't (TimeSeriesMetadataCache L362).
  2. [P1] The missing-device error message describes the opposite inconsistency (L491).
  3. [P2] Metadata IO size and load counts are no longer comparable with the per-column path (L483, L370).
  4. [P2] The I/O saving only exists when the selected columns span more than one measurement leaf group (more than 256 series by default), and the gate doesn't reflect that (FileLoaderUtils L447). Could the same batching be done inside the existing get() miss path instead of a parallel path (L342)?
  5. The PR description is out of date. It still describes a configurable device_metadata_batch_scan_ratio (default 0.8), an IoTDBConfig change, an ignoreNotExists gate and readDeviceMetadata. The code hard-codes 0.6 with no switch, has no such gate, and uses getDeviceTimeseriesMetadata. Could you update it and add benchmark numbers for a wide device (e.g. 800 columns, cold and warm cache) compared with master? That would make it easier to judge the 0.6 threshold.

Reuse cached measurements and batch-read only the missing ones, probing
the per-measurement bloom filter first so absent columns do not force a
whole-device read; the all-or-nothing cache check is gone. Count load
statistics per series so they stay comparable with the per-column path,
and document the batch read's missing IO-size accounting.

Report a dedicated error naming the device and the TsFile when the
resource records a device that the file metadata lacks (en + zh), share
the load statistics recording between both paths, and pass the external
flag through the bloom filter lookup.

Only batch when the table has more field columns than the measurement
index fan-out, so small devices stay on the per-column path.

Add a real 700-column batch-vs-per-column comparison, a 300-column
end-to-end batch scan spanning two leaf groups, and gate boundary cases.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved batch-path issues can trigger unnecessary device reads and absent-device I/O errors, while regression tests leave sparse all-null rows and multi-leaf field values unverified.

5 open issues
What changed in this PR

This PR reduces repeated metadata-index traversal in DataNode wide-table queries by loading a device’s metadata in one batch when most field columns are selected.

Changes:

  • Adds cache- and bloom-filter-aware batch loading with per-series query statistics.
  • Gates batching by table width and a 60% projection threshold, excluding DEBUG and offset-addressed scans.
  • Shares aligned metadata assembly, adds localized missing-device errors, and introduces regression tests.
File Description
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​storageengine/​dataregion/​read/​reader/​series/​AlignedSeriesScanBatchTest.java Adds a real-file wide-device batch scan regression.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​storageengine/​buffer/​TimeSeriesMetadataCacheTest.java Tests batch loading, caching, missing devices, and metadata equivalence.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​FileLoaderUtilsTest.java Tests batch eligibility boundaries and exclusions.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​storageengine/​buffer/​TimeSeriesMetadataCache.java Implements cache-aware device metadata loading and statistics accounting.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​plan/​planner/​plan/​parameter/​SeriesScanOptions.java Carries the table’s total field-column count.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​plan/​planner/​DataNodeTableOperatorGenerator.java Supplies field counts for eligible table scans.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​relational/​ExternalTsFileSeriesScanUtil.java Keeps external offset-addressed scans on the per-column path.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​FileLoaderUtils.java Selects the batch path and shares aligned metadata assembly.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​queryengine/​execution/​operator/​source/​AlignedSeriesScanUtil.java Passes the field count into metadata loading.
iotdb-core/​datanode/​src/​main/​i18n/​zh/​org/​apache/​iotdb/​db/​i18n/​StorageEngineMessages.java Adds the Chinese missing-device error message.
iotdb-core/​datanode/​src/​main/​i18n/​en/​org/​apache/​iotdb/​db/​i18n/​StorageEngineMessages.java Adds the English error naming the device and TsFile.

Review effort: Max

Skip the device read when the time column is known absent, so a bloom
filter false positive on a field cannot force an absent-device read.
Fall back to per-column lookups for a small remainder (one or two
measurements), which reads a single leaf group instead of the whole
device, and count the cold-call cache probe and lock wait in the disk
lookup time.

Strengthen the wide-scan tests: assert timestamps and every field value,
compare a sparse batch scan against the per-column path, cover an
all-null projection under ignoreAllNullRows, and add cache regressions
for the filtered time column and the single-miss fallback.
TsFileSequenceReader is an external apache/tsfile dependency and has no
batch API that accepts a measurement set, so the batch path still reads
the whole device and filters afterwards. Note the constraint and the
existing mitigations (bloom filter, per-column fallback) at the call
site; a truly selective reader needs an upstream change.
Load the remaining misses through the per-column get() when the cache
already holds most of the request: it touches only their leaf groups
instead of re-reading the whole device. Replace the GC-sensitive
WeakHashMap monitor with fixed lock stripes, and make the returned map
complete for allSensors on every early return.

Give aggregation table scans a totalFieldColumnCount so they can use the
batch path, and make the fan-out gate span two leaf groups (>=), fixing
its off-by-one and comment.

Tests: production ignoreAllNullRows=false coverage including all-null
projections, batch negative-caching parity, copy-on-return, a three-miss
eviction case, the multi-leaf premise, and per-chunk offset/statistics
comparison.
@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 16, 2026 03:37
Copilot stopped reviewing on behalf of ColinLeeo due to an error November 16, 2026 03:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Aggregation scans can unintentionally take the whole-device batch path, while cold-cache fallback underreports metadata-load time by omitting initial cache probes and device-lock waits.

2 open issues

Review effort: Balanced

When a cold batch call delegates the remaining measurements to the
per-column get(), the cache probes and lock wait were assigned to cache
time while the cache hit count was zero, so neither branch reported
them. Attribute the phase to disk time when there are no cache hits and
allow a disk time with a zero count to be recorded.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. Status of my earlier comments: 9 of 12 are fixed, including both P1s. Still open by design: the batch read's IO bytes are not reported, the 0.6 ratio has no off switch, and the selective reader is an upstream follow-up.

Since this PR is a performance change, I benchmarked it against its base. A JUnit harness (not pushed) drives FileLoaderUtils.loadAlignedTimeSeriesMetadata over real TsFiles written with TsFileWriter (12 files, aligned devices, 4 chunks per series). It counts the bytes and read calls that reach the file and runs each variant in its own JVM (surefire defaults with -Xmx1024m, which gives a ~61 MB TimeSeriesMetadataCache). I measured f8b9c83 against ca8074d. The only batch-path change since then, 469394f, is time accounting, so the numbers apply to 6ab4b8e. Loaded metadata was identical to base in every scenario (checksums), so there is no correctness issue here.

Two runs per variant; ms per query (one device across 12 files), cold / warm:

Scenario base PR PR + the two changes below
Dense, 700 fields, select all, cache fits 5.1–5.3 / 1.7–1.8 8.5 / 1.5–2.7 5.5–5.9 / 1.5–1.7
Dense, 700 fields, select first 60% 3.4–3.6 / 1.0–1.1 5.3–5.4 / 1.0 4.0–4.4 / 1.1
Repeated select-all, working set larger than the metadata cache (steady state) 4.5–9.9 ms, 1.7–4.1 MB per query 10.1–16.1 ms, 11–15 MB per query 4.4–5.4 ms, 1.0–2.0 MB per query
Sparse, ~20 of 700 fields per file (the motivating case) 4.8–4.9 / 4.7 2.8–2.9 / 2.1 1.9–2.0 / 2.1
100-field table (below the gate) 0.9 / 0.27 unchanged unchanged
Cold load, 96 devices x 300 fields, 8 / 32 threads (wall ms) 225–232 / 237–247 208–220 / 193–210 199–213 / 186–189

The sparse case gains about 2x as intended, and the lock stripes do not hurt concurrency. There are two regressions, though. Each has an inline comment with the change that removed it locally:

  1. [P1] Under metadata-cache pressure, the "cache holds most of the request, so delegate to per-column get()" fallback multiplies metadata reads 3–8x.
  2. [P1] For dense wide devices, a cold call costs 45–70% more time than base. It moves the same bytes in fewer reads, so the extra cost is CPU: every selected measurement goes through two cache passes and a BloomFilter probe before the device read.

Two smaller points:

  • With a 60% prefix projection, the device-wide read still loads a leaf group that no selected column needs, so it stays 10–30% slower even after both changes. That cost remains until the selective reader lands.
  • The description says getDeviceTimeseriesMetadata costs about 2x IO. In these runs the batch read moved the same bytes as the per-column path (2019.9 KB vs 2021.0 KB per query, in 48 vs 72 reads). Also, the gate in the code is >=, while the description says "more than".

I'm happy to share the harness if it helps.

// (later misses in the same group hit the cache) instead of re-reading the whole device.
// Those lookups record their own statistics.
if (possiblyPresent.size() <= MAX_PER_COLUMN_FALLBACK
|| (cacheHitCount > 0 && possiblyPresent.size() <= cacheHitCount)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] This fallback multiplies metadata IO 3–8x under cache pressure

Setup: repeated SELECT * over 8 devices x 12 files x 701 series, which is more than the ~61 MB TimeSeriesMetadataCache holds. In that steady state, most calls arrive with more than half of their keys cached and take this branch. Per query, the PR read 11–15 MB in 393–514 reads (1,520–2,006 readTimeseriesMetadata calls per round), while base read 1.7–4.1 MB in 61–144 reads. That made the PR 10–16 ms per query against base's 4.5–9.9 ms.

Each get() here re-reads and re-inserts the whole leaf group of the missing measurement (up to 256 series, with chunk metadata). Under eviction pressure, those entries do not survive until the next missing measurement of the same group is looked up. Visiting the misses in sorted order did not help (13 MB per query).

Dropping only the majority clause brought it down to 1.0–2.0 MB and 4.3–5.4 ms per query, at or below base. That means keeping possiblyPresent.size() <= MAX_PER_COLUMN_FALLBACK and otherwise doing the single device read. Partial-eviction runs with a cache large enough stayed at base level. Could you remove the clause, or replace it with a group-aware load that reads each needed leaf group once and serves the misses from that read instead of from the cache?

loadBloomFilter(filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, false);
List<String> possiblyPresent = new ArrayList<>(missing.size());
for (String measurement : missing) {
if (bloomFilterMayContain(bloomFilter, device, measurement)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Cold calls on dense wide devices are 45–70% slower than base

With the cache cleared (and large enough for the working set), loading a 700-field device across 12 files took 8.5 ms per query against base's 5.1–5.3 ms. A 60% projection took 5.3–5.4 ms against 3.4–3.6 ms. The bytes were the same and the batch path made fewer reads (48 vs 72 per query), so the extra cost is CPU. On a cold call, every selected measurement goes through:

  • two collectCachedMetadata passes;
  • this probe, which builds device.toString() + "." + measurement and runs k hashes, 700 times;
  • the copies into the possiblyPresent HashSet, the inner result map, putAll and putIfAbsent.

The per-column path probes the filter once per leaf group.

What removed the gap locally, when cacheHitCount == 0:

  • under the lock, re-check only the time-column key, and fall back to the current path if it appeared;
  • probe the BloomFilter for the time column only, then read the device once and fill result directly;
  • consult the filter only for selected measurements the read did not find, to decide whether to cache a placeholder (as get() does).

With that change, dense select-all came out at 5.5–5.9 ms and the 60% projection at 4.0–4.4 ms. It also sped up the sparse case (1.9–2.0 ms cold vs 2.8–2.9 ms), because it drops the second cache pass and the per-measurement toString(). Building the device.toString() + "." prefix once per call would help the warm path too.

* Fraction of a table's field columns that a scan must select for it to use the batch
* device-metadata read instead of the per-column path.
*/
private static final double DEVICE_METADATA_BATCH_SCAN_RATIO = 0.6;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 60% prefix projections still lose 10–30% cold, and there is no off switch

Even with both changes from the inline comments on TimeSeriesMetadataCache, a projection of the first 60% of the fields (s000–s429 of s000–s699) is 10–30% slower cold than base: 4.0–4.4 ms vs 3.4–3.6 ms. The selected columns fall into two of the device's three leaf groups, but the device-wide read loads all three: 2.0 MB vs 1.5 MB per query. That cost stays until the selective reader exists.

Since the batch path has no runtime switch, I'd lean toward a more conservative threshold, or a config flag, until benchmark numbers are in the description.

Replace the device-wide batch read and its 0.6 projection gate with one
getTimeseriesMetadata(Set) entry point: resolve cached columns first,
then read each measurement leaf group that holds a selected column
once, bloom-probing only the group probe and answering every requested
column of the group from that read. get() becomes a single-column
wrapper over it.

This removes the static ratio, the parallel batch branch, the
totalFieldColumnCount planner plumbing and the now-unused i18n
exception constant. The cache and wide-device scan tests are rewritten
for the new semantics.
An aligned device always has its time column, so when the requested set
contains it and the cache or the bloom filter proves it absent, no
selected series of that device can be read from this file. The
per-column loader returned null at that point; the set-based loader kept
probing the remaining measurements, and a bloom-filter false positive on
any of them sent the device metadata index lookup into the reader, which
throws when the device is not in the file (e.g. a seq file that only
holds a subset of the devices read by cross compaction).

Probe the time column first and stop the load when it resolves to null.
Found by ReadPointAlignedCrossCompactionTest.test21.
The requested measurements were only added to the pending set by the
cache pass. The cache-disabled and external-TsFile branch skipped that
pass without seeding the set, so it stayed empty, no leaf group was
read and the call returned nothing: read_tsfile queries returned zero
rows on every external TsFile.

Seed pending from the requested set in that branch and add a regression
test that loads an external file through getTimeseriesMetadata with the
device metadata offset.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues alter per-series counters, allow repeated metadata reads and unnecessary sibling loading, and leave shared test state dependent on execution order.

4 open issues
2 resolved since last review

Review effort: Balanced

Count disk lookups per requested measurement, cache hits included in
the same total, as the per-column path did, so EXPLAIN ANALYZE and the
monitoring counters stay comparable; leaf-read behavior stays covered
by the cache tests rather than by the counter.

When the metadata cache is disabled, filter the group read by the
requested measurements instead of allSensors, so a non-aligned get()
does not deserialize and drop its siblings' chunk metadata. Restore the
shared test context to SERIES_SCAN after the external-file test, and
document that the once-per-group guarantee needs a bloom filter.
Keep only the helpers with several call sites or real logic: inline the
time-column and bloom predicates, rebuild the probe order where it is
used, and return the cache-hit count from loadLeafGroups instead of
wrapping it in a one-field class.
Replace the per-probe leaf-group loop with one call to
TsFileSequenceReader.readTimeseriesMetadata(device, offset,
measurements, returnedMeasurements, ...), which resolves the pending
measurements against the leaf groups that hold them, reads each visited
group once, and also returns the group siblings a single-column call
needs for cache warming.

The cache keeps the cache pass, the bloom-filter device gate, the
negative placeholder and the per-series counters; the probe loop,
probeFound, the group filter and the per-column bloom probes are gone.
The aligned loader now always requests the time column.

This needs the tsfile change that adds the set-based overload; the
tsfile dependency must be bumped to a version containing it before this
branch can be pushed.
The set-based readTimeseriesMetadata overload is not in the dated tsfile
snapshot the build used before. Point the dependency at the 2.5.1
snapshot, which will contain it once apache/tsfile#1123 is merged and
published.

The IoTDB change that uses the new API must not be pushed until that
snapshot exists in the configured repositories.
The build must not resolve the shared 2.5.1-SNAPSHOT, whose content can
change under it. Depend on the dated 2.5.1-261117-SNAPSHOT that carries
the selective metadata reader, and publish that snapshot before pushing
the branch so CI no longer resolves a shared moving artifact.
Cache the TimeSeriesMetadataCacheKey hash in the constructor instead of
allocating the Objects.hash argument array on every lookup and write,
and pass the pending set straight to the tsfile reader when the aligned
batch path requests exactly the set it caches.
@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 18, 2026 02:34

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The set-based reader is a nice redesign, and it resolves two of the three regressions from my previous review:

  • Under cache pressure, steady-state metadata IO is now below base. Repeated select-all reads 1.0–1.3 MB per query against base's 1.7–2.0 MB, and takes 4.3–4.5 ms against 5.1–7.0 ms.
  • A 60% prefix projection now reads the same bytes as base.

I reran the same harness on aa52c51 and on its merge base d5c1534. It uses real TsFiles (12 files, aligned and non-aligned devices, 4 chunks per series), counts bytes and reads at the file, and runs each benchmark in its own JVM (two runs per build, -Xmx1024m). Loaded metadata matched base in every scenario. The PR's tests plus ReadPointAlignedCrossCompactionTest (37 cases) pass locally.

ms per query (one device across 12 files), cold / warm:

Scenario base PR
Dense, 700 fields, select all, cache fits 5.7 / 1.9–2.2 6.6–7.5 / 1.9–2.0
Dense, 700 fields, first 60% 3.5–3.8 / 1.0–1.4 4.0–4.2 / 1.0–1.2
Dense, 1 column 1.0 0.5–0.6
Re-query after evicting 1% / 30% 5.5–5.9 / 6.9–7.0 3.0–3.3 / 5.2–6.0
Repeated select-all, working set larger than the cache (steady state) 5.1–7.0, 1.7–2.0 MB per query 4.3–4.5, 1.0–1.3 MB per query
10 random fields absent per file, warm 2.1–2.5, 0 KB 3.0–3.2, 1.9 MB / 46 reads per query
10 fields added later (absent in every file), warm 2.1–2.2, 0 KB 2.4–2.5, 517 KB / 24 reads per query
Sparse, ~20 of 700 fields per file 5.0–5.2 / 4.9–5.0, warm 0 KB 2.1–2.3 / 2.2–2.3, warm 59 KB / 24 reads
100-field table 0.9–1.0 / 0.3 1.0 / 0.25
Tree model: 300 non-aligned series, loaded one by one 2.9–3.1 / 1.1 4.1–4.4 / 1.7–2.1
Tree model: 5 of 300 non-aligned series 0.53 / 0.01 0.29–0.40 / 0.03
Cold load, 96 devices x 300 fields, 8 / 32 threads (wall ms) 221–227 / 222–233 163–189 / 160–161

Still open (each has an inline comment):

  1. [P1] A selected column that is absent from a file is never cached as absent, so every warm query re-reads the leaf groups it maps into. Base does no metadata IO for such columns once warm. Adding columns to a wide table leaves every older file without them, so this will be common.
  2. [P1] The single-column get() is 33–53% slower cold and 48–96% slower warm. This is the tree-model path (300 non-aligned series per device). Bytes are identical; the cost is per-call allocation, even on cache hits.
  3. [P2] Dense cold loads are still 15–32% slower, and 7–18% for the 60% projection, in line with the numbers in the description.
  4. [P2] Offset-addressed (external) reads now consult the BloomFilter. The javadoc says this is skipped, and master did skip it.

Also, tsfile.version points at 2.5.1-261117-SNAPSHOT. That snapshot is built from apache/tsfile#1123, which is still open with no maintainer approval and was republished on 11-18. This PR can't merge until that change is reviewed, merged and released.

}
if (timeseriesMetadata == null || timeseriesMetadata == NULL_EXISTS_CACHE_PLACE_HOLDER) {
}
fillMissingWithNull(requested, result);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Requested measurements that are absent from the file are never remembered, so warm queries keep re-reading their leaf groups

readAndMerge caches a placeholder only for series the reader returned with count == 0. A requested measurement that the file does not contain at all only gets null here; nothing goes into the LRU. On the next query it misses the cache again, which triggers readTimeseriesMetadata again, and the reader re-reads the leaf group the measurement maps into.

Master avoids this in two ways. The per-measurement BloomFilter check rejects such a column without IO, and on a false positive the old get() caches NULL_EXISTS_CACHE_PLACE_HOLDER. So once warm, master does no metadata IO for absent columns.

Measured warm, with a cache large enough for the working set (4 devices x 12 files x 700 fields):

  • 10 random fields absent per file: PR 1.9 MB and 46 reads per query, 3.0–3.2 ms; base 0 KB and 2.1–2.5 ms (15 KB with a file-level time index).
  • 10 fields added later, so absent in every file: PR 517 KB and 24 reads per query, which is the last group of every file, re-read each time; base 0 KB.
  • Sparse (20 of 700 present): PR 59 KB and 24 reads on every query; base 0 KB. It is still about 2x faster on CPU.

Adding columns to a wide table leaves every older file without them, so this hits ordinary SELECT * queries.

Suggestion: after the read, cache NULL_EXISTS_CACHE_PLACE_HOLDER for every requested measurement that is still missing, when !ignoreNotExists && isMayCacheNonExistSeries() (the same rule as before). With ignoreNotExists no placeholders are allowed, so in calls that are already mostly cache hits, the remaining measurements could go through the BloomFilter before deciding to read.

QueryContext queryContext,
long[] deviceMetadataIndexNodeOffset)
throws IOException {
return getTimeseriesMetadata(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Single-column get() is 33–53% slower cold and 48–96% slower warm (tree-model path)

Benchmark: 8 non-aligned devices x 300 series x 12 files, with each series loaded through FileLoaderUtils.loadTimeSeriesMetadata as tree-model scans do. Cold went from 2.9–3.1 to 4.1–4.4 ms per device, and warm from 1.1 to 1.7–2.1 ms. Bytes and reads are identical, so the extra time is per-call CPU.

Every call, including a cache hit, now:

  • concatenates device.toString() + ".";
  • allocates a HashMap, a HashSet, a singleton set and a second TimeSeriesMetadataCacheKey;
  • goes through getTimeseriesMetadata and its timing block, then looks the result up in the map.

On a miss it also copies allSensors into returned.

Could get() keep a fast path: call lruCache.getIfPresent(key), return a copy (plus statistics) on a hit, and delegate to the set-based path only on a miss? Computing devicePrefix only when the BloomFilter is actually consulted would also help the set-based path.

|| resource.getTimeIndexType() == ITimeIndex.FILE_TIME_INDEX_TYPE;

Map<String, TimeseriesMetadata> metadata =
cache.getTimeseriesMetadata(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Dense cold loads are still 15–32% slower than base

For a 700-field device, select-all cold measured 6.6–7.5 ms against base's 5.7 ms, and the first-60% projection 4.0–4.2 ms against 3.5–3.8 ms. Bytes are equal, and this matches the 17–37% in the description.

The extra work appears to be per call, repeated for every device and file:

  • building the pending and result sets for 701 names;
  • the reader turning measurements into a TreeSet and back into a list, i.e. sorting 701 strings;
  • getChildIndexEntries over that list.

The selected set is identical for every device and file of a scan. Sorting it once per scan and passing the sorted list through (which needs the tsfile API to accept a pre-sorted list) should remove most of the repeated cost. A profile would confirm which part dominates.

BloomFilter bloomFilter =
loadBloomFilter(
filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, externalTsFile);
if (deviceAbsent(requested, result, bloomFilter, devicePrefix)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Offset-addressed (external) reads now depend on the file's BloomFilter

The javadoc of deviceMetadataIndexNodeOffset says BloomFilter filtering is skipped for external TsFiles, but this branch still calls loadBloomFilter and deviceAbsent. An external device can therefore be skipped as absent based on that file's BloomFilter.

Master skipped the BloomFilter whenever an offset was given (if (deviceMetadataIndexNodeOffset == null) in both branches of the old get()). Presence is already known in that case, because the offset comes from iterating the file's device index. If an external file's BloomFilter does not contain device.toString() + "." (for example, a file from another writer), read_tsfile would silently drop that device.

Could you skip the BloomFilter when deviceMetadataIndexNodeOffset != null, as before?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread pom.xml
Comment on lines +146 to 149
<tsfile.version>2.5.1-261117-SNAPSHOT</tsfile.version>
<i18n.locale>en</i18n.locale>
<tsfile.locale.opt/>
</properties>
* Load statistics are counted per requested measurement.
*
* @param deviceMetadataIndexNodeOffset non-null for external tsfiles, which address the device
* metadata index by offset; bloom filtering is skipped in that case
Comment on lines 216 to +220
long startTime = System.nanoTime();
long cachePhaseEndTime = 0;
long loadBloomFilterTime = 0;
long cacheHitCount = 0;
long diskLookupCount = 0;
Comment on lines +250 to +251
BloomFilter bloomFilter =
loadBloomFilter(filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, false);
Comment on lines +279 to +281
BloomFilter bloomFilter =
loadBloomFilter(
filePath, tsFileID, queryContext, bloomFilterIoSizeRecorder, externalTsFile);
Comment on lines +318 to +324
recordLoadStatistics(
queryContext,
cacheHitCount,
cacheLookupTime,
diskLookupCount,
diskLookupTime,
loadBloomFilterTime);
Comment on lines +857 to +906
try (TsFileWriter writer = new TsFileWriter(file)) {
writer.registerAlignedTimeseries(
deviceID,
Arrays.asList(
new MeasurementSchema("s1", TSDataType.INT32),
new MeasurementSchema("s2", TSDataType.INT32)));
for (int time = 0; time < 3; time++) {
TSRecord record = new TSRecord(deviceID, time);
record.addPoint("s1", time);
record.addPoint("s2", time);
writer.writeRecord(record);
}
}
cache.clear();
bloomFilterCache.clear();
long[] offset;
try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath())) {
LazyTsFileDeviceIterator iterator = new LazyTsFileDeviceIterator(reader);
assertTrue(iterator.hasNext());
iterator.next();
offset = iterator.getCurrentDeviceMeasurementNodeOffset();
}
// external tsfiles bypass the metadata cache: every requested measurement must still be read
EnvironmentUtils.TEST_QUERY_FI_CONTEXT.setQueryDataSourceType(
QueryDataSourceType.EXTERNAL_TSFILE_SCAN);
try {
Set<String> allSensors = new HashSet<>(Arrays.asList("", "s1", "s2"));
TsFileResource resource = new TsFileResource(file);
resource.setStatusForTest(TsFileResourceStatus.NORMAL);
Map<String, TimeseriesMetadata> result =
cache.getTimeseriesMetadata(
file.getAbsolutePath(),
resource.getTsFileID(),
deviceID,
allSensors,
allSensors,
false,
false,
EnvironmentUtils.TEST_QUERY_FI_CONTEXT,
offset);
assertEquals(allSensors.size(), result.size());
for (String measurement : allSensors) {
assertNotNull(measurement, result.get(measurement));
assertEquals(measurement, result.get(measurement).getMeasurementId());
}
} finally {
EnvironmentUtils.TEST_QUERY_FI_CONTEXT.setQueryDataSourceType(
QueryDataSourceType.SERIES_SCAN);
cache.clear();
bloomFilterCache.clear();
EnvironmentUtils.TEST_QUERY_FI_CONTEXT.setQueryDataSourceType(
QueryDataSourceType.SERIES_SCAN);
cache.clear();
bloomFilterCache.clear();
Comment on lines +247 to +252
while (true) {
Optional<Boolean> hasNextFile = seriesScanUtil.hasNextFile();
if (hasNextFile.isPresent()) {
return hasNextFile.get();
}
}
Comment on lines +348 to +352
if (!allSensors.contains("")) {
// the aligned loader always needs the time column, even if the caller's set omits it
allSensors = new HashSet<>(allSensors);
allSensors.add("");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allSensors will never contain "", so just allSensors.add("") is enough.

queryContext.getQueryStatistics().getLoadBloomFilterActualIOSize()::addAndGet;
boolean cacheHit = true;
boolean externalTsFile = queryContext.isExternalTsFileScan();
String devicePrefix = device.toString() + IoTDBConstant.PATH_SEPARATOR;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lazily init it while you really need this.

cachePhaseEndTime = System.nanoTime();
// serialize reads of the same device+file (same lock key as get) so threads don't each
// walk the device metadata index and race on the shared reader
synchronized (deviceLock(device.toString(), filePath)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create a local field to save device.toString() and reuse it.

* absent, no selected series of the device can be read from this file. For a single requested
* series, the bloom filter can rule that series out directly.
*/
private static boolean deviceAbsent(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to use bloomfilter to judge whether each timesereis exists and directly update result if bloomfilter say one sensor doesn't exists.

* Reads the pending measurements from the leaf groups that hold them. When the cache is enabled
* the group siblings are returned as well, so a later single-column get() finds them in the LRU.
*/
private void readAndMerge(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add java doc to explain what's the meaning of each parameter, especially for the three set

if (requested.contains(measurement)) {
result.put(measurement, null);
}
cacheNonExistenceIfNeeded(tsFileID, device, measurement, cacheEnabled, ignoreNotExists);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

count = 0 doesn't mean this sensor doesn't exist

if (cacheEnabled) {
lruCache.put(new TimeSeriesMetadataCacheKey(tsFileID, device, measurement), metadata);
}
if (requested.contains(measurement)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to do this contains again here, all loaded entries should only contains what you really need.

}
if (timeseriesMetadata == null || timeseriesMetadata == NULL_EXISTS_CACHE_PLACE_HOLDER) {
}
fillMissingWithNull(requested, result);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove sensor from pending set in tsfile function call, after which, the remaining is what really not existing sensors.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And for these really not existing sensors, besides putting them in result map, you should also put them back in cache

Comment on lines +427 to +432
private static void fillMissingWithNull(
Set<String> requested, Map<String, TimeseriesMetadata> result) {
for (String measurement : requested) {
result.putIfAbsent(measurement, null);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

costly

Comment on lines +574 to +576
// a key is built for every lookup and every write: compute the hash once instead of
// allocating the Objects.hash argument array on each call
this.hash = Objects.hash(tsFileID, device, measurement);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually you don't need to do so, jvm will store it for each object

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants