Skip to content

Add a selective metadata reader for a set of measurements - #1123

Open
ColinLeeo wants to merge 2 commits into
apache:developfrom
ColinLeeo:colin/measurement-selective-metadata
Open

ColinLeeo wants to merge 2 commits into
apache:developfrom
ColinLeeo:colin/measurement-selective-metadata

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Description

Reading the TimeseriesMetadata of N measurements of one device currently walks the device metadata index and deserializes the containing leaf group once per measurement. A wide aligned projection therefore pays one index walk and one leaf deserialization per column, and an absent measurement that a bloom filter cannot rule out re-reads the same leaf group.

This PR adds a set-based metadata reader to TsFileSequenceReader that resolves a whole measurement set against the leaf groups that hold it.

What changed

  • MetadataIndexNode.getChildIndexEntries(keys, exactSearch): one-pass floor search of a sorted key set over a node's children. It picks a monotonic binary search or a merge scan by cost and returns one shared Pair per child, so callers can group consecutive keys by identity.
  • TsFileSequenceReader.readTimeseriesMetadata(device, offset, measurements, returnedMeasurements, ignoreNotExistDevice, ioSizeRecorder): locates the device's measurement metadata index node (or reads the supplied offset for external files), groups the measurements by child with the batch search, reads each visited internal node and each visited leaf group once, and returns the metadata of returnedMeasurements found in those groups. Measurements absent from the file are absent from the returned map.
    • returnedMeasurements may be a superset of measurements, which lets a single-measurement call warm its group siblings.
  • Convenience overloads without the offset and without returnedMeasurements.

Example

A file whose device has 700 series spread over 3 measurement leaf groups; read all 700 measurements, 680 of which are absent from the file and cannot be ruled out by a bloom filter:

  • Before: one readTimeseriesMetadata call per measurement; each absent name re-reads and re-deserializes its floor leaf.
  • After: one call; the 3 leaf groups are read once each and the walk resolves the 680 absent names.

In the IoTDB query benchmark that motivated this change (700-column device, 20 columns per file, 12 files, cold metadata cache), a sparse 20-of-700 projection dropped from 8.0 ms to 2.0–2.6 ms per query.

Tests

  • MetadataIndexNodeTest.testGetChildIndexEntries: both search strategies, a key before the first child, the last child's end offset, exact search, and an empty node.
  • TimeSeriesMetadataReadTest.testReadTimeseriesMetadataInBatch: batch results match per-measurement reads (statistics and chunk metadata), absent measurement and absent device, the offset-addressed variant, and sibling warming through returnedMeasurements.

Compatibility

All additions are new methods; existing overloads are unchanged. The batch index search follows the approach of #550.

🤖 Generated with Claude Code

MetadataIndexNode.getChildIndexEntries() searches the floor child entry
of a sorted key set in one pass, choosing between a monotonic binary
search and a merge scan and reusing one pair instance per child. The new
TsFileSequenceReader.readTimeseriesMetadata(device, offset,
measurements, returnedMeasurements, ...) uses it to visit only the leaf
groups that hold a searched measurement, reads each visited group once,
and returns the metadata of the returned measurements.

A caller can resolve a whole projection without one index walk and one
leaf deserialization per column, and a single-measurement call can warm
its group siblings through returnedMeasurements.

Follows the batch index-search approach of apache#550.
@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 17, 2026 08:52
ColinLeeo added a commit to ColinLeeo/iotdb that referenced this pull request Nov 17, 2026
The set-based readTimeseriesMetadata overload is not in the dated tsfile
snapshot the build used before. Point the dependency at the 2.5.1
snapshot, which will contain it once apache/tsfile#1123 is merged and
published.

The IoTDB change that uses the new API must not be pushed until that
snapshot exists in the configured repositories.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Batch lookups mishandle duplicate keys and mixed device ID types, while filtering decodes unnecessary chunk metadata and tests lack valid ordering and internal-node coverage.

5 open issues
What changed in this PR

Adds batch metadata lookup to the Java reader, reducing repeated index traversal and leaf-group reads when selecting multiple measurements from one device.

Changes:

  • Adds batch child-index searches using binary or merge strategies.
  • Adds device- and offset-based reader overloads, including optional sibling metadata retrieval.
  • Adds tests for batch results, missing measurements and devices, offsets, and sibling retrieval.
File Description
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​read/​TimeSeriesMetadataReadTest.java Tests batch reader results and API variants.
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​file/​metadata/​MetadataIndexNodeTest.java Tests batch child lookup and boundary cases.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​read/​TsFileSequenceReader.java Adds grouped metadata traversal and reader overloads.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​file/​metadata/​MetadataIndexNode.java Adds batch searches with shared child-entry results.

Review effort: Balanced

Compare the index child to the key in the merge search, like the binary
searches do, so a plain and a string-array device id can be compared
without throwing. Keep the matched child eligible for duplicate keys in
both batch strategies.

Decode the chunk metadata of a leaf group only for the measurements the
call returns, using the existing filter-aware TimeseriesMetadata
deserializer (and a stream counterpart for groups larger than one
buffer), so a sparse request does not build chunk metadata for every
series it walks past.

Cover the internal measurement node with a 20-measurement fixture, assert
that absent names add no reads, and make the index-node tests use
lexicographically ordered children plus duplicate and mixed-device-id
cases.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving the selective reader into TsFile. I checked out cb46a2d and ran MetadataIndexNodeTest and TimeSeriesMetadataReadTest locally (6 cases pass). Since this API exists for performance, I also wrote throwaway differential tests and a reader-level benchmark (not pushed).

Correctness: no wrong result found.

  • Batch index search: 600,000 random node/key sets (duplicate keys, empty nodes, exact and floor mode), both strategies against a brute-force floor. No mismatch.
  • Batch read against the single-measurement reader, name by name (id, type, statistics, and the offset and statistics of every chunk): 467,805 names on tree-model files (aligned and non-aligned devices, index degree 2/3/4/7/256, multi-level measurement index, absent names and devices, the offset-addressed variant) and 153,615 names on table-model files (null tags, all-null columns). No mismatch.

Performance: one call per (device, file), TsFile API only. Real files written with TsFileWriter: one aligned device with 700 INT64 columns (three measurement leaf groups), 3 chunks per series, and a sparse file in which the same device has 20 of the 700 columns. The harness counts positional reads and bytes at the TsFileInput and reports the median of 5 timed rounds after warm-up (JDK 21, files in the page cache; rounds are mostly within ±3% of the median).

Scenario master flow this PR prototype without the sort
Dense device, select all 700 columns 100.6 µs, 6 reads / 179 KB 152.6 µs, 4 reads / 179 KB 93.0 µs
Dense device, first 60% of the columns 70.0 µs 91.2 µs 60.7 µs
Sparse file (20 of 700 columns), select all 700 56.1 µs, 2 reads / 5 KB 55.0 µs, 2 reads / 5 KB 3.5 µs
Same, in a file with 1,000 other devices (19 BloomFilter false positives) 533 µs, 60 reads / 225 KB 81.7 µs, 3 reads / 11 KB 23.1 µs
Dense device, 5 columns 29.1 µs 19.2 µs 16.3 µs
2 present columns + 18 absent names that sort into the other two leaf groups 29.9 µs, 2 reads / 65 KB 60.7 µs, 4 reads / 179 KB 17.3 µs, 2 reads / 65 KB [a]
Warm query: 100 never-written columns still unresolved 8.4 µs, no read 65.4 µs, 4 reads / 179 KB 9.5 µs, no leaf-group read [a]
20-column device, all present 3.7 µs 4.5 µs 3.4 µs
Offset-addressed read (external TsFile), select all 700 columns 18,643 µs, 1,402 reads / 42 MB 149.6 µs, 4 reads / 179 KB n/a
  • "master flow" is what IoTDB master's TimeSeriesMetadataCache.get() does per column on the default path: cache probe, per-measurement BloomFilter check, then readTimeseriesMetadata(device, measurement, allSensors, ...). It already reads each leaf group once.
  • The last row is master's externalTsFile / cache-disabled branch, which calls the single-measurement reader once per column. This is the "Before" of the description.
  • [a] with the existence predicate proposed in the inline comment on the Javadoc.

What this shows:

  • The reader is a large win where master really reads per column: 80–160x for offset-addressed reads (1, 3 and 10 chunks per series), and 6.5x with 20x fewer bytes when the BloomFilter has false positives. Narrow projections also gain, because chunk metadata is decoded only for the returned series.
  • On the default path it is not faster than what master does today. The sparse case that motivated it is level (55.0 vs 56.1 µs, same bytes), so the gain measured in apache/iotdb#19198 comes from the caller making one call per file instead of a cache/lock/BloomFilter cycle per column. Dense requests are slower by a constant of about 51 µs per call: 117.3 vs 65.7 µs with 1 chunk per series, 152.6 vs 100.6 µs with 3, 243 vs 192 µs with 10.

Requesting changes for the two P1s. Each point has an inline comment:

  1. [P1] Sorting the request accounts for 85–90% of that constant. Bucketing names by child removes it and makes the reader faster than the master flow in every dense case I measured.
  2. [P1] An absent name makes the reader load the leaf group it sorts into, and the API gives callers no cheap way to prevent that. The Javadoc and the test claim more than the reader guarantees.
  3. [P2] The new overloads make a literal null argument ambiguous.
  4. [P2] Test gaps: weak read-count assertions, no aligned or table-model device.
  5. [P2] TsFile's own query path does not use the new reader, although it has the per-absent-name re-read.
  6. [P3] Semantics and documentation: floor semantics differ from getChildIndexEntry, an empty request skips the device check, @return wording.
  7. [P3] Small code points: double contains, duplicated deserializer, an unreachable branch.

Not covered: the branch for leaf groups larger than 2 GB was only read, not run. The numbers above isolate the reader; end-to-end numbers are in apache/iotdb#19198. I can share the harness and the prototype if that helps.

throw new IOException(
Messages.format("error.read.device_not_in_metadata_file", device, file));
}
List<String> sortedMeasurements = new ArrayList<>(new TreeSet<>(measurements));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Sorting the request dominates the per-call cost; bucket the names by child instead

new ArrayList<>(new TreeSet<>(measurements)) costs 43–51 µs for 701 names when measured alone. That is 85–90% of the ~51 µs by which a dense select-all call is slower than the master flow (152.6 vs 100.6 µs with 3 chunks per series, 117.3 vs 65.7 µs with 1, 243 vs 192 µs with 10; bytes are equal). In the sparse case this PR targets (20 of 700 columns written), about 46 of the 55 µs per call go into sorting names that are not in the file. With 12 files per query that would be roughly 0.6 ms of the 0.9–1.8 ms that a dense cold query loses in apache/iotdb#19198, and it is the sort I suspected there.

The traversal does not need a globally sorted list. An index node has at most max_degree_of_index_node children, so each name can be routed with a floor binary search over the children (about 8 comparisons at degree 256) and collected per child. On a LEAF_MEASUREMENT node only the set of selected groups matters, so the loop can stop once every group is selected, which happens after a few names for a dense request:

boolean[] selected = new boolean[children.size()];
int remaining = children.size();
for (String name : names) {
  int index = floor(children, name); // last child whose key <= name, or -1
  if (index >= 0 && !selected[index]) {
    selected[index] = true;
    if (--remaining == 0) {
      break;
    }
  }
}

A prototype of this returned exactly the same metadata as this PR's reader on 50,400 randomized requests (tree and table model, index degree 2 to 256), with the same reads and bytes. It takes 93.0 µs for the dense select-all (7.6% faster than the master flow instead of 52% slower), 60.7 µs for the 60% projection (70.0 µs for the master flow) and 3.5 µs for the sparse case (55.0 µs here).

With that, MetadataIndexNode.getChildIndexEntries, the two search strategies and the shared-Pair identity contract are no longer needed by this PR. If you prefer to keep the sorted traversal, skipping the sort for an input that is already sorted would help callers that can sort once per scan, but the bucket approach needs no cooperation from the caller.

Comment on lines +1127 to +1129
* Reads the {@link TimeseriesMetadata} of the measurements that share a leaf group with one of
* {@code measurements}. Leaf groups that no searched measurement maps into are not read, and each
* visited leaf group is read once.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] An absent name makes the reader load the leaf group it sorts into, and callers have no cheap way to avoid that

A name that is not in the file is still routed to its floor child, and that leaf group is read and deserialized. So the sentence here holds literally, but a group that contains none of the requested series is read whenever an absent name sorts into it.

Measured on the 700-column device (three leaf groups):

  • 2 present columns plus 18 absent names that sort into the other two groups: 4 reads / 179 KB / 60.7 µs, against 2 reads / 65 KB / 29.9 µs for the master flow, whose per-measurement BloomFilter check rejects the absent names before any read.
  • 100 never-written columns requested again by a warm query: 4 reads / 179 KB on every call (490 KB with 10 chunks per series). The master flow does no read, apart from BloomFilter false positives.

This is the reader-side cause of the warm re-reads in apache/iotdb#19198. Today a caller can only avoid it by probing the BloomFilter for every name before the call, and that is expensive for a dense request: 322 µs instead of 152.6 µs per call for 701 present names, about 0.24 µs per name that passes, because every hash function has to be evaluated.

Suggestion: let the reader take an optional Predicate<String> mayExist (the caller wraps the BloomFilter) and test a name only while the group it sorts into has not been selected yet. A dense request then needs a few probes in total, and a group that only absent names point to is not read. In the prototype this gives 92.5 µs for the dense select-all, 2 reads / 65 KB / 17.3 µs for the first case above, and no leaf-group read for the second. Every requested name that is missing from the result is then known to be absent: it was rejected by the filter, or its group was read and did not contain it. Offset-addressed reads of external files should pass no predicate, since their BloomFilter may not cover the series.

Whatever you decide, please reword this Javadoc so that it says what happens for absent names.

* @return the metadata of the measurements found in this file; absent measurements are missing
* from the map
*/
public Map<String, TimeseriesMetadata> readTimeseriesMetadata(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] The new overloads make a literal null argument ambiguous

reader.readTimeseriesMetadata(device, null, someSet, false, null) does not compile:

error: reference to readTimeseriesMetadata is ambiguous
  both method readTimeseriesMetadata(IDeviceID,String,Set<String>,boolean,LongConsumer)
  and method readTimeseriesMetadata(IDeviceID,long[],Collection<String>,boolean,LongConsumer) match

It resolves only when the third argument is not a Set, or when the null is cast to long[]. The three-argument overload has the same conflict with (IDeviceID, String, boolean): reader.readTimeseriesMetadata(device, null, false) compiled before this PR and is ambiguous after it.

readTimeseriesMetadata now has 8 overloads with three return types (TimeseriesMetadata, List, Map). A distinct name for the set-based reader would avoid both problems and make call sites easier to read.

Comment on lines +1135 to +1136
* @return the metadata of the returned measurements found in this file; absent measurements are
* missing from the map

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] @return is inaccurate when returnedMeasurements is a superset

A name in returnedMeasurements that exists in the file but lives in a leaf group no searched measurement sorts into is missing from the map as well. For those names a missing key does not mean "absent from the file". Suggest: "the metadata of the returned measurements found in the visited leaf groups".

boolean ignoreNotExistDevice,
LongConsumer ioSizeRecorder)
throws IOException {
if (measurements.isEmpty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] An empty request skips the device check

With an empty measurements the method returns before the device is looked up, so an absent device with ignoreNotExistDevice = false yields an empty map instead of the IOException that a non-empty request and the single-measurement reader throw. This is harmless for the current caller, but it should be documented, or the check moved below the device lookup.

Comment on lines +1251 to +1252
TimeseriesMetadata metadata = deserializeTimeseriesMetadata(buffer, requested);
if (requested.contains(metadata.getMeasurementId())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Each series is tested against requested twice, through a method reference created per series

deserializeTimeseriesMetadata builds requested::contains for every series (L1277, L1287), and this loop then calls requested.contains again. For a narrow projection this can cost more than the chunk-metadata skipping saves: selecting 5 of 700 columns with one chunk per series takes 18.3 µs here, against 15.4 µs for the master flow and 13.8 µs for the prototype, which creates the predicate once per group and reuses its result. I did not measure this part separately. With 3 chunks per series this PR is ahead (19.2 vs 29.1 µs).

Two related small costs in the entry method: the result HashMap is not pre-sized (L1161), and new HashSet<>(sortedMeasurements) copies the request even when measurements is already a Set (L1160).

* @param keys keys sorted in ascending order
* @param exactSearch whether to return null instead of the floor child when the key is missing
*/
public List<Pair<IMetadataIndexEntry, Long>> getChildIndexEntries(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] The batch search and getChildIndexEntry disagree for a key before the first child

In floor mode getChildIndexEntry(key, false) returns child 0 for a key that sorts before the first child (low == 0 ? low : low - 1), while this method returns null for it. On an empty node the single-key method throws IndexOutOfBoundsException and this one returns null.

Returning null is the better answer, because it saves reading a group that cannot contain the key, and the test asserts it. But two adjacent methods with different floor semantics will surprise the next caller, so could the Javadoc state the difference? testGetChildIndexEntriesComparesChildrenWithMixedDeviceIds compares the two methods only for keys at or after the first child.

* Deserializes one TimeseriesMetadata from a file input and decodes its chunk metadata only when
* {@code decodeChunkMetadata} accepts the measurement.
*/
public static TimeseriesMetadata deserializeFrom(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Duplicated deserializer

This is a copy of deserializeFrom(TsFileInput, boolean, boolean) (L166) with the predicate added. The ByteBuffer variant was refactored so that the old overload delegates to the new one. Could the TsFileInput one do the same, with id -> true? Otherwise the two bodies will drift.

Assert.assertEquals(measurementCount, all.size());
// the batch read resolves several measurements per node/group read
Assert.assertFalse(allReads.isEmpty());
Assert.assertTrue(allReads.size() < measurementCount);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] The read-count assertions are too weak, and nothing covers aligned or table-model devices

allReads.size() < measurementCount (and the same check at L262) still passes if every node and leaf group is read twice. The tree is deterministic here: 20 series at degree 3 give exactly 11 reads (the root, 3 leaf index nodes and 7 leaf groups). Asserting that number, or comparing the recorded sizes with the expected ranges, would actually pin down "each visited node and group is read once".

The batch path is exercised only on non-aligned tree-model devices, while its only caller is IoTDB's aligned loader, which also passes the time column "". My differential runs on aligned and table-model files found no problem, so this is a coverage gap and not a bug. An aligned fixture with the time column in the request and a table-model fixture would be cheap regression guards.

Comment on lines +239 to +248
// absent names must not add reads
Set<String> withAbsent = new HashSet<>(present);
withAbsent.add("a");
withAbsent.add("m99");
withAbsent.add("zzz");
List<Long> readsWithAbsent = new ArrayList<>();
Map<String, TimeseriesMetadata> withAbsentResult =
reader.readTimeseriesMetadata(device, offset, withAbsent, false, readsWithAbsent::add);
Assert.assertEquals(measurementCount, withAbsentResult.size());
Assert.assertEquals(allReads, readsWithAbsent);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] This assertion holds only for the names chosen

a sorts before the device's first series, so it has no floor child. m99 and zzz sort after the last series, so they fall into the last leaf group, which the present names already select. An absent name that sorts into a group no present name selects does add reads: on this fixture m00 alone takes 3 reads, and m00 plus the absent m10x takes 5 (one more index node and one more leaf group).

Could this request a subset of the present names plus an absent name that falls into another group, and assert the contract the reader really has, whichever it ends up being after the comment on the reader's Javadoc?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants