You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(python): union table-model columns across TsFiles in TsFileDataFrame - #1120
A physical TsFile registers only the columns it actually wrote: IoTDB compaction outputs record the fields present in each compaction, and independently written files may come from different schema snapshots. As a result one table can legitimately span files whose per-file column sets differ. TsFileDataFrame rejected such load sets outright with Incompatible schema for table ....
This change makes the table-model dataset treat the file set as a union: the same table name across files is merged into one logical view instead of rejected.
Merge semantics
The dataset does not compare authored FIELD declarations. Reads go through each file's own catalog; the global view only needs the table name, the positional tag values that identify a device, the field names it can expose, and where every (device, field) shard lives.
FIELD columns are the union of the readable (numeric) fields in first-seen (file load) order. A device/field absent from a file simply contributes no shard.
TAG columns are positional; a device is keyed by its ordered tag values, never by column names. Files may only append trailing tags — a longer tag list adds the new tag at the end, and devices from files without it read it as null, in either load order. Files must agree on the tag columns they share: a different name at the same position means the same table name points at a different tag layout (IoTDB never reorders tags), and merging by position would conflate unrelated devices, so it is rejected. Tag types are not compared.
No FIELD comparison or validation is performed: field declared types and tag/field categories are not compared.
The merged entry carries no field_types/schema_columns, so persistent-index construction (dataset/index.py) derives column types from the physical data through its existing fallback, and its cross-shard "incompatible physical type" check remains the only type guard (index mode, as before this PR).
One further rejection: a name that is a TAG in one file and a readable (numeric) FIELD in another — a namespace conflict inside one table, since the persistent index keys columns by (table, name). Non-numeric fields are never exposed or indexed, so they do not take part in that check.
The tree-model merge path is untouched, and mixed tree/table load sets are still rejected.
Dataset index
With the union merge, a numeric field that a file declares but no device ever writes enters the merged field_columns without a physical type. The index column fallback packed a -1 placeholder into COLUMN_SCHEMA (unsigned short) and the build failed with struct.error. The fallback now emits columns only for fields that actually have a physical type; such a field has no series and is unreadable through the dataset, so leaving it out matches the non-indexed view. This path was previously unreachable (tree fields always carry data; table entries used the declared schema_columns branch).
The index cache identity includes the caller's load order, so each order builds its own index and the merged field order follows first-seen order rather than cache history. Device identity in the index is the canonical tag-value path (names are never part of it), and series bind by (device, column name). Indexed, non-indexed and index-reuse views were compared on a mixed load set (disjoint fields, appended tags, non-numeric column) and on a longer-tag-first / shorter-tag-second order; both produced identical tags, field sets, timestamps and values.
Tests
python/tests/test_tsfile_dataset.py, parametrized over use_index=False/True:
extra field across files → union columns, per-device values verified
disjoint field sets across three files
overlapping subsets for different devices
trailing tag appended in either load order ([region] then [region, site], and the reverse) with null tag values in list_timeseries_metadata
a three-file tag-append chain, where the second merge runs against an entry that already unions two files
a differing shared tag prefix (same table name) is rejected in either load order
non-numeric declared columns with different names union without appearing in the numeric surface
a merged table's persisted index columns are exactly its tags plus readable fields (no declaration in memory)
same field name with different declared types merges without an index; the index path still rejects the physical-type mismatch ("incompatible physical type")
a name that is TAG in one file and FIELD in another is rejected
a declared numeric field that is never written stays unreadable and no longer breaks index construction
Verification
Full Maven build from a clean tree, the command CI's unit-test-python job runs (plus lzma2 as the repo requires):
Python: 637 passed, 16 skipped (pre-existing macOS/Linux-only and sanitizer-only skips)
spotless:check for tsfile-cpp and tsfile-python, Apache RAT license check, wheel build: all passed
BUILD SUCCESS
The first two attempts failed on dependency downloads (utfcpp, googletest) over a slow network; reruns used the repository's local dependency cache and the SHA-256-verified googletest archive. No source or build configuration was changed to work around it.
Remaining limitations
Duplicate timestamps for one logical series across files are still rejected.
Declared types are ignored for multi-file table datasets; only the index path's physical-type check can reject a conflict, so indexed and non-indexed loads can differ for a field whose shards store different physical types.
Non-numeric declared columns no longer appear in the persistent index for multi-file table datasets (they were never readable through the dataset).
Devices written under a shorter tag schema stay distinct from devices with real values for the added tag (the former reads with a null tag); this matches IoTDB.
A tag renamed across snapshots is rejected (it is indistinguishable from a recreated table with a different tag layout), rather than reconciled.
Docs
skills/tsfile/references/dataframe-arrow.md documents the union semantics in the lazy multi-file dataset section.
…rame
A physical TsFile registers only the columns it actually wrote, so one
table can span files whose per-file column sets differ (compaction
outputs, files written at different schema snapshots). TsFileDataFrame
rejected such load sets outright.
Merge table entries instead: FIELD columns are unioned in first-seen
order and a device/field missing from a file reads back as NaN; TAG
columns are prefix-compatible (extra trailing tags are appended and
shorter devices read with a null tag), and only a differing shared tag
prefix, a tag-vs-field category change, or a same-named FIELD column
with a different data type is rejected, with an error naming the table,
file, column and types.
The merged entry rebuilds schema_columns so the persistent dataset index
keeps defining columns and validating physical types. Shard registration
already keys logical series by the global field index resolved by name,
so table-model reader-local field indices no longer need to coincide
with the merged global ones.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The index cache ignores source-path order, so reversed file loads can reuse an incorrect column order and make positional selection depend on cache history.
This PR lets Python’s TsFileDataFrame combine table-model TsFiles with different column sets into one logical table, supporting files written from evolving schemas or separate compactions.
Changes:
Unions FIELD columns and prefix-compatible TAG schemas while rejecting incompatible types or categories.
Adds indexed and non-indexed tests for unions, missing values, tag padding, and schema conflicts.
Documents merge semantics and remaining limitations.
File
Description
skills/tsfile/references/dataframe-arrow.md
Documents multi-file schema unions and compatibility rules.
python/tsfile/dataset/dataframe.py
Implements table-schema merging and corrects the shard-index comment.
python/tests/test_tsfile_dataset.py
Adds union coverage and updates schema-conflict assertions.
Device identity in the dataset is the ordered tuple of tag values, not the
tag column names, and IoTDB keeps tag ordinals across RENAME COLUMN, so a
tag renamed at the same position is the same tag: keep the first-seen name
for display and let both files' devices line up position by position.
Only a data-type change at a shared tag position is still rejected, since
it would make the shared device ids uninterpretable.
The dataset only consumes the table name, the positional tag values that
identify a device, the readable field names and where each shard lives;
reads go through every file's own catalog and never use an authored
declaration. Drop the remaining schema comparisons: tag types, field
declared types and tag/field categories are no longer validated, and a
tag renamed at the same position simply keeps the first-seen name.
The merged entry carries no field_types or schema_columns, so the
persistent index derives column types from the physical data (its
cross-shard physical-type check is unchanged). One rejection remains: a
name that is a TAG in one file and a FIELD in another, because the index
keys columns by (table, name) and a duplicate makes lookups ambiguous.
With the pure-union table merge, a numeric field that one file declares
but no device ever writes enters the merged field_columns without any
physical type. The index column fallback packed a -1 placeholder into
COLUMN_SCHEMA (unsigned short) and the build failed with struct.error.
Emit columns only for fields that actually have a physical type. Such a
field has no series and is unreadable through the dataset anyway, so
leaving it out matches the non-indexed view. This path was previously
unreachable: tree fields always carry data, and table entries took the
declared schema_columns branch.
Positional device identity only works while files agree on the tag
columns they share. IoTDB never reorders tags, so a same table name with
a different tag name at the same position means the table was recreated
with a different tag layout; merging by position would conflate
unrelated devices. Require the shared tag prefix to match by name and
keep rejecting a name that is a TAG in one file and a FIELD in another.
Appending trailing tags in either load order still unions correctly
(devices from files without the new tag read it as null), and FIELD
declarations remain unvalidated.
Address the review follow-ups on the table-model union:
- exercise a three-file tag-append chain, where the second merge runs
against an entry that already unions two files;
- assert the persisted index defines a merged table as tags plus readable
fields only (and that the in-memory merged entry carries no
declaration);
- share the first-seen field union between the table and tree merges;
- correct a test comment that still described tag column names as part of
device identity.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
TAG/FIELD conflicts involving nonnumeric fields are not rejected, and order-insensitive cache reuse violates first-seen field ordering, making positional selection depend on cache history.
After building an index for [A, B] with disjoint fields temperature and humidity, reopening [B, A] reuses the original field order because index_path_for() and index_matches_paths() ignore input order. This violates the new first-seen ordering and makes positional selection depend on cache history. Make cache reuse order-aware, including invalidation of existing order-insensitive caches, and add a reversed-order reuse test.
The persistent index identity hashed the sorted source paths, so loading
the same files in reverse order reused an index whose merged field order
came from the other order; positional selection then depended on cache
history instead of the documented first-seen order. Hash the caller's
order instead. index_matches_paths keeps validating the canonical sorted
file list, and sorted loads keep their existing cache identity.
Also scope the TAG/FIELD namespace guard to readable fields explicitly:
non-numeric fields are never exposed or indexed, so they cannot collide
in the index, and the docs no longer promise they are rejected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A physical TsFile registers only the columns it actually wrote: IoTDB compaction outputs record the fields present in each compaction, and independently written files may come from different schema snapshots. As a result one table can legitimately span files whose per-file column sets differ.
TsFileDataFramerejected such load sets outright withIncompatible schema for table ....This change makes the table-model dataset treat the file set as a union: the same table name across files is merged into one logical view instead of rejected.
Merge semantics
The dataset does not compare authored FIELD declarations. Reads go through each file's own catalog; the global view only needs the table name, the positional tag values that identify a device, the field names it can expose, and where every (device, field) shard lives.
field_types/schema_columns, so persistent-index construction (dataset/index.py) derives column types from the physical data through its existing fallback, and its cross-shard "incompatible physical type" check remains the only type guard (index mode, as before this PR).Dataset index
With the union merge, a numeric field that a file declares but no device ever writes enters the merged
field_columnswithout a physical type. The index column fallback packed a-1placeholder intoCOLUMN_SCHEMA(unsigned short) and the build failed withstruct.error. The fallback now emits columns only for fields that actually have a physical type; such a field has no series and is unreadable through the dataset, so leaving it out matches the non-indexed view. This path was previously unreachable (tree fields always carry data; table entries used the declaredschema_columnsbranch).The index cache identity includes the caller's load order, so each order builds its own index and the merged field order follows first-seen order rather than cache history. Device identity in the index is the canonical tag-value path (names are never part of it), and series bind by (device, column name). Indexed, non-indexed and index-reuse views were compared on a mixed load set (disjoint fields, appended tags, non-numeric column) and on a longer-tag-first / shorter-tag-second order; both produced identical tags, field sets, timestamps and values.
Tests
python/tests/test_tsfile_dataset.py, parametrized overuse_index=False/True:[region]then[region, site], and the reverse) with null tag values inlist_timeseries_metadataVerification
Full Maven build from a clean tree, the command CI's
unit-test-pythonjob runs (pluslzma2as the repo requires):spotless:checkfortsfile-cppandtsfile-python, Apache RAT license check, wheel build: all passedThe first two attempts failed on dependency downloads (utfcpp, googletest) over a slow network; reruns used the repository's local dependency cache and the SHA-256-verified googletest archive. No source or build configuration was changed to work around it.
Remaining limitations
Docs
skills/tsfile/references/dataframe-arrow.mddocuments the union semantics in the lazy multi-file dataset section.