Repository navigation
feat(core): support TIME in primary-key tables, ORC and Avro - #547
Merged
Merged
Conversation
SteNicholas
force-pushed
the
PAIMON-540
branch
3 times, most recently
from
November 11, 2026 05:04
384be3c to
1da6a8f
Compare
lucasfang
reviewed
Nov 11, 2026
Support TIME columns in primary-key tables, as partition and bucket keys, in predicates, and in ORC, Avro and Mosaic files, following Paimon Java. TIME stays an Arrow time32[ms] holding the milliseconds of the day, with its declared precision in the paimon.time.precision field metadata. - Add FieldType::TIME and Literal(FieldType::TIME, millis). - Read and write TIME as int32 in BinaryRow, serializers, comparators, the sort buffer, columnar rows and row-to-Arrow conversion. - Validate and evaluate TIME predicates, and push them down to Parquet and ORC. Parse the TIME literals of query auth row filters like Java LocalTime.parse. - Allow TIME partition keys. Format their values like Java TimeToStringCastRule, or as milliseconds with partition.legacy-name, and parse them like BinaryStringUtils.toTime. Partition value errors now name the partition key. - Hash TIME bucket keys as int32 in the default and Hive bucket functions and in bucket pruning. - Map TIME to an ORC int and to an Avro int with the time-millis logical type, rejecting Avro precisions above 3, and report TIME file statistics as int statistics. - Reject time32 units other than milliseconds in predicates, partition and bucket keys, variant_get targets, and ORC and Avro reads and writes. - Cast between TIME and STRING, TIMESTAMP or TIMESTAMP_LTZ in schema evolution, for both data and statistics, like the Java TIME cast rules. A TIME field is cast to STRING with the precision it declares, which also formats TIME values in system tables, and a TIMESTAMP_LTZ field is cast to TIME in the local time zone. variant_get casts strings and timestamps to TIME. - Support TIME in min/max aggregation, the realtime store and the hash index. - Document the TIME support of each file format.
SteNicholas
force-pushed
the
PAIMON-540
branch
from
November 11, 2026 08:48
1da6a8f to
6ada347
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #540
Complete the TIME type support in Paimon C++, following Paimon Java. TIME stays an Arrow
time32[ms]holding the milliseconds of the day, with its declared precision (TIME(0)toTIME(9)) in thepaimon.time.precisionfield metadata.FieldType::TIMEandLiteral(FieldType::TIME, millis).BinaryRow, serializers, comparators, the sort buffer, columnar rows and row-to-Arrow conversion, so primary-key tables support TIME as a primary key and as a value column.IN/NOT IN, null checks), and push them down to Parquet and ORC. Query auth row filters of REST catalog tables parse TIME literals like JavaLocalTime.parse.TimeToStringCastRule, or as milliseconds withpartition.legacy-name, and parsed likeBinaryStringUtils.toTime. Partition value errors now name the partition key.intand to an Avrointwith thetime-millislogical type, rejecting Avro precisions above 3 like Java, and report TIME file statistics as int statistics (Parquet, ORC, Avro, Mosaic).time32units other than milliseconds in predicates, partition and bucket keys,variant_gettargets, and ORC and Avro reads and writes.variant_getcasts strings and timestamps to TIME.Not covered yet, as documented: TIME column default values, file or global indexes on TIME columns, and verifying that Paimon Java reads tables written by Paimon C++ and that Paimon C++ reads Java-written primary-key tables with TIME keys.
Tests
00:00:00.123,23:59:59.999, offsets, out-of-range and malformed input), literals, literal conversion, predicate validation and evaluation, row writers, serializers, comparators, columnar rows, row-to-Arrow conversion, the sort buffer, aggregation and the row equalizer.CastedRow, field mapping, statistics evolution andvariant_get, including non-millisecond targets.$filessystem table; and reading Java-written TIME tables in Parquet, ORC and Avro.ReadResultCollector::SortArraynow concatenates the chunks before sorting, which avoids an Arrow multi-chunk sort bug on TIME columns (Segmentation fault upon multi-key sorting of concatenated tables arrow#47252).API and Format
FieldType::TIMEininclude/paimon/defs.h, andLiteral(FieldType, int32_t)now also acceptsFieldType::TIME.intand to Avro asintwith thetime-millislogical type, both matching Paimon Java. TIME partition values and bucket ids follow Paimon Java.Documentation
Yes.
docs/source/user_guide/data_types.rstdocuments the TIME support of primary-key and append-only tables, partition and bucket keys, predicates and query auth row filters, each file format and its statistics, string parsing, schema evolution casts, and what is not supported yet.Generative AI tooling
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with Claude Code