Skip to content

feat(core): support TIME in primary-key tables, ORC and Avro - #547

Merged
lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-540
Nov 12, 2026
Merged

lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-540

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #540

Complete the TIME type support in Paimon C++, following Paimon Java. TIME stays an Arrow time32[ms] holding the milliseconds of the day, with its declared precision (TIME(0) to TIME(9)) in the paimon.time.precision field metadata.

  • Add FieldType::TIME and Literal(FieldType::TIME, millis).
  • Read and write TIME as int32 in BinaryRow, serializers, comparators, the sort buffer, columnar rows and row-to-Arrow conversion, so primary-key tables support TIME as a primary key and as a value column.
  • Validate and evaluate TIME predicates (comparisons, IN/NOT IN, null checks), and push them down to Parquet and ORC. Query auth row filters of REST catalog tables parse TIME literals like Java LocalTime.parse.
  • Allow TIME partition keys. Partition values are formatted like Java TimeToStringCastRule, or as milliseconds with partition.legacy-name, and parsed like BinaryStringUtils.toTime. Partition value errors now name the partition key.
  • Hash TIME bucket keys as int32 in the default and Hive bucket functions and in bucket pruning.
  • Map TIME to an ORC int and to an Avro int with the time-millis logical type, rejecting Avro precisions above 3 like Java, and report TIME file statistics as int statistics (Parquet, ORC, Avro, Mosaic).
  • Reject time32 units other than milliseconds in predicates, partition and bucket keys, variant_get targets, and ORC and Avro reads and writes.
  • Cast between TIME and STRING, TIMESTAMP or TIMESTAMP_LTZ in schema evolution, for both data and statistics, like the Java TIME cast rules. A TIME field is cast to STRING with the precision it declares, which also formats TIME values in system tables, and a TIMESTAMP_LTZ field is cast to TIME in the local time zone. variant_get casts strings and timestamps to TIME.
  • Support TIME in min/max aggregation, the realtime store and the hash index.

Not covered yet, as documented: TIME column default values, file or global indexes on TIME columns, and verifying that Paimon Java reads tables written by Paimon C++ and that Paimon C++ reads Java-written primary-key tables with TIME keys.

Tests

  • Unit tests for TIME string parsing and formatting (precisions 0/3/6/9, midnight, 00:00:00.123, 23:59:59.999, offsets, out-of-range and malformed input), literals, literal conversion, predicate validation and evaluation, row writers, serializers, comparators, columnar rows, row-to-Arrow conversion, the sort buffer, aggregation and the row equalizer.
  • Partition computer tests for TIME partition keys of precisions 0/3/6/9, legacy partition names, default partitions and invalid values; bucket tests for the default and Hive bucket functions and bucket pruning.
  • Cast tests for STRING/TIMESTAMP/TIMESTAMP_LTZ to and from TIME for arrays and literals, the cast executor factory, CastedRow, field mapping, statistics evolution and variant_get, including non-millisecond targets.
  • Query auth tests for TIME row filter literals, valid and malformed, and for evaluating TIME row filters and copying TIME columns in masks.
  • Format tests for ORC, Avro, Parquet and Mosaic TIME reads, writes and statistics, and the realtime store and hash index.
  • Integration tests: primary-key tables with TIME partition, primary and bucket keys across buckets, with upserts, deletes, full compaction, bucket and statistics pruning, and deletion vectors; append-only tables with TIME partition and value predicates; TIME schema evolution; the $files system table; and reading Java-written TIME tables in Parquet, ORC and Avro.
  • ReadResultCollector::SortArray now concatenates the chunks before sorting, which avoids an Arrow multi-chunk sort bug on TIME columns (Segmentation fault upon multi-key sorting of concatenated tables arrow#47252).

API and Format

  • API: adds FieldType::TIME in include/paimon/defs.h, and Literal(FieldType, int32_t) now also accepts FieldType::TIME.
  • Format: TIME is written to ORC as int and to Avro as int with the time-millis logical type, both matching Paimon Java. TIME partition values and bucket ids follow Paimon Java.

Documentation

Yes. docs/source/user_guide/data_types.rst documents the TIME support of primary-key and append-only tables, partition and bucket keys, predicates and query auth row filters, each file format and its statistics, string parsing, schema evolution casts, and what is not supported yet.

Generative AI tooling

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

@SteNicholas
SteNicholas force-pushed the PAIMON-540 branch 3 times, most recently from 384be3c to 1da6a8f Compare November 11, 2026 05:04
Support TIME columns in primary-key tables, as partition and bucket
keys, in predicates, and in ORC, Avro and Mosaic files, following Paimon
Java. TIME stays an Arrow time32[ms] holding the milliseconds of the
day, with its declared precision in the paimon.time.precision field
metadata.

- Add FieldType::TIME and Literal(FieldType::TIME, millis).
- Read and write TIME as int32 in BinaryRow, serializers, comparators,
  the sort buffer, columnar rows and row-to-Arrow conversion.
- Validate and evaluate TIME predicates, and push them down to Parquet
  and ORC. Parse the TIME literals of query auth row filters like Java
  LocalTime.parse.
- Allow TIME partition keys. Format their values like Java
  TimeToStringCastRule, or as milliseconds with partition.legacy-name,
  and parse them like BinaryStringUtils.toTime. Partition value errors
  now name the partition key.
- Hash TIME bucket keys as int32 in the default and Hive bucket
  functions and in bucket pruning.
- Map TIME to an ORC int and to an Avro int with the time-millis
  logical type, rejecting Avro precisions above 3, and report TIME
  file statistics as int statistics.
- Reject time32 units other than milliseconds in predicates, partition
  and bucket keys, variant_get targets, and ORC and Avro reads and
  writes.
- Cast between TIME and STRING, TIMESTAMP or TIMESTAMP_LTZ in schema
  evolution, for both data and statistics, like the Java TIME cast
  rules. A TIME field is cast to STRING with the precision it declares,
  which also formats TIME values in system tables, and a TIMESTAMP_LTZ
  field is cast to TIME in the local time zone. variant_get casts
  strings and timestamps to TIME.
- Support TIME in min/max aggregation, the realtime store and the hash
  index.
- Document the TIME support of each file format.

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 2edb77e into apache:main Nov 12, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Complete TIME type support

3 participants