Skip to content

[Feature] Complete TIME type support #540

Closed

Description

@SteNicholas

Search before asking

  • I searched in the issues and found no existing issue tracking the remaining end-to-end TIME support.

Related completed work: #292 / #378 added Parquet TIME reading, and #477 / #478 added Lance TIME support. This issue tracks the remaining gaps.

Motivation

Sub-issue of #519: Core Feature Alignment — Complete TIME type support.

Paimon C++ already parses TIME(p) and supports TIME in parts of the append-table and file-format paths. The current data type documentation still lists important restrictions:

  • Primary-key tables cannot use TIME as either a key or a value column.
  • TIME cannot be a partition or bucket key.
  • Predicates cannot reference TIME columns.
  • ORC and Avro TIME support, and end-to-end read/write/statistics coverage across formats, remain incomplete.

These gaps prevent Java-written TIME schemas from being used consistently through the C++ table APIs. The roadmap item should be completed through focused changes with explicit compatibility checks.

Solution

Preserve the existing Java-compatible representation: TIME(p), for p = 0..9, uses Arrow time32[ms] and an integer number of milliseconds since midnight. Preserve declared precision through paimon.time.precision metadata, including the default TIME(0). Higher declared precision must continue to follow Paimon Java's millisecond storage semantics.

Suggested implementation order:

  • Complete scalar and row plumbing. Add the missing public/internal type mappings and literal conversions; support TIME in internal row/array access, binary serialization, comparisons, Arrow row conversion, statistics, and the casting/default-value paths needed for Java compatibility. Preserve existing public enum values and serialization compatibility when extending type handling.
  • Enable predicates. Support equality, ordering, IN / NOT IN, and null checks through predicate validation, row evaluation, Arrow batch evaluation, and statistics pruning. Validate that pruning never removes matching rows.
  • Enable primary-key tables. Support TIME value columns and TIME primary keys through write, commit, read, upsert, delete, and compaction, covering the existing applicable merge and deletion-vector paths.
  • Enable partition and bucket keys. Align TIME partition string conversion and bucket hashing with Java, and validate routing and pruning using Java-generated tables.
  • Complete file-format support. Audit the existing TIME paths and close read/write/statistics gaps, including Parquet table writes and ORC/Avro mappings. Preserve working Lance/Vortex paths and cover other enabled formats where their Java counterparts support TIME.
  • Update documentation. Replace the current restrictions with the verified support matrix and document any remaining format-specific limitations.

Useful implementation entry points:

Anything else?

Acceptance and validation:

  • Add focused unit and integration tests for the implemented paths.
  • Cover NULL, midnight, 00:00:00.123, 23:59:59.999, declared precisions 0/3/6/9, precision-metadata round trips, and invalid precision/unit inputs.
  • Cover TIME predicates with and without pruning, primary-key upserts/deletes and compaction, partition/bucket routing, and format read/write finalization.
  • Validate Java-written data from C++ and C++-written data from Java for supported formats and table modes; include key hashing, partition values, nulls, and declared precision.
  • Keep existing compatibility tests passing and update the documented support matrix.

This issue may be implemented in multiple focused PRs, starting with scalar/predicate support and primary-key value columns. Keep the parent roadmap item open until the full agreed scope and compatibility validation are complete.

Are you willing to submit a PR?

  • I'm willing to submit a PR!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions