Repository navigation
Description
Search before asking
- I searched in the issues and found no existing issue tracking the remaining end-to-end TIME support.
Related completed work: #292 / #378 added Parquet TIME reading, and #477 / #478 added Lance TIME support. This issue tracks the remaining gaps.
Motivation
Sub-issue of #519: Core Feature Alignment — Complete TIME type support.
Paimon C++ already parses TIME(p) and supports TIME in parts of the append-table and file-format paths. The current data type documentation still lists important restrictions:
- Primary-key tables cannot use TIME as either a key or a value column.
- TIME cannot be a partition or bucket key.
- Predicates cannot reference TIME columns.
- ORC and Avro TIME support, and end-to-end read/write/statistics coverage across formats, remain incomplete.
These gaps prevent Java-written TIME schemas from being used consistently through the C++ table APIs. The roadmap item should be completed through focused changes with explicit compatibility checks.
Solution
Preserve the existing Java-compatible representation: TIME(p), for p = 0..9, uses Arrow time32[ms] and an integer number of milliseconds since midnight. Preserve declared precision through paimon.time.precision metadata, including the default TIME(0). Higher declared precision must continue to follow Paimon Java's millisecond storage semantics.
Suggested implementation order:
- Complete scalar and row plumbing. Add the missing public/internal type mappings and literal conversions; support TIME in internal row/array access, binary serialization, comparisons, Arrow row conversion, statistics, and the casting/default-value paths needed for Java compatibility. Preserve existing public enum values and serialization compatibility when extending type handling.
- Enable predicates. Support equality, ordering,
IN/NOT IN, and null checks through predicate validation, row evaluation, Arrow batch evaluation, and statistics pruning. Validate that pruning never removes matching rows. - Enable primary-key tables. Support TIME value columns and TIME primary keys through write, commit, read, upsert, delete, and compaction, covering the existing applicable merge and deletion-vector paths.
- Enable partition and bucket keys. Align TIME partition string conversion and bucket hashing with Java, and validate routing and pruning using Java-generated tables.
- Complete file-format support. Audit the existing TIME paths and close read/write/statistics gaps, including Parquet table writes and ORC/Avro mappings. Preserve working Lance/Vortex paths and cover other enabled formats where their Java counterparts support TIME.
- Update documentation. Replace the current restrictions with the verified support matrix and document any remaining format-specific limitations.
Useful implementation entry points:
- Public field types and field type mapping.
- Literal conversion and predicate validation.
- Internal row access, binary serializers, and Arrow row conversion.
- Schema validation, partition conversion, bucket calculation, and format-specific adapters/statistics.
Anything else?
Acceptance and validation:
- Add focused unit and integration tests for the implemented paths.
- Cover
NULL, midnight,00:00:00.123,23:59:59.999, declared precisions0/3/6/9, precision-metadata round trips, and invalid precision/unit inputs. - Cover TIME predicates with and without pruning, primary-key upserts/deletes and compaction, partition/bucket routing, and format read/write finalization.
- Validate Java-written data from C++ and C++-written data from Java for supported formats and table modes; include key hashing, partition values, nulls, and declared precision.
- Keep existing compatibility tests passing and update the documented support matrix.
This issue may be implemented in multiple focused PRs, starting with scalar/predicate support and primary-key value columns. Keep the parent roadmap item open until the full agreed scope and compatibility validation are complete.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity