Repository navigation
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Using the snapshot instant for both system tables addresses the JVM-dependent results in #11219. All 13 Core snapshot and tag tests passed on a clean integration with master. I found two reader compatibility issues with the type change.
[P1] Preserve the epoch value with Spark legacy timestamp mapping. I committed a row, created its tag, and joined $snapshots to $tags on Spark 3.5.9 with an Asia/Shanghai JVM and a UTC SQL session. With spark.paimon.legacy-timestamp-mapping.enabled=false, both unix_millis(commit_time) values matched the stored 1795146612121. Setting it to true returned 1795117812121 for both, eight hours early. On master, the legacy-mapping query returned the correct instant. AbstractSparkInternalRow.getTimestampMicros calls the converter without the Paimon type. The legacy branch calls Timestamp.toSQLTimestamp(), applying the JVM zone to the new epoch-backed value. Spark 3.2 and 3.3 always use that branch. Please pass through epoch micros for TIMESTAMP_LTZ and exercise both mapping settings in the SQL regression.
[P2] Coordinate the StarRocks reader before upgrading its Paimon dependency. StarRocks reads $snapshots.commit_time for unpartitioned table freshness and converts it using commitTime.toLocalDateTime().atZone(ZoneId.systemDefault()).toInstant(). I ran that conversion with an Asia/Shanghai JVM and both Paimon timestamp representations. A commit at 2026-11-17T06:31:24.761Z produced the same value before this change and 2026-11-16T22:31:24.761Z afterward. The earlier timestamp can affect materialized-view refresh and staleness decisions. Please document the type transition and track a StarRocks reader update that uses epoch milliseconds for LTZ and retains the existing conversion for older Paimon schemas. StarRocks consumer
|
@JingsongLi Thanks for the repro. [P1] Preserve the epoch value with Spark legacy timestamp mapping. This matches the historical behavior. Paimon 0.8 mapped both Paimon [P2] Coordinate the StarRocks reader before upgrading its Paimon dependency. |
Implements the java language proposed in #11219
Purpose
Fixes time zone handling for
commit_timein the$snapshotsand$tagssystem tables. Both columns represent the snapshot commit instant. Previously, the Java implementation converted the stored epoch milliseconds to a localTIMESTAMP(3)using the JVM time zone. When the JVM and Spark session used different time zones, comparisons against an absolute timestamp could return incorrect results.This PR changes both columns to
TIMESTAMP_LTZ(3)in Java and constructs their values directly from the stored epoch milliseconds.PyPaimon will depend on paimon-rust. Follow-up PRs will update both paimon-rust and PyPaimon to use the same commit_time type and time zone semantics as the Java implementation.
Tests
SnapshotsTableTestandTagsTableTestto verify thecommit_timecolumn type and value.Asia/Shanghaiand the session time zone set toUTC. It verifies that a timestamp filter includes the expected snapshot and thatunix_millis(commit_time)returns the original snapshot epoch milliseconds for both the snapshot and its tag.