Skip to content

[core] use TIMESTAMP_LTZ for epoch-based system table timestamps in J… - #11286

Open
fengruotj wants to merge 1 commit into
apache:masterfrom
fengruotj:fix/snapshot-commit-time
Open

fengruotj wants to merge 1 commit into
apache:masterfrom
fengruotj:fix/snapshot-commit-time

Conversation

@fengruotj

@fengruotj fengruotj commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Implements the java language proposed in #11219

Purpose

Fixes time zone handling for commit_time in the $snapshots and $tags system tables. Both columns represent the snapshot commit instant. Previously, the Java implementation converted the stored epoch milliseconds to a local TIMESTAMP(3) using the JVM time zone. When the JVM and Spark session used different time zones, comparisons against an absolute timestamp could return incorrect results.

This PR changes both columns to TIMESTAMP_LTZ(3) in Java and constructs their values directly from the stored epoch milliseconds.

PyPaimon will depend on paimon-rust. Follow-up PRs will update both paimon-rust and PyPaimon to use the same commit_time type and time zone semantics as the Java implementation.

Tests

  • Updates SnapshotsTableTest and TagsTableTest to verify the commit_time column type and value.
  • Adds a Spark SQL regression test with the JVM time zone set to Asia/Shanghai and the session time zone set to UTC. It verifies that a timestamp filter includes the expected snapshot and that unix_millis(commit_time) returns the original snapshot epoch milliseconds for both the snapshot and its tag.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the snapshot instant for both system tables addresses the JVM-dependent results in #11219. All 13 Core snapshot and tag tests passed on a clean integration with master. I found two reader compatibility issues with the type change.

[P1] Preserve the epoch value with Spark legacy timestamp mapping. I committed a row, created its tag, and joined $snapshots to $tags on Spark 3.5.9 with an Asia/Shanghai JVM and a UTC SQL session. With spark.paimon.legacy-timestamp-mapping.enabled=false, both unix_millis(commit_time) values matched the stored 1795146612121. Setting it to true returned 1795117812121 for both, eight hours early. On master, the legacy-mapping query returned the correct instant. AbstractSparkInternalRow.getTimestampMicros calls the converter without the Paimon type. The legacy branch calls Timestamp.toSQLTimestamp(), applying the JVM zone to the new epoch-backed value. Spark 3.2 and 3.3 always use that branch. Please pass through epoch micros for TIMESTAMP_LTZ and exercise both mapping settings in the SQL regression.

[P2] Coordinate the StarRocks reader before upgrading its Paimon dependency. StarRocks reads $snapshots.commit_time for unpartitioned table freshness and converts it using commitTime.toLocalDateTime().atZone(ZoneId.systemDefault()).toInstant(). I ran that conversion with an Asia/Shanghai JVM and both Paimon timestamp representations. A commit at 2026-11-17T06:31:24.761Z produced the same value before this change and 2026-11-16T22:31:24.761Z afterward. The earlier timestamp can affect materialized-view refresh and staleness decisions. Please document the type transition and track a StarRocks reader update that uses epoch milliseconds for LTZ and retains the existing conversion for older Paimon schemas. StarRocks consumer

@fengruotj fengruotj closed this Nov 20, 2026
@fengruotj fengruotj reopened this Nov 20, 2026
@fengruotj

fengruotj commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

@JingsongLi Thanks for the repro.

[P1] Preserve the epoch value with Spark legacy timestamp mapping.
I confirmed the eight-hour shift with spark-sql on Spark 3.5.9, using an Asia/Shanghai JVM and a UTC SQL session. I also checked an ordinary TIMESTAMP_LTZ column and an element in ARRAY<TIMESTAMP_LTZ>. With legacy mapping enabled, both read eight hours early on the base commit and on this PR. With legacy mapping disabled, both preserved the stored epoch.

This matches the historical behavior. Paimon 0.8 mapped both Paimon TimestampType and LocalZonedTimestampType to Spark TimestampType. The newer mapping distinguishes them on Spark 3.4+, and spark.paimon.legacy-timestamp-mapping.enabled=true restores the old mapping for workloads that depend on it. The Paimon docs describe the legacy conversion as correct for Paimon TimestampType only. On Spark 3.3 and earlier, the connector always selects that conversion, so readers may need to adjust for the JVM time-zone offset manually when reading LTZ values. I confirmed that setting the option to false made no difference on Spark 3.2.4 or 3.3.4.

[P2] Coordinate the StarRocks reader before upgrading its Paimon dependency.
Got it. This change looks like it has compatibility implications for several downstream readers. I'll hold off on moving it forward for now and leave the PR open for discussion.

@fengruotj fengruotj closed this Nov 20, 2026
@fengruotj fengruotj reopened this Nov 20, 2026
@fengruotj fengruotj closed this Nov 20, 2026
@fengruotj fengruotj reopened this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants