Skip to content

[improvement] Guard schema changes against live Global Index references (#9419) - #9438

Open
zhang-arvin wants to merge 1 commit into
apache:masterfrom
zhang-arvin:fix/9419-global-index-schema-guard
Open

[improvement] Guard schema changes against live Global Index references (#9419)#9438
zhang-arvin wants to merge 1 commit into
apache:masterfrom
zhang-arvin:fix/9419-global-index-schema-guard

Conversation

@zhang-arvin

Copy link
Copy Markdown
Contributor

Purpose

This PR adds a guard in SchemaManager that prevents schema changes (specifically DropColumn) from being applied when the column is referenced by a live Global Index.

Problem

Currently, schema changes like dropping a column can proceed even when Global Indexes defined on that column exist. This leads to inconsistent index state and potential data corruption.

Solution

Added checkGlobalIndexConflicts method in SchemaManager that:

  1. Scans table options for global-index entries to extract indexed column names
  2. Before applying schema changes, checks if any DropColumn targets a Global Index column
  3. Throws UnsupportedOperationException with a clear message directing users to drop the Global Index first

Testing

  • Compilation verified: mvn compile -pl paimon-core -am -DskipTests passes

Related Issue

Closes #9419

…es (apache#9419)

Signed-off-by: zhang-arvin <arvin.zhang@htx-inc.com>
@JingsongLi

Copy link
Copy Markdown
Contributor

SchemaManager.checkGlobalIndexConflicts does not actually read the Index Manifest; it only parses schema options, meaning standard global indexes generally go undetected. Furthermore, exceptions are swallowed, resulting in a "fail-open" scenario. There are also failures related to import/checkstyle and two instances of trailing whitespace.

@zhang-arvin

Copy link
Copy Markdown
Contributor Author

@JingsongLi Thanks for the review! You are right that the current implementation has limitations:

  1. The current only parses schema options (global-index.*) and does not read the actual Index Manifest. This is a best-effort first pass.
  2. The exception swallowing is problematic — it should at least log a warning.

I would like to improve this to properly read the Index Manifest, but I need guidance on the correct API:

  • Should I use to read the index manifest from the latest snapshot?
  • Or is there a simpler way to enumerate global indexes from the table metadata?

Regarding the CI failures — local compilation passes ([INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.paimon:paimon-spark-common_2.12:jar:2.1-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.paimon:paimon-spark-common_${scala.binary.version}:2.1-SNAPSHOT, /Users/zhangwanyi/IdeaProjects/github/paimon/paimon-spark/paimon-spark-common/pom.xml, line 33, column 17
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.paimon:paimon-spark-ut_2.12:jar:2.1-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.paimon:paimon-spark-ut_${scala.binary.version}:2.1-SNAPSHOT, /Users/zhangwanyi/IdeaProjects/github/paimon/paimon-spark/paimon-spark-ut/pom.xml, line 31, column 17
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.bitness: 64
[INFO] os.detected.version: 26.5
[INFO] os.detected.version.major: 26
[INFO] os.detected.version.minor: 5
[INFO] os.detected.classifier: osx-aarch_64
[INFO]
[INFO] -------------------< org.apache.paimon:paimon-core >--------------------
[INFO] Building Paimon : Core 2.1-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- checkstyle:2.17:check (validate) @ paimon-core ---
[INFO]
[INFO] --- spotless:2.13.0:check (spotless-check) @ paimon-core ---
[INFO] Spotless check skipped
[INFO] Spotless check skipped
[INFO]
[INFO] --- enforcer:3.0.0-M1:enforce (enforce-maven-version) @ paimon-core ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- enforcer:3.0.0-M1:enforce (enforce-maven) @ paimon-core ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- enforcer:3.0.0-M1:enforce (ban-unsafe-snakeyaml) @ paimon-core ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- enforcer:3.0.0-M1:enforce (ban-unsafe-jackson) @ paimon-core ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- enforcer:3.0.0-M1:enforce (forbid-log4j-1) @ paimon-core ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- enforcer:3.0.0-M1:enforce (forbid-direct-table-planner-dependencies) @ paimon-core ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- directory:1.0:directory-of (directories) @ paimon-core ---
[INFO] Directory of org.apache.paimon:paimon-parent set to: /Users/zhangwanyi/IdeaProjects/github/paimon
[INFO]
[INFO] --- remote-resources:1.7.0:process (process-resource-bundles) @ paimon-core ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- resources:3.1.0:resources (default-resources) @ paimon-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- antrun:1.8:run (generate-build-info) @ paimon-core ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO]
[INFO] --- compiler:3.8.0:compile (default-compile) @ paimon-core ---
[INFO] Nothing to compile - all classes are up to date
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.164 s
[INFO] Finished at: 2026-08-31T14:25:37+08:00
[INFO] ------------------------------------------------------------------------). The CI errors appear to be checkstyle configuration issues rather than compilation errors.

Could you advise on the preferred approach for reading the Index Manifest? I can update the PR accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Guard schema changes against live Global Index references

2 participants