Skip to content

[spark] Match BINARY partition values by content in partition management - #11226

Open
LuciferYang wants to merge 2 commits into
apache:masterfrom
LuciferYang:spark-binary-partition-identifiers
Open

LuciferYang wants to merge 2 commits into
apache:masterfrom
LuciferYang:spark-binary-partition-identifiers

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PaimonPartitionManagement.listPartitionIdentifiers compared values with Objects.equals, so BINARY values (byte[]) never matched: SHOW PARTITIONS with a spec found nothing, full-spec DROP PARTITION and Spark 4.2's DESCRIBE ... PARTITION failed, ADD PARTITION of an existing value succeeded, and partial TRUNCATE failed. It now uses Objects.deepEquals.

DROP and TRUNCATE passed string specs to the commit, which parses them back, so a blank value (named like NULL) or a TIMESTAMP's legacy name in a non-UTC JVM removed the wrong partition or none. They now remove the stored partitions matching by value, by BinaryRow, through an empty static overwrite, which runs maintenance such as snapshot expiration as INSERT OVERWRITE does. A partial TRUNCATE matching nothing no longer fails; a literal default partition name matching nothing fails, suggesting p = NULL. Hive tables with metastore.partitioned-table are dropped by name and reject such values. SHOW TABLE EXTENDED matches by value too.

DELETE has the same problem: #11291.

Tests

  • PaimonPartitionManagementTest: SHOW PARTITIONS with a spec, ADD PARTITION of an existing value, DROP PARTITION and partial TRUNCATEs on a BINARY column, one matching nothing. Next to a NULL partition, dropping a blank BINARY or STRING value removes only it, and SHOW TABLE EXTENDED reports its row count. A partial TRUNCATE removes blank and non-UTF-8 BINARY partitions, a TRUNCATE removes a partition a batch read skips, dropping a TIMESTAMP whose legacy name parses as another partition removes the right one, and the literal default partition name fails.
  • DDLWithHiveCatalogTestBase: through the Hive catalog, dropping a blank partition removes only it, or with metastore.partitioned-table fails and keeps every row.
  • Spark42CompatibilityTest: DESCRIBE ... PARTITION on a BINARY partition column, with spark.sql.binaryOutputStyle set to UTF-8 so the value renders as text.

The paimon-spark-ut tests fail on master except the batch-read TRUNCATE, which checks partitions come from the snapshot.

@Akash3121 Akash3121 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The normal Spark 3.5.9 partition-management suite passed all 14 tests locally, and Spark 4.2 CI passed its 39-test suite including the new BINARY DESCRIBE case. A separate Spark 3.5.9 SQL test reproduced deletion of the wrong partition.

val structField = partitionSchema.fields(internalRowIndex)
Objects.equals(
// `deepEquals`: a BINARY partition value is a `byte[]`.
Objects.deepEquals(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] The new match can delete the NULL partition when the user drops a non-NULL BINARY partition. I reproduced this on Spark 3.5.9 using a native table T (a INT, p BINARY) PARTITIONED BY (p) with (1, CAST(' ' AS BINARY)), (2, NULL), (3, CAST('a' AS BINARY)). ALTER TABLE T DROP PARTITION (p = ' ') succeeds, then SELECT a FROM T returns 1, 3 (expected 2, 3). deepEquals accepts the one-space bytes in the existence check. The commit then converts the same identifier through generatePartValues, maps whitespace to the default partition name, and rebuilds a NULL BinaryRow for the delete. Please carry the typed BINARY identifier through the destructive operation, or reject identifiers that cannot round-trip before committing. Add a regression with both partitions populated.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced; the STRING version drops the NULL partition on master too. DROP and TRUNCATE now carry the typed identifier through: they match the table's partitions by value, as listPartitionIdentifiers does, and remove the matches by BinaryRow with withOverwriteStaticPartitions. A Hive table with metastore.partitioned-table is dropped through PartitionModification.dropPartitions, which takes partition names, so there a value whose name does not parse back is rejected before anything is removed.

The new tests put ' ', NULL and 'a' in one table, for BINARY and for STRING, and check that dropping ' ' leaves the other two. A Hive catalog test covers the STRING case with and without metastore.partitioned-table.

The same spec made SHOW TABLE EXTENDED ... PARTITION (p = ' ') show the NULL partition's statistics; that lookup now matches by value as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants