Repository navigation
[spark] Match BINARY partition values by content in partition management - #11226
LuciferYang wants to merge 2 commits into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
The normal Spark 3.5.9 partition-management suite passed all 14 tests locally, and Spark 4.2 CI passed its 39-test suite including the new BINARY DESCRIBE case. A separate Spark 3.5.9 SQL test reproduced deletion of the wrong partition.
| val structField = partitionSchema.fields(internalRowIndex) | ||
| Objects.equals( | ||
| // `deepEquals`: a BINARY partition value is a `byte[]`. | ||
| Objects.deepEquals( |
There was a problem hiding this comment.
[P1] The new match can delete the NULL partition when the user drops a non-NULL BINARY partition. I reproduced this on Spark 3.5.9 using a native table T (a INT, p BINARY) PARTITIONED BY (p) with (1, CAST(' ' AS BINARY)), (2, NULL), (3, CAST('a' AS BINARY)). ALTER TABLE T DROP PARTITION (p = ' ') succeeds, then SELECT a FROM T returns 1, 3 (expected 2, 3). deepEquals accepts the one-space bytes in the existence check. The commit then converts the same identifier through generatePartValues, maps whitespace to the default partition name, and rebuilds a NULL BinaryRow for the delete. Please carry the typed BINARY identifier through the destructive operation, or reject identifiers that cannot round-trip before committing. Add a regression with both partitions populated.
There was a problem hiding this comment.
Reproduced; the STRING version drops the NULL partition on master too. DROP and TRUNCATE now carry the typed identifier through: they match the table's partitions by value, as listPartitionIdentifiers does, and remove the matches by BinaryRow with withOverwriteStaticPartitions. A Hive table with metastore.partitioned-table is dropped through PartitionModification.dropPartitions, which takes partition names, so there a value whose name does not parse back is rejected before anything is removed.
The new tests put ' ', NULL and 'a' in one table, for BINARY and for STRING, and check that dropping ' ' leaves the other two. A Hive catalog test covers the STRING case with and without metastore.partitioned-table.
The same spec made SHOW TABLE EXTENDED ... PARTITION (p = ' ') show the NULL partition's statistics; that lookup now matches by value as well.
Purpose
PaimonPartitionManagement.listPartitionIdentifierscompared values withObjects.equals, so BINARY values (byte[]) never matched:SHOW PARTITIONSwith a spec found nothing, full-specDROP PARTITIONand Spark 4.2'sDESCRIBE ... PARTITIONfailed,ADD PARTITIONof an existing value succeeded, and partialTRUNCATEfailed. It now usesObjects.deepEquals.DROP and TRUNCATE passed string specs to the commit, which parses them back, so a blank value (named like NULL) or a TIMESTAMP's legacy name in a non-UTC JVM removed the wrong partition or none. They now remove the stored partitions matching by value, by
BinaryRow, through an empty static overwrite, which runs maintenance such as snapshot expiration asINSERT OVERWRITEdoes. A partialTRUNCATEmatching nothing no longer fails; a literal default partition name matching nothing fails, suggestingp = NULL. Hive tables withmetastore.partitioned-tableare dropped by name and reject such values.SHOW TABLE EXTENDEDmatches by value too.DELETEhas the same problem: #11291.Tests
PaimonPartitionManagementTest:SHOW PARTITIONSwith a spec,ADD PARTITIONof an existing value,DROP PARTITIONand partialTRUNCATEs on a BINARY column, one matching nothing. Next to a NULL partition, dropping a blank BINARY or STRING value removes only it, andSHOW TABLE EXTENDEDreports its row count. A partialTRUNCATEremoves blank and non-UTF-8 BINARY partitions, aTRUNCATEremoves a partition a batch read skips, dropping a TIMESTAMP whose legacy name parses as another partition removes the right one, and the literal default partition name fails.DDLWithHiveCatalogTestBase: through the Hive catalog, dropping a blank partition removes only it, or withmetastore.partitioned-tablefails and keeps every row.Spark42CompatibilityTest:DESCRIBE ... PARTITIONon a BINARY partition column, withspark.sql.binaryOutputStyleset toUTF-8so the value renders as text.The
paimon-spark-uttests fail on master except the batch-readTRUNCATE, which checks partitions come from the snapshot.