You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] DELETE on partition columns removes the wrong partition when a partition name does not parse back #11291
Compute Engine
Spark 4.1.4. The blank STRING case also reproduces on Spark 3.5.9.
Minimal reproduce step
On a native table, a DELETE whose condition covers only partition columns and converts to a Paimon predicate plans TruncatePaimonTableWithFilterExec (unless delete.force-produce-changelog is set). The exec finds the matching partitions, turns each one back into a string spec with InternalRowPartitionComputer.generatePartValues, and calls truncatePartitions, which parses the specs again. Values that do not survive that round trip remove the wrong partition:
CREATETABLET (a INT, p STRING) PARTITIONED BY (p);
INSERT INTO T VALUES (1, ''), (2, NULL), (3, 'a');
DELETEFROM T WHERE p ='';
SELECT a FROM T; -- 1, 3
The blank value is written as __DEFAULT_PARTITION__, which parses back as NULL. A blank BINARY value does the same when the condition is on another partition column:
CREATETABLET (a INT, p1 STRING, p2 BINARY) PARTITIONED BY (p1, p2);
INSERT INTO T VALUES (1, 'x', CAST(''AS BINARY)), (2, 'x', NULL), (3, 'y', NULL);
DELETEFROM T WHERE p1 ='x';
SELECT a FROM T; -- 1, 3
With the default partition.legacy-name, a TIMESTAMP value is written as its UTC wall-clock time and parsed back in the JVM time zone. In an Asia/Shanghai JVM:
CREATETABLET (a INT, p TIMESTAMP) PARTITIONED BY (p);
INSERT INTO T VALUES (1, CAST('2024-06-01 10:30:00'ASTIMESTAMP)),
(2, CAST('2024-06-01 02:30:00'ASTIMESTAMP));
DELETEFROM T WHERE p =TIMESTAMP'2024-06-01 10:30:00';
SELECT a FROM T; -- 1
What doesn't meet your expectations?
The three statements should leave 2, 3, 3 and 2. Each reports success but keeps rows its condition matches, and the first and third also remove rows it does not match.
Anything else?
The exec already holds the matched partitions as BinaryRows, so it can commit them with withOverwriteStaticPartitions (with dynamic-partition-overwrite off) instead of rebuilding string specs. #11226 does the same for ALTER TABLE ... DROP PARTITION and TRUNCATE TABLE ... PARTITION.
Activity