Skip to content

[Bug] DELETE on partition columns removes the wrong partition when a partition name does not parse back #11291

Open

Description

@LuciferYang

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version
master @ 2bf10c1

Compute Engine
Spark 4.1.4. The blank STRING case also reproduces on Spark 3.5.9.

Minimal reproduce step
On a native table, a DELETE whose condition covers only partition columns and converts to a Paimon predicate plans TruncatePaimonTableWithFilterExec (unless delete.force-produce-changelog is set). The exec finds the matching partitions, turns each one back into a string spec with InternalRowPartitionComputer.generatePartValues, and calls truncatePartitions, which parses the specs again. Values that do not survive that round trip remove the wrong partition:

CREATE TABLE T (a INT, p STRING) PARTITIONED BY (p);
INSERT INTO T VALUES (1, ' '), (2, NULL), (3, 'a');
DELETE FROM T WHERE p = ' ';
SELECT a FROM T;  -- 1, 3

The blank value is written as __DEFAULT_PARTITION__, which parses back as NULL. A blank BINARY value does the same when the condition is on another partition column:

CREATE TABLE T (a INT, p1 STRING, p2 BINARY) PARTITIONED BY (p1, p2);
INSERT INTO T VALUES (1, 'x', CAST(' ' AS BINARY)), (2, 'x', NULL), (3, 'y', NULL);
DELETE FROM T WHERE p1 = 'x';
SELECT a FROM T;  -- 1, 3

With the default partition.legacy-name, a TIMESTAMP value is written as its UTC wall-clock time and parsed back in the JVM time zone. In an Asia/Shanghai JVM:

CREATE TABLE T (a INT, p TIMESTAMP) PARTITIONED BY (p);
INSERT INTO T VALUES (1, CAST('2024-06-01 10:30:00' AS TIMESTAMP)),
  (2, CAST('2024-06-01 02:30:00' AS TIMESTAMP));
DELETE FROM T WHERE p = TIMESTAMP '2024-06-01 10:30:00';
SELECT a FROM T;  -- 1

What doesn't meet your expectations?
The three statements should leave 2, 3, 3 and 2. Each reports success but keeps rows its condition matches, and the first and third also remove rows it does not match.

Anything else?
The exec already holds the matched partitions as BinaryRows, so it can commit them with withOverwriteStaticPartitions (with dynamic-partition-overwrite off) instead of rebuilding string specs. #11226 does the same for ALTER TABLE ... DROP PARTITION and TRUNCATE TABLE ... PARTITION.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions