Skip to content

[spark] Delete partitions by value in metadata-only DELETE - #11296

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:spark-delete-partitions-by-value
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:spark-delete-partitions-by-value

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

Fixes #11291. A DELETE whose condition covers only partition columns runs TruncatePaimonTableWithFilterExec, which turned the matched partitions back into string specs and let the commit parse them again. A blank value is written as the default partition name and parses back as NULL, a non-UTF-8 BINARY value comes back with other bytes, and a TIMESTAMP's legacy name shifts by a non-UTC JVM's zone offset, so the statement kept rows it matched and could remove rows it did not.

The exec now commits the matched BinaryRows through an empty static overwrite. Like INSERT OVERWRITE, that commit runs maintenance such as snapshot expiration afterwards. #11226 uses the same commit for DROP PARTITION and TRUNCATE PARTITION; whichever merges second can share the helper.

Tests

PaimonOptimizationTestBase: metadata-only deletes of a blank STRING partition next to a NULL one, of blank and non-UTF-8 BINARY partitions reached through another partition column, and of a TIMESTAMP partition in an Asia/Shanghai JVM whose legacy name parses back as another existing partition. Each checks the truncate exec was planned and only the matched rows are gone. They fail on master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] DELETE on partition columns removes the wrong partition when a partition name does not parse back

1 participant