Spark 4.1: Implement RepairTable action - #17622
Draft
rahulsmahadev wants to merge 2 commits into
Draft
Conversation
Adds a Spark implementation of the RepairTable action, which repairs manifest entries whose statistics disagree with the files they refer to. The statistics of every live entry are compared against the file by reading its footer, and only the manifests that contain at least one incorrect entry are rewritten, so the cost of the commit is proportional to the number of incorrect entries rather than to the size of the table. Entries are rewritten with ManifestWriter#existing, carrying through the original snapshot id and data and file sequence numbers. This preserves the lineage of the files, and therefore which delete files apply to them, so the repair leaves the contents of the table unchanged. Repairing statistics does not change the number of live files, so the commit goes through the existing rewrite manifests validation unchanged. - RepairMetrics reads and compares statistics per format (Parquet, ORC and Avro) and rebuilds a file with corrected statistics. - Files whose statistics cannot be read are carried through unchanged and counted as incorrect but not repaired, which is what distinguishes entryStatsIncorrectCount from entryStatsRepairedCount. - The repair-column-metrics option skips the footer reads and repairs only record counts and file sizes, for tables where only those are suspect. - dryRun reports what would be repaired without committing.
Covers what happens when the table changes underneath a repair, and what is left behind when the commit does not succeed: - a concurrent append commits between planning and the repair commit: the repair succeeds and the appended records survive, since the appended data lands in a new manifest and the manifests being repaired are still present - a concurrent operation replaces the very manifest being repaired: the commit fails validation in BaseRewriteManifests#validateDeletedManifests and the table is left untouched, rather than dropping the concurrent change - a failed commit deletes the manifests the repair wrote - a commit reported as CommitStateUnknownException keeps them, as the commit may have succeeded - a dry run leaves none of them behind Note the cleanup tests assert on the manifests written by the action itself. A failed commit on a format version 1 table can also leave behind a copy of a manifest made by the core staging path in BaseRewriteManifests, which is only cleaned up after a successful commit and is not owned by this action.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spark implementation of the
RepairTableaction whose API was added in #17399.repairFileMetrics()compares the metrics recorded in each live manifest entry (record count, file size, column bounds/null/nan/value counts) against the underlying data and delete files, and rewrites only the manifests that contain an incorrect entry. Rewritten entries carry through their original snapshot id and sequence numbers, so the repair does not change which delete files apply.dryRun()reports what would be repaired without committing.Scoped to Spark 4.1 for now; happy to backport once this lands.
Tests in
TestRepairTableActioncover: no-op on correct stats, record-count/file-size/column-metric repair, dry run, partitioned tables, and concurrency (concurrent append, a conflicting manifest rewrite, commit-state-unknown, and cleanup of manifests written by a failed or dry-run repair).