fix: respect path boundary in compareToFileList location prefix matching (#16493) - #17607
Open
waterWang wants to merge 1 commit into
Open
fix: respect path boundary in compareToFileList location prefix matching (#16493)#17607waterWang wants to merge 1 commit into
waterWang wants to merge 1 commit into
Conversation
…ing (apache#16493) When using compareToFileList(Dataset), the file_list_view is filtered by startsWith(tableLocation). This raw string prefix match incorrectly includes sibling paths that share a prefix with the table location, e.g. s3://bucket/table-backup/... would match a table at s3://bucket/table. Fix: normalize the location to ensure it ends with path separator before using startsWith, and also accept exact match for the location itself. This prevents orphan file cleanup from operating on files outside the intended table directory when the user provides a file_list_view. Test: testCompareToFileListDoesNotMatchSiblingPaths verifies that sibling paths (table-backup/..., table_old/...) are not treated as in-scope orphan candidates.
uros-b
reviewed
Aug 13, 2026
uros-b
left a comment
uros-b
left a comment
Member
There was a problem hiding this comment.
@waterWang Please note that this might already be in progress #16498.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #16493
Why
When using
compareToFileList(Dataset), thefile_list_viewis filtered bystartsWith(tableLocation). This raw string prefix match incorrectly includes sibling paths that share a prefix with the table location. For example, a table ats3://bucket/tablewould also matchs3://bucket/table-backup/data/file.parquetors3://bucket/table_old/data/file.parquet, treating files outside the intended directory as in-scope orphan candidates.Summary
Normalize the location to ensure it ends with a path separator (
/) before usingstartsWith, and also accept an exact match for the location itself. This ensures that only files actually under the table directory (or equal to the location) are treated as in-scope.How to Test
New test
testCompareToFileListDoesNotMatchSiblingPathsverifies that:tableLocation + "-backup/"(sibling prefix) are NOT in scopetableLocation + "_old/"(sibling prefix) are NOT in scopetableLocation + "/data/"are correctly in scopeType
Checklist