Skip to content

[Fix-18780][Storage-OSS] Delete non-empty OSS directories recursively - #18802

Open
det101 wants to merge 9 commits into
apache:devfrom
det101:fix-18780-oss-recursive-delete
Open

det101 wants to merge 9 commits into
apache:devfrom
det101:fix-18780-oss-recursive-delete

Conversation

@det101

@det101 det101 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Was this PR generated or assisted by AI?

YES. Both the code and the PR description were AI-assisted.

Purpose of the pull request

Deleting a non-empty directory in the resource center with resource.storage.type=OSS only removed the directory marker. Child objects stayed in the bucket, so the directory prefix was still listed.

Fixes #18780.

Brief change log

  • When recursive is true, OssStorageOperator#delete lists every object under <path>/ and deletes them with deleteObjects in pages of 1000.
  • File deletes and recursive=false still delete a single object.
  • Add unit tests with a mocked OSS client.
  • OBS, COS, and GCS still ignore recursive in delete.

Verify this pull request

This change added tests and can be verified as follows:

  • Added OssStorageOperatorTest with a mocked OSS client (5 tests passed).
  • Ran ./mvnw -pl dolphinscheduler-storage-plugin/dolphinscheduler-storage-oss spotless:check.

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

Resource center deletion passes recursive=true, but OssStorageOperator
only removed the directory marker and left child objects in the bucket.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please follow the pull request template first.

@det101

det101 commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I've updated the PR description to follow the pull request template.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allow Resource Center to delete OSS directories without a marker

The old implementation can leave child objects behind after deleting the
directory marker. Resource Center still lists the directory as a common
prefix, but DeleteResourceDtoValidator calls OssStorageOperator.exists(),
which checks only the marker. After upgrading, the delete request still fails
before reaching the new recursive deletion code.

For recursive OSS deletes, treat a prefix with child objects as an existing
directory. Add a regression test through the Resource Center delete path with
a child object and no directory marker.

Surface partial OSS batch deletion failures

OssStorageOperator.deleteObjectsByPrefix() enables quiet mode and discards
the batch result. If OSS leaves a child object undeleted, Resource Center can
report success while the directory remains visible.

Use verbose mode, compare the confirmed keys with the requested keys, and
report a failure that identifies the unconfirmed keys. Add a regression test
where OSS confirms only a subset of the batch.

OSS batch delete API

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] [Storage-OSS] Deleting a non-empty directory in the resource center leaves child objects on Aliyun OSS

2 participants