Skip to content

[Improvement] Avoid an unnecessary S3 PUT when opening a data file - #1465

Open
ZENOTME wants to merge 1 commit into
apache:mainfrom
ZENOTME:codex/skip-s3-directory-marker
Open

ZENOTME wants to merge 1 commit into
apache:mainfrom
ZENOTME:codex/skip-s3-directory-marker

Conversation

@ZENOTME

@ZENOTME ZENOTME commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Closes #1464. Avoid an extra S3 PUT when opening a data file.

Brief change log

Remove the explicit mkdirs() calls from the append and primary-key data writers. OpenDAL’s local filesystem writer already creates missing parent directories when it opens a file. The S3 writer can write the object directly, so removing mkdirs() avoids a separate PUT for the directory marker. Custom OpenDAL operators that require parent directories should create them in their own write path.

Tests

Add a regression test that writes append and primary-key Parquet files into fresh buckets through a custom FileIO provider backed by OpenDAL FS. It verifies that the files exist and that neither writer requests the bucket directory separately.

@ZENOTME ZENOTME changed the title fix(io): skip S3 directory markers when opening data files [Improvement] Avoid an unnecessary S3 PUT when opening a data file Nov 19, 2026
@ZENOTME
ZENOTME force-pushed the codex/skip-s3-directory-marker branch 8 times, most recently from 9d237a0 to b795f49 Compare November 20, 2026 02:03
@ZENOTME
ZENOTME force-pushed the codex/skip-s3-directory-marker branch from b795f49 to 2ea74d4 Compare November 20, 2026 02:17
.new_file_name(&self.data_file_prefix, &self.file_format);
let location = self.paths.new_path(&file_name)?;
let bucket_dir = location.parent();
self.file_io.mkdirs(&format!("{bucket_dir}/")).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we skip mkdirs() for S3 in FileIO and keep it for backends that need it? I tested a FileIOProvider backed by OpenDAL WebDAV with automatic parent creation disabled. Both append and primary-key writes to a new bucket-0 failed with HTTP 409 during prepare_commit(). The same writes passed after explicitly creating the bucket directory.

Could we also add a regression test that checks both S3 write paths produce a Parquet file without sending a directory marker PUT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that each operator handles parent directories as needed. E.g. FsWriter::create() calls ensure_write_abs_path(), which creates missing parent directories. The WebDAV writer does the same by default in WebdavWriter::write_once().

Is there a use case for disabling automatic directory creation when the WebDAV server requires the parent to exist, or is the concern mainly backward compatibility with providers that rely on Paimon to create it?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly backward compatibility. I don’t have a concrete use case for that WebDAV setup. I used it to reproduce what happens when a custom provider relies on Paimon to create the bucket directory.

I’m fine with leaving parent directory creation to the operator and documenting that in FileIOProvider. @JingsongLi does that match what you’d expect from FileIOProvider?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Avoid an unnecessary S3 PUT when opening a data file

2 participants