Repository navigation
Conversation
9d237a0 to
b795f49
Compare
b795f49 to
2ea74d4
Compare
| .new_file_name(&self.data_file_prefix, &self.file_format); | ||
| let location = self.paths.new_path(&file_name)?; | ||
| let bucket_dir = location.parent(); | ||
| self.file_io.mkdirs(&format!("{bucket_dir}/")).await?; |
There was a problem hiding this comment.
Could we skip mkdirs() for S3 in FileIO and keep it for backends that need it? I tested a FileIOProvider backed by OpenDAL WebDAV with automatic parent creation disabled. Both append and primary-key writes to a new bucket-0 failed with HTTP 409 during prepare_commit(). The same writes passed after explicitly creating the bucket directory.
Could we also add a regression test that checks both S3 write paths produce a Parquet file without sending a directory marker PUT?
There was a problem hiding this comment.
My understanding is that each operator handles parent directories as needed. E.g. FsWriter::create() calls ensure_write_abs_path(), which creates missing parent directories. The WebDAV writer does the same by default in WebdavWriter::write_once().
Is there a use case for disabling automatic directory creation when the WebDAV server requires the parent to exist, or is the concern mainly backward compatibility with providers that rely on Paimon to create it?
There was a problem hiding this comment.
Mainly backward compatibility. I don’t have a concrete use case for that WebDAV setup. I used it to reproduce what happens when a custom provider relies on Paimon to create the bucket directory.
I’m fine with leaving parent directory creation to the operator and documenting that in FileIOProvider. @JingsongLi does that match what you’d expect from FileIOProvider?
Purpose
Closes #1464. Avoid an extra S3 PUT when opening a data file.
Brief change log
Remove the explicit
mkdirs()calls from the append and primary-key data writers. OpenDAL’s local filesystem writer already creates missing parent directories when it opens a file. The S3 writer can write the object directly, so removingmkdirs()avoids a separatePUTfor the directory marker. Custom OpenDAL operators that require parent directories should create them in their own write path.Tests
Add a regression test that writes append and primary-key Parquet files into fresh buckets through a custom FileIO provider backed by OpenDAL FS. It verifies that the files exist and that neither writer requests the bucket directory separately.