[common] Release the BTree index file on every path that abandons the writer - #9474
Open
PDGGK wants to merge 2 commits into
Open
[common] Release the BTree index file on every path that abandons the writer#9474PDGGK wants to merge 2 commits into
PDGGK wants to merge 2 commits into
Conversation
finish() writes the null bitmap, the bloom filter, the index block and the footer before closing the output stream, and all of that sits in one try whose catch only rethrows. A write that fails part way through - a full disk is the ordinary cause - skips out.close() and leaves the handle open. The constructor has the same gap in the other direction: it opens the output stream and then builds the serializer, comparator and SstFileWriter, so a failure there strands a file nothing else references. BTreeIndexReader guards its own construction the same way.
PDGGK
force-pushed
the
fix-btree-writer-leak
branch
from
September 1, 2026 12:03
28973e4 to
3e3e6fb
Compare
BTreeIndexWriter is the only GlobalIndexSingleColumnWriter in src/main that does not implement Closeable; the other seven do. Its constructor opens a PositionOutputStream unconditionally, and the only out.close() is on the success path of finish(). The owner cleanup paths - PkSortedIndexFile#build and SortedSingleColumnIndexWriter#close - reach a writer only through 'instanceof AutoCloseable', so for this one they evaluate false and skip it silently. A build abandoned before finish(), because the row source failed or a Flink or Spark index build task was cancelled, leaves that stream open with nothing able to reach it. Implement Closeable with an idempotent close(), so a build that never reaches finish() releases the file through the path its owners already use.
JingsongLi
reviewed
Sep 2, 2026
| this.comparator = keySerializer.createComparator(); | ||
| // todo: we may enable bf to accelerate equal and in predicate in the future | ||
| this.writer = new SstFileWriter(out, blockSize, null, compressionFactory); | ||
| } catch (RuntimeException e) { |
Contributor
There was a problem hiding this comment.
[P2] Close the stream when constructor setup throws an Error
This constructor-cleanup guard still misses Error. In particular, new SstFileWriter(...) allocates its data and index block buffers after out has been opened, so an OutOfMemoryError leaves the constructor without returning an AutoCloseable object and strands the output stream. A task/runtime may catch that failure while keeping the JVM alive, just like the abandoned-build paths this PR addresses. Please catch and rethrow RuntimeException | Error (or use the usual catch (Throwable) constructor-cleanup pattern) so every post-open setup failure closes out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
BTreeIndexWriteropens aPositionOutputStreamin its constructor and, before this change, gave it back on exactly one path: the success path offinish(). Three ways that leaves the file open.1. The writer is the only one its owners cannot release. It is the only
GlobalIndexSingleColumnWriterinsrc/mainthat does not implementCloseable:BitmapGlobalIndexWriterMultiValueBitmapIndexWriterFMGlobalIndexWriterNativeFullTextGlobalIndexWriterNativeVectorGlobalIndexWriterLuminaVectorGlobalIndexWriterESIndexGlobalIndexWriterBTreeIndexWriterBoth cleanup paths a btree writer can reach guard on that:
For a btree writer both evaluate false, so the writer is skipped without a word. A build abandoned before
finish()— the row source fails, or a Flink or SparkCREATE GLOBAL INDEX ... btreetask is cancelled — leaves the stream open with nothing left that can reach it: the class exposes neither aclose()nor any accessor forout. On thePkSortedIndexFilepath,deleteCreatedFiles()then unlinks the target while that stream is still open.2. A partially built writer keeps the file. The constructor opens the stream, then builds the comparator and the
SstFileWriter. If either throws, the stream is already open and the object is never handed back.3.
finish()keeps the file when it fails. It writes the null bitmap, the bloom filter, the index block and the footer before closing. Anything failing part way — a full disk is the ordinary cause — went tocatch (IOException e)and rethrew without closing.Tests
BTreeIndexWriterCloseTest, three cases against aPositionOutputStreamthat counts closes and fails past a fixed capacity:testTheOwnerCleanupPathReleasesAnAbandonedWriter— runs the exactinstanceof AutoCloseableidiom the owners use, against a writer that never reachesfinish().testCloseIsIdempotent—SortedGlobalIndexWriterholds the task writer in a try-with-resources, soclose()can follow a successfulfinish(); the stream must not be closed twice.testFinishReleasesTheFileWhenWritingFails— a write that fails mid-finish().On
masterthe first two fail and the third errors. Removing onlyCloseablefrom the class while keeping theclose()method — so it still compiles and still has a way to be closed, just not one the owners can see — fails exactly one test,testTheOwnerCleanupPathReleasesAnAbandonedWriterwithExpecting AtomicInteger(0) to have value: 1, which is what pins that the test discriminates the property being fixed rather than merely the presence of a method.Worth noting for review:
SortedGlobalIndexWriterTestalready asserts this cleanup, intestSingleColumnWriterClosesActiveWriterandtestBuildForSinglePartitionClosesWriterAfterFailure. Both build the subject withmock(GlobalIndexSingleColumnWriter.class, withSettings().extraInterfaces(Closeable.class))— they add to the mock the one property the real class lacks — and the second names indexTypebtreebut then overridescreateWriter()to return that mock, soBTreeIndexWriteris never constructed. Both are green today.mvn test -pl paimon-common -Dtest='org.apache.paimon.globalindex.**'— 349 tests, all passing.spotless:checkandcheckstyle:checkclean.Tests API and Compatibility
No API, format or configuration change.
close()is additive and idempotent, so a successfulfinish()behaves exactly as before.