Skip to content

Close the reader pool before deleting files on rollback - #16643

Open
manduinca wants to merge 1 commit into
apache:mainfrom
manduinca:fix/rollback-reader-pool
Open

manduinca wants to merge 1 commit into
apache:mainfrom
manduinca:fix/rollback-reader-pool

Conversation

@manduinca

@manduinca manduinca commented Sep 7, 2026

Copy link
Copy Markdown

Description

rollbackInternalNoCommit asks the deleter to remove unreferenced files while the pooled readers are still open, so an uncommitted segment gets deleted from under them. On a directory that checks for this the rollback fails outright:

java.io.IOException: MockDirectoryWrapper: file "_2.cfs" is still open: cannot delete
    at org.apache.lucene.util.FileDeleter.delete(FileDeleter.java:234)
    at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:620)
    at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:578)
    at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2628)

Moving readerPool.close() ahead of deleter.checkpoint(...) is the fix, and as @LuXugang notes in the issue it isn't enough on its own: the checkpoint's own diagnostic logging goes back to the pool through segStringnumDeletedDocsgetPooledInstance(info, false), which throws AlreadyClosedException once the pool is closed.

That second part turns out to be a lookup answering with an exception where it could answer with the truth. ReaderPool.get(info, create) throws for a closed pool even when create == false, but a closed pool holds no readers (its own assert says the map is empty), so null is the accurate answer for a lookup. So get now returns null in that case and keeps throwing when asked to create.

On the callers: there are eight create == false lookups in IndexWriter. Five null-check the result (numDeletedDocs falls back to info.getDelCount(...); the others skip the segment). The remaining three sit on the merge path (commitMergedDeletesAndUpdates, buildMappedDVUpdatesFromDisk, closeMergeReaders), hold their own ref on the reader, and two of them already assert rld != null on that basis. Reaching them with a closed pool was an invalid state before this change too — abortMerges() runs before the pool is closed on rollback — so the difference there is only which exception reports it.

Tests

TestIndexWriterRollbackReaderPool is the reproducer from the issue. It fails on main with the stack trace above and passes with this change.

The full lucene/core suite is green with the change: 8689 tests, 319 skipped, 0 failures — which is where the AlreadyClosedException mentioned in the issue would show up.

Closes #16625

The deleter removed uncommitted segment files while pooled readers still
held them open. Close the pool first, and let ReaderPool.get return null
for a lookup on a closed pool instead of throwing, so the checkpoint's
diagnostic logging keeps working.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IndexWriter rollback deletes uncommitted segment files while pooled readers are still open

1 participant