Conversation
The deleter removed uncommitted segment files while pooled readers still held them open. Close the pool first, and let ReaderPool.get return null for a lookup on a closed pool instead of throwing, so the checkpoint's diagnostic logging keeps working.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
rollbackInternalNoCommitasks the deleter to remove unreferenced files while the pooled readers are still open, so an uncommitted segment gets deleted from under them. On a directory that checks for this the rollback fails outright:Moving
readerPool.close()ahead ofdeleter.checkpoint(...)is the fix, and as @LuXugang notes in the issue it isn't enough on its own: the checkpoint's own diagnostic logging goes back to the pool throughsegString→numDeletedDocs→getPooledInstance(info, false), which throwsAlreadyClosedExceptiononce the pool is closed.That second part turns out to be a lookup answering with an exception where it could answer with the truth.
ReaderPool.get(info, create)throws for a closed pool even whencreate == false, but a closed pool holds no readers (its own assert says the map is empty), sonullis the accurate answer for a lookup. Sogetnow returnsnullin that case and keeps throwing when asked to create.On the callers: there are eight
create == falselookups inIndexWriter. Five null-check the result (numDeletedDocsfalls back toinfo.getDelCount(...); the others skip the segment). The remaining three sit on the merge path (commitMergedDeletesAndUpdates,buildMappedDVUpdatesFromDisk,closeMergeReaders), hold their own ref on the reader, and two of them alreadyassert rld != nullon that basis. Reaching them with a closed pool was an invalid state before this change too —abortMerges()runs before the pool is closed on rollback — so the difference there is only which exception reports it.Tests
TestIndexWriterRollbackReaderPoolis the reproducer from the issue. It fails onmainwith the stack trace above and passes with this change.The full
lucene/coresuite is green with the change: 8689 tests, 319 skipped, 0 failures — which is where theAlreadyClosedExceptionmentioned in the issue would show up.Closes #16625