AVRO-4253: [java] Bound FastReaderBuilder record-reader cache to fix memory leak - #3963
Open
iemejia wants to merge 1 commit into
Open
AVRO-4253: [java] Bound FastReaderBuilder record-reader cache to fix memory leak#3963iemejia wants to merge 1 commit into
iemejia wants to merge 1 commit into
Conversation
…memory leak The RecordReader cache was a weak-identity map keyed on the reader Schema, but each cached RecordReader holds a strong reference to that same Schema (needed by its InstanceSupplier at read time). Because the cached value strongly referenced its own weak key, the weak entries could never be reclaimed, so the cache grew without bound whenever many distinct Schema instances were used (e.g. a schema re-parsed for every file or message), leading to unbounded memory growth. Replace the weak two-level map with a bounded LRU cache keyed on an identity-based (reader, writer) schema pair. Entries that are still being initialized are never evicted, so recursive schema resolution still terminates by resolving back to the same in-flight instance. The bound defaults to 2048 and is configurable via the org.apache.avro.fastreader.recordReaderCacheSize system property.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
FastReaderBuilder'sRecordReadercache was a weak-identity map keyed on the readerSchema:However, each cached
RecordReaderholds a strong reference to that same readerSchema(RecordReader.schema, needed by itsInstanceSupplierat read time —getNewRecordSupplierreturnsthis::newRecord, which takes the schema as a parameter). Because the cached value strongly references its own weak key, the weak entries can never be reclaimed. The cache therefore grows without bound whenever many distinctSchemainstances are used — e.g. when a schema is re-parsed for every file or message in a long-running process — leading to unbounded memory growth (reports of caches reaching 100GB).This is the memory leak tracked by AVRO-4253 (and its duplicate AVRO-3524). Note the earlier commit referencing AVRO-4253 (#3764) actually addressed an unrelated logging concern; the leak itself was still present.
How was this patch fixed?
Replace the weak two-level map with a bounded LRU cache keyed on an identity-based
(reader, writer)schema pair:Schema.equals/hashCodeon large schemas.INITIALIZINGare never evicted, so recursive schema resolution still terminates by resolving back to the same in-flight instance instead of rebuilding endlessly.DatumReaderdirectly, so eviction only means a future lookup rebuilds it.2048and is configurable via theorg.apache.avro.fastreader.recordReaderCacheSizesystem property.How was this patch tested?
New
TestFastReaderBuilderCacheBounded:cacheStaysBoundedAcrossDistinctSchemaInstances— 3048 freshly-parsed (distinct-identity) schemas leave the cache bounded at<= 2048(fails on the old unbounded code).reusedSchemaInstanceHitsCache— a reused schema instance keeps a single cache entry (cache still effective).recursiveSchemaReadsCorrectly— a self-referential (linked-list) schema round-trips correctly.Also verified no regressions across the read path:
TestResolvingIO(816),TestResolvingIOResolving(192),TestGenericDatumReader,TestGenericData,TestDataFile,TestResolver, and the FastReaderBuilder tests all pass under the default, custom-coders, and without-fast-reader surefire profiles.