Skip to content

AVRO-4253: [java] Bound FastReaderBuilder record-reader cache to fix memory leak - #3963

Open
iemejia wants to merge 1 commit into
apache:mainfrom
iemejia:AVRO-4253-bound-fastreader-cache
Open

AVRO-4253: [java] Bound FastReaderBuilder record-reader cache to fix memory leak#3963
iemejia wants to merge 1 commit into
apache:mainfrom
iemejia:AVRO-4253-bound-fastreader-cache

Conversation

@iemejia

@iemejia iemejia commented Aug 24, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

FastReaderBuilder's RecordReader cache was a weak-identity map keyed on the reader Schema:

private final Map<Schema, Map<Schema, RecordReader>> readerCache =
    Collections.synchronizedMap(new WeakIdentityHashMap<>());

However, each cached RecordReader holds a strong reference to that same reader Schema (RecordReader.schema, needed by its InstanceSupplier at read time — getNewRecordSupplier returns this::newRecord, which takes the schema as a parameter). Because the cached value strongly references its own weak key, the weak entries can never be reclaimed. The cache therefore grows without bound whenever many distinct Schema instances are used — e.g. when a schema is re-parsed for every file or message in a long-running process — leading to unbounded memory growth (reports of caches reaching 100GB).

This is the memory leak tracked by AVRO-4253 (and its duplicate AVRO-3524). Note the earlier commit referencing AVRO-4253 (#3764) actually addressed an unrelated logging concern; the leak itself was still present.

How was this patch fixed?

Replace the weak two-level map with a bounded LRU cache keyed on an identity-based (reader, writer) schema pair:

  • Bounds memory unconditionally, independent of the value-references-key cycle.
  • Identity-based key preserves the original cache semantics and avoids the cost of Schema.equals/hashCode on large schemas.
  • Recursion-safe: entries that are still INITIALIZING are never evicted, so recursive schema resolution still terminates by resolving back to the same in-flight instance instead of rebuilding endlessly.
  • Evicting a cached reader is safe for in-use readers: the caller holds the returned DatumReader directly, so eviction only means a future lookup rebuilds it.
  • The bound defaults to 2048 and is configurable via the org.apache.avro.fastreader.recordReaderCacheSize system property.

How was this patch tested?

New TestFastReaderBuilderCacheBounded:

  • cacheStaysBoundedAcrossDistinctSchemaInstances — 3048 freshly-parsed (distinct-identity) schemas leave the cache bounded at <= 2048 (fails on the old unbounded code).
  • reusedSchemaInstanceHitsCache — a reused schema instance keeps a single cache entry (cache still effective).
  • recursiveSchemaReadsCorrectly — a self-referential (linked-list) schema round-trips correctly.

Also verified no regressions across the read path: TestResolvingIO (816), TestResolvingIOResolving (192), TestGenericDatumReader, TestGenericData, TestDataFile, TestResolver, and the FastReaderBuilder tests all pass under the default, custom-coders, and without-fast-reader surefire profiles.

…memory leak

The RecordReader cache was a weak-identity map keyed on the reader Schema,
but each cached RecordReader holds a strong reference to that same Schema
(needed by its InstanceSupplier at read time). Because the cached value
strongly referenced its own weak key, the weak entries could never be
reclaimed, so the cache grew without bound whenever many distinct Schema
instances were used (e.g. a schema re-parsed for every file or message),
leading to unbounded memory growth.

Replace the weak two-level map with a bounded LRU cache keyed on an
identity-based (reader, writer) schema pair. Entries that are still being
initialized are never evicted, so recursive schema resolution still
terminates by resolving back to the same in-flight instance. The bound
defaults to 2048 and is configurable via the
org.apache.avro.fastreader.recordReaderCacheSize system property.
@github-actions github-actions Bot added the Java Pull Requests for Java binding label Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Java Pull Requests for Java binding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant