Repository navigation
[format] Keep Parquet codec classes out of Hadoop's ReflectionUtils cache - #10974
Merged
JingsongLi merged 1 commit intoNov 4, 2026
Merged
Conversation
…ache Parquet's CodecFactory creates codecs with Hadoop's ReflectionUtils.newInstance, which caches each constructor in a static map with strong class keys. When Hadoop is on a shared class loader (for example /opt/flink/lib of a Flink session cluster) and Paimon comes with the job jar, the cached keys are Parquet codec classes loaded by the job class loader, so every finished job that reads or writes Parquet keeps its class loader and Metaspace alive. With the default zstd compression this is ZstandardCodec, as found in apache#9795. Add ParquetCodecFactory, which instantiates the codec directly and keeps the rest of CodecFactory unchanged, and use it for both the Parquet writer and the read options. Fixes apache#9795
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #9795
Parquet's
CodecFactory.getCodeccreates codecs with Hadoop'sReflectionUtils.newInstance. That method stores the constructor of every class it creates in the staticCONSTRUCTOR_CACHE, which has strong class keys and is never cleared. In a Flink session cluster, Hadoop usually comes from/opt/flink/libwhile Paimon comes with the job jar. The cached keys are then Parquet's codec classes loaded by the job's class loader, for exampleZstandardCodecfor the defaultfile.compression = zstd. As a result, every finished job that reads or writes Parquet keeps its class loader, and its Metaspace, reachable until the TaskManager restarts. The heap dump in #9795 shows this chain.This is the change to the codec instantiation path asked for in the review of #9827. Upgrading Parquet does not help, because 1.16.0 still calls
ReflectionUtils.newInstance.The change adds
ParquetCodecFactory, aCodecFactorysubclass.getCodecinstantiates the codec directly: it calls the no-arg constructor, thenConfigurable.setConf. This is whatReflectionUtils.newInstancedoes for a codec.CODEC_BY_NAMEcache and its key.Paimon creates a codec factory in two places, and both now use the new class:
ParquetWriter.Builder#build;ParquetUtil#getParquetReadOptionsBuilder, which every Parquet reader goes through.To check the fix, I reproduced the leak locally:
FileFormat.Results for 10 jobs with zstd (the default):
ZstandardCodeckeys inCONSTRUCTOR_CACHEThis PR does not cover
snappyorlz4_raw. With those codecs, Parquet's heap compressor and decompressor also returnSnappyCompressor/Lz4RawCompressor(and the matching decompressors) to Hadoop's staticCodecPool. That pool is keyed by these job-loaded classes. With this change, those codecs no longer reachCONSTRUCTOR_CACHE, but the same reproduction still retains 3 of 3 job loaders throughCodecPool. Removing that path changes how compressors are reused, so it needs its own decision and is not part of this PR.Tests
ParquetCodecFactoryTestwrites and reads a Parquet file throughFileFormatwithzstd,snappyandlz4_raw. After the write and again after the read, it checks that Hadoop'sReflectionUtils.CONSTRUCTOR_CACHE(read through reflection) does not contain the codec class.CODEC_BY_NAME, so codecs created by earlier tests in the same JVM cannot hide a regression.new CodecFactory(...), all three cases fail at the first check, for example with{org.apache.parquet.hadoop.codec.ZstandardCodec=public org.apache.parquet.hadoop.codec.ZstandardCodec()}in the cache.Ran locally on JDK 17:
ParquetCodecFactoryTest;org.apache.paimon.format.parquetinpaimon-format: 325 run, 0 failures, 1 skipped by an existing@Disabled;paimon-format.Other modules and JDK versions are left to CI.