Skip to content

[format] Keep Parquet codec classes out of Hadoop's ReflectionUtils cache - #10974

Merged
JingsongLi merged 1 commit into
apache:masterfrom
PDGGK:parquet-codec-skip-hadoop-constructor-cache
Nov 4, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
PDGGK:parquet-codec-skip-hadoop-constructor-cache

Conversation

@PDGGK

@PDGGK PDGGK commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fixes #9795

Parquet's CodecFactory.getCodec creates codecs with Hadoop's ReflectionUtils.newInstance. That method stores the constructor of every class it creates in the static CONSTRUCTOR_CACHE, which has strong class keys and is never cleared. In a Flink session cluster, Hadoop usually comes from /opt/flink/lib while Paimon comes with the job jar. The cached keys are then Parquet's codec classes loaded by the job's class loader, for example ZstandardCodec for the default file.compression = zstd. As a result, every finished job that reads or writes Parquet keeps its class loader, and its Metaspace, reachable until the TaskManager restarts. The heap dump in #9795 shows this chain.

This is the change to the codec instantiation path asked for in the review of #9827. Upgrading Parquet does not help, because 1.16.0 still calls ReflectionUtils.newInstance.

The change adds ParquetCodecFactory, a CodecFactory subclass.

  • Its getCodec instantiates the codec directly: it calls the no-arg constructor, then Configurable.setConf. This is what ReflectionUtils.newInstance does for a codec.
  • Everything else is unchanged, including the shared CODEC_BY_NAME cache and its key.

Paimon creates a codec factory in two places, and both now use the new class:

  • the vendored ParquetWriter.Builder#build;
  • ParquetUtil#getParquetReadOptionsBuilder, which every Parquet reader goes through.

To check the fix, I reproduced the leak locally:

  • Hadoop client 3.3.6 is on the shared class path.
  • Each "job" runs in its own thread, with its own child class loader that holds the Paimon jars.
  • The job writes and reads one Parquet file through FileFormat.
  • The job's loader is then dropped and GC runs.

Results for 10 jobs with zstd (the default):

job class loaders reachable after GC Metaspace growth after GC ZstandardCodec keys in CONSTRUCTOR_CACHE
paimon-bundle 2.1.0 10 of 10 +130.4 MB 10
master (d85253b) 10 of 10 +130.5 MB 10
this PR 0 of 10 +3.2 MB 0

This PR does not cover snappy or lz4_raw. With those codecs, Parquet's heap compressor and decompressor also return SnappyCompressor / Lz4RawCompressor (and the matching decompressors) to Hadoop's static CodecPool. That pool is keyed by these job-loaded classes. With this change, those codecs no longer reach CONSTRUCTOR_CACHE, but the same reproduction still retains 3 of 3 job loaders through CodecPool. Removing that path changes how compressors are reused, so it needs its own decision and is not part of this PR.

Tests

ParquetCodecFactoryTest writes and reads a Parquet file through FileFormat with zstd, snappy and lz4_raw. After the write and again after the read, it checks that Hadoop's ReflectionUtils.CONSTRUCTOR_CACHE (read through reflection) does not contain the codec class.

  • Before each case, the test removes the class from that cache and empties CODEC_BY_NAME, so codecs created by earlier tests in the same JVM cannot hide a regression.
  • With both call sites reverted to new CodecFactory(...), all three cases fail at the first check, for example with {org.apache.parquet.hadoop.codec.ZstandardCodec=public org.apache.parquet.hadoop.codec.ZstandardCodec()} in the cache.
  • Reverting only the writer fails the check after the write. Reverting only the reader fails the check after the read.

Ran locally on JDK 17:

  • ParquetCodecFactoryTest;
  • every test class under org.apache.paimon.format.parquet in paimon-format: 325 run, 0 failures, 1 skipped by an existing @Disabled;
  • checkstyle and spotless for paimon-format.

Other modules and JDK versions are left to CI.

…ache

Parquet's CodecFactory creates codecs with Hadoop's
ReflectionUtils.newInstance, which caches each constructor in a static
map with strong class keys. When Hadoop is on a shared class loader (for
example /opt/flink/lib of a Flink session cluster) and Paimon comes with
the job jar, the cached keys are Parquet codec classes loaded by the job
class loader, so every finished job that reads or writes Parquet keeps
its class loader and Metaspace alive. With the default zstd compression
this is ZstandardCodec, as found in apache#9795.

Add ParquetCodecFactory, which instantiates the codec directly and keeps
the rest of CodecFactory unchanged, and use it for both the Parquet
writer and the read options.

Fixes apache#9795
@JingsongLi
JingsongLi merged commit 396ba05 into apache:master Nov 4, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants