Skip to content

[fix][sec] Upgrade avro to 1.12.2 - #24992

Merged
merlimat merged 25 commits into
apache:masterfrom
lhotari:lh-avro-1.12.1
Aug 26, 2026
Merged

[fix][sec] Upgrade avro to 1.12.2#24992
merlimat merged 25 commits into
apache:masterfrom
lhotari:lh-avro-1.12.1

Conversation

@lhotari

@lhotari lhotari commented Nov 17, 2025

Copy link
Copy Markdown
Member

Fixes #25783

Motivation

Avro 1.12.1 contains 4 security fixes: https://avro.apache.org/blog/2025/10/16/avro-1.12.1/

The upgrade was blocked on AVRO-4209, a StackOverflowError in ReflectData.getSchema for a POJO with a
recursive field, introduced by AVRO-3940 in 1.12.1. That is fixed in 1.12.2, so this goes from 1.12.0
straight to 1.12.2 rather than stopping at 1.12.1.

Going to 1.12.2 pulls in three behaviour changes that Pulsar has to handle.

1. Avro 1.12.2 refuses to reflect over classes that are not explicitly trusted (AVRO-4189).

It is worth separating the two releases, since the change that causes this landed in 1.12.2 rather than in
1.12.1. In 1.12.0 and 1.12.1 the trusted-class allow-list was consulted in exactly one place,
SpecificDatumReader.findStringClass, so it only guarded classes named by the java-class /
java-key-class schema properties. AVRO-4189 introduced ClassSecurityValidator and moved the check into
ClassUtils.forName, so it now guards every reflective class resolution, and the default trusted
packages are gone: what is left is 16 hardcoded JDK classes and no packages at all.

That matters because SpecificData.getClass(Schema) resolves a class for every RECORD, ENUM and FIXED
schema, and ReflectData.populateEncoderCache calls it from getCustomEncoding on both
ReflectDatumWriter.write and ReflectDatumReader.read (those getCustomEncoding call sites arrived
separately, in 1.12.1, via AVRO-4165). Pulsar's AvroWriter and AvroReader are built on
ReflectDatum*, so without an allow-list every Schema.AVRO(...) encode and decode throws
SecurityException — including the schemas Pulsar uses internally for its own system topics.

Scope, since it is narrower than it first looks: Schema.JSON is not affected — it derives an Avro
schema for the SchemaInfo but reads and writes through Jackson, so it never resolves a class
reflectively. Schema.PROTOBUF is affected, but at schema-construction time rather than on produce.

2. The fast reader is enabled by default (AVRO-3230, 1.12.1).

GenericData.fastReaderEnabled now defaults to true. The fast path reports malformed data as
AvroTypeException where the classic path raised IndexOutOfBoundsException, so decoding errors escaped
GenericAvroReader.read instead of being wrapped in SchemaSerializationException.

3. avro-protobuf emits a default for enum types (AVRO-4133, 1.12.1).

The schema JSON generated from a protobuf message with an enum field gains a "default" attribute.

Modifications

  • Upgrade Avro from 1.12.0 to 1.12.2 in the version catalog, and update the jar names in the
    distribution/server and distribution/shell LICENSE.bin.txt files.

  • Add org.apache.pulsar.client.schema.AvroTrustedClasses, and declare the application's classes
    automatically
    . Passing a class to Schema.AVRO(...) is the application naming that class, so Pulsar
    trusts it and every type the derived schema references — nested records and enums, including ones in
    other packages, and the declared collection and @Stringable types the fields carry. Most applications
    therefore need no code change at all. Pulsar Functions and connectors are covered the same way, since
    they build their schemas through the same call, and so is the broker for the types it writes to its own
    system topics — no separate declaration is needed for either. Schema.PROTOBUF declares its message
    class too, before deriving the schema, since deriving is itself a reflective resolution.

    The hook is in AvroSchema.of, and only when the SchemaDefinition carries a POJO and no
    jsonDef. That condition is the trust boundary rather than a convenience: SchemaUtil.createAvroSchema
    gives jsonDef precedence, and AutoConsumeSchema feeds it raw schema-registry bytes, so expanding
    from a schema Pulsar did not derive would let whoever supplied that document choose which class names
    become trusted. Expansion works from the SchemaInfo just derived rather than by re-deriving from the
    class, since a differently configured ReflectData does not name the same types, and is cached by the
    schema that was walked.

    For what auto-registration deliberately does not cover, AvroTrustedClasses is the supported entry
    point: trust(Class...) (expands through the schema), trustExactly(Class...) (only what you named),
    trustClasses(String...) for classes you cannot reference at compile time, trustPackages(String...)
    and trust(Predicate<Class<?>>). Its signatures use only Class, String and
    java.util.function.Predicate — deliberately no Avro types, because Avro's own API is
    unusable from the shaded clients (see below).

    Declarations are process-wide and accumulate, so code making a temporary change takes a snapshot()
    and hands it back to restore(State) afterwards; resetToDefaults() returns to the built-in
    baseline. The declarations live in a nested AvroTrustedClasses.State that the facade swaps
    atomically, and both snapshot() and restore(...) copy, so a snapshot never shares mutable state
    with what is live and can be restored more than once. It installs by composing with the validator already in
    place, so Avro's defaults, the SERIALIZABLE_* system properties and any policy the application
    installed itself all keep working. It also seeds the protobuf runtime types and the collection types
    Avro records as java-class properties, so those never have to be rediscovered by a caller.

    ClassSecurityValidator only exists from Avro 1.12.2, and an application using
    pulsar-client-original can pin an older Avro, so every reference to it sits behind a lazily-loaded
    holder: on such an Avro there is nothing to enforce and declaring trust is a no-op, rather than a
    NoClassDefFoundError out of Schema.AVRO(...).

  • Catch AvroRuntimeException in both GenericAvroReader.read overloads so fast-reader decoding failures
    are still reported as SchemaSerializationException. The InputStream overload is the one consumers
    reach, via MessageImpl.decodeBySchemaAbstractStructSchema.decodeAbstractMultiVersionReader.

  • Update ProtobufSchemaTest for the enum default attribute.

  • The test JVMs are granted no Avro trust. Declaring classes where the application hands one over
    covers the fixture POJOs the suites serialize just as it covers an application's own, so granting the
    whole Pulsar namespace would only give tests a safety net production does not have — a path that failed
    to declare something would pass in CI and fail for users. Reverting a change is the test's own
    responsibility rather than shared test infrastructure's: a test that needs a known baseline takes a
    snapshot(), pins the global validator, and puts both back afterwards, so declarations belonging to
    a broker shared across test classes survive.

Upgrade note (for the release notes)

Avro 1.12.2 no longer reflects over arbitrary classes, but for most applications there is nothing to
do: passing a class to Schema.AVRO(...) declares it, along with everything the derived schema
references. Pulsar Functions and connectors are covered the same way.

Where a class is named only by a schema that arrived from the registry, Pulsar deliberately does not
declare it on the application's behalf — that is the case the allow-list exists to constrain. The main
one is Schema.AUTO_CONSUME() against a topic whose schema is a bare enum, which resolves the enum class
named in the writer's schema. An application that hits it declares the class itself, once, before the
first producer or consumer:

import org.apache.pulsar.client.schema.AvroTrustedClasses;

AvroTrustedClasses.trust(Colour.class);

If something is rejected that should not be, the exception names the class, and trust(...) on it is the
fix.

Schema.JSON is unaffected. Schema.PROTOBUF is affected at schema-construction time.

A warning about Avro's own error message, because it is the first thing users will read. It says to
set org.apache.avro.SERIALIZABLE_CLASSES / SERIALIZABLE_PACKAGES "or set them via the API (see
org.apache.avro.util.ClassSecurityValidator)". For anyone on the shaded client — pulsar-client or
pulsar-client-all, the usual dependency — both are wrong: shading relocates the class, so the snippet
does not compile (and adding org.apache.avro:avro to make it compile is worse — it compiles, runs,
configures a second copy, and the exception fires unchanged), and the property name is relocated with it,
so the -D is silently inert. Following the exception produces a byte-identical failure.

The system properties do work with the right name for the artifact:

# pulsar-client / pulsar-client-all (shaded - the usual dependency)
-Dorg.apache.pulsar.shade.org.apache.avro.SERIALIZABLE_PACKAGES=com.example.model

# pulsar-client-original (unshaded), broker, proxy and other components
-Dorg.apache.avro.SERIALIZABLE_PACKAGES=com.example.model

They are read in a static initializer, so they must be set before anything in the JVM has touched Avro — a
command-line -D always works, System.setProperty() only if it runs first. Note that a package alone
will not cover the java.util.List above; that is the bookkeeping trust(Class...) does for you.

Setting SERIALIZABLE_PACKAGES=* (same two spellings) restores the pre-1.12.2 behaviour, if the upgrade
breaks an application and you need it running while working out which classes to trust. Whether to keep it
that way is a policy decision for each deployment to make on its own assessment.

Two things that are easy to miss: nested classes must be named with the binary $ form
(com.example.Outer$Inner), and Schema.AUTO_CONSUME() against a topic whose schema is
Schema.JSON(SomeEnum.class) needs that enum trusted — an enum is the one POJO shape that produces a
non-RECORD top-level schema, and the consumer resolves the class named in the writer's schema.

Verifying this change

This change added tests and can be verified as follows:

  • AvroTrustedClassesTest pins the global validator to Avro's hardcoded DEFAULT_TRUSTED_CLASSES, so
    what is trusted is only ever what the test declares. It covers: Schema.AVRO(Order.class) round-tripping
    with no declaration at all, where Order reaches an enum, a record in another package, a List and a
    URI; that a definition built with withJsonDef — how AutoConsumeSchema passes a registry-fetched
    schema — does not expand trust; that a schema is walked only once; trustExactly not following
    references; interfaces, including one with no derivable schema; expansion through an @Union of
    implementations; a nested enum registering under its binary $ name; the predicate form; the seeded
    collection types; snapshot()/restore(...) round-tripping and resetToDefaults() clearing both the
    declarations and the walk cache; and that declaring more trust does not compose into the global
    validator again.
  • AvroTrustedClassesWithoutValidatorTest loads the class in a class loader that hides
    ClassSecurityValidator, so the older-Avro path is exercised from a JVM whose own Avro is current.
  • PulsarInternalAvroTypesTrustTest round-trips MetadataEvent, PulsarEvent with a fully populated
    TopicPolicies, and both transaction buffer snapshot formats under that same pinned validator — the
    guard that the broker's own system-topic types stay serializable with nothing declared in advance. It
    also pins that Schema.JSON over the broker's load-balancer types needs no declaration, and that trust
    does not leak to unrelated application classes.
  • Personal CI on this branch is green (40/40 jobs), including CI - System - Pulsar Connectors - Thread
    and - Process, which run a function using Schema.AVRO over a function-owned POJO.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • Anything that affects deployment

Adds org.apache.pulsar.client.schema.AvroTrustedClasses. Client applications using Schema.AVRO with
their own POJOs must declare those classes after this upgrade — see the upgrade note above. Pulsar's own
types, and Pulsar Functions and connectors, are handled automatically.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari added this to the 4.2.0 milestone Nov 17, 2025
@lhotari lhotari self-assigned this Nov 17, 2025
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Nov 17, 2025
@lhotari

lhotari commented Nov 20, 2025

Copy link
Copy Markdown
Member Author

The change apache/avro#3304 causes the StackOverflowError issue.

  Caused by: java.lang.StackOverflowError
  	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:492)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:673)
  	at org.apache.avro.reflect.ReflectData.createNonStringMapSchema(ReflectData.java:548)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:642)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:649)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createNonStringMapSchema(ReflectData.java:549)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:642)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:649)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createNonStringMapSchema(ReflectData.java:549)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:642)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:649)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createNonStringMapSchema(ReflectData.java:549)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:642)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:649)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createNonStringMapSchema(ReflectData.java:549)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:642)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:649)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)
  	at org.apache.avro.reflect.ReflectData.createNonStringMapSchema(ReflectData.java:549)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:642)
  	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:894)
  	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:98)
  	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:744)

@lhotari

lhotari commented Nov 20, 2025

Copy link
Copy Markdown
Member Author

Issue reported to Avro project: https://issues.apache.org/jira/browse/AVRO-4209

@codecov-commenter

codecov-commenter commented Nov 21, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.63%. Comparing base (212ee6a) to head (e1ac129).
⚠️ Report is 6 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (212ee6a) and HEAD (e1ac129). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (212ee6a) HEAD (e1ac129)
unittests 2 1
Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #24992       +/-   ##
=============================================
- Coverage     74.28%   38.63%   -35.66%     
+ Complexity    34065    13325    -20740     
=============================================
  Files          1920     1863       -57     
  Lines        150302   146139     -4163     
  Branches      17450    16964      -486     
=============================================
- Hits         111656    56458    -55198     
- Misses        29740    82057    +52317     
+ Partials       8906     7624     -1282     
Flag Coverage Δ
inttests 26.41% <ø> (-0.05%) ⬇️
systests 22.95% <ø> (+0.04%) ⬆️
unittests 34.79% <ø> (-39.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1417 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Avro 1.12.2 (AVRO-4189) moved the trusted-class check into ClassUtils.forName,
so every reflective class resolution is now denied unless allow-listed. In
1.12.0 and 1.12.1 the check only guarded classes named by the java-class /
java-key-class schema properties. Since SpecificData.getClass resolves a class
for every RECORD/ENUM/FIXED schema, and ReflectData.populateEncoderCache calls
it on both ReflectDatumWriter.write and ReflectDatumReader.read, Pulsar's own
system-topic schemas no longer serialize.

Add PulsarAvroClassSecurity, which trusts the types Pulsar itself serializes
with Avro and installs itself by composing with the validator already in place,
so an application's own policy and Avro's system properties are preserved.
PulsarService.start() installs it. Application POJOs are deliberately not
covered; they stay the application's own to trust.

Also catch AvroRuntimeException in GenericAvroReader.read: Avro 1.12.1 enabled
the fast reader by default (AVRO-3230), which reports malformed data as
AvroTypeException where the classic path raised IndexOutOfBoundsException, so
decoding errors escaped instead of becoming SchemaSerializationException.

For the Kubernetes function runtime, pass SERIALIZABLE_PACKAGES=* to the
function JVM, where the effect is confined to a single pod. The process and
thread runtimes share a JVM and are left for the operator to configure.

Assisted-by: Claude Code (Opus 5)
# Conflicts:
#	build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts
…s arg assertions

GenericAvroReader has two read overloads and only the byte[] one was widened to
catch AvroRuntimeException. The InputStream overload is the one consumers
actually reach, through MessageImpl.decodeBySchema -> AbstractStructSchema.decode
-> AbstractMultiVersionReader, so a fast-reader AvroTypeException still escaped
there instead of becoming SchemaSerializationException.

KubernetesRuntimeTest asserts the java command line by argument count and
position, so the two new -D arguments shift its expectations.

Assisted-by: Claude Code (Opus 5)
Functions that use Schema.AVRO/JSON over their own POJOs were rejected by Avro
1.12.2, since those classes belong to the deployment rather than to Pulsar and so
are not in the allow-list. Trusting every package for the Kubernetes runtime
covered only that runtime, and did it with a wildcard.

Add PulsarAvroClassSecurity.trustClassLoader, which trusts the classes a given
class loader defined, and register the function class loader in ThreadRuntime.
That covers all three runtimes: the process and Kubernetes runtimes reach the
same code through JavaInstanceStarter inside the function's own JVM. It is also
narrower than the wildcard it replaces, trusting only the classes that were
actually deployed rather than everything on the JVM's class path. The trust is
revoked when the instance stops, and the registry holds loaders weakly so a
stopped function's classes can still be collected.

Assisted-by: Claude Code (Opus 5)
Schema.JSON derives an Avro schema from the POJO but reads and writes it with
Jackson, so unlike Schema.AVRO it never resolves classes reflectively and its
types do not need trusting. Schema generation writes the java-class property
without resolving it: every ReflectData.getClassProp call site is on the
read/write path, not in createSchema.

Cover the broker's own JSON-schema types from the extensible load balancer, so
a future change to the JSON read/write path cannot start requiring trust
without a test noticing.

Assisted-by: Claude Code (Opus 5)
@lhotari
lhotari marked this pull request as draft August 20, 2026 11:23
lhotari added 14 commits August 20, 2026 15:19
Avro's own trusted-class API is unreachable from the shaded clients: shading
relocates org.apache.avro.util.ClassSecurityValidator, so an application on
pulsar-client or pulsar-client-all cannot name it, and adding its own Avro
dependency only configures a second copy that the shaded client never consults.
The Avro system property is renamed by the same relocation, so the advice in
Avro's own SecurityException message does not apply there either. That left
applications with no portable way to declare their POJOs.

Add org.apache.pulsar.client.schema.AvroTrustedClasses, which owns the trust
and installs it into Avro's global validator. Its signatures avoid Avro types
entirely, so it works the same shaded or not. trust(Class...) is the primary
entry point: it walks the schema Avro generates and covers the nested records
and enums, the ones in other packages, and the declared collection and
@Stringable types a package-based rule would miss.

Split the callers out of the client, since neither is a client concern:
BrokerAvroTrustedClasses declares the broker's own system-topic types, and
FunctionAvroTrustedClasses trusts a deployed function's class loader. This
replaces PulsarAvroClassSecurity, which held both and lived in the client
module while its allow-list was entirely server-side.

Assisted-by: Claude Code (Opus 5)
…nces

trust(Class...) expands through the schema Avro derives, which is what makes it
usable but also means it trusts more than the caller named. Add trustExactly for
when the precise set is the point, mirroring trustClasses(String...) for callers
who have the Class in hand.

Interfaces work through either. Avro derives an empty record for a user
interface, so trust() has nothing to expand but still registers it, which is
what a field declaring that interface needs. Some interfaces have no derivable
schema at all - a bare java.util.List fails with "Can't find element type of
Collection" - and there the schema walk is skipped and the class itself is still
trusted. Trusting an interface does not trust its implementations: Avro names
the concrete class in the schema, so those are reached by trusting a holder
whose field declares them with @union.

Assisted-by: Claude Code (Opus 5)
…from them

Passing a class to Schema.AVRO(...) is the application naming that class, so
Pulsar can trust it, and the types the derived schema references, without the
application declaring anything. That covers the overwhelming majority of cases,
including Pulsar Functions and connectors, which build their schemas through
the same call.

The hook sits in AvroSchema.of, guarded on the SchemaDefinition carrying a POJO,
rather than at the shared SchemaUtil choke point. That guard is the trust
boundary: createAvroSchema also has a withJsonDef arm, and AutoConsumeSchema
feeds it raw schema-registry bytes, so a hook there would let whoever registered
a schema add the class names in it to the allow-list - which is what the
allow-list exists to prevent.

Trust is expanded from the SchemaInfo that was just derived rather than by
re-deriving from the class, since ReflectData configured differently does not
name the same types. Traversal is cached per class and JSR-310 setting; the
setting has to be part of the key because a registered conversion replaces a
named type with a logical primitive, so one setting yields fewer names than the
other and keying on the class alone could under-trust.

With this, the class-loader trust for Pulsar Functions is redundant and grants
more than the application asked for - a NAR's whole contents rather than the
classes it actually uses with a schema - so it is removed.

Assisted-by: Claude Code (Opus 5)
ClassSecurityValidator arrived in Avro 1.12.2, and an application using
pulsar-client-original can pin an older Avro. Since AvroSchema.of now declares
the application's class on every schema construction, a missing validator would
have surfaced as NoClassDefFoundError out of Schema.AVRO(...) itself.

Move every reference to Avro's validator types into a nested holder, so they
resolve only when that class is first used, and gate its use on a one-time
Class.forName check. On an Avro without the validator there is nothing to
enforce, so declaring trust is simply a no-op.

The test loads AvroTrustedClasses in a class loader that hides the validator,
which is the only way to exercise the absent case from a JVM whose own Avro is
current.

Assisted-by: Claude Code (Opus 5)
Avro records a field's declared collection type as a "java-class" property and
resolves it reflectively. Trusting a class expands through the derived schema
and picks those up, but declaring a package or a bare class name does not, which
made "trust my model package" quietly insufficient for any POJO with a List
field. They are plain containers with nothing exploitable in construction, and
Avro's own build trusts the same set, so seed them rather than making every
caller rediscover them. That also lets the test JVMs stop listing them.

With that, BrokerAvroTrustedClasses had nothing left to say: every type the
broker writes to a system topic is serialized through Schema.AVRO(SomeClass),
so building the schema already trusts it and everything it references -
including CommandSubscribe$SubType, reached through TopicPolicies. Remove it and
the PulsarService call; the broker is just an application of the client here.

PulsarInternalAvroTypesTrustTest keeps the property honest: it round-trips the
transaction buffer snapshots, topic policy events and metadata events with the
global validator dropped to Avro's hardcoded defaults, so a broker type that
ever stops going through Schema.AVRO(Class) fails there.

Assisted-by: Claude Code (Opus 5)
…ypes in tests

Removing SERIALIZABLE_CLASSES from the test JVMs surfaced a real gap: Schema.PROTOBUF
does not go through AvroSchema.of, so nothing installed the validator and even the
seeded com.google.protobuf never took effect. A client using only Schema.PROTOBUF
would have failed on com.google.protobuf.Any.

Declare the message class in ProtobufSchema.of, and do it *before* deriving the
schema rather than after: createProtobufAvroSchema is itself a reflective
resolution, so a call placed after it is too late. ProtobufNativeSchema needs
nothing - it serializes a descriptor and never reaches Avro reflection.

The @Stringable types the test JVMs used to list (URI, URL, File) are recorded as
java-class properties on the derived schema, so trusting a class already picks them
up. Nothing needs them declared separately any more.

Assisted-by: Claude Code (Opus 5)
The build trusted the whole org.apache.pulsar namespace for every test JVM, which
gave tests a safety net production does not have: a path that failed to declare a
class would pass in CI and fail for users. Removing SERIALIZABLE_CLASSES already
exposed one such gap in Schema.PROTOBUF; this removes the rest of the net.

Nothing needs it now. Building a schema from a class declares that class and
everything the derived schema references, which covers the fixture POJOs the test
suites serialize just as it covers an application's own.

The two tests that pin the validator to Avro's hardcoded defaults keep doing so,
now to isolate themselves from trust another test declared rather than to defeat
the build's own grant.

Assisted-by: Claude Code (Opus 5)
trustApplicationSchema(pojo, null) read as "trust this schema" while passing no
schema. Protobuf wants the class trusted and nothing expanded, which is what
trustExactly already means.

There is genuinely no derived schema to expand from at that point: the call has
to precede deriving it, because deriving is itself a reflective resolution. It
does not need one either - encoding and decoding go through protobuf's own
reader and writer, so the message's nested types are never resolved by Avro.

Assisted-by: Claude Code (Opus 5)
Blank lines left behind when the class-loader trust was removed. The file now
matches master, so it drops out of the diff entirely.

Assisted-by: Claude Code (Opus 5)
Trust boundary: only expand from the derived schema when the definition actually
carries no jsonDef. SchemaUtil.createAvroSchema gives jsonDef precedence, and only
the stock builder forbids setting both, so a custom SchemaDefinition pairing a
local POJO with registry JSON could have fed wire-chosen names into the trusted
set. Check the same condition rather than assuming the invariant holds.

Resolve names against the POJO's own class loader rather than the thread context
loader. A plugin POJO built under the system TCCL would otherwise register the
dotted schema name while the reader resolves the '$' binary name through the
plugin loader, and be refused.

Key the traversal cache on the schema that was walked rather than the class name
and JSR-310 flag. Two loaders can define different versions of the same class
name, naming different types; keying on the name alone could cache the smaller
set and then refuse a type the other needs. The schema subsumes the flag.

Also correct the SingletonCleanerListener comment: it does not clear declarations
made through AvroTrustedClasses, deliberately, since a broker shared across test
classes still needs them. Say so, and say what it costs.

Assisted-by: Claude Code (Opus 5)
testTrustDoesNotLeakToUnrelatedApplicationClasses could never fail: the setup
pins the global validator to Avro's own defaults, and nothing in the method
installed Pulsar's predicate on top, so the assertion was answered by the pinned
baseline alone and would have passed however wide auto-registration got. Build a
schema first, and assert that class IS trusted, so the negative assertion has
something to disprove.

trustClassLoader's javadoc claimed Pulsar Functions and connectors use it. They
did until the class-loader grant was replaced by auto-registration; there is no
caller now. Say what it is actually for, and steer callers to the narrower option.

The seeded defaults - the protobuf runtime types and the collection types Avro
records as java-class properties - were inert in a JVM whose schemas are all
built from a schema document, because nothing installed the predicate there.
trustExactly now installs even when it names nothing.

Where a definition carries both a POJO and a jsonDef, trust the class the
application named rather than nothing at all, while still declining to expand
from a schema Pulsar did not derive.

Also correct the predicate javadoc: holding one instance keeps installs cheap but
does not make duplicate references impossible if something composes on top.

Assisted-by: Claude Code (Opus 5)
Avro's trusted-class declarations are process-wide, so a test that needs a
known baseline was reaching for a reset that also discarded whatever else had
been declared -- including the types a broker sharing the JVM still needs to
read and write its system topics. SingletonCleanerListener was carrying the
other half of that problem, restoring Avro's global validator between test
classes on the suite's behalf.

Reverting that listener and giving tests snapshot() and restore(Snapshot)
instead puts the responsibility where it belongs: a test saves what was
declared, changes it, and puts it back. The state now lives in a nested
Snapshot instance holding the trusted sets and the trust check, with
AvroTrustedClasses as a static facade over the current one, so restoring is an
atomic reference swap. Both snapshot() and restore() copy, so a snapshot never
shares mutable state with what is live and can be restored more than once.

resetForTesting() and forgetInstalledForTesting() are gone. Once a test
restores the global validator itself, the recorded install can never match the
current global, so the next install composes again on its own.
Nothing used trustClassLoader/untrustClassLoader. Functions and connectors
build their schemas through Schema.AVRO(...) like any other application, so
their classes are already declared by name when the schema is built, and the
only caller left was the test written for the feature. Trusting a loader is
also the bluntest thing this class could offer, since it covers every
third-party class packaged alongside the code that was meant to be trusted.

Removing it takes the weak class-loader set with it -- the one piece of state
whose copy() needed synchronization and care not to let a snapshot pin a
loader.
One instance is live and answers Avro's trust check; snapshot() hands out a
detached copy of it. Naming the type after the copy made the mutators read as
though they changed a frozen thing -- snapshot.markWalked(...) -- when they are
changing the declarations in force. State says what the type is, and snapshot()
and restore() say what you do with a copy of it.

restore(State snapshot) keeps snapshot as the parameter name: there the
argument really is a snapshot the caller took earlier.
@lhotari
lhotari marked this pull request as ready for review August 26, 2026 16:08
@merlimat
merlimat merged commit 91226e4 into apache:master Aug 26, 2026
82 of 84 checks passed
merlimat pushed a commit that referenced this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[sec] org.apache.avro:avro contains CVE-2025-33042

7 participants