[dpol1]

As discussed, this replaces the Xerces restore with docs. The archetypes now pass -Djdk.xml.maxGeneralEntitySizeLimit=0 -Djdk.xml.totalEntitySizeLimit=0 to the workers. The configuration docs also cover local mode (STORM_JAR_JVM_OPTS) and the Tika fork (parser.tika.pipes.jvmargs), and the 4.0 migration notes point there.

[rzo1]

Thanks for tracking this down. I'd rather not bring Xerces back, though: 2.12.2 is from 2022, Xerces hasn't had a release since, and putting it on the classpath swaps out the SAX parser for everything in the worker, not just sitemaps and feeds.

The JDK parser can handle this with two system properties. I tried it on JDK 25.0.3 with a sitemap of 34,000 URLs and three & each. With the defaults it fails as you describe. Raising only one of the limits isn't enough, because the second one kicks in at 100,000 too. With both it parses fine:

-Djdk.xml.maxGeneralEntitySizeLimit=0 -Djdk.xml.totalEntitySizeLimit=0

So my suggestion would be to document these for the worker JVMs (topology.worker.childopts) in the configuration docs and the archetypes' crawler-conf.yaml, with a note in the 4.0 migration notes for anyone running on Java 24+.

Longer term, crawler-commons could set the two limits on its own parser in SiteMapParser.processXml() when DOCTYPEs are disallowed (the default). Without a DTD there are no custom entities, so it's safe, and then nobody needs the worker options for sitemaps. ROME would need the same for feeds.

[dpol1]

heya, makes sense, with 4.0 being a major release it's a good moment to drop Xerces. I'll rework this PR into docs for the two properties. Mainly wanted to flag the regression before the release 🙂

[dpol1]

ping @rzo1

[dpol1]

I’ve expanded the wording to cover XML-based documents parsed by Tika, including OpenDocument files, and added a note for applications using Xerces classes directly.

Separately, the 4.0 migration guide is missing several breaking changes already documented elsewhere. I’ll prepare a separate docs PR to collect the upgrade steps and link to the existing module documentation.