Repository navigation
#158 remove the DOM from the ParseFilters - #2262
Merged
Merged
Conversation
ParseFilter.filter() no longer takes a DocumentFragment and needsDOM() is removed. The parser bolts do not build a W3C DOM any more: extracting data from the markup of a document is done with the JSoupFilters, which are now applied by both the JSoupParserBolt and the Tika ParserBolt. - remove the DOM-based XPathFilter and the DebugParseFilter - remove DocumentFragmentBuilder (core) and DOMBuilder / XMLCharacterRecognizer (Tika), as well as the now unused parser.uppercase.element.names option of the Tika ParserBolt - drop the xercesImpl dependency from core - update the ParseFilter implementations, tests and documentation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rzo1
approved these changes
Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2252, #2253 and #2256 for #158.
Building a W3C DOM is costly, and since the JSoupFilters are now applied by both the
JSoupParserBoltand the TikaParserBolt, nothing in StormCrawler needs it any more. This removes the DOM from the parse filtering altogether.Changes
ParseFilter.filter(String url, byte[] content, ParseResult parse)no longer takes aDocumentFragment;needsDOM()is removed.ParseFiltersno longer skips filters for lack of a DOM.parse.filter.XPathFilter(usejsoup.XPathFilter) andDebugParseFilter.JSoupParserBoltand the TikaParserBoltno longer build a DOM.DocumentFragmentBuilder(core) andDOMBuilder/XMLCharacterRecognizer(Tika) are removed, along with the Tika optionparser.uppercase.element.names, which only applied to the DOM.xercesImplis dropped from core.ParseFilterimplementations (core, langid, playwright, opensearchJSONResourceWrapper), the tests and the documentation.Breaking changes (for the 4.0 migration notes)
ParseFilters must drop theDocumentFragmentargument and anyneedsDOM()override. Filters which extracted data from the DOM should be rewritten asJSoupFilters.org.apache.stormcrawler.parse.filter.XPathFiltermust move toorg.apache.stormcrawler.jsoup.XPathFilterinjsoupfilters.json. The expression syntax differs:string(...)/serialize(...)become/tidyText(),/allText(),/html()or/@attr.parser.uppercase.element.namesis no longer used.Testing
Core parse / bolt / jsoup tests, all Tika tests,
JsRenderingDetectorTestandDelegateRefresherTestpass locally.🤖 Generated with Claude Code