Repository navigation
Tika ParserBolt: apply the JSoupFilters to the documents parsed by Tika - #2256
Conversation
A new JSoupBuilder turns the XHTML SAX events produced by Tika into a JSoup document, the same way DOMBuilder does for the W3C DOM given to the parse filters. It is teed with the link and text handlers, so the document is parsed once, and it also receives the content replayed from a Tika Pipes fork. The document is only built when JSoupFilters are configured (jsoup.filters.config.file, as for JSoupParserBolt); they run before the parse filters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tika only sends the ParseContext components listed in its META-INF/tika/*.idx files to the forked JVM, which used its DefaultHtmlMapper as a result and dropped elements such as span and most attributes from the documents given to the filters. Register IdentityHtmlMapper in a parse-context.idx shipped by this module and pass it with each parse when it is the configured mapper (the default). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rzo1
left a comment
rzo1
left a comment
There was a problem hiding this comment.
Thanks, LGTM. A few things inline, none of them blocking. Also, configuration.adoc still says the HtmlMapper is ignored under parser.tika.timeout.
| // Tika only sends the components registered in its index files to the forked JVM, | ||
| // which falls back to its default mapper otherwise | ||
| if (IdentityHtmlMapper.class.equals(htmlMapperClass)) { | ||
| pipesHtmlMapper = IdentityHtmlMapper.INSTANCE; |
There was a problem hiding this comment.
My local AI flagged this one, and I think it's valid:
This changes which HtmlMapper the forked JVM uses. Before, it fell back to Tika's
DefaultHtmlMapper, which drops most attributes. Now it gets theIdentityHtmlMapper, which keeps all of them, including ones with a prefix likexlink:href(used by inline SVG icons, e.g.<svg><use xlink:href="#icon"></use></svg>) orx-on:click(Alpine.js).The fork sends its result back as XML, written by
ToXMLContentHandler. That writer outputsxlink:href="#icon"as it is, but without declaring thexlinknamespace. On our side,reparseIntoHandlerreads that XML back with a namespace-aware SAX parser, which rejects an undeclared prefix with "prefix not bound". The page is then reported as a parse error and nothing is emitted, not even the text or the outlinks.To reproduce: set
parser.tika.timeout, leave everything else at the defaults (no jsoup filters needed), and parse<html><body><svg><use xlink:href="#i"></use></svg><a href="l.html">l</a></body></html>. On main this page parses fine and emits the outlink; with this PR it fails. In-process parsing (no timeout) is not affected.
Only Pipes mode is affected, but inline SVG is everywhere, so we should fix it before 4.0, here or in a follow-up. One option is to re-read the XML with a non-namespace-aware parser and pass the qName as the local name. A Pipes test with an xlink:href and an x-on:click attribute would cover it.
| ParseContext parseContext = new ParseContext(); | ||
| setPdfParserConfig(parseContext); | ||
| if (pipesHtmlMapper != null) { | ||
| parseContext.set(HtmlMapper.class, pipesHtmlMapper); |
There was a problem hiding this comment.
If parse-context.idx isn't merged into the shaded jar (e.g. a project not built from the archetype, whose shade config doesn't include it), every parse in Pipes mode fails, because the fork can't deserialise the mapper we pass here. prepare() doesn't notice, since startFork() runs its test parse without the mapper. Could we make this fail-safe? For example, check in prepare() whether IdentityHtmlMapper is registered (ComponentNameResolver.getFriendlyName(IdentityHtmlMapper.class) != null, worth testing it both with and without the idx file). If not, log a warning that points to the AppendingTransformer for META-INF/tika/parse-context.idx, and don't send the mapper, so the fork falls back to DefaultHtmlMapper as before.
|
|
||
| parseFilters = ParseFilters.fromConf(conf); | ||
|
|
||
| jsoupFilters = JSoupFilters.fromConf(conf); |
There was a problem hiding this comment.
With the archetype config, the jsoup filters now also run on PDFs and Office files. An untitled PDF ends up with an empty parse.title, because Tika always writes a <title> element and XPathFilter keeps the first match even if it's blank, so the META fallback is never tried. HTML pages with an empty <title> have the same problem today. I'd suggest skipping blank values in XPathFilter (both the jsoup and the DOM one), so the next expression gets tried and empty values never get stored. Probably better as a small separate PR. Not sure the shared config key is what we want here, but fine either way.
…under Tika Pipes - Re-read the XML returned by the fork without namespace processing and restore the namespaces with NamespaceRestoringHandler: the fork writes names kept by the IdentityHtmlMapper, such as xlink:href or x-on:click, without declaring their prefix, which failed the whole document. - Only send the IdentityHtmlMapper to the fork when it is registered with Tika, otherwise log a warning and let the fork use its DefaultHtmlMapper instead of failing every parse. - configuration.adoc and README: describe which HtmlMapper the fork uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@rzo1 thanks for your comments. They should be addressed in the latest commit. Would you mind having a look? |
Relates to #158: a first step towards parse filters that no longer deal with the W3C DOM. With this, JSoupFilters work on everything Tika parses, not only on what the
JSoupParserBolthandles.Changes
JSoupBuilder(new, inexternal/tika): a SAX handler which builds a JSoupDocumentfrom the XHTML events produced by Tika, likeDOMBuilderdoes for the W3C DOM. It is teed with the link and text handlers, so the document is still parsed once, and it also receives the content replayed from a Tika Pipes fork. Element names are lowercased, adjacent text is merged into a single node, script/style content is kept as data, and the document keeps what it received if the parse stops early.ParserBoltloads the JSoupFilters fromjsoup.filters.config.file, the same key asJSoupParserBolt, and runs them before the parse filters. The JSoup document is only built when at least one JSoupFilter is configured (newJSoupFilters.isEmpty()).IdentityHtmlMapper: Tika only sends theParseContextcomponents listed inMETA-INF/tika/*.idxto the forked JVM, so the fork used to fall back toDefaultHtmlMapper, which drops elements such asspanand most attributes. The module now registersIdentityHtmlMapperin its ownMETA-INF/tika/parse-context.idxand passes it with each parse when it is the configured mapper (the default). Shaded jars must merge that file, which the archetype already does.parser.tika.timeout.Things to be aware of
<div class="page">/<p>elements, and Tika rebuilds the<head>of an HTML page from its metadata (e.g.<meta name="dc:subject">instead of<meta name="keywords">). This was already the case for the DOM given to the parse filters.ParseFilter(and removingDOMBuilder,DocumentFragmentBuilderand xerces) is left for a separate PR.Tests
JSoupBuilderTest: structure, text merging, script data, base URI, truncated input.ParserBoltJSoupFiltersTest: core's XPath test page and a generated PDF, both in-process and through a Tika Pipes fork, plus no filters configured. The Pipes HTML test fails without theIdentityHtmlMapperchange.ParserBoltTestandParserBoltPipesTimeoutTestpass.🤖 Generated with Claude Code