Tika ParserBolt: apply the JSoupFilters to the documents parsed by Tika

core/src/main/java/org/apache/stormcrawler/parse/JSoupFilters.java

@@ -111,6 +111,11 @@ public void configure(@NotNull Map<String, Object> stormConf, @NotNull JsonNode
         filters = list.toArray(new JSoupFilter[0]);
     }
 
+    /** Returns true if no filters are configured, so callers can skip building a document. */
+    public boolean isEmpty() {
+        return filters.length == 0;
+    }
+
     @Override
     public void filter(
             @NotNull String url,

docs/src/main/asciidoc/configuration.adoc

@@ -570,7 +570,7 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/tika[tika
 | parser.extract.embedded | false | Whether to parse embedded documents. Since Tika 4 embedded documents are no longer parsed unless this is set to `true`.
 | parser.tika.text.maxlength | -1 | Maximum number of characters of text extracted from a document, -1 (or any negative value) for no limit. When the limit is reached the parse stops, the text and outlinks extracted so far are kept and the metadata `parse.text.trimmed` is set to `true`.
 | parser.tika.pdf.maxpages | -1 | Maximum number of pages of a PDF document to parse, 0 or less to keep the `maxPages` of the Tika configuration (no limit by default). The pages after it are skipped, their text and outlinks are not extracted. Other settings of `pdf-parser` in the `parse-context` section of the Tika configuration are kept.
-| parser.tika.timeout | -1 | Maximum time in milliseconds a document may take to parse, 0 or less for no limit. When set, the parse runs in a forked JVM (Tika Pipes), one per bolt instance; a parse that exceeds this is killed outright and reported to the status stream as an `ERROR` with the message `parse timeout`. A partial result returned between embedded documents is kept and marked `parse.text.trimmed`. When `parser.tika.text.maxlength` is negative (the default), the text is limited to 20,000,000 characters in this mode. Keep it well below `topology.message.timeout.secs`: the time a tuple waits behind a slow parse or a fork restart counts against that too. `parser.htmlmapper.classname` is ignored in this mode, see the README of the module.
+| parser.tika.timeout | -1 | Maximum time in milliseconds a document may take to parse, 0 or less for no limit. When set, the parse runs in a forked JVM (Tika Pipes), one per bolt instance; a parse that exceeds this is killed outright and reported to the status stream as an `ERROR` with the message `parse timeout`. A partial result returned between embedded documents is kept and marked `parse.text.trimmed`. When `parser.tika.text.maxlength` is negative (the default), the text is limited to 20,000,000 characters in this mode. Keep it well below `topology.message.timeout.secs`: the time a tuple waits behind a slow parse or a fork restart counts against that too. The forked JVM uses the `IdentityHtmlMapper` (the default), any other value of `parser.htmlmapper.classname` is ignored in this mode and Tika's `DefaultHtmlMapper` is used instead, see the README of the module.
 | parser.tika.pipes.jvmargs | -Xmx512m | JVM arguments passed to each forked process under `parser.tika.timeout`. `-Xmx512m` is added when they set no maximum heap (`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxRAM*`).
 | parser.tika.pipes.maxfilesperprocess | - | Restart a forked process after this many documents under `parser.tika.timeout`, to bound slow leaks in parsing libraries. Unset uses Tika's default, 10000.
 | parser.tika.pipes.plugins.dir | - | Directory holding Tika Pipes plugin zips under `parser.tika.timeout`, not needed by default. Unset uses Tika's default plugin directory resolution.

external/tika/README.md

@@ -45,6 +45,10 @@ The time spent parsing a document can be limited with `parser.tika.timeout` (mil
 
 Each `ParserBolt` instance runs one fork, so a host needs the worker heap plus one fork heap per `ParserBolt` instance on it, and some overhead for each JVM. When several forks share a host, consider `-XX:ActiveProcessorCount` as well as a heap limit. A handful of related keys tune the forked JVMs: `parser.tika.pipes.jvmargs` (`-Xmx512m` is added when they set no maximum heap), `parser.tika.pipes.maxfilesperprocess` (restart a fork after this many documents, to bound slow leaks in parsing libraries; Tika's default is 10000), and `parser.tika.pipes.plugins.dir` (Tika Pipes plugins, not needed by default). Documents over 10MB are handed to the forked JVM through a temporary file in the worker's `java.io.tmpdir`.
 
-`parser.htmlmapper.classname` is not applied to parses running under `parser.tika.timeout`: an `HtmlMapper` cannot be passed to the forked JVM, which always uses Tika's `DefaultHtmlMapper`. That mapper drops the elements it does not consider safe from the DOM given to the parse filters, so XPath expressions written against the default `IdentityHtmlMapper` output may need adjusting.
+## JSoup filters
+
+The JSoupFilters configured with `jsoup.filters.config.file` are applied to every document parsed by Tika, not only HTML, the same way as with the `JSoupParserBolt`; they run before the parse filters. The JSoup document is built from the XHTML Tika produces, not from the original content: a PDF becomes `<div class="page">` and `<p>` elements, and the `<head>` of an HTML page holds Tika's metadata (e.g. `<meta name="dc:subject">` instead of `<meta name="keywords">`), so expressions written for the original HTML may need adjusting. Element names are lowercased. The document is only built if at least one JSoupFilter is configured.
+
+Under `parser.tika.timeout`, the default `IdentityHtmlMapper` is passed to the forked JVM, so the JSoup and parse filters get the same documents as with an in-process parse. Tika can only send the components registered in a `META-INF/tika/*.idx` file to the forked JVM: this module registers `IdentityHtmlMapper` in its `META-INF/tika/parse-context.idx`, which must be merged with the ones from Tika when building a shaded jar (see the `AppendingTransformer` entries in the archetype's `pom.xml`). If it is missing, the bolt logs a warning and the forked JVM uses Tika's `DefaultHtmlMapper`. Any other value of `parser.htmlmapper.classname` is ignored in this mode and the forked JVM uses Tika's `DefaultHtmlMapper`, which drops the elements it does not consider safe.
 
 Since Tika 4, Tika metadata keys use namespaced names, which surface as renamed `parse.*` keys, e.g. `parse.resourceName` is now `parse.tk:resource-name`.

external/tika/src/main/java/org/apache/stormcrawler/tika/JSoupBuilder.java

@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.tika;
+
+import org.apache.commons.lang3.StringUtils;
+import org.jsoup.nodes.DataNode;
+import org.jsoup.nodes.Document;
+import org.jsoup.nodes.Element;
+import org.jsoup.nodes.TextNode;
+import org.jsoup.parser.ParseSettings;
+import org.jsoup.parser.Tag;
+import org.xml.sax.Attributes;
+import org.xml.sax.helpers.DefaultHandler;
+
+/**
+ * Builds a JSoup document from the XHTML SAX events produced by Tika, so that JSoupFilters can be
+ * applied to any format Tika can parse. Element names are lowercased, namespaces are dropped. The
+ * document holds whatever was received so far if the parse stops early, e.g. when the text limit is
+ * reached.
+ */
+public class JSoupBuilder extends DefaultHandler {
+
+    private final Document document;
+
+    private Element current;
+
+    private final StringBuilder text = new StringBuilder();
+
+    /**
+     * @param baseUri URL of the document, used to resolve relative links
+     */
+    public JSoupBuilder(String baseUri) {
+        document = new Document(baseUri);
+        current = document;
+    }
+
+    public Document getDocument() {
+        flushText();
+        return document;
+    }
+
+    @Override
+    public void startElement(String uri, String localName, String qualifiedName, Attributes atts) {
+        flushText();
+        String name = StringUtils.isNotEmpty(localName) ? localName : qualifiedName;
+        Element element =
+                new Element(Tag.valueOf(name, ParseSettings.htmlDefault), document.baseUri());
+        for (int i = 0; i < atts.getLength(); i++) {
+            String attName = atts.getLocalName(i);
+            if (StringUtils.isEmpty(attName)) {
+                attName = atts.getQName(i);
+            }
+            if (attName.startsWith("xmlns")) {
+                continue;
+            }
+            element.attr(attName, atts.getValue(i));
+        }
+        current.appendChild(element);
+        current = element;
+    }
+
+    @Override
+    public void endElement(String uri, String localName, String qualifiedName) {
+        flushText();
+        if (current != document) {
+            current = current.parent();
+        }
+    }
+
+    @Override
+    public void characters(char[] ch, int start, int length) {
+        text.append(ch, start, length);
+    }
+
+    @Override
+    public void ignorableWhitespace(char[] ch, int start, int length) {
+        text.append(ch, start, length);
+    }
+
+    /** Adds the text received since the last element event as a single node. */
+    private void flushText() {
+        if (text.length() == 0) {
+            return;
+        }
+        String value = text.toString();
+        text.setLength(0);
+        // nothing outside of the root element
+        if (current == document) {
+            return;
+        }
+        // keep the content of scripts and styles as is, like the JSoup parser does
+        String name = current.normalName();
+        if ("script".equals(name) || "style".equals(name)) {
+            current.appendChild(new DataNode(value));
+        } else {
+            current.appendChild(new TextNode(value));
+        }
+    }
+}

external/tika/src/main/java/org/apache/stormcrawler/tika/NamespaceRestoringHandler.java

@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.tika;
+
+import java.util.Enumeration;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.tika.sax.ContentHandlerDecorator;
+import org.xml.sax.Attributes;
+import org.xml.sax.ContentHandler;
+import org.xml.sax.SAXException;
+import org.xml.sax.helpers.AttributesImpl;
+import org.xml.sax.helpers.NamespaceSupport;
+
+/**
+ * Resolves the namespaces of the events produced by a SAX parser which is not namespace aware, so
+ * that the XML returned by a Tika Pipes fork reaches the handlers as an in-process parse would. The
+ * fork writes the names kept by the IdentityHtmlMapper as they are, e.g. {@code xlink:href} or
+ * {@code x-on:click}, without declaring their prefix, which a namespace aware parser rejects. A
+ * name whose prefix is not declared is kept whole as the local name, an element without a declared
+ * prefix gets the default namespace in scope.
+ */
+class NamespaceRestoringHandler extends ContentHandlerDecorator {
+
+    private static final String XMLNS = "xmlns";
+
+    private final NamespaceSupport namespaces = new NamespaceSupport();
+
+    private final String[] parts = new String[3];
+
+    NamespaceRestoringHandler(ContentHandler handler) {
+        super(handler);
+    }
+
+    @Override
+    public void startElement(String uri, String localName, String qualifiedName, Attributes atts)
+            throws SAXException {
+        namespaces.pushContext();
+        for (int i = 0; i < atts.getLength(); i++) {
+            String prefix = declaredPrefix(atts.getQName(i));
+            if (prefix != null) {
+                namespaces.declarePrefix(prefix, atts.getValue(i));
+                super.startPrefixMapping(prefix, atts.getValue(i));
+            }
+        }
+
+        AttributesImpl resolved = new AttributesImpl();
+        for (int i = 0; i < atts.getLength(); i++) {
+            String name = atts.getQName(i);
+            if (declaredPrefix(name) != null) {
+                continue;
+            }
+            if (namespaces.processName(name, parts, true) != null) {
+                resolved.addAttribute(parts[0], parts[1], name, atts.getType(i), atts.getValue(i));
+            } else {
+                resolved.addAttribute("", name, name, atts.getType(i), atts.getValue(i));
+            }
+        }
+
+        resolveElement(qualifiedName);
+        super.startElement(parts[0], parts[1], qualifiedName, resolved);
+    }
+
+    @Override
+    public void endElement(String uri, String localName, String qualifiedName) throws SAXException {
+        resolveElement(qualifiedName);
+        super.endElement(parts[0], parts[1], qualifiedName);
+        Enumeration<?> declared = namespaces.getDeclaredPrefixes();
+        while (declared.hasMoreElements()) {
+            super.endPrefixMapping((String) declared.nextElement());
+        }
+        namespaces.popContext();
+    }
+
+    /** Sets the namespace and local name of an element in {@link #parts}. */
+    private void resolveElement(String qualifiedName) {
+        if (namespaces.processName(qualifiedName, parts, false) == null) {
+            parts[0] = StringUtils.defaultString(namespaces.getURI(""));
+            parts[1] = qualifiedName;
+        }
+    }
+
+    /** Returns the prefix declared by an attribute, "" for the default namespace, or null. */
+    private static String declaredPrefix(String attributeName) {
+        if (attributeName.equals(XMLNS)) {
+            return "";
+        }
+        if (attributeName.startsWith(XMLNS + ":")) {
+            return attributeName.substring(XMLNS.length() + 1);
+        }
+        return null;
+    }
+}