Let a middleware run on a media type other than HTML

CHANGELOG.md

@@ -90,6 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript` or `application/json`, written as a bare `type/subtype` in lower case. A response that declares a charset other than UTF-8, or whose status is not `200`, gets no middleware. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then the serve phase, and an action that decides about HTML there is refused by name. A response an entry covers loses the origin's `ETag` and `Last-Modified`, and one a `[[serve]]` entry covers is sent `private, no-store` when a module left state on the request.
 - The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, `cmp_id` unless it is a whole number from 1 to 4095, and entries that leave any page without the prompt are refused. The new `Middleware::template_key` keys a stored page by what a middleware writes that no setting decides, so a change to a stub stores pages again. The Trusted Server maintainers hold the crate until InMobi adopt it.
 - The stock modules' fetch middleware give as their `template_key` the text compiled into their crates that they write into a page, so a change to it stores pages again. The Google ad tag gives its head scripts, the `tsjs.adInit` bootstrap among them, and the GPT script's first-party path, Prebid its configuration script and bundle tag, Didomi its settings script, and Sourcepoint its head scripts and its CDN's first-party path. Permutive, Lockr, DataDome and Google Tag Manager give the first-party path they point a vendor's address at.
 - `[robots-txt]` serves a publisher's `/robots.txt` from the settings. `modules` names what makes the file, in order. `refuse_all` and `allow_all` are Trusted Server's own, and any other name is a module that contributes rules through the new `RobotsTxtContributor` extension point, declared with `with_robots_txt_contributor`. Trusted Server writes the file, so a contributor cannot produce a malformed one, a refusal cannot be loosened, and a path in `always_allow` stays open to every crawler refused the whole site. With `refuse_all` selected every response an adapter finalizes carries `X-Robots-Tag: noindex, nofollow`. A module is asked for the host the request was addressed to, never one a forwarded header names, and is given nothing of the crawler's connection. Its answer is held for that host in the request's key-value store until its refresh is due, and no adapter gives a request a working one, so every adapter asks each module on every request. A module that cannot answer has its last held answer served, and with none held, or with no rule in the whole file, the answer is `503` and never an empty file. On Fastly the file is answered before the Edge Cookie steps and the request filters. Without the section `/robots.txt` reaches the publisher's origin as before.

crates/cmp/inmobi/src/lib.rs

@@ -401,10 +401,14 @@ mod tests {
     /// The head markup `middleware` write in `phase`, in the order written.
     fn head_markup(phase: MiddlewarePhase, middleware: Vec<Arc<dyn Middleware>>) -> Vec<String> {
         let document_state = IntegrationDocumentState::default();
-        MiddlewareChain::new(phase, middleware)
-            .plan(&context(phase, &document_state))
-            .expect("the chain should plan")
-            .head_inserts
+        MiddlewareChain::new(
+            phase,
+            trusted_server_core::middleware::HTML_MEDIA_TYPE,
+            middleware,
+        )
+        .plan(&context(phase, &document_state))
+        .expect("the chain should plan")
+        .head_inserts
     }
 
     /// The one piece of head markup the prompt writes in `phase`.

crates/trusted-server-core/src/integrations/js_asset_proxy.rs

@@ -672,6 +672,7 @@ mod tests {
     ) -> String {
         let chain = MiddlewareChain::new(
             MiddlewarePhase::Fetch,
+            crate::middleware::HTML_MEDIA_TYPE,
             vec![Arc::new(ScriptAddress(integration))],
         );
         let config = HtmlProcessorConfig {

crates/trusted-server-core/src/integrations/registry.rs

@@ -175,15 +175,17 @@ impl IntegrationDocumentState {
 ///
 /// A request preparer or a request filter leaves a value under its
 /// integration id with [`IntegrationRequestState::insert`]. Where the
-/// request's reader is served an HTML document, every value is copied into
-/// the [`IntegrationDocumentState`] the `[[serve]]` entry's middleware share,
+/// request's reader is served an HTML document, or another body a
+/// `[[serve]]` entry covers, every value is copied into the
+/// [`IntegrationDocumentState`] the `[[serve]]` entry's middleware share,
 /// so the module's serve middleware reads it there, and the same values are
 /// handed to the module's response finalizer.
 ///
 /// A request that carries any value keeps to the origin path. Its document is
 /// never read from a shared template and never stored as one, and an HTML
-/// response with a body is sent `private, no-store`. A fetch middleware is
-/// handed none of these values.
+/// response with a body, or another response a `[[serve]]` entry covers, is
+/// sent `private, no-store`. A fetch middleware is handed none of these
+/// values.
 #[derive(Clone, Default)]
 pub struct IntegrationRequestState {
     values: IntegrationDocumentStateMap,
@@ -1396,6 +1398,13 @@ fn check_middleware(
                         phases_of(middleware).join(" and ")
                     )));
                 }
+                if !middleware.handles_media_type(&entry.media_type) {
+                    return Err(refuse(format!(
+                        "{at} covers `{}` and names `{name}`, which does not work on that media \
+                         type",
+                        entry.media_type
+                    )));
+                }
             }
         }
     }
@@ -1944,7 +1953,7 @@ impl IntegrationRegistry {
             .iter()
             .filter_map(|name| self.inner.middleware_named(name).map(Arc::clone))
             .collect();
-        MiddlewareChain::new(phase, middleware)
+        MiddlewareChain::new(phase, media_type, middleware)
     }
 
     /// Every integration id the registry was built from, named or not, in
@@ -2954,6 +2963,18 @@ pub(crate) mod test_support {
         pub(crate) const LINK_TARGET: &str = "/fixture/link";
         /// The selector the broken middleware asks for.
         pub(crate) const BROKEN_SELECTOR: &str = "a[";
+        /// The name of the fetch middleware that works on `text/plain`.
+        pub(crate) const TEXT: &str = "testing.middleware-fixture.text";
+        /// The media type the text middleware works on.
+        pub(crate) const TEXT_MEDIA_TYPE: &str = "text/plain";
+        /// The line the text middleware adds at the end of a text body.
+        pub(crate) const TEXT_LINE: &str = "# added by the middleware fixture\n";
+        /// The name of the serve middleware that writes a header on a text
+        /// response.
+        pub(crate) const TEXT_HEADER: &str = "testing.middleware-fixture.text-header";
+        /// The header the text header middleware writes, name and value.
+        pub(crate) const TEXT_HEADER_NAME: &str = "x-fixture-text";
+        pub(crate) const TEXT_HEADER_VALUE: &str = "served";
 
         /// The builder core's test build lists beside its own.
         pub(crate) const BUILDER: IntegrationBuilder =
@@ -3075,6 +3096,72 @@ pub(crate) mod test_support {
             }
         }
 
+        /// Passes a text body through and adds a line at its end.
+        struct AppendLine;
+
+        impl crate::streaming_processor::StreamProcessor for AppendLine {
+            fn process_chunk(
+                &mut self,
+                chunk: &[u8],
+                is_last: bool,
+            ) -> Result<Vec<u8>, std::io::Error> {
+                let mut output = chunk.to_vec();
+                if is_last {
+                    output.extend_from_slice(TEXT_LINE.as_bytes());
+                }
+                Ok(output)
+            }
+        }
+
+        struct Text;
+
+        impl Middleware for Text {
+            fn middleware_id(&self) -> &'static str {
+                TEXT
+            }
+
+            fn handles_media_type(&self, media_type: &str) -> bool {
+                media_type == TEXT_MEDIA_TYPE
+            }
+
+            fn phases(&self) -> &[MiddlewarePhase] {
+                &[MiddlewarePhase::Fetch]
+            }
+
+            fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction {
+                MiddlewareAction {
+                    stream: Some(Box::new(AppendLine)),
+                    ..MiddlewareAction::pass()
+                }
+            }
+        }
+
+        struct TextHeader;
+
+        impl Middleware for TextHeader {
+            fn middleware_id(&self) -> &'static str {
+                TEXT_HEADER
+            }
+
+            fn handles_media_type(&self, media_type: &str) -> bool {
+                media_type == TEXT_MEDIA_TYPE
+            }
+
+            fn phases(&self) -> &[MiddlewarePhase] {
+                &[MiddlewarePhase::Serve]
+            }
+
+            fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction {
+                MiddlewareAction {
+                    response_headers: vec![(
+                        http::HeaderName::from_static(TEXT_HEADER_NAME),
+                        http::HeaderValue::from_static(TEXT_HEADER_VALUE),
+                    )],
+                    ..MiddlewareAction::pass()
+                }
+            }
+        }
+
         struct Broken;
 
         impl Middleware for Broken {
@@ -3114,6 +3201,8 @@ pub(crate) mod test_support {
                     .with_middleware(Arc::new(Broken))
                     .with_middleware(Arc::new(Reader))
                     .with_middleware(Arc::new(BrokenReader))
+                    .with_middleware(Arc::new(Text))
+                    .with_middleware(Arc::new(TextHeader))
                     .build(),
             ))
         }
@@ -6134,20 +6223,28 @@ mod tests {
                 fixture::BROKEN,
                 fixture::READER,
                 fixture::BROKEN_READER,
+                fixture::TEXT,
+                fixture::TEXT_HEADER,
                 crate::service_version::MODULE,
             ],
             "should list the middleware the selected module supplies, and core's own after them"
         );
         assert_eq!(
             registry.middleware_in(MiddlewarePhase::Fetch),
-            [fixture::HEAD, fixture::LINKS, fixture::BROKEN],
+            [
+                fixture::HEAD,
+                fixture::LINKS,
+                fixture::BROKEN,
+                fixture::TEXT
+            ],
             "should list the middleware that run in the fetch phase"
         );
         assert_eq!(
             registry.middleware_in(MiddlewarePhase::Serve),
             [
                 fixture::READER,
                 fixture::BROKEN_READER,
+                fixture::TEXT_HEADER,
                 crate::service_version::MODULE
             ],
             "should list the middleware that run in the serve phase, core's own last"
@@ -6169,6 +6266,73 @@ mod tests {
         );
     }
 
+    /// An entry for `text/plain` naming the middleware that works on it.
+    fn text_entry(names: &[&str]) -> PhaseEntry {
+        PhaseEntry {
+            media_type: fixture::TEXT_MEDIA_TYPE.to_owned(),
+            path: None,
+            middleware: names.iter().map(|name| (*name).to_owned()).collect(),
+        }
+    }
+
+    #[test]
+    fn an_entry_runs_a_middleware_on_the_media_type_it_handles() {
+        let settings = fixture_settings_with_entries(vec![
+            html_entry(None, &[fixture::HEAD]),
+            text_entry(&[fixture::TEXT]),
+        ]);
+
+        let registry = IntegrationRegistry::new(&settings).expect("should build a registry");
+
+        let chain_for = |media_type: &str| {
+            registry
+                .middleware_chain(
+                    &settings.fetch,
+                    MiddlewarePhase::Fetch,
+                    media_type,
+                    "/a.txt",
+                )
+                .ids()
+        };
+        assert_eq!(
+            chain_for(fixture::TEXT_MEDIA_TYPE),
+            [fixture::TEXT],
+            "should run the text middleware on a text response"
+        );
+        assert_eq!(
+            chain_for(HTML_MEDIA_TYPE),
+            [fixture::HEAD],
+            "should run the page middleware on a page and not the text one"
+        );
+    }
+
+    #[test]
+    fn an_entry_naming_a_middleware_under_a_media_type_it_does_not_handle_is_refused() {
+        for (media_type, name) in [
+            (fixture::TEXT_MEDIA_TYPE, fixture::HEAD),
+            (HTML_MEDIA_TYPE, fixture::TEXT),
+        ] {
+            let settings = fixture_settings_with_entries(vec![PhaseEntry {
+                media_type: media_type.to_owned(),
+                path: None,
+                middleware: vec![name.to_owned()],
+            }]);
+
+            let error = IntegrationRegistry::new(&settings)
+                .err()
+                .expect("should refuse a middleware under a media type it does not handle");
+
+            let message = error.to_string();
+            assert!(
+                message.contains(&format!(
+                    "[[fetch]] entry 1 covers `{media_type}` and names `{name}`, which does not \
+                     work on that media type"
+                )),
+                "should name the entry, the media type and the middleware: {message}"
+            );
+        }
+    }
+
     #[test]
     fn a_module_no_section_selects_supplies_no_middleware() {
         let settings = crate::test_support::tests::create_test_settings();
@@ -6197,12 +6361,14 @@ mod tests {
             message.contains(
                 "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies"
             ) && message.contains(&format!(
-                "[{}, {}, {}, {}, {}, {}]",
+                "[{}, {}, {}, {}, {}, {}, {}, {}]",
                 fixture::HEAD,
                 fixture::LINKS,
                 fixture::BROKEN,
                 fixture::READER,
                 fixture::BROKEN_READER,
+                fixture::TEXT,
+                fixture::TEXT_HEADER,
                 crate::service_version::MODULE
             )),
             "should name the entry and list what could be named: {message}"
@@ -6255,6 +6421,8 @@ mod tests {
                 fixture::BROKEN,
                 fixture::READER,
                 fixture::BROKEN_READER,
+                fixture::TEXT,
+                fixture::TEXT_HEADER,
             ],
             "should list every middleware when there are no entries"
         );
@@ -6263,7 +6431,13 @@ mod tests {
                 html_entry(Some("/news/"), &[fixture::LINKS]),
                 html_entry(None, &[fixture::HEAD]),
             ]),
-            [fixture::BROKEN, fixture::READER, fixture::BROKEN_READER],
+            [
+                fixture::BROKEN,
+                fixture::READER,
+                fixture::BROKEN_READER,
+                fixture::TEXT,
+                fixture::TEXT_HEADER
+            ],
             "should leave out a middleware any entry names"
         );
     }
@@ -6326,7 +6500,13 @@ mod tests {
             .collect();
         assert_eq!(
             unnamed,
-            [fixture::LINKS, fixture::BROKEN, fixture::BROKEN_READER],
+            [
+                fixture::LINKS,
+                fixture::BROKEN,
+                fixture::BROKEN_READER,
+                fixture::TEXT,
+                fixture::TEXT_HEADER
+            ],
             "should count a middleware as named only by an entry of a phase it runs in"
         );
     }