Make a file the origin does not have from the middleware named for its path

CHANGELOG.md

@@ -91,7 +91,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the end of the publisher's `ads.txt`, from `crates/ads-txt/sellers`, where a `[[fetch]]` entry for `text/plain` names it. Each line is checked against the ads.txt grammar as the settings load, and a selected module no entry places is refused.
+- A file the origin does not have is made by the middleware a `[[fetch]]` entry with a `path`, for a media type other than HTML, names for it. When the origin answers `404 Not Found` for that path itself, which names a file and not a folder, the middleware runs on an empty document of the entry's media type and its output is the file, so `ads.txt` is made from the sellers lines alone where the publisher has none. With no such entry, or for a request that names no host, the origin's answer stands.
+- `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the publisher's `ads.txt`, from `crates/ads-txt/sellers`. The module is a middleware for `text/plain`, named in a `[[fetch]]` entry on the file's path, which passes the origin's file through and writes the lines after its last one, so the publisher's file stays authoritative, and a publisher whose origin has no file gets one made of the lines. Each line is checked as the settings load as a data record or a variable of the ads.txt grammar, and a selected module that no entry for `text/plain` on `/ads.txt` places is refused.
 - A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript`, `application/json` or `image/svg+xml`, written as a bare `type/subtype` in lower case. A response that declares a charset other than UTF-8, or whose status is not `200`, gets no middleware. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then, for a response that is not stored, the serve phase, and an action that decides about HTML there is refused by name. A response an entry covers loses the origin's `ETag` and `Last-Modified`, and one a `[[serve]]` entry covers is sent `private, no-store` when a module left state on the request.
 - The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, `cmp_id` unless it is a whole number from 1 to 4095, and entries that leave any page without the prompt are refused. The new `Middleware::template_key` keys a stored page by what a middleware writes that no setting decides, so a change to a stub stores pages again. The Trusted Server maintainers hold the crate until InMobi adopt it.
 - The stock modules' fetch middleware give as their `template_key` the text compiled into their crates that they write into a page, so a change to it stores pages again. The Google ad tag gives its head scripts, the `tsjs.adInit` bootstrap among them, and the GPT script's first-party path, Prebid its configuration script and bundle tag, Didomi its settings script, and Sourcepoint its head scripts and its CDN's first-party path. Permutive, Lockr, DataDome and Google Tag Manager give the first-party path they point a vendor's address at.

crates/ads-txt/sellers/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "trusted-server-ads-txt-sellers"
-description = "Adds a publisher's sellers to the origin's ads.txt as it passes through, from the lines the settings give."
+description = "Adds a publisher's sellers to the origin's ads.txt from the lines the settings give, or makes the file from them where the origin has none."
 authors = { workspace = true }
 edition = { workspace = true }
 license = { workspace = true }

crates/ads-txt/sellers/src/lib.rs

@@ -1,6 +1,7 @@
 //! The sellers a deployment adds to the publisher's `ads.txt`.
 //!
-//! The publisher's own file at the origin stays authoritative.
+//! The publisher's own file at the origin stays authoritative. Where the
+//! origin has none, the file is made of the lines alone.
 
 use std::sync::Arc;
 
@@ -28,6 +29,10 @@ pub const MODULE: &str = "ads-txt.sellers";
 /// works on.
 pub const MEDIA_TYPE: &str = "text/plain";
 
+/// The path of the file, which the entry placing the middleware names, so
+/// that the file is also made where the origin has none.
+const PATH: &str = "/ads.txt";
+
 /// The variables the specification names, each written as `NAME=value`.
 const VARIABLES: [&str; 5] = [
     "CONTACT",
@@ -138,21 +143,25 @@ fn validate_lines(config: &SellersConfig) -> Result<(), Report<TrustedServerErro
     Ok(())
 }
 
-/// Refuses a document that selects the module but names it in no entry, where
-/// the lines would never be written.
+/// Refuses a document that selects the module but names it in no entry for
+/// the file's path, where the lines would never be written into the file or
+/// make it where the origin has none.
 fn require_an_entry(settings: &Settings) -> Result<(), Report<TrustedServerError>> {
     let placed = settings.fetch.entries().iter().any(|entry| {
-        entry.media_type == MEDIA_TYPE && entry.middleware.iter().any(|name| name == MODULE)
+        entry.media_type == MEDIA_TYPE
+            && entry.path.as_deref() == Some(PATH)
+            && entry.middleware.iter().any(|name| name == MODULE)
     });
     if placed {
         return Ok(());
     }
     Err(Report::new(TrustedServerError::Settings {
         message: format!(
-            "[ads-txt] selects `sellers`, but no [[fetch]] entry for `{MEDIA_TYPE}` names \
-             `{MODULE}`, so the lines would never be written. Add an entry with media_type = \
-             `{MEDIA_TYPE}`, path = `/ads.txt` and the middleware. The path is a prefix, so \
-             the entry covers every path that starts with `/ads.txt`"
+            "[ads-txt] selects `sellers`, but no [[fetch]] entry for `{MEDIA_TYPE}` with path \
+             = `{PATH}` names `{MODULE}`, so the lines would never be written into the file or \
+             make it where the origin has none. Add an entry with media_type = \
+             `{MEDIA_TYPE}`, path = `{PATH}` and the middleware. The path is a prefix, so the \
+             entry also covers every path that starts with `{PATH}`"
         ),
     }))
 }
@@ -392,13 +401,39 @@ mod tests {
 
         let refused = validate(&settings).expect_err("should refuse a module no entry places");
         assert!(
-            format!("{refused:?}")
-                .contains("no [[fetch]] entry for `text/plain` names `ads-txt.sellers`"),
+            format!("{refused:?}").contains(
+                "no [[fetch]] entry for `text/plain` with path = `/ads.txt` names \
+                 `ads-txt.sellers`"
+            ),
             "{refused:?}"
         );
         assert!(register(&settings).is_err());
     }
 
+    #[test]
+    fn a_document_placing_the_module_off_the_file_s_path_is_refused() {
+        for path in [None, Some("/robots.txt"), Some("/ads"), Some("/")] {
+            let path_line = path.map_or_else(String::new, |path| format!("path = \"{path}\"\n"));
+            let settings = page_settings(&format!(
+                "[ads-txt]\nmodules = [\"sellers\"]\n\n[ads-txt.sellers]\nlines = \
+                 [\"{RECORD}\"]\n\n[[fetch]]\nmedia_type = \"text/plain\"\n{path_line}\
+                 middleware = [\"ads-txt.sellers\"]\n"
+            ));
+
+            let Err(refused) = validate(&settings) else {
+                panic!("should refuse the module placed at {path:?}, where no file is made");
+            };
+            assert!(
+                format!("{refused:?}").contains("with path = `/ads.txt` names `ads-txt.sellers`"),
+                "should ask for the entry on `/ads.txt`: {refused:?}"
+            );
+            assert!(
+                register(&settings).is_err(),
+                "should not register the module placed at {path:?}"
+            );
+        }
+    }
+
     #[test]
     fn a_line_that_is_not_a_record_or_a_variable_is_refused_for_its_reason() {
         let cases = [

crates/trusted-server-adapter-axum/tests/routes.rs

@@ -20,7 +20,12 @@ const LEGACY_ADMIN_DENY_METHODS: &[&str] =
 /// every route into a startup error page (and its route table into the
 /// fallback-only set).
 fn test_settings() -> trusted_server_core::settings::Settings {
-    trusted_server_core::settings::Settings::from_toml(
+    test_settings_with("")
+}
+
+/// The route tests' settings with `extra` appended.
+fn test_settings_with(extra: &str) -> trusted_server_core::settings::Settings {
+    trusted_server_core::settings::Settings::from_toml(&format!(
         r#"
             [publisher]
             domain = "test-publisher.example.com"
@@ -36,8 +41,9 @@ fn test_settings() -> trusted_server_core::settings::Settings {
 
             [geo]
             assume_single_jurisdiction = true
-        "#,
-    )
+            {extra}
+        "#
+    ))
     .expect("should parse route test settings")
     // A loaded domain carries its policy, and these tests decide under the
     // repository's sample.
@@ -832,3 +838,83 @@ async fn health_carries_no_version() {
         "should write no version on /health, which the service_version middleware writes on a page"
     );
 }
+
+/// A publisher whose origin has no `ads.txt` gets one made of the sellers
+/// lines, through this adapter's router and the stock sellers module.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn the_sellers_module_makes_an_ads_txt_the_origin_does_not_have() {
+    use std::sync::Arc;
+
+    use trusted_server_core::platform::PlatformHttpClient;
+    use trusted_server_core::platform::test_support::{
+        StubHttpClient, build_services_with_http_client,
+    };
+
+    const SELLERS: &str = r#"
+        [ads-txt]
+        modules = ["sellers"]
+
+        [ads-txt.sellers]
+        lines = ["ssp.example, 12345, DIRECT, f08c47fec0942fa0"]
+
+        [[fetch]]
+        media_type = "text/plain"
+        path = "/ads.txt"
+        middleware = ["ads-txt.sellers"]
+    "#;
+
+    let origin = Arc::new(StubHttpClient::new());
+    origin.push_response_with_headers(
+        404,
+        b"<html><body>Not found</body></html>".to_vec(),
+        vec![
+            ("content-type", "text/html; charset=utf-8"),
+            ("set-cookie", "session=origin; Path=/"),
+            ("cache-control", "public, max-age=600"),
+        ],
+    );
+    let router = TrustedServerApp::routes_with_settings_and_services(
+        test_settings_with(SELLERS),
+        build_services_with_http_client(Arc::clone(&origin) as Arc<dyn PlatformHttpClient>),
+    )
+    .expect("should build the router with the sellers module");
+    let request = edgezero_core::http::request_builder()
+        .method("GET")
+        .uri("https://test-publisher.example.com/ads.txt")
+        .header("host", "test-publisher.example.com")
+        .body(edgezero_core::body::Body::empty())
+        .expect("should build the ads.txt request");
+
+    let response = router
+        .oneshot(request)
+        .await
+        .expect("should answer the ads.txt request");
+
+    assert_eq!(
+        response.status(),
+        200,
+        "should make the file the origin does not have"
+    );
+    assert_eq!(
+        response
+            .headers()
+            .get("content-type")
+            .and_then(|value| value.to_str().ok()),
+        Some("text/plain; charset=utf-8")
+    );
+    assert!(
+        response.headers().get("set-cookie").is_none()
+            && response.headers().get("cache-control").is_none(),
+        "should carry none of the headers the origin sent with its 404: {:?}",
+        response.headers()
+    );
+    let body = response
+        .into_body()
+        .into_bytes()
+        .expect("should buffer the made file");
+    assert_eq!(
+        body.as_ref(),
+        b"ssp.example, 12345, DIRECT, f08c47fec0942fa0\n",
+        "should be the sellers lines alone"
+    );
+}

crates/trusted-server-core/src/middleware.rs

@@ -482,6 +482,25 @@ impl PhaseEntries {
         names
     }
 
+    /// The entry that makes the document the origin does not have at
+    /// `path`, being the first written, for a media type core streams other
+    /// than HTML, whose `path` is `path` itself and that names a middleware.
+    /// A longer path the entry's `path` begins, and a folder, written with its
+    /// closing slash, get nothing made, so a page the origin does not have
+    /// keeps the origin's answer.
+    #[must_use]
+    pub fn making(&self, path: &str) -> Option<&PhaseEntry> {
+        if path.ends_with('/') {
+            return None;
+        }
+        self.0.iter().find(|entry| {
+            entry.media_type != HTML_MEDIA_TYPE
+                && is_dispatched_media_type(&entry.media_type)
+                && entry.path.as_deref() == Some(path)
+                && !entry.middleware.is_empty()
+        })
+    }
+
     /// Checks each entry's media type, path and names, and that no entry is
     /// hidden behind an earlier one, or covers the same pages as one.
     ///

crates/trusted-server-core/src/middleware/tests.rs

@@ -1874,3 +1874,144 @@ fn deploy_validation_refuses_a_misshapen_entry() {
         "should name the entry: {error}"
     );
 }
+
+#[test]
+fn the_making_entry_is_for_the_request_path_and_names_a_middleware() {
+    let entries = PhaseEntries::new(vec![
+        PhaseEntry {
+            media_type: "text/plain".to_owned(),
+            path: Some("/ads.txt".to_owned()),
+            middleware: Vec::new(),
+        },
+        PhaseEntry {
+            media_type: "text/plain".to_owned(),
+            path: Some("/ads.txt".to_owned()),
+            middleware: vec!["example.sellers".to_owned()],
+        },
+        PhaseEntry {
+            media_type: HTML_MEDIA_TYPE.to_owned(),
+            path: None,
+            middleware: vec!["example.page".to_owned()],
+        },
+    ]);
+
+    assert_eq!(
+        entries
+            .making("/ads.txt")
+            .map(|entry| entry.middleware.as_slice()),
+        Some(&["example.sellers".to_owned()][..]),
+        "should pass the entry naming no middleware and take the first that names one"
+    );
+    assert!(
+        entries.making("/robots.txt").is_none(),
+        "should make nothing where no entry is for the request path, whatever covers every path"
+    );
+}
+
+#[test]
+fn an_html_entry_with_a_path_makes_nothing() {
+    let entries = PhaseEntries::new(vec![PhaseEntry {
+        media_type: HTML_MEDIA_TYPE.to_owned(),
+        path: Some("/news/missing".to_owned()),
+        middleware: vec!["example.page".to_owned()],
+    }]);
+
+    assert!(
+        entries.making("/news/missing").is_none(),
+        "should leave a page the origin does not have as the origin's answer"
+    );
+}
+
+#[test]
+fn a_file_is_made_at_the_entry_path_alone() {
+    let entries = PhaseEntries::new(vec![
+        PhaseEntry {
+            media_type: "text/css".to_owned(),
+            path: Some("/".to_owned()),
+            middleware: vec!["example.styles".to_owned()],
+        },
+        PhaseEntry {
+            media_type: "text/plain".to_owned(),
+            path: Some("/ads.txt".to_owned()),
+            middleware: vec!["example.sellers".to_owned()],
+        },
+        PhaseEntry {
+            media_type: "text/plain".to_owned(),
+            path: Some("/feeds/".to_owned()),
+            middleware: vec!["example.feeds".to_owned()],
+        },
+    ]);
+
+    assert_eq!(
+        entries
+            .making("/ads.txt")
+            .map(|entry| entry.media_type.as_str()),
+        Some("text/plain"),
+        "should make the file from the entry for its own path"
+    );
+    for path in [
+        "/site.css",
+        "/missing-page",
+        "/ads.txt.bak",
+        "/feeds/latest",
+    ] {
+        assert!(
+            entries.making(path).is_none(),
+            "should make nothing at {path}, which an entry's path only begins"
+        );
+    }
+    for folder in ["/", "/feeds/"] {
+        assert!(
+            entries.making(folder).is_none(),
+            "should make nothing for the folder {folder}"
+        );
+    }
+}
+
+#[test]
+fn the_first_entry_written_for_the_path_makes_the_file() {
+    let entries = PhaseEntries::new(vec![
+        PhaseEntry {
+            media_type: "text/css".to_owned(),
+            path: Some("/ads.txt".to_owned()),
+            middleware: vec!["example.styles".to_owned()],
+        },
+        PhaseEntry {
+            media_type: "text/plain".to_owned(),
+            path: Some("/ads.txt".to_owned()),
+            middleware: vec!["example.sellers".to_owned()],
+        },
+    ]);
+
+    assert_eq!(
+        entries
+            .making("/ads.txt")
+            .map(|entry| entry.media_type.as_str()),
+        Some("text/css"),
+        "should take the entry written first among those for the path"
+    );
+}
+
+#[test]
+fn an_html_entry_ahead_does_not_shadow_the_entry_that_makes_the_file() {
+    let entries = PhaseEntries::new(vec![
+        PhaseEntry {
+            media_type: HTML_MEDIA_TYPE.to_owned(),
+            path: Some("/".to_owned()),
+            middleware: vec!["example.page".to_owned()],
+        },
+        PhaseEntry {
+            media_type: "text/plain".to_owned(),
+            path: Some("/ads.txt".to_owned()),
+            middleware: vec!["example.sellers".to_owned()],
+        },
+    ]);
+
+    assert_eq!(
+        entries
+            .making("/ads.txt")
+            .map(|entry| entry.media_type.as_str()),
+        Some("text/plain"),
+        "should make the file from the text/plain entry, not as HTML"
+    );
+}