Make robots.txt a middleware core supplies

CHANGELOG.md

@@ -62,6 +62,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Removed
 
+- **Breaking:** The `/robots.txt` route on the four adapters, with `handle_robots_txt` and the route's own `Cache-Control: public, max-age=3600`. The file is made by the `robots_txt` middleware where an entry names it, and takes the deployment's cache policy for the path as every file does. A configuration with `[robots-txt]` and no `[[fetch]]` entry naming `robots_txt` on `/robots.txt` is refused when it loads, and a build from before this change refuses that entry, so the new configuration and the new build go live together.
 - **Breaking:** HTTP Basic authentication is removed, with the `[[handlers]]` list that configured it. No request is asked for a password, and nothing is served beneath `/_ts/admin`. A configuration that still carries `[[handlers]]` is refused, with directions, when it is validated and when the service starts, so that no deployment believes a path is protected when it is not. A build before this change refuses a configuration without a handler covering `/_ts/admin`, and one with `[request_signing]` and no store ids, so no one configuration starts both builds: the new configuration and the new build go live together. A staged version (`ts deploy --staging`) reads the same configuration key from the config store it links, so a staging store of its own, written with `ts config push --staging`, lets the pair be checked together first. Put a password on a path of the publisher's own at its origin or in the hosting platform. The `handler_password` secret is no longer read, and its entries in `fastly.toml`, the Spin manifest and the smoke scripts are removed.
 - **Breaking:** `POST /_ts/admin/cache/purge` is removed. It let anyone holding one shared password empty the service's cache over the publisher's own domain. Purge with `ts cache purge`, which asks the hosting platform. A request to the removed path is answered `404` by the server itself and is never forwarded to the origin.
 - The Fastly adapter's management API client is removed. The deployed service no longer reads a Fastly API token from a secret store named `api-keys`, which it used to write its own config store and secret store when a key was rotated. A deployment that linked that store can unlink it and revoke the token. `fastly.toml` and the Viceroy template drop their `api-keys` entries.
@@ -93,6 +94,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `robots_txt`, core's own middleware for the file `[robots-txt]` describes, named in a `[[fetch]]` entry for `text/plain` on `/robots.txt`. Its preparation makes the file from the selected modules through their holding, and its action replaces the origin's file with it. The file is made in place of any other answer the origin gives to a `GET` or `HEAD`, such as a `404`, a page, an error or a redirect. The origin is asked for its whole file, without a crawler's conditional or range headers, and the middleware works on `/robots.txt` alone. A module that cannot answer with nothing held refuses the request with `503` and `Retry-After` before any document is processed. A section with no such entry, or with an entry naming the middleware on another path or media type, is refused when the settings load.
+- A middleware may say which request paths it works on, with `Middleware::handles_path`, that it writes the whole document, with `writes_whole_document`, so core makes the document in place of any answer it would not hand the middleware as a `200` of the entry's media type, and that it needs the origin's whole body, with `needs_whole_body`, so the origin is asked without the reader's conditional and range headers. By default a middleware works on every path its entry covers and says neither of the other two.
 - A middleware may prepare, and may refuse. `Middleware::prepare` runs in the request's handler before the document is processed, as the middleware's module with the request's services, and what it answers is kept under the phase and the middleware's name for `create` in that phase, so a decision that needs a store or the network is made there while `create` stays synchronous. The fetch phase of a document that may be stored is prepared with no client IP, TLS details or client signals in the services. A refusal answers the request with its status and `Retry-After`, with no document, and nothing is stored. Core logs a refusal as a warning naming the middleware, and answers a status that is not a client or server error with `500`.
 - A loading cache. `PlatformLoadingCache` in core loads a missing value once per key however many requests ask for it at the same moment, gives every waiting request the value or the error, and holds nothing that failed. A load is handed services with nothing of the reader in them, from `RuntimeServices::without_connection`, because its value serves every reader. A crate offers one through `IntegrationBuilder::with_loading_cache`, each feature that holds answers selects the module whose cache holds them in its own `[cache.<name>]` table, and an adapter takes the selections with `loading_caches_from`. A feature that selects none takes the one loading cache module the build links, and gets `NoLoadingCache`, which loads every time, when the build links none or several. The new `crates/loading-cache/51degrees`, the module `loading-cache.51degrees`, implements it with the published `fiftyone-caching` package as a brief copy in memory over the host's value cache on Fastly or its key-value store on Cloudflare and Spin, with the memory copy alone on Axum, for a deployment that links it, because no stock build does. Each application state keeps a memory copy of its own, which lasts one request where an adapter builds its state for each request, as Cloudflare and Spin do. Every load runs in its caller.
 - `[cache.robots-txt]` selects, with `module`, the loading cache that holds each robots.txt module's answer, and sets the lifetimes of its copies, in seconds: `memory_seconds` (default 60), `store_seconds` (default 604800) and `store_idle_seconds` (default 2592000). A memory lifetime may be 0, the others must be at least 1, and none may be more than a year. A `module` no linked crate offers a cache from is refused by `ts config validate` and when the service starts.

crates/trusted-server-adapter-axum/src/app.rs

@@ -40,7 +40,6 @@ use trusted_server_core::publisher::{
 use trusted_server_core::request_signing::{
     handle_trusted_server_discovery, handle_verify_signature,
 };
-use trusted_server_core::robots_txt::{ROBOTS_TXT_PATH, handle_robots_txt};
 use trusted_server_core::settings::Settings;
 use trusted_server_core::settings_data::{
     default_config_key, default_config_store_name, get_settings_from_config_store_with,
@@ -484,7 +483,6 @@ enum NamedRouteHandler {
     FirstPartyClick,
     FirstPartySign,
     FirstPartyProxyRebuild,
-    RobotsTxt,
 }
 
 struct NamedRoute {
@@ -493,7 +491,7 @@ struct NamedRoute {
     handler: NamedRouteHandler,
 }
 
-fn named_routes() -> [NamedRoute; 15] {
+fn named_routes() -> [NamedRoute; 14] {
     [
         NamedRoute {
             path: "/.well-known/trusted-server.json",
@@ -578,23 +576,9 @@ fn named_routes() -> [NamedRoute; 15] {
             primary_methods: &[Method::GET, Method::POST],
             handler: NamedRouteHandler::FirstPartyProxyRebuild,
         },
-        // Only when the settings carry `[robots-txt]`.
-        NamedRoute {
-            path: ROBOTS_TXT_PATH,
-            primary_methods: &[Method::GET, Method::HEAD],
-            handler: NamedRouteHandler::RobotsTxt,
-        },
     ]
 }
 
-/// Whether a named route is served for these settings.
-fn named_route_is_served(settings: &Settings, handler: NamedRouteHandler) -> bool {
-    match handler {
-        NamedRouteHandler::RobotsTxt => settings.robots_txt.is_some(),
-        _ => true,
-    }
-}
-
 fn named_route_handler(
     state: Arc<AppState>,
     handler: NamedRouteHandler,
@@ -669,9 +653,6 @@ fn named_route_handler(
                     NamedRouteHandler::FirstPartyProxyRebuild => {
                         handle_first_party_proxy_rebuild(&state.settings, &services, req).await
                     }
-                    NamedRouteHandler::RobotsTxt => {
-                        handle_robots_txt(&state.settings, &services, &state.registry, req).await
-                    }
                 }
             },
         ))
@@ -810,6 +791,25 @@ impl TrustedServerApp {
         let state = build_state_with_services(settings, Some(services))?;
         Ok(build_router(&state))
     }
+
+    /// Build the full router as
+    /// [`TrustedServerApp::routes_with_registrations`] does, with every
+    /// request handed a clone of `services`, as
+    /// [`TrustedServerApp::routes_with_settings_and_services`] hands them.
+    ///
+    /// # Errors
+    ///
+    /// Returns the errors [`TrustedServerApp::routes_with_registrations`]
+    /// returns.
+    pub fn routes_with_registrations_and_services(
+        settings: Settings,
+        integrations: &[IntegrationBuilder],
+        services: RuntimeServices,
+    ) -> Result<RouterService, Report<TrustedServerError>> {
+        let state =
+            build_state_with_registrations_and_services(settings, integrations, Some(services))?;
+        Ok(build_router(&state))
+    }
 }
 
 fn build_router(state: &Arc<AppState>) -> RouterService {
@@ -834,9 +834,6 @@ fn build_router(state: &Arc<AppState>) -> RouterService {
     });
 
     for route in named_routes() {
-        if !named_route_is_served(&state.settings, route.handler) {
-            continue;
-        }
         for method in route.primary_methods {
             router = router.route(
                 route.path,

crates/trusted-server-adapter-axum/tests/routes.rs

@@ -918,3 +918,139 @@ async fn the_sellers_module_makes_an_ads_txt_the_origin_does_not_have() {
         "should be the sellers lines alone"
     );
 }
+
+// ---------------------------------------------------------------------------
+// robots.txt
+// ---------------------------------------------------------------------------
+
+/// The route test settings with a `[robots-txt]` section that refuses every
+/// crawler and keeps one path open, and the entry that places the file.
+fn settings_refusing_every_crawler() -> trusted_server_core::settings::Settings {
+    trusted_server_core::settings::Settings::from_toml(
+        r#"
+            [publisher]
+            domain = "test-publisher.example.com"
+            cookie_domain = ".test-publisher.example.com"
+            origin_url = "https://origin.test-publisher.example.com"
+            proxy_secret = "integration-test-proxy-secret"
+
+            [ec]
+            module = "hmac"
+
+            [ec.hmac]
+            passphrase = "test-secret-key-32-bytes-minimum"
+
+            [geo]
+            assume_single_jurisdiction = true
+
+            [robots-txt]
+            modules = "refuse_all"
+            always_allow = ["/ads.txt"]
+
+            [[fetch]]
+            media_type = "text/plain"
+            path = "/robots.txt"
+            middleware = ["robots_txt"]
+        "#,
+    )
+    .expect("should parse robots.txt test settings")
+    .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample())
+}
+
+/// `/robots.txt` is the publisher path's, answered against an origin that
+/// answers each request in turn. A `GET` has the origin's file replaced by
+/// the refusal, a `HEAD` has the file made where the origin has none and
+/// gets no body, and a `POST` keeps the origin's `404`, because a file is
+/// made only for a `GET` or a `HEAD`. Each carries the refusal's
+/// `X-Robots-Tag`, the file's own answer included.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn robots_txt_is_served_through_the_publisher_path_with_its_tag() {
+    use std::sync::Arc;
+
+    use edgezero_core::http::Response;
+    use trusted_server_core::platform::PlatformHttpClient;
+    use trusted_server_core::platform::test_support::{
+        StubHttpClient, build_services_with_http_client,
+    };
+
+    let origin = Arc::new(StubHttpClient::new());
+    origin.push_response_with_headers(
+        200,
+        b"User-agent: *\nAllow: /\n".to_vec(),
+        vec![("content-type", "text/plain")],
+    );
+    for _ in 0..2 {
+        origin.push_response_with_headers(
+            404,
+            b"<html><body>not found</body></html>".to_vec(),
+            vec![("content-type", "text/html")],
+        );
+    }
+    let router = TrustedServerApp::routes_with_settings_and_services(
+        settings_refusing_every_crawler(),
+        build_services_with_http_client(Arc::clone(&origin) as Arc<dyn PlatformHttpClient>),
+    )
+    .expect("should build router with a stub origin");
+    let robots_txt = |method: &str| {
+        edgezero_core::http::request_builder()
+            .method(method)
+            .uri("https://test-publisher.example.com/robots.txt")
+            .header("host", "test-publisher.example.com")
+            .body(edgezero_core::body::Body::empty())
+            .expect("should build request")
+    };
+    let tag = |response: &Response| {
+        response
+            .headers()
+            .get("x-robots-tag")
+            .and_then(|value| value.to_str().ok())
+            .map(str::to_owned)
+    };
+    let body = |response: Response| {
+        let bytes = response
+            .into_body()
+            .into_bytes()
+            .expect("should hold the answer in memory");
+        String::from_utf8(bytes.to_vec()).expect("the body is text")
+    };
+
+    let get = router
+        .clone()
+        .oneshot(robots_txt("GET"))
+        .await
+        .expect("should serve the GET");
+    assert_eq!(get.status(), 200);
+    assert_eq!(tag(&get).as_deref(), Some("noindex, nofollow"));
+    assert_eq!(body(get), "User-agent: *\nAllow: /ads.txt\nDisallow: /\n");
+
+    let head = router
+        .clone()
+        .oneshot(robots_txt("HEAD"))
+        .await
+        .expect("should serve the HEAD");
+    assert_eq!(
+        head.status(),
+        200,
+        "the file is made where the origin has none"
+    );
+    assert_eq!(tag(&head).as_deref(), Some("noindex, nofollow"));
+    assert_eq!(body(head), "", "a HEAD gets no body");
+
+    let post = router
+        .clone()
+        .oneshot(robots_txt("POST"))
+        .await
+        .expect("should serve the POST");
+    assert_eq!(
+        post.status(),
+        404,
+        "a file is made only for a GET or a HEAD"
+    );
+    assert_eq!(tag(&post).as_deref(), Some("noindex, nofollow"));
+
+    assert_eq!(
+        origin.recorded_request_methods(),
+        ["GET", "HEAD", "POST"],
+        "each request reaches the origin"
+    );
+}