Add robots.txt rules from modules

CHANGELOG.md

@@ -86,6 +86,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `[robots-txt]` serves a publisher's `/robots.txt` from the settings. `modules` names what makes the file, in order. `refuse_all` and `allow_all` are Trusted Server's own, and any other name is a module that contributes rules through the new `RobotsTxtContributor` extension point, declared with `with_robots_txt_contributor`. Trusted Server writes the file, so a contributor cannot produce a malformed one, a refusal cannot be loosened, and a path in `always_allow` stays open to every crawler refused the whole site. With `refuse_all` selected every response an adapter finalizes carries `X-Robots-Tag: noindex, nofollow`. A module is asked for the host the request was addressed to, never one a forwarded header names, and is given nothing of the crawler's connection. Its answer is held for that host in the request's key-value store until its refresh is due, and no adapter gives a request a working one, so every adapter asks each module on every request. A module that cannot answer has its last held answer served, and with none held, or with no rule in the whole file, the answer is `503` and never an empty file. On Fastly the file is answered before the Edge Cookie steps and the request filters. Without the section `/robots.txt` reaches the publisher's origin as before.
 - `ts keys rotate` and `ts keys deactivate` rotate and retire the request signing keys from an operator's machine. They write the two stores the service links as `jwks_store` and `signing_keys` through the `fastly` CLI, with the token from `FASTLY_API_TOKEN`. A private key reaches `fastly` on its standard input and never in its arguments, and a key id no platform can store is refused before anything is written. A command that cannot read the config store writes nothing, and a delete that failed part way can be run again.
 - One Fastly service can serve several publishers. A `TS__CONFIG__KEY` entry in the `trusted_server_config` store that contains `{host}` names an app-config blob for each host, pushed with `ts config push --key <host>`, and each request reads the blob under its own host. Each host's blob names its permission policy file and pins its SHA-256, and the push keeps each file `--permissions` supplies once, in a shared entry under `permissions_<sha256>`, written before the blob, which a live push writes a minute after it writes an entry. A host with no blob is answered `421 Misdirected Request`, never from the logical store ID or from another host's blob, a malformed selector is answered `500`, and a sandbox that serves several requests keeps an application for each blob, found only by that blob's key. Without the entry, or with a selector that has no placeholder, the service reads one blob. `scripts/config-by-host-local-test.sh` runs it under Viceroy.
 - `ts audit generate` writes the `[[fetch]]` and `[[serve]]` entries that run the page changes of the modules it selects on every HTML page, and `trusted-server.example.toml` documents the entries with every middleware the stock modules supply. `trusted-server-modules` lists those middleware with `middleware()` and `middleware_for(...)`.

crates/testing/seam-probe/src/lib.rs

@@ -2,11 +2,12 @@
 //! part of the integration seam from a vendor crate's position.
 //!
 //! One registration carries a browser module, a proxy route, a geo module,
-//! an Edge Cookie identity module, a device module and a middleware for
-//! each phase of a page, alongside its own configuration block, and the builder
-//! adds a request preparer, a demand implementation `[demand]` can name and
-//! an ad server implementation `[ad-server]` can name. The round-trip
-//! tests in `crates/trusted-server-adapter-axum/tests/seam_probe.rs` and
+//! an Edge Cookie identity module, a device module, a robots.txt contributor
+//! and a middleware for each phase of a page, alongside its own configuration
+//! block, and the builder adds a request preparer, a demand implementation
+//! `[demand]` can name and an ad server implementation `[ad-server]` can
+//! name. The round-trip tests in
+//! `crates/trusted-server-adapter-axum/tests/seam_probe.rs` and
 //! `crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs` drive
 //! each of those through a real adapter, so the seam is proven by a caller that
 //! core does not know about.
@@ -46,10 +47,11 @@ use trusted_server_core::integrations::{
 use trusted_server_core::middleware::{
     AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, TextHandler,
 };
-use trusted_server_core::module_context::ModuleCall;
+use trusted_server_core::module_context::{ModuleCall, ModuleRequest};
 use trusted_server_core::platform::{
     GeoInfo, PlatformError, PlatformGeo, PlatformResponse, RuntimeServices,
 };
+use trusted_server_core::robots_txt::{Contribution, RobotsTxtContributor};
 use trusted_server_core::settings::{IntegrationConfig, Settings};
 use validator::Validate;
 
@@ -59,12 +61,16 @@ pub const SEAM_PROBE_ID: &str = "seam_probe";
 /// The name this module is selected by, its crate folder below `crates/`, so
 /// `[testing] modules = ["seam-probe"]` selects it. Its geo, Edge Cookie and
 /// device modules carry the same name, so `[geo] module`, `[ec] module` and
-/// `[device] module` select each of them as `testing.seam-probe`.
+/// `[device] module` select each of them as `testing.seam-probe`, and so does
+/// `[robots-txt] modules` its robots.txt contributor.
 #[must_use]
 pub fn module_name() -> &'static str {
     trusted_server_core::module_name!()
 }
 
+/// The crawler the probe's robots.txt rules are addressed to.
+pub const SEAM_PROBE_ROBOTS_AGENT: &str = "SeamProbeBot";
+
 /// Source label the registry uses in duplicate-id errors.
 pub const SEAM_PROBE_SOURCE: &str = "trusted-server-integration-seam-probe";
 
@@ -404,6 +410,42 @@ impl DeviceModule for SeamProbeDevice {
     }
 }
 
+/// The probe's robots.txt contributor, declared on the same registration.
+///
+/// Its one rule names the request it was asked for, so a test can read back
+/// that the contributor was handed the host and the path and not the query.
+#[derive(Debug)]
+pub struct SeamProbeRobotsTxt;
+
+impl SeamProbeRobotsTxt {
+    fn rules(&self, request: ModuleRequest<'_>) -> Contribution {
+        Contribution::parse(&format!(
+            "User-Agent: {SEAM_PROBE_ROBOTS_AGENT}\nDisallow: /{}{}?{}",
+            request.host(),
+            request.path(),
+            request.query()
+        ))
+    }
+}
+
+#[async_trait::async_trait(?Send)]
+impl RobotsTxtContributor for SeamProbeRobotsTxt {
+    fn refresh(&self) -> core::time::Duration {
+        core::time::Duration::from_secs(60)
+    }
+
+    fn fingerprint(&self) -> String {
+        "seam-probe".to_owned()
+    }
+
+    async fn contribute(
+        &self,
+        call: ModuleCall<'_>,
+    ) -> Result<Contribution, Report<TrustedServerError>> {
+        Ok(call.inject(self, Self::rules)?)
+    }
+}
+
 #[async_trait::async_trait(?Send)]
 impl PlatformGeo for SeamProbeGeo {
     async fn lookup(
@@ -694,11 +736,13 @@ pub fn register(
             .with_geo_module(module_name(), Arc::new(SeamProbeGeo::new(config.country)));
     }
     {
-        // Identity and device are declared the same way location is, so one
-        // module supplies all three and there is no second mechanism.
+        // Identity, device and robots.txt rules are declared the same way
+        // location is, so one module supplies all four and there is no second
+        // mechanism.
         registration = registration
             .with_ec_module(module_name(), Arc::new(SeamProbeEc))
-            .with_device_module(module_name(), Arc::new(SeamProbeDevice));
+            .with_device_module(module_name(), Arc::new(SeamProbeDevice))
+            .with_robots_txt_contributor(module_name(), Arc::new(SeamProbeRobotsTxt));
     }
 
     Ok(Some(registration.build()))

crates/trusted-server-adapter-axum/src/app.rs

@@ -38,6 +38,7 @@ use trusted_server_core::publisher::{
 use trusted_server_core::request_signing::{
     handle_trusted_server_discovery, handle_verify_signature,
 };
+use trusted_server_core::robots_txt::{ROBOTS_TXT_PATH, handle_robots_txt};
 use trusted_server_core::settings::Settings;
 use trusted_server_core::settings_data::{
     default_config_key, default_config_store_name, get_settings_from_config_store_with,
@@ -461,6 +462,7 @@ enum NamedRouteHandler {
     FirstPartyClick,
     FirstPartySign,
     FirstPartyProxyRebuild,
+    RobotsTxt,
 }
 
 struct NamedRoute {
@@ -469,7 +471,7 @@ struct NamedRoute {
     handler: NamedRouteHandler,
 }
 
-fn named_routes() -> [NamedRoute; 14] {
+fn named_routes() -> [NamedRoute; 15] {
     [
         NamedRoute {
             path: "/.well-known/trusted-server.json",
@@ -554,9 +556,23 @@ fn named_routes() -> [NamedRoute; 14] {
             primary_methods: &[Method::GET, Method::POST],
             handler: NamedRouteHandler::FirstPartyProxyRebuild,
         },
+        // Only when the settings carry `[robots-txt]`.
+        NamedRoute {
+            path: ROBOTS_TXT_PATH,
+            primary_methods: &[Method::GET, Method::HEAD],
+            handler: NamedRouteHandler::RobotsTxt,
+        },
     ]
 }
 
+/// Whether a named route is served for these settings.
+fn named_route_is_served(settings: &Settings, handler: NamedRouteHandler) -> bool {
+    match handler {
+        NamedRouteHandler::RobotsTxt => settings.robots_txt.is_some(),
+        _ => true,
+    }
+}
+
 fn named_route_handler(
     state: Arc<AppState>,
     handler: NamedRouteHandler,
@@ -631,6 +647,9 @@ fn named_route_handler(
                     NamedRouteHandler::FirstPartyProxyRebuild => {
                         handle_first_party_proxy_rebuild(&state.settings, &services, req).await
                     }
+                    NamedRouteHandler::RobotsTxt => {
+                        handle_robots_txt(&state.settings, &services, &state.registry, req).await
+                    }
                 }
             },
         ))
@@ -793,6 +812,9 @@ fn build_router(state: &Arc<AppState>) -> RouterService {
     });
 
     for route in named_routes() {
+        if !named_route_is_served(&state.settings, route.handler) {
+            continue;
+        }
         for method in route.primary_methods {
             router = router.route(
                 route.path,

crates/trusted-server-adapter-axum/src/middleware.rs

@@ -103,6 +103,10 @@ pub(crate) fn apply_finalize_headers(settings: &Settings, response: &mut Respons
     trusted_server_core::response_privacy::apply_response_headers_with_cache_privacy(
         settings, response,
     );
+
+    // Last, so that nothing an origin or an operator's header set can loosen
+    // a site taken out of search.
+    trusted_server_core::robots_txt::apply_response_robots_tag(settings, response);
 }
 
 // ---------------------------------------------------------------------------