diff --git a/CHANGELOG.md b/CHANGELOG.md index b5dd83df1f..8f63e52030 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -86,6 +86,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `[robots-txt]` serves a publisher's `/robots.txt` from the settings. `modules` names what makes the file, in order. `refuse_all` and `allow_all` are Trusted Server's own, and any other name is a module that contributes rules through the new `RobotsTxtContributor` extension point, declared with `with_robots_txt_contributor`. Trusted Server writes the file, so a contributor cannot produce a malformed one, a refusal cannot be loosened, and a path in `always_allow` stays open to every crawler refused the whole site. With `refuse_all` selected every response an adapter finalizes carries `X-Robots-Tag: noindex, nofollow`. A module is asked for the host the request was addressed to, never one a forwarded header names, and is given nothing of the crawler's connection. Its answer is held for that host in the request's key-value store until its refresh is due, and no adapter gives a request a working one, so every adapter asks each module on every request. A module that cannot answer has its last held answer served, and with none held, or with no rule in the whole file, the answer is `503` and never an empty file. On Fastly the file is answered before the Edge Cookie steps and the request filters. Without the section `/robots.txt` reaches the publisher's origin as before. - `ts keys rotate` and `ts keys deactivate` rotate and retire the request signing keys from an operator's machine. They write the two stores the service links as `jwks_store` and `signing_keys` through the `fastly` CLI, with the token from `FASTLY_API_TOKEN`. A private key reaches `fastly` on its standard input and never in its arguments, and a key id no platform can store is refused before anything is written. A command that cannot read the config store writes nothing, and a delete that failed part way can be run again. - One Fastly service can serve several publishers. A `TS__CONFIG__KEY` entry in the `trusted_server_config` store that contains `{host}` names an app-config blob for each host, pushed with `ts config push --key `, and each request reads the blob under its own host. Each host's blob names its permission policy file and pins its SHA-256, and the push keeps each file `--permissions` supplies once, in a shared entry under `permissions_`, written before the blob, which a live push writes a minute after it writes an entry. A host with no blob is answered `421 Misdirected Request`, never from the logical store ID or from another host's blob, a malformed selector is answered `500`, and a sandbox that serves several requests keeps an application for each blob, found only by that blob's key. Without the entry, or with a selector that has no placeholder, the service reads one blob. `scripts/config-by-host-local-test.sh` runs it under Viceroy. - `ts audit generate` writes the `[[fetch]]` and `[[serve]]` entries that run the page changes of the modules it selects on every HTML page, and `trusted-server.example.toml` documents the entries with every middleware the stock modules supply. `trusted-server-modules` lists those middleware with `middleware()` and `middleware_for(...)`. diff --git a/crates/testing/seam-probe/src/lib.rs b/crates/testing/seam-probe/src/lib.rs index 2012e502f1..0e65a0648c 100644 --- a/crates/testing/seam-probe/src/lib.rs +++ b/crates/testing/seam-probe/src/lib.rs @@ -2,11 +2,12 @@ //! part of the integration seam from a vendor crate's position. //! //! One registration carries a browser module, a proxy route, a geo module, -//! an Edge Cookie identity module, a device module and a middleware for -//! each phase of a page, alongside its own configuration block, and the builder -//! adds a request preparer, a demand implementation `[demand]` can name and -//! an ad server implementation `[ad-server]` can name. The round-trip -//! tests in `crates/trusted-server-adapter-axum/tests/seam_probe.rs` and +//! an Edge Cookie identity module, a device module, a robots.txt contributor +//! and a middleware for each phase of a page, alongside its own configuration +//! block, and the builder adds a request preparer, a demand implementation +//! `[demand]` can name and an ad server implementation `[ad-server]` can +//! name. The round-trip tests in +//! `crates/trusted-server-adapter-axum/tests/seam_probe.rs` and //! `crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs` drive //! each of those through a real adapter, so the seam is proven by a caller that //! core does not know about. @@ -46,10 +47,11 @@ use trusted_server_core::integrations::{ use trusted_server_core::middleware::{ AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, TextHandler, }; -use trusted_server_core::module_context::ModuleCall; +use trusted_server_core::module_context::{ModuleCall, ModuleRequest}; use trusted_server_core::platform::{ GeoInfo, PlatformError, PlatformGeo, PlatformResponse, RuntimeServices, }; +use trusted_server_core::robots_txt::{Contribution, RobotsTxtContributor}; use trusted_server_core::settings::{IntegrationConfig, Settings}; use validator::Validate; @@ -59,12 +61,16 @@ pub const SEAM_PROBE_ID: &str = "seam_probe"; /// The name this module is selected by, its crate folder below `crates/`, so /// `[testing] modules = ["seam-probe"]` selects it. Its geo, Edge Cookie and /// device modules carry the same name, so `[geo] module`, `[ec] module` and -/// `[device] module` select each of them as `testing.seam-probe`. +/// `[device] module` select each of them as `testing.seam-probe`, and so does +/// `[robots-txt] modules` its robots.txt contributor. #[must_use] pub fn module_name() -> &'static str { trusted_server_core::module_name!() } +/// The crawler the probe's robots.txt rules are addressed to. +pub const SEAM_PROBE_ROBOTS_AGENT: &str = "SeamProbeBot"; + /// Source label the registry uses in duplicate-id errors. pub const SEAM_PROBE_SOURCE: &str = "trusted-server-integration-seam-probe"; @@ -404,6 +410,42 @@ impl DeviceModule for SeamProbeDevice { } } +/// The probe's robots.txt contributor, declared on the same registration. +/// +/// Its one rule names the request it was asked for, so a test can read back +/// that the contributor was handed the host and the path and not the query. +#[derive(Debug)] +pub struct SeamProbeRobotsTxt; + +impl SeamProbeRobotsTxt { + fn rules(&self, request: ModuleRequest<'_>) -> Contribution { + Contribution::parse(&format!( + "User-Agent: {SEAM_PROBE_ROBOTS_AGENT}\nDisallow: /{}{}?{}", + request.host(), + request.path(), + request.query() + )) + } +} + +#[async_trait::async_trait(?Send)] +impl RobotsTxtContributor for SeamProbeRobotsTxt { + fn refresh(&self) -> core::time::Duration { + core::time::Duration::from_secs(60) + } + + fn fingerprint(&self) -> String { + "seam-probe".to_owned() + } + + async fn contribute( + &self, + call: ModuleCall<'_>, + ) -> Result> { + Ok(call.inject(self, Self::rules)?) + } +} + #[async_trait::async_trait(?Send)] impl PlatformGeo for SeamProbeGeo { async fn lookup( @@ -694,11 +736,13 @@ pub fn register( .with_geo_module(module_name(), Arc::new(SeamProbeGeo::new(config.country))); } { - // Identity and device are declared the same way location is, so one - // module supplies all three and there is no second mechanism. + // Identity, device and robots.txt rules are declared the same way + // location is, so one module supplies all four and there is no second + // mechanism. registration = registration .with_ec_module(module_name(), Arc::new(SeamProbeEc)) - .with_device_module(module_name(), Arc::new(SeamProbeDevice)); + .with_device_module(module_name(), Arc::new(SeamProbeDevice)) + .with_robots_txt_contributor(module_name(), Arc::new(SeamProbeRobotsTxt)); } Ok(Some(registration.build())) diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index a2d3dd0696..f418265fef 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -38,6 +38,7 @@ use trusted_server_core::publisher::{ use trusted_server_core::request_signing::{ handle_trusted_server_discovery, handle_verify_signature, }; +use trusted_server_core::robots_txt::{ROBOTS_TXT_PATH, handle_robots_txt}; use trusted_server_core::settings::Settings; use trusted_server_core::settings_data::{ default_config_key, default_config_store_name, get_settings_from_config_store_with, @@ -461,6 +462,7 @@ enum NamedRouteHandler { FirstPartyClick, FirstPartySign, FirstPartyProxyRebuild, + RobotsTxt, } struct NamedRoute { @@ -469,7 +471,7 @@ struct NamedRoute { handler: NamedRouteHandler, } -fn named_routes() -> [NamedRoute; 14] { +fn named_routes() -> [NamedRoute; 15] { [ NamedRoute { path: "/.well-known/trusted-server.json", @@ -554,9 +556,23 @@ fn named_routes() -> [NamedRoute; 14] { primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartyProxyRebuild, }, + // Only when the settings carry `[robots-txt]`. + NamedRoute { + path: ROBOTS_TXT_PATH, + primary_methods: &[Method::GET, Method::HEAD], + handler: NamedRouteHandler::RobotsTxt, + }, ] } +/// Whether a named route is served for these settings. +fn named_route_is_served(settings: &Settings, handler: NamedRouteHandler) -> bool { + match handler { + NamedRouteHandler::RobotsTxt => settings.robots_txt.is_some(), + _ => true, + } +} + fn named_route_handler( state: Arc, handler: NamedRouteHandler, @@ -631,6 +647,9 @@ fn named_route_handler( NamedRouteHandler::FirstPartyProxyRebuild => { handle_first_party_proxy_rebuild(&state.settings, &services, req).await } + NamedRouteHandler::RobotsTxt => { + handle_robots_txt(&state.settings, &services, &state.registry, req).await + } } }, )) @@ -793,6 +812,9 @@ fn build_router(state: &Arc) -> RouterService { }); for route in named_routes() { + if !named_route_is_served(&state.settings, route.handler) { + continue; + } for method in route.primary_methods { router = router.route( route.path, diff --git a/crates/trusted-server-adapter-axum/src/middleware.rs b/crates/trusted-server-adapter-axum/src/middleware.rs index 9bc715c820..41c226abeb 100644 --- a/crates/trusted-server-adapter-axum/src/middleware.rs +++ b/crates/trusted-server-adapter-axum/src/middleware.rs @@ -103,6 +103,10 @@ pub(crate) fn apply_finalize_headers(settings: &Settings, response: &mut Respons trusted_server_core::response_privacy::apply_response_headers_with_cache_privacy( settings, response, ); + + // Last, so that nothing an origin or an operator's header set can loosen + // a site taken out of search. + trusted_server_core::robots_txt::apply_response_robots_tag(settings, response); } // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-adapter-axum/tests/seam_probe.rs b/crates/trusted-server-adapter-axum/tests/seam_probe.rs index 1ffc1b8355..6a97cd7727 100644 --- a/crates/trusted-server-adapter-axum/tests/seam_probe.rs +++ b/crates/trusted-server-adapter-axum/tests/seam_probe.rs @@ -23,12 +23,13 @@ use trusted_server_core::evidence::OwnedRequestInfo; use trusted_server_core::integrations::{ IntegrationBuilder, IntegrationRegistration, IntegrationRegistry, }; -use trusted_server_core::module_context::{ModuleContext, ModuleRequest}; +use trusted_server_core::module_context::{ModuleCall, ModuleContext, ModuleRequest}; use trusted_server_core::platform::{ BackendNamingPolicy, ClientInfo, DisabledGeo, PlatformBackend, PlatformBackendSpec, PlatformConfigStore, PlatformError, PlatformSecretStore, RuntimeServices, StoreId, StoreName, UnavailableHttpClient, UnavailableKvStore, }; +use trusted_server_core::robots_txt::{Contribution, RobotsTxtContributor}; use trusted_server_core::settings::Settings; use trusted_server_core::tsjs::tsjs_script_src; use trusted_server_core::tsjs_bundle::{JsModulePart, compile_time_parts}; @@ -258,6 +259,352 @@ async fn proxy_route_reports_the_modules_geo_and_that_the_preparer_ran() { ); } +/// `[robots-txt] modules` naming the probe serves the rules the probe gives, +/// asked for the request's host and path and never its query, because the +/// answer is held and served to every crawler. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn robots_txt_selection_naming_a_module_serves_the_rules_it_gives() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = ["testing.seam-probe", "allow_all"] + sitemap = "https://test-publisher.example.com/sitemap.xml" + {PROBE_BLOCK} + "# + )); + let mut service = service_with(settings, &[seam_probe::builder()]); + + let response = get(&mut service, "/robots.txt?crawler=one").await; + + assert_eq!( + response.status().as_u16(), + 200, + "the file should be served here and not by the origin" + ); + assert_eq!( + response + .headers() + .get("content-type") + .and_then(|value| value.to_str().ok()), + Some("text/plain; charset=utf-8") + ); + let file = body_text(response).await; + let agent = seam_probe::SEAM_PROBE_ROBOTS_AGENT; + assert!( + file.starts_with(&format!("User-Agent: {agent}\nDisallow: /")), + "the probe's rules should come first, as `modules` orders them: {file}" + ); + assert!( + file.contains("/robots.txt?\n"), + "the probe should be asked for the path with no query: {file}" + ); + assert!( + !file.contains("crawler=one"), + "nothing one crawler put in the address should reach a held answer: {file}" + ); + assert!( + file.ends_with( + "User-Agent: *\nAllow: /\n\nSitemap: https://test-publisher.example.com/sitemap.xml\n" + ), + "core's own rules and the sitemap should follow: {file}" + ); +} + +/// With `refuse_all` selected the file is the refusal, and every response +/// this adapter finalizes says not to index, the response of another route +/// included. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn refusing_every_crawler_tags_every_response() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = "refuse_all" + always_allow = ["/ads.txt"] + {PROBE_BLOCK} + "# + )); + let mut service = service_with(settings, &[seam_probe::builder()]); + let tag = |response: &axum::response::Response| { + response + .headers() + .get("x-robots-tag") + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + }; + + let file = get(&mut service, "/robots.txt").await; + + assert_eq!(file.status().as_u16(), 200, "the refusal should be served"); + assert_eq!(tag(&file).as_deref(), Some("noindex, nofollow")); + assert_eq!( + body_text(file).await, + "User-agent: *\nAllow: /ads.txt\nDisallow: /\n" + ); + + let other = get(&mut service, seam_probe::SEAM_PROBE_REPORT_PATH).await; + + assert_eq!( + tag(&other).as_deref(), + Some("noindex, nofollow"), + "no route may be the one page a crawler is allowed to index" + ); +} + +/// A name `[robots-txt] modules` gives that nothing the deployment runs +/// supplies stops the adapter at startup, so the file does not fail on the +/// first crawler to ask for it. +#[test] +fn robots_txt_selection_naming_a_module_that_is_not_running_fails_at_startup() { + let settings = settings_with( + r#" + [robots-txt] + modules = ["testing.seam-probe"] + "#, + ); + + let error = TrustedServerApp::routes_with_registrations(settings, &[seam_probe::builder()]) + .err() + .expect("should refuse a contributor whose module no section selects"); + let message = format!("{error:?}"); + + assert!( + message.contains("`[robots-txt] modules` names `testing.seam-probe`"), + "should name the selection: {message}" + ); + assert!( + message.contains("refuse_all") && message.contains("allow_all"), + "should say what can be named: {message}" + ); +} + +/// The methods this adapter registers for `/robots.txt` under `settings`. +fn robots_txt_methods(settings: Settings) -> Vec { + TrustedServerApp::routes_with_registrations(settings, &[seam_probe::builder()]) + .expect("should build the routes") + .routes() + .into_iter() + .filter(|route| route.path() == "/robots.txt") + .map(|route| route.method().to_string()) + .collect() +} + +/// With no `[robots-txt]` section the path is the publisher's, and the probe +/// being on offer changes nothing. With the section the path is this +/// server's for `GET` and `HEAD`. +#[test] +fn robots_txt_is_this_servers_only_when_the_settings_carry_the_section() { + assert_eq!( + robots_txt_methods(settings_with(PROBE_BLOCK)), + Vec::::new(), + "without the section no route of this server names the path" + ); + + let methods = robots_txt_methods(settings_with(&format!( + r#" + [robots-txt] + modules = "refuse_all" + {PROBE_BLOCK} + "# + ))); + for method in ["GET", "HEAD"] { + assert!( + methods.iter().any(|registered| registered == method), + "{method} /robots.txt should be answered here: {methods:?}" + ); + } +} + +/// Sends one request with `method` and `headers` and returns the response. +async fn send( + service: &mut EdgeZeroAxumService, + method: &str, + uri: &str, + headers: &[(&str, &str)], +) -> axum::response::Response { + let mut builder = Request::builder().method(method).uri(uri); + for &(name, value) in headers { + builder = builder.header(name, value); + } + let request = builder + .body(AxumBody::empty()) + .expect("should build request"); + + service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should respond") +} + +/// The value of the header `name`, as text. +fn header_text(response: &axum::response::Response, name: &str) -> Option { + response + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) +} + +/// A `HEAD` for the file is answered here with the headers a `GET` is given +/// and no body. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_head_of_robots_txt_is_answered_with_the_headers_and_no_body() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = "refuse_all" + {PROBE_BLOCK} + "# + )); + let mut service = service_with(settings, &[seam_probe::builder()]); + + let response = send(&mut service, "HEAD", "/robots.txt", &[]).await; + + assert_eq!(response.status().as_u16(), 200); + assert_eq!( + header_text(&response, "content-type").as_deref(), + Some("text/plain; charset=utf-8") + ); + assert_eq!( + header_text(&response, "cache-control").as_deref(), + Some("public, max-age=3600") + ); + assert!( + body_text(response).await.is_empty(), + "a HEAD carries no body" + ); +} + +/// The id of the module that answers once. +const ANSWERS_ONCE_ID: &str = "robots-txt-answers-once"; + +/// A robots.txt contributor that answers the first time it is asked and fails +/// every time after, whose answer is due for a refresh at once. +#[derive(Debug, Default)] +struct AnswersOnce { + asked: std::sync::atomic::AtomicUsize, +} + +#[async_trait::async_trait(?Send)] +impl RobotsTxtContributor for AnswersOnce { + fn refresh(&self) -> std::time::Duration { + std::time::Duration::ZERO + } + + fn fingerprint(&self) -> String { + "answers-once".to_owned() + } + + async fn contribute( + &self, + _call: ModuleCall<'_>, + ) -> Result> { + if self.asked.fetch_add(1, std::sync::atomic::Ordering::SeqCst) == 0 { + Ok(Contribution::parse("User-Agent: OnceBot\nDisallow: /")) + } else { + Err(Report::new(TrustedServerError::Proxy { + message: "the rules service is down".to_owned(), + })) + } + } +} + +fn register_answers_once( + _settings: &Settings, +) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(ANSWERS_ONCE_ID) + .with_robots_txt_contributor( + "robots-txt.answers-once", + Arc::new(AnswersOnce::default()), + ) + .build(), + )) +} + +fn accept_answers_once(_settings: &Settings) -> Result> { + Ok(true) +} + +/// Settings whose file is made by the module that answers once. +const ANSWERS_ONCE_BLOCK: &str = r#" + [robots-txt] + modules = ["answers-once"] + + [testing] + modules = ["robots-txt-answers-once"] +"#; + +/// This adapter holds no answer, so every request for the file asks each +/// module, and a module that fails with nothing held makes the file a `503` +/// that tells a crawler to come back in ten minutes and is not stored. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn with_nothing_held_a_module_that_fails_makes_the_file_unavailable() { + let builder = IntegrationBuilder::new( + ANSWERS_ONCE_ID, + ANSWERS_ONCE_ID, + register_answers_once, + accept_answers_once, + ) + .with_module_name("testing.robots-txt-answers-once"); + let mut service = service_with(settings_with(ANSWERS_ONCE_BLOCK), &[builder]); + + let first = get(&mut service, "/robots.txt").await; + assert_eq!(first.status().as_u16(), 200); + assert_eq!(body_text(first).await, "User-Agent: OnceBot\nDisallow: /\n"); + + let second = get(&mut service, "/robots.txt").await; + assert_eq!( + second.status().as_u16(), + 503, + "nothing is held here, so the module is asked again and its failure fails the file" + ); + assert_eq!(header_text(&second, "retry-after").as_deref(), Some("600")); + assert_eq!( + header_text(&second, "cache-control").as_deref(), + Some("no-store") + ); +} + +/// This adapter leaves `Forwarded` and `X-Forwarded-Host` in place, and the +/// probe's rule names the host it was asked for, so the file shows that a +/// claimed host never reaches a module. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_forwarded_host_never_reaches_a_robots_txt_module() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = ["testing.seam-probe"] + {PROBE_BLOCK} + "# + )); + let mut service = service_with(settings, &[seam_probe::builder()]); + + let response = send( + &mut service, + "GET", + "/robots.txt", + &[ + ("host", "test-publisher.example.com"), + ("x-forwarded-host", "elsewhere.example"), + ("forwarded", "host=elsewhere.example"), + ], + ) + .await; + let file = body_text(response).await; + + assert!( + file.contains("Disallow: /test-publisher.example.com/robots.txt?"), + "the probe should be asked for the host addressed: {file}" + ); + assert!( + !file.contains("elsewhere.example"), + "a claimed host should never reach the file: {file}" + ); +} + /// A request prepares exactly once whether it is served by a named route or by /// the fallback, so the invariant holds across the whole route table rather /// than only on the path the probe's own proxy route sits on. diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index a56c5f8a9e..f0b6942e62 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -42,6 +42,7 @@ use trusted_server_core::publisher::{ use trusted_server_core::request_signing::{ handle_trusted_server_discovery, handle_verify_signature, }; +use trusted_server_core::robots_txt::{ROBOTS_TXT_PATH, handle_robots_txt}; use trusted_server_core::settings::Settings; use crate::middleware::{FinalizeResponseMiddleware, SanitizeRequestMiddleware}; @@ -825,6 +826,20 @@ fn build_router(state: &Arc) -> RouterService { }); router = router.route(DATA_PAGE_PATH, Method::GET, data); + // Only when the settings carry `[robots-txt]`. + if state.settings.robots_txt.is_some() { + let robots_txt = make_handler(Arc::clone(&state), |s, services, req| async move { + handle_robots_txt(&s.settings, &services, &s.registry, req).await + }); + for method in publisher_fallback_methods() { + if matches!(method, Method::GET | Method::HEAD) { + router = router.route(ROBOTS_TXT_PATH, method, robots_txt.clone()); + } else { + router = router.route(ROBOTS_TXT_PATH, method, fallback.clone()); + } + } + } + for method in publisher_fallback_methods() { router = router.route("/", method.clone(), fallback.clone()); router = router.route("/{*rest}", method, fallback.clone()); diff --git a/crates/trusted-server-adapter-cloudflare/src/middleware.rs b/crates/trusted-server-adapter-cloudflare/src/middleware.rs index 3a58cd63d5..e8cc4e414e 100644 --- a/crates/trusted-server-adapter-cloudflare/src/middleware.rs +++ b/crates/trusted-server-adapter-cloudflare/src/middleware.rs @@ -114,6 +114,10 @@ pub(crate) fn apply_finalize_headers( trusted_server_core::response_privacy::apply_response_headers_with_cache_privacy( settings, response, ); + + // Last, so that nothing an origin or an operator's header set can loosen + // a site taken out of search. + trusted_server_core::robots_txt::apply_response_robots_tag(settings, response); } // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index 79877360d4..ab07e52833 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -874,3 +874,401 @@ async fn nextjs_auction_output_holds_until_the_structural_body_close() { "should not leak generated placeholders: {html}" ); } + +// --------------------------------------------------------------------------- +// robots.txt +// --------------------------------------------------------------------------- + +/// The route test settings with a `[robots-txt]` section that refuses every +/// crawler and keeps one path open. +fn router_refusing_every_crawler() -> RouterService { + let settings = Settings::from_toml( + r#" + [publisher] + domain = "test-publisher.example.com" + cookie_domain = ".test-publisher.example.com" + origin_url = "https://origin.test-publisher.example.com" + proxy_secret = "route-test-proxy-secret" + + [ec] + module = "hmac" + + [ec.hmac] + passphrase = "test-secret-key-32-bytes-minimum" + + [geo] + assume_single_jurisdiction = true + + [robots-txt] + modules = "refuse_all" + always_allow = ["/ads.txt"] + "#, + ) + .expect("should parse route test settings") + // A loaded domain carries its policy. + .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()); + + TrustedServerApp::routes_with_settings(settings) + .expect("should build router from test settings") +} + +fn robots_request(method: &str, path: &str) -> Request { + request_builder() + .method(method) + .uri(path) + .body(edgezero_core::body::Body::empty()) + .expect("should build request") +} + +fn robots_tag(response: &Response) -> Option { + response + .headers() + .get("x-robots-tag") + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn refusing_every_crawler_serves_the_refusal_and_tags_every_response() { + let file = route( + router_refusing_every_crawler(), + robots_request("GET", "/robots.txt"), + ) + .await; + + assert_eq!( + file.status().as_u16(), + 200, + "the refusal should be served here" + ); + assert_eq!(robots_tag(&file).as_deref(), Some("noindex, nofollow")); + let body = file.into_body().into_bytes().unwrap_or_default(); + assert_eq!(&body[..], b"User-agent: *\nAllow: /ads.txt\nDisallow: /\n"); + + let other = route( + router_refusing_every_crawler(), + robots_request("GET", "/.well-known/trusted-server.json"), + ) + .await; + + assert_eq!( + robots_tag(&other).as_deref(), + Some("noindex, nofollow"), + "no route may be the one page a crawler is allowed to index" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_head_of_robots_txt_is_answered_here_with_no_body() { + let response = route( + router_refusing_every_crawler(), + robots_request("HEAD", "/robots.txt"), + ) + .await; + + assert_eq!( + response.status().as_u16(), + 200, + "a HEAD should be answered here as a GET is" + ); + assert_eq!( + response + .headers() + .get("content-type") + .and_then(|value| value.to_str().ok()), + Some("text/plain; charset=utf-8") + ); + let body = response.into_body().into_bytes().unwrap_or_default(); + assert!(body.is_empty(), "a HEAD carries no body"); +} + +/// A method that is neither `GET` nor `HEAD` is the publisher's, as it is on +/// any other path, so it is not answered with the file. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn another_method_on_robots_txt_is_not_answered_with_the_file() { + let response = route( + router_refusing_every_crawler(), + robots_request("POST", "/robots.txt"), + ) + .await; + + let body = response.into_body().into_bytes().unwrap_or_default(); + assert!( + !body.starts_with(b"User-agent:"), + "a POST should reach the publisher fallback and not be given the file" + ); +} + +#[test] +fn robots_txt_is_this_servers_only_when_the_settings_carry_the_section() { + let registered = |router: RouterService| -> Vec<(String, String)> { + router + .routes() + .into_iter() + .map(|route| (route.method().to_string(), route.path().to_string())) + .collect() + }; + + let without = registered(test_router()); + assert!( + !without.iter().any(|(_, path)| path == "/robots.txt"), + "without the section the path is the publisher's: {without:?}" + ); + + let with = registered(router_refusing_every_crawler()); + for method in ["GET", "HEAD"] { + assert!( + with.iter() + .any(|(registered, path)| registered == method && path == "/robots.txt"), + "{method} /robots.txt should be answered here: {with:?}" + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_response_carries_no_blanket_rule_without_the_section() { + let response = route( + test_router(), + robots_request("GET", "/.well-known/trusted-server.json"), + ) + .await; + + assert_eq!(robots_tag(&response), None); +} + +/// A module's rules served through this adapter's router, held in the +/// key-value store the runtime hands the application. +mod robots_txt_from_a_module { + use std::collections::HashMap; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex}; + use std::time::Duration; + + use bytes::Bytes; + use edgezero_core::http::Request; + use edgezero_core::key_value_store::{KvError, KvHandle, KvPage, KvStore}; + use edgezero_core::router::RouterService; + use edgezero_core::store_registry::KvRegistry; + use error_stack::Report; + use trusted_server_adapter_cloudflare::app::TrustedServerApp; + use trusted_server_core::error::TrustedServerError; + use trusted_server_core::integrations::{IntegrationBuilder, IntegrationRegistration}; + use trusted_server_core::module_context::ModuleCall; + use trusted_server_core::robots_txt::{Contribution, RobotsTxtContributor}; + use trusted_server_core::settings::Settings; + + use super::robots_request; + + /// The id of the module that answers once. + const ANSWERS_ONCE_ID: &str = "robots-txt-answers-once"; + + /// A robots.txt contributor that answers the first time it is asked and + /// fails every time after, whose answer is due for a refresh at once. + #[derive(Debug, Default)] + struct AnswersOnce { + asked: AtomicUsize, + } + + #[async_trait::async_trait(?Send)] + impl RobotsTxtContributor for AnswersOnce { + fn refresh(&self) -> Duration { + Duration::ZERO + } + + fn fingerprint(&self) -> String { + "answers-once".to_owned() + } + + async fn contribute( + &self, + _call: ModuleCall<'_>, + ) -> Result> { + if self.asked.fetch_add(1, Ordering::SeqCst) == 0 { + Ok(Contribution::parse("User-Agent: OnceBot\nDisallow: /")) + } else { + Err(Report::new(TrustedServerError::Proxy { + message: "the rules service is down".to_owned(), + })) + } + } + } + + fn register_answers_once( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(ANSWERS_ONCE_ID) + .with_robots_txt_contributor( + "robots-txt.answers-once", + Arc::new(AnswersOnce::default()), + ) + .build(), + )) + } + + fn accept_answers_once(_settings: &Settings) -> Result> { + Ok(true) + } + + /// The route test settings with a file made by the module that answers + /// once. + fn router_with_a_module_that_answers_once() -> RouterService { + let settings = Settings::from_toml( + r#" + [publisher] + domain = "test-publisher.example.com" + cookie_domain = ".test-publisher.example.com" + origin_url = "https://origin.test-publisher.example.com" + proxy_secret = "route-test-proxy-secret" + + [ec] + module = "hmac" + + [ec.hmac] + passphrase = "test-secret-key-32-bytes-minimum" + + [geo] + assume_single_jurisdiction = true + + [robots-txt] + modules = ["answers-once"] + + [testing] + modules = ["robots-txt-answers-once"] + "#, + ) + .expect("should parse route test settings") + // A loaded domain carries its policy. + .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()); + let builder = IntegrationBuilder::new( + ANSWERS_ONCE_ID, + ANSWERS_ONCE_ID, + register_answers_once, + accept_answers_once, + ) + .with_module_name("testing.robots-txt-answers-once"); + + TrustedServerApp::routes_with_registrations(settings, &[builder]) + .expect("should build router from test settings") + } + + /// A key-value store kept in memory for one test, so what one request + /// writes is what the next one reads. + #[derive(Default)] + struct MemoryStore { + entries: Mutex>, + } + + #[async_trait::async_trait(?Send)] + impl KvStore for MemoryStore { + async fn delete(&self, key: &str) -> Result<(), KvError> { + self.entries + .lock() + .expect("should lock the store") + .remove(key); + Ok(()) + } + + async fn get_bytes(&self, key: &str) -> Result, KvError> { + Ok(self + .entries + .lock() + .expect("should lock the store") + .get(key) + .cloned()) + } + + async fn list_keys_page( + &self, + _prefix: &str, + _cursor: Option<&str>, + _limit: usize, + ) -> Result { + Ok(KvPage::default()) + } + + async fn put_bytes(&self, key: &str, value: Bytes) -> Result<(), KvError> { + self.entries + .lock() + .expect("should lock the store") + .insert(key.to_owned(), value); + Ok(()) + } + + async fn put_bytes_with_ttl( + &self, + key: &str, + value: Bytes, + _ttl: Duration, + ) -> Result<(), KvError> { + self.put_bytes(key, value).await + } + } + + /// A crawler's `GET` for the file, carrying `store` as the key-value store + /// the runtime hands the application, when there is one. + fn robots_get(store: Option<&Arc>) -> Request { + let mut request = robots_request("GET", "/robots.txt"); + if let Some(store) = store { + let handle = KvHandle::new(Arc::clone(store) as Arc); + request.extensions_mut().insert(KvRegistry::single_id( + "trusted_server_kv".to_owned(), + handle, + )); + } + request + } + + /// With the store the runtime hands the application, a module's answer + /// is held, so when the module next fails the file is made from the + /// answer it gave before. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn a_held_answer_is_served_when_its_module_fails() { + let router = router_with_a_module_that_answers_once(); + let store = Arc::new(MemoryStore::default()); + + for asked in ["answers", "fails"] { + let response = router + .oneshot(robots_get(Some(&store))) + .await + .expect("should route request"); + + assert_eq!(response.status().as_u16(), 200, "the module {asked}"); + let body = response.into_body().into_bytes().unwrap_or_default(); + assert_eq!( + &body[..], + b"User-Agent: OnceBot\nDisallow: /\n", + "the module {asked}" + ); + } + } + + /// With no store from the runtime nothing is held, so a module that fails + /// makes the file a `503` that tells a crawler to come back in ten + /// minutes and is not stored. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn with_nothing_held_a_module_that_fails_makes_the_file_unavailable() { + let router = router_with_a_module_that_answers_once(); + + let first = router + .oneshot(robots_get(None)) + .await + .expect("should route request"); + assert_eq!(first.status().as_u16(), 200); + + let second = router + .oneshot(robots_get(None)) + .await + .expect("should route request"); + assert_eq!(second.status().as_u16(), 503); + let header = |name: &str| { + second + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + }; + assert_eq!(header("retry-after").as_deref(), Some("600")); + assert_eq!(header("cache-control").as_deref(), Some("no-store")); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index aa8903bf17..62711b20fb 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -138,6 +138,7 @@ use trusted_server_core::publisher::{ use trusted_server_core::request_signing::{ handle_trusted_server_discovery, handle_verify_signature, }; +use trusted_server_core::robots_txt::{ROBOTS_TXT_PATH, handle_robots_txt}; use trusted_server_core::settings::{ProxyAssetRoute, Settings}; use trusted_server_core::settings_data::{ DEFAULT_CONFIG_STORE_ID, get_settings_from_config_store_with, @@ -834,6 +835,17 @@ async fn execute_named( return Ok(http_error(&report)); } + // A crawler asking for robots.txt is answered once the modules have + // prepared the request, before the Edge Cookie steps and the request + // filters, so nothing is read from its cookies or written back, and no + // filter's answer stands in for the file. + if matches!(handler, NamedRouteHandler::RobotsTxt) { + let response = handle_robots_txt(&state.settings, &services, &state.registry, req) + .await + .unwrap_or_else(|error| http_error(&error)); + return Ok(response); + } + let mut ec = build_ec_request_state(&state.settings, &services, &req).await; // EcContext creation errors short-circuit before filters, mirroring legacy: // the legacy path returns its error response before running filter_request. @@ -977,6 +989,9 @@ async fn run_named_route( NamedRouteHandler::FirstPartyProxyRebuild => { handle_first_party_proxy_rebuild(&state.settings, services, req).await } + NamedRouteHandler::RobotsTxt => { + unreachable!("robots.txt is answered before EC setup") + } } } @@ -1330,6 +1345,7 @@ enum NamedRouteHandler { FirstPartyClick, FirstPartySign, FirstPartyProxyRebuild, + RobotsTxt, } struct NamedRoute { @@ -1448,8 +1464,22 @@ const NAMED_ROUTES: &[NamedRoute] = &[ primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartyProxyRebuild, }, + // Only when the settings carry `[robots-txt]`. + NamedRoute { + path: ROBOTS_TXT_PATH, + primary_methods: &[Method::GET, Method::HEAD], + handler: NamedRouteHandler::RobotsTxt, + }, ]; +/// Whether a named route is served for these settings. +fn named_route_is_served(settings: &Settings, handler: NamedRouteHandler) -> bool { + match handler { + NamedRouteHandler::RobotsTxt => settings.robots_txt.is_some(), + _ => true, + } +} + fn named_route_handler( state: Arc, handler: NamedRouteHandler, @@ -1512,6 +1542,9 @@ impl TrustedServerApp { // non-primary publisher fallback method is registered from the same // row. Adding a named route now requires editing only this table. for route in NAMED_ROUTES { + if !named_route_is_served(&state.settings, route.handler) { + continue; + } for method in route.primary_methods { router = router.route( route.path, diff --git a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs index e564d6a076..a1211f9b32 100644 --- a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs +++ b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs @@ -16,16 +16,20 @@ use error_stack::Report; use futures::executor::block_on; use trusted_server_core::error::TrustedServerError; use trusted_server_core::evidence::OwnedRequestInfo; -use trusted_server_core::integrations::{IntegrationBuilder, IntegrationRegistration}; -use trusted_server_core::module_context::{ModuleContext, ModuleRequest}; +use trusted_server_core::integrations::{ + IntegrationBuilder, IntegrationRegistration, IntegrationRequestFilter, RequestFilterDecision, + RequestFilterEffects, RequestFilterInput, +}; +use trusted_server_core::module_context::{ModuleCall, ModuleContext, ModuleRequest}; +use trusted_server_core::robots_txt::{Contribution, RobotsTxtContributor}; use trusted_server_core::settings::Settings; use trusted_server_core::tsjs::tsjs_script_src; use trusted_server_core::tsjs_bundle::{JsModulePart, compile_time_parts}; use trusted_server_integration_seam_probe as seam_probe; use super::{ - AppState, TrustedServerApp, build_finalize_services, build_state_from_settings, - build_state_with_registrations, register_integrations, + AppState, EcFinalizeState, TrustedServerApp, build_finalize_services, + build_state_from_settings, build_state_with_registrations, register_integrations, }; /// Largest response body these tests read, ample for the served script. @@ -210,6 +214,372 @@ fn proxy_route_reports_the_modules_geo_and_that_the_preparer_ran() { ); } +/// `[robots-txt] modules` naming the probe serves the rules the probe gives, +/// asked for the request's host and path and never its query, because the +/// answer is held and served to every crawler. +#[test] +fn robots_txt_selection_naming_a_module_serves_the_rules_it_gives() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = ["testing.seam-probe", "allow_all"] + {HMAC_BLOCK} + {PROBE_BLOCK} + "# + )); + let router = router_with(settings, &[seam_probe::builder()]); + + let response = get(&router, "/robots.txt?crawler=one", None); + + assert_eq!( + response.status().as_u16(), + 200, + "the file should be served here and not by the origin" + ); + assert!( + response.headers().get("x-robots-tag").is_none(), + "a file made from contributors puts no blanket rule on responses" + ); + let file = body_text(response); + let agent = seam_probe::SEAM_PROBE_ROBOTS_AGENT; + assert!( + file.starts_with(&format!("User-Agent: {agent}\nDisallow: /")), + "the probe's rules should come first, as `modules` orders them: {file}" + ); + assert!( + file.contains("/robots.txt?\n"), + "the probe should be asked for the path with no query: {file}" + ); + assert!( + !file.contains("crawler=one"), + "nothing one crawler put in the address should reach a held answer: {file}" + ); + assert!( + file.ends_with("User-Agent: *\nAllow: /\n"), + "core's own rules should follow: {file}" + ); +} + +/// With `refuse_all` selected the file is the refusal, and every response +/// this adapter finalizes says not to index, the response of another route +/// included. +#[test] +fn refusing_every_crawler_tags_every_response() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = "refuse_all" + always_allow = ["/ads.txt"] + {HMAC_BLOCK} + {PROBE_BLOCK} + "# + )); + let router = router_with(settings, &[seam_probe::builder()]); + let tag = |response: &Response| { + response + .headers() + .get("x-robots-tag") + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + }; + + let file = get(&router, "/robots.txt", None); + + assert_eq!(file.status().as_u16(), 200, "the refusal should be served"); + assert_eq!(tag(&file).as_deref(), Some("noindex, nofollow")); + assert_eq!( + body_text(file), + "User-agent: *\nAllow: /ads.txt\nDisallow: /\n" + ); + + let other = get(&router, seam_probe::SEAM_PROBE_REPORT_PATH, None); + + assert_eq!( + tag(&other).as_deref(), + Some("noindex, nofollow"), + "no route may be the one page a crawler is allowed to index" + ); +} + +/// The methods this adapter registers for `/robots.txt`. +fn robots_txt_methods(router: &RouterService) -> Vec { + router + .routes() + .into_iter() + .filter(|route| route.path() == "/robots.txt") + .map(|route| route.method().to_string()) + .collect() +} + +/// With no `[robots-txt]` section the path is the publisher's, and the probe +/// being on offer changes nothing. With the section the path is this +/// server's for `GET` and `HEAD`. +#[test] +fn robots_txt_is_this_servers_only_when_the_settings_carry_the_section() { + let without = router_with( + settings_with(&format!("{HMAC_BLOCK}{PROBE_BLOCK}")), + &[seam_probe::builder()], + ); + + assert_eq!( + robots_txt_methods(&without), + Vec::::new(), + "without the section no route of this server names the path" + ); + let other = get(&without, seam_probe::SEAM_PROBE_REPORT_PATH, None); + assert!( + other.headers().get("x-robots-tag").is_none(), + "no blanket rule is placed on responses without the section" + ); + + let with = router_with( + settings_with(&format!( + r#" + [robots-txt] + modules = "refuse_all" + {HMAC_BLOCK} + {PROBE_BLOCK} + "# + )), + &[seam_probe::builder()], + ); + + let methods = robots_txt_methods(&with); + for method in ["GET", "HEAD"] { + assert!( + methods.iter().any(|registered| registered == method), + "{method} /robots.txt should be answered here: {methods:?}" + ); + } +} + +/// The id of a module whose request filter answers every request with a +/// challenge, as bot protection does. +const CHALLENGE_ID: &str = "robots-txt-challenge"; + +/// The module name that module is selected by. +const CHALLENGE_MODULE: &str = "testing.robots-txt-challenge"; + +/// Answers every request with a `403` challenge. +struct Challenging; + +#[async_trait::async_trait(?Send)] +impl IntegrationRequestFilter for Challenging { + fn integration_id(&self) -> &'static str { + CHALLENGE_ID + } + + async fn filter_request( + &self, + _input: RequestFilterInput<'_>, + ) -> Result> { + let response = edgezero_core::http::response_builder() + .status(403) + .body(Body::from("a challenge")) + .expect("should build the challenge"); + Ok(RequestFilterDecision::Respond { + response: Box::new(response), + effects: RequestFilterEffects::default(), + }) + } +} + +fn register_challenge( + _settings: &Settings, +) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(CHALLENGE_ID) + .with_request_filter(Arc::new(Challenging)) + .build(), + )) +} + +fn accept_challenge(_settings: &Settings) -> Result> { + Ok(true) +} + +/// A crawler asking for robots.txt is answered before the Edge Cookie steps +/// and the request filters, so a filter that answers every other request with +/// a challenge never stands in for the file, and no Edge Cookie state rides +/// on the answer. +#[test] +fn robots_txt_is_answered_before_the_edge_cookie_steps_and_the_filters() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = "refuse_all" + {HMAC_BLOCK} + + [testing] + modules = ["robots-txt-challenge"] + "# + )); + let challenge = IntegrationBuilder::new( + CHALLENGE_ID, + CHALLENGE_ID, + register_challenge, + accept_challenge, + ) + .with_module_name(CHALLENGE_MODULE); + let router = router_with(settings, &[challenge]); + + let other = get(&router, "/.well-known/trusted-server.json", None); + assert_eq!( + other.status().as_u16(), + 403, + "the filter should answer another named route" + ); + + let file = get(&router, "/robots.txt", None); + assert_eq!( + file.status().as_u16(), + 200, + "the file should be served, not the filter's challenge" + ); + assert!( + file.extensions().get::().is_none(), + "no Edge Cookie state should ride on the file" + ); + assert_eq!(body_text(file), "User-agent: *\nDisallow: /\n"); +} + +/// Sends one request with `method` and returns the response. +fn send(router: &RouterService, method: Method, path: &str) -> Response { + let request = request_builder() + .method(method) + .uri(format!("https://test-publisher.com{path}")) + .body(Body::empty()) + .expect("should build request"); + block_on(router.oneshot(request)).expect("should route request") +} + +/// The value of the header `name`, as text. +fn header_text<'r>(response: &'r Response, name: &str) -> Option<&'r str> { + response + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) +} + +/// A `HEAD` for the file is answered here with the headers a `GET` is given +/// and no body. +#[test] +fn a_head_of_robots_txt_is_answered_with_the_headers_and_no_body() { + let router = router_with( + settings_with(&format!( + r#" + [robots-txt] + modules = "refuse_all" + {HMAC_BLOCK} + {PROBE_BLOCK} + "# + )), + &[seam_probe::builder()], + ); + + let response = send(&router, Method::HEAD, "/robots.txt"); + + assert_eq!(response.status().as_u16(), 200); + assert_eq!( + header_text(&response, "content-type"), + Some("text/plain; charset=utf-8") + ); + assert_eq!( + header_text(&response, "cache-control"), + Some("public, max-age=3600") + ); + assert!(body_text(response).is_empty(), "a HEAD carries no body"); +} + +/// The id of the module that answers once. +const ANSWERS_ONCE_ID: &str = "robots-txt-answers-once"; + +/// A robots.txt contributor that answers the first time it is asked and fails +/// every time after, whose answer is due for a refresh at once. +#[derive(Debug, Default)] +struct AnswersOnce { + asked: std::sync::atomic::AtomicUsize, +} + +#[async_trait::async_trait(?Send)] +impl RobotsTxtContributor for AnswersOnce { + fn refresh(&self) -> std::time::Duration { + std::time::Duration::ZERO + } + + fn fingerprint(&self) -> String { + "answers-once".to_owned() + } + + async fn contribute( + &self, + _call: ModuleCall<'_>, + ) -> Result> { + if self.asked.fetch_add(1, std::sync::atomic::Ordering::SeqCst) == 0 { + Ok(Contribution::parse("User-Agent: OnceBot\nDisallow: /")) + } else { + Err(Report::new(TrustedServerError::Proxy { + message: "the rules service is down".to_owned(), + })) + } + } +} + +fn register_answers_once( + _settings: &Settings, +) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(ANSWERS_ONCE_ID) + .with_robots_txt_contributor( + "robots-txt.answers-once", + Arc::new(AnswersOnce::default()), + ) + .build(), + )) +} + +fn accept_answers_once(_settings: &Settings) -> Result> { + Ok(true) +} + +/// This adapter holds no answer, so every request for the file asks each +/// module, and a module that fails with nothing held makes the file a `503` +/// that tells a crawler to come back in ten minutes and is not stored. +#[test] +fn with_nothing_held_a_module_that_fails_makes_the_file_unavailable() { + let settings = settings_with(&format!( + r#" + [robots-txt] + modules = ["answers-once"] + {HMAC_BLOCK} + + [testing] + modules = ["robots-txt-answers-once"] + "# + )); + let builder = IntegrationBuilder::new( + ANSWERS_ONCE_ID, + ANSWERS_ONCE_ID, + register_answers_once, + accept_answers_once, + ) + .with_module_name("testing.robots-txt-answers-once"); + let router = router_with(settings, &[builder]); + + let first = get(&router, "/robots.txt", None); + assert_eq!(first.status().as_u16(), 200); + assert_eq!(body_text(first), "User-Agent: OnceBot\nDisallow: /\n"); + + let second = get(&router, "/robots.txt", None); + assert_eq!( + second.status().as_u16(), + 503, + "nothing is held here, so the module is asked again and its failure fails the file" + ); + assert_eq!(header_text(&second, "retry-after"), Some("600")); + assert_eq!(header_text(&second, "cache-control"), Some("no-store")); +} + /// A request prepares exactly once whether a named route or the fallback /// serves it. #[test] diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 2bab1b58a3..b2d38884ec 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -161,6 +161,8 @@ pub(crate) fn geo_allowed_for_response(response: &Response) -> bool { /// 5. `settings.response_headers` — operator-configured overrides, except the /// cache-controlling headers are skipped on uncacheable (`private`/`no-store`) /// responses so operators cannot re-enable shared caching for per-user payloads +/// 6. `X-Robots-Tag: noindex, nofollow` when `[robots-txt] modules` selects +/// `refuse_all` pub(crate) fn apply_finalize_headers( settings: &Settings, geo_info: Option<&GeoInfo>, @@ -196,6 +198,10 @@ pub(crate) fn apply_finalize_headers( trusted_server_core::response_privacy::apply_response_headers_with_cache_privacy( settings, response, ); + + // Last, so that nothing an origin or an operator's header set can loosen + // a site taken out of search. + trusted_server_core::robots_txt::apply_response_robots_tag(settings, response); } /// Forces cookie-bearing responses to stay private to shared caches. diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 5b9c8e4b66..83460f2a8a 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -47,6 +47,7 @@ use trusted_server_core::publisher::{ use trusted_server_core::request_signing::{ handle_trusted_server_discovery, handle_verify_signature, }; +use trusted_server_core::robots_txt::{ROBOTS_TXT_PATH, handle_robots_txt}; use trusted_server_core::settings::Settings; #[cfg(all(feature = "spin", target_arch = "wasm32"))] use trusted_server_core::settings_data::{default_config_key, default_secret_store_name}; @@ -1098,6 +1099,29 @@ fn build_router(state: &Arc) -> RouterService { } } } + // Only when the settings carry `[robots-txt]`. + if state.settings.robots_txt.is_some() { + let s = Arc::clone(&state); + let robots_txt_handler = move |ctx: RequestContext| { + let s = Arc::clone(&s); + async move { + let services = s.services_for_request(&ctx); + Ok::( + handle_robots_txt(&s.settings, &services, &s.registry, ctx.into_request()) + .await + .unwrap_or_else(|e| http_error(&e)), + ) + } + }; + for method in publisher_fallback_methods() { + if matches!(method, Method::GET | Method::HEAD) { + builder = builder.route(ROBOTS_TXT_PATH, method, robots_txt_handler.clone()); + } else { + builder = builder.route(ROBOTS_TXT_PATH, method, fallback.clone()); + } + } + } + for method in publisher_fallback_methods() { builder = builder.route("/", method.clone(), fallback.clone()); builder = builder.route("/{*rest}", method, fallback.clone()); diff --git a/crates/trusted-server-adapter-spin/src/middleware.rs b/crates/trusted-server-adapter-spin/src/middleware.rs index 4e5140a654..9418a4dc92 100644 --- a/crates/trusted-server-adapter-spin/src/middleware.rs +++ b/crates/trusted-server-adapter-spin/src/middleware.rs @@ -138,6 +138,10 @@ pub(crate) fn apply_finalize_headers( trusted_server_core::response_privacy::apply_response_headers_with_cache_privacy( settings, response, ); + + // Last, so that nothing an origin or an operator's header set can loosen + // a site taken out of search. + trusted_server_core::robots_txt::apply_response_robots_tag(settings, response); } // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index 83c36a0319..3f686305b6 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -1208,3 +1208,401 @@ async fn nextjs_auction_output_holds_until_the_structural_body_close() { "should not leak generated placeholders: {html}" ); } + +// --------------------------------------------------------------------------- +// robots.txt +// --------------------------------------------------------------------------- + +/// The route test settings with a `[robots-txt]` section that refuses every +/// crawler and keeps one path open. +fn router_refusing_every_crawler() -> RouterService { + let settings = Settings::from_toml( + r#" + [publisher] + domain = "test-publisher.example.com" + cookie_domain = ".test-publisher.example.com" + origin_url = "https://origin.test-publisher.example.com" + proxy_secret = "route-test-proxy-secret" + + [ec] + module = "hmac" + + [ec.hmac] + passphrase = "test-secret-key-32-bytes-minimum" + + [geo] + assume_single_jurisdiction = true + + [robots-txt] + modules = "refuse_all" + always_allow = ["/ads.txt"] + "#, + ) + .expect("should parse route test settings") + // A loaded domain carries its policy. + .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()); + + TrustedServerApp::routes_with_settings(settings) + .expect("should build router from test settings") +} + +fn robots_request(method: &str, path: &str) -> Request { + request_builder() + .method(method) + .uri(path) + .body(edgezero_core::body::Body::empty()) + .expect("should build request") +} + +fn robots_tag(response: &Response) -> Option { + response + .headers() + .get("x-robots-tag") + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn refusing_every_crawler_serves_the_refusal_and_tags_every_response() { + let file = route( + router_refusing_every_crawler(), + robots_request("GET", "/robots.txt"), + ) + .await; + + assert_eq!( + file.status().as_u16(), + 200, + "the refusal should be served here" + ); + assert_eq!(robots_tag(&file).as_deref(), Some("noindex, nofollow")); + let body = file.into_body().into_bytes().unwrap_or_default(); + assert_eq!(&body[..], b"User-agent: *\nAllow: /ads.txt\nDisallow: /\n"); + + let other = route( + router_refusing_every_crawler(), + robots_request("GET", "/.well-known/trusted-server.json"), + ) + .await; + + assert_eq!( + robots_tag(&other).as_deref(), + Some("noindex, nofollow"), + "no route may be the one page a crawler is allowed to index" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_head_of_robots_txt_is_answered_here_with_no_body() { + let response = route( + router_refusing_every_crawler(), + robots_request("HEAD", "/robots.txt"), + ) + .await; + + assert_eq!( + response.status().as_u16(), + 200, + "a HEAD should be answered here as a GET is" + ); + assert_eq!( + response + .headers() + .get("content-type") + .and_then(|value| value.to_str().ok()), + Some("text/plain; charset=utf-8") + ); + let body = response.into_body().into_bytes().unwrap_or_default(); + assert!(body.is_empty(), "a HEAD carries no body"); +} + +/// A method that is neither `GET` nor `HEAD` is the publisher's, as it is on +/// any other path, so it is not answered with the file. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn another_method_on_robots_txt_is_not_answered_with_the_file() { + let response = route( + router_refusing_every_crawler(), + robots_request("POST", "/robots.txt"), + ) + .await; + + let body = response.into_body().into_bytes().unwrap_or_default(); + assert!( + !body.starts_with(b"User-agent:"), + "a POST should reach the publisher fallback and not be given the file" + ); +} + +#[test] +fn robots_txt_is_this_servers_only_when_the_settings_carry_the_section() { + let registered = |router: RouterService| -> Vec<(String, String)> { + router + .routes() + .into_iter() + .map(|route| (route.method().to_string(), route.path().to_string())) + .collect() + }; + + let without = registered(test_router()); + assert!( + !without.iter().any(|(_, path)| path == "/robots.txt"), + "without the section the path is the publisher's: {without:?}" + ); + + let with = registered(router_refusing_every_crawler()); + for method in ["GET", "HEAD"] { + assert!( + with.iter() + .any(|(registered, path)| registered == method && path == "/robots.txt"), + "{method} /robots.txt should be answered here: {with:?}" + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_response_carries_no_blanket_rule_without_the_section() { + let response = route( + test_router(), + robots_request("GET", "/.well-known/trusted-server.json"), + ) + .await; + + assert_eq!(robots_tag(&response), None); +} + +/// A module's rules served through this adapter's router, held in the +/// key-value store the runtime hands the application. +mod robots_txt_from_a_module { + use std::collections::HashMap; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex}; + use std::time::Duration; + + use bytes::Bytes; + use edgezero_core::http::Request; + use edgezero_core::key_value_store::{KvError, KvHandle, KvPage, KvStore}; + use edgezero_core::router::RouterService; + use edgezero_core::store_registry::KvRegistry; + use error_stack::Report; + use trusted_server_adapter_spin::app::TrustedServerApp; + use trusted_server_core::error::TrustedServerError; + use trusted_server_core::integrations::{IntegrationBuilder, IntegrationRegistration}; + use trusted_server_core::module_context::ModuleCall; + use trusted_server_core::robots_txt::{Contribution, RobotsTxtContributor}; + use trusted_server_core::settings::Settings; + + use super::robots_request; + + /// The id of the module that answers once. + const ANSWERS_ONCE_ID: &str = "robots-txt-answers-once"; + + /// A robots.txt contributor that answers the first time it is asked and + /// fails every time after, whose answer is due for a refresh at once. + #[derive(Debug, Default)] + struct AnswersOnce { + asked: AtomicUsize, + } + + #[async_trait::async_trait(?Send)] + impl RobotsTxtContributor for AnswersOnce { + fn refresh(&self) -> Duration { + Duration::ZERO + } + + fn fingerprint(&self) -> String { + "answers-once".to_owned() + } + + async fn contribute( + &self, + _call: ModuleCall<'_>, + ) -> Result> { + if self.asked.fetch_add(1, Ordering::SeqCst) == 0 { + Ok(Contribution::parse("User-Agent: OnceBot\nDisallow: /")) + } else { + Err(Report::new(TrustedServerError::Proxy { + message: "the rules service is down".to_owned(), + })) + } + } + } + + fn register_answers_once( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(ANSWERS_ONCE_ID) + .with_robots_txt_contributor( + "robots-txt.answers-once", + Arc::new(AnswersOnce::default()), + ) + .build(), + )) + } + + fn accept_answers_once(_settings: &Settings) -> Result> { + Ok(true) + } + + /// The route test settings with a file made by the module that answers + /// once. + fn router_with_a_module_that_answers_once() -> RouterService { + let settings = Settings::from_toml( + r#" + [publisher] + domain = "test-publisher.example.com" + cookie_domain = ".test-publisher.example.com" + origin_url = "https://origin.test-publisher.example.com" + proxy_secret = "route-test-proxy-secret" + + [ec] + module = "hmac" + + [ec.hmac] + passphrase = "test-secret-key-32-bytes-minimum" + + [geo] + assume_single_jurisdiction = true + + [robots-txt] + modules = ["answers-once"] + + [testing] + modules = ["robots-txt-answers-once"] + "#, + ) + .expect("should parse route test settings") + // A loaded domain carries its policy. + .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()); + let builder = IntegrationBuilder::new( + ANSWERS_ONCE_ID, + ANSWERS_ONCE_ID, + register_answers_once, + accept_answers_once, + ) + .with_module_name("testing.robots-txt-answers-once"); + + TrustedServerApp::routes_with_registrations(settings, &[builder]) + .expect("should build router from test settings") + } + + /// A key-value store kept in memory for one test, so what one request + /// writes is what the next one reads. + #[derive(Default)] + struct MemoryStore { + entries: Mutex>, + } + + #[async_trait::async_trait(?Send)] + impl KvStore for MemoryStore { + async fn delete(&self, key: &str) -> Result<(), KvError> { + self.entries + .lock() + .expect("should lock the store") + .remove(key); + Ok(()) + } + + async fn get_bytes(&self, key: &str) -> Result, KvError> { + Ok(self + .entries + .lock() + .expect("should lock the store") + .get(key) + .cloned()) + } + + async fn list_keys_page( + &self, + _prefix: &str, + _cursor: Option<&str>, + _limit: usize, + ) -> Result { + Ok(KvPage::default()) + } + + async fn put_bytes(&self, key: &str, value: Bytes) -> Result<(), KvError> { + self.entries + .lock() + .expect("should lock the store") + .insert(key.to_owned(), value); + Ok(()) + } + + async fn put_bytes_with_ttl( + &self, + key: &str, + value: Bytes, + _ttl: Duration, + ) -> Result<(), KvError> { + self.put_bytes(key, value).await + } + } + + /// A crawler's `GET` for the file, carrying `store` as the key-value store + /// the runtime hands the application, when there is one. + fn robots_get(store: Option<&Arc>) -> Request { + let mut request = robots_request("GET", "/robots.txt"); + if let Some(store) = store { + let handle = KvHandle::new(Arc::clone(store) as Arc); + request.extensions_mut().insert(KvRegistry::single_id( + "trusted_server_kv".to_owned(), + handle, + )); + } + request + } + + /// With the store the runtime hands the application, a module's answer + /// is held, so when the module next fails the file is made from the + /// answer it gave before. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn a_held_answer_is_served_when_its_module_fails() { + let router = router_with_a_module_that_answers_once(); + let store = Arc::new(MemoryStore::default()); + + for asked in ["answers", "fails"] { + let response = router + .oneshot(robots_get(Some(&store))) + .await + .expect("should route request"); + + assert_eq!(response.status().as_u16(), 200, "the module {asked}"); + let body = response.into_body().into_bytes().unwrap_or_default(); + assert_eq!( + &body[..], + b"User-Agent: OnceBot\nDisallow: /\n", + "the module {asked}" + ); + } + } + + /// With no store from the runtime nothing is held, so a module that fails + /// makes the file a `503` that tells a crawler to come back in ten + /// minutes and is not stored. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn with_nothing_held_a_module_that_fails_makes_the_file_unavailable() { + let router = router_with_a_module_that_answers_once(); + + let first = router + .oneshot(robots_get(None)) + .await + .expect("should route request"); + assert_eq!(first.status().as_u16(), 200); + + let second = router + .oneshot(robots_get(None)) + .await + .expect("should route request"); + assert_eq!(second.status().as_u16(), 503); + let header = |name: &str| { + second + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + }; + assert_eq!(header("retry-after").as_deref(), Some("600")); + assert_eq!(header("cache-control").as_deref(), Some("no-store")); + } +} diff --git a/crates/trusted-server-core/src/http_util.rs b/crates/trusted-server-core/src/http_util.rs index 8b61a73568..163091aaa7 100644 --- a/crates/trusted-server-core/src/http_util.rs +++ b/crates/trusted-server-core/src/http_util.rs @@ -165,6 +165,27 @@ pub(crate) fn request_host(req: &Request) -> &str { extract_request_host(req) } +/// The host a request was addressed to, lowercased with any port removed. +/// +/// Read from `Host`, or from the URI's host when there is no `Host` header, +/// and never from `Forwarded` or `X-Forwarded-Host`, which a client can set to +/// any host it likes. What is signed for a host, or held for every request to +/// it, is keyed on this, so a forwarded header cannot put another host in its +/// place. +pub(crate) fn addressed_host(req: &Request) -> String { + let raw = req + .headers() + .get(header::HOST) + .and_then(|value| value.to_str().ok()) + .or_else(|| req.uri().host()) + .unwrap_or_default(); + let without_port = match raw.rsplit_once(':') { + Some((host, port)) if port.bytes().all(|b| b.is_ascii_digit()) => host, + _ => raw, + }; + without_port.to_ascii_lowercase() +} + /// The scheme a request was made over, read as /// [`RequestInfo::from_request`] reads it. pub(crate) fn request_scheme(req: &Request, client_info: &ClientInfo) -> &'static str { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index e55e1f0ac0..48d4e3a013 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -21,6 +21,7 @@ use crate::http_util::is_navigation_request; use crate::middleware::{Middleware, MiddlewareChain, MiddlewarePhase, PhaseEntries}; use crate::module_context::{ModuleCall, ModuleContext, ResolvedRequest}; use crate::platform::{DisabledGeo, PlatformGeo, RuntimeServices}; +use crate::robots_txt::{ALLOW_ALL, REFUSE_ALL, RobotsTxtContributor}; use crate::settings::Settings; /// What a middleware's element handler decides about one attribute of an @@ -666,6 +667,12 @@ pub struct IntegrationRegistration { /// Declaring one does not make it active, because the module is only asked /// to classify a request when `[device] module` names it. pub device_module: Option<(&'static str, Arc)>, + /// The rules this registration adds to `/robots.txt`, with the name + /// `[robots-txt] modules` selects them by. + /// + /// Declaring them does not put them in the file, because the file is only + /// built from the contributors `[robots-txt] modules` names. + pub robots_txt_contributor: Option<(&'static str, Arc)>, } impl IntegrationRegistration { @@ -695,6 +702,7 @@ impl IntegrationRegistrationBuilder { geo_module: None, ec_module: None, device_module: None, + robots_txt_contributor: None, }, } } @@ -776,6 +784,21 @@ impl IntegrationRegistrationBuilder { self } + /// Declare the rules this registration adds to `/robots.txt`, under their + /// `name`. + /// + /// They are only asked for when `[robots-txt] modules` names them. Core + /// holds what they return and writes the file, see [`crate::robots_txt`]. + #[must_use] + pub fn with_robots_txt_contributor( + mut self, + name: &'static str, + contributor: Arc, + ) -> Self { + self.registration.robots_txt_contributor = Some((name, contributor)); + self + } + /// Mark this integration's JS module for deferred loading via /// `