diff --git a/CHANGELOG.md b/CHANGELOG.md index d7dcb6004a..74b1befda6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -91,7 +91,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the end of the publisher's `ads.txt`, from `crates/ads-txt/sellers`, where a `[[fetch]]` entry for `text/plain` names it. Each line is checked against the ads.txt grammar as the settings load, and a selected module no entry places is refused. +- A file the origin does not have is made by the middleware a `[[fetch]]` entry with a `path`, for a media type other than HTML, names for it. When the origin answers `404 Not Found` for that path itself, which names a file and not a folder, the middleware runs on an empty document of the entry's media type and its output is the file, so `ads.txt` is made from the sellers lines alone where the publisher has none. With no such entry, or for a request that names no host, the origin's answer stands. +- `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the publisher's `ads.txt`, from `crates/ads-txt/sellers`. The module is a middleware for `text/plain`, named in a `[[fetch]]` entry on the file's path, which passes the origin's file through and writes the lines after its last one, so the publisher's file stays authoritative, and a publisher whose origin has no file gets one made of the lines. Each line is checked as the settings load as a data record or a variable of the ads.txt grammar, and a selected module that no entry for `text/plain` on `/ads.txt` places is refused. - A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript`, `application/json` or `image/svg+xml`, written as a bare `type/subtype` in lower case. A response that declares a charset other than UTF-8, or whose status is not `200`, gets no middleware. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then, for a response that is not stored, the serve phase, and an action that decides about HTML there is refused by name. A response an entry covers loses the origin's `ETag` and `Last-Modified`, and one a `[[serve]]` entry covers is sent `private, no-store` when a module left state on the request. - The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, `cmp_id` unless it is a whole number from 1 to 4095, and entries that leave any page without the prompt are refused. The new `Middleware::template_key` keys a stored page by what a middleware writes that no setting decides, so a change to a stub stores pages again. The Trusted Server maintainers hold the crate until InMobi adopt it. - The stock modules' fetch middleware give as their `template_key` the text compiled into their crates that they write into a page, so a change to it stores pages again. The Google ad tag gives its head scripts, the `tsjs.adInit` bootstrap among them, and the GPT script's first-party path, Prebid its configuration script and bundle tag, Didomi its settings script, and Sourcepoint its head scripts and its CDN's first-party path. Permutive, Lockr, DataDome and Google Tag Manager give the first-party path they point a vendor's address at. diff --git a/crates/ads-txt/sellers/Cargo.toml b/crates/ads-txt/sellers/Cargo.toml index 1bab501e44..48f16ca1f8 100644 --- a/crates/ads-txt/sellers/Cargo.toml +++ b/crates/ads-txt/sellers/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "trusted-server-ads-txt-sellers" -description = "Adds a publisher's sellers to the origin's ads.txt as it passes through, from the lines the settings give." +description = "Adds a publisher's sellers to the origin's ads.txt from the lines the settings give, or makes the file from them where the origin has none." authors = { workspace = true } edition = { workspace = true } license = { workspace = true } diff --git a/crates/ads-txt/sellers/src/lib.rs b/crates/ads-txt/sellers/src/lib.rs index 9f79039526..6d6437384e 100644 --- a/crates/ads-txt/sellers/src/lib.rs +++ b/crates/ads-txt/sellers/src/lib.rs @@ -1,6 +1,7 @@ //! The sellers a deployment adds to the publisher's `ads.txt`. //! -//! The publisher's own file at the origin stays authoritative. +//! The publisher's own file at the origin stays authoritative. Where the +//! origin has none, the file is made of the lines alone. use std::sync::Arc; @@ -28,6 +29,10 @@ pub const MODULE: &str = "ads-txt.sellers"; /// works on. pub const MEDIA_TYPE: &str = "text/plain"; +/// The path of the file, which the entry placing the middleware names, so +/// that the file is also made where the origin has none. +const PATH: &str = "/ads.txt"; + /// The variables the specification names, each written as `NAME=value`. const VARIABLES: [&str; 5] = [ "CONTACT", @@ -138,21 +143,25 @@ fn validate_lines(config: &SellersConfig) -> Result<(), Report Result<(), Report> { let placed = settings.fetch.entries().iter().any(|entry| { - entry.media_type == MEDIA_TYPE && entry.middleware.iter().any(|name| name == MODULE) + entry.media_type == MEDIA_TYPE + && entry.path.as_deref() == Some(PATH) + && entry.middleware.iter().any(|name| name == MODULE) }); if placed { return Ok(()); } Err(Report::new(TrustedServerError::Settings { message: format!( - "[ads-txt] selects `sellers`, but no [[fetch]] entry for `{MEDIA_TYPE}` names \ - `{MODULE}`, so the lines would never be written. Add an entry with media_type = \ - `{MEDIA_TYPE}`, path = `/ads.txt` and the middleware. The path is a prefix, so \ - the entry covers every path that starts with `/ads.txt`" + "[ads-txt] selects `sellers`, but no [[fetch]] entry for `{MEDIA_TYPE}` with path \ + = `{PATH}` names `{MODULE}`, so the lines would never be written into the file or \ + make it where the origin has none. Add an entry with media_type = \ + `{MEDIA_TYPE}`, path = `{PATH}` and the middleware. The path is a prefix, so the \ + entry also covers every path that starts with `{PATH}`" ), })) } @@ -392,13 +401,39 @@ mod tests { let refused = validate(&settings).expect_err("should refuse a module no entry places"); assert!( - format!("{refused:?}") - .contains("no [[fetch]] entry for `text/plain` names `ads-txt.sellers`"), + format!("{refused:?}").contains( + "no [[fetch]] entry for `text/plain` with path = `/ads.txt` names \ + `ads-txt.sellers`" + ), "{refused:?}" ); assert!(register(&settings).is_err()); } + #[test] + fn a_document_placing_the_module_off_the_file_s_path_is_refused() { + for path in [None, Some("/robots.txt"), Some("/ads"), Some("/")] { + let path_line = path.map_or_else(String::new, |path| format!("path = \"{path}\"\n")); + let settings = page_settings(&format!( + "[ads-txt]\nmodules = [\"sellers\"]\n\n[ads-txt.sellers]\nlines = \ + [\"{RECORD}\"]\n\n[[fetch]]\nmedia_type = \"text/plain\"\n{path_line}\ + middleware = [\"ads-txt.sellers\"]\n" + )); + + let Err(refused) = validate(&settings) else { + panic!("should refuse the module placed at {path:?}, where no file is made"); + }; + assert!( + format!("{refused:?}").contains("with path = `/ads.txt` names `ads-txt.sellers`"), + "should ask for the entry on `/ads.txt`: {refused:?}" + ); + assert!( + register(&settings).is_err(), + "should not register the module placed at {path:?}" + ); + } + } + #[test] fn a_line_that_is_not_a_record_or_a_variable_is_refused_for_its_reason() { let cases = [ diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index 085a4c0466..4d6b5ad867 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -20,7 +20,12 @@ const LEGACY_ADMIN_DENY_METHODS: &[&str] = /// every route into a startup error page (and its route table into the /// fallback-only set). fn test_settings() -> trusted_server_core::settings::Settings { - trusted_server_core::settings::Settings::from_toml( + test_settings_with("") +} + +/// The route tests' settings with `extra` appended. +fn test_settings_with(extra: &str) -> trusted_server_core::settings::Settings { + trusted_server_core::settings::Settings::from_toml(&format!( r#" [publisher] domain = "test-publisher.example.com" @@ -36,8 +41,9 @@ fn test_settings() -> trusted_server_core::settings::Settings { [geo] assume_single_jurisdiction = true - "#, - ) + {extra} + "# + )) .expect("should parse route test settings") // A loaded domain carries its policy, and these tests decide under the // repository's sample. @@ -832,3 +838,83 @@ async fn health_carries_no_version() { "should write no version on /health, which the service_version middleware writes on a page" ); } + +/// A publisher whose origin has no `ads.txt` gets one made of the sellers +/// lines, through this adapter's router and the stock sellers module. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_sellers_module_makes_an_ads_txt_the_origin_does_not_have() { + use std::sync::Arc; + + use trusted_server_core::platform::PlatformHttpClient; + use trusted_server_core::platform::test_support::{ + StubHttpClient, build_services_with_http_client, + }; + + const SELLERS: &str = r#" + [ads-txt] + modules = ["sellers"] + + [ads-txt.sellers] + lines = ["ssp.example, 12345, DIRECT, f08c47fec0942fa0"] + + [[fetch]] + media_type = "text/plain" + path = "/ads.txt" + middleware = ["ads-txt.sellers"] + "#; + + let origin = Arc::new(StubHttpClient::new()); + origin.push_response_with_headers( + 404, + b"Not found".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("set-cookie", "session=origin; Path=/"), + ("cache-control", "public, max-age=600"), + ], + ); + let router = TrustedServerApp::routes_with_settings_and_services( + test_settings_with(SELLERS), + build_services_with_http_client(Arc::clone(&origin) as Arc), + ) + .expect("should build the router with the sellers module"); + let request = edgezero_core::http::request_builder() + .method("GET") + .uri("https://test-publisher.example.com/ads.txt") + .header("host", "test-publisher.example.com") + .body(edgezero_core::body::Body::empty()) + .expect("should build the ads.txt request"); + + let response = router + .oneshot(request) + .await + .expect("should answer the ads.txt request"); + + assert_eq!( + response.status(), + 200, + "should make the file the origin does not have" + ); + assert_eq!( + response + .headers() + .get("content-type") + .and_then(|value| value.to_str().ok()), + Some("text/plain; charset=utf-8") + ); + assert!( + response.headers().get("set-cookie").is_none() + && response.headers().get("cache-control").is_none(), + "should carry none of the headers the origin sent with its 404: {:?}", + response.headers() + ); + let body = response + .into_body() + .into_bytes() + .expect("should buffer the made file"); + assert_eq!( + body.as_ref(), + b"ssp.example, 12345, DIRECT, f08c47fec0942fa0\n", + "should be the sellers lines alone" + ); +} diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index cfa659ce47..fcdb4def9d 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -482,6 +482,25 @@ impl PhaseEntries { names } + /// The entry that makes the document the origin does not have at + /// `path`, being the first written, for a media type core streams other + /// than HTML, whose `path` is `path` itself and that names a middleware. + /// A longer path the entry's `path` begins, and a folder, written with its + /// closing slash, get nothing made, so a page the origin does not have + /// keeps the origin's answer. + #[must_use] + pub fn making(&self, path: &str) -> Option<&PhaseEntry> { + if path.ends_with('/') { + return None; + } + self.0.iter().find(|entry| { + entry.media_type != HTML_MEDIA_TYPE + && is_dispatched_media_type(&entry.media_type) + && entry.path.as_deref() == Some(path) + && !entry.middleware.is_empty() + }) + } + /// Checks each entry's media type, path and names, and that no entry is /// hidden behind an earlier one, or covers the same pages as one. /// diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index 1c986b6aaf..e9070619bb 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -1874,3 +1874,144 @@ fn deploy_validation_refuses_a_misshapen_entry() { "should name the entry: {error}" ); } + +#[test] +fn the_making_entry_is_for_the_request_path_and_names_a_middleware() { + let entries = PhaseEntries::new(vec![ + PhaseEntry { + media_type: "text/plain".to_owned(), + path: Some("/ads.txt".to_owned()), + middleware: Vec::new(), + }, + PhaseEntry { + media_type: "text/plain".to_owned(), + path: Some("/ads.txt".to_owned()), + middleware: vec!["example.sellers".to_owned()], + }, + PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec!["example.page".to_owned()], + }, + ]); + + assert_eq!( + entries + .making("/ads.txt") + .map(|entry| entry.middleware.as_slice()), + Some(&["example.sellers".to_owned()][..]), + "should pass the entry naming no middleware and take the first that names one" + ); + assert!( + entries.making("/robots.txt").is_none(), + "should make nothing where no entry is for the request path, whatever covers every path" + ); +} + +#[test] +fn an_html_entry_with_a_path_makes_nothing() { + let entries = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: Some("/news/missing".to_owned()), + middleware: vec!["example.page".to_owned()], + }]); + + assert!( + entries.making("/news/missing").is_none(), + "should leave a page the origin does not have as the origin's answer" + ); +} + +#[test] +fn a_file_is_made_at_the_entry_path_alone() { + let entries = PhaseEntries::new(vec![ + PhaseEntry { + media_type: "text/css".to_owned(), + path: Some("/".to_owned()), + middleware: vec!["example.styles".to_owned()], + }, + PhaseEntry { + media_type: "text/plain".to_owned(), + path: Some("/ads.txt".to_owned()), + middleware: vec!["example.sellers".to_owned()], + }, + PhaseEntry { + media_type: "text/plain".to_owned(), + path: Some("/feeds/".to_owned()), + middleware: vec!["example.feeds".to_owned()], + }, + ]); + + assert_eq!( + entries + .making("/ads.txt") + .map(|entry| entry.media_type.as_str()), + Some("text/plain"), + "should make the file from the entry for its own path" + ); + for path in [ + "/site.css", + "/missing-page", + "/ads.txt.bak", + "/feeds/latest", + ] { + assert!( + entries.making(path).is_none(), + "should make nothing at {path}, which an entry's path only begins" + ); + } + for folder in ["/", "/feeds/"] { + assert!( + entries.making(folder).is_none(), + "should make nothing for the folder {folder}" + ); + } +} + +#[test] +fn the_first_entry_written_for_the_path_makes_the_file() { + let entries = PhaseEntries::new(vec![ + PhaseEntry { + media_type: "text/css".to_owned(), + path: Some("/ads.txt".to_owned()), + middleware: vec!["example.styles".to_owned()], + }, + PhaseEntry { + media_type: "text/plain".to_owned(), + path: Some("/ads.txt".to_owned()), + middleware: vec!["example.sellers".to_owned()], + }, + ]); + + assert_eq!( + entries + .making("/ads.txt") + .map(|entry| entry.media_type.as_str()), + Some("text/css"), + "should take the entry written first among those for the path" + ); +} + +#[test] +fn an_html_entry_ahead_does_not_shadow_the_entry_that_makes_the_file() { + let entries = PhaseEntries::new(vec![ + PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: Some("/".to_owned()), + middleware: vec!["example.page".to_owned()], + }, + PhaseEntry { + media_type: "text/plain".to_owned(), + path: Some("/ads.txt".to_owned()), + middleware: vec!["example.sellers".to_owned()], + }, + ]); + + assert_eq!( + entries + .making("/ads.txt") + .map(|entry| entry.media_type.as_str()), + Some("text/plain"), + "should make the file from the text/plain entry, not as HTML" + ); +} diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 7b494e2e78..352c88d9db 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -1974,6 +1974,23 @@ pub(crate) fn classify_response_route( ResponseRoute::Stream } +/// An empty document of `media_type`, which the middleware a `[[fetch]]` +/// entry names for a path the origin has nothing at is run on. +fn empty_document(media_type: &str) -> Result, Report> { + let content_type = if media_type.starts_with("text/") { + format!("{media_type}; charset=utf-8") + } else { + media_type.to_owned() + }; + Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, content_type) + .body(EdgeBody::empty()) + .change_context(TrustedServerError::Proxy { + message: format!("failed to build an empty {media_type} document"), + }) +} + fn response_cache_control_is_private_or_no_store(response: &Response) -> bool { cache_control_headers_are_private_or_no_store(response.headers()) } @@ -5676,6 +5693,22 @@ pub async fn handle_publisher_request( response.headers().len() ); + // A file the origin does not have is made, see `PhaseEntries::making`. + // Core runs no middleware for a request that names no host, so there the + // made file would go out empty and the origin's answer stands instead. + if response.status() == StatusCode::NOT_FOUND + && (request_method == Method::GET || request_method == Method::HEAD) + && !request_host.is_empty() + && let Some(entry) = settings.fetch.making(&request_path) + { + log::debug!( + "Making {} for {request_path}, which the origin does not have, with {}", + entry.media_type, + entry.middleware.join(", ") + ); + response = empty_document(&entry.media_type)?; + } + if should_run_ad_stack && response.status() == StatusCode::NOT_MODIFIED { if let Some(dispatched) = dispatched_auction.take() { emit_abandoned_auction( @@ -22124,6 +22157,430 @@ mod tests { ); } + /// What the publisher path answered one request with. + struct Proxied { + settings: Settings, + registry: IntegrationRegistry, + services: crate::platform::RuntimeServices, + orchestrator: AuctionOrchestrator, + response: PublisherResponse, + } + + /// A request through the publisher path, the origin answering `status`. + async fn proxy(fetch: Vec, method: Method, path: &str, status: u16) -> Proxied { + let settings = settings_running_the_middleware_fixture(fetch); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + status, + b"the origin's page".to_vec(), + vec![("content-type", "text/html; charset=utf-8")], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let consent = crate::consent::ConsentContext { + jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, + ..Default::default() + }; + let mut ec_context = EcContext::new_for_test_with_ip(None, consent, None); + let orchestrator = AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + ); + let req = HttpRequest::builder() + .method(method) + .uri(format!("https://publisher.example{path}")) + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let response = handle_publisher_request( + AppContext { + settings: &settings, + integration_registry: ®istry, + }, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[], + registry: None, + }, + req, + EdgeCacheHeader::SMaxageFallback, + ) + .await + .expect("should proxy the request"); + Proxied { + settings, + registry, + services, + orchestrator, + response, + } + } + + /// One entry naming the fixture's text middleware, for `path` or for + /// every path. + fn text_entry_for(path: Option<&str>) -> Vec { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: vec![fixture::TEXT.to_owned()], + }] + } + + /// The response of a proxied request, however it is to be sent. + fn response_of(response: PublisherResponse) -> Response { + match response { + PublisherResponse::Buffered(response) + | PublisherResponse::Stream { response, .. } + | PublisherResponse::PassThrough { response, .. } => response, + PublisherResponse::AssembleTemplate { .. } => { + panic!("a template is not an answer to these requests") + } + } + } + + fn content_type_of(response: &Response) -> Option<&str> { + response + .headers() + .get(header::CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + } + + #[tokio::test] + async fn a_file_the_origin_does_not_have_is_made_by_the_middleware_named_for_its_path() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let proxied = proxy( + text_entry_for(Some("/ads.txt")), + Method::GET, + "/ads.txt", + 404, + ) + .await; + let PublisherResponse::Stream { + response, + body, + params, + } = proxied.response + else { + panic!("a made file should stream through the entry's middleware"); + }; + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + content_type_of(&response), + Some("text/plain; charset=utf-8"), + "should be a document of the entry's media type" + ); + let mut params = *params; + let mut output = Vec::new(); + stream_publisher_body_async( + body, + &mut output, + &mut params, + &proxied.settings, + &proxied.registry, + &proxied.orchestrator, + &proxied.services, + ) + .await + .expect("should stream the made file"); + assert_eq!( + String::from_utf8(output).expect("text"), + fixture::TEXT_LINE, + "should be what the middleware writes on an empty document" + ); + } + + #[tokio::test] + async fn a_head_request_for_a_made_file_gets_the_same_answer() { + let proxied = proxy( + text_entry_for(Some("/ads.txt")), + Method::HEAD, + "/ads.txt", + 404, + ) + .await; + let response = buffer_publisher_response_async( + proxied.response, + &Method::HEAD, + &proxied.settings, + &proxied.registry, + &proxied.orchestrator, + &proxied.services, + ) + .await + .expect("should buffer the HEAD answer"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + content_type_of(&response), + Some("text/plain; charset=utf-8"), + "should answer a HEAD as the GET is answered" + ); + assert!( + response + .into_body() + .into_bytes() + .expect("should hold the HEAD answer in memory") + .is_empty(), + "should answer a HEAD with no body" + ); + } + + #[tokio::test] + async fn an_html_entry_with_a_path_makes_no_missing_page() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::HTML_MEDIA_TYPE; + + let proxied = proxy( + vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: Some("/news/".to_owned()), + middleware: vec![fixture::HEAD.to_owned()], + }], + Method::GET, + "/news/missing", + 404, + ) + .await; + + assert_eq!( + response_of(proxied.response).status(), + StatusCode::NOT_FOUND, + "should keep the origin's answer for a page it does not have" + ); + } + + #[tokio::test] + async fn an_html_entry_ahead_does_not_shadow_the_entry_for_a_missing_file() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::HTML_MEDIA_TYPE; + + let mut fetch = vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: Some("/".to_owned()), + middleware: vec![fixture::HEAD.to_owned()], + }]; + fetch.extend(text_entry_for(Some("/ads.txt"))); + let proxied = proxy(fetch, Method::GET, "/ads.txt", 404).await; + let response = response_of(proxied.response); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + content_type_of(&response), + Some("text/plain; charset=utf-8"), + "should make the file from the text/plain entry, not as HTML" + ); + } + + #[tokio::test] + async fn an_entry_covering_every_path_makes_no_missing_page() { + let proxied = proxy(text_entry_for(None), Method::GET, "/missing.txt", 404).await; + + assert_eq!( + response_of(proxied.response).status(), + StatusCode::NOT_FOUND, + "should keep the origin's answer, because a page the origin does not have is its answer" + ); + } + + #[tokio::test] + async fn a_missing_page_under_an_entry_path_keeps_the_origin_answer() { + for (entry_path, path) in [("/", "/missing-page"), ("/ads.txt", "/ads.txt.bak")] { + let proxied = proxy(text_entry_for(Some(entry_path)), Method::GET, path, 404).await; + + assert_eq!( + response_of(proxied.response).status(), + StatusCode::NOT_FOUND, + "should keep the origin's answer for {path}, which the entry for {entry_path} \ + covers but does not name" + ); + } + } + + #[tokio::test] + async fn an_entry_for_a_folder_makes_no_missing_page() { + for folder in ["/", "/feeds/"] { + let proxied = proxy(text_entry_for(Some(folder)), Method::GET, folder, 404).await; + + assert_eq!( + response_of(proxied.response).status(), + StatusCode::NOT_FOUND, + "should keep the origin's answer for the folder {folder}" + ); + } + } + + #[tokio::test] + async fn a_missing_file_outside_the_entry_path_keeps_the_origin_answer() { + let proxied = proxy( + text_entry_for(Some("/ads.txt")), + Method::GET, + "/robots.txt", + 404, + ) + .await; + + assert_eq!( + response_of(proxied.response).status(), + StatusCode::NOT_FOUND + ); + } + + #[tokio::test] + async fn a_request_that_names_no_host_keeps_the_origin_answer() { + let settings = settings_running_the_middleware_fixture(text_entry_for(Some("/ads.txt"))); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 404, + b"the origin's page".to_vec(), + vec![("content-type", "text/html; charset=utf-8")], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let request = build_request(Method::GET, "/ads.txt"); + + let response = run_publisher_proxy(&settings, &services, request).await; + + assert_eq!( + response_of(response).status(), + StatusCode::NOT_FOUND, + "should keep the origin's answer, because core runs no middleware for a request \ + that names no host and the made file would go out empty" + ); + } + + #[tokio::test] + async fn a_method_other_than_get_or_head_keeps_the_origin_answer() { + for method in [ + Method::POST, + Method::PUT, + Method::PATCH, + Method::DELETE, + Method::OPTIONS, + ] { + let proxied = proxy( + text_entry_for(Some("/ads.txt")), + method.clone(), + "/ads.txt", + 404, + ) + .await; + + assert_eq!( + response_of(proxied.response).status(), + StatusCode::NOT_FOUND, + "should keep the origin's answer to a {method}" + ); + } + } + + #[tokio::test] + async fn a_made_file_carries_none_of_the_headers_of_the_origin_s_404() { + let settings = settings_running_the_middleware_fixture(text_entry_for(Some("/ads.txt"))); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 404, + b"the origin's page".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("set-cookie", "session=origin; Path=/"), + ("cache-control", "public, max-age=600"), + ("etag", "\"origin-404\""), + ("x-origin", "not found"), + ], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/ads.txt") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + + let response = response_of(run_publisher_proxy(&settings, &services, request).await); + + assert_eq!(response.status(), StatusCode::OK); + let names: Vec<&str> = response + .headers() + .keys() + .map(http::HeaderName::as_str) + .collect(); + assert_eq!( + names, + [header::CONTENT_TYPE.as_str()], + "should carry the Content-Type the file is made with and nothing the origin sent" + ); + } + + #[tokio::test] + async fn a_made_file_streams_through_the_finalizer_fastly_uses() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let proxied = proxy( + text_entry_for(Some("/ads.txt")), + Method::GET, + "/ads.txt", + 404, + ) + .await; + let response = publisher_response_into_streaming_response( + proxied.response, + &Method::GET, + Arc::new(proxied.settings), + &proxied.registry, + Arc::new(proxied.orchestrator), + proxied.services, + ) + .await + .expect("should finalize the made file"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + content_type_of(&response), + Some("text/plain; charset=utf-8") + ); + let mut body = Vec::new(); + match response.into_body() { + EdgeBody::Stream(mut stream) => { + while let Some(chunk) = futures::StreamExt::next(&mut stream).await { + body.extend_from_slice(&chunk.expect("should read the made file")); + } + } + other => body.extend_from_slice(&other.into_bytes().expect("should hold the file")), + } + assert_eq!( + String::from_utf8(body).expect("text"), + fixture::TEXT_LINE, + "should be what the middleware writes on an empty document" + ); + } + + #[tokio::test] + async fn an_origin_error_on_the_entry_path_keeps_the_origin_answer() { + let proxied = proxy( + text_entry_for(Some("/ads.txt")), + Method::GET, + "/ads.txt", + 500, + ) + .await; + + assert_eq!( + response_of(proxied.response).status(), + StatusCode::INTERNAL_SERVER_ERROR, + "should make a file only where the origin has none" + ); + } + #[test] fn a_text_body_in_a_charset_other_than_utf8_is_left_as_it_was() { use crate::integrations::registry_test_support::middleware_fixture as fixture; diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index d6b9c61ad0..6dead8b236 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1205,13 +1205,17 @@ it passes through untouched. | `lines` | Array[String] | Yes | The lines written at the end of the file, each a data record (domain, account id, `DIRECT` or `RESELLER`, and an optional certification authority id) or a variable | **How the file is made.** Where a `[[fetch]]` entry for `text/plain` names -the module, it writes the lines after the origin's last one. The entry's -`path` is a prefix, so `/ads.txt` also covers `/ads.txt.bak`. +the module, it writes the lines after the origin's last one, and where the +origin has no file the file is made of the lines alone, see A file the origin +does not have below. The entry's `path` is a prefix, so `/ads.txt` also +covers `/ads.txt.bak`, but only `/ads.txt` itself is made where the origin +has none. **Refused when the settings load**: a line that is not a data record or a variable, naming which part is wrong, a comment, an empty list, and a document that selects the module but names it in no `[[fetch]]` entry for -`text/plain`, where the lines would never be written. +`text/plain` with `path = "/ads.txt"`, where the lines would never be +written into the file or make it. **Example**: @@ -1915,6 +1919,17 @@ covers loses the origin's `ETag` and `Last-Modified`, because a middleware may change its body, and when a module left state on the request, one a `[[serve]]` entry covers is sent `private, no-store`, as a page is. +**A file the origin does not have.** When the origin answers `404 Not Found` +for the `path` of a `[[fetch]]` entry for a media type other than HTML, +that entry's middleware runs on an empty document of its media type and its +output is the file, so `ads.txt` is made from the sellers lines alone where +the publisher has none. The file is made at that path itself and never at a +longer path the entry covers, and where two such entries name the path the +first written makes it. With no such entry, on any other answer from the +origin, or for a request that names no host, the origin's answer stands, and +an entry for HTML, without a path or for a folder makes nothing, because a +page the origin does not have is the origin's answer. + ```toml [[fetch]] media_type = "text/plain" diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index ffdd8f542d..eec5583f04 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -212,6 +212,12 @@ markup and the element and text handlers are decisions about HTML, and an action that makes one on another media type is refused by name when the chain is planned. +A middleware named in an entry with a path, for a media type other than +HTML, also runs where the origin answers `404 Not Found` for that path +itself, on an empty document of the entry's media type, and what it writes +is the file. A middleware that adds to a file therefore writes the same on an +empty one, with nothing before its first line. + ```rust struct Sellers;