diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ea9f13a5b..350b3bd331 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -90,6 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript` or `application/json`, written as a bare `type/subtype` in lower case. A response that declares a charset other than UTF-8, or whose status is not `200`, gets no middleware. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then the serve phase, and an action that decides about HTML there is refused by name. A response an entry covers loses the origin's `ETag` and `Last-Modified`, and one a `[[serve]]` entry covers is sent `private, no-store` when a module left state on the request. - The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, `cmp_id` unless it is a whole number from 1 to 4095, and entries that leave any page without the prompt are refused. The new `Middleware::template_key` keys a stored page by what a middleware writes that no setting decides, so a change to a stub stores pages again. The Trusted Server maintainers hold the crate until InMobi adopt it. - The stock modules' fetch middleware give as their `template_key` the text compiled into their crates that they write into a page, so a change to it stores pages again. The Google ad tag gives its head scripts, the `tsjs.adInit` bootstrap among them, and the GPT script's first-party path, Prebid its configuration script and bundle tag, Didomi its settings script, and Sourcepoint its head scripts and its CDN's first-party path. Permutive, Lockr, DataDome and Google Tag Manager give the first-party path they point a vendor's address at. - `[robots-txt]` serves a publisher's `/robots.txt` from the settings. `modules` names what makes the file, in order. `refuse_all` and `allow_all` are Trusted Server's own, and any other name is a module that contributes rules through the new `RobotsTxtContributor` extension point, declared with `with_robots_txt_contributor`. Trusted Server writes the file, so a contributor cannot produce a malformed one, a refusal cannot be loosened, and a path in `always_allow` stays open to every crawler refused the whole site. With `refuse_all` selected every response an adapter finalizes carries `X-Robots-Tag: noindex, nofollow`. A module is asked for the host the request was addressed to, never one a forwarded header names, and is given nothing of the crawler's connection. Its answer is held for that host in the request's key-value store until its refresh is due, and no adapter gives a request a working one, so every adapter asks each module on every request. A module that cannot answer has its last held answer served, and with none held, or with no rule in the whole file, the answer is `503` and never an empty file. On Fastly the file is answered before the Edge Cookie steps and the request filters. Without the section `/robots.txt` reaches the publisher's origin as before. diff --git a/crates/cmp/inmobi/src/lib.rs b/crates/cmp/inmobi/src/lib.rs index d036d38f9e..96cfe0a536 100644 --- a/crates/cmp/inmobi/src/lib.rs +++ b/crates/cmp/inmobi/src/lib.rs @@ -401,10 +401,14 @@ mod tests { /// The head markup `middleware` write in `phase`, in the order written. fn head_markup(phase: MiddlewarePhase, middleware: Vec>) -> Vec { let document_state = IntegrationDocumentState::default(); - MiddlewareChain::new(phase, middleware) - .plan(&context(phase, &document_state)) - .expect("the chain should plan") - .head_inserts + MiddlewareChain::new( + phase, + trusted_server_core::middleware::HTML_MEDIA_TYPE, + middleware, + ) + .plan(&context(phase, &document_state)) + .expect("the chain should plan") + .head_inserts } /// The one piece of head markup the prompt writes in `phase`. diff --git a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs index 5c8a52fcd5..51b3710dbe 100644 --- a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs +++ b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs @@ -672,6 +672,7 @@ mod tests { ) -> String { let chain = MiddlewareChain::new( MiddlewarePhase::Fetch, + crate::middleware::HTML_MEDIA_TYPE, vec![Arc::new(ScriptAddress(integration))], ); let config = HtmlProcessorConfig { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index fc03657531..4ade57c677 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -175,15 +175,17 @@ impl IntegrationDocumentState { /// /// A request preparer or a request filter leaves a value under its /// integration id with [`IntegrationRequestState::insert`]. Where the -/// request's reader is served an HTML document, every value is copied into -/// the [`IntegrationDocumentState`] the `[[serve]]` entry's middleware share, +/// request's reader is served an HTML document, or another body a +/// `[[serve]]` entry covers, every value is copied into the +/// [`IntegrationDocumentState`] the `[[serve]]` entry's middleware share, /// so the module's serve middleware reads it there, and the same values are /// handed to the module's response finalizer. /// /// A request that carries any value keeps to the origin path. Its document is /// never read from a shared template and never stored as one, and an HTML -/// response with a body is sent `private, no-store`. A fetch middleware is -/// handed none of these values. +/// response with a body, or another response a `[[serve]]` entry covers, is +/// sent `private, no-store`. A fetch middleware is handed none of these +/// values. #[derive(Clone, Default)] pub struct IntegrationRequestState { values: IntegrationDocumentStateMap, @@ -1396,6 +1398,13 @@ fn check_middleware( phases_of(middleware).join(" and ") ))); } + if !middleware.handles_media_type(&entry.media_type) { + return Err(refuse(format!( + "{at} covers `{}` and names `{name}`, which does not work on that media \ + type", + entry.media_type + ))); + } } } } @@ -1944,7 +1953,7 @@ impl IntegrationRegistry { .iter() .filter_map(|name| self.inner.middleware_named(name).map(Arc::clone)) .collect(); - MiddlewareChain::new(phase, middleware) + MiddlewareChain::new(phase, media_type, middleware) } /// Every integration id the registry was built from, named or not, in @@ -2954,6 +2963,18 @@ pub(crate) mod test_support { pub(crate) const LINK_TARGET: &str = "/fixture/link"; /// The selector the broken middleware asks for. pub(crate) const BROKEN_SELECTOR: &str = "a["; + /// The name of the fetch middleware that works on `text/plain`. + pub(crate) const TEXT: &str = "testing.middleware-fixture.text"; + /// The media type the text middleware works on. + pub(crate) const TEXT_MEDIA_TYPE: &str = "text/plain"; + /// The line the text middleware adds at the end of a text body. + pub(crate) const TEXT_LINE: &str = "# added by the middleware fixture\n"; + /// The name of the serve middleware that writes a header on a text + /// response. + pub(crate) const TEXT_HEADER: &str = "testing.middleware-fixture.text-header"; + /// The header the text header middleware writes, name and value. + pub(crate) const TEXT_HEADER_NAME: &str = "x-fixture-text"; + pub(crate) const TEXT_HEADER_VALUE: &str = "served"; /// The builder core's test build lists beside its own. pub(crate) const BUILDER: IntegrationBuilder = @@ -3075,6 +3096,72 @@ pub(crate) mod test_support { } } + /// Passes a text body through and adds a line at its end. + struct AppendLine; + + impl crate::streaming_processor::StreamProcessor for AppendLine { + fn process_chunk( + &mut self, + chunk: &[u8], + is_last: bool, + ) -> Result, std::io::Error> { + let mut output = chunk.to_vec(); + if is_last { + output.extend_from_slice(TEXT_LINE.as_bytes()); + } + Ok(output) + } + } + + struct Text; + + impl Middleware for Text { + fn middleware_id(&self) -> &'static str { + TEXT + } + + fn handles_media_type(&self, media_type: &str) -> bool { + media_type == TEXT_MEDIA_TYPE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + stream: Some(Box::new(AppendLine)), + ..MiddlewareAction::pass() + } + } + } + + struct TextHeader; + + impl Middleware for TextHeader { + fn middleware_id(&self) -> &'static str { + TEXT_HEADER + } + + fn handles_media_type(&self, media_type: &str) -> bool { + media_type == TEXT_MEDIA_TYPE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + response_headers: vec![( + http::HeaderName::from_static(TEXT_HEADER_NAME), + http::HeaderValue::from_static(TEXT_HEADER_VALUE), + )], + ..MiddlewareAction::pass() + } + } + } + struct Broken; impl Middleware for Broken { @@ -3114,6 +3201,8 @@ pub(crate) mod test_support { .with_middleware(Arc::new(Broken)) .with_middleware(Arc::new(Reader)) .with_middleware(Arc::new(BrokenReader)) + .with_middleware(Arc::new(Text)) + .with_middleware(Arc::new(TextHeader)) .build(), )) } @@ -6134,13 +6223,20 @@ mod tests { fixture::BROKEN, fixture::READER, fixture::BROKEN_READER, + fixture::TEXT, + fixture::TEXT_HEADER, crate::service_version::MODULE, ], "should list the middleware the selected module supplies, and core's own after them" ); assert_eq!( registry.middleware_in(MiddlewarePhase::Fetch), - [fixture::HEAD, fixture::LINKS, fixture::BROKEN], + [ + fixture::HEAD, + fixture::LINKS, + fixture::BROKEN, + fixture::TEXT + ], "should list the middleware that run in the fetch phase" ); assert_eq!( @@ -6148,6 +6244,7 @@ mod tests { [ fixture::READER, fixture::BROKEN_READER, + fixture::TEXT_HEADER, crate::service_version::MODULE ], "should list the middleware that run in the serve phase, core's own last" @@ -6169,6 +6266,73 @@ mod tests { ); } + /// An entry for `text/plain` naming the middleware that works on it. + fn text_entry(names: &[&str]) -> PhaseEntry { + PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: None, + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + } + } + + #[test] + fn an_entry_runs_a_middleware_on_the_media_type_it_handles() { + let settings = fixture_settings_with_entries(vec![ + html_entry(None, &[fixture::HEAD]), + text_entry(&[fixture::TEXT]), + ]); + + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + + let chain_for = |media_type: &str| { + registry + .middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + media_type, + "/a.txt", + ) + .ids() + }; + assert_eq!( + chain_for(fixture::TEXT_MEDIA_TYPE), + [fixture::TEXT], + "should run the text middleware on a text response" + ); + assert_eq!( + chain_for(HTML_MEDIA_TYPE), + [fixture::HEAD], + "should run the page middleware on a page and not the text one" + ); + } + + #[test] + fn an_entry_naming_a_middleware_under_a_media_type_it_does_not_handle_is_refused() { + for (media_type, name) in [ + (fixture::TEXT_MEDIA_TYPE, fixture::HEAD), + (HTML_MEDIA_TYPE, fixture::TEXT), + ] { + let settings = fixture_settings_with_entries(vec![PhaseEntry { + media_type: media_type.to_owned(), + path: None, + middleware: vec![name.to_owned()], + }]); + + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse a middleware under a media type it does not handle"); + + let message = error.to_string(); + assert!( + message.contains(&format!( + "[[fetch]] entry 1 covers `{media_type}` and names `{name}`, which does not \ + work on that media type" + )), + "should name the entry, the media type and the middleware: {message}" + ); + } + } + #[test] fn a_module_no_section_selects_supplies_no_middleware() { let settings = crate::test_support::tests::create_test_settings(); @@ -6197,12 +6361,14 @@ mod tests { message.contains( "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies" ) && message.contains(&format!( - "[{}, {}, {}, {}, {}, {}]", + "[{}, {}, {}, {}, {}, {}, {}, {}]", fixture::HEAD, fixture::LINKS, fixture::BROKEN, fixture::READER, fixture::BROKEN_READER, + fixture::TEXT, + fixture::TEXT_HEADER, crate::service_version::MODULE )), "should name the entry and list what could be named: {message}" @@ -6255,6 +6421,8 @@ mod tests { fixture::BROKEN, fixture::READER, fixture::BROKEN_READER, + fixture::TEXT, + fixture::TEXT_HEADER, ], "should list every middleware when there are no entries" ); @@ -6263,7 +6431,13 @@ mod tests { html_entry(Some("/news/"), &[fixture::LINKS]), html_entry(None, &[fixture::HEAD]), ]), - [fixture::BROKEN, fixture::READER, fixture::BROKEN_READER], + [ + fixture::BROKEN, + fixture::READER, + fixture::BROKEN_READER, + fixture::TEXT, + fixture::TEXT_HEADER + ], "should leave out a middleware any entry names" ); } @@ -6326,7 +6500,13 @@ mod tests { .collect(); assert_eq!( unnamed, - [fixture::LINKS, fixture::BROKEN, fixture::BROKEN_READER], + [ + fixture::LINKS, + fixture::BROKEN, + fixture::BROKEN_READER, + fixture::TEXT, + fixture::TEXT_HEADER + ], "should count a middleware as named only by an entry of a phase it runs in" ); } diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index e4e270b24b..cfa659ce47 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -41,10 +41,49 @@ use crate::integrations::{AttributeRewriteAction, IntegrationDocumentState, Scri use crate::platform::PlatformIdentity; use crate::streaming_processor::StreamProcessor; -/// The media type a document is matched as, and the only one a middleware -/// runs on. +/// The media type a document is matched as, and the one a middleware runs on +/// unless it says it handles another. pub const HTML_MEDIA_TYPE: &str = "text/html"; +/// Whether a middleware may run on a response of `media_type`, written as a +/// bare `type/subtype` in lower case. +/// +/// HTML and the text assets that stream through core as text: every `text/` +/// type except `text/x-component`, and JavaScript and JSON. +#[must_use] +pub fn is_dispatched_media_type(media_type: &str) -> bool { + media_type != "text/x-component" + && (media_type.starts_with("text/") + || matches!(media_type, "application/javascript" | "application/json")) +} + +/// The type and subtype of a content type header, in lower case and without +/// parameters, which is what an entry's `media_type` is matched against. +/// `text/html; charset=utf-8` gives `text/html`. +#[must_use] +pub fn media_type_of(content_type: &str) -> String { + content_type + .split(';') + .next() + .unwrap_or_default() + .trim() + .to_ascii_lowercase() +} + +/// Whether a content type declares no charset or declares UTF-8, the one a +/// middleware's stream processor is handed. A body in another charset passes +/// through untouched, because a processor that writes UTF-8 into it would +/// corrupt it. +#[must_use] +pub fn is_utf8_or_undeclared_charset(content_type: &str) -> bool { + content_type.split(';').skip(1).all(|parameter| { + parameter.split_once('=').is_none_or(|(name, value)| { + !name.trim().eq_ignore_ascii_case("charset") + || value.trim().trim_matches('"').eq_ignore_ascii_case("utf-8") + }) + }) +} + /// When a middleware runs, relative to the store of shared templates. #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum MiddlewarePhase { @@ -253,12 +292,17 @@ impl MiddlewareAction { /// Whether the action leaves the document as it arrived. #[must_use] pub fn is_pass(&self) -> bool { + self.is_stream_only() && self.stream.is_none() && self.response_headers.is_empty() + } + + /// Whether the action decides nothing about HTML, so it can run on a + /// media type that is not HTML. + #[must_use] + pub fn is_stream_only(&self) -> bool { self.head_inserts.is_empty() && self.after_bundle_inserts.is_empty() && self.element_handlers.is_empty() && self.text_handlers.is_empty() - && self.stream.is_none() - && self.response_headers.is_empty() } /// Checks every handler's selector parses, before a handler is built @@ -328,6 +372,12 @@ pub trait Middleware: Send + Sync { /// names its own by a bare word. fn middleware_id(&self) -> &'static str; + /// Whether this middleware works on a response of `media_type`, given as + /// a bare `type/subtype`. + fn handles_media_type(&self, media_type: &str) -> bool { + media_type == HTML_MEDIA_TYPE + } + /// The phases an entry may name this middleware in. fn phases(&self) -> &[MiddlewarePhase]; @@ -441,10 +491,21 @@ impl PhaseEntries { pub fn validate(&self, phase: MiddlewarePhase) -> Result<(), String> { for (index, entry) in self.0.iter().enumerate() { let at = format!("[[{phase}]] entry {}", index + 1); - if entry.media_type != HTML_MEDIA_TYPE { + if entry.media_type != entry.media_type.to_ascii_lowercase() + || entry + .media_type + .contains(|c: char| c == ';' || c == '*' || c.is_whitespace()) + || entry + .media_type + .split_once('/') + .is_none_or(|(_, subtype)| subtype.is_empty()) + || !is_dispatched_media_type(&entry.media_type) + { return Err(format!( - "{at} covers `{}`, and a middleware runs on `{HTML_MEDIA_TYPE}` alone. Write \ - media_type = \"{HTML_MEDIA_TYPE}\", in lower case with no parameters", + "{at} covers `{}`, which is not a media type a middleware runs on. Write a \ + bare `type/subtype` in lower case with no parameters, being `text/html`, \ + another `text/` type other than `text/x-component`, \ + `application/javascript` or `application/json`", entry.media_type )); } @@ -529,10 +590,12 @@ impl Serialize for PhaseEntries { } } -/// The middleware one entry selects, in the order they run. +/// The middleware one entry selects, in the order they run, for one media +/// type. #[derive(Clone)] pub struct MiddlewareChain { phase: MiddlewarePhase, + media_type: String, middleware: Vec>, } @@ -541,16 +604,26 @@ impl fmt::Debug for MiddlewareChain { formatter .debug_struct("MiddlewareChain") .field("phase", &self.phase) + .field("media_type", &self.media_type) .field("middleware", &self.ids()) .finish() } } impl MiddlewareChain { - /// A chain of middleware already in the order they run. + /// A chain of middleware already in the order they run, for responses of + /// `media_type`. #[must_use] - pub fn new(phase: MiddlewarePhase, middleware: Vec>) -> Self { - Self { phase, middleware } + pub fn new( + phase: MiddlewarePhase, + media_type: &str, + middleware: Vec>, + ) -> Self { + Self { + phase, + media_type: media_type.to_owned(), + middleware, + } } /// The phase this chain runs in. @@ -559,6 +632,12 @@ impl MiddlewareChain { self.phase } + /// The media type of the responses this chain runs on. + #[must_use] + pub fn media_type(&self) -> &str { + &self.media_type + } + /// The middleware names, in the order they run. #[must_use] pub fn ids(&self) -> Vec<&'static str> { @@ -577,11 +656,13 @@ impl MiddlewareChain { /// Asks every middleware in turn what it will do with one document. /// /// Checks each handler's selector, refusing the document when one does - /// not parse. + /// not parse. On a media type other than HTML, an action that decides + /// about HTML is refused the same way. /// /// # Errors /// - /// A message naming the middleware and the selector that does not parse. + /// A message naming the middleware and the selector that does not parse, + /// or the HTML decision it made about a document that is not HTML. pub fn plan(&self, context: &MiddlewareContext<'_>) -> Result { let mut plan = MiddlewarePlan::default(); for middleware in &self.middleware { @@ -593,6 +674,16 @@ impl MiddlewareChain { middleware.middleware_id() ) })?; + if self.media_type != HTML_MEDIA_TYPE && !action.is_stream_only() { + return Err(format!( + "the {} middleware `{}` cannot run on `{}`, because it decides about HTML \ + and that is not HTML. On another media type an action carries a stream \ + processor alone", + self.phase, + middleware.middleware_id(), + self.media_type + )); + } if !action.response_headers.is_empty() && self.phase != MiddlewarePhase::Serve { return Err(format!( "the {} middleware `{}` cannot run, because it writes a response header, \ diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index 4b47ffa08b..1c986b6aaf 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -11,7 +11,7 @@ use crate::html_processor::{ create_html_processor_with_middleware, create_serve_processor, }; use crate::integrations::registry_test_support::request_fixture; -use crate::integrations::{IntegrationRegistry, IntegrationRequestState}; +use crate::integrations::{IntegrationDocumentState, IntegrationRegistry, IntegrationRequestState}; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use crate::test_support::tests::create_test_settings; @@ -54,11 +54,11 @@ fn middleware( } fn chain_of(middleware: Vec>) -> MiddlewareChain { - MiddlewareChain::new(MiddlewarePhase::Fetch, middleware) + MiddlewareChain::new(MiddlewarePhase::Fetch, HTML_MEDIA_TYPE, middleware) } fn serve_chain_of(middleware: Vec>) -> MiddlewareChain { - MiddlewareChain::new(MiddlewarePhase::Serve, middleware) + MiddlewareChain::new(MiddlewarePhase::Serve, HTML_MEDIA_TYPE, middleware) } /// What a serve chain is told of a reader whose request left `request_state`. @@ -324,22 +324,150 @@ fn entries_from_the_longest_path_to_every_path_are_accepted() { } #[test] -fn an_entry_for_a_media_type_other_than_html_is_refused() { - for media_type in ["text/css", "text/html; charset=utf-8", "TEXT/HTML", "*/*"] { - let mut css = entry(None, &["example.a"]); - css.media_type = media_type.to_owned(); +fn an_entry_for_a_media_type_a_middleware_does_not_run_on_is_refused() { + for media_type in [ + "text/html; charset=utf-8", + "TEXT/HTML", + "text/PLAIN", + "*/*", + "text/*", + "image/png", + "application/octet-stream", + "text/x-component", + "image/svg+xml", + "text/", + "text", + "text/plain\t", + "text/ plain", + ] { + let mut other = entry(None, &["example.a"]); + other.media_type = media_type.to_owned(); - let message = refusal(vec![css]); + let message = refusal(vec![other]); assert!( message.contains("[[fetch]] entry 1") && message.contains(&format!("covers `{media_type}`")) - && message.contains("media_type = \"text/html\""), - "should name the entry and the media type to write: {message}" + && message.contains("is not a media type a middleware runs on") + && message.contains("`text/html`"), + "should name the entry, the media type and what to write: {message}" ); } } +#[test] +fn an_entry_may_cover_a_text_media_type_that_is_not_html() { + for media_type in [ + "text/plain", + "text/css", + "application/javascript", + "application/json", + ] { + let mut text = entry(None, &["example.a"]); + text.media_type = media_type.to_owned(); + + PhaseEntries::new(vec![text]) + .validate(MiddlewarePhase::Fetch) + .unwrap_or_else(|message| panic!("{media_type} should be accepted: {message}")); + } +} + +#[test] +fn a_media_type_is_the_type_and_subtype_of_a_content_type_in_lower_case() { + assert_eq!(media_type_of("text/html; charset=utf-8"), "text/html"); + assert_eq!(media_type_of(" Text/Plain "), "text/plain"); + assert_eq!(media_type_of("application/json"), "application/json"); + assert_eq!(media_type_of(""), ""); + assert!(is_dispatched_media_type("text/plain")); + assert!(!is_dispatched_media_type("image/svg+xml")); + assert!( + !is_dispatched_media_type("text/x-component"), + "a length prefixed stream is not changed" + ); + assert!(!is_dispatched_media_type("image/png")); +} + +#[test] +fn a_body_is_handed_to_a_middleware_only_in_utf8_or_an_undeclared_charset() { + assert!(is_utf8_or_undeclared_charset("text/plain")); + assert!(is_utf8_or_undeclared_charset("text/plain; charset=utf-8")); + assert!(is_utf8_or_undeclared_charset( + "text/plain; Charset=\"UTF-8\"" + )); + assert!(is_utf8_or_undeclared_charset("text/plain; format=flowed")); + assert!(!is_utf8_or_undeclared_charset("text/plain; charset=utf-16")); + assert!(!is_utf8_or_undeclared_charset( + "text/plain; charset=iso-8859-1" + )); +} + +#[test] +fn a_chain_on_another_media_type_runs_a_stream_processor_alone() { + let document_state = IntegrationDocumentState::default(); + let context = super::test_support::context(MiddlewarePhase::Fetch, &document_state); + let head = middleware("example.head", |_context| MiddlewareAction { + head_inserts: vec!["".to_owned()], + ..MiddlewareAction::pass() + }); + let stream = middleware("example.stream", |_context| MiddlewareAction { + stream: Some(Box::new(Append("# more".to_owned()))), + ..MiddlewareAction::pass() + }); + + let refused = MiddlewareChain::new(MiddlewarePhase::Fetch, "text/plain", vec![head]) + .plan(&context) + .expect_err("a head insert is a decision about HTML"); + assert!( + refused.contains("`example.head`") + && refused.contains("cannot run on `text/plain`") + && refused.contains("decides about HTML"), + "should name the middleware, the media type and the reason: {refused}" + ); + + let plan = MiddlewareChain::new(MiddlewarePhase::Fetch, "text/plain", vec![stream]) + .plan(&context) + .expect("a stream processor runs on any media type"); + assert_eq!(plan.processors.len(), 1); + assert!(plan.head_inserts.is_empty()); +} + +#[test] +fn a_serve_chain_on_another_media_type_may_write_a_header() { + let document_state = IntegrationDocumentState::default(); + let context = super::test_support::context(MiddlewarePhase::Serve, &document_state); + + let plan = MiddlewareChain::new( + MiddlewarePhase::Serve, + "text/plain", + vec![header("example.version", "x-example", "1")], + ) + .plan(&context) + .expect("a header is not a decision about HTML"); + + assert_eq!( + plan.response_headers.len(), + 1, + "should keep the header for the response" + ); + assert!( + plan.processors.is_empty() && plan.head_inserts.is_empty(), + "should change nothing in the body" + ); +} + +/// A stream processor that adds `text` at the end of a body. +struct Append(String); + +impl crate::streaming_processor::StreamProcessor for Append { + fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> Result, io::Error> { + let mut output = chunk.to_vec(); + if is_last { + output.extend_from_slice(self.0.as_bytes()); + } + Ok(output) + } +} + #[test] fn an_entry_whose_path_could_cover_nothing_is_refused() { let message = refusal(vec![entry(Some("news/"), &["example.a"])]); @@ -1680,8 +1808,8 @@ fn settings_without_entries_write_no_fetch_key() { fn settings_with_a_misshapen_entry_are_refused() { let cases = [ ( - "\n[[fetch]]\nmedia_type = \"text/css\"\nmiddleware = [\"example.all\"]\n", - "[[fetch]] entry 1 covers `text/css`", + "\n[[fetch]]\nmedia_type = \"image/png\"\nmiddleware = [\"example.all\"]\n", + "[[fetch]] entry 1 covers `image/png`", ), ( "\n[fetch]\nmedia_type = \"text/html\"\nmiddleware = [\"example.all\"]\n", diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index e5228df42b..7b494e2e78 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -75,8 +75,8 @@ use crate::html_processor::ServePass; use crate::http_util::{ RequestInfo, is_navigation_request, remove_deployment_headers, serve_static_with_etag, }; -use crate::integrations::{IntegrationRegistry, IntegrationRequestState}; -use crate::middleware::{ResponseHeader, write_response_headers}; +use crate::integrations::{IntegrationDocumentState, IntegrationRegistry, IntegrationRequestState}; +use crate::middleware::{MiddlewarePhase, ResponseHeader, media_type_of, write_response_headers}; use crate::permissions::PermissionState; use crate::platform::{ GeoInfo, PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, @@ -645,6 +645,8 @@ struct ProcessResponseParams<'a> { request_path: &'a str, settings: &'a Settings, content_type: &'a str, + /// See [`OwnedProcessResponseParams::status`]. + status: StatusCode, integration_registry: &'a IntegrationRegistry, /// Head script carrying this request's permission state, or [`None`] under a /// shared-template mode. See [`template_permissions_script`]. @@ -666,6 +668,163 @@ struct PublisherBodyProcessor { response_headers: Vec, } +/// A response that is not HTML, as the middleware the entries covering it +/// are told of it. +struct TextResponse<'a> { + /// The response's `Content-Type`, parameters and all. + content_type: &'a str, + status: StatusCode, + request_path: &'a str, + request_host: &'a str, + request_scheme: &'a str, + origin_host: &'a str, + request_state: &'a IntegrationRequestState, + max_buffered_script_bytes: usize, +} + +/// The stream processors the entries covering a response that is not HTML +/// run on its body, the fetch entry's and then the serve entry's, with the +/// headers the serve entry's middleware write on the response, or none +/// when [`text_entry_media_type`] gives no media type or no entry covers +/// it. Both phases run here because the template cache stores HTML alone. +/// +/// # Errors +/// +/// When a middleware decides about HTML on a response that is not HTML, or +/// asks for a selector that does not parse. +fn middleware_stream_processors( + response: &TextResponse<'_>, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) -> Result> { + use crate::middleware::MiddlewareContext; + + let Some(media_type) = text_entry_media_type(response.content_type, response.status) else { + return Ok(TextPass::default()); + }; + let mut processors = Vec::new(); + let mut response_headers = Vec::new(); + for phase in &MiddlewarePhase::ALL { + let chain = integration_registry.middleware_chain( + settings.phase_entries(*phase), + *phase, + &media_type, + response.request_path, + ); + if chain.is_empty() { + continue; + } + let document_state = IntegrationDocumentState::default(); + if *phase == MiddlewarePhase::Serve { + response.request_state.seed(&document_state); + } + let plan = chain + .plan(&MiddlewareContext { + phase: *phase, + request_host: response.request_host, + request_scheme: response.request_scheme, + origin_host: response.origin_host, + platform: integration_registry.platform(), + document_state: &document_state, + max_buffered_script_bytes: response.max_buffered_script_bytes, + }) + .map_err(|message| Report::new(TrustedServerError::Configuration { message }))?; + processors.extend(plan.processors); + response_headers.extend(plan.response_headers); + } + Ok(TextPass { + processors, + response_headers, + }) +} + +/// The media type the entries match a response that is not HTML by, or +/// `None` when no middleware runs on it: HTML and the React Server +/// Components stream, which have processors of their own, a type no entry +/// may cover, a charset other than UTF-8, and a status other than `200`. +/// Only a `200` carries the whole file, so a partial `206` or an error +/// body is left as the origin sent it. +fn text_entry_media_type(content_type: &str, status: StatusCode) -> Option { + use crate::middleware::{is_dispatched_media_type, is_utf8_or_undeclared_charset}; + + let media_type = media_type_of(content_type); + let runs = status == StatusCode::OK + && !is_html_content_type(content_type) + && !content_type_contains_ascii_case_insensitive(content_type, "text/x-component") + && is_dispatched_media_type(&media_type) + && is_utf8_or_undeclared_charset(content_type); + runs.then_some(media_type) +} + +/// Fits the headers of a response that is not HTML to the entries covering +/// it. A body an entry covers loses the origin's `ETag` and `Last-Modified`, +/// because its middleware may change it while the origin's file stays the +/// same, so a conditional request carrying them would bring a `304` for the +/// body as it was. A body a `[[serve]]` entry covers is also kept out of +/// shared caches when a module left state on the request, because the +/// entry's middleware are handed that state. Called once the serve +/// middleware's headers are written, so none of them can undo it. +fn apply_text_entry_cache_rules( + response: &mut Response, + params: &OwnedProcessResponseParams, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) { + let Some(media_type) = text_entry_media_type(¶ms.content_type, params.status) else { + return; + }; + let covered = |phase: MiddlewarePhase| { + !integration_registry + .middleware_chain( + settings.phase_entries(phase), + phase, + &media_type, + ¶ms.request_path, + ) + .is_empty() + }; + let served = covered(MiddlewarePhase::Serve); + if served && !params.request_state.is_empty() { + // Removes the validators too. + enforce_terminal_private_cache_privacy(response); + } else if served || covered(MiddlewarePhase::Fetch) { + response.headers_mut().remove(header::ETAG); + response.headers_mut().remove(header::LAST_MODIFIED); + } +} + +/// What the entries covering a text response do to it: the processors over +/// its body and the headers the serve entry's middleware write. +#[derive(Default)] +struct TextPass { + processors: Vec>, + response_headers: Vec, +} + +/// A processor followed by the ones the entries add, each handed what the +/// one before produced. +struct WithStreamProcessors { + inner: Box, + processors: Vec>, +} + +impl StreamProcessor for WithStreamProcessors { + fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> Result, std::io::Error> { + let mut output = self.inner.process_chunk(chunk, is_last)?; + for processor in &mut self.processors { + output = processor.process_chunk(&output, is_last)?; + } + Ok(output) + } + + fn reset(&mut self) { + self.inner.reset(); + for processor in &mut self.processors { + processor.reset(); + } + } +} + impl PublisherBodyProcessor { /// Build the body processor, returning any deferred inline seam token it /// installed alongside it. @@ -716,12 +875,30 @@ impl PublisherBodyProcessor { ¶ms.request_scheme, )) } else { - Box::new(create_url_replacer( - ¶ms.origin_host, - ¶ms.origin_url, - ¶ms.request_host, - ¶ms.request_scheme, - )) + let pass = middleware_stream_processors( + &TextResponse { + content_type: ¶ms.content_type, + status: params.status, + request_path: ¶ms.request_path, + request_host: ¶ms.request_host, + request_scheme: ¶ms.request_scheme, + origin_host: ¶ms.origin_host, + request_state: ¶ms.request_state, + max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, + }, + settings, + integration_registry, + )?; + response_headers = pass.response_headers; + Box::new(WithStreamProcessors { + inner: Box::new(create_url_replacer( + ¶ms.origin_host, + ¶ms.origin_url, + ¶ms.request_host, + ¶ms.request_scheme, + )), + processors: pass.processors, + }) }; Ok(( @@ -822,13 +999,31 @@ fn process_response_streaming( .with_max_pending_decoded_bytes(max_pending_decoded_bytes) .process(body_as_reader(body)?, output)?; } else { - let replacer = create_url_replacer( - params.origin_host, - params.origin_url, - params.request_host, - params.request_scheme, - ); - StreamingPipeline::new(config, replacer) + let pass = middleware_stream_processors( + &TextResponse { + content_type: params.content_type, + status: params.status, + request_path: params.request_path, + request_host: params.request_host, + request_scheme: params.request_scheme, + origin_host: params.origin_host, + request_state: params.request_state, + max_buffered_script_bytes: params.settings.publisher.max_buffered_body_bytes, + }, + params.settings, + params.integration_registry, + )?; + response_headers = pass.response_headers; + let processor = WithStreamProcessors { + inner: Box::new(create_url_replacer( + params.origin_host, + params.origin_url, + params.request_host, + params.request_scheme, + )), + processors: pass.processors, + }; + StreamingPipeline::new(config, processor) .with_max_pending_decoded_bytes(max_pending_decoded_bytes) .process(body_as_reader(body)?, output)?; } @@ -1848,6 +2043,9 @@ pub struct OwnedProcessResponseParams { /// `[[serve]]` entries that cover the document. pub(crate) request_path: String, pub(crate) content_type: String, + /// The origin's status, which decides whether the entries' middleware + /// run on a body that is not HTML. See [`text_entry_media_type`]. + pub(crate) status: StatusCode, pub(crate) ad_slots_script: Option, pub(crate) ad_bids_state: AdBidsState, /// Observation context for the in-flight auction. @@ -1974,6 +2172,7 @@ pub async fn buffer_publisher_response_async( ) .await?; write_response_headers(&mut response, response_headers); + apply_text_entry_cache_rules(&mut response, ¶ms, settings, integration_registry); // Authorized template cache transforms are emitted as identity by // `process_response_streaming_async`; inline transforms retain the origin // coding. This avoids recompressing and immediately decoding a full document. @@ -2347,6 +2546,7 @@ fn build_template_assembly_params( // Set by the caller, with the rest of what belongs to the request. request_path: String::new(), content_type: entry.metadata.content_type.clone(), + status: StatusCode::OK, // The template already carries the head seam; re-injecting would duplicate it. permissions_json, ad_slots_script: None, @@ -2809,6 +3009,7 @@ pub async fn publisher_response_into_streaming_response( } }; write_response_headers(&mut response, processor.take_response_headers()); + apply_text_entry_cache_rules(&mut response, ¶ms, &settings, integration_registry); // The guard is created before the lazy stream so an auction whose // response body is dropped unpolled still logs the loss. let dispatched_auction = params.dispatched_auction.take().map(|dispatched| { @@ -3141,6 +3342,7 @@ pub fn stream_publisher_body( request_path: ¶ms.request_path, settings, content_type: ¶ms.content_type, + status: params.status, integration_registry, permissions_script: permissions_script.as_deref(), ad_slots_script: params.ad_slots_script.as_deref(), @@ -5782,6 +5984,7 @@ pub async fn handle_publisher_request( request_scheme: request_scheme.to_string(), request_path: request_path.clone(), content_type, + status, permissions_json, ad_slots_script: ad_slots_script.clone(), ad_bids_state: ad_bids_state.clone(), @@ -7886,6 +8089,7 @@ mod tests { use crate::integrations::{ CarriedJsModule, IntegrationBuilder, IntegrationRegistration, IntegrationRegistry, }; + use crate::middleware::{PhaseEntries, PhaseEntry}; use crate::permissions::{Permission, PermissionSet}; use crate::platform::test_support::{ StubHttpClient, build_services_with_http_client, noop_services, @@ -9425,6 +9629,7 @@ mod tests { request_scheme: "https".to_owned(), request_path: "/".to_owned(), content_type: "application/json".to_owned(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -21463,6 +21668,481 @@ mod tests { )) } + /// The settings of a deployment running the middleware fixture, with the + /// `[[fetch]]` entry given. + fn settings_running_the_middleware_fixture(fetch: Vec) -> Settings { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.fetch = PhaseEntries::new(fetch); + settings + } + + /// A text body of `content_type` streamed through the publisher path, as + /// a reader gets it. + fn text_body_through_the_publisher( + settings: &Settings, + content_type: &str, + body: &str, + ) -> String { + let registry = IntegrationRegistry::new(settings).expect("should build a registry"); + let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: "identity".to_string(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/a.txt".to_owned(), + content_type: content_type.to_string(), + status: http::StatusCode::OK, + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }; + let mut output = Vec::new(); + stream_publisher_body( + EdgeBody::from(body.as_bytes().to_vec()), + &mut output, + ¶ms, + settings, + ®istry, + ) + .expect("should stream a text body"); + String::from_utf8(output).expect("text") + } + + #[test] + fn a_text_body_passes_through_the_middleware_of_the_entry_covering_it() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::TEXT.to_owned()], + }]); + let body = "User-agent: *\nSee https://origin.example.com/more\n"; + + let output = text_body_through_the_publisher(&settings, "text/plain; charset=utf-8", body); + + assert_eq!( + output, + format!( + "User-agent: *\nSee https://proxy.example.com/more\n{}", + fixture::TEXT_LINE + ), + "should move the origin's address and then run the entry's middleware" + ); + } + + /// The same through the response path a streamed body takes, which + /// builds its processor apart from the buffered path's. + #[tokio::test] + async fn a_streamed_text_body_passes_through_the_middleware_of_the_entry_covering_it() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::platform::test_support::noop_services; + + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::TEXT.to_owned()], + }]); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let orchestrator = AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + ); + let services = noop_services(); + let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/a.txt".to_owned(), + content_type: "text/plain; charset=utf-8".to_string(), + status: http::StatusCode::OK, + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }; + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"User-agent: *\nSee https://origin.example.com/"), + bytes::Bytes::from_static(b"more\n"), + ])); + let mut output = Vec::new(); + + stream_publisher_body_async( + body, + &mut output, + &mut params, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await + .expect("should stream a text body"); + + assert_eq!( + String::from_utf8(output).expect("text"), + format!( + "User-agent: *\nSee https://proxy.example.com/more\n{}", + fixture::TEXT_LINE + ), + "should move the origin's address and then run the entry's middleware" + ); + } + + /// The parameters of a text response at `/a.txt`, from the origin as + /// `content_type` says, for a reader of `proxy.example.com`. + fn text_params(content_type: &str) -> OwnedProcessResponseParams { + OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: "identity".to_string(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/a.txt".to_owned(), + content_type: content_type.to_string(), + status: http::StatusCode::OK, + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + } + } + + /// Settings running the fixture with a serve entry on `text/plain` + /// naming its header-writing middleware. + fn settings_serving_the_text_header() -> Settings { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_running_the_middleware_fixture(Vec::new()); + settings.serve = PhaseEntries::new(vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::TEXT_HEADER.to_owned()], + }]); + settings + } + + fn named_headers(headers: &[ResponseHeader]) -> Vec<(&str, &[u8])> { + headers + .iter() + .map(|(name, value)| (name.as_str(), value.as_bytes())) + .collect() + } + + #[test] + fn a_text_body_s_serve_middleware_write_their_headers_on_the_response() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let settings = settings_serving_the_text_header(); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let params = text_params("text/plain; charset=utf-8"); + let mut output = Vec::new(); + + let headers = stream_publisher_body( + EdgeBody::from(b"plain\n".to_vec()), + &mut output, + ¶ms, + &settings, + ®istry, + ) + .expect("should stream a text body"); + + assert_eq!( + named_headers(&headers), + [( + fixture::TEXT_HEADER_NAME, + fixture::TEXT_HEADER_VALUE.as_bytes() + )], + "should hand back the header the serve middleware writes on a text response" + ); + assert_eq!(String::from_utf8(output).expect("text"), "plain\n"); + } + + #[tokio::test] + async fn a_streamed_text_body_s_serve_middleware_write_their_headers_on_the_response() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::platform::test_support::noop_services; + + let settings = settings_serving_the_text_header(); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let orchestrator = AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + ); + let services = noop_services(); + let mut params = text_params("text/plain; charset=utf-8"); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"pla"), + bytes::Bytes::from_static(b"in\n"), + ])); + let mut output = Vec::new(); + + let headers = stream_publisher_body_async( + body, + &mut output, + &mut params, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await + .expect("should stream a text body"); + + assert_eq!( + named_headers(&headers), + [( + fixture::TEXT_HEADER_NAME, + fixture::TEXT_HEADER_VALUE.as_bytes() + )], + "should hand back the header the serve middleware writes on a streamed text response" + ); + assert_eq!(String::from_utf8(output).expect("text"), "plain\n"); + } + + /// The answers to `GET /a.txt` through the publisher path, the origin + /// answering `status` with `plain\n` as UTF-8 text and the headers given, + /// finalized as the adapters that buffer do and then as Fastly does. + /// With `marked`, a module leaves state on the request. + async fn text_answers_through_the_publisher( + settings: Settings, + marked: bool, + status: u16, + origin_headers: &[(&str, &str)], + ) -> Vec<(Response, String)> { + let settings = Arc::new(settings); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + )); + let mut answers = Vec::new(); + for streaming in [false, true] { + let stub = Arc::new(StubHttpClient::new()); + let mut headers = vec![("content-type", "text/plain; charset=utf-8")]; + headers.extend_from_slice(origin_headers); + stub.push_response_with_headers(status, b"plain\n".to_vec(), headers); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/a.txt") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + if marked { + crate::integrations::registry_test_support::request_fixture::mark(&mut request); + } + let consent = crate::consent::ConsentContext { + jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, + ..Default::default() + }; + let mut ec_context = EcContext::new_for_test_with_ip(None, consent, None); + let proxied = handle_publisher_request( + AppContext { + settings: &settings, + integration_registry: ®istry, + }, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[], + registry: None, + }, + request, + EdgeCacheHeader::SMaxageFallback, + ) + .await + .expect("should proxy the request"); + let response = if streaming { + publisher_response_into_streaming_response( + proxied, + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + services.clone(), + ) + .await + } else { + buffer_publisher_response_async( + proxied, + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await + } + .expect("should finalize the response"); + let (parts, body) = response.into_parts(); + let body = match body { + EdgeBody::Stream(mut stream) => { + let mut out = Vec::new(); + while let Some(chunk) = futures::StreamExt::next(&mut stream).await { + out.extend_from_slice(&chunk.expect("should read a chunk")); + } + out + } + other => other.into_bytes().expect("should hold the bytes").to_vec(), + }; + answers.push(( + Response::from_parts(parts, EdgeBody::empty()), + String::from_utf8(body).expect("text"), + )); + } + answers + } + + #[tokio::test] + async fn a_text_response_whose_serve_entry_is_handed_request_state_is_private() { + let origin = [("cache-control", "public, max-age=600"), ("etag", "\"v1\"")]; + for (marked, cache_control) in [(true, "no-store, private"), (false, "public, max-age=600")] + { + let answers = text_answers_through_the_publisher( + settings_serving_the_text_header(), + marked, + 200, + &origin, + ) + .await; + for (response, _) in &answers { + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some(cache_control), + "should keep a text response shaped from one reader's state out of \ + shared caches, and no other" + ); + } + } + } + + #[tokio::test] + async fn a_text_body_is_handed_to_a_middleware_only_when_the_status_is_200() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::TEXT.to_owned()], + }]); + for (status, body) in [ + (200, format!("plain\n{}", fixture::TEXT_LINE)), + (206, "plain\n".to_owned()), + (404, "plain\n".to_owned()), + (500, "plain\n".to_owned()), + ] { + let answers = + text_answers_through_the_publisher(settings.clone(), false, status, &[]).await; + for (response, text) in &answers { + assert_eq!(response.status().as_u16(), status); + assert_eq!( + text, &body, + "should change a whole file and leave a {status} as the origin sent it" + ); + } + } + } + + #[tokio::test] + async fn a_text_body_a_middleware_changes_loses_the_origin_validators() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let origin = [ + ("etag", "\"v1\""), + ("last-modified", "Wed, 21 Oct 2015 07:28:00 GMT"), + ]; + for (path, keeps_validators) in [(None, false), (Some("/elsewhere/"), true)] { + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: vec![fixture::TEXT.to_owned()], + }]); + let answers = text_answers_through_the_publisher(settings, false, 200, &origin).await; + for (response, _) in &answers { + for name in [header::ETAG, header::LAST_MODIFIED] { + assert_eq!( + response.headers().contains_key(&name), + keeps_validators, + "{name} should stay on a body no middleware changes, and on no other" + ); + } + } + } + } + + #[test] + fn a_text_body_no_entry_covers_is_left_as_it_was() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: Some("/elsewhere/".to_owned()), + middleware: vec![fixture::TEXT.to_owned()], + }]); + + let output = text_body_through_the_publisher(&settings, "text/plain", "plain\n"); + + assert_eq!( + output, "plain\n", + "should run nothing on a path the entry does not cover" + ); + } + + #[test] + fn a_text_body_in_a_charset_other_than_utf8_is_left_as_it_was() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::TEXT_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::TEXT.to_owned()], + }]); + + let output = + text_body_through_the_publisher(&settings, "text/plain; charset=utf-16", "plain\n"); + + assert_eq!( + output, "plain\n", + "should hand a middleware no body it would corrupt by writing UTF-8" + ); + } + #[test] fn tsjs_dynamic_serves_a_standalone_module_by_registration_flag_not_by_name() { // The module the registration marks standalone is one a vendor crate @@ -21650,6 +22330,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -21713,6 +22394,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -21797,6 +22479,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -21890,6 +22573,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -21998,6 +22682,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22047,6 +22732,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22176,6 +22862,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22243,6 +22930,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22313,6 +23001,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22383,6 +23072,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22453,6 +23143,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22503,6 +23194,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22729,6 +23421,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: Some( r#""# @@ -22807,6 +23500,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: Some( r#""# @@ -22887,6 +23581,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -22961,6 +23656,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/css".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -23117,6 +23813,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: Some( r#""# @@ -23793,6 +24490,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -23991,6 +24689,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: Some( r#""# @@ -24073,6 +24772,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "Text/HTML; Charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: Some( r#""# @@ -24138,6 +24838,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -24258,6 +24959,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), @@ -24318,6 +25020,7 @@ mod tests { request_scheme: "https".to_string(), request_path: "/".to_owned(), content_type: "text/html".to_string(), + status: http::StatusCode::OK, permissions_json: String::new(), ad_slots_script: None, ad_bids_state: AdBidsState::default(), diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index a4aa28491c..341cfc3ef3 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1808,7 +1808,7 @@ An entry of either list has the same three fields. | Field | Type | Required | Description | | ------------ | ---------------- | -------- | ---------------------------------------------------------------------------------------------- | -| `media_type` | String | Yes | The media type the entry covers. Only `text/html` is accepted | +| `media_type` | String | Yes | The media type the entry covers, a bare `type/subtype` in lower case, see below | | `path` | String | No | A prefix of the request path, compared as written. Absent, or `/`, the entry covers every path | | `middleware` | Array of strings | Yes | The middleware to run, in order, each handed the page as the one before it left it | @@ -1847,6 +1847,38 @@ script bundle, after the fetch middleware's. A middleware takes its settings from its module's own `[
.]` table. An entry carries none and switches nothing on. +### Entries for a media type other than HTML + +An entry covers `text/html`, another `text/` type other than +`text/x-component`, `application/javascript` or `application/json`, written +as a bare `type/subtype` in lower case with no parameters. A response is +matched by the type and subtype of its `Content-Type`, without the +parameters, so `text/plain; charset=utf-8` takes the entry for +`text/plain`. Anything else, an image or a length prefixed stream among +them, passes through untouched and no entry can cover it. So does a +response that declares a charset other than UTF-8 or whose status is not +`200`, such as a partial `206` or an error page. + +A middleware says which media types it works on, HTML alone unless it says +otherwise, and an entry naming one under a media type it does not handle is +refused when the registry is built, with the entry and the middleware named. +On a media type other than HTML a middleware changes the body as a stream, +after core has moved the origin's addresses in it, in the order the entry +names them. Head markup and the element and text handlers are decisions +about HTML, and a middleware that makes one there is refused by name. The +template cache stores HTML alone, so the fetch entry's middleware and then +the serve entry's run on each response as it is fetched. A response an entry +covers loses the origin's `ETag` and `Last-Modified`, because a middleware +may change its body, and when a module left state on the request, one a +`[[serve]]` entry covers is sent `private, no-store`, as a page is. + +```toml +[[fetch]] +media_type = "text/plain" +path = "/ads.txt" +middleware = ["example.sellers"] +``` + The modules this repository ships supply the middleware below. A deployment that runs several of them names them all in one entry's list, because a page takes one entry, and the order written is the order they run in. The table @@ -1873,7 +1905,7 @@ the names in and the order `trusted-server.example.toml` shows them in. An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type -other than `text/html`, a path that does not start with `/` or that holds a +no middleware runs on, a path that does not start with `/` or that holds a `*`, a `?` or a `#`, an entry that names no middleware or names one twice, a key an entry does not read, and an entry that an earlier one already covers. An entry naming a middleware that no running module supplies refuses startup, diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index 57c57e6fea..ffdd8f542d 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -200,6 +200,47 @@ Production adapters replace every unavailable or fixture service with the target implementation. The builder deliberately panics when a required service is omitted, so adapter startup must construct the complete service graph. +### A middleware for a media type other than HTML + +A middleware works on `text/html` unless it says otherwise, with +`handles_media_type`, and an entry naming it under a media type it does not +handle is refused when the registry is built. On a media type other than +HTML an action may carry a stream processor alone, which is handed the body +after core has moved the origin's addresses in it, chunk by chunk, with the +last chunk marked, and only for a `200`, whose body is the whole file. Head +markup and the element and text handlers are decisions about HTML, and an +action that makes one on another media type is refused by name when the +chain is planned. + +```rust +struct Sellers; + +impl Middleware for Sellers { + fn middleware_id(&self) -> &'static str { + "example.sellers" + } + + fn handles_media_type(&self, media_type: &str) -> bool { + media_type == "text/plain" + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + stream: Some(Box::new(AppendLines::new(["example-ssp.com, 1, DIRECT"]))), + ..MiddlewareAction::pass() + } + } +} +``` + +The entry that places it names the media type and, where the change is for +one file, its path. See "Entries for a media type other than HTML" in the +[configuration guide](/guide/configuration#entries-for-a-media-type-other-than-html). + ## Proxy implementation rules An `IntegrationProxy::handle` implementation receives the request and its