diff --git a/AGENTS.md b/AGENTS.md index cf80a644a3..318fc212a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -447,8 +447,7 @@ Integrations register in Rust via: ```rust IntegrationRegistration::builder(ID) .with_proxy() - .with_attribute_rewriter() - .with_head_injector() + .with_middleware() .build() ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index 49f8f01287..57cf1428fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware. A deployment adds the entries when it upgrades. Until it does, a module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. To keep a page as it was, name the middleware of the modules a deployment selects in one `[[fetch]]` entry and one `[[serve]]` entry for `text/html`, in the order `trusted-server.example.toml` shows them, which is the order the modules' hooks ran in. +- **Breaking:** The four page traits are removed, being `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector`, with their context types, `ScriptTextAccumulator`, `AttributeRewriteOutcome` and the registration methods that took them. A module that implemented one supplies a `Middleware` instead, whose action carries the same decisions. A browser module's attribute on the script bundle's tag is stated with `with_bundle_tag_attribute`. - A pushed configuration keeps DataDome's two deprecated secret store selectors, `server_side_key_secret_store` and `protection_test_bypass.credential_secret_store`, where they were removed as the settings were read. DataDome still accepts and discards both, so a configuration that carries one loads as before. - Integration modules have moved out of `trusted-server-core` into crates of their own under `crates///`, each named by that folder, so `crates/cmp/osano` is the module `cmp.osano`. The settings that select and configure a module do not change. `crates/trusted-server-modules` lists the modules a stock build ships from crates of their own, in the order their hooks run. Every adapter builds its state and loads its settings with that list, and `ts config validate` and `ts config push` run each stock module's own rules. Code that named a moved module under `trusted_server_core::integrations` names its crate instead, such as `trusted_server_cmp_osano`. Moved: Osano, Lockr, Permutive, Sourcepoint, Didomi, Google Tag Manager, Testlight, Next.js, Google Publisher Tags and its diagnostics, DataDome, the mock ad server, APS, Prebid, the Prebid Server demand and the plain OpenRTB demand. Core keeps the JavaScript asset proxy, which is its own. - The Fastly adapter is a library with a thin binary over it. A deployment that ships a vendor crate writes a binary of its own that calls `trusted_server_adapter_fastly::run_with(vec![...])` with that crate's integration builder, in place of editing the adapter. The builders are composed with the built-in ones, and a builder runs only when the settings select its module. The Wasm artifact keeps its name, `trusted-server-adapter-fastly.wasm`. @@ -62,6 +64,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- The origin's address in a page's `href`, `src`, `action`, `srcset` and `imagesrcset` is now replaced in one pass. A request host that begins with the origin host, such as `publisher.example.localhost` for `publisher.example`, was matched again inside its own replacement, so `https://publisher.example/` served over `http` came out as `http://publisher.example.localhost.localhost.localhost/`. A `srcset` served on a `www.` host in front of an apex origin came out with `www.www.`. - Publisher responses now map origin URLs in `Location`, `Content-Location`, `Refresh`, and `Link` headers to the serving host, and each CSP source naming the origin gains a serving-host source beside it (#1128). A redirect to the current URL under a different scheme, or a `GET`/`HEAD` redirect to the current URL that sets no cookie (or any such redirect under a `Host` override), is left unchanged to avoid a loop. `` refresh and CSP tags in the page body are not rewritten. See "Origin URLs in proxied response headers" in the configuration guide. - Image Optimizer configuration now rejects profile and profile-set names within the same map that collide after trimming, even when their values match, instead of silently overwriting an entry. Empty names continue to be discarded. - Browser integration bundles now share context providers and the configured log level with TSJS core (#1196). With Permutive enabled and cohorts available, `tsjs.requestAds()` now sends `config.permutive_segments` to `/auction`; the server retains that key only when `auction.allowed_context_keys` includes `"permutive_segments"`. This restores a previously inactive data flow, not a consent gate: review publisher consent handling before relying on it. Integration logs now follow `tsjs.setConfig` and `tsjs.log.setLevel`, and creative no longer raises the default `warn` level to `info`. The `?tsdebug=1` query parameter or `localStorage['tsdebug'] = '1'` intentionally enables `debug` logging across all bundles when creative installs, overriding any log level set earlier through `tsjs.setConfig` or `tsjs.log.setLevel`. Later calls to either API can change the level again. @@ -73,9 +76,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `ts audit generate` writes the `[[fetch]]` and `[[serve]]` entries that run the page changes of the modules it selects on every HTML page, and `trusted-server.example.toml` documents the entries with every middleware the stock modules supply. `trusted-server-modules` lists those middleware with `middleware()` and `middleware_for(...)`. +- Page changes as middleware. A module registers a change it makes to a page with `.with_middleware(...)`, and the ordered `[[fetch]]` and `[[serve]]` entries of the settings say which pages each one runs on and in what order. A fetch middleware runs on the page as the origin sent it and what it writes is stored with a shared template. A serve middleware runs on each reader's copy, whether the page came from the store or from the origin, and nothing it writes is stored. Where text a middleware writes into a script would end the script, or begin `` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery. - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary. - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape. diff --git a/crates/ad-tag/google/.gitattributes b/crates/ad-tag/google/.gitattributes new file mode 100644 index 0000000000..edbb160800 --- /dev/null +++ b/crates/ad-tag/google/.gitattributes @@ -0,0 +1,9 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf + +# The bootstrap script is embedded as it is on disk and written into the head +# of every page, so it must arrive with the line endings it was committed +# with on every platform. +src/gpt_bootstrap.js text eol=lf diff --git a/crates/ad-tag/google/src/diagnostics.rs b/crates/ad-tag/google/src/diagnostics.rs index 3299a90584..29cc5d2cf6 100644 --- a/crates/ad-tag/google/src/diagnostics.rs +++ b/crates/ad-tag/google/src/diagnostics.rs @@ -16,8 +16,10 @@ use validator::Validate; use trusted_server_core::error::TrustedServerError; use trusted_server_core::http_util::is_navigation_request; use trusted_server_core::integrations::{ - IntegrationBuilder, IntegrationHeadInjector, IntegrationHtmlContext, IntegrationRegistration, - IntegrationRequestState, + IntegrationBuilder, IntegrationDocumentState, IntegrationRegistration, IntegrationRequestState, +}; +use trusted_server_core::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, }; use trusted_server_core::response_privacy::enforce_terminal_private_cache_privacy; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -255,44 +257,51 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(GPT_DIAGNOSTICS_INTEGRATION_ID) .with_standalone_js() - .with_head_injector(Arc::new(DiagnosticsHead)) + .with_middleware(Arc::new(DiagnosticsHead)) .build(), )) } -/// Writes what an active decision adds to a document's head. +/// Writes what an active decision adds to a reader's copy of a document, on +/// the pages a `[[serve]]` entry names [`MODULE`] for. +/// +/// It runs for each reader, and not once for a stored page, because the +/// decision is made for one request. struct DiagnosticsHead; impl DiagnosticsHead { /// The decision the preparer left for this document, when it left one. - fn decision(ctx: &IntegrationHtmlContext<'_>) -> Option> { - ctx.document_state - .get::(GPT_DIAGNOSTICS_INTEGRATION_ID) + fn decision( + document_state: &IntegrationDocumentState, + ) -> Option> { + document_state.get::(GPT_DIAGNOSTICS_INTEGRATION_ID) } } -impl IntegrationHeadInjector for DiagnosticsHead { - fn integration_id(&self) -> &'static str { - GPT_DIAGNOSTICS_INTEGRATION_ID +impl Middleware for DiagnosticsHead { + fn middleware_id(&self) -> &'static str { + MODULE } - fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - Self::decision(ctx) - .and_then(|decision| decision.bootstrap_script()) - .into_iter() - .collect() + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] } - /// The module loads synchronously after the bundle, so its listeners are - /// in place before the publisher's scripts in the origin head run. - fn after_bundle_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - Self::decision(ctx) - .and_then(|decision| { - JsModulePart::compile_time(GPT_DIAGNOSTICS_INTEGRATION_ID) - .and_then(|module| decision.module_script_tag(&module)) - }) - .into_iter() - .collect() + /// The bootstrap goes ahead of the bundle. The module loads synchronously + /// after the bundle, so its listeners are in place before the publisher's + /// scripts in the origin head run. + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + let Some(decision) = Self::decision(context.document_state) else { + return MiddlewareAction::pass(); + }; + MiddlewareAction { + head_inserts: decision.bootstrap_script().into_iter().collect(), + after_bundle_inserts: JsModulePart::compile_time(GPT_DIAGNOSTICS_INTEGRATION_ID) + .and_then(|module| decision.module_script_tag(&module)) + .into_iter() + .collect(), + ..MiddlewareAction::pass() + } } } @@ -511,9 +520,16 @@ mod tests { use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::test_support::tests::create_test_settings; + /// Settings that select the module and run its middleware on every + /// reader's copy of a page. fn settings() -> Settings { let mut settings = create_test_settings(); settings.select_module("ad-tag", "ad-tag.google.diagnostics"); + trusted_server_core::html_processor::test_support::place_on_every_page( + &mut settings, + MiddlewarePhase::Serve, + &[MODULE], + ); settings } @@ -826,7 +842,8 @@ mod tests { #[test] fn an_active_document_loads_the_module_after_the_unified_bundle_once() { - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::create_page_processor; use trusted_server_core::streaming_processor::StreamProcessor as _; let settings = settings(); @@ -846,7 +863,7 @@ mod tests { "https", ) .with_request_state(IntegrationRequestState::of(&request)); - let mut processor = create_html_processor(config); + let mut processor = create_page_processor(&settings, ®istry, config); let output = processor .process_chunk( @@ -895,7 +912,8 @@ mod tests { #[test] fn an_inactive_document_gets_neither_script() { - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::create_page_processor; use trusted_server_core::streaming_processor::StreamProcessor as _; let settings = settings(); @@ -911,7 +929,7 @@ mod tests { "https", ) .with_request_state(IntegrationRequestState::of(&request)); - let mut processor = create_html_processor(config); + let mut processor = create_page_processor(&settings, ®istry, config); let output = processor .process_chunk( diff --git a/crates/ad-tag/google/src/fixtures/page-change.input.html b/crates/ad-tag/google/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..6a9f667691 --- /dev/null +++ b/crates/ad-tag/google/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Google Publisher Tag + + + + + + + +An article on the origin + +
+ + diff --git a/crates/ad-tag/google/src/fixtures/page-change.recorded.html b/crates/ad-tag/google/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..3d3a347ffd --- /dev/null +++ b/crates/ad-tag/google/src/fixtures/page-change.recorded.html @@ -0,0 +1,947 @@ + + + + +Google Publisher Tag + + + + + + + +An article on the origin + +
+ + diff --git a/crates/ad-tag/google/src/fixtures/page-change.settings.toml b/crates/ad-tag/google/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..50ef4c9659 --- /dev/null +++ b/crates/ad-tag/google/src/fixtures/page-change.settings.toml @@ -0,0 +1,13 @@ +[ad-tag] +modules = ["google", "google.diagnostics"] + +[ad-tag.google] +gam_attribution_enabled = false + +[[fetch]] +media_type = "text/html" +middleware = ["ad-tag.google"] + +[[serve]] +media_type = "text/html" +middleware = ["ad-tag.google.diagnostics"] diff --git a/crates/ad-tag/google/src/lib.rs b/crates/ad-tag/google/src/lib.rs index 9d29c6b780..8017393fc4 100644 --- a/crates/ad-tag/google/src/lib.rs +++ b/crates/ad-tag/google/src/lib.rs @@ -18,7 +18,7 @@ //! //! ## How It Works //! -//! 1. **HTML rewriting** – The [`IntegrationAttributeRewriter`] swaps `src`/`href` +//! 1. **HTML rewriting** – The module's middleware swaps `src`/`href` //! attributes pointing at Google's GPT script with a first-party URL //! (`/integrations/gpt/script`). //! 2. **Script proxy** – [`IntegrationProxy`] endpoints serve `gpt.js` @@ -49,6 +49,7 @@ pub mod diagnostics; +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -64,9 +65,11 @@ use trusted_server_core::constants::{ }; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationEndpoint, IntegrationHeadInjector, IntegrationHtmlContext, IntegrationProxy, - IntegrationRegistration, + AttributeRewriteAction, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{ProxyRequestConfig, proxy_request}; @@ -431,13 +434,13 @@ pub fn register( return Ok(None); }; - Ok(Some( - IntegrationRegistration::builder(GPT_INTEGRATION_ID) - .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_head_injector(integration) - .build(), - )) + let mut registration = IntegrationRegistration::builder(GPT_INTEGRATION_ID) + .with_proxy(integration.clone()) + .with_middleware(Arc::new(PageChange(integration.clone()))); + if let Some((name, value)) = integration.bundle_tag_attribute() { + registration = registration.with_bundle_tag_attribute(name, value); + } + Ok(Some(registration.build())) } impl GptIntegration { @@ -487,25 +490,39 @@ impl IntegrationProxy for GptIntegration { } } -impl IntegrationAttributeRewriter for GptIntegration { - fn integration_id(&self) -> &'static str { - GPT_INTEGRATION_ID +/// GPT's change to a page, on the pages a `[[fetch]]` entry names [`MODULE`] +/// for. It writes the `tsjs.adInit` bootstrap into the head and, when +/// `rewrite_script` is set, points the GPT script's address at the +/// first-party path. +struct PageChange(Arc); + +impl Middleware for PageChange { + fn middleware_id(&self) -> &'static str { + MODULE } - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_script && matches!(attribute, "src" | "href") + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if !self.config.rewrite_script { - return AttributeRewriteAction::keep(); + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let mut action = MiddlewareAction { + head_inserts: self.0.head_markup(), + ..MiddlewareAction::pass() + }; + if self.0.config.rewrite_script { + let decide: Rc = + Rc::new(|matched| GptIntegration::rewrite_script_address(matched.value)); + action.element_handlers = AttributeRewrite::each(&["src", "href"], &decide); } + action + } +} +impl GptIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party path when it is the GPT script's address. + fn rewrite_script_address(attr_value: &str) -> AttributeRewriteAction { if Self::is_gpt_script_url(attr_value) { // Root-relative so the browser resolves it against the page host. // Note: a page-level `` participates in this resolution, @@ -517,14 +534,8 @@ impl IntegrationAttributeRewriter for GptIntegration { AttributeRewriteAction::keep() } } -} - -impl IntegrationHeadInjector for GptIntegration { - fn integration_id(&self) -> &'static str { - GPT_INTEGRATION_ID - } - /// Injects the `tsjs.adInit` bootstrap script into ``. + /// The `tsjs.adInit` bootstrap scripts written into ``. /// /// ## Scroll / refresh handoff contract (Phase 1) /// @@ -541,7 +552,7 @@ impl IntegrationHeadInjector for GptIntegration { /// auction via `GET /_ts/page-bids` on pushState / replaceState / popstate /// route changes (see `auction/endpoints.rs`). /// The `POST /auction` endpoint is not involved in scroll or refresh flows. - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { + fn head_markup(&self) -> Vec { let gam_attribution_flag = if self.config.gam_attribution_enabled { "window.__tsjs_gam_attribution_enabled=true;" } else { @@ -571,12 +582,12 @@ impl IntegrationHeadInjector for GptIntegration { scripts } - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - if self.config.gam_attribution_enabled { - vec![("data-ts-gam-attribution", "true")] - } else { - Vec::new() - } + /// The attribute on the script bundle's tag that lets the bundle apply + /// GAM attribution, when `gam_attribution_enabled` is set. + fn bundle_tag_attribute(&self) -> Option<(&'static str, &'static str)> { + self.config + .gam_attribution_enabled + .then_some(("data-ts-gam-attribution", "true")) } } @@ -622,14 +633,14 @@ mod tests { } } - fn test_context() -> IntegrationAttributeContext<'static> { - IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - } + /// What the module's middleware decides to do with a page. + fn page_action(integration: Arc) -> MiddlewareAction { + let document_state = IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + PageChange(integration).create(&context) } fn build_http_request(method: Method, uri: &str) -> http::Request { @@ -713,17 +724,12 @@ mod tests { ); } - // -- Attribute rewriter -- + // -- Element handler -- #[test] - fn attribute_rewriter_rewrites_gpt_urls() { - let integration = GptIntegration::new(test_config()); - let ctx = test_context(); - - let result = integration.rewrite( - "src", + fn element_handler_rewrites_gpt_urls() { + let result = GptIntegration::rewrite_script_address( "https://securepubads.g.doubleclick.net/tag/js/gpt.js", - &ctx, ); match result { @@ -738,11 +744,8 @@ mod tests { } #[test] - fn attribute_rewriter_keeps_non_gpt_urls() { - let integration = GptIntegration::new(test_config()); - let ctx = test_context(); - - let result = integration.rewrite("src", "https://cdn.example.com/analytics.js", &ctx); + fn element_handler_keeps_non_gpt_urls() { + let result = GptIntegration::rewrite_script_address("https://cdn.example.com/analytics.js"); assert_eq!( result, @@ -752,50 +755,45 @@ mod tests { } #[test] - fn attribute_rewriter_noop_when_disabled() { + fn element_handler_noop_when_disabled() { let config = GptConfig { rewrite_script: false, ..test_config() }; - let integration = GptIntegration::new(config); - let ctx = test_context(); + let action = page_action(GptIntegration::new(config)); - let result = integration.rewrite( - "src", - "https://securepubads.g.doubleclick.net/tag/js/gpt.js", - &ctx, + assert!( + action.element_handlers.is_empty(), + "should judge no address when rewrite_script is disabled" ); - assert_eq!( - result, - AttributeRewriteAction::Keep, - "should keep GPT URLs when rewrite_script is disabled" + action.head_inserts.len(), + 2, + "should still write the bootstrap into the head" ); } #[test] fn handles_attribute_respects_config() { - let enabled = GptIntegration::new(test_config()); - assert!( - enabled.handles_attribute("src"), - "should handle src when rewrite_script is true" - ); - assert!( - enabled.handles_attribute("href"), - "should handle href when rewrite_script is true" - ); - assert!( - !enabled.handles_attribute("action"), - "should not handle action attribute" + let enabled = page_action(GptIntegration::new(test_config())); + let judged: Vec<&str> = enabled + .element_handlers + .iter() + .map(|handler| handler.attribute()) + .collect(); + assert_eq!( + judged, + ["src", "href"], + "should judge src and href, and no other attribute, when rewrite_script is true" ); - let disabled = GptIntegration::new(GptConfig { + let disabled = page_action(GptIntegration::new(GptConfig { rewrite_script: false, ..test_config() - }); + })); assert!( - !disabled.handles_attribute("src"), - "should not handle src when rewrite_script is false" + disabled.element_handlers.is_empty(), + "should judge no attribute when rewrite_script is false" ); } @@ -1205,20 +1203,12 @@ mod tests { ); } - // -- Head injector -- + // -- Head markup -- #[test] - fn head_injector_emits_enable_flag() { + fn head_markup_emits_enable_flag() { let integration = GptIntegration::new(test_config()); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should emit exactly two head inserts"); assert_eq!( @@ -1227,7 +1217,7 @@ mod tests { "should set the enable flag and call the GPT shim activation function" ); assert!( - integration.tsjs_script_tag_attributes().is_empty(), + integration.bundle_tag_attribute().is_none(), "should not authorize GAM attribution metadata by default" ); } @@ -1238,15 +1228,7 @@ mod tests { gam_attribution_enabled: true, ..test_config() }); - let document_state = IntegrationDocumentState::default(); - let context = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&context); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should not add another head insert"); assert!( @@ -1254,8 +1236,8 @@ mod tests { "should activate the early bootstrap marker" ); assert_eq!( - integration.tsjs_script_tag_attributes(), - vec![("data-ts-gam-attribution", "true")], + integration.bundle_tag_attribute(), + Some(("data-ts-gam-attribution", "true")), "should authorize the bundle fallback on the publisher tag" ); } @@ -1264,14 +1246,7 @@ mod tests { fn head_inserts_includes_ts_ad_init_with_synchronous_bids_read() { let config = test_config(); let integration = GptIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let combined = inserts.join(""); assert!(combined.contains("ts.adInit"), "should define tsjs.adInit"); assert!( @@ -1322,14 +1297,7 @@ mod tests { // fallback lives in the Vitest suite (gpt_bootstrap.test.ts). let config = test_config(); let integration = GptIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let combined = integration.head_inserts(&ctx).join(""); + let combined = integration.head_markup().join(""); assert!( combined.contains("ts.scheduleInitialAdInit"), "should install the fallback scheduler for bundle-load failures" @@ -1356,14 +1324,7 @@ mod tests { fn head_inserts_bootstrap_uses_css_safe_div_prefix_lookup() { let config = test_config(); let integration = GptIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let combined = integration.head_inserts(&ctx).join(""); + let combined = integration.head_markup().join(""); assert!( combined.contains("querySelectorAll(\"[id]\")"), "bootstrap should scan ID-bearing elements instead of interpolating div_id into CSS" @@ -1381,14 +1342,7 @@ mod tests { #[test] fn head_inserts_bootstrap_installs_inner_div_slot_handoff() { let integration = GptIntegration::new(test_config()); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let combined = integration.head_inserts(&ctx).join(""); + let combined = integration.head_markup().join(""); assert!( combined.contains("gptSlotHandoffs"), "bootstrap should keep late publisher slot handoff state on window.tsjs" @@ -1411,14 +1365,7 @@ mod tests { fn head_inserts_bootstrap_guards_enable_services_with_idempotency_flag() { let config = test_config(); let integration = GptIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let combined = integration.head_inserts(&ctx).join(""); + let combined = integration.head_markup().join(""); assert!( combined.contains("ts.servicesEnabled"), "should guard enableServices/enableSingleRequest with the tsjs.servicesEnabled flag" @@ -1438,14 +1385,7 @@ mod tests { // slots or they render blank. let config = test_config(); let integration = GptIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let combined = integration.head_inserts(&ctx).join(""); + let combined = integration.head_markup().join(""); assert!( combined.contains("gpt.setConfig"), "bootstrap should wrap googletag.setConfig() to detect the disabled state" @@ -1498,11 +1438,36 @@ mod tests { } #[test] - fn head_injector_integration_id() { + fn the_middleware_is_named_as_the_module() { let integration = GptIntegration::new(test_config()); + assert_eq!(PageChange(integration).middleware_id(), MODULE); + } + + #[test] + fn a_registration_states_the_attribution_attribute_only_when_it_is_enabled() { + let attributes_with = |gam_attribution_enabled: bool| { + let mut settings = create_test_settings(); + settings + .insert_module_config( + "ad-tag", + MODULE, + &serde_json::json!({ "gam_attribution_enabled": gam_attribution_enabled }), + ) + .expect("should select GPT"); + register(&settings) + .expect("should register GPT") + .expect("should run GPT") + .bundle_tag_attributes + }; + assert_eq!( - IntegrationHeadInjector::integration_id(integration.as_ref()), - "gpt" + attributes_with(true), + [("data-ts-gam-attribution", "true")], + "should put the attribute on the bundle's tag" + ); + assert!( + attributes_with(false).is_empty(), + "should put nothing on the bundle's tag by default" ); } @@ -1513,15 +1478,7 @@ mod tests { ..test_config() }; let integration = GptIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!( inserts.len(), @@ -1543,15 +1500,7 @@ mod tests { ..test_config() }; let integration = GptIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); // The injected `` must be neutralised: the only // `` left is the tag's own legitimate closer. @@ -1576,15 +1525,7 @@ mod tests { #[test] fn head_inserts_omits_slim_prebid_url_when_not_configured() { let integration = GptIntegration::new(test_config()); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!( inserts.len(), @@ -1607,4 +1548,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder(), super::diagnostics::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/auction/aps/.gitattributes b/crates/auction/aps/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/auction/aps/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/auction/aps/src/fixtures/page-change.input.html b/crates/auction/aps/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..6a58b86ede --- /dev/null +++ b/crates/auction/aps/src/fixtures/page-change.input.html @@ -0,0 +1,13 @@ + + + + +APS + + + +An article on the origin + +
+ + diff --git a/crates/auction/aps/src/fixtures/page-change.recorded.html b/crates/auction/aps/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..c33ea015d7 --- /dev/null +++ b/crates/auction/aps/src/fixtures/page-change.recorded.html @@ -0,0 +1,13 @@ + + + + +APS + + + +An article on the origin + +
+ + diff --git a/crates/auction/aps/src/fixtures/page-change.settings.toml b/crates/auction/aps/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..0bc023ecf3 --- /dev/null +++ b/crates/auction/aps/src/fixtures/page-change.settings.toml @@ -0,0 +1,9 @@ +[demand] +modules = ["aps_main"] + +[demand.aps_main] +implementation = "auction.aps" +endpoint = "https://aps.example.com/e/pb/bid" +routing = "all_eligible" +account_id = "example-aps-account-id" +rendering_mode = "publisher_native" diff --git a/crates/auction/aps/src/lib.rs b/crates/auction/aps/src/lib.rs index 7d8692d2ff..3d7b8b11f1 100644 --- a/crates/auction/aps/src/lib.rs +++ b/crates/auction/aps/src/lib.rs @@ -44,8 +44,8 @@ use trusted_server_core::auction::types::{AdSlot, AuctionContext, AuctionRequest use trusted_server_core::auction::types::{AuctionResponse, Bid, BidRenderer, MediaType}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - IntegrationEndpoint, IntegrationHeadInjector, IntegrationHtmlContext, IntegrationProxy, - IntegrationRegistration, UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_response_bounded, + IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, + UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_response_bounded, }; #[cfg(test)] use trusted_server_core::integrations::{ @@ -2078,22 +2078,10 @@ impl IntegrationProxy for ApsRendererIntegration { } } -impl IntegrationHeadInjector for ApsRendererIntegration { - fn integration_id(&self) -> &'static str { - APS_INTEGRATION_ID - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - Vec::new() - } - - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - (self.rendering_mode == ApsRenderingMode::PublisherNative) - .then_some(("data-ts-aps-rendering-mode", "publisher_native")) - .into_iter() - .collect() - } -} +/// The attribute on the script bundle's tag that lets the browser render an +/// APS creative itself. It is written by the server alone, so a page cannot +/// claim the mode for itself. +const NATIVE_RENDERING_ATTRIBUTE: (&str, &str) = ("data-ts-aps-rendering-mode", "publisher_native"); /// Register renderer support when the plan selects an APS demand source. /// @@ -2134,14 +2122,13 @@ pub fn register_for_plan( let Some((_, rendering_mode)) = selected else { return Ok(None); }; - let integration = Arc::new(ApsRendererIntegration { rendering_mode }); - let registration = IntegrationRegistration::builder(APS_INTEGRATION_ID) - .without_js() - .with_head_injector(integration.clone()); - let registration = if rendering_mode == ApsRenderingMode::TrustedServer { - registration.with_proxy(integration) - } else { - registration + let registration = IntegrationRegistration::builder(APS_INTEGRATION_ID).without_js(); + let registration = match rendering_mode { + ApsRenderingMode::TrustedServer => { + registration.with_proxy(Arc::new(ApsRendererIntegration { rendering_mode })) + } + ApsRenderingMode::PublisherNative => registration + .with_bundle_tag_attribute(NATIVE_RENDERING_ATTRIBUTE.0, NATIVE_RENDERING_ATTRIBUTE.1), }; Ok(Some(registration.build())) } @@ -2154,7 +2141,6 @@ mod tests { use trusted_server_core::auction::types::{ AdFormat, AdSlot, AuctionContext, AuctionRequest, BidStatus, PublisherInfo, UserInfo, }; - use trusted_server_core::integrations::IntegrationDocumentState; use trusted_server_core::platform::test_support::{ StubHttpClient, build_services_with_http_client, noop_services, }; @@ -3280,11 +3266,8 @@ mod tests { 1, "a selected APS source should register the trusted-server renderer route" ); - assert_eq!(registration.head_injectors.len(), 1); assert!( - registration.head_injectors[0] - .tsjs_script_tag_attributes() - .is_empty(), + registration.bundle_tag_attributes.is_empty(), "the default rendering mode should not authorize publisher-native rendering" ); assert!(registration.js_disabled); @@ -3317,7 +3300,16 @@ mod tests { registration.proxies.is_empty(), "publisher-native rendering should not register the static renderer" ); - assert_eq!(registration.head_injectors.len(), 1); + assert!( + registration.middleware.is_empty(), + "should change no page, so a page cannot be made to carry a forged native-mode \ + marker" + ); + assert_eq!( + registration.bundle_tag_attributes, + [("data-ts-aps-rendering-mode", "publisher_native")], + "should authorize native mode on the publisher bundle tag" + ); let integration = ApsRendererIntegration { rendering_mode: ApsRenderingMode::PublisherNative, @@ -3326,22 +3318,6 @@ mod tests { integration.routes().is_empty(), "should expose no renderer route" ); - let document_state = IntegrationDocumentState::default(); - let context = IntegrationHtmlContext { - request_host: "publisher.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - assert!( - integration.head_inserts(&context).is_empty(), - "should not inject a forgeable native-mode marker" - ); - assert_eq!( - integration.tsjs_script_tag_attributes(), - vec![("data-ts-aps-rendering-mode", "publisher_native")], - "should authorize native mode on the publisher bundle tag" - ); } #[test] @@ -4639,4 +4615,21 @@ mod engine_tests { .expect("should serialize routing metadata"); assert!(!routing.contains("example-bidder") && !routing.contains("private")); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/auction/prebid/.gitattributes b/crates/auction/prebid/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/auction/prebid/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/auction/prebid/src/fixtures/page-change.input.html b/crates/auction/prebid/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..c7454e5cfc --- /dev/null +++ b/crates/auction/prebid/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Prebid + + + + + + + +An article on the origin + +A link whose path matches + + diff --git a/crates/auction/prebid/src/fixtures/page-change.recorded.html b/crates/auction/prebid/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..b8d0b8e4eb --- /dev/null +++ b/crates/auction/prebid/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Prebid + + + + + + + +An article on the origin + + + + diff --git a/crates/auction/prebid/src/fixtures/page-change.settings.toml b/crates/auction/prebid/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..9e65dba815 --- /dev/null +++ b/crates/auction/prebid/src/fixtures/page-change.settings.toml @@ -0,0 +1,15 @@ +[proxy] +allowed_domains = ["assets.example"] + +[auction] +modules = ["prebid"] + +[auction.prebid] +external_bundle_url = "https://assets.example/prebid/trusted-prebid.js" +timeout_ms = 1000 +script_patterns = ["/prebid.js", "/prebid.min.js"] +debug = false + +[[fetch]] +media_type = "text/html" +middleware = ["auction.prebid"] diff --git a/crates/auction/prebid/src/lib.rs b/crates/auction/prebid/src/lib.rs index 40489e459f..9ab8cf69f0 100644 --- a/crates/auction/prebid/src/lib.rs +++ b/crates/auction/prebid/src/lib.rs @@ -18,6 +18,7 @@ )] use std::collections::HashSet; +use std::rc::Rc; use std::sync::{Arc, LazyLock}; use async_trait::async_trait; @@ -41,9 +42,11 @@ use trusted_server_core::auction::plan::AuctionPlan; use trusted_server_core::cache_policy::{CacheControlPolicy, EdgeCacheHeader}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationEndpoint, IntegrationHeadInjector, IntegrationHtmlContext, IntegrationProxy, - IntegrationRegistration, + AttributeRewriteAction, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{ProxyRequestConfig, is_host_allowed, proxy_request}; @@ -1100,8 +1103,7 @@ pub fn register_for_plan( Ok(Some( IntegrationRegistration::builder(PREBID_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_head_injector(integration) + .with_middleware(Arc::new(PageChange(integration))) .with_deferred_js() .build(), )) @@ -1119,8 +1121,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(PREBID_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_head_injector(integration) + .with_middleware(Arc::new(PageChange(integration))) .with_deferred_js() .build(), )) @@ -1186,22 +1187,38 @@ impl IntegrationProxy for PrebidIntegration { } } -impl IntegrationAttributeRewriter for PrebidIntegration { - fn integration_id(&self) -> &'static str { - PREBID_INTEGRATION_ID +/// Prebid's change to a page, on the pages a `[[fetch]]` entry names +/// [`MODULE`] for. It writes the configuration Prebid.js reads and the tag +/// that loads the bundle into the head, and removes the publisher's own +/// Prebid script. +struct PageChange(Arc); + +impl Middleware for PageChange { + fn middleware_id(&self) -> &'static str { + MODULE } - fn handles_attribute(&self, attribute: &str) -> bool { - matches!(attribute, "src" | "href") + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if self.matches_script_url(attr_value) { + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_script_address(matched.value)); + MiddlewareAction { + head_inserts: self.0.head_markup(), + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } + } +} + +impl PrebidIntegration { + /// What becomes of a `src` or an `href`, whose element is removed when it + /// loads the publisher's own Prebid script. + fn rewrite_script_address(&self, value: &str) -> AttributeRewriteAction { + if self.matches_script_url(value) { AttributeRewriteAction::remove_element() } else { AttributeRewriteAction::keep() @@ -1225,12 +1242,10 @@ fn injected_prebid_config_script(config_json: &str) -> String { ) } -impl IntegrationHeadInjector for PrebidIntegration { - fn integration_id(&self) -> &'static str { - PREBID_INTEGRATION_ID - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { +impl PrebidIntegration { + /// The scripts written into the head, being the configuration Prebid.js + /// reads and the tag that loads the bundle. + fn head_markup(&self) -> Vec { if let Some(inserts) = &self.planned_head_inserts { return inserts.clone(); } @@ -1350,7 +1365,10 @@ mod tests { use trusted_server_core::auction::plan::{BidderId, BidderRouteConfig, ProviderId}; - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::{ AttributeRewriteAction, IntegrationDocumentState, IntegrationRegistry, }; @@ -1417,6 +1435,7 @@ mod tests { }), ) .expect("should select Prebid"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); settings } @@ -1854,14 +1873,21 @@ server_url = "https://prebid.example/openrtb2/auction" IntegrationRegistry::with_plan_and_registrations(&settings, plan, &[builder()]) .expect("should build integration registry"); let document_state = IntegrationDocumentState::default(); - let context = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); - let inserts = registry.head_inserts(&context); + let inserts = registry + .middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + trusted_server_core::middleware::HTML_MEDIA_TYPE, + "/", + ) + .plan(&context) + .expect("should plan the page's middleware") + .head_inserts; let config_insert = inserts .iter() .find(|insert| insert.contains("window.__tsjs_prebid")) @@ -2136,36 +2162,20 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] } #[test] - fn attribute_rewriter_removes_prebid_scripts() { + fn element_handler_removes_prebid_scripts() { let integration = PrebidIntegration::new(base_config()); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - }; - - let rewritten = integration.rewrite("src", "https://cdn.prebid.org/prebid.min.js", &ctx); + let rewritten = integration.rewrite_script_address("https://cdn.prebid.org/prebid.min.js"); assert!(matches!(rewritten, AttributeRewriteAction::RemoveElement)); - let untouched = integration.rewrite("src", "https://cdn.example.com/app.js", &ctx); + let untouched = integration.rewrite_script_address("https://cdn.example.com/app.js"); assert!(matches!(untouched, AttributeRewriteAction::Keep)); } #[test] - fn attribute_rewriter_handles_query_strings_and_links() { + fn element_handler_handles_query_strings_and_links() { let integration = PrebidIntegration::new(base_config()); - let ctx = IntegrationAttributeContext { - attribute_name: "href", - element_name: "a", - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - }; - let rewritten = - integration.rewrite("href", "https://cdn.prebid.org/prebid.js?v=1.2.3", &ctx); + integration.rewrite_script_address("https://cdn.prebid.org/prebid.js?v=1.2.3"); assert!(matches!(rewritten, AttributeRewriteAction::RemoveElement)); } @@ -2192,7 +2202,7 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -2241,7 +2251,7 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -2963,17 +2973,9 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_emits_config_script() { + fn head_markup_emits_config_script() { let integration = PrebidIntegration::new(base_config()); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should produce config and bundle inserts"); let script = &inserts[0]; @@ -3028,17 +3030,15 @@ external_bundle_sri = "sha384-AAAA" let plan = trusted_server_core::auction::compile_auction_plan(&settings) .expect("should compile auction plan"); let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); for enabled in [true, false] { let registration = register_for_plan(&settings, &plan.clone().with_enabled(enabled)) .expect("should register prebid") .expect("should enable prebid"); - let inserts = registration.head_injectors[0].head_inserts(&ctx); + let inserts = registration.middleware[0].create(&context).head_inserts; let script = &inserts[0]; assert!( script.contains(r#""managedUserIds":[{"name":"exampleId""#), @@ -3060,7 +3060,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_includes_managed_user_ids() { + fn head_markup_includes_managed_user_ids() { let mut config = base_config(); config.managed_user_ids = vec![PrebidManagedUserIdConfig { name: "exampleId".to_string(), @@ -3076,15 +3076,7 @@ external_bundle_sri = "sha384-AAAA" }), }]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3096,7 +3088,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_omits_optional_managed_user_id_fields_when_unset() { + fn head_markup_omits_optional_managed_user_id_fields_when_unset() { let mut config = base_config(); config.managed_user_ids = vec![PrebidManagedUserIdConfig { name: "exampleId".to_string(), @@ -3104,15 +3096,7 @@ external_bundle_sri = "sha384-AAAA" storage: None, }]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3122,17 +3106,9 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_omits_managed_user_ids_when_none_configured() { + fn head_markup_omits_managed_user_ids_when_none_configured() { let integration = PrebidIntegration::new(base_config()); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3207,7 +3183,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_escapes_script_breakout_in_managed_user_ids() { + fn head_markup_escapes_script_breakout_in_managed_user_ids() { let mut config = base_config(); config.managed_user_ids = vec![PrebidManagedUserIdConfig { params: serde_json::Map::from_iter([( @@ -3217,15 +3193,7 @@ external_bundle_sri = "sha384-AAAA" ..valid_managed_user_id() }]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3248,20 +3216,12 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_includes_excluded_gam_ad_unit_path_suffixes() { + fn head_markup_includes_excluded_gam_ad_unit_path_suffixes() { let mut config = base_config(); config.excluded_gam_ad_unit_path_suffixes = vec!["/trackingonly".to_string(), "/measurement-only".to_string()]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains( @@ -3273,19 +3233,11 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_handles_missing_account_id() { + fn head_markup_handles_missing_account_id() { let mut config = base_config(); config.account_id = None; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains(r#""accountId":"""#), @@ -3295,7 +3247,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_emits_external_bundle_script_with_hash_and_integrity() { + fn head_markup_emits_external_bundle_script_with_hash_and_integrity() { let sha256 = "a".repeat(64); let mut config = base_config(); config.external_bundle_url = @@ -3303,15 +3255,7 @@ external_bundle_sri = "sha384-AAAA" config.external_bundle_sha256 = Some(sha256.clone()); config.external_bundle_sri = Some(test_sri("sha384", &[0; 48])); let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should emit config and bundle scripts"); assert!( @@ -3332,20 +3276,12 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_emits_external_bundle_script_without_hash_query_when_unhashed() { + fn head_markup_emits_external_bundle_script_without_hash_query_when_unhashed() { let mut config = base_config(); config.external_bundle_url = Some("https://assets.example/prebid/trusted-prebid.js".to_string()); let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should emit config and bundle scripts"); assert!( @@ -3361,19 +3297,11 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_escapes_less_than_signs_in_values() { + fn head_markup_escapes_less_than_signs_in_values() { let mut config = base_config(); config.account_id = Some("".to_string()); let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains(r#""accountId":"\u003c/script>\u003cscript>alert(1)\u003c/script>""#), @@ -3383,17 +3311,9 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_omits_client_side_bidders_when_empty() { + fn head_markup_omits_client_side_bidders_when_empty() { let integration = PrebidIntegration::new(base_config()); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( !script.contains("clientSideBidders"), @@ -3403,19 +3323,11 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_includes_client_side_bidders_when_configured() { + fn head_markup_includes_client_side_bidders_when_configured() { let mut config = base_config(); config.client_side_bidders = vec!["rubicon".to_string(), "magnite".to_string()]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains(r#""clientSideBidders":["rubicon","magnite"]"#), @@ -3471,4 +3383,21 @@ server_url = "https://prebid.example" "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/audience/permutive/.gitattributes b/crates/audience/permutive/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/audience/permutive/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/audience/permutive/src/fixtures/page-change.input.html b/crates/audience/permutive/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..cd143b0737 --- /dev/null +++ b/crates/audience/permutive/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Permutive + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/audience/permutive/src/fixtures/page-change.recorded.html b/crates/audience/permutive/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..e2d9fdfbc7 --- /dev/null +++ b/crates/audience/permutive/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Permutive + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/audience/permutive/src/fixtures/page-change.settings.toml b/crates/audience/permutive/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..3463a0e97d --- /dev/null +++ b/crates/audience/permutive/src/fixtures/page-change.settings.toml @@ -0,0 +1,10 @@ +[audience] +modules = ["permutive"] + +[audience.permutive] +organization_id = "myorg" +workspace_id = "abc123" + +[[fetch]] +media_type = "text/html" +middleware = ["audience.permutive"] diff --git a/crates/audience/permutive/src/lib.rs b/crates/audience/permutive/src/lib.rs index 452f85ea05..60c0e4ee20 100644 --- a/crates/audience/permutive/src/lib.rs +++ b/crates/audience/permutive/src/lib.rs @@ -15,6 +15,7 @@ ) )] +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -28,10 +29,13 @@ use validator::Validate; use trusted_server_core::constants::INTERNAL_HEADERS; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, - UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, collect_response_bounded, - ensure_integration_backend, + AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, + IntegrationRegistration, UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, + collect_response_bounded, ensure_integration_backend, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::{PlatformHttpRequest, RuntimeServices}; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -352,7 +356,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(PERMUTIVE_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(SdkAddress(integration))) .build(), )) } @@ -458,26 +462,11 @@ impl IntegrationProxy for PermutiveIntegration { } } -impl IntegrationAttributeRewriter for PermutiveIntegration { - fn integration_id(&self) -> &'static str { - PERMUTIVE_INTEGRATION_ID - } - - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_sdk && matches!(attribute, "src" | "href") - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if !self.config.rewrite_sdk { - return AttributeRewriteAction::keep(); - } - - if self.is_permutive_sdk_url(attr_value) { +impl PermutiveIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party endpoint when it is the SDK's address. + fn rewrite_sdk_address(&self, value: &str) -> AttributeRewriteAction { + if self.is_permutive_sdk_url(value) { // Rewrite to first-party SDK endpoint. // Root-relative so the browser resolves it against the page host. // Note: a page-level `` participates in this resolution, @@ -491,6 +480,33 @@ impl IntegrationAttributeRewriter for PermutiveIntegration { } } +/// Loads the SDK from the first-party endpoint, on the pages a `[[fetch]]` +/// entry names [`MODULE`] for. It does nothing unless `rewrite_sdk` is set. +struct SdkAddress(Arc); + +impl Middleware for SdkAddress { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + if !self.0.config.rewrite_sdk { + return MiddlewareAction::pass(); + } + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_sdk_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } + } +} + // Default value functions fn default_api_endpoint() -> String { "https://api.permutive.com".to_string() @@ -565,7 +581,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_rewrites_sdk_urls() { + fn test_element_handler_rewrites_sdk_urls() { let config = PermutiveConfig { organization_id: "myorg".to_string(), workspace_id: "workspace-123".to_string(), @@ -577,19 +593,8 @@ mod tests { }; let integration = PermutiveIntegration::new(config); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - - let rewritten = integration.rewrite( - "src", - "https://myorg.edge.permutive.app/workspace-123-web.js", - &ctx, - ); + let rewritten = integration + .rewrite_sdk_address("https://myorg.edge.permutive.app/workspace-123-web.js"); assert!(matches!(rewritten, AttributeRewriteAction::Replace(_))); if let AttributeRewriteAction::Replace(url) = rewritten { @@ -598,7 +603,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_noop_when_disabled() { + fn test_element_handler_noop_when_disabled() { let config = PermutiveConfig { organization_id: "myorg".to_string(), workspace_id: "workspace-123".to_string(), @@ -609,22 +614,16 @@ mod tests { rewrite_sdk: false, // Disabled }; let integration = PermutiveIntegration::new(config); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - - let rewritten = integration.rewrite( - "src", - "https://myorg.edge.permutive.app/workspace-123-web.js", - &ctx, + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, ); - assert!(matches!(rewritten, AttributeRewriteAction::Keep)); + assert!( + SdkAddress(integration).create(&context).is_pass(), + "should leave every page alone when SDK rewriting is off" + ); } #[test] @@ -762,6 +761,23 @@ mod tests { ); } + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } + #[test] fn permutive_proxy_does_not_forward_host_signal_headers() { // Every proxied route forwards the reader's custom `x-` headers, so diff --git a/crates/bot-protection/datadome/.gitattributes b/crates/bot-protection/datadome/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/bot-protection/datadome/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/bot-protection/datadome/src/fixtures/page-change.input.html b/crates/bot-protection/datadome/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..add3e2e96b --- /dev/null +++ b/crates/bot-protection/datadome/src/fixtures/page-change.input.html @@ -0,0 +1,16 @@ + + + + +DataDome + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/bot-protection/datadome/src/fixtures/page-change.recorded.html b/crates/bot-protection/datadome/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..3c0e736fae --- /dev/null +++ b/crates/bot-protection/datadome/src/fixtures/page-change.recorded.html @@ -0,0 +1,16 @@ + + + + +DataDome + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/bot-protection/datadome/src/fixtures/page-change.settings.toml b/crates/bot-protection/datadome/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..008b8fe6cb --- /dev/null +++ b/crates/bot-protection/datadome/src/fixtures/page-change.settings.toml @@ -0,0 +1,13 @@ +[bot-protection] +modules = ["datadome"] + +[bot-protection.datadome] +client_side_key = "test-client-key" + +[[fetch]] +media_type = "text/html" +middleware = ["bot-protection.datadome"] + +[[serve]] +media_type = "text/html" +middleware = ["bot-protection.datadome.tag"] diff --git a/crates/bot-protection/datadome/src/lib.rs b/crates/bot-protection/datadome/src/lib.rs index e8994d2ba2..fcba7c68a7 100644 --- a/crates/bot-protection/datadome/src/lib.rs +++ b/crates/bot-protection/datadome/src/lib.rs @@ -50,8 +50,8 @@ //! //! # HTML Attribute Rewriting //! -//! When `rewrite_sdk = true`, the integration implements [`IntegrationAttributeRewriter`] to -//! automatically rewrite `DataDome` script URLs in HTML responses: +//! When `rewrite_sdk = true`, the module's fetch middleware rewrites `DataDome` script URLs in +//! the pages a `[[fetch]]` entry names it for: //! //! - `" )] } -} -impl IntegrationAttributeRewriter for DataDomeIntegration { - fn integration_id(&self) -> &'static str { - DATADOME_INTEGRATION_ID - } - - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_sdk && matches!(attribute, "src" | "href") - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party path when it is a `DataDome` script's address. + fn rewrite_sdk_address(attr_value: &str) -> AttributeRewriteAction { // Check if this is a DataDome script URL let is_datadome = attr_value.contains("js.datadome.co") || attr_value.contains("datadome.co/tags.js"); @@ -1084,8 +1129,8 @@ pub fn register( let mut builder = IntegrationRegistration::builder(DATADOME_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_head_injector(integration.clone()); + .with_middleware(Arc::new(SdkAddress(integration.clone()))) + .with_middleware(Arc::new(ClientTag(integration.clone()))); if integration.config.enable_protection { builder = builder.with_request_filter(integration); @@ -1286,17 +1331,6 @@ mod tests { ); } - fn html_context_for_tests( - document_state: &trusted_server_core::integrations::IntegrationDocumentState, - ) -> IntegrationHtmlContext<'_> { - IntegrationHtmlContext { - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - document_state, - } - } - #[test] fn protection_secrets_are_absent_by_default() { let config = DataDomeConfig::default(); @@ -1517,15 +1551,13 @@ mod tests { } #[test] - fn head_injector_escapes_client_side_tag_url_attribute() { + fn head_markup_escapes_client_side_tag_url_attribute() { let mut config = test_config(); config.client_side_key = "test-client-key".to_string(); config.client_side_tag_url = "/integrations/datadome/tags.js?one=1&two=2".to_string(); let integration = DataDomeIntegration::new(config); - let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); - let ctx = html_context_for_tests(&document_state); - - let inserts = integration.head_inserts(&ctx); + let document_state = IntegrationDocumentState::default(); + let inserts = integration.client_tag(&document_state); assert!( inserts[0].contains( @@ -1536,15 +1568,13 @@ mod tests { } #[test] - fn head_injector_emits_client_side_tag_when_key_configured() { + fn head_markup_emits_client_side_tag_when_key_configured() { let mut config = test_config(); config.client_side_key = "test-client-key".to_string(); config.client_side_configuration = serde_json::json!({ "ajaxListenerPath": true }); let integration = DataDomeIntegration::new(config); - let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); - let ctx = html_context_for_tests(&document_state); - - let inserts = integration.head_inserts(&ctx); + let document_state = IntegrationDocumentState::default(); + let inserts = integration.client_tag(&document_state); assert_eq!(inserts.len(), 1, "should emit one combined DataDome insert"); assert!( @@ -1562,18 +1592,16 @@ mod tests { } #[test] - fn head_injector_omits_client_side_tag_when_disabled_or_blank() { + fn head_markup_omits_client_side_tag_when_disabled_or_blank() { let mut suppressed = test_config(); suppressed.client_side_key = "test-client-key".to_string(); let suppressed_integration = DataDomeIntegration::new(suppressed); - let suppressed_state = - trusted_server_core::integrations::IntegrationDocumentState::default(); + let suppressed_state = IntegrationDocumentState::default(); suppressed_state .get_or_insert_with(DATADOME_INTEGRATION_ID, || DataDomeClientTagSuppressed); - let suppressed_ctx = html_context_for_tests(&suppressed_state); assert!( suppressed_integration - .head_inserts(&suppressed_ctx) + .client_tag(&suppressed_state) .is_empty(), "should omit the tag when the request is IP-excluded" ); @@ -1581,10 +1609,9 @@ mod tests { let mut blank_key = test_config(); blank_key.client_side_key = " ".to_string(); let integration = DataDomeIntegration::new(blank_key); - let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); - let ctx = html_context_for_tests(&document_state); + let document_state = IntegrationDocumentState::default(); assert!( - integration.head_inserts(&ctx).is_empty(), + integration.client_tag(&document_state).is_empty(), "should not inject a tag without a client-side key" ); @@ -1593,7 +1620,7 @@ mod tests { disabled.inject_client_side_tag = false; let integration = DataDomeIntegration::new(disabled); assert!( - integration.head_inserts(&ctx).is_empty(), + integration.client_tag(&document_state).is_empty(), "should not inject a tag when injection is disabled" ); } @@ -1641,24 +1668,50 @@ mod tests { } #[test] - fn attribute_rewriter_matches_datadome() { - let integration = DataDomeIntegration::new(test_config()); + fn the_tag_middleware_is_named_after_the_module() { + assert_eq!( + TAG_MIDDLEWARE, + trusted_server_core::module_name!("tag"), + "should be the module's name with a part of its own" + ); + } - // Should handle both src and href attributes - assert!(integration.handles_attribute("src")); - assert!(integration.handles_attribute("href")); - assert!(!integration.handles_attribute("data-src")); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "publisher.com", - request_scheme: "https", - origin_host: "origin.publisher.com", - }; + #[test] + fn the_sdk_rewrite_is_off_without_rewrite_sdk() { + let mut config = test_config(); + config.rewrite_sdk = false; + let document_state = IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + + assert!( + SdkAddress(DataDomeIntegration::new(config)) + .create(&context) + .is_pass(), + "should leave every page alone when rewrite_sdk is false" + ); + } + + #[test] + fn element_handler_matches_datadome() { + // Should judge both src and href attributes, and no other + let document_state = IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + let action = SdkAddress(DataDomeIntegration::new(test_config())).create(&context); + let judged: Vec<&str> = action + .element_handlers + .iter() + .map(|handler| handler.attribute()) + .collect(); + assert_eq!(judged, ["src", "href"]); // Should rewrite DataDome URLs in src - let action = integration.rewrite("src", "https://js.datadome.co/tags.js", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co/tags.js"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/tags.js"); @@ -1667,7 +1720,7 @@ mod tests { } // Should rewrite DataDome URLs in href (for link preload/prefetch) - let action = integration.rewrite("href", "https://js.datadome.co/tags.js", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co/tags.js"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/tags.js"); @@ -1676,24 +1729,14 @@ mod tests { } // Should not rewrite other URLs - let action = integration.rewrite("src", "https://example.com/script.js", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://example.com/script.js"); assert!(matches!(action, AttributeRewriteAction::Keep)); } #[test] - fn attribute_rewriter_preserves_path() { - let integration = DataDomeIntegration::new(test_config()); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "publisher.com", - request_scheme: "https", - origin_host: "origin.publisher.com", - }; - + fn element_handler_preserves_path() { // Should preserve /js/... paths for signal collection API - let action = integration.rewrite("src", "https://js.datadome.co/js/check", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co/js/check"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/js/check"); @@ -1702,7 +1745,7 @@ mod tests { } // Should handle protocol-relative URLs - let action = integration.rewrite("href", "//js.datadome.co/js/signal", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("//js.datadome.co/js/signal"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/js/signal"); @@ -1711,7 +1754,7 @@ mod tests { } // Bare domain without path should default to /tags.js - let action = integration.rewrite("src", "https://js.datadome.co", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/tags.js"); @@ -1763,7 +1806,10 @@ mod tests { const PUBLISHER_TAGGED_DOCUMENT: &[u8] = br#"content"#; fn process_document(suppress: bool) -> String { - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::{IntegrationRegistry, IntegrationRequestState}; use trusted_server_core::streaming_processor::StreamProcessor as _; @@ -1775,6 +1821,8 @@ mod tests { &serde_json::json!({ "client_side_key": "test-client-key" }), ) .expect("should configure DataDome integration"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); + place_on_every_page(&mut settings, MiddlewarePhase::Serve, &[TAG_MIDDLEWARE]); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create integration registry with DataDome"); let mut request = http::Request::builder() @@ -1792,7 +1840,7 @@ mod tests { "https", ) .with_request_state(IntegrationRequestState::of(&request)); - let mut processor = create_html_processor(config); + let mut processor = create_page_processor(&settings, ®istry, config); let output = processor .process_chunk(PUBLISHER_TAGGED_DOCUMENT, true) @@ -2043,4 +2091,21 @@ mod tests { "should identify the removed field: {error:?}" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/bot-protection/datadome/src/protection.rs b/crates/bot-protection/datadome/src/protection.rs index 17c8760a2f..eb6a0a8936 100644 --- a/crates/bot-protection/datadome/src/protection.rs +++ b/crates/bot-protection/datadome/src/protection.rs @@ -44,9 +44,9 @@ impl DataDomeIntegration { let test_bypass_matched = self.take_protection_test_bypass_header(input.request); if test_bypass_matched { // Both markers travel together. The first is DataDome's own - // tag-suppression signal, left for its head injector. The second tells - // core the response is personalized to this request and cannot be - // shared through a cache or a template. + // tag-suppression signal, left for its serve middleware. The second + // tells core the response is personalized to this request and cannot + // be shared through a cache or a template. super::suppress_client_tag(input.request); input .request @@ -165,9 +165,9 @@ impl DataDomeIntegration { } => { if suppress_client_tag { // Both markers travel together. The first is DataDome's own - // tag-suppression signal, left for its head injector. The second - // tells core the response is personalized to this request and - // cannot be shared through a cache or a template. + // tag-suppression signal, left for its serve middleware. The + // second tells core the response is personalized to this request + // and cannot be shared through a cache or a template. super::suppress_client_tag(input.request); input .request diff --git a/crates/cmp/didomi/.gitattributes b/crates/cmp/didomi/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/cmp/didomi/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/cmp/didomi/src/fixtures/page-change.input.html b/crates/cmp/didomi/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..e6a40bef11 --- /dev/null +++ b/crates/cmp/didomi/src/fixtures/page-change.input.html @@ -0,0 +1,13 @@ + + + + +Didomi + + + + +An article on the origin + + + diff --git a/crates/cmp/didomi/src/fixtures/page-change.recorded.html b/crates/cmp/didomi/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..c7e21c1ab9 --- /dev/null +++ b/crates/cmp/didomi/src/fixtures/page-change.recorded.html @@ -0,0 +1,13 @@ + + + + +Didomi + + + + +An article on the origin + + + diff --git a/crates/cmp/didomi/src/fixtures/page-change.settings.toml b/crates/cmp/didomi/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..7540bfd8a3 --- /dev/null +++ b/crates/cmp/didomi/src/fixtures/page-change.settings.toml @@ -0,0 +1,11 @@ +[cmp] +modules = ["didomi"] + +[cmp.didomi] +geo_query_parameters = false +sdk_origin = "https://sdk.example.com" +api_origin = "https://api.example.com" + +[[fetch]] +media_type = "text/html" +middleware = ["cmp.didomi"] diff --git a/crates/cmp/didomi/src/lib.rs b/crates/cmp/didomi/src/lib.rs index b8b7aa6461..5082608d46 100644 --- a/crates/cmp/didomi/src/lib.rs +++ b/crates/cmp/didomi/src/lib.rs @@ -22,9 +22,11 @@ use validator::{Validate, ValidationError}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationHeadInjector, - IntegrationHtmlContext, IntegrationProxy, IntegrationRegistration, collect_body_bounded, - ensure_integration_backend, + INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, + collect_body_bounded, ensure_integration_backend, +}; +use trusted_server_core::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, }; use trusted_server_core::platform::{GeoInfo, PlatformHttpRequest, RuntimeServices}; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -437,7 +439,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(DIDOMI_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_head_injector(integration) + .with_middleware(Arc::new(ClientConfig(integration))) .build(), )) } @@ -565,12 +567,10 @@ impl IntegrationProxy for DidomiIntegration { } } -impl IntegrationHeadInjector for DidomiIntegration { - fn integration_id(&self) -> &'static str { - DIDOMI_INTEGRATION_ID - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { +impl DidomiIntegration { + /// The script that hands the browser module the path this deployment + /// serves Didomi under. + fn client_config_script(&self) -> String { #[derive(Serialize)] #[serde(rename_all = "camelCase")] struct InjectedDidomiClientConfig { @@ -589,9 +589,28 @@ impl IntegrationHeadInjector for DidomiIntegration { }) .replace("window.__tsjs_didomi={config_json};"# - )] + format!(r#""#) + } +} + +/// Writes the browser module's settings into the head, on the pages a +/// `[[fetch]]` entry names [`MODULE`] for. +struct ClientConfig(Arc); + +impl Middleware for ClientConfig { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + head_inserts: vec![self.0.client_config_script()], + ..MiddlewareAction::pass() + } } } @@ -602,7 +621,7 @@ mod tests { use super::*; use http::Method; - use trusted_server_core::integrations::{IntegrationDocumentState, IntegrationRegistry}; + use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::platform::PlatformCacheIntent; use trusted_server_core::platform::test_support::{ NoopConfigStore, NoopSecretStore, StubBackend, StubHttpClient, @@ -1353,7 +1372,7 @@ mod tests { } #[test] - fn head_injector_emits_proxy_path() { + fn head_markup_emits_proxy_path() { let custom_config = DidomiIntegrationConfig { geo_query_parameters: false, proxy_path: Some("my-consent".to_string()), @@ -1361,18 +1380,25 @@ mod tests { api_origin: default_api_origin(), }; let integration = DidomiIntegration::new(Arc::new(custom_config)); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let inserts = integration.head_inserts(&ctx); - assert_eq!(inserts.len(), 1); + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + + let action = ClientConfig(integration).create(&context); + assert_eq!( - inserts[0], - r#""# + action.head_inserts, + [r#""#], + "should write the one script, with the path this deployment serves Didomi under" + ); + assert!( + action.after_bundle_inserts.is_empty() + && action.element_handlers.is_empty() + && action.text_handlers.is_empty() + && action.stream.is_none(), + "should change nothing else of the page" ); } @@ -1435,18 +1461,11 @@ mod tests { } #[test] - fn head_injector_default_path() { + fn head_markup_default_path() { let integration = DidomiIntegration::new(Arc::new(config())); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let inserts = integration.head_inserts(&ctx); + assert_eq!( - inserts[0], + integration.client_config_script(), r#""# ); } @@ -1459,4 +1478,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/cmp/osano/src/lib.rs b/crates/cmp/osano/src/lib.rs index ec8fdbc83c..6e236b316e 100644 --- a/crates/cmp/osano/src/lib.rs +++ b/crates/cmp/osano/src/lib.rs @@ -120,8 +120,8 @@ mod tests { "Osano v1 should not register Rust proxy routes" ); assert!( - registration.head_injectors.is_empty(), - "Osano v1 should not inject HTML from Rust" + registration.middleware.is_empty(), + "Osano v1 should not change a page from Rust" ); } diff --git a/crates/cmp/sourcepoint/.gitattributes b/crates/cmp/sourcepoint/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/cmp/sourcepoint/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/cmp/sourcepoint/src/fixtures/page-change.input.html b/crates/cmp/sourcepoint/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..66137ff8cb --- /dev/null +++ b/crates/cmp/sourcepoint/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Sourcepoint + + + + + + + +An article on the origin + +A link to another host + + diff --git a/crates/cmp/sourcepoint/src/fixtures/page-change.recorded.html b/crates/cmp/sourcepoint/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..925a486535 --- /dev/null +++ b/crates/cmp/sourcepoint/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Sourcepoint + + + + + + + +An article on the origin + +A link to another host + + diff --git a/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml b/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..7c05107150 --- /dev/null +++ b/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml @@ -0,0 +1,9 @@ +[cmp] +modules = ["sourcepoint"] + +[cmp.sourcepoint] +rewrite_sdk = true + +[[fetch]] +media_type = "text/html" +middleware = ["cmp.sourcepoint"] diff --git a/crates/cmp/sourcepoint/src/lib.rs b/crates/cmp/sourcepoint/src/lib.rs index 4fc401ba96..8b0ea6c8eb 100644 --- a/crates/cmp/sourcepoint/src/lib.rs +++ b/crates/cmp/sourcepoint/src/lib.rs @@ -7,10 +7,10 @@ //! //! | Layer | Mechanism | What it catches | //! |-------|-----------|-----------------| -//! | HTML attributes | `IntegrationAttributeRewriter` | Static `", self.config.rewrite_sdk @@ -1187,7 +1203,7 @@ mod tests { use super::*; use std::sync::atomic::{AtomicUsize, Ordering}; use trusted_server_core::error::IntoHttpResponse as _; - use trusted_server_core::integrations::{IntegrationDocumentState, IntegrationRegistry}; + use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::platform::test_support::{ StubHttpClient, build_services_with_http_client, }; @@ -2014,18 +2030,8 @@ mod tests { #[test] fn rewrites_cdn_urls_to_first_party_paths() { let integration = SourcepointIntegration::new(Arc::new(config())); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - - let rewritten = integration.rewrite( - "src", + let rewritten = integration.rewrite_sdk_address( "https://cdn.privacy-mgmt.com/mms/v2/get_site_data?account_id=821", - &ctx, ); assert_eq!( @@ -2039,16 +2045,9 @@ mod tests { #[test] fn leaves_non_sourcepoint_urls_unchanged() { let integration = SourcepointIntegration::new(Arc::new(config())); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; assert_eq!( - integration.rewrite("src", "https://example.com/script.js", &ctx), + integration.rewrite_sdk_address("https://example.com/script.js"), AttributeRewriteAction::keep() ); } @@ -2314,18 +2313,26 @@ mod tests { } #[test] - fn attribute_rewriter_skips_when_rewrite_disabled() { + fn element_handler_skips_when_rewrite_disabled() { let mut cfg = config(); cfg.rewrite_sdk = false; let integration = SourcepointIntegration::new(Arc::new(cfg)); + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + + let action = PageChange(integration).create(&context); assert!( - !integration.handles_attribute("src"), - "should not handle src when rewrite_sdk is false" + action.element_handlers.is_empty(), + "should judge no address when rewrite_sdk is false" ); - assert!( - !integration.handles_attribute("href"), - "should not handle href when rewrite_sdk is false" + assert_eq!( + action.head_inserts.len(), + 1, + "should still write the browser module's settings" ); } @@ -2355,17 +2362,10 @@ mod tests { } #[test] - fn head_injector_emits_config_script_plus_trap_when_enabled() { + fn head_markup_emits_config_script_plus_trap_when_enabled() { let integration = SourcepointIntegration::new(Arc::new(config())); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "ts.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - document_state: &document_state, - }; - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!( inserts.len(), 2, @@ -2416,19 +2416,12 @@ mod tests { } #[test] - fn head_injector_returns_config_when_rewrite_disabled() { + fn head_markup_returns_config_when_rewrite_disabled() { let mut cfg = config(); cfg.rewrite_sdk = false; let integration = SourcepointIntegration::new(Arc::new(cfg)); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "ts.prospecta.com", - request_scheme: "https", - origin_host: "origin.prospecta.com", - document_state: &document_state, - }; - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!( inserts.len(), 1, @@ -3084,4 +3077,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/framework/nextjs/.gitattributes b/crates/framework/nextjs/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/framework/nextjs/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/framework/nextjs/src/fixtures/page-change.input.html b/crates/framework/nextjs/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..66a36e3ca4 --- /dev/null +++ b/crates/framework/nextjs/src/fixtures/page-change.input.html @@ -0,0 +1,15 @@ + + + + +Next.js + + +An article on the origin + + + + + + + diff --git a/crates/framework/nextjs/src/fixtures/page-change.recorded.html b/crates/framework/nextjs/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..11f71f6cc0 --- /dev/null +++ b/crates/framework/nextjs/src/fixtures/page-change.recorded.html @@ -0,0 +1,15 @@ + + + + +Next.js + + +An article on the origin + + + + + + + diff --git a/crates/framework/nextjs/src/fixtures/page-change.settings.toml b/crates/framework/nextjs/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..b5fc71ebf8 --- /dev/null +++ b/crates/framework/nextjs/src/fixtures/page-change.settings.toml @@ -0,0 +1,9 @@ +[framework] +modules = ["nextjs"] + +[framework.nextjs] +rewrite_attributes = ["href", "link", "url"] + +[[fetch]] +media_type = "text/html" +middleware = ["framework.nextjs"] diff --git a/crates/framework/nextjs/src/lib.rs b/crates/framework/nextjs/src/lib.rs index 4c06663909..4d31ebbd5d 100644 --- a/crates/framework/nextjs/src/lib.rs +++ b/crates/framework/nextjs/src/lib.rs @@ -17,6 +17,9 @@ use validator::Validate; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::IntegrationRegistration; +use trusted_server_core::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, +}; use trusted_server_core::settings::{IntegrationConfig, Settings}; const NEXTJS_INTEGRATION_ID: &str = "nextjs"; @@ -45,9 +48,10 @@ mod shared; pub use rsc::rewrite_rsc_scripts_combined; -use rsc_placeholders::NextJsRscPlaceholderRewriter; +use rsc_placeholders::{NextJsRscPlaceholderRewriter, RscPlaceholderText}; use rsc_stream::NextJsRscStreamProcessorFactory; -use script_rewriter::NextJsNextDataRewriter; +use script_rewriter::{NextDataText, NextJsNextDataRewriter}; +use shared::ScriptDocument; #[derive(Debug, Clone, Deserialize, Serialize, Validate)] #[serde(deny_unknown_fields)] @@ -114,21 +118,59 @@ pub fn register( log::info!("NextJS integration not registered (no section selects it)"); return Ok(None); }; - // Register a structured (Pages Router __NEXT_DATA__) rewriter. - let structured = Arc::new(NextJsNextDataRewriter::new(config.clone())?); + let page_change = PageChange { + // The Pages Router's `__NEXT_DATA__` script. + structured: Arc::new(NextJsNextDataRewriter::new(config.clone())?), + // Placeholders for the App Router's RSC payload scripts, written + // during the HTML rewrite pass and substituted by the bounded output + // stream processor. + placeholders: Arc::new(NextJsRscPlaceholderRewriter::new(config.clone())), + stream: NextJsRscStreamProcessorFactory::new(config), + }; - // Insert placeholders for App Router RSC payload scripts during the HTML rewrite pass, - // then substitute them through the bounded output stream processor. - let placeholders = Arc::new(NextJsRscPlaceholderRewriter::new(config.clone())); + Ok(Some( + IntegrationRegistration::builder(NEXTJS_INTEGRATION_ID) + .with_middleware(Arc::new(page_change)) + .build(), + )) +} - let stream_processor = Arc::new(NextJsRscStreamProcessorFactory::new(config.clone())); +/// Moves the origin's address to the publisher's in the data Next.js writes +/// into a page, on the pages a `[[fetch]]` entry names [`MODULE`] for. +struct PageChange { + structured: Arc, + placeholders: Arc, + stream: NextJsRscStreamProcessorFactory, +} - let builder = IntegrationRegistration::builder(NEXTJS_INTEGRATION_ID) - .with_script_rewriter(structured) - .with_script_rewriter(placeholders) - .with_html_stream_processor(stream_processor); +impl Middleware for PageChange { + fn middleware_id(&self) -> &'static str { + MODULE + } - Ok(Some(builder.build())) + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + /// A `__NEXT_DATA__` script is put to the structured rewriter before the + /// placeholder rewriter, which is asked about every script. The stream + /// processor then works on the document the two left. + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + text_handlers: vec![ + Box::new(NextDataText { + rewriter: Arc::clone(&self.structured), + document: ScriptDocument::new(context), + }), + Box::new(RscPlaceholderText { + rewriter: Arc::clone(&self.placeholders), + document: ScriptDocument::new(context), + }), + ], + stream: Some(self.stream.create(context)), + ..MiddlewareAction::pass() + } + } } fn build( @@ -145,13 +187,24 @@ mod tests { use super::*; use serde_json::json; use std::io::Cursor; - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::streaming_processor::{ Compression, PipelineConfig, StreamingPipeline, }; use trusted_server_core::test_support::tests::create_test_settings; + /// Test settings whose `[[fetch]]` entry runs the module's middleware on + /// every page. + fn settings_placing_the_module() -> Settings { + let mut settings = create_test_settings(); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); + settings + } + fn config_from_settings( settings: &Settings, registry: &IntegrationRegistry, @@ -173,7 +226,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -186,7 +239,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -227,6 +280,50 @@ mod tests { ); } + #[test] + fn html_processor_keeps_what_a_rewritten_next_data_script_holds() { + // The payload is rewritten because it names the origin, and the rest + // of it has to come back as the page wrote it. + let html = r#""#; + + let mut settings = settings_placing_the_module(); + settings + .insert_module_config( + "framework", + "framework.nextjs", + &json!({ + "rewrite_attributes": ["href", "link", "url"], + }), + ) + .expect("should update nextjs config"); + let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) + .expect("should create registry"); + let config = config_from_settings(&settings, ®istry); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size: 8192, + }, + create_page_processor(&settings, ®istry, config), + ); + + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("pipeline should process HTML"); + let processed = String::from_utf8_lossy(&output); + + assert!( + processed.contains(r#""href":"https://test.example.com/search?q=shoes&page=2""#), + "should move the address and keep its query string as written: {processed}" + ); + assert!( + processed.contains(r#""note":"a < b && c > d""#), + "should keep a value it did not rewrite as written: {processed}" + ); + } + #[test] fn html_processor_rewrites_next_data_fixture_like_payload() { let html = r#" @@ -253,7 +350,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -272,7 +369,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -329,7 +426,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -342,7 +439,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -375,7 +472,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -388,7 +485,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -423,7 +520,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -437,7 +534,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -486,7 +583,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -500,7 +597,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -556,7 +653,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -569,7 +666,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); // Use small chunk size to force fragmentation let pipeline_config = PipelineConfig { input_compression: Compression::None, @@ -617,7 +714,7 @@ mod tests { // Build a __NEXT_DATA__ payload large enough to cross a 32-byte chunk boundary. let html = r#""#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -630,7 +727,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); // Use a very small chunk size to force fragmentation. let pipeline_config = PipelineConfig { @@ -672,7 +769,7 @@ mod tests { "x".repeat(400), // pad to guarantee chunk-boundary fragmentation ); - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -685,7 +782,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, @@ -729,7 +826,7 @@ mod tests { /// Build the production HTML processor for the Next.js integration and run /// `html` through it at `chunk_size`, returning the streamed output. fn stream_nextjs_html(html: &str, chunk_size: usize) -> String { - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -741,7 +838,11 @@ mod tests { .expect("should update nextjs config"); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); - let processor = create_html_processor(config_from_settings(&settings, ®istry)); + let processor = create_page_processor( + &settings, + ®istry, + config_from_settings(&settings, ®istry), + ); let mut pipeline = StreamingPipeline::new( PipelineConfig { input_compression: Compression::None, @@ -834,7 +935,7 @@ mod tests { " ); - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -848,7 +949,11 @@ mod tests { .expect("should update nextjs config"); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); - let processor = create_html_processor(config_from_settings(&settings, ®istry)); + let processor = create_page_processor( + &settings, + ®istry, + config_from_settings(&settings, ®istry), + ); let mut pipeline = StreamingPipeline::new( PipelineConfig { input_compression: Compression::None, @@ -906,7 +1011,7 @@ mod tests { fn output_overflow_restores_an_in_progress_script_at_every_split() { use trusted_server_core::streaming_processor::StreamProcessor as _; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -922,7 +1027,11 @@ mod tests { let expected = format!("{first}{padding}"); for split in 1..script.len() { - let mut processor = create_html_processor(config_from_settings(&settings, ®istry)); + let mut processor = create_page_processor( + &settings, + ®istry, + config_from_settings(&settings, ®istry), + ); let mut output = processor .process_chunk(first.as_bytes(), false) .expect("should process unresolved RSC group"); @@ -945,4 +1054,21 @@ mod tests { ); } } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/framework/nextjs/src/rsc_placeholders.rs b/crates/framework/nextjs/src/rsc_placeholders.rs index 08d81b3932..6a542270ea 100644 --- a/crates/framework/nextjs/src/rsc_placeholders.rs +++ b/crates/framework/nextjs/src/rsc_placeholders.rs @@ -1,9 +1,9 @@ use std::sync::Arc; -use trusted_server_core::integrations::{ - IntegrationScriptContext, IntegrationScriptRewriter, ScriptRewriteAction, -}; +use trusted_server_core::integrations::ScriptRewriteAction; +use trusted_server_core::middleware::TextHandler; +use super::NextJsIntegrationConfig; use super::rsc::DEFAULT_MAX_COMBINED_PAYLOAD_BYTES; #[cfg(test)] pub(super) use super::rsc_stream::RSC_PAYLOAD_PLACEHOLDER_PREFIX; @@ -12,10 +12,9 @@ use super::rsc_stream::{ capture_fragment, classify_rsc_group, document_state, rsc_payload_placeholder, }; use super::shared::{ - RSC_RECEIVER_CONTEXT_BYTES, find_rsc_push_payload_range, find_trimmed_rsc_push_payload_range, - receiver_context_is_flight_push, + RSC_RECEIVER_CONTEXT_BYTES, ScriptContext, ScriptDocument, find_rsc_push_payload_range, + find_trimmed_rsc_push_payload_range, receiver_context_is_flight_push, }; -use super::{NEXTJS_INTEGRATION_ID, NextJsIntegrationConfig}; pub(super) struct NextJsRscPlaceholderRewriter { config: Arc, @@ -169,16 +168,25 @@ impl NextJsRscPlaceholderRewriter { } } -impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { - fn integration_id(&self) -> &'static str { - NEXTJS_INTEGRATION_ID - } +/// The RSC placeholder rewriter at work on one document. +pub(super) struct RscPlaceholderText { + pub(super) rewriter: Arc, + pub(super) document: ScriptDocument, +} - fn selector(&self) -> &'static str { +impl TextHandler for RscPlaceholderText { + fn selector(&self) -> &str { "script" } - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + self.rewriter.rewrite(text, &self.document.chunk(is_last)) + } +} + +impl NextJsRscPlaceholderRewriter { + /// Decides what happens to one chunk of a script's text. + fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if self.config.rewrite_attributes.is_empty() { return ScriptRewriteAction::keep(); } @@ -344,15 +352,15 @@ fn remember_released(context: &mut String, released: &str) { #[cfg(test)] mod tests { use super::*; + use crate::NEXTJS_INTEGRATION_ID; use crate::rsc_stream::NextJsDocumentState; use trusted_server_core::integrations::IntegrationDocumentState; fn ctx( is_last_in_text_node: bool, document_state: &IntegrationDocumentState, - ) -> IntegrationScriptContext<'_> { - IntegrationScriptContext { - selector: "script", + ) -> ScriptContext<'_> { + ScriptContext { request_host: "proxy.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -732,7 +740,7 @@ mod tests { let state = IntegrationDocumentState::default(); let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); - let context = IntegrationScriptContext { + let context = ScriptContext { max_buffered_script_bytes: budget, ..ctx(true, &state) }; diff --git a/crates/framework/nextjs/src/rsc_stream.rs b/crates/framework/nextjs/src/rsc_stream.rs index c927e70585..b8e77e441d 100644 --- a/crates/framework/nextjs/src/rsc_stream.rs +++ b/crates/framework/nextjs/src/rsc_stream.rs @@ -2,9 +2,8 @@ use std::collections::VecDeque; use std::io; use std::sync::{Arc, Mutex}; -use trusted_server_core::integrations::{ - IntegrationDocumentState, IntegrationHtmlStreamContext, IntegrationHtmlStreamProcessorFactory, -}; +use trusted_server_core::integrations::IntegrationDocumentState; +use trusted_server_core::middleware::MiddlewareContext; use trusted_server_core::streaming_processor::StreamProcessor; use super::rsc::{ @@ -151,24 +150,19 @@ impl NextJsRscStreamProcessorFactory { pub(super) fn new(config: Arc) -> Self { Self { config } } -} - -impl IntegrationHtmlStreamProcessorFactory for NextJsRscStreamProcessorFactory { - fn integration_id(&self) -> &'static str { - NEXTJS_INTEGRATION_ID - } - fn create(&self, context: IntegrationHtmlStreamContext) -> Box { + /// The processor of the one document `context` describes. + pub(super) fn create(&self, context: &MiddlewareContext<'_>) -> Box { let limit = if self.config.max_combined_payload_bytes == 0 { DEFAULT_MAX_COMBINED_PAYLOAD_BYTES } else { self.config.max_combined_payload_bytes }; Box::new(NextJsRscStreamProcessor::new( - document_state(&context.document_state), - context.origin_host, - context.request_host, - context.request_scheme, + document_state(context.document_state), + context.origin_host.to_owned(), + context.request_host.to_owned(), + context.request_scheme.to_owned(), limit, )) } diff --git a/crates/framework/nextjs/src/script_rewriter.rs b/crates/framework/nextjs/src/script_rewriter.rs index 37de438f84..9243cb3cc4 100644 --- a/crates/framework/nextjs/src/script_rewriter.rs +++ b/crates/framework/nextjs/src/script_rewriter.rs @@ -4,13 +4,12 @@ use error_stack::Report; use regex::{Regex, escape}; use trusted_server_core::error::TrustedServerError; -use trusted_server_core::integrations::{ - IntegrationScriptContext, IntegrationScriptRewriter, ScriptRewriteAction, -}; +use trusted_server_core::integrations::ScriptRewriteAction; +use trusted_server_core::middleware::TextHandler; +use super::NextJsIntegrationConfig; use super::rsc_stream::{FragmentCapture, capture_fragment, document_state}; -use super::shared::strip_origin_host_with_optional_port; -use super::{NEXTJS_INTEGRATION_ID, NextJsIntegrationConfig}; +use super::shared::{ScriptContext, ScriptDocument, strip_origin_host_with_optional_port}; pub(super) struct NextJsNextDataRewriter { config: Arc, @@ -27,11 +26,7 @@ impl NextJsNextDataRewriter { }) } - fn rewrite_structured( - &self, - content: &str, - ctx: &IntegrationScriptContext<'_>, - ) -> ScriptRewriteAction { + fn rewrite_structured(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if ctx.origin_host.is_empty() || ctx.request_host.is_empty() || self.config.rewrite_attributes.is_empty() @@ -52,16 +47,25 @@ impl NextJsNextDataRewriter { } } -impl IntegrationScriptRewriter for NextJsNextDataRewriter { - fn integration_id(&self) -> &'static str { - NEXTJS_INTEGRATION_ID - } +/// The `__NEXT_DATA__` rewriter at work on one document. +pub(super) struct NextDataText { + pub(super) rewriter: Arc, + pub(super) document: ScriptDocument, +} - fn selector(&self) -> &'static str { +impl TextHandler for NextDataText { + fn selector(&self) -> &str { "script#__NEXT_DATA__" } - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + self.rewriter.rewrite(text, &self.document.chunk(is_last)) + } +} + +impl NextJsNextDataRewriter { + /// Decides what happens to one chunk of the `__NEXT_DATA__` script. + fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if self.config.rewrite_attributes.is_empty() { return ScriptRewriteAction::keep(); } @@ -218,12 +222,8 @@ mod tests { }) } - fn ctx<'a>( - selector: &'static str, - document_state: &'a IntegrationDocumentState, - ) -> IntegrationScriptContext<'a> { - IntegrationScriptContext { - selector, + fn ctx(document_state: &IntegrationDocumentState) -> ScriptContext<'_> { + ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -244,9 +244,9 @@ mod tests { // Document one: cut off before its final fragment arrives. let first_document = IntegrationDocumentState::default(); - let interrupted = IntegrationScriptContext { + let interrupted = ScriptContext { is_last_in_text_node: false, - ..ctx("script#__NEXT_DATA__", &first_document) + ..ctx(&first_document) }; let secret = r#"{"props":{"pageProps":{"sessionToken":"SESSION-ONE-SECRET","href":"#; @@ -259,7 +259,7 @@ mod tests { // Document two: fresh document state, same rewriter. let second_document = IntegrationDocumentState::default(); - let fresh = ctx("script#__NEXT_DATA__", &second_document); + let fresh = ctx(&second_document); let benign = r#"{"props":{"pageProps":{"href":"https://origin.example.com/reviews"}}}"#; let action = rewriter.rewrite(benign, &fresh); @@ -286,7 +286,7 @@ mod tests { let rewriter = NextJsNextDataRewriter::new(test_config()) .expect("should build Next.js structured rewriter"); let document_state = IntegrationDocumentState::default(); - let result = rewriter.rewrite(payload, &ctx("script#__NEXT_DATA__", &document_state)); + let result = rewriter.rewrite(payload, &ctx(&document_state)); match result { ScriptRewriteAction::Replace(value) => { @@ -544,8 +544,7 @@ mod tests { let fragment1 = r#"{"props":{"pageProps":{"href":"https://origin."#; let fragment2 = r#"example.com/reviews"}}}"#; - let ctx_intermediate = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let ctx_intermediate = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -553,7 +552,7 @@ mod tests { max_buffered_script_bytes: 16 * 1024 * 1024, document_state: &document_state, }; - let ctx_last = IntegrationScriptContext { + let ctx_last = ScriptContext { is_last_in_text_node: true, ..ctx_intermediate }; @@ -591,8 +590,7 @@ mod tests { let document_state = IntegrationDocumentState::default(); let payload = r#"{"props":{"pageProps":{"href":"https://origin.example.com/page"}}}"#; - let ctx_single = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let ctx_single = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -622,8 +620,7 @@ mod tests { let fragment1 = r#"{"props":{"pageProps":{"title":"Hello"#; let fragment2 = r#" World","count":42}}}"#; - let ctx_intermediate = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let ctx_intermediate = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -631,7 +628,7 @@ mod tests { max_buffered_script_bytes: 16 * 1024 * 1024, document_state: &document_state, }; - let ctx_last = IntegrationScriptContext { + let ctx_last = ScriptContext { is_last_in_text_node: true, ..ctx_intermediate }; @@ -658,8 +655,7 @@ mod tests { fn fragmented_next_data_releases_suppressed_prefix_on_overflow_and_resets() { let rewriter = NextJsNextDataRewriter::new(test_config()).expect("should build rewriter"); let document_state = IntegrationDocumentState::default(); - let first = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let first = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -681,7 +677,7 @@ mod tests { assert_eq!( rewriter.rewrite( "-tail", - &IntegrationScriptContext { + &ScriptContext { is_last_in_text_node: true, ..first }, @@ -701,8 +697,7 @@ mod tests { let rewriter = NextJsNextDataRewriter::new(test_config()).expect("should build rewriter"); let first_state = IntegrationDocumentState::default(); let second_state = IntegrationDocumentState::default(); - let first_context = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let first_context = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -710,7 +705,7 @@ mod tests { max_buffered_script_bytes: 64, document_state: &first_state, }; - let second_context = IntegrationScriptContext { + let second_context = ScriptContext { document_state: &second_state, ..first_context }; @@ -728,7 +723,7 @@ mod tests { let ScriptRewriteAction::Replace(first_output) = rewriter.rewrite( "done", - &IntegrationScriptContext { + &ScriptContext { is_last_in_text_node: true, ..first_context }, diff --git a/crates/framework/nextjs/src/shared.rs b/crates/framework/nextjs/src/shared.rs index 1ae78a7cab..a9483099d0 100644 --- a/crates/framework/nextjs/src/shared.rs +++ b/crates/framework/nextjs/src/shared.rs @@ -8,6 +8,60 @@ use std::sync::LazyLock; use regex::Regex; use trusted_server_core::host_rewrite::rewrite_bare_host_at_boundaries; +use trusted_server_core::integrations::IntegrationDocumentState; +use trusted_server_core::middleware::MiddlewareContext; + +/// What a script rewriter is told about the chunk of text in front of it. +#[derive(Debug)] +pub(crate) struct ScriptContext<'a> { + /// The host the page is served on. + pub(crate) request_host: &'a str, + /// The scheme the page is served over. + pub(crate) request_scheme: &'a str, + /// The host the page was fetched from. + pub(crate) origin_host: &'a str, + /// Whether the chunk is the last of its text node. + pub(crate) is_last_in_text_node: bool, + /// The most script text a rewriter may hold while it waits for the rest. + pub(crate) max_buffered_script_bytes: usize, + /// The state kept for the document the script is in. + pub(crate) document_state: &'a IntegrationDocumentState, +} + +/// What a script rewriter keeps of the document it works on, to tell it about +/// each chunk. +pub(crate) struct ScriptDocument { + request_host: String, + request_scheme: String, + origin_host: String, + max_buffered_script_bytes: usize, + document_state: IntegrationDocumentState, +} + +impl ScriptDocument { + /// What `context` says of the document. + pub(crate) fn new(context: &MiddlewareContext<'_>) -> Self { + Self { + request_host: context.request_host.to_owned(), + request_scheme: context.request_scheme.to_owned(), + origin_host: context.origin_host.to_owned(), + max_buffered_script_bytes: context.max_buffered_script_bytes, + document_state: context.document_state.clone(), + } + } + + /// The context of one chunk of a script's text. + pub(crate) fn chunk(&self, is_last_in_text_node: bool) -> ScriptContext<'_> { + ScriptContext { + request_host: &self.request_host, + request_scheme: &self.request_scheme, + origin_host: &self.origin_host, + is_last_in_text_node, + max_buffered_script_bytes: self.max_buffered_script_bytes, + document_state: &self.document_state, + } + } +} // These are static code-defined literals, not config-derived patterns, so they // intentionally remain lazy statics instead of participating in diff --git a/crates/identity/lockr/.gitattributes b/crates/identity/lockr/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/identity/lockr/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/identity/lockr/src/fixtures/page-change.input.html b/crates/identity/lockr/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..4e816c68d6 --- /dev/null +++ b/crates/identity/lockr/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Lockr + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/identity/lockr/src/fixtures/page-change.recorded.html b/crates/identity/lockr/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..199fbc145e --- /dev/null +++ b/crates/identity/lockr/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Lockr + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/identity/lockr/src/fixtures/page-change.settings.toml b/crates/identity/lockr/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..f1878e4b19 --- /dev/null +++ b/crates/identity/lockr/src/fixtures/page-change.settings.toml @@ -0,0 +1,9 @@ +[identity] +modules = ["lockr"] + +[identity.lockr] +app_id = "test-app-id" + +[[fetch]] +media_type = "text/html" +middleware = ["identity.lockr"] diff --git a/crates/identity/lockr/src/lib.rs b/crates/identity/lockr/src/lib.rs index 74e26f02ff..8f4269c219 100644 --- a/crates/identity/lockr/src/lib.rs +++ b/crates/identity/lockr/src/lib.rs @@ -19,6 +19,7 @@ ) )] +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -32,10 +33,13 @@ use validator::Validate; use trusted_server_core::constants::INTERNAL_HEADERS; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, - UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, collect_response_bounded, - ensure_integration_backend, + AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, + IntegrationRegistration, UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, + collect_response_bounded, ensure_integration_backend, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::{PlatformHttpRequest, RuntimeServices}; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -373,7 +377,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(LOCKR_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(SdkAddress(integration))) .build(), )) } @@ -419,26 +423,11 @@ impl IntegrationProxy for LockrIntegration { } } -impl IntegrationAttributeRewriter for LockrIntegration { - fn integration_id(&self) -> &'static str { - LOCKR_INTEGRATION_ID - } - - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_sdk && matches!(attribute, "src" | "href") - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if !self.config.rewrite_sdk { - return AttributeRewriteAction::Keep; - } - - if self.is_lockr_sdk_url(attr_value) { +impl LockrIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party endpoint when it is the SDK's address. + fn rewrite_sdk_address(&self, value: &str) -> AttributeRewriteAction { + if self.is_lockr_sdk_url(value) { // Root-relative so the browser resolves it against the page host. // Note: a page-level `` participates in this resolution, // so on pages that set an external base URL these resolve against @@ -453,6 +442,33 @@ impl IntegrationAttributeRewriter for LockrIntegration { } } +/// Loads the SDK from the first-party endpoint, on the pages a `[[fetch]]` +/// entry names [`MODULE`] for. It does nothing unless `rewrite_sdk` is set. +struct SdkAddress(Arc); + +impl Middleware for SdkAddress { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + if !self.0.config.rewrite_sdk { + return MiddlewareAction::pass(); + } + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_sdk_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } + } +} + fn default_api_endpoint() -> String { "https://identity.loc.kr".to_string() } @@ -493,16 +509,6 @@ mod tests { } } - fn test_context() -> IntegrationAttributeContext<'static> { - IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - } - } - #[test] fn test_lockr_sdk_url_detection() { let integration = LockrIntegration::new(test_config()); @@ -543,11 +549,10 @@ mod tests { } #[test] - fn test_attribute_rewriter_rewrites_sdk_urls() { + fn test_element_handler_rewrites_sdk_urls() { let integration = LockrIntegration::new(test_config()); - let ctx = test_context(); - let result = integration.rewrite("src", "https://aim.loc.kr/identity-lockr-v1.0.js", &ctx); + let result = integration.rewrite_sdk_address("https://aim.loc.kr/identity-lockr-v1.0.js"); assert_eq!( result, @@ -557,11 +562,10 @@ mod tests { } #[test] - fn test_attribute_rewriter_keeps_non_lockr_urls() { + fn test_element_handler_keeps_non_lockr_urls() { let integration = LockrIntegration::new(test_config()); - let ctx = test_context(); - let result = integration.rewrite("src", "https://example.com/other.js", &ctx); + let result = integration.rewrite_sdk_address("https://example.com/other.js"); assert_eq!( result, @@ -571,20 +575,21 @@ mod tests { } #[test] - fn test_attribute_rewriter_noop_when_disabled() { + fn test_element_handler_noop_when_disabled() { let config = LockrConfig { rewrite_sdk: false, ..test_config() }; let integration = LockrIntegration::new(config); - let ctx = test_context(); - - let result = integration.rewrite("src", "https://aim.loc.kr/identity-lockr-v1.0.js", &ctx); + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); - assert_eq!( - result, - AttributeRewriteAction::Keep, - "should keep all URLs when rewrite_sdk is disabled" + assert!( + SdkAddress(integration).create(&context).is_pass(), + "should leave every page alone when rewrite_sdk is disabled" ); } @@ -798,4 +803,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/tag/google-tag-manager/.gitattributes b/crates/tag/google-tag-manager/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/tag/google-tag-manager/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.input.html b/crates/tag/google-tag-manager/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..134e4914c3 --- /dev/null +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.input.html @@ -0,0 +1,21 @@ + + + + +Google Tag Manager + + + + + + + +An article on the origin + + + + diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html b/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..cbb6bebe36 --- /dev/null +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html @@ -0,0 +1,21 @@ + + + + +Google Tag Manager + + + + + + + +An article on the origin + + + + diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml b/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..8520eb5692 --- /dev/null +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml @@ -0,0 +1,10 @@ +[tag] +modules = ["google-tag-manager"] + +[tag.google-tag-manager] +container_id = "GTM-TEST1234" +upstream_url = "https://www.googletagmanager.com" + +[[fetch]] +media_type = "text/html" +middleware = ["tag.google-tag-manager"] diff --git a/crates/tag/google-tag-manager/src/lib.rs b/crates/tag/google-tag-manager/src/lib.rs index 5dc84ff739..c97cd58aca 100644 --- a/crates/tag/google-tag-manager/src/lib.rs +++ b/crates/tag/google-tag-manager/src/lib.rs @@ -24,6 +24,7 @@ ) )] +use std::rc::Rc; use std::sync::{Arc, LazyLock}; use async_trait::async_trait; @@ -36,12 +37,13 @@ use serde::{Deserialize, Serialize}; use validator::{Validate, ValidationError}; use trusted_server_core::error::TrustedServerError; -use trusted_server_core::integrations::ScriptTextAccumulator; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, IntegrationScriptContext, - IntegrationScriptRewriter, ScriptRewriteAction, UPSTREAM_SDK_MAX_RESPONSE_BYTES, - collect_response_bounded, + AttributeRewriteAction, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, + ScriptRewriteAction, UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_response_bounded, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, TextHandler, }; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{ @@ -312,9 +314,9 @@ fn default_max_beacon_body_size() -> usize { 65536 // 64KB - prevents memory pressure from oversized payloads } -/// GTM domain markers the script rewriter looks for. Kept in one place so the +/// GTM domain markers the script handler looks for. Kept in one place so the /// boundary-safe prefix check in [`might_contain_gtm_prefix`] and the full-match -/// check in [`GoogleTagManagerIntegration::rewrite`] cannot drift apart. +/// check in [`InlineSnippet`] cannot drift apart. const GTM_SCRIPT_MARKERS: &[&str] = &["googletagmanager.com", "google-analytics.com"]; /// Minimum trailing-prefix length required to engage the accumulation path. @@ -331,13 +333,11 @@ const GTM_MIN_PREFIX_LEN: usize = 6; /// Return true if `text` either already contains a GTM marker or ends with a /// proper prefix of one of length ≥ [`GTM_MIN_PREFIX_LEN`] — i.e., the next -/// fragment could still complete a match. Used to gate whether the GTM -/// rewriter should claim ownership of a script's output via -/// `RemoveNode`/`Replace`. +/// fragment could still complete a match. It gates whether the script +/// handler holds a script's text via `RemoveNode`/`Replace`. /// -/// Returning false means the rewriter can safely return `Keep` and leave the -/// script untouched, preserving any replacement a more-specific rewriter -/// (e.g., `NextJsNextDataRewriter`) made on the same element. +/// Returning false means the handler can safely return `Keep` and leave the +/// script untouched. fn might_contain_gtm_prefix(text: &str) -> bool { for marker in GTM_SCRIPT_MARKERS { if text.contains(marker) { @@ -941,8 +941,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(GTM_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_script_rewriter(integration) + .with_middleware(integration) .build(), )) } @@ -1032,58 +1031,68 @@ impl IntegrationProxy for GoogleTagManagerIntegration { } } -impl IntegrationAttributeRewriter for GoogleTagManagerIntegration { - fn integration_id(&self) -> &'static str { - GTM_INTEGRATION_ID +impl GoogleTagManagerIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party path when it is one of Google's tag addresses. + fn rewrite_address(value: &str) -> AttributeRewriteAction { + if Self::is_rewritable_url(value) { + AttributeRewriteAction::replace(Self::rewrite_gtm_urls(value)) + } else { + AttributeRewriteAction::keep() + } } +} - fn handles_attribute(&self, attribute: &str) -> bool { - matches!(attribute, "src" | "href") +/// Google Tag Manager's change to a page, on the pages a `[[fetch]]` entry +/// names [`MODULE`] for. It points Google's tag addresses at the first-party +/// path, in `src` and `href` attributes and in the text of inline scripts. +impl Middleware for GoogleTagManagerIntegration { + fn middleware_id(&self) -> &'static str { + MODULE } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if Self::is_rewritable_url(attr_value) { - AttributeRewriteAction::replace(Self::rewrite_gtm_urls(attr_value)) - } else { - AttributeRewriteAction::keep() + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let decide: Rc = + Rc::new(|matched| Self::rewrite_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + text_handlers: vec![Box::new(InlineSnippet::default())], + ..MiddlewareAction::pass() } } } -impl IntegrationScriptRewriter for GoogleTagManagerIntegration { - fn integration_id(&self) -> &'static str { - GTM_INTEGRATION_ID - } +/// Rewrites Google's tag addresses in the text of inline scripts. +/// +/// Made for one document, so text held for a script that never ends cannot +/// reach another document. +#[derive(Default)] +struct InlineSnippet { + /// Text held back while it might still turn out to hold a marker. + held: String, +} - fn selector(&self) -> &'static str { +impl TextHandler for InlineSnippet { + fn selector(&self) -> &str { "script" // Match all scripts to find inline GTM snippets } - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { - // Per document, never per registry: a registry-lifetime buffer would - // carry one document's partial script into the next. - let accumulator = ctx - .document_state - .get_or_insert_with(GTM_INTEGRATION_ID, ScriptTextAccumulator::default); - let mut buf = accumulator.buffer(); - - // Cheap gate: only engage the accumulation path for scripts whose - // running text could plausibly contain a GTM/GA domain. Unrelated - // scripts (the vast majority) return Keep on every fragment so they - // stay visible in `lol_html`'s output unchanged — and, critically, - // so a Replace from this rewriter does not clobber a Replace from - // another rewriter on the same element (e.g., `NextJsNextDataRewriter` - // on `script#__NEXT_DATA__`). See `might_contain_gtm_prefix` for the - // boundary-safe substring check. + fn decide(&mut self, content: &str, is_last: bool) -> ScriptRewriteAction { + let buf = &mut self.held; + + // Cheap gate: only hold text for scripts whose running text could + // plausibly contain a GTM/GA domain. Unrelated scripts (the vast + // majority) are kept on every fragment, so they pass through + // untouched. See `might_contain_gtm_prefix` for the boundary-safe + // substring check. let prior_and_current_might_match = - might_contain_gtm_prefix(&buf) || might_contain_gtm_prefix(content); + might_contain_gtm_prefix(buf) || might_contain_gtm_prefix(content); - if !ctx.is_last_in_text_node { + if !is_last { // Intermediate fragment. Only accumulate + suppress when the script // might still resolve to a GTM match. Otherwise return Keep so the // fragment is emitted as-is by `lol_html` and untouched by us. @@ -1091,6 +1100,13 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { buf.push_str(content); return ScriptRewriteAction::RemoveNode; } + // Nothing held can become a marker now, so what was held goes out + // ahead of this fragment. Left for the last fragment, it would be + // written after text that came later in the script. + if !buf.is_empty() { + buf.push_str(content); + return ScriptRewriteAction::replace(std::mem::take(buf)); + } return ScriptRewriteAction::keep(); } @@ -1099,7 +1115,7 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { None } else { buf.push_str(content); - Some(std::mem::take(&mut *buf)) + Some(std::mem::take(buf)) }; let text = full_content.as_deref().unwrap_or(content); @@ -1112,7 +1128,9 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { .iter() .any(|marker| text.contains(marker)) { - return ScriptRewriteAction::replace(Self::rewrite_gtm_urls(text)); + return ScriptRewriteAction::replace(GoogleTagManagerIntegration::rewrite_gtm_urls( + text, + )); } // No GTM content — if we accumulated fragments, emit them unchanged. @@ -1128,11 +1146,12 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { #[cfg(test)] mod tests { use super::*; - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationDocumentState, IntegrationRegistry, IntegrationScriptContext, - IntegrationScriptRewriter, ScriptRewriteAction, + AttributeRewriteAction, IntegrationDocumentState, IntegrationRegistry, ScriptRewriteAction, }; use trusted_server_core::platform::test_support::{ StubHttpClient, build_services_with_http_client, @@ -2365,24 +2384,10 @@ mod tests { } #[test] - fn test_attribute_rewriter() { - let config = tag_config("GTM-TEST1234", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - + fn test_element_handler() { // Case 1: Standard HTTPS URL - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/gtm.js?id=GTM-TEST1234", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!( @@ -2394,11 +2399,8 @@ mod tests { } // Case 2: Protocol-relative URL - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", + let action = GoogleTagManagerIntegration::rewrite_address( "//www.googletagmanager.com/gtm.js?id=GTM-TEST1234", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!( @@ -2413,11 +2415,8 @@ mod tests { } // Case 3: gtag/js URL in href (preload link) - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/gtag/js?id=G-DQMZGMPHXN", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!( @@ -2432,11 +2431,8 @@ mod tests { } // Case 4: google-analytics.com URL in href - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.google-analytics.com/g/collect", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!(val, "/integrations/google_tag_manager/g/collect"); @@ -2448,11 +2444,8 @@ mod tests { } // Case 5: analytics.google.com URL in href - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://analytics.google.com/g/collect?v=2", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!(val, "/integrations/google_tag_manager/g/collect?v=2"); @@ -2464,36 +2457,17 @@ mod tests { } // Case 6: Other URL (should be kept) - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", - "https://other.com/script.js", - &ctx, - ); + let action = GoogleTagManagerIntegration::rewrite_address("https://other.com/script.js"); assert!(matches!(action, AttributeRewriteAction::Keep)); } #[test] - fn test_attribute_rewriter_rejects_false_positives() { + fn test_element_handler_rejects_false_positives() { // Test that URLs with GTM domains in query parameters or paths are NOT rewritten // This verifies the fix for P2: proper URL parsing instead of substring matching - let config = tag_config("GTM-TEST1234", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let ctx = IntegrationAttributeContext { - attribute_name: "href", - element_name: "a", - request_host: "example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - // Case 1: GTM domain in query parameter - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://evil.com/?redirect=https://www.google-analytics.com/collect", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2501,11 +2475,8 @@ mod tests { ); // Case 2: GTM domain in path component - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://example.com/www.googletagmanager.com/gtm.js", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2513,11 +2484,8 @@ mod tests { ); // Case 3: Unsupported path on valid GTM domain - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/ns.html", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2525,11 +2493,8 @@ mod tests { ); // Case 4: Fragment with GTM domain - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://example.com/page#https://www.googletagmanager.com/gtm.js", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2537,11 +2502,8 @@ mod tests { ); // Case 5: Valid GTM URL should STILL be rewritten (sanity check) - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/gtm.js?id=GTM-TEST", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Replace(_)), @@ -2550,20 +2512,8 @@ mod tests { } #[test] - fn test_script_rewriter() { - let config = tag_config("GTM-TEST1234", &[]); - let integration = GoogleTagManagerIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - - let ctx = IntegrationScriptContext { - selector: "script", - request_host: "example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: true, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &doc_state, - }; + fn test_text_handler() { + let mut script = InlineSnippet::default(); // Case 1: Inline GTM snippet let snippet = r#"(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': @@ -2572,7 +2522,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-XXXX');"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, snippet, &ctx); + let action = script.decide(snippet, true); if let ScriptRewriteAction::Replace(val) = action { assert!(val.contains("/integrations/google_tag_manager/gtm.js")); assert!(!val.contains("https://www.googletagmanager.com/gtm.js")); @@ -2582,7 +2532,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= // Case 2: Protocol relative let snippet_proto = r#"j.src='//www.googletagmanager.com/gtm.js?id='+i+dl;"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, snippet_proto, &ctx); + let action = script.decide(snippet_proto, true); if let ScriptRewriteAction::Replace(val) = action { assert!(val.contains("/integrations/google_tag_manager/gtm.js")); assert!(!val.contains("//www.googletagmanager.com/gtm.js")); @@ -2595,7 +2545,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= // Case 3: Irrelevant script let other_script = "console.log('hello');"; - let action = IntegrationScriptRewriter::rewrite(&*integration, other_script, &ctx); + let action = script.decide(other_script, true); assert!(matches!(action, ScriptRewriteAction::Keep)); } @@ -3076,8 +3026,12 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= assert!(!rewritten.contains("https://www.google-analytics.com")); } + /// Settings that run this module's middleware on every page, for the + /// tests that select the module. fn make_settings() -> Settings { - create_test_settings() + let mut settings = create_test_settings(); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); + settings } fn config_from_settings( @@ -3190,7 +3144,7 @@ assume_single_jurisdiction = true let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -3230,7 +3184,7 @@ assume_single_jurisdiction = true let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -3295,7 +3249,7 @@ assume_single_jurisdiction = true let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -3336,6 +3290,136 @@ assume_single_jurisdiction = true ); } + /// The page a reader receives when `html` arrives from the origin + /// `chunk_size` bytes at a time, with Google Tag Manager running. + fn page_with_gtm(html: &str, chunk_size: usize) -> String { + let mut settings = make_settings(); + settings + .insert_module_config( + "tag", + MODULE, + &serde_json::json!({ + "container_id": "GTM-12345", + "upstream_url": "https://www.googletagmanager.com" + }), + ) + .expect("should update config"); + let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) + .expect("should create registry"); + let config = config_from_settings(&settings, ®istry); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + create_page_processor(&settings, ®istry, config), + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should process"); + String::from_utf8(output).expect("should stay UTF-8") + } + + #[test] + fn the_rewritten_snippet_keeps_its_data_layer_parameter() { + // The standard snippet adds `'&l='+l` to the container's address when + // a page names its own data layer. A browser decodes no entity inside + // a script, so `&l=` would send a parameter called `amp;l`, and + // the container would read the default data layer instead. + let processed = page_with_gtm( + r#""#, + 8192, + ); + + assert!( + processed.contains("'/integrations/google_tag_manager/gtm.js?id='+i+dl"), + "should point the snippet at the first-party path: {processed}" + ); + assert!( + processed.contains("dl=l!='dataLayer'?'&l='+l:''"), + "should keep the data layer parameter as the snippet wrote it: {processed}" + ); + assert!( + !processed.contains("&"), + "should escape nothing inside the script: {processed}" + ); + } + + #[test] + fn a_loader_of_the_page_s_own_keeps_its_operators() { + let processed = page_with_gtm( + r#""#, + 8192, + ); + + assert!( + processed.contains( + "" + ), + "should change the address and leave the script's operators as they were: \ + {processed}" + ); + } + + #[test] + fn a_script_held_for_a_marker_that_never_comes_is_written_back_as_it_was() { + // The first chunk ends on `google`, which could be the start of a + // marker, so the script's text is held until its last chunk shows it + // names no Google Tag Manager host. + let script = r#"var engine = "google"; if (a && b < c) { search(engine); }"#; + let html = format!(""); + let first_chunk_ends = html.find("google").expect("should find the word") + "google".len(); + + let processed = page_with_gtm(&html, first_chunk_ends); + + assert!( + processed.contains(&format!("")), + "should write a script it held and did not change back as the page wrote it: \ + {processed}" + ); + } + + #[test] + fn a_script_held_and_then_let_go_keeps_its_order() { + // The first chunk ends on `google`, so the script's text is held. Two + // chunks later nothing held can become a marker any more, and what was + // held has to go out ahead of the text that follows it. + let script = r#"var engine = "google"; var one = 1; var two = 2; var three = 3; var four = 4; var five = 5; var six = 6;"#; + let html = format!(""); + let first_chunk_ends = html.find("google").expect("should find the word") + "google".len(); + + let processed = page_with_gtm(&html, first_chunk_ends); + + assert!( + processed.contains(&format!("")), + "should write the script back in the order the page wrote it: {processed}" + ); + } + + #[test] + fn a_script_that_writes_markup_keeps_its_order() { + // The page arrives in one piece. The parser still hands a script's + // text over in parts, split where a `<` might close the script, so the + // part ending on `google` is held without any help from the network. + let script = + r#"footer.innerHTML = 'Search with google or another engine';"#; + let html = format!(""); + + let processed = page_with_gtm(&html, 8192); + + assert!( + processed.contains(&format!("")), + "should write the script back in the order the page wrote it: {processed}" + ); + } + #[test] fn test_container_id_validation_accepts_valid_ids() { // Valid container IDs with different lengths @@ -3434,32 +3518,14 @@ assume_single_jurisdiction = true #[test] fn fragmented_gtm_snippet_is_accumulated_and_rewritten() { - let config = tag_config("GTM-FRAG1", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let document_state = IntegrationDocumentState::default(); - // Simulate lol_html splitting the GTM snippet mid-domain. let fragment1 = r#"(function(w,d,s,l,i){j.src='https://www.google"#; let fragment2 = r#"tagmanager.com/gtm.js?id='+i;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-FRAG1');"#; - let ctx_intermediate = IntegrationScriptContext { - selector: "script", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: false, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &document_state, - }; - let ctx_last = IntegrationScriptContext { - is_last_in_text_node: true, - ..ctx_intermediate - }; + let mut script = InlineSnippet::default(); // Intermediate fragment: should be suppressed. - let action1 = - IntegrationScriptRewriter::rewrite(&*integration, fragment1, &ctx_intermediate); + let action1 = script.decide(fragment1, false); assert_eq!( action1, ScriptRewriteAction::RemoveNode, @@ -3467,7 +3533,7 @@ assume_single_jurisdiction = true ); // Last fragment: should emit full rewritten content. - let action2 = IntegrationScriptRewriter::rewrite(&*integration, fragment2, &ctx_last); + let action2 = script.decide(fragment2, true); match action2 { ScriptRewriteAction::Replace(rewritten) => { assert!( @@ -3485,32 +3551,31 @@ assume_single_jurisdiction = true #[test] fn an_interrupted_document_leaves_no_residue_for_the_next_document() { - // The registry holds one `Arc` for the - // lifetime of the application, so both documents below go through the - // SAME rewriter instance. Only the document state differs. With the - // buffer owned by the rewriter this test fails: document two emits - // document one's secret. + // The registry holds one middleware for the lifetime of the + // application, so both documents below go through the SAME instance. + // Each is given a handler of its own, which is what holds the text. + // With the text held by the middleware this test fails: document two + // emits document one's secret. let integration = GoogleTagManagerIntegration::new(tag_config("GTM-LEAK01", &[])); + let handler_for = |document_state: &IntegrationDocumentState| { + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + document_state, + ); + integration.create(&context).text_handlers.remove(0) + }; // Document one: a GTM snippet that is cut off before its final // fragment ever arrives, as a client disconnect or truncated origin // body would do. let first_document = IntegrationDocumentState::default(); - let interrupted = IntegrationScriptContext { - selector: "script", - request_host: "first.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: false, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &first_document, - }; + let mut interrupted = handler_for(&first_document); // Must end mid-domain: that is what makes the cheap prefix gate // accumulate rather than pass the fragment through untouched. let secret = r#"(function(w,d,s,l,i){var token='SESSION-ONE-SECRET'; j.src='https://www.google"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, secret, &interrupted); + let action = interrupted.decide(secret, false); assert_eq!( action, ScriptRewriteAction::RemoveNode, @@ -3518,20 +3583,12 @@ assume_single_jurisdiction = true ); // Document two: a different request, a fresh document state, the same - // registry and the same rewriter. + // registry and the same middleware. let second_document = IntegrationDocumentState::default(); - let fresh = IntegrationScriptContext { - selector: "script", - request_host: "second.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: true, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &second_document, - }; + let mut fresh = handler_for(&second_document); let benign = r#"(function(w,d,s,l,i){j.src='https://www.googletagmanager.com/gtm.js?id='+i;})(window,document,'script','dataLayer','GTM-LEAK01');"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, benign, &fresh); + let action = fresh.decide(benign, true); let emitted = match action { ScriptRewriteAction::Replace(rewritten) => rewritten, @@ -3556,40 +3613,21 @@ assume_single_jurisdiction = true #[test] fn non_gtm_fragmented_script_returns_keep_on_every_fragment() { // A script with no GTM marker in flight (and no plausible prefix) must - // return `Keep` on every fragment. Returning `RemoveNode` + - // `Replace(unchanged)` would stomp on other rewriters' replacements - // when selectors overlap (see `GoogleTagManagerIntegration::rewrite`). - let config = tag_config("GTM-PASS1", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let document_state = IntegrationDocumentState::default(); - + // return `Keep` on every fragment, so a script that is none of this + // module's business is never held or written again. let fragment1 = "console.log('hel"; let fragment2 = "lo world');"; - let ctx_intermediate = IntegrationScriptContext { - selector: "script", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: false, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &document_state, - }; - let ctx_last = IntegrationScriptContext { - is_last_in_text_node: true, - ..ctx_intermediate - }; + let mut script = InlineSnippet::default(); - let action1 = - IntegrationScriptRewriter::rewrite(&*integration, fragment1, &ctx_intermediate); + let action1 = script.decide(fragment1, false); assert_eq!( action1, ScriptRewriteAction::Keep, "non-GTM intermediate fragment should not be claimed" ); - let action2 = IntegrationScriptRewriter::rewrite(&*integration, fragment2, &ctx_last); + let action2 = script.decide(fragment2, true); assert_eq!( action2, ScriptRewriteAction::Keep, @@ -3602,33 +3640,16 @@ assume_single_jurisdiction = true /// is a fragmented non-GTM script. Both must produce correct output. #[test] fn accumulation_buffer_drains_between_consecutive_script_elements() { - let config = tag_config("GTM-MULTI1", &[]); - let integration = GoogleTagManagerIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - // --- First ", + "", + "", + ); + + /// [`PROBE_PAGE`] as a reader receives it from a deployment running + /// `settings` with the probe registered, read `chunk_size` bytes at a + /// time. + fn probe_page(settings: &Settings, chunk_size: usize) -> String { + use trusted_server_core::html_processor::test_support::processed_page; + use trusted_server_core::integrations::IntegrationRegistry; + + let registry = IntegrationRegistry::with_registrations(settings, &[builder()]) + .expect("should build a registry with the probe registered"); + processed_page(settings, ®istry, PROBE_PAGE, chunk_size) + } + + #[test] + fn the_middleware_changes_a_page_where_an_entry_names_it() { + use trusted_server_core::html_processor::test_support::REQUEST_HOST; + + let settings = settings_with_probe( + r#"country = "NZ" + + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] + "#, + ); + + // Read whole, and then seven bytes at a time so the script's text + // reaches the handler in pieces. + for chunk_size in [8192, 7] { + let page = probe_page(&settings, chunk_size); + + assert_eq!( + page.matches(&seam_probe_head_marker("NZ")).count(), + 1, + "should mark the head once, with the country the probe is set to: {page}" + ); + assert!( + page.contains(&format!( + "probe" + )), + "should point the probe's link at its report route: {page}" + ); + assert!( + page.contains(&format!( + "other" + )), + "should leave another link to core, which moves it to the reader's host: {page}" + ); + assert!( + page.contains(""), + "should write the country into the probe's script: {page}" + ); + assert!( + page.contains(""), + "should leave another script as it was: {page}" + ); + } + } + + #[test] + fn the_serve_middleware_marks_a_reader_s_copy_where_an_entry_names_it() { + use trusted_server_core::html_processor::test_support::REQUEST_HOST; + + let settings = settings_with_probe( + r#"country = "NZ" + + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] + + [[serve]] + media_type = "text/html" + middleware = ["testing.seam-probe.reader"] + "#, + ); + + let page = probe_page(&settings, 8192); + + let country = page + .find(&seam_probe_head_marker("NZ")) + .expect("should mark the head with the country"); + let reader = page + .find(&seam_probe_reader_marker(REQUEST_HOST)) + .expect("should mark the head with the host the reader asked for"); + let bundle = page + .find("id=\"trustedserver-js\"") + .expect("should carry the script bundle"); + assert!( + country < reader && reader < bundle, + "should write the serve middleware's mark after the fetch one's and ahead of the \ + bundle: {page}" + ); + assert_eq!( + reader_middleware_name(), + "testing.seam-probe.reader", + "should name the serve middleware after the module" + ); + } + + #[test] + fn the_middleware_changes_nothing_where_no_entry_names_it() { + use trusted_server_core::html_processor::test_support::REQUEST_HOST; + + let page = probe_page(&settings_with_probe(r#"country = "NZ""#), 8192); + + assert!( + !page.contains("seam-probe-country") && !page.contains("seam-probe-reader"), + "should not mark the head of a page no entry places it on: {page}" + ); + assert!( + page.contains(&format!( + "probe" + )), + "should leave the probe's link to core: {page}" + ); + assert!( + page.contains(""), + "should leave the probe's script as it was: {page}" + ); + } + /// The registry rejects a carried module whose declared hash is wrong, so /// this keeps the fixture's literal honest rather than letting every other /// test fail with a startup error. diff --git a/crates/testing/testlight/.gitattributes b/crates/testing/testlight/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/testing/testlight/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/testing/testlight/src/fixtures/page-change.input.html b/crates/testing/testlight/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..74055212dc --- /dev/null +++ b/crates/testing/testlight/src/fixtures/page-change.input.html @@ -0,0 +1,16 @@ + + + + +Testlight + + + + + + +An article on the origin + +

testlight.js is named here in text, which is left as it is.

+ + diff --git a/crates/testing/testlight/src/fixtures/page-change.recorded.html b/crates/testing/testlight/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..386f2effe4 --- /dev/null +++ b/crates/testing/testlight/src/fixtures/page-change.recorded.html @@ -0,0 +1,16 @@ + + + + +Testlight + + + + + + +An article on the origin + +

testlight.js is named here in text, which is left as it is.

+ + diff --git a/crates/testing/testlight/src/fixtures/page-change.settings.toml b/crates/testing/testlight/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..aa1a4e0c53 --- /dev/null +++ b/crates/testing/testlight/src/fixtures/page-change.settings.toml @@ -0,0 +1,11 @@ +[auction] +modules = ["testing.testlight"] + +[auction.testing.testlight] +endpoint = "https://example.com/openrtb2/auction" +rewrite_scripts = true +shim_src = "https://edge.example.com/static/testlight.js" + +[[fetch]] +media_type = "text/html" +middleware = ["testing.testlight"] diff --git a/crates/testing/testlight/src/lib.rs b/crates/testing/testlight/src/lib.rs index 884af941a6..f819feafe6 100644 --- a/crates/testing/testlight/src/lib.rs +++ b/crates/testing/testlight/src/lib.rs @@ -9,6 +9,7 @@ reason = "tests use direct diagnostics and panic-on-failure helpers" ) )] +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -23,9 +24,13 @@ use validator::Validate; use trusted_server_core::edge_cookie::recognized_ec_id; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, - UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_body_bounded, collect_response_bounded, + AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, + IntegrationRegistration, UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_body_bounded, + collect_response_bounded, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{ProxyRequestConfig, proxy_request}; @@ -192,7 +197,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(TESTLIGHT_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(ShimScript(integration))) .build(), )) } @@ -280,30 +285,43 @@ impl IntegrationProxy for TestlightIntegration { } } -impl IntegrationAttributeRewriter for TestlightIntegration { - fn integration_id(&self) -> &'static str { - TESTLIGHT_INTEGRATION_ID +impl TestlightIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the shim + /// when it is the vendor's script. + fn rewrite_script_address(&self, value: &str) -> AttributeRewriteAction { + let lowered = value.to_ascii_lowercase(); + if lowered.contains("testlight.js") { + AttributeRewriteAction::replace(self.config.shim_src.clone()) + } else { + AttributeRewriteAction::keep() + } } +} + +/// Loads the shim in place of the vendor's script, on the pages a `[[fetch]]` +/// entry names [`MODULE`] for. It does nothing unless `rewrite_scripts` is +/// set. +struct ShimScript(Arc); - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_scripts && matches!(attribute, "src" | "href") +impl Middleware for ShimScript { + fn middleware_id(&self) -> &'static str { + MODULE } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if !self.config.rewrite_scripts { - return AttributeRewriteAction::keep(); - } + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } - let lowered = attr_value.to_ascii_lowercase(); - if lowered.contains("testlight.js") { - AttributeRewriteAction::replace(self.config.shim_src.clone()) - } else { - AttributeRewriteAction::keep() + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + if !self.0.config.rewrite_scripts { + return MiddlewareAction::pass(); + } + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_script_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() } } } @@ -353,16 +371,8 @@ mod tests { }; let integration = TestlightIntegration::new(config); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - let rewritten = - integration.rewrite("src", "https://cdn.testlight.net/v1/testlight.js", &ctx); + integration.rewrite_script_address("https://cdn.testlight.net/v1/testlight.js"); assert!( matches!( rewritten, @@ -382,18 +392,16 @@ mod tests { rewrite_scripts: false, }; let integration = TestlightIntegration::new(config); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); - assert!(matches!( - integration.rewrite("src", "https://cdn.testlight.net/script.js", &ctx), - AttributeRewriteAction::Keep - )); + assert!( + ShimScript(integration).create(&context).is_pass(), + "should leave every page alone when script rewriting is off" + ); } #[test] @@ -613,7 +621,10 @@ mod tests { fn a_served_page_loads_the_shim_in_place_of_the_vendor_script() { use std::io::Cursor; - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::streaming_processor::{ Compression, PipelineConfig, StreamingPipeline, @@ -636,6 +647,7 @@ mod tests { }), ) .expect("should insert testlight config"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); @@ -652,7 +664,7 @@ mod tests { output_compression: Compression::None, chunk_size: 8192, }, - create_html_processor(config), + create_page_processor(&settings, ®istry, config), ); let mut output = Vec::new(); @@ -670,4 +682,21 @@ mod tests { "the vendor's own URL should be gone from the page" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/trusted-server-adapter-axum/tests/seam_probe.rs b/crates/trusted-server-adapter-axum/tests/seam_probe.rs index ae5fd82717..ecd158bf25 100644 --- a/crates/trusted-server-adapter-axum/tests/seam_probe.rs +++ b/crates/trusted-server-adapter-axum/tests/seam_probe.rs @@ -380,6 +380,112 @@ fn geo_selector_naming_a_module_without_a_geo_module_fails_at_startup() { ); } +/// A `[[fetch]]` entry naming the probe's middleware starts the adapter, and +/// the registry runs that middleware on the paths the entry covers. +#[test] +fn fetch_entry_naming_a_modules_middleware_starts_the_adapter() { + use trusted_server_core::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; + + let settings = || { + settings_with(&format!( + r#" + {PROBE_BLOCK} + + [[fetch]] + media_type = "text/html" + path = "/news/" + middleware = ["testing.seam-probe"] + "# + )) + }; + + let registry = registry_with_probe(&settings()); + let chain_for = |path: &str| { + registry + .middleware_chain( + &settings().fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + path, + ) + .ids() + }; + assert_eq!( + chain_for("/news/today"), + [seam_probe::module_name()], + "should run the module's middleware under the entry's path" + ); + assert!( + chain_for("/sport/today").is_empty(), + "should run nothing on a path no entry covers" + ); + + TrustedServerApp::routes_with_registrations(settings(), &[seam_probe::builder()]) + .expect("should start with an entry naming a middleware the module supplies"); +} + +/// A `[[serve]]` entry starts the adapter when it names a middleware that +/// runs in that phase, and is a startup error when it names one that does +/// not. +#[test] +fn serve_entry_starts_the_adapter_only_with_a_middleware_of_that_phase() { + let settings = |name: &str| { + settings_with(&format!( + r#" + {PROBE_BLOCK} + + [[serve]] + media_type = "text/html" + middleware = ["{name}"] + "# + )) + }; + + TrustedServerApp::routes_with_registrations( + settings(seam_probe::reader_middleware_name()), + &[seam_probe::builder()], + ) + .expect("should start with a serve entry naming the module's serve middleware"); + + let error = TrustedServerApp::routes_with_registrations( + settings(seam_probe::module_name()), + &[seam_probe::builder()], + ) + .err() + .expect("should refuse to start when a serve entry names a fetch middleware"); + + let message = error.to_string(); + assert!( + message.contains("[[serve]] entry 1 names `testing.seam-probe`") + && message.contains("does not run in that phase. It runs in [[fetch]]"), + "should say which phase the middleware runs in: {message}" + ); +} + +/// A `[[fetch]]` entry naming the middleware of a module no section selects +/// is a startup error, raised where the adapter builds its routes. +#[test] +fn fetch_entry_naming_the_middleware_of_an_unselected_module_fails_at_startup() { + let settings = settings_with( + r#" + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] + "#, + ); + + let error = TrustedServerApp::routes_with_registrations(settings, &[seam_probe::builder()]) + .err() + .expect("should refuse to start when an entry names a middleware that does not run"); + + let message = error.to_string(); + assert!( + message.contains("[[fetch]] entry 1 names `testing.seam-probe`") + && message.contains("`testing.seam-probe` is a module no section selects"), + "should name the entry and say why the middleware is not running: {message}" + ); +} + // --------------------------------------------------------------------------- // 5. Two builders claiming one id are rejected, naming both sources // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs index 8be7ef8aae..5b3dce2ad9 100644 --- a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs +++ b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs @@ -258,6 +258,75 @@ fn a_named_route_and_the_fallback_each_prepare_the_request_exactly_once() { ); } +/// A `[[fetch]]` entry naming the probe's middleware builds this adapter's +/// state, and one naming the middleware of a module no section selects is a +/// startup error. +#[test] +fn fetch_entry_builds_state_only_when_its_middleware_is_running() { + const ENTRY: &str = r#" + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] +"#; + + let selected = settings_with(&format!("{HMAC_BLOCK}{PROBE_BLOCK}{ENTRY}")); + assert!( + build_state_with_registrations(selected, &[seam_probe::builder()]).is_ok(), + "should build state with an entry naming a middleware the module supplies" + ); + + let unselected = settings_with(&format!("{HMAC_BLOCK}{ENTRY}")); + let error = build_state_with_registrations(unselected, &[seam_probe::builder()]) + .err() + .expect("should refuse to start when an entry names a middleware that does not run"); + + let message = error.to_string(); + assert!( + message.contains("[[fetch]] entry 1 names `testing.seam-probe`") + && message.contains("`testing.seam-probe` is a module no section selects"), + "should name the entry and say why the middleware is not running: {message}" + ); +} + +/// A `[[serve]]` entry builds this adapter's state when it names a +/// middleware that runs in that phase, and is a startup error when it names +/// one that does not. +#[test] +fn serve_entry_builds_state_only_with_a_middleware_of_that_phase() { + let settings = |name: &str| { + settings_with(&format!( + r#"{HMAC_BLOCK}{PROBE_BLOCK} + [[serve]] + media_type = "text/html" + middleware = ["{name}"] +"# + )) + }; + + assert!( + build_state_with_registrations( + settings(seam_probe::reader_middleware_name()), + &[seam_probe::builder()] + ) + .is_ok(), + "should build state with a serve entry naming the module's serve middleware" + ); + + let error = build_state_with_registrations( + settings(seam_probe::module_name()), + &[seam_probe::builder()], + ) + .err() + .expect("should refuse to start when a serve entry names a fetch middleware"); + + let message = error.to_string(); + assert!( + message.contains("[[serve]] entry 1 names `testing.seam-probe`") + && message.contains("does not run in that phase. It runs in [[fetch]]"), + "should say which phase the middleware runs in: {message}" + ); +} + /// `[geo] module` naming a selected module that supplies no geo module is a /// startup error, raised where this adapter builds its state. #[test] diff --git a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs index 16f3017354..93762a011a 100644 --- a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs +++ b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs @@ -19,6 +19,7 @@ use serde::Serialize; use trusted_server_core::creative_opportunities::{ CreativeOpportunitiesConfig, validate_page_pattern, }; +use trusted_server_core::middleware::MiddlewarePhase; use url::Url; use crate::commands::audit::ad_templates::{origin_changed, without_fragment}; @@ -160,6 +161,7 @@ struct DraftConfig { /// Id of the integration the audit writes its discovered assets for. const JS_ASSET_PROXY_ID: &str = "js_asset_proxy"; +const GOOGLE_TAG_MANAGER_ID: &str = "google_tag_manager"; #[derive(Debug, Clone)] struct JsAssetProxySection { @@ -480,7 +482,9 @@ fn build_draft_config_with_generator( } // The template documents every module as a commented example, so the - // sections the audit fills in are appended after it. + // sections the audit fills in are appended after it. The integrations + // whose modules it selects are kept, so their page changes can be placed. + let mut selected: Vec<&str> = Vec::new(); for id in ["datadome", "didomi", "gpt"] { if detected.contains(id) && let Some((section, written)) = module_section(id) @@ -493,16 +497,22 @@ fn build_draft_config_with_generator( modules = [\"{written}\"]\n" ), ); + selected.push(id); } } if let Some(container_id) = >m_container_id { append_section(&mut draft, &build_google_tag_manager_section(container_id)); + selected.push(GOOGLE_TAG_MANAGER_ID); } if asset_proxy_section.candidate_count > 0 { append_section(&mut draft, &asset_proxy_section.toml); + selected.push(JS_ASSET_PROXY_ID); } else { append_section(&mut draft, &asset_proxy_section.notes); } + if let Some(entries) = build_page_entries(&selected) { + append_section(&mut draft, &entries); + } let mut manual_review = Vec::new(); if detected.contains("google_tag_manager") && gtm_container_id.is_none() { @@ -652,6 +662,40 @@ fn build_google_tag_manager_section(container_id: &str) -> String { ) } +/// The entries that run the middleware of the selected modules on every +/// HTML page, or `None` when none of them changes a page. +/// +/// A module changes a page only where an entry names its middleware, so a +/// draft that selected a module and wrote no entry would leave the module +/// changing nothing. The names and their order come from the stock list. +fn build_page_entries(integration_ids: &[&str]) -> Option { + let mut entries = String::new(); + for phase in MiddlewarePhase::ALL { + let names = trusted_server_modules::middleware_for(integration_ids, phase); + if names.is_empty() { + continue; + } + if entries.is_empty() { + entries.push_str( + "# Written by `ts audit`. A module changes a page only where an entry names\n\ + # its middleware, so these run what the modules selected above supply on\n\ + # every HTML page, in the order written.\n", + ); + } else { + entries.push('\n'); + } + let listed = names + .iter() + .map(|name| toml_string(name)) + .collect::>() + .join(", "); + entries.push_str(&format!( + "[[{phase}]]\nmedia_type = \"text/html\"\nmiddleware = [{listed}]\n" + )); + } + (!entries.is_empty()).then_some(entries) +} + /// Selects the JavaScript asset proxy in the template's `[proxy]` section by /// uncommenting the selection the template documents there. fn select_js_asset_proxy(draft: &str) -> CliResult { @@ -2741,6 +2785,146 @@ mod tests { ); } + /// What the draft's entries of `phase` hold, each as its media type, + /// whether it has a path, and the middleware it names. + fn drafted_entries(parsed: &toml::Value, phase: &str) -> Vec<(String, bool, Vec)> { + parsed + .get(phase) + .and_then(toml::Value::as_array) + .map(|entries| { + entries + .iter() + .map(|entry| { + ( + entry["media_type"] + .as_str() + .expect("should write a media type") + .to_owned(), + entry.get("path").is_some(), + entry["middleware"] + .as_array() + .expect("should write the middleware") + .iter() + .filter_map(toml::Value::as_str) + .map(str::to_owned) + .collect(), + ) + }) + .collect() + }) + .unwrap_or_default() + } + + #[test] + fn build_draft_config_places_the_page_changes_of_the_modules_it_selects() { + let url = Url::parse("https://publisher.example.com/page").expect("should parse URL"); + let mut detected_integrations: Vec = ["gpt", "datadome", "didomi"] + .into_iter() + .map(|id| DetectedIntegration { + id: id.to_string(), + evidence: format!("https://{id}.example.com/tag.js"), + }) + .collect(); + detected_integrations.push(DetectedIntegration { + id: "google_tag_manager".to_string(), + evidence: "GTM-ABC123".to_string(), + }); + let artifact = AuditArtifact { + audited_url: url.to_string(), + page_title: Some("Example".to_string()), + js_asset_count: 1, + third_party_asset_count: 1, + detected_integrations, + assets: vec![audited_asset( + "https://cdn.vendor.example.com/sdk.js", + AssetParty::ThirdParty, + None, + )], + warnings: Vec::new(), + }; + let mut generator = FixedPathGenerator::new(&["/assets/aaaaaaaaaaaaaaaaaaaaaaaa.js"]); + + let draft = build_draft_config_with_generator( + &url, + &artifact, + &gpt_slots::DiscoveredSlots::default(), + &mut generator, + ) + .expect("should build draft config"); + + let parsed = + toml::from_str::(&draft.toml).expect("draft should parse as TOML"); + assert_eq!( + drafted_entries(&parsed, "fetch"), + vec![( + "text/html".to_owned(), + false, + vec![ + "js_asset_proxy".to_owned(), + "cmp.didomi".to_owned(), + "tag.google-tag-manager".to_owned(), + "bot-protection.datadome".to_owned(), + "ad-tag.google".to_owned(), + ], + )], + "should run the fetch middleware of every module it selected on every HTML \ + page, in the order the stock modules register" + ); + assert_eq!( + drafted_entries(&parsed, "serve"), + vec![( + "text/html".to_owned(), + false, + vec!["bot-protection.datadome.tag".to_owned()], + )], + "should run the one serve middleware the selected modules supply" + ); + trusted_server_core::settings::Settings::from_toml( + &draft + .toml + .replace( + "password = \"handler_password\"", + "password = \"test-admin-password-32-bytes-minimum\"", + ) + .replace( + "passphrase = \"ec_passphrase\"", + "passphrase = \"test-ec-passphrase-32-bytes-minimum\"", + ) + .replace( + "proxy_secret = \"publisher_proxy_secret\"", + "proxy_secret = \"test-proxy-secret-32-bytes-minimum\"", + ), + ) + .expect("should load the draft, its entries included, as settings"); + } + + #[test] + fn build_draft_config_writes_no_entry_when_it_selects_no_module() { + let url = Url::parse("https://publisher.example/path").expect("should parse URL"); + let artifact = AuditArtifact { + audited_url: url.to_string(), + page_title: None, + js_asset_count: 0, + third_party_asset_count: 0, + detected_integrations: vec![DetectedIntegration { + id: "prebid".to_string(), + evidence: "inline script matched `prebid`".to_string(), + }], + assets: Vec::new(), + warnings: Vec::new(), + }; + + let draft = build_draft_config(&url, &artifact, &gpt_slots::DiscoveredSlots::default()) + .expect("should build draft config"); + + let parsed = toml::from_str::(&draft).expect("draft should parse as TOML"); + assert!( + drafted_entries(&parsed, "fetch").is_empty() + && drafted_entries(&parsed, "serve").is_empty(), + "should place nothing for a module it leaves to manual review" + ); + } + #[test] fn build_draft_config_extracts_gtm_container_id_from_url_evidence() { let url = Url::parse("https://publisher.example.com").expect("should parse URL"); diff --git a/crates/trusted-server-core/.gitattributes b/crates/trusted-server-core/.gitattributes new file mode 100644 index 0000000000..6e7a419fbc --- /dev/null +++ b/crates/trusted-server-core/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/integrations/fixtures/*.html text eol=lf diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index 8dfbaaf794..ca34ca3568 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -356,6 +356,7 @@ pub fn validate_settings_for_deploy_with( // The selection is checked first, so a block nothing runs is reported as // that rather than as whatever its unread settings fail next. settings.validate_module_sections()?; + settings.validate_phase_entries()?; validate_secret_key_references(settings, extra_integrations)?; validate_non_secret_deploy_placeholders(settings)?; diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index c77b678b10..7fcc4c2e1b 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -1,7 +1,7 @@ //! Simplified HTML processor that combines URL replacement and integration injection //! //! This module provides a `StreamProcessor` implementation for HTML content. -use std::cell::Cell; +use std::cell::{Cell, RefCell}; use std::io; use std::rc::Rc; use std::sync::Arc; @@ -9,20 +9,27 @@ use std::sync::atomic::{AtomicBool, Ordering}; use lol_html::{ EndTagHandler, Settings as RewriterSettings, element, end, - html_content::{ContentType, EndTag}, + html_content::{ContentType, EndTag, TextChunk, TextType}, text, }; use crate::integrations::{ - AttributeRewriteOutcome, IntegrationAttributeContext, IntegrationDocumentState, - IntegrationHtmlContext, IntegrationRegistry, IntegrationRequestState, IntegrationScriptContext, + AttributeRewriteAction, IntegrationDocumentState, IntegrationRegistry, IntegrationRequestState, ScriptRewriteAction, }; +use crate::middleware::{ + ElementHandler, MatchedAttribute, MiddlewareChain, MiddlewareContext, MiddlewarePlan, + TextHandler, +}; use crate::publisher::build_empty_bids_script; use crate::settings::Settings; use crate::streaming_processor::{HtmlRewriterAdapter, StreamProcessor}; use crate::tsjs; +mod script_text; + +use script_text::ScriptTextGuard; + struct HtmlWithStreamingProcessors { inner: Box, processors: Vec>, @@ -96,10 +103,10 @@ pub struct HtmlProcessorConfig { /// Maximum bytes an integration may retain while processing one script or /// unresolved streaming group. pub max_buffered_body_bytes: usize, - /// What modules left on the request for their page hooks, copied into the - /// document's state before parsing starts. Empty for a document that may - /// be stored and served to other readers, where nothing made for one - /// request may appear. + /// What modules left on the request, which the serve chain is handed + /// where it runs on the same pass as the fetch chain. The fetch chain is + /// handed none of it, because what that chain writes may be stored and + /// served to other readers. pub request_state: IntegrationRequestState, /// What the `` seam injects. Decided by the caller rather than inferred /// from [`Self::ad_slots_script`]. @@ -176,7 +183,7 @@ impl HtmlProcessorConfig { self } - /// Attach what modules left on the request for their page hooks. + /// Attach what modules left on the request, for the serve chain. #[must_use] pub fn with_request_state(mut self, request_state: IntegrationRequestState) -> Self { self.request_state = request_state; @@ -194,48 +201,352 @@ impl HtmlProcessorConfig { } } -/// Create an HTML processor with URL replacement and integration hooks. +/// The chunk of text being rewritten, as the last handler to change it left +/// it, shared by every text handler of one document. +/// +/// The parser hands every handler a chunk as the origin sent it and writes +/// only the last replacement, so without this a handler would undo the change +/// of each handler before it. +#[derive(Default)] +struct TextEdits(RefCell); + +impl TextEdits { + /// The chunk as the handlers before this one left it. + fn current(&self, chunk: &TextChunk<'_>) -> String { + if chunk.removed() { + self.0.borrow().clone() + } else { + chunk.as_str().to_owned() + } + } + + /// Writes `text` in the chunk's place, as it is. + /// + /// Written as markup, because a handler is handed the text as the page + /// wrote it. Writing it as text would escape `&`, `<` and `>`, and a + /// browser decodes none of those inside a script. + fn replace(&self, chunk: &mut TextChunk<'_>, text: &str) { + chunk.replace(text, ContentType::Html); + text.clone_into(&mut self.0.borrow_mut()); + } + + /// Leaves the chunk out, along with anything an earlier handler wrote in + /// its place. + fn remove(&self, chunk: &mut TextChunk<'_>) { + self.replace(chunk, ""); + } +} + +/// Create an HTML processor that moves the origin's address to the +/// publisher's and writes the script bundle, with no middleware. /// /// # Panics /// /// Panics if the `ad_bids_state` `Mutex` is poisoned. This cannot happen in /// normal operation since no code holds the lock across a panic boundary. #[must_use] +pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcessor { + build_html_processor(config, MiddlewarePlan::default()) +} + +/// [`create_html_processor`], with the middleware of the `fetch` chain run +/// on the document as well, and then those of the `serve` chain where the +/// document goes to one reader and is not stored. +/// +/// `serve` is `None` for a document that may be stored, whose serve chain +/// runs on each reader's copy instead, through [`create_serve_processor`]. +/// +/// # Errors +/// +/// When a handler's selector does not parse, naming the selector. +/// +/// # Panics +/// +/// As [`create_html_processor`]. +pub fn create_html_processor_with_middleware( + config: HtmlProcessorConfig, + fetch: &MiddlewareChain, + serve: Option<&MiddlewareChain>, +) -> Result, String> { + let document_state = IntegrationDocumentState::default(); + let plan = fetch.plan(&MiddlewareContext { + phase: fetch.phase(), + request_host: &config.request_host, + request_scheme: &config.request_scheme, + origin_host: &config.origin_host, + document_state: &document_state, + max_buffered_script_bytes: config.max_buffered_body_bytes, + })?; + let reader = match serve { + Some(serve) => create_serve_processor( + serve, + &ReaderDocument { + request_host: &config.request_host, + request_scheme: &config.request_scheme, + origin_host: &config.origin_host, + request_state: &config.request_state, + max_buffered_script_bytes: config.max_buffered_body_bytes, + }, + )?, + None => None, + }; + let mut processor = build_html_processor(config, plan); + processor.processors.extend(reader); + Ok(processor) +} + +/// What a serve chain is told about the reader's copy it works on. +#[derive(Debug, Clone, Copy)] +pub struct ReaderDocument<'a> { + /// Publisher-facing host the reader asked for. + pub request_host: &'a str, + /// Publisher-facing scheme the reader asked for. + pub request_scheme: &'a str, + /// Host the document was fetched from. + pub origin_host: &'a str, + /// What modules left on this reader's request for their page changes. + pub request_state: &'a IntegrationRequestState, + /// The most a middleware may hold of one script while it decides. + pub max_buffered_script_bytes: usize, +} + +/// A processor that carries out a serve chain alone, on one reader's copy of +/// a document core has already rewritten, or `None` when the chain has +/// nothing to do. +/// +/// Nothing of core's own is written or rewritten a second time. Head markup +/// goes straight before the script bundle's tag and markup for after the +/// bundle straight after it, so a document core wrote no bundle into gets +/// neither. +/// +/// # Errors +/// +/// When a handler's selector does not parse, naming the selector. +pub fn create_serve_processor( + chain: &MiddlewareChain, + document: &ReaderDocument<'_>, +) -> Result>, String> { + if chain.is_empty() { + return Ok(None); + } + let document_state = IntegrationDocumentState::default(); + document.request_state.seed(&document_state); + let plan = chain.plan(&MiddlewareContext { + phase: chain.phase(), + request_host: document.request_host, + request_scheme: document.request_scheme, + origin_host: document.origin_host, + document_state: &document_state, + max_buffered_script_bytes: document.max_buffered_script_bytes, + })?; + if plan.is_empty() { + return Ok(None); + } + Ok(Some(Box::new(build_serve_processor(plan)))) +} + +/// The tag of the script bundle, which core writes once at the start of +/// ``. +const BUNDLE_SELECTOR: &str = "script#trustedserver-js"; + +/// One selector and what a handler does with what it matches. +type ContentHandler = ( + std::borrow::Cow<'static, lol_html::Selector>, + lol_html::ElementContentHandlers<'static>, +); + +/// The handlers that carry out a plan's element and text handlers. +/// +/// The text handlers share `edits` with every text handler registered before +/// them, so each is handed a chunk as the one before it left it. A +/// [`ScriptTextGuard`] registered after them writes what they left in a +/// script, so that their text cannot end the script or keep its end tag from +/// closing it. +/// +/// A selector that does not parse panics here, which +/// [`MiddlewareChain::plan`] rules out by refusing to plan one. +fn middleware_content_handlers( + element_handlers: Vec>, + text_handlers: Vec>, + edits: &Rc, +) -> Vec { + let mut handlers: Vec = Vec::new(); + for mut handler in element_handlers { + let selector = handler.selector().to_owned(); + let attribute = handler.attribute().to_owned(); + handlers.push(element!(selector, move |el| { + if el.removed() { + return Ok(()); + } + let Some(value) = el.get_attribute(&attribute) else { + return Ok(()); + }; + let element_name = el.tag_name(); + match handler.decide(&MatchedAttribute { + element_name: &element_name, + attribute_name: &attribute, + value: &value, + }) { + AttributeRewriteAction::Keep => {} + AttributeRewriteAction::Replace(replacement) => { + if replacement != value { + el.set_attribute(&attribute, &replacement)?; + } + } + AttributeRewriteAction::RemoveElement => el.remove(), + } + Ok(()) + })); + } + let guarded = !text_handlers.is_empty(); + for mut handler in text_handlers { + let selector = handler.selector().to_owned(); + let edits = Rc::clone(edits); + handlers.push(text!(selector, move |text| { + let current = edits.current(text); + match handler.decide(¤t, text.last_in_text_node()) { + ScriptRewriteAction::Keep => {} + ScriptRewriteAction::Replace(rewritten) => edits.replace(text, &rewritten), + ScriptRewriteAction::RemoveNode => edits.remove(text), + } + Ok(()) + })); + } + if guarded { + let edits = Rc::clone(edits); + let mut guard = ScriptTextGuard::default(); + handlers.push(text!("script", move |text| { + if text.text_type() == TextType::ScriptData { + let written = edits.current(text); + if let Some(safe) = guard.write(&written, text.removed(), text.last_in_text_node()) + { + edits.replace(text, &safe); + } + } + Ok(()) + })); + } + handlers +} + +/// The processor behind [`create_serve_processor`], carrying out `plan` and +/// nothing else. +fn build_serve_processor(plan: MiddlewarePlan) -> HtmlWithStreamingProcessors { + let MiddlewarePlan { + head_inserts, + after_bundle_inserts, + element_handlers, + text_handlers, + processors, + } = plan; + let mut element_content_handlers: Vec = Vec::new(); + if !head_inserts.is_empty() || !after_bundle_inserts.is_empty() { + let before = head_inserts.concat(); + let after = after_bundle_inserts.concat(); + let written = Cell::new(false); + element_content_handlers.push(element!(BUNDLE_SELECTOR, move |el| { + // Core's tag is the first in the document. An element of the + // page's own under the same id is left alone. + if !written.replace(true) { + el.before(&before, ContentType::Html); + el.after(&after, ContentType::Html); + } + Ok(()) + })); + } + let edits = Rc::new(TextEdits::default()); + element_content_handlers.extend(middleware_content_handlers( + element_handlers, + text_handlers, + &edits, + )); + HtmlWithStreamingProcessors { + inner: Box::new(HtmlRewriterAdapter::new(RewriterSettings { + element_content_handlers, + ..RewriterSettings::default() + })), + processors, + } +} + +/// `value` with each occurrence of a form's text replaced by that form's +/// replacement, in one pass from the start, so no replacement is read again. +/// +/// The earliest match is taken each time, and the longer of two that start at +/// the same place. A form with empty text is ignored. +fn replace_each_once(value: &str, forms: &[(String, String)]) -> String { + let mut out = String::with_capacity(value.len()); + let mut rest = value; + loop { + let next = forms + .iter() + .filter(|(text, _)| !text.is_empty()) + .filter_map(|(text, replacement)| { + rest.find(text.as_str()) + .map(|at| (at, text.len(), replacement)) + }) + .min_by(|a, b| a.0.cmp(&b.0).then_with(|| b.1.cmp(&a.1))); + let Some((at, len, replacement)) = next else { + break; + }; + out.push_str(&rest[..at]); + out.push_str(replacement); + rest = &rest[at + len..]; + } + out.push_str(rest); + out +} + +/// The processor behind [`create_html_processor`] and +/// [`create_html_processor_with_middleware`], carrying out `plan`. +/// +/// A handler in `plan` whose selector does not parse panics here, which +/// [`MiddlewareChain::plan`] rules out by refusing to plan one. #[allow( clippy::needless_pass_by_value, reason = "the returned processor owns request configuration captured by its handlers" )] -pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcessor { - let stream_processor_factories = config.integrations.html_stream_processor_factories(); - let document_state = IntegrationDocumentState::default(); - config.request_state.seed(&document_state); - - // Simplified URL patterns structure - stores only core data and generates variants on-demand +fn build_html_processor( + config: HtmlProcessorConfig, + plan: MiddlewarePlan, +) -> HtmlWithStreamingProcessors { + let MiddlewarePlan { + head_inserts: middleware_head_inserts, + after_bundle_inserts: middleware_after_bundle_inserts, + element_handlers: middleware_element_handlers, + text_handlers: middleware_text_handlers, + processors: middleware_processors, + } = plan; + // The spellings of the origin's address in a URL attribute, each with its + // replacement, built once per document. They are replaced in one pass and a + // replacement is never read again, because a request host that begins with + // the origin host would otherwise match a later spelling inside an earlier + // spelling's replacement. struct UrlPatterns { origin_host: String, request_host: String, - request_scheme: String, + /// The `https://`, `http://` and protocol-relative spellings. + url_forms: Vec<(String, String)>, + /// The same and the bare host, which a `srcset` candidate may name. + srcset_forms: Vec<(String, String)>, } impl UrlPatterns { - fn https_origin(&self) -> String { - format!("https://{}", self.origin_host) - } - - fn http_origin(&self) -> String { - format!("http://{}", self.origin_host) - } - - fn protocol_relative_origin(&self) -> String { - format!("//{}", self.origin_host) - } - - fn replacement_url(&self) -> String { - format!("{}://{}", self.request_scheme, self.request_host) - } - - fn protocol_relative_replacement(&self) -> String { - format!("//{}", self.request_host) + fn new(origin_host: &str, request_host: &str, request_scheme: &str) -> Self { + let replacement_url = format!("{request_scheme}://{request_host}"); + let url_forms = vec![ + (format!("https://{origin_host}"), replacement_url.clone()), + (format!("http://{origin_host}"), replacement_url), + (format!("//{origin_host}"), format!("//{request_host}")), + ]; + let mut srcset_forms = url_forms.clone(); + srcset_forms.push((origin_host.to_owned(), request_host.to_owned())); + Self { + origin_host: origin_host.to_owned(), + request_host: request_host.to_owned(), + url_forms, + srcset_forms, + } } fn rewrite_url_value(&self, value: &str) -> Option { @@ -243,40 +554,36 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso return None; } - let https_origin = self.https_origin(); - let http_origin = self.http_origin(); - let protocol_relative_origin = self.protocol_relative_origin(); - let replacement_url = self.replacement_url(); - let protocol_relative_replacement = self.protocol_relative_replacement(); - - let mut rewritten = value - .replace(&https_origin, &replacement_url) - .replace(&http_origin, &replacement_url) - .replace(&protocol_relative_origin, &protocol_relative_replacement); - - if rewritten.starts_with(&self.origin_host) { - let suffix = &rewritten[self.origin_host.len()..]; - let boundary_ok = suffix.is_empty() - || matches!(suffix.as_bytes().first(), Some(b'/' | b'?' | b'#')); - if boundary_ok { - rewritten = format!("{}{}", self.request_host, suffix); + // A value that begins with the bare host is moved when the host + // ends there, so `origin.example.com/news` moves and + // `origin.example.com.mirror.example` does not. + let rewritten = match value.strip_prefix(self.origin_host.as_str()) { + Some(rest) + if rest.is_empty() + || matches!(rest.as_bytes().first(), Some(b'/' | b'?' | b'#')) => + { + format!( + "{}{}", + self.request_host, + replace_each_once(rest, &self.url_forms) + ) } - } + _ => replace_each_once(value, &self.url_forms), + }; (rewritten != value).then_some(rewritten) } } - let patterns = Rc::new(UrlPatterns { - origin_host: config.origin_host.clone(), - request_host: config.request_host.clone(), - request_scheme: config.request_scheme.clone(), - }); + let patterns = Rc::new(UrlPatterns::new( + &config.origin_host, + &config.request_host, + &config.request_scheme, + )); let injected_tsjs = Rc::new(Cell::new(false)); let injected_bids = Arc::new(AtomicBool::new(false)); let integration_registry = config.integrations.clone(); - let script_rewriters = integration_registry.script_rewriters(); let ad_slots_script = config.ad_slots_script.clone(); let permissions_script = config.permissions_script.clone(); let body_close = config.body_close.clone(); @@ -307,8 +614,6 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso element!("head", { let injected_tsjs = injected_tsjs.clone(); let integrations = integration_registry.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); let ad_slots_script = ad_slots_script.clone(); let permissions_script = permissions_script.clone(); move |el| { @@ -324,16 +629,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso if let Some(ref slots_script) = ad_slots_script { snippet.push_str(slots_script); } - let ctx = IntegrationHtmlContext { - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - document_state: &document_state, - }; - // First inject integration-specific config (e.g., window.__tsjs_prebid) - // so it's available when the bundle's auto-init code reads it. - for insert in integrations.head_inserts(&ctx) { - snippet.push_str(&insert); + // What the middleware write ahead of the bundle, such as + // a module's configuration, which is then in place when + // the bundle's own start-up code reads it. + for insert in &middleware_head_inserts { + snippet.push_str(insert); } // Main bundle: core + non-deferred integrations (synchronous). let immediate_parts = integrations.js_parts_immediate(); @@ -342,10 +642,10 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso &immediate_parts, &script_attributes, )); - // What an integration loads after the bundle and ahead of + // What a middleware loads after the bundle and ahead of // the page's own scripts in the origin head. - for insert in integrations.after_bundle_inserts(&ctx) { - snippet.push_str(&insert); + for insert in &middleware_after_bundle_inserts { + snippet.push_str(insert); } // Deferred bundles: large modules like prebid loaded after // HTML parsing completes. Empty when none are enabled. @@ -424,39 +724,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in href attributes element!("[href]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut href) = el.get_attribute("href") { - let original_href = href.clone(); - let element_name = el.tag_name(); - if let Some(rewritten) = patterns.rewrite_url_value(&href) { - href = rewritten; - } - - match integrations.rewrite_attribute( - "href", - &href, - &IntegrationAttributeContext { - attribute_name: "href", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_href) => { - href = integration_href; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if href != original_href { - el.set_attribute("href", &href)?; - } + if let Some(href) = el.get_attribute("href") + && let Some(rewritten) = patterns.rewrite_url_value(&href) + { + el.set_attribute("href", &rewritten)?; } Ok(()) } @@ -464,38 +736,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in src attributes element!("[src]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut src) = el.get_attribute("src") { - let original_src = src.clone(); - let element_name = el.tag_name(); - if let Some(rewritten) = patterns.rewrite_url_value(&src) { - src = rewritten; - } - match integrations.rewrite_attribute( - "src", - &src, - &IntegrationAttributeContext { - attribute_name: "src", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_src) => { - src = integration_src; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if src != original_src { - el.set_attribute("src", &src)?; - } + if let Some(src) = el.get_attribute("src") + && let Some(rewritten) = patterns.rewrite_url_value(&src) + { + el.set_attribute("src", &rewritten)?; } Ok(()) } @@ -503,39 +748,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in action attributes element!("[action]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut action) = el.get_attribute("action") { - let original_action = action.clone(); - let element_name = el.tag_name(); - if let Some(rewritten) = patterns.rewrite_url_value(&action) { - action = rewritten; - } - - match integrations.rewrite_attribute( - "action", - &action, - &IntegrationAttributeContext { - attribute_name: "action", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_action) => { - action = integration_action; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if action != original_action { - el.set_attribute("action", &action)?; - } + if let Some(action) = el.get_attribute("action") + && let Some(rewritten) = patterns.rewrite_url_value(&action) + { + el.set_attribute("action", &rewritten)?; } Ok(()) } @@ -543,46 +760,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in srcset attributes (for responsive images) element!("[srcset]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut srcset) = el.get_attribute("srcset") { - let original_srcset = srcset.clone(); - let element_name = el.tag_name(); - let new_srcset = srcset - .replace(&patterns.https_origin(), &patterns.replacement_url()) - .replace(&patterns.http_origin(), &patterns.replacement_url()) - .replace( - &patterns.protocol_relative_origin(), - &patterns.protocol_relative_replacement(), - ) - .replace(&patterns.origin_host, &patterns.request_host); - if new_srcset != srcset { - srcset = new_srcset; - } - - match integrations.rewrite_attribute( - "srcset", - &srcset, - &IntegrationAttributeContext { - attribute_name: "srcset", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_srcset) => { - srcset = integration_srcset; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if srcset != original_srcset { - el.set_attribute("srcset", &srcset)?; + if let Some(srcset) = el.get_attribute("srcset") { + let rewritten = replace_each_once(&srcset, &patterns.srcset_forms); + if rewritten != srcset { + el.set_attribute("srcset", &rewritten)?; } } Ok(()) @@ -591,45 +773,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in imagesrcset attributes (for link preload) element!("[imagesrcset]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut imagesrcset) = el.get_attribute("imagesrcset") { - let original_imagesrcset = imagesrcset.clone(); - let element_name = el.tag_name(); - let new_imagesrcset = imagesrcset - .replace(&patterns.https_origin(), &patterns.replacement_url()) - .replace(&patterns.http_origin(), &patterns.replacement_url()) - .replace( - &patterns.protocol_relative_origin(), - &patterns.protocol_relative_replacement(), - ); - if new_imagesrcset != imagesrcset { - imagesrcset = new_imagesrcset; - } - - match integrations.rewrite_attribute( - "imagesrcset", - &imagesrcset, - &IntegrationAttributeContext { - attribute_name: "imagesrcset", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_imagesrcset) => { - imagesrcset = integration_imagesrcset; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if imagesrcset != original_imagesrcset { - el.set_attribute("imagesrcset", &imagesrcset)?; + if let Some(imagesrcset) = el.get_attribute("imagesrcset") { + let rewritten = replace_each_once(&imagesrcset, &patterns.url_forms); + if rewritten != imagesrcset { + el.set_attribute("imagesrcset", &rewritten)?; } } Ok(()) @@ -668,38 +816,17 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso })); } - for script_rewriter in script_rewriters { - let selector = script_rewriter.selector(); - let rewriter = script_rewriter.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); - element_content_handlers.push(text!(selector, { - let rewriter = rewriter.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); - move |text| { - let ctx = IntegrationScriptContext { - selector, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - is_last_in_text_node: text.last_in_text_node(), - max_buffered_script_bytes: config.max_buffered_body_bytes, - document_state: &document_state, - }; - match rewriter.rewrite(text.as_str(), &ctx) { - ScriptRewriteAction::Keep => {} - ScriptRewriteAction::Replace(rewritten) => { - text.replace(&rewritten, ContentType::Text); - } - ScriptRewriteAction::RemoveNode => { - text.remove(); - } - } - Ok(()) - } - })); - } + // Shared by every text handler, so each is handed a chunk as the one + // before it left it. + let edits = Rc::new(TextEdits::default()); + + // Registered after core's own handlers, so an element handler judges an + // address as it will be served. + element_content_handlers.extend(middleware_content_handlers( + middleware_element_handlers, + middleware_text_handlers, + &edits, + )); let rewriter_settings = RewriterSettings { document_content_handlers, @@ -709,29 +836,296 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso let inner = HtmlRewriterAdapter::new(rewriter_settings); - let stream_context = crate::integrations::IntegrationHtmlStreamContext { - request_host: config.request_host.clone(), - request_scheme: config.request_scheme.clone(), - origin_host: config.origin_host.clone(), - document_state: document_state.clone(), - }; - let processors = stream_processor_factories - .into_iter() - .map(|factory| factory.create(stream_context.clone())) - .collect(); HtmlWithStreamingProcessors { inner: Box::new(inner), - processors, + processors: middleware_processors, + } +} + +/// Running a page through the document pipeline and comparing what comes out +/// with a recording, for a module crate's own tests. +/// +/// A recording is the page a reader receives, kept as a file beside the test. +/// It is how a change to the way a module makes its page change is shown to +/// leave the page as it was. +#[cfg(any(test, feature = "test-utils"))] +pub mod test_support { + use std::io::Cursor; + + use super::{HtmlProcessorConfig, create_html_processor_with_middleware}; + use crate::integrations::IntegrationRegistry; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase, PhaseEntries, PhaseEntry}; + use crate::settings::Settings; + use crate::streaming_processor::{ + Compression, PipelineConfig, StreamProcessor, StreamingPipeline, + }; + + /// The host a test page is fetched from. + pub const ORIGIN_HOST: &str = "origin.example.com"; + + /// The host a test page is served on, over HTTPS. + pub const REQUEST_HOST: &str = "publisher.example.com"; + + /// Set in the environment to write recordings in place of comparing with + /// them. + pub const RECORD_ENV: &str = "TS_RECORD_PAGES"; + + /// What stands in a recording for the hash of the script bundle, which + /// changes whenever any browser module does. + const BUNDLE_HASH: &str = "BUNDLE-HASH"; + + /// The path a test page is asked for at, which decides the entry of the + /// settings that covers it. + pub const REQUEST_PATH: &str = "/"; + + /// Has `settings` run `names` on every page in `phase`, in the order + /// given, in place of any entries that phase held. + pub fn place_on_every_page(settings: &mut Settings, phase: MiddlewarePhase, names: &[&str]) { + let entries = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + }]); + match phase { + MiddlewarePhase::Fetch => settings.fetch = entries, + MiddlewarePhase::Serve => settings.serve = entries, + } + } + + /// The processor of a page asked for at [`REQUEST_PATH`] from a + /// deployment running `settings` with `registry`'s modules, built from + /// `config`, on which a test sets what it is about. The page is not + /// stored, so both phases' middleware run on the one pass. + /// + /// # Panics + /// + /// When a middleware asks for a selector that does not parse. + #[must_use] + pub fn create_page_processor( + settings: &Settings, + registry: &IntegrationRegistry, + config: HtmlProcessorConfig, + ) -> impl StreamProcessor + use<> { + let fetch = registry.middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + REQUEST_PATH, + ); + let serve = registry.middleware_chain( + &settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + REQUEST_PATH, + ); + create_html_processor_with_middleware(config, &fetch, Some(&serve)) + .expect("should plan the page's middleware") + } + + /// The page a reader of `html` receives from a deployment running + /// `settings` with `registry`'s modules, read from the origin `chunk_size` + /// bytes at a time and asked for at [`REQUEST_PATH`]. The page is not + /// stored, so both phases' middleware run on the one pass. + /// + /// The script bundle's `?v=` hash is replaced by a fixed word, so a + /// recording changes when a module's page change does and not when a + /// browser module is rebuilt. + /// + /// # Panics + /// + /// When the pipeline refuses the page, or leaves something that is not + /// UTF-8. + #[must_use] + pub fn processed_page( + settings: &Settings, + registry: &IntegrationRegistry, + html: &str, + chunk_size: usize, + ) -> String { + let config = HtmlProcessorConfig::from_settings( + settings, + registry, + ORIGIN_HOST, + REQUEST_HOST, + "https", + ); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + create_page_processor(settings, registry, config), + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should process the page"); + without_bundle_hashes(&String::from_utf8(output).expect("should leave the page UTF-8")) + } + + /// `page` with each `?v=` hash of a script bundle address replaced. + fn without_bundle_hashes(page: &str) -> String { + const MARK: &str = "/static/tsjs="; + let mut out = String::with_capacity(page.len()); + let mut rest = page; + while let Some(at) = rest.find(MARK) { + let (before, address) = rest.split_at(at + MARK.len()); + out.push_str(before); + let end = address + .find(['"', '\'', ' ', '>', '<']) + .unwrap_or(address.len()); + let (address, after) = address.split_at(end); + match address.split_once("?v=") { + Some((file, hash)) + if !hash.is_empty() && hash.bytes().all(|b| b.is_ascii_hexdigit()) => + { + out.push_str(file); + out.push_str("?v="); + out.push_str(BUNDLE_HASH); + } + _ => out.push_str(address), + } + rest = after; + } + out.push_str(rest); + out + } + + /// The least a deployment's settings must say, which a module's own + /// settings are written after. + const BASE_SETTINGS: &str = r#" +[[handlers]] +path = "^/_ts/admin" +username = "admin" +password = "page-recording-password" + +[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +origin_url = "https://origin.example.com" +proxy_secret = "page-recording-proxy-secret" + +[geo] +assume_single_jurisdiction = true + +[ec] +module = "hmac" + +[ec.hmac] +passphrase = "page-recording-passphrase-32-bytes" +"#; + + /// The settings of a deployment that writes `module_settings`, a TOML + /// document selecting a module and holding its table, after the least a + /// deployment must say. + /// + /// # Panics + /// + /// When the two together are not settings a deployment could load. + #[must_use] + pub fn page_settings(module_settings: &str) -> Settings { + Settings::from_toml(&format!("{BASE_SETTINGS}\n{module_settings}")) + .expect("should read the page's settings") + } + + /// The size of the chunks a recorded page is read from the origin in, + /// large enough that a test page arrives whole. + const RECORDING_CHUNK_SIZE: usize = 8192; + + /// Checks the page `html` becomes is the one in `recorded`, byte for + /// byte. + /// + /// The deployment writes `module_settings`, see [`page_settings`], and + /// links `builders` beside core's own modules. `recorded` is the recording + /// as the test embeds it and `path` is where it is kept. With + /// [`RECORD_ENV`] set the recording is written to `path` instead, which is + /// how a recording is made. A recording is made again only when the page + /// is meant to change for every reader. + /// + /// # Panics + /// + /// When the page is not the recorded one, or the recording cannot be + /// written. + pub fn assert_page_is_recorded( + module_settings: &str, + builders: &[crate::integrations::IntegrationBuilder], + html: &str, + recorded: &str, + path: &str, + ) { + let settings = &page_settings(module_settings); + let registry = &IntegrationRegistry::with_registrations(settings, builders) + .expect("should build the registry"); + let page = processed_page(settings, registry, html, RECORDING_CHUNK_SIZE); + #[cfg(not(target_arch = "wasm32"))] + if std::env::var_os(RECORD_ENV).is_some() { + std::fs::write(path, &page).expect("should write the recording"); + return; + } + assert!( + recorded == page, + "the page is not the one recorded at {path}.\n--- recorded\n{recorded}\n--- now\n{page}\n\ + ---\nA recording changes only when the page is meant to change for every reader, \ + by running the test with {RECORD_ENV} set." + ); + } + + #[cfg(test)] + mod tests { + use super::{place_on_every_page, processed_page, without_bundle_hashes}; + use crate::integrations::IntegrationRegistry; + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::MiddlewarePhase; + + #[test] + fn a_test_page_runs_the_middleware_placed_on_every_page() { + let mut settings = crate::test_support::tests::create_test_settings(); + settings.select_module("testing", fixture::MODULE); + let html = ""; + + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let unplaced = processed_page(&settings, ®istry, html, 8192); + + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[fixture::HEAD]); + place_on_every_page(&mut settings, MiddlewarePhase::Serve, &[fixture::READER]); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let placed = processed_page(&settings, ®istry, html, 8192); + + assert!( + !unplaced.contains("middleware-fixture"), + "should run no middleware an entry does not name: {unplaced}" + ); + assert!( + placed.contains(&fixture::head_marker("")) + && placed.contains("middleware-fixture-reader"), + "should run the middleware of both phases on a test page: {placed}" + ); + } + + #[test] + fn a_bundle_hash_is_replaced_and_nothing_else_is() { + let page = "\ + \ + kept\ + kept"; + + let replaced = without_bundle_hashes(page); + + assert_eq!( + replaced, + "\ + \ + kept\ + kept" + ); + } } } #[cfg(test)] mod tests { use super::*; - use crate::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationHeadInjector, IntegrationHtmlContext, - }; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use crate::test_support::tests::create_test_settings; use serde_json::json; @@ -760,137 +1154,12 @@ mod tests { } #[test] - fn integration_attribute_rewriter_can_remove_elements() { - struct RemovingLinkRewriter; - - impl IntegrationAttributeRewriter for RemovingLinkRewriter { - fn integration_id(&self) -> &'static str { - "removing" - } - - fn handles_attribute(&self, attribute: &str) -> bool { - attribute == "href" - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if attr_value.contains("remove-me") { - AttributeRewriteAction::remove_element() - } else { - AttributeRewriteAction::keep() - } - } - } - - let html = r#" - remove - keep - "#; - - let mut config = create_test_config(); - config.integrations = - IntegrationRegistry::from_rewriters(vec![Arc::new(RemovingLinkRewriter)], Vec::new()); - - let processor = create_html_processor(config); - let pipeline_config = PipelineConfig { - input_compression: Compression::None, - output_compression: Compression::None, - chunk_size: 8192, - }; - let mut pipeline = StreamingPipeline::new(pipeline_config, processor); - - let mut output = Vec::new(); - pipeline - .process(Cursor::new(html.as_bytes()), &mut output) - .expect("pipeline should process HTML"); - let processed = String::from_utf8(output).expect("output should be valid UTF-8"); - - assert!(processed.contains("keep-me")); - assert!(!processed.contains("remove-me")); - } - - #[test] - fn integration_head_injector_prepends_after_tsjs_once() { - struct TestHeadInjector; - - impl IntegrationHeadInjector for TestHeadInjector { - fn integration_id(&self) -> &'static str { - "test" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - vec!["".to_owned()] - } - } - - let html = "Test"; - - let mut config = create_test_config(); - config.integrations = IntegrationRegistry::from_rewriters_with_head_injectors( - Vec::new(), - Vec::new(), - vec![Arc::new(TestHeadInjector)], - ); - - let processor = create_html_processor(config); - let pipeline_config = PipelineConfig { - input_compression: Compression::None, - output_compression: Compression::None, - chunk_size: 8192, - }; - let mut pipeline = StreamingPipeline::new(pipeline_config, processor); - - let mut output = Vec::new(); - pipeline - .process(Cursor::new(html.as_bytes()), &mut output) - .expect("pipeline should process HTML"); - let processed = String::from_utf8(output).expect("output should be valid UTF-8"); - - let tsjs_marker = "id=\"trustedserver-js\""; - let head_marker = "window.__testHeadInjector=true"; - - assert_eq!( - processed.matches(tsjs_marker).count(), - 1, - "should inject unified tsjs tag once" - ); - assert_eq!( - processed.matches(head_marker).count(), - 1, - "should inject head snippet once" - ); - - let tsjs_index = processed - .find(tsjs_marker) - .expect("should include unified tsjs tag"); - let head_index = processed - .find(head_marker) - .expect("should include head snippet"); - let title_index = processed - .find("") - .expect("should keep existing head content"); - - assert!( - head_index < tsjs_index, - "should inject config before tsjs bundle so auto-init can read it" - ); - assert!( - tsjs_index < title_index, - "should prepend all injected content before existing head content" - ); - } - - #[test] - fn integration_head_injector_marks_only_the_bundle_it_asks_to_mark() { + fn a_registration_marks_only_the_bundle_it_asks_to_mark() { use crate::integrations::registry_test_support::tag_fixture as tag; fn process(mark_bundle: Option<bool>) -> String { - let integrations = if let Some(mark_bundle) = mark_bundle { - let mut settings = create_test_settings(); + let mut settings = create_test_settings(); + if let Some(mark_bundle) = mark_bundle { settings .insert_module_config( "testing", @@ -900,13 +1169,16 @@ mod tests { }), ) .expect("should insert the stand-in's settings"); - IntegrationRegistry::new(&settings).expect("should build the registry") - } else { - IntegrationRegistry::empty_for_tests() - }; + test_support::place_on_every_page( + &mut settings, + crate::middleware::MiddlewarePhase::Fetch, + &[tag::MODULE], + ); + } + let registry = IntegrationRegistry::new(&settings).expect("should build the registry"); let mut config = create_test_config(); - config.integrations = integrations; - let mut processor = create_html_processor(config); + config.integrations = registry.clone(); + let mut processor = test_support::create_page_processor(&settings, ®istry, config); let output = processor .process_chunk(b"<html><head></head><body></body></html>", true) .expect("should process HTML"); @@ -928,11 +1200,11 @@ mod tests { let attribute = format!("{}=\"true\"", tag::BUNDLE_ATTRIBUTE); assert!( marked.contains(&attribute), - "should mark the bundle an injector asks to mark" + "should mark the bundle a registration asks to mark" ); assert!( !unmarked.contains(tag::BUNDLE_ATTRIBUTE), - "should leave the bundle unmarked when the injector asks for nothing" + "should leave the bundle unmarked when the registration asks for nothing" ); assert!( !without_the_integration.contains(tag::BUNDLE_ATTRIBUTE), @@ -947,7 +1219,7 @@ mod tests { .expect("should include the publisher bundle"); assert!( head_insert_index < publisher_bundle_index, - "should keep integration head inserts before the publisher bundle" + "should keep a middleware's head markup before the publisher bundle" ); } @@ -957,13 +1229,14 @@ mod tests { let html = "<html><head><title>Test"; let mut settings = create_test_settings(); - settings.select_module("testing", request_fixture::MODULE); - let mut config = create_test_config(); - config.integrations = + request_fixture::select_and_place(&mut settings); + let registry = IntegrationRegistry::new(&settings).expect("should build integration registry"); + let mut config = create_test_config(); + config.integrations = registry.clone(); config.request_state = request_fixture::marked(); - let processor = create_html_processor(config); + let processor = test_support::create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -1053,6 +1326,146 @@ mod tests { assert!(!result.contains("origin.example.com")); } + #[test] + fn the_origin_s_address_is_moved_in_srcset_and_imagesrcset() { + let mut processor = create_html_processor(create_test_config()); + + let output = processor + .process_chunk( + br#""#, + true, + ) + .expect("should process the document"); + let processed = String::from_utf8(output).expect("should stay UTF-8"); + + assert!( + processed.contains( + r#"imagesrcset="https://test.example.com/a.png 1x, https://test.example.com/b.png 2x, //test.example.com/c.png 3x""# + ), + "should move the origin's address in every candidate of an imagesrcset: {processed}" + ); + assert!( + processed.contains( + r#"srcset="https://test.example.com/a.png 1x, https://test.example.com/b.png 2x, //test.example.com/c.png 3x, test.example.com/d.png 4x""# + ), + "should move the origin's address in every candidate of a srcset, a bare host \ + included: {processed}" + ); + assert!( + processed.contains(r#""#), + "should leave a srcset that names another host as it is: {processed}" + ); + } + + /// A request host that begins with the origin host, as a local host named + /// after the publisher does, is written once in every address over either + /// scheme, rather than matched again inside its own replacement. + #[test] + fn a_request_host_that_begins_with_the_origin_host_is_written_once() { + for scheme in ["http", "https"] { + let mut config = create_test_config(); + config.origin_host = "publisher.example".to_owned(); + config.request_host = "publisher.example.localhost".to_owned(); + config.request_scheme = scheme.to_owned(); + let mut processor = create_html_processor(config); + + let output = processor + .process_chunk( + br#"nob
"#, + true, + ) + .expect("should process the document"); + let processed = String::from_utf8(output).expect("should stay UTF-8"); + + assert!( + !processed.contains(".localhost.localhost"), + "should write the request host once over {scheme}: {processed}" + ); + for expected in [ + format!(r#"href="{scheme}://publisher.example.localhost/news""#), + format!(r#"href="{scheme}://publisher.example.localhost/old""#), + r#"href="publisher.example.localhost/bare""#.to_owned(), + r#"src="//publisher.example.localhost/c.png""#.to_owned(), + format!( + r#"srcset="{scheme}://publisher.example.localhost/d.png 1x, publisher.example.localhost/e.png 2x""# + ), + format!(r#"action="{scheme}://publisher.example.localhost/search""#), + format!( + r#"imagesrcset="{scheme}://publisher.example.localhost/a.png 1x, //publisher.example.localhost/b.png 2x""# + ), + ] { + assert!( + processed.contains(&expected), + "should contain {expected} over {scheme}: {processed}" + ); + } + } + } + + /// Hosts that do not begin with the origin host are written as before, + /// both one with no part of the origin host in it and a `www.` host in + /// front of an apex origin, whose `srcset` must not become `www.www.`. + #[test] + fn a_request_host_that_does_not_begin_with_the_origin_host_is_written_as_before() { + for (origin, request) in [ + ("publisher.example", "publisher-example.localhost"), + ("example.com", "www.example.com"), + ] { + let mut config = create_test_config(); + config.origin_host = origin.to_owned(); + config.request_host = request.to_owned(); + config.request_scheme = "https".to_owned(); + let mut processor = create_html_processor(config); + + let input = format!( + r#"ab"# + ); + let output = processor + .process_chunk(input.as_bytes(), true) + .expect("should process the document"); + let processed = String::from_utf8(output).expect("should stay UTF-8"); + + for expected in [ + format!(r#"href="https://{request}/a""#), + format!(r#"href="https://{request}/b""#), + format!(r#"src="//{request}/c.png""#), + format!(r#"srcset="https://{request}/d.png 1x, {request}/e.png 2x""#), + ] { + assert!( + processed.contains(&expected), + "should contain {expected} for {origin} served as {request}: {processed}" + ); + } + } + } + + #[test] + fn each_form_is_replaced_once_from_the_start() { + let forms = vec![ + ( + "https://a.example".to_owned(), + "http://a.example.localhost".to_owned(), + ), + ( + "http://a.example".to_owned(), + "http://a.example.localhost".to_owned(), + ), + ("//a.example".to_owned(), "//a.example.localhost".to_owned()), + ("a.example".to_owned(), "a.example.localhost".to_owned()), + (String::new(), "never".to_owned()), + ]; + + assert_eq!( + replace_each_once( + "https://a.example/1 http://a.example/2 //a.example/3 a.example/4", + &forms + ), + "http://a.example.localhost/1 http://a.example.localhost/2 //a.example.localhost/3 a.example.localhost/4" + ); + assert_eq!(replace_each_once("nothing here", &forms), "nothing here"); + assert_eq!(replace_each_once("", &forms), ""); + } + #[test] fn test_html_processor_config_from_settings() { let settings = create_test_settings(); diff --git a/crates/trusted-server-core/src/html_processor/script_text.rs b/crates/trusted-server-core/src/html_processor/script_text.rs new file mode 100644 index 0000000000..507673ad5c --- /dev/null +++ b/crates/trusted-server-core/src/html_processor/script_text.rs @@ -0,0 +1,357 @@ +//! Script text a middleware writes, kept inside its `')", "a('\\u003c/script>')"), + ("a('')", "a('\\u003c/SCRIPT >')"), + ("a('')", "a('", + "a('')"], true), + "a('"; + assert_eq!(written(&[legacy], false), legacy); + let chunks = [ + "", + ]; + assert_eq!(written(&chunks, false), chunks.concat()); + } +} diff --git a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.input.html b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.input.html new file mode 100644 index 0000000000..14c7aa4842 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.input.html @@ -0,0 +1,17 @@ + + + + +JS asset proxy + + + + + + + +An article on the origin + +A link to the same address + + diff --git a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.recorded.html b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.recorded.html new file mode 100644 index 0000000000..4168885df0 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.recorded.html @@ -0,0 +1,17 @@ + + + + +JS asset proxy + + + + + + + +An article on the origin + +A link to the same address + + diff --git a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml new file mode 100644 index 0000000000..2cedf868f5 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml @@ -0,0 +1,13 @@ +[proxy] +modules = ["js_asset_proxy"] + +[proxy.js_asset_proxy] + +[[proxy.js_asset_proxy.assets]] +path = "/assets/example-vendor-loader.js" +origin_url = "https://cdn.example.com/vendor-loader.js" +proxy = "enabled" + +[[fetch]] +media_type = "text/html" +middleware = ["js_asset_proxy"] diff --git a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs index fb80c60b20..7a650f1648 100644 --- a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs +++ b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs @@ -6,6 +6,7 @@ //! proxying, or block matching script tags from publisher HTML. use std::collections::HashSet; +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -22,8 +23,11 @@ use crate::constants::{ }; use crate::error::TrustedServerError; use crate::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, + AttributeRewriteAction, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, +}; +use crate::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use crate::platform::RuntimeServices; use crate::proxy::{ProxyRequestConfig, proxy_request}; @@ -278,14 +282,32 @@ impl JsAssetProxyIntegration { fn asset_for_script_src( &self, script_src: &str, - ctx: &IntegrationAttributeContext<'_>, + request_scheme: &str, ) -> Option<&JsAssetProxyAsset> { self.asset_for_origin_url(script_src).or_else(|| { - let normalized_src = normalize_script_src(script_src, ctx.request_scheme)?; + let normalized_src = normalize_script_src(script_src, request_scheme)?; self.asset_for_origin_url(&normalized_src) }) } + /// What happens to a script whose address is `script_src`, on a page + /// served over `request_scheme`. + fn rewrite_script_address( + &self, + script_src: &str, + request_scheme: &str, + ) -> AttributeRewriteAction { + let Some(asset) = self.asset_for_script_src(script_src, request_scheme) else { + return AttributeRewriteAction::keep(); + }; + + match asset.proxy { + JsAssetProxyMode::Enabled => AttributeRewriteAction::replace(asset.path.clone()), + JsAssetProxyMode::Disabled => AttributeRewriteAction::keep(), + JsAssetProxyMode::Blocked => AttributeRewriteAction::remove_element(), + } + } + fn build_proxy_config<'a>( origin_url: &'a str, req: &Request, @@ -492,7 +514,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(JS_ASSET_PROXY_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(ScriptAddress(integration))) .build(), )) } @@ -564,33 +586,36 @@ impl IntegrationProxy for JsAssetProxyIntegration { } } -impl IntegrationAttributeRewriter for JsAssetProxyIntegration { - fn integration_id(&self) -> &'static str { - JS_ASSET_PROXY_INTEGRATION_ID +/// Points a configured script at its first-party path, or removes one that +/// is blocked, on the pages a `[[fetch]]` entry names [`MODULE`] for. +/// +/// It is asked about a script's address as the middleware named before it +/// left it, so it is named first where it has to see the address the page +/// wrote. +struct ScriptAddress(Arc); + +impl Middleware for ScriptAddress { + fn middleware_id(&self) -> &'static str { + MODULE } - fn handles_attribute(&self, attribute: &str) -> bool { - attribute == "src" + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - attr_name: &str, - attr_value: &str, - ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if attr_name != "src" || !ctx.element_name.eq_ignore_ascii_case("script") { - return AttributeRewriteAction::keep(); - } - - let Some(asset) = self.asset_for_script_src(attr_value, ctx) else { - return AttributeRewriteAction::keep(); - }; - - match asset.proxy { - JsAssetProxyMode::Enabled => AttributeRewriteAction::replace(asset.path.clone()), - JsAssetProxyMode::Disabled => AttributeRewriteAction::keep(), - JsAssetProxyMode::Blocked => AttributeRewriteAction::remove_element(), + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + let integration = Arc::clone(&self.0); + let request_scheme = context.request_scheme.to_owned(); + let decide: Rc = Rc::new(move |matched| { + integration.rewrite_script_address(matched.value, &request_scheme) + }); + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + "script[src]", + "src", + decide, + ))], + ..MiddlewareAction::pass() } } } @@ -602,10 +627,13 @@ mod tests { use std::sync::Arc; use crate::constants::{HEADER_REFERER, HEADER_X_FORWARDED_FOR, HEADER_X_TS_EC}; - use crate::html_processor::{BodyCloseInjection, HtmlProcessorConfig, create_html_processor}; - use crate::integrations::{ - AttributeRewriteAction, IntegrationAttributeRewriter, IntegrationRegistry, + use crate::html_processor::test_support::{place_on_every_page, processed_page}; + use crate::html_processor::{ + BodyCloseInjection, HtmlProcessorConfig, create_html_processor_with_middleware, }; + use crate::integrations::registry_test_support::tag_fixture as tag; + use crate::integrations::{AttributeRewriteAction, IntegrationRegistry}; + use crate::middleware::MiddlewareChain; use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use crate::test_support::tests::create_test_settings; @@ -636,41 +664,31 @@ mod tests { } } - fn rewrite_context() -> IntegrationAttributeContext<'static> { - IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - } - } - + /// The page a reader of `html` receives when `integration`'s middleware + /// alone runs on it. fn process_html_with_integration( html: &str, integration: Arc, ) -> String { - let rewriter: Arc = integration; - process_html_with_registry( - html, - IntegrationRegistry::from_rewriters(vec![rewriter], Vec::new()), - ) - } - - fn process_html_with_registry(html: &str, integrations: IntegrationRegistry) -> String { - let processor = create_html_processor(HtmlProcessorConfig { + let chain = MiddlewareChain::new( + MiddlewarePhase::Fetch, + vec![Arc::new(ScriptAddress(integration))], + ); + let config = HtmlProcessorConfig { csp_nonce_observed: None, body_close: BodyCloseInjection::None, origin_host: "origin.example.com".to_string(), request_host: "publisher.example.com".to_string(), request_scheme: "https".to_string(), - integrations, + integrations: IntegrationRegistry::default(), ad_slots_script: None, ad_bids_state: Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, request_state: crate::integrations::IntegrationRequestState::default(), permissions_script: None, - }); + }; + let processor = create_html_processor_with_middleware(config, &chain, None) + .expect("should plan the asset proxy's middleware"); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -685,6 +703,35 @@ mod tests { String::from_utf8(output).expect("should produce UTF-8 HTML") } + /// The page a reader of `html` receives from a deployment running + /// `settings`. + fn process_html_with_settings(html: &str, settings: &Settings) -> String { + let registry = IntegrationRegistry::new(settings).expect("should build registry"); + processed_page(settings, ®istry, html, 8192) + } + + /// Settings that run the tag stand-in and proxy its script in `mode`, + /// with the two middleware named in the order given. + fn settings_proxying_the_tag(mode: &str, middleware: &[&str]) -> Settings { + let mut settings = create_test_settings(); + settings.select_module("testing", tag::MODULE); + settings + .insert_module_config( + "proxy", + "js_asset_proxy", + &json!({ + "assets": [{ + "path": "/assets/sdk.js", + "origin_url": tag::SCRIPT_URL, + "proxy": mode + }] + }), + ) + .expect("should insert JS asset proxy config"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, middleware); + settings + } + #[test] fn an_integration_that_is_not_named_registers_no_routes() { let settings = create_test_settings(); @@ -739,17 +786,16 @@ mod tests { assert_eq!(routes[0].method, Method::GET); assert_eq!(routes[0].path, "/assets/enabled.js"); - let ctx = rewrite_context(); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/enabled.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/enabled.js", "https"), AttributeRewriteAction::Replace(ref value) if value == "/assets/enabled.js" )); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/disabled.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/disabled.js", "https"), AttributeRewriteAction::Keep )); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/blocked.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/blocked.js", "https"), AttributeRewriteAction::RemoveElement )); } @@ -761,46 +807,13 @@ mod tests { "https://cdn.example.com/vendor.js", JsAssetProxyMode::Enabled, )])); - let ctx = rewrite_context(); - - assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/vendor.js?v=1", &ctx), - AttributeRewriteAction::Keep - )); - assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/other.js", &ctx), - AttributeRewriteAction::Keep - )); - } - - #[test] - fn non_script_src_matches_are_not_rewritten_or_blocked() { - let integration = JsAssetProxyIntegration::new(config_with_assets(vec![ - asset( - "/assets/enabled.js", - "https://cdn.example.com/enabled.js", - JsAssetProxyMode::Enabled, - ), - asset( - "/assets/blocked.js", - "https://cdn.example.com/blocked.js", - JsAssetProxyMode::Blocked, - ), - ])); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "img", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/enabled.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/vendor.js?v=1", "https"), AttributeRewriteAction::Keep )); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/blocked.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/other.js", "https"), AttributeRewriteAction::Keep )); } @@ -834,6 +847,25 @@ mod tests { assert!(processed.contains(r#""#)); } + #[test] + fn a_script_tag_written_in_upper_case_is_a_script() { + let integration = JsAssetProxyIntegration::new(config_with_assets(vec![asset( + "/assets/enabled.js", + "https://cdn.example.com/enabled.js", + JsAssetProxyMode::Enabled, + )])); + + let processed = process_html_with_integration( + r#""#, + integration, + ); + + assert!( + processed.contains(r#"SRC="/assets/enabled.js""#), + "should rewrite a script whatever case its tag is written in: {processed}" + ); + } + #[test] fn script_src_matching_normalizes_common_browser_url_forms() { let integration = JsAssetProxyIntegration::new(config_with_assets(vec![asset( @@ -841,7 +873,6 @@ mod tests { "https://cdn.example.com/vendor.js", JsAssetProxyMode::Enabled, )])); - let ctx = rewrite_context(); for script_src in [ "//cdn.example.com/vendor.js", @@ -850,7 +881,7 @@ mod tests { ] { assert!( matches!( - integration.rewrite("src", script_src, &ctx), + integration.rewrite_script_address(script_src, "https"), AttributeRewriteAction::Replace(ref value) if value == "/assets/vendor.js" ), "script src {script_src} should normalize to the configured origin URL" @@ -859,119 +890,84 @@ mod tests { } #[test] - fn js_asset_proxy_rewriter_takes_precedence_over_native_rewriters() { - let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::tag_fixture::MODULE, - ); - settings - .insert_module_config( - "proxy", - "js_asset_proxy", - &json!({ - "assets": [{ - "path": "/assets/sdk.js", - "origin_url": crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL, - "proxy": "enabled" - }] - }), - ) - .expect("should insert JS asset proxy config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); + fn the_asset_proxy_named_first_rewrites_a_script_ahead_of_a_module() { + let settings = settings_proxying_the_tag("enabled", &[MODULE, tag::MODULE]); let html = format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL + tag::SCRIPT_URL ); - let processed = process_html_with_registry(&html, registry); + let processed = process_html_with_settings(&html, &settings); assert!( processed.contains(r#""#), - "JS asset proxy should rewrite before the integration's own rewriter: {processed}" + "JS asset proxy should rewrite before the module named after it: {processed}" ); assert!( - !processed.contains( - crate::integrations::registry_test_support::tag_fixture::FIRST_PARTY_SCRIPT - ), - "the integration's own rewrite should not override JS asset proxy" + !processed.contains(tag::FIRST_PARTY_SCRIPT), + "the module's own rewrite should not override JS asset proxy" ); } #[test] - fn js_asset_proxy_blocking_takes_precedence_over_native_rewriters() { - let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::tag_fixture::MODULE, + fn a_module_named_ahead_of_the_asset_proxy_rewrites_the_script_first() { + let settings = settings_proxying_the_tag("enabled", &[tag::MODULE, MODULE]); + let html = format!( + r#""#, + tag::SCRIPT_URL ); - settings - .insert_module_config( - "proxy", - "js_asset_proxy", - &json!({ - "assets": [{ - "path": "/assets/sdk.js", - "origin_url": crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL, - "proxy": "blocked" - }] - }), - ) - .expect("should insert JS asset proxy config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); + + let processed = process_html_with_settings(&html, &settings); + + assert!( + processed.contains(&format!( + r#""#, + tag::FIRST_PARTY_SCRIPT + )), + "the module named first should rewrite the script: {processed}" + ); + assert!( + !processed.contains("/assets/sdk.js"), + "JS asset proxy should not know an address the module has already moved" + ); + } + + #[test] + fn the_asset_proxy_named_first_blocks_a_script_ahead_of_a_module() { + let settings = settings_proxying_the_tag("blocked", &[MODULE, tag::MODULE]); let html = format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL + tag::SCRIPT_URL ); - let processed = process_html_with_registry(&html, registry); + let processed = process_html_with_settings(&html, &settings); assert!( !processed.contains("vendor.cmd"), - "blocked JS asset should remove the script element before the integration can rewrite it" + "blocked JS asset should remove the script element before the module can rewrite it" ); assert!( - !processed.contains( - crate::integrations::registry_test_support::tag_fixture::FIRST_PARTY_SCRIPT - ), - "the integration's own rewrite should not keep a blocked script" + !processed.contains(tag::FIRST_PARTY_SCRIPT), + "the module's own rewrite should not keep a blocked script" ); } #[test] fn disabled_js_asset_proxy_candidate_allows_native_rewriters() { - let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::tag_fixture::MODULE, - ); - settings - .insert_module_config( - "proxy", - "js_asset_proxy", - &json!({ - "assets": [{ - "path": "/assets/sdk.js", - "origin_url": crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL, - "proxy": "disabled" - }] - }), - ) - .expect("should insert JS asset proxy config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); + let settings = settings_proxying_the_tag("disabled", &[MODULE, tag::MODULE]); let html = format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL + tag::SCRIPT_URL ); - let processed = process_html_with_registry(&html, registry); + let processed = process_html_with_settings(&html, &settings); assert!( processed.contains(&format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::FIRST_PARTY_SCRIPT + tag::FIRST_PARTY_SCRIPT )), - "disabled JS asset proxy entries should not suppress an integration's own rewrite" + "disabled JS asset proxy entries should not suppress a module's own rewrite" ); } @@ -1320,10 +1316,10 @@ mod tests { }), ) .expect("should insert integration config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); - let processed = process_html_with_registry( + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); + let processed = process_html_with_settings( r#""#, - registry, + &settings, ); assert!(processed.contains(r#""#)); @@ -1694,4 +1690,21 @@ mod tests { "should identify the unknown asset field" ); } + + /// The page a reader receives with the script proxy running, kept as a + /// file so that changing how the page change is made can be shown to + /// leave the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + crate::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/js-asset-proxy.settings.toml"), + &[], + include_str!("fixtures/js-asset-proxy.input.html"), + include_str!("fixtures/js-asset-proxy.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/integrations/fixtures/js-asset-proxy.recorded.html" + ), + ); + } } diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index f12f4671b5..c100a74336 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -20,15 +20,12 @@ mod registry; #[cfg(test)] pub(crate) use registry::test_support as registry_test_support; pub use registry::{ - AttributeRewriteAction, AttributeRewriteOutcome, CarriedJsModule, HeaderMutation, - HeaderMutationMode, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationDocumentState, IntegrationEndpoint, IntegrationHeadInjector, IntegrationHtmlContext, - IntegrationHtmlStreamContext, IntegrationHtmlStreamProcessorFactory, IntegrationMetadata, - IntegrationProxy, IntegrationRegistration, IntegrationRegistrationBuilder, IntegrationRegistry, - IntegrationRequestFilter, IntegrationRequestState, IntegrationScriptContext, - IntegrationScriptRewriter, ProxyDispatchInput, RequestFilterDecision, RequestFilterEffects, - RequestFilterInput, RequestFilterRegistryInput, RequestFilterRegistryOutcome, - ScriptRewriteAction, ScriptTextAccumulator, + AttributeRewriteAction, CarriedJsModule, HeaderMutation, HeaderMutationMode, + IntegrationDocumentState, IntegrationEndpoint, IntegrationMetadata, IntegrationProxy, + IntegrationRegistration, IntegrationRegistrationBuilder, IntegrationRegistry, + IntegrationRequestFilter, IntegrationRequestState, ProxyDispatchInput, RequestFilterDecision, + RequestFilterEffects, RequestFilterInput, RequestFilterRegistryInput, + RequestFilterRegistryOutcome, ScriptRewriteAction, }; /// Registers or retrieves a platform backend for the given URL. @@ -697,10 +694,10 @@ impl IntegrationBuilder { } } -/// The built-in integrations, in hook order. +/// The built-in integrations, in the order `trusted-server-modules` lists them. const BUILT_IN_BUILDERS: &[IntegrationBuilder] = &[ - // This must remain the first module a section selects: attribute - // rewriters chain replacements and short-circuit removals. + // This stays the first module a section selects, so its middleware is + // the first of theirs in the order the registry lists them. js_asset_proxy::BUILDER, // A stand-in for an integration that streams, which core's own tests // select where they need one. @@ -717,6 +714,10 @@ const BUILT_IN_BUILDERS: &[IntegrationBuilder] = &[ // the same tests. #[cfg(test)] registry_test_support::deferred_fixture::BUILDER, + // A stand-in for a module that changes a page through middleware, for + // the same tests. + #[cfg(test)] + registry_test_support::middleware_fixture::BUILDER, // A stand-in for the plainest demand implementation there can be, which // core's own tests name where they need a source. #[cfg(test)] @@ -743,13 +744,14 @@ const BUILT_IN_BUILDERS: &[IntegrationBuilder] = &[ .with_adserver(&crate::auction::test_support::adserver_fixture::ADSERVER), ]; -/// The built-in integration builders, in hook order. +/// The built-in integration builders, in the order `trusted-server-modules` +/// lists them. pub(crate) fn builders() -> &'static [IntegrationBuilder] { BUILT_IN_BUILDERS } /// Every builder the registry will consider: the built-in set followed by -/// `extra`, in that order, so hook order for the built-ins never changes. +/// `extra`, in that order, so the built-ins keep their order. pub(crate) fn all_builders( extra: &[IntegrationBuilder], ) -> impl Iterator + '_ { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 88d949f5da..f904b9fc35 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -1,6 +1,6 @@ use std::any::{Any, TypeId}; use std::collections::BTreeMap; -use std::sync::{Arc, Mutex, MutexGuard, PoisonError}; +use std::sync::{Arc, Mutex}; use async_trait::async_trait; use edgezero_core::body::Body as EdgeBody; @@ -18,12 +18,13 @@ use crate::ec::module::{EcModuleSelection, EdgeCookieModule}; use crate::error::TrustedServerError; use crate::geo::GeoInfo; use crate::http_util::is_navigation_request; +use crate::middleware::{Middleware, MiddlewareChain, MiddlewarePhase, PhaseEntries}; use crate::module_context::{ModuleCall, ModuleContext, ResolvedRequest}; use crate::platform::{DisabledGeo, PlatformGeo, RuntimeServices}; use crate::settings::Settings; -use crate::streaming_processor::StreamProcessor; -/// Action returned by attribute rewriters to describe how the runtime should mutate the element. +/// What a middleware's element handler decides about one attribute of an +/// element. #[derive(Debug, Clone, PartialEq, Eq)] pub enum AttributeRewriteAction { /// Leave the attribute and element untouched. @@ -51,15 +52,8 @@ impl AttributeRewriteAction { } } -/// Outcome returned by the registry after running every matching attribute rewriter. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum AttributeRewriteOutcome { - Unchanged, - Replaced(String), - RemoveElement, -} - -/// Action returned by inline script rewriters to describe how to mutate the node. +/// What a middleware's text handler decides about one chunk of the text +/// inside an element. #[derive(Debug, Clone, PartialEq, Eq)] pub enum ScriptRewriteAction { Keep, @@ -84,33 +78,13 @@ impl ScriptRewriteAction { } } -/// Context provided to integration HTML attribute rewriters. -#[derive(Debug)] -pub struct IntegrationAttributeContext<'a> { - pub attribute_name: &'a str, - pub element_name: &'a str, - pub request_host: &'a str, - pub request_scheme: &'a str, - pub origin_host: &'a str, -} - -/// Context passed to script/text rewriters for inline HTML handling. -#[derive(Debug)] -pub struct IntegrationScriptContext<'a> { - pub selector: &'a str, - pub request_host: &'a str, - pub request_scheme: &'a str, - pub origin_host: &'a str, - pub is_last_in_text_node: bool, - pub max_buffered_script_bytes: usize, - pub document_state: &'a IntegrationDocumentState, -} - type IntegrationDocumentStateMap = BTreeMap<(&'static str, TypeId), Arc>; -/// Per-document state shared between HTML/script rewriters and post-processors. +/// State shared by the middleware working on one document in one phase. /// -/// This exists to support multi-phase HTML processing without requiring a second HTML parse. +/// What one handler leaves here a later handler or a stream processor reads, +/// so the document is parsed once. In the serve phase it starts with what +/// the request's hooks left, see [`IntegrationRequestState`]. #[derive(Clone, Default)] pub struct IntegrationDocumentState { inner: Arc>, @@ -195,20 +169,20 @@ impl IntegrationDocumentState { } } -/// Values a module's request hooks leave for its own page hooks, for one -/// request. +/// Values a module's request hooks leave for its own serve middleware, for +/// one request. /// /// A request preparer or a request filter leaves a value under its -/// integration id with [`IntegrationRequestState::insert`]. When the request -/// produces an HTML document for that one reader, every value is copied into -/// the document's [`IntegrationDocumentState`] before parsing starts, so the -/// module's head injector, rewriters and stream processors read it there, and -/// the same values are handed to the module's response finalizer. +/// integration id with [`IntegrationRequestState::insert`]. Where the +/// request's reader is served an HTML document, every value is copied into +/// the [`IntegrationDocumentState`] the `[[serve]]` entry's middleware share, +/// so the module's serve middleware reads it there, and the same values are +/// handed to the module's response finalizer. /// /// A request that carries any value keeps to the origin path. Its document is /// never read from a shared template and never stored as one, and an HTML -/// response with a body is sent `private, no-store`. A document built to be -/// shared starts with none of these values. +/// response with a body is sent `private, no-store`. A fetch middleware is +/// handed none of these values. #[derive(Clone, Default)] pub struct IntegrationRequestState { values: IntegrationDocumentStateMap, @@ -223,8 +197,8 @@ impl std::fmt::Debug for IntegrationRequestState { } impl IntegrationRequestState { - /// Leaves `value` on `request` for the page hooks of `integration_id`, - /// in place of a value of the same type left earlier. + /// Leaves `value` on `request` for the serve middleware of + /// `integration_id`, in place of a value of the same type left earlier. pub fn insert(request: &mut Request, integration_id: &'static str, value: T) where T: Any + Send + Sync + 'static, @@ -237,8 +211,8 @@ impl IntegrationRequestState { request.extensions_mut().insert(state); } - /// Holds `value` for the page hooks of `integration_id`, in place of a - /// value of the same type held earlier. + /// Holds `value` for the serve middleware of `integration_id`, in place + /// of a value of the same type held earlier. pub fn set(&mut self, integration_id: &'static str, value: T) where T: Any + Send + Sync + 'static, @@ -293,38 +267,6 @@ impl IntegrationRequestState { } } -/// Per-document buffer for script text fragments split across chunks. -/// -/// `lol_html` can deliver one text node as several chunks, so a rewriter that -/// needs the whole script must accumulate until `is_last_in_text_node`. -/// -/// This lives in [`IntegrationDocumentState`] rather than on the rewriter. -/// Rewriters are registered once as `Arc` and -/// live as long as the [`IntegrationRegistry`], so a buffer owned by a -/// rewriter is shared by every document that registry serves. A document whose -/// stream ends before the final fragment — client disconnect, origin error, -/// truncated body — leaves its partial script in that buffer, and the next -/// document prepends the residue to its own accumulation. That corrupts the -/// response and can disclose the previous document's content. -/// -/// Keyed per integration id, so each integration gets its own buffer, and -/// dropped with the document state at end of document. -#[derive(Debug, Default)] -pub struct ScriptTextAccumulator { - buffer: Mutex, -} - -impl ScriptTextAccumulator { - /// Locks the buffer. - /// - /// Recovers from poisoning rather than panicking: a poisoned buffer holds - /// at worst a partial script, and the caller's `is_last_in_text_node` - /// handling already tolerates unexpected contents. - pub fn buffer(&self) -> MutexGuard<'_, String> { - self.buffer.lock().unwrap_or_else(PoisonError::into_inner) - } -} - /// Describes an HTTP endpoint exposed by an integration. #[derive(Clone, Debug)] pub struct IntegrationEndpoint { @@ -669,83 +611,6 @@ fn apply_header_mutation_to_response(response: &mut Response, mutation } } -/// Trait for integration-provided HTML attribute rewrite hooks. -pub trait IntegrationAttributeRewriter: Send + Sync { - /// Identifier for logging/diagnostics. - fn integration_id(&self) -> &'static str; - /// Return true when this rewriter wants to inspect a given attribute. - fn handles_attribute(&self, attribute: &str) -> bool; - /// Attempt to rewrite the attribute value. Return `AttributeRewriteAction::Replace` - /// to update the attribute, `Keep` to leave it untouched, or `RemoveElement` to drop the node. - fn rewrite( - &self, - attr_name: &str, - attr_value: &str, - ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction; -} - -/// Trait for integration-provided inline script/text rewrite hooks. -pub trait IntegrationScriptRewriter: Send + Sync { - /// Identifier for logging/diagnostics. - fn integration_id(&self) -> &'static str; - /// CSS selector (e.g. `script#__NEXT_DATA__`) that should trigger this rewriter. - fn selector(&self) -> &'static str; - /// Attempt to rewrite the inline text content for the selector. - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction; -} - -/// Context for HTML post-processors. -#[derive(Debug)] -pub struct IntegrationHtmlContext<'a> { - pub request_host: &'a str, - pub request_scheme: &'a str, - pub origin_host: &'a str, - pub document_state: &'a IntegrationDocumentState, -} - -/// Owned request data supplied when an integration creates an HTML stream processor. -#[derive(Clone)] -pub struct IntegrationHtmlStreamContext { - /// Publisher-facing host used for rewritten URLs. - pub request_host: String, - /// Publisher-facing scheme used for rewritten URLs. - pub request_scheme: String, - /// Origin host whose URLs may be rewritten. - pub origin_host: String, - /// Request-local state shared with the document's integration rewriters. - pub document_state: IntegrationDocumentState, -} - -/// Creates one mutable HTML output processor for each document. -pub trait IntegrationHtmlStreamProcessorFactory: Send + Sync { - /// Identifier for logging and diagnostics. - fn integration_id(&self) -> &'static str; - - /// Create a request-local streaming processor. - fn create(&self, context: IntegrationHtmlStreamContext) -> Box; -} - -/// Trait for integration-provided HTML head injections. -pub trait IntegrationHeadInjector: Send + Sync { - /// Identifier for logging/diagnostics. - fn integration_id(&self) -> &'static str; - /// Return HTML snippets to insert at the start of ``. - fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec; - - /// Return HTML snippets to insert straight after the main script bundle - /// and before any deferred one, for a script that needs the bundle to - /// have run and has to run before the page's own scripts. - fn after_bundle_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - Vec::new() - } - - /// Return attributes to add to the publisher TSJS bundle tag. - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - Vec::new() - } -} - /// A browser module a registration carries, for a module built outside /// `trusted-server-js`. The crate embeds its built IIFE with `include_str!` /// and states its SHA-256 as a literal next to it; the registry verifies the @@ -771,11 +636,16 @@ pub struct IntegrationRegistration { /// integration injects the tag itself when it decides to. pub js_standalone: bool, pub proxies: Vec>, - pub attribute_rewriters: Vec>, - pub script_rewriters: Vec>, - pub html_stream_processors: Vec>, - pub head_injectors: Vec>, pub request_filters: Vec>, + /// The page changes this registration supplies, see [`crate::middleware`]. + /// + /// Declaring one does not make it run, because a middleware changes a + /// page only where an entry of the settings names it. + pub middleware: Vec>, + /// Attributes this registration puts on the script bundle's tag, each a + /// name and a value, for a browser module that reads a setting from the + /// tag it was loaded by. + pub bundle_tag_attributes: Vec<(&'static str, &'static str)>, /// Geo module this registration supplies, with the name `[geo] module` /// selects it by. /// @@ -819,11 +689,9 @@ impl IntegrationRegistrationBuilder { js_module: None, js_standalone: false, proxies: Vec::new(), - attribute_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters: Vec::new(), + middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), geo_module: None, ec_module: None, device_module: None, @@ -838,38 +706,33 @@ impl IntegrationRegistrationBuilder { } #[must_use] - pub fn with_attribute_rewriter( - mut self, - rewriter: Arc, - ) -> Self { - self.registration.attribute_rewriters.push(rewriter); - self - } - - #[must_use] - pub fn with_script_rewriter(mut self, rewriter: Arc) -> Self { - self.registration.script_rewriters.push(rewriter); - self - } - - #[must_use] - pub fn with_html_stream_processor( - mut self, - processor: Arc, - ) -> Self { - self.registration.html_stream_processors.push(processor); + pub fn with_request_filter(mut self, filter: Arc) -> Self { + self.registration.request_filters.push(filter); self } + /// Declare a page change this registration supplies, see + /// [`crate::middleware`]. Called once for each one. + /// + /// The middleware changes a page only where an entry of the settings + /// names it, and one no entry names is logged as a warning when the + /// registry is built. #[must_use] - pub fn with_head_injector(mut self, injector: Arc) -> Self { - self.registration.head_injectors.push(injector); + pub fn with_middleware(mut self, middleware: Arc) -> Self { + self.registration.middleware.push(middleware); self } + /// Put an attribute on the script bundle's tag, on every page the bundle + /// is written into. + /// + /// The name is lower case letters, digits and hyphens, and the value + /// holds none of `"`, `&`, `<` and `>`, or startup is refused. Where two + /// registrations give one name different values the first is kept, and + /// the other is logged as a warning. #[must_use] - pub fn with_request_filter(mut self, filter: Arc) -> Self { - self.registration.request_filters.push(filter); + pub fn with_bundle_tag_attribute(mut self, name: &'static str, value: &'static str) -> Self { + self.registration.bundle_tag_attributes.push((name, value)); self } @@ -987,11 +850,14 @@ struct IntegrationRegistryInner { // Modules carried by their registrations, verified against their // declared hash at construction. carried_js: Vec<(&'static str, CarriedJsModule)>, - html_rewriters: Vec>, - script_rewriters: Vec>, - html_stream_processors: Vec>, - head_injectors: Vec>, request_filters: Vec>, + // The middleware the running registrations supply, each against the + // integration that supplied it, in registration order. + middleware: Vec<(&'static str, Arc)>, + // The attributes the running registrations put on the script bundle's + // tag, each against the integration that asked for it, in registration + // order. + bundle_tag_attributes: Vec<(&'static str, (&'static str, &'static str))>, /// JS module IDs to include in the bundle that come from a source other than /// a registered integration, for example a module tied to the selected Edge /// Cookie module. Populated in [`IntegrationRegistry::new`] from settings. @@ -1038,11 +904,9 @@ impl Default for IntegrationRegistryInner { disabled_js_ids: Vec::new(), standalone_js_ids: Vec::new(), carried_js: Vec::new(), - html_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters: Vec::new(), + middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), extra_js_module_ids: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), @@ -1081,6 +945,35 @@ const DEVICE_TYPE: &str = "device"; const GEO_TYPE: &str = "geo"; impl IntegrationRegistryInner { + /// The registered middleware an entry means by `name`. + fn middleware_named(&self, name: &str) -> Option<&Arc> { + self.middleware + .iter() + .map(|(_, middleware)| middleware) + .find(|middleware| middleware.middleware_id() == name) + } + + /// A sentence to follow the refusal of a middleware name an entry wrote, + /// when the name is that of a module no section selects, or begins with + /// one. Empty otherwise. + fn unselected_module_note(&self, name: &str) -> String { + self.builder_modules + .iter() + .find(|(module, selected)| { + !*selected + && name + .strip_prefix(*module) + .is_some_and(|rest| rest.is_empty() || rest.starts_with('.')) + }) + .map(|(module, _)| { + format!( + ". `{module}` is a module no section selects, so nothing it supplies is \ + running" + ) + }) + .unwrap_or_default() + } + /// A sentence to follow the refusal of a name a selector wrote, when the /// name is a builder's module: one no section selects, which is why /// nothing it supplies is running, or one that is selected and supplies no @@ -1333,9 +1226,8 @@ fn resolve_geo_module( pub struct IntegrationMetadata { pub id: &'static str, pub routes: Vec, - pub attribute_rewriters: usize, - pub script_selectors: Vec<&'static str>, - pub head_injectors: usize, + /// The names of the middleware the integration supplies. + pub middleware: Vec<&'static str>, pub request_filters: usize, } @@ -1344,9 +1236,7 @@ impl IntegrationMetadata { Self { id, routes: Vec::new(), - attribute_rewriters: 0, - script_selectors: Vec::new(), - head_injectors: 0, + middleware: Vec::new(), request_filters: 0, } } @@ -1367,6 +1257,108 @@ pub struct ProxyDispatchInput<'a> { pub req: Request, } +/// Refuses a registered middleware no entry could name, and an entry this +/// deployment cannot run. +/// +/// A middleware no entry names is left alone and logged. +/// +/// # Errors +/// +/// Naming the module or the entry at fault. +fn check_middleware( + settings: &Settings, + inner: &IntegrationRegistryInner, +) -> Result<(), Report> { + let refuse = |message: String| Report::new(TrustedServerError::Configuration { message }); + let phases_of = |middleware: &Arc| -> Vec { + middleware + .phases() + .iter() + .map(|phase| format!("[[{phase}]]")) + .collect() + }; + for (position, (integration, middleware)) in inner.middleware.iter().enumerate() { + let id = middleware.middleware_id(); + if !crate::module_name::is_valid(id) { + return Err(refuse(format!( + "module `{integration}` supplies a middleware named `{id}`, which is not a \ + name an entry can write. A name is parts joined by `.`, each in lower case \ + letters, digits, `_` or `-`" + ))); + } + if middleware.phases().is_empty() { + return Err(refuse(format!( + "module `{integration}` supplies the middleware `{id}`, which says it runs \ + in no phase, so no entry could name it" + ))); + } + if let Some((earlier, _)) = inner.middleware[..position] + .iter() + .find(|(_, earlier)| earlier.middleware_id() == id) + { + return Err(refuse(format!( + "modules `{earlier}` and `{integration}` both supply a middleware named \ + `{id}`, so an entry could not say which one it means" + ))); + } + } + for phase in MiddlewarePhase::ALL { + for (index, entry) in settings.phase_entries(phase).entries().iter().enumerate() { + let at = format!("[[{phase}]] entry {}", index + 1); + for name in &entry.middleware { + let Some(middleware) = inner.middleware_named(name) else { + let supplied: Vec<&str> = inner + .middleware + .iter() + .map(|(_, middleware)| middleware.middleware_id()) + .collect(); + return Err(refuse(format!( + "{at} names `{name}`, which no module that runs supplies. The \ + middleware the running modules supply is [{}]{}", + supplied.join(", "), + inner.unselected_module_note(name) + ))); + }; + if !middleware.phases().contains(&phase) { + return Err(refuse(format!( + "{at} names `{name}`, which does not run in that phase. It runs in {}", + phases_of(middleware).join(" and ") + ))); + } + } + } + } + for (integration, middleware) in unnamed_middleware(settings, inner) { + log::warn!( + "Module `{integration}` supplies the middleware `{}` and no {} entry names \ + it, so it changes no page", + middleware.middleware_id(), + phases_of(middleware).join(" or ") + ); + } + Ok(()) +} + +/// The middleware no entry names in a phase it runs in, each with the +/// integration that supplies it, in registration order. +fn unnamed_middleware<'a>( + settings: &Settings, + inner: &'a IntegrationRegistryInner, +) -> Vec<(&'static str, &'a Arc)> { + inner + .middleware + .iter() + .filter(|(_, middleware)| { + let id = middleware.middleware_id(); + !middleware + .phases() + .iter() + .any(|phase| settings.phase_entries(*phase).names().contains(&id)) + }) + .map(|(integration, middleware)| (*integration, middleware)) + .collect() +} + /// In-memory registry of integrations discovered from settings. #[derive(Clone, Default)] pub struct IntegrationRegistry { @@ -1629,15 +1621,37 @@ impl IntegrationRegistry { inner.routes.push((route, registration.integration_id)); } } - inner - .html_rewriters - .extend(registration.attribute_rewriters); - inner.script_rewriters.extend(registration.script_rewriters); - inner - .html_stream_processors - .extend(registration.html_stream_processors); - inner.head_injectors.extend(registration.head_injectors); inner.request_filters.extend(registration.request_filters); + for middleware in registration.middleware { + inner + .middleware + .push((registration.integration_id, middleware)); + } + for (name, value) in registration.bundle_tag_attributes { + // The tag is written as markup with no escaping, so a name + // or a value that could end the attribute is refused. + let name_is_plain = !name.is_empty() + && name.bytes().all(|byte| { + byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-' + }); + let value_is_plain = !value + .bytes() + .any(|byte| matches!(byte, b'"' | b'&' | b'<' | b'>')); + if !name_is_plain || !value_is_plain { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "integration `{}` puts the attribute `{name}` on the script \ + bundle's tag with the value `{value}`. A name is lower case \ + letters, digits and hyphens, and a value holds none of `\"`, `&`, \ + `<` and `>`", + registration.integration_id + ), + })); + } + inner + .bundle_tag_attributes + .push((registration.integration_id, (name, value))); + } if let Some((name, module)) = registration.geo_module { claim_module_name(&mut claimed, "geo", name, registration.integration_id)?; inner.geo_modules.push((name, module)); @@ -1717,6 +1731,7 @@ impl IntegrationRegistry { inner.ec_module = resolve_ec_module(settings, &inner); inner.device_module = resolve_device_module(settings, &inner)?; inner.geo_module = geo_module; + check_middleware(settings, &inner)?; Ok(Self { inner: Arc::new(inner), @@ -1754,6 +1769,48 @@ impl IntegrationRegistry { self.inner.device_module.clone() } + /// The name of every middleware a running module supplies, in + /// registration order. + #[must_use] + pub fn middleware_ids(&self) -> Vec<&'static str> { + self.inner + .middleware + .iter() + .map(|(_, middleware)| middleware.middleware_id()) + .collect() + } + + /// The names of the middleware that run in `phase`, in registration + /// order. + #[must_use] + pub fn middleware_in(&self, phase: MiddlewarePhase) -> Vec<&'static str> { + self.inner + .middleware + .iter() + .filter(|(_, middleware)| middleware.phases().contains(&phase)) + .map(|(_, middleware)| middleware.middleware_id()) + .collect() + } + + /// The middleware the first entry covering a response selects, in the + /// order they run, for a response of `media_type` to a request for `path`. + /// A response no entry covers gives an empty chain. + #[must_use] + pub fn middleware_chain( + &self, + entries: &PhaseEntries, + phase: MiddlewarePhase, + media_type: &str, + path: &str, + ) -> MiddlewareChain { + let middleware = entries + .for_response(media_type, path) + .iter() + .filter_map(|name| self.inner.middleware_named(name).map(Arc::clone)) + .collect(); + MiddlewareChain::new(phase, middleware) + } + /// Every integration id the registry was built from, named or not, in /// registration order. A registry test enumerates this to check every /// builder was considered, while the deploy validation test iterates the @@ -1946,102 +2003,29 @@ impl IntegrationRegistry { } } - /// Give integrations a chance to rewrite HTML attributes. - #[must_use] - pub fn rewrite_attribute( - &self, - attr_name: &str, - attr_value: &str, - ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteOutcome { - let mut current = attr_value.to_owned(); - let mut changed = false; - for rewriter in &self.inner.html_rewriters { - if !rewriter.handles_attribute(attr_name) { - continue; - } - match rewriter.rewrite(attr_name, ¤t, ctx) { - AttributeRewriteAction::Keep => {} - AttributeRewriteAction::Replace(next_value) => { - current = next_value; - changed = true; - } - AttributeRewriteAction::RemoveElement => { - return AttributeRewriteOutcome::RemoveElement; - } - } - } - - if changed { - AttributeRewriteOutcome::Replaced(current) - } else { - AttributeRewriteOutcome::Unchanged - } - } - - /// Expose registered script/text rewriters for HTML processing. - #[must_use] - pub fn script_rewriters(&self) -> Vec> { - self.inner.script_rewriters.clone() - } - - /// Expose registered per-document HTML stream processor factories. - #[must_use] - pub fn html_stream_processor_factories( - &self, - ) -> Vec> { - self.inner.html_stream_processors.clone() - } - - /// Collect HTML snippets for insertion at the start of ``. - #[must_use] - pub fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - let mut inserts = Vec::new(); - for injector in &self.inner.head_injectors { - let mut next = injector.head_inserts(ctx); - if !next.is_empty() { - inserts.append(&mut next); - } - } - inserts - } - - /// Collect HTML snippets for insertion straight after the main script - /// bundle. - #[must_use] - pub fn after_bundle_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - self.inner - .head_injectors - .iter() - .flat_map(|injector| injector.after_bundle_inserts(ctx)) - .collect() - } - - /// Collect static attributes for the publisher TSJS bundle tag. + /// Collect the attributes the registrations put on the publisher TSJS + /// bundle tag, each keeping the first value a name is given. #[must_use] pub fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { let mut attributes: Vec<(&'static str, &'static str)> = Vec::new(); - for injector in &self.inner.head_injectors { - for attribute in injector.tsjs_script_tag_attributes() { - let existing = attributes - .iter() - .find(|(name, _)| *name == attribute.0) - .copied(); - match existing { - None => attributes.push(attribute), - Some((_, kept_value)) if kept_value != attribute.1 => log::warn!( - "Integration `{}` emits conflicting value for publisher tag attribute `{}`; keeping the first", - injector.integration_id(), - attribute.0 - ), - Some(_) => {} - } + for (integration, attribute) in self.inner.bundle_tag_attributes.iter().copied() { + let existing = attributes + .iter() + .find(|(name, _)| *name == attribute.0) + .copied(); + match existing { + None => attributes.push(attribute), + Some((_, kept_value)) if kept_value != attribute.1 => log::warn!( + "Integration `{integration}` emits conflicting value for publisher tag attribute `{}`; keeping the first", + attribute.0 + ), + Some(_) => {} } } attributes } - /// Provide a snapshot of registered integrations and their hooks. + /// Provide a snapshot of registered integrations and what each supplies. #[must_use] pub fn registered_integrations(&self) -> Vec { let mut map: BTreeMap<&'static str, IntegrationMetadata> = BTreeMap::new(); @@ -2061,25 +2045,11 @@ impl IntegrationRegistry { )); } - for rewriter in &self.inner.html_rewriters { - let entry = map - .entry(rewriter.integration_id()) - .or_insert_with(|| IntegrationMetadata::new(rewriter.integration_id())); - entry.attribute_rewriters += 1; - } - - for rewriter in &self.inner.script_rewriters { - let entry = map - .entry(rewriter.integration_id()) - .or_insert_with(|| IntegrationMetadata::new(rewriter.integration_id())); - entry.script_selectors.push(rewriter.selector()); - } - - for injector in &self.inner.head_injectors { + for (integration_id, middleware) in &self.inner.middleware { let entry = map - .entry(injector.integration_id()) - .or_insert_with(|| IntegrationMetadata::new(injector.integration_id())); - entry.head_injectors += 1; + .entry(*integration_id) + .or_insert_with(|| IntegrationMetadata::new(integration_id)); + entry.middleware.push(middleware.middleware_id()); } for filter in &self.inner.request_filters { @@ -2276,91 +2246,6 @@ impl IntegrationRegistry { } } - #[cfg(test)] - #[must_use] - pub fn from_rewriters( - attribute_rewriters: Vec>, - script_rewriters: Vec>, - ) -> Self { - Self { - inner: Arc::new(IntegrationRegistryInner { - get_router: Router::new(), - post_router: Router::new(), - put_router: Router::new(), - delete_router: Router::new(), - patch_router: Router::new(), - head_router: Router::new(), - options_router: Router::new(), - routes: Vec::new(), - builder_ids: Vec::new(), - running_integration_ids: Vec::new(), - html_rewriters: attribute_rewriters, - script_rewriters, - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), - request_filters: Vec::new(), - request_preparers: Vec::new(), - response_finalizers: Vec::new(), - deferred_js_ids: Vec::new(), - disabled_js_ids: Vec::new(), - extra_js_module_ids: Vec::new(), - standalone_js_ids: Vec::new(), - carried_js: Vec::new(), - geo_modules: Vec::new(), - ec_modules: Vec::new(), - device_modules: Vec::new(), - builder_modules: Vec::new(), - geo_module: None, - ec_module: None, - device_module: None, - }), - plan: None, - } - } - - #[cfg(test)] - #[must_use] - pub fn from_rewriters_with_head_injectors( - attribute_rewriters: Vec>, - script_rewriters: Vec>, - head_injectors: Vec>, - ) -> Self { - Self { - inner: Arc::new(IntegrationRegistryInner { - get_router: Router::new(), - post_router: Router::new(), - put_router: Router::new(), - delete_router: Router::new(), - patch_router: Router::new(), - head_router: Router::new(), - options_router: Router::new(), - routes: Vec::new(), - builder_ids: Vec::new(), - running_integration_ids: Vec::new(), - html_rewriters: attribute_rewriters, - script_rewriters, - html_stream_processors: Vec::new(), - head_injectors, - request_filters: Vec::new(), - request_preparers: Vec::new(), - response_finalizers: Vec::new(), - deferred_js_ids: Vec::new(), - disabled_js_ids: Vec::new(), - extra_js_module_ids: Vec::new(), - standalone_js_ids: Vec::new(), - carried_js: Vec::new(), - geo_modules: Vec::new(), - ec_modules: Vec::new(), - device_modules: Vec::new(), - builder_modules: Vec::new(), - geo_module: None, - ec_module: None, - device_module: None, - }), - plan: None, - } - } - #[cfg(any(test, feature = "test-utils"))] #[must_use] pub fn from_request_filters(request_filters: Vec>) -> Self { @@ -2376,11 +2261,9 @@ impl IntegrationRegistry { routes: Vec::new(), builder_ids: Vec::new(), running_integration_ids: Vec::new(), - html_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters, + middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2455,11 +2338,9 @@ impl IntegrationRegistry { routes: Vec::new(), builder_ids: Vec::new(), running_integration_ids: Vec::new(), - html_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters: Vec::new(), + middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2534,10 +2415,10 @@ pub(crate) mod test_support { /// does, the way a module that reserves a cookie does. Selected, its /// request filter leaves the same mark on a request that carries its /// header, which is the route a module that decides in its filter takes. - /// Selected, the - /// stand-in's head injector reads the mark from the document's state and - /// writes one script at the start of `` and the tag of its - /// standalone module after the bundle. Its response finalizer sets a + /// Selected, it supplies one serve middleware under its own name, which + /// reads the mark from the state the request left and writes one script + /// ahead of the bundle and the tag of its standalone module after it, + /// into that reader's copy alone. Its response finalizer sets a /// cookie on the response to a marked request, and its builder declares /// that it reads the auction token. pub(crate) mod request_fixture { @@ -2549,11 +2430,12 @@ pub(crate) mod test_support { use crate::error::TrustedServerError; use crate::integrations::registry::{ - CarriedJsModule, IntegrationHeadInjector, IntegrationHtmlContext, - IntegrationRegistration, IntegrationRequestFilter, IntegrationRequestState, - RequestFilterDecision, RequestFilterEffects, RequestFilterInput, + CarriedJsModule, IntegrationRegistration, IntegrationRequestFilter, + IntegrationRequestState, RequestFilterDecision, RequestFilterEffects, + RequestFilterInput, }; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase}; use crate::settings::Settings; use crate::tsjs_bundle::JsModulePart; @@ -2707,40 +2589,46 @@ pub(crate) mod test_support { struct Head; - impl Head { - fn marked(ctx: &IntegrationHtmlContext<'_>) -> bool { - ctx.document_state.get::(ID).is_some() - } - } - - impl IntegrationHeadInjector for Head { - fn integration_id(&self) -> &'static str { - ID + impl Middleware for Head { + fn middleware_id(&self) -> &'static str { + MODULE } - fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - if !Self::marked(ctx) { - return Vec::new(); - } - vec![format!("")] + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] } - fn after_bundle_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - if !Self::marked(ctx) { - return Vec::new(); + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + if context.document_state.get::(ID).is_none() { + return MiddlewareAction::pass(); } let module = JsModulePart { id: ID, source: JS, sha256: JS_SHA256, }; - vec![format!( - "", - crate::tsjs::tsjs_single_module_script_src(&module) - )] + MiddlewareAction { + head_inserts: vec![format!("")], + after_bundle_inserts: vec![format!( + "", + crate::tsjs::tsjs_single_module_script_src(&module) + )], + ..MiddlewareAction::pass() + } } } + /// Has `settings` select the stand-in and run its middleware, and no + /// other, on every reader's copy of a page. + pub(crate) fn select_and_place(settings: &mut Settings) { + settings.select_module("testing", MODULE); + crate::html_processor::test_support::place_on_every_page( + settings, + MiddlewarePhase::Serve, + &[MODULE], + ); + } + fn register( settings: &Settings, ) -> Result, Report> { @@ -2754,7 +2642,7 @@ pub(crate) mod test_support { sha256: JS_SHA256, }) .with_standalone_js() - .with_head_injector(Arc::new(Head)) + .with_middleware(Arc::new(Head)) .with_request_filter(Arc::new(Filter)) .build(), )) @@ -2771,19 +2659,18 @@ pub(crate) mod test_support { /// core's own tests of how modules are divided between the bundle and /// deferred tags, and of how a module's settings reach a page template. /// - /// Selected, it carries a deferred browser module and writes its two - /// settings into ``. + /// Selected, it carries a deferred browser module and supplies one fetch + /// middleware under its own name, which writes its two settings into + /// ``. pub(crate) mod deferred_fixture { use std::sync::Arc; use error_stack::Report; use crate::error::TrustedServerError; - use crate::integrations::registry::{ - CarriedJsModule, IntegrationHeadInjector, IntegrationHtmlContext, - IntegrationRegistration, - }; + use crate::integrations::registry::{CarriedJsModule, IntegrationRegistration}; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase}; use crate::settings::Settings; /// The integration id the stand-in registers under. @@ -2817,22 +2704,39 @@ pub(crate) mod test_support { timeout_ms: u32, } - impl IntegrationHeadInjector for Head { - fn integration_id(&self) -> &'static str { - ID + impl Middleware for Head { + fn middleware_id(&self) -> &'static str { + MODULE } - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { let config = serde_json::json!({ "label": self.label, "timeoutMs": self.timeout_ms, }); - vec![format!( - "" - )] + MiddlewareAction { + head_inserts: vec![format!( + "" + )], + ..MiddlewareAction::pass() + } } } + /// Has `settings` run the stand-in's middleware, and no other, on + /// every page. + pub(crate) fn place(settings: &mut Settings) { + crate::html_processor::test_support::place_on_every_page( + settings, + MiddlewarePhase::Fetch, + &[MODULE], + ); + } + fn register( settings: &Settings, ) -> Result, Report> { @@ -2846,7 +2750,7 @@ pub(crate) mod test_support { sha256: JS_SHA256, }) .with_deferred_js() - .with_head_injector(Arc::new(Head { + .with_middleware(Arc::new(Head { label: config.label, timeout_ms: config.timeout_ms, })) @@ -2861,38 +2765,54 @@ pub(crate) mod test_support { } } - /// A stand-in for an integration that tags a page, for core's own tests - /// of the HTML processor and the JavaScript asset proxy. + /// A stand-in for a module that changes a page through middleware, for + /// core's own tests of the entries and of the page path. /// - /// Selected, it inserts one script at the start of `` and rewrites - /// the address of its own script to a first-party path. With - /// `mark_bundle` set it also asks for an attribute on the publisher - /// bundle tag. - pub(crate) mod tag_fixture { + /// Selected, it supplies three fetch middleware. The one under the + /// module's own name writes a marker carrying the module's `label` + /// setting at the start of ``. The one named `.links` points every + /// link that carries `data-fixture` at a fixed path. The one named + /// `.broken` asks for a selector that does not parse. + /// + /// It supplies two serve middleware as well. The one named `.reader` + /// writes a marker ahead of the script bundle, saying which origin it was + /// told of and whether the request stand-in left its mark, and a script + /// straight after the bundle. The one named `.broken-reader` asks for a + /// selector that does not parse. + pub(crate) mod middleware_fixture { + use std::rc::Rc; use std::sync::Arc; use error_stack::Report; use crate::error::TrustedServerError; - use crate::integrations::registry::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationHeadInjector, IntegrationHtmlContext, IntegrationRegistration, - }; + use crate::integrations::registry::{AttributeRewriteAction, IntegrationRegistration}; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{ + AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, + }; use crate::settings::Settings; /// The integration id the stand-in registers under. - pub(crate) const ID: &str = "tag_fixture"; + pub(crate) const ID: &str = "middleware_fixture"; /// The name a test's settings select the stand-in by, in `[testing]`. - pub(crate) const MODULE: &str = "testing.tag-fixture"; - /// The script the stand-in's vendor would serve. - pub(crate) const SCRIPT_URL: &str = "https://cdn.tag-fixture.example/sdk.js"; - /// The first-party path the stand-in rewrites that script to. - pub(crate) const FIRST_PARTY_SCRIPT: &str = "/integrations/tag_fixture/script"; - /// What the stand-in's head insert sets, so a test can find it. - pub(crate) const HEAD_FLAG: &str = "window.__ts_tag_fixture=true;"; - /// The attribute the stand-in asks for on the publisher bundle tag. - pub(crate) const BUNDLE_ATTRIBUTE: &str = "data-ts-tag-fixture"; + pub(crate) const MODULE: &str = "testing.middleware-fixture"; + /// The name of the middleware that marks the head. + pub(crate) const HEAD: &str = MODULE; + /// The name of the middleware that moves links. + pub(crate) const LINKS: &str = "testing.middleware-fixture.links"; + /// The name of the middleware whose selector does not parse. + pub(crate) const BROKEN: &str = "testing.middleware-fixture.broken"; + /// The name of the serve middleware that marks a reader's copy. + pub(crate) const READER: &str = "testing.middleware-fixture.reader"; + /// The name of the serve middleware whose selector does not parse. + pub(crate) const BROKEN_READER: &str = "testing.middleware-fixture.broken-reader"; + /// What the reader middleware writes straight after the bundle. + pub(crate) const READER_SCRIPT: &str = ""; + /// Where the links middleware points a link. + pub(crate) const LINK_TARGET: &str = "/fixture/link"; + /// The selector the broken middleware asks for. + pub(crate) const BROKEN_SELECTOR: &str = "a["; /// The builder core's test build lists beside its own. pub(crate) const BUILDER: IntegrationBuilder = @@ -2902,52 +2822,137 @@ pub(crate) mod test_support { #[serde(deny_unknown_fields)] struct FixtureSettings { #[serde(default)] - mark_bundle: bool, + label: String, } impl crate::settings::IntegrationConfig for FixtureSettings {} - struct Tag { - mark_bundle: bool, + /// The marker the head middleware writes for `label`. + pub(crate) fn head_marker(label: &str) -> String { + format!("") } - impl IntegrationHeadInjector for Tag { - fn integration_id(&self) -> &'static str { - ID + struct Head { + label: String, + } + + impl Middleware for Head { + fn middleware_id(&self) -> &'static str { + HEAD } - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - vec![format!("")] + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - if self.mark_bundle { - vec![(BUNDLE_ATTRIBUTE, "true")] - } else { - Vec::new() + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + head_inserts: vec![head_marker(&self.label)], + ..MiddlewareAction::pass() } } } - impl IntegrationAttributeRewriter for Tag { - fn integration_id(&self) -> &'static str { - ID + /// The marker the reader middleware writes when told of `origin_host`, + /// for a request the request stand-in marked or did not. + pub(crate) fn reader_marker(origin_host: &str, marked: bool) -> String { + format!( + "" + ) + } + + struct Reader; + + impl Middleware for Reader { + fn middleware_id(&self) -> &'static str { + READER } - fn handles_attribute(&self, attribute: &str) -> bool { - attribute == "src" + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if attr_value == SCRIPT_URL { - AttributeRewriteAction::Replace(FIRST_PARTY_SCRIPT.to_owned()) - } else { - AttributeRewriteAction::Keep + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + use super::request_fixture; + + let marked = context + .document_state + .get::(request_fixture::ID) + .is_some(); + MiddlewareAction { + head_inserts: vec![reader_marker(context.origin_host, marked)], + after_bundle_inserts: vec![READER_SCRIPT.to_owned()], + ..MiddlewareAction::pass() + } + } + } + + struct BrokenReader; + + impl Middleware for BrokenReader { + fn middleware_id(&self) -> &'static str { + BROKEN_READER + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + BROKEN_SELECTOR, + "href", + Rc::new(|_matched| AttributeRewriteAction::keep()), + ))], + ..MiddlewareAction::pass() + } + } + } + + struct Links; + + impl Middleware for Links { + fn middleware_id(&self) -> &'static str { + LINKS + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + "a[data-fixture]", + "href", + Rc::new(|_matched| AttributeRewriteAction::replace(LINK_TARGET)), + ))], + ..MiddlewareAction::pass() + } + } + } + + struct Broken; + + impl Middleware for Broken { + fn middleware_id(&self) -> &'static str { + BROKEN + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + BROKEN_SELECTOR, + "href", + Rc::new(|_matched| AttributeRewriteAction::keep()), + ))], + ..MiddlewareAction::pass() } } } @@ -2958,14 +2963,16 @@ pub(crate) mod test_support { let Some(config) = settings.module_config::(MODULE)? else { return Ok(None); }; - let tag = Arc::new(Tag { - mark_bundle: config.mark_bundle, - }); Ok(Some( IntegrationRegistration::builder(ID) .without_js() - .with_attribute_rewriter(tag.clone()) - .with_head_injector(tag) + .with_middleware(Arc::new(Head { + label: config.label, + })) + .with_middleware(Arc::new(Links)) + .with_middleware(Arc::new(Broken)) + .with_middleware(Arc::new(Reader)) + .with_middleware(Arc::new(BrokenReader)) .build(), )) } @@ -2977,16 +2984,114 @@ pub(crate) mod test_support { } } + /// A stand-in for an integration that tags a page, for core's own tests + /// of the HTML processor and the JavaScript asset proxy. + /// + /// Selected, it supplies one fetch middleware under its own name, which + /// writes one script in `` ahead of the bundle and rewrites the + /// address of its own script to a first-party path. With `mark_bundle` + /// set it also asks for an attribute on the publisher bundle tag. + pub(crate) mod tag_fixture { + use std::rc::Rc; + use std::sync::Arc; + + use error_stack::Report; + + use crate::error::TrustedServerError; + use crate::integrations::registry::{AttributeRewriteAction, IntegrationRegistration}; + use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, + }; + use crate::settings::Settings; + + /// The integration id the stand-in registers under. + pub(crate) const ID: &str = "tag_fixture"; + /// The name a test's settings select the stand-in by, in `[testing]`. + pub(crate) const MODULE: &str = "testing.tag-fixture"; + /// The script the stand-in's vendor would serve. + pub(crate) const SCRIPT_URL: &str = "https://cdn.tag-fixture.example/sdk.js"; + /// The first-party path the stand-in rewrites that script to. + pub(crate) const FIRST_PARTY_SCRIPT: &str = "/integrations/tag_fixture/script"; + /// What the stand-in's head insert sets, so a test can find it. + pub(crate) const HEAD_FLAG: &str = "window.__ts_tag_fixture=true;"; + /// The attribute the stand-in asks for on the publisher bundle tag. + pub(crate) const BUNDLE_ATTRIBUTE: &str = "data-ts-tag-fixture"; + + /// The builder core's test build lists beside its own. + pub(crate) const BUILDER: IntegrationBuilder = + IntegrationBuilder::new(ID, CORE_SOURCE, register, validate).with_module_name(MODULE); + + #[derive(Debug, serde::Deserialize, validator::Validate)] + #[serde(deny_unknown_fields)] + struct FixtureSettings { + #[serde(default)] + mark_bundle: bool, + } + + impl crate::settings::IntegrationConfig for FixtureSettings {} + + struct Tag; + + impl Middleware for Tag { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let decide: Rc = Rc::new(|matched| { + if matched.value == SCRIPT_URL { + AttributeRewriteAction::Replace(FIRST_PARTY_SCRIPT.to_owned()) + } else { + AttributeRewriteAction::Keep + } + }); + MiddlewareAction { + head_inserts: vec![format!("")], + element_handlers: vec![Box::new(AttributeRewrite::new("src", decide))], + ..MiddlewareAction::pass() + } + } + } + + fn register( + settings: &Settings, + ) -> Result, Report> { + let Some(config) = settings.module_config::(MODULE)? else { + return Ok(None); + }; + let mut registration = IntegrationRegistration::builder(ID) + .without_js() + .with_middleware(Arc::new(Tag)); + if config.mark_bundle { + registration = registration.with_bundle_tag_attribute(BUNDLE_ATTRIBUTE, "true"); + } + Ok(Some(registration.build())) + } + + fn validate(settings: &Settings) -> Result> { + settings + .module_config::(MODULE) + .map(|config| config.is_some()) + } + } + /// A stand-in for an integration that rewrites script payloads in two /// passes, for core's own tests of the page pipeline. /// - /// The script rewriter swaps the payload of each `fixture_payload("...")` - /// call for a placeholder that carries a namespace made per document, and - /// keeps the payload in the document state. The stream processor swaps - /// each placeholder back for its payload with the origin host rewritten. - /// A payload pushed with `fixture_payload_open` leaves its group - /// unresolved, and the processor holds its output from that placeholder - /// on until a `fixture_payload_close` arrives. + /// Selected, it supplies one fetch middleware under its own name. Its + /// text handler swaps the payload of each `fixture_payload("...")` call + /// for a placeholder that carries a namespace made per document, and + /// keeps the payload. Its stream processor swaps each placeholder back + /// for its payload with the origin host rewritten. A payload pushed with + /// `fixture_payload_open` leaves its group unresolved, and the processor + /// holds its output from that placeholder on until a + /// `fixture_payload_close` arrives. pub(crate) mod payload_fixture { use std::io; use std::sync::{Arc, Mutex, PoisonError}; @@ -2994,12 +3099,11 @@ pub(crate) mod test_support { use error_stack::Report; use crate::error::TrustedServerError; - use crate::integrations::registry::{ - IntegrationHtmlStreamContext, IntegrationHtmlStreamProcessorFactory, - IntegrationRegistration, IntegrationScriptContext, IntegrationScriptRewriter, - ScriptRewriteAction, ScriptTextAccumulator, - }; + use crate::integrations::registry::{IntegrationRegistration, ScriptRewriteAction}; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, TextHandler, + }; use crate::settings::Settings; use crate::streaming_processor::StreamProcessor; @@ -3024,7 +3128,7 @@ pub(crate) mod test_support { unresolved: bool, } - /// What one document's script rewriter has captured so far. + /// What one document's text handler has captured so far. struct Captured { namespace: String, payloads: Vec, @@ -3039,18 +3143,16 @@ pub(crate) mod test_support { } } - fn captured( - state: &crate::integrations::registry::IntegrationDocumentState, - ) -> Arc> { - state.get_or_insert_with(ID, || Mutex::new(Captured::default())) + /// The scripts of one document, each held until its last chunk. + struct Scripts { + captured: Arc>, + held: String, } - struct ScriptRewriter; - - impl ScriptRewriter { + impl Scripts { /// Swaps the payload of a whole script for a placeholder, or /// leaves a script that pushes no payload as it is. - fn rewrite_whole(script: &str, ctx: &IntegrationScriptContext<'_>) -> Option { + fn rewrite_whole(&self, script: &str) -> Option { let (call, opens, closes) = [ ("fixture_payload_open(\"", true, false), ("fixture_payload_close(\"", false, true), @@ -3064,8 +3166,7 @@ pub(crate) mod test_support { return None; } - let shared = captured(ctx.document_state); - let mut captured = shared.lock().unwrap_or_else(PoisonError::into_inner); + let mut captured = self.captured.lock().unwrap_or_else(PoisonError::into_inner); let placeholder = format!( "{PLACEHOLDER_PREFIX}{}_{}{PLACEHOLDER_END}", captured.namespace, @@ -3088,55 +3189,61 @@ pub(crate) mod test_support { } } - impl IntegrationScriptRewriter for ScriptRewriter { - fn integration_id(&self) -> &'static str { - ID - } - - fn selector(&self) -> &'static str { + impl TextHandler for Scripts { + fn selector(&self) -> &str { "script" } - fn rewrite( - &self, - content: &str, - ctx: &IntegrationScriptContext<'_>, - ) -> ScriptRewriteAction { - let accumulator = ctx - .document_state - .get_or_insert_with(ID, ScriptTextAccumulator::default); - let mut buffer = accumulator.buffer(); - let claimed = !buffer.is_empty() || content.contains("fixture_payload"); + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + let claimed = !self.held.is_empty() || text.contains("fixture_payload"); if !claimed { return ScriptRewriteAction::Keep; } - buffer.push_str(content); - if !ctx.is_last_in_text_node { + self.held.push_str(text); + if !is_last { return ScriptRewriteAction::RemoveNode; } - let script = std::mem::take(&mut *buffer); - let rewritten = Self::rewrite_whole(&script, ctx).unwrap_or(script); + let script = std::mem::take(&mut self.held); + let rewritten = self.rewrite_whole(&script).unwrap_or(script); ScriptRewriteAction::replace(rewritten) } } - struct StreamFactory; + struct Payloads; - impl IntegrationHtmlStreamProcessorFactory for StreamFactory { - fn integration_id(&self) -> &'static str { - ID + impl Middleware for Payloads { + fn middleware_id(&self) -> &'static str { + MODULE } - fn create(&self, context: IntegrationHtmlStreamContext) -> Box { - Box::new(Processor { - context, - held: Vec::new(), - }) + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + /// The text handler and the stream processor of one document + /// share what the handler captures. + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + let captured = Arc::new(Mutex::new(Captured::default())); + MiddlewareAction { + text_handlers: vec![Box::new(Scripts { + captured: Arc::clone(&captured), + held: String::new(), + })], + stream: Some(Box::new(Processor { + captured, + origin_host: context.origin_host.to_owned(), + request_host: context.request_host.to_owned(), + held: Vec::new(), + })), + ..MiddlewareAction::pass() + } } } struct Processor { - context: IntegrationHtmlStreamContext, + captured: Arc>, + origin_host: String, + request_host: String, /// Output not yet released, because it ends inside a placeholder /// or starts at one whose group is unresolved. held: Vec, @@ -3153,8 +3260,7 @@ pub(crate) mod test_support { impl StreamProcessor for Processor { fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> Result, io::Error> { self.held.extend_from_slice(chunk); - let shared = captured(&self.context.document_state); - let captured = shared.lock().unwrap_or_else(PoisonError::into_inner); + let captured = self.captured.lock().unwrap_or_else(PoisonError::into_inner); let prefix = PLACEHOLDER_PREFIX.as_bytes(); let mut out = Vec::with_capacity(self.held.len()); let mut at = 0; @@ -3188,7 +3294,7 @@ pub(crate) mod test_support { out.extend_from_slice( payload .original - .replace(&self.context.origin_host, &self.context.request_host) + .replace(&self.origin_host, &self.request_host) .as_bytes(), ); at = end; @@ -3222,6 +3328,17 @@ pub(crate) mod test_support { } } + /// Has `settings` select the stand-in and run its middleware, and no + /// other, on every page. + pub(crate) fn select_and_place(settings: &mut Settings) { + settings.select_module("testing", MODULE); + crate::html_processor::test_support::place_on_every_page( + settings, + MiddlewarePhase::Fetch, + &[MODULE], + ); + } + fn register( settings: &Settings, ) -> Result, Report> { @@ -3230,8 +3347,7 @@ pub(crate) mod test_support { } Ok(Some( IntegrationRegistration::builder(ID) - .with_script_rewriter(Arc::new(ScriptRewriter)) - .with_html_stream_processor(Arc::new(StreamFactory)) + .with_middleware(Arc::new(Payloads)) .build(), )) } @@ -3254,88 +3370,17 @@ pub(crate) mod test_support { #[cfg(test)] mod tests { + use super::test_support::middleware_fixture as fixture; use super::test_support::{ PROBE_JS, PROBE_JS_SHA256, carried_probe_registration, probe_registration, validate_nothing, }; use super::*; use crate::constants::COOKIE_TS_EC; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewareAction, MiddlewareContext, PhaseEntry}; use crate::permissions::{Permission, PermissionSet, PermissionState}; use crate::platform::test_support::noop_services; use http::{HeaderValue, StatusCode, header}; - struct DefaultMetadataHeadInjector; - - impl IntegrationHeadInjector for DefaultMetadataHeadInjector { - fn integration_id(&self) -> &'static str { - "default-metadata" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - Vec::new() - } - } - - struct StaticMetadataHeadInjector; - - impl IntegrationHeadInjector for StaticMetadataHeadInjector { - fn integration_id(&self) -> &'static str { - "static-metadata" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - Vec::new() - } - - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - vec![ - ("data-ts-gam-attribution", "true"), - ("data-test-order", "second"), - ] - } - } - - struct ConflictingMetadataHeadInjector; - - impl IntegrationHeadInjector for ConflictingMetadataHeadInjector { - fn integration_id(&self) -> &'static str { - "conflicting-metadata" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - Vec::new() - } - - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - vec![ - ("data-ts-gam-attribution", "false"), - ("data-third-attribute", "third"), - ] - } - } - - #[test] - fn tsjs_script_tag_attributes_preserve_registration_order_and_default_empty() { - let registry = IntegrationRegistry::from_rewriters_with_head_injectors( - Vec::new(), - Vec::new(), - vec![ - Arc::new(DefaultMetadataHeadInjector), - Arc::new(StaticMetadataHeadInjector), - Arc::new(ConflictingMetadataHeadInjector), - ], - ); - - assert_eq!( - registry.tsjs_script_tag_attributes(), - vec![ - ("data-ts-gam-attribution", "true"), - ("data-test-order", "second"), - ("data-third-attribute", "third"), - ], - "should keep the first value for duplicate names and preserve attribute order" - ); - } - // Mock integration proxy for testing struct MockProxy; @@ -3476,76 +3521,6 @@ mod tests { ); } - struct CountingStreamFactory(&'static str); - - impl IntegrationHtmlStreamProcessorFactory for CountingStreamFactory { - fn integration_id(&self) -> &'static str { - self.0 - } - - fn create(&self, _context: IntegrationHtmlStreamContext) -> Box { - struct CountingStreamProcessor(usize); - - impl StreamProcessor for CountingStreamProcessor { - fn process_chunk( - &mut self, - chunk: &[u8], - _is_last: bool, - ) -> std::io::Result> { - self.0 += 1; - let mut output = self.0.to_string().into_bytes(); - output.extend_from_slice(chunk); - Ok(output) - } - } - - Box::new(CountingStreamProcessor(0)) - } - } - - #[test] - fn html_stream_factories_preserve_order_and_create_isolated_sessions() { - let registration = IntegrationRegistration::builder("test") - .with_html_stream_processor(Arc::new(CountingStreamFactory("first"))) - .with_html_stream_processor(Arc::new(CountingStreamFactory("second"))) - .build(); - let identifiers: Vec<_> = registration - .html_stream_processors - .iter() - .map(|factory| factory.integration_id()) - .collect(); - assert_eq!( - identifiers, - ["first", "second"], - "should preserve factory registration order", - ); - - let context = IntegrationHtmlStreamContext { - request_host: "proxy.example.com".to_owned(), - request_scheme: "https".to_owned(), - origin_host: "origin.example.com".to_owned(), - document_state: IntegrationDocumentState::default(), - }; - let factory = ®istration.html_stream_processors[0]; - let mut first = factory.create(context.clone()); - let mut second = factory.create(context); - - assert_eq!( - first - .process_chunk(b"a", false) - .expect("should process first session"), - b"1a", - "should initialize the first session counter", - ); - assert_eq!( - second - .process_chunk(b"b", true) - .expect("should process second session"), - b"1b", - "should initialize an independent second session counter", - ); - } - #[test] fn handle_proxy_passes_http_request_without_fastly_round_trip() { let settings = create_test_settings(); @@ -4932,33 +4907,20 @@ mod tests { ); } - /// Writes one fixed insert, so a test can read the order hooks ran in. - struct FixedHeadInsert { - id: &'static str, - insert: &'static str, - } - - impl IntegrationHeadInjector for FixedHeadInsert { - fn integration_id(&self) -> &'static str { - self.id - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - vec![self.insert.to_owned()] - } - } - - const SECTION_INSERT: &str = ""; - const PLAN_INSERT: &str = ""; + // Each probe supplies a middleware that changes nothing, so a test can + // read the order registrations were made in from the names the registry + // lists. + const SECTION_MIDDLEWARE: &str = "testing.probe-section"; + const PLAN_MIDDLEWARE: &str = "testing.probe-plan"; fn section_probe_registration( _settings: &Settings, ) -> Result, Report> { Ok(Some( IntegrationRegistration::builder("probe-section") - .with_head_injector(Arc::new(FixedHeadInsert { - id: "probe-section", - insert: SECTION_INSERT, + .with_middleware(Arc::new(Named { + id: SECTION_MIDDLEWARE, + phases: FETCH_ONLY, })) .build(), )) @@ -4972,9 +4934,9 @@ mod tests { ) -> Result, Report> { Ok(plan.enabled().then(|| { IntegrationRegistration::builder("probe-plan") - .with_head_injector(Arc::new(FixedHeadInsert { - id: "probe-plan", - insert: PLAN_INSERT, + .with_middleware(Arc::new(Named { + id: PLAN_MIDDLEWARE, + phases: FETCH_ONLY, })) .build() })) @@ -5002,17 +4964,11 @@ mod tests { .with_module_name("testing.probe-plan") .with_plan_registration(plan_probe_registration), ]; - let probe_inserts = |registry: &IntegrationRegistry| { - let document_state = IntegrationDocumentState::default(); + let probe_middleware = |registry: &IntegrationRegistry| { registry - .head_inserts(&IntegrationHtmlContext { - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - document_state: &document_state, - }) + .middleware_ids() .into_iter() - .filter(|insert| insert == SECTION_INSERT || insert == PLAN_INSERT) + .filter(|name| *name == SECTION_MIDDLEWARE || *name == PLAN_MIDDLEWARE) .collect::>() }; @@ -5023,7 +4979,7 @@ mod tests { !registry.integration_runs("probe-plan"), "should register nothing where the function finds nothing in the plan" ); - assert_eq!(probe_inserts(®istry), vec![SECTION_INSERT]); + assert_eq!(probe_middleware(®istry), vec![SECTION_MIDDLEWARE]); settings.auction.enabled = true; let registry = IntegrationRegistry::with_registrations(&settings, &extra) @@ -5033,9 +4989,9 @@ mod tests { "should register from the plan a module no section selects" ); assert_eq!( - probe_inserts(®istry), - vec![PLAN_INSERT, SECTION_INSERT], - "should run the hooks of what the plan registered ahead of a section's module" + probe_middleware(®istry), + vec![PLAN_MIDDLEWARE, SECTION_MIDDLEWARE], + "should register what the plan supplies ahead of a section's module" ); } @@ -5888,4 +5844,503 @@ mod tests { "error should name the module and both integrations: {message}" ); } + + fn html_entry(path: Option<&str>, names: &[&str]) -> PhaseEntry { + PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + } + } + + /// Settings that select core's middleware stand-in and hold `entries`. + fn fixture_settings_with_entries(entries: Vec) -> Settings { + let mut settings = settings_naming(fixture::MODULE); + settings.fetch = PhaseEntries::new(entries); + settings + } + + #[test] + fn a_selected_module_s_middleware_run_where_an_entry_names_them() { + let settings = fixture_settings_with_entries(vec![ + html_entry(Some("/news/"), &[fixture::LINKS, fixture::HEAD]), + html_entry(None, &[fixture::HEAD]), + ]); + + let registry = IntegrationRegistry::new(&settings) + .expect("should build a registry whose entries name registered middleware"); + + assert_eq!( + registry.middleware_ids(), + [ + fixture::HEAD, + fixture::LINKS, + fixture::BROKEN, + fixture::READER, + fixture::BROKEN_READER, + ], + "should list the middleware the selected module supplies" + ); + assert_eq!( + registry.middleware_in(MiddlewarePhase::Fetch), + [fixture::HEAD, fixture::LINKS, fixture::BROKEN], + "should list the middleware that run in the fetch phase" + ); + assert_eq!( + registry.middleware_in(MiddlewarePhase::Serve), + [fixture::READER, fixture::BROKEN_READER], + "should list the middleware that run in the serve phase" + ); + let chain_for = |media_type: &str, path: &str| { + registry + .middleware_chain(&settings.fetch, MiddlewarePhase::Fetch, media_type, path) + .ids() + }; + assert_eq!( + chain_for(HTML_MEDIA_TYPE, "/news/today"), + [fixture::LINKS, fixture::HEAD], + "should run the first covering entry's middleware in the order it names them" + ); + assert_eq!(chain_for(HTML_MEDIA_TYPE, "/"), [fixture::HEAD]); + assert!( + chain_for("text/css", "/news/site.css").is_empty(), + "should run nothing on a media type no entry covers" + ); + } + + #[test] + fn a_module_no_section_selects_supplies_no_middleware() { + let settings = crate::test_support::tests::create_test_settings(); + + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + + assert!( + !registry.middleware_ids().contains(&fixture::HEAD), + "should register nothing of a module that does not run" + ); + } + + #[test] + fn an_entry_naming_a_middleware_nothing_supplies_is_refused_with_what_is_supplied() { + let settings = fixture_settings_with_entries(vec![ + html_entry(Some("/news/"), &[fixture::HEAD]), + html_entry(None, &[fixture::HEAD, "testing.nothing"]), + ]); + + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse an entry naming a middleware nothing supplies"); + + let message = error.to_string(); + assert!( + message.contains( + "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies" + ) && message.contains(&format!( + "[{}, {}, {}, {}, {}]", + fixture::HEAD, + fixture::LINKS, + fixture::BROKEN, + fixture::READER, + fixture::BROKEN_READER + )), + "should name the entry and list what could be named: {message}" + ); + assert!( + !message.contains("no section selects"), + "should not blame a selection when the name is no module's: {message}" + ); + } + + #[test] + fn an_entry_naming_the_middleware_of_a_module_no_section_selects_says_so() { + let mut settings = crate::test_support::tests::create_test_settings(); + settings.fetch = PhaseEntries::new(vec![html_entry(None, &[fixture::LINKS])]); + + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse an entry naming the middleware of a module that does not run"); + + let message = error.to_string(); + assert!( + message.contains(&format!("[[fetch]] entry 1 names `{}`", fixture::LINKS)) + && message.contains(&format!( + "`{}` is a module no section selects, so nothing it supplies is running", + fixture::MODULE + )), + "should say the module is not selected: {message}" + ); + } + + #[test] + fn a_middleware_no_entry_names_is_listed_for_the_startup_warning() { + let unnamed = |entries: Vec| -> Vec<&'static str> { + let settings = fixture_settings_with_entries(entries); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + unnamed_middleware(&settings, ®istry.inner) + .into_iter() + .map(|(integration, middleware)| { + assert_eq!(integration, fixture::ID, "should name the supplier"); + middleware.middleware_id() + }) + .collect() + }; + + assert_eq!( + unnamed(Vec::new()), + [ + fixture::HEAD, + fixture::LINKS, + fixture::BROKEN, + fixture::READER, + fixture::BROKEN_READER, + ], + "should list every middleware when there are no entries" + ); + assert_eq!( + unnamed(vec![ + html_entry(Some("/news/"), &[fixture::LINKS]), + html_entry(None, &[fixture::HEAD]), + ]), + [fixture::BROKEN, fixture::READER, fixture::BROKEN_READER], + "should leave out a middleware any entry names" + ); + } + + #[test] + fn an_entry_naming_a_middleware_in_a_phase_it_does_not_run_in_is_refused() { + let mut settings = settings_naming(fixture::MODULE); + settings.serve = PhaseEntries::new(vec![html_entry(None, &[fixture::HEAD])]); + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse a fetch middleware named in a serve entry"); + let message = error.to_string(); + assert!( + message.contains(&format!("[[serve]] entry 1 names `{}`", fixture::HEAD)) + && message.contains("does not run in that phase. It runs in [[fetch]]"), + "should say which phase the middleware runs in: {message}" + ); + + let mut settings = settings_naming(fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![html_entry(None, &[fixture::READER])]); + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse a serve middleware named in a fetch entry"); + let message = error.to_string(); + assert!( + message.contains(&format!("[[fetch]] entry 1 names `{}`", fixture::READER)) + && message.contains("does not run in that phase. It runs in [[serve]]"), + "should say which phase the middleware runs in: {message}" + ); + } + + #[test] + fn each_phase_has_entries_of_its_own() { + let mut settings = settings_naming(fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![html_entry(None, &[fixture::HEAD])]); + settings.serve = PhaseEntries::new(vec![html_entry(Some("/news/"), &[fixture::READER])]); + let registry = IntegrationRegistry::new(&settings) + .expect("should build a registry with an entry in each phase"); + + let chain_for = |phase: MiddlewarePhase, path: &str| { + registry + .middleware_chain(settings.phase_entries(phase), phase, HTML_MEDIA_TYPE, path) + .ids() + }; + assert_eq!( + chain_for(MiddlewarePhase::Fetch, "/news/today"), + [fixture::HEAD] + ); + assert_eq!( + chain_for(MiddlewarePhase::Serve, "/news/today"), + [fixture::READER] + ); + assert!( + chain_for(MiddlewarePhase::Serve, "/sport/today").is_empty(), + "should run no serve middleware on a path only the fetch entries cover" + ); + let unnamed: Vec<&str> = unnamed_middleware(&settings, ®istry.inner) + .into_iter() + .map(|(_, middleware)| middleware.middleware_id()) + .collect(); + assert_eq!( + unnamed, + [fixture::LINKS, fixture::BROKEN, fixture::BROKEN_READER], + "should count a middleware as named only by an entry of a phase it runs in" + ); + } + + /// A middleware under a name of the test's choosing, running in the + /// phases given. + struct Named { + id: &'static str, + phases: &'static [MiddlewarePhase], + } + + impl Middleware for Named { + fn middleware_id(&self) -> &'static str { + self.id + } + + fn phases(&self) -> &[MiddlewarePhase] { + self.phases + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction::pass() + } + } + + const FETCH_ONLY: &[MiddlewarePhase] = &[MiddlewarePhase::Fetch]; + + fn first_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("first_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "testing.shared-name", + phases: FETCH_ONLY, + })) + .build(), + )) + } + + fn second_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("second_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "testing.shared-name", + phases: FETCH_ONLY, + })) + .build(), + )) + } + + fn misnamed_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("misnamed_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "Not A Name", + phases: FETCH_ONLY, + })) + .build(), + )) + } + + fn phaseless_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("phaseless_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "testing.phaseless", + phases: &[], + })) + .build(), + )) + } + + /// A builder for `register`, selected in `settings` under a module name + /// made from its id. + fn selected_supplier( + settings: &mut Settings, + id: &'static str, + module: &'static str, + register: crate::integrations::IntegrationBuilderFn, + ) -> crate::integrations::IntegrationBuilder { + settings.select_module("testing", module); + crate::integrations::IntegrationBuilder::new( + id, + "a-vendor-crate", + register, + validate_nothing, + ) + .with_module_name(module) + } + + #[test] + fn two_modules_supplying_one_middleware_name_are_refused() { + let mut settings = crate::test_support::tests::create_test_settings(); + let extra = [ + selected_supplier( + &mut settings, + "first_supplier", + "testing.first-supplier", + first_supplier, + ), + selected_supplier( + &mut settings, + "second_supplier", + "testing.second-supplier", + second_supplier, + ), + ]; + + let error = IntegrationRegistry::with_registrations(&settings, &extra) + .err() + .expect("should refuse one middleware name supplied twice"); + + let message = error.to_string(); + assert!( + message.contains("`first_supplier` and `second_supplier` both supply a middleware") + && message.contains("`testing.shared-name`"), + "should name both suppliers and the name: {message}" + ); + } + + fn marking_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("marking") + .without_js() + .with_bundle_tag_attribute("data-example-mode", "first") + .with_bundle_tag_attribute("data-example-flag", "true") + .build(), + )) + } + + fn disagreeing_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("disagreeing") + .without_js() + .with_bundle_tag_attribute("data-example-mode", "second") + .with_bundle_tag_attribute("data-example-other", "kept") + .build(), + )) + } + + fn unsafe_name_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("unsafe_name") + .without_js() + .with_bundle_tag_attribute("onload=alert(1) data-x", "true") + .build(), + )) + } + + fn unsafe_value_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("unsafe_value") + .without_js() + .with_bundle_tag_attribute("data-example-mode", "\"> + "#; + + let mut plain = create_html_processor(config()); + let without = String::from_utf8( + crate::streaming_processor::StreamProcessor::process_chunk( + &mut plain, + html.as_bytes(), + true, + ) + .expect("should process the document"), + ) + .expect("should leave the document UTF-8"); + + assert_eq!(page(&chain_of(Vec::new()), html), without); +} + +#[test] +fn head_markup_goes_ahead_of_the_bundle_in_the_order_the_middleware_are_named() { + let chain = chain_of(vec![ + head("example.first", ""), + head("example.second", ""), + ]); + + let page = page(&chain, PAGE); + + let first = index_of(&page, "name=\"first\""); + let second = index_of(&page, "name=\"second\""); + let bundle = index_of(&page, BUNDLE_TAG); + let title = index_of(&page, ""); + assert!( + first < second && second < bundle && bundle < title, + "should write head markup in the order named, ahead of the bundle and of the \ + page's own head: {page}" + ); + assert_eq!( + page.matches("name=\"first\"").count(), + 1, + "should write head markup once" + ); +} + +#[test] +fn markup_for_after_the_bundle_goes_between_the_bundle_and_the_page_s_own_head() { + let chain = chain_of(vec![middleware("example.after", |_| MiddlewareAction { + after_bundle_inserts: vec!["<script id=\"after\"></script>".to_owned()], + ..MiddlewareAction::pass() + })]); + + let page = page(&chain, PAGE); + + let bundle = index_of(&page, BUNDLE_TAG); + let after = index_of(&page, "id=\"after\""); + let title = index_of(&page, "<title>"); + assert!( + bundle < after && after < title, + "should write the markup straight after the bundle: {page}" + ); +} + +#[test] +fn a_page_with_no_head_gets_no_head_markup() { + let chain = chain_of(vec![head("example.first", "<meta name=\"first\">")]); + + let page = page(&chain, "<p>a fragment</p>"); + + assert_eq!( + page, "<p>a fragment</p>", + "should leave a document with no head as it arrived" + ); +} + +#[test] +fn an_element_handler_is_asked_about_an_address_as_it_will_be_served() { + let chain = chain_of(vec![element("example.seen", "a.x", "href", |matched| { + AttributeRewriteAction::replace(format!( + "seen:{}:{}:{}", + matched.element_name, matched.attribute_name, matched.value + )) + })]); + + let page = page( + &chain, + r#"<html><body><a class="x" href="https://origin.example.com/page">x</a><a href="https://origin.example.com/other">y</a></body></html>"#, + ); + + assert!( + page.contains(r#"<a class="x" href="seen:a:href:https://test.example.com/page">"#), + "should hand the handler the address after core moved it to the reader's host: {page}" + ); + assert!( + page.contains(r#"<a href="https://test.example.com/other">"#), + "should leave an element the selector does not match to core: {page}" + ); +} + +#[test] +fn one_decision_can_be_asked_about_each_of_several_attributes() { + let chain = chain_of(vec![middleware("example.each", |_| { + let decide: Rc<AttributeRewriteFn> = Rc::new(|matched| { + if matched.value == "/vendor.js" { + AttributeRewriteAction::replace(format!( + "/first-party.js#{}", + matched.attribute_name + )) + } else { + AttributeRewriteAction::keep() + } + }); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } + })]); + + let page = page( + &chain, + r#"<html><body><script src="/vendor.js"></script><link href="/vendor.js"><img src="/other.png"><a data-src="/vendor.js">x</a></body></html>"#, + ); + + assert!( + page.contains(r#"<script src="/first-party.js#src"></script>"#) + && page.contains(r#"<link href="/first-party.js#href">"#), + "should ask the one decision about each attribute named: {page}" + ); + assert!( + page.contains(r#"<img src="/other.png">"#) && page.contains(r#"<a data-src="/vendor.js">"#), + "should leave another value and another attribute alone: {page}" + ); +} + +#[test] +fn a_module_s_tests_can_call_a_middleware_with_a_test_page_s_context() { + use crate::html_processor::test_support::{ORIGIN_HOST, REQUEST_HOST}; + + let document_state = crate::integrations::IntegrationDocumentState::default(); + let context = test_support::context(MiddlewarePhase::Serve, &document_state); + + assert_eq!(context.phase, MiddlewarePhase::Serve); + assert_eq!( + ( + context.request_scheme, + context.request_host, + context.origin_host + ), + ("https", REQUEST_HOST, ORIGIN_HOST), + "should describe the page the recording helpers serve" + ); +} + +#[test] +fn an_element_handler_can_remove_the_element() { + let chain = chain_of(vec![element("example.remove", "a.gone", "href", |_| { + AttributeRewriteAction::remove_element() + })]); + + let page = page( + &chain, + r#"<html><body><a class="gone" href="/x">removed</a><a class="gone">no address</a><a href="/y">kept</a></body></html>"#, + ); + + assert!( + !page.contains("removed"), + "should remove the element and what it holds: {page}" + ); + assert!( + page.contains(r#"<a class="gone">no address</a>"#), + "should leave an element that carries no such attribute as it is: {page}" + ); + assert!(page.contains(r#"<a href="/y">kept</a>"#)); +} + +#[test] +fn handlers_on_one_element_are_asked_in_order_each_seeing_the_last_value() { + let chain = chain_of(vec![ + element("example.one", "a", "href", |_| { + AttributeRewriteAction::replace("/one") + }), + element("example.two", "a[href]", "href", |matched| { + AttributeRewriteAction::replace(format!("{}/two", matched.value)) + }), + ]); + + let page = page( + &chain, + r#"<html><body><a href="/start">x</a></body></html>"#, + ); + + assert!( + page.contains(r#"<a href="/one/two">"#), + "should hand the second handler the value the first left: {page}" + ); +} + +#[test] +fn an_element_an_earlier_handler_removed_is_not_asked_about_again() { + let asked = Arc::new(AtomicUsize::new(0)); + let counted = Arc::clone(&asked); + let chain = chain_of(vec![ + element("example.remove", "a", "href", |_| { + AttributeRewriteAction::remove_element() + }), + element("example.count", "a", "href", move |_| { + counted.fetch_add(1, Ordering::SeqCst); + AttributeRewriteAction::keep() + }), + ]); + + let page = page(&chain, r#"<html><body><a href="/x">x</a></body></html>"#); + + assert!(!page.contains("<a "), "should remove the element: {page}"); + assert_eq!( + asked.load(Ordering::SeqCst), + 0, + "should not ask a later handler about an element that is already gone" + ); +} + +#[test] +fn a_text_handler_is_handed_the_text_as_the_one_before_it_left_it() { + let chain = chain_of(vec![ + text("example.one", "script.x", || { + |text: &str, _is_last: bool| { + if text.contains("alpha") { + ScriptRewriteAction::replace(text.replace("alpha", "beta")) + } else { + ScriptRewriteAction::keep() + } + } + }), + text("example.two", "script", || { + |text: &str, _is_last: bool| { + if text.contains("beta") { + ScriptRewriteAction::replace(text.replace("beta", "gamma")) + } else { + ScriptRewriteAction::keep() + } + } + }), + ]); + + let page = page( + &chain, + r#"<html><body><script class="x">var a = "alpha";</script><script>var b = "alpha";</script></body></html>"#, + ); + + assert!( + page.contains(r#"<script class="x">var a = "gamma";</script>"#), + "should hand the second handler what the first wrote: {page}" + ); + assert!( + page.contains(r#"<script>var b = "alpha";</script>"#), + "should leave a script only the second matches, which holds nothing it changes: {page}" + ); +} + +#[test] +fn text_a_handler_removed_is_empty_for_the_next_which_may_write_in_its_place() { + let chain = chain_of(vec![ + text("example.remove", "script", || { + |_text: &str, _is_last: bool| ScriptRewriteAction::remove_node() + }), + text("example.write", "script", || { + |text: &str, is_last: bool| { + assert_eq!(text, "", "should hand over nothing where text was removed"); + if is_last { + ScriptRewriteAction::replace("written()") + } else { + ScriptRewriteAction::keep() + } + } + }), + ]); + + let page = page( + &chain, + "<html><body><script>removed()</script></body></html>", + ); + + assert!( + page.contains("<script>written()</script>"), + "should write the later handler's text in place of what was removed: {page}" + ); +} + +#[test] +fn a_text_handler_s_replacement_is_written_as_it_is() { + let chain = chain_of(vec![text("example.script", "script", || { + |text: &str, _is_last: bool| { + if text.is_empty() { + ScriptRewriteAction::keep() + } else { + ScriptRewriteAction::replace("if (a && b < c || d > e) { go(\"?x=1&y=2\"); }") + } + } + })]); + + let page = page(&chain, "<html><body><script>old()</script></body></html>"); + + assert!( + page.contains("<script>if (a && b < c || d > e) { go(\"?x=1&y=2\"); }</script>"), + "should write a script's text with no character escaped: {page}" + ); +} + +#[test] +fn a_handler_that_holds_a_script_writes_it_back_whole() { + // The script reaches the handler in pieces, because the document is read + // seven bytes at a time, and the word to change lies across pieces. + let chain = chain_of(vec![text("example.hold", "script", || { + let mut held = String::new(); + move |text: &str, is_last: bool| { + held.push_str(text); + if !is_last { + return ScriptRewriteAction::remove_node(); + } + ScriptRewriteAction::replace(std::mem::take(&mut held).replace("ORIGINAL", "CHANGED")) + } + })]); + + let page = page_with( + config(), + &chain, + "<html><body><script>one(); two(\"ORIGINAL\"); three();</script><p>after</p></body></html>", + 7, + ); + + assert!( + page.contains("<script>one(); two(\"CHANGED\"); three();</script><p>after</p>"), + "should write the whole script once, in order, with the word changed: {page}" + ); +} + +const SCRIPT_PAGE: &str = "<html><body><script>one(); two();</script><p>after</p></body></html>"; + +/// A middleware that writes `script` as the text of every script. +fn writes_script(script: &str) -> Arc<dyn Middleware> { + let script = script.to_owned(); + text("example.script", "script", move || { + let script = script.clone(); + move |_text: &str, is_last: bool| { + if is_last { + ScriptRewriteAction::replace(script.clone()) + } else { + ScriptRewriteAction::remove_node() + } + } + }) +} + +/// The text of each script in `page`, and the name of each element, as a +/// browser's tokenizer reads the page. +fn as_a_browser_reads(page: &str) -> (Vec<String>, Vec<String>) { + let scripts = std::cell::RefCell::new((String::new(), Vec::new())); + let elements = std::cell::RefCell::new(Vec::new()); + lol_html::rewrite_str( + page, + lol_html::RewriteStrSettings { + element_content_handlers: vec![ + lol_html::element!("*", |element| { + elements.borrow_mut().push(element.tag_name()); + Ok(()) + }), + lol_html::text!("script", |text| { + let mut scripts = scripts.borrow_mut(); + let (script, read) = &mut *scripts; + script.push_str(text.as_str()); + if text.last_in_text_node() { + read.push(std::mem::take(script)); + } + Ok(()) + }), + ], + ..lol_html::RewriteStrSettings::new() + }, + ) + .expect("should read the page"); + (scripts.into_inner().1, elements.into_inner()) +} + +/// Asserts that a browser reads `page` as one script holding `script`, +/// followed by the paragraph, and no other element. +fn assert_script_stays_whole(page: &str, script: &str) { + let (scripts, elements) = as_a_browser_reads(page); + assert_eq!( + scripts, + [script], + "should read the script's text as written, up to its own end tag: {page}" + ); + assert_eq!( + elements, + ["html", "body", "script", "p"], + "should read what follows the script as markup, and nothing in it: {page}" + ); +} + +#[test] +fn script_text_that_would_end_the_script_is_written_so_the_script_stays_whole() { + for (written, expected) in [ + ( + "var s = \"</script><img src=x onerror=alert(1)>\";", + "var s = \"\\u003c/script><img src=x onerror=alert(1)>\";", + ), + ( + "var s = \"</SCRIPT ><img src=x onerror=alert(1)>\";", + "var s = \"\\u003c/SCRIPT ><img src=x onerror=alert(1)>\";", + ), + ] { + let page = page(&chain_of(vec![writes_script(written)]), SCRIPT_PAGE); + + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should write the `<` of the closing tag as `\\u003c`: {page}" + ); + assert_script_stays_whole(&page, expected); + } +} + +#[test] +fn script_text_with_a_comment_opening_leaves_the_end_tag_closing_the_script() { + // After `<!--`, `<script` would keep the script's own end tag from + // closing it, and the rest of the page would be read as script. + for (written, expected) in [ + ("var s = \"<!--\";", "var s = \"<!--\";"), + ( + "var s = \"<!--<script>\";", + "var s = \"<!--\\u003cscript>\";", + ), + ] { + let page = page(&chain_of(vec![writes_script(written)]), SCRIPT_PAGE); + + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should write `<!--` as it is, and the `<` of a script tag after it as \ + `\\u003c`: {page}" + ); + assert_script_stays_whole(&page, expected); + } +} + +#[test] +fn script_text_written_in_pieces_cannot_end_the_script_between_them() { + // The page is read a byte at a time, so the script's text arrives in + // several chunks. The handler writes the `<` of a closing tag in place + // of the first chunk and the rest of the tag in place of the last. + let chain = chain_of(vec![text("example.pieces", "script", || { + let mut first = true; + move |_text: &str, is_last: bool| { + if is_last { + ScriptRewriteAction::replace("/script><img src=x onerror=alert(1)>") + } else if std::mem::take(&mut first) { + ScriptRewriteAction::replace("<") + } else { + ScriptRewriteAction::remove_node() + } + } + })]); + + let page = page_with(config(), &chain, SCRIPT_PAGE, 1); + + let expected = "\\u003c/script><img src=x onerror=alert(1)>"; + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should escape a closing tag whose `<` was written with an earlier chunk: {page}" + ); + assert_script_stays_whole(&page, expected); +} + +#[test] +fn script_text_a_serve_middleware_writes_cannot_end_the_script_either() { + let chain = serve_chain_of(vec![writes_script( + "var s = \"</script><img src=x onerror=alert(1)>\";", + )]); + + let page = served(&chain, SCRIPT_PAGE); + + let expected = "var s = \"\\u003c/script><img src=x onerror=alert(1)>\";"; + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should keep the text a serve middleware writes inside its script: {page}" + ); + assert_script_stays_whole(&page, expected); +} + +#[test] +fn a_script_no_middleware_changes_is_written_as_the_page_wrote_it() { + // An old page hides a script inside `<!--` and writes a script tag from + // within it, which the parser reads as the script's own text. + let html = "<html><body><script><!--\ndocument.write('<script src=\"a.js\"></script>');\n\ + //--></script><p>after</p></body></html>"; + let chain = chain_of(vec![text("example.keep", "script", || { + |_text: &str, _is_last: bool| ScriptRewriteAction::keep() + })]); + + for chunk_size in [1, 7, 8192] { + assert_eq!( + page_with(config(), &chain, html, chunk_size), + html, + "should leave a script no middleware changed byte for byte, read {chunk_size} \ + bytes at a time" + ); + } +} + +/// Appends a comment to the document, and swaps one word for another. +struct Rewording { + from: &'static str, + to: &'static str, + comment: &'static str, +} + +impl crate::streaming_processor::StreamProcessor for Rewording { + fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> io::Result<Vec<u8>> { + let mut text = String::from_utf8_lossy(chunk).replace(self.from, self.to); + if is_last { + text.push_str(self.comment); + } + Ok(text.into_bytes()) + } +} + +#[test] +fn a_stream_processor_is_handed_the_document_the_handlers_left() { + let chain = chain_of(vec![ + middleware("example.first", |_| MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + "a", + "href", + Rc::new(|_| AttributeRewriteAction::replace("/from-handler")), + ))], + stream: Some(Box::new(Rewording { + from: "/from-handler", + to: "/from-first", + comment: "<!--first-->", + })), + ..MiddlewareAction::pass() + }), + middleware("example.second", |_| MiddlewareAction { + stream: Some(Box::new(Rewording { + from: "/from-first", + to: "/from-second", + comment: "<!--second-->", + })), + ..MiddlewareAction::pass() + }), + ]); + + let page = page( + &chain, + r#"<html><body><a href="/start">x</a></body></html>"#, + ); + + assert!( + page.contains(r#"<a href="/from-second">"#), + "should hand each processor what the handlers and the processor before it left: {page}" + ); + assert!( + page.ends_with("<!--first--><!--second-->"), + "should run the processors in the order their middleware are named: {page}" + ); +} + +/// What one middleware leaves in the document's state for another. +struct Shared(&'static str); + +#[test] +fn the_middleware_of_one_document_share_its_state_and_see_nothing_a_request_left() { + const KEY: &str = "example"; + let chain = chain_of(vec![ + middleware("example.leaves", |context| { + context + .document_state + .get_or_insert_with(KEY, || Shared("left by the first")); + MiddlewareAction::pass() + }), + middleware("example.reads", |context| { + let shared = context + .document_state + .get::<Shared>(KEY) + .map_or("nothing", |shared| shared.0); + let request_mark = context + .document_state + .get::<request_fixture::Mark>(request_fixture::ID) + .is_some(); + MiddlewareAction { + head_inserts: vec![format!( + "<meta name=\"state\" content=\"{shared}; request mark {request_mark}\">" + )], + ..MiddlewareAction::pass() + } + }), + ]); + // The request left a mark, which a serve middleware may read. + let config = config().with_request_state(request_fixture::marked()); + + let page = page_with(config, &chain, PAGE, 8192); + + assert!( + page.contains("content=\"left by the first; request mark false\""), + "should share the document's state along the chain, with nothing of the request in \ + it: {page}" + ); +} + +#[test] +fn each_document_is_planned_afresh() { + let created = Arc::new(AtomicUsize::new(0)); + let counted = Arc::clone(&created); + let chain = chain_of(vec![middleware("example.counts", move |context| { + let nth = counted.fetch_add(1, Ordering::SeqCst) + 1; + let earlier = context.document_state.get::<Shared>("example").is_some(); + context + .document_state + .get_or_insert_with("example", || Shared("this document")); + MiddlewareAction { + head_inserts: vec![format!( + "<meta name=\"nth\" content=\"{nth}; earlier state {earlier}\">" + )], + ..MiddlewareAction::pass() + } + })]); + + let first = page(&chain, PAGE); + let second = page(&chain, PAGE); + + assert!(first.contains("content=\"1; earlier state false\"")); + assert!( + second.contains("content=\"2; earlier state false\""), + "should ask the middleware again for the next document, with a state of its own: \ + {second}" + ); + assert_eq!(created.load(Ordering::SeqCst), 2); +} + +#[test] +fn a_middleware_is_told_the_phase_the_hosts_and_how_much_it_may_hold() { + let chain = chain_of(vec![middleware("example.context", |context| { + MiddlewareAction { + head_inserts: vec![format!( + "<meta name=\"context\" content=\"{} {}://{} from {} holding {}\">", + context.phase, + context.request_scheme, + context.request_host, + context.origin_host, + context.max_buffered_script_bytes + )], + ..MiddlewareAction::pass() + } + })]); + let most = create_test_settings().publisher.max_buffered_body_bytes; + + let page = page(&chain, PAGE); + + assert!( + page.contains(&format!( + "content=\"fetch https://{REQUEST_HOST} from {ORIGIN_HOST} holding {most}\"" + )), + "should tell the middleware about the document: {page}" + ); +} + +#[test] +fn a_selector_that_does_not_parse_refuses_the_document_naming_the_middleware() { + let chains = [ + chain_of(vec![ + head("example.fine", "<meta name=\"fine\">"), + element("example.broken", "a[", "href", |_| { + AttributeRewriteAction::keep() + }), + ]), + chain_of(vec![text("example.broken", "a[", || { + |_text: &str, _is_last: bool| ScriptRewriteAction::keep() + })]), + ]; + + for chain in chains { + let message = create_html_processor_with_middleware(config(), &chain, None) + .err() + .expect("should refuse to plan the document"); + + assert!( + message.contains("the fetch middleware `example.broken` cannot run") + && message.contains("`a[` is not a usable selector"), + "should name the middleware and the selector: {message}" + ); + } +} + +#[test] +fn a_chain_reports_its_phase_and_the_names_in_the_order_they_run() { + let chain = chain_of(vec![ + head("example.b", "<meta name=\"b\">"), + head("example.a", "<meta name=\"a\">"), + ]); + + assert_eq!(chain.phase(), MiddlewarePhase::Fetch); + assert_eq!(chain.ids(), ["example.b", "example.a"]); + assert!(!chain.is_empty()); + assert!(chain_of(Vec::new()).is_empty()); +} + +// --------------------------------------------------------------------------- +// What the document pipeline does with a serve chain +// --------------------------------------------------------------------------- + +/// [`PAGE`] as core leaves it for a store, with the script bundle in its +/// head and one link. +fn rewritten_page() -> String { + page( + &chain_of(Vec::new()), + r#"<html><head><title>Pagex"#, + ) +} + +#[test] +fn a_serve_chain_writes_head_markup_straight_before_the_bundle_and_the_rest_straight_after() { + let chain = serve_chain_of(vec![ + middleware("example.first", |_| MiddlewareAction { + head_inserts: vec!["".to_owned()], + after_bundle_inserts: vec!["".to_owned()], + ..MiddlewareAction::pass() + }), + middleware("example.second", |_| MiddlewareAction { + head_inserts: vec!["".to_owned()], + after_bundle_inserts: vec!["".to_owned()], + ..MiddlewareAction::pass() + }), + ]); + let rewritten = rewritten_page(); + + let page = served(&chain, &rewritten); + + assert!( + page.contains("" + ), + "should write the markup for after the bundle straight after it: {page}" + ); + assert_eq!( + page.matches(BUNDLE_TAG).count(), + 1, + "should write no second bundle: {page}" + ); + assert_eq!( + page.replace("", "") + .replace( + "", + "" + ), + rewritten, + "should change nothing else of a document core has already rewritten" + ); +} + +#[test] +fn a_serve_chain_writes_head_markup_once_where_the_page_repeats_the_bundle_s_id() { + let chain = serve_chain_of(vec![head("example.first", "")]); + let rewritten = rewritten_page().replace( + "", + "", + ); + + let page = served(&chain, &rewritten); + + assert_eq!( + page.matches("name=\"first\"").count(), + 1, + "should write head markup at core's tag alone: {page}" + ); + assert!( + index_of(&page, "name=\"first\"") < index_of(&page, ""), + "should write it in the head: {page}" + ); +} + +#[test] +fn a_reader_s_copy_with_no_bundle_gets_no_head_markup() { + let chain = serve_chain_of(vec![head("example.first", "<meta name=\"first\">")]); + + let page = served(&chain, "<p>a fragment</p>"); + + assert_eq!(page, "<p>a fragment</p>"); +} + +#[test] +fn a_serve_chain_s_handlers_work_on_the_page_as_core_left_it() { + let chain = serve_chain_of(vec![ + element("example.link", "a", "href", |matched| { + AttributeRewriteAction::replace(format!("{}?reader=1", matched.value)) + }), + text("example.script", "script", || { + |text: &str, _is_last: bool| { + if text.contains("var a = 1;") { + ScriptRewriteAction::replace(text.replace("var a = 1;", "var a = 2 && 3;")) + } else { + ScriptRewriteAction::keep() + } + } + }), + ]); + + let page = served(&chain, &rewritten_page()); + + assert!( + page.contains(r#"<a href="https://test.example.com/x?reader=1">"#), + "should hand an element handler the address core already moved: {page}" + ); + assert!( + page.contains("<script>var a = 2 && 3;</script>"), + "should write a text handler's replacement as it is: {page}" + ); +} + +#[test] +fn a_serve_chain_s_stream_processor_runs_over_the_reader_s_copy() { + let chain = serve_chain_of(vec![middleware("example.stream", |_| MiddlewareAction { + stream: Some(Box::new(Rewording { + from: "<title>Page", + to: "Reader", + comment: "", + })), + ..MiddlewareAction::pass() + })]); + + let page = served(&chain, &rewritten_page()); + + assert!(page.contains("Reader") && page.ends_with("")); +} + +#[test] +fn a_serve_middleware_reads_what_the_request_left() { + let chain = serve_chain_of(vec![middleware("example.reads", |context| { + let marked = context + .document_state + .get::(request_fixture::ID) + .is_some(); + MiddlewareAction { + head_inserts: vec![format!( + "", + context.phase, context.request_scheme, context.request_host, context.origin_host + )], + ..MiddlewareAction::pass() + } + })]); + let rewritten = rewritten_page(); + + let marked = served_with(&chain, &request_fixture::marked(), &rewritten, 8192); + let unmarked = served(&chain, &rewritten); + + let told = format!("serve https://{REQUEST_HOST} from {ORIGIN_HOST}"); + assert!( + marked.contains(&format!("content=\"{told}; marked true\"")), + "should hand a serve middleware what the request left: {marked}" + ); + assert!( + unmarked.contains(&format!("content=\"{told}; marked false\"")), + "should hand it nothing where the request left nothing: {unmarked}" + ); +} + +#[test] +fn a_serve_chain_with_nothing_to_do_makes_no_processor() { + let state = IntegrationRequestState::default(); + + let none = create_serve_processor(&serve_chain_of(Vec::new()), &reader(&state)) + .expect("should plan an empty chain"); + assert!(none.is_none(), "should make no processor for no middleware"); + + let passes = serve_chain_of(vec![middleware("example.pass", |_| { + MiddlewareAction::pass() + })]); + let none = + create_serve_processor(&passes, &reader(&state)).expect("should plan a chain that passes"); + assert!( + none.is_none(), + "should make no processor where every middleware leaves the page alone" + ); +} + +#[test] +fn a_serve_selector_that_does_not_parse_refuses_the_reader_s_copy() { + let chain = serve_chain_of(vec![element("example.broken", "a[", "href", |_| { + AttributeRewriteAction::keep() + })]); + + let message = create_serve_processor(&chain, &reader(&IntegrationRequestState::default())) + .err() + .expect("should refuse to plan the reader's copy"); + + assert!( + message.contains("the serve middleware `example.broken` cannot run") + && message.contains("`a[` is not a usable selector"), + "should name the phase, the middleware and the selector: {message}" + ); +} + +#[test] +fn on_a_page_that_is_not_stored_the_serve_chain_follows_the_fetch_chain_on_the_one_pass() { + let fetch = chain_of(vec![ + head("example.fetch-head", ""), + element("example.fetch-link", "a", "href", |_| { + AttributeRewriteAction::replace("/fetch") + }), + ]); + let serve = serve_chain_of(vec![ + head("example.serve-head", ""), + element("example.serve-link", "a", "href", |matched| { + AttributeRewriteAction::replace(format!("{}/serve", matched.value)) + }), + ]); + let html = + r#"Pagex"#; + + // Read whole, and then seven bytes at a time, so the second rewriter is + // fed the first one's output in pieces. + for chunk_size in [8192, 7] { + let processor = create_html_processor_with_middleware(config(), &fetch, Some(&serve)) + .expect("should plan both chains"); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + processor, + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should process the document"); + let page = String::from_utf8(output).expect("should leave the document UTF-8"); + + assert!( + page.contains(r#""#), + "should hand the serve chain the page as the fetch chain left it: {page}" + ); + let fetch_head = index_of(&page, "name=\"fetch\""); + let serve_head = index_of(&page, "name=\"serve\""); + assert!( + fetch_head < serve_head && serve_head < index_of(&page, BUNDLE_TAG), + "should write the serve chain's head markup after the fetch chain's and ahead of \ + the bundle: {page}" + ); + } +} + +// --------------------------------------------------------------------------- +// The settings +// --------------------------------------------------------------------------- + +fn settings_with_document(extra: &str) -> Result { + let document = format!( + "{}\n{extra}", + crate::test_support::tests::crate_test_settings_str() + ); + crate::settings::Settings::from_toml(&document).map_err(|error| format!("{error:?}")) +} + +#[test] +fn fetch_entries_are_read_from_the_settings_in_the_order_written() { + let settings = settings_with_document( + r#" +[[fetch]] +media_type = "text/html" +path = "/news/" +middleware = ["example.news", "example.all"] + +[[fetch]] +media_type = "text/html" +middleware = ["example.all"] +"#, + ) + .expect("should read settings with fetch entries"); + + assert_eq!( + settings.phase_entries(MiddlewarePhase::Fetch), + &PhaseEntries::new(vec![ + entry(Some("/news/"), &["example.news", "example.all"]), + entry(None, &["example.all"]), + ]) + ); +} + +#[test] +fn serve_entries_are_read_from_the_settings_beside_the_fetch_entries() { + let settings = settings_with_document( + r#" +[[fetch]] +media_type = "text/html" +middleware = ["example.all"] + +[[serve]] +media_type = "text/html" +path = "/news/" +middleware = ["example.reader"] +"#, + ) + .expect("should read settings with an entry in each phase"); + + assert_eq!( + settings.phase_entries(MiddlewarePhase::Fetch), + &PhaseEntries::new(vec![entry(None, &["example.all"])]) + ); + assert_eq!( + settings.phase_entries(MiddlewarePhase::Serve), + &PhaseEntries::new(vec![entry(Some("/news/"), &["example.reader"])]) + ); + let written = serde_json::to_value(&settings).expect("should write the settings"); + assert_eq!( + written.get("serve"), + Some(&json!([ + { "media_type": "text/html", "path": "/news/", "middleware": ["example.reader"] } + ])), + "should write the serve entries a document holds" + ); +} + +#[test] +fn settings_with_a_misshapen_serve_entry_are_refused_naming_the_phase() { + let message = + settings_with_document("\n[[serve]]\nmedia_type = \"text/html\"\nmiddleware = []\n") + .expect_err("should refuse the settings"); + + assert!( + message.contains("[[serve]] entry 1 names no middleware"), + "should name the phase and the entry: {message}" + ); +} + +#[test] +fn settings_without_entries_write_no_fetch_key() { + let settings = settings_with_document("").expect("should read settings with no entries"); + let written = serde_json::to_value(&settings).expect("should write the settings"); + assert!( + written.get("fetch").is_none() && written.get("serve").is_none(), + "should leave both keys out of a stored document that wrote no entries" + ); + + let settings = settings_with_document( + "\n[[fetch]]\nmedia_type = \"text/html\"\nmiddleware = [\"example.all\"]\n", + ) + .expect("should read settings with one entry"); + let written = serde_json::to_value(&settings).expect("should write the settings"); + assert_eq!( + written.get("fetch"), + Some(&json!([{ "media_type": "text/html", "middleware": ["example.all"] }])), + "should write the entries a document holds" + ); +} + +#[test] +fn settings_with_a_misshapen_entry_are_refused() { + let cases = [ + ( + "\n[[fetch]]\nmedia_type = \"text/css\"\nmiddleware = [\"example.all\"]\n", + "[[fetch]] entry 1 covers `text/css`", + ), + ( + "\n[fetch]\nmedia_type = \"text/html\"\nmiddleware = [\"example.all\"]\n", + "A phase is a list of entries, each written as [[fetch]]", + ), + ( + "\n[[fetch]]\nmedia_type = \"text/html\"\nmiddleware = [\"example.all\"]\nenabled = true\n", + "unknown field `enabled`", + ), + ( + "\n[[fetch]]\nmedia_type = \"text/html\"\n", + "missing field `middleware`", + ), + ]; + + for (document, expected) in cases { + let message = settings_with_document(document).expect_err("should refuse the settings"); + assert!( + message.contains(expected), + "should say `{expected}`: {message}" + ); + } +} + +#[test] +fn settings_with_an_entry_after_one_for_the_path_slash_are_refused() { + let message = settings_with_document( + r#" +[[fetch]] +media_type = "text/html" +path = "/" +middleware = ["example.root"] + +[[fetch]] +media_type = "text/html" +middleware = ["example.all"] +"#, + ) + .expect_err("should refuse settings with an entry that is never reached"); + + assert!( + message.contains("[[fetch]] entry 2 covers the same pages as [[fetch]] entry 1"), + "should name the entry that is never reached: {message}" + ); +} + +#[test] +fn deploy_validation_refuses_a_misshapen_entry() { + let mut settings = create_test_settings(); + settings.fetch = PhaseEntries::new(vec![ + entry(None, &["example.all"]), + entry(Some("/news/"), &["example.news"]), + ]); + + let error = crate::config::validate_settings_for_deploy(&settings) + .expect_err("should refuse entries with one that is never reached"); + + assert!( + error + .to_string() + .contains("[[fetch]] entry 2 is never reached"), + "should name the entry: {error}" + ); +} diff --git a/crates/trusted-server-core/src/platform/template_cache.rs b/crates/trusted-server-core/src/platform/template_cache.rs index 203bba1625..e5c9c1de5a 100644 --- a/crates/trusted-server-core/src/platform/template_cache.rs +++ b/crates/trusted-server-core/src/platform/template_cache.rs @@ -59,7 +59,7 @@ pub struct TemplateCacheKey { /// mutate first. pub url: String, /// Host and scheme. The post-processed output is host-dependent by construction: - /// both reach `IntegrationHtmlContext` and drive URL rewriting. + /// both are handed to every middleware and drive URL rewriting. pub request_host: String, /// See [`Self::request_host`]. pub request_scheme: String, diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 23c326d32c..0c5871d650 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -637,6 +637,8 @@ struct ProcessResponseParams<'a> { origin_url: &'a str, request_host: &'a str, request_scheme: &'a str, + /// See [`OwnedProcessResponseParams::request_path`]. + request_path: &'a str, settings: &'a Settings, content_type: &'a str, integration_registry: &'a IntegrationRegistry, @@ -683,6 +685,7 @@ impl PublisherBodyProcessor { origin_host: ¶ms.origin_host, request_host: ¶ms.request_host, request_scheme: ¶ms.request_scheme, + request_path: ¶ms.request_path, settings, integration_registry, permissions_script: permissions_script_for(params, settings), @@ -767,6 +770,7 @@ fn process_response_streaming( origin_host: params.origin_host, request_host: params.request_host, request_scheme: params.request_scheme, + request_path: params.request_path, settings: params.settings, integration_registry: params.integration_registry, permissions_script: params.permissions_script.map(str::to_string), @@ -1324,6 +1328,8 @@ struct HtmlStreamProcessorParams<'a> { origin_host: &'a str, request_host: &'a str, request_scheme: &'a str, + /// See [`OwnedProcessResponseParams::request_path`]. + request_path: &'a str, settings: &'a Settings, integration_registry: &'a IntegrationRegistry, /// Head script carrying this request's permission state, or [`None`] under a @@ -1331,8 +1337,8 @@ struct HtmlStreamProcessorParams<'a> { permissions_script: Option, ad_slots_script: Option, ad_bids_state: Arc>>, - /// What modules left on the request for their page hooks. See - /// [`template_request_state`]. + /// What modules left on the request, which the `[[serve]]` entry's + /// middleware read where that entry runs on this pass. request_state: IntegrationRequestState, /// Whether a shared template was authorized for this response. /// @@ -1345,25 +1351,6 @@ struct HtmlStreamProcessorParams<'a> { deferred_inline_marker: Option, } -/// The request state a document's page hooks may read. -/// -/// What a module leaves on a request is made for that one request, so it -/// must not reach a template that is stored and served to other readers. -/// -/// [`handle_publisher_request`] already keeps a request that carries any off -/// the shared template path, so a shared mode never sees one today. This -/// makes the same guarantee where the document is built, so that neither -/// place depends on the other. -pub(crate) fn template_request_state( - mode: AssemblyMode, - request_state: IntegrationRequestState, -) -> IntegrationRequestState { - match mode { - AssemblyMode::Inline => request_state, - AssemblyMode::Esi => IntegrationRequestState::default(), - } -} - /// The marker emitted at the `` seam under [`AssemblyMode::Esi`], reserving the /// place this reader's slots and bids are spliced into. /// @@ -1520,7 +1507,8 @@ fn deferred_inline_seam_token( fn create_html_stream_processor( params: HtmlStreamProcessorParams<'_>, ) -> Result, Report> { - use crate::html_processor::{HtmlProcessorConfig, create_html_processor}; + use crate::html_processor::{HtmlProcessorConfig, create_html_processor_with_middleware}; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; let config = HtmlProcessorConfig::from_settings( params.settings, @@ -1538,8 +1526,6 @@ fn create_html_stream_processor( _ => body_close_injection(assembly_mode, params.ad_slots_script.is_some()), }; - let request_state = template_request_state(assembly_mode, params.request_state); - // Only a response that can be stored has a consumer for the observation, so the // handlers are not registered for ordinary inline traffic. let csp_nonce_observed = params @@ -1550,11 +1536,100 @@ fn create_html_stream_processor( let config = config .with_permissions_script(params.permissions_script) .with_ad_state(params.ad_slots_script, params.ad_bids_state) - .with_request_state(request_state) + .with_request_state(params.request_state) .with_body_close(body_close) .with_csp_nonce_observer(csp_nonce_observed); - Ok(create_html_processor(config)) + // The middleware of the first [[fetch]] entry covering this path. + let fetch = params.integration_registry.middleware_chain( + ¶ms.settings.fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + params.request_path, + ); + // A response that may be stored leaves this processor as the template + // every reader shares, so its [[serve]] entry runs on each reader's copy + // once that copy is assembled. See `run_serve_pass`. + let serve = (!params.shared_template_authorized).then(|| { + params.integration_registry.middleware_chain( + ¶ms.settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + params.request_path, + ) + }); + create_html_processor_with_middleware(config, &fetch, serve.as_ref()) + .map_err(|message| Report::new(TrustedServerError::Configuration { message })) +} + +/// The processor that carries out the `[[serve]]` entry covering this request +/// on a reader's copy that is assembled and not yet encoded, or `None` when +/// no entry covers the request or its middleware have nothing to do. +/// +/// The origin host is read from the settings. A page served from the store +/// keeps no record of it, and a serve middleware is told the same about a +/// page from the store as about one fetched for this request. +fn reader_serve_processor( + params: &OwnedProcessResponseParams, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) -> Result>, Report> { + use crate::html_processor::{ReaderDocument, create_serve_processor}; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; + + if !is_html_content_type(¶ms.content_type) { + return Ok(None); + } + let chain = integration_registry.middleware_chain( + &settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + ¶ms.request_path, + ); + let origin_host = settings.publisher.origin_host(); + create_serve_processor( + &chain, + &ReaderDocument { + request_host: ¶ms.request_host, + request_scheme: ¶ms.request_scheme, + origin_host: &origin_host, + request_state: ¶ms.request_state, + max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, + }, + ) + .map_err(|message| Report::new(TrustedServerError::Configuration { message })) +} + +/// Runs the `[[serve]]` entry covering this request over a reader's whole +/// copy, assembled and not yet encoded. +fn run_serve_pass( + bytes: Vec, + params: &OwnedProcessResponseParams, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) -> Result, Report> { + let Some(mut processor) = reader_serve_processor(params, settings, integration_registry)? + else { + return Ok(bytes); + }; + processor + .process_chunk(&bytes, true) + .change_context(TrustedServerError::Proxy { + message: "a serve middleware could not rewrite the page".to_string(), + }) +} + +/// One piece of a streamed reader's copy through the `[[serve]]` entry, when +/// one covers the request. +fn serve_chunk( + serve: &mut Option>, + bytes: Vec, + is_last: bool, +) -> Result, std::io::Error> { + match serve { + Some(processor) => processor.process_chunk(&bytes, is_last), + None => Ok(bytes), + } } /// Result of publisher request handling, indicating whether the response body @@ -1738,6 +1813,9 @@ pub struct OwnedProcessResponseParams { pub(crate) origin_url: String, pub(crate) request_host: String, pub(crate) request_scheme: String, + /// The path the reader asked for, which decides the `[[fetch]]` and + /// `[[serve]]` entries that cover the document. + pub(crate) request_path: String, pub(crate) content_type: String, pub(crate) ad_slots_script: Option, pub(crate) ad_bids_state: AdBidsState, @@ -1751,7 +1829,8 @@ pub struct OwnedProcessResponseParams { pub(crate) dispatched_auction: Option, /// Price granularity used to bucket bids when building `tsjs.bids`. pub(crate) price_granularity: PriceGranularity, - /// What modules left on the request for their page hooks. + /// What modules left on the request, for their serve middleware and + /// their response finalizers. pub(crate) request_state: IntegrationRequestState, /// Set by the transform when the document carries a response-bound CSP nonce. /// @@ -1967,6 +2046,9 @@ pub async fn buffer_publisher_response_async( set_assembly_response_state(&mut response, state); } let bytes = if was_authorized { + // The [[serve]] entry runs here on a shared template, and + // inside the transform on a page that was never authorized. + let bytes = run_serve_pass(bytes, ¶ms, settings, integration_registry)?; encode_complete_body(bytes, response_compression(&response))? } else { bytes @@ -2016,6 +2098,7 @@ pub async fn buffer_publisher_response_async( assembled.extend_from_slice(head); assembled.extend_from_slice(seam.as_bytes()); assembled.extend_from_slice(tail); + let assembled = run_serve_pass(assembled, ¶ms, settings, integration_registry)?; let assembled = encode_complete_body(assembled, response_compression(&response))?; response.headers_mut().insert( http::header::CONTENT_LENGTH, @@ -2208,6 +2291,8 @@ fn build_template_assembly_params( origin_url: settings.publisher.origin_url.clone(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), + // Set by the caller, with the rest of what belongs to the request. + request_path: String::new(), content_type: entry.metadata.content_type.clone(), // The template already carries the head seam; re-injecting would duplicate it. permissions_json, @@ -2516,6 +2601,23 @@ pub async fn publisher_response_into_streaming_response( return Ok(response); } + // Planned before the lazy body is built, so a serve entry that + // cannot run fails the response while it still can be failed. + let mut serve = match reader_serve_processor(¶ms, &settings, integration_registry) { + Ok(serve) => serve, + Err(err) => { + if let Some(dispatched) = params.dispatched_auction.take() { + emit_abandoned_auction( + &services, + params.auction_observation.take(), + dispatched, + "processor_init_error", + ) + .await; + } + return Err(err); + } + }; let services = services.clone(); let settings = Arc::clone(&settings); let orchestrator = Arc::clone(&orchestrator); @@ -2539,8 +2641,10 @@ pub async fn publisher_response_into_streaming_response( let (head, tail) = split_template_at_seam(&template) .map_err(|e| std::io::Error::other(e.to_string()))?; let mut encoder = BodyStreamEncoder::new(compression); + // Each piece goes through the serve entry as it leaves, so + // the head is still written before the auction is awaited. let encoded_head = encoder - .encode_chunk(head.to_vec()) + .encode_chunk(serve_chunk(&mut serve, head.to_vec(), false)?) .map_err(publisher_stream_error)?; if !encoded_head.is_empty() { yield bytes::Bytes::from(encoded_head); @@ -2571,14 +2675,14 @@ pub async fn publisher_response_into_streaming_response( let seam = seam_script_for(¶ms); if !seam.is_empty() { let encoded_seam = encoder - .encode_chunk(seam.into_bytes()) + .encode_chunk(serve_chunk(&mut serve, seam.into_bytes(), false)?) .map_err(publisher_stream_error)?; if !encoded_seam.is_empty() { yield bytes::Bytes::from(encoded_seam); } } let encoded_tail = encoder - .encode_chunk(tail.to_vec()) + .encode_chunk(serve_chunk(&mut serve, tail.to_vec(), true)?) .map_err(publisher_stream_error)?; if !encoded_tail.is_empty() { yield bytes::Bytes::from(encoded_tail); @@ -2977,6 +3081,7 @@ pub fn stream_publisher_body( origin_url: ¶ms.origin_url, request_host: ¶ms.request_host, request_scheme: ¶ms.request_scheme, + request_path: ¶ms.request_path, settings, content_type: ¶ms.content_type, integration_registry, @@ -3083,6 +3188,7 @@ pub async fn stream_publisher_body_async( origin_host: ¶ms.origin_host, request_host: ¶ms.request_host, request_scheme: ¶ms.request_scheme, + request_path: ¶ms.request_path, settings, integration_registry, permissions_script: permissions_script_for(params, settings), @@ -5227,6 +5333,7 @@ pub async fn handle_publisher_request( ad_bids_state.clone(), permissions_json.clone(), ); + params.request_path.clone_from(&request_path); params.seam_ad_slots = seam_ad_slots.clone(); params.dispatched_auction = dispatched_auction.take(); params.auction_observation = auction_observation.take(); @@ -5617,6 +5724,7 @@ pub async fn handle_publisher_request( origin_url: settings.publisher.origin_url.clone(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), + request_path: request_path.clone(), content_type, permissions_json, ad_slots_script: ad_slots_script.clone(), @@ -7934,9 +8042,8 @@ mod tests { &[SCHEDULING_PROVIDER], ); settings.proxy.allowed_domains = vec!["*.example".to_owned(), "*.example.com".to_owned()]; - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); settings } @@ -9263,6 +9370,7 @@ mod tests { origin_url: settings.publisher.origin_url.clone(), request_host: settings.publisher.domain.clone(), request_scheme: "https".to_owned(), + request_path: "/".to_owned(), content_type: "application/json".to_owned(), permissions_json: String::new(), ad_slots_script: None, @@ -9315,9 +9423,8 @@ mod tests { #[test] fn stream_publisher_body_writes_what_a_module_left_on_the_request_into_materialized_html() { let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); let integration_registry = IntegrationRegistry::new(&settings).expect("should create integration registry"); @@ -9495,17 +9602,19 @@ mod tests { //! passing code: the decisions were right and nothing checked what the //! composition of them *renders*. //! - //! These tests render whole documents through `create_html_processor`, - //! composing the same three decisions `create_html_stream_processor` uses, - //! and compare bytes. A future request-dependent injection added at either - //! seam fails here even if every decision function is left untouched. + //! These tests render whole documents through the processor + //! `create_html_stream_processor` builds, composing the same decisions it + //! uses, and compare bytes. A future request-dependent injection added at + //! either seam fails here even if every decision function is left untouched. use super::template_neutrality_tests::{settings_with_slots, slot}; use super::*; use crate::creative_opportunities::AssemblyMode; - use crate::html_processor::{HtmlProcessorConfig, create_html_processor}; + use crate::html_processor::{HtmlProcessorConfig, create_html_processor_with_middleware}; use crate::integrations::IntegrationRegistry; + use crate::integrations::registry_test_support::middleware_fixture as fixture; use crate::integrations::registry_test_support::request_fixture; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; const DOCUMENT: &[u8] = b"t

content

"; @@ -9516,7 +9625,7 @@ mod tests { struct RequestShape { /// Folds in consent, bot classification, prefetch and the kill switch. ad_stack_ran: bool, - /// A module left state on the request for its page hooks. + /// A module left state on the request for its serve middleware. module_state_left: bool, /// A resolved auction, present only when one was dispatched. bids_available: bool, @@ -9528,22 +9637,29 @@ mod tests { let mut settings = settings_with_slots(); settings.proxy.allowed_domains = vec!["*.example".to_string(), "*.example.com".to_string()]; - // The stand-in is what writes a marked request's state into a - // document, so it runs for every shape. + // The request stand-in is what writes a marked request's state + // into a document, so it runs for every shape. The reader + // middleware beside it writes into every reader's copy, marked or + // not, so a template that ran the serve entry would show it. settings.select_module("testing", request_fixture::MODULE); + settings.select_module("testing", fixture::MODULE); + crate::html_processor::test_support::place_on_every_page( + &mut settings, + MiddlewarePhase::Serve, + &[request_fixture::MODULE, fixture::READER], + ); + let registry = IntegrationRegistry::new(&settings) + .expect("should build a registry that runs the stand-in"); let slots = [slot()]; let ad_slots_script = template_ad_slots_script(mode, shape.ad_stack_ran, &settings, &slots, "/"); let body_close = body_close_injection(mode, ad_slots_script.is_some()); - let request_state = template_request_state( - mode, - if shape.module_state_left { - request_fixture::marked() - } else { - IntegrationRequestState::default() - }, - ); + let request_state = if shape.module_state_left { + request_fixture::marked() + } else { + IntegrationRequestState::default() + }; let ad_bids_state = std::sync::Arc::new(std::sync::Mutex::new(shape.bids_available.then(|| { @@ -9555,8 +9671,7 @@ mod tests { origin_host: "origin.example.com".to_string(), request_host: "example.com".to_string(), request_scheme: "https".to_string(), - integrations: IntegrationRegistry::new(&settings) - .expect("should build a registry that runs the stand-in"), + integrations: registry.clone(), permissions_script: template_permissions_script(mode, &permissions_json_fixture()), ad_slots_script, ad_bids_state, @@ -9565,7 +9680,27 @@ mod tests { body_close, }; - let mut processor = create_html_processor(config); + let fetch = registry.middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + "/", + ); + // A shared mode builds the template every reader shares, which is + // the response `create_html_stream_processor` runs no serve entry + // on. + let shared_template_authorized = matches!(mode, AssemblyMode::Esi); + let serve = (!shared_template_authorized).then(|| { + registry.middleware_chain( + &settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + "/", + ) + }); + let mut processor = + create_html_processor_with_middleware(config, &fetch, serve.as_ref()) + .expect("should plan the document's middleware"); let out = processor .process_chunk(DOCUMENT, true) .expect("should process the document"); @@ -9646,6 +9781,7 @@ mod tests { "permissions", request_fixture::HEAD_FLAG, request_fixture::MODULE_FILE, + fixture::READER_SCRIPT, ] { assert!( !rendered.contains(forbidden), @@ -9728,6 +9864,12 @@ mod tests { && !unmarked.contains(request_fixture::MODULE_FILE), "a request no module left state on should get neither:\n{unmarked}" ); + assert!( + rendered.contains(fixture::READER_SCRIPT) + && unmarked.contains(fixture::READER_SCRIPT), + "an inline document is one reader's copy, so the serve entry should run on \ + it whatever the request left:\n{unmarked}" + ); } #[test] @@ -9884,6 +10026,50 @@ mod tests { settings } + #[test] + fn the_fetch_entries_change_the_fingerprint() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + // What a fetch middleware writes is stored with the template, so + // which run, in what order and on which paths all select one. + let fingerprint_with = |entries: &[(Option<&str>, &[&str])]| { + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.fetch = PhaseEntries::new( + entries + .iter() + .map(|(path, names)| PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + }) + .collect(), + ); + fingerprint(&settings) + }; + + let fingerprints = [ + fingerprint_with(&[]), + fingerprint_with(&[(None, &[fixture::HEAD])]), + fingerprint_with(&[(None, &[fixture::HEAD, fixture::LINKS])]), + fingerprint_with(&[(None, &[fixture::LINKS, fixture::HEAD])]), + fingerprint_with(&[(Some("/news/"), &[fixture::HEAD])]), + ]; + + for (index, fingerprint) in fingerprints.iter().enumerate() { + assert!( + !fingerprints[..index].contains(fingerprint), + "entries {index} must select a template of their own" + ); + } + assert_eq!( + fingerprint_with(&[(None, &[fixture::HEAD])]), + fingerprints[1], + "the same entries must select the same template" + ); + } + #[test] fn production_fingerprint_uses_the_generated_build_digest() { let settings = settings_with_integration(true, 1000); @@ -9936,8 +10122,8 @@ mod tests { #[test] fn reconfiguring_an_integration_changes_the_fingerprint() { - // Config reaches the template directly: an integration's head insert - // carries its settings into bytes shared between readers. + // Config reaches the template directly: a middleware's head markup + // carries its module's settings into bytes shared between readers. assert_ne!( fingerprint(&settings_with_integration(true, 1000)), fingerprint(&settings_with_integration(true, 2500)), @@ -10817,8 +11003,9 @@ mod tests { request } - /// A navigation a module left state on for its page hooks, the way a - /// real one arrives: on the request, left by a preparer or a filter. + /// A navigation a module left state on for its serve middleware, the + /// way a real one arrives: on the request, left by a preparer or a + /// filter. fn marked_navigation_request() -> Request { let mut request = navigation_request(); crate::integrations::registry_test_support::request_fixture::mark(&mut request); @@ -11144,23 +11331,485 @@ mod tests { } #[tokio::test] - async fn harness_emits_a_summary_row_for_an_ad_serving_navigation() { - let stub = Arc::new(StubHttpClient::new()); - let sink = Arc::new(RecordingTelemetrySink::default()); - let services = services_with_cache_and_telemetry( - Arc::clone(&stub), - Arc::new(MemoryTemplateCache::default()), - Arc::clone(&sink), - ); - let settings = Arc::new(settings_with_mode("esi")); - queue_shareable_html(&stub); + async fn harness_emits_a_summary_row_for_an_ad_serving_navigation() { + let stub = Arc::new(StubHttpClient::new()); + let sink = Arc::new(RecordingTelemetrySink::default()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::clone(&sink), + ); + let settings = Arc::new(settings_with_mode("esi")); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, navigation_request()).await; + + assert!( + last_summary_row(&sink).is_some(), + "the harness must emit a summary row, or every assertion built on it is vacuous" + ); + } + + /// [`settings_with_mode`], with core's middleware stand-in selected + /// under `label` and one `[[fetch]]` entry naming `names` for `path`. + fn settings_with_fetch_entry( + mode: &str, + label: &str, + path: Option<&str>, + names: &[&str], + ) -> Settings { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_mode(mode); + settings + .insert_module_config( + "testing", + fixture::MODULE, + &serde_json::json!({ "label": label }), + ) + .expect("should select the middleware stand-in"); + settings.fetch = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + }]); + settings + } + + /// [`navigation_request`], for another path of the same site. + fn navigation_request_for(path: &str) -> Request { + let mut request = navigation_request(); + *request.uri_mut() = format!("https://ts.example.com{path}") + .parse() + .expect("should build the address"); + request + } + + async fn page_of(response: Response) -> String { + String::from_utf8(body_of(response).await).expect("should serve a UTF-8 page") + } + + #[tokio::test] + async fn a_fetch_middleware_changes_a_page_under_the_path_its_entry_covers() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_fetch_entry( + "inline", + label, + Some("/article"), + &[fixture::HEAD], + )); + let marker = fixture::head_marker(label); + + queue_shareable_html(&stub); + let covered = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + assert_eq!( + covered.matches(&marker).count(), + 1, + "should run the entry's middleware once on a {label} page under its path, \ + with the module's own setting: {covered}" + ); + + queue_shareable_html(&stub); + let elsewhere = page_of( + run_via( + &settings, + &services, + navigation_request_for("/elsewhere"), + finalizer, + ) + .await, + ) + .await; + assert!( + elsewhere.contains("origin") && !elsewhere.contains(&marker), + "should run nothing on a {label} page no entry covers: {elsewhere}" + ); + } + } + + #[tokio::test] + async fn what_a_fetch_middleware_writes_is_stored_with_the_shared_template() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_fetch_entry( + "esi", + label, + None, + &[fixture::HEAD], + )); + let marker = fixture::head_marker(label); + // One answer from the origin, so the second reader can only + // be served from the store. + queue_shareable_html(&stub); + + let first = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + let second = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + + assert_eq!( + stub.recorded_cache_intents().len(), + 1, + "should serve the second {label} reader from the stored template" + ); + for (reader, page) in [("first", &first), ("second", &second)] { + assert_eq!( + page.matches(&marker).count(), + 1, + "should serve the {reader} {label} reader what the middleware wrote, \ + once: {page}" + ); + } + } + } + + /// One `[[serve]]` entry naming `names` for `path`. + fn serve_entry(path: Option<&str>, names: &[&str]) -> crate::middleware::PhaseEntries { + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + }]) + } + + /// Settings in `mode` whose fetch entry runs the stand-in's head + /// middleware on every page and whose serve entry runs its reader + /// middleware under `serve_path`. + fn settings_with_both_phases( + mode: &str, + label: &str, + serve_path: Option<&str>, + ) -> Settings { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_fetch_entry(mode, label, None, &[fixture::HEAD]); + settings.serve = serve_entry(serve_path, &[fixture::READER]); + settings + } + + /// What the stand-in's reader middleware writes for a deployment + /// running `settings`, for a request that left a mark or did not. + fn reader_marker_for(settings: &Settings, marked: bool) -> String { + crate::integrations::registry_test_support::middleware_fixture::reader_marker( + &settings.publisher.origin_host(), + marked, + ) + } + + fn index_in(page: &str, needle: &str) -> usize { + page.find(needle) + .unwrap_or_else(|| panic!("should find `{needle}` in: {page}")) + } + + #[tokio::test] + async fn a_reader_of_a_page_that_is_not_stored_gets_the_serve_entry() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_both_phases("inline", label, None)); + let reader = reader_marker_for(&settings, false); + queue_shareable_html(&stub); + + let page = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + + assert_eq!( + page.matches(&reader).count(), + 1, + "should run the serve entry once on a {label} page: {page}" + ); + assert_eq!(page.matches(fixture::READER_SCRIPT).count(), 1); + let fetch_at = index_in(&page, &fixture::head_marker(label)); + let reader_at = index_in(&page, &reader); + let bundle_at = index_in(&page, "id=\"trustedserver-js\""); + let after_at = index_in(&page, fixture::READER_SCRIPT); + assert!( + fetch_at < reader_at && reader_at < bundle_at && bundle_at < after_at, + "should write the serve entry's markup around the bundle, after the fetch \ + entry's: {page}" + ); + } + } + + #[tokio::test] + async fn a_reader_of_a_shared_template_gets_the_serve_entry_and_the_store_never_does() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::clone(&cache), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_both_phases("esi", label, None)); + // Told the same origin on the miss as on the hit, where the + // page's own record of it is empty. + let reader = reader_marker_for(&settings, false); + // One answer from the origin, so the second reader can only + // be served from the store. + queue_shareable_html(&stub); + + let miss = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + let hit = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + + assert_eq!( + stub.recorded_cache_intents().len(), + 1, + "should serve the second {label} reader from the stored template" + ); + for (reader_of, page) in [("miss", &miss), ("hit", &hit)] { + assert_eq!( + page.matches(&reader).count(), + 1, + "should run the serve entry once for the {label} reader of a \ + {reader_of}: {page}" + ); + assert_eq!( + page.matches(fixture::READER_SCRIPT).count(), + 1, + "should write its markup after the bundle once on a {reader_of}: {page}" + ); + assert_eq!( + page.matches(&fixture::head_marker(label)).count(), + 1, + "should serve what the fetch entry wrote once on a {reader_of}: {page}" + ); + } + let stored: Vec = cache + .entries + .lock() + .expect("should lock the stored templates") + .values() + .map(|entry| String::from_utf8_lossy(&entry.body).into_owned()) + .collect(); + assert_eq!(stored.len(), 1, "should store one template"); + assert!( + stored[0].contains(&fixture::head_marker(label)), + "should store what the fetch entry wrote: {}", + stored[0] + ); + assert!( + !stored[0].contains("middleware-fixture-reader") + && !stored[0].contains("data-fixture-reader"), + "should store nothing the serve entry wrote: {}", + stored[0] + ); + } + } + + #[tokio::test] + async fn a_page_from_the_store_sends_its_head_through_the_serve_entry_before_the_auction() { + // The streaming finalizer writes a stored page's head at once and + // waits for the auction only at the seam. The serve entry is a + // rewriter in that stream, so this checks it hands the head on + // instead of holding it until the page ends. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_both_phases("esi", "streamed", None)); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, navigation_request()).await; + let hit = run(&settings, &services, navigation_request()).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 1, + "the second request must be a hit, or this asserts nothing" + ); + let EdgeBody::Stream(mut stream) = hit.into_body() else { + panic!("a page from the store must stream, not buffer"); + }; + let first = stream + .next() + .await + .expect("the stream should yield a first chunk") + .expect("the first chunk should read"); + let first = String::from_utf8_lossy(&first); + + assert!( + first.contains(&reader_marker_for(&settings, false)) + && first.contains("id=\"trustedserver-js\""), + "the first chunk must be the head, with what the serve entry wrote in it: \ + {first}" + ); + assert!( + !first.contains("window.tsjs"), + "the first chunk must come before the bids script: {first}" + ); + } + + #[tokio::test] + async fn the_request_path_picks_the_serve_entry() { + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = + Arc::new(settings_with_both_phases("inline", label, Some("/article"))); + let reader = reader_marker_for(&settings, false); + + queue_shareable_html(&stub); + let covered = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + queue_shareable_html(&stub); + let elsewhere = page_of( + run_via( + &settings, + &services, + navigation_request_for("/elsewhere"), + finalizer, + ) + .await, + ) + .await; + + assert_eq!( + covered.matches(&reader).count(), + 1, + "should run the serve entry on a {label} page under its path: {covered}" + ); + assert!( + !elsewhere.contains("middleware-fixture-reader"), + "should run no serve entry on a {label} page no entry covers: {elsewhere}" + ); + } + } + + #[tokio::test] + async fn a_serve_middleware_reads_what_a_request_left_and_that_page_is_not_stored() { + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::clone(&cache), + Arc::new(RecordingTelemetrySink::default()), + ); + // A shared mode, which a request that carries state of its + // own never takes. + let settings = Arc::new(settings_with_both_phases("esi", label, None)); + queue_shareable_html(&stub); + + let page = page_of( + run_via(&settings, &services, marked_navigation_request(), finalizer).await, + ) + .await; + + assert_eq!( + page.matches(&reader_marker_for(&settings, true)).count(), + 1, + "should hand the serve middleware what the {label} request left: {page}" + ); + assert!( + cache + .entries + .lock() + .expect("should lock the stored templates") + .is_empty(), + "should store nothing of a page made for one request" + ); + } + } + + #[tokio::test] + async fn a_reader_with_an_auction_gets_the_serve_entry_and_its_bids() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; - let _ = run(&settings, &services, navigation_request()).await; + for mode in ["inline", "esi"] { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let mut settings = settings_with_bidder(mode); + settings.select_module("testing", fixture::MODULE); + settings.serve = serve_entry(None, &[fixture::READER]); + let settings = Arc::new(settings); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + let reader = reader_marker_for(&settings, false); - assert!( - last_summary_row(&sink).is_some(), - "the harness must emit a summary row, or every assertion built on it is vacuous" - ); + // Bid first, page second: one queue, and the auction is + // dispatched before the origin fetch. + queue_bid_response(&stub); + queue_shareable_html(&stub); + let first = + page_of(run_bidding(&settings, &services, navigation_request()).await).await; + // A second reader, whose page comes from the store under a + // shared mode and from the origin otherwise. + queue_bid_response(&stub); + if mode == "inline" { + queue_shareable_html(&stub); + } + let second = + page_of(run_bidding(&settings, &services, navigation_request()).await).await; + + for (which, page) in [("first", &first), ("second", &second)] { + assert_eq!( + page.matches(&reader).count(), + 1, + "should run the serve entry once for the {which} {mode} reader: {page}" + ); + let bids_at = index_in(page, "3.50"); + assert!( + bids_at < index_in(page, ""), + "should write the {which} {mode} reader's bid ahead of the end of the \ + body: {page}" + ); + } + } } /// Drive `handle_publisher_request` through the **EC-preload** fetch path. @@ -13166,14 +13815,20 @@ mod tests { async fn two_pass_integration_templates_are_identical_across_request_shapes() { let mut settings = settings_with_mode("esi"); settings.publisher.origin_url = "https://origin.example.com".to_owned(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::new(&settings).expect("should create registry"); assert!( - !registry.html_stream_processor_factories().is_empty(), - "should exercise the stand-in's stream processor" + !registry + .middleware_chain( + &settings.fetch, + crate::middleware::MiddlewarePhase::Fetch, + crate::middleware::HTML_MEDIA_TYPE, + "/", + ) + .is_empty(), + "should exercise the stand-in's middleware, which supplies a stream processor" ); let settings = Arc::new(settings); let html = br#"
"#; @@ -13565,6 +14220,7 @@ mod tests { }), ) .expect("should insert the stand-in's table"); + crate::integrations::registry_test_support::deferred_fixture::place(&mut settings); settings } @@ -13607,9 +14263,20 @@ mod tests { queue_shareable_html(&stub); queue_shareable_html(&stub); - let _ = run(&first, &services, navigation_request()).await; - let _ = run(&second, &services, navigation_request()).await; + let first_page = String::from_utf8( + body_of(run(&first, &services, navigation_request()).await).await, + ) + .expect("should serve the first configuration's page as UTF-8"); + let second_page = String::from_utf8( + body_of(run(&second, &services, navigation_request()).await).await, + ) + .expect("should serve the second configuration's page as UTF-8"); + assert!( + first_page.contains("\"timeoutMs\":1000") + && second_page.contains("\"timeoutMs\":2500"), + "each page should carry the markup its own configuration writes" + ); let stored = stored_cache_keys(&cache); assert_eq!( stored.len(), @@ -14920,10 +15587,7 @@ mod tests { let stub = Arc::new(StubHttpClient::new()); let cache = Arc::new(MemoryTemplateCache::default()); let mut raw = settings_with_mode("esi"); - raw.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, - ); + crate::integrations::registry_test_support::request_fixture::select_and_place(&mut raw); let settings = Arc::new(raw); let services = services(Arc::clone(&stub), Arc::clone(&cache)); queue_shareable_html(&stub); @@ -15012,10 +15676,7 @@ mod tests { let stub = Arc::new(StubHttpClient::new()); let cache = Arc::new(MemoryTemplateCache::default()); let mut raw = settings_with_mode("esi"); - raw.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, - ); + crate::integrations::registry_test_support::request_fixture::select_and_place(&mut raw); let settings = Arc::new(raw); let services = services(Arc::clone(&stub), Arc::clone(&cache)); queue_shareable_html(&stub); @@ -15082,10 +15743,7 @@ mod tests { .as_mut() .expect("fixture configures creative opportunities") .origin_is_cookie_independent = Some(true); - raw.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, - ); + crate::integrations::registry_test_support::request_fixture::select_and_place(&mut raw); let settings = Arc::new(raw); let services = services(Arc::clone(&stub), Arc::clone(&cache)); queue_shareable_html(&stub); @@ -18237,9 +18895,8 @@ mod tests { async fn inactive_ad_stack_preserves_the_cache_privacy_of_a_request_a_module_acts_on() { // Arrange let mut settings = settings_with_disabled_ad_templates(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); let stub = Arc::new(StubHttpClient::new()); queue_html_response_with_cache_control(&stub, "no-cache"); @@ -18906,9 +19563,8 @@ mod tests { #[tokio::test] async fn what_a_request_filter_leaves_survives_into_the_publisher_html_pipeline() { let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::new(&settings) .expect("should create integration registry with the stand-in"); @@ -20746,6 +21402,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20780,6 +21437,294 @@ mod tests { ); } + /// A selector is chosen for each document, so one that does not parse + /// cannot be refused when the settings load. The response is refused + /// instead of the parser panicking. + #[test] + fn a_middleware_whose_selector_does_not_parse_refuses_the_response() { + use crate::error::IntoHttpResponse as _; + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::BROKEN.to_owned()], + }]); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/".to_owned(), + content_type: "text/html; charset=utf-8".to_string(), + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }; + + let mut output = Vec::new(); + let error = stream_publisher_body( + EdgeBody::from(b"
x".to_vec()), + &mut output, + ¶ms, + &settings, + ®istry, + ) + .expect_err("should refuse a page whose middleware cannot run"); + + let message = error.to_string(); + assert!( + message.contains(&format!( + "the fetch middleware `{}` cannot run", + fixture::BROKEN + )) && message.contains(&format!( + "`{}` is not a usable selector", + fixture::BROKEN_SELECTOR + )), + "should name the middleware and the selector: {message}" + ); + assert_eq!( + error.current_context().status_code(), + StatusCode::INTERNAL_SERVER_ERROR, + "should answer as a fault of the deployment's own" + ); + assert!(output.is_empty(), "should write nothing of the page"); + } + + /// The same refusal through each finalizer, so an adapter answers with an + /// error where it would otherwise send the headers of a page it cannot + /// deliver. + #[tokio::test] + async fn a_middleware_whose_selector_does_not_parse_fails_the_response_through_each_finalizer() + { + use crate::error::IntoHttpResponse as _; + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::BROKEN.to_owned()], + }]); + let settings = Arc::new(settings); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + )); + let services = noop_services(); + let page = || PublisherResponse::Stream { + response: Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build the response"), + body: EdgeBody::from( + b"x".to_vec(), + ), + params: Box::new(OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/".to_owned(), + content_type: "text/html; charset=utf-8".to_string(), + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }), + }; + + let streamed = publisher_response_into_streaming_response( + page(), + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + services.clone(), + ) + .await; + let buffered = buffer_publisher_response_async( + page(), + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await; + + for (finalizer, result) in [("streaming", streamed), ("buffered", buffered)] { + let error = result + .err() + .unwrap_or_else(|| panic!("should fail the response in the {finalizer} finalizer")); + assert!( + error.to_string().contains(&format!( + "the fetch middleware `{}` cannot run", + fixture::BROKEN + )), + "should name the middleware in the {finalizer} finalizer: {error}" + ); + assert_eq!( + error.current_context().status_code(), + StatusCode::INTERNAL_SERVER_ERROR, + "should answer as a fault of the deployment's own in the {finalizer} finalizer" + ); + } + } + + /// A serve entry that cannot run fails the response at each place a page + /// leaves from, being a page that is not stored and a page from the + /// store, through each finalizer. + #[tokio::test] + async fn a_serve_middleware_whose_selector_does_not_parse_fails_the_response_wherever_it_leaves() + { + use crate::error::IntoHttpResponse as _; + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.serve = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::BROKEN_READER.to_owned()], + }]); + let settings = Arc::new(settings); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + )); + let services = noop_services(); + let response = || { + Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build the response") + }; + let params = || { + Box::new(OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/".to_owned(), + content_type: "text/html; charset=utf-8".to_string(), + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }) + }; + let from_the_origin = || PublisherResponse::Stream { + response: response(), + body: EdgeBody::from( + b"x".to_vec(), + ), + params: params(), + }; + let from_the_store = || PublisherResponse::AssembleTemplate { + response: response(), + template: format!( + "x{AD_ASSEMBLY_SEAM}" + ) + .into_bytes(), + params: params(), + }; + + let mut results = Vec::new(); + let pages: [(&str, &dyn Fn() -> PublisherResponse); 2] = [ + ("from the origin", &from_the_origin), + ("from the store", &from_the_store), + ]; + for (page, publisher_response) in pages { + results.push(( + page, + "streaming", + publisher_response_into_streaming_response( + publisher_response(), + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + services.clone(), + ) + .await, + )); + results.push(( + page, + "buffered", + buffer_publisher_response_async( + publisher_response(), + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await, + )); + } + + for (page, finalizer, result) in results { + let error = result.err().unwrap_or_else(|| { + panic!("should fail a page {page} in the {finalizer} finalizer") + }); + assert!( + error.to_string().contains(&format!( + "the serve middleware `{}` cannot run", + fixture::BROKEN_READER + )), + "should name the middleware for a page {page} in the {finalizer} finalizer: \ + {error}" + ); + assert_eq!( + error.current_context().status_code(), + StatusCode::INTERNAL_SERVER_ERROR, + "should answer as a fault of the deployment's own for a page {page} in the \ + {finalizer} finalizer" + ); + } + } + /// Empty origin body on the streaming route must produce no output /// without erroring. Exercises the `Ok(0)` branch of `process_chunks` /// plus the processor's `is_last=true, chunk=[]` terminal call. @@ -20805,6 +21750,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20853,6 +21799,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20981,6 +21928,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21047,6 +21995,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21116,6 +22065,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21185,6 +22135,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21254,6 +22205,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21303,6 +22255,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21528,6 +22481,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -21605,6 +22559,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -21684,6 +22639,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21757,6 +22713,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21912,6 +22869,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -21941,6 +22899,11 @@ mod tests { }), ) .expect("should insert the stand-in's settings"); + crate::html_processor::test_support::place_on_every_page( + &mut settings, + crate::middleware::MiddlewarePhase::Fetch, + &[tag::MODULE], + ); let body = streaming_finalize_response_with_settings( html_stream_params("", None), @@ -22008,9 +22971,8 @@ mod tests { // request token emitted by lol_html at the structural end is a seam. let page = br#"
still streaming
"#; let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let params = html_stream_params( "", @@ -22185,9 +23147,8 @@ mod tests { settings.auction.enabled = true; settings.auction.provider_names = vec!["seam_test".to_owned()]; settings.auction.timeout_ms = 60_000; - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let client = Arc::new(GatedAuctionHttpClient { inner: StubHttpClient::new(), @@ -22584,6 +23545,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -22781,6 +23743,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -22862,6 +23825,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "Text/HTML; Charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -22926,6 +23890,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -22992,11 +23957,11 @@ mod tests { /// routes through `Stream`, and the shared processor pipeline applies it. #[test] fn streaming_html_with_stream_processors_rewrites_body() { - // Select the stand-in so a stream processor is registered. + // Select the stand-in and place its middleware, so the page has a + // stream processor. let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::with_plan( @@ -23009,8 +23974,15 @@ mod tests { .expect("should create integration registry"); assert!( - !registry.html_stream_processor_factories().is_empty(), - "the stand-in must register an HTML stream processor" + !registry + .middleware_chain( + &settings.fetch, + crate::middleware::MiddlewarePhase::Fetch, + crate::middleware::HTML_MEDIA_TYPE, + "/", + ) + .is_empty(), + "the stand-in's middleware, which supplies an HTML stream processor, must run on the page" ); assert_eq!( classify_response_route( @@ -23038,6 +24010,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -23067,16 +24040,14 @@ mod tests { ); } - /// Document-state survives from the parser pass into the stream processor. - /// A script rewriter writes into `IntegrationDocumentState` during - /// parsing, and the request-local stream processor reads it and substitutes. + /// What a text handler captures during parsing reaches the stream + /// processor of the same document, which substitutes it. /// Regression test: placeholders must be inserted and removed from final output. #[test] - fn document_state_placeholders_substitute_through_streaming_path() { + fn a_handler_s_placeholders_are_substituted_through_the_streaming_path() { let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::with_plan( &settings, @@ -23099,6 +24070,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html".to_string(), permissions_json: String::new(), ad_slots_script: None, diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index e4920d9805..395ac0dd22 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -26,6 +26,7 @@ use crate::ec::module::{ }; use crate::error::TrustedServerError; use crate::host_header::validate_host_header_override_value; +use crate::middleware::{MiddlewarePhase, PhaseEntries}; use crate::platform::PlatformImageOptimizerRegion; use crate::provider_table::{ProviderChoice, ProviderList, SectionModules}; use crate::redacted::Redacted; @@ -4072,6 +4073,14 @@ pub struct Settings { skip_serializing_if = "crate::inspect::config::InspectConfig::is_default" )] pub inspect: crate::inspect::config::InspectConfig, + /// The page changes run on a document fetched from the origin, written + /// as `[[fetch]]` entries. See [`crate::middleware`]. + #[serde(default, skip_serializing_if = "PhaseEntries::is_empty")] + pub fetch: PhaseEntries, + /// The page changes run on each reader's copy of a document, written as + /// `[[serve]]` entries. + #[serde(default, skip_serializing_if = "PhaseEntries::is_empty")] + pub serve: PhaseEntries, /// Where the loader wrote secrets, which the configuration view masks. /// Never read from a document and never written to one. #[serde(skip)] @@ -4258,6 +4267,7 @@ impl Settings { settings.ec.validate_module_selection()?; settings.ec.validate_resolve_allowed_origins()?; settings.validate_module_sections()?; + settings.validate_phase_entries()?; settings.device.validate_module_selection()?; settings.geo.validate_module_selection()?; settings @@ -4754,6 +4764,32 @@ impl Settings { Ok(()) } + /// The entries of `phase`, in the order written. + #[must_use] + pub fn phase_entries(&self, phase: MiddlewarePhase) -> &PhaseEntries { + match phase { + MiddlewarePhase::Fetch => &self.fetch, + MiddlewarePhase::Serve => &self.serve, + } + } + + /// Checks the shape of each phase's entries, see + /// [`PhaseEntries::validate`]. Whether a name is one a running module + /// supplies is only knowable where the registry is built, so it is + /// checked there. + /// + /// # Errors + /// + /// Naming the entry at fault. + pub fn validate_phase_entries(&self) -> Result<(), Report> { + for phase in MiddlewarePhase::ALL { + self.phase_entries(phase) + .validate(phase) + .map_err(|message| Report::new(TrustedServerError::Configuration { message }))?; + } + Ok(()) + } + /// Checks every section's selection and tables, and that no module is /// selected in two sections. /// diff --git a/crates/trusted-server-core/src/streaming_processor.rs b/crates/trusted-server-core/src/streaming_processor.rs index d95e6ebf85..e3d9272e6b 100644 --- a/crates/trusted-server-core/src/streaming_processor.rs +++ b/crates/trusted-server-core/src/streaming_processor.rs @@ -310,8 +310,8 @@ impl lol_html::OutputSink for RcVecSink { /// Adapter to use `lol_html` [`HtmlRewriter`](lol_html::HtmlRewriter) as a [`StreamProcessor`]. /// /// Output is emitted incrementally on every [`process_chunk`](StreamProcessor::process_chunk) -/// call. Script rewriters that receive text from `lol_html` must be fragment-safe — -/// they accumulate text fragments internally until `is_last_in_text_node` is true. +/// call. A text handler that receives text from `lol_html` must be fragment-safe, +/// so it accumulates text fragments until the last chunk of the text node arrives. /// /// The adapter is single-use: one adapter per request. Calling [`StreamProcessor::reset`] /// is a no-op because the rewriter consumes its settings on construction. @@ -1731,8 +1731,8 @@ mod tests { } /// Verify that `lol_html` fragments text nodes when input chunks split - /// mid-text-node. Script rewriters must be fragment-safe — they accumulate - /// text fragments internally until `is_last_in_text_node` is true. + /// mid-text-node. A text handler must be fragment-safe, so it accumulates + /// text fragments until the last chunk of the text node arrives. #[test] fn lol_html_fragments_text_across_chunk_boundaries() { use std::cell::RefCell; diff --git a/crates/trusted-server-core/src/test_support.rs b/crates/trusted-server-core/src/test_support.rs index 59c3f2187e..d3cb084928 100644 --- a/crates/trusted-server-core/src/test_support.rs +++ b/crates/trusted-server-core/src/test_support.rs @@ -350,6 +350,12 @@ pub mod nextjs_auction { # fails closed on an unknown jurisdiction. [geo] module = "platform" + + # What Next.js changes in a page is a middleware, which runs on + # the pages an entry names it for. + [[fetch]] + media_type = "text/html" + middleware = ["framework.nextjs"] "#, ) .expect("should parse Next.js auction fixture settings"); diff --git a/crates/trusted-server-integration-tests/README.md b/crates/trusted-server-integration-tests/README.md index 2d3981721e..8ca3655a32 100644 --- a/crates/trusted-server-integration-tests/README.md +++ b/crates/trusted-server-integration-tests/README.md @@ -7,7 +7,7 @@ end-to-end publisher behavior. It is not linked into an adapter artifact. - `tests/parity.rs` calls Axum, Cloudflare, and Spin routers in process and compares their shared route behavior. -- `tests/documentation_snippets.rs` extracts the checked integration-guide +- `tests/documentation_snippets.rs` extracts each checked integration-guide fixture and compiles it in an isolated offline crate. - `tests/integration.rs` exercises the Fastly/Viceroy and Axum paths against the WordPress and Next.js fixture containers. diff --git a/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml index c3c81bb69e..5b8c643a94 100644 --- a/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml +++ b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml @@ -50,6 +50,12 @@ secret_store_id = "secrets" [ad-tag] modules = ["google.diagnostics"] +# What the overlay writes into a page is a serve middleware, which runs on +# each reader's copy of the pages an entry names it for. +[[serve]] +media_type = "text/html" +middleware = ["ad-tag.google.diagnostics"] + [proxy] certificate_check = false diff --git a/crates/trusted-server-integration-tests/tests/documentation_snippets.rs b/crates/trusted-server-integration-tests/tests/documentation_snippets.rs index 0e56dbc119..b215d47388 100644 --- a/crates/trusted-server-integration-tests/tests/documentation_snippets.rs +++ b/crates/trusted-server-integration-tests/tests/documentation_snippets.rs @@ -3,15 +3,30 @@ use std::process::Command; use std::time::{SystemTime, UNIX_EPOCH}; const GUIDE: &str = include_str!("../../../docs/guide/integration-guide.md"); -const START: &str = ""; -const END: &str = ""; #[test] fn integration_guide_runtime_services_fixture_compiles() { + assert_documented_fixture_compiles("runtime-services"); +} + +#[test] +fn integration_guide_middleware_fixture_compiles() { + assert_documented_fixture_compiles("middleware"); +} + +/// Compiles the fence the guide marks as the snippet `name`, exactly as +/// written, as an isolated crate that depends on core alone. +#[allow( + clippy::panic, + reason = "a helper of the tests above, which fails the test that called it" +)] +fn assert_documented_fixture_compiles(name: &str) { + let start = format!(""); + let end = format!(""); let marked = GUIDE - .split_once(START) - .and_then(|(_, tail)| tail.split_once(END).map(|(body, _)| body)) - .expect("should contain one bounded runtime-services snippet"); + .split_once(&start) + .and_then(|(_, tail)| tail.split_once(&end).map(|(body, _)| body)) + .unwrap_or_else(|| panic!("should contain one bounded {name} snippet")); let source = marked .trim() .strip_prefix("```rust\n") @@ -23,7 +38,7 @@ fn integration_guide_runtime_services_fixture_compiles() { .expect("should have a current system clock") .as_nanos(); let fixture_root = std::env::temp_dir().join(format!( - "trusted-server-documentation-snippet-{}-{nonce}", + "trusted-server-documentation-snippet-{name}-{}-{nonce}", std::process::id() )); let source_dir = fixture_root.join("src"); @@ -82,7 +97,7 @@ fn integration_guide_runtime_services_fixture_compiles() { assert!( output.status.success(), - "documented integration fixture must compile:\nstdout:\n{}\nstderr:\n{}", + "the documented {name} fixture must compile:\nstdout:\n{}\nstderr:\n{}", String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) ); diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 84fa14e4cc..c45f1dc572 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -47,7 +47,7 @@ type PbjsGlobal = typeof _pbjsDefault & { // Prebid.js itself is NOT bundled into this module. It is served as the // external bundle configured via `auction.prebid.external_bundle_url` // (required whenever the prebid integration runs) and owns the -// `window.pbjs` global. The Rust head injector emits a stub +// `window.pbjs` global. The Rust module's middleware emits a stub // (`window.pbjs = window.pbjs || {que:[],cmd:[]}`) before any script runs and // Prebid.js installs its API onto that same object, so capturing the reference // at module scope is safe regardless of evaluation order. @@ -179,7 +179,7 @@ export interface PrebidNpmConfig { /** * Shape of the server-injected config at `window.__tsjs_prebid`. - * Set by the Rust IntegrationHeadInjector from trusted-server.toml values. + * Set by the Rust module's middleware from trusted-server.toml values. */ interface InjectedPrebidConfig { accountId?: string; diff --git a/crates/trusted-server-modules/src/lib.rs b/crates/trusted-server-modules/src/lib.rs index 2951f12b33..c55f0ba443 100644 --- a/crates/trusted-server-modules/src/lib.rs +++ b/crates/trusted-server-modules/src/lib.rs @@ -5,6 +5,10 @@ //! hooks run. Offering a module does not run it, because the registry builds a //! module only when a section of the settings selects it. //! +//! It also says which middleware those modules supply, so a tool that writes +//! a configuration can place a module's page changes without naming one +//! itself. +//! //! A deployment that ships a module of its own hands that module's builder to //! its adapter, which runs it after these. @@ -14,6 +18,7 @@ )] use trusted_server_core::integrations::IntegrationBuilder; +use trusted_server_core::middleware::MiddlewarePhase; /// The builders of the modules a stock build ships from crates of their own, /// in hook order. @@ -68,11 +73,276 @@ pub fn selection_of(id: &str) -> Option<(&'static str, &'static str)> { )) } +/// A middleware a stock module supplies. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct StockMiddleware { + /// The id of the integration that supplies it. + pub integration: &'static str, + /// The phase it runs in. + pub phase: MiddlewarePhase, + /// The name an entry writes. + pub name: &'static str, +} + +/// The middleware a stock build's modules supply, core's script proxy among +/// them, in the order a deployment that runs them all names them in an +/// entry. +/// +/// That is the order the registry registers the modules in. Prebid registers +/// from the auction plan and so comes first, core's script proxy is the first +/// module a section selects, and the rest follow [`builders`]. +#[must_use] +pub fn middleware() -> Vec { + use MiddlewarePhase::{Fetch, Serve}; + use trusted_server_ad_tag_google as google; + use trusted_server_auction_prebid as prebid; + use trusted_server_audience_permutive as permutive; + use trusted_server_bot_protection_datadome as datadome; + use trusted_server_cmp_didomi as didomi; + use trusted_server_cmp_sourcepoint as sourcepoint; + use trusted_server_core::integrations::js_asset_proxy; + use trusted_server_framework_nextjs as nextjs; + use trusted_server_identity_lockr as lockr; + use trusted_server_tag_google_tag_manager as google_tag_manager; + use trusted_server_testing_testlight as testlight; + + let row = |integration, phase, name| StockMiddleware { + integration, + phase, + name, + }; + vec![ + row(prebid::builder().id(), Fetch, prebid::MODULE), + // Core registers its script proxy under the module's own name. + row(js_asset_proxy::MODULE, Fetch, js_asset_proxy::MODULE), + row(testlight::builder().id(), Fetch, testlight::MODULE), + row(nextjs::builder().id(), Fetch, nextjs::MODULE), + row(permutive::builder().id(), Fetch, permutive::MODULE), + row(lockr::builder().id(), Fetch, lockr::MODULE), + row(didomi::builder().id(), Fetch, didomi::MODULE), + row(sourcepoint::builder().id(), Fetch, sourcepoint::MODULE), + row( + google_tag_manager::builder().id(), + Fetch, + google_tag_manager::MODULE, + ), + row(datadome::builder().id(), Fetch, datadome::MODULE), + row(datadome::builder().id(), Serve, datadome::TAG_MIDDLEWARE), + row(google::builder().id(), Fetch, google::MODULE), + row( + google::diagnostics::builder().id(), + Serve, + google::diagnostics::MODULE, + ), + ] +} + +/// The names of the middleware that the stock modules with the integration +/// ids `ids` supply for `phase`, in the order an entry names them. +/// +/// A tool that knows a module by its integration id, as the `ts` audit does +/// from what it finds on a page, writes the entry that places the module's +/// page changes with this and names no middleware itself. +#[must_use] +pub fn middleware_for(ids: &[&str], phase: MiddlewarePhase) -> Vec<&'static str> { + middleware() + .into_iter() + .filter(|row| row.phase == phase && ids.contains(&row.integration)) + .map(|row| row.name) + .collect() +} + #[cfg(test)] mod tests { - use trusted_server_core::integrations::IntegrationBuilder; + use trusted_server_core::integrations::{IntegrationBuilder, IntegrationRegistry}; + use trusted_server_core::middleware::MiddlewarePhase; + + use super::{builders, builders_with, middleware, middleware_for, selection_of}; + + /// The settings each stock module's recorded page is made with, which + /// select the module and place every middleware it supplies. + const PAGE_SETTINGS: &[&str] = &[ + include_str!("../../auction/prebid/src/fixtures/page-change.settings.toml"), + include_str!( + "../../trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml" + ), + include_str!("../../testing/testlight/src/fixtures/page-change.settings.toml"), + include_str!("../../framework/nextjs/src/fixtures/page-change.settings.toml"), + include_str!("../../audience/permutive/src/fixtures/page-change.settings.toml"), + include_str!("../../identity/lockr/src/fixtures/page-change.settings.toml"), + include_str!("../../cmp/didomi/src/fixtures/page-change.settings.toml"), + include_str!("../../cmp/sourcepoint/src/fixtures/page-change.settings.toml"), + include_str!("../../tag/google-tag-manager/src/fixtures/page-change.settings.toml"), + include_str!("../../bot-protection/datadome/src/fixtures/page-change.settings.toml"), + include_str!("../../ad-tag/google/src/fixtures/page-change.settings.toml"), + ]; + + #[test] + fn the_listed_middleware_are_the_ones_the_modules_register() { + let listed = middleware(); + let mut checked: Vec<&str> = Vec::new(); + + for module_settings in PAGE_SETTINGS { + let settings = + trusted_server_core::html_processor::test_support::page_settings(module_settings); + let registry = IntegrationRegistry::with_registrations(&settings, &builders()) + .expect("should build a registry from a recorded page's settings"); + let running: Vec<&str> = registry + .registered_integrations() + .iter() + .map(|integration| integration.id) + .collect(); + + for phase in MiddlewarePhase::ALL { + let expected: Vec<&str> = listed + .iter() + .filter(|row| row.phase == phase && running.contains(&row.integration)) + .map(|row| row.name) + .collect(); + assert_eq!( + registry.middleware_in(phase), + expected, + "should list for {phase} what the modules {running:?} register for it" + ); + } + checked.extend(running); + } + + for row in &listed { + assert!( + checked.contains(&row.integration), + "should check `{}` against a module that registers it, and none of the \ + recorded pages runs `{}`", + row.name, + row.integration + ); + } + } + + #[test] + fn the_listed_middleware_follow_the_order_the_modules_register_in() { + // What registers from the auction plan goes first, core's own module + // is the first a section selects, and the stock modules follow. + let mut order = vec![ + trusted_server_auction_prebid::builder().id(), + trusted_server_core::integrations::js_asset_proxy::MODULE, + ]; + for builder in builders() { + if !order.contains(&builder.id()) { + order.push(builder.id()); + } + } + let place = |integration: &str| { + order + .iter() + .position(|id| *id == integration) + .unwrap_or_else(|| panic!("`{integration}` should be a stock module")) + }; + + for phase in MiddlewarePhase::ALL { + let places: Vec = middleware() + .iter() + .filter(|row| row.phase == phase) + .map(|row| place(row.integration)) + .collect(); + assert!( + places.is_sorted(), + "should list the {phase} middleware in the order their modules register: \ + {places:?}" + ); + } + } + + #[test] + fn the_template_s_entries_name_the_listed_middleware() { + use trusted_server_core::settings::Settings; + use trusted_server_core::test_support::template::{ + template_with_resolved_required_secrets, uncomment_block, + }; + + let template = uncomment_block( + &uncomment_block(&template_with_resolved_required_secrets(), "[[fetch]]"), + "[[serve]]", + ); + let settings = + Settings::from_toml(&template).expect("should load the template with its entries"); + + for phase in MiddlewarePhase::ALL { + let listed: Vec<&str> = middleware() + .iter() + .filter(|row| row.phase == phase) + .map(|row| row.name) + .collect(); + let entries = settings.phase_entries(phase).entries(); + assert_eq!( + entries.len(), + 1, + "should document one [[{phase}]] entry for every page" + ); + assert!( + entries[0].path.is_none(), + "should document the [[{phase}]] entry that covers every page" + ); + assert_eq!( + entries[0].middleware, listed, + "should document every {phase} middleware the stock modules supply, in the \ + listed order" + ); + } + } + + #[test] + fn the_guide_s_table_lists_the_same_middleware() { + const GUIDE: &str = include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../docs/guide/configuration.md" + )); + + // A row of the table is a name and then the list it is written in. + let documented: Vec<(String, String)> = GUIDE + .lines() + .filter_map(|line| { + let mut cells = line.split('|').map(str::trim).skip(1); + let name = cells.next()?.strip_prefix('`')?.strip_suffix('`')?; + let list = cells.next()?.strip_prefix("`[[")?.strip_suffix("]]`")?; + Some((name.to_owned(), list.to_owned())) + }) + .collect(); + let listed: Vec<(String, String)> = middleware() + .iter() + .map(|row| (row.name.to_owned(), row.phase.to_string())) + .collect(); + + assert_eq!( + documented, listed, + "should document the middleware the stock modules supply, each with its phase, \ + in the listed order" + ); + } + + #[test] + fn a_tool_is_handed_a_module_s_middleware_by_its_integration_id() { + use trusted_server_bot_protection_datadome as datadome; + use trusted_server_cmp_didomi as didomi; - use super::{builders, builders_with, selection_of}; + let ids = [datadome::builder().id(), didomi::builder().id()]; + + assert_eq!( + middleware_for(&ids, MiddlewarePhase::Fetch), + [didomi::MODULE, datadome::MODULE], + "should name the modules' fetch middleware in the order an entry runs them, \ + whatever order the ids came in" + ); + assert_eq!( + middleware_for(&ids, MiddlewarePhase::Serve), + [datadome::TAG_MIDDLEWARE], + "should name the one serve middleware the two supply" + ); + assert!( + middleware_for(&["no_such_module"], MiddlewarePhase::Fetch).is_empty(), + "should name nothing for an id no stock module has" + ); + } #[test] fn stock_modules_are_offered_in_hook_order() { diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md index d60345f08a..1f0a0e4f92 100644 --- a/docs/guide/api-reference.md +++ b/docs/guide/api-reference.md @@ -1290,7 +1290,7 @@ All integration modules are built at compile time. At runtime, the server concat Every row below records a compiled integration registration predicate. An integration with no HTTP route can still contribute a browser module, -rewriter, injector, post-processor, request filter, or ad server. In the +middleware, request filter, or ad server. In the predicates below, `named` means a section selects the integration's module. | Integration | Registration predicate | HTTP routes | diff --git a/docs/guide/cli.md b/docs/guide/cli.md index c0b10aa495..5c949a2fcf 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -355,7 +355,10 @@ ts config validate ``` The draft also selects, in the section of each module's type, the modules it -can configure from what it found, and writes their tables. Where it found +can configure from what it found, and writes their tables. It writes the +`[[fetch]]` and `[[serve]]` entries that run those modules' page changes on +every HTML page, because a module changes a page only where an entry names +its middleware. Where it found third-party scripts it writes `[proxy.js_asset_proxy]` with each one `proxy = "disabled"`, so they are inventory only, and nothing is served or rewritten until you review a candidate and change its `proxy` value to diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index d27b2e8055..bfa81ed085 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -237,6 +237,7 @@ fail and the service will return its startup-error response. | `[demand]` | several modules | The auction's demand sources | | `[device]` | one module | Device classification | | `[ec]` | one module | Edge Cookie identity, persistence, and partner sync | +| `[[fetch]]` | nothing | Which page changes run on a page as it is fetched, and in what order | | `[geo]` | one module | Which module resolves location, if any | | `[[handlers]]` | nothing | Ordered HTTP Basic-auth rules | | `[image_optimizer]` | nothing | Reusable Fastly Image Optimizer profiles | @@ -248,6 +249,7 @@ fail and the service will return its startup-error response. | `[request_signing]` | nothing | Outbound Ed25519 request signing and management-store IDs | | `[response_headers]` | nothing | Headers added to Trusted Server responses | | `[rewrite]` | nothing | First-party URL rewrite exclusions | +| `[[serve]]` | nothing | Which page changes run on each reader's copy of a page, and in what order | | `[tester_cookie]` | nothing | Optional tester-cookie endpoints | | `[tinybird]` | nothing | Direct Tinybird auction telemetry | | `[trusted_client_ip]` | nothing | Authenticated front-door client-IP forwarding | @@ -1858,6 +1860,97 @@ not page integrations, so a section that names one refuses startup like any name no module supplies. Their settings live in `[demand.]` and `[ad-server.]`. +### Placing page changes + +A module changes a page through a middleware, and a middleware runs only on +the pages where an entry names it. Selecting the module makes its middleware +available, and the `[[fetch]]` and `[[serve]]` entries say which pages each +one runs on and in what order. + +The two lists are the two phases of a page. A middleware runs in the phase +its module built it for, and an entry of the other list that names it refuses +startup. + +| Phase | Runs | Told about the reader | Stored in a shared template | +| ----------- | ------------------------------------------------------------------------------ | --------------------- | --------------------------- | +| `[[fetch]]` | On the page as the origin sent it, once for each fetch | No | Yes | +| `[[serve]]` | On each reader's copy, whether the page came from the store or from the origin | Yes | No | + +An entry of either list has the same three fields. + +| Field | Type | Required | Description | +| ------------ | ---------------- | -------- | ---------------------------------------------------------------------------------------------- | +| `media_type` | String | Yes | The media type the entry covers. Only `text/html` is accepted | +| `path` | String | No | A prefix of the request path, compared as written. Absent, or `/`, the entry covers every path | +| `middleware` | Array of strings | Yes | The middleware to run, in order, each handed the page as the one before it left it | + +A page takes the first entry that covers it and runs that entry's middleware +alone. Entries are therefore written from the longest path to the entry with +no path, and a folder is written with its closing slash, because `/news` also +begins `/newsletter`. + +```toml +[[fetch]] +media_type = "text/html" +path = "/news/" +middleware = ["tag.example.cleanup", "tag.example"] + +[[fetch]] +media_type = "text/html" +middleware = ["tag.example"] + +[[serve]] +media_type = "text/html" +middleware = ["tag.example.reader"] +``` + +A fetch middleware runs on the page as the origin sent it. It is told nothing +about the reader, because what it leaves is what a +[shared template](#shared-template-assembly-assembly-mode-esi) stores for +every reader, and the entries are part of what selects a stored template, so +changing them never serves a page built under the old ones. + +A serve middleware runs on one reader's copy, after the fetch middleware and +after a stored page is assembled for that reader. It is told what the +module's own request hooks left on that reader's request, and nothing it +writes is stored. The markup it adds to the head goes straight before the +script bundle, after the fetch middleware's. + +A middleware takes its settings from its module's own `[
.]` +table. An entry carries none and switches nothing on. + +The modules this repository ships supply the middleware below. A deployment +that runs several of them names them all in one entry's list, because a page +takes one entry, and the order written is the order they run in. The table +is in the order the modules register, which is the order `ts audit` writes +the names in and the order `trusted-server.example.toml` shows them in. + +| Middleware | Phase | What it changes | +| ----------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | +| `js_asset_proxy` | `[[fetch]]` | Points each configured script at its first-party path and removes one that is blocked. It is asked about an address as the middleware named before it left it, so name it ahead of one that moves the same script | +| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | +| `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | +| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | +| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | +| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | +| `cmp.sourcepoint` | `[[fetch]]` | Writes `window.__tsjs_sourcepoint` into the head and, when `rewrite_sdk` is set, the trap on `window._sp_`, and points a `src` or `href` on Sourcepoint's CDN at `/integrations/sourcepoint/cdn` | +| `tag.google-tag-manager` | `[[fetch]]` | Points Google Tag Manager and Google Analytics addresses at `/integrations/google_tag_manager`, in `src` and `href` attributes and in the text of inline scripts | +| `bot-protection.datadome` | `[[fetch]]` | Points a `src` or `href` that is a DataDome script's address at `/integrations/datadome`, when `rewrite_sdk` is set | +| `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | +| `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | +| `ad-tag.google.diagnostics` | `[[serve]]` | For a request that activated diagnostics, writes the bootstrap ahead of the script bundle and the tag that loads the diagnostics module straight after it | + +An entry that could not do what it says refuses the configuration, both when +a deployment is validated and when the settings load. That is a media type +other than `text/html`, a path that does not start with `/` or that holds a +`*`, a `?` or a `#`, an entry that names no middleware or names one twice, a +key an entry does not read, and an entry that an earlier one already covers. +An entry naming a middleware that no running module supplies refuses startup, +and the message lists the names that could be written. A middleware that no +entry of its phase names changes no page, and startup logs a warning that +names it. + The sections below give each module's settings, and the integration guides describe what each one does. @@ -1981,6 +2074,12 @@ Each `[[proxy.js_asset_proxy.assets]]` entry: | `proxy` | String | `enabled` | `enabled`, `disabled`, or `blocked` (removes script tags) | | `cache_ttl_seconds` | Integer | None | Optional per-asset downstream cache TTL override | +Rewriting and removing script tags is the middleware `js_asset_proxy`, which +runs on the pages a `[[fetch]]` entry names it for. It knows a script by the +address it is handed, which is the address as the middleware named before it +left it, so name it ahead of a middleware that moves the address of a script +it is to proxy or block. See [Placing page changes](#placing-page-changes). + ### lockr Integration **Section**: `[identity.lockr]` @@ -1995,7 +2094,10 @@ Each `[[proxy.js_asset_proxy.assets]]` entry: | `rewrite_sdk_host` | Boolean or null | `null` | Deprecated compatibility input | | `origin_override` | URL or null | `null` | Optional upstream `Origin` override | -See [lockr](/guide/integrations/lockr). +The SDK rewrite is the middleware `identity.lockr`, which runs on the pages +a `[[fetch]]` entry names it for. See +[Placing page changes](#placing-page-changes) and +[lockr](/guide/integrations/lockr). ### Prebid Integration @@ -3288,6 +3390,12 @@ from the file, and startup checks the rest and runs the first set again. - Each integration validates its own block, selected or not, so a typo in a block that is switched off is still caught +**Page changes**: + +- Every `[[fetch]]` and `[[serve]]` entry covers `text/html`, has a path no + earlier entry of its list already covers, and names at least one + middleware, each once + ### Checked when the service starts Everything above runs again on the loaded configuration, and these join it: @@ -3296,6 +3404,10 @@ Everything above runs again on the loaded configuration, and these join it: this build does not have, a missing settings table, or a table the selector does not name, stops the service on its next start. A passing `ts config validate` is not proof that a change to those four will start +- The middleware each `[[fetch]]` and `[[serve]]` entry names. A name no + running module supplies, or one named in the phase it does not run in, + stops the service on its next start, because which middleware a build has + is only known where its modules are registered - Resolved secret values, so a passphrase shorter than 32 bytes, a placeholder or a weak handler password fails here - The compiled `permissions.yaml` policy, and the `assume_single_jurisdiction` diff --git a/docs/guide/creative-processing.md b/docs/guide/creative-processing.md index 9d23119982..626ffce5e6 100644 --- a/docs/guide/creative-processing.md +++ b/docs/guide/creative-processing.md @@ -698,86 +698,67 @@ exclude_domains = ["assets.publisher.com"] Skip resources already on your domain. -## Integration Hooks +## Page Changes by Modules -### Attribute Rewriters +A module changes a page through a middleware, which runs on the pages a +`[[fetch]]` or `[[serve]]` entry names it for. A middleware may do any of +these to a document. -Integrations can override attribute rewriting: - -**Example**: Next.js integration rewrites origin URLs - -```rust -impl IntegrationAttributeRewriter for NextJsIntegration { - fn rewrite(&self, attr_name: &str, attr_value: &str, ctx: &Context) - -> AttributeRewriteAction - { - if attr_name == "href" && attr_value.contains(&ctx.origin_host) { - let rewritten = attr_value.replace(&ctx.origin_host, &ctx.request_host); - return AttributeRewriteAction::replace(rewritten); - } - AttributeRewriteAction::keep() - } -} -``` - -**Actions**: +**Decide about an attribute** of the elements a CSS selector matches: - `keep()` - Leave attribute unchanged - `replace(value)` - Change attribute value - `remove_element()` - Delete entire element -### Script Rewriters +**Decide about the text** inside the elements a selector matches, such as a +script's: + +- `keep()` - Leave the text unchanged +- `replace(content)` - Replace the text +- `remove_node()` - Remove the text -Integrations can modify `"#, - ctx.request_host - )] + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + let origin = context.origin_host.to_owned(); + let host = context.request_host.to_owned(); + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::new( + "data-link", + Rc::new(move |matched| { + if matched.value.contains(&origin) { + AttributeRewriteAction::replace(matched.value.replace(&origin, &host)) + } else { + AttributeRewriteAction::keep() + } + }), + ))], + ..MiddlewareAction::pass() + } } } ``` -**Behavior**: - -- Snippets are prepended into `` after the TSJS bundle tag -- Called once per HTML response -- Multiple integrations can each contribute snippets -- If no snippets are returned, no extra markup is added - -See [Integration Guide](/guide/integration-guide) for creating custom rewriters. +See [Changing a page](/guide/integration-guide#changing-a-page) for what a +middleware can rely on, and +[Placing page changes](/guide/configuration#placing-page-changes) for the +entries that place one. ## TSJS Injection diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index aa8631a362..2321fb8dc7 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -188,6 +188,7 @@ only the asset proxy entries you want to serve or block. Then validate it. Edit `trusted-server.toml` to configure: - the page modules that run, each selected in the section of its type (`[cmp]`, `[tag]`, `[ad-tag]`, `[bot-protection]`, `[identity]`, `[audience]`, `[framework]`, and `[auction]` for Prebid), with their settings under `[
.]` +- the `[[fetch]]` and `[[serve]]` entries that say which pages each selected module's page changes run on, which the audit's draft writes for the modules it selects - the demand sources, in `[demand] modules`, each with its settings under `[demand.]` - the ad server, if one runs, in `[ad-server] module`, with its settings under `[ad-server.]` - server bidder routes under `[auction.bidders.]` diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index d10be83bc5..5b47769c6d 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -11,12 +11,10 @@ Axum, or Spin SDK types. EdgeZero-neutral request and its call into the request's module context, from which it names the `Settings`, the `RuntimeServices` and whatever else it needs. -- `IntegrationAttributeRewriter` inspects selected HTML attributes. -- `IntegrationScriptRewriter` handles one declared selector. -- `IntegrationHeadInjector` inserts deterministic head markup. -- `IntegrationHtmlPostProcessor` is for bounded whole-document work that - cannot be performed during streaming. - `IntegrationRequestFilter` makes an early request decision. +- `Middleware` changes a page, on the pages where the operator's settings + place it, and is the one way a module changes a page. See + [Changing a page](#changing-a-page). Build one `IntegrationRegistration` with only the hooks the feature needs. Use `with_deferred_js()` only for a separately loaded integration module and @@ -25,7 +23,7 @@ namespaced and bounded; do not introduce a general outbound proxy. ## Compiling core-neutral fixture -The fixture below registers an attribute rewriter and constructs every +The fixture below registers a middleware and constructs every required `RuntimeServices` service without an adapter dependency. The documentation test extracts this exact fence and compiles it as an isolated crate. @@ -34,12 +32,16 @@ crate. ```rust use std::net::IpAddr; +use std::rc::Rc; use std::sync::Arc; use error_stack::Report; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationRegistration, + AttributeRewriteAction, IntegrationRegistration, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, + MiddlewareContext, MiddlewarePhase, }; use trusted_server_core::platform::{ BackendNamingPolicy, ClientInfo, GeoInfo, PlatformBackend, @@ -143,33 +145,35 @@ impl PlatformGeo for FixtureGeo { } } -struct AssetRewriter; +struct AssetAddress; -impl IntegrationAttributeRewriter for AssetRewriter { - fn integration_id(&self) -> &'static str { - "example" +impl Middleware for AssetAddress { + fn middleware_id(&self) -> &'static str { + "testing.example" } - fn handles_attribute(&self, attribute: &str) -> bool { - matches!(attribute, "src" | "href") + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - _attribute: &str, - value: &str, - _context: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - value - .strip_prefix("https://assets.example/") - .map(|path| AttributeRewriteAction::replace(format!("/assets/{path}"))) - .unwrap_or_else(AttributeRewriteAction::keep) + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let decide: Rc = Rc::new(|matched| { + matched + .value + .strip_prefix("https://assets.example/") + .map(|path| AttributeRewriteAction::replace(format!("/assets/{path}"))) + .unwrap_or_else(AttributeRewriteAction::keep) + }); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } } } pub fn registration() -> IntegrationRegistration { IntegrationRegistration::builder("example") - .with_attribute_rewriter(Arc::new(AssetRewriter)) + .with_middleware(Arc::new(AssetAddress)) .build() } @@ -298,21 +302,19 @@ this repository that leaves the declaration out. The build function returns an `IntegrationRegistration`, built with the builder every integration uses. -| Declaration | What it does | -| ----------------------------------------------------- | ----------------------------------------------------------------------------------- | -| `.with_proxy(...)` | Routes the paths the proxy declares | -| `.with_head_injector(...)` | Emits markup at the start of `` and, if it chooses, after the script bundle | -| `.with_attribute_rewriter(...)` | Rewrites attribute values in publisher HTML | -| `.with_script_rewriter(...)` | Rewrites inline script contents | -| `.with_html_stream_processor(...)` | Works on the document as it streams | -| `.with_request_filter(...)` | Inspects a request and can turn it back before it reaches the origin | -| `.with_js_module(CarriedJsModule { source, sha256 })` | Carries the integration's own browser script, built outside `trusted-server-js` | -| `.with_deferred_js()` | Serves the script as its own `