Add page changes as middleware

AGENTS.md

@@ -447,8 +447,7 @@ Integrations register in Rust via:
 ```rust
 IntegrationRegistration::builder(ID)
     .with_proxy()
-    .with_attribute_rewriter()
-    .with_head_injector()
+    .with_middleware()
     .build()
 ```
 

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware. A deployment adds the entries when it upgrades. Until it does, a module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. To keep a page as it was, name the middleware of the modules a deployment selects in one `[[fetch]]` entry and one `[[serve]]` entry for `text/html`, in the order `trusted-server.example.toml` shows them, which is the order the modules' hooks ran in.
+- **Breaking:** The four page traits are removed, being `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector`, with their context types, `ScriptTextAccumulator`, `AttributeRewriteOutcome` and the registration methods that took them. A module that implemented one supplies a `Middleware` instead, whose action carries the same decisions. A browser module's attribute on the script bundle's tag is stated with `with_bundle_tag_attribute`.
 - A pushed configuration keeps DataDome's two deprecated secret store selectors, `server_side_key_secret_store` and `protection_test_bypass.credential_secret_store`, where they were removed as the settings were read. DataDome still accepts and discards both, so a configuration that carries one loads as before.
 - Integration modules have moved out of `trusted-server-core` into crates of their own under `crates/<type>/<vendor>/`, each named by that folder, so `crates/cmp/osano` is the module `cmp.osano`. The settings that select and configure a module do not change. `crates/trusted-server-modules` lists the modules a stock build ships from crates of their own, in the order their hooks run. Every adapter builds its state and loads its settings with that list, and `ts config validate` and `ts config push` run each stock module's own rules. Code that named a moved module under `trusted_server_core::integrations` names its crate instead, such as `trusted_server_cmp_osano`. Moved: Osano, Lockr, Permutive, Sourcepoint, Didomi, Google Tag Manager, Testlight, Next.js, Google Publisher Tags and its diagnostics, DataDome, the mock ad server, APS, Prebid, the Prebid Server demand and the plain OpenRTB demand. Core keeps the JavaScript asset proxy, which is its own.
 - The Fastly adapter is a library with a thin binary over it. A deployment that ships a vendor crate writes a binary of its own that calls `trusted_server_adapter_fastly::run_with(vec![...])` with that crate's integration builder, in place of editing the adapter. The builders are composed with the built-in ones, and a builder runs only when the settings select its module. The Wasm artifact keeps its name, `trusted-server-adapter-fastly.wasm`.
@@ -62,6 +64,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- The origin's address in a page's `href`, `src`, `action`, `srcset` and `imagesrcset` is now replaced in one pass. A request host that begins with the origin host, such as `publisher.example.localhost` for `publisher.example`, was matched again inside its own replacement, so `https://publisher.example/` served over `http` came out as `http://publisher.example.localhost.localhost.localhost/`. A `srcset` served on a `www.` host in front of an apex origin came out with `www.www.`.
 - Publisher responses now map origin URLs in `Location`, `Content-Location`, `Refresh`, and `Link` headers to the serving host, and each CSP source naming the origin gains a serving-host source beside it (#1128). A redirect to the current URL under a different scheme, or a `GET`/`HEAD` redirect to the current URL that sets no cookie (or any such redirect under a `Host` override), is left unchanged to avoid a loop. `<meta>` refresh and CSP tags in the page body are not rewritten. See "Origin URLs in proxied response headers" in the configuration guide.
 - Image Optimizer configuration now rejects profile and profile-set names within the same map that collide after trimming, even when their values match, instead of silently overwriting an entry. Empty names continue to be discarded.
 - Browser integration bundles now share context providers and the configured log level with TSJS core (#1196). With Permutive enabled and cohorts available, `tsjs.requestAds()` now sends `config.permutive_segments` to `/auction`; the server retains that key only when `auction.allowed_context_keys` includes `"permutive_segments"`. This restores a previously inactive data flow, not a consent gate: review publisher consent handling before relying on it. Integration logs now follow `tsjs.setConfig` and `tsjs.log.setLevel`, and creative no longer raises the default `warn` level to `info`. The `?tsdebug=1` query parameter or `localStorage['tsdebug'] = '1'` intentionally enables `debug` logging across all bundles when creative installs, overriding any log level set earlier through `tsjs.setConfig` or `tsjs.log.setLevel`. Later calls to either API can change the level again.
@@ -73,9 +76,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `ts audit generate` writes the `[[fetch]]` and `[[serve]]` entries that run the page changes of the modules it selects on every HTML page, and `trusted-server.example.toml` documents the entries with every middleware the stock modules supply. `trusted-server-modules` lists those middleware with `middleware()` and `middleware_for(...)`.
+- Page changes as middleware. A module registers a change it makes to a page with `.with_middleware(...)`, and the ordered `[[fetch]]` and `[[serve]]` entries of the settings say which pages each one runs on and in what order. A fetch middleware runs on the page as the origin sent it and what it writes is stored with a shared template. A serve middleware runs on each reader's copy, whether the page came from the store or from the origin, and nothing it writes is stored. Where text a middleware writes into a script would end the script, or begin `<script` after `<!--`, its `<` is written as `\u003c`. An entry with no path after one with `path = "/"` is refused, because both cover every page.
 - A builder registers its module from the compiled auction plan with `with_plan_registration` and checks its settings against the plan with `with_plan_validator`, so a module whose page support follows what the plan selects is not named in `trusted-server-core`. Prebid and APS register this way. A renderer descriptor carries its builder's statement of which payload key holds the identifier the renderer picks its bid by, made with `BidRenderer::picking_bid_by`, and the page's `hb_adid` is read through that statement.
 - A module declares the settings in its own table that hold the name of a secret, with `with_secret_settings` on its builder, each with a rule for when the table puts it to use. A setting in use is looked up in the default secret store as the settings load and has to name a key for deploy validation to pass, and one not in use is cleared and never looked up. DataDome's server-side key and its test bypass credential are declared this way, and `trusted-server-core` no longer names them.
-- A module can act on one request from start to finish without `trusted-server-core` naming it. A request preparer or a request filter leaves a value in the request's `IntegrationRequestState`, which is copied into the document's state for the module's page hooks and handed to the response finalizer the builder declares with `with_response_finalizer`. A request that carries any value keeps to the origin path, so its document is never read from or stored as a shared template, and its HTML response is sent `private, no-store`. A head injector can also write straight after the script bundle with `after_bundle_inserts`, and a builder declares with `with_auction_token` that its browser script reads the token an auction publishes with its winning bids. The registry runs the request preparers once for a request.
+- A module can act on one request from start to finish without `trusted-server-core` naming it. A request preparer or a request filter leaves a value in the request's `IntegrationRequestState`, which the module's serve middleware reads from the state of the reader's copy and which is handed to the response finalizer the builder declares with `with_response_finalizer`. A request that carries any value keeps to the origin path, so its document is never read from or stored as a shared template, and its HTML response is sent `private, no-store`. A middleware can also write straight after the script bundle with `after_bundle_inserts`, and a builder declares with `with_auction_token` that its browser script reads the token an auction publishes with its winning bids. The registry runs the request preparers once for a request.
 - Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `<base>` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
 - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary.
 - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape.

crates/ad-tag/google/.gitattributes

@@ -0,0 +1,9 @@
+# The recording is compared byte for byte with the page made from the input
+# beside it, so both must arrive with the line endings they were committed
+# with on every platform.
+src/fixtures/page-change.*.html text eol=lf
+
+# The bootstrap script is embedded as it is on disk and written into the head
+# of every page, so it must arrive with the line endings it was committed
+# with on every platform.
+src/gpt_bootstrap.js text eol=lf

crates/ad-tag/google/src/diagnostics.rs

@@ -16,8 +16,10 @@ use validator::Validate;
 use trusted_server_core::error::TrustedServerError;
 use trusted_server_core::http_util::is_navigation_request;
 use trusted_server_core::integrations::{
-    IntegrationBuilder, IntegrationHeadInjector, IntegrationHtmlContext, IntegrationRegistration,
-    IntegrationRequestState,
+    IntegrationBuilder, IntegrationDocumentState, IntegrationRegistration, IntegrationRequestState,
+};
+use trusted_server_core::middleware::{
+    Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase,
 };
 use trusted_server_core::response_privacy::enforce_terminal_private_cache_privacy;
 use trusted_server_core::settings::{IntegrationConfig, Settings};
@@ -255,44 +257,51 @@ pub fn register(
     Ok(Some(
         IntegrationRegistration::builder(GPT_DIAGNOSTICS_INTEGRATION_ID)
             .with_standalone_js()
-            .with_head_injector(Arc::new(DiagnosticsHead))
+            .with_middleware(Arc::new(DiagnosticsHead))
             .build(),
     ))
 }
 
-/// Writes what an active decision adds to a document's head.
+/// Writes what an active decision adds to a reader's copy of a document, on
+/// the pages a `[[serve]]` entry names [`MODULE`] for.
+///
+/// It runs for each reader, and not once for a stored page, because the
+/// decision is made for one request.
 struct DiagnosticsHead;
 
 impl DiagnosticsHead {
     /// The decision the preparer left for this document, when it left one.
-    fn decision(ctx: &IntegrationHtmlContext<'_>) -> Option<Arc<GptDiagnosticsRequestDecision>> {
-        ctx.document_state
-            .get::<GptDiagnosticsRequestDecision>(GPT_DIAGNOSTICS_INTEGRATION_ID)
+    fn decision(
+        document_state: &IntegrationDocumentState,
+    ) -> Option<Arc<GptDiagnosticsRequestDecision>> {
+        document_state.get::<GptDiagnosticsRequestDecision>(GPT_DIAGNOSTICS_INTEGRATION_ID)
     }
 }
 
-impl IntegrationHeadInjector for DiagnosticsHead {
-    fn integration_id(&self) -> &'static str {
-        GPT_DIAGNOSTICS_INTEGRATION_ID
+impl Middleware for DiagnosticsHead {
+    fn middleware_id(&self) -> &'static str {
+        MODULE
     }
 
-    fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec<String> {
-        Self::decision(ctx)
-            .and_then(|decision| decision.bootstrap_script())
-            .into_iter()
-            .collect()
+    fn phases(&self) -> &[MiddlewarePhase] {
+        &[MiddlewarePhase::Serve]
     }
 
-    /// The module loads synchronously after the bundle, so its listeners are
-    /// in place before the publisher's scripts in the origin head run.
-    fn after_bundle_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec<String> {
-        Self::decision(ctx)
-            .and_then(|decision| {
-                JsModulePart::compile_time(GPT_DIAGNOSTICS_INTEGRATION_ID)
-                    .and_then(|module| decision.module_script_tag(&module))
-            })
-            .into_iter()
-            .collect()
+    /// The bootstrap goes ahead of the bundle. The module loads synchronously
+    /// after the bundle, so its listeners are in place before the publisher's
+    /// scripts in the origin head run.
+    fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction {
+        let Some(decision) = Self::decision(context.document_state) else {
+            return MiddlewareAction::pass();
+        };
+        MiddlewareAction {
+            head_inserts: decision.bootstrap_script().into_iter().collect(),
+            after_bundle_inserts: JsModulePart::compile_time(GPT_DIAGNOSTICS_INTEGRATION_ID)
+                .and_then(|module| decision.module_script_tag(&module))
+                .into_iter()
+                .collect(),
+            ..MiddlewareAction::pass()
+        }
     }
 }
 
@@ -511,9 +520,16 @@ mod tests {
     use trusted_server_core::integrations::IntegrationRegistry;
     use trusted_server_core::test_support::tests::create_test_settings;
 
+    /// Settings that select the module and run its middleware on every
+    /// reader's copy of a page.
     fn settings() -> Settings {
         let mut settings = create_test_settings();
         settings.select_module("ad-tag", "ad-tag.google.diagnostics");
+        trusted_server_core::html_processor::test_support::place_on_every_page(
+            &mut settings,
+            MiddlewarePhase::Serve,
+            &[MODULE],
+        );
         settings
     }
 
@@ -826,7 +842,8 @@ mod tests {
 
     #[test]
     fn an_active_document_loads_the_module_after_the_unified_bundle_once() {
-        use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor};
+        use trusted_server_core::html_processor::HtmlProcessorConfig;
+        use trusted_server_core::html_processor::test_support::create_page_processor;
         use trusted_server_core::streaming_processor::StreamProcessor as _;
 
         let settings = settings();
@@ -846,7 +863,7 @@ mod tests {
             "https",
         )
         .with_request_state(IntegrationRequestState::of(&request));
-        let mut processor = create_html_processor(config);
+        let mut processor = create_page_processor(&settings, &registry, config);
 
         let output = processor
             .process_chunk(
@@ -895,7 +912,8 @@ mod tests {
 
     #[test]
     fn an_inactive_document_gets_neither_script() {
-        use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor};
+        use trusted_server_core::html_processor::HtmlProcessorConfig;
+        use trusted_server_core::html_processor::test_support::create_page_processor;
         use trusted_server_core::streaming_processor::StreamProcessor as _;
 
         let settings = settings();
@@ -911,7 +929,7 @@ mod tests {
             "https",
         )
         .with_request_state(IntegrationRequestState::of(&request));
-        let mut processor = create_html_processor(config);
+        let mut processor = create_page_processor(&settings, &registry, config);
 
         let output = processor
             .process_chunk(

crates/ad-tag/google/src/fixtures/page-change.input.html

@@ -0,0 +1,17 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Google Publisher Tag</title>
+<link rel="preload" as="script" href="https://securepubads.g.doubleclick.net/tag/js/gpt.js">
+<script async src="https://securepubads.g.doubleclick.net/tag/js/gpt.js"></script>
+<script src="//securepubads.g.doubleclick.net/tag/js/gpt.js"></script>
+<script src="https://securepubads.g.doubleclick.net/pagead/other.js"></script>
+<script src="https://cdn.example.net/unrelated.js"></script>
+</head>
+<body>
+<a href="https://origin.example.com/article">An article on the origin</a>
+<img src="https://origin.example.com/image.png" alt="">
+<div id="ad-slot-1"></div>
+</body>
+</html>