Serve asset routes on every adapter

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- The Axum, Cloudflare and Spin adapters serve `[[proxy.asset_routes]]`, where only the Fastly adapter did, so a `GET` or `HEAD` under a route's prefix is answered from the route's source, with its signing and with the answers a selected asset cache module holds for it, and no longer by the publisher's origin. The response skips the Edge Cookie steps, so it neither writes nor expires the Edge Cookie, and takes the route's cache policy after the finalize headers. Those adapters hold the asset body whole, up to 10 MiB on Cloudflare and 8 MiB on Spin. Their HTTP clients refuse Image Optimizer metadata, which the Axum client now does on both of its send paths, so there a route that enables the Image Optimizer answers each request it would optimize with `502` and `no-store, private`, after a signed route has asked its origin for the object with a `HEAD`, and serves its SVG paths and requests that set the debug parameter unoptimized. Each adapter logs a warning naming every such route, once for each instance of the adapter, which on Axum is the process.
+- **Breaking:** on Spin, a request to a source whose `host_header` names a host other than the one in its `url` answers `502` before its origin is asked, the publisher's `[source.origin]` as well as an asset route's source, because Spin sends the host in `url` as the `Host`. Before, the request reached its origin with the host in `url` in place of `host_header`. A Spin deployment whose publisher source sets such a `host_header` answers every page with `502`, robots.txt included. Before upgrading, check that no source of a Spin deployment sets a `host_header` naming another host, or remove it once its origin answers requests that carry the host in `url` as the `Host`.
 - The Edge Cookie is written without an identity graph as it is with one, because the graph is one optional store of the data kept with the cookie, such as partner IDs. A Fastly service with no `[ec] ec_store` now writes the cookie a real browser's page navigation creates, and `POST /_ts/api/v1/ec/resolve` answers there as it does with a graph, where neither wrote one. So the identifier auction providers get as `user.id`, where storage and personalized-ad selection are both set, is one value for the life of the cookie, where a server-side module such as `hmac` gave a new value on each page view and a client-side module gave none. Naming an `ec_store` later gives each browser holding a cookie a server-side module wrote without one a new identifier on its first page view, because the identity graph replaces an identifier it has no row for. A client-side module creates nothing at the edge to replace its cookie with, so that cookie keeps its identifier with no row for as long as the browser keeps the cookie and its resolved marker, and each of that browser's page views pays for an extra identity-graph read and a strongly consistent existence check. **Breaking for crates outside core:** `ec_finalize_response` and `handle_ec_resolve` read the bot gate from the context's device signals (`EcContext::set_device_signals`), so a context with none gets no created cookie.
 - The Axum, Cloudflare and Spin adapters create and write the Edge Cookie and serve `POST /_ts/api/v1/ec/resolve`, where only the Fastly adapter did. A real browser's page navigation is given an Edge Cookie where its permissions allow one, and every route in the request cycle finalizes the Edge Cookie once the response is finalized, which writes a created cookie under the request's public prefix and path and expires one on a withdrawal. Those adapters open no identity graph, so the cookie is written with no graph row, by the resolve route too, the graph's syncs are skipped, and startup logs once that the store `[ec] ec_store` names is not opened. The bot gate there passes any client that sends a browser's `User-Agent`, and where storage and personalized-ad selection are both set a real browser's identifier reaches auction providers as `user.id`, from its first page view with a server-side module and from the requests after its resolve with a client-side one.
 - The Axum, Cloudflare and Spin adapters run the integration request filters and classify each request with the device module `[device]` selects, as the Fastly adapter does. Before, they ran no filter and asked a selected device module only for a bid request's device attributes. Every named route but the inspection endpoints, and the publisher fallback, reads the request's Edge Cookie context with the device classification and then runs the filters before the route, so a filter that answers, such as DataDome's challenge, stands in for the route, and the response headers a filter names are written after the finalize headers, with the Fastly adapter's privacy guards applied after them. The 51Degrees client hints middleware and DataDome's protection therefore work on every adapter, through HTTP clients that apply no module's own timeout on Axum, Cloudflare and Spin, and DataDome's protection checks `/auction` and `GET` on both page-bids paths as well as the publisher's pages. On those adapters `[device] module = "platform"` classifies from the `User-Agent`, a response core marks private loses any `ETag`, `Last-Modified`, `Expires` or `Age` that `[response_headers]` writes, robots.txt is answered before the Edge Cookie steps and the filters with an empty Edge Cookie context, and on Spin the discovery, signature verification and first-party routes and the page-bids preflight now run the modules' preparers.
@@ -118,6 +120,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- The Cloudflare and Spin adapters install a logger, so the warnings and errors they log are written. EdgeZero installs none on either platform, so every line was dropped, startup failures and each error answer's report included. The Worker writes each line with the console method of its level, which `wrangler dev` prints and `wrangler tail` and the dashboard's live logs show, and the Spin component writes each line to standard error, which Spin keeps as the component's log. Both write `info` and above.
 - The origin's address in a page's `href`, `src`, `action`, `srcset` and `imagesrcset` is now replaced in one pass. A request host that begins with the origin host, such as `publisher.example.localhost` for `publisher.example`, was matched again inside its own replacement, so `https://publisher.example/` served over `http` came out as `http://publisher.example.localhost.localhost.localhost/`. A `srcset` served on a `www.` host in front of an apex origin came out with `www.www.`.
 - Publisher responses now map origin URLs in `Location`, `Content-Location`, `Refresh`, and `Link` headers to the serving host, and each CSP source naming the origin gains a serving-host source beside it (#1128). A redirect to the current URL under a different scheme, or a `GET`/`HEAD` redirect to the current URL that sets no cookie (or any such redirect under a `Host` override), is left unchanged to avoid a loop. `<meta>` refresh and CSP tags in the page body are not rewritten. See "Origin URLs in proxied response headers" in the configuration guide.
 - Image Optimizer configuration now rejects profile and profile-set names within the same map that collide after trimming, even when their values match, instead of silently overwriting an entry. Empty names continue to be discarded.

crates/trusted-server-adapter-axum/src/app.rs

@@ -37,8 +37,8 @@ use trusted_server_core::integrations::{
 };
 use trusted_server_core::module_context::{ModuleContext, ModuleRequest};
 use trusted_server_core::proxy::{
-    handle_first_party_click, handle_first_party_proxy, handle_first_party_proxy_rebuild,
-    handle_first_party_proxy_sign,
+    AssetProxyCachePolicy, handle_asset_proxy_request, handle_first_party_click,
+    handle_first_party_proxy, handle_first_party_proxy_rebuild, handle_first_party_proxy_sign,
 };
 use trusted_server_core::publisher::{
     AppContext, AuctionDispatch, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH,
@@ -120,6 +120,38 @@ fn warn_once_when_the_identity_graph_is_unavailable(settings: &Settings, logged:
     );
 }
 
+/// Set once startup has logged that an asset route enables the Image
+/// Optimizer, which this adapter does not have. The state can be built more
+/// than once, so the flag keeps the warning to one for each instance.
+static IMAGE_OPTIMIZER_UNAVAILABLE_LOGGED: AtomicBool = AtomicBool::new(false);
+
+/// Logs once, at warning level, the prefix of each asset route that enables
+/// the Image Optimizer, which this adapter does not have.
+///
+/// The HTTP client refuses a request carrying Image Optimizer metadata, so
+/// such a route answers each request it would optimize with `502`. The route
+/// is not refused at startup, because its SVG paths and debug requests are
+/// still answered, as are the deployment's other routes and pages.
+fn warn_once_when_an_asset_route_enables_the_image_optimizer(settings: &Settings) {
+    let prefixes: Vec<String> = settings
+        .proxy
+        .asset_routes
+        .iter()
+        .filter(|route| route.image_optimizer.as_ref().is_some_and(|io| io.enabled))
+        .map(|route| format!("`{}`", route.prefix))
+        .collect();
+    if prefixes.is_empty() || IMAGE_OPTIMIZER_UNAVAILABLE_LOGGED.swap(true, Ordering::Relaxed) {
+        return;
+    }
+    log::warn!(
+        "the Axum adapter has no Image Optimizer, so each request these asset \
+         routes would optimize answers 502 Bad Gateway, and their SVG paths and \
+         requests that set the profile set's debug parameter are served \
+         unoptimized: {}",
+        prefixes.join(", ")
+    );
+}
+
 /// The permission signal modules this adapter links, in the order they run
 /// when configuration names none. Global Privacy Control is first because it
 /// is a browser setting with no interface of its own, and the three that
@@ -231,7 +263,7 @@ fn build_state_with_settings(
 }
 
 /// The header core writes a shared cache policy in on this adapter.
-const EDGE_CACHE_HEADER: EdgeCacheHeader = EdgeCacheHeader::SMaxageFallback;
+pub(crate) const EDGE_CACHE_HEADER: EdgeCacheHeader = EdgeCacheHeader::SMaxageFallback;
 
 /// Build the application state from explicit settings, composing the modules
 /// this adapter links with the externally supplied builders in
@@ -329,6 +361,7 @@ fn build_state_with_registrations_and_services(
     );
     warn_when_certificate_check_is_off(&settings);
     warn_once_when_the_identity_graph_is_unavailable(&settings, &IDENTITY_GRAPH_UNAVAILABLE_LOGGED);
+    warn_once_when_an_asset_route_enables_the_image_optimizer(&settings);
     let permission_policy = settings.required_permission_policy()?;
 
     Ok(Arc::new(AppState {
@@ -849,6 +882,15 @@ async fn dispatch_fallback(
                 }))
             })
     } else {
+        // A `GET` or `HEAD` under an asset route goes to the route's own
+        // source in place of the publisher's origin. Its response skips the
+        // Edge Cookie steps, so it carries no Edge Cookie state.
+        let matched_asset_route = matches!(method, Method::GET | Method::HEAD)
+            .then(|| state.settings.asset_route_index_for_path(&path))
+            .flatten();
+        if let Some(index) = matched_asset_route {
+            return dispatch_asset_fallback(state, services, req, index, effects).await;
+        }
         // A real browser's page navigation is given an Edge Cookie when it has
         // none. Another request may lack consent signals such as `Sec-GPC`.
         if ec.is_real_browser
@@ -908,6 +950,64 @@ async fn dispatch_fallback(
     )
 }
 
+/// Whether an asset response carries a body.
+///
+/// `HEAD` responses and the bodiless statuses (204, 304) keep the origin's
+/// `Content-Length` and carry no body, so a body stream is dropped for them.
+fn asset_response_carries_body(method: &Method, status: StatusCode) -> bool {
+    *method != Method::HEAD
+        && status != StatusCode::NO_CONTENT
+        && status != StatusCode::NOT_MODIFIED
+}
+
+/// Answers a request under the asset route at `index` from the route's own
+/// source.
+///
+/// The route's source module locates the asset, its signer signs the request
+/// where the source's `auth` table names one, and core answers from a copy it
+/// holds before it asks the origin. The response carries its cache policy,
+/// which [`FinalizeResponseMiddleware`] applies after the finalize headers,
+/// and the request filters' effects, and no Edge Cookie state, because an
+/// asset response skips the Edge Cookie steps.
+async fn dispatch_asset_fallback(
+    state: &AppState,
+    services: &RuntimeServices,
+    req: Request,
+    index: usize,
+    effects: RequestFilterEffects,
+) -> Response {
+    let method = req.method().clone();
+    let asset_route = &state.settings.proxy.asset_routes[index];
+    let answer = match state.registry.route_source(&state.settings, index) {
+        Ok(source) => {
+            handle_asset_proxy_request(&state.settings, services, req, asset_route, source.as_ref())
+                .await
+        }
+        Err(report) => Err(report),
+    };
+    let response = match answer {
+        Ok(asset_response) => {
+            let cache_policy = asset_response.cache_policy();
+            let (mut response, stream_body) = asset_response.into_response_and_body();
+            if let Some(body) = stream_body
+                && asset_response_carries_body(&method, response.status())
+            {
+                *response.body_mut() = body;
+            }
+            response.extensions_mut().insert(cache_policy);
+            response
+        }
+        Err(report) => {
+            let mut response = http_error(&report);
+            response
+                .extensions_mut()
+                .insert(AssetProxyCachePolicy::NoStorePrivate);
+            response
+        }
+    };
+    attach_request_filter_effects(response, effects)
+}
+
 /// Whether the request is a crawler's `GET` or `HEAD` of the robots.txt that
 /// `[robots-txt]` makes.
 fn asks_for_robots_txt(settings: &Settings, method: &Method, path: &str) -> bool {