Map folder origin URLs in headers, add a serve headers step

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- The mapping of origin URLs in response headers (#1128) follows a path in `origin_url`. A URL on the origin is mapped to the serving host only when it is in that folder, and it loses the folder, so the reader of a site served from a folder of object storage is sent to the right address. A URL on the origin's host outside the folder is unchanged, because no address on the serving host leads there, and a CSP source with no path still gains the serving host beside it. A reader of a publisher whose request addressing hook removes a path prefix is sent to an address without the prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers `421` if every selector value misses.
 - **Breaking:** a path in `publisher.origin_url` is kept, and the request path is added after it with one `/`. The path was discarded before, so a deployment whose `origin_url` has one now sends each request to the folder the path names, and answers `400` to a request path with a `.` or `..` segment, with or without `origin_auth`. A deployment whose `origin_url` has no path sends its requests as before. Before upgrading, check that the origin serves the site at that path, or remove the path.
 - **Breaking:** the check that closes `/_ts/admin` and `/admin/keys` on every adapter now also answers `404` to a request path still changing after four rounds of percent-decoding and to a path that reaches either once an escape that is not valid UTF-8 is replaced. The publisher received both before, and no setting turns the check off.
 - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServicesBuilder::geo`, `RuntimeServices::with_geo` and `IntegrationRegistrationBuilder::with_geo_module` take an `Arc<dyn GeoModule>`, and `RuntimeServices::geo()`, `IntegrationRegistry::geo_module()`, the `IntegrationRegistration::geo_module` field and `build_geo_module` return or hold one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths build the call from the request.
@@ -102,6 +103,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- A serve middleware may set headers on a response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. The method is not called for a refusal made before the origin is asked, such as the `405` for a method a signed origin does not take, for a page served from the template cache or the refusal of such a page, or for an answer made from an error.
 - A publisher's site may be served from a folder of object storage through the publisher proxy, so its pages pass through the page pipeline, which an asset route skips. The folder is the path of `origin_url`, and two optional `[publisher]` settings do the rest, each off unless set. `origin_auth` takes the table an asset route's `auth` takes and signs every origin request with it, through the same signer. `directory_index` names a file added to a request path ending in `/`, for the origin request only. A signed origin is only read, so a method other than `GET` or `HEAD` is answered `405`, and a signed request carries none of the reader's cookies, credentials, referrer, conditional validators or range. A malformed `origin_auth` or `directory_index` is refused when the settings load, with the setting named.
 - `[ad-tag.google] write_loader` writes the loader tag, for a page that carries no loader of its own, through the serve middleware `ad-tag.google.loader`, which a `[[serve]]` entry names and `ts audit` writes where it finds GPT. The tag goes in the head after the bootstrap and asks for the loader on the module's own route. Only a reader whose permissions allow storage on the device (`necessary.operations.storage`) gets it, because GPT loaded from Google's standard address may use the device's storage at any time. That is TCF Purpose 1, which core's server-side auction requires where GDPR applies and which the Google tag module's loaders require. Off by default, because a page that already carries the loader would otherwise load it twice.
 - An asset cache module may answer for a file before core asks its origin. Core offers each `GET` for a file that the publisher's origin sends or an asset route fetches, other than one the platform's image optimizer answers, to the module that `[asset-cache]` selects, with what it knows of the reader's request and the request it would send. The module serves a response, which core handles as it would the origin's, reports that its own origin request failed, which core answers as a failed origin request, or passes the file back, and a deployment that selects none has every file asked of its origin. A registration supplies one with `with_asset_cache_module`, and the registry refuses one from a builder that `[asset-cache]` does not select, an `[asset-cache]` that selects two, and one that selects a builder that supplies none, naming that builder. The new `crates/asset-cache/51degrees`, the module `asset-cache.51degrees`, which no stock build links, holds each file as its origin sent it, under what the origin's `Vary` names and for the origin's own lifetime, in two loading caches it declares, `51degrees-assets` for the copies and `51degrees-assets-vary` for each address's record, each selecting its own loading cache module and both sized by `[cache.51degrees-assets]`. It keeps nothing that may not be shared, serves a copy with an `Age` that counts from the origin's, and on the publisher path leaves a document, and every request that carries a cookie, to the origin. A load that fails asking the origin is the answer to every request waiting on it, so a failing origin is asked once. A table that names `none` is refused, as one naming any module no linked crate offers is.

crates/trusted-server-core/src/integrations/registry.rs

@@ -2297,6 +2297,22 @@ impl IntegrationRegistry {
         MiddlewareChain::new(phase, media_type, middleware)
     }
 
+    /// The middleware the serve phase's headers step asks for `path`: every
+    /// one an entry whose path covers it names, see
+    /// [`PhaseEntries::for_path`].
+    #[must_use]
+    pub fn middleware_for_path(
+        &self,
+        entries: &PhaseEntries,
+        path: &str,
+    ) -> Vec<Arc<dyn Middleware>> {
+        entries
+            .for_path(path)
+            .into_iter()
+            .filter_map(|name| self.inner.middleware_named(name).map(Arc::clone))
+            .collect()
+    }
+
     /// Whether a middleware that an entry of either phase names for a request
     /// for `path`, under any media type, needs the origin's whole body, see
     /// [`Middleware::needs_whole_body`]. Asked before the origin answers, so
@@ -3307,7 +3323,7 @@ pub(crate) mod test_support {
         use crate::integrations::{CORE_SOURCE, IntegrationBuilder};
         use crate::middleware::{
             AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase,
-            Preparation, Refusal,
+            Preparation, Refusal, ResponseHeader, ResponseView,
         };
         use crate::module_context::ModuleCall;
         use crate::settings::Settings;
@@ -3497,6 +3513,62 @@ pub(crate) mod test_support {
                 MiddlewareAction::pass()
             }
         }
+        /// The name of the serve middleware that sets headers through the
+        /// headers step alone.
+        pub(crate) const LOCATING: &str = "testing.middleware-fixture.locating";
+        pub(crate) const LOCATION_PREFIX: &str = "/fixture";
+        /// The header the locating middleware sets on every response it is
+        /// shown, carrying that response's status.
+        pub(crate) const SHOWN_STATUS_NAME: &str = "x-fixture-shown-status";
+
+        /// Puts a prefix on a redirect's root-relative `Location`, which is
+        /// what a deployment that names the publisher in the first part of the
+        /// address needs, and names the status of every response it is shown.
+        struct Locating;
+
+        impl Middleware for Locating {
+            fn middleware_id(&self) -> &'static str {
+                LOCATING
+            }
+
+            fn phases(&self) -> &[MiddlewarePhase] {
+                &[MiddlewarePhase::Serve]
+            }
+
+            fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction {
+                MiddlewareAction::pass()
+            }
+
+            fn response_headers(
+                &self,
+                _context: &MiddlewareContext<'_>,
+                response: &ResponseView<'_>,
+            ) -> Vec<ResponseHeader> {
+                let mut headers = vec![(
+                    http::HeaderName::from_static(SHOWN_STATUS_NAME),
+                    http::HeaderValue::from(response.status.as_u16()),
+                )];
+                if !response.status.is_redirection() {
+                    return headers;
+                }
+                let Some(location) = response
+                    .headers
+                    .get(http::header::LOCATION)
+                    .and_then(|value| value.to_str().ok())
+                else {
+                    return headers;
+                };
+                if location.starts_with('/')
+                    && !location.starts_with("//")
+                    && let Ok(value) =
+                        http::HeaderValue::from_str(&format!("{LOCATION_PREFIX}{location}"))
+                {
+                    headers.push((http::header::LOCATION, value));
+                }
+                headers
+            }
+        }
+
         /// The name of the serve middleware that writes a header on a text
         /// response.
         pub(crate) const TEXT_HEADER: &str = "testing.middleware-fixture.text-header";
@@ -3733,6 +3805,7 @@ pub(crate) mod test_support {
                     .with_middleware(Arc::new(Preparing))
                     .with_middleware(Arc::new(Refusing))
                     .with_middleware(Arc::new(TextHeader))
+                    .with_middleware(Arc::new(Locating))
                     .build(),
             ))
         }
@@ -7106,6 +7179,7 @@ mod tests {
                 fixture::PREPARING,
                 fixture::REFUSING,
                 fixture::TEXT_HEADER,
+                fixture::LOCATING,
                 crate::service_version::MODULE,
             ],
             "should list the middleware the selected module supplies, and core's own after them"
@@ -7129,6 +7203,7 @@ mod tests {
                 fixture::BROKEN_READER,
                 fixture::PREPARING,
                 fixture::TEXT_HEADER,
+                fixture::LOCATING,
                 crate::service_version::MODULE
             ],
             "should list the middleware that run in the serve phase, core's own last"
@@ -7150,6 +7225,37 @@ mod tests {
         );
     }
 
+    #[test]
+    fn middleware_for_path_gives_every_covering_entry_s_middleware_once_in_entry_order() {
+        let registry = IntegrationRegistry::new(&fixture_settings_with_entries(Vec::new()))
+            .expect("should build a registry");
+        let entries = PhaseEntries::new(vec![
+            html_entry(Some("/news/"), &[fixture::LOCATING, fixture::READER]),
+            text_entry(&[fixture::TEXT_HEADER, fixture::LOCATING]),
+            html_entry(Some("/sport/"), &[fixture::BROKEN_READER]),
+            html_entry(None, &["testing.nothing", fixture::READER]),
+        ]);
+
+        let ids = |path: &str| -> Vec<&'static str> {
+            registry
+                .middleware_for_path(&entries, path)
+                .iter()
+                .map(|middleware| middleware.middleware_id())
+                .collect()
+        };
+        assert_eq!(
+            ids("/news/today"),
+            [fixture::LOCATING, fixture::READER, fixture::TEXT_HEADER],
+            "should give each middleware once, in entry order, whatever the entry's media type"
+        );
+        assert_eq!(
+            ids("/weather/"),
+            [fixture::TEXT_HEADER, fixture::LOCATING, fixture::READER],
+            "should leave out an entry whose path does not cover the request, and a name \
+             nothing supplies"
+        );
+    }
+
     /// An entry for `text/plain` naming the middleware that works on it.
     fn text_entry(names: &[&str]) -> PhaseEntry {
         PhaseEntry {
@@ -7245,7 +7351,7 @@ mod tests {
             message.contains(
                 "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies"
             ) && message.contains(&format!(
-                "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}]",
+                "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}]",
                 fixture::HEAD,
                 fixture::LINKS,
                 fixture::BROKEN,
@@ -7255,6 +7361,7 @@ mod tests {
                 fixture::PREPARING,
                 fixture::REFUSING,
                 fixture::TEXT_HEADER,
+                fixture::LOCATING,
                 crate::service_version::MODULE
             )),
             "should name the entry and list what could be named: {message}"
@@ -7311,6 +7418,7 @@ mod tests {
                 fixture::PREPARING,
                 fixture::REFUSING,
                 fixture::TEXT_HEADER,
+                fixture::LOCATING,
             ],
             "should list every middleware when there are no entries"
         );
@@ -7326,7 +7434,8 @@ mod tests {
                 fixture::TEXT,
                 fixture::PREPARING,
                 fixture::REFUSING,
-                fixture::TEXT_HEADER
+                fixture::TEXT_HEADER,
+                fixture::LOCATING
             ],
             "should leave out a middleware any entry names"
         );
@@ -7397,7 +7506,8 @@ mod tests {
                 fixture::TEXT,
                 fixture::PREPARING,
                 fixture::REFUSING,
-                fixture::TEXT_HEADER
+                fixture::TEXT_HEADER,
+                fixture::LOCATING
             ],
             "should count a middleware as named only by an entry of a phase it runs in"
         );

crates/trusted-server-core/src/middleware.rs

@@ -276,6 +276,15 @@ pub trait TextHandler {
 /// A header a middleware writes on the reader's response.
 pub type ResponseHeader = (HeaderName, HeaderValue);
 
+/// What a serve middleware is told about the response it may set a header
+/// on, read only: the status and the headers as they stand after core's own
+/// rewrites, before the adapter finalizes the response.
+#[derive(Debug, Clone, Copy)]
+pub struct ResponseView<'a> {
+    pub status: StatusCode,
+    pub headers: &'a http::HeaderMap,
+}
+
 /// Writes `headers` on `response`, each replacing a header of the same name.
 pub fn write_response_headers<T>(response: &mut http::Response<T>, headers: Vec<ResponseHeader>) {
     for (name, value) in headers {
@@ -572,6 +581,22 @@ pub trait Middleware: Send + Sync {
     /// Decides what to do with one document.
     fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction;
 
+    /// The headers a serve middleware sets on the response the publisher path
+    /// sends, a redirect with no body included, for a request path its entry
+    /// covers. Each replaces a
+    /// header of the same name, and one the action from
+    /// [`create`](Self::create) sets in
+    /// [`MiddlewareAction::response_headers`] replaces it in turn. None by
+    /// default.
+    fn response_headers(
+        &self,
+        context: &MiddlewareContext<'_>,
+        response: &ResponseView<'_>,
+    ) -> Vec<ResponseHeader> {
+        let _ = (context, response);
+        Vec::new()
+    }
+
     /// Text a stored page is keyed by, being what this middleware writes
     /// that no setting decides, such as markup compiled into its crate. A
     /// change to it stores pages again, where otherwise a page stored before
@@ -661,6 +686,23 @@ impl PhaseEntries {
             .map_or(&[][..], |entry| entry.middleware.as_slice())
     }
 
+    /// The middleware every entry whose path covers `path` names, in entry
+    /// order and each once, whatever the entries' media types. A header is
+    /// not about a body, so this is what the headers step of the serve phase
+    /// asks.
+    #[must_use]
+    pub fn for_path(&self, path: &str) -> Vec<&str> {
+        let mut names: Vec<&str> = Vec::new();
+        for entry in self.0.iter().filter(|entry| entry.covers_path(path)) {
+            for name in &entry.middleware {
+                if !names.contains(&name.as_str()) {
+                    names.push(name);
+                }
+            }
+        }
+        names
+    }
+
     /// Every name the entries select, each once, in the order first written.
     #[must_use]
     pub fn names(&self) -> Vec<&str> {

crates/trusted-server-core/src/middleware/tests.rs

@@ -312,6 +312,33 @@ fn names_lists_each_middleware_once_in_the_order_first_written() {
     assert_eq!(entries.names(), ["example.b", "example.a", "example.c"]);
 }
 
+#[test]
+fn for_path_names_every_entry_covering_the_path_whatever_its_media_type() {
+    let mut text = entry(Some("/news/"), &["example.text", "example.a"]);
+    text.media_type = "text/plain".to_owned();
+    let entries = PhaseEntries::new(vec![
+        entry(Some("/news/"), &["example.a", "example.b"]),
+        text,
+        entry(Some("/sport/"), &["example.sport"]),
+        entry(None, &["example.b", "example.all"]),
+    ]);
+
+    assert_eq!(
+        entries.for_path("/news/today"),
+        ["example.a", "example.b", "example.text", "example.all"],
+        "should name each middleware once, in entry order, from every entry covering the path"
+    );
+    assert_eq!(
+        entries.for_path("/weather/"),
+        ["example.b", "example.all"],
+        "should leave out an entry whose path does not cover the request"
+    );
+    assert!(
+        PhaseEntries::default().for_path("/").is_empty(),
+        "should name nothing when there are no entries"
+    );
+}
+
 #[test]
 fn entries_from_the_longest_path_to_every_path_are_accepted() {
     PhaseEntries::new(vec![