diff --git a/CHANGELOG.md b/CHANGELOG.md index 26cdd03768..a659c9ef79 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- The mapping of origin URLs in response headers (#1128) follows a path in `origin_url`. A URL on the origin is mapped to the serving host only when it is in that folder, and it loses the folder, so the reader of a site served from a folder of object storage is sent to the right address. A URL on the origin's host outside the folder is unchanged, because no address on the serving host leads there, and a CSP source with no path still gains the serving host beside it. A reader of a publisher whose request addressing hook removes a path prefix is sent to an address without the prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers `421` if every selector value misses. - **Breaking:** a path in `publisher.origin_url` is kept, and the request path is added after it with one `/`. The path was discarded before, so a deployment whose `origin_url` has one now sends each request to the folder the path names, and answers `400` to a request path with a `.` or `..` segment, with or without `origin_auth`. A deployment whose `origin_url` has no path sends its requests as before. Before upgrading, check that the origin serves the site at that path, or remove the path. - **Breaking:** the check that closes `/_ts/admin` and `/admin/keys` on every adapter now also answers `404` to a request path still changing after four rounds of percent-decoding and to a path that reaches either once an escape that is not valid UTF-8 is replaced. The publisher received both before, and no setting turns the check off. - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServicesBuilder::geo`, `RuntimeServices::with_geo` and `IntegrationRegistrationBuilder::with_geo_module` take an `Arc`, and `RuntimeServices::geo()`, `IntegrationRegistry::geo_module()`, the `IntegrationRegistration::geo_module` field and `build_geo_module` return or hold one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths build the call from the request. @@ -102,6 +103,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- A serve middleware may set headers on a response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. The method is not called for a refusal made before the origin is asked, such as the `405` for a method a signed origin does not take, for a page served from the template cache or the refusal of such a page, or for an answer made from an error. - A publisher's site may be served from a folder of object storage through the publisher proxy, so its pages pass through the page pipeline, which an asset route skips. The folder is the path of `origin_url`, and two optional `[publisher]` settings do the rest, each off unless set. `origin_auth` takes the table an asset route's `auth` takes and signs every origin request with it, through the same signer. `directory_index` names a file added to a request path ending in `/`, for the origin request only. A signed origin is only read, so a method other than `GET` or `HEAD` is answered `405`, and a signed request carries none of the reader's cookies, credentials, referrer, conditional validators or range. A malformed `origin_auth` or `directory_index` is refused when the settings load, with the setting named. - `[ad-tag.google] write_loader` writes the loader tag, for a page that carries no loader of its own, through the serve middleware `ad-tag.google.loader`, which a `[[serve]]` entry names and `ts audit` writes where it finds GPT. The tag goes in the head after the bootstrap and asks for the loader on the module's own route. Only a reader whose permissions allow storage on the device (`necessary.operations.storage`) gets it, because GPT loaded from Google's standard address may use the device's storage at any time. That is TCF Purpose 1, which core's server-side auction requires where GDPR applies and which the Google tag module's loaders require. Off by default, because a page that already carries the loader would otherwise load it twice. - An asset cache module may answer for a file before core asks its origin. Core offers each `GET` for a file that the publisher's origin sends or an asset route fetches, other than one the platform's image optimizer answers, to the module that `[asset-cache]` selects, with what it knows of the reader's request and the request it would send. The module serves a response, which core handles as it would the origin's, reports that its own origin request failed, which core answers as a failed origin request, or passes the file back, and a deployment that selects none has every file asked of its origin. A registration supplies one with `with_asset_cache_module`, and the registry refuses one from a builder that `[asset-cache]` does not select, an `[asset-cache]` that selects two, and one that selects a builder that supplies none, naming that builder. The new `crates/asset-cache/51degrees`, the module `asset-cache.51degrees`, which no stock build links, holds each file as its origin sent it, under what the origin's `Vary` names and for the origin's own lifetime, in two loading caches it declares, `51degrees-assets` for the copies and `51degrees-assets-vary` for each address's record, each selecting its own loading cache module and both sized by `[cache.51degrees-assets]`. It keeps nothing that may not be shared, serves a copy with an `Age` that counts from the origin's, and on the publisher path leaves a document, and every request that carries a cookie, to the origin. A load that fails asking the origin is the answer to every request waiting on it, so a failing origin is asked once. A table that names `none` is refused, as one naming any module no linked crate offers is. diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 55ac9a53b3..5419fa1b54 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -2297,6 +2297,22 @@ impl IntegrationRegistry { MiddlewareChain::new(phase, media_type, middleware) } + /// The middleware the serve phase's headers step asks for `path`: every + /// one an entry whose path covers it names, see + /// [`PhaseEntries::for_path`]. + #[must_use] + pub fn middleware_for_path( + &self, + entries: &PhaseEntries, + path: &str, + ) -> Vec> { + entries + .for_path(path) + .into_iter() + .filter_map(|name| self.inner.middleware_named(name).map(Arc::clone)) + .collect() + } + /// Whether a middleware that an entry of either phase names for a request /// for `path`, under any media type, needs the origin's whole body, see /// [`Middleware::needs_whole_body`]. Asked before the origin answers, so @@ -3307,7 +3323,7 @@ pub(crate) mod test_support { use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; use crate::middleware::{ AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, - Preparation, Refusal, + Preparation, Refusal, ResponseHeader, ResponseView, }; use crate::module_context::ModuleCall; use crate::settings::Settings; @@ -3497,6 +3513,62 @@ pub(crate) mod test_support { MiddlewareAction::pass() } } + /// The name of the serve middleware that sets headers through the + /// headers step alone. + pub(crate) const LOCATING: &str = "testing.middleware-fixture.locating"; + pub(crate) const LOCATION_PREFIX: &str = "/fixture"; + /// The header the locating middleware sets on every response it is + /// shown, carrying that response's status. + pub(crate) const SHOWN_STATUS_NAME: &str = "x-fixture-shown-status"; + + /// Puts a prefix on a redirect's root-relative `Location`, which is + /// what a deployment that names the publisher in the first part of the + /// address needs, and names the status of every response it is shown. + struct Locating; + + impl Middleware for Locating { + fn middleware_id(&self) -> &'static str { + LOCATING + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction::pass() + } + + fn response_headers( + &self, + _context: &MiddlewareContext<'_>, + response: &ResponseView<'_>, + ) -> Vec { + let mut headers = vec![( + http::HeaderName::from_static(SHOWN_STATUS_NAME), + http::HeaderValue::from(response.status.as_u16()), + )]; + if !response.status.is_redirection() { + return headers; + } + let Some(location) = response + .headers + .get(http::header::LOCATION) + .and_then(|value| value.to_str().ok()) + else { + return headers; + }; + if location.starts_with('/') + && !location.starts_with("//") + && let Ok(value) = + http::HeaderValue::from_str(&format!("{LOCATION_PREFIX}{location}")) + { + headers.push((http::header::LOCATION, value)); + } + headers + } + } + /// The name of the serve middleware that writes a header on a text /// response. pub(crate) const TEXT_HEADER: &str = "testing.middleware-fixture.text-header"; @@ -3733,6 +3805,7 @@ pub(crate) mod test_support { .with_middleware(Arc::new(Preparing)) .with_middleware(Arc::new(Refusing)) .with_middleware(Arc::new(TextHeader)) + .with_middleware(Arc::new(Locating)) .build(), )) } @@ -7106,6 +7179,7 @@ mod tests { fixture::PREPARING, fixture::REFUSING, fixture::TEXT_HEADER, + fixture::LOCATING, crate::service_version::MODULE, ], "should list the middleware the selected module supplies, and core's own after them" @@ -7129,6 +7203,7 @@ mod tests { fixture::BROKEN_READER, fixture::PREPARING, fixture::TEXT_HEADER, + fixture::LOCATING, crate::service_version::MODULE ], "should list the middleware that run in the serve phase, core's own last" @@ -7150,6 +7225,37 @@ mod tests { ); } + #[test] + fn middleware_for_path_gives_every_covering_entry_s_middleware_once_in_entry_order() { + let registry = IntegrationRegistry::new(&fixture_settings_with_entries(Vec::new())) + .expect("should build a registry"); + let entries = PhaseEntries::new(vec![ + html_entry(Some("/news/"), &[fixture::LOCATING, fixture::READER]), + text_entry(&[fixture::TEXT_HEADER, fixture::LOCATING]), + html_entry(Some("/sport/"), &[fixture::BROKEN_READER]), + html_entry(None, &["testing.nothing", fixture::READER]), + ]); + + let ids = |path: &str| -> Vec<&'static str> { + registry + .middleware_for_path(&entries, path) + .iter() + .map(|middleware| middleware.middleware_id()) + .collect() + }; + assert_eq!( + ids("/news/today"), + [fixture::LOCATING, fixture::READER, fixture::TEXT_HEADER], + "should give each middleware once, in entry order, whatever the entry's media type" + ); + assert_eq!( + ids("/weather/"), + [fixture::TEXT_HEADER, fixture::LOCATING, fixture::READER], + "should leave out an entry whose path does not cover the request, and a name \ + nothing supplies" + ); + } + /// An entry for `text/plain` naming the middleware that works on it. fn text_entry(names: &[&str]) -> PhaseEntry { PhaseEntry { @@ -7245,7 +7351,7 @@ mod tests { message.contains( "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies" ) && message.contains(&format!( - "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}]", + "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}]", fixture::HEAD, fixture::LINKS, fixture::BROKEN, @@ -7255,6 +7361,7 @@ mod tests { fixture::PREPARING, fixture::REFUSING, fixture::TEXT_HEADER, + fixture::LOCATING, crate::service_version::MODULE )), "should name the entry and list what could be named: {message}" @@ -7311,6 +7418,7 @@ mod tests { fixture::PREPARING, fixture::REFUSING, fixture::TEXT_HEADER, + fixture::LOCATING, ], "should list every middleware when there are no entries" ); @@ -7326,7 +7434,8 @@ mod tests { fixture::TEXT, fixture::PREPARING, fixture::REFUSING, - fixture::TEXT_HEADER + fixture::TEXT_HEADER, + fixture::LOCATING ], "should leave out a middleware any entry names" ); @@ -7397,7 +7506,8 @@ mod tests { fixture::TEXT, fixture::PREPARING, fixture::REFUSING, - fixture::TEXT_HEADER + fixture::TEXT_HEADER, + fixture::LOCATING ], "should count a middleware as named only by an entry of a phase it runs in" ); diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index fbf8b1c726..1af2de7ecd 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -276,6 +276,15 @@ pub trait TextHandler { /// A header a middleware writes on the reader's response. pub type ResponseHeader = (HeaderName, HeaderValue); +/// What a serve middleware is told about the response it may set a header +/// on, read only: the status and the headers as they stand after core's own +/// rewrites, before the adapter finalizes the response. +#[derive(Debug, Clone, Copy)] +pub struct ResponseView<'a> { + pub status: StatusCode, + pub headers: &'a http::HeaderMap, +} + /// Writes `headers` on `response`, each replacing a header of the same name. pub fn write_response_headers(response: &mut http::Response, headers: Vec) { for (name, value) in headers { @@ -572,6 +581,22 @@ pub trait Middleware: Send + Sync { /// Decides what to do with one document. fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction; + /// The headers a serve middleware sets on the response the publisher path + /// sends, a redirect with no body included, for a request path its entry + /// covers. Each replaces a + /// header of the same name, and one the action from + /// [`create`](Self::create) sets in + /// [`MiddlewareAction::response_headers`] replaces it in turn. None by + /// default. + fn response_headers( + &self, + context: &MiddlewareContext<'_>, + response: &ResponseView<'_>, + ) -> Vec { + let _ = (context, response); + Vec::new() + } + /// Text a stored page is keyed by, being what this middleware writes /// that no setting decides, such as markup compiled into its crate. A /// change to it stores pages again, where otherwise a page stored before @@ -661,6 +686,23 @@ impl PhaseEntries { .map_or(&[][..], |entry| entry.middleware.as_slice()) } + /// The middleware every entry whose path covers `path` names, in entry + /// order and each once, whatever the entries' media types. A header is + /// not about a body, so this is what the headers step of the serve phase + /// asks. + #[must_use] + pub fn for_path(&self, path: &str) -> Vec<&str> { + let mut names: Vec<&str> = Vec::new(); + for entry in self.0.iter().filter(|entry| entry.covers_path(path)) { + for name in &entry.middleware { + if !names.contains(&name.as_str()) { + names.push(name); + } + } + } + names + } + /// Every name the entries select, each once, in the order first written. #[must_use] pub fn names(&self) -> Vec<&str> { diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index 087accdd48..5265ba03c9 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -312,6 +312,33 @@ fn names_lists_each_middleware_once_in_the_order_first_written() { assert_eq!(entries.names(), ["example.b", "example.a", "example.c"]); } +#[test] +fn for_path_names_every_entry_covering_the_path_whatever_its_media_type() { + let mut text = entry(Some("/news/"), &["example.text", "example.a"]); + text.media_type = "text/plain".to_owned(); + let entries = PhaseEntries::new(vec![ + entry(Some("/news/"), &["example.a", "example.b"]), + text, + entry(Some("/sport/"), &["example.sport"]), + entry(None, &["example.b", "example.all"]), + ]); + + assert_eq!( + entries.for_path("/news/today"), + ["example.a", "example.b", "example.text", "example.all"], + "should name each middleware once, in entry order, from every entry covering the path" + ); + assert_eq!( + entries.for_path("/weather/"), + ["example.b", "example.all"], + "should leave out an entry whose path does not cover the request" + ); + assert!( + PhaseEntries::default().for_path("/").is_empty(), + "should name nothing when there are no entries" + ); +} + #[test] fn entries_from_the_longest_path_to_every_path_are_accepted() { PhaseEntries::new(vec![ diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index d176448509..db0548fcb3 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3885,6 +3885,44 @@ fn publisher_origin_path_and_query<'a>( Cow::Owned(built) } +/// Writes on `response` the headers each serve middleware an entry names for +/// `request_path` sets, see +/// [`Middleware::response_headers`](crate::middleware::Middleware::response_headers). +fn write_serve_middleware_headers( + response: &mut Response, + reader: &crate::html_processor::ReaderDocument<'_>, + request_path: &str, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) { + let middleware = integration_registry + .middleware_for_path(settings.phase_entries(MiddlewarePhase::Serve), request_path); + if middleware.is_empty() { + return; + } + let document_state = IntegrationDocumentState::default(); + reader.request_state.seed(&document_state); + let context = crate::middleware::MiddlewareContext { + phase: MiddlewarePhase::Serve, + request_host: reader.request_host, + request_scheme: reader.request_scheme, + origin_host: reader.origin_host, + platform: reader.platform, + document_state: &document_state, + max_buffered_script_bytes: reader.max_buffered_script_bytes, + prepared: reader.prepared, + }; + let view = crate::middleware::ResponseView { + status: response.status(), + headers: response.headers(), + }; + let headers = middleware + .iter() + .flat_map(|middleware| middleware.response_headers(&context, &view)) + .collect(); + write_response_headers(response, headers); +} + /// Returns the status a publisher origin request is refused with before it is /// sent, or `None` when it may be sent. /// @@ -6217,6 +6255,20 @@ pub async fn handle_publisher_request( response = empty_document(&entry.media_type)?; } + // Every serve middleware an entry names for this path is asked for the + // headers it sets on the response that is sent, whatever it is, so a + // redirect with no body reaches a middleware that rewrites its Location. + let header_prepared = Prepared::default(); + let header_reader = crate::html_processor::ReaderDocument { + request_host, + request_scheme, + origin_host: &origin_host, + platform: integration_registry.platform(), + request_state: &request_state, + max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, + prepared: &header_prepared, + }; + if should_run_ad_stack && response.status() == StatusCode::NOT_MODIFIED { if let Some(dispatched) = dispatched_auction.take() { emit_abandoned_auction( @@ -6237,10 +6289,16 @@ pub async fn handle_publisher_request( .change_context(TrustedServerError::Proxy { message: "failed to build unexpected origin 304 response".to_string(), })?; + write_serve_middleware_headers( + &mut response, + &header_reader, + &request_path, + settings, + integration_registry, + ); enforce_terminal_private_cache_privacy(&mut response); return Ok(PublisherResponse::Buffered(response)); } - // Map origin URLs in navigation, preload, and CSP headers to the serving host, // matching the body rewrite, on every route. Runs before the template-cache gate // so a stored template replays the rewritten policy headers; the cache key is @@ -6250,6 +6308,7 @@ pub async fn handle_publisher_request( &OriginHeaderRewrite { origin_host: &origin_host, origin_host_header: &origin_host_header, + origin_path: parsed_origin.path(), request_host, request_scheme, origin_scheme: &origin_scheme, @@ -6257,6 +6316,15 @@ pub async fn handle_publisher_request( request_is_get_or_head: is_get_or_head, }, ); + // Asked after the rewrite, so a serve middleware sees the headers the + // reader receives and a header it sets is the one sent. + write_serve_middleware_headers( + &mut response, + &header_reader, + &request_path, + settings, + integration_registry, + ); let template_cache_policy = TemplateCachePolicy::from_settings(settings); let gate_content_type = response @@ -6474,6 +6542,13 @@ pub async fn handle_publisher_request( .await; } let mut response = refusal_response(&refusal)?; + write_serve_middleware_headers( + &mut response, + &header_reader, + &request_path, + settings, + integration_registry, + ); enforce_terminal_private_cache_privacy(&mut response); return Ok(PublisherResponse::Buffered(response)); } @@ -20988,6 +21063,53 @@ mod tests { } } + #[tokio::test] + async fn the_502_for_an_unexpected_origin_304_carries_a_serve_middlewares_headers() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_dispatching_provider(); + settings.select_module("testing", fixture::MODULE); + settings.serve = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::LOCATING.to_owned()], + }]); + let mut orchestrator = AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + ); + orchestrator.register_provider(Arc::new(DispatchingTestProvider)); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, b"unused provider response".to_vec()); + stub.push_response_with_headers(304, Vec::new(), vec![("etag", ORIGIN_ETAG)]); + let services = services_with_telemetry( + Arc::clone(&stub) as Arc, + Arc::new(RecordingTelemetrySink::default()), + ); + + let response = run_with_orchestrator( + &settings, + &services, + &orchestrator, + &[article_slot()], + conditional_navigation_request(), + ) + .await; + + let PublisherResponse::Buffered(response) = response else { + panic!("unexpected origin 304 should return a buffered response") + }; + assert_eq!(response.status(), StatusCode::BAD_GATEWAY); + assert_eq!( + response + .headers() + .get(fixture::SHOWN_STATUS_NAME) + .and_then(|value| value.to_str().ok()), + Some("502"), + "should set the middleware's headers on the 502, which is what is sent" + ); + } + #[tokio::test] async fn inactive_ad_stack_304_uses_short_browser_cache_policy() { // Arrange @@ -24132,6 +24254,32 @@ mod tests { ); } + #[tokio::test] + async fn a_made_file_carries_the_headers_a_serve_middleware_sets() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = + settings_running_the_middleware_fixture(text_entry_for(Some("/ads.txt"))); + settings.serve = PhaseEntries::new(vec![PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::LOCATING.to_owned()], + }]); + + let proxied = proxy_with(settings, Method::GET, "/ads.txt", 404).await; + let response = response_of(proxied.response); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response + .headers() + .get(fixture::SHOWN_STATUS_NAME) + .and_then(|value| value.to_str().ok()), + Some("200"), + "should set the middleware's headers on the made file, which is what is sent" + ); + } + #[tokio::test] async fn an_html_entry_with_a_path_makes_no_missing_page() { use crate::integrations::registry_test_support::middleware_fixture as fixture; @@ -24979,7 +25127,7 @@ mod tests { .to_owned() } - fn response_status(response: &PublisherResponse) -> StatusCode { + pub(super) fn response_status(response: &PublisherResponse) -> StatusCode { match response { PublisherResponse::Buffered(response) | PublisherResponse::PassThrough { response, .. } @@ -25458,6 +25606,451 @@ mod tests { } } + /// A redirect the origin sends to itself, named for the reader on every + /// route the publisher proxy returns an origin response through. + mod origin_redirect_tests { + use super::object_storage_origin_tests::response_status; + use super::*; + use crate::platform::PlatformHttpClient; + + const ORIGIN_URL: &str = "https://origin.example"; + const STORAGE_URL: &str = "https://storage.example.com/bucket/site"; + const READER_HOST: &str = "publisher.example"; + + fn settings_with_origin(origin_url: &str) -> Settings { + let mut settings = create_test_settings(); + settings.publisher.origin_url = origin_url.to_owned(); + settings + } + + /// A reader request for `uri` on [`READER_HOST`], made over `scheme`. + fn reader_request(method: Method, uri: &str, scheme: &str) -> HttpRequest { + HttpRequest::builder() + .method(method) + .uri(uri) + .header(header::HOST, READER_HOST) + .header("x-forwarded-proto", scheme) + .body(EdgeBody::empty()) + .expect("should build reader request") + } + + /// What the publisher proxy returns for `request` when the origin + /// answers with `status` and `headers`, and the addresses the origin + /// was sent. + async fn proxied( + settings: &Settings, + request: HttpRequest, + status: u16, + headers: Vec<(&str, &str)>, + ) -> (PublisherResponse, Vec) { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers(status, b"origin response".to_vec(), headers); + let services = + build_services_with_http_client(Arc::clone(&stub) as Arc); + let response = run_publisher_proxy(settings, &services, request).await; + (response, stub.recorded_request_uris()) + } + + /// The `Location` the reader receives. + fn location(response: &PublisherResponse) -> Option<&str> { + let response = match response { + PublisherResponse::Buffered(response) + | PublisherResponse::PassThrough { response, .. } + | PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } => response, + }; + response + .headers() + .get(header::LOCATION) + .and_then(|value| value.to_str().ok()) + } + + fn route(response: &PublisherResponse) -> &'static str { + match response { + PublisherResponse::Buffered(_) => "buffered", + PublisherResponse::PassThrough { .. } => "pass-through", + PublisherResponse::Stream { .. } => "stream", + PublisherResponse::AssembleTemplate { .. } => "template", + } + } + + #[tokio::test] + async fn the_reader_receives_the_rewritten_location_on_every_route() { + let settings = settings_with_origin(ORIGIN_URL); + // (what is asked for, method, the origin's status, the origin's + // media type, the route the answer takes) + let cases = [ + ("a page, buffered", Method::GET, 301, None, "buffered"), + ( + "a page, streamed", + Method::GET, + 301, + Some("text/html; charset=utf-8"), + "stream", + ), + ("a found page", Method::GET, 302, None, "buffered"), + ("a see other", Method::POST, 303, None, "buffered"), + ("a HEAD", Method::HEAD, 301, None, "buffered"), + ( + "a HEAD of a page", + Method::HEAD, + 302, + Some("text/html"), + "stream", + ), + ("a POST", Method::POST, 307, None, "buffered"), + ("a PUT", Method::PUT, 308, None, "buffered"), + ("a DELETE", Method::DELETE, 307, None, "buffered"), + ]; + for (what, method, status, media_type, expected_route) in cases { + let mut headers = vec![("location", "https://origin.example/news/?page=2#latest")]; + if let Some(media_type) = media_type { + headers.push(("content-type", media_type)); + } + + let (response, sent) = proxied( + &settings, + reader_request(method, "https://publisher.example/news?page=2", "https"), + status, + headers, + ) + .await; + + assert_eq!( + sent, + vec!["https://origin.example/news?page=2"], + "should ask the origin for {what}" + ); + assert_eq!( + route(&response), + expected_route, + "should return {what} through the expected route" + ); + assert_eq!( + location(&response), + Some("https://publisher.example/news/?page=2#latest"), + "should name the redirect for the reader on {what}" + ); + } + } + + #[tokio::test] + async fn the_reader_s_own_scheme_is_used() { + let settings = settings_with_origin(ORIGIN_URL); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "http://publisher.example/news", "http"), + 301, + vec![("location", "https://origin.example/news/")], + ) + .await; + + assert_eq!( + location(&response), + Some("http://publisher.example/news/"), + "should send a reader on plain HTTP to plain HTTP" + ); + } + + #[tokio::test] + async fn an_origin_on_plain_http_or_on_a_port_is_named_for_the_reader() { + // (origin URL, the origin's `Location`, what the reader receives) + let cases = [ + ( + "http://origin.example", + "http://origin.example/news/", + "https://publisher.example/news/", + ), + ( + "http://origin.example", + "https://origin.example/news/", + "https://publisher.example/news/", + ), + ( + "http://origin.example:8080", + "http://origin.example:8080/news/", + "https://publisher.example/news/", + ), + ( + "http://origin.example:8080", + "http://origin.example/news/", + "http://origin.example/news/", + ), + ( + "https://origin.example:8443", + "https://origin.example:8443/news/", + "https://publisher.example/news/", + ), + ( + "https://origin.example:8443", + "https://origin.example/news/", + "https://origin.example/news/", + ), + ( + "https://origin.example", + "https://origin.example:8443/news/", + "https://origin.example:8443/news/", + ), + ]; + for (origin_url, sent, expected) in cases { + let settings = settings_with_origin(origin_url); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", sent)], + ) + .await; + + assert_eq!( + location(&response), + Some(expected), + "should give the reader `{expected}` for `{sent}` from `{origin_url}`" + ); + } + } + + #[tokio::test] + async fn under_a_host_override_the_host_of_origin_url_is_the_origin_s_name() { + let mut settings = settings_with_origin(ORIGIN_URL); + settings.publisher.origin_host_header_override = Some("www.example.com".to_owned()); + // (the origin's `Location`, what the reader receives) + let cases = [ + ( + "https://origin.example/news/", + "https://publisher.example/news/", + ), + ( + "https://www.example.com/news/", + "https://www.example.com/news/", + ), + ]; + for (sent, expected) in cases { + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", sent)], + ) + .await; + + assert_eq!( + location(&response), + Some(expected), + "should treat the host of `origin_url` as the origin's name for `{sent}`" + ); + } + } + + #[tokio::test] + async fn a_redirect_under_the_origin_path_reaches_the_reader_without_it() { + let settings = settings_with_origin(STORAGE_URL); + + let (response, sent) = proxied( + &settings, + reader_request( + Method::GET, + "https://publisher.example/news?page=2", + "https", + ), + 301, + vec![( + "location", + "https://storage.example.com/bucket/site/news/?page=2", + )], + ) + .await; + + assert_eq!( + sent, + vec!["https://storage.example.com/bucket/site/news?page=2"], + "should ask the origin under the origin path" + ); + assert_eq!( + location(&response), + Some("https://publisher.example/news/?page=2"), + "should send the reader to their own path, without the origin path" + ); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", "https://storage.example.com/bucket/other/news/")], + ) + .await; + + assert_eq!( + location(&response), + Some("https://storage.example.com/bucket/other/news/"), + "should leave a redirect outside the origin path as the origin wrote it" + ); + } + + #[tokio::test] + async fn a_redirect_that_is_not_the_origin_s_own_reaches_the_reader_unchanged() { + let settings = settings_with_origin(ORIGIN_URL); + // (why, the origin's `Location`) + let cases = [ + ("another host", "https://somewhere.else/news/"), + ("a path from the root", "/news/"), + ("a path beside the one asked for", "news/"), + ("a query alone", "?page=2"), + ]; + for (why, sent) in cases { + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + vec![("location", sent)], + ) + .await; + + assert_eq!( + response_status(&response), + StatusCode::FOUND, + "should relay the redirect for {why}" + ); + assert_eq!( + location(&response), + Some(sent), + "should leave {why} as the origin wrote it" + ); + } + } + + #[tokio::test] + async fn a_location_on_an_answer_that_is_no_redirect_is_named_for_the_reader() { + let settings = settings_with_origin(ORIGIN_URL); + // (the origin's status, the origin's media type, the route the + // answer takes) + let cases = [ + (200, None, "pass-through"), + (200, Some("text/html; charset=utf-8"), "stream"), + (201, None, "pass-through"), + (404, None, "buffered"), + (503, Some("text/html"), "stream"), + ]; + for (status, media_type, expected_route) in cases { + let mut headers = vec![("location", "https://origin.example/news/")]; + if let Some(media_type) = media_type { + headers.push(("content-type", media_type)); + } + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + status, + headers, + ) + .await; + + assert_eq!( + route(&response), + expected_route, + "should return a {status} through the expected route" + ); + assert_eq!( + location(&response), + Some("https://publisher.example/news/"), + "should name the `Location` of a {status} for the reader" + ); + } + } + + #[tokio::test] + async fn a_serve_middleware_named_for_the_path_sets_a_redirects_location() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_origin(ORIGIN_URL); + settings + .insert_module_config( + "testing", + fixture::MODULE, + &serde_json::json!({ "label": "locating" }), + ) + .expect("should select the middleware stand-in"); + settings.serve = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::LOCATING.to_owned()], + }]); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + vec![("location", "/uk")], + ) + .await; + assert_eq!( + location(&response), + Some("/fixture/uk"), + "should hand a bodiless redirect to the middleware and keep the header it set" + ); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", "https://somewhere.else/news/")], + ) + .await; + assert_eq!( + location(&response), + Some("https://somewhere.else/news/"), + "should leave a Location the middleware does not set" + ); + + let mut elsewhere = settings.clone(); + elsewhere.serve = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: Some("/other/".to_owned()), + middleware: vec![fixture::LOCATING.to_owned()], + }]); + let (response, _) = proxied( + &elsewhere, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + vec![("location", "/uk")], + ) + .await; + assert_eq!( + location(&response), + Some("/uk"), + "should not ask a middleware whose entry covers another path" + ); + } + + #[tokio::test] + async fn a_redirect_with_no_location_gains_none() { + let settings = settings_with_origin(ORIGIN_URL); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + Vec::new(), + ) + .await; + + assert_eq!( + response_status(&response), + StatusCode::FOUND, + "should relay the redirect" + ); + assert_eq!( + location(&response), + None, + "should add no `Location` the origin did not send" + ); + } + } + #[test] fn stream_publisher_body_preserves_gzip_round_trip() { use flate2::write::GzEncoder; diff --git a/crates/trusted-server-core/src/response_header_rewrite.rs b/crates/trusted-server-core/src/response_header_rewrite.rs index c2a1d1c247..52149d2733 100644 --- a/crates/trusted-server-core/src/response_header_rewrite.rs +++ b/crates/trusted-server-core/src/response_header_rewrite.rs @@ -12,7 +12,8 @@ //! `Refresh`, `Link`): an absolute or protocol-relative URL whose authority is //! the publisher origin is mapped to the serving host, using the same //! scheme mapping as the HTML body rewriter. Relative URLs and other hosts -//! are left unchanged. +//! are left unchanged. When the origin is a folder of its host, only a URL in +//! that folder is mapped, and it loses the folder. //! - **Content Security Policy** (`Content-Security-Policy`, //! `Content-Security-Policy-Report-Only`): every source expression that names //! the publisher origin gains a sibling expression for the serving host. @@ -24,6 +25,8 @@ //! preserved. Operator `[response_headers]` overrides are applied later and //! still take precedence. +use std::borrow::Cow; + use edgezero_core::http::{HeaderMap, HeaderName, HeaderValue, header}; use url::Url; @@ -50,6 +53,13 @@ pub(crate) struct OriginHeaderRewrite<'a> { /// point of view, so the retry through Trusted Server is redirected again /// whatever cookies the response sets. pub origin_host_header: &'a str, + /// Path of the publisher origin's folder (`/bucket/site`), or empty or `/` + /// when the origin is the whole host. + /// + /// A URL on the origin host is the origin's only at or beneath this path, + /// and maps to the serving host without it. One outside the folder has no + /// address on the serving host and is left unchanged. + pub origin_path: &'a str, /// Serving host authority the browser addressed. pub request_host: &'a str, /// Scheme the browser used to reach the serving host. @@ -84,23 +94,57 @@ impl OriginHeaderRewrite<'_> { /// Rewrite a URL reference that points at the publisher origin. /// - /// Returns [`None`] when `url` is relative, names another host, or names the - /// origin host with a different port. + /// Returns [`None`] when `url` is relative, names another host, names the + /// origin host with a different port, or names a path outside the origin + /// folder. fn rewrite_url(&self, url: &str) -> Option { let rest = strip_prefix_ignore_ascii_case(url, "https://") .or_else(|| strip_prefix_ignore_ascii_case(url, "http://")); if let Some(rest) = rest { - let suffix = self.strip_origin_authority(rest)?; + let suffix = self.serving_suffix(self.strip_origin_authority(rest)?)?; return Some(format!( "{}://{}{suffix}", self.request_scheme, self.request_host )); } - let suffix = self.strip_origin_authority(url.strip_prefix("//")?)?; + let suffix = self.serving_suffix(self.strip_origin_authority(url.strip_prefix("//")?)?)?; Some(format!("//{}{suffix}", self.request_host)) } + /// What follows the serving host for `suffix`, the path, query and + /// fragment that follow the origin authority in a URL on the origin. + /// + /// With an [`Self::origin_path`], dot segments are resolved first, as the + /// browser resolves them, so a path that climbs out of the folder is + /// outside it, and [`None`] is returned for a path outside the folder. + fn serving_suffix<'b>(&self, suffix: &'b str) -> Option> { + if self.origin_path.trim_end_matches('/').is_empty() { + return Some(Cow::Borrowed(suffix)); + } + let target = Url::parse(&format!( + "{}://{}{suffix}", + self.origin_scheme, self.origin_host + )) + .ok()?; + let path = self.serving_path(target.path())?; + Some(Cow::Owned(format!( + "{path}{}", + &target[url::Position::AfterPath..] + ))) + } + + /// The serving-host path for `path`, a path on the origin host, being + /// `path` without [`Self::origin_path`], or [`None`] when `path` is outside + /// that folder. The folder counts only where it ends at a `/`, so + /// `/bucket/site` does not come off `/bucket/sitemap.xml`. + fn serving_path<'b>(&self, path: &'b str) -> Option<&'b str> { + let path = path + .strip_prefix(self.origin_path.trim_end_matches('/')) + .filter(|path| path.is_empty() || path.starts_with('/'))?; + Some(if path.is_empty() { "/" } else { path }) + } + /// Rewrite a navigation target (`Location`, `Refresh`), unless rewriting /// it would loop. /// @@ -195,7 +239,7 @@ impl OriginHeaderRewrite<'_> { Some((path, query)) => (path, Some(query)), None => (self.request_path_and_query, None), }; - target.path() == request_path && target.query() == request_query + self.serving_path(target.path()) == Some(request_path) && target.query() == request_query } /// Strip the origin authority from the start of `rest`, returning the @@ -208,15 +252,27 @@ impl OriginHeaderRewrite<'_> { /// Build the serving-host CSP source for a source expression naming the /// origin, or [`None`] when the expression names something else. + /// + /// A source with no path permits every path on the origin, so its sibling + /// names the whole serving host even when the origin is a folder. fn csp_source_for(&self, source: &str) -> Option { if source.starts_with('\'') { return None; } + let rest = source.split_once("://").map_or(source, |(_, rest)| rest); + let mapping = if self.strip_origin_authority(rest)?.is_empty() { + OriginHeaderRewrite { + origin_path: "", + ..*self + } + } else { + *self + }; if source.contains("://") { - return self.rewrite_url(source); + return mapping.rewrite_url(source); } - let suffix = self.strip_origin_authority(source)?; - Some(format!("{}{suffix}", self.request_host)) + let suffix = mapping.serving_suffix(mapping.strip_origin_authority(source)?)?; + Some(format!("{}{suffix}", mapping.request_host)) } } @@ -232,6 +288,7 @@ impl OriginHeaderRewrite<'_> { /// let rewrite = OriginHeaderRewrite { /// origin_host: "origin.example.com", /// origin_host_header: "origin.example.com", +/// origin_path: "", /// request_host: "www.example.com", /// request_scheme: "https", /// origin_scheme: "https", @@ -564,6 +621,7 @@ mod tests { const REWRITE: OriginHeaderRewrite<'static> = OriginHeaderRewrite { origin_host: "origin.example.com", origin_host_header: "origin.example.com", + origin_path: "", request_host: "www.example.com", request_scheme: "https", origin_scheme: "http", @@ -571,6 +629,15 @@ mod tests { request_is_get_or_head: false, }; + /// An origin that is a folder of object storage, served at `www.example.com`. + const FOLDER: OriginHeaderRewrite<'static> = OriginHeaderRewrite { + origin_host: "storage.example.com", + origin_host_header: "storage.example.com", + origin_path: "/bucket/site", + origin_scheme: "https", + ..REWRITE + }; + fn rewritten(name: &HeaderName, values: &[&str]) -> Vec { let mut headers = HeaderMap::new(); for value in values { @@ -645,6 +712,7 @@ mod tests { let rewrite = OriginHeaderRewrite { origin_host: "127.0.0.1:8301", origin_host_header: "127.0.0.1:8301", + origin_path: "", request_host: "127.0.0.1:3031", request_scheme: "http", origin_scheme: "http", @@ -1064,6 +1132,163 @@ mod tests { } } + #[test] + fn folder_origin_maps_urls_in_the_folder_without_it() { + // (why, URL on the origin, URL on the serving host) + let cases = [ + ( + "a path in the folder loses the folder", + "https://storage.example.com/bucket/site/about/", + "https://www.example.com/about/", + ), + ( + "the query and the fragment are kept", + "https://storage.example.com/bucket/site/about/index.html?v=2#team", + "https://www.example.com/about/index.html?v=2#team", + ), + ( + "the folder itself is the serving host's root", + "https://storage.example.com/bucket/site", + "https://www.example.com/", + ), + ( + "the folder with a closing slash is the serving host's root", + "https://storage.example.com/bucket/site/", + "https://www.example.com/", + ), + ( + "a dot segment that stays in the folder is resolved first", + "https://storage.example.com/bucket/site/about/../news/", + "https://www.example.com/news/", + ), + ( + "a protocol-relative URL stays protocol-relative", + "//storage.example.com/bucket/site/about/", + "//www.example.com/about/", + ), + ]; + for (why, url, expected) in cases { + assert_eq!( + FOLDER.rewrite_url(url).as_deref(), + Some(expected), + "should map `{url}`: {why}" + ); + } + + let closing_slash = OriginHeaderRewrite { + origin_path: "/bucket/site/", + ..FOLDER + }; + assert_eq!( + closing_slash + .rewrite_url("https://storage.example.com/bucket/site/about/") + .as_deref(), + Some("https://www.example.com/about/"), + "should read a folder written with a closing slash as the same folder" + ); + let whole_host = OriginHeaderRewrite { + origin_path: "/", + ..FOLDER + }; + assert_eq!( + whole_host + .rewrite_url("https://storage.example.com/bucket/site/about/") + .as_deref(), + Some("https://www.example.com/bucket/site/about/"), + "should take nothing off for an origin path of `/`" + ); + } + + #[test] + fn folder_origin_leaves_urls_outside_the_folder_unchanged() { + // (why, URL on the origin host) + let cases = [ + ( + "another folder of the host", + "https://storage.example.com/bucket/other/about/", + ), + ( + "the folder counts only where it ends at a separator", + "https://storage.example.com/bucket/sitemap.xml", + ), + ( + "a folder whose name starts with the folder's is another folder", + "https://storage.example.com/bucket/site-two/about/", + ), + ( + "the host's root is above the folder", + "https://storage.example.com/", + ), + ( + "a URL that stops at the host names its root", + "https://storage.example.com", + ), + ( + "a dot segment that climbs out of the folder is outside it", + "https://storage.example.com/bucket/site/../other/about/", + ), + ( + "a path in another letter case is another path", + "https://storage.example.com/Bucket/Site/about/", + ), + ]; + for (why, url) in cases { + assert_eq!( + FOLDER.rewrite_url(url), + None, + "should leave `{url}` unchanged: {why}" + ); + } + } + + #[test] + fn folder_origin_keeps_redirects_to_the_current_request_that_would_loop() { + let rewrite = OriginHeaderRewrite { + request_path_and_query: "/news?page=2", + request_is_get_or_head: true, + ..FOLDER + }; + assert_eq!( + rewrite.rewrite_location("http://storage.example.com/bucket/site/news?page=2", true), + None, + "should keep a scheme change to the current request in the folder" + ); + assert_eq!( + rewrite.rewrite_location("https://storage.example.com/bucket/site/news?page=2", false), + None, + "should keep a cookieless GET redirect to the current request in the folder" + ); + assert_eq!( + rewrite + .rewrite_location( + "https://storage.example.com/bucket/site/news/?page=2", + false + ) + .as_deref(), + Some("https://www.example.com/news/?page=2"), + "should map a redirect to another address in the folder" + ); + } + + #[test] + fn folder_origin_csp_sources_follow_the_folder() { + assert_eq!( + rewrite_csp( + "img-src https://storage.example.com storage.example.com/bucket/site/img/ \ + storage.example.com/bucket/other/", + &FOLDER, + ) + .as_deref(), + Some( + "img-src https://storage.example.com https://www.example.com \ + storage.example.com/bucket/site/img/ www.example.com/img/ \ + storage.example.com/bucket/other/" + ), + "should give a source with no path the whole serving host, map one in the \ + folder without it, and add nothing beside one outside it" + ); + } + #[test] fn skips_rewrite_when_hosts_match_or_are_missing() { let mut headers = HeaderMap::new(); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index d040526fb5..9ebe17e3b4 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -461,6 +461,17 @@ is sent as it arrived, as before. When `origin_url` has a path, a request path with a `.` or `..` segment, in any percent-encoded spelling, is refused with `400` so no request reaches outside the folder. +**Response headers**: When `origin_url` has a path, a URL in a response +header that names the origin is mapped to the serving host only when it is in +that folder, and it loses the folder. With the folder above, `Location: +https://storage.publisher.com/bucket/site/news/` reaches a reader of +`www.publisher.com` as `https://www.publisher.com/news/`, and a URL outside +the folder is left as the origin wrote it (see Origin URLs in proxied response +headers). Where a request addressing hook removes a path prefix, the reader is +sent to an address without the prefix, which a Fastly service whose `__KEY` +selector carries `{host}` answers `421` if every selector value misses (see +One blob for each path). + #### `origin_host_header_override` **Purpose**: Optional Host header to send to the publisher origin while still @@ -1222,6 +1233,15 @@ Location: https://www.example.com/landing.html Content-Security-Policy: img-src https://origin.example.com https://www.example.com data: ``` +When `publisher.origin_url` has a path, such as a folder of object storage, +only a URL in that folder is mapped, and it loses the folder. With +`origin_url = "https://storage.example.com/bucket/site"`, +`https://storage.example.com/bucket/site/news/` becomes +`https://www.example.com/news/`, and `https://storage.example.com/bucket/other/` +is unchanged, because no address on the serving host leads there. A CSP source +with no path still gains the serving host beside it, and the two exceptions +below compare the path without the folder. + Two exceptions prevent a redirect loop. Rewriting either target would send the browser back to the same URL, and the origin would redirect again: @@ -2160,6 +2180,16 @@ alone. Entries are therefore written from the longest path to the entry with no path, and a folder is written with its closing slash, because `/news` also begins `/newsletter`. +A serve middleware may set response headers as well as change the page. For +that it is asked about a response the publisher path sends for a request path +its entry covers, whatever the response's media type and whether or not it +has a body, with the response's status and headers in hand, so a redirect +reaches a middleware that rewrites its `Location`. A header it sets replaces +one of the same name. A serve middleware is not asked about a refusal made +before the origin is asked, such as the `405` for a method a signed origin +does not take, about a page served from the template cache or the refusal of +such a page, or about an answer made from an error. + ```toml [[fetch]] media_type = "text/html" diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md new file mode 100644 index 0000000000..3c1262b2da --- /dev/null +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -0,0 +1,171 @@ +# Design Spec: A folder origin's addresses in response headers, and a headers step for serve middleware + +**Status:** Proposed, 2026-10-13. Raised as draft pull request IABTechLab/trusted-server#1378 (from jwrosewell/trusted-server#32, which is closed), for Tech Lab review. +**Author:** 51Degrees (contributed), for Tech Lab review +**Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` and `2026-10-13-request-addressing-design.md`, both in IABTechLab/trusted-server#1342, which holds every design document of the stack +**Related PRs:** IABTechLab/trusted-server#1378, IABTechLab/trusted-server#1238 (merged, the header rewrite this change extends), jwrosewell/trusted-server#32 (closed) +**Issues:** IABTechLab/trusted-server#1330, IABTechLab/trusted-server#1128 +**Last updated:** 2026-10-19 + +> **Why this spec exists.** The header rewrite merged in main (#1238) maps an address on the origin in a response header to the serving host, and knows the origin only as a host. Since the object storage design, the origin may be a folder of its host, and an address in that folder keeps the folder when it is mapped, so the reader is sent to an address that does not exist. The same change gives a serve middleware a way to set headers on a response that holds no page, a redirect among them. + +## 1. Problem + +The header rewrite (#1238) maps a URL on the origin in `Location`, `Content-Location`, `Refresh`, `Link` and the two Content Security Policy headers to the serving host, keeping its path, on every route of `handle_publisher_request`. The object storage design (`2026-10-13-publisher-object-storage-origin-design.md`) lets `origin_url` name a folder, such as `https://storage.example.com/bucket/site`, and a request for `/news` is sent to the origin as `/bucket/site/news`. An origin that redirects that request to `https://storage.example.com/bucket/site/news/` has the redirect mapped to `https://publisher.example/bucket/site/news/`. The reader's browser asks for that address, which the publisher proxy sends to the origin as `/bucket/site/bucket/site/news/`, an object that does not exist. An address on the origin's host outside the folder, such as `/bucket/other/`, is mapped the same way, though no address on the serving host leads there. The two exceptions the header rewrite makes to avoid a redirect loop compare the path the reader asked for with the path in the header, folder and all, so neither matches under a folder. + +A second gap sits beside the first. A serve middleware works on the page in a response and writes headers through the action it returns for that page, so a response with no page in it, such as most redirects, reaches no middleware. A deployment that wants a policy of its own for the `Location` of a redirect has nowhere to put the policy. + +## 2. Design + +There is no setting. The header rewrite is given the path of `origin_url`, the folder the object storage design sends requests into, and maps a URL on the origin only when the URL is in that folder, without the folder. With no path in `origin_url`, or a path of `/`, the mapping is the header rewrite's as merged. + +With `origin_url = "https://storage.example.com/bucket/site"` and a reader who asked for `https://publisher.example/news`: + +| The origin answers | The reader receives | +| -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| `301` with `Location: https://storage.example.com/bucket/site/news/` | `Location: https://publisher.example/news/` | +| `302` with `Location: https://storage.example.com/bucket/site/news/?page=2#latest` | `Location: https://publisher.example/news/?page=2#latest` | +| `301` with `Location: https://storage.example.com/bucket/site` | `Location: https://publisher.example/` | +| `301` with `Location: https://storage.example.com/bucket/other/news/` | The header as the origin wrote it | +| `301` with `Location: https://storage.example.com/bucket/sitemap.xml` | The header as the origin wrote it | +| `200` with `Link: ; rel=preload` | `Link: ; rel=preload` | +| `200` with `Content-Security-Policy: img-src https://storage.example.com storage.example.com/bucket/site/img/` | `Content-Security-Policy: img-src https://storage.example.com https://publisher.example storage.example.com/bucket/site/img/ publisher.example/img/` | + +**Which addresses are in the folder.** A URL on the origin, being one whose host and port the header rewrite already takes as the origin's, is in the folder when its path is the folder or beneath it. Dot segments are resolved first, as the reader's browser resolves them, so `/bucket/site/about/../news/` is `/bucket/site/news/` and in the folder, and `/bucket/site/../other/` is outside it. The folder counts only where it ends at a `/`, so `/bucket/site` does not come off `/bucket/sitemap.xml` or `/bucket/site-two/`. Paths are compared exactly, so `/Bucket/Site/` is not the folder. A folder written with a closing `/` is the same folder. The folder itself, with or without its closing `/`, is the serving host's root, `/`. A URL that stops at the origin's host names the host's root, which is outside the folder. + +**What the reader receives.** The URL as the header rewrite maps it, with the folder taken off the path and the query and fragment kept. A protocol-relative URL stays protocol-relative. + +**What is left alone.** A URL on the origin's host outside the folder is left as the origin wrote it, because no address on the serving host leads there, and the reader's browser follows it to the origin's host. + +**Content Security Policy sources.** A source expression with no path permits every path on the origin's host, so its serving-host sibling has no path either and permits the whole serving host, as without a folder. A source with a path in the folder gains a sibling without the folder. A source with a path outside the folder gains no sibling. + +**The loop exceptions.** The header rewrite leaves a `Location` or `Refresh` target unchanged where following it would bring the reader back to the address they asked for (#1238, and "Origin URLs in proxied response headers" in the configuration guide). With a folder, the path the reader asked for is compared with the path of the target without the folder, so both exceptions apply under a folder as they do without one. + +**What names the origin.** The header rewrite decides, and this change does not alter it. The origin is the host of `origin_url` with its port, in any letter case, with or without a `Host` override, and the header rewrite maps a `Location` on a response of any status. + +**A publisher addressed by a path prefix.** The request addressing design (`2026-10-13-request-addressing-design.md`) lets one host serve several publishers, each under a prefix of its own. The prefix is removed from the path before the request is routed, so the origin is asked for the path without it, and core never writes the removed prefix into a response. The header rewrite writes the request's scheme and host and the path of the URL, and nothing in `publisher.rs` reads the removed prefix, so the reader of such a publisher is sent to an address without the prefix, with or without a folder. With the example hook in the request addressing design, on a Fastly service whose `__KEY` selector is `{host}`, a reader of `https://front.example/publisher-a.example/news` whose origin answers `Location: https://origin.example/news/` receives `Location: https://front.example/news/`. The hook reads `news` as the publisher, so the reader is answered `421 Misdirected Request` unless a publisher has that name. A serve middleware puts the prefix back, through the headers step (section 3), and the step is given the rewritten `Location` and the request state a module's request preparer left. Core does not. No module in this repository reads the removed prefix, so none puts it back yet, and no test sends a request addressed by a path prefix through the header rewrite. A root-relative `Location`, such as `/news/`, is resolved by the reader's browser against the root of the host, so it loses the prefix in the same way. + +**Where it happens.** In the header rewrite, `rewrite_origin_urls_in_headers` in `response_header_rewrite.rs`, which `handle_publisher_request` in `publisher.rs` calls where #1238 placed it. The folder is the `origin_path` field this change adds to `OriginHeaderRewrite`, which the handler fills from the path of `origin_url`, the path the object storage design builds the origin address from. The call comes after a file that core makes where the origin has none has replaced the origin's answer, and before the template cache gate, so every route the handler returns an origin answer through passes it. A copy of an asset from the asset cache enters at the same point as the origin's own answer. The `502` core sends in place of an unexpected `304` returns before the call, and carries no header from the origin. The serve middleware headers step (section 3) runs straight after the header rewrite. + +**Adapters.** No adapter changes. The header rewrite needs a `3xx` to reach core with its `Location` as the origin wrote it, and that holds on all four adapters. The scheme and host it writes are the two values `RequestInfo::from_request` reads for the page pipeline, and how each adapter fills them differs. + +| Adapter | A `3xx` reaches core because | The scheme comes from | The host comes from | +| ---------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | +| Fastly | The client has no mode that follows a redirect | The connection's TLS signal, with `Forwarded`, `X-Forwarded-Host`, `X-Forwarded-Proto` and `Fastly-SSL` stripped first | The `Host` header | +| Spin | The client has no mode that follows a redirect | The runtime's own URL for the request, with the same headers stripped | The runtime's own URL for the request | +| Axum | The client's redirect policy is none | `Forwarded`, `X-Forwarded-Proto` or `Fastly-SSL` where the request carries one, otherwise `http` | `Forwarded` or `X-Forwarded-Host` where the request carries one, otherwise `Host` | +| Cloudflare | The client asks for manual redirect handling | As Axum | As Axum | + +The Axum and Cloudflare adapters set no TLS signal and strip only the configured client IP headers, so a client of either can choose the host its own redirect names, as it can already choose the host the page pipeline writes into a page. Each adapter already has a test that a redirect from its client is left for core to see, and those tests are not part of this change. + +**Consent and privacy.** The mapped URL is made from the request's scheme and host and the path, query and fragment of the origin's URL, and the folder only shortens the path. Nothing is added to the address, no consent state is read and no request is made, so the change has no effect on consent. + +**Why core and not a middleware.** The folder is part of the origin `[publisher]` names, and the header rewrite that maps the origin's addresses is in core (#1238). A middleware is told the host of `origin_url` and no origin path, and runs only where a `[[serve]]` entry covers the request path, so a deployment that named none would still send its readers to an address that does not exist. What a deployment decides for itself, such as where a redirect to another name of the site goes or whether to put a removed path prefix back, is a middleware's, through the headers step in section 3. + +**What changes for a deployment.** A deployment whose `origin_url` has no path sees no change. For one with a path, a URL in the folder reaches the reader without the folder, where the header rewrite alone kept it, and a URL on the origin's host outside the folder is relayed as the origin wrote it. No setting is added or changed. + +## 3. The serve middleware headers step + +A serve middleware may set headers on a response the publisher path sends, a redirect included, except for the responses listed under "Where the step is called" below. The header rewrite in section 2 does not use the step. A deployment's own policy for a redirect does, such as where a `Location` for another name of the site goes, or putting a removed path prefix back. + +**What it adds.** A method on `Middleware`, with a default that returns no headers, and the read only view of the response that the method is given, both in `crates/trusted-server-core/src/middleware.rs`. A middleware written before the method compiles and behaves as before, and a request path no `[[serve]]` entry covers sees no change. + +```rust +pub trait Middleware: Send + Sync { + fn response_headers( + &self, + context: &MiddlewareContext<'_>, + response: &ResponseView<'_>, + ) -> Vec { + Vec::new() + } +} + +pub struct ResponseView<'a> { + pub status: StatusCode, + pub headers: &'a http::HeaderMap, +} +``` + +`ResponseView` holds the status of the response and a borrowed view of its headers. + +**Which middleware are asked.** Every middleware named by an entry of `[[serve]]` whose path covers the request path, whatever the entry's media type, in entry order and each once. A header is not about a body, so the media type, which says what the body is, is not compared. `PhaseEntry::covers_path` says whether an entry covers a path, `PhaseEntries::for_path` gives the names, and `IntegrationRegistry::middleware_for_path` gives the middleware, leaving out a name no running module supplies. Where no entry covers the path, nothing is built and nothing is asked. + +**What a middleware is given.** A `MiddlewareContext` for the serve phase, made as for a page. The context carries the request's host and scheme, the host of `origin_url`, the platform identity, and the request state the modules' request preparers left, copied into a document state of its own. Nothing a middleware prepared is available, because the step runs no preparation. + +**What a middleware may do.** Return headers. Each is written on the response in middleware order, replacing any header of the same name, so a later middleware replaces an earlier one's header of that name. The method cannot change the status or the body. A header the same middleware sets for a page through the action it returns from `create` is written later, in the page pass, and replaces the header set here. + +**Where it runs.** In `handle_publisher_request`, once the origin's answer is in hand, straight after the header rewrite (section 2), which itself follows a file that core makes where the origin has none, and before core's cache rules, the modules' `finalize_response` hooks and the adapter's finalizing. A middleware therefore sees the headers as the header rewrite left them, and a header it sets is not rewritten. A cache header it sets can be replaced by the rules that follow, such as the `private, no-store` that core gives a page it has changed for one reader. + +**Where the step is called.** The step is called at three places in `handle_publisher_request`: + +1. The response the handler goes on with, being the origin's answer, a copy of an asset from the asset cache, or the file core makes where the origin has none. +2. The `502` core sends in place of an unexpected `304` from the origin. +3. The refusal of a streamed page by a middleware's preparation. + +The step is not called for three kinds of response: + +1. A refusal made before the origin is asked, being the `405` for a method a signed origin does not take and the `400` for a path with a dot segment under an origin folder. +2. A page served from the template cache, and the refusal of such a page by a middleware's preparation. That page has its headers built from the stored template, which replays a fixed list of policy headers, so a header the step set outside that list is missing from the page. The template cache is used only in the `Esi` assembly mode, so a deployment in the default `Inline` mode has no page the step misses on this account. +3. An answer made from an error, such as a failure to reach the origin, which `handle_publisher_request` returns to the adapter as an error. + +## 4. What is not in this change + +- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the origin's, so the header rewrite relays it and the reader follows it. A deployment whose origin does so names the host the origin redirects to in `origin_url`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a serve middleware of its own, through `Middleware::response_headers` (section 3). The step is asked about a redirect with no body, though not about every response the publisher path sends. +- **An origin that redirects an index file back to its folder.** With `directory_index` set, the index file is added to every request path that ends in `/`, so a request for `/news/` is sent to the origin as `/bucket/site/news/index.html`. An origin that answers that request with a redirect to `/bucket/site/news/` names the address the reader asked for. Under the header rewrite's second loop exception, a `GET` or `HEAD` redirect that sets no cookie is relayed as the origin wrote it, so the reader's browser leaves the deployment for the origin's address. One that sets a cookie is mapped, so the reader's browser asks again each time. +- **A `Location` that names no host, from an origin in a folder.** An origin under a path in `origin_url` that answers `Location: /bucket/site/news/` has the header relayed as written, as the header rewrite relays any relative URL, and the reader's browser then asks for a path with the folder in it. +- **Putting a removed path prefix back.** A serve middleware does so through the headers step, and no module in this repository reads the removed prefix yet. See "A publisher addressed by a path prefix" in section 2. +- **Following a redirect.** The publisher proxy follows no redirect itself. The reader's browser makes the next request. +- **Asset routes and the first-party proxy.** Their redirects are handled in `proxy.rs` and are unchanged. + +## 5. Acceptance + +Tests in `trusted-server-core`, where the code is, for the folder, in `response_header_rewrite.rs`: + +- A URL in the folder loses the folder and keeps its query and fragment, the folder itself with or without its closing `/` becomes `/`, a dot segment that stays in the folder is resolved, a protocol-relative URL stays protocol-relative, a folder written with a closing `/` is the same folder, and an origin path of `/` takes nothing off (`folder_origin_maps_urls_in_the_folder_without_it`). +- Another folder of the host, a folder whose name only begins with the folder's, a name that begins as the folder does but does not end at a `/`, the host's root, a URL that stops at the host, a dot segment that climbs out of the folder and a path in another letter case are each left unchanged (`folder_origin_leaves_urls_outside_the_folder_unchanged`). +- Under a folder, a scheme change to the address the reader asked for and a cookieless `GET` redirect to it are kept, and a redirect to another address in the folder is mapped (`folder_origin_keeps_redirects_to_the_current_request_that_would_loop`). +- A source with no path gains the whole serving host, a source in the folder gains a sibling without the folder, and a source outside the folder gains none (`folder_origin_csp_sources_follow_the_folder`). + +Tests through `handle_publisher_request` with a scripted origin, in `publisher.rs`: + +- The reader receives the mapped `Location` on a buffered answer and on a streamed page, for a `GET`, a `HEAD`, a `POST`, a `PUT` and a `DELETE`, for `301`, `302`, `303`, `307` and `308` (`the_reader_receives_the_rewritten_location_on_every_route`), and on the request's own scheme (`the_reader_s_own_scheme_is_used`). +- With a folder in `origin_url`, the origin is asked under the folder, a redirect in the folder reaches the reader without it, and one outside it reaches the reader as written (`a_redirect_under_the_origin_path_reaches_the_reader_without_it`). +- An origin on plain HTTP or on a port is named for the reader, and the same host on another port is not (`an_origin_on_plain_http_or_on_a_port_is_named_for_the_reader`). +- Under a `Host` override, a `Location` naming the host of `origin_url` is mapped and one naming the override is not (`under_a_host_override_the_host_of_origin_url_is_the_origin_s_name`). +- A `Location` on a `200`, a `201`, a `404` and a `503` is mapped on the pass-through, streamed and buffered routes (`a_location_on_an_answer_that_is_no_redirect_is_named_for_the_reader`). +- A redirect to another host and one naming no host reach the reader as the origin wrote them, and a redirect with no `Location` gains none (`a_redirect_that_is_not_the_origin_s_own_reaches_the_reader_unchanged` and `a_redirect_with_no_location_gains_none`). +- The header rewrite's own tests pass on this branch, both its handler tests for the two loop exceptions and its tests in `response_header_rewrite.rs`. + +Tests in `trusted-server-core` for the headers step: + +- `PhaseEntries::for_path` names every entry that covers a path whatever its media type, each middleware once in entry order, leaves out an entry whose path does not cover the request, and names nothing when there are no entries (`for_path_names_every_entry_covering_the_path_whatever_its_media_type`, in `middleware/tests.rs`). +- `IntegrationRegistry::middleware_for_path` gives the middleware those names stand for, once each and in entry order, and leaves out a name no module supplies (`middleware_for_path_gives_every_covering_entry_s_middleware_once_in_entry_order`, in `integrations/registry.rs`). +- Through `handle_publisher_request`, a middleware named for the path sets the `Location` of a redirect with no body and keeps the header it set, leaves a `Location` it does not set as written, and is not asked when its entry covers another path (`a_serve_middleware_named_for_the_path_sets_a_redirects_location`, in `publisher.rs`). +- The file core makes where the origin has none carries the headers a middleware sets, and so does the `502` for an unexpected origin `304` (`a_made_file_carries_the_headers_a_serve_middleware_sets` and `the_502_for_an_unexpected_origin_304_carries_a_serve_middlewares_headers`, in `publisher.rs`). + +Nothing tests the following: + +- The step on the refusal of a streamed page by a middleware's preparation, which is the third place the step is called. +- That the step is not called for a refusal made before the origin is asked, for a page from the template cache, or for an answer made from an error. +- Several middleware setting a header of one name on one response. +- The folder on `Content-Location`, `Refresh` and `Link` through the handler. Each takes the same mapping the folder tests call directly. +- A request addressed by a path prefix, sent through the header rewrite. +- Any adapter, because no adapter changes, although each already tests that its client leaves a redirect for core to see. + +Five deliberate faults were made, one in each run, and each was caught by a test: the handler giving the header rewrite no folder (`a_redirect_under_the_origin_path_reaches_the_reader_without_it`), the folder counted where it does not end at a `/` (`folder_origin_leaves_urls_outside_the_folder_unchanged`), a source with no path taking the folder rule (`folder_origin_csp_sources_follow_the_folder`), the loop exceptions comparing the path with the folder still in it (`folder_origin_keeps_redirects_to_the_current_request_that_would_loop`), and the folder compared before dot segments are resolved (`folder_origin_maps_urls_in_the_folder_without_it`). + +## Revision record + +| Date | Change | +| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-10-13 | First draft. | +| 2026-10-14 | Names its pull request and issue, and points a deployment that serves two names of the site at the serve middleware headers step. | +| 2026-10-15 | Fixes from review round 1. The headers step as a section with its tests and the responses it skips, who puts a removed path prefix back, the adapters, the setting, consent, failure and the index file loop, a table of what names the origin, the title now saying host, and the same sentence corrected in the changelog and the configuration guide. | +| 2026-10-15 | Corrections from review round 2. The two loops and the `421` under a removed path prefix, and the three kinds of response the headers step is not called for, named in the changelog and the configuration guide. | +| 2026-10-19 | Follows the header rewrite merged in main (#1238) in place of a `Location` rule of its own, and gives the header rewrite the origin folder. The title, sections 1, 2, 4 and 5 and where the headers step runs change to match, and so do the changelog and the configuration guide. | + +## Review log + +1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/37-origin-redirect` at `73801ddb7`. Issues found, three of them blocking. The design called the serve middleware headers step outside the change though the series adds it, said the step is asked about every response the publisher path sends though three kinds of response skip it, and said nothing of the reader of a publisher served under a path prefix, who is sent to an address without the prefix. This revision adds a section for the step with its tests, lists the responses the step skips, says a serve middleware and not core puts the prefix back, corrects the partly true claims, fills the gaps in coverage, takes the advice the round gave, and corrects the same sentence in the changelog and the configuration guide. +2. **Round 2, 2026-10-15.** A lighter review in a fresh session, against this page at `9dafb089e`, of the round 1 fixes and the new text. Approved, with no blocking issues. The changelog and the configuration guide now name the two loops and the `421` for a reader under a removed path prefix, and name all three kinds of response the headers step is not called for. diff --git a/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md index f48bca4dba..8ec908b502 100644 --- a/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md +++ b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md @@ -43,7 +43,7 @@ The page that comes back is a publisher page like any other, so it is fetched, c **The origin path.** A path in `origin_url` is kept, and the request path is added after it with exactly one `/` between them, whichever side carries one. The request's query is kept unless `origin_auth` sets `origin_query = "strip"` (see "The query"). An `origin_url` with no path sends the request path as it arrived, apart from the rebuilding of a signed address (see "Signing"). A query in `origin_url` is ignored, as it was. The path is no setting, because `origin_url` is required, and before this change the path was discarded. A deployment whose existing document has a path in `origin_url` therefore sends different requests to its origin, whether or not the document sets the two settings, because each request now goes to the folder the path names and a request path with a `.` or `..` segment is answered `400`. The path is a breaking change for such a deployment. -The publisher proxy builds the origin address in one place, `handle_publisher_request` in `publisher.rs`, and that address now comes from `publisher_origin_path_and_query`. The streamed page, the buffered page, the template cache key, a `HEAD` and every other method take the address built there. The publisher path follows no redirect and compares no `Location` against the origin, so there was nothing of that kind to change. +The publisher proxy builds the origin address in one place, `handle_publisher_request` in `publisher.rs`, and that address now comes from `publisher_origin_path_and_query`. The streamed page, the buffered page, the template cache key, a `HEAD` and every other method take the address built there. The publisher path follows no redirect. A URL in the folder in a response header, a redirect's `Location` among them, reaches the reader without the folder, and one outside the folder is left as the origin wrote it, as `2026-10-13-origin-redirect-design.md` sets out. **Signing.** `origin_auth` is the table an asset route's `auth` takes, read as the same type, with the same fields, defaults and secret store key references, and it signs through the same function, `apply_asset_origin_auth`. There is one signing path. The request is signed last, once its address and `Host` are final. Signing rebuilds the address through the URL parser, `url::Url`, signs the rebuilt address and sends the request to it, so the two cannot differ. The parser writes the address in its own form. It percent-encodes an apostrophe or a non-ASCII character in the query, and a double quote, a curly brace or a non-ASCII character in the path, all of which `http::Uri` accepts as they are, and it resolves dot segments in the path, which are refused with `400` first when `origin_url` has a path. The table still accepts the deprecated `secret_store` field that an asset route's `auth` accepts. The field is ignored with a logged warning and is never written back.