From 0e9172e3b41be71b0b08b5f62e85ba7fba794273 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Tue, 13 Oct 2026 23:44:26 +0100 Subject: [PATCH 01/14] An origin's redirect to itself keeps the reader on the address they asked for The publisher proxy relayed an origin's redirect as the origin wrote it. An origin redirects to itself for ordinary reasons, a request path without its trailing slash being enough, and it names itself in the Location, so the reader's browser left the deployment for the origin. Where the Location of a 3xx response names the origin, the reader is now sent to the same path, query and fragment on the scheme and host they asked for. The origin is named by the Host it was sent, port included, which is origin_host_header_override where one is set and the host of origin_url otherwise. A Location naming any other host, one that names no host, and the Location of a response that is not a redirect are left as the origin wrote them. With a path in origin_url, a Location at or beneath that path comes back without it, and one outside it is left alone, so no response carries a path the reader has no address for. The header is rewritten in the one place handle_publisher_request receives the origin's response, so the streamed page, the buffered page, a HEAD and every other method carry it. The guide's origin_url entry, the changelog and a design page describe the behavior, and the object storage design page's sentence on redirects now points at that page. --- CHANGELOG.md | 1 + crates/trusted-server-core/src/publisher.rs | 826 ++++++++++++++++++ docs/guide/configuration.md | 15 + .../2026-10-13-origin-redirect-design.md | 73 ++ ...-publisher-object-storage-origin-design.md | 2 +- 5 files changed, 916 insertions(+), 1 deletion(-) create mode 100644 docs/superpowers/specs/2026-10-13-origin-redirect-design.md diff --git a/CHANGELOG.md b/CHANGELOG.md index d83d9256f2..b12cadee27 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- A redirect the publisher origin sends to itself keeps the reader on the address they asked for. Where the `Location` of a `3xx` response names the origin, being the host of `origin_url` with its port, or `origin_host_header_override` where one is set, the publisher proxy sends the reader to the same path, query and fragment on the request's scheme and host, where it relayed the origin's own address and the reader's browser left the deployment. With a path in `origin_url`, a `Location` under that path comes back without it and one outside it is unchanged. A `Location` naming any other host, one that names no host, and the `Location` of a response that is not a redirect are relayed as before. - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServices::geo` and `IntegrationRegistration::with_geo_module` take an `Arc`, and `build_geo_module` returns one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths, and the Fastly adapter's two finalize paths, build the call from the request. - A module declares the caches it loads through, by name, and a deployment selects a loading cache for each in its own `[cache.]` table, the way `[cache.robots-txt]` selects one for robots.txt. A `[cache.]` table naming a cache no linked module declares is refused by `ts config validate` and when the deployment starts, naming the ones it does, and a module declaring `robots-txt` or a name another module declares is refused. `IntegrationBuilder::with_cache_features` declares them, `CacheSettings::loading_caches` lists core's own and the declared ones, and `LoadingCaches::insert` takes any name. - **Breaking for crates outside core:** the cookie constants `COOKIE_TS_EC`, `COOKIE_TS_EC_RESOLVED`, `COOKIE_TS_EC_PULL_COMPLETE`, `COOKIE_TS_EIDS` and `COOKIE_TS_TESTER` are `COOKIE_EC`, `COOKIE_EC_RESOLVED`, `COOKIE_EC_PULL_COMPLETE`, `COOKIE_EIDS` and `COOKIE_TESTER`, holding the entry's name alone, and a cookie is named through the request's public prefix with `services.public_prefix().cookie_name(COOKIE_EC)`. A module's response header naming a cookie under the public prefix is refused as one under `ts-` was. diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index bb99ec8750..0ac467c67f 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3742,6 +3742,71 @@ fn publisher_origin_path_and_query<'a>( Cow::Owned(built) } +/// Returns the `Location` that keeps a reader on the address they asked for +/// when the origin redirects to itself, or `None` to leave the header as the +/// origin wrote it. +/// +/// A `Location` is the origin's own when it names `origin_host`, the `Host` +/// the origin was sent, port included, and its path is `origin_path` or +/// beneath it. The reader is sent to the same place on `request_scheme` and +/// `request_host`, with `origin_path` taken off the path and the query and +/// fragment kept. Relayed as written it would take the reader's browser off +/// the deployment, and a missing trailing slash is enough to make an origin +/// send one. +/// +/// A `Location` naming another host is the origin sending the reader +/// elsewhere, and one outside `origin_path` has no address on the request's +/// host, so both are left alone. So is one that names no host, which the +/// reader's browser resolves against the address it asked for. +fn rewritten_origin_redirect( + location: &str, + origin_host: &str, + origin_path: &str, + request_scheme: &str, + request_host: &str, +) -> Option { + // A request that names no host has no address to send the reader to. + if request_host.is_empty() { + return None; + } + let target = match url::Url::parse(location) { + Ok(target) => target, + // `//host/path` names a host and takes the scheme of the request. + Err(url::ParseError::RelativeUrlWithoutBase) if location.starts_with("//") => { + url::Url::parse(&format!("{request_scheme}:{location}")).ok()? + } + Err(_) => return None, + }; + if !matches!(target.scheme(), "http" | "https") { + return None; + } + + // Compared with the port, because `origin.example` and + // `origin.example:8443` are different origins and `host_str` leaves the + // port off. + let host = target.host_str()?; + let target_host = match target.port() { + Some(port) => format!("{host}:{port}"), + None => host.to_owned(), + }; + if !target_host.eq_ignore_ascii_case(origin_host) { + return None; + } + + // The origin path counts only where it ends at a separator, so `/site` + // does not come off the front of `/sitemap.xml`. + let path = target + .path() + .strip_prefix(origin_path.trim_end_matches('/')) + .filter(|rest| rest.is_empty() || rest.starts_with('/'))?; + let path = if path.is_empty() { "/" } else { path }; + let query_and_fragment = &target[url::Position::AfterPath..]; + + Some(format!( + "{request_scheme}://{request_host}{path}{query_and_fragment}" + )) +} + /// Most percent-decoding rounds applied when looking for a dot segment. /// /// Bounds the work a `%25`-chained path can force, as the closed-path check @@ -6025,6 +6090,28 @@ pub async fn handle_publisher_request( response.headers().len() ); + // Every route below returns this response, so a redirect the origin sent + // to itself is named for the reader here, once. + if response.status().is_redirection() + && let Some(rewritten) = response + .headers() + .get(header::LOCATION) + .and_then(|value| value.to_str().ok()) + .and_then(|location| { + rewritten_origin_redirect( + location, + &origin_host_header, + parsed_origin.path(), + request_scheme, + request_host, + ) + }) + && let Ok(value) = HeaderValue::from_str(&rewritten) + { + log::debug!("Origin redirect named for the request's host"); + response.headers_mut().insert(header::LOCATION, value); + } + // A file the origin does not have is made, see `PhaseEntries::making`. An // origin page where the robots.txt middleware is placed is no file either. if (request_method == Method::GET || request_method == Method::HEAD) @@ -23487,6 +23574,745 @@ mod tests { } } + /// A redirect the origin sends to itself, named for the reader on every + /// route the publisher proxy returns an origin response through. + mod origin_redirect_tests { + use super::*; + use crate::platform::PlatformHttpClient; + + const ORIGIN_URL: &str = "https://origin.example"; + const STORAGE_URL: &str = "https://storage.example.com/bucket/site"; + const READER_HOST: &str = "publisher.example"; + + #[test] + fn a_redirect_naming_the_origin_is_named_for_the_reader() { + // (why, location, origin host, the rewritten location) + let cases = [ + ( + "an absolute redirect to the origin comes back on the reader's host", + "https://origin.example/news", + "origin.example", + "https://publisher.example/news", + ), + ( + "the path, the query and the fragment are kept", + "https://origin.example/news/archive/?page=2&sort=new#latest", + "origin.example", + "https://publisher.example/news/archive/?page=2&sort=new#latest", + ), + ( + "a query alone is kept", + "https://origin.example/news?utm_source=x&page=2", + "origin.example", + "https://publisher.example/news?utm_source=x&page=2", + ), + ( + "a fragment alone is kept", + "https://origin.example/news#latest", + "origin.example", + "https://publisher.example/news#latest", + ), + ( + "a redirect to the origin's root is the reader's root", + "https://origin.example", + "origin.example", + "https://publisher.example/", + ), + ( + "an origin reached over plain HTTP is named for a reader on HTTPS", + "http://origin.example/news", + "origin.example", + "https://publisher.example/news", + ), + ( + "an origin on a port is named with that port", + "https://origin.example:8443/news", + "origin.example:8443", + "https://publisher.example/news", + ), + ( + "a plain HTTP origin on a port is named with that port", + "http://192.0.2.1:9000/news", + "192.0.2.1:9000", + "https://publisher.example/news", + ), + ( + "a scheme's own port written out is the same origin", + "https://origin.example:443/news", + "origin.example", + "https://publisher.example/news", + ), + ( + "a Host header override is the origin's name when one is set", + "https://www.example.com/news", + "www.example.com", + "https://publisher.example/news", + ), + ( + "a Host header override is compared in any letter case", + "https://www.example.com/news", + "WWW.Example.COM", + "https://publisher.example/news", + ), + ( + "an origin named by an address in brackets is named with them", + "http://[::1]:3000/news", + "[::1]:3000", + "https://publisher.example/news", + ), + ( + "a host is the same host in any letter case, and the path keeps its own", + "https://ORIGIN.Example/News", + "origin.example", + "https://publisher.example/News", + ), + ( + "a redirect that leaves the scheme off still names the origin", + "//origin.example/news?page=2", + "origin.example", + "https://publisher.example/news?page=2", + ), + ]; + for (why, location, origin_host, expected) in cases { + assert_eq!( + rewritten_origin_redirect(location, origin_host, "", "https", READER_HOST) + .as_deref(), + Some(expected), + "should rewrite `{location}`: {why}", + ); + } + } + + #[test] + fn the_reader_is_sent_to_the_scheme_and_host_they_asked_for() { + // (why, location, the reader's scheme, the reader's host, the + // rewritten location) + let cases = [ + ( + "the reader's own scheme is used, so a deployment served over \ + plain HTTP is not sent to a scheme it does not answer", + "https://origin.example/news", + "http", + "publisher.example", + "http://publisher.example/news", + ), + ( + "the reader's host keeps its port", + "https://origin.example/news", + "http", + "localhost:8080", + "http://localhost:8080/news", + ), + ]; + for (why, location, scheme, host, expected) in cases { + assert_eq!( + rewritten_origin_redirect(location, "origin.example", "", scheme, host) + .as_deref(), + Some(expected), + "should rewrite `{location}`: {why}", + ); + } + } + + #[test] + fn a_location_that_does_not_name_the_origin_is_left_alone() { + // (why, location, origin host) + let cases = [ + ( + "a redirect anywhere else is the origin sending the reader off \ + site on purpose", + "https://somewhere.else/news", + "origin.example", + ), + ( + "a host beneath the origin's is another host", + "https://www.origin.example/news", + "origin.example", + ), + ( + "a host that only starts with the origin's is another host", + "https://origin.example.elsewhere.test/news", + "origin.example", + ), + ( + "a host that only ends with the origin's is another host", + "https://notorigin.example/news", + "origin.example", + ), + ( + "the origin's name written as a user name names another host", + "https://origin.example@elsewhere.test/news", + "origin.example", + ), + ( + "an origin on a port is a different origin from the same host \ + without one", + "https://origin.example/news", + "origin.example:8443", + ), + ( + "an origin without a port is a different origin from the same \ + host on one", + "https://origin.example:8443/news", + "origin.example", + ), + ( + "with a Host header override set, the host connected to is not \ + the origin's name", + "https://origin.example/news", + "www.example.com", + ), + ( + "a path from the root names no host", + "/news", + "origin.example", + ), + ( + "a path beside the one asked for names no host", + "news/latest", + "origin.example", + ), + ( + "a path that climbs names no host", + "../news", + "origin.example", + ), + ("a query alone names no host", "?page=2", "origin.example"), + ( + "a fragment alone names no host", + "#latest", + "origin.example", + ), + ("an empty value names no host", "", "origin.example"), + ( + "a redirect that leaves the scheme off may name another host", + "//somewhere.else/news", + "origin.example", + ), + ( + "an address that is not a web address is not the origin's own", + "ftp://origin.example/news", + "origin.example", + ), + ( + "an address with no host is not the origin's own", + "mailto:someone@origin.example", + "origin.example", + ), + ( + "a value that is no address is left as written", + "https://", + "origin.example", + ), + ]; + for (why, location, origin_host) in cases { + assert_eq!( + rewritten_origin_redirect(location, origin_host, "", "https", READER_HOST), + None, + "should leave `{location}` alone: {why}", + ); + } + } + + #[test] + fn a_reader_with_no_host_is_sent_nowhere_new() { + assert_eq!( + rewritten_origin_redirect( + "https://origin.example/news", + "origin.example", + "", + "https", + "", + ), + None, + "should leave the redirect alone when the request names no host" + ); + } + + #[test] + fn a_redirect_under_the_origin_path_loses_that_path() { + // (why, location, origin path, the rewritten location) + let cases = [ + ( + "a path beneath the origin path is the reader's path without it", + "https://storage.example.com/bucket/site/about/", + "/bucket/site", + "https://publisher.example/about/", + ), + ( + "the query and the fragment are kept", + "https://storage.example.com/bucket/site/about/index.html?v=2#team", + "/bucket/site", + "https://publisher.example/about/index.html?v=2#team", + ), + ( + "the origin path itself is the reader's root", + "https://storage.example.com/bucket/site", + "/bucket/site", + "https://publisher.example/", + ), + ( + "the origin path with a trailing slash is the reader's root", + "https://storage.example.com/bucket/site/", + "/bucket/site", + "https://publisher.example/", + ), + ( + "an origin path written with a trailing slash is the same path", + "https://storage.example.com/bucket/site/about/", + "/bucket/site/", + "https://publisher.example/about/", + ), + ( + "a dot segment that stays beneath the origin path is resolved first", + "https://storage.example.com/bucket/site/about/../news/", + "/bucket/site", + "https://publisher.example/news/", + ), + ( + "an origin path of `/` keeps nothing, so nothing comes off", + "https://storage.example.com/bucket/site/about/", + "/", + "https://publisher.example/bucket/site/about/", + ), + ]; + for (why, location, origin_path, expected) in cases { + assert_eq!( + rewritten_origin_redirect( + location, + "storage.example.com", + origin_path, + "https", + READER_HOST, + ) + .as_deref(), + Some(expected), + "should rewrite `{location}`: {why}", + ); + } + } + + #[test] + fn a_redirect_outside_the_origin_path_is_left_alone() { + // (why, location) + let cases = [ + ( + "another folder of the same host is not the publisher's site", + "https://storage.example.com/bucket/other/about/", + ), + ( + "the origin path counts only where it ends at a separator", + "https://storage.example.com/bucket/sitemap.xml", + ), + ( + "a folder whose name starts with the origin path's last is another folder", + "https://storage.example.com/bucket/site-two/about/", + ), + ( + "the host's root is above the origin path", + "https://storage.example.com/", + ), + ( + "a dot segment that climbs out of the origin path is outside it", + "https://storage.example.com/bucket/site/../other/about/", + ), + ( + "a path in another letter case is another path", + "https://storage.example.com/Bucket/Site/about/", + ), + ]; + for (why, location) in cases { + assert_eq!( + rewritten_origin_redirect( + location, + "storage.example.com", + "/bucket/site", + "https", + READER_HOST, + ), + None, + "should leave `{location}` alone: {why}", + ); + } + } + + fn settings_with_origin(origin_url: &str) -> Settings { + let mut settings = create_test_settings(); + settings.publisher.origin_url = origin_url.to_owned(); + settings + } + + /// A reader request for `uri` on [`READER_HOST`], made over `scheme`. + fn reader_request(method: Method, uri: &str, scheme: &str) -> HttpRequest { + HttpRequest::builder() + .method(method) + .uri(uri) + .header(header::HOST, READER_HOST) + .header("x-forwarded-proto", scheme) + .body(EdgeBody::empty()) + .expect("should build reader request") + } + + /// What the publisher proxy returns for `request` when the origin + /// answers with `status` and `headers`, and the addresses the origin + /// was sent. + async fn proxied( + settings: &Settings, + request: HttpRequest, + status: u16, + headers: Vec<(&str, &str)>, + ) -> (PublisherResponse, Vec) { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers(status, b"origin response".to_vec(), headers); + let services = + build_services_with_http_client(Arc::clone(&stub) as Arc); + let response = run_publisher_proxy(settings, &services, request).await; + (response, stub.recorded_request_uris()) + } + + /// The `Location` the reader receives. + fn location(response: &PublisherResponse) -> Option<&str> { + let response = match response { + PublisherResponse::Buffered(response) + | PublisherResponse::PassThrough { response, .. } + | PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } => response, + }; + response + .headers() + .get(header::LOCATION) + .and_then(|value| value.to_str().ok()) + } + + fn route(response: &PublisherResponse) -> &'static str { + match response { + PublisherResponse::Buffered(_) => "buffered", + PublisherResponse::PassThrough { .. } => "pass-through", + PublisherResponse::Stream { .. } => "stream", + PublisherResponse::AssembleTemplate { .. } => "template", + } + } + + #[tokio::test] + async fn the_reader_receives_the_rewritten_location_on_every_route() { + let settings = settings_with_origin(ORIGIN_URL); + // (what is asked for, method, the origin's status, the origin's + // media type, the route the answer takes) + let cases = [ + ("a page, buffered", Method::GET, 301, None, "buffered"), + ( + "a page, streamed", + Method::GET, + 301, + Some("text/html; charset=utf-8"), + "stream", + ), + ("a found page", Method::GET, 302, None, "buffered"), + ("a see other", Method::POST, 303, None, "buffered"), + ("a HEAD", Method::HEAD, 301, None, "buffered"), + ( + "a HEAD of a page", + Method::HEAD, + 302, + Some("text/html"), + "stream", + ), + ("a POST", Method::POST, 307, None, "buffered"), + ("a PUT", Method::PUT, 308, None, "buffered"), + ("a DELETE", Method::DELETE, 307, None, "buffered"), + ]; + for (what, method, status, media_type, expected_route) in cases { + let mut headers = vec![("location", "https://origin.example/news/?page=2#latest")]; + if let Some(media_type) = media_type { + headers.push(("content-type", media_type)); + } + + let (response, sent) = proxied( + &settings, + reader_request(method, "https://publisher.example/news?page=2", "https"), + status, + headers, + ) + .await; + + assert_eq!( + sent, + vec!["https://origin.example/news?page=2"], + "should ask the origin for {what}" + ); + assert_eq!( + route(&response), + expected_route, + "should return {what} through the expected route" + ); + assert_eq!( + location(&response), + Some("https://publisher.example/news/?page=2#latest"), + "should name the redirect for the reader on {what}" + ); + } + } + + #[tokio::test] + async fn the_reader_s_own_scheme_is_used() { + let settings = settings_with_origin(ORIGIN_URL); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "http://publisher.example/news", "http"), + 301, + vec![("location", "https://origin.example/news/")], + ) + .await; + + assert_eq!( + location(&response), + Some("http://publisher.example/news/"), + "should send a reader on plain HTTP to plain HTTP" + ); + } + + #[tokio::test] + async fn an_origin_on_plain_http_or_on_a_port_is_named_for_the_reader() { + // (origin URL, the origin's `Location`, what the reader receives) + let cases = [ + ( + "http://origin.example", + "http://origin.example/news/", + "https://publisher.example/news/", + ), + ( + "http://origin.example", + "https://origin.example/news/", + "https://publisher.example/news/", + ), + ( + "http://origin.example:8080", + "http://origin.example:8080/news/", + "https://publisher.example/news/", + ), + ( + "http://origin.example:8080", + "http://origin.example/news/", + "http://origin.example/news/", + ), + ( + "https://origin.example:8443", + "https://origin.example:8443/news/", + "https://publisher.example/news/", + ), + ( + "https://origin.example:8443", + "https://origin.example/news/", + "https://origin.example/news/", + ), + ( + "https://origin.example", + "https://origin.example:8443/news/", + "https://origin.example:8443/news/", + ), + ]; + for (origin_url, sent, expected) in cases { + let settings = settings_with_origin(origin_url); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", sent)], + ) + .await; + + assert_eq!( + location(&response), + Some(expected), + "should give the reader `{expected}` for `{sent}` from `{origin_url}`" + ); + } + } + + #[tokio::test] + async fn the_host_header_override_is_the_origin_s_name_when_set() { + let mut settings = settings_with_origin(ORIGIN_URL); + settings.publisher.origin_host_header_override = Some("www.example.com".to_owned()); + // (the origin's `Location`, what the reader receives) + let cases = [ + ( + "https://www.example.com/news/", + "https://publisher.example/news/", + ), + ( + "https://origin.example/news/", + "https://origin.example/news/", + ), + ]; + for (sent, expected) in cases { + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", sent)], + ) + .await; + + assert_eq!( + location(&response), + Some(expected), + "should treat the override as the origin's name for `{sent}`" + ); + } + } + + #[tokio::test] + async fn a_redirect_under_the_origin_path_reaches_the_reader_without_it() { + let settings = settings_with_origin(STORAGE_URL); + + let (response, sent) = proxied( + &settings, + reader_request( + Method::GET, + "https://publisher.example/news?page=2", + "https", + ), + 301, + vec![( + "location", + "https://storage.example.com/bucket/site/news/?page=2", + )], + ) + .await; + + assert_eq!( + sent, + vec!["https://storage.example.com/bucket/site/news?page=2"], + "should ask the origin under the origin path" + ); + assert_eq!( + location(&response), + Some("https://publisher.example/news/?page=2"), + "should send the reader to their own path, without the origin path" + ); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", "https://storage.example.com/bucket/other/news/")], + ) + .await; + + assert_eq!( + location(&response), + Some("https://storage.example.com/bucket/other/news/"), + "should leave a redirect outside the origin path as the origin wrote it" + ); + } + + #[tokio::test] + async fn a_redirect_that_is_not_the_origin_s_own_reaches_the_reader_unchanged() { + let settings = settings_with_origin(ORIGIN_URL); + // (why, the origin's `Location`) + let cases = [ + ("another host", "https://somewhere.else/news/"), + ("a path from the root", "/news/"), + ("a path beside the one asked for", "news/"), + ("a query alone", "?page=2"), + ]; + for (why, sent) in cases { + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + vec![("location", sent)], + ) + .await; + + assert_eq!( + response_status(&response), + StatusCode::FOUND, + "should relay the redirect for {why}" + ); + assert_eq!( + location(&response), + Some(sent), + "should leave {why} as the origin wrote it" + ); + } + } + + #[tokio::test] + async fn a_location_on_an_answer_that_is_no_redirect_is_unchanged() { + let settings = settings_with_origin(ORIGIN_URL); + // (the origin's status, the origin's media type, the route the + // answer takes) + let cases = [ + (200, None, "pass-through"), + (200, Some("text/html; charset=utf-8"), "stream"), + (201, None, "pass-through"), + (404, None, "buffered"), + (503, Some("text/html"), "stream"), + ]; + for (status, media_type, expected_route) in cases { + let mut headers = vec![("location", "https://origin.example/news/")]; + if let Some(media_type) = media_type { + headers.push(("content-type", media_type)); + } + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + status, + headers, + ) + .await; + + assert_eq!( + route(&response), + expected_route, + "should return a {status} through the expected route" + ); + assert_eq!( + location(&response), + Some("https://origin.example/news/"), + "should leave the `Location` of a {status} as the origin wrote it" + ); + } + } + + #[tokio::test] + async fn a_redirect_with_no_location_gains_none() { + let settings = settings_with_origin(ORIGIN_URL); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + Vec::new(), + ) + .await; + + assert_eq!( + response_status(&response), + StatusCode::FOUND, + "should relay the redirect" + ); + assert_eq!( + location(&response), + None, + "should add no `Location` the origin did not send" + ); + } + + fn response_status(response: &PublisherResponse) -> StatusCode { + match response { + PublisherResponse::Buffered(response) + | PublisherResponse::PassThrough { response, .. } + | PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } => response.status(), + } + } + } + #[test] fn stream_publisher_body_preserves_gzip_round_trip() { use flate2::write::GzEncoder; diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 496d39f9d3..c1b67dd438 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -448,6 +448,19 @@ is sent as it arrived, as before. When `origin_url` has a path, a request path with a `.` or `..` segment, in any percent-encoded spelling, is refused with `400` so no request reaches outside the folder. +**Redirects**: When the origin answers with a redirect whose `Location` names +the origin itself, the reader is sent to the same path, query and fragment on +the scheme and host they asked for, so their browser stays on the deployment. +An origin asked for `/news` that answers `Location: +https://origin.publisher.com/news/` sends a reader of `www.publisher.com` to +`https://www.publisher.com/news/`. The origin is named by the host in +`origin_url`, with its port, or by `origin_host_header_override` where one is +set. When `origin_url` has a path, a `Location` under that path comes back +without it, and one outside it is left as the origin wrote it. A `Location` +naming any other host, one that names no host (such as `/news/`), and the +`Location` of a response that is not a redirect are also left as the origin +wrote them. + #### `origin_host_header_override` **Purpose**: Optional Host header to send to the publisher origin while still @@ -457,6 +470,8 @@ connecting to the host in `origin_url`. - Connects, uses SNI, and checks certificates against `origin_url` - Sends the configured value as the outbound HTTP `Host` header +- Is the name a redirect from the origin is compared with (see Redirects under + `origin_url`) - Useful when the origin endpoint expects a canonical publisher hostname **Format**: Hostname with optional port, without protocol, path, query, or fragment diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md new file mode 100644 index 0000000000..19a80bab3d --- /dev/null +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -0,0 +1,73 @@ +# Design Spec: An origin's redirect to itself keeps the reader on the address they asked for + +**Status:** Proposed, 2026-10-13. Implemented on a local working branch, with no pull request raised yet. +**Author:** 51Degrees (contributed), for Tech Lab review +**Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` +**Related PRs:** none yet +**Last updated:** 2026-10-13 + +> **Why this spec exists.** The publisher proxy relays an origin's redirect exactly as the origin wrote it, so a redirect the origin sends to itself hands the reader the origin's own address and their browser leaves the deployment. + +## 1. Problem + +An origin redirects to itself for ordinary reasons, and a request path without its trailing slash is enough. It writes its own name in the `Location` header, because that is the name it was asked for. The publisher proxy relays the header as written, so the reader's browser follows it to the origin. From then on each page is the origin's own answer, with none of the publisher's `[[fetch]]` or `[[serve]]` middleware run on it, at an address the publisher never gave the reader. + +## 2. Design + +There is no setting. When the origin answers a request with a redirect, the publisher proxy reads the `Location` header, and where it names the origin the reader is sent to the same place on the scheme and host they asked for. + +With `origin_url = "https://origin.example"` and a reader who asked for `https://publisher.example/news`: + +| The origin answers | The reader receives | +| ----------------------------------------------------------------- | --------------------------------------------------------- | +| `301` with `Location: https://origin.example/news/` | `Location: https://publisher.example/news/` | +| `302` with `Location: https://origin.example/news/?page=2#latest` | `Location: https://publisher.example/news/?page=2#latest` | +| `307` with `Location: //origin.example/news/` | `Location: https://publisher.example/news/` | +| `301` with `Location: https://www.origin.example/news/` | The header as the origin wrote it | +| `301` with `Location: https://elsewhere.example/news/` | The header as the origin wrote it | +| `301` with `Location: /news/` | The header as the origin wrote it | +| `200` or `201` with `Location: https://origin.example/news/` | The header as the origin wrote it | + +**Which responses.** A response with a status from `300` to `399` and a `Location` header. A `Location` on any other status is left as written, so a `201` naming what was created is not touched. + +**What names the origin.** The host of the `Location`, with its port, is compared with the `Host` the origin was sent, which is `origin_host_header_override` where one is set and otherwise the host of `origin_url` with its port. Letter case is ignored. A port that is the usual one for the scheme the `Location` is written with counts as no port, so `https://origin.example:443/news` names `origin.example`. An origin on a port and the same host without one are different origins. The scheme of the `Location` is not compared, so an origin asked over plain HTTP that answers with its `https` address still names itself. Only an `http` or `https` address counts. A `Location` that leaves the scheme off, as `//origin.example/news/` does, names a host and is read on the scheme of the request. With an override set, the host in `origin_url` is where the connection goes and not what the origin is called, so a `Location` naming that host is left as written. + +**What the reader receives.** The scheme and host of the request, which are the two values the page pipeline already uses to rewrite the origin's addresses in a page, then the path, query and fragment of the `Location`. Where the origin's name is the request's own host, the header comes back naming the same host on the request's scheme. A request that names no host has nowhere to send the reader, so its redirect is left as written. + +**The origin path.** Where `origin_url` has a path, the origin is that folder and not the whole host. A `Location` whose path is the origin path, or beneath it, comes back with the origin path removed, which reverses the way the origin address was built. A request for `/news` is sent to `/bucket/site/news`, and a redirect to `/bucket/site/news/` reaches the reader as `/news/`. The origin path counts only where it ends at a `/`, so `/bucket/site` does not come off `/bucket/sitemap.xml`. Dot segments in the `Location` are resolved before the comparison, so a path that climbs out of the folder is outside it. A `Location` on the origin's host and outside the origin path is left as written, because no address on the reader's host leads there. + +**What is left alone.** A `Location` naming any other host is the origin sending the reader somewhere else, and a host beneath the origin's, such as `www.origin.example`, is another host. A `Location` that names no host, such as `/news/`, `news/` or `?page=2`, is resolved by the reader's browser against the address it asked for, so it stays on the deployment as it is. A value that cannot be read as an address is relayed as it arrived. + +**Where it happens.** In one place, `handle_publisher_request` in `publisher.rs`, at the point the origin's response is received and before anything else reads it. The streamed page, the buffered page, a `HEAD` and every other method return that one response, so each carries the rewritten header. The header is written for each request from that request's own scheme and host, after the origin's response has been returned to core. The rule itself is `rewritten_origin_redirect`, which takes the `Location`, the origin's host and path and the request's scheme and host, and returns the new value or nothing. + +**Why core.** The redirect answers the request core made to the one origin `[publisher]` names, and the rule uses only that origin's own name and path and the request's own scheme and host. It names no vendor and reads no module's settings. + +**Why no setting.** A `Location` naming the origin takes the reader off the deployment that is serving them, which no deployment asks for by naming an origin. A deployment whose origin sends no redirect to itself sees no change. + +## 3. What is not in this change + +- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. +- **An origin that redirects away from the scheme it is asked on.** The scheme of the `Location` is not compared, and the reader is sent to their own scheme. An origin that `origin_url` reaches over plain HTTP, and that answers each request with the same address on `https`, sends the reader back to the address they asked for, which gets the same answer again. `origin_url` names the scheme the origin serves its pages on. +- **A `Location` that names no host, from an origin in a folder.** An origin under a path in `origin_url` that answers `Location: /bucket/site/news/` has the header relayed as written, and the reader's browser then asks for a path with the folder in it. +- **Following a redirect.** The publisher proxy follows no redirect itself. The reader's browser makes the next request. +- **Any header but `Location`.** An address in `Content-Location`, `Link` or `Refresh` is relayed as before. +- **Asset routes and the first-party proxy.** Their redirects are handled in `proxy.rs` and are unchanged. + +## 4. Acceptance + +Tests in `trusted-server-core`, where the code is: + +- A `Location` naming the origin comes back on the request's scheme and host with its path, query and fragment, for an origin on `https` and on plain HTTP, with and without a port, named by an address in brackets, in any letter case, and with the scheme left off. +- With `origin_host_header_override` set, a `Location` naming the override is rewritten and one naming the host of `origin_url` is not. +- A `Location` naming another host, a host beneath the origin's, a host that only starts or ends with the origin's name, or the origin's name on a different port is left as written. So is a path, a query or a fragment alone, an address that is not `http` or `https`, a value that is no address, and any `Location` when the request names no host. +- With a path in `origin_url`, a `Location` at or beneath that path comes back without it. One beside it, above it or climbing out of it is left as written, and the origin path comes off only where it ends at a `/`. +- Through `handle_publisher_request` with a scripted origin, the reader receives the rewritten `Location` on a buffered answer and on a streamed page, for a `GET`, a `HEAD`, a `POST`, a `PUT` and a `DELETE`, for `301`, `302`, `303`, `307` and `308`, and on the request's own scheme. +- Through the same handler, a redirect to another host, a redirect naming no host, a `Location` on a `200`, a `201`, a `404` and a `503`, and a redirect with no `Location` each reach the reader as the origin sent them. + +Sixteen deliberate faults were made, one in each run, and each was caught by a test: a `Location` rewritten on any status, the port left out of the comparison, the origin path not removed, the origin path removed where it does not end at a `/`, a `Location` outside the origin path rewritten with its path whole, the fragment dropped, the query dropped, an address that is not `http` or `https` rewritten, a request with no host rewritten, a `Location` naming no host resolved against the origin, the host of `origin_url` compared where an override is set, a `Location` with the scheme left off not read, the host compared in exact letter case, the reader always sent to `https`, the origin path not passed by the handler, and the origin path itself given an empty path. + +## Revision record + +| Date | Change | +| ---------- | ------------ | +| 2026-10-13 | First draft. | diff --git a/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md index 5148b587c4..e7014f3eb6 100644 --- a/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md +++ b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md @@ -44,7 +44,7 @@ The page that comes back is a publisher page like any other, so it is fetched, c **The origin path.** A path in `origin_url` is kept, and the request path is added after it with exactly one `/` between them, whichever side carries one. The request's query is kept. An `origin_url` with no path, or the path `/`, sends the request path as it arrived, which is what every existing document does. A query in `origin_url` is ignored, as it was. -The publisher proxy builds the origin address in one place, `handle_publisher_request` in `publisher.rs`, and that address now comes from `publisher_origin_path_and_query`. The streamed page, the buffered page, the template cache key, a `HEAD` and every other method take the address built there. The publisher path follows no redirect and compares no `Location` against the origin, so there was nothing of that kind to change. +The publisher proxy builds the origin address in one place, `handle_publisher_request` in `publisher.rs`, and that address now comes from `publisher_origin_path_and_query`. The streamed page, the buffered page, the template cache key, a `HEAD` and every other method take the address built there. The publisher path follows no redirect. A redirect the origin sends to itself has the origin path taken off its `Location`, as `2026-10-13-origin-redirect-design.md` sets out. **Signing.** `origin_auth` is the table an asset route's `auth` takes, read as the same type, with the same fields, defaults and secret store key references, and it signs through the same function, `apply_asset_origin_auth`. There is one signing path. The request is signed last, once its address and `Host` are final, and it is sent to the address that was signed, so the two cannot differ. From 0affe093f074f8105865de5062518bd52a90da9a Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Tue, 13 Oct 2026 23:50:46 +0100 Subject: [PATCH 02/14] The origin redirect design names what a redirect to another name needs --- docs/superpowers/specs/2026-10-13-origin-redirect-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index 19a80bab3d..58c3e92625 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -46,7 +46,7 @@ With `origin_url = "https://origin.example"` and a reader who asked for `https:/ ## 3. What is not in this change -- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. +- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a middleware of its own, which needs a serve-phase way to change a header the origin set and is outside this change. - **An origin that redirects away from the scheme it is asked on.** The scheme of the `Location` is not compared, and the reader is sent to their own scheme. An origin that `origin_url` reaches over plain HTTP, and that answers each request with the same address on `https`, sends the reader back to the address they asked for, which gets the same answer again. `origin_url` names the scheme the origin serves its pages on. - **A `Location` that names no host, from an origin in a folder.** An origin under a path in `origin_url` that answers `Location: /bucket/site/news/` has the header relayed as written, and the reader's browser then asks for a path with the folder in it. - **Following a redirect.** The publisher proxy follows no redirect itself. The reader's browser makes the next request. From 002b96e945e72e6053f7a758326eb71ecd012d46 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 14 Oct 2026 08:48:28 +0100 Subject: [PATCH 03/14] The origin-redirect design names its pull request and issue --- .../superpowers/specs/2026-10-13-origin-redirect-design.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index 58c3e92625..d406914bdd 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -1,10 +1,11 @@ # Design Spec: An origin's redirect to itself keeps the reader on the address they asked for -**Status:** Proposed, 2026-10-13. Implemented on a local working branch, with no pull request raised yet. +**Status:** Proposed, 2026-10-13. Raised as draft pull request jwrosewell/trusted-server#32, for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review **Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` -**Related PRs:** none yet -**Last updated:** 2026-10-13 +**Related PRs:** jwrosewell/trusted-server#32 +**Issues:** IABTechLab/trusted-server#1330 +**Last updated:** 2026-10-14 > **Why this spec exists.** The publisher proxy relays an origin's redirect exactly as the origin wrote it, so a redirect the origin sends to itself hands the reader the origin's own address and their browser leaves the deployment. From 3dd7955a1aeab7cb37cfeee28e907466f9bd079f Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 14 Oct 2026 12:27:20 +0100 Subject: [PATCH 04/14] A serve middleware sets headers on every response, a bodiless redirect included A serve middleware could set a response header only through the action it returns for a page, so it ran on the responses the phase processes and on no other. An origin's redirect to a root-relative address, which the redirect rule leaves as written, reached the reader untouched on a deployment that names the publisher in the first part of the address, and the reader left the publisher, because the middleware that would put the prefix on the Location never saw the response: a 301 or 302 from such an origin has no body and no media type an entry covers. The Middleware trait gains response_headers, given the response's status and headers through a ResponseView and answering the headers to set, nothing by default. The publisher path asks it of every serve middleware an entry whose path covers the request names, whatever the entries' media types, on every response the origin path sends, after core's own redirect rewrite and before the response goes on, and a header set replaces one of the same name through write_response_headers. PhaseEntries::for_path and IntegrationRegistry::middleware_for_path give that list. A fixture middleware puts a prefix on a redirect's root-relative Location through the step alone, and a test through handle_publisher_request shows a bodiless 302 reaching it and the reader receiving the header it set, a Location it does not set left alone, and a middleware whose entry covers another path not asked. The guide's page changes section says what the step is for, and the changelog says what it adds. --- CHANGELOG.md | 1 + .../src/integrations/registry.rs | 77 ++++++++++++++- crates/trusted-server-core/src/middleware.rs | 49 ++++++++++ crates/trusted-server-core/src/publisher.rs | 98 +++++++++++++++++++ docs/guide/configuration.md | 7 ++ 5 files changed, 228 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f05234657..d16de7206f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -90,6 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- A serve middleware may set headers on every response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. - A publisher's site may be served from a folder of object storage through the publisher proxy, so its pages pass through the page pipeline, which an asset route skips. Three optional `[publisher]` settings do it, each off unless set. A path in `origin_url` is kept and the request path is added after it with one `/`. `origin_auth` takes the table an asset route's `auth` takes and signs every origin request with it, through the same signer. `directory_index` names a file added to a request path ending in `/`, for the origin request only. A signed origin is only read, so a method other than `GET` or `HEAD` is answered `405`, and a signed request carries none of the reader's cookies, credentials, referrer, conditional validators or range. With a path in `origin_url`, a request path with a `.` or `..` segment is answered `400`. A malformed `origin_auth` or `directory_index` is refused when the settings load, with the setting named. - `[ad-tag.google] write_loader` writes the loader tag into the head, last among the module's inserts and asking for the loader on the module's own route, for a page that carries no loader of its own. Off by default, because a page that already carries the loader would otherwise load it twice. - The publisher's assets and the answers of the asset routes are held through the loading cache, selected in `[cache.assets]`, so an asset is fetched once however many readers and instances ask for it at the same moment, and the copy in the host's shared store answers every instance until the origin's own lifetime ends. A copy is kept under what the origin says it varies on, served with the headers that describe it and an `Age` that counts from the origin's, and nothing is kept that may not be shared. A deployment that names no module takes the one loading cache module the build links, and a build that links none holds nothing, as before. diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 27ab083690..33a76af021 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -2122,6 +2122,22 @@ impl IntegrationRegistry { MiddlewareChain::new(phase, media_type, middleware) } + /// The middleware the serve phase's headers step asks for `path`: every + /// one an entry whose path covers it names, see + /// [`PhaseEntries::for_path`]. + #[must_use] + pub fn middleware_for_path( + &self, + entries: &PhaseEntries, + path: &str, + ) -> Vec> { + entries + .for_path(path) + .into_iter() + .filter_map(|name| self.inner.middleware_named(name).map(Arc::clone)) + .collect() + } + /// Every integration id the registry was built from, named or not, in /// registration order. A registry test enumerates this to check every /// builder was considered, while the deploy validation test iterates the @@ -3112,7 +3128,7 @@ pub(crate) mod test_support { use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; use crate::middleware::{ AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, - Preparation, Refusal, + Preparation, Refusal, ResponseHeader, ResponseView, }; use crate::module_context::ModuleCall; use crate::settings::Settings; @@ -3227,6 +3243,52 @@ pub(crate) mod test_support { } /// The name of the serve middleware that writes a header on a text /// response. + pub(crate) const LOCATING: &str = "testing.middleware-fixture.locating"; + pub(crate) const LOCATION_PREFIX: &str = "/fixture"; + + /// Puts a prefix on a redirect's root-relative `Location`, through the + /// headers step alone, which is what a deployment that names the + /// publisher in the first part of the address needs. + struct Locating; + + impl Middleware for Locating { + fn middleware_id(&self) -> &'static str { + LOCATING + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction::pass() + } + + fn response_headers( + &self, + _context: &MiddlewareContext<'_>, + response: &ResponseView<'_>, + ) -> Vec { + if !response.status.is_redirection() { + return Vec::new(); + } + let Some(location) = response + .headers + .get(http::header::LOCATION) + .and_then(|value| value.to_str().ok()) + else { + return Vec::new(); + }; + if !location.starts_with('/') || location.starts_with("//") { + return Vec::new(); + } + match http::HeaderValue::from_str(&format!("{LOCATION_PREFIX}{location}")) { + Ok(value) => vec![(http::header::LOCATION, value)], + Err(_) => Vec::new(), + } + } + } + pub(crate) const TEXT_HEADER: &str = "testing.middleware-fixture.text-header"; /// The header the text header middleware writes, name and value. pub(crate) const TEXT_HEADER_NAME: &str = "x-fixture-text"; @@ -3461,6 +3523,7 @@ pub(crate) mod test_support { .with_middleware(Arc::new(Preparing)) .with_middleware(Arc::new(Refusing)) .with_middleware(Arc::new(TextHeader)) + .with_middleware(Arc::new(Locating)) .build(), )) } @@ -6719,6 +6782,7 @@ mod tests { fixture::PREPARING, fixture::REFUSING, fixture::TEXT_HEADER, + fixture::LOCATING, crate::service_version::MODULE, ], "should list the middleware the selected module supplies, and core's own after them" @@ -6742,6 +6806,7 @@ mod tests { fixture::BROKEN_READER, fixture::PREPARING, fixture::TEXT_HEADER, + fixture::LOCATING, crate::service_version::MODULE ], "should list the middleware that run in the serve phase, core's own last" @@ -6858,7 +6923,7 @@ mod tests { message.contains( "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies" ) && message.contains(&format!( - "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}]", + "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}]", fixture::HEAD, fixture::LINKS, fixture::BROKEN, @@ -6868,6 +6933,7 @@ mod tests { fixture::PREPARING, fixture::REFUSING, fixture::TEXT_HEADER, + fixture::LOCATING, crate::service_version::MODULE )), "should name the entry and list what could be named: {message}" @@ -6924,6 +6990,7 @@ mod tests { fixture::PREPARING, fixture::REFUSING, fixture::TEXT_HEADER, + fixture::LOCATING, ], "should list every middleware when there are no entries" ); @@ -6939,7 +7006,8 @@ mod tests { fixture::TEXT, fixture::PREPARING, fixture::REFUSING, - fixture::TEXT_HEADER + fixture::TEXT_HEADER, + fixture::LOCATING ], "should leave out a middleware any entry names" ); @@ -7010,7 +7078,8 @@ mod tests { fixture::TEXT, fixture::PREPARING, fixture::REFUSING, - fixture::TEXT_HEADER + fixture::TEXT_HEADER, + fixture::LOCATING ], "should count a middleware as named only by an entry of a phase it runs in" ); diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index b978f68bd9..de27ef8874 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -276,6 +276,15 @@ pub trait TextHandler { /// A header a middleware writes on the reader's response. pub type ResponseHeader = (HeaderName, HeaderValue); +/// What a serve middleware is told about the response it may set a header +/// on, read only: the status and the headers as they stand after core's own +/// rewrites, before the adapter finalizes the response. +#[derive(Debug, Clone, Copy)] +pub struct ResponseView<'a> { + pub status: StatusCode, + pub headers: &'a http::HeaderMap, +} + /// Writes `headers` on `response`, each replacing a header of the same name. pub fn write_response_headers(response: &mut http::Response, headers: Vec) { for (name, value) in headers { @@ -520,6 +529,22 @@ pub trait Middleware: Send + Sync { /// Decides what to do with one document. fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction; + + /// The headers a serve middleware sets on `response`. Asked for every + /// response the publisher path sends for a request path the + /// middleware's entry covers, whatever the response's media type and + /// whether or not it has a body, so a redirect reaches a middleware + /// that rewrites its `Location`. A header returned replaces one of the + /// same name. Nothing by default, and a middleware that answers only + /// here returns a passing action from [`create`](Self::create). + fn response_headers( + &self, + context: &MiddlewareContext<'_>, + response: &ResponseView<'_>, + ) -> Vec { + let _ = (context, response); + Vec::new() + } } /// One entry of `[[fetch]]` or `[[serve]]`, being which responses it covers @@ -589,6 +614,30 @@ impl PhaseEntries { .map_or(&[][..], |entry| entry.middleware.as_slice()) } + /// The middleware every entry whose path covers `path` names, in entry + /// order and each once, whatever the entries' media types. A header is + /// not about a body, so this is what the headers step of the serve phase + /// asks. + #[must_use] + pub fn for_path(&self, path: &str) -> Vec<&str> { + let mut names: Vec<&str> = Vec::new(); + for entry in &self.0 { + if !entry + .path + .as_deref() + .is_none_or(|prefix| path.starts_with(prefix)) + { + continue; + } + for name in &entry.middleware { + if !names.contains(&name.as_str()) { + names.push(name); + } + } + } + names + } + /// Every name the entries select, each once, in the order first written. #[must_use] pub fn names(&self) -> Vec<&str> { diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index a0b2bea267..bf34c41fad 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -6198,6 +6198,38 @@ pub async fn handle_publisher_request( response.headers_mut().insert(header::LOCATION, value); } + // Every serve middleware an entry names for this path is asked for the + // headers it sets on this response, whatever the response is, so a + // redirect with no body reaches a middleware that rewrites its Location. + let header_middleware = integration_registry.middleware_for_path( + settings.phase_entries(MiddlewarePhase::Serve), + &request_path, + ); + if !header_middleware.is_empty() { + let document_state = IntegrationDocumentState::default(); + request_state.seed(&document_state); + let prepared = Prepared::default(); + let context = crate::middleware::MiddlewareContext { + phase: MiddlewarePhase::Serve, + request_host, + request_scheme, + origin_host: &origin_host, + platform: integration_registry.platform(), + document_state: &document_state, + max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, + prepared: &prepared, + }; + let view = crate::middleware::ResponseView { + status: response.status(), + headers: response.headers(), + }; + let mut headers = Vec::new(); + for middleware in &header_middleware { + headers.extend(middleware.response_headers(&context, &view)); + } + write_response_headers(&mut response, headers); + } + // A file the origin does not have is made, see `PhaseEntries::making`. An // origin page where the robots.txt middleware is placed is no file either. if (request_method == Method::GET || request_method == Method::HEAD) @@ -25024,6 +25056,72 @@ mod tests { } } + #[tokio::test] + async fn a_serve_middleware_named_for_the_path_sets_a_redirects_location() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_origin(ORIGIN_URL); + settings + .insert_module_config( + "testing", + fixture::MODULE, + &serde_json::json!({ "label": "locating" }), + ) + .expect("should select the middleware stand-in"); + settings.serve = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::LOCATING.to_owned()], + }]); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + vec![("location", "/uk")], + ) + .await; + assert_eq!( + location(&response), + Some("/fixture/uk"), + "should hand a bodiless redirect to the middleware and keep the header it set" + ); + + let (response, _) = proxied( + &settings, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 301, + vec![("location", "https://somewhere.else/news/")], + ) + .await; + assert_eq!( + location(&response), + Some("https://somewhere.else/news/"), + "should leave a Location the middleware does not set" + ); + + let mut elsewhere = settings.clone(); + elsewhere.serve = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: Some("/other/".to_owned()), + middleware: vec![fixture::LOCATING.to_owned()], + }]); + let (response, _) = proxied( + &elsewhere, + reader_request(Method::GET, "https://publisher.example/news", "https"), + 302, + vec![("location", "/uk")], + ) + .await; + assert_eq!( + location(&response), + Some("/uk"), + "should not ask a middleware whose entry covers another path" + ); + } + #[tokio::test] async fn a_redirect_with_no_location_gains_none() { let settings = settings_with_origin(ORIGIN_URL); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 9a4851bae8..a7af326058 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1993,6 +1993,13 @@ alone. Entries are therefore written from the longest path to the entry with no path, and a folder is written with its closing slash, because `/news` also begins `/newsletter`. +A serve middleware may set response headers as well as change the page. For +that it is asked about every response the publisher path sends for a request +path its entry covers, whatever the response's media type and whether or not +it has a body, with the response's status and headers in hand, so a redirect +reaches a middleware that rewrites its `Location`. A header it sets replaces +one of the same name. + ```toml [[fetch]] media_type = "text/html" From 43a55ddfa8820edc906c3f314d452194a572046c Mon Sep 17 00:00:00 2001 From: James Rogers Date: Wed, 14 Oct 2026 13:51:21 +0100 Subject: [PATCH 05/14] A Host override written with the scheme's own port still names the origin The origin's name is read on the scheme of the Location, so a port that is the usual one for it counts as no port, as url writes the Location. --- crates/trusted-server-core/src/publisher.rs | 25 ++++++++++++++------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index bf34c41fad..feadf0c28f 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3783,14 +3783,11 @@ fn rewritten_origin_redirect( } // Compared with the port, because `origin.example` and - // `origin.example:8443` are different origins and `host_str` leaves the - // port off. - let host = target.host_str()?; - let target_host = match target.port() { - Some(port) => format!("{host}:{port}"), - None => host.to_owned(), - }; - if !target_host.eq_ignore_ascii_case(origin_host) { + // `origin.example:8443` are different origins. The origin's name is read + // on the scheme of the `Location`, so a port that is the usual one for it + // counts as no port on both sides, as `url` writes the `Location`. + let origin = url::Url::parse(&format!("{}://{origin_host}", target.scheme())).ok()?; + if target.host_str()? != origin.host_str()? || target.port() != origin.port() { return None; } @@ -24425,6 +24422,13 @@ mod tests { "www.example.com", "https://publisher.example/news", ), + ( + "a Host header override written with the scheme's own port is \ + the same name", + "https://www.example.com/news", + "www.example.com:443", + "https://publisher.example/news", + ), ( "a Host header override is compared in any letter case", "https://www.example.com/news", @@ -24533,6 +24537,11 @@ mod tests { "https://origin.example:8443/news", "origin.example", ), + ( + "a port that is the usual one for another scheme is a port", + "http://origin.example/news", + "origin.example:443", + ), ( "with a Host header override set, the host connected to is not \ the origin's name", From fd0ba29787aa15ad9f0ad88c250557eb164f4c0a Mon Sep 17 00:00:00 2001 From: James Rogers Date: Wed, 14 Oct 2026 14:11:38 +0100 Subject: [PATCH 06/14] The design page names the serve middleware headers step for another name of the site --- docs/superpowers/specs/2026-10-13-origin-redirect-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index d406914bdd..d1657744f6 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -47,7 +47,7 @@ With `origin_url = "https://origin.example"` and a reader who asked for `https:/ ## 3. What is not in this change -- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a middleware of its own, which needs a serve-phase way to change a header the origin set and is outside this change. +- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a serve middleware of its own, through `Middleware::response_headers`, which is asked about every response the publisher path sends, a redirect with no body included. - **An origin that redirects away from the scheme it is asked on.** The scheme of the `Location` is not compared, and the reader is sent to their own scheme. An origin that `origin_url` reaches over plain HTTP, and that answers each request with the same address on `https`, sends the reader back to the address they asked for, which gets the same answer again. `origin_url` names the scheme the origin serves its pages on. - **A `Location` that names no host, from an origin in a folder.** An origin under a path in `origin_url` that answers `Location: /bucket/site/news/` has the header relayed as written, and the reader's browser then asks for a path with the folder in it. - **Following a redirect.** The publisher proxy follows no redirect itself. The reader's browser makes the next request. From 61f1b6f6629d56afe056d58482dc42eac1b03d72 Mon Sep 17 00:00:00 2001 From: James Rogers Date: Wed, 14 Oct 2026 14:11:42 +0100 Subject: [PATCH 07/14] An entry's path check is shared and the headers step says which header wins PhaseEntries::for_path and PhaseEntry::covers call one covers_path check. The rustdoc of Middleware::response_headers is cut to what an implementer needs and says a header the action from create sets replaces one set here. for_path is tested with entries of several media types and a repeated middleware. --- crates/trusted-server-core/src/middleware.rs | 38 +++++++++---------- .../src/middleware/tests.rs | 27 +++++++++++++ 2 files changed, 45 insertions(+), 20 deletions(-) diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index de27ef8874..f9e3390f96 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -530,13 +530,13 @@ pub trait Middleware: Send + Sync { /// Decides what to do with one document. fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction; - /// The headers a serve middleware sets on `response`. Asked for every - /// response the publisher path sends for a request path the - /// middleware's entry covers, whatever the response's media type and - /// whether or not it has a body, so a redirect reaches a middleware - /// that rewrites its `Location`. A header returned replaces one of the - /// same name. Nothing by default, and a middleware that answers only - /// here returns a passing action from [`create`](Self::create). + /// The headers a serve middleware sets on the response the publisher path + /// sends, a redirect with no body included, for a request path its entry + /// covers. Each replaces a + /// header of the same name, and one the action from + /// [`create`](Self::create) sets in + /// [`MiddlewareAction::response_headers`] replaces it in turn. None by + /// default. fn response_headers( &self, context: &MiddlewareContext<'_>, @@ -569,11 +569,16 @@ impl PhaseEntry { /// `path`. #[must_use] pub fn covers(&self, media_type: &str, path: &str) -> bool { - self.media_type == media_type - && self - .path - .as_deref() - .is_none_or(|prefix| path.starts_with(prefix)) + self.media_type == media_type && self.covers_path(path) + } + + /// Whether the entry covers a request for `path`, whatever the media + /// type. + #[must_use] + pub fn covers_path(&self, path: &str) -> bool { + self.path + .as_deref() + .is_none_or(|prefix| path.starts_with(prefix)) } } @@ -621,14 +626,7 @@ impl PhaseEntries { #[must_use] pub fn for_path(&self, path: &str) -> Vec<&str> { let mut names: Vec<&str> = Vec::new(); - for entry in &self.0 { - if !entry - .path - .as_deref() - .is_none_or(|prefix| path.starts_with(prefix)) - { - continue; - } + for entry in self.0.iter().filter(|entry| entry.covers_path(path)) { for name in &entry.middleware { if !names.contains(&name.as_str()) { names.push(name); diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index 73930460bf..37c9d94364 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -312,6 +312,33 @@ fn names_lists_each_middleware_once_in_the_order_first_written() { assert_eq!(entries.names(), ["example.b", "example.a", "example.c"]); } +#[test] +fn for_path_names_every_entry_covering_the_path_whatever_its_media_type() { + let mut text = entry(Some("/news/"), &["example.text", "example.a"]); + text.media_type = "text/plain".to_owned(); + let entries = PhaseEntries::new(vec![ + entry(Some("/news/"), &["example.a", "example.b"]), + text, + entry(Some("/sport/"), &["example.sport"]), + entry(None, &["example.b", "example.all"]), + ]); + + assert_eq!( + entries.for_path("/news/today"), + ["example.a", "example.b", "example.text", "example.all"], + "should name each middleware once, in entry order, from every entry covering the path" + ); + assert_eq!( + entries.for_path("/weather/"), + ["example.b", "example.all"], + "should leave out an entry whose path does not cover the request" + ); + assert!( + PhaseEntries::default().for_path("/").is_empty(), + "should name nothing when there are no entries" + ); +} + #[test] fn entries_from_the_longest_path_to_every_path_are_accepted() { PhaseEntries::new(vec![ From d0148227e7fcbf5c4e86824fe94c8775e76724c5 Mon Sep 17 00:00:00 2001 From: James Rogers Date: Wed, 14 Oct 2026 14:11:46 +0100 Subject: [PATCH 08/14] Serve middleware headers are written on the response that is sent The headers step ran before a made file or the 502 for an unexpected 304 replaced the response, so its headers were lost. It now runs on the made file, the 502 and a middleware refusal. The locating fixture names the status it was shown, so tests show a made file and the 502 carry it. middleware_for_path is tested directly, the TEXT_HEADER rustdoc is back on TEXT_HEADER, and the redirect tests share one response_status helper. --- .../src/integrations/registry.rs | 67 ++++-- crates/trusted-server-core/src/publisher.rs | 190 ++++++++++++++---- 2 files changed, 202 insertions(+), 55 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 33a76af021..24259ef5e1 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -3241,14 +3241,17 @@ pub(crate) mod test_support { MiddlewareAction::pass() } } - /// The name of the serve middleware that writes a header on a text - /// response. + /// The name of the serve middleware that sets headers through the + /// headers step alone. pub(crate) const LOCATING: &str = "testing.middleware-fixture.locating"; pub(crate) const LOCATION_PREFIX: &str = "/fixture"; + /// The header the locating middleware sets on every response it is + /// shown, carrying that response's status. + pub(crate) const SHOWN_STATUS_NAME: &str = "x-fixture-shown-status"; - /// Puts a prefix on a redirect's root-relative `Location`, through the - /// headers step alone, which is what a deployment that names the - /// publisher in the first part of the address needs. + /// Puts a prefix on a redirect's root-relative `Location`, which is + /// what a deployment that names the publisher in the first part of the + /// address needs, and names the status of every response it is shown. struct Locating; impl Middleware for Locating { @@ -3269,26 +3272,33 @@ pub(crate) mod test_support { _context: &MiddlewareContext<'_>, response: &ResponseView<'_>, ) -> Vec { + let mut headers = vec![( + http::HeaderName::from_static(SHOWN_STATUS_NAME), + http::HeaderValue::from(response.status.as_u16()), + )]; if !response.status.is_redirection() { - return Vec::new(); + return headers; } let Some(location) = response .headers .get(http::header::LOCATION) .and_then(|value| value.to_str().ok()) else { - return Vec::new(); + return headers; }; - if !location.starts_with('/') || location.starts_with("//") { - return Vec::new(); - } - match http::HeaderValue::from_str(&format!("{LOCATION_PREFIX}{location}")) { - Ok(value) => vec![(http::header::LOCATION, value)], - Err(_) => Vec::new(), + if location.starts_with('/') + && !location.starts_with("//") + && let Ok(value) = + http::HeaderValue::from_str(&format!("{LOCATION_PREFIX}{location}")) + { + headers.push((http::header::LOCATION, value)); } + headers } } + /// The name of the serve middleware that writes a header on a text + /// response. pub(crate) const TEXT_HEADER: &str = "testing.middleware-fixture.text-header"; /// The header the text header middleware writes, name and value. pub(crate) const TEXT_HEADER_NAME: &str = "x-fixture-text"; @@ -6828,6 +6838,37 @@ mod tests { ); } + #[test] + fn middleware_for_path_gives_every_covering_entry_s_middleware_once_in_entry_order() { + let registry = IntegrationRegistry::new(&fixture_settings_with_entries(Vec::new())) + .expect("should build a registry"); + let entries = PhaseEntries::new(vec![ + html_entry(Some("/news/"), &[fixture::LOCATING, fixture::READER]), + text_entry(&[fixture::TEXT_HEADER, fixture::LOCATING]), + html_entry(Some("/sport/"), &[fixture::BROKEN_READER]), + html_entry(None, &["testing.nothing", fixture::READER]), + ]); + + let ids = |path: &str| -> Vec<&'static str> { + registry + .middleware_for_path(&entries, path) + .iter() + .map(|middleware| middleware.middleware_id()) + .collect() + }; + assert_eq!( + ids("/news/today"), + [fixture::LOCATING, fixture::READER, fixture::TEXT_HEADER], + "should give each middleware once, in entry order, whatever the entry's media type" + ); + assert_eq!( + ids("/weather/"), + [fixture::TEXT_HEADER, fixture::LOCATING, fixture::READER], + "should leave out an entry whose path does not cover the request, and a name \ + nothing supplies" + ); + } + /// An entry for `text/plain` naming the middleware that works on it. fn text_entry(names: &[&str]) -> PhaseEntry { PhaseEntry { diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index feadf0c28f..0b75e963e7 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3743,6 +3743,44 @@ fn publisher_origin_path_and_query<'a>( Cow::Owned(built) } +/// Writes on `response` the headers each serve middleware an entry names for +/// `request_path` sets, see +/// [`Middleware::response_headers`](crate::middleware::Middleware::response_headers). +fn write_serve_middleware_headers( + response: &mut Response, + reader: &crate::html_processor::ReaderDocument<'_>, + request_path: &str, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) { + let middleware = integration_registry + .middleware_for_path(settings.phase_entries(MiddlewarePhase::Serve), request_path); + if middleware.is_empty() { + return; + } + let document_state = IntegrationDocumentState::default(); + reader.request_state.seed(&document_state); + let context = crate::middleware::MiddlewareContext { + phase: MiddlewarePhase::Serve, + request_host: reader.request_host, + request_scheme: reader.request_scheme, + origin_host: reader.origin_host, + platform: reader.platform, + document_state: &document_state, + max_buffered_script_bytes: reader.max_buffered_script_bytes, + prepared: reader.prepared, + }; + let view = crate::middleware::ResponseView { + status: response.status(), + headers: response.headers(), + }; + let headers = middleware + .iter() + .flat_map(|middleware| middleware.response_headers(&context, &view)) + .collect(); + write_response_headers(response, headers); +} + /// Returns the `Location` that keeps a reader on the address they asked for /// when the origin redirects to itself, or `None` to leave the header as the /// origin wrote it. @@ -6195,38 +6233,6 @@ pub async fn handle_publisher_request( response.headers_mut().insert(header::LOCATION, value); } - // Every serve middleware an entry names for this path is asked for the - // headers it sets on this response, whatever the response is, so a - // redirect with no body reaches a middleware that rewrites its Location. - let header_middleware = integration_registry.middleware_for_path( - settings.phase_entries(MiddlewarePhase::Serve), - &request_path, - ); - if !header_middleware.is_empty() { - let document_state = IntegrationDocumentState::default(); - request_state.seed(&document_state); - let prepared = Prepared::default(); - let context = crate::middleware::MiddlewareContext { - phase: MiddlewarePhase::Serve, - request_host, - request_scheme, - origin_host: &origin_host, - platform: integration_registry.platform(), - document_state: &document_state, - max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, - prepared: &prepared, - }; - let view = crate::middleware::ResponseView { - status: response.status(), - headers: response.headers(), - }; - let mut headers = Vec::new(); - for middleware in &header_middleware { - headers.extend(middleware.response_headers(&context, &view)); - } - write_response_headers(&mut response, headers); - } - // A file the origin does not have is made, see `PhaseEntries::making`. An // origin page where the robots.txt middleware is placed is no file either. if (request_method == Method::GET || request_method == Method::HEAD) @@ -6242,6 +6248,20 @@ pub async fn handle_publisher_request( response = empty_document(&entry.media_type)?; } + // Every serve middleware an entry names for this path is asked for the + // headers it sets on the response that is sent, whatever it is, so a + // redirect with no body reaches a middleware that rewrites its Location. + let header_prepared = Prepared::default(); + let header_reader = crate::html_processor::ReaderDocument { + request_host, + request_scheme, + origin_host: &origin_host, + platform: integration_registry.platform(), + request_state: &request_state, + max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, + prepared: &header_prepared, + }; + if should_run_ad_stack && response.status() == StatusCode::NOT_MODIFIED { if let Some(dispatched) = dispatched_auction.take() { emit_abandoned_auction( @@ -6262,9 +6282,23 @@ pub async fn handle_publisher_request( .change_context(TrustedServerError::Proxy { message: "failed to build unexpected origin 304 response".to_string(), })?; + write_serve_middleware_headers( + &mut response, + &header_reader, + &request_path, + settings, + integration_registry, + ); enforce_terminal_private_cache_privacy(&mut response); return Ok(PublisherResponse::Buffered(response)); } + write_serve_middleware_headers( + &mut response, + &header_reader, + &request_path, + settings, + integration_registry, + ); let template_cache_policy = TemplateCachePolicy::from_settings(settings); let gate_content_type = response @@ -6476,6 +6510,13 @@ pub async fn handle_publisher_request( .await; } let mut response = refusal_response(&refusal)?; + write_serve_middleware_headers( + &mut response, + &header_reader, + &request_path, + settings, + integration_registry, + ); enforce_terminal_private_cache_privacy(&mut response); return Ok(PublisherResponse::Buffered(response)); } @@ -20554,6 +20595,53 @@ mod tests { } } + #[tokio::test] + async fn the_502_for_an_unexpected_origin_304_carries_a_serve_middlewares_headers() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_dispatching_provider(); + settings.select_module("testing", fixture::MODULE); + settings.serve = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::LOCATING.to_owned()], + }]); + let mut orchestrator = AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + ); + orchestrator.register_provider(Arc::new(DispatchingTestProvider)); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, b"unused provider response".to_vec()); + stub.push_response_with_headers(304, Vec::new(), vec![("etag", ORIGIN_ETAG)]); + let services = services_with_telemetry( + Arc::clone(&stub) as Arc, + Arc::new(RecordingTelemetrySink::default()), + ); + + let response = run_with_orchestrator( + &settings, + &services, + &orchestrator, + &[article_slot()], + conditional_navigation_request(), + ) + .await; + + let PublisherResponse::Buffered(response) = response else { + panic!("unexpected origin 304 should return a buffered response") + }; + assert_eq!(response.status(), StatusCode::BAD_GATEWAY); + assert_eq!( + response + .headers() + .get(fixture::SHOWN_STATUS_NAME) + .and_then(|value| value.to_str().ok()), + Some("502"), + "should set the middleware's headers on the 502, which is what is sent" + ); + } + #[tokio::test] async fn inactive_ad_stack_304_uses_short_browser_cache_policy() { // Arrange @@ -23231,6 +23319,32 @@ mod tests { ); } + #[tokio::test] + async fn a_made_file_carries_the_headers_a_serve_middleware_sets() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = + settings_running_the_middleware_fixture(text_entry_for(Some("/ads.txt"))); + settings.serve = PhaseEntries::new(vec![PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::LOCATING.to_owned()], + }]); + + let proxied = proxy_with(settings, Method::GET, "/ads.txt", 404).await; + let response = response_of(proxied.response); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response + .headers() + .get(fixture::SHOWN_STATUS_NAME) + .and_then(|value| value.to_str().ok()), + Some("200"), + "should set the middleware's headers on the made file, which is what is sent" + ); + } + #[tokio::test] async fn an_html_entry_with_a_path_makes_no_missing_page() { use crate::integrations::registry_test_support::middleware_fixture as fixture; @@ -23842,7 +23956,7 @@ mod tests { .to_owned() } - fn response_status(response: &PublisherResponse) -> StatusCode { + pub(super) fn response_status(response: &PublisherResponse) -> StatusCode { match response { PublisherResponse::Buffered(response) | PublisherResponse::PassThrough { response, .. } @@ -24351,6 +24465,7 @@ mod tests { /// A redirect the origin sends to itself, named for the reader on every /// route the publisher proxy returns an origin response through. mod origin_redirect_tests { + use super::object_storage_origin_tests::response_status; use super::*; use crate::platform::PlatformHttpClient; @@ -25154,15 +25269,6 @@ mod tests { "should add no `Location` the origin did not send" ); } - - fn response_status(response: &PublisherResponse) -> StatusCode { - match response { - PublisherResponse::Buffered(response) - | PublisherResponse::PassThrough { response, .. } - | PublisherResponse::Stream { response, .. } - | PublisherResponse::AssembleTemplate { response, .. } => response.status(), - } - } } #[test] From 9dafb089e0afdf3c926eb59f85e21004820ac896 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 17:06:12 +0100 Subject: [PATCH 09/14] The origin redirect design names the headers step it adds and the responses that skip it Review round 1 found that the design called the serve middleware headers step outside the change although the series adds it, said the step is asked about every response the publisher path sends when a refusal before the origin is asked, a page from the template cache and an answer made from an error skip it, and said nothing of a publisher served under a path prefix, whose reader the rewritten Location sends to an address without the prefix. The design now has a section for the step with its tests, says a serve middleware and not core puts the prefix back, and covers the adapters, the setting, consent, failure and the loop with an index file. The same sentence is corrected in the changelog and the configuration guide. --- CHANGELOG.md | 2 +- docs/guide/configuration.md | 10 +- .../2026-10-13-origin-redirect-design.md | 145 +++++++++++++++--- 3 files changed, 132 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d16de7206f..1c60a5f97d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -90,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- A serve middleware may set headers on every response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. +- A serve middleware may set headers on a response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. The method is not called for a refusal made before the origin is asked, such as the `405` for a method a signed origin does not take, or for a page served from the template cache. - A publisher's site may be served from a folder of object storage through the publisher proxy, so its pages pass through the page pipeline, which an asset route skips. Three optional `[publisher]` settings do it, each off unless set. A path in `origin_url` is kept and the request path is added after it with one `/`. `origin_auth` takes the table an asset route's `auth` takes and signs every origin request with it, through the same signer. `directory_index` names a file added to a request path ending in `/`, for the origin request only. A signed origin is only read, so a method other than `GET` or `HEAD` is answered `405`, and a signed request carries none of the reader's cookies, credentials, referrer, conditional validators or range. With a path in `origin_url`, a request path with a `.` or `..` segment is answered `400`. A malformed `origin_auth` or `directory_index` is refused when the settings load, with the setting named. - `[ad-tag.google] write_loader` writes the loader tag into the head, last among the module's inserts and asking for the loader on the module's own route, for a page that carries no loader of its own. Off by default, because a page that already carries the loader would otherwise load it twice. - The publisher's assets and the answers of the asset routes are held through the loading cache, selected in `[cache.assets]`, so an asset is fetched once however many readers and instances ask for it at the same moment, and the copy in the host's shared store answers every instance until the origin's own lifetime ends. A copy is kept under what the origin says it varies on, served with the headers that describe it and an `Age` that counts from the origin's, and nothing is kept that may not be shared. A deployment that names no module takes the one loading cache module the build links, and a build that links none holds nothing, as before. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 2f0042a4d7..da215b4802 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1995,11 +1995,13 @@ no path, and a folder is written with its closing slash, because `/news` also begins `/newsletter`. A serve middleware may set response headers as well as change the page. For -that it is asked about every response the publisher path sends for a request -path its entry covers, whatever the response's media type and whether or not -it has a body, with the response's status and headers in hand, so a redirect +that it is asked about a response the publisher path sends for a request path +its entry covers, whatever the response's media type and whether or not it +has a body, with the response's status and headers in hand, so a redirect reaches a middleware that rewrites its `Location`. A header it sets replaces -one of the same name. +one of the same name. A serve middleware is not asked about a refusal made +before the origin is asked, such as the `405` for a method a signed origin +does not take, or about a page served from the template cache. ```toml [[fetch]] diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index d1657744f6..7814ace391 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -1,18 +1,20 @@ -# Design Spec: An origin's redirect to itself keeps the reader on the address they asked for +# Design Spec: An origin's redirect to itself keeps the reader on the host they asked for **Status:** Proposed, 2026-10-13. Raised as draft pull request jwrosewell/trusted-server#32, for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review -**Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` +**Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` and `2026-10-13-request-addressing-design.md`, both in IABTechLab/trusted-server#1084, which holds every design document of the stack **Related PRs:** jwrosewell/trusted-server#32 **Issues:** IABTechLab/trusted-server#1330 -**Last updated:** 2026-10-14 +**Last updated:** 2026-10-15 -> **Why this spec exists.** The publisher proxy relays an origin's redirect exactly as the origin wrote it, so a redirect the origin sends to itself hands the reader the origin's own address and their browser leaves the deployment. +> **Why this spec exists.** The publisher proxy relays an origin's redirect exactly as the origin wrote it, so a redirect the origin sends to itself hands the reader the origin's own address and their browser leaves the deployment. The same change gives a serve middleware a way to set headers on a response that holds no page, a redirect among them. ## 1. Problem An origin redirects to itself for ordinary reasons, and a request path without its trailing slash is enough. It writes its own name in the `Location` header, because that is the name it was asked for. The publisher proxy relays the header as written, so the reader's browser follows it to the origin. From then on each page is the origin's own answer, with none of the publisher's `[[fetch]]` or `[[serve]]` middleware run on it, at an address the publisher never gave the reader. +A second gap sits beside the first. A serve middleware works on the page in a response and writes headers through the action it returns for that page, so a response with no page in it, such as most redirects, reaches no middleware. A deployment that wants a policy of its own for the `Location` of a redirect has nowhere to put the policy. + ## 2. Design There is no setting. When the origin answers a request with a redirect, the publisher proxy reads the `Location` header, and where it names the origin the reader is sent to the same place on the scheme and host they asked for. @@ -31,44 +33,147 @@ With `origin_url = "https://origin.example"` and a reader who asked for `https:/ **Which responses.** A response with a status from `300` to `399` and a `Location` header. A `Location` on any other status is left as written, so a `201` naming what was created is not touched. -**What names the origin.** The host of the `Location`, with its port, is compared with the `Host` the origin was sent, which is `origin_host_header_override` where one is set and otherwise the host of `origin_url` with its port. Letter case is ignored. A port that is the usual one for the scheme the `Location` is written with counts as no port, so `https://origin.example:443/news` names `origin.example`. An origin on a port and the same host without one are different origins. The scheme of the `Location` is not compared, so an origin asked over plain HTTP that answers with its `https` address still names itself. Only an `http` or `https` address counts. A `Location` that leaves the scheme off, as `//origin.example/news/` does, names a host and is read on the scheme of the request. With an override set, the host in `origin_url` is where the connection goes and not what the origin is called, so a `Location` naming that host is left as written. +**What names the origin.** The host of the `Location`, with its port, is compared with the `Host` the origin was sent, which is `origin_host_header_override` where one is set and otherwise the host of `origin_url` with its port. Letter case in a host is ignored and the scheme of the `Location` is not compared. The table gives the cases, with `origin_url = "https://origin.example"` unless a row says otherwise. + +| The `Location` names | The origin's name | Because | +| --------------------------------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------- | +| `https://origin.example/news/` | Yes | The host and port are the origin's | +| `https://ORIGIN.example/news/` | Yes | Letter case in a host is ignored | +| `https://origin.example:443/news/` | Yes | The usual port for the scheme counts as no port | +| `http://origin.example/news/` | Yes | The scheme of the `Location` is not compared | +| `//origin.example/news/` | Yes | A `Location` with no scheme names a host and is read on the scheme of the request | +| `https://origin.example:8443/news/` | No | The same host on another port is another origin | +| `https://www.origin.example/news/` | No | A host beneath the origin's is another host | +| `https://origin.example.elsewhere.test/news/` | No | A host that only begins with the origin's name is another host | +| `https://origin.example@elsewhere.test/news/` | No | The text before an `@` is a user name, and the host is `elsewhere.test` | +| `ftp://origin.example/news/` | No | Only an `http` or `https` address counts | +| `https://origin.example/news/`, with `origin_url = "https://origin.example:8443"` | No | The same host without the port is another origin | +| `https://www.example.com/news/`, with `origin_host_header_override = "www.example.com"` | Yes | The override is the name the origin was sent | +| `https://origin.example/news/`, with `origin_host_header_override = "www.example.com"` | No | The host in `origin_url` is where the connection goes and not what the origin is called | + +**What the reader receives.** The scheme and host of the request, then the path, query and fragment of the `Location`. The page pipeline writes the same scheme and host in place of the origin's addresses in a page, but the page pipeline takes the host of `origin_url` as the origin's name even where an override is set, whereas this rule takes the override, which is the name the origin was sent. With an override set, the page pipeline and this rule therefore disagree on what the origin is called, and the page pipeline is not changed here. Where the origin's name is the request's own host, the header comes back naming the same host on the request's scheme. A request that names no host has nowhere to send the reader, so its redirect is left as written. + +**The origin path.** Where `origin_url` has a path, the origin is that folder and not the whole host. A `Location` whose path is the origin path, or beneath it, comes back with the origin path removed, which reverses the way the origin address was built. A request for `/news` is sent to `/bucket/site/news`, and a redirect to `/bucket/site/news/` reaches the reader as `/news/`. A `Location` that is the origin path itself reaches the reader as `/`. The origin path counts only where it ends at a `/`, so `/bucket/site` does not come off `/bucket/sitemap.xml`. Paths are compared exactly, so `/Bucket/Site` is not `/bucket/site`. Dot segments in the `Location` are resolved before the comparison, so a path that climbs out of the folder is outside it. A `Location` on the origin's host and outside the origin path is left as written, because no address on the reader's host leads there. + +**A publisher addressed by a path prefix.** The request addressing design (`2026-10-13-request-addressing-design.md`) lets one host serve several publishers, each under a prefix of its own. The prefix is removed from the path before the request is routed, so the origin is asked for the path without it, and core never writes the removed prefix into a response. The rewritten `Location` is made from the request's scheme and host and the path of the `Location`, and nothing in `publisher.rs` reads the removed prefix, so the reader of such a publisher is sent to an address without the prefix. With the example hook in the request addressing design, on a Fastly service whose `__KEY` selector is `{host}`, a reader of `https://front.example/publisher-a.example/news` whose origin answers `Location: https://origin.example/news/` receives `Location: https://front.example/news/`. The hook reads `news` as the publisher, so the reader is answered `421 Misdirected Request` unless a publisher has that name. Before this change the same reader followed the origin's own address to the origin. A serve middleware puts the prefix back, through the headers step (section 3), and the step is given the rewritten `Location` and the request state a module's request preparer left. Core does not. No module in this repository reads the removed prefix, so none puts it back yet, and no test sends a request addressed by a path prefix through the rule. A root-relative `Location`, such as `/news/`, is resolved by the reader's browser against the root of the host, so it loses the prefix in the same way with or without this change. + +**What is left alone.** A `Location` naming any other host is the origin sending the reader somewhere else. A `Location` that names no host, such as `/news/`, `news/` or `?page=2`, is resolved by the reader's browser against the address it asked for, so it stays on the deployment as it is. A value that cannot be read as an address is relayed as it arrived. + +**When the header cannot be rewritten.** A `Location` with bytes outside printable ASCII, and a rewritten value that is not a valid header value, are each relayed as the origin wrote them. When a response carries several `Location` headers, only the first is read, and when the first names the origin the rewritten value replaces all of them. None of these three cases has a test. + +**Where it happens.** In one place, `handle_publisher_request` in `publisher.rs`, as soon as the origin's response is received. Two things come first and neither reads the `Location`. They are the removal of the three headers that describe the deployment (`X-Cache`, `X-Cache-Hits` and `X-Served-By`) and one debug log line that names the status and the number of headers. The streamed page, the buffered page, a `HEAD` and every other method return that one response, so each carries the rewritten header. A copy of an asset from the asset cache enters at the same point as the origin's own answer. Neither the template cache nor the asset cache keeps a response other than `200`, so no stored copy holds a redirect. The header is written for each request from that request's own scheme and host, after the platform has returned the origin's response to core. The rule itself is `rewritten_origin_redirect`, which takes the `Location`, the origin's host and path and the request's scheme and host, and returns the new value or nothing. + +**Adapters.** No adapter changes. The rule needs a `3xx` to reach core with its `Location` as the origin wrote it, and that holds on all four adapters. The scheme and host the rule writes are the two values `RequestInfo::from_request` reads for the page pipeline, and how each adapter fills them differs. + +| Adapter | A `3xx` reaches core because | The scheme comes from | The host comes from | +| ---------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | +| Fastly | The client has no mode that follows a redirect | The connection's TLS signal, with `Forwarded`, `X-Forwarded-Host`, `X-Forwarded-Proto` and `Fastly-SSL` stripped first | The `Host` header | +| Spin | The client has no mode that follows a redirect | The runtime's own URL for the request, with the same headers stripped | The runtime's own URL for the request | +| Axum | The client's redirect policy is none | `Forwarded`, `X-Forwarded-Proto` or `Fastly-SSL` where the request carries one, otherwise `http` | `Forwarded` or `X-Forwarded-Host` where the request carries one, otherwise `Host` | +| Cloudflare | The client asks for manual redirect handling | As Axum | As Axum | + +The Axum and Cloudflare adapters set no TLS signal and strip only the configured client IP headers, so a client of either can choose the host its own redirect names, as it can already choose the host the page pipeline writes into a page. Each adapter already has a test that a redirect from its client is left for core to see, and those tests are not part of this change. + +**Consent and privacy.** The rewritten `Location` is made from the request's scheme and host and the path, query and fragment of the `Location`. The rule adds nothing to the address, reads no consent state and makes no request, so it has no effect on consent. Its debug log line is a fixed sentence that prints neither address. + +**Why core and not a middleware.** The redirect answers the request core made to the one origin `[publisher]` names, and the rule uses only that origin's own name and path and the request's own scheme and host. The rule names no vendor and reads no module's settings. Rewriting the origin's own address is part of the origin fetch, the step that turns the origin's answer into the deployment's. Core already moves the origin's address in a page to the reader's host before any middleware sees the page, and a `Location` naming the origin is the same address in a header. The headers step in section 3 lets a middleware change a header, and three things keep this rule out of a middleware. A middleware is told the host of `origin_url` and no origin path, and with an override set that host is not the name the origin was sent. A middleware runs only where a `[[serve]]` entry covers the request path, so a deployment that named none would still send its readers off the deployment. And the headers step is not asked about every response (section 3). What a deployment decides for itself, such as where a redirect to another name of the site goes or whether to put a removed path prefix back, is a middleware's, through that step. + +**Why no setting, and what changes for a deployment.** A `Location` naming the origin takes the reader off the deployment that is serving them, which no deployment asks for by naming an origin. A deployment whose origin sends no redirect to itself sees no change. For a deployment whose origin does, the reader stays on the deployment where before the reader's browser left it. Section 4 gives the two cases where the reader's browser loops instead. No setting is added or changed. `origin_host_header_override` gains a second job, being the name a redirect is compared with, and the configuration guide says so under `origin_url` and under the override. Loading the settings already refuses an override that is not a host name, an IPv4 address or a bracketed IPv6 address with an optional port, so the comparison never reads anything else. -**What the reader receives.** The scheme and host of the request, which are the two values the page pipeline already uses to rewrite the origin's addresses in a page, then the path, query and fragment of the `Location`. Where the origin's name is the request's own host, the header comes back naming the same host on the request's scheme. A request that names no host has nowhere to send the reader, so its redirect is left as written. +## 3. The serve middleware headers step -**The origin path.** Where `origin_url` has a path, the origin is that folder and not the whole host. A `Location` whose path is the origin path, or beneath it, comes back with the origin path removed, which reverses the way the origin address was built. A request for `/news` is sent to `/bucket/site/news`, and a redirect to `/bucket/site/news/` reaches the reader as `/news/`. The origin path counts only where it ends at a `/`, so `/bucket/site` does not come off `/bucket/sitemap.xml`. Dot segments in the `Location` are resolved before the comparison, so a path that climbs out of the folder is outside it. A `Location` on the origin's host and outside the origin path is left as written, because no address on the reader's host leads there. +A serve middleware may set headers on a response the publisher path sends, a redirect included, except for the responses listed under "Where the step is called" below. The redirect rule in section 2 does not use the step. A deployment's own policy for a redirect does, such as where a `Location` for another name of the site goes, or putting a removed path prefix back. -**What is left alone.** A `Location` naming any other host is the origin sending the reader somewhere else, and a host beneath the origin's, such as `www.origin.example`, is another host. A `Location` that names no host, such as `/news/`, `news/` or `?page=2`, is resolved by the reader's browser against the address it asked for, so it stays on the deployment as it is. A value that cannot be read as an address is relayed as it arrived. +**What it adds.** A method on `Middleware`, with a default that returns no headers, and the read only view of the response that the method is given, both in `crates/trusted-server-core/src/middleware.rs`. A middleware written before the method compiles and behaves as before, and a request path no `[[serve]]` entry covers sees no change. -**Where it happens.** In one place, `handle_publisher_request` in `publisher.rs`, at the point the origin's response is received and before anything else reads it. The streamed page, the buffered page, a `HEAD` and every other method return that one response, so each carries the rewritten header. The header is written for each request from that request's own scheme and host, after the origin's response has been returned to core. The rule itself is `rewritten_origin_redirect`, which takes the `Location`, the origin's host and path and the request's scheme and host, and returns the new value or nothing. +```rust +pub trait Middleware: Send + Sync { + fn response_headers( + &self, + context: &MiddlewareContext<'_>, + response: &ResponseView<'_>, + ) -> Vec { + Vec::new() + } +} -**Why core.** The redirect answers the request core made to the one origin `[publisher]` names, and the rule uses only that origin's own name and path and the request's own scheme and host. It names no vendor and reads no module's settings. +pub struct ResponseView<'a> { + pub status: StatusCode, + pub headers: &'a http::HeaderMap, +} +``` -**Why no setting.** A `Location` naming the origin takes the reader off the deployment that is serving them, which no deployment asks for by naming an origin. A deployment whose origin sends no redirect to itself sees no change. +`ResponseView` holds the status of the response and a borrowed view of its headers. -## 3. What is not in this change +**Which middleware are asked.** Every middleware named by an entry of `[[serve]]` whose path covers the request path, whatever the entry's media type, in entry order and each once. A header is not about a body, so the media type, which says what the body is, is not compared. `PhaseEntry::covers_path` says whether an entry covers a path, `PhaseEntries::for_path` gives the names, and `IntegrationRegistry::middleware_for_path` gives the middleware, leaving out a name no running module supplies. Where no entry covers the path, nothing is built and nothing is asked. -- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a serve middleware of its own, through `Middleware::response_headers`, which is asked about every response the publisher path sends, a redirect with no body included. +**What a middleware is given.** A `MiddlewareContext` for the serve phase, made as for a page. The context carries the request's host and scheme, the host of `origin_url`, the platform identity, and the request state the modules' request preparers left, copied into a document state of its own. Nothing a middleware prepared is available, because the step runs no preparation. + +**What a middleware may do.** Return headers. Each is written on the response in middleware order, replacing any header of the same name, so a later middleware replaces an earlier one's header of that name. The method cannot change the status or the body. A header the same middleware sets for a page through the action it returns from `create` is written later, in the page pass, and replaces the header set here. + +**Where it runs.** In `handle_publisher_request`, once the origin's answer is in hand, after the redirect rule, after a file that core makes where the origin has none has replaced the origin's answer, and before core's cache rules, the modules' `finalize_response` hooks and the adapter's finalizing. A middleware therefore sees the rewritten `Location`, and a cache header it sets can be replaced by the rules that follow, such as the `private, no-store` that core gives a page it has changed for one reader. + +**Where the step is called.** The step is called at three places in `handle_publisher_request`: + +1. The response the handler goes on with, being the origin's answer, a copy of an asset from the asset cache, or the file core makes where the origin has none. +2. The `502` core sends in place of an unexpected `304` from the origin. +3. The refusal of a streamed page by a middleware's preparation. + +The step is not called for three kinds of response: + +1. A refusal made before the origin is asked, being the `405` for a method a signed origin does not take and the `400` for a path with a dot segment under an origin folder. +2. A page served from the template cache, and the refusal of such a page by a middleware's preparation. That page has its headers built from the stored template, which replays a fixed list of policy headers, so a header the step set outside that list is missing from the page. The template cache is used only in the `Esi` assembly mode, so a deployment in the default `Inline` mode has no page the step misses on this account. +3. An answer made from an error, such as a failure to reach the origin, which `handle_publisher_request` returns to the adapter as an error. + +## 4. What is not in this change + +- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a serve middleware of its own, through `Middleware::response_headers` (section 3). The step is asked about a redirect with no body, though not about every response the publisher path sends. - **An origin that redirects away from the scheme it is asked on.** The scheme of the `Location` is not compared, and the reader is sent to their own scheme. An origin that `origin_url` reaches over plain HTTP, and that answers each request with the same address on `https`, sends the reader back to the address they asked for, which gets the same answer again. `origin_url` names the scheme the origin serves its pages on. +- **An origin that redirects an index file back to its folder.** With `directory_index` set, the index file is added to every request path that ends in `/`, so a request for `/news/` is sent to the origin as `/news/index.html`. An origin that answers that request with a redirect to `/news/` sends the reader back to the address they asked for, which gets the same answer again. Before this change the reader followed the origin's own address and left the deployment, and now the reader's browser asks again each time. - **A `Location` that names no host, from an origin in a folder.** An origin under a path in `origin_url` that answers `Location: /bucket/site/news/` has the header relayed as written, and the reader's browser then asks for a path with the folder in it. +- **Putting a removed path prefix back.** A serve middleware does so through the headers step, and no module in this repository reads the removed prefix yet. See "A publisher addressed by a path prefix" in section 2. +- **The name the page pipeline gives the origin.** With an override set, the page pipeline still moves the host of `origin_url` in a page and not the override. This change leaves the page pipeline as it was. - **Following a redirect.** The publisher proxy follows no redirect itself. The reader's browser makes the next request. - **Any header but `Location`.** An address in `Content-Location`, `Link` or `Refresh` is relayed as before. - **Asset routes and the first-party proxy.** Their redirects are handled in `proxy.rs` and are unchanged. -## 4. Acceptance +## 5. Acceptance -Tests in `trusted-server-core`, where the code is: +Tests in `trusted-server-core`, where the code is, for the redirect rule: - A `Location` naming the origin comes back on the request's scheme and host with its path, query and fragment, for an origin on `https` and on plain HTTP, with and without a port, named by an address in brackets, in any letter case, and with the scheme left off. - With `origin_host_header_override` set, a `Location` naming the override is rewritten and one naming the host of `origin_url` is not. - A `Location` naming another host, a host beneath the origin's, a host that only starts or ends with the origin's name, or the origin's name on a different port is left as written. So is a path, a query or a fragment alone, an address that is not `http` or `https`, a value that is no address, and any `Location` when the request names no host. -- With a path in `origin_url`, a `Location` at or beneath that path comes back without it. One beside it, above it or climbing out of it is left as written, and the origin path comes off only where it ends at a `/`. +- With a path in `origin_url`, a `Location` at or beneath that path comes back without it, and the origin path itself comes back as `/`. One beside it, above it, in another letter case or climbing out of it is left as written, and the origin path comes off only where it ends at a `/`. - Through `handle_publisher_request` with a scripted origin, the reader receives the rewritten `Location` on a buffered answer and on a streamed page, for a `GET`, a `HEAD`, a `POST`, a `PUT` and a `DELETE`, for `301`, `302`, `303`, `307` and `308`, and on the request's own scheme. - Through the same handler, a redirect to another host, a redirect naming no host, a `Location` on a `200`, a `201`, a `404` and a `503`, and a redirect with no `Location` each reach the reader as the origin sent them. -Sixteen deliberate faults were made, one in each run, and each was caught by a test: a `Location` rewritten on any status, the port left out of the comparison, the origin path not removed, the origin path removed where it does not end at a `/`, a `Location` outside the origin path rewritten with its path whole, the fragment dropped, the query dropped, an address that is not `http` or `https` rewritten, a request with no host rewritten, a `Location` naming no host resolved against the origin, the host of `origin_url` compared where an override is set, a `Location` with the scheme left off not read, the host compared in exact letter case, the reader always sent to `https`, the origin path not passed by the handler, and the origin path itself given an empty path. +Tests in `trusted-server-core` for the headers step: + +- `PhaseEntries::for_path` names every entry that covers a path whatever its media type, each middleware once in entry order, leaves out an entry whose path does not cover the request, and names nothing when there are no entries (`for_path_names_every_entry_covering_the_path_whatever_its_media_type`, in `middleware/tests.rs`). +- `IntegrationRegistry::middleware_for_path` gives the middleware those names stand for, once each and in entry order, and leaves out a name no module supplies (`middleware_for_path_gives_every_covering_entry_s_middleware_once_in_entry_order`, in `integrations/registry.rs`). +- Through `handle_publisher_request`, a middleware named for the path sets the `Location` of a redirect with no body and keeps the header it set, leaves a `Location` it does not set as written, and is not asked when its entry covers another path (`a_serve_middleware_named_for_the_path_sets_a_redirects_location`, in `publisher.rs`). +- The file core makes where the origin has none carries the headers a middleware sets, and so does the `502` for an unexpected origin `304` (`a_made_file_carries_the_headers_a_serve_middleware_sets` and `the_502_for_an_unexpected_origin_304_carries_a_serve_middlewares_headers`, in `publisher.rs`). + +Nothing tests the following: + +- The step on the refusal of a streamed page by a middleware's preparation, which is the third place the step is called. +- That the step is not called for a refusal made before the origin is asked, for a page from the template cache, or for an answer made from an error. +- Several middleware setting a header of one name on one response. +- A `Location` with bytes outside printable ASCII, a rewritten value that is not a valid header value, and several `Location` headers on one response. +- A request addressed by a path prefix, sent through the redirect rule. +- Any adapter, because no adapter changes, although each already tests that its client leaves a redirect for core to see. + +Sixteen deliberate faults were made, one in each run, and each was caught by a test: a `Location` rewritten on any status, the port left out of the comparison, the origin path not removed, the origin path removed where it does not end at a `/`, a `Location` outside the origin path rewritten with its path whole, the fragment dropped, the query dropped, an address that is not `http` or `https` rewritten, a request with no host rewritten, a `Location` naming no host resolved against the origin, the host of `origin_url` compared where an override is set, a `Location` with the scheme left off not read, the host compared in exact letter case, the reader always sent to `https`, the origin path not passed by the handler, and a `Location` that is the origin path itself rewritten with an empty path and not `/`. All sixteen are in the redirect rule and the handler's use of the rule, and none is recorded for the headers step. + +## Review log + +1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/37-origin-redirect` at `73801ddb7`. Issues found, three of them blocking. The design called the serve middleware headers step outside the change though the series adds it, said the step is asked about every response the publisher path sends though three kinds of response skip it, and said nothing of the reader of a publisher served under a path prefix, who is sent to an address without the prefix. This revision adds a section for the step with its tests, lists the responses the step skips, says a serve middleware and not core puts the prefix back, corrects the partly true claims, fills the gaps in coverage, takes the advice the round gave, and corrects the same sentence in the changelog and the configuration guide. ## Revision record -| Date | Change | -| ---------- | ------------ | -| 2026-10-13 | First draft. | +| Date | Change | +| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-10-13 | First draft. | +| 2026-10-14 | Names its pull request and issue, and points a deployment that serves two names of the site at the serve middleware headers step. | +| 2026-10-15 | Fixes from review round 1. The headers step as a section with its tests and the responses it skips, who puts a removed path prefix back, the adapters, the setting, consent, failure and the index file loop, a table of what names the origin, the title now saying host, and the same sentence corrected in the changelog and the configuration guide. | From 2e12310e5665a7825c81ddc83c26b2688142dc8c Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 17:28:19 +0100 Subject: [PATCH 10/14] The origin redirect design takes its round 2 corrections Review round 2 approved the design with no blocking issues and found two gaps in the changelog and the configuration guide. Neither named the two origins that make the reader's browser loop, being one reached over plain HTTP that redirects to its own https address and one that redirects its index file back to its folder, or that a reader under a removed path prefix is sent to an address without the prefix, which a Fastly service whose selector carries {host} answers 421. Both named only two of the three kinds of response the headers step is not called for. The sentences now name an answer made from an error and the refusal of a template cache page as well. The design records the round. --- CHANGELOG.md | 4 ++-- docs/guide/configuration.md | 10 ++++++++-- .../specs/2026-10-13-origin-redirect-design.md | 2 ++ 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c60a5f97d..df46a28f15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- A redirect the publisher origin sends to itself keeps the reader on the address they asked for. Where the `Location` of a `3xx` response names the origin, being the host of `origin_url` with its port, or `origin_host_header_override` where one is set, the publisher proxy sends the reader to the same path, query and fragment on the request's scheme and host, where it relayed the origin's own address and the reader's browser left the deployment. With a path in `origin_url`, a `Location` under that path comes back without it and one outside it is unchanged. A `Location` naming any other host, one that names no host, and the `Location` of a response that is not a redirect are relayed as before. +- A redirect the publisher origin sends to itself keeps the reader on the address they asked for. Where the `Location` of a `3xx` response names the origin, being the host of `origin_url` with its port, or `origin_host_header_override` where one is set, the publisher proxy sends the reader to the same path, query and fragment on the request's scheme and host, where it relayed the origin's own address and the reader's browser left the deployment. With a path in `origin_url`, a `Location` under that path comes back without it and one outside it is unchanged. A `Location` naming any other host, one that names no host, and the `Location` of a response that is not a redirect are relayed as before. An origin reached over plain HTTP that redirects each request to the same address on `https`, and one that redirects its `directory_index` file back to its folder, make the reader's browser loop. Where a request addressing hook removes a path prefix, the reader is sent to an address without the prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers `421` if every selector value misses. - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServices::geo` and `IntegrationRegistration::with_geo_module` take an `Arc`, and `build_geo_module` returns one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths, and the Fastly adapter's two finalize paths, build the call from the request. - A module declares the caches it loads through, by name, and a deployment selects a loading cache for each in its own `[cache.]` table, the way `[cache.robots-txt]` selects one for robots.txt. A `[cache.]` table naming a cache no linked module declares is refused by `ts config validate` and when the deployment starts, naming the ones it does, and a module declaring `robots-txt` or a name another module declares is refused. `IntegrationBuilder::with_cache_features` declares them, `CacheSettings::loading_caches` lists core's own and the declared ones, and `LoadingCaches::insert` takes any name. - **Breaking for crates outside core:** the cookie constants `COOKIE_TS_EC`, `COOKIE_TS_EC_RESOLVED`, `COOKIE_TS_EC_PULL_COMPLETE`, `COOKIE_TS_EIDS` and `COOKIE_TS_TESTER` are `COOKIE_EC`, `COOKIE_EC_RESOLVED`, `COOKIE_EC_PULL_COMPLETE`, `COOKIE_EIDS` and `COOKIE_TESTER`, holding the entry's name alone, and a cookie is named through the request's public prefix with `services.public_prefix().cookie_name(COOKIE_EC)`. A module's response header naming a cookie under the public prefix is refused as one under `ts-` was. @@ -90,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- A serve middleware may set headers on a response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. The method is not called for a refusal made before the origin is asked, such as the `405` for a method a signed origin does not take, or for a page served from the template cache. +- A serve middleware may set headers on a response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. The method is not called for a refusal made before the origin is asked, such as the `405` for a method a signed origin does not take, for a page served from the template cache or the refusal of such a page, or for an answer made from an error. - A publisher's site may be served from a folder of object storage through the publisher proxy, so its pages pass through the page pipeline, which an asset route skips. Three optional `[publisher]` settings do it, each off unless set. A path in `origin_url` is kept and the request path is added after it with one `/`. `origin_auth` takes the table an asset route's `auth` takes and signs every origin request with it, through the same signer. `directory_index` names a file added to a request path ending in `/`, for the origin request only. A signed origin is only read, so a method other than `GET` or `HEAD` is answered `405`, and a signed request carries none of the reader's cookies, credentials, referrer, conditional validators or range. With a path in `origin_url`, a request path with a `.` or `..` segment is answered `400`. A malformed `origin_auth` or `directory_index` is refused when the settings load, with the setting named. - `[ad-tag.google] write_loader` writes the loader tag into the head, last among the module's inserts and asking for the loader on the module's own route, for a page that carries no loader of its own. Off by default, because a page that already carries the loader would otherwise load it twice. - The publisher's assets and the answers of the asset routes are held through the loading cache, selected in `[cache.assets]`, so an asset is fetched once however many readers and instances ask for it at the same moment, and the copy in the host's shared store answers every instance until the origin's own lifetime ends. A copy is kept under what the origin says it varies on, served with the headers that describe it and an `Age` that counts from the origin's, and nothing is kept that may not be shared. A deployment that names no module takes the one loading cache module the build links, and a build that links none holds nothing, as before. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index da215b4802..ef96cafaec 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -459,7 +459,12 @@ set. When `origin_url` has a path, a `Location` under that path comes back without it, and one outside it is left as the origin wrote it. A `Location` naming any other host, one that names no host (such as `/news/`), and the `Location` of a response that is not a redirect are also left as the origin -wrote them. +wrote them. An origin reached over plain HTTP that redirects each request to +the same address on `https`, and one that redirects its `directory_index` file +back to its folder, make the reader's browser loop. Where a request addressing +hook removes a path prefix, the reader is sent to an address without the +prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers +`421` if every selector value misses (see One blob for each path). #### `origin_host_header_override` @@ -2001,7 +2006,8 @@ has a body, with the response's status and headers in hand, so a redirect reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. A serve middleware is not asked about a refusal made before the origin is asked, such as the `405` for a method a signed origin -does not take, or about a page served from the template cache. +does not take, about a page served from the template cache or the refusal of +such a page, or about an answer made from an error. ```toml [[fetch]] diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index 7814ace391..3597bf3d7b 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -169,6 +169,7 @@ Sixteen deliberate faults were made, one in each run, and each was caught by a t ## Review log 1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/37-origin-redirect` at `73801ddb7`. Issues found, three of them blocking. The design called the serve middleware headers step outside the change though the series adds it, said the step is asked about every response the publisher path sends though three kinds of response skip it, and said nothing of the reader of a publisher served under a path prefix, who is sent to an address without the prefix. This revision adds a section for the step with its tests, lists the responses the step skips, says a serve middleware and not core puts the prefix back, corrects the partly true claims, fills the gaps in coverage, takes the advice the round gave, and corrects the same sentence in the changelog and the configuration guide. +2. **Round 2, 2026-10-15.** A lighter review in a fresh session, against this page at `9dafb089e`, of the round 1 fixes and the new text. Approved, with no blocking issues. The changelog and the configuration guide now name the two loops and the `421` for a reader under a removed path prefix, and name all three kinds of response the headers step is not called for. ## Revision record @@ -177,3 +178,4 @@ Sixteen deliberate faults were made, one in each run, and each was caught by a t | 2026-10-13 | First draft. | | 2026-10-14 | Names its pull request and issue, and points a deployment that serves two names of the site at the serve middleware headers step. | | 2026-10-15 | Fixes from review round 1. The headers step as a section with its tests and the responses it skips, who puts a removed path prefix back, the adapters, the setting, consent, failure and the index file loop, a table of what names the origin, the title now saying host, and the same sentence corrected in the changelog and the configuration guide. | +| 2026-10-15 | Corrections from review round 2. The two loops and the `421` under a removed path prefix, and the three kinds of response the headers step is not called for, named in the changelog and the configuration guide. | From ef10ee326fcc689241cc1659f416b58868891a67 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 18:09:12 +0100 Subject: [PATCH 11/14] The origin redirect design names the specs pull request that holds it The related specs line named IABTechLab/trusted-server#1084 as the pull request that holds every design document of the stack. That pull request was closed on 2026-10-15 and the specs pull request is now IABTechLab/trusted-server#1342, so the line names the new one. Only the number changes. --- docs/superpowers/specs/2026-10-13-origin-redirect-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index 3597bf3d7b..293cf07e88 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -2,7 +2,7 @@ **Status:** Proposed, 2026-10-13. Raised as draft pull request jwrosewell/trusted-server#32, for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review -**Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` and `2026-10-13-request-addressing-design.md`, both in IABTechLab/trusted-server#1084, which holds every design document of the stack +**Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` and `2026-10-13-request-addressing-design.md`, both in IABTechLab/trusted-server#1342, which holds every design document of the stack **Related PRs:** jwrosewell/trusted-server#32 **Issues:** IABTechLab/trusted-server#1330 **Last updated:** 2026-10-15 From a8a1d2ab15769673bb2da3bb9a7531c2329e7adc Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 20:51:35 +0100 Subject: [PATCH 12/14] The design header names the pull request that carries it --- docs/superpowers/specs/2026-10-13-origin-redirect-design.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index 293cf07e88..a0ef209ac8 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -1,11 +1,11 @@ # Design Spec: An origin's redirect to itself keeps the reader on the host they asked for -**Status:** Proposed, 2026-10-13. Raised as draft pull request jwrosewell/trusted-server#32, for Tech Lab review. +**Status:** Proposed, 2026-10-13. Raised as draft pull request IABTechLab/trusted-server#1378 (from jwrosewell/trusted-server#32, which is closed), for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review **Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` and `2026-10-13-request-addressing-design.md`, both in IABTechLab/trusted-server#1342, which holds every design document of the stack -**Related PRs:** jwrosewell/trusted-server#32 +**Related PRs:** IABTechLab/trusted-server#1378, jwrosewell/trusted-server#32 (closed) **Issues:** IABTechLab/trusted-server#1330 -**Last updated:** 2026-10-15 +**Last updated:** 2026-10-16 > **Why this spec exists.** The publisher proxy relays an origin's redirect exactly as the origin wrote it, so a redirect the origin sends to itself hands the reader the origin's own address and their browser leaves the deployment. The same change gives a serve middleware a way to set headers on a response that holds no page, a redirect among them. From 873bb6611784b6dd7825e18e234af31a1e10b00c Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Mon, 19 Oct 2026 14:07:17 +0100 Subject: [PATCH 13/14] The header rewrite maps a folder origin's URLs and replaces the redirect rule The header rewrite merged in main (#1238) maps an origin URL in Location, Content-Location, Refresh, Link and CSP to the serving host, on every route of handle_publisher_request. Its loop exceptions, and its rules for a Location on any status and under a Host override, differ from the Location rule this change added. With both in place the two loop tests from #1238 failed, and so did three of this change's own tests. This change now follows the header rewrite. Its own rule, rewritten_origin_redirect, the call to it and its unit tests are removed. The header rewrite gains origin_path, so with a folder in origin_url a URL on the origin host is mapped only when it is in the folder, and it loses the folder. Dot segments are resolved first, the folder counts only where it ends at a slash, and the loop exceptions compare the reader's path with the URL's path without the folder. A CSP source with no path still gains the whole serving host. The folder cases of the removed unit tests move to the header rewrite's tests. The serve middleware headers step now runs after the header rewrite, so a middleware sees the headers the reader receives and a header it sets is the one sent. Two handler tests take the header rewrite's behavior. A Location on an answer that is not a redirect is named for the reader, and under a Host override the host of origin_url is the origin's name. cargo test -p trusted-server-core --lib -- redirect response_header_rewrite passes 58 tests. The same run on this branch before the change, with #1238 merged, failed five. --- crates/trusted-server-core/src/publisher.rs | 482 +----------------- .../src/response_header_rewrite.rs | 243 ++++++++- 2 files changed, 252 insertions(+), 473 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 933324d8ad..03c974e2be 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3913,68 +3913,6 @@ fn write_serve_middleware_headers( write_response_headers(response, headers); } -/// Returns the `Location` that keeps a reader on the address they asked for -/// when the origin redirects to itself, or `None` to leave the header as the -/// origin wrote it. -/// -/// A `Location` is the origin's own when it names `origin_host`, the `Host` -/// the origin was sent, port included, and its path is `origin_path` or -/// beneath it. The reader is sent to the same place on `request_scheme` and -/// `request_host`, with `origin_path` taken off the path and the query and -/// fragment kept. Relayed as written it would take the reader's browser off -/// the deployment, and a missing trailing slash is enough to make an origin -/// send one. -/// -/// A `Location` naming another host is the origin sending the reader -/// elsewhere, and one outside `origin_path` has no address on the request's -/// host, so both are left alone. So is one that names no host, which the -/// reader's browser resolves against the address it asked for. -fn rewritten_origin_redirect( - location: &str, - origin_host: &str, - origin_path: &str, - request_scheme: &str, - request_host: &str, -) -> Option { - // A request that names no host has no address to send the reader to. - if request_host.is_empty() { - return None; - } - let target = match url::Url::parse(location) { - Ok(target) => target, - // `//host/path` names a host and takes the scheme of the request. - Err(url::ParseError::RelativeUrlWithoutBase) if location.starts_with("//") => { - url::Url::parse(&format!("{request_scheme}:{location}")).ok()? - } - Err(_) => return None, - }; - if !matches!(target.scheme(), "http" | "https") { - return None; - } - - // Compared with the port, because `origin.example` and - // `origin.example:8443` are different origins. The origin's name is read - // on the scheme of the `Location`, so a port that is the usual one for it - // counts as no port on both sides, as `url` writes the `Location`. - let origin = url::Url::parse(&format!("{}://{origin_host}", target.scheme())).ok()?; - if target.host_str()? != origin.host_str()? || target.port() != origin.port() { - return None; - } - - // The origin path counts only where it ends at a separator, so `/site` - // does not come off the front of `/sitemap.xml`. - let path = target - .path() - .strip_prefix(origin_path.trim_end_matches('/')) - .filter(|rest| rest.is_empty() || rest.starts_with('/'))?; - let path = if path.is_empty() { "/" } else { path }; - let query_and_fragment = &target[url::Position::AfterPath..]; - - Some(format!( - "{request_scheme}://{request_host}{path}{query_and_fragment}" - )) -} - /// Returns the status a publisher origin request is refused with before it is /// sent, or `None` when it may be sent. /// @@ -6296,28 +6234,6 @@ pub async fn handle_publisher_request( response.headers().len() ); - // Every route below returns this response, so a redirect the origin sent - // to itself is named for the reader here, once. - if response.status().is_redirection() - && let Some(rewritten) = response - .headers() - .get(header::LOCATION) - .and_then(|value| value.to_str().ok()) - .and_then(|location| { - rewritten_origin_redirect( - location, - &origin_host_header, - parsed_origin.path(), - request_scheme, - request_host, - ) - }) - && let Ok(value) = HeaderValue::from_str(&rewritten) - { - log::debug!("Origin redirect named for the request's host"); - response.headers_mut().insert(header::LOCATION, value); - } - // A file the origin does not have is made, see `entry_making_the_document`. // Core runs no middleware for a request that names no host, so there the // made file would go out empty and the origin's answer stands instead. @@ -6383,14 +6299,6 @@ pub async fn handle_publisher_request( enforce_terminal_private_cache_privacy(&mut response); return Ok(PublisherResponse::Buffered(response)); } - write_serve_middleware_headers( - &mut response, - &header_reader, - &request_path, - settings, - integration_registry, - ); - // Map origin URLs in navigation, preload, and CSP headers to the serving host, // matching the body rewrite, on every route. Runs before the template-cache gate // so a stored template replays the rewritten policy headers; the cache key is @@ -6400,6 +6308,7 @@ pub async fn handle_publisher_request( &OriginHeaderRewrite { origin_host: &origin_host, origin_host_header: &origin_host_header, + origin_path: parsed_origin.path(), request_host, request_scheme, origin_scheme: &origin_scheme, @@ -6407,6 +6316,15 @@ pub async fn handle_publisher_request( request_is_get_or_head: is_get_or_head, }, ); + // Asked after the rewrite, so a serve middleware sees the headers the + // reader receives and a header it sets is the one sent. + write_serve_middleware_headers( + &mut response, + &header_reader, + &request_path, + settings, + integration_registry, + ); let template_cache_policy = TemplateCachePolicy::from_settings(settings); let gate_content_type = response @@ -26073,370 +25991,6 @@ mod tests { const STORAGE_URL: &str = "https://storage.example.com/bucket/site"; const READER_HOST: &str = "publisher.example"; - #[test] - fn a_redirect_naming_the_origin_is_named_for_the_reader() { - // (why, location, origin host, the rewritten location) - let cases = [ - ( - "an absolute redirect to the origin comes back on the reader's host", - "https://origin.example/news", - "origin.example", - "https://publisher.example/news", - ), - ( - "the path, the query and the fragment are kept", - "https://origin.example/news/archive/?page=2&sort=new#latest", - "origin.example", - "https://publisher.example/news/archive/?page=2&sort=new#latest", - ), - ( - "a query alone is kept", - "https://origin.example/news?utm_source=x&page=2", - "origin.example", - "https://publisher.example/news?utm_source=x&page=2", - ), - ( - "a fragment alone is kept", - "https://origin.example/news#latest", - "origin.example", - "https://publisher.example/news#latest", - ), - ( - "a redirect to the origin's root is the reader's root", - "https://origin.example", - "origin.example", - "https://publisher.example/", - ), - ( - "an origin reached over plain HTTP is named for a reader on HTTPS", - "http://origin.example/news", - "origin.example", - "https://publisher.example/news", - ), - ( - "an origin on a port is named with that port", - "https://origin.example:8443/news", - "origin.example:8443", - "https://publisher.example/news", - ), - ( - "a plain HTTP origin on a port is named with that port", - "http://192.0.2.1:9000/news", - "192.0.2.1:9000", - "https://publisher.example/news", - ), - ( - "a scheme's own port written out is the same origin", - "https://origin.example:443/news", - "origin.example", - "https://publisher.example/news", - ), - ( - "a Host header override is the origin's name when one is set", - "https://www.example.com/news", - "www.example.com", - "https://publisher.example/news", - ), - ( - "a Host header override written with the scheme's own port is \ - the same name", - "https://www.example.com/news", - "www.example.com:443", - "https://publisher.example/news", - ), - ( - "a Host header override is compared in any letter case", - "https://www.example.com/news", - "WWW.Example.COM", - "https://publisher.example/news", - ), - ( - "an origin named by an address in brackets is named with them", - "http://[::1]:3000/news", - "[::1]:3000", - "https://publisher.example/news", - ), - ( - "a host is the same host in any letter case, and the path keeps its own", - "https://ORIGIN.Example/News", - "origin.example", - "https://publisher.example/News", - ), - ( - "a redirect that leaves the scheme off still names the origin", - "//origin.example/news?page=2", - "origin.example", - "https://publisher.example/news?page=2", - ), - ]; - for (why, location, origin_host, expected) in cases { - assert_eq!( - rewritten_origin_redirect(location, origin_host, "", "https", READER_HOST) - .as_deref(), - Some(expected), - "should rewrite `{location}`: {why}", - ); - } - } - - #[test] - fn the_reader_is_sent_to_the_scheme_and_host_they_asked_for() { - // (why, location, the reader's scheme, the reader's host, the - // rewritten location) - let cases = [ - ( - "the reader's own scheme is used, so a deployment served over \ - plain HTTP is not sent to a scheme it does not answer", - "https://origin.example/news", - "http", - "publisher.example", - "http://publisher.example/news", - ), - ( - "the reader's host keeps its port", - "https://origin.example/news", - "http", - "localhost:8080", - "http://localhost:8080/news", - ), - ]; - for (why, location, scheme, host, expected) in cases { - assert_eq!( - rewritten_origin_redirect(location, "origin.example", "", scheme, host) - .as_deref(), - Some(expected), - "should rewrite `{location}`: {why}", - ); - } - } - - #[test] - fn a_location_that_does_not_name_the_origin_is_left_alone() { - // (why, location, origin host) - let cases = [ - ( - "a redirect anywhere else is the origin sending the reader off \ - site on purpose", - "https://somewhere.else/news", - "origin.example", - ), - ( - "a host beneath the origin's is another host", - "https://www.origin.example/news", - "origin.example", - ), - ( - "a host that only starts with the origin's is another host", - "https://origin.example.elsewhere.test/news", - "origin.example", - ), - ( - "a host that only ends with the origin's is another host", - "https://notorigin.example/news", - "origin.example", - ), - ( - "the origin's name written as a user name names another host", - "https://origin.example@elsewhere.test/news", - "origin.example", - ), - ( - "an origin on a port is a different origin from the same host \ - without one", - "https://origin.example/news", - "origin.example:8443", - ), - ( - "an origin without a port is a different origin from the same \ - host on one", - "https://origin.example:8443/news", - "origin.example", - ), - ( - "a port that is the usual one for another scheme is a port", - "http://origin.example/news", - "origin.example:443", - ), - ( - "with a Host header override set, the host connected to is not \ - the origin's name", - "https://origin.example/news", - "www.example.com", - ), - ( - "a path from the root names no host", - "/news", - "origin.example", - ), - ( - "a path beside the one asked for names no host", - "news/latest", - "origin.example", - ), - ( - "a path that climbs names no host", - "../news", - "origin.example", - ), - ("a query alone names no host", "?page=2", "origin.example"), - ( - "a fragment alone names no host", - "#latest", - "origin.example", - ), - ("an empty value names no host", "", "origin.example"), - ( - "a redirect that leaves the scheme off may name another host", - "//somewhere.else/news", - "origin.example", - ), - ( - "an address that is not a web address is not the origin's own", - "ftp://origin.example/news", - "origin.example", - ), - ( - "an address with no host is not the origin's own", - "mailto:someone@origin.example", - "origin.example", - ), - ( - "a value that is no address is left as written", - "https://", - "origin.example", - ), - ]; - for (why, location, origin_host) in cases { - assert_eq!( - rewritten_origin_redirect(location, origin_host, "", "https", READER_HOST), - None, - "should leave `{location}` alone: {why}", - ); - } - } - - #[test] - fn a_reader_with_no_host_is_sent_nowhere_new() { - assert_eq!( - rewritten_origin_redirect( - "https://origin.example/news", - "origin.example", - "", - "https", - "", - ), - None, - "should leave the redirect alone when the request names no host" - ); - } - - #[test] - fn a_redirect_under_the_origin_path_loses_that_path() { - // (why, location, origin path, the rewritten location) - let cases = [ - ( - "a path beneath the origin path is the reader's path without it", - "https://storage.example.com/bucket/site/about/", - "/bucket/site", - "https://publisher.example/about/", - ), - ( - "the query and the fragment are kept", - "https://storage.example.com/bucket/site/about/index.html?v=2#team", - "/bucket/site", - "https://publisher.example/about/index.html?v=2#team", - ), - ( - "the origin path itself is the reader's root", - "https://storage.example.com/bucket/site", - "/bucket/site", - "https://publisher.example/", - ), - ( - "the origin path with a trailing slash is the reader's root", - "https://storage.example.com/bucket/site/", - "/bucket/site", - "https://publisher.example/", - ), - ( - "an origin path written with a trailing slash is the same path", - "https://storage.example.com/bucket/site/about/", - "/bucket/site/", - "https://publisher.example/about/", - ), - ( - "a dot segment that stays beneath the origin path is resolved first", - "https://storage.example.com/bucket/site/about/../news/", - "/bucket/site", - "https://publisher.example/news/", - ), - ( - "an origin path of `/` keeps nothing, so nothing comes off", - "https://storage.example.com/bucket/site/about/", - "/", - "https://publisher.example/bucket/site/about/", - ), - ]; - for (why, location, origin_path, expected) in cases { - assert_eq!( - rewritten_origin_redirect( - location, - "storage.example.com", - origin_path, - "https", - READER_HOST, - ) - .as_deref(), - Some(expected), - "should rewrite `{location}`: {why}", - ); - } - } - - #[test] - fn a_redirect_outside_the_origin_path_is_left_alone() { - // (why, location) - let cases = [ - ( - "another folder of the same host is not the publisher's site", - "https://storage.example.com/bucket/other/about/", - ), - ( - "the origin path counts only where it ends at a separator", - "https://storage.example.com/bucket/sitemap.xml", - ), - ( - "a folder whose name starts with the origin path's last is another folder", - "https://storage.example.com/bucket/site-two/about/", - ), - ( - "the host's root is above the origin path", - "https://storage.example.com/", - ), - ( - "a dot segment that climbs out of the origin path is outside it", - "https://storage.example.com/bucket/site/../other/about/", - ), - ( - "a path in another letter case is another path", - "https://storage.example.com/Bucket/Site/about/", - ), - ]; - for (why, location) in cases { - assert_eq!( - rewritten_origin_redirect( - location, - "storage.example.com", - "/bucket/site", - "https", - READER_HOST, - ), - None, - "should leave `{location}` alone: {why}", - ); - } - } - fn settings_with_origin(origin_url: &str) -> Settings { let mut settings = create_test_settings(); settings.publisher.origin_url = origin_url.to_owned(); @@ -26633,18 +26187,18 @@ mod tests { } #[tokio::test] - async fn the_host_header_override_is_the_origin_s_name_when_set() { + async fn under_a_host_override_the_host_of_origin_url_is_the_origin_s_name() { let mut settings = settings_with_origin(ORIGIN_URL); settings.publisher.origin_host_header_override = Some("www.example.com".to_owned()); // (the origin's `Location`, what the reader receives) let cases = [ ( - "https://www.example.com/news/", + "https://origin.example/news/", "https://publisher.example/news/", ), ( - "https://origin.example/news/", - "https://origin.example/news/", + "https://www.example.com/news/", + "https://www.example.com/news/", ), ]; for (sent, expected) in cases { @@ -26659,7 +26213,7 @@ mod tests { assert_eq!( location(&response), Some(expected), - "should treat the override as the origin's name for `{sent}`" + "should treat the host of `origin_url` as the origin's name for `{sent}`" ); } } @@ -26742,7 +26296,7 @@ mod tests { } #[tokio::test] - async fn a_location_on_an_answer_that_is_no_redirect_is_unchanged() { + async fn a_location_on_an_answer_that_is_no_redirect_is_named_for_the_reader() { let settings = settings_with_origin(ORIGIN_URL); // (the origin's status, the origin's media type, the route the // answer takes) @@ -26774,8 +26328,8 @@ mod tests { ); assert_eq!( location(&response), - Some("https://origin.example/news/"), - "should leave the `Location` of a {status} as the origin wrote it" + Some("https://publisher.example/news/"), + "should name the `Location` of a {status} for the reader" ); } } diff --git a/crates/trusted-server-core/src/response_header_rewrite.rs b/crates/trusted-server-core/src/response_header_rewrite.rs index c2a1d1c247..52149d2733 100644 --- a/crates/trusted-server-core/src/response_header_rewrite.rs +++ b/crates/trusted-server-core/src/response_header_rewrite.rs @@ -12,7 +12,8 @@ //! `Refresh`, `Link`): an absolute or protocol-relative URL whose authority is //! the publisher origin is mapped to the serving host, using the same //! scheme mapping as the HTML body rewriter. Relative URLs and other hosts -//! are left unchanged. +//! are left unchanged. When the origin is a folder of its host, only a URL in +//! that folder is mapped, and it loses the folder. //! - **Content Security Policy** (`Content-Security-Policy`, //! `Content-Security-Policy-Report-Only`): every source expression that names //! the publisher origin gains a sibling expression for the serving host. @@ -24,6 +25,8 @@ //! preserved. Operator `[response_headers]` overrides are applied later and //! still take precedence. +use std::borrow::Cow; + use edgezero_core::http::{HeaderMap, HeaderName, HeaderValue, header}; use url::Url; @@ -50,6 +53,13 @@ pub(crate) struct OriginHeaderRewrite<'a> { /// point of view, so the retry through Trusted Server is redirected again /// whatever cookies the response sets. pub origin_host_header: &'a str, + /// Path of the publisher origin's folder (`/bucket/site`), or empty or `/` + /// when the origin is the whole host. + /// + /// A URL on the origin host is the origin's only at or beneath this path, + /// and maps to the serving host without it. One outside the folder has no + /// address on the serving host and is left unchanged. + pub origin_path: &'a str, /// Serving host authority the browser addressed. pub request_host: &'a str, /// Scheme the browser used to reach the serving host. @@ -84,23 +94,57 @@ impl OriginHeaderRewrite<'_> { /// Rewrite a URL reference that points at the publisher origin. /// - /// Returns [`None`] when `url` is relative, names another host, or names the - /// origin host with a different port. + /// Returns [`None`] when `url` is relative, names another host, names the + /// origin host with a different port, or names a path outside the origin + /// folder. fn rewrite_url(&self, url: &str) -> Option { let rest = strip_prefix_ignore_ascii_case(url, "https://") .or_else(|| strip_prefix_ignore_ascii_case(url, "http://")); if let Some(rest) = rest { - let suffix = self.strip_origin_authority(rest)?; + let suffix = self.serving_suffix(self.strip_origin_authority(rest)?)?; return Some(format!( "{}://{}{suffix}", self.request_scheme, self.request_host )); } - let suffix = self.strip_origin_authority(url.strip_prefix("//")?)?; + let suffix = self.serving_suffix(self.strip_origin_authority(url.strip_prefix("//")?)?)?; Some(format!("//{}{suffix}", self.request_host)) } + /// What follows the serving host for `suffix`, the path, query and + /// fragment that follow the origin authority in a URL on the origin. + /// + /// With an [`Self::origin_path`], dot segments are resolved first, as the + /// browser resolves them, so a path that climbs out of the folder is + /// outside it, and [`None`] is returned for a path outside the folder. + fn serving_suffix<'b>(&self, suffix: &'b str) -> Option> { + if self.origin_path.trim_end_matches('/').is_empty() { + return Some(Cow::Borrowed(suffix)); + } + let target = Url::parse(&format!( + "{}://{}{suffix}", + self.origin_scheme, self.origin_host + )) + .ok()?; + let path = self.serving_path(target.path())?; + Some(Cow::Owned(format!( + "{path}{}", + &target[url::Position::AfterPath..] + ))) + } + + /// The serving-host path for `path`, a path on the origin host, being + /// `path` without [`Self::origin_path`], or [`None`] when `path` is outside + /// that folder. The folder counts only where it ends at a `/`, so + /// `/bucket/site` does not come off `/bucket/sitemap.xml`. + fn serving_path<'b>(&self, path: &'b str) -> Option<&'b str> { + let path = path + .strip_prefix(self.origin_path.trim_end_matches('/')) + .filter(|path| path.is_empty() || path.starts_with('/'))?; + Some(if path.is_empty() { "/" } else { path }) + } + /// Rewrite a navigation target (`Location`, `Refresh`), unless rewriting /// it would loop. /// @@ -195,7 +239,7 @@ impl OriginHeaderRewrite<'_> { Some((path, query)) => (path, Some(query)), None => (self.request_path_and_query, None), }; - target.path() == request_path && target.query() == request_query + self.serving_path(target.path()) == Some(request_path) && target.query() == request_query } /// Strip the origin authority from the start of `rest`, returning the @@ -208,15 +252,27 @@ impl OriginHeaderRewrite<'_> { /// Build the serving-host CSP source for a source expression naming the /// origin, or [`None`] when the expression names something else. + /// + /// A source with no path permits every path on the origin, so its sibling + /// names the whole serving host even when the origin is a folder. fn csp_source_for(&self, source: &str) -> Option { if source.starts_with('\'') { return None; } + let rest = source.split_once("://").map_or(source, |(_, rest)| rest); + let mapping = if self.strip_origin_authority(rest)?.is_empty() { + OriginHeaderRewrite { + origin_path: "", + ..*self + } + } else { + *self + }; if source.contains("://") { - return self.rewrite_url(source); + return mapping.rewrite_url(source); } - let suffix = self.strip_origin_authority(source)?; - Some(format!("{}{suffix}", self.request_host)) + let suffix = mapping.serving_suffix(mapping.strip_origin_authority(source)?)?; + Some(format!("{}{suffix}", mapping.request_host)) } } @@ -232,6 +288,7 @@ impl OriginHeaderRewrite<'_> { /// let rewrite = OriginHeaderRewrite { /// origin_host: "origin.example.com", /// origin_host_header: "origin.example.com", +/// origin_path: "", /// request_host: "www.example.com", /// request_scheme: "https", /// origin_scheme: "https", @@ -564,6 +621,7 @@ mod tests { const REWRITE: OriginHeaderRewrite<'static> = OriginHeaderRewrite { origin_host: "origin.example.com", origin_host_header: "origin.example.com", + origin_path: "", request_host: "www.example.com", request_scheme: "https", origin_scheme: "http", @@ -571,6 +629,15 @@ mod tests { request_is_get_or_head: false, }; + /// An origin that is a folder of object storage, served at `www.example.com`. + const FOLDER: OriginHeaderRewrite<'static> = OriginHeaderRewrite { + origin_host: "storage.example.com", + origin_host_header: "storage.example.com", + origin_path: "/bucket/site", + origin_scheme: "https", + ..REWRITE + }; + fn rewritten(name: &HeaderName, values: &[&str]) -> Vec { let mut headers = HeaderMap::new(); for value in values { @@ -645,6 +712,7 @@ mod tests { let rewrite = OriginHeaderRewrite { origin_host: "127.0.0.1:8301", origin_host_header: "127.0.0.1:8301", + origin_path: "", request_host: "127.0.0.1:3031", request_scheme: "http", origin_scheme: "http", @@ -1064,6 +1132,163 @@ mod tests { } } + #[test] + fn folder_origin_maps_urls_in_the_folder_without_it() { + // (why, URL on the origin, URL on the serving host) + let cases = [ + ( + "a path in the folder loses the folder", + "https://storage.example.com/bucket/site/about/", + "https://www.example.com/about/", + ), + ( + "the query and the fragment are kept", + "https://storage.example.com/bucket/site/about/index.html?v=2#team", + "https://www.example.com/about/index.html?v=2#team", + ), + ( + "the folder itself is the serving host's root", + "https://storage.example.com/bucket/site", + "https://www.example.com/", + ), + ( + "the folder with a closing slash is the serving host's root", + "https://storage.example.com/bucket/site/", + "https://www.example.com/", + ), + ( + "a dot segment that stays in the folder is resolved first", + "https://storage.example.com/bucket/site/about/../news/", + "https://www.example.com/news/", + ), + ( + "a protocol-relative URL stays protocol-relative", + "//storage.example.com/bucket/site/about/", + "//www.example.com/about/", + ), + ]; + for (why, url, expected) in cases { + assert_eq!( + FOLDER.rewrite_url(url).as_deref(), + Some(expected), + "should map `{url}`: {why}" + ); + } + + let closing_slash = OriginHeaderRewrite { + origin_path: "/bucket/site/", + ..FOLDER + }; + assert_eq!( + closing_slash + .rewrite_url("https://storage.example.com/bucket/site/about/") + .as_deref(), + Some("https://www.example.com/about/"), + "should read a folder written with a closing slash as the same folder" + ); + let whole_host = OriginHeaderRewrite { + origin_path: "/", + ..FOLDER + }; + assert_eq!( + whole_host + .rewrite_url("https://storage.example.com/bucket/site/about/") + .as_deref(), + Some("https://www.example.com/bucket/site/about/"), + "should take nothing off for an origin path of `/`" + ); + } + + #[test] + fn folder_origin_leaves_urls_outside_the_folder_unchanged() { + // (why, URL on the origin host) + let cases = [ + ( + "another folder of the host", + "https://storage.example.com/bucket/other/about/", + ), + ( + "the folder counts only where it ends at a separator", + "https://storage.example.com/bucket/sitemap.xml", + ), + ( + "a folder whose name starts with the folder's is another folder", + "https://storage.example.com/bucket/site-two/about/", + ), + ( + "the host's root is above the folder", + "https://storage.example.com/", + ), + ( + "a URL that stops at the host names its root", + "https://storage.example.com", + ), + ( + "a dot segment that climbs out of the folder is outside it", + "https://storage.example.com/bucket/site/../other/about/", + ), + ( + "a path in another letter case is another path", + "https://storage.example.com/Bucket/Site/about/", + ), + ]; + for (why, url) in cases { + assert_eq!( + FOLDER.rewrite_url(url), + None, + "should leave `{url}` unchanged: {why}" + ); + } + } + + #[test] + fn folder_origin_keeps_redirects_to_the_current_request_that_would_loop() { + let rewrite = OriginHeaderRewrite { + request_path_and_query: "/news?page=2", + request_is_get_or_head: true, + ..FOLDER + }; + assert_eq!( + rewrite.rewrite_location("http://storage.example.com/bucket/site/news?page=2", true), + None, + "should keep a scheme change to the current request in the folder" + ); + assert_eq!( + rewrite.rewrite_location("https://storage.example.com/bucket/site/news?page=2", false), + None, + "should keep a cookieless GET redirect to the current request in the folder" + ); + assert_eq!( + rewrite + .rewrite_location( + "https://storage.example.com/bucket/site/news/?page=2", + false + ) + .as_deref(), + Some("https://www.example.com/news/?page=2"), + "should map a redirect to another address in the folder" + ); + } + + #[test] + fn folder_origin_csp_sources_follow_the_folder() { + assert_eq!( + rewrite_csp( + "img-src https://storage.example.com storage.example.com/bucket/site/img/ \ + storage.example.com/bucket/other/", + &FOLDER, + ) + .as_deref(), + Some( + "img-src https://storage.example.com https://www.example.com \ + storage.example.com/bucket/site/img/ www.example.com/img/ \ + storage.example.com/bucket/other/" + ), + "should give a source with no path the whole serving host, map one in the \ + folder without it, and add nothing beside one outside it" + ); + } + #[test] fn skips_rewrite_when_hosts_match_or_are_missing() { let mut headers = HeaderMap::new(); From 6ec3e077b3ee027b56771044afde8218739ae9b6 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Mon, 19 Oct 2026 14:21:55 +0100 Subject: [PATCH 14/14] The changelog, the guide and the design follow the header rewrite The changelog entry, the configuration guide and the origin redirect design describe the header rewrite merged in main (#1238) with the origin folder this change gives it, in place of the Location rule the change no longer has. The guide's section on origin URLs in response headers says how a folder is mapped, and the override is no longer named as the name a redirect is compared with. The design records five deliberate faults, each caught by a test. The object storage design's sentence on redirects names the mapping it now follows. --- CHANGELOG.md | 2 +- docs/guide/configuration.md | 38 +++--- .../2026-10-13-origin-redirect-design.md | 126 ++++++++---------- ...-publisher-object-storage-origin-design.md | 2 +- 4 files changed, 79 insertions(+), 89 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fead1ff55..4cd8586f80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- A redirect the publisher origin sends to itself keeps the reader on the address they asked for. Where the `Location` of a `3xx` response names the origin, being the host of `origin_url` with its port, or `origin_host_header_override` where one is set, the publisher proxy sends the reader to the same path, query and fragment on the request's scheme and host, where it relayed the origin's own address and the reader's browser left the deployment. With a path in `origin_url`, a `Location` under that path comes back without it and one outside it is unchanged. A `Location` naming any other host, one that names no host, and the `Location` of a response that is not a redirect are relayed as before. An origin reached over plain HTTP that redirects each request to the same address on `https`, and one that redirects its `directory_index` file back to its folder, make the reader's browser loop. Where a request addressing hook removes a path prefix, the reader is sent to an address without the prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers `421` if every selector value misses. +- The mapping of origin URLs in response headers (#1128) follows a path in `origin_url`. A URL on the origin is mapped to the serving host only when it is in that folder, and it loses the folder, so the reader of a site served from a folder of object storage is sent to the right address. A URL on the origin's host outside the folder is unchanged, because no address on the serving host leads there, and a CSP source with no path still gains the serving host beside it. A reader of a publisher whose request addressing hook removes a path prefix is sent to an address without the prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers `421` if every selector value misses. - **Breaking:** a path in `publisher.origin_url` is kept, and the request path is added after it with one `/`. The path was discarded before, so a deployment whose `origin_url` has one now sends each request to the folder the path names, and answers `400` to a request path with a `.` or `..` segment, with or without `origin_auth`. A deployment whose `origin_url` has no path sends its requests as before. Before upgrading, check that the origin serves the site at that path, or remove the path. - **Breaking:** the check that closes `/_ts/admin` and `/admin/keys` on every adapter now also answers `404` to a request path still changing after four rounds of percent-decoding and to a path that reaches either once an escape that is not valid UTF-8 is replaced. The publisher received both before, and no setting turns the check off. - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServicesBuilder::geo`, `RuntimeServices::with_geo` and `IntegrationRegistrationBuilder::with_geo_module` take an `Arc`, and `RuntimeServices::geo()`, `IntegrationRegistry::geo_module()`, the `IntegrationRegistration::geo_module` field and `build_geo_module` return or hold one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths build the call from the request. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index a91a8c265d..4e4dd964c9 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -448,23 +448,16 @@ is sent as it arrived, as before. When `origin_url` has a path, a request path with a `.` or `..` segment, in any percent-encoded spelling, is refused with `400` so no request reaches outside the folder. -**Redirects**: When the origin answers with a redirect whose `Location` names -the origin itself, the reader is sent to the same path, query and fragment on -the scheme and host they asked for, so their browser stays on the deployment. -An origin asked for `/news` that answers `Location: -https://origin.publisher.com/news/` sends a reader of `www.publisher.com` to -`https://www.publisher.com/news/`. The origin is named by the host in -`origin_url`, with its port, or by `origin_host_header_override` where one is -set. When `origin_url` has a path, a `Location` under that path comes back -without it, and one outside it is left as the origin wrote it. A `Location` -naming any other host, one that names no host (such as `/news/`), and the -`Location` of a response that is not a redirect are also left as the origin -wrote them. An origin reached over plain HTTP that redirects each request to -the same address on `https`, and one that redirects its `directory_index` file -back to its folder, make the reader's browser loop. Where a request addressing -hook removes a path prefix, the reader is sent to an address without the -prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers -`421` if every selector value misses (see One blob for each path). +**Response headers**: When `origin_url` has a path, a URL in a response +header that names the origin is mapped to the serving host only when it is in +that folder, and it loses the folder. With the folder above, `Location: +https://storage.publisher.com/bucket/site/news/` reaches a reader of +`www.publisher.com` as `https://www.publisher.com/news/`, and a URL outside +the folder is left as the origin wrote it (see Origin URLs in proxied response +headers). Where a request addressing hook removes a path prefix, the reader is +sent to an address without the prefix, which a Fastly service whose `__KEY` +selector carries `{host}` answers `421` if every selector value misses (see +One blob for each path). #### `origin_host_header_override` @@ -475,8 +468,6 @@ connecting to the host in `origin_url`. - Connects, uses SNI, and checks certificates against `origin_url` - Sends the configured value as the outbound HTTP `Host` header -- Is the name a redirect from the origin is compared with (see Redirects under - `origin_url`) - Useful when the origin endpoint expects a canonical publisher hostname **Format**: Hostname with optional port, without protocol, path, query, or fragment @@ -1218,6 +1209,15 @@ Location: https://www.example.com/landing.html Content-Security-Policy: img-src https://origin.example.com https://www.example.com data: ``` +When `publisher.origin_url` has a path, such as a folder of object storage, +only a URL in that folder is mapped, and it loses the folder. With +`origin_url = "https://storage.example.com/bucket/site"`, +`https://storage.example.com/bucket/site/news/` becomes +`https://www.example.com/news/`, and `https://storage.example.com/bucket/other/` +is unchanged, because no address on the serving host leads there. A CSP source +with no path still gains the serving host beside it, and the two exceptions +below compare the path without the folder. + Two exceptions prevent a redirect loop. Rewriting either target would send the browser back to the same URL, and the origin would redirect again: diff --git a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md index a0ef209ac8..3c1262b2da 100644 --- a/docs/superpowers/specs/2026-10-13-origin-redirect-design.md +++ b/docs/superpowers/specs/2026-10-13-origin-redirect-design.md @@ -1,69 +1,53 @@ -# Design Spec: An origin's redirect to itself keeps the reader on the host they asked for +# Design Spec: A folder origin's addresses in response headers, and a headers step for serve middleware **Status:** Proposed, 2026-10-13. Raised as draft pull request IABTechLab/trusted-server#1378 (from jwrosewell/trusted-server#32, which is closed), for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review **Related specs:** `2026-10-13-publisher-object-storage-origin-design.md` and `2026-10-13-request-addressing-design.md`, both in IABTechLab/trusted-server#1342, which holds every design document of the stack -**Related PRs:** IABTechLab/trusted-server#1378, jwrosewell/trusted-server#32 (closed) -**Issues:** IABTechLab/trusted-server#1330 -**Last updated:** 2026-10-16 +**Related PRs:** IABTechLab/trusted-server#1378, IABTechLab/trusted-server#1238 (merged, the header rewrite this change extends), jwrosewell/trusted-server#32 (closed) +**Issues:** IABTechLab/trusted-server#1330, IABTechLab/trusted-server#1128 +**Last updated:** 2026-10-19 -> **Why this spec exists.** The publisher proxy relays an origin's redirect exactly as the origin wrote it, so a redirect the origin sends to itself hands the reader the origin's own address and their browser leaves the deployment. The same change gives a serve middleware a way to set headers on a response that holds no page, a redirect among them. +> **Why this spec exists.** The header rewrite merged in main (#1238) maps an address on the origin in a response header to the serving host, and knows the origin only as a host. Since the object storage design, the origin may be a folder of its host, and an address in that folder keeps the folder when it is mapped, so the reader is sent to an address that does not exist. The same change gives a serve middleware a way to set headers on a response that holds no page, a redirect among them. ## 1. Problem -An origin redirects to itself for ordinary reasons, and a request path without its trailing slash is enough. It writes its own name in the `Location` header, because that is the name it was asked for. The publisher proxy relays the header as written, so the reader's browser follows it to the origin. From then on each page is the origin's own answer, with none of the publisher's `[[fetch]]` or `[[serve]]` middleware run on it, at an address the publisher never gave the reader. +The header rewrite (#1238) maps a URL on the origin in `Location`, `Content-Location`, `Refresh`, `Link` and the two Content Security Policy headers to the serving host, keeping its path, on every route of `handle_publisher_request`. The object storage design (`2026-10-13-publisher-object-storage-origin-design.md`) lets `origin_url` name a folder, such as `https://storage.example.com/bucket/site`, and a request for `/news` is sent to the origin as `/bucket/site/news`. An origin that redirects that request to `https://storage.example.com/bucket/site/news/` has the redirect mapped to `https://publisher.example/bucket/site/news/`. The reader's browser asks for that address, which the publisher proxy sends to the origin as `/bucket/site/bucket/site/news/`, an object that does not exist. An address on the origin's host outside the folder, such as `/bucket/other/`, is mapped the same way, though no address on the serving host leads there. The two exceptions the header rewrite makes to avoid a redirect loop compare the path the reader asked for with the path in the header, folder and all, so neither matches under a folder. A second gap sits beside the first. A serve middleware works on the page in a response and writes headers through the action it returns for that page, so a response with no page in it, such as most redirects, reaches no middleware. A deployment that wants a policy of its own for the `Location` of a redirect has nowhere to put the policy. ## 2. Design -There is no setting. When the origin answers a request with a redirect, the publisher proxy reads the `Location` header, and where it names the origin the reader is sent to the same place on the scheme and host they asked for. +There is no setting. The header rewrite is given the path of `origin_url`, the folder the object storage design sends requests into, and maps a URL on the origin only when the URL is in that folder, without the folder. With no path in `origin_url`, or a path of `/`, the mapping is the header rewrite's as merged. -With `origin_url = "https://origin.example"` and a reader who asked for `https://publisher.example/news`: +With `origin_url = "https://storage.example.com/bucket/site"` and a reader who asked for `https://publisher.example/news`: -| The origin answers | The reader receives | -| ----------------------------------------------------------------- | --------------------------------------------------------- | -| `301` with `Location: https://origin.example/news/` | `Location: https://publisher.example/news/` | -| `302` with `Location: https://origin.example/news/?page=2#latest` | `Location: https://publisher.example/news/?page=2#latest` | -| `307` with `Location: //origin.example/news/` | `Location: https://publisher.example/news/` | -| `301` with `Location: https://www.origin.example/news/` | The header as the origin wrote it | -| `301` with `Location: https://elsewhere.example/news/` | The header as the origin wrote it | -| `301` with `Location: /news/` | The header as the origin wrote it | -| `200` or `201` with `Location: https://origin.example/news/` | The header as the origin wrote it | +| The origin answers | The reader receives | +| -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| `301` with `Location: https://storage.example.com/bucket/site/news/` | `Location: https://publisher.example/news/` | +| `302` with `Location: https://storage.example.com/bucket/site/news/?page=2#latest` | `Location: https://publisher.example/news/?page=2#latest` | +| `301` with `Location: https://storage.example.com/bucket/site` | `Location: https://publisher.example/` | +| `301` with `Location: https://storage.example.com/bucket/other/news/` | The header as the origin wrote it | +| `301` with `Location: https://storage.example.com/bucket/sitemap.xml` | The header as the origin wrote it | +| `200` with `Link: ; rel=preload` | `Link: ; rel=preload` | +| `200` with `Content-Security-Policy: img-src https://storage.example.com storage.example.com/bucket/site/img/` | `Content-Security-Policy: img-src https://storage.example.com https://publisher.example storage.example.com/bucket/site/img/ publisher.example/img/` | -**Which responses.** A response with a status from `300` to `399` and a `Location` header. A `Location` on any other status is left as written, so a `201` naming what was created is not touched. +**Which addresses are in the folder.** A URL on the origin, being one whose host and port the header rewrite already takes as the origin's, is in the folder when its path is the folder or beneath it. Dot segments are resolved first, as the reader's browser resolves them, so `/bucket/site/about/../news/` is `/bucket/site/news/` and in the folder, and `/bucket/site/../other/` is outside it. The folder counts only where it ends at a `/`, so `/bucket/site` does not come off `/bucket/sitemap.xml` or `/bucket/site-two/`. Paths are compared exactly, so `/Bucket/Site/` is not the folder. A folder written with a closing `/` is the same folder. The folder itself, with or without its closing `/`, is the serving host's root, `/`. A URL that stops at the origin's host names the host's root, which is outside the folder. -**What names the origin.** The host of the `Location`, with its port, is compared with the `Host` the origin was sent, which is `origin_host_header_override` where one is set and otherwise the host of `origin_url` with its port. Letter case in a host is ignored and the scheme of the `Location` is not compared. The table gives the cases, with `origin_url = "https://origin.example"` unless a row says otherwise. +**What the reader receives.** The URL as the header rewrite maps it, with the folder taken off the path and the query and fragment kept. A protocol-relative URL stays protocol-relative. -| The `Location` names | The origin's name | Because | -| --------------------------------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------- | -| `https://origin.example/news/` | Yes | The host and port are the origin's | -| `https://ORIGIN.example/news/` | Yes | Letter case in a host is ignored | -| `https://origin.example:443/news/` | Yes | The usual port for the scheme counts as no port | -| `http://origin.example/news/` | Yes | The scheme of the `Location` is not compared | -| `//origin.example/news/` | Yes | A `Location` with no scheme names a host and is read on the scheme of the request | -| `https://origin.example:8443/news/` | No | The same host on another port is another origin | -| `https://www.origin.example/news/` | No | A host beneath the origin's is another host | -| `https://origin.example.elsewhere.test/news/` | No | A host that only begins with the origin's name is another host | -| `https://origin.example@elsewhere.test/news/` | No | The text before an `@` is a user name, and the host is `elsewhere.test` | -| `ftp://origin.example/news/` | No | Only an `http` or `https` address counts | -| `https://origin.example/news/`, with `origin_url = "https://origin.example:8443"` | No | The same host without the port is another origin | -| `https://www.example.com/news/`, with `origin_host_header_override = "www.example.com"` | Yes | The override is the name the origin was sent | -| `https://origin.example/news/`, with `origin_host_header_override = "www.example.com"` | No | The host in `origin_url` is where the connection goes and not what the origin is called | +**What is left alone.** A URL on the origin's host outside the folder is left as the origin wrote it, because no address on the serving host leads there, and the reader's browser follows it to the origin's host. -**What the reader receives.** The scheme and host of the request, then the path, query and fragment of the `Location`. The page pipeline writes the same scheme and host in place of the origin's addresses in a page, but the page pipeline takes the host of `origin_url` as the origin's name even where an override is set, whereas this rule takes the override, which is the name the origin was sent. With an override set, the page pipeline and this rule therefore disagree on what the origin is called, and the page pipeline is not changed here. Where the origin's name is the request's own host, the header comes back naming the same host on the request's scheme. A request that names no host has nowhere to send the reader, so its redirect is left as written. +**Content Security Policy sources.** A source expression with no path permits every path on the origin's host, so its serving-host sibling has no path either and permits the whole serving host, as without a folder. A source with a path in the folder gains a sibling without the folder. A source with a path outside the folder gains no sibling. -**The origin path.** Where `origin_url` has a path, the origin is that folder and not the whole host. A `Location` whose path is the origin path, or beneath it, comes back with the origin path removed, which reverses the way the origin address was built. A request for `/news` is sent to `/bucket/site/news`, and a redirect to `/bucket/site/news/` reaches the reader as `/news/`. A `Location` that is the origin path itself reaches the reader as `/`. The origin path counts only where it ends at a `/`, so `/bucket/site` does not come off `/bucket/sitemap.xml`. Paths are compared exactly, so `/Bucket/Site` is not `/bucket/site`. Dot segments in the `Location` are resolved before the comparison, so a path that climbs out of the folder is outside it. A `Location` on the origin's host and outside the origin path is left as written, because no address on the reader's host leads there. +**The loop exceptions.** The header rewrite leaves a `Location` or `Refresh` target unchanged where following it would bring the reader back to the address they asked for (#1238, and "Origin URLs in proxied response headers" in the configuration guide). With a folder, the path the reader asked for is compared with the path of the target without the folder, so both exceptions apply under a folder as they do without one. -**A publisher addressed by a path prefix.** The request addressing design (`2026-10-13-request-addressing-design.md`) lets one host serve several publishers, each under a prefix of its own. The prefix is removed from the path before the request is routed, so the origin is asked for the path without it, and core never writes the removed prefix into a response. The rewritten `Location` is made from the request's scheme and host and the path of the `Location`, and nothing in `publisher.rs` reads the removed prefix, so the reader of such a publisher is sent to an address without the prefix. With the example hook in the request addressing design, on a Fastly service whose `__KEY` selector is `{host}`, a reader of `https://front.example/publisher-a.example/news` whose origin answers `Location: https://origin.example/news/` receives `Location: https://front.example/news/`. The hook reads `news` as the publisher, so the reader is answered `421 Misdirected Request` unless a publisher has that name. Before this change the same reader followed the origin's own address to the origin. A serve middleware puts the prefix back, through the headers step (section 3), and the step is given the rewritten `Location` and the request state a module's request preparer left. Core does not. No module in this repository reads the removed prefix, so none puts it back yet, and no test sends a request addressed by a path prefix through the rule. A root-relative `Location`, such as `/news/`, is resolved by the reader's browser against the root of the host, so it loses the prefix in the same way with or without this change. +**What names the origin.** The header rewrite decides, and this change does not alter it. The origin is the host of `origin_url` with its port, in any letter case, with or without a `Host` override, and the header rewrite maps a `Location` on a response of any status. -**What is left alone.** A `Location` naming any other host is the origin sending the reader somewhere else. A `Location` that names no host, such as `/news/`, `news/` or `?page=2`, is resolved by the reader's browser against the address it asked for, so it stays on the deployment as it is. A value that cannot be read as an address is relayed as it arrived. +**A publisher addressed by a path prefix.** The request addressing design (`2026-10-13-request-addressing-design.md`) lets one host serve several publishers, each under a prefix of its own. The prefix is removed from the path before the request is routed, so the origin is asked for the path without it, and core never writes the removed prefix into a response. The header rewrite writes the request's scheme and host and the path of the URL, and nothing in `publisher.rs` reads the removed prefix, so the reader of such a publisher is sent to an address without the prefix, with or without a folder. With the example hook in the request addressing design, on a Fastly service whose `__KEY` selector is `{host}`, a reader of `https://front.example/publisher-a.example/news` whose origin answers `Location: https://origin.example/news/` receives `Location: https://front.example/news/`. The hook reads `news` as the publisher, so the reader is answered `421 Misdirected Request` unless a publisher has that name. A serve middleware puts the prefix back, through the headers step (section 3), and the step is given the rewritten `Location` and the request state a module's request preparer left. Core does not. No module in this repository reads the removed prefix, so none puts it back yet, and no test sends a request addressed by a path prefix through the header rewrite. A root-relative `Location`, such as `/news/`, is resolved by the reader's browser against the root of the host, so it loses the prefix in the same way. -**When the header cannot be rewritten.** A `Location` with bytes outside printable ASCII, and a rewritten value that is not a valid header value, are each relayed as the origin wrote them. When a response carries several `Location` headers, only the first is read, and when the first names the origin the rewritten value replaces all of them. None of these three cases has a test. +**Where it happens.** In the header rewrite, `rewrite_origin_urls_in_headers` in `response_header_rewrite.rs`, which `handle_publisher_request` in `publisher.rs` calls where #1238 placed it. The folder is the `origin_path` field this change adds to `OriginHeaderRewrite`, which the handler fills from the path of `origin_url`, the path the object storage design builds the origin address from. The call comes after a file that core makes where the origin has none has replaced the origin's answer, and before the template cache gate, so every route the handler returns an origin answer through passes it. A copy of an asset from the asset cache enters at the same point as the origin's own answer. The `502` core sends in place of an unexpected `304` returns before the call, and carries no header from the origin. The serve middleware headers step (section 3) runs straight after the header rewrite. -**Where it happens.** In one place, `handle_publisher_request` in `publisher.rs`, as soon as the origin's response is received. Two things come first and neither reads the `Location`. They are the removal of the three headers that describe the deployment (`X-Cache`, `X-Cache-Hits` and `X-Served-By`) and one debug log line that names the status and the number of headers. The streamed page, the buffered page, a `HEAD` and every other method return that one response, so each carries the rewritten header. A copy of an asset from the asset cache enters at the same point as the origin's own answer. Neither the template cache nor the asset cache keeps a response other than `200`, so no stored copy holds a redirect. The header is written for each request from that request's own scheme and host, after the platform has returned the origin's response to core. The rule itself is `rewritten_origin_redirect`, which takes the `Location`, the origin's host and path and the request's scheme and host, and returns the new value or nothing. - -**Adapters.** No adapter changes. The rule needs a `3xx` to reach core with its `Location` as the origin wrote it, and that holds on all four adapters. The scheme and host the rule writes are the two values `RequestInfo::from_request` reads for the page pipeline, and how each adapter fills them differs. +**Adapters.** No adapter changes. The header rewrite needs a `3xx` to reach core with its `Location` as the origin wrote it, and that holds on all four adapters. The scheme and host it writes are the two values `RequestInfo::from_request` reads for the page pipeline, and how each adapter fills them differs. | Adapter | A `3xx` reaches core because | The scheme comes from | The host comes from | | ---------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | @@ -74,15 +58,15 @@ With `origin_url = "https://origin.example"` and a reader who asked for `https:/ The Axum and Cloudflare adapters set no TLS signal and strip only the configured client IP headers, so a client of either can choose the host its own redirect names, as it can already choose the host the page pipeline writes into a page. Each adapter already has a test that a redirect from its client is left for core to see, and those tests are not part of this change. -**Consent and privacy.** The rewritten `Location` is made from the request's scheme and host and the path, query and fragment of the `Location`. The rule adds nothing to the address, reads no consent state and makes no request, so it has no effect on consent. Its debug log line is a fixed sentence that prints neither address. +**Consent and privacy.** The mapped URL is made from the request's scheme and host and the path, query and fragment of the origin's URL, and the folder only shortens the path. Nothing is added to the address, no consent state is read and no request is made, so the change has no effect on consent. -**Why core and not a middleware.** The redirect answers the request core made to the one origin `[publisher]` names, and the rule uses only that origin's own name and path and the request's own scheme and host. The rule names no vendor and reads no module's settings. Rewriting the origin's own address is part of the origin fetch, the step that turns the origin's answer into the deployment's. Core already moves the origin's address in a page to the reader's host before any middleware sees the page, and a `Location` naming the origin is the same address in a header. The headers step in section 3 lets a middleware change a header, and three things keep this rule out of a middleware. A middleware is told the host of `origin_url` and no origin path, and with an override set that host is not the name the origin was sent. A middleware runs only where a `[[serve]]` entry covers the request path, so a deployment that named none would still send its readers off the deployment. And the headers step is not asked about every response (section 3). What a deployment decides for itself, such as where a redirect to another name of the site goes or whether to put a removed path prefix back, is a middleware's, through that step. +**Why core and not a middleware.** The folder is part of the origin `[publisher]` names, and the header rewrite that maps the origin's addresses is in core (#1238). A middleware is told the host of `origin_url` and no origin path, and runs only where a `[[serve]]` entry covers the request path, so a deployment that named none would still send its readers to an address that does not exist. What a deployment decides for itself, such as where a redirect to another name of the site goes or whether to put a removed path prefix back, is a middleware's, through the headers step in section 3. -**Why no setting, and what changes for a deployment.** A `Location` naming the origin takes the reader off the deployment that is serving them, which no deployment asks for by naming an origin. A deployment whose origin sends no redirect to itself sees no change. For a deployment whose origin does, the reader stays on the deployment where before the reader's browser left it. Section 4 gives the two cases where the reader's browser loops instead. No setting is added or changed. `origin_host_header_override` gains a second job, being the name a redirect is compared with, and the configuration guide says so under `origin_url` and under the override. Loading the settings already refuses an override that is not a host name, an IPv4 address or a bracketed IPv6 address with an optional port, so the comparison never reads anything else. +**What changes for a deployment.** A deployment whose `origin_url` has no path sees no change. For one with a path, a URL in the folder reaches the reader without the folder, where the header rewrite alone kept it, and a URL on the origin's host outside the folder is relayed as the origin wrote it. No setting is added or changed. ## 3. The serve middleware headers step -A serve middleware may set headers on a response the publisher path sends, a redirect included, except for the responses listed under "Where the step is called" below. The redirect rule in section 2 does not use the step. A deployment's own policy for a redirect does, such as where a `Location` for another name of the site goes, or putting a removed path prefix back. +A serve middleware may set headers on a response the publisher path sends, a redirect included, except for the responses listed under "Where the step is called" below. The header rewrite in section 2 does not use the step. A deployment's own policy for a redirect does, such as where a `Location` for another name of the site goes, or putting a removed path prefix back. **What it adds.** A method on `Middleware`, with a default that returns no headers, and the read only view of the response that the method is given, both in `crates/trusted-server-core/src/middleware.rs`. A middleware written before the method compiles and behaves as before, and a request path no `[[serve]]` entry covers sees no change. @@ -111,7 +95,7 @@ pub struct ResponseView<'a> { **What a middleware may do.** Return headers. Each is written on the response in middleware order, replacing any header of the same name, so a later middleware replaces an earlier one's header of that name. The method cannot change the status or the body. A header the same middleware sets for a page through the action it returns from `create` is written later, in the page pass, and replaces the header set here. -**Where it runs.** In `handle_publisher_request`, once the origin's answer is in hand, after the redirect rule, after a file that core makes where the origin has none has replaced the origin's answer, and before core's cache rules, the modules' `finalize_response` hooks and the adapter's finalizing. A middleware therefore sees the rewritten `Location`, and a cache header it sets can be replaced by the rules that follow, such as the `private, no-store` that core gives a page it has changed for one reader. +**Where it runs.** In `handle_publisher_request`, once the origin's answer is in hand, straight after the header rewrite (section 2), which itself follows a file that core makes where the origin has none, and before core's cache rules, the modules' `finalize_response` hooks and the adapter's finalizing. A middleware therefore sees the headers as the header rewrite left them, and a header it sets is not rewritten. A cache header it sets can be replaced by the rules that follow, such as the `private, no-store` that core gives a page it has changed for one reader. **Where the step is called.** The step is called at three places in `handle_publisher_request`: @@ -127,26 +111,31 @@ The step is not called for three kinds of response: ## 4. What is not in this change -- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the one it was asked for, so the header is relayed and the reader follows it. A deployment whose origin does so names the host the origin redirects to, in `origin_url` or in `origin_host_header_override`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a serve middleware of its own, through `Middleware::response_headers` (section 3). The step is asked about a redirect with no body, though not about every response the publisher path sends. -- **An origin that redirects away from the scheme it is asked on.** The scheme of the `Location` is not compared, and the reader is sent to their own scheme. An origin that `origin_url` reaches over plain HTTP, and that answers each request with the same address on `https`, sends the reader back to the address they asked for, which gets the same answer again. `origin_url` names the scheme the origin serves its pages on. -- **An origin that redirects an index file back to its folder.** With `directory_index` set, the index file is added to every request path that ends in `/`, so a request for `/news/` is sent to the origin as `/news/index.html`. An origin that answers that request with a redirect to `/news/` sends the reader back to the address they asked for, which gets the same answer again. Before this change the reader followed the origin's own address and left the deployment, and now the reader's browser asks again each time. -- **A `Location` that names no host, from an origin in a folder.** An origin under a path in `origin_url` that answers `Location: /bucket/site/news/` has the header relayed as written, and the reader's browser then asks for a path with the folder in it. +- **Another name for the same site.** An origin at `https://publisher.example` that redirects to `https://www.publisher.example` names a host that is not the origin's, so the header rewrite relays it and the reader follows it. A deployment whose origin does so names the host the origin redirects to in `origin_url`. A deployment that serves both names, and wants the reader kept on it across that redirect, decides where such a `Location` goes with a serve middleware of its own, through `Middleware::response_headers` (section 3). The step is asked about a redirect with no body, though not about every response the publisher path sends. +- **An origin that redirects an index file back to its folder.** With `directory_index` set, the index file is added to every request path that ends in `/`, so a request for `/news/` is sent to the origin as `/bucket/site/news/index.html`. An origin that answers that request with a redirect to `/bucket/site/news/` names the address the reader asked for. Under the header rewrite's second loop exception, a `GET` or `HEAD` redirect that sets no cookie is relayed as the origin wrote it, so the reader's browser leaves the deployment for the origin's address. One that sets a cookie is mapped, so the reader's browser asks again each time. +- **A `Location` that names no host, from an origin in a folder.** An origin under a path in `origin_url` that answers `Location: /bucket/site/news/` has the header relayed as written, as the header rewrite relays any relative URL, and the reader's browser then asks for a path with the folder in it. - **Putting a removed path prefix back.** A serve middleware does so through the headers step, and no module in this repository reads the removed prefix yet. See "A publisher addressed by a path prefix" in section 2. -- **The name the page pipeline gives the origin.** With an override set, the page pipeline still moves the host of `origin_url` in a page and not the override. This change leaves the page pipeline as it was. - **Following a redirect.** The publisher proxy follows no redirect itself. The reader's browser makes the next request. -- **Any header but `Location`.** An address in `Content-Location`, `Link` or `Refresh` is relayed as before. - **Asset routes and the first-party proxy.** Their redirects are handled in `proxy.rs` and are unchanged. ## 5. Acceptance -Tests in `trusted-server-core`, where the code is, for the redirect rule: +Tests in `trusted-server-core`, where the code is, for the folder, in `response_header_rewrite.rs`: + +- A URL in the folder loses the folder and keeps its query and fragment, the folder itself with or without its closing `/` becomes `/`, a dot segment that stays in the folder is resolved, a protocol-relative URL stays protocol-relative, a folder written with a closing `/` is the same folder, and an origin path of `/` takes nothing off (`folder_origin_maps_urls_in_the_folder_without_it`). +- Another folder of the host, a folder whose name only begins with the folder's, a name that begins as the folder does but does not end at a `/`, the host's root, a URL that stops at the host, a dot segment that climbs out of the folder and a path in another letter case are each left unchanged (`folder_origin_leaves_urls_outside_the_folder_unchanged`). +- Under a folder, a scheme change to the address the reader asked for and a cookieless `GET` redirect to it are kept, and a redirect to another address in the folder is mapped (`folder_origin_keeps_redirects_to_the_current_request_that_would_loop`). +- A source with no path gains the whole serving host, a source in the folder gains a sibling without the folder, and a source outside the folder gains none (`folder_origin_csp_sources_follow_the_folder`). -- A `Location` naming the origin comes back on the request's scheme and host with its path, query and fragment, for an origin on `https` and on plain HTTP, with and without a port, named by an address in brackets, in any letter case, and with the scheme left off. -- With `origin_host_header_override` set, a `Location` naming the override is rewritten and one naming the host of `origin_url` is not. -- A `Location` naming another host, a host beneath the origin's, a host that only starts or ends with the origin's name, or the origin's name on a different port is left as written. So is a path, a query or a fragment alone, an address that is not `http` or `https`, a value that is no address, and any `Location` when the request names no host. -- With a path in `origin_url`, a `Location` at or beneath that path comes back without it, and the origin path itself comes back as `/`. One beside it, above it, in another letter case or climbing out of it is left as written, and the origin path comes off only where it ends at a `/`. -- Through `handle_publisher_request` with a scripted origin, the reader receives the rewritten `Location` on a buffered answer and on a streamed page, for a `GET`, a `HEAD`, a `POST`, a `PUT` and a `DELETE`, for `301`, `302`, `303`, `307` and `308`, and on the request's own scheme. -- Through the same handler, a redirect to another host, a redirect naming no host, a `Location` on a `200`, a `201`, a `404` and a `503`, and a redirect with no `Location` each reach the reader as the origin sent them. +Tests through `handle_publisher_request` with a scripted origin, in `publisher.rs`: + +- The reader receives the mapped `Location` on a buffered answer and on a streamed page, for a `GET`, a `HEAD`, a `POST`, a `PUT` and a `DELETE`, for `301`, `302`, `303`, `307` and `308` (`the_reader_receives_the_rewritten_location_on_every_route`), and on the request's own scheme (`the_reader_s_own_scheme_is_used`). +- With a folder in `origin_url`, the origin is asked under the folder, a redirect in the folder reaches the reader without it, and one outside it reaches the reader as written (`a_redirect_under_the_origin_path_reaches_the_reader_without_it`). +- An origin on plain HTTP or on a port is named for the reader, and the same host on another port is not (`an_origin_on_plain_http_or_on_a_port_is_named_for_the_reader`). +- Under a `Host` override, a `Location` naming the host of `origin_url` is mapped and one naming the override is not (`under_a_host_override_the_host_of_origin_url_is_the_origin_s_name`). +- A `Location` on a `200`, a `201`, a `404` and a `503` is mapped on the pass-through, streamed and buffered routes (`a_location_on_an_answer_that_is_no_redirect_is_named_for_the_reader`). +- A redirect to another host and one naming no host reach the reader as the origin wrote them, and a redirect with no `Location` gains none (`a_redirect_that_is_not_the_origin_s_own_reaches_the_reader_unchanged` and `a_redirect_with_no_location_gains_none`). +- The header rewrite's own tests pass on this branch, both its handler tests for the two loop exceptions and its tests in `response_header_rewrite.rs`. Tests in `trusted-server-core` for the headers step: @@ -160,16 +149,11 @@ Nothing tests the following: - The step on the refusal of a streamed page by a middleware's preparation, which is the third place the step is called. - That the step is not called for a refusal made before the origin is asked, for a page from the template cache, or for an answer made from an error. - Several middleware setting a header of one name on one response. -- A `Location` with bytes outside printable ASCII, a rewritten value that is not a valid header value, and several `Location` headers on one response. -- A request addressed by a path prefix, sent through the redirect rule. +- The folder on `Content-Location`, `Refresh` and `Link` through the handler. Each takes the same mapping the folder tests call directly. +- A request addressed by a path prefix, sent through the header rewrite. - Any adapter, because no adapter changes, although each already tests that its client leaves a redirect for core to see. -Sixteen deliberate faults were made, one in each run, and each was caught by a test: a `Location` rewritten on any status, the port left out of the comparison, the origin path not removed, the origin path removed where it does not end at a `/`, a `Location` outside the origin path rewritten with its path whole, the fragment dropped, the query dropped, an address that is not `http` or `https` rewritten, a request with no host rewritten, a `Location` naming no host resolved against the origin, the host of `origin_url` compared where an override is set, a `Location` with the scheme left off not read, the host compared in exact letter case, the reader always sent to `https`, the origin path not passed by the handler, and a `Location` that is the origin path itself rewritten with an empty path and not `/`. All sixteen are in the redirect rule and the handler's use of the rule, and none is recorded for the headers step. - -## Review log - -1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/37-origin-redirect` at `73801ddb7`. Issues found, three of them blocking. The design called the serve middleware headers step outside the change though the series adds it, said the step is asked about every response the publisher path sends though three kinds of response skip it, and said nothing of the reader of a publisher served under a path prefix, who is sent to an address without the prefix. This revision adds a section for the step with its tests, lists the responses the step skips, says a serve middleware and not core puts the prefix back, corrects the partly true claims, fills the gaps in coverage, takes the advice the round gave, and corrects the same sentence in the changelog and the configuration guide. -2. **Round 2, 2026-10-15.** A lighter review in a fresh session, against this page at `9dafb089e`, of the round 1 fixes and the new text. Approved, with no blocking issues. The changelog and the configuration guide now name the two loops and the `421` for a reader under a removed path prefix, and name all three kinds of response the headers step is not called for. +Five deliberate faults were made, one in each run, and each was caught by a test: the handler giving the header rewrite no folder (`a_redirect_under_the_origin_path_reaches_the_reader_without_it`), the folder counted where it does not end at a `/` (`folder_origin_leaves_urls_outside_the_folder_unchanged`), a source with no path taking the folder rule (`folder_origin_csp_sources_follow_the_folder`), the loop exceptions comparing the path with the folder still in it (`folder_origin_keeps_redirects_to_the_current_request_that_would_loop`), and the folder compared before dot segments are resolved (`folder_origin_maps_urls_in_the_folder_without_it`). ## Revision record @@ -179,3 +163,9 @@ Sixteen deliberate faults were made, one in each run, and each was caught by a t | 2026-10-14 | Names its pull request and issue, and points a deployment that serves two names of the site at the serve middleware headers step. | | 2026-10-15 | Fixes from review round 1. The headers step as a section with its tests and the responses it skips, who puts a removed path prefix back, the adapters, the setting, consent, failure and the index file loop, a table of what names the origin, the title now saying host, and the same sentence corrected in the changelog and the configuration guide. | | 2026-10-15 | Corrections from review round 2. The two loops and the `421` under a removed path prefix, and the three kinds of response the headers step is not called for, named in the changelog and the configuration guide. | +| 2026-10-19 | Follows the header rewrite merged in main (#1238) in place of a `Location` rule of its own, and gives the header rewrite the origin folder. The title, sections 1, 2, 4 and 5 and where the headers step runs change to match, and so do the changelog and the configuration guide. | + +## Review log + +1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/37-origin-redirect` at `73801ddb7`. Issues found, three of them blocking. The design called the serve middleware headers step outside the change though the series adds it, said the step is asked about every response the publisher path sends though three kinds of response skip it, and said nothing of the reader of a publisher served under a path prefix, who is sent to an address without the prefix. This revision adds a section for the step with its tests, lists the responses the step skips, says a serve middleware and not core puts the prefix back, corrects the partly true claims, fills the gaps in coverage, takes the advice the round gave, and corrects the same sentence in the changelog and the configuration guide. +2. **Round 2, 2026-10-15.** A lighter review in a fresh session, against this page at `9dafb089e`, of the round 1 fixes and the new text. Approved, with no blocking issues. The changelog and the configuration guide now name the two loops and the `421` for a reader under a removed path prefix, and name all three kinds of response the headers step is not called for. diff --git a/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md index f6409668af..8ec908b502 100644 --- a/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md +++ b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md @@ -43,7 +43,7 @@ The page that comes back is a publisher page like any other, so it is fetched, c **The origin path.** A path in `origin_url` is kept, and the request path is added after it with exactly one `/` between them, whichever side carries one. The request's query is kept unless `origin_auth` sets `origin_query = "strip"` (see "The query"). An `origin_url` with no path sends the request path as it arrived, apart from the rebuilding of a signed address (see "Signing"). A query in `origin_url` is ignored, as it was. The path is no setting, because `origin_url` is required, and before this change the path was discarded. A deployment whose existing document has a path in `origin_url` therefore sends different requests to its origin, whether or not the document sets the two settings, because each request now goes to the folder the path names and a request path with a `.` or `..` segment is answered `400`. The path is a breaking change for such a deployment. -The publisher proxy builds the origin address in one place, `handle_publisher_request` in `publisher.rs`, and that address now comes from `publisher_origin_path_and_query`. The streamed page, the buffered page, the template cache key, a `HEAD` and every other method take the address built there. The publisher path follows no redirect. A redirect the origin sends to itself has the origin path taken off its `Location`, as `2026-10-13-origin-redirect-design.md` sets out. +The publisher proxy builds the origin address in one place, `handle_publisher_request` in `publisher.rs`, and that address now comes from `publisher_origin_path_and_query`. The streamed page, the buffered page, the template cache key, a `HEAD` and every other method take the address built there. The publisher path follows no redirect. A URL in the folder in a response header, a redirect's `Location` among them, reaches the reader without the folder, and one outside the folder is left as the origin wrote it, as `2026-10-13-origin-redirect-design.md` sets out. **Signing.** `origin_auth` is the table an asset route's `auth` takes, read as the same type, with the same fields, defaults and secret store key references, and it signs through the same function, `apply_asset_origin_auth`. There is one signing path. The request is signed last, once its address and `Host` are final. Signing rebuilds the address through the URL parser, `url::Url`, signs the rebuilt address and sends the request to it, so the two cannot differ. The parser writes the address in its own form. It percent-encodes an apostrophe or a non-ASCII character in the query, and a double quote, a curly brace or a non-ASCII character in the path, all of which `http::Uri` accepts as they are, and it resolves dot segments in the path, which are refused with `400` first when `origin_url` has a path. The table still accepts the deprecated `secret_store` field that an asset route's `auth` accepts. The field is ignored with a logged warning and is never written back.