diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e2086a6a1..26cdd03768 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** a path in `publisher.origin_url` is kept, and the request path is added after it with one `/`. The path was discarded before, so a deployment whose `origin_url` has one now sends each request to the folder the path names, and answers `400` to a request path with a `.` or `..` segment, with or without `origin_auth`. A deployment whose `origin_url` has no path sends its requests as before. Before upgrading, check that the origin serves the site at that path, or remove the path. +- **Breaking:** the check that closes `/_ts/admin` and `/admin/keys` on every adapter now also answers `404` to a request path still changing after four rounds of percent-decoding and to a path that reaches either once an escape that is not valid UTF-8 is replaced. The publisher received both before, and no setting turns the check off. - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServicesBuilder::geo`, `RuntimeServices::with_geo` and `IntegrationRegistrationBuilder::with_geo_module` take an `Arc`, and `RuntimeServices::geo()`, `IntegrationRegistry::geo_module()`, the `IntegrationRegistration::geo_module` field and `build_geo_module` return or hold one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths build the call from the request. - **Breaking:** no response carries an `x-geo-*` header, `x-geo-info-available` included, unless the deployment selects `headers.geo` (`[headers] modules = ["geo"]`) and names it in a `[[serve]]` entry. That serve middleware writes the location already resolved for the request on the responses the entry covers, and no adapter calls a geo module or writes the headers when it finishes a response. For crates outside core, `GeoInfo::set_response_headers` and the `HEADER_X_GEO_*` constants are gone from core. - The Axum dev server and Spin, which have no geo lookup, refuse `[geo] module = "platform"` at startup naming the setting. The selection located no request there and still counted as a geo module, so a deployment running an Edge Cookie module was never asked to set `[geo] assume_single_jurisdiction`. The configuration and Cloudflare guides say that Cloudflare places a visitor in a region only from the `cf-region-code` request header. @@ -100,6 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- A publisher's site may be served from a folder of object storage through the publisher proxy, so its pages pass through the page pipeline, which an asset route skips. The folder is the path of `origin_url`, and two optional `[publisher]` settings do the rest, each off unless set. `origin_auth` takes the table an asset route's `auth` takes and signs every origin request with it, through the same signer. `directory_index` names a file added to a request path ending in `/`, for the origin request only. A signed origin is only read, so a method other than `GET` or `HEAD` is answered `405`, and a signed request carries none of the reader's cookies, credentials, referrer, conditional validators or range. A malformed `origin_auth` or `directory_index` is refused when the settings load, with the setting named. - `[ad-tag.google] write_loader` writes the loader tag, for a page that carries no loader of its own, through the serve middleware `ad-tag.google.loader`, which a `[[serve]]` entry names and `ts audit` writes where it finds GPT. The tag goes in the head after the bootstrap and asks for the loader on the module's own route. Only a reader whose permissions allow storage on the device (`necessary.operations.storage`) gets it, because GPT loaded from Google's standard address may use the device's storage at any time. That is TCF Purpose 1, which core's server-side auction requires where GDPR applies and which the Google tag module's loaders require. Off by default, because a page that already carries the loader would otherwise load it twice. - An asset cache module may answer for a file before core asks its origin. Core offers each `GET` for a file that the publisher's origin sends or an asset route fetches, other than one the platform's image optimizer answers, to the module that `[asset-cache]` selects, with what it knows of the reader's request and the request it would send. The module serves a response, which core handles as it would the origin's, reports that its own origin request failed, which core answers as a failed origin request, or passes the file back, and a deployment that selects none has every file asked of its origin. A registration supplies one with `with_asset_cache_module`, and the registry refuses one from a builder that `[asset-cache]` does not select, an `[asset-cache]` that selects two, and one that selects a builder that supplies none, naming that builder. The new `crates/asset-cache/51degrees`, the module `asset-cache.51degrees`, which no stock build links, holds each file as its origin sent it, under what the origin's `Vary` names and for the origin's own lifetime, in two loading caches it declares, `51degrees-assets` for the copies and `51degrees-assets-vary` for each address's record, each selecting its own loading cache module and both sized by `[cache.51degrees-assets]`. It keeps nothing that may not be shared, serves a copy with an `Age` that counts from the origin's, and on the publisher path leaves a document, and every request that carries a cookie, to the origin. A load that fails asking the origin is the answer to every request waiting on it, so a failing origin is asked once. A table that names `none` is refused, as one naming any module no linked crate offers is. - A Google tag module, `[tag] modules = ["google-tag"]` with `[tag.google-tag]`, from `crates/tag/google-tag`, which writes the consent mode defaults, Tag Manager's container script, the gtag.js snippet and the AdSense loader in the head immediately before the script bundle's tag, and Tag Manager's noscript iframe after the opening body tag, each copied from Google's documentation. Every reader gets the consent defaults, and only a reader whose permissions allow storage on the device (`necessary.operations.storage`) gets the loaders and the noscript iframe. The consent defaults come first, the four consent types Google requires for its own services default to denied, the tag waits `consent_defaults.wait_for_update` milliseconds (500 unless set) for a consent management platform's update before it fires with them, and `consent_defaults.region` scopes them to country or region codes, with every other visitor given all four denied. Where the Tag Manager module serves the same tag or container the loader names the first-party path, and a page's own loader for the configured tag is removed for every reader. diff --git a/crates/trusted-server-core/src/closed_paths.rs b/crates/trusted-server-core/src/closed_paths.rs index bac9b398c2..808883b134 100644 --- a/crates/trusted-server-core/src/closed_paths.rs +++ b/crates/trusted-server-core/src/closed_paths.rs @@ -7,8 +7,6 @@ //! every adapter asks [`closed_path_response`] before its publisher fallback //! and the request is answered `404` here. -use std::borrow::Cow; - use edgezero_core::body::Body as EdgeBody; use http::{Request, Response, StatusCode, header}; @@ -50,30 +48,28 @@ pub fn closed_path_response(req: &Request) -> Option bool { - if is_closed_as_written(path) { - return true; - } + matches_any_decoding(path, is_closed_as_written) +} +/// Returns whether `matches` holds for `path` as written or after any of +/// [`MAX_PERCENT_DECODE_ROUNDS`] rounds of percent-decoding. +/// +/// A path still changing after the last round counts as a match, so a check +/// built on this fails closed. +pub(crate) fn matches_any_decoding(path: &str, matches: impl Fn(&str) -> bool) -> bool { let mut current = path.to_owned(); - for _ in 0..MAX_PERCENT_DECODE_ROUNDS { - let Some(decoded) = percent_decoded_path(¤t) else { - return false; - }; - - // A path whose remaining `%` sequences are not decodable escapes is a - // fixed point, so further rounds would repeat the same comparison. + for _ in 0..=MAX_PERCENT_DECODE_ROUNDS { + if matches(¤t) { + return true; + } + let decoded = + String::from_utf8_lossy(&urlencoding::decode_binary(current.as_bytes())).into_owned(); if decoded == current { return false; } - - if is_closed_as_written(&decoded) { - return true; - } - current = decoded; } - - false + true } fn is_closed_as_written(path: &str) -> bool { @@ -84,16 +80,6 @@ fn is_closed_as_written(path: &str) -> bool { .is_some_and(|remainder| remainder.starts_with('/')) } -/// Percent-decodes `path` once, returning `None` when the path contains no -/// escape sequence or decodes to invalid UTF-8. -fn percent_decoded_path(path: &str) -> Option { - if !path.contains('%') { - return None; - } - - urlencoding::decode(path).ok().map(Cow::into_owned) -} - fn not_found() -> Response { Response::builder() .status(StatusCode::NOT_FOUND) @@ -216,6 +202,11 @@ mod tests { } } + #[test] + fn a_path_still_decoding_after_the_last_round_is_closed() { + assert_closed("/admin%2525252525252Fkeys/rotate"); + } + #[test] fn other_publisher_paths_are_left_to_the_fallback() { for path in [ diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index bff6d6e98e..7c8b8602a2 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -279,37 +279,32 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig { ], true, ), + ]; + // An asset route's `auth` and `[publisher]`'s `origin_auth` take the + // same table, so each holds the same three key references. + let origin_auth_keys = ["access_key_id", "secret_access_key", "session_token"]; + fields.extend(origin_auth_keys.map(|key| { field( vec![ optional_object("proxy"), optional_object("asset_routes"), SecretPathSegment::ArrayEach, optional_object("auth"), - object("access_key_id"), - ], - true, - ), - field( - vec![ - optional_object("proxy"), - optional_object("asset_routes"), - SecretPathSegment::ArrayEach, - optional_object("auth"), - object("secret_access_key"), + object(key), ], true, - ), + ) + })); + fields.extend(origin_auth_keys.map(|key| { field( vec![ - optional_object("proxy"), - optional_object("asset_routes"), - SecretPathSegment::ArrayEach, - optional_object("auth"), - object("session_token"), + object("publisher"), + optional_object("origin_auth"), + object(key), ], true, - ), - ]; + ) + })); // The settings a module declares in its own table, for core's // modules and the ones a tool registered for deploy validation. let builders = crate::integrations::all_builders(deploy_integrations()).collect::>(); @@ -551,28 +546,40 @@ fn validate_secret_key_references( } for (index, route) in settings.proxy.asset_routes.iter().enumerate() { - let Some(AssetOriginAuth::S3SigV4(auth)) = route.auth.as_ref() else { - continue; - }; - validate_secret_key_reference( - &format!("proxy.asset_routes[{index}].auth.access_key_id"), - auth.access_key_id.expose(), - )?; - validate_secret_key_reference( - &format!("proxy.asset_routes[{index}].auth.secret_access_key"), - auth.secret_access_key.expose(), - )?; - if let Some(token) = &auth.session_token { - validate_secret_key_reference( - &format!("proxy.asset_routes[{index}].auth.session_token"), - token.expose(), + if let Some(auth) = route.auth.as_ref() { + validate_origin_auth_key_references( + &format!("proxy.asset_routes[{index}].auth"), + auth, )?; } } + if let Some(auth) = settings.publisher.origin_auth.as_ref() { + validate_origin_auth_key_references("publisher.origin_auth", auth)?; + } Ok(()) } +/// Refuses an empty secret key reference in an origin auth table at `path`. +fn validate_origin_auth_key_references( + path: &str, + auth: &AssetOriginAuth, +) -> Result<(), Report> { + let AssetOriginAuth::S3SigV4(auth) = auth; + validate_secret_key_reference( + &format!("{path}.access_key_id"), + auth.access_key_id.expose(), + )?; + validate_secret_key_reference( + &format!("{path}.secret_access_key"), + auth.secret_access_key.expose(), + )?; + if let Some(token) = &auth.session_token { + validate_secret_key_reference(&format!("{path}.session_token"), token.expose())?; + } + Ok(()) +} + fn validate_secret_key_reference( path: &str, key_name: &str, @@ -1044,6 +1051,9 @@ formats = [{ width = 300, height = 250 }] true, ), ("proxy.asset_routes[*].auth.session_token".to_owned(), true), + ("publisher.origin_auth.access_key_id".to_owned(), true), + ("publisher.origin_auth.secret_access_key".to_owned(), true), + ("publisher.origin_auth.session_token".to_owned(), true), ], "should expose the native EdgeZero secret metadata contract" ); @@ -1398,6 +1408,94 @@ passphrase = "production-secret-key-32-bytes-min" ); } + /// Valid settings whose `[publisher]` signs its origin requests. + fn settings_signing_the_publisher_origin( + access_key_id: &str, + secret_access_key: &str, + session_token: Option<&str>, + ) -> Settings { + let mut settings = valid_settings(); + settings.publisher.origin_url = "https://storage.example.com/bucket/site".to_string(); + settings.publisher.origin_auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "test-region-1".to_string(), + secret_store: None, + access_key_id: Redacted::new(access_key_id.to_string()), + secret_access_key: Redacted::new(secret_access_key.to_string()), + session_token: session_token.map(|token| Redacted::new(token.to_string())), + origin_query: None, + })); + settings + } + + #[test] + fn deploy_validation_accepts_a_publisher_origin_auth_naming_its_keys() { + validate_settings_for_deploy(&settings_signing_the_publisher_origin( + "origin_access_key_id", + "origin_secret_access_key", + Some("origin_session_token"), + )) + .expect("should accept an origin_auth whose key references are named"); + } + + #[test] + fn deploy_validation_refuses_an_empty_publisher_origin_auth_key_reference() { + for (settings, path) in [ + ( + settings_signing_the_publisher_origin(" ", "origin_secret_access_key", None), + "publisher.origin_auth.access_key_id", + ), + ( + settings_signing_the_publisher_origin("origin_access_key_id", "", None), + "publisher.origin_auth.secret_access_key", + ), + ( + settings_signing_the_publisher_origin( + "origin_access_key_id", + "origin_secret_access_key", + Some(" "), + ), + "publisher.origin_auth.session_token", + ), + ] { + let err = validate_settings_for_deploy(&settings) + .expect_err("should refuse an empty origin_auth key reference"); + assert!( + format!("{err:?}").contains(&format!( + "secret key reference at `{path}` must not be empty" + )), + "should name `{path}`: {err:?}" + ); + } + } + + #[test] + fn documented_publisher_object_storage_settings_validate_when_uncommented() { + let toml = uncomment_block( + &template_with_resolved_required_secrets(), + "[publisher.origin_auth]", + ) + .replacen( + "# directory_index = \"index.html\"", + "directory_index = \"index.html\"", + 1, + ); + let settings = Settings::from_toml(&toml) + .expect("the documented origin_auth and directory_index should parse and validate"); + + assert!( + matches!( + settings.publisher.origin_auth, + Some(AssetOriginAuth::S3SigV4(_)) + ), + "the documented origin_auth should be read" + ); + assert_eq!( + settings.publisher.directory_index.as_deref(), + Some("index.html"), + "the documented directory_index should be read" + ); + } + /// A table written for a module its section does not select is refused /// when the settings load, so `ts config validate` reports it before the /// configuration reaches a deployment. diff --git a/crates/trusted-server-core/src/config_payload.rs b/crates/trusted-server-core/src/config_payload.rs index 46dad7d959..c671740469 100644 --- a/crates/trusted-server-core/src/config_payload.rs +++ b/crates/trusted-server-core/src/config_payload.rs @@ -884,6 +884,45 @@ mod tests { ); } + #[test] + fn publisher_origin_auth_resolves_its_credentials_from_the_store() { + let mut original = test_settings(); + original.publisher.origin_url = "https://storage.example.com/bucket/site".to_string(); + original.publisher.origin_auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "test-region-1".to_string(), + secret_store: None, + access_key_id: Redacted::new("s3-access-key".to_string()), + secret_access_key: Redacted::new("s3-secret-key".to_string()), + session_token: Some(Redacted::new("s3-session-key".to_string())), + origin_query: None, + })); + + let reconstructed = settings_from_config_blob( + &envelope_json(&original), + &UnifiedSecretStore, + &StoreName::from("ts_secrets"), + PolicyOrigin::Blob, + ) + .expect("should resolve the publisher origin credentials from the mapped store"); + + let Some(AssetOriginAuth::S3SigV4(auth)) = reconstructed.publisher.origin_auth.as_ref() + else { + panic!("should keep the publisher origin_auth"); + }; + assert_eq!(auth.access_key_id.expose(), "AKIAIOSFODNN7EXAMPLE"); + assert_eq!( + auth.secret_access_key.expose(), + "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY" + ); + assert_eq!( + auth.session_token + .as_ref() + .map(Redacted::expose) + .map(String::as_str), + Some("resolved-session-token") + ); + } + #[test] fn partner_without_api_token_loads_without_secret_resolution() { let mut original = test_settings(); diff --git a/crates/trusted-server-core/src/inspect/config/tests.rs b/crates/trusted-server-core/src/inspect/config/tests.rs index f92ab52a7d..e79ccc2af2 100644 --- a/crates/trusted-server-core/src/inspect/config/tests.rs +++ b/crates/trusted-server-core/src/inspect/config/tests.rs @@ -212,6 +212,13 @@ fn a_key_the_inspect_section_does_not_have_is_refused_naming_it() { fn documents_holding_every_declared_secret() -> Vec<(&'static str, Value)> { let mut main = document(""); main["publisher"]["proxy_secret"] = json!("proxy"); + main["publisher"]["origin_auth"] = json!({ + "type": "s3_sigv4", + "region": "test-region-1", + "access_key_id": "origin-access", + "secret_access_key": "origin-secret", + "session_token": "origin-session", + }); main["ec"]["hmac"]["passphrase"] = json!("hmac"); main["ec"]["partners"] = json!([{ "name": "Example Partner", diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs index 72d4f6e1da..7a5f5aa8ee 100644 --- a/crates/trusted-server-core/src/proxy.rs +++ b/crates/trusted-server-core/src/proxy.rs @@ -919,7 +919,11 @@ fn asset_origin_host_header( }) } -fn apply_asset_origin_auth( +/// Signs an origin request whose address and headers are otherwise final. +/// +/// The one signing path for origin auth, used by asset routes and by the +/// publisher origin alike. +pub(crate) fn apply_asset_origin_auth( method: &Method, target_url: &url::Url, headers: &mut http::HeaderMap, @@ -946,6 +950,17 @@ fn apply_asset_origin_auth( } } +/// Removes every header in `auth`'s signing namespace, so a header a reader +/// sent there cannot reach the origin beside the signature. +pub(crate) fn remove_origin_auth_namespace_headers( + headers: &mut http::HeaderMap, + auth: &AssetOriginAuth, +) { + match auth { + AssetOriginAuth::S3SigV4(_) => s3_sigv4::remove_namespace_headers(headers), + } +} + fn build_asset_platform_request( method: &Method, target_url: &url::Url, diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 45c483b976..d176448509 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -93,8 +93,9 @@ use crate::response_privacy::{ }; use crate::rsc_flight::RscFlightUrlRewriter; use crate::settings::{ - AUCTION_DEBUG_METADATA_ALLOWLIST, AUCTION_DEBUG_UPSTREAM_METADATA_KEYS, - AuctionDebugCommentFormat, AuctionDebugCommentOptions, AuctionDebugCommentVerbosity, Settings, + AUCTION_DEBUG_METADATA_ALLOWLIST, AUCTION_DEBUG_UPSTREAM_METADATA_KEYS, AssetOriginAuth, + AuctionDebugCommentFormat, AuctionDebugCommentOptions, AuctionDebugCommentVerbosity, Publisher, + Settings, }; use crate::streaming_processor::{ BodyStreamDecoder, BodyStreamEncoder, Compression, GzipDecodeReader, PipelineConfig, @@ -3838,6 +3839,149 @@ fn rewrite_origin_request( Ok(()) } +/// Builds the path and query of a publisher origin request. +/// +/// With none of `[publisher]`'s path settings in use this is the reader's path +/// and query exactly as sent. A path in `origin_url` comes first, joined to +/// the reader's path with one `/`. `directory_index` is added to a path that +/// ends in `/`, and the query is left off when `origin_auth` strips it. +fn publisher_origin_path_and_query<'a>( + publisher: &Publisher, + origin_path: &str, + uri: &'a Uri, +) -> Cow<'a, str> { + let path_and_query = uri + .path_and_query() + .map_or("/", http::uri::PathAndQuery::as_str); + let path = uri + .path_and_query() + .map_or("/", http::uri::PathAndQuery::path); + let folder = origin_path.trim_end_matches('/'); + let index = publisher + .directory_index + .as_deref() + .filter(|_| path.ends_with('/')); + let strip_query = publisher.origin_query_is_stripped(); + if folder.is_empty() && index.is_none() && !strip_query { + return Cow::Borrowed(path_and_query); + } + + let mut built = + String::with_capacity(folder.len() + path_and_query.len() + index.map_or(0, str::len) + 1); + if folder.is_empty() { + built.push_str(path); + } else { + built.push_str(folder); + built.push('/'); + built.push_str(path.strip_prefix('/').unwrap_or(path)); + } + if let Some(index) = index { + built.push_str(index); + } + if let Some(query) = uri.query().filter(|_| !strip_query) { + built.push('?'); + built.push_str(query); + } + Cow::Owned(built) +} + +/// Returns the status a publisher origin request is refused with before it is +/// sent, or `None` when it may be sent. +/// +/// A signed origin is only asked to read, so any method but `GET` or `HEAD` is +/// refused with `405` rather than signed with the publisher's credentials. +/// When `origin_url` names a folder, a reader path with a `.` or `..` segment +/// in any spelling is refused with `400`, so no request climbs out of it. +fn refused_origin_request( + publisher: &Publisher, + origin_path: &str, + req: &Request, +) -> Option { + if publisher.origin_auth.is_some() && !matches!(*req.method(), Method::GET | Method::HEAD) { + return Some(StatusCode::METHOD_NOT_ALLOWED); + } + if !origin_path.trim_end_matches('/').is_empty() && path_has_dot_segment(req.uri().path()) { + return Some(StatusCode::BAD_REQUEST); + } + None +} + +/// Returns `true` when `path` has a `.` or `..` segment as written or after +/// any of a bounded number of percent-decoding rounds. A backslash counts as a +/// separator, and a path still changing after the last round counts as having +/// one. +fn path_has_dot_segment(path: &str) -> bool { + crate::closed_paths::matches_any_decoding(path, |current| { + current + .split(['/', '\\']) + .any(|segment| segment == "." || segment == "..") + }) +} + +/// The plain-text answer to a publisher origin request refused before it is +/// sent. +fn origin_refusal_response( + status: StatusCode, +) -> Result, Report> { + let mut builder = Response::builder() + .status(status) + .header(header::CONTENT_TYPE, "text/plain; charset=utf-8"); + if status == StatusCode::METHOD_NOT_ALLOWED { + builder = builder.header(header::ALLOW, "GET, HEAD"); + } + let mut response = builder + .body(EdgeBody::from( + status.canonical_reason().unwrap_or_default(), + )) + .change_context(TrustedServerError::Proxy { + message: "failed to build publisher origin refusal".to_string(), + })?; + enforce_terminal_private_cache_privacy(&mut response); + Ok(response) +} + +/// Reader headers a signed publisher origin request never carries. +const READER_HEADERS_NOT_SIGNED: [header::HeaderName; 10] = [ + header::COOKIE, + header::AUTHORIZATION, + header::PROXY_AUTHORIZATION, + header::REFERER, + header::IF_NONE_MATCH, + header::IF_MODIFIED_SINCE, + header::IF_MATCH, + header::IF_UNMODIFIED_SINCE, + header::IF_RANGE, + header::RANGE, +]; + +/// Signs a publisher origin request in place with `[publisher]`'s +/// `origin_auth`, once the reader's headers a signed read must not carry are +/// removed. +/// +/// The request is sent to the address that was signed, the target as a parsed +/// URL, so the signature and the request line agree byte for byte. +fn sign_publisher_origin_request( + req: &mut Request, + auth: &AssetOriginAuth, +) -> Result<(), Report> { + for name in READER_HEADERS_NOT_SIGNED { + req.headers_mut().remove(name); + } + crate::proxy::remove_origin_auth_namespace_headers(req.headers_mut(), auth); + let target = + url::Url::parse(&req.uri().to_string()).change_context(TrustedServerError::Proxy { + message: "invalid publisher origin uri".to_string(), + })?; + *req.uri_mut() = target + .as_str() + .parse::() + .change_context(TrustedServerError::Proxy { + message: "invalid publisher origin uri".to_string(), + })?; + let method = req.method().clone(); + crate::proxy::apply_asset_origin_auth(&method, &target, req.headers_mut(), auth) +} + /// Build a minimal [`AuctionContext`] for the collect phase. /// /// See [`AuctionContext::request`]: the orchestrator's collect path runs @@ -5262,12 +5406,20 @@ pub async fn handle_publisher_request( })?; let origin_host = settings.publisher.origin_host(); let origin_host_header = settings.publisher.origin_host_header(); - let origin_path_and_query = req + if let Some(status) = refused_origin_request(&settings.publisher, parsed_origin.path(), &req) { + log::debug!("Publisher origin request refused before sending: {status}"); + return Ok(PublisherResponse::Buffered(origin_refusal_response( + status, + )?)); + } + let request_path_and_query = req .uri() .path_and_query() .map(http::uri::PathAndQuery::as_str) - .unwrap_or("/"); - let request_path_and_query = origin_path_and_query.to_string(); + .unwrap_or("/") + .to_string(); + let origin_path_and_query = + publisher_origin_path_and_query(&settings.publisher, parsed_origin.path(), req.uri()); let target_uri = format!("{origin_scheme}://{origin_host}{origin_path_and_query}") .parse::() .change_context(TrustedServerError::Proxy { @@ -5554,6 +5706,10 @@ pub async fn handle_publisher_request( .then(|| target_uri.to_string()); let reader_carries_cookie = req.headers().contains_key(header::COOKIE); rewrite_origin_request(&mut req, target_uri, &origin_host_header)?; + // Signed last, once nothing else changes the request the origin is sent. + if let Some(auth) = settings.publisher.origin_auth.as_ref() { + sign_publisher_origin_request(&mut req, auth)?; + } let request_method = req.method().clone(); let mut origin_request = Some(req); @@ -24681,6 +24837,627 @@ mod tests { ); } + /// The publisher origin request as `origin_url`'s path, `directory_index` + /// and `origin_auth` shape it, for an origin such as object storage. + mod object_storage_origin_tests { + use super::*; + use crate::platform::PlatformHttpClient; + use crate::redacted::Redacted; + use crate::s3_sigv4::{S3Credentials, sign_headers}; + use crate::settings::{OriginQueryPolicy, S3SigV4AuthConfig}; + + const STORAGE_URL: &str = "https://storage.example.com/bucket/site"; + const REGION: &str = "test-region-1"; + const ACCESS_KEY_ID: &str = "TESTACCESSKEYID"; + const SECRET_ACCESS_KEY: &str = "test-secret-access-key"; + const READER_AUTHORIZATION: &str = "Bearer reader-token"; + + fn signing_auth(origin_query: Option) -> AssetOriginAuth { + AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: REGION.to_owned(), + secret_store: None, + access_key_id: Redacted::new(ACCESS_KEY_ID.to_owned()), + secret_access_key: Redacted::new(SECRET_ACCESS_KEY.to_owned()), + session_token: None, + origin_query, + }) + } + + fn reader_request(method: Method, uri: &str) -> HttpRequest { + HttpRequest::builder() + .method(method) + .uri(uri) + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build reader request") + } + + /// A reader request carrying every header a signed read leaves off, + /// and one it keeps. + fn reader_request_with_private_headers(method: Method, uri: &str) -> HttpRequest { + HttpRequest::builder() + .method(method) + .uri(uri) + .header(header::HOST, "publisher.example") + .header(header::COOKIE, "session=reader-session") + .header(header::AUTHORIZATION, READER_AUTHORIZATION) + .header(header::PROXY_AUTHORIZATION, "Basic cmVhZGVyOnNlY3JldA==") + .header(header::REFERER, "https://elsewhere.example/page") + .header(header::IF_NONE_MATCH, "\"reader-etag\"") + .header(header::IF_MODIFIED_SINCE, "Tue, 01 Sep 2026 00:00:00 GMT") + .header(header::IF_MATCH, "\"reader-etag\"") + .header(header::IF_UNMODIFIED_SINCE, "Tue, 01 Sep 2026 00:00:00 GMT") + .header(header::IF_RANGE, "\"reader-etag\"") + .header(header::RANGE, "bytes=0-99") + .header("x-amz-meta-reader", "reader-chosen") + .header(header::USER_AGENT, "reader-agent") + .body(EdgeBody::empty()) + .expect("should build reader request") + } + + fn stub_with_responses(count: usize) -> Arc { + let stub = Arc::new(StubHttpClient::new()); + for _ in 0..count { + stub.push_response(200, b"origin response".to_vec()); + } + stub + } + + fn services_for(stub: &Arc) -> RuntimeServices { + build_services_with_http_client(Arc::clone(stub) as Arc) + } + + /// The origin addresses sent for reader requests to `uris`, in order. + async fn origin_uris(settings: &Settings, uris: &[&str]) -> Vec { + let stub = stub_with_responses(uris.len()); + let services = services_for(&stub); + for uri in uris { + let _ = run_publisher_proxy(settings, &services, reader_request(Method::GET, uri)) + .await; + } + stub.recorded_request_uris() + } + + fn sent_header<'a>(sent: &'a [(String, String)], name: &str) -> Option<&'a str> { + sent.iter() + .find(|(sent_name, _)| sent_name.eq_ignore_ascii_case(name)) + .map(|(_, value)| value.as_str()) + } + + /// Asserts the one origin request `stub` recorded is signed as `method` + /// for exactly the address and `Host` it was sent with. + fn assert_signed(stub: &StubHttpClient, method: &Method) { + let uris = stub.recorded_request_uris(); + let recorded = stub.recorded_request_headers(); + let sent = recorded.first().expect("should send one origin request"); + let host = sent_header(sent, "host").expect("should send a Host header"); + assert_eq!( + host, "storage.example.com", + "should sign for the origin host" + ); + let signed_at = sent_header(sent, "x-amz-date").expect("should send the signing time"); + assert_eq!( + sent_header(sent, "authorization"), + Some(expected_authorization(method, &uris[0], host, signed_at).as_str()), + "should sign exactly the address and host that were sent" + ); + } + + /// The `Authorization` the signer gives `method` on `uri` with `host` at + /// the moment the signed request records. + fn expected_authorization( + method: &Method, + uri: &str, + host: &str, + signed_at: &str, + ) -> String { + let signed_at = chrono::NaiveDateTime::parse_from_str(signed_at, "%Y%m%dT%H%M%SZ") + .expect("should read the signing time") + .and_utc(); + let mut headers = http::HeaderMap::new(); + headers.insert( + header::HOST, + HeaderValue::from_str(host).expect("should build host header"), + ); + sign_headers( + method, + &url::Url::parse(uri).expect("should parse sent address"), + &mut headers, + REGION, + &S3Credentials { + access_key_id: ACCESS_KEY_ID.to_owned(), + secret_access_key: Redacted::new(SECRET_ACCESS_KEY.to_owned()), + session_token: None, + }, + SystemTime::from(signed_at), + ) + .expect("should sign the expected request"); + headers + .get(header::AUTHORIZATION) + .and_then(|value| value.to_str().ok()) + .expect("should produce an Authorization header") + .to_owned() + } + + fn response_status(response: &PublisherResponse) -> StatusCode { + match response { + PublisherResponse::Buffered(response) + | PublisherResponse::PassThrough { response, .. } + | PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } => response.status(), + } + } + + #[tokio::test] + async fn origin_address_without_the_new_settings_is_unchanged() { + let settings = create_test_settings(); + + let sent = origin_uris( + &settings, + &[ + "https://publisher.example/", + "https://publisher.example/page", + "https://publisher.example/news/", + "https://publisher.example/page?a=1&b=2", + "https://publisher.example/a/../b", + "https://publisher.example//double", + ], + ) + .await; + + assert_eq!( + sent, + vec![ + "https://origin.test-publisher.com/", + "https://origin.test-publisher.com/page", + "https://origin.test-publisher.com/news/", + "https://origin.test-publisher.com/page?a=1&b=2", + "https://origin.test-publisher.com/a/../b", + "https://origin.test-publisher.com//double", + ], + "should send the reader's path and query to the origin host as before" + ); + } + + #[test] + fn origin_path_and_query_without_the_new_settings_is_the_reader_s_own() { + let publisher = Publisher::default(); + for request in [ + "/", + "/page", + "/news/", + "/page?a=1&b=2", + "/a/../b", + "//double", + "/%2e%2e/x", + "/x?", + ] { + let uri: Uri = request.parse().expect("should parse reader path"); + for origin_path in ["", "/"] { + let built = publisher_origin_path_and_query(&publisher, origin_path, &uri); + assert!( + matches!(built, Cow::Borrowed(_)), + "should reuse the reader's own path for `{request}`" + ); + assert_eq!( + built, + uri.path_and_query() + .map_or("/", http::uri::PathAndQuery::as_str), + "should send `{request}` as written" + ); + } + } + } + + #[tokio::test] + async fn origin_path_is_kept_and_joined_with_one_slash() { + let mut settings = create_test_settings(); + settings.publisher.origin_url = STORAGE_URL.to_owned(); + + let sent = origin_uris( + &settings, + &[ + "https://publisher.example/", + "https://publisher.example/page", + "https://publisher.example/news/", + "https://publisher.example/page?a=1&b=2", + ], + ) + .await; + + assert_eq!( + sent, + vec![ + "https://storage.example.com/bucket/site/", + "https://storage.example.com/bucket/site/page", + "https://storage.example.com/bucket/site/news/", + "https://storage.example.com/bucket/site/page?a=1&b=2", + ], + "should send the reader's path under the origin path, query kept" + ); + } + + #[test] + fn origin_path_join_has_one_slash_whichever_side_carries_one() { + let publisher = Publisher::default(); + for (origin_path, request, expected) in [ + ("/bucket/site", "/page", "/bucket/site/page"), + ("/bucket/site/", "/page", "/bucket/site/page"), + ("/bucket/site", "/news/", "/bucket/site/news/"), + ("/bucket/site/", "/news/", "/bucket/site/news/"), + ("/bucket/site", "/", "/bucket/site/"), + ("/bucket/site/", "/", "/bucket/site/"), + ("/bucket/site", "/page?a=1&b=2", "/bucket/site/page?a=1&b=2"), + ("/bucket/site/", "/news/?a=1", "/bucket/site/news/?a=1"), + ] { + let uri: Uri = request.parse().expect("should parse reader path"); + assert_eq!( + publisher_origin_path_and_query(&publisher, origin_path, &uri), + expected, + "should join `{origin_path}` and `{request}` with one slash" + ); + } + } + + #[test] + fn directory_index_is_added_only_to_a_path_ending_in_a_slash() { + let publisher = Publisher { + directory_index: Some("index.html".to_owned()), + ..Publisher::default() + }; + for (origin_path, request, expected) in [ + ("", "/", "/index.html"), + ("", "/news/", "/news/index.html"), + ("", "/news", "/news"), + ("", "/news/?a=1", "/news/index.html?a=1"), + ("", "/news?a=1", "/news?a=1"), + ("", "/style.css", "/style.css"), + ("/bucket/site", "/", "/bucket/site/index.html"), + ("/bucket/site", "/news/", "/bucket/site/news/index.html"), + ("/bucket/site", "/news", "/bucket/site/news"), + ] { + let uri: Uri = request.parse().expect("should parse reader path"); + assert_eq!( + publisher_origin_path_and_query(&publisher, origin_path, &uri), + expected, + "should give `{request}` under `{origin_path}` the expected origin path" + ); + } + } + + #[tokio::test] + async fn directory_index_changes_the_origin_address_and_not_the_reader_s() { + let mut settings = create_test_settings(); + settings.publisher.directory_index = Some("index.html".to_owned()); + + let sent = origin_uris( + &settings, + &[ + "https://publisher.example/", + "https://publisher.example/news/", + "https://publisher.example/news", + "https://publisher.example/style.css", + ], + ) + .await; + assert_eq!( + sent, + vec![ + "https://origin.test-publisher.com/index.html", + "https://origin.test-publisher.com/news/index.html", + "https://origin.test-publisher.com/news", + "https://origin.test-publisher.com/style.css", + ], + "should add the index file only to a path ending in a slash" + ); + + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"news".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let services = services_for(&stub); + let response = run_publisher_proxy( + &settings, + &services, + reader_request(Method::GET, "https://publisher.example/news/"), + ) + .await; + let PublisherResponse::Stream { params, .. } = response else { + panic!("should stream the origin page through the page pipeline"); + }; + assert_eq!( + params.request_path, "/news/", + "should give the page pipeline the reader's own path" + ); + } + + #[tokio::test] + async fn signed_origin_request_carries_none_of_the_reader_s_private_headers() { + let mut settings = create_test_settings(); + settings.publisher.origin_url = STORAGE_URL.to_owned(); + settings.publisher.directory_index = Some("index.html".to_owned()); + settings.publisher.origin_auth = Some(signing_auth(None)); + let stub = stub_with_responses(1); + let services = services_for(&stub); + + let _ = run_publisher_proxy( + &settings, + &services, + reader_request_with_private_headers( + Method::GET, + "https://publisher.example/news/?a=1", + ), + ) + .await; + + let uris = stub.recorded_request_uris(); + assert_eq!( + uris, + vec!["https://storage.example.com/bucket/site/news/index.html?a=1"], + "should send the signed read to the object's address" + ); + let recorded = stub.recorded_request_headers(); + let sent = recorded.first().expect("should send one origin request"); + assert_eq!( + sent_header(sent, "cookie"), + None, + "should not send the reader's cookies" + ); + assert_ne!( + sent_header(sent, "authorization"), + Some(READER_AUTHORIZATION), + "should not send the reader's Authorization" + ); + assert_eq!( + sent_header(sent, "proxy-authorization"), + None, + "should not send the reader's proxy credentials" + ); + assert_eq!( + sent_header(sent, "referer"), + None, + "should not send the reader's referrer" + ); + assert_eq!( + sent_header(sent, "if-none-match"), + None, + "should not send the reader's If-None-Match" + ); + assert_eq!( + sent_header(sent, "if-modified-since"), + None, + "should not send the reader's If-Modified-Since" + ); + assert_eq!( + sent_header(sent, "if-match"), + None, + "should not send the reader's If-Match" + ); + assert_eq!( + sent_header(sent, "if-unmodified-since"), + None, + "should not send the reader's If-Unmodified-Since" + ); + assert_eq!( + sent_header(sent, "if-range"), + None, + "should not send the reader's If-Range" + ); + assert_eq!( + sent_header(sent, "range"), + None, + "should not send the reader's Range" + ); + assert_eq!( + sent_header(sent, "x-amz-meta-reader"), + None, + "should not send a header the reader set in the signing namespace" + ); + assert_eq!( + sent_header(sent, "user-agent"), + Some("reader-agent"), + "should still send headers a signed read may carry" + ); + assert_signed(&stub, &Method::GET); + } + + #[tokio::test] + async fn signed_head_request_is_signed_as_a_head() { + let mut settings = create_test_settings(); + settings.publisher.origin_url = STORAGE_URL.to_owned(); + settings.publisher.origin_auth = Some(signing_auth(None)); + let stub = stub_with_responses(1); + let services = services_for(&stub); + + let _ = run_publisher_proxy( + &settings, + &services, + reader_request(Method::HEAD, "https://publisher.example/page"), + ) + .await; + + assert_eq!(stub.recorded_request_methods(), vec!["HEAD"]); + assert_signed(&stub, &Method::HEAD); + } + + #[tokio::test] + async fn signed_origin_strips_the_query_when_origin_auth_says_so() { + let mut settings = create_test_settings(); + settings.publisher.origin_url = STORAGE_URL.to_owned(); + settings.publisher.origin_auth = Some(signing_auth(Some(OriginQueryPolicy::Strip))); + let stub = stub_with_responses(1); + let services = services_for(&stub); + + let _ = run_publisher_proxy( + &settings, + &services, + reader_request(Method::GET, "https://publisher.example/page?utm=1"), + ) + .await; + + let uris = stub.recorded_request_uris(); + assert_eq!( + uris, + vec!["https://storage.example.com/bucket/site/page"], + "should leave the reader's query off the signed read" + ); + assert_signed(&stub, &Method::GET); + } + + #[tokio::test] + async fn signed_origin_refuses_a_method_that_is_not_a_read() { + let mut settings = create_test_settings(); + settings.publisher.origin_auth = Some(signing_auth(None)); + for method in [Method::POST, Method::PUT, Method::DELETE, Method::OPTIONS] { + let stub = stub_with_responses(1); + let services = services_for(&stub); + + let response = run_publisher_proxy( + &settings, + &services, + reader_request(method.clone(), "https://publisher.example/page"), + ) + .await; + + assert_eq!( + response_status(&response), + StatusCode::METHOD_NOT_ALLOWED, + "should refuse {method} rather than sign it" + ); + let PublisherResponse::Buffered(response) = response else { + panic!("should answer {method} without the origin"); + }; + assert_eq!( + response + .headers() + .get(header::ALLOW) + .and_then(|value| value.to_str().ok()), + Some("GET, HEAD"), + "should name the methods a signed origin answers" + ); + assert!( + stub.recorded_request_uris().is_empty(), + "should send nothing to the origin for {method}" + ); + } + } + + #[tokio::test] + async fn unsigned_origin_still_receives_the_reader_s_headers_and_methods() { + let settings = create_test_settings(); + let stub = stub_with_responses(2); + let services = services_for(&stub); + + let _ = run_publisher_proxy( + &settings, + &services, + reader_request_with_private_headers(Method::GET, "https://publisher.example/page"), + ) + .await; + let _ = run_publisher_proxy( + &settings, + &services, + reader_request(Method::POST, "https://publisher.example/form"), + ) + .await; + + let recorded = stub.recorded_request_headers(); + let sent = recorded.first().expect("should send the GET to the origin"); + assert_eq!( + sent_header(sent, "cookie"), + Some("session=reader-session"), + "should forward the reader's cookies as before" + ); + assert_eq!( + sent_header(sent, "authorization"), + Some(READER_AUTHORIZATION), + "should forward the reader's Authorization as before" + ); + assert_eq!( + sent_header(sent, "referer"), + Some("https://elsewhere.example/page"), + "should forward the reader's referrer as before" + ); + assert_eq!( + stub.recorded_request_methods(), + vec!["GET", "POST"], + "should forward a POST as before" + ); + } + + #[tokio::test] + async fn origin_path_refuses_a_reader_path_that_climbs_out_of_it() { + let mut settings = create_test_settings(); + settings.publisher.origin_url = STORAGE_URL.to_owned(); + for path in [ + "/../other-site/page", + "/news/../../other-site/page", + "/news/./page", + "/%2e%2e/other-site/page", + "/%2E%2E/other-site/page", + "/%252e%252e/other-site/page", + "/..%2fother-site/page", + "/news/..", + ] { + let stub = stub_with_responses(1); + let services = services_for(&stub); + + let response = run_publisher_proxy( + &settings, + &services, + reader_request(Method::GET, &format!("https://publisher.example{path}")), + ) + .await; + + assert_eq!( + response_status(&response), + StatusCode::BAD_REQUEST, + "should refuse `{path}`" + ); + assert!( + stub.recorded_request_uris().is_empty(), + "should send nothing to the origin for `{path}`" + ); + } + } + + #[test] + fn dot_segments_are_found_in_any_spelling() { + for path in [ + "/..", + "/.", + "/a/../b", + "/a/./b", + "/%2e%2e/b", + "/%2E%2e/b", + "/%252e%252e/b", + "/a%2f..%2fb", + "/a\\..\\b", + ] { + assert!( + path_has_dot_segment(path), + "should find a dot segment in `{path}`" + ); + } + for path in [ + "/", + "/page", + "/news/", + "/.well-known/x", + "/a..b", + "/...", + "/%2e%2ex", + ] { + assert!( + !path_has_dot_segment(path), + "should find no dot segment in `{path}`" + ); + } + } + } + #[test] fn stream_publisher_body_preserves_gzip_round_trip() { use flate2::write::GzEncoder; diff --git a/crates/trusted-server-core/src/s3_sigv4.rs b/crates/trusted-server-core/src/s3_sigv4.rs index dc4dc25ad2..633aa797a5 100644 --- a/crates/trusted-server-core/src/s3_sigv4.rs +++ b/crates/trusted-server-core/src/s3_sigv4.rs @@ -28,6 +28,24 @@ const ALGORITHM: &str = "AWS4-HMAC-SHA256"; const SERVICE: &str = "s3"; const TERMINATOR: &str = "aws4_request"; const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD"; +/// Prefix of every header name in this signing scheme's namespace. +const NAMESPACE_PREFIX: &str = "x-amz-"; + +/// Removes every header in this signing scheme's namespace. +/// +/// [`sign_headers`] replaces the few it writes. A caller that forwards a +/// reader's headers removes the rest first, so no header the reader chose can +/// change how the origin reads the signed request. +pub(crate) fn remove_namespace_headers(headers: &mut HeaderMap) { + let names: Vec<_> = headers + .keys() + .filter(|name| name.as_str().starts_with(NAMESPACE_PREFIX)) + .cloned() + .collect(); + for name in names { + headers.remove(&name); + } +} /// `AWS` credentials used to sign an `S3` request. /// diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index d1ab17f9ec..bc9cd907dc 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -91,6 +91,15 @@ pub struct Publisher { #[serde(default = "default_max_buffered_body_bytes")] #[validate(range(min = 1, message = "must be at least 1 byte"))] pub max_buffered_body_bytes: usize, + /// Optional signing of publisher origin reads, taking an asset route's `auth` table. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub origin_auth: Option, + /// Optional file name added to a request path that ends in `/` on the way + /// to the origin, for an origin such as object storage that does not answer + /// a folder with its index file. The reader's address is unchanged. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[validate(custom(function = validate_directory_index))] + pub directory_index: Option, } fn default_max_buffered_body_bytes() -> usize { @@ -111,6 +120,8 @@ impl Default for Publisher { origin_host_header_override: None, proxy_secret: Redacted::default(), max_buffered_body_bytes: default_max_buffered_body_bytes(), + origin_auth: None, + directory_index: None, } } } @@ -202,6 +213,8 @@ impl Publisher { /// origin_host_header_override: None, /// proxy_secret: Redacted::new("proxy-secret".to_string()), /// max_buffered_body_bytes: 16 * 1024 * 1024, + /// origin_auth: None, + /// directory_index: None, /// }; /// assert_eq!(publisher.origin_host(), "origin.example.com:8080"); /// ``` @@ -226,8 +239,48 @@ impl Publisher { .clone() .unwrap_or_else(|| self.origin_host()) } + + /// Returns `true` when [`Self::origin_auth`] says the reader's query is + /// left off origin requests, as an asset route's `origin_query` does. + #[must_use] + pub fn origin_query_is_stripped(&self) -> bool { + self.origin_auth + .as_ref() + .and_then(AssetOriginAuth::origin_query_policy) + == Some(OriginQueryPolicy::Strip) + } + + fn normalize(&mut self) { + if let Some(auth) = &mut self.origin_auth { + auth.normalize(); + } + } + + /// Refuses an `origin_auth` the signer cannot use, naming the setting. + /// + /// As for an asset route, a signed origin must use `https`. + fn prepare_runtime(&self) -> Result<(), Report> { + let Some(auth) = &self.origin_auth else { + return Ok(()); + }; + let uses_https = Url::parse(&self.origin_url).is_ok_and(|url| url.scheme() == "https"); + if !uses_https { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "{PUBLISHER_ORIGIN_AUTH_SETTING} requires publisher.origin_url to use https" + ), + })); + } + auth.prepare_runtime(PUBLISHER_ORIGIN_AUTH_SETTING) + } } +/// The setting named in every refusal of `[publisher]`'s `origin_auth`. +const PUBLISHER_ORIGIN_AUTH_SETTING: &str = "publisher.origin_auth"; + +/// The setting named in every refusal of an asset route's `auth`. +const ASSET_ROUTE_AUTH_SETTING: &str = "proxy.asset_routes auth"; + /// Which integrations run, and the settings each one is given. /// /// The sections of module types core does not read itself, such as `[cmp]` or @@ -1768,7 +1821,7 @@ pub enum OriginQueryPolicy { Strip, } -/// Authentication configuration for an asset origin. +/// Authentication configuration for an asset origin or the publisher origin. #[derive(Debug, Clone, Deserialize, Serialize)] #[serde(tag = "type", rename_all = "snake_case", deny_unknown_fields)] pub enum AssetOriginAuth { @@ -1784,9 +1837,10 @@ impl AssetOriginAuth { } } - fn prepare_runtime(&self) -> Result<(), Report> { + /// Refuses a table the signer cannot use, naming `setting` in the message. + fn prepare_runtime(&self, setting: &str) -> Result<(), Report> { match self { - Self::S3SigV4(config) => config.prepare_runtime(), + Self::S3SigV4(config) => config.prepare_runtime(setting), } } @@ -1852,23 +1906,24 @@ impl S3SigV4AuthConfig { }); } - fn prepare_runtime(&self) -> Result<(), Report> { + fn prepare_runtime(&self, setting: &str) -> Result<(), Report> { if self.region.is_empty() { return Err(Report::new(TrustedServerError::Configuration { - message: "proxy.asset_routes auth s3_sigv4 region must not be empty".to_string(), + message: format!("{setting} s3_sigv4 region must not be empty"), })); } if !s3_region_is_valid(&self.region) { return Err(Report::new(TrustedServerError::Configuration { - message: - "proxy.asset_routes auth s3_sigv4 region must contain only lowercase letters, digits, and '-'" - .to_string(), + message: format!( + "{setting} s3_sigv4 region must contain only lowercase letters, digits, and '-'" + ), })); } if self.access_key_id.expose().is_empty() || self.secret_access_key.expose().is_empty() { return Err(Report::new(TrustedServerError::Configuration { - message: "proxy.asset_routes auth s3_sigv4 credentials must not be empty after secret resolution" - .to_string(), + message: format!( + "{setting} s3_sigv4 credentials must not be empty after secret resolution" + ), })); } Ok(()) @@ -2636,7 +2691,7 @@ impl ProxyAssetRoute { } if let Some(auth) = &self.auth { - auth.prepare_runtime()?; + auth.prepare_runtime(ASSET_ROUTE_AUTH_SETTING)?; } if let Some(image_optimizer) = &self.image_optimizer { image_optimizer.prepare_runtime()?; @@ -4549,6 +4604,7 @@ impl Settings { /// Returns a configuration error if image optimizer keys collide after trimming. pub(crate) fn normalize_deserialized(&mut self) -> Result<(), Report> { self.cache.normalize(); + self.publisher.normalize(); self.proxy.normalize(); self.image_optimizer.normalize()?; self.debug.auction_html_comment_options.normalize(); @@ -4595,11 +4651,13 @@ impl Settings { /// /// Returns a configuration error if any cached runtime artifact cannot be /// prepared, if any handler path regex does not compile, if a creative - /// opportunity slot is invalid, or if - /// [`AuctionDebugCommentOptions::metadata_keys`] names an unsupported key. + /// opportunity slot is invalid, if `publisher.origin_auth` cannot sign, or + /// if [`AuctionDebugCommentOptions::metadata_keys`] names an unsupported + /// key. pub fn prepare_runtime(&mut self) -> Result<(), Report> { self.image_optimizer.prepare_runtime()?; self.cache.prepare_runtime()?; + self.publisher.prepare_runtime()?; self.proxy.prepare_runtime()?; if let Some(analytics) = &mut self.analytics { analytics.prepare_runtime()?; @@ -5099,6 +5157,56 @@ fn validate_host_header_override(value: &str) -> Result<(), ValidationError> { Ok(()) } +/// Longest `directory_index` accepted, the usual limit on one file name. +const MAX_DIRECTORY_INDEX_BYTES: usize = 255; + +/// Accepts a single file name that can be added to a path as written. +/// +/// The name is added to the origin request without encoding, so it is limited +/// to the characters a path segment carries unescaped. A `/` would reach into +/// another folder, `%` would let an escape stand for one, and `.` or `..` +/// would name a folder rather than a file. +fn validate_directory_index(value: &str) -> Result<(), ValidationError> { + let is_segment_byte = |byte: u8| { + byte.is_ascii_alphanumeric() + || matches!( + byte, + b'-' | b'.' + | b'_' + | b'~' + | b'!' + | b'$' + | b'&' + | b'\'' + | b'(' + | b')' + | b'*' + | b'+' + | b',' + | b';' + | b'=' + | b':' + | b'@' + ) + }; + let is_file_name = !value.is_empty() + && value.len() <= MAX_DIRECTORY_INDEX_BYTES + && value != "." + && value != ".." + && value.bytes().all(is_segment_byte); + if !is_file_name { + let mut err = ValidationError::new("invalid_directory_index"); + err.add_param("value".into(), &value); + err.message = Some( + "directory_index must be one file name such as `index.html`, without `/`, `%`, \ + spaces, a query or a fragment" + .into(), + ); + return Err(err); + } + Ok(()) +} + fn validation_error_summary(errors: &validator::ValidationErrors) -> String { fn walk(errors: &validator::ValidationErrors, prefix: &str, messages: &mut Vec) { let mut fields = errors @@ -9169,6 +9277,8 @@ source_domain = "partner.example.com" origin_host_header_override: None, proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host(), "origin.example.com:8080"); @@ -9180,6 +9290,8 @@ source_domain = "partner.example.com" origin_host_header_override: None, proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host(), "origin.example.com"); @@ -9191,6 +9303,8 @@ source_domain = "partner.example.com" origin_host_header_override: None, proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host(), "localhost:9090"); @@ -9202,6 +9316,8 @@ source_domain = "partner.example.com" origin_host_header_override: None, proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host(), "localhost:9090"); @@ -9213,6 +9329,8 @@ source_domain = "partner.example.com" origin_host_header_override: None, proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host(), "192.168.1.1:8080"); @@ -9224,6 +9342,8 @@ source_domain = "partner.example.com" origin_host_header_override: None, proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host(), "[::1]:8080"); } @@ -9237,6 +9357,8 @@ source_domain = "partner.example.com" origin_host_header_override: None, proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host_header(), "origin.example.com:8443"); @@ -9251,6 +9373,8 @@ source_domain = "partner.example.com" origin_host_header_override: Some("www.example.com".to_string()), proxy_secret: Redacted::new("test-secret".to_string()), max_buffered_body_bytes: 16 * 1024 * 1024, + origin_auth: None, + directory_index: None, }; assert_eq!(publisher.origin_host_header(), "www.example.com"); @@ -9293,6 +9417,258 @@ source_domain = "partner.example.com" ); } + /// The fixture settings with `lines` added to `[publisher]` and its + /// `origin_url` replaced by `origin_url`. + fn publisher_settings_with(origin_url: &str, lines: &str) -> String { + crate_test_settings_str() + .replacen( + "origin_url = \"https://origin.test-publisher.com\"", + &format!("origin_url = \"{origin_url}\""), + 1, + ) + .replacen( + "proxy_secret = \"unit-test-proxy-secret\"", + &format!("proxy_secret = \"unit-test-proxy-secret\"\n{lines}"), + 1, + ) + } + + const STORAGE_ORIGIN_URL: &str = "https://storage.example.com/bucket/site"; + + #[test] + fn publisher_object_storage_settings_are_off_by_default() { + let settings = + Settings::from_toml(&crate_test_settings_str()).expect("should parse test settings"); + + assert!( + settings.publisher.origin_auth.is_none(), + "origin_auth should default to unset" + ); + assert!( + settings.publisher.directory_index.is_none(), + "directory_index should default to unset" + ); + let serialized = + serde_json::to_value(&settings.publisher).expect("should serialize publisher"); + assert!( + serialized.get("origin_auth").is_none() && serialized.get("directory_index").is_none(), + "unset settings should serialize as a document written before they existed: \ + {serialized}" + ); + } + + #[test] + fn publisher_origin_auth_reads_the_asset_route_auth_table() { + let settings = Settings::from_toml(&publisher_settings_with( + STORAGE_ORIGIN_URL, + r#"origin_auth = { type = "s3_sigv4", region = "test-region-1", access_key_id = "origin_access_key_id", secret_access_key = "origin_secret_access_key", session_token = "origin_session_token", origin_query = "strip" } + directory_index = "index.html""#, + )) + .expect("should parse a publisher with origin_auth and directory_index"); + + let Some(AssetOriginAuth::S3SigV4(auth)) = settings.publisher.origin_auth.as_ref() else { + panic!("should read origin_auth as the asset route auth table"); + }; + assert_eq!(auth.region, "test-region-1"); + assert_eq!(auth.access_key_id.expose(), "origin_access_key_id"); + assert_eq!(auth.secret_access_key.expose(), "origin_secret_access_key"); + assert_eq!( + auth.session_token + .as_ref() + .map(Redacted::expose) + .map(String::as_str), + Some("origin_session_token") + ); + assert!( + settings.publisher.origin_query_is_stripped(), + "origin_query = \"strip\" should leave the query off origin requests" + ); + assert_eq!( + settings.publisher.directory_index.as_deref(), + Some("index.html") + ); + + let defaulted = Settings::from_toml(&publisher_settings_with( + STORAGE_ORIGIN_URL, + r#"origin_auth = { type = "s3_sigv4", region = "test-region-1" }"#, + )) + .expect("should parse origin_auth naming only its region"); + let Some(AssetOriginAuth::S3SigV4(auth)) = defaulted.publisher.origin_auth.as_ref() else { + panic!("should read origin_auth with defaulted key references"); + }; + assert_eq!( + ( + auth.access_key_id.expose().as_str(), + auth.secret_access_key.expose().as_str() + ), + ("access_key_id", "secret_access_key"), + "omitted key references should default as an asset route's do" + ); + assert!( + !defaulted.publisher.origin_query_is_stripped(), + "the query should be kept unless origin_query strips it" + ); + } + + #[test] + fn publisher_directory_index_accepts_a_file_name() { + for name in [ + "index.html", + "default.htm", + "index", + "index_v2.html", + "home~1.html", + ] { + let lines = format!("directory_index = {}", toml::Value::String(name.to_owned())); + let settings = Settings::from_toml(&publisher_settings_with( + "https://origin.test-publisher.com", + &lines, + )) + .unwrap_or_else(|err| panic!("should accept directory_index `{name}`: {err:?}")); + assert_eq!(settings.publisher.directory_index.as_deref(), Some(name)); + } + } + + #[test] + fn publisher_directory_index_refuses_anything_but_a_file_name() { + let too_long = "a".repeat(super::MAX_DIRECTORY_INDEX_BYTES + 1); + for name in [ + "", + ".", + "..", + "pages/index.html", + "/index.html", + "index.html/", + "index\\html", + "index.html?lang=en", + "index.html#top", + "index page.html", + " index.html", + "%2e%2e", + "index%2Ehtml", + "\u{e9}t\u{e9}.html", + too_long.as_str(), + ] { + let lines = format!("directory_index = {}", toml::Value::String(name.to_owned())); + let err = Settings::from_toml(&publisher_settings_with( + "https://origin.test-publisher.com", + &lines, + )) + .expect_err("should refuse a directory_index that is not one file name"); + assert!( + format!("{err:?}").contains("publisher.directory_index"), + "the refusal of `{name}` should name the setting: {err:?}" + ); + } + } + + #[test] + fn publisher_origin_auth_refuses_a_table_it_cannot_sign_with() { + for (origin_url, table, expected) in [ + ( + STORAGE_ORIGIN_URL, + r#"{ type = "s3_sigv4", region = "" }"#, + "publisher.origin_auth s3_sigv4 region must not be empty", + ), + ( + STORAGE_ORIGIN_URL, + r#"{ type = "s3_sigv4", region = "Test_Region" }"#, + "publisher.origin_auth s3_sigv4 region must contain only lowercase letters", + ), + ( + STORAGE_ORIGIN_URL, + r#"{ type = "s3_sigv4", region = "test-region-1", access_key_id = " " }"#, + "publisher.origin_auth s3_sigv4 credentials must not be empty", + ), + ( + STORAGE_ORIGIN_URL, + r#"{ type = "s3_sigv4", region = "test-region-1", secret_access_key = "" }"#, + "publisher.origin_auth s3_sigv4 credentials must not be empty", + ), + ( + "http://storage.example.com/bucket/site", + r#"{ type = "s3_sigv4", region = "test-region-1" }"#, + "publisher.origin_auth requires publisher.origin_url to use https", + ), + ( + STORAGE_ORIGIN_URL, + r#"{ type = "unknown_scheme", region = "test-region-1" }"#, + "origin_auth", + ), + ( + STORAGE_ORIGIN_URL, + r#"{ type = "s3_sigv4", region = "test-region-1", bucket = "site" }"#, + "origin_auth", + ), + (STORAGE_ORIGIN_URL, r#""s3_sigv4""#, "origin_auth"), + ] { + let err = Settings::from_toml(&publisher_settings_with( + origin_url, + &format!("origin_auth = {table}"), + )) + .expect_err("should refuse an origin_auth the signer cannot use"); + assert!( + format!("{err:?}").contains(expected), + "the refusal of `{table}` should say `{expected}`: {err:?}" + ); + } + } + + #[test] + fn asset_route_auth_refusals_still_name_the_asset_route() { + let mut route = ProxyAssetRoute::new("/assets/", "https://storage.example.com"); + route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: String::new(), + secret_store: None, + access_key_id: Redacted::new("access_key_id".to_owned()), + secret_access_key: Redacted::new("secret_access_key".to_owned()), + session_token: None, + origin_query: None, + })); + + let err = route + .prepare_runtime() + .expect_err("should refuse an asset route auth with no region"); + assert!( + format!("{err:?}") + .contains("proxy.asset_routes auth s3_sigv4 region must not be empty"), + "the asset route refusal should keep its wording: {err:?}" + ); + } + + #[test] + fn publisher_origin_auth_credentials_stay_out_of_debug_output() { + let canaries = [ + "canary-origin-access", + "canary-origin-secret", + "canary-origin-session", + ]; + let publisher = Publisher { + origin_auth: Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "test-region-1".to_owned(), + secret_store: None, + access_key_id: Redacted::new(canaries[0].to_owned()), + secret_access_key: Redacted::new(canaries[1].to_owned()), + session_token: Some(Redacted::new(canaries[2].to_owned())), + origin_query: None, + })), + ..Publisher::default() + }; + + let debug = format!("{publisher:?}"); + + assert!( + debug.contains("test-region-1"), + "should leave the region visible in debug output: {debug}" + ); + for canary in canaries { + assert!( + !debug.contains(canary), + "should keep `{canary}` out of debug output" + ); + } + } + #[test] fn rejects_zero_max_buffered_body_bytes() { // A zero-byte cap fails every non-empty buffered publisher response at diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 5fef48beba..d040526fb5 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -87,6 +87,7 @@ publisher, trusted-client-IP, EC, Tinybird, DataDome, and S3 fields: - `bot-protection.datadome.server_side_key_secret_name`, when protection is enabled - `bot-protection.datadome.protection_test_bypass.credential_secret_name`, when the bypass is enabled - `proxy.asset_routes[*].auth.access_key_id`, `secret_access_key`, and optional `session_token` +- `publisher.origin_auth.access_key_id`, `secret_access_key`, and optional `session_token`, when `[publisher]` sets `origin_auth` Their values belong in the logical `trusted_server_secrets` store and are resolved only while an instance builds runtime settings. An adapter can map the @@ -364,10 +365,12 @@ Core publisher settings for domain, origin, and proxy configuration. | ----------------------------- | ------- | -------- | --------------------------------------------------------------------------- | | `domain` | String | Yes | Publisher's apex domain name | | `cookie_domain` | String | Yes | Domain for non-EC cookies (typically with leading dot) | -| `origin_url` | String | Yes | Full URL of publisher origin server | +| `origin_url` | String | Yes | Publisher origin URL, whose path (if any) comes before the request path | | `origin_host_header_override` | String | No | Outbound Host header to send while connecting to `origin_url` | | `proxy_secret` | String | Yes | Secret-store key name for the proxy URL secret | | `max_buffered_body_bytes` | Integer | No | Buffered-body cap / Fastly stream raw+decoded byte ceiling (default 16 MiB) | +| `origin_auth` | Table | No | Signs every origin request with the table an asset route's `auth` takes | +| `directory_index` | String | No | File name added to an origin request path that ends in `/` | > **Note:** EC cookies (`ts-ec`) derive their domain automatically as `.{domain}` and > do not use `cookie_domain`. The `cookie_domain` field is used by other cookie helpers. @@ -444,10 +447,20 @@ TRUSTED_SERVER__PUBLISHER__MAX_BUFFERED_BODY_BYTES=16777216 - ✅ `https://origin.publisher.com` - ✅ `https://origin.publisher.com:8080` - ✅ `http://192.168.1.1:9000` +- ✅ `https://storage.publisher.com/bucket/site` (a folder of object storage) - ❌ `origin.publisher.com` (missing protocol) +- ❌ `https://origin.publisher.com/` (trailing slash) **Port Handling**: Includes port if non-standard (not 80/443). +**Path Handling**: A path in `origin_url` is kept, and the request path is +added after it with exactly one `/` between them, so a request for `/news/page` +with the folder above is sent to `/bucket/site/news/page`. The request query is +kept. A query in `origin_url` itself is ignored. With no path, the request path +is sent as it arrived, as before. When `origin_url` has a path, a request path +with a `.` or `..` segment, in any percent-encoded spelling, is refused with +`400` so no request reaches outside the folder. + #### `origin_host_header_override` **Purpose**: Optional Host header to send to the publisher origin while still @@ -532,6 +545,77 @@ a zero-byte cap fails every non-empty publisher response. TRUSTED_SERVER__PUBLISHER__MAX_BUFFERED_BODY_BYTES=16777216 ``` +#### `origin_auth` + +**Purpose**: Sign every publisher origin request, for an origin such as +private object storage that answers only signed reads. Pages still pass through +the page pipeline, which an [asset route](#proxy-asset-routes) skips. + +**Format**: The same table as +[`[proxy.asset_routes.auth]`](#proxy-asset-routes-auth), with the same fields, +defaults and secret-store key references, signed by the same signer. + +```toml +[publisher] +domain = "publisher.com" +cookie_domain = ".publisher.com" +origin_url = "https://storage.publisher.com/bucket/site" +proxy_secret = "publisher_proxy_secret" +directory_index = "index.html" + +[publisher.origin_auth] +type = "s3_sigv4" +region = "example-region-1" +access_key_id = "publisher_origin_access_key_id" +secret_access_key = "publisher_origin_secret_access_key" +# session_token = "publisher_origin_session_token" +# origin_query = "strip" +``` + +**Behavior**: + +- Only `GET` and `HEAD` are signed. Any other method is answered + `405 Method Not Allowed` with `Allow: GET, HEAD` and never reaches the origin, + so the publisher's credentials are only ever used to read. +- A signed request carries none of the reader's `Cookie`, `Authorization`, + `Proxy-Authorization`, `Referer`, `If-None-Match`, `If-Modified-Since`, + `If-Match`, `If-Unmodified-Since`, `If-Range` or `Range` headers, and none in + the signing scheme's own header namespace. +- `origin_query = "strip"` leaves the reader's query off the origin request. + By default the query is kept. +- `origin_url` must use `https`. A table the signer cannot use, such as an + empty or malformed `region`, is refused at startup naming + `publisher.origin_auth`. + +**Default**: Unset, and origin requests are sent unsigned with the reader's +headers, as before. + +#### `directory_index` + +**Purpose**: A file name added to the origin request path when the request +path ends in `/`, for an origin such as object storage that does not answer a +folder with its index file. + +**Usage**: + +- `/` is sent to the origin as `/index.html`, and `/news/` as + `/news/index.html`. +- `/news` and `/style.css` are sent as they are. +- Only the origin request changes. The reader's address, and the path the page + pipeline and its middleware see, are unchanged. + +**Format**: One file name, made of letters, digits and `-._~!$&'()*+,;=:@`, +with no `/`, `%`, space, query or fragment, and neither `.` nor `..`. Anything +else is refused at startup naming `publisher.directory_index`. + +**Default**: Unset, and every path is sent as it arrived. + +**Environment Override**: + +```bash +TRUSTED_SERVER__PUBLISHER__DIRECTORY_INDEX=index.html +``` + ## Trusted Client IP Configuration Use this optional section when a trusted CDN service forwards requests to the diff --git a/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md new file mode 100644 index 0000000000..f48bca4dba --- /dev/null +++ b/docs/superpowers/specs/2026-10-13-publisher-object-storage-origin-design.md @@ -0,0 +1,152 @@ +# Design Spec: A publisher's site served from a folder of object storage + +**Status:** Proposed, 2026-10-13. Raised as draft pull request IABTechLab/trusted-server#1377 (from jwrosewell/trusted-server#31, which is closed), for Tech Lab review. +**Author:** 51Degrees (contributed), for Tech Lab review +**Related specs:** `2026-10-12-page-middleware-design.md`, `2026-10-12-middleware-creates-a-missing-file-design.md`, `2026-04-28-multi-backend-asset-proxy-design.md` and `2026-10-14-origin-signing-design.md`, with every design document of the stack held in IABTechLab/trusted-server#1342 +**Related PRs:** IABTechLab/trusted-server#1377, jwrosewell/trusted-server#31 (closed) +**Issues:** IABTechLab/trusted-server#1329 +**Last updated:** 2026-10-16 + +## 1. Problem + +A publisher whose site is static files in object storage can only be served by an asset route, and an asset route skips the page pipeline, so none of the publisher's `[[fetch]]` or `[[serve]]` middleware runs on its pages. The publisher proxy does run the pipeline, and three things keep it from reaching such a site. It sends the request path to the origin's host and discards any path in `origin_url`, so a site kept in a folder of a bucket has no address. It sends its requests unsigned, and only an asset route can sign. And object storage answers `folder/` with nothing, where a web server answers with `folder/index.html`. + +## 2. Design + +Two optional settings in `[publisher]`, `origin_auth` and `directory_index`, and a new meaning for the path of `origin_url`, which is required and so is no setting. Each setting is off unless set, and a document that sets neither serializes to the same bytes as before, so a build that predates them still reads that document. A document that sets one is refused by such a build, because `[publisher]` refuses a field it does not know, so the build is deployed before the document is pushed. Two changes also reach a deployment that sets neither setting. A path already in `origin_url` now decides where every request goes, and the check that answers `404` to the administration paths now closes more paths on every adapter. Each change is breaking for the requests it reaches (see "Rollout"). + +```toml +[publisher] +domain = "publisher.example" +cookie_domain = ".publisher.example" +origin_url = "https://storage.example/bucket/site" +proxy_secret = "publisher_proxy_secret" +directory_index = "index.html" + +[publisher.origin_auth] +type = "s3_sigv4" +region = "example-region-1" +access_key_id = "publisher_origin_access_key_id" +secret_access_key = "publisher_origin_secret_access_key" +``` + +| A request for | Is sent to the origin as | +| -------------------------------------------- | ------------------------------------------------ | +| `/` | `/bucket/site/index.html`, signed | +| `/news/` | `/bucket/site/news/index.html`, signed | +| `/news` | `/bucket/site/news`, signed | +| `/style.css?v=2` | `/bucket/site/style.css?v=2`, signed | +| `/news/../../other-site/page` | Not sent. Answered `400` | +| A `POST`, or any method but `GET` and `HEAD` | Not sent. Answered `405` with `Allow: GET, HEAD` | + +The page that comes back is a publisher page like any other, so it is fetched, changed by the middleware named for it, stored and served as one from a web server is. + +**The origin path.** A path in `origin_url` is kept, and the request path is added after it with exactly one `/` between them, whichever side carries one. The request's query is kept unless `origin_auth` sets `origin_query = "strip"` (see "The query"). An `origin_url` with no path sends the request path as it arrived, apart from the rebuilding of a signed address (see "Signing"). A query in `origin_url` is ignored, as it was. The path is no setting, because `origin_url` is required, and before this change the path was discarded. A deployment whose existing document has a path in `origin_url` therefore sends different requests to its origin, whether or not the document sets the two settings, because each request now goes to the folder the path names and a request path with a `.` or `..` segment is answered `400`. The path is a breaking change for such a deployment. + +The publisher proxy builds the origin address in one place, `handle_publisher_request` in `publisher.rs`, and that address now comes from `publisher_origin_path_and_query`. The streamed page, the buffered page, the template cache key, a `HEAD` and every other method take the address built there. The publisher path follows no redirect and compares no `Location` against the origin, so there was nothing of that kind to change. + +**Signing.** `origin_auth` is the table an asset route's `auth` takes, read as the same type, with the same fields, defaults and secret store key references, and it signs through the same function, `apply_asset_origin_auth`. There is one signing path. The request is signed last, once its address and `Host` are final. Signing rebuilds the address through the URL parser, `url::Url`, signs the rebuilt address and sends the request to it, so the two cannot differ. The parser writes the address in its own form. It percent-encodes an apostrophe or a non-ASCII character in the query, and a double quote, a curly brace or a non-ASCII character in the path, all of which `http::Uri` accepts as they are, and it resolves dot segments in the path, which are refused with `400` first when `origin_url` has a path. The table still accepts the deprecated `secret_store` field that an asset route's `auth` accepts. The field is ignored with a logged warning and is never written back. + +**The query.** By default a signed read passes the reader's query to storage and signs it, in the parser's form described under "Signing". AWS's documentation of the S3 `GetObject` request lists query parameters that select a version of an object or set headers of the answer, `versionId` and `response-content-type` among them. A reader's query reaches storage with the publisher's signature on it. No test here calls S3, so what S3 does with those parameters rests on AWS's documentation. `origin_query` is a field of the `origin_auth` table that takes `preserve`, the default, or `strip`. The value `strip` leaves the query off before the request is signed, and a site that has no use for a query string sets `strip`. The cost is that `/style.css?v=2` and `/style.css?v=3` are sent to storage as the same address, so a version number in the query no longer changes the request. + +**The Host that is signed.** The signer reads the `Host` header of the request, which is `origin_host_header_override` when that setting is set and the host of `origin_url` otherwise, with its port when the port is not the default. The signer lists `host` among the headers it signs, so the signature belongs to that value. Every adapter but Spin passes that value on (see "The adapters"). + +What keeps a signed read safe: + +| The risk | What prevents it | +| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The reader's credentials reach the storage service | `Cookie`, `Authorization` and `Proxy-Authorization` are removed before signing | +| The reader's browsing history reaches the storage service | `Referer` is removed | +| A reader's cache decides what the publisher's credentials ask for | `If-None-Match`, `If-Modified-Since`, `If-Match`, `If-Unmodified-Since`, `If-Range` and `Range` are removed, so every signed read asks for the whole object | +| The reader sets a header the signing scheme reads | Every header in the scheme's own namespace is removed, and the signer writes its own | +| The publisher's credentials are used to write or delete | Only `GET` and `HEAD` are signed. Any other method is answered `405` and nothing is sent | +| A reader climbs out of the folder `origin_url` names | With a path in `origin_url`, a request path with a `.` or `..` segment, as written, after up to four rounds of percent-decoding, or while still changing after the fourth round, is answered `400` and nothing is sent | +| The reader's query reaches storage with the publisher's signature on it | Nothing by default, because the query is kept and signed. `origin_query = "strip"` leaves the query off | +| The signed request, with any session token, travels in the clear | `origin_url` must use `https` when `origin_auth` is set | + +An origin with no `origin_auth` is sent the reader's headers and methods as before. The `400` for a dot segment does not depend on `origin_auth`. Whenever `origin_url` has a path, an unsigned origin answers `400` for every method, and a signed origin answers `400` for `GET` and `HEAD`, because any other method is answered `405` first. + +**One decoding function for two checks.** The dot segment check and the closed path check in `closed_paths.rs` share one function, `matches_any_decoding`. The function tests a path as written and after each of four rounds of percent-decoding, which turns `%2e` into `.`, and counts a path that is still changing after the fourth round as a match, so each check fails closed. A match is a `400` for the dot segment check and a `404` for the closed path check, and every adapter runs the closed path check on every deployment (see "Rollout"). + +**What storage is sent, and what the reader gets back.** A signed read leaves off the ten reader headers the table above lists and every `x-amz-` header. The other headers of the request are sent, with `Host` set to the origin's and `Accept-Encoding` limited to what the page pipeline can decode, as for any origin. `User-Agent` therefore stays, and so does any client address header still on the request, such as `X-Forwarded-For`, because the list names none. What storage sends back is handled as any origin's answer is. Core has no rule that removes a storage service's response headers, so they reach the reader as any origin's do. AWS's documentation lists `x-amz-request-id` and `x-amz-id-2` among the headers S3 adds to a response, and no test here sends a storage response that carries them. This series changes no consent code, so consent is handled as before. + +**When a read is refused.** The `405` and the `400` are plain text answers with `Cache-Control: no-store, private`, and the `405` carries `Allow: GET, HEAD`. Nothing is sent to storage for either. A read that storage refuses reaches the reader with the status, headers and body storage sent. A body that is not text, JavaScript or JSON is returned unchanged, and a text body goes through the page pipeline like any other error page the origin sends. AWS's documentation describes an error response from S3 as an XML document. No test here sends an S3 error response, so what a reader sees for one follows from the rule above and has not been tested. A `404` from storage can be made into a file as for any origin, and a `403` is not (see section 3). If signing fails, for example because the session token holds a character a header cannot carry, nothing is sent and the failure leaves `handle_publisher_request` as an error. The adapter answers that error as it answers any failed request, with `400` when the signer cannot write a header value and `502` for the other failures. + +**The index file.** `directory_index` names one file. A request path ending in `/`, the root among them, has the name added for the origin request only, ahead of the query. A path without the trailing `/` is sent as asked, so `/news` is not guessed to be a folder. The reader's address does not change, and neither does the path the middleware and the creative opportunity rules see. The key is written in the document, because an environment override such as `TRUSTED_SERVER__PUBLISHER__DIRECTORY_INDEX` replaces a key the document already has and is ignored when the key is missing (see "Environment Variable Overrides (Typed CLI)" in `docs/guide/configuration.md`). + +**What is refused when the settings load.** + +- An `origin_auth` with an empty or malformed `region`, with credentials that are empty once resolved, or beside an `origin_url` that is not `https`. The message names `publisher.origin_auth`. +- An `origin_auth` with a `type` or a field its table does not know, or that is not a table. The parser refuses the document with its own message, to which this change adds nothing. +- An `origin_auth` with an empty secret key reference, by `ts config validate` and `ts config push`, naming the reference, such as `publisher.origin_auth.access_key_id`. +- A `directory_index` that is not one file name. A name may hold only ASCII letters and digits and the characters `-._~!$&'()*+,;=:@`, may not be `.` or `..`, and may not be longer than 255 bytes. An empty name, or one with a `/`, a `%`, a space, a query, a fragment, a backslash or a letter outside ASCII, is refused. The message names `publisher.directory_index`. + +An asset route's `auth` is refused in the words it was, naming the asset route. + +**The adapters.** All four adapters reach the origin through `handle_publisher_request` in core, and no adapter file changed, so the two settings and the path in `origin_url` work the same way on each. The adapters differ in what they do with the `Host` header of the outbound request. The Fastly adapter sets the `Host` it sends through the backend it registers for the origin, from `origin_host_header_override` when that is set. The Axum and Cloudflare adapters copy the request's headers, `Host` among them, onto the outbound request. The Spin adapter drops `Host`, because WASI HTTP forbids it on an outbound request, and takes the host from the request address. On Spin an `origin_host_header_override` that differs from the host of `origin_url` therefore makes the signature cover one `Host` while the request carries another, so a Spin deployment that signs sets no override. + +**Rollout.** Two changes reach a deployment that sets neither `origin_auth` nor `directory_index`, and each is breaking for the requests it reaches. + +1. A path in `origin_url` was discarded before and is now kept. A deployment whose `origin_url` has one sends each request to the folder the path names, and answers `400` to a request path with a `.` or `..` segment, with or without `origin_auth`. A deployment whose `origin_url` has no path sends its requests as before. +2. The closed path check in `closed_paths.rs` closes `/_ts/admin` and `/admin/keys`, with the paths beneath them, by answering `404`. All four adapters run it before their publisher fallback, so no setting turns it off and no adapter file changed. A request path still changing after four rounds of percent-decoding is now answered `404`, where the publisher received it before. A path with an escape that is not valid UTF-8 is now read with the bad bytes replaced and checked again, so it is answered `404` when it reaches a closed path, where the check stopped at that escape and the publisher received the request before. + +The build is deployed before a document that sets `origin_auth` or `directory_index` is pushed, because a build that predates them refuses the document. A deployment whose `origin_url` already has a path first checks that its origin serves the site at that path, or removes the path. Rolling back is the reverse. A build that predates this change refuses a document that sets either setting, so the document is changed first, and that build also discards the path in `origin_url` and sends unsigned requests, so it cannot serve a site kept in a folder of storage. + +**Why the publisher proxy and not a route of its own.** The page pipeline, the template cache, the making of a file the origin does not have and the cache policy for a path all hang off the publisher path. A second path for a site kept in storage would carry each of them again, and a page would behave differently according to where its bytes are kept. + +**Why the settings and not a module.** Each change acts on the request that core already makes to the one origin `[publisher]` names, and the signer is the one core already has for asset routes. The path in `origin_url` and `directory_index` name no vendor. `origin_auth` does, because AWS signing is a vendor's and does not belong in core. Core already kept AWS signing for asset routes. This series adds the `origin_auth` setting to `[publisher]` and, in `s3_sigv4.rs`, the `x-amz-` header rule that removes a reader's headers from the signing namespace. A later series, whose design is `2026-10-14-origin-signing-design.md` (IABTechLab/trusted-server#1391), moves the signer, its settings type and that header rule out of core into a shared crate. + +## 3. What is not in this change + +- **Validators and ranges on a signed origin.** A browser revalidating a file through a signed publisher origin is sent the file again, and a range request is answered with the whole file. An asset route forwards both, and stays the place for large media. +- **A missing object read as a missing file.** A file the origin does not have is made when the origin answers `404`. Some object storage answers a missing object with `403`, and with `404` only where the credentials may also list the bucket. No storage service was called in testing this change, so a deployment that wants a file made checks what its storage answers for an object it does not have. +- **Addresses in a page that name the storage host.** A site that links to itself by relative or root-relative addresses is unaffected. An absolute address naming the storage host is rewritten to the reader's host as before. In a style sheet or a script the folder is removed with it, because the whole of `origin_url` is matched there. In an HTML document only the host is matched, so the folder stays in the path, and the address then names nothing beneath the folder. +- **Media types.** The page pipeline runs on what the origin says a file is, so each object is stored with its media type. +- **A trailing slash on `origin_url`** is refused, as it was. The join gives one `/` whether or not the path ends in one. +- **A second signing scheme.** `origin_auth` takes the one type an asset route takes. + +## 4. Acceptance + +Tests in `trusted-server-core`, where the code is: + +- A publisher with neither `origin_auth` nor `directory_index`, and an `origin_url` with no path, sends each origin address it sent before, a path with a dot segment and a doubled slash among them, with the reader's cookies, `Authorization` and referrer, and a `POST` as a `POST`. +- A path in `origin_url` is kept and joined with one `/`, with and without a trailing `/` on either side, and the query is kept. +- `directory_index` is added to `/` and to a path ending in `/`, and to no other path, and the page pipeline is given the reader's own path. +- A signed request carries the signature the signer gives for the address, `Host` and method that were sent, and none of the reader's headers the table above lists, with one assertion for each. A `HEAD` is signed as a `HEAD`, and `origin_query = "strip"` leaves the query off before signing. +- A signed origin answers `POST`, `PUT`, `DELETE` and `OPTIONS` with `405` and `Allow: GET, HEAD`, and a request path with a dot segment under a path in `origin_url` is answered `400` in each of eight spellings, and neither is sent. +- A path still changing after the last round of percent-decoding is closed by the closed path check in `crates/trusted-server-core/src/closed_paths.rs`. +- Each malformed `origin_auth` and `directory_index` is refused when the settings load, with the setting named, and the documented example in `trusted-server.example.toml` loads when uncommented. +- The credentials of `origin_auth` are resolved from the secret store as an asset route's are, masked in the configuration view, never served by it, and kept out of debug output. + +The tests of the publisher path are in the `object_storage_origin_tests` module of `crates/trusted-server-core/src/publisher.rs`, and each behavior in the table is asserted by the test beside it. + +| Behavior | Test | +| ----------------------------------------------- | -------------------------------------------------------------------- | +| The reader's headers are left off a signed read | `signed_origin_request_carries_none_of_the_reader_s_private_headers` | +| Headers in the signing namespace are left off | `signed_origin_request_carries_none_of_the_reader_s_private_headers` | +| A `HEAD` is signed as a `HEAD` | `signed_head_request_is_signed_as_a_head` | +| The path of `origin_url` is kept | `origin_path_is_kept_and_joined_with_one_slash` | +| `directory_index` is added | `directory_index_changes_the_origin_address_and_not_the_reader_s` | +| A method that is not a read is answered `405` | `signed_origin_refuses_a_method_that_is_not_a_read` | +| A dot segment is answered `400` | `origin_path_refuses_a_reader_path_that_climbs_out_of_it` | + +**Tests that are missing.** Four gaps have no test. + +1. No test sends a page from a signed folder origin through `[[fetch]]`, `[[serve]]` or the template cache. The signed tests answer with a plain `200` and check the request sent to storage, and the test that streams a page through the pipeline sets `directory_index` and no `origin_auth`. +2. The signed tests set `origin_auth` on the settings struct, so none loads a document that carries `[publisher.origin_auth]` and then sends a request through it. +3. The refusals of a `type` or a field the table does not know, and of an `origin_auth` that is not a table, are checked only for the text `origin_auth`, which the parser's own echo of the document line satisfies, so no test checks the full name `publisher.origin_auth` for them. +4. No test combines a dot segment with `origin_auth`, sets `origin_host_header_override` on a signed origin, gives the closed path check an escape that is not valid UTF-8, or sends a signed read whose query or path holds a character the URL parser encodes. + +## Review log + +1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/36-object-storage` at `f5f8ac2d0`. Issues found, four of them blocking. Section 2 counted the path of `origin_url` as an optional setting that is off unless set, when it changes the requests of every deployment whose document has one, left out that the closed path check now answers `404` to a path still changing after four rounds of percent-decoding on every deployment, said the query is always kept when `origin_query = "strip"` leaves it off, and said no setting is a vendor's when AWS signing is. This revision states both changes as breaking with the upgrade and rollback order, puts `origin_query` in the design, says that this series adds the AWS signing setting where core already kept AWS signing and that a later series moves it, and adds the adapters, the `Host` that is signed, what a signed read sends, what a reader sees when a read is refused, the exact rule for `directory_index`, the tests that are missing and the pull request that holds the stack's design documents. +2. **Round 2, 2026-10-15.** A lighter review in a fresh session, against this page at `1465a29d1`, of the round 1 fixes and the new text. One blocking issue, in the changelog, and three smaller ones. This revision moves the path in `origin_url` and the closed path change into the Changed section of the changelog as breaking entries and corrects the changelog entry's count of settings, says that this series adds the `x-amz-` header rule to core, says that signing rebuilds the address through the URL parser and that a signed origin answers a method other than `GET` or `HEAD` with `405` before the dot segment check, and attributes what the page says about S3 to AWS's documentation. + +## Revision record + +| Date | Change | +| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-10-13 | First draft. | +| 2026-10-14 | Names its draft pull request and issue. | +| 2026-10-15 | Fixes from review round 1. The path of `origin_url` and the closed path check as breaking changes with the upgrade and rollback order, `origin_query`, the vendor claim, the adapters and the `Host` that is signed, what a signed read sends and what a reader sees on a refusal, the rule for `directory_index`, the missing tests and the header. | +| 2026-10-15 | Corrections from review round 2. The changelog entry, the `x-amz-` rule this series adds, the rebuilt address, the methods the `400` applies to, and the statements about S3 attributed to AWS's documentation. | diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 12e852c136..98edaaf1c2 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -57,7 +57,9 @@ domain = "example.com" # Cookie scope for first-party identity cookies (leading dot = include subdomains). cookie_domain = ".example.com" -# Upstream origin to proxy publisher content from. No trailing slash. +# Upstream origin to proxy publisher content from. No trailing slash. A path is +# kept and the request path is added after it, so a site held in a folder of +# object storage is "https://storage.example.com/bucket/site". origin_url = "https://origin.example.com" # Secret Store key for the HMAC secret used to sign first-party proxy URLs. proxy_secret = "publisher_proxy_secret" @@ -67,6 +69,20 @@ proxy_secret = "publisher_proxy_secret" # rewriting/injection) instead of streamed. Default 16 MiB; larger responses # return 502. Raise for deployments serving very large publisher pages. # max_buffered_body_bytes = 16777216 # 16 MiB +# Optional: file name added to a request path ending in "/" on the way to the +# origin, for an origin such as object storage that does not answer a folder +# with its index file. The reader's address is unchanged. +# directory_index = "index.html" +# Optional: sign every origin request, for an origin such as private object +# storage. The same table as [proxy.asset_routes.auth]. Only GET and HEAD are +# signed, other methods are refused, and the reader's cookies, Authorization, +# Referer, conditional validators and Range are not sent. origin_url must use +# https. Keep this table last in [publisher]. +# [publisher.origin_auth] +# type = "s3_sigv4" +# region = "example-region-1" +# access_key_id = "publisher_origin_access_key_id" +# secret_access_key = "publisher_origin_secret_access_key" # -----------------------------------------------------------------------------