From 67533606e783b28d0f30636e9b018e0e584c9f47 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 14 Oct 2026 20:46:38 +0100 Subject: [PATCH 01/17] A source module says where the publisher path and asset routes fetch from Where content comes from was written twice, in [publisher] and in every asset route, and fetched by code that knew one kind of source. A source module now answers it. [source] selects the publisher's, and each [[proxy.asset_routes]] entry selects its own in a source table of the same shape. Core supplies origin, an HTTP origin, and a module a build links offers another through with_source_module. For each request a source answers a SourceLocation, being the address and folder, the Host to send, the signing and the index file. The publisher path and the asset routes fetch, cache, rewrite the origin host and follow the redirect rule from that answer, so a source that answers a different origin for each request is kept apart everywhere. The origin fields move into [source.origin] and [proxy.asset_routes.source.origin]. A document writing one where it was is refused naming the new place, and a document with no [source] is refused. A path in an asset route's url names a folder the target path is added under, a target path with a dot segment under a folder is answered 400, and a route applies directory_index as the publisher does. Stored page and asset keys are unchanged for an existing deployment. The configuration page masks the origin's url and host_header by default, and the signing secrets are declared at their new paths. Design: docs/superpowers/specs/2026-10-14-source-modules-design.md --- .../setup-integration-test-env/action.yml | 6 +- .github/workflows/test.yml | 2 +- CHANGELOG.md | 11 + crates/auction/prebid-server/src/lib.rs | 7 +- crates/auction/prebid/src/lib.rs | 7 +- crates/tag/google-tag-manager/src/lib.rs | 14 +- crates/testing/seam-probe/src/lib.rs | 14 +- crates/trusted-server-adapter-axum/src/app.rs | 14 +- .../src/middleware.rs | 7 +- .../tests/routes.rs | 14 +- .../tests/seam_probe.rs | 14 +- .../src/app.rs | 35 +- .../src/middleware.rs | 7 +- .../tests/routes.rs | 28 +- .../trusted-server-adapter-fastly/src/app.rs | 64 +- .../src/app/seam_probe_tests.rs | 7 +- .../trusted-server-adapter-fastly/src/lib.rs | 7 +- .../src/middleware.rs | 7 +- .../src/sandbox.rs | 7 +- crates/trusted-server-adapter-spin/src/app.rs | 21 +- .../src/middleware.rs | 7 +- .../tests/routes.rs | 21 +- .../src/commands/audit/generate/mod.rs | 11 +- .../tests/config_store_defaults.rs | 2 +- crates/trusted-server-core/src/config.rs | 160 ++- .../trusted-server-core/src/config_payload.rs | 68 +- .../trusted-server-core/src/html_processor.rs | 7 +- .../trusted-server-core/src/inspect/config.rs | 6 +- .../src/inspect/config/tests.rs | 99 +- .../src/integrations/js_asset_proxy.rs | 7 +- .../src/integrations/registry.rs | 154 +++ crates/trusted-server-core/src/lib.rs | 1 + .../trusted-server-core/src/provider_table.rs | 12 + crates/trusted-server-core/src/proxy.rs | 554 ++++++-- crates/trusted-server-core/src/publisher.rs | 356 +++-- .../src/response_privacy.rs | 7 +- crates/trusted-server-core/src/settings.rs | 1174 +++++++++-------- crates/trusted-server-core/src/source.rs | 671 ++++++++++ .../trusted-server-core/src/test_support.rs | 62 +- .../configs/trusted-server.integration.toml | 7 +- .../src/bin/generate-viceroy-config.rs | 12 +- .../tests/common/config.rs | 7 +- .../tests/common/runtime.rs | 2 +- .../tests/parity.rs | 7 +- docs/guide/api-reference.md | 9 +- docs/guide/asset-routes.md | 50 +- docs/guide/configuration-rules.md | 26 +- docs/guide/configuration.md | 425 +++--- docs/guide/error-reference.md | 29 +- docs/guide/first-party-proxy.md | 14 +- .../specs/2026-10-14-source-modules-design.md | 87 ++ scripts/attestation-local-test.sh | 2 +- scripts/template-cache-local-test.sh | 4 +- trusted-server.example.toml | 50 +- 54 files changed, 3224 insertions(+), 1179 deletions(-) create mode 100644 crates/trusted-server-core/src/source.rs create mode 100644 docs/superpowers/specs/2026-10-14-source-modules-design.md diff --git a/.github/actions/setup-integration-test-env/action.yml b/.github/actions/setup-integration-test-env/action.yml index 841d8d5bd9..34d7d5b62d 100644 --- a/.github/actions/setup-integration-test-env/action.yml +++ b/.github/actions/setup-integration-test-env/action.yml @@ -75,7 +75,7 @@ runs: if: ${{ inputs.build-wasm == 'true' }} shell: bash env: - TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:${{ inputs.origin-port }} + TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:${{ inputs.origin-port }} TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32 TRUSTED_SERVER__EC__PARTNERS: >- @@ -87,7 +87,7 @@ runs: if: ${{ inputs.build-axum == 'true' }} shell: bash env: - TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:${{ inputs.origin-port }} + TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:${{ inputs.origin-port }} TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32 TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK: "false" @@ -118,7 +118,7 @@ runs: if: ${{ inputs.build-cloudflare == 'true' }} shell: bash env: - TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:${{ inputs.origin-port }} + TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:${{ inputs.origin-port }} TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32 TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK: "false" diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 37d0d2148f..c73744f8fd 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -214,7 +214,7 @@ jobs: # rejects (falling back to the generic 503 router). This proves the # documented Spin artifact can boot, not just compile. env: - TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:8080 + TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:8080 TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32 TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK: "false" diff --git a/CHANGELOG.md b/CHANGELOG.md index 9cc54e5ae5..6373a277bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** Where content comes from is a source module. `[source]` selects the publisher's, and each `[[proxy.asset_routes]]` entry selects its own in a `source` table of the same shape, with `module` naming the source and its settings in the table beneath. Core supplies `origin`, an HTTP origin, and a module a build links can offer another through `IntegrationRegistrationBuilder::with_source_module`. A source answers for each request where that request's content comes from, and core fetches, caches, rewrites the origin host and follows the redirect rule from that answer. `[source]` is required. The origin settings move into the `origin` table, and a document that still writes one where it was is refused at startup naming the new place. An environment variable that set the address is now `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The old name is refused where the build reads variables, and `ts config push` ignores it, as it ignores any variable for a key the document does not hold. A path in an asset route's `url` names a folder the target path is added under, where a path was refused, a target path with a `.` or `..` segment under a folder is answered `400`, and a route applies `directory_index` as the publisher does. An existing deployment's stored pages and assets keep their keys. **For crates outside core**, `handle_asset_proxy_request` takes the route's source, which `IntegrationRegistry::route_source` gives, and `Publisher::origin_host`, `origin_host_header` and `origin_query_is_stripped` are replaced by `Settings::origin_source` and the `SourceLocation` a source answers. + + | Previous | Now | + | ----------------------------------------- | ----------------------------------------- | + | `[publisher] origin_url` | `[source.origin] url` | + | `[publisher] origin_host_header_override` | `[source.origin] host_header` | + | `[publisher] origin_auth` | `[source.origin] auth` | + | `[publisher] directory_index` | `[source.origin] directory_index` | + | `[[proxy.asset_routes]] origin_url` | `[proxy.asset_routes.source.origin] url` | + | `[[proxy.asset_routes]] auth` | `[proxy.asset_routes.source.origin] auth` | + - A redirect the publisher origin sends to itself keeps the reader on the address they asked for. Where the `Location` of a `3xx` response names the origin, being the host of `origin_url` with its port, or `origin_host_header_override` where one is set, the publisher proxy sends the reader to the same path, query and fragment on the request's scheme and host, where it relayed the origin's own address and the reader's browser left the deployment. With a path in `origin_url`, a `Location` under that path comes back without it and one outside it is unchanged. A `Location` naming any other host, one that names no host, and the `Location` of a response that is not a redirect are relayed as before. - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServices::geo` and `IntegrationRegistration::with_geo_module` take an `Arc`, and `build_geo_module` returns one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths, and the Fastly adapter's two finalize paths, build the call from the request. - A module declares the caches it loads through, by name, and a deployment selects a loading cache for each in its own `[cache.]` table, the way `[cache.robots-txt]` selects one for robots.txt. A `[cache.]` table naming a cache no linked module declares is refused by `ts config validate` and when the deployment starts, naming the ones it does, and a module declaring `robots-txt` or a name another module declares is refused. `IntegrationBuilder::with_cache_features` declares them, `CacheSettings::loading_caches` lists core's own and the declared ones, and `LoadingCaches::insert` takes any name. diff --git a/crates/auction/prebid-server/src/lib.rs b/crates/auction/prebid-server/src/lib.rs index 44d7b9da7b..afedc6627a 100644 --- a/crates/auction/prebid-server/src/lib.rs +++ b/crates/auction/prebid-server/src/lib.rs @@ -3040,10 +3040,15 @@ mod tests { [auction] modules = ["prebid"] +[source] +module = "origin" + +[source.origin] +url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" -origin_url = "https://origin.test-publisher.com" proxy_secret = "test-secret" [ec] diff --git a/crates/auction/prebid/src/lib.rs b/crates/auction/prebid/src/lib.rs index bebe707333..a1d4fdbf7d 100644 --- a/crates/auction/prebid/src/lib.rs +++ b/crates/auction/prebid/src/lib.rs @@ -1722,10 +1722,15 @@ mod tests { [auction] modules = ["prebid"] +[source] +module = "origin" + +[source.origin] +url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" -origin_url = "https://origin.test-publisher.com" proxy_secret = "test-secret" [ec] diff --git a/crates/tag/google-tag-manager/src/lib.rs b/crates/tag/google-tag-manager/src/lib.rs index de36072699..eee2c0259f 100644 --- a/crates/tag/google-tag-manager/src/lib.rs +++ b/crates/tag/google-tag-manager/src/lib.rs @@ -3112,10 +3112,15 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= #[test] fn test_config_parsing() { let toml_str = r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" -origin_url = "https://origin.test-publisher.com" proxy_secret = "test-secret" [ec] @@ -3147,10 +3152,15 @@ assume_single_jurisdiction = true #[test] fn a_block_for_an_unnamed_integration_is_refused() { let toml_str = r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" -origin_url = "https://origin.test-publisher.com" proxy_secret = "test-secret" [ec] diff --git a/crates/testing/seam-probe/src/lib.rs b/crates/testing/seam-probe/src/lib.rs index 942a58e156..611ac8d0aa 100644 --- a/crates/testing/seam-probe/src/lib.rs +++ b/crates/testing/seam-probe/src/lib.rs @@ -1115,10 +1115,15 @@ mod tests { fn settings_with_probe(body: &str) -> Settings { Settings::from_toml(&format!( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "seam-probe-test-proxy-secret" [ec] @@ -1358,10 +1363,15 @@ mod tests { fn validate_reports_not_selected_without_a_module_entry() { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "seam-probe-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 3e57f2627e..f4854514eb 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -887,10 +887,15 @@ mod tests { /// inject, with the `[ec.acme]` block that module's settings live in. /// `acme` is a fictional vendor key. const UNINJECTED_MODULE_TOML: &str = r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "unit-test-proxy-secret" [ec] @@ -912,10 +917,15 @@ mod tests { fn a_requests_services_carry_the_publishers_prefixes() { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "www.publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "fictional-proxy-secret-32-bytes-ok" [geo] diff --git a/crates/trusted-server-adapter-axum/src/middleware.rs b/crates/trusted-server-adapter-axum/src/middleware.rs index afeb310356..eec062eb8f 100644 --- a/crates/trusted-server-adapter-axum/src/middleware.rs +++ b/crates/trusted-server-adapter-axum/src/middleware.rs @@ -150,10 +150,15 @@ mod tests { fn settings_with_response_headers(headers: Vec<(&str, &str)>) -> Settings { let mut s = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "unit-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index c3be7159c2..3a7fed2425 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -25,10 +25,15 @@ const LEGACY_ADMIN_DENY_METHODS: &[&str] = fn test_settings() -> trusted_server_core::settings::Settings { trusted_server_core::settings::Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "integration-test-proxy-secret" [ec] @@ -650,10 +655,15 @@ async fn first_party_proxy_rebuild_is_routed() { /// inject, with the `[ec.acme]` block that module's settings live in. /// `acme` is a fictional vendor key. const UNINJECTED_MODULE_TOML: &str = r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "integration-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-axum/tests/seam_probe.rs b/crates/trusted-server-adapter-axum/tests/seam_probe.rs index 404f183f8a..507ae2b4ab 100644 --- a/crates/trusted-server-adapter-axum/tests/seam_probe.rs +++ b/crates/trusted-server-adapter-axum/tests/seam_probe.rs @@ -58,10 +58,15 @@ assume_single_jurisdiction = true }; Settings::from_toml(&format!( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "seam-probe-test-proxy-secret" [ec] @@ -845,10 +850,15 @@ fn deploy_validation_accepts_a_modules_ad_server_only_with_its_builder() { fn settings_selecting_module(extra: &str) -> Settings { Settings::from_toml(&format!( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "seam-probe-test-proxy-secret" [geo] diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index 6e7742bf55..86467deb92 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -946,10 +946,15 @@ mod tests { /// inject, with the `[ec.acme]` block that module's settings live in. /// `acme` is a fictional vendor key. const UNINJECTED_MODULE_TOML: &str = r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "unit-test-proxy-secret" [ec] @@ -1002,10 +1007,15 @@ mod tests { fn aps_profile_settings() -> Settings { let mut settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "fictional-proxy-secret" [ec] @@ -1088,10 +1098,15 @@ mod tests { fn a_requests_services_carry_the_publishers_prefixes() { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "www.publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "fictional-proxy-secret-32-bytes-ok" [geo] @@ -1134,10 +1149,15 @@ mod tests { fn disabled_startup_accepts_dormant_multi_provider_auction_plan() { let mut settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "fictional-proxy-secret" [ec] @@ -1159,10 +1179,15 @@ mod tests { fn startup_rejects_multi_provider_auction_plan() { let mut settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "fictional-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-cloudflare/src/middleware.rs b/crates/trusted-server-adapter-cloudflare/src/middleware.rs index 3fa9f06aaa..42cc575097 100644 --- a/crates/trusted-server-adapter-cloudflare/src/middleware.rs +++ b/crates/trusted-server-adapter-cloudflare/src/middleware.rs @@ -166,10 +166,15 @@ mod tests { // by design. let mut s = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "unit-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index 6a4b58a84f..5499fc4a82 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -31,10 +31,15 @@ fn test_router() -> RouterService { fn test_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] @@ -78,10 +83,15 @@ fn assert_route_registered(method: &str, path: &str) { fn make_router() -> RouterService { let settings = trusted_server_core::settings::Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "integration-test-proxy-secret" [geo] @@ -445,10 +455,15 @@ async fn tsjs_route_prefix_is_handled_not_5xx() { /// inject, with the `[ec.acme]` block that module's settings live in. /// `acme` is a fictional vendor key. const UNINJECTED_MODULE_TOML: &str = r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] @@ -586,10 +601,15 @@ fn every_fixed_route_beneath_an_underscore_prefix_is_reserved() { fn router_refusing_every_crawler() -> RouterService { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 79a87238d4..6cb1762199 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -143,7 +143,7 @@ use trusted_server_core::publisher::{ use trusted_server_core::request_signing::{ handle_trusted_server_discovery, handle_verify_signature, }; -use trusted_server_core::settings::{ProxyAssetRoute, Settings}; +use trusted_server_core::settings::Settings; use trusted_server_core::settings_data::{ DEFAULT_CONFIG_STORE_ID, get_settings_from_config_store_with, }; @@ -1095,10 +1095,10 @@ async fn dispatch_fallback( // before the publisher fallback. Asset responses skip EC finalization // (no EcFinalizeState attached), matching legacy's `should_finalize_ec = false`. let matched_asset_route = matches!(method, Method::GET | Method::HEAD) - .then(|| state.settings.asset_route_for_path(&path)) + .then(|| state.settings.asset_route_index_for_path(&path)) .flatten(); - if let Some(asset_route) = matched_asset_route { - return dispatch_asset_fallback(state, services, req, asset_route, &effects).await; + if let Some(index) = matched_asset_route { + return dispatch_asset_fallback(state, services, req, index, &effects).await; } // Generate an EC ID if needed — mirrors the legacy catch-all arm. @@ -1207,14 +1207,22 @@ async fn dispatch_asset_fallback( state: &AppState, services: &RuntimeServices, req: Request, - asset_route: &ProxyAssetRoute, + index: usize, effects: &RequestFilterEffects, ) -> Response { log::info!("No explicit route matched; proxying via configured asset route"); let method = req.method().clone(); + let asset_route = &state.settings.proxy.asset_routes[index]; + let answer = match state.registry.route_source(&state.settings, index) { + Ok(source) => { + handle_asset_proxy_request(&state.settings, services, req, asset_route, source.as_ref()) + .await + } + Err(report) => Err(report), + }; - match handle_asset_proxy_request(&state.settings, services, req, asset_route).await { + match answer { Ok(asset_response) => { let cache_policy = asset_response.cache_policy(); let (mut response, stream_body) = asset_response.into_response_and_body(); @@ -1769,10 +1777,15 @@ mod tests { fn test_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [proxy] @@ -2143,10 +2156,15 @@ mod tests { // answered here. let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [ec] @@ -3047,10 +3065,15 @@ mod tests { // ran instead of the publisher fallback (which always attaches one). let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [ec] @@ -3069,7 +3092,8 @@ mod tests { [[proxy.asset_routes]] prefix = "/.image/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" "#, ) .expect("should parse asset-route settings"); @@ -3180,10 +3204,15 @@ mod tests { // pipe straight to the client via `stream_to_client`. let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [ec] @@ -3202,7 +3231,8 @@ mod tests { [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" "#, ) .expect("should parse asset-route settings"); @@ -3210,18 +3240,14 @@ mod tests { let services = streaming_runtime_services(); let req = empty_request(Method::GET, "/.images/logo.png"); - let asset_route = state + let index = state .settings - .asset_route_for_path("/.images/logo.png") + .asset_route_index_for_path("/.images/logo.png") .expect("should match the configured asset route"); let effects = RequestFilterEffects::default(); let response = block_on(super::dispatch_asset_fallback( - &state, - &services, - req, - asset_route, - &effects, + &state, &services, req, index, &effects, )); assert_eq!( diff --git a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs index d900fe7524..678ab0914b 100644 --- a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs +++ b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs @@ -66,10 +66,15 @@ fn settings_with(extra: &str) -> Settings { }; Settings::from_toml(&format!( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "seam-probe-test-proxy-secret" {extra} diff --git a/crates/trusted-server-adapter-fastly/src/lib.rs b/crates/trusted-server-adapter-fastly/src/lib.rs index 68362df0d0..7b83bf157f 100644 --- a/crates/trusted-server-adapter-fastly/src/lib.rs +++ b/crates/trusted-server-adapter-fastly/src/lib.rs @@ -1339,10 +1339,15 @@ mod tests { fn test_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [geo] diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index f9e780ebdb..22ef6d5f8d 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -357,10 +357,15 @@ mod tests { fn test_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [geo] diff --git a/crates/trusted-server-adapter-fastly/src/sandbox.rs b/crates/trusted-server-adapter-fastly/src/sandbox.rs index 7154c3ec84..17bdb04649 100644 --- a/crates/trusted-server-adapter-fastly/src/sandbox.rs +++ b/crates/trusted-server-adapter-fastly/src/sandbox.rs @@ -613,10 +613,15 @@ mod tests { fn test_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example" + [publisher] domain = "test-publisher.example" cookie_domain = ".test-publisher.example" - origin_url = "https://origin.test-publisher.example" proxy_secret = "unit-test-proxy-secret" [geo] diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index b083ae056a..c785f759e6 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -1170,10 +1170,15 @@ mod tests { fn multi_provider_settings() -> Settings { let mut settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "fictional-proxy-secret" [ec] @@ -1254,10 +1259,15 @@ mod tests { fn a_requests_services_carry_the_publishers_prefixes() { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "www.publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "fictional-proxy-secret-32-bytes-ok" [geo] @@ -1330,10 +1340,15 @@ mod tests { /// inject, with the `[ec.acme]` block that module's settings live in. /// `acme` is a fictional vendor key. const UNINJECTED_MODULE_TOML: &str = r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "unit-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-spin/src/middleware.rs b/crates/trusted-server-adapter-spin/src/middleware.rs index ecb104b7fb..29a624a697 100644 --- a/crates/trusted-server-adapter-spin/src/middleware.rs +++ b/crates/trusted-server-adapter-spin/src/middleware.rs @@ -190,10 +190,15 @@ mod tests { // by design. let mut s = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "unit-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index b74c1f4a93..100fcc87cf 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -30,10 +30,15 @@ fn test_router() -> RouterService { fn test_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] @@ -783,10 +788,15 @@ async fn data_endpoint_answers_a_browser_opening_the_page_and_nothing_else() { /// inject, with the `[ec.acme]` block that module's settings live in. /// `acme` is a fictional vendor key. const UNINJECTED_MODULE_TOML: &str = r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] @@ -924,10 +934,15 @@ fn every_fixed_route_beneath_an_underscore_prefix_is_reserved() { fn router_refusing_every_crawler() -> RouterService { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] diff --git a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs index bb81eef875..55425fec54 100644 --- a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs +++ b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs @@ -457,9 +457,9 @@ fn build_draft_config_with_generator( )?; draft = replace_key_in_section( &draft, - "publisher", - "origin_url", - &format!("origin_url = \"{origin}\""), + "source.origin", + "url", + &format!("url = \"{origin}\""), )?; let detected = artifact @@ -2757,7 +2757,10 @@ mod tests { assert!(draft.contains("domain = \"www.publisher.example\"")); assert!(draft.contains("cookie_domain = \".www.publisher.example\"")); - assert!(draft.contains("origin_url = \"https://www.publisher.example:8443\"")); + assert!(draft.contains( + " +url = \"https://www.publisher.example:8443\"" + )); assert!(draft.contains("Detected prebid")); let parsed = toml::from_str::(&draft).expect("draft should parse as TOML"); let selected = |section: &str| { diff --git a/crates/trusted-server-cli/tests/config_store_defaults.rs b/crates/trusted-server-cli/tests/config_store_defaults.rs index 23c5f86e9b..070ba8e119 100644 --- a/crates/trusted-server-cli/tests/config_store_defaults.rs +++ b/crates/trusted-server-cli/tests/config_store_defaults.rs @@ -50,7 +50,7 @@ fn push(project: &TempDir, args: &[&str], overrides: &[(&str, &str)]) -> Output ".publisher.example.com", ) .env( - "TRUSTED_SERVER__PUBLISHER__ORIGIN_URL", + "TRUSTED_SERVER__SOURCE__ORIGIN__URL", "https://upstream.example.com", ) .envs(overrides.iter().copied()) diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index 475ed96102..f6c6f76bcc 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -280,8 +280,8 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig { true, ), ]; - // An asset route's `auth` and `[publisher]`'s `origin_auth` take the - // same table, so each holds the same three key references. + // An asset route's source and the publisher's take the same signing + // table, so each holds the same three key references. let origin_auth_keys = ["access_key_id", "secret_access_key", "session_token"]; fields.extend(origin_auth_keys.map(|key| { field( @@ -289,6 +289,8 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig { optional_object("proxy"), optional_object("asset_routes"), SecretPathSegment::ArrayEach, + optional_object("source"), + optional_object("origin"), optional_object("auth"), object(key), ], @@ -298,8 +300,9 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig { fields.extend(origin_auth_keys.map(|key| { field( vec![ - object("publisher"), - optional_object("origin_auth"), + optional_object("source"), + optional_object("origin"), + optional_object("auth"), object(key), ], true, @@ -448,8 +451,11 @@ fn validate_non_secret_deploy_placeholders( if crate::settings::Publisher::is_placeholder_cookie_domain(&settings.publisher.cookie_domain) { insecure_fields.push("publisher.cookie_domain"); } - if crate::settings::Publisher::is_placeholder_origin_url(&settings.publisher.origin_url) { - insecure_fields.push("publisher.origin_url"); + if settings + .origin_source() + .is_some_and(|origin| origin.is_placeholder()) + { + insecure_fields.push("source.origin.url"); } if insecure_fields.is_empty() { @@ -526,15 +532,15 @@ fn validate_secret_key_references( } for (index, route) in settings.proxy.asset_routes.iter().enumerate() { - if let Some(auth) = route.auth.as_ref() { + if let Some(auth) = route.origin_source().and_then(|origin| origin.auth) { validate_origin_auth_key_references( - &format!("proxy.asset_routes[{index}].auth"), - auth, + &format!("proxy.asset_routes[{index}].source.origin.auth"), + &auth, )?; } } - if let Some(auth) = settings.publisher.origin_auth.as_ref() { - validate_origin_auth_key_references("publisher.origin_auth", auth)?; + if let Some(auth) = settings.origin_source().and_then(|origin| origin.auth) { + validate_origin_auth_key_references("source.origin.auth", &auth)?; } Ok(()) @@ -599,7 +605,8 @@ mod tests { }; use crate::test_support::template::{template_with_resolved_required_secrets, uncomment_block}; use crate::test_support::tests::{ - crate_test_settings_str, crate_test_settings_str_with_ec_section, select_hmac_module, + crate_test_settings_str, crate_test_settings_str_with_ec_section, edit_origin, + select_hmac_module, set_route_auth, }; use edgezero_core::app_config::AppConfigMeta; use edgezero_core::blob_envelope::BlobEnvelope; @@ -1025,15 +1032,21 @@ formats = [{ width = 300, height = 250 }] ("ec.partners[*].ts_pull_token".to_owned(), true), ("trusted_client_ip.shared_secret".to_owned(), false), ("analytics.tinybird.auction_token_secret".to_owned(), true), - ("proxy.asset_routes[*].auth.access_key_id".to_owned(), true), ( - "proxy.asset_routes[*].auth.secret_access_key".to_owned(), + "proxy.asset_routes[*].source.origin.auth.access_key_id".to_owned(), + true + ), + ( + "proxy.asset_routes[*].source.origin.auth.secret_access_key".to_owned(), true, ), - ("proxy.asset_routes[*].auth.session_token".to_owned(), true), - ("publisher.origin_auth.access_key_id".to_owned(), true), - ("publisher.origin_auth.secret_access_key".to_owned(), true), - ("publisher.origin_auth.session_token".to_owned(), true), + ( + "proxy.asset_routes[*].source.origin.auth.session_token".to_owned(), + true + ), + ("source.origin.auth.access_key_id".to_owned(), true), + ("source.origin.auth.secret_access_key".to_owned(), true), + ("source.origin.auth.session_token".to_owned(), true), ], "should expose the native EdgeZero secret metadata contract" ); @@ -1095,14 +1108,17 @@ formats = [{ width = 300, height = 250 }] "/assets/", "https://examplebucket.s3.us-east-1.amazonaws.com", ); - route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "us-east-1".to_string(), - secret_store: Some("legacy-s3-store".to_string()), - access_key_id: Redacted::new("s3-access-key".to_string()), - secret_access_key: Redacted::new("s3-secret-key".to_string()), - session_token: None, - origin_query: None, - })); + set_route_auth( + &mut route, + Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "us-east-1".to_string(), + secret_store: Some("legacy-s3-store".to_string()), + access_key_id: Redacted::new("s3-access-key".to_string()), + secret_access_key: Redacted::new("s3-secret-key".to_string()), + session_token: None, + origin_query: None, + })), + ); settings.proxy.asset_routes.push(route); settings @@ -1341,10 +1357,15 @@ gam_network_id = "99999" fn runtime_validation_rejects_placeholders() { let settings = Settings::from_toml( r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" + [publisher] domain = "example.com" cookie_domain = ".example.com" -origin_url = "https://origin.example.com" proxy_secret = "change-me-proxy-secret" [geo] @@ -1374,7 +1395,9 @@ passphrase = "production-secret-key-32-bytes-min" let mut settings = valid_settings(); settings.publisher.domain = "example.com".to_string(); settings.publisher.cookie_domain = ".example.com".to_string(); - settings.publisher.origin_url = "https://origin.example.com".to_string(); + edit_origin(&mut settings, |origin| { + origin.url = "https://origin.example.com".to_string() + }); let err = validate_settings_for_deploy(&settings) .expect_err("should reject unedited example publisher hosts"); @@ -1383,62 +1406,58 @@ passphrase = "production-secret-key-32-bytes-min" assert!( text.contains("publisher.domain") && text.contains("publisher.cookie_domain") - && text.contains("publisher.origin_url"), + && text.contains("source.origin.url"), "should flag all three example publisher placeholders: {err:?}" ); } - /// Valid settings whose `[publisher]` signs its origin requests. + /// Valid settings whose `[source.origin]` signs its requests. fn settings_signing_the_publisher_origin( access_key_id: &str, secret_access_key: &str, session_token: Option<&str>, ) -> Settings { let mut settings = valid_settings(); - settings.publisher.origin_url = "https://storage.example.com/bucket/site".to_string(); - settings.publisher.origin_auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "test-region-1".to_string(), - secret_store: None, - access_key_id: Redacted::new(access_key_id.to_string()), - secret_access_key: Redacted::new(secret_access_key.to_string()), - session_token: session_token.map(|token| Redacted::new(token.to_string())), - origin_query: None, - })); + edit_origin(&mut settings, |origin| { + origin.url = "https://storage.example.com/bucket/site".to_string() + }); + edit_origin(&mut settings, |origin| { + origin.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "test-region-1".to_string(), + secret_store: None, + access_key_id: Redacted::new(access_key_id.to_string()), + secret_access_key: Redacted::new(secret_access_key.to_string()), + session_token: session_token.map(|token| Redacted::new(token.to_string())), + origin_query: None, + })) + }); settings } #[test] - fn deploy_validation_accepts_a_publisher_origin_auth_naming_its_keys() { + fn deploy_validation_accepts_a_source_origin_auth_naming_its_keys() { validate_settings_for_deploy(&settings_signing_the_publisher_origin( "origin_access_key_id", "origin_secret_access_key", Some("origin_session_token"), )) - .expect("should accept an origin_auth whose key references are named"); + .expect("should accept an auth whose key references are named"); } #[test] - fn deploy_validation_refuses_an_empty_publisher_origin_auth_key_reference() { + fn deploy_validation_refuses_an_empty_source_origin_auth_key_reference() { for (settings, path) in [ ( settings_signing_the_publisher_origin(" ", "origin_secret_access_key", None), - "publisher.origin_auth.access_key_id", + "source.origin.auth.access_key_id", ), ( settings_signing_the_publisher_origin("origin_access_key_id", "", None), - "publisher.origin_auth.secret_access_key", - ), - ( - settings_signing_the_publisher_origin( - "origin_access_key_id", - "origin_secret_access_key", - Some(" "), - ), - "publisher.origin_auth.session_token", + "source.origin.auth.secret_access_key", ), ] { let err = validate_settings_for_deploy(&settings) - .expect_err("should refuse an empty origin_auth key reference"); + .expect_err("should refuse an empty auth key reference"); assert!( format!("{err:?}").contains(&format!( "secret key reference at `{path}` must not be empty" @@ -1449,10 +1468,27 @@ passphrase = "production-secret-key-32-bytes-min" } #[test] - fn documented_publisher_object_storage_settings_validate_when_uncommented() { + fn a_blank_source_origin_session_token_is_read_as_none() { + let settings = settings_signing_the_publisher_origin( + "origin_access_key_id", + "origin_secret_access_key", + Some(" "), + ); + validate_settings_for_deploy(&settings) + .expect("a blank session token is no token, as a document's is"); + let Some(AssetOriginAuth::S3SigV4(auth)) = + settings.origin_source().and_then(|origin| origin.auth) + else { + panic!("should keep the origin auth"); + }; + assert!(auth.session_token.is_none()); + } + + #[test] + fn documented_source_origin_object_storage_settings_validate_when_uncommented() { let toml = uncomment_block( &template_with_resolved_required_secrets(), - "[publisher.origin_auth]", + "[source.origin.auth]", ) .replacen( "# directory_index = \"index.html\"", @@ -1460,17 +1496,17 @@ passphrase = "production-secret-key-32-bytes-min" 1, ); let settings = Settings::from_toml(&toml) - .expect("the documented origin_auth and directory_index should parse and validate"); + .expect("the documented auth and directory_index should parse and validate"); + let origin = settings + .origin_source() + .expect("the template should select the origin source"); assert!( - matches!( - settings.publisher.origin_auth, - Some(AssetOriginAuth::S3SigV4(_)) - ), - "the documented origin_auth should be read" + matches!(origin.auth, Some(AssetOriginAuth::S3SigV4(_))), + "the documented auth should be read" ); assert_eq!( - settings.publisher.directory_index.as_deref(), + origin.directory_index.as_deref(), Some("index.html"), "the documented directory_index should be read" ); diff --git a/crates/trusted-server-core/src/config_payload.rs b/crates/trusted-server-core/src/config_payload.rs index ca9122b61f..b1baa3a518 100644 --- a/crates/trusted-server-core/src/config_payload.rs +++ b/crates/trusted-server-core/src/config_payload.rs @@ -146,7 +146,8 @@ mod tests { TinybirdSettings, TrustedClientIpConfig, }; use crate::test_support::tests::{ - crate_test_settings_str, hmac_passphrase, select_hmac_module, select_host_signals_module, + crate_test_settings_str, edit_origin, hmac_passphrase, select_hmac_module, + select_host_signals_module, set_route_auth, }; fn test_settings() -> Settings { @@ -521,14 +522,17 @@ mod tests { "/assets/", "https://examplebucket.s3.us-east-1.amazonaws.com", ); - route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "us-east-1".to_string(), - secret_store: Some("legacy-s3-store".to_string()), - access_key_id: Redacted::new("s3-access-key".to_string()), - secret_access_key: Redacted::new("s3-secret-key".to_string()), - session_token: Some(Redacted::new("s3-session-key".to_string())), - origin_query: None, - })); + set_route_auth( + &mut route, + Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "us-east-1".to_string(), + secret_store: Some("legacy-s3-store".to_string()), + access_key_id: Redacted::new("s3-access-key".to_string()), + secret_access_key: Redacted::new("s3-secret-key".to_string()), + session_token: Some(Redacted::new("s3-session-key".to_string())), + origin_query: None, + })), + ); original.proxy.asset_routes.push(route); original .ec @@ -551,8 +555,8 @@ mod tests { Some("resolved-tinybird-token") ); let auth = reconstructed.proxy.asset_routes[0] - .auth - .as_ref() + .origin_source() + .and_then(|origin| origin.auth) .expect("should preserve S3 auth"); let AssetOriginAuth::S3SigV4(auth) = auth; assert_eq!(auth.access_key_id.expose(), "AKIAIOSFODNN7EXAMPLE"); @@ -752,9 +756,12 @@ mod tests { "/default-s3/", "https://examplebucket.s3.us-east-1.amazonaws.com", ); - route.auth = Some(AssetOriginAuth::S3SigV4( - toml::from_str("region = \"us-east-1\"").expect("should apply S3 secret defaults"), - )); + set_route_auth( + &mut route, + Some(AssetOriginAuth::S3SigV4( + toml::from_str("region = \"us-east-1\"").expect("should apply S3 secret defaults"), + )), + ); original.proxy.asset_routes.push(route); let reconstructed = settings_from_config_blob( @@ -765,8 +772,8 @@ mod tests { .expect("should resolve default S3 secret keys"); let AssetOriginAuth::S3SigV4(auth) = reconstructed.proxy.asset_routes[0] - .auth - .as_ref() + .origin_source() + .and_then(|origin| origin.auth) .expect("should preserve S3 auth"); assert_eq!(auth.access_key_id.expose(), "AKIAIOSFODNN7EXAMPLE"); assert_eq!( @@ -776,17 +783,21 @@ mod tests { } #[test] - fn publisher_origin_auth_resolves_its_credentials_from_the_store() { + fn source_origin_auth_resolves_its_credentials_from_the_store() { let mut original = test_settings(); - original.publisher.origin_url = "https://storage.example.com/bucket/site".to_string(); - original.publisher.origin_auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "test-region-1".to_string(), - secret_store: None, - access_key_id: Redacted::new("s3-access-key".to_string()), - secret_access_key: Redacted::new("s3-secret-key".to_string()), - session_token: Some(Redacted::new("s3-session-key".to_string())), - origin_query: None, - })); + edit_origin(&mut original, |origin| { + origin.url = "https://storage.example.com/bucket/site".to_string() + }); + edit_origin(&mut original, |origin| { + origin.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "test-region-1".to_string(), + secret_store: None, + access_key_id: Redacted::new("s3-access-key".to_string()), + secret_access_key: Redacted::new("s3-secret-key".to_string()), + session_token: Some(Redacted::new("s3-session-key".to_string())), + origin_query: None, + })) + }); let reconstructed = settings_from_config_blob( &envelope_json(&original), @@ -795,9 +806,10 @@ mod tests { ) .expect("should resolve the publisher origin credentials from the mapped store"); - let Some(AssetOriginAuth::S3SigV4(auth)) = reconstructed.publisher.origin_auth.as_ref() + let Some(AssetOriginAuth::S3SigV4(auth)) = + reconstructed.origin_source().and_then(|origin| origin.auth) else { - panic!("should keep the publisher origin_auth"); + panic!("should keep the origin auth"); }; assert_eq!(auth.access_key_id.expose(), "AKIAIOSFODNN7EXAMPLE"); assert_eq!( diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 931199aebf..7d48594772 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -1054,10 +1054,15 @@ pub mod test_support { /// The least a deployment's settings must say, which a module's own /// settings are written after. const BASE_SETTINGS: &str = r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" + [publisher] domain = "publisher.example.com" cookie_domain = ".publisher.example.com" -origin_url = "https://origin.example.com" proxy_secret = "page-recording-proxy-secret" [geo] diff --git a/crates/trusted-server-core/src/inspect/config.rs b/crates/trusted-server-core/src/inspect/config.rs index 0e0a5171bf..fce32e2ae4 100644 --- a/crates/trusted-server-core/src/inspect/config.rs +++ b/crates/trusted-server-core/src/inspect/config.rs @@ -44,7 +44,7 @@ const NOT_PUBLISHED: &str = "This publisher does not publish its configuration." const PATTERN_FORM: &str = "A pattern is keys joined by `.`, with `[]` for every element of a \ list or `[N]` for one, such as \ - `proxy.asset_routes[0].origin_url`"; + `proxy.asset_routes[0].source.origin.url`"; /// The `[inspect]` section, which says what the configuration page shows. #[derive(Debug, Clone, PartialEq, Eq, Deserialize, Serialize)] @@ -93,7 +93,7 @@ pub enum PathStep { } /// A concrete path written the way a pattern is, such as -/// `proxy.asset_routes[0].origin_url`. +/// `proxy.asset_routes[0].source.origin.url`. #[must_use] pub fn render_path(steps: &[PathStep]) -> String { let mut out = String::new(); @@ -122,7 +122,7 @@ enum PatternStep { /// Values in the view named by keys joined by `.`, with `[]` for every /// element of a list and `[N]` for one, such as -/// `proxy.asset_routes[].origin_url` or `proxy.asset_routes[0].prefix`. +/// `proxy.asset_routes[].source.origin.url` or `proxy.asset_routes[0].prefix`. /// /// A pattern names the values at exactly its own depth. A key holding `.`, /// `[` or `]` cannot be named. diff --git a/crates/trusted-server-core/src/inspect/config/tests.rs b/crates/trusted-server-core/src/inspect/config/tests.rs index c434011653..4b19b49cea 100644 --- a/crates/trusted-server-core/src/inspect/config/tests.rs +++ b/crates/trusted-server-core/src/inspect/config/tests.rs @@ -105,7 +105,7 @@ fn served(settings: &Settings) -> (String, String) { ) } -/// The value at a concrete path, such as `proxy.asset_routes[0].origin_url`. +/// The value at a concrete path, such as `proxy.asset_routes[0].prefix`. fn at<'v>(value: &'v Value, path: &str) -> Option<&'v Value> { let pattern = PathPattern::parse(path).expect("a concrete path"); let mut node = value; @@ -141,14 +141,14 @@ fn unmarked(value: &Value) -> Value { #[test] fn a_pattern_names_keys_every_element_or_one_element() { - let every = PathPattern::parse("proxy.asset_routes[].origin_url").expect("well formed"); - let one = PathPattern::parse("proxy.asset_routes[1].origin_url").expect("well formed"); + let every = PathPattern::parse("proxy.asset_routes[].prefix").expect("well formed"); + let one = PathPattern::parse("proxy.asset_routes[1].prefix").expect("well formed"); let route = |index| { vec![ PathStep::Key("proxy".to_owned()), PathStep::Key("asset_routes".to_owned()), PathStep::Index(index), - PathStep::Key("origin_url".to_owned()), + PathStep::Key("prefix".to_owned()), ] }; @@ -159,7 +159,7 @@ fn a_pattern_names_keys_every_element_or_one_element() { !every.matches(&route(0)[..3]), "should name only values at its own depth" ); - assert_eq!(render_path(&route(1)), "proxy.asset_routes[1].origin_url"); + assert_eq!(render_path(&route(1)), "proxy.asset_routes[1].prefix"); } #[test] @@ -208,7 +208,7 @@ fn a_key_the_inspect_section_does_not_have_is_refused_naming_it() { fn documents_holding_every_declared_secret() -> Vec<(&'static str, Value)> { let mut main = document(""); main["publisher"]["proxy_secret"] = json!("proxy"); - main["publisher"]["origin_auth"] = json!({ + main["source"]["origin"]["auth"] = json!({ "type": "s3_sigv4", "region": "test-region-1", "access_key_id": "origin-access", @@ -232,13 +232,18 @@ fn documents_holding_every_declared_secret() -> Vec<(&'static str, Value)> { }); main["proxy"]["asset_routes"] = json!([{ "prefix": "/assets/", - "origin_url": "https://assets.example.com", - "auth": { - "type": "s3_sigv4", - "region": "us-east-1", - "access_key_id": "route-access", - "secret_access_key": "route-secret", - "session_token": "route-session", + "source": { + "module": "origin", + "origin": { + "url": "https://assets.example.com", + "auth": { + "type": "s3_sigv4", + "region": "us-east-1", + "access_key_id": "route-access", + "secret_access_key": "route-secret", + "session_token": "route-session", + }, + }, }, }]); main["analytics"] = json!({ @@ -346,13 +351,10 @@ fn every_declared_core_secret_is_masked_and_never_served() { /// sensitive, with those paths and values. fn settings_with_every_sensitive_field() -> (Settings, Vec<(&'static str, &'static str)>) { let expected = vec![ - ("publisher.origin_url", "https://origin-canary.example.com"), + ("source.origin.url", "https://origin-canary.example.com"), + ("source.origin.host_header", "host-canary.example.com"), ( - "publisher.origin_host_header_override", - "host-canary.example.com", - ), - ( - "proxy.asset_routes[0].origin_url", + "proxy.asset_routes[0].source.origin.url", "https://bucket-canary.example.com", ), ("ec.ec_store", "identity-store-canary"), @@ -366,12 +368,13 @@ fn settings_with_every_sensitive_field() -> (Settings, Vec<(&'static str, &'stat .expect("an expected path") }; let mut settings = settings(); - settings.publisher.origin_url = value("publisher.origin_url"); - settings.publisher.origin_host_header_override = - Some(value("publisher.origin_host_header_override")); + settings.set_origin_source(&crate::source::OriginSettings { + host_header: Some(value("source.origin.host_header")), + ..crate::source::OriginSettings::new(value("source.origin.url")) + }); settings.proxy.asset_routes = vec![ProxyAssetRoute::new( "/assets/", - value("proxy.asset_routes[0].origin_url"), + value("proxy.asset_routes[0].source.origin.url"), )]; settings.ec.ec_store = Some(value("ec.ec_store")); settings.auction.creative_store = value("auction.creative_store"); @@ -584,9 +587,10 @@ fn a_path_masked_beneath_a_key_holding_a_secret_lists_the_renamed_key() { const ASSET_ROUTES: &str = r#" [[proxy.asset_routes]] prefix = "/assets/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" - [proxy.asset_routes.auth] + [proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "us-east-1" access_key_id = "route-access" @@ -594,29 +598,30 @@ const ASSET_ROUTES: &str = r#" [[proxy.asset_routes]] prefix = "/media/" - origin_url = "https://media.example.com" + source.module = "origin" + source.origin.url = "https://media.example.com" "#; #[test] fn show_reveals_a_value_masked_by_default() { let settings = load(document(&format!( - "{ASSET_ROUTES}\n[inspect]\nshow = [\"publisher.origin_url\", \ - \"proxy.asset_routes[].origin_url\"]" + "{ASSET_ROUTES}\n[inspect]\nshow = [\"source.origin.url\", \ + \"proxy.asset_routes[].source.origin.url\"]" ))) .expect("should load"); let view = view(&settings); assert_eq!( - at(&view.settings, "publisher.origin_url"), + at(&view.settings, "source.origin.url"), Some(&json!("https://origin.test-publisher.com")) ); assert_eq!( - at(&view.settings, "proxy.asset_routes[1].origin_url"), + at(&view.settings, "proxy.asset_routes[1].source.origin.url"), Some(&json!("https://media.example.com")) ); - assert!(!view.masked.iter().any(|path| path.ends_with(".origin_url"))); - assert!(!is_masked(&view, "publisher.origin_url")); + assert!(!view.masked.iter().any(|path| path.ends_with(".origin.url"))); + assert!(!is_masked(&view, "source.origin.url")); } #[test] @@ -640,16 +645,16 @@ fn a_pattern_not_honored_as_written_is_refused_saying_why() { "show names `publisher.proxy_secret`, which is a secret", ), ( - r#"show = ["proxy.asset_routes[].auth.secret_access_key"]"#, - "show names `proxy.asset_routes[].auth.secret_access_key`, which is a secret", + r#"show = ["proxy.asset_routes[].source.origin.auth.secret_access_key"]"#, + "show names `proxy.asset_routes[].source.origin.auth.secret_access_key`, which is a secret", ), ( r#"hide = ["publisher.no_such_setting"]"#, "hide names `publisher.no_such_setting`, which matches no value", ), ( - r#"show = ["proxy.asset_routes[9].origin_url"]"#, - "show names `proxy.asset_routes[9].origin_url`, which matches no value", + r#"show = ["proxy.asset_routes[9].source.origin.url"]"#, + "show names `proxy.asset_routes[9].source.origin.url`, which matches no value", ), ( r#"show = ["publisher.domain"]"#, @@ -657,8 +662,8 @@ fn a_pattern_not_honored_as_written_is_refused_saying_why() { it names is masked by default", ), ( - r#"hide = ["publisher.origin_url"]"#, - "hide names `publisher.origin_url`, which changes nothing, because \ + r#"hide = ["source.origin.url"]"#, + "hide names `source.origin.url`, which changes nothing, because \ everything it names is masked or empty already", ), ( @@ -667,20 +672,20 @@ fn a_pattern_not_honored_as_written_is_refused_saying_why() { everything it names is masked or empty already", ), ( - r#"show = ["publisher.origin_url", "publisher.origin_url"]"#, - "show names `publisher.origin_url`, which changes nothing, because \ - `publisher.origin_url` in the same list already names everything it does", + r#"show = ["source.origin.url", "source.origin.url"]"#, + "show names `source.origin.url`, which changes nothing, because \ + `source.origin.url` in the same list already names everything it does", ), ( - r#"show = ["proxy.asset_routes[].origin_url", "proxy.asset_routes[0].origin_url"]"#, - "show names `proxy.asset_routes[0].origin_url`, which changes nothing, because \ - `proxy.asset_routes[].origin_url` in the same list already names everything \ + r#"show = ["proxy.asset_routes[].source.origin.url", "proxy.asset_routes[0].source.origin.url"]"#, + "show names `proxy.asset_routes[0].source.origin.url`, which changes nothing, because \ + `proxy.asset_routes[].source.origin.url` in the same list already names everything \ it does", ), ( - "show = [\"publisher.origin_url\"]\nhide = [\"publisher.origin_url\"]", - "show names `publisher.origin_url`, which changes nothing, because hide \ - names it as `publisher.origin_url` and hide wins over show", + "show = [\"source.origin.url\"]\nhide = [\"source.origin.url\"]", + "show names `source.origin.url`, which changes nothing, because hide \ + names it as `source.origin.url` and hide wins over show", ), ( "config = false\nhide = [\"publisher.domain\"]", diff --git a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs index 183288a568..8ba50ac61f 100644 --- a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs +++ b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs @@ -1060,10 +1060,15 @@ mod tests { #[test] fn rejects_unknown_proxy_mode() { let toml = r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [ec] diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 1e19746cc2..56d0fd9737 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -25,6 +25,7 @@ use crate::platform::{DisabledGeo, GeoModule, PlatformGeoModule, RuntimeServices use crate::prefix::{PrefixModule, Prefixes, SelectedPrefixModule}; use crate::robots_txt::{ALLOW_ALL, REFUSE_ALL, RobotsTxtContributor}; use crate::settings::Settings; +use crate::source::{Source, SourceFactory}; /// What a middleware's element handler decides about one attribute of an /// element. @@ -681,6 +682,12 @@ pub struct IntegrationRegistration { /// Declaring them does not put them in the file, because the file is only /// built from the contributors `[robots-txt] modules` names. pub robots_txt_contributor: Option<(&'static str, Arc)>, + /// Source module this registration supplies, with the name a source table + /// selects it by, see [`crate::source`]. + /// + /// Declaring one does not make it serve anything, because a source is only + /// built where `[source]` or an asset route's `source` table names it. + pub source_module: Option<(&'static str, Arc)>, } impl IntegrationRegistration { @@ -712,6 +719,7 @@ impl IntegrationRegistrationBuilder { device_module: None, prefix_module: None, robots_txt_contributor: None, + source_module: None, }, } } @@ -805,6 +813,19 @@ impl IntegrationRegistrationBuilder { self } + /// Declare a source module under `name`, which `[source]` or an asset + /// route's `source` table selects. The factory builds a source from the + /// selecting table's settings, see [`crate::source`]. + #[must_use] + pub fn with_source_module( + mut self, + name: &'static str, + factory: Arc, + ) -> Self { + self.registration.source_module = Some((name, factory)); + self + } + /// Declare the rules this registration adds to `/robots.txt`, under their /// `name`. /// @@ -941,6 +962,18 @@ struct IntegrationRegistryInner { // The prefix module `[prefix] module` resolved to, core's own when // nothing is selected. prefix_module: SelectedPrefixModule, + // The source modules the running registrations supply, each with its name. + source_modules: Vec<(&'static str, Arc)>, + // The publisher's source and each asset route's, in route order, built + // from the settings. `None` for a registry built without settings. + sources: Option, +} + +/// The sources a deployment's settings name, built once. +#[derive(Clone)] +struct BuiltSources { + publisher: Arc, + routes: Vec>, } impl Default for IntegrationRegistryInner { @@ -977,6 +1010,8 @@ impl Default for IntegrationRegistryInner { device_module: None, prefix_modules: Vec::new(), prefix_module: SelectedPrefixModule::default(), + source_modules: Vec::new(), + sources: None, } } } @@ -1590,6 +1625,12 @@ fn check_section_selections( .filter_map(|builder| builder.module_name()) .collect(); for (section, modules) in settings.module_sections() { + // A source is offered by a registration under a name of its own, as + // a prefix module is, so building the sources checks the selection + // against the sources on offer. + if section == crate::source::SOURCE_SECTION { + continue; + } for written in modules.selected() { if crate::module_name::resolve(section, written, &offered).is_some() { continue; @@ -1628,7 +1669,111 @@ fn check_section_selections( Ok(()) } +/// The source `selection` names, built from its table, where `setting` names +/// the table in a refusal. Core's `origin` is always on offer, beside the +/// sources the registrations supply. +fn build_source( + selection: &crate::provider_table::SectionModules, + setting: &str, + offered: &[(&'static str, Arc)], +) -> Result, Report> { + let (written, table) = crate::source::selected(selection, setting)?; + let table_setting = format!("{setting}.{written}"); + let names: Vec<&str> = offered + .iter() + .map(|(name, _)| *name) + .chain(std::iter::once(crate::source::ORIGIN_MODULE)) + .collect(); + let Some(found) = crate::module_name::resolve(crate::source::SOURCE_SECTION, written, &names) + else { + let short: Vec<&str> = names + .iter() + .map(|name| crate::module_name::short_form(crate::source::SOURCE_SECTION, name)) + .collect(); + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "[{setting}] selects `{written}`, which no module this deployment supplies. \ + The source modules it supplies are [{}]", + short.join(", ") + ), + })); + }; + if found == crate::source::ORIGIN_MODULE { + return crate::source::OriginFactory.build(&table, &table_setting); + } + let factory = offered + .iter() + .find(|(name, _)| *name == found) + .map(|(_, factory)| Arc::clone(factory)) + .ok_or_else(|| { + Report::new(TrustedServerError::Configuration { + message: format!("[{setting}] selects `{written}`, which nothing builds"), + }) + })?; + factory.build(&table, &table_setting) +} + +/// The publisher's source and each asset route's, built from `settings`. +fn build_sources( + settings: &Settings, + offered: &[(&'static str, Arc)], +) -> Result> { + let publisher = build_source(&settings.source, crate::source::SOURCE_SECTION, offered)?; + let routes = settings + .proxy + .asset_routes + .iter() + .map(|route| build_source(&route.source, &route.source_setting(), offered)) + .collect::, _>>()?; + Ok(BuiltSources { publisher, routes }) +} + impl IntegrationRegistry { + /// The publisher's source, built when the registry was built from the + /// settings, or built now from `settings` with core's sources alone for a + /// registry built without them. + /// + /// # Errors + /// + /// When the registry holds none and `settings` name a source core does + /// not supply, or one it cannot build. + pub fn publisher_source( + &self, + settings: &Settings, + ) -> Result, Report> { + match &self.inner.sources { + Some(sources) => Ok(Arc::clone(&sources.publisher)), + None => build_source(&settings.source, crate::source::SOURCE_SECTION, &[]), + } + } + + /// The source of the asset route at `index` in `settings`, built as for + /// [`Self::publisher_source`]. + /// + /// # Errors + /// + /// When there is no route at `index`, or as [`Self::publisher_source`]. + pub fn route_source( + &self, + settings: &Settings, + index: usize, + ) -> Result, Report> { + if let Some(source) = self + .inner + .sources + .as_ref() + .and_then(|sources| sources.routes.get(index)) + { + return Ok(Arc::clone(source)); + } + let route = settings.proxy.asset_routes.get(index).ok_or_else(|| { + Report::new(TrustedServerError::Configuration { + message: format!("there is no asset route at {index}"), + }) + })?; + build_source(&route.source, &route.source_setting(), &[]) + } + /// Tells the registry what the adapter reports about the platform, which /// a middleware is told about every document. #[must_use] @@ -1963,6 +2108,10 @@ impl IntegrationRegistry { claim_module_name(&mut claimed, "prefix", name, registration.integration_id)?; inner.prefix_modules.push((name, module)); } + if let Some((name, factory)) = registration.source_module { + claim_module_name(&mut claimed, "source", name, registration.integration_id)?; + inner.source_modules.push((name, factory)); + } if let Some((name, contributor)) = registration.robots_txt_contributor { claim_module_name( &mut claimed, @@ -2021,6 +2170,7 @@ impl IntegrationRegistry { inner.ec_module = resolve_ec_module(settings, &inner); inner.device_module = resolve_device_module(settings, &inner)?; inner.prefix_module = resolve_prefix_module(settings, &inner)?; + inner.sources = Some(build_sources(settings, &inner.source_modules)?); inner.geo_module = geo_module; check_robots_txt_selection(settings, &inner)?; if let Some(config) = settings.robots_txt.as_ref() { @@ -2632,6 +2782,8 @@ impl IntegrationRegistry { device_module: None, prefix_modules: Vec::new(), prefix_module: SelectedPrefixModule::default(), + source_modules: Vec::new(), + sources: None, }), plan: None, platform: Arc::default(), @@ -2714,6 +2866,8 @@ impl IntegrationRegistry { device_module: None, prefix_modules: Vec::new(), prefix_module: SelectedPrefixModule::default(), + source_modules: Vec::new(), + sources: None, }), plan: None, platform: Arc::default(), diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 13c00b2984..3a84b26c25 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -79,6 +79,7 @@ pub mod secret_resolution; pub mod service_version; pub mod settings; pub mod settings_data; +pub mod source; pub mod source_cache; pub mod streaming_processor; pub mod streaming_replacer; diff --git a/crates/trusted-server-core/src/provider_table.rs b/crates/trusted-server-core/src/provider_table.rs index 9715accdc7..b8222a319e 100644 --- a/crates/trusted-server-core/src/provider_table.rs +++ b/crates/trusted-server-core/src/provider_table.rs @@ -433,6 +433,18 @@ impl SectionModules { Some(table) } + /// A section selecting `written` alone, with `module`, and holding its + /// settings table. + #[must_use] + pub fn one(written: &str, settings: Map) -> Self { + let mut section = Self { + key: Some(SelectionKey::Module), + ..Self::default() + }; + section.insert(written, settings); + section + } + /// Selects `written` and stores its settings table, which is what writing /// both in a document does. pub fn insert(&mut self, written: &str, settings: Map) { diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs index 364c07b63a..223dde0c33 100644 --- a/crates/trusted-server-core/src/proxy.rs +++ b/crates/trusted-server-core/src/proxy.rs @@ -29,6 +29,7 @@ use crate::platform::{ }; use crate::s3_sigv4::{self, S3Credentials}; use crate::settings::{AssetOriginAuth, OriginQueryPolicy, ProxyAssetRoute, Settings}; +use crate::source::{Source, SourceLocation, SourceRequest}; use crate::streaming_processor::{Compression, PipelineConfig, StreamProcessor, StreamingPipeline}; /// Chunk size used for streaming content through the rewrite pipeline. @@ -856,29 +857,29 @@ fn default_port_for_scheme(scheme: &str) -> Option { fn build_asset_proxy_target_url( route: &ProxyAssetRoute, + location: &SourceLocation, path: &str, query: &str, ) -> Result> { - let mut target_url = - url::Url::parse(&route.origin_url).change_context(TrustedServerError::Proxy { - message: format!("Invalid asset origin_url: {}", route.origin_url), - })?; + let mut target_url = location.parsed().clone(); + // A folder in the source's address comes before the route's target path, + // and a target path that could climb out of it is refused. + let folder = target_url.path().trim_end_matches('/').to_owned(); - let scheme = target_url.scheme(); - if scheme != "http" && scheme != "https" { - return Err(Report::new(TrustedServerError::Proxy { - message: format!("Unsupported asset origin_url scheme: {scheme}"), + let mut target_path = route.target_path_for(path)?; + if !folder.is_empty() && crate::publisher::path_has_dot_segment(&target_path) { + return Err(Report::new(TrustedServerError::BadRequest { + message: "an asset path under a source folder must not have a `.` or `..` segment" + .to_string(), })); } - - if target_url.host_str().is_none() { - return Err(Report::new(TrustedServerError::Proxy { - message: "Missing host in asset origin_url".to_string(), - })); + if let Some(index) = location + .directory_index() + .filter(|_| target_path.ends_with('/')) + { + target_path.push_str(index); } - - let target_path = route.target_path_for(path)?; - target_url.set_path(&target_path); + target_url.set_path(&format!("{folder}{target_path}")); if query.is_empty() { target_url.set_query(None); } else { @@ -1031,24 +1032,24 @@ fn apply_no_store_cache_control(response: &mut Response) { } fn should_preflight_s3( - route: &ProxyAssetRoute, + location: &SourceLocation, image_optimizer_enabled: bool, request_method: &Method, ) -> bool { image_optimizer_enabled - && matches!(route.auth.as_ref(), Some(AssetOriginAuth::S3SigV4(_))) + && matches!(location.auth(), Some(AssetOriginAuth::S3SigV4(_))) && (request_method == Method::GET || request_method == Method::HEAD) } async fn preflight_s3_origin_for_image_optimizer( services: &RuntimeServices, - route: &ProxyAssetRoute, + location: &SourceLocation, target_url: &url::Url, request_method: &Method, unsigned_headers: &http::HeaderMap, backend_name: &str, ) -> Result, Report> { - let Some(auth @ AssetOriginAuth::S3SigV4(_)) = route.auth.as_ref() else { + let Some(auth @ AssetOriginAuth::S3SigV4(_)) = location.auth() else { return Ok(None); }; @@ -1116,10 +1117,23 @@ pub async fn handle_asset_proxy_request( services: &RuntimeServices, req: Request, route: &ProxyAssetRoute, + source: &dyn Source, ) -> Result> { let incoming_path = req.uri().path(); let incoming_query = req.uri().query().unwrap_or(""); - let mut target_url = build_asset_proxy_target_url(route, incoming_path, incoming_query)?; + let removed_prefix = req + .extensions() + .get::() + .map(|prefix| prefix.as_str().to_owned()); + let location = source.locate(&SourceRequest { + method: req.method(), + path: incoming_path, + query: incoming_query, + headers: req.headers(), + removed_prefix: removed_prefix.as_deref(), + })?; + let mut target_url = + build_asset_proxy_target_url(route, &location, incoming_path, incoming_query)?; let skip_image_optimizer = asset_path_skips_image_optimizer(incoming_path) || asset_path_skips_image_optimizer(target_url.path()); let image_optimizer = if skip_image_optimizer { @@ -1133,7 +1147,7 @@ pub async fn handle_asset_proxy_request( crate::asset_image_optimizer::options_for_asset_request(settings, route, incoming_query)? }; - if route.origin_query_policy() == OriginQueryPolicy::Strip { + if route.origin_query_policy_at(&location) == OriginQueryPolicy::Strip { target_url.set_query(None); } @@ -1150,7 +1164,7 @@ pub async fn handle_asset_proxy_request( scheme: scheme.to_string(), host: host.to_string(), port: target_url.port(), - host_header_override: None, + host_header_override: location.host_header_override().map(str::to_owned), certificate_check: settings.proxy.certificate_check, first_byte_timeout: DEFAULT_FIRST_BYTE_TIMEOUT, between_bytes_timeout: DEFAULT_FIRST_BYTE_TIMEOUT, @@ -1166,12 +1180,18 @@ pub async fn handle_asset_proxy_request( outbound_headers.insert(header_name, value.clone()); } } - outbound_headers.insert(header::HOST, asset_origin_host_header(&target_url)?); + let host_header = match location.host_header_override() { + Some(host) => HeaderValue::from_str(host).change_context(TrustedServerError::Proxy { + message: "invalid source host header".to_string(), + })?, + None => asset_origin_host_header(&target_url)?, + }; + outbound_headers.insert(header::HOST, host_header); - if should_preflight_s3(route, image_optimizer.is_some(), req.method()) + if should_preflight_s3(&location, image_optimizer.is_some(), req.method()) && let Some(response) = preflight_s3_origin_for_image_optimizer( services, - route, + &location, &target_url, req.method(), &outbound_headers, @@ -1182,7 +1202,7 @@ pub async fn handle_asset_proxy_request( return Ok(response); } - if let Some(auth) = &route.auth { + if let Some(auth) = location.auth() { apply_asset_origin_auth(req.method(), &target_url, &mut outbound_headers, auth)?; } @@ -1194,7 +1214,7 @@ pub async fn handle_asset_proxy_request( // answer the platform's image optimizer makes is not kept, because the // platform holds it. let copy_key = if image_optimizer.is_none() { - asset_route_copy_key(services, &req, route, &target_url) + asset_route_copy_key(services, &req, route, &location, &target_url) } else { None }; @@ -1260,6 +1280,16 @@ pub async fn handle_asset_proxy_request( Ok(response) } +/// What tells one route's copies apart from another's: the address its +/// source answered and the route's prefix, so two routes to one address keep +/// their own copies, and the `Host` where the source sets one. +fn route_origin_identity(route: &ProxyAssetRoute, location: &SourceLocation) -> String { + match location.host_header_override() { + Some(host) => format!("{}\0{}\0{host}", location.url(), route.prefix), + None => format!("{}\0{}", location.url(), route.prefix), + } +} + /// The key an asset route's answer is held under, or nothing for a request /// whose answer is not kept: one that is not a `GET`, one carrying /// `Authorization`, a conditional header or a range, which a copy cannot @@ -1271,6 +1301,7 @@ fn asset_route_copy_key( services: &RuntimeServices, req: &Request, route: &ProxyAssetRoute, + location: &SourceLocation, target_url: &url::Url, ) -> Option { if req.method() != Method::GET @@ -1300,7 +1331,7 @@ fn asset_route_copy_key( url: target_url.to_string(), request_host, request_scheme: req.uri().scheme_str().unwrap_or("https").to_owned(), - origin_identity: format!("{}\0{}", route.origin_url, route.prefix), + origin_identity: route_origin_identity(route, location), fingerprint: String::new(), }) } @@ -2362,7 +2393,8 @@ mod tests { asset_path_skips_image_optimizer, build_asset_proxy_target_url, handle_asset_proxy_request, handle_first_party_click, handle_first_party_proxy, handle_first_party_proxy_rebuild, handle_first_party_proxy_sign, is_host_allowed, is_host_permitted, proxy_request, - rebuild_response_with_body, reconstruct_and_validate_signed_target, stream_asset_body, + rebuild_response_with_body, reconstruct_and_validate_signed_target, route_origin_identity, + stream_asset_body, }; use crate::cache_policy::{CachePolicy, EdgeCacheHeader}; use crate::constants::{HEADER_ACCEPT, HEADER_X_FORWARDED_FOR}; @@ -2383,7 +2415,9 @@ mod tests { ImageOptimizerCropOffsetsConfig, ImageOptimizerProfileSet, ImageOptimizerSettings, OriginQueryPolicy, ProxyAssetRoute, S3SigV4AuthConfig, Settings, UnknownProfilePolicy, }; - use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + use crate::test_support::tests::{ + crate_test_settings_str, create_test_settings, set_route_auth, + }; use bytes::Bytes; use edgezero_core::body::Body as EdgeBody; use edgezero_core::http::response_builder as edge_response_builder; @@ -4989,12 +5023,128 @@ mod tests { ); } + /// Where `route`'s `origin` source fetches from. + fn location_of(route: &ProxyAssetRoute) -> crate::source::SourceLocation { + route + .origin_source() + .expect("the route should select the origin source") + .location() + .expect("the origin source should give a location") + } + + /// `route`'s `origin` source, as the registry builds it. + fn origin_source_of(route: &ProxyAssetRoute) -> crate::source::OriginSource { + crate::source::OriginSource::new( + &route + .origin_source() + .expect("the route should select the origin source"), + ) + .expect("should build the route's source") + } + #[test] - fn build_asset_proxy_target_url_preserves_path_and_query() { - let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com"); + fn a_route_copy_is_named_by_its_address_prefix_and_any_host_its_source_sets() { + let route = ProxyAssetRoute::new("/assets/", "https://assets.example.com"); + let location = location_of(&route); + assert_eq!( + route_origin_identity(&route, &location), + "https://assets.example.com\0/assets/", + "a source setting no Host should keep the key existing copies have" + ); + + let named = location.with_host_header(Some("www.example.com".to_owned())); + assert_eq!( + route_origin_identity(&route, &named), + "https://assets.example.com\0/assets/\0www.example.com", + "a source setting a Host should keep copies apart from one that does not" + ); + } + + #[test] + fn build_asset_proxy_target_url_puts_the_path_under_the_source_folder() { + let mut route = ProxyAssetRoute::new("/assets/", "https://storage.example.com/bucket/site"); let target_url = - build_asset_proxy_target_url(&route, "/.images/foo.jpg", "auto=webp&width=800") + build_asset_proxy_target_url(&route, &location_of(&route), "/assets/app.js", "v=1") .expect("should build asset target URL"); + assert_eq!( + target_url.as_str(), + "https://storage.example.com/bucket/site/assets/app.js?v=1", + "should send the path under the folder the source names" + ); + + route.path_pattern = Some(r"^/assets/(.*)$".to_string()); + route.target_path = Some("/static/$1".to_string()); + let target_url = + build_asset_proxy_target_url(&route, &location_of(&route), "/assets/app.js", "") + .expect("should build rewritten asset target URL"); + assert_eq!( + target_url.as_str(), + "https://storage.example.com/bucket/site/static/app.js", + "should put a rewritten path under the folder too" + ); + } + + #[test] + fn build_asset_proxy_target_url_keeps_requests_inside_the_source_folder() { + let route = ProxyAssetRoute::new("/assets/", "https://storage.example.com/bucket/site"); + for path in [ + "/assets/../../other/secret.json", + "/assets/%2e%2e/%2e%2e/other/secret.json", + "/assets/./app.js", + "/assets/%252e%252e/secret.json", + ] { + let err = build_asset_proxy_target_url(&route, &location_of(&route), path, "") + .expect_err("should refuse a path that could leave the folder"); + assert_eq!( + err.current_context().status_code(), + StatusCode::BAD_REQUEST, + "should answer `{path}` with 400" + ); + } + + let root = ProxyAssetRoute::new("/assets/", "https://assets.example.com"); + build_asset_proxy_target_url(&root, &location_of(&root), "/assets/./app.js", "") + .expect("a source with no folder has nothing to climb out of"); + } + + #[test] + fn build_asset_proxy_target_url_adds_the_source_index_to_a_folder_path() { + let mut route = ProxyAssetRoute::new("/docs/", "https://storage.example.com/bucket/site"); + route.set_origin_source(&crate::source::OriginSettings { + directory_index: Some("index.html".to_owned()), + ..crate::source::OriginSettings::new("https://storage.example.com/bucket/site") + }); + let location = location_of(&route); + for (path, expected) in [ + ( + "/docs/", + "https://storage.example.com/bucket/site/docs/index.html", + ), + ( + "/docs/guide/", + "https://storage.example.com/bucket/site/docs/guide/index.html", + ), + ( + "/docs/guide", + "https://storage.example.com/bucket/site/docs/guide", + ), + ] { + let target_url = build_asset_proxy_target_url(&route, &location, path, "") + .expect("should build asset target URL"); + assert_eq!(target_url.as_str(), expected, "the target of `{path}`"); + } + } + + #[test] + fn build_asset_proxy_target_url_preserves_path_and_query() { + let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com"); + let target_url = build_asset_proxy_target_url( + &route, + &location_of(&route), + "/.images/foo.jpg", + "auto=webp&width=800", + ) + .expect("should build asset target URL"); assert_eq!( target_url.as_str(), @@ -5010,6 +5160,7 @@ mod tests { route.target_path = Some("/image/upload/$1.$2".to_string()); let target_url = build_asset_proxy_target_url( &route, + &location_of(&route), "/.image/c_fit,w_1440/MjA/example.jpg", "auto=webp", ) @@ -5027,8 +5178,13 @@ mod tests { let mut route = ProxyAssetRoute::new("/_next/static/", "https://static-assets.example.com"); route.path_pattern = Some(r"^(.*)$".to_string()); route.target_path = Some("/_network$1".to_string()); - let target_url = build_asset_proxy_target_url(&route, "/_next/static/chunks/app.js", "") - .expect("should build rewritten static asset target URL"); + let target_url = build_asset_proxy_target_url( + &route, + &location_of(&route), + "/_next/static/chunks/app.js", + "", + ) + .expect("should build rewritten static asset target URL"); assert_eq!( target_url.as_str(), @@ -5042,7 +5198,7 @@ mod tests { let mut route = ProxyAssetRoute::new("/.image/", "https://assets.example.com"); route.path_pattern = Some(r"^/\.image/(.*)\.jpg$".to_string()); route.target_path = Some("/image/upload/$1.jpg".to_string()); - let err = build_asset_proxy_target_url(&route, "/.image/foo.png", "") + let err = build_asset_proxy_target_url(&route, &location_of(&route), "/.image/foo.png", "") .expect_err("should reject paths that do not match the configured rewrite"); assert!( @@ -5056,7 +5212,7 @@ mod tests { let mut route = ProxyAssetRoute::new("/assets/", "https://assets.example.com"); route.path_pattern = Some(r"^/assets/(.*)$".to_string()); route.target_path = Some("$1".to_string()); - let err = build_asset_proxy_target_url(&route, "/assets/app.js", "") + let err = build_asset_proxy_target_url(&route, &location_of(&route), "/assets/app.js", "") .expect_err("should reject rewritten paths without a leading slash"); assert!( @@ -5153,11 +5309,17 @@ mod tests { ) -> Response { let settings = create_test_settings(); let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com"); - handle_asset_proxy_request(&settings, services, request, &route) - .await - .expect("should answer the asset request") - .into_response() - .expect("should return a buffered asset response") + handle_asset_proxy_request( + &settings, + services, + request, + &route, + &origin_source_of(&route), + ) + .await + .expect("should answer the asset request") + .into_response() + .expect("should return a buffered asset response") } #[test] @@ -5382,11 +5544,17 @@ mod tests { ); let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com:8443"); - let response = handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy asset request") - .into_response() - .expect("should return buffered asset response"); + let response = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy asset request") + .into_response() + .expect("should return buffered asset response"); assert_eq!(response.status(), StatusCode::OK); assert_eq!( stub.recorded_stream_response_flags(), @@ -5472,9 +5640,15 @@ mod tests { let req = build_http_request(Method::GET, "https://www.example.com/.images/foo.jpg"); let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com"); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy streaming asset response"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy streaming asset response"); assert_eq!(stub.recorded_stream_response_flags(), vec![true]); }); @@ -5505,11 +5679,17 @@ mod tests { let req = build_http_request(Method::GET, "https://www.example.com/.images/foo.jpg"); let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com"); - let response = handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy asset request") - .into_response() - .expect("should return buffered asset response"); + let response = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy asset request") + .into_response() + .expect("should return buffered asset response"); assert!( response.headers().get(header::SET_COOKIE).is_none(), @@ -5568,9 +5748,15 @@ mod tests { ); let route = ProxyAssetRoute::new("/assets/", "https://assets.example.com"); - let asset_response = handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy asset request"); + let asset_response = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy asset request"); assert_eq!( asset_response.cache_policy(), AssetProxyCachePolicy::Normalized(CachePolicy::public_immutable( @@ -5675,9 +5861,15 @@ mod tests { let req = build_http_request(Method::GET, "https://www.example.com/assets/app.js"); let route = ProxyAssetRoute::new("/assets/", "https://assets.example.com"); - let asset_response = handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy asset request"); + let asset_response = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy asset request"); assert_eq!( asset_response.cache_policy(), @@ -5740,16 +5932,19 @@ mod tests { "/.images/", "https://examplebucket.s3.us-east-1.amazonaws.com", ); - route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "us-east-1".to_string(), - secret_store: None, - access_key_id: Redacted::new("AKIAIOSFODNN7EXAMPLE".to_string()), - secret_access_key: Redacted::new( - "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY".to_string(), - ), - session_token: None, - origin_query: None, - })); + set_route_auth( + &mut route, + Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "us-east-1".to_string(), + secret_store: None, + access_key_id: Redacted::new("AKIAIOSFODNN7EXAMPLE".to_string()), + secret_access_key: Redacted::new( + "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY".to_string(), + ), + session_token: None, + origin_query: None, + })), + ); route.image_optimizer = Some(AssetImageOptimizerConfig { enabled: true, region: "us_east".to_string(), @@ -5786,20 +5981,29 @@ mod tests { "/.images/", "https://examplebucket.s3.us-east-1.amazonaws.com", ); - route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "us-east-1".to_string(), - secret_store: None, - access_key_id: Redacted::new("AKIAIOSFODNN7EXAMPLE".to_string()), - secret_access_key: Redacted::new( - "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY".to_string(), - ), - session_token: None, - origin_query: Some(OriginQueryPolicy::Strip), - })); + set_route_auth( + &mut route, + Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "us-east-1".to_string(), + secret_store: None, + access_key_id: Redacted::new("AKIAIOSFODNN7EXAMPLE".to_string()), + secret_access_key: Redacted::new( + "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY".to_string(), + ), + session_token: None, + origin_query: Some(OriginQueryPolicy::Strip), + })), + ); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy signed S3 asset request"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy signed S3 asset request"); let uris = stub.recorded_request_uris(); assert_eq!( @@ -5852,20 +6056,29 @@ mod tests { "https://www.example.com/.images/foo.jpg?profile=medium", ); let mut route = test_s3_image_optimizer_route(); - route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "us-east-1".to_string(), - secret_store: None, - access_key_id: Redacted::new("AKIAIOSFODNN7EXAMPLE".to_string()), - secret_access_key: Redacted::new( - "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY".to_string(), - ), - session_token: Some(Redacted::new("session-token".to_string())), - origin_query: None, - })); + set_route_auth( + &mut route, + Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "us-east-1".to_string(), + secret_store: None, + access_key_id: Redacted::new("AKIAIOSFODNN7EXAMPLE".to_string()), + secret_access_key: Redacted::new( + "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY".to_string(), + ), + session_token: Some(Redacted::new("session-token".to_string())), + origin_query: None, + })), + ); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy optimized S3 asset request"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy optimized S3 asset request"); assert_eq!( stub.recorded_request_methods(), @@ -5915,9 +6128,15 @@ mod tests { origin_query: None, }); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy optimized asset request"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy optimized asset request"); let uris = stub.recorded_request_uris(); assert_eq!( @@ -5972,9 +6191,15 @@ mod tests { origin_query: None, }); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy SVG asset without Image Optimizer profile parsing"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy SVG asset without Image Optimizer profile parsing"); assert_eq!( stub.recorded_request_uris(), @@ -6017,9 +6242,15 @@ mod tests { origin_query: None, }); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy incoming SVG asset without Image Optimizer profile parsing"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy incoming SVG asset without Image Optimizer profile parsing"); assert_eq!( stub.recorded_request_uris(), @@ -6060,9 +6291,15 @@ mod tests { origin_query: None, }); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy rewritten SVG asset without Image Optimizer metadata"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy rewritten SVG asset without Image Optimizer metadata"); assert_eq!( stub.recorded_request_uris(), @@ -6093,11 +6330,17 @@ mod tests { ); let route = test_s3_image_optimizer_route(); - let response = handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy raw SVG asset through S3 route") - .into_response() - .expect("should return buffered asset response"); + let response = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy raw SVG asset through S3 route") + .into_response() + .expect("should return buffered asset response"); assert_eq!(response_body_string(response), "raw-svg"); assert_eq!( @@ -6139,11 +6382,16 @@ mod tests { ); let route = test_s3_image_optimizer_route(); - let (response, stream_body) = - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy optimized S3 asset request") - .into_response_and_body(); + let (response, stream_body) = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy optimized S3 asset request") + .into_response_and_body(); assert_eq!(response.status(), StatusCode::OK, "should proxy the asset"); let mut output = Vec::new(); @@ -6218,9 +6466,15 @@ mod tests { ); let route = test_s3_image_optimizer_route(); - let asset_response = handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should return raw S3 error response"); + let asset_response = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should return raw S3 error response"); assert_eq!( asset_response.cache_policy(), AssetProxyCachePolicy::NoStorePrivate, @@ -6290,11 +6544,16 @@ mod tests { ); let route = test_s3_image_optimizer_route(); - let (response, stream_body) = - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy debug S3 asset request") - .into_response_and_body(); + let (response, stream_body) = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy debug S3 asset request") + .into_response_and_body(); assert_eq!(response.status(), StatusCode::OK, "should proxy the asset"); let mut output = Vec::new(); @@ -6347,9 +6606,15 @@ mod tests { origin_query: None, }); - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy debug asset request"); + handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy debug asset request"); let options = stub.recorded_image_optimizer_options(); assert!( @@ -6369,11 +6634,16 @@ mod tests { let req = build_http_request(Method::GET, "https://www.example.com/.images/foo.jpg"); let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com"); - let (response, stream_body) = - handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy a streaming asset response") - .into_response_and_body(); + let (response, stream_body) = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy a streaming asset response") + .into_response_and_body(); assert_eq!(response.status(), StatusCode::OK); assert!( response.headers().get(header::SET_COOKIE).is_none(), @@ -6503,9 +6773,15 @@ mod tests { let settings = create_test_settings(); let req = build_http_request(Method::GET, "https://www.example.com/.images/foo.jpg"); let route = ProxyAssetRoute::new("/.images/", "https://assets.example.com"); - let asset_response = handle_asset_proxy_request(&settings, &services, req, &route) - .await - .expect("should proxy a streaming asset response"); + let asset_response = handle_asset_proxy_request( + &settings, + &services, + req, + &route, + &origin_source_of(&route), + ) + .await + .expect("should proxy a streaming asset response"); let err = asset_response .into_response() diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index f89973ed79..890351127a 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -92,9 +92,9 @@ use crate::response_privacy::{ use crate::rsc_flight::RscFlightUrlRewriter; use crate::settings::{ AUCTION_DEBUG_METADATA_ALLOWLIST, AUCTION_DEBUG_UPSTREAM_METADATA_KEYS, AssetOriginAuth, - AuctionDebugCommentFormat, AuctionDebugCommentOptions, AuctionDebugCommentVerbosity, Publisher, - Settings, + AuctionDebugCommentFormat, AuctionDebugCommentOptions, AuctionDebugCommentVerbosity, Settings, }; +use crate::source::{SourceLocation, SourceRequest}; use crate::source_cache::{Found, Lookup, SourceAddress, SourceCache, SourceVary}; use crate::streaming_processor::{ BodyStreamDecoder, BodyStreamEncoder, Compression, GzipDecodeReader, PipelineConfig, @@ -1867,9 +1867,9 @@ fn create_html_stream_processor( /// that is assembled and not yet encoded, and to the response, which is /// nothing when no entry covers the request. /// -/// The origin host is read from the settings. A page served from the store -/// keeps no record of it, and a serve middleware is told the same about a -/// page from the store as about one fetched for this request. +/// The origin host is the one the publisher's source answered for this +/// request, which the parameters of a page from the store carry as well as +/// those of one fetched for it. fn reader_serve_processor( params: &OwnedProcessResponseParams, settings: &Settings, @@ -1887,13 +1887,13 @@ fn reader_serve_processor( HTML_MEDIA_TYPE, ¶ms.request_path, ); - let origin_host = settings.publisher.origin_host(); + let origin_host = ¶ms.origin_host; create_serve_processor( &chain, &ReaderDocument { request_host: ¶ms.request_host, request_scheme: ¶ms.request_scheme, - origin_host: &origin_host, + origin_host, platform: integration_registry.platform(), request_state: ¶ms.request_state, max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, @@ -2639,7 +2639,7 @@ fn seam_script_for(params: &OwnedProcessResponseParams) -> String { /// leave through the seam, never through the cached head. fn build_template_assembly_params( entry: &crate::platform::TemplateEntry, - settings: &Settings, + location: &SourceLocation, request_host: &str, request_scheme: &str, price_granularity: PriceGranularity, @@ -2654,8 +2654,8 @@ fn build_template_assembly_params( seam_ad_slots: None, policy_headers: Vec::new(), content_encoding: entry.metadata.content_encoding.clone(), - origin_host: String::new(), - origin_url: settings.publisher.origin_url.clone(), + origin_host: location.host(), + origin_url: location.url().to_owned(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), // Set by the caller, with the rest of what belongs to the request. @@ -3699,12 +3699,13 @@ fn rewrite_origin_request( /// Builds the path and query of a publisher origin request. /// -/// With none of `[publisher]`'s path settings in use this is the reader's path -/// and query exactly as sent. A path in `origin_url` comes first, joined to -/// the reader's path with one `/`. `directory_index` is added to a path that -/// ends in `/`, and the query is left off when `origin_auth` strips it. +/// With none of the location's path settings in use this is the reader's +/// path and query exactly as sent. A path in the location's address comes +/// first, joined to the reader's path with one `/`. The directory index is +/// added to a path that ends in `/`, and the query is left off when the +/// location's signing strips it. fn publisher_origin_path_and_query<'a>( - publisher: &Publisher, + location: &SourceLocation, origin_path: &str, uri: &'a Uri, ) -> Cow<'a, str> { @@ -3715,11 +3716,8 @@ fn publisher_origin_path_and_query<'a>( .path_and_query() .map_or("/", http::uri::PathAndQuery::path); let folder = origin_path.trim_end_matches('/'); - let index = publisher - .directory_index - .as_deref() - .filter(|_| path.ends_with('/')); - let strip_query = publisher.origin_query_is_stripped(); + let index = location.directory_index().filter(|_| path.ends_with('/')); + let strip_query = location.query_is_stripped(); if folder.is_empty() && index.is_none() && !strip_query { return Cow::Borrowed(path_and_query); } @@ -3848,14 +3846,15 @@ fn rewritten_origin_redirect( /// /// A signed origin is only asked to read, so any method but `GET` or `HEAD` is /// refused with `405` rather than signed with the publisher's credentials. -/// When `origin_url` names a folder, a reader path with a `.` or `..` segment -/// in any spelling is refused with `400`, so no request climbs out of it. +/// When the location's address names a folder, a reader path with a `.` or +/// `..` segment in any spelling is refused with `400`, so no request climbs out +/// of it. fn refused_origin_request( - publisher: &Publisher, + location: &SourceLocation, origin_path: &str, req: &Request, ) -> Option { - if publisher.origin_auth.is_some() && !matches!(*req.method(), Method::GET | Method::HEAD) { + if location.auth().is_some() && !matches!(*req.method(), Method::GET | Method::HEAD) { return Some(StatusCode::METHOD_NOT_ALLOWED); } if !origin_path.trim_end_matches('/').is_empty() && path_has_dot_segment(req.uri().path()) { @@ -3868,7 +3867,7 @@ fn refused_origin_request( /// any of a bounded number of percent-decoding rounds. A backslash counts as a /// separator, and a path still changing after the last round counts as having /// one. -fn path_has_dot_segment(path: &str) -> bool { +pub(crate) fn path_has_dot_segment(path: &str) -> bool { crate::closed_paths::matches_any_decoding(path, |current| { current .split(['/', '\\']) @@ -5332,11 +5331,21 @@ pub async fn handle_publisher_request( // recomputed per delivery point. let permissions_json = ec_context.permissions().page_json(); - let parsed_origin = url::Url::parse(&settings.publisher.origin_url).change_context( - TrustedServerError::Proxy { - message: format!("Invalid origin_url: {}", settings.publisher.origin_url), - }, - )?; + // Where this request's content comes from, as the publisher's source + // answers. Everything below that names the origin reads this. + let source = integration_registry.publisher_source(settings)?; + let removed_prefix = req + .extensions() + .get::() + .map(|prefix| prefix.as_str().to_owned()); + let location = source.locate(&SourceRequest { + method: req.method(), + path: req.uri().path(), + query: req.uri().query().unwrap_or_default(), + headers: req.headers(), + removed_prefix: removed_prefix.as_deref(), + })?; + let parsed_origin = location.parsed().clone(); let origin_scheme = parsed_origin.scheme().to_string(); let origin_host_without_port = parsed_origin.host_str().ok_or_else(|| { Report::new(TrustedServerError::Proxy { @@ -5349,7 +5358,7 @@ pub async fn handle_publisher_request( scheme: origin_scheme.clone(), host: origin_host_without_port.to_string(), port: parsed_origin.port(), - host_header_override: settings.publisher.origin_host_header_override.clone(), + host_header_override: location.host_header_override().map(str::to_owned), certificate_check: settings.proxy.certificate_check, first_byte_timeout: DEFAULT_PUBLISHER_FIRST_BYTE_TIMEOUT, between_bytes_timeout: DEFAULT_PUBLISHER_FIRST_BYTE_TIMEOUT, @@ -5358,16 +5367,16 @@ pub async fn handle_publisher_request( .change_context(TrustedServerError::Proxy { message: "backend registration failed".to_string(), })?; - let origin_host = settings.publisher.origin_host(); - let origin_host_header = settings.publisher.origin_host_header(); - if let Some(status) = refused_origin_request(&settings.publisher, parsed_origin.path(), &req) { + let origin_host = location.host(); + let origin_host_header = location.host_header(); + if let Some(status) = refused_origin_request(&location, parsed_origin.path(), &req) { log::debug!("Publisher origin request refused before sending: {status}"); return Ok(PublisherResponse::Buffered(origin_refusal_response( status, )?)); } let origin_path_and_query = - publisher_origin_path_and_query(&settings.publisher, parsed_origin.path(), req.uri()); + publisher_origin_path_and_query(&location, parsed_origin.path(), req.uri()); let target_uri = format!("{origin_scheme}://{origin_host}{origin_path_and_query}") .parse::() .change_context(TrustedServerError::Proxy { @@ -5630,7 +5639,7 @@ pub async fn handle_publisher_request( .path_and_query() .map(|path_and_query| path_and_query.as_str().to_owned()) .unwrap_or_else(|| "/".to_owned()), - origin_identity: format!("{}\0{}", settings.publisher.origin_url, origin_host_header), + origin_identity: location.identity(), assembly_mode, vary_values: settings .creative_opportunities @@ -5661,14 +5670,14 @@ pub async fn handle_publisher_request( url: target_uri.to_string(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), - origin_identity: format!("{}\0{}", settings.publisher.origin_url, origin_host_header), + origin_identity: location.identity(), fingerprint: template_fingerprint(settings, integration_registry, services.public_prefix()), }); let mut template_cache_response_state = matches!(assembly_mode, AssemblyMode::Esi) .then_some(TemplateCacheResponseState::BypassRequest); rewrite_origin_request(&mut req, target_uri, &origin_host_header)?; // Signed last, once nothing else changes the request the origin is sent. - if let Some(auth) = settings.publisher.origin_auth.as_ref() { + if let Some(auth) = location.auth() { sign_publisher_origin_request(&mut req, auth)?; } @@ -5984,7 +5993,7 @@ pub async fn handle_publisher_request( let response = build_cached_template_response(&entry, reader_compression)?; let mut params = build_template_assembly_params( &entry, - settings, + &location, request_host, request_scheme, price_granularity, @@ -6007,7 +6016,7 @@ pub async fn handle_publisher_request( let module_context = ModuleContext::new(module_request) .with_settings(settings) .with_request_state(ec_context, services); - let stored_origin_host = settings.publisher.origin_host(); + let stored_origin_host = location.host(); let document = TextResponse { content_type: crate::middleware::HTML_MEDIA_TYPE, request_path: &request_path, @@ -6550,7 +6559,7 @@ pub async fn handle_publisher_request( policy_headers, content_encoding, origin_host, - origin_url: settings.publisher.origin_url.clone(), + origin_url: location.url().to_owned(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), request_path: request_path.clone(), @@ -9500,7 +9509,9 @@ mod tests { StubHttpClient, build_services_with_http_client, noop_services, noop_services_with_telemetry_sink, }; - use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + use crate::test_support::tests::{ + crate_test_settings_str, create_test_settings, edit_origin, origin_host, + }; use edgezero_core::body::Body as EdgeBody; use http::{Method, Request as HttpRequest, StatusCode, header}; use std::sync::Arc; @@ -11032,8 +11043,11 @@ mod tests { seam_ad_slots: None, policy_headers: Vec::new(), content_encoding: content_encoding.to_owned(), - origin_host: settings.publisher.origin_host(), - origin_url: settings.publisher.origin_url.clone(), + origin_host: origin_host(settings), + origin_url: settings + .origin_source() + .expect("settings should select the origin source") + .url, request_host: settings.publisher.domain.clone(), request_scheme: "https".to_owned(), request_path: "/".to_owned(), @@ -11233,17 +11247,25 @@ mod tests { settings: &Settings, services: &RuntimeServices, req: Request, + ) -> PublisherResponse { + run_publisher_proxy_with(settings, &test_registry(settings), services, req).await + } + + async fn run_publisher_proxy_with( + settings: &Settings, + registry: &IntegrationRegistry, + services: &RuntimeServices, + req: Request, ) -> PublisherResponse { let orchestrator = AuctionOrchestrator::new( crate::auction::test_support::legacy_auction_config(settings), ); let mut ec_context = EcContext::read_from_request(settings, &req, services).expect("should read EC context"); - let registry = test_registry(settings); handle_publisher_request( AppContext { settings, - integration_registry: ®istry, + integration_registry: registry, }, services, None, @@ -11260,6 +11282,160 @@ mod tests { .expect("should proxy publisher request") } + mod source_tests { + //! The publisher path asks its source where each request's content + //! comes from. + + use super::*; + use crate::integrations::{IntegrationBuilder, IntegrationRegistration}; + use crate::source::{Source, SourceFactory, SourceLocation, SourceRequest}; + + /// The module whose registration offers the source, which a + /// section selects to run it. + const ADDRESSED_MODULE: &str = "testing.addressed"; + + /// The source's own name, which `[source]` selects. + const ADDRESSED_SOURCE: &str = "testing.addressed.source"; + + /// A source answering the origin named by the prefix request + /// addressing removed, as a source reading the address does. + #[derive(Debug)] + struct AddressedSource; + + impl Source for AddressedSource { + fn locate( + &self, + request: &SourceRequest<'_>, + ) -> Result> { + let name = request + .removed_prefix + .unwrap_or("/default.example") + .trim_start_matches('/'); + SourceLocation::new(format!("https://{name}")) + } + } + + struct AddressedFactory; + + impl SourceFactory for AddressedFactory { + fn build( + &self, + _settings: &serde_json::Map, + _setting: &str, + ) -> Result, Report> { + Ok(Arc::new(AddressedSource)) + } + } + + fn addressed_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("addressed") + .with_source_module(ADDRESSED_SOURCE, Arc::new(AddressedFactory)) + .build(), + )) + } + + fn addressed_builder() -> IntegrationBuilder { + IntegrationBuilder::new( + "addressed", + "seam-probe", + addressed_registration, + |_settings: &Settings| Ok(true), + ) + .with_module_name(ADDRESSED_MODULE) + } + + /// Settings whose `[source]` selects the addressed source. + fn addressed_settings() -> Settings { + let mut settings = create_test_settings(); + settings + .insert_module_config("testing", ADDRESSED_MODULE, &serde_json::json!({})) + .expect("should run the test module"); + settings.source = crate::provider_table::SectionModules::one( + ADDRESSED_SOURCE, + serde_json::Map::new(), + ); + settings + } + + fn addressed_request(prefix: &str) -> HttpRequest { + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build reader request"); + request + .extensions_mut() + .insert(crate::addressing::RemovedPathPrefix::new(prefix.to_owned())); + request + } + + #[tokio::test] + async fn a_source_answering_per_request_sends_each_request_to_its_own_origin() { + let settings = addressed_settings(); + let registry = + IntegrationRegistry::with_registrations(&settings, &[addressed_builder()]) + .expect("should build the registry offering the test source"); + let stub = Arc::new(StubHttpClient::new()); + for _ in 0..2 { + stub.push_response_with_headers( + 200, + b"page".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + } + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + let mut origin_hosts = Vec::new(); + for prefix in ["/a.example", "/b.example"] { + let response = run_publisher_proxy_with( + &settings, + ®istry, + &services, + addressed_request(prefix), + ) + .await; + let PublisherResponse::Stream { params, .. } = response else { + panic!("should stream the origin page through the page pipeline"); + }; + origin_hosts.push(params.origin_host.clone()); + } + + assert_eq!( + stub.recorded_request_uris(), + vec!["https://a.example/page", "https://b.example/page"], + "each request should go to the origin its source answered" + ); + assert_eq!( + origin_hosts, + vec!["a.example", "b.example"], + "the page pipeline should be told each request's own origin host" + ); + } + + #[test] + fn a_source_the_build_does_not_offer_is_refused_naming_those_it_does() { + let mut settings = create_test_settings(); + settings.source = + crate::provider_table::SectionModules::one("nosuch", serde_json::Map::new()); + + let Err(error) = IntegrationRegistry::new(&settings) else { + panic!("should refuse a source no module supplies"); + }; + + let message = format!("{error:?}"); + assert!( + message.contains("nosuch") && message.contains("origin"), + "should name the selection and the sources on offer: {message}" + ); + } + } + mod rendered_template_identity_tests { //! The gate the plan's Task 3 Step 2 actually asks for. //! @@ -11884,7 +12060,9 @@ mod tests { assert_ne!(fingerprint(&base), fingerprint(&creative)); let mut origin = base.clone(); - origin.publisher.origin_host_header_override = Some("tenant.example.com".to_string()); + edit_origin(&mut origin, |settings| { + settings.host_header = Some("tenant.example.com".to_string()) + }); assert_ne!(fingerprint(&base), fingerprint(&origin)); } @@ -13206,7 +13384,7 @@ mod tests { /// running `settings`, for a request that left a mark or did not. fn reader_marker_for(settings: &Settings, marked: bool) -> String { crate::integrations::registry_test_support::middleware_fixture::reader_marker( - &settings.publisher.origin_host(), + &origin_host(settings), marked, ) } @@ -15457,7 +15635,9 @@ mod tests { #[tokio::test] async fn two_pass_integration_templates_are_identical_across_request_shapes() { let mut settings = settings_with_mode("esi"); - settings.publisher.origin_url = "https://origin.example.com".to_owned(); + edit_origin(&mut settings, |origin| { + origin.url = "https://origin.example.com".to_owned() + }); crate::integrations::registry_test_support::payload_fixture::select_and_place( &mut settings, ); @@ -21970,20 +22150,6 @@ mod tests { ); } - #[test] - fn test_publisher_origin_host_extraction() { - let settings = create_test_settings(); - let origin_host = settings.publisher.origin_host(); - assert_eq!(origin_host, "origin.test-publisher.com"); - - let mut settings_with_port = create_test_settings(); - settings_with_port.publisher.origin_url = "origin.test-publisher.com:8080".to_string(); - assert_eq!( - settings_with_port.publisher.origin_host(), - "origin.test-publisher.com:8080" - ); - } - #[test] fn test_invalid_utf8_handling() { let invalid_utf8_bytes = vec![0xFF, 0xFE, 0xFD]; @@ -23891,7 +24057,9 @@ mod tests { #[tokio::test] async fn publisher_request_sends_configured_host_header_override() { let mut settings = create_test_settings(); - settings.publisher.origin_host_header_override = Some("www.example.com".to_string()); + edit_origin(&mut settings, |origin| { + origin.host_header = Some("www.example.com".to_string()) + }); let stub = Arc::new(StubHttpClient::new()); stub.push_response(200, b"origin response".to_vec()); let services = build_services_with_http_client( @@ -24104,9 +24272,14 @@ mod tests { ); } + /// A location with none of the path settings in use. + fn plain_location() -> SourceLocation { + SourceLocation::new("https://origin.example.com").expect("should accept the address") + } + #[test] fn origin_path_and_query_without_the_new_settings_is_the_reader_s_own() { - let publisher = Publisher::default(); + let location = plain_location(); for request in [ "/", "/page", @@ -24119,7 +24292,7 @@ mod tests { ] { let uri: Uri = request.parse().expect("should parse reader path"); for origin_path in ["", "/"] { - let built = publisher_origin_path_and_query(&publisher, origin_path, &uri); + let built = publisher_origin_path_and_query(&location, origin_path, &uri); assert!( matches!(built, Cow::Borrowed(_)), "should reuse the reader's own path for `{request}`" @@ -24137,7 +24310,7 @@ mod tests { #[tokio::test] async fn origin_path_is_kept_and_joined_with_one_slash() { let mut settings = create_test_settings(); - settings.publisher.origin_url = STORAGE_URL.to_owned(); + edit_origin(&mut settings, |origin| origin.url = STORAGE_URL.to_owned()); let sent = origin_uris( &settings, @@ -24164,7 +24337,7 @@ mod tests { #[test] fn origin_path_join_has_one_slash_whichever_side_carries_one() { - let publisher = Publisher::default(); + let location = plain_location(); for (origin_path, request, expected) in [ ("/bucket/site", "/page", "/bucket/site/page"), ("/bucket/site/", "/page", "/bucket/site/page"), @@ -24177,7 +24350,7 @@ mod tests { ] { let uri: Uri = request.parse().expect("should parse reader path"); assert_eq!( - publisher_origin_path_and_query(&publisher, origin_path, &uri), + publisher_origin_path_and_query(&location, origin_path, &uri), expected, "should join `{origin_path}` and `{request}` with one slash" ); @@ -24186,10 +24359,7 @@ mod tests { #[test] fn directory_index_is_added_only_to_a_path_ending_in_a_slash() { - let publisher = Publisher { - directory_index: Some("index.html".to_owned()), - ..Publisher::default() - }; + let location = plain_location().with_directory_index(Some("index.html".to_owned())); for (origin_path, request, expected) in [ ("", "/", "/index.html"), ("", "/news/", "/news/index.html"), @@ -24203,7 +24373,7 @@ mod tests { ] { let uri: Uri = request.parse().expect("should parse reader path"); assert_eq!( - publisher_origin_path_and_query(&publisher, origin_path, &uri), + publisher_origin_path_and_query(&location, origin_path, &uri), expected, "should give `{request}` under `{origin_path}` the expected origin path" ); @@ -24213,7 +24383,9 @@ mod tests { #[tokio::test] async fn directory_index_changes_the_origin_address_and_not_the_reader_s() { let mut settings = create_test_settings(); - settings.publisher.directory_index = Some("index.html".to_owned()); + edit_origin(&mut settings, |origin| { + origin.directory_index = Some("index.html".to_owned()) + }); let sent = origin_uris( &settings, @@ -24261,9 +24433,13 @@ mod tests { #[tokio::test] async fn signed_origin_request_carries_none_of_the_reader_s_private_headers() { let mut settings = create_test_settings(); - settings.publisher.origin_url = STORAGE_URL.to_owned(); - settings.publisher.directory_index = Some("index.html".to_owned()); - settings.publisher.origin_auth = Some(signing_auth(None)); + edit_origin(&mut settings, |origin| origin.url = STORAGE_URL.to_owned()); + edit_origin(&mut settings, |origin| { + origin.directory_index = Some("index.html".to_owned()) + }); + edit_origin(&mut settings, |origin| { + origin.auth = Some(signing_auth(None)) + }); let stub = stub_with_responses(1); let services = services_for(&stub); @@ -24351,8 +24527,10 @@ mod tests { #[tokio::test] async fn signed_head_request_is_signed_as_a_head() { let mut settings = create_test_settings(); - settings.publisher.origin_url = STORAGE_URL.to_owned(); - settings.publisher.origin_auth = Some(signing_auth(None)); + edit_origin(&mut settings, |origin| origin.url = STORAGE_URL.to_owned()); + edit_origin(&mut settings, |origin| { + origin.auth = Some(signing_auth(None)) + }); let stub = stub_with_responses(1); let services = services_for(&stub); @@ -24370,8 +24548,10 @@ mod tests { #[tokio::test] async fn signed_origin_strips_the_query_when_origin_auth_says_so() { let mut settings = create_test_settings(); - settings.publisher.origin_url = STORAGE_URL.to_owned(); - settings.publisher.origin_auth = Some(signing_auth(Some(OriginQueryPolicy::Strip))); + edit_origin(&mut settings, |origin| origin.url = STORAGE_URL.to_owned()); + edit_origin(&mut settings, |origin| { + origin.auth = Some(signing_auth(Some(OriginQueryPolicy::Strip))) + }); let stub = stub_with_responses(1); let services = services_for(&stub); @@ -24394,7 +24574,9 @@ mod tests { #[tokio::test] async fn signed_origin_refuses_a_method_that_is_not_a_read() { let mut settings = create_test_settings(); - settings.publisher.origin_auth = Some(signing_auth(None)); + edit_origin(&mut settings, |origin| { + origin.auth = Some(signing_auth(None)) + }); for method in [Method::POST, Method::PUT, Method::DELETE, Method::OPTIONS] { let stub = stub_with_responses(1); let services = services_for(&stub); @@ -24475,7 +24657,7 @@ mod tests { #[tokio::test] async fn origin_path_refuses_a_reader_path_that_climbs_out_of_it() { let mut settings = create_test_settings(); - settings.publisher.origin_url = STORAGE_URL.to_owned(); + edit_origin(&mut settings, |origin| origin.url = STORAGE_URL.to_owned()); for path in [ "/../other-site/page", "/news/../../other-site/page", @@ -24920,7 +25102,7 @@ mod tests { fn settings_with_origin(origin_url: &str) -> Settings { let mut settings = create_test_settings(); - settings.publisher.origin_url = origin_url.to_owned(); + edit_origin(&mut settings, |origin| origin.url = origin_url.to_owned()); settings } @@ -25116,7 +25298,9 @@ mod tests { #[tokio::test] async fn the_host_header_override_is_the_origin_s_name_when_set() { let mut settings = settings_with_origin(ORIGIN_URL); - settings.publisher.origin_host_header_override = Some("www.example.com".to_owned()); + edit_origin(&mut settings, |origin| { + origin.host_header = Some("www.example.com".to_owned()) + }); // (the origin's `Location`, what the reader receives) let cases = [ ( diff --git a/crates/trusted-server-core/src/response_privacy.rs b/crates/trusted-server-core/src/response_privacy.rs index 301b8f799f..fc7105b2ef 100644 --- a/crates/trusted-server-core/src/response_privacy.rs +++ b/crates/trusted-server-core/src/response_privacy.rs @@ -234,10 +234,15 @@ mod tests { fn settings_with_response_headers(headers: &[(&str, &str)]) -> Settings { let mut s = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "unit-test-proxy-secret" [geo] diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 63f7c0071b..de58560579 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -45,13 +45,21 @@ pub struct Publisher { /// (see [`ec_cookie_domain`](Self::ec_cookie_domain)). #[validate(custom(function = validate_cookie_domain))] pub cookie_domain: String, - #[serde(serialize_with = "crate::redacted::sensitive")] - #[validate(custom(function = validate_no_trailing_slash))] - pub origin_url: String, - /// Optional outbound Host header to send while connecting to `origin_url`. - #[serde(default, serialize_with = "crate::redacted::sensitive")] - #[validate(custom(function = validate_host_header_override))] - pub origin_host_header_override: Option, + /// Moved to the publisher's source, kept so a document carrying it is + /// refused with directions. + #[serde(default, skip_serializing)] + #[allow( + dead_code, + reason = "the field exists so that reading the moved setting fails with directions" + )] + origin_url: MovedOriginUrl, + /// Moved to the publisher's source, as for `origin_url`. + #[serde(default, skip_serializing)] + #[allow( + dead_code, + reason = "the field exists so that reading the moved setting fails with directions" + )] + origin_host_header_override: MovedOriginHostHeader, /// Secret used to encrypt/decrypt proxied URLs in `/first-party/proxy`. /// Keep this secret stable to allow existing links to decode. #[validate(custom(function = validate_redacted_not_empty))] @@ -85,15 +93,20 @@ pub struct Publisher { #[serde(default = "default_max_buffered_body_bytes")] #[validate(range(min = 1, message = "must be at least 1 byte"))] pub max_buffered_body_bytes: usize, - /// Optional signing of publisher origin reads, taking an asset route's `auth` table. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub origin_auth: Option, - /// Optional file name added to a request path that ends in `/` on the way - /// to the origin, for an origin such as object storage that does not answer - /// a folder with its index file. The reader's address is unchanged. - #[serde(default, skip_serializing_if = "Option::is_none")] - #[validate(custom(function = validate_directory_index))] - pub directory_index: Option, + /// Moved to the publisher's source, as for `origin_url`. + #[serde(default, skip_serializing)] + #[allow( + dead_code, + reason = "the field exists so that reading the moved setting fails with directions" + )] + origin_auth: MovedOriginAuth, + /// Moved to the publisher's source, as for `origin_url`. + #[serde(default, skip_serializing)] + #[allow( + dead_code, + reason = "the field exists so that reading the moved setting fails with directions" + )] + directory_index: MovedDirectoryIndex, } fn default_max_buffered_body_bytes() -> usize { @@ -110,12 +123,12 @@ impl Default for Publisher { Self { domain: String::default(), cookie_domain: String::default(), - origin_url: String::default(), - origin_host_header_override: None, + origin_url: MovedOriginUrl, + origin_host_header_override: MovedOriginHostHeader, proxy_secret: Redacted::default(), max_buffered_body_bytes: default_max_buffered_body_bytes(), - origin_auth: None, - directory_index: None, + origin_auth: MovedOriginAuth, + directory_index: MovedDirectoryIndex, } } } @@ -192,88 +205,24 @@ impl Publisher { .any(|p| p.eq_ignore_ascii_case(&host)) }) } - - /// Extracts the host (including port if present) from the `origin_url`. - /// - /// # Examples - /// - /// ``` - /// # use trusted_server_core::settings::Publisher; - /// # use trusted_server_core::redacted::Redacted; - /// let publisher = Publisher { - /// domain: "example.com".to_string(), - /// cookie_domain: ".example.com".to_string(), - /// origin_url: "https://origin.example.com:8080".to_string(), - /// origin_host_header_override: None, - /// proxy_secret: Redacted::new("proxy-secret".to_string()), - /// max_buffered_body_bytes: 16 * 1024 * 1024, - /// origin_auth: None, - /// directory_index: None, - /// }; - /// assert_eq!(publisher.origin_host(), "origin.example.com:8080"); - /// ``` - #[allow(dead_code)] - #[must_use] - pub fn origin_host(&self) -> String { - Url::parse(&self.origin_url) - .ok() - .and_then(|url| { - url.host_str().map(|host| match url.port() { - Some(port) => format!("{}:{}", host, port), - None => host.to_string(), - }) - }) - .unwrap_or_else(|| self.origin_url.clone()) - } - - /// Returns the outbound Host header for proxied publisher-origin requests. - #[must_use] - pub fn origin_host_header(&self) -> String { - self.origin_host_header_override - .clone() - .unwrap_or_else(|| self.origin_host()) - } - - /// Returns `true` when [`Self::origin_auth`] says the reader's query is - /// left off origin requests, as an asset route's `origin_query` does. - #[must_use] - pub fn origin_query_is_stripped(&self) -> bool { - self.origin_auth - .as_ref() - .and_then(AssetOriginAuth::origin_query_policy) - == Some(OriginQueryPolicy::Strip) - } - - fn normalize(&mut self) { - if let Some(auth) = &mut self.origin_auth { - auth.normalize(); - } - } - - /// Refuses an `origin_auth` the signer cannot use, naming the setting. - /// - /// As for an asset route, a signed origin must use `https`. - fn prepare_runtime(&self) -> Result<(), Report> { - let Some(auth) = &self.origin_auth else { - return Ok(()); - }; - let uses_https = Url::parse(&self.origin_url).is_ok_and(|url| url.scheme() == "https"); - if !uses_https { - return Err(Report::new(TrustedServerError::Configuration { - message: format!( - "{PUBLISHER_ORIGIN_AUTH_SETTING} requires publisher.origin_url to use https" - ), - })); - } - auth.prepare_runtime(PUBLISHER_ORIGIN_AUTH_SETTING) - } } -/// The setting named in every refusal of `[publisher]`'s `origin_auth`. -const PUBLISHER_ORIGIN_AUTH_SETTING: &str = "publisher.origin_auth"; +/// `value` as a `Host` header a source may send, or why it cannot be one. +pub(crate) fn check_host_header(value: &str) -> Result<(), String> { + validate_host_header_override_value(value) + .map_err(|reason| format!("must be a host or host:port alone, but {reason}")) +} -/// The setting named in every refusal of an asset route's `auth`. -const ASSET_ROUTE_AUTH_SETTING: &str = "proxy.asset_routes auth"; +/// `value` as one file name a source may add to a folder's path, or why it +/// cannot be one. +pub(crate) fn check_directory_index(value: &str) -> Result<(), String> { + validate_directory_index(value).map_err(|error| { + error.message.map_or_else( + || "is not one file name".to_owned(), + |message| message.to_string(), + ) + }) +} /// Which integrations run, and the settings each one is given. /// @@ -1905,14 +1854,14 @@ pub enum AssetOriginAuth { } impl AssetOriginAuth { - fn normalize(&mut self) { + pub(crate) fn normalize(&mut self) { match self { Self::S3SigV4(config) => config.normalize(), } } /// Refuses a table the signer cannot use, naming `setting` in the message. - fn prepare_runtime(&self, setting: &str) -> Result<(), Report> { + pub(crate) fn prepare_runtime(&self, setting: &str) -> Result<(), Report> { match self { Self::S3SigV4(config) => config.prepare_runtime(setting), } @@ -2585,13 +2534,22 @@ pub struct ProxyAssetRoute { /// a trailing `/` unless you intentionally want `/static` to match paths such /// as `/staticfile.js`. pub prefix: String, - /// Absolute `http` or `https` origin used for upstream requests. - /// - /// Only the scheme, host, and port are used. Any path or query configured on - /// this URL is rejected because the incoming request path/query, or the - /// configured rewrite result, replaces them at runtime. - #[serde(serialize_with = "crate::redacted::sensitive")] - pub origin_url: String, + /// The source the route's content comes from, in the shape `[source]` + /// has: `module` naming it and its settings in the table beneath. + #[serde( + default, + skip_serializing_if = "SectionModules::is_unset", + serialize_with = "crate::source::serialize_selection" + )] + pub source: SectionModules, + /// Moved to the route's source, kept so a document carrying it is refused + /// with directions. + #[serde(default, skip_serializing)] + #[allow( + dead_code, + reason = "the field exists so that reading the moved setting fails with directions" + )] + origin_url: MovedRouteOriginUrl, /// Optional regex matched against the incoming request path before proxying. pub path_pattern: Option, /// Optional regex replacement used with [`Self::path_pattern`] to build the upstream path. @@ -2599,9 +2557,13 @@ pub struct ProxyAssetRoute { /// Must be configured together with [`Self::path_pattern`] and must produce a /// path that starts with `/`. pub target_path: Option, - /// Optional origin authentication configuration. - #[serde(default)] - pub auth: Option, + /// Moved to the route's source, as for `origin_url`. + #[serde(default, skip_serializing)] + #[allow( + dead_code, + reason = "the field exists so that reading the moved setting fails with directions" + )] + auth: MovedRouteAuth, /// Optional Image Optimizer configuration. #[serde(default)] pub image_optimizer: Option, @@ -2610,19 +2572,40 @@ pub struct ProxyAssetRoute { } impl ProxyAssetRoute { - /// Create an asset route with the required prefix and origin URL. + /// An asset route at `prefix` whose source is core's `origin` at + /// `origin_url`. #[must_use] pub fn new(prefix: impl Into, origin_url: impl Into) -> Self { Self { prefix: prefix.into(), - origin_url: origin_url.into(), + source: crate::source::OriginSettings::new(origin_url).selection(), ..Self::default() } } + /// Sets the route's source to core's `origin` with `origin`. + pub fn set_origin_source(&mut self, origin: &crate::source::OriginSettings) { + self.source = origin.selection(); + } + + /// The route's `origin` source settings as written, when its source is + /// `origin`. [`Self::prepare_runtime`] checks them. + #[must_use] + pub fn origin_source(&self) -> Option { + let (name, table) = crate::source::selected(&self.source, "source").ok()?; + (name == crate::source::ORIGIN) + .then(|| crate::source::OriginSettings::parse(&table, "source.origin").ok()) + .flatten() + } + + /// The setting named in a refusal of this route's source. + #[must_use] + pub fn source_setting(&self) -> String { + format!("proxy.asset_routes {} source", self.prefix) + } + fn normalize(&mut self) { self.prefix = self.prefix.trim().to_string(); - self.origin_url = self.origin_url.trim().to_string(); self.path_pattern = self .path_pattern .take() @@ -2633,9 +2616,6 @@ impl ProxyAssetRoute { .take() .map(|value| value.trim().to_string()) .filter(|value| !value.is_empty()); - if let Some(auth) = &mut self.auth { - auth.normalize(); - } if let Some(image_optimizer) = &mut self.image_optimizer { image_optimizer.normalize(); } @@ -2724,33 +2704,7 @@ impl ProxyAssetRoute { }) })?; - validate_proxy_origin_url(&self.origin_url).map_err(|err| { - Report::new(TrustedServerError::Configuration { - message: format!( - "proxy.asset_routes origin_url `{}` is invalid: {err}", - self.origin_url - ), - }) - })?; - - if matches!(&self.auth, Some(AssetOriginAuth::S3SigV4(_))) { - let parsed_origin = Url::parse(&self.origin_url).map_err(|err| { - Report::new(TrustedServerError::Configuration { - message: format!( - "proxy.asset_routes origin_url `{}` is invalid: {err}", - self.origin_url - ), - }) - })?; - if parsed_origin.scheme() != "https" { - return Err(Report::new(TrustedServerError::Configuration { - message: format!( - "proxy.asset_routes origin_url `{}` must use https when auth type is s3_sigv4", - self.origin_url - ), - })); - } - } + crate::source::check_selection(&self.source, &self.source_setting())?; match (&self.path_pattern, &self.target_path) { (None, None) | (Some(_), Some(_)) => {} @@ -2764,9 +2718,6 @@ impl ProxyAssetRoute { } } - if let Some(auth) = &self.auth { - auth.prepare_runtime(ASSET_ROUTE_AUTH_SETTING)?; - } if let Some(image_optimizer) = &self.image_optimizer { image_optimizer.prepare_runtime()?; } @@ -2799,11 +2750,22 @@ impl ProxyAssetRoute { /// enabled Image Optimizer routes and `preserve` otherwise. #[must_use] pub fn origin_query_policy(&self) -> OriginQueryPolicy { - if let Some(policy) = self - .auth - .as_ref() - .and_then(AssetOriginAuth::origin_query_policy) - { + self.query_policy_with(self.origin_source().and_then(|origin| origin.auth).as_ref()) + } + + /// The effective origin query policy for a request to `location`, as + /// [`Self::origin_query_policy`] gives it, with the signing the location + /// carries. + #[must_use] + pub fn origin_query_policy_at( + &self, + location: &crate::source::SourceLocation, + ) -> OriginQueryPolicy { + self.query_policy_with(location.auth()) + } + + fn query_policy_with(&self, auth: Option<&AssetOriginAuth>) -> OriginQueryPolicy { + if let Some(policy) = auth.and_then(AssetOriginAuth::origin_query_policy) { return policy; } if let Some(policy) = self @@ -2939,20 +2901,29 @@ impl Proxy { /// Resolve the longest matching asset route for the given request path. #[must_use] pub fn asset_route_for_path(&self, path: &str) -> Option<&ProxyAssetRoute> { - let mut best_match: Option<&ProxyAssetRoute> = None; + self.asset_route_index_for_path(path) + .map(|index| &self.asset_routes[index]) + } - for route in &self.asset_routes { + /// The position in `asset_routes` of the route + /// [`Self::asset_route_for_path`] gives, which names the route's source + /// to [`IntegrationRegistry::route_source`](crate::integrations::IntegrationRegistry::route_source). + #[must_use] + pub fn asset_route_index_for_path(&self, path: &str) -> Option { + let mut best_match: Option<(usize, usize)> = None; + + for (index, route) in self.asset_routes.iter().enumerate() { if !path.starts_with(&route.prefix) { continue; } match best_match { - Some(current) if current.prefix.len() >= route.prefix.len() => {} - _ => best_match = Some(route), + Some((_, length)) if length >= route.prefix.len() => {} + _ => best_match = Some((index, route.prefix.len())), } } - best_match + best_match.map(|(index, _)| index) } } @@ -4034,6 +4005,43 @@ macro_rules! refused_table { }; } +refused_table! { + /// `[publisher] origin_url`, moved to the publisher's source. + MovedOriginUrl => "`[publisher] origin_url` has moved to the publisher's source. Write \ + `[source] module = \"origin\"` and the address as `[source.origin] url`" +} + +refused_table! { + /// `[publisher] origin_host_header_override`, moved to the publisher's source. + MovedOriginHostHeader => "`[publisher] origin_host_header_override` has moved to the \ + publisher's source, as `[source.origin] host_header`" +} + +refused_table! { + /// `[publisher] origin_auth`, moved to the publisher's source. + MovedOriginAuth => "`[publisher] origin_auth` has moved to the publisher's source, as \ + `[source.origin] auth`" +} + +refused_table! { + /// `[publisher] directory_index`, moved to the publisher's source. + MovedDirectoryIndex => "`[publisher] directory_index` has moved to the publisher's \ + source, as `[source.origin] directory_index`" +} + +refused_table! { + /// An asset route's `origin_url`, moved to the route's source. + MovedRouteOriginUrl => "`origin_url` of a `[[proxy.asset_routes]]` entry has moved to \ + the route's source. Write `[proxy.asset_routes.source] module = \"origin\"` and the \ + address as `[proxy.asset_routes.source.origin] url`" +} + +refused_table! { + /// An asset route's `auth`, moved to the route's source. + MovedRouteAuth => "`auth` of a `[[proxy.asset_routes]]` entry has moved to the route's \ + source, as `[proxy.asset_routes.source.origin] auth`" +} + refused_table! { /// The `[adserver]` table, renamed `[ad-server]`. RenamedAdServerTable => "Configuration table `[adserver]` is now `[ad-server]`. Select the \ @@ -4368,6 +4376,14 @@ fn validate_trusted_client_ip_shared_secret( pub struct Settings { #[validate(nested)] pub publisher: Publisher, + /// The source the publisher's pages come from: `module` naming it and its + /// settings in the table beneath, `[source.origin]` for core's `origin`. + #[serde( + default, + skip_serializing_if = "SectionModules::is_unset", + serialize_with = "crate::source::serialize_selection" + )] + pub source: SectionModules, #[serde(default)] #[validate(nested)] pub tester_cookie: TesterCookieConfig, @@ -4646,7 +4662,6 @@ impl Settings { /// Returns a configuration error if image optimizer keys collide after trimming. pub(crate) fn normalize_deserialized(&mut self) -> Result<(), Report> { self.cache.normalize(); - self.publisher.normalize(); self.proxy.normalize(); self.image_optimizer.normalize()?; self.debug.auction_html_comment_options.normalize(); @@ -4704,19 +4719,35 @@ impl Settings { Ok(settings) } + /// The publisher's `origin` source settings as written, when `[source]` + /// selects core's `origin`. [`Self::prepare_runtime`] checks them. + #[must_use] + pub fn origin_source(&self) -> Option { + let (name, table) = + crate::source::selected(&self.source, crate::source::SOURCE_SECTION).ok()?; + (name == crate::source::ORIGIN) + .then(|| crate::source::OriginSettings::parse(&table, "source.origin").ok()) + .flatten() + } + + /// Sets the publisher's source to core's `origin` with `origin`. + pub fn set_origin_source(&mut self, origin: &crate::source::OriginSettings) { + self.source = origin.selection(); + } + /// Eagerly prepare runtime-only settings artifacts. /// /// # Errors /// /// Returns a configuration error if any cached runtime artifact cannot be /// prepared, if any handler path regex does not compile, if a creative - /// opportunity slot is invalid, if `publisher.origin_auth` cannot sign, or + /// opportunity slot is invalid, if the publisher's source cannot be used, or /// if [`AuctionDebugCommentOptions::metadata_keys`] names an unsupported /// key. pub fn prepare_runtime(&mut self) -> Result<(), Report> { self.image_optimizer.prepare_runtime()?; self.cache.prepare_runtime()?; - self.publisher.prepare_runtime()?; + crate::source::check_selection(&self.source, crate::source::SOURCE_SECTION)?; self.proxy.prepare_runtime()?; if let Some(analytics) = &mut self.analytics { analytics.prepare_runtime()?; @@ -4886,8 +4917,11 @@ impl Settings { if Publisher::is_placeholder_cookie_domain(&self.publisher.cookie_domain) { insecure_fields.push("publisher.cookie_domain".to_owned()); } - if Publisher::is_placeholder_origin_url(&self.publisher.origin_url) { - insecure_fields.push("publisher.origin_url".to_owned()); + if self + .origin_source() + .is_some_and(|origin| origin.is_placeholder()) + { + insecure_fields.push("source.origin.url".to_owned()); } if insecure_fields.is_empty() { @@ -4943,12 +4977,19 @@ impl Settings { self.proxy.asset_route_for_path(path) } + /// The position of the asset route [`Self::asset_route_for_path`] gives. + #[must_use] + pub fn asset_route_index_for_path(&self, path: &str) -> Option { + self.proxy.asset_route_index_for_path(path) + } + /// Every section that selects modules, with its name: `[proxy]`, /// `[auction]` and each module type's own section. pub fn module_sections(&self) -> impl Iterator { [ ("proxy", &self.proxy.modules), ("auction", &self.auction.modules), + (crate::source::SOURCE_SECTION, &self.source), ] .into_iter() .chain(self.sections.iter()) @@ -5217,31 +5258,6 @@ fn validate_cookie_domain(value: &str) -> Result<(), ValidationError> { Ok(()) } -fn validate_no_trailing_slash(value: &str) -> Result<(), ValidationError> { - if value.ends_with('/') { - let mut err = ValidationError::new("trailing_slash"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include a trailing slash".into()); - return Err(err); - } - Ok(()) -} - -fn validate_host_header_override(value: &str) -> Result<(), ValidationError> { - if let Err(reason) = validate_host_header_override_value(value) { - let mut err = ValidationError::new("invalid_host_header_override"); - err.add_param("value".into(), &value); - err.add_param("reason".into(), &reason); - err.message = Some( - "origin_host_header_override must be a valid host or host:port without scheme, path, query, or fragment" - .into(), - ); - return Err(err); - } - - Ok(()) -} - /// Longest `directory_index` accepted, the usual limit on one file name. const MAX_DIRECTORY_INDEX_BYTES: usize = 255; @@ -5357,63 +5373,6 @@ fn validate_asset_route_prefix(value: &str) -> Result<(), ValidationError> { Ok(()) } -fn validate_proxy_origin_url(value: &str) -> Result<(), ValidationError> { - validate_no_trailing_slash(value)?; - - let parsed = Url::parse(value).map_err(|parse_error| { - let mut err = ValidationError::new("invalid_origin_url"); - err.add_param("value".into(), &value); - err.add_param("message".into(), &parse_error.to_string()); - err.message = Some("origin_url must be an absolute http or https URL".into()); - err - })?; - - if !matches!(parsed.scheme(), "http" | "https") { - let mut err = ValidationError::new("invalid_origin_url_scheme"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must use http or https".into()); - return Err(err); - } - - if parsed.host_str().is_none() { - let mut err = ValidationError::new("missing_origin_host"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must include a host".into()); - return Err(err); - } - - if !parsed.username().is_empty() || parsed.password().is_some() { - let mut err = ValidationError::new("origin_url_has_userinfo"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include username or password".into()); - return Err(err); - } - - if parsed.fragment().is_some() { - let mut err = ValidationError::new("origin_url_has_fragment"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include a fragment".into()); - return Err(err); - } - - if !matches!(parsed.path(), "" | "/") { - let mut err = ValidationError::new("origin_url_has_path"); - err.add_param("value".into(), &value); - err.message = - Some("origin_url must not include a path; only scheme/host/port are used".into()); - return Err(err); - } - - if parsed.query().is_some() { - let mut err = ValidationError::new("origin_url_has_query"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include a query string".into()); - return Err(err); - } - - Ok(()) -} - fn from_value_or_str<'de, D, T>(deserializer: D) -> Result where D: Deserializer<'de>, @@ -5886,8 +5845,10 @@ mod tests { // like a configured `trusted_client_ip`, needs a compatible blob restored // before rolling back to a binary that predates it. The shared test // config sets `[geo]` because the permission model requires a default - // country, so both selector tables are reset to unset here. + // country and `[source]` because every deployment names one, so those + // tables are reset to unset here. settings.geo = GeoConfig::default(); + settings.source = SectionModules::default(); settings.device = DeviceConfig::default(); settings.sections = TypeSections::default(); settings.auction.modules.clear(); @@ -6033,14 +5994,17 @@ module = \"none\"", }); let mut asset_route = ProxyAssetRoute::new("/s3-assets/", "https://s3.canary.example"); - asset_route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: "us-east-1".to_string(), - secret_store: None, - access_key_id: Redacted::new(CANARY_S3_ACCESS_KEY_ID.to_string()), - secret_access_key: Redacted::new(CANARY_S3_SECRET_ACCESS_KEY.to_string()), - session_token: Some(Redacted::new(CANARY_S3_SESSION_TOKEN.to_string())), - origin_query: None, - })); + asset_route.set_origin_source(&crate::source::OriginSettings { + auth: Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: "us-east-1".to_string(), + secret_store: None, + access_key_id: Redacted::new(CANARY_S3_ACCESS_KEY_ID.to_string()), + secret_access_key: Redacted::new(CANARY_S3_SECRET_ACCESS_KEY.to_string()), + session_token: Some(Redacted::new(CANARY_S3_SESSION_TOKEN.to_string())), + origin_query: None, + })), + ..crate::source::OriginSettings::new("https://s3.canary.example") + }); settings.proxy.asset_routes = vec![asset_route]; settings.analytics = Some(AnalyticsConfig { @@ -7161,11 +7125,11 @@ module = \"none\"", ".test-publisher.com", "EC cookie domain should be computed as .{{domain}}" ); - assert_eq!( - settings.publisher.origin_url, - "https://origin.test-publisher.com" - ); - assert_eq!(settings.publisher.origin_host_header_override, None); + let origin = settings + .origin_source() + .expect("the fixture should select the origin source"); + assert_eq!(origin.url, "https://origin.test-publisher.com"); + assert_eq!(origin.host_header, None); assert_eq!( settings.ec.module.as_ref(), Some(&EcModuleSelection::from(HMAC_MODULE_KEY)), @@ -8519,16 +8483,16 @@ passphrase = "another-test-secret-key-32-bytes" } #[test] - fn validate_rejects_trailing_slash_in_origin_url() { + fn validate_rejects_trailing_slash_in_source_origin_url() { let toml_str = crate_test_settings_str().replace( - r#"origin_url = "https://origin.test-publisher.com""#, - r#"origin_url = "https://origin.test-publisher.com/""#, + r#"url = "https://origin.test-publisher.com""#, + r#"url = "https://origin.test-publisher.com/""#, ); - let result = Settings::from_toml(&toml_str); + let err = Settings::from_toml(&toml_str).expect_err("a url ending in '/' should fail"); assert!( - result.is_err(), - "origin_url ending with '/' should fail validation" + format!("{err:?}").contains("[source.origin] url"), + "the refusal should name the setting: {err:?}" ); } @@ -8632,16 +8596,21 @@ passphrase = "another-test-secret-key-32-bytes" } #[test] - fn validate_accepts_origin_host_header_override() { + fn validate_accepts_source_origin_host_header() { let toml_str = crate_test_settings_str().replace( - r#"origin_url = "https://origin.test-publisher.com""#, - r#"origin_url = "https://origin.test-publisher.com" -origin_host_header_override = "www.example.com:8443""#, + r#"url = "https://origin.test-publisher.com""#, + r#"url = "https://origin.test-publisher.com" +host_header = "www.example.com:8443""#, ); let settings = Settings::from_toml(&toml_str).expect("should accept host header override"); assert_eq!( - settings.publisher.origin_host_header(), + settings + .origin_source() + .expect("should select the origin source") + .location() + .expect("should give a location") + .host_header(), "www.example.com:8443", "should use configured host header override" ); @@ -8650,8 +8619,8 @@ origin_host_header_override = "www.example.com:8443""#, #[test] fn publisher_rejects_unknown_fields() { let toml_str = crate_test_settings_str().replace( - r#"origin_url = "https://origin.test-publisher.com""#, - r#"origin_url = "https://origin.test-publisher.com" + r#"proxy_secret = "unit-test-proxy-secret""#, + r#"proxy_secret = "unit-test-proxy-secret" origin_host_header_overide = "www.example.com""#, ); @@ -8664,7 +8633,23 @@ origin_host_header_overide = "www.example.com""#, } #[test] - fn validate_rejects_invalid_origin_host_header_overrides() { + fn source_origin_rejects_unknown_fields() { + let toml_str = crate_test_settings_str().replace( + r#"url = "https://origin.test-publisher.com""#, + r#"url = "https://origin.test-publisher.com" +host_headr = "www.example.com""#, + ); + + let err = Settings::from_toml(&toml_str) + .expect_err("unknown origin fields should fail configuration loading"); + assert!( + format!("{err:?}").contains("host_headr"), + "error should identify the misspelled origin field: {err:?}" + ); + } + + #[test] + fn validate_rejects_invalid_source_origin_host_headers() { for override_value in [ "", " www.example.com", @@ -8684,23 +8669,23 @@ origin_host_header_overide = "www.example.com""#, "[::1", ] { let toml_str = crate_test_settings_str().replace( - r#"origin_url = "https://origin.test-publisher.com""#, + r#"url = "https://origin.test-publisher.com""#, &format!( - "origin_url = \"https://origin.test-publisher.com\"\norigin_host_header_override = {override_value:?}" + "url = \"https://origin.test-publisher.com\"\nhost_header = {override_value:?}" ), ); let result = Settings::from_toml(&toml_str); assert!( result.is_err(), - "origin_host_header_override {override_value:?} should fail validation" + "host_header {override_value:?} should fail validation" ); } } #[test] fn test_settings_missing_required_fields() { - let re = Regex::new(r"origin_url = .*").expect("regex should compile"); + let re = Regex::new(r"url = .*").expect("regex should compile"); let toml_str = crate_test_settings_str(); let toml_str = re.replace(&toml_str, ""); @@ -9015,9 +9000,10 @@ source_domain = "partner.example.com" let toml_str = crate_test_settings_str(); let origin_key = format!( - "{}{}PUBLISHER{}ORIGIN_URL", + "{}{}SOURCE{}ORIGIN{}URL", ENVIRONMENT_VARIABLE_PREFIX, ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, ENVIRONMENT_VARIABLE_SEPARATOR ); let partner_0_name_key = format!( @@ -9161,9 +9147,10 @@ source_domain = "partner.example.com" let toml_str = crate_test_settings_str(); let origin_key = format!( - "{}{}PUBLISHER{}ORIGIN_URL", + "{}{}SOURCE{}ORIGIN{}URL", ENVIRONMENT_VARIABLE_PREFIX, ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, ENVIRONMENT_VARIABLE_SEPARATOR ); let path_key = format!( @@ -9233,9 +9220,10 @@ source_domain = "partner.example.com" fn test_set_env() { temp_env::with_var( format!( - "{}{}PUBLISHER{}ORIGIN_URL", + "{}{}SOURCE{}ORIGIN{}URL", ENVIRONMENT_VARIABLE_PREFIX, ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, ENVIRONMENT_VARIABLE_SEPARATOR ), Some("https://change-publisher.com"), @@ -9244,22 +9232,48 @@ source_domain = "partner.example.com" assert!(settings.is_ok(), "Settings should load from embedded TOML"); assert_eq!( - settings.expect("should load settings").publisher.origin_url, + settings + .expect("should load settings") + .origin_source() + .expect("should select the origin source") + .url, "https://change-publisher.com" ); }, ); } + #[test] + fn the_old_environment_variable_for_the_address_is_refused_naming_the_source() { + temp_env::with_var( + format!( + "{}{}PUBLISHER{}ORIGIN_URL", + ENVIRONMENT_VARIABLE_PREFIX, + ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR + ), + Some("https://change-publisher.com"), + || { + let error = Settings::from_toml_and_env(&crate_test_settings_str()) + .expect_err("should refuse the old variable"); + assert!( + format!("{error:?}").contains("[source.origin] url"), + "should name where the address is set now: {error:?}" + ); + }, + ); + } + #[test] fn test_override_env() { let toml_str = crate_test_settings_str(); temp_env::with_var( format!( - "{}{}PUBLISHER{}ORIGIN_URL", + "{}{}SOURCE{}ORIGIN{}URL", ENVIRONMENT_VARIABLE_PREFIX, ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, ENVIRONMENT_VARIABLE_SEPARATOR ), Some("https://change-publisher.com"), @@ -9268,7 +9282,11 @@ source_domain = "partner.example.com" assert!(settings.is_ok(), "Settings should load from embedded TOML"); assert_eq!( - settings.expect("should load settings").publisher.origin_url, + settings + .expect("should load settings") + .origin_source() + .expect("should select the origin source") + .url, "https://change-publisher.com" ); }, @@ -9276,11 +9294,12 @@ source_domain = "partner.example.com" } #[test] - fn test_origin_host_header_override_env() { + fn test_source_origin_host_header_env() { let env_key = format!( - "{}{}PUBLISHER{}ORIGIN_HOST_HEADER_OVERRIDE", + "{}{}SOURCE{}ORIGIN{}HOST_HEADER", ENVIRONMENT_VARIABLE_PREFIX, ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, ENVIRONMENT_VARIABLE_SEPARATOR ); @@ -9288,20 +9307,27 @@ source_domain = "partner.example.com" let settings = Settings::from_toml_and_env(&crate_test_settings_str()) .expect("should load settings with host header override env"); + let origin = settings + .origin_source() + .expect("should select the origin source"); + assert_eq!(origin.host_header.as_deref(), Some("www.example.com")); assert_eq!( - settings.publisher.origin_host_header_override.as_deref(), - Some("www.example.com") + origin + .location() + .expect("should give a location") + .host_header(), + "www.example.com" ); - assert_eq!(settings.publisher.origin_host_header(), "www.example.com"); }); } #[test] - fn test_origin_host_header_override_env_typo_fails_closed() { + fn test_source_origin_host_header_env_typo_fails_closed() { let env_key = format!( - "{}{}PUBLISHER{}ORIGIN_HOST_HEADER_OVERIDE", + "{}{}SOURCE{}ORIGIN{}HOST_HEADR", ENVIRONMENT_VARIABLE_PREFIX, ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, ENVIRONMENT_VARIABLE_SEPARATOR ); @@ -9309,125 +9335,12 @@ source_domain = "partner.example.com" let err = Settings::from_toml_and_env(&crate_test_settings_str()) .expect_err("misspelled host override env var should fail configuration loading"); assert!( - format!("{err:?}").contains("origin_host_header_overide"), - "error should identify the misspelled publisher env field: {err:?}" + format!("{err:?}").contains("host_headr"), + "error should identify the misspelled origin env field: {err:?}" ); }); } - #[test] - fn test_publisher_origin_host() { - // Test with full URL including port - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "https://origin.example.com:8080".to_string(), - origin_host_header_override: None, - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - assert_eq!(publisher.origin_host(), "origin.example.com:8080"); - - // Test with URL without port (default HTTPS port) - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "https://origin.example.com".to_string(), - origin_host_header_override: None, - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - assert_eq!(publisher.origin_host(), "origin.example.com"); - - // Test with HTTP URL with explicit port - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "http://localhost:9090".to_string(), - origin_host_header_override: None, - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - assert_eq!(publisher.origin_host(), "localhost:9090"); - - // Test with URL without protocol (fallback to original) - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "localhost:9090".to_string(), - origin_host_header_override: None, - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - assert_eq!(publisher.origin_host(), "localhost:9090"); - - // Test with IPv4 address - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "http://192.168.1.1:8080".to_string(), - origin_host_header_override: None, - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - assert_eq!(publisher.origin_host(), "192.168.1.1:8080"); - - // Test with IPv6 address - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "http://[::1]:8080".to_string(), - origin_host_header_override: None, - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - assert_eq!(publisher.origin_host(), "[::1]:8080"); - } - - #[test] - fn test_publisher_origin_host_header_defaults_to_origin_host() { - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "https://origin.example.com:8443".to_string(), - origin_host_header_override: None, - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - - assert_eq!(publisher.origin_host_header(), "origin.example.com:8443"); - } - - #[test] - fn test_publisher_origin_host_header_uses_override() { - let publisher = Publisher { - domain: "example.com".to_string(), - cookie_domain: ".example.com".to_string(), - origin_url: "https://origin.example.com".to_string(), - origin_host_header_override: Some("www.example.com".to_string()), - proxy_secret: Redacted::new("test-secret".to_string()), - max_buffered_body_bytes: 16 * 1024 * 1024, - origin_auth: None, - directory_index: None, - }; - - assert_eq!(publisher.origin_host_header(), "www.example.com"); - } - #[test] fn publisher_default_max_buffered_body_bytes_matches_config_default() { // The manual `Default` impl must agree with the serde default applied @@ -9441,10 +9354,15 @@ source_domain = "partner.example.com" let from_toml = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.example.com" + [publisher] domain = "example.com" cookie_domain = ".example.com" - origin_url = "https://origin.example.com" proxy_secret = "unit-test-proxy-secret" [ec] @@ -9465,57 +9383,53 @@ source_domain = "partner.example.com" ); } - /// The fixture settings with `lines` added to `[publisher]` and its - /// `origin_url` replaced by `origin_url`. - fn publisher_settings_with(origin_url: &str, lines: &str) -> String { - crate_test_settings_str() - .replacen( - "origin_url = \"https://origin.test-publisher.com\"", - &format!("origin_url = \"{origin_url}\""), - 1, - ) - .replacen( - "proxy_secret = \"unit-test-proxy-secret\"", - &format!("proxy_secret = \"unit-test-proxy-secret\"\n{lines}"), - 1, - ) + /// The fixture settings with `[source.origin]` at `url` and `lines` added + /// to it. + fn origin_settings_with(url: &str, lines: &str) -> String { + crate_test_settings_str().replacen( + "url = \"https://origin.test-publisher.com\"", + &format!("url = \"{url}\"\n{lines}"), + 1, + ) } const STORAGE_ORIGIN_URL: &str = "https://storage.example.com/bucket/site"; #[test] - fn publisher_object_storage_settings_are_off_by_default() { + fn source_origin_object_storage_settings_are_off_by_default() { let settings = Settings::from_toml(&crate_test_settings_str()).expect("should parse test settings"); + let origin = settings + .origin_source() + .expect("the fixture should select the origin source"); + assert!(origin.auth.is_none(), "auth should default to unset"); assert!( - settings.publisher.origin_auth.is_none(), - "origin_auth should default to unset" - ); - assert!( - settings.publisher.directory_index.is_none(), + origin.directory_index.is_none(), "directory_index should default to unset" ); - let serialized = - serde_json::to_value(&settings.publisher).expect("should serialize publisher"); + let serialized = serde_json::to_value(&settings.source).expect("should serialize source"); assert!( - serialized.get("origin_auth").is_none() && serialized.get("directory_index").is_none(), - "unset settings should serialize as a document written before they existed: \ - {serialized}" + serialized["origin"].get("auth").is_none() + && serialized["origin"].get("directory_index").is_none(), + "unset settings should not be written: {serialized}" ); } #[test] - fn publisher_origin_auth_reads_the_asset_route_auth_table() { - let settings = Settings::from_toml(&publisher_settings_with( + fn source_origin_auth_reads_the_asset_route_auth_table() { + let settings = Settings::from_toml(&origin_settings_with( STORAGE_ORIGIN_URL, - r#"origin_auth = { type = "s3_sigv4", region = "test-region-1", access_key_id = "origin_access_key_id", secret_access_key = "origin_secret_access_key", session_token = "origin_session_token", origin_query = "strip" } + r#"auth = { type = "s3_sigv4", region = "test-region-1", access_key_id = "origin_access_key_id", secret_access_key = "origin_secret_access_key", session_token = "origin_session_token", origin_query = "strip" } directory_index = "index.html""#, )) - .expect("should parse a publisher with origin_auth and directory_index"); + .expect("should parse an origin with auth and directory_index"); + let origin = settings + .origin_source() + .expect("should select the origin source"); - let Some(AssetOriginAuth::S3SigV4(auth)) = settings.publisher.origin_auth.as_ref() else { - panic!("should read origin_auth as the asset route auth table"); + let Some(AssetOriginAuth::S3SigV4(auth)) = origin.auth.as_ref() else { + panic!("should read auth as the asset route auth table"); }; assert_eq!(auth.region, "test-region-1"); assert_eq!(auth.access_key_id.expose(), "origin_access_key_id"); @@ -9528,21 +9442,24 @@ source_domain = "partner.example.com" Some("origin_session_token") ); assert!( - settings.publisher.origin_query_is_stripped(), + origin + .location() + .expect("should give a location") + .query_is_stripped(), "origin_query = \"strip\" should leave the query off origin requests" ); - assert_eq!( - settings.publisher.directory_index.as_deref(), - Some("index.html") - ); + assert_eq!(origin.directory_index.as_deref(), Some("index.html")); - let defaulted = Settings::from_toml(&publisher_settings_with( + let defaulted = Settings::from_toml(&origin_settings_with( STORAGE_ORIGIN_URL, - r#"origin_auth = { type = "s3_sigv4", region = "test-region-1" }"#, + r#"auth = { type = "s3_sigv4", region = "test-region-1" }"#, )) - .expect("should parse origin_auth naming only its region"); - let Some(AssetOriginAuth::S3SigV4(auth)) = defaulted.publisher.origin_auth.as_ref() else { - panic!("should read origin_auth with defaulted key references"); + .expect("should parse auth naming only its region"); + let defaulted = defaulted + .origin_source() + .expect("should select the origin source"); + let Some(AssetOriginAuth::S3SigV4(auth)) = defaulted.auth.as_ref() else { + panic!("should read auth with defaulted key references"); }; assert_eq!( ( @@ -9553,13 +9470,16 @@ source_domain = "partner.example.com" "omitted key references should default as an asset route's do" ); assert!( - !defaulted.publisher.origin_query_is_stripped(), + !defaulted + .location() + .expect("should give a location") + .query_is_stripped(), "the query should be kept unless origin_query strips it" ); } #[test] - fn publisher_directory_index_accepts_a_file_name() { + fn source_origin_directory_index_accepts_a_file_name() { for name in [ "index.html", "default.htm", @@ -9568,17 +9488,24 @@ source_domain = "partner.example.com" "home~1.html", ] { let lines = format!("directory_index = {}", toml::Value::String(name.to_owned())); - let settings = Settings::from_toml(&publisher_settings_with( + let settings = Settings::from_toml(&origin_settings_with( "https://origin.test-publisher.com", &lines, )) .unwrap_or_else(|err| panic!("should accept directory_index `{name}`: {err:?}")); - assert_eq!(settings.publisher.directory_index.as_deref(), Some(name)); + assert_eq!( + settings + .origin_source() + .expect("should select the origin source") + .directory_index + .as_deref(), + Some(name) + ); } } #[test] - fn publisher_directory_index_refuses_anything_but_a_file_name() { + fn source_origin_directory_index_refuses_anything_but_a_file_name() { let too_long = "a".repeat(super::MAX_DIRECTORY_INDEX_BYTES + 1); for name in [ "", @@ -9598,63 +9525,63 @@ source_domain = "partner.example.com" too_long.as_str(), ] { let lines = format!("directory_index = {}", toml::Value::String(name.to_owned())); - let err = Settings::from_toml(&publisher_settings_with( + let err = Settings::from_toml(&origin_settings_with( "https://origin.test-publisher.com", &lines, )) .expect_err("should refuse a directory_index that is not one file name"); assert!( - format!("{err:?}").contains("publisher.directory_index"), + format!("{err:?}").contains("[source.origin] directory_index"), "the refusal of `{name}` should name the setting: {err:?}" ); } } #[test] - fn publisher_origin_auth_refuses_a_table_it_cannot_sign_with() { + fn source_origin_auth_refuses_a_table_it_cannot_sign_with() { for (origin_url, table, expected) in [ ( STORAGE_ORIGIN_URL, r#"{ type = "s3_sigv4", region = "" }"#, - "publisher.origin_auth s3_sigv4 region must not be empty", + "source.origin.auth s3_sigv4 region must not be empty", ), ( STORAGE_ORIGIN_URL, r#"{ type = "s3_sigv4", region = "Test_Region" }"#, - "publisher.origin_auth s3_sigv4 region must contain only lowercase letters", + "source.origin.auth s3_sigv4 region must contain only lowercase letters", ), ( STORAGE_ORIGIN_URL, r#"{ type = "s3_sigv4", region = "test-region-1", access_key_id = " " }"#, - "publisher.origin_auth s3_sigv4 credentials must not be empty", + "source.origin.auth s3_sigv4 credentials must not be empty", ), ( STORAGE_ORIGIN_URL, r#"{ type = "s3_sigv4", region = "test-region-1", secret_access_key = "" }"#, - "publisher.origin_auth s3_sigv4 credentials must not be empty", + "source.origin.auth s3_sigv4 credentials must not be empty", ), ( "http://storage.example.com/bucket/site", r#"{ type = "s3_sigv4", region = "test-region-1" }"#, - "publisher.origin_auth requires publisher.origin_url to use https", + "[source.origin] signs with auth, so url must use https", ), ( STORAGE_ORIGIN_URL, r#"{ type = "unknown_scheme", region = "test-region-1" }"#, - "origin_auth", + "source.origin", ), ( STORAGE_ORIGIN_URL, r#"{ type = "s3_sigv4", region = "test-region-1", bucket = "site" }"#, - "origin_auth", + "bucket", ), - (STORAGE_ORIGIN_URL, r#""s3_sigv4""#, "origin_auth"), + (STORAGE_ORIGIN_URL, r#""s3_sigv4""#, "source.origin"), ] { - let err = Settings::from_toml(&publisher_settings_with( + let err = Settings::from_toml(&origin_settings_with( origin_url, - &format!("origin_auth = {table}"), + &format!("auth = {table}"), )) - .expect_err("should refuse an origin_auth the signer cannot use"); + .expect_err("should refuse an auth the signer cannot use"); assert!( format!("{err:?}").contains(expected), "the refusal of `{table}` should say `{expected}`: {err:?}" @@ -9663,36 +9590,40 @@ source_domain = "partner.example.com" } #[test] - fn asset_route_auth_refusals_still_name_the_asset_route() { + fn asset_route_auth_refusals_name_the_asset_route() { let mut route = ProxyAssetRoute::new("/assets/", "https://storage.example.com"); - route.auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { - region: String::new(), - secret_store: None, - access_key_id: Redacted::new("access_key_id".to_owned()), - secret_access_key: Redacted::new("secret_access_key".to_owned()), - session_token: None, - origin_query: None, - })); + route.set_origin_source(&crate::source::OriginSettings { + auth: Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + region: String::new(), + secret_store: None, + access_key_id: Redacted::new("access_key_id".to_owned()), + secret_access_key: Redacted::new("secret_access_key".to_owned()), + session_token: None, + origin_query: None, + })), + ..crate::source::OriginSettings::new("https://storage.example.com") + }); let err = route .prepare_runtime() .expect_err("should refuse an asset route auth with no region"); assert!( - format!("{err:?}") - .contains("proxy.asset_routes auth s3_sigv4 region must not be empty"), - "the asset route refusal should keep its wording: {err:?}" + format!("{err:?}").contains( + "proxy.asset_routes /assets/ source.origin.auth s3_sigv4 region must not be empty" + ), + "the asset route refusal should name the route: {err:?}" ); } #[test] - fn publisher_origin_auth_credentials_stay_out_of_debug_output() { + fn source_origin_auth_credentials_stay_out_of_debug_output() { let canaries = [ "canary-origin-access", "canary-origin-secret", "canary-origin-session", ]; - let publisher = Publisher { - origin_auth: Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { + let origin = crate::source::OriginSettings { + auth: Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig { region: "test-region-1".to_owned(), secret_store: None, access_key_id: Redacted::new(canaries[0].to_owned()), @@ -9700,10 +9631,12 @@ source_domain = "partner.example.com" session_token: Some(Redacted::new(canaries[2].to_owned())), origin_query: None, })), - ..Publisher::default() + ..crate::source::OriginSettings::new("https://storage.example.com/bucket/site") }; + let mut settings = Settings::default(); + settings.set_origin_source(&origin); - let debug = format!("{publisher:?}"); + let debug = format!("{origin:?} {settings:?}"); assert!( debug.contains("test-region-1"), @@ -9724,10 +9657,15 @@ source_domain = "partner.example.com" // silently breaking traffic. let result = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.example.com" + [publisher] domain = "example.com" cookie_domain = ".example.com" - origin_url = "https://origin.example.com" proxy_secret = "unit-test-proxy-secret" max_buffered_body_bytes = 0 @@ -9944,6 +9882,97 @@ source_domain = "partner.example.com" } } + /// The fields that said where the publisher's pages come from are refused + /// naming where they are written now. + #[test] + fn the_moved_publisher_origin_fields_are_refused_naming_the_source() { + for (line, expected) in [ + ( + r#"origin_url = "https://origin.example.com""#, + "[source.origin] url", + ), + ( + r#"origin_host_header_override = "www.example.com""#, + "[source.origin] host_header", + ), + ( + r#"directory_index = "index.html""#, + "[source.origin] directory_index", + ), + ( + r#"origin_auth = { type = "s3_sigv4", region = "test-region-1" }"#, + "[source.origin] auth", + ), + ] { + let toml = crate_test_settings_str().replacen( + "proxy_secret = \"unit-test-proxy-secret\"", + &format!("proxy_secret = \"unit-test-proxy-secret\"\n{line}"), + 1, + ); + + let error = Settings::from_toml(&toml).expect_err("should refuse the moved field"); + assert!( + format!("{error:?}").contains(expected), + "`{line}` should be refused naming `{expected}`: {error:?}" + ); + } + } + + /// An asset route's origin fields are refused naming the route's source. + #[test] + fn the_moved_asset_route_origin_fields_are_refused_naming_the_source() { + for (lines, expected) in [ + ( + r#"origin_url = "https://assets.example.com""#, + "[proxy.asset_routes.source.origin] url", + ), + ( + r#"source.module = "origin" + source.origin.url = "https://assets.example.com" + auth = { type = "s3_sigv4", region = "us-east-1" }"#, + "[proxy.asset_routes.source.origin] auth", + ), + ] { + let toml = format!( + "{}\n[proxy]\n\n[[proxy.asset_routes]]\nprefix = \"/assets/\"\n{lines}\n", + crate_test_settings_str() + ); + + let error = Settings::from_toml(&toml).expect_err("should refuse the moved field"); + assert!( + format!("{error:?}").contains(expected), + "the route should be refused naming `{expected}`: {error:?}" + ); + } + } + + /// A configuration blob in the old shape is refused the same way. + #[test] + fn json_settings_refuse_the_moved_publisher_origin_url() { + let mut value = serde_json::to_value(create_test_settings()) + .expect("should serialize the test settings fixture to JSON"); + value["publisher"]["origin_url"] = json!("https://origin.example.com"); + + let error = + Settings::from_json_value(value).expect_err("should refuse the moved field in JSON"); + assert!( + format!("{error:?}").contains("[source.origin] url"), + "should name where the address is written now: {error:?}" + ); + } + + #[test] + fn a_document_naming_no_source_is_refused() { + let toml = + crate_test_settings_str().replacen("[source]\n module = \"origin\"", "", 1); + + let error = Settings::from_toml(&toml).expect_err("should refuse a missing source"); + assert!( + format!("{error:?}").contains("[source] names no source"), + "should say the document names no source: {error:?}" + ); + } + #[test] fn invalid_settings_for_a_selected_module_fail_registry_startup() { fn build( @@ -10198,7 +10227,8 @@ source_domain = "partner.example.com" allowed_domains: vec![], asset_routes: vec![ProxyAssetRoute { prefix: " /.images/ ".to_string(), - origin_url: " https://assets.example.com ".to_string(), + source: crate::source::OriginSettings::new(" https://assets.example.com ") + .selection(), ..Default::default() }], modules: SectionModules::default(), @@ -10209,8 +10239,12 @@ source_domain = "partner.example.com" "should trim asset-route prefix" ); assert_eq!( - proxy.asset_routes[0].origin_url, "https://assets.example.com", - "should trim asset-route origin_url" + proxy.asset_routes[0] + .origin_source() + .expect("should select the origin source") + .url, + "https://assets.example.com", + "reading the source should trim its url" ); } @@ -10221,7 +10255,8 @@ source_domain = "partner.example.com" allowed_domains: vec![], asset_routes: vec![ProxyAssetRoute { prefix: "/.images/".to_string(), - origin_url: "https://assets.example.com".to_string(), + source: crate::source::OriginSettings::new("https://assets.example.com") + .selection(), path_pattern: Some(" ^/(.*)$ ".to_string()), target_path: Some(" /rewritten/$1 ".to_string()), ..Default::default() @@ -10250,7 +10285,8 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.image/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" path_pattern = "^/\\.image/(.*)/[^/]+\\.([^/.]+)$" target_path = "/image/upload/$1.$2" "#; @@ -10297,9 +10333,10 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.image/" - origin_url = "https://bucket.s3.us-east-1.amazonaws.com" + source.module = "origin" + source.origin.url = "https://bucket.s3.us-east-1.amazonaws.com" - [proxy.asset_routes.auth] + [proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "us-east-1" origin_query = "strip" @@ -10317,7 +10354,11 @@ source_domain = "partner.example.com" .expect("should match configured route"); assert!(route.image_optimizer_enabled()); assert_eq!(route.origin_query_policy(), OriginQueryPolicy::Strip); - match route.auth.as_ref().expect("should configure route auth") { + match route + .origin_source() + .and_then(|origin| origin.auth) + .expect("should configure route auth") + { AssetOriginAuth::S3SigV4(config) => { assert_eq!(config.region, "us-east-1"); assert_eq!(config.secret_store, None); @@ -10335,9 +10376,10 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "http://bucket.s3.us-east-1.amazonaws.com" + source.module = "origin" + source.origin.url = "http://bucket.s3.us-east-1.amazonaws.com" - [proxy.asset_routes.auth] + [proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "us-east-1" "#; @@ -10346,7 +10388,7 @@ source_domain = "partner.example.com" .expect_err("should reject cleartext S3 SigV4 origin URLs"); assert!( - format!("{err:?}").contains("must use https when auth type is s3_sigv4"), + format!("{err:?}").contains("signs with auth, so url must use https"), "should mention the S3 SigV4 HTTPS requirement: {err:?}" ); } @@ -10361,9 +10403,10 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://bucket.s3.us-east-1.amazonaws.com" + source.module = "origin" + source.origin.url = "https://bucket.s3.us-east-1.amazonaws.com" - [proxy.asset_routes.auth] + [proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "{region}" "# @@ -10387,9 +10430,10 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://bucket.s3.us-east-1.amazonaws.com" + source.module = "origin" + source.origin.url = "https://bucket.s3.us-east-1.amazonaws.com" - [proxy.asset_routes.auth] + [proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "us-east-1" secret_access_key_name = "secret_access_key" @@ -10419,7 +10463,8 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.image/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" [proxy.asset_routes.image_optimizer] enabled = true @@ -10579,7 +10624,8 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/images/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" [proxy.asset_routes.image_optimizer] region = " us_east " @@ -10717,25 +10763,31 @@ source_domain = "partner.example.com" ), ( format!( - "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}ORIGIN_URL" + "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}SOURCE{separator}MODULE" + ), + Some("origin"), + ), + ( + format!( + "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}SOURCE{separator}ORIGIN{separator}URL" ), Some("https://bucket.s3.us-west-2.amazonaws.com"), ), ( format!( - "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}AUTH{separator}TYPE" + "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}SOURCE{separator}ORIGIN{separator}AUTH{separator}TYPE" ), Some("s3_sigv4"), ), ( format!( - "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}AUTH{separator}REGION" + "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}SOURCE{separator}ORIGIN{separator}AUTH{separator}REGION" ), Some("us-west-2"), ), ( format!( - "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}AUTH{separator}ORIGIN_QUERY" + "{ENVIRONMENT_VARIABLE_PREFIX}{separator}PROXY{separator}ASSET_ROUTES{separator}0{separator}SOURCE{separator}ORIGIN{separator}AUTH{separator}ORIGIN_QUERY" ), Some("strip"), ), @@ -10858,7 +10910,8 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.image/" - origin_url = "https://bucket.s3.us-east-1.amazonaws.com" + source.module = "origin" + source.origin.url = "https://bucket.s3.us-east-1.amazonaws.com" [proxy.asset_routes.image_optimizer] enabled = true @@ -10880,7 +10933,7 @@ source_domain = "partner.example.com" fn proxy_asset_route_disabled_image_optimizer_does_not_override_origin_query_policy() { let route = ProxyAssetRoute { prefix: "/.image/".to_string(), - origin_url: "https://assets.example.com".to_string(), + source: crate::source::OriginSettings::new("https://assets.example.com").selection(), image_optimizer: Some(AssetImageOptimizerConfig { enabled: false, region: "us_east".to_string(), @@ -10901,7 +10954,8 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.image/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" path_pattern = "^/\\.image/(.*)$" "#; let err = Settings::from_toml(&toml_str) @@ -10921,7 +10975,8 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.image/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" path_pattern = "[" target_path = "/image/upload/$1" "#; @@ -10942,12 +10997,12 @@ source_domain = "partner.example.com" asset_routes: vec![ ProxyAssetRoute { prefix: "/.images/".to_string(), - origin_url: "https://a.example.com".to_string(), + source: crate::source::OriginSettings::new("https://a.example.com").selection(), ..Default::default() }, ProxyAssetRoute { prefix: "/.images/special/".to_string(), - origin_url: "https://b.example.com".to_string(), + source: crate::source::OriginSettings::new("https://b.example.com").selection(), ..Default::default() }, ], @@ -10958,7 +11013,11 @@ source_domain = "partner.example.com" .asset_route_for_path("/.images/special/banner.png") .expect("should match a configured asset route"); assert_eq!( - route.origin_url, "https://b.example.com", + route + .origin_source() + .expect("should select the origin source") + .url, + "https://b.example.com", "should prefer the most specific prefix" ); } @@ -10971,12 +11030,14 @@ source_domain = "partner.example.com" asset_routes: vec![ ProxyAssetRoute { prefix: "/.images/".to_string(), - origin_url: "https://first.example.com".to_string(), + source: crate::source::OriginSettings::new("https://first.example.com") + .selection(), ..Default::default() }, ProxyAssetRoute { prefix: "/.images/".to_string(), - origin_url: "https://second.example.com".to_string(), + source: crate::source::OriginSettings::new("https://second.example.com") + .selection(), ..Default::default() }, ], @@ -10987,7 +11048,11 @@ source_domain = "partner.example.com" .asset_route_for_path("/.images/banner.png") .expect("should match duplicate prefixes deterministically"); assert_eq!( - route.origin_url, "https://first.example.com", + route + .origin_source() + .expect("should select the origin source") + .url, + "https://first.example.com", "should keep the first configured duplicate prefix" ); } @@ -11018,7 +11083,8 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = ".images/" - origin_url = "https://assets.example.com" + source.module = "origin" + source.origin.url = "https://assets.example.com" "#; let err = Settings::from_toml(&toml_str).expect_err("should reject invalid asset-route prefix"); @@ -11036,31 +11102,36 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "ftp://assets.example.com" + source.module = "origin" + source.origin.url = "ftp://assets.example.com" "#; let err = Settings::from_toml(&toml_str) .expect_err("should reject non-http asset-route origin_url"); assert!( - format!("{err:?}").contains("origin_url must use http or https"), + format!("{err:?}").contains("must use http or https"), "should mention the origin_url validation failure: {err:?}" ); } #[test] - fn proxy_asset_route_validation_rejects_origin_url_path() { + fn proxy_asset_route_accepts_an_origin_folder() { let toml_str = crate_test_settings_str() + r#" [proxy] [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://assets.example.com/api" + source.module = "origin" + source.origin.url = "https://assets.example.com/api" "#; - let err = Settings::from_toml(&toml_str) - .expect_err("should reject asset-route origin_url with path"); - assert!( - format!("{err:?}").contains("origin_url must not include a path"), - "should mention the origin_url path validation failure: {err:?}" + let settings = Settings::from_toml(&toml_str) + .expect("a path in an asset route's origin url should name a folder"); + assert_eq!( + settings.proxy.asset_routes[0] + .origin_source() + .expect("should select the origin source") + .url, + "https://assets.example.com/api" ); } @@ -11072,12 +11143,13 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://assets.example.com?token=abc" + source.module = "origin" + source.origin.url = "https://assets.example.com?token=abc" "#; let err = Settings::from_toml(&toml_str) .expect_err("should reject asset-route origin_url with query"); assert!( - format!("{err:?}").contains("origin_url must not include a query string"), + format!("{err:?}").contains("must not carry a query or a fragment"), "should mention the origin_url query validation failure: {err:?}" ); } @@ -11090,12 +11162,13 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://user:pass@assets.example.com" + source.module = "origin" + source.origin.url = "https://user:pass@assets.example.com" "#; let err = Settings::from_toml(&toml_str) .expect_err("should reject asset-route origin_url with userinfo"); assert!( - format!("{err:?}").contains("origin_url must not include username or password"), + format!("{err:?}").contains("must not carry a user name or password"), "should mention the origin_url userinfo validation failure: {err:?}" ); } @@ -11108,12 +11181,13 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://assets.example.com#fragment" + source.module = "origin" + source.origin.url = "https://assets.example.com#fragment" "#; let err = Settings::from_toml(&toml_str) .expect_err("should reject asset-route origin_url with fragment"); assert!( - format!("{err:?}").contains("origin_url must not include a fragment"), + format!("{err:?}").contains("must not carry a query or a fragment"), "should mention the origin_url fragment validation failure: {err:?}" ); } @@ -11126,12 +11200,17 @@ source_domain = "partner.example.com" [[proxy.asset_routes]] prefix = "/.images/" - origin_url = "https://assets.example.com:8443" + source.module = "origin" + source.origin.url = "https://assets.example.com:8443" "#; let settings = Settings::from_toml(&toml_str).expect("should accept asset-route origin host and port"); assert_eq!( - settings.proxy.asset_routes[0].origin_url, "https://assets.example.com:8443", + settings.proxy.asset_routes[0] + .origin_source() + .expect("should select the origin source") + .url, + "https://assets.example.com:8443", "should preserve valid origin URL with non-standard port" ); } @@ -11144,9 +11223,10 @@ source_domain = "partner.example.com" allowed_domains = [" AD.EXAMPLE.COM ", " ", "*.CDN.Example.Com"] "#; let origin_key = format!( - "{}{}PUBLISHER{}ORIGIN_URL", + "{}{}SOURCE{}ORIGIN{}URL", ENVIRONMENT_VARIABLE_PREFIX, ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, ENVIRONMENT_VARIABLE_SEPARATOR ); temp_env::with_var( @@ -11196,10 +11276,15 @@ source_domain = "partner.example.com" #[test] fn settings_parses_creative_opportunities_section() { let toml = r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" + [publisher] domain = "example.com" cookie_domain = ".example.com" -origin_url = "https://origin.example.com" proxy_secret = "secret" [geo] @@ -11281,10 +11366,15 @@ formats = [{ width = 300, height = 250 }] #[test] fn settings_rejects_invalid_creative_opportunity_slot_id() { let toml = r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" + [publisher] domain = "example.com" cookie_domain = ".example.com" -origin_url = "https://origin.example.com" proxy_secret = "secret" [geo] @@ -11318,10 +11408,15 @@ formats = [{ width = 300, height = 250 }] // id injected via env is rejected by from_toml_and_env (the build-time // path uses the same validation against the merged config). let toml = r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" + [publisher] domain = "example.com" cookie_domain = ".example.com" -origin_url = "https://origin.example.com" proxy_secret = "secret" [geo] @@ -11361,10 +11456,15 @@ gam_network_id = "21765378893" fn creative_opportunity_settings_toml(slot_body: &str) -> String { format!( r#" +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" + [publisher] domain = "example.com" cookie_domain = ".example.com" -origin_url = "https://origin.example.com" proxy_secret = "secret" [geo] diff --git a/crates/trusted-server-core/src/source.rs b/crates/trusted-server-core/src/source.rs new file mode 100644 index 0000000000..d2925f8e35 --- /dev/null +++ b/crates/trusted-server-core/src/source.rs @@ -0,0 +1,671 @@ +//! Where the content the publisher path and the asset routes serve comes from. +//! +//! A source module supplies it. `[source]` selects the publisher's, and each +//! `[[proxy.asset_routes]]` entry its own in a `source` table of the same +//! shape: `module` names the source and the table beneath it holds the +//! source's settings. +//! +//! ```toml +//! [source] +//! module = "origin" +//! +//! [source.origin] +//! url = "https://origin.example.com" +//! ``` +//! +//! For each request a [`Source`] answers where its content comes from, as a +//! [`SourceLocation`]. Core fetches that location over HTTP, keys its caches +//! on it, rewrites the origin's host in the page from it and tells middleware +//! the origin host from it. +//! +//! Core supplies one source, `origin`, an HTTP origin named in its table. A +//! module offers another through +//! [`IntegrationRegistrationBuilder::with_source_module`](crate::integrations::IntegrationRegistrationBuilder::with_source_module). + +use std::fmt; +use std::sync::Arc; + +use error_stack::Report; +use http::{HeaderMap, Method}; +use serde::{Deserialize, Serialize, Serializer}; +use serde_json::{Map, Value}; +use url::Url; + +use crate::error::TrustedServerError; +use crate::provider_table::SectionModules; +use crate::redacted::{Redacted, SENSITIVE_MARKER}; +use crate::settings::{AssetOriginAuth, OriginQueryPolicy}; + +/// The section that selects the publisher's source. +pub const SOURCE_SECTION: &str = "source"; + +/// The name of the source core supplies, as a source table writes it. +pub const ORIGIN: &str = "origin"; + +/// The full name of the source core supplies. +pub const ORIGIN_MODULE: &str = "source.origin"; + +/// What a source is told about the request it answers for. +#[derive(Debug, Clone, Copy)] +pub struct SourceRequest<'a> { + /// The request's method. + pub method: &'a Method, + /// The path the request continues with, without its query. + pub path: &'a str, + /// The request's query, empty when it has none. + pub query: &'a str, + /// The request's headers. + pub headers: &'a HeaderMap, + /// The path prefix request addressing removed, if it removed one. + pub removed_prefix: Option<&'a str>, +} + +/// Where the content for a request comes from. +#[derive(Clone)] +pub struct SourceLocation { + url: String, + parsed: Url, + host_header: Option, + auth: Option, + directory_index: Option, +} + +impl fmt::Debug for SourceLocation { + // The address and the signing can name private storage and credentials, + // so only the host is shown. + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SourceLocation") + .field("host", &self.host()) + .finish_non_exhaustive() + } +} + +impl SourceLocation { + /// A location at `url`, an absolute `http` or `https` address whose path + /// is the origin's folder. + /// + /// # Errors + /// + /// When `url` is not one [`validate_url`] accepts. + pub fn new(url: impl Into) -> Result> { + let url = url.into().trim().to_owned(); + let parsed = validate_url(&url) + .map_err(|message| Report::new(TrustedServerError::Configuration { message }))?; + Ok(Self { + url, + parsed, + host_header: None, + auth: None, + directory_index: None, + }) + } + + /// The same location sending `host_header` as the `Host`. + #[must_use] + pub fn with_host_header(mut self, host_header: Option) -> Self { + self.host_header = host_header; + self + } + + /// The same location signing each request with `auth`. + #[must_use] + pub fn with_auth(mut self, auth: Option) -> Self { + self.auth = auth; + self + } + + /// The same location adding `directory_index` to a path ending in `/`. + #[must_use] + pub fn with_directory_index(mut self, directory_index: Option) -> Self { + self.directory_index = directory_index; + self + } + + /// The address as written. + #[must_use] + pub fn url(&self) -> &str { + &self.url + } + + /// The address, parsed. + #[must_use] + pub fn parsed(&self) -> &Url { + &self.parsed + } + + /// The origin's host, with its port where the address names one. + #[must_use] + pub fn host(&self) -> String { + match (self.parsed.host_str(), self.parsed.port()) { + (Some(host), Some(port)) => format!("{host}:{port}"), + (Some(host), None) => host.to_owned(), + (None, _) => self.url.clone(), + } + } + + /// The `Host` header sent to the origin. + #[must_use] + pub fn host_header(&self) -> String { + self.host_header.clone().unwrap_or_else(|| self.host()) + } + + /// The `Host` header written for this location, where it is not the + /// address's host. + #[must_use] + pub fn host_header_override(&self) -> Option<&str> { + self.host_header.as_deref() + } + + /// The signing applied to each request, if any. + #[must_use] + pub fn auth(&self) -> Option<&AssetOriginAuth> { + self.auth.as_ref() + } + + /// The file added to a path ending in `/`, if any. + #[must_use] + pub fn directory_index(&self) -> Option<&str> { + self.directory_index.as_deref() + } + + /// Whether the reader's query is left off the request, as the signing + /// says. + #[must_use] + pub fn query_is_stripped(&self) -> bool { + self.auth + .as_ref() + .and_then(AssetOriginAuth::origin_query_policy) + == Some(OriginQueryPolicy::Strip) + } + + /// What tells this location's responses apart from another's in a cache: + /// the address and the `Host` it is asked with. + #[must_use] + pub fn identity(&self) -> String { + format!("{}\0{}", self.url, self.host_header()) + } +} + +/// A source of content, built from its settings when the application is +/// built. +pub trait Source: Send + Sync + fmt::Debug { + /// Where the content for `request` comes from. + /// + /// # Errors + /// + /// When the source cannot say where the request's content comes from. + /// Core answers the request with the error's status. + fn locate( + &self, + request: &SourceRequest<'_>, + ) -> Result>; +} + +/// Builds the source a settings table configures. +pub trait SourceFactory: Send + Sync { + /// The source `settings` configure, where `setting` names the table in a + /// refusal. + /// + /// # Errors + /// + /// When the table cannot be read or fails validation. + fn build( + &self, + settings: &Map, + setting: &str, + ) -> Result, Report>; +} + +/// The settings of core's `origin` source. +#[derive(Clone, Default, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct OriginSettings { + /// The origin's absolute `http` or `https` address. A path names the + /// folder requests are sent under. + #[serde(serialize_with = "crate::redacted::sensitive")] + pub url: String, + /// The `Host` header to send, where it is not the address's host. + #[serde( + default, + skip_serializing_if = "Option::is_none", + serialize_with = "crate::redacted::sensitive" + )] + pub host_header: Option, + /// The file added to a request path ending in `/`, for an origin such as + /// object storage that does not answer a folder with its index file. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub directory_index: Option, + /// Signing of every request, for an origin that answers only signed + /// reads. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub auth: Option, +} + +impl fmt::Debug for OriginSettings { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("OriginSettings") + .field("url", &Redacted::new(self.url.clone())) + .field("host_header", &self.host_header) + .field("directory_index", &self.directory_index) + .field("auth", &self.auth) + .finish() + } +} + +impl OriginSettings { + /// Settings for the origin at `url`, with nothing else set. + #[must_use] + pub fn new(url: impl Into) -> Self { + Self { + url: url.into(), + ..Self::default() + } + } + + /// A source table selecting core's `origin` with these settings, as + /// `[source]` or a route's `source` holds it. + #[must_use] + pub fn selection(&self) -> SectionModules { + let table = match serde_json::to_value(self) { + Ok(Value::Object(table)) => table, + _ => Map::new(), + }; + SectionModules::one(ORIGIN, table) + } + + /// Reads the table at `setting` as written, trimmed, without checking + /// what it names, so a check that runs before the table is used can see + /// it. + /// + /// # Errors + /// + /// When the table has an unknown key or a value of the wrong type, naming + /// `setting`. + pub fn parse( + settings: &Map, + setting: &str, + ) -> Result> { + let mut origin: Self = + serde_json::from_value(Value::Object(settings.clone())).map_err(|error| { + Report::new(TrustedServerError::Configuration { + message: format!("[{setting}] could not be read: {error}"), + }) + })?; + origin.url = origin.url.trim().to_owned(); + if let Some(auth) = &mut origin.auth { + auth.normalize(); + } + Ok(origin) + } + + /// Reads and checks the table at `setting`. + /// + /// # Errors + /// + /// When the table cannot be read, or names an address, a `Host`, an index + /// file or signing core cannot use, naming `setting`. + pub fn read( + settings: &Map, + setting: &str, + ) -> Result> { + let origin = Self::parse(settings, setting)?; + origin.check(setting)?; + Ok(origin) + } + + fn check(&self, setting: &str) -> Result<(), Report> { + let refuse = |message: String| { + Report::new(TrustedServerError::Configuration { + message: format!("[{setting}] {message}"), + }) + }; + let parsed = validate_url(&self.url).map_err(&refuse)?; + if let Some(host_header) = &self.host_header { + crate::settings::check_host_header(host_header) + .map_err(|reason| refuse(format!("host_header {reason}")))?; + } + if let Some(index) = &self.directory_index { + crate::settings::check_directory_index(index) + .map_err(|reason| refuse(format!("directory_index {reason}")))?; + } + if let Some(auth) = &self.auth { + if parsed.scheme() != "https" { + return Err(refuse("signs with auth, so url must use https".to_owned())); + } + auth.prepare_runtime(&format!("{setting}.auth"))?; + } + Ok(()) + } + + /// Whether the address is the unedited template placeholder host. + #[must_use] + pub fn is_placeholder(&self) -> bool { + crate::settings::Publisher::is_placeholder_origin_url(&self.url) + } + + /// The location these settings name. + /// + /// # Errors + /// + /// When the address cannot be read, which [`Self::read`] has already + /// refused for settings it returned. + pub fn location(&self) -> Result> { + Ok(SourceLocation::new(self.url.clone())? + .with_host_header(self.host_header.clone()) + .with_auth(self.auth.clone()) + .with_directory_index(self.directory_index.clone())) + } +} + +/// Core's source, an HTTP origin named in its table, the same for every +/// request. +#[derive(Debug, Clone)] +pub struct OriginSource { + location: SourceLocation, +} + +impl OriginSource { + /// The source `settings` name. + /// + /// # Errors + /// + /// When the settings' address cannot be read. + pub fn new(settings: &OriginSettings) -> Result> { + Ok(Self { + location: settings.location()?, + }) + } +} + +impl Source for OriginSource { + fn locate( + &self, + _request: &SourceRequest<'_>, + ) -> Result> { + Ok(self.location.clone()) + } +} + +/// Builds [`OriginSource`] from an `origin` table. +#[derive(Debug, Clone, Copy, Default)] +pub struct OriginFactory; + +impl SourceFactory for OriginFactory { + fn build( + &self, + settings: &Map, + setting: &str, + ) -> Result, Report> { + let origin = OriginSettings::read(settings, setting)?; + Ok(Arc::new(OriginSource::new(&origin)?)) + } +} + +/// The one source `selection` names, as written, with its settings table, or +/// a refusal naming `setting`. +/// +/// # Errors +/// +/// When the table selects no source, or more than one. +pub fn selected<'a>( + selection: &'a SectionModules, + setting: &str, +) -> Result<(&'a str, Map), Report> { + match selection.selected() { + [one] => Ok(( + one.as_str(), + selection.settings_of(one).cloned().unwrap_or_default(), + )), + [] => Err(Report::new(TrustedServerError::Configuration { + message: format!( + "[{setting}] names no source. Write `module = \"origin\"` and the origin's \ + address under [{setting}.origin] url" + ), + })), + _ => Err(Report::new(TrustedServerError::Configuration { + message: format!("[{setting}] names more than one source. Write `module` for one"), + })), + } +} + +/// Checks a selection of core's `origin` source, which needs no module on +/// offer to read. A selection of another source is checked where the +/// registry builds it. +/// +/// # Errors +/// +/// When the selection names no source or several, or its `origin` table +/// cannot be used. +pub fn check_selection( + selection: &SectionModules, + setting: &str, +) -> Result<(), Report> { + let (name, table) = selected(selection, setting)?; + if name == ORIGIN { + OriginSettings::read(&table, &format!("{setting}.{ORIGIN}"))?; + } + Ok(()) +} + +/// Serializes a source selection, for +/// `#[serde(serialize_with = "crate::source::serialize_selection")]`. While +/// [`crate::redacted::to_marked_value`] runs, the `origin` table's `url` and +/// `host_header` are marked sensitive, as they would be read through +/// [`OriginSettings`], so the configuration page masks them by default. +/// +/// # Errors +/// +/// As the serializer's own. +pub fn serialize_selection(selection: &SectionModules, serializer: S) -> Result +where + S: Serializer, +{ + if !crate::redacted::is_marking() { + return selection.serialize(serializer); + } + let mut value = serde_json::to_value(selection).map_err(serde::ser::Error::custom)?; + if let Some(Value::Object(origin)) = value.get_mut(ORIGIN) { + for key in ["url", "host_header"] { + if let Some(field) = origin.remove(key) { + let mut marked = Map::new(); + marked.insert(SENSITIVE_MARKER.to_owned(), field); + origin.insert(key.to_owned(), Value::Object(marked)); + } + } + } + value.serialize(serializer) +} + +/// Whether `url` is an address a source location can be at: absolute `http` +/// or `https` with a host, no credentials, query, fragment or trailing `/`. +/// A path names a folder. +/// +/// # Errors +/// +/// A message naming what is wrong. +pub fn validate_url(url: &str) -> Result { + if url.ends_with('/') { + return Err(format!("url `{url}` must not end in `/`")); + } + let parsed = Url::parse(url) + .map_err(|error| format!("url `{url}` is not an absolute address: {error}"))?; + if !matches!(parsed.scheme(), "http" | "https") { + return Err(format!("url `{url}` must use http or https")); + } + if parsed.host_str().is_none() { + return Err(format!("url `{url}` must name a host")); + } + if !parsed.username().is_empty() || parsed.password().is_some() { + return Err(format!( + "url `{url}` must not carry a user name or password" + )); + } + if parsed.query().is_some() || parsed.fragment().is_some() { + return Err(format!("url `{url}` must not carry a query or a fragment")); + } + Ok(parsed) +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::*; + + fn table(value: Value) -> Map { + match value { + Value::Object(table) => table, + _ => unreachable!("the tests write objects"), + } + } + + #[test] + fn a_location_host_keeps_a_port_the_address_names() { + for (url, host) in [ + ("https://origin.example.com:8080", "origin.example.com:8080"), + ("https://origin.example.com", "origin.example.com"), + ("https://origin.example.com:443", "origin.example.com"), + ("http://localhost:9090", "localhost:9090"), + ("http://192.168.1.1:8080", "192.168.1.1:8080"), + ("http://[::1]:8080", "[::1]:8080"), + ] { + let location = SourceLocation::new(url).expect("should accept the address"); + assert_eq!(location.host(), host, "the host of `{url}`"); + } + } + + #[test] + fn a_location_refuses_an_address_core_cannot_fetch() { + for url in [ + "localhost:9090", + "ftp://assets.example.com", + "https://assets.example.com/", + "https://user:pass@assets.example.com", + "https://assets.example.com?token=abc", + "https://assets.example.com#fragment", + "/relative/path", + ] { + assert!(SourceLocation::new(url).is_err(), "should refuse `{url}`"); + } + } + + #[test] + fn a_location_path_names_a_folder() { + let location = SourceLocation::new("https://storage.example.com/bucket/site") + .expect("should accept a folder"); + assert_eq!(location.parsed().path(), "/bucket/site"); + assert_eq!(location.host(), "storage.example.com"); + } + + #[test] + fn the_host_header_defaults_to_the_address_host() { + let location = + SourceLocation::new("https://origin.example.com:8443").expect("should accept"); + assert_eq!(location.host_header(), "origin.example.com:8443"); + assert_eq!(location.host_header_override(), None); + + let location = location.with_host_header(Some("www.example.com".to_owned())); + assert_eq!(location.host_header(), "www.example.com"); + assert_eq!(location.host_header_override(), Some("www.example.com")); + } + + #[test] + fn the_identity_tells_host_headers_apart() { + let plain = SourceLocation::new("https://origin.example.com").expect("should accept"); + let named = plain + .clone() + .with_host_header(Some("www.example.com".to_owned())); + assert_ne!(plain.identity(), named.identity()); + assert_eq!( + plain.identity(), + "https://origin.example.com\0origin.example.com", + "the identity should match the key the caches used before sources" + ); + } + + #[test] + fn origin_settings_read_trim_and_check() { + let origin = OriginSettings::read( + &table(json!({ "url": " https://origin.example.com ", "host_header": "www.example.com" })), + "source.origin", + ) + .expect("should read the table"); + assert_eq!(origin.url, "https://origin.example.com"); + assert_eq!(origin.host_header.as_deref(), Some("www.example.com")); + + for (value, expected) in [ + ( + json!({ "url": "https://origin.example.com", "host_headr": "x" }), + "host_headr", + ), + (json!({ "host_header": "www.example.com" }), "url"), + ( + json!({ "url": "https://origin.example.com", "host_header": "https://x" }), + "[source.origin] host_header", + ), + ( + json!({ "url": "https://origin.example.com", "directory_index": "a/b.html" }), + "[source.origin] directory_index", + ), + ( + json!({ "url": "http://origin.example.com", "auth": { "type": "s3_sigv4", "region": "test-region-1" } }), + "[source.origin] signs with auth, so url must use https", + ), + ] { + let err = OriginSettings::read(&table(value.clone()), "source.origin") + .expect_err("should refuse the table"); + assert!( + format!("{err:?}").contains(expected), + "the refusal of {value} should say `{expected}`: {err:?}" + ); + } + } + + #[test] + fn a_selection_reads_back_as_the_settings_it_was_built_from() { + let origin = OriginSettings { + host_header: Some("www.example.com".to_owned()), + directory_index: Some("index.html".to_owned()), + ..OriginSettings::new("https://storage.example.com/site") + }; + let selection = origin.selection(); + + let (name, table) = selected(&selection, SOURCE_SECTION).expect("should select one"); + assert_eq!(name, ORIGIN); + let read = OriginSettings::read(&table, "source.origin").expect("should read back"); + assert_eq!(read.url, origin.url); + assert_eq!(read.host_header, origin.host_header); + assert_eq!(read.directory_index, origin.directory_index); + check_selection(&selection, SOURCE_SECTION).expect("should pass the check"); + } + + #[test] + fn a_selection_must_name_exactly_one_source() { + let err = check_selection(&SectionModules::default(), SOURCE_SECTION) + .expect_err("should refuse a section naming no source"); + assert!(format!("{err:?}").contains("names no source"), "{err:?}"); + } + + #[test] + fn the_origin_source_answers_the_same_location_for_every_request() { + let source = OriginFactory + .build( + &table(json!({ "url": "https://origin.example.com" })), + "source.origin", + ) + .expect("should build"); + let headers = HeaderMap::new(); + let location = source + .locate(&SourceRequest { + method: &Method::GET, + path: "/any/path", + query: "a=1", + headers: &headers, + removed_prefix: Some("/www.example.com"), + }) + .expect("should locate"); + assert_eq!(location.url(), "https://origin.example.com"); + } +} diff --git a/crates/trusted-server-core/src/test_support.rs b/crates/trusted-server-core/src/test_support.rs index 840d07dd9e..12d1e27164 100644 --- a/crates/trusted-server-core/src/test_support.rs +++ b/crates/trusted-server-core/src/test_support.rs @@ -6,15 +6,68 @@ pub mod tests { use crate::redacted::Redacted; use crate::settings::{Ec, EcModuleBlock, HmacModuleConfig, HostSignalsModuleConfig, Settings}; + /// Changes the publisher's `origin` source settings with `edit`. + /// + /// # Panics + /// + /// When `settings` do not select core's `origin` source. + pub fn edit_origin( + settings: &mut Settings, + edit: impl FnOnce(&mut crate::source::OriginSettings), + ) { + let mut origin = settings + .origin_source() + .expect("settings should select the origin source"); + edit(&mut origin); + settings.set_origin_source(&origin); + } + + /// Sets the signing of `route`'s `origin` source to `auth`. + /// + /// # Panics + /// + /// When `route` does not select core's `origin` source. + pub fn set_route_auth( + route: &mut crate::settings::ProxyAssetRoute, + auth: Option, + ) { + let mut origin = route + .origin_source() + .expect("the route should select the origin source"); + origin.auth = auth; + route.set_origin_source(&origin); + } + + /// The host of the publisher's `origin` source, with its port where its + /// address names one. + /// + /// # Panics + /// + /// When `settings` do not select core's `origin` source. + #[must_use] + pub fn origin_host(settings: &Settings) -> String { + settings + .origin_source() + .expect("settings should select the origin source") + .location() + .expect("the origin source should give a location") + .host() + } + #[must_use] pub fn crate_test_settings_str() -> String { r#" [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [geo] # A gdpr-eu country, where every permission requires a signal. This # reproduces the prior no-default floor, so existing tests are @@ -363,9 +416,14 @@ pub mod nextjs_auction { [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "fixture-test-proxy-secret" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [ec] passphrase = "test-secret-key-32-bytes-minimum" diff --git a/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml index e49c170a2a..38971c4440 100644 --- a/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml +++ b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml @@ -1,7 +1,12 @@ +[source] +module = "origin" + +[source.origin] +url = "http://127.0.0.1:8888" + [publisher] domain = "localhost" cookie_domain = "localhost" -origin_url = "http://127.0.0.1:8888" proxy_secret = "integration_proxy_secret" [geo] diff --git a/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs b/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs index f5bb7e80f5..f080966466 100644 --- a/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs +++ b/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs @@ -119,7 +119,11 @@ fn build_app_config_envelope( .map_err(|error| error_box(format!("invalid Trusted Server app config: {error}")))?; let mut settings = app_config.into_settings(); if let Some(origin_url) = origin_url { - settings.publisher.origin_url = origin_url.to_string(); + let mut origin = settings.origin_source().ok_or_else(|| { + error_box("the app config should select the origin source".to_string()) + })?; + origin.url = origin_url.to_string(); + settings.set_origin_source(&origin); } // The fixture selects modules a stock build ships, so validation needs // the stock list as the adapters and the `ts` tool have it. @@ -325,7 +329,11 @@ mod tests { .expect("should verify blob"); assert_eq!( - settings.publisher.origin_url, "http://127.0.0.1:9999", + settings + .origin_source() + .expect("should select the origin source") + .url, + "http://127.0.0.1:9999", "should apply origin override before envelope creation" ); } diff --git a/crates/trusted-server-integration-tests/tests/common/config.rs b/crates/trusted-server-integration-tests/tests/common/config.rs index 0ee777c35e..6f1fa45270 100644 --- a/crates/trusted-server-integration-tests/tests/common/config.rs +++ b/crates/trusted-server-integration-tests/tests/common/config.rs @@ -32,7 +32,12 @@ fn app_config_envelope( )) })?; let mut settings = app_config.into_settings(); - settings.publisher.origin_url = origin_url; + let mut origin = settings.origin_source().ok_or_else(|| { + Report::new(TestError::ConfigGeneration) + .attach("the integration config should select the origin source") + })?; + origin.url = origin_url; + settings.set_origin_source(&origin); // The fixture selects modules a stock build ships, so validation needs // the stock list as the adapters and the `ts` tool have it. trusted_server_core::config::register_deploy_integrations(trusted_server_modules::builders()); diff --git a/crates/trusted-server-integration-tests/tests/common/runtime.rs b/crates/trusted-server-integration-tests/tests/common/runtime.rs index 048c76d44b..00e9b0f30d 100644 --- a/crates/trusted-server-integration-tests/tests/common/runtime.rs +++ b/crates/trusted-server-integration-tests/tests/common/runtime.rs @@ -123,7 +123,7 @@ pub fn wasm_binary_path() -> PathBuf { /// Get the fixed origin port used for Docker container port mapping. /// /// This must match the port baked into the WASM binary via -/// `TRUSTED_SERVER__PUBLISHER__ORIGIN_URL` at build time. +/// `TRUSTED_SERVER__SOURCE__ORIGIN__URL` at build time. pub fn origin_port() -> u16 { match std::env::var("INTEGRATION_ORIGIN_PORT") { Ok(value) => value diff --git a/crates/trusted-server-integration-tests/tests/parity.rs b/crates/trusted-server-integration-tests/tests/parity.rs index 65b8f9fabd..4815e2d393 100644 --- a/crates/trusted-server-integration-tests/tests/parity.rs +++ b/crates/trusted-server-integration-tests/tests/parity.rs @@ -31,10 +31,15 @@ use trusted_server_core::test_support::nextjs_auction; fn test_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "parity-test-proxy-secret" [ec] diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md index a265acdbb4..181fa9833a 100644 --- a/docs/guide/api-reference.md +++ b/docs/guide/api-reference.md @@ -116,13 +116,18 @@ Endpoint-specific CORS: `Access-Control-Allow-Origin: *`. ```json { - "masked": ["publisher.origin_url", "publisher.proxy_secret"], + "masked": ["publisher.proxy_secret", "source.origin.url"], "settings": { "publisher": { "cookie_domain": ".example.com", "domain": "example.com", - "origin_url": "XXXX", "proxy_secret": "XXXX" + }, + "source": { + "module": "origin", + "origin": { + "url": "XXXX" + } } }, "version": "0.1.0" diff --git a/docs/guide/asset-routes.md b/docs/guide/asset-routes.md index 405bc1d1bb..8499812d28 100644 --- a/docs/guide/asset-routes.md +++ b/docs/guide/asset-routes.md @@ -12,7 +12,7 @@ For a matching `GET` or `HEAD` request, Trusted Server runs this sequence: 2. Optionally rewrite the request path with `path_pattern` and `target_path`. 3. Build Image Optimizer metadata from the request query when the route enables it. 4. Apply the origin query policy. -5. Add origin authentication headers when the route configures auth. +5. Add origin authentication headers when the route's source configures auth. 6. Send the request to the resolved backend origin. 7. Return the origin status, body, and safe headers. @@ -20,12 +20,19 @@ The origin query policy runs before S3 signing, so the signature covers the exac ## Basic route +Each route names where its content comes from in a `source` table, in the same shape as the publisher's [`[source]`](/guide/configuration#source). `module = "origin"` selects core's HTTP origin, and `[proxy.asset_routes.source.origin]` holds its settings. Write the route's own keys, `prefix`, `path_pattern` and `target_path`, before the source tables, because a table header ends the route's own keys. + ```toml [proxy] [[proxy.asset_routes]] prefix = "/assets/" -origin_url = "https://assets.example.com" + +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://assets.example.com" ``` A request for `/assets/logo.png?v=1` is sent to: @@ -35,7 +42,9 @@ https://assets.example.com/assets/logo.png?v=1 ``` Plain asset routes preserve the incoming query string by default. -`origin_url` must be only an absolute `http` or `https` origin (scheme, host, and optional port), without userinfo, path, query, or fragment components. +`url` must be an absolute `http` or `https` address with a host, without a user name or password, a query, a fragment, or a trailing `/`. A path in `url` names a folder, and the route's target path is added under it. With `url = "https://storage.example.com/bucket/site"`, a request for `/assets/logo.png` is sent to `https://storage.example.com/bucket/site/assets/logo.png`. + +`host_header` sets the outbound `Host` header where it is not the host of `url`. Trusted Server still connects, uses SNI, and checks certificates against the host in `url`. ## Path rewrite @@ -44,23 +53,33 @@ Use `path_pattern` and `target_path` when the public path shape differs from the ```toml [[proxy.asset_routes]] prefix = "/.image/" -origin_url = "https://assets-cdn.example.com" path_pattern = "^/\\.image/(.*)/[^/]+\\.([^/.]+)$" target_path = "/image/upload/$1.$2" + +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://assets-cdn.example.com" ``` -Both fields must be configured together. The rewritten path must start with `/`. +Both fields must be configured together. The rewritten path must start with `/`. When `url` has a path, the rewritten path is added under that folder. ## Private S3 origins -Add an auth block when the asset origin is a private S3 bucket. +Add an auth block to the route's source when the asset origin is a private S3 bucket. ```toml [[proxy.asset_routes]] prefix = "/.image/" -origin_url = "https://bucket.s3.us-east-1.amazonaws.com" -[proxy.asset_routes.auth] +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://bucket.s3.us-east-1.amazonaws.com" + +[proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "us-east-1" origin_query = "strip" @@ -71,8 +90,8 @@ secret_access_key = "s3_secret_access_key" ### S3 requirements -- `origin_url` must be the real S3 host that AWS validates in the SigV4 canonical request. -- Use `https` origins for authenticated routes. `http` origins are accepted by the generic asset route validator, but they would send SigV4 headers in clear text. +- `url` must be the real S3 host that AWS validates in the SigV4 canonical request. +- `url` must use `https` when the route signs. A signing table the signer cannot use, such as an empty or malformed `region`, is refused at startup naming the route, as in `proxy.asset_routes /.image/ source.origin.auth`. - S3 support is for `GET` and `HEAD` asset reads. - Signing uses header-based AWS SigV4, not presigned URLs. - The signer uses `x-amz-content-sha256: UNSIGNED-PAYLOAD`. @@ -103,7 +122,7 @@ Place those values in the logical `trusted_server_secrets` store. Adapter config Precedence: -1. `proxy.asset_routes.auth.origin_query`, when auth is configured. +1. `proxy.asset_routes.source.origin.auth.origin_query`, when auth is configured. 2. `proxy.asset_routes.image_optimizer.origin_query`, when Image Optimizer is enabled. 3. The route default. @@ -148,9 +167,14 @@ when_missing = "smart" [[proxy.asset_routes]] prefix = "/.image/" -origin_url = "https://bucket.s3.us-east-1.amazonaws.com" -[proxy.asset_routes.auth] +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://bucket.s3.us-east-1.amazonaws.com" + +[proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "us-east-1" origin_query = "strip" diff --git a/docs/guide/configuration-rules.md b/docs/guide/configuration-rules.md index abb932fba9..49a0ca516b 100644 --- a/docs/guide/configuration-rules.md +++ b/docs/guide/configuration-rules.md @@ -8,10 +8,10 @@ request, and why the file is built this way. ## One rule for every type Everything Trusted Server can switch on is a module, selected the same way: -the Edge Cookie identity, the location and device lookups, the permission -signals, the demand sources in an auction, the ad server that picks the -winner, the rules in `robots.txt`, where auction telemetry is sent, and the -page integrations. +where the publisher's pages come from, the Edge Cookie identity, the location +and device lookups, the permission signals, the demand sources in an auction, +the ad server that picks the winner, the rules in `robots.txt`, where auction +telemetry is sent, and the page integrations. ```toml [] @@ -22,10 +22,9 @@ setting = "value" ``` 1. **The type is the job.** Each type is one top-level table, named for what - its modules do: `ec`, `geo`, `device`, `permission-signal`, `demand`, - `ad-server`, `robots-txt`, `ads-txt` and `analytics`, and the section of each - module - type, such as `cmp` or `tag`. + its modules do: `source`, `ec`, `geo`, `device`, `permission-signal`, + `demand`, `ad-server`, `robots-txt`, `ads-txt` and `analytics`, and the + section of each module type, such as `cmp` or `tag`. 2. **The selector chooses what runs.** A type that runs one module takes `module`, a string. A type that runs several takes `modules`, a list. 3. **`[.]` holds the settings.** A name with nothing to set @@ -48,6 +47,7 @@ setting = "value" | Type | Runs | Selector | Implementations in this repository | | ------------------- | ----------------- | ------------------ | ------------------------------------------------------------------------------------------------------- | +| `source` | one | `module`, a string | `origin`, an HTTP origin, or a module that supplies a source | | `ec` | one | `module`, a string | `hmac`, `host_signals`, `client_fixed` (demonstration builds), or an integration that supplies identity | | `geo` | one | `module`, a string | `platform`, `none`, or an integration that supplies location | | `device` | one | `module`, a string | `builtin` (the default), `fastly`, or an integration that supplies device signals | @@ -83,6 +83,7 @@ partner registry and the cluster thresholds, and `[geo]` holds | Type | With no `module` or `modules` line | | ----------------------------------------------- | --------------------------------------------------------------------------------- | +| `source` | the configuration is refused, because `[source]` is required | | `ec` | no Edge Cookie is created | | `geo` | no location is resolved and no host geo service is called | | `device` | `builtin` runs, which reads the User-Agent only | @@ -166,7 +167,7 @@ They keep their own keys and have no selector. | Table | Configures | | ------------------------------------------ | --------------------------------------------------------------------- | -| `[publisher]` | the site, its origin and the proxy secret | +| `[publisher]` | the site and the proxy secret | | `[auction]` | whether auctions run, the whole-auction timeout and creative handling | | `[auction.bidders.]` | which demand provider a browser bidder code is sent to | | `[creative_opportunities]` | server-rendered ad slots | @@ -203,9 +204,14 @@ A site with every kind of component the rules cover. [publisher] domain = "example.com" cookie_domain = ".example.com" -origin_url = "https://origin.example.com" proxy_secret = "publisher_proxy_secret" # key name in trusted_server_secrets +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" + [proxy] allowed_domains = ["assets.example.com"] diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 67837a4885..0ff62313a1 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -27,17 +27,23 @@ setting = "value" ### Minimal Configuration -Create `trusted-server.toml` in your project root. Generate both secret values -first with `openssl rand -base64 32`. The placeholders below are intentionally -rejected until replaced. +Create `trusted-server.toml` in your project root. Two sections are required, +`[publisher]` for the site and [`[source]`](#source) for where its pages come +from. Generate both secret values first with `openssl rand -base64 32`. The +placeholders below are intentionally rejected until replaced. ```toml [publisher] domain = "publisher.com" cookie_domain = ".publisher.com" -origin_url = "https://origin.publisher.com" proxy_secret = "publisher_proxy_secret" +[source] +module = "origin" + +[source.origin] +url = "https://origin.publisher.com" + [ec] module = "hmac" @@ -54,7 +60,7 @@ read by the deployed application at request time. ```bash # Format: TRUSTED_SERVER__SECTION__FIELD export TRUSTED_SERVER__PUBLISHER__DOMAIN=publisher.com -export TRUSTED_SERVER__PUBLISHER__ORIGIN_URL=https://origin.publisher.com +export TRUSTED_SERVER__SOURCE__ORIGIN__URL=https://origin.publisher.com # Secret overrides, when needed, are key names, not secret values. export TRUSTED_SERVER__PUBLISHER__PROXY_SECRET=publisher_proxy_secret export TRUSTED_SERVER__EC__MODULE=hmac @@ -79,13 +85,14 @@ publisher, trusted-client-IP, EC, Tinybird, DataDome, and S3 fields: - `analytics.tinybird.auction_token_secret`, when `[analytics] module = "tinybird"` - `bot-protection.datadome.server_side_key_secret_name`, when protection is enabled - `bot-protection.datadome.protection_test_bypass.credential_secret_name`, when the bypass is enabled -- `proxy.asset_routes[*].auth.access_key_id`, `secret_access_key`, and optional `session_token` -- `publisher.origin_auth.access_key_id`, `secret_access_key`, and optional `session_token`, when `[publisher]` sets `origin_auth` +- `proxy.asset_routes[*].source.origin.auth.access_key_id`, `secret_access_key`, and optional `session_token`, when a route's `[proxy.asset_routes.source.origin]` sets `auth` +- `source.origin.auth.access_key_id`, `secret_access_key`, and optional `session_token`, when `[source.origin]` sets `auth` Their values belong in the logical `trusted_server_secrets` store and are resolved only while an instance builds runtime settings. An adapter can map the logical ID to a different physical name. For example, Fastly commonly maps -`trusted_server_secrets` to physical store `ts_secrets`. +`trusted_server_secrets` to physical store `ts_secrets`. A blank +`session_token` in either signing table is read as no token. Two accepted secret-shaped fields are deliberately different: @@ -213,7 +220,7 @@ fail and the service will return its startup-error response. ## Key Sections -12 of these sections select what runs, with `module` where one runs and +13 of these sections select what runs, with `module` where one runs and `modules` where several run, and each gives every selected name its own `[.]` settings table, as [Configuration Rules](/guide/configuration-rules) describes. @@ -238,13 +245,14 @@ fail and the service will return its startup-error response. | `[permission-signal]` | several modules | Which permission signals are acted on, in order | | `[prefix]` | one module | Which module names the browser's cookies and the shared store keys | | `[proxy]` | several modules | Proxy allowlist, TLS policy, asset routes, and the first-party script proxy module | -| `[publisher]` | nothing | Publisher domain, origin, and proxy signing key | +| `[publisher]` | nothing | Publisher domain, cookie domain, and proxy signing key | | `[request_signing]` | nothing | Outbound Ed25519 request signing | | `[response_headers]` | nothing | Headers added to Trusted Server responses | | `[rewrite]` | nothing | First-party URL rewrite exclusions | | `[robots-txt]` | several modules | What `/robots.txt` answers, and `X-Robots-Tag` on its responses when every crawler is refused | | `[[serve]]` | nothing | Which page changes run on each reader's copy of a page, and in what order | | `[shared]` | nothing | What one vendor's modules share, a table per vendor, which runs that vendor's builder | +| `[source]` | one module | Where the publisher's pages come from | | `[tester_cookie]` | nothing | Optional tester-cookie endpoints | | `[trusted_client_ip]` | nothing | Authenticated front-door client-IP forwarding | | `[]`, such as `[cmp]`, `[tag]`, `[ad-tag]`, `[bot-protection]`, `[identity]`, `[audience]` or `[framework]` | one module or several | The section of a module type, selecting the modules of that type that run | @@ -258,9 +266,14 @@ deployment. [publisher] domain = "publisher.com" cookie_domain = ".publisher.com" -origin_url = "https://origin.publisher.com" proxy_secret = "publisher_proxy_secret" +[source] +module = "origin" + +[source.origin] +url = "https://origin.publisher.com" + [ec] module = "hmac" @@ -344,20 +357,19 @@ this way. ## Publisher Configuration -Core publisher settings for domain, origin, and proxy configuration. +Core publisher settings for the domain, cookies, and proxy signing, and the +source the publisher's pages come from. ### `[publisher]` -| Field | Type | Required | Description | -| ----------------------------- | ------- | -------- | --------------------------------------------------------------------------- | -| `domain` | String | Yes | Publisher's apex domain name | -| `cookie_domain` | String | Yes | Domain for non-EC cookies (typically with leading dot) | -| `origin_url` | String | Yes | Publisher origin URL, whose path (if any) comes before the request path | -| `origin_host_header_override` | String | No | Outbound Host header to send while connecting to `origin_url` | -| `proxy_secret` | String | Yes | Secret-store key name for the proxy URL secret | -| `max_buffered_body_bytes` | Integer | No | Buffered-body cap / Fastly stream raw+decoded byte ceiling (default 16 MiB) | -| `origin_auth` | Table | No | Signs every origin request with the table an asset route's `auth` takes | -| `directory_index` | String | No | File name added to an origin request path that ends in `/` | +| Field | Type | Required | Description | +| ------------------------- | ------- | -------- | --------------------------------------------------------------------------- | +| `domain` | String | Yes | Publisher's apex domain name | +| `cookie_domain` | String | Yes | Domain for non-EC cookies (typically with leading dot) | +| `proxy_secret` | String | Yes | Secret-store key name for the proxy URL secret | +| `max_buffered_body_bytes` | Integer | No | Buffered-body cap / Fastly stream raw+decoded byte ceiling (default 16 MiB) | + +Where the publisher's pages come from is set in [`[source]`](#source). > **Note:** EC cookies (`ts-ec`) derive their domain automatically as `.{domain}` and > do not use `cookie_domain`. The `cookie_domain` field is used by other cookie helpers. @@ -368,9 +380,6 @@ Core publisher settings for domain, origin, and proxy configuration. [publisher] domain = "publisher.com" cookie_domain = ".publisher.com" -origin_url = "https://origin.publisher.com" -# Optional: connect to origin_url but send this outbound Host header. -# origin_host_header_override = "www.publisher.com" proxy_secret = "publisher_proxy_secret" ``` @@ -379,8 +388,6 @@ proxy_secret = "publisher_proxy_secret" ```bash TRUSTED_SERVER__PUBLISHER__DOMAIN=publisher.com TRUSTED_SERVER__PUBLISHER__COOKIE_DOMAIN=.publisher.com -TRUSTED_SERVER__PUBLISHER__ORIGIN_URL=https://origin.publisher.com -TRUSTED_SERVER__PUBLISHER__ORIGIN_HOST_HEADER_OVERRIDE=www.publisher.com TRUSTED_SERVER__PUBLISHER__PROXY_SECRET=publisher_proxy_secret TRUSTED_SERVER__PUBLISHER__MAX_BUFFERED_BODY_BYTES=16777216 ``` @@ -419,70 +426,6 @@ TRUSTED_SERVER__PUBLISHER__MAX_BUFFERED_BODY_BYTES=16777216 **Best Practice**: Use leading dot (`.publisher.com`) for subdomain sharing. -#### `origin_url` - -**Purpose**: Backend origin server URL for publisher content. - -**Usage**: - -- Fallback proxy target for non-integration requests -- HTML processing rewrites origin URLs to request host -- Base for relative URL resolution - -**Format**: Full URL with protocol - -- ✅ `https://origin.publisher.com` -- ✅ `https://origin.publisher.com:8080` -- ✅ `http://192.168.1.1:9000` -- ✅ `https://storage.publisher.com/bucket/site` (a folder of object storage) -- ❌ `origin.publisher.com` (missing protocol) -- ❌ `https://origin.publisher.com/` (trailing slash) - -**Port Handling**: Includes port if non-standard (not 80/443). - -**Path Handling**: A path in `origin_url` is kept, and the request path is -added after it with exactly one `/` between them, so a request for `/news/page` -with the folder above is sent to `/bucket/site/news/page`. The request query is -kept. A query in `origin_url` itself is ignored. With no path, the request path -is sent as it arrived, as before. When `origin_url` has a path, a request path -with a `.` or `..` segment, in any percent-encoded spelling, is refused with -`400` so no request reaches outside the folder. - -**Redirects**: When the origin answers with a redirect whose `Location` names -the origin itself, the reader is sent to the same path, query and fragment on -the scheme and host they asked for, so their browser stays on the deployment. -An origin asked for `/news` that answers `Location: -https://origin.publisher.com/news/` sends a reader of `www.publisher.com` to -`https://www.publisher.com/news/`. The origin is named by the host in -`origin_url`, with its port, or by `origin_host_header_override` where one is -set. When `origin_url` has a path, a `Location` under that path comes back -without it, and one outside it is left as the origin wrote it. A `Location` -naming any other host, one that names no host (such as `/news/`), and the -`Location` of a response that is not a redirect are also left as the origin -wrote them. - -#### `origin_host_header_override` - -**Purpose**: Optional Host header to send to the publisher origin while still -connecting to the host in `origin_url`. - -**Usage**: - -- Connects, uses SNI, and checks certificates against `origin_url` -- Sends the configured value as the outbound HTTP `Host` header -- Is the name a redirect from the origin is compared with (see Redirects under - `origin_url`) -- Useful when the origin endpoint expects a canonical publisher hostname - -**Format**: Hostname with optional port, without protocol, path, query, or fragment - -- ✅ `www.publisher.com` -- ✅ `www.publisher.com:8443` -- ❌ `https://www.publisher.com` -- ❌ `www.publisher.com/path` - -**Default**: When omitted, Trusted Server sends the host from `origin_url`. - #### `proxy_secret` **Purpose**: Secret-store key name for the HMAC-SHA256 value used to sign proxy URLs. @@ -547,50 +490,122 @@ a zero-byte cap fails every non-empty publisher response. TRUSTED_SERVER__PUBLISHER__MAX_BUFFERED_BODY_BYTES=16777216 ``` -#### `origin_auth` +### `[source]` -**Purpose**: Sign every publisher origin request, for an origin such as -private object storage that answers only signed reads. Pages still pass through -the page pipeline, which an [asset route](#proxy-asset-routes) skips. +`[source]` is required. It says where the publisher's pages come from. A +source answers, for each request, where that request's content comes from. +`module` names the source, and the table beneath it holds the source's +settings, as in every section that selects a module. Core supplies one source, +`origin`, an HTTP origin. A module a build links can offer another source under +its own name, selected the same way, with `module = ""` and its settings +in `[source.]`. Each asset route names its own source in the same shape, +in [`[proxy.asset_routes.source.origin]`](#proxy-asset-routes-source-origin). -**Format**: The same table as -[`[proxy.asset_routes.auth]`](#proxy-asset-routes-auth), with the same fields, -defaults and secret-store key references, signed by the same signer. +The `origin` source takes these settings in `[source.origin]`: + +| Field | Type | Required | Description | +| ----------------- | ------ | -------- | ------------------------------------------------------------------------- | +| `url` | String | Yes | Origin address, whose path (if any) names the folder requests are sent to | +| `host_header` | String | No | Outbound Host header, where it is not the host of `url` | +| `directory_index` | String | No | File name added to an origin request path that ends in `/` | +| `auth` | Table | No | Signs every origin request, in the same table an asset route's source has | + +A configuration with no `[source]` is refused with `[source] names no source`. +A configuration that writes one of the fields below is refused, and the message +names the setting to write instead. + +| Refused field | Write instead | +| ----------------------------------------- | ----------------------------------------- | +| `[publisher] origin_url` | `[source.origin] url` | +| `[publisher] origin_host_header_override` | `[source.origin] host_header` | +| `[publisher] origin_auth` | `[source.origin] auth` | +| `[publisher] directory_index` | `[source.origin] directory_index` | +| `[[proxy.asset_routes]] origin_url` | `[proxy.asset_routes.source.origin] url` | +| `[[proxy.asset_routes]] auth` | `[proxy.asset_routes.source.origin] auth` | + +**Example**: ```toml -[publisher] -domain = "publisher.com" -cookie_domain = ".publisher.com" -origin_url = "https://storage.publisher.com/bucket/site" -proxy_secret = "publisher_proxy_secret" -directory_index = "index.html" +[source] +module = "origin" -[publisher.origin_auth] -type = "s3_sigv4" -region = "example-region-1" -access_key_id = "publisher_origin_access_key_id" -secret_access_key = "publisher_origin_secret_access_key" -# session_token = "publisher_origin_session_token" -# origin_query = "strip" +[source.origin] +url = "https://origin.publisher.com" +# Optional: connect to url but send this outbound Host header. +# host_header = "www.publisher.com" ``` -**Behavior**: +**Environment Override**: -- Only `GET` and `HEAD` are signed. Any other method is answered - `405 Method Not Allowed` with `Allow: GET, HEAD` and never reaches the origin, - so the publisher's credentials are only ever used to read. -- A signed request carries none of the reader's `Cookie`, `Authorization`, - `Proxy-Authorization`, `Referer`, `If-None-Match`, `If-Modified-Since`, - `If-Match`, `If-Unmodified-Since`, `If-Range` or `Range` headers, and none in - the signing scheme's own header namespace. -- `origin_query = "strip"` leaves the reader's query off the origin request. - By default the query is kept. -- `origin_url` must use `https`. A table the signer cannot use, such as an - empty or malformed `region`, is refused at startup naming - `publisher.origin_auth`. +```bash +TRUSTED_SERVER__SOURCE__ORIGIN__URL=https://origin.publisher.com +TRUSTED_SERVER__SOURCE__ORIGIN__HOST_HEADER=www.publisher.com +``` -**Default**: Unset, and origin requests are sent unsigned with the reader's -headers, as before. +#### `url` + +**Purpose**: Address of the origin server for publisher content. + +**Usage**: + +- Fallback proxy target for non-integration requests +- HTML processing rewrites origin URLs to request host +- Base for relative URL resolution + +**Format**: An absolute `http` or `https` address with a host. It carries no +user name or password, no query, no fragment, and no trailing `/`. + +- ✅ `https://origin.publisher.com` +- ✅ `https://origin.publisher.com:8080` +- ✅ `http://192.168.1.1:9000` +- ✅ `https://storage.publisher.com/bucket/site` (a folder of object storage) +- ❌ `origin.publisher.com` (missing protocol) +- ❌ `https://origin.publisher.com/` (trailing slash) +- ❌ `https://origin.publisher.com?site=1` (query) +- ❌ `https://user:pass@origin.publisher.com` (user name and password) + +**Port Handling**: Includes port if non-standard (not 80/443). + +**Path Handling**: A path in `url` names a folder, and the request path is +added after it with exactly one `/` between them, so a request for `/news/page` +with the folder above is sent to `/bucket/site/news/page`. The request query is +kept. With no path, the request path is sent as it arrived. When `url` has a +path, a request path with a `.` or `..` segment, in any percent-encoded +spelling, is refused with `400` so no request reaches outside the folder. + +**Redirects**: When the origin answers with a redirect whose `Location` names +the origin itself, the reader is sent to the same path, query and fragment on +the scheme and host they asked for, so their browser stays on the deployment. +An origin asked for `/news` that answers `Location: +https://origin.publisher.com/news/` sends a reader of `www.publisher.com` to +`https://www.publisher.com/news/`. The origin is named by the host in `url`, +with its port, or by `host_header` where one is set. When `url` has a path, a +`Location` under that path comes back without it, and one outside it is left +as the origin wrote it. A `Location` naming any other host, one that names no +host (such as `/news/`), and the `Location` of a response that is not a +redirect are also left as the origin wrote them. + +#### `host_header` + +**Purpose**: Optional outbound `Host` header, where it is not the host of +`url`. Trusted Server still connects to the host in `url`. + +**Usage**: + +- Connects, uses SNI, and checks certificates against the host in `url` +- Sends the configured value as the outbound HTTP `Host` header +- Is the name a redirect from the origin is compared with (see Redirects under + `url`) +- Useful when the origin endpoint expects a canonical publisher hostname + +**Format**: Hostname with optional port, without protocol, path, query, or fragment + +- ✅ `www.publisher.com` +- ✅ `www.publisher.com:8443` +- ❌ `https://www.publisher.com` +- ❌ `www.publisher.com/path` + +**Default**: When omitted, Trusted Server sends the host from `url`. #### `directory_index` @@ -608,16 +623,62 @@ folder with its index file. **Format**: One file name, made of letters, digits and `-._~!$&'()*+,;=:@`, with no `/`, `%`, space, query or fragment, and neither `.` nor `..`. Anything -else is refused at startup naming `publisher.directory_index`. +else is refused at startup naming `[source.origin] directory_index`. **Default**: Unset, and every path is sent as it arrived. **Environment Override**: ```bash -TRUSTED_SERVER__PUBLISHER__DIRECTORY_INDEX=index.html +TRUSTED_SERVER__SOURCE__ORIGIN__DIRECTORY_INDEX=index.html ``` +#### `auth` + +**Purpose**: Sign every publisher origin request, for an origin such as +private object storage that answers only signed reads. Pages still pass through +the page pipeline, which an [asset route](#proxy-asset-routes) skips. + +**Format**: The same table as +[`[proxy.asset_routes.source.origin.auth]`](#proxy-asset-routes-source-origin-auth), +with the same fields, defaults and secret-store key references, signed by the +same signer. Write `[source.origin.auth]` after the other `[source.origin]` +keys, because a table header ends the keys of the table before it. + +```toml +[source] +module = "origin" + +[source.origin] +url = "https://storage.publisher.com/bucket/site" +directory_index = "index.html" + +[source.origin.auth] +type = "s3_sigv4" +region = "example-region-1" +access_key_id = "publisher_origin_access_key_id" +secret_access_key = "publisher_origin_secret_access_key" +# session_token = "publisher_origin_session_token" +# origin_query = "strip" +``` + +**Behavior**: + +- Only `GET` and `HEAD` are signed. Any other method is answered + `405 Method Not Allowed` with `Allow: GET, HEAD` and never reaches the origin, + so the publisher's credentials are only ever used to read. +- A signed request carries none of the reader's `Cookie`, `Authorization`, + `Proxy-Authorization`, `Referer`, `If-None-Match`, `If-Modified-Since`, + `If-Match`, `If-Unmodified-Since`, `If-Range` or `Range` headers, and none in + the signing scheme's own header namespace. +- `origin_query = "strip"` leaves the reader's query off the origin request. + By default the query is kept. +- `url` must use `https`. A table the signer cannot use, such as an empty or + malformed `region`, is refused at startup naming `source.origin.auth`. + +**Default**: Unset, and origin requests are sent unsigned with the reader's +headers. + ## Trusted Client IP Configuration Use this optional section when a trusted CDN service forwards requests to the @@ -779,19 +840,19 @@ defaults. Each entry of `show` and `hide` is a path pattern, being keys joined by `.`, with `[]` for every element of a list and `[N]` for one, such as -`publisher.origin_url`, `proxy.asset_routes[].origin_url` or +`source.origin.url`, `proxy.asset_routes[].source.origin.url` or `proxy.asset_routes[0].prefix`. A pattern names the values at exactly its own depth. A masked value shows as `XXXX`, and the page lists every masked path. -| What | Masked | Can `show` reveal it | -| ------------------------------------------------------------------------- | ---------- | -------------------------------- | -| A secret, meaning a value the settings loader fills from the secret store | Always | No | -| `publisher.origin_url` and `publisher.origin_host_header_override` | By default | Yes | -| `proxy.asset_routes[].origin_url` | By default | Yes | -| `ec.ec_store` and `auction.creative_store` | By default | Yes | -| Anything `hide` names | When named | It is the publisher's own choice | +| What | Masked | Can `show` reveal it | +| --------------------------------------------------------------------------------------------- | ---------- | -------------------------------- | +| A secret, meaning a value the settings loader fills from the secret store | Always | No | +| `source.origin.url` and `source.origin.host_header` | By default | Yes | +| `proxy.asset_routes[].source.origin.url` and `proxy.asset_routes[].source.origin.host_header` | By default | Yes | +| `ec.ec_store` and `auction.creative_store` | By default | Yes | +| Anything `hide` names | When named | It is the publisher's own choice | A pattern that would not be honored as written refuses the configuration, both when a deployment is validated and when the settings load. That is a @@ -804,7 +865,7 @@ pattern that changes nothing the page shows, and any pattern at all beside ```toml [inspect] -show = ["publisher.origin_url"] +show = ["source.origin.url"] hide = ["response_headers"] ``` @@ -1532,21 +1593,28 @@ Set this to `false` only for local development with self-signed certificates. Asset routes proxy selected first-party paths to an alternate asset origin without requiring signed `/first-party/proxy` URLs. -| Field | Type | Required | Description | -| ----------------- | ------ | -------- | ----------------------------------------------- | -| `prefix` | String | Yes | Request path prefix to match | -| `origin_url` | String | Yes | Absolute `http` or `https` origin URL | -| `path_pattern` | String | No | Regex matched against the incoming request path | -| `target_path` | String | No | Replacement path used with `path_pattern` | -| `auth` | Table | No | Optional origin authentication | -| `image_optimizer` | Table | No | Optional route-level Image Optimizer settings | +| Field | Type | Required | Description | +| ----------------- | ------ | -------- | ----------------------------------------------------------------- | +| `prefix` | String | Yes | Request path prefix to match | +| `source` | Table | Yes | Where the route's content comes from, in the shape `[source]` has | +| `path_pattern` | String | No | Regex matched against the incoming request path | +| `target_path` | String | No | Replacement path used with `path_pattern` | +| `image_optimizer` | Table | No | Optional route-level Image Optimizer settings | + +Write `prefix`, `path_pattern` and `target_path` before the route's source +tables, because a table header ends the route's own keys. **Example**: ```toml [[proxy.asset_routes]] prefix = "/assets/" -origin_url = "https://assets.example.com" + +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://assets.example.com" ``` **Path rewrite example**: @@ -1554,9 +1622,14 @@ origin_url = "https://assets.example.com" ```toml [[proxy.asset_routes]] prefix = "/.image/" -origin_url = "https://assets-cdn.example.com" path_pattern = "^/\\.image/(.*)/[^/]+\\.([^/.]+)$" target_path = "/image/upload/$1.$2" + +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://assets-cdn.example.com" ``` **Behavior**: @@ -1565,10 +1638,36 @@ target_path = "/image/upload/$1.$2" - Built-in and integration routes take precedence. - The longest matching asset-route prefix wins. - `path_pattern` and `target_path` must be configured together. -- `origin_url` must not include userinfo, a path, a query string, or a fragment. - Unsafe origin response headers such as `Set-Cookie` are stripped before the response reaches the browser. -### `[proxy.asset_routes.auth]` +### `[proxy.asset_routes.source.origin]` + +A route selects its source the way [`[source]`](#source) does, with `module` +in `[proxy.asset_routes.source]` and the source's settings in the table +beneath. Core's `origin` source takes these settings: + +| Field | Type | Required | Description | +| ----------------- | ------ | -------- | ------------------------------------------------------- | +| `url` | String | Yes | Absolute `http` or `https` origin address | +| `host_header` | String | No | Outbound Host header, where it is not the host of `url` | +| `directory_index` | String | No | File name added to a target path that ends in `/` | +| `auth` | Table | No | Signs every request to the origin, see the table below | + +- `url` must name a host, and must not carry a user name or password, a query, + a fragment, or a trailing `/`. +- A path in `url` names a folder. The route's target path, after any + `path_pattern` and `target_path` rewrite, is added under that folder, so with + `url = "https://storage.example.com/bucket/site"` a request for + `/assets/app.js` is sent to + `https://storage.example.com/bucket/site/assets/app.js`. +- When `url` names a folder, a target path with a `.` or `..` segment in any + spelling is answered `400`, so no request reaches outside the folder. +- `directory_index` follows the same rules as the publisher's + [`directory_index`](#directory-index). +- With `host_header`, Trusted Server still connects, uses SNI, and checks + certificates against the host in `url`. + +### `[proxy.asset_routes.source.origin.auth]` The first supported origin auth type is `s3_sigv4`. @@ -1586,9 +1685,14 @@ The first supported origin auth type is `s3_sigv4`. ```toml [[proxy.asset_routes]] prefix = "/.image/" -origin_url = "https://bucket.s3.us-east-1.amazonaws.com" -[proxy.asset_routes.auth] +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://bucket.s3.us-east-1.amazonaws.com" + +[proxy.asset_routes.source.origin.auth] type = "s3_sigv4" region = "us-east-1" origin_query = "strip" @@ -1597,7 +1701,9 @@ secret_access_key = "s3_secret_access_key" # session_token = "s3_session_token" ``` -S3 auth uses header-based AWS SigV4 with `UNSIGNED-PAYLOAD`. It is scoped to read-only asset requests and expects `origin_url` to use the S3 host that AWS validates. Credential references resolve from `trusted_server_secrets` at startup, and request signing performs no secret-store reads. +S3 auth uses header-based AWS SigV4 with `UNSIGNED-PAYLOAD`. It is scoped to read-only asset requests and expects `url` to use the S3 host that AWS validates. Credential references resolve from `trusted_server_secrets` at startup, and request signing performs no secret-store reads. + +Signing requires `url` to use `https`. A table the signer cannot use, such as an empty or malformed `region`, is refused at startup naming the route, as in `proxy.asset_routes /.image/ source.origin.auth`. Effective `origin_query` precedence is auth-level `origin_query`, then enabled Image Optimizer `origin_query`, then the route default. @@ -3768,10 +3874,12 @@ from the file, and startup checks the rest and runs the first set again. ### Checked when the configuration is validated or pushed -**Publisher**: +**Publisher and source**: -- All fields non-empty -- `origin_url` is a valid URL +- All `[publisher]` fields non-empty +- `[source]` names exactly one source +- `[source.origin] url` is an absolute `http` or `https` address with a host, + and carries no user name or password, query, fragment, or trailing `/` **EC Validation**: @@ -3840,8 +3948,13 @@ Configuration error: [demand.pbs_main] endpoint must be HTTPS, or HTTP to 127.0. # trusted-server.dev.toml [publisher] domain = "localhost" -origin_url = "http://localhost:3000" proxy_secret = "publisher_proxy_secret" + +[source] +module = "origin" + +[source.origin] +url = "http://localhost:3000" ``` **Staging and production**: diff --git a/docs/guide/error-reference.md b/docs/guide/error-reference.md index 67c25e0e57..0487a22772 100644 --- a/docs/guide/error-reference.md +++ b/docs/guide/error-reference.md @@ -62,8 +62,13 @@ before validation. ```toml [publisher] domain = "your-publisher-domain.com" -origin_url = "https://origin.your-publisher-domain.com" proxy_secret = "publisher_proxy_secret" + +[source] +module = "origin" + +[source.origin] +url = "https://origin.your-publisher-domain.com" ``` `proxy_secret` names an entry in the Trusted Server secret store. Provision a @@ -73,12 +78,32 @@ TOML file. **Required Fields:** - `publisher.domain` -- `publisher.origin_url` - `publisher.proxy_secret` +- `source.module` +- `source.origin.url` (when `source.module = "origin"`) - `ec.hmac.passphrase` (when `ec.module = "hmac"`) --- +### No source named + +**Error Message:** + +``` +Configuration error: [source] names no source. Write `module = "origin"` and the origin's address under [source.origin] url +``` + +**Cause:** The configuration has no `[source]` section, or has one with no +`module` line, such as a `[source.origin]` table written on its own. `[source]` +is required, because it says where the publisher's pages come from. + +**Solution:** Add the section with its `module` line, as in the example above. A configuration that +names the origin in `[publisher]` is refused with a message naming the +`[source.origin]` setting to write instead. See +[`[source]`](/guide/configuration#source). + +--- + ### Invalid URL format **Error Message:** diff --git a/docs/guide/first-party-proxy.md b/docs/guide/first-party-proxy.md index 69d4ed7c6a..a711d832e2 100644 --- a/docs/guide/first-party-proxy.md +++ b/docs/guide/first-party-proxy.md @@ -442,8 +442,13 @@ Configure proxy behavior in `trusted-server.toml`: [publisher] domain = "publisher.com" cookie_domain = ".publisher.com" -origin_url = "https://origin.publisher.com" proxy_secret = "publisher_proxy_secret" + +[source] +module = "origin" + +[source.origin] +url = "https://origin.publisher.com" ``` `proxy_secret` is the key name in the Trusted Server secret store. Provision a @@ -456,7 +461,12 @@ Use `[[proxy.asset_routes]]` when a first-party path prefix should proxy directl ```toml [[proxy.asset_routes]] prefix = "/assets/" -origin_url = "https://assets.example.com" + +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://assets.example.com" ``` Asset routes are intended for publisher-owned paths, not third-party creative URLs. They support optional path rewrites, private S3 origin signing, and Fastly Image Optimizer metadata. See [Asset Routes](/guide/asset-routes) for full configuration and operational guidance. diff --git a/docs/superpowers/specs/2026-10-14-source-modules-design.md b/docs/superpowers/specs/2026-10-14-source-modules-design.md new file mode 100644 index 0000000000..1c99a41d18 --- /dev/null +++ b/docs/superpowers/specs/2026-10-14-source-modules-design.md @@ -0,0 +1,87 @@ +# Design Spec: A source module says where the publisher path and the asset routes fetch from + +**Status:** Proposed, 2026-10-14. Implemented on a working branch of the fork, with no pull request raised yet. +**Author:** 51Degrees (contributed), for Tech Lab review +**Related specs:** `2026-10-13-publisher-object-storage-origin-design.md`, `2026-10-13-origin-redirect-design.md`, `2026-10-13-request-addressing-design.md`, `2026-04-28-multi-backend-asset-proxy-design.md` +**Related PRs:** none yet +**Issues:** IABTechLab/trusted-server#1332 +**Last updated:** 2026-10-14 + +> **Why this spec exists.** Core's publisher path fetches from the origin `[publisher]` names and each asset route from the origin it names, so no module can say where a request's content comes from, and where it comes from is written in two places in two shapes. + +## 1. Problem + +Where a deployment's content comes from is written twice. `[publisher]` carries `origin_url`, `origin_host_header_override`, `origin_auth` and `directory_index` beside who the publisher is, and every `[[proxy.asset_routes]]` entry carries its own `origin_url` and `auth`. Both are fetched by code in core that knows one kind of source, an HTTP origin. + +A deployment whose origin depends on the request has no place for the code that chooses it. An origin chosen from the request's own address, as on a host that serves many publishers by path, or by a rule of the deployer's own, would otherwise be one more branch in core's fetch, running for every deployment. + +The fetch and serve phases change a response. They do not choose where it comes from, so a middleware is the wrong place too. + +## 2. Design + +**A source module says where content comes from.** Core declares the interface. The HTTP origin fetch is the one source core supplies, as the module `origin`. A module a build links offers another under its own name, as every other module type is offered. + +**One shape for where content comes from.** A source is selected in a module section, `module` naming it and a table beneath it holding its settings, the shape `[geo]` and `[prefix]` already use. The publisher's source is `[source]`. An asset route's is the route's own `source` table. + +```toml +[publisher] +domain = "www.example.com" +cookie_domain = ".example.com" +proxy_secret = "proxy_secret_name" + +[source] +module = "origin" + +[source.origin] +url = "https://origin.example.com" +host_header = "www.example.com" + +[[proxy.asset_routes]] +prefix = "/static/" + +[proxy.asset_routes.source] +module = "origin" + +[proxy.asset_routes.source.origin] +url = "https://bucket.object-store.example/site" +auth = { type = "s3_sigv4", region = "eu-west-1", access_key_id = "id_name", secret_access_key = "key_name" } +``` + +`[publisher]` keeps who the publisher is and how its pages are processed. Every field that said where the pages come from moves to the `origin` source's table, under the names the table above shows: + +| Was | Is | +| ----------------------------------------- | ----------------------------------------- | +| `[publisher] origin_url` | `[source.origin] url` | +| `[publisher] origin_host_header_override` | `[source.origin] host_header` | +| `[publisher] origin_auth` | `[source.origin] auth` | +| `[publisher] directory_index` | `[source.origin] directory_index` | +| `[[proxy.asset_routes]] origin_url` | `[proxy.asset_routes.source.origin] url` | +| `[[proxy.asset_routes]] auth` | `[proxy.asset_routes.source.origin] auth` | + +An asset route keeps `prefix`, `path_pattern`, `target_path` and `image_optimizer`, which say which requests it answers and what path it asks for. A path in an asset route's `url` names a folder the target path is added under, as a path in the publisher's `url` does. A document that writes a field where it used to be is refused at startup, with the new place named in the message, and a document with no `[source]` is refused. A source table naming a module the build does not link is refused with the names on offer, as every other module selection is. + +The configuration page masks `url` and `host_header` by default, as it masked the fields they replace, and the signing secrets are declared at their new paths. + +**What a source answers.** For each request a source first answers where the content comes from: the origin's scheme, host, port and folder, the `Host` header to send, how the request is signed and whether its query is kept, and the folder's index document. Core's caches key on that answer, and core rewrites the origin's host in the page, follows the redirect rule and tells middleware the origin host from it, so a source that answers differently for each request is kept apart in every one of those places. + +Core fetches the location a source answers through the same send, the same early send, streaming and the same cache paths as today. The `origin` source answers the address, `Host` and signing the old fields named, and the cache keys are made from them in the same form, so an existing deployment's stored pages and assets keep their keys. + +**One call for every path.** The publisher's page, the publisher's assets held in the asset cache, and each asset route ask their source the same way, so content a source answers for is fetched and cached on the same terms whichever path asked. + +**The request a source is given.** The method, the path and query the request continues with, its headers, and the prefix request addressing removed, if any. That lets a source take where to fetch from out of the address. + +## 3. What is not in this change + +- Any source other than `origin`. A test supplies one that answers a different origin for each request, which is what a source reading the address does. +- A source supplying the response itself, for content that has no HTTP origin. That is a second method on the interface, added with the first source that needs it. +- Selecting a source per path for the publisher's pages. A deployment wanting that writes an asset route. + +## 4. Tests + +- Each old field is refused naming its new place, in a document and in a configuration blob, and a document with no `[source]` is refused. +- `[source.origin]` reads, trims and checks its address, `Host`, index file and signing, and refuses an unknown key, naming the setting. Environment variables set the same fields at their new paths. +- A location gives the host with its port, the `Host` it sends, a folder from the address's path, and an identity in the form the caches used before, so the stored keys do not change. +- A source answering a different origin for each request sends each request to its own origin and tells the page pipeline that request's origin host. +- An unknown source name is refused with the names on offer. +- An asset route reads its source, fetches its target path under the source's folder, and signs with the source's signing. +- The configuration page masks the origin's address and `Host` by default and declares the signing secrets at their new paths. diff --git a/scripts/attestation-local-test.sh b/scripts/attestation-local-test.sh index 57023096b9..294a966a18 100755 --- a/scripts/attestation-local-test.sh +++ b/scripts/attestation-local-test.sh @@ -155,7 +155,7 @@ TOMLEOF cd "$WORK" TRUSTED_SERVER__PUBLISHER__DOMAIN="$PUBLISHER" \ TRUSTED_SERVER__PUBLISHER__COOKIE_DOMAIN=".$PUBLISHER" \ - TRUSTED_SERVER__PUBLISHER__ORIGIN_URL="http://127.0.0.1:$ORIGIN_PORT" \ + TRUSTED_SERVER__SOURCE__ORIGIN__URL="http://127.0.0.1:$ORIGIN_PORT" \ "$TS" config push --adapter fastly --local \ --manifest "$WORK/edgezero.toml" --app-config "$WORK/app.toml" \ --no-diff --yes >/dev/null diff --git a/scripts/template-cache-local-test.sh b/scripts/template-cache-local-test.sh index 6c5f832381..7c40f585be 100755 --- a/scripts/template-cache-local-test.sh +++ b/scripts/template-cache-local-test.sh @@ -251,8 +251,8 @@ def replace_once(content, old, new, description): s = replace_once( s, - 'origin_url = "https://origin.example.com"', - f'origin_url = "http://127.0.0.1:{origin_port}"', + 'url = "https://origin.example.com"', + f'url = "http://127.0.0.1:{origin_port}"', "publisher origin", ) s = replace_once( diff --git a/trusted-server.example.toml b/trusted-server.example.toml index fc662a28e5..54e9a6996d 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -6,8 +6,8 @@ # `trusted-server.toml`. Copy it (`ts config init`), fill in the required # values, and push it (`ts config push`) as an EdgeZero app-config blob. # -# Only one section is REQUIRED for the server to start and pass validation, -# which is [publisher] (domain + origin). +# Two sections are REQUIRED for the server to start and pass validation, +# being [publisher] (the domain) and [source] (where its pages come from). # # Edge Cookie identity is optional and stays off until an [ec] module is # selected, so no identity secret is needed to start. See the Edge Cookie @@ -50,35 +50,46 @@ # ----------------------------------------------------------------------------- -# REQUIRED - Publisher / origin +# REQUIRED - Publisher # ----------------------------------------------------------------------------- [publisher] # Public domain Trusted Server is fronting. domain = "example.com" # Cookie scope for first-party identity cookies (leading dot = include subdomains). cookie_domain = ".example.com" -# Upstream origin to proxy publisher content from. No trailing slash. A path is -# kept and the request path is added after it, so a site held in a folder of -# object storage is "https://storage.example.com/bucket/site". -origin_url = "https://origin.example.com" # Secret Store key for the HMAC secret used to sign first-party proxy URLs. proxy_secret = "publisher_proxy_secret" -# Optional: override the outbound Host header sent to origin_url. -# origin_host_header_override = "www.example.com" # Optional: max bytes buffered when a response is post-processed in full (HTML # rewriting/injection) instead of streamed. Default 16 MiB; larger responses # return 502. Raise for deployments serving very large publisher pages. # max_buffered_body_bytes = 16777216 # 16 MiB + + +# ----------------------------------------------------------------------------- +# REQUIRED - Source of the publisher's pages +# ----------------------------------------------------------------------------- +# `module` names the source and the table beneath it holds the source's +# settings. Core supplies `origin`, an HTTP origin. A module can offer another. +[source] +module = "origin" + +[source.origin] +# Upstream origin to proxy publisher content from. No trailing slash. A path is +# kept and the request path is added after it, so a site held in a folder of +# object storage is "https://storage.example.com/bucket/site". +url = "https://origin.example.com" +# Optional: the outbound Host header, where it is not the host of url. +# host_header = "www.example.com" # Optional: file name added to a request path ending in "/" on the way to the # origin, for an origin such as object storage that does not answer a folder # with its index file. The reader's address is unchanged. # directory_index = "index.html" # Optional: sign every origin request, for an origin such as private object -# storage. The same table as [proxy.asset_routes.auth]. Only GET and HEAD are -# signed, other methods are refused, and the reader's cookies, Authorization, -# Referer, conditional validators and Range are not sent. origin_url must use -# https. Keep this table last in [publisher]. -# [publisher.origin_auth] +# storage. The same table as an asset route's. Only GET and HEAD are signed, +# other methods are refused, and the reader's cookies, Authorization, Referer, +# conditional validators and Range are not sent. url must use https. Keep this +# table last in [source.origin]. +# [source.origin.auth] # type = "s3_sigv4" # region = "example-region-1" # access_key_id = "publisher_origin_access_key_id" @@ -423,14 +434,21 @@ pull_sync_concurrency = 3 # prefix wins; only GET/HEAD participate; query string is preserved. # [[proxy.asset_routes]] # prefix = "/.image/" -# origin_url = "https://assets.example.com" # Optional path rewrite before sending upstream (the prefix must match what the # pattern expects, and here both use `/.image/`): # path_pattern = "^/\\.image/(.*)/[^/]+\\.([^/.]+)$" # target_path = "/image/upload/$1.$2" # +# The route's source, in the shape [source] has. A path in url names a folder +# the target path is added under. +# [proxy.asset_routes.source] +# module = "origin" +# +# [proxy.asset_routes.source.origin] +# url = "https://assets.example.com" +# # Optional S3 SigV4 auth for a private origin: -# [proxy.asset_routes.auth] +# [proxy.asset_routes.source.origin.auth] # type = "s3_sigv4" # region = "us-east-1" # origin_query = "strip" From dc7306d5115c60faac150240ade44d7bb6913a47 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 14 Oct 2026 20:52:41 +0100 Subject: [PATCH 02/17] A module sees the prefix request addressing removed Request addressing takes a prefix off the path and keeps it as a request extension, so a source can say where content comes from. A module was handed the continuing path only, so a module that names things after the tenant a path addressed, such as a prefix module naming cookies after the publisher, could not tell which one it was. ModuleRequest::of now reads the removed prefix and removed_prefix() gives it, with_removed_prefix sets it, and ResolvedRequest keeps it, so a held request still has it. A request addressing nothing carries none. --- .../trusted-server-core/src/module_context.rs | 37 +++++++++++++++---- .../src/module_context/tests.rs | 27 ++++++++++++++ 2 files changed, 56 insertions(+), 8 deletions(-) diff --git a/crates/trusted-server-core/src/module_context.rs b/crates/trusted-server-core/src/module_context.rs index e53ef8231e..a660bcbbdd 100644 --- a/crates/trusted-server-core/src/module_context.rs +++ b/crates/trusted-server-core/src/module_context.rs @@ -71,6 +71,7 @@ pub struct ModuleRequest<'r> { query: &'r str, host: &'r str, scheme: &'r str, + removed_prefix: Option<&'r str>, } impl<'r> ModuleRequest<'r> { @@ -83,6 +84,7 @@ impl<'r> ModuleRequest<'r> { query: "", host, scheme, + removed_prefix: None, } } @@ -97,6 +99,12 @@ impl<'r> ModuleRequest<'r> { request.uri().path(), ) .with_query(request.uri().query().unwrap_or_default()) + .with_removed_prefix( + request + .extensions() + .get::() + .map(crate::addressing::RemovedPathPrefix::as_str), + ) } /// The same request with the query string, without its leading `?`. @@ -105,6 +113,15 @@ impl<'r> ModuleRequest<'r> { Self { query, ..self } } + /// The same request with the path prefix request addressing removed. + #[must_use] + pub fn with_removed_prefix(self, removed_prefix: Option<&'r str>) -> Self { + Self { + removed_prefix, + ..self + } + } + /// The request method. #[must_use] pub fn method(&self) -> &'r Method { @@ -134,6 +151,14 @@ impl<'r> ModuleRequest<'r> { pub fn scheme(&self) -> &'r str { self.scheme } + + /// The path prefix request addressing removed, such as `/www.example.com` + /// on a host that serves several publishers by path, if it removed one. + /// [`Self::path`] is the path the request continues with. + #[must_use] + pub fn removed_prefix(&self) -> Option<&'r str> { + self.removed_prefix + } } /// A [`ModuleRequest`] that owns its parts, for a caller that must keep them @@ -148,6 +173,7 @@ pub struct ResolvedRequest { query: String, host: String, scheme: String, + removed_prefix: Option, } impl ResolvedRequest { @@ -155,14 +181,7 @@ impl ResolvedRequest { /// scheme read as the publisher path reads them. #[must_use] pub fn of(request: &http::Request, client: &ClientInfo) -> Self { - let view = ModuleRequest::of(request, client); - Self { - method: view.method.clone(), - path: view.path.to_owned(), - query: view.query.to_owned(), - host: view.host.to_owned(), - scheme: view.scheme.to_owned(), - } + Self::from_view(&ModuleRequest::of(request, client)) } /// The method, address, host and scheme `view` carries. @@ -174,6 +193,7 @@ impl ResolvedRequest { query: view.query.to_owned(), host: view.host.to_owned(), scheme: view.scheme.to_owned(), + removed_prefix: view.removed_prefix.map(str::to_owned), } } @@ -191,6 +211,7 @@ impl ResolvedRequest { pub fn view(&self) -> ModuleRequest<'_> { ModuleRequest::new(&self.method, &self.host, &self.scheme, &self.path) .with_query(&self.query) + .with_removed_prefix(self.removed_prefix.as_deref()) } } diff --git a/crates/trusted-server-core/src/module_context/tests.rs b/crates/trusted-server-core/src/module_context/tests.rs index dff4fbc923..394fc23395 100644 --- a/crates/trusted-server-core/src/module_context/tests.rs +++ b/crates/trusted-server-core/src/module_context/tests.rs @@ -454,3 +454,30 @@ fn a_route_answers_forbidden_only_for_a_value_its_permissions_withhold() { http::StatusCode::INTERNAL_SERVER_ERROR ); } + +#[test] +fn a_module_request_carries_the_prefix_request_addressing_removed() { + let mut request = http::Request::builder() + .method(Method::GET) + .uri("https://yoursite.example/news/") + .body(EdgeBody::empty()) + .expect("should build the request"); + let client = ClientInfo::default(); + assert_eq!(ModuleRequest::of(&request, &client).removed_prefix(), None); + + request + .extensions_mut() + .insert(crate::addressing::RemovedPathPrefix::new( + "/www.example.com".to_owned(), + )); + let view = ModuleRequest::of(&request, &client); + assert_eq!(view.removed_prefix(), Some("/www.example.com")); + assert_eq!(view.path(), "/news/"); + + let held = ResolvedRequest::of(&request, &client); + assert_eq!( + held.view().removed_prefix(), + Some("/www.example.com"), + "a held request should keep the removed prefix" + ); +} From 5494af9ca37b2ee8399a067db0b3086022afef17 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 14 Oct 2026 08:21:26 +0100 Subject: [PATCH 03/17] An integration route is handed its request's evidence A route reads its request through its module call, and the call carried the request as resolved, the settings and the request state, but no evidence. A route that asks for the request as evidence, the client address and headers a module reads, was refused with "there is no request evidence where this module runs", so every such route answered 500 in production while its own tests, which build the context by hand, passed. handle_proxy now takes the evidence before the route takes the request, in the form the permission path reads it: the normalized client address, the headers after the caller's EC header is removed, and the address asked for. The test goes through handle_proxy, so it fails if the evidence is dropped again. (cherry picked from commit efad1ece83c3ec214e7d60c3e0754fb1053c5b41) --- .../src/integrations/registry.rs | 117 +++++++++++++++++- 1 file changed, 116 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 56d0fd9737..d04fa4ba3f 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -17,6 +17,7 @@ use crate::ec::device::DeviceModule; use crate::ec::kv::KvIdentityGraph; use crate::ec::module::{EcModuleSelection, EdgeCookieModule}; use crate::error::TrustedServerError; +use crate::evidence::OwnedRequestInfo; use crate::geo::GeoInfo; use crate::http_util::is_navigation_request; use crate::middleware::{Middleware, MiddlewareChain, MiddlewarePhase, PhaseEntries}; @@ -2484,10 +2485,25 @@ impl IntegrationRegistry { req.headers_mut().remove(HEADER_X_TS_EC.clone()); // The route takes the request, so the context holds a copy of - // what it resolved to, beside what was resolved for it. + // what it resolved to and of its evidence, beside what was + // resolved for it. The evidence is read as the permission path + // reads it, after the caller's EC header is gone. let resolved = ResolvedRequest::of(&req, services.client_info()); + let evidence = OwnedRequestInfo::new( + services + .client_info() + .client_ip + .map(crate::ec::generation::normalize_ip) + .unwrap_or_default(), + req.headers().clone(), + ) + .with_request_target( + req.uri().path().to_owned(), + req.uri().query().unwrap_or_default().to_owned(), + ); let context = ModuleContext::new(resolved.view()) .with_settings(settings) + .with_evidence(&evidence) .with_request_state(ec_context, services); Some( proxy @@ -4859,6 +4875,105 @@ mod tests { ); } + /// A route that answers with the request evidence its module call + /// handed it. + struct EvidenceReadingRoute; + + impl EvidenceReadingRoute { + fn report( + &self, + _req: Request, + evidence: &dyn crate::evidence::RequestInfo, + ) -> Result, Report> { + let body = format!( + "ip={} path={} query={} session={} ec={}", + evidence.client_ip(), + evidence.path(), + evidence.query(), + evidence.cookie("session").unwrap_or("none"), + evidence.header("x-ts-ec").unwrap_or("none"), + ); + Ok(Response::builder() + .status(StatusCode::OK) + .body(EdgeBody::from(body)) + .expect("should build the response")) + } + } + + #[async_trait(?Send)] + impl IntegrationProxy for EvidenceReadingRoute { + fn integration_name(&self) -> &'static str { + "evidence" + } + + fn routes(&self) -> Vec { + vec![IntegrationEndpoint::post("/integrations/test/evidence")] + } + + async fn handle( + &self, + call: ModuleCall<'_>, + req: Request, + ) -> Result, Report> { + call.inject_with(self, req, Self::report)? + } + } + + /// A route is handed its request's evidence, being the client address, + /// the headers and the address it was asked at, without the caller's EC + /// header, so a route that reads the request as evidence can answer. + #[test] + fn a_route_is_handed_its_requests_evidence() { + let settings = create_test_settings(); + let registry = IntegrationRegistry::from_routes(vec![( + Method::POST, + "/integrations/test/evidence", + ( + Arc::new(EvidenceReadingRoute) as Arc, + "evidence", + ), + )]); + let req = Request::builder() + .method(Method::POST) + .uri("https://test-publisher.com/integrations/test/evidence?from=page") + .header(header::HOST, "test-publisher.com") + .header(header::COOKIE, "session=abc") + .header(HEADER_X_TS_EC.clone(), "caller-supplied") + .body(EdgeBody::empty()) + .expect("should build request"); + let mut ec_context = + EcContext::new_for_test(None, crate::consent::ConsentContext::default()); + let services = + crate::platform::test_support::noop_services_with_client_ip(std::net::IpAddr::from([ + 203, 0, 113, 7, + ])); + + let response = futures::executor::block_on(registry.handle_proxy(ProxyDispatchInput { + method: &Method::POST, + path: "/integrations/test/evidence", + settings: &settings, + kv: None, + ec_context: &mut ec_context, + services: &services, + req, + })) + .expect("should find the route") + .expect("a route reading the evidence should answer"); + let body = String::from_utf8( + response + .into_body() + .into_bytes() + .unwrap_or_default() + .to_vec(), + ) + .expect("the answer is text"); + + assert_eq!( + body, + "ip=203.0.113.7 path=/integrations/test/evidence query=from=page session=abc ec=none" + ); + } + #[test] fn handle_proxy_rejects_invalid_ec_request_header() { let settings = create_test_settings(); From d6d0da0ff9047c2b4e17ca5e2b8d7bc7de93d63d Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 14 Oct 2026 23:14:00 +0100 Subject: [PATCH 04/17] The testlight test gives its settings an origin source A document without [source] is refused since the source series, and the test built its registry from default settings. --- crates/testing/testlight/src/lib.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/crates/testing/testlight/src/lib.rs b/crates/testing/testlight/src/lib.rs index f819feafe6..4c8f75f85c 100644 --- a/crates/testing/testlight/src/lib.rs +++ b/crates/testing/testlight/src/lib.rs @@ -635,6 +635,9 @@ mod tests { "#; let mut settings = Settings::default(); + settings.set_origin_source(&trusted_server_core::source::OriginSettings::new( + "https://origin.example.com", + )); let shim_src = "https://edge.example.com/static/testlight.js".to_owned(); settings .insert_module_config( From a7a04281bda069ce24e4d71d5003fdc9a7b1c0dc Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 01:25:31 +0100 Subject: [PATCH 05/17] The local test scripts set the origin's address under [source.origin] The source series moved the origin's address out of [publisher], and the overlay ignores a variable for a key the document lacks, so the scripts that still set TRUSTED_SERVER__PUBLISHER__ORIGIN_URL left the template cache harness, the config by host harness and the integration runs on the placeholder origin, which the settings refuse. --- scripts/config-by-host-local-test.sh | 2 +- scripts/integration-tests-browser.sh | 2 +- scripts/integration-tests.sh | 2 +- scripts/smoke-common.sh | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/config-by-host-local-test.sh b/scripts/config-by-host-local-test.sh index ed14f24e73..ff640e5f2c 100755 --- a/scripts/config-by-host-local-test.sh +++ b/scripts/config-by-host-local-test.sh @@ -205,7 +205,7 @@ push_blob() { # push_blob [--key ] cd "$WORK" TRUSTED_SERVER__PUBLISHER__DOMAIN="$domain" \ TRUSTED_SERVER__PUBLISHER__COOKIE_DOMAIN=".$domain" \ - TRUSTED_SERVER__PUBLISHER__ORIGIN_URL="http://127.0.0.1:$port" \ + TRUSTED_SERVER__SOURCE__ORIGIN__URL="http://127.0.0.1:$port" \ "$TS" config push --adapter fastly --local \ --manifest "$WORK/edgezero.toml" --app-config "$WORK/app.toml" \ --no-diff --yes "$@" >/dev/null diff --git a/scripts/integration-tests-browser.sh b/scripts/integration-tests-browser.sh index e81767be32..d9a715bea9 100755 --- a/scripts/integration-tests-browser.sh +++ b/scripts/integration-tests-browser.sh @@ -28,7 +28,7 @@ fi # --- Build WASM binary --- echo "==> Building WASM binary (origin=http://127.0.0.1:$ORIGIN_PORT)..." -TRUSTED_SERVER__PUBLISHER__ORIGIN_URL="http://127.0.0.1:$ORIGIN_PORT" \ +TRUSTED_SERVER__SOURCE__ORIGIN__URL="http://127.0.0.1:$ORIGIN_PORT" \ TRUSTED_SERVER__PUBLISHER__PROXY_SECRET="integration-test-proxy-secret" \ TRUSTED_SERVER__EC__PASSPHRASE="integration-test-ec-secret-padded-32" \ TRUSTED_SERVER__EC__PARTNERS='[{"name":"Integration Test Partner","source_domain":"inttest.example.com","bidstream_enabled":true,"api_token":"integration-test-token-alpha-32-bytes-ok"},{"name":"Integration Test Partner 2","source_domain":"inttest2.example.com","bidstream_enabled":true,"api_token":"integration-test-token-bravo-32-bytes-ok"}]' \ diff --git a/scripts/integration-tests.sh b/scripts/integration-tests.sh index 96e492f40a..59a56e4ef8 100755 --- a/scripts/integration-tests.sh +++ b/scripts/integration-tests.sh @@ -51,7 +51,7 @@ echo "==> Building WASM binary..." cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1 echo "==> Building Axum native binary (origin=http://127.0.0.1:$ORIGIN_PORT)..." -TRUSTED_SERVER__PUBLISHER__ORIGIN_URL="http://127.0.0.1:$ORIGIN_PORT" \ +TRUSTED_SERVER__SOURCE__ORIGIN__URL="http://127.0.0.1:$ORIGIN_PORT" \ TRUSTED_SERVER__PUBLISHER__PROXY_SECRET="integration-test-proxy-secret" \ TRUSTED_SERVER__EDGE_COOKIE__SECRET_KEY="integration-test-secret-key" \ TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK=false \ diff --git a/scripts/smoke-common.sh b/scripts/smoke-common.sh index e262b36cc8..03ceb660fe 100644 --- a/scripts/smoke-common.sh +++ b/scripts/smoke-common.sh @@ -164,7 +164,7 @@ smoke_initialize_config() { "$SMOKE_TS_BIN" config init --app-config "$SMOKE_APP_CONFIG" export TRUSTED_SERVER__PUBLISHER__DOMAIN="127.0.0.1" export TRUSTED_SERVER__PUBLISHER__COOKIE_DOMAIN="127.0.0.1" - export TRUSTED_SERVER__PUBLISHER__ORIGIN_URL="http://127.0.0.1:$origin_port" + export TRUSTED_SERVER__SOURCE__ORIGIN__URL="http://127.0.0.1:$origin_port" "$SMOKE_TS_BIN" config validate \ --manifest "$repository/edgezero.toml" \ --app-config "$SMOKE_APP_CONFIG" \ From ff33fea7459ca714da7ed84887b89329fab9f63b Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 10:17:35 +0100 Subject: [PATCH 06/17] A page names itself by the Host its source sends A source may fetch through another host and send the publisher's own name as the Host, as for an origin behind a proxy at a different address. The page then names that Host's name and never the address's host. Core looked for the address's host, which the page does not carry, so it rewrote none of the page's own addresses, and every middleware was told the proxy's host as the page's origin. Core now hands the page pipeline and every middleware the Host the source sends as the page's origin host, for a page fetched for the request and for one served from the store alike. Where a source sends no other Host the two names are the same, so nothing else changes. The fetch itself still goes to the source's address. Checked: core's library tests pass natively, including a new test that fetches a page through another address and finds its own links pointed at the request's host, and cargo test-axum, test-cloudflare, test-spin and test-fastly pass. Clippy on core with -D warnings is clean. --- .../trusted-server-core/src/html_processor.rs | 3 +- crates/trusted-server-core/src/middleware.rs | 5 +- crates/trusted-server-core/src/publisher.rs | 69 +++++++++++++++++-- 3 files changed, 69 insertions(+), 8 deletions(-) diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 7d48594772..11fd8839c4 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -329,7 +329,8 @@ pub struct ReaderDocument<'a> { pub request_host: &'a str, /// Publisher-facing scheme the reader asked for. pub request_scheme: &'a str, - /// Host the document was fetched from. + /// The host the document names as its own, being the `Host` its source + /// sends. pub origin_host: &'a str, /// What the adapter reports about the platform the deployment runs on. pub platform: &'a PlatformIdentity, diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index f9e3390f96..77d716f9a2 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -140,8 +140,9 @@ pub struct MiddlewareContext<'a> { pub request_host: &'a str, /// Publisher-facing scheme the reader asked for. pub request_scheme: &'a str, - /// Host the document was fetched from, the same for a page served from - /// the store as for one fetched for this request. + /// The host the document names as its own, being the `Host` its source + /// sends, the same for a page served from the store as for one fetched + /// for this request. pub origin_host: &'a str, /// What the adapter reports about the platform the deployment runs on, /// as the attestation evidence carries it. diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 890351127a..01642a5ed4 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -2654,7 +2654,7 @@ fn build_template_assembly_params( seam_ad_slots: None, policy_headers: Vec::new(), content_encoding: entry.metadata.content_encoding.clone(), - origin_host: location.host(), + origin_host: location.host_header(), origin_url: location.url().to_owned(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), @@ -5368,6 +5368,10 @@ pub async fn handle_publisher_request( message: "backend registration failed".to_string(), })?; let origin_host = location.host(); + // The name the origin answers to, sent as its `Host`. A page names itself + // by it, so its own addresses are looked for under this name and not the + // address's host, which differ where the source fetches through another + // host. let origin_host_header = location.host_header(); if let Some(status) = refused_origin_request(&location, parsed_origin.path(), &req) { log::debug!("Publisher origin request refused before sending: {status}"); @@ -6016,7 +6020,7 @@ pub async fn handle_publisher_request( let module_context = ModuleContext::new(module_request) .with_settings(settings) .with_request_state(ec_context, services); - let stored_origin_host = location.host(); + let stored_origin_host = location.host_header(); let document = TextResponse { content_type: crate::middleware::HTML_MEDIA_TYPE, request_path: &request_path, @@ -6200,7 +6204,7 @@ pub async fn handle_publisher_request( let header_reader = crate::html_processor::ReaderDocument { request_host, request_scheme, - origin_host: &origin_host, + origin_host: &origin_host_header, platform: integration_registry.platform(), request_state: &request_state, max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, @@ -6429,7 +6433,7 @@ pub async fn handle_publisher_request( request_path: &request_path, request_host, request_scheme, - origin_host: &origin_host, + origin_host: &origin_host_header, request_state: &request_state, max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, prepared: &prepared, @@ -6558,7 +6562,7 @@ pub async fn handle_publisher_request( seam_ad_slots, policy_headers, content_encoding, - origin_host, + origin_host: origin_host_header, origin_url: location.url().to_owned(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), @@ -24090,6 +24094,61 @@ mod tests { ); } + #[tokio::test] + async fn a_page_fetched_through_another_host_is_rewritten_under_its_own_name() { + let mut settings = create_test_settings(); + edit_origin(&mut settings, |origin| { + origin.url = "https://fetch.example/through".to_string(); + origin.host_header = Some("www.example.com".to_string()); + }); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + br#"news"# + .to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let req = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + + let PublisherResponse::Stream { body, params, .. } = + run_publisher_proxy(&settings, &services, req).await + else { + panic!("should stream the page through the page pipeline"); + }; + assert_eq!( + stub.recorded_request_uris(), + vec!["https://fetch.example/through/page"], + "should fetch the page from the source's address" + ); + assert_eq!( + params.origin_host, "www.example.com", + "the page's own host is the Host its source sends" + ); + let mut output = Vec::new(); + stream_publisher_body( + body, + &mut output, + ¶ms, + &settings, + &test_registry(&settings), + ) + .expect("should rewrite the page"); + let html = String::from_utf8(output).expect("should be UTF-8"); + assert!( + html.contains("publisher.example/news"), + "should point the page's own address at the request host: {html}" + ); + assert!(!html.contains("www.example.com"), "{html}"); + } + /// The publisher origin request as `origin_url`'s path, `directory_index` /// and `origin_auth` shape it, for an origin such as object storage. mod object_storage_origin_tests { From 6eabcc695c588456a33365793aa8030329bf94d7 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 17:26:22 +0100 Subject: [PATCH 07/17] The source modules design states the stored keys and page host it changes, and its limits Review round 1 found that the keys of stored pages and of the publisher's assets change with this series, that a page and every middleware are now told the Host the source sends as the page's origin host, and that the behavior the series adds to asset routes and to modules was missing from the design. The design now says all three, adds configuration, adapters, consent and privacy, failure handling and rollout, lists the paths with no test, records the spelling of core's source as a known gap and records the review round. The changelog entry no longer says stored keys are unchanged and now states the host change. --- CHANGELOG.md | 2 +- .../specs/2026-10-14-source-modules-design.md | 115 +++++++++++++++--- 2 files changed, 100 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6373a277bd..f64abd3e19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- **Breaking:** Where content comes from is a source module. `[source]` selects the publisher's, and each `[[proxy.asset_routes]]` entry selects its own in a `source` table of the same shape, with `module` naming the source and its settings in the table beneath. Core supplies `origin`, an HTTP origin, and a module a build links can offer another through `IntegrationRegistrationBuilder::with_source_module`. A source answers for each request where that request's content comes from, and core fetches, caches, rewrites the origin host and follows the redirect rule from that answer. `[source]` is required. The origin settings move into the `origin` table, and a document that still writes one where it was is refused at startup naming the new place. An environment variable that set the address is now `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The old name is refused where the build reads variables, and `ts config push` ignores it, as it ignores any variable for a key the document does not hold. A path in an asset route's `url` names a folder the target path is added under, where a path was refused, a target path with a `.` or `..` segment under a folder is answered `400`, and a route applies `directory_index` as the publisher does. An existing deployment's stored pages and assets keep their keys. **For crates outside core**, `handle_asset_proxy_request` takes the route's source, which `IntegrationRegistry::route_source` gives, and `Publisher::origin_host`, `origin_host_header` and `origin_query_is_stripped` are replaced by `Settings::origin_source` and the `SourceLocation` a source answers. +- **Breaking:** Where content comes from is a source module. `[source]` selects the publisher's, and each `[[proxy.asset_routes]]` entry selects its own in a `source` table of the same shape, with `module` naming the source and its settings in the table beneath. Core supplies `origin`, an HTTP origin, and a module a build links can offer another through `IntegrationRegistrationBuilder::with_source_module`. A source answers for each request where that request's content comes from, and core fetches, caches, rewrites the origin host and follows the redirect rule from that answer. `[source]` is required. The origin settings move into the `origin` table, and a document that still writes one where it was is refused at startup naming the new place. An environment variable that set the address is now `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The old name is refused where the build reads variables, and `ts config push` ignores it, as it ignores any variable for a key the document does not hold. A path in an asset route's `url` names a folder the target path is added under, where a path was refused, a target path with a `.` or `..` segment under a folder is answered `400`, and a route applies `directory_index` as the publisher does. Stored pages and the publisher's stored assets are fetched again after the upgrade, because their keys hold a code worked out from the settings, but an asset route's stored copies keep their keys. Where `host_header` is set, the page pipeline and every middleware are now told that `Host` as the page's origin host, where they were told the address's host, so the page's own addresses under that name are rewritten to the request's host. **For crates outside core**, `handle_asset_proxy_request` takes the route's source, which `IntegrationRegistry::route_source` gives, and `Publisher::origin_host`, `origin_host_header` and `origin_query_is_stripped` are replaced by `Settings::origin_source` and the `SourceLocation` a source answers. | Previous | Now | | ----------------------------------------- | ----------------------------------------- | diff --git a/docs/superpowers/specs/2026-10-14-source-modules-design.md b/docs/superpowers/specs/2026-10-14-source-modules-design.md index 1c99a41d18..825ff4a5d2 100644 --- a/docs/superpowers/specs/2026-10-14-source-modules-design.md +++ b/docs/superpowers/specs/2026-10-14-source-modules-design.md @@ -1,11 +1,11 @@ # Design Spec: A source module says where the publisher path and the asset routes fetch from -**Status:** Proposed, 2026-10-14. Implemented on a working branch of the fork, with no pull request raised yet. +**Status:** Proposed, 2026-10-14. Raised as draft pull request jwrosewell/trusted-server#34, for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review **Related specs:** `2026-10-13-publisher-object-storage-origin-design.md`, `2026-10-13-origin-redirect-design.md`, `2026-10-13-request-addressing-design.md`, `2026-04-28-multi-backend-asset-proxy-design.md` -**Related PRs:** none yet +**Related PRs:** jwrosewell/trusted-server#34 **Issues:** IABTechLab/trusted-server#1332 -**Last updated:** 2026-10-14 +**Last updated:** 2026-10-15 > **Why this spec exists.** Core's publisher path fetches from the origin `[publisher]` names and each asset route from the origin it names, so no module can say where a request's content comes from, and where it comes from is written in two places in two shapes. @@ -47,7 +47,7 @@ url = "https://bucket.object-store.example/site" auth = { type = "s3_sigv4", region = "eu-west-1", access_key_id = "id_name", secret_access_key = "key_name" } ``` -`[publisher]` keeps who the publisher is and how its pages are processed. Every field that said where the pages come from moves to the `origin` source's table, under the names the table above shows: +`[publisher]` keeps who the publisher is and how its pages are processed. Every field that said where the pages come from moves to the `origin` source's table, under the names in this table: | Was | Is | | ----------------------------------------- | ----------------------------------------- | @@ -58,30 +58,113 @@ auth = { type = "s3_sigv4", region = "eu-west-1", access_key_id = "id_name", sec | `[[proxy.asset_routes]] origin_url` | `[proxy.asset_routes.source.origin] url` | | `[[proxy.asset_routes]] auth` | `[proxy.asset_routes.source.origin] auth` | -An asset route keeps `prefix`, `path_pattern`, `target_path` and `image_optimizer`, which say which requests it answers and what path it asks for. A path in an asset route's `url` names a folder the target path is added under, as a path in the publisher's `url` does. A document that writes a field where it used to be is refused at startup, with the new place named in the message, and a document with no `[source]` is refused. A source table naming a module the build does not link is refused with the names on offer, as every other module selection is. +The two earlier designs for the origin, `2026-10-13-publisher-object-storage-origin-design.md` and `2026-10-13-origin-redirect-design.md`, name these settings by the places they had in `[publisher]`. + +An asset route keeps `prefix`, `path_pattern`, `target_path` and `image_optimizer`, which say which requests it answers and what path it asks for. A document that writes a moved field where it used to be is refused at startup, with the new place named in the message, and so is a configuration blob, being the stored copy of the settings that a running service loads. A document with no `[source]` is refused, and so is an asset route with no `source`, naming the route. A source table naming a module the build does not link is refused with the names on offer, as every other module selection is. The configuration page masks `url` and `host_header` by default, as it masked the fields they replace, and the signing secrets are declared at their new paths. -**What a source answers.** For each request a source first answers where the content comes from: the origin's scheme, host, port and folder, the `Host` header to send, how the request is signed and whether its query is kept, and the folder's index document. Core's caches key on that answer, and core rewrites the origin's host in the page, follows the redirect rule and tells middleware the origin host from it, so a source that answers differently for each request is kept apart in every one of those places. +**What an asset route gains.** A route's source takes the same settings as the publisher's, so a route gains the two it had no place for, `host_header` and `directory_index`. A path in a route's `url`, which was refused before, now names a folder the target path is added under, as a path in the publisher's `url` does. Where the address has a folder, the route refuses a target path with a `.` or `..` segment, written plainly or percent-encoded, and answers `400 Bad Request` without sending anything, as the publisher path does for a reader's path. The route checks the target path after its rewrite, and the publisher path checks the reader's path. A target path ending in `/` has the source's `directory_index` added. + +**Configuration.** Core's source is selected with `module = "origin"`, and its settings sit under `[source.origin]`, or under `[proxy.asset_routes.source.origin]` for a route. An unknown key in either table is refused, and the message names the key. + +| Setting | Required | When omitted | Accepted | +| ----------------- | -------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `url` | Yes | The settings are refused | An absolute `http` or `https` address with a host, and no user name, password, query, fragment or trailing `/`. A path names a folder. Spaces around the address are trimmed | +| `host_header` | No | The `Host` sent is the host of `url`, with its port unless that is the usual port for the scheme | A host, or a host and port, and nothing else | +| `directory_index` | No | No file is added, and every path is sent as it arrived | One file name, with no `/`, `%`, space, query or fragment, and not `.` or `..` | +| `auth` | No | Requests are sent unsigned | A signing table the signer can use, beside a `url` that uses `https` | + +Environment variables name the settings of the publisher's source with the prefix `TRUSTED_SERVER` and `__` between the parts, such as `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. When the `ts` command reads a document it applies the environment variables as an overlay, which replaces the value of a setting the document already holds and adds no setting, so a variable cannot create a table the document lacks. The overlay replaces single values and reaches no element of a list, so no variable sets an asset route's source. + +**What a source answers.** For each request a source answers where the content comes from, as a location made of the origin's address (scheme, host, port and folder), the `Host` header to send, how the request is signed, and the index file added to a path ending in `/`. Whether the reader's query goes to the origin is not a separate answer. The signing's `origin_query` setting says so, and for an asset route the route's Image Optimizer settings and its default say so where the signing does not. Core uses the location in five ways: + +1. It fetches the address, sending the `Host` and the signing. +2. It keys the stored pages and assets on the address and the `Host`. +3. It looks for the page's own addresses under the `Host` and rewrites them to the request's host. +4. It compares the `Location` of a redirect with the `Host`, to find a redirect the origin sends to itself. +5. It tells every middleware the `Host` as the page's origin host. + +A source that answers differently for each request is kept apart in all five. + +Core fetches the location with the same HTTP client as before, streaming the response where the platform offers it, and reads and writes the same stored pages and stored assets. Where no stored page or stored asset has to be looked up first, core starts the origin request in the background while the Edge Cookie record is read, as before. + +**What the stored copies are keyed on.** The origin part of each key is made from the location in the form the old settings gave, so it does not change for an existing deployment. For a page and for the publisher's assets it is the address and the `Host` sent. For a copy held for an asset route it is the address and the route's prefix, and the `Host` as well where the source sets one, which no route could before. The two forms differ, so the publisher path and an asset route never share a stored copy, even for one address. + +The key of a stored page and of a stored publisher asset also holds a digest of the settings as written out and of core's source files. A digest is a short code worked out from content, so it changes whenever the content does. This series changes both, because the settings now write out a `[source]` section where `[publisher]` held the origin settings, and core's source changed. Every stored page and every stored asset of the publisher is therefore fetched from the origin again on its first request after the upgrade, as it is after any release that changes core's source. The key of an asset route's copy holds no digest, so the copies of asset routes survive the upgrade. + +**The host a page names itself by.** A source may fetch through one address and send the publisher's own name as the `Host`, as for an origin behind a proxy at another address. The page then names the `Host` and never the address's host. Before this series core looked for the address's host, so for a deployment that set `origin_host_header_override` the page pipeline found none of the page's own addresses to rewrite, and every middleware was told the address's host as the page's origin host. Core now hands the page pipeline and every middleware the `Host` the source sends, for a page fetched for the request and for one served from the store alike. The page's own addresses are pointed at the request's host, and `MiddlewareContext::origin_host` holds that `Host`. An HTML page whose links name only the address's host, and not the `Host`, was rewritten before and is not now. Where a source sends no other `Host` the two names are the same and nothing changes. The fetch still goes to the address, and the redirect rule compared a `Location` with the `Host` before and still does. + +**One call for every path.** The publisher's page, the publisher's assets held in the asset cache, and each asset route ask their source with the same call, `locate`, and the same description of the request. The rules after that call are not all shared. The keys of their stored copies differ, as the paragraphs above set out. The publisher path answers `405` to a request that is not `GET` or `HEAD` when its location signs, where the Fastly adapter sends an asset route only `GET` and `HEAD`. Both refuse a `.` or `..` segment where the address has a folder, though the publisher path checks the reader's path and an asset route checks its target path after the rewrite. + +**The request a source is given.** The method, the path and query the request continues with, the request's headers, and the prefix request addressing removed, if any. That lets a source take where to fetch from out of the address. The same prefix is on the module request that every module is given, through `ModuleRequest::removed_prefix`, so a module that names things after the tenant a path addressed, such as a prefix module naming cookies after the publisher, can tell which tenant it is. No module in this repository reads that accessor yet, and only a test does. -Core fetches the location a source answers through the same send, the same early send, streaming and the same cache paths as today. The `origin` source answers the address, `Host` and signing the old fields named, and the cache keys are made from them in the same form, so an existing deployment's stored pages and assets keep their keys. +**Consent and privacy.** A source is not governed by the permission model. A module declares the permissions its data use needs, and core runs the module only when each is set for the request (see "Module Permissions" in `docs/guide/configuration.md`). A source declares none, so core asks the publisher's source about every request, and an asset route's source about every request the route answers, whatever the request's permissions are. The request a source is given holds the headers of the request as core received it, which include `Cookie` and `Authorization`. Core's own `origin` source reads none of them. A source from a module can read all of them, and core does not limit which it reads. A source that sends a header to another service to decide where content comes from therefore answers for itself whether the reader's permissions allow that. -**One call for every path.** The publisher's page, the publisher's assets held in the asset cache, and each asset route ask their source the same way, so content a source answers for is fetched and cached on the same terms whichever path asked. +**How each adapter serves a source.** All four adapters read `[source]` and `[[proxy.asset_routes]]`, and build every source when they build the integration registry, which holds the modules a build links. -**The request a source is given.** The method, the path and query the request continues with, its headers, and the prefix request addressing removed, if any. That lets a source take where to fetch from out of the address. +| Adapter | Asks the publisher's source for a page | Serves asset routes and asks each route's source | +| ---------- | -------------------------------------- | ------------------------------------------------ | +| Fastly | Yes | Yes, for `GET` and `HEAD` | +| Axum | Yes | No | +| Cloudflare | Yes | No | +| Spin | Yes | No | + +Only the Fastly adapter dispatched asset routes before this change, and it still does. This change altered production code in that adapter alone, in `dispatch_fallback` and `dispatch_asset_fallback`, which now match a route to its position in the list and take that route's source from `IntegrationRegistry::route_source`. The Axum, Cloudflare and Spin adapters changed only in the settings their tests write, so on those three a `[[proxy.asset_routes]]` entry is read and never served, as before. + +**When it fails.** A source that cannot be built from its settings fails the building of the integration registry, and every adapter builds the registry. The Fastly adapter then answers each request with the startup error, being `500` for a configuration error. A source that cannot say where one request's content comes from returns an error from `locate`, and core answers that request with the status of the error, being `400` for a bad request, `502` for a proxy error and `500` for a configuration error. A bad request puts its own message in the body, and the other errors answer with a short generic one. On the publisher path `locate` runs before any stored page is read, any auction starts and any origin request is sent. On the Fastly adapter an asset route's error response is marked so that nothing stores it. Core fetches the location as returned and does not check again what core's own `origin` checks when it reads its settings. A location that carries signing is signed even when its address is `http`, where `origin` refuses `auth` beside an `http` address, so a source that signs keeps to `https` itself. + +**Rollout.** This change is breaking, and the changelog labels it so. Every deployment's settings change shape, so the build and the settings go live together. A new build refuses a configuration blob of the old shape, naming the new place, and an old build refuses a blob of the new shape, because that blob has no `publisher.origin_url`. Neither order of the two works, and a rollback restores the old build and the old blob together, as the changelog entry for the move of the auction providers also sets out. Four things change with the move: + +1. A variable that set a moved setting under its old name, such as `TRUSTED_SERVER__PUBLISHER__ORIGIN_URL`, names a setting the moved document no longer holds, so the overlay the `ts` command applies ignores it without a message. The new name is `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The settings refuse the template's placeholder address when a deployment is validated, which stops a pipeline that sets the old name over an unedited template. A pipeline that sets the old name over a document holding a real address uses the document's address and not the variable's, with no message. +2. An `[inspect]` pattern in `show` or `hide` that names an old path, such as `publisher.origin_url` or `proxy.asset_routes[].origin_url`, matches no value in the new settings, because the moved fields are no longer written out. The section refuses a pattern that matches no value (`validate_patterns` in `crates/trusted-server-core/src/inspect/config.rs`), so the configuration is refused until the pattern names the new path. +3. Stored pages and the publisher's stored assets are fetched again after the upgrade, as the paragraph on the stored copies sets out. The stored copies of asset routes keep their keys. +4. A deployment that set `origin_host_header_override` has its pages rewritten under the `Host`, as the paragraph on the host a page names itself by sets out. + +**Also in this series.** `handle_proxy` hands an integration route its request's evidence, being the client address, the headers after the caller's Edge Cookie header is removed and the address asked for. A route that asked for the request as evidence was refused before and answered `500`. The fix was made on another working branch of the fork and is carried here, and its test goes through `handle_proxy`. ## 3. What is not in this change - Any source other than `origin`. A test supplies one that answers a different origin for each request, which is what a source reading the address does. - A source supplying the response itself, for content that has no HTTP origin. That is a second method on the interface, added with the first source that needs it. -- Selecting a source per path for the publisher's pages. A deployment wanting that writes an asset route. +- **Selecting a source per path for the publisher's pages.** A deployment that wants one can write an asset route, which asks a source of its own, but only on the Fastly adapter, which alone serves asset routes. An asset route answers `GET` and `HEAD` only and passes the origin's status, body and safe headers through without the page pipeline. The paths it answers therefore lose the page changes of the `[[fetch]]` and `[[serve]]` middleware, the rewrite of the origin's own addresses in a page, the rule that keeps a redirect to the origin on the reader's address, the stored pages, the auction and the Edge Cookie work done on a page. +- **Core's source under its full name.** Write `module = "origin"`, with the settings in `[source.origin]`. The registry also resolves the full name `module = "source.origin"`, which reads its settings from `[source.source.origin]`, but the code that reads core's source by the bare name `origin` does not see such a selection. That code is the early check of the address when the settings load (`check_selection`), `Settings::origin_source` and the checks built on it when a deployment is validated, being the placeholder address and the signing key references, the default masking of `url` and `host_header` on the configuration page (`serialize_selection`), and the declaration of the signing secrets, which names `source.origin.auth` and `proxy.asset_routes[*].source.origin.auth` only. The address is still checked when the registry is built. No test writes the full name. + +## 4. Acceptance -## 4. Tests +Tests in `trusted-server-core`, where the code is, unless a file is named: -- Each old field is refused naming its new place, in a document and in a configuration blob, and a document with no `[source]` is refused. -- `[source.origin]` reads, trims and checks its address, `Host`, index file and signing, and refuses an unknown key, naming the setting. Environment variables set the same fields at their new paths. -- A location gives the host with its port, the `Host` it sends, a folder from the address's path, and an identity in the form the caches used before, so the stored keys do not change. -- A source answering a different origin for each request sends each request to its own origin and tells the page pipeline that request's origin host. +- Each moved field is refused naming its new place. In a document all six are tested, the publisher's four in one test and a route's two in another. In a configuration blob only the publisher's `origin_url` is tested, and the other five are refused by the same code with no blob test of their own. A document with no `[source]` is refused, and a section naming no source is refused by the early check. +- `[source.origin]` reads, trims and checks its address, `Host`, index file and signing, and refuses an unknown key, naming the setting. +- Environment variables set the same settings at their new paths, and an old name is refused. These tests read variables with a helper that exists only in test builds, `Settings::from_toml_and_env`. The helper adds a setting the document lacks and reaches the elements of a list, which the overlay the `ts` command applies does not, so the helper refuses an old name and sets a route's settings where the overlay ignores the one and cannot do the other. No test sets an old name through the overlay. +- A location gives the host with its port, the `Host` it sends, a folder from the address's path, and an identity in the form the caches used before, so the origin part of the stored keys does not change. A route's copy is named by its address and prefix, and by the `Host` as well where its source sets one. +- A source answering a different origin for each request sends each request to its own origin and tells the page pipeline that request's origin host. The test keeps no stored copies. - An unknown source name is refused with the names on offer. -- An asset route reads its source, fetches its target path under the source's folder, and signs with the source's signing. +- An asset route reads its source and signs with the source's signing. The function that builds its target address puts the path under the source's folder, adds the source's index file to a path ending in `/`, and refuses a target path with a `.` or `..` segment under a folder with `400`, in four spellings. +- A page fetched through another address, with the `Host` its source sends, names itself by that `Host`, and its own addresses are pointed at the request's host. - The configuration page masks the origin's address and `Host` by default and declares the signing secrets at their new paths. +- A route is handed its request's evidence through `handle_proxy`, and a module request carries the removed prefix, which a held request keeps. + +Nothing tests these paths: + +- A source answering a different origin for each request under the stored pages and the stored assets, so no test shows two such origins kept apart in either store. +- A redirect under such a source. +- An asset route under such a source. Every route test uses core's `origin` source. +- A source that returns an error from `locate`, on the publisher path or on a route, and the response each adapter gives. +- The route's `400` and index file as answers of the handler. The tests stop at the function that builds the target address. +- The host a page names itself by for a page served from the store, and the `origin_host` a middleware is told. The test covers a page fetched for the request. +- An asset route with no `source`. +- The full name `module = "source.origin"`. +- An `[inspect]` pattern that names an old path. +- An old environment variable name through the overlay the `ts` command applies. + +## Review log + +1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/39-source-modules` at `ff33fea74`. Issues found, three of them blocking. The claim that stored pages and assets keep their keys was false for pages and the publisher's assets, the change to the host that a page and every middleware are told as the page's origin host was missing, and the behavior the series adds to asset routes and to modules was missing. This revision corrects the claim and states the other two, and adds the configuration, adapter, consent and privacy, failure and rollout paragraphs, the paths that have no test and a known gap in the spelling of core's source. + +## Revision record + +| Date | Change | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-10-14 | First draft. | +| 2026-10-15 | Fixes from review round 1. What the stored keys hold, the host a page names itself by, the behavior added to asset routes and modules, configuration, adapters, consent and privacy, failure handling, rollout, the untested paths, the spelling of core's source and the review log. | From 0f51b33fef57fd580b6515e7745d931e50fc9b4e Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 17:40:56 +0100 Subject: [PATCH 08/17] The source modules design records its round 2 review Round 2 approved the design with no blocking issues. It could not confirm that the per-request source test keeps no stored copies, and that holds. The test runs in the default assembly mode, which stores no page, with test services that select no cache, which store no asset, and their template cache is the unavailable one. The design names no design documents pull request, so no reference changed. The review log and the revision record carry the round. --- docs/superpowers/specs/2026-10-14-source-modules-design.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/superpowers/specs/2026-10-14-source-modules-design.md b/docs/superpowers/specs/2026-10-14-source-modules-design.md index 825ff4a5d2..3b67394074 100644 --- a/docs/superpowers/specs/2026-10-14-source-modules-design.md +++ b/docs/superpowers/specs/2026-10-14-source-modules-design.md @@ -161,6 +161,7 @@ Nothing tests these paths: ## Review log 1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/39-source-modules` at `ff33fea74`. Issues found, three of them blocking. The claim that stored pages and assets keep their keys was false for pages and the publisher's assets, the change to the host that a page and every middleware are told as the page's origin host was missing, and the behavior the series adds to asset routes and to modules was missing. This revision corrects the claim and states the other two, and adds the configuration, adapter, consent and privacy, failure and rollout paragraphs, the paths that have no test and a known gap in the spelling of core's source. +2. **Round 2, 2026-10-15.** A lighter review in a fresh session, against this page at `6eabcc695`, of the round 1 fixes and the new text. Approved, with no blocking issues. No change was needed. ## Revision record @@ -168,3 +169,4 @@ Nothing tests these paths: | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-10-14 | First draft. | | 2026-10-15 | Fixes from review round 1. What the stored keys hold, the host a page names itself by, the behavior added to asset routes and modules, configuration, adapters, consent and privacy, failure handling, rollout, the untested paths, the spelling of core's source and the review log. | +| 2026-10-15 | Review round 2 recorded, with no change to the design. | From 878941c16471b7766acc558002b529f34723bba9 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 12:35:58 +0100 Subject: [PATCH 09/17] Set the config push tests' origin under [source.origin] This branch moves the origin's address from [publisher] origin_url to [source.origin] url, and the local test scripts set it there. The config push tests from the config by host change still set TRUSTED_SERVER__PUBLISHER__ORIGIN_URL, so every push they ran was refused for the example's insecure default origin. They now set TRUSTED_SERVER__SOURCE__ORIGIN__URL, as config_store_defaults.rs does. --- crates/trusted-server-cli/tests/config_push_by_key.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/trusted-server-cli/tests/config_push_by_key.rs b/crates/trusted-server-cli/tests/config_push_by_key.rs index 443b43e1ff..7d198a3e33 100644 --- a/crates/trusted-server-cli/tests/config_push_by_key.rs +++ b/crates/trusted-server-cli/tests/config_push_by_key.rs @@ -88,7 +88,7 @@ fn push(project: &TempDir, args: &[&str]) -> Output { format!(".{HOST}"), ) .env( - "TRUSTED_SERVER__PUBLISHER__ORIGIN_URL", + "TRUSTED_SERVER__SOURCE__ORIGIN__URL", "https://origin.publisher.example", ) .output() From 14dfd6b6d3523537a55b4d3d6c35e8e40d956d66 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 17:48:01 +0100 Subject: [PATCH 10/17] The CMP id test names its origin under the origin source The test of the InMobi module's CMP id range gave its settings a real origin by writing the publisher table's origin_url, which this branch moves to [source.origin] and no longer exposes, so the module's tests did not compile from here up. The test now sets the origin through Settings::set_origin_source, as the other tests here do. --- crates/cmp/inmobi/src/lib.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/crates/cmp/inmobi/src/lib.rs b/crates/cmp/inmobi/src/lib.rs index ad5ea528cf..1125d7ec67 100644 --- a/crates/cmp/inmobi/src/lib.rs +++ b/crates/cmp/inmobi/src/lib.rs @@ -1150,7 +1150,9 @@ mod tests { "cmp_id = {value}\n{}", html_entry(None, &[MODULE]) )); - settings.publisher.origin_url = "https://origin.publisher.example".to_owned(); + settings.set_origin_source(&trusted_server_core::source::OriginSettings::new( + "https://origin.publisher.example", + )); settings }; let load = |settings: &Settings| { From c0efb11a53e82c4e1b1b16ae71f0a22bcad3c0d0 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 20:52:30 +0100 Subject: [PATCH 11/17] The design header names the pull request that carries it --- docs/superpowers/specs/2026-10-14-source-modules-design.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-14-source-modules-design.md b/docs/superpowers/specs/2026-10-14-source-modules-design.md index 3b67394074..d31a7af3fb 100644 --- a/docs/superpowers/specs/2026-10-14-source-modules-design.md +++ b/docs/superpowers/specs/2026-10-14-source-modules-design.md @@ -1,11 +1,11 @@ # Design Spec: A source module says where the publisher path and the asset routes fetch from -**Status:** Proposed, 2026-10-14. Raised as draft pull request jwrosewell/trusted-server#34, for Tech Lab review. +**Status:** Proposed, 2026-10-14. Raised as draft pull request IABTechLab/trusted-server#1380 (from jwrosewell/trusted-server#34), for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review **Related specs:** `2026-10-13-publisher-object-storage-origin-design.md`, `2026-10-13-origin-redirect-design.md`, `2026-10-13-request-addressing-design.md`, `2026-04-28-multi-backend-asset-proxy-design.md` -**Related PRs:** jwrosewell/trusted-server#34 +**Related PRs:** jwrosewell/trusted-server#34, IABTechLab/trusted-server#1380 **Issues:** IABTechLab/trusted-server#1332 -**Last updated:** 2026-10-15 +**Last updated:** 2026-10-16 > **Why this spec exists.** Core's publisher path fetches from the origin `[publisher]` names and each asset route from the origin it names, so no module can say where a request's content comes from, and where it comes from is written in two places in two shapes. From 460aa83b32ddd0a431ecfa8109b0b02fb5a0e70d Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 22:19:40 +0100 Subject: [PATCH 12/17] Three tests name their origin in a source section The publisher's origin has moved from `[publisher] origin_url` to `[source.origin] url`, and settings refuse it in the old place. The Axum and Fastly tests of `/robots.txt` through the publisher path and the Cloudflare test of the attestation pages still wrote it in `[publisher]`, so each now selects the origin module in `[source]` and names the address there, as the other tests do. --- crates/trusted-server-adapter-axum/tests/routes.rs | 7 ++++++- crates/trusted-server-adapter-cloudflare/tests/routes.rs | 7 ++++++- crates/trusted-server-adapter-fastly/src/app.rs | 7 ++++++- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index fd7a2f4f88..c33cfa219b 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -973,10 +973,15 @@ async fn the_sellers_module_makes_an_ads_txt_the_origin_does_not_have() { fn settings_refusing_every_crawler() -> trusted_server_core::settings::Settings { trusted_server_core::settings::Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "integration-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index 5741eab7a4..0bd366f7f2 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -655,10 +655,15 @@ async fn attestation_routes_skip_the_request_preparers() { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 1246245f08..3b51ee0d7a 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -3262,10 +3262,15 @@ mod tests { let settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.com" + [publisher] domain = "test-publisher.com" cookie_domain = ".test-publisher.com" - origin_url = "https://origin.test-publisher.com" proxy_secret = "unit-test-proxy-secret" [ec] From 6f7238f5c39232a3ba7cdde181cb4f0ed4d5bd09 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Sun, 18 Oct 2026 22:23:16 +0100 Subject: [PATCH 13/17] The failed asset load test passes the route's source handle_asset_proxy_request takes the route's source since sources became modules, and the test added with the failed load fix called it without one, so the core library tests did not compile. The test now passes origin_source_of(&route), as the other asset route tests do. --- crates/trusted-server-core/src/proxy.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs index 13eead23ec..05b3aa0ce8 100644 --- a/crates/trusted-server-core/src/proxy.rs +++ b/crates/trusted-server-core/src/proxy.rs @@ -5525,8 +5525,14 @@ mod tests { build_http_request(Method::GET, "https://www.example.com/.images/site.css"); // Nothing is queued, so the stub fails the origin request. - let Err(error) = - handle_asset_proxy_request(&settings, &services, request, &route).await + let Err(error) = handle_asset_proxy_request( + &settings, + &services, + request, + &route, + &origin_source_of(&route), + ) + .await else { panic!("should fail when the origin request fails"); }; From 315cdf03b3e9c7b012134ac8e947cae9c09e0756 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Mon, 19 Oct 2026 14:30:45 +0100 Subject: [PATCH 14/17] The header rewrite's messages, tests and guide name the source settings The header rewrite merged in main (#1238) names publisher.origin_url and publisher.origin_host_header_override in two warnings, a doc comment and the configuration guide, and its handler tests set those fields. This change moves both settings to source.origin.url and source.origin.host_header, so the warnings, the comment and the guide name the new settings and the tests set them through edit_origin, as the change's own tests do. Without the test change core's tests do not compile here. cargo test -p trusted-server-core --lib -- redirect response_header_rewrite passes 58 tests. --- crates/trusted-server-core/src/publisher.rs | 8 +++++--- crates/trusted-server-core/src/response_header_rewrite.rs | 6 +++--- docs/guide/configuration.md | 6 +++--- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index e3c27bfbc6..e517fc2d4b 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -19960,7 +19960,7 @@ mod tests { location: &str, ) -> String { let mut settings = create_test_settings(); - settings.publisher.origin_url = origin_url.to_string(); + edit_origin(&mut settings, |origin| origin.url = origin_url.to_owned()); let stub = Arc::new(StubHttpClient::new()); stub.push_response_with_headers( 303, @@ -20014,7 +20014,7 @@ mod tests { #[tokio::test] async fn cookieless_get_redirect_to_current_url_is_not_rewritten_into_a_loop() { - // A canonical-host redirect triggered by `origin_host_header_override` + // A canonical-host redirect triggered by `source.origin.host_header` // answers the retry through Trusted Server the same way, so it must leave. for method in [Method::GET, Method::HEAD] { let location = redirect_location_for_current_url( @@ -20034,7 +20034,9 @@ mod tests { #[tokio::test] async fn overridden_host_redirect_with_cookie_is_not_rewritten_into_a_loop() { let mut settings = create_test_settings(); - settings.publisher.origin_host_header_override = Some("publisher.example".to_string()); + edit_origin(&mut settings, |origin| { + origin.host_header = Some("publisher.example".to_owned()); + }); let stub = Arc::new(StubHttpClient::new()); stub.push_response_with_headers( 301, diff --git a/crates/trusted-server-core/src/response_header_rewrite.rs b/crates/trusted-server-core/src/response_header_rewrite.rs index 52149d2733..9dd8e07220 100644 --- a/crates/trusted-server-core/src/response_header_rewrite.rs +++ b/crates/trusted-server-core/src/response_header_rewrite.rs @@ -163,7 +163,7 @@ impl OriginHeaderRewrite<'_> { if self.is_scheme_change_to_current_request(url) { log::warn!( "Keeping origin scheme-change redirect to the current request URL; the \ - browser leaves the serving host. Check that `publisher.origin_url` uses \ + browser leaves the serving host. Check that `source.origin.url` uses \ the scheme the origin enforces" ); return None; @@ -178,7 +178,7 @@ impl OriginHeaderRewrite<'_> { /// sets no cookie. Nothing the browser sends on the retry would differ, so /// the origin is redirecting because of something Trusted Server sends /// unchanged. When the outbound `Host` differs from the origin host (from - /// `publisher.origin_host_header_override`), the redirect is a host change + /// `source.origin.host_header`), the redirect is a host change /// that a retry cannot satisfy, so it is left unchanged even if the /// response sets a cookie. fn rewrite_location(&self, url: &str, response_sets_cookie: bool) -> Option { @@ -194,7 +194,7 @@ impl OriginHeaderRewrite<'_> { log::warn!( "Keeping origin redirect to the current request URL under a `Host` \ override; the browser leaves the serving host. Check that the origin \ - accepts the `Host` from `publisher.origin_host_header_override`" + accepts the `Host` from `source.origin.host_header`" ); } else { log::warn!( diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 62566d0a74..c6de30d911 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1271,7 +1271,7 @@ Location: https://www.example.com/landing.html Content-Security-Policy: img-src https://origin.example.com https://www.example.com data: ``` -When `[source.origin] url` has a path, such as a folder of object storage, +When `source.origin.url` has a path, such as a folder of object storage, only a URL in that folder is mapped, and it loses the folder. With `url = "https://storage.example.com/bucket/site"`, `https://storage.example.com/bucket/site/news/` becomes @@ -1284,11 +1284,11 @@ Two exceptions prevent a redirect loop. Rewriting either target would send the browser back to the same URL, and the origin would redirect again: - When the origin redirects to the URL being requested but under a different - scheme from `publisher.origin_url` (for example, an `http://` origin forcing + scheme from `source.origin.url` (for example, an `http://` origin forcing HTTPS), the `Location` or `Refresh` target is left unchanged. - When the origin answers a `GET` or `HEAD` with a `Location` to the URL being requested, the target is left unchanged if the response sets no cookie, or - whatever cookies it sets when `publisher.origin_host_header_override` sends a + whatever cookies it sets when `source.origin.host_header` sends a different `Host`. The origin is redirecting because of something Trusted Server sends unchanged on every retry, such as that `Host`. From 4f1cf830cb7a5711ff777a980c630eb57d753cf3 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 28 Oct 2026 11:30:06 +0000 Subject: [PATCH 15/17] The environment files set the origin's address under [source.origin] --- .env.dev | 4 +++- .env.example | 6 +++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.env.dev b/.env.dev index 347bc0a359..838a24c7c8 100644 --- a/.env.dev +++ b/.env.dev @@ -5,7 +5,9 @@ # [publisher] TRUSTED_SERVER__PUBLISHER__DOMAIN=localhost TRUSTED_SERVER__PUBLISHER__COOKIE_DOMAIN=localhost -TRUSTED_SERVER__PUBLISHER__ORIGIN_URL=http://localhost:9090 + +# [source.origin] +TRUSTED_SERVER__SOURCE__ORIGIN__URL=http://localhost:9090 # [synthetic] TRUSTED_SERVER__SYNTHETIC__COUNTER_STORE=counter_store diff --git a/.env.example b/.env.example index 6dcbfa9c4c..3f0738ff87 100644 --- a/.env.example +++ b/.env.example @@ -15,7 +15,11 @@ # ============================================================================= TRUSTED_SERVER__PUBLISHER__DOMAIN=publisher.com TRUSTED_SERVER__PUBLISHER__COOKIE_DOMAIN=.publisher.com -TRUSTED_SERVER__PUBLISHER__ORIGIN_URL=https://origin.publisher.com + +# ============================================================================= +# Source Settings +# ============================================================================= +TRUSTED_SERVER__SOURCE__ORIGIN__URL=https://origin.publisher.com # ============================================================================= # Synthetic ID Settings From 0ca07b1ed4c05745079b01200d00dabeb3a07d09 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 28 Oct 2026 11:30:07 +0000 Subject: [PATCH 16/17] The policy and endpoint fixtures name their origin in a source section --- crates/trusted-server-adapter-axum/src/app.rs | 7 ++++++- crates/trusted-server-adapter-cloudflare/tests/routes.rs | 7 ++++++- crates/trusted-server-adapter-spin/tests/routes.rs | 7 ++++++- crates/trusted-server-core/src/test_support.rs | 7 ++++++- 4 files changed, 24 insertions(+), 4 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index ba8ed043a0..d67ee2eac7 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -1098,10 +1098,15 @@ mod tests { fn startup_settings() -> Settings { Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "publisher_proxy_secret" "#, ) diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index c2bff297d7..39459690aa 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -658,10 +658,15 @@ mod module_endpoints { fn settings_with(extra: &str) -> Settings { Settings::from_toml(&format!( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index 1929b59304..11032fac92 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -696,10 +696,15 @@ mod module_endpoints { fn settings_with(extra: &str) -> Settings { Settings::from_toml(&format!( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.test-publisher.example.com" + [publisher] domain = "test-publisher.example.com" cookie_domain = ".test-publisher.example.com" - origin_url = "https://origin.test-publisher.example.com" proxy_secret = "route-test-proxy-secret" [ec] diff --git a/crates/trusted-server-core/src/test_support.rs b/crates/trusted-server-core/src/test_support.rs index f383ce4f3e..807a543f01 100644 --- a/crates/trusted-server-core/src/test_support.rs +++ b/crates/trusted-server-core/src/test_support.rs @@ -892,10 +892,15 @@ pub mod policy_failure { pub fn load_error() -> Report { let mut settings = Settings::from_toml( r#" + [source] + module = "origin" + + [source.origin] + url = "https://origin.publisher.example" + [publisher] domain = "publisher.example" cookie_domain = ".publisher.example" - origin_url = "https://origin.publisher.example" proxy_secret = "publisher_proxy_secret" "#, ) From 0d4f41698a88ebacd3673a569d5de9502ff6046e Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 28 Oct 2026 11:30:56 +0000 Subject: [PATCH 17/17] The changelog says only stored pages are fetched again --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b80fb259a..0072b857c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- **Breaking:** Where content comes from is a source module. `[source]` selects the publisher's, and each `[[proxy.asset_routes]]` entry selects its own in a `source` table of the same shape, with `module` naming the source and its settings in the table beneath. Core supplies `origin`, an HTTP origin, and a module a build links can offer another through `IntegrationRegistrationBuilder::with_source_module`. A source answers for each request where that request's content comes from, and core fetches, caches, rewrites the origin host and follows the redirect rule from that answer. `[source]` is required. The origin settings move into the `origin` table, and a document that still writes one where it was is refused at startup naming the new place. An environment variable that set the address is now `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The old name is refused where the build reads variables, and `ts config push` ignores it, as it ignores any variable for a key the document does not hold. A path in an asset route's `url` names a folder the target path is added under, where a path was refused, a target path with a `.` or `..` segment under a folder is answered `400`, and a route applies `directory_index` as the publisher does. Stored pages and the publisher's stored assets are fetched again after the upgrade, because their keys hold a code worked out from the settings, but an asset route's stored copies keep their keys. Where `host_header` is set, the page pipeline and every middleware are now told that `Host` as the page's origin host, where they were told the address's host, so the page's own addresses under that name are rewritten to the request's host. **For crates outside core**, `handle_asset_proxy_request` takes the route's source, which `IntegrationRegistry::route_source` gives, and `Publisher::origin_host`, `origin_host_header` and `origin_query_is_stripped` are replaced by `Settings::origin_source` and the `SourceLocation` a source answers. +- **Breaking:** Where content comes from is a source module. `[source]` selects the publisher's, and each `[[proxy.asset_routes]]` entry selects its own in a `source` table of the same shape, with `module` naming the source and its settings in the table beneath. Core supplies `origin`, an HTTP origin, and a module a build links can offer another through `IntegrationRegistrationBuilder::with_source_module`. A source answers for each request where that request's content comes from, and core fetches, caches, rewrites the origin host and follows the redirect rule from that answer. `[source]` is required. The origin settings move into the `origin` table, and a document that still writes one where it was is refused at startup naming the new place. An environment variable that set the address is now `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The old name is refused where the build reads variables, and `ts config push` ignores it, as it ignores any variable for a key the document does not hold. A path in an asset route's `url` names a folder the target path is added under, where a path was refused, a target path with a `.` or `..` segment under a folder is answered `400`, and a route applies `directory_index` as the publisher does. Stored pages are fetched again after the upgrade, because their keys hold a code worked out from the settings. Where `host_header` is set, the page pipeline and every middleware are now told that `Host` as the page's origin host, where they were told the address's host, so the page's own addresses under that name are rewritten to the request's host. **For crates outside core**, `handle_asset_proxy_request` takes the route's source, which `IntegrationRegistry::route_source` gives, and `Publisher::origin_host`, `origin_host_header` and `origin_query_is_stripped` are replaced by `Settings::origin_source` and the `SourceLocation` a source answers. | Previous | Now | | ----------------------------------------- | ----------------------------------------- |