diff --git a/CHANGELOG.md b/CHANGELOG.md index 177bef1ed3..4fdba05db6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- The Edge Cookie is written without an identity graph as it is with one, because the graph is one optional store of the data kept with the cookie, such as partner IDs. A Fastly service with no `[ec] ec_store` now writes the cookie a real browser's page navigation creates, and `POST /_ts/api/v1/ec/resolve` answers there as it does with a graph, where neither wrote one. So the identifier auction providers get as `user.id`, where storage and personalized-ad selection are both set, is one value for the life of the cookie, where a server-side module such as `hmac` gave a new value on each page view and a client-side module gave none. Naming an `ec_store` later gives each browser holding a cookie a server-side module wrote without one a new identifier on its first page view, because the identity graph replaces an identifier it has no row for. A client-side module creates nothing at the edge to replace its cookie with, so that cookie keeps its identifier with no row for as long as the browser keeps the cookie and its resolved marker, and each of that browser's page views pays for an extra identity-graph read and a strongly consistent existence check. **Breaking for crates outside core:** `ec_finalize_response` and `handle_ec_resolve` read the bot gate from the context's device signals (`EcContext::set_device_signals`), so a context with none gets no created cookie. +- The Axum, Cloudflare and Spin adapters create and write the Edge Cookie and serve `POST /_ts/api/v1/ec/resolve`, where only the Fastly adapter did. A real browser's page navigation is given an Edge Cookie where its permissions allow one, and every route in the request cycle finalizes the Edge Cookie once the response is finalized, which writes a created cookie under the request's public prefix and path and expires one on a withdrawal. Those adapters open no identity graph, so the cookie is written with no graph row, by the resolve route too, the graph's syncs are skipped, and startup logs once that the store `[ec] ec_store` names is not opened. The bot gate there passes any client that sends a browser's `User-Agent`, and where storage and personalized-ad selection are both set a real browser's identifier reaches auction providers as `user.id`, from its first page view with a server-side module and from the requests after its resolve with a client-side one. - The Axum, Cloudflare and Spin adapters run the integration request filters and classify each request with the device module `[device]` selects, as the Fastly adapter does. Before, they ran no filter and asked a selected device module only for a bid request's device attributes. Every named route but the inspection endpoints, and the publisher fallback, reads the request's Edge Cookie context with the device classification and then runs the filters before the route, so a filter that answers, such as DataDome's challenge, stands in for the route, and the response headers a filter names are written after the finalize headers, with the Fastly adapter's privacy guards applied after them. The 51Degrees client hints middleware and DataDome's protection therefore work on every adapter, through HTTP clients that apply no module's own timeout on Axum, Cloudflare and Spin, and DataDome's protection checks `/auction` and `GET` on both page-bids paths as well as the publisher's pages. On those adapters `[device] module = "platform"` classifies from the `User-Agent`, a response core marks private loses any `ETag`, `Last-Modified`, `Expires` or `Age` that `[response_headers]` writes, robots.txt is answered before the Edge Cookie steps and the filters with an empty Edge Cookie context, and on Spin the discovery, signature verification and first-party routes and the page-bids preflight now run the modules' preparers. - **Breaking:** the platform's own device module is selected with `[device] module = "platform"`, as `[geo] module = "platform"` selects the platform's geo lookup, where it was `module = "fastly"`. A document that still writes `fastly` is refused at startup as naming a device module no running module supplies. **For crates outside core**, `build_device_module` takes a `build_platform` factory. - **Breaking for crates outside core:** an adapter declares the request headers its platform sets. `platform::declare_platform_tls_headers` names a header the platform sets on a request that arrived over TLS, which core reads for the scheme and never forwards to an origin. `platform::declare_platform_spoofable_headers` names the platform's other headers a client could also send, which `sanitize_forwarded_headers` strips at entry with the standard forwarding headers, in an adapter that calls it. `platform::declare_platform_internal_headers` names headers that are never passed to a third party. The Fastly adapter declares `fastly-ssl`, `fastly-client-ip` and `x-debug-fastly-pop` before it serves. `SPOOFABLE_FORWARDED_HEADERS` holds the three standard forwarding headers, `HEADER_X_DEBUG_FASTLY_POP` leaves core, and `constants::is_internal_header` answers for the fixed list and the declared headers together. A `[trusted_client_ip]` `ip_header` may start with `x-` or end with `-client-ip`, so `fastly-client-ip` stays valid. On the Axum, Cloudflare and Spin adapters, which declare none of these headers, a client's `fastly-ssl` is no longer removed from the request to the publisher's origin, and on Spin a client's `fastly-ssl` and `fastly-client-ip` are no longer stripped at entry. On those adapters the `x-debug-fastly-pop` header from a client is also no longer kept from third parties, because `INTERNAL_HEADERS` no longer holds it, and is no longer refused as a `[trusted_client_ip]` header name. On the Axum and Cloudflare adapters a client's `fastly-ssl` also no longer makes the request HTTPS. On Fastly, `ts config validate`, which runs with no adapter's declarations, no longer refuses `x-debug-fastly-pop` as a `[trusted_client_ip]` header name, and the service refuses it when it loads the document. diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 639cdfb63d..97ef9f96b0 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -1,5 +1,5 @@ use std::sync::Arc; -use std::sync::atomic::AtomicBool; +use std::sync::atomic::{AtomicBool, Ordering}; use edgezero_core::app::Hooks; use edgezero_core::context::RequestContext; @@ -22,8 +22,10 @@ use trusted_server_core::ec::device::{ BuiltinDeviceModule, DeviceModule, DeviceSignals, build_device_module, }; use trusted_server_core::ec::module::ensure_module_available; +use trusted_server_core::ec::resolve::handle_ec_resolve; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::evidence::BorrowedRequestInfo; +use trusted_server_core::http_util::is_navigation_request; use trusted_server_core::inspect::config::{CONFIG_JSON_PATH, CONFIG_PAGE_PATH, handle_config}; use trusted_server_core::inspect::data::{DATA_PAGE_PATH, handle_data}; use trusted_server_core::inspect::permissions::{ @@ -92,6 +94,32 @@ pub struct AppState { /// Set once startup has logged that this adapter sends no auction telemetry. static UNSENT_ANALYTICS_LOGGED: AtomicBool = AtomicBool::new(false); +/// Set once startup has logged that this adapter opens no identity graph. The +/// state can be built more than once, so the flag keeps the warning to one for +/// each instance. +static IDENTITY_GRAPH_UNAVAILABLE_LOGGED: AtomicBool = AtomicBool::new(false); + +/// Logs once, at warning level, that `[ec] ec_store` names an identity graph +/// this adapter does not open, so the Edge Cookie is still written but the +/// data the graph would keep with it, such as partner IDs, is not. `logged` +/// is set as the line is logged, and once set, nothing more is logged. +/// +/// The identity graph needs strongly consistent reads and conditional writes, +/// which the key-value store this adapter can open does not offer. +fn warn_once_when_the_identity_graph_is_unavailable(settings: &Settings, logged: &AtomicBool) { + let Some(store) = settings.ec.ec_store.as_deref() else { + return; + }; + if logged.swap(true, Ordering::Relaxed) { + return; + } + log::warn!( + "[ec] ec_store names `{store}`, but the Axum adapter opens no identity \ + graph, so the Edge Cookie is still written but the data the graph would \ + keep with it, such as partner IDs, is not" + ); +} + /// The permission signal modules this adapter links, in the order they run /// when configuration names none. Global Privacy Control is first because it /// is a browser setting with no interface of its own, and the three that @@ -300,6 +328,7 @@ fn build_state_with_registrations_and_services( &UNSENT_ANALYTICS_LOGGED, ); warn_when_certificate_check_is_off(&settings); + warn_once_when_the_identity_graph_is_unavailable(&settings, &IDENTITY_GRAPH_UNAVAILABLE_LOGGED); let permission_policy = settings.required_permission_policy()?; Ok(Arc::new(AppState { @@ -511,6 +540,65 @@ async fn build_ec_context( Ok(context) } +/// The request's Edge Cookie state, read before its route runs. +struct EcRequestState { + ec_context: EcContext, + /// Whether the request passed the bot gate, which creating an Edge Cookie + /// on a page navigation sits behind. + is_real_browser: bool, +} + +/// The Edge Cookie state a route leaves on its response. +/// [`FinalizeResponseMiddleware`] takes it off the response and finalizes the +/// Edge Cookie once the response is finalized. +#[derive(Clone)] +pub(crate) struct EcFinalizeState { + pub(crate) ec_context: EcContext, + /// The request's services, which finalizing hands to the selected Edge + /// Cookie module. + pub(crate) services: RuntimeServices, +} + +/// Reads the request's Edge Cookie state, with the bot gate's answer taken +/// from the device classification. +/// +/// # Errors +/// +/// The errors of [`build_ec_context`]. +async fn build_ec_request_state( + state: &AppState, + services: &RuntimeServices, + req: &Request, +) -> Result> { + let ec_context = build_ec_context(state, services, req).await?; + let is_real_browser = ec_context + .device_signals() + .is_some_and(|signals| signals.looks_like_browser); + if !is_real_browser { + log::info!("Bot gate: blocking EC operations"); + } + Ok(EcRequestState { + ec_context, + is_real_browser, + }) +} + +/// Leaves the Edge Cookie state and the non-empty request-filter response +/// effects on a response of the request cycle, for +/// [`FinalizeResponseMiddleware`]. +fn attach_dispatch_extensions( + mut response: Response, + ec: EcRequestState, + services: &RuntimeServices, + effects: RequestFilterEffects, +) -> Response { + response.extensions_mut().insert(EcFinalizeState { + ec_context: ec.ec_context, + services: services.clone(), + }); + attach_request_filter_effects(response, effects) +} + /// Result of the request-filter step that runs before a route. enum PreRoute { /// A filter answered, or failed, and that answer stands in for the @@ -605,21 +693,21 @@ async fn execute_named( _ => {} } - let mut ec_context = match build_ec_context(&state, &services, &req).await { - Ok(context) => context, + let mut ec = match build_ec_request_state(&state, &services, &req).await { + Ok(ec) => ec, Err(report) => return Ok(http_error(&report)), }; - let effects = match run_pre_route_filters(&state, &services, &mut req, &ec_context).await { + let effects = match run_pre_route_filters(&state, &services, &mut req, &ec.ec_context).await { PreRoute::ShortCircuit { response, effects } => { - return Ok(attach_request_filter_effects(response, effects)); + return Ok(attach_dispatch_extensions(response, ec, &services, effects)); } PreRoute::Continue { effects } => effects, }; - let response = run_named_route(&state, &services, req, handler, &mut ec_context) + let response = run_named_route(&state, &services, req, handler, &mut ec.ec_context) .await .unwrap_or_else(|error| http_error(&error)); - Ok(attach_request_filter_effects(response, effects)) + Ok(attach_dispatch_extensions(response, ec, &services, effects)) } async fn run_named_route( @@ -639,6 +727,11 @@ async fn run_named_route( NamedRouteHandler::Permissions | NamedRouteHandler::Config | NamedRouteHandler::Data => { unreachable!("the inspection endpoints are answered before the request cycle") } + NamedRouteHandler::EcResolve => { + // This adapter opens no identity graph, so a created Edge Cookie + // is set with no graph row. + handle_ec_resolve(&state.settings, req, ec_context, None, services).await + } NamedRouteHandler::Auction => { // The auction consent gate reads the caller's jurisdiction from // the geo-aware context, where `EcContext::default()` would fail it @@ -722,13 +815,13 @@ async fn dispatch_fallback( return answer_robots_txt(state, services, req, &method).await; } - let mut ec_context = match build_ec_context(state, services, &req).await { - Ok(context) => context, + let mut ec = match build_ec_request_state(state, services, &req).await { + Ok(ec) => ec, Err(report) => return http_error(&report), }; - let effects = match run_pre_route_filters(state, services, &mut req, &ec_context).await { + let effects = match run_pre_route_filters(state, services, &mut req, &ec.ec_context).await { PreRoute::ShortCircuit { response, effects } => { - return attach_request_filter_effects(response, effects); + return attach_dispatch_extensions(response, ec, services, effects); } PreRoute::Continue { effects } => effects, }; @@ -745,7 +838,7 @@ async fn dispatch_fallback( path: &path, settings: &state.settings, kv: None, - ec_context: &mut ec_context, + ec_context: &mut ec.ec_context, services, req, }) @@ -756,6 +849,17 @@ async fn dispatch_fallback( })) }) } else { + // A real browser's page navigation is given an Edge Cookie when it has + // none. Another request may lack consent signals such as `Sec-GPC`. + if ec.is_real_browser + && is_navigation_request(&req) + && let Err(err) = ec + .ec_context + .generate_if_needed(&state.settings, None, services) + .await + { + log::error!("EC generation failed for publisher proxy: {err:?}"); + } // Run the server-side auction with the configured creative-opportunity // slots; `handle_publisher_request` matches them against the request // path. @@ -771,7 +875,7 @@ async fn dispatch_fallback( }, services, None, - &mut ec_context, + &mut ec.ec_context, auction, req, EDGE_CACHE_HEADER, @@ -796,7 +900,12 @@ async fn dispatch_fallback( } }; - attach_request_filter_effects(result.unwrap_or_else(|error| http_error(&error)), effects) + attach_dispatch_extensions( + result.unwrap_or_else(|error| http_error(&error)), + ec, + services, + effects, + ) } /// Whether the request is a crawler's `GET` or `HEAD` of the robots.txt that @@ -875,6 +984,7 @@ enum NamedRouteHandler { Permissions, Config, Data, + EcResolve, Auction, PageBids, FirstPartyProxy, @@ -889,7 +999,7 @@ struct NamedRoute { handler: NamedRouteHandler, } -fn named_routes() -> [NamedRoute; 14] { +fn named_routes() -> [NamedRoute; 15] { [ NamedRoute { path: "/.well-known/trusted-server.json", @@ -930,6 +1040,11 @@ fn named_routes() -> [NamedRoute; 14] { primary_methods: &[Method::GET], handler: NamedRouteHandler::Data, }, + NamedRoute { + path: "/_ts/api/v1/ec/resolve", + primary_methods: &[Method::POST], + handler: NamedRouteHandler::EcResolve, + }, NamedRoute { path: "/auction", primary_methods: &[Method::POST], diff --git a/crates/trusted-server-adapter-axum/src/app/request_cycle_tests.rs b/crates/trusted-server-adapter-axum/src/app/request_cycle_tests.rs index b09080baf3..5cd104ab5c 100644 --- a/crates/trusted-server-adapter-axum/src/app/request_cycle_tests.rs +++ b/crates/trusted-server-adapter-axum/src/app/request_cycle_tests.rs @@ -3,16 +3,22 @@ //! route in the cycle, and their response effects after the finalize headers, //! without making the response less private, the route's module call carries //! the device classification, an Edge Cookie context that cannot be read -//! answers before the filters, and robots.txt and the inspection endpoints -//! are answered without them. +//! answers before the filters, robots.txt and the inspection endpoints are +//! answered without them, and the Edge Cookie is created and written for a +//! real browser's page view, set by the resolve route for a client-side +//! module and expired on a withdrawal. +use std::net::{IpAddr, Ipv4Addr}; +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; use edgezero_core::body::Body; use edgezero_core::http::{HeaderValue, Method, Request, Response, StatusCode, request_builder}; use edgezero_core::router::RouterService; use error_stack::Report; +use trusted_server_core::addressing::addressed_router; use trusted_server_core::ec::device::{DeviceModule, DeviceSignals}; +use trusted_server_core::ec::module::{EdgeCookieModule, GeneratedEdgeCookie, ModuleCode}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::evidence::RequestInfo; use trusted_server_core::integrations::{ @@ -21,13 +27,18 @@ use trusted_server_core::integrations::{ RequestFilterInput, }; use trusted_server_core::module_context::ModuleCall; -use trusted_server_core::platform::PlatformHttpClient; +use trusted_server_core::permissions::{Permission, PermissionSet}; use trusted_server_core::platform::test_support::{ - StubHttpClient, build_services_with_http_client, + NoopConfigStore, NoopSecretStore, StubBackend, StubHttpClient, build_services_with_http_client, +}; +use trusted_server_core::platform::{ + ClientInfo, GeoInfo, PlatformError, PlatformGeo, PlatformGeoModule, PlatformHttpClient, + RuntimeServices, UnavailableKvStore, }; use trusted_server_core::publisher::{PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH}; use trusted_server_core::response_privacy::enforce_terminal_private_cache_privacy; use trusted_server_core::settings::Settings; +use trusted_server_core::test_support::addressing::FirstSegment; use super::TrustedServerApp; @@ -391,6 +402,17 @@ fn settings_with(extra: &str) -> Settings { .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()) } +/// The router built through the composition root with `builders`, every +/// request handed `services`. +fn router_for( + settings: Settings, + builders: &[IntegrationBuilder], + services: RuntimeServices, +) -> RouterService { + TrustedServerApp::routes_with_registrations_and_services(settings, builders, services) + .expect("should build the router") +} + /// The router built through the composition root with `builders`, every /// request sent to `origin`. fn router_with( @@ -398,12 +420,11 @@ fn router_with( builders: &[IntegrationBuilder], origin: &Arc, ) -> RouterService { - TrustedServerApp::routes_with_registrations_and_services( + router_for( settings, builders, build_services_with_http_client(Arc::clone(origin) as Arc), ) - .expect("should build the router") } fn get(path: &str, cookie: Option<&str>) -> Request { @@ -644,6 +665,410 @@ async fn robots_txt_is_answered_before_the_request_filters() { assert_eq!(origin.recorded_request_methods(), ["GET"]); } +// --------------------------------------------------------------------------- +// The Edge Cookie +// --------------------------------------------------------------------------- + +/// The `User-Agent` of a desktop browser, which the bot gate passes. +const BROWSER: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"; + +/// The `User-Agent` of a crawler, which the bot gate stops. +const CRAWLER: &str = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"; + +/// A page the publisher's origin answers. +const PAGE: &[u8] = b"article"; + +/// An Edge Cookie identifier in the shape the `hmac` module gives one. +const CARRIED_EC: &str = + "hmac~0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef.abc123"; + +/// A TC String that consents to no purpose, so it refuses storage, Purpose 1. +/// It is version 2 from CMP 1, in English, for vendor list 1 and no vendor, +/// last updated at time zero, so the settings that read it turn the TCF +/// module's expiry check off. +const STORAGE_REFUSING_TC_STRING: &str = "CAAAAAAAAAAAAABABAENABCAAAAAAAAAAAhoAAA"; + +/// A location in the US, where the sample permission rules grant the Edge +/// Cookie with no signal. +struct UsGeo; + +#[async_trait::async_trait(?Send)] +impl PlatformGeo for UsGeo { + async fn lookup( + &self, + _client_ip: Option, + _services: &RuntimeServices, + ) -> Result, Report> { + Ok(Some(GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: String::new(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + })) + } +} + +/// The services of a reader in the US with an address, carrying the +/// permission signal modules this adapter links, every request sent to +/// `origin`. +fn services_for_a_reader(origin: &Arc) -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(UnavailableKvStore)) + .backend(Arc::new(StubBackend)) + .http_client(Arc::clone(origin) as Arc) + .geo(Arc::new(PlatformGeoModule::new( + "platform", + Arc::new(UsGeo), + ))) + .client_info(ClientInfo { + client_ip: Some(IpAddr::V4(Ipv4Addr::new(192, 0, 2, 10))), + ..ClientInfo::default() + }) + .permission_signal_modules(super::shipped_signal_modules().into()) + .permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()) + .build() +} + +/// Settings whose `[geo] module = "platform"` leaves the location the +/// reader's services give standing, where an unset selector resolves the +/// disabled module. +fn settings_for_a_reader() -> Settings { + let mut settings = settings_with(""); + settings.geo.module = Some("platform".to_owned()); + settings +} + +/// An origin answering each of `pages` requests with [`PAGE`]. +fn origin_answering(pages: usize) -> Arc { + let origin = Arc::new(StubHttpClient::new()); + for _ in 0..pages { + origin.push_response_with_headers( + 200, + PAGE.to_vec(), + vec![("content-type", "text/html; charset=utf-8")], + ); + } + origin +} + +/// A page navigation to `path` by a client with `user_agent`, carrying +/// `cookie` when there is one. +fn page_view(path: &str, user_agent: &str, cookie: Option<&str>) -> Request { + let mut builder = request_builder() + .method("GET") + .uri(format!("https://test-publisher.example.com{path}")) + .header("host", "test-publisher.example.com") + .header("user-agent", user_agent) + .header("accept", "text/html") + .header("sec-fetch-dest", "document") + .header("sec-fetch-mode", "navigate"); + if let Some(cookie) = cookie { + builder = builder.header("cookie", cookie); + } + builder + .body(Body::empty()) + .expect("should build the page view") +} + +/// The `Set-Cookie` line that writes the Edge Cookie, if the response has one. +/// A response writes it at most once, because core alone writes it. +fn edge_cookie(response: &Response) -> Option { + let lines: Vec = response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .filter(|line| line.starts_with("ts-ec=")) + .map(str::to_owned) + .collect(); + assert!( + lines.len() <= 1, + "the Edge Cookie should be written at most once: {lines:?}" + ); + lines.into_iter().next() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_first_page_view_gets_the_edge_cookie_under_the_public_prefix_and_path() { + let origin = origin_answering(1); + let router = addressed_router( + router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)), + Arc::new(FirstSegment), + ); + + let response = route( + &router, + page_view("/publisher.example/article", BROWSER, None), + ) + .await; + + assert_eq!(response.status(), StatusCode::OK); + let cookie = edge_cookie(&response).expect("a first page view should get the Edge Cookie"); + assert!( + cookie.contains("Path=/publisher.example"), + "the Edge Cookie should be written under the publisher's path: {cookie}" + ); + assert_eq!( + origin.recorded_request_uris(), + ["https://origin.test-publisher.example.com/article"], + "the page should be asked for at the path the addressing continues with" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_returning_page_view_keeps_its_edge_cookie() { + let origin = origin_answering(2); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + + let first = route(&router, page_view("/article", BROWSER, None)).await; + let written = edge_cookie(&first).expect("the first page view should get the Edge Cookie"); + assert!( + written.contains("Path=/;") || written.ends_with("Path=/"), + "a request addressed by host writes the Edge Cookie under the root: {written}" + ); + let pair = written + .split(';') + .next() + .expect("a Set-Cookie line starts with its pair") + .to_owned(); + + let returning = route(&router, page_view("/article", BROWSER, Some(&pair))).await; + + assert_eq!(returning.status(), StatusCode::OK); + assert_eq!( + edge_cookie(&returning), + None, + "a returning page view should keep the Edge Cookie it carries" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_bot_s_page_view_gets_no_edge_cookie() { + let origin = origin_answering(1); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + + let response = route(&router, page_view("/article", CRAWLER, None)).await; + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + edge_cookie(&response), + None, + "the bot gate should keep a crawler from getting an Edge Cookie" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_withdrawal_expires_the_edge_cookie_the_browser_carries() { + // No `[geo]` module, so the reader resolves at the top of the sample + // rules, the `gdpr-eu` group, where storage needs a signal, so a TC String + // refusing storage withdraws it. + let origin = origin_answering(1); + let router = router_for( + settings_with("[consent]\ncheck_expiration = false"), + &[], + services_for_a_reader(&origin), + ); + let cookies = format!("ts-ec={CARRIED_EC}; euconsent-v2={STORAGE_REFUSING_TC_STRING}"); + + let response = route(&router, page_view("/article", BROWSER, Some(&cookies))).await; + + assert_eq!(response.status(), StatusCode::OK); + let expired = + edge_cookie(&response).expect("a withdrawal should expire the Edge Cookie it carries"); + assert!( + expired.starts_with("ts-ec=;") && expired.contains("Max-Age=0"), + "the Edge Cookie should be expired: {expired}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_resolve_route_answers() { + let origin = Arc::new(StubHttpClient::new()); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + let resolve = |page_origin: &str| { + request_builder() + .method("POST") + .uri("https://test-publisher.example.com/_ts/api/v1/ec/resolve") + .header("host", "test-publisher.example.com") + .header("origin", page_origin) + .header("content-type", "text/plain") + .header("user-agent", BROWSER) + .body(Body::from("posted value")) + .expect("should build the resolve request") + }; + + let foreign = route(&router, resolve("https://elsewhere.example")).await; + assert_eq!( + foreign.status(), + StatusCode::FORBIDDEN, + "a foreign Origin should be refused" + ); + assert_eq!(header(&foreign, "cache-control"), Some("no-store")); + + let own = route(&router, resolve("https://test-publisher.example.com")).await; + assert_eq!( + own.status(), + StatusCode::NO_CONTENT, + "a server-side module creates nothing from a posted value" + ); + assert_eq!(edge_cookie(&own), None); + + assert!( + origin.recorded_request_uris().is_empty(), + "the resolve route should be answered here, never by the origin" + ); +} + +/// The id and module name of an Edge Cookie module that creates nothing at +/// the edge and takes its identifier from the value the page posts, as a +/// client-side module does. +const POSTED_VALUE_ID: &str = "posted_value"; +const POSTED_VALUE_MODULE: &str = "testing.posted-value"; + +/// The value the page posts, which the module takes as its identifier. +const POSTED_VALUE: &str = "a-posted-value"; + +#[derive(Debug)] +struct PostedValueModule; + +#[async_trait::async_trait(?Send)] +impl EdgeCookieModule for PostedValueModule { + fn id(&self) -> &'static str { + POSTED_VALUE_MODULE + } + + fn code(&self) -> ModuleCode { + trusted_server_core::module_code!("t0pv") + } + + async fn generate( + &self, + _call: ModuleCall<'_>, + ) -> Result> { + Ok(GeneratedEdgeCookie::default()) + } + + async fn resolve_from_client( + &self, + _call: ModuleCall<'_>, + payload: &[u8], + ) -> Result> { + Ok(GeneratedEdgeCookie { + id: (payload == POSTED_VALUE.as_bytes()).then(|| POSTED_VALUE.to_owned()), + response_headers: Vec::new(), + }) + } + + fn accepts_id(&self, value: &str) -> bool { + value == POSTED_VALUE + } + + fn required_permissions(&self) -> PermissionSet { + PermissionSet::none().with(Permission::StoreOnDevice) + } +} + +fn register_posted_value( + _settings: &Settings, +) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(POSTED_VALUE_ID) + .without_js() + .with_ec_module(POSTED_VALUE_MODULE, Arc::new(PostedValueModule)) + .build(), + )) +} + +fn posted_value_builder() -> IntegrationBuilder { + IntegrationBuilder::new( + POSTED_VALUE_ID, + "adapter-tests", + register_posted_value, + always, + ) + .with_module_name(POSTED_VALUE_MODULE) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_resolve_route_sets_a_client_side_module_s_edge_cookie() { + let mut settings = settings_for_a_reader(); + settings.select_module("testing", POSTED_VALUE_MODULE); + settings.ec.module = Some(POSTED_VALUE_MODULE.into()); + settings.ec.module_blocks.clear(); + let origin = Arc::new(StubHttpClient::new()); + let router = router_for( + settings, + &[posted_value_builder()], + services_for_a_reader(&origin), + ); + let resolve = request_builder() + .method("POST") + .uri("https://test-publisher.example.com/_ts/api/v1/ec/resolve") + .header("host", "test-publisher.example.com") + .header("origin", "https://test-publisher.example.com") + .header("content-type", "text/plain") + .header("user-agent", BROWSER) + .body(Body::from(POSTED_VALUE)) + .expect("should build the resolve request"); + + let response = route(&router, resolve).await; + + assert_eq!( + response.status(), + StatusCode::OK, + "the module's identifier should be set with no identity graph" + ); + assert_eq!(header(&response, "cache-control"), Some("no-store")); + let cookie = edge_cookie(&response).expect("the resolve route should set the Edge Cookie"); + assert!( + cookie.starts_with("ts-ec=t0pv~a-posted-value;"), + "the Edge Cookie should carry the module's coded identifier: {cookie}" + ); + assert!( + response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .any(|line| line.starts_with("ts-ecr=1;")), + "the resolved marker should be set beside the Edge Cookie" + ); + assert!( + origin.recorded_request_uris().is_empty(), + "the resolve route should be answered here, never by the origin" + ); +} + +#[test] +fn startup_warns_that_this_adapter_opens_no_identity_graph() { + // The flag is set as the warning is logged, so a deployment whose + // `[ec] ec_store` names no store logs nothing. + let logged = AtomicBool::new(false); + super::warn_once_when_the_identity_graph_is_unavailable(&settings_for_a_reader(), &logged); + assert!( + !logged.load(Ordering::Relaxed), + "with no ec_store there is no identity graph to warn about" + ); + + let mut settings = settings_for_a_reader(); + settings.ec.ec_store = Some("ec_identity_store".to_owned()); + let origin = Arc::new(StubHttpClient::new()); + + let _router = router_for(settings, &[], services_for_a_reader(&origin)); + + assert!( + super::IDENTITY_GRAPH_UNAVAILABLE_LOGGED.load(Ordering::Relaxed), + "startup should warn that the identity graph is unavailable" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn robots_txt_shows_the_device_module_none_of_the_crawler_s_cookies() { let origin = Arc::new(StubHttpClient::new()); diff --git a/crates/trusted-server-adapter-axum/src/middleware.rs b/crates/trusted-server-adapter-axum/src/middleware.rs index 3eb75b8dbe..e9b5a17879 100644 --- a/crates/trusted-server-adapter-axum/src/middleware.rs +++ b/crates/trusted-server-adapter-axum/src/middleware.rs @@ -5,6 +5,8 @@ use edgezero_core::context::RequestContext; use edgezero_core::error::EdgeError; use edgezero_core::http::Response; use edgezero_core::middleware::{Middleware, Next}; +use trusted_server_core::ec::finalize::ec_finalize_response; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::http_util::{ ROUTE_NAMESPACE, sanitize_trusted_client_ip_headers, strip_deployment_headers, }; @@ -15,6 +17,8 @@ use trusted_server_core::response_privacy::{ }; use trusted_server_core::settings::Settings; +use crate::app::EcFinalizeState; + // --------------------------------------------------------------------------- // SanitizeRequestMiddleware // --------------------------------------------------------------------------- @@ -62,7 +66,8 @@ impl Middleware for SanitizeRequestMiddleware { /// /// Registered directly inside [`SanitizeRequestMiddleware`], so that every /// outgoing response carries a consistent set of headers. Once the response -/// is finalized it applies the response effects the request filters left. +/// is finalized it finalizes the Edge Cookie and then applies the response +/// effects the request filters left. pub struct FinalizeResponseMiddleware { settings: Arc, } @@ -80,15 +85,49 @@ impl Middleware for FinalizeResponseMiddleware { async fn handle(&self, ctx: RequestContext, next: Next<'_>) -> Result { let path = ctx.request().uri().path().to_owned(); let mut response = next.run(ctx).await?; + let ec_state = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); // Before finalizing, so a header finalizing writes is kept. strip_deployment_headers(&path, ROUTE_NAMESPACE, &mut response); apply_finalize_headers(&self.settings, &mut response); + if let Some(mut ec_state) = ec_state { + finalize_edge_cookie(&self.settings, &mut ec_state, &mut response).await; + } apply_terminal_response_effects(&mut response, request_filter_effects.as_ref()); Ok(response) } } +/// Finalizes the Edge Cookie on a response with no identity graph, because +/// this adapter opens none. +/// +/// Core writes a created Edge Cookie without a graph as it does with one, for +/// a request that passed the bot gate and whose permissions allow it. A +/// partner registry that cannot be built skips finalizing. +async fn finalize_edge_cookie( + settings: &Settings, + ec_state: &mut EcFinalizeState, + response: &mut Response, +) { + let partner_registry = match PartnerRegistry::from_config(&settings.ec.partners) { + Ok(registry) => registry, + Err(report) => { + log::error!("EC finalize skipped: failed to build partner registry: {report:?}"); + return; + } + }; + ec_finalize_response( + settings, + &mut ec_state.ec_context, + None, + &partner_registry, + None, + response, + &ec_state.services, + ) + .await; +} + /// Applies the request filters' response effects to a finalized response, /// then restores the privacy guards a late header could loosen. fn apply_terminal_response_effects( diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index 130ea92e90..8290ac5207 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -1,7 +1,7 @@ use core::future::Future; use core::pin::Pin; use std::sync::Arc; -use std::sync::atomic::AtomicBool; +use std::sync::atomic::{AtomicBool, Ordering}; use edgezero_core::app::{Hooks, StoreMetadata, StoresMetadata}; use edgezero_core::context::RequestContext; @@ -26,8 +26,10 @@ use trusted_server_core::ec::device::{ BuiltinDeviceModule, DeviceModule, DeviceSignals, build_device_module, }; use trusted_server_core::ec::module::{EdgeCookieModule, build_reusable_module}; +use trusted_server_core::ec::resolve::handle_ec_resolve; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::evidence::BorrowedRequestInfo; +use trusted_server_core::http_util::is_navigation_request; use trusted_server_core::inspect::config::{CONFIG_PATHS, handle_config}; use trusted_server_core::inspect::data::{DATA_PAGE_PATH, handle_data}; use trusted_server_core::inspect::permissions::{PERMISSIONS_PATHS, handle_permissions}; @@ -288,6 +290,7 @@ fn build_state_with_registrations_and_services( "Cloudflare", &UNSENT_ANALYTICS_LOGGED, ); + warn_once_when_the_identity_graph_is_unavailable(&settings, &IDENTITY_GRAPH_UNAVAILABLE_LOGGED); let permission_policy = settings.required_permission_policy()?; Ok(Arc::new(AppState { @@ -307,6 +310,32 @@ fn build_state_with_registrations_and_services( /// one for each instance. static UNSENT_ANALYTICS_LOGGED: AtomicBool = AtomicBool::new(false); +/// Set once startup has logged that this adapter opens no identity graph. The +/// state can be built more than once, so the flag keeps the warning to one for +/// each instance. +static IDENTITY_GRAPH_UNAVAILABLE_LOGGED: AtomicBool = AtomicBool::new(false); + +/// Logs once, at warning level, that `[ec] ec_store` names an identity graph +/// this adapter does not open, so the Edge Cookie is still written but the +/// data the graph would keep with it, such as partner IDs, is not. `logged` +/// is set as the line is logged, and once set, nothing more is logged. +/// +/// The identity graph needs strongly consistent reads and conditional writes, +/// which the key-value store this adapter can open does not offer. +fn warn_once_when_the_identity_graph_is_unavailable(settings: &Settings, logged: &AtomicBool) { + let Some(store) = settings.ec.ec_store.as_deref() else { + return; + }; + if logged.swap(true, Ordering::Relaxed) { + return; + } + log::warn!( + "[ec] ec_store names `{store}`, but the Cloudflare adapter opens no identity \ + graph, so the Edge Cookie is still written but the data the graph would \ + keep with it, such as partner IDs, is not" + ); +} + /// The permission signal modules this adapter links, in the order they run /// when configuration names none. Global Privacy Control is first because it /// is a browser setting with no interface of its own, and the three that @@ -526,6 +555,65 @@ async fn build_ec_context( Ok(context) } +/// The request's Edge Cookie state, read before its route runs. +struct EcRequestState { + ec_context: EcContext, + /// Whether the request passed the bot gate, which creating an Edge Cookie + /// on a page navigation sits behind. + is_real_browser: bool, +} + +/// The Edge Cookie state a route leaves on its response. +/// [`FinalizeResponseMiddleware`] takes it off the response and finalizes the +/// Edge Cookie once the response is finalized. +#[derive(Clone)] +pub(crate) struct EcFinalizeState { + pub(crate) ec_context: EcContext, + /// The request's services, which finalizing hands to the selected Edge + /// Cookie module. + pub(crate) services: RuntimeServices, +} + +/// Reads the request's Edge Cookie state, with the bot gate's answer taken +/// from the device classification. +/// +/// # Errors +/// +/// The errors of [`build_ec_context`]. +async fn build_ec_request_state( + settings: &Settings, + services: &RuntimeServices, + req: &Request, +) -> Result> { + let ec_context = build_ec_context(settings, services, req).await?; + let is_real_browser = ec_context + .device_signals() + .is_some_and(|signals| signals.looks_like_browser); + if !is_real_browser { + log::info!("Bot gate: blocking EC operations"); + } + Ok(EcRequestState { + ec_context, + is_real_browser, + }) +} + +/// Leaves the Edge Cookie state and the non-empty request-filter response +/// effects on a response of the request cycle, for +/// [`FinalizeResponseMiddleware`]. +fn attach_dispatch_extensions( + mut response: Response, + ec: EcRequestState, + services: &RuntimeServices, + effects: RequestFilterEffects, +) -> Response { + response.extensions_mut().insert(EcFinalizeState { + ec_context: ec.ec_context, + services: services.clone(), + }); + attach_request_filter_effects(response, effects) +} + /// Result of the request-filter step that runs before a route. enum PreRoute { /// A filter answered, or failed, and that answer stands in for the @@ -633,6 +721,7 @@ where enum NamedRouteHandler { TrustedServerDiscovery, VerifySignature, + EcResolve, Auction, PageBids, FirstPartyProxy, @@ -667,21 +756,21 @@ async fn execute_named( return Ok(http_error(&error)); } - let mut ec_context = match build_ec_context(&state.settings, &services, &req).await { - Ok(context) => context, + let mut ec = match build_ec_request_state(&state.settings, &services, &req).await { + Ok(ec) => ec, Err(report) => return Ok(http_error(&report)), }; - let effects = match run_pre_route_filters(&state, &services, &mut req, &ec_context).await { + let effects = match run_pre_route_filters(&state, &services, &mut req, &ec.ec_context).await { PreRoute::ShortCircuit { response, effects } => { - return Ok(attach_request_filter_effects(response, effects)); + return Ok(attach_dispatch_extensions(response, ec, &services, effects)); } PreRoute::Continue { effects } => effects, }; - let response = run_named_route(&state, &services, req, handler, &mut ec_context) + let response = run_named_route(&state, &services, req, handler, &mut ec.ec_context) .await .unwrap_or_else(|error| http_error(&error)); - Ok(attach_request_filter_effects(response, effects)) + Ok(attach_dispatch_extensions(response, ec, &services, effects)) } async fn run_named_route( @@ -698,6 +787,11 @@ async fn run_named_route( NamedRouteHandler::VerifySignature => { handle_verify_signature(&state.settings, services, req) } + NamedRouteHandler::EcResolve => { + // This adapter opens no identity graph, so a created Edge Cookie + // is set with no graph row. + handle_ec_resolve(&state.settings, req, ec_context, None, services).await + } NamedRouteHandler::Auction => { // The auction consent gate reads the caller's jurisdiction from // the geo-aware context, where `EcContext::default()` would fail it @@ -785,13 +879,13 @@ async fn dispatch_fallback( // Identity could not be established (for example the selected Edge // Cookie module is unavailable). Answer with an error rather than // serving the request with no identity. - let mut ec_context = match build_ec_context(&state.settings, services, &req).await { - Ok(context) => context, + let mut ec = match build_ec_request_state(&state.settings, services, &req).await { + Ok(ec) => ec, Err(report) => return http_error(&report), }; - let effects = match run_pre_route_filters(state, services, &mut req, &ec_context).await { + let effects = match run_pre_route_filters(state, services, &mut req, &ec.ec_context).await { PreRoute::ShortCircuit { response, effects } => { - return attach_request_filter_effects(response, effects); + return attach_dispatch_extensions(response, ec, services, effects); } PreRoute::Continue { effects } => effects, }; @@ -809,7 +903,7 @@ async fn dispatch_fallback( path: &path, settings: &state.settings, kv: None, - ec_context: &mut ec_context, + ec_context: &mut ec.ec_context, services, req, }) @@ -820,6 +914,17 @@ async fn dispatch_fallback( })) }) } else { + // A real browser's page navigation is given an Edge Cookie when it has + // none. Another request may lack consent signals such as `Sec-GPC`. + if ec.is_real_browser + && is_navigation_request(&req) + && let Err(err) = ec + .ec_context + .generate_if_needed(&state.settings, None, services) + .await + { + log::error!("EC generation failed for publisher proxy: {err:?}"); + } let auction = AuctionDispatch { orchestrator: &state.orchestrator, slots: state.settings.creative_opportunity_slots(), @@ -832,7 +937,7 @@ async fn dispatch_fallback( }, services, None, - &mut ec_context, + &mut ec.ec_context, auction, req, EDGE_CACHE_HEADER, @@ -854,7 +959,12 @@ async fn dispatch_fallback( } }; - attach_request_filter_effects(result.unwrap_or_else(|error| http_error(&error)), effects) + attach_dispatch_extensions( + result.unwrap_or_else(|error| http_error(&error)), + ec, + services, + effects, + ) } /// Whether the request is a crawler's `GET` or `HEAD` of the robots.txt that @@ -1174,6 +1284,10 @@ fn build_router(state: &Arc) -> RouterService { "/verify-signature", named(NamedRouteHandler::VerifySignature), ) + .post( + "/_ts/api/v1/ec/resolve", + named(NamedRouteHandler::EcResolve), + ) .post("/auction", named(NamedRouteHandler::Auction)) .get( "/first-party/proxy", diff --git a/crates/trusted-server-adapter-cloudflare/src/app/request_cycle_tests.rs b/crates/trusted-server-adapter-cloudflare/src/app/request_cycle_tests.rs index 7eb8a9c850..9c94d57873 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app/request_cycle_tests.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app/request_cycle_tests.rs @@ -3,16 +3,22 @@ //! route in the cycle, and their response effects after the finalize headers, //! without making the response less private, the route's module call carries //! the device classification, an Edge Cookie context that cannot be read -//! answers before the filters, and robots.txt and the inspection endpoints -//! are answered without them. +//! answers before the filters, robots.txt and the inspection endpoints are +//! answered without them, and the Edge Cookie is created and written for a +//! real browser's page view, set by the resolve route for a client-side +//! module and expired on a withdrawal. +use std::net::{IpAddr, Ipv4Addr}; +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; use edgezero_core::body::Body; use edgezero_core::http::{HeaderValue, Method, Request, Response, StatusCode, request_builder}; use edgezero_core::router::RouterService; use error_stack::Report; +use trusted_server_core::addressing::addressed_router; use trusted_server_core::ec::device::{DeviceModule, DeviceSignals}; +use trusted_server_core::ec::module::{EdgeCookieModule, GeneratedEdgeCookie, ModuleCode}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::evidence::RequestInfo; use trusted_server_core::integrations::{ @@ -21,13 +27,18 @@ use trusted_server_core::integrations::{ RequestFilterInput, }; use trusted_server_core::module_context::ModuleCall; -use trusted_server_core::platform::PlatformHttpClient; +use trusted_server_core::permissions::{Permission, PermissionSet}; use trusted_server_core::platform::test_support::{ - StubHttpClient, build_services_with_http_client, + NoopConfigStore, NoopSecretStore, StubBackend, StubHttpClient, build_services_with_http_client, +}; +use trusted_server_core::platform::{ + ClientInfo, GeoInfo, PlatformError, PlatformGeo, PlatformGeoModule, PlatformHttpClient, + RuntimeServices, UnavailableKvStore, }; use trusted_server_core::publisher::{PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH}; use trusted_server_core::response_privacy::enforce_terminal_private_cache_privacy; use trusted_server_core::settings::Settings; +use trusted_server_core::test_support::addressing::FirstSegment; use super::{build_router, build_state_with_registrations_and_services}; @@ -391,6 +402,18 @@ fn settings_with(extra: &str) -> Settings { .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()) } +/// The router built through the composition root with `builders`, every +/// request handed `services`. +fn router_for( + settings: Settings, + builders: &[IntegrationBuilder], + services: RuntimeServices, +) -> RouterService { + let state = build_state_with_registrations_and_services(settings, builders, Some(services)) + .expect("should build the state"); + build_router(&state) +} + /// The router built through the composition root with `builders`, every /// request sent to `origin`. fn router_with( @@ -398,15 +421,11 @@ fn router_with( builders: &[IntegrationBuilder], origin: &Arc, ) -> RouterService { - let state = build_state_with_registrations_and_services( + router_for( settings, builders, - Some(build_services_with_http_client( - Arc::clone(origin) as Arc - )), + build_services_with_http_client(Arc::clone(origin) as Arc), ) - .expect("should build the state"); - build_router(&state) } fn get(path: &str, cookie: Option<&str>) -> Request { @@ -647,6 +666,410 @@ async fn robots_txt_is_answered_before_the_request_filters() { assert_eq!(origin.recorded_request_methods(), ["GET"]); } +// --------------------------------------------------------------------------- +// The Edge Cookie +// --------------------------------------------------------------------------- + +/// The `User-Agent` of a desktop browser, which the bot gate passes. +const BROWSER: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"; + +/// The `User-Agent` of a crawler, which the bot gate stops. +const CRAWLER: &str = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"; + +/// A page the publisher's origin answers. +const PAGE: &[u8] = b"article"; + +/// An Edge Cookie identifier in the shape the `hmac` module gives one. +const CARRIED_EC: &str = + "hmac~0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef.abc123"; + +/// A TC String that consents to no purpose, so it refuses storage, Purpose 1. +/// It is version 2 from CMP 1, in English, for vendor list 1 and no vendor, +/// last updated at time zero, so the settings that read it turn the TCF +/// module's expiry check off. +const STORAGE_REFUSING_TC_STRING: &str = "CAAAAAAAAAAAAABABAENABCAAAAAAAAAAAhoAAA"; + +/// A location in the US, where the sample permission rules grant the Edge +/// Cookie with no signal. +struct UsGeo; + +#[async_trait::async_trait(?Send)] +impl PlatformGeo for UsGeo { + async fn lookup( + &self, + _client_ip: Option, + _services: &RuntimeServices, + ) -> Result, Report> { + Ok(Some(GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: String::new(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + })) + } +} + +/// The services of a reader in the US with an address, carrying the +/// permission signal modules this adapter links, every request sent to +/// `origin`. +fn services_for_a_reader(origin: &Arc) -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(UnavailableKvStore)) + .backend(Arc::new(StubBackend)) + .http_client(Arc::clone(origin) as Arc) + .geo(Arc::new(PlatformGeoModule::new( + "platform", + Arc::new(UsGeo), + ))) + .client_info(ClientInfo { + client_ip: Some(IpAddr::V4(Ipv4Addr::new(192, 0, 2, 10))), + ..ClientInfo::default() + }) + .permission_signal_modules(super::shipped_signal_modules().into()) + .permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()) + .build() +} + +/// Settings whose `[geo] module = "platform"` leaves the location the +/// reader's services give standing, where an unset selector resolves the +/// disabled module. +fn settings_for_a_reader() -> Settings { + let mut settings = settings_with(""); + settings.geo.module = Some("platform".to_owned()); + settings +} + +/// An origin answering each of `pages` requests with [`PAGE`]. +fn origin_answering(pages: usize) -> Arc { + let origin = Arc::new(StubHttpClient::new()); + for _ in 0..pages { + origin.push_response_with_headers( + 200, + PAGE.to_vec(), + vec![("content-type", "text/html; charset=utf-8")], + ); + } + origin +} + +/// A page navigation to `path` by a client with `user_agent`, carrying +/// `cookie` when there is one. +fn page_view(path: &str, user_agent: &str, cookie: Option<&str>) -> Request { + let mut builder = request_builder() + .method("GET") + .uri(format!("https://test-publisher.example.com{path}")) + .header("host", "test-publisher.example.com") + .header("user-agent", user_agent) + .header("accept", "text/html") + .header("sec-fetch-dest", "document") + .header("sec-fetch-mode", "navigate"); + if let Some(cookie) = cookie { + builder = builder.header("cookie", cookie); + } + builder + .body(Body::empty()) + .expect("should build the page view") +} + +/// The `Set-Cookie` line that writes the Edge Cookie, if the response has one. +/// A response writes it at most once, because core alone writes it. +fn edge_cookie(response: &Response) -> Option { + let lines: Vec = response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .filter(|line| line.starts_with("ts-ec=")) + .map(str::to_owned) + .collect(); + assert!( + lines.len() <= 1, + "the Edge Cookie should be written at most once: {lines:?}" + ); + lines.into_iter().next() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_first_page_view_gets_the_edge_cookie_under_the_public_prefix_and_path() { + let origin = origin_answering(1); + let router = addressed_router( + router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)), + Arc::new(FirstSegment), + ); + + let response = route( + &router, + page_view("/publisher.example/article", BROWSER, None), + ) + .await; + + assert_eq!(response.status(), StatusCode::OK); + let cookie = edge_cookie(&response).expect("a first page view should get the Edge Cookie"); + assert!( + cookie.contains("Path=/publisher.example"), + "the Edge Cookie should be written under the publisher's path: {cookie}" + ); + assert_eq!( + origin.recorded_request_uris(), + ["https://origin.test-publisher.example.com/article"], + "the page should be asked for at the path the addressing continues with" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_returning_page_view_keeps_its_edge_cookie() { + let origin = origin_answering(2); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + + let first = route(&router, page_view("/article", BROWSER, None)).await; + let written = edge_cookie(&first).expect("the first page view should get the Edge Cookie"); + assert!( + written.contains("Path=/;") || written.ends_with("Path=/"), + "a request addressed by host writes the Edge Cookie under the root: {written}" + ); + let pair = written + .split(';') + .next() + .expect("a Set-Cookie line starts with its pair") + .to_owned(); + + let returning = route(&router, page_view("/article", BROWSER, Some(&pair))).await; + + assert_eq!(returning.status(), StatusCode::OK); + assert_eq!( + edge_cookie(&returning), + None, + "a returning page view should keep the Edge Cookie it carries" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_bot_s_page_view_gets_no_edge_cookie() { + let origin = origin_answering(1); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + + let response = route(&router, page_view("/article", CRAWLER, None)).await; + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + edge_cookie(&response), + None, + "the bot gate should keep a crawler from getting an Edge Cookie" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_withdrawal_expires_the_edge_cookie_the_browser_carries() { + // No `[geo]` module, so the reader resolves at the top of the sample + // rules, the `gdpr-eu` group, where storage needs a signal, so a TC String + // refusing storage withdraws it. + let origin = origin_answering(1); + let router = router_for( + settings_with("[consent]\ncheck_expiration = false"), + &[], + services_for_a_reader(&origin), + ); + let cookies = format!("ts-ec={CARRIED_EC}; euconsent-v2={STORAGE_REFUSING_TC_STRING}"); + + let response = route(&router, page_view("/article", BROWSER, Some(&cookies))).await; + + assert_eq!(response.status(), StatusCode::OK); + let expired = + edge_cookie(&response).expect("a withdrawal should expire the Edge Cookie it carries"); + assert!( + expired.starts_with("ts-ec=;") && expired.contains("Max-Age=0"), + "the Edge Cookie should be expired: {expired}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_resolve_route_answers() { + let origin = Arc::new(StubHttpClient::new()); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + let resolve = |page_origin: &str| { + request_builder() + .method("POST") + .uri("https://test-publisher.example.com/_ts/api/v1/ec/resolve") + .header("host", "test-publisher.example.com") + .header("origin", page_origin) + .header("content-type", "text/plain") + .header("user-agent", BROWSER) + .body(Body::from("posted value")) + .expect("should build the resolve request") + }; + + let foreign = route(&router, resolve("https://elsewhere.example")).await; + assert_eq!( + foreign.status(), + StatusCode::FORBIDDEN, + "a foreign Origin should be refused" + ); + assert_eq!(header(&foreign, "cache-control"), Some("no-store")); + + let own = route(&router, resolve("https://test-publisher.example.com")).await; + assert_eq!( + own.status(), + StatusCode::NO_CONTENT, + "a server-side module creates nothing from a posted value" + ); + assert_eq!(edge_cookie(&own), None); + + assert!( + origin.recorded_request_uris().is_empty(), + "the resolve route should be answered here, never by the origin" + ); +} + +/// The id and module name of an Edge Cookie module that creates nothing at +/// the edge and takes its identifier from the value the page posts, as a +/// client-side module does. +const POSTED_VALUE_ID: &str = "posted_value"; +const POSTED_VALUE_MODULE: &str = "testing.posted-value"; + +/// The value the page posts, which the module takes as its identifier. +const POSTED_VALUE: &str = "a-posted-value"; + +#[derive(Debug)] +struct PostedValueModule; + +#[async_trait::async_trait(?Send)] +impl EdgeCookieModule for PostedValueModule { + fn id(&self) -> &'static str { + POSTED_VALUE_MODULE + } + + fn code(&self) -> ModuleCode { + trusted_server_core::module_code!("t0pv") + } + + async fn generate( + &self, + _call: ModuleCall<'_>, + ) -> Result> { + Ok(GeneratedEdgeCookie::default()) + } + + async fn resolve_from_client( + &self, + _call: ModuleCall<'_>, + payload: &[u8], + ) -> Result> { + Ok(GeneratedEdgeCookie { + id: (payload == POSTED_VALUE.as_bytes()).then(|| POSTED_VALUE.to_owned()), + response_headers: Vec::new(), + }) + } + + fn accepts_id(&self, value: &str) -> bool { + value == POSTED_VALUE + } + + fn required_permissions(&self) -> PermissionSet { + PermissionSet::none().with(Permission::StoreOnDevice) + } +} + +fn register_posted_value( + _settings: &Settings, +) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(POSTED_VALUE_ID) + .without_js() + .with_ec_module(POSTED_VALUE_MODULE, Arc::new(PostedValueModule)) + .build(), + )) +} + +fn posted_value_builder() -> IntegrationBuilder { + IntegrationBuilder::new( + POSTED_VALUE_ID, + "adapter-tests", + register_posted_value, + always, + ) + .with_module_name(POSTED_VALUE_MODULE) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_resolve_route_sets_a_client_side_module_s_edge_cookie() { + let mut settings = settings_for_a_reader(); + settings.select_module("testing", POSTED_VALUE_MODULE); + settings.ec.module = Some(POSTED_VALUE_MODULE.into()); + settings.ec.module_blocks.clear(); + let origin = Arc::new(StubHttpClient::new()); + let router = router_for( + settings, + &[posted_value_builder()], + services_for_a_reader(&origin), + ); + let resolve = request_builder() + .method("POST") + .uri("https://test-publisher.example.com/_ts/api/v1/ec/resolve") + .header("host", "test-publisher.example.com") + .header("origin", "https://test-publisher.example.com") + .header("content-type", "text/plain") + .header("user-agent", BROWSER) + .body(Body::from(POSTED_VALUE)) + .expect("should build the resolve request"); + + let response = route(&router, resolve).await; + + assert_eq!( + response.status(), + StatusCode::OK, + "the module's identifier should be set with no identity graph" + ); + assert_eq!(header(&response, "cache-control"), Some("no-store")); + let cookie = edge_cookie(&response).expect("the resolve route should set the Edge Cookie"); + assert!( + cookie.starts_with("ts-ec=t0pv~a-posted-value;"), + "the Edge Cookie should carry the module's coded identifier: {cookie}" + ); + assert!( + response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .any(|line| line.starts_with("ts-ecr=1;")), + "the resolved marker should be set beside the Edge Cookie" + ); + assert!( + origin.recorded_request_uris().is_empty(), + "the resolve route should be answered here, never by the origin" + ); +} + +#[test] +fn startup_warns_that_this_adapter_opens_no_identity_graph() { + // The flag is set as the warning is logged, so a deployment whose + // `[ec] ec_store` names no store logs nothing. + let logged = AtomicBool::new(false); + super::warn_once_when_the_identity_graph_is_unavailable(&settings_for_a_reader(), &logged); + assert!( + !logged.load(Ordering::Relaxed), + "with no ec_store there is no identity graph to warn about" + ); + + let mut settings = settings_for_a_reader(); + settings.ec.ec_store = Some("ec_identity_store".to_owned()); + let origin = Arc::new(StubHttpClient::new()); + + let _router = router_for(settings, &[], services_for_a_reader(&origin)); + + assert!( + super::IDENTITY_GRAPH_UNAVAILABLE_LOGGED.load(Ordering::Relaxed), + "startup should warn that the identity graph is unavailable" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn robots_txt_shows_the_device_module_none_of_the_crawler_s_cookies() { let origin = Arc::new(StubHttpClient::new()); diff --git a/crates/trusted-server-adapter-cloudflare/src/middleware.rs b/crates/trusted-server-adapter-cloudflare/src/middleware.rs index 06f2b45358..c254009cec 100644 --- a/crates/trusted-server-adapter-cloudflare/src/middleware.rs +++ b/crates/trusted-server-adapter-cloudflare/src/middleware.rs @@ -5,6 +5,8 @@ use edgezero_core::context::RequestContext; use edgezero_core::error::EdgeError; use edgezero_core::http::Response; use edgezero_core::middleware::{Middleware, Next}; +use trusted_server_core::ec::finalize::ec_finalize_response; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::http_util::{ ROUTE_NAMESPACE, sanitize_trusted_client_ip_headers, strip_deployment_headers, }; @@ -15,6 +17,8 @@ use trusted_server_core::response_privacy::{ }; use trusted_server_core::settings::Settings; +use crate::app::EcFinalizeState; + // --------------------------------------------------------------------------- // SanitizeRequestMiddleware // --------------------------------------------------------------------------- @@ -59,7 +63,8 @@ impl Middleware for SanitizeRequestMiddleware { /// /// Registered directly inside [`SanitizeRequestMiddleware`], so that every /// outgoing response carries a consistent set of headers. Once the response -/// is finalized it applies the response effects the request filters left. +/// is finalized it finalizes the Edge Cookie and then applies the response +/// effects the request filters left. pub struct FinalizeResponseMiddleware { settings: Arc, } @@ -77,15 +82,49 @@ impl Middleware for FinalizeResponseMiddleware { async fn handle(&self, ctx: RequestContext, next: Next<'_>) -> Result { let path = ctx.request().uri().path().to_owned(); let mut response = next.run(ctx).await?; + let ec_state = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); // Before finalizing, so a header finalizing writes is kept. strip_deployment_headers(&path, ROUTE_NAMESPACE, &mut response); apply_finalize_headers(&self.settings, &mut response); + if let Some(mut ec_state) = ec_state { + finalize_edge_cookie(&self.settings, &mut ec_state, &mut response).await; + } apply_terminal_response_effects(&mut response, request_filter_effects.as_ref()); Ok(response) } } +/// Finalizes the Edge Cookie on a response with no identity graph, because +/// this adapter opens none. +/// +/// Core writes a created Edge Cookie without a graph as it does with one, for +/// a request that passed the bot gate and whose permissions allow it. A +/// partner registry that cannot be built skips finalizing. +async fn finalize_edge_cookie( + settings: &Settings, + ec_state: &mut EcFinalizeState, + response: &mut Response, +) { + let partner_registry = match PartnerRegistry::from_config(&settings.ec.partners) { + Ok(registry) => registry, + Err(report) => { + log::error!("EC finalize skipped: failed to build partner registry: {report:?}"); + return; + } + }; + ec_finalize_response( + settings, + &mut ec_state.ec_context, + None, + &partner_registry, + None, + response, + &ec_state.services, + ) + .await; +} + /// Applies the request filters' response effects to a finalized response, /// then restores the privacy guards a late header could loosen. fn apply_terminal_response_effects( diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 63a9fb4aa3..9aadcb9c9e 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -1053,9 +1053,9 @@ async fn run_named_route( unreachable!("the inspection endpoints are answered before EC setup") } NamedRouteHandler::EcResolve => { - // The resolve endpoint persists the identity-graph row before - // creating, so it takes the same bot-gated graph as generation: an - // unrecognized client gets no graph, and therefore no new Edge Cookie. + // The resolve endpoint takes the same bot-gated graph as + // generation, so an unrecognized client writes no graph row, and + // core's bot gate gives it no new Edge Cookie. handle_ec_resolve( &state.settings, req, @@ -2966,6 +2966,115 @@ mod tests { ); } + /// A location in the US, where the sample permission rules grant the Edge + /// Cookie with no signal. + struct UsGeo; + + #[async_trait::async_trait(?Send)] + impl trusted_server_core::platform::PlatformGeo for UsGeo { + async fn lookup( + &self, + _client_ip: Option, + _services: &RuntimeServices, + ) -> Result, Report> { + Ok(Some(trusted_server_core::platform::GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: String::new(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + })) + } + } + + #[test] + fn with_no_ec_store_the_entry_point_writes_a_created_edge_cookie() { + // The test settings name no `[ec] ec_store`, so the service opens no + // identity graph. A browser's page view still gets the Edge Cookie it + // creates, because a missing graph does not stop the cookie. + use trusted_server_core::platform::test_support::{ + NoopBackend, NoopConfigStore, NoopHttpClient, NoopSecretStore, + }; + + let settings = test_settings(); + assert_eq!( + settings.ec.ec_store, None, + "the test settings should name no ec_store" + ); + let services = RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(NoopKvStore)) + .backend(Arc::new(NoopBackend)) + .http_client(Arc::new(NoopHttpClient)) + .geo(Arc::new( + trusted_server_core::platform::PlatformGeoModule::new("platform", Arc::new(UsGeo)), + )) + .client_info(ClientInfo { + client_ip: Some(IpAddr::V4(Ipv4Addr::new(192, 0, 2, 10))), + ..ClientInfo::default() + }) + .permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()) + .build(); + let request = browser_request(Method::GET, "/article", "document"); + + let mut ec = block_on(super::build_ec_request_state( + &settings, &services, &request, + )); + assert!( + ec.is_real_browser, + "the browser's page view should pass the bot gate" + ); + assert!( + ec.kv_graph.is_none() && ec.finalize_kv_graph.is_none(), + "with no ec_store the service should open no identity graph" + ); + // The publisher fallback's step for a browser's page navigation. + block_on( + ec.ec_context + .generate_if_needed(&settings, ec.kv_graph.as_ref(), &services), + ) + .expect("should create the Edge Cookie"); + let ec_id = ec + .ec_context + .ec_value() + .expect("the page view should create an Edge Cookie") + .to_owned(); + let mut finalize = ec.into_finalize_state(); + let mut response = edgezero_core::http::response_builder() + .status(StatusCode::OK) + .body(Body::empty()) + .expect("should build the page response"); + + block_on(crate::apply_edgezero_ec_finalize( + &settings, + &mut finalize, + &mut response, + )) + .expect("the partner registry should build"); + + let cookies: Vec<&str> = response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .filter(|cookie| cookie.starts_with("ts-ec=")) + .collect(); + assert_eq!( + cookies.len(), + 1, + "the entry point should write the created Edge Cookie once, got {cookies:?}" + ); + assert!( + cookies[0].starts_with(&format!("ts-ec={ec_id};")), + "the Edge Cookie should carry the created identifier, got {}", + cookies[0] + ); + } + #[test] fn entry_point_client_info_reaches_request_filters() { // Regression guard for the EdgeZero bot-protection metadata loss: diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index b10d22b73d..13931ae28c 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -1,6 +1,6 @@ use std::net::{IpAddr, SocketAddr}; use std::sync::Arc; -use std::sync::atomic::AtomicBool; +use std::sync::atomic::{AtomicBool, Ordering}; #[cfg(all(feature = "spin", target_arch = "wasm32"))] use edgezero_adapter_spin::config_store::SpinConfigStore; @@ -30,9 +30,10 @@ use trusted_server_core::ec::device::{ BuiltinDeviceModule, DeviceModule, DeviceSignals, build_device_module, }; use trusted_server_core::ec::module::{EdgeCookieModule, build_reusable_module}; +use trusted_server_core::ec::resolve::handle_ec_resolve; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::evidence::BorrowedRequestInfo; -use trusted_server_core::http_util::sanitize_forwarded_headers; +use trusted_server_core::http_util::{is_navigation_request, sanitize_forwarded_headers}; use trusted_server_core::inspect::config::{CONFIG_JSON_PATH, CONFIG_PAGE_PATH, handle_config}; use trusted_server_core::inspect::data::{DATA_PAGE_PATH, handle_data}; use trusted_server_core::inspect::permissions::{ @@ -277,6 +278,7 @@ fn build_state_with_registrations_and_services( "Spin", &UNSENT_ANALYTICS_LOGGED, ); + warn_once_when_the_identity_graph_is_unavailable(&settings, &IDENTITY_GRAPH_UNAVAILABLE_LOGGED); let permission_policy = settings.required_permission_policy()?; Ok(Arc::new(AppState { @@ -315,6 +317,32 @@ fn refuse_platform_geo(settings: &Settings) -> Result<(), Report [Method; 7] { // publisher-fallback method on these paths is routed to the fallback so it // reaches the publisher origin rather than a router-level 405. // -// The EC API routes that the Fastly entry point registers — POST -// `/_ts/api/v1/batch-sync`, GET/OPTIONS `/_ts/api/v1/identify`, POST -// `/_ts/api/v1/ec/resolve`, are -// intentionally absent here, matching the Axum and Cloudflare adapters: those -// handlers require a platform KV `ec_store` (and, for batch-sync, a partner -// registry and rate limiter) that the portability adapters do not yet wire. -// On Spin these paths fall through to the publisher/integration fallback, -// identical to the other non-Fastly adapters. -fn named_fallback_paths() -> [(&'static str, &'static [Method]); 14] { +// POST `/_ts/api/v1/batch-sync` and GET/OPTIONS `/_ts/api/v1/identify` are +// absent, because their handlers need the identity graph this adapter does +// not open (and, for batch-sync, a partner registry and rate limiter). On +// Spin these paths fall through to the publisher/integration fallback. +fn named_fallback_paths() -> [(&'static str, &'static [Method]); 15] { [ ("/.well-known/trusted-server.json", &[Method::GET]), ("/verify-signature", &[Method::POST]), @@ -520,6 +544,7 @@ fn named_fallback_paths() -> [(&'static str, &'static [Method]); 14] { (CONFIG_PAGE_PATH, &[Method::GET]), (CONFIG_JSON_PATH, &[Method::GET]), (DATA_PAGE_PATH, &[Method::GET]), + ("/_ts/api/v1/ec/resolve", &[Method::POST]), ("/auction", &[Method::POST]), (PAGE_BIDS_PATH, &[Method::GET, Method::OPTIONS]), (PAGE_BIDS_LEGACY_PATH, &[Method::GET, Method::OPTIONS]), @@ -781,6 +806,65 @@ async fn build_ec_context( Ok(context) } +/// The request's Edge Cookie state, read before its route runs. +struct EcRequestState { + ec_context: EcContext, + /// Whether the request passed the bot gate, which creating an Edge Cookie + /// on a page navigation sits behind. + is_real_browser: bool, +} + +/// The Edge Cookie state a route leaves on its response. +/// [`FinalizeResponseMiddleware`] takes it off the response and finalizes the +/// Edge Cookie once the response is finalized. +#[derive(Clone)] +pub(crate) struct EcFinalizeState { + pub(crate) ec_context: EcContext, + /// The request's services, which finalizing hands to the selected Edge + /// Cookie module. + pub(crate) services: RuntimeServices, +} + +/// Reads the request's Edge Cookie state, with the bot gate's answer taken +/// from the device classification. +/// +/// # Errors +/// +/// The errors of [`build_ec_context`]. +async fn build_ec_request_state( + settings: &Settings, + services: &RuntimeServices, + req: &Request, +) -> Result> { + let ec_context = build_ec_context(settings, services, req).await?; + let is_real_browser = ec_context + .device_signals() + .is_some_and(|signals| signals.looks_like_browser); + if !is_real_browser { + log::info!("Bot gate: blocking EC operations"); + } + Ok(EcRequestState { + ec_context, + is_real_browser, + }) +} + +/// Leaves the Edge Cookie state and the non-empty request-filter response +/// effects on a response of the request cycle, for +/// [`FinalizeResponseMiddleware`]. +fn attach_dispatch_extensions( + mut response: Response, + ec: EcRequestState, + services: &RuntimeServices, + effects: RequestFilterEffects, +) -> Response { + response.extensions_mut().insert(EcFinalizeState { + ec_context: ec.ec_context, + services: services.clone(), + }); + attach_request_filter_effects(response, effects) +} + /// Result of the request-filter step that runs before a route. enum PreRoute { /// A filter answered, or failed, and that answer stands in for the @@ -873,6 +957,7 @@ pub(crate) fn http_error(report: &Report) -> Response { enum NamedRouteHandler { TrustedServerDiscovery, VerifySignature, + EcResolve, Auction, PageBids, FirstPartyProxy, @@ -907,21 +992,21 @@ async fn execute_named( return Ok(http_error(&error)); } - let mut ec_context = match build_ec_context(&state.settings, &services, &req).await { - Ok(context) => context, + let mut ec = match build_ec_request_state(&state.settings, &services, &req).await { + Ok(ec) => ec, Err(report) => return Ok(http_error(&report)), }; - let effects = match run_pre_route_filters(&state, &services, &mut req, &ec_context).await { + let effects = match run_pre_route_filters(&state, &services, &mut req, &ec.ec_context).await { PreRoute::ShortCircuit { response, effects } => { - return Ok(attach_request_filter_effects(response, effects)); + return Ok(attach_dispatch_extensions(response, ec, &services, effects)); } PreRoute::Continue { effects } => effects, }; - let response = run_named_route(&state, &services, req, handler, &mut ec_context) + let response = run_named_route(&state, &services, req, handler, &mut ec.ec_context) .await .unwrap_or_else(|error| http_error(&error)); - Ok(attach_request_filter_effects(response, effects)) + Ok(attach_dispatch_extensions(response, ec, &services, effects)) } async fn run_named_route( @@ -938,6 +1023,11 @@ async fn run_named_route( NamedRouteHandler::VerifySignature => { handle_verify_signature(&state.settings, services, req) } + NamedRouteHandler::EcResolve => { + // This adapter opens no identity graph, so a created Edge Cookie + // is set with no graph row. + handle_ec_resolve(&state.settings, req, ec_context, None, services).await + } NamedRouteHandler::Auction => { // The auction consent gate reads the caller's jurisdiction from // the geo-aware context, where `EcContext::default()` would fail it @@ -1028,13 +1118,13 @@ async fn dispatch_fallback( // Identity could not be established (for example the selected Edge // Cookie module is unavailable). Answer with an error rather than // serving the request with no identity. - let mut ec_context = match build_ec_context(&state.settings, services, &req).await { - Ok(context) => context, + let mut ec = match build_ec_request_state(&state.settings, services, &req).await { + Ok(ec) => ec, Err(report) => return http_error(&report), }; - let effects = match run_pre_route_filters(state, services, &mut req, &ec_context).await { + let effects = match run_pre_route_filters(state, services, &mut req, &ec.ec_context).await { PreRoute::ShortCircuit { response, effects } => { - return attach_request_filter_effects(response, effects); + return attach_dispatch_extensions(response, ec, services, effects); } PreRoute::Continue { effects } => effects, }; @@ -1052,7 +1142,7 @@ async fn dispatch_fallback( path: &path, settings: &state.settings, kv: None, - ec_context: &mut ec_context, + ec_context: &mut ec.ec_context, services, req, }) @@ -1063,6 +1153,17 @@ async fn dispatch_fallback( })) }) } else { + // A real browser's page navigation is given an Edge Cookie when it has + // none. Another request may lack consent signals such as `Sec-GPC`. + if ec.is_real_browser + && is_navigation_request(&req) + && let Err(err) = ec + .ec_context + .generate_if_needed(&state.settings, None, services) + .await + { + log::error!("EC generation failed for publisher proxy: {err:?}"); + } let auction = AuctionDispatch { orchestrator: &state.orchestrator, slots: state.settings.creative_opportunity_slots(), @@ -1075,7 +1176,7 @@ async fn dispatch_fallback( }, services, None, - &mut ec_context, + &mut ec.ec_context, auction, req, EDGE_CACHE_HEADER, @@ -1097,7 +1198,12 @@ async fn dispatch_fallback( } }; - attach_request_filter_effects(result.unwrap_or_else(|error| http_error(&error)), effects) + attach_dispatch_extensions( + result.unwrap_or_else(|error| http_error(&error)), + ec, + services, + effects, + ) } /// Whether the request is a crawler's `GET` or `HEAD` of the robots.txt that @@ -1413,6 +1519,10 @@ fn build_router(state: &Arc) -> RouterService { "/verify-signature", named(NamedRouteHandler::VerifySignature), ) + .post( + "/_ts/api/v1/ec/resolve", + named(NamedRouteHandler::EcResolve), + ) .get(PERMISSIONS_PAGE_PATH, permissions_handler.clone()) .get(PERMISSIONS_JSON_PATH, permissions_handler) .get(CONFIG_PAGE_PATH, config_handler.clone()) diff --git a/crates/trusted-server-adapter-spin/src/app/request_cycle_tests.rs b/crates/trusted-server-adapter-spin/src/app/request_cycle_tests.rs index 7eb8a9c850..9c94d57873 100644 --- a/crates/trusted-server-adapter-spin/src/app/request_cycle_tests.rs +++ b/crates/trusted-server-adapter-spin/src/app/request_cycle_tests.rs @@ -3,16 +3,22 @@ //! route in the cycle, and their response effects after the finalize headers, //! without making the response less private, the route's module call carries //! the device classification, an Edge Cookie context that cannot be read -//! answers before the filters, and robots.txt and the inspection endpoints -//! are answered without them. +//! answers before the filters, robots.txt and the inspection endpoints are +//! answered without them, and the Edge Cookie is created and written for a +//! real browser's page view, set by the resolve route for a client-side +//! module and expired on a withdrawal. +use std::net::{IpAddr, Ipv4Addr}; +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; use edgezero_core::body::Body; use edgezero_core::http::{HeaderValue, Method, Request, Response, StatusCode, request_builder}; use edgezero_core::router::RouterService; use error_stack::Report; +use trusted_server_core::addressing::addressed_router; use trusted_server_core::ec::device::{DeviceModule, DeviceSignals}; +use trusted_server_core::ec::module::{EdgeCookieModule, GeneratedEdgeCookie, ModuleCode}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::evidence::RequestInfo; use trusted_server_core::integrations::{ @@ -21,13 +27,18 @@ use trusted_server_core::integrations::{ RequestFilterInput, }; use trusted_server_core::module_context::ModuleCall; -use trusted_server_core::platform::PlatformHttpClient; +use trusted_server_core::permissions::{Permission, PermissionSet}; use trusted_server_core::platform::test_support::{ - StubHttpClient, build_services_with_http_client, + NoopConfigStore, NoopSecretStore, StubBackend, StubHttpClient, build_services_with_http_client, +}; +use trusted_server_core::platform::{ + ClientInfo, GeoInfo, PlatformError, PlatformGeo, PlatformGeoModule, PlatformHttpClient, + RuntimeServices, UnavailableKvStore, }; use trusted_server_core::publisher::{PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH}; use trusted_server_core::response_privacy::enforce_terminal_private_cache_privacy; use trusted_server_core::settings::Settings; +use trusted_server_core::test_support::addressing::FirstSegment; use super::{build_router, build_state_with_registrations_and_services}; @@ -391,6 +402,18 @@ fn settings_with(extra: &str) -> Settings { .with_permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()) } +/// The router built through the composition root with `builders`, every +/// request handed `services`. +fn router_for( + settings: Settings, + builders: &[IntegrationBuilder], + services: RuntimeServices, +) -> RouterService { + let state = build_state_with_registrations_and_services(settings, builders, Some(services)) + .expect("should build the state"); + build_router(&state) +} + /// The router built through the composition root with `builders`, every /// request sent to `origin`. fn router_with( @@ -398,15 +421,11 @@ fn router_with( builders: &[IntegrationBuilder], origin: &Arc, ) -> RouterService { - let state = build_state_with_registrations_and_services( + router_for( settings, builders, - Some(build_services_with_http_client( - Arc::clone(origin) as Arc - )), + build_services_with_http_client(Arc::clone(origin) as Arc), ) - .expect("should build the state"); - build_router(&state) } fn get(path: &str, cookie: Option<&str>) -> Request { @@ -647,6 +666,410 @@ async fn robots_txt_is_answered_before_the_request_filters() { assert_eq!(origin.recorded_request_methods(), ["GET"]); } +// --------------------------------------------------------------------------- +// The Edge Cookie +// --------------------------------------------------------------------------- + +/// The `User-Agent` of a desktop browser, which the bot gate passes. +const BROWSER: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"; + +/// The `User-Agent` of a crawler, which the bot gate stops. +const CRAWLER: &str = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"; + +/// A page the publisher's origin answers. +const PAGE: &[u8] = b"article"; + +/// An Edge Cookie identifier in the shape the `hmac` module gives one. +const CARRIED_EC: &str = + "hmac~0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef.abc123"; + +/// A TC String that consents to no purpose, so it refuses storage, Purpose 1. +/// It is version 2 from CMP 1, in English, for vendor list 1 and no vendor, +/// last updated at time zero, so the settings that read it turn the TCF +/// module's expiry check off. +const STORAGE_REFUSING_TC_STRING: &str = "CAAAAAAAAAAAAABABAENABCAAAAAAAAAAAhoAAA"; + +/// A location in the US, where the sample permission rules grant the Edge +/// Cookie with no signal. +struct UsGeo; + +#[async_trait::async_trait(?Send)] +impl PlatformGeo for UsGeo { + async fn lookup( + &self, + _client_ip: Option, + _services: &RuntimeServices, + ) -> Result, Report> { + Ok(Some(GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: String::new(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + })) + } +} + +/// The services of a reader in the US with an address, carrying the +/// permission signal modules this adapter links, every request sent to +/// `origin`. +fn services_for_a_reader(origin: &Arc) -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(UnavailableKvStore)) + .backend(Arc::new(StubBackend)) + .http_client(Arc::clone(origin) as Arc) + .geo(Arc::new(PlatformGeoModule::new( + "platform", + Arc::new(UsGeo), + ))) + .client_info(ClientInfo { + client_ip: Some(IpAddr::V4(Ipv4Addr::new(192, 0, 2, 10))), + ..ClientInfo::default() + }) + .permission_signal_modules(super::shipped_signal_modules().into()) + .permission_policy(trusted_server_core::permissions::PermissionPolicy::sample()) + .build() +} + +/// Settings whose `[geo] module = "platform"` leaves the location the +/// reader's services give standing, where an unset selector resolves the +/// disabled module. +fn settings_for_a_reader() -> Settings { + let mut settings = settings_with(""); + settings.geo.module = Some("platform".to_owned()); + settings +} + +/// An origin answering each of `pages` requests with [`PAGE`]. +fn origin_answering(pages: usize) -> Arc { + let origin = Arc::new(StubHttpClient::new()); + for _ in 0..pages { + origin.push_response_with_headers( + 200, + PAGE.to_vec(), + vec![("content-type", "text/html; charset=utf-8")], + ); + } + origin +} + +/// A page navigation to `path` by a client with `user_agent`, carrying +/// `cookie` when there is one. +fn page_view(path: &str, user_agent: &str, cookie: Option<&str>) -> Request { + let mut builder = request_builder() + .method("GET") + .uri(format!("https://test-publisher.example.com{path}")) + .header("host", "test-publisher.example.com") + .header("user-agent", user_agent) + .header("accept", "text/html") + .header("sec-fetch-dest", "document") + .header("sec-fetch-mode", "navigate"); + if let Some(cookie) = cookie { + builder = builder.header("cookie", cookie); + } + builder + .body(Body::empty()) + .expect("should build the page view") +} + +/// The `Set-Cookie` line that writes the Edge Cookie, if the response has one. +/// A response writes it at most once, because core alone writes it. +fn edge_cookie(response: &Response) -> Option { + let lines: Vec = response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .filter(|line| line.starts_with("ts-ec=")) + .map(str::to_owned) + .collect(); + assert!( + lines.len() <= 1, + "the Edge Cookie should be written at most once: {lines:?}" + ); + lines.into_iter().next() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_first_page_view_gets_the_edge_cookie_under_the_public_prefix_and_path() { + let origin = origin_answering(1); + let router = addressed_router( + router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)), + Arc::new(FirstSegment), + ); + + let response = route( + &router, + page_view("/publisher.example/article", BROWSER, None), + ) + .await; + + assert_eq!(response.status(), StatusCode::OK); + let cookie = edge_cookie(&response).expect("a first page view should get the Edge Cookie"); + assert!( + cookie.contains("Path=/publisher.example"), + "the Edge Cookie should be written under the publisher's path: {cookie}" + ); + assert_eq!( + origin.recorded_request_uris(), + ["https://origin.test-publisher.example.com/article"], + "the page should be asked for at the path the addressing continues with" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_returning_page_view_keeps_its_edge_cookie() { + let origin = origin_answering(2); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + + let first = route(&router, page_view("/article", BROWSER, None)).await; + let written = edge_cookie(&first).expect("the first page view should get the Edge Cookie"); + assert!( + written.contains("Path=/;") || written.ends_with("Path=/"), + "a request addressed by host writes the Edge Cookie under the root: {written}" + ); + let pair = written + .split(';') + .next() + .expect("a Set-Cookie line starts with its pair") + .to_owned(); + + let returning = route(&router, page_view("/article", BROWSER, Some(&pair))).await; + + assert_eq!(returning.status(), StatusCode::OK); + assert_eq!( + edge_cookie(&returning), + None, + "a returning page view should keep the Edge Cookie it carries" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_bot_s_page_view_gets_no_edge_cookie() { + let origin = origin_answering(1); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + + let response = route(&router, page_view("/article", CRAWLER, None)).await; + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + edge_cookie(&response), + None, + "the bot gate should keep a crawler from getting an Edge Cookie" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_withdrawal_expires_the_edge_cookie_the_browser_carries() { + // No `[geo]` module, so the reader resolves at the top of the sample + // rules, the `gdpr-eu` group, where storage needs a signal, so a TC String + // refusing storage withdraws it. + let origin = origin_answering(1); + let router = router_for( + settings_with("[consent]\ncheck_expiration = false"), + &[], + services_for_a_reader(&origin), + ); + let cookies = format!("ts-ec={CARRIED_EC}; euconsent-v2={STORAGE_REFUSING_TC_STRING}"); + + let response = route(&router, page_view("/article", BROWSER, Some(&cookies))).await; + + assert_eq!(response.status(), StatusCode::OK); + let expired = + edge_cookie(&response).expect("a withdrawal should expire the Edge Cookie it carries"); + assert!( + expired.starts_with("ts-ec=;") && expired.contains("Max-Age=0"), + "the Edge Cookie should be expired: {expired}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_resolve_route_answers() { + let origin = Arc::new(StubHttpClient::new()); + let router = router_for(settings_for_a_reader(), &[], services_for_a_reader(&origin)); + let resolve = |page_origin: &str| { + request_builder() + .method("POST") + .uri("https://test-publisher.example.com/_ts/api/v1/ec/resolve") + .header("host", "test-publisher.example.com") + .header("origin", page_origin) + .header("content-type", "text/plain") + .header("user-agent", BROWSER) + .body(Body::from("posted value")) + .expect("should build the resolve request") + }; + + let foreign = route(&router, resolve("https://elsewhere.example")).await; + assert_eq!( + foreign.status(), + StatusCode::FORBIDDEN, + "a foreign Origin should be refused" + ); + assert_eq!(header(&foreign, "cache-control"), Some("no-store")); + + let own = route(&router, resolve("https://test-publisher.example.com")).await; + assert_eq!( + own.status(), + StatusCode::NO_CONTENT, + "a server-side module creates nothing from a posted value" + ); + assert_eq!(edge_cookie(&own), None); + + assert!( + origin.recorded_request_uris().is_empty(), + "the resolve route should be answered here, never by the origin" + ); +} + +/// The id and module name of an Edge Cookie module that creates nothing at +/// the edge and takes its identifier from the value the page posts, as a +/// client-side module does. +const POSTED_VALUE_ID: &str = "posted_value"; +const POSTED_VALUE_MODULE: &str = "testing.posted-value"; + +/// The value the page posts, which the module takes as its identifier. +const POSTED_VALUE: &str = "a-posted-value"; + +#[derive(Debug)] +struct PostedValueModule; + +#[async_trait::async_trait(?Send)] +impl EdgeCookieModule for PostedValueModule { + fn id(&self) -> &'static str { + POSTED_VALUE_MODULE + } + + fn code(&self) -> ModuleCode { + trusted_server_core::module_code!("t0pv") + } + + async fn generate( + &self, + _call: ModuleCall<'_>, + ) -> Result> { + Ok(GeneratedEdgeCookie::default()) + } + + async fn resolve_from_client( + &self, + _call: ModuleCall<'_>, + payload: &[u8], + ) -> Result> { + Ok(GeneratedEdgeCookie { + id: (payload == POSTED_VALUE.as_bytes()).then(|| POSTED_VALUE.to_owned()), + response_headers: Vec::new(), + }) + } + + fn accepts_id(&self, value: &str) -> bool { + value == POSTED_VALUE + } + + fn required_permissions(&self) -> PermissionSet { + PermissionSet::none().with(Permission::StoreOnDevice) + } +} + +fn register_posted_value( + _settings: &Settings, +) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder(POSTED_VALUE_ID) + .without_js() + .with_ec_module(POSTED_VALUE_MODULE, Arc::new(PostedValueModule)) + .build(), + )) +} + +fn posted_value_builder() -> IntegrationBuilder { + IntegrationBuilder::new( + POSTED_VALUE_ID, + "adapter-tests", + register_posted_value, + always, + ) + .with_module_name(POSTED_VALUE_MODULE) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_resolve_route_sets_a_client_side_module_s_edge_cookie() { + let mut settings = settings_for_a_reader(); + settings.select_module("testing", POSTED_VALUE_MODULE); + settings.ec.module = Some(POSTED_VALUE_MODULE.into()); + settings.ec.module_blocks.clear(); + let origin = Arc::new(StubHttpClient::new()); + let router = router_for( + settings, + &[posted_value_builder()], + services_for_a_reader(&origin), + ); + let resolve = request_builder() + .method("POST") + .uri("https://test-publisher.example.com/_ts/api/v1/ec/resolve") + .header("host", "test-publisher.example.com") + .header("origin", "https://test-publisher.example.com") + .header("content-type", "text/plain") + .header("user-agent", BROWSER) + .body(Body::from(POSTED_VALUE)) + .expect("should build the resolve request"); + + let response = route(&router, resolve).await; + + assert_eq!( + response.status(), + StatusCode::OK, + "the module's identifier should be set with no identity graph" + ); + assert_eq!(header(&response, "cache-control"), Some("no-store")); + let cookie = edge_cookie(&response).expect("the resolve route should set the Edge Cookie"); + assert!( + cookie.starts_with("ts-ec=t0pv~a-posted-value;"), + "the Edge Cookie should carry the module's coded identifier: {cookie}" + ); + assert!( + response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .any(|line| line.starts_with("ts-ecr=1;")), + "the resolved marker should be set beside the Edge Cookie" + ); + assert!( + origin.recorded_request_uris().is_empty(), + "the resolve route should be answered here, never by the origin" + ); +} + +#[test] +fn startup_warns_that_this_adapter_opens_no_identity_graph() { + // The flag is set as the warning is logged, so a deployment whose + // `[ec] ec_store` names no store logs nothing. + let logged = AtomicBool::new(false); + super::warn_once_when_the_identity_graph_is_unavailable(&settings_for_a_reader(), &logged); + assert!( + !logged.load(Ordering::Relaxed), + "with no ec_store there is no identity graph to warn about" + ); + + let mut settings = settings_for_a_reader(); + settings.ec.ec_store = Some("ec_identity_store".to_owned()); + let origin = Arc::new(StubHttpClient::new()); + + let _router = router_for(settings, &[], services_for_a_reader(&origin)); + + assert!( + super::IDENTITY_GRAPH_UNAVAILABLE_LOGGED.load(Ordering::Relaxed), + "startup should warn that the identity graph is unavailable" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn robots_txt_shows_the_device_module_none_of_the_crawler_s_cookies() { let origin = Arc::new(StubHttpClient::new()); diff --git a/crates/trusted-server-adapter-spin/src/middleware.rs b/crates/trusted-server-adapter-spin/src/middleware.rs index 81c8f1cac0..eb12d6f65c 100644 --- a/crates/trusted-server-adapter-spin/src/middleware.rs +++ b/crates/trusted-server-adapter-spin/src/middleware.rs @@ -5,6 +5,8 @@ use edgezero_core::context::RequestContext; use edgezero_core::error::EdgeError; use edgezero_core::http::Response; use edgezero_core::middleware::{Middleware, Next}; +use trusted_server_core::ec::finalize::ec_finalize_response; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::http_util::{ ROUTE_NAMESPACE, sanitize_trusted_client_ip_headers, strip_deployment_headers, }; @@ -15,6 +17,8 @@ use trusted_server_core::response_privacy::{ }; use trusted_server_core::settings::Settings; +use crate::app::EcFinalizeState; + // --------------------------------------------------------------------------- // SanitizeRequestMiddleware // --------------------------------------------------------------------------- @@ -59,7 +63,8 @@ impl Middleware for SanitizeRequestMiddleware { /// /// Registered directly inside [`SanitizeRequestMiddleware`], so that every /// outgoing response carries a consistent set of headers. Once the response -/// is finalized it applies the response effects the request filters left. +/// is finalized it finalizes the Edge Cookie and then applies the response +/// effects the request filters left. pub struct FinalizeResponseMiddleware { settings: Arc, } @@ -77,15 +82,49 @@ impl Middleware for FinalizeResponseMiddleware { async fn handle(&self, ctx: RequestContext, next: Next<'_>) -> Result { let path = ctx.request().uri().path().to_owned(); let mut response = next.run(ctx).await?; + let ec_state = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); // Before finalizing, so a header finalizing writes is kept. strip_deployment_headers(&path, ROUTE_NAMESPACE, &mut response); apply_finalize_headers(&self.settings, &mut response); + if let Some(mut ec_state) = ec_state { + finalize_edge_cookie(&self.settings, &mut ec_state, &mut response).await; + } apply_terminal_response_effects(&mut response, request_filter_effects.as_ref()); Ok(response) } } +/// Finalizes the Edge Cookie on a response with no identity graph, because +/// this adapter opens none. +/// +/// Core writes a created Edge Cookie without a graph as it does with one, for +/// a request that passed the bot gate and whose permissions allow it. A +/// partner registry that cannot be built skips finalizing. +async fn finalize_edge_cookie( + settings: &Settings, + ec_state: &mut EcFinalizeState, + response: &mut Response, +) { + let partner_registry = match PartnerRegistry::from_config(&settings.ec.partners) { + Ok(registry) => registry, + Err(report) => { + log::error!("EC finalize skipped: failed to build partner registry: {report:?}"); + return; + } + }; + ec_finalize_response( + settings, + &mut ec_state.ec_context, + None, + &partner_registry, + None, + response, + &ec_state.services, + ) + .await; +} + /// Applies the request filters' response effects to a finalized response, /// then restores the privacy guards a late header could loosen. fn apply_terminal_response_effects( diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index 59959c4034..6297279ba3 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -30,6 +30,13 @@ use super::{EcKvSnapshot, EidSyncSource, current_timestamp, log_id}; /// ingestion of the identifiers the request carried, and cookie writes for /// new EC generation. /// +/// A newly created Edge Cookie is written for a request that passed the bot +/// gate and whose permissions allow it, whether or not `kv` holds an identity +/// graph. The graph is one optional store of the data kept with the cookie, +/// such as partner IDs. With one, the cookie's row is written first and the +/// cookie only once that row is authoritative. Without one, the graph steps +/// are skipped. +/// /// When the request carries an explicit withdrawal signal (a storage opt-out or /// a record refusing storage) and the client presented a cookie, the browser /// response clears the EC cookie immediately and the EC identity-graph KV @@ -171,39 +178,71 @@ pub async fn ec_finalize_response( return; } - // Newly generated EC in this request. Do not emit a generated EC when - // there is no KV graph: that would mint a browser cookie with no backing - // identity-graph row, producing a phantom ID on later requests. + // Newly generated EC in this request. if ec_context.ec_generated() { - let (Some(graph), Some(kv_key)) = (kv, ec_context.ec_kv_key()) else { - log::info!( - "Skipping generated EC response write because the KV graph or the \ - identity-graph key is unavailable" - ); - reconcile_pull_sync_marker(settings, registry, ec_context, response); - return; - }; - - let updates = gate_eid_updates_by_permissions( - collect_eid_updates( - ec_context.module_eids(), - sharedid_cookie, - ec_context.client_eids(), - registry, - ), + write_generated_ec( + settings, ec_context, + kv, + registry, + sharedid_cookie, + response, ); - sync_eid_cookie_updates(graph, ec_context, &kv_key, &updates, EidSyncSource::NewEc); - if ec_context.kv_snapshot().entry_for(&kv_key).is_some() { - set_ec_cookie_on_response(settings, ec_context, response); - } else { - log::warn!("Skipping generated EC cookie because backing row is not authoritative"); - } } reconcile_pull_sync_marker(settings, registry, ec_context, response); } +/// Writes the Edge Cookie this request created, for a request that passed +/// the bot gate. +/// +/// With an identity graph, the identifiers the request carried are recorded +/// on the cookie's row first, and the cookie is written only once that row is +/// authoritative, so every cookie written with a graph has a row in it. +/// Without one, the cookie is written with no row, because a missing graph +/// does not stop the cookie. +fn write_generated_ec( + settings: &Settings, + ec_context: &mut EcContext, + kv: Option<&KvIdentityGraph>, + registry: &PartnerRegistry, + sharedid_cookie: Option<&str>, + response: &mut Response, +) { + if !ec_context.passed_bot_gate() { + log::info!( + "Skipping generated EC response write because the request did not pass the bot gate" + ); + return; + } + let Some(graph) = kv else { + set_ec_cookie_on_response(settings, ec_context, response); + return; + }; + let Some(kv_key) = ec_context.ec_kv_key() else { + log::info!( + "Skipping generated EC response write because the identity-graph key is unavailable" + ); + return; + }; + + let updates = gate_eid_updates_by_permissions( + collect_eid_updates( + ec_context.module_eids(), + sharedid_cookie, + ec_context.client_eids(), + registry, + ), + ec_context, + ); + sync_eid_cookie_updates(graph, ec_context, &kv_key, &updates, EidSyncSource::NewEc); + if ec_context.kv_snapshot().entry_for(&kv_key).is_some() { + set_ec_cookie_on_response(settings, ec_context, response); + } else { + log::warn!("Skipping generated EC cookie because backing row is not authoritative"); + } +} + fn sync_eid_cookie_updates( graph: &KvIdentityGraph, ec_context: &mut EcContext, @@ -1184,41 +1223,151 @@ mod tests { } #[tokio::test] - async fn finalize_generated_ec_without_kv_skips_cookie_and_header() { + async fn finalize_generated_ec_without_kv_writes_the_cookie_and_skips_the_graph() { + // The identity graph is one optional store of the data kept with the + // Edge Cookie, so a missing graph does not stop the cookie. The + // response gets what it gets with a graph, and the graph steps are + // skipped, so the partner ID a graph records is kept nowhere. let settings = create_test_settings(); let generated_ec = sample_ec_id("gen123"); - let mut ec_context = make_context( - Some(&generated_ec), + let created_ec = || { + let mut ec_context = make_context( + Some(&generated_ec), + None, + false, + true, + Jurisdiction::NonRegulated, + true, + ); + ec_context.set_device_signals(crate::ec::tests::browser_signals()); + ec_context + }; + let partners = vec![make_partner("sharedid.org")]; + let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); + + let graph = KvIdentityGraph::in_memory("test_store"); + let live = KvEntry::new( + &granting_consent(), None, - false, - true, - Jurisdiction::NonRegulated, - true, + current_timestamp(), + &settings.publisher.domain, ); - let mut response = empty_response(); + graph + .create(&generated_ec, &live) + .expect("should seed the generated identifier's row"); + let mut with_graph = created_ec(); + with_graph.set_kv_snapshot(graph.load_snapshot(&generated_ec)); + let mut graph_response = empty_response(); + ec_finalize_response( + &settings, + &mut with_graph, + Some(&graph), + ®istry, + Some("shared-cookie-id"), + &mut graph_response, + &noop_services(), + ) + .await; - let test_registry = PartnerRegistry::empty(); + let mut ec_context = created_ec(); + let mut response = empty_response(); ec_finalize_response( &settings, &mut ec_context, None, - &test_registry, - None, + ®istry, + Some("shared-cookie-id"), &mut response, &noop_services(), ) .await; + let set_cookie = get_header_str(&response, "set-cookie") + .expect("generated EC without KV should set the cookie"); assert!( - get_header(&response, "x-ts-ec").is_none(), - "generated EC without KV should not set response header" + set_cookie.starts_with(&format!("ts-ec={generated_ec};")), + "generated EC without KV should write its identifier, got: {set_cookie}" + ); + assert_eq!( + response.headers(), + graph_response.headers(), + "generated EC without KV should get the cookie and headers a graph gets" ); assert!( - get_header(&response, "set-cookie").is_none(), - "generated EC without KV should not set cookie" + matches!(ec_context.kv_snapshot(), EcKvSnapshot::NotRead), + "generated EC without KV should run no graph step" + ); + let (row, _) = graph + .get(&generated_ec) + .expect("should read the graph") + .expect("the row should remain"); + assert_eq!( + row.ids.get("sharedid.org").map(|id| id.uid.as_str()), + Some("shared-cookie-id"), + "with a graph the request's partner ID should be recorded on the row" ); } + #[tokio::test] + async fn a_generated_ec_is_written_only_for_a_request_that_passed_the_bot_gate() { + // The bot gate reads the request's device signals, so a crawler or a + // request with none gets no created Edge Cookie, with a graph or + // without, and a browser gets one either way. + let settings = create_test_settings(); + for (case, signals, written) in [ + ("a browser", Some(crate::ec::tests::browser_signals()), true), + ( + "a crawler", + Some(crate::ec::tests::crawler_signals()), + false, + ), + ("a request with no device signals", None, false), + ] { + for with_graph in [true, false] { + let ec_id = sample_ec_id("botgt1"); + let graph = KvIdentityGraph::in_memory("test_store"); + graph + .create(&ec_id, &live_entry()) + .expect("should seed the generated identifier's row"); + let mut ec_context = make_context( + Some(&ec_id), + None, + false, + true, + Jurisdiction::NonRegulated, + true, + ); + if let Some(signals) = signals.clone() { + ec_context.set_device_signals(signals); + } + ec_context.set_kv_snapshot(EcKvSnapshot::Present { + ec_id: ec_id.clone(), + entry: Box::new(live_entry()), + generation: None, + }); + let mut response = empty_response(); + + ec_finalize_response( + &settings, + &mut ec_context, + with_graph.then_some(&graph), + &PartnerRegistry::empty(), + None, + &mut response, + &noop_services(), + ) + .await; + + assert_eq!( + get_header(&response, "set-cookie").is_some(), + written, + "{case}, with a graph {with_graph}: the created Edge Cookie should be \ + written only when the request passed the bot gate" + ); + } + } + } + #[tokio::test] async fn finalize_rotates_orphaned_cookie_to_new_backed_ec() { let settings = create_test_settings(); @@ -1494,6 +1643,7 @@ mod tests { Jurisdiction::NonRegulated, true, ); + ec_context.set_device_signals(crate::ec::tests::browser_signals()); ec_context.set_kv_snapshot(graph.load_snapshot(&ec_id)); let partners = vec![make_partner("sharedid.org")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); @@ -2109,7 +2259,16 @@ mod tests { // the orphan's key, so nothing later treats the miss as authoritative. let settings = create_test_settings(); let hmac_orphan = sample_ec_id("orphn2"); - let cases: [(&str, EcContext, KvIdentityGraph); 3] = [ + let cases: [(&str, EcContext, KvIdentityGraph); 4] = [ + ( + "a client-side module, which creates nothing at the edge", + orphan_context( + "cfix~an-ec", + Some("192.0.2.10"), + std::sync::Arc::new(crate::ec::module::ClientFixedModule), + ), + KvIdentityGraph::in_memory("test_store"), + ), ( "the HMAC module with no client IP", orphan_context(&hmac_orphan, None, crate::ec::tests::hmac_module()), @@ -2278,6 +2437,7 @@ mod tests { Jurisdiction::NonRegulated, true, ); + ec_context.set_device_signals(crate::ec::tests::browser_signals()); ec_context.set_kv_snapshot(EcKvSnapshot::Missing { ec_id: generated_ec, }); @@ -2422,6 +2582,7 @@ mod tests { Jurisdiction::NonRegulated, gate_open, ); + ec_context.set_device_signals(crate::ec::tests::browser_signals()); ec_context.set_kv_snapshot(EcKvSnapshot::Present { ec_id: ec_id.clone(), entry: Box::new(live_entry()), @@ -2610,6 +2771,7 @@ mod tests { }; let mut ec_context = make_context_with_consent(None, None, false, false, consent, true) .with_module_for_test(std::sync::Arc::new(EvidenceHeaderModule)); + ec_context.set_device_signals(crate::ec::tests::browser_signals()); ec_context .generate_if_needed(&settings, Some(&graph), &noop_services()) .await diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index 0fa1c61cd8..27f011a9c1 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -1015,7 +1015,8 @@ impl EcContext { /// Sets the device signals derived from the adapter layer. /// /// Must be called before [`generate_if_needed`] so that new entries - /// include the [`KvDevice`] record. The adapter derives these from + /// include the [`KvDevice`] record, and before the Edge Cookie is + /// finalized, whose bot gate reads them. The adapter derives these from /// `req.get_tls_ja4()`, `req.get_client_h2_fingerprint()`, and UA. /// /// [`KvDevice`]: super::kv_types::KvDevice @@ -1038,6 +1039,17 @@ impl EcContext { self.device_signals.as_ref() } + /// Whether the request passed the bot gate, being whether its device + /// signals say it looks like a real browser. A created Edge Cookie is + /// written only for a request that passed, and a request with no device + /// signals has not. + #[must_use] + pub(crate) fn passed_bot_gate(&self) -> bool { + self.device_signals + .as_ref() + .is_some_and(|signals| signals.looks_like_browser) + } + /// Returns the normalized client IP, if available. #[must_use] pub fn client_ip(&self) -> Option<&str> { @@ -2406,9 +2418,9 @@ pub(crate) mod tests { } } - /// Runs generation through `module` and hands back the context whether or - /// not generation succeeded, so a test can finalize a response on a context - /// whose generation returned an error. + /// Runs generation through `module` for a browser and hands back the + /// context whether or not generation succeeded, so a test can finalize a + /// response on a context whose generation returned an error. async fn generate_with_header_setting_module( module: HeaderSettingModule, graph: Option<&KvIdentityGraph>, @@ -2421,6 +2433,7 @@ pub(crate) mod tests { let req = create_test_request(&[]); let mut ec = EcContext::read_from_request(&settings, &req, &services) .expect("should read EC context"); + ec.set_device_signals(browser_signals()); let outcome = ec.generate_if_needed(&settings, graph, &services).await; (settings, ec, outcome) } @@ -2606,6 +2619,7 @@ pub(crate) mod tests { let req = addressed_test_request(&[]); let mut ec = EcContext::read_from_request(&settings, &req, &services) .expect("should read EC context"); + ec.set_device_signals(browser_signals()); let graph = KvIdentityGraph::in_memory("test-ec-store"); ec.generate_if_needed(&settings, Some(&graph), &services) .await @@ -2967,6 +2981,44 @@ pub(crate) mod tests { )) } + /// The device signals of a desktop browser, which the bot gate passes. + /// + /// A created Edge Cookie is written only for a request whose device + /// signals pass the bot gate, so a test that expects one sets these, as + /// an adapter sets the signals it derives on every request. + pub(crate) fn browser_signals() -> DeviceSignals { + DeviceSignals::derive_ua_only( + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36", + ) + } + + /// The device signals of a crawler, which the bot gate stops. + pub(crate) fn crawler_signals() -> DeviceSignals { + DeviceSignals::derive_ua_only( + "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", + ) + } + + #[test] + fn the_bot_gate_passes_a_browser_and_stops_a_crawler_or_a_request_with_no_signals() { + let mut context = EcContext::new_for_test(None, ConsentContext::default()); + assert!( + !context.passed_bot_gate(), + "a request with no device signals has not passed the bot gate" + ); + context.set_device_signals(crawler_signals()); + assert!( + !context.passed_bot_gate(), + "the bot gate should stop a crawler" + ); + context.set_device_signals(browser_signals()); + assert!( + context.passed_bot_gate(), + "the bot gate should pass a browser" + ); + } + #[tokio::test] async fn generate_keys_the_identity_graph_by_the_normalized_identifier() { use crate::platform::test_support::noop_services_with_ec_module; diff --git a/crates/trusted-server-core/src/ec/resolve.rs b/crates/trusted-server-core/src/ec/resolve.rs index b54fcf4961..c237ab842a 100644 --- a/crates/trusted-server-core/src/ec/resolve.rs +++ b/crates/trusted-server-core/src/ec/resolve.rs @@ -42,23 +42,21 @@ const MAX_BODY_SIZE: usize = 64 * 1024; /// sets identity state, so a foreign page must not be able to drive it) and a /// `text/plain` or `application/json` body. Gates on the configured module's /// required permissions, then asks the module to create an Edge Cookie from -/// the posted payload. A created identifier is persisted to the identity graph -/// before the cookie is set, so withdrawal reaches a client-set identity the -/// same way it reaches an edge-created one. With no graph available this -/// endpoint sets no cookie at all, which is stricter than the organic -/// generation path, where the identifier is committed and only the row write -/// is skipped. On -/// success the EC cookie and its `non-HttpOnly` resolved marker are set and the -/// status is `200`. +/// the posted payload. With an identity graph, a created identifier is +/// persisted to it before the cookie is set, so withdrawal reaches a +/// client-set identity the same way it reaches an edge-created one. Without +/// one, the cookie is set with no row, as on the organic generation path, +/// because a missing graph does not stop the cookie. On success the EC cookie +/// and its `non-HttpOnly` resolved marker are set and the status is `200`. /// /// Rejections: `403` for a missing or foreign `Origin`, `415` for another /// content type, `413` for an oversized body, `400` when the module creates an /// identifier outside the identifier bounds, `409` when the request already /// carries a different identity (a resolve must not silently replace one), and /// `503` when the identity-graph write fails. When the permission gate is -/// closed, no module is configured, no graph is available, or the module -/// creates nothing, the response is `204` with no cookie. Every response this -/// handler builds carries `Cache-Control: no-store`; a module or +/// closed, no module is configured, the module creates nothing, or the request +/// did not pass the bot gate, the response is `204` with no cookie. Every +/// response this handler builds carries `Cache-Control: no-store`; a module or /// configuration error propagates to the adapter's error response instead, /// and so does a module asking for a response header inside core's reserved /// surface, which is a broken module contract rather than a bad request. @@ -207,30 +205,32 @@ pub async fn handle_ec_resolve( return Ok(status_only(StatusCode::CONFLICT)); } - // Persist the identity-graph row before setting the cookie, keyed by the - // module's canonical form, exactly like the organic generation path. - // Without a row, withdrawal could never reach this identity, so with no - // graph available this endpoint creates no cookie at all, which is stricter - // than the organic generation path, where the identifier is committed and - // only the row write is skipped. - let Some(graph) = kv else { - log::warn!("EC resolve skipped: no identity graph available, so no cookie is created"); + // The same bot gate the organic generation path writes a created cookie + // behind. + if !ec_context.passed_bot_gate() { + log::info!("EC resolve skipped: the request did not pass the bot gate"); return Ok(status_only(StatusCode::NO_CONTENT)); - }; - let now = super::current_timestamp(); - let mut entry = KvEntry::new( - ec_context.consent(), - ec_context.geo_info(), - now, - &settings.publisher.domain, - ); - entry.device = ec_context - .device_signals() - .map(super::device::DeviceSignals::to_kv_device); - let kv_key = super::module::module_kv_key(module.as_ref(), &ec_id); - if let Err(err) = graph.create_or_revive(&kv_key, &entry) { - log::error!("EC resolve failed to write the identity-graph row: {err:?}"); - return Ok(status_only(StatusCode::SERVICE_UNAVAILABLE)); + } + + // With an identity graph, persist the row before setting the cookie, + // keyed by the module's canonical form, exactly like the organic + // generation path, so withdrawal reaches this identity. + if let Some(graph) = kv { + let now = super::current_timestamp(); + let mut entry = KvEntry::new( + ec_context.consent(), + ec_context.geo_info(), + now, + &settings.publisher.domain, + ); + entry.device = ec_context + .device_signals() + .map(super::device::DeviceSignals::to_kv_device); + let kv_key = super::module::module_kv_key(module.as_ref(), &ec_id); + if let Err(err) = graph.create_or_revive(&kv_key, &entry) { + log::error!("EC resolve failed to write the identity-graph row: {err:?}"); + return Ok(status_only(StatusCode::SERVICE_UNAVAILABLE)); + } } let mut response = status_only(StatusCode::OK); @@ -478,13 +478,15 @@ mod tests { crate::ec::kv::KvIdentityGraph::in_memory("test-ec-store") } - /// A context whose gate is set by hand, carrying the module the settings - /// select, resolved as a request's own context resolves it. + /// A browser's context whose gate is set by hand, carrying the module the + /// settings select, resolved as a request's own context resolves it. fn gated(settings: &Settings, ec_allowed: bool) -> EcContext { - with_selected_module( + let mut context = with_selected_module( settings, EcContext::new_for_test_gated(None, ConsentContext::default(), ec_allowed), - ) + ); + context.set_device_signals(crate::ec::tests::browser_signals()); + context } fn with_selected_module(settings: &Settings, context: EcContext) -> EcContext { @@ -655,8 +657,8 @@ mod tests { resolve_request_with_header_module(headers, id, post(HEADER_MODULE_ID), graph).await } - /// Drives `request` through [`ResolveHeaderModule`], threaded the way a - /// composition root threads a module. + /// Drives a browser's `request` through [`ResolveHeaderModule`], threaded + /// the way a composition root threads a module. async fn resolve_request_with_header_module( headers: &'static [(&'static str, &'static str)], id: Option<&'static str>, @@ -671,8 +673,9 @@ mod tests { .uri("https://edge.example.com/") .body(EdgeBody::empty()) .expect("should build organic request"); - let ec = EcContext::read_from_request(&settings, &organic, &services) + let mut ec = EcContext::read_from_request(&settings, &organic, &services) .expect("should read EC context"); + ec.set_device_signals(crate::ec::tests::browser_signals()); handle_ec_resolve(&settings, request, &ec, graph, &services).await } @@ -856,6 +859,7 @@ mod tests { .expect("should build organic request"); let mut ec = EcContext::read_from_request(&settings, &organic, &services) .expect("should read EC context"); + ec.set_device_signals(crate::ec::tests::browser_signals()); assert!( ec.ec_value().is_none(), "no EC should exist on the first visit" @@ -998,29 +1002,19 @@ mod tests { async fn resolve_answers_204_with_no_cookie_when_it_creates_nothing() { let mut no_module = create_test_settings(); no_module.ec.module = None; - for (case, settings, body, allowed, has_graph) in [ + for (case, settings, body, allowed) in [ ( "the permission gate is closed", settings_with_client_fixed(), FIXED_WORD, false, - true, - ), - ("no module is configured", no_module, "123", true, true), - ( - "there is no identity graph to hold the row, so a cookie would be a \ - phantom identity", - settings_with_client_fixed(), - FIXED_WORD, - true, - false, ), + ("no module is configured", no_module, "123", true), ( "the posted value fails verification", settings_with_client_fixed(), "not-the-word", true, - true, ), ] { let graph = in_memory_graph(); @@ -1028,7 +1022,7 @@ mod tests { &settings, post(body), &gated(&settings, allowed), - has_graph.then_some(&graph), + Some(&graph), &noop_services(), ) .await @@ -1042,6 +1036,103 @@ mod tests { } } + /// The identity graph is one optional store of the data kept with the + /// Edge Cookie, so without one the endpoint answers as it does with one, + /// setting the cookie and its marker with no row behind them. + #[tokio::test] + async fn resolve_answers_as_it_does_with_a_graph_when_there_is_none() { + let settings = settings_with_client_fixed(); + let graph = in_memory_graph(); + let with_graph = handle_ec_resolve( + &settings, + post(FIXED_WORD), + &gated(&settings, true), + Some(&graph), + &noop_services(), + ) + .await + .expect("should handle resolve with a graph"); + + let response = handle_ec_resolve( + &settings, + post(FIXED_WORD), + &gated(&settings, true), + None, + &noop_services(), + ) + .await + .expect("should handle resolve without a graph"); + + assert_eq!( + response.status(), + StatusCode::OK, + "a verified value should return 200 without a graph" + ); + assert_eq!( + response.headers(), + with_graph.headers(), + "the Edge Cookie, its marker and no-store should be the ones a graph gets" + ); + assert!( + response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .any(|value| value + .to_str() + .is_ok_and(|cookie| cookie.starts_with("ts-ec=cfix~an-ec;"))), + "should set the coded verified word as the Edge Cookie" + ); + } + + /// A request the bot gate stops, being a crawler's or one with no device + /// signals, creates no Edge Cookie and no row, with a graph or without. + #[tokio::test] + async fn resolve_creates_nothing_for_a_request_that_did_not_pass_the_bot_gate() { + let settings = settings_with_client_fixed(); + for (case, signals) in [ + ("a crawler", Some(crate::ec::tests::crawler_signals())), + ("a request with no device signals", None), + ] { + for with_graph in [true, false] { + let mut ec = with_selected_module( + &settings, + EcContext::new_for_test_gated(None, ConsentContext::default(), true), + ); + if let Some(signals) = signals.clone() { + ec.set_device_signals(signals); + } + let graph = in_memory_graph(); + let response = handle_ec_resolve( + &settings, + post(FIXED_WORD), + &ec, + with_graph.then_some(&graph), + &noop_services(), + ) + .await + .expect("should handle resolve"); + + assert_eq!( + response.status(), + StatusCode::NO_CONTENT, + "{case}, with a graph {with_graph}" + ); + assert!( + response.headers().get(header::SET_COOKIE).is_none(), + "{case}, with a graph {with_graph}: should set no cookie" + ); + assert!( + graph + .get("cfix~an-ec") + .expect("should read the graph") + .is_none(), + "{case}, with a graph {with_graph}: should write no row" + ); + } + } + } + /// A location whose rules set storage without a signal, so the gate is /// open. fn open_gate_geo() -> crate::platform::GeoInfo { @@ -1071,8 +1162,10 @@ mod tests { .body(EdgeBody::empty()) .expect("should build organic request"); let geo = open_gate_geo(); - let ec = EcContext::read_from_request_with_geo(settings, &organic, &services, Some(&geo)) - .expect("should read EC context"); + let mut ec = + EcContext::read_from_request_with_geo(settings, &organic, &services, Some(&geo)) + .expect("should read EC context"); + ec.set_device_signals(crate::ec::tests::browser_signals()); assert!( ec.ec_allowed(), "the gate must be open for this to test anything" @@ -1137,13 +1230,14 @@ mod tests { .insert(crate::addressing::RemovedPathPrefix::new( "/publisher-a.example".to_owned(), )); - let ec = EcContext::read_from_request_with_geo( + let mut ec = EcContext::read_from_request_with_geo( &settings, &req, &services, Some(&open_gate_geo()), ) .expect("should read EC context"); + ec.set_device_signals(crate::ec::tests::browser_signals()); let response = handle_ec_resolve(&settings, req, &ec, Some(&in_memory_graph()), &services) .await @@ -1494,8 +1588,9 @@ mod tests { } /// The demo page script's post as the adapter handles it: the context is - /// read from the post, the handler answers, and the response is - /// finalized before the browser stores what it sets. + /// read from the post with a browser's device signals, the handler + /// answers, and the response is finalized before the browser stores what + /// it sets. async fn post_fixed_word( settings: &Settings, services: &RuntimeServices, @@ -1506,6 +1601,7 @@ mod tests { send_cookies(&mut req, jar); let mut ec = EcContext::read_from_request(settings, &req, services).expect("should read EC context"); + ec.set_device_signals(crate::ec::tests::browser_signals()); let mut response = handle_ec_resolve(settings, req, &ec, Some(graph), services) .await .expect("should handle resolve"); diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 1c7496e22e..f4e7466b7c 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -32680,6 +32680,7 @@ mod tests { }; let mut ec_context = EcContext::new_for_test(None, consent) .with_module_for_test(Arc::new(CanonicalizingModule)); + ec_context.set_device_signals(crate::ec::tests::browser_signals()); ec_context .generate_if_needed( &settings, diff --git a/crates/trusted-server-js/lib/src/integrations/ec_client_fixed/index.ts b/crates/trusted-server-js/lib/src/integrations/ec_client_fixed/index.ts index c8dee2ecbc..6845107e88 100644 --- a/crates/trusted-server-js/lib/src/integrations/ec_client_fixed/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/ec_client_fixed/index.ts @@ -3,8 +3,8 @@ // Client and server share one fixed, known word. When the resolved marker is // absent, this posts that word to the resolve endpoint. With the `client_fixed` // module selected, the server verifies the word and, on a match, persists the -// identity-graph row and sets the coded form of the word (cfix~an-ec) as an -// HttpOnly Edge Cookie on the +// identity-graph row where the deployment has an identity graph, and sets the +// coded form of the word (cfix~an-ec) as an HttpOnly Edge Cookie on the // response, together with a non-HttpOnly marker cookie. The Edge Cookie itself // is HttpOnly, so this script can never see it; the marker is what tells it a // resolve already succeeded, so it does not post again on every page view. diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md index 1c2fa98f2b..2a37930bc4 100644 --- a/docs/guide/api-reference.md +++ b/docs/guide/api-reference.md @@ -39,6 +39,7 @@ authority, scheme, and client-address headers in the innermost middleware. | normal | `/__ts/page-bids` | `GET` | literal | `always` | real | real | real | real | | normal | `/__ts/page-bids` | `OPTIONS` | literal | `always` | guarded | guarded | guarded | guarded | | normal | `/_ts/api/v1/batch-sync` | `POST` | literal | `always` | real | — | — | — | +| normal | `/_ts/api/v1/ec/resolve` | `POST` | literal | `always` | real | real | real | real | | normal | `/_ts/api/v1/identify` | `GET`, `OPTIONS` | literal | `always` | real | — | — | — | | normal | `/_ts/clear-tester` | `GET` | literal | `settings.tester_cookie.enabled` | real | — | — | — | | normal | `/_ts/config.json` | `GET` | literal | `always` | real | real | real | real | @@ -644,7 +645,7 @@ Resolve endpoint for client-side Edge Cookie modules. The page posts a value tha **Request Body:** the module's value, opaque to the core. For `client_fixed` this is the fixed known word sent as `text/plain`. -**Behavior:** gated by the [permission model](/guide/permission-model) exactly like organic generation. On success the identifier is written to the identity graph first, then the EC cookie is set on this response (`HttpOnly`, `Secure`, `SameSite=Lax`) together with the `ts-ecr` marker cookie the page script can read, and the status is `200`. When the gate is closed, no client-side module is configured, no identity graph is available, or the module produces no identifier, the response is `204` with no cookie. Rejections: `403` for a missing or foreign `Origin`, `415` for a content type other than `text/plain` or `application/json`, `413` for an oversized body, `400` when the created identifier is outside the identifier bounds, `409` when the request already carries a different identity, and `503` when the identity-graph write fails. Every response the handler builds carries `Cache-Control: no-store`. +**Behavior:** gated by the [permission model](/guide/permission-model) exactly like organic generation. The module is asked before the [bot gate](/guide/configuration#device-configuration) is checked, so a request that does not pass the bot gate still reaches the module and can get the `400` and `409` rejections below. On success the identifier is written to the identity graph first, where there is one, then the EC cookie is set on this response (`HttpOnly`, `Secure`, `SameSite=Lax`) together with the `ts-ecr` marker cookie the page script can read, and the status is `200`. When the permission gate is closed, the request does not pass the bot gate, no client-side module is configured, or the module produces no identifier, the response is `204` with no cookie. With no identity graph, as on the Axum, Cloudflare and Spin adapters and on a Fastly service with no `[ec] ec_store`, the cookie is set with no graph row. Rejections: `403` for a missing or foreign `Origin`, `415` for a content type other than `text/plain` or `application/json`, `413` for an oversized body, `400` when the created identifier is outside the identifier bounds, `409` when the request already carries a different identity, and `503` when the identity-graph write fails. Every response the handler builds carries `Cache-Control: no-store`. --- diff --git a/docs/guide/axum-dev.md b/docs/guide/axum-dev.md index 62b2d7c577..4636ac4422 100644 --- a/docs/guide/axum-dev.md +++ b/docs/guide/axum-dev.md @@ -73,7 +73,9 @@ Axum sanitizes forwarded client-IP headers but does not resolve [adapter matrix](./api-reference#adapter-and-startup-support). The modules' request filters and the device module `[device]` selects run as on Fastly, and `[device] module = "platform"` classifies from the `User-Agent`, because -the dev server supplies no TLS or HTTP/2 signals. +the dev server supplies no TLS or HTTP/2 signals. An Edge Cookie module +creates and writes its cookie with no identity graph behind it, and +`POST /_ts/api/v1/ec/resolve` sets one the same way. Axum sends no [auction telemetry](./telemetry#adapter-support), and warns at startup when `[analytics]` selects a module. diff --git a/docs/guide/cloudflare.md b/docs/guide/cloudflare.md index d83b918407..fabe89cb8f 100644 --- a/docs/guide/cloudflare.md +++ b/docs/guide/cloudflare.md @@ -81,8 +81,10 @@ is declared, and EC KV lookup is not available. Forwarded client-IP headers are sanitized, but `[trusted_client_ip]` is not resolved. The modules' request filters and the device module `[device]` selects run as on Fastly, and `[device] module = "platform"` classifies from -the `User-Agent`, because the Worker supplies no TLS or HTTP/2 signals. See -the [API reference](./api-reference) for route-level behavior. +the `User-Agent`, because the Worker supplies no TLS or HTTP/2 signals. An +Edge Cookie module creates and writes its cookie with no identity graph +behind it, and `POST /_ts/api/v1/ec/resolve` sets one the same way. +See the [API reference](./api-reference) for route-level behavior. The Worker sends no [auction telemetry](./telemetry#adapter-support), and warns at startup when `[analytics]` selects a module. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 803387b50d..05b4b5b40e 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -907,7 +907,7 @@ Legacy consent-store records are not read or migrated into `ec.ec_store`. Their | ------------------------- | -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `module` | String or null | No | Name of the active Edge Cookie module: `"hmac"` (built-in), `"host_signals"` (opt-in), `"none"` (explicitly stateless), or the name of a module a crate outside core supplies. Omit to run statelessly with no Edge Cookie. The `"client_fixed"` demonstration module needs the `client-fixed-demo` build feature | | `resolve_allowed_origins` | Array | No | Extra exact origins allowed to POST the client resolve endpoint, beyond `https://{publisher.domain}` | -| `ec_store` | String or null | No | Fastly KV store name for EC identity graph and withdrawal state | +| `ec_store` | String or null | No | Fastly KV store name for EC identity graph and withdrawal state. The Axum, Cloudflare and Spin adapters open no identity graph, and log that once at startup | | `pull_sync_concurrency` | Integer | No | Maximum concurrent pull-sync requests per organic response | | `cluster_trust_threshold` | Integer | No | Cluster size threshold for identity trust decisions | | `cluster_recheck_secs` | Integer | No | Legacy compatibility setting, because cluster rechecks no longer use timestamps | diff --git a/docs/guide/edge-cookies.md b/docs/guide/edge-cookies.md index c8f5b49a12..ce17904e4c 100644 --- a/docs/guide/edge-cookies.md +++ b/docs/guide/edge-cookies.md @@ -110,6 +110,8 @@ flowchart TD The `ec_identity_store` KV store is the only EC lifecycle store. It holds identity graph state, source-domain keyed partner UIDs, a minimal consent snapshot used for EC entry metadata, withdrawal tombstones, and completion markers that prevent stale point-read misses from rewriting completed tombstones. A marker key records the original tombstone's validity bound and is ignored at or after that time, even if its KV row has not expired yet. If the clock is unusable, the marker is ignored and withdrawal falls back to a strong root-existence check; only a confirmed existing root can be written. This prevents a stale marker from suppressing withdrawal when an expired EC key is created again. With a healthy store, repeated withdrawal of an already tombstoned EC ID leaves the row unchanged instead of refreshing its 24-hour TTL. Permission resolution for each request is based on the live request signals listed above. +The store is optional. The identity graph is one store of the data kept with the Edge Cookie, such as partner UIDs, so without it the Edge Cookie is still created and written, behind the same permission gate and [bot gate](/guide/configuration#device-configuration), and that data is not kept. A Fastly service with no `[ec] ec_store` therefore writes the Edge Cookie a real browser's page navigation creates, and so do the Axum, Cloudflare and Spin adapters, which open no identity graph. The identifier itself is used and shared as it is with the store, for example as the bid request's `user.id` where the storage and personalised-ad permissions are both set. Adding the store later gives each browser holding a cookie a server-side module wrote without it a new identifier on its first page view, because the identity graph replaces an identifier it has no row for. A client-side module creates nothing at the edge to replace its cookie with, so that cookie keeps its identifier with no row, and nothing is kept for that reader, for as long as the browser keeps the cookie and its `ts-ecr` marker. Each of that reader's page views meanwhile pays for an extra read of the store and a strongly consistent existence check. + ## Module Types: Server-Side and Client-Side The Edge Cookie identifier comes from a configurable module, selected by `[ec] module`. A module is one of two types, and the permission gate above applies to both. The two reach the **same outcome** (a `ts-ec` cookie set and carried on every later request) by **different routes**. @@ -146,7 +148,7 @@ The two types differ only in route and in the methods they use: | Endpoint | none | `POST /_ts/api/v1/ec/resolve` | | Cookie set on | the page response | the resolve response | -The resolve endpoint requires an `Origin` on the publisher's domain and a `text/plain` or `application/json` body, and it answers `409` rather than silently replacing an identity the request already carries. A created identifier is persisted to the identity graph before the cookie is set, so withdrawal reaches a client-set identity the same way it reaches an edge-created one. On success the cookie is set on the endpoint's own first-party `200` response, so the value is live for every subsequent request without a second navigation. The cookie is `HttpOnly`, so the page script never reads it back, and a non-`HttpOnly` marker cookie (`ts-ecr=1`, carrying no identity) tells the script a resolve succeeded so it does not post again on every page view. Every resolve response carries `Cache-Control: no-store`. The `client_fixed` demonstration module is compiled only behind the `client-fixed-demo` cargo feature, so a production build rejects selecting it at startup. +The resolve endpoint requires an `Origin` on the publisher's domain and a `text/plain` or `application/json` body, and it answers `409` rather than silently replacing an identity the request already carries. Where there is an identity graph, a created identifier is persisted to it before the cookie is set, so withdrawal reaches a client-set identity the same way it reaches an edge-created one. On success the cookie is set on the endpoint's own first-party `200` response, so the value is live for every subsequent request without a second navigation. The cookie is `HttpOnly`, so the page script never reads it back, and a non-`HttpOnly` marker cookie (`ts-ecr=1`, carrying no identity) tells the script a resolve succeeded so it does not post again on every page view. Every resolve response carries `Cache-Control: no-store`. Without an identity graph the endpoint sets the cookie with no graph row, as the page response does. The `client_fixed` demonstration module is compiled only behind the `client-fixed-demo` cargo feature, so a production build rejects selecting it at startup. Because the posted value comes from the browser, **verification is the module's responsibility**. A client-side module must verify the payload (for example a signature) before creating an identifier, or a client could forge an Edge Cookie. The endpoint itself is module-agnostic. It bounds the body, applies the same permission gate as organic generation, calls the module, and writes the cookie. diff --git a/docs/guide/fastly.md b/docs/guide/fastly.md index 936bd1fa0f..4860c60990 100644 --- a/docs/guide/fastly.md +++ b/docs/guide/fastly.md @@ -388,10 +388,12 @@ When upgrading an existing service: ## Create EC KV Store -Edge Cookie flows require one KV store: +The data kept with the Edge Cookie needs one KV store. Without it the Edge Cookie is still created and written, and that data is not kept: - Identity graph store (`ec_store`) - EC identity graph, source-domain keyed partner UIDs, minimal consent metadata, and withdrawal tombstones +Adding the store to a service that ran without it gives each browser holding a cookie that a server-side module, such as `hmac`, wrote without it a new identifier on its first page view, because the identity graph replaces an identifier it has no row for. A client-side module, such as `client_fixed`, creates nothing at the edge to replace its cookie with, so that cookie keeps its identifier with no row, and nothing is kept for that reader, for as long as the browser keeps the cookie and its `ts-ecr` marker. Each of that reader's page views meanwhile pays for an extra read of the store and a strongly consistent existence check. + Partners are configured statically in `[[ec.partners]]` and loaded into an in-memory registry at startup. There is no separate consent KV store. Consent is interpreted from live request cookies, headers, geolocation, and policy defaults. Create it: diff --git a/docs/guide/spin.md b/docs/guide/spin.md index 2ed894596d..6e69d51462 100644 --- a/docs/guide/spin.md +++ b/docs/guide/spin.md @@ -89,7 +89,8 @@ normalization layer. Spin has no geo lookup, so `[geo] module = "platform"` is refused at startup. The modules' request filters and the device module `[device]` selects run as on Fastly, and `[device] module = "platform"` classifies from the `User-Agent`, because Spin supplies no TLS or HTTP/2 -signals. +signals. An Edge Cookie module creates and writes its cookie with no identity +graph behind it, and `POST /_ts/api/v1/ec/resolve` sets one the same way. Spin sends no [auction telemetry](./telemetry#adapter-support), and warns at startup when `[analytics]` selects a module. diff --git a/docs/superpowers/specs/2026-10-17-edge-cookie-on-every-adapter-design.md b/docs/superpowers/specs/2026-10-17-edge-cookie-on-every-adapter-design.md new file mode 100644 index 0000000000..1573a42d9b --- /dev/null +++ b/docs/superpowers/specs/2026-10-17-edge-cookie-on-every-adapter-design.md @@ -0,0 +1,130 @@ +# Design Spec: The Edge Cookie is created and written on every adapter + +**Status:** Proposed, 2026-10-17. A pull request above the one that runs the request filters on every adapter, not yet raised, for Tech Lab review. +**Author:** 51Degrees (contributed), for Tech Lab review +**Related specs:** `2026-10-17-request-filters-on-every-adapter-design.md`, `2026-10-15-cookies-by-path-design.md`, `2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md`, and the design documents of the stack in IABTechLab/trusted-server#1342 +**Related PRs:** none yet +**Issues:** none +**Last updated:** 2026-10-18 + +> **Why this spec exists.** Only the Fastly adapter creates and writes the Edge Cookie and serves the resolve route. On Axum, Cloudflare and Spin an Edge Cookie module reads the cookie a request carries and the integration proxy creates one for a navigation to a module route, but no other page view creates one and no response writes one, so the stack's cookies-by-path change is never reached there. This change runs the Fastly adapter's Edge Cookie steps on those three adapters, which open no identity graph, and core writes a created Edge Cookie with or without a graph, so the cookie is written on every host, the graph is optional, and without one the graph's syncs are skipped. + +## 1. Problem + +The Fastly adapter runs the Edge Cookie lifecycle in four places (`crates/trusted-server-adapter-fastly/src/app.rs` and `src/lib.rs`). + +1. `build_ec_request_state` reads the request's Edge Cookie context with the device classification, takes the bot gate's answer from the classification (`DeviceSignals::looks_like_browser`), and keeps the identity graph, when `[ec] ec_store` names one, for a request that passes the gate. +2. `dispatch_fallback` creates an Edge Cookie for a real browser's page navigation (`EcContext::generate_if_needed`) before the publisher path runs. The integration proxy creates one for a navigation to a module route (`IntegrationRegistry::handle_proxy` in core). +3. Every response a route in the request cycle makes carries an `EcFinalizeState`, and the entry point runs `ec_finalize_response` on it once the response is finalized. That writes a created Edge Cookie, expires one on a withdrawal and reconciles the markers. +4. `POST /_ts/api/v1/ec/resolve` is a row of `NAMED_ROUTES`, answered by `handle_ec_resolve`. + +Axum, Cloudflare and Spin have only the integration proxy's creation in step 2, because each hands the request's Edge Cookie context to `IntegrationRegistry::handle_proxy` as Fastly does, so a navigation to a module route creates an identifier there that the module call can read and no response writes. They have none of the other steps. Spin's `named_fallback_paths` says the resolve route is left out because the identity graph is not wired there. + +Core writes a created Edge Cookie only with its identity-graph row. `ec_finalize_response` in `crates/trusted-server-core/src/ec/finalize.rs` skips the write when it is given no graph, so that on a deployment with a graph no browser carries an identifier the graph does not know. The three adapters open no graph, because the graph needs strongly consistent reads and conditional writes (`EcKvStore::key_exists` and `EcKvWriteMode::IfGenerationMatch` in `ec/kv_backend.rs`), which the platform key-value store these adapters can open, edgezero's `KvStore`, does not offer. It has no strongly consistent read and no conditional write, because its `exists` reads through `get_bytes` and `KvHandle::read_modify_write` is documented as not atomic (`edgezero_core::key_value_store`). Porting the Fastly steps alone would therefore create an Edge Cookie on these adapters and never write it. A Fastly service whose `[ec] ec_store` names no store writes none either, and the resolve route answers `204` with no cookie wherever it has no graph. + +## 2. Design + +**The Edge Cookie request state.** Each adapter's `build_ec_request_state` reads the context as before, with the device classification, and keeps the bot gate's answer beside it in an `EcRequestState`. A request the gate stops is logged with Fastly's message, without the JA4, platform and mobile fields Fastly adds. No identity graph is kept, because none is opened. + +**The bot gate on these adapters.** The gate is the answer of the device module `[device]` selects (`DeviceSignals::looks_like_browser`). On these adapters `platform` classifies from the `User-Agent` alone, as the built-in default does, so any client that sends a browser's `User-Agent` passes, where Fastly's `platform` module requires a TLS signal as well as a recognized platform (`DeviceSignals::derive`). Before this change that answer decided nothing about the Edge Cookie on these adapters, and now it decides whether a page view creates one, so a `[device]` module that answers `DeviceSignals::unknown()` stops every created Edge Cookie, there as on Fastly. + +**Creating the Edge Cookie.** `dispatch_fallback` calls `EcContext::generate_if_needed` for a real browser's page navigation (`is_navigation_request`) before the publisher path runs, with no graph, as the Fastly adapter does when it has none. The integration proxy creates one for a navigation to a module route as before, in core, for any client, because it does not read the bot gate, and finalizing writes it only for a request that passed. + +**Writing it.** Every response a route in the request cycle makes carries an `EcFinalizeState`, holding the context and the request's services. These are the responses the Fastly adapter attaches it to, being the named routes but the inspection endpoints, the fallback's tsjs, module route and publisher answers, and a filter's answer. robots.txt carries none, as on Fastly. `FinalizeResponseMiddleware` takes the state off the response once it has written the finalize headers and runs `ec_finalize_response` with no graph, which applies a module's response headers, writes an Edge Cookie the request created, expires the cookie and its markers on a withdrawal, and reconciles the pull-sync marker. The request filters' response effects are applied after it, as on Fastly. The `sharedId` cookie the Fastly adapter carries to finalizing is read only by the graph's identifier writes, so it is not carried. + +**Core writes the cookie on every host.** `ec_finalize_response` writes a created Edge Cookie for a request that passed the bot gate and whose permissions allow the Edge Cookie (`EcContext::ec_allowed`), with or without an identity graph. With one, it records the request's identifiers on the cookie's row and writes the cookie once that row is authoritative, as before. Without one, it skips the graph steps and writes the cookie through `set_ec_cookie_on_response`, which takes the request's public prefix and, on a request addressed by path, the removed prefix as its `Path`, as core writes every cookie it owns. So the invariant of `2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md` that a cookie is emitted only after its backing row exists now holds only where there is a graph, for the resolve route as for finalizing. Core reads the bot gate from the device signals the adapter sets on the context (`EcContext::passed_bot_gate`, which a request with no device signals does not pass), because a missing graph no longer stops the cookie and so can no longer stand in for the gate. The Fastly adapter still opens no graph for a request the gate stops, so no graph row is written for it. + +| Step | Fastly | Axum, Cloudflare and Spin | +| ------------------------------------ | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | +| Bot gate | `build_ec_request_state` | `build_ec_request_state` | +| Identity graph for the request | The `[ec] ec_store` graph, for a real browser | None | +| Creating on a page navigation | `dispatch_fallback`, for a real browser | `dispatch_fallback`, for a real browser | +| Finalizing | The entry point, after the finalize headers | `FinalizeResponseMiddleware`, after the finalize headers | +| Writing a created Edge Cookie | `ec_finalize_response`, once its graph row exists, or with no row where `ec_store` names no store | `ec_finalize_response`, with no row | +| Pull sync after the response is sent | When the graph and the partners allow it | Skipped | +| `POST /_ts/api/v1/ec/resolve` | A row of `NAMED_ROUTES` | A named route on each adapter, in the request cycle | + +**The resolve route.** Each adapter registers `POST /_ts/api/v1/ec/resolve` as a named route the way it registers its other named routes, so the request passes the preparers, the Edge Cookie context and the request filters, as on Fastly. It is answered by `handle_ec_resolve` with no graph, which answers as it does with one, so its refusals are the Fastly route's, being `403` for a missing or foreign `Origin` and `415` for another content type among them, and a client-side module's created Edge Cookie is set with its marker and no graph row. The bot gate is checked after the module has been asked and after the `400` and `409` answers, at the point where, before this change, the Fastly route found no graph for such a request, so a request that did not pass it still drives the module and can still get those answers, and otherwise gets `204` with no cookie, as on Fastly. + +**One startup warning.** Where `[ec] ec_store` names an identity graph store, each adapter logs once, as it builds its state, that it opens no identity graph, so the Edge Cookie is still written but the data the graph would keep with it, such as partner IDs, is not. Cloudflare and Spin build their state for every request, so a flag keeps the line to one for each instance. + +**Configuration.** No key is added. `[ec] ec_store` is accepted on these adapters as before, and logged once. `[device]` now decides through the bot gate whether a page view creates an Edge Cookie on these adapters, as described above. + +**Consent and privacy.** A created Edge Cookie is written only where core's permission gate allows the Edge Cookie module and the request passed the bot gate, on every host, with a graph or without. A withdrawal expires the cookie and its markers, as on Fastly, and there is no graph row to tombstone. Without a graph no partner identifier is written against the Edge Cookie, and no pull sync, batch sync or identify call reads one. + +The identifier itself leaves the edge without a graph as it does with one. Where the request's permissions set what the Edge Cookie module requires together with storage and personalized-ad selection (`EcContext::ec_sharing_allowed`, being `StoreOnDevice` and `SelectPersonalisedAds`), the publisher path's server-side auction and `GET /_ts/page-bids` send it to each auction provider as the auction request's `user.id`, and as `ts-` in that request's `id`, and `POST /auction` sends it as `user.id` (`publisher.rs`, `auction/endpoints.rs` and `auction/formats.rs` in core). A module call handed the request's state, such as a module route's handler or a document's serve-phase preparation, can read it behind the permissions the Edge Cookie module declares (`ModuleContext::with_request_state`), which for `hmac` is storage alone. That includes the identifier the integration proxy creates for a navigation to a module route by any client whose permissions allow the Edge Cookie, which the route's module call can read even when the bot gate stopped the request, though no response writes it, as on Fastly. The sample rules set storage and personalized-ad selection in the US with no signal. So this change discloses more in three cases. + +- On Axum, Cloudflare and Spin the publisher path created no identifier and no response wrote one, so an auction there carried a `user.id` only for a browser that already carried an Edge Cookie, which nothing on these adapters wrote. Now a server-side module, such as `hmac`, creates one on a real browser's first page navigation before the publisher path runs its auction, and the same value reaches the auction providers on that page view and on every later one whose permissions set both, for the year the cookie lasts. +- On a Fastly service with no `ec_store`, `hmac` created a new value on every page view, the HMAC of the client IP with a fresh random suffix, and no response wrote it, so the `user.id` of each page view was a new value whose 64-character HMAC part was the same for every page view from one address. Now one value follows the browser across page views and addresses for the year the cookie lasts. +- A client-side module, such as `client_fixed`, gave a browser no identifier on Axum, Cloudflare and Spin or on a Fastly service with no `ec_store`, because its `generate` creates nothing on a page navigation, those three adapters had no resolve route, and the Fastly route answered `204` with no graph. Now the resolve route sets the Edge Cookie with no graph for a browser that passes the bot gate where the permissions allow the module, and every later auction whose permissions set both sends that value to the auction providers as `user.id`, for the year the cookie lasts. + +This disclosure is intended. It is what a Fastly service with an `ec_store` does for the same browser, because the Edge Cookie is the identifier and the permission model, not the identity graph, decides whether it is shared. + +**Failure handling.** A failure to create the Edge Cookie is logged and the page is served without one, as on Fastly. A partner registry that cannot be built from `[ec] partners` skips the finalize step with a logged error, as on Fastly. + +**Rollout.** A deployment that selects no Edge Cookie module creates and writes no Edge Cookie, as before, but finalizing now runs on these adapters, so there a withdrawal expires an Edge Cookie or pull-sync marker the browser carries, as on Fastly. One that selects a server-side module, such as `hmac`, now sets the Edge Cookie on a real browser's first page navigation where the permissions allow it, on these adapters and on a Fastly service with no `ec_store`, and that response is kept out of shared caches because it sets a cookie (`enforce_set_cookie_cache_privacy`). A returning browser keeps its cookie while no identity graph sees it. A client-side module, such as `client_fixed`, gets its cookie from the resolve route on those hosts too. + +**A graph configured later replaces a server-side module's cookies.** When a Fastly service that ran with no `ec_store` names one, or a deployment on one of these adapters moves to a Fastly service with a graph, each browser holding a cookie written without a graph by a server-side module such as `hmac`, which creates its identifier at the edge, gets a new identifier on the first page navigation it makes there that passes the bot gate and whose permissions allow the Edge Cookie. The publisher path reads the cookie's row for that navigation, because no valid pull-sync marker lets it skip the read and a marker is only set from a row, and finalizing proves the row absent and gives the browser a new identifier from the module, with a row (`confirm_then_recover_orphaned_ec` in `ec/finalize.rs`). This design keeps that rather than have the graph adopt such a cookie by creating its row. An identifier a browser carries cannot be authenticated, because the `hmac` module's random suffix is unsigned, so the graph cannot tell a cookie written without a graph from one a client made up or one whose withdrawal tombstone, which lasts 24 hours, has expired, and adopting it would start a graph row from an unverified cookie, which `2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md` rules out. A deployer that adds a graph later therefore resets once the identifier of each reader holding such a cookie, and the `user.id` the auction providers see changes with it. + +**A client-side module's cookie keeps its identifier with no row.** A client-side module, such as `client_fixed`, creates nothing at the edge, so when finalizing proves such a cookie's row absent, recovery gets no identifier from the module, logs that and leaves the cookie as it is. Nothing else gives that cookie a row. Organic creation does nothing for a request that carries an identifier, a partner ID is not written against a missing row, and the page script posts to the resolve route, where such a module's rows are created, only while the resolved marker the route set beside the cookie, with the same one-year lifetime, is absent. So once a graph is configured, a cookie such a module got without one keeps its identifier with no row, the graph keeps nothing for that reader for as long as the browser keeps the cookie and its marker, and each of that reader's page navigations that passes the bot gate and whose permissions allow the module pays for a second point read and a strongly consistent existence check, a strong-mode list on Fastly, before recovery gives up again. Giving such a cookie its row would start a row from a cookie the graph cannot verify, the reason a graph does not adopt a server-side module's cookie, so it is listed under What is not in this change. + +**Crates outside core.** A crate outside core that finalizes the Edge Cookie or answers the resolve route sets the device signals on the context, or no created cookie is written. This is breaking for such a crate, and no compiler catches it. `generate_if_needed` does not read the bot gate, so such a crate that passes a graph and sets no device signals gets a graph row for an identifier no response writes. The adapters in this repository cannot reach that, because Fastly passes no graph for a request the gate stops and the others pass none. Orphan recovery reads whether the crate let the request recover (`EcContext::set_recovery_eligible`) rather than the bot gate, so such a crate that lets a request recover without its own gate having checked it gets a replacement cookie written with no device signals. In this repository only Fastly lets a request recover, and only for a real browser's publisher navigation. + +**Tests.** In `src/app/request_cycle_tests.rs` of each of the three adapters, through the adapter's router, with a stub origin and a reader whose services place them in the US and carry the permission signal modules the adapter links, where the sample permission rules grant the Edge Cookie with no signal, and `[geo] module = "platform"`, which leaves that location standing: + +- `a_first_page_view_gets_the_edge_cookie_under_the_public_prefix_and_path`: a page view addressed by path gets `ts-ec` with `Path=/publisher.example`, and the origin is asked at the continuing path. +- `a_returning_page_view_keeps_its_edge_cookie`: a page view addressed by host gets `ts-ec` under `/`, and the same page view carrying that cookie gets none. +- `a_bot_s_page_view_gets_no_edge_cookie`: a page view whose `User-Agent` names a crawler gets none. +- `a_withdrawal_expires_the_edge_cookie_the_browser_carries`: with no `[geo]` module instead, so the reader resolves at the top of the sample rules, where storage needs a signal, a page view carrying `ts-ec` and a TC String that refuses storage gets `ts-ec` expired with `Max-Age=0`. +- `the_resolve_route_answers`: a foreign `Origin` gets `403` with `Cache-Control: no-store` and the publisher's gets `204` with no cookie, the answer of a server-side module, which creates nothing from a posted value, and neither reaches the origin. +- `the_resolve_route_sets_a_client_side_module_s_edge_cookie`: with a test module selected by `[ec] module` that creates nothing at the edge and takes its identifier from the posted value, as a client-side module does, the publisher's post of that value gets `200` with `ts-ec`, its `ts-ecr` marker and `Cache-Control: no-store`, and does not reach the origin. +- `startup_warns_that_this_adapter_opens_no_identity_graph`: building the state with `[ec] ec_store` set sets the flag the warning is logged under, and the warning's check with no `ec_store` sets none. The flag is set as the line is logged, so the test shows the warning's branch runs, not that a line reaches the log. + +The tests' `edge_cookie` helper also fails a response that writes the Edge Cookie more than once, so the first page view shows core alone writes it. + +In core: + +- `finalize_generated_ec_without_kv_writes_the_cookie_and_skips_the_graph` in `ec/finalize.rs`: a created Edge Cookie finalized with no graph gets the headers the same cookie gets with one, and no graph step runs, while with a graph the request's partner ID is recorded on the row. +- `a_generated_ec_is_written_only_for_a_request_that_passed_the_bot_gate` in `ec/finalize.rs`: a browser's created Edge Cookie is written, and a crawler's or that of a request with no device signals is not, with a graph and without. +- `resolve_answers_as_it_does_with_a_graph_when_there_is_none` in `ec/resolve.rs`: with no graph the route answers `200` with the Edge Cookie, its marker and `Cache-Control: no-store`, the headers it gives with a graph. +- `resolve_creates_nothing_for_a_request_that_did_not_pass_the_bot_gate` in `ec/resolve.rs`: a crawler, or a request with no device signals, gets `204` with no cookie and no row, with a graph and without. +- `the_bot_gate_passes_a_browser_and_stops_a_crawler_or_a_request_with_no_signals` in `ec/mod.rs`. +- `finalize_rotates_orphaned_cookie_to_new_backed_ec` in `ec/finalize.rs`, which this change leaves as it is: a well-formed cookie whose row is missing, finalized with a graph on a page view that may recover, gets a new identifier with a row, which is the replacement Rollout describes. +- `an_orphan_recovery_that_cannot_rotate_leaves_a_failed_snapshot` in `ec/finalize.rs`, which gains a client-side module's case: a `client_fixed` cookie whose row is missing, finalized with a graph on a page view that may recover, keeps its identifier with no cookie written and a failed snapshot for its key, which is the case Rollout describes for such a module. + +In the Fastly adapter, under Viceroy, `with_no_ec_store_the_entry_point_writes_a_created_edge_cookie` in `src/app.rs`: with no `[ec] ec_store`, `build_ec_request_state` opens no graph for a browser's page view, `generate_if_needed`, the step `dispatch_fallback` runs for one, creates an Edge Cookie, and `apply_edgezero_ec_finalize`, the entry point's finalize step, writes it once. The test calls the three steps in turn rather than routing a page view through `dispatch_fallback` and `edgezero_main`. + +The adapters' test of a `200` from the resolve route supplies its own client-side module through a registration, as a crate outside core would, because the `client_fixed` module is compiled only behind core's `client-fixed-demo` feature, which no adapter passes on. + +**Decisions.** Two questions this design left open, whether the resolve route should write a client-side module's cookie on an adapter that opens no graph and whether a Fastly service with no `ec_store` should write a created Edge Cookie, are both answered yes. The Edge Cookie is the identifier, and the identity graph is one store of the data associated with it. The permission state comes from the permission model, whose sources may be values the graph holds, signals the client sends or any other source, so a missing graph does not stop the cookie. + +So the Edge Cookie is the identifier, the identity graph is one optional store of the data associated with it, the partner IDs among it, and the permission state comes from the permission model whatever its sources. A missing graph does not stop the cookie. + +Where a graph is configured it still replaces a server-side module's identifier it has no row for, and leaves a client-side module's identifier with no row, as Rollout describes. Both are the graph refusing to start a row from an identifier it cannot verify, not a condition on the cookie, and without a graph nothing replaces a cookie. + +## 3. What is not in this change + +- An identity graph on Axum, Cloudflare or Spin, which needs a strongly consistent store with conditional writes. +- `/_ts/api/v1/identify`, `/_ts/api/v1/batch-sync` and pull sync on those adapters, which need the graph, and the tester routes, which no other adapter registers. +- Asset routes on these adapters, which the next pull request of the stack covers. +- A graph adopting an Edge Cookie it has no row for by creating the row instead of replacing the identifier, which needs a way to tell a cookie written without a graph from one a client made up or one whose withdrawal tombstone has expired, such as a signed identifier. +- Giving a client-side module's cookie written without a graph its row once a graph is configured, where recovery has no identifier to replace it with. It needs the same way to tell such a cookie from one a client made up or one whose withdrawal tombstone has expired, because the row would otherwise start from an unverified cookie. +- A bot gate in `generate_if_needed`, which would keep a crate outside core that passes a graph and sets no device signals from writing a row for an identifier no response writes. + +## 4. Acceptance + +The seven tests pass on each of the three adapters, the core and Fastly tests above pass, and the existing tests of each adapter, core's Edge Cookie tests and the cross-adapter parity tests pass. + +## Revision record + +| Date | Change | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-10-17 | First draft. | +| 2026-10-18 | Core writes a created Edge Cookie with or without an identity graph, behind the bot gate it reads from the device signals, and the resolve route answers without a graph as it does with one, which answers both open questions. | +| 2026-10-18 | Fixes from review round 1. The identifier's disclosure to auction providers and module calls with no graph and the two cases where it grows, the replacement of these cookies once a graph sees them and why a graph does not adopt one, the integration proxy's creation the three adapters already ran, the bot gate on these adapters, the order of the resolve route's checks, the store's methods, the log line, a deployment with no module, crates outside core, a test of a withdrawal through each router, the startup warning's case with no `ec_store`, what the Fastly test does, the answer no adapter test covers, what is not in this change and the review log. | +| 2026-10-18 | Fixes from review round 2. A client-side module's cookie, which a graph configured later leaves with no row, and what that costs, the replacement limited to a server-side module's cookie, the third case where the disclosure grows, the 2026-07-10 invariant that now holds only with a graph, orphan recovery for crates outside core, a test of a `200` from the resolve route through each adapter's router, a client-side module's case in core's orphan recovery test, what is not in this change and the review log. | + +## Review log + +1. **Round 1, 2026-10-18.** A review in a fresh session, against `split/53-edge-cookie` at `198ce56db`. Issues found, two of them blocking. The two were that the spec said the Edge Cookie stays a first-party identifier in the browser without a graph, when core sends it to the auction providers as `user.id` and in the auction request's `id` and lets module calls read it, so this change gives the providers one value for a year where they had none or a new one on each page view, and that the spec said a returning browser keeps its cookie without saying that a graph configured later replaces every cookie written without one, which Decisions also left unexplained. Fixed in `960d9c61f`, which corrects the changelog and the Edge Cookie and Fastly guides, in `dbbbcfa14`, which tests a withdrawal through each adapter's router and the startup warning with no `ec_store`, and in `3c9e9f913`, which states the disclosure and why it is intended, states the replacement and why the graph keeps it rather than adopt such a cookie, and fixes the non-blocking findings that were cheap and certain. A graph that adopts such a cookie, a bot gate in `generate_if_needed` and a `200` from the resolve route in the adapters' tests are listed under What is not in this change. +2. **Round 2, 2026-10-18.** A review in a fresh session, against `split/53-edge-cookie` at `3c9e9f913`, of the round 1 fixes and the new text. Issues found, two of them blocking. The two were that the spec, the changelog and both guides said a graph configured later replaces each cookie written without one, when a client-side module creates nothing at the edge, so recovery leaves its cookie with no row and each of that reader's page navigations pays for a second read and an existence check, and that the spec listed two cases where the disclosure grows, leaving out a client-side module on a host with no graph, whose readers now get an identifier from the resolve route where they got none. Fixed in `fa4181b19`, which corrects the changelog, the Edge Cookie, Fastly and API reference guides and the page script's comment, in `fd7227aa1`, which tests a `200` from the resolve route through each adapter's router and a client-side module's orphaned cookie in core, and in this revision, which limits the replacement to a server-side module, states the client-side case and its cost, adds the third case, and fixes the non-blocking findings that were cheap and certain. Giving a client-side module's cookie its row is listed under What is not in this change, beside a graph that adopts a cookie.