diff --git a/CHANGELOG.md b/CHANGELOG.md index f691c6a005..a7fc880408 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Consent decoding for identity providers is stricter: a raw consent string that cannot be decoded blocks a provider whose policy is strict, and the GPP sharing, targeted-advertising and GPC fields of the US sections now block it too. A header-only GPP string with no applicable section does not block where TCF does not apply. +- The Lockr proxy now logs only the path of the upstream URL, never its query string. +- Identity KV is written at most once per EC-bearing request. Only a full page navigation from a returning visitor (the writer request) writes KV, once, after the response is sent: browser EIDs, pull partner results and the cluster size all go in that one write, with a generation check and no retry of the whole write. Every other route reads KV and stages but writes nothing. A consent-withdrawal tombstone and the removal-only retry of a failed write are the only exceptions. Mid-branch builds drop browser ingest until the whole change ships. +- `POST /auction`, `GET /_ts/page-bids` and `GET /_ts/api/v1/identify` no longer write KV, and `/_ts/api/v1/identify` omits expired IDs. The `/identify` response shape is unchanged: one UID per source. +- Pull partners now run from a visitor's second page load, from the writer request after send, and back off on errors. They are paced by `[ec.identity.providers.pull-]` (`refresh_after_sec`, `max_calls_per_minute`), so the old default of 10 calls per hour becomes an effective 60 per hour. That 60 per hour is a ceiling, not a rate: `refresh_after_sec` and error backoff gate each partner first. Runtime startup now builds each pull partner's descriptor: a partner whose allowed host fails the descriptor host check stops startup. See [Identity Providers](docs/guide/configuration.md#identity-providers-ec-identity). +- **Breaking:** `ec.pull_sync_concurrency`, and `pull_sync_ttl_sec` and `pull_sync_rate_limit` on `[[ec.partners]]`, are removed, and validation now rejects them with an error that names the replacement. Use `[ec.identity].max_concurrency` (the example config's value moves from 3 to the default 4), `refresh_after_sec`, and `max_calls_per_minute = ceil(N / 60)` for `pull_sync_rate_limit = N`, the last two in `[ec.identity.providers.]`. +- Identity providers that resolve must now declare the TCF purposes their GVL entry lists (`ConsentPolicy::with_tcf_purposes`) and their GVL ID (`with_tcf_vendor`), or opt out of vendor consent with `without_tcf_vendor`; `ConsentPolicy::strict()` alone now requires only Purpose 1, and `build()` rejects an undeclared resolver. +- A new visitor's first navigation that runs an auction now runs hold-mode identity providers that do not read the EC ID, and creates its row with their answers in the one create write, so a visitor who leaves after one page can still get a provider ID. It waits up to `hold_budget_ms`; without such a provider the create runs before routing as before. +- A same-origin `/_ts/page-bids` call from a real browser now writes browser ingest and after-send provider results, at most once per `ec.identity.spa_writer_interval_sec` (default 300, `0` turns it off), so single-page apps do not wait for a full navigation. +- A removal whose KV writes all fail now waits in a `pending:` KV key (one-day TTL) for the next writer navigation to merge, counted in `removal_deferred`; `removal_lost` counts only a removal that could not reach that key either. Positive IDs a capture sees on a request that is not the writer wait there too, so a capture's full entry set reaches KV. Every returning writer navigation reads that key once, a second KV read. +- A global revoke (`revoke_scope = "global"`) now also sets an `HttpOnly` `ts-ec-revoked` cookie with a fingerprint of the visitor's consent, and no EC is minted until that consent changes, so a CMP that still grants does not undo the revoke on the next page. +- `max_calls_per_ip_per_minute` (default 10) on each `[ec.identity.providers.]` caps calls to that provider from one client address across visitors, next to the per-visitor cap. +- `[ec.identity] max_backend_requests` (default 32, Fastly's per-execution limit) bounds the origin, auction and identity provider calls of one request together. A provider call past it is skipped and is not counted as a failure, so it never opens the provider's circuit breaker. Hold-mode calls also cap their connect timeout at their call timeout. +- A provider opt-out clears on a TCF re-grant only when the opt-out was recorded under a TCF denial that is now granted; a later CMP re-save that was already granting no longer clears it. A clear records the opt-out it lifted rather than its own time, so it cannot lift an opt-out written after its request read the row. +- `Resolves::with_refresh_grace` lets an identity provider without private state keep an expired ID it wrote for a source it owns as a refresh anchor until the expiry plus the grace. The ID is never served meanwhile. +- With `[ec.identity] mirror_enabled = true` (off by default), `/auction` responses carry `ext.ts_eids_s` and `/_ts/page-bids` responses carry `ts_eids_s`: the payload (`value` with `max_age`, `clear`, or absent) for the client-only `__Host-ts-eids-s` mirror cookie, at most 2,048 bytes and one day. The shim now writes only `[[ec.partners]]` sources into `ts-eids`, merging each auction's entries per source instead of replacing the cookie, and no longer clears it when a request carries no EIDs (a TCF block still clears it). +- The `ts-ec-pull-complete` cookie is removed: Trusted Server no longer sets or reads it, and a copy a browser holds expires within one hour. +- **Breaking:** The KV identity row moves to schema version 2. A release before this one cannot read a version 2 row, so rolling back past this release is unsupported once it has written one. See [Rolling back](docs/guide/configuration.md#rolling-back). This release should merge after the EID-fidelity changes, whose behaviour its golden tests pin. +- The `EID sync measurement: ...` log line is no longer logged for returning visitors; only a new EC still logs it. Each request now logs one `identity request measurement: ...` line after the response is sent, with the inbound `Cookie` header size, KV reads and writes by route kind, the flush outcome, lost removals and the after-send withdrawal run. It holds no identifiers. Operator dashboards that read the old line should read the new one. +- **Breaking:** Prebid Server no longer receives any Trusted Server identity cookie (`ts-ec`, `ts-eids`, and the `__Host-ts-eids-s` mirror that this release's shim writes); today it receives `ts-ec` and `ts-eids`. The publisher origin still receives `ts-ec` and `ts-eids` but never the mirror. `[ec] strip_eids_cookie_from_origin = true` strips `ts-eids` from origin requests too. - **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER__GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header. - **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias. - The S2S `/_ts/api/v1/batch-sync` endpoint now validates the full batch and calls the CAS-protected update path once per distinct normalized EC ID. The last valid UID wins within a group, and infrastructure failures reject the failing and each unprocessed group, so accepted and `kv_unavailable` input indexes may interleave. @@ -44,6 +64,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Server-side identity provider framework. An integration registers an `IdentityProvider` with `try_with_identity_provider`, and the coordinator handles consent, transport limits, backoff and the single KV write. Provider authors get a test harness (`ProviderHarness`, `check_conformance`, `output_eids`). See [Writing an Identity Provider](docs/guide/identity-providers.md). No vendor ships with the framework. +- Source ownership: `owner` on an `[[ec.partners]]` entry names the provider that owns a source. An owner's result replaces the source's whole entry set, and batch sync writes an owned source only when the row holds no value for it (an expired owned record counts as present and is not filled). Validation rejects an owner that is not enabled, is capture-only, does not claim the source, or collides with another provider. See [Source ownership](docs/guide/configuration.md#source-ownership-ec-partners-owner). +- Hold mode: `mode = "hold"` in `[ec.identity.providers.]` runs a provider before the navigation's auction, within the shared `hold_budget_ms`, with `timeout_ms` and `after_send_timeout_ms` as call timeouts. +- Revoke-only passive capture for providers that proxy a vendor's API: a captured removal or opt-out is written to KV, a positive result clears an opt-out, and positive IDs are only counted. `revoke_scope` (`provider` by default, or `global` to run the consent-withdrawal tombstone) sets how far an opt-out reaches. Removals and opt-outs apply from any client, as consent withdrawals do; only a clear needs a real browser. A capture-only provider must list its `sources`. See the [capture provider checklist](docs/guide/edge-cookies.md#capture-provider-checklist). +- Complete provider EIDs: a provider can return several entries and UIDs per source, with UID-level `atype` and `ext` and EID-level `inserter`, `matcher`, `mm` and `ext`. KV, the bidstream, `x-ts-eids` and the `ts-eids-s` mirror carry the whole set in order; `/identify` still returns one UID per source. See [Complete provider EIDs](docs/guide/edge-cookies.md#complete-provider-eids). +- `/_ts/api/v1/identify` returns an ID a server-side provider wrote only to a browser call from the publisher's site (an allowed `Origin` and `Sec-Fetch-Site: same-origin` or `same-site`), because the EC ID is a bearer value partners and the origin hold. Other IDs are returned as before. +- The `capture_unpersisted` and `removal_lost` counters in the `identity request measurement: ...` log line, and a per-provider circuit breaker (`breaker_failures_per_minute`) that skips a provider for every visitor after repeated transport failures, timeouts, 429s or 5xx responses. + - Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery. - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary. - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape. diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index caba714d68..76300c48b0 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -18,6 +18,7 @@ use trusted_server_core::ec::EcContext; use trusted_server_core::ec::admin::{ admin_ec_lookup_not_supported, deny_admin_diagnostic_fallback, handle_admin_eids_lookup, }; +use trusted_server_core::ec::finalize::BrowserSeed; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; @@ -232,6 +233,8 @@ async fn dispatch_fallback( kv: None, ec_context: &mut ec_context, services, + is_real_browser: false, + seed: &BrowserSeed::empty(), req, }) .await @@ -325,7 +328,7 @@ const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ Method::DELETE, ]; -fn named_routes() -> [NamedRoute; 17] { +fn named_routes() -> [NamedRoute; 18] { [ NamedRoute { path: "/.well-known/trusted-server.json", @@ -371,6 +374,13 @@ fn named_routes() -> [NamedRoute; 17] { primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcNotSupported, }, + // The provider opt-out clear writes the EC identity graph, so it + // answers like the lookup: not available on this adapter. + NamedRoute { + path: "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + primary_methods: &[Method::DELETE], + handler: NamedRouteHandler::AdminEcNotSupported, + }, // Admin EIDs echo: pure request inspection (no KV), so the dev // server serves the real handler. NamedRoute { diff --git a/crates/trusted-server-adapter-axum/src/platform.rs b/crates/trusted-server-adapter-axum/src/platform.rs index e2713f47f2..db20b898f9 100644 --- a/crates/trusted-server-adapter-axum/src/platform.rs +++ b/crates/trusted-server-adapter-axum/src/platform.rs @@ -666,6 +666,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_secs(15), between_bytes_timeout: Duration::from_secs(15), + connect_timeout: None, host_header_override: None, discriminator: None, }; @@ -687,6 +688,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_secs(15), between_bytes_timeout: Duration::from_secs(15), + connect_timeout: None, host_header_override: None, discriminator: None, }; diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index bcb31b7e44..fa445e5a2d 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -114,6 +114,7 @@ fn all_explicit_routes_are_registered() { ("POST", "/_ts/admin/keys/deactivate"), ("GET", "/_ts/admin/ec"), ("GET", "/_ts/admin/ec/{id}"), + ("DELETE", "/_ts/admin/ec/{id}/providers/{provider}/opt-out"), ("GET", "/_ts/admin/eids"), ("POST", "/admin/keys/rotate"), ("POST", "/admin/keys/deactivate"), @@ -388,6 +389,55 @@ async fn authenticated_admin_ec_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_opt_out_clear_route_returns_501() { + // The clear writes the EC identity graph, which is Fastly KV backed, so + // the Axum dev server answers the DELETE locally with 501 like the EC + // lookup. + let path = format!( + "/_ts/admin/ec/{}.abc123/providers/pull-ssp-example-com/opt-out", + "a".repeat(64) + ); + let send = |method: &'static str| { + let path = path.clone(); + async move { + let req = Request::builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(AxumBody::empty()) + .expect("should build request"); + make_service() + .ready() + .await + .expect("should be ready") + .call(req) + .await + .expect("should respond") + } + }; + + let deleted = send("DELETE").await; + assert_eq!( + deleted.status().as_u16(), + 501, + "DELETE {path} should report that the opt-out clear is unsupported" + ); + assert_eq!( + deleted + .headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json"), + "should answer with the same JSON as the EC lookup" + ); + assert_eq!( + send("GET").await.status().as_u16(), + 404, + "control: GET {path} should be denied locally as an unknown admin route" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn admin_ec_route_without_credentials_returns_401() { let mut svc = make_service(); diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index 87b9567e7f..5e3571c289 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -22,6 +22,7 @@ use trusted_server_core::ec::admin::{ admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, deny_admin_diagnostic_fallback, handle_admin_eids_lookup, }; +use trusted_server_core::ec::finalize::BrowserSeed; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; @@ -544,6 +545,8 @@ fn build_router(state: &Arc) -> RouterService { kv: None, ec_context: &mut ec_context, services: &services, + is_real_browser: false, + seed: &BrowserSeed::empty(), req, }) .await @@ -643,6 +646,14 @@ fn build_router(state: &Arc) -> RouterService { .get("/_ts/admin/ec/{id}", |_ctx: RequestContext| async { Ok::(admin_ec_lookup_not_supported()) }) + // The provider opt-out clear writes the EC identity graph, so it + // answers like the lookup. + .delete( + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + |_ctx: RequestContext| async { + Ok::(admin_ec_lookup_not_supported()) + }, + ) // Admin EIDs echo: pure request inspection (no KV), so this // adapter serves the real handler. .get( diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index f385bf5295..b4ba6a457c 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -254,6 +254,7 @@ fn all_explicit_routes_are_registered() { ("POST", "/_ts/admin/keys/deactivate"), ("GET", "/_ts/admin/ec"), ("GET", "/_ts/admin/ec/{id}"), + ("DELETE", "/_ts/admin/ec/{id}/providers/{provider}/opt-out"), ("GET", "/_ts/admin/eids"), ("POST", "/auction"), // SPA re-auction endpoint, plus its deprecated `/__ts/` alias. Both @@ -353,6 +354,49 @@ async fn authenticated_admin_ec_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_opt_out_clear_route_returns_501() { + // The clear writes the EC identity graph, which is Fastly KV backed, so + // Cloudflare answers the DELETE locally with 501 like the EC lookup. + let path = format!( + "/_ts/admin/ec/{}.abc123/providers/pull-ssp-example-com/opt-out", + "a".repeat(64) + ); + let delete = request_builder() + .method("DELETE") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), delete).await; + + assert_eq!( + resp.status().as_u16(), + 501, + "DELETE {path} should report that the opt-out clear is unsupported" + ); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json"), + "should answer with the same JSON as the EC lookup" + ); + + let get = request_builder() + .method("GET") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), get).await; + assert_eq!( + resp.status().as_u16(), + 404, + "control: GET {path} should be denied locally as an unknown admin route" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn admin_ec_route_without_credentials_returns_401() { let req = request_builder() diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index bf09c2086e..7c664946bd 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -24,6 +24,7 @@ //! | POST | `/_ts/admin/keys/deactivate` | [`handle_deactivate_key`] | //! | GET | `/_ts/admin/ec` | [`handle_admin_ec_lookup`] | //! | GET | `/_ts/admin/ec/{id}` | [`handle_admin_ec_lookup`] | +//! | DELETE | `/_ts/admin/ec/{id}/providers/{provider}/opt-out` | [`handle_admin_opt_out_clear`] | //! | GET | `/_ts/admin/eids` | [`handle_admin_eids_lookup`] | //! | POST | `/_ts/api/v1/batch-sync` | [`handle_batch_sync`] | //! | GET | `/_ts/api/v1/identify` | [`handle_identify`] | @@ -108,11 +109,14 @@ use trusted_server_core::config_payload::DEFAULT_SECRET_STORE_ID; use trusted_server_core::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; use trusted_server_core::ec::admin::{ deny_admin_diagnostic_fallback, handle_admin_ec_lookup, handle_admin_eids_lookup, + handle_admin_opt_out_clear, }; use trusted_server_core::ec::batch_sync::handle_batch_sync; use trusted_server_core::ec::consent::ec_consent_withdrawn; use trusted_server_core::ec::device::DeviceSignals; +use trusted_server_core::ec::finalize::{BrowserSeed, browser_seed}; use trusted_server_core::ec::identify::{cors_preflight_identify, handle_identify}; +use trusted_server_core::ec::identity::IdentityRuntime; use trusted_server_core::ec::kv::KvIdentityGraph; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::ec::{EcContext, EidSyncSource}; @@ -123,7 +127,8 @@ use trusted_server_core::integrations::{ RequestFilterRegistryOutcome, }; use trusted_server_core::platform::{ - ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices, StoreName, + ClientInfo, GeoInfo, PlatformGeo, PlatformHttpClient, PlatformKvStore, RuntimeServices, + StoreName, }; use trusted_server_core::proxy::{ AssetProxyCachePolicy, handle_asset_proxy_request, handle_first_party_click, @@ -189,6 +194,18 @@ pub(crate) struct AppState { pub(crate) registry: Arc, pub(crate) default_kv_store: Arc, pub(crate) auction_telemetry_sink: Arc, + /// Identity providers, keys and policy, built once per settings load. + pub(crate) identity: Arc, + /// The identity graph over `ec.ec_store`, built once. `None` when no + /// store is configured. + pub(crate) identity_graph: Option, + /// Replaces the Fastly HTTP client in tests, so a request reaches a + /// scripted origin and bidder instead of the network. + #[cfg(test)] + pub(crate) http_client_for_test: Option>, + /// Replaces the Fastly geolocation in tests, which Viceroy does not serve. + #[cfg(test)] + pub(crate) geo_for_test: Option>, } /// Build the application state, loading settings and constructing all per-application components. @@ -215,6 +232,49 @@ pub(crate) fn load_settings_from_config_store( ) } +/// Builds the identity runtime for one settings load. +/// +/// # Errors +/// +/// Returns an error when a provider cannot be described or the catalog rules fail. +pub(crate) fn build_identity_runtime( + settings: &Settings, + integrations: &IntegrationRegistry, +) -> Result, Report> { + IdentityRuntime::from_settings(settings, integrations).map(Arc::new) +} + +/// Builds the identity runtime for settings loaded without application +/// state, from the integration registry those settings describe. +/// +/// A plan or registry failure is logged and the runtime is built with no +/// registered providers, so generic pull providers still run. +/// +/// # Errors +/// +/// Returns an error when the identity runtime cannot be built. +pub(crate) fn build_identity_runtime_from_reloaded_settings( + settings: &Settings, +) -> Result, Report> { + let registry = compile_auction_plan(settings) + .and_then(|plan| IntegrationRegistry::with_plan(settings, Arc::new(plan))); + identity_runtime_from_registry_result(settings, registry) +} + +fn identity_runtime_from_registry_result( + settings: &Settings, + registry: Result>, +) -> Result, Report> { + let registry = registry.unwrap_or_else(|error| { + log::warn!( + "identity runtime built without registered providers: integration registry unavailable after settings reload: {}", + error.current_context() + ); + IntegrationRegistry::default() + }); + build_identity_runtime(settings, ®istry) +} + pub(crate) fn build_state_from_settings( settings: Settings, ) -> Result, Report> { @@ -224,19 +284,71 @@ pub(crate) fn build_state_from_settings( plan.validate_for_target(trusted_server_core::platform::AuctionTargetId::Fastly)?; let orchestrator = build_orchestrator_with_plan(Arc::clone(&plan), &settings)?; let registry = IntegrationRegistry::with_plan(&settings, plan)?; + let identity = build_identity_runtime(&settings, ®istry)?; let auction_telemetry_sink = crate::tinybird::auction_sink_from_settings(&settings); let default_kv_store = Arc::new(UnavailableKvStore) as Arc; + let identity_graph = crate::maybe_identity_graph(&settings); Ok(Arc::new(AppState { settings: Arc::new(settings), orchestrator: Arc::new(orchestrator), registry: Arc::new(registry), default_kv_store, + identity_graph, auction_telemetry_sink, + identity, + #[cfg(test)] + http_client_for_test: None, + #[cfg(test)] + geo_for_test: None, })) } +/// Builds application state from `settings` with `graph` as its identity +/// graph, so routing, finalize and the after-send flush all use one store. +#[cfg(test)] +pub(crate) fn app_state_with_graph_for_test( + settings: Settings, + graph: KvIdentityGraph, +) -> Arc { + let state = build_state_from_settings(settings).expect("should build app state from settings"); + let mut state = Arc::try_unwrap(state) + .unwrap_or_else(|_| unreachable!("a freshly built state should have one owner")); + state.identity_graph = Some(graph); + Arc::new(state) +} + +/// `state` with `identity` as its identity runtime, for a request that +/// reaches a provider no configuration can name. +#[cfg(test)] +pub(crate) fn app_state_with_identity_for_test( + state: Arc, + identity: IdentityRuntime, +) -> Arc { + let mut state = Arc::try_unwrap(state) + .unwrap_or_else(|_| unreachable!("a freshly built state should have one owner")); + state.identity = Arc::new(identity); + Arc::new(state) +} + +/// As [`app_state_with_graph_for_test`], with `client` answering every +/// outbound request the routes make and `geo` locating every client. +#[cfg(test)] +pub(crate) fn app_state_with_network_for_test( + settings: Settings, + graph: KvIdentityGraph, + client: Arc, + geo: Arc, +) -> Arc { + let state = app_state_with_graph_for_test(settings, graph); + let mut state = Arc::try_unwrap(state) + .unwrap_or_else(|_| unreachable!("a freshly built state should have one owner")); + state.http_client_for_test = Some(client); + state.geo_for_test = Some(geo); + Arc::new(state) +} + fn warn_if_certificate_check_disabled(settings: &Settings) { if !settings.proxy.certificate_check { log::warn!( @@ -261,6 +373,13 @@ fn warn_if_certificate_check_disabled(settings: &Settings) { /// continues to rely on the trusted `fastly-ssl` header injected by /// `edgezero_main` after sanitization. fn build_per_request_services(state: &AppState, ctx: &RequestContext) -> RuntimeServices { + let http_client: Arc = Arc::new(FastlyPlatformHttpClient::default()); + #[cfg(test)] + let http_client = state.http_client_for_test.clone().unwrap_or(http_client); + let geo: Arc = Arc::new(FastlyPlatformGeo); + #[cfg(test)] + let geo = state.geo_for_test.clone().unwrap_or(geo); + let client_info = ctx .request() .extensions() @@ -281,8 +400,8 @@ fn build_per_request_services(state: &AppState, ctx: &RequestContext) -> Runtime .template_cache(Arc::new(crate::template_cache::FastlyTemplateCache::new())) .template_assembler(Arc::new(crate::esi_assembly::FastlyTemplateAssembler)) .backend(Arc::new(FastlyPlatformBackend)) - .http_client(Arc::new(FastlyPlatformHttpClient)) - .geo(Arc::new(FastlyPlatformGeo)) + .http_client(http_client) + .geo(geo) .auction_telemetry_sink(Arc::clone(&state.auction_telemetry_sink)) .client_info(client_info) .build() @@ -313,21 +432,20 @@ fn uses_dynamic_tsjs_fallback(method: &Method, path: &str) -> bool { /// /// `edgezero_main` pops this from the response after dispatch and runs /// [`trusted_server_core::ec::finalize::ec_finalize_response`] plus the -/// pull-sync hook on the converted fastly response — the same EC response +/// after-send identity flush on the converted fastly response — the same EC response /// lifecycle the legacy path drives through `RouteResult`. #[derive(Clone)] pub(crate) struct EcFinalizeState { pub(crate) ec_context: EcContext, /// Whether EC finalization may write to the KV identity graph. - /// `KvIdentityGraph` wraps a non-`Sync` `dyn EcKvStore` and cannot ride - /// in response extensions, so `edgezero_main` rebuilds the graph from - /// settings when this is set. + /// `edgezero_main` takes the graph from application state when this is + /// set. pub(crate) use_finalize_kv: bool, pub(crate) eids_cookie: Option, pub(crate) sharedid_cookie: Option, pub(crate) is_real_browser: bool, - /// Per-request services carried to the entry point so the pull-sync - /// dispatcher can reuse the same platform HTTP client. + /// Per-request services carried to the entry point so the after-send + /// providers reuse the same platform HTTP client. pub(crate) services: RuntimeServices, } @@ -339,6 +457,11 @@ struct EcRequestState { ec_context: EcContext, kv_graph: Option, finalize_kv_graph: Option, + /// The graph integration proxy dispatch gets, for every client: a + /// proxied capture's removals and opt-outs are revocations and keep the + /// write path, as consent withdrawals do (spec 4.7). EC generation on + /// that route checks the real-browser flag itself. + proxy_kv_graph: Option, eids_cookie: Option, sharedid_cookie: Option, is_real_browser: bool, @@ -356,9 +479,12 @@ struct EcRequestState { impl EcRequestState { fn into_finalize_state(self) -> EcFinalizeState { + // A global revoke is a withdrawal from any client (spec 8). + let use_finalize_kv = + self.finalize_kv_graph.is_some() || self.ec_context.global_revoke_requested(); EcFinalizeState { ec_context: self.ec_context, - use_finalize_kv: self.finalize_kv_graph.is_some(), + use_finalize_kv, eids_cookie: self.eids_cookie, sharedid_cookie: self.sharedid_cookie, is_real_browser: self.is_real_browser, @@ -367,6 +493,30 @@ impl EcRequestState { } } +/// Collects the request's browser ingest for a new EC's one create. +/// +/// An unreadable partner registry yields an empty seed: the create still +/// happens, without the ingest. +fn browser_seed_for(state: &AppState, ec: &EcRequestState) -> BrowserSeed { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| elapsed.as_secs()); + match PartnerRegistry::from_config(&state.settings.ec.partners) { + Ok(registry) => browser_seed( + &state.settings, + ®istry, + ec.ec_context.consent(), + ec.eids_cookie.as_deref(), + ec.sharedid_cookie.as_deref(), + now, + ), + Err(err) => { + log::warn!("Browser ingest skipped: partner registry is unavailable: {err:?}"); + BrowserSeed::empty() + } + } +} + /// Derives device signals from the request's `User-Agent` header. /// /// Used as the fallback when no entry-point-captured [`DeviceSignals`] are @@ -402,10 +552,11 @@ fn device_signals_for(req: &Request) -> DeviceSignals { /// Builds the per-request EC state, mirroring the pre-routing prelude of the /// legacy `route_request` step by step. fn build_ec_request_state( - settings: &Settings, + state: &AppState, services: &RuntimeServices, req: &Request, ) -> EcRequestState { + let settings = &*state.settings; let device_signals = device_signals_for(req); let is_real_browser = device_signals.looks_like_browser(); if !is_real_browser { @@ -432,6 +583,10 @@ fn build_ec_request_state( match EcContext::read_from_request_with_geo(settings, req, services, geo_info.as_ref()) { Ok(mut context) => { context.set_device_signals(device_signals); + context.set_real_browser(is_real_browser); + context.set_identity_runtime(Arc::clone(&state.identity)); + context + .set_identity_rate_limiter(Arc::new(FastlyRateLimiter::new(RATE_COUNTER_NAME))); // Orphan-recovery eligibility is intentionally left false here. // Authorizing it during generic pre-routing would let named // routes and request-filter short circuits (e.g. a DataDome @@ -447,7 +602,12 @@ fn build_ec_request_state( // Bot gate: suppress KV-backed EC writes for unrecognized clients, except // consent withdrawals. Revocations keep the write path so tombstones stay // authoritative even for privacy-extension-heavy clients. - let kv_graph = crate::maybe_identity_graph(settings); + let kv_graph = state.identity_graph.clone(); + let proxy_kv_graph = if setup_error.is_none() { + kv_graph.clone() + } else { + None + }; let finalize_kv_graph = if setup_error.is_none() && (is_real_browser || ec_consent_withdrawn(ec_context.consent())) { @@ -461,6 +621,7 @@ fn build_ec_request_state( ec_context, kv_graph, finalize_kv_graph, + proxy_kv_graph, eids_cookie, sharedid_cookie, is_real_browser, @@ -572,6 +733,15 @@ async fn execute_named( // These diagnostics are read-only. Running the normal EC lifecycle would // attach finalization state and could ingest request cookies into KV after // the handler returns, violating that contract. + // The opt-out clear writes one KV row through the removal write, never the + // request's cookies, so it skips the EC lifecycle for the same reason. + if matches!(handler, NamedRouteHandler::AdminOptOutClear) { + return Ok(handle_admin_opt_out_clear( + state.identity_graph.as_ref(), + &req, + )); + } + if matches!( handler, NamedRouteHandler::AdminEcLookup | NamedRouteHandler::AdminEidsLookup @@ -582,8 +752,13 @@ async fn execute_named( // Deliberately do not use an EC request-state graph: that // copy is bot-gated, while operators use curl for this // authenticated diagnostic. - let kv = crate::maybe_identity_graph(&state.settings); - handle_admin_ec_lookup(kv.as_ref(), ®istry, &req) + let kv = state.identity_graph.clone(); + handle_admin_ec_lookup( + kv.as_ref(), + ®istry, + Some(state.identity.as_ref()), + &req, + ) } NamedRouteHandler::AdminEidsLookup => handle_admin_eids_lookup(®istry, &req), _ => unreachable!("admin diagnostics should use early dispatch"), @@ -599,7 +774,7 @@ async fn execute_named( return Ok(http_error(&report)); } - let mut ec = build_ec_request_state(&state.settings, &services, &req); + let mut ec = build_ec_request_state(&state, &services, &req); // EcContext creation errors short-circuit before filters, mirroring legacy: // the legacy path returns its error response before running filter_request. if let Some(report) = ec.setup_error.take() { @@ -647,6 +822,9 @@ async fn run_named_route( NamedRouteHandler::AdminCachePurge => { unreachable!("cache purge should be handled before EC setup") } + NamedRouteHandler::AdminOptOutClear => { + unreachable!("opt-out clear should be handled before EC setup") + } NamedRouteHandler::LegacyAdminDenied => Ok(legacy_admin_alias_denied()), NamedRouteHandler::BatchSync => { // Dispatched by execute_named before EC state is built. @@ -656,7 +834,13 @@ async fn run_named_route( if req.method() == Method::OPTIONS { cors_preflight_identify(&state.settings, &req) } else { - let kv = crate::require_identity_graph(&state.settings)?; + // The state's graph is the one over `ec.ec_store` that + // `require_identity_graph` would build, so the answer reads the + // same store the rest of the request counts against. + let kv = match state.identity_graph.clone() { + Some(graph) => graph, + None => crate::require_identity_graph(&state.settings)?, + }; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; handle_identify( &state.settings, @@ -735,15 +919,14 @@ async fn run_named_route( /// graph + partner registry + rate limiter, with a default EC context for /// response finalization. fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> Response { - let device_signals = device_signals_for(&req); - let is_real_browser = device_signals.looks_like_browser(); + let is_real_browser = device_signals_for(&req).looks_like_browser(); let eids_cookie = crate::extract_cookie_value(&req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(&req, COOKIE_SHAREDID); let result = crate::require_identity_graph(&state.settings).and_then(|kv| { let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); - handle_batch_sync(&kv, &partner_registry, &limiter, req) + handle_batch_sync(&kv, &partner_registry, &limiter, &state.identity, req) }); let mut response = result.unwrap_or_else(|e| http_error(&e)); @@ -788,7 +971,7 @@ async fn dispatch_fallback( return http_error(&report); } - let mut ec = build_ec_request_state(&state.settings, services, &req); + let mut ec = build_ec_request_state(state, services, &req); if let Some(report) = ec.setup_error.take() { let response = http_error(&report); return attach_dispatch_extensions(response, ec, RequestFilterEffects::default()); @@ -810,15 +993,19 @@ async fn dispatch_fallback( // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the // publisher-specific streaming finalizer instead. + ec.ec_context.decide_writer(&req, true, ec.is_real_browser); + let seed = browser_seed_for(state, &ec); state .registry .handle_proxy(ProxyDispatchInput { method: &method, path: &path, settings: &state.settings, - kv: ec.kv_graph.as_ref(), + kv: ec.proxy_kv_graph.as_ref(), ec_context: &mut ec.ec_context, services, + is_real_browser: ec.is_real_browser, + seed: &seed, req, }) .await @@ -844,15 +1031,23 @@ async fn dispatch_fallback( // Only for document navigations by recognised browsers; subresource // requests may lack consent signals such as Sec-GPC. let is_navigation = is_navigation_request(&req); + ec.ec_context.decide_writer(&req, true, ec.is_real_browser); if is_navigation { ec.ec_context.set_eid_sync_source(EidSyncSource::Navigation); } let is_publisher_navigation = ec.is_real_browser && is_navigation; - if is_publisher_navigation - && let Err(err) = ec - .ec_context - .generate_if_needed(&state.settings, ec.kv_graph.as_ref()) - { + let seed = browser_seed_for(state, &ec); + // With a hold-mode provider that can run before the row exists, the + // create waits for it inside the publisher handler (spec 4.7). + let minted = if !is_publisher_navigation { + Ok(()) + } else if state.identity.holds_on_first_navigation() { + ec.ec_context.mint_deferred(&state.settings, &seed) + } else { + ec.ec_context + .generate_if_needed(&state.settings, ec.kv_graph.as_ref(), &seed) + }; + if let Err(err) = minted { log::warn!("EC generation failed for publisher proxy: {err:?}"); } @@ -1076,6 +1271,7 @@ enum NamedRouteHandler { DeactivateKey, AdminEcLookup, AdminEidsLookup, + AdminOptOutClear, AdminCachePurge, /// Legacy `/admin/keys/*` aliases — denied locally with 404 so they never /// reach the publisher fallback (which would leak admin credentials). @@ -1155,6 +1351,14 @@ const NAMED_ROUTES: &[NamedRoute] = &[ primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, }, + // Operator clear of a provider's sticky opt-out (`ps.o`). Only DELETE is + // claimed: any other method reaches the publisher fallback, which denies + // the reserved admin namespace locally. + NamedRoute { + path: "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + primary_methods: &[Method::DELETE], + handler: NamedRouteHandler::AdminOptOutClear, + }, // Admin EIDs echo: decodes the request's ts-eids/sharedId cookies with // an ingestion preview. Pure request inspection — no KV access. NamedRoute { @@ -1293,7 +1497,7 @@ impl TrustedServerApp { (Self::routes_for_state(&state), Some(state)) } - fn routes_for_state(state: &Arc) -> RouterService { + pub(crate) fn routes_for_state(state: &Arc) -> RouterService { let mut router = RouterService::builder() .middleware(FinalizeResponseMiddleware::new( Arc::clone(&state.settings), @@ -1372,9 +1576,9 @@ mod tests { use super::{ AppState, AuctionDispatch, EcContext, EdgeCacheHeader, EidSyncSource, HandlerFuture, NAMED_ROUTES, NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, RuntimeStoreConfig, - TrustedServerApp, build_orchestrator_with_plan, build_per_request_services, - build_state_from_settings, compile_auction_plan, handle_publisher_request, - publisher_response_into_streaming_response, startup_error_router, + TrustedServerApp, build_identity_runtime, build_orchestrator_with_plan, + build_per_request_services, build_state_from_settings, compile_auction_plan, + handle_publisher_request, publisher_response_into_streaming_response, startup_error_router, }; use base64::Engine as _; use bytes::Bytes; @@ -1388,6 +1592,7 @@ mod tests { use edgezero_core::params::PathParams; use edgezero_core::router::RouterService; use std::net::{IpAddr, Ipv4Addr}; + use trusted_server_core::ec::identity::test_support::ec_partner; use error_stack::Report; use futures::executor::block_on; @@ -1478,6 +1683,28 @@ mod tests { block_on(router.oneshot(request)).expect("should route request") } + #[test] + fn reloaded_identity_runtime_keeps_pull_providers_when_the_registry_fails() { + let mut settings = test_settings(); + settings.ec.partners = vec![ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + "example-token-value-0123456789abcdef", + )]; + let failed = Err(Report::new(TrustedServerError::Configuration { + message: "registry failed".to_owned(), + })); + + let runtime = super::identity_runtime_from_registry_result(&settings, failed) + .expect("should still build the pull runtime"); + + assert_eq!( + format!("{runtime:?}"), + "IdentityRuntime { providers: 1, .. }", + "should hold the generic pull provider" + ); + } + fn test_settings() -> Settings { Settings::from_toml( r#" @@ -1566,6 +1793,8 @@ mod tests { filters: Vec>, ) -> RouterService { let settings = test_settings(); + let identity = build_identity_runtime(&settings, &IntegrationRegistry::default()) + .expect("should build the identity runtime"); let plan = Arc::new( trusted_server_core::auction::compile_auction_plan(&settings) .expect("should compile auction plan"), @@ -1584,6 +1813,10 @@ mod tests { orchestrator: Arc::new(orchestrator), registry: Arc::new(registry), default_kv_store, + identity, + identity_graph: None, + http_client_for_test: None, + geo_for_test: None, }); TrustedServerApp::routes_for_state(&state) } @@ -1936,6 +2169,35 @@ mod tests { ); } + #[test] + fn admin_opt_out_clear_route_is_registered() { + // The operator clear is a DELETE on its own route. The route must call + // the core handler, and `/_ts/admin/ec/{id}` must stay GET-only so the + // clear is reachable only through the dedicated path. + let route = NAMED_ROUTES + .iter() + .find(|route| route.path == "/_ts/admin/ec/{id}/providers/{provider}/opt-out") + .expect("should register the provider opt-out clear route"); + assert!( + matches!(route.handler, NamedRouteHandler::AdminOptOutClear), + "the route must map to the opt-out clear handler" + ); + assert_eq!( + route.primary_methods, + &[Method::DELETE], + "the route must have DELETE as its only primary method" + ); + let lookup = NAMED_ROUTES + .iter() + .find(|route| route.path == "/_ts/admin/ec/{id}") + .expect("should register the lookup route"); + assert_eq!( + lookup.primary_methods, + &[Method::GET], + "control: the lookup route must stay GET-only" + ); + } + #[test] fn page_bids_serves_canonical_path_and_deprecated_alias() { // The SPA re-auction endpoint lives at the canonical single-underscore diff --git a/crates/trusted-server-adapter-fastly/src/backend.rs b/crates/trusted-server-adapter-fastly/src/backend.rs index 0151cbb165..a0d08faab6 100644 --- a/crates/trusted-server-adapter-fastly/src/backend.rs +++ b/crates/trusted-server-adapter-fastly/src/backend.rs @@ -82,6 +82,8 @@ fn compute_host_header(scheme: &str, host: &str, port: u16) -> String { pub(crate) const DEFAULT_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(15); /// Default timeout between response body bytes for backends (10 seconds). pub(crate) const DEFAULT_BETWEEN_BYTES_TIMEOUT: Duration = Duration::from_secs(10); +/// Connect timeout of a backend whose spec sets none. +pub(crate) const DEFAULT_CONNECT_TIMEOUT: Duration = Duration::from_secs(1); /// Configuration for creating a dynamic Fastly backend. /// @@ -94,6 +96,7 @@ pub struct BackendConfig<'a> { certificate_check: bool, first_byte_timeout: Duration, between_bytes_timeout: Duration, + connect_timeout: Option, host_header_override: Option<&'a str>, discriminator: Option<&'a str>, } @@ -112,6 +115,7 @@ impl<'a> BackendConfig<'a> { certificate_check: true, first_byte_timeout: DEFAULT_FIRST_BYTE_TIMEOUT, between_bytes_timeout: DEFAULT_BETWEEN_BYTES_TIMEOUT, + connect_timeout: None, host_header_override: None, discriminator: None, } @@ -154,6 +158,14 @@ impl<'a> BackendConfig<'a> { self } + /// Set the maximum time to wait for the connection. `None`, the default, + /// uses [`DEFAULT_CONNECT_TIMEOUT`]. + #[must_use] + pub fn connect_timeout(mut self, timeout: Option) -> Self { + self.connect_timeout = timeout; + self + } + /// Set the outbound Host header sent to the backend origin. #[must_use] pub fn host_header_override(mut self, host: Option<&'a str>) -> Self { @@ -183,6 +195,7 @@ impl<'a> BackendConfig<'a> { certificate_check: self.certificate_check, first_byte_timeout: self.first_byte_timeout, between_bytes_timeout: self.between_bytes_timeout, + connect_timeout: self.connect_timeout, discriminator: self.discriminator.map(str::to_owned), } } @@ -237,7 +250,7 @@ impl<'a> BackendConfig<'a> { // Target base is host[:port]; SSL is enabled only for https scheme let mut builder = Backend::builder(&backend_name, &host_with_port) .override_host(&host_header) - .connect_timeout(Duration::from_secs(1)) + .connect_timeout(self.connect_timeout.unwrap_or(DEFAULT_CONNECT_TIMEOUT)) .first_byte_timeout(self.first_byte_timeout) .between_bytes_timeout(self.between_bytes_timeout); if self.scheme.eq_ignore_ascii_case("https") { diff --git a/crates/trusted-server-adapter-fastly/src/ec_kv.rs b/crates/trusted-server-adapter-fastly/src/ec_kv.rs index c427ede96b..660b51d0ed 100644 --- a/crates/trusted-server-adapter-fastly/src/ec_kv.rs +++ b/crates/trusted-server-adapter-fastly/src/ec_kv.rs @@ -7,7 +7,7 @@ use error_stack::{Report, ResultExt}; use fastly::kv_store::{InsertMode, KVStore, KVStoreError, ListPage}; use trusted_server_core::ec::kv_backend::{ - EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome, + EcKvLookup, EcKvRateLimited, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome, }; use trusted_server_core::ec::log_id; use trusted_server_core::error::TrustedServerError; @@ -151,6 +151,15 @@ impl EcKvStore for FastlyEcKvStore { Err(fastly::kv_store::KVStoreError::ItemPreconditionFailed) => { Ok(EcKvWriteOutcome::PreconditionFailed) } + Err(KVStoreError::TooManyRequests) => Err(Report::new(KVStoreError::TooManyRequests) + .change_context(TrustedServerError::KvStore { + store_name: self.store_name.clone(), + message: format!( + "Per-key write limit refused the entry for key '{}'", + log_id(key) + ), + }) + .attach_opaque(EcKvRateLimited)), Err(err) => Err( Report::new(err).change_context(TrustedServerError::KvStore { store_name: self.store_name.clone(), @@ -187,6 +196,10 @@ impl EcKvStore for FastlyEcKvStore { message: format!("Failed to delete key '{}'", log_id(key)), }) } + + fn pause(&self, duration: std::time::Duration) { + std::thread::sleep(duration); + } } #[cfg(test)] @@ -227,6 +240,18 @@ mod tests { .expect("should build a list page") } + #[test] + fn pause_blocks_for_the_duration() { + let started = std::time::Instant::now(); + + store().pause(Duration::from_millis(50)); + + assert!( + started.elapsed() >= Duration::from_millis(50), + "should block for at least the requested duration" + ); + } + #[test] fn exact_existence_follows_cursors_and_rejects_prefix_collisions() { let mut cursors = Vec::new(); diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index b8f814a01c..949cffa770 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -13,12 +13,18 @@ use fastly::{Request as FastlyRequest, Response as FastlyResponse}; use trusted_server_core::cache_policy::{EdgeCacheHeader, cache_control_headers_have_directive}; use trusted_server_core::constants::HEADER_X_TS_VERSION; +use trusted_server_core::ec::EcKvSnapshot; use trusted_server_core::ec::device::DeviceSignals; -use trusted_server_core::ec::finalize::ec_finalize_response; -use trusted_server_core::ec::kv::KvIdentityGraph; -use trusted_server_core::ec::pull_sync::{ - PullSyncContext, build_pull_sync_context, dispatch_pull_sync, +use trusted_server_core::ec::finalize::{ec_finalize_response, retry_failed_withdrawals}; +use trusted_server_core::ec::identity::IdentityRuntime; +use trusted_server_core::ec::identity::flush::{ + AfterSendInput, FlushOutcome, defer_capture_writes, lose_pending_removals, may_flush, + retry_pending_removals, run_after_send, }; +use trusted_server_core::ec::identity::measurement::{IdentityMeasurement, MeasurementInput}; +use trusted_server_core::ec::kv::KvIdentityGraph; +use trusted_server_core::ec::kv_backend::KvOperationCounts; +use trusted_server_core::ec::rate_limiter::RateLimiter; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::RequestFilterEffects; @@ -44,7 +50,7 @@ mod tinybird; use crate::app::{ AppState, EcFinalizeState, RuntimeStoreConfig, TrustedServerApp, - load_settings_from_config_store, + build_identity_runtime_from_reloaded_settings, load_settings_from_config_store, }; use crate::ec_kv::FastlyEcKvStore; use crate::middleware::{HEADER_X_TS_FINALIZED, apply_finalize_headers, resolve_geo_for_response}; @@ -506,50 +512,72 @@ fn edgezero_main(mut req: FastlyRequest, sandbox: &mut Sandbox, ordinal: u64, re } if let Some(mut ec_state) = ec_state { - if let Some(settings) = settings_snapshot.as_deref() { - match apply_edgezero_ec_finalize(settings, &mut ec_state, &mut response) { - Ok(partner_registry) => { - send_edgezero_response( - response, - request_filter_effects.as_ref(), - counters.as_ref(), - ); - run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); - return; - } - Err(e) => { - log::error!( - "EdgeZero EC finalize skipped: failed to build partner registry: {e:?}" - ); + let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); + let mut send = |response: HttpResponse| { + send_edgezero_response(response, request_filter_effects.as_ref(), counters.as_ref()); + }; + if let Some(state) = app_state.as_deref() { + let graph = || { + if state.identity_graph.is_none() { + log::warn!("identity graph unavailable: ec.ec_store is not configured"); } - } - } else { - match load_settings_from_config_store(&runtime_stores) { - Ok(settings) => { - match apply_edgezero_ec_finalize(&settings, &mut ec_state, &mut response) { - Ok(partner_registry) => { - send_edgezero_response( - response, - request_filter_effects.as_ref(), - counters.as_ref(), - ); - run_edgezero_pull_sync_after_send( - &settings, - &partner_registry, - &ec_state, - ); - return; - } - Err(e) => { - log::error!( - "EdgeZero EC finalize skipped: failed to build partner registry: {e:?}" - ); - } + state.identity_graph.clone() + }; + finalize_send_and_flush( + PostRouteInput { + runtime: Some(&state.identity), + settings: &state.settings, + ec_state: &mut ec_state, + graph: &graph, + rate_limiter: &limiter, + }, + response, + &mut send, + ); + return; + } + match load_settings_from_config_store(&runtime_stores) { + Ok(settings) => { + // No application state was built, so the identity runtime + // comes from the reloaded settings. Without one, finalize and + // the withdrawal retry still run; only phase 4 is skipped. + let identity = match build_identity_runtime_from_reloaded_settings(&settings) { + Ok(identity) => { + ec_state + .ec_context + .set_identity_runtime(Arc::clone(&identity)); + Some(identity) } - } - Err(e) => { - log::warn!("EdgeZero EC finalize skipped: failed to reload settings: {e:?}"); - } + Err(e) => { + log::warn!( + "identity flush skipped: identity runtime unavailable after settings reload: {}", + e.current_context() + ); + None + } + }; + let graph = || { + require_identity_graph(&settings) + .map_err(|e| { + log::warn!("identity graph unavailable: {}", e.current_context()); + }) + .ok() + }; + finalize_send_and_flush( + PostRouteInput { + runtime: identity.as_deref(), + settings: &settings, + ec_state: &mut ec_state, + graph: &graph, + rate_limiter: &limiter, + }, + response, + &mut send, + ); + return; + } + Err(e) => { + log::warn!("EdgeZero EC finalize skipped: failed to reload settings: {e:?}"); } } } @@ -592,14 +620,73 @@ fn apply_entry_point_finalize_headers( apply_finalize_headers(settings, geo_info.as_ref(), response); } -fn apply_edgezero_ec_finalize( - settings: &Settings, - ec_state: &mut EcFinalizeState, - response: &mut HttpResponse, -) -> Result> { - let partner_registry = PartnerRegistry::from_config(&settings.ec.partners)?; +/// What the post-route work reads. It takes the runtime and settings rather +/// than [`AppState`], because the settings-reload branch has none. +struct PostRouteInput<'a> { + /// `None` when the runtime could not be built: finalize and the + /// withdrawal retry still run, and phase 4 is skipped. + runtime: Option<&'a IdentityRuntime>, + settings: &'a Settings, + ec_state: &'a mut EcFinalizeState, + /// Builds the identity graph. Called only when finalize or the after-send + /// work needs one, and at most once. + graph: &'a dyn Fn() -> Option, + rate_limiter: &'a dyn RateLimiter, +} + +/// Finalizes the response, hands it to `send`, then runs the after-send work: +/// the withdrawal retry first, then phase 4's one flush (spec 5.2), and logs +/// the request's `identity request measurement` line. +/// +/// `send` is a callback so phase 4 runs only after the response is sent. +/// Returns phase 4's outcome, or `None` when finalize could not run. +fn finalize_send_and_flush( + input: PostRouteInput<'_>, + response: HttpResponse, + send: &mut dyn FnMut(HttpResponse), +) -> Option { + let (outcome, measurement) = finalize_send_and_flush_measured(input, response, send); + if let Some(measurement) = measurement { + measurement.log(); + } + outcome +} + +/// As [`finalize_send_and_flush`], returning the measurement instead of +/// logging it. The measurement is `None` when finalize could not run. +fn finalize_send_and_flush_measured( + input: PostRouteInput<'_>, + mut response: HttpResponse, + send: &mut dyn FnMut(HttpResponse), +) -> (Option, Option) { + let PostRouteInput { + runtime, + settings, + ec_state, + graph, + rate_limiter, + } = input; + let partner_registry = match PartnerRegistry::from_config(&settings.ec.partners) { + Ok(registry) => registry, + Err(e) => { + log::error!("EdgeZero EC finalize skipped: failed to build partner registry: {e:?}"); + send(response); + return (None, None); + } + }; + // Every KV operation from here on counts into this; phase 1's read is + // known from the snapshot the route left. + let kv_counts = KvOperationCounts::default(); + let phase_one_read = !matches!(ec_state.ec_context.kv_snapshot(), EcKvSnapshot::NotRead); + let mut built: Option> = None; + let mut graph_once = || { + built + .get_or_insert_with(|| graph().map(|graph| graph.with_operation_counts(&kv_counts))) + .clone() + }; + let finalize_kv_graph = if ec_state.use_finalize_kv { - maybe_identity_graph(settings) + graph_once() } else { None }; @@ -610,54 +697,89 @@ fn apply_edgezero_ec_finalize( &partner_registry, ec_state.eids_cookie.as_deref(), ec_state.sharedid_cookie.as_deref(), - response, + &mut response, ); - Ok(partner_registry) -} + let status = response.status(); + send(response); -fn run_edgezero_pull_sync_after_send( - settings: &Settings, - partner_registry: &PartnerRegistry, - ec_state: &EcFinalizeState, -) { - if !ec_state.is_real_browser { - return; + // A retry skipped because the graph is unavailable leaves the ID failed, so + // it counts as a failed run. + let withdrawal_after_send = ec_state.ec_context.has_failed_withdrawals(); + if withdrawal_after_send { + match graph_once() { + Some(graph) => retry_failed_withdrawals(&mut ec_state.ec_context, &graph), + None => log::warn!("withdrawal retry skipped: identity graph unavailable"), + } } + let withdrawal_after_send_failed = ec_state.ec_context.has_failed_withdrawals(); - let prepared_context = build_pull_sync_context(&ec_state.ec_context, partner_registry); - let Some((context, kv)) = - prepare_pull_sync_after_send(prepared_context, || require_identity_graph(settings)) - else { - return; - }; + // A capture removal write that failed before send gets one more attempt. + if ec_state.ec_context.has_pending_removals() { + match graph_once() { + Some(graph) => retry_pending_removals(&mut ec_state.ec_context, &graph), + None => { + log::warn!("capture removal retry skipped: identity graph unavailable"); + lose_pending_removals(&mut ec_state.ec_context); + } + } + } + + // Positive capture IDs from a request that is not the writer wait in the + // pending key for the next writer. + if ec_state.ec_context.has_capture_writes() { + match graph_once() { + Some(graph) => defer_capture_writes(&mut ec_state.ec_context, &graph), + None => log::warn!("capture writes dropped: identity graph unavailable"), + } + } - let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); - dispatch_pull_sync( + let outcome = flush_after_send( + runtime, settings, - &kv, - partner_registry, - &limiter, - &context, - &ec_state.services, + ec_state, + &mut graph_once, + status, + rate_limiter, ); + let measurement = ec_state.ec_context.identity_measurement(&MeasurementInput { + phase_one_read, + kv_reads: kv_counts.reads(), + kv_writes: kv_counts.writes(), + flush: Some(&outcome), + withdrawal_after_send, + withdrawal_after_send_failed, + }); + (Some(outcome), Some(measurement)) } -fn prepare_pull_sync_after_send( - context: Option, - graph_factory: F, -) -> Option<(PullSyncContext, KvIdentityGraph)> -where - F: FnOnce() -> Result>, -{ - let context = context?; - let kv = match graph_factory() { - Ok(kv) => kv, - Err(err) => { - log::debug!("Pull sync: identity graph unavailable, skipping: {err:?}"); - return None; - } +/// Phase 4's one flush, or [`FlushOutcome::Skipped`] when it cannot run. +fn flush_after_send( + runtime: Option<&IdentityRuntime>, + settings: &Settings, + ec_state: &mut EcFinalizeState, + graph_once: &mut dyn FnMut() -> Option, + status: edgezero_core::http::StatusCode, + rate_limiter: &dyn RateLimiter, +) -> FlushOutcome { + let Some(runtime) = runtime else { + return FlushOutcome::Skipped; + }; + if !ec_state.is_real_browser || !may_flush(&ec_state.ec_context) { + return FlushOutcome::Skipped; + } + let Some(graph) = graph_once() else { + log::debug!("identity flush skipped: identity graph unavailable"); + return FlushOutcome::Skipped; }; - Some((context, kv)) + run_after_send(AfterSendInput { + runtime, + settings, + services: &ec_state.services, + graph: &graph, + context: &mut ec_state.ec_context, + status, + rate_limiter, + }) } /// Sends a finalized `EdgeZero` response to the client. @@ -845,15 +967,15 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { use super::*; + use crate::app::TrustedServerApp; use edgezero_core::body::Body as EdgeBody; use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use trusted_server_core::ec::identity::measurement::RouteKind; use trusted_server_core::integrations::HeaderMutation; - fn test_settings() -> Settings { - Settings::from_toml( - r#" + const TEST_SETTINGS_TOML: &str = r#" [[handlers]] path = "^/_ts/admin" username = "admin" @@ -872,26 +994,404 @@ mod tests { enabled = false config_store_id = "test-config-store-id" secret_store_id = "test-secret-store-id" - "#, + "#; + + fn test_settings() -> Settings { + Settings::from_toml(TEST_SETTINGS_TOML).expect("should parse test settings") + } + + const EC_ID: &str = "0000000000000000000000000000000000000000000000000000000000000000.ABC123"; + + /// Allows every call. + struct AllowAll; + + impl RateLimiter for AllowAll { + fn exceeded(&self, _key: &str, _limit: u32) -> Result> { + Ok(false) + } + + fn record(&self, _key: &str) -> Result<(), Report> { + Ok(()) + } + + fn count(&self, _key: &str) -> Result> { + Ok(0) + } + } + + /// Marks `request` as sent by a real browser at a client address. + fn as_browser(request: &mut HttpRequest) { + request.extensions_mut().insert(DeviceSignals::derive( + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36", + Some("t13d1516h2_8daaf6152771_b186095e22b6"), + Some("1:65536;2:0;4:6291456;6:262144"), + )); + request + .extensions_mut() + .insert(trusted_server_core::platform::ClientInfo { + client_ip: Some("192.0.2.10".parse().expect("should parse the client IP")), + ..trusted_server_core::platform::ClientInfo::default() + }); + } + + /// A browser document navigation carrying `cookie`. + fn navigation(cookie: Option<&str>) -> HttpRequest { + let mut builder = edgezero_core::http::request_builder() + .method(edgezero_core::http::Method::GET) + .uri("https://test-publisher.com/some-page") + .header("sec-fetch-dest", "document"); + if let Some(cookie) = cookie { + builder = builder.header("cookie", cookie); + } + let mut request = builder + .body(EdgeBody::empty()) + .expect("should build request"); + as_browser(&mut request); + request + } + + /// Polls `future` to completion on this thread without entering an + /// executor, so code under test that blocks on its own futures, such as + /// the hold dispatch of an auction navigation, can run inside it. The + /// scripted network is ready at once, so a future that stays pending is a + /// test bug. + fn poll_ready(future: F) -> F::Output { + let waker = futures::task::noop_waker(); + let mut context = core::task::Context::from_waker(&waker); + let mut future = core::pin::pin!(future); + for _ in 0..10_000 { + if let core::task::Poll::Ready(output) = future.as_mut().poll(&mut context) { + return output; + } + } + unreachable!("the future should complete against the scripted network") + } + + /// Routes `request`, then finalizes, sends and flushes it as + /// `edgezero_main` does, keeping the sent response. + fn drive_request_for_test( + state: &Arc, + request: HttpRequest, + ) -> (HttpResponse, Option) { + drive_with_runtime(state, request, Some(&state.identity), |_| {}) + } + + /// As [`drive_request_for_test`], with the runtime the post-route work + /// gets: `None` is the settings-reload branch without a runtime. + fn drive_with_runtime( + state: &Arc, + request: HttpRequest, + runtime: Option<&IdentityRuntime>, + prepare: impl FnOnce(&mut EcFinalizeState), + ) -> (HttpResponse, Option) { + let router = TrustedServerApp::routes_for_state(state); + let mut response = poll_ready(router.oneshot(request)).expect("should route the request"); + let Some(mut ec_state) = response.extensions_mut().remove::() else { + return (response, None); + }; + prepare(&mut ec_state); + let mut sent = None; + let graph = || state.identity_graph.clone(); + let outcome = finalize_send_and_flush( + PostRouteInput { + runtime, + settings: &state.settings, + ec_state: &mut ec_state, + graph: &graph, + rate_limiter: &AllowAll, + }, + response, + &mut |response| sent = Some(response), + ); + (sent.expect("should send the response"), outcome) + } + + /// As [`drive_request_for_test`], keeping the request's measurement. + fn drive_measured_for_test( + state: &Arc, + request: HttpRequest, + ) -> (Option, IdentityMeasurement) { + let router = TrustedServerApp::routes_for_state(state); + let mut response = + futures::executor::block_on(router.oneshot(request)).expect("should route the request"); + let mut ec_state = response + .extensions_mut() + .remove::() + .expect("should leave EC state on the response"); + let graph = || state.identity_graph.clone(); + let (outcome, measurement) = finalize_send_and_flush_measured( + PostRouteInput { + runtime: Some(&state.identity), + settings: &state.settings, + ec_state: &mut ec_state, + graph: &graph, + rate_limiter: &AllowAll, + }, + response, + &mut |_| {}, + ); + ( + outcome, + measurement.expect("should measure a finalized request"), + ) + } + + /// A returning visitor's live row, without a cluster size so the flush + /// has one change to write. + fn seeded_state() -> ( + Arc, + Arc, + ) { + let (graph, store) = KvIdentityGraph::write_counting("drive-store"); + let consent = trusted_server_core::ec::identity::test_support::allowing_consent(); + let entry = trusted_server_core::ec::kv_types::KvEntry::new( + &consent, + None, + 1_000, + "test-publisher.com", + ); + graph.create(EC_ID, &entry).expect("should seed the row"); + store.reset_counts(); + let state = crate::app::app_state_with_graph_for_test(test_settings(), graph); + (state, store) + } + + #[test] + fn identity_measurement_records_counts_for_a_writer_navigation() { + let (state, _store) = seeded_state(); + let cookie = format!("ts-ec={EC_ID}"); + + let (outcome, measurement) = drive_measured_for_test(&state, navigation(Some(&cookie))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush once"); + assert_eq!( + measurement.cookie_header_bytes(), + cookie.len(), + "should hold the inbound Cookie header size" + ); + assert_eq!( + measurement.kv_reads(), + 2, + "should count the phase 1 read and the pending-key read" + ); + assert_eq!(measurement.kv_writes(), 1, "should count the one write"); + assert_eq!( + measurement.route(), + RouteKind::Writer, + "should label a returning navigation as the writer" + ); + assert_eq!( + measurement.flush(), + Some(FlushOutcome::Written), + "should hold the phase 4 outcome" + ); + assert_eq!(measurement.removal_lost(), 0, "should lose no removal"); + assert!( + !measurement.withdrawal_after_send() && !measurement.withdrawal_after_send_failed(), + "should need no after-send withdrawal run" + ); + } + + #[test] + fn identity_measurement_holds_no_identifier() { + let (state, _store) = seeded_state(); + let cookie = format!("ts-ec={EC_ID}; ts-eids=secret-uid-value"); + + let (_, measurement) = drive_measured_for_test(&state, navigation(Some(&cookie))); + let line = measurement.to_string(); + + assert!( + line.contains(&format!("cookie_header_bytes={}", cookie.len())), + "should still carry the cookie size: {line}" + ); + assert!(!line.contains(EC_ID), "should not hold the EC ID: {line}"); + assert!( + !line.contains("secret-uid-value"), + "should not hold a cookie value: {line}" + ); + } + + #[test] + fn drive_request_for_test_counts_routing_finalize_and_flush_against_one_store() { + let (state, store) = seeded_state(); + + let (_, outcome) = + drive_request_for_test(&state, navigation(Some(&format!("ts-ec={EC_ID}")))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush once"); + let writes = store.writes(); + assert_eq!(writes.lookups, 1, "should read the row once, in phase 1"); + assert_eq!( + writes.if_generation_match, 1, + "should write once with a CAS" + ); + assert_eq!(writes.total(), 1, "should make no other write"); + } + + #[test] + fn finalize_send_and_flush_sends_before_it_flushes() { + let (state, store) = seeded_state(); + let router = TrustedServerApp::routes_for_state(&state); + let mut response = futures::executor::block_on( + router.oneshot(navigation(Some(&format!("ts-ec={EC_ID}")))), ) - .expect("should parse test settings") + .expect("should route the request"); + let mut ec_state = response + .extensions_mut() + .remove::() + .expect("should carry EC state"); + let writes_at_send = std::cell::Cell::new(None); + let graph = || state.identity_graph.clone(); + + let outcome = finalize_send_and_flush( + PostRouteInput { + runtime: Some(&state.identity), + settings: &state.settings, + ec_state: &mut ec_state, + graph: &graph, + rate_limiter: &AllowAll, + }, + response, + &mut |_| writes_at_send.set(Some(store.writes().total())), + ); + + assert_eq!( + writes_at_send.get(), + Some(0), + "should send before any write" + ); + assert_eq!(outcome, Some(FlushOutcome::Written), "should then flush"); + assert_eq!(store.writes().total(), 1, "should write once"); } #[test] - fn pull_sync_noop_states_skip_post_send_graph_factory() { + fn after_send_noop_states_skip_post_send_graph_factory() { + let (state, _) = seeded_state(); + let router = TrustedServerApp::routes_for_state(&state); + // A named route is never the writer, so phase 4 has nothing to do. + let request = edgezero_core::http::request_builder() + .method(edgezero_core::http::Method::GET) + .uri("https://test-publisher.com/.well-known/trusted-server.json") + .body(EdgeBody::empty()) + .expect("should build request"); + let mut response = + futures::executor::block_on(router.oneshot(request)).expect("should route"); + let mut ec_state = response + .extensions_mut() + .remove::() + .expect("should carry EC state"); + ec_state.use_finalize_kv = false; let calls = std::cell::Cell::new(0); - let result = prepare_pull_sync_after_send(None, || { + let graph = || { calls.set(calls.get() + 1); - Err(Report::new(TrustedServerError::KvStore { - store_name: "unexpected".to_owned(), - message: "graph factory should not run".to_owned(), - })) - }); + None + }; + + let outcome = finalize_send_and_flush( + PostRouteInput { + runtime: Some(&state.identity), + settings: &state.settings, + ec_state: &mut ec_state, + graph: &graph, + rate_limiter: &AllowAll, + }, + response, + &mut drop, + ); + + assert_eq!(outcome, Some(FlushOutcome::Skipped), "should skip phase 4"); + assert_eq!(calls.get(), 0, "should not build the graph"); + } + + #[test] + fn no_response_sets_or_reads_the_pull_complete_cookie() { + let (state, store) = seeded_state(); + let cookie = format!("ts-ec={EC_ID}; ts-ec-pull-complete=v1.example"); + + let (response, _) = drive_request_for_test(&state, navigation(Some(&cookie))); + + assert!( + response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .all(|value| !value.starts_with("ts-ec-pull-complete")), + "should set no pull-complete cookie" + ); + assert_eq!( + store.writes().lookups, + 1, + "should read the row whatever the old marker cookie says" + ); + + let (withdrawn, _) = drive_with_runtime( + &state, + navigation(Some(&cookie)), + Some(&state.identity), + withdraw, + ); + let cookies = set_cookies(&withdrawn); + assert!( + cookies + .iter() + .any(|value| value.starts_with("ts-ec=;") && value.contains("Max-Age=0")), + "control: finalize should still expire ts-ec on a withdrawal: {cookies:?}" + ); + assert!( + cookies + .iter() + .all(|value| !value.starts_with("ts-ec-pull-complete")), + "should not expire a pull-complete cookie either: {cookies:?}" + ); + } + + /// Turns the request's consent into an explicit withdrawal (GPC in a US + /// state). Viceroy has no geo, so routing alone cannot produce one. + fn withdraw(ec_state: &mut EcFinalizeState) { + let consent = ec_state.ec_context.consent_mut(); + consent.jurisdiction = + trusted_server_core::consent::jurisdiction::Jurisdiction::UsState("CA".to_owned()); + consent.gpc = true; + } + + fn set_cookies(response: &HttpResponse) -> Vec { + response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok().map(str::to_owned)) + .collect() + } + + #[test] + fn reload_without_identity_runtime_still_finalizes() { + let (state, _) = seeded_state(); + + let (response, outcome) = drive_with_runtime( + &state, + navigation(Some(&format!("ts-ec={EC_ID}"))), + None, + withdraw, + ); + + let cookies = set_cookies(&response); assert!( - result.is_none(), - "a skipped pull-sync plan should return none" + cookies + .iter() + .any(|value| value.starts_with("ts-ec=;") && value.contains("Max-Age=0")), + "should still finalize and expire ts-ec without a runtime: {cookies:?}" ); - assert_eq!(calls.get(), 0, "should not invoke the graph factory"); + let (row, _) = state + .identity_graph + .as_ref() + .expect("should hold the graph") + .get(EC_ID) + .expect("should read the row") + .expect("should keep the tombstone"); + assert!(!row.consent.ok, "should still tombstone the row"); + assert_eq!(outcome, Some(FlushOutcome::Skipped), "should skip phase 4"); } #[test] @@ -1281,4 +1781,3665 @@ mod tests { "should include sec-ch-ua-platform fallback" ); } + + /// End-to-end proof of the one-write invariant, the reads and G10 and + /// G13 (spec 13). Every test routes a request through the real router, + /// finalizes and sends it, and runs phase 4, against one counting store. + mod identity_writes { + use super::*; + use std::collections::BTreeMap; + use std::sync::Mutex; + + use trusted_server_core::ec::identity::IdentityProviderError; + use trusted_server_core::ec::identity::test_support::{ + MULTI_ENTRY_SOURCE, MultiEntryTestProvider, WriteCountingEcKv, allowing_consent, + }; + use trusted_server_core::ec::kv_backend::EcKvStore as _; + use trusted_server_core::ec::kv_types::{ + KvEid, KvEntry, KvNetwork, KvOptOut, KvPartnerId, KvProviderState, KvUid, + }; + use trusted_server_core::integrations::IntegrationRegistration; + use trusted_server_core::platform::{ + GeoInfo, PlatformError, PlatformGeo, PlatformHttpClient, PlatformHttpRequest, + PlatformPendingRequest, PlatformResponse, PlatformSelectResult, + }; + use trusted_server_core::settings::{EcPartner, ProviderMode, ProviderSettings}; + + /// The host the publisher origin answers on. + const ORIGIN_HOST: &str = "origin.test-publisher.com"; + + /// A cookie value no test treats as a real ID. + const SHARED_ID: &str = "shared-id-1"; + + /// One answer of the scripted network. + #[derive(Clone)] + struct Reply { + status: u16, + headers: Vec<(String, String)>, + body: Vec, + } + + /// One request the scripted network received. + #[derive(Clone)] + struct Seen { + host: String, + headers: edgezero_core::http::HeaderMap, + body: Vec, + } + + struct Pending { + reply: Reply, + backend_name: String, + } + + /// A network that answers per host and records what it was sent. It + /// stands in for the publisher origin, the bidders and the pull + /// partners. + struct ScriptedNetwork { + replies: Mutex>, + seen: Mutex>, + } + + impl ScriptedNetwork { + fn new() -> Self { + Self { + replies: Mutex::new(BTreeMap::new()), + seen: Mutex::new(Vec::new()), + } + } + + fn respond(&self, host: &str, status: u16, content_type: &str, body: &str) { + self.respond_with_headers(host, status, &[("content-type", content_type)], body); + } + + fn respond_with_headers( + &self, + host: &str, + status: u16, + headers: &[(&str, &str)], + body: &str, + ) { + self.replies + .lock() + .expect("should lock the replies") + .insert( + host.to_owned(), + Reply { + status, + headers: headers + .iter() + .map(|(name, value)| ((*name).to_owned(), (*value).to_owned())) + .collect(), + body: body.as_bytes().to_vec(), + }, + ); + } + + /// Forgets the answer for `host`, so a call to it fails in transport. + fn clear_reply(&self, host: &str) { + self.replies + .lock() + .expect("should lock the replies") + .remove(host); + } + + /// Every request that went to `host`, in order. + fn requests_to(&self, host: &str) -> Vec { + self.seen + .lock() + .expect("should lock the requests") + .iter() + .filter(|seen| seen.host == host) + .cloned() + .collect() + } + + /// Records `request` and returns its answer with the backend it named. + fn record( + &self, + request: PlatformHttpRequest, + ) -> Result<(Reply, String), Report> { + let backend_name = request.backend_name; + let (parts, body) = request.request.into_parts(); + let host = parts.uri.host().unwrap_or_default().to_owned(); + self.seen + .lock() + .expect("should lock the requests") + .push(Seen { + host: host.clone(), + headers: parts.headers, + body: body + .into_bytes() + .map(|bytes| bytes.to_vec()) + .unwrap_or_default(), + }); + self.replies + .lock() + .expect("should lock the replies") + .get(&host) + .cloned() + .map(|reply| (reply, backend_name)) + .ok_or_else(|| Report::new(PlatformError::HttpClient)) + } + } + + fn platform_response( + reply: Reply, + backend_name: String, + ) -> Result> { + let mut builder = edgezero_core::http::response_builder().status(reply.status); + for (name, value) in reply.headers { + builder = builder.header(name, value); + } + builder + .body(EdgeBody::from(reply.body)) + .map(|response| PlatformResponse::new(response).with_backend_name(backend_name)) + .map_err(|_| Report::new(PlatformError::HttpClient)) + } + + #[async_trait::async_trait(?Send)] + impl PlatformHttpClient for ScriptedNetwork { + async fn send( + &self, + request: PlatformHttpRequest, + ) -> Result> { + let (reply, backend_name) = self.record(request)?; + platform_response(reply, backend_name) + } + + async fn send_async( + &self, + request: PlatformHttpRequest, + ) -> Result> { + let (reply, backend_name) = self.record(request)?; + Ok(PlatformPendingRequest::new(Pending { + reply, + backend_name: backend_name.clone(), + }) + .with_backend_name(backend_name)) + } + + async fn select( + &self, + mut pending_requests: Vec, + ) -> Result> { + if pending_requests.is_empty() { + return Err(Report::new(PlatformError::HttpClient)); + } + let first = pending_requests.remove(0); + let ready = match first.downcast::() { + Ok(pending) => platform_response(pending.reply, pending.backend_name), + Err(_) => Err(Report::new(PlatformError::HttpClient)), + }; + Ok(PlatformSelectResult { + ready, + remaining: pending_requests, + failed_backend_name: None, + }) + } + } + + /// Locates every client in a country no privacy law of the default + /// consent config covers, so EC and EID consent is granted. + struct NonRegulatedGeo; + + impl PlatformGeo for NonRegulatedGeo { + fn lookup( + &self, + _client_ip: Option, + ) -> Result, Report> { + Ok(Some(GeoInfo { + city: "Example City".to_owned(), + country: "CA".to_owned(), + continent: "NA".to_owned(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + })) + } + } + + /// What every scenario shares: the router state, the counting store + /// behind its identity graph and the scripted network. + struct Scenario { + state: Arc, + store: Arc, + network: Arc, + } + + /// The partner of `source`, registered for the bidstream with no pull. + fn partner_toml(source: &str) -> String { + format!( + r#" + [[ec.partners]] + name = "Partner {source}" + source_domain = "{source}" + bidstream_enabled = true + api_token = "token-for-{source}-0123456789abcdef" + "# + ) + } + + impl Scenario { + /// A scenario whose settings are the test settings plus `extra` + /// TOML, with `sharedid.org` and `id5-sync.com` registered. + fn with_settings(extra: &str) -> Self { + let toml = format!( + "{TEST_SETTINGS_TOML}{}{}{extra}", + partner_toml("sharedid.org"), + partner_toml("id5-sync.com"), + ); + Self::from_toml(&toml) + } + + fn new() -> Self { + Self::with_settings("") + } + + fn from_toml(toml: &str) -> Self { + let (graph, store) = KvIdentityGraph::write_counting("identity-writes-store"); + Self::from_parts(toml, graph, store) + } + + /// A scenario over `graph`, which must count into `store`. + fn from_parts( + toml: &str, + graph: KvIdentityGraph, + store: Arc, + ) -> Self { + let mut settings = Settings::from_toml(toml).expect("should parse the settings"); + // These scenarios exercise the mirror; its default-off gate + // has its own test. + settings.ec.identity.mirror_enabled = true; + let network = Arc::new(ScriptedNetwork::new()); + network.respond( + ORIGIN_HOST, + 200, + "text/html", + "page", + ); + let state = crate::app::app_state_with_network_for_test( + settings, + graph, + Arc::clone(&network) as Arc, + Arc::new(NonRegulatedGeo), + ); + Self { + state, + store, + network, + } + } + + fn graph(&self) -> &KvIdentityGraph { + self.state + .identity_graph + .as_ref() + .expect("should hold the graph") + } + + /// Stores `entry` under [`EC_ID`] and zeroes the counts. + fn seed(&self, entry: &KvEntry) { + self.graph() + .create(EC_ID, entry) + .expect("should seed the row"); + self.store.reset_counts(); + } + + /// Seeds a live row that already holds its cluster size, so the + /// flush has no change of its own to write (C2). + fn seed_live(&self) -> KvEntry { + let entry = live_entry(); + self.seed(&entry); + entry + } + + /// Drives `request`, and checks that no response of any scenario + /// sets a server identity cookie (spec 13). + fn drive(&self, request: HttpRequest) -> (HttpResponse, Option) { + let (response, outcome) = drive_request_for_test(&self.state, request); + assert_no_server_identity_cookie(&response); + (response, outcome) + } + + /// The row under `ec_id`. + fn row_of(&self, ec_id: &str) -> KvEntry { + self.graph() + .get(ec_id) + .expect("should read the row") + .expect("should hold the row") + .0 + } + + /// The row under [`EC_ID`]. + fn row(&self) -> KvEntry { + self.row_of(EC_ID) + } + } + + /// A live row for [`EC_ID`] with its cluster size present. + fn live_entry() -> KvEntry { + let mut entry = KvEntry::new(&allowing_consent(), None, 1_000, "test-publisher.com"); + entry.network = Some(KvNetwork { + cluster_size: Some(1), + }); + entry + } + + fn with_header(mut request: HttpRequest, name: &'static str, value: &str) -> HttpRequest { + request.headers_mut().insert( + name, + HeaderValue::from_str(value).expect("should encode the header"), + ); + request + } + + fn without_header(mut request: HttpRequest, name: &'static str) -> HttpRequest { + request.headers_mut().remove(name); + request + } + + /// The cookies every response of every scenario must leave unset: + /// the server sets `ts-ec` and no other identity cookie. + fn is_server_identity_cookie(name: &str) -> bool { + name == "ts-eids" + || name == "__Host-ts-eids-s" + || name.starts_with("ts-eids-r") + || name == "ts-ec-pull-complete" + } + + fn cookie_name(set_cookie: &str) -> &str { + set_cookie.split('=').next().unwrap_or_default().trim() + } + + fn assert_no_server_identity_cookie(response: &HttpResponse) { + for cookie in set_cookies(response) { + assert!( + !is_server_identity_cookie(cookie_name(&cookie)), + "should set no identity cookie but ts-ec: {cookie}" + ); + } + } + + /// The value of the `name` cookie the response sets. + fn set_cookie_value(response: &HttpResponse, name: &str) -> Option { + set_cookies(response).into_iter().find_map(|cookie| { + let pair = cookie.split(';').next()?; + let (found, value) = pair.split_once('=')?; + (found.trim() == name).then(|| value.trim().to_owned()) + }) + } + + /// The cookie of a returning visitor whose browser holds a sharedId. + fn returning_with_shared_id() -> String { + format!("ts-ec={EC_ID}; sharedId={SHARED_ID}") + } + + /// Asserts every write mode count, so no mode hides behind another. + fn assert_writes(store: &WriteCountingEcKv, expected: (usize, usize, usize), what: &str) { + let writes = store.writes(); + assert_eq!( + (writes.add, writes.if_generation_match, writes.overwrite), + expected, + "should make exactly (Add, IfGenerationMatch, Overwrite) writes: {what}" + ); + } + + fn assert_no_writes(store: &WriteCountingEcKv, what: &str) { + assert_writes(store, (0, 0, 0), what); + } + + /// The UID the row holds for `source`. + fn stored_uid(row: &KvEntry, source: &str) -> Option { + row.ids + .get(source) + .and_then(KvPartnerId::first_uid) + .map(str::to_owned) + } + + #[test] + fn returning_visitor_with_changes_writes_once() { + let scenario = Scenario::new(); + scenario.seed_live(); + + let (response, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!(response.status(), 200, "should serve the origin page"); + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_writes(&scenario.store, (0, 1, 0), "one CAS write"); + assert_eq!( + stored_uid(&scenario.row(), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should hold the browser's sharedId in KV after the one write" + ); + } + + /// A store that refuses its first `collisions` `Add` writes without + /// storing them, as a minted ID that is already taken does. + struct CollidingStore { + inner: Arc, + collisions: Mutex, + refused: Arc, + } + + impl trusted_server_core::ec::kv_backend::EcKvStore for CollidingStore { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup( + &self, + key: &str, + ) -> Result< + Option, + Report, + > { + self.inner.lookup(key) + } + + fn key_exists(&self, key: &str) -> Result> { + self.inner.key_exists(key) + } + + fn insert( + &self, + key: &str, + write: trusted_server_core::ec::kv_backend::EcKvWrite<'_>, + ) -> Result< + trusted_server_core::ec::kv_backend::EcKvWriteOutcome, + Report, + > { + if matches!( + write.mode, + trusted_server_core::ec::kv_backend::EcKvWriteMode::Add + ) { + let mut remaining = self.collisions.lock().expect("should lock the count"); + if *remaining > 0 { + *remaining -= 1; + self.refused + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); + return Ok( + trusted_server_core::ec::kv_backend::EcKvWriteOutcome::PreconditionFailed, + ); + } + } + self.inner.insert(key, write) + } + + fn list_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result, Report> { + self.inner.list_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + self.inner.delete(key) + } + } + + /// The same navigation with one header replaced. + fn navigation_with_header( + cookie: Option<&str>, + name: &'static str, + value: &str, + ) -> HttpRequest { + with_header(navigation(cookie), name, value) + } + + #[test] + fn identity_measurement_counts_the_create_write_of_a_new_ec_navigation() { + let scenario = Scenario::new(); + let cookie = format!("sharedId={SHARED_ID}"); + + let (outcome, measurement) = + drive_measured_for_test(&scenario.state, navigation(Some(&cookie))); + + assert_eq!( + measurement.route(), + RouteKind::NewEc, + "should label a first visit as a new EC" + ); + assert_eq!( + measurement.kv_writes(), + 1, + "should count the create write and no other" + ); + assert_eq!( + measurement.kv_reads(), + 0, + "should count no read: the create reads nothing" + ); + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should not flush a new visitor" + ); + } + + #[test] + fn new_visitor_navigation_writes_once() { + let scenario = Scenario::new(); + let cookie = format!("sharedId={SHARED_ID}"); + + let (response, outcome) = scenario.drive(navigation(Some(&cookie))); + + assert_eq!(response.status(), 200, "should serve the origin page"); + let minted = set_cookie_value(&response, "ts-ec").expect("should mint an EC"); + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should leave a new visitor nothing to flush" + ); + assert_writes(&scenario.store, (1, 0, 0), "one Add and no flush"); + assert_eq!( + stored_uid(&scenario.row_of(&minted), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should carry the browser ingest in the create" + ); + } + + #[test] + fn new_visitor_collision_remints_and_writes_once() { + let (_, counting) = KvIdentityGraph::write_counting("collision-store"); + let refused = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let graph = KvIdentityGraph::new(CollidingStore { + inner: Arc::clone(&counting), + collisions: Mutex::new(1), + refused: Arc::clone(&refused), + }); + let toml = format!("{TEST_SETTINGS_TOML}{}", partner_toml("sharedid.org")); + let scenario = Scenario::from_parts(&toml, graph, counting); + let cookie = format!("sharedId={SHARED_ID}"); + + let (response, _) = scenario.drive(navigation(Some(&cookie))); + + assert_eq!( + refused.load(std::sync::atomic::Ordering::SeqCst), + 1, + "control: should have met one taken ID first" + ); + assert_writes( + &scenario.store, + (1, 0, 0), + "one Add lands after the collision", + ); + let minted: Vec = set_cookies(&response) + .into_iter() + .filter(|cookie| cookie_name(cookie) == "ts-ec") + .collect(); + assert_eq!( + minted.len(), + 1, + "should set one ts-ec, for the ID that landed" + ); + let id = set_cookie_value(&response, "ts-ec").expect("should mint an EC"); + assert_eq!( + stored_uid(&scenario.row_of(&id), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should hold the ingest under the re-minted ID" + ); + } + + #[test] + fn returning_visitor_without_changes_writes_nothing() { + let scenario = Scenario::new(); + let mut entry = live_entry(); + entry.ids.insert( + "sharedid.org".to_owned(), + KvPartnerId::browser(SHARED_ID, Some(1_500)), + ); + scenario.seed(&entry); + + let (response, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!(response.status(), 200, "should serve the origin page"); + assert_eq!( + outcome, + Some(FlushOutcome::NoChange), + "should find the row already holds the ID" + ); + assert_no_writes(&scenario.store, "the browser's ID is already stored"); + + let (_, nothing) = scenario.drive(navigation(Some(&format!("ts-ec={EC_ID}")))); + assert_eq!( + nothing, + Some(FlushOutcome::Skipped), + "should skip a navigation that carries nothing" + ); + assert_no_writes(&scenario.store, "nothing to ingest"); + + let control = Scenario::new(); + control.seed_live(); + control.drive(navigation(Some(&returning_with_shared_id()))); + assert_writes( + &control.store, + (0, 1, 0), + "control: the same request writes when the row lacks the ID", + ); + } + + #[test] + fn revalidation_304_with_changes_writes_once() { + let scenario = Scenario::new(); + scenario.seed_live(); + scenario + .network + .respond_with_headers(ORIGIN_HOST, 304, &[("etag", "\"page-1\"")], ""); + let request = navigation_with_header( + Some(&returning_with_shared_id()), + "if-none-match", + "\"page-1\"", + ); + + let (response, outcome) = scenario.drive(request); + + assert_eq!(response.status(), 304, "should answer the revalidation"); + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_writes(&scenario.store, (0, 1, 0), "one CAS write"); + assert_eq!( + stored_uid(&scenario.row(), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should hold the sharedId" + ); + } + + #[test] + fn returning_navigation_answered_3xx_4xx_5xx_writes_nothing() { + let control = Scenario::new(); + control.seed_live(); + control.drive(navigation(Some(&returning_with_shared_id()))); + assert_writes(&control.store, (0, 1, 0), "control: a 200 writes once"); + + for status in [301_u16, 302, 400, 404, 410, 500, 502, 503] { + let scenario = Scenario::new(); + scenario.seed_live(); + scenario.network.respond_with_headers( + ORIGIN_HOST, + status, + &[ + ("content-type", "text/html"), + ("location", "https://origin.test-publisher.com/next"), + ], + "answer", + ); + + let (response, outcome) = + scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + response.status().as_u16(), + status, + "should pass the origin status through: {status}" + ); + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should have nothing to flush after {status}" + ); + assert_no_writes(&scenario.store, &format!("origin answered {status}")); + assert_eq!( + stored_uid(&scenario.row(), "sharedid.org"), + None, + "should not store the ID the browser sent, after {status}" + ); + } + } + + #[test] + fn new_visitor_prefetch_writes_once() { + let scenario = Scenario::new(); + + let (response, _) = scenario.drive(navigation_with_header( + Some(&format!("sharedId={SHARED_ID}")), + "sec-purpose", + "prefetch", + )); + + assert!( + set_cookie_value(&response, "ts-ec").is_some(), + "should mint an EC for a prefetched navigation" + ); + assert_writes(&scenario.store, (1, 0, 0), "one Add"); + } + + #[test] + fn returning_prefetch_writes_nothing() { + for header in ["sec-purpose", "purpose"] { + let scenario = Scenario::new(); + scenario.seed_live(); + + let (_, outcome) = scenario.drive(navigation_with_header( + Some(&returning_with_shared_id()), + header, + "prefetch", + )); + + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should not be the writer: {header}" + ); + assert_no_writes(&scenario.store, header); + } + + let control = Scenario::new(); + control.seed_live(); + control.drive(navigation(Some(&returning_with_shared_id()))); + assert_writes( + &control.store, + (0, 1, 0), + "control: the same navigation without prefetch writes once", + ); + } + + #[test] + fn new_visitor_without_sec_fetch_dest_writes_once() { + let scenario = Scenario::new(); + let request = with_header( + without_header( + navigation(Some(&format!("sharedId={SHARED_ID}"))), + "sec-fetch-dest", + ), + "accept", + "text/html,application/xhtml+xml", + ); + + let (response, _) = scenario.drive(request); + + assert!( + set_cookie_value(&response, "ts-ec").is_some(), + "should mint an EC" + ); + assert_writes(&scenario.store, (1, 0, 0), "one Add"); + } + + #[test] + fn returning_iframe_or_no_sec_fetch_dest_writes_nothing() { + let requests = [ + ( + "iframe", + navigation_with_header( + Some(&returning_with_shared_id()), + "sec-fetch-dest", + "iframe", + ), + ), + ( + "no Sec-Fetch-Dest", + with_header( + without_header( + navigation(Some(&returning_with_shared_id())), + "sec-fetch-dest", + ), + "accept", + "text/html", + ), + ), + ]; + for (name, request) in requests { + let scenario = Scenario::new(); + scenario.seed_live(); + + let (_, outcome) = scenario.drive(request); + + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should not be the writer: {name}" + ); + assert_no_writes(&scenario.store, name); + } + + let control = Scenario::new(); + control.seed_live(); + control.drive(navigation(Some(&returning_with_shared_id()))); + assert_writes( + &control.store, + (0, 1, 0), + "control: a document navigation writes once", + ); + } + + /// The settings of an enabled Didomi proxy, whose upstream answers on + /// `sdk.example.com`. + const DIDOMI_TOML: &str = r#" + [integrations.didomi] + enabled = true + sdk_origin = "https://sdk.example.com" + api_origin = "https://api.example.com" + "#; + + fn didomi_scenario() -> Scenario { + let scenario = Scenario::with_settings(DIDOMI_TOML); + scenario + .network + .respond("sdk.example.com", 200, "application/javascript", "loader()"); + scenario + } + + fn didomi_navigation(cookie: Option<&str>) -> HttpRequest { + let mut request = navigation(cookie); + *request.uri_mut() = "https://test-publisher.com/integrations/didomi/consent/loader.js" + .parse() + .expect("should parse the proxy URI"); + request + } + + #[test] + fn integration_proxy_navigation_minting_ec_writes_once() { + let scenario = didomi_scenario(); + + let (response, _) = + scenario.drive(didomi_navigation(Some(&format!("sharedId={SHARED_ID}")))); + + assert_eq!(response.status(), 200, "should serve the proxied answer"); + assert_eq!( + scenario.network.requests_to("sdk.example.com").len(), + 1, + "control: should have reached the integration, not the publisher" + ); + assert!( + scenario.network.requests_to(ORIGIN_HOST).is_empty(), + "should not reach the publisher origin" + ); + let minted = set_cookie_value(&response, "ts-ec").expect("should mint an EC"); + assert_writes(&scenario.store, (1, 0, 0), "one Add"); + assert_eq!( + stored_uid(&scenario.row_of(&minted), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should carry the ingest in the create" + ); + } + + /// An enabled auction with one bidder, and one slot for page bids. + const AUCTION_TOML: &str = r#" + [auction] + enabled = true + timeout_ms = 2000 + + [auction.providers.bidder-main] + protocol = "openrtb-2.6" + profile = "standard" + endpoint = "https://bidder.example.com/auction" + routing = "all_eligible" + + [creative_opportunities] + enabled = true + gam_network_id = "123456789" + + [[creative_opportunities.slot]] + id = "leaderboard" + gam_unit_path = "/123456789/leaderboard" + div_id = "div-gpt-ad-leaderboard" + page_patterns = ["/"] + formats = [{ width = 728, height = 90 }] + "#; + + /// A first-party asset route to a scripted origin. + const ASSET_TOML: &str = r#" + [[proxy.asset_routes]] + prefix = "/assets/" + origin_url = "https://assets.example.com" + "#; + + /// A pull partner whose answers the scripted network serves. + const PULL_PARTNER_TOML: &str = r#" + [[ec.partners]] + name = "Pull SSP" + source_domain = "ssp.example.com" + bidstream_enabled = true + pull_sync_enabled = true + pull_sync_url = "https://sync.ssp.example.com/pull" + pull_sync_allowed_domains = ["sync.ssp.example.com"] + ts_pull_token = "example-token-value-0123456789abcdef" + "#; + + /// The provider the pull partner becomes. + const PULL_PROVIDER: &str = "pull-ssp-example-com"; + + const BIDDER_HOST: &str = "bidder.example.com"; + const PULL_HOST: &str = "sync.ssp.example.com"; + + /// A scenario with the auction, the Didomi proxy, an asset route and + /// the pull partner, and a network that answers all of them. + fn full_scenario() -> Scenario { + let scenario = Scenario::with_settings(&format!( + "{AUCTION_TOML}{DIDOMI_TOML}{ASSET_TOML}{PULL_PARTNER_TOML}" + )); + scenario.network.respond( + BIDDER_HOST, + 200, + "application/json", + r#"{"id":"response","seatbid":[]}"#, + ); + scenario + .network + .respond("sdk.example.com", 200, "application/javascript", "loader()"); + scenario.network.respond( + "assets.example.com", + 200, + "application/javascript", + "asset()", + ); + scenario.network.respond( + PULL_HOST, + 200, + "application/json", + r#"{"uid":"pull-uid-1"}"#, + ); + scenario + } + + /// A scenario with the auction and no pull partner. + fn auction_scenario() -> Scenario { + let scenario = Scenario::with_settings(AUCTION_TOML); + scenario.network.respond( + BIDDER_HOST, + 200, + "application/json", + r#"{"id":"response","seatbid":[]}"#, + ); + scenario + } + + fn auction_body(eids: Option<&serde_json::Value>) -> serde_json::Value { + let mut body = serde_json::json!({ + "adUnits": [{ + "code": "div-1", + "mediaTypes": { "banner": { "sizes": [[300, 250]] } } + }] + }); + if let Some(eids) = eids { + body["eids"] = eids.clone(); + } + body + } + + /// A `/auction` request from a real browser. + fn auction_request(cookie: Option<&str>, body: &serde_json::Value) -> HttpRequest { + let mut builder = edgezero_core::http::request_builder() + .method(edgezero_core::http::Method::POST) + .uri("https://test-publisher.com/auction") + .header("content-type", "application/json"); + if let Some(cookie) = cookie { + builder = builder.header("cookie", cookie); + } + let mut request = builder + .body(EdgeBody::from( + serde_json::to_vec(body).expect("should serialize the body"), + )) + .expect("should build request"); + as_browser(&mut request); + request + } + + /// A browser's `/_ts/page-bids` call for the home page. + fn page_bids_request(cookie: &str) -> HttpRequest { + let mut request = edgezero_core::http::request_builder() + .method(edgezero_core::http::Method::GET) + .uri("https://test-publisher.com/_ts/page-bids?path=%2F") + .header("x-tsjs-page-bids", "1") + .header("cookie", cookie) + .body(EdgeBody::empty()) + .expect("should build request"); + as_browser(&mut request); + request + } + + /// The partner `source` asking who the visitor is. + fn identify_request(cookie: &str, source: &str) -> HttpRequest { + let mut request = edgezero_core::http::request_builder() + .method(edgezero_core::http::Method::GET) + .uri("https://test-publisher.com/_ts/api/v1/identify") + .header( + "authorization", + format!("Bearer token-for-{source}-0123456789abcdef"), + ) + .header("cookie", cookie) + .header("origin", "https://www.test-publisher.com") + .header("sec-fetch-site", "same-site") + .body(EdgeBody::empty()) + .expect("should build request"); + as_browser(&mut request); + request + } + + fn asset_request(cookie: &str) -> HttpRequest { + let mut request = edgezero_core::http::request_builder() + .method(edgezero_core::http::Method::GET) + .uri("https://test-publisher.com/assets/app.js") + .header("sec-fetch-dest", "script") + .header("cookie", cookie) + .body(EdgeBody::empty()) + .expect("should build request"); + as_browser(&mut request); + request + } + + fn json_of(response: HttpResponse) -> serde_json::Value { + let bytes = response + .into_body() + .into_bytes() + .expect("should buffer the body"); + serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null) + } + + /// The last request body the bidder received. + fn bidder_body(scenario: &Scenario) -> serde_json::Value { + let seen = scenario + .network + .requests_to(BIDDER_HOST) + .pop() + .expect("should have called the bidder"); + serde_json::from_slice(&seen.body).expect("should parse the bidder request") + } + + /// The EIDs of a bidder request, from `user.ext.eids`. + fn bidder_eids(body: &serde_json::Value) -> serde_json::Value { + body["user"]["ext"]["eids"].clone() + } + + /// What `x-ts-eids` carries. + fn header_eids(response: &HttpResponse) -> serde_json::Value { + use base64::Engine as _; + let value = response + .headers() + .get("x-ts-eids") + .and_then(|value| value.to_str().ok()) + .expect("should carry x-ts-eids"); + let bytes = base64::engine::general_purpose::STANDARD + .decode(value) + .expect("should decode x-ts-eids"); + serde_json::from_slice(&bytes).expect("should parse x-ts-eids") + } + + /// The decoded `ts-eids-s` payload of an `/auction` body, or its + /// `clear` marker. + fn mirror_of(body: &serde_json::Value) -> serde_json::Value { + use base64::Engine as _; + let member = &body["ext"]["ts_eids_s"]; + match member["value"].as_str() { + Some(value) => { + let bytes = base64::engine::general_purpose::URL_SAFE_NO_PAD + .decode(value) + .expect("should decode the mirror"); + serde_json::from_slice(&bytes).expect("should parse the mirror") + } + None => member.clone(), + } + } + + #[test] + fn auction_page_bids_identify_proxy_and_assets_write_nothing() { + let scenario = full_scenario(); + scenario.seed_live(); + let cookie = returning_with_shared_id(); + let body = auction_body(None); + + let requests = [ + ("auction", auction_request(Some(&cookie), &body)), + ("page bids", page_bids_request(&cookie)), + ("identify", identify_request(&cookie, "sharedid.org")), + ("integration proxy subresource", { + let mut request = didomi_navigation(Some(&cookie)); + request + .headers_mut() + .insert("sec-fetch-dest", HeaderValue::from_static("script")); + request + }), + ("asset", asset_request(&cookie)), + ]; + for (name, request) in requests { + let (response, _) = scenario.drive(request); + + assert!( + response.status().is_success(), + "should answer {name} with a success, got {}", + response.status() + ); + assert_no_writes(&scenario.store, name); + } + assert_eq!( + scenario.network.requests_to(BIDDER_HOST).len(), + 2, + "control: the auction and the page bids should both have reached the bidder" + ); + for host in ["sdk.example.com", "assets.example.com"] { + assert_eq!( + scenario.network.requests_to(host).len(), + 1, + "control: the proxy and the asset route should each have reached {host}" + ); + } + + let control = full_scenario(); + control.seed_live(); + control.drive(navigation(Some(&cookie))); + assert_writes( + &control.store, + (0, 1, 0), + "control: a navigation with the same cookies writes once", + ); + } + + #[test] + fn returning_writer_navigation_reads_once() { + let changed = Scenario::new(); + changed.seed_live(); + + changed.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + changed.store.writes().lookups, + 1, + "should read once, with a write" + ); + assert_writes(&changed.store, (0, 1, 0), "one CAS write"); + + let unchanged = Scenario::new(); + let mut entry = live_entry(); + entry.ids.insert( + "sharedid.org".to_owned(), + KvPartnerId::browser(SHARED_ID, Some(1_500)), + ); + unchanged.seed(&entry); + + unchanged.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + unchanged.store.writes().lookups, + 1, + "should read once, with nothing to write" + ); + assert_no_writes(&unchanged.store, "the row already holds the ID"); + + unchanged.drive(navigation(Some(&returning_with_shared_id()))); + assert_eq!( + unchanged.store.writes().lookups, + 2, + "control: the counter should count each navigation's read" + ); + } + + #[test] + fn due_after_send_provider_adds_one_reread() { + let scenario = full_scenario(); + scenario.seed_live(); + + let (_, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_eq!( + scenario.network.requests_to(PULL_HOST).len(), + 1, + "should call the due pull provider after send" + ); + assert_eq!( + scenario.store.writes().lookups, + 2, + "should read in phase 1 and once more before dispatching" + ); + assert_writes( + &scenario.store, + (0, 1, 0), + "the browser ingest and the pull in one write", + ); + let row = scenario.row(); + assert_eq!( + stored_uid(&row, "sharedid.org").as_deref(), + Some(SHARED_ID), + "should hold the browser ID" + ); + assert_eq!( + stored_uid(&row, "ssp.example.com").as_deref(), + Some("pull-uid-1"), + "should hold the pulled ID" + ); + + let idle = full_scenario(); + let mut entry = live_entry(); + entry.ps.insert( + PULL_PROVIDER.to_owned(), + trusted_server_core::ec::kv_types::KvProviderState { + n: Some(4_000_000_000), + ..Default::default() + }, + ); + idle.seed(&entry); + + idle.drive(navigation(Some(&returning_with_shared_id()))); + + assert!( + idle.network.requests_to(PULL_HOST).is_empty(), + "control: should not call a provider that is not due" + ); + assert_eq!( + idle.store.writes().lookups, + 1, + "control: should read once when no provider is due" + ); + } + + #[test] + fn auction_and_page_bids_keep_one_read() { + let cookie = returning_with_shared_id(); + let auction = full_scenario(); + auction.seed_live(); + + auction.drive(auction_request(Some(&cookie), &auction_body(None))); + + assert_eq!( + auction.store.writes().lookups, + 1, + "/auction should read once" + ); + assert_no_writes(&auction.store, "/auction"); + assert_eq!( + auction.network.requests_to(BIDDER_HOST).len(), + 1, + "control: the auction should have run" + ); + + let page_bids = full_scenario(); + page_bids.seed_live(); + + page_bids.drive(page_bids_request(&cookie)); + + assert_eq!( + page_bids.store.writes().lookups, + 1, + "/_ts/page-bids should read once" + ); + assert_no_writes(&page_bids.store, "/_ts/page-bids"); + assert_eq!( + page_bids.network.requests_to(BIDDER_HOST).len(), + 1, + "control: the page-bids auction should have run" + ); + } + + #[test] + fn same_origin_page_bids_is_the_spa_writer_once_the_interval_passes() { + let spa_request = || { + let mut request = page_bids_request(&returning_with_shared_id()); + request.headers_mut().insert( + "sec-fetch-site", + edgezero_core::http::HeaderValue::from_static("same-origin"), + ); + request + }; + + let due = full_scenario(); + due.seed_live(); + let (_, outcome) = due.drive(spa_request()); + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should write once as the SPA writer" + ); + assert_eq!( + stored_uid(&due.row(), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should ingest the browser's sharedId" + ); + assert_eq!( + due.network.requests_to(PULL_HOST).len(), + 1, + "should run the due after-send provider" + ); + + let recent = full_scenario(); + let mut row = live_entry(); + row.wa = Some( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("should read the clock") + .as_secs(), + ); + recent.seed(&row); + let (_, outcome) = recent.drive(spa_request()); + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should wait out the interval since the last writer flush" + ); + assert_no_writes(&recent.store, "an SPA call within the interval"); + } + + #[test] + fn no_response_sets_an_identity_cookie_other_than_ts_ec() { + assert!( + is_server_identity_cookie("ts-eids") + && is_server_identity_cookie("__Host-ts-eids-s") + && is_server_identity_cookie("ts-eids-r1") + && is_server_identity_cookie("ts-ec-pull-complete") + && !is_server_identity_cookie("ts-ec"), + "control: the check should name exactly the cookies the server never sets" + ); + let browser_cookies = format!( + "{}; ts-eids=ignored; __Host-ts-eids-s=ignored; ts-eids-r1=ignored; \ + ts-ec-pull-complete=v1.example", + returning_with_shared_id() + ); + + let new_visitor = full_scenario(); + let (minted, _) = new_visitor.drive(navigation(Some("sharedId=shared-id-1"))); + assert!( + set_cookie_value(&minted, "ts-ec").is_some(), + "control: a new visitor's response should set ts-ec" + ); + + let returning = full_scenario(); + returning.seed_live(); + let body = auction_body(None); + for request in [ + navigation(Some(&browser_cookies)), + auction_request(Some(&browser_cookies), &body), + page_bids_request(&browser_cookies), + identify_request(&browser_cookies, "sharedid.org"), + ] { + let (response, _) = returning.drive(request); + let names: Vec = set_cookies(&response) + .iter() + .map(|cookie| cookie_name(cookie).to_owned()) + .collect(); + assert!( + names.iter().all(|name| name == "ts-ec"), + "should set at most ts-ec: {names:?}" + ); + } + } + + /// A source `id5-sync.com` entry set with two entries, three UIDs and + /// every UID-level and EID-level field. + fn full_entry_set() -> Vec { + use trusted_server_core::ec::kv_types::{KvEid, KvUid}; + vec![ + KvEid { + uids: vec![ + KvUid { + id: "set-uid-a".to_owned(), + atype: Some(1), + ext: Some(serde_json::json!({ "rtiPartner": "A" })), + }, + KvUid { + id: "set-uid-b".to_owned(), + atype: Some(3), + ext: Some(serde_json::json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("inserter-a.example.com".to_owned()), + matcher: Some("matcher-a.example.com".to_owned()), + mm: Some(3), + ext: Some(serde_json::json!({ "segments": ["s1", "s2"] })), + }, + KvEid { + uids: vec![KvUid { + id: "set-uid-c".to_owned(), + atype: Some(1), + ext: Some(serde_json::json!({ "label": "c" })), + }], + inserter: Some("inserter-b.example.com".to_owned()), + matcher: Some("matcher-b.example.com".to_owned()), + mm: Some(1), + ext: Some(serde_json::json!({ "tier": "gold" })), + }, + ] + } + + /// What bidders receive for [`full_entry_set`]. + fn full_entry_set_eids() -> serde_json::Value { + serde_json::json!([ + { + "source": "id5-sync.com", + "inserter": "inserter-a.example.com", + "matcher": "matcher-a.example.com", + "mm": 3, + "ext": { "segments": ["s1", "s2"] }, + "uids": [ + { "id": "set-uid-a", "atype": 1, "ext": { "rtiPartner": "A" } }, + { "id": "set-uid-b", "atype": 3, "ext": { "stype": "ppuid" } } + ] + }, + { + "source": "id5-sync.com", + "inserter": "inserter-b.example.com", + "matcher": "matcher-b.example.com", + "mm": 1, + "ext": { "tier": "gold" }, + "uids": [{ "id": "set-uid-c", "atype": 1, "ext": { "label": "c" } }] + } + ]) + } + + /// A live row holding `eids` for `id5-sync.com`, with no writer. + fn row_with_set(exp: Option) -> KvEntry { + let mut entry = live_entry(); + entry.ids.insert( + "id5-sync.com".to_owned(), + KvPartnerId { + at: Some(1_500), + exp, + eids: full_entry_set(), + ..KvPartnerId::default() + }, + ); + entry + } + + /// A browser entry of a source no partner registers. + fn browser_entry() -> serde_json::Value { + serde_json::json!({ + "source": "browser.example.com", + "uids": [{ "id": "browser-uid", "atype": 1 }] + }) + } + + const FAR_FUTURE: u64 = 4_000_000_000; + + fn ec_cookie() -> String { + format!("ts-ec={EC_ID}") + } + + #[test] + fn entry_set_round_trips_from_kv_to_bidders() { + let scenario = auction_scenario(); + scenario.seed(&row_with_set(Some(FAR_FUTURE))); + let body = auction_body(Some(&serde_json::json!([browser_entry()]))); + + let (response, _) = scenario.drive(auction_request(Some(&ec_cookie()), &body)); + + let mut expected = vec![browser_entry()]; + expected.extend( + full_entry_set_eids() + .as_array() + .expect("should be a list") + .clone(), + ); + let expected = serde_json::Value::Array(expected); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + expected, + "should send the browser entry, then both stored entries in order, every field intact" + ); + assert_eq!( + header_eids(&response), + expected, + "x-ts-eids should carry what was sent" + ); + let mirror = mirror_of(&json_of(response)); + assert_eq!( + mirror, + serde_json::json!({ "v": 1, "eids": full_entry_set_eids() }), + "the mirror should hold the same stored entries and no browser entry" + ); + assert_no_writes(&scenario.store, "/auction"); + } + + #[test] + fn entry_set_survives_an_unrelated_flush() { + let scenario = Scenario::new(); + let seeded = row_with_set(Some(FAR_FUTURE)); + scenario.seed(&seeded); + + let (_, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should flush the sharedId" + ); + assert_writes(&scenario.store, (0, 1, 0), "one CAS write"); + let row = scenario.row(); + assert_eq!( + stored_uid(&row, "sharedid.org").as_deref(), + Some(SHARED_ID), + "control: the flush should have written the sharedId fill" + ); + assert_eq!( + row.ids.get("id5-sync.com"), + seeded.ids.get("id5-sync.com"), + "should leave the stored entry set exactly as it was" + ); + } + + #[test] + fn expired_entry_set_is_never_served_or_mirrored() { + let live = auction_scenario(); + live.seed(&row_with_set(Some(FAR_FUTURE))); + let (live_response, _) = + live.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + assert_eq!( + bidder_eids(&bidder_body(&live)), + full_entry_set_eids(), + "control: an unexpired set should be served" + ); + assert_ne!( + mirror_of(&json_of(live_response)), + serde_json::json!({ "clear": true }), + "control: an unexpired set should be mirrored" + ); + + let expired = auction_scenario(); + expired.seed(&row_with_set(Some(1_500))); + let (response, _) = + expired.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + assert_eq!( + bidder_eids(&bidder_body(&expired)), + serde_json::Value::Null, + "should send no EIDs from an expired set" + ); + assert_eq!( + mirror_of(&json_of(response)), + serde_json::json!({ "clear": true }), + "should mirror nothing for an expired set" + ); + } + + /// A version 1 row: `uid` records and no entry sets. + fn seed_version_1_row(scenario: &Scenario) { + scenario.seed(&live_entry()); + let lookup = scenario + .store + .lookup(EC_ID) + .expect("should read the row") + .expect("should hold the row"); + let body = serde_json::json!({ + "v": 1, + "created": 1_000, + "consent": { "ok": true, "updated": 1_000 }, + "geo": { "country": "CA" }, + "network": { "cluster_size": 1 }, + "ids": { "id5-sync.com": { "uid": "version-1-uid", "at": 1_500 } } + }) + .to_string(); + let metadata = lookup + .metadata + .map(|bytes| String::from_utf8(bytes).expect("should be UTF-8")) + .unwrap_or_else(|| "{}".to_owned()); + scenario + .store + .insert( + EC_ID, + trusted_server_core::ec::kv_backend::EcKvWrite { + body: &body, + metadata: &metadata, + ttl: std::time::Duration::from_secs(3_600), + mode: trusted_server_core::ec::kv_backend::EcKvWriteMode::Overwrite, + }, + ) + .expect("should store the version 1 row"); + scenario.store.reset_counts(); + } + + fn stored_body(scenario: &Scenario) -> serde_json::Value { + let lookup = scenario + .store + .lookup(EC_ID) + .expect("should read the row") + .expect("should hold the row"); + serde_json::from_slice(&lookup.body).expect("should parse the stored row") + } + + #[test] + fn version_1_row_is_served_and_upgraded_by_the_next_write() { + let scenario = auction_scenario(); + seed_version_1_row(&scenario); + + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + serde_json::json!([{ + "source": "id5-sync.com", + "uids": [{ + "id": "version-1-uid", + "atype": trusted_server_core::settings::EcPartner::default_openrtb_atype() + }] + }]), + "should serve the version 1 record as one entry with one UID" + ); + assert_no_writes(&scenario.store, "/auction"); + assert_eq!( + stored_body(&scenario)["v"], + 1, + "control: the auction should leave the row at version 1" + ); + + let (_, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should flush the sharedId" + ); + assert_writes(&scenario.store, (0, 1, 0), "one CAS write"); + let written = stored_body(&scenario); + assert_eq!(written["v"], 2, "should write version 2"); + let record = &written["ids"]["id5-sync.com"]; + assert!( + record.get("uid").is_none(), + "should drop the version 1 uid field: {record}" + ); + assert_eq!( + record["eids"][0]["uids"][0]["id"], "version-1-uid", + "should keep the UID as an entry" + ); + } + + #[test] + fn generic_pull_uid_written_after_send_is_served_on_the_next_auction() { + let scenario = full_scenario(); + scenario.seed_live(); + + let (_, outcome) = scenario.drive(navigation(Some(&ec_cookie()))); + + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should flush the pull" + ); + let record = scenario.row().ids.get("ssp.example.com").cloned(); + assert_eq!( + record.as_ref().and_then(|record| record.w.as_deref()), + Some(PULL_PROVIDER), + "should record the pull provider as the writer" + ); + + let (response, _) = + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + let atype = trusted_server_core::settings::EcPartner::default_openrtb_atype(); + let served = serde_json::json!([{ + "source": "ssp.example.com", + "uids": [{ "id": "pull-uid-1", "atype": atype }] + }]); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + served, + "should serve the pulled UID to bidders" + ); + assert_eq!( + mirror_of(&json_of(response)), + serde_json::json!({ "v": 1, "eids": served }), + "should mirror the pulled UID" + ); + } + + /// As [`drive_with_runtime`], running `after_send` between the send + /// and phase 4, where another request can land. + fn drive_with_send_hook( + state: &Arc, + request: HttpRequest, + after_send: &mut dyn FnMut(), + ) -> Option { + let router = TrustedServerApp::routes_for_state(state); + let mut response = futures::executor::block_on(router.oneshot(request)) + .expect("should route the request"); + let mut ec_state = response + .extensions_mut() + .remove::() + .expect("should carry EC state"); + let graph = || state.identity_graph.clone(); + finalize_send_and_flush( + PostRouteInput { + runtime: Some(&state.identity), + settings: &state.settings, + ec_state: &mut ec_state, + graph: &graph, + rate_limiter: &AllowAll, + }, + response, + &mut |_| after_send(), + ) + } + + /// Withdraws consent on an `/auction` call, as a second tab can. + fn withdraw_on_auction(state: &Arc) { + let (response, _) = drive_with_runtime( + state, + auction_request(Some(&ec_cookie()), &auction_body(None)), + Some(&state.identity), + withdraw, + ); + assert_no_server_identity_cookie(&response); + assert!( + set_cookies(&response) + .iter() + .any(|cookie| cookie.starts_with("ts-ec=;") && cookie.contains("Max-Age=0")), + "control: the withdrawal should expire ts-ec: {:?}", + set_cookies(&response) + ); + } + + #[test] + fn withdrawal_on_auction_after_navigation_send_makes_phase_four_drop_and_call_nothing() { + let control = full_scenario(); + control.seed_live(); + let (_, outcome) = control.drive(navigation(Some(&returning_with_shared_id()))); + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "control: should flush" + ); + assert_eq!( + control.network.requests_to(PULL_HOST).len(), + 1, + "control: should call the due provider" + ); + + let scenario = full_scenario(); + scenario.seed_live(); + let state = Arc::clone(&scenario.state); + + let outcome = drive_with_send_hook( + &state, + navigation(Some(&returning_with_shared_id())), + &mut || withdraw_on_auction(&state), + ); + + assert_eq!( + outcome, + Some(FlushOutcome::Dropped("tombstone")), + "should drop the navigation's mutation" + ); + assert!( + scenario.network.requests_to(PULL_HOST).is_empty(), + "should call no provider after the tombstone" + ); + assert_writes( + &scenario.store, + (1, 1, 0), + "the withdrawal's marker and tombstone only, nothing from phase 4", + ); + let row = scenario.row(); + assert!(!row.consent.ok, "should leave the tombstone"); + assert!(row.ids.is_empty(), "should hold no ID after the tombstone"); + } + + #[test] + fn withdrawal_returns_clear_for_the_mirror() { + let scenario = auction_scenario(); + scenario.seed(&row_with_set(Some(FAR_FUTURE))); + let (before, _) = + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + assert_eq!( + mirror_of(&json_of(before)), + serde_json::json!({ "v": 1, "eids": full_entry_set_eids() }), + "control: a live row should be mirrored" + ); + + withdraw_on_auction(&scenario.state); + let (after, _) = + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + assert_eq!( + mirror_of(&json_of(after)), + serde_json::json!({ "clear": true }), + "should tell the shim to expire the mirror" + ); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + serde_json::Value::Null, + "should send bidders no ID from a tombstoned row" + ); + } + + #[test] + fn consent_withdrawal_writes_tombstone_and_marker() { + // (refused conditional writes, tombstone CAS attempts, marker + // writes, whether the row ends tombstoned) + let cases = [ + (0_usize, 1_usize, 1_usize, true), + (5, 6, 1, true), + (10, 10, 0, false), + ]; + for (refused, tombstone_attempts, markers, tombstoned) in cases { + let scenario = Scenario::new(); + scenario.seed_live(); + if refused > 0 { + scenario.store.refuse_conditional_writes(EC_ID, refused); + } + + let (response, outcome) = drive_with_runtime( + &scenario.state, + navigation(Some(&ec_cookie())), + Some(&scenario.state.identity), + withdraw, + ); + + assert_no_server_identity_cookie(&response); + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should flush nothing after a withdrawal: {refused} refused" + ); + assert_writes( + &scenario.store, + (markers, tombstone_attempts, 0), + &format!("{refused} refused writes: the marker, then every tombstone attempt"), + ); + assert_eq!( + !scenario.row().consent.ok, + tombstoned, + "should end tombstoned only when an attempt landed: {refused} refused" + ); + } + } + + #[test] + fn cas_conflict_on_flush_is_one_attempt() { + let scenario = Scenario::new(); + scenario.seed_live(); + scenario.store.refuse_conditional_writes(EC_ID, 5); + + let (_, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + outcome, + Some(FlushOutcome::WriteFailed { + removals: trusted_server_core::ec::identity::flush::RemovalWriteOutcome::NothingToWrite + }), + "should fail the flush with no removal to write" + ); + assert_writes( + &scenario.store, + (0, 1, 0), + "one refused attempt, and no retry although four more would be refused", + ); + + let control = Scenario::new(); + control.seed_live(); + control.drive(navigation(Some(&returning_with_shared_id()))); + assert_writes(&control.store, (0, 1, 0), "control: one attempt that lands"); + } + + #[test] + fn refused_flush_is_rederived_by_the_next_navigation() { + let scenario = Scenario::new(); + scenario.seed_live(); + scenario.store.refuse_conditional_writes(EC_ID, 1); + + scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + stored_uid(&scenario.row(), "sharedid.org"), + None, + "should not hold the ID after the refused write" + ); + assert_writes(&scenario.store, (0, 1, 0), "the refused attempt"); + + let (_, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should write the second time" + ); + assert_writes(&scenario.store, (0, 2, 0), "one attempt per navigation"); + assert_eq!( + stored_uid(&scenario.row(), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should hold the ingest the second navigation re-derived from the same cookies" + ); + } + + #[test] + fn deleted_row_is_found_by_the_next_writer_and_rotated() { + let scenario = Scenario::new(); + scenario.seed_live(); + scenario + .graph() + .delete(EC_ID) + .expect("should delete the row out of band"); + scenario.store.reset_counts(); + + let (response, _) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + let rotated = set_cookie_value(&response, "ts-ec").expect("should set the new EC"); + assert_ne!(rotated, EC_ID, "should rotate to a new ID"); + assert_writes( + &scenario.store, + (1, 0, 0), + "one Add creates the replacement", + ); + assert_eq!( + stored_uid(&scenario.row_of(&rotated), "sharedid.org").as_deref(), + Some(SHARED_ID), + "should carry the ingest in the replacement row" + ); + assert!( + scenario.graph().get(EC_ID).expect("should read").is_none(), + "should not resurrect the deleted ID" + ); + } + + #[test] + fn stager_that_sees_missing_row_returns_clear_and_next_writer_recovers() { + let live = auction_scenario(); + live.seed(&row_with_set(Some(FAR_FUTURE))); + let (served, _) = live.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + assert_ne!( + mirror_of(&json_of(served)), + serde_json::json!({ "clear": true }), + "control: a live row should be mirrored" + ); + + let scenario = auction_scenario(); + let (response, _) = + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + assert_eq!( + mirror_of(&json_of(response)), + serde_json::json!({ "clear": true }), + "should return clear for a missing row" + ); + assert_no_writes(&scenario.store, "the stager stores nothing"); + assert!( + scenario.graph().get(EC_ID).expect("should read").is_none(), + "should not create the row" + ); + + let (navigated, _) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + let rotated = set_cookie_value(&navigated, "ts-ec").expect("should set the new EC"); + assert_ne!(rotated, EC_ID, "the next writer should recover by rotating"); + assert_writes(&scenario.store, (1, 0, 0), "one Add by the writer"); + } + + const REGISTERED_ONLY_COOKIE: &str = + include_str!("../../trusted-server-js/lib/test/fixtures/ts-eids-registered-only.txt"); + const ORTB2_ONLY_COOKIE: &str = + include_str!("../../trusted-server-js/lib/test/fixtures/ts-eids-ortb2-only.txt"); + + /// A provider that answers every call by invalidating the ID of its + /// one source, so the flush carries a removal. + struct InvalidatingProvider { + descriptor: trusted_server_core::ec::identity::ProviderDescriptor, + source: trusted_server_core::ec::identity::SourceDomain, + } + + impl InvalidatingProvider { + fn new(id: &str, source: &str) -> Self { + use trusted_server_core::ec::identity::{ProviderDescriptor, Resolves}; + let source: trusted_server_core::ec::identity::SourceDomain = + source.parse().expect("should parse the source"); + let descriptor = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_source(source.clone()) + .with_resolves( + Resolves::new().with_endpoint("https://probe.example.com/resolve"), + ) + .with_consent_policy( + trusted_server_core::ec::identity::test_support::resolver_policy(), + ) + .build() + .expect("should build the descriptor"); + Self { descriptor, source } + } + } + + impl trusted_server_core::ec::identity::IdentityProvider for InvalidatingProvider { + fn descriptor(&self) -> &trusted_server_core::ec::identity::ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &trusted_server_core::ec::identity::ResolveInput<'_>, + ) -> Result< + Option, + Report, + > { + Ok(Some( + trusted_server_core::ec::identity::ProviderRequest::get(input.endpoint()?), + )) + } + + fn response( + &self, + response: &trusted_server_core::ec::identity::ProviderResponse, + _input: &trusted_server_core::ec::identity::ResolveInput<'_>, + ) -> Result< + trusted_server_core::ec::identity::Resolution, + Report, + > { + response.success_text()?; + Ok(trusted_server_core::ec::identity::Resolution::invalidate( + &self.source, + )) + } + } + + /// A scenario whose one provider invalidates the ID it wrote for + /// `ssp.example.com`, which the seeded row holds and is due to refresh. + fn invalidating_scenario() -> Scenario { + let Scenario { + state, + store, + network, + } = Scenario::new(); + let provider: Arc = Arc::new( + InvalidatingProvider::new("removal-probe", "ssp.example.com"), + ); + let runtime = IdentityRuntime::from_settings_with_providers( + &state.settings, + vec![provider], + BTreeMap::from([( + "ssp.example.com".parse().expect("should parse the source"), + "removal-probe" + .parse() + .expect("should parse the provider ID"), + )]), + ) + .expect("should build the identity runtime"); + let scenario = Scenario { + state: crate::app::app_state_with_identity_for_test(state, runtime), + store, + network, + }; + scenario + .network + .respond("probe.example.com", 200, "text/plain", "ok"); + let mut entry = live_entry(); + entry.ids.insert( + "ssp.example.com".to_owned(), + KvPartnerId { + w: Some("removal-probe".to_owned()), + rv: Some("0123456789abcdef".to_owned()), + at: Some(1_000), + ra: Some(1_500), + ..KvPartnerId::new("probe-uid") + }, + ); + scenario.seed(&entry); + scenario + } + + #[test] + fn refused_flush_writes_its_removals_alone() { + let control = invalidating_scenario(); + control.drive(navigation(Some(&returning_with_shared_id()))); + assert_writes(&control.store, (0, 1, 0), "control: one write holds both"); + let row = control.row(); + assert_eq!( + stored_uid(&row, "ssp.example.com"), + None, + "control: the removal should be in the one write" + ); + assert_eq!( + stored_uid(&row, "sharedid.org").as_deref(), + Some(SHARED_ID), + "control: so should the ingest" + ); + + let refused_once = invalidating_scenario(); + refused_once.store.refuse_conditional_writes(EC_ID, 1); + + let (_, outcome) = refused_once.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + outcome, + Some(FlushOutcome::WriteFailed { + removals: + trusted_server_core::ec::identity::flush::RemovalWriteOutcome::Written + }), + "should write the removals after the flush is refused" + ); + assert_writes( + &refused_once.store, + (0, 2, 0), + "the refused flush, then the removals alone", + ); + let row = refused_once.row(); + assert_eq!( + stored_uid(&row, "ssp.example.com"), + None, + "should have removed the invalidated ID" + ); + assert_eq!( + stored_uid(&row, "sharedid.org"), + None, + "should not write the refused ingest with the removals" + ); + + let refused_always = invalidating_scenario(); + refused_always.store.refuse_conditional_writes(EC_ID, 6); + + let (_, outcome) = refused_always.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!( + outcome, + Some(FlushOutcome::WriteFailed { + removals: trusted_server_core::ec::identity::flush::RemovalWriteOutcome::Failed + }), + "should report the removal write failed when every attempt is refused" + ); + assert_writes( + &refused_always.store, + (1, 6, 0), + "the flush once, the removals up to five times, then the pending key", + ); + assert_eq!( + stored_uid(&refused_always.row(), "ssp.example.com").as_deref(), + Some("probe-uid"), + "should leave the row as it was" + ); + + let (_, next) = refused_always.drive(navigation(Some(&format!("ts-ec={EC_ID}")))); + + assert_eq!( + next, + Some(FlushOutcome::Written), + "should merge the pending key on the next writer" + ); + assert_eq!( + stored_uid(&refused_always.row(), "ssp.example.com"), + None, + "should apply the removal the earlier flush missed" + ); + } + + /// A scenario whose Didomi proxy answers `body`, with a capture-only + /// provider listening on that proxy's path and a row holding a + /// browser-written `ssp.example.com` ID for it to invalidate. + fn capture_scenario(body: &str) -> Scenario { + capture_scenario_with(body, false) + } + + /// As [`capture_scenario`], with the provider's `revoke_scope` set to + /// `"global"` when `global`. The settings are changed after the app + /// state is built, as config validation knows no capture-only + /// provider. + fn capture_scenario_with(body: &str, global: bool) -> Scenario { + let Scenario { + mut state, + store, + network, + } = Scenario::with_settings(DIDOMI_TOML); + if global { + let owned = Arc::get_mut(&mut state).expect("should own the fresh state"); + Arc::make_mut(&mut owned.settings) + .ec + .identity + .providers + .insert( + "didomi".to_owned(), + trusted_server_core::settings::ProviderSettings { + revoke_scope: Some(trusted_server_core::settings::RevokeScope::Global), + ..trusted_server_core::settings::ProviderSettings::default() + }, + ); + } + let provider = + trusted_server_core::ec::identity::test_support::answering_capture_provider( + "didomi", + &["ssp.example.com"], + None, + ); + let runtime = IdentityRuntime::from_settings_with_providers( + &state.settings, + vec![provider], + BTreeMap::new(), + ) + .expect("should build the identity runtime"); + let scenario = Scenario { + state: crate::app::app_state_with_identity_for_test(state, runtime), + store, + network, + }; + scenario + .network + .respond("sdk.example.com", 200, "application/json", body); + let mut entry = live_entry(); + entry.ids.insert( + "ssp.example.com".to_owned(), + KvPartnerId { + rv: Some("0123456789abcdef".to_owned()), + at: Some(1_000), + ..KvPartnerId::new("probe-uid") + }, + ); + scenario.seed(&entry); + scenario + } + + const CAPTURE_INVALIDATION: &str = r#"{"invalidate":["ssp.example.com"]}"#; + + #[test] + fn capture_removal_is_a_separate_write_beside_the_one_flush() { + let control = capture_scenario(r#"{}"#); + control.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &control.store, + (0, 1, 0), + "control: a proxied navigation with nothing to capture flushes once", + ); + + let scenario = capture_scenario(CAPTURE_INVALIDATION); + + let (response, outcome) = + scenario.drive(didomi_navigation(Some(&returning_with_shared_id()))); + + assert_eq!(response.status(), 200, "should serve the proxied answer"); + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should still flush the writer's ingest" + ); + assert_writes( + &scenario.store, + (0, 2, 0), + "the capture removal write, then the writer's one flush", + ); + let row = scenario.row(); + assert_eq!( + stored_uid(&row, "ssp.example.com"), + None, + "should have removed the captured invalidation" + ); + assert_eq!( + stored_uid(&row, "sharedid.org").as_deref(), + Some(SHARED_ID), + "should keep the writer's own ingest in its flush" + ); + } + + #[test] + fn capture_global_revoke_tombstones_the_row_and_expires_the_ec_cookie() { + let scenario = capture_scenario_with(r#"{"opted_out":true}"#, true); + + let (response, _) = + scenario.drive(didomi_navigation(Some(&returning_with_shared_id()))); + + assert_eq!(response.status(), 200, "should serve the proxied answer"); + let row = scenario.row(); + assert!(!row.consent.ok, "should tombstone the row"); + assert!(row.ids.is_empty(), "should remove every ID"); + assert!( + set_cookies(&response).iter().any(|cookie| { + cookie_name(cookie) == "ts-ec" && cookie.contains("Max-Age=0") + }), + "should expire the EC cookie" + ); + let writes = scenario.store.writes(); + assert_eq!( + (writes.if_generation_match, writes.overwrite), + (1, 0), + "should write one tombstone and no flush" + ); + } + + #[test] + fn failed_capture_removal_write_runs_once_more_after_send() { + let scenario = capture_scenario(CAPTURE_INVALIDATION); + scenario.store.refuse_conditional_writes(EC_ID, 5); + + let (outcome, measurement) = drive_measured_for_test( + &scenario.state, + didomi_navigation(Some(&returning_with_shared_id())), + ); + + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should flush the ingest too" + ); + assert_writes( + &scenario.store, + (0, 7, 0), + "five refused attempts before send, the retry after send, the flush", + ); + assert_eq!( + stored_uid(&scenario.row(), "ssp.example.com"), + None, + "should have written the removal after send" + ); + assert_eq!(measurement.removal_lost(), 0, "should lose no removal"); + } + + #[test] + fn capture_removal_after_both_attempts_reaches_the_row_through_the_pending_key() { + let scenario = capture_scenario(CAPTURE_INVALIDATION); + scenario.store.refuse_conditional_writes(EC_ID, 10); + + let (outcome, measurement) = drive_measured_for_test( + &scenario.state, + didomi_navigation(Some(&returning_with_shared_id())), + ); + + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should flush the merge" + ); + assert_eq!(measurement.removal_lost(), 0, "should lose no removal"); + assert!( + measurement.to_string().contains("removal_deferred=1"), + "should print the deferral on the measurement line" + ); + assert_eq!( + stored_uid(&scenario.row(), "ssp.example.com"), + None, + "should apply the removal in the writer's flush" + ); + } + + #[test] + fn capture_removal_lost_after_both_attempts_is_counted() { + let scenario = capture_scenario(CAPTURE_INVALIDATION); + scenario.store.refuse_conditional_writes(EC_ID, 10); + scenario.store.fail_all_writes(&format!("pending:{EC_ID}")); + + let (_, measurement) = drive_measured_for_test( + &scenario.state, + didomi_navigation(Some(&returning_with_shared_id())), + ); + + assert_eq!( + measurement.removal_lost(), + 1, + "should count the removal lost after both attempts" + ); + assert!( + measurement.to_string().contains("removal_lost=1"), + "should print the count on the measurement line" + ); + } + + #[test] + fn capture_positive_ids_are_counted_on_the_measurement_line_and_not_stored() { + let scenario = capture_scenario( + r#"{"tokens":[{"source":"ssp.example.com","value":"captured-secret-uid"}]}"#, + ); + + let (_, measurement) = drive_measured_for_test( + &scenario.state, + didomi_navigation(Some(&returning_with_shared_id())), + ); + + let line = measurement.to_string(); + assert!( + line.contains("capture_unpersisted=didomi/ssp.example.com:1"), + "should count the positive ID: {line}" + ); + assert!( + !line.contains("captured-secret-uid"), + "should print no value: {line}" + ); + assert_eq!( + stored_uid(&scenario.row(), "ssp.example.com").as_deref(), + Some("probe-uid"), + "should not store the captured ID" + ); + } + + #[test] + fn registered_source_beyond_todays_trim_reaches_kv_on_next_navigation() { + use base64::Engine as _; + let cookie_value = REGISTERED_ONLY_COOKIE.trim_end(); + let decoded: serde_json::Value = serde_json::from_slice( + &base64::engine::general_purpose::STANDARD + .decode(cookie_value) + .expect("should decode the shim's cookie"), + ) + .expect("should parse the shim's cookie"); + let sources: Vec<&str> = decoded + .as_array() + .expect("should be a list") + .iter() + .filter_map(|entry| entry["source"].as_str()) + .collect(); + assert_eq!( + sources, + ["registered.example.com"], + "control: the shim should keep only the registered source" + ); + let scenario = Scenario::with_settings(&partner_toml("registered.example.com")); + scenario.seed_live(); + + let (_, outcome) = scenario.drive(navigation(Some(&format!( + "ts-ec={EC_ID}; ts-eids={cookie_value}" + )))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_writes(&scenario.store, (0, 1, 0), "one write"); + let row = scenario.row(); + assert_eq!( + stored_uid(&row, "registered.example.com"), + Some("x".repeat(400)), + "should hold the registered source the old trim would have dropped" + ); + assert!( + (0..7).all(|index| !row.ids.contains_key(&format!("u{index}.example.com"))), + "should hold no unregistered source: {:?}", + row.ids.keys().collect::>() + ); + } + + #[test] + fn ortb2_only_eid_reaches_kv() { + let scenario = Scenario::new(); + scenario.seed_live(); + assert_eq!( + stored_uid(&scenario.row(), "id5-sync.com"), + None, + "control: id5-sync.com should be absent from KV before the navigation" + ); + let cookie = format!("ts-ec={EC_ID}; ts-eids={}", ORTB2_ONLY_COOKIE.trim_end()); + + let (_, outcome) = scenario.drive(navigation(Some(&cookie))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_writes(&scenario.store, (0, 1, 0), "one write"); + assert_eq!( + stored_uid(&scenario.row(), "id5-sync.com").as_deref(), + Some("fictional-id5"), + "should hold the ID5 UID the shim copied from the ortb2 config" + ); + } + + /// Everything a visitor's navigation, `/auction` and `/identify` + /// produce, for comparing runs. + #[derive(Debug, PartialEq)] + struct Observed { + navigation: (u16, Option), + origin_cookie: Option, + row_uids: BTreeMap>, + bidder_request: serde_json::Value, + auction_header_eids: serde_json::Value, + auction_mirror: serde_json::Value, + identify: (u16, serde_json::Value), + writes: (usize, usize, usize), + } + + fn observe_with_mirror_cookie(mirror: Option<&str>) -> Observed { + let scenario = auction_scenario(); + let mut entry = live_entry(); + entry.ids.insert( + "id5-sync.com".to_owned(), + KvPartnerId::browser("server-id5", Some(1_500)), + ); + scenario.seed(&entry); + let mut cookie = returning_with_shared_id(); + if let Some(mirror) = mirror { + cookie.push_str(&format!("; __Host-ts-eids-s={mirror}")); + } + + let (nav, nav_outcome) = scenario.drive(navigation(Some(&cookie))); + let (auction, _) = scenario.drive(auction_request(Some(&cookie), &auction_body(None))); + let (identify, _) = scenario.drive(identify_request(&cookie, "sharedid.org")); + + let mut bidder_request = bidder_body(&scenario); + bidder_request + .as_object_mut() + .expect("should be an object") + .remove("id"); + let row = scenario.row(); + let writes = scenario.store.writes(); + let origin_cookie = scenario + .network + .requests_to(ORIGIN_HOST) + .first() + .and_then(|seen| seen.headers.get("cookie")) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + Observed { + navigation: (nav.status().as_u16(), nav_outcome), + origin_cookie, + row_uids: row + .ids + .keys() + .map(|source| (source.clone(), stored_uid(&row, source))) + .collect(), + bidder_request, + auction_header_eids: header_eids(&auction), + auction_mirror: mirror_of(&json_of(auction)), + identify: (identify.status().as_u16(), json_of(identify)), + writes: (writes.add, writes.if_generation_match, writes.overwrite), + } + } + + #[test] + fn forged_mirror_cookie_changes_nothing() { + use base64::Engine as _; + let encode = |value: &serde_json::Value| { + base64::engine::general_purpose::URL_SAFE_NO_PAD + .encode(serde_json::to_vec(value).expect("should serialize the mirror")) + }; + let forged = encode(&serde_json::json!({ + "v": 1, + "eids": [ + { "source": "sharedid.org", "uids": [{ "id": "forged-uid", "atype": 1 }] }, + { "source": "forged.example.com", "uids": [{ "id": "forged-uid", "atype": 1 }] } + ] + })); + let stale = encode(&serde_json::json!({ + "v": 1, + "eids": [{ "source": "id5-sync.com", "uids": [{ "id": "stale-id5", "atype": 1 }] }] + })); + let baseline = observe_with_mirror_cookie(None); + let shown = format!("{baseline:?}"); + assert!( + shown.contains("server-id5") && shown.contains(SHARED_ID), + "control: the baseline should hold the server and browser IDs: {shown}" + ); + assert_eq!( + baseline.writes, + (0, 1, 0), + "control: the navigation should write once" + ); + + for (name, value) in [ + ("forged", forged.as_str()), + ("stale", stale.as_str()), + ("malformed", "%%%not-base64"), + ("empty", ""), + ] { + let observed = observe_with_mirror_cookie(Some(value)); + + assert_eq!( + observed, baseline, + "should change no bidder request, answer or write: {name}" + ); + let shown = format!("{observed:?}"); + assert!( + !shown.contains("forged-uid") && !shown.contains("stale-id5"), + "should carry nothing from the {name} mirror: {shown}" + ); + } + } + + /// What the code before the identity work, run on this input, + /// stores: the first UID of the last entry of each registered + /// source, with the sharedId cookie last, and nothing for an + /// unregistered source. + const GOLDEN_INGEST_COOKIE: &str = "W3siZXh0Ijp7InNlZ21lbnRzIjpbInMxIl19LCJpbnNlcnRlciI6Imluc2VydGVyLmV4YW1wbGUuY29tIiwibW0iOjIsInNvdXJjZSI6ImlkNS1zeW5jLmNvbSIsInVpZHMiOlt7ImF0eXBlIjoxLCJpZCI6ImlkNS1maXJzdCJ9LHsiYXR5cGUiOjEsImlkIjoiaWQ1LXNlY29uZCJ9XX0seyJzb3VyY2UiOiJ1bnJlZ2lzdGVyZWQuZXhhbXBsZS5jb20iLCJ1aWRzIjpbeyJhdHlwZSI6MSwiaWQiOiJ1bnJlZy0xIn1dfSx7InNvdXJjZSI6InNoYXJlZGlkLm9yZyIsInVpZHMiOlt7ImF0eXBlIjoxLCJpZCI6InByZWJpZC1zaGFyZWQifV19LHsiaW5zZXJ0ZXIiOiJsYXRlci5leGFtcGxlLmNvbSIsIm1tIjoxLCJzb3VyY2UiOiJpZDUtc3luYy5jb20iLCJ1aWRzIjpbeyJhdHlwZSI6MSwiaWQiOiJpZDUtbGFzdC1maXJzdCJ9LHsiYXR5cGUiOjEsImlkIjoiaWQ1LWxhc3Qtc2Vjb25kIn1dfV0="; + + /// The `x-ts-eids` value the code before the identity work builds + /// for [`GOLDEN_ROW_EIDS`]. + const GOLDEN_ROW_HEADER: &str = "W3sic291cmNlIjoiaWQ1LXN5bmMuY29tIiwidWlkcyI6W3siaWQiOiJpZDUtdWlkIiwiYXR5cGUiOjN9XX0seyJzb3VyY2UiOiJzaGFyZWRpZC5vcmciLCJ1aWRzIjpbeyJpZCI6InNoYXJlZC11aWQiLCJhdHlwZSI6M31dfV0="; + + /// A body with repeated sources and EID-level fields. + const GOLDEN_BODY_EIDS: &str = r#"[ + {"inserter":"adserver.org","matcher":"adserver.org","mm":4,"source":"adserver.org","uids":[{"atype":1,"ext":{"rtiPartner":"TDID"},"id":"body-tdid"}]}, + {"ext":{"segments":["s1"]},"inserter":"other.example.com","mm":1,"source":"adserver.org","uids":[{"atype":1,"id":"body-tdid-2"}]}, + {"source":"ids.example.com","uids":[{"atype":3,"id":"ids-uid"}]} + ]"#; + + /// What the code before the identity work forwards for + /// [`GOLDEN_BODY_EIDS`] and an empty row. + const GOLDEN_BODY_FORWARDED: &str = r#"[ + {"source":"adserver.org","uids":[{"id":"body-tdid","atype":1,"ext":{"rtiPartner":"TDID"}}],"inserter":"adserver.org","matcher":"adserver.org","mm":4}, + {"source":"adserver.org","uids":[{"id":"body-tdid-2","atype":1}],"inserter":"other.example.com","mm":1,"ext":{"segments":["s1"]}}, + {"source":"ids.example.com","uids":[{"id":"ids-uid","atype":3}]} + ]"#; + + fn golden_expected_ingest() -> BTreeMap { + BTreeMap::from([ + ("id5-sync.com".to_owned(), "id5-last-first".to_owned()), + ("sharedid.org".to_owned(), "shared-1".to_owned()), + ]) + } + + /// Each source's first UID, with a check that the record is one + /// entry holding one UID and nothing else. + fn assert_one_uid_records(row: &KvEntry) -> BTreeMap { + row.ids + .iter() + .map(|(source, record)| { + assert_eq!(record.w, None, "should write no provider: {source}"); + assert_eq!(record.eids.len(), 1, "should hold one entry: {source}"); + let entry = &record.eids[0]; + assert_eq!(entry.uids.len(), 1, "should hold one UID: {source}"); + assert_eq!( + (&entry.inserter, &entry.matcher, entry.mm, &entry.ext), + (&None, &None, None, &None), + "should hold no EID-level field: {source}" + ); + assert_eq!( + (entry.uids[0].atype, &entry.uids[0].ext), + (None, &None), + "should hold no UID-level field: {source}" + ); + (source.clone(), entry.uids[0].id.clone()) + }) + .collect() + } + + #[test] + fn golden_browser_ingest_writes_the_first_uid() { + let cookie = format!("ts-eids={GOLDEN_INGEST_COOKIE}; sharedId=shared-1"); + + let returning = Scenario::new(); + returning.seed_live(); + let (_, outcome) = + returning.drive(navigation(Some(&format!("ts-ec={EC_ID}; {cookie}")))); + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_writes(&returning.store, (0, 1, 0), "one flush"); + assert_eq!( + assert_one_uid_records(&returning.row()), + golden_expected_ingest(), + "the flushed row should hold the UIDs the code before this spec stored" + ); + + let new_visitor = Scenario::new(); + let (response, _) = new_visitor.drive(navigation(Some(&cookie))); + let minted = set_cookie_value(&response, "ts-ec").expect("should mint an EC"); + assert_writes(&new_visitor.store, (1, 0, 0), "one create"); + assert_eq!( + assert_one_uid_records(&new_visitor.row_of(&minted)), + golden_expected_ingest(), + "the created row should hold the same UIDs" + ); + } + + #[test] + fn golden_one_uid_row_goes_out_unchanged() { + let scenario = auction_scenario(); + let mut entry = live_entry(); + for (source, uid) in [("id5-sync.com", "id5-uid"), ("sharedid.org", "shared-uid")] { + entry + .ids + .insert(source.to_owned(), KvPartnerId::browser(uid, Some(1_500))); + } + scenario.seed(&entry); + + let (response, _) = + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + let expected: serde_json::Value = serde_json::json!([ + { "source": "id5-sync.com", "uids": [{ "id": "id5-uid", "atype": 3 }] }, + { "source": "sharedid.org", "uids": [{ "id": "shared-uid", "atype": 3 }] } + ]); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + expected, + "user.ext.eids should equal what the code before this spec sent" + ); + assert_eq!( + response + .headers() + .get("x-ts-eids") + .and_then(|value| value.to_str().ok()), + Some(GOLDEN_ROW_HEADER), + "x-ts-eids should equal what the code before this spec built, byte for byte" + ); + } + + #[test] + fn golden_auction_body_forwarding_is_unchanged() { + let scenario = auction_scenario(); + scenario.seed_live(); + let eids: serde_json::Value = + serde_json::from_str(GOLDEN_BODY_EIDS).expect("should parse the body EIDs"); + let forwarded: serde_json::Value = serde_json::from_str(GOLDEN_BODY_FORWARDED) + .expect("should parse the forwarded EIDs"); + assert!( + forwarded.to_string().contains("rtiPartner") + && forwarded.to_string().contains("other.example.com"), + "control: the golden should hold UID-level and EID-level fields" + ); + + scenario.drive(auction_request( + Some(&ec_cookie()), + &auction_body(Some(&eids)), + )); + + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + forwarded, + "user.ext.eids should equal what the code before this spec forwarded" + ); + assert_no_writes(&scenario.store, "/auction"); + } + + // ---- A registered provider's complete set, end to end (Task 39b) ---- + + /// The provider `MultiEntryTestProvider` registers as. + const MULTI_PROVIDER: &str = "exampleid"; + + /// The host `MultiEntryTestProvider` calls. + const MULTI_HOST: &str = "api.exampleid.example.com"; + + /// The registration `exampleid` makes for `provider`. + fn multi_registration(provider: MultiEntryTestProvider) -> IntegrationRegistration { + IntegrationRegistration::builder("exampleid") + .try_with_identity_provider(Ok::<_, Report>(provider)) + .expect("should register the provider") + .build() + } + + /// A scenario with the auction, the `exampleid` source registered for + /// the bidstream, and `registrations` contributing providers. + /// `exampleid` owns [`MULTI_ENTRY_SOURCE`] and runs in `mode`. + fn registered_scenario( + registrations: Vec, + mode: ProviderMode, + ) -> Scenario { + let Scenario { + mut state, + store, + network, + } = Scenario::with_settings(&format!( + "{AUCTION_TOML}{}", + partner_toml(MULTI_ENTRY_SOURCE) + )); + let owned = Arc::get_mut(&mut state).expect("should own the fresh state"); + Arc::make_mut(&mut owned.settings) + .ec + .identity + .providers + .insert( + MULTI_PROVIDER.to_owned(), + ProviderSettings { + mode, + ..ProviderSettings::default() + }, + ); + let providers = registrations + .into_iter() + .flat_map(|registration| registration.identity_providers) + .collect(); + let runtime = IdentityRuntime::from_settings_with_providers( + &state.settings, + providers, + BTreeMap::from([( + MULTI_ENTRY_SOURCE.parse().expect("should parse the source"), + MULTI_PROVIDER + .parse() + .expect("should parse the provider ID"), + )]), + ) + .expect("should build the identity runtime"); + let scenario = Scenario { + state: crate::app::app_state_with_identity_for_test(state, runtime), + store, + network, + }; + scenario.network.respond( + BIDDER_HOST, + 200, + "application/json", + r#"{"id":"response","seatbid":[]}"#, + ); + answer_with(&scenario, "full"); + scenario + } + + /// A scenario whose one provider is `MultiEntryTestProvider`. + fn multi_scenario(provider: MultiEntryTestProvider, mode: ProviderMode) -> Scenario { + registered_scenario(vec![multi_registration(provider)], mode) + } + + /// Makes the provider's endpoint answer with the `entries` shape. + fn answer_with(scenario: &Scenario, entries: &str) { + scenario.network.respond( + MULTI_HOST, + 200, + "application/json", + &serde_json::json!({ "entries": entries }).to_string(), + ); + } + + fn kv_uid(id: &str, atype: Option, ext: Option) -> KvUid { + KvUid { + id: id.to_owned(), + atype, + ext, + } + } + + /// What the provider's `full` answer stores. + fn multi_full_set() -> Vec { + vec![ + KvEid { + uids: vec![ + kv_uid( + "uid-a", + Some(1), + Some(serde_json::json!({ "rtiPartner": "example" })), + ), + kv_uid("uid-b", None, None), + ], + inserter: Some("inserter-a.example.com".to_owned()), + matcher: Some("matcher-a.example.com".to_owned()), + mm: Some(2), + ext: None, + }, + KvEid { + uids: vec![kv_uid("uid-c", None, None)], + inserter: None, + matcher: None, + mm: None, + ext: Some(serde_json::json!({ "segments": ["example"] })), + }, + ] + } + + /// The first entry of the `full` answer, which the `first` answer + /// returns alone. + fn multi_first_set() -> Vec { + multi_full_set().into_iter().take(1).collect() + } + + /// The first entry as bidders see it. + fn multi_first_eid() -> serde_json::Value { + serde_json::json!({ + "source": MULTI_ENTRY_SOURCE, + "inserter": "inserter-a.example.com", + "matcher": "matcher-a.example.com", + "mm": 2, + "uids": [ + { "id": "uid-a", "atype": 1, "ext": { "rtiPartner": "example" } }, + { "id": "uid-b", "atype": EcPartner::default_openrtb_atype() } + ] + }) + } + + /// The second entry as bidders see it, with the provenance default. + fn multi_second_eid() -> serde_json::Value { + serde_json::json!({ + "source": MULTI_ENTRY_SOURCE, + "inserter": "default-inserter.example.com", + "matcher": "default-matcher.example.com", + "ext": { "segments": ["example"] }, + "uids": [{ "id": "uid-c", "atype": EcPartner::default_openrtb_atype() }] + }) + } + + /// A row holding `eids` for [`MULTI_ENTRY_SOURCE`], written by the + /// provider and due for a refresh. + fn multi_row(eids: Vec) -> KvEntry { + let mut entry = live_entry(); + entry.ids.insert( + MULTI_ENTRY_SOURCE.to_owned(), + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: Some("0123456789abcdef".to_owned()), + at: Some(1_000), + ra: Some(1_500), + eids, + ..KvPartnerId::default() + }, + ); + entry + } + + /// The record under [`MULTI_ENTRY_SOURCE`], with its fresh `rv` and `at` + /// checked and copied into the expectation of the caller. + fn fresh_record(row: &KvEntry) -> KvPartnerId { + let record = row + .ids + .get(MULTI_ENTRY_SOURCE) + .expect("should hold the provider's record") + .clone(); + let revision = record.rv.as_deref().expect("should carry a revision"); + assert!( + revision.len() == 16 + && revision + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)), + "should carry a 16 digit lowercase hex rv" + ); + assert_ne!( + revision, "0123456789abcdef", + "should carry a fresh rv, not the seeded one" + ); + assert!(record.at.is_some(), "should record when it was acquired"); + record + } + + #[test] + fn registered_provider_full_set_reaches_kv_mirror_and_bidstream_in_order() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + scenario.seed_live(); + + let (_, outcome) = scenario.drive(navigation(Some(&ec_cookie()))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "should call the registered provider once" + ); + assert_writes(&scenario.store, (0, 1, 0), "one flush holds the whole set"); + let record = fresh_record(&scenario.row()); + assert_eq!( + record, + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: record.rv.clone(), + at: record.at, + exp: None, + ra: None, + ru: None, + eids: multi_full_set(), + }, + "should store both entries as returned, written by the provider" + ); + + scenario.store.reset_counts(); + let body = auction_body(Some(&serde_json::json!([browser_entry()]))); + let (response, _) = scenario.drive(auction_request(Some(&ec_cookie()), &body)); + + let expected = + serde_json::json!([browser_entry(), multi_first_eid(), multi_second_eid()]); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + expected, + "user.ext.eids should carry the browser entry, then both entries in order" + ); + assert_eq!( + header_eids(&response), + expected, + "x-ts-eids should carry the same" + ); + assert_eq!( + mirror_of(&json_of(response)), + serde_json::json!({ "v": 1, "eids": [multi_first_eid(), multi_second_eid()] }), + "the mirror should decode to the same two entries, defaults included" + ); + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "control: the /auction should serve from KV without calling the provider" + ); + assert_no_writes(&scenario.store, "/auction"); + } + + #[test] + fn new_visitor_hold_answer_is_created_with_the_row_in_one_write() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + let mut request = navigation(None); + *request.uri_mut() = "https://test-publisher.com/" + .parse() + .expect("should parse the home page URI"); + + let (response, outcome) = scenario.drive(request); + + assert_eq!( + outcome, + Some(FlushOutcome::Skipped), + "should leave nothing for a flush" + ); + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "should call the hold-mode provider on the first navigation" + ); + assert_writes(&scenario.store, (1, 0, 0), "one create holds the answer"); + let ec_id = set_cookie_value(&response, "ts-ec").expect("should set the EC cookie"); + let (row, _) = scenario + .graph() + .get(&ec_id) + .expect("should read the row") + .expect("should create the row"); + assert_eq!( + row.ids + .values() + .find(|record| record.w.as_deref() == Some(MULTI_PROVIDER)) + .map(|record| record.eids.clone()), + Some(multi_full_set()), + "should create the row with the provider's answer" + ); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + serde_json::json!([multi_first_eid(), multi_second_eid()]), + "should hand this navigation's auction the answer" + ); + } + + #[test] + fn one_entry_one_uid_result_is_written_as_one_entry() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + scenario.seed_live(); + answer_with(&scenario, "single"); + + scenario.drive(navigation(Some(&ec_cookie()))); + + assert_writes(&scenario.store, (0, 1, 0), "one flush"); + let record = fresh_record(&scenario.row()); + assert_eq!( + record, + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: record.rv.clone(), + at: record.at, + exp: None, + ra: None, + ru: None, + eids: vec![KvEid { + uids: vec![kv_uid( + "uid-a", + Some(1), + Some(serde_json::json!({ "rtiPartner": "example" })) + )], + inserter: None, + matcher: None, + mm: None, + ext: None, + }], + }, + "should store one entry holding one UID" + ); + let stored = stored_body(&scenario); + assert_eq!( + stored["ids"][MULTI_ENTRY_SOURCE], + serde_json::json!({ + "w": MULTI_PROVIDER, + "rv": record.rv, + "at": record.at, + "eids": [{ + "uids": [{ + "id": "uid-a", + "atype": 1, + "ext": { "rtiPartner": "example" } + }] + }] + }), + "should write the one shape of spec 6.1" + ); + } + + #[test] + fn later_result_without_an_entry_removes_it() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + let mut row = multi_row(multi_full_set()); + row.ids.insert( + "id5-sync.com".to_owned(), + KvPartnerId::browser("id5-uid", Some(1_500)), + ); + scenario.seed(&row); + answer_with(&scenario, "first"); + + let (_, outcome) = scenario.drive(navigation(Some(&ec_cookie()))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "control: the due record should have been refreshed" + ); + let after = scenario.row(); + let record = fresh_record(&after); + assert_eq!( + record, + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: record.rv.clone(), + at: record.at, + exp: None, + ra: None, + ru: None, + eids: multi_first_set(), + }, + "should leave exactly the returned entry in KV" + ); + assert_eq!( + after.ids.get("id5-sync.com"), + row.ids.get("id5-sync.com"), + "should keep the record of a source the result does not name" + ); + + let (_, _) = scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + serde_json::json!([ + multi_first_eid(), + { "source": "id5-sync.com", "uids": [{ "id": "id5-uid", "atype": 3 }] } + ]), + "should serve exactly the remaining entry, and the other source's record" + ); + } + + #[test] + fn stateful_result_with_an_oversized_set_keeps_only_its_removals() { + let scenario = multi_scenario(MultiEntryTestProvider::stateful(), ProviderMode::Hold); + let mut row = multi_row(multi_first_set()); + row.ps.insert( + MULTI_PROVIDER.to_owned(), + KvProviderState { + st: Some(serde_json::json!({ "seen": false })), + ..KvProviderState::default() + }, + ); + scenario.seed(&row); + answer_with(&scenario, "oversized"); + + scenario.drive(navigation(Some(&ec_cookie()))); + + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "control: the provider should have been called" + ); + let after = scenario.row(); + assert!( + !after.ids.contains_key(MULTI_ENTRY_SOURCE), + "should apply the also_invalidate removal and store none of the oversized set" + ); + let state = after + .ps + .get(MULTI_PROVIDER) + .expect("should keep the provider's row state"); + assert!( + state.n.is_some_and(|wait_until| wait_until > 1_700_000_000), + "should back off the rejected answer" + ); + assert_eq!( + state, + &KvProviderState { + n: state.n, + f: 1, + o: None, + oc: None, + iv: state.iv.clone(), + ov: Vec::new(), + st: Some(serde_json::json!({ "seen": false })), + }, + "should keep the old state, not the rejected replacement, and count one failure" + ); + assert_eq!( + state.iv.len(), + 2, + "should record the two invalidated UIDs, so the removal is not undone" + ); + } + + #[test] + fn identify_returns_the_first_uid_of_a_full_record() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + let mut row = multi_row(multi_full_set()); + row.ids + .get_mut(MULTI_ENTRY_SOURCE) + .expect("should hold the record") + .ra = None; + row.ids.insert( + "id5-sync.com".to_owned(), + KvPartnerId::browser("id5-uid", Some(1_500)), + ); + scenario.seed(&row); + let atype = EcPartner::default_openrtb_atype(); + + let full = json_of( + scenario + .drive(identify_request(&ec_cookie(), MULTI_ENTRY_SOURCE)) + .0, + ); + let one_uid = json_of( + scenario + .drive(identify_request(&ec_cookie(), "id5-sync.com")) + .0, + ); + + assert_eq!(full["uid"], "uid-a", "should return the first UID"); + assert_eq!( + full["eid"], + serde_json::json!({ + "source": MULTI_ENTRY_SOURCE, + "uids": [{ "id": "uid-a", "atype": atype }] + }), + "should return the first UID with the partner's openrtb_atype" + ); + assert_eq!( + one_uid["eid"], + serde_json::json!({ + "source": "id5-sync.com", + "uids": [{ "id": "id5-uid", "atype": atype }] + }), + "control: a one-UID record should have the same shape" + ); + let keys = |body: &serde_json::Value| { + body.as_object() + .expect("should be an object") + .keys() + .cloned() + .collect::>() + }; + assert_eq!( + keys(&full), + keys(&one_uid), + "should answer in the same shape" + ); + } + + #[test] + fn rollback_to_part_1_config_keeps_the_owners_opt_out() { + let scenario = Scenario::with_settings(&format!( + "{AUCTION_TOML}{}", + partner_toml(MULTI_ENTRY_SOURCE) + )); + scenario.network.respond( + BIDDER_HOST, + 200, + "application/json", + r#"{"id":"response","seatbid":[]}"#, + ); + let mut row = multi_row(multi_full_set()); + row.ps.insert( + MULTI_PROVIDER.to_owned(), + KvProviderState { + n: Some(FAR_FUTURE), + f: 2, + o: Some(KvOptOut { + at: 1_200, + fp: "0".to_owned(), + }), + oc: Some(1_100), + iv: vec!["00112233445566ff".to_owned()], + ov: vec!["8899aabbccddeeff".to_owned()], + st: Some(serde_json::json!({ "seen": true })), + }, + ); + scenario.seed(&row); + + let (_, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + let after = scenario.row(); + assert_eq!( + after.ps.get(MULTI_PROVIDER), + Some(&KvProviderState { + n: None, + f: 0, + o: Some(KvOptOut { + at: 1_200, + fp: "0".to_owned(), + }), + oc: Some(1_100), + iv: Vec::new(), + ov: vec!["8899aabbccddeeff".to_owned()], + st: None, + }), + "should keep o, oc and ov and lose n, f, iv and st" + ); + assert_eq!( + after.ids.get(MULTI_ENTRY_SOURCE), + row.ids.get(MULTI_ENTRY_SOURCE), + "should keep the multi-entry record readable" + ); + assert_eq!( + stored_uid(&after, "sharedid.org").as_deref(), + Some(SHARED_ID), + "control: the flush should have written the ingest" + ); + + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + let sent = bidder_eids(&bidder_body(&scenario)); + assert!( + !sent.to_string().contains(MULTI_ENTRY_SOURCE), + "should not serve a record whose provider the catalog no longer has: {sent}" + ); + assert!( + sent.to_string().contains("sharedid.org"), + "control: should still serve the browser record: {sent}" + ); + } + + /// A scenario whose Didomi proxy answers `body`, with a capture-only + /// provider registered through an `IntegrationRegistration`, and a + /// row holding a browser-written `ssp.example.com` ID. + fn registered_capture_scenario(body: &str, global: bool) -> Scenario { + let Scenario { + mut state, + store, + network, + } = Scenario::with_settings(DIDOMI_TOML); + if global { + let owned = Arc::get_mut(&mut state).expect("should own the fresh state"); + Arc::make_mut(&mut owned.settings) + .ec + .identity + .providers + .insert( + "didomi".to_owned(), + ProviderSettings { + revoke_scope: Some(trusted_server_core::settings::RevokeScope::Global), + ..ProviderSettings::default() + }, + ); + } + let registration = IntegrationRegistration::builder("didomi") + .with_identity_provider( + trusted_server_core::ec::identity::test_support::answering_capture_provider( + "didomi", + &["ssp.example.com"], + None, + ), + ) + .build(); + let runtime = IdentityRuntime::from_settings_with_providers( + &state.settings, + registration.identity_providers, + BTreeMap::new(), + ) + .expect("should build the identity runtime"); + let scenario = Scenario { + state: crate::app::app_state_with_identity_for_test(state, runtime), + store, + network, + }; + scenario + .network + .respond("sdk.example.com", 200, "application/json", body); + let mut entry = live_entry(); + entry.ids.insert( + "ssp.example.com".to_owned(), + KvPartnerId { + rv: Some("0123456789abcdef".to_owned()), + at: Some(1_000), + ..KvPartnerId::new("probe-uid") + }, + ); + scenario.seed(&entry); + scenario + } + + #[test] + fn captured_revocation_writes_with_cas() { + let clean = registered_capture_scenario(CAPTURE_INVALIDATION, false); + clean.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &clean.store, + (0, 2, 0), + "one CAS write for the removal before send, then the writer's flush", + ); + assert_eq!( + stored_uid(&clean.row(), "ssp.example.com"), + None, + "should remove the captured ID" + ); + + let fifth = registered_capture_scenario(CAPTURE_INVALIDATION, false); + fifth.store.refuse_conditional_writes(EC_ID, 4); + fifth.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &fifth.store, + (0, 6, 0), + "four refused attempts and the fifth before send, no retry after send, the flush", + ); + assert_eq!( + stored_uid(&fifth.row(), "ssp.example.com"), + None, + "should remove the ID on the fifth attempt" + ); + + let retried = registered_capture_scenario(CAPTURE_INVALIDATION, false); + retried.store.refuse_conditional_writes(EC_ID, 5); + retried.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &retried.store, + (0, 7, 0), + "five refused attempts before send, one more after send, the flush", + ); + assert_eq!( + stored_uid(&retried.row(), "ssp.example.com"), + None, + "should remove the ID on the attempt after send" + ); + } + + #[test] + fn global_revoke_from_real_browser_writes_as_withdrawal() { + let withdrawn = Scenario::new(); + withdrawn.seed_live(); + drive_with_runtime( + &withdrawn.state, + navigation(Some(&ec_cookie())), + Some(&withdrawn.state.identity), + withdraw, + ); + let withdrawal_writes = withdrawn.store.writes(); + + let scenario = registered_capture_scenario(r#"{"opted_out":true}"#, true); + + let (response, _) = + scenario.drive(didomi_navigation(Some(&returning_with_shared_id()))); + + let row = scenario.row(); + assert!(!row.consent.ok, "should tombstone the row"); + assert!(row.ids.is_empty(), "should remove every ID"); + assert_eq!( + scenario.row().consent.ok, + withdrawn.row().consent.ok, + "should end as a consent withdrawal does" + ); + let writes = scenario.store.writes(); + assert_eq!( + (writes.add, writes.if_generation_match, writes.overwrite), + ( + withdrawal_writes.add, + withdrawal_writes.if_generation_match, + withdrawal_writes.overwrite + ), + "should write the tombstone and its marker as a withdrawal does" + ); + assert!( + set_cookies(&response).iter().any(|cookie| { + cookie_name(cookie) == "ts-ec" && cookie.contains("Max-Age=0") + }), + "should expire the EC cookie" + ); + } + + /// A proxied navigation from a client that fails the real-browser + /// check: no `Sec-Fetch-Dest`, a non-browser user agent. + fn non_browser_didomi_navigation() -> HttpRequest { + let mut request = didomi_navigation(Some(&returning_with_shared_id())); + request.extensions_mut().clear(); + request.headers_mut().remove("sec-fetch-dest"); + request + .headers_mut() + .insert("user-agent", HeaderValue::from_static("curl/8.0")); + request + } + + #[test] + fn non_browser_capture_writes_its_removal() { + let scenario = registered_capture_scenario(CAPTURE_INVALIDATION, false); + assert!( + scenario.row().ids.contains_key("ssp.example.com"), + "control: the seeded row holds the record" + ); + + let (response, _) = scenario.drive(non_browser_didomi_navigation()); + + assert_eq!(response.status(), 200, "should serve the proxied answer"); + assert!( + !scenario.row().ids.contains_key("ssp.example.com"), + "should apply the removal whatever the client" + ); + assert!( + !set_cookies(&response) + .iter() + .any(|cookie| cookie_name(cookie) == "ts-ec" && cookie.contains("Max-Age=0")), + "should not expire the EC cookie" + ); + } + + #[test] + fn non_browser_global_revoke_writes_as_withdrawal() { + let scenario = registered_capture_scenario(r#"{"opted_out":true}"#, true); + + let (response, _) = scenario.drive(non_browser_didomi_navigation()); + + assert!(!scenario.row().consent.ok, "should tombstone the row"); + assert!( + set_cookies(&response).iter().any(|cookie| { + cookie_name(cookie) == "ts-ec" && cookie.contains("Max-Age=0") + }), + "should expire the EC cookie" + ); + } + + #[test] + fn returning_navigation_answered_404_flushes_only_hold_removals_and_backoff() { + let removal_row = || multi_row(multi_first_set()); + let drive_home = |scenario: &Scenario, status: u16| { + scenario.network.respond( + ORIGIN_HOST, + status, + "text/html", + "page", + ); + let mut request = navigation(Some(&ec_cookie())); + *request.uri_mut() = "https://test-publisher.com/" + .parse() + .expect("should parse the home page URI"); + scenario.drive(request) + }; + + let ok = multi_scenario(MultiEntryTestProvider::stateful(), ProviderMode::Hold); + ok.seed(&removal_row()); + answer_with(&ok, "oversized"); + drive_home(&ok, 200); + assert!( + !ok.row().ids.contains_key(MULTI_ENTRY_SOURCE), + "control: a 200 should flush the removal" + ); + + let not_found = multi_scenario(MultiEntryTestProvider::stateful(), ProviderMode::Hold); + not_found.seed(&removal_row()); + answer_with(¬_found, "oversized"); + drive_home(¬_found, 404); + + assert_eq!( + not_found.network.requests_to(MULTI_HOST).len(), + 1, + "should still have run the hold call" + ); + let after = not_found.row(); + assert!( + !after.ids.contains_key(MULTI_ENTRY_SOURCE), + "should flush the hold answer's removal on a 404" + ); + assert_eq!( + after.ps.get(MULTI_PROVIDER).map(|state| state.iv.len()), + Some(2), + "should flush the removal's record of the two invalidated UIDs" + ); + + let backoff = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + backoff.seed_live(); + answer_with(&backoff, "none"); + drive_home(&backoff, 404); + assert!( + backoff + .row() + .ps + .get(MULTI_PROVIDER) + .and_then(|state| state.n) + .is_some_and(|wait_until| wait_until > 1_700_000_000), + "should flush the no_match backoff on a 404" + ); + + let positive = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + positive.seed_live(); + answer_with(&positive, "full"); + drive_home(&positive, 404); + assert!( + !positive.row().ids.contains_key(MULTI_ENTRY_SOURCE), + "should store no positive hold answer on a 404" + ); + } + + /// A scenario like [`Scenario::new`] with `MultiEntryTestProvider` + /// registered on its own source, and its endpoint left unanswered so + /// the provider writes nothing the golden tests would see. + fn golden_registered_scenario() -> Scenario { + let scenario = registered_scenario( + vec![multi_registration(MultiEntryTestProvider::new())], + ProviderMode::AfterSend, + ); + scenario.network.clear_reply(MULTI_HOST); + scenario + } + + #[test] + fn golden_browser_ingest_with_a_registered_provider_writes_the_first_uid() { + let cookie = format!("ts-eids={GOLDEN_INGEST_COOKIE}; sharedId=shared-1"); + + let returning = golden_registered_scenario(); + returning.seed_live(); + let (_, outcome) = + returning.drive(navigation(Some(&format!("ts-ec={EC_ID}; {cookie}")))); + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_writes(&returning.store, (0, 1, 0), "one flush"); + assert_eq!( + assert_one_uid_records(&returning.row()), + golden_expected_ingest(), + "the flushed row should hold the UIDs the code before this spec stored" + ); + + let new_visitor = golden_registered_scenario(); + let (response, _) = new_visitor.drive(navigation(Some(&cookie))); + let minted = set_cookie_value(&response, "ts-ec").expect("should mint an EC"); + assert_writes(&new_visitor.store, (1, 0, 0), "one create"); + assert_eq!( + assert_one_uid_records(&new_visitor.row_of(&minted)), + golden_expected_ingest(), + "the created row should hold the same UIDs" + ); + } + + #[test] + fn golden_one_uid_row_with_a_registered_provider_goes_out_unchanged() { + let scenario = golden_registered_scenario(); + let mut entry = live_entry(); + for (source, uid) in [("id5-sync.com", "id5-uid"), ("sharedid.org", "shared-uid")] { + entry + .ids + .insert(source.to_owned(), KvPartnerId::browser(uid, Some(1_500))); + } + scenario.seed(&entry); + + let (response, _) = + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + serde_json::json!([ + { "source": "id5-sync.com", "uids": [{ "id": "id5-uid", "atype": 3 }] }, + { "source": "sharedid.org", "uids": [{ "id": "shared-uid", "atype": 3 }] } + ]), + "user.ext.eids should equal what the code before this spec sent" + ); + assert_eq!( + response + .headers() + .get("x-ts-eids") + .and_then(|value| value.to_str().ok()), + Some(GOLDEN_ROW_HEADER), + "x-ts-eids should equal what the code before this spec built, byte for byte" + ); + } + + #[test] + fn golden_auction_body_forwarding_with_a_registered_provider_is_unchanged() { + let scenario = golden_registered_scenario(); + scenario.seed_live(); + let eids: serde_json::Value = + serde_json::from_str(GOLDEN_BODY_EIDS).expect("should parse the body EIDs"); + let forwarded: serde_json::Value = serde_json::from_str(GOLDEN_BODY_FORWARDED) + .expect("should parse the forwarded EIDs"); + + scenario.drive(auction_request( + Some(&ec_cookie()), + &auction_body(Some(&eids)), + )); + + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + forwarded, + "user.ext.eids should equal what the code before this spec forwarded" + ); + assert_no_writes(&scenario.store, "/auction"); + } + } } diff --git a/crates/trusted-server-adapter-fastly/src/platform.rs b/crates/trusted-server-adapter-fastly/src/platform.rs index d0f4ee9d0e..af395eda2b 100644 --- a/crates/trusted-server-adapter-fastly/src/platform.rs +++ b/crates/trusted-server-adapter-fastly/src/platform.rs @@ -7,6 +7,7 @@ use fastly::geo::{Geo, geo_lookup}; use fastly::{ConfigStore, Request, SecretStore}; use std::io::Read as _; use std::net::IpAddr; +use std::sync::atomic::{AtomicUsize, Ordering}; use crate::backend::BackendConfig; pub(crate) use trusted_server_core::platform::UnavailableKvStore; @@ -159,6 +160,7 @@ fn backend_config_from_spec(spec: &PlatformBackendSpec) -> BackendConfig<'_> { .certificate_check(spec.certificate_check) .first_byte_timeout(spec.first_byte_timeout) .between_bytes_timeout(spec.between_bytes_timeout) + .connect_timeout(spec.connect_timeout) .discriminator(spec.discriminator.as_deref()) } @@ -492,7 +494,13 @@ fn apply_fastly_cache_intent(request: &mut fastly::Request, intent: &PlatformCac /// environment cannot register a real Fastly backend, so the common /// `StubHttpClient` driver test records the one-send 3xx behavior while adapter /// tests cover request conversion and the single-send boundary. -pub struct FastlyPlatformHttpClient; +/// +/// One client is built per request, so it counts the backend requests the +/// execution has started, for Fastly's per-execution backend request limit. +#[derive(Default)] +pub struct FastlyPlatformHttpClient { + started: AtomicUsize, +} #[async_trait::async_trait(?Send)] impl PlatformHttpClient for FastlyPlatformHttpClient { @@ -504,10 +512,15 @@ impl PlatformHttpClient for FastlyPlatformHttpClient { true } + fn requests_started(&self) -> Option { + Some(self.started.load(Ordering::Relaxed)) + } + async fn send( &self, request: PlatformHttpRequest, ) -> Result> { + self.started.fetch_add(1, Ordering::Relaxed); let backend_name = request.backend_name.clone(); let image_optimizer = request.image_optimizer; let stream_response = request.stream_response; @@ -528,6 +541,7 @@ impl PlatformHttpClient for FastlyPlatformHttpClient { &self, request: PlatformHttpRequest, ) -> Result> { + self.started.fetch_add(1, Ordering::Relaxed); let backend_name = request.backend_name.clone(); if request.image_optimizer.is_some() { return Err(Report::new(PlatformError::HttpClient) @@ -1013,6 +1027,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_secs(15), between_bytes_timeout: Duration::from_secs(15), + connect_timeout: None, discriminator: None, }; @@ -1037,6 +1052,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_secs(15), between_bytes_timeout: Duration::from_secs(15), + connect_timeout: None, discriminator: None, }; @@ -1063,6 +1079,7 @@ mod tests { certificate_check: false, first_byte_timeout: Duration::from_secs(15), between_bytes_timeout: Duration::from_secs(15), + connect_timeout: None, discriminator: None, }; @@ -1087,6 +1104,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_secs(15), between_bytes_timeout: Duration::from_secs(15), + connect_timeout: None, discriminator: None, }; @@ -1106,6 +1124,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_millis(2000), between_bytes_timeout: Duration::from_millis(2000), + connect_timeout: None, discriminator: None, }; @@ -1135,6 +1154,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_millis(750), between_bytes_timeout: Duration::from_millis(750), + connect_timeout: None, discriminator: None, }; @@ -1162,6 +1182,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_millis(750), between_bytes_timeout: Duration::from_millis(750), + connect_timeout: None, discriminator: Some("ipv6-provider".to_string()), }; let mut bare = bracketed.clone(); @@ -1185,7 +1206,7 @@ mod tests { #[test] fn auction_http_capabilities_are_explicit() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let capabilities = trusted_server_core::platform::AuctionTargetId::Fastly .descriptor() .capabilities(); @@ -1355,7 +1376,7 @@ mod tests { #[test] fn fastly_platform_http_client_send_returns_error_for_unregistered_backend() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let request = request_builder() .method("GET") .uri("https://example.com/") @@ -1375,7 +1396,7 @@ mod tests { #[test] fn fastly_platform_http_client_send_async_returns_error_for_unregistered_backend() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let request = request_builder() .method("GET") .uri("https://example.com/") @@ -1395,7 +1416,7 @@ mod tests { #[test] fn fastly_platform_http_client_select_returns_error_for_empty_list() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let err = futures::executor::block_on(client.select(vec![])) .expect_err("should return error for empty pending list"); @@ -1408,7 +1429,7 @@ mod tests { #[test] fn fastly_platform_http_client_select_returns_error_for_wrong_inner_type() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); // Wrap a non-PendingRequest type to trigger the downcast failure. let wrong = PlatformPendingRequest::new(42u32).with_backend_name("origin-a"); let err = futures::executor::block_on(client.select(vec![wrong])) @@ -1427,7 +1448,7 @@ mod tests { #[test] fn fastly_platform_http_client_rejects_stream_marked_pending_from_select() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let pending = PlatformPendingRequest::new(42_u32) .with_backend_name("origin-a") .with_response_handling(true, edgezero_core::http::Method::GET); @@ -1442,7 +1463,7 @@ mod tests { #[test] fn fastly_platform_http_client_send_returns_error_for_streaming_body() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let request = request_builder() .method("POST") .uri("https://example.com/") @@ -1469,7 +1490,7 @@ mod tests { #[test] fn fastly_platform_http_client_send_async_rejects_image_optimizer_metadata() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let request = request_builder() .method("GET") .uri("https://example.com/image.jpg") @@ -1492,7 +1513,7 @@ mod tests { #[test] fn fastly_platform_http_client_supports_pending_streaming_responses() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); assert!( client.supports_pending_streaming_responses(), "should advertise direct pending-response streaming" @@ -1517,7 +1538,7 @@ mod tests { #[test] fn fastly_platform_http_client_send_async_returns_error_for_streaming_body() { - let client = FastlyPlatformHttpClient; + let client = FastlyPlatformHttpClient::default(); let request = request_builder() .method("POST") .uri("https://example.com/") @@ -1703,6 +1724,7 @@ mod tests { certificate_check: true, first_byte_timeout: Duration::from_millis(750), between_bytes_timeout: Duration::from_millis(750), + connect_timeout: None, discriminator: Some("prebid".to_string()), }; let prebid_name = backend diff --git a/crates/trusted-server-adapter-fastly/src/rate_limiter.rs b/crates/trusted-server-adapter-fastly/src/rate_limiter.rs index 67520925ed..44431e907c 100644 --- a/crates/trusted-server-adapter-fastly/src/rate_limiter.rs +++ b/crates/trusted-server-adapter-fastly/src/rate_limiter.rs @@ -22,6 +22,31 @@ fn effective_hourly_limit(hourly_limit: u32) -> u32 { hourly_limit_to_per_minute_limit(hourly_limit).saturating_mul(60) } +/// The two counter calls the breaker uses, so tests can fake them. +trait CounterOps { + fn increment(&self, key: &str) -> Result<(), String>; + fn lookup_count(&self, key: &str) -> Result; +} + +fn counter_error(action: &str, message: &str) -> Report { + Report::new(TrustedServerError::KvStore { + store_name: RATE_COUNTER_NAME.to_owned(), + message: format!("Failed to {action} sync rate counter: {message}"), + }) +} + +fn record_with(counter: &dyn CounterOps, key: &str) -> Result<(), Report> { + counter + .increment(key) + .map_err(|message| counter_error("increment", &message)) +} + +fn count_with(counter: &dyn CounterOps, key: &str) -> Result> { + counter + .lookup_count(key) + .map_err(|message| counter_error("read", &message)) +} + /// Fastly Edge Rate Limiting implementation of [`RateLimiter`]. pub struct FastlyRateLimiter { counter: RateCounter, @@ -72,12 +97,126 @@ impl RateLimiter for FastlyRateLimiter { Ok(false) } + + fn record(&self, key: &str) -> Result<(), Report> { + record_with(&self.counter, key) + } + + fn count(&self, key: &str) -> Result> { + count_with(&self.counter, key) + } +} + +impl CounterOps for RateCounter { + fn increment(&self, key: &str) -> Result<(), String> { + RateCounter::increment(self, key, 1).map_err(|error| error.to_string()) + } + + fn lookup_count(&self, key: &str) -> Result { + RateCounter::lookup_count(self, key, CounterDuration::SixtySecs) + .map_err(|error| error.to_string()) + } } #[cfg(test)] mod tests { + use std::cell::RefCell; + use super::*; + struct FakeCounter { + increments: RefCell>, + lookups: RefCell>, + count: u32, + fail: bool, + } + + impl FakeCounter { + fn new(count: u32, fail: bool) -> Self { + Self { + increments: RefCell::new(Vec::new()), + lookups: RefCell::new(Vec::new()), + count, + fail, + } + } + } + + impl CounterOps for FakeCounter { + fn increment(&self, key: &str) -> Result<(), String> { + self.increments.borrow_mut().push(key.to_owned()); + if self.fail { + return Err("counter unavailable".to_owned()); + } + Ok(()) + } + + fn lookup_count(&self, key: &str) -> Result { + self.lookups.borrow_mut().push(key.to_owned()); + if self.fail { + return Err("counter unavailable".to_owned()); + } + Ok(self.count) + } + } + + #[test] + fn record_increments_once() { + let counter = FakeCounter::new(0, false); + + record_with(&counter, "identity-fail:example") + .expect("should record against a working counter"); + + assert_eq!( + *counter.increments.borrow(), + vec!["identity-fail:example".to_owned()], + "should increment the key exactly once" + ); + assert!( + counter.lookups.borrow().is_empty(), + "should not read the count while recording" + ); + } + + #[test] + fn count_reads_the_sixty_second_window() { + let counter = FakeCounter::new(7, false); + + let count = count_with(&counter, "identity-fail:example") + .expect("should count against a working counter"); + + assert_eq!(count, 7, "should return the counter's value"); + assert_eq!( + *counter.lookups.borrow(), + vec!["identity-fail:example".to_owned()], + "should look the key up once" + ); + assert!( + counter.increments.borrow().is_empty(), + "should not change the count while reading it" + ); + } + + #[test] + fn counter_errors_map_to_kv_store_errors() { + let counter = FakeCounter::new(0, true); + + let record_error = record_with(&counter, "identity-fail:example") + .expect_err("should fail when the increment fails"); + let count_error = count_with(&counter, "identity-fail:example") + .expect_err("should fail when the lookup fails"); + + for error in [record_error, count_error] { + assert!( + matches!( + error.current_context(), + TrustedServerError::KvStore { store_name, .. } if store_name == RATE_COUNTER_NAME + ), + "should map the failure to a KV store error naming the counter" + ); + } + } + #[test] fn zero_hourly_limit_denies_all() { assert_eq!( diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 504dbd9fba..4ab358a2fa 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -191,6 +191,7 @@ fn tinybird_backend_spec(api_host: &str) -> PlatformBackendSpec { certificate_check: true, first_byte_timeout: TINYBIRD_FIRST_BYTE_TIMEOUT, between_bytes_timeout: TINYBIRD_BETWEEN_BYTES_TIMEOUT, + connect_timeout: None, discriminator: None, } } diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 804a23b620..03bcbfb323 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -24,6 +24,7 @@ use trusted_server_core::ec::admin::{ admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, deny_admin_diagnostic_fallback, handle_admin_eids_lookup, }; +use trusted_server_core::ec::finalize::BrowserSeed; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::http_util::sanitize_forwarded_headers; @@ -221,7 +222,7 @@ const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ Method::DELETE, ]; -fn named_fallback_paths() -> [(&'static str, &'static [Method]); 17] { +fn named_fallback_paths() -> [(&'static str, &'static [Method]); 18] { [ ("/.well-known/trusted-server.json", &[Method::GET]), ("/verify-signature", &[Method::POST]), @@ -229,6 +230,10 @@ fn named_fallback_paths() -> [(&'static str, &'static [Method]); 17] { ("/_ts/admin/keys/deactivate", &[Method::POST]), ("/_ts/admin/ec", &[Method::GET]), ("/_ts/admin/ec/{id}", &[Method::GET]), + ( + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + &[Method::DELETE], + ), ("/_ts/admin/eids", &[Method::GET]), ("/_ts/admin/cache/purge", LEGACY_ADMIN_DENY_METHODS), ("/admin/keys/rotate", LEGACY_ADMIN_DENY_METHODS), @@ -825,6 +830,8 @@ fn build_router(state: &Arc) -> RouterService { kv: None, ec_context: &mut ec_context, services: &services, + is_real_browser: false, + seed: &BrowserSeed::empty(), req, }) .await @@ -919,6 +926,12 @@ fn build_router(state: &Arc) -> RouterService { // adapter has no store to read. .get("/_ts/admin/ec", admin_ec_not_supported_handler) .get("/_ts/admin/ec/{id}", admin_ec_not_supported_handler) + // The provider opt-out clear writes the EC identity graph, so it + // answers like the lookup. + .delete( + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + admin_ec_not_supported_handler, + ) .get("/_ts/admin/eids", admin_eids_handler) .post("/auction", auction_handler) .get(PAGE_BIDS_PATH, page_bids_handler.clone()) diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index 613832b059..cb62546dec 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -151,6 +151,49 @@ async fn authenticated_admin_ec_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_opt_out_clear_route_returns_501() { + // The clear writes the EC identity graph, which is Fastly KV backed, so + // Spin answers the DELETE locally with 501 like the EC lookup. + let path = format!( + "/_ts/admin/ec/{}.abc123/providers/pull-ssp-example-com/opt-out", + "a".repeat(64) + ); + let delete = request_builder() + .method("DELETE") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), delete).await; + + assert_eq!( + resp.status().as_u16(), + 501, + "DELETE {path} should report that the opt-out clear is unsupported" + ); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json"), + "should answer with the same JSON as the EC lookup" + ); + + let get = request_builder() + .method("GET") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), get).await; + assert_eq!( + resp.status().as_u16(), + 404, + "control: GET {path} should be denied locally as an unknown admin route" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn admin_ec_route_without_credentials_returns_401() { let req = request_builder() diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index c21caa1b0a..28a6de49f4 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -9,13 +9,26 @@ use serde_json::value::RawValue; use crate::auction::formats::AdRequest; use crate::auction::orchestrator::OrchestrationResult; -use crate::consent::{consent_allows_server_side_auction, gate_eids_by_consent}; +use crate::consent::{ + allows_eid_persistence, consent_allows_server_side_auction, gate_eids_by_consent, +}; use crate::constants::COOKIE_TS_EIDS; use crate::cookies::extract_cookie_value; use crate::ec::EcContext; use crate::ec::EcKvSnapshot; +use crate::ec::checked_current_timestamp; use crate::ec::client_eids::parse_client_eids; -use crate::ec::eids::{resolve_partner_ids, to_eids}; +use crate::ec::current_timestamp; +use crate::ec::eids::{ServerSet, server_set}; +use crate::ec::identity::IdentityRuntime; +use crate::ec::identity::UnixTime; +use crate::ec::identity::catalog::ProviderCatalog; +use crate::ec::identity::coordinator::HoldResults; +use crate::ec::identity::denials::{Denials, denials, filter_browser_candidates}; +use crate::ec::identity::mirror::{MirrorPayload, MirrorRead, MirrorSource, mirror_payload}; +use crate::ec::identity::policy::PolicySignals; +use crate::ec::identity::request::opted_out; +use crate::ec::identity::servable::ServeContext; use crate::ec::kv::KvIdentityGraph; use crate::ec::partner::normalize_partner_source_domain; use crate::ec::prebid_eids::parse_prebid_eids_cookie; @@ -27,7 +40,7 @@ use crate::settings::Settings; use super::AuctionOrchestrator; use super::formats::{ - convert_to_openrtb_response, convert_to_openrtb_response_with_report, + convert_to_openrtb_response_with_mirror, convert_to_openrtb_response_with_report, convert_tsjs_to_auction_request, }; use super::telemetry::{ @@ -213,11 +226,17 @@ pub async fn handle_auction( total_time_ms: 0, metadata: HashMap::new(), }; - return convert_to_openrtb_response( + return convert_to_openrtb_response_with_mirror( &empty_result, settings, &auction_request, ec_context.ec_allowed(), + mirror_member( + &EcKvSnapshot::NotRead, + None, + ec_context, + PolicySignals::from_headers(http_req.headers()), + ), ); } @@ -266,11 +285,14 @@ pub async fn handle_auction( total_time_ms: 0, metadata: HashMap::new(), }; - return convert_to_openrtb_response( + // A denial clears the mirror so a withdrawal reaches the cookie + // (spec 5.6). + return convert_to_openrtb_response_with_mirror( &empty_result, settings, &auction_request, ec_context.ec_allowed(), + Some(MirrorPayload::Clear.to_json()), ); } @@ -295,12 +317,8 @@ pub async fn handle_auction( } else { None }; - // Carry the full request-local EID set to response finalization so KV - // ingestion uses what this request actually sent, not just whatever fits - // in the size-capped `ts-eids` cookie (see `ec::finalize::ec_finalize_response`). - if let Some(eids) = &client_eids { - ec_context.set_client_eids(eids.clone()); - } + // The browser's EIDs feed the merge below and nothing else: `/auction` is + // a stager, so they never reach KV (spec 5.3, G10). // Resolve partner EIDs from the KV identity graph when the user has a valid // EC and both KV and partner stores are available. Gate the read on a @@ -310,14 +328,15 @@ pub async fn handle_auction( (Some(graph), Some(ec_id), Some(_)) => graph.load_snapshot(ec_id), _ => EcKvSnapshot::NotRead, }; - // Hand the loaded row to the request context so response finalization — - // which runs on an EC context the adapter owns, after this handler returns - // — ingests `ts-eids`/`sharedId` updates from this read instead of paying - // for a second lookup. + // Hand the loaded row to the request context so response finalization, + // which runs on an EC context the adapter owns after this handler + // returns, reuses this read instead of paying for a second lookup. if !matches!(auction_kv_snapshot, EcKvSnapshot::NotRead) { ec_context.set_kv_snapshot(auction_kv_snapshot.clone()); } - let eids = resolve_auction_eids(&auction_kv_snapshot, registry, ec_context); + let signals = PolicySignals::from_headers(http_req.headers()); + let identity = auction_identity(&auction_kv_snapshot, registry, ec_context, signals); + let mirror = identity.mirror_member(ec_context); // Look up geo for device info. let geo = services @@ -339,9 +358,10 @@ pub async fn handle_auction( geo, )?; - // Forward the browser entries as sent, append KV entries for registered - // sources the browser did not send, then apply consent gating. - let merged_eids = merge_auction_eids(client_eids, eids); + // Forward the browser entries that pass the provider denials, append the + // server set for sources the browser did not send, then apply consent + // gating. + let merged_eids = identity.merge(client_eids.as_deref()); let had_eids = merged_eids.as_ref().is_some_and(|v| !v.is_empty()); auction_request.user.eids = gate_eids_by_consent(merged_eids, auction_request.user.consent.as_ref()); @@ -393,6 +413,7 @@ pub async fn handle_auction( settings, &auction_request, ec_context.ec_allowed(), + mirror, ) { Ok(conversion) => conversion, Err(error) => { @@ -436,34 +457,249 @@ pub async fn handle_auction( Ok(conversion.response) } -/// Resolves partner EIDs from the KV identity graph for bidstream decoration. +/// What the row a request read means for the `ts-eids-s` mirror (spec 6.2). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MirrorState { + /// No KV read for this request's EC: no mirror member. + NotRead, + /// EC use is denied, or the row is missing, a tombstone or withdrawn. + Cleared, + /// A live row: the mirror carries the servable server set. + Live, +} + +/// What one request's KV read gives the auction, resolved once (spec 7.1). /// -/// Returns `None` when any prerequisite is missing (no KV store, no partner -/// store, no EC, consent denied). On KV or partner-resolution errors, logs a -/// warning and returns empty EIDs so the auction can proceed in degraded mode. -pub(crate) fn resolve_auction_eids( +/// Servability is decided once, at one clock read, and both the bidstream +/// merge and the `ts-eids-s` mirror read the same [`ServerSet`]. +pub(crate) struct AuctionIdentity { + /// `None` when the request may not use KV IDs at all (no registry, no + /// EC, or EC use denied). + server: Option, + denials: Denials, + mirror: MirrorState, + /// The clock read servability used, when there was one. + now: Option, +} + +impl AuctionIdentity { + /// The server set's EIDs, before browser precedence. + #[cfg(test)] + #[must_use] + pub(crate) fn server_eids(&self) -> Option> { + self.server.as_ref().map(ServerSet::eids) + } + + /// Overlays the answers of the hold dispatch before this navigation's + /// auction onto the server set. A request that may not use KV IDs stays + /// without a set. + pub(crate) fn add_hold_results( + &mut self, + results: HoldResults, + registry: &PartnerRegistry, + catalog: &ProviderCatalog, + ) { + if let Some(server) = self.server.as_mut() { + server.with_hold_results(results, registry, catalog); + } + } + + /// The request's bidstream EIDs before consent gating: the browser's + /// candidates after the denials, then the server set (spec 7.1). + #[must_use] + pub(crate) fn merge(&self, client_eids: Option<&[Eid]>) -> Option> { + let empty = ServerSet::default(); + let merged = merge_auction_eids( + client_eids.unwrap_or_default(), + self.server.as_ref().unwrap_or(&empty), + &self.denials, + ); + if merged.is_empty() { + None + } else { + Some(merged) + } + } + + /// The mirror member of an `/auction` or `/_ts/page-bids` response + /// (spec 6.2). + /// + /// The mirror carries the same server set as the bidstream, provenance + /// defaults included, and never an ID consent would strip. + /// + /// `None` while `[ec.identity] mirror_enabled` is off or the request has + /// no identity runtime: the response then carries no mirror member. + #[must_use] + pub(crate) fn mirror_member(&self, ec_context: &EcContext) -> Option { + if !mirror_enabled(ec_context) { + return None; + } + let sources = match self.mirror { + MirrorState::NotRead | MirrorState::Cleared => Vec::new(), + MirrorState::Live if allows_eid_persistence(ec_context.consent()) => self + .server + .as_ref() + .map(|server| mirror_sources(server, ec_context)) + .unwrap_or_default(), + MirrorState::Live => Vec::new(), + }; + let read = if self.mirror == MirrorState::NotRead { + MirrorRead::NotRead + } else if sources.is_empty() { + MirrorRead::Cleared + } else { + MirrorRead::Servable(&sources) + }; + let now = self.now.unwrap_or_else(current_timestamp); + mirror_payload(read, now).map(|payload| payload.to_json()) + } +} + +/// Whether the operator turned the `ts-eids-s` mirror on (spec 6.2, 15). +pub(crate) fn mirror_enabled(ec_context: &EcContext) -> bool { + ec_context + .identity_runtime() + .is_some_and(|runtime| runtime.identity().mirror_enabled) +} + +/// One [`MirrorSource`] per source of `server`, ranked by its writer's +/// `cookie_priority`. +fn mirror_sources(server: &ServerSet, ec_context: &EcContext) -> Vec { + let identity = ec_context.identity_runtime().map(IdentityRuntime::identity); + server + .sources() + .iter() + .map(|source| { + let priority = match (&source.writer, identity) { + (Some(writer), Some(identity)) => identity + .providers + .get(writer.as_str()) + .map_or(0, |settings| settings.cookie_priority), + _ => 0, + }; + MirrorSource { + source: source.source_domain.clone(), + provider: source.writer.clone(), + priority, + expires_at: source.expires_at, + eids: source.eids.clone(), + } + }) + .collect() +} + +fn mirror_state( snapshot: &EcKvSnapshot, registry: Option<&PartnerRegistry>, ec_context: &EcContext, -) -> Option> { - let registry = registry?; - +) -> MirrorState { if !ec_context.ec_allowed() { - return None; + return MirrorState::Cleared; } - - let ec_id = ec_context.ec_value()?; - - let Some(entry) = snapshot.entry_for(ec_id) else { - return Some(Vec::new()); + let (Some(ec_id), Some(_)) = (ec_context.ec_value(), registry) else { + return MirrorState::NotRead; }; + match snapshot { + EcKvSnapshot::Missing { ec_id: read } if read == ec_id => MirrorState::Cleared, + EcKvSnapshot::Present { + ec_id: read, entry, .. + } if read == ec_id => { + if entry.consent.ok { + MirrorState::Live + } else { + MirrorState::Cleared + } + } + _ => MirrorState::NotRead, + } +} - if !entry.consent.ok { - return Some(Vec::new()); +/// Resolves what the request's KV read gives the auction, once. +/// +/// The server set is `None` when any prerequisite is missing (no partner +/// store, no EC, consent denied), and empty on a KV miss, a failed read, a +/// withdrawn row or a failed clock, so the auction proceeds in degraded +/// mode. A stored record is in it only when the shared servability check +/// passes under this request's consent and `signals`. The denials come from +/// the row read, whatever its consent state. +pub(crate) fn auction_identity( + snapshot: &EcKvSnapshot, + registry: Option<&PartnerRegistry>, + ec_context: &EcContext, + signals: PolicySignals, +) -> AuctionIdentity { + let mut identity = AuctionIdentity { + server: None, + denials: Denials::default(), + mirror: mirror_state(snapshot, registry, ec_context), + now: None, + }; + let Some(registry) = registry else { + return identity; + }; + if !ec_context.ec_allowed() { + return identity; } + let Some(ec_id) = ec_context.ec_value() else { + return identity; + }; + let entry = snapshot.entry_for(ec_id); + // Without a runtime no provider is configured, so only unowned records + // without a writer are servable, and no provider denies anything. + let empty_catalog; + let catalog = match ec_context.identity_runtime() { + Some(runtime) => runtime.catalog(), + None => { + empty_catalog = ProviderCatalog::empty(); + &empty_catalog + } + }; + if let Some(entry) = entry { + identity.denials = denials(entry, catalog); + } + // A zero time would make every expiry check pass (spec 4.4), so a failed + // clock resolves no KV IDs. + let Some(now) = checked_current_timestamp() else { + identity.server = Some(ServerSet::default()); + return identity; + }; + identity.now = Some(now); + let opted_out = entry.map(opted_out).unwrap_or_default(); + let serve = ServeContext { + now: UnixTime::from_secs(now), + consent: ec_context.consent(), + signals, + catalog, + opted_out: &opted_out, + }; + identity.server = Some(server_set(entry, registry, &serve)); + identity +} + +/// Resolves partner EIDs from the KV identity graph for bidstream decoration. +/// +/// Returns `None` when any prerequisite is missing (no KV store, no partner +/// store, no EC, consent denied), and empty EIDs on a KV miss or error so +/// the auction can proceed in degraded mode. See [`auction_identity`]. +#[cfg(test)] +pub(crate) fn resolve_auction_eids( + snapshot: &EcKvSnapshot, + registry: Option<&PartnerRegistry>, + ec_context: &EcContext, + signals: PolicySignals, +) -> Option> { + auction_identity(snapshot, registry, ec_context, signals).server_eids() +} - let resolved = resolve_partner_ids(registry, entry); - Some(to_eids(&resolved)) +/// The mirror member of an `/auction` or `/_ts/page-bids` response, from the +/// row this request read (spec 6.2). See [`AuctionIdentity::mirror_member`]. +pub(crate) fn mirror_member( + snapshot: &EcKvSnapshot, + registry: Option<&PartnerRegistry>, + ec_context: &EcContext, + signals: PolicySignals, +) -> Option { + auction_identity(snapshot, registry, ec_context, signals).mirror_member(ec_context) } /// Resolves the request's browser EIDs: the `/auction` body when it has a @@ -492,46 +728,60 @@ fn parse_cookie_auction_eids(cookie_value: Option<&str>) -> Option> { } } -/// Merges the request's browser EIDs with KV-resolved partner EIDs. +/// Merges the request's browser EIDs with the server set (spec 7.1). /// -/// Browser entries are forwarded as parsed, in wire order, never altered or -/// combined: they are what Prebid sent the `trustedServer` adapter, and a -/// source may repeat with different EID-level fields. KV entries are appended -/// after them, only for registered sources no browser entry carries. Sources -/// are compared as the partner registry normalizes them (lowercase, trailing -/// dot removed). For a source the browser sends, KV's UIDs, `atype` and `ext` -/// are not used: KV keeps the first value it stored, so it is the staler copy. -/// Entries without UIDs are dropped. +/// The browser's candidates first pass the provider `denials`. Those left +/// are forwarded as parsed, in wire order, never altered beyond a denied +/// UID: they are what Prebid sent the `trustedServer` adapter, and a source +/// may repeat with different EID-level fields. A server source's whole entry +/// set is appended after them, and the two are never mixed: +/// +/// - For a source whose owner wrote a valid value, the owner's set replaces +/// every browser entry of the source, however often the browser repeated +/// it. +/// - For any other source, the server's set goes out only when no browser +/// entry carries the source, so the browser's current value wins over a +/// browser-seeded record and over any unowned record. +/// +/// Sources are compared as the partner registry normalizes them (lowercase, +/// trailing dot removed). Entries without UIDs are dropped. pub(crate) fn merge_auction_eids( - client_eids: Option>, - resolved_eids: Option>, -) -> Option> { - let mut merged: Vec = client_eids + client_eids: &[Eid], + server: &ServerSet, + denials: &Denials, +) -> Vec { + let owned_sources: HashSet = server + .sources() + .iter() + .filter(|source| source.owned && source.eids.iter().any(|eid| !eid.uids.is_empty())) + .filter_map(|source| normalize_partner_source_domain(&source.source_domain).ok()) + .collect(); + let is_owned = |eid: &Eid| { + normalize_partner_source_domain(&eid.source) + .is_ok_and(|source| owned_sources.contains(&source)) + }; + let mut merged: Vec = filter_browser_candidates(client_eids, denials) .into_iter() - .flatten() - .filter(|eid| !eid.uids.is_empty()) + .filter(|eid| !eid.uids.is_empty() && !is_owned(eid)) .collect(); let browser_sources: HashSet = merged .iter() .filter_map(|eid| normalize_partner_source_domain(&eid.source).ok()) .collect(); - for eid in resolved_eids.into_iter().flatten() { - if eid.uids.is_empty() { - continue; - } - let carried_by_browser = normalize_partner_source_domain(&eid.source) - .is_ok_and(|source| browser_sources.contains(&source)); - if !carried_by_browser { - merged.push(eid); + for source in server.sources() { + for eid in &source.eids { + if eid.uids.is_empty() { + continue; + } + let carried_by_browser = normalize_partner_source_domain(&eid.source) + .is_ok_and(|source| browser_sources.contains(&source)); + if source.owned || !carried_by_browser { + merged.push(eid.clone()); + } } } - - if merged.is_empty() { - None - } else { - Some(merged) - } + merged } #[cfg(test)] @@ -543,6 +793,11 @@ mod tests { use crate::auction::types::{AuctionRequest, AuctionResponse}; use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::ConsentContext; + use crate::ec::identity::revision::uid_hash; + use crate::ec::identity::test_support::{ + ec_partner, example_entry_set, full_record, pid, provider_from, source, + }; + use crate::ec::kv_types::{KvEid, KvEntry, KvPartnerId, KvProviderState, KvUid}; use crate::error::IntoHttpResponse as _; use crate::openrtb::Uid; use crate::platform::test_support::{ @@ -555,6 +810,7 @@ mod tests { use base64::engine::general_purpose::STANDARD as BASE64; use serde_json::Value as JsonValue; use serde_json::json; + use std::collections::BTreeMap; use std::sync::{Arc, Mutex}; #[derive(Default)] @@ -614,8 +870,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: crate::settings::EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: crate::settings::EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -706,14 +963,13 @@ mod tests { 1, "finalization must reuse the endpoint snapshot instead of reading again" ); - let (stored, _) = graph - .get(&ec_id) - .expect("should read store") - .expect("row should exist"); - assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some("shared-cookie-id"), - "the sharedId update must still be ingested from the shared snapshot" + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "/auction is a stager and must not write the sharedId" + ); + assert!( + ec_context.identity().mutation().is_empty(), + "/auction is a stager and must not queue the sharedId" ); } @@ -752,12 +1008,17 @@ mod tests { stored .ids .iter() - .map(|(source, id)| (source.clone(), id.uid.clone())) + .map(|(source, id)| { + ( + source.clone(), + id.first_uid().unwrap_or_default().to_owned(), + ) + }) .collect() } #[tokio::test] - async fn auction_ingests_its_body_eids_and_not_a_cookie_only_source() { + async fn auction_ingests_nothing_from_its_body_or_the_cookie() { let settings = create_test_settings(); let mut orchestrator = AuctionOrchestrator::new(AuctionConfig { enabled: true, @@ -809,15 +1070,18 @@ mod tests { &mut response, ); - assert_eq!( - stored_ids(&graph, &ec_id), - vec![("body.example.com".to_owned(), "body-uid".to_owned())], - "KV should learn the body's source and not the cookie-only source" + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "KV should learn nothing from an /auction" + ); + assert!( + ec_context.identity().mutation().is_empty(), + "/auction should queue nothing" ); } #[tokio::test] - async fn disabled_auction_ingests_nothing_from_the_cookie_but_keeps_shared_id() { + async fn disabled_auction_ingests_nothing() { let settings = create_test_settings(); let orchestrator = AuctionOrchestrator::new(AuctionConfig { enabled: false, @@ -857,23 +1121,35 @@ mod tests { &mut response, ); - assert_eq!( - stored_ids(&graph, &ec_id), - vec![("sharedid.org".to_owned(), "shared-cookie-id".to_owned())], - "an /auction that resolved no client set should ingest only the sharedId cookie" + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "a disabled /auction should ingest nothing" + ); + assert!( + ec_context.identity().mutation().is_empty(), + "a disabled /auction should queue nothing" ); } #[test] - fn navigation_ingests_the_cookie_only_source() { + fn writer_navigation_queues_the_cookie_only_source() { // Control for the two tests above: the registry does match the - // cookie-only source when the route reads the cookie. + // cookie-only source when the writer navigation reads the cookie. let settings = create_test_settings(); let registry = ingestion_registry("body.example.com"); let ec_id = format!("{}.ingnav", "a".repeat(64)); let graph = seeded_graph(&ec_id); let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); ec_context.set_eid_sync_source(crate::ec::EidSyncSource::Navigation); + ec_context.identity_mut().set_writer_decision( + crate::ec::identity::request::WriterDecision { + writer: true, + is_real_browser: true, + user_agent: None, + dnt: false, + page_url: None, + }, + ); let cookie = cookie_only_ts_eids(); let mut response = http::Response::new(EdgeBody::empty()); @@ -887,21 +1163,26 @@ mod tests { &mut response, ); - assert_eq!( - stored_ids(&graph, &ec_id), - vec![(COOKIE_ONLY_SOURCE.to_owned(), "cookie-uid".to_owned())], - "a navigation should ingest the cookie's registered source" + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "a navigation should leave the write to phase 4" + ); + let queued = ec_context.identity().mutation().positive_changes(); + assert!( + matches!( + queued, + [crate::ec::identity::mutation::PartnerIdChange::FillMissing { source, record }] + if source.as_str() == COOKIE_ONLY_SOURCE + && record.first_uid() == Some("cookie-uid") + ), + "a navigation should queue the cookie's registered source" ); } #[tokio::test] - async fn auction_body_eids_reach_kv_even_when_the_ts_eids_cookie_is_absent() { - // Regression test for #1184: `/auction` sends every EID it has in the - // request body, but response finalization used to ingest identity - // graph updates only from the `ts-eids` cookie — which the browser - // caps in size and may not have sent at all. `handle_auction` must - // hand its parsed body EIDs to `ec_context` so finalization ingests - // them regardless of what the cookie carried. + async fn auction_body_eids_never_reach_kv() { + // The `/auction` body reaches bidders but is not an ingest input + // (spec G10): finalization neither writes nor queues it. let settings = create_test_settings(); let mut orchestrator = AuctionOrchestrator::new(AuctionConfig { enabled: true, @@ -956,15 +1237,6 @@ mod tests { ) .await; - assert_eq!( - ec_context.client_eids().map(|eids| eids - .iter() - .map(|eid| eid.source.as_str()) - .collect::>()), - Some(vec!["id5-sync.com"]), - "the endpoint must hand its parsed body EIDs to the request context" - ); - let mut response = http::Response::new(EdgeBody::empty()); crate::ec::finalize::ec_finalize_response( &settings, @@ -976,14 +1248,13 @@ mod tests { &mut response, ); - let (stored, _) = graph - .get(&ec_id) - .expect("should read store") - .expect("row should exist"); - assert_eq!( - stored.ids.get("id5-sync.com").map(|id| id.uid.as_str()), - Some("ID5_from_body"), - "the body's EID must be ingested into KV without a ts-eids cookie" + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "the body's EID must not be ingested into KV" + ); + assert!( + ec_context.identity().mutation().is_empty(), + "the body's EID must not be queued" ); } @@ -1669,7 +1940,12 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + PolicySignals::default(), + ); assert!( result.is_some_and(|eids| eids.is_empty()), "should degrade to empty EIDs without a snapshot" @@ -1681,7 +1957,12 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, None, &ec_context); + let result = resolve_auction_eids( + &EcKvSnapshot::NotRead, + None, + &ec_context, + PolicySignals::default(), + ); assert!( result.is_none(), "should return None when registry is missing" @@ -1694,7 +1975,12 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::Unknown, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + PolicySignals::default(), + ); assert!( result.is_none(), "should return None when consent is denied" @@ -1706,7 +1992,12 @@ mod tests { let registry = PartnerRegistry::empty(); let ec_context = make_ec_context(Jurisdiction::NonRegulated, None); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + PolicySignals::default(), + ); assert!( result.is_none(), "should return None when no EC value is present" @@ -1722,7 +2013,12 @@ mod tests { let snapshot = EcKvSnapshot::Failed { ec_id: ec_id.clone(), }; - let result = resolve_auction_eids(&snapshot, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &snapshot, + Some(®istry), + &ec_context, + PolicySignals::default(), + ); let eids = result.expect("should return Some on KV error (degraded mode)"); assert!( eids.is_empty(), @@ -1730,6 +2026,131 @@ mod tests { ); } + #[test] + fn generic_pull_record_with_writer_is_served_after_flush() { + use crate::ec::identity::IdentityRuntime; + use crate::ec::identity::test_support::{ec_partner, record_for_test}; + use crate::test_support::tests::create_test_settings; + + let mut partner = ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + "example-token-value-0123456789abcdef", + ); + partner.api_token = Some(crate::redacted::Redacted::new( + "token-ssp-example-com-32-bytes-minimum".to_owned(), + )); + let registry = PartnerRegistry::from_config(std::slice::from_ref(&partner)) + .expect("should build registry"); + let mut settings = create_test_settings(); + settings.ec.partners = vec![partner]; + let with_provider = std::sync::Arc::new( + IdentityRuntime::from_settings( + &settings, + &crate::integrations::IntegrationRegistry::default(), + ) + .expect("should build the runtime"), + ); + settings.ec.partners = Vec::new(); + let without_provider = std::sync::Arc::new( + IdentityRuntime::from_settings( + &settings, + &crate::integrations::IntegrationRegistry::default(), + ) + .expect("should build the runtime"), + ); + + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = crate::ec::kv_types::KvEntry::minimal("other.example.com", "other", 1); + entry.ids.insert( + "ssp.example.com".to_owned(), + record_for_test("pulled-uid", Some("pull-ssp-example-com"), 1), + ); + let snapshot = EcKvSnapshot::Present { + ec_id: ec_id.clone(), + entry: Box::new(entry), + generation: None, + }; + let resolve = |runtime: Option>| { + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + if let Some(runtime) = runtime { + ec_context.set_identity_runtime(runtime); + } + resolve_auction_eids( + &snapshot, + Some(®istry), + &ec_context, + PolicySignals::default(), + ) + .expect("should resolve EIDs") + }; + + let served = resolve(Some(with_provider)); + let dropped = resolve(Some(without_provider)); + let no_runtime = resolve(None); + + assert!( + served.iter().any(|eid| eid.source == "ssp.example.com"), + "should send the pulled UID when the runtime's catalog holds its writer" + ); + assert!( + dropped.iter().all(|eid| eid.source != "ssp.example.com"), + "control: should drop the record when the catalog lacks its writer" + ); + assert!( + no_runtime.iter().all(|eid| eid.source != "ssp.example.com"), + "control: should drop the record without a runtime" + ); + } + + #[test] + fn resolve_auction_eids_without_providers_matches_pre_identity_output() { + let partners = vec![ + counting_test_partner("a.example.com"), + counting_test_partner("b.example.com"), + ]; + let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + let mut entry = crate::ec::kv_types::KvEntry::minimal("a.example.com", "ua", 1); + entry.ids.insert( + "b.example.com".to_owned(), + crate::ec::kv_types::KvPartnerId::new("ub"), + ); + let snapshot = EcKvSnapshot::Present { + ec_id: ec_id.clone(), + entry: Box::new(entry), + generation: None, + }; + + let eids = resolve_auction_eids( + &snapshot, + Some(®istry), + &ec_context, + PolicySignals::default(), + ) + .expect("should resolve EIDs"); + + let atype = crate::settings::EcPartner::default_openrtb_atype(); + let expected: Vec = [("a.example.com", "ua"), ("b.example.com", "ub")] + .into_iter() + .map(|(source, uid)| { + Eid::new( + source.to_owned(), + vec![crate::openrtb::Uid { + id: uid.to_owned(), + atype: Some(atype), + ext: None, + }], + ) + }) + .collect(); + assert_eq!( + eids, expected, + "should send exactly the EIDs sent before identity providers existed" + ); + } + #[test] fn resolve_client_auction_eids_falls_back_to_ts_eids_cookie() { let cookie_payload = json!([ @@ -1827,88 +2248,844 @@ mod tests { } fn merged_json(client: Option>, kv: Option>) -> JsonValue { - serde_json::to_value(merge_auction_eids(client, kv)).expect("should serialize merged EIDs") + let merged = merge_auction_eids( + &client.unwrap_or_default(), + &ServerSet::from_eids(kv.unwrap_or_default()), + &Denials::default(), + ); + let merged = (!merged.is_empty()).then_some(merged); + serde_json::to_value(merged).expect("should serialize merged EIDs") + } + + const MERGE_EC_ID_PREFIX: &str = "merge"; + + fn merge_ec_id() -> String { + format!("{}.{MERGE_EC_ID_PREFIX}", "a".repeat(64)) + } + + fn merge_registry(sources: &[&str]) -> PartnerRegistry { + let partners: Vec<_> = sources + .iter() + .map(|source| counting_test_partner(source)) + .collect(); + PartnerRegistry::from_config(&partners).expect("should build registry") + } + + fn merge_entry(records: Vec<(&str, KvPartnerId)>) -> KvEntry { + let mut entry = KvEntry::minimal("unused.example.com", "unused", 1); + entry.ids.clear(); + for (source, record) in records { + entry.ids.insert(source.to_owned(), record); + } + entry + } + + /// The request's [`AuctionIdentity`] for `entry`, with no providers. + fn merge_identity(entry: KvEntry, registry: &PartnerRegistry) -> AuctionIdentity { + let ec_id = merge_ec_id(); + let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + let snapshot = EcKvSnapshot::Present { + ec_id, + entry: Box::new(entry), + generation: None, + }; + auction_identity( + &snapshot, + Some(registry), + &ec_context, + PolicySignals::default(), + ) + } + + fn default_atype() -> i32 { + crate::settings::EcPartner::default_openrtb_atype() + } + + fn plain_eid(source: &str, id: &str, atype: Option) -> Eid { + Eid { + source: source.to_owned(), + uids: vec![Uid { + id: id.to_owned(), + atype, + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + } } #[test] - fn merge_case_1_forwards_a_browser_entry_with_its_fields() { - let merged = merged_json( - Some(vec![browser_eid( - SOURCE_S, - "u1", - None, - Some("inserter-1.example.com"), - Some(1), - )]), - None, + fn merge_browser_value_wins_for_unowned_source() { + let registry = merge_registry(&[SOURCE_S]); + let identity = merge_identity( + merge_entry(vec![(SOURCE_S, KvPartnerId::new("server"))]), + ®istry, ); + let merged = identity.merge(Some(&[plain_eid(SOURCE_S, "browser", Some(1))])); + assert_eq!( merged, - json!([{ "source": SOURCE_S, "inserter": "inserter-1.example.com", "mm": 1, "uids": [{ "id": "u1" }] }]), - "a browser entry should go out with its EID-level fields" + Some(vec![plain_eid(SOURCE_S, "browser", Some(1))]), + "the browser's value should win for an unowned source" ); } #[test] - fn merge_case_2_keeps_the_browser_atype_for_a_shared_uid() { - let merged = merged_json( - Some(vec![browser_eid( - SOURCE_S, - "u1", - Some(1), - Some("inserter-1.example.com"), - Some(1), - )]), - Some(vec![kv_eid(SOURCE_S, "u1", 3)]), + fn merge_server_value_goes_out_when_the_browser_did_not_send_the_source() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let identity = merge_identity( + merge_entry(vec![(SOURCE_T, KvPartnerId::new("server"))]), + ®istry, ); + let merged = identity.merge(Some(&[plain_eid(SOURCE_S, "browser", Some(1))])); + assert_eq!( merged, - json!([{ "source": SOURCE_S, "inserter": "inserter-1.example.com", "mm": 1, "uids": [{ "id": "u1", "atype": 1 }] }]), - "KV's operator-configured atype should not override the browser's" + Some(vec![ + plain_eid(SOURCE_S, "browser", Some(1)), + plain_eid(SOURCE_T, "server", Some(default_atype())), + ]), + "the server value should follow the browser entries" ); } #[test] - fn merge_case_3_drops_the_stale_kv_uid_for_a_browser_source() { - let merged = merged_json( - Some(vec![browser_eid( - SOURCE_S, - "u1", - None, - Some("inserter-1.example.com"), - Some(1), - )]), - Some(vec![kv_eid(SOURCE_S, "u0", 3)]), + fn merge_expired_entry_is_not_sent() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let expired = KvPartnerId { + exp: Some(1), + ..KvPartnerId::new("expired") + }; + let identity = merge_identity( + merge_entry(vec![ + (SOURCE_S, expired), + (SOURCE_T, KvPartnerId::new("live")), + ]), + ®istry, ); assert_eq!( - merged, - json!([{ "source": SOURCE_S, "inserter": "inserter-1.example.com", "mm": 1, "uids": [{ "id": "u1" }] }]), - "KV's value for a source the browser sends should not be forwarded" + identity.merge(None), + Some(vec![plain_eid(SOURCE_T, "live", Some(default_atype()))]), + "an expired record should not be sent" ); } #[test] - fn merge_case_4_appends_kv_sources_the_browser_did_not_send() { - let merged = merged_json( - Some(vec![browser_eid(SOURCE_S, "u1", None, None, None)]), - Some(vec![kv_eid(SOURCE_T, "t1", 3)]), + fn merge_stored_atype_and_ext_are_emitted() { + let registry = merge_registry(&[SOURCE_S]); + let identity = merge_identity( + merge_entry(vec![(SOURCE_S, full_record(None, example_entry_set()))]), + ®istry, ); assert_eq!( - merged, - json!([ - { "source": SOURCE_S, "uids": [{ "id": "u1" }] }, - { "source": SOURCE_T, "uids": [{ "id": "t1", "atype": 3 }] } + identity.merge(None), + Some(vec![ + Eid { + source: SOURCE_S.to_owned(), + uids: vec![ + Uid { + id: "example-a".to_owned(), + atype: Some(1), + ext: None, + }, + Uid { + id: "example-b".to_owned(), + atype: Some(3), + ext: Some(json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("example.com".to_owned()), + matcher: None, + mm: Some(3), + ext: None, + }, + Eid { + source: SOURCE_S.to_owned(), + uids: vec![Uid { + id: "example-c".to_owned(), + atype: Some(1), + ext: None, + }], + inserter: None, + matcher: Some("partner.example.com".to_owned()), + mm: None, + ext: None, + }, ]), - "browser entries should come first and KV-only sources after" + "the stored atype, UID ext and EID-level fields should go out" ); } #[test] - fn merge_case_5_keeps_same_source_entries_with_different_fields_apart() { + fn merge_entry_set_goes_out_one_eid_per_entry_in_stored_order_after_browser_entries() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let identity = merge_identity( + merge_entry(vec![(SOURCE_S, full_record(None, example_entry_set()))]), + ®istry, + ); + + let merged = identity + .merge(Some(&[plain_eid(SOURCE_T, "browser", None)])) + .expect("should merge EIDs"); + + let shape: Vec<(String, Vec)> = merged + .iter() + .map(|eid| { + ( + eid.source.clone(), + eid.uids.iter().map(|uid| uid.id.clone()).collect(), + ) + }) + .collect(); + assert_eq!( + shape, + vec![ + (SOURCE_T.to_owned(), vec!["browser".to_owned()]), + ( + SOURCE_S.to_owned(), + vec!["example-a".to_owned(), "example-b".to_owned()] + ), + (SOURCE_S.to_owned(), vec!["example-c".to_owned()]), + ], + "the stored set should go out one EID per entry, in order, after the browser" + ); + } + + fn decode_eids_header(value: &str) -> Vec { + serde_json::from_slice(&BASE64.decode(value).expect("should be base64")) + .expect("should be EID JSON") + } + + #[test] + fn merge_x_ts_eids_matches_what_was_sent() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let identity = merge_identity( + merge_entry(vec![(SOURCE_S, full_record(None, example_entry_set()))]), + ®istry, + ); + let merged = identity + .merge(Some(&[plain_eid(SOURCE_T, "browser", Some(1))])) + .expect("should merge EIDs"); + + let (header, truncated) = + crate::ec::eids::encode_eids_header(&merged).expect("should encode the header"); + + assert!(!truncated, "should fit without truncation"); + assert_eq!( + decode_eids_header(&header), + merged, + "x-ts-eids should carry exactly what was sent" + ); + } + + #[test] + fn merge_header_still_truncates_per_entry() { + let registry = merge_registry(&[SOURCE_S]); + let entries: Vec = (0..40) + .map(|index| KvEid { + uids: vec![KvUid { + id: format!("{index:03}-{}", "x".repeat(100)), + atype: Some(1), + ext: None, + }], + ..KvEid::default() + }) + .collect(); + let identity = merge_identity( + merge_entry(vec![(SOURCE_S, full_record(None, entries))]), + ®istry, + ); + let merged = identity.merge(None).expect("should merge EIDs"); + + let (header, truncated) = + crate::ec::eids::encode_eids_header(&merged).expect("should encode the header"); + + assert_eq!(merged.len(), 40, "user.eids should hold the whole set"); + assert!(truncated, "x-ts-eids-truncated should be set"); + let carried = decode_eids_header(&header); + assert!( + !carried.is_empty() && carried.len() < merged.len(), + "x-ts-eids should hold the entries that fit" + ); + assert_eq!( + carried, + merged[..carried.len()].to_vec(), + "the header should hold a prefix of whole entries" + ); + } + + #[test] + fn merge_denials_apply_through_the_request_identity() { + let partner = ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + "example-token-value-0123456789abcdef", + ); + let mut settings = create_test_settings(); + settings.ec.partners = vec![partner]; + let runtime = std::sync::Arc::new( + IdentityRuntime::from_settings( + &settings, + &crate::integrations::IntegrationRegistry::default(), + ) + .expect("should build the runtime"), + ); + let registry = merge_registry(&["ssp.example.com"]); + let ec_id = merge_ec_id(); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + ec_context.set_identity_runtime(runtime); + let mut entry = merge_entry(Vec::new()); + entry.ps.insert( + "pull-ssp-example-com".to_owned(), + KvProviderState { + iv: vec![uid_hash("invalidated")], + ..KvProviderState::default() + }, + ); + let snapshot = EcKvSnapshot::Present { + ec_id, + entry: Box::new(entry), + generation: None, + }; + let identity = auction_identity( + &snapshot, + Some(®istry), + &ec_context, + PolicySignals::default(), + ); + + let merged = identity.merge(Some(&[ + plain_eid("ssp.example.com", "invalidated", Some(1)), + plain_eid(SOURCE_T, "kept", Some(1)), + ])); + + assert_eq!( + merged, + Some(vec![plain_eid(SOURCE_T, "kept", Some(1))]), + "a browser UID the claiming provider invalidated should not be sent" + ); + } + + /// The request identity for `entry` when `provider` owns each of `owned`. + fn owned_identity( + entry: KvEntry, + registry: &PartnerRegistry, + provider: Arc, + owned: &[&str], + consent: ConsentContext, + ) -> (AuctionIdentity, EcContext) { + let owners = owned + .iter() + .map(|name| (source(name), provider.descriptor().id().clone())) + .collect(); + let runtime = Arc::new( + IdentityRuntime::from_settings_with_providers( + &create_test_settings(), + vec![provider], + owners, + ) + .expect("should build the runtime"), + ); + let ec_id = merge_ec_id(); + let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); + ec_context.set_identity_runtime(runtime); + ec_context.enable_mirror_for_test(); + let snapshot = EcKvSnapshot::Present { + ec_id, + entry: Box::new(entry), + generation: None, + }; + let identity = auction_identity( + &snapshot, + Some(registry), + &ec_context, + PolicySignals::default(), + ); + (identity, ec_context) + } + + fn non_regulated_consent() -> ConsentContext { + ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + ..ConsentContext::default() + } + } + + fn owner_provider(expiring: bool) -> Arc { + let mut builder = + crate::ec::identity::descriptor::ProviderDescriptor::builder(pid("owner")) + .with_source(source(SOURCE_S)) + .with_resolves( + crate::ec::identity::descriptor::Resolves::new() + .with_endpoint("https://api.example.com/fetch"), + ) + .with_consent_policy(crate::ec::identity::test_support::resolver_policy()); + if expiring { + builder = builder.with_expiring_ids(); + } + provider_from(builder.build().expect("should build the descriptor")) + } + + fn owner_record(uid: &str) -> KvPartnerId { + KvPartnerId { + w: Some("owner".to_owned()), + ..KvPartnerId::new(uid) + } + } + + #[test] + fn owner_valid_value_beats_browser_value() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let (identity, _) = owned_identity( + merge_entry(vec![(SOURCE_S, owner_record("owner-uid"))]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let merged = identity.merge(Some(&[ + plain_eid(SOURCE_S, "browser", Some(1)), + plain_eid(SOURCE_T, "other", Some(1)), + ])); + + assert_eq!( + merged, + Some(vec![ + plain_eid(SOURCE_T, "other", Some(1)), + plain_eid(SOURCE_S, "owner-uid", Some(default_atype())), + ]), + "the owner's value should replace the browser's, after the browser entries" + ); + } + + #[test] + fn owner_valid_set_replaces_every_repeated_browser_entry() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let (identity, _) = owned_identity( + merge_entry(vec![( + SOURCE_S, + full_record(Some("owner"), example_entry_set()), + )]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let merged = identity.merge(Some(&[ + plain_eid(SOURCE_S, "browser-one", Some(1)), + plain_eid(SOURCE_T, "other", Some(1)), + Eid { + inserter: Some("inserter.example.com".to_owned()), + ..plain_eid(SOURCE_S, "browser-two", Some(3)) + }, + ])); + + assert_eq!( + merged, + Some(vec![ + plain_eid(SOURCE_T, "other", Some(1)), + Eid { + source: SOURCE_S.to_owned(), + uids: vec![ + Uid { + id: "example-a".to_owned(), + atype: Some(1), + ext: None, + }, + Uid { + id: "example-b".to_owned(), + atype: Some(3), + ext: Some(json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("example.com".to_owned()), + matcher: None, + mm: Some(3), + ext: None, + }, + Eid { + source: SOURCE_S.to_owned(), + uids: vec![Uid { + id: "example-c".to_owned(), + atype: Some(1), + ext: None, + }], + inserter: None, + matcher: Some("partner.example.com".to_owned()), + mm: None, + ext: None, + }, + ]), + "exactly the owner's two entries should go out, in stored order, and no browser entry of the source" + ); + } + + #[test] + fn browser_seeded_value_never_beats_browser_current_value() { + let registry = merge_registry(&[SOURCE_S]); + let (identity, _) = owned_identity( + merge_entry(vec![(SOURCE_S, KvPartnerId::new("seeded"))]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let sent = identity.merge(Some(&[plain_eid(SOURCE_S, "current", Some(1))])); + let silent = identity.merge(None); + + assert_eq!( + sent, + Some(vec![plain_eid(SOURCE_S, "current", Some(1))]), + "the browser's current value should beat a browser-seeded record" + ); + assert_eq!( + silent, + Some(vec![plain_eid(SOURCE_S, "seeded", Some(default_atype()))]), + "control: a seeded value of a source that does not expire should go out when the browser is silent" + ); + } + + #[test] + fn expiring_owned_source_never_serves_browser_seeded_value() { + let registry = merge_registry(&[SOURCE_S]); + let (expiring, _) = owned_identity( + merge_entry(vec![(SOURCE_S, KvPartnerId::new("seeded"))]), + ®istry, + owner_provider(true), + &[SOURCE_S], + non_regulated_consent(), + ); + let (owned_expiring, _) = owned_identity( + merge_entry(vec![( + SOURCE_S, + KvPartnerId { + exp: Some(current_timestamp() + 1_000), + ..owner_record("owner-uid") + }, + )]), + ®istry, + owner_provider(true), + &[SOURCE_S], + non_regulated_consent(), + ); + + assert_eq!( + expiring.merge(None), + None, + "should not serve a browser-seeded value of an expiring owned source" + ); + assert_eq!( + owned_expiring.merge(None), + Some(vec![plain_eid( + SOURCE_S, + "owner-uid", + Some(default_atype()) + )]), + "control: the owner's unexpired value should go out" + ); + } + + #[test] + fn unowned_source_keeps_the_browser_first_rule() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let (identity, _) = owned_identity( + merge_entry(vec![ + (SOURCE_S, owner_record("owner-uid")), + (SOURCE_T, owner_record("claimed-uid")), + ]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let merged = identity.merge(Some(&[plain_eid(SOURCE_T, "browser", Some(1))])); + + assert_eq!( + merged, + Some(vec![ + plain_eid(SOURCE_T, "browser", Some(1)), + plain_eid(SOURCE_S, "owner-uid", Some(default_atype())), + ]), + "the browser should still win for a source with no owner" + ); + } + + #[test] + fn targeted_opt_out_blocks_server_ids_flush_and_mirror() { + let registry = merge_registry(&[SOURCE_S]); + let consent_with = |targeted: bool| ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + gpp: Some(crate::consent::types::GppConsent { + version: 1, + section_ids: vec![7], + eu_tcf: None, + us_sale_opt_out: Some(false), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: Some(targeted), + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }), + ..ConsentContext::default() + }; + let provider = owner_provider(false); + let entry = || merge_entry(vec![(SOURCE_S, owner_record("owner-uid"))]); + let (blocked, blocked_context) = owned_identity( + entry(), + ®istry, + provider.clone(), + &[SOURCE_S], + consent_with(true), + ); + let (allowed, allowed_context) = owned_identity( + entry(), + ®istry, + provider.clone(), + &[SOURCE_S], + consent_with(false), + ); + + assert_eq!( + blocked.merge(None), + None, + "a targeted-advertising opt-out should leave no server-side ID in user.eids" + ); + assert_eq!( + blocked.mirror_member(&blocked_context), + Some(json!({ "clear": true })), + "the ts-eids-s mirror should carry none of the provider's entries" + ); + assert!( + allowed.merge(None).is_some() + && allowed + .mirror_member(&allowed_context) + .is_some_and(|member| member.get("value").is_some()), + "control: without the opt-out the ID should go out and be mirrored" + ); + + let catalog = ProviderCatalog::new(vec![provider.clone()], BTreeMap::new()); + let settings = crate::settings::ProviderSettings::default(); + let identity_settings = crate::settings::IdentitySettings::default(); + let check = |consent: &ConsentContext| { + crate::ec::identity::coordinator::eligibility( + &crate::ec::identity::coordinator::EligibilityInput { + provider: provider.as_ref(), + settings: &settings, + identity: &identity_settings, + catalog: &catalog, + consent, + }, + &crate::ec::identity::coordinator::EligibilityView::empty(), + &crate::ec::identity::test_support::facts_for_writer(), + crate::ec::identity::coordinator::Phase::AfterSend, + ) + }; + assert!( + matches!( + check(&consent_with(true)), + crate::ec::identity::coordinator::Eligibility::Skip( + crate::ec::identity::coordinator::SkipReason::ConsentPolicy + ) + ), + "the provider should not be dispatched, so nothing of it is flushed" + ); + assert!( + matches!( + check(&consent_with(false)), + crate::ec::identity::coordinator::Eligibility::Eligible { .. } + ), + "control: without the opt-out the provider should be eligible" + ); + } + + #[test] + fn golden_merge_one_uid_row_goes_out_as_on_main() { + let registry = merge_registry(&["a.example.com", "b.example.com"]); + let identity = merge_identity( + merge_entry(vec![ + ("b.example.com", KvPartnerId::new("ub")), + ("a.example.com", KvPartnerId::new("ua")), + ]), + ®istry, + ); + + // Written from `main`'s `merge_auction_eids(None, Some(to_eids(..)))`. + let expected = vec![ + Eid { + source: "a.example.com".to_owned(), + uids: vec![Uid { + id: "ua".to_owned(), + atype: Some(3), + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + }, + Eid { + source: "b.example.com".to_owned(), + uids: vec![Uid { + id: "ub".to_owned(), + atype: Some(3), + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + }, + ]; + assert_eq!( + identity.merge(None), + Some(expected), + "a one-UID row should go out exactly as on main" + ); + } + + #[test] + fn golden_merge_browser_entries_are_forwarded_unchanged() { + let client = vec![ + Eid { + source: SOURCE_S.to_owned(), + uids: vec![ + Uid { + id: "s1".to_owned(), + atype: Some(1), + ext: Some(json!({ "stype": "ppuid" })), + }, + Uid { + id: "s2".to_owned(), + atype: None, + ext: None, + }, + ], + inserter: Some("inserter-1.example.com".to_owned()), + matcher: Some("matcher.example.com".to_owned()), + mm: Some(1), + ext: Some(json!({ "segments": ["a"] })), + }, + Eid { + source: SOURCE_T.to_owned(), + uids: Vec::new(), + inserter: None, + matcher: None, + mm: None, + ext: None, + }, + Eid { + source: SOURCE_S.to_owned(), + uids: vec![Uid { + id: "s1".to_owned(), + atype: Some(3), + ext: None, + }], + inserter: Some("inserter-2.example.com".to_owned()), + matcher: None, + mm: Some(2), + ext: None, + }, + ]; + + let merged = merge_auction_eids(&client, &ServerSet::default(), &Denials::default()); + + assert_eq!( + merged, + vec![client[0].clone(), client[2].clone()], + "browser entries should go out as today, minus entries without UIDs" + ); + } + + #[test] + fn merge_case_1_forwards_a_browser_entry_with_its_fields() { + let merged = merged_json( + Some(vec![browser_eid( + SOURCE_S, + "u1", + None, + Some("inserter-1.example.com"), + Some(1), + )]), + None, + ); + + assert_eq!( + merged, + json!([{ "source": SOURCE_S, "inserter": "inserter-1.example.com", "mm": 1, "uids": [{ "id": "u1" }] }]), + "a browser entry should go out with its EID-level fields" + ); + } + + #[test] + fn merge_case_2_keeps_the_browser_atype_for_a_shared_uid() { + let merged = merged_json( + Some(vec![browser_eid( + SOURCE_S, + "u1", + Some(1), + Some("inserter-1.example.com"), + Some(1), + )]), + Some(vec![kv_eid(SOURCE_S, "u1", 3)]), + ); + + assert_eq!( + merged, + json!([{ "source": SOURCE_S, "inserter": "inserter-1.example.com", "mm": 1, "uids": [{ "id": "u1", "atype": 1 }] }]), + "KV's operator-configured atype should not override the browser's" + ); + } + + #[test] + fn merge_case_3_drops_the_stale_kv_uid_for_a_browser_source() { + let merged = merged_json( + Some(vec![browser_eid( + SOURCE_S, + "u1", + None, + Some("inserter-1.example.com"), + Some(1), + )]), + Some(vec![kv_eid(SOURCE_S, "u0", 3)]), + ); + + assert_eq!( + merged, + json!([{ "source": SOURCE_S, "inserter": "inserter-1.example.com", "mm": 1, "uids": [{ "id": "u1" }] }]), + "KV's value for a source the browser sends should not be forwarded" + ); + } + + #[test] + fn merge_case_4_appends_kv_sources_the_browser_did_not_send() { + let merged = merged_json( + Some(vec![browser_eid(SOURCE_S, "u1", None, None, None)]), + Some(vec![kv_eid(SOURCE_T, "t1", 3)]), + ); + + assert_eq!( + merged, + json!([ + { "source": SOURCE_S, "uids": [{ "id": "u1" }] }, + { "source": SOURCE_T, "uids": [{ "id": "t1", "atype": 3 }] } + ]), + "browser entries should come first and KV-only sources after" + ); + } + + #[test] + fn merge_case_5_keeps_same_source_entries_with_different_fields_apart() { let merged = merged_json( Some(vec![ browser_eid( @@ -2144,4 +3321,479 @@ mod tests { ); }); } + + mod mirror { + use super::*; + use crate::ec::kv_types::{KvEntry, KvPartnerId}; + use base64::engine::general_purpose::URL_SAFE_NO_PAD; + + const MIRROR_SOURCE_A: &str = "a.example.com"; + const MIRROR_SOURCE_B: &str = "b.example.com"; + + fn mirror_registry() -> PartnerRegistry { + let mut disabled = counting_test_partner("off.example.com"); + disabled.bidstream_enabled = false; + PartnerRegistry::from_config(&[ + counting_test_partner(MIRROR_SOURCE_A), + counting_test_partner(MIRROR_SOURCE_B), + disabled, + ]) + .expect("should build partner registry") + } + + fn row_with(records: &[(&str, &str, Option)]) -> KvEntry { + let mut entry = KvEntry::minimal("unregistered.example.com", "x", 1); + entry.ids.clear(); + for (source, uid, exp) in records { + entry.ids.insert( + (*source).to_owned(), + KvPartnerId { + exp: *exp, + ..KvPartnerId::browser(*uid, Some(1)) + }, + ); + } + entry + } + + fn ec_id_for(tag: &str) -> String { + format!("{}.{tag}", "a".repeat(64)) + } + + /// Records the EIDs its request carried, then answers no-bid so the + /// auction completes and the response can be inspected. + struct NoBidRecordingProvider { + eids: Arc>>, + } + + #[async_trait::async_trait(?Send)] + impl AuctionProvider for NoBidRecordingProvider { + fn provider_name(&self) -> &str { + "eid_recording_provider" + } + + async fn request_bids( + &self, + request: &AuctionRequest, + _context: &AuctionContext<'_>, + ) -> Result> { + *self.eids.lock().expect("should lock recorded eids") = Some( + serde_json::to_value(&request.user.eids) + .expect("should serialize recorded eids"), + ); + Ok(ProviderRequestOutcome::Immediate(AuctionResponse::no_bid( + self.provider_name(), + 0, + ))) + } + + async fn parse_response( + &self, + _response: PlatformResponse, + _response_time_ms: u64, + ) -> Result> { + panic!("parse_response must not run for an immediate answer"); + } + + fn timeout_ms(&self) -> u32 { + 100 + } + + fn backend_name( + &self, + _services: &RuntimeServices, + _timeout_ms: u32, + ) -> Option { + Some("record-backend".to_string()) + } + } + + fn recording_orchestrator(recorded: &Arc>>) -> AuctionOrchestrator { + let mut orchestrator = AuctionOrchestrator::new(AuctionConfig { + enabled: true, + providers: AuctionConfig::legacy_provider_map(&["eid_recording_provider"]), + timeout_ms: 2000, + mediator: None, + ..Default::default() + }); + orchestrator.register_provider(Arc::new(NoBidRecordingProvider { + eids: Arc::clone(recorded), + })); + orchestrator + } + + async fn run_auction( + ec_context: &mut EcContext, + graph: Option<&KvIdentityGraph>, + registry: Option<&PartnerRegistry>, + body: Vec, + ) -> JsonValue { + let recorded = Arc::new(Mutex::new(None)); + let orchestrator = recording_orchestrator(&recorded); + let req = Request::builder() + .method("POST") + .uri("https://test-publisher.com/auction") + .body(EdgeBody::from(body)) + .expect("should build auction request"); + let response = handle_auction( + &create_test_settings(), + &orchestrator, + graph, + registry, + ec_context, + &noop_services(), + req, + ) + .await + .expect("should answer the auction"); + let bytes = response + .into_body() + .into_bytes() + .expect("should buffer the body"); + serde_json::from_slice(&bytes).expect("should parse the response") + } + + fn mirror_of(response: &JsonValue) -> Option<&JsonValue> { + response.get("ext").and_then(|ext| ext.get("ts_eids_s")) + } + + fn decoded(mirror: &JsonValue) -> JsonValue { + let value = mirror["value"].as_str().expect("should hold a value"); + let bytes = URL_SAFE_NO_PAD + .decode(value) + .expect("should be unpadded base64url"); + serde_json::from_slice(&bytes).expect("should be JSON") + } + + fn live_context(ec_id: &str, graph: &KvIdentityGraph) -> EcContext { + let _ = graph; + let mut context = make_ec_context(Jurisdiction::NonRegulated, Some(ec_id)); + context.enable_mirror_for_test(); + context + } + + #[tokio::test] + async fn auction_body_eids_are_not_written_to_kv() { + let (graph, store) = KvIdentityGraph::write_counting("test_store"); + let ec_id = ec_id_for("nowrit"); + graph + .create(&ec_id, &row_with(&[])) + .expect("should seed the live row"); + store.reset_counts(); + let registry = mirror_registry(); + let recorded = Arc::new(Mutex::new(None)); + let orchestrator = recording_orchestrator(&recorded); + let mut ec_context = live_context(&ec_id, &graph); + let req = Request::builder() + .method("POST") + .uri("https://test-publisher.com/auction") + .body(EdgeBody::from( + serde_json::to_vec(&json!({ + "adUnits": [{ "code": "div-gpt-1", "mediaTypes": { "banner": { "sizes": [[300, 250]] } } }], + "eids": [{ "source": MIRROR_SOURCE_A, "uids": [{ "id": "body-uid", "atype": 1 }] }] + })) + .expect("should serialize body"), + )) + .expect("should build auction request"); + + let _ = handle_auction( + &create_test_settings(), + &orchestrator, + Some(&graph), + Some(®istry), + &mut ec_context, + &noop_services(), + req, + ) + .await; + let mut response = http::Response::new(EdgeBody::empty()); + crate::ec::finalize::ec_finalize_response( + &create_test_settings(), + &mut ec_context, + Some(&graph), + ®istry, + None, + None, + &mut response, + ); + + let writes = store.writes(); + assert_eq!( + (writes.add, writes.if_generation_match, writes.overwrite), + (0, 0, 0), + "should write nothing to KV" + ); + assert_eq!( + recorded.lock().expect("should lock recorded eids").clone(), + Some( + json!([{ "source": MIRROR_SOURCE_A, "uids": [{ "id": "body-uid", "atype": 1 }] }]) + ), + "should still send the body's UID to the bidder" + ); + } + + #[tokio::test] + async fn auction_omits_the_mirror_while_it_is_disabled() { + let ec_id = ec_id_for("mirror-off"); + let graph = KvIdentityGraph::in_memory("test_store"); + graph + .create(&ec_id, &row_with(&[(MIRROR_SOURCE_A, "uid-a", None)])) + .expect("should seed the live row"); + let registry = mirror_registry(); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + + let response = run_auction( + &mut ec_context, + Some(&graph), + Some(®istry), + ad_units_only_body(), + ) + .await; + + assert_eq!( + mirror_of(&response), + None, + "should return no mirror member while [ec.identity] mirror_enabled is off" + ); + } + + #[tokio::test] + async fn auction_returns_the_mirror_of_its_servable_set() { + let ec_id = ec_id_for("mirror"); + let graph = KvIdentityGraph::in_memory("test_store"); + graph + .create(&ec_id, &row_with(&[(MIRROR_SOURCE_A, "uid-a", None)])) + .expect("should seed the live row"); + let registry = mirror_registry(); + let mut ec_context = live_context(&ec_id, &graph); + + let response = run_auction( + &mut ec_context, + Some(&graph), + Some(®istry), + ad_units_only_body(), + ) + .await; + + let mirror = mirror_of(&response).expect("should carry the mirror"); + let atype = crate::settings::EcPartner::default_openrtb_atype(); + assert_eq!( + decoded(mirror), + json!({ "v": 1, "eids": [{ "source": MIRROR_SOURCE_A, "uids": [{ "id": "uid-a", "atype": atype }] }] }), + "should hold exactly the servable record's EID" + ); + assert_eq!( + mirror["max_age"], 86_400, + "should live a day without an exp" + ); + } + + #[tokio::test] + async fn auction_mirror_max_age_follows_the_earliest_exp() { + let now = crate::ec::current_timestamp(); + let ec_id = ec_id_for("maxage"); + let graph = KvIdentityGraph::in_memory("test_store"); + graph + .create( + &ec_id, + &row_with(&[ + (MIRROR_SOURCE_A, "uid-a", Some(now + 600)), + (MIRROR_SOURCE_B, "uid-b", Some(now + 999_999)), + ]), + ) + .expect("should seed the live row"); + let registry = mirror_registry(); + let mut ec_context = live_context(&ec_id, &graph); + + let response = run_auction( + &mut ec_context, + Some(&graph), + Some(®istry), + ad_units_only_body(), + ) + .await; + + let max_age = mirror_of(&response).expect("should carry the mirror")["max_age"] + .as_u64() + .expect("should hold a number"); + assert!( + (595..=600).contains(&max_age), + "should take the earliest exp, got {max_age}" + ); + } + + #[tokio::test] + async fn auction_mirror_leaves_out_bidstream_disabled_sources() { + let ec_id = ec_id_for("bidoff"); + let graph = KvIdentityGraph::in_memory("test_store"); + graph + .create( + &ec_id, + &row_with(&[ + (MIRROR_SOURCE_A, "uid-a", None), + ("off.example.com", "uid-off", None), + ]), + ) + .expect("should seed the live row"); + let registry = mirror_registry(); + let mut ec_context = live_context(&ec_id, &graph); + + let response = run_auction( + &mut ec_context, + Some(&graph), + Some(®istry), + ad_units_only_body(), + ) + .await; + + let sources: Vec = + decoded(mirror_of(&response).expect("should carry the mirror"))["eids"] + .as_array() + .expect("should hold EIDs") + .iter() + .map(|eid| eid["source"].as_str().unwrap_or_default().to_owned()) + .collect(); + assert_eq!( + sources, + vec![MIRROR_SOURCE_A.to_owned()], + "should leave out the source bidstream disabled" + ); + } + + #[tokio::test] + async fn auction_mirror_leaves_out_ids_consent_filters_out() { + // TCF Purpose 1 granted, Purpose 4 refused: the auction runs, but + // `gate_eids_by_consent` strips every EID the bidder would get. + let ec_id = ec_id_for("gated"); + let graph = KvIdentityGraph::in_memory("test_store"); + graph + .create(&ec_id, &row_with(&[(MIRROR_SOURCE_A, "uid-a", None)])) + .expect("should seed the live row"); + let registry = mirror_registry(); + let mut ec_context = + EcContext::new_for_test(Some(ec_id.clone()), gdpr_consent_with_purposes(&[1], 4)); + ec_context.enable_mirror_for_test(); + + let response = run_auction( + &mut ec_context, + Some(&graph), + Some(®istry), + ad_units_only_body(), + ) + .await; + + assert_eq!( + mirror_of(&response), + Some(&json!({ "clear": true })), + "should not mirror an ID consent filtered out" + ); + } + + fn gdpr_consent_with_purposes(granted: &[usize], _refused: usize) -> ConsentContext { + let mut consent = + crate::ec::identity::test_support::test_consent_gdpr_without_vendor(9_999); + let tcf = consent.tcf.as_mut().expect("should hold a TCF string"); + tcf.purpose_consents = (1..=24).map(|purpose| granted.contains(&purpose)).collect(); + consent + } + + #[tokio::test] + async fn auction_returns_clear_on_consent_denial_tombstone_and_missing_row() { + let registry = mirror_registry(); + let live = row_with(&[(MIRROR_SOURCE_A, "uid-a", None)]); + let mut withdrawn = live.clone(); + withdrawn.consent.ok = false; + let cases: [(&str, Option, bool, JsonValue); 5] = [ + ("live row", Some(live.clone()), false, json!("value")), + ( + "tombstone", + Some(KvEntry::tombstone(1_000)), + false, + json!({ "clear": true }), + ), + ( + "consent.ok false", + Some(withdrawn), + false, + json!({ "clear": true }), + ), + ("missing row", None, false, json!({ "clear": true })), + ("consent denial", Some(live), true, json!({ "clear": true })), + ]; + + for (label, row, deny_consent, expected) in cases { + let ec_id = ec_id_for("clears"); + let graph = KvIdentityGraph::in_memory("test_store"); + if let Some(row) = &row { + graph.create(&ec_id, row).expect("should seed the row"); + } + let mut ec_context = if deny_consent { + // No TCF Purpose 1: the auction consent gate answers early. + EcContext::new_for_test(Some(ec_id.clone()), gdpr_consent_with_purposes(&[], 4)) + } else { + live_context(&ec_id, &graph) + }; + + let response = run_auction( + &mut ec_context, + Some(&graph), + Some(®istry), + ad_units_only_body(), + ) + .await; + + let mirror = mirror_of(&response) + .unwrap_or_else(|| panic!("{label}: should carry a mirror")); + if expected == json!("value") { + assert!( + mirror.get("value").is_some(), + "{label}: should carry a value" + ); + } else { + assert_eq!(mirror, &expected, "{label}: should clear"); + } + } + } + + #[tokio::test] + async fn auction_returns_clear_when_the_servable_set_is_empty() { + let ec_id = ec_id_for("empty"); + let graph = KvIdentityGraph::in_memory("test_store"); + graph + .create(&ec_id, &row_with(&[("off.example.com", "uid-off", None)])) + .expect("should seed the live row"); + let registry = mirror_registry(); + let mut ec_context = live_context(&ec_id, &graph); + + let response = run_auction( + &mut ec_context, + Some(&graph), + Some(®istry), + ad_units_only_body(), + ) + .await; + + assert_eq!( + mirror_of(&response), + Some(&json!({ "clear": true })), + "should clear an empty servable set" + ); + } + + #[tokio::test] + async fn auction_without_a_kv_read_omits_the_mirror() { + let ec_id = ec_id_for("noread"); + let registry = mirror_registry(); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + ec_context.enable_mirror_for_test(); + + let response = + run_auction(&mut ec_context, None, Some(®istry), ad_units_only_body()).await; + + assert!( + response["ext"].get("ts_eids_s").is_none(), + "should leave the mirror cookie as it is" + ); + } + } } diff --git a/crates/trusted-server-core/src/auction/formats.rs b/crates/trusted-server-core/src/auction/formats.rs index cec4823005..de037e9d37 100644 --- a/crates/trusted-server-core/src/auction/formats.rs +++ b/crates/trusted-server-core/src/auction/formats.rs @@ -341,10 +341,30 @@ pub fn convert_to_openrtb_response( auction_request: &AuctionRequest, ec_allowed: bool, ) -> Result, Report> { - Ok( - convert_to_openrtb_response_with_report(result, settings, auction_request, ec_allowed)? - .response, - ) + convert_to_openrtb_response_with_mirror(result, settings, auction_request, ec_allowed, None) +} + +/// Like [`convert_to_openrtb_response`], carrying the `ts-eids-s` mirror +/// payload as `ext.ts_eids_s` (spec 6.2). +/// +/// # Errors +/// +/// Returns an error if response serialization fails. +pub(crate) fn convert_to_openrtb_response_with_mirror( + result: &OrchestrationResult, + settings: &Settings, + auction_request: &AuctionRequest, + ec_allowed: bool, + ts_eids_s: Option, +) -> Result, Report> { + Ok(convert_to_openrtb_response_with_report( + result, + settings, + auction_request, + ec_allowed, + ts_eids_s, + )? + .response) } pub(crate) fn convert_to_openrtb_response_with_report( @@ -352,6 +372,7 @@ pub(crate) fn convert_to_openrtb_response_with_report( settings: &Settings, auction_request: &AuctionRequest, ec_allowed: bool, + ts_eids_s: Option, ) -> Result> { let mut seatbids = Vec::with_capacity(result.winning_bids.len()); let rewrite_creatives = settings.auction.rewrite_creatives; @@ -512,6 +533,7 @@ pub(crate) fn convert_to_openrtb_response_with_report( dropped_winner_count: delivery.dropped_winner_count, dropped_winner_reasons: delivery.dropped_winner_reasons.clone(), }, + ts_eids_s, } .to_ext(), ..Default::default() @@ -1601,9 +1623,14 @@ mod tests { metadata: HashMap::new(), }; - let conversion = - convert_to_openrtb_response_with_report(&result, &settings, &auction_request, false) - .expect("should omit invalid winners and preserve valid slots"); + let conversion = convert_to_openrtb_response_with_report( + &result, + &settings, + &auction_request, + false, + None, + ) + .expect("should omit invalid winners and preserve valid slots"); assert_eq!( conversion.delivery.delivered_winner_slots, HashSet::from(["ordinary".to_string(), "renderer".to_string()]), @@ -1926,9 +1953,14 @@ mod tests { let auction_request = make_auction_request(); let result = make_result(make_bid("div-gpt-top", "appnexus", None)); - let conversion = - convert_to_openrtb_response_with_report(&result, &settings, &auction_request, false) - .expect("should omit a winner without a decoded price"); + let conversion = convert_to_openrtb_response_with_report( + &result, + &settings, + &auction_request, + false, + None, + ) + .expect("should omit a winner without a decoded price"); assert!(conversion.delivery.delivered_winner_slots.is_empty()); assert_eq!(conversion.delivery.dropped_winner_count, 1); assert_eq!( diff --git a/crates/trusted-server-core/src/auction/plan.rs b/crates/trusted-server-core/src/auction/plan.rs index df36fc09b7..9bd739fc5a 100644 --- a/crates/trusted-server-core/src/auction/plan.rs +++ b/crates/trusted-server-core/src/auction/plan.rs @@ -396,6 +396,7 @@ impl ProviderPlan { certificate_check: true, first_byte_timeout: timeout, between_bytes_timeout: timeout, + connect_timeout: None, discriminator: Some(self.id.as_str().to_owned()), } } diff --git a/crates/trusted-server-core/src/auction/routing.rs b/crates/trusted-server-core/src/auction/routing.rs index 2b8135da3c..97f74b334d 100644 --- a/crates/trusted-server-core/src/auction/routing.rs +++ b/crates/trusted-server-core/src/auction/routing.rs @@ -203,15 +203,16 @@ impl ProviderSlotInput { /// the same legacy handling. Client-supplied `X-Forwarded-For` is never read. #[derive(Debug, Clone, Default)] pub(crate) struct PrebidTransportHeaders { - cookie: Option, + cookies: Vec, user_agent: Option, referer: Option, accept_language: Option, } impl PrebidTransportHeaders { - pub(crate) fn cookie(&self) -> Option<&HeaderValue> { - self.cookie.as_ref() + /// Every inbound `Cookie` field, in order. + pub(crate) fn cookies(&self) -> &[HeaderValue] { + &self.cookies } pub(crate) fn user_agent(&self) -> Option<&HeaderValue> { @@ -228,7 +229,12 @@ impl PrebidTransportHeaders { fn snapshot(request: &Request) -> Self { Self { - cookie: request.headers().get(header::COOKIE).cloned(), + cookies: request + .headers() + .get_all(header::COOKIE) + .iter() + .cloned() + .collect(), user_agent: request.headers().get(header::USER_AGENT).cloned(), referer: request.headers().get(header::REFERER).cloned(), accept_language: request.headers().get(header::ACCEPT_LANGUAGE).cloned(), @@ -1336,7 +1342,14 @@ mod tests { Some(attested), ); let headers = routed.prebid_transport_headers(); - assert_eq!(headers.cookie(), Some(&HeaderValue::from_static("first=1"))); + assert_eq!( + headers.cookies(), + [ + HeaderValue::from_static("first=1"), + HeaderValue::from_static("second=2") + ], + "should retain every Cookie field in order" + ); assert_eq!( headers.user_agent().expect("should retain UA").as_bytes(), b"agent-\x80" diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index e48f5c2f03..6b9d86cd73 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -9,10 +9,13 @@ use std::borrow::Cow; use edgezero_core::app_config::{SecretField, SecretKind, SecretPathSegment}; -use error_stack::Report; +use error_stack::{Report, ResultExt as _}; +use serde::de::DeserializeOwned; use serde::{Deserialize, Deserializer, Serialize, Serializer}; use validator::{Validate, ValidationError, ValidationErrors}; +use crate::ec::identity::ProviderDescriptor; +use crate::ec::identity::provider::IdentityProviderConfig; use crate::ec::registry::PartnerRegistry; use crate::error::TrustedServerError; use crate::integrations::{ @@ -32,7 +35,15 @@ use crate::integrations::{ sourcepoint::SourcepointConfig, testlight::TestlightConfig, }; -use crate::settings::{AssetOriginAuth, IntegrationConfig, Settings}; +use crate::settings::{ + AssetOriginAuth, IdentitySettings, IntegrationConfig, ProviderMode, Settings, +}; + +/// Largest `[ec.identity].hold_budget_ms`. +const MAX_HOLD_BUDGET_MS: u64 = 1000; + +/// Largest `after_send_timeout_ms` of a provider. +const MAX_AFTER_SEND_TIMEOUT_MS: u64 = 10_000; const DEPLOY_VALIDATION_FIELD: &str = "trusted_server"; #[cfg(test)] @@ -258,6 +269,13 @@ pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report Result, Report> { + #[allow(unused_mut)] + let mut descriptors = + crate::ec::identity::generic_pull::generic_pull_descriptors(&settings.ec.partners)?; + #[cfg(test)] + descriptors.extend(validate_identity_integration::< + crate::ec::identity::examples::ExampleIdConfig, + >( + settings, + crate::ec::identity::examples::EXAMPLEID_INTEGRATION_ID, + )?); + Ok(descriptors) +} + +/// Parses `[integrations.]` as `T` and, when it is enabled, describes the +/// provider it enables. +/// +/// It never builds a provider and never reads a secret, so it runs at deploy +/// time, when secret fields still hold secret-store key names. +/// +/// # Errors +/// +/// Returns [`TrustedServerError::Configuration`] when the integration's +/// config does not parse or validate, or its descriptor cannot be built. +/// Returns `Ok(None)` when the integration is absent or disabled. +// Temporary: the first production identity integration adds the first caller. +#[allow(dead_code)] +pub(crate) fn validate_identity_integration( + settings: &Settings, + id: &str, +) -> Result, Report> { + let Some(config) = settings.integration_config::(id)? else { + return Ok(None); + }; + config + .build_descriptor() + .map(Some) + .change_context(TrustedServerError::Configuration { + message: format!("integrations.{id}: the identity provider cannot be described"), + }) +} + +fn identity_configuration_error(message: String) -> Report { + Report::new(TrustedServerError::Configuration { message }) +} + +/// Validates the numeric rules of `[ec.identity]`. +/// +/// Rules that need provider descriptors live in the identity catalog +/// validation instead. +/// +/// # Errors +/// +/// Returns [`TrustedServerError::Configuration`] naming the first key that is +/// out of range. +pub(crate) fn validate_identity_settings( + settings: &IdentitySettings, +) -> Result<(), Report> { + if settings.hold_budget_ms > MAX_HOLD_BUDGET_MS { + return Err(identity_configuration_error(format!( + "ec.identity.hold_budget_ms must be at most {MAX_HOLD_BUDGET_MS}, got {}", + settings.hold_budget_ms + ))); + } + if settings.max_concurrency == 0 { + return Err(identity_configuration_error( + "ec.identity.max_concurrency must be at least 1".to_owned(), + )); + } + for (id, provider) in &settings.providers { + if provider.refresh_after_sec == 0 { + return Err(identity_configuration_error(format!( + "ec.identity.providers.{id}.refresh_after_sec must be greater than 0" + ))); + } + if !(1..=MAX_AFTER_SEND_TIMEOUT_MS).contains(&provider.after_send_timeout_ms) { + return Err(identity_configuration_error(format!( + "ec.identity.providers.{id}.after_send_timeout_ms must be between 1 and {MAX_AFTER_SEND_TIMEOUT_MS}, got {}", + provider.after_send_timeout_ms + ))); + } + if provider.mode == ProviderMode::Hold + && provider + .timeout_ms + .is_some_and(|timeout| timeout > settings.hold_budget_ms) + { + return Err(identity_configuration_error(format!( + "ec.identity.providers.{id}.timeout_ms must not exceed hold_budget_ms ({})", + settings.hold_budget_ms + ))); + } + } + Ok(()) +} + +/// Rejects config keys that were removed instead of deprecated. +/// +/// # Errors +/// +/// Returns [`TrustedServerError::Configuration`] naming the replacement of the +/// first removed key that is set. +pub(crate) fn validate_removed_keys(settings: &Settings) -> Result<(), Report> { + if settings.ec.removed_pull_sync_concurrency.is_some() { + return Err(identity_configuration_error( + "ec.pull_sync_concurrency was removed; set [ec.identity].max_concurrency instead" + .to_owned(), + )); + } + for partner in &settings.ec.partners { + let id = crate::ec::identity::generic_pull::generic_pull_id(&partner.source_domain); + if partner.removed_pull_sync_ttl_sec.is_some() { + return Err(identity_configuration_error(format!( + "ec.partners `{}`: pull_sync_ttl_sec was removed; set refresh_after_sec in [ec.identity.providers.{id}] instead", + partner.source_domain + ))); + } + if partner.removed_pull_sync_rate_limit.is_some() { + return Err(identity_configuration_error(format!( + "ec.partners `{}`: pull_sync_rate_limit was removed; set max_calls_per_minute = ceil(N / 60) in [ec.identity.providers.{id}] instead", + partner.source_domain + ))); + } + } Ok(()) } @@ -518,8 +683,18 @@ mod tests { use super::*; use crate::auction_config_types::{NotificationConfig, ProviderConfig, RoutingMode}; + use crate::ec::identity::catalog::validate_identity_catalog; + use crate::ec::identity::descriptor::Resolves; + use crate::ec::identity::examples::ExampleCaptureProvider; + use crate::ec::identity::provider::{IdentityProvider as _, IdentityProviderConfig}; + use crate::ec::identity::resolution::IdentityProviderError; + use crate::ec::identity::runtime::IdentityRuntime; + use crate::ec::identity::test_support::ec_partner; + use crate::integrations::IntegrationRegistry; use crate::redacted::Redacted; - use crate::settings::{ProxyAssetRoute, S3SigV4AuthConfig, TrustedClientIpConfig}; + use crate::settings::{ + ProviderSettings, ProxyAssetRoute, RevokeScope, S3SigV4AuthConfig, TrustedClientIpConfig, + }; use crate::test_support::tests::crate_test_settings_str; use edgezero_core::app_config::AppConfigMeta; use edgezero_core::blob_envelope::BlobEnvelope; @@ -1616,4 +1791,275 @@ password = "production-admin-password-32-bytes" "validation error should mention invalid provider" ); } + + #[test] + fn removed_pull_sync_concurrency_is_rejected_with_its_replacement() { + let toml = crate_test_settings_str().replace("[ec]\n", "[ec]\npull_sync_concurrency = 3\n"); + let mut settings = Settings::from_toml(&toml).expect("should parse the removed key"); + settings.proxy.allowed_domains = vec!["*.example".to_string(), "*.example.com".to_string()]; + + let deploy = validate_settings_for_deploy(&settings).expect_err("should reject at deploy"); + assert!( + format!("{deploy:?}").contains("[ec.identity].max_concurrency"), + "should name the replacement at deploy, got: {deploy:?}" + ); + let runtime = + validate_settings_for_runtime(&settings).expect_err("should reject at runtime"); + assert!( + format!("{runtime:?}").contains("[ec.identity].max_concurrency"), + "should name the replacement at runtime, got: {runtime:?}" + ); + + let control = crate_test_settings_str() + .replace("[ec]\n", "[ec.identity]\nmax_concurrency = 3\n\n[ec]\n"); + let mut control = Settings::from_toml(&control).expect("should parse the replacement key"); + control.proxy.allowed_domains = vec!["*.example".to_string(), "*.example.com".to_string()]; + validate_settings_for_deploy(&control).expect("control: should pass at deploy"); + } + + #[test] + fn removed_partner_pull_keys_are_rejected_with_their_replacements() { + let base = crate_test_settings_str(); + let partner = |extra: &str| { + format!( + "{base}\n[[ec.partners]]\nname = \"Example SSP\"\nsource_domain = \"ssp.example.com\"\npull_sync_enabled = true\npull_sync_url = \"https://sync.ssp.example.com/pull\"\npull_sync_allowed_domains = [\"sync.ssp.example.com\"]\nts_pull_token = \"example-token-value-0123456789abcdef\"\n{extra}\n" + ) + }; + let parse = |toml: &str| { + let mut settings = Settings::from_toml(toml).expect("should parse the settings"); + settings.proxy.allowed_domains = + vec!["*.example".to_string(), "*.example.com".to_string()]; + settings + }; + + for (key, replacement) in [ + ("pull_sync_ttl_sec = 3600", "refresh_after_sec"), + ("pull_sync_rate_limit = 20", "max_calls_per_minute"), + ] { + let settings = parse(&partner(key)); + let deploy = + validate_settings_for_deploy(&settings).expect_err("should reject at deploy"); + assert!( + format!("{deploy:?}").contains(replacement), + "should name {replacement} at deploy, got: {deploy:?}" + ); + let runtime = + validate_settings_for_runtime(&settings).expect_err("should reject at runtime"); + assert!( + format!("{runtime:?}").contains(replacement), + "should name {replacement} at runtime, got: {runtime:?}" + ); + } + + let control = parse(&partner("")); + validate_settings_for_deploy(&control).expect("control: should pass at deploy"); + } + + #[derive(Debug, Deserialize, Validate)] + struct SecretedExampleConfig { + enabled: bool, + // Held so the config has a secret field, which no descriptor reads. + #[allow(dead_code)] + api_key: Redacted, + } + + impl IntegrationConfig for SecretedExampleConfig { + fn is_enabled(&self) -> bool { + self.enabled + } + } + + impl IdentityProviderConfig for SecretedExampleConfig { + fn build_descriptor(&self) -> Result> { + ProviderDescriptor::builder("secreted".parse()?) + .with_source("secreted.example.com".parse()?) + .with_resolves(Resolves::new().with_endpoint("https://api.example.com/fetch")) + .with_consent_policy(crate::ec::identity::test_support::resolver_policy()) + .build() + } + } + + fn exampleid_settings(partner_owner: Option<&str>) -> Settings { + let mut settings = valid_settings(); + settings.integrations.insert( + "exampleid".to_owned(), + serde_json::json!({ + "enabled": true, + "endpoint": "https://api.example.com/fetch", + "partner_id": 7, + "gvl_id": 1, + }), + ); + let mut partner = ec_partner( + "exampleid.example.com", + "https://sync.exampleid.example.com/pull", + "example-token-value-0123456789abcdef", + ); + partner.pull_sync_enabled = false; + partner.pull_sync_url = None; + partner.pull_sync_allowed_domains = Vec::new(); + partner.owner = partner_owner.map(ToOwned::to_owned); + settings.ec.partners = vec![partner]; + settings + } + + #[test] + fn config_without_ec_identity_is_accepted() { + let settings = valid_settings(); + + assert!( + settings.ec.identity.providers.is_empty(), + "should hold no provider tables" + ); + validate_settings_for_deploy(&settings).expect("should accept a config with no identity"); + } + + #[test] + fn identity_integration_descriptor_joins_deploy_validation() { + let settings = exampleid_settings(Some("exampleid")); + + let descriptors = + identity_descriptors_for_deploy(&settings).expect("should describe the deploy"); + assert_eq!( + descriptors + .iter() + .map(|descriptor| descriptor.id().as_str()) + .collect::>(), + ["exampleid"], + "should hold the integration's descriptor" + ); + validate_settings_for_deploy(&settings).expect("should accept a declared owner"); + + let unknown_owner = exampleid_settings(Some("missing")); + let error = validate_settings_for_deploy(&unknown_owner) + .expect_err("should reject an owner no enabled provider matches"); + assert!( + format!("{error}").contains("missing"), + "should name the owner: {error}" + ); + + let mut disabled = exampleid_settings(None); + disabled.integrations.insert( + "exampleid".to_owned(), + serde_json::json!({ "enabled": false }), + ); + assert!( + identity_descriptors_for_deploy(&disabled) + .expect("should describe the deploy") + .is_empty(), + "control: a disabled integration declares no descriptor" + ); + } + + #[test] + fn build_descriptor_runs_in_deploy_validation_without_secrets() { + let mut settings = valid_settings(); + settings.integrations.insert( + "secreted".to_owned(), + serde_json::json!({ "enabled": true, "api_key": "secret-store-key-name" }), + ); + + let descriptor = + validate_identity_integration::(&settings, "secreted") + .expect("should validate a config whose secret holds a key name") + .expect("should describe an enabled integration"); + assert_eq!( + descriptor.id().as_str(), + "secreted", + "should return the descriptor" + ); + + settings.integrations.insert( + "secreted".to_owned(), + serde_json::json!({ "enabled": false, "api_key": "secret-store-key-name" }), + ); + assert!( + validate_identity_integration::(&settings, "secreted") + .expect("should accept a disabled integration") + .is_none(), + "control: a disabled integration has no descriptor" + ); + assert!( + validate_identity_integration::(&settings, "absent") + .expect("should accept an absent integration") + .is_none(), + "control: an absent integration has no descriptor" + ); + } + + #[test] + fn capture_settings_on_a_provider_without_captures_are_rejected_at_deploy() { + let mut settings = exampleid_settings(None); + settings.ec.identity.providers.insert( + "exampleid".to_owned(), + ProviderSettings { + revoke_scope: Some(RevokeScope::Global), + ..ProviderSettings::default() + }, + ); + + let error = validate_settings_for_deploy(&settings) + .expect_err("should reject revoke_scope on a resolve-only provider"); + assert!( + format!("{error}").contains("exampleid"), + "should name the provider: {error}" + ); + + let capturing = ExampleCaptureProvider::new().expect("should build the provider"); + let partner = ec_partner( + "examplecapture.example.com", + "https://sync.examplecapture.example.com/pull", + "example-token-value-0123456789abcdef", + ); + let mut identity = IdentitySettings::default(); + identity.providers.insert( + "examplecapture".to_owned(), + ProviderSettings { + revoke_scope: Some(RevokeScope::Global), + sources: Some(vec!["examplecapture.example.com".to_owned()]), + ..ProviderSettings::default() + }, + ); + validate_identity_catalog(&[capturing.descriptor().clone()], &[partner], &identity) + .expect("control: the same key on a capturing provider should pass"); + } + + #[test] + fn capture_enabled_has_no_effect() { + let base = format!( + "{}\n[[ec.partners]]\nname = \"Example SSP\"\nsource_domain = \"ssp.example.com\"\npull_sync_enabled = true\npull_sync_url = \"https://sync.ssp.example.com/pull\"\npull_sync_allowed_domains = [\"sync.ssp.example.com\"]\nts_pull_token = \"example-token-value-0123456789abcdef\"\n", + crate_test_settings_str() + ); + let parse = |toml: &str| { + let mut settings = Settings::from_toml(toml).expect("should parse the settings"); + settings.proxy.allowed_domains = + vec!["*.example".to_string(), "*.example.com".to_string()]; + settings + }; + let plain = parse(&base); + let flagged = parse(&format!( + "{base}\n[ec.identity.providers.pull-ssp-example-com]\ncapture_enabled = true\n" + )); + + validate_settings_for_deploy(&flagged).expect("should accept the lenient key"); + let plain_runtime = IdentityRuntime::from_settings(&plain, &IntegrationRegistry::default()) + .expect("should build"); + let flagged_runtime = + IdentityRuntime::from_settings(&flagged, &IntegrationRegistry::default()) + .expect("should build"); + let id = "pull-ssp-example-com".parse().expect("should parse the ID"); + assert_eq!( + flagged_runtime.catalog().sources_of(&id), + plain_runtime.catalog().sources_of(&id), + "should change no provider source" + ); + assert!( + !flagged_runtime.catalog().is_capture_only(&id) + && flagged_runtime + .catalog() + .owner_of(&"ssp.example.com".parse().expect("should parse")) + .is_none(), + "should make no provider a capturer and give no source an owner" + ); + } } diff --git a/crates/trusted-server-core/src/config_payload.rs b/crates/trusted-server-core/src/config_payload.rs index c9690b01f5..85f53dba82 100644 --- a/crates/trusted-server-core/src/config_payload.rs +++ b/crates/trusted-server-core/src/config_payload.rs @@ -291,7 +291,7 @@ mod tests { "should preserve publisher domain" ); assert_eq!( - reconstructed.ec.pull_sync_concurrency, original.ec.pull_sync_concurrency, + reconstructed.ec.identity.max_concurrency, original.ec.identity.max_concurrency, "should preserve numeric fields" ); assert_eq!( diff --git a/crates/trusted-server-core/src/consent/gpp.rs b/crates/trusted-server-core/src/consent/gpp.rs index 59e39613ab..e84069446f 100644 --- a/crates/trusted-server-core/src/consent/gpp.rs +++ b/crates/trusted-server-core/src/consent/gpp.rs @@ -23,7 +23,7 @@ use error_stack::Report; -use super::types::{ConsentDecodeError, GppConsent, TcfConsent}; +use super::types::{ConsentDecodeError, GppConsent, TcfConsent, UsPrivacy}; /// Maximum length of a raw GPP string before parsing. /// @@ -71,7 +71,9 @@ pub fn decode_gpp_string(gpp_string: &str) -> Result Result Option { /// 21=UsNj, 22=UsTn, 23=UsMn. const US_SECTION_ID_RANGE: std::ops::RangeInclusive = 7..=23; -/// Extracts the `sale_opt_out` signal across all US sections in a parsed GPP -/// string. +/// Decodes every US section (IDs 7–23) in a parsed GPP string. /// -/// Iterates through section IDs looking for any in the US range (7–23), -/// decodes each US section, and aggregates the result conservatively: -/// -/// - `Some(true)` if any decodable US section says the user opted out of sale -/// - `Some(false)` if at least one decodable US section says they did not opt -/// out and none say they opted out -/// - `None` if no US section is present or no decodable US section yields a -/// usable `sale_opt_out` signal -fn decode_us_sale_opt_out(parsed: &iab_gpp::v1::GPPString) -> Option { - let mut result = None; - - for us_section_id in parsed +/// A section that fails to decode is logged and skipped, so one malformed +/// section never hides a valid one. The flag is `true` when any US section +/// failed to decode, so its opt-outs are unknown. +fn decode_us_sections(parsed: &iab_gpp::v1::GPPString) -> (Vec, bool) { + let mut failed = false; + let sections = parsed .section_ids() .filter(|id| US_SECTION_ID_RANGE.contains(&(**id as u16))) - { - match parsed.decode_section(*us_section_id) { - Ok(section) => match us_sale_opt_out_from_section(§ion) { - Some(true) => return Some(true), - Some(false) => result = Some(false), - None => {} + .filter_map( + |us_section_id| match parsed.decode_section(*us_section_id) { + Ok(section) => Some(section), + Err(e) => { + log::warn!("Failed to decode US GPP section {us_section_id}: {e}"); + failed = true; + None + } }, - Err(e) => { - log::warn!("Failed to decode US GPP section {us_section_id}: {e}"); - } + ) + .collect(); + (sections, failed) +} + +/// Aggregates one boolean signal across decoded US sections, conservatively: +/// +/// - `Some(true)` if any section reports the signal set +/// - `Some(false)` if at least one section reports it unset and none report +/// it set +/// - `None` if no section yields the signal +fn aggregate_us_signal( + sections: &[iab_gpp::sections::Section], + extract: fn(&iab_gpp::sections::Section) -> Option, +) -> Option { + let mut result = None; + for section in sections { + match extract(section) { + Some(true) => return Some(true), + Some(false) => result = Some(false), + None => {} } } - result } @@ -177,6 +200,110 @@ fn us_sale_opt_out_from_section(section: &iab_gpp::sections::Section) -> Option< Some(*sale_opt_out == OptOut::OptedOut) } +/// Reads the sharing opt-out, defined by the US National and California +/// sections only. +fn us_sharing_opt_out_from_section(section: &iab_gpp::sections::Section) -> Option { + use iab_gpp::sections::Section; + use iab_gpp::sections::us_common::OptOut; + + let sharing_opt_out = match section { + Section::UsNat(s) => match &s.core { + iab_gpp::sections::usnat::Core::V1(c) => &c.sharing_opt_out, + iab_gpp::sections::usnat::Core::V2(c) => &c.sharing_opt_out, + _ => return None, + }, + Section::UsCa(s) => &s.core.sharing_opt_out, + _ => return None, + }; + + Some(*sharing_opt_out == OptOut::OptedOut) +} + +/// Reads the targeted-advertising opt-out from the US National section and +/// every state section that defines it (all but California). +fn us_targeted_advertising_opt_out_from_section( + section: &iab_gpp::sections::Section, +) -> Option { + use iab_gpp::sections::Section; + use iab_gpp::sections::us_common::OptOut; + + // Keep this match in sync with new US-state variants added by `iab_gpp`. + let targeted_advertising_opt_out = match section { + Section::UsNat(s) => match &s.core { + iab_gpp::sections::usnat::Core::V1(c) => &c.targeted_advertising_opt_out, + iab_gpp::sections::usnat::Core::V2(c) => &c.targeted_advertising_opt_out, + _ => return None, + }, + Section::UsVa(s) => &s.core.targeted_advertising_opt_out, + Section::UsCo(s) => &s.core.targeted_advertising_opt_out, + Section::UsUt(s) => &s.core.targeted_advertising_opt_out, + Section::UsCt(s) => &s.core.targeted_advertising_opt_out, + Section::UsFl(s) => &s.core.targeted_advertising_opt_out, + Section::UsMt(s) => &s.core.targeted_advertising_opt_out, + Section::UsOr(s) => &s.core.targeted_advertising_opt_out, + Section::UsTx(s) => &s.core.targeted_advertising_opt_out, + Section::UsDe(s) => &s.core.targeted_advertising_opt_out, + Section::UsIa(s) => &s.core.targeted_advertising_opt_out, + Section::UsNe(s) => &s.core.targeted_advertising_opt_out, + Section::UsNh(s) => &s.core.targeted_advertising_opt_out, + Section::UsNj(s) => &s.core.targeted_advertising_opt_out, + Section::UsTn(s) => &s.core.targeted_advertising_opt_out, + Section::UsMn(s) => &s.core.targeted_advertising_opt_out, + _ => return None, + }; + + Some(*targeted_advertising_opt_out == OptOut::OptedOut) +} + +/// Reads the Global Privacy Control segment of the US National section. +fn us_gpc_from_section(section: &iab_gpp::sections::Section) -> Option { + match section { + iab_gpp::sections::Section::UsNat(s) => s.gpc, + _ => None, + } +} + +/// Reads whether the US National section marks a known child without consent. +fn us_known_child_from_section(section: &iab_gpp::sections::Section) -> Option { + use iab_gpp::sections::Section; + use iab_gpp::sections::us_common::Consent; + + let Section::UsNat(s) = section else { + return None; + }; + match &s.core { + iab_gpp::sections::usnat::Core::V1(c) => { + let child = &c.known_child_sensitive_data_consents; + Some(child.from_13_to_16 == Consent::NoConsent || child.under_13 == Consent::NoConsent) + } + iab_gpp::sections::usnat::Core::V2(c) => { + let child = &c.known_child_sensitive_data_consents; + Some( + child.process_sensitive_data_from_13_to_16 == Consent::NoConsent + || child.process_sensitive_data_under_13 == Consent::NoConsent + || child.process_personal_data_from_16_to_17 == Consent::NoConsent, + ) + } + _ => None, + } +} + +/// Reads GPP section 6, the US Privacy string, through the standalone +/// decoder. Returns `None` when absent, and `Err` when present but +/// undecodable. +fn decode_us_privacy_section(parsed: &iab_gpp::v1::GPPString) -> Result, ()> { + let Some(raw) = parsed.section(iab_gpp::sections::SectionId::UspV1) else { + return Ok(None); + }; + match super::us_privacy::decode_us_privacy(raw) { + Ok(us_privacy) => Ok(Some(us_privacy)), + Err(e) => { + log::warn!("GPP contains a US Privacy section but decoding failed: {e}"); + Err(()) + } + } +} + /// Parses a `__gpp_sid` cookie value into a vector of section IDs. /// /// The cookie is a comma-separated list of integer section IDs, e.g. `"2,6"`. @@ -210,6 +337,8 @@ pub fn parse_gpp_sid_cookie(raw: &str) -> Option> { #[cfg(test)] mod tests { use super::*; + use crate::consent::types::PrivacyFlag; + use crate::consent::us_privacy::decode_us_privacy; // A known-good GPP string with US Privacy section (section ID 6). // Header "DBABTA" encodes: version=1, section IDs=[6] (UspV1). @@ -461,4 +590,309 @@ mod tests { "should return None when no decodable US section yields sale_opt_out" ); } + + // Encodes a US core segment: a 6-bit version, then 2-bit fields. + fn us_core_segment(version: u8, fields: &[u8]) -> String { + const BASE64_URL: &[u8; 64] = + b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; + + let mut bits = format!("{version:06b}"); + for field in fields { + bits.push_str(&format!("{field:02b}")); + } + while bits.len() % 6 != 0 { + bits.push('0'); + } + bits.as_bytes() + .chunks(6) + .map(|chunk| { + let value = u8::from_str_radix( + core::str::from_utf8(chunk).expect("should encode segment bits as utf8"), + 2, + ) + .expect("should parse 6-bit chunk"); + char::from(BASE64_URL[value as usize]) + }) + .collect() + } + + // The US National core, version 1. `child` holds the two known-child + // consent fields. Opt-out values: 1 opted out, 2 did not. + fn us_nat_section(sale: u8, sharing: u8, targeted: u8, child: [u8; 2], gpc: bool) -> String { + let mut fields = vec![1, 1, 1, 1, 1, 1, sale, sharing, targeted]; + fields.extend([2; 12]); + fields.extend(child); + // Personal data consent, MSPA covered transaction and the two modes. + fields.extend([2, 2, 1, 1]); + let core = us_core_segment(1, &fields); + let gpc_segment = if gpc { "YA" } else { "QA" }; + format!("{core}.{gpc_segment}") + } + + // The California core: sale, then sharing. + fn us_ca_section(sale: u8, sharing: u8) -> String { + let mut fields = vec![1, 1, 1, sale, sharing]; + fields.extend([2; 9]); + fields.extend([2, 2]); + fields.extend([2, 2, 1, 1]); + format!("{}.QA", us_core_segment(1, &fields)) + } + + // The Virginia core: sale, then targeted advertising. + fn us_va_section(sale: u8, targeted: u8) -> String { + let mut fields = vec![1, 1, 1, sale, targeted]; + fields.extend([2; 8]); + fields.push(2); + fields.extend([2, 1, 1]); + us_core_segment(1, &fields) + } + + #[test] + fn decodes_the_sharing_opt_out_from_us_national_and_california() { + let nat = gpp_with_sections(&[(7, &us_nat_section(2, 1, 2, [2, 2], false))]); + let result = decode_gpp_string(&nat).expect("should decode US National"); + assert_eq!( + result.us_sharing_opt_out, + Some(true), + "should read the US National sharing opt-out" + ); + assert_eq!( + result.us_sale_opt_out, + Some(false), + "should keep the sale signal apart from the sharing signal" + ); + + let nat_control = gpp_with_sections(&[(7, &us_nat_section(2, 2, 2, [2, 2], false))]); + assert_eq!( + decode_gpp_string(&nat_control) + .expect("should decode US National control") + .us_sharing_opt_out, + Some(false), + "should read no sharing opt-out when it is not set" + ); + + let california = gpp_with_sections(&[(8, &us_ca_section(2, 1))]); + assert_eq!( + decode_gpp_string(&california) + .expect("should decode California") + .us_sharing_opt_out, + Some(true), + "should read the California sharing opt-out" + ); + let california_control = gpp_with_sections(&[(8, &us_ca_section(2, 2))]); + assert_eq!( + decode_gpp_string(&california_control) + .expect("should decode California control") + .us_sharing_opt_out, + Some(false), + "should read no California sharing opt-out when it is not set" + ); + + let virginia = gpp_with_sections(&[(9, &us_va_section(2, 2))]); + assert_eq!( + decode_gpp_string(&virginia) + .expect("should decode Virginia") + .us_sharing_opt_out, + None, + "should leave sharing unset for a section that does not define it" + ); + } + + #[test] + fn decodes_the_targeted_advertising_opt_out_from_sections_that_define_it() { + let nat = gpp_with_sections(&[(7, &us_nat_section(2, 2, 1, [2, 2], false))]); + let result = decode_gpp_string(&nat).expect("should decode US National"); + assert_eq!( + result.us_targeted_advertising_opt_out, + Some(true), + "should read the US National targeted-advertising opt-out" + ); + assert_eq!( + result.us_sharing_opt_out, + Some(false), + "should keep the sharing signal apart from the targeted signal" + ); + + let virginia = gpp_with_sections(&[(9, &us_va_section(2, 1))]); + assert_eq!( + decode_gpp_string(&virginia) + .expect("should decode Virginia") + .us_targeted_advertising_opt_out, + Some(true), + "should read the Virginia targeted-advertising opt-out" + ); + let virginia_control = gpp_with_sections(&[(9, &us_va_section(2, 2))]); + assert_eq!( + decode_gpp_string(&virginia_control) + .expect("should decode Virginia control") + .us_targeted_advertising_opt_out, + Some(false), + "should read no targeted-advertising opt-out when it is not set" + ); + + let california = gpp_with_sections(&[(8, &us_ca_section(2, 2))]); + assert_eq!( + decode_gpp_string(&california) + .expect("should decode California") + .us_targeted_advertising_opt_out, + None, + "should leave targeted advertising unset for a section that does not define it" + ); + } + + #[test] + fn any_section_opt_out_wins_for_the_new_opt_outs() { + let gpp = gpp_with_sections(&[ + (7, &us_nat_section(2, 2, 2, [2, 2], false)), + (9, &us_va_section(2, 1)), + ]); + let result = decode_gpp_string(&gpp).expect("should decode both sections"); + assert_eq!( + result.us_targeted_advertising_opt_out, + Some(true), + "should treat any decodable targeted opt-out as authoritative" + ); + } + + #[test] + fn decodes_the_us_national_gpc_field() { + let set = gpp_with_sections(&[(7, &us_nat_section(2, 2, 2, [2, 2], true))]); + assert_eq!( + decode_gpp_string(&set) + .expect("should decode US National with GPC") + .us_gpc, + Some(true), + "should read the GPC field when it is set" + ); + let unset = gpp_with_sections(&[(7, &us_nat_section(2, 2, 2, [2, 2], false))]); + assert_eq!( + decode_gpp_string(&unset) + .expect("should decode US National without GPC") + .us_gpc, + Some(false), + "should read GPC false when the segment says so" + ); + assert_eq!( + decode_gpp_string(GPP_USP_ONLY) + .expect("should decode USP only") + .us_gpc, + None, + "should leave GPC unset without a US National section" + ); + } + + #[test] + fn decodes_a_known_child_without_consent() { + // Known-child fields: 1 means no consent, 2 consent, 0 not applicable. + let no_consent = gpp_with_sections(&[(7, &us_nat_section(2, 2, 2, [2, 1], false))]); + assert_eq!( + decode_gpp_string(&no_consent) + .expect("should decode known child") + .us_known_child, + Some(true), + "should flag a known child without consent" + ); + let consent = gpp_with_sections(&[(7, &us_nat_section(2, 2, 2, [2, 2], false))]); + assert_eq!( + decode_gpp_string(&consent) + .expect("should decode consenting child") + .us_known_child, + Some(false), + "should not flag a known child who has consent" + ); + let not_applicable = gpp_with_sections(&[(7, &us_nat_section(2, 2, 2, [0, 0], false))]); + assert_eq!( + decode_gpp_string(¬_applicable) + .expect("should decode not-applicable child") + .us_known_child, + Some(false), + "should not flag a section that marks no known child" + ); + assert_eq!( + decode_gpp_string(GPP_USP_ONLY) + .expect("should decode USP only") + .us_known_child, + None, + "should leave known child unset without a US National section" + ); + } + + #[test] + fn decodes_section_6_through_the_us_privacy_decoder() { + let result = decode_gpp_string(GPP_USP_ONLY).expect("should decode USP only"); + assert_eq!( + result.us_privacy_section, + Some(decode_us_privacy("1YNN").expect("should decode 1YNN")), + "should read section 6 as a US Privacy string" + ); + let opted_out = decode_gpp_string(&gpp_with_sections(&[(6, "1YYN")])) + .expect("should decode an opted-out section 6"); + assert_eq!( + opted_out + .us_privacy_section + .expect("should carry section 6") + .opt_out_sale, + PrivacyFlag::Yes, + "should read the section 6 sale opt-out" + ); + assert_eq!( + opted_out.us_sale_opt_out, None, + "should keep us_sale_opt_out to the US state sections" + ); + let no_section = decode_gpp_string(&gpp_with_sections(&[( + 7, + &us_nat_section(2, 2, 2, [2, 2], false), + )])) + .expect("should decode US National"); + assert!( + no_section.us_privacy_section.is_none(), + "should leave section 6 unset when absent" + ); + } + + #[test] + fn header_only_gpp_decodes_with_no_section_fields() { + let result = decode_gpp_string("DBAA").expect("should decode a header-only GPP string"); + assert!(result.section_ids.is_empty(), "should list no section IDs"); + assert_eq!(result.us_sale_opt_out, None, "should leave sale unset"); + assert_eq!( + result.us_sharing_opt_out, None, + "should leave sharing unset" + ); + assert_eq!( + result.us_targeted_advertising_opt_out, None, + "should leave targeted advertising unset" + ); + assert_eq!(result.us_gpc, None, "should leave GPC unset"); + assert_eq!( + result.us_known_child, None, + "should leave known child unset" + ); + assert!( + result.us_privacy_section.is_none(), + "should leave section 6 unset" + ); + assert!(result.eu_tcf.is_none(), "should carry no TCF section"); + } + + #[test] + fn flags_a_us_section_or_section_6_that_fails_to_decode() { + let corrupt_us = decode_gpp_string(&gpp_with_sections(&[(7, "not-a-valid-usnat-section")])) + .expect("should decode the header"); + assert!( + corrupt_us.us_section_decode_failed, + "should flag an undecodable US section" + ); + let corrupt_usp = decode_gpp_string(&gpp_with_sections(&[(6, "garbage")])) + .expect("should decode the header"); + assert!( + corrupt_usp.us_section_decode_failed, + "should flag an undecodable section 6" + ); + let valid = decode_gpp_string(GPP_USP_ONLY).expect("should decode USP only"); + assert!( + !valid.us_section_decode_failed, + "should not flag a decodable string" + ); + } } diff --git a/crates/trusted-server-core/src/consent/mod.rs b/crates/trusted-server-core/src/consent/mod.rs index 76aa6ad859..e78d3f4b12 100644 --- a/crates/trusted-server-core/src/consent/mod.rs +++ b/crates/trusted-server-core/src/consent/mod.rs @@ -280,13 +280,24 @@ fn has_eu_tcf_signal(raw_tc_present: bool, gpp_section_ids: Option<&[u16]>) -> b /// Returns the effective decoded TCF consent for enforcement decisions. #[must_use] -fn effective_tcf(ctx: &ConsentContext) -> Option<&types::TcfConsent> { +pub(crate) fn effective_tcf(ctx: &ConsentContext) -> Option<&types::TcfConsent> { ctx.tcf.as_ref().or_else(|| { let g = ctx.gpp.as_ref()?; g.eu_tcf.as_ref() }) } +/// Whether TCF applies to this request: `gdpr_applies`, or a jurisdiction of +/// GDPR or unknown. +#[must_use] +pub(crate) fn tcf_applies(ctx: &ConsentContext) -> bool { + ctx.gdpr_applies + || matches!( + ctx.jurisdiction, + jurisdiction::Jurisdiction::Gdpr | jurisdiction::Jurisdiction::Unknown + ) +} + /// Returns whether a server-side auction may be dispatched for this request. /// /// Fails closed for GDPR-relevant traffic: when an EU TCF signal is present @@ -296,12 +307,7 @@ fn effective_tcf(ctx: &ConsentContext) -> Option<&types::TcfConsent> { /// non-GDPR jurisdiction with no EU TCF signal are freely allowed. #[must_use] pub fn consent_allows_server_side_auction(ctx: &ConsentContext) -> bool { - let requires_tcf_purpose1 = ctx.gdpr_applies - || matches!( - ctx.jurisdiction, - jurisdiction::Jurisdiction::Gdpr | jurisdiction::Jurisdiction::Unknown - ); - if !requires_tcf_purpose1 { + if !tcf_applies(ctx) { return true; } effective_tcf(ctx).is_some_and(|tcf| tcf.has_purpose_consent(1)) @@ -780,6 +786,12 @@ mod tests { section_ids: vec![2], eu_tcf: Some(make_tcf(gpp_last_updated_ds, gpp_allows_eids)), us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), ..ConsentContext::default() } @@ -792,6 +804,12 @@ mod tests { section_ids: vec![2], eu_tcf: Some(make_tcf(0, allows_eids)), us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }; let cases = [ ( @@ -952,6 +970,12 @@ mod tests { section_ids: vec![2], eu_tcf: Some(TcfBuilder::new().with_storage(true).build()), us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), ..ConsentContext::default() }; @@ -1113,6 +1137,12 @@ mod tests { section_ids: vec![2], eu_tcf: Some(make_tcf(0, true)), us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), ..ConsentContext::default() }; @@ -1226,6 +1256,12 @@ mod tests { section_ids: vec![2], eu_tcf: Some(make_tcf_with_storage(true)), us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), gdpr_applies: true, ..ConsentContext::default() @@ -1441,6 +1477,12 @@ mod tests { section_ids: vec![7], eu_tcf: None, us_sale_opt_out: Some(false), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), ..ConsentContext::default() }; @@ -1459,6 +1501,12 @@ mod tests { section_ids: vec![7], eu_tcf: None, us_sale_opt_out: Some(true), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), ..ConsentContext::default() }; @@ -1482,6 +1530,12 @@ mod tests { section_ids: vec![7], eu_tcf: None, us_sale_opt_out: Some(false), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), ..ConsentContext::default() }; @@ -1501,6 +1555,12 @@ mod tests { section_ids: vec![7], eu_tcf: None, us_sale_opt_out: Some(true), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), ..ConsentContext::default() }; @@ -1523,6 +1583,12 @@ mod tests { section_ids: vec![7], eu_tcf: None, us_sale_opt_out: Some(false), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), us_privacy: Some(UsPrivacy { version: 1, @@ -1551,6 +1617,12 @@ mod tests { section_ids: vec![2], eu_tcf: None, us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, }), us_privacy: Some(UsPrivacy { version: 1, diff --git a/crates/trusted-server-core/src/consent/types.rs b/crates/trusted-server-core/src/consent/types.rs index 7d1b1902aa..f9e0912e37 100644 --- a/crates/trusted-server-core/src/consent/types.rs +++ b/crates/trusted-server-core/src/consent/types.rs @@ -105,7 +105,7 @@ impl fmt::Display for RawConsentSignals { /// /// Built from [`RawConsentSignals`] by the decoding pipeline in /// [`super::build_consent_context`]. -#[derive(Debug, Clone, Default)] +#[derive(Clone, Default)] pub struct ConsentContext { /// Raw TC String from `euconsent-v2` cookie, passed as-is in `user.consent`. pub raw_tc_string: Option, @@ -148,6 +148,33 @@ pub struct ConsentContext { pub source: ConsentSource, } +/// Prints `[redacted]` in place of every raw consent string, so a stray +/// `{:?}` never writes a consent string to a log. +impl fmt::Debug for ConsentContext { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + fn redacted(raw: &Option) -> Option<&'static str> { + raw.as_ref().map(|_| "[redacted]") + } + + formatter + .debug_struct("ConsentContext") + .field("raw_tc_string", &redacted(&self.raw_tc_string)) + .field("raw_gpp_string", &redacted(&self.raw_gpp_string)) + .field("gpp_section_ids", &self.gpp_section_ids) + .field("raw_us_privacy", &redacted(&self.raw_us_privacy)) + .field("raw_ac_string", &redacted(&self.raw_ac_string)) + .field("gdpr_applies", &self.gdpr_applies) + .field("tcf", &self.tcf) + .field("gpp", &self.gpp) + .field("us_privacy", &self.us_privacy) + .field("expired", &self.expired) + .field("gpc", &self.gpc) + .field("jurisdiction", &self.jurisdiction) + .field("source", &self.source) + .finish() + } +} + impl ConsentContext { /// Returns `true` when no consent signals are present. #[must_use] @@ -308,6 +335,28 @@ pub struct GppConsent { /// - `Some(false)` — a US section is present and user did not opt out /// - `None` — no US section exists in the GPP string pub us_sale_opt_out: Option, + /// Whether the user opted out of sharing for cross-context behavioural + /// advertising via a US GPP section that defines it (US National, + /// California). `None` when no such section is present or decodable. + pub us_sharing_opt_out: Option, + /// Whether the user opted out of targeted advertising via a US GPP + /// section that defines it (US National and most state sections). `None` + /// when no such section is present or decodable. + pub us_targeted_advertising_opt_out: Option, + /// The Global Privacy Control field of the US National section. `None` + /// when that section or its GPC segment is absent. + pub us_gpc: Option, + /// Whether the US National section marks a known child without consent. + /// `Some(false)` when the section is present and marks none, `None` when + /// it is absent or undecodable. + pub us_known_child: Option, + /// The US Privacy string carried as GPP section 6, read through the + /// standalone US Privacy decoder. `None` when absent or undecodable. + pub us_privacy_section: Option, + /// Whether a US section (IDs 7–23) or section 6 was present but failed to + /// decode, so its opt-outs are unknown. Strict identity policies fail + /// closed on it. + pub us_section_decode_failed: bool, } // --------------------------------------------------------------------------- @@ -414,6 +463,45 @@ impl core::error::Error for ConsentDecodeError {} mod tests { use super::*; + #[test] + fn consent_context_debug_redacts_raw_strings() { + let context = ConsentContext { + raw_tc_string: Some("tc-string-example".to_owned()), + raw_gpp_string: Some("gpp-string-example".to_owned()), + raw_us_privacy: Some("usp-string-example".to_owned()), + raw_ac_string: Some("ac-string-example".to_owned()), + gdpr_applies: true, + gpc: true, + ..ConsentContext::default() + }; + + let rendered = format!("{context:?}"); + + for raw in [ + "tc-string-example", + "gpp-string-example", + "usp-string-example", + "ac-string-example", + ] { + assert!( + !rendered.contains(raw), + "should not print the raw string {raw}" + ); + } + assert!( + rendered.contains("[redacted]"), + "should mark the raw strings as redacted" + ); + assert!( + rendered.contains("gdpr_applies: true") && rendered.contains("gpc: true"), + "should still print the non-secret fields" + ); + assert!( + format!("{:?}", ConsentContext::default()).contains("raw_tc_string: None"), + "should print an absent raw string as None" + ); + } + #[test] fn empty_signals() { let signals = RawConsentSignals::default(); diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index 1fd3c2382c..da9ea740e8 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -1,11 +1,15 @@ use http::header::HeaderName; pub const COOKIE_TS_EC: &str = "ts-ec"; -/// Short-lived signed proof that the current EC row has every pull-partner UID. -pub const COOKIE_TS_EC_PULL_COMPLETE: &str = "ts-ec-pull-complete"; +/// Marks a global revoke: holds the consent fingerprint the revoke saw, and +/// stops EC minting while the request's consent is unchanged. +pub const COOKIE_TS_EC_REVOKED: &str = "ts-ec-revoked"; /// Cookie written by the Trusted Server JS SDK containing a standard-base64-encoded /// JSON array of Extended User IDs (`[{ source, uids }]`) from identity providers. pub const COOKIE_TS_EIDS: &str = "ts-eids"; +/// Client-only mirror of the servable server-side EIDs, written by the shim +/// and never read by the server (spec 6.2). +pub const COOKIE_TS_EIDS_S: &str = "__Host-ts-eids-s"; pub const COOKIE_TS_TESTER: &str = "ts-tester"; pub const COOKIE_SHAREDID: &str = "sharedId"; diff --git a/crates/trusted-server-core/src/cookies.rs b/crates/trusted-server-core/src/cookies.rs index 3fe6719ef7..43105eb6ae 100644 --- a/crates/trusted-server-core/src/cookies.rs +++ b/crates/trusted-server-core/src/cookies.rs @@ -11,7 +11,10 @@ use error_stack::{Report, ResultExt as _}; use http::Request; use http::header; -use crate::constants::{COOKIE_EUCONSENT_V2, COOKIE_GPP, COOKIE_GPP_SID, COOKIE_US_PRIVACY}; +use crate::constants::{ + COOKIE_EUCONSENT_V2, COOKIE_GPP, COOKIE_GPP_SID, COOKIE_TS_EC, COOKIE_TS_EIDS, + COOKIE_TS_EIDS_S, COOKIE_US_PRIVACY, +}; use crate::error::TrustedServerError; /// Cookie names carrying privacy consent signals. @@ -26,6 +29,23 @@ pub const CONSENT_COOKIE_NAMES: &[&str] = &[ COOKIE_US_PRIVACY, ]; +/// Trusted Server identity cookies that never reach a third party. +/// +/// Stripped from Prebid Server requests and never added to proxy or provider +/// requests. +pub const TS_IDENTITY_COOKIE_NAMES: &[&str] = &[COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_TS_EIDS_S]; + +/// Identity cookies removed from publisher-origin requests by default. +/// +/// `ts-ec` and `ts-eids` still reach the origin, as they did before the +/// `ts-eids-s` mirror existed, so the origin's `Cookie` header stays at its +/// earlier size. See [`TS_ORIGIN_STRIPPED_COOKIE_NAMES_WITH_EIDS`]. +pub const TS_ORIGIN_STRIPPED_COOKIE_NAMES: &[&str] = &[COOKIE_TS_EIDS_S]; + +/// Identity cookies removed from publisher-origin requests when the operator +/// sets `ec.strip_eids_cookie_from_origin`: `ts-eids` as well as the mirror. +pub const TS_ORIGIN_STRIPPED_COOKIE_NAMES_WITH_EIDS: &[&str] = &[COOKIE_TS_EIDS, COOKIE_TS_EIDS_S]; + /// Parses a cookie string into a [`CookieJar`]. /// /// Returns an empty jar if the cookie string is unparseable. @@ -104,6 +124,52 @@ pub fn strip_cookies(cookie_header: &str, cookie_names: &[&str]) -> String { .join("; ") } +/// Strips cookies whose name is exactly one of `names` from a `Cookie` header +/// value. +/// +/// Returns an empty string when nothing is left. +#[must_use] +pub fn strip_cookie_pairs(cookie_header: &str, names: &[&str]) -> String { + cookie_header + .split(';') + .map(str::trim) + .filter(|pair| !pair.is_empty()) + .filter(|pair| { + let name = pair.split('=').next().unwrap_or_default().trim(); + !names.contains(&name) + }) + .collect::>() + .join("; ") +} + +/// Strips matching cookies from every `Cookie` field in `headers`. +/// +/// All `Cookie` fields are read, matching pairs are removed, and the rest is +/// written back as one field. A field that is not valid UTF-8 is dropped +/// rather than forwarded, because it cannot be checked. The header is removed +/// when nothing is left. +pub fn strip_cookie_headers(headers: &mut http::HeaderMap, names: &[&str]) { + let retained = headers + .get_all(header::COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .map(|value| strip_cookie_pairs(value, names)) + .filter(|value| !value.is_empty()) + .collect::>() + .join("; "); + + headers.remove(header::COOKIE); + if retained.is_empty() { + return; + } + match http::HeaderValue::from_str(&retained) { + Ok(value) => { + headers.insert(header::COOKIE, value); + } + Err(_) => log::debug!("Dropping Cookie header that is invalid after stripping"), + } +} + /// Copies the `Cookie` header from one request to another, optionally /// stripping consent cookies. /// @@ -451,4 +517,70 @@ mod tests { let stripped = strip_cookies(header, CONSENT_COOKIE_NAMES); assert_eq!(stripped, "session=abc=123=def"); } + + #[test] + fn strip_cookie_pairs_removes_exact_names_only() { + let header = + "ts-ec=abc; ts-eids=W10; __Host-ts-eids-s=eyJ2IjoxfQ; other=1; ts-eids-extra=keep"; + + let stripped = strip_cookie_pairs(header, TS_ORIGIN_STRIPPED_COOKIE_NAMES_WITH_EIDS); + + assert_eq!( + stripped, "ts-ec=abc; other=1; ts-eids-extra=keep", + "should drop the ID cookies and keep ts-ec and a cookie that only shares a prefix" + ); + + assert_eq!( + strip_cookie_pairs(header, TS_ORIGIN_STRIPPED_COOKIE_NAMES), + "ts-ec=abc; ts-eids=W10; other=1; ts-eids-extra=keep", + "should drop only the mirror by default" + ); + } + + #[test] + fn strip_cookie_headers_joins_every_cookie_field_and_drops_non_utf8() { + let mut headers = http::HeaderMap::new(); + headers.append( + header::COOKIE, + http::HeaderValue::from_static("ts-ec=abc; ts-eids=W10"), + ); + headers.append( + header::COOKIE, + http::HeaderValue::from_bytes(b"bad=\xff").expect("should build an opaque header"), + ); + headers.append(header::COOKIE, http::HeaderValue::from_static("other=1")); + + strip_cookie_headers(&mut headers, TS_IDENTITY_COOKIE_NAMES); + + let values: Vec<_> = headers.get_all(header::COOKIE).iter().collect(); + assert_eq!(values.len(), 1, "should collapse into one Cookie header"); + assert_eq!( + values[0], "other=1", + "should keep only the unrelated cookie" + ); + } + + #[test] + fn strip_cookie_headers_removes_header_when_nothing_is_left() { + let mut headers = http::HeaderMap::new(); + headers.insert( + header::COOKIE, + http::HeaderValue::from_static("ts-ec=abc; __Host-ts-eids-s=eyJ2IjoxfQ"), + ); + headers.insert( + header::USER_AGENT, + http::HeaderValue::from_static("ExampleBrowser/1.0"), + ); + + strip_cookie_headers(&mut headers, TS_IDENTITY_COOKIE_NAMES); + + assert!( + headers.get(header::COOKIE).is_none(), + "should remove the empty Cookie header" + ); + assert!( + headers.get(header::USER_AGENT).is_some(), + "should leave other headers alone" + ); + } } diff --git a/crates/trusted-server-core/src/cookies/template_cache_policy.rs b/crates/trusted-server-core/src/cookies/template_cache_policy.rs index 61c3b9139c..7c42a8d6f1 100644 --- a/crates/trusted-server-core/src/cookies/template_cache_policy.rs +++ b/crates/trusted-server-core/src/cookies/template_cache_policy.rs @@ -2,7 +2,7 @@ use std::collections::{HashMap, HashSet}; -use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EC, COOKIE_TS_EIDS}; +use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_TS_EIDS_S}; use crate::platform::TemplateCookieValue; /// Whether the request can share a template and its explicit cookie dimensions. @@ -32,7 +32,13 @@ pub(crate) fn validate_cookie_names( return Err(format!("{field} contains invalid cookie name `{name}`")); } if field == "template_cache_key_cookies" - && [COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_SHAREDID].contains(&name.as_str()) + && [ + COOKIE_TS_EC, + COOKIE_TS_EIDS, + COOKIE_TS_EIDS_S, + COOKIE_SHAREDID, + ] + .contains(&name.as_str()) { return Err(format!( "{field} must not key on Trusted Server identity cookie `{name}`" @@ -246,6 +252,20 @@ mod tests { } } + #[test] + fn template_cache_key_rejects_the_mirror_cookie() { + let error = validate_cookie_names(&names(&["__Host-ts-eids-s"]), &[]) + .expect_err("should reject the mirror cookie as a cache key"); + assert!( + error.contains("must not key on Trusted Server identity cookie"), + "should explain the rejection, got {error}" + ); + assert!( + validate_cookie_names(&names(&["session"]), &[]).is_ok(), + "should still accept an unrelated cookie" + ); + } + #[test] fn template_cookie_policy_allows_only_independent_unlisted_duplicates() { for fields in [ diff --git a/crates/trusted-server-core/src/ec/admin.rs b/crates/trusted-server-core/src/ec/admin.rs index 18b5563307..c8b5df52f3 100644 --- a/crates/trusted-server-core/src/ec/admin.rs +++ b/crates/trusted-server-core/src/ec/admin.rs @@ -6,6 +6,9 @@ //! attach, so operators can debug KV-to-auction propagation without KV //! console access. //! +//! Also serves `DELETE /_ts/admin/ec/{id}/providers/{provider}/opt-out`, +//! which clears a provider's sticky opt-out. +//! //! Also serves `GET /_ts/admin/eids`, which echoes the request's `ts-eids` //! and `sharedId` cookies with an ingestion preview — the client-side half //! of EID propagation that is never stored server-side. @@ -26,13 +29,26 @@ use serde_json::Value as JsonValue; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt as _}; +use crate::consent::ConsentContext; +use crate::consent::jurisdiction::Jurisdiction; use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; use crate::cookies::extract_cookie_value; use crate::error::TrustedServerError; use crate::openrtb::Eid; +use super::checked_current_timestamp; use super::eids::{resolve_partner_ids, to_eids}; use super::generation::is_valid_ec_id; +use super::identity::IdentityProviderId; +use super::identity::IdentityRuntime; +use super::identity::UnixTime; +use super::identity::catalog::ProviderCatalog; +use super::identity::input::ThreadRandom; +use super::identity::mutation::{ApplyContext, IdentityMutation, ProviderStateChange}; +use super::identity::policy::PolicySignals; +use super::identity::removal::{RemovalTiming, RemovalWriteOutcome, write_removals}; +use super::identity::request::opted_out; +use super::identity::servable::ServeContext; use super::kv::KvIdentityGraph; use super::kv_backend::EcKvLookup; use super::kv_types::{KvEntry, KvMetadata}; @@ -276,6 +292,7 @@ struct SkippedPartnerId { pub fn handle_admin_ec_lookup( kv: Option<&KvIdentityGraph>, registry: &PartnerRegistry, + runtime: Option<&IdentityRuntime>, req: &Request, ) -> Result, Report> { let Some(kv) = kv else { @@ -297,7 +314,7 @@ pub fn handle_admin_ec_lookup( }; log::info!("Admin EC lookup: returning entry for '{}'", log_id(&ec_id)); - let payload = build_lookup_response(registry, kv.store_name(), ec_id, &lookup); + let payload = build_lookup_response(registry, runtime, kv.store_name(), ec_id, &lookup); let body = serde_json::to_string(&payload).change_context(TrustedServerError::Configuration { message: "failed to serialize admin EC lookup response".to_owned(), @@ -305,6 +322,104 @@ pub fn handle_admin_ec_lookup( Ok(json_response(StatusCode::OK, body)) } +/// Handles `DELETE /_ts/admin/ec/{id}/providers/{provider}/opt-out`. +/// +/// Clears the provider's sticky opt-out `ps.o` by writing `oc = now`, which +/// also empties `ps.ov` (spec 8). It runs the removal write's CAS loop and +/// never reads the request's cookies or consent. Responds: +/// +/// - `200 OK` with `{"cleared": true}` when the clear was written; +/// - `400 Bad Request` for an invalid EC ID or provider ID; +/// - `404 Not Found` when the row is missing or the provider holds no opt-out; +/// - `405 Method Not Allowed` for any method but `DELETE`; +/// - `501 Not Implemented` when no EC identity graph is configured; +/// - `503 Service Unavailable` when the write did not land. +#[must_use] +pub fn handle_admin_opt_out_clear( + kv: Option<&KvIdentityGraph>, + req: &Request, +) -> Response { + if req.method() != Method::DELETE { + let mut response = json_error(StatusCode::METHOD_NOT_ALLOWED, "method not allowed"); + response + .headers_mut() + .insert(header::ALLOW, HeaderValue::from_static("DELETE")); + return response; + } + let Some(kv) = kv else { + return admin_ec_lookup_not_supported(); + }; + let Some((ec_id, provider)) = opt_out_path_parts(req.uri().path()) else { + return json_error(StatusCode::NOT_FOUND, "admin diagnostic route not found"); + }; + if !is_valid_ec_id(ec_id) { + return json_error( + StatusCode::BAD_REQUEST, + "invalid EC ID format (expected {64hex}.{6alnum})", + ); + } + let Ok(provider) = provider.parse::() else { + return json_error(StatusCode::BAD_REQUEST, "invalid identity provider ID"); + }; + + let Some(now) = checked_current_timestamp() else { + return json_error( + StatusCode::SERVICE_UNAVAILABLE, + "the clock is unavailable, so the opt-out was not cleared", + ); + }; + let mut clear = IdentityMutation::default(); + clear.merge_state( + provider.clone(), + ProviderStateChange { + opt_out_clear: Some(now), + ..ProviderStateChange::default() + }, + ); + let catalog = ProviderCatalog::empty(); + let context = ApplyContext { + now, + random: &ThreadRandom, + catalog: &catalog, + }; + match write_removals(kv, ec_id, &clear, &context, RemovalTiming::BeforeSend) { + RemovalWriteOutcome::Written => { + log::info!( + "Admin cleared the opt-out of identity provider '{provider}' for '{}'", + log_id(ec_id) + ); + json_response(StatusCode::OK, r#"{"cleared":true}"#.to_owned()) + } + RemovalWriteOutcome::NothingToWrite => json_error( + StatusCode::NOT_FOUND, + "no EC entry, or the provider holds no opt-out to clear", + ), + RemovalWriteOutcome::Failed => json_error( + StatusCode::SERVICE_UNAVAILABLE, + "the opt-out clear could not be written; retry", + ), + } +} + +/// Splits `/_ts/admin/ec/{id}/providers/{provider}/opt-out` into its EC ID +/// and provider ID, or `None` for any other shape. +fn opt_out_path_parts(path: &str) -> Option<(&str, &str)> { + let mut segments = path + .strip_prefix(ADMIN_EC_PATH)? + .strip_prefix('/')? + .split('/'); + let ec_id = segments.next()?; + let (Some("providers"), Some(provider), Some("opt-out"), None) = ( + segments.next(), + segments.next(), + segments.next(), + segments.next(), + ) else { + return None; + }; + Some((ec_id, provider)) +} + /// Returns the portable response used when an adapter has no EC KV backend. #[must_use] pub fn admin_ec_lookup_not_supported() -> Response { @@ -368,12 +483,26 @@ fn requested_ec_id(req: &Request) -> Result.st` with a marker. Provider state is never +/// returned by the admin lookup (spec 6.1, 9.3). +fn redact_provider_state(entry: &mut JsonValue) { + let Some(states) = entry.get_mut("ps").and_then(JsonValue::as_object_mut) else { + return; + }; + for state in states.values_mut() { + if let Some(st) = state.get_mut("st") { + *st = JsonValue::String("[redacted]".to_owned()); + } + } +} + /// Builds the success payload from a raw KV lookup. /// /// Parse failures are reported in the payload rather than propagated, so /// corrupt entries remain inspectable. fn build_lookup_response( registry: &PartnerRegistry, + runtime: Option<&IdentityRuntime>, store_name: &str, ec_id: String, lookup: &EcKvLookup, @@ -393,14 +522,26 @@ fn build_lookup_response( match serde_json::from_slice::(&lookup.body) { Ok(mut entry_json) => { + redact_provider_state(&mut entry_json); add_iso_timestamp_companions(&mut entry_json); payload.entry = Some(entry_json); match serde_json::from_slice::(&lookup.body) { - Ok(entry) => { + Ok(mut entry) => { payload.tombstone = Some(!entry.consent.ok); - match entry.validate() { - Ok(()) => payload.auction = Some(build_auction_view(registry, &entry)), + // Upgrade first so the auction view matches what the read + // path serves, with invalid records dropped. + let upgraded = entry + .upgrade_for_read() + .and_then(|dropped| entry.validate().map(|()| dropped)); + match upgraded { + Ok(dropped) => { + payload.auction = Some(build_auction_view(registry, &entry, runtime)); + if !dropped.is_empty() { + payload.entry_error = + Some(format!("dropped on read: {}", dropped.join("; "))); + } + } Err(message) => { payload.entry_error = Some(format!( "entry failed validation (auction reads fail closed \ @@ -481,19 +622,57 @@ fn iso_timestamp(unix_seconds: u64) -> Option { /// Derives the auction EID view for a valid entry, mirroring the filters in /// [`resolve_partner_ids`] and reporting why each stored ID was skipped. -fn build_auction_view(registry: &PartnerRegistry, entry: &KvEntry) -> AuctionEidsView { - let resolved = resolve_partner_ids(registry, entry); +fn build_auction_view( + registry: &PartnerRegistry, + entry: &KvEntry, + runtime: Option<&IdentityRuntime>, +) -> AuctionEidsView { + // The admin view has no request, so it reads with permissive consent. A + // failed clock resolves no KV IDs, as the auction does (spec 4.4). + let consent = ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + ..ConsentContext::default() + }; + let empty_catalog; + let catalog = match runtime { + Some(runtime) => runtime.catalog(), + None => { + empty_catalog = ProviderCatalog::empty(); + &empty_catalog + } + }; + let opted_out = opted_out(entry); + let now = checked_current_timestamp(); + let serve = ServeContext { + now: UnixTime::from_secs(now.unwrap_or(0)), + consent: &consent, + signals: PolicySignals::default(), + catalog, + opted_out: &opted_out, + }; + let resolved = if now.is_some() { + resolve_partner_ids(registry, entry, &serve) + } else { + Vec::new() + }; let eids = to_eids(&resolved); let mut skipped = Vec::new(); for (source_domain, partner_uid) in &entry.ids { - let reason = if partner_uid.uid.is_empty() { + let reason = if partner_uid.first_uid().is_none_or(str::is_empty) { "empty_uid" } else { match registry.get(source_domain) { None => "not_in_registry", Some(partner) if !partner.bidstream_enabled => "bidstream_disabled", - Some(_) => continue, + Some(_) + if resolved + .iter() + .any(|item| item.source_domain == *source_domain) => + { + continue; + } + Some(_) => "not_servable", } }; skipped.push(SkippedPartnerId { @@ -684,7 +863,7 @@ mod tests { use super::*; use crate::ec::kv_backend::test_support::InMemoryEcKv; use crate::ec::kv_backend::{EcKvStore as _, EcKvWrite, EcKvWriteMode}; - use crate::ec::kv_types::KvPartnerId; + use crate::ec::kv_types::{KvPartnerId, KvProviderState}; use crate::redacted::Redacted; use crate::settings::EcPartner; @@ -703,8 +882,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -778,15 +958,11 @@ mod tests { let mut entry = KvEntry::minimal("bidstream.example", "uid-live", 1_741_824_000); entry.ids.insert( "disabled.example".to_owned(), - KvPartnerId { - uid: "uid-disabled".to_owned(), - }, + KvPartnerId::new("uid-disabled"), ); entry.ids.insert( "unknown.example".to_owned(), - KvPartnerId { - uid: "uid-unknown".to_owned(), - }, + KvPartnerId::new("uid-unknown"), ); entry } @@ -971,13 +1147,156 @@ mod tests { } } + fn admin_lookup_body_for_test(entry: &KvEntry) -> serde_json::Value { + let ec_id = test_ec_id(); + let kv = kv_with_entry(&ec_id, entry); + let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); + + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) + .expect("should handle lookup"); + response_json(response) + } + + #[test] + fn admin_lookup_redacts_provider_state() { + let mut entry = KvEntry::minimal("example.com", "example-uid", 1_760_000_000); + entry.ps.insert( + "id5".to_owned(), + KvProviderState { + n: Some(5), + st: Some(serde_json::json!({ "signature": "secret-state" })), + ..KvProviderState::default() + }, + ); + let body = admin_lookup_body_for_test(&entry); + + assert_eq!( + body["entry"]["ps"]["id5"]["st"], + serde_json::json!("[redacted]"), + "should redact the state" + ); + assert_eq!( + body["entry"]["ps"]["id5"]["n"], + serde_json::json!(5), + "should keep the rest of ps" + ); + assert_eq!( + body["entry"]["ids"]["example.com"]["eids"][0]["uids"][0]["id"], + serde_json::json!("example-uid"), + "should keep the ids" + ); + assert!( + !body.to_string().contains("secret-state"), + "should never return the provider state" + ); + } + + #[test] + fn admin_lookup_reports_records_dropped_on_read() { + let ec_id = test_ec_id(); + let body = serde_json::json!({ + "v": 2, + "created": 1_741_824_000_u64, + "consent": { "ok": true, "updated": 1_741_824_000_u64 }, + "geo": { "country": "US" }, + "ids": { + "bidstream.example": { "eids": [{ "uids": [{ "id": "uid-live" }] }] }, + "blank.example": { "eids": [{ "uids": [{ "id": " " }] }] } + } + }) + .to_string(); + let kv = kv_with_raw_body(&ec_id, &body); + let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); + + let json = response_json( + handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) + .expect("should handle lookup"), + ); + + let error = json["entry_error"].as_str().expect("should report a drop"); + assert!( + error.starts_with("dropped on read: ") && error.contains("blank.example"), + "should name the dropped source, got {error}" + ); + assert_eq!( + json["auction"]["eids"][0]["source"], "bidstream.example", + "should build the auction view from the surviving record" + ); + } + + #[test] + fn admin_view_serves_generic_pull_records_with_the_runtime_catalog() { + use crate::ec::identity::test_support::{ec_partner, record_for_test}; + use crate::test_support::tests::create_test_settings; + + let mut partner = ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + "example-token-value-0123456789abcdef", + ); + partner.api_token = Some(crate::redacted::Redacted::new( + "token-ssp-example-com-32-bytes-minimum".to_owned(), + )); + let registry = PartnerRegistry::from_config(std::slice::from_ref(&partner)) + .expect("should build registry"); + let mut settings = create_test_settings(); + settings.ec.partners = vec![partner]; + let runtime = IdentityRuntime::from_settings( + &settings, + &crate::integrations::IntegrationRegistry::default(), + ) + .expect("should build the runtime"); + let mut entry = KvEntry::minimal("other.example.com", "other", 1); + entry.ids.insert( + "ssp.example.com".to_owned(), + record_for_test("pulled-uid", Some("pull-ssp-example-com"), 1), + ); + + let with_runtime = build_auction_view(®istry, &entry, Some(&runtime)); + let without_runtime = build_auction_view(®istry, &entry, None); + + assert!( + with_runtime + .eids + .iter() + .any(|eid| eid.source == "ssp.example.com"), + "should serve the pulled record when the runtime's catalog holds its writer" + ); + assert!( + without_runtime + .eids + .iter() + .all(|eid| eid.source != "ssp.example.com"), + "control: should not serve it without a runtime" + ); + } + + #[test] + fn auction_view_reports_an_expired_record_as_not_servable() { + let mut entry = KvEntry::minimal("bidstream.example", "uid-expired", 1_741_824_000); + entry + .ids + .get_mut("bidstream.example") + .expect("should hold the source") + .exp = Some(50); + + let view = build_auction_view(&test_registry(), &entry, None); + + assert!(view.eids.is_empty(), "should not resolve the expired ID"); + assert_eq!(view.skipped.len(), 1, "should explain the dropped ID"); + assert_eq!( + view.skipped[0].reason, "not_servable", + "should report the servability reason" + ); + } + #[test] fn returns_entry_with_auction_view() { let ec_id = test_ec_id(); let kv = kv_with_entry(&ec_id, &sample_entry()); let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::OK); @@ -995,7 +1314,7 @@ mod tests { assert_eq!(json["store"], "test-store"); assert_eq!(json["tombstone"], false); assert_eq!( - json["entry"]["ids"]["bidstream.example"]["uid"], "uid-live", + json["entry"]["ids"]["bidstream.example"]["eids"][0]["uids"][0]["id"], "uid-live", "should echo the stored entry verbatim" ); assert_eq!( @@ -1073,7 +1392,7 @@ mod tests { let kv = kv_with_raw_body_and_metadata(&ec_id, &body, &metadata); let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); let json = response_json(response); @@ -1100,7 +1419,7 @@ mod tests { let kv = kv_with_raw_body(&ec_id, &body); let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); let json = response_json(response); @@ -1132,7 +1451,7 @@ mod tests { ); let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::OK); @@ -1152,7 +1471,7 @@ mod tests { let kv = KvIdentityGraph::in_memory("test-store"); let req = get_request(&format!("/_ts/admin/ec/{}", test_ec_id())); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::NOT_FOUND); @@ -1163,7 +1482,7 @@ mod tests { let kv = KvIdentityGraph::in_memory("test-store"); let req = get_request("/_ts/admin/ec/not-a-valid-id"); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::BAD_REQUEST); @@ -1175,7 +1494,7 @@ mod tests { let kv = kv_with_raw_body(&ec_id, "not json at all"); let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!( @@ -1216,7 +1535,7 @@ mod tests { let kv = kv_with_raw_body(&ec_id, &body); let req = get_request(&format!("/_ts/admin/ec/{ec_id}")); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::OK); @@ -1241,7 +1560,7 @@ mod tests { let kv = kv_with_entry(&ec_id, &sample_entry()); let req = get_request_with_cookie("/_ts/admin/ec", &format!("other=1; ts-ec={ec_id}; x=2")); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::OK); @@ -1266,7 +1585,7 @@ mod tests { &format!("ts-ec={stale_ec_id}; ts-ec={live_ec_id}"), ); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::OK); @@ -1283,7 +1602,7 @@ mod tests { let kv = KvIdentityGraph::in_memory("test-store"); let req = get_request("/_ts/admin/ec"); - let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req) + let response = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req) .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::NOT_FOUND); @@ -1301,8 +1620,8 @@ mod tests { fn missing_identity_graph_returns_501() { let req = get_request(&format!("/_ts/admin/ec/{}", test_ec_id())); - let response = - handle_admin_ec_lookup(None, &test_registry(), &req).expect("should handle lookup"); + let response = handle_admin_ec_lookup(None, &test_registry(), None, &req) + .expect("should handle lookup"); assert_eq!(response.status(), StatusCode::NOT_IMPLEMENTED); } @@ -1336,7 +1655,7 @@ mod tests { let kv = KvIdentityGraph::failing("broken-store"); let req = get_request(&format!("/_ts/admin/ec/{}", test_ec_id())); - let result = handle_admin_ec_lookup(Some(&kv), &test_registry(), &req); + let result = handle_admin_ec_lookup(Some(&kv), &test_registry(), None, &req); assert!(result.is_err(), "should propagate KV read failures"); } @@ -1560,4 +1879,137 @@ mod tests { assert_eq!(matched[0]["source_domain"], "sharedid.org"); assert_eq!(matched[0]["uid"], "shared-uid-123"); } + + const OPT_OUT_PROVIDER: &str = "pull-ssp-example-com"; + + fn opt_out_path(ec_id: &str, provider: &str) -> String { + format!("/_ts/admin/ec/{ec_id}/providers/{provider}/opt-out") + } + + fn opted_out_entry() -> KvEntry { + let mut entry = sample_entry(); + entry.ps.insert( + OPT_OUT_PROVIDER.to_owned(), + KvProviderState { + o: Some(crate::ec::kv_types::KvOptOut { + at: 1_000, + fp: "800".to_owned(), + }), + ov: vec!["0123456789abcdef".to_owned()], + ..KvProviderState::default() + }, + ); + entry.ps.insert( + "other-provider".to_owned(), + KvProviderState { + ov: vec!["fedcba9876543210".to_owned()], + ..KvProviderState::default() + }, + ); + entry + } + + #[test] + fn admin_clears_provider_opt_out() { + let ec_id = test_ec_id(); + let kv = kv_with_entry(&ec_id, &opted_out_entry()); + + let response = handle_admin_opt_out_clear( + Some(&kv), + &request_with_method( + http::Method::DELETE, + &opt_out_path(&ec_id, OPT_OUT_PROVIDER), + ), + ); + + assert_eq!( + response.status(), + StatusCode::OK, + "should clear the opt-out" + ); + assert_eq!( + response_json(response), + serde_json::json!({ "cleared": true }), + "should report the clear" + ); + let (stored, _) = kv + .get(&ec_id) + .expect("should read the row") + .expect("should hold the row"); + let state = stored + .ps + .get(OPT_OUT_PROVIDER) + .expect("should keep the provider state"); + assert!( + state.oc.is_some_and(|clear| clear > 1_000), + "should store the clear as oc" + ); + assert!(state.ov.is_empty(), "should empty ps.ov"); + assert_eq!( + stored + .ps + .get("other-provider") + .map(|other| other.ov.clone()), + Some(vec!["fedcba9876543210".to_owned()]), + "control: another provider's state should be untouched" + ); + } + + #[test] + fn admin_opt_out_clear_rejects_what_it_cannot_clear() { + let ec_id = test_ec_id(); + let kv = kv_with_entry(&ec_id, &opted_out_entry()); + let delete = |path: &str| { + handle_admin_opt_out_clear(Some(&kv), &request_with_method(http::Method::DELETE, path)) + .status() + }; + + assert_eq!( + delete(&opt_out_path(&ec_id, "other-provider")), + StatusCode::NOT_FOUND, + "should not find an opt-out the provider does not hold" + ); + assert_eq!( + delete(&opt_out_path("not-an-ec-id", OPT_OUT_PROVIDER)), + StatusCode::BAD_REQUEST, + "should reject an invalid EC ID" + ); + assert_eq!( + delete(&opt_out_path(&ec_id, "Not_A_Provider")), + StatusCode::BAD_REQUEST, + "should reject an invalid provider ID" + ); + let get = handle_admin_opt_out_clear( + Some(&kv), + &get_request(&opt_out_path(&ec_id, OPT_OUT_PROVIDER)), + ); + assert_eq!( + get.status(), + StatusCode::METHOD_NOT_ALLOWED, + "should accept only DELETE" + ); + let (stored, _) = kv + .get(&ec_id) + .expect("should read the row") + .expect("should hold the row"); + assert!( + stored + .ps + .get(OPT_OUT_PROVIDER) + .is_some_and(|state| state.oc.is_none() && !state.ov.is_empty()), + "control: no rejected request should clear the opt-out" + ); + assert_eq!( + handle_admin_opt_out_clear( + None, + &request_with_method( + http::Method::DELETE, + &opt_out_path(&ec_id, OPT_OUT_PROVIDER) + ), + ) + .status(), + StatusCode::NOT_IMPLEMENTED, + "should report an unconfigured identity graph" + ); + } } diff --git a/crates/trusted-server-core/src/ec/auth.rs b/crates/trusted-server-core/src/ec/auth.rs index b7db6f1fa8..eec58d9a2e 100644 --- a/crates/trusted-server-core/src/ec/auth.rs +++ b/crates/trusted-server-core/src/ec/auth.rs @@ -57,8 +57,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } diff --git a/crates/trusted-server-core/src/ec/batch_sync.rs b/crates/trusted-server-core/src/ec/batch_sync.rs index 4903589b29..fce040af47 100644 --- a/crates/trusted-server-core/src/ec/batch_sync.rs +++ b/crates/trusted-server-core/src/ec/batch_sync.rs @@ -24,6 +24,8 @@ use crate::error::TrustedServerError; use super::auth::authenticate_bearer; use super::generation::{is_valid_ec_id, normalize_ec_id_for_kv}; +use super::identity::ids::SourceDomain; +use super::identity::runtime::IdentityRuntime; use super::kv::{KvIdentityGraph, UpsertResult}; use super::log_id; use super::rate_limiter::RateLimiter; @@ -45,6 +47,7 @@ trait BatchSyncWriter { ec_id: &str, partner_id: &str, uid: &str, + only_when_absent: bool, ) -> Result>; } @@ -54,8 +57,15 @@ impl BatchSyncWriter for KvIdentityGraph { ec_id: &str, partner_id: &str, uid: &str, + only_when_absent: bool, ) -> Result> { - KvIdentityGraph::upsert_partner_id_if_exists(self, ec_id, partner_id, uid) + KvIdentityGraph::upsert_partner_id_if_exists_policy( + self, + ec_id, + partner_id, + uid, + only_when_absent, + ) } } @@ -108,6 +118,10 @@ struct MappingGroup { /// Handles `POST /_ts/api/v1/batch-sync`. /// +/// A source an identity provider owns is written only when the row holds no +/// value for it; a stored value, including a provider's entry set, is never +/// replaced. +/// /// # Errors /// /// Returns [`TrustedServerError`] on serialization or KV store failures. @@ -115,15 +129,17 @@ pub fn handle_batch_sync( kv: &KvIdentityGraph, registry: &PartnerRegistry, rate_limiter: &dyn RateLimiter, + identity: &IdentityRuntime, req: Request, ) -> Result, Report> { - handle_batch_sync_with_writer(kv, registry, rate_limiter, req) + handle_batch_sync_with_writer(kv, registry, rate_limiter, identity, req) } fn handle_batch_sync_with_writer( writer: &dyn BatchSyncWriter, registry: &PartnerRegistry, rate_limiter: &dyn RateLimiter, + identity: &IdentityRuntime, req: Request, ) -> Result, Report> { // 1. Authenticate @@ -167,7 +183,13 @@ fn handle_batch_sync_with_writer( } // 4. Process mappings with per-item validation and rejection reasons. - let (accepted, errors) = process_mappings(writer, &partner.source_domain, &body.mappings); + let owned = partner + .source_domain + .parse::() + .ok() + .is_some_and(|source| identity.catalog().owner_of(&source).is_some()); + let (accepted, errors) = + process_mappings(writer, &partner.source_domain, owned, &body.mappings); let rejected = errors.len(); let status = if rejected > 0 { @@ -202,6 +224,7 @@ fn content_length_exceeds_limit(req: &Request, max_body_size: usize) - fn process_mappings( writer: &dyn BatchSyncWriter, partner_id: &str, + owned: bool, mappings: &[SyncMapping], ) -> (usize, Vec) { let mut errors = Vec::new(); @@ -244,7 +267,12 @@ fn process_mappings( let mut accepted = 0; for (group_index, group) in groups.iter().enumerate() { - match writer.upsert_partner_id_if_exists(&group.ec_id, partner_id, &group.partner_uid) { + match writer.upsert_partner_id_if_exists( + &group.ec_id, + partner_id, + &group.partner_uid, + owned, + ) { Ok(UpsertResult::Written | UpsertResult::Unchanged) => { accepted += group.indexes.len(); } @@ -308,6 +336,10 @@ mod tests { use super::*; use std::collections::VecDeque; + use crate::ec::identity::test_support::{ + example_entry_set, fixed_provider, full_record, pid, source as source_domain, + }; + use crate::ec::kv_types::{KvEntry, KvPartnerId}; use crate::error::TrustedServerError; use crate::redacted::Redacted; use crate::settings::EcPartner; @@ -341,6 +373,14 @@ mod tests { ) -> Result> { Ok(self.should_exceed) } + + fn record(&self, _key: &str) -> Result<(), Report> { + Ok(()) + } + + fn count(&self, _key: &str) -> Result> { + Ok(0) + } } #[derive(Clone, Debug, PartialEq, Eq)] @@ -374,6 +414,7 @@ mod tests { ec_id: &str, partner_id: &str, uid: &str, + _only_when_absent: bool, ) -> Result> { self.calls.borrow_mut().push(WriterCall { ec_id: ec_id.to_owned(), @@ -406,8 +447,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -429,6 +471,136 @@ mod tests { serde_json::from_slice(&body).expect("should deserialize batch-sync response") } + fn identity_with_owner(owned: &[(&str, &str)]) -> IdentityRuntime { + let settings = crate::test_support::tests::create_test_settings(); + let providers = owned + .iter() + .map(|(id, source)| fixed_provider(id, &[source])) + .collect(); + let owners = owned + .iter() + .map(|(id, source)| (source_domain(source), pid(id))) + .collect(); + IdentityRuntime::from_settings_with_providers(&settings, providers, owners) + .expect("should build the identity runtime") + } + + fn empty_identity() -> IdentityRuntime { + identity_with_owner(&[]) + } + + fn batch_body(ec_id: &str, uid: &str) -> String { + serde_json::json!({ + "mappings": [{ "ec_id": ec_id, "partner_uid": uid, "timestamp": 1 }] + }) + .to_string() + } + + fn seeded_graph(ec_id: &str, record: Option) -> KvIdentityGraph { + let graph = KvIdentityGraph::in_memory("batch-store"); + let mut entry = KvEntry::tombstone(1000); + entry.consent.ok = true; + if let Some(record) = record { + entry.ids.insert("ssp.example.com".to_owned(), record); + } + graph.create(ec_id, &entry).expect("should seed the row"); + graph + } + + fn stored_record(graph: &KvIdentityGraph, ec_id: &str) -> Option { + let (entry, _) = graph + .get(ec_id) + .expect("should read the row") + .expect("should hold the row"); + entry.ids.get("ssp.example.com").cloned() + } + + fn run_batch( + graph: &KvIdentityGraph, + identity: &IdentityRuntime, + ec_id: &str, + uid: &str, + ) -> serde_json::Value { + let limiter = MockRateLimiter { + should_exceed: false, + }; + let response = handle_batch_sync( + graph, + &test_registry(), + &limiter, + identity, + authorized_batch_request(&batch_body(ec_id, uid)), + ) + .expect("should handle the batch"); + response_json(response) + } + + #[test] + fn batch_sync_update_to_owned_source_applies_only_when_absent() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let owned = identity_with_owner(&[("example-provider", "ssp.example.com")]); + + let absent = seeded_graph(&ec_id, None); + let summary = run_batch(&absent, &owned, &ec_id, "batch-uid"); + assert_eq!(summary["accepted"], 1, "should accept the absent write"); + assert_eq!( + stored_record(&absent, &ec_id), + Some(KvPartnerId::new("batch-uid")), + "should write an owned source that holds no value" + ); + + let present = seeded_graph(&ec_id, Some(KvPartnerId::new("existing-uid"))); + let summary = run_batch(&present, &owned, &ec_id, "batch-uid"); + assert_eq!(summary["accepted"], 1, "should report the skip as accepted"); + assert_eq!( + stored_record(&present, &ec_id), + Some(KvPartnerId::new("existing-uid")), + "should keep the value an owned source already holds" + ); + + let control = seeded_graph(&ec_id, Some(KvPartnerId::new("existing-uid"))); + let summary = run_batch(&control, &empty_identity(), &ec_id, "batch-uid"); + assert_eq!(summary["accepted"], 1, "should accept the unowned update"); + assert_eq!( + stored_record(&control, &ec_id), + Some(KvPartnerId::new("batch-uid")), + "should keep replacing the value of an unowned source" + ); + } + + #[test] + fn batch_sync_never_touches_an_owners_entry_set() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let owned = identity_with_owner(&[("example-provider", "ssp.example.com")]); + let owners_record = full_record(Some("example-provider"), example_entry_set()); + let graph = seeded_graph(&ec_id, Some(owners_record.clone())); + + let before = serde_json::to_vec(&owners_record.eids).expect("should serialize"); + let summary = run_batch(&graph, &owned, &ec_id, "batch-uid"); + let after = stored_record(&graph, &ec_id).expect("should keep the record"); + + assert_eq!(summary["accepted"], 1, "should report the skip as accepted"); + assert_eq!( + serde_json::to_vec(&after.eids).expect("should serialize"), + before, + "should leave the owner's eids byte-equal" + ); + assert_eq!(after.eids.len(), 2, "should keep both entries"); + assert_eq!( + after.w.as_deref(), + Some("example-provider"), + "should keep the owner's writer" + ); + + let unowned = seeded_graph(&ec_id, Some(owners_record)); + run_batch(&unowned, &empty_identity(), &ec_id, "batch-uid"); + assert_eq!( + stored_record(&unowned, &ec_id), + Some(KvPartnerId::new("batch-uid")), + "should write a one-entry, one-UID record without w or rv for an unowned source" + ); + } + fn test_registry() -> PartnerRegistry { let partners = vec![make_test_partner( "ssp.example.com", @@ -489,8 +661,9 @@ mod tests { .body(EdgeBody::from("not-json")) .expect("should build test request"); - let response = handle_batch_sync_with_writer(&writer, ®istry, &limiter, req) - .expect("should return oversized response"); + let response = + handle_batch_sync_with_writer(&writer, ®istry, &limiter, &empty_identity(), req) + .expect("should return oversized response"); assert_eq!( response.status(), @@ -515,8 +688,9 @@ mod tests { .body(EdgeBody::from(oversized_body)) .expect("should build test request"); - let response = handle_batch_sync_with_writer(&writer, ®istry, &limiter, req) - .expect("should return oversized response"); + let response = + handle_batch_sync_with_writer(&writer, ®istry, &limiter, &empty_identity(), req) + .expect("should return oversized response"); assert_eq!( response.status(), @@ -534,7 +708,7 @@ mod tests { mapping(&format!("{}.ABC123", "a".repeat(64)), "u3", 1), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 1, "should count successful writes as accepted"); assert_eq!(errors.len(), 2, "should reject invalid mappings only"); @@ -560,7 +734,7 @@ mod tests { mapping(&format!("{}.ABC123", "c".repeat(64)), "u3", 1), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 1, "should keep accepted count before failure"); assert_eq!( @@ -592,8 +766,8 @@ mod tests { .body(EdgeBody::empty()) .expect("should build test request"); - let response = - handle_batch_sync(&kv, ®istry, &limiter, req).expect("should return response"); + let response = handle_batch_sync(&kv, ®istry, &limiter, &empty_identity(), req) + .expect("should return response"); assert_eq!( response.status(), StatusCode::UNAUTHORIZED, @@ -657,7 +831,7 @@ mod tests { mapping(&withdrawn_ec_id, "uid-2", 101), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 0, "should not accept ineligible mappings"); assert_eq!(errors.len(), 2, "should report both errors"); @@ -674,7 +848,7 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let mappings = vec![mapping(&ec_id, "uid-1", 100), mapping(&ec_id, "uid-1", 101)]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 2, "should accept every unchanged group member"); assert!( @@ -693,7 +867,7 @@ mod tests { mapping(&ec_id, "uid-old", 100), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!( accepted, 2, @@ -723,7 +897,7 @@ mod tests { mapping(&ec_id_a_upper, "a-last", 3), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 3, "should accept every valid group member"); assert!(errors.is_empty(), "should report no errors"); @@ -753,7 +927,7 @@ mod tests { .map(|index| mapping(&ec_id, &format!("uid-{index}"), index as u64)) .collect::>(); - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!( accepted, MAX_BATCH_SIZE, @@ -781,7 +955,7 @@ mod tests { mapping(&mixed_suffix, "mixed", 2), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 2, "should accept both distinct EC IDs"); assert!(errors.is_empty(), "should report no errors"); @@ -813,7 +987,7 @@ mod tests { mapping(&ec_id, " ", 3), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 2, "should accept valid group members"); assert_eq!(errors.len(), 1, "should retain the invalid UID error"); @@ -841,7 +1015,7 @@ mod tests { mapping(&ec_id_b, "b-2", 4), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 0, "should reject all ineligible group members"); assert_eq!( @@ -882,7 +1056,7 @@ mod tests { mapping(&ec_id_c, "", 6), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!( accepted, 2, @@ -929,6 +1103,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return success response"); @@ -959,6 +1134,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return multi-status response"); @@ -997,6 +1173,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return validation response"); @@ -1042,6 +1219,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return infrastructure failure response"); diff --git a/crates/trusted-server-core/src/ec/client_eids.rs b/crates/trusted-server-core/src/ec/client_eids.rs index 30951b9d21..91e824f916 100644 --- a/crates/trusted-server-core/src/ec/client_eids.rs +++ b/crates/trusted-server-core/src/ec/client_eids.rs @@ -216,11 +216,10 @@ fn parse_client_eid_ext(value: &JsonValue) -> Option { #[cfg(test)] mod tests { - use std::sync::{Mutex, Once}; - use serde_json::json; use super::*; + use crate::test_support::logs::{captured_logs, install_capturing_logger}; fn parse(text: &str) -> Vec { let raw = RawValue::from_string(text.to_owned()).expect("should capture raw JSON"); @@ -245,42 +244,6 @@ mod tests { format!(r#"{{"k":"{}"}}"#, "x".repeat(bytes - 8)) } - static CAPTURED_LOGS: Mutex> = Mutex::new(Vec::new()); - static INSTALL_LOGGER: Once = Once::new(); - - struct CapturingLogger; - - impl log::Log for CapturingLogger { - fn enabled(&self, metadata: &log::Metadata<'_>) -> bool { - metadata.level() <= log::Level::Debug - } - - fn log(&self, record: &log::Record<'_>) { - if self.enabled(record.metadata()) { - CAPTURED_LOGS - .lock() - .expect("should lock captured logs") - .push(record.args().to_string()); - } - } - - fn flush(&self) {} - } - - fn install_capturing_logger() { - INSTALL_LOGGER.call_once(|| { - log::set_logger(&CapturingLogger).expect("should install the capturing test logger"); - log::set_max_level(log::LevelFilter::Debug); - }); - } - - fn captured_logs() -> Vec { - CAPTURED_LOGS - .lock() - .expect("should lock captured logs") - .clone() - } - #[test] fn keeps_a_valid_entry_with_every_field() { let parsed = parsed_json( diff --git a/crates/trusted-server-core/src/ec/cookies.rs b/crates/trusted-server-core/src/ec/cookies.rs index ac0e0c05ba..29cb0188b9 100644 --- a/crates/trusted-server-core/src/ec/cookies.rs +++ b/crates/trusted-server-core/src/ec/cookies.rs @@ -18,7 +18,10 @@ use std::borrow::Cow; use edgezero_core::body::Body as EdgeBody; use http::{HeaderValue, Response, header}; -use crate::constants::COOKIE_TS_EC; +use crate::consent::ConsentContext; +use crate::constants::{COOKIE_TS_EC, COOKIE_TS_EC_REVOKED}; +use crate::ec::identity::descriptor::ConsentPolicy; +use crate::ec::identity::policy::{FP_PURPOSES, PolicySignals, fingerprint_bits}; use crate::settings::Settings; /// Maximum age for the EC cookie (1 year in seconds). @@ -153,6 +156,54 @@ pub fn set_ec_cookie(settings: &Settings, response: &mut Response, ec_ } } +/// The value of the global-revoke marker for `consent`: a consent +/// fingerprint over every TCF purpose, the US opt-outs and GPC, hex. It +/// changes when any of the visitor's purpose decisions changes, and not +/// when a CMP re-saves the same decisions (spec 8). +#[must_use] +pub(crate) fn revoke_marker_value(consent: &ConsentContext) -> String { + let policy = ConsentPolicy::strict().with_tcf_purposes(1..=FP_PURPOSES); + fingerprint_bits(&policy, consent, PolicySignals::default()) +} + +fn append_revoke_marker( + settings: &Settings, + response: &mut Response, + value: &str, + max_age: i32, +) { + let cookie = format!( + "{COOKIE_TS_EC_REVOKED}={value}; Domain={}; Path=/; Secure; SameSite=Lax; Max-Age={max_age}; HttpOnly", + settings.publisher.ec_cookie_domain() + ); + match HeaderValue::from_str(&cookie) { + Ok(val) => { + response.headers_mut().append(header::SET_COOKIE, val); + } + Err(e) => log::warn!("Skipping revoke marker Set-Cookie: invalid header value: {e}"), + } +} + +/// Sets the global-revoke marker to the fingerprint of `consent`, so no EC +/// is minted until the visitor's consent changes (spec 8). +pub(crate) fn set_revoke_marker( + settings: &Settings, + response: &mut Response, + consent: &ConsentContext, +) { + append_revoke_marker( + settings, + response, + &revoke_marker_value(consent), + COOKIE_MAX_AGE, + ); +} + +/// Expires the global-revoke marker once a new EC has been minted. +pub(crate) fn expire_revoke_marker(settings: &Settings, response: &mut Response) { + append_revoke_marker(settings, response, "", 0); +} + /// Expires the EC cookie by setting `Max-Age=0`. /// /// Used when a user revokes consent — the browser will delete the cookie @@ -177,9 +228,32 @@ pub fn expire_ec_cookie(settings: &Settings, response: &mut Response) #[cfg(test)] mod tests { use super::*; + use crate::ec::identity::test_support::test_consent_gdpr_without_vendor; use crate::test_support::tests::create_test_settings; use http::header; + #[test] + fn revoke_marker_changes_with_any_purpose_decision() { + let granted = test_consent_gdpr_without_vendor(5); + let mut withdrawn = granted.clone(); + withdrawn + .tcf + .as_mut() + .expect("should carry a TC string") + .purpose_consents[1] = false; + + assert_ne!( + revoke_marker_value(&granted), + revoke_marker_value(&withdrawn), + "should change when Purpose 2 alone changes" + ); + assert_eq!( + revoke_marker_value(&granted), + revoke_marker_value(&granted.clone()), + "should not change when nothing changes" + ); + } + fn empty_response() -> Response { Response::builder() .status(200) diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index 0493f5dec1..5d7e763f73 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -2,7 +2,9 @@ //! //! Used by both `/_ts/api/v1/identify` and `/auction` to resolve source-domain //! keyed IDs from KV entries, convert them to `OpenRTB` EID structures, and -//! build base64-encoded response headers. +//! build base64-encoded response headers. KV IDs are filtered by the shared +//! servability check in [`super::identity::servable`], and a stored record +//! goes out as one EID per stored entry. use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; use error_stack::{Report, ResultExt as _}; @@ -10,36 +12,54 @@ use error_stack::{Report, ResultExt as _}; use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; -use super::kv_types::KvEntry; +use super::identity::catalog::ProviderCatalog; +use super::identity::coordinator::HoldResults; +use super::identity::servable::{ServeContext, is_servable}; +use super::identity::{IdentityProviderId, SourceDomain}; +use super::kv_types::{KvEid, KvEntry}; use super::registry::PartnerRegistry; /// Maximum size (in bytes) for the base64-encoded `x-ts-eids` header value. pub const MAX_EIDS_HEADER_BYTES: usize = 4096; -/// A source-domain keyed ID resolved from a KV entry against the partner registry. +/// A source-domain keyed record resolved from a KV entry against the partner +/// registry and the servability check. /// -/// Only includes partners with `bidstream_enabled = true` and a non-empty UID. +/// Only includes partners with `bidstream_enabled = true`, a non-empty UID and +/// a servable record. pub struct ResolvedPartnerId { /// The partner's identity source domain and EC KV `ids` key. pub source_domain: String, - /// The synced user ID value. + /// The first UID of the first stored entry. pub uid: String, - /// `OpenRTB` agent type for this partner's identifiers. + /// `OpenRTB` agent type, used for a UID without a stored one. pub openrtb_atype: i32, + /// The source's stored entry set, in order. One entry for a one-UID record. + pub entries: Vec, + /// The provider that wrote the record. `None` for browser ingest and + /// version 1 rows. + pub writer: Option, + /// Unix seconds at which the record stops being servable, when it has an + /// `exp`. + pub expires_at: Option, } /// Resolves source-domain keyed IDs from a KV entry against the partner registry. /// -/// Filters to partners with `bidstream_enabled = true` and non-empty UIDs, -/// sorted deterministically by source domain. +/// Filters to partners with `bidstream_enabled = true`, non-empty UIDs and +/// servable records, sorted deterministically by source domain. #[must_use] -pub fn resolve_partner_ids(registry: &PartnerRegistry, entry: &KvEntry) -> Vec { +pub(crate) fn resolve_partner_ids( + registry: &PartnerRegistry, + entry: &KvEntry, + serve: &ServeContext<'_>, +) -> Vec { let mut resolved = Vec::new(); for (source_domain, partner_uid) in &entry.ids { - if partner_uid.uid.is_empty() { + let Some(uid) = partner_uid.first_uid().filter(|uid| !uid.is_empty()) else { continue; - } + }; let Some(partner) = registry.get(source_domain) else { continue; @@ -48,10 +68,23 @@ pub fn resolve_partner_ids(registry: &PartnerRegistry, entry: &KvEntry) -> Vec() else { + continue; + }; + if !is_servable(serve, &source, partner_uid) { + continue; + } + resolved.push(ResolvedPartnerId { source_domain: partner.source_domain.clone(), - uid: partner_uid.uid.clone(), + uid: uid.to_owned(), openrtb_atype: partner.openrtb_atype, + entries: partner_uid.eids.clone(), + writer: partner_uid + .w + .as_deref() + .and_then(|writer| writer.parse().ok()), + expires_at: partner_uid.exp, }); } @@ -59,24 +92,230 @@ pub fn resolve_partner_ids(registry: &PartnerRegistry, entry: &KvEntry) -> Vec Vec { resolved .iter() - .map(|item| { - Eid::new( - item.source_domain.clone(), - vec![Uid { - id: item.uid.clone(), - atype: Some(item.openrtb_atype), - ext: None, - }], - ) + .flat_map(|item| { + item.entries + .iter() + .map(move |entry| stored_entry_to_eid(item, entry)) }) .collect() } +fn stored_entry_to_eid(item: &ResolvedPartnerId, entry: &KvEid) -> Eid { + let uids = entry + .uids + .iter() + .map(|uid| Uid { + id: uid.id.clone(), + atype: Some(uid.atype.unwrap_or(item.openrtb_atype)), + ext: uid.ext.clone(), + }) + .collect(); + Eid { + source: item.source_domain.clone(), + uids, + inserter: entry.inserter.clone(), + matcher: entry.matcher.clone(), + mm: entry.mm, + ext: entry.ext.clone(), + } +} + +/// One source of a [`ServerSet`]: a servable record and its outbound EIDs. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct ServerSource { + /// The partner's source domain, as the registry holds it. + pub(crate) source_domain: String, + /// The provider that wrote the record, if any. + pub(crate) writer: Option, + /// Unix seconds at which the record stops being servable. + pub(crate) expires_at: Option, + /// Whether the record is the source owner's valid value: the source has + /// an owner and the record's writer is that owner (spec 7.1). + pub(crate) owned: bool, + /// One EID per stored entry, in stored order, provenance defaults filled. + pub(crate) eids: Vec, +} + +/// The servable server set of the KV row a request read (spec 7.1). +/// +/// Built once per request; the bidstream merge and the `ts-eids-s` mirror +/// both read it. It never holds anything from `ts-eids-s`. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct ServerSet { + sources: Vec, +} + +impl ServerSet { + /// The set's sources, ordered by source domain. + #[must_use] + pub(crate) fn sources(&self) -> &[ServerSource] { + &self.sources + } + + /// Every source's EIDs, in order. + #[cfg(test)] + #[must_use] + pub(crate) fn eids(&self) -> Vec { + self.sources + .iter() + .flat_map(|source| source.eids.iter().cloned()) + .collect() + } + + /// A set of one source per EID, for tests that start from EIDs. + #[cfg(test)] + #[must_use] + pub(crate) fn from_eids(eids: Vec) -> Self { + Self { + sources: eids + .into_iter() + .map(|eid| ServerSource { + source_domain: eid.source.clone(), + writer: None, + expires_at: None, + owned: false, + eids: vec![eid], + }) + .collect(), + } + } +} + +impl ServerSet { + /// Overlays the answers of this navigation's hold dispatch (spec 4.7), + /// replacing the stored value of each source they name. + /// + /// Each source gets one `Eid` per entry, in the order the provider + /// returned them, with the writer's provenance default applied + /// ([`fill_provenance`]). A source the partner registry does not list for + /// the bidstream is left out, as a stored record would be. + pub(crate) fn with_hold_results( + &mut self, + results: HoldResults, + registry: &PartnerRegistry, + catalog: &ProviderCatalog, + ) { + let HoldResults { + additions, + mut writer, + } = results; + for (source, entries) in additions { + let Some(partner) = registry + .get(source.as_str()) + .filter(|partner| partner.bidstream_enabled) + else { + continue; + }; + let writer = writer.remove(&source); + let mut eids: Vec = entries + .iter() + .map(|entry| Eid { + source: partner.source_domain.clone(), + uids: entry + .uids + .iter() + .map(|uid| Uid { + id: uid.id.clone(), + atype: Some(uid.atype.unwrap_or(partner.openrtb_atype)), + ext: uid.ext.clone(), + }) + .collect(), + inserter: entry.inserter.clone(), + matcher: entry.matcher.clone(), + mm: entry.mm, + ext: entry.ext.clone(), + }) + .collect(); + fill_provenance(&mut eids, writer.as_ref(), &source, catalog); + let held = ServerSource { + source_domain: partner.source_domain.clone(), + owned: writer.is_some() && catalog.owner_of(&source) == writer.as_ref(), + writer, + expires_at: None, + eids, + }; + match self + .sources + .binary_search_by(|stored| stored.source_domain.cmp(&held.source_domain)) + { + Ok(index) => self.sources[index] = held, + Err(index) => self.sources.insert(index, held), + } + } + } +} + +/// The servable server set of `entry` (spec 7.1). +/// +/// Without a read row, or for a row that is not live and consenting, the set +/// is empty. Each record goes through [`resolve_partner_ids`] and +/// [`to_eids`], then [`fill_provenance`]. +#[must_use] +pub(crate) fn server_set( + entry: Option<&KvEntry>, + registry: &PartnerRegistry, + serve: &ServeContext<'_>, +) -> ServerSet { + let Some(entry) = entry.filter(|entry| entry.consent.ok) else { + return ServerSet::default(); + }; + let sources = resolve_partner_ids(registry, entry, serve) + .into_iter() + .map(|record| { + let mut eids = to_eids(core::slice::from_ref(&record)); + let mut owned = false; + if let Ok(source) = record.source_domain.parse::() { + fill_provenance(&mut eids, record.writer.as_ref(), &source, serve.catalog); + owned = record.writer.is_some() + && serve.catalog.owner_of(&source) == record.writer.as_ref(); + } + ServerSource { + source_domain: record.source_domain, + writer: record.writer, + expires_at: record.expires_at, + owned, + eids, + } + }) + .collect(); + ServerSet { sources } +} + +/// Applies the writer's `with_provenance` default for `source` to the EIDs +/// of a record it wrote (spec 4.3, 7.1). +/// +/// Only an entry that sets none of `inserter`, `matcher` and `mm` gets the +/// default; an entry that sets any of them is the vendor's complete +/// statement and is left as stored. A record without a configured writer +/// gets nothing. +pub(crate) fn fill_provenance( + eids: &mut [Eid], + writer: Option<&IdentityProviderId>, + source: &SourceDomain, + catalog: &ProviderCatalog, +) { + let Some(default) = writer + .and_then(|writer| catalog.get(writer)) + .and_then(|provider| provider.descriptor().provenance(source)) + else { + return; + }; + for eid in eids + .iter_mut() + .filter(|eid| eid.inserter.is_none() && eid.matcher.is_none() && eid.mm.is_none()) + { + eid.inserter = default.inserter().map(str::to_owned); + eid.matcher = default.matcher().map(str::to_owned); + eid.mm = default.mm(); + } +} + /// Builds a base64-encoded EID header value, truncating if needed. /// /// Returns `(encoded_value, was_truncated)`. If the full set of EIDs exceeds @@ -146,9 +385,58 @@ pub fn encode_eids_header(eids: &[Eid]) -> Result<(String, bool), Report, + } + + impl ServeFixture { + fn new() -> Self { + Self { + consent: test_consent_allowing_everything(), + catalog: ProviderCatalog::empty(), + opted_out: BTreeSet::new(), + } + } + + fn at(&self, now: u64) -> ServeContext<'_> { + ServeContext { + now: UnixTime::from_secs(now), + consent: &self.consent, + signals: PolicySignals::default(), + catalog: &self.catalog, + opted_out: &self.opted_out, + } + } + } + + fn registry_with(sources: &[&str]) -> PartnerRegistry { + let partners: Vec = sources + .iter() + .map(|source| EcPartner { + openrtb_atype: 99, + ..make_test_partner(source) + }) + .collect(); + PartnerRegistry::from_config(&partners).expect("should build registry") + } fn make_test_partner(source_domain: &str) -> EcPartner { EcPartner { @@ -163,8 +451,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -179,20 +468,15 @@ mod tests { let mut entry = KvEntry::tombstone(1000); entry.consent.ok = true; - entry.ids.insert( - "zeta.example.com".to_owned(), - super::super::kv_types::KvPartnerId { - uid: "uid-z".to_owned(), - }, - ); - entry.ids.insert( - "alpha.example.com".to_owned(), - super::super::kv_types::KvPartnerId { - uid: "uid-a".to_owned(), - }, - ); - - let resolved = resolve_partner_ids(®istry, &entry); + entry + .ids + .insert("zeta.example.com".to_owned(), KvPartnerId::new("uid-z")); + entry + .ids + .insert("alpha.example.com".to_owned(), KvPartnerId::new("uid-a")); + + let fixture = ServeFixture::new(); + let resolved = resolve_partner_ids(®istry, &entry, &fixture.at(100)); let source_domains: Vec<&str> = resolved .iter() .map(|item| item.source_domain.as_str()) @@ -205,6 +489,111 @@ mod tests { ); } + #[test] + fn resolve_partner_ids_skips_expired_records_and_uses_stored_atype() { + let registry = registry_with(&["a.example.com", "b.example.com"]); + let mut entry = KvEntry::minimal("a.example.com", "ua", 1); + entry.ids.insert( + "b.example.com".to_owned(), + KvPartnerId { + exp: Some(50), + ..KvPartnerId::new("ub") + }, + ); + entry + .ids + .get_mut("a.example.com") + .expect("should hold source a") + .eids[0] + .uids[0] + .atype = Some(3); + let fixture = ServeFixture::new(); + + let resolved = resolve_partner_ids(®istry, &entry, &fixture.at(100)); + + assert_eq!(resolved.len(), 1, "should drop the expired b"); + assert_eq!(resolved[0].source_domain, "a.example.com", "should keep a"); + assert_eq!( + to_eids(&resolved)[0].uids[0].atype, + Some(3), + "should prefer the stored atype" + ); + } + + #[test] + fn a_one_uid_record_goes_out_as_on_main() { + let registry = registry_with(&["a.example.com"]); + let entry = KvEntry::minimal("a.example.com", "ua", 1); + let fixture = ServeFixture::new(); + + let eids = to_eids(&resolve_partner_ids(®istry, &entry, &fixture.at(100))); + + assert_eq!( + eids, + vec![Eid::new( + "a.example.com".to_owned(), + vec![Uid { + id: "ua".to_owned(), + atype: Some(99), + ext: None + }], + )], + "should be the EID main sends" + ); + } + + #[test] + fn a_full_record_goes_out_as_one_eid_per_stored_entry() { + let registry = registry_with(&["a.example.com"]); + let mut set = example_entry_set(); + set[1].uids[0].atype = None; + let mut entry = KvEntry::minimal("b.example.com", "unregistered", 1); + entry + .ids + .insert("a.example.com".to_owned(), full_record(None, set)); + let fixture = ServeFixture::new(); + + let eids = to_eids(&resolve_partner_ids(®istry, &entry, &fixture.at(100))); + + assert_eq!( + eids, + vec![ + Eid { + source: "a.example.com".to_owned(), + uids: vec![ + Uid { + id: "example-a".to_owned(), + atype: Some(1), + ext: None + }, + Uid { + id: "example-b".to_owned(), + atype: Some(3), + ext: Some(json!({ "stype": "ppuid" })) + }, + ], + inserter: Some("example.com".to_owned()), + matcher: None, + mm: Some(3), + ext: None, + }, + Eid { + source: "a.example.com".to_owned(), + uids: vec![Uid { + id: "example-c".to_owned(), + atype: Some(99), + ext: None + }], + inserter: None, + matcher: Some("partner.example.com".to_owned()), + mm: None, + ext: None, + }, + ], + "should send every entry in stored order, a UID without atype taking the partner's" + ); + } + #[test] fn to_eids_maps_resolved_ids_correctly() { let resolved = vec![ @@ -212,16 +601,25 @@ mod tests { uid: "LR_xyz".to_owned(), source_domain: "liveramp.com".to_owned(), openrtb_atype: 3, + entries: KvPartnerId::new("LR_xyz".to_owned()).eids, + writer: None, + expires_at: None, }, ResolvedPartnerId { uid: "ID5_abc".to_owned(), source_domain: "id5-sync.com".to_owned(), openrtb_atype: 1, + entries: KvPartnerId::new("ID5_abc".to_owned()).eids, + writer: None, + expires_at: None, }, ResolvedPartnerId { uid: "pair-id".to_owned(), source_domain: "google.com".to_owned(), openrtb_atype: 571187, + entries: KvPartnerId::new("pair-id".to_owned()).eids, + writer: None, + expires_at: None, }, ]; @@ -251,6 +649,9 @@ mod tests { uid: format!("uid_{}", "x".repeat(100)), source_domain: format!("partner-{idx}.example.com"), openrtb_atype: 3, + entries: KvPartnerId::new(format!("uid_{}", "x".repeat(100))).eids, + writer: None, + expires_at: None, }); } @@ -270,6 +671,9 @@ mod tests { uid: "u1".to_owned(), source_domain: "ssp.com".to_owned(), openrtb_atype: 3, + entries: KvPartnerId::new("u1".to_owned()).eids, + writer: None, + expires_at: None, }]; let (encoded, truncated) = @@ -278,4 +682,120 @@ mod tests { assert!(!truncated, "should not truncate small payload"); assert!(!encoded.is_empty(), "should produce non-empty value"); } + + fn provenance_catalog() -> ProviderCatalog { + ProviderCatalog::new( + vec![provenance_provider( + "pa", + &["a.example.com", "b.example.com"], + vec![ + ( + "a.example.com", + EidProvenance::new() + .with_inserter("default-inserter.example.com") + .with_matcher("default-matcher.example.com") + .with_mm(2), + ), + ( + "b.example.com", + EidProvenance::new().with_inserter("default-inserter.example.com"), + ), + ], + )], + BTreeMap::new(), + ) + } + + #[test] + fn merge_provenance_applies_only_to_an_entry_with_no_eid_level_fields() { + let registry = registry_with(&["a.example.com", "b.example.com"]); + let kv_uid = |id: &str| KvUid { + id: id.to_owned(), + atype: None, + ext: None, + }; + let mut entry = KvEntry::minimal("unused.example.com", "unused", 1); + entry.ids.clear(); + entry.ids.insert( + "a.example.com".to_owned(), + full_record( + Some("pa"), + vec![ + KvEid { + uids: vec![kv_uid("bare")], + ..KvEid::default() + }, + KvEid { + uids: vec![kv_uid("vendor")], + inserter: Some("vendor.example.com".to_owned()), + ..KvEid::default() + }, + ], + ), + ); + entry + .ids + .insert("b.example.com".to_owned(), KvPartnerId::new("seeded")); + let fixture = ServeFixture { + catalog: provenance_catalog(), + ..ServeFixture::new() + }; + + let set = server_set(Some(&entry), ®istry, &fixture.at(100)); + + let uid = |id: &str| Uid { + id: id.to_owned(), + atype: Some(99), + ext: None, + }; + assert_eq!( + set.eids(), + vec![ + Eid { + source: "a.example.com".to_owned(), + uids: vec![uid("bare")], + inserter: Some("default-inserter.example.com".to_owned()), + matcher: Some("default-matcher.example.com".to_owned()), + mm: Some(2), + ext: None, + }, + Eid { + source: "a.example.com".to_owned(), + uids: vec![uid("vendor")], + inserter: Some("vendor.example.com".to_owned()), + matcher: None, + mm: None, + ext: None, + }, + Eid { + source: "b.example.com".to_owned(), + uids: vec![uid("seeded")], + inserter: None, + matcher: None, + mm: None, + ext: None, + }, + ], + "should fill the default only on the provider-written entry with no EID-level field" + ); + } + + #[test] + fn merge_server_set_is_empty_without_a_live_consenting_row() { + let registry = registry_with(&["a.example.com"]); + let fixture = ServeFixture::new(); + let mut entry = KvEntry::minimal("a.example.com", "ua", 1); + entry.consent.ok = false; + + assert_eq!( + server_set(None, ®istry, &fixture.at(100)), + ServerSet::default(), + "should be empty without a KV read" + ); + assert_eq!( + server_set(Some(&entry), ®istry, &fixture.at(100)), + ServerSet::default(), + "should be empty for a withdrawn row" + ); + } } diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index e41a8180d4..0508b5a2b1 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -10,21 +10,21 @@ use http::Response; use super::consent::{ec_consent_granted, ec_consent_withdrawn}; use crate::consent::{ConsentContext, allows_eid_persistence}; -use crate::openrtb::Eid; use crate::settings::Settings; use super::EcContext; -use super::cookies::{expire_ec_cookie, set_ec_cookie}; +use super::cookies::{expire_ec_cookie, expire_revoke_marker, set_ec_cookie, set_revoke_marker}; use super::generation::{generate_ec_id, is_valid_ec_id}; +use super::identity::ids::SourceDomain; +use super::identity::mutation::{IdentityMutation, PartnerIdChange}; use super::kv::{ CreateIfAbsentOutcome, EidCookieSyncOutcome, KvIdentityGraph, PartnerIdUpdate, apply_partner_id_updates, }; -use super::kv_types::KvEntry; +use super::kv_types::{KvEntry, KvPartnerId}; use super::prebid_eids::collect_eid_updates; -use super::pull_sync_marker::{expire_marker, reconcile_marker}; use super::registry::PartnerRegistry; -use super::{EcKvSnapshot, EidSyncSource, current_timestamp, log_id}; +use super::{EcKvSnapshot, EidSyncSource, checked_current_timestamp, current_timestamp, log_id}; /// TS-managed response headers tied to EC identity output. const EC_RESPONSE_HEADERS: &[&str] = &[ @@ -36,16 +36,20 @@ const EC_RESPONSE_HEADERS: &[&str] = &[ /// Finalizes EC response behavior for all routes. /// -/// Applies withdrawal handling, last-seen updates, cookie reconciliation, -/// Prebid EID ingestion, and cookie writes for new EC generation. +/// Applies withdrawal handling, cookie reconciliation, orphan recovery and the +/// `ts-ec` cookie for a new EC. +/// +/// A returning visitor's response writes no KV row. On the writer request the +/// browser ingest from `ts-eids` and sharedId joins the request's identity +/// mutation, which phase 4 flushes; every other request ingests nothing. A new +/// EC's ingest rode in its create ([`BrowserSeed`]). /// /// On consent withdrawal, the browser response clears the EC cookie /// immediately and the EC identity-graph KV tombstone is the authoritative /// revocation marker. There is no separate consent KV store to clean up. /// /// `eids_cookie` should be the raw value of the `ts-eids` cookie extracted -/// from the request *before* routing consumes it. `/auction` responses ignore -/// it in favor of the request's own client set. +/// from the request *before* routing consumes it. pub fn ec_finalize_response( settings: &Settings, ec_context: &mut EcContext, @@ -55,17 +59,24 @@ pub fn ec_finalize_response( sharedid_cookie: Option<&str>, response: &mut Response, ) { - ec_context.validate_pull_sync_marker(settings, registry); + // A minted EC whose navigation never reached its create gets it here, + // before its cookie is set (spec 4.7). + if let Err(err) = ec_context.complete_deferred_create(settings, kv) { + log::warn!("Deferred EC create failed at finalize: {err:?}"); + } let consent_allows_ec = ec_consent_granted(ec_context.consent()); - let consent_withdrawn = ec_consent_withdrawn(ec_context.consent()); - - if !consent_allows_ec { - // Expire the request-local marker independently of the EC cookie: a - // withdrawal must stop any pending pull-sync disclosure window. - if consent_withdrawn && ec_context.pull_sync_marker().was_present() { - expire_marker(ec_context.pull_sync_marker_mut(), response); + // A global revoke captured from any client (spec 8) is a withdrawal: + // the same tombstone, cookie expiry and after-send run, with whatever the + // request staged dropped. + let global_revoke = ec_context.global_revoke_requested(); + let consent_withdrawn = ec_consent_withdrawn(ec_context.consent()) || global_revoke; + + if !consent_allows_ec || global_revoke { + if global_revoke { + *ec_context.identity_mut().mutation_mut() = IdentityMutation::default(); + // No EC is minted until the visitor's consent changes (spec 8). + set_revoke_marker(settings, response, ec_context.consent()); } - finalize_unusable_consent( settings, ec_context, @@ -80,21 +91,23 @@ pub fn ec_finalize_response( // Returning user: consent is granted and EC came from request. if ec_context.ec_was_present() && !ec_context.ec_generated() && consent_allows_ec { if let (Some(graph), Some(ec_id)) = (kv, ec_context.ec_value().map(str::to_owned)) { - let source = ec_context.eid_sync_source(); - let collected = source - .map(|_| { - let (cookie_input, client_input) = browser_eid_input(ec_context, eids_cookie); - collect_eid_updates(cookie_input, sharedid_cookie, client_input, registry) - }) - .unwrap_or_default(); + // Only the writer request ingests browser data (spec 5.3). Every + // other request leaves `ts-eids` and sharedId for the writer. + let writer = ec_context.identity().writer(); + let collected = if writer { + collect_eid_updates( + eids_cookie, + sharedid_cookie, + None, + registry, + checked_current_timestamp(), + ) + } else { + Vec::new() + }; let had_eid_updates = !collected.is_empty(); let updates = gate_eid_updates_by_consent(collected, ec_context.consent()); - // `sync_eid_cookie_updates` early-returns without refreshing the - // snapshot when `updates` is empty, which is correct when there - // was never anything to write (an unconfigured registry, no sync - // source, or no EID cookies/body this request — the case - // `finalize_not_read_snapshot_does_not_rotate` covers). But when - // consent gating (e.g. TCF Purpose 4 denial) is what emptied + // When consent gating (e.g. TCF Purpose 4 denial) is what emptied // `updates`, the request *did* have EID data, and orphan recovery // below still needs an actual `Missing` read to detect an // orphaned cookie. An unread `NotRead` snapshot proves nothing, so @@ -107,8 +120,6 @@ pub fn ec_finalize_response( { let snapshot = graph.load_snapshot(&ec_id); ec_context.set_kv_snapshot(snapshot); - } else if let Some(source) = source { - sync_eid_cookie_updates(graph, ec_context, &ec_id, &updates, source); } if matches!(ec_context.kv_snapshot(), EcKvSnapshot::Missing { .. }) && ec_context.recovery_eligible() @@ -116,11 +127,11 @@ pub fn ec_finalize_response( confirm_then_recover_orphaned_ec( settings, ec_context, graph, &ec_id, &updates, response, ); + } else { + queue_browser_ingest(ec_context, &ec_id, &updates); } } - reconcile_pull_sync_marker(settings, registry, ec_context, response); - // Ordinary returning-user page views no longer refresh the browser // cookie, emit the EC header, or update KV TTL. return; @@ -130,126 +141,132 @@ pub fn ec_finalize_response( // there is no KV graph: that would mint a browser cookie with no backing // identity-graph row, producing a phantom ID on later requests. if ec_context.ec_generated() { - let (Some(graph), Some(ec_id)) = (kv, ec_context.ec_value().map(str::to_owned)) else { + let (Some(_), Some(ec_id)) = (kv, ec_context.ec_value().map(str::to_owned)) else { log::info!("Skipping generated EC response write because KV graph is unavailable"); - reconcile_pull_sync_marker(settings, registry, ec_context, response); return; }; - let (cookie_input, client_input) = browser_eid_input(ec_context, eids_cookie); - let updates = gate_eid_updates_by_consent( - collect_eid_updates(cookie_input, sharedid_cookie, client_input, registry), - ec_context.consent(), - ); - sync_eid_cookie_updates(graph, ec_context, &ec_id, &updates, EidSyncSource::NewEc); - if ec_context.kv_snapshot().entry_for(&ec_id).is_some() { + // The create in `generate_if_needed` (or the integration proxy's) + // already carried the browser ingest, so finalize writes nothing. + if let Some(entry) = ec_context.kv_snapshot().entry_for(&ec_id) { + let seeded = gate_eid_updates_by_consent( + collect_eid_updates( + eids_cookie, + sharedid_cookie, + None, + registry, + checked_current_timestamp(), + ), + ec_context.consent(), + ); + if !seeded.is_empty() && partner_id_updates_stored(entry, &seeded) { + record_eid_sync_terminal(EidSyncSource::NewEc, EidCookieSyncOutcome::Written); + } set_ec_cookie_on_response(settings, ec_context, response); + if ec_context.has_revoke_marker() { + expire_revoke_marker(settings, response); + } } else { log::warn!("Skipping generated EC cookie because backing row is not authoritative"); } } - - reconcile_pull_sync_marker(settings, registry, ec_context, response); } -/// Chooses the browser EID input KV ingestion reads for this request. +/// The browser ingest of one request, ready to seed a new identity row. /// -/// `/auction` ingests only the client set the auction itself used (its body, -/// else its `ts-eids` cookie, as `handle_auction` resolved it). A cookie from -/// an earlier page is then never ingested next to a scoped body, and an -/// `/auction` that resolved no client set ingests nothing from the cookie. -/// Every other route, and requests with no sync source, read the cookie. -fn browser_eid_input<'a>( - ec_context: &'a EcContext, - eids_cookie: Option<&'a str>, -) -> (Option<&'a str>, Option<&'a [Eid]>) { - match ec_context.eid_sync_source() { - Some(EidSyncSource::Auction) => (None, ec_context.client_eids()), - _ => (eids_cookie, None), +/// Opaque on purpose: it holds UIDs, so it prints only a count. Build it with +/// [`browser_seed`], or use [`BrowserSeed::empty`] when there is nothing to seed. +#[derive(Clone, Default)] +pub struct BrowserSeed(Vec); + +impl BrowserSeed { + /// A seed that carries nothing. + #[must_use] + pub fn empty() -> Self { + Self::default() + } + + /// The updates to apply to the new row. + pub(crate) fn updates(&self) -> &[PartnerIdUpdate] { + &self.0 } } -fn sync_eid_cookie_updates( - graph: &KvIdentityGraph, - ec_context: &mut EcContext, - ec_id: &str, - updates: &[PartnerIdUpdate], - source: EidSyncSource, -) { - if updates.is_empty() { - return; +impl core::fmt::Debug for BrowserSeed { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("BrowserSeed") + .field("updates", &self.0.len()) + .finish() } +} - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - ec_id, - updates, - ec_context.kv_snapshot().clone(), - ); - ec_context.set_kv_snapshot(snapshot); - record_eid_sync_terminal(source, outcome); +/// Collects the consent-gated browser ingest from `ts-eids` and sharedId. +/// +/// One UID per registered source: the `/auction` body is not an ingest input. +/// `now` is the acquisition time stored on each record. +#[must_use] +pub fn browser_seed( + _settings: &Settings, + registry: &PartnerRegistry, + consent: &ConsentContext, + eids_cookie: Option<&str>, + sharedid_cookie: Option<&str>, + now: u64, +) -> BrowserSeed { + BrowserSeed(gate_eid_updates_by_consent( + collect_eid_updates(eids_cookie, sharedid_cookie, None, registry, Some(now)), + consent, + )) } -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -struct EidSyncMeasurement { - source: EidSyncSource, - outcome: EidCookieSyncOutcome, - already_matched: u8, - written: u8, - conflict_duplicate: u8, - deferred: u8, +/// Whether `entry` already holds every update's UID. +fn partner_id_updates_stored(entry: &KvEntry, updates: &[PartnerIdUpdate]) -> bool { + updates.iter().all(|update| { + entry + .ids + .get(&update.partner_id) + .is_some_and(|stored| stored.first_uid() == Some(update.uid.as_str())) + }) } -impl EidSyncMeasurement { - fn new(source: EidSyncSource, outcome: EidCookieSyncOutcome) -> Self { - Self { +/// Adds the browser ingest to the writer's mutation, for phase 4 to flush. +/// +/// Nothing is queued against a withdrawn, missing or failed row. A `NotRead` +/// snapshot still queues: phase 4 re-reads before it writes. +fn queue_browser_ingest(ec_context: &mut EcContext, ec_id: &str, updates: &[PartnerIdUpdate]) { + if updates.is_empty() || !ec_context.identity().writer() { + return; + } + let usable = match ec_context.kv_snapshot() { + EcKvSnapshot::NotRead => true, + snapshot => snapshot + .entry_for(ec_id) + .is_some_and(|entry| entry.consent.ok), + }; + if !usable { + return; + } + let mutation = ec_context.identity_mut().mutation_mut(); + for update in updates { + let Ok(source) = update.partner_id.parse::() else { + log::warn!("Browser ingest skipped a source that is not a valid domain"); + continue; + }; + mutation.add_change(PartnerIdChange::FillMissing { source, - outcome, - already_matched: u8::from(matches!(outcome, EidCookieSyncOutcome::AlreadyMatched)), - written: u8::from(matches!( - outcome, - EidCookieSyncOutcome::Written | EidCookieSyncOutcome::WrittenWithDeferredFreshness - )), - conflict_duplicate: u8::from(matches!(outcome, EidCookieSyncOutcome::ConflictMatched)), - deferred: u8::from(matches!( - outcome, - EidCookieSyncOutcome::WrittenWithDeferredFreshness - | EidCookieSyncOutcome::DeferredConflict - | EidCookieSyncOutcome::DeferredFreshness - | EidCookieSyncOutcome::DeferredStaleRead - )), - } + record: KvPartnerId::browser(update.uid.as_str(), update.acquired_at), + }); } } +/// Logs the one remaining `EID sync measurement` line: a new EC whose +/// create carried the browser ingest. Returning visitors log the +/// `identity request measurement` line instead. fn record_eid_sync_terminal(source: EidSyncSource, outcome: EidCookieSyncOutcome) { - let measurement = EidSyncMeasurement::new(source, outcome); log::info!( - "EID sync measurement: source={} outcome={} attempted=1 already_matched={} written={} \ - conflict_duplicate={} deferred={}", - measurement.source, - measurement.outcome, - measurement.already_matched, - measurement.written, - measurement.conflict_duplicate, - measurement.deferred, - ); -} - -fn reconcile_pull_sync_marker( - settings: &Settings, - registry: &PartnerRegistry, - ec_context: &mut EcContext, - response: &mut Response, -) { - let ec_id = ec_context.ec_value().map(str::to_owned); - let snapshot = ec_context.kv_snapshot().clone(); - reconcile_marker( - settings, - registry, - ec_id.as_deref(), - &snapshot, - ec_context.pull_sync_marker_mut(), - response, + "EID sync measurement: source={source} outcome={outcome} attempted=1 already_matched=0 \ + written=1 conflict_duplicate=0 deferred=0", ); } @@ -330,6 +347,9 @@ fn recover_orphaned_ec( generation: None, }; ec_context.replace_with_generated(ec_id, snapshot); + // The create carried this request's ingest: it is the page + // load's one write, so phase 4 has nothing left to flush. + *ec_context.identity_mut().mutation_mut() = IdentityMutation::default(); set_ec_cookie_on_response(settings, ec_context, response); return; } @@ -380,9 +400,9 @@ fn confirm_then_recover_orphaned_ec( match confirmed { EcKvSnapshot::Present { .. } => { // The row became visible after the origin round trip: adopt it and - // merge any pending updates rather than rotating a valid identity. + // queue the ingest rather than rotating a valid identity. ec_context.set_kv_snapshot(confirmed); - sync_eid_cookie_updates(graph, ec_context, ec_id, updates, EidSyncSource::Navigation); + queue_browser_ingest(ec_context, ec_id, updates); } EcKvSnapshot::Missing { .. } => match graph.key_exists_confirmed(ec_id) { Ok(false) => recover_orphaned_ec(settings, ec_context, graph, updates, response), @@ -477,35 +497,69 @@ fn finalize_unusable_consent( // for subsequent EC behavior. if let Some(graph) = kv { apply_withdrawal_tombstones(&ids_to_withdraw, |ec_id| { - // The graph hands back the post-withdrawal snapshot rather than - // leaving the caller to rebuild it, so post-send work that reads - // the context — pull sync discloses the raw EC ID to partners — - // sees the tombstone that was just written. Only the active ID has - // a snapshot in the context to correct. - let initial = if ec_context.kv_snapshot().belongs_to(ec_id) { - ec_context.kv_snapshot().clone() - } else { - EcKvSnapshot::NotRead - }; - let outcome = graph.tombstone_existing_from_snapshot(ec_id, initial); // The browser cookie is already cleared, so a failed tombstone // leaves a live row that server-side consumers still read as // consented. Report every failure, including the non-active cookie // ID whose outcome is not retained on the request context. - if matches!(outcome, EcKvSnapshot::Failed { .. }) { + if !tombstone_withdrawn_id(ec_context, graph, ec_id) { log::warn!( "EC withdrawal tombstone failed for '{}': the identity-graph row may \ still be live with consent granted", log_id(ec_id) ); - } - if ec_context.ec_value() == Some(ec_id) { - ec_context.set_kv_snapshot(outcome); + ec_context.record_failed_withdrawal(ec_id); } }); } } +/// Runs the tombstone path once for `ec_id` and returns whether it succeeded. +/// +/// The graph hands back the post-withdrawal snapshot rather than leaving the +/// caller to rebuild it, so post-send work that reads the context — pull sync +/// discloses the raw EC ID to partners — sees the tombstone that was just +/// written. Only the active ID has a snapshot in the context to correct. +fn tombstone_withdrawn_id( + ec_context: &mut EcContext, + graph: &KvIdentityGraph, + ec_id: &str, +) -> bool { + let initial = if ec_context.kv_snapshot().belongs_to(ec_id) { + ec_context.kv_snapshot().clone() + } else { + EcKvSnapshot::NotRead + }; + let outcome = graph.tombstone_existing_from_snapshot(ec_id, initial); + let succeeded = !matches!(outcome, EcKvSnapshot::Failed { .. }); + if ec_context.ec_value() == Some(ec_id) { + ec_context.set_kv_snapshot(outcome); + } + succeeded +} + +/// Retries each withdrawal tombstone that failed before send, once, after send. +/// +/// Each failed ID re-reads its row (a [`EcKvSnapshot::NotRead`] start) and runs +/// the same tombstone path as before send, with its own CAS attempts and +/// completion marker. This is the last attempt: a success clears the ID from +/// the context's failed list, and a second failure is logged and leaves it +/// there for the failure counters. +/// An ID that was never recorded as failed is not touched. +pub fn retry_failed_withdrawals(ec_context: &mut EcContext, graph: &KvIdentityGraph) { + let failed = ec_context.failed_withdrawals().to_vec(); + for ec_id in &failed { + if tombstone_withdrawn_id(ec_context, graph, ec_id) { + ec_context.clear_failed_withdrawal(ec_id); + } else { + log::warn!( + "EC withdrawal tombstone failed again after send for '{}': the identity-graph \ + row may still be live with consent granted", + log_id(ec_id) + ); + } + } +} + fn withdrawal_ec_ids(ec_context: &EcContext) -> HashSet { let mut hashes = HashSet::new(); @@ -542,11 +596,48 @@ mod tests { use super::*; use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::{ConsentContext, ConsentSource}; - use crate::openrtb::{Eid, Uid}; + use crate::ec::identity::request::WriterDecision; use crate::redacted::Redacted; use crate::settings::EcPartner; use crate::test_support::tests::{create_test_settings, purpose_one_only_consent}; + #[test] + fn finalize_creates_a_deferred_ec_its_navigation_did_not() { + let settings = create_test_settings(); + let graph = KvIdentityGraph::in_memory("ec-store"); + let mut context = + EcContext::new_for_test_with_ip(None, granting_consent(), Some("192.0.2.5".to_owned())); + context + .mint_deferred(&settings, &BrowserSeed::empty()) + .expect("should mint"); + let mut response = empty_response(); + + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &PartnerRegistry::empty(), + None, + None, + &mut response, + ); + + let ec_id = context.ec_value().expect("should keep the EC").to_owned(); + assert!( + graph.get(&ec_id).expect("should read").is_some(), + "should create the row" + ); + assert!( + response + .headers() + .get_all(http::header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .any(|cookie| cookie.starts_with(&format!("ts-ec={ec_id};"))), + "should set the EC cookie once its row exists" + ); + } + fn empty_response() -> Response { Response::builder() .status(200) @@ -625,8 +716,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -1125,7 +1217,7 @@ mod tests { Some("192.0.2.11".to_owned()), ); ec_context.set_recovery_eligible(true); - ec_context.set_eid_sync_source(EidSyncSource::Navigation); + set_writer(&mut ec_context, true); let partners = vec![make_partner("id5-sync.com")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1165,58 +1257,6 @@ mod tests { ); } - #[test] - fn valid_marker_with_unread_snapshot_defers_orphan_recovery() { - let settings = create_test_settings(); - let orphaned_ec = sample_ec_id("orphn2"); - let consent = ConsentContext { - jurisdiction: Jurisdiction::NonRegulated, - source: ConsentSource::Cookie, - ..Default::default() - }; - let mut ec_context = EcContext::new_for_test_with_ip( - Some(orphaned_ec.clone()), - consent, - Some("192.0.2.10".to_owned()), - ); - ec_context.set_recovery_eligible(true); - ec_context.set_pull_sync_marker_for_test( - crate::ec::pull_sync_marker::PullSyncMarkerState::Valid { expires_at: 4_600 }, - ); - let mut partner = make_partner("pull.example.com"); - partner.pull_sync_enabled = true; - partner.pull_sync_url = Some("https://sync.example.com/pull".to_owned()); - partner.pull_sync_allowed_domains = vec!["sync.example.com".to_owned()]; - partner.ts_pull_token = Some(Redacted::new("pull-token".to_owned())); - let registry = PartnerRegistry::from_config(&[partner]).expect("should build registry"); - let graph = KvIdentityGraph::in_memory("test_store"); - let mut response = empty_response(); - - ec_finalize_response( - &settings, - &mut ec_context, - Some(&graph), - ®istry, - None, - None, - &mut response, - ); - - assert_eq!( - ec_context.ec_value(), - Some(orphaned_ec.as_str()), - "an unread snapshot should defer orphan rotation until marker expiry" - ); - assert!( - matches!(ec_context.kv_snapshot(), EcKvSnapshot::NotRead), - "a marker-skipped request should leave the snapshot unread" - ); - assert!( - response.headers().get(http::header::SET_COOKIE).is_none(), - "bounded orphan deferral should not rewrite browser identity state" - ); - } - #[test] fn finalize_returning_user_subresource_does_not_persist_eid_updates() { let settings = create_test_settings(); @@ -1257,7 +1297,7 @@ mod tests { } #[test] - fn finalize_navigation_routes_persist_returning_user_eid_updates() { + fn finalize_navigation_routes_never_write_returning_user_eid_updates_directly() { for (source, suffix, cookie_id) in [ (EidSyncSource::Navigation, "naveid", "navigation-cookie-id"), (EidSyncSource::PageBids, "spaeid", "page-bids-cookie-id"), @@ -1276,6 +1316,7 @@ mod tests { .expect("should seed the live row"); let mut ec_context = returning_user_context(&ec_id, graph.load_snapshot(&ec_id), true); ec_context.set_eid_sync_source(source); + set_writer(&mut ec_context, true); let partners = vec![make_partner("sharedid.org")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); let mut response = empty_response(); @@ -1294,31 +1335,27 @@ mod tests { .get(&ec_id) .expect("should read store") .expect("row should remain"); + assert!( + !stored.ids.contains_key("sharedid.org"), + "{source} should leave the flush to phase 4" + ); assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some(cookie_id), - "{source} should persist the returning-user EID cookie" + ec_context.identity().mutation().change_count(), + 1, + "{source} should queue the returning-user EID cookie" ); } } #[test] - fn finalize_generated_ec_without_sync_source_ingests_the_ts_eids_cookie() { + fn generate_if_needed_seeds_the_ts_eids_cookie() { let settings = create_test_settings(); - let ec_id = sample_ec_id("genck1"); let graph = KvIdentityGraph::in_memory("test_store"); - let live = KvEntry::new( - &granting_consent(), + let mut ec_context = EcContext::new_for_test_with_ip( None, - current_timestamp(), - &settings.publisher.domain, + granting_consent(), + Some("192.0.2.10".to_owned()), ); - graph - .create(&ec_id, &live) - .expect("should seed generated row"); - let mut ec_context = - make_context(Some(&ec_id), None, false, true, Jurisdiction::NonRegulated); - ec_context.set_kv_snapshot(graph.load_snapshot(&ec_id)); let registry = PartnerRegistry::from_config(&[make_partner("ids.example.com")]) .expect("should build registry"); let eids_cookie = base64::engine::general_purpose::STANDARD.encode( @@ -1327,35 +1364,37 @@ mod tests { ])) .expect("should serialize test cookie payload"), ); - let mut response = empty_response(); - - ec_finalize_response( + let seed = browser_seed( &settings, - &mut ec_context, - Some(&graph), ®istry, + ec_context.consent(), Some(&eids_cookie), None, - &mut response, + current_timestamp(), ); + ec_context + .generate_if_needed(&settings, Some(&graph), &seed) + .expect("should generate an EC"); + + let ec_id = ec_context.ec_value().expect("should hold the new EC"); let (stored, _) = graph - .get(&ec_id) + .get(ec_id) .expect("should read store") - .expect("row should remain"); + .expect("row should exist"); assert_eq!( stored .ids .iter() - .map(|(source, id)| (source.as_str(), id.uid.as_str())) + .map(|(source, id)| (source.as_str(), id.first_uid().unwrap_or_default())) .collect::>(), vec![("ids.example.com", "cookie-uid")], - "a new EC with no sync source should ingest the cookie" + "a new EC should carry the ts-eids cookie in its create" ); } #[test] - fn finalize_generated_ec_persists_eid_updates() { + fn finalize_generated_ec_writes_no_ingest_of_its_own() { let settings = create_test_settings(); let ec_id = sample_ec_id("geneid"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1389,9 +1428,13 @@ mod tests { .get(&ec_id) .expect("should read store") .expect("row should remain"); - assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some("generated-cookie-id") + assert!( + !stored.ids.contains_key("sharedid.org"), + "finalize should leave a new EC's ingest to the create" + ); + assert!( + get_header(&response, "set-cookie").is_some(), + "should still set the ts-ec cookie" ); } @@ -1403,99 +1446,23 @@ mod tests { EidSyncSource::PageBids, EidSyncSource::NewEc, ]; - let outcomes = [ - EidCookieSyncOutcome::AlreadyMatched, - EidCookieSyncOutcome::Written, - EidCookieSyncOutcome::WrittenWithDeferredFreshness, - EidCookieSyncOutcome::ConflictMatched, - EidCookieSyncOutcome::DeferredConflict, - EidCookieSyncOutcome::DeferredFreshness, - EidCookieSyncOutcome::DeferredStaleRead, - EidCookieSyncOutcome::Missing, - EidCookieSyncOutcome::ConsentWithdrawn, - EidCookieSyncOutcome::Failed, - ]; assert_eq!( sources.map(|source| source.to_string()), - ["navigation", "auction", "page_bids", "new_ec"] - ); - assert_eq!( - outcomes.map(|outcome| outcome.to_string()), - [ - "already_matched", - "written", - "written_with_deferred_freshness", - "conflict_matched", - "deferred_conflict", - "deferred_freshness", - "deferred_stale_read", - "missing", - "consent_withdrawn", - "failed", - ] - ); - - assert_eq!( - EidSyncMeasurement::new( - EidSyncSource::Navigation, - EidCookieSyncOutcome::AlreadyMatched, - ), - EidSyncMeasurement { - source: EidSyncSource::Navigation, - outcome: EidCookieSyncOutcome::AlreadyMatched, - already_matched: 1, - written: 0, - conflict_duplicate: 0, - deferred: 0, - } - ); - assert_eq!( - EidSyncMeasurement::new( - EidSyncSource::Auction, - EidCookieSyncOutcome::WrittenWithDeferredFreshness, - ), - EidSyncMeasurement { - source: EidSyncSource::Auction, - outcome: EidCookieSyncOutcome::WrittenWithDeferredFreshness, - already_matched: 0, - written: 1, - conflict_duplicate: 0, - deferred: 1, - } - ); - assert_eq!( - EidSyncMeasurement::new(EidSyncSource::NewEc, EidCookieSyncOutcome::ConflictMatched,), - EidSyncMeasurement { - source: EidSyncSource::NewEc, - outcome: EidCookieSyncOutcome::ConflictMatched, - already_matched: 0, - written: 0, - conflict_duplicate: 1, - deferred: 0, - } + ["navigation", "auction", "page_bids", "new_ec"], + "should label each source with a bounded name" ); assert_eq!( - EidSyncMeasurement::new( - EidSyncSource::PageBids, - EidCookieSyncOutcome::DeferredStaleRead, - ), - EidSyncMeasurement { - source: EidSyncSource::PageBids, - outcome: EidCookieSyncOutcome::DeferredStaleRead, - already_matched: 0, - written: 0, - conflict_duplicate: 0, - deferred: 1, - } + EidCookieSyncOutcome::Written.to_string(), + "written", + "should label the one outcome the new-EC line reports" ); } #[test] - fn finalize_auction_transient_miss_still_persists_eid_updates() { - // `/auction` saves its first lookup into the context and is never - // recovery eligible, so a stale miss there has no later chance to - // retry. Finalization must revalidate before dropping collected IDs. + fn finalize_auction_stale_miss_neither_ingests_nor_writes() { + // `/auction` is a stager: even with a stale endpoint miss in its + // snapshot it does not re-read, ingest or write. let settings = create_test_settings(); let ec_id = sample_ec_id("named1"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1535,93 +1502,13 @@ mod tests { .get(&ec_id) .expect("should read store") .expect("row should remain"); - assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some("shared-cookie-id"), - "a stale endpoint miss must not suppress EID persistence" - ); - } - - #[test] - fn finalize_persists_every_configured_partner_from_client_eids_over_a_trimmed_cookie() { - // Regression test for #1184: the `ts-eids` cookie only carries what - // the browser could fit under its size cap, but `/auction` also - // hands finalization the full EID set from the request body via - // `EcContext::set_client_eids`. That full set must land in KV even - // when the cookie alone would have dropped a configured partner. - let settings = create_test_settings(); - let ec_id = sample_ec_id("cleids1"); - let graph = KvIdentityGraph::in_memory("test_store"); - let live = KvEntry::new( - &granting_consent(), - None, - current_timestamp(), - &settings.publisher.domain, - ); - graph - .create(&ec_id, &live) - .expect("should seed the live row this request updates"); - let mut ec_context = returning_user_context( - &ec_id, - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - false, - ); - ec_context.set_eid_sync_source(EidSyncSource::Auction); - // Only `id5-sync.com` "fit" in the (simulated) trimmed cookie; - // `liveramp.com` was dropped by the browser's size cap. - let eids_cookie = base64::engine::general_purpose::STANDARD.encode( - serde_json::to_vec(&serde_json::json!([ - {"source": "id5-sync.com", "uids": [{"id": "ID5_from_cookie", "atype": 1}]} - ])) - .expect("should serialize test cookie payload"), - ); - ec_context.set_client_eids(vec![ - Eid::new( - "id5-sync.com".to_owned(), - vec![Uid { - id: "ID5_from_body".to_owned(), - atype: Some(1), - ext: None, - }], - ), - Eid::new( - "liveramp.com".to_owned(), - vec![Uid { - id: "LR_from_body".to_owned(), - atype: Some(3), - ext: None, - }], - ), - ]); - let partners = vec![make_partner("id5-sync.com"), make_partner("liveramp.com")]; - let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); - let mut response = empty_response(); - - ec_finalize_response( - &settings, - &mut ec_context, - Some(&graph), - ®istry, - Some(&eids_cookie), - None, - &mut response, - ); - - let (stored, _) = graph - .get(&ec_id) - .expect("should read store") - .expect("row should exist after ingestion"); - assert_eq!( - stored.ids.get("id5-sync.com").map(|id| id.uid.as_str()), - Some("ID5_from_body"), - "the request body's EID should win over the cookie's stale value" + assert!( + !stored.ids.contains_key("sharedid.org"), + "a stager must not write ingest" ); - assert_eq!( - stored.ids.get("liveramp.com").map(|id| id.uid.as_str()), - Some("LR_from_body"), - "a partner the cookie trimmed must still be ingested from the body" + assert!( + ec_context.identity().mutation().is_empty(), + "a stager must not queue ingest" ); } @@ -1652,18 +1539,14 @@ mod tests { false, purpose1_only_consent, ); - ec_context.set_kv_snapshot(EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }); - ec_context.set_eid_sync_source(EidSyncSource::Auction); - ec_context.set_client_eids(vec![Eid::new( - "id5-sync.com".to_owned(), - vec![Uid { - id: "ID5_should_not_persist".to_owned(), - atype: Some(1), - ext: None, - }], - )]); + ec_context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + set_writer(&mut ec_context, true); + let eids_cookie = base64::engine::general_purpose::STANDARD.encode( + serde_json::to_vec(&serde_json::json!([ + { "source": "id5-sync.com", "uids": [{ "id": "ID5_should_not_persist", "atype": 1 }] } + ])) + .expect("should serialize test cookie payload"), + ); let partners = vec![make_partner("id5-sync.com")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); let mut response = empty_response(); @@ -1673,7 +1556,7 @@ mod tests { &mut ec_context, Some(&graph), ®istry, - None, + Some(&eids_cookie), None, &mut response, ); @@ -1687,6 +1570,10 @@ mod tests { "denying TCF Purpose 4 must keep the EID out of KV even though Purpose 1 \ (EC) consent is granted" ); + assert!( + ec_context.identity().mutation().is_empty(), + "denying TCF Purpose 4 must keep the EID out of the mutation" + ); } #[test] @@ -2043,6 +1930,18 @@ mod tests { ec } + fn set_writer(ec_context: &mut EcContext, writer: bool) { + ec_context + .identity_mut() + .set_writer_decision(WriterDecision { + writer, + is_real_browser: writer, + user_agent: None, + dnt: false, + page_url: None, + }); + } + fn assert_did_not_rotate(ec_context: &EcContext, orphan: &str, response: &Response) { assert_eq!( ec_context.ec_value(), @@ -2315,9 +2214,6 @@ mod tests { }; let mut ec_context = make_context_with_consent(Some(&ec_id), Some(&ec_id), true, false, consent); - ec_context.set_pull_sync_marker_for_test( - crate::ec::pull_sync_marker::PullSyncMarkerState::Invalid, - ); let graph = KvIdentityGraph::failing("unavailable-store"); let mut response = empty_response(); @@ -2348,71 +2244,10 @@ mod tests { .any(|cookie| { cookie.starts_with("ts-ec=;") && cookie.contains("Max-Age=0") }), "KV failure should not prevent EC cookie deletion" ); - assert!( - cookies.iter().any(|cookie| { - cookie.starts_with("ts-ec-pull-complete=;") && cookie.contains("Max-Age=0") - }), - "KV failure should not prevent marker deletion" - ); } #[test] - fn finalize_sets_marker_for_complete_pull_partner_snapshot() { - let settings = create_test_settings(); - let ec_id = sample_ec_id("compl1"); - let mut partner = make_partner("ssp.example.com"); - partner.pull_sync_enabled = true; - partner.pull_sync_url = Some("https://sync.example.com/pull".to_owned()); - partner.pull_sync_allowed_domains = vec!["sync.example.com".to_owned()]; - partner.ts_pull_token = Some(Redacted::new("pull-token".to_owned())); - let registry = PartnerRegistry::from_config(&[partner]).expect("should build registry"); - let mut ec_context = make_context( - Some(&ec_id), - Some(&ec_id), - true, - false, - Jurisdiction::NonRegulated, - ); - let mut entry = live_entry(); - entry.ids.insert( - "ssp.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "partner-uid".to_owned(), - }, - ); - ec_context.set_kv_snapshot(EcKvSnapshot::Present { - ec_id, - entry: Box::new(entry), - generation: Some(1), - }); - let mut response = empty_response(); - - ec_finalize_response( - &settings, - &mut ec_context, - None, - ®istry, - None, - None, - &mut response, - ); - - let cookies = response - .headers() - .get_all(http::header::SET_COOKIE) - .iter() - .filter_map(|value| value.to_str().ok()) - .collect::>(); - assert!( - cookies - .iter() - .any(|cookie| cookie.starts_with("ts-ec-pull-complete=v1.")), - "complete snapshot should issue the marker" - ); - } - - #[test] - fn explicit_withdrawal_without_marker_or_ec_cookie_does_not_set_cookie() { + fn explicit_withdrawal_without_ec_cookie_does_not_set_cookie() { let settings = create_test_settings(); let consent = ConsentContext { jurisdiction: Jurisdiction::UsState("CA".to_owned()), @@ -2439,47 +2274,992 @@ mod tests { ); } - #[test] - fn explicit_withdrawal_expires_marker_without_ec_cookie() { - let settings = create_test_settings(); - let consent = ConsentContext { - jurisdiction: Jurisdiction::UsState("CA".to_owned()), - gpc: true, - source: ConsentSource::Cookie, - ..Default::default() + mod identity_flow { + use serde_json::json; + + use super::*; + use crate::ec::identity::catalog::ProviderCatalog; + use crate::ec::identity::ids::SourceDomain; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::mutation::{ + ApplyContext, IdentityMutation, PartnerIdChange, apply, }; - let mut ec_context = make_context_with_consent(None, None, false, false, consent); - ec_context.set_pull_sync_marker_for_test( - crate::ec::pull_sync_marker::PullSyncMarkerState::Invalid, - ); - let mut response = empty_response(); + use crate::ec::kv_types::KvPartnerId; - ec_finalize_response( - &settings, - &mut ec_context, - None, - &PartnerRegistry::empty(), - None, - None, - &mut response, - ); + const SHARED_ID_SOURCE: &str = "sharedid.org"; - let cookies = response - .headers() - .get_all(http::header::SET_COOKIE) - .iter() - .filter_map(|value| value.to_str().ok()) - .collect::>(); - assert!( - cookies.iter().any(|cookie| { - cookie.starts_with("ts-ec-pull-complete=;") && cookie.contains("Max-Age=0") - }), - "withdrawal should expire the marker independently of EC cookie state" - ); - assert!( - cookies.iter().all(|cookie| !cookie.starts_with("ts-ec=;")), - "missing EC cookie should not add an EC-cookie expiry" - ); + fn settings_with_partner(_source: &str) -> Settings { + create_test_settings() + } + + fn registry_with_partner(source: &str) -> PartnerRegistry { + PartnerRegistry::from_config(&[make_partner(source)]).expect("should build registry") + } + + fn ts_eids_cookie(value: &serde_json::Value) -> String { + base64::engine::general_purpose::STANDARD + .encode(serde_json::to_vec(value).expect("should serialize test cookie payload")) + } + + fn queued_records(mutation: &IdentityMutation) -> Vec<(SourceDomain, KvPartnerId)> { + mutation + .positive_changes() + .iter() + .filter_map(|change| match change { + PartnerIdChange::FillMissing { source, record } + | PartnerIdChange::OwnerWrite { source, record, .. } => { + Some((source.clone(), record.clone())) + } + PartnerIdChange::OwnerRemove { .. } | PartnerIdChange::WriterRemove { .. } => { + None + } + }) + .collect() + } + + fn set_cookie_names(response: &Response) -> Vec { + response + .headers() + .get_all(http::header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .filter_map(|cookie| cookie.split('=').next()) + .map(str::to_owned) + .collect() + } + + fn html_response() -> Response { + Response::builder() + .status(200) + .header(http::header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build test response") + } + + fn seed_live_row(graph: &KvIdentityGraph, entry: &KvEntry) -> String { + let ec_id = sample_ec_id("idflow"); + graph + .create(&ec_id, entry) + .expect("should seed the live row"); + ec_id + } + + fn live_row() -> KvEntry { + KvEntry::minimal("other.example.com", "x", 1) + } + + fn returning_context(ec_id: &str, writer: bool) -> EcContext { + let mut context = returning_user_context(ec_id, EcKvSnapshot::NotRead, false); + set_writer(&mut context, writer); + context + } + + fn finalize_with_shared_id( + context: &mut EcContext, + graph: &KvIdentityGraph, + shared_id: Option<&str>, + ) -> Response { + let mut response = html_response(); + ec_finalize_response( + &settings_with_partner(SHARED_ID_SOURCE), + context, + Some(graph), + ®istry_with_partner(SHARED_ID_SOURCE), + None, + shared_id, + &mut response, + ); + response + } + + #[test] + fn returning_writer_navigation_adds_ingest_to_the_mutation_and_writes_nothing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &live_row()); + store.reset_counts(); + let mut context = returning_context(&ec_id, true); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + + let response = finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + assert_eq!( + store.writes().total(), + 0, + "should not write in finalize for a returning writer" + ); + let records = queued_records(context.identity().mutation()); + assert_eq!(records.len(), 1, "should queue the sharedId fill"); + assert_eq!( + records[0].1.first_uid(), + Some("example-shared-id"), + "should queue the cookie's UID" + ); + assert!( + set_cookie_names(&response).is_empty(), + "should set no cookie on a returning HTML response" + ); + } + + #[test] + fn writer_ingest_applied_to_the_live_row_matches_the_old_direct_write() { + let (graph, _store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &live_row()); + let mut context = returning_context(&ec_id, true); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + let eids_cookie = ts_eids_cookie(&json!([ + { "source": SHARED_ID_SOURCE, "uids": [{ "id": "example-prebid" }] } + ])); + let mut response = html_response(); + let now = checked_current_timestamp().expect("should read the clock"); + + ec_finalize_response( + &settings_with_partner(SHARED_ID_SOURCE), + &mut context, + Some(&graph), + ®istry_with_partner(SHARED_ID_SOURCE), + Some(&eids_cookie), + Some("example-shared-id"), + &mut response, + ); + + let live = live_row(); + let mut direct = live.clone(); + apply_partner_id_updates( + &mut direct, + &[PartnerIdUpdate::new(SHARED_ID_SOURCE, "example-shared-id") + .with_acquired_at(now)], + ); + let catalog = ProviderCatalog::empty(); + let random = SeededRandom::new(7); + let applied = apply( + context.identity().mutation(), + &live, + &ApplyContext { + now, + random: &random, + catalog: &catalog, + }, + ) + .expect("should apply to a live row"); + assert!(applied.changed, "should change the live row"); + assert_eq!( + applied.entry.ids.keys().collect::>(), + direct.ids.keys().collect::>(), + "should store the same sources as the old direct write" + ); + let staged = applied + .entry + .ids + .get(SHARED_ID_SOURCE) + .expect("should store the source"); + let written = direct + .ids + .get(SHARED_ID_SOURCE) + .expect("should store the source"); + assert_eq!( + staged.first_uid(), + written.first_uid(), + "should store the same UID" + ); + assert_eq!( + staged.at, written.at, + "should store the same acquisition time" + ); + } + + #[test] + fn golden_browser_ingest_stays_one_uid_whatever_the_browser_sent() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &live_row()); + store.reset_counts(); + let mut context = returning_context(&ec_id, true); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + let eids_cookie = ts_eids_cookie(&json!([ + { + "source": "partner.example.com", + "uids": [ + { "id": "example-a", "atype": 1, "ext": { "stype": "ppuid" } }, + { "id": "example-b", "atype": 3 } + ], + "inserter": "example.com", + "mm": 3 + }, + { "source": "unregistered.example.com", "uids": [{ "id": "example-c", "atype": 1 }] } + ])); + let mut response = html_response(); + + ec_finalize_response( + &settings_with_partner("partner.example.com"), + &mut context, + Some(&graph), + ®istry_with_partner("partner.example.com"), + Some(&eids_cookie), + None, + &mut response, + ); + + assert_eq!(store.writes().total(), 0, "should not write in finalize"); + let records = queued_records(context.identity().mutation()); + assert_eq!( + records.len(), + 1, + "should queue only the registered source, as today" + ); + let (source, record) = &records[0]; + assert_eq!( + source.as_str(), + "partner.example.com", + "should queue the registered source" + ); + assert_eq!( + serde_json::to_value(record).expect("should serialize the record"), + json!({ "at": record.at, "eids": [{ "uids": [{ "id": "example-a" }] }] }), + "should store the UID collect_eid_updates picks today, with no atype, ext or EID-level field" + ); + } + + #[test] + fn stagers_ingest_nothing_and_write_nothing() { + for source in [ + Some(EidSyncSource::Auction), + Some(EidSyncSource::PageBids), + None, + ] { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &live_row()); + store.reset_counts(); + let mut context = returning_context(&ec_id, false); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + if let Some(source) = source { + context.set_eid_sync_source(source); + } + + let response = + finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + assert_eq!( + store.writes().total(), + 0, + "{source:?} should not write in finalize" + ); + assert!( + context.identity().mutation().is_empty(), + "{source:?} should leave the mutation empty" + ); + assert!( + set_cookie_names(&response).is_empty(), + "{source:?} should set no cookie" + ); + } + } + + #[test] + fn unusable_consent_writes_nothing_and_queues_nothing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &live_row()); + store.reset_counts(); + let mut context = make_context( + Some(&ec_id), + Some(&ec_id), + true, + false, + Jurisdiction::Unknown, + ); + set_writer(&mut context, true); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + + finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + assert_eq!(store.writes().total(), 0, "should not write"); + assert!( + context.identity().mutation().is_empty(), + "should queue nothing" + ); + } + + #[test] + fn withdrawal_still_writes_its_tombstone_on_any_request() { + for writer in [true, false] { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &live_row()); + store.reset_counts(); + let consent = ConsentContext { + jurisdiction: Jurisdiction::UsState("CA".to_owned()), + gpc: true, + source: ConsentSource::Cookie, + ..Default::default() + }; + let mut context = + make_context_with_consent(Some(&ec_id), Some(&ec_id), true, false, consent); + set_writer(&mut context, writer); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + + finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + assert!( + store.writes().total() > 0, + "writer={writer} should write the tombstone" + ); + let (stored, _) = graph + .get(&ec_id) + .expect("should read store") + .expect("row should remain"); + assert!( + !stored.consent.ok, + "writer={writer} should tombstone the row" + ); + assert!( + context.identity().mutation().is_empty(), + "writer={writer} should queue nothing" + ); + } + } + + #[test] + fn tombstone_read_leaves_the_mutation_empty() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &KvEntry::tombstone(1000)); + store.reset_counts(); + let mut context = returning_context(&ec_id, true); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + + finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + assert_eq!(store.writes().total(), 0, "should not write"); + assert!( + context.identity().mutation().is_empty(), + "should leave the mutation empty" + ); + } + + #[test] + fn orphan_rotation_carries_browser_ingest_in_one_create() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let orphan = sample_ec_id("orphn2"); + let mut context = returning_context(&orphan, true); + context.set_recovery_eligible(true); + context.set_kv_snapshot(EcKvSnapshot::Missing { + ec_id: orphan.clone(), + }); + + let response = finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + let replacement = context.ec_value().expect("should keep an EC ID").to_owned(); + assert_ne!(replacement, orphan, "should rotate the orphan"); + let writes = store.writes(); + assert_eq!(writes.add, 1, "should write one create"); + assert_eq!(writes.total(), 1, "should write nothing else"); + let (stored, _) = graph + .get(&replacement) + .expect("should read store") + .expect("should hold the replacement row"); + assert_eq!( + stored + .ids + .get(SHARED_ID_SOURCE) + .and_then(KvPartnerId::first_uid), + Some("example-shared-id"), + "should carry the sharedId as a one-UID record" + ); + assert!( + context.identity().mutation().is_empty(), + "should leave the mutation empty" + ); + assert_eq!( + set_cookie_names(&response), + vec!["ts-ec".to_owned()], + "should set only the replacement cookie" + ); + } + + #[test] + fn confirm_read_that_finds_the_row_queues_ingest_instead_of_writing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, &live_row()); + store.reset_counts(); + let mut context = returning_context(&ec_id, true); + context.set_recovery_eligible(true); + context.set_kv_snapshot(EcKvSnapshot::Missing { + ec_id: ec_id.clone(), + }); + + let response = finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + assert_eq!(store.writes().total(), 0, "should not write"); + assert_eq!( + context.ec_value(), + Some(ec_id.as_str()), + "should not rotate" + ); + assert_eq!( + queued_records(context.identity().mutation()).len(), + 1, + "should queue the ingest" + ); + assert!( + context.kv_snapshot().entry_for(&ec_id).is_some(), + "should adopt the confirmed row as the snapshot" + ); + assert!( + set_cookie_names(&response).is_empty(), + "should set no cookie" + ); + } + + #[test] + fn integration_proxy_writer_never_rotates_on_missing_row() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let orphan = sample_ec_id("orphn3"); + let mut context = returning_context(&orphan, true); + context.set_recovery_eligible(false); + context.set_kv_snapshot(EcKvSnapshot::Missing { + ec_id: orphan.clone(), + }); + + let response = finalize_with_shared_id(&mut context, &graph, Some("example-shared-id")); + + assert_eq!(store.writes().total(), 0, "should not create"); + assert_did_not_rotate(&context, &orphan, &response); + assert!( + context.identity().mutation().is_empty(), + "should queue nothing against a missing row" + ); + } + + #[test] + fn new_visitor_create_carries_browser_ingest_in_one_write() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let settings = create_test_settings(); + let registry = registry_with_partner(SHARED_ID_SOURCE); + let mut context = EcContext::new_for_test_with_ip( + None, + granting_consent(), + Some("192.0.2.10".to_owned()), + ); + let seed = browser_seed( + &settings, + ®istry, + context.consent(), + None, + Some("example-shared-id"), + 1_760_000_000, + ); + + context + .generate_if_needed(&settings, Some(&graph), &seed) + .expect("should generate an EC"); + let mut response = html_response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + None, + Some("example-shared-id"), + &mut response, + ); + + let writes = store.writes(); + assert_eq!(writes.add, 1, "should write one create"); + assert_eq!(writes.total(), 1, "should write nothing else"); + let ec_id = context.ec_value().expect("should hold the new EC"); + let (stored, _) = graph + .get(ec_id) + .expect("should read store") + .expect("should hold the new row"); + assert_eq!( + stored + .ids + .get(SHARED_ID_SOURCE) + .and_then(KvPartnerId::first_uid), + Some("example-shared-id"), + "should carry the sharedId as a one-UID record" + ); + assert_eq!( + set_cookie_names(&response), + vec!["ts-ec".to_owned()], + "should set the ts-ec cookie as today" + ); + } + } + + mod after_send_withdrawal_retry { + use super::*; + use crate::ec::kv::MAX_CAS_RETRIES; + + fn withdrawn_context(active: &str, cookie: &str) -> EcContext { + let consent = ConsentContext { + jurisdiction: Jurisdiction::UsState("CA".to_owned()), + gpc: true, + source: ConsentSource::Cookie, + ..Default::default() + }; + make_context_with_consent(Some(active), Some(cookie), true, false, consent) + } + + fn seed(graph: &KvIdentityGraph, ec_id: &str) { + graph + .create(ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + } + + fn finalize(context: &mut EcContext, graph: &KvIdentityGraph) { + let mut response = empty_response(); + ec_finalize_response( + &create_test_settings(), + context, + Some(graph), + &PartnerRegistry::empty(), + None, + None, + &mut response, + ); + } + + fn is_tombstone(graph: &KvIdentityGraph, ec_id: &str) -> bool { + graph + .get(ec_id) + .expect("should read the row") + .is_some_and(|(entry, _)| !entry.consent.ok) + } + + #[test] + fn failed_withdrawal_is_recorded_for_after_send() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("fail01"); + seed(&graph, &ec_id); + store.refuse_conditional_writes(&ec_id, usize::MAX); + let mut context = withdrawn_context(&ec_id, &ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + + finalize(&mut context, &graph); + + assert_eq!( + context.failed_withdrawals(), + std::slice::from_ref(&ec_id), + "should record the active ID whose tombstone failed" + ); + assert!( + !is_tombstone(&graph, &ec_id), + "the row should still be live" + ); + } + + #[test] + fn successful_withdrawal_records_nothing() { + let (graph, _store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("good01"); + seed(&graph, &ec_id); + let mut context = withdrawn_context(&ec_id, &ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + + finalize(&mut context, &graph); + + assert!( + context.failed_withdrawals().is_empty(), + "should record nothing for a tombstone that was written" + ); + assert!(is_tombstone(&graph, &ec_id), "should tombstone the row"); + } + + #[test] + fn nothing_failed_means_the_retry_touches_nothing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("none01"); + seed(&graph, &ec_id); + store.reset_counts(); + let mut context = withdrawn_context(&ec_id, &ec_id); + + retry_failed_withdrawals(&mut context, &graph); + + assert_eq!( + store.writes().total(), + 0, + "should write nothing of any mode" + ); + assert_eq!(store.writes().lookups, 0, "should read nothing"); + } + + #[test] + fn after_send_retry_writes_the_tombstone_once() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("retry1"); + seed(&graph, &ec_id); + store.refuse_conditional_writes(&ec_id, MAX_CAS_RETRIES as usize); + let mut context = withdrawn_context(&ec_id, &ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + finalize(&mut context, &graph); + assert_eq!( + context.failed_withdrawals(), + std::slice::from_ref(&ec_id), + "should record the failure before send" + ); + store.reset_counts(); + + retry_failed_withdrawals(&mut context, &graph); + + assert!(is_tombstone(&graph, &ec_id), "should read as a tombstone"); + assert!( + context.failed_withdrawals().is_empty(), + "should clear an ID whose retry succeeded" + ); + let writes = store.writes(); + assert_eq!( + writes.if_generation_match, 1, + "should write the tombstone once" + ); + assert_eq!(writes.add, 1, "should write the completion marker once"); + assert_eq!(writes.overwrite, 0, "should not overwrite"); + assert!( + context + .kv_snapshot() + .entry_for(&ec_id) + .is_some_and(|entry| !entry.consent.ok), + "should update the active snapshot to the tombstone" + ); + } + + #[test] + fn after_send_retry_is_the_last_attempt() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("last01"); + seed(&graph, &ec_id); + store.refuse_conditional_writes(&ec_id, usize::MAX); + let mut context = withdrawn_context(&ec_id, &ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + finalize(&mut context, &graph); + store.reset_counts(); + + retry_failed_withdrawals(&mut context, &graph); + + let writes = store.writes(); + assert_eq!( + writes.if_generation_match, MAX_CAS_RETRIES as usize, + "should make one run of the CAS attempts, never a second retry" + ); + assert_eq!(writes.add, 0, "should write no completion marker"); + assert_eq!(writes.overwrite, 0, "should not overwrite"); + assert_eq!( + context.failed_withdrawals(), + std::slice::from_ref(&ec_id), + "should leave the failure recorded for the counters" + ); + assert!( + !is_tombstone(&graph, &ec_id), + "the row should still be live" + ); + } + + #[test] + fn only_the_id_that_fails_twice_remains_after_the_retry() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let active = sample_ec_id("activ5"); + let cookie = sample_ec_id("cook5e"); + seed(&graph, &active); + seed(&graph, &cookie); + store.refuse_conditional_writes(&cookie, MAX_CAS_RETRIES as usize); + store.refuse_conditional_writes(&active, usize::MAX); + let mut context = withdrawn_context(&active, &cookie); + context.set_kv_snapshot(graph.load_snapshot(&active)); + finalize(&mut context, &graph); + assert_eq!( + context.failed_withdrawals().len(), + 2, + "should record both IDs before send" + ); + + retry_failed_withdrawals(&mut context, &graph); + + assert_eq!( + context.failed_withdrawals(), + std::slice::from_ref(&active), + "should keep only the ID that failed twice" + ); + assert!( + is_tombstone(&graph, &cookie), + "should withdraw the cookie row" + ); + assert!( + !is_tombstone(&graph, &active), + "the active row should be live" + ); + } + + #[test] + fn withdrawal_of_two_ids_retries_only_the_failed_one() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let active = sample_ec_id("activ4"); + let cookie = sample_ec_id("cook4e"); + seed(&graph, &active); + seed(&graph, &cookie); + store.refuse_conditional_writes(&cookie, MAX_CAS_RETRIES as usize); + let mut context = withdrawn_context(&active, &cookie); + context.set_kv_snapshot(graph.load_snapshot(&active)); + finalize(&mut context, &graph); + assert_eq!( + context.failed_withdrawals(), + std::slice::from_ref(&cookie), + "should record only the cookie ID" + ); + assert!( + is_tombstone(&graph, &active), + "the active row should be withdrawn" + ); + let active_generation = graph.get(&active).expect("should read").map(|row| row.1); + store.reset_counts(); + + retry_failed_withdrawals(&mut context, &graph); + + assert!( + is_tombstone(&graph, &cookie), + "should withdraw the cookie row" + ); + assert!( + context.failed_withdrawals().is_empty(), + "should hold no ID once the only failed one succeeds" + ); + assert_eq!( + graph.get(&active).expect("should read").map(|row| row.1), + active_generation, + "should not rewrite the active row" + ); + assert_eq!( + store.writes().if_generation_match, + 1, + "should write one tombstone, for the cookie ID only" + ); + } + } + + mod capture_revoke { + use super::*; + use crate::ec::identity::mutation::{IdentityMutation, PartnerIdChange}; + use crate::ec::identity::revision::Revision; + use crate::ec::identity::test_support::{pid, source}; + + fn finalize(context: &mut EcContext, graph: &KvIdentityGraph) -> Response { + let mut response = empty_response(); + ec_finalize_response( + &create_test_settings(), + context, + Some(graph), + &PartnerRegistry::empty(), + None, + None, + &mut response, + ); + response + } + + fn consenting_context(ec_id: &str) -> EcContext { + let mut context = make_context_with_consent( + Some(ec_id), + Some(ec_id), + true, + false, + granting_consent(), + ); + context + .identity_mut() + .mutation_mut() + .add_change(PartnerIdChange::OwnerRemove { + source: source("a.example.com"), + provider: pid("lockr"), + expected: Revision::Unset, + }); + context + } + + fn expires_the_ec_cookie(response: &Response) -> bool { + response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .any(|cookie| cookie.starts_with("ts-ec=") && cookie.contains("Max-Age=0")) + } + + #[test] + fn capture_global_revoke_from_a_real_browser_tombstones_the_row_and_clears_the_ec_cookie() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("revoke"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.reset_counts(); + let mut context = consenting_context(&ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.request_global_revoke(); + + let response = finalize(&mut context, &graph); + + let (stored, _) = graph + .get(&ec_id) + .expect("should read the row") + .expect("should keep the tombstone"); + assert!(!stored.consent.ok, "should tombstone the row"); + assert!(stored.ids.is_empty(), "should remove every ID"); + assert_eq!( + store.writes().if_generation_match, + 1, + "should write one tombstone" + ); + assert!( + expires_the_ec_cookie(&response), + "should expire the EC cookie" + ); + assert!( + context.failed_withdrawals().is_empty(), + "should record no failed tombstone" + ); + assert_eq!( + context.identity().mutation(), + &IdentityMutation::default(), + "should drop the request's mutation" + ); + + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.reset_counts(); + let mut control = consenting_context(&ec_id); + control.set_kv_snapshot(graph.load_snapshot(&ec_id)); + let response = finalize(&mut control, &graph); + assert!( + graph + .get(&ec_id) + .expect("should read the row") + .is_some_and(|(row, _)| row.consent.ok), + "control: without a revoke the row stays live" + ); + assert!( + !expires_the_ec_cookie(&response), + "control: without a revoke the cookie stays" + ); + assert_eq!(store.writes().total(), 0, "control: writes nothing"); + } + + fn revoke_marker_cookie(response: &Response) -> Option { + response + .headers() + .get_all(http::header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .find(|cookie| cookie.starts_with("ts-ec-revoked=")) + .map(str::to_owned) + } + + #[test] + fn capture_global_revoke_sets_the_minting_marker() { + let graph = KvIdentityGraph::in_memory("ec-store"); + let ec_id = sample_ec_id("marker"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + let mut context = consenting_context(&ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.request_global_revoke(); + + let response = finalize(&mut context, &graph); + + let marker = revoke_marker_cookie(&response).expect("should set the marker"); + assert!( + marker.starts_with(&format!( + "ts-ec-revoked={};", + crate::ec::cookies::revoke_marker_value(context.consent()) + )), + "should hold the fingerprint of the consent the revoke saw" + ); + assert!( + marker.contains("HttpOnly") && marker.contains("Secure"), + "should be HttpOnly and Secure" + ); + + let mut control = consenting_context(&ec_id); + control.set_kv_snapshot(graph.load_snapshot(&ec_id)); + assert!( + revoke_marker_cookie(&finalize(&mut control, &graph)).is_none(), + "control: without a revoke no marker is set" + ); + } + + #[test] + fn minting_under_changed_consent_expires_a_stale_marker() { + let graph = KvIdentityGraph::in_memory("ec-store"); + let mut context = EcContext::new_for_test_with_ip( + None, + granting_consent(), + Some("192.0.2.5".to_owned()), + ); + context.set_revoke_marker_for_test("ffff"); + context + .generate_if_needed(&create_test_settings(), Some(&graph), &BrowserSeed::empty()) + .expect("should mint"); + + let response = finalize(&mut context, &graph); + + assert!( + revoke_marker_cookie(&response).is_some_and(|cookie| cookie.contains("Max-Age=0")), + "should expire the marker once a new EC is minted" + ); + } + + #[test] + fn capture_global_revoke_failed_tombstone_gets_the_after_send_run() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("rvfail"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.refuse_conditional_writes(&ec_id, 5); + let mut context = consenting_context(&ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.request_global_revoke(); + + let response = finalize(&mut context, &graph); + + assert_eq!( + context.failed_withdrawals(), + std::slice::from_ref(&ec_id), + "should record the failed tombstone for the after-send run" + ); + assert!( + expires_the_ec_cookie(&response), + "should still expire the cookie" + ); + + retry_failed_withdrawals(&mut context, &graph); + + assert!( + graph + .get(&ec_id) + .expect("should read the row") + .is_some_and(|(row, _)| !row.consent.ok), + "should tombstone the row after send" + ); + } + + #[test] + fn capture_global_revoke_without_an_ec_cookie_expires_nothing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("nocook"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.reset_counts(); + let mut context = + make_context_with_consent(Some(&ec_id), None, false, true, granting_consent()); + context.request_global_revoke(); + + let response = finalize(&mut context, &graph); + + assert!( + !expires_the_ec_cookie(&response), + "should expire no cookie the request did not carry" + ); + assert_eq!(store.writes().total(), 0, "should tombstone no row"); + } } fn live_entry() -> KvEntry { diff --git a/crates/trusted-server-core/src/ec/identify.rs b/crates/trusted-server-core/src/ec/identify.rs index c22a592461..2aaeaa5e55 100644 --- a/crates/trusted-server-core/src/ec/identify.rs +++ b/crates/trusted-server-core/src/ec/identify.rs @@ -16,14 +16,33 @@ use crate::openrtb::{Eid, Uid}; use crate::settings::Settings; use super::EcContext; +use super::checked_current_timestamp; +use super::identity::SourceDomain; +use super::identity::UnixTime; +use super::identity::catalog::ProviderCatalog; +use super::identity::policy::PolicySignals; +use super::identity::request::opted_out; +use super::identity::servable::{ServeContext, is_servable}; use super::kv::KvIdentityGraph; +use super::kv_types::KvPartnerId; use super::log_id; use super::registry::PartnerRegistry; +/// Whether the browser marked the request as a same-origin or same-site +/// fetch. A missing `Sec-Fetch-Site` header fails closed. +fn is_same_site_fetch(req: &Request) -> bool { + req.headers() + .get("sec-fetch-site") + .and_then(|value| value.to_str().ok()) + .is_some_and(|site| matches!(site, "same-origin" | "same-site")) +} + /// Handles `GET /_ts/api/v1/identify`. /// /// Requires Bearer token authentication. Returns only the requesting -/// partner's UID for the active EC ID. +/// partner's UID for the active EC ID. A provider-written UID is returned only +/// to a call with an allowed `Origin` and a same-origin or same-site +/// `Sec-Fetch-Site`, because the EC ID is a bearer value servers hold too. /// /// # Errors /// @@ -83,35 +102,79 @@ pub fn handle_identify( )); }; + // The EC ID is a bearer value partners and the origin hold (spec 9.5). + // A provider-written ID goes only to a call the visitor's browser made + // from the publisher's site. + let browser_context = allowed_origin.is_some() && is_same_site_fetch(req); let mut degraded = false; let mut uid: Option = None; let mut cluster_size: Option = None; match kv.get(ec_id) { - Ok(Some((entry, generation))) => { + Ok(Some((entry, _generation))) => { if !entry.consent.ok { // Tombstone entries preserve the withdrawal signal for 24 hours. - // Do not extract IDs or evaluate cluster size because that would - // write back with the live-entry TTL. + // Do not extract IDs or evaluate cluster size. log::trace!("Identify found tombstone for '{}'", log_id(ec_id)); } else { - // Extract only this partner's UID. - if let Some(partner_uid) = entry.ids.get(&partner.source_domain) - && !partner_uid.uid.is_empty() - { - uid = Some(partner_uid.uid.clone()); - } - - // Evaluate cluster size lazily for identify responses. Existing - // stored cluster_size values are reused without a prefix-list call. - match kv.evaluate_cluster(ec_id, &entry, generation) { - Ok(size) => { - cluster_size = size; - } - Err(err) => { - log::warn!("Cluster evaluation failed for '{}': {err:?}", log_id(ec_id)); + // Extract only this partner's UID, and only when the shared + // servability check passes under this request's consent, never + // the row's stored snapshot. A failed clock serves nothing, + // because a zero time would make every expiry check pass. + if let Some(now) = checked_current_timestamp() { + // Without a runtime no provider is configured. + let empty_catalog; + let catalog = match ec_context.identity_runtime() { + Some(runtime) => runtime.catalog(), + None => { + empty_catalog = ProviderCatalog::empty(); + &empty_catalog + } + }; + let opted_out = opted_out(&entry); + let serve = ServeContext { + now: UnixTime::from_secs(now), + consent: ec_context.consent(), + signals: PolicySignals::from_headers(req.headers()), + catalog, + opted_out: &opted_out, + }; + if let Some(partner_uid) = entry + .ids + .get(&partner.source_domain) + .filter(|record| record.w.is_none() || browser_context) + .filter(|record| { + partner + .source_domain + .parse::() + .is_ok_and(|source| is_servable(&serve, &source, record)) + }) + .and_then(KvPartnerId::first_uid) + .filter(|partner_uid| !partner_uid.is_empty()) + { + uid = Some(partner_uid.to_owned()); } } + + // Reuse a stored cluster_size; otherwise count without writing. + // The writer stores the size, never this read path. + cluster_size = match entry + .network + .as_ref() + .and_then(|network| network.cluster_size) + { + Some(size) => Some(size), + None => match kv.compute_cluster_size(ec_id) { + Ok(size) => Some(size), + Err(err) => { + log::warn!( + "Cluster evaluation failed for '{}': {err:?}", + log_id(ec_id) + ); + None + } + }, + }; } } Ok(None) => {} @@ -336,6 +399,12 @@ mod tests { use super::*; use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::{ConsentContext, ConsentSource}; + use std::collections::BTreeMap; + use std::sync::Arc; + + use crate::ec::identity::runtime::IdentityRuntime; + use crate::ec::identity::test_support::{example_entry_set, fixed_provider, full_record}; + use crate::ec::kv_types::{KvEid, KvEntry, KvUid}; use crate::ec::registry::PartnerRegistry; use crate::redacted::Redacted; use crate::settings::EcPartner; @@ -374,8 +443,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -641,6 +711,255 @@ mod tests { ); } + fn identify_body(kv: &KvIdentityGraph, ec_id: &str) -> serde_json::Value { + identify_body_for( + kv, + ec_id, + make_test_partner("ssp.example.com", VALID_API_TOKEN), + ) + } + + fn identify_body_for( + kv: &KvIdentityGraph, + ec_id: &str, + partner: EcPartner, + ) -> serde_json::Value { + let settings = create_test_settings(); + let partners = vec![partner]; + let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); + let req = Request::builder() + .method("GET") + .uri("https://edge.test-publisher.com/identify") + .header("authorization", format!("Bearer {VALID_API_TOKEN}")) + .body(EdgeBody::empty()) + .expect("should build test request"); + let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(ec_id)); + + let response = handle_identify(&settings, kv, ®istry, &req, &ec_context) + .expect("should construct identify response"); + assert_eq!(response.status(), StatusCode::OK, "should answer 200"); + serde_json::from_slice(&response.into_body().into_bytes().unwrap_or_default()) + .expect("should decode identify response JSON") + } + + #[test] + fn handle_identify_omits_an_expired_record_and_keeps_a_valid_one() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = KvEntry::minimal("ssp.example.com", "expired-uid", 1); + entry + .ids + .get_mut("ssp.example.com") + .expect("should hold the partner source") + .exp = Some(50); + entry.ids.insert( + "other.example.com".to_owned(), + KvPartnerId::new("valid-uid"), + ); + let kv = KvIdentityGraph::in_memory("identify-store"); + kv.create(&ec_id, &entry).expect("should seed KV entry"); + + let body = identify_body(&kv, &ec_id); + + assert!( + body.get("uid").is_none() && body.get("eid").is_none(), + "should not return the expired ID" + ); + + let mut live = KvEntry::minimal("ssp.example.com", "live-uid", 1); + live.ids + .get_mut("ssp.example.com") + .expect("should hold the partner source") + .exp = Some(u64::MAX); + let kv = KvIdentityGraph::in_memory("identify-store"); + kv.create(&ec_id, &live).expect("should seed KV entry"); + + let body = identify_body(&kv, &ec_id); + + assert_eq!(body["uid"], "live-uid", "should return the unexpired ID"); + } + + /// Calls `/identify` for a row whose `ssp.example.com` record `writer` + /// wrote, with the provider `prov` configured, and `headers` on the + /// request. Returns the `uid` the response carries. + fn identify_uid_with_headers(writer: Option<&str>, headers: &[(&str, &str)]) -> Option { + let settings = create_test_settings(); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = KvEntry::minimal("other.example.com", "other-uid", 1); + entry.ids.insert( + "ssp.example.com".to_owned(), + KvPartnerId { + w: writer.map(str::to_owned), + ..KvPartnerId::new("written-uid") + }, + ); + let kv = KvIdentityGraph::in_memory("identify-store"); + kv.create(&ec_id, &entry).expect("should seed KV entry"); + let partners = vec![make_test_partner("ssp.example.com", VALID_API_TOKEN)]; + let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); + let mut builder = Request::builder() + .method("GET") + .uri("https://edge.test-publisher.com/identify") + .header("authorization", format!("Bearer {VALID_API_TOKEN}")); + for (name, value) in headers { + builder = builder.header(*name, *value); + } + let req = builder + .body(EdgeBody::empty()) + .expect("should build test request"); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + ec_context.set_identity_runtime(Arc::new( + IdentityRuntime::from_settings_with_providers( + &settings, + vec![fixed_provider("prov", &["ssp.example.com"])], + BTreeMap::new(), + ) + .expect("should build the runtime"), + )); + + let response = handle_identify(&settings, &kv, ®istry, &req, &ec_context) + .expect("should construct identify response"); + assert_eq!(response.status(), StatusCode::OK, "should answer 200"); + let body: serde_json::Value = + serde_json::from_slice(&response.into_body().into_bytes().unwrap_or_default()) + .expect("should decode identify response JSON"); + body["uid"].as_str().map(str::to_owned) + } + + #[test] + fn identify_returns_a_provider_written_id_only_with_browser_context() { + const ORIGIN: (&str, &str) = ("origin", "https://www.test-publisher.com"); + + assert_eq!( + identify_uid_with_headers(Some("prov"), &[ORIGIN, ("sec-fetch-site", "same-site")]) + .as_deref(), + Some("written-uid"), + "should return it to a same-site browser call" + ); + assert_eq!( + identify_uid_with_headers(Some("prov"), &[ORIGIN, ("sec-fetch-site", "same-origin")]) + .as_deref(), + Some("written-uid"), + "should return it to a same-origin browser call" + ); + for (headers, case) in [ + (&[][..], "no Origin"), + (&[ORIGIN][..], "no Sec-Fetch-Site"), + ( + &[ORIGIN, ("sec-fetch-site", "cross-site")][..], + "a cross-site fetch", + ), + ( + &[ORIGIN, ("sec-fetch-site", "none")][..], + "a user-initiated fetch", + ), + ] { + assert_eq!( + identify_uid_with_headers(Some("prov"), headers), + None, + "should withhold it for {case}" + ); + } + } + + #[test] + fn identify_returns_a_browser_seeded_id_without_browser_context() { + assert_eq!( + identify_uid_with_headers(None, &[]).as_deref(), + Some("written-uid"), + "should keep today's behaviour for an ID no provider wrote" + ); + } + + #[test] + fn identify_computes_cluster_size_without_writing() { + let (kv, store) = KvIdentityGraph::write_counting("identify-store"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let entry = KvEntry::minimal("ssp.example.com", "example-uid", 1); + assert!( + entry.network.is_none(), + "should seed a row without a stored cluster size" + ); + kv.create(&ec_id, &entry).expect("should seed KV entry"); + store.reset_counts(); + + let body = identify_body(&kv, &ec_id); + + assert!(body["cluster_size"].is_number(), "should report a size"); + assert_eq!(store.writes().total(), 0, "should not write"); + } + + #[test] + fn identify_returns_the_first_uid_of_a_multi_entry_record_with_the_partner_atype() { + let (kv, _store) = KvIdentityGraph::write_counting("identify-store"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = KvEntry::minimal("other.example.com", "other-uid", 1); + let set = [&["example-a", "example-b"][..], &["example-c"][..]] + .iter() + .enumerate() + .map(|(index, ids)| KvEid { + uids: ids + .iter() + .map(|id| KvUid { + id: (*id).to_owned(), + atype: Some(1), + ext: None, + }) + .collect(), + inserter: (index == 0).then(|| "example.com".to_owned()), + ..KvEid::default() + }) + .collect(); + entry + .ids + .insert("partner.example.com".to_owned(), full_record(None, set)); + kv.create(&ec_id, &entry).expect("should seed KV entry"); + let mut partner = make_test_partner("partner.example.com", VALID_API_TOKEN); + partner.openrtb_atype = 3; + partner.bidstream_enabled = true; + + let body = identify_body_for(&kv, &ec_id, partner); + + assert_eq!( + body["uid"], + serde_json::json!("example-a"), + "should return the first UID of the first entry" + ); + assert_eq!( + body["eid"], + serde_json::json!({ "source": "partner.example.com", "uids": [{ "id": "example-a", "atype": 3 }] }), + "should return one UID with the partner's openrtb_atype, as for a one-UID record" + ); + } + + #[test] + fn handle_identify_returns_one_uid_for_a_full_record() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = KvEntry::minimal("other.example.com", "other-uid", 1); + entry.ids.insert( + "ssp.example.com".to_owned(), + full_record(None, example_entry_set()), + ); + let kv = KvIdentityGraph::in_memory("identify-store"); + kv.create(&ec_id, &entry).expect("should seed KV entry"); + + let body = identify_body(&kv, &ec_id); + + assert_eq!(body["uid"], "example-a", "should return the first UID"); + let expected = Eid::new( + "ssp.example.com".to_owned(), + vec![Uid { + id: "example-a".to_owned(), + atype: Some(EcPartner::default_openrtb_atype()), + ext: None, + }], + ); + assert_eq!( + body["eid"], + serde_json::to_value(expected).expect("should serialize the expected EID"), + "should return the one-UID shape, never the full set" + ); + } + #[test] fn handle_identify_denies_mismatched_browser_origin() { let settings = create_test_settings(); diff --git a/crates/trusted-server-core/src/ec/identity/catalog.rs b/crates/trusted-server-core/src/ec/identity/catalog.rs new file mode 100644 index 0000000000..92f6081119 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/catalog.rs @@ -0,0 +1,825 @@ +//! The enabled identity providers and source ownership. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use error_stack::{Report, ResultExt as _}; + +use super::descriptor::ProviderDescriptor; +use super::generic_pull::{GenericPullProvider, generic_pull_id}; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::provider::IdentityProvider; +use crate::error::TrustedServerError; +use crate::settings::{EcPartner, IdentitySettings, Settings}; + +/// Most providers one deployment may enable. +const MAX_PROVIDERS: usize = 32; + +fn catalog_error(message: String) -> Report { + Report::new(TrustedServerError::Configuration { message }) +} + +/// Validates the catalog rules of spec 10.2 against every provider +/// descriptor. +/// +/// It rejects a duplicate provider ID, more than 32 providers, an +/// `[ec.identity.providers.]` table that names no enabled provider, and +/// `revoke_scope` on a provider whose descriptor declares no `captures`. It +/// also checks source ownership, after each provider's `sources` narrowing: a +/// `sources` list must name declared sources and keep at least one, a +/// capture-only provider must list its `sources`, an `owner` must name an +/// enabled provider that is not capture-only and keeps the source, every +/// claimed source needs an `[[ec.partners]]` entry, and no second provider +/// may claim an owned source. It is the one place these rules live. It reads +/// descriptors only, so it runs on unresolved deploy-time settings. +/// +/// # Errors +/// +/// Returns [`TrustedServerError::Configuration`] naming the first rule +/// that fails. +pub(crate) fn validate_identity_catalog( + descriptors: &[ProviderDescriptor], + partners: &[EcPartner], + identity: &IdentitySettings, +) -> Result<(), Report> { + let mut ids = BTreeSet::new(); + for descriptor in descriptors { + if !ids.insert(descriptor.id().as_str()) { + return Err(catalog_error(format!( + "identity provider `{}` is declared more than once", + descriptor.id() + ))); + } + } + if descriptors.len() > MAX_PROVIDERS { + return Err(catalog_error(format!( + "at most {MAX_PROVIDERS} identity providers may be enabled, got {}", + descriptors.len() + ))); + } + + for (id, settings) in &identity.providers { + let Some(descriptor) = descriptors + .iter() + .find(|descriptor| descriptor.id().as_str() == id) + else { + let configured = if ids.is_empty() { + "none".to_owned() + } else { + ids.iter().copied().collect::>().join(", ") + }; + let hint = partners + .iter() + .find(|partner| { + !partner.pull_sync_enabled + && generic_pull_id(&partner.source_domain).as_str() == id + }) + .map_or_else(String::new, |partner| { + format!( + "; partner `{}` has pull_sync_enabled = false", + partner.source_domain + ) + }); + return Err(catalog_error(format!( + "ec.identity.providers.{id} names no enabled identity provider{hint}; \ + configured providers: {configured}" + ))); + }; + if settings.revoke_scope.is_some() && descriptor.captures().is_none() { + return Err(catalog_error(format!( + "ec.identity.providers.{id}.revoke_scope is set, but provider `{id}` declares no captures" + ))); + } + } + + validate_source_ownership(descriptors, partners, identity) +} + +/// Whether `descriptor` captures and never resolves. +fn is_capture_only(descriptor: &ProviderDescriptor) -> bool { + descriptor.captures().is_some() && descriptor.resolves().is_none() +} + +/// Each descriptor's sources after its `sources` narrowing, rejecting a list +/// that names an undeclared source or keeps none, and a capture-only +/// provider that has no list. +fn narrowed_sources<'a>( + descriptors: &'a [ProviderDescriptor], + identity: &IdentitySettings, +) -> Result>, Report> { + let mut narrowed = BTreeMap::new(); + for descriptor in descriptors { + let id = descriptor.id().as_str(); + let listed = identity + .providers + .get(id) + .and_then(|table| table.sources.as_ref()); + let Some(listed) = listed else { + if is_capture_only(descriptor) { + return Err(catalog_error(format!( + "provider `{id}` is capture-only, so ec.identity.providers.{id}.sources must \ + list the sources it is the delivery path for" + ))); + } + narrowed.insert(id, descriptor.sources().to_vec()); + continue; + }; + let mut kept = Vec::with_capacity(listed.len()); + for name in listed { + let declared = name + .parse::() + .ok() + .filter(|source| descriptor.sources().contains(source)); + let Some(source) = declared else { + return Err(catalog_error(format!( + "ec.identity.providers.{id}.sources names `{name}`, which provider `{id}` does not declare" + ))); + }; + kept.push(source); + } + if kept.is_empty() { + return Err(catalog_error(format!( + "ec.identity.providers.{id}.sources narrows provider `{id}` to no source" + ))); + } + narrowed.insert(id, kept); + } + Ok(narrowed) +} + +/// The ownership rules of spec 10.2, on narrowed sources. +fn validate_source_ownership( + descriptors: &[ProviderDescriptor], + partners: &[EcPartner], + identity: &IdentitySettings, +) -> Result<(), Report> { + let narrowed = narrowed_sources(descriptors, identity)?; + + // A partner whose source does not parse is reported by partner validation. + let partner_sources: BTreeMap> = partners + .iter() + .filter_map(|partner| { + Some(( + partner.source_domain.parse().ok()?, + partner.owner.as_deref(), + )) + }) + .collect(); + + for descriptor in descriptors { + let id = descriptor.id().as_str(); + for source in &narrowed[id] { + if !partner_sources.contains_key(source) { + return Err(catalog_error(format!( + "identity provider `{id}` claims source `{source}`, which has no [[ec.partners]] entry" + ))); + } + } + } + + for (source, owner) in &partner_sources { + let Some(owner) = owner else { + continue; + }; + let Some(descriptor) = descriptors + .iter() + .find(|descriptor| descriptor.id().as_str() == *owner) + else { + return Err(catalog_error(format!( + "ec.partners source `{source}` names owner `{owner}`, which is not an enabled identity provider" + ))); + }; + if is_capture_only(descriptor) { + return Err(catalog_error(format!( + "ec.partners source `{source}` names owner `{owner}`, which is capture-only; \ + a capture-only provider stores no positive ID and cannot own a source" + ))); + } + if !narrowed[*owner].contains(source) { + return Err(catalog_error(format!( + "ec.partners source `{source}` names owner `{owner}`, which does not claim that source" + ))); + } + let other = descriptors + .iter() + .map(|other| other.id().as_str()) + .find(|other| other != owner && narrowed[*other].contains(source)); + if let Some(other) = other { + return Err(catalog_error(format!( + "identity provider `{other}` claims source `{source}`, which `{owner}` owns; \ + use ec.identity.providers.{other}.sources to narrow one of them" + ))); + } + } + Ok(()) +} + +/// Enabled providers, their narrowed sources and source owners. Built once +/// per settings load. +#[derive(Clone)] +pub(crate) struct ProviderCatalog { + providers: BTreeMap>, + sources: BTreeMap>, + owners: BTreeMap, +} + +impl ProviderCatalog { + /// Builds the catalog from settings, without resolving any secret. + /// + /// It reads `settings.ec.partners` directly and never builds a + /// `PartnerRegistry`, so a deploy-time `api_token` that holds a secret + /// store key name is fine. It holds the generic pull providers, then + /// the `registered` providers integrations contributed, with each + /// source owner from `[[ec.partners]].owner`. It validates every descriptor with + /// [`validate_identity_catalog`] before building, so a duplicate ID is + /// reported by that one validator. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::Configuration`] when a pull partner + /// cannot be described or the catalog rules fail. + pub(crate) fn from_settings( + settings: &Settings, + registered: &[Arc], + ) -> Result> { + let mut providers = settings + .ec + .partners + .iter() + .filter(|partner| partner.pull_sync_enabled) + .map(|partner| { + GenericPullProvider::from_partner(partner) + .map(|provider| Arc::new(provider) as Arc) + .change_context(TrustedServerError::Configuration { + message: format!( + "ec.partners: pull sync of '{}' cannot be described as an identity provider", + partner.source_domain + ), + }) + }) + .collect::, _>>()?; + providers.extend(registered.iter().cloned()); + let descriptors: Vec<_> = providers + .iter() + .map(|provider| provider.descriptor().clone()) + .collect(); + validate_identity_catalog(&descriptors, &settings.ec.partners, &settings.ec.identity)?; + let owners = settings + .ec + .partners + .iter() + .filter_map(|partner| { + let owner = partner.owner.as_deref()?; + Some((partner.source_domain.parse().ok()?, owner.parse().ok()?)) + }) + .collect(); + Ok(Self::new(providers, owners)) + } + + /// A catalog with no providers. + #[must_use] + pub(crate) fn empty() -> Self { + Self::new(Vec::new(), BTreeMap::new()) + } + + /// Builds the catalog. Validation (ownership, duplicates) happens in + /// config validation before this is called. + #[must_use] + pub(crate) fn new( + providers: Vec>, + owners: BTreeMap, + ) -> Self { + let providers: BTreeMap<_, _> = providers + .into_iter() + .map(|provider| (provider.descriptor().id().clone(), provider)) + .collect(); + let sources = providers + .iter() + .map(|(id, provider)| (id.clone(), provider.descriptor().sources().to_vec())) + .collect(); + Self { + providers, + sources, + owners, + } + } + + /// Narrows providers to the listed sources their descriptors declare + /// (spec 10.1). Config validation has rejected a source a descriptor + /// does not declare and a narrowing to nothing. + #[must_use] + pub(crate) fn with_narrowing( + mut self, + narrowing: BTreeMap>, + ) -> Self { + for (id, listed) in narrowing { + if let Some(sources) = self.sources.get_mut(&id) { + sources.retain(|source| listed.contains(source)); + } + } + self + } + + /// A provider by ID. + #[must_use] + pub(crate) fn get(&self, id: &IdentityProviderId) -> Option<&Arc> { + self.providers.get(id) + } + + /// Every provider, ordered by ID. + pub(crate) fn providers(&self) -> impl Iterator> { + self.providers.values() + } + + /// Whether `id` is an enabled provider. + #[must_use] + pub(crate) fn is_configured(&self, id: &IdentityProviderId) -> bool { + self.providers.contains_key(id) + } + + /// The owner of `source`, if any. + #[must_use] + pub(crate) fn owner_of(&self, source: &SourceDomain) -> Option<&IdentityProviderId> { + self.owners.get(source) + } + + /// A provider's sources after narrowing. Empty for an unknown provider. + #[must_use] + pub(crate) fn sources_of(&self, id: &IdentityProviderId) -> &[SourceDomain] { + self.sources.get(id).map_or(&[], Vec::as_slice) + } + + /// Every provider that claims `source` after narrowing, ordered by ID. + pub(crate) fn claimants<'a>( + &'a self, + source: &'a SourceDomain, + ) -> impl Iterator { + self.sources + .iter() + .filter(move |(_, sources)| sources.contains(source)) + .map(|(id, _)| id) + } + + /// Whether `id` captures and never resolves. + #[must_use] + pub(crate) fn is_capture_only(&self, id: &IdentityProviderId) -> bool { + self.get(id).is_some_and(|provider| { + let descriptor = provider.descriptor(); + descriptor.captures().is_some() && descriptor.resolves().is_none() + }) + } + + /// The narrowed sources of a capture-only provider. Empty for any other. + #[must_use] + pub(crate) fn capture_only_sources(&self, id: &IdentityProviderId) -> &[SourceDomain] { + if self.is_capture_only(id) { + self.sources_of(id) + } else { + &[] + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::descriptor::{Captures, Resolves}; + use crate::ec::identity::examples::{ExampleIdProvider, example_config}; + use crate::ec::identity::generic_pull::generic_pull_descriptors; + use crate::ec::identity::test_support::{ec_partner, fixed_capture_provider, resolver_policy}; + use crate::settings::{ProviderSettings, RevokeScope}; + use crate::test_support::tests::create_test_settings; + + const TOKEN: &str = "example-token-value-0123456789abcdef"; + + fn partner() -> EcPartner { + ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + TOKEN, + ) + } + + fn owned_partner(source: &str, owner: Option<&str>) -> EcPartner { + let mut partner = ec_partner(source, &format!("https://sync.{source}/pull"), TOKEN); + partner.pull_sync_enabled = false; + partner.owner = owner.map(ToOwned::to_owned); + partner + } + + fn resolving_descriptor(id: &str, sources: &[&str]) -> ProviderDescriptor { + let mut builder = ProviderDescriptor::builder(id.parse().expect("should parse the ID")) + .with_resolves(Resolves::new().with_endpoint("https://api.example.com/fetch")) + .with_consent_policy(resolver_policy()); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse the source")); + } + builder.build().expect("should build the descriptor") + } + + fn capture_descriptor(id: &str, sources: &[&str]) -> ProviderDescriptor { + fixed_capture_provider(id, sources).descriptor().clone() + } + + fn listing(sources: &[&str]) -> ProviderSettings { + ProviderSettings { + sources: Some(sources.iter().map(|source| (*source).to_owned()).collect()), + ..ProviderSettings::default() + } + } + + fn identity_tables(tables: Vec<(&str, ProviderSettings)>) -> IdentitySettings { + let mut identity = IdentitySettings::default(); + for (id, settings) in tables { + identity.providers.insert(id.to_owned(), settings); + } + identity + } + + fn validate_with( + descriptors: &[ProviderDescriptor], + partners: &[EcPartner], + identity: &IdentitySettings, + ) -> Result<(), String> { + validate_identity_catalog(descriptors, partners, identity).map_err(|e| format!("{e}")) + } + + fn identity_with(id: &str, settings: ProviderSettings) -> IdentitySettings { + let mut identity = IdentitySettings::default(); + identity.providers.insert(id.to_owned(), settings); + identity + } + + fn validate(partners: &[EcPartner], identity: &IdentitySettings) -> Result<(), String> { + let descriptors = generic_pull_descriptors(partners).expect("should describe"); + validate_identity_catalog(&descriptors, partners, identity).map_err(|e| format!("{e:?}")) + } + + #[test] + fn validate_identity_catalog_rejects_unknown_provider_tables() { + let partners = [partner()]; + + let error = validate( + &partners, + &identity_with("pull-other-example-com", ProviderSettings::default()), + ) + .expect_err("should reject a table naming no provider"); + assert!( + error.contains("pull-other-example-com") && error.contains("pull-ssp-example-com"), + "should name the table and list the configured IDs, got: {error}" + ); + + validate( + &partners, + &identity_with("pull-ssp-example-com", ProviderSettings::default()), + ) + .expect("control: a table for the real pull ID should pass"); + } + + #[test] + fn validate_identity_catalog_rejects_revoke_scope_without_captures() { + let partners = [partner()]; + let with_scope = ProviderSettings { + revoke_scope: Some(RevokeScope::Provider), + ..ProviderSettings::default() + }; + + validate( + &partners, + &identity_with("pull-ssp-example-com", with_scope.clone()), + ) + .expect_err("should reject revoke_scope on a provider without captures"); + validate( + &partners, + &identity_with("pull-ssp-example-com", ProviderSettings::default()), + ) + .expect("control: the same table without revoke_scope should pass"); + + let capturing = fixed_capture_provider("capturer", &["capture.example.com"]); + assert!( + capturing + .descriptor() + .captures() + .is_some_and(|c| *c != Captures::new()), + "should hold a capturing descriptor for the control" + ); + let capturing_scope = ProviderSettings { + sources: Some(vec!["capture.example.com".to_owned()]), + ..with_scope + }; + validate_identity_catalog( + &[capturing.descriptor().clone()], + &[owned_partner("capture.example.com", None)], + &identity_with("capturer", capturing_scope), + ) + .expect("control: revoke_scope on a capturing provider should pass"); + } + + #[test] + fn validate_identity_catalog_rejects_duplicate_ids_and_more_than_32_providers() { + let one = partner(); + let descriptors = + generic_pull_descriptors(std::slice::from_ref(&one)).expect("should describe"); + let duplicated = [descriptors[0].clone(), descriptors[0].clone()]; + let _ = validate_identity_catalog(&duplicated, &[one], &IdentitySettings::default()) + .expect_err("should reject a duplicate ID"); + + let many = |count: usize| -> Vec { + (0..count) + .map(|n| { + let source = format!("ssp{n}.example.com"); + ec_partner(&source, &format!("https://sync.{source}/pull"), TOKEN) + }) + .collect() + }; + validate(&many(32), &IdentitySettings::default()) + .expect("control: 32 providers should pass"); + validate(&many(33), &IdentitySettings::default()).expect_err("should reject 33 providers"); + } + + #[test] + fn from_settings_builds_from_unresolved_settings() { + let mut settings = create_test_settings(); + let mut unresolved = partner(); + unresolved.api_token = Some(crate::redacted::Redacted::new("key-name".to_owned())); + settings.ec.partners = vec![unresolved]; + + assert!( + crate::ec::registry::PartnerRegistry::from_config(&settings.ec.partners).is_err(), + "should hold settings the registry rejects" + ); + let catalog = ProviderCatalog::from_settings(&settings, &[]).expect("should build"); + + let id = "pull-ssp-example-com".parse().expect("should parse the ID"); + assert!(catalog.is_configured(&id), "should hold the pull provider"); + assert_eq!(catalog.sources_of(&id).len(), 1, "should claim its source"); + assert!( + catalog + .owner_of(&"ssp.example.com".parse().expect("should parse")) + .is_none(), + "should have no owner for a partner that names none" + ); + } + + #[test] + fn registered_provider_with_a_generic_pull_id_is_rejected_by_the_shared_validator() { + let mut settings = create_test_settings(); + settings.ec.partners = vec![partner()]; + + let duplicate = fixed_capture_provider("pull-ssp-example-com", &["other.example.com"]); + let error = ProviderCatalog::from_settings(&settings, &[duplicate]) + .err() + .expect("should reject a registered ID that a pull partner already uses"); + assert!( + format!("{error}") + .contains("identity provider `pull-ssp-example-com` is declared more than once"), + "should report the duplicate ID: {error}" + ); + + settings + .ec + .partners + .push(owned_partner("exampleid.example.com", None)); + let registered = + ExampleIdProvider::new(&example_config()).expect("should build the example provider"); + let catalog = ProviderCatalog::from_settings(&settings, &[Arc::new(registered)]) + .expect("control: a distinct registered ID should build"); + assert!( + catalog.is_configured(&"exampleid".parse().expect("should parse")), + "should hold the registered provider" + ); + assert!( + catalog.is_configured(&"pull-ssp-example-com".parse().expect("should parse")), + "should keep the pull provider" + ); + } + + #[test] + fn owner_must_name_an_enabled_provider_that_declares_the_source() { + let descriptors = vec![resolving_descriptor( + "exampleid", + &["exampleid.example.com"], + )]; + let ok = vec![owned_partner("exampleid.example.com", Some("exampleid"))]; + validate_with(&descriptors, &ok, &IdentitySettings::default()) + .expect("should accept an owner that declares the source"); + + let unknown = vec![owned_partner("exampleid.example.com", Some("missing"))]; + let error = validate_with(&descriptors, &unknown, &IdentitySettings::default()) + .expect_err("should reject an owner that is not enabled"); + assert!(error.contains("missing"), "should name the owner: {error}"); + + let wrong_source = vec![ + owned_partner("exampleid.example.com", None), + owned_partner("other.example.com", Some("exampleid")), + ]; + let error = validate_with(&descriptors, &wrong_source, &IdentitySettings::default()) + .expect_err("should reject an owner whose descriptor lacks the source"); + assert!( + error.contains("other.example.com"), + "should name the source: {error}" + ); + } + + #[test] + fn owner_uses_the_narrowed_descriptor() { + let descriptors = vec![resolving_descriptor( + "exampleid", + &["a.example.com", "b.example.com"], + )]; + let partners = vec![ + owned_partner("a.example.com", Some("exampleid")), + owned_partner("b.example.com", None), + ]; + validate_with( + &descriptors, + &partners, + &identity_tables(vec![("exampleid", listing(&["a.example.com"]))]), + ) + .expect("control: an owner keeping its source should pass"); + + let error = validate_with( + &descriptors, + &partners, + &identity_tables(vec![("exampleid", listing(&["b.example.com"]))]), + ) + .expect_err("should reject an owner narrowed away from its source"); + assert!( + error.contains("a.example.com"), + "should name the source: {error}" + ); + } + + #[test] + fn provider_claiming_a_source_without_a_partner_entry_is_rejected() { + let descriptors = vec![resolving_descriptor( + "exampleid", + &["a.example.com", "b.example.com"], + )]; + let partners = vec![owned_partner("a.example.com", None)]; + let error = validate_with(&descriptors, &partners, &IdentitySettings::default()) + .expect_err("should reject a claimed source with no partner entry"); + assert!( + error.contains("b.example.com") && error.contains("exampleid"), + "should name the source and the provider: {error}" + ); + + validate_with( + &descriptors, + &partners, + &identity_tables(vec![("exampleid", listing(&["a.example.com"]))]), + ) + .expect("control: narrowing to the declared partner should pass"); + validate_with( + &descriptors, + &[ + owned_partner("a.example.com", None), + owned_partner("b.example.com", None), + ], + &IdentitySettings::default(), + ) + .expect("control: a partner entry per source should pass"); + } + + #[test] + fn two_providers_claiming_an_owned_source_are_rejected() { + let descriptors = vec![ + resolving_descriptor("first", &["a.example.com"]), + resolving_descriptor("second", &["a.example.com", "b.example.com"]), + ]; + let owned = vec![ + owned_partner("a.example.com", Some("first")), + owned_partner("b.example.com", None), + ]; + let error = validate_with(&descriptors, &owned, &IdentitySettings::default()) + .expect_err("should reject a second claimant of an owned source"); + assert!( + error.contains("a.example.com") && error.contains("second"), + "should name the source and the other claimant: {error}" + ); + + validate_with( + &descriptors, + &owned, + &identity_tables(vec![("second", listing(&["b.example.com"]))]), + ) + .expect("control: narrowing the other claimant away should pass"); + + let unowned = vec![ + owned_partner("a.example.com", None), + owned_partner("b.example.com", None), + ]; + validate_with(&descriptors, &unowned, &IdentitySettings::default()) + .expect("control: two claimants of an unowned source should pass"); + } + + #[test] + fn sources_must_name_declared_sources_and_not_narrow_to_nothing() { + let descriptors = vec![resolving_descriptor("exampleid", &["a.example.com"])]; + let partners = vec![ + owned_partner("a.example.com", None), + owned_partner("c.example.com", None), + ]; + validate_with( + &descriptors, + &partners, + &identity_tables(vec![("exampleid", listing(&["a.example.com"]))]), + ) + .expect("control: a declared source should pass"); + + let error = validate_with( + &descriptors, + &partners, + &identity_tables(vec![("exampleid", listing(&["c.example.com"]))]), + ) + .expect_err("should reject an undeclared source"); + assert!( + error.contains("c.example.com") && error.contains("exampleid"), + "should name the source and the provider: {error}" + ); + + let error = validate_with( + &descriptors, + &partners, + &identity_tables(vec![("exampleid", listing(&[]))]), + ) + .expect_err("should reject narrowing to nothing"); + assert!( + error.contains("exampleid") && error.contains("no source"), + "should name the provider: {error}" + ); + } + + #[test] + fn capture_only_provider_without_sources_is_rejected() { + let descriptors = vec![capture_descriptor("capturer", &["capture.example.com"])]; + let partners = vec![owned_partner("capture.example.com", None)]; + let error = validate_with(&descriptors, &partners, &IdentitySettings::default()) + .expect_err("should reject a capture-only provider with no sources list"); + assert!( + error.contains("capturer") && error.contains("sources"), + "should name the provider and the key: {error}" + ); + + validate_with( + &descriptors, + &partners, + &identity_tables(vec![("capturer", ProviderSettings::default())]), + ) + .expect_err("should reject a provider table that lists no sources"); + + validate_with( + &descriptors, + &partners, + &identity_tables(vec![("capturer", listing(&["capture.example.com"]))]), + ) + .expect("control: a sources list should pass"); + + let resolving = vec![resolving_descriptor("exampleid", &["capture.example.com"])]; + validate_with(&resolving, &partners, &IdentitySettings::default()) + .expect("control: a resolving provider needs no sources list"); + } + + #[test] + fn capture_only_provider_cannot_own_a_source() { + let descriptors = vec![capture_descriptor("capturer", &["capture.example.com"])]; + let identity = identity_tables(vec![("capturer", listing(&["capture.example.com"]))]); + let owned = vec![owned_partner("capture.example.com", Some("capturer"))]; + let error = validate_with(&descriptors, &owned, &identity) + .expect_err("should reject a capture-only owner"); + assert!( + error.contains("capturer") && error.contains("capture-only"), + "should name the provider and the reason: {error}" + ); + + let unowned = vec![owned_partner("capture.example.com", None)]; + validate_with(&descriptors, &unowned, &identity) + .expect("control: the same provider without ownership should pass"); + + let resolving = vec![resolving_descriptor("capturer", &["capture.example.com"])]; + validate_with(&resolving, &owned, &IdentitySettings::default()) + .expect("control: a resolving provider may own the source"); + } + + #[test] + fn from_settings_records_owners_from_partners() { + let mut settings = create_test_settings(); + settings.ec.partners = vec![owned_partner("exampleid.example.com", Some("exampleid"))]; + let registered = + ExampleIdProvider::new(&example_config()).expect("should build the example provider"); + let catalog = ProviderCatalog::from_settings(&settings, &[Arc::new(registered)]) + .expect("should build"); + + assert_eq!( + catalog.owner_of(&"exampleid.example.com".parse().expect("should parse")), + Some(&"exampleid".parse().expect("should parse the ID")), + "should record the partner's owner" + ); + assert!( + catalog + .owner_of(&"other.example.com".parse().expect("should parse")) + .is_none(), + "control: an unowned source has no owner" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs new file mode 100644 index 0000000000..46bbb23a6e --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -0,0 +1,6980 @@ +//! Coordinator eligibility, dispatch and outcomes (spec 4.7 and 8). +//! +//! Which of a provider's sources need a call, how the calls are sent under +//! the transport limits, and what stored changes a call's outcome becomes. + +use std::collections::{BTreeMap, BTreeSet}; +use std::time::Duration; + +use edgezero_core::body::Body as EdgeBody; +use error_stack::Report; +use http::{StatusCode, header}; +use web_time::Instant; + +use super::catalog::ProviderCatalog; +use super::descriptor::{Input, ProviderDescriptor, ResolveKind, Resolves}; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::flush::{DispatchInput, DispatchRun, add_dispatch_results, may_flush, request_facts}; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::input::{ + CaptureInput, ProviderConsent, RandomSource, ResolveInput, ResolveSignals, StoredId, + ThreadRandom, +}; +use super::mutation::{ + ApplyContext, IdentityMutation, PartnerIdChange, ProviderResult, ProviderStateChange, + StateUpdate, apply, +}; +use super::policy::{ + FP_PURPOSES, FP_TCF, FP_US_OPT_OUT_MASK, FP_VENDOR, PolicySignals, fingerprint_bits, + policy_allows, +}; +use super::provider::IdentityProvider; +use super::removal::{RemovalTiming, RemovalWriteOutcome, write_removals_counted}; +use super::resolution::{ + EntrySet, EntrySets, IdentityProviderError, Resolution, ResolvedEid, StateChange, entry_sets, +}; +use super::revision::{IdentityKeys, StateRevision, record_revision, uid_hash, visitor_key}; +use crate::consent::{ConsentContext, effective_tcf}; +use crate::ec::kv::KvIdentityGraph; +use crate::ec::kv_types::{KvEid, KvEntry, KvOptOut, KvPartnerId}; +use crate::ec::rate_limiter::RateLimiter; +use crate::ec::{EcContext, EcKvSnapshot, current_timestamp, log_id}; +use crate::platform::{ + PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, + RuntimeServices, +}; +use crate::settings::{IdentitySettings, ProviderMode, ProviderSettings, RevokeScope, Settings}; +use crate::streaming_processor::{BodyStreamDecoder, Compression}; + +/// The longest connect timeout a hold call asks for: the adapters' default. +const HOLD_CONNECT_TIMEOUT_CAP: Duration = Duration::from_secs(1); + +/// How long a burst-capped provider waits: one limiter window. +const BURST_WAIT_SECS: u64 = 60; +/// The wait after a `no_match` that gives no `retry_in`. +const NO_MATCH_WAIT_SECS: u64 = 86_400; +/// The longest `retry_in` the coordinator honours. +const MAX_RETRY_SECS: u64 = 7 * 86_400; +/// First wait after an input-caused failure. +const INPUT_BACKOFF_BASE_SECS: u64 = 3_600; +/// Longest wait after an input-caused failure. +const INPUT_BACKOFF_CAP_SECS: u64 = 86_400; +/// First wait after a transport failure. +const TRANSPORT_BACKOFF_BASE_SECS: u64 = 300; +/// Longest wait after a transport failure. +const TRANSPORT_BACKOFF_CAP_SECS: u64 = 3_600; + +/// One provider's `ps` entry, as the request read it. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +#[derive(Debug, Clone, Default)] +struct ProviderView { + n: Option, + f: u32, + o: Option, + oc: Option, + iv: Vec, + ov: Vec, + st: Option, +} + +/// What eligibility and outcomes read from the KV row the request read +/// (spec 4.7), and nothing else. It never reads `ts-eids-s`: every removal is +/// already in KV (spec 5.3). +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +#[derive(Debug, Clone, Default)] +pub(crate) struct EligibilityView { + records: BTreeMap, + providers: BTreeMap, +} + +impl EligibilityView { + /// The view of the row the request read. A key that is not a valid + /// source domain or provider ID cannot name a configured one and is + /// skipped. + #[must_use] + // Temporary: the request wiring, the first non-test caller, arrives in a later task. + #[allow(dead_code)] + pub(crate) fn from_kv(entry: &KvEntry) -> Self { + let records = entry + .ids + .iter() + .filter_map(|(source, record)| { + Some((source.parse::().ok()?, record.clone())) + }) + .collect(); + let providers = entry + .ps + .iter() + .filter_map(|(provider, state)| { + let view = ProviderView { + n: state.n, + f: state.f, + o: state.o.clone(), + oc: state.oc, + iv: state.iv.clone(), + ov: state.ov.clone(), + st: state.st.clone(), + }; + Some((provider.parse::().ok()?, view)) + }) + .collect(); + Self { records, providers } + } + + /// A view of a row with no records and no provider state. + #[must_use] + // Temporary: the request wiring, the first non-test caller, arrives in a later task. + #[allow(dead_code)] + pub(crate) fn empty() -> Self { + Self::default() + } + + /// The whole record of `source`, if the row holds one. + #[must_use] + pub(crate) fn record(&self, source: &SourceDomain) -> Option<&KvPartnerId> { + self.records.get(source) + } + + /// Every stored record. + pub(crate) fn records(&self) -> impl Iterator { + self.records.iter() + } + + /// The provider's consecutive failures, `f`. + #[must_use] + pub(crate) fn failures(&self, provider: &IdentityProviderId) -> u32 { + self.providers.get(provider).map_or(0, |view| view.f) + } + + /// The provider's `n`: the earliest time it may be called. + #[must_use] + pub(crate) fn next_eligible(&self, provider: &IdentityProviderId) -> Option { + self.providers.get(provider).and_then(|view| view.n) + } + + /// The provider's private state `st`, which a call reads. + #[must_use] + // Temporary: the request wiring, the first non-test caller, arrives in a later task. + #[allow(dead_code)] + pub(crate) fn state(&self, provider: &IdentityProviderId) -> Option<&serde_json::Value> { + self.providers + .get(provider) + .and_then(|view| view.st.as_ref()) + } + + /// Whether the provider's opt-out removed a UID with this hash, `ps.ov`. + #[must_use] + // Temporary: the request wiring, the first non-test caller, arrives in a later task. + #[allow(dead_code)] + pub(crate) fn has_opt_out_hash(&self, provider: &IdentityProviderId, hash: &str) -> bool { + self.providers + .get(provider) + .is_some_and(|view| view.ov.iter().any(|known| known == hash)) + } + + /// Whether the provider invalidated a UID with this hash, `ps.iv`. + fn has_invalidated_hash(&self, provider: &IdentityProviderId, hash: &str) -> bool { + self.providers + .get(provider) + .is_some_and(|view| view.iv.iter().any(|known| known == hash)) + } + + /// Whether `o` is set and no clear's watermark `oc` reaches `o.at`. + #[must_use] + pub(crate) fn is_opted_out(&self, provider: &IdentityProviderId) -> bool { + self.providers.get(provider).is_some_and(|view| { + view.o + .as_ref() + .is_some_and(|opt_out| view.oc.is_none_or(|clear| clear < opt_out.at)) + }) + } + + #[cfg(test)] + fn provider_mut(&mut self, provider: &str) -> &mut ProviderView { + self.providers + .entry(provider.parse().expect("should parse the provider ID")) + .or_default() + } + + #[cfg(test)] + pub(crate) fn set_record_for_test(&mut self, source: &SourceDomain, record: KvPartnerId) { + self.records.insert(source.clone(), record); + } + + #[cfg(test)] + pub(crate) fn set_failures_for_test(&mut self, provider: &str, failures: u32) { + self.provider_mut(provider).f = failures; + } + + #[cfg(test)] + pub(crate) fn set_next_for_test(&mut self, provider: &str, next: u64) { + self.provider_mut(provider).n = Some(next); + } + + #[cfg(test)] + pub(crate) fn set_invalidated_for_test(&mut self, provider: &str, hashes: Vec) { + self.provider_mut(provider).iv = hashes; + } + + #[cfg(test)] + pub(crate) fn set_opt_out_for_test(&mut self, provider: &str, at: u64, clear: Option) { + let view = self.provider_mut(provider); + view.o = Some(KvOptOut { + at, + fp: "0".to_owned(), + }); + view.oc = clear; + } +} + +/// What the request is, as eligibility reads it. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +#[derive(Debug, Clone)] +pub(crate) struct RequestFacts { + /// The client looks like a real browser (step 1). + pub(crate) is_real_browser: bool, + /// Global consent allows EC and EID persistence (step 2). + pub(crate) ec_persistence_allowed: bool, + /// The signals the consent context does not carry (step 3). + pub(crate) policy_signals: PolicySignals, + /// This is the writer request, not a stager (step 7). + pub(crate) writer: bool, + /// This is a new visitor's first navigation (step 7). + pub(crate) new_visitor: bool, + /// The navigation runs an auction (step 7). + pub(crate) runs_auction: bool, + /// The request's time. + pub(crate) now: UnixTime, + /// The inputs this request can supply (step 4). + pub(crate) available_inputs: BTreeSet, +} + +/// When a provider is dispatched, relative to the navigation response. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Phase { + /// Before the auction, within the hold budget. + Hold, + /// After the response is sent. + AfterSend, +} + +/// Why a provider was not eligible. Each variant names the step of spec 4.7 +/// that stopped it. The breaker (step 6) and the burst cap (step 8) are +/// checked in [`dispatch`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum SkipReason { + /// The provider declares no `resolves`, so it never makes a call. + NotAResolver, + /// Step 1: the client does not look like a real browser. + NotRealBrowser, + /// Step 2: global consent does not allow EID persistence. + EcPersistenceNotAllowed, + /// Step 3: the provider's consent policy is not met. + ConsentPolicy, + /// Step 4: no source needs a call this time. + NoSourceDue, + /// Step 5: `ps.n` has not passed. + NotYetEligible, + /// Step 5: the visitor opted out of the provider. + OptedOut, + /// Step 7: this is a new visitor's first navigation. + NewVisitor, + /// Step 7: this request is a stager. + NotWriter, + /// Step 7: the provider's mode runs it in the other phase. + WrongPhase, +} + +/// Whether a provider is eligible. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum Eligibility { + /// It is, for these sources. + Eligible { + kinds: BTreeMap, + }, + /// It is not. + Skip(SkipReason), +} + +/// What [`eligibility`] reads about one provider. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +pub(crate) struct EligibilityInput<'a> { + pub(crate) provider: &'a dyn IdentityProvider, + pub(crate) settings: &'a ProviderSettings, + pub(crate) identity: &'a IdentitySettings, + pub(crate) catalog: &'a ProviderCatalog, + pub(crate) consent: &'a ConsentContext, +} + +/// Whether the provider is eligible for this request, steps 1 to 7 of spec +/// 4.7 in order, and which sources need which kind of call. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +#[must_use] +pub(crate) fn eligibility( + input: &EligibilityInput<'_>, + view: &EligibilityView, + facts: &RequestFacts, + phase: Phase, +) -> Eligibility { + let descriptor = input.provider.descriptor(); + let id = descriptor.id(); + if descriptor.resolves().is_none() { + return Eligibility::Skip(SkipReason::NotAResolver); + } + if !facts.is_real_browser { + return Eligibility::Skip(SkipReason::NotRealBrowser); + } + if !facts.ec_persistence_allowed { + return Eligibility::Skip(SkipReason::EcPersistenceNotAllowed); + } + if !policy_allows( + descriptor.consent_policy(), + input.consent, + facts.policy_signals, + ) { + return Eligibility::Skip(SkipReason::ConsentPolicy); + } + let kinds = source_kinds(input, view, facts); + if kinds.is_empty() { + log::debug!("Skipping identity provider '{id}': no source needs a call"); + return Eligibility::Skip(SkipReason::NoSourceDue); + } + if view + .next_eligible(id) + .is_some_and(|next| facts.now.as_secs() < next) + { + return Eligibility::Skip(SkipReason::NotYetEligible); + } + if view.is_opted_out(id) { + return Eligibility::Skip(SkipReason::OptedOut); + } + if facts.new_visitor { + return Eligibility::Skip(SkipReason::NewVisitor); + } + if !facts.writer { + return Eligibility::Skip(SkipReason::NotWriter); + } + let runs_in_phase = match (phase, input.settings.mode) { + (Phase::Hold, ProviderMode::Hold) => facts.runs_auction, + (Phase::AfterSend, ProviderMode::AfterSend) => true, + (Phase::AfterSend, ProviderMode::Hold) => !facts.runs_auction, + (Phase::Hold, ProviderMode::AfterSend) => false, + }; + if !runs_in_phase { + return Eligibility::Skip(SkipReason::WrongPhase); + } + Eligibility::Eligible { kinds } +} + +/// Step 4: the kind each claimed source needs, leaving out a source whose +/// kind the provider does not list or whose needs the request cannot meet. +fn source_kinds( + input: &EligibilityInput<'_>, + view: &EligibilityView, + facts: &RequestFacts, +) -> BTreeMap { + let descriptor = input.provider.descriptor(); + let Some(resolves) = descriptor.resolves() else { + return BTreeMap::new(); + }; + input + .catalog + .sources_of(descriptor.id()) + .iter() + .filter_map(|source| { + let kind = source_kind(input, view, facts, source)?; + let listed = resolves.kinds().contains(&kind); + let met = resolves.needs(kind).is_subset(&facts.available_inputs); + (listed && met).then(|| (source.clone(), kind)) + }) + .collect() +} + +fn source_kind( + input: &EligibilityInput<'_>, + view: &EligibilityView, + facts: &RequestFacts, + source: &SourceDomain, +) -> Option { + let id = input.provider.descriptor().id(); + let record = view.record(source); + match input.catalog.owner_of(source) { + Some(owner) if owner == id => match record { + None => Some(ResolveKind::Initial), + Some(record) => { + owned_record_is_due(input, view, facts, record).then_some(ResolveKind::Refresh) + } + }, + // Config validation lets one provider own a source, so another + // owner means this provider may not resolve it. + Some(_) => None, + None => { + let present = + record.is_some_and(|record| has_unexpired_id(input.catalog, record, facts.now)); + (!present).then_some(ResolveKind::Initial) + } + } +} + +/// Whether a stored record of an owned source is due (spec 4.7 step 4). +fn owned_record_is_due( + input: &EligibilityInput<'_>, + view: &EligibilityView, + facts: &RequestFacts, + record: &KvPartnerId, +) -> bool { + let id = input.provider.descriptor().id(); + let now = facts.now.as_secs(); + let invalidated = record.w.is_none() + && record + .eids + .iter() + .flat_map(|entry| &entry.uids) + .any(|uid| view.has_invalidated_hash(id, &uid_hash(&uid.id))); + if invalidated { + return false; + } + if record.exp.is_some_and(|exp| now >= exp) || record.w.as_deref() != Some(id.as_str()) { + return true; + } + match record.ra { + Some(refresh_at) => now >= refresh_at, + None => record.at.is_none_or(|acquired| { + acquired.saturating_add(input.settings.refresh_after_sec) <= now + }), + } +} + +/// Whether KV holds an unexpired ID in this record for an unowned source. A +/// record hidden only by consent or an opt-out still counts: `FillMissing` +/// would not replace it. +fn has_unexpired_id(catalog: &ProviderCatalog, record: &KvPartnerId, now: UnixTime) -> bool { + if record.eids.is_empty() || record.exp.is_some_and(|exp| now.as_secs() >= exp) { + return false; + } + record.w.as_deref().is_none_or(|writer| { + writer + .parse::() + .is_ok_and(|writer| catalog.is_configured(&writer)) + }) +} + +/// What one provider call came to. +#[derive(Debug)] +pub(crate) enum CallOutcome { + /// The provider returned a resolution. + Resolved(Resolution), + /// `request` returned `None`. + Skipped, + /// A declared input was unavailable. + InputUnavailable, + /// The stored state did not decode, and the resolution the provider + /// still returned, if any. + InvalidState(Option), + /// The hold budget ran out before the answer. + HoldCutOff, + /// The call failed because of this visitor's input (a 4xx other than 429, + /// `Config`, `UndeclaredInput` or `Malformed`). + InputFailure, + /// A transport failure, timeout, 429 or 5xx. + TransportFailure, + /// The circuit breaker was open, so nothing was sent. + BreakerOpen, + /// The burst cap stopped the call. + BurstCapped, + /// A limiter error meant nothing was sent. + NotDispatched, + /// The request's backend budget had no room, so nothing was sent. + OverBackendBudget, +} + +impl CallOutcome { + /// Whether the call produced a positive answer: a resolution that carries + /// at least one ID. + #[must_use] + pub(crate) fn is_found(&self) -> bool { + matches!( + self, + Self::Resolved(resolution) | Self::InvalidState(Some(resolution)) + if !resolution.eids().is_empty() + ) + } +} + +/// What an outcome needs besides the provider and the view. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +pub(crate) struct OutcomeContext<'a> { + pub(crate) now: UnixTime, + pub(crate) catalog: &'a ProviderCatalog, + pub(crate) settings: &'a ProviderSettings, + pub(crate) identity: &'a IdentitySettings, + /// The state revision the call read, for a provider with private state. + pub(crate) expected_state: Option, + /// The consent bits an opt-out records, from [`super::policy::fingerprint_bits`]. + pub(crate) fingerprint: &'a str, +} + +/// The stored changes of one outcome. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct OutcomeChanges { + /// Every removal, and the positive changes of a provider without private + /// state. + pub(crate) changes: Vec, + /// The positive changes, state change, `n` and `f` of a provider with + /// private state. They apply as one unit. + pub(crate) result: Option, + /// `n` and `f` of a provider without a result. + pub(crate) state: ProviderStateChange, +} + +/// The stored changes of an outcome: the table of spec 8, row by row. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +#[must_use] +pub(crate) fn outcome_to_changes( + provider: &dyn IdentityProvider, + outcome: CallOutcome, + view: &EligibilityView, + context: &OutcomeContext<'_>, +) -> OutcomeChanges { + let id = provider.descriptor().id(); + let now = context.now.as_secs(); + match outcome { + CallOutcome::Resolved(resolution) => resolved_changes(provider, &resolution, view, context), + CallOutcome::Skipped | CallOutcome::InputUnavailable => { + wait_until(now.saturating_add(context.settings.refresh_after_sec)) + } + CallOutcome::InvalidState(resolution) => { + let mut changes = resolution.map_or_else(OutcomeChanges::default, |resolution| { + resolved_changes(provider, &resolution, view, context) + }); + let slot = state_slot(&mut changes); + slot.st = StateUpdate::Clear; + slot.f = None; + if slot.n.is_none() { + slot.n = Some(now.saturating_add(context.settings.refresh_after_sec)); + } + changes + } + CallOutcome::HoldCutOff => wait_until(now.saturating_add(context.identity.hold_retry_sec)), + CallOutcome::InputFailure => failure( + view.failures(id), + now, + INPUT_BACKOFF_BASE_SECS, + INPUT_BACKOFF_CAP_SECS, + ), + CallOutcome::TransportFailure => failure( + view.failures(id), + now, + TRANSPORT_BACKOFF_BASE_SECS, + TRANSPORT_BACKOFF_CAP_SECS, + ), + CallOutcome::BurstCapped => wait_until(now.saturating_add(BURST_WAIT_SECS)), + CallOutcome::BreakerOpen | CallOutcome::NotDispatched | CallOutcome::OverBackendBudget => { + OutcomeChanges::default() + } + } +} + +/// The state change the outcome's `st`, `n` and `f` belong to: the result's +/// when there is one, the stand-alone one otherwise. +fn state_slot(changes: &mut OutcomeChanges) -> &mut ProviderStateChange { + match changes.result.as_mut() { + Some(result) => &mut result.state, + None => &mut changes.state, + } +} + +fn wait_until(next: u64) -> OutcomeChanges { + OutcomeChanges { + state: ProviderStateChange { + n: Some(next), + ..ProviderStateChange::default() + }, + ..OutcomeChanges::default() + } +} + +/// `base × 2^(failures-1)`, at most `cap`, in seconds. +fn backoff_secs(failures: u32, base: u64, cap: u64) -> u64 { + base.saturating_mul(2_u64.saturating_pow(failures.saturating_sub(1))) + .min(cap) +} + +fn failure(failures_before: u32, now: u64, base: u64, cap: u64) -> OutcomeChanges { + let failures = failures_before.saturating_add(1); + OutcomeChanges { + state: ProviderStateChange { + n: Some(now.saturating_add(backoff_secs(failures, base, cap))), + f: Some(failures), + ..ProviderStateChange::default() + }, + ..OutcomeChanges::default() + } +} + +/// Whether the provider declares private state. +fn keeps_private_state(descriptor: &ProviderDescriptor) -> bool { + descriptor + .resolves() + .is_some_and(Resolves::has_private_state) +} + +/// How `Resolved(resolution)` becomes changes (spec 4.5 and 8). +fn resolved_changes( + provider: &dyn IdentityProvider, + resolution: &Resolution, + view: &EligibilityView, + context: &OutcomeContext<'_>, +) -> OutcomeChanges { + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let now = context.now.as_secs(); + let stateful = keeps_private_state(descriptor); + + let (kept, dropped_unclaimed) = drop_unclaimed(id, resolution.eids(), context.catalog); + let mut sets = entry_sets(&kept, context.now); + log_entry_sets(id, &sets); + let discarded = discarded_sets(descriptor, &mut sets, view); + let malformed = resolution.state_rejected() + || reject_positive_part(stateful, &discarded) + || all_sets_discarded(&sets, &discarded, dropped_unclaimed); + if malformed { + log::warn!("Identity provider '{id}' returned a result the coordinator rejects"); + } + + let positive = if malformed { + Vec::new() + } else { + positive_changes(provider, &sets, &discarded, view, context) + }; + let mut removals = invalidations(id, resolution, view, context.catalog); + let removed_any = !removals.is_empty(); + let mut state = ProviderStateChange::default(); + if resolution.is_opted_out() { + removals.extend(opt_out_removals(id, view, context.catalog)); + state.opt_out = Some(KvOptOut { + at: now, + fp: context.fingerprint.to_owned(), + }); + state.st = StateUpdate::Clear; + } + + let mut next = None; + if malformed { + let failures = view.failures(id).saturating_add(1); + state.f = Some(failures); + next = Some(now.saturating_add(backoff_secs( + failures, + INPUT_BACKOFF_BASE_SECS, + INPUT_BACKOFF_CAP_SECS, + ))); + } else { + if !resolution.eids().is_empty() { + state.f = Some(0); + } + if stateful && state.st == StateUpdate::Keep { + state.st = state_update(id, resolution); + } + if resolution.is_no_match() { + let wait = resolution + .retry_in() + .map_or(NO_MATCH_WAIT_SECS, |delay| delay.as_secs()) + .min(MAX_RETRY_SECS); + next = Some(now.saturating_add(wait)); + } else if removed_any { + next = Some(now); + } + } + if let Some(delay) = resolution.retry_in() { + let earliest = now.saturating_add(delay.as_secs().min(MAX_RETRY_SECS)); + next = Some(next.map_or(earliest, |current| current.max(earliest))); + } + state.n = next; + + if stateful && !malformed { + let result = (!positive.is_empty() || state != ProviderStateChange::default()).then(|| { + ProviderResult { + provider: id.clone(), + ids: positive, + state, + expected_state: context.expected_state.clone(), + } + }); + return OutcomeChanges { + changes: removals, + result, + state: ProviderStateChange::default(), + }; + } + removals.extend(positive); + OutcomeChanges { + changes: removals, + result: None, + state, + } +} + +/// Step 1: drops the entries of sources the provider does not claim after +/// narrowing. Logs each source with the provider ID only. Also reports +/// whether it dropped anything, which step 5 needs. +fn drop_unclaimed( + provider: &IdentityProviderId, + eids: &[ResolvedEid], + catalog: &ProviderCatalog, +) -> (Vec, bool) { + let claimed = catalog.sources_of(provider); + let mut dropped = false; + let kept = eids + .iter() + .filter(|eid| { + let keep = claimed.contains(eid.source()); + if !keep { + dropped = true; + log::debug!( + "Dropping an entry of identity provider '{provider}' for unclaimed source '{}'", + eid.source() + ); + } + keep + }) + .cloned() + .collect(); + (kept, dropped) +} + +/// Step 2: logs each dropped part and oversized source, never a value. +fn log_entry_sets(provider: &IdentityProviderId, sets: &EntrySets) { + for drop in &sets.drops { + log::debug!( + "Identity provider '{provider}' source '{}': dropped {}", + drop.source, + drop.part + ); + } + for source in &sets.oversized { + log::warn!( + "Identity provider '{provider}' source '{source}': discarded an oversized entry set" + ); + } +} + +/// Step 3: the sources whose set is discarded. An oversized set is. A set of +/// an expiring source that carries no expiry keeps the stored `exp` when its +/// entries equal the stored `eids`, and is discarded otherwise. +fn discarded_sets( + descriptor: &ProviderDescriptor, + sets: &mut EntrySets, + view: &EligibilityView, +) -> BTreeSet { + let mut discarded: BTreeSet = sets.oversized.iter().cloned().collect(); + for (source, set) in &mut sets.sets { + if !descriptor.is_expiring(source) || set.expires_at.is_some() { + continue; + } + let stored_exp = view + .record(source) + .filter(|record| record.eids == set.entries) + .and_then(|record| record.exp); + match stored_exp { + Some(exp) => set.expires_at = Some(UnixTime::from_secs(exp)), + None => { + log::warn!( + "Identity provider '{}' source '{source}': discarded a set that carries no expiry", + descriptor.id() + ); + discarded.insert(source.clone()); + } + } + } + discarded +} + +/// Step 4: whether a discarded set rejects the whole positive part. It does +/// for a provider with private state, whose IDs and state apply as a unit. +fn reject_positive_part(stateful: bool, discarded: &BTreeSet) -> bool { + stateful && !discarded.is_empty() +} + +/// Step 5: whether the result named a source and every set is discarded, +/// including by dropping entries of unclaimed sources (spec 4.5). +fn all_sets_discarded( + sets: &EntrySets, + discarded: &BTreeSet, + dropped_unclaimed: bool, +) -> bool { + sets.all_discarded() + || (sets.sets.is_empty() && dropped_unclaimed) + || (!sets.sets.is_empty() && sets.sets.keys().all(|source| discarded.contains(source))) +} + +/// Steps 6 to 8: each kept set becomes one record, written whole. +fn positive_changes( + provider: &dyn IdentityProvider, + sets: &EntrySets, + discarded: &BTreeSet, + view: &EligibilityView, + context: &OutcomeContext<'_>, +) -> Vec { + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let stateful = keeps_private_state(descriptor); + sets.sets + .iter() + .filter(|(source, _)| !discarded.contains(*source)) + .map(|(source, set)| { + if set.refresh_until.is_some() && !stateful { + log::warn!( + "Identity provider '{id}' source '{source}': ignoring refresh_until without private state" + ); + } + let record = record_of(id, set, stateful, context.now); + if context.catalog.owner_of(source) == Some(id) { + PartnerIdChange::OwnerWrite { + source: source.clone(), + record, + expected: view.record(source).map(record_revision), + } + } else { + PartnerIdChange::FillMissing { + source: source.clone(), + record, + } + } + }) + .collect() +} + +fn record_of( + provider: &IdentityProviderId, + set: &EntrySet, + stateful: bool, + now: UnixTime, +) -> KvPartnerId { + KvPartnerId { + w: Some(provider.to_string()), + rv: None, + at: Some(now.as_secs()), + exp: set.expires_at.map(UnixTime::as_secs), + ra: set.refresh_at.map(UnixTime::as_secs), + ru: stateful + .then_some(set.refresh_until) + .flatten() + .map(UnixTime::as_secs), + eids: set.entries.clone(), + } +} + +/// Step 9: an `OwnerRemove` for each claimed source the result invalidated +/// that has a record. It carries no hash: applying it hashes the live record. +fn invalidations( + provider: &IdentityProviderId, + resolution: &Resolution, + view: &EligibilityView, + catalog: &ProviderCatalog, +) -> Vec { + let claimed = catalog.sources_of(provider); + resolution + .invalidated() + .iter() + .filter_map(|source| { + if !claimed.contains(source) { + log::debug!( + "Ignoring invalidation by identity provider '{provider}' of unclaimed source '{source}'" + ); + return None; + } + Some(PartnerIdChange::OwnerRemove { + source: source.clone(), + provider: provider.clone(), + expected: record_revision(view.record(source)?), + }) + }) + .collect() +} + +/// Step 10: a `WriterRemove` for every record this provider wrote, and for a +/// capture-only provider every record of its sources that has no writer. +fn opt_out_removals( + provider: &IdentityProviderId, + view: &EligibilityView, + catalog: &ProviderCatalog, +) -> Vec { + let capture_only = catalog.capture_only_sources(provider); + view.records() + .filter(|(source, record)| match record.w.as_deref() { + Some(writer) => writer == provider.as_str(), + None => capture_only.contains(source), + }) + .map(|(source, _)| PartnerIdChange::WriterRemove { + source: source.clone(), + writer: provider.clone(), + }) + .collect() +} + +/// The private-state change the resolution asks for. +fn state_update(provider: &IdentityProviderId, resolution: &Resolution) -> StateUpdate { + match resolution.state_change() { + StateChange::Keep => StateUpdate::Keep, + StateChange::Clear => StateUpdate::Clear, + StateChange::Replace(json) => match serde_json::from_str(json) { + Ok(value) => StateUpdate::Replace(value), + Err(_) => { + log::warn!("Identity provider '{provider}' returned state that is not JSON"); + StateUpdate::Keep + } + }, + } +} + +/// Everything a dispatch reads that is not per call. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +pub(crate) struct IdentityDeps<'a> { + pub(crate) settings: &'a Settings, + pub(crate) catalog: &'a ProviderCatalog, + pub(crate) keys: &'a IdentityKeys, + /// Draws record revisions when a mutation applies. + // Temporary: first read when the mutations are applied. + #[allow(dead_code)] + pub(crate) random: &'a dyn RandomSource, + pub(crate) services: &'a RuntimeServices, + pub(crate) rate_limiter: Option<&'a dyn RateLimiter>, +} + +/// One provider call to dispatch: the provider and the kind each of its +/// sources needs. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +#[derive(Debug, Clone)] +pub(crate) struct Call { + pub(crate) provider: IdentityProviderId, + pub(crate) kinds: BTreeMap, + /// This provider's stored private state `ps.st`, as JSON. Only this + /// call's `ResolveInput` sees it. + pub(crate) state_json: Option, +} + +/// A call that was sent and is waiting for its answer. +struct InFlightCall<'a> { + provider: &'a dyn IdentityProvider, + input: ResolveInput<'a>, + call_context: Option, + /// When the request was sent. + sent_at: Instant, + /// How long the backend waits for the first byte. + timeout: Duration, + /// The remaining hold budget, not the provider's `timeout_ms`, set the + /// timeout. + budget_limited: bool, +} + +/// What preparing one call came to. +enum Prepared<'a> { + /// The call is settled without being sent. + Settled(CallOutcome), + /// The call was sent. + Sent(Box>, PlatformPendingRequest), +} + +/// Whether the provider is a generic pull provider, the only kind given the +/// full EC ID. +fn is_generic_pull(provider: &dyn IdentityProvider) -> bool { + provider + .descriptor() + .resolves() + .is_some_and(Resolves::gets_ec_id) +} + +/// The order calls are dispatched in. Generic pull providers keep the +/// fairness rule of spec 10.4: sorted by source domain, then rotated left by +/// `(now / 3600) % len`, so the starting partner changes each hour when +/// `max_concurrency` limits a batch. Every other provider follows, by ID. +fn dispatch_order(calls: Vec, catalog: &ProviderCatalog, now: UnixTime) -> Vec { + let (mut generic, mut others): (Vec, Vec) = calls.into_iter().partition(|call| { + catalog + .get(&call.provider) + .is_some_and(|provider| is_generic_pull(provider.as_ref())) + }); + generic.sort_by(|left, right| { + let source = |call: &Call| { + catalog + .get(&call.provider) + .map(|provider| provider.descriptor().primary_source().clone()) + }; + source(left) + .cmp(&source(right)) + .then_with(|| left.provider.cmp(&right.provider)) + }); + if !generic.is_empty() { + let hour = usize::try_from(now.as_secs() / 3600).unwrap_or(0); + let offset = hour % generic.len(); + generic.rotate_left(offset); + } + others.sort_by(|left, right| left.provider.cmp(&right.provider)); + generic.extend(others); + generic +} + +/// Allows an `https` request to a declared host, with no `Cookie` header. +/// +/// # Errors +/// +/// [`IdentityProviderError::Config`] for any other request. +pub(crate) fn check_request( + request: &ProviderRequest, + descriptor: &ProviderDescriptor, +) -> Result<(), Report> { + if request.url().scheme() != "https" { + return Err(IdentityProviderError::config( + "a provider request must use https", + )); + } + let host = request.url().host_str().unwrap_or_default(); + let declared = descriptor.resolves().is_some_and(|resolves| { + resolves + .hosts() + .iter() + .any(|allowed| allowed.eq_ignore_ascii_case(host)) + }); + if !declared { + return Err(IdentityProviderError::config( + "a provider request must go to a declared host", + )); + } + if request.has_cookie_header() { + return Err(IdentityProviderError::config( + "a provider request must not set a Cookie header", + )); + } + Ok(()) +} + +/// Decodes a response body to at most `cap` bytes: identity, `gzip` or `br`. +/// +/// `None` when the decoded body is over `cap`, does not decode, or the +/// encoding is unknown. The cap applies while decoding, so a small body +/// cannot expand past it. +pub(crate) fn decode_body(encoding: Option<&str>, body: Vec, cap: usize) -> Option> { + let compression = match encoding.map(str::trim) { + None | Some("") => Compression::None, + Some(name) if name.eq_ignore_ascii_case("identity") => Compression::None, + Some(name) if name.eq_ignore_ascii_case("gzip") => Compression::Gzip, + Some(name) if name.eq_ignore_ascii_case("br") => Compression::Brotli, + Some(_) => return None, + }; + if compression == Compression::None { + return (body.len() <= cap).then_some(body); + } + let mut decoder = BodyStreamDecoder::new(compression, cap); + let mut decoded = decoder + .decode_chunk(bytes::Bytes::from(body)) + .ok()? + .to_vec(); + decoded.extend(decoder.finish().ok()?); + (decoded.len() <= cap).then_some(decoded) +} + +/// Whether the response declares a body over `cap`, or an unreadable length. +pub(crate) fn content_length_exceeds_limit( + response: &PlatformResponse, + cap: usize, + label: &str, +) -> bool { + let Some(value) = response.response.headers().get(header::CONTENT_LENGTH) else { + return false; + }; + let Some(length) = value + .to_str() + .ok() + .and_then(|value| value.parse::().ok()) + else { + log::warn!("'{label}' returned an unreadable Content-Length header, rejecting"); + return true; + }; + if length > cap { + log::warn!("'{label}' returned an oversized Content-Length ({length} bytes), rejecting"); + return true; + } + false +} + +/// Buffers a response body of at most `cap` bytes, as sent (still encoded). +/// `None` when the declared or actual length is over `cap`. +pub(crate) fn buffered_body( + response: PlatformResponse, + cap: usize, + label: &str, +) -> Option<(http::response::Parts, bytes::Bytes)> { + if content_length_exceeds_limit(&response, cap, label) { + return None; + } + let (parts, body) = response.response.into_parts(); + let body = body.into_bytes().unwrap_or_default(); + if body.len() > cap { + log::warn!( + "'{label}' returned an oversized response ({} bytes), rejecting", + body.len() + ); + return None; + } + Some((parts, body)) +} + +/// What waiting for one in-flight request came to. +pub(crate) enum Waited { + /// The response arrived. + Response(PlatformResponse), + /// The request failed or timed out. + Failed, + /// The caller stopped waiting, so the request was dropped. + Dropped, +} + +/// Waits for each in-flight request in turn and empties `in_flight`. All were +/// sent up front, so waiting in turn does not change concurrency. +/// +/// `keep_waiting` is asked before each wait. Once it returns `false`, every +/// request still pending is dropped without waiting. +pub(crate) async fn drain_in_flight( + in_flight: &mut Vec<(T, PlatformPendingRequest)>, + services: &RuntimeServices, + mut keep_waiting: impl FnMut() -> bool, +) -> Vec<(T, Waited)> { + let mut waiting = true; + let mut waited = Vec::with_capacity(in_flight.len()); + for (label, pending) in in_flight.drain(..) { + waiting = waiting && keep_waiting(); + if !waiting { + waited.push((label, Waited::Dropped)); + continue; + } + match services.http_client().wait(pending).await { + Ok(response) => waited.push((label, Waited::Response(response))), + Err(error) => { + log::warn!("Request failed for '{label}': {error:?}"); + waited.push((label, Waited::Failed)); + } + } + } + waited +} + +/// Logs a provider `Config` failure. Only the error's own context is logged, +/// never its attachments. +fn log_config(provider: &IdentityProviderId, error: &Report) { + log::error!( + "Identity provider '{provider}' configuration error: {}", + error.current_context() + ); +} + +/// The outcome a `request` error becomes. +fn request_error_outcome( + provider: &IdentityProviderId, + error: &Report, +) -> CallOutcome { + if matches!( + error.current_context(), + IdentityProviderError::InputUnavailable { .. } + ) { + return CallOutcome::InputUnavailable; + } + log_config(provider, error); + CallOutcome::InputFailure +} + +/// The outcome a `response` error becomes. +fn response_error_outcome(error: &Report) -> CallOutcome { + match error.current_context() { + IdentityProviderError::Rejected { status } + if *status == StatusCode::TOO_MANY_REQUESTS || status.is_server_error() => + { + CallOutcome::TransportFailure + } + IdentityProviderError::InvalidState => CallOutcome::InvalidState(None), + _ => CallOutcome::InputFailure, + } +} + +/// The first-byte timeout of a call, or `None` when a hold dispatch has no +/// budget left. +fn call_timeout( + deps: &IdentityDeps<'_>, + settings: &ProviderSettings, + remaining: Option, +) -> Option<(Duration, bool)> { + let policy = deps.services.backend().naming_policy(); + let millis = |value: u64| u32::try_from(value).unwrap_or(u32::MAX); + let mut budget_limited = false; + let timeout_ms = match remaining { + None => { + let configured = millis(settings.after_send_timeout_ms); + policy.canonicalize_transport_timeout_ms(configured, configured) + } + Some(remaining) => { + let configured = millis( + settings + .timeout_ms + .unwrap_or(deps.settings.ec.identity.hold_budget_ms), + ); + let remaining_ms = u32::try_from(remaining.as_millis()).unwrap_or(u32::MAX); + budget_limited = remaining_ms < configured; + policy.canonicalize_transport_timeout_ms(remaining_ms, configured) + } + }; + (timeout_ms > 0).then(|| (Duration::from_millis(u64::from(timeout_ms)), budget_limited)) +} + +/// The limiter key counting a provider's transport failures. +fn breaker_key(provider: &IdentityProviderId) -> String { + format!("identity-fail:{provider}") +} + +/// Whether the provider's circuit breaker is open: its failure count in the +/// last minute has reached `breaker_failures_per_minute`. A count that cannot +/// be read leaves the breaker closed. +fn breaker_open( + deps: &IdentityDeps<'_>, + provider: &IdentityProviderId, + settings: &ProviderSettings, +) -> bool { + let Some(limiter) = deps.rate_limiter else { + return false; + }; + match limiter.count(&breaker_key(provider)) { + Ok(count) => count >= settings.breaker_failures_per_minute, + Err(error) => { + log::warn!( + "Identity provider '{provider}' breaker count unreadable, treating it as closed: {}", + error.current_context() + ); + false + } + } +} + +/// Feeds a transport failure to the circuit breaker. Nothing else is +/// recorded: a hold cut-off is a budget decision, and an input failure only +/// backs off its own visitor. A limiter error is logged and the outcome kept. +fn record_failure(deps: &IdentityDeps<'_>, provider: &IdentityProviderId, outcome: &CallOutcome) { + if !matches!(outcome, CallOutcome::TransportFailure) { + return; + } + let Some(limiter) = deps.rate_limiter else { + return; + }; + if let Err(error) = limiter.record(&breaker_key(provider)) { + log::warn!( + "Identity provider '{provider}' failure not recorded for the breaker: {}", + error.current_context() + ); + } +} + +/// Whether one more call would pass `[ec.identity] max_backend_requests`. +/// A hold call runs before the origin fetch may have started and before +/// the auction, so it leaves room for the origin and one call per auction +/// provider. An adapter that does not count its requests is not budgeted. +fn over_backend_budget(deps: &IdentityDeps<'_>, hold: bool) -> bool { + let Some(started) = deps.services.http_client().requests_started() else { + return false; + }; + let auction = &deps.settings.auction; + let reserved = if hold { + 1 + if auction.enabled { + auction.providers.len() + } else { + 0 + } + } else { + 0 + }; + started + reserved >= deps.settings.ec.identity.max_backend_requests +} + +/// Whether the burst cap stops the call. `Err` when it could not be read. +fn burst_capped( + deps: &IdentityDeps<'_>, + provider: &IdentityProviderId, + ec_id: Option<&str>, + settings: &ProviderSettings, +) -> Result { + let Some(limiter) = deps.rate_limiter else { + return Ok(false); + }; + let Some(ec_id) = ec_id else { + log::debug!("Identity provider '{provider}' not dispatched: the request has no EC ID"); + return Err(()); + }; + let key = format!("identity:{provider}:{}", visitor_key(deps.keys, ec_id)); + limiter + .exceeded_per_minute(&key, settings.max_calls_per_minute) + .map_err(|error| { + log::warn!( + "Identity provider '{provider}' not dispatched, burst cap unreadable: {error:?}" + ); + }) +} + +/// Whether the per-IP ceiling stops the call: `max_calls_per_ip_per_minute` +/// calls to one provider from one client address, across every visitor +/// behind it, so minting a fresh EC per request cannot multiply calls. The +/// key holds a keyed hash of the address, never the address. `Err` when it +/// could not be read. A request without a client IP, or a ceiling of 0, is +/// not capped here; the per-visitor cap still applies. +fn ip_capped( + deps: &IdentityDeps<'_>, + provider: &IdentityProviderId, + settings: &ProviderSettings, +) -> Result { + let (Some(limiter), Some(ip)) = (deps.rate_limiter, deps.services.client_info().client_ip) + else { + return Ok(false); + }; + if settings.max_calls_per_ip_per_minute == 0 { + return Ok(false); + } + let key = format!( + "identity-ip:{provider}:{}", + visitor_key(deps.keys, &format!("ip:{ip}")) + ); + limiter + .exceeded_per_minute(&key, settings.max_calls_per_ip_per_minute) + .map_err(|error| { + log::warn!( + "Identity provider '{provider}' not dispatched, per-IP ceiling unreadable: {error:?}" + ); + }) +} + +/// Builds, checks and sends one call. +async fn prepare_call<'a>( + deps: &IdentityDeps<'a>, + call: &'a Call, + signals: &'a ResolveSignals<'a>, + started: Instant, + budget: Option, +) -> Prepared<'a> { + let Some(provider) = deps.catalog.get(&call.provider) else { + log::error!( + "Identity provider '{}' is not in the catalog, not dispatched", + call.provider + ); + return Prepared::Settled(CallOutcome::InputFailure); + }; + let provider: &dyn IdentityProvider = provider.as_ref(); + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let settings = deps.settings.ec.identity.provider(id); + + if breaker_open(deps, id, &settings) { + log::debug!("Identity provider '{id}' not dispatched: its circuit breaker is open"); + return Prepared::Settled(CallOutcome::BreakerOpen); + } + + let input = ResolveInput::new(descriptor, call.kinds.clone(), signals) + .with_state_json(call.state_json.clone()); + let request = provider.request(&input); + if !input.recorded_errors().is_empty() { + log::error!("Identity provider '{id}' read an input it did not declare"); + return Prepared::Settled(CallOutcome::InputFailure); + } + let request = match request { + Ok(Some(request)) => request, + Ok(None) => return Prepared::Settled(CallOutcome::Skipped), + Err(error) => return Prepared::Settled(request_error_outcome(id, &error)), + }; + if let Err(error) = check_request(&request, descriptor) { + log_config(id, &error); + return Prepared::Settled(CallOutcome::InputFailure); + } + + let remaining = budget.map(|budget| budget.saturating_sub(started.elapsed())); + let Some((timeout, budget_limited)) = call_timeout(deps, &settings, remaining) else { + return Prepared::Settled(CallOutcome::HoldCutOff); + }; + if over_backend_budget(deps, budget.is_some()) { + log::debug!( + "Identity provider '{id}' not dispatched: the request's backend budget is spent" + ); + return Prepared::Settled(CallOutcome::OverBackendBudget); + } + match burst_capped(deps, id, signals.ec_id.as_deref(), &settings) { + Ok(true) => return Prepared::Settled(CallOutcome::BurstCapped), + Ok(false) => {} + Err(()) => return Prepared::Settled(CallOutcome::NotDispatched), + } + match ip_capped(deps, id, &settings) { + Ok(true) => { + log::debug!("Identity provider '{id}' not dispatched: the per-IP ceiling is reached"); + return Prepared::Settled(CallOutcome::BurstCapped); + } + Ok(false) => {} + Err(()) => return Prepared::Settled(CallOutcome::NotDispatched), + } + + let url = request.url(); + let backend = deps.services.backend().ensure(&PlatformBackendSpec { + scheme: url.scheme().to_owned(), + host: url.host_str().unwrap_or_default().to_owned(), + port: url.port(), + host_header_override: None, + certificate_check: deps.settings.proxy.certificate_check, + first_byte_timeout: timeout, + between_bytes_timeout: timeout, + // A hold call must not wait past the budget to connect, so its + // connect timeout is capped at the call timeout (spec 3.4). + connect_timeout: budget.map(|_| timeout.min(HOLD_CONNECT_TIMEOUT_CAP)), + discriminator: Some(id.to_string()), + }); + let backend = match backend { + Ok(name) => name, + Err(error) => { + log::warn!("Identity provider '{id}': failed to resolve the backend: {error:?}"); + return Prepared::Settled(CallOutcome::TransportFailure); + } + }; + + let mut builder = http::Request::builder() + .method(request.method().clone()) + .uri(url.as_str()); + for (name, value) in request.headers() { + builder = builder.header(name, value); + } + let body = if request.body().is_empty() { + EdgeBody::empty() + } else { + EdgeBody::from(request.body().to_vec()) + }; + let Ok(outbound) = builder.body(body) else { + log::error!("Identity provider '{id}': the request could not be built"); + return Prepared::Settled(CallOutcome::InputFailure); + }; + let call_context = request.call_context_json().map(str::to_owned); + + match deps + .services + .http_client() + .send_async(PlatformHttpRequest::new(outbound, backend)) + .await + { + Ok(pending) => Prepared::Sent( + Box::new(InFlightCall { + provider, + input, + call_context, + sent_at: Instant::now(), + timeout, + budget_limited, + }), + pending, + ), + Err(error) => { + log::warn!("Identity provider '{id}': failed to send: {error:?}"); + Prepared::Settled(CallOutcome::TransportFailure) + } + } +} + +/// The outcome of a call whose response arrived. +fn finish_call( + call: &InFlightCall<'_>, + response: PlatformResponse, + max_response_bytes: usize, +) -> CallOutcome { + let id = call.provider.descriptor().id(); + let Some((parts, raw)) = buffered_body(response, max_response_bytes, id.as_str()) else { + return CallOutcome::TransportFailure; + }; + let encoding = parts + .headers + .get(header::CONTENT_ENCODING) + .and_then(|value| value.to_str().ok()); + let Some(body) = decode_body(encoding, raw.to_vec(), max_response_bytes) else { + log::warn!("Identity provider '{id}' returned a body that is over the cap or undecodable"); + return CallOutcome::TransportFailure; + }; + let response = + ProviderResponse::new(parts.status, parts.headers, body, call.call_context.clone()); + let result = call.provider.response(&response, &call.input); + if !call.input.recorded_errors().is_empty() { + log::error!("Identity provider '{id}' read an input it did not declare"); + return CallOutcome::InputFailure; + } + match result { + Ok(resolution) if call.input.state_decode_failed() => { + CallOutcome::InvalidState(Some(resolution)) + } + Ok(resolution) => CallOutcome::Resolved(resolution), + Err(error) => { + if matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ) { + log_config(id, &error); + } + response_error_outcome(&error) + } + } +} + +/// Dispatches `calls` and returns each provider's outcome (spec 4.7). +/// +/// At most `max_concurrency` calls are in flight. `budget` is `Some(the +/// remaining hold budget)` for a hold dispatch before the auction, and +/// `None` after the response is sent: there, a call that misses its timeout is +/// an ordinary transport failure. +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +pub(crate) async fn dispatch( + deps: &IdentityDeps<'_>, + calls: Vec, + signals: &ResolveSignals<'_>, + budget: Option, +) -> Vec<(IdentityProviderId, CallOutcome)> { + let started = Instant::now(); + let max_concurrency = deps.settings.ec.identity.max_concurrency.max(1); + let calls = dispatch_order(calls, deps.catalog, signals.now); + let mut outcomes = Vec::with_capacity(calls.len()); + let mut batch = InFlightBatch::default(); + let clock = DispatchClock { started, budget }; + + for call in &calls { + match prepare_call(deps, call, signals, started, budget).await { + Prepared::Settled(outcome) => { + record_failure(deps, &call.provider, &outcome); + outcomes.push((call.provider.clone(), outcome)); + } + Prepared::Sent(sent_call, pending) => { + batch.sent.insert(call.provider.clone(), sent_call); + batch.in_flight.push((call.provider.clone(), pending)); + if batch.in_flight.len() >= max_concurrency { + drain_batch(deps, &mut batch, clock, &mut outcomes).await; + } + } + } + } + drain_batch(deps, &mut batch, clock, &mut outcomes).await; + outcomes +} + +/// The calls [`dispatch`] has sent and not yet waited for. +#[derive(Default)] +struct InFlightBatch<'a> { + in_flight: Vec<(IdentityProviderId, PlatformPendingRequest)>, + sent: BTreeMap>>, +} + +/// When a [`dispatch`] started and the hold budget it shares, if any. +#[derive(Clone, Copy)] +struct DispatchClock { + started: Instant, + budget: Option, +} + +/// Waits for every call in `batch` and adds each outcome to `outcomes`. +async fn drain_batch( + deps: &IdentityDeps<'_>, + batch: &mut InFlightBatch<'_>, + clock: DispatchClock, + outcomes: &mut Vec<(IdentityProviderId, CallOutcome)>, +) { + let DispatchClock { started, budget } = clock; + let max_response_bytes = deps.settings.ec.identity.max_response_bytes; + let within_budget = || budget.is_none_or(|budget| started.elapsed() < budget); + for (id, waited) in drain_in_flight(&mut batch.in_flight, deps.services, within_budget).await { + let Some(call) = batch.sent.remove(&id) else { + continue; + }; + let outcome = match waited { + Waited::Response(response) => finish_call(&call, response, max_response_bytes), + // A failed wait is a cut off only when the hold budget was + // spent: the whole budget, or a timeout the remaining budget + // set. Any other failure counts against the provider. + Waited::Failed + if budget.is_some_and(|budget| started.elapsed() >= budget) + || (budget.is_some() + && call.budget_limited + && call.sent_at.elapsed() >= call.timeout) => + { + CallOutcome::HoldCutOff + } + Waited::Failed => CallOutcome::TransportFailure, + Waited::Dropped => CallOutcome::HoldCutOff, + }; + record_failure(deps, &id, &outcome); + outcomes.push((id, outcome)); + } +} + +/// Whether the visitor's consent now re-grants `descriptor`'s provider after +/// the sticky opt-out `opt_out` (spec 8). Each rule needs a denial the +/// opt-out's `fp` recorded to turn into a grant, so a CMP re-save that was +/// already granting never clears a vendor-reported opt-out. +/// +/// - A provider with a GVL ID: `fp` records a decoded TC string that denied +/// the vendor or a required purpose, and the current TC string grants the +/// vendor and every required purpose. TCF timestamps are day-level, so +/// `LastUpdated` is not compared. +/// - Any provider, under a GPP string with a readable US section: `fp` records +/// at least one US opt-out bit and the current fingerprint holds none of +/// the US opt-out bits `fp` recorded. Other bits may differ. +/// +/// A jurisdiction change, a newly present GPP string without a US section, a +/// relaxed signal or an `fp` that does not parse never clears. +#[must_use] +pub(crate) fn regrant_clears_opt_out( + descriptor: &ProviderDescriptor, + opt_out: &KvOptOut, + consent: &ConsentContext, + signals: PolicySignals, +) -> bool { + tcf_regrants(descriptor, opt_out, consent) + || gpp_regrants(descriptor, opt_out, consent, signals) +} + +/// Whether the opt-out's `fp` recorded a TCF denial of the provider's GVL +/// vendor or a purpose its policy requires, and the current TC string grants +/// them all. +fn tcf_regrants( + descriptor: &ProviderDescriptor, + opt_out: &KvOptOut, + consent: &ConsentContext, +) -> bool { + let policy = descriptor.consent_policy(); + let (Some(vendor), Some(tcf)) = (policy.tcf_vendor(), effective_tcf(consent)) else { + return false; + }; + let Ok(recorded) = u64::from_str_radix(&opt_out.fp, 16) else { + return false; + }; + let required = policy + .tcf_purposes() + .iter() + .filter(|purpose| (1..=FP_PURPOSES).contains(*purpose)) + .fold(1_u64 << FP_VENDOR, |mask, purpose| { + mask | 1 << (purpose - 1) + }); + let recorded_denial = recorded & (1 << FP_TCF) != 0 && recorded & required != required; + recorded_denial + && tcf.has_vendor_consent(vendor) + && policy + .tcf_purposes() + .iter() + .all(|purpose| tcf.has_purpose_consent(usize::from(*purpose))) +} + +/// Whether the GPP US section now shows none of the US opt-outs `fp` +/// recorded. GPP carries no creation time, so this is judged by section +/// values; without a readable US section nothing is judged. +fn gpp_regrants( + descriptor: &ProviderDescriptor, + opt_out: &KvOptOut, + consent: &ConsentContext, + signals: PolicySignals, +) -> bool { + let Some(gpp) = consent.gpp.as_ref() else { + return false; + }; + let has_us_section = gpp.us_sale_opt_out.is_some() + || gpp.us_sharing_opt_out.is_some() + || gpp.us_targeted_advertising_opt_out.is_some() + || gpp.us_privacy_section.is_some(); + if gpp.us_section_decode_failed || !has_us_section { + return false; + } + let Ok(recorded) = u64::from_str_radix(&opt_out.fp, 16) else { + return false; + }; + if recorded & FP_US_OPT_OUT_MASK == 0 { + return false; + } + let Ok(current) = u64::from_str_radix( + &fingerprint_bits(descriptor.consent_policy(), consent, signals), + 16, + ) else { + return false; + }; + current & recorded & FP_US_OPT_OUT_MASK == 0 +} + +/// The providers whose sticky opt-out the request's consent re-grants, each +/// with its clear's watermark: the `at` of the opt-out the request read +/// (spec 8). An opt-out written after that read is newer and stays. +#[must_use] +pub(crate) fn opt_out_clears( + view: &EligibilityView, + catalog: &ProviderCatalog, + consent: &ConsentContext, + signals: PolicySignals, +) -> Vec<(IdentityProviderId, u64)> { + catalog + .providers() + .filter_map(|provider| { + let descriptor = provider.descriptor(); + let state = view.providers.get(descriptor.id())?; + let opt_out = state.o.as_ref()?; + (view.is_opted_out(descriptor.id()) + && regrant_clears_opt_out(descriptor, opt_out, consent, signals)) + .then(|| (descriptor.id().clone(), opt_out.at)) + }) + .collect() +} + +/// Runs [`run_hold`] with the request's runtime and limiter. `None` when the +/// adapter set no identity runtime (no provider is configured) or no limiter +/// (fails closed). +pub(crate) async fn hold_before_auction( + settings: &Settings, + services: &RuntimeServices, + context: &mut EcContext, +) -> Option { + let runtime = context.identity().runtime().cloned()?; + let Some(limiter) = context.identity().rate_limiter() else { + // Without a limiter the burst cap and breaker cannot hold, so fail + // closed: dispatch nothing before the response. + log::warn!("Skipping the identity hold dispatch: no rate limiter is set for the request"); + return None; + }; + let random = ThreadRandom; + let deps = runtime.deps( + settings, + services, + &random, + Some(limiter.as_ref() as &dyn RateLimiter), + ); + Some(run_hold(&deps, context).await) +} + +/// What the hold dispatch before a navigation's auction learned (spec 4.7). +/// +/// The answers this auction's server set adds, per source, and the provider +/// that wrote each, so the output rules can apply that provider's provenance +/// default. Nothing here is stored: the same answers ride the writer's one +/// flush. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct HoldResults { + /// The entry set each source now holds, in the order the provider + /// returned it. + pub(crate) additions: BTreeMap>, + /// The provider that wrote each source in `additions`. + pub(crate) writer: BTreeMap, +} + +/// Dispatches the hold-mode providers of a returning writer navigation that +/// runs an auction (spec 4.7). +/// +/// Every provider that [`eligibility`] passes in [`Phase::Hold`] on the phase 1 +/// view is called within `hold_budget_ms`, measured from the first dispatch, +/// each waiting at most `min(hold_timeout_ms, remaining budget)`. The phase 1 +/// read has already confirmed a live, consenting row, so no EC ID is +/// disclosed before consent is confirmed. Each outcome joins the writer's +/// mutation, and every answer that mutation would store is returned for this +/// auction. No answer goes into a cookie. +/// +/// A new visitor's writer navigation, whose EC was minted with +/// [`EcContext::mint_deferred`], dispatches on the row it will create, +/// skipping every provider that reads the EC ID, since no row backs it yet. +/// Its answers go into that row instead of a mutation, so the create is the +/// one write (spec 4.7, new visitors). +/// +/// A request that is neither, runs no auction, or whose phase 1 read is not a +/// live, consenting row dispatches nothing. +pub(crate) async fn run_hold(deps: &IdentityDeps<'_>, context: &mut EcContext) -> HoldResults { + let mut results = HoldResults::default(); + let new_visitor = context.identity().writer() && context.deferred_entry().is_some(); + if !context.identity().runs_auction() || !(new_visitor || may_flush(context)) { + return results; + } + let Some(ec_id) = context.ec_value().map(str::to_owned) else { + return results; + }; + let entry = if new_visitor { + context.deferred_entry().cloned() + } else { + context + .kv_snapshot() + .entry_for(&ec_id) + .filter(|entry| entry.consent.ok) + .cloned() + }; + let Some(entry) = entry else { + return results; + }; + + let now = current_timestamp(); + let facts = request_facts(context, now, true); + context.identity_mut().set_hold_dispatched(); + let view = EligibilityView::from_kv(&entry); + let calls: Vec = deps + .catalog + .providers() + .filter_map(|provider| { + let id = provider.descriptor().id(); + let reads_ec_id = provider + .descriptor() + .resolves() + .is_some_and(Resolves::gets_ec_id); + if new_visitor && reads_ec_id { + return None; + } + let provider_settings = deps.settings.ec.identity.provider(id); + let Eligibility::Eligible { kinds } = eligibility( + &EligibilityInput { + provider: provider.as_ref(), + settings: &provider_settings, + identity: &deps.settings.ec.identity, + catalog: deps.catalog, + consent: context.consent(), + }, + &view, + &facts, + Phase::Hold, + ) else { + return None; + }; + Some(Call { + provider: id.clone(), + kinds, + state_json: view + .state(id) + .and_then(|state| serde_json::to_string(state).ok()), + }) + }) + .collect(); + if calls.is_empty() { + return results; + } + + let mut held = IdentityMutation::default(); + add_dispatch_results( + &mut held, + deps, + &DispatchInput { + context, + ec_id: &ec_id, + live: &entry, + view: &view, + facts: &facts, + }, + DispatchRun { + calls, + budget: Some(Duration::from_millis( + deps.settings.ec.identity.hold_budget_ms, + )), + }, + ) + .await; + + // Computed on the phase 1 row; the flush applies the same mutation to the + // phase 4 live row, which may differ. + let apply_context = ApplyContext { + now, + random: deps.random, + catalog: deps.catalog, + }; + let applied = apply(&held, &entry, &apply_context); + if let Some(applied) = &applied { + for (source, record) in &applied.entry.ids { + if entry.ids.get(source) == Some(record) { + continue; + } + let (Ok(source), Some(writer)) = ( + source.parse::(), + record.w.as_deref().and_then(|writer| writer.parse().ok()), + ) else { + log::debug!( + "Skipping a hold answer for source '{source}': no parseable writer on the record" + ); + continue; + }; + results + .additions + .insert(source.clone(), record.eids.clone()); + results.writer.insert(source, writer); + } + } + if new_visitor { + if let Some(applied) = applied { + context.set_deferred_entry(applied.entry); + } + } else { + context.identity_mut().mutation_mut().absorb(held); + } + results +} + +/// One proxied response offered for capture (spec 4.7). +/// +/// There is no request body here: capture never reads it. +pub(crate) struct CaptureCall<'a> { + pub(crate) method: &'a http::Method, + pub(crate) path: &'a str, + pub(crate) status: StatusCode, + pub(crate) headers: &'a http::HeaderMap, + /// The decoded body, at most `max_response_bytes`. + pub(crate) body: &'a [u8], +} + +/// Whether any enabled provider's capture rule matches the request. +#[must_use] +pub(crate) fn has_capture_rule( + catalog: &ProviderCatalog, + method: &http::Method, + path: &str, +) -> bool { + catalog.providers().any(|provider| { + provider + .descriptor() + .captures() + .is_some_and(|captures| captures.matches(method, path)) + }) +} + +/// What a capture reads of the row. +struct CaptureView { + view: EligibilityView, + /// The row could not be read, so no removal can be built. + unreadable: bool, + /// Reads this view made. + reads: u32, +} + +/// The row a capture reads its view from: the request's own snapshot when it +/// holds one for this EC ID, else one read. Empty when the row is missing or +/// not a live row. +fn capture_view( + graph: Option<&KvIdentityGraph>, + context: &EcContext, + ec_id: Option<&str>, +) -> CaptureView { + let empty = |unreadable, reads| CaptureView { + view: EligibilityView::empty(), + unreadable, + reads, + }; + let (Some(graph), Some(ec_id)) = (graph, ec_id) else { + return empty(false, 0); + }; + let (snapshot, reads) = if context.kv_snapshot().belongs_to(ec_id) { + (context.kv_snapshot().clone(), 0) + } else { + (graph.load_snapshot(ec_id), 1) + }; + match snapshot { + EcKvSnapshot::Present { entry, .. } if entry.consent.ok => CaptureView { + view: EligibilityView::from_kv(&entry), + unreadable: false, + reads, + }, + EcKvSnapshot::Failed { .. } => empty(true, reads), + _ => empty(false, reads), + } +} + +/// Counts and logs the source invalidations a capture result asked for when +/// the row could not be read: each needs the record's revision, so none can be +/// built or retried (spec 5.3). An opt-out needs no view and is not counted +/// here. Holds no value. +fn lose_unbuildable_invalidations( + context: &mut EcContext, + ec_id: Option<&str>, + provider: &IdentityProviderId, + resolution: &Resolution, + catalog: &ProviderCatalog, +) { + let claimed = catalog.sources_of(provider); + for source in resolution + .invalidated() + .iter() + .filter(|source| claimed.contains(source)) + { + log::error!( + "Identity removal lost for '{}': provider '{provider}', source '{source}'", + log_id(ec_id.unwrap_or_default()), + ); + context.identity_mut().record_removal_lost(); + } +} + +/// Whether eligibility steps 1 to 3 of spec 4.7 pass for the request. +fn passes_consent_steps(descriptor: &ProviderDescriptor, context: &EcContext, now: u64) -> bool { + let facts = request_facts(context, now, false); + facts.is_real_browser + && facts.ec_persistence_allowed + && policy_allows( + descriptor.consent_policy(), + context.consent(), + facts.policy_signals, + ) +} + +/// What one capture result asks for, once the real-browser gate has passed. +struct CaptureAsk { + removals: Vec, + state: ProviderStateChange, + /// Positive changes that pass the consent steps and differ from what KV + /// holds, for the writer's flush or the pending key (spec 5.3). + positives: Vec, +} + +/// The source and record of a positive change. +fn positive_record(change: &PartnerIdChange) -> Option<(&SourceDomain, &KvPartnerId)> { + match change { + PartnerIdChange::FillMissing { source, record } + | PartnerIdChange::OwnerWrite { source, record, .. } => Some((source, record)), + PartnerIdChange::OwnerRemove { .. } | PartnerIdChange::WriterRemove { .. } => None, + } +} + +/// The sources the provider claims that a result carries IDs for, once each. +fn positive_sources( + provider: &IdentityProviderId, + resolution: &Resolution, + catalog: &ProviderCatalog, +) -> BTreeSet { + let claimed = catalog.sources_of(provider); + resolution + .eids() + .iter() + .map(|eid| eid.source().clone()) + .filter(|source| claimed.contains(source)) + .collect() +} + +/// Reads one result for its removals, opt-out and clear, and for the +/// positive changes worth keeping. +fn capture_ask( + deps: &IdentityDeps<'_>, + context: &EcContext, + provider: &dyn IdentityProvider, + resolution: Resolution, + view: &EligibilityView, + positive: bool, +) -> CaptureAsk { + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let now = UnixTime::from_secs(current_timestamp()); + if resolution.changes_state() { + log::warn!("Identity provider '{id}' returned a state change from capture, ignoring it"); + } + let opted_out = resolution.is_opted_out(); + let facts = request_facts(context, now.as_secs(), false); + let fingerprint = fingerprint_bits( + descriptor.consent_policy(), + context.consent(), + facts.policy_signals, + ); + let provider_settings = deps.settings.ec.identity.provider(id); + let changes = outcome_to_changes( + provider, + CallOutcome::Resolved(resolution), + view, + &OutcomeContext { + now, + catalog: deps.catalog, + settings: &provider_settings, + identity: &deps.settings.ec.identity, + expected_state: None, + fingerprint: &fingerprint, + }, + ); + let OutcomeChanges { + changes, + result, + state, + } = changes; + let (resulting, result_ids) = match result { + Some(result) => (result.state, result.ids), + None => (state, Vec::new()), + }; + let (removals, positives): (Vec<_>, Vec<_>) = changes + .into_iter() + .chain(result_ids) + .partition(PartnerIdChange::is_removal); + let keeps_positives = positive + && !opted_out + && !view.is_opted_out(id) + && passes_consent_steps(descriptor, context, now.as_secs()); + let mut ask = CaptureAsk { + removals, + state: ProviderStateChange::default(), + positives: if keeps_positives { + positives + .into_iter() + .filter(|change| { + positive_record(change).is_some_and(|(source, record)| { + view.record(source) + .is_none_or(|stored| stored.eids != record.eids) + }) + }) + .collect() + } else { + Vec::new() + }, + }; + if let Some(opt_out) = resulting.opt_out.filter(|_| opted_out) { + ask.state.opt_out = Some(opt_out); + ask.state.st = StateUpdate::Clear; + } + // Only a capture-only provider without a GVL ID has no other way to clear: + // a vendor that issues again has been re-consented through (spec 8). + let only_capture_clears = + deps.catalog.is_capture_only(id) && descriptor.consent_policy().tcf_vendor().is_none(); + if positive + && only_capture_clears + && view.is_opted_out(id) + && passes_consent_steps(descriptor, context, now.as_secs()) + { + // Clear only the opt-out this request read (spec 8). + ask.state.opt_out_clear = view + .providers + .get(id) + .and_then(|state| state.o.as_ref()) + .map(|opt_out| opt_out.at); + } + ask +} + +/// Offers a proxied 2xx response to each provider whose capture rule matches +/// (spec 4.7). +/// +/// Capture is revoke-only. Removals and opt-outs are written at once with the +/// removal write, whatever the consent and whatever the client, as a consent +/// withdrawal is. A clear re-enables IDs, so only a real browser's clear is +/// written ([`passes_consent_steps`]). A positive ID is counted and never +/// stored. A `revoke_scope = "global"` opt-out asks finalize for the +/// consent-withdrawal tombstone path instead. +pub(crate) fn run_capture( + deps: &IdentityDeps<'_>, + graph: Option<&KvIdentityGraph>, + context: &mut EcContext, + call: &CaptureCall<'_>, +) { + let matching: Vec<&std::sync::Arc> = deps + .catalog + .providers() + .filter(|provider| { + provider + .descriptor() + .captures() + .is_some_and(|captures| captures.matches(call.method, call.path)) + }) + .collect(); + if matching.is_empty() { + return; + } + let ec_id = context.ec_value().map(str::to_owned); + let CaptureView { + view, + unreadable, + reads: view_reads, + } = capture_view(graph, context, ec_id.as_deref()); + context.identity_mut().record_capture_kv(view_reads, 0); + let response = + ProviderResponse::new(call.status, call.headers.clone(), call.body.to_vec(), None); + let consent = ProviderConsent::from_context(context.consent()); + let now = UnixTime::from_secs(current_timestamp()); + + let mut mutation = IdentityMutation::default(); + for provider in matching { + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let stored: BTreeMap = view + .records() + .filter(|(source, _)| deps.catalog.sources_of(id).contains(source)) + .filter_map(|(source, record)| { + StoredId::from_record(source, record).map(|stored| (source.clone(), stored)) + }) + .collect(); + let input = CaptureInput::new( + descriptor, + call.path, + &response, + &stored, + &consent, + &deps.settings.publisher.domain, + now, + ); + let resolution = match provider.capture(&input) { + Ok(resolution) => resolution, + Err(error) => { + if !error.current_context().is_not_implemented() { + log::warn!( + "Identity provider '{id}' failed to capture: {}", + error.current_context() + ); + } + continue; + } + }; + + let positive_sources = positive_sources(id, &resolution, deps.catalog); + for source in &positive_sources { + context + .identity_mut() + .record_unpersisted(id.clone(), source.clone()); + } + if resolution.is_opted_out() + && deps.settings.ec.identity.provider(id).revoke_scope() == RevokeScope::Global + { + context.request_global_revoke(); + continue; + } + + if unreadable { + lose_unbuildable_invalidations( + context, + ec_id.as_deref(), + id, + &resolution, + deps.catalog, + ); + } + let ask = capture_ask( + deps, + context, + provider.as_ref(), + resolution, + &view, + !positive_sources.is_empty(), + ); + for removal in ask.removals { + mutation.add_change(removal); + } + mutation.merge_state(id.clone(), ask.state); + // The writer's own flush stores them; any other request leaves them + // to the pending key after send (spec 5.3, Pending key). + if !unreadable { + for change in ask.positives { + if context.identity().writer() { + context.identity_mut().mutation_mut().add_change(change); + } else { + context + .identity_mut() + .capture_writes_mut() + .add_change(change); + } + } + } + } + + // The tombstone the finalize step writes supersedes every other removal. + if context.global_revoke_requested() || !mutation.has_removals() { + return; + } + let (Some(graph), Some(ec_id)) = (graph, ec_id.as_deref()) else { + log::debug!("Dropping capture removals: no identity graph or EC ID for the request"); + return; + }; + let apply_context = ApplyContext { + now: now.as_secs(), + random: deps.random, + catalog: deps.catalog, + }; + let (outcome, cost) = write_removals_counted( + graph, + ec_id, + &mutation, + &apply_context, + RemovalTiming::BeforeSend, + ); + context + .identity_mut() + .record_capture_kv(cost.reads, cost.writes); + if outcome == RemovalWriteOutcome::Failed { + if let Some(earlier) = context.identity().pending_removals().cloned() { + mutation.absorb(earlier); + } + context.identity_mut().set_pending_removals(Some(mutation)); + } +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + use std::sync::Arc; + use std::time::Duration; + + use std::io::Write as _; + + use serde_json::json; + + use super::*; + use crate::consent::types::GppConsent; + use crate::consent::us_privacy::decode_us_privacy; + use crate::ec::identity::descriptor::{ + ConsentPolicy, Input, ProviderDescriptor, ResolveKind, Resolves, UsOptOut, + }; + use crate::ec::identity::flush::defer_capture_writes; + use crate::ec::identity::input::{ProviderConsent, SeededRandom, ThreadRandom}; + use crate::ec::identity::pending::{load_pending, pending_key}; + use crate::ec::identity::policy::fingerprint_bits; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::resolution::{ResolvedEid, ResolvedUid}; + use crate::ec::identity::revision::{record_revision, uid_hash}; + use crate::ec::identity::test_support::FakeRateLimiter; + use crate::ec::identity::test_support::fake_http::{ + FakeBackend, FakeHttpClient, ScriptedProvider, fake_services, fake_services_from, + }; + use crate::ec::identity::test_support::{ + OutcomeFixture, allowing_consent, facts_for_writer, fixed_capture_provider, fixed_provider, + pid, provider_from, record_for_test, resolver_policy, source, stateful_fixed_provider, + test_consent_gdpr_without_vendor, + }; + use crate::ec::kv_types::{KvEid, KvUid}; + use crate::platform::ClientInfo; + use crate::settings::ProviderMode; + use crate::test_support::tests::create_test_settings; + + /// The six lines every eligibility test repeats: one provider, its + /// catalog and the settings and consent the call reads. + struct EligibilityFixture { + provider: Arc, + catalog: ProviderCatalog, + settings: ProviderSettings, + identity: IdentitySettings, + consent: ConsentContext, + } + + impl EligibilityFixture { + /// A catalog holding `provider` alone, with no source owners. + fn new(provider: &Arc) -> Self { + Self::with_catalog( + provider, + ProviderCatalog::new(vec![provider.clone()], BTreeMap::new()), + ) + } + + /// A catalog holding `provider` alone, owning each of `sources`. + fn owning(provider: &Arc, sources: &[&str]) -> Self { + let owners = sources + .iter() + .map(|name| (source(name), provider.descriptor().id().clone())) + .collect(); + Self::with_catalog( + provider, + ProviderCatalog::new(vec![provider.clone()], owners), + ) + } + + fn with_catalog(provider: &Arc, catalog: ProviderCatalog) -> Self { + Self { + provider: provider.clone(), + catalog, + settings: ProviderSettings::default(), + identity: IdentitySettings::default(), + consent: allowing_consent(), + } + } + + fn with_settings(mut self, settings: ProviderSettings) -> Self { + self.settings = settings; + self + } + + fn with_consent(mut self, consent: ConsentContext) -> Self { + self.consent = consent; + self + } + + fn check(&self, view: &EligibilityView, facts: &RequestFacts, phase: Phase) -> Eligibility { + let input = EligibilityInput { + provider: self.provider.as_ref(), + settings: &self.settings, + identity: &self.identity, + catalog: &self.catalog, + consent: &self.consent, + }; + eligibility(&input, view, facts, phase) + } + + fn is_eligible(&self, view: &EligibilityView, facts: &RequestFacts, phase: Phase) -> bool { + matches!(self.check(view, facts, phase), Eligibility::Eligible { .. }) + } + + /// The kinds of a writer navigation, after send. + fn kinds(&self, view: &EligibilityView) -> BTreeMap { + match self.check(view, &facts_for_writer(), Phase::AfterSend) { + Eligibility::Eligible { kinds } => kinds, + Eligibility::Skip(reason) => panic!("should be eligible, skipped for {reason:?}"), + } + } + + fn skip_reason(&self, view: &EligibilityView, facts: &RequestFacts) -> SkipReason { + match self.check(view, facts, Phase::AfterSend) { + Eligibility::Skip(reason) => reason, + Eligibility::Eligible { .. } => panic!("should be skipped"), + } + } + } + + fn custom_provider( + id: &str, + sources: &[&str], + resolves: Resolves, + policy: ConsentPolicy, + ) -> Arc { + let mut builder = ProviderDescriptor::builder(pid(id)) + .with_resolves(resolves.with_endpoint("https://api.example.com")) + .with_consent_policy(policy); + for name in sources { + builder = builder.with_source(source(name)); + } + provider_from(builder.build().expect("should build the descriptor")) + } + + fn expiring_provider(id: &str, sources: &[&str]) -> Arc { + let mut builder = ProviderDescriptor::builder(pid(id)) + .with_resolves(Resolves::new().with_endpoint("https://api.example.com")) + .with_consent_policy(ConsentPolicy::eid_persistence_only()) + .with_expiring_ids(); + for name in sources { + builder = builder.with_source(source(name)); + } + provider_from(builder.build().expect("should build the descriptor")) + } + + fn positive_record(change: &PartnerIdChange) -> Option<&KvPartnerId> { + match change { + PartnerIdChange::FillMissing { record, .. } + | PartnerIdChange::OwnerWrite { record, .. } => Some(record), + PartnerIdChange::OwnerRemove { .. } | PartnerIdChange::WriterRemove { .. } => None, + } + } + + fn kinds_of(entries: &[(&str, ResolveKind)]) -> BTreeMap { + entries + .iter() + .map(|(name, kind)| (source(name), *kind)) + .collect() + } + + fn one_entry(uid: &str) -> KvEid { + KvEid { + uids: vec![KvUid { + id: uid.to_owned(), + atype: None, + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + } + } + + fn large_entries(source_domain: &SourceDomain) -> Vec { + // Three entries of about 2 KB each: the set is over MAX_ENTRY_SET_BYTES. + let large_ext = json!({ "segments": "x".repeat(2_000) }); + (0..3) + .map(|index| { + ResolvedEid::new(source_domain, ResolvedUid::new(format!("example-{index}"))) + .with_ext(large_ext.clone()) + }) + .collect() + } + + // Eligibility + + #[test] + fn skips_when_not_a_real_browser() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])); + let facts = RequestFacts { + is_real_browser: false, + ..facts_for_writer() + }; + + assert_eq!( + fixture.skip_reason(&EligibilityView::empty(), &facts), + SkipReason::NotRealBrowser, + "should skip a client that does not look like a browser" + ); + assert!( + fixture.is_eligible( + &EligibilityView::empty(), + &facts_for_writer(), + Phase::AfterSend + ), + "control: a real browser should be eligible" + ); + } + + #[test] + fn skips_when_ec_persistence_is_not_allowed() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])); + let facts = RequestFacts { + ec_persistence_allowed: false, + ..facts_for_writer() + }; + + assert_eq!( + fixture.skip_reason(&EligibilityView::empty(), &facts), + SkipReason::EcPersistenceNotAllowed, + "should skip when global consent blocks EID persistence" + ); + } + + #[test] + fn skips_when_the_consent_policy_blocks() { + let provider = custom_provider( + "p", + &["a.example.com"], + Resolves::new(), + resolver_policy().with_tcf_vendor(5), + ); + let blocked = + EligibilityFixture::new(&provider).with_consent(test_consent_gdpr_without_vendor(5)); + let allowed = + EligibilityFixture::new(&provider).with_consent(test_consent_gdpr_without_vendor(6)); + let view = EligibilityView::empty(); + + assert_eq!( + blocked.skip_reason(&view, &facts_for_writer()), + SkipReason::ConsentPolicy, + "should skip when the policy's vendor has no consent" + ); + assert!( + allowed.is_eligible(&view, &facts_for_writer(), Phase::AfterSend), + "control: should run when the vendor has consent" + ); + } + + #[test] + fn skips_a_provider_that_does_not_resolve() { + let fixture = EligibilityFixture::new(&fixed_capture_provider("p", &["a.example.com"])); + + assert_eq!( + fixture.skip_reason(&EligibilityView::empty(), &facts_for_writer()), + SkipReason::NotAResolver, + "should never run a capture-only provider" + ); + } + + #[test] + fn skips_new_visitor_in_every_mode() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])); + let view = EligibilityView::empty(); + for phase in [Phase::Hold, Phase::AfterSend] { + let facts = RequestFacts { + new_visitor: true, + ..facts_for_writer() + }; + assert!( + matches!( + fixture.check(&view, &facts, phase), + Eligibility::Skip(SkipReason::NewVisitor) + ), + "should run no provider on a new visitor's first navigation ({phase:?})" + ); + } + let returning = facts_for_writer(); + assert!( + matches!( + fixture.check(&view, &returning, Phase::AfterSend), + Eligibility::Eligible { .. } + ), + "control: a returning writer should be eligible" + ); + } + + #[test] + fn stager_runs_no_provider_in_either_phase() { + let provider = fixed_provider("p", &["a.example.com"]); + let view = EligibilityView::empty(); + let stager = RequestFacts { + writer: false, + runs_auction: true, + ..facts_for_writer() + }; + for mode in [ProviderMode::Hold, ProviderMode::AfterSend] { + let fixture = EligibilityFixture::new(&provider).with_settings(ProviderSettings { + mode, + ..ProviderSettings::default() + }); + for phase in [Phase::Hold, Phase::AfterSend] { + assert!( + matches!( + fixture.check(&view, &stager, phase), + Eligibility::Skip(SkipReason::NotWriter) + ), + "should run no provider on a stager ({mode:?}, {phase:?})" + ); + } + } + } + + #[test] + fn owned_record_written_by_someone_else_is_due_for_refresh() { + let fixture = EligibilityFixture::owning( + &fixed_provider("p", &["a.example.com", "b.example.com"]), + &["a.example.com", "b.example.com"], + ); + let mut view = EligibilityView::empty(); + // Written by the browser, and acquired recently. + view.set_record_for_test( + &source("a.example.com"), + record_for_test("example-a", None, 999), + ); + // Written by this provider, acquired recently. + view.set_record_for_test( + &source("b.example.com"), + record_for_test("example-b", Some("p"), 999), + ); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[("a.example.com", ResolveKind::Refresh)]), + "should refresh the browser-seeded record and leave the fresh owned one" + ); + } + + #[test] + fn owned_source_without_a_record_is_initial() { + let fixture = EligibilityFixture::owning( + &fixed_provider("p", &["a.example.com"]), + &["a.example.com"], + ); + + assert_eq!( + fixture.kinds(&EligibilityView::empty()), + kinds_of(&[("a.example.com", ResolveKind::Initial)]), + "should ask for an initial resolution when there is no record" + ); + } + + #[test] + fn owned_record_is_due_by_refresh_anchor_or_age() { + let fixture = EligibilityFixture::owning( + &fixed_provider("p", &["a.example.com", "b.example.com", "c.example.com"]), + &["a.example.com", "b.example.com", "c.example.com"], + ) + .with_settings(ProviderSettings { + refresh_after_sec: 100, + ..ProviderSettings::default() + }); + let mut view = EligibilityView::empty(); + // `ra` has passed although `at` is recent. + let mut passed = record_for_test("example-a", Some("p"), 999); + passed.ra = Some(1_000); + view.set_record_for_test(&source("a.example.com"), passed); + // `ra` is ahead although `at` is old, so `ra` wins. + let mut ahead = record_for_test("example-b", Some("p"), 100); + ahead.ra = Some(5_000); + view.set_record_for_test(&source("b.example.com"), ahead); + // No `ra`, and `at + refresh_after_sec` has passed. + view.set_record_for_test( + &source("c.example.com"), + record_for_test("example-c", Some("p"), 900), + ); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[ + ("a.example.com", ResolveKind::Refresh), + ("c.example.com", ResolveKind::Refresh) + ]), + "should refresh when ra passed, or when there is no ra and the age passed" + ); + } + + #[test] + fn owned_record_without_at_or_ra_is_due() { + let fixture = EligibilityFixture::owning( + &fixed_provider("p", &["a.example.com"]), + &["a.example.com"], + ); + let mut view = EligibilityView::empty(); + let mut record = record_for_test("example-a", Some("p"), 999); + record.at = None; + view.set_record_for_test(&source("a.example.com"), record); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[("a.example.com", ResolveKind::Refresh)]), + "should refresh a record with no acquisition time" + ); + } + + #[test] + fn browser_seeded_uid_in_iv_is_never_due() { + let fixture = EligibilityFixture::owning( + &fixed_provider("p", &["a.example.com"]), + &["a.example.com"], + ); + let mut view = EligibilityView::empty(); + view.set_record_for_test( + &source("a.example.com"), + record_for_test("example-bad", None, 10), + ); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[("a.example.com", ResolveKind::Refresh)]), + "control: a browser-seeded record should be due" + ); + + view.set_invalidated_for_test("p", vec![uid_hash("example-bad")]); + assert!( + matches!( + fixture.check(&view, &facts_for_writer(), Phase::AfterSend), + Eligibility::Skip(SkipReason::NoSourceDue) + ), + "should never refresh a browser-seeded value the provider invalidated" + ); + } + + #[test] + fn expired_refresh_anchor_stays_refresh() { + let fixture = EligibilityFixture::owning( + &expiring_provider("p", &["a.example.com"]), + &["a.example.com"], + ); + let mut view = EligibilityView::empty(); + let mut anchor = record_for_test("example-a", Some("p"), 100); + anchor.exp = Some(500); + anchor.ru = Some(9_000); + view.set_record_for_test(&source("a.example.com"), anchor); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[("a.example.com", ResolveKind::Refresh)]), + "should keep the kind Refresh for an expired record that is still stored" + ); + } + + #[test] + fn unowned_source_with_unexpired_record_is_not_initial_even_when_consent_hides_it() { + let hiding = custom_provider( + "q", + &["a.example.com"], + Resolves::new(), + resolver_policy().with_tcf_vendor(5), + ); + let asking = fixed_provider("p", &["a.example.com"]); + let catalog = ProviderCatalog::new(vec![hiding, asking.clone()], BTreeMap::new()); + // Consent withholds vendor 5, so `q`'s record is not servable. + let fixture = EligibilityFixture::with_catalog(&asking, catalog) + .with_consent(test_consent_gdpr_without_vendor(5)); + let mut view = EligibilityView::empty(); + view.set_record_for_test( + &source("a.example.com"), + record_for_test("example-a", Some("q"), 900), + ); + + assert!( + matches!( + fixture.check(&view, &facts_for_writer(), Phase::AfterSend), + Eligibility::Skip(SkipReason::NoSourceDue) + ), + "should not call to fill a source that already holds an unexpired ID" + ); + assert_eq!( + fixture.kinds(&EligibilityView::empty()), + kinds_of(&[("a.example.com", ResolveKind::Initial)]), + "control: an absent record should be Initial" + ); + } + + #[test] + fn expired_unowned_record_counts_as_absent() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])); + let mut view = EligibilityView::empty(); + let mut record = record_for_test("example-a", None, 100); + record.exp = Some(1_000); + view.set_record_for_test(&source("a.example.com"), record); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[("a.example.com", ResolveKind::Initial)]), + "should treat a record expired at now as absent" + ); + } + + #[test] + fn record_of_an_unconfigured_writer_counts_as_absent() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])); + let mut view = EligibilityView::empty(); + view.set_record_for_test( + &source("a.example.com"), + record_for_test("example-a", Some("gone"), 900), + ); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[("a.example.com", ResolveKind::Initial)]), + "should treat a record written by an unconfigured provider as absent" + ); + + let mut browser_view = EligibilityView::empty(); + browser_view.set_record_for_test( + &source("a.example.com"), + record_for_test("example-a", None, 900), + ); + assert!( + matches!( + fixture.check(&browser_view, &facts_for_writer(), Phase::AfterSend), + Eligibility::Skip(SkipReason::NoSourceDue) + ), + "control: a browser-seeded unexpired record should count as present" + ); + } + + #[test] + fn hold_provider_runs_before_the_auction_and_not_after_send() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])) + .with_settings(ProviderSettings { + mode: ProviderMode::Hold, + ..ProviderSettings::default() + }); + let view = EligibilityView::empty(); + let facts = RequestFacts { + runs_auction: true, + ..facts_for_writer() + }; + + assert!( + fixture.is_eligible(&view, &facts, Phase::Hold), + "should hold before the auction" + ); + assert!( + !fixture.is_eligible(&view, &facts, Phase::AfterSend), + "should not run again after send" + ); + } + + #[test] + fn hold_provider_runs_after_send_when_the_navigation_runs_no_auction() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])) + .with_settings(ProviderSettings { + mode: ProviderMode::Hold, + ..ProviderSettings::default() + }); + let view = EligibilityView::empty(); + + let no_auction = RequestFacts { + runs_auction: false, + ..facts_for_writer() + }; + let auction = RequestFacts { + runs_auction: true, + ..facts_for_writer() + }; + let stager = RequestFacts { + writer: false, + runs_auction: true, + ..facts_for_writer() + }; + + assert!( + fixture.is_eligible(&view, &no_auction, Phase::AfterSend), + "should move after send without an auction" + ); + assert!( + !fixture.is_eligible(&view, &no_auction, Phase::Hold), + "should not hold without an auction" + ); + assert!( + fixture.is_eligible(&view, &auction, Phase::Hold), + "control: should hold before an auction" + ); + assert!( + !fixture.is_eligible(&view, &auction, Phase::AfterSend), + "should not also run after send" + ); + assert!( + !fixture.is_eligible(&view, &stager, Phase::Hold), + "should never run on a stager" + ); + } + + #[test] + fn after_send_provider_never_runs_in_hold() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])); + let view = EligibilityView::empty(); + for runs_auction in [true, false] { + let facts = RequestFacts { + runs_auction, + ..facts_for_writer() + }; + assert!( + !fixture.is_eligible(&view, &facts, Phase::Hold), + "should not hold an after-send provider (auction: {runs_auction})" + ); + assert!( + fixture.is_eligible(&view, &facts, Phase::AfterSend), + "control: should run it after send (auction: {runs_auction})" + ); + } + } + + #[test] + fn narrowed_away_source_is_never_a_kind() { + let provider = fixed_provider("p", &["a.example.com", "b.example.com"]); + let catalog = ProviderCatalog::new(vec![provider.clone()], BTreeMap::new()) + .with_narrowing(BTreeMap::from([(pid("p"), vec![source("a.example.com")])])); + let fixture = EligibilityFixture::with_catalog(&provider, catalog); + + assert_eq!( + fixture.kinds(&EligibilityView::empty()), + kinds_of(&[("a.example.com", ResolveKind::Initial)]), + "should leave out a source the operator narrowed away" + ); + } + + #[test] + fn kinds_the_provider_does_not_list_are_left_out() { + let provider = custom_provider( + "p", + &["a.example.com", "b.example.com"], + Resolves::new().with_kinds([ResolveKind::Initial]), + ConsentPolicy::eid_persistence_only(), + ); + let fixture = EligibilityFixture::owning(&provider, &["a.example.com", "b.example.com"]); + let mut view = EligibilityView::empty(); + view.set_record_for_test( + &source("a.example.com"), + record_for_test("example-a", None, 10), + ); + + assert_eq!( + fixture.kinds(&view), + kinds_of(&[("b.example.com", ResolveKind::Initial)]), + "should leave out the Refresh source of an Initial-only provider" + ); + } + + #[test] + fn source_whose_needs_are_unavailable_is_left_out() { + let provider = custom_provider( + "p", + &["a.example.com"], + Resolves::new() + .with_needs_for(ResolveKind::Initial, [Input::ClientIp]) + .with_needs_for(ResolveKind::Refresh, [Input::HashedEmail]), + ConsentPolicy::eid_persistence_only(), + ); + let fixture = EligibilityFixture::owning(&provider, &["a.example.com"]); + let initial = EligibilityView::empty(); + let mut refresh = EligibilityView::empty(); + refresh.set_record_for_test( + &source("a.example.com"), + record_for_test("example-a", None, 10), + ); + let mut without_email = facts_for_writer(); + without_email.available_inputs.remove(&Input::HashedEmail); + let mut without_address = facts_for_writer(); + without_address.available_inputs.remove(&Input::ClientIp); + + assert_eq!( + fixture.skip_reason(&refresh, &without_email), + SkipReason::NoSourceDue, + "should leave out a Refresh whose needs are missing" + ); + assert_eq!( + fixture.skip_reason(&initial, &without_address), + SkipReason::NoSourceDue, + "should leave out an Initial whose needs are missing" + ); + assert!( + fixture.is_eligible(&initial, &without_email, Phase::AfterSend), + "control: the Initial should not need the email" + ); + assert!( + fixture.is_eligible(&refresh, &facts_for_writer(), Phase::AfterSend), + "control: should run when the inputs are available" + ); + } + + #[test] + fn future_n_and_sticky_opt_out_block() { + let fixture = EligibilityFixture::new(&fixed_provider("p", &["a.example.com"])); + let facts = facts_for_writer(); + + let mut waiting = EligibilityView::empty(); + waiting.set_next_for_test("p", 1_001); + assert_eq!( + fixture.skip_reason(&waiting, &facts), + SkipReason::NotYetEligible, + "should wait for n" + ); + + let mut due = EligibilityView::empty(); + due.set_next_for_test("p", 1_000); + assert!( + fixture.is_eligible(&due, &facts, Phase::AfterSend), + "control: should run once n has passed" + ); + + let mut opted_out = EligibilityView::empty(); + opted_out.set_opt_out_for_test("p", 500, None); + assert_eq!( + fixture.skip_reason(&opted_out, &facts), + SkipReason::OptedOut, + "should not call a provider the visitor opted out of" + ); + } + + #[test] + fn a_clear_watermark_lifts_the_opt_out_it_saw() { + let mut view = EligibilityView::empty(); + view.set_opt_out_for_test("p", 500, Some(600)); + assert!( + !view.is_opted_out(&pid("p")), + "should lift the opt-out when oc is later than o.at" + ); + + view.set_opt_out_for_test("p", 500, Some(500)); + assert!( + !view.is_opted_out(&pid("p")), + "should lift the opt-out when oc names it as the opt-out the clear saw" + ); + + view.set_opt_out_for_test("p", 500, Some(400)); + assert!( + view.is_opted_out(&pid("p")), + "should keep the opt-out when oc is older" + ); + assert!( + !view.is_opted_out(&pid("other")), + "should not opt out a provider with no state" + ); + } + + #[test] + fn view_reads_only_the_kv_row() { + let mut entry = KvEntry::new(&allowing_consent(), None, 10, "example.com"); + entry.ids.insert( + "a.example.com".to_owned(), + record_for_test("example-a", Some("p"), 900), + ); + entry.ps.insert( + "p".to_owned(), + crate::ec::kv_types::KvProviderState { + n: Some(2_000), + f: 2, + o: Some(KvOptOut { + at: 5, + fp: "1".to_owned(), + }), + oc: Some(4), + iv: vec!["0123456789abcdef".to_owned()], + ov: Vec::new(), + st: None, + }, + ); + + let view = EligibilityView::from_kv(&entry); + + assert!( + view.record(&source("a.example.com")).is_some(), + "should hold the source's whole record" + ); + assert_eq!(view.failures(&pid("p")), 2, "should hold f"); + assert_eq!(view.next_eligible(&pid("p")), Some(2_000), "should hold n"); + assert!(view.is_opted_out(&pid("p")), "should hold o and oc"); + } + + // Fingerprint + + #[test] + fn fingerprint_bits_is_a_plain_hex_bitmask_and_changes_with_each_signal() { + let policy = ConsentPolicy::strict() + .with_tcf_purposes([2, 4, 10]) + .with_tcf_vendor(5); + let signals = PolicySignals::default(); + let gdpr = test_consent_gdpr_without_vendor(6); + let bits = |consent: &ConsentContext, signals: PolicySignals| { + let value = fingerprint_bits(&policy, consent, signals); + assert!( + value.len() <= 16 && value == value.to_lowercase(), + "should be at most 16 lowercase hex digits" + ); + u64::from_str_radix(&value, 16).expect("should parse as hex") + }; + + // Purposes 1, 2, 4 and 10 are read and granted: bits 0, 1, 3, 9. + // Vendor 5 is granted: bit 10. A TC string was decoded: bit 16. + let control = bits(&gdpr, signals); + assert_eq!( + control, + (1 << 0) | (1 << 1) | (1 << 3) | (1 << 9) | (1 << 10) | (1 << 16), + "should set one bit per granted purpose, the vendor bit and the TCF bit" + ); + + let mut without_purpose_two = gdpr.clone(); + if let Some(tcf) = without_purpose_two.tcf.as_mut() { + tcf.purpose_consents[1] = false; + } + assert_eq!( + bits(&without_purpose_two, signals), + control & !(1 << 1), + "should clear only bit 1 when purpose 2 is not granted" + ); + + let without_vendor = test_consent_gdpr_without_vendor(5); + assert_eq!( + bits(&without_vendor, signals), + control & !(1 << 10), + "should clear only the vendor bit when the vendor is not granted" + ); + + let mut sale = gdpr.clone(); + sale.us_privacy = Some(decode_us_privacy("1YYN").expect("should decode us_privacy")); + assert_eq!( + bits(&sale, signals), + control | (1 << 11), + "should set the sale bit for a US Privacy sale opt-out" + ); + + let mut gpp_sale = gdpr.clone(); + gpp_sale.gpp = Some(GppConsent { + version: 1, + section_ids: vec![7], + eu_tcf: None, + us_sale_opt_out: Some(true), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }); + assert_eq!( + bits(&gpp_sale, signals), + control | (1 << 11), + "should set the sale bit for a GPP sale opt-out" + ); + + let mut gpc = gdpr.clone(); + gpc.gpc = true; + assert_eq!( + bits(&gpc, signals), + control | (1 << 12), + "should set the GPC bit" + ); + + assert_eq!( + bits(&gdpr, PolicySignals { dnt: true }), + control | (1 << 13), + "should set the DNT bit" + ); + } + + #[test] + fn fingerprint_bits_leaves_a_relaxed_signal_at_zero_and_never_moves_a_bit() { + let strict = resolver_policy(); + let relaxed = resolver_policy() + .allowing_gpc() + .allowing_dnt() + .allowing_us_opt_out(UsOptOut::Sale); + let mut consent = allowing_consent(); + consent.gpc = true; + let signals = PolicySignals { dnt: true }; + + assert_eq!( + fingerprint_bits(&strict, &consent, signals), + "3000", + "should set the GPC and DNT bits at positions 12 and 13" + ); + assert_eq!( + fingerprint_bits(&relaxed, &consent, signals), + "0", + "should leave a relaxed signal at zero" + ); + assert_eq!( + fingerprint_bits( + &resolver_policy().with_tcf_purposes([7]), + &consent, + PolicySignals::default() + ), + "1000", + "should keep GPC at bit 12 whatever purposes the policy reads" + ); + } + + // Section 8 rows + + #[test] + fn no_match_waits_retry_in_capped_at_seven_days() { + let provider = fixed_provider("p", &["a.example.com"]); + let fixture = OutcomeFixture::new(); + let context = fixture.context_at(1_000); + let default = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::no_match()), + &EligibilityView::empty(), + &context, + ); + assert_eq!( + default.state.n, + Some(1_000 + 86_400), + "should wait the 24 h default" + ); + let long = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved( + Resolution::no_match().with_retry_in(Duration::from_secs(30 * 86_400)), + ), + &EligibilityView::empty(), + &context, + ); + assert_eq!( + long.state.n, + Some(1_000 + 7 * 86_400), + "should cap the wait at 7 days" + ); + } + + #[test] + fn found_resets_failures_and_retry_in_applies_to_every_resolution() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 3); + view.set_record_for_test(&a, record_for_test("example-old", Some("p"), 900)); + let fixture = OutcomeFixture::new(); + let context = fixture.context_at(1_000); + + let found = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved( + Resolution::found([ResolvedEid::new(&a, ResolvedUid::new("u"))]) + .with_retry_in(Duration::from_secs(600)), + ), + &view, + &context, + ); + let invalidated = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved( + Resolution::invalidate(&a).with_retry_in(Duration::from_secs(600)), + ), + &view, + &context, + ); + + assert_eq!( + found.state.f, + Some(0), + "should reset the failure count on found" + ); + assert_eq!( + found.state.n, + Some(1_000 + 600), + "should wait at least retry_in after found" + ); + assert_eq!( + invalidated.changes.len(), + 1, + "control: should remove the stored record" + ); + assert_eq!( + invalidated.state.n, + Some(1_000 + 600), + "should raise invalidate's n = now to retry_in" + ); + } + + #[test] + fn found_without_retry_in_sets_no_n() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("u"), + )])), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.state.n, None, + "should leave the refresh schedule to the record's own timing" + ); + } + + #[test] + fn input_failure_backs_off_exponentially_to_a_day() { + let provider = fixed_provider("p", &["a.example.com"]); + let fixture = OutcomeFixture::new(); + let context = fixture.context_at(1_000); + let cases = [ + (0, 1, 3_600), + (1, 2, 7_200), + (4, 5, 57_600), + (5, 6, 86_400), + (u32::MAX, u32::MAX, 86_400), + ]; + for (before, after, wait) in cases { + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", before); + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::InputFailure, + &view, + &context, + ); + assert_eq!( + changes.state.f, + Some(after), + "should count the failure (f was {before})" + ); + assert_eq!( + changes.state.n, + Some(1_000 + wait), + "should back off min(1 h x 2^(f-1), 24 h) (f was {before})" + ); + } + } + + #[test] + fn transport_failure_backs_off_to_an_hour() { + let provider = fixed_provider("p", &["a.example.com"]); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 5); + let fixture = OutcomeFixture::new(); + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::TransportFailure, + &view, + &fixture.context_at(1_000), + ); + assert_eq!(changes.state.f, Some(6), "should count the failure"); + assert_eq!( + changes.state.n, + Some(1_000 + 3_600), + "should back off min(5 min x 2^5, 1 h) = 1 h" + ); + + let first = outcome_to_changes( + provider.as_ref(), + CallOutcome::TransportFailure, + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + assert_eq!( + first.state.n, + Some(1_000 + 300), + "should start at 5 minutes" + ); + } + + #[test] + fn hold_cut_off_waits_hold_retry_without_counting_a_failure() { + let provider = fixed_provider("p", &["a.example.com"]); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 2); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::HoldCutOff, + &view, + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.state.n, + Some(1_000 + 300), + "should wait hold_retry_sec" + ); + assert_eq!(changes.state.f, None, "should not count a failure"); + assert!(changes.changes.is_empty(), "should change no stored ID"); + } + + #[test] + fn skip_waits_refresh_after() { + let provider = fixed_provider("p", &["a.example.com"]); + let fixture = OutcomeFixture::new().with_settings(ProviderSettings { + refresh_after_sec: 777, + ..ProviderSettings::default() + }); + for outcome in [CallOutcome::Skipped, CallOutcome::InputUnavailable] { + let changes = outcome_to_changes( + provider.as_ref(), + outcome, + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + assert_eq!( + changes.state.n, + Some(1_000 + 777), + "should wait refresh_after_sec" + ); + assert_eq!(changes.state.f, None, "should not count a failure"); + } + } + + #[test] + fn burst_cap_waits_one_window_and_never_counts_a_failure() { + let provider = fixed_provider("p", &["a.example.com"]); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 2); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::BurstCapped, + &view, + &fixture.context_at(1_000), + ); + + assert_eq!(changes.state.n, Some(1_060), "should wait one 60 s window"); + assert_eq!(changes.state.f, None, "should never count a failure"); + } + + #[test] + fn breaker_open_and_not_dispatched_change_nothing() { + let provider = fixed_provider("p", &["a.example.com"]); + let fixture = OutcomeFixture::new(); + for outcome in [CallOutcome::BreakerOpen, CallOutcome::NotDispatched] { + let changes = outcome_to_changes( + provider.as_ref(), + outcome, + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + assert_eq!( + changes, + OutcomeChanges::default(), + "should change nothing when the call was not made" + ); + } + } + + #[test] + fn invalid_state_clears_state_and_keeps_f() { + let provider = stateful_fixed_provider("p", &["a.example.com"]); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 2); + let fixture = OutcomeFixture::new().with_settings(ProviderSettings { + refresh_after_sec: 777, + ..ProviderSettings::default() + }); + let context = fixture.context_at(1_000); + + let bare = outcome_to_changes( + provider.as_ref(), + CallOutcome::InvalidState(None), + &view, + &context, + ); + assert_eq!(bare.state.st, StateUpdate::Clear, "should clear st"); + assert_eq!(bare.state.f, None, "should keep f"); + assert_eq!( + bare.state.n, + Some(1_000 + 777), + "should wait refresh_after_sec" + ); + + let with_result = outcome_to_changes( + provider.as_ref(), + CallOutcome::InvalidState(Some( + Resolution::no_match().with_retry_in(Duration::from_secs(60)), + )), + &view, + &context, + ); + let result = with_result + .result + .expect("should carry the result's own state change"); + assert_eq!(result.state.st, StateUpdate::Clear, "should clear st"); + assert_eq!(result.state.f, None, "should keep f"); + assert_eq!( + result.state.n, + Some(1_000 + 60), + "should keep the n the result set" + ); + } + + #[test] + fn oversized_state_rejects_positive_part_but_keeps_removals() { + let provider = stateful_fixed_provider("p", &["a.example.com", "b.example.com"]); + let a = source("a.example.com"); + let b = source("b.example.com"); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&b, record_for_test("example-old", Some("p"), 900)); + let resolution = Resolution::found([ResolvedEid::new(&a, ResolvedUid::new("example-a"))]) + .also_invalidate(&b) + .replacing_state(&"x".repeat(2_000)); + + let fixture = OutcomeFixture::new(); + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(resolution), + &view, + &fixture.context_at(1_000), + ); + + assert!( + changes.result.is_none(), + "should store no ID and no state change" + ); + assert_eq!( + changes.changes.len(), + 1, + "should keep only the invalidation" + ); + assert!( + matches!(&changes.changes[0], PartnerIdChange::OwnerRemove { source, .. } if *source == b), + "should still apply the invalidation" + ); + assert_eq!( + changes.state.st, + StateUpdate::Keep, + "should keep the old state" + ); + assert_eq!(changes.state.f, Some(1), "should handle it as Malformed"); + assert_eq!( + changes.state.n, + Some(1_000 + 3_600), + "should back off min(1 h x 2^0, 24 h)" + ); + } + + #[test] + fn opted_out_removes_what_it_wrote_and_clears_state() { + let provider = fixed_provider("p", &["a.example.com", "b.example.com"]); + let a = source("a.example.com"); + let b = source("b.example.com"); + let c = source("c.example.com"); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&a, record_for_test("example-a", Some("p"), 900)); + view.set_record_for_test(&b, record_for_test("example-b", None, 900)); + view.set_record_for_test(&c, record_for_test("example-c", Some("p"), 900)); + let fixture = OutcomeFixture::new().with_fingerprint("5"); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::opted_out()), + &view, + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.changes, + vec![ + PartnerIdChange::WriterRemove { + source: a, + writer: pid("p"), + }, + PartnerIdChange::WriterRemove { + source: c, + writer: pid("p"), + }, + ], + "should remove every record the provider wrote, claimed or not, and no browser record" + ); + assert_eq!( + changes.state.opt_out, + Some(KvOptOut { + at: 1_000, + fp: "5".to_owned() + }), + "should record the opt-out" + ); + assert_eq!(changes.state.st, StateUpdate::Clear, "should clear st"); + } + + #[test] + fn capture_only_opt_out_also_removes_browser_records_of_its_sources() { + let provider = fixed_capture_provider("p", &["a.example.com", "b.example.com"]); + let a = source("a.example.com"); + let b = source("b.example.com"); + let c = source("c.example.com"); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&a, record_for_test("example-a", None, 900)); + view.set_record_for_test(&b, record_for_test("example-b", Some("q"), 900)); + view.set_record_for_test(&c, record_for_test("example-c", None, 900)); + let catalog = ProviderCatalog::new(vec![provider.clone()], BTreeMap::new()); + let fixture = OutcomeFixture::new().with_catalog(catalog); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::opted_out()), + &view, + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.changes, + vec![PartnerIdChange::WriterRemove { + source: a, + writer: pid("p"), + }], + "should remove the browser record of its own source only" + ); + } + + #[test] + fn opted_out_records_the_fingerprint_of_the_current_consent() { + let provider = fixed_provider("p", &["a.example.com"]); + let policy = resolver_policy(); + let mut consent = allowing_consent(); + consent.gpc = true; + let fingerprint = fingerprint_bits(&policy, &consent, PolicySignals::default()); + let fixture = OutcomeFixture::new().with_fingerprint(&fingerprint); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::opted_out()), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.state.opt_out, + Some(KvOptOut { + at: 1_000, + fp: "1000".to_owned() + }), + "should store the fingerprint of the consent in force" + ); + } + + #[test] + fn stateful_opt_out_goes_into_the_result_with_its_removals_outside() { + let provider = stateful_fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&a, record_for_test("example-a", Some("p"), 900)); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::opted_out()), + &view, + &fixture.context_at(1_000), + ); + + assert_eq!(changes.changes.len(), 1, "should list the removal"); + let result = changes + .result + .expect("should carry the opt-out in one result"); + assert!(result.ids.is_empty(), "should hold no positive change"); + assert!(result.state.opt_out.is_some(), "should record the opt-out"); + assert_eq!(result.state.st, StateUpdate::Clear, "should clear st"); + } + + #[test] + fn invalidate_removes_the_set_and_sets_n_now() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let record = record_for_test("example-a", Some("p"), 900); + let expected = record_revision(&record); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&a, record); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::invalidate(&a)), + &view, + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.changes, + vec![PartnerIdChange::OwnerRemove { + source: a, + provider: pid("p"), + expected, + }], + "should remove the set at the revision read" + ); + assert_eq!(changes.state.n, Some(1_000), "should set n = now"); + assert_eq!(changes.state.f, None, "should not touch f"); + } + + #[test] + fn invalidating_a_source_without_a_record_makes_no_change() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::invalidate(&a)), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + assert_eq!( + changes, + OutcomeChanges::default(), + "should change nothing for a source with no record" + ); + } + + #[test] + fn invalidating_an_unclaimed_source_makes_no_change() { + let provider = fixed_provider("p", &["a.example.com"]); + let c = source("c.example.com"); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&c, record_for_test("example-c", Some("q"), 900)); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::invalidate(&c)), + &view, + &fixture.context_at(1_000), + ); + + assert!( + changes.changes.is_empty(), + "should not let a provider remove a source it does not claim" + ); + } + + // Entry sets + + #[test] + fn found_with_several_entries_becomes_one_whole_set_per_source() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let resolution = Resolution::found([ + ResolvedEid::new(&a, ResolvedUid::new("example-a").with_atype(1)) + .with_uid( + ResolvedUid::new("example-b") + .with_atype(3) + .with_ext(json!({ "stype": "ppuid" })), + ) + .with_inserter("example.com") + .with_mm(3), + ResolvedEid::new(&a, ResolvedUid::new("example-c")).with_matcher("partner.example.com"), + ]); + + let fixture = OutcomeFixture::new(); + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(resolution), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.changes.len(), + 1, + "should make one change for the one source" + ); + let record = positive_record(&changes.changes[0]).expect("should be a positive change"); + assert_eq!(record.w.as_deref(), Some("p"), "should name the writer"); + assert_eq!(record.at, Some(1_000), "should stamp the write time"); + assert_eq!( + record.eids, + vec![ + KvEid { + uids: vec![ + KvUid { + id: "example-a".to_owned(), + atype: Some(1), + ext: None + }, + KvUid { + id: "example-b".to_owned(), + atype: Some(3), + ext: Some(json!({ "stype": "ppuid" })) + }, + ], + inserter: Some("example.com".to_owned()), + matcher: None, + mm: Some(3), + ext: None, + }, + KvEid { + uids: vec![KvUid { + id: "example-c".to_owned(), + atype: None, + ext: None + }], + inserter: None, + matcher: Some("partner.example.com".to_owned()), + mm: None, + ext: None, + }, + ], + "should store the whole set, in the order the provider returned it" + ); + } + + #[test] + fn one_uid_result_without_eid_fields_becomes_a_one_entry_record() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("example-a"), + )])), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + let record = positive_record(&changes.changes[0]).expect("should be a positive change"); + assert_eq!( + record.eids, + KvPartnerId::new("example-a").eids, + "should write the shape browser ingest writes" + ); + assert_eq!(record.rv, None, "should leave the revision to the merge"); + } + + #[test] + fn owned_source_becomes_owner_write_and_unowned_becomes_fill_missing() { + let provider = fixed_provider("p", &["a.example.com", "b.example.com"]); + let a = source("a.example.com"); + let b = source("b.example.com"); + let catalog = ProviderCatalog::new( + vec![provider.clone()], + BTreeMap::from([(a.clone(), pid("p"))]), + ); + let stored = record_for_test("example-old", Some("p"), 900); + let expected = record_revision(&stored); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&a, stored); + let fixture = OutcomeFixture::new().with_catalog(catalog); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ + ResolvedEid::new(&a, ResolvedUid::new("example-a")), + ResolvedEid::new(&b, ResolvedUid::new("example-b")), + ])), + &view, + &fixture.context_at(1_000), + ); + + assert!( + changes.changes.iter().any(|change| matches!( + change, + PartnerIdChange::OwnerWrite { source, expected: Some(revision), .. } + if *source == a && *revision == expected + )), + "should write the owned source at the revision read" + ); + assert!( + changes.changes.iter().any(|change| matches!( + change, + PartnerIdChange::FillMissing { source, .. } if *source == b + )), + "should only fill the unowned source" + ); + + let fresh = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("example-a"), + )])), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + assert!( + matches!( + &fresh.changes[0], + PartnerIdChange::OwnerWrite { expected: None, .. } + ), + "should expect no record when the owned source had none" + ); + } + + #[test] + fn stateful_found_goes_into_one_provider_result_with_expected_state() { + let provider = stateful_fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let expected_state = StateRevision::of(Some(&json!({ "token": "example-old" }))); + let fixture = OutcomeFixture::new().with_expected_state(expected_state.clone()); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved( + Resolution::found([ResolvedEid::new(&a, ResolvedUid::new("example-a"))]) + .replacing_state(&json!({ "token": "example-new" })), + ), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + assert!( + changes.changes.is_empty(), + "should hold no stand-alone change" + ); + assert_eq!( + changes.state, + ProviderStateChange::default(), + "should hold no stand-alone state" + ); + let result = changes.result.expect("should produce one result"); + assert_eq!(result.provider, pid("p"), "should name the provider"); + assert_eq!(result.ids.len(), 1, "should carry the positive change"); + assert_eq!( + result.state.st, + StateUpdate::Replace(json!({ "token": "example-new" })), + "should replace st" + ); + assert_eq!(result.state.f, Some(0), "should reset f in the result"); + assert_eq!( + result.expected_state, + Some(expected_state), + "should expect the state revision the call read" + ); + } + + #[test] + fn unclaimed_source_entries_are_discarded() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let c = source("c.example.com"); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ + ResolvedEid::new(&a, ResolvedUid::new("example-a")), + ResolvedEid::new(&c, ResolvedUid::new("example-c")), + ])), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.changes.len(), + 1, + "should keep only the claimed source" + ); + assert!( + matches!(&changes.changes[0], PartnerIdChange::FillMissing { source, .. } if *source == a), + "should drop the source the provider does not claim" + ); + } + + #[test] + fn found_naming_only_unclaimed_sources_is_malformed() { + let provider = fixed_provider("p", &["a.example.com"]); + let c = source("c.example.com"); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 3); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &c, + ResolvedUid::new("example-c"), + )])), + &view, + &fixture.context_at(1_000), + ); + + assert!(changes.changes.is_empty(), "should store nothing"); + assert_eq!(changes.state.f, Some(4), "should count a failure"); + assert_eq!( + changes.state.n, + Some(1_000 + 28_800), + "should back off min(1 h x 2^3, 24 h)" + ); + } + + #[test] + fn one_claimed_and_one_unclaimed_source_still_resets_failures() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let c = source("c.example.com"); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 3); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ + ResolvedEid::new(&a, ResolvedUid::new("example-a")), + ResolvedEid::new(&c, ResolvedUid::new("example-c")), + ])), + &view, + &fixture.context_at(1_000), + ); + + assert_eq!(changes.changes.len(), 1, "should store the claimed source"); + assert_eq!(changes.state.f, Some(0), "should reset f on found"); + } + + #[test] + fn oversized_set_from_a_stateless_provider_leaves_only_that_source_unchanged() { + let provider = fixed_provider("p", &["a.example.com", "b.example.com"]); + let a = source("a.example.com"); + let b = source("b.example.com"); + let mut entries = large_entries(&a); + entries.push(ResolvedEid::new(&b, ResolvedUid::new("example-b"))); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found(entries)), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + assert_eq!(changes.changes.len(), 1, "should keep the other source"); + assert!( + matches!(&changes.changes[0], PartnerIdChange::FillMissing { source, .. } if *source == b), + "should leave the oversized source as it is" + ); + assert_eq!(changes.state.f, Some(0), "should still count as found"); + } + + #[test] + fn oversized_set_from_a_stateful_provider_rejects_the_positive_part_and_keeps_removals() { + let provider = stateful_fixed_provider("p", &["a.example.com", "b.example.com"]); + let a = source("a.example.com"); + let b = source("b.example.com"); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&b, record_for_test("example-old", Some("p"), 900)); + let resolution = Resolution::found(large_entries(&a)) + .also_invalidate(&b) + .replacing_state(&json!({ "token": "example-token" })); + + let fixture = OutcomeFixture::new(); + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(resolution), + &view, + &fixture.context_at(1_000), + ); + + assert!( + changes.result.is_none(), + "should store no ID and no state change" + ); + assert!( + changes.changes.iter().any( + |change| matches!(change, PartnerIdChange::OwnerRemove { source, .. } if *source == b) + ), + "should still apply the invalidation" + ); + assert!( + changes + .changes + .iter() + .all(|change| positive_record(change).is_none()), + "should store no positive change" + ); + assert!( + matches!(changes.state.st, StateUpdate::Keep), + "should keep the old state" + ); + assert_eq!(changes.state.f, Some(1), "should handle it as Malformed"); + assert_eq!( + changes.state.n, + Some(1_000 + 3_600), + "should back off min(1 h x 2^0, 24 h)" + ); + } + + #[test] + fn result_whose_every_set_is_discarded_is_malformed() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 1); + view.set_record_for_test( + &source("b.example.com"), + record_for_test("example-b", Some("p"), 900), + ); + let fixture = OutcomeFixture::new(); + + let oversized = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved( + Resolution::found(large_entries(&a)).also_invalidate(&source("b.example.com")), + ), + &view, + &fixture.context_at(1_000), + ); + let dropped = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new(" "), + )])), + &view, + &fixture.context_at(1_000), + ); + + for (name, changes) in [("oversized", &oversized), ("dropped", &dropped)] { + assert_eq!(changes.state.f, Some(2), "should count a failure ({name})"); + assert_eq!( + changes.state.n, + Some(1_000 + 7_200), + "should back off min(1 h x 2^1, 24 h) ({name})" + ); + assert!( + changes + .changes + .iter() + .all(|change| positive_record(change).is_none()), + "should store no positive change ({name})" + ); + } + assert_eq!( + oversized.changes.len(), + 1, + "should still apply the removal of an oversized result" + ); + } + + #[test] + fn invalidating_a_set_carries_no_hash_until_it_applies() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let fixture = OutcomeFixture::new(); + let mut record = record_for_test("example-0-0", Some("p"), 900); + record.eids = (0..4) + .map(|chunk| KvEid { + uids: (0..8) + .map(|index| KvUid { + id: format!("example-{chunk}-{index}"), + atype: None, + ext: None, + }) + .collect(), + inserter: None, + matcher: None, + mm: None, + ext: None, + }) + .collect(); + let expected = record_revision(&record); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&a, record); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::invalidate(&a)), + &view, + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.changes.len(), + 1, + "should make one removal for the 32-UID set" + ); + assert!( + matches!( + &changes.changes[0], + PartnerIdChange::OwnerRemove { source, provider, expected: revision } + if *source == a && provider.as_str() == "p" && *revision == expected + ), + "should remove the whole set at the revision it read" + ); + assert_eq!(changes.state.n, Some(1_000), "should set n = now"); + } + + #[test] + fn expiring_set_without_expiry_keeps_stored_exp_only_when_unchanged() { + let provider = expiring_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let catalog = ProviderCatalog::new( + vec![provider.clone()], + BTreeMap::from([(a.clone(), pid("p"))]), + ); + let fixture = OutcomeFixture::new().with_catalog(catalog); + let mut stored = record_for_test("example-a", Some("p"), 900); + stored.exp = Some(5_000); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&a, stored); + + let unchanged = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("example-a"), + )])), + &view, + &fixture.context_at(1_000), + ); + let changed = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("example-other"), + )])), + &view, + &fixture.context_at(1_000), + ); + let with_expiry = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("example-other"), + ) + .with_expires_in(Duration::from_secs(60))])), + &view, + &fixture.context_at(1_000), + ); + + let kept = positive_record(&unchanged.changes[0]).expect("should store the unchanged set"); + assert_eq!(kept.exp, Some(5_000), "should keep the stored exp"); + assert_eq!(unchanged.state.f, Some(0), "should count as found"); + + assert!( + changed.changes.is_empty(), + "should discard a changed set that carries no expiry" + ); + assert_eq!( + changed.state.f, + Some(1), + "should handle a result whose every set is discarded as Malformed" + ); + + let fresh = positive_record(&with_expiry.changes[0]).expect("should store the set"); + assert_eq!(fresh.exp, Some(1_060), "should use the result's own expiry"); + } + + #[test] + fn result_without_a_source_leaves_its_set_alone() { + let provider = fixed_provider("p", &["a.example.com", "b.example.com"]); + let a = source("a.example.com"); + let b = source("b.example.com"); + let mut view = EligibilityView::empty(); + view.set_record_for_test(&b, record_for_test("example-b", Some("p"), 900)); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("example-a"), + )])), + &view, + &fixture.context_at(1_000), + ); + + assert_eq!( + changes.changes.len(), + 1, + "should change only the named source" + ); + assert!( + matches!(&changes.changes[0], PartnerIdChange::FillMissing { source, .. } if *source == a), + "should leave the source the result did not name" + ); + } + + #[test] + fn record_carries_the_earliest_times_of_its_set() { + let provider = stateful_fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let fixture = OutcomeFixture::new(); + let resolution = Resolution::found([ + ResolvedEid::new(&a, ResolvedUid::new("example-a")) + .with_expires_in(Duration::from_secs(600)) + .with_refresh_in(Duration::from_secs(300)) + .with_refresh_until(UnixTime::from_secs(9_000)), + ResolvedEid::new(&a, ResolvedUid::new("example-b")) + .with_expires_in(Duration::from_secs(60)) + .with_refresh_in(Duration::from_secs(900)) + .with_refresh_until(UnixTime::from_secs(8_000)), + ]); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(resolution), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + let result = changes.result.expect("should produce one result"); + let record = positive_record(&result.ids[0]).expect("should be a positive change"); + assert_eq!(record.exp, Some(1_060), "should keep the earliest expiry"); + assert_eq!(record.ra, Some(1_300), "should keep the earliest refresh"); + assert_eq!( + record.ru, + Some(8_000), + "should keep the earliest refresh limit" + ); + } + + #[test] + fn refresh_until_is_ignored_for_a_stateless_provider() { + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ResolvedEid::new( + &a, + ResolvedUid::new("example-a"), + ) + .with_refresh_until(UnixTime::from_secs(9_000))])), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + let record = positive_record(&changes.changes[0]).expect("should be a positive change"); + assert_eq!( + record.ru, None, + "should ignore refresh_until without private state" + ); + } + + #[test] + fn duplicate_entries_collapse_into_one_entry() { + // A duplicate entry is dropped, so the kept set has one entry. + let provider = fixed_provider("p", &["a.example.com"]); + let a = source("a.example.com"); + let fixture = OutcomeFixture::new(); + + let changes = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::found([ + ResolvedEid::new(&a, ResolvedUid::new("example-a")), + ResolvedEid::new(&a, ResolvedUid::new("example-a")), + ])), + &EligibilityView::empty(), + &fixture.context_at(1_000), + ); + + let record = positive_record(&changes.changes[0]).expect("should be a positive change"); + assert_eq!( + record.eids, + vec![one_entry("example-a")], + "should drop the duplicate" + ); + } + + // Dispatch + + struct RecordingLimiter { + exceeded: bool, + fail: bool, + keys: std::sync::Mutex>, + } + + impl RecordingLimiter { + fn new(exceeded: bool, fail: bool) -> Self { + Self { + exceeded, + fail, + keys: std::sync::Mutex::new(Vec::new()), + } + } + } + + impl RateLimiter for RecordingLimiter { + fn exceeded( + &self, + key: &str, + _hourly_limit: u32, + ) -> Result> { + self.keys + .lock() + .expect("should lock the keys") + .push(key.to_owned()); + if self.fail { + return Err(Report::new(crate::error::TrustedServerError::Proxy { + message: "limiter unavailable".to_owned(), + })); + } + Ok(self.exceeded) + } + + fn record(&self, _key: &str) -> Result<(), Report> { + Ok(()) + } + + fn count(&self, _key: &str) -> Result> { + Ok(0) + } + } + + /// Providers, a fake client and everything `dispatch` reads. + struct DispatchFixture { + settings: Settings, + catalog: ProviderCatalog, + keys: IdentityKeys, + client: Arc, + backend: Arc, + services: RuntimeServices, + } + + impl DispatchFixture { + fn new(providers: Vec>) -> Self { + let mut settings = create_test_settings(); + // One fixture client counts the sends of every run, where a real + // request starts from zero, so the budget tests set their own. + settings.ec.identity.max_backend_requests = usize::MAX; + let client = Arc::new(FakeHttpClient::new()); + let backend = Arc::new(FakeBackend::new()); + let services = fake_services(&client, &backend); + Self { + keys: IdentityKeys::from_settings(&settings), + catalog: ProviderCatalog::new(providers, BTreeMap::new()), + settings, + client, + backend, + services, + } + } + + fn run( + &self, + ids: &[&str], + now: u64, + budget: Option, + limiter: Option<&dyn RateLimiter>, + ) -> Vec<(IdentityProviderId, CallOutcome)> { + self.run_with_state(ids, &[], now, budget, limiter) + } + + fn run_with_state( + &self, + ids: &[&str], + states: &[(&str, &str)], + now: u64, + budget: Option, + limiter: Option<&dyn RateLimiter>, + ) -> Vec<(IdentityProviderId, CallOutcome)> { + let deps = IdentityDeps { + settings: &self.settings, + catalog: &self.catalog, + keys: &self.keys, + random: &SeededRandom::new(1), + services: &self.services, + rate_limiter: limiter, + }; + let calls = ids + .iter() + .map(|id| { + let provider = self + .catalog + .get(&pid(id)) + .expect("should hold the provider"); + Call { + state_json: states + .iter() + .find(|(owner, _)| owner == id) + .map(|(_, state)| (*state).to_owned()), + provider: pid(id), + kinds: BTreeMap::from([( + provider.descriptor().primary_source().clone(), + ResolveKind::Initial, + )]), + } + }) + .collect(); + let signals = ResolveSignals { + client_ip: None, + user_agent: Some("ExampleBrowser/1.0"), + page_url: None, + region: None, + hashed_email: None, + consent: ProviderConsent::default(), + publisher_domain: "www.example.com", + now: UnixTime::from_secs(now), + random: &ThreadRandom, + ec_id: Some(format!("{}.ABC123", "a".repeat(64))), + stored: BTreeMap::new(), + }; + futures::executor::block_on(dispatch(&deps, calls, &signals, budget)) + } + } + + fn outcome_of<'a>( + outcomes: &'a [(IdentityProviderId, CallOutcome)], + id: &str, + ) -> &'a CallOutcome { + &outcomes + .iter() + .find(|(provider, _)| *provider == pid(id)) + .expect("should hold an outcome for the provider") + .1 + } + + fn scripted(id: &str, host: &str, source: &str) -> Arc { + ScriptedProvider::new(id, host, source, false).into_provider() + } + + #[test] + fn check_request_enforces_https_host_and_no_cookie() { + let descriptor = fixed_provider("p", &["a.example.com"]).descriptor().clone(); + let good = ProviderRequest::get( + &"https://api.example.com/x" + .parse() + .expect("should parse the URL"), + ); + check_request(&good, &descriptor) + .expect("should allow an https request to a declared host (control)"); + for (name, request) in [ + ( + "http", + ProviderRequest::get( + &"http://api.example.com/x" + .parse() + .expect("should parse the URL"), + ), + ), + ( + "host", + ProviderRequest::get( + &"https://other.example.net/x" + .parse() + .expect("should parse the URL"), + ), + ), + ( + "cookie", + good.clone() + .try_with_header("cookie", "a=1") + .expect("should add the header"), + ), + ] { + assert!( + check_request(&request, &descriptor).is_err(), + "should reject the {name} case" + ); + } + } + + #[test] + fn decode_body_caps_decoded_size() { + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder + .write_all(&vec![b'a'; 10_000]) + .expect("should write to the encoder"); + let gzipped = encoder.finish().expect("should finish the gzip stream"); + assert_eq!( + decode_body(Some("gzip"), gzipped.clone(), 10_000).map(|body| body.len()), + Some(10_000), + "should decode a body that fits" + ); + assert!( + decode_body(Some("gzip"), gzipped, 9_999).is_none(), + "should reject a body over the cap after decoding" + ); + assert!( + decode_body(Some("zstd"), vec![1], 10).is_none(), + "should reject an unknown encoding" + ); + assert_eq!( + decode_body(None, b"ok".to_vec(), 10).as_deref(), + Some(&b"ok"[..]), + "should pass an identity body through" + ); + assert!( + decode_body(None, vec![0; 11], 10).is_none(), + "should reject an identity body over the cap" + ); + } + + #[test] + fn dispatch_maps_status_codes_to_outcomes() { + let mut fixture = DispatchFixture::new(vec![ + scripted("ok", "ok.example.com", "ok.example.com"), + scripted("busy", "busy.example.com", "busy.example.com"), + scripted("bad", "bad.example.com", "bad.example.com"), + scripted("big", "big.example.com", "big.example.com"), + scripted("long", "long.example.com", "long.example.com"), + ]); + fixture.settings.ec.identity.max_response_bytes = 8; + fixture.client.respond("ok.example.com", 200, b"{}"); + fixture.client.respond("busy.example.com", 503, b"{}"); + fixture.client.respond("bad.example.com", 400, b"{}"); + fixture + .client + .respond("big.example.com", 200, b"0123456789"); + fixture.client.respond_with_headers( + "long.example.com", + 200, + b"{}", + &[("content-length", "9999")], + ); + + let outcomes = fixture.run(&["ok", "busy", "bad", "big", "long"], 1_000, None, None); + + assert!( + matches!(outcome_of(&outcomes, "ok"), CallOutcome::Resolved(_)), + "should resolve a 200" + ); + assert!( + matches!(outcome_of(&outcomes, "busy"), CallOutcome::TransportFailure), + "should count a 503 as a transport failure" + ); + assert!( + matches!(outcome_of(&outcomes, "bad"), CallOutcome::InputFailure), + "should count a 400 as an input failure" + ); + assert!( + matches!(outcome_of(&outcomes, "big"), CallOutcome::TransportFailure), + "should count a body over the cap as a transport failure" + ); + assert!( + matches!(outcome_of(&outcomes, "long"), CallOutcome::TransportFailure), + "should count a Content-Length over the cap as a transport failure" + ); + } + + #[test] + fn dispatch_rejects_a_body_that_decodes_over_the_cap() { + let mut fixture = + DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.settings.ec.identity.max_response_bytes = 4_096; + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder + .write_all(&vec![b'a'; 100_000]) + .expect("should write to the encoder"); + let gzipped = encoder.finish().expect("should finish the gzip stream"); + assert!( + gzipped.len() < 4_096, + "should compress under the cap (control)" + ); + fixture.client.respond_with_headers( + "p.example.com", + 200, + &gzipped, + &[("content-encoding", "gzip")], + ); + + let outcomes = fixture.run(&["p"], 1_000, None, None); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::TransportFailure), + "should count a decompression bomb as a transport failure" + ); + } + + #[test] + fn after_send_timeout_is_a_transport_failure_not_a_cut_off() { + let fixture = DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.client.time_out("p.example.com"); + + let outcomes = fixture.run(&["p"], 1_000, None, None); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::TransportFailure), + "should count a missed after-send timeout as a transport failure" + ); + assert_eq!( + fixture.backend.first_byte_timeouts(), + vec![Duration::from_millis( + ProviderSettings::default_after_send_timeout_ms() + )], + "should wait the after-send timeout for the first byte" + ); + } + + #[test] + fn hold_dispatch_cuts_off_a_call_the_budget_ran_out_on() { + let fixture = DispatchFixture::new(vec![ + scripted("a", "a.example.com", "a.example.com"), + scripted("b", "b.example.com", "b.example.com"), + scripted("c", "c.example.com", "c.example.com"), + ]); + fixture.client.respond("a.example.com", 200, b"{}"); + fixture.client.respond("b.example.com", 200, b"{}"); + fixture.client.respond("c.example.com", 200, b"{}"); + fixture + .client + .delay("b.example.com", Duration::from_millis(60)); + fixture.client.time_out("c.example.com"); + + let outcomes = fixture.run( + &["a", "b", "c"], + 1_000, + Some(Duration::from_millis(20)), + None, + ); + + assert!( + matches!(outcome_of(&outcomes, "a"), CallOutcome::Resolved(_)), + "should keep a response that arrived in flight" + ); + assert!( + matches!(outcome_of(&outcomes, "b"), CallOutcome::HoldCutOff), + "should drop a call still pending when the budget ran out" + ); + assert!( + matches!(outcome_of(&outcomes, "c"), CallOutcome::HoldCutOff), + "should drop a call the budget ran out before waiting on" + ); + assert_eq!( + fixture.backend.first_byte_timeouts().len(), + 3, + "should have sent all three before the budget ran out" + ); + } + + #[test] + fn hold_failure_before_the_budget_is_spent_is_a_transport_failure() { + let mut fixture = + DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.settings.ec.identity.providers.insert( + "p".to_owned(), + ProviderSettings { + timeout_ms: Some(10), + ..ProviderSettings::default() + }, + ); + fixture.client.time_out("p.example.com"); + + let outcomes = fixture.run(&["p"], 1_000, Some(Duration::from_secs(60)), None); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::TransportFailure), + "should count a failure under timeout_ms as a transport failure, not a cut off" + ); + } + + #[test] + fn each_call_reads_only_its_own_providers_state() { + let fixture = DispatchFixture::new(vec![ + ScriptedProvider::new("one", "one.example.com", "one.example.com", false) + .stateful() + .into_provider(), + ScriptedProvider::new("two", "two.example.com", "two.example.com", false) + .stateful() + .into_provider(), + ScriptedProvider::new("three", "three.example.com", "three.example.com", false) + .stateful() + .into_provider(), + ]); + for host in ["one", "two", "three"] { + fixture + .client + .respond(&format!("{host}.example.com"), 200, b"{}"); + } + + fixture.run_with_state( + &["one", "two", "three"], + &[ + ("one", r#"{"token":"first-secret"}"#), + ("two", r#"{"token":"second-secret"}"#), + ], + 1_000, + None, + None, + ); + + let uris = fixture.client.sent_uris(); + let uri_for = |host: &str| { + uris.iter() + .find(|uri| uri.contains(host)) + .expect("should have sent to the host") + .clone() + }; + let first = uri_for("one.example.com"); + assert!( + first.contains("first-secret") && !first.contains("second-secret"), + "should give the first provider only its own state" + ); + let second = uri_for("two.example.com"); + assert!( + second.contains("second-secret") && !second.contains("first-secret"), + "should give the second provider only its own state" + ); + assert!( + uri_for("three.example.com").contains("state=none"), + "should give a provider with no stored state none" + ); + } + + #[test] + fn hold_dispatch_with_no_budget_sends_nothing() { + let fixture = DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.client.respond("p.example.com", 200, b"{}"); + + let outcomes = fixture.run(&["p"], 1_000, Some(Duration::ZERO), None); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::HoldCutOff), + "should cut off a hold call with no budget left" + ); + assert!( + fixture.client.sent_hosts().is_empty(), + "should send nothing" + ); + } + + #[test] + fn hold_dispatch_waits_the_smaller_of_the_hold_timeout_and_the_budget() { + let mut fixture = + DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.settings.ec.identity.hold_budget_ms = 80; + fixture.client.respond("p.example.com", 200, b"{}"); + + fixture.run(&["p"], 1_000, Some(Duration::from_secs(10)), None); + fixture.run(&["p"], 1_000, Some(Duration::from_millis(30)), None); + + let timeouts = fixture.backend.first_byte_timeouts(); + assert_eq!( + timeouts[0], + Duration::from_millis(80), + "should wait the hold timeout when the budget is larger" + ); + assert!( + timeouts[1] <= Duration::from_millis(30), + "should wait no longer than the remaining budget" + ); + } + + #[test] + fn per_ip_ceiling_caps_calls_from_one_address() { + let mut fixture = + DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.settings.ec.identity.providers.insert( + "p".to_owned(), + ProviderSettings { + max_calls_per_minute: 10_000, + max_calls_per_ip_per_minute: 2, + ..ProviderSettings::default() + }, + ); + fixture.client.respond("p.example.com", 200, b"{}"); + let from = |ip: &str| ClientInfo { + client_ip: Some(ip.parse().expect("should parse the IP")), + ..ClientInfo::default() + }; + let limiter = FakeRateLimiter::new(); + + fixture.services = fake_services_from(&fixture.client, &fixture.backend, from("192.0.2.1")); + let first_two: Vec<_> = (0..2) + .map(|_| fixture.run(&["p"], 1_000, None, Some(&limiter))) + .collect(); + let third = fixture.run(&["p"], 1_000, None, Some(&limiter)); + fixture.services = + fake_services_from(&fixture.client, &fixture.backend, from("198.51.100.1")); + let elsewhere = fixture.run(&["p"], 1_000, None, Some(&limiter)); + + assert!( + first_two + .iter() + .all(|outcomes| matches!(outcome_of(outcomes, "p"), CallOutcome::Resolved(_))), + "should send while the address is under its ceiling" + ); + assert!( + matches!(outcome_of(&third, "p"), CallOutcome::BurstCapped), + "should cap a third call from the same address within the minute" + ); + assert!( + matches!(outcome_of(&elsewhere, "p"), CallOutcome::Resolved(_)), + "control: another address has its own ceiling" + ); + assert!( + limiter.keys().iter().all(|key| !key.contains("192.0.2.1")), + "should key the ceiling on a keyed hash, never the raw IP" + ); + } + + #[test] + fn after_send_dispatch_stops_at_the_backend_request_budget() { + let mut fixture = DispatchFixture::new(vec![ + scripted("p", "p.example.com", "p.example.com"), + scripted("q", "q.example.com", "q.example.com"), + ]); + fixture.settings.ec.identity.max_backend_requests = 3; + fixture.client.started_earlier(2); + fixture.client.respond("p.example.com", 200, b"{}"); + fixture.client.respond("q.example.com", 200, b"{}"); + let limiter = FakeRateLimiter::new(); + + let outcomes = fixture.run(&["p", "q"], 1_000, None, Some(&limiter)); + + assert_eq!( + fixture.client.sent_hosts().len(), + 1, + "should send only what the budget leaves after the origin and auction" + ); + assert!( + outcomes + .iter() + .any(|(_, outcome)| matches!(outcome, CallOutcome::OverBackendBudget)), + "should report the call it did not send" + ); + assert!( + limiter + .keys() + .iter() + .all(|key| !key.starts_with("identity-fail:")), + "should not count a skipped call as a breaker failure" + ); + } + + #[test] + fn hold_dispatch_reserves_the_origin_and_the_auction() { + let mut fixture = + DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.settings.ec.identity.max_backend_requests = 3; + fixture.settings.auction.enabled = true; + fixture.settings.auction.mediator = None; + fixture.settings.auction.providers = + crate::auction_config_types::AuctionConfig::legacy_provider_map(&["one", "two"]); + fixture.client.respond("p.example.com", 200, b"{}"); + + let held = fixture.run(&["p"], 1_000, Some(Duration::from_secs(1)), None); + let after_send = fixture.run(&["p"], 1_000, None, None); + + assert!( + matches!(outcome_of(&held, "p"), CallOutcome::OverBackendBudget), + "should not hold a call that would leave no room for the origin and two auction calls" + ); + assert!( + !matches!(outcome_of(&after_send, "p"), CallOutcome::OverBackendBudget), + "control: after send nothing is reserved" + ); + } + + #[test] + fn hold_dispatch_caps_the_connect_timeout_at_the_call_timeout() { + let mut fixture = + DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.settings.ec.identity.hold_budget_ms = 80; + fixture.client.respond("p.example.com", 200, b"{}"); + + fixture.run(&["p"], 1_000, Some(Duration::from_secs(10)), None); + fixture.run(&["p"], 1_000, None, None); + + assert_eq!( + fixture.backend.connect_timeouts(), + vec![Some(Duration::from_millis(80)), None], + "should cap a hold call's connect timeout at its call timeout and leave an after-send call on the adapter default" + ); + } + + #[test] + fn burst_cap_skips_without_counting_a_failure() { + let fixture = DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.client.respond("p.example.com", 200, b"{}"); + let capped = RecordingLimiter::new(true, false); + let broken = RecordingLimiter::new(false, true); + + let outcomes = fixture.run(&["p"], 1_000, None, Some(&capped)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::BurstCapped), + "should report a burst-capped call" + ); + assert!( + fixture.client.sent_hosts().is_empty(), + "should not send a burst-capped call" + ); + let keys = capped.keys.lock().expect("should lock the keys"); + assert!( + keys[0].starts_with("identity:p:") && !keys[0].contains(&"a".repeat(64)), + "should key the cap by provider and the keyed visitor hash, not the EC ID" + ); + drop(keys); + + let outcomes = fixture.run(&["p"], 1_000, None, Some(&broken)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::NotDispatched), + "should not dispatch when the limiter cannot be read" + ); + assert!( + fixture.client.sent_hosts().is_empty(), + "should not send when the limiter cannot be read" + ); + + let allowing = RecordingLimiter::new(false, false); + let outcomes = fixture.run(&["p"], 1_000, None, Some(&allowing)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::Resolved(_)), + "should dispatch under the cap (control)" + ); + } + + /// A fixture with provider `p` whose breaker opens at `threshold` and + /// whose burst cap never stops a test. + fn breaker_fixture(threshold: u32) -> DispatchFixture { + let mut fixture = + DispatchFixture::new(vec![scripted("p", "p.example.com", "p.example.com")]); + fixture.settings.ec.identity.providers.insert( + "p".to_owned(), + ProviderSettings { + breaker_failures_per_minute: threshold, + max_calls_per_minute: 10_000, + ..ProviderSettings::default() + }, + ); + fixture + } + + #[test] + fn breaker_opens_at_threshold_and_skips_every_visitor() { + let fixture = breaker_fixture(3); + fixture.client.respond("p.example.com", 503, b"{}"); + let limiter = FakeRateLimiter::new(); + + for call in 0..3 { + let outcomes = fixture.run(&["p"], 1_000, None, Some(&limiter)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::TransportFailure), + "should dispatch while the breaker is closed (call {call})" + ); + } + assert_eq!( + limiter.count_of("identity-fail:p"), + 3, + "should record each transport failure" + ); + let sent_before = fixture.client.sent_hosts().len(); + let burst_before = limiter.keys(); + + let outcomes = fixture.run(&["p"], 1_000, None, Some(&limiter)); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::BreakerOpen), + "should skip the provider once the count reaches the threshold" + ); + assert_eq!( + fixture.client.sent_hosts().len(), + sent_before, + "should send nothing while the breaker is open" + ); + assert_eq!( + limiter.count_of("identity-fail:p"), + 3, + "should not record a failure for a call it never made" + ); + assert_eq!( + limiter.keys(), + burst_before, + "should not touch the burst counter of an open breaker" + ); + } + + #[test] + fn breaker_is_per_provider() { + let mut fixture = breaker_fixture(1); + fixture.catalog = ProviderCatalog::new( + vec![ + scripted("p", "p.example.com", "p.example.com"), + scripted("q", "q.example.com", "q.example.com"), + ], + BTreeMap::new(), + ); + fixture.client.respond("q.example.com", 200, b"{}"); + let limiter = FakeRateLimiter::new().with_count("identity-fail:p", 1); + + let outcomes = fixture.run(&["p", "q"], 1_000, None, Some(&limiter)); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::BreakerOpen), + "should open the breaker of the failing provider" + ); + assert!( + matches!(outcome_of(&outcomes, "q"), CallOutcome::Resolved(_)), + "should leave another provider's breaker closed (control)" + ); + } + + #[test] + fn four_xx_and_malformed_never_open_the_breaker() { + let fixture = breaker_fixture(50); + fixture.client.respond("p.example.com", 400, b"{}"); + let limiter = FakeRateLimiter::new(); + + for _ in 0..1_000 { + let outcomes = fixture.run(&["p"], 1_000, None, Some(&limiter)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::InputFailure), + "should count a 400 as an input failure" + ); + } + assert_eq!( + limiter.count_of("identity-fail:p"), + 0, + "should never record an input failure" + ); + + fixture.client.respond("p.example.com", 503, b"{}"); + fixture.run(&["p"], 1_000, None, Some(&limiter)); + assert_eq!( + limiter.count_of("identity-fail:p"), + 1, + "should record a 503 (control)" + ); + } + + #[test] + fn hold_cut_off_does_not_record_a_failure() { + let fixture = breaker_fixture(50); + fixture.client.respond("p.example.com", 200, b"{}"); + let limiter = FakeRateLimiter::new(); + + let outcomes = fixture.run(&["p"], 1_000, Some(Duration::ZERO), Some(&limiter)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::HoldCutOff), + "should cut off a hold call with no budget" + ); + assert_eq!( + limiter.count_of("identity-fail:p"), + 0, + "should not record a budget cut as a provider failure" + ); + + fixture.client.time_out("p.example.com"); + let outcomes = fixture.run(&["p"], 1_000, None, Some(&limiter)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::TransportFailure), + "should count an after-send timeout as a transport failure" + ); + assert_eq!( + limiter.count_of("identity-fail:p"), + 1, + "should record a timeout (control)" + ); + } + + #[test] + fn burst_cap_is_checked_after_every_other_eligibility_step() { + let mut fixture = breaker_fixture(1); + fixture.catalog = ProviderCatalog::new( + vec![ + scripted("p", "p.example.com", "p.example.com"), + ScriptedProvider::new("u", "u.example.com", "u.example.com", false) + .reading_undeclared_user_agent() + .into_provider(), + ], + BTreeMap::new(), + ); + fixture.client.respond("p.example.com", 200, b"{}"); + fixture.client.respond("u.example.com", 200, b"{}"); + let limiter = FakeRateLimiter::new().with_count("identity-fail:p", 1); + + // The breaker is open for `p`, and `u` fails its input check. + let outcomes = fixture.run(&["p", "u"], 1_000, None, Some(&limiter)); + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::BreakerOpen), + "should stop `p` at the breaker" + ); + assert!( + matches!(outcome_of(&outcomes, "u"), CallOutcome::InputFailure), + "should stop `u` at its input check" + ); + // A hold call with no budget left is cut off. + fixture.run(&["u"], 1_000, Some(Duration::ZERO), Some(&limiter)); + assert_eq!( + limiter.keys(), + vec!["identity-fail:p".to_owned()], + "should never touch the burst counter of a provider that stopped earlier" + ); + + let eligible = breaker_fixture(1); + eligible.client.respond("p.example.com", 200, b"{}"); + let open_limiter = FakeRateLimiter::new(); + eligible.run(&["p"], 1_000, None, Some(&open_limiter)); + assert!( + open_limiter + .keys() + .iter() + .any(|key| key.starts_with("identity:p:")), + "should increment the burst counter of a provider that reaches dispatch (control)" + ); + } + + #[test] + fn breaker_count_error_treats_the_breaker_as_closed() { + let fixture = breaker_fixture(1); + fixture.client.respond("p.example.com", 200, b"{}"); + let limiter = FakeRateLimiter::new() + .with_count("identity-fail:p", 100) + .failing_count(); + + let outcomes = fixture.run(&["p"], 1_000, None, Some(&limiter)); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::Resolved(_)), + "should dispatch when the failure count cannot be read" + ); + assert_eq!(fixture.client.sent_hosts().len(), 1, "should send the call"); + } + + #[test] + fn burst_cap_error_skips_the_dispatch() { + let fixture = breaker_fixture(50); + fixture.client.respond("p.example.com", 200, b"{}"); + let limiter = FakeRateLimiter::new().failing_exceeded(); + + let outcomes = fixture.run(&["p"], 1_000, None, Some(&limiter)); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::NotDispatched), + "should not dispatch when the burst cap cannot be read" + ); + assert!( + fixture.client.sent_hosts().is_empty(), + "should send nothing" + ); + } + + #[test] + fn record_error_keeps_the_outcome() { + let fixture = breaker_fixture(50); + fixture.client.respond("p.example.com", 503, b"{}"); + let limiter = FakeRateLimiter::new().failing_record(); + + let outcomes = fixture.run(&["p"], 1_000, None, Some(&limiter)); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::TransportFailure), + "should keep the transport failure when recording it fails" + ); + assert_eq!( + limiter.count_of("identity-fail:p"), + 0, + "should not have recorded it" + ); + } + + #[test] + fn undeclared_read_is_an_input_failure_even_when_ignored() { + let fixture = DispatchFixture::new(vec![ + ScriptedProvider::new("p", "p.example.com", "p.example.com", false) + .reading_undeclared_user_agent() + .into_provider(), + ]); + fixture.client.respond("p.example.com", 200, b"{}"); + + let outcomes = fixture.run(&["p"], 1_000, None, None); + + assert!( + matches!(outcome_of(&outcomes, "p"), CallOutcome::InputFailure), + "should fail a provider that read an undeclared input, though it ignored the error" + ); + assert!( + fixture.client.sent_hosts().is_empty(), + "should not send the call" + ); + } + + #[test] + fn generic_pull_dispatch_order_rotates_hourly() { + let generic = + |id: &str, host: &str| ScriptedProvider::new(id, host, host, true).into_provider(); + let mut fixture = DispatchFixture::new(vec![ + generic("gamma", "gamma.example.com"), + generic("alpha", "alpha.example.com"), + generic("beta", "beta.example.com"), + scripted("other", "other.example.com", "other.example.com"), + ]); + fixture.settings.ec.identity.max_concurrency = 1; + for host in ["alpha", "beta", "gamma", "other"] { + fixture + .client + .respond(&format!("{host}.example.com"), 200, b"{}"); + } + let ids = ["other", "gamma", "beta", "alpha"]; + + fixture.run(&ids, 100, None, None); + fixture.run(&ids, 3_600, None, None); + fixture.run(&ids, 7_200, None, None); + + let sent = fixture.client.sent_hosts(); + assert_eq!( + sent, + [ + "alpha.example.com", + "beta.example.com", + "gamma.example.com", + "other.example.com", + "beta.example.com", + "gamma.example.com", + "alpha.example.com", + "other.example.com", + "gamma.example.com", + "alpha.example.com", + "beta.example.com", + "other.example.com", + ], + "should sort generic pull by source, rotate the start each hour, and send other providers after" + ); + } + + fn regrant_descriptor(policy: ConsentPolicy) -> ProviderDescriptor { + custom_provider("regrant", &["a.example.com"], Resolves::new(), policy) + .descriptor() + .clone() + } + + fn stored_opt_out(at: u64, fp: &str) -> KvOptOut { + KvOptOut { + at, + fp: fp.to_owned(), + } + } + + fn gpp_consent(sale: Option, targeted: Option) -> ConsentContext { + let mut consent = allowing_consent(); + consent.gpp = Some(GppConsent { + version: 1, + section_ids: vec![7], + eu_tcf: None, + us_sale_opt_out: sale, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: targeted, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }); + consent + } + + #[test] + fn tcf_regrant_clears_only_a_denial_the_opt_out_recorded() { + let descriptor = regrant_descriptor(resolver_policy().with_tcf_vendor(5)); + let signals = PolicySignals::default(); + // `test_consent_gdpr_without_vendor(v)` grants every purpose and every + // vendor but `v`, so 99 grants vendor 5 and 5 denies it. + let tcf_with = |denied_vendor: u16, last_updated_ds: u64| { + let mut consent = test_consent_gdpr_without_vendor(denied_vendor); + consent + .tcf + .as_mut() + .expect("should carry a TC string") + .last_updated_ds = last_updated_ds; + consent + }; + let recorded_under = |consent: &ConsentContext| { + stored_opt_out( + 1_000, + &fingerprint_bits(descriptor.consent_policy(), consent, signals), + ) + }; + let denied = recorded_under(&tcf_with(5, 0)); + let granted = recorded_under(&tcf_with(99, 0)); + + assert!( + regrant_clears_opt_out(&descriptor, &denied, &tcf_with(99, 0), signals), + "should clear when the vendor goes from denied to granted, even on a day-level LastUpdated before the opt-out" + ); + assert!( + !regrant_clears_opt_out(&descriptor, &denied, &tcf_with(5, 20_000), signals), + "should not clear while the vendor stays denied" + ); + assert!( + !regrant_clears_opt_out(&descriptor, &granted, &tcf_with(99, 20_000), signals), + "should not clear a vendor-reported opt-out on a later CMP re-save that was already granting" + ); + assert!( + !regrant_clears_opt_out( + &descriptor, + &stored_opt_out(1_000, "0"), + &tcf_with(99, 20_000), + signals + ), + "should not clear an opt-out recorded where TCF did not apply" + ); + } + + #[test] + fn opt_out_clears_carry_the_opt_out_they_saw() { + let catalog = ProviderCatalog::new( + vec![custom_provider( + "regrant", + &["a.example.com"], + Resolves::new(), + resolver_policy(), + )], + BTreeMap::new(), + ); + let mut view = EligibilityView::empty(); + let sale = gpp_consent(Some(true), None); + let fp = fingerprint_bits(&resolver_policy(), &sale, PolicySignals::default()); + view.set_opt_out_for_test("regrant", 700, None); + view.providers + .get_mut(&pid("regrant")) + .expect("should hold the provider") + .o = Some(stored_opt_out(700, &fp)); + + assert_eq!( + opt_out_clears( + &view, + &catalog, + &gpp_consent(Some(false), None), + PolicySignals::default() + ), + vec![(pid("regrant"), 700)], + "should clear up to the opt-out the request read, not to its own time" + ); + } + + #[test] + fn jurisdiction_change_never_clears_it() { + let descriptor = regrant_descriptor(resolver_policy()); + let signals = PolicySignals::default(); + let sale = stored_opt_out(1_000, "800"); + + let mut eu_without_gpp = allowing_consent(); + eu_without_gpp.jurisdiction = crate::consent::jurisdiction::Jurisdiction::Gdpr; + let mut header_only = allowing_consent(); + header_only.gpp = Some(GppConsent { + version: 1, + section_ids: vec![2], + eu_tcf: None, + us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }); + + assert!( + !regrant_clears_opt_out(&descriptor, &sale, &eu_without_gpp, signals), + "should not clear when the visitor moved to a jurisdiction with no GPP string" + ); + assert!( + !regrant_clears_opt_out(&descriptor, &sale, &header_only, signals), + "should not clear on a newly present GPP string with no US section" + ); + assert!( + regrant_clears_opt_out(&descriptor, &sale, &gpp_consent(Some(false), None), signals), + "control: a US section showing no opt-out should clear" + ); + } + + #[test] + fn gpp_regrant_clears_when_none_of_the_recorded_opt_outs_remain() { + let descriptor = regrant_descriptor(resolver_policy()); + let signals = PolicySignals::default(); + let sale_opt_out = gpp_consent(Some(true), None); + let stored = fingerprint_bits(descriptor.consent_policy(), &sale_opt_out, signals); + assert_eq!(stored, "800", "should record the sale opt-out bit"); + // Purpose 4 was also granted when the opt-out was stored (bit 3). + let with_purpose_four = format!("{:x}", 0x800_u64 | 0x8); + let opt_out = stored_opt_out(1_000, &with_purpose_four); + + assert!( + regrant_clears_opt_out( + &descriptor, + &opt_out, + &gpp_consent(Some(false), None), + signals + ), + "should clear when the section shows no opt-out, even with purpose 4 changed" + ); + assert!( + !regrant_clears_opt_out(&descriptor, &opt_out, &sale_opt_out, signals), + "control: a section that still shows the sale opt-out should not clear" + ); + assert!( + !regrant_clears_opt_out( + &descriptor, + &stored_opt_out(1_000, "8"), + &gpp_consent(Some(false), None), + signals + ), + "control: an fp with no US opt-out bit should never clear by GPP" + ); + assert!( + !regrant_clears_opt_out( + &descriptor, + &stored_opt_out(1_000, "not-hex"), + &gpp_consent(Some(false), None), + signals + ), + "control: an fp that does not parse should never clear" + ); + assert!( + regrant_clears_opt_out( + &descriptor, + &opt_out, + &gpp_consent(Some(false), Some(true)), + signals + ), + "should judge only the opt-outs the fingerprint recorded, not a new one" + ); + } + + /// Hold dispatch before a writer navigation's auction (spec 4.7). + mod hold_mode { + use http::StatusCode; + + use super::*; + use crate::ec::identity::flush::FlushOutcome; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::test_support::TEST_EC_ID; + use crate::ec::identity::test_support::fake_http::AnsweringProvider; + use crate::ec::identity::test_support::hold_fixture::{HoldFixture, live_row}; + use crate::ec::kv_types::{KvOptOut, KvProviderState, KvUid}; + + const HOLD_HOST: &str = "p.example.com"; + const HOLD_SOURCE: &str = "ids.example.com"; + + fn hold_provider( + id: &str, + host: &str, + source: &str, + ) -> (Arc, ProviderMode) { + ( + AnsweringProvider::new(id, host, source, None).into_provider(), + ProviderMode::Hold, + ) + } + + fn found_body(uid: &str) -> Vec { + serde_json::to_vec(&serde_json::json!({ + "answer": "found", + "entries": [{ "uids": [{ "id": uid, "atype": 1 }] }], + })) + .expect("should serialize the answer") + } + + fn single_entry(uid: &str) -> Vec { + vec![KvEid { + uids: vec![KvUid { + id: uid.to_owned(), + atype: Some(1), + ext: None, + }], + ..KvEid::default() + }] + } + + #[test] + fn new_visitor_hold_answer_feeds_the_auction_and_rides_the_create() { + let mut fixture = HoldFixture::new_visitor(vec![ + hold_provider("p", HOLD_HOST, HOLD_SOURCE), + ( + AnsweringProvider::reading_ec_id( + "pull", + "pull.example.com", + "pull.example.com", + ) + .into_provider(), + ProviderMode::Hold, + ), + ]); + fixture + .client + .respond(HOLD_HOST, 200, &found_body("held-uid")); + fixture + .client + .respond("pull.example.com", 200, &found_body("pulled-uid")); + + let results = fixture.run_hold(); + + assert_eq!( + results.additions, + BTreeMap::from([(source(HOLD_SOURCE), single_entry("held-uid"))]), + "should hand this auction the answer" + ); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should call no provider that reads the EC ID before the row exists" + ); + assert_eq!(fixture.store.writes().total(), 0, "should not create yet"); + assert!( + fixture.context.identity().mutation().is_empty(), + "should stage nothing for a flush the new EC never runs" + ); + + fixture + .context + .complete_deferred_create(&fixture.settings, Some(&fixture.graph)) + .expect("should create the row"); + + let ec_id = fixture + .context + .ec_value() + .expect("should keep the EC ID") + .to_owned(); + let (row, _) = fixture + .graph + .get(&ec_id) + .expect("should read the row") + .expect("should hold the row"); + assert_eq!( + row.ids.get(HOLD_SOURCE).map(|record| record.eids.clone()), + Some(single_entry("held-uid")), + "should create the row with the hold answer" + ); + assert_eq!( + fixture.store.writes().add, + 1, + "should create with one write" + ); + } + + #[test] + fn hold_answer_feeds_this_auction_and_joins_the_mutation() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + fixture + .client + .respond(HOLD_HOST, 200, &found_body("held-uid")); + + let results = fixture.run_hold(); + + assert_eq!( + results, + HoldResults { + additions: BTreeMap::from([(source(HOLD_SOURCE), single_entry("held-uid"))]), + writer: BTreeMap::from([(source(HOLD_SOURCE), pid("p"))]), + }, + "should hand this auction the answer and its writer" + ); + let staged = fixture.context.identity().mutation().positive_changes(); + assert!( + matches!( + staged, + [PartnerIdChange::FillMissing { source: staged_source, record }] + if *staged_source == source(HOLD_SOURCE) + && record.w.as_deref() == Some("p") + && record.eids == single_entry("held-uid") + ), + "should stage the answer in the writer's mutation: {staged:?}" + ); + + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!(outcome, FlushOutcome::Written, "should write the answer"); + assert_eq!( + fixture.row().ids[HOLD_SOURCE].eids, + single_entry("held-uid"), + "should store the answer" + ); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should not dispatch the provider again after send" + ); + let writes = fixture.store.writes(); + assert_eq!( + (writes.if_generation_match, writes.total()), + (1, 1), + "should make the one flush write and no other" + ); + } + + #[test] + fn auction_starts_at_the_hold_budget_when_a_provider_is_slow() { + let mut fixture = HoldFixture::new( + vec![ + hold_provider("a", "a.example.com", "a.example.com"), + hold_provider("b", "b.example.com", "b.example.com"), + ], + &live_row(), + true, + ); + fixture.settings.ec.identity.hold_budget_ms = 50; + fixture + .client + .respond("a.example.com", 200, &found_body("fast-uid")); + fixture + .client + .respond("b.example.com", 200, &found_body("late-uid")); + let delay = Duration::from_millis(400); + fixture.client.delay("b.example.com", delay); + + let before = current_timestamp(); + let started = Instant::now(); + let results = fixture.run_hold(); + let elapsed = started.elapsed(); + let outcome = fixture.flush(StatusCode::OK); + + assert!( + // The call's timeout is the budget left when it was sent. + elapsed >= Duration::from_millis(45) && elapsed < delay, + "should start the auction at the budget, not at the slow answer: {elapsed:?}" + ); + assert_eq!( + results.additions.keys().collect::>(), + [&source("a.example.com")], + "should keep the fast answer and drop the slow one" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write the flush"); + let row = fixture.row(); + let retry = fixture.settings.ec.identity.hold_retry_sec; + assert!( + row.ps["b"] + .n + .is_some_and(|n| n >= before + retry && n <= before + retry + 5), + "should wait hold_retry_sec after the cut-off: {:?}", + row.ps["b"] + ); + assert_eq!(row.ps["b"].f, 0, "should count no failure for the cut-off"); + assert_eq!( + fixture.limiter.count_of("identity-fail:b"), + 0, + "should not feed the breaker" + ); + assert!( + row.ids.contains_key("a.example.com") && !row.ids.contains_key("b.example.com"), + "should store the fast answer and not the slow one" + ); + } + + #[test] + fn hold_providers_still_run_after_send_when_hold_found_no_usable_row() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + fixture + .client + .respond(HOLD_HOST, 200, &found_body("late-uid")); + fixture + .context + .set_kv_snapshot(crate::ec::EcKvSnapshot::NotRead); + + let held = fixture.run_hold(); + let before_send = fixture.client.sent_hosts(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!( + (held, before_send), + (HoldResults::default(), Vec::::new()), + "should dispatch nothing in hold without a usable row" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write the answer"); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should run the hold-mode provider once, after send" + ); + assert_eq!( + fixture.row().ids[HOLD_SOURCE].eids, + single_entry("late-uid"), + "should store the answer" + ); + } + + #[test] + fn cut_off_hold_call_sets_hold_retry_without_counting_a_failure() { + let mut row = live_row(); + row.ps.insert( + "p".to_owned(), + KvProviderState { + f: 2, + ..KvProviderState::default() + }, + ); + let mut fixture = + HoldFixture::new(vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], &row, true); + fixture.settings.ec.identity.hold_budget_ms = 0; + fixture.client.respond(HOLD_HOST, 200, &found_body("never")); + + let before = current_timestamp(); + let results = fixture.run_hold(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!(results, HoldResults::default(), "should add nothing"); + assert_eq!( + outcome, + FlushOutcome::Written, + "should write the retry wait" + ); + let state = fixture.row().ps["p"].clone(); + let retry = fixture.settings.ec.identity.hold_retry_sec; + assert!( + state + .n + .is_some_and(|n| n >= before + retry && n <= before + retry + 5), + "should wait hold_retry_sec: {state:?}" + ); + assert_eq!(state.f, 2, "should leave the failure count as it was"); + assert_eq!( + fixture.limiter.count_of("identity-fail:p"), + 0, + "should not feed the breaker" + ); + } + + #[test] + fn hold_kv_opt_out_blocks_dispatch() { + let mut opted_out = live_row(); + opted_out.ps.insert( + "p".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 900, + fp: "0".to_owned(), + }), + ..KvProviderState::default() + }, + ); + let mut blocked = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &opted_out, + true, + ); + blocked + .client + .respond(HOLD_HOST, 200, &found_body("blocked")); + let mut control = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + control + .client + .respond(HOLD_HOST, 200, &found_body("allowed")); + + let blocked_results = blocked.run_hold(); + let control_results = control.run_hold(); + + assert_eq!( + (blocked_results, blocked.client.sent_hosts()), + (HoldResults::default(), Vec::::new()), + "should call no provider the row opted out of" + ); + assert_eq!( + (control_results.additions.len(), control.client.sent_hosts()), + (1, vec![HOLD_HOST.to_owned()]), + "control: should call the provider with no opt-out" + ); + } + + #[test] + fn no_hold_on_a_stager() { + let mut stager = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + stager.client.respond(HOLD_HOST, 200, &found_body("stager")); + stager + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: false, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + let mut writer = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + writer.client.respond(HOLD_HOST, 200, &found_body("writer")); + + let stager_results = stager.run_hold(); + let writer_results = writer.run_hold(); + + assert_eq!( + (stager_results, stager.client.sent_hosts()), + (HoldResults::default(), Vec::::new()), + "should make no provider call on a request that is not the writer" + ); + assert_eq!( + (writer_results.additions.len(), writer.client.sent_hosts()), + (1, vec![HOLD_HOST.to_owned()]), + "control: should call the provider on the writer" + ); + } + + #[test] + fn hold_no_match_on_a_writer_stops_recalls_from_the_spa_session() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + fixture + .client + .respond(HOLD_HOST, 200, br#"{"answer":"no_match"}"#); + fixture.run_hold(); + assert_eq!( + fixture.flush(StatusCode::OK), + FlushOutcome::Written, + "should write the no_match wait" + ); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "control: the writer navigation called the provider once" + ); + assert!( + fixture.row().ps["p"] + .n + .is_some_and(|n| n > current_timestamp()), + "should hold the provider back with ps.n" + ); + + // The SPA session's stagers (`/auction`, `/_ts/page-bids`) read + // the stored row and are never the writer. + for _ in 0..3 { + fixture + .context + .set_kv_snapshot(fixture.graph.load_snapshot(TEST_EC_ID)); + fixture + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: false, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + fixture.run_hold(); + } + // A second writer navigation inside ps.n is held back too. + fixture + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + fixture.run_hold(); + + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should never call the provider again before ps.n" + ); + } + + #[test] + fn hold_provider_moves_after_send_on_a_navigation_without_an_auction() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + false, + ); + fixture.client.time_out(HOLD_HOST); + + let held = fixture.run_hold(); + let before_send = fixture.client.sent_hosts(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!( + (held, before_send), + (HoldResults::default(), Vec::::new()), + "should dispatch nothing before the response" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write the failure"); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should dispatch after send" + ); + assert_eq!( + fixture.backend.first_byte_timeouts(), + [Duration::from_millis( + fixture + .settings + .ec + .identity + .provider(&pid("p")) + .after_send_timeout_ms + )], + "should wait the after-send timeout, not the hold budget" + ); + assert_eq!( + fixture.row().ps["p"].f, + 1, + "should count a miss as a transport failure" + ); + assert_eq!( + fixture.limiter.count_of("identity-fail:p"), + 1, + "should feed the breaker, which a cut-off never does" + ); + } + + #[test] + fn stateful_hold_answer_flushes_its_state_with_its_ids() { + let mut fixture = HoldFixture::new( + vec![( + AnsweringProvider::stateful("p", HOLD_HOST, HOLD_SOURCE).into_provider(), + ProviderMode::Hold, + )], + &live_row(), + true, + ); + fixture + .client + .respond(HOLD_HOST, 200, &found_body("stateful-uid")); + + let results = fixture.run_hold(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!( + results.additions.get(&source(HOLD_SOURCE)), + Some(&single_entry("stateful-uid")), + "should feed this auction" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write once"); + let row = fixture.row(); + assert_eq!( + row.ids[HOLD_SOURCE].eids, + single_entry("stateful-uid"), + "should store the IDs" + ); + assert_eq!( + row.ps["p"].st, + Some(serde_json::json!({ "seen": true })), + "should store the state with its IDs" + ); + assert_eq!(fixture.store.writes().total(), 1, "should make one write"); + } + + #[test] + fn writer_navigation_with_non_2xx_status_flushes_hold_removals_and_backoff_only() { + let mut row = live_row(); + row.ids.insert( + "i.example.com".to_owned(), + KvPartnerId { + exp: Some(600), + ..record_for_test("old-uid", Some("i"), 500) + }, + ); + row.ps.insert( + "f".to_owned(), + KvProviderState { + f: 2, + ..KvProviderState::default() + }, + ); + let mut fixture = HoldFixture::new( + vec![ + hold_provider("f", "f.example.com", "f.example.com"), + hold_provider("n", "n.example.com", "n.example.com"), + hold_provider("i", "i.example.com", "i.example.com"), + ], + &row, + true, + ); + fixture + .client + .respond("f.example.com", 200, &found_body("found-uid")); + fixture + .client + .respond("n.example.com", 200, br#"{"answer":"no_match"}"#); + fixture + .client + .respond("i.example.com", 200, br#"{"answer":"invalidate"}"#); + + let results = fixture.run_hold(); + let outcome = fixture.flush(StatusCode::NOT_FOUND); + + assert_eq!( + results.additions.keys().collect::>(), + [&source("f.example.com")], + "should still feed this auction the found answer" + ); + assert_eq!( + outcome, + FlushOutcome::Written, + "should write the cut mutation" + ); + let stored = fixture.row(); + assert!( + !stored.ids.contains_key("f.example.com"), + "should store no ID from the found" + ); + assert!( + !stored.ids.contains_key("i.example.com"), + "should store the invalidate's removal" + ); + assert!( + stored.ps["n"].n.is_some_and(|n| n > current_timestamp()), + "should store the no_match backoff" + ); + assert_eq!( + stored.ps["f"].f, 2, + "should leave the found's failure count alone" + ); + assert_eq!(fixture.store.writes().total(), 1, "should make one write"); + } + } + + /// Revoke-only capture from proxied responses (spec 4.7). + mod capture_run { + use serde_json::json; + + use super::*; + use crate::consent::jurisdiction::Jurisdiction; + use crate::ec::identity::revision::set_hashes; + use crate::ec::identity::test_support::capture_fixture::{CaptureFixture, capture_path}; + use crate::ec::identity::test_support::hold_fixture::live_row; + use crate::ec::identity::test_support::{ + TEST_EC_ID, answering_capture_provider, record_for_test, + }; + use crate::ec::kv_types::KvProviderState; + use crate::test_support::logs::{captured_logs, install_capturing_logger}; + + const FIRST: &str = "a.example.com"; + const SECOND: &str = "b.example.com"; + + fn lockr() -> Arc { + answering_capture_provider("lockr", &[FIRST, SECOND], None) + } + + fn lockr_path() -> String { + capture_path("lockr") + } + + fn row_with(records: &[(&str, &str, Option<&str>)]) -> KvEntry { + let mut row = live_row(); + for (source, uid, writer) in records { + row.ids + .insert((*source).to_owned(), record_for_test(uid, *writer, 500)); + } + row + } + + fn opted_out_row(at: u64) -> KvEntry { + let mut row = live_row(); + row.ps.insert( + "lockr".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at, + fp: "0".to_owned(), + }), + ..KvProviderState::default() + }, + ); + row + } + + fn denied_consent() -> ConsentContext { + ConsentContext { + gdpr_applies: true, + jurisdiction: Jurisdiction::Gdpr, + ..ConsentContext::default() + } + } + + fn kept_sources(mutation: &IdentityMutation) -> Vec<&str> { + mutation + .stand_alone_changes() + .iter() + .filter_map(|change| { + crate::ec::identity::coordinator::positive_record(change) + .map(|(source, _)| source.as_str()) + }) + .collect() + } + + #[test] + fn capture_positive_from_another_request_waits_in_the_pending_key() { + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&live_row())); + + fixture.capture( + &lockr_path(), + &json!({"tokens": [{"source": FIRST, "value": "secret-uid-1"}]}), + ); + + assert_eq!( + kept_sources(fixture.context.identity().capture_writes()), + vec![FIRST], + "should keep the positive for after send" + ); + assert_eq!( + fixture.store.writes().total(), + 0, + "should write nothing before send" + ); + + defer_capture_writes(&mut fixture.context, &fixture.graph); + + let merge = load_pending(&fixture.graph, TEST_EC_ID, 0).expect("should hold the write"); + assert_eq!( + kept_sources(&merge.mutation), + vec![FIRST], + "should append the positive to the pending key" + ); + assert_eq!( + fixture.row(), + live_row(), + "should leave the row to the next writer" + ); + } + + #[test] + fn capture_positive_from_the_writer_joins_its_flush() { + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&live_row())); + fixture + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + + fixture.capture( + &lockr_path(), + &json!({"tokens": [{"source": FIRST, "value": "secret-uid-1"}]}), + ); + + assert_eq!( + kept_sources(fixture.context.identity().mutation()), + vec![FIRST], + "should stage the positive in the writer's mutation" + ); + assert!( + fixture.context.identity().capture_writes().is_empty(), + "should keep nothing for the pending key" + ); + } + + #[test] + fn capture_positive_is_kept_only_from_a_browser_and_when_it_changes_kv() { + let body = json!({"tokens": [{"source": FIRST, "value": "secret-uid-1"}]}); + let mut bot = CaptureFixture::new(vec![lockr()], Some(&live_row())).not_a_browser(); + bot.capture(&lockr_path(), &body); + assert!( + bot.context.identity().capture_writes().is_empty(), + "should keep nothing from a client that is not a browser" + ); + + let mut first = CaptureFixture::new(vec![lockr()], Some(&live_row())); + first.capture(&lockr_path(), &body); + let (_, record) = first + .context + .identity() + .capture_writes() + .stand_alone_changes() + .iter() + .find_map(crate::ec::identity::coordinator::positive_record) + .expect("control: a first capture keeps the positive"); + let mut stored = live_row(); + stored.ids.insert(FIRST.to_owned(), record.clone()); + + let mut again = CaptureFixture::new(vec![lockr()], Some(&stored)); + again.capture(&lockr_path(), &body); + + assert!( + again.context.identity().capture_writes().is_empty(), + "should keep nothing KV already holds" + ); + } + + #[test] + fn capture_stores_no_positive_id_and_counts_it() { + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&live_row())); + + fixture.capture( + &lockr_path(), + &json!({"tokens": [ + {"source": FIRST, "value": "secret-uid-1"}, + {"source": SECOND, "value": "secret-uid-2"}, + ]}), + ); + + assert_eq!( + fixture.row(), + live_row(), + "should store neither positive ID" + ); + assert_eq!(fixture.store.writes().total(), 0, "should make no KV write"); + let counted = fixture.context.identity().capture_unpersisted(); + assert_eq!( + counted, + &BTreeMap::from([ + ((pid("lockr"), source(FIRST)), 1), + ((pid("lockr"), source(SECOND)), 1), + ]), + "should count one per source" + ); + assert!( + !format!("{counted:?}").contains("secret-uid"), + "should hold no value" + ); + } + + #[test] + fn capture_removal_is_written_before_send() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture( + &lockr_path(), + &json!({"tokens": [{"source": SECOND, "value": "secret-uid-b"}], + "invalidate": [FIRST]}), + ); + + let writes = fixture.store.writes(); + assert_eq!( + writes.if_generation_match, 1, + "should write once with a CAS" + ); + assert_eq!(writes.total(), 1, "should make no other write"); + let row = fixture.row(); + assert!(!row.ids.contains_key(FIRST), "should remove the record"); + assert!( + !row.ids.contains_key(SECOND), + "should store no positive ID alongside" + ); + assert_eq!( + row.ps["lockr"].iv, + set_hashes(&seeded.ids[FIRST].eids), + "should hash the removed record into iv" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should leave nothing to retry" + ); + } + + #[test] + fn capture_removal_applies_whatever_the_consent() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let body = json!({"invalidate": [FIRST]}); + + let mut granted = CaptureFixture::new(vec![lockr()], Some(&seeded)); + granted.capture(&lockr_path(), &body); + let mut denied = + CaptureFixture::new(vec![lockr()], Some(&seeded)).with_consent(denied_consent()); + denied.capture(&lockr_path(), &body); + + assert_eq!( + denied.row(), + granted.row(), + "should write the same removal under denied consent" + ); + assert!( + !denied.row().ids.contains_key(FIRST), + "should remove the record" + ); + assert_eq!(denied.store.writes().total(), 1, "should write once"); + } + + #[test] + fn capture_removal_write_retries_only_on_generation_mismatch() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let body = json!({"invalidate": [FIRST]}); + + let mut mismatched = CaptureFixture::new(vec![lockr()], Some(&seeded)); + mismatched.store.refuse_conditional_writes(TEST_EC_ID, 2); + mismatched.capture(&lockr_path(), &body); + assert_eq!( + mismatched.store.writes().if_generation_match, + 3, + "should retry each generation mismatch" + ); + assert!( + !mismatched.row().ids.contains_key(FIRST), + "should win on the third write" + ); + + let mut failing = CaptureFixture::new(vec![lockr()], Some(&seeded)); + failing.store.fail_conditional_writes(TEST_EC_ID); + failing.capture(&lockr_path(), &body); + assert_eq!( + failing.store.writes().if_generation_match, + 1, + "should not retry a non-CAS error" + ); + } + + #[test] + fn failed_capture_removal_write_runs_once_more_after_send() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.store.refuse_conditional_writes(TEST_EC_ID, 5); + + fixture.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + + assert!( + fixture.row().ids.contains_key(FIRST), + "should leave the record after five refusals" + ); + assert!( + fixture.context.identity().pending_removals().is_some(), + "should keep the removal for the retry" + ); + assert_eq!( + fixture.context.identity().removal_lost(), + 0, + "should not count a removal lost yet" + ); + + fixture.retry_after_send(); + + assert!( + !fixture.row().ids.contains_key(FIRST), + "should write the removal after send" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should clear the pending removal" + ); + assert_eq!( + fixture.context.identity().removal_lost(), + 0, + "should count nothing lost" + ); + } + + #[test] + fn capture_removal_after_both_attempts_goes_to_the_pending_key() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.store.fail_conditional_writes(TEST_EC_ID); + + fixture.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + fixture.retry_after_send(); + + assert_eq!( + fixture.context.identity().removal_lost(), + 0, + "should lose nothing" + ); + assert_eq!( + fixture.context.identity().removal_deferred(), + 1, + "should defer the removal" + ); + let merge = + load_pending(&fixture.graph, TEST_EC_ID, 0).expect("should hold the removal"); + assert_eq!( + merge.mutation.removal_labels(), + vec![(pid("lockr"), Some(source(FIRST)))], + "should hold the capture's removal" + ); + } + + #[test] + fn capture_removal_lost_after_both_attempts_is_counted_and_logged_without_a_value() { + install_capturing_logger(); + let provider = answering_capture_provider("lostcap", &[FIRST], None); + let seeded = row_with(&[(FIRST, "secret-uid-lost-9137", None)]); + let mut fixture = CaptureFixture::new(vec![provider], Some(&seeded)); + fixture.store.fail_conditional_writes(TEST_EC_ID); + fixture.store.fail_all_writes(&pending_key(TEST_EC_ID)); + + fixture.capture(&capture_path("lostcap"), &json!({"invalidate": [FIRST]})); + fixture.retry_after_send(); + + assert_eq!( + fixture.context.identity().removal_lost(), + 1, + "should count the lost removal" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should keep nothing after the last attempt" + ); + assert!( + fixture.row().ids.contains_key(FIRST), + "control: the record is still there" + ); + let logs = captured_logs(); + assert!( + logs.iter().any(|line| line.contains("removal lost") + && line.contains("lostcap") + && line.contains(FIRST)), + "should log the provider and source: {logs:?}" + ); + assert!( + !logs + .iter() + .any(|line| line.contains("secret-uid-lost-9137")), + "should log no value" + ); + } + + #[test] + fn capture_removal_write_skips_a_tombstone_a_missing_row_and_an_applied_removal() { + let body = json!({"invalidate": [FIRST]}); + + let mut tombstone = + CaptureFixture::new(vec![lockr()], Some(&KvEntry::tombstone(1_000))); + tombstone.capture(&lockr_path(), &body); + let mut missing = CaptureFixture::new(vec![lockr()], None); + missing.capture(&lockr_path(), &body); + let mut applied = CaptureFixture::new(vec![lockr()], Some(&live_row())); + applied.capture(&lockr_path(), &body); + + for (name, fixture) in [ + ("a tombstone", &tombstone), + ("a missing row", &missing), + ("an applied removal", &applied), + ] { + assert_eq!( + fixture.store.writes().total(), + 0, + "should write nothing for {name}" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should keep nothing to retry for {name}" + ); + } + let mut control = CaptureFixture::new( + vec![lockr()], + Some(&row_with(&[(FIRST, "secret-uid-a", None)])), + ); + control.capture(&lockr_path(), &body); + assert_eq!( + control.store.writes().total(), + 1, + "control: a live record is written" + ); + } + + #[test] + fn capture_opt_out_removes_every_record_the_provider_wrote() { + let seeded = row_with(&[ + (FIRST, "secret-uid-a", Some("lockr")), + (SECOND, "secret-uid-b", Some("other")), + ]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture(&lockr_path(), &json!({"opted_out": true})); + + let row = fixture.row(); + assert!( + !row.ids.contains_key(FIRST), + "should remove the record the provider wrote" + ); + assert!( + row.ids.contains_key(SECOND), + "control: should keep another writer's record" + ); + assert!(row.ps["lockr"].o.is_some(), "should store the opt-out"); + assert_eq!( + row.ps["lockr"].ov, + set_hashes(&seeded.ids[FIRST].eids), + "should hash the removed record into ov" + ); + assert_eq!(fixture.store.writes().total(), 1, "should write once"); + } + + #[test] + fn capture_only_opt_out_also_removes_browser_records_of_its_sources() { + let seeded = row_with(&[ + (FIRST, "secret-uid-a", None), + ("unrelated.example.com", "secret-uid-c", None), + ]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture(&lockr_path(), &json!({"opted_out": true})); + + let row = fixture.row(); + assert!( + !row.ids.contains_key(FIRST), + "should remove the browser record of its source" + ); + assert!( + row.ids.contains_key("unrelated.example.com"), + "control: should keep a browser record of another source" + ); + } + + #[test] + fn capture_only_provider_clears_its_opt_out_on_a_consenting_positive_result() { + let seeded = opted_out_row(100); + let positive = json!({"tokens": [{"source": FIRST, "value": "secret-uid-a"}]}); + + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.capture(&lockr_path(), &positive); + let row = fixture.row(); + assert_eq!( + row.ps["lockr"].oc, + Some(100), + "should store the clear with the opt-out it saw as its watermark" + ); + assert!(row.ids.is_empty(), "should store no ID"); + assert_eq!(fixture.store.writes().total(), 1, "should write once"); + + let mut denied = + CaptureFixture::new(vec![lockr()], Some(&seeded)).with_consent(denied_consent()); + denied.capture(&lockr_path(), &positive); + assert_eq!( + denied.row().ps["lockr"].oc, + None, + "control: should not clear under denied consent" + ); + assert_eq!(denied.store.writes().total(), 0, "should not write"); + + let mut vendor = CaptureFixture::new( + vec![answering_capture_provider("lockr", &[FIRST], Some(7))], + Some(&seeded), + ); + vendor.capture(&lockr_path(), &positive); + assert_eq!( + vendor.row().ps["lockr"].oc, + None, + "control: a provider with a GVL ID never clears by capture" + ); + + let mut empty = CaptureFixture::new(vec![lockr()], Some(&seeded)); + empty.capture(&lockr_path(), &json!({})); + assert_eq!( + empty.row().ps["lockr"].oc, + None, + "control: a result without IDs does not clear" + ); + } + + #[test] + fn capture_global_revoke_from_a_real_browser_requests_the_tombstone_and_writes_nothing() { + let seeded = row_with(&[(FIRST, "secret-uid-a", Some("lockr"))]); + let mut global = CaptureFixture::with_scopes(vec![lockr()], Some(&seeded), &["lockr"]); + + global.capture(&lockr_path(), &json!({"opted_out": true})); + + assert!( + global.context.global_revoke_requested(), + "should request the global revoke" + ); + assert_eq!(global.row(), seeded, "should leave the row to finalize"); + assert_eq!(global.store.writes().total(), 0, "should write nothing"); + + let mut provider = CaptureFixture::new(vec![lockr()], Some(&seeded)); + provider.capture(&lockr_path(), &json!({"opted_out": true})); + assert!( + !provider.context.global_revoke_requested(), + "control: the provider scope requests no global revoke" + ); + assert!( + provider.row().ps["lockr"].o.is_some(), + "control: the provider scope stores the opt-out" + ); + } + + #[test] + fn capture_from_a_non_browser_applies_its_removals_and_opt_out() { + let seeded = row_with(&[(FIRST, "secret-uid-a", Some("lockr"))]); + + let mut invalidated = CaptureFixture::new(vec![lockr()], Some(&seeded)).not_a_browser(); + invalidated.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + assert!( + !invalidated.row().ids.contains_key(FIRST), + "should remove the invalidated record" + ); + assert_eq!( + invalidated.store.writes().if_generation_match, + 1, + "should write the removal at once" + ); + + let mut opted_out = CaptureFixture::new(vec![lockr()], Some(&seeded)).not_a_browser(); + opted_out.capture(&lockr_path(), &json!({"opted_out": true})); + let row = opted_out.row(); + assert!(row.ps["lockr"].o.is_some(), "should store the opt-out"); + assert!( + !row.ids.contains_key(FIRST), + "should remove what the provider wrote" + ); + + let mut global = CaptureFixture::with_scopes(vec![lockr()], Some(&seeded), &["lockr"]) + .not_a_browser(); + global.capture(&lockr_path(), &json!({"opted_out": true})); + assert!( + global.context.global_revoke_requested(), + "should request the global revoke, as a withdrawal from any client does" + ); + } + + #[test] + fn capture_from_a_non_browser_never_clears_an_opt_out() { + let seeded = opted_out_row(100); + let positive = json!({"tokens": [{"source": FIRST, "value": "secret-uid-a"}]}); + + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)).not_a_browser(); + fixture.capture(&lockr_path(), &positive); + + assert_eq!( + fixture.row().ps["lockr"].oc, + None, + "should not clear the opt-out" + ); + assert_eq!(fixture.store.writes().total(), 0, "should write nothing"); + assert_eq!( + fixture.context.identity().capture_unpersisted().len(), + 1, + "should still count the positive ID" + ); + } + + #[test] + fn capture_on_a_proxy_writer_navigation_uses_the_removal_write_not_the_mutation() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.context.identity_mut().set_writer_decision( + crate::ec::identity::request::WriterDecision { + writer: true, + is_real_browser: true, + user_agent: None, + dnt: false, + page_url: None, + }, + ); + + fixture.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + + assert!( + fixture.context.identity().mutation().is_empty(), + "should leave the writer's mutation alone" + ); + assert_eq!( + fixture.store.writes().if_generation_match, + 1, + "should write the removal at once" + ); + assert!( + !fixture.row().ids.contains_key(FIRST), + "should remove the record" + ); + } + + #[test] + fn capture_never_touches_n_f_or_the_breaker() { + let mut seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + seeded.ps.insert( + "lockr".to_owned(), + KvProviderState { + n: Some(9_999_999_999), + f: 2, + ..KvProviderState::default() + }, + ); + for body in [ + json!({"invalidate": [FIRST]}), + json!({"opted_out": true}), + json!({}), + ] { + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture(&lockr_path(), &body); + + let state = &fixture.row().ps["lockr"]; + assert_eq!(state.n, Some(9_999_999_999), "should keep n for {body}"); + assert_eq!(state.f, 2, "should keep f for {body}"); + } + } + + #[test] + fn capture_removal_lost_when_the_row_cannot_be_read() { + install_capturing_logger(); + let provider = answering_capture_provider("unreadcap", &[FIRST], None); + let seeded = row_with(&[(FIRST, "secret-uid-unread-5521", None)]); + let unreadable = |fixture: &mut CaptureFixture| { + fixture.context.set_kv_snapshot(EcKvSnapshot::Failed { + ec_id: TEST_EC_ID.to_owned(), + }); + }; + + // An invalidation needs the row's revision, so it is lost. + let mut invalidating = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + unreadable(&mut invalidating); + invalidating.capture(&capture_path("unreadcap"), &json!({"invalidate": [FIRST]})); + assert_eq!( + invalidating.context.identity().removal_lost(), + 1, + "should count the invalidation it could not build" + ); + assert_eq!( + invalidating.store.writes().total(), + 0, + "should write nothing" + ); + + // An opt-out needs no view: the removal write re-reads and writes it. + let mut opting_out = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + unreadable(&mut opting_out); + opting_out.capture(&capture_path("unreadcap"), &json!({"opted_out": true})); + assert_eq!( + opting_out.context.identity().removal_lost(), + 0, + "should not count the opt-out lost" + ); + let row = opting_out.row(); + assert!(row.ps["unreadcap"].o.is_some(), "should store the opt-out"); + assert!( + !row.ids.contains_key(FIRST), + "should remove the provider's source record" + ); + + // A failed opt-out write gets the after-send retry, then is lost + // when the pending key cannot take it either. + let mut failing = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + unreadable(&mut failing); + failing.store.fail_conditional_writes(TEST_EC_ID); + failing.store.fail_all_writes(&pending_key(TEST_EC_ID)); + failing.capture(&capture_path("unreadcap"), &json!({"opted_out": true})); + assert!( + failing.context.identity().pending_removals().is_some(), + "should stage the opt-out for the retry" + ); + assert_eq!(failing.context.identity().removal_lost(), 0, "not lost yet"); + failing.retry_after_send(); + assert_eq!( + failing.context.identity().removal_lost(), + 1, + "should count the opt-out lost once its retry fails" + ); + + let mut empty = CaptureFixture::new(vec![provider], Some(&seeded)); + unreadable(&mut empty); + empty.capture(&capture_path("unreadcap"), &json!({})); + assert_eq!( + empty.context.identity().removal_lost(), + 0, + "control: a result without removals loses nothing" + ); + + let logs = captured_logs(); + assert!( + logs.iter().any(|line| line.contains("removal lost") + && line.contains("unreadcap") + && line.contains(FIRST)), + "should log the provider and source: {logs:?}" + ); + assert!( + !logs + .iter() + .any(|line| line.contains("secret-uid-unread-5521")), + "should log no value" + ); + } + + #[test] + fn capture_counts_its_kv_operations_in_the_measurement() { + use crate::ec::identity::measurement::MeasurementInput; + + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.context.set_kv_snapshot(EcKvSnapshot::NotRead); + fixture.store.refuse_conditional_writes(TEST_EC_ID, 2); + + fixture.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + + let measurement = fixture.context.identity_measurement(&MeasurementInput { + phase_one_read: false, + kv_reads: 0, + kv_writes: 0, + flush: None, + withdrawal_after_send: false, + withdrawal_after_send_failed: false, + }); + assert_eq!( + measurement.kv_reads(), + 4, + "should count the view read and three removal-write reads" + ); + assert_eq!( + measurement.kv_writes(), + 3, + "should count each removal-write attempt" + ); + assert_eq!( + fixture.store.writes().if_generation_match, + 3, + "control: the store saw the same writes" + ); + } + + #[test] + fn capture_global_revoke_skips_the_other_providers_removals() { + let shared = "/integrations/shared/"; + let global = crate::ec::identity::test_support::answering_capture_provider_at( + "lockr", + &[FIRST], + None, + shared, + ); + let local = crate::ec::identity::test_support::answering_capture_provider_at( + "localcap", + &[SECOND], + None, + shared, + ); + let seeded = row_with(&[ + (FIRST, "secret-uid-a", Some("lockr")), + (SECOND, "secret-uid-b", None), + ]); + let mut fixture = + CaptureFixture::with_scopes(vec![global, local], Some(&seeded), &["lockr"]); + + fixture.capture( + "/integrations/shared/api", + &json!({"opted_out": true, "invalidate": [SECOND]}), + ); + + assert!( + fixture.context.global_revoke_requested(), + "should request the global revoke" + ); + assert_eq!( + fixture.store.writes().total(), + 0, + "should leave every removal to the tombstone" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should keep nothing to retry" + ); + } + + #[test] + fn capture_state_change_is_ignored() { + let mut seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + seeded.ps.insert( + "statecap".to_owned(), + KvProviderState { + st: Some(json!("old-state")), + ..KvProviderState::default() + }, + ); + install_capturing_logger(); + let provider = answering_capture_provider("statecap", &[FIRST], None); + let mut fixture = CaptureFixture::new(vec![provider], Some(&seeded)); + + fixture.capture( + &capture_path("statecap"), + &json!({"invalidate": [FIRST], "state": "new-state"}), + ); + assert!( + captured_logs() + .iter() + .any(|line| line.contains("'statecap'") + && line.contains("state change from capture")), + "should warn about the ignored state change" + ); + + let row = fixture.row(); + assert_eq!( + row.ps["statecap"].st, + Some(json!("old-state")), + "should keep the stored state" + ); + assert!( + !row.ids.contains_key(FIRST), + "control: should still apply the removal" + ); + } + } +} diff --git a/crates/trusted-server-core/src/ec/identity/denials.rs b/crates/trusted-server-core/src/ec/identity/denials.rs new file mode 100644 index 0000000000..1c4e10dc6c --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/denials.rs @@ -0,0 +1,311 @@ +//! Provider denials of browser EID candidates, read from KV (spec 7.1, +//! "Provider denials apply to browser candidates"). + +use std::collections::{BTreeMap, BTreeSet}; + +use super::catalog::ProviderCatalog; +use super::ids::SourceDomain; +use super::request::opted_out; +use super::revision::uid_hash; +use crate::ec::kv_types::KvEntry; +use crate::openrtb::Eid; + +/// What the providers that claim a source deny of the browser's candidates. +/// +/// Empty without a KV read. +#[derive(Debug, Default)] +pub(crate) struct Denials { + /// Per source, the `ps.iv` and `ps.ov` hashes of every claiming provider. + hashes: BTreeMap>, + /// Sources whose browser value is dropped whole: those of an opted-out + /// owner, and every source of an opted-out capture-only provider. + dropped: BTreeSet, +} + +impl Denials { + fn is_empty(&self) -> bool { + self.hashes.is_empty() && self.dropped.is_empty() + } +} + +/// The denials `entry` holds for the providers in `catalog` (spec 7.1). +#[must_use] +pub(crate) fn denials(entry: &KvEntry, catalog: &ProviderCatalog) -> Denials { + let mut result = Denials::default(); + let sources: BTreeSet<&SourceDomain> = catalog + .providers() + .flat_map(|provider| catalog.sources_of(provider.descriptor().id())) + .collect(); + for source in sources { + let hashes: BTreeSet = catalog + .claimants(source) + .filter_map(|provider| entry.ps.get(provider.as_str())) + .flat_map(|state| state.iv.iter().chain(&state.ov).cloned()) + .collect(); + if !hashes.is_empty() { + result.hashes.insert(source.clone(), hashes); + } + } + for provider in opted_out(entry) { + let capture_only = catalog.is_capture_only(&provider); + for source in catalog.sources_of(&provider) { + if capture_only || catalog.owner_of(source) == Some(&provider) { + result.dropped.insert(source.clone()); + } + } + } + result +} + +/// The browser's candidates after `denials` (spec 7.1). +/// +/// Drops each entry of a dropped-whole source; otherwise drops each UID whose +/// hash a claiming provider holds, then each entry left with no UID. Every +/// other entry goes through untouched, in order. +#[must_use] +pub(crate) fn filter_browser_candidates(client: &[Eid], denials: &Denials) -> Vec { + if denials.is_empty() { + return client.to_vec(); + } + let mut kept = Vec::with_capacity(client.len()); + for eid in client { + let Ok(source) = eid.source.parse::() else { + kept.push(eid.clone()); + continue; + }; + if denials.dropped.contains(&source) { + continue; + } + let Some(hashes) = denials.hashes.get(&source) else { + kept.push(eid.clone()); + continue; + }; + let mut filtered = eid.clone(); + filtered + .uids + .retain(|uid| !hashes.contains(&uid_hash(&uid.id))); + if !filtered.uids.is_empty() { + kept.push(filtered); + } + } + kept +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use super::*; + use crate::ec::identity::test_support::{fixed_capture_provider, fixed_provider, pid, source}; + use crate::ec::kv_types::{KvOptOut, KvProviderState}; + use crate::openrtb::Uid; + + fn uid(id: &str) -> Uid { + Uid { + id: id.to_owned(), + atype: Some(1), + ext: None, + } + } + + fn browser(source: &str, ids: &[&str]) -> Eid { + Eid { + source: source.to_owned(), + uids: ids.iter().map(|id| uid(id)).collect(), + inserter: Some("inserter.example.com".to_owned()), + matcher: None, + mm: Some(1), + ext: None, + } + } + + fn entry_with(states: &[(&str, KvProviderState)]) -> KvEntry { + let mut entry = KvEntry::minimal("unused.example.com", "unused", 1); + for (provider, state) in states { + entry.ps.insert((*provider).to_owned(), state.clone()); + } + entry + } + + fn invalidated(uids: &[&str]) -> KvProviderState { + KvProviderState { + iv: uids.iter().map(|uid| uid_hash(uid)).collect(), + ..KvProviderState::default() + } + } + + fn opted_out_state() -> KvProviderState { + KvProviderState { + o: Some(KvOptOut { + at: 10, + fp: "0".to_owned(), + }), + ..KvProviderState::default() + } + } + + fn unowned_catalog() -> ProviderCatalog { + ProviderCatalog::new( + vec![ + fixed_provider("pa", &["a.example.com"]), + fixed_provider("pb", &["b.example.com"]), + ], + BTreeMap::new(), + ) + } + + #[test] + fn denial_drops_a_browser_uid_in_a_claimants_iv_for_an_unowned_source() { + let entry = entry_with(&[("pa", invalidated(&["a-old"]))]); + let denials = denials(&entry, &unowned_catalog()); + + let kept = filter_browser_candidates( + &[ + browser("a.example.com", &["a-old"]), + browser("b.example.com", &["b1"]), + ], + &denials, + ); + + assert_eq!( + kept, + vec![browser("b.example.com", &["b1"])], + "should drop the invalidated UID and its emptied entry" + ); + } + + #[test] + fn denial_drops_a_browser_uid_in_a_claimants_ov() { + let state = KvProviderState { + ov: vec![uid_hash("a-removed")], + ..KvProviderState::default() + }; + let entry = entry_with(&[("pa", state)]); + let denials = denials(&entry, &unowned_catalog()); + + let kept = filter_browser_candidates(&[browser("a.example.com", &["a-removed"])], &denials); + + assert_eq!( + kept, + Vec::::new(), + "should drop a UID the opt-out removed" + ); + } + + #[test] + fn denial_filters_uid_by_uid_and_drops_an_emptied_entry() { + let entry = entry_with(&[("pa", invalidated(&["A"]))]); + let denials = denials(&entry, &unowned_catalog()); + + let kept = filter_browser_candidates( + &[ + browser("a.example.com", &["A", "B"]), + browser("a.example.com", &["A"]), + ], + &denials, + ); + + assert_eq!( + kept, + vec![Eid { + source: "a.example.com".to_owned(), + uids: vec![uid("B")], + inserter: Some("inserter.example.com".to_owned()), + matcher: None, + mm: Some(1), + ext: None, + }], + "should keep B alone and drop the entry holding only A" + ); + } + + #[test] + fn denial_ignores_hashes_of_a_provider_that_does_not_claim_the_source() { + let entry = entry_with(&[("pb", invalidated(&["shared"]))]); + let denials = denials(&entry, &unowned_catalog()); + + let kept = filter_browser_candidates(&[browser("a.example.com", &["shared"])], &denials); + + assert_eq!( + kept, + vec![browser("a.example.com", &["shared"])], + "should not apply another source's hashes" + ); + } + + #[test] + fn denial_drops_every_owned_source_while_the_owner_is_opted_out() { + let catalog = ProviderCatalog::new( + vec![fixed_provider("owner", &["a.example.com", "b.example.com"])], + BTreeMap::from([(source("a.example.com"), pid("owner"))]), + ); + let entry = entry_with(&[("owner", opted_out_state())]); + let denials = denials(&entry, &catalog); + + let kept = filter_browser_candidates( + &[ + browser("a.example.com", &["a1"]), + browser("b.example.com", &["b1"]), + ], + &denials, + ); + + assert_eq!( + kept, + vec![browser("b.example.com", &["b1"])], + "should drop the owned source and keep the unowned one" + ); + } + + #[test] + fn denial_drops_every_source_of_an_opted_out_capture_only_provider() { + let catalog = ProviderCatalog::new( + vec![ + fixed_capture_provider("cap", &["a.example.com", "b.example.com"]), + fixed_provider("other", &["c.example.com"]), + ], + BTreeMap::from([(source("a.example.com"), pid("cap"))]), + ); + let client = [ + browser("a.example.com", &["a1"]), + browser("b.example.com", &["b1"]), + browser("c.example.com", &["c1"]), + ]; + let entry = entry_with(&[("cap", opted_out_state())]); + + let kept = filter_browser_candidates(&client, &denials(&entry, &catalog)); + + assert_eq!( + kept, + vec![browser("c.example.com", &["c1"])], + "should drop owned and unowned sources of the capture-only provider" + ); + + let cleared = KvProviderState { + oc: Some(20), + ..opted_out_state() + }; + let entry = entry_with(&[("cap", cleared)]); + let kept = filter_browser_candidates(&client, &denials(&entry, &catalog)); + assert_eq!( + kept, + client.to_vec(), + "control: after a later oc the browser values go out again" + ); + } + + #[test] + fn denial_mirror_id_fed_back_through_the_body_is_filtered() { + let entry = entry_with(&[("pa", invalidated(&["mirror-only"]))]); + let denials = denials(&entry, &unowned_catalog()); + + let kept = + filter_browser_candidates(&[browser("a.example.com", &["mirror-only"])], &denials); + + assert_eq!( + kept, + Vec::::new(), + "a stale mirror ID fed back as a browser candidate should not be sent" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/descriptor.rs b/crates/trusted-server-core/src/ec/identity/descriptor.rs new file mode 100644 index 0000000000..908ea9df39 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/descriptor.rs @@ -0,0 +1,1176 @@ +//! What a provider is, may write, needs and must respect. + +use core::time::Duration; +use std::collections::{BTreeMap, BTreeSet}; + +use error_stack::Report; +use http::Method; +use url::Url; + +use super::ids::{IdentityProviderId, SourceDomain}; +use super::resolution::IdentityProviderError; +use crate::ec::kv_types::{MAX_EID_FIELD_BYTES, is_valid_eid_field}; +use crate::ec::partner::normalize_partner_source_domain; + +/// What a provider is, may write, needs and must respect. +/// +/// Built from config with no secrets, so `ts config validate` and the running +/// coordinator read the same value. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ProviderDescriptor { + id: IdentityProviderId, + sources: Vec, + resolves: Option, + endpoint: Option, + captures: Option, + consent_policy: ConsentPolicy, + provenance: BTreeMap, + expiring: BTreeSet, +} + +impl ProviderDescriptor { + /// Starts a descriptor for `id`. + #[must_use] + pub fn builder(id: IdentityProviderId) -> ProviderDescriptorBuilder { + ProviderDescriptorBuilder { + id, + sources: Vec::new(), + resolves: None, + captures: None, + consent_policy: ConsentPolicy::strict(), + provenance: BTreeMap::new(), + expiring_all: false, + expiring: BTreeSet::new(), + } + } + + /// The provider's ID. + #[must_use] + pub fn id(&self) -> &IdentityProviderId { + &self.id + } + + /// The sources this provider may write, in declaration order. + #[must_use] + pub fn sources(&self) -> &[SourceDomain] { + &self.sources + } + + /// The first source added. Every built descriptor has one. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + &self.sources[0] + } + + /// Finds a declared source by a vendor's spelling of it, normalized. + #[must_use] + pub fn source_named(&self, name: &str) -> Option<&SourceDomain> { + self.sources.iter().find(|source| source.matches(name)) + } + + /// Active resolution, when declared. + #[must_use] + pub fn resolves(&self) -> Option<&Resolves> { + self.resolves.as_ref() + } + + /// The endpoint declared with [`Resolves::with_endpoint`], if any. + #[must_use] + pub fn endpoint(&self) -> Option<&Url> { + self.endpoint.as_ref() + } + + /// Passive capture, when declared. + #[must_use] + pub fn captures(&self) -> Option<&Captures> { + self.captures.as_ref() + } + + /// The consent this provider requires. + #[must_use] + pub fn consent_policy(&self) -> &ConsentPolicy { + &self.consent_policy + } + + /// Default EID-level fields for `source`, if declared. + #[must_use] + pub fn provenance(&self, source: &SourceDomain) -> Option<&EidProvenance> { + self.provenance.get(source) + } + + /// Whether IDs of `source` are declared as expiring. + #[must_use] + pub fn is_expiring(&self, source: &SourceDomain) -> bool { + self.expiring.contains(source) + } +} + +/// Builds a [`ProviderDescriptor`]. +#[derive(Debug, Clone)] +pub struct ProviderDescriptorBuilder { + id: IdentityProviderId, + sources: Vec, + resolves: Option, + captures: Option, + consent_policy: ConsentPolicy, + provenance: BTreeMap, + expiring_all: bool, + expiring: BTreeSet, +} + +impl ProviderDescriptorBuilder { + /// Adds a source this provider may write. Repeatable. A repeat is ignored. + #[must_use] + pub fn with_source(mut self, source: SourceDomain) -> Self { + if !self.sources.contains(&source) { + self.sources.push(source); + } + self + } + + /// Declares active resolution. + #[must_use] + pub fn with_resolves(mut self, resolves: Resolves) -> Self { + self.resolves = Some(resolves); + self + } + + /// Declares passive capture. + #[must_use] + pub fn with_captures(mut self, captures: Captures) -> Self { + self.captures = Some(captures); + self + } + + /// Replaces the default policy, [`ConsentPolicy::strict`]. + #[must_use] + pub fn with_consent_policy(mut self, policy: ConsentPolicy) -> Self { + self.consent_policy = policy; + self + } + + /// Marks the IDs of every declared source as expiring (spec 6.1). + #[must_use] + pub fn with_expiring_ids(mut self) -> Self { + self.expiring_all = true; + self + } + + /// Marks the IDs of one declared source as expiring. + #[must_use] + pub fn with_expiring_source(mut self, source: &SourceDomain) -> Self { + self.expiring.insert(source.clone()); + self + } + + /// Default EID-level fields for a declared source. At output, the + /// coordinator applies them to a stored entry of that source, written by + /// this provider, only when the entry sets none of `inserter`, `matcher` + /// and `mm`. An entry that sets any of them is the vendor's complete + /// statement, so vendor and config provenance never mix in one entry. + /// The defaults are not stored, so a config change applies at once. + #[must_use] + pub fn with_provenance(mut self, source: &SourceDomain, provenance: EidProvenance) -> Self { + self.provenance.insert(source.clone(), provenance); + self + } + + /// Validates and builds the descriptor. + /// + /// # Errors + /// + /// Returns `Config` when the descriptor has no source, no capability, a + /// resolver with no endpoint or host, an endpoint that is not an `https` + /// URL, a host that is not a bare hostname, provenance or an expiring + /// mark for a source it does not declare, a provenance `inserter` or + /// `matcher` that is blank or over 255 bytes, or a provenance `mm` below 0. + pub fn build(self) -> Result> { + let id = self.id; + let fail = |what: &str| IdentityProviderError::config(format!("provider `{id}`: {what}")); + + if self.sources.is_empty() { + return Err(fail("declares no source")); + } + if self.resolves.is_none() && self.captures.is_none() { + return Err(fail("declares neither resolves nor captures")); + } + for source in self.provenance.keys().chain(self.expiring.iter()) { + if !self.sources.contains(source) { + return Err(fail(&format!("names undeclared source `{source}`"))); + } + } + for (source, provenance) in &self.provenance { + let fields = [ + ("inserter", provenance.inserter()), + ("matcher", provenance.matcher()), + ]; + for (name, value) in fields { + if let Some(value) = value + && !is_valid_eid_field(value) + { + return Err(fail(&format!( + "provenance `{name}` for `{source}` is blank or over {MAX_EID_FIELD_BYTES} bytes" + ))); + } + } + if provenance.mm().is_some_and(|mm| mm < 0) { + return Err(fail(&format!("provenance `mm` for `{source}` is below 0"))); + } + } + + let mut endpoint = None; + let mut resolves = self.resolves; + if let Some(resolves) = resolves.as_mut() { + if let Some(raw) = resolves.raw_endpoint.as_deref() { + let url = Url::parse(raw).map_err(|_| fail("endpoint is not a URL"))?; + if url.scheme() != "https" { + return Err(fail("endpoint must use https")); + } + let host = url + .host_str() + .ok_or_else(|| fail("endpoint has no host"))? + .to_owned(); + if !resolves.hosts.contains(&host) { + resolves.hosts.insert(0, host); + } + endpoint = Some(url); + } + if resolves.hosts.is_empty() { + return Err(fail("resolves declares no endpoint or host")); + } + if let Some(grace) = resolves.refresh_grace { + if !resolves.kinds.contains(&ResolveKind::Refresh) { + return Err(fail("declares a refresh grace but not the refresh kind")); + } + if grace.is_zero() { + return Err(fail("declares a zero refresh grace")); + } + } + for host in &resolves.hosts { + if normalize_partner_source_domain(host).ok().as_deref() != Some(host.as_str()) { + return Err(fail(&format!( + "host `{host}` is not a bare lowercase hostname" + ))); + } + } + if let Some(gap) = self.consent_policy.resolver_gap() { + return Err(fail(gap)); + } + } + + let expiring = if self.expiring_all { + self.sources.iter().cloned().collect() + } else { + self.expiring + }; + + Ok(ProviderDescriptor { + id, + sources: self.sources, + resolves, + endpoint, + captures: self.captures, + consent_policy: self.consent_policy, + provenance: self.provenance, + expiring, + }) + } +} + +/// The kind of resolution one source needs. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum ResolveKind { + /// No stored record for the source. + #[display("initial")] + Initial, + /// A stored record exists, valid or expired, and is due. + #[display("refresh")] + Refresh, +} + +/// Active resolution: the provider calls an endpoint. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Resolves { + raw_endpoint: Option, + hosts: Vec, + needs: BTreeSet, + needs_for: BTreeMap>, + wants: BTreeSet, + kinds: Vec, + private_state: bool, + refresh_grace: Option, + ec_id: bool, +} + +impl Default for Resolves { + fn default() -> Self { + Self::new() + } +} + +impl Resolves { + /// A resolver with no inputs, handling `Initial` and `Refresh`. + #[must_use] + pub fn new() -> Self { + Self { + raw_endpoint: None, + hosts: Vec::new(), + needs: BTreeSet::new(), + needs_for: BTreeMap::new(), + wants: BTreeSet::new(), + kinds: vec![ResolveKind::Initial, ResolveKind::Refresh], + private_state: false, + refresh_grace: None, + ec_id: false, + } + } + + /// The endpoint the provider calls, as config spells it. `build()` parses + /// it and requires `https`. Its host becomes an allowed host. + #[must_use] + pub fn with_endpoint(mut self, endpoint: &str) -> Self { + self.raw_endpoint = Some(endpoint.to_owned()); + self + } + + /// Adds another host the provider may call. Repeatable. + #[must_use] + pub fn with_host(mut self, host: &str) -> Self { + if !self.hosts.iter().any(|known| known == host) { + self.hosts.push(host.to_owned()); + } + self + } + + /// Inputs required for every kind. + #[must_use] + pub fn with_needs(mut self, inputs: impl IntoIterator) -> Self { + self.needs.extend(inputs); + self + } + + /// Inputs required for one kind only. + #[must_use] + pub fn with_needs_for( + mut self, + kind: ResolveKind, + inputs: impl IntoIterator, + ) -> Self { + self.needs_for.entry(kind).or_default().extend(inputs); + self + } + + /// Inputs passed when available, never required. + #[must_use] + pub fn with_wants(mut self, inputs: impl IntoIterator) -> Self { + self.wants.extend(inputs); + self + } + + /// The kinds this provider handles. Default: `Initial` and `Refresh`. + #[must_use] + pub fn with_kinds(mut self, kinds: impl IntoIterator) -> Self { + self.kinds = kinds.into_iter().collect(); + self + } + + /// This provider keeps private state in KV (`ps.st`, spec 6.1). It can + /// then read the stored state and return a state change. + #[must_use] + pub fn with_private_state(mut self) -> Self { + self.private_state = true; + self + } + + /// Keeps an expired record this provider wrote for a source it owns as + /// a refresh anchor until `exp` plus `grace` (spec 5.4, step 4). The + /// record is never served, but the next eligible request still selects + /// `Refresh` for it rather than `Initial`. For providers whose refresh + /// call accepts an expired ID and that keep no private state, so cannot + /// set `refresh_until`. `build()` requires `Refresh` among the kinds and a + /// grace above zero. + #[must_use] + pub fn with_refresh_grace(mut self, grace: Duration) -> Self { + self.refresh_grace = Some(grace); + self + } + + /// Gives the provider the full EC ID. Only generic pull may use this. + #[must_use] + pub(in crate::ec::identity) fn with_ec_id(mut self) -> Self { + self.ec_id = true; + self + } + + /// Hosts the provider may call. + #[must_use] + pub(crate) fn hosts(&self) -> &[String] { + &self.hosts + } + + /// Inputs required for `kind`: the common needs plus that kind's. + #[must_use] + pub(crate) fn needs(&self, kind: ResolveKind) -> BTreeSet { + let mut needs = self.needs.clone(); + if let Some(extra) = self.needs_for.get(&kind) { + needs.extend(extra.iter().copied()); + } + needs + } + + /// Every input declared for any kind, needed or wanted. + #[must_use] + pub(crate) fn declared_inputs(&self) -> BTreeSet { + let mut all = self.needs.clone(); + all.extend(self.wants.iter().copied()); + for extra in self.needs_for.values() { + all.extend(extra.iter().copied()); + } + all + } + + /// Inputs passed when available. + #[must_use] + pub(crate) fn wants(&self) -> &BTreeSet { + &self.wants + } + + /// The kinds this provider handles. + #[must_use] + pub(crate) fn kinds(&self) -> &[ResolveKind] { + &self.kinds + } + + /// How long an expired record this provider wrote stays a refresh + /// anchor, if it declared a grace. + #[must_use] + pub(crate) fn refresh_grace(&self) -> Option { + self.refresh_grace + } + + /// Whether the provider keeps private state. + #[must_use] + pub(crate) fn has_private_state(&self) -> bool { + self.private_state + } + + /// Whether the provider receives the full EC ID. + #[must_use] + pub(crate) fn gets_ec_id(&self) -> bool { + self.ec_id + } +} + +/// Revoke-only capture from responses TS proxies for this provider: it +/// can remove IDs, opt out or clear an opt-out, and never stores a positive +/// ID (spec 4.7). +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Captures { + paths: Vec<(Method, String)>, + prefixes: Vec, +} + +impl Captures { + /// No capture rules yet. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Offers proxied responses to exactly this method and path. Repeatable. + #[must_use] + pub fn with_path(mut self, method: Method, path: &str) -> Self { + self.paths.push((method, path.to_owned())); + self + } + + /// Offers proxied responses whose path starts with `prefix`, for any + /// method. Repeatable. + #[must_use] + pub fn with_path_prefix(mut self, prefix: &str) -> Self { + self.prefixes.push(prefix.to_owned()); + self + } + + /// Whether a proxied request matches a rule. + #[must_use] + pub(crate) fn matches(&self, method: &Method, path: &str) -> bool { + self.paths + .iter() + .any(|(rule_method, rule_path)| rule_method == method && rule_path == path) + || self + .prefixes + .iter() + .any(|prefix| path.starts_with(prefix.as_str())) + } + + /// Exact path rules. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn paths(&self) -> &[(Method, String)] { + &self.paths + } + + /// Prefix rules. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn prefixes(&self) -> &[String] { + &self.prefixes + } +} + +/// Default EID-level fields (`OpenRTB` 2.6 §3.2.27) that bidders read, such +/// as who inserted an ID and how it was matched. They apply only to an entry +/// that sets none of the three. A provider whose vendor returns these per +/// response sets them on each `ResolvedEid` instead (spec 4.5). +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct EidProvenance { + inserter: Option, + matcher: Option, + mm: Option, +} + +impl EidProvenance { + /// No fields set. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Sets `inserter`. + #[must_use] + pub fn with_inserter(mut self, inserter: &str) -> Self { + self.inserter = Some(inserter.to_owned()); + self + } + + /// Sets `matcher`. + #[must_use] + pub fn with_matcher(mut self, matcher: &str) -> Self { + self.matcher = Some(matcher.to_owned()); + self + } + + /// Sets `mm`. + #[must_use] + pub fn with_mm(mut self, mm: i32) -> Self { + self.mm = Some(mm); + self + } + + /// `inserter`, if set. + #[must_use] + pub fn inserter(&self) -> Option<&str> { + self.inserter.as_deref() + } + + /// `matcher`, if set. + #[must_use] + pub fn matcher(&self) -> Option<&str> { + self.matcher.as_deref() + } + + /// `mm`, if set. + #[must_use] + pub fn mm(&self) -> Option { + self.mm + } +} + +/// A US opt-out a policy can relax. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum UsOptOut { + /// Sale of personal information. + #[display("sale")] + Sale, + /// Sharing for cross-context behavioural advertising. + #[display("sharing")] + Sharing, + /// Targeted advertising. + #[display("targeted_advertising")] + TargetedAdvertising, +} + +/// The consent a provider requires. Evaluated by the coordinator (Task 12). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ConsentPolicy { + tcf_purposes: BTreeSet, + tcf_vendor: Option, + allow_dnt: bool, + allow_gpc: bool, + allowed_us_opt_outs: BTreeSet, + eid_persistence_only: bool, + purposes_declared: bool, + without_vendor: bool, +} + +impl ConsentPolicy { + /// The default. Blocked when any consent signal present cannot be + /// decoded. Where TCF applies, requires Purpose 1 and whatever the + /// provider declares with [`with_tcf_purposes`](Self::with_tcf_purposes) + /// and [`with_tcf_vendor`](Self::with_tcf_vendor). Blocked by every US + /// opt-out TS decodes, by `Sec-GPC: 1` and by `DNT: 1`. + /// + /// A resolver's policy must declare its purposes and its vendor, or say + /// it has none with [`without_tcf_vendor`](Self::without_tcf_vendor); + /// [`ProviderDescriptorBuilder::build`] rejects it otherwise. + #[must_use] + pub fn strict() -> Self { + Self { + tcf_purposes: [1].into_iter().collect(), + tcf_vendor: None, + allow_dnt: false, + allow_gpc: false, + allowed_us_opt_outs: BTreeSet::new(), + eid_persistence_only: false, + purposes_declared: false, + without_vendor: false, + } + } + + /// Declares the TCF purposes the provider processes under consent, as + /// its GVL entry lists them. Each is required where TCF applies, beside + /// Purpose 1, which is always required. An empty list declares Purpose 1 + /// alone. + #[must_use] + pub fn with_tcf_purposes(mut self, purposes: impl IntoIterator) -> Self { + self.tcf_purposes.extend(purposes); + self.purposes_declared = true; + self + } + + /// Declares that the provider has no IAB Global Vendor List ID, so no + /// vendor consent is checked: purpose consent alone lets it run where + /// TCF applies. Needs a documented legal basis (TCF Policies, Chapter + /// III, Policy 14(3)). + #[must_use] + pub fn without_tcf_vendor(mut self) -> Self { + self.without_vendor = true; + self + } + + /// Requires vendor consent for this IAB Global Vendor List ID. + #[must_use] + pub fn with_tcf_vendor(mut self, vendor_id: u16) -> Self { + self.tcf_vendor = Some(vendor_id); + self + } + + /// Does not block on `DNT: 1`. Needs a documented legal basis. + #[must_use] + pub fn allowing_dnt(mut self) -> Self { + self.allow_dnt = true; + self + } + + /// Does not block on GPC. Needs a documented legal basis. + #[must_use] + pub fn allowing_gpc(mut self) -> Self { + self.allow_gpc = true; + self + } + + /// Does not block on one US opt-out. Needs a documented legal basis. + #[must_use] + pub fn allowing_us_opt_out(mut self, opt_out: UsOptOut) -> Self { + self.allowed_us_opt_outs.insert(opt_out); + self + } + + /// Blocks exactly what pull sync blocks today: only the global EC and + /// EID-persistence gates. For generic pull's parity (spec 10.4). + // Temporary: first used by a later identity task. + #[allow(dead_code)] + #[must_use] + pub(in crate::ec::identity) fn eid_persistence_only() -> Self { + Self { + eid_persistence_only: true, + purposes_declared: true, + without_vendor: true, + ..Self::strict() + } + } + + /// Required TCF purposes. + #[must_use] + pub fn tcf_purposes(&self) -> &BTreeSet { + &self.tcf_purposes + } + + /// Required GVL vendor, if any. + #[must_use] + pub fn tcf_vendor(&self) -> Option { + self.tcf_vendor + } + + /// Whether `DNT: 1` is allowed. + #[must_use] + pub fn allows_dnt(&self) -> bool { + self.allow_dnt + } + + /// Whether GPC is allowed. + #[must_use] + pub fn allows_gpc(&self) -> bool { + self.allow_gpc + } + + /// Whether a US opt-out is allowed. + #[must_use] + pub fn allows_us_opt_out(&self, opt_out: UsOptOut) -> bool { + self.allowed_us_opt_outs.contains(&opt_out) + } + + /// Whether this is generic pull's parity policy. + #[must_use] + pub(crate) fn is_eid_persistence_only(&self) -> bool { + self.eid_persistence_only + } + + /// Why a resolver may not use this policy, if it may not. + fn resolver_gap(&self) -> Option<&'static str> { + if !self.purposes_declared { + Some("consent policy declares no TCF purposes; call with_tcf_purposes") + } else if self.tcf_vendor.is_none() && !self.without_vendor { + Some( + "consent policy names no TCF vendor; call with_tcf_vendor, or without_tcf_vendor with a documented legal basis", + ) + } else { + None + } + } +} + +/// An input a provider can ask for. +/// +/// Consent strings, the provider's stored IDs, the publisher domain, the +/// time and the declared endpoint are always available. Private state is +/// available to providers declared `with_private_state()`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum Input { + /// The platform's client address. + #[display("client_ip")] + ClientIp, + /// The request's `User-Agent`. + #[display("user_agent")] + UserAgent, + /// The top-level page URL without query string or fragment. + #[display("page_url")] + PageUrl, + /// Country and subdivision only. + #[display("region")] + Region, + /// A SHA-256 of a normalized email address. Never available in v1. + #[display("hashed_email")] + HashedEmail, +} + +/// Something a provider read without declaring it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum Access { + /// An undeclared [`Input`]. + #[display("{_0}")] + Input(Input), + /// The endpoint, when the descriptor declares no `resolves`. + #[display("endpoint")] + Endpoint, + /// Private state, without `with_private_state()`. + #[display("private_state")] + PrivateState, + /// The full EC ID, without `with_ec_id()`. Only generic pull declares it. + #[display("ec_id")] + EcId, +} + +#[cfg(test)] +mod tests { + use http::Method; + + use super::*; + use crate::ec::kv_types::MAX_EID_FIELD_BYTES; + + fn source(name: &str) -> SourceDomain { + name.parse().expect("should parse source") + } + + fn id(name: &str) -> IdentityProviderId { + name.parse().expect("should parse id") + } + + #[test] + fn builds_the_documented_example() { + let descriptor = ProviderDescriptor::builder(id("exampleid")) + .with_source(source("exampleid.example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.exampleid.example.com/v1/fetch") + .with_needs([Input::ClientIp, Input::UserAgent]) + .with_wants([Input::PageUrl]), + ) + .with_consent_policy( + ConsentPolicy::strict() + .with_tcf_purposes([4]) + .with_tcf_vendor(9999), + ) + .build() + .expect("should build"); + + assert_eq!(descriptor.id().as_str(), "exampleid", "should keep the id"); + assert_eq!( + descriptor.primary_source(), + &source("exampleid.example.com"), + "should keep the source" + ); + assert_eq!( + descriptor.endpoint().map(Url::as_str), + Some("https://api.exampleid.example.com/v1/fetch"), + "should parse the endpoint" + ); + let resolves = descriptor.resolves().expect("should resolve"); + assert_eq!( + resolves.hosts(), + &["api.exampleid.example.com".to_owned()], + "should allow the endpoint host" + ); + assert_eq!( + resolves.kinds(), + &[ResolveKind::Initial, ResolveKind::Refresh], + "should default to Initial and Refresh" + ); + assert!(descriptor.captures().is_none(), "should not capture"); + assert!( + !descriptor.is_expiring(&source("exampleid.example.com")), + "should not expire by default" + ); + assert_eq!( + descriptor.consent_policy().tcf_vendor(), + Some(9999), + "should keep the vendor" + ); + assert_eq!( + descriptor + .consent_policy() + .tcf_purposes() + .iter() + .copied() + .collect::>(), + vec![1, 4], + "should require Purpose 1 and the declared Purpose 4" + ); + } + + #[test] + fn strict_requires_only_purpose_one_until_purposes_are_declared() { + let policy = ConsentPolicy::strict(); + + assert_eq!( + policy.tcf_purposes().iter().copied().collect::>(), + vec![1], + "should require Purpose 1 alone" + ); + assert_eq!( + ConsentPolicy::strict() + .with_tcf_purposes([3, 5]) + .tcf_purposes() + .iter() + .copied() + .collect::>(), + vec![1, 3, 5], + "should add the declared purposes to Purpose 1" + ); + } + + #[test] + fn a_resolver_must_declare_its_purposes_and_vendor() { + let resolver = |policy: ConsentPolicy| { + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new().with_endpoint("https://a.example.com")) + .with_consent_policy(policy) + .build() + }; + + let undeclared = resolver(ConsentPolicy::strict().with_tcf_vendor(5)) + .expect_err("should reject a resolver that declares no purposes"); + let no_vendor = resolver(ConsentPolicy::strict().with_tcf_purposes([4])) + .expect_err("should reject a resolver that names no vendor"); + let relaxed = resolver( + ConsentPolicy::strict() + .with_tcf_purposes([]) + .without_tcf_vendor(), + ) + .expect("should build a resolver that opts out of vendor consent"); + let capture = ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .build(); + + assert!( + format!("{undeclared:?}").contains("with_tcf_purposes"), + "should name the missing purposes" + ); + assert!( + format!("{no_vendor:?}").contains("without_tcf_vendor"), + "should name the vendor relaxation" + ); + assert_eq!( + relaxed.consent_policy().tcf_vendor(), + None, + "should check no vendor" + ); + assert!( + capture.is_ok(), + "should not require a policy of a capture-only provider" + ); + } + + #[test] + fn build_rejects_invalid_descriptors() { + let cases: Vec<(&str, ProviderDescriptorBuilder)> = vec![ + ( + "no source", + ProviderDescriptor::builder(id("p")) + .with_resolves(Resolves::new().with_endpoint("https://a.example.com")), + ), + ( + "no capability", + ProviderDescriptor::builder(id("p")).with_source(source("example.com")), + ), + ( + "no endpoint or host", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new()), + ), + ( + "http endpoint", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new().with_endpoint("http://a.example.com")), + ), + ( + "bad host", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new().with_host("a.example.com/path")), + ), + ( + "provenance for undeclared source", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_provenance( + &source("other.example.com"), + EidProvenance::new().with_mm(2), + ), + ), + ( + "refresh grace without refresh", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_kinds([ResolveKind::Initial]) + .with_refresh_grace(core::time::Duration::from_secs(60)), + ), + ), + ( + "zero refresh grace", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_refresh_grace(core::time::Duration::ZERO), + ), + ), + ( + "expiring undeclared source", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_expiring_source(&source("other.example.com")), + ), + ( + "blank provenance inserter", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_provenance( + &source("example.com"), + EidProvenance::new().with_inserter(" "), + ), + ), + ( + "blank provenance matcher", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_provenance( + &source("example.com"), + EidProvenance::new().with_matcher(""), + ), + ), + ( + "oversized provenance matcher", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_provenance( + &source("example.com"), + EidProvenance::new().with_matcher(&"m".repeat(MAX_EID_FIELD_BYTES + 1)), + ), + ), + ( + "negative provenance mm", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_provenance(&source("example.com"), EidProvenance::new().with_mm(-1)), + ), + ]; + for (name, builder) in cases { + let error = builder + .build() + .expect_err(&format!("should reject a descriptor with {name}")); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ), + "should be a Config error: {name}" + ); + } + } + + #[test] + fn expiring_and_provenance_are_per_source() { + let a = source("a.example.com"); + let b = source("b.example.com"); + let descriptor = ProviderDescriptor::builder(id("p")) + .with_source(a.clone()) + .with_source(b.clone()) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_expiring_source(&b) + .with_provenance( + &a, + EidProvenance::new() + .with_inserter("example-inserter") + .with_mm(2), + ) + .build() + .expect("should build"); + + assert!(!descriptor.is_expiring(&a), "should not mark a as expiring"); + assert!(descriptor.is_expiring(&b), "should mark b as expiring"); + let provenance = descriptor + .provenance(&a) + .expect("should have provenance for a"); + assert_eq!( + provenance.inserter(), + Some("example-inserter"), + "should keep the inserter" + ); + assert_eq!(provenance.matcher(), None, "should leave the matcher unset"); + assert_eq!(provenance.mm(), Some(2), "should keep mm"); + assert!( + descriptor.provenance(&b).is_none(), + "should have no provenance for b" + ); + assert_eq!( + descriptor.source_named("A.EXAMPLE.COM"), + Some(&a), + "should find a by any spelling" + ); + } + + #[test] + fn expiring_ids_marks_every_source() { + let descriptor = ProviderDescriptor::builder(id("p")) + .with_source(source("a.example.com")) + .with_source(source("b.example.com")) + .with_expiring_ids() + .with_captures(Captures::new().with_path_prefix("/integrations/p/")) + .build() + .expect("should build"); + assert!( + descriptor + .sources() + .iter() + .all(|s| descriptor.is_expiring(s)), + "should mark every source" + ); + } + + #[test] + fn captures_match_exact_paths_and_prefixes() { + let captures = Captures::new() + .with_path(Method::POST, "/integrations/p/api/token") + .with_path_prefix("/integrations/p/stream/"); + assert!( + captures.matches(&Method::POST, "/integrations/p/api/token"), + "should match the exact path" + ); + assert!( + !captures.matches(&Method::GET, "/integrations/p/api/token"), + "should require the method" + ); + assert!( + !captures.matches(&Method::POST, "/integrations/p/api/token/x"), + "should not treat an exact path as a prefix" + ); + assert!( + captures.matches(&Method::GET, "/integrations/p/stream/1"), + "should match a prefix for any method" + ); + assert!( + !captures.matches(&Method::GET, "/integrations/p/settings"), + "should not match others" + ); + } + + #[test] + fn needs_differ_by_kind() { + let resolves = Resolves::new() + .with_endpoint("https://a.example.com") + .with_needs([Input::ClientIp]) + .with_needs_for(ResolveKind::Initial, [Input::HashedEmail]); + assert_eq!( + resolves.needs(ResolveKind::Initial), + [Input::ClientIp, Input::HashedEmail].into_iter().collect(), + "should need both inputs for Initial" + ); + assert_eq!( + resolves.needs(ResolveKind::Refresh), + [Input::ClientIp].into_iter().collect(), + "should need only the IP for Refresh" + ); + } + + #[test] + fn valid_provenance_builds() { + let descriptor = ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_provenance( + &source("example.com"), + EidProvenance::new() + .with_inserter("example-inserter") + .with_matcher("example-matcher") + .with_mm(0), + ) + .build() + .expect("should accept non-blank provenance and mm 0"); + assert!( + descriptor.provenance(&source("example.com")).is_some(), + "should keep the provenance" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/examples.rs b/crates/trusted-server-core/src/ec/identity/examples.rs new file mode 100644 index 0000000000..24363b873b --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/examples.rs @@ -0,0 +1,1026 @@ +//! Fictional providers from spec 4.6, for tests of registration, coordination +//! and capture. + +use std::time::Duration; + +use error_stack::Report; +use http::Method; +use serde::{Deserialize, Serialize}; +use serde_json::json; +use url::Url; +use validator::Validate; + +use super::descriptor::{Captures, ConsentPolicy, Input, ProviderDescriptor, Resolves}; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::ids::UnixTime; +use super::input::{CaptureInput, ResolveInput, StoredId}; +use super::provider::{IdentityProvider, IdentityProviderConfig}; +use super::resolution::{IdentityProviderError, Resolution, ResolvedEid, ResolvedUid}; +use crate::settings::IntegrationConfig; + +/// Integration ID of [`ExampleIdProvider`]. +pub(crate) const EXAMPLEID_INTEGRATION_ID: &str = "exampleid"; + +/// Integration ID of [`ExampleCaptureProvider`]. +pub(crate) const EXAMPLECAPTURE_INTEGRATION_ID: &str = "examplecapture"; + +/// Config of the fictional `exampleid` integration. +#[derive(Debug, Clone, Deserialize, Validate)] +pub(crate) struct ExampleIdConfig { + pub enabled: bool, + #[validate(url)] + pub endpoint: String, + pub partner_id: u32, + pub gvl_id: u16, +} + +impl IntegrationConfig for ExampleIdConfig { + fn is_enabled(&self) -> bool { + self.enabled + } +} + +impl IdentityProviderConfig for ExampleIdConfig { + fn build_descriptor(&self) -> Result> { + ProviderDescriptor::builder(EXAMPLEID_INTEGRATION_ID.parse()?) + .with_source("exampleid.example.com".parse()?) + .with_resolves( + Resolves::new() + .with_endpoint(&self.endpoint) + .with_needs([Input::ClientIp, Input::UserAgent]) + .with_wants([Input::PageUrl]) + .with_private_state(), + ) + .with_consent_policy( + ConsentPolicy::strict() + .with_tcf_purposes([4]) + .with_tcf_vendor(self.gvl_id), + ) + .build() + } +} + +/// A valid [`ExampleIdConfig`]. +pub(crate) fn example_config() -> ExampleIdConfig { + ExampleIdConfig { + enabled: true, + endpoint: "https://api.example.com/fetch".to_owned(), + partner_id: 7, + gvl_id: 1, + } +} + +/// A resolving provider that round-trips private state. +pub(crate) struct ExampleIdProvider { + descriptor: ProviderDescriptor, + partner_id: u32, +} + +/// The private state [`ExampleIdProvider`] keeps between calls. +#[derive(Serialize, Deserialize)] +pub(crate) struct ExampleIdState { + signature: String, +} + +#[derive(Serialize)] +struct FetchBody<'a> { + partner: u32, + ip: String, + ua: &'a str, + page: Option<&'a str>, + gdpr: bool, + consent: Option<&'a str>, + signature: Option, +} + +#[derive(Deserialize)] +struct FetchReply { + uid: Option, + signature: Option, + link_type: Option, + max_age_sec: Option, +} + +impl ExampleIdProvider { + /// Builds the provider from its validated config. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Config`] when the config cannot + /// describe a valid provider. + pub(crate) fn new(config: &ExampleIdConfig) -> Result> { + Ok(Self { + descriptor: config.build_descriptor()?, + partner_id: config.partner_id, + }) + } +} + +impl IdentityProvider for ExampleIdProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + let consent = input.consent(); + let body = FetchBody { + partner: self.partner_id, + ip: input.client_ip()?.to_string(), + ua: input.user_agent()?, + page: input.page_url().ok().map(Url::as_str), + gdpr: consent.gdpr_applies(), + consent: consent.raw_tc_string(), + signature: input + .state::()? + .map(|state| state.signature), + }; + ProviderRequest::post_json(input.endpoint()?, &body).map(Some) + } + + fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, + ) -> Result> { + let reply: FetchReply = response.success_json()?; + let Some(uid) = reply.uid else { + return Ok(Resolution::no_match()); + }; + + let mut uid = ResolvedUid::new(uid).with_atype(1); + if let Some(link_type) = reply.link_type { + uid = uid.with_ext(json!({ "linkType": link_type })); + } + let mut eid = ResolvedEid::new(input.primary_source(), uid); + if let Some(max_age) = reply.max_age_sec { + eid = eid.with_refresh_in(Duration::from_secs(max_age)); + } + + let resolution = Resolution::found([eid]); + Ok(match reply.signature { + Some(signature) => resolution.replacing_state(&ExampleIdState { signature }), + None => resolution, + }) + } +} + +/// A capture-only provider that reads tokens from the API it proxies. +pub(crate) struct ExampleCaptureProvider { + descriptor: ProviderDescriptor, +} + +#[derive(Deserialize)] +struct TokenReply { + tokens: Vec, +} + +#[derive(Deserialize)] +struct Token { + source: String, + value: String, + expires: u64, +} + +impl ExampleCaptureProvider { + /// Builds the provider with the capture path of spec 4.6. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Config`] when the descriptor is invalid. + pub(crate) fn new() -> Result> { + Self::with_capture_path("/integrations/examplecapture/api/token") + } + + /// Builds the provider capturing from `path` instead. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Config`] when the descriptor is invalid. + pub(crate) fn with_capture_path(path: &str) -> Result> { + let descriptor = ProviderDescriptor::builder(EXAMPLECAPTURE_INTEGRATION_ID.parse()?) + .with_source("examplecapture.example.com".parse()?) + .with_captures(Captures::new().with_path(Method::POST, path)) + .build()?; + Ok(Self { descriptor }) + } +} + +impl IdentityProvider for ExampleCaptureProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn capture( + &self, + input: &CaptureInput<'_>, + ) -> Result> { + let Ok(reply) = input.response().json::() else { + return Ok(Resolution::no_match()); + }; + Ok(Resolution::found(reply.tokens.into_iter().filter_map( + |token| { + let source = input.source_named(&token.source)?; + Some( + ResolvedEid::new(source, ResolvedUid::new(token.value)) + .with_expires_at(UnixTime::from_secs(token.expires)), + ) + }, + ))) + } +} + +#[cfg(test)] +mod tests { + use http::header::COOKIE; + use http::{HeaderValue, StatusCode}; + + use super::*; + use crate::ec::identity::descriptor::{EidProvenance, ProviderDescriptorBuilder}; + use crate::ec::identity::ids::SourceDomain; + use crate::ec::identity::test_support::{ + CaptureFixture, ConformanceCases, ConformanceFailure, HarnessSetup, ProviderHarness, + TestConsent, check_descriptor_ignores_secrets, resolver_policy, + }; + use crate::openrtb::{Eid, Uid}; + + type RequestFn = + fn(&ResolveInput<'_>) -> Result, Report>; + type ResponseFn = fn( + &ProviderResponse, + &ResolveInput<'_>, + ) -> Result>; + type CaptureFn = fn(&CaptureInput<'_>) -> Result>; + + /// A resolver whose behaviour is two plain functions. + struct TestResolver { + descriptor: ProviderDescriptor, + request: RequestFn, + response: ResponseFn, + } + + impl IdentityProvider for TestResolver { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + (self.request)(input) + } + + fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, + ) -> Result> { + (self.response)(response, input) + } + } + + /// A capturer whose behaviour is one plain function. + struct TestCapturer { + descriptor: ProviderDescriptor, + capture: CaptureFn, + } + + impl IdentityProvider for TestCapturer { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn capture( + &self, + input: &CaptureInput<'_>, + ) -> Result> { + (self.capture)(input) + } + } + + /// A provider that declares a capability and overrides nothing. + struct DeclaresOnly(ProviderDescriptor); + + impl IdentityProvider for DeclaresOnly { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } + } + + const CAPTURE_PATH: &str = "/integrations/exampleid/api/token"; + + fn source() -> SourceDomain { + "exampleid.example.com" + .parse() + .expect("should parse the source") + } + + fn builder() -> ProviderDescriptorBuilder { + ProviderDescriptor::builder( + EXAMPLEID_INTEGRATION_ID + .parse() + .expect("should parse the provider id"), + ) + .with_source(source()) + .with_consent_policy(resolver_policy()) + } + + fn resolves() -> Resolves { + Resolves::new().with_endpoint("https://api.exampleid.example.com/v1/fetch") + } + + fn capturer(capture: CaptureFn) -> TestCapturer { + TestCapturer { + descriptor: builder() + .with_captures(Captures::new().with_path(Method::POST, CAPTURE_PATH)) + .build() + .expect("should build the descriptor"), + capture, + } + } + + fn capture_cases() -> ConformanceCases { + ConformanceCases::default() + .with_capture_case(CaptureFixture::new(Method::POST, CAPTURE_PATH)) + } + + fn plain_request( + input: &ResolveInput<'_>, + ) -> Result, Report> { + ProviderRequest::post_json(input.endpoint()?, &json!({})).map(Some) + } + + fn one_uid( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ResolvedEid::new( + &source(), + ResolvedUid::new("uid-1"), + )])) + } + + fn resolver( + builder: ProviderDescriptorBuilder, + resolves: Resolves, + request: RequestFn, + response: ResponseFn, + ) -> TestResolver { + TestResolver { + descriptor: builder + .with_resolves(resolves) + .build() + .expect("should build the descriptor"), + request, + response, + } + } + + fn resolve_cases() -> ConformanceCases { + ConformanceCases::default().with_resolve_case( + HarnessSetup::default(), + StatusCode::OK, + &json!({}), + ) + } + + fn failure( + provider: &dyn IdentityProvider, + cases: &ConformanceCases, + ) -> Report { + ProviderHarness::new(provider) + .try_check_conformance(cases) + .expect_err("should fail the conformance check") + } + + #[test] + fn example_provider_round_trip() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + let mut harness = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0"); + + let request = harness + .request() + .expect("should build a request") + .expect("should not skip the request"); + assert_eq!( + request.url().host_str(), + Some("api.example.com"), + "should call the declared host" + ); + + let resolution = harness + .respond( + StatusCode::OK, + &json!({ "uid": "example-uid", "signature": "example-signature" }), + ) + .expect("should parse the response"); + assert_eq!( + resolution.eids()[0].uids()[0].id(), + "example-uid", + "should return the uid" + ); + let state: ExampleIdState = harness.state_of(&resolution).expect("should replace state"); + assert_eq!( + state.signature, "example-signature", + "should keep the signature" + ); + } + + #[test] + fn example_provider_conforms() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + ProviderHarness::new(&provider).check_conformance_with_state::( + &ConformanceCases::default().with_resolve_case( + HarnessSetup::default() + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0"), + StatusCode::OK, + &json!({ "uid": "example-uid", "signature": "s" }), + ), + ); + } + + #[test] + fn stateful_provider_is_pointed_at_the_state_check() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::StatefulNeedsStateType + ), + "should point at check_conformance_with_state" + ); + } + + #[test] + fn capture_provider_conforms() { + let provider = ExampleCaptureProvider::new().expect("should build the provider"); + + ProviderHarness::new(&provider).check_conformance( + &ConformanceCases::default().with_capture_case( + CaptureFixture::new(Method::POST, "/integrations/examplecapture/api/token") + .with_response( + StatusCode::OK, + &json!({ "tokens": [{ + "source": "examplecapture.example.com", + "value": "token-1", + "expires": 1_760_086_400_u64, + }] }), + ), + ), + ); + } + + #[test] + fn conformance_catches_an_undeclared_read() { + fn reads_user_agent( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _ignored = input.user_agent().ok(); + plain_request(input) + } + let provider = resolver(builder(), resolves(), reads_user_agent, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::UndeclaredInput { .. } + ), + "should report the undeclared input" + ); + } + + #[test] + fn conformance_catches_an_undeclared_state_read() { + fn reads_state( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _ignored = input.state::().ok(); + plain_request(input) + } + let provider = resolver(builder(), resolves(), reads_state, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::UndeclaredInput { .. } + ), + "should report the undeclared state read" + ); + } + + #[test] + fn harness_skips_the_request_when_the_consent_policy_refuses() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + let allowed = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0") + .request() + .expect("should build a request"); + let refused = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0") + .with_consent(TestConsent::default().with_gdpr_applies(true)) + .request() + .expect("should not fail the call"); + + assert!(allowed.is_some(), "should build a request without a signal"); + assert!( + refused.is_none(), + "should skip the request when GDPR applies and no TCF consent is present" + ); + } + + #[test] + fn descriptor_ignores_secrets() { + check_descriptor_ignores_secrets(&example_config(), &example_config()); + } + + #[test] + fn output_eids_applies_the_provenance_default_only_to_bare_entries() { + fn two_entries( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ + ResolvedEid::new(&source(), ResolvedUid::new("uid-a")) + .with_inserter("vendor-inserter"), + ResolvedEid::new(&source(), ResolvedUid::new("uid-b")), + ])) + } + let provider = resolver( + builder().with_provenance( + &source(), + EidProvenance::new() + .with_inserter("default-inserter") + .with_matcher("default-matcher"), + ), + resolves(), + plain_request, + two_entries, + ); + let mut harness = ProviderHarness::new(&provider); + harness + .request() + .expect("should build a request") + .expect("should not skip the request"); + let resolution = harness + .respond(StatusCode::OK, &json!({})) + .expect("should resolve"); + + let eids = harness.output_eids(&resolution); + + assert_eq!(eids.len(), 2, "should output both entries"); + assert_eq!( + eids[0].inserter.as_deref(), + Some("vendor-inserter"), + "should keep the entry's own fields" + ); + assert_eq!( + eids[0].matcher, None, + "should not mix the default into an entry that sets a field" + ); + assert_eq!( + eids[1].inserter.as_deref(), + Some("default-inserter"), + "should apply the default to a bare entry" + ); + assert_eq!( + eids[1].matcher.as_deref(), + Some("default-matcher"), + "should apply the whole default to a bare entry" + ); + } + + #[test] + fn conformance_catches_a_cookie_header() { + fn sets_cookie( + input: &ResolveInput<'_>, + ) -> Result, Report> { + Ok(plain_request(input)?.map(|request| { + request.with_header(COOKIE, HeaderValue::from_static("session=example")) + })) + } + let provider = resolver(builder(), resolves(), sets_cookie, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::CookieHeader), + "should report the Cookie header" + ); + } + + #[test] + fn conformance_catches_a_request_to_an_undeclared_host() { + fn wrong_host( + _input: &ResolveInput<'_>, + ) -> Result, Report> { + let url = Url::parse("https://other.example.org/v1/fetch").expect("should parse"); + Ok(Some(ProviderRequest::get(&url))) + } + let provider = resolver(builder(), resolves(), wrong_host, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::UndeclaredHost), + "should report the undeclared host" + ); + } + + #[test] + fn conformance_catches_a_request_over_plain_http() { + fn plain_http( + _input: &ResolveInput<'_>, + ) -> Result, Report> { + let url = + Url::parse("http://api.exampleid.example.com/v1/fetch").expect("should parse"); + Ok(Some(ProviderRequest::get(&url))) + } + let provider = resolver(builder(), resolves(), plain_http, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::InsecureScheme), + "should report the plain http request" + ); + } + + #[test] + fn conformance_catches_an_entry_for_an_undeclared_source() { + fn other_source( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let other: SourceDomain = "other.example.com".parse().expect("should parse"); + Ok(Resolution::found([ResolvedEid::new( + &other, + ResolvedUid::new("uid-1"), + )])) + } + let provider = resolver(builder(), resolves(), plain_request, other_source); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::UndeclaredSource { .. } + ), + "should report the undeclared source" + ); + } + + #[test] + fn conformance_catches_a_33rd_uid() { + fn thirty_three( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let mut eid = ResolvedEid::new(&source(), ResolvedUid::new("uid-0")); + for index in 1..33 { + eid = eid.with_uid(ResolvedUid::new(format!("uid-{index}"))); + } + Ok(Resolution::found([eid])) + } + let provider = resolver(builder(), resolves(), plain_request, thirty_three); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::LimitHit { .. }), + "should report the dropped UID" + ); + } + + #[test] + fn conformance_catches_a_blank_inserter() { + fn blank_inserter( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ResolvedEid::new( + &source(), + ResolvedUid::new("uid-1"), + ) + .with_inserter(" ")])) + } + let provider = resolver(builder(), resolves(), plain_request, blank_inserter); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::LimitHit { .. }), + "should report the dropped inserter" + ); + } + + #[test] + fn conformance_catches_an_oversized_set() { + fn eight_large_entries( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let entries = (0..8).map(|index| { + ResolvedEid::new( + &source(), + ResolvedUid::new(format!("uid-{index}")) + .with_ext(json!({ "pad": "x".repeat(1_000) })), + ) + }); + Ok(Resolution::found(entries)) + } + let provider = resolver(builder(), resolves(), plain_request, eight_large_entries); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::SetTooLarge { .. } + ), + "should report the set over MAX_ENTRY_SET_BYTES" + ); + } + + #[test] + fn conformance_catches_a_capability_declared_but_not_overridden() { + let resolving = DeclaresOnly( + builder() + .with_resolves(resolves()) + .build() + .expect("should build the descriptor"), + ); + let capturing = DeclaresOnly( + builder() + .with_captures(Captures::new().with_path(Method::POST, CAPTURE_PATH)) + .build() + .expect("should build the descriptor"), + ); + + let resolve_error = failure(&resolving, &resolve_cases()); + let capture_error = failure(&capturing, &capture_cases()); + + assert!( + matches!( + resolve_error.current_context(), + ConformanceFailure::NotImplemented { method: "request" } + ), + "should report the missing request override" + ); + assert!( + matches!( + capture_error.current_context(), + ConformanceFailure::NotImplemented { method: "capture" } + ), + "should report the missing capture override" + ); + } + + #[test] + fn conformance_catches_a_resolver_that_never_builds_a_request() { + fn skips( + _input: &ResolveInput<'_>, + ) -> Result, Report> { + Ok(None) + } + let provider = resolver(builder(), resolves(), skips, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::ResolverNeverRequests + ), + "should report a resolver that never requests" + ); + } + + #[test] + fn conformance_catches_a_capturer_that_never_captures() { + fn nothing(_input: &CaptureInput<'_>) -> Result> { + Ok(Resolution::no_match()) + } + + let error = failure(&capturer(nothing), &capture_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::CapturerNeverMatches + ), + "should report a capturer that captures nothing" + ); + } + + #[test] + fn conformance_catches_state_from_a_stateless_provider() { + fn returns_state( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok( + Resolution::found([ResolvedEid::new(&source(), ResolvedUid::new("uid-1"))]) + .replacing_state(&json!({ "signature": "s" })), + ) + } + let provider = resolver(builder(), resolves(), plain_request, returns_state); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::StatelessState), + "should report state from a provider without private state" + ); + } + + #[test] + fn conformance_catches_a_capture_that_changes_state() { + fn clears_state( + _input: &CaptureInput<'_>, + ) -> Result> { + Ok( + Resolution::found([ResolvedEid::new(&source(), ResolvedUid::new("uid-1"))]) + .clearing_state(), + ) + } + + let error = failure(&capturer(clears_state), &capture_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::CaptureStateChange + ), + "should report a capture that changes state" + ); + } + + #[test] + fn conformance_catches_a_stateful_provider_that_accepts_a_bare_string_state() { + fn reads_any_state( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _state = input.state::()?; + plain_request(input) + } + fn keeps_state( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok( + Resolution::found([ResolvedEid::new(&source(), ResolvedUid::new("uid-1"))]) + .replacing_state(&json!({ "signature": "s" })), + ) + } + let provider = resolver( + builder(), + resolves().with_private_state(), + reads_any_state, + keeps_state, + ); + + let error = ProviderHarness::new(&provider) + .try_check_conformance_with_state::(&resolve_cases()) + .expect_err("should fail a state type that decodes a bare string"); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::BareStringStateAccepted + ), + "should report the accepted bare string" + ); + } + + #[test] + fn conformance_catches_an_expiring_source_without_an_expiry() { + let provider = resolver( + builder().with_expiring_source(&source()), + resolves(), + plain_request, + one_uid, + ); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::MissingExpiry { .. } + ), + "should report the missing expiry" + ); + } + + #[test] + fn conformance_accepts_a_complete_set_within_the_limits() { + fn complete_set( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ + ResolvedEid::new( + &source(), + ResolvedUid::new("uid-a") + .with_atype(1) + .with_ext(json!({ "stype": "ppuid" })), + ) + .with_uid(ResolvedUid::new("uid-b")) + .with_inserter("inserter.example.com") + .with_matcher("matcher.example.com") + .with_mm(2) + .with_ext(json!({ "segments": ["s"] })), + ResolvedEid::new(&source(), ResolvedUid::new("uid-c")) + .with_ext(json!({ "segments": ["t"] })), + ])) + } + let provider = resolver(builder(), resolves(), plain_request, complete_set); + + let result = ProviderHarness::new(&provider).try_check_conformance(&resolve_cases()); + + assert!( + result.is_ok(), + "should accept two entries and three UIDs with every field" + ); + } + + #[test] + fn stored_full_set_reaches_a_refreshing_provider() { + fn echoes_stored_set( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let stored = input + .stored(&source()) + .map(|stored| stored.eids().to_vec()) + .unwrap_or_default(); + ProviderRequest::post_json(input.endpoint()?, &stored).map(Some) + } + let provider = resolver(builder(), resolves(), echoes_stored_set, one_uid); + let set = vec![ + Eid { + source: "exampleid.example.com".to_owned(), + uids: vec![ + Uid { + id: "uid-a".to_owned(), + atype: Some(1), + ext: None, + }, + Uid { + id: "uid-b".to_owned(), + atype: None, + ext: Some(json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("inserter.example.com".to_owned()), + matcher: None, + mm: Some(2), + ext: None, + }, + Eid { + source: "exampleid.example.com".to_owned(), + uids: vec![Uid { + id: "uid-c".to_owned(), + atype: None, + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: Some(json!({ "segments": ["s"] })), + }, + ]; + let mut harness = ProviderHarness::new(&provider) + .with_stored(StoredId::for_test(&source(), "uid-a").with_eids(set.clone())); + + harness + .request() + .expect("should build a request") + .expect("should not skip the refresh"); + + assert_eq!( + harness.request_json::>(), + set, + "should give the provider the stored full set" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/exchange.rs b/crates/trusted-server-core/src/ec/identity/exchange.rs new file mode 100644 index 0000000000..87911bd704 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/exchange.rs @@ -0,0 +1,445 @@ +//! What a provider sends and receives. The coordinator does all I/O. + +use core::fmt; + +use error_stack::Report; +use http::header::{AUTHORIZATION, CONTENT_TYPE, COOKIE}; +use http::{HeaderMap, HeaderName, HeaderValue, Method, StatusCode}; +use serde::Serialize; +use serde::de::DeserializeOwned; +use url::Url; + +use super::resolution::IdentityProviderError; +use crate::redacted::Redacted; + +/// Maximum JSON bytes of a call context. +pub const MAX_CALL_CONTEXT_BYTES: usize = 1024; + +/// One call a provider wants made. +#[derive(Clone)] +pub struct ProviderRequest { + method: Method, + url: Url, + headers: HeaderMap, + body: Vec, + call_context: Option, +} + +impl fmt::Debug for ProviderRequest { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderRequest") + .field("method", &self.method) + .field("host", &self.url.host_str()) + .field("path", &self.url.path()) + .field("headers", &self.headers.keys().collect::>()) + .field("body_len", &self.body.len()) + .finish_non_exhaustive() + } +} + +impl ProviderRequest { + /// Any method, with no body. + #[must_use] + pub fn new(method: Method, url: &Url) -> Self { + Self { + method, + url: url.clone(), + headers: HeaderMap::new(), + body: Vec::new(), + call_context: None, + } + } + + /// A `GET`. + #[must_use] + pub fn get(url: &Url) -> Self { + Self::new(Method::GET, url) + } + + /// A `POST` with a JSON body. + /// + /// # Errors + /// + /// `Config` when `body` does not serialize. + pub fn post_json( + url: &Url, + body: &impl Serialize, + ) -> Result> { + let bytes = serde_json::to_vec(body) + .map_err(|_| IdentityProviderError::config("request body does not serialize"))?; + Self::new(Method::POST, url).with_body("application/json", bytes) + } + + /// A `POST` with a `text/plain` body. + #[must_use] + pub fn post_text(url: &Url, body: &str) -> Self { + let mut request = Self::new(Method::POST, url); + request + .headers + .insert(CONTENT_TYPE, HeaderValue::from_static("text/plain")); + request.body = body.as_bytes().to_vec(); + request + } + + /// Sets the body and its `Content-Type`. + /// + /// # Errors + /// + /// `Config` when `content_type` is not a valid header value. + pub fn with_body( + mut self, + content_type: &str, + body: Vec, + ) -> Result> { + let value = HeaderValue::from_str(content_type) + .map_err(|_| IdentityProviderError::config("invalid content type"))?; + self.headers.insert(CONTENT_TYPE, value); + self.body = body; + Ok(self) + } + + /// Appends a query parameter, URL-encoded. + #[must_use] + pub fn with_query(mut self, name: &str, value: &str) -> Self { + self.url.query_pairs_mut().append_pair(name, value); + self + } + + /// Sets a header. A `Cookie` header makes the coordinator refuse the call. + #[must_use] + pub fn with_header(mut self, name: HeaderName, value: HeaderValue) -> Self { + self.headers.insert(name, value); + self + } + + /// Sets a header from strings. + /// + /// # Errors + /// + /// `Config` when the name or value is invalid. + pub fn try_with_header( + self, + name: &str, + value: &str, + ) -> Result> { + let name = HeaderName::from_bytes(name.as_bytes()) + .map_err(|_| IdentityProviderError::config("invalid header name"))?; + let value = HeaderValue::from_str(value) + .map_err(|_| IdentityProviderError::config("invalid header value"))?; + Ok(self.with_header(name, value)) + } + + /// Adds `Authorization: Bearer `. + #[must_use] + pub fn with_bearer(mut self, token: &Redacted) -> Self { + if let Ok(mut value) = HeaderValue::from_str(&format!("Bearer {}", token.expose())) { + value.set_sensitive(true); + self.headers.insert(AUTHORIZATION, value); + } + self + } + + /// Data `response` needs from this call. Kept in memory, never sent. + /// + /// # Errors + /// + /// `Config` when it does not serialize or is over 1,024 bytes of JSON. + pub fn try_with_call_context( + mut self, + context: &impl Serialize, + ) -> Result> { + let json = serde_json::to_string(context) + .map_err(|_| IdentityProviderError::config("call context does not serialize"))?; + if json.len() > MAX_CALL_CONTEXT_BYTES { + return Err(IdentityProviderError::config( + "call context exceeds 1024 bytes", + )); + } + self.call_context = Some(json); + Ok(self) + } + + /// The method. + #[must_use] + pub fn method(&self) -> &Method { + &self.method + } + + /// The URL. + #[must_use] + pub fn url(&self) -> &Url { + &self.url + } + + /// The headers. + #[must_use] + pub fn headers(&self) -> &HeaderMap { + &self.headers + } + + /// The body. + #[must_use] + pub fn body(&self) -> &[u8] { + &self.body + } + + /// The call context JSON. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn call_context_json(&self) -> Option<&str> { + self.call_context.as_deref() + } + + /// Whether the provider set a `Cookie` header. + #[must_use] + pub(crate) fn has_cookie_header(&self) -> bool { + self.headers.contains_key(COOKIE) + } +} + +/// A buffered, decoded provider response. +#[derive(Clone)] +pub struct ProviderResponse { + status: StatusCode, + headers: HeaderMap, + body: Vec, + call_context: Option, +} + +impl fmt::Debug for ProviderResponse { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderResponse") + .field("status", &self.status) + .field("body_len", &self.body.len()) + .finish_non_exhaustive() + } +} + +impl ProviderResponse { + /// Built by the coordinator after buffering and decoding. + #[must_use] + pub(crate) fn new( + status: StatusCode, + headers: HeaderMap, + body: Vec, + call_context: Option, + ) -> Self { + Self { + status, + headers, + body, + call_context, + } + } + + /// A response for tests. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn for_test(status: StatusCode, body: &[u8]) -> Self { + Self::new(status, HeaderMap::new(), body.to_vec(), None) + } + + /// The status. + #[must_use] + pub fn status(&self) -> StatusCode { + self.status + } + + /// A header as a string, if present and visible ASCII. + #[must_use] + pub fn header(&self, name: &str) -> Option<&str> { + self.headers.get(name).and_then(|value| value.to_str().ok()) + } + + /// The decoded body. + #[must_use] + pub fn body(&self) -> &[u8] { + &self.body + } + + /// Parses the body as JSON, whatever the status. + /// + /// # Errors + /// + /// `Malformed` when it does not parse. + pub fn json(&self) -> Result> { + serde_json::from_slice(&self.body).map_err(|_| IdentityProviderError::malformed()) + } + + /// Requires 2xx, then parses JSON. + /// + /// # Errors + /// + /// `Rejected` for a non-2xx status, `Malformed` for a bad body. + pub fn success_json(&self) -> Result> { + self.require_success()?; + self.json() + } + + /// Requires 2xx and a UTF-8 body. + /// + /// # Errors + /// + /// `Rejected` for a non-2xx status, `Malformed` for non-UTF-8. + pub fn success_text(&self) -> Result<&str, Report> { + self.require_success()?; + core::str::from_utf8(&self.body).map_err(|_| IdentityProviderError::malformed()) + } + + /// The context the request carried, decoded. + /// + /// # Errors + /// + /// `Config` when it does not decode as `T`. + pub fn call_context( + &self, + ) -> Result, Report> { + self.call_context + .as_deref() + .map(|json| { + serde_json::from_str(json) + .map_err(|_| IdentityProviderError::config("call context does not decode")) + }) + .transpose() + } + + fn require_success(&self) -> Result<(), Report> { + if self.status.is_success() { + Ok(()) + } else { + Err(IdentityProviderError::rejected(self.status)) + } + } +} + +#[cfg(test)] +mod tests { + use http::{HeaderName, HeaderValue, Method, StatusCode}; + use serde_json::json; + + use super::*; + + fn url() -> Url { + Url::parse("https://api.example.com/v1/fetch").expect("should parse") + } + + #[test] + fn post_json_sets_body_and_content_type() { + let request = ProviderRequest::post_json(&url(), &json!({ "a": 1 })) + .expect("should build a JSON request"); + assert_eq!(request.method(), Method::POST, "should use POST"); + assert_eq!( + request.headers()["content-type"], + "application/json", + "should set the JSON content type" + ); + assert_eq!(request.body(), br#"{"a":1}"#, "should serialize the body"); + } + + #[test] + fn query_bearer_and_headers() { + let request = ProviderRequest::get(&url()) + .with_query("pid", "1 2") + .with_bearer(&Redacted::new("token".to_owned())) + .try_with_header("x-example", "v") + .expect("should accept a valid header"); + assert_eq!( + request.url().as_str(), + "https://api.example.com/v1/fetch?pid=1+2", + "should encode the query" + ); + assert_eq!( + request.headers()["authorization"], + "Bearer token", + "should set the bearer token" + ); + assert_eq!(request.headers()["x-example"], "v", "should set the header"); + let _ = ProviderRequest::get(&url()) + .try_with_header("bad header", "v") + .expect_err("should reject a bad name"); + let _ = ProviderRequest::get(&url()) + .try_with_header("x", "bad\nvalue") + .expect_err("should reject a bad value"); + } + + #[test] + fn cookie_headers_are_flagged_from_either_helper() { + let typed = ProviderRequest::get(&url()).with_header( + HeaderName::from_static("cookie"), + HeaderValue::from_static("a=1"), + ); + let dynamic = ProviderRequest::get(&url()) + .try_with_header("Cookie", "a=1") + .expect("should accept a Cookie header from strings"); + assert!( + typed.has_cookie_header(), + "should flag a Cookie header from the typed helper" + ); + assert!( + dynamic.has_cookie_header(), + "should flag a Cookie header from the string helper" + ); + assert!( + !ProviderRequest::get(&url()).has_cookie_header(), + "should not flag a request without one" + ); + } + + #[test] + fn call_context_round_trips_and_is_capped() { + let request = ProviderRequest::get(&url()) + .try_with_call_context(&json!({ "nonce": [1, 2] })) + .expect("should accept a small call context"); + let response = ProviderResponse::new( + StatusCode::OK, + http::HeaderMap::new(), + Vec::new(), + request.call_context_json().map(str::to_owned), + ); + let context: serde_json::Value = response + .call_context() + .expect("should decode the call context") + .expect("should carry the call context"); + assert_eq!(context, json!({ "nonce": [1, 2] }), "should round-trip"); + + let error = ProviderRequest::get(&url()) + .try_with_call_context(&"x".repeat(MAX_CALL_CONTEXT_BYTES)) + .expect_err("should reject an oversized call context"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ), + "should report an oversized call context as Config" + ); + } + + #[test] + fn success_helpers_check_status_and_encoding() { + let ok = ProviderResponse::for_test(StatusCode::OK, br#"{"uid":"u"}"#); + let value: serde_json::Value = ok.success_json().expect("should parse a 2xx JSON body"); + assert_eq!(value["uid"], "u", "should read the uid"); + + let rejected = ProviderResponse::for_test(StatusCode::FORBIDDEN, br#"{}"#); + let error = rejected + .success_json::() + .expect_err("should reject a non-2xx status"); + assert!( + matches!(error.current_context(), IdentityProviderError::Rejected { status } if *status == StatusCode::FORBIDDEN), + "should report Rejected with the status" + ); + rejected + .json::() + .expect("should parse JSON whatever the status"); + + let binary = ProviderResponse::for_test(StatusCode::OK, &[0xff, 0xfe]); + let error = binary.success_text().expect_err("should reject non-UTF-8"); + assert!( + matches!(error.current_context(), IdentityProviderError::Malformed), + "should report non-UTF-8 as Malformed" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/flush.rs b/crates/trusted-server-core/src/ec/identity/flush.rs new file mode 100644 index 0000000000..c2969b4fb8 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/flush.rs @@ -0,0 +1,1713 @@ +//! Phase 4: the writer's one flush after send (spec 5.2). +//! +//! The writer request stages every identity change in one +//! [`IdentityMutation`]. After the response is sent, this module dispatches +//! the after-send providers and writes the whole mutation once, with +//! `IfGenerationMatch`. A failed flush is not retried as a whole; only its +//! removals are written again, alone (spec 5.2 phase 4 step 5). + +use std::collections::{BTreeMap, BTreeSet}; +use std::net::IpAddr; +use std::time::Duration; + +use http::StatusCode; + +use super::coordinator::{ + Call, Eligibility, EligibilityInput, EligibilityView, IdentityDeps, OutcomeChanges, + OutcomeContext, Phase, RequestFacts, dispatch, eligibility, opt_out_clears, outcome_to_changes, +}; +use super::descriptor::{Input, Resolves}; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::input::{ProviderConsent, Region, ResolveSignals, StoredId, ThreadRandom}; +use super::mutation::{ApplyContext, IdentityMutation, ProviderStateChange, apply}; +use super::pending::{append_pending, load_pending}; +use super::policy::{PolicySignals, fingerprint_bits}; +use super::provider::IdentityProvider; +use super::removal::{RemovalTiming, log_removal_lost, write_removals}; +use super::revision::StateRevision; +use super::runtime::IdentityRuntime; +use crate::consent::allows_eid_persistence; +use crate::ec::kv::KvIdentityGraph; +use crate::ec::kv_backend::EcKvWriteOutcome; +use crate::ec::kv_types::KvEntry; +use crate::ec::rate_limiter::RateLimiter; +use crate::ec::{EcContext, EcKvSnapshot, current_timestamp, log_id}; +use crate::platform::RuntimeServices; +use crate::settings::{ProviderMode, Settings}; + +pub use super::removal::RemovalWriteOutcome; + +/// What phase 4 needs. The catalog and keys come from `runtime`. +pub struct AfterSendInput<'a> { + /// The identity runtime of this settings load. + pub runtime: &'a IdentityRuntime, + /// The settings the request ran with. + pub settings: &'a Settings, + /// The request's platform services, for the after-send calls. + pub services: &'a RuntimeServices, + /// The identity graph to read and write. + pub graph: &'a KvIdentityGraph, + /// The request's EC state, holding the writer decision and the mutation. + pub context: &'a mut EcContext, + /// The status of the response that was sent. + pub status: StatusCode, + /// The per-visitor burst cap. + pub rate_limiter: &'a dyn RateLimiter, +} + +/// How phase 4 ended. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum FlushOutcome { + /// Not a returning EC's writer, or nothing to do. Nothing was read. + Skipped, + /// The mutation changed nothing in the row, so nothing was written. + NoChange, + /// The one write landed. + Written, + /// The row was a tombstone, missing or unreadable, or the merged row was + /// invalid. The mutation was dropped and no provider was called after it. + Dropped(&'static str), + /// The one write failed. `removals` is how the removal-only retry ended. + WriteFailed { + /// The removal-only retry's outcome. + removals: RemovalWriteOutcome, + }, +} + +/// Whether phase 4 can have work for this request: it is a returning EC's +/// writer. A caller checks this before building the identity graph. +#[must_use] +pub fn may_flush(context: &EcContext) -> bool { + context.identity().writer() && context.ec_was_present() && !context.ec_generated() +} + +/// Runs phase 4 for the writer request (spec 5.2, steps 1 to 5). +/// +/// It reads the row at most once, calls the due after-send providers only +/// after that read confirms a live, consenting row, and writes at most once +/// with the generation it read. Only a failed write that holds removals adds +/// the removal-only write of spec 5.3. +#[must_use] +pub fn run_after_send(input: AfterSendInput<'_>) -> FlushOutcome { + let AfterSendInput { + runtime, + settings, + services, + graph, + context, + status, + rate_limiter, + } = input; + + let returning_writer = may_flush(context); + let Some(ec_id) = context.ec_value().map(str::to_owned) else { + return FlushOutcome::Skipped; + }; + if !returning_writer { + return FlushOutcome::Skipped; + } + + let mut mutation = core::mem::take(context.identity_mut().mutation_mut()); + let status_ok = status.is_success() || status == StatusCode::NOT_MODIFIED; + if !status_ok { + mutation.retain_for_error_status(); + } + + let now = current_timestamp(); + // Hold-mode providers run in phase 4 unless the hold dispatch already + // considered them on a usable row (spec 4.7 step 7). + let facts = request_facts(context, now, context.identity().hold_dispatched()); + let phase_one = context.kv_snapshot().clone(); + let phase_one_entry = phase_one.entry_for(&ec_id); + let phase_one_view = + phase_one_entry.map_or_else(EligibilityView::empty, EligibilityView::from_kv); + let due = if status_ok { + due_providers(runtime, settings, context, &phase_one_view, &facts) + } else { + Vec::new() + }; + let needs_cluster_size = phase_one_entry.is_some_and(lacks_cluster_size); + // The SPA writer writes only on a row its auction read, and only once + // the interval since the last writer flush has passed (spec 5.1). + if context.identity().spa_writer() { + let interval = settings.ec.identity.spa_writer_interval_sec; + let waiting = phase_one_entry + .is_none_or(|entry| entry.wa.is_some_and(|at| now < at.saturating_add(interval))); + if waiting { + return FlushOutcome::Skipped; + } + } + // Every returning writer reads the pending key: what earlier requests + // could not write waits there for it (spec 5.3, Pending key). + let phase_one_applied = phase_one_entry.and_then(|entry| entry.pg).unwrap_or(0); + let mut pending = load_pending(graph, &ec_id, phase_one_applied); + // A re-grant clears a sticky opt-out on the writer's one flush (spec 8). + let regranted = opt_out_clears( + &phase_one_view, + runtime.catalog(), + context.consent(), + facts.policy_signals, + ); + + if mutation.is_empty() + && due.is_empty() + && !needs_cluster_size + && regranted.is_empty() + && pending.is_none() + { + return FlushOutcome::Skipped; + } + + // Phase 1's read decides when no provider is due (spec 5.2 phase 4 + // step 2). A failed or absent phase 1 read counts as no read. + let reusable = match &phase_one { + EcKvSnapshot::Present { + entry, + generation: Some(generation), + .. + } if due.is_empty() && phase_one.belongs_to(&ec_id) => { + Some((entry.as_ref().clone(), *generation)) + } + EcKvSnapshot::Missing { .. } if due.is_empty() && phase_one.belongs_to(&ec_id) => { + return drop_mutation(&ec_id, "missing"); + } + _ => None, + }; + let (live, generation) = match reusable { + Some(snapshot) => snapshot, + None => match graph.get(&ec_id) { + Ok(Some(row)) => row, + Ok(None) => return drop_mutation(&ec_id, "missing"), + Err(error) => { + log::warn!( + "Identity flush for '{}' could not read the row: {}", + log_id(&ec_id), + error.current_context() + ); + return drop_mutation(&ec_id, "read failed"); + } + }, + }; + if !live.consent.ok { + return drop_mutation(&ec_id, "tombstone"); + } + let live_applied = live.pg.unwrap_or(0); + if pending.is_some() && live_applied != phase_one_applied { + pending = load_pending(graph, &ec_id, live_applied); + } + + // The row is live and consenting: only now may the EC ID leave TS. + let live_view = EligibilityView::from_kv(&live); + let calls: Vec = due + .iter() + .filter_map(|id| live_call(runtime, settings, context, &live_view, &facts, id)) + .collect(); + if !calls.is_empty() { + let random = ThreadRandom; + let deps = runtime.deps(settings, services, &random, Some(rate_limiter)); + // Phase 4 runs after the response is sent, outside any executor, so + // blocking here is safe. The hold dispatch awaits instead: it runs + // inside the request handler's executor. + futures::executor::block_on(add_dispatch_results( + &mut mutation, + &deps, + &DispatchInput { + context, + ec_id: &ec_id, + live: &live, + view: &live_view, + facts: &facts, + }, + DispatchRun { + calls, + budget: None, + }, + )); + } + + for (provider, watermark) in regranted { + mutation.merge_state( + provider, + ProviderStateChange { + opt_out_clear: Some(watermark), + ..ProviderStateChange::default() + }, + ); + } + + if lacks_cluster_size(&live) { + match graph.compute_cluster_size(&ec_id) { + Ok(size) => mutation.set_cluster_size(size), + Err(error) => log::warn!( + "Identity flush for '{}' could not compute the cluster size: {}", + log_id(&ec_id), + error.current_context() + ), + } + } + + // Only this request's own removals go back to the pending key if the + // write fails: the merged items are still there. + let own_removals = mutation.removals(); + let pending_generation = pending.map(|merge| { + let mut merged = merge.mutation; + merged.absorb(core::mem::take(&mut mutation)); + mutation = merged; + merge.generation + }); + + let random = ThreadRandom; + let apply_context = ApplyContext { + now, + random: &random, + catalog: runtime.catalog(), + }; + let Some(mut applied) = apply(&mutation, &live, &apply_context) else { + return drop_mutation(&ec_id, "invalid"); + }; + if let Some(generation) = pending_generation { + applied.entry.pg = Some(generation); + applied.changed = true; + } + if !applied.changed { + return FlushOutcome::NoChange; + } + applied.entry.wa = Some(now); + + match graph.write_if_generation(&ec_id, &applied.entry, generation) { + Ok(EcKvWriteOutcome::Written) => FlushOutcome::Written, + Ok(EcKvWriteOutcome::PreconditionFailed) => { + log::warn!( + "Identity flush for '{}' lost its generation check; not retried", + log_id(&ec_id) + ); + write_failed( + graph, + context, + &ec_id, + &mutation, + &own_removals, + &apply_context, + ) + } + Err(error) => { + log::warn!( + "Identity flush for '{}' failed to write: {}", + log_id(&ec_id), + error.current_context() + ); + write_failed( + graph, + context, + &ec_id, + &mutation, + &own_removals, + &apply_context, + ) + } + } +} + +fn drop_mutation(ec_id: &str, reason: &'static str) -> FlushOutcome { + log::debug!( + "Identity flush for '{}' dropped its mutation: {reason}", + log_id(ec_id) + ); + FlushOutcome::Dropped(reason) +} + +fn lacks_cluster_size(entry: &KvEntry) -> bool { + entry.consent.ok + && entry + .network + .as_ref() + .and_then(|network| network.cluster_size) + .is_none() +} + +/// Writes the failed flush's removals alone. When that fails too, the +/// request's own removals go to the pending key (spec 5.2 phase 4 step 5). +fn write_failed( + graph: &KvIdentityGraph, + context: &mut EcContext, + ec_id: &str, + mutation: &IdentityMutation, + own_removals: &IdentityMutation, + apply_context: &ApplyContext<'_>, +) -> FlushOutcome { + let removals = mutation.removals(); + if !removals.has_removals() { + return FlushOutcome::WriteFailed { + removals: RemovalWriteOutcome::NothingToWrite, + }; + } + // Phase 4 runs after send, right after the failed flush. + let outcome = write_removals( + graph, + ec_id, + &removals, + apply_context, + RemovalTiming::AfterSend, + ); + if outcome == RemovalWriteOutcome::Failed { + defer_or_lose(graph, context, ec_id, own_removals); + } + FlushOutcome::WriteFailed { removals: outcome } +} + +/// Appends removals the row write missed to the pending key and counts each +/// as deferred, or, when that fails too, logs and counts each as lost. +fn defer_or_lose( + graph: &KvIdentityGraph, + context: &mut EcContext, + ec_id: &str, + removals: &IdentityMutation, +) { + if !removals.has_removals() { + return; + } + let deferred = append_pending(graph, ec_id, removals, current_timestamp()); + if !deferred { + log_removal_lost(ec_id, removals); + } + for _ in removals.removal_labels() { + if deferred { + context.identity_mut().record_removal_deferred(); + } else { + context.identity_mut().record_removal_lost(); + } + } +} + +/// Appends the positive writes a capture kept, on a request that is not the +/// writer, to the pending key for the next writer (spec 5.3, Pending key). +/// Without an EC ID, or after a global revoke, it drops them. +pub fn defer_capture_writes(context: &mut EcContext, graph: &KvIdentityGraph) { + let writes = core::mem::take(context.identity_mut().capture_writes_mut()); + if writes.is_empty() || context.global_revoke_requested() { + return; + } + let Some(ec_id) = context.ec_value().map(str::to_owned) else { + return; + }; + if !append_pending(graph, &ec_id, &writes, current_timestamp()) { + log::warn!( + "Capture writes for '{}' could not reach the pending key", + log_id(&ec_id) + ); + } +} + +/// Tries once more, after send, the capture removal write that failed before +/// send (spec 5.3). +/// +/// This is the last row attempt: the pending removals are taken, and when +/// they still cannot be written they go to the pending key, or are logged +/// and counted as lost if that fails too. Without a pending removal it does +/// nothing. +pub fn retry_pending_removals(context: &mut EcContext, graph: &KvIdentityGraph) { + let Some(removals) = context.identity().pending_removals().cloned() else { + return; + }; + let ec_id = context.ec_value().map(str::to_owned); + let runtime = context.identity().runtime().cloned(); + let (Some(ec_id), Some(runtime)) = (ec_id, runtime) else { + log::warn!("Capture removal retry skipped: no EC ID or identity runtime"); + lose_pending_removals(context); + return; + }; + let random = ThreadRandom; + let outcome = write_removals( + graph, + &ec_id, + &removals, + &ApplyContext { + now: current_timestamp(), + random: &random, + catalog: runtime.catalog(), + }, + RemovalTiming::AfterSend, + ); + context.identity_mut().set_pending_removals(None); + if outcome == RemovalWriteOutcome::Failed { + defer_or_lose(graph, context, &ec_id, &removals); + } +} + +/// Gives up the capture removal write that failed before send: logs and +/// counts each removal as lost. For an after-send run that cannot even try, +/// such as one without an identity graph. +pub fn lose_pending_removals(context: &mut EcContext) { + let Some(removals) = context.identity().pending_removals().cloned() else { + return; + }; + context.identity_mut().set_pending_removals(None); + log_removal_lost(context.ec_value().unwrap_or_default(), &removals); + for _ in removals.removal_labels() { + context.identity_mut().record_removal_lost(); + } +} + +/// What eligibility reads about the request itself. +pub(super) fn request_facts(context: &EcContext, now: u64, runs_auction: bool) -> RequestFacts { + let identity = context.identity(); + let mut available_inputs = BTreeSet::new(); + for (input, present) in [ + (Input::ClientIp, client_ip(context).is_some()), + (Input::UserAgent, identity.user_agent().is_some()), + (Input::PageUrl, identity.page_url().is_some()), + (Input::Region, region(context).is_some()), + ] { + if present { + available_inputs.insert(input); + } + } + RequestFacts { + is_real_browser: identity.is_real_browser(), + ec_persistence_allowed: context.ec_allowed() && allows_eid_persistence(context.consent()), + policy_signals: PolicySignals { + dnt: identity.dnt(), + }, + writer: identity.writer(), + // A minted EC whose create waits for the hold dispatch may run + // hold-mode providers (spec 4.7, new visitors). + new_visitor: context.ec_generated() && context.deferred_entry().is_none(), + runs_auction, + now: UnixTime::from_secs(now), + available_inputs, + } +} + +pub(super) fn client_ip(context: &EcContext) -> Option { + context.client_ip().and_then(|ip| ip.parse().ok()) +} + +pub(super) fn region(context: &EcContext) -> Option { + context + .geo_info() + .and_then(|geo| Region::from_geo(&geo.country, geo.region.as_deref())) +} + +/// The eligibility of `provider` for phase 4 against `view`. +fn after_send_eligibility( + provider: &dyn IdentityProvider, + runtime: &IdentityRuntime, + settings: &Settings, + context: &EcContext, + view: &EligibilityView, + facts: &RequestFacts, +) -> Eligibility { + let provider_settings = settings.ec.identity.provider(provider.descriptor().id()); + eligibility( + &EligibilityInput { + provider, + settings: &provider_settings, + identity: &settings.ec.identity, + catalog: runtime.catalog(), + consent: context.consent(), + }, + view, + facts, + Phase::AfterSend, + ) +} + +/// The after-send providers due on the phase 1 view. +fn due_providers( + runtime: &IdentityRuntime, + settings: &Settings, + context: &EcContext, + view: &EligibilityView, + facts: &RequestFacts, +) -> Vec { + let spa_writer = context.identity().spa_writer(); + runtime + .catalog() + .providers() + // The SPA writer never runs a hold-mode provider (spec 5.1). + .filter(|provider| { + !spa_writer + || settings + .ec + .identity + .provider(provider.descriptor().id()) + .mode + == ProviderMode::AfterSend + }) + .filter(|provider| { + matches!( + after_send_eligibility(provider.as_ref(), runtime, settings, context, view, facts), + Eligibility::Eligible { .. } + ) + }) + .map(|provider| provider.descriptor().id().clone()) + .collect() +} + +/// The call of a provider still due on the live row, with its live `ps.st`. +fn live_call( + runtime: &IdentityRuntime, + settings: &Settings, + context: &EcContext, + view: &EligibilityView, + facts: &RequestFacts, + id: &IdentityProviderId, +) -> Option { + let provider = runtime.catalog().get(id)?; + let Eligibility::Eligible { kinds } = + after_send_eligibility(provider.as_ref(), runtime, settings, context, view, facts) + else { + return None; + }; + Some(Call { + provider: id.clone(), + kinds, + state_json: view + .state(id) + .and_then(|state| serde_json::to_string(state).ok()), + }) +} + +/// What dispatching the due providers reads. +pub(super) struct DispatchInput<'a> { + pub(super) context: &'a EcContext, + pub(super) ec_id: &'a str, + pub(super) live: &'a KvEntry, + pub(super) view: &'a EligibilityView, + pub(super) facts: &'a RequestFacts, +} + +/// The calls to send and, for a hold dispatch before the auction, the budget +/// they share. +pub(super) struct DispatchRun { + pub(super) calls: Vec, + pub(super) budget: Option, +} + +/// Dispatches `run.calls` and adds their outcomes to `mutation`. +pub(super) async fn add_dispatch_results( + mutation: &mut IdentityMutation, + deps: &IdentityDeps<'_>, + input: &DispatchInput<'_>, + run: DispatchRun, +) { + let DispatchRun { calls, budget } = run; + let resolving: BTreeSet<&SourceDomain> = + calls.iter().flat_map(|call| call.kinds.keys()).collect(); + let stored: BTreeMap = input + .view + .records() + .filter(|(source, _)| resolving.contains(source)) + .filter_map(|(source, record)| { + StoredId::from_record(source, record).map(|stored| (source.clone(), stored)) + }) + .collect(); + let identity = input.context.identity(); + let signals = ResolveSignals { + client_ip: client_ip(input.context), + user_agent: identity.user_agent(), + page_url: identity.page_url().cloned(), + region: region(input.context), + hashed_email: None, + consent: ProviderConsent::from_context(input.context.consent()), + publisher_domain: &deps.settings.publisher.domain, + now: input.facts.now, + random: deps.random, + ec_id: Some(input.ec_id.to_owned()), + stored, + }; + let catalog = deps.catalog; + for (id, outcome) in dispatch(deps, calls, &signals, budget).await { + let Some(provider) = catalog.get(&id) else { + continue; + }; + let provider: &dyn IdentityProvider = provider.as_ref(); + let descriptor = provider.descriptor(); + let provider_settings = deps.settings.ec.identity.provider(&id); + let fingerprint = fingerprint_bits( + descriptor.consent_policy(), + input.context.consent(), + input.facts.policy_signals, + ); + let stateful = descriptor + .resolves() + .is_some_and(Resolves::has_private_state); + let outcome_context = OutcomeContext { + now: input.facts.now, + catalog, + settings: &provider_settings, + identity: &deps.settings.ec.identity, + expected_state: stateful.then(|| { + StateRevision::of( + input + .live + .ps + .get(id.as_str()) + .and_then(|state| state.st.as_ref()), + ) + }), + fingerprint: &fingerprint, + }; + let found = outcome.is_found(); + let changes = outcome_to_changes(provider, outcome, input.view, &outcome_context); + add_outcome(mutation, &id, changes, found); + } +} + +/// Adds one outcome's changes to the mutation. +/// +/// A `found` outcome's state change is tagged, so an error status drops its +/// `n` and `f` with its positive answer (spec 5.1). +fn add_outcome( + mutation: &mut IdentityMutation, + id: &IdentityProviderId, + changes: OutcomeChanges, + found: bool, +) { + let OutcomeChanges { + changes, + result, + state, + } = changes; + for change in changes { + mutation.add_change(change); + } + if let Some(result) = result { + mutation.add_result(result); + } + if found { + mutation.merge_found_state(id.clone(), state); + } else { + mutation.merge_state(id.clone(), state); + } +} + +#[cfg(test)] +mod tests { + use std::cell::RefCell; + use std::sync::Arc; + + use error_stack::Report; + + use super::*; + use crate::consent::types::GppConsent; + use crate::ec::identity::denials::{denials, filter_browser_candidates}; + use crate::ec::identity::mutation::PartnerIdChange; + use crate::ec::identity::pending::pending_key; + use crate::ec::identity::removal::PER_KEY_WRITE_WINDOW; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::revision::{record_revision, uid_hash, visitor_key}; + use crate::ec::identity::test_support::fake_http::{ + FakeBackend, FakeHttpClient, fake_services, + }; + use crate::ec::identity::test_support::{ + TEST_EC_ID, WriteCountingEcKv, allowing_consent, ec_partner, example_entry_set, + full_record, pid, record_for_test, source, + }; + use crate::ec::kv_types::{KvNetwork, KvOptOut, KvPartnerId, KvProviderState}; + use crate::error::TrustedServerError; + use crate::openrtb::{Eid, Uid}; + use crate::settings::ProviderSettings; + use crate::test_support::tests::create_test_settings; + + const PULL_HOST: &str = "sync.ssp.example.com"; + const PULL_SOURCE: &str = "ssp.example.com"; + const PULL_ID: &str = "pull-ssp-example-com"; + + /// Answers every burst-cap read with `exceeded`, and records the keys. + struct ScriptedLimiter { + exceeded: bool, + keys: RefCell>, + } + + impl ScriptedLimiter { + fn allowing() -> Self { + Self { + exceeded: false, + keys: RefCell::new(Vec::new()), + } + } + + fn capped() -> Self { + Self { + exceeded: true, + keys: RefCell::new(Vec::new()), + } + } + } + + impl RateLimiter for ScriptedLimiter { + fn exceeded(&self, key: &str, _limit: u32) -> Result> { + self.keys.borrow_mut().push(key.to_owned()); + Ok(self.exceeded) + } + + fn record(&self, _key: &str) -> Result<(), Report> { + Ok(()) + } + + fn count(&self, _key: &str) -> Result> { + Ok(0) + } + } + + struct Fixture { + settings: Settings, + runtime: IdentityRuntime, + client: Arc, + services: RuntimeServices, + graph: KvIdentityGraph, + store: Arc, + context: EcContext, + } + + impl Fixture { + /// A returning writer whose phase 1 read found `row`. With + /// `pull_partner`, one generic pull partner is configured. + fn new(row: Option<&KvEntry>, pull_partner: bool) -> Self { + Self::with_settings(row, Self::settings(pull_partner)) + } + + fn settings(pull_partner: bool) -> Settings { + let mut settings = create_test_settings(); + settings.ec.partners = if pull_partner { + vec![ec_partner( + PULL_SOURCE, + &format!("https://{PULL_HOST}/pull"), + "example-token-value-0123456789abcdef", + )] + } else { + Vec::new() + }; + settings + } + + fn with_settings(row: Option<&KvEntry>, settings: Settings) -> Self { + let runtime = IdentityRuntime::from_settings( + &settings, + &crate::integrations::IntegrationRegistry::default(), + ) + .expect("should build runtime"); + let client = Arc::new(FakeHttpClient::new()); + let backend = Arc::new(FakeBackend::new()); + let services = fake_services(&client, &backend); + let (graph, store) = KvIdentityGraph::write_counting("flush-store"); + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + allowing_consent(), + Some("192.0.2.10".to_owned()), + ); + context.identity_mut().set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + if let Some(row) = row { + graph.create(TEST_EC_ID, row).expect("should seed the row"); + context.set_kv_snapshot(graph.load_snapshot(TEST_EC_ID)); + } + store.reset_counts(); + Self { + settings, + runtime, + client, + services, + graph, + store, + context, + } + } + + fn run(&mut self, limiter: &ScriptedLimiter) -> FlushOutcome { + self.run_with_status(limiter, StatusCode::OK) + } + + fn run_with_status( + &mut self, + limiter: &ScriptedLimiter, + status: StatusCode, + ) -> FlushOutcome { + run_after_send(AfterSendInput { + runtime: &self.runtime, + settings: &self.settings, + services: &self.services, + graph: &self.graph, + context: &mut self.context, + status, + rate_limiter: limiter, + }) + } + + fn stage(&mut self, change: PartnerIdChange) { + self.context + .identity_mut() + .mutation_mut() + .add_change(change); + } + + fn stage_sharedid(&mut self) { + self.stage(PartnerIdChange::FillMissing { + source: source("sharedid.org"), + record: KvPartnerId::browser("shared-uid", Some(current_timestamp())), + }); + } + + fn row(&self) -> KvEntry { + self.graph + .get(TEST_EC_ID) + .expect("should read the row") + .expect("should hold the row") + .0 + } + } + + /// A live, consenting row; `cluster_size` is set unless `sized` is false. + fn live_row(sized: bool) -> KvEntry { + let mut entry = KvEntry::new(&allowing_consent(), None, 1_000, "test-publisher.com"); + if sized { + entry.network = Some(KvNetwork { + cluster_size: Some(1), + }); + } + entry + } + + fn pull_answers(fixture: &Fixture, uid: &str) { + fixture + .client + .respond(PULL_HOST, 200, format!(r#"{{"uid":"{uid}"}}"#).as_bytes()); + } + + #[test] + fn writer_flushes_mutation_once_with_generation_match() { + let mut fixture = Fixture::new(Some(&live_row(true)), false); + fixture.stage_sharedid(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the mutation"); + let writes = fixture.store.writes(); + assert_eq!( + writes.if_generation_match, 1, + "should write once with a CAS" + ); + assert_eq!(writes.total(), 1, "should make no other write"); + assert_eq!(writes.lookups, 0, "should reuse the phase 1 read"); + assert!( + fixture.row().ids.contains_key("sharedid.org"), + "should store the sharedId fill" + ); + } + + #[test] + fn writer_with_nothing_to_do_reads_and_writes_nothing() { + let mut fixture = Fixture::new(Some(&live_row(true)), false); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Skipped, "should have nothing to do"); + let writes = fixture.store.writes(); + assert_eq!( + (writes.lookups, writes.total()), + (0, 0), + "should touch no KV" + ); + } + + #[test] + fn non_2xx_or_304_status_drops_ingest_and_dispatches_nothing() { + let mut fixture = Fixture::new(Some(&live_row(true)), true); + pull_answers(&fixture, "pulled-uid"); + fixture.stage_sharedid(); + + let outcome = fixture.run_with_status( + &ScriptedLimiter::allowing(), + StatusCode::INTERNAL_SERVER_ERROR, + ); + + assert_eq!(outcome, FlushOutcome::Skipped, "should drop the ingest"); + assert!( + fixture.client.sent_hosts().is_empty(), + "should call no provider" + ); + let writes = fixture.store.writes(); + assert_eq!( + (writes.lookups, writes.total()), + (0, 0), + "should touch no KV" + ); + } + + #[test] + fn not_modified_status_still_flushes() { + let mut fixture = Fixture::new(Some(&live_row(true)), false); + fixture.stage_sharedid(); + + let outcome = + fixture.run_with_status(&ScriptedLimiter::allowing(), StatusCode::NOT_MODIFIED); + + assert_eq!( + outcome, + FlushOutcome::Written, + "should treat 304 as success" + ); + } + + #[test] + fn tombstone_on_reread_drops_mutation_and_calls_no_provider() { + let mut fixture = Fixture::new(Some(&live_row(true)), true); + pull_answers(&fixture, "leaked-uid"); + fixture.stage_sharedid(); + let tombstoned = fixture + .graph + .write_withdrawal_tombstone(TEST_EC_ID, drop) + .expect("should tombstone the row"); + assert_eq!( + tombstoned, + crate::ec::kv::TombstoneOutcome::Written, + "should tombstone the live row" + ); + fixture.store.reset_counts(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!( + outcome, + FlushOutcome::Dropped("tombstone"), + "should drop the mutation on a tombstone" + ); + assert!( + fixture.client.sent_hosts().is_empty(), + "should disclose the EC ID to no provider" + ); + assert_eq!(fixture.store.writes().total(), 0, "should write nothing"); + assert!( + fixture.row().ids.is_empty(), + "should leave the tombstone bare" + ); + } + + #[test] + fn failed_phase_one_read_rereads_before_writing() { + let mut fixture = Fixture::new(Some(&live_row(true)), false); + fixture.context.set_kv_snapshot(EcKvSnapshot::Failed { + ec_id: TEST_EC_ID.to_owned(), + }); + fixture.stage_sharedid(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!( + outcome, + FlushOutcome::Written, + "should write after the re-read" + ); + let writes = fixture.store.writes(); + assert_eq!(writes.lookups, 1, "should re-read once"); + assert_eq!(writes.if_generation_match, 1, "should write once"); + } + + #[test] + fn ingest_queued_against_an_unread_snapshot_is_written_after_a_read() { + let mut fixture = Fixture::new(Some(&live_row(true)), false); + fixture.context.set_kv_snapshot(EcKvSnapshot::NotRead); + fixture.stage_sharedid(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the ingest"); + assert_eq!( + fixture.store.writes().lookups, + 1, + "should read before writing" + ); + assert!( + fixture.row().ids.contains_key("sharedid.org"), + "should store the sharedId the writer queued" + ); + } + + #[test] + fn due_generic_pull_provider_dispatches_after_reread_and_its_uid_lands_in_the_same_write() { + let mut fixture = Fixture::new(Some(&live_row(true)), true); + pull_answers(&fixture, "pulled-uid"); + fixture.stage_sharedid(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write once"); + let writes = fixture.store.writes(); + assert_eq!(writes.lookups, 1, "should re-read before dispatch"); + assert_eq!(writes.total(), 1, "should make one write"); + assert_eq!( + fixture.client.sent_hosts(), + [PULL_HOST], + "should call the partner" + ); + let row = fixture.row(); + let record = row + .ids + .get(PULL_SOURCE) + .expect("should store the pulled UID"); + assert_eq!(record.eids.len(), 1, "should store a one-entry record"); + assert_eq!( + record.first_uid(), + Some("pulled-uid"), + "should store the UID" + ); + assert_eq!( + record.w.as_deref(), + Some(PULL_ID), + "should name the provider" + ); + assert!( + row.ids.contains_key("sharedid.org"), + "should land the ingest too" + ); + } + + #[test] + fn generic_pull_results_join_the_single_flush() { + let mut settings = Fixture::settings(true); + settings.ec.partners.push(ec_partner( + "dsp.example.com", + "https://sync.dsp.example.com/pull", + "example-token-value-0123456789abcdef", + )); + settings.ec.identity.max_concurrency = 1; + let mut fixture = Fixture::with_settings(Some(&live_row(true)), settings); + pull_answers(&fixture, "pulled-uid"); + fixture + .client + .respond("sync.dsp.example.com", 200, br#"{"uid":"other-uid"}"#); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write once"); + assert_eq!(fixture.store.writes().total(), 1, "should make one write"); + let row = fixture.row(); + assert!( + row.ids.contains_key(PULL_SOURCE) && row.ids.contains_key("dsp.example.com"), + "should land both partners' UIDs in the one write" + ); + } + + /// A live row whose last writer flush was `age` seconds ago. + fn row_written_ago(age: u64) -> KvEntry { + let mut row = live_row(true); + row.wa = Some(current_timestamp().saturating_sub(age)); + row + } + + #[test] + fn spa_writer_waits_out_the_interval_since_the_last_write() { + let mut recent = Fixture::new(Some(&row_written_ago(10)), false); + recent.context.identity_mut().set_spa_writer(true); + recent.stage_sharedid(); + assert_eq!( + recent.run(&ScriptedLimiter::allowing()), + FlushOutcome::Skipped, + "should not write within the interval" + ); + assert_eq!(recent.store.writes().total(), 0, "should write nothing"); + + let mut unread = Fixture::new(None, false); + unread.context.identity_mut().set_spa_writer(true); + unread.stage_sharedid(); + assert_eq!( + unread.run(&ScriptedLimiter::allowing()), + FlushOutcome::Skipped, + "should not read the row for an SPA writer whose auction did not" + ); + assert_eq!(unread.store.writes().lookups, 0, "should read nothing"); + + let mut due = Fixture::new(Some(&row_written_ago(400)), false); + due.context.identity_mut().set_spa_writer(true); + due.stage_sharedid(); + let before = current_timestamp(); + assert_eq!( + due.run(&ScriptedLimiter::allowing()), + FlushOutcome::Written, + "should write once the interval has passed" + ); + assert!( + due.row().wa.is_some_and(|at| at >= before), + "should store the time of this write" + ); + } + + #[test] + fn spa_writer_runs_no_hold_mode_provider() { + let hold = || { + let mut settings = Fixture::settings(true); + settings.ec.identity.providers.insert( + PULL_ID.to_owned(), + ProviderSettings { + mode: ProviderMode::Hold, + ..ProviderSettings::default() + }, + ); + settings + }; + let mut spa = Fixture::with_settings(Some(&row_written_ago(400)), hold()); + spa.context.identity_mut().set_spa_writer(true); + pull_answers(&spa, "pulled-uid"); + + spa.run(&ScriptedLimiter::allowing()); + + assert!( + spa.client.sent_hosts().is_empty(), + "should not call a hold-mode provider" + ); + + let mut after_send = Fixture::new(Some(&row_written_ago(400)), true); + after_send.context.identity_mut().set_spa_writer(true); + pull_answers(&after_send, "pulled-uid"); + after_send.run(&ScriptedLimiter::allowing()); + assert_eq!( + after_send.client.sent_hosts(), + [PULL_HOST], + "control: should call an after-send provider" + ); + } + + #[test] + fn hold_provider_on_a_navigation_without_an_auction_runs_in_phase_four() { + let mut settings = Fixture::settings(true); + settings.ec.identity.providers.insert( + PULL_ID.to_owned(), + ProviderSettings { + mode: ProviderMode::Hold, + ..ProviderSettings::default() + }, + ); + let mut fixture = Fixture::with_settings(Some(&live_row(true)), settings); + pull_answers(&fixture, "pulled-uid"); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the result"); + assert_eq!( + fixture.client.sent_hosts(), + [PULL_HOST], + "should call the provider" + ); + } + + #[test] + fn cas_conflict_is_not_retried_as_a_whole() { + let mut fixture = Fixture::new(Some(&live_row(true)), false); + fixture.stage_sharedid(); + fixture.store.refuse_conditional_writes(TEST_EC_ID, 1); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!( + outcome, + FlushOutcome::WriteFailed { + removals: RemovalWriteOutcome::NothingToWrite + }, + "should fail without a removal retry" + ); + assert_eq!( + fixture.store.writes().if_generation_match, + 1, + "should attempt the flush once" + ); + } + + /// A row holding `PULL_SOURCE`, and a mutation with a fill, a removal of + /// that record and an opt-out of the pull provider. + fn removal_fixture() -> (Fixture, u64) { + let mut row = live_row(true); + let record = record_for_test("old-uid", Some(PULL_ID), 900); + let expected = record_revision(&record); + row.ids.insert(PULL_SOURCE.to_owned(), record); + let mut fixture = Fixture::new(Some(&row), true); + fixture.stage_sharedid(); + fixture.stage(PartnerIdChange::OwnerRemove { + source: source(PULL_SOURCE), + provider: pid(PULL_ID), + expected, + }); + let at = current_timestamp(); + fixture.context.identity_mut().mutation_mut().merge_state( + pid(PULL_ID), + ProviderStateChange { + opt_out: Some(KvOptOut { + at, + fp: "0".to_owned(), + }), + ..ProviderStateChange::default() + }, + ); + (fixture, at) + } + + #[test] + fn failed_flush_writes_its_removals_alone() { + let (mut fixture, at) = removal_fixture(); + fixture.store.refuse_conditional_writes(TEST_EC_ID, 1); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!( + outcome, + FlushOutcome::WriteFailed { + removals: RemovalWriteOutcome::Written + }, + "should write the removals alone" + ); + assert_eq!( + fixture.store.writes().if_generation_match, + 2, + "should make one flush attempt and one removal write" + ); + let row = fixture.row(); + assert!( + !row.ids.contains_key(PULL_SOURCE), + "should apply the removal" + ); + assert!( + !row.ids.contains_key("sharedid.org"), + "should drop the fill" + ); + assert_eq!( + row.ps + .get(PULL_ID) + .and_then(|state| state.o.as_ref()) + .map(|o| o.at), + Some(at), + "should store the opt-out" + ); + } + + #[test] + fn failed_flush_waits_the_per_key_window_before_its_removal_retry() { + let (mut fixture, _) = removal_fixture(); + fixture.store.refuse_conditional_writes(TEST_EC_ID, 1); + + fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!( + fixture.store.pauses(), + vec![PER_KEY_WRITE_WINDOW], + "should wait one per-key window after the failed flush" + ); + } + + #[test] + fn written_flush_never_waits() { + let (mut fixture, _) = removal_fixture(); + + fixture.run(&ScriptedLimiter::allowing()); + + assert!( + fixture.store.pauses().is_empty(), + "should not wait when the flush is written" + ); + } + + #[test] + fn a_removal_the_row_write_missed_goes_to_the_pending_key() { + let (mut fixture, _) = removal_fixture(); + fixture + .store + .refuse_conditional_writes(TEST_EC_ID, usize::MAX); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!( + outcome, + FlushOutcome::WriteFailed { + removals: RemovalWriteOutcome::Failed + }, + "should miss the row write" + ); + assert_eq!( + fixture.context.identity().removal_lost(), + 0, + "should lose nothing" + ); + assert_eq!( + fixture.context.identity().removal_deferred(), + 2, + "should defer the removal and the opt-out" + ); + let merge = load_pending(&fixture.graph, TEST_EC_ID, 0).expect("should hold the item"); + assert_eq!( + merge.mutation.removal_labels(), + vec![ + (pid(PULL_ID), None), + (pid(PULL_ID), Some(source(PULL_SOURCE))) + ], + "should hold the request's removals only" + ); + } + + #[test] + fn the_next_writer_merges_the_pending_key_once() { + let mut row = live_row(true); + let record = record_for_test("old-uid", Some(PULL_ID), 900); + let expected = record_revision(&record); + row.ids.insert(PULL_SOURCE.to_owned(), record); + let mut fixture = Fixture::new(Some(&row), false); + let mut removal = IdentityMutation::default(); + removal.add_change(PartnerIdChange::OwnerRemove { + source: source(PULL_SOURCE), + provider: pid(PULL_ID), + expected, + }); + assert!( + append_pending(&fixture.graph, TEST_EC_ID, &removal, current_timestamp()), + "should seed the pending key" + ); + + fixture.store.reset_counts(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the merge"); + assert_eq!( + fixture.store.writes().pending_lookups, + 1, + "should read the pending key once" + ); + let merged = fixture.row(); + assert!( + !merged.ids.contains_key(PULL_SOURCE), + "should apply the pending removal" + ); + let watermark = merged.pg.expect("should store the watermark"); + assert!( + load_pending(&fixture.graph, TEST_EC_ID, watermark).is_none(), + "should cover every pending item" + ); + + fixture + .context + .set_kv_snapshot(fixture.graph.load_snapshot(TEST_EC_ID)); + fixture.store.reset_counts(); + let again = fixture.run(&ScriptedLimiter::allowing()); + assert_eq!(again, FlushOutcome::Skipped, "should merge nothing twice"); + assert_eq!( + fixture.store.writes().if_generation_match, + 0, + "should write nothing twice" + ); + } + + #[test] + fn failed_removal_retry_counts_removal_lost() { + let (mut fixture, _) = removal_fixture(); + fixture + .store + .refuse_conditional_writes(TEST_EC_ID, usize::MAX); + fixture.store.fail_all_writes(&pending_key(TEST_EC_ID)); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!( + outcome, + FlushOutcome::WriteFailed { + removals: RemovalWriteOutcome::Failed + }, + "should lose the removals" + ); + assert_eq!( + fixture.store.writes().if_generation_match, + 1 + 5, + "should make one flush attempt and five removal attempts" + ); + assert_eq!( + fixture.context.identity().removal_lost(), + 2, + "should count the removal and the opt-out" + ); + } + + #[test] + fn burst_capped_provider_gets_sixty_second_backoff_in_the_write() { + let mut fixture = Fixture::new(Some(&live_row(true)), true); + pull_answers(&fixture, "pulled-uid"); + let before = current_timestamp(); + + let outcome = fixture.run(&ScriptedLimiter::capped()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the backoff"); + assert!( + fixture.client.sent_hosts().is_empty(), + "should not call the provider" + ); + let next = fixture + .row() + .ps + .get(PULL_ID) + .and_then(|state| state.n) + .expect("should store n"); + assert!( + (before + 60..=current_timestamp() + 60).contains(&next), + "should wait one 60-second window" + ); + } + + #[test] + fn missing_cluster_size_is_added_to_the_one_write() { + let mut fixture = Fixture::new(Some(&live_row(false)), false); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the size"); + assert_eq!(fixture.store.writes().total(), 1, "should write once"); + assert_eq!( + fixture + .row() + .network + .and_then(|network| network.cluster_size), + Some(1), + "should store the cluster size" + ); + } + + #[test] + fn untouched_entry_set_is_preserved_by_the_flush() { + let mut row = live_row(true); + let multi = full_record(Some("other"), example_entry_set()); + row.ids.insert("multi.example.com".to_owned(), multi); + let mut fixture = Fixture::new(Some(&row), false); + let live = fixture + .graph + .get(TEST_EC_ID) + .expect("should read") + .expect("should hold the row") + .0; + fixture.store.reset_counts(); + fixture.stage_sharedid(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the fill"); + assert_eq!( + fixture.row().ids.get("multi.example.com"), + live.ids.get("multi.example.com"), + "should keep the untouched record field for field" + ); + } + + #[test] + fn generic_pull_runs_only_on_returning_writer() { + let mut stager = Fixture::new(Some(&live_row(true)), true); + pull_answers(&stager, "pulled-uid"); + stager + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: false, + is_real_browser: true, + user_agent: None, + dnt: false, + page_url: None, + }); + + assert_eq!( + stager.run(&ScriptedLimiter::allowing()), + FlushOutcome::Skipped, + "should skip a stager" + ); + assert!( + stager.client.sent_hosts().is_empty(), + "should call no partner" + ); + } + + #[test] + fn new_visitor_reaches_pull_partners_from_second_page_load() { + let mut first = Fixture::new(None, true); + pull_answers(&first, "pulled-uid"); + let snapshot = EcKvSnapshot::Present { + ec_id: TEST_EC_ID.to_owned(), + entry: Box::new(live_row(true)), + generation: None, + }; + first + .context + .replace_with_generated(TEST_EC_ID.to_owned(), snapshot); + + assert_eq!( + first.run(&ScriptedLimiter::allowing()), + FlushOutcome::Skipped, + "should run no provider on the first navigation" + ); + assert!( + first.client.sent_hosts().is_empty(), + "should call no partner" + ); + + let mut second = Fixture::new(Some(&live_row(true)), true); + pull_answers(&second, "pulled-uid"); + assert_eq!( + second.run(&ScriptedLimiter::allowing()), + FlushOutcome::Written, + "should pull on the second navigation" + ); + assert_eq!( + second.client.sent_hosts(), + [PULL_HOST], + "should call the partner" + ); + } + + #[test] + fn failing_pull_partner_backs_off_through_ps() { + let mut fixture = Fixture::new(Some(&live_row(true)), true); + fixture.client.respond(PULL_HOST, 503, b""); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should write the backoff"); + let row = fixture.row(); + let state = row.ps.get(PULL_ID).expect("should store provider state"); + assert_eq!(state.f, 1, "should count the failure"); + assert!( + state.n.is_some_and(|next| next > current_timestamp()), + "should push n into the future" + ); + + let mut again = Fixture::new(Some(&row), true); + again.client.respond(PULL_HOST, 503, b""); + assert_eq!( + again.run(&ScriptedLimiter::allowing()), + FlushOutcome::Skipped, + "should not retry the partner before n" + ); + assert!( + again.client.sent_hosts().is_empty(), + "should call no partner" + ); + } + + #[test] + fn rate_limit_key_is_per_visitor_not_per_ip() { + let mut fixture = Fixture::new(Some(&live_row(true)), true); + pull_answers(&fixture, "pulled-uid"); + let limiter = ScriptedLimiter::allowing(); + + let _ = fixture.run(&limiter); + + let expected = format!( + "identity:{PULL_ID}:{}", + visitor_key(fixture.runtime.keys(), TEST_EC_ID) + ); + assert_eq!(*limiter.keys.borrow(), [expected], "should key by visitor"); + } + + #[test] + fn eid_persistence_denied_dispatches_no_pull_partner() { + let mut fixture = Fixture::new(Some(&live_row(true)), true); + pull_answers(&fixture, "pulled-uid"); + *fixture.context.consent_mut() = crate::test_support::tests::purpose_one_only_consent(); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Skipped, "should have nothing to do"); + assert!( + fixture.client.sent_hosts().is_empty(), + "should call no partner" + ); + } + + #[test] + fn pull_partner_with_a_stored_uid_is_not_called() { + let mut row = live_row(true); + row.ids + .insert(PULL_SOURCE.to_owned(), KvPartnerId::new("already-known")); + let mut fixture = Fixture::new(Some(&row), true); + pull_answers(&fixture, "pulled-uid"); + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Skipped, "should have nothing to do"); + assert!( + fixture.client.sent_hosts().is_empty(), + "should call no partner" + ); + assert_eq!(fixture.store.writes().lookups, 0, "should not re-read"); + } + + #[test] + fn regrant_clear_empties_ov() { + let held = "browser-held-uid"; + let mut row = live_row(true); + row.ps.insert( + PULL_ID.to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 1_000, + fp: "800".to_owned(), + }), + ov: vec![uid_hash(held)], + ..KvProviderState::default() + }, + ); + let browser = Eid { + source: PULL_SOURCE.to_owned(), + uids: vec![Uid { + id: held.to_owned(), + atype: Some(1), + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + }; + let mut fixture = Fixture::new(Some(&row), true); + let before = denials(&row, fixture.runtime.catalog()); + let mut consent = allowing_consent(); + consent.gpp = Some(GppConsent { + version: 1, + section_ids: vec![7], + eu_tcf: None, + us_sale_opt_out: Some(false), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }); + *fixture.context.consent_mut() = consent; + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should flush the clear"); + assert_eq!( + fixture.store.writes().if_generation_match, + 1, + "should ride the one write" + ); + let stored = fixture.row(); + let state = stored + .ps + .get(PULL_ID) + .expect("should keep the provider state"); + assert_eq!( + state.oc, + Some(1_000), + "should store the clear with the opt-out it read as its watermark" + ); + assert!(state.ov.is_empty(), "should empty ps.ov"); + assert!( + filter_browser_candidates(core::slice::from_ref(&browser), &before).is_empty(), + "control: before the clear the browser UID should be denied" + ); + let after = denials(&stored, fixture.runtime.catalog()); + assert_eq!( + filter_browser_candidates(core::slice::from_ref(&browser), &after), + vec![browser], + "should send the browser UID again after the clear" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/generic_pull.rs b/crates/trusted-server-core/src/ec/identity/generic_pull.rs new file mode 100644 index 0000000000..9c34b05ba7 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/generic_pull.rs @@ -0,0 +1,565 @@ +//! Today's pull sync, modelled as an identity provider. +//! +//! A pull-enabled `[[ec.partners]]` entry becomes one [`GenericPullProvider`]: +//! a `GET` of the partner's `pull_sync_url` with the EC ID, answered with one +//! UID. The coordinator owns eligibility, pacing, consent and write-back. + +use error_stack::{Report, ResultExt as _}; +use serde::Deserialize; +use sha2::{Digest as _, Sha256}; +use url::Url; + +use super::descriptor::{ConsentPolicy, ProviderDescriptor, Resolves}; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::input::ResolveInput; +use super::provider::IdentityProvider; +use super::resolution::{IdentityProviderError, Resolution, ResolvedEid, ResolvedUid}; +use crate::ec::partner::normalize_partner_source_domain; +use crate::ec::prebid_eids::is_valid_eid_uid; +use crate::ec::pull_sync::{build_pull_request_url, validated_pull_sync_url}; +use crate::error::TrustedServerError; +use crate::redacted::Redacted; +use crate::settings::EcPartner; + +/// Longest provider ID, from `^[a-z][a-z0-9_-]{0,62}$`. +const MAX_PROVIDER_ID_LENGTH: usize = 63; + +/// Readable characters kept when the ID is cut, leaving room for `-` and the +/// 8 hash characters. +const CUT_READABLE_LENGTH: usize = MAX_PROVIDER_ID_LENGTH - 9; + +/// Derives the provider ID of the pull partner with `source_domain`. +/// +/// It is `pull-` plus the normalized domain with `.` replaced by `-`. A +/// domain that already contains `-` also gets `-` and the first 8 hex digits +/// of the SHA-256 of the domain, so `a-b.example.com` and `a.b.example.com` +/// differ. An ID over 63 characters is cut to 54 readable characters, then +/// `-` and the hash. +#[must_use] +pub(crate) fn generic_pull_id(source_domain: &str) -> IdentityProviderId { + let domain = normalize_partner_source_domain(source_domain) + .unwrap_or_else(|_| source_domain.to_ascii_lowercase()); + let readable: String = format!("pull-{domain}") + .chars() + .map(|character| { + if character.is_ascii_lowercase() || character.is_ascii_digit() || character == '-' { + character + } else { + '-' + } + }) + .collect(); + let digest = Sha256::digest(domain.as_bytes()); + let hash = hex::encode(&digest[..4]); + + let id = if domain.contains('-') { + format!("{readable}-{hash}") + } else { + readable.clone() + }; + let id = if id.len() > MAX_PROVIDER_ID_LENGTH { + format!("{}-{hash}", &readable[..CUT_READABLE_LENGTH]) + } else { + id + }; + id.parse() + .expect("should derive a valid provider ID from a normalized domain") +} + +/// The descriptors of every pull-enabled partner. +/// +/// Reads no secret, so it works on unresolved deploy-time settings. +/// +/// # Errors +/// +/// Returns [`TrustedServerError::Configuration`] when a pull-enabled partner +/// cannot be described, for example a `pull_sync_allowed_domains` entry that +/// is not a hostname. +pub(crate) fn generic_pull_descriptors( + partners: &[EcPartner], +) -> Result, Report> { + partners + .iter() + .filter(|partner| partner.pull_sync_enabled) + .map(|partner| { + describe(partner).change_context(TrustedServerError::Configuration { + message: format!( + "ec.partners: pull sync of '{}' cannot be described as an identity provider", + partner.source_domain + ), + }) + }) + .collect() +} + +fn describe(partner: &EcPartner) -> Result> { + let mut resolves = Resolves::new().with_ec_id(); + for host in &partner.pull_sync_allowed_domains { + resolves = resolves.with_host(&host.trim().trim_end_matches('.').to_ascii_lowercase()); + } + let source: SourceDomain = partner.source_domain.parse()?; + ProviderDescriptor::builder(generic_pull_id(&partner.source_domain)) + .with_source(source) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::eid_persistence_only()) + .build() +} + +/// The answer a pull partner gives. +#[derive(Debug, Deserialize)] +struct PullAnswer { + uid: Option, +} + +/// One pull-enabled partner, called as an identity provider. +pub(crate) struct GenericPullProvider { + descriptor: ProviderDescriptor, + source: SourceDomain, + pull_sync_url: Option, + allowed_domains: Vec, + token: Option>, +} + +impl GenericPullProvider { + /// Builds the provider of `partner`. + /// + /// The descriptor reads only `source_domain`, `pull_sync_url` and + /// `pull_sync_allowed_domains`, which are not secret, so this works on + /// unresolved deploy-time settings. The token is kept for + /// [`request`](IdentityProvider::request) only. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Config`] when the partner cannot be + /// described. + pub(crate) fn from_partner(partner: &EcPartner) -> Result> { + let descriptor = describe(partner)?; + let source = descriptor.primary_source().clone(); + Ok(Self { + descriptor, + source, + pull_sync_url: partner.pull_sync_url.clone(), + allowed_domains: partner.pull_sync_allowed_domains.clone(), + token: partner.ts_pull_token.clone(), + }) + } +} + +impl IdentityProvider for GenericPullProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + let Some(mut url) = + validated_pull_sync_url(self.pull_sync_url.as_deref(), &self.allowed_domains) + else { + return Ok(None); + }; + let Some(token) = self + .token + .as_ref() + .filter(|token| !token.expose().trim().is_empty()) + else { + log::warn!( + "Pull sync: partner '{}' enabled but missing ts_pull_token", + self.source + ); + return Ok(None); + }; + + // The allowlist match ignores a trailing dot, and the descriptor's + // hosts carry none, so call the dotless spelling. + let host = url + .host_str() + .unwrap_or_default() + .trim_end_matches('.') + .to_owned(); + if url.host_str() != Some(host.as_str()) && url.set_host(Some(&host)).is_err() { + return Ok(None); + } + let url: Url = build_pull_request_url(url, input.ec_id()?); + Ok(Some(ProviderRequest::get(&url).with_bearer(token))) + } + + fn response( + &self, + response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + // A 404 is the partner's "no user" answer, as pull sync treats it. + if response.status() == http::StatusCode::NOT_FOUND { + return Ok(Resolution::no_match()); + } + let answer: PullAnswer = response.success_json()?; + match answer.uid { + Some(uid) if is_valid_eid_uid(&uid) => Ok(Resolution::found([ResolvedEid::new( + &self.source, + ResolvedUid::new(uid), + )])), + _ => Ok(Resolution::no_match()), + } + } +} + +#[cfg(test)] +mod tests { + use http::StatusCode; + use serde_json::json; + + use super::*; + use crate::ec::identity::test_support::{ProviderHarness, TEST_EC_ID, ec_partner}; + use crate::settings::{ProviderSettings, Settings}; + use crate::test_support::tests::create_test_settings; + + const TOKEN: &str = "example-token-value-0123456789abcdef"; + + fn partner() -> crate::settings::EcPartner { + ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + TOKEN, + ) + } + + #[test] + fn calls_the_partner_url_with_the_ec_id_and_bearer() { + let provider = GenericPullProvider::from_partner(&partner()).expect("should build"); + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + + let request = harness + .request() + .expect("should build") + .expect("should not skip"); + + assert_eq!( + request.url().host_str(), + Some("sync.ssp.example.com"), + "should call the partner host" + ); + assert_eq!( + request + .url() + .query_pairs() + .find(|(name, _)| name == "ec_id") + .map(|(_, value)| value.into_owned()), + Some(TEST_EC_ID.to_owned()), + "should pass the EC ID" + ); + assert_eq!( + request.headers()["authorization"], + format!("Bearer {TOKEN}"), + "should send the bearer token" + ); + assert_eq!(request.method(), http::Method::GET, "should use GET"); + + let resolution = harness + .respond(StatusCode::OK, &json!({ "uid": "partner-uid" })) + .expect("should parse the answer"); + assert_eq!(resolution.eids().len(), 1, "should return one entry"); + let eid = &resolution.eids()[0]; + assert_eq!( + eid.source().as_str(), + "ssp.example.com", + "should name the source" + ); + assert_eq!(eid.uids().len(), 1, "should return one UID"); + assert_eq!( + eid.uids()[0].id(), + "partner-uid", + "should return the partner's UID" + ); + assert_eq!( + ( + eid.uids()[0].atype(), + eid.uids()[0].ext(), + eid.inserter(), + eid.matcher(), + eid.mm(), + eid.ext() + ), + (None, None, None, None, None, None), + "should set nothing beyond the UID, so it is stored as one entry with one UID" + ); + } + + #[test] + fn keeps_the_query_the_partner_url_already_has() { + let partner = ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull?x=1", + TOKEN, + ); + let provider = GenericPullProvider::from_partner(&partner).expect("should build"); + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + + let request = harness + .request() + .expect("should build") + .expect("should not skip"); + + let query = request.url().query().expect("should have a query"); + assert!(query.contains("x=1"), "should keep the existing query"); + assert!(query.contains("ec_id="), "should append the EC ID"); + assert!(!query.contains("ip="), "should not forward the client IP"); + } + + #[test] + fn not_found_and_null_uid_are_no_match_as_today() { + let provider = GenericPullProvider::from_partner(&partner()).expect("should build"); + let oversized = "x".repeat(513); + for (name, status, body) in [ + ("404", StatusCode::NOT_FOUND, json!({})), + ( + "404 with a body", + StatusCode::NOT_FOUND, + json!({ "uid": "u" }), + ), + ("null uid", StatusCode::OK, json!({ "uid": null })), + ("missing uid", StatusCode::OK, json!({})), + ("empty uid", StatusCode::OK, json!({ "uid": "" })), + ("blank uid", StatusCode::OK, json!({ "uid": " " })), + ("oversized uid", StatusCode::OK, json!({ "uid": oversized })), + ] { + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + harness + .request() + .expect("should build") + .expect("should not skip"); + let resolution = harness + .respond(status, &body) + .expect("should not be an error"); + assert!(resolution.is_no_match(), "should be no_match for {name}"); + } + } + + #[test] + fn a_uid_at_the_size_limit_is_kept() { + let provider = GenericPullProvider::from_partner(&partner()).expect("should build"); + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + harness + .request() + .expect("should build") + .expect("should not skip"); + + let resolution = harness + .respond(StatusCode::OK, &json!({ "uid": "x".repeat(512) })) + .expect("should parse"); + + assert_eq!( + resolution.eids()[0].uids()[0].id().len(), + 512, + "should keep a UID of MAX_UID_LENGTH" + ); + } + + #[test] + fn other_failures_are_errors_not_no_match() { + let provider = GenericPullProvider::from_partner(&partner()).expect("should build"); + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + harness + .request() + .expect("should build") + .expect("should not skip"); + + let rejected = harness + .respond(StatusCode::INTERNAL_SERVER_ERROR, &json!({})) + .expect_err("should reject a 500"); + assert!( + matches!( + rejected.current_context(), + IdentityProviderError::Rejected { status } if *status == StatusCode::INTERNAL_SERVER_ERROR + ), + "should report Rejected" + ); + let malformed = harness + .respond_raw(StatusCode::OK, &[], b"not json") + .expect_err("should reject a body that is not JSON"); + assert!( + matches!( + malformed.current_context(), + IdentityProviderError::Malformed + ), + "should report Malformed" + ); + } + + #[test] + fn skips_without_a_valid_url_or_token() { + let mut http_url = partner(); + http_url.pull_sync_url = Some("http://sync.ssp.example.com/pull".to_owned()); + let mut other_host = partner(); + other_host.pull_sync_url = Some("https://evil.example.net/pull".to_owned()); + let mut no_url = partner(); + no_url.pull_sync_url = None; + let mut no_token = partner(); + no_token.ts_pull_token = None; + + for (name, partner) in [ + ("http scheme", http_url), + ("host outside the allowlist", other_host), + ("no URL", no_url), + ("no token", no_token), + ] { + let provider = GenericPullProvider::from_partner(&partner).expect("should build"); + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + + let request = harness.request().expect("should not be an error"); + + assert!(request.is_none(), "should skip for {name}"); + } + } + + #[test] + fn accepts_an_allowlist_match_after_normalization() { + let mut partner = partner(); + partner.pull_sync_url = Some("https://SYNC.SSP.EXAMPLE.COM./pull".to_owned()); + partner.pull_sync_allowed_domains = vec!["Sync.Ssp.Example.Com.".to_owned()]; + let provider = GenericPullProvider::from_partner(&partner).expect("should build"); + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + + let request = harness.request().expect("should build"); + + assert!(request.is_some(), "should call a host the allowlist names"); + } + + #[test] + fn a_stored_record_makes_the_source_a_refresh_and_the_call_still_goes_out() { + let provider = GenericPullProvider::from_partner(&partner()).expect("should build"); + let source = provider.descriptor().primary_source().clone(); + let stored = crate::ec::identity::test_support::stored_for_test(&source, "stored-uid"); + let mut harness = ProviderHarness::new(&provider) + .with_ec_id(TEST_EC_ID) + .with_stored(stored); + + let request = harness.request().expect("should build"); + + assert!(request.is_some(), "should refresh a stored source"); + } + + #[test] + fn the_descriptor_reads_no_secret() { + let mut partner = partner(); + partner.ts_pull_token = Some(crate::redacted::Redacted::new("secret-key-name".to_owned())); + let with_key_name = GenericPullProvider::from_partner(&partner).expect("should build"); + let with_token = GenericPullProvider::from_partner(&self::partner()).expect("should build"); + + assert_eq!( + with_key_name.descriptor(), + with_token.descriptor(), + "should describe the same provider whatever the token holds" + ); + let descriptor = with_token.descriptor(); + assert_eq!( + descriptor.id().as_str(), + "pull-ssp-example-com", + "should derive the ID" + ); + assert_eq!(descriptor.sources().len(), 1, "should claim one source"); + assert_eq!( + descriptor.primary_source().as_str(), + "ssp.example.com", + "should claim the partner's source domain" + ); + let resolves = descriptor.resolves().expect("should resolve"); + assert_eq!( + resolves.hosts(), + ["sync.ssp.example.com".to_owned()], + "should allow only the allowlisted hosts" + ); + assert!(resolves.gets_ec_id(), "should get the EC ID"); + assert!(descriptor.captures().is_none(), "should not capture"); + assert!( + descriptor.consent_policy().is_eid_persistence_only(), + "should require the EID persistence consent pull sync requires today" + ); + } + + #[test] + fn descriptors_cover_only_pull_enabled_partners() { + let mut disabled = ec_partner("off.example.com", "https://sync.off.example.com/p", TOKEN); + disabled.pull_sync_enabled = false; + let partners = [partner(), disabled]; + + let descriptors = generic_pull_descriptors(&partners).expect("should build"); + + assert_eq!( + descriptors.len(), + 1, + "should skip a partner without pull sync" + ); + assert_eq!( + descriptors[0].id().as_str(), + "pull-ssp-example-com", + "should describe the pull-enabled partner" + ); + } + + #[test] + fn provider_ids_are_derived_and_bounded() { + assert_eq!( + generic_pull_id("ssp.example.com").as_str(), + "pull-ssp-example-com", + "should keep a dot-only domain readable" + ); + let long = format!("{}.example.com", "a".repeat(70)); + let id = generic_pull_id(&long); + assert!( + id.as_str().len() <= 63, + "should bound the ID to 63 characters" + ); + assert_ne!( + id, + generic_pull_id(&format!("{}.example.com", "a".repeat(71))), + "should keep long IDs unique by hash" + ); + assert_ne!( + generic_pull_id("a-b.example.com"), + generic_pull_id("a.b.example.com"), + "should not collide when a domain already contains a hyphen" + ); + assert_eq!( + generic_pull_id("SSP.Example.Com."), + generic_pull_id("ssp.example.com"), + "should derive the ID from the normalized domain" + ); + let hyphenated_long = format!("{}-b.example.com", "a".repeat(70)); + assert!( + generic_pull_id(&hyphenated_long).as_str().len() <= 63, + "should bound a long hyphenated domain too" + ); + } + + #[test] + fn generic_pull_pacing_comes_from_its_provider_table() { + let mut settings: Settings = create_test_settings(); + let id = generic_pull_id("ssp.example.com"); + + let defaults = settings.ec.identity.provider(&id); + assert_eq!( + defaults.refresh_after_sec, 86_400, + "should default the refresh" + ); + assert_eq!( + defaults.max_calls_per_minute, 1, + "should default the pacing" + ); + + settings.ec.identity.providers.insert( + id.as_str().to_owned(), + ProviderSettings { + max_calls_per_minute: 2, + ..ProviderSettings::default() + }, + ); + let configured = settings.ec.identity.provider(&id); + assert_eq!( + configured.max_calls_per_minute, 2, + "should read the pacing from the pull provider's table" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/ids.rs b/crates/trusted-server-core/src/ec/identity/ids.rs new file mode 100644 index 0000000000..21c2516cea --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/ids.rs @@ -0,0 +1,223 @@ +//! Identifiers shared by the identity provider API. + +use core::ops::Add; +use core::str::FromStr; +use std::time::Duration; + +use error_stack::Report; +use serde::{Deserialize, Serialize}; + +use super::resolution::IdentityProviderError; +use crate::ec::partner::normalize_partner_source_domain; + +/// An identity provider's ID, such as `id5` or `pull-ssp-example-com`. +/// +/// Matches `^[a-z][a-z0-9_-]{0,62}$`, so it can be a bare TOML key and can +/// equal an integration ID that uses `_`. Separate from +/// [`crate::auction::ProviderId`]. +#[derive( + Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, Serialize, Deserialize, +)] +#[serde(try_from = "String", into = "String")] +pub struct IdentityProviderId(String); + +impl IdentityProviderId { + /// The ID as written in config. + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl FromStr for IdentityProviderId { + type Err = Report; + + fn from_str(value: &str) -> Result { + let mut bytes = value.bytes(); + let valid_first = bytes.next().is_some_and(|byte| byte.is_ascii_lowercase()); + let valid_rest = bytes.all(|byte| { + byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'_' || byte == b'-' + }); + if valid_first && valid_rest && value.len() <= 63 { + Ok(Self(value.to_owned())) + } else { + Err(IdentityProviderError::config(format!( + "identity provider id `{value}` must match ^[a-z][a-z0-9_-]{{0,62}}$" + ))) + } + } +} + +impl TryFrom for IdentityProviderId { + type Error = Report; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(id: IdentityProviderId) -> Self { + id.0 + } +} + +/// An EID `source` domain, normalized. Also the key of the KV `ids` map. +#[derive( + Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, Serialize, Deserialize, +)] +#[serde(try_from = "String", into = "String")] +pub struct SourceDomain(String); + +impl SourceDomain { + /// The normalized domain. + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } + + /// Whether a vendor's spelling of a source names this one, after + /// normalization. + #[must_use] + pub fn matches(&self, name: &str) -> bool { + normalize_partner_source_domain(name).is_ok_and(|normalized| normalized == self.0) + } +} + +impl FromStr for SourceDomain { + type Err = Report; + + fn from_str(value: &str) -> Result { + normalize_partner_source_domain(value) + .map(Self) + .map_err(IdentityProviderError::config) + } +} + +impl TryFrom for SourceDomain { + type Error = Report; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(source: SourceDomain) -> Self { + source.0 + } +} + +/// Unix seconds. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub struct UnixTime(u64); + +impl UnixTime { + /// From Unix seconds. + #[must_use] + pub fn from_secs(secs: u64) -> Self { + Self(secs) + } + + /// From Unix milliseconds, for vendors such as Lockr and `UID2`. Floors to + /// whole seconds. + #[must_use] + pub fn from_millis(millis: u64) -> Self { + Self(millis / 1000) + } + + /// As Unix seconds. + #[must_use] + pub fn as_secs(self) -> u64 { + self.0 + } +} + +impl Add for UnixTime { + type Output = UnixTime; + + fn add(self, rhs: Duration) -> UnixTime { + UnixTime(self.0.saturating_add(rhs.as_secs())) + } +} +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn provider_id_accepts_the_documented_grammar() { + for id in ["id5", "pull-ssp-example-com", "lockr_aim", "a"] { + assert!( + id.parse::().is_ok(), + "should accept `{id}`" + ); + } + let longest = format!("a{}", "b".repeat(62)); + longest + .parse::() + .expect("should accept 63 characters"); + } + + #[test] + fn provider_id_rejects_everything_else() { + let too_long = format!("a{}", "b".repeat(63)); + for id in ["", "1id", "Id5", "id 5", "id.5", too_long.as_str()] { + let error = id + .parse::() + .expect_err("should reject the id"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ), + "should be a Config error for `{id}`" + ); + } + } + + #[test] + fn source_domain_normalizes_and_matches() { + let source: SourceDomain = "ID5-Sync.com.".parse().expect("should parse"); + assert_eq!( + source.as_str(), + "id5-sync.com", + "should lower-case and drop the trailing dot" + ); + assert!( + source.matches("id5-sync.COM"), + "should match another spelling" + ); + assert!( + !source.matches("example.com"), + "should not match another domain" + ); + assert!( + !source.matches("https://id5-sync.com"), + "should not match a URL" + ); + } + + #[test] + fn source_domain_round_trips_through_serde() { + let source: SourceDomain = "example.com".parse().expect("should parse"); + let json = serde_json::to_string(&source).expect("should serialize"); + assert_eq!(json, "\"example.com\"", "should serialize as a bare string"); + let back: SourceDomain = serde_json::from_str(&json).expect("should deserialize"); + assert_eq!(back, source, "should round-trip"); + serde_json::from_str::("\"not a domain\"") + .expect_err("should reject an invalid domain on deserialize"); + } + + #[test] + fn unix_time_converts_millis_and_adds_durations() { + assert_eq!( + UnixTime::from_millis(1_760_000_000_999).as_secs(), + 1_760_000_000, + "should floor millis" + ); + let later = UnixTime::from_secs(100) + Duration::from_secs(50); + assert_eq!(later, UnixTime::from_secs(150), "should add seconds"); + let saturated = UnixTime::from_secs(u64::MAX) + Duration::from_secs(1); + assert_eq!(saturated, UnixTime::from_secs(u64::MAX), "should saturate"); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/input.rs b/crates/trusted-server-core/src/ec/identity/input.rs new file mode 100644 index 0000000000..ef64652df4 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/input.rs @@ -0,0 +1,1246 @@ +//! What a provider may read for one call. + +use core::cell::RefCell; +use core::fmt; +use std::collections::{BTreeMap, BTreeSet}; +use std::net::IpAddr; + +use base64::Engine as _; +use base64::engine::general_purpose::STANDARD; +use error_stack::Report; +use rand::RngCore as _; +use serde::de::DeserializeOwned; +use url::Url; + +use super::descriptor::{Access, Input, ProviderDescriptor, ResolveKind, Resolves}; +use super::exchange::ProviderResponse; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::resolution::IdentityProviderError; +use crate::consent::ConsentContext; +use crate::ec::kv_types::KvPartnerId; +use crate::openrtb::{Eid, Uid}; + +/// The raw consent strings a provider forwards, and `gdpr_applies`. +/// +/// Providers never decide consent. The coordinator has already applied the +/// descriptor's policy. +#[derive(Clone, Default, PartialEq, Eq)] +pub struct ProviderConsent { + gdpr_applies: bool, + tc_string: Option, + gpp_string: Option, + gpp_section_ids: Option>, + us_privacy: Option, +} + +impl fmt::Debug for ProviderConsent { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderConsent") + .field("gdpr_applies", &self.gdpr_applies) + .field("tc_string", &self.tc_string.as_ref().map(|_| "[redacted]")) + .field( + "gpp_string", + &self.gpp_string.as_ref().map(|_| "[redacted]"), + ) + .field("gpp_section_ids", &self.gpp_section_ids) + .field( + "us_privacy", + &self.us_privacy.as_ref().map(|_| "[redacted]"), + ) + .finish() + } +} + +impl ProviderConsent { + /// The view of a request's consent context. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn from_context(context: &ConsentContext) -> Self { + Self { + gdpr_applies: context.gdpr_applies, + tc_string: context.raw_tc_string.clone(), + gpp_string: context.raw_gpp_string.clone(), + gpp_section_ids: context.gpp_section_ids.clone(), + us_privacy: context.raw_us_privacy.clone(), + } + } + + /// Whether GDPR applies to this request. + #[must_use] + pub fn gdpr_applies(&self) -> bool { + self.gdpr_applies + } + + /// The raw TCF string. + #[must_use] + pub fn raw_tc_string(&self) -> Option<&str> { + self.tc_string.as_deref() + } + + /// The raw GPP string. + #[must_use] + pub fn raw_gpp_string(&self) -> Option<&str> { + self.gpp_string.as_deref() + } + + /// The applicable GPP section IDs. + #[must_use] + pub fn gpp_section_ids(&self) -> Option<&[u16]> { + self.gpp_section_ids.as_deref() + } + + /// The raw US Privacy string. + #[must_use] + pub fn raw_us_privacy(&self) -> Option<&str> { + self.us_privacy.as_deref() + } +} + +/// Country and subdivision, upper case. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Region { + country: String, + subdivision: Option, +} + +impl Region { + /// Normalizes platform geo: upper case, and an ISO 3166-2 `US-CA` style + /// region loses its country prefix. `None` without a two-letter country. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn from_geo(country: &str, region: Option<&str>) -> Option { + let country = country.trim().to_ascii_uppercase(); + if country.len() != 2 || !country.bytes().all(|byte| byte.is_ascii_alphabetic()) { + return None; + } + let subdivision = region + .map(|value| value.trim().to_ascii_uppercase()) + .map(|value| match value.split_once('-') { + Some((prefix, rest)) if prefix == country => rest.to_owned(), + _ => value, + }) + .filter(|value| { + !value.is_empty() && value.bytes().all(|byte| byte.is_ascii_alphanumeric()) + }); + Some(Self { + country, + subdivision, + }) + } + + /// ISO 3166-1 alpha-2, upper case, for example `"US"`. + #[must_use] + pub fn country(&self) -> &str { + &self.country + } + + /// The ISO 3166-2 subdivision without the country prefix, for example + /// `"CA"`. `None` when the platform does not know it. + #[must_use] + pub fn subdivision(&self) -> Option<&str> { + self.subdivision.as_deref() + } + + /// Whether this is `country` and `subdivision`. + #[must_use] + pub fn is(&self, country: &str, subdivision: &str) -> bool { + self.country == country && self.subdivision.as_deref() == Some(subdivision) + } +} + +/// The 32 raw bytes of a SHA-256 of a normalized email. Personal data. +#[derive(Clone, PartialEq, Eq)] +pub struct HashedEmail([u8; 32]); + +impl fmt::Debug for HashedEmail { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("HashedEmail([redacted])") + } +} + +impl HashedEmail { + /// From digest bytes. Test support only: v1 has no source of hashed + /// emails, so nothing else builds one. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn from_bytes(bytes: [u8; 32]) -> Self { + Self(bytes) + } + + /// From the SHA-256 digest of a normalized address. Test support only, + /// and the one constructor over [`Self::from_bytes`]. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn for_test(bytes: [u8; 32]) -> Self { + Self::from_bytes(bytes) + } + + /// The digest. + #[must_use] + pub fn as_bytes(&self) -> &[u8; 32] { + &self.0 + } + + /// Lowercase hex, as `LiveRamp` takes it. + #[must_use] + pub fn to_hex(&self) -> String { + hex::encode(self.0) + } + + /// Standard base64, as `UID2` takes it. + #[must_use] + pub fn to_base64(&self) -> String { + STANDARD.encode(self.0) + } +} + +/// A stored record for one of this provider's sources, valid or expired. +/// +/// It holds the source's whole entry set, which is never empty. +#[derive(Clone, PartialEq)] +pub struct StoredId { + eids: Vec, + uid: String, + writer: Option, + acquired_at: Option, + expires_at: Option, + refresh_at: Option, +} + +impl fmt::Debug for StoredId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("StoredId") + .field("entries", &self.eids.len()) + .field("uid", &"[redacted]") + .field("writer", &self.writer) + .field("acquired_at", &self.acquired_at) + .field("expires_at", &self.expires_at) + .field("refresh_at", &self.refresh_at) + .finish_non_exhaustive() + } +} + +impl StoredId { + /// From the KV record of `source`. A writer that does not parse is + /// treated as none. `None` for a record with no entry, which + /// `KvEntry::validate` never lets through. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn from_record(source: &SourceDomain, record: &KvPartnerId) -> Option { + let uid = record.first_uid()?.to_owned(); + let eids = record + .eids + .iter() + .map(|entry| Eid { + source: source.as_str().to_owned(), + uids: entry + .uids + .iter() + .map(|uid| Uid { + id: uid.id.clone(), + atype: uid.atype, + ext: uid.ext.clone(), + }) + .collect(), + inserter: entry.inserter.clone(), + matcher: entry.matcher.clone(), + mm: entry.mm, + ext: entry.ext.clone(), + }) + .collect(); + Some(Self { + eids, + uid, + writer: record.w.as_deref().and_then(|writer| writer.parse().ok()), + acquired_at: record.at.map(UnixTime::from_secs), + expires_at: record.exp.map(UnixTime::from_secs), + refresh_at: record.ra.map(UnixTime::from_secs), + }) + } + + fn first_uid(&self) -> Option<&Uid> { + self.eids.first().and_then(|eid| eid.uids.first()) + } + + /// A stored one-entry, one-UID record of `source`. Test support only. + /// + /// # Panics + /// + /// Panics when `uid` is not a valid stored UID. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn for_test(source: &SourceDomain, uid: &str) -> Self { + Self::from_record(source, &KvPartnerId::new(uid)) + .expect("should build a stored ID from a record with a UID") + } + + /// Replaces the entry set with a full one, and `uid()` becomes its first + /// UID. Test support only, so a test can give a refreshing provider a + /// stored full set. + /// + /// # Panics + /// + /// Panics when `eids` holds no UID. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn with_eids(mut self, eids: Vec) -> Self { + self.uid = eids + .first() + .and_then(|eid| eid.uids.first()) + .map(|uid| uid.id.clone()) + .expect("should give the stored set at least one UID"); + self.eids = eids; + self + } + + /// Records `writer` as the provider that wrote it. Test support only. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn written_by(mut self, writer: &IdentityProviderId) -> Self { + self.writer = Some(writer.clone()); + self + } + + /// Sets the time it must stop being sent. Test support only. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn expiring_at(mut self, at: UnixTime) -> Self { + self.expires_at = Some(at); + self + } + + /// Every stored entry for the source, in stored order, with its UIDs and + /// EID-level fields. A record that browser ingest, generic pull, batch + /// sync or a version 1 row wrote is one entry with one UID. + #[must_use] + pub fn eids(&self) -> &[Eid] { + &self.eids + } + + /// The first UID of the first entry. What a single-UID refresh, such as + /// `LiveRamp`'s, sends back. + #[must_use] + pub fn uid(&self) -> &str { + &self.uid + } + + /// The `atype` of that first UID. + #[must_use] + pub fn atype(&self) -> Option { + self.first_uid().and_then(|uid| uid.atype) + } + + /// The UID-level ext of that first UID, as bidders read it in + /// `uids[].ext`. + #[must_use] + pub fn ext(&self) -> Option<&serde_json::Value> { + self.first_uid().and_then(|uid| uid.ext.as_ref()) + } + + /// The provider that wrote it. `None` for browser ingest and legacy rows. + #[must_use] + pub fn writer(&self) -> Option<&IdentityProviderId> { + self.writer.as_ref() + } + + /// When it was acquired or last refreshed. + #[must_use] + pub fn acquired_at(&self) -> Option { + self.acquired_at + } + + /// When it must stop being sent. + #[must_use] + pub fn expires_at(&self) -> Option { + self.expires_at + } + + /// When it is due for refresh. + #[must_use] + pub fn refresh_at(&self) -> Option { + self.refresh_at + } + + /// Whether `now` is at or past its expiry. + #[must_use] + pub fn is_expired(&self, now: UnixTime) -> bool { + self.expires_at.is_some_and(|expires| now >= expires) + } +} + +/// A source of random bytes. +pub(crate) trait RandomSource: Send + Sync { + /// Fills `bytes`. + fn fill(&self, bytes: &mut [u8]); +} + +/// The platform generator EC generation uses (`rand::thread_rng`). +// Temporary: first used by a later identity task. +#[allow(dead_code)] +pub(crate) struct ThreadRandom; + +impl RandomSource for ThreadRandom { + fn fill(&self, bytes: &mut [u8]) { + rand::thread_rng().fill_bytes(bytes); + } +} + +/// Deterministic bytes for tests: xorshift64* from a seed. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug)] +pub struct SeededRandom(std::sync::Mutex); + +#[cfg(any(test, feature = "test-utils"))] +impl SeededRandom { + /// A generator that yields the same bytes for the same seed. A zero + /// seed is treated as one, since xorshift never leaves zero. + #[must_use] + pub fn new(seed: u64) -> Self { + Self(std::sync::Mutex::new(seed.max(1))) + } +} + +#[cfg(any(test, feature = "test-utils"))] +impl RandomSource for SeededRandom { + fn fill(&self, bytes: &mut [u8]) { + let mut state = self.0.lock().expect("should lock the seed"); + for byte in bytes { + *state ^= *state >> 12; + *state ^= *state << 25; + *state ^= *state >> 27; + *byte = state.wrapping_mul(0x2545_F491_4F6C_DD1D).to_be_bytes()[0]; + } + } +} + +/// Every signal the coordinator holds for one request. Crate-private: a +/// provider only ever sees a [`ResolveInput`] filtered by its descriptor. +pub(crate) struct ResolveSignals<'a> { + pub(crate) client_ip: Option, + pub(crate) user_agent: Option<&'a str>, + pub(crate) page_url: Option, + pub(crate) region: Option, + pub(crate) hashed_email: Option, + pub(crate) consent: ProviderConsent, + pub(crate) publisher_domain: &'a str, + pub(crate) now: UnixTime, + pub(crate) random: &'a dyn RandomSource, + pub(crate) ec_id: Option, + pub(crate) stored: BTreeMap, +} + +/// What one provider may read for one call. +pub struct ResolveInput<'a> { + descriptor: &'a ProviderDescriptor, + kinds: BTreeMap, + signals: &'a ResolveSignals<'a>, + declared: BTreeSet, + available: BTreeSet, + state_json: Option, + undeclared: RefCell>, + state_failed: RefCell, + state_decoded: RefCell, +} + +impl fmt::Debug for ResolveInput<'_> { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ResolveInput") + .field("provider", self.descriptor.id()) + .field("kinds", &self.kinds) + .finish_non_exhaustive() + } +} + +impl<'a> ResolveInput<'a> { + /// The input for one call. `kinds` holds only the sources to resolve. + /// The inputs of every kind present are available, plus the wanted ones. + /// An input declared only for a kind this call does not hold is + /// declared, so reading it is not an error of the provider, but it is + /// unavailable (spec 4.4). + #[must_use] + pub(crate) fn new( + descriptor: &'a ProviderDescriptor, + kinds: BTreeMap, + signals: &'a ResolveSignals<'a>, + ) -> Self { + let (declared, available) = + descriptor + .resolves() + .map_or_else(Default::default, |resolves| { + let mut available = resolves.wants().clone(); + for kind in kinds.values().copied().collect::>() { + available.extend(resolves.needs(kind)); + } + (resolves.declared_inputs(), available) + }); + Self { + descriptor, + kinds, + signals, + declared, + available, + state_json: None, + undeclared: RefCell::new(Vec::new()), + state_failed: RefCell::new(false), + state_decoded: RefCell::new(false), + } + } + + /// Gives the call its own provider's stored private state `ps.st`, as + /// JSON. State is per provider and sensitive, so it never rides on the + /// signals every call shares. + #[must_use] + pub(crate) fn with_state_json(mut self, state_json: Option) -> Self { + self.state_json = state_json; + self + } + + fn record(&self, access: Access) -> Report { + self.undeclared.borrow_mut().push(access); + Report::new(IdentityProviderError::UndeclaredInput { access }) + } + + fn declared( + &self, + input: Input, + value: Option, + ) -> Result> { + if !self.declared.contains(&input) { + return Err(self.record(Access::Input(input))); + } + if !self.available.contains(&input) { + return Err(Report::new(IdentityProviderError::InputUnavailable { + input, + })); + } + value.ok_or_else(|| Report::new(IdentityProviderError::InputUnavailable { input })) + } + + /// Undeclared reads recorded so far. The coordinator logs them as + /// `Config`, and the harness fails the test. + #[must_use] + pub(crate) fn recorded_errors(&self) -> Vec { + self.undeclared.borrow().clone() + } + + /// Whether a `state()` call failed to decode. + #[must_use] + pub(crate) fn state_decode_failed(&self) -> bool { + *self.state_failed.borrow() + } + + /// Whether a `state()` call decoded the stored state. The conformance + /// check of the test harness uses it to see a provider accept a state + /// that no valid state looks like. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub(crate) fn state_was_decoded(&self) -> bool { + *self.state_decoded.borrow() + } + + /// The sources to resolve this time, each with its own kind. + pub fn sources(&self) -> impl Iterator + '_ { + self.kinds.iter().map(|(source, kind)| (source, *kind)) + } + + /// Whether `source` is being resolved for the first time. + #[must_use] + pub fn is_initial(&self, source: &SourceDomain) -> bool { + self.kinds.get(source) == Some(&ResolveKind::Initial) + } + + /// Whether `source` is being refreshed. + #[must_use] + pub fn is_refresh(&self, source: &SourceDomain) -> bool { + self.kinds.get(source) == Some(&ResolveKind::Refresh) + } + + /// The descriptor's primary source. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + self.descriptor.primary_source() + } + + /// The stored record for one of this provider's sources. + #[must_use] + pub fn stored(&self, source: &SourceDomain) -> Option<&StoredId> { + self.signals.stored.get(source) + } + + /// This provider's private state, decoded. `Ok(None)` when there is none. + /// + /// # Errors + /// + /// `InvalidState` when the stored state does not decode as `T`. The + /// failure is recorded, and the coordinator clears the state whatever + /// the provider returns. `UndeclaredInput` without `with_private_state()`. + pub fn state(&self) -> Result, Report> { + if !self + .descriptor + .resolves() + .is_some_and(Resolves::has_private_state) + { + return Err(self.record(Access::PrivateState)); + } + let Some(raw) = self.state_json.as_deref() else { + return Ok(None); + }; + serde_json::from_str(raw) + .map(|state| { + *self.state_decoded.borrow_mut() = true; + Some(state) + }) + .map_err(|_| { + *self.state_failed.borrow_mut() = true; + Report::new(IdentityProviderError::InvalidState) + }) + } + + /// The endpoint declared with `Resolves::with_endpoint`. + /// + /// # Errors + /// + /// `UndeclaredInput { access: Endpoint }` when none was declared. + pub fn endpoint(&self) -> Result<&Url, Report> { + self.descriptor + .endpoint() + .ok_or_else(|| self.record(Access::Endpoint)) + } + + /// The consent strings to forward. + #[must_use] + pub fn consent(&self) -> &ProviderConsent { + &self.signals.consent + } + + /// The publisher's domain. + #[must_use] + pub fn publisher_domain(&self) -> &str { + self.signals.publisher_domain + } + + /// The request's time. + #[must_use] + pub fn now(&self) -> UnixTime { + self.signals.now + } + + /// Random bytes from the platform generator. Fixed by the harness. + #[must_use] + pub fn random_bytes(&self) -> [u8; N] { + let mut bytes = [0u8; N]; + self.signals.random.fill(&mut bytes); + bytes + } + + /// The platform's client address. + /// + /// # Errors + /// + /// `UndeclaredInput` when not declared, `InputUnavailable` when missing. + pub fn client_ip(&self) -> Result> { + self.declared(Input::ClientIp, self.signals.client_ip) + } + + /// The `User-Agent`. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn user_agent(&self) -> Result<&str, Report> { + self.declared(Input::UserAgent, self.signals.user_agent) + } + + /// The page URL, without query string or fragment. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn page_url(&self) -> Result<&Url, Report> { + self.declared(Input::PageUrl, self.signals.page_url.as_ref()) + } + + /// Country and subdivision. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn region(&self) -> Result<&Region, Report> { + self.declared(Input::Region, self.signals.region.as_ref()) + } + + /// A hashed email. Never available in v1. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn hashed_email(&self) -> Result<&HashedEmail, Report> { + self.declared(Input::HashedEmail, self.signals.hashed_email.as_ref()) + } + + /// The full EC ID. Only for a resolver declared `with_ec_id()`. + /// + /// # Errors + /// + /// `UndeclaredInput { access: EcId }` otherwise. `Config` when the + /// request has no EC ID, which the coordinator never allows: it does not + /// dispatch generic pull without one. + pub(in crate::ec::identity) fn ec_id(&self) -> Result<&str, Report> { + if !self.descriptor.resolves().is_some_and(Resolves::gets_ec_id) { + return Err(self.record(Access::EcId)); + } + self.signals.ec_id.as_deref().ok_or_else(|| { + IdentityProviderError::config("generic pull was called without an EC ID") + }) + } +} + +/// What a capture may read from one proxied response. +pub struct CaptureInput<'a> { + descriptor: &'a ProviderDescriptor, + path: &'a str, + response: &'a ProviderResponse, + stored: &'a BTreeMap, + consent: &'a ProviderConsent, + publisher_domain: &'a str, + now: UnixTime, +} + +impl fmt::Debug for CaptureInput<'_> { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("CaptureInput") + .field("provider", self.descriptor.id()) + .field("path", &self.path) + .finish_non_exhaustive() + } +} + +impl<'a> CaptureInput<'a> { + /// The input for one capture. + #[must_use] + pub(crate) fn new( + descriptor: &'a ProviderDescriptor, + path: &'a str, + response: &'a ProviderResponse, + stored: &'a BTreeMap, + consent: &'a ProviderConsent, + publisher_domain: &'a str, + now: UnixTime, + ) -> Self { + Self { + descriptor, + path, + response, + stored, + consent, + publisher_domain, + now, + } + } + + /// The proxied request path, without query string. + #[must_use] + pub fn path(&self) -> &str { + self.path + } + + /// The proxied response, decoded. + #[must_use] + pub fn response(&self) -> &ProviderResponse { + self.response + } + + /// The descriptor's primary source. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + self.descriptor.primary_source() + } + + /// Finds a declared source by a vendor's spelling. + #[must_use] + pub fn source_named(&self, name: &str) -> Option<&SourceDomain> { + self.descriptor.source_named(name) + } + + /// The stored record for a source. + #[must_use] + pub fn stored(&self, source: &SourceDomain) -> Option<&StoredId> { + self.stored.get(source) + } + + /// The request's consent, from its cookies and headers. The request + /// body is never read (spec 4.7). + #[must_use] + pub fn consent(&self) -> &ProviderConsent { + self.consent + } + + /// The publisher's domain. + #[must_use] + pub fn publisher_domain(&self) -> &str { + self.publisher_domain + } + + /// The request's time. + #[must_use] + pub fn now(&self) -> UnixTime { + self.now + } +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + use std::net::IpAddr; + + use super::*; + use crate::ec::identity::descriptor::{Captures, ProviderDescriptor, Resolves}; + use crate::ec::identity::exchange::ProviderResponse; + use crate::ec::kv_types::{KvEid, KvPartnerId, KvUid}; + + fn descriptor(resolves: Resolves) -> ProviderDescriptor { + ProviderDescriptor::builder("p".parse().expect("should parse the provider id")) + .with_source("example.com".parse().expect("should parse the source")) + .with_resolves(resolves) + .with_consent_policy(crate::ec::identity::test_support::resolver_policy()) + .build() + .expect("should build") + } + + fn signals() -> ResolveSignals<'static> { + ResolveSignals { + client_ip: Some("192.0.2.10".parse().expect("should parse ip")), + user_agent: Some("ExampleBrowser/1.0"), + page_url: None, + region: Region::from_geo("us", Some("us-ca")), + hashed_email: None, + consent: ProviderConsent::default(), + publisher_domain: "www.example.com", + now: UnixTime::from_secs(1_760_000_000), + random: &ThreadRandom, + ec_id: Some("a".repeat(64) + ".ABC123"), + stored: BTreeMap::new(), + } + } + + #[test] + fn declared_inputs_are_available_and_wanted_missing_ones_are_unavailable() { + let descriptor = descriptor( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_needs([Input::ClientIp, Input::Region]) + .with_wants([Input::PageUrl]), + ); + let signals = signals(); + let input = ResolveInput::new( + &descriptor, + kinds(&descriptor, ResolveKind::Initial), + &signals, + ); + + assert_eq!( + input.client_ip().expect("should read the ip"), + "192.0.2.10".parse::().expect("should parse the ip"), + "should return the client ip" + ); + let region = input.region().expect("should read region"); + assert_eq!( + (region.country(), region.subdivision()), + ("US", Some("CA")), + "should normalize the region" + ); + assert!(region.is("US", "CA"), "should test country and subdivision"); + let error = input.page_url().expect_err("should be unavailable"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::InputUnavailable { + input: Input::PageUrl + } + ), + "should report a declared but missing input as InputUnavailable" + ); + assert!( + input.recorded_errors().is_empty(), + "should record nothing for declared reads" + ); + } + + #[test] + fn input_declared_only_for_an_absent_kind_is_unavailable() { + let descriptor = descriptor( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_needs([Input::ClientIp]) + .with_needs_for(ResolveKind::Refresh, [Input::UserAgent]), + ); + let signals = signals(); + + let initial = ResolveInput::new( + &descriptor, + kinds(&descriptor, ResolveKind::Initial), + &signals, + ); + let refresh = ResolveInput::new( + &descriptor, + kinds(&descriptor, ResolveKind::Refresh), + &signals, + ); + + let error = initial + .user_agent() + .expect_err("should hide a Refresh-only input on an Initial call"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::InputUnavailable { + input: Input::UserAgent + } + ), + "should report InputUnavailable, not an undeclared read" + ); + assert!( + initial.recorded_errors().is_empty(), + "should record no undeclared read" + ); + assert_eq!( + refresh + .user_agent() + .expect("should read the user agent on a Refresh call"), + "ExampleBrowser/1.0", + "should return the user agent for the kind that declares it" + ); + } + + #[test] + fn undeclared_reads_are_recorded_even_when_ignored() { + let descriptor = descriptor(Resolves::new().with_endpoint("https://a.example.com")); + let signals = signals(); + let input = ResolveInput::new( + &descriptor, + kinds(&descriptor, ResolveKind::Initial), + &signals, + ); + + let ignored = input.user_agent().ok(); + let state = input.state::(); + + assert!(ignored.is_none(), "should not return an undeclared input"); + assert!(state.is_err(), "should need with_private_state for state"); + assert_eq!( + input.recorded_errors(), + vec![Access::Input(Input::UserAgent), Access::PrivateState], + "should record both undeclared reads" + ); + input + .endpoint() + .expect("should always return the declared endpoint"); + } + + #[test] + fn state_decodes_and_a_bad_state_is_recorded() { + let descriptor = descriptor( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_private_state(), + ); + let good = signals(); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Refresh), &good) + .with_state_json(Some(r#"{"signature":"s"}"#.to_owned())); + let state: serde_json::Value = input + .state() + .expect("should decode the state") + .expect("should have a state"); + assert_eq!(state["signature"], "s", "should decode the state"); + assert!( + !input.state_decode_failed(), + "should not record a good state as a failure" + ); + + let bad = signals(); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Refresh), &bad) + .with_state_json(Some(r#""a bare string""#.to_owned())); + let error = input + .state::() + .expect_err("should not decode a bare string"); + assert!( + matches!(error.current_context(), IdentityProviderError::InvalidState), + "should report InvalidState" + ); + assert!(input.state_decode_failed(), "should record the failure"); + } + + #[test] + fn ec_id_is_only_for_providers_declared_with_ec_id() { + let generic = descriptor(Resolves::new().with_host("a.example.com").with_ec_id()); + let other = descriptor(Resolves::new().with_host("a.example.com")); + let with_ec = signals(); + ResolveInput::new(&generic, kinds(&generic, ResolveKind::Initial), &with_ec) + .ec_id() + .expect("should let generic pull read the EC ID"); + let input = ResolveInput::new(&other, kinds(&other, ResolveKind::Initial), &with_ec); + assert!( + input.ec_id().is_err(), + "should refuse the EC ID to other providers" + ); + assert_eq!( + input.recorded_errors(), + vec![Access::EcId], + "should record an EC ID read" + ); + + let mut no_ec = signals(); + no_ec.ec_id = None; + let error = ResolveInput::new(&generic, kinds(&generic, ResolveKind::Initial), &no_ec) + .ec_id() + .expect_err("should fail without an EC ID"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ), + "should report a missing EC ID as Config, not as a missing input" + ); + } + + #[test] + fn stored_ids_and_expiry() { + let record = KvPartnerId { + w: Some("p".to_owned()), + rv: Some("0123456789abcdef".to_owned()), + at: Some(10), + exp: Some(20), + ra: Some(15), + ru: None, + eids: vec![KvEid { + uids: vec![KvUid { + id: "u".to_owned(), + atype: Some(3), + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + }], + }; + let source: SourceDomain = "example.com".parse().expect("should parse the source"); + let stored = + StoredId::from_record(&source, &record).expect("should read a record with an entry"); + assert_eq!( + stored.writer().map(IdentityProviderId::as_str), + Some("p"), + "should parse the writer" + ); + assert!( + !stored.is_expired(UnixTime::from_secs(19)), + "should not be expired before exp" + ); + assert!( + stored.is_expired(UnixTime::from_secs(20)), + "should be expired at exp" + ); + assert_eq!( + stored.refresh_at(), + Some(UnixTime::from_secs(15)), + "should read ra" + ); + assert_eq!( + stored.eids().len(), + 1, + "should expose a one-UID record as one entry" + ); + let eid = &stored.eids()[0]; + assert_eq!(eid.source, "example.com", "should fill in the source"); + assert_eq!(eid.uids.len(), 1, "should expose its one UID"); + assert_eq!(eid.uids[0].id, "u", "should keep the UID"); + assert_eq!(eid.uids[0].atype, Some(3), "should keep the atype"); + assert_eq!( + ( + eid.inserter.as_deref(), + eid.matcher.as_deref(), + eid.mm, + eid.ext.as_ref() + ), + (None, None, None, None), + "should expose no EID-level fields when none are stored" + ); + assert_eq!( + (stored.uid(), stored.atype()), + ("u", Some(3)), + "should describe the first UID" + ); + + let empty = KvPartnerId { + eids: Vec::new(), + ..record + }; + assert!( + StoredId::from_record(&source, &empty).is_none(), + "should not read a record with no entry" + ); + } + + #[test] + fn a_full_record_exposes_every_entry_and_its_first_uid() { + let record = KvPartnerId { + w: Some("p".to_owned()), + rv: Some("0123456789abcdef".to_owned()), + at: Some(10), + exp: None, + ra: None, + ru: None, + eids: vec![ + KvEid { + uids: vec![ + KvUid { + id: "first".to_owned(), + atype: Some(1), + ext: Some(serde_json::json!({ "rtiPartner": "TDID" })), + }, + KvUid { + id: "second".to_owned(), + atype: Some(3), + ext: None, + }, + ], + inserter: Some("inserter.example.com".to_owned()), + matcher: None, + mm: Some(2), + ext: Some(serde_json::json!({ "segments": ["a"] })), + }, + KvEid { + uids: vec![KvUid { + id: "third".to_owned(), + atype: None, + ext: None, + }], + inserter: None, + matcher: Some("matcher.example.com".to_owned()), + mm: None, + ext: None, + }, + ], + }; + let source: SourceDomain = "example.com".parse().expect("should parse the source"); + + let stored = StoredId::from_record(&source, &record).expect("should read the record"); + + let ids: Vec> = stored + .eids() + .iter() + .map(|eid| eid.uids.iter().map(|uid| uid.id.as_str()).collect()) + .collect(); + assert_eq!( + ids, + vec![vec!["first", "second"], vec!["third"]], + "should keep every entry in stored order" + ); + assert!( + stored.eids().iter().all(|eid| eid.source == "example.com"), + "should fill in the source" + ); + assert_eq!( + stored.eids()[0].inserter.as_deref(), + Some("inserter.example.com"), + "should keep the inserter" + ); + assert_eq!(stored.eids()[0].mm, Some(2), "should keep mm"); + assert_eq!( + stored.eids()[0].ext, + Some(serde_json::json!({ "segments": ["a"] })), + "should keep the EID-level ext" + ); + assert_eq!( + stored.eids()[1].matcher.as_deref(), + Some("matcher.example.com"), + "should keep the matcher" + ); + assert_eq!( + stored.uid(), + "first", + "should return the first UID of the first entry" + ); + assert_eq!(stored.atype(), Some(1), "should return that UID's atype"); + assert_eq!( + stored.ext(), + Some(&serde_json::json!({ "rtiPartner": "TDID" })), + "should return that UID's ext" + ); + assert!( + !format!("{stored:?}").contains("second"), + "should redact every UID in Debug" + ); + } + + #[test] + fn hashed_email_debug_is_redacted() { + let email = HashedEmail::from_bytes([0xab; 32]); + assert_eq!( + email.to_hex(), + "ab".repeat(32), + "should encode lowercase hex" + ); + assert_eq!( + format!("{email:?}"), + "HashedEmail([redacted])", + "should redact Debug" + ); + } + + #[test] + fn capture_input_exposes_path_and_never_the_request_body() { + let descriptor = + ProviderDescriptor::builder("c".parse().expect("should parse the provider id")) + .with_source("example.com".parse().expect("should parse the source")) + .with_captures(Captures::new().with_path(http::Method::POST, "/integrations/c/api")) + .build() + .expect("should build"); + let response = ProviderResponse::for_test(http::StatusCode::OK, b"{}"); + let consent = ProviderConsent::default(); + let stored = BTreeMap::new(); + let input = CaptureInput::new( + &descriptor, + "/integrations/c/api", + &response, + &stored, + &consent, + "www.example.com", + UnixTime::from_secs(1), + ); + assert_eq!( + input.path(), + "/integrations/c/api", + "should expose the proxied path" + ); + assert_eq!( + input.primary_source().as_str(), + "example.com", + "should expose the primary source" + ); + } + + fn kinds( + descriptor: &ProviderDescriptor, + kind: ResolveKind, + ) -> BTreeMap { + descriptor + .sources() + .iter() + .map(|source| (source.clone(), kind)) + .collect() + } + + #[derive(Debug, serde::Deserialize)] + struct TypedState { + #[allow(dead_code)] + signature: String, + } +} diff --git a/crates/trusted-server-core/src/ec/identity/measurement.rs b/crates/trusted-server-core/src/ec/identity/measurement.rs new file mode 100644 index 0000000000..b9b4edd844 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/measurement.rs @@ -0,0 +1,267 @@ +//! The one `identity request measurement` line per request (spec 14, step 1). +//! +//! It carries sizes, counts and labels only: no EC ID, no UID and no cookie +//! value can reach it (spec 9.3). + +use core::fmt::Write as _; + +use std::collections::BTreeMap; + +use super::flush::FlushOutcome; +use super::ids::{IdentityProviderId, SourceDomain}; + +/// What kind of identity route the request was. +#[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] +pub enum RouteKind { + /// The one browser-driven writer request. + #[display("writer")] + Writer, + /// Any other request: it reads and stages but never writes after send. + #[display("stager")] + Stager, + /// The request generated a new EC. + #[display("new_ec")] + NewEc, +} + +/// What the adapter knows about the request after phase 4. +#[derive(Debug, Clone, Copy)] +pub struct MeasurementInput<'a> { + /// Whether phase 1 read the identity row. + pub phase_one_read: bool, + /// KV reads made after phase 1. + pub kv_reads: u32, + /// KV writes made after phase 1. + pub kv_writes: u32, + /// How phase 4 ended, or `None` when it did not run. + pub flush: Option<&'a FlushOutcome>, + /// Whether a withdrawal tombstone needed its after-send run. + pub withdrawal_after_send: bool, + /// Whether that run left the tombstone unwritten. + pub withdrawal_after_send_failed: bool, +} + +/// The measurement of one request, formatted by [`core::fmt::Display`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct IdentityMeasurement { + pub(crate) cookie_header_bytes: usize, + pub(crate) route: RouteKind, + pub(crate) kv_reads: u32, + pub(crate) kv_writes: u32, + pub(crate) flush: Option, + pub(crate) removal_lost: u32, + pub(crate) removal_deferred: u32, + pub(crate) withdrawal_after_send: bool, + pub(crate) withdrawal_after_send_failed: bool, + pub(crate) capture_unpersisted: BTreeMap<(IdentityProviderId, SourceDomain), u32>, +} + +impl IdentityMeasurement { + /// The inbound `Cookie` header size in bytes. + #[must_use] + pub fn cookie_header_bytes(&self) -> usize { + self.cookie_header_bytes + } + + /// The route kind label. + #[must_use] + pub fn route(&self) -> RouteKind { + self.route + } + + /// KV reads this request made. + #[must_use] + pub fn kv_reads(&self) -> u32 { + self.kv_reads + } + + /// KV writes this request made. + #[must_use] + pub fn kv_writes(&self) -> u32 { + self.kv_writes + } + + /// How phase 4 ended, or `None` when it did not run. + #[must_use] + pub fn flush(&self) -> Option { + self.flush + } + + /// Removals every attempt failed to write. + #[must_use] + pub fn removal_lost(&self) -> u32 { + self.removal_lost + } + + /// Removals the row write missed that went to the pending key. + #[must_use] + pub fn removal_deferred(&self) -> u32 { + self.removal_deferred + } + + /// Whether a withdrawal tombstone needed its after-send run. + #[must_use] + pub fn withdrawal_after_send(&self) -> bool { + self.withdrawal_after_send + } + + /// Whether the after-send withdrawal run left the tombstone unwritten. + #[must_use] + pub fn withdrawal_after_send_failed(&self) -> bool { + self.withdrawal_after_send_failed + } + + /// Positive capture IDs counted and never stored, per provider and + /// source. Counts only: no value is held. + #[must_use] + pub fn capture_unpersisted(&self) -> &BTreeMap<(IdentityProviderId, SourceDomain), u32> { + &self.capture_unpersisted + } + + /// Logs the measurement at info level. + pub fn log(&self) { + log::info!("identity request measurement: {self}"); + } +} + +impl core::fmt::Display for IdentityMeasurement { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + write!( + formatter, + "cookie_header_bytes={} route={} kv_reads={} kv_writes={} flush={} removal_lost={} \ + removal_deferred={} withdrawal_after_send={} withdrawal_after_send_failed={} capture_unpersisted={}", + self.cookie_header_bytes, + self.route, + self.kv_reads, + self.kv_writes, + FlushLabel(self.flush), + self.removal_lost, + self.removal_deferred, + u8::from(self.withdrawal_after_send), + u8::from(self.withdrawal_after_send_failed), + CaptureUnpersisted(&self.capture_unpersisted), + ) + } +} + +/// The capture counts as `provider/source:count`, comma separated, or `none`. +struct CaptureUnpersisted<'a>(&'a BTreeMap<(IdentityProviderId, SourceDomain), u32>); + +impl core::fmt::Display for CaptureUnpersisted<'_> { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + if self.0.is_empty() { + return formatter.write_str("none"); + } + for (index, ((provider, source), count)) in self.0.iter().enumerate() { + if index > 0 { + formatter.write_char(',')?; + } + write!(formatter, "{provider}/{source}:{count}")?; + } + Ok(()) + } +} + +/// A phase 4 outcome as a lower snake case label, with a drop reason +/// appended. +pub(crate) struct FlushLabel(pub(crate) Option); + +impl core::fmt::Display for FlushLabel { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + match self.0 { + None => formatter.write_str("not_run"), + Some(FlushOutcome::Skipped) => formatter.write_str("skipped"), + Some(FlushOutcome::NoChange) => formatter.write_str("no_change"), + Some(FlushOutcome::Written) => formatter.write_str("written"), + Some(FlushOutcome::Dropped(reason)) => { + formatter.write_str("dropped_")?; + reason.chars().try_for_each(|character| { + formatter.write_char(if character == ' ' { '_' } else { character }) + }) + } + Some(FlushOutcome::WriteFailed { .. }) => formatter.write_str("write_failed"), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::removal::RemovalWriteOutcome; + + #[test] + fn flush_label_covers_every_outcome() { + let cases = [ + (None, "not_run"), + (Some(FlushOutcome::Skipped), "skipped"), + (Some(FlushOutcome::NoChange), "no_change"), + (Some(FlushOutcome::Written), "written"), + ( + Some(FlushOutcome::Dropped("read failed")), + "dropped_read_failed", + ), + ( + Some(FlushOutcome::WriteFailed { + removals: RemovalWriteOutcome::Failed, + }), + "write_failed", + ), + ]; + + for (outcome, expected) in cases { + assert_eq!( + FlushLabel(outcome).to_string(), + expected, + "should label {outcome:?}" + ); + } + } + + fn measurement( + capture_unpersisted: BTreeMap<(IdentityProviderId, SourceDomain), u32>, + ) -> IdentityMeasurement { + IdentityMeasurement { + cookie_header_bytes: 10, + route: RouteKind::Stager, + kv_reads: 0, + kv_writes: 0, + flush: None, + removal_lost: 0, + removal_deferred: 0, + withdrawal_after_send: false, + withdrawal_after_send_failed: false, + capture_unpersisted, + } + } + + #[test] + fn measurement_line_counts_capture_unpersisted_per_provider_and_source() { + let line = measurement(BTreeMap::from([ + ( + ( + "lockr".parse().expect("should parse the provider ID"), + "a.example.com".parse().expect("should parse the source"), + ), + 2, + ), + ( + ( + "lockr".parse().expect("should parse the provider ID"), + "b.example.com".parse().expect("should parse the source"), + ), + 1, + ), + ])) + .to_string(); + let empty = measurement(BTreeMap::new()).to_string(); + + assert!( + line.ends_with("capture_unpersisted=lockr/a.example.com:2,lockr/b.example.com:1"), + "should list each provider and source with its count: {line}" + ); + assert!( + empty.ends_with("capture_unpersisted=none"), + "control: should print none without a capture: {empty}" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mirror.rs b/crates/trusted-server-core/src/ec/identity/mirror.rs new file mode 100644 index 0000000000..35937cfffb --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/mirror.rs @@ -0,0 +1,282 @@ +//! The `ts-eids-s` mirror payload: the servable server set for page scripts, +//! capped to one cookie (spec 6.2). The server never reads the cookie back. + +use base64::Engine as _; +use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use serde_json::json; + +use super::ids::IdentityProviderId; +use crate::openrtb::Eid; + +/// Maximum bytes of the encoded cookie value. +pub(crate) const MIRROR_MAX_VALUE_BYTES: usize = 2048; +/// Maximum cookie lifetime: that of `ts-eids`. +pub(crate) const MIRROR_MAX_AGE_SEC: u64 = 86_400; + +/// One source's servable server entries, as bidders receive them. +#[derive(Debug, Clone)] +pub(crate) struct MirrorSource { + pub(crate) source: String, + /// The provider whose `cookie_priority` ranks the source, for the log. + pub(crate) provider: Option, + pub(crate) priority: i32, + /// The source's earliest `exp`, in Unix seconds. + pub(crate) expires_at: Option, + pub(crate) eids: Vec, +} + +/// What the request's KV read gave the mirror. +#[derive(Debug, Clone, Copy)] +pub(crate) enum MirrorRead<'a> { + /// No KV read: the response carries no mirror member. + NotRead, + /// Consent denies EID persistence, or the row is a tombstone, withdrawn + /// or missing. + Cleared, + /// The servable server set, in output order. + Servable(&'a [MirrorSource]), +} + +/// The mirror member of an `/auction` or `/_ts/page-bids` response. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum MirrorPayload { + Value { value: String, max_age: u64 }, + Clear, +} + +impl MirrorPayload { + /// The JSON the shim reads. + #[must_use] + pub(crate) fn to_json(&self) -> serde_json::Value { + match self { + Self::Value { value, max_age } => json!({ "value": value, "max_age": max_age }), + Self::Clear => json!({ "clear": true }), + } + } +} + +fn encode(sources: &[&MirrorSource]) -> String { + let eids: Vec<&Eid> = sources.iter().flat_map(|source| &source.eids).collect(); + let bytes = + serde_json::to_vec(&json!({ "v": 1, "eids": eids })).expect("should serialize EIDs"); + URL_SAFE_NO_PAD.encode(bytes) +} + +fn source_bytes(source: &MirrorSource) -> usize { + serde_json::to_vec(&source.eids).map_or(usize::MAX, |bytes| bytes.len()) +} + +/// The mirror payload for a request (spec 6.2). +/// +/// # Performance +/// +/// Encodes the set once per omission. A set holds at most one entry set per +/// configured source, so this stays small. +#[must_use] +pub(crate) fn mirror_payload(read: MirrorRead<'_>, now: u64) -> Option { + let sources = match read { + MirrorRead::NotRead => return None, + MirrorRead::Cleared => return Some(MirrorPayload::Clear), + MirrorRead::Servable(sources) => sources, + }; + let mut kept: Vec<&MirrorSource> = sources.iter().collect(); + let mut value = encode(&kept); + while value.len() > MIRROR_MAX_VALUE_BYTES && !kept.is_empty() { + let index = (0..kept.len()) + .min_by_key(|&index| { + ( + kept[index].priority, + core::cmp::Reverse(source_bytes(kept[index])), + core::cmp::Reverse(index), + ) + }) + .expect("should hold a source"); + let left_out = kept.remove(index); + log::info!( + "Leaving source '{}' of provider '{}' out of the ts-eids-s mirror, which exceeds {MIRROR_MAX_VALUE_BYTES} bytes", + left_out.source, + left_out + .provider + .as_ref() + .map_or("none", IdentityProviderId::as_str), + ); + value = encode(&kept); + } + if kept.is_empty() { + return Some(MirrorPayload::Clear); + } + let max_age = kept + .iter() + .filter_map(|source| source.expires_at) + .map(|exp| exp.saturating_sub(now).max(1)) + .fold(MIRROR_MAX_AGE_SEC, u64::min); + Some(MirrorPayload::Value { value, max_age }) +} + +#[cfg(test)] +mod tests { + use base64::Engine as _; + use base64::engine::general_purpose::URL_SAFE_NO_PAD; + use serde_json::json; + + use super::*; + use crate::openrtb::{Eid, Uid}; + + const NOW: u64 = 1_760_000_000; + + fn source(name: &str, priority: i32, expires_at: Option, ids: &[&str]) -> MirrorSource { + MirrorSource { + source: name.to_owned(), + provider: Some("p".parse().expect("should parse the provider ID")), + priority, + expires_at, + eids: vec![Eid::new( + name.to_owned(), + ids.iter() + .map(|id| Uid { + id: (*id).to_owned(), + atype: Some(1), + ext: None, + }) + .collect(), + )], + } + } + + fn decode(payload: &MirrorPayload) -> (serde_json::Value, u64) { + let MirrorPayload::Value { value, max_age } = payload else { + panic!("should hold a value"); + }; + let bytes = URL_SAFE_NO_PAD + .decode(value) + .expect("should be unpadded base64url"); + ( + serde_json::from_slice(&bytes).expect("should be JSON"), + *max_age, + ) + } + + #[test] + fn no_read_is_absent_and_a_denial_or_empty_set_is_clear() { + assert_eq!( + mirror_payload(MirrorRead::NotRead, NOW), + None, + "should leave the cookie as it is" + ); + assert_eq!( + mirror_payload(MirrorRead::Cleared, NOW), + Some(MirrorPayload::Clear), + "should clear on a denial" + ); + assert_eq!( + mirror_payload(MirrorRead::Servable(&[]), NOW), + Some(MirrorPayload::Clear), + "should clear an empty set" + ); + assert_eq!( + MirrorPayload::Clear.to_json(), + json!({ "clear": true }), + "should serialize the clear shape" + ); + } + + #[test] + fn value_is_the_v1_entries_in_unpadded_base64url() { + let sources = [source("example.com", 0, None, &["example-a"])]; + + let payload = + mirror_payload(MirrorRead::Servable(&sources), NOW).expect("should return a payload"); + + let (decoded, max_age) = decode(&payload); + assert_eq!( + decoded, + json!({ "v": 1, "eids": [{ "source": "example.com", "uids": [{ "id": "example-a", "atype": 1 }] }] }), + "should hold the entries exactly as bidders receive them" + ); + assert_eq!( + max_age, MIRROR_MAX_AGE_SEC, + "should live a day without an exp" + ); + let MirrorPayload::Value { value, .. } = &payload else { + panic!("should hold a value") + }; + assert_eq!( + payload.to_json(), + json!({ "value": value, "max_age": MIRROR_MAX_AGE_SEC }), + "should serialize" + ); + } + + #[test] + fn max_age_never_outlives_the_earliest_exp() { + let sources = [ + source("a.example.com", 0, Some(NOW + 3_600), &["a"]), + source("b.example.com", 0, Some(NOW + 600), &["b"]), + source("c.example.com", 0, Some(NOW + 999_999), &["c"]), + ]; + + let (_, max_age) = decode( + &mirror_payload(MirrorRead::Servable(&sources), NOW).expect("should return a payload"), + ); + + assert_eq!(max_age, 600, "should take the earliest exp"); + } + + #[test] + fn an_oversized_set_drops_whole_sources_lowest_priority_then_largest() { + let big = |name: &str, priority, count| { + let ids: Vec = (0..count) + .map(|index| format!("{name}-{index:040}")) + .collect(); + source( + name, + priority, + None, + &ids.iter().map(String::as_str).collect::>(), + ) + }; + let sources = [ + big("keep-high", 10, 6), + big("drop-low-large", 0, 12), + big("keep-low-small", 0, 6), + big("keep-mid", 5, 6), + ]; + + let payload = + mirror_payload(MirrorRead::Servable(&sources), NOW).expect("should return a payload"); + + let MirrorPayload::Value { value, .. } = &payload else { + panic!("should hold a value") + }; + assert!(value.len() <= MIRROR_MAX_VALUE_BYTES, "should fit the cap"); + let (decoded, _) = decode(&payload); + let kept: Vec<&str> = decoded["eids"] + .as_array() + .expect("should be an array") + .iter() + .map(|eid| eid["source"].as_str().expect("should name the source")) + .collect(); + assert_eq!( + kept, + vec!["keep-high", "keep-low-small", "keep-mid"], + "should drop the largest lowest-priority source whole" + ); + } + + #[test] + fn a_set_where_nothing_fits_is_clear() { + let ids: Vec = (0..80).map(|index| format!("only-{index:040}")).collect(); + let sources = [source( + "only.example.com", + 0, + None, + &ids.iter().map(String::as_str).collect::>(), + )]; + + assert_eq!( + mirror_payload(MirrorRead::Servable(&sources), NOW), + Some(MirrorPayload::Clear), + "should clear rather than keep an empty or partial set" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs new file mode 100644 index 0000000000..cd940b6863 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -0,0 +1,46 @@ +//! Server-side identity providers. +//! +//! An [`IdentityProvider`] turns request signals into `OpenRTB` EIDs. +//! Providers only build requests and parse buffered responses. The +//! coordinator owns eligibility, consent, transport limits and KV +//! write-back, so every page load makes at most one browser-driven KV +//! write. See `docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md`. + +pub(crate) mod catalog; +pub(crate) mod coordinator; +pub(crate) mod denials; +pub mod descriptor; +#[cfg(test)] +pub(crate) mod examples; +pub mod exchange; +pub mod flush; +pub(crate) mod generic_pull; +pub mod ids; +pub mod input; +pub mod measurement; +pub(crate) mod mirror; +pub(crate) mod mutation; +pub(crate) mod pending; +pub(crate) mod policy; +pub mod provider; +pub(crate) mod removal; +pub(crate) mod request; +pub mod resolution; +pub(crate) mod revision; +pub mod runtime; +pub(crate) mod servable; +#[cfg(any(test, feature = "test-utils"))] +pub mod test_support; + +pub use descriptor::{ + Access, Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, + ProviderDescriptorBuilder, ResolveKind, Resolves, UsOptOut, +}; +pub use exchange::{ProviderRequest, ProviderResponse}; +pub use ids::{IdentityProviderId, SourceDomain, UnixTime}; +pub use input::{CaptureInput, HashedEmail, ProviderConsent, Region, ResolveInput, StoredId}; +pub use provider::{IdentityProvider, IdentityProviderConfig}; +pub use resolution::{ + IdentityProviderError, Resolution, ResolvedEid, ResolvedUid, StateChange, When, +}; +pub use runtime::IdentityRuntime; diff --git a/crates/trusted-server-core/src/ec/identity/mutation.rs b/crates/trusted-server-core/src/ec/identity/mutation.rs new file mode 100644 index 0000000000..f074429c68 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/mutation.rs @@ -0,0 +1,2340 @@ +//! One request's identity changes as a single mutation, and its merge into +//! the live KV row (spec 5.4). + +use std::collections::{BTreeMap, BTreeSet}; + +use super::catalog::ProviderCatalog; +use super::descriptor::Resolves; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::input::RandomSource; +use super::revision::{ + Revision, StateRevision, new_revision, push_hashes, record_revision, set_hashes, +}; +use crate::ec::kv_types::{ + KvEntry, KvNetwork, KvOptOut, KvPartnerId, KvProviderState, MAX_INVALIDATED_UIDS, + MAX_PROVIDER_STATES, +}; + +/// A change to one source's record (spec 5.4). +/// +/// Each `record` is the source's whole record: its `eids`, `w`, the timing +/// fields and `rv`. Every change writes or removes a source's set whole. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum PartnerIdChange { + /// From browser ingest (`record.w` is `None`) or an unowned provider + /// (`record.w` names it). Fills a source that holds nothing usable. + FillMissing { + source: SourceDomain, + record: KvPartnerId, + }, + /// From the owning provider, whose ID `record.w` holds. + OwnerWrite { + source: SourceDomain, + record: KvPartnerId, + expected: Option, + }, + /// From `Resolution::invalidate` or `also_invalidate`. `provider` is the + /// provider that invalidated, whose `ps.iv` receives the hashes. + OwnerRemove { + source: SourceDomain, + provider: IdentityProviderId, + expected: Revision, + }, + /// From `Resolution::opted_out`. Removes the source's record if `writer` + /// wrote it, at any revision, and for a capture-only `writer` also a + /// record no provider wrote. Its hashes go to the writer's `ps.ov`. + WriterRemove { + source: SourceDomain, + writer: IdentityProviderId, + }, +} + +impl PartnerIdChange { + /// Whether the change removes a record. + pub(crate) fn is_removal(&self) -> bool { + matches!(self, Self::OwnerRemove { .. } | Self::WriterRemove { .. }) + } +} + +/// What a change does to a provider's private state `ps.st`. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) enum StateUpdate { + /// Leave `st` as it is. + #[default] + Keep, + /// Replace `st`. + Replace(serde_json::Value), + /// Remove `st`. + Clear, +} + +/// A change to one provider's `ps` entry. It has no `iv` or `ov` field: +/// those hashes come only from applied removals. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct ProviderStateChange { + pub(crate) n: Option, + pub(crate) f: Option, + pub(crate) opt_out: Option, + pub(crate) opt_out_clear: Option, + pub(crate) st: StateUpdate, +} + +impl ProviderStateChange { + // Temporary: first used by a later identity task. + #[allow(dead_code)] + fn is_noop(&self) -> bool { + *self == Self::default() + } + + // Temporary: first used by a later identity task. + #[allow(dead_code)] + fn has_removal(&self) -> bool { + self.opt_out.is_some() || self.opt_out_clear.is_some() + } + + // Temporary: first used by a later identity task. + #[allow(dead_code)] + fn removals(&self) -> Self { + Self { + opt_out: self.opt_out.clone(), + opt_out_clear: self.opt_out_clear, + ..Self::default() + } + } + + // Temporary: first used by a later identity task. + #[allow(dead_code)] + fn merge(&mut self, later: Self) { + self.n = self.n.max(later.n); + self.f = later.f.or(self.f); + if later.opt_out.as_ref().map(|o| o.at) >= self.opt_out.as_ref().map(|o| o.at) { + self.opt_out = later.opt_out.or_else(|| self.opt_out.take()); + } + self.opt_out_clear = self.opt_out_clear.max(later.opt_out_clear); + if later.st != StateUpdate::Keep { + self.st = later.st; + } + } +} + +/// One provider call's outcome. It applies as a unit: every ID change must +/// apply on its own and the state revision must still match. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct ProviderResult { + pub(crate) provider: IdentityProviderId, + /// Positive changes only. Removals go to the mutation's own list. + pub(crate) ids: Vec, + pub(crate) state: ProviderStateChange, + /// `None` means the call read no state, so the state check always passes. + pub(crate) expected_state: Option, +} + +/// Everything one request changes in the identity row. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct IdentityMutation { + partner_ids: Vec, + provider_state: BTreeMap, + results: Vec, + cluster_size: Option, + /// Providers whose stand-alone state change comes from a `found` outcome. + found_states: BTreeSet, +} + +impl IdentityMutation { + /// Adds a change that stands alone. + pub(crate) fn add_change(&mut self, change: PartnerIdChange) { + self.partner_ids.push(change); + } + + /// Adds a provider call's result. + /// + /// A removal in `result.ids` is a provider-side mistake: removals go + /// through `partner_ids` and `provider_state`. It is logged by provider + /// and source, never by UID. + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn add_result(&mut self, result: ProviderResult) { + for change in result.ids.iter().filter(|change| change.is_removal()) { + let (PartnerIdChange::OwnerRemove { source, .. } + | PartnerIdChange::WriterRemove { source, .. }) = change + else { + continue; + }; + log::warn!( + "Identity provider '{}' put a removal for source '{}' in its result IDs; removals belong in partner_ids", + result.provider, + source.as_str(), + ); + } + self.results.push(result); + } + + /// Merges a state change into any earlier one for the same provider: + /// later `n`, last `f`, the later opt-out and clear, last non-`Keep` + /// `st`. + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn merge_state( + &mut self, + provider: IdentityProviderId, + change: ProviderStateChange, + ) { + self.provider_state + .entry(provider) + .or_default() + .merge(change); + } + + /// Merges the state change of a `found` outcome and tags it, so + /// [`retain_for_error_status`](Self::retain_for_error_status) can drop + /// its `n` and `f` with the positive answer it belongs to. + pub(crate) fn merge_found_state( + &mut self, + provider: IdentityProviderId, + change: ProviderStateChange, + ) { + self.found_states.insert(provider.clone()); + self.merge_state(provider, change); + } + + /// Adds everything `other` holds, as if it had been staged here after + /// the changes already present. + pub(crate) fn absorb(&mut self, other: Self) { + let Self { + partner_ids, + provider_state, + results, + cluster_size, + found_states, + } = other; + self.partner_ids.extend(partner_ids); + for (provider, change) in provider_state { + self.merge_state(provider, change); + } + self.results.extend(results); + self.cluster_size = cluster_size.or(self.cluster_size); + self.found_states.extend(found_states); + } + + /// Sets the cluster size to write. + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn set_cluster_size(&mut self, size: u32) { + self.cluster_size = Some(size); + } + + /// How many changes the mutation holds: stand-alone ones, provider + /// results and provider state changes. Holds no UID, so it is safe to log. + #[must_use] + pub(crate) fn change_count(&self) -> usize { + self.partner_ids.len() + self.results.len() + self.provider_state.len() + } + + /// The stand-alone changes, for tests that check what a caller staged. + #[cfg(test)] + pub(crate) fn positive_changes(&self) -> &[PartnerIdChange] { + &self.partner_ids + } + + /// The stand-alone changes, removals and positives alike, for the + /// pending key. + #[must_use] + pub(crate) fn stand_alone_changes(&self) -> &[PartnerIdChange] { + &self.partner_ids + } + + /// Each provider state change that holds an opt-out or clear, for the + /// pending key. + pub(crate) fn removal_states( + &self, + ) -> impl Iterator { + self.all_states().filter(|(_, state)| state.has_removal()) + } + + /// Whether nothing would change. + #[must_use] + pub(crate) fn is_empty(&self) -> bool { + self.partner_ids.is_empty() + && self.results.is_empty() + && self.cluster_size.is_none() + && self + .provider_state + .values() + .all(ProviderStateChange::is_noop) + } + + /// Whether any removal, opt-out or clear is present. + #[must_use] + pub(crate) fn has_removals(&self) -> bool { + self.partner_ids.iter().any(PartnerIdChange::is_removal) + || self + .provider_state + .values() + .any(ProviderStateChange::has_removal) + || self.results.iter().any(|result| result.state.has_removal()) + } + + /// The provider and source of every removal, opt-out and clear, sorted and + /// de-duplicated. An opt-out or clear has no source. Holds no UID, so it + /// is safe to log. + #[must_use] + pub(crate) fn removal_labels(&self) -> Vec<(IdentityProviderId, Option)> { + let mut labels: Vec<_> = self + .partner_ids + .iter() + .filter_map(|change| match change { + PartnerIdChange::OwnerRemove { + source, provider, .. + } => Some((provider.clone(), Some(source.clone()))), + PartnerIdChange::WriterRemove { source, writer } => { + Some((writer.clone(), Some(source.clone()))) + } + _ => None, + }) + .collect(); + labels.extend( + self.all_states() + .filter(|(_, state)| state.has_removal()) + .map(|(provider, _)| (provider.clone(), None)), + ); + labels.sort(); + labels.dedup(); + labels + } + + /// A copy holding only the removals, opt-outs and clears, for the + /// removal write and the removal-only retry. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn removals(&self) -> Self { + let mut copy = Self { + partner_ids: self + .partner_ids + .iter() + .filter(|change| change.is_removal()) + .cloned() + .collect(), + ..Self::default() + }; + let states = self.provider_state.iter().chain( + self.results + .iter() + .map(|result| (&result.provider, &result.state)), + ); + for (provider, state) in states { + if state.has_removal() { + copy.merge_state(provider.clone(), state.removals()); + } + } + copy + } + + /// Cuts the mutation down for a navigation whose origin status was not + /// 2xx or 304 (spec 5.1). + /// + /// Keeps every removal, opt-out and clear, the `n` and `f` of each + /// stand-alone state change that does not come from a `found` outcome, + /// and the cluster size. Drops browser ingest, every other positive + /// change, any stand-alone `st`, the `n` and `f` of a `found` outcome, and + /// each provider result whole, keeping only its opt-out and clear. + /// + /// A stand-alone state change carries the `n` and `f` of an outcome other + /// than a stateful `found`, whose scheduling rides in its result. A + /// stateless provider's `found` lands here too, and its tag + /// ([`merge_found_state`](Self::merge_found_state)) marks it for the cut. + pub(crate) fn retain_for_error_status(&mut self) { + self.partner_ids.retain(PartnerIdChange::is_removal); + for (provider, state) in &mut self.provider_state { + if self.found_states.contains(provider) { + *state = state.removals(); + } + state.st = StateUpdate::Keep; + } + self.found_states.clear(); + for result in core::mem::take(&mut self.results) { + if result.state.has_removal() { + self.merge_state(result.provider, result.state.removals()); + } + } + } + + /// Every provider state change that applies regardless of its result: + /// the standalone ones, and each result's. + // Temporary: first used by a later identity task. + #[allow(dead_code)] + fn all_states(&self) -> impl Iterator { + self.provider_state.iter().chain( + self.results + .iter() + .map(|result| (&result.provider, &result.state)), + ) + } +} + +/// What the merge needs besides the mutation and the live row. +// Temporary: first used by a later identity task. +#[allow(dead_code)] +pub(crate) struct ApplyContext<'a> { + pub(crate) now: u64, + pub(crate) random: &'a dyn RandomSource, + pub(crate) catalog: &'a ProviderCatalog, +} + +/// The merged row. +#[derive(Debug, Clone, PartialEq)] +// Temporary: first used by a later identity task. +#[allow(dead_code)] +pub(crate) struct Applied { + pub(crate) entry: KvEntry, + /// Whether the result differs from the live row. A caller skips the + /// write when it is `false`. + pub(crate) changed: bool, + /// Whether only the removals were applied, because the full result + /// failed validation. + pub(crate) removals_only: bool, +} + +/// Merges `mutation` into `live` (spec 5.4). +/// +/// A `ps` entry holding only default values is dropped during cleanup. +/// +/// Apply order: +/// +/// 1. A tombstone (`consent.ok == false`) gives `None`. +/// 2. Removals, never skipped: +/// - Every `OwnerRemove` whose `expected` equals `record_revision(live_record)`. +/// A version 1 record matches `Revision::Unset`. It appends +/// `set_hashes(&live_record.eids)` to `ps[provider].iv`. +/// - Every `WriterRemove` whose live record has `w == writer`, or, when +/// `writer` is capture-only and the source is one of its +/// `capture_only_sources`, has no `w`. It appends the removed record's +/// hashes to `ps[writer].ov`. +/// - Each new opt-out applies when `at > live.o.at` (or there is no live +/// `o`) and `at >= live.oc` (or there is no live `oc`). Applying it +/// stores `o` and removes every live record that provider wrote, at any +/// revision, and for a capture-only provider every record of its sources +/// that no provider wrote, each adding its hashes to `ps.ov`. +/// - Then each clear (`opt_out_clear`) that is strictly later than the +/// provider's live `o.at` is stored as `oc` and empties `ps.ov`. A clear +/// that is not later, or that finds no live opt-out, is dropped and +/// writes nothing. `ps.iv` is never emptied. +/// - Hashes always come from the record KV held. A removal that no longer +/// applies adds none. +/// 3. Positive changes: +/// - A provider is opted out when its live `o` is set and its `oc` is not +/// strictly later than `o.at`. Every change whose `record.w` is opted out +/// is skipped, `OwnerWrite` and `FillMissing` alike. +/// - For an owned source, every positive change is skipped while the owner +/// is opted out, whoever wrote it, and an `OwnerWrite` or `FillMissing` +/// any of whose UIDs hashes into the owner's `ps.iv` is skipped too. Both +/// read the entry after step 2. +/// - A `FillMissing` applies when the source is absent, or its record is +/// expired, or its record's writer is not configured. It is skipped when +/// the hash of any UID of its set is in the `ps.iv` or `ps.ov` of any +/// provider in `catalog.claimants(source)`. A browser `FillMissing` +/// (`record.w` is `None`) is also skipped when the source belongs to an +/// opted-out capture-only provider's `capture_only_sources`. +/// - An `OwnerWrite` applies when the live revision equals `expected`, or +/// when `expected` is `None` and the source has no valid record with +/// `w == record.w`. A browser-seeded value never blocks the owner. +/// - A written record replaces the live record whole, never merges into it, +/// so no entry of the old set survives. It gets a fresh `rv` and +/// `at = record.at.or(now)`, and keeps the change's `w`. +/// - Each `ProviderResult` applies only when every one of its ID changes +/// would apply on its own and, when `expected_state` is `Some`, +/// `StateRevision::of(live ps.st)` equals it. Its state change, `n` and +/// `f` then apply with its IDs. Otherwise the whole result is dropped. +/// 4. `ps` changes: `n` takes the later value, `f` is replaced, `st` is +/// replaced or cleared. Then `cluster_size`. Then truncate each `ps.iv` and +/// `ps.ov` from the front to `MAX_INVALIDATED_UIDS`. Then prune the `ps` +/// entries of providers the catalog does not hold: each keeps only `o`, +/// `oc` and `ov`, so a retired provider's opt-out survives its removal from +/// config and holds again if it comes back, and an entry with none of them +/// is dropped. While `ps` is over `MAX_PROVIDER_STATES`, drop retained +/// entries first, oldest `o.at` first. Then drop expired records, except +/// refresh anchors: the source's owner is configured and holds a `st`, and +/// the record's `ru` has not passed. +/// 5. `validate()`. On failure, rebuild from the live entry through +/// [`apply_removals_only`], and if that also fails, give `None` and log. +/// `validate()` checks the entry-set limits of spec 4.5, so a set that +/// breaks one is dropped whole here and never trimmed. +/// 6. `changed` is whether the result differs from `live`. +#[must_use] +// Temporary: first used by a later identity task. +#[allow(dead_code)] +pub(crate) fn apply( + mutation: &IdentityMutation, + live: &KvEntry, + context: &ApplyContext<'_>, +) -> Option { + if !live.consent.ok { + return None; + } + + let mut entry = live.clone(); + apply_removals(&mut entry, mutation, context); + let accepted = apply_positive(&mut entry, mutation, context); + apply_state(&mut entry, mutation, &accepted); + finish_removal_fields(&mut entry, context); + drop_expired_records(&mut entry, context); + + if let Err(message) = entry.validate() { + log::warn!("Identity mutation produced an invalid row, keeping removals only: {message}"); + return apply_removals_only(mutation, live, context).map(|applied| Applied { + removals_only: true, + ..applied + }); + } + let changed = &entry != live; + Some(Applied { + entry, + changed, + removals_only: false, + }) +} + +/// Applies only spec 5.4's steps 1, 2, 4 (`o`, `oc`, `iv` and `ov`), 5 and 6, +/// for the capture removal write and the removal-only retry. It ignores every +/// positive change, `n`, `f`, `st` and `cluster_size`. +#[must_use] +// Temporary: first used by a later identity task. +#[allow(dead_code)] +pub(crate) fn apply_removals_only( + mutation: &IdentityMutation, + live: &KvEntry, + context: &ApplyContext<'_>, +) -> Option { + if !live.consent.ok { + return None; + } + + let mut entry = live.clone(); + apply_removals(&mut entry, mutation, context); + finish_removal_fields(&mut entry, context); + + if let Err(message) = entry.validate() { + log::warn!("Identity removals produced an invalid row, dropping them: {message}"); + return None; + } + let changed = &entry != live; + Some(Applied { + entry, + changed, + removals_only: true, + }) +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn state_mut<'a>(entry: &'a mut KvEntry, provider: &IdentityProviderId) -> &'a mut KvProviderState { + entry.ps.entry(provider.to_string()).or_default() +} + +/// Removes the record at `source` when `matches` accepts it, and hashes its +/// set into the list `list` picks from `provider`'s state. +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn remove_record( + entry: &mut KvEntry, + source: &str, + matches: impl FnOnce(&KvPartnerId) -> bool, + provider: &IdentityProviderId, + list: fn(&mut KvProviderState) -> &mut Vec, +) { + if !entry.ids.get(source).is_some_and(matches) { + return; + } + if let Some(record) = entry.ids.remove(source) { + let hashes = set_hashes(&record.eids); + push_hashes(list(state_mut(entry, provider)), hashes); + } +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn invalidated(state: &mut KvProviderState) -> &mut Vec { + &mut state.iv +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn opted_out_hashes(state: &mut KvProviderState) -> &mut Vec { + &mut state.ov +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn apply_removals(entry: &mut KvEntry, mutation: &IdentityMutation, context: &ApplyContext<'_>) { + for change in &mutation.partner_ids { + match change { + PartnerIdChange::OwnerRemove { + source, + provider, + expected, + } => remove_record( + entry, + source.as_str(), + |record| record_revision(record) == *expected, + provider, + invalidated, + ), + PartnerIdChange::WriterRemove { source, writer } => { + let capture_only = context + .catalog + .capture_only_sources(writer) + .contains(source); + remove_record( + entry, + source.as_str(), + |record| match record.w.as_deref() { + Some(written_by) => written_by == writer.as_str(), + None => capture_only, + }, + writer, + opted_out_hashes, + ); + } + PartnerIdChange::FillMissing { .. } | PartnerIdChange::OwnerWrite { .. } => {} + } + } + + // A clear's watermark names an opt-out the clearing request read, so it + // applies before this mutation's own opt-outs, which it never saw. + for (provider, change) in mutation.all_states() { + if let Some(clear) = change.opt_out_clear { + apply_clear(entry, provider, clear); + } + } + for (provider, change) in mutation.all_states() { + if let Some(opt_out) = &change.opt_out { + apply_opt_out(entry, provider, opt_out, context); + } + } +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn apply_opt_out( + entry: &mut KvEntry, + provider: &IdentityProviderId, + opt_out: &KvOptOut, + context: &ApplyContext<'_>, +) { + let live = entry.ps.get(provider.as_str()); + let newer = live + .and_then(|state| state.o.as_ref()) + .is_none_or(|existing| opt_out.at > existing.at); + let not_cleared_later = live + .and_then(|state| state.oc) + .is_none_or(|cleared| opt_out.at > cleared); + if !(newer && not_cleared_later) { + return; + } + state_mut(entry, provider).o = Some(opt_out.clone()); + + let capture_only_sources = context.catalog.capture_only_sources(provider); + let doomed: Vec = entry + .ids + .iter() + .filter(|(source, record)| match record.w.as_deref() { + Some(written_by) => written_by == provider.as_str(), + None => capture_only_sources + .iter() + .any(|owned| owned.as_str() == source.as_str()), + }) + .map(|(source, _)| source.clone()) + .collect(); + for source in doomed { + remove_record(entry, &source, |_| true, provider, opted_out_hashes); + } +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +/// Lifts the opt-out when `watermark`, the `at` of the newest opt-out the +/// clearing request read, reaches it (spec 8). An opt-out the request never +/// saw is newer than its watermark and stays. +fn apply_clear(entry: &mut KvEntry, provider: &IdentityProviderId, watermark: u64) { + let Some(state) = entry.ps.get_mut(provider.as_str()) else { + return; + }; + let lifts = state + .o + .as_ref() + .is_some_and(|opt_out| opt_out.at <= watermark) + && state.oc.is_none_or(|cleared| cleared < watermark); + if lifts { + state.oc = Some(watermark); + state.ov.clear(); + } +} + +/// Whether `provider` has a sticky opt-out in `entry` that no later clear lifts. +pub(crate) fn is_opted_out(entry: &KvEntry, provider: &str) -> bool { + entry + .ps + .get(provider) + .and_then(|state| state.o.as_ref().map(|o| (o, state.oc))) + .is_some_and(|(opt_out, cleared)| cleared.is_none_or(|cleared| cleared < opt_out.at)) +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn is_expired(record: &KvPartnerId, now: u64) -> bool { + record.exp.is_some_and(|exp| exp <= now) +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn change_applies(entry: &KvEntry, change: &PartnerIdChange, context: &ApplyContext<'_>) -> bool { + if let PartnerIdChange::FillMissing { source, record } + | PartnerIdChange::OwnerWrite { source, record, .. } = change + && owner_denies(entry, source, record, context) + { + return false; + } + match change { + PartnerIdChange::FillMissing { source, record } => { + fill_applies(entry, source, record, context) + } + PartnerIdChange::OwnerWrite { + source, + record, + expected, + } => { + if record + .w + .as_deref() + .is_some_and(|writer| is_opted_out(entry, writer)) + { + return false; + } + let live = entry.ids.get(source.as_str()); + match expected { + Some(expected) => live.is_some_and(|live| record_revision(live) == *expected), + None => !live.is_some_and(|live| { + live.w.is_some() && live.w == record.w && !is_expired(live, context.now) + }), + } + } + PartnerIdChange::OwnerRemove { .. } | PartnerIdChange::WriterRemove { .. } => false, + } +} + +/// Whether the source's owner denies a positive change: while the owner is +/// opted out, nothing fills its source, and no writer, the owner included, +/// stores a UID the owner invalidated. Both are read after step 2, so an +/// opt-out or invalidation in the same mutation counts. +fn owner_denies( + entry: &KvEntry, + source: &SourceDomain, + record: &KvPartnerId, + context: &ApplyContext<'_>, +) -> bool { + let Some(owner) = context.catalog.owner_of(source) else { + return false; + }; + if is_opted_out(entry, owner.as_str()) { + return true; + } + entry.ps.get(owner.as_str()).is_some_and(|state| { + set_hashes(&record.eids) + .iter() + .any(|hash| state.iv.contains(hash)) + }) +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn fill_applies( + entry: &KvEntry, + source: &SourceDomain, + record: &KvPartnerId, + context: &ApplyContext<'_>, +) -> bool { + match &record.w { + Some(writer) if is_opted_out(entry, writer) => return false, + Some(_) => {} + None => { + let blocked = context.catalog.providers().any(|provider| { + let id = provider.descriptor().id(); + is_opted_out(entry, id.as_str()) + && context.catalog.capture_only_sources(id).contains(source) + }); + if blocked { + return false; + } + } + } + + let open = entry.ids.get(source.as_str()).is_none_or(|live| { + is_expired(live, context.now) + || live.w.as_deref().is_some_and(|writer| { + !writer + .parse::() + .is_ok_and(|id| context.catalog.is_configured(&id)) + }) + }); + if !open { + return false; + } + + let hashes = set_hashes(&record.eids); + !context.catalog.claimants(source).any(|claimant| { + entry.ps.get(claimant.as_str()).is_some_and(|state| { + hashes + .iter() + .any(|hash| state.iv.contains(hash) || state.ov.contains(hash)) + }) + }) +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn write_change(entry: &mut KvEntry, change: &PartnerIdChange, context: &ApplyContext<'_>) { + let (PartnerIdChange::FillMissing { source, record } + | PartnerIdChange::OwnerWrite { source, record, .. }) = change + else { + return; + }; + let mut written = record.clone(); + written.rv = Some(new_revision(context.random).as_str().to_owned()); + written.at = record.at.or(Some(context.now)); + entry.ids.insert(source.to_string(), written); +} + +/// Applies the positive changes and returns the state changes that go with +/// them: every standalone one, and those of the results that applied. +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn apply_positive<'a>( + entry: &mut KvEntry, + mutation: &'a IdentityMutation, + context: &ApplyContext<'_>, +) -> Vec<(&'a IdentityProviderId, &'a ProviderStateChange)> { + for change in &mutation.partner_ids { + if !change.is_removal() && change_applies(entry, change, context) { + write_change(entry, change, context); + } + } + + let mut accepted: Vec<_> = mutation.provider_state.iter().collect(); + for result in &mutation.results { + let changes = || result.ids.iter().filter(|change| !change.is_removal()); + let state_matches = result.expected_state.as_ref().is_none_or(|expected| { + let live = entry + .ps + .get(result.provider.as_str()) + .and_then(|state| state.st.as_ref()); + StateRevision::of(live) == *expected + }); + if state_matches && changes().all(|change| change_applies(entry, change, context)) { + for change in changes() { + write_change(entry, change, context); + } + accepted.push((&result.provider, &result.state)); + } + } + accepted +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn apply_state( + entry: &mut KvEntry, + mutation: &IdentityMutation, + accepted: &[(&IdentityProviderId, &ProviderStateChange)], +) { + for (provider, change) in accepted { + if change.n.is_some() || change.f.is_some() || change.st != StateUpdate::Keep { + let state = state_mut(entry, provider); + state.n = state.n.max(change.n); + if let Some(failures) = change.f { + state.f = failures; + } + match &change.st { + StateUpdate::Keep => {} + StateUpdate::Replace(value) => state.st = Some(value.clone()), + StateUpdate::Clear => state.st = None, + } + } + } + if let Some(size) = mutation.cluster_size { + entry.network = Some(KvNetwork { + cluster_size: Some(size), + }); + } +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn is_configured(entry_key: &str, context: &ApplyContext<'_>) -> bool { + entry_key + .parse::() + .is_ok_and(|id| context.catalog.is_configured(&id)) +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn finish_removal_fields(entry: &mut KvEntry, context: &ApplyContext<'_>) { + for state in entry.ps.values_mut() { + for list in [&mut state.iv, &mut state.ov] { + let excess = list.len().saturating_sub(MAX_INVALIDATED_UIDS); + list.drain(..excess); + } + } + + for (provider, state) in &mut entry.ps { + if !is_configured(provider, context) { + *state = KvProviderState { + o: state.o.take(), + oc: state.oc.take(), + ov: std::mem::take(&mut state.ov), + ..KvProviderState::default() + }; + } + } + entry + .ps + .retain(|_, state| *state != KvProviderState::default()); + + while entry.ps.len() > MAX_PROVIDER_STATES { + let oldest = entry + .ps + .iter() + .filter(|(provider, _)| !is_configured(provider, context)) + .min_by_key(|(_, state)| state.o.as_ref().map(|o| o.at)) + .map(|(provider, _)| provider.clone()); + let Some(oldest) = oldest else { + break; + }; + entry.ps.remove(&oldest); + } +} + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +fn drop_expired_records(entry: &mut KvEntry, context: &ApplyContext<'_>) { + let ps = &entry.ps; + entry.ids.retain(|source, record| { + if !is_expired(record, context.now) { + return true; + } + let Some(owner) = source + .parse::() + .ok() + .and_then(|source| context.catalog.owner_of(&source).cloned()) + .filter(|owner| context.catalog.is_configured(owner)) + else { + return false; + }; + // A stateful owner anchors the record until its `ru`. + let state_anchor = ps + .get(owner.as_str()) + .is_some_and(|state| state.st.is_some()) + && record.ru.is_some_and(|ru| context.now < ru); + // A stateless owner anchors a record it wrote until `exp` plus the + // refresh grace its descriptor declares. + let grace_anchor = record.w.as_deref() == Some(owner.as_str()) + && record.exp.is_some_and(|exp| { + context + .catalog + .get(&owner) + .and_then(|provider| provider.descriptor().resolves()) + .and_then(Resolves::refresh_grace) + .is_some_and(|grace| context.now < exp.saturating_add(grace.as_secs())) + }); + state_anchor || grace_anchor + }); +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::*; + use crate::ec::identity::catalog::ProviderCatalog; + use crate::ec::identity::descriptor::{ConsentPolicy, ProviderDescriptor}; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::revision::{RecordRevision, StateRevision, set_hashes, uid_hash}; + use crate::ec::identity::test_support::{ + example_entry_set, fixed_capture_provider, fixed_provider, full_record, provider_from, + }; + use crate::ec::kv_types::{ + KvEntry, KvOptOut, KvPartnerId, KvProviderState, MAX_INVALIDATED_UIDS, MAX_PROVIDER_STATES, + }; + + const NOW: u64 = 1_000; + + fn src(name: &str) -> SourceDomain { + name.parse().expect("should parse the source") + } + + fn pid(name: &str) -> IdentityProviderId { + name.parse().expect("should parse the provider ID") + } + + fn rev(value: &str) -> Revision { + Revision::Set(RecordRevision::from_stored(value)) + } + + /// `owner` owns a.example.com, `other` claims b.example.com unowned, and + /// the capture-only `cap` is narrowed to c.example.com. + fn catalog() -> ProviderCatalog { + ProviderCatalog::new( + vec![ + fixed_provider("owner", &["a.example.com"]), + fixed_provider("other", &["b.example.com"]), + fixed_capture_provider("cap", &["c.example.com"]), + ], + [(src("a.example.com"), pid("owner"))].into_iter().collect(), + ) + } + + fn run(mutation: &IdentityMutation, live: &KvEntry) -> Option { + let random = SeededRandom::new(1); + let catalog = catalog(); + apply( + mutation, + live, + &ApplyContext { + now: NOW, + random: &random, + catalog: &catalog, + }, + ) + } + + fn run_removals(mutation: &IdentityMutation, live: &KvEntry) -> Option { + let random = SeededRandom::new(1); + let catalog = catalog(); + apply_removals_only( + mutation, + live, + &ApplyContext { + now: NOW, + random: &random, + catalog: &catalog, + }, + ) + } + + fn written(writer: &str, uid: &str, rv: &str) -> KvPartnerId { + KvPartnerId { + w: Some(writer.to_owned()), + rv: Some(rv.to_owned()), + at: Some(900), + ..KvPartnerId::new(uid) + } + } + + fn opt_out(at: u64) -> ProviderStateChange { + ProviderStateChange { + opt_out: Some(KvOptOut { + at, + fp: "1".to_owned(), + }), + ..ProviderStateChange::default() + } + } + + fn fill(source: &str, record: KvPartnerId) -> PartnerIdChange { + PartnerIdChange::FillMissing { + source: src(source), + record, + } + } + + #[test] + fn retain_for_error_status_keeps_removals_and_backoff() { + let removal = PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: Revision::Unset, + }; + let backoff = ProviderStateChange { + n: Some(NOW + 86_400), + ..ProviderStateChange::default() + }; + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill("b.example.com", KvPartnerId::browser("u", Some(NOW)))); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: written("owner", "found-uid", "1111111111111111"), + expected: None, + }); + mutation.add_change(removal.clone()); + mutation.merge_state(pid("other"), backoff.clone()); + + mutation.retain_for_error_status(); + + let mut expected = IdentityMutation::default(); + expected.add_change(removal); + expected.merge_state(pid("other"), backoff); + assert_eq!( + mutation, expected, + "should keep only the removal and the no_match backoff" + ); + } + + #[test] + fn retain_for_error_status_drops_the_scheduling_of_a_found_and_keeps_its_opt_out() { + let opt_out = KvOptOut { + at: NOW, + fp: "1".to_owned(), + }; + let found = ProviderStateChange { + n: Some(NOW + 60), + f: Some(0), + opt_out: Some(opt_out.clone()), + ..ProviderStateChange::default() + }; + let backoff = ProviderStateChange { + n: Some(NOW + 86_400), + f: Some(3), + ..ProviderStateChange::default() + }; + let mut mutation = IdentityMutation::default(); + mutation.merge_found_state(pid("found"), found); + mutation.merge_state(pid("other"), backoff.clone()); + + mutation.retain_for_error_status(); + + let mut expected = IdentityMutation::default(); + expected.merge_state( + pid("found"), + ProviderStateChange { + opt_out: Some(opt_out), + ..ProviderStateChange::default() + }, + ); + expected.merge_state(pid("other"), backoff); + assert_eq!( + mutation, expected, + "should keep the found's opt-out and the other outcome's n and f, and drop the found's n and f" + ); + } + + #[test] + fn absorb_adds_changes_states_and_found_tags_after_the_existing_ones() { + let mut base = IdentityMutation::default(); + base.add_change(fill("a.example.com", KvPartnerId::browser("u", Some(NOW)))); + base.merge_state( + pid("p"), + ProviderStateChange { + n: Some(10), + ..ProviderStateChange::default() + }, + ); + let mut other = IdentityMutation::default(); + other.add_change(fill("b.example.com", KvPartnerId::browser("v", Some(NOW)))); + other.merge_found_state( + pid("p"), + ProviderStateChange { + n: Some(20), + f: Some(0), + ..ProviderStateChange::default() + }, + ); + + base.absorb(other); + + let mut expected = IdentityMutation::default(); + expected.add_change(fill("a.example.com", KvPartnerId::browser("u", Some(NOW)))); + expected.add_change(fill("b.example.com", KvPartnerId::browser("v", Some(NOW)))); + expected.merge_found_state( + pid("p"), + ProviderStateChange { + n: Some(20), + f: Some(0), + ..ProviderStateChange::default() + }, + ); + assert_eq!( + base, expected, + "should hold both mutations, the found tag included" + ); + } + + #[test] + fn tombstone_drops_everything() { + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill("b.example.com", KvPartnerId::browser("u", Some(NOW)))); + assert!( + run(&mutation, &KvEntry::tombstone(1)).is_none(), + "should drop on a tombstone" + ); + assert!( + run_removals(&mutation, &KvEntry::tombstone(1)).is_none(), + "should drop removals on a tombstone" + ); + } + + #[test] + fn fill_missing_fills_absent_expired_and_unconfigured_only() { + let mut live = KvEntry::minimal("b.example.com", "existing", 1); + live.ids.insert( + "d.example.com".to_owned(), + KvPartnerId { + exp: Some(NOW), + ..KvPartnerId::new("old") + }, + ); + live.ids.insert( + "f.example.com".to_owned(), + written("retired", "stale", "1111111111111111"), + ); + let mut mutation = IdentityMutation::default(); + for source in [ + "b.example.com", + "d.example.com", + "e.example.com", + "f.example.com", + ] { + mutation.add_change(fill(source, KvPartnerId::browser("new", Some(NOW)))); + } + + let applied = run(&mutation, &live).expect("should apply"); + + let uid = |source: &str| applied.entry.ids[source].first_uid().map(str::to_owned); + assert_eq!( + uid("b.example.com").as_deref(), + Some("existing"), + "should not replace a live value" + ); + assert_eq!( + uid("d.example.com").as_deref(), + Some("new"), + "should replace an expired value" + ); + assert_eq!( + uid("e.example.com").as_deref(), + Some("new"), + "should fill an absent source" + ); + assert_eq!( + uid("f.example.com").as_deref(), + Some("new"), + "should treat an unconfigured writer as absent" + ); + let record = &applied.entry.ids["e.example.com"]; + assert_eq!( + (record.at, record.w.as_deref()), + (Some(NOW), None), + "should be a browser record" + ); + assert!( + record.rv.is_some(), + "should give every record it writes an rv" + ); + } + + #[test] + fn owner_write_checks_revision_and_gets_a_fresh_rv() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", "v1", "1111111111111111"), + ); + let write = |expected| { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId { + w: Some("owner".to_owned()), + ..KvPartnerId::new("v2") + }, + expected, + }); + run(&mutation, &live).expect("should apply") + }; + + assert!( + !write(Some(rev("2222222222222222"))).changed, + "should lose a stale revision to the newer write" + ); + let applied = write(Some(rev("1111111111111111"))); + let record = &applied.entry.ids["a.example.com"]; + assert_eq!(record.first_uid(), Some("v2"), "should write"); + assert_eq!(record.w.as_deref(), Some("owner"), "should keep the writer"); + assert_ne!( + record.rv.as_deref(), + Some("1111111111111111"), + "should draw a new rv" + ); + assert_eq!(record.at, Some(NOW), "should stamp at"); + } + + #[test] + fn owner_write_replaces_the_whole_set_and_leaves_other_sources() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + rv: Some("1111111111111111".to_owned()), + ..full_record(Some("owner"), example_entry_set()) + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId { + w: Some("owner".to_owned()), + ..KvPartnerId::new("only") + }, + expected: Some(rev("1111111111111111")), + }); + + let applied = run(&mutation, &live).expect("should apply"); + + assert_eq!( + applied.entry.ids["a.example.com"], + KvPartnerId { + rv: applied.entry.ids["a.example.com"].rv.clone(), + at: Some(NOW), + w: Some("owner".to_owned()), + ..KvPartnerId::new("only") + }, + "should keep no entry of the old set" + ); + assert_eq!( + applied.entry.ids["b.example.com"].first_uid(), + Some("x"), + "should leave other sources alone" + ); + + let mut full = IdentityMutation::default(); + full.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: full_record(Some("owner"), example_entry_set()), + expected: None, + }); + let applied = run(&full, &KvEntry::minimal("b.example.com", "x", 1)).expect("should apply"); + assert_eq!( + applied.entry.ids["a.example.com"].eids, + example_entry_set(), + "should write every entry, in order" + ); + } + + #[test] + fn browser_seeded_and_version_1_values_never_block_the_owner() { + let live = KvEntry::minimal("a.example.com", "seeded", 1); + let mut write = IdentityMutation::default(); + write.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId { + w: Some("owner".to_owned()), + ..KvPartnerId::new("owned") + }, + expected: None, + }); + assert_eq!( + run(&write, &live).expect("should apply").entry.ids["a.example.com"].first_uid(), + Some("owned"), + "should let the owner take over a seeded value" + ); + + let mut remove = IdentityMutation::default(); + remove.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: Revision::Unset, + }); + let applied = run(&remove, &live).expect("should apply"); + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should match a record without rv as Unset" + ); + assert_eq!( + applied.entry.ps["owner"].iv, + vec![uid_hash("seeded")], + "should hash the removed UID into iv" + ); + } + + #[test] + fn an_applied_invalidation_hashes_every_uid_of_the_live_set() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + rv: Some("1111111111111111".to_owned()), + ..full_record(Some("owner"), example_entry_set()) + }, + ); + let invalidate = |expected: &str| { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: rev(expected), + }); + run(&mutation, &live).expect("should apply") + }; + + let applied = invalidate("1111111111111111"); + let stale = invalidate("2222222222222222"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should remove the set" + ); + assert_eq!( + applied.entry.ps["owner"].iv, + set_hashes(&example_entry_set()), + "should hash every UID it removed" + ); + assert!( + stale.entry.ids.contains_key("a.example.com"), + "should keep a set at another revision" + ); + assert!( + stale + .entry + .ps + .get("owner") + .is_none_or(|state| state.iv.is_empty()), + "should add no hash" + ); + } + + /// A live row where `owner` wrote `uid` on a.example.com at rv `1111…`. + fn owned_row(uid: &str) -> KvEntry { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", uid, "1111111111111111"), + ); + live + } + + #[test] + fn an_invalidation_and_a_browser_refill_of_the_same_uid_leave_it_removed() { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: rev("1111111111111111"), + }); + mutation.add_change(fill( + "a.example.com", + KvPartnerId::browser("uid-a", Some(950)), + )); + + let applied = run(&mutation, &owned_row("uid-a")).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should not refill the UID the same mutation invalidated" + ); + assert!( + !applied.entry.ps["owner"].iv.is_empty(), + "should record the invalidation" + ); + } + + #[test] + fn an_opted_out_owner_blocks_every_positive_change_to_its_sources() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 500, + fp: "1".to_owned(), + }), + ..KvProviderState::default() + }, + ); + let browser = fill("a.example.com", KvPartnerId::browser("new-uid", Some(950))); + let other_writer = fill( + "a.example.com", + written("other", "new-uid", "2222222222222222"), + ); + + for change in [browser, other_writer] { + let mut mutation = IdentityMutation::default(); + mutation.add_change(change.clone()); + + let applied = run(&mutation, &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should skip {change:?} while the owner is opted out" + ); + } + + let mut control = IdentityMutation::default(); + control.add_change(fill( + "b.example.com", + KvPartnerId::browser("new-uid", Some(950)), + )); + let applied = + run(&control, &KvEntry::minimal("c.example.com", "x", 1)).expect("should apply"); + assert!( + applied.entry.ids.contains_key("b.example.com"), + "control: should fill a source the opted-out owner does not own" + ); + } + + #[test] + fn an_owner_opt_out_in_the_same_mutation_blocks_a_browser_refill() { + let mut mutation = IdentityMutation::default(); + mutation.merge_state(pid("owner"), opt_out(1_000)); + mutation.add_change(fill( + "a.example.com", + KvPartnerId::browser("new-uid", Some(950)), + )); + + let applied = run(&mutation, &owned_row("uid-a")).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should neither keep the owner's record nor take the browser's" + ); + } + + #[test] + fn an_owner_write_of_an_invalidated_uid_is_skipped() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + iv: vec![uid_hash("uid-a")], + ..KvProviderState::default() + }, + ); + let owner_write = |uid: &str| { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: written("owner", uid, "3333333333333333"), + expected: None, + }); + run(&mutation, &live).expect("should apply") + }; + + assert!( + !owner_write("uid-a").entry.ids.contains_key("a.example.com"), + "should not store a UID the owner invalidated" + ); + assert!( + owner_write("uid-b").entry.ids.contains_key("a.example.com"), + "control: should store another UID" + ); + } + + #[test] + fn iv_keeps_the_128_most_recent_hashes() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + let old: Vec = (0..MAX_INVALIDATED_UIDS) + .map(|index| format!("{index:016x}")) + .collect(); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + iv: old.clone(), + ..KvProviderState::default() + }, + ); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", "u", "1111111111111111"), + ); + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: rev("1111111111111111"), + }); + + let applied = run(&mutation, &live).expect("should apply"); + + let iv = &applied.entry.ps["owner"].iv; + assert_eq!(iv.len(), MAX_INVALIDATED_UIDS, "should cap at 128"); + assert_eq!(iv[..127], old[1..], "should drop the oldest hash"); + assert_eq!(iv[127], uid_hash("u"), "should append the new hash"); + } + + #[test] + fn opt_out_removes_what_the_writer_wrote_fills_ov_and_skips_its_changes() { + let mut live = KvEntry::minimal("e.example.com", "browser", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", "v1", "1111111111111111"), + ); + live.ids.insert( + "b.example.com".to_owned(), + written("owner", "unowned-by-owner", "2222222222222222"), + ); + let mut mutation = IdentityMutation::default(); + mutation.merge_state(pid("owner"), opt_out(NOW)); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId { + w: Some("owner".to_owned()), + ..KvPartnerId::new("v2") + }, + expected: None, + }); + + let applied = run(&mutation, &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should remove the owned record" + ); + assert!( + !applied.entry.ids.contains_key("b.example.com"), + "should remove the unowned record it wrote" + ); + assert_eq!( + applied.entry.ids["e.example.com"].first_uid(), + Some("browser"), + "should leave other records" + ); + let state = &applied.entry.ps["owner"]; + assert_eq!( + state.o.as_ref().map(|o| o.at), + Some(NOW), + "should store the opt-out" + ); + assert_eq!( + state.ov, + vec![uid_hash("v1"), uid_hash("unowned-by-owner")], + "should hash what it removed into ov" + ); + assert!(state.iv.is_empty(), "should leave iv for invalidations"); + } + + #[test] + fn a_capture_only_opt_out_removes_browser_records_of_its_sources_and_blocks_their_fill() { + let mut live = KvEntry::minimal("c.example.com", "browser-c", 1); + live.ids + .insert("e.example.com".to_owned(), KvPartnerId::new("browser-e")); + let mut mutation = IdentityMutation::default(); + mutation.merge_state(pid("cap"), opt_out(NOW)); + + let applied = run(&mutation, &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("c.example.com"), + "should remove a browser record of its source" + ); + assert!( + applied.entry.ids.contains_key("e.example.com"), + "should leave a source it is not narrowed to" + ); + + let mut refill = IdentityMutation::default(); + refill.add_change(fill( + "c.example.com", + KvPartnerId::browser("another", Some(NOW)), + )); + let applied = run(&refill, &applied.entry).expect("should apply"); + assert!( + !applied.entry.ids.contains_key("c.example.com"), + "should skip a browser fill while it is opted out" + ); + } + + #[test] + fn a_clear_lifts_only_the_opt_out_it_saw() { + let mut opted = KvEntry::minimal("b.example.com", "x", 1); + opted.ps.insert( + "owner".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 900, + fp: "1".to_owned(), + }), + ov: vec![uid_hash("opted-out")], + ..KvProviderState::default() + }, + ); + let clear_up_to = |watermark: u64| { + let mut clear = IdentityMutation::default(); + clear.merge_state( + pid("owner"), + ProviderStateChange { + opt_out_clear: Some(watermark), + ..ProviderStateChange::default() + }, + ); + run(&clear, &opted) + }; + + let seen = clear_up_to(900).expect("should apply"); + assert_eq!( + seen.entry.ps["owner"].oc, + Some(900), + "should store the at of the opt-out the clear saw" + ); + assert!( + !is_opted_out(&seen.entry, "owner"), + "should lift the opt-out it saw" + ); + assert!( + clear_up_to(899).is_none_or(|applied| is_opted_out(&applied.entry, "owner")), + "should leave an opt-out newer than the one the clear saw" + ); + + let mut reopted = IdentityMutation::default(); + reopted.merge_state(pid("owner"), opt_out(901)); + let again = run(&reopted, &seen.entry).expect("should apply"); + assert!( + is_opted_out(&again.entry, "owner"), + "should apply a later opt-out over the clear" + ); + } + + #[test] + fn opt_out_and_clear_ordering() { + let live = KvEntry::minimal("b.example.com", "x", 1); + let mut same_second = IdentityMutation::default(); + same_second.merge_state( + pid("owner"), + ProviderStateChange { + opt_out_clear: Some(NOW), + ..opt_out(NOW) + }, + ); + let applied = run(&same_second, &live).expect("should apply"); + assert!( + applied.entry.ps["owner"].o.is_some(), + "should stay opted out" + ); + assert_eq!( + applied.entry.ps["owner"].oc, None, + "should drop a clear that is not later" + ); + + let mut no_opt_out = IdentityMutation::default(); + no_opt_out.merge_state( + pid("owner"), + ProviderStateChange { + opt_out_clear: Some(NOW), + ..ProviderStateChange::default() + }, + ); + assert!( + !run(&no_opt_out, &live).expect("should apply").changed, + "should drop a clear with nothing to clear" + ); + + let mut opted = live.clone(); + opted.ps.insert( + "owner".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 900, + fp: "1".to_owned(), + }), + iv: vec![uid_hash("invalidated")], + ov: vec![uid_hash("opted-out")], + ..KvProviderState::default() + }, + ); + let mut clear = IdentityMutation::default(); + clear.merge_state( + pid("owner"), + ProviderStateChange { + opt_out_clear: Some(NOW), + ..ProviderStateChange::default() + }, + ); + let state = &run(&clear, &opted).expect("should apply").entry.ps["owner"]; + assert_eq!(state.oc, Some(NOW), "should store the clear's watermark"); + assert!(state.ov.is_empty(), "should empty ov"); + assert_eq!( + state.iv, + vec![uid_hash("invalidated")], + "should never empty iv" + ); + + let mut older = IdentityMutation::default(); + older.merge_state(pid("owner"), opt_out(899)); + assert!( + !run(&older, &opted).expect("should apply").changed, + "should drop an opt-out not after the live one" + ); + } + + #[test] + fn fill_missing_is_skipped_when_any_uid_is_in_a_claimants_iv_or_ov() { + let mut live = KvEntry::minimal("e.example.com", "x", 1); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + iv: vec![uid_hash("example-b")], + ..KvProviderState::default() + }, + ); + live.ps.insert( + "other".to_owned(), + KvProviderState { + ov: vec![uid_hash("removed")], + ..KvProviderState::default() + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill( + "a.example.com", + full_record(None, example_entry_set()), + )); + mutation.add_change(fill( + "b.example.com", + KvPartnerId::browser("removed", Some(NOW)), + )); + mutation.add_change(fill( + "d.example.com", + KvPartnerId::browser("removed", Some(NOW)), + )); + + let applied = run(&mutation, &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should block an owned set on its second UID" + ); + assert!( + !applied.entry.ids.contains_key("b.example.com"), + "should block an unowned source by its claimant's ov" + ); + assert!( + applied.entry.ids.contains_key("d.example.com"), + "should not block a source no denying provider claims" + ); + } + + #[test] + fn provider_result_applies_atomically() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", "newer", "3333333333333333"), + ); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + st: Some(json!({ "token": "B" })), + ..KvProviderState::default() + }, + ); + let result = |expected, expected_state| ProviderResult { + provider: pid("owner"), + ids: vec![PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId { + w: Some("owner".to_owned()), + ..KvPartnerId::new("mine") + }, + expected: Some(expected), + }], + state: ProviderStateChange { + st: StateUpdate::Replace(json!({ "token": "A" })), + f: Some(0), + ..ProviderStateChange::default() + }, + expected_state, + }; + + let mut lost_id = IdentityMutation::default(); + lost_id.add_result(result( + rev("1111111111111111"), + Some(StateRevision::of(Some(&json!({ "token": "B" })))), + )); + let mut lost_state = IdentityMutation::default(); + lost_state.add_result(result( + rev("3333333333333333"), + Some(StateRevision::of(Some(&json!({ "token": "A0" })))), + )); + let mut wins = IdentityMutation::default(); + wins.add_result(result( + rev("3333333333333333"), + Some(StateRevision::of(Some(&json!({ "token": "B" })))), + )); + + for (name, mutation) in [("an ID", lost_id), ("the state", lost_state)] { + assert!( + !run(&mutation, &live).expect("should apply").changed, + "should drop the whole result when {name} lost" + ); + } + let applied = run(&wins, &live).expect("should apply"); + assert_eq!( + applied.entry.ids["a.example.com"].first_uid(), + Some("mine"), + "should write the ID" + ); + assert_eq!( + applied.entry.ps["owner"].st, + Some(json!({ "token": "A" })), + "should write the state with it" + ); + } + + #[test] + fn refresh_anchor_survives_until_ru_or_state_loss() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + exp: Some(NOW - 1), + ru: Some(NOW + 100), + ..written("owner", "expired", "1111111111111111") + }, + ); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + st: Some(json!({ "r": 1 })), + ..KvProviderState::default() + }, + ); + let mut unrelated = IdentityMutation::default(); + unrelated.set_cluster_size(1); + + assert!( + run(&unrelated, &live) + .expect("should apply") + .entry + .ids + .contains_key("a.example.com"), + "should keep the anchor" + ); + live.ps + .get_mut("owner") + .expect("should hold the owner's state") + .st = None; + assert!( + !run(&unrelated, &live) + .expect("should apply") + .entry + .ids + .contains_key("a.example.com"), + "should drop it" + ); + } + + /// Runs `mutation` with an owner of `a.example.com` that keeps no + /// private state and declares `grace` as its refresh grace. + fn run_with_graced_owner( + mutation: &IdentityMutation, + live: &KvEntry, + grace: Option, + ) -> Option { + let mut resolves = Resolves::new().with_endpoint("https://api.example.com"); + if let Some(grace) = grace { + resolves = resolves.with_refresh_grace(core::time::Duration::from_secs(grace)); + } + let owner = provider_from( + ProviderDescriptor::builder(pid("owner")) + .with_source(src("a.example.com")) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::eid_persistence_only()) + .build() + .expect("should build the owner"), + ); + let catalog = ProviderCatalog::new( + vec![owner, fixed_provider("other", &["b.example.com"])], + [(src("a.example.com"), pid("owner"))].into_iter().collect(), + ); + let random = SeededRandom::new(1); + apply( + mutation, + live, + &ApplyContext { + now: NOW, + random: &random, + catalog: &catalog, + }, + ) + } + + fn keeps_expired_owned_record(writer: &str, expired_for: u64, grace: Option) -> bool { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + exp: Some(NOW - expired_for), + ..written(writer, "expired", "1111111111111111") + }, + ); + let mut unrelated = IdentityMutation::default(); + unrelated.set_cluster_size(1); + run_with_graced_owner(&unrelated, &live, grace) + .expect("should apply") + .entry + .ids + .contains_key("a.example.com") + } + + #[test] + fn stateless_refresh_anchor_survives_until_exp_plus_grace() { + assert!( + keeps_expired_owned_record("owner", 50, Some(100)), + "should keep the owner's expired record inside its refresh grace" + ); + assert!( + !keeps_expired_owned_record("owner", 100, Some(100)), + "should drop it once exp plus the grace has passed" + ); + assert!( + !keeps_expired_owned_record("owner", 50, None), + "should drop it when the owner declares no grace, as before" + ); + assert!( + !keeps_expired_owned_record("other", 50, Some(100)), + "should drop an expired record another provider wrote" + ); + } + + #[test] + fn unconfigured_provider_state_keeps_o_oc_and_ov() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + let kept = KvProviderState { + o: Some(KvOptOut { + at: 7, + fp: "1".to_owned(), + }), + oc: Some(6), + ov: vec![uid_hash("removed")], + ..KvProviderState::default() + }; + live.ps.insert( + "retired".to_owned(), + KvProviderState { + n: Some(5), + iv: vec![uid_hash("invalidated")], + st: Some(json!(1)), + ..kept.clone() + }, + ); + live.ps.insert( + "gone".to_owned(), + KvProviderState { + n: Some(5), + ..KvProviderState::default() + }, + ); + live.ps.insert( + "other".to_owned(), + KvProviderState { + n: Some(5), + ..KvProviderState::default() + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.set_cluster_size(1); + + let applied = run(&mutation, &live).expect("should apply"); + + assert_eq!( + applied.entry.ps["retired"], kept, + "should keep only o, oc and ov" + ); + assert!( + !applied.entry.ps.contains_key("gone"), + "should drop an entry with none of them" + ); + assert_eq!( + applied.entry.ps["other"].n, + Some(5), + "should leave a configured provider untouched" + ); + } + + #[test] + fn retained_opt_outs_are_pruned_first_when_over_the_state_cap() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "other".to_owned(), + KvProviderState { + n: Some(5), + ..KvProviderState::default() + }, + ); + for (at, index) in (100_u64..).zip(0..MAX_PROVIDER_STATES) { + live.ps.insert( + format!("retired-{index:02}"), + KvProviderState { + o: Some(KvOptOut { + at, + fp: "1".to_owned(), + }), + ..KvProviderState::default() + }, + ); + } + let mut mutation = IdentityMutation::default(); + mutation.set_cluster_size(1); + + let applied = run(&mutation, &live).expect("should apply"); + + assert_eq!( + applied.entry.ps.len(), + MAX_PROVIDER_STATES, + "should fit the cap" + ); + assert!( + applied.entry.ps.contains_key("other"), + "should keep the configured provider" + ); + assert!( + !applied.entry.ps.contains_key("retired-00"), + "should drop the oldest retained opt-out" + ); + assert!( + applied.entry.ps.contains_key("retired-01"), + "should keep the newer ones" + ); + } + + #[test] + fn an_invalid_result_falls_back_to_removals_only() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", "v1", "1111111111111111"), + ); + let mut oversized = example_entry_set(); + oversized[0].ext = Some(json!({ "segments": "x".repeat(2_000) })); + oversized[1].ext = Some(json!({ "segments": "y".repeat(2_000) })); + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill("d.example.com", full_record(Some("other"), oversized))); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: rev("1111111111111111"), + }); + + let applied = run(&mutation, &live).expect("should apply removals"); + + assert!( + applied.removals_only, + "should fail validation for a set over MAX_ENTRY_SET_BYTES" + ); + assert!( + !applied.entry.ids.contains_key("d.example.com"), + "should drop the set whole, never trim it" + ); + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should still apply the removal" + ); + } + + #[test] + fn apply_removals_only_ignores_everything_but_removals() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", "v1", "1111111111111111"), + ); + live.ids.insert( + "d.example.com".to_owned(), + KvPartnerId { + exp: Some(NOW - 1), + ..KvPartnerId::new("expired") + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill( + "e.example.com", + KvPartnerId::browser("new", Some(NOW)), + )); + mutation.merge_state( + pid("other"), + ProviderStateChange { + n: Some(5), + f: Some(2), + ..opt_out(NOW) + }, + ); + mutation.set_cluster_size(3); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: rev("1111111111111111"), + }); + + let applied = run_removals(&mutation, &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should apply the removal" + ); + assert!( + !applied.entry.ids.contains_key("e.example.com"), + "should ignore a positive change" + ); + assert!( + applied.entry.ids.contains_key("d.example.com"), + "should not prune expired records" + ); + let state = &applied.entry.ps["other"]; + assert_eq!((state.n, state.f), (None, 0), "should never touch n or f"); + assert!(state.o.is_some(), "should store the opt-out"); + assert_eq!( + applied.entry.network, live.network, + "should not touch cluster_size" + ); + + let nothing = IdentityMutation::default(); + assert!( + !run_removals(¬hing, &live).expect("should apply").changed, + "should report no change" + ); + } + + fn writer_remove(source: &str, writer: &str) -> IdentityMutation { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::WriterRemove { + source: src(source), + writer: pid(writer), + }); + mutation + } + + #[test] + fn a_writer_remove_removes_a_record_the_writer_owns() { + let mut live = KvEntry::minimal("e.example.com", "browser", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("owner", "v1", "1111111111111111"), + ); + + let applied = + run_removals(&writer_remove("a.example.com", "owner"), &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should remove the record the writer wrote" + ); + assert_eq!( + applied.entry.ps["owner"].ov, + vec![uid_hash("v1")], + "should hash the removed UID into ov" + ); + } + + #[test] + fn a_capture_only_writer_remove_removes_a_browser_record() { + let mut live = KvEntry::minimal("e.example.com", "other", 1); + live.ids + .insert("c.example.com".to_owned(), KvPartnerId::new("browser-c")); + + let applied = + run_removals(&writer_remove("c.example.com", "cap"), &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("c.example.com"), + "should remove the browser record of a capture-only source" + ); + assert_eq!( + applied.entry.ps["cap"].ov, + vec![uid_hash("browser-c")], + "should hash the removed UID into ov" + ); + } + + #[test] + fn a_writer_remove_keeps_another_writers_record() { + let mut live = KvEntry::minimal("e.example.com", "browser", 1); + live.ids.insert( + "a.example.com".to_owned(), + written("other", "v1", "1111111111111111"), + ); + + let applied = + run_removals(&writer_remove("a.example.com", "owner"), &live).expect("should apply"); + + assert!( + applied.entry.ids.contains_key("a.example.com"), + "should keep a record another writer wrote" + ); + assert!(!applied.changed, "should report no change"); + } + + #[test] + fn removals_and_has_removals_round_trip() { + let mut mutation = writer_remove("a.example.com", "owner"); + mutation.add_change(fill("b.example.com", KvPartnerId::browser("u", Some(NOW)))); + mutation.merge_state(pid("other"), opt_out(NOW)); + mutation.merge_state( + pid("cap"), + ProviderStateChange { + n: Some(3), + ..ProviderStateChange::default() + }, + ); + + assert!(mutation.has_removals(), "should see the removals"); + let removals = mutation.removals(); + assert!(removals.has_removals(), "should keep the removals"); + assert_eq!( + removals.removal_labels(), + mutation.removal_labels(), + "should keep every removal label" + ); + assert!( + removals.partner_ids.iter().all(PartnerIdChange::is_removal), + "should drop the positive changes" + ); + assert!( + !removals.provider_state.contains_key(&pid("cap")), + "should drop a state change that holds no removal" + ); + assert!( + !IdentityMutation::default().has_removals(), + "should see no removals in an empty mutation" + ); + } + + #[test] + fn removal_labels_are_sorted_and_deduplicated() { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: rev("1111111111111111"), + }); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + provider: pid("owner"), + expected: rev("2222222222222222"), + }); + mutation.add_change(PartnerIdChange::WriterRemove { + source: src("b.example.com"), + writer: pid("cap"), + }); + mutation.merge_state(pid("owner"), opt_out(NOW)); + mutation.merge_state( + pid("other"), + ProviderStateChange { + opt_out_clear: Some(NOW), + ..ProviderStateChange::default() + }, + ); + mutation.add_result(ProviderResult { + provider: pid("owner"), + ids: Vec::new(), + state: opt_out(NOW), + expected_state: None, + }); + + assert_eq!( + mutation.removal_labels(), + vec![ + (pid("cap"), Some(src("b.example.com"))), + (pid("other"), None), + (pid("owner"), None), + (pid("owner"), Some(src("a.example.com"))), + ], + "should list each provider and source once, sorted, with no UID" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/pending.rs b/crates/trusted-server-core/src/ec/identity/pending.rs new file mode 100644 index 0000000000..461cb22216 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/pending.rs @@ -0,0 +1,451 @@ +//! The pending key: removals and capture results that missed the row write, +//! kept for the next writer to merge (spec 5.3, Pending key). +//! +//! `pending:` holds a list of items, each with a generation. An item +//! is appended with a CAS, and the next writer request merges every item +//! newer than the row's `pg` watermark into its one row write, then stores +//! the newest generation as `pg`. The key is never deleted: its one-day TTL +//! ends it. Auctions never read it. + +use std::collections::BTreeMap; +use std::time::Duration; + +use serde::{Deserialize, Serialize}; + +use super::ids::{IdentityProviderId, SourceDomain}; +use super::mutation::{IdentityMutation, PartnerIdChange, ProviderStateChange}; +use super::revision::{RecordRevision, Revision}; +use crate::ec::kv::{KvIdentityGraph, MAX_CAS_RETRIES}; +use crate::ec::kv_backend::{EcKvWriteMode, EcKvWriteOutcome}; +use crate::ec::kv_types::{KvOptOut, KvPartnerId}; +use crate::ec::log_id; + +/// How long a pending key lives after its last append. +pub(crate) const PENDING_TTL: Duration = Duration::from_secs(24 * 60 * 60); + +/// Items a pending key holds at most. An append to a full key fails. +pub(crate) const MAX_PENDING_ITEMS: usize = 16; + +/// Generations per second. A generation is at least `now * 1000`, so a key +/// that expired and comes back never reuses a generation a row has applied. +const GENERATIONS_PER_SECOND: u64 = 1_000; + +/// The pending key of an EC ID. +#[must_use] +pub(crate) fn pending_key(ec_id: &str) -> String { + format!("pending:{ec_id}") +} + +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +struct KvPending { + #[serde(default)] + items: Vec, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +struct KvPendingItem { + /// The item's generation. + g: u64, + /// When the item was appended, Unix seconds. + at: u64, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + ch: Vec, + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + ps: BTreeMap, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +enum KvRevision { + Unset, + Set(String), +} + +impl From<&Revision> for KvRevision { + fn from(revision: &Revision) -> Self { + match revision { + Revision::Unset => Self::Unset, + Revision::Set(value) => Self::Set(value.as_str().to_owned()), + } + } +} + +impl From<&KvRevision> for Revision { + fn from(revision: &KvRevision) -> Self { + match revision { + KvRevision::Unset => Self::Unset, + KvRevision::Set(value) => Self::Set(RecordRevision::from_stored(value)), + } + } +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "k", rename_all = "snake_case")] +enum KvPendingChange { + FillMissing { + s: SourceDomain, + r: KvPartnerId, + }, + OwnerWrite { + s: SourceDomain, + r: KvPartnerId, + #[serde(default, skip_serializing_if = "Option::is_none")] + e: Option, + }, + OwnerRemove { + s: SourceDomain, + p: IdentityProviderId, + e: KvRevision, + }, + WriterRemove { + s: SourceDomain, + p: IdentityProviderId, + }, +} + +impl From<&PartnerIdChange> for KvPendingChange { + fn from(change: &PartnerIdChange) -> Self { + match change { + PartnerIdChange::FillMissing { source, record } => Self::FillMissing { + s: source.clone(), + r: record.clone(), + }, + PartnerIdChange::OwnerWrite { + source, + record, + expected, + } => Self::OwnerWrite { + s: source.clone(), + r: record.clone(), + e: expected.as_ref().map(KvRevision::from), + }, + PartnerIdChange::OwnerRemove { + source, + provider, + expected, + } => Self::OwnerRemove { + s: source.clone(), + p: provider.clone(), + e: KvRevision::from(expected), + }, + PartnerIdChange::WriterRemove { source, writer } => Self::WriterRemove { + s: source.clone(), + p: writer.clone(), + }, + } + } +} + +impl From<&KvPendingChange> for PartnerIdChange { + fn from(change: &KvPendingChange) -> Self { + match change { + KvPendingChange::FillMissing { s, r } => Self::FillMissing { + source: s.clone(), + record: r.clone(), + }, + KvPendingChange::OwnerWrite { s, r, e } => Self::OwnerWrite { + source: s.clone(), + record: r.clone(), + expected: e.as_ref().map(Revision::from), + }, + KvPendingChange::OwnerRemove { s, p, e } => Self::OwnerRemove { + source: s.clone(), + provider: p.clone(), + expected: Revision::from(e), + }, + KvPendingChange::WriterRemove { s, p } => Self::WriterRemove { + source: s.clone(), + writer: p.clone(), + }, + } + } +} + +/// The removal part of a provider's state change: its opt-out and clear. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +struct KvPendingState { + #[serde(default, skip_serializing_if = "Option::is_none")] + o: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + oc: Option, +} + +/// What the next writer merges: the pending items newer than its row's +/// watermark, and the generation to store as the new watermark. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct PendingMerge { + pub(crate) mutation: IdentityMutation, + pub(crate) generation: u64, +} + +fn to_item(mutation: &IdentityMutation, generation: u64, now: u64) -> KvPendingItem { + KvPendingItem { + g: generation, + at: now, + ch: mutation + .stand_alone_changes() + .iter() + .map(KvPendingChange::from) + .collect(), + ps: mutation + .removal_states() + .map(|(provider, state)| { + ( + provider.clone(), + KvPendingState { + o: state.opt_out.clone(), + oc: state.opt_out_clear, + }, + ) + }) + .collect(), + } +} + +/// Appends `mutation`'s stand-alone changes and opt-outs to the EC ID's +/// pending key with a CAS, retrying a generation mismatch. Items older than +/// [`PENDING_TTL`] are dropped first. +/// +/// Returns `false` when the item was not stored: the key is full, unreadable +/// or every attempt lost. The caller then counts what it held as lost. +#[must_use] +pub(crate) fn append_pending( + graph: &KvIdentityGraph, + ec_id: &str, + mutation: &IdentityMutation, + now: u64, +) -> bool { + let key = pending_key(ec_id); + for _attempt in 0..MAX_CAS_RETRIES { + let (mut pending, mode) = match graph.lookup_key(&key) { + Ok(Some(lookup)) => match serde_json::from_slice::(&lookup.body) { + Ok(pending) => (pending, EcKvWriteMode::IfGenerationMatch(lookup.generation)), + Err(error) => { + log::warn!("Pending key for '{}' is unreadable: {error}", log_id(ec_id)); + return false; + } + }, + Ok(None) => (KvPending::default(), EcKvWriteMode::Add), + Err(error) => { + log::warn!( + "Pending key for '{}' could not be read: {}", + log_id(ec_id), + error.current_context() + ); + return false; + } + }; + pending + .items + .retain(|item| item.at.saturating_add(PENDING_TTL.as_secs()) > now); + if pending.items.len() >= MAX_PENDING_ITEMS { + log::warn!("Pending key for '{}' is full", log_id(ec_id)); + return false; + } + let last = pending.items.last().map_or(0, |item| item.g); + let generation = last + .saturating_add(1) + .max(now.saturating_mul(GENERATIONS_PER_SECOND)); + pending.items.push(to_item(mutation, generation, now)); + let Ok(body) = serde_json::to_string(&pending) else { + return false; + }; + match graph.write_key(&key, &body, PENDING_TTL, mode) { + Ok(EcKvWriteOutcome::Written) => return true, + Ok(EcKvWriteOutcome::PreconditionFailed) => {} + Err(error) => { + log::warn!( + "Pending key for '{}' could not be written: {}", + log_id(ec_id), + error.current_context() + ); + return false; + } + } + } + false +} + +/// Reads the EC ID's pending key and builds the merge of every item newer +/// than `applied`, the row's watermark. `None` when there is nothing newer, +/// or the key cannot be read. +#[must_use] +pub(crate) fn load_pending( + graph: &KvIdentityGraph, + ec_id: &str, + applied: u64, +) -> Option { + let lookup = match graph.lookup_key(&pending_key(ec_id)) { + Ok(lookup) => lookup?, + Err(error) => { + log::warn!( + "Pending key for '{}' could not be read: {}", + log_id(ec_id), + error.current_context() + ); + return None; + } + }; + let pending: KvPending = match serde_json::from_slice(&lookup.body) { + Ok(pending) => pending, + Err(error) => { + log::warn!("Pending key for '{}' is unreadable: {error}", log_id(ec_id)); + return None; + } + }; + let mut mutation = IdentityMutation::default(); + let mut generation = applied; + for item in pending.items.iter().filter(|item| item.g > applied) { + for change in &item.ch { + mutation.add_change(PartnerIdChange::from(change)); + } + for (provider, state) in &item.ps { + mutation.merge_state( + provider.clone(), + ProviderStateChange { + opt_out: state.o.clone(), + opt_out_clear: state.oc, + ..ProviderStateChange::default() + }, + ); + } + generation = generation.max(item.g); + } + (generation > applied).then_some(PendingMerge { + mutation, + generation, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::test_support::{pid, record_for_test, source}; + use crate::ec::kv_backend::test_support::InMemoryEcKv; + + const EC_ID: &str = "0000000000000000000000000000000000000000000000000000000000000000.ABC123"; + + fn removal(name: &str) -> IdentityMutation { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::WriterRemove { + source: source(name), + writer: pid("lockr"), + }); + mutation + } + + #[test] + fn appended_items_merge_once_past_the_watermark() { + let graph = KvIdentityGraph::new(InMemoryEcKv::new("ec-store")); + let mut opt_out = removal("a.example.com"); + opt_out.merge_state( + pid("lockr"), + ProviderStateChange { + opt_out: Some(KvOptOut { + at: 90, + fp: "1".to_owned(), + }), + ..ProviderStateChange::default() + }, + ); + + assert!( + append_pending(&graph, EC_ID, &opt_out, 100), + "should store the first item" + ); + assert!( + append_pending(&graph, EC_ID, &removal("b.example.com"), 100), + "should store the second item" + ); + + let merge = load_pending(&graph, EC_ID, 0).expect("should merge both items"); + assert_eq!( + merge.generation, + 100 * GENERATIONS_PER_SECOND + 1, + "should hand back the newest generation" + ); + assert_eq!( + merge.mutation.removal_labels().len(), + 3, + "should merge both removals and the opt-out" + ); + let first_only = load_pending(&graph, EC_ID, 100 * GENERATIONS_PER_SECOND) + .expect("should merge the newer item"); + assert_eq!( + first_only.mutation.removal_labels(), + vec![(pid("lockr"), Some(source("b.example.com")))], + "should skip an item the watermark covers" + ); + assert!( + load_pending(&graph, EC_ID, merge.generation).is_none(), + "should merge nothing once the watermark covers every item" + ); + } + + #[test] + fn a_capture_write_round_trips_with_its_expected_revision() { + let graph = KvIdentityGraph::new(InMemoryEcKv::new("ec-store")); + let mut capture = IdentityMutation::default(); + let change = PartnerIdChange::OwnerWrite { + source: source("a.example.com"), + record: record_for_test("uid-1", Some("lockr"), 100), + expected: Some(Revision::Set(RecordRevision::from_stored( + "0123456789abcdef", + ))), + }; + capture.add_change(change.clone()); + + assert!( + append_pending(&graph, EC_ID, &capture, 100), + "should store the capture" + ); + + let merge = load_pending(&graph, EC_ID, 0).expect("should merge the capture"); + assert_eq!( + merge.mutation.stand_alone_changes(), + std::slice::from_ref(&change), + "should give back the change it stored" + ); + } + + #[test] + fn a_full_key_refuses_and_expired_items_make_room() { + let graph = KvIdentityGraph::new(InMemoryEcKv::new("ec-store")); + for _ in 0..MAX_PENDING_ITEMS { + assert!( + append_pending(&graph, EC_ID, &removal("a.example.com"), 100), + "should store items up to the cap" + ); + } + + assert!( + !append_pending(&graph, EC_ID, &removal("a.example.com"), 100), + "should refuse an item past the cap" + ); + assert!( + append_pending( + &graph, + EC_ID, + &removal("a.example.com"), + 100 + PENDING_TTL.as_secs() + ), + "should drop expired items and store the new one" + ); + } + + #[test] + fn a_generation_never_falls_below_the_clock() { + let graph = KvIdentityGraph::new(InMemoryEcKv::new("ec-store")); + let applied_before_expiry = 500 * GENERATIONS_PER_SECOND + 3; + + assert!( + append_pending(&graph, EC_ID, &removal("a.example.com"), 600), + "should store the item" + ); + + assert!( + load_pending(&graph, EC_ID, applied_before_expiry).is_some(), + "should merge an item appended after a key expired and came back" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/policy.rs b/crates/trusted-server-core/src/ec/identity/policy.rs new file mode 100644 index 0000000000..51d08d14b2 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/policy.rs @@ -0,0 +1,856 @@ +//! Evaluates a provider's [`ConsentPolicy`] against a request's consent. + +use super::descriptor::{ConsentPolicy, UsOptOut}; +use crate::consent::types::PrivacyFlag; +use crate::consent::{ConsentContext, effective_tcf, tcf_applies}; + +/// Signals the consent context does not carry. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) struct PolicySignals { + /// `DNT: 1` was present. + pub(crate) dnt: bool, +} + +impl PolicySignals { + /// Reads the signals a request's headers carry. + #[must_use] + pub(crate) fn from_headers(headers: &http::HeaderMap) -> Self { + Self { + dnt: headers + .get("dnt") + .and_then(|value| value.to_str().ok()) + .is_some_and(|value| value.trim() == "1"), + } + } +} + +/// The highest TCF purpose the fingerprint has a bit for. +pub(crate) const FP_PURPOSES: u8 = 10; +/// Fingerprint bit of the vendor consent. Purposes 1 to 10 hold bits 0 to 9. +pub(crate) const FP_VENDOR: u32 = 10; +/// Fingerprint bit of the US sale opt-out. +pub(crate) const FP_US_SALE: u32 = 11; +/// Fingerprint bit of Global Privacy Control. +pub(crate) const FP_GPC: u32 = 12; +/// Fingerprint bit of `DNT: 1`. +const FP_DNT: u32 = 13; +/// Fingerprint bit of the US sharing opt-out. +pub(crate) const FP_US_SHARING: u32 = 14; +/// Fingerprint bit of the US targeted-advertising opt-out. +pub(crate) const FP_US_TARGETED: u32 = 15; +/// Fingerprint bit set when TCF applied and a TC string was decoded, so the +/// purpose and vendor bits record a grant or a denial rather than nothing. +pub(crate) const FP_TCF: u32 = 16; +/// The US opt-out bits: sale, sharing, targeted advertising and GPC. +pub(crate) const FP_US_OPT_OUT_MASK: u64 = + (1 << FP_US_SALE) | (1 << FP_US_SHARING) | (1 << FP_US_TARGETED) | (1 << FP_GPC); + +/// Whether the request carries a decoded US sale opt-out, from a GPP US +/// section, GPP section 6 or the US Privacy string. +fn us_sale_opt_out(consent: &ConsentContext) -> bool { + let section_6 = consent + .gpp + .as_ref() + .and_then(|gpp| gpp.us_privacy_section.as_ref()); + consent.gpp.as_ref().and_then(|gpp| gpp.us_sale_opt_out) == Some(true) + || section_6 + .into_iter() + .chain(consent.us_privacy.as_ref()) + .any(|usp| usp.opt_out_sale == PrivacyFlag::Yes) +} + +/// Whether the request carries a decoded US sharing opt-out from GPP. +fn us_sharing_opt_out(consent: &ConsentContext) -> bool { + consent.gpp.as_ref().and_then(|gpp| gpp.us_sharing_opt_out) == Some(true) +} + +/// Whether the request carries a decoded US targeted-advertising opt-out from +/// GPP. +fn us_targeted_advertising_opt_out(consent: &ConsentContext) -> bool { + consent + .gpp + .as_ref() + .and_then(|gpp| gpp.us_targeted_advertising_opt_out) + == Some(true) +} + +/// Whether Global Privacy Control is set, by `Sec-GPC` or GPP's GPC field. +fn gpc_signal(consent: &ConsentContext) -> bool { + consent.gpc || consent.gpp.as_ref().and_then(|gpp| gpp.us_gpc) == Some(true) +} + +/// Whether GPP marks a known child without consent. +fn known_child_without_consent(consent: &ConsentContext) -> bool { + consent.gpp.as_ref().and_then(|gpp| gpp.us_known_child) == Some(true) +} + +/// The consent an opt-out's `fp` records, as a lowercase hex bitmask (spec 8). +/// +/// Each signal has a fixed bit, whatever the policy reads: +/// +/// - bits 0 to 9: TCF purposes 1 to 10, set when consent grants the purpose +/// - bit 10: the policy's vendor, set when consent grants it +/// - bit 11: the US sale opt-out, from `us_privacy`, a GPP US section or GPP +/// section 6 +/// - bit 12: GPC +/// - bit 13: DNT +/// - bit 14: the US sharing opt-out +/// - bit 15: the US targeted-advertising opt-out +/// +/// The purpose bits are 0 when TCF does not apply or no TCF consent is +/// present, so a request with no TCF consent and one that grants nothing +/// share the same purpose bits. +/// +/// A signal the policy does not read or relaxes stays 0. Later parts append +/// bits after these and keep these where they are. +#[must_use] +// Temporary: the request wiring, the first non-test caller, arrives in a later task. +#[allow(dead_code)] +pub(crate) fn fingerprint_bits( + policy: &ConsentPolicy, + consent: &ConsentContext, + signals: PolicySignals, +) -> String { + let mut bits: u64 = 0; + if !policy.is_eid_persistence_only() { + if tcf_applies(consent) + && let Some(tcf) = effective_tcf(consent) + { + bits |= 1 << FP_TCF; + for purpose in policy.tcf_purposes() { + if (1..=FP_PURPOSES).contains(purpose) + && tcf.has_purpose_consent(usize::from(*purpose)) + { + bits |= 1 << (purpose - 1); + } + } + if let Some(vendor) = policy.tcf_vendor() + && tcf.has_vendor_consent(vendor) + { + bits |= 1 << FP_VENDOR; + } + } + if !policy.allows_us_opt_out(UsOptOut::Sale) && us_sale_opt_out(consent) { + bits |= 1 << FP_US_SALE; + } + if !policy.allows_us_opt_out(UsOptOut::Sharing) && us_sharing_opt_out(consent) { + bits |= 1 << FP_US_SHARING; + } + if !policy.allows_us_opt_out(UsOptOut::TargetedAdvertising) + && us_targeted_advertising_opt_out(consent) + { + bits |= 1 << FP_US_TARGETED; + } + if !policy.allows_gpc() && gpc_signal(consent) { + bits |= 1 << FP_GPC; + } + if !policy.allows_dnt() && signals.dnt { + bits |= 1 << FP_DNT; + } + } + format!("{bits:x}") +} + +/// Whether `policy` allows a provider under this request's consent. The +/// global EC and EID-persistence gates are checked separately (eligibility +/// steps 1 and 2) and are not repeated here. +#[must_use] +pub(crate) fn policy_allows( + policy: &ConsentPolicy, + consent: &ConsentContext, + signals: PolicySignals, +) -> bool { + if policy.is_eid_persistence_only() { + return true; + } + // Fail closed: a raw string with no decoded form blocks. + if (consent.raw_tc_string.is_some() && consent.tcf.is_none()) + || (consent.raw_gpp_string.is_some() && consent.gpp.is_none()) + || (consent.raw_us_privacy.is_some() && consent.us_privacy.is_none()) + { + return false; + } + // A known child, or a US section whose opt-outs could not be read, blocks + // every provider whatever the policy relaxes. + if known_child_without_consent(consent) + || consent + .gpp + .as_ref() + .is_some_and(|gpp| gpp.us_section_decode_failed) + { + return false; + } + if tcf_applies(consent) { + // A TC string or a GPP TCF section. A header-only GPP string has + // neither, so TCF consent is still missing. + let Some(tcf) = effective_tcf(consent) else { + return false; + }; + if !policy + .tcf_purposes() + .iter() + .all(|purpose| tcf.has_purpose_consent(usize::from(*purpose))) + { + return false; + } + if let Some(vendor) = policy.tcf_vendor() + && !tcf.has_vendor_consent(vendor) + { + return false; + } + } + if !policy.allows_us_opt_out(UsOptOut::Sale) && us_sale_opt_out(consent) { + return false; + } + if !policy.allows_us_opt_out(UsOptOut::Sharing) && us_sharing_opt_out(consent) { + return false; + } + if !policy.allows_us_opt_out(UsOptOut::TargetedAdvertising) + && us_targeted_advertising_opt_out(consent) + { + return false; + } + if !policy.allows_gpc() && gpc_signal(consent) { + return false; + } + if !policy.allows_dnt() && signals.dnt { + return false; + } + true +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::consent::ConsentContext; + use crate::consent::gpp::decode_gpp_string; + use crate::consent::jurisdiction::Jurisdiction; + use crate::consent::types::{GppConsent, TcfConsent}; + use crate::consent::us_privacy::decode_us_privacy; + use crate::ec::identity::descriptor::{ConsentPolicy, UsOptOut}; + use crate::ec::identity::test_support::resolver_policy; + + #[derive(Default)] + struct Signals { + gdpr: bool, + purposes: Vec, + vendors: Vec, + us_sale_opt_out: bool, + gpc: bool, + raw_tc_without_decode: bool, + gpp_header_only: bool, + tcf_in_gpp: bool, + } + + fn tcf_consent(purposes: &[usize], vendors: &[u16]) -> TcfConsent { + let mut purpose_consents = vec![false; 24]; + for purpose in purposes { + purpose_consents[purpose - 1] = true; + } + TcfConsent { + version: 2, + cmp_id: 1, + cmp_version: 1, + consent_screen: 1, + consent_language: "EN".to_owned(), + vendor_list_version: 1, + tcf_policy_version: 4, + created_ds: 0, + last_updated_ds: 0, + purpose_consents, + purpose_legitimate_interests: vec![false; 24], + vendor_consents: vendors.to_vec(), + vendor_legitimate_interests: Vec::new(), + special_feature_opt_ins: vec![false; 12], + } + } + + // Takes the builder by value so call sites read as a struct literal. + #[allow(clippy::needless_pass_by_value)] + fn consent_with(signals: Signals) -> ConsentContext { + let mut consent = ConsentContext { + gdpr_applies: signals.gdpr, + gpc: signals.gpc, + jurisdiction: if signals.gdpr { + Jurisdiction::Gdpr + } else { + Jurisdiction::UsState("CA".to_owned()) + }, + ..ConsentContext::default() + }; + if signals.gdpr && !signals.tcf_in_gpp && !signals.purposes.is_empty() { + consent.tcf = Some(tcf_consent(&signals.purposes, &signals.vendors)); + } + if signals.us_sale_opt_out { + consent.raw_us_privacy = Some("1YYN".to_owned()); + consent.us_privacy = + Some(decode_us_privacy("1YYN").expect("should decode a US Privacy string")); + } + if signals.raw_tc_without_decode { + consent.raw_tc_string = Some("undecodable".to_owned()); + } + if signals.gpp_header_only || signals.tcf_in_gpp { + consent.raw_gpp_string = Some("DBAA".to_owned()); + let mut gpp = + decode_gpp_string("DBAA").expect("should decode a header-only GPP string"); + if signals.tcf_in_gpp { + gpp.eu_tcf = Some(tcf_consent(&signals.purposes, &signals.vendors)); + } + consent.gpp = Some(gpp); + consent.gpp_section_ids = Some(Vec::new()); + } + consent + } + + #[test] + fn strict_blocks_each_signal_it_decodes() { + let strict = resolver_policy(); + let allowed = consent_with(Signals::default()); + assert!( + policy_allows(&strict, &allowed, PolicySignals { dnt: false }), + "should allow a request with no signal set" + ); + + for (name, signals, dnt) in [ + ( + "gdpr without purpose 4", + Signals { + gdpr: true, + purposes: vec![1], + ..Signals::default() + }, + false, + ), + ( + "us sale opt-out", + Signals { + us_sale_opt_out: true, + ..Signals::default() + }, + false, + ), + ( + "gpc", + Signals { + gpc: true, + ..Signals::default() + }, + false, + ), + ("dnt", Signals::default(), true), + ( + "undecodable tc string", + Signals { + raw_tc_without_decode: true, + ..Signals::default() + }, + false, + ), + ] { + assert!( + !policy_allows(&strict, &consent_with(signals), PolicySignals { dnt }), + "should block under strict: {name}" + ); + } + } + + #[test] + fn relaxations_and_vendor_consent() { + let gpc = consent_with(Signals { + gpc: true, + ..Signals::default() + }); + assert!( + policy_allows( + &resolver_policy().allowing_gpc(), + &gpc, + PolicySignals { dnt: false } + ), + "should allow GPC when the policy relaxes it" + ); + let sale = consent_with(Signals { + us_sale_opt_out: true, + ..Signals::default() + }); + assert!( + policy_allows( + &resolver_policy().allowing_us_opt_out(UsOptOut::Sale), + &sale, + PolicySignals { dnt: false } + ), + "should allow a sale opt-out when the policy relaxes it" + ); + let gdpr = consent_with(Signals { + gdpr: true, + purposes: vec![1, 4], + vendors: vec![9999], + ..Signals::default() + }); + assert!( + policy_allows( + &resolver_policy().with_tcf_vendor(9999), + &gdpr, + PolicySignals { dnt: false } + ), + "should allow a granted vendor" + ); + assert!( + !policy_allows( + &resolver_policy().with_tcf_vendor(1234), + &gdpr, + PolicySignals { dnt: false } + ), + "should block a vendor without consent" + ); + } + + #[test] + fn eid_persistence_only_ignores_provider_signals() { + let consent = consent_with(Signals { + gpc: true, + us_sale_opt_out: true, + ..Signals::default() + }); + assert!( + policy_allows( + &ConsentPolicy::eid_persistence_only(), + &consent, + PolicySignals { dnt: true } + ), + "should check only the global gates for generic pull's parity policy, which the coordinator applies separately" + ); + } + + #[test] + fn header_only_gpp_does_not_block_strict_where_tcf_does_not_apply() { + let header_only = consent_with(Signals { + gpp_header_only: true, + ..Signals::default() + }); + assert!( + policy_allows( + &resolver_policy(), + &header_only, + PolicySignals { dnt: false } + ), + "should not block on a decoded GPP string with no applicable section" + ); + + let under_gdpr = consent_with(Signals { + gdpr: true, + gpp_header_only: true, + ..Signals::default() + }); + assert!( + !policy_allows( + &resolver_policy(), + &under_gdpr, + PolicySignals { dnt: false } + ), + "should still need TCF consent where TCF applies" + ); + + let gpp_tcf = consent_with(Signals { + gdpr: true, + purposes: vec![1, 4], + tcf_in_gpp: true, + ..Signals::default() + }); + assert!( + policy_allows(&resolver_policy(), &gpp_tcf, PolicySignals { dnt: false }), + "should accept TCF consent from a GPP TCF section" + ); + + let mut undecodable = consent_with(Signals::default()); + undecodable.raw_gpp_string = Some("not-a-gpp-string".to_owned()); + assert!( + !policy_allows( + &resolver_policy(), + &undecodable, + PolicySignals { dnt: false } + ), + "should fail closed on a GPP string that has no decoded form" + ); + } + + #[test] + fn corrupt_us_section_blocks_strict_and_ignores_relaxations() { + let corrupt = decode_gpp_string("DBABL~not-a-valid-usnat-section") + .expect("should decode the GPP header with a raw section"); + assert!( + corrupt.us_section_decode_failed, + "should flag the undecodable US section" + ); + let mut consent = consent_with(Signals::default()); + consent.raw_gpp_string = Some("corrupt".to_owned()); + consent.gpp = Some(corrupt); + let relaxed = resolver_policy() + .allowing_gpc() + .allowing_dnt() + .allowing_us_opt_out(UsOptOut::Sale) + .allowing_us_opt_out(UsOptOut::Sharing) + .allowing_us_opt_out(UsOptOut::TargetedAdvertising); + assert!( + !allows(&resolver_policy(), &consent), + "should block strict on an undecodable US section" + ); + assert!( + !allows(&relaxed, &consent), + "should block whatever the policy relaxes" + ); + assert!( + allows(&ConsentPolicy::eid_persistence_only(), &consent), + "should leave generic pull's parity policy at the global gates" + ); + let control = consent_with_gpp(|_| {}); + assert!( + allows(&resolver_policy(), &control), + "should allow a decodable GPP string under the same policy" + ); + } + + fn consent_with_gpp(edit: impl FnOnce(&mut GppConsent)) -> ConsentContext { + let mut consent = consent_with(Signals { + gpp_header_only: true, + ..Signals::default() + }); + edit(consent.gpp.as_mut().expect("should carry a decoded GPP")); + consent + } + + fn allows(policy: &ConsentPolicy, consent: &ConsentContext) -> bool { + policy_allows(policy, consent, PolicySignals { dnt: false }) + } + + #[test] + fn strict_blocks_sharing_targeted_gpc_field_and_known_child() { + let strict = resolver_policy(); + let control = consent_with_gpp(|gpp| { + gpp.us_sharing_opt_out = Some(false); + gpp.us_targeted_advertising_opt_out = Some(false); + gpp.us_gpc = Some(false); + gpp.us_known_child = Some(false); + }); + assert!( + allows(&strict, &control), + "should allow a GPP string that sets none of the new signals" + ); + + let sharing = consent_with_gpp(|gpp| gpp.us_sharing_opt_out = Some(true)); + assert!( + !allows(&strict, &sharing), + "should block on the sharing opt-out" + ); + assert!( + allows( + &resolver_policy().allowing_us_opt_out(UsOptOut::Sharing), + &sharing + ), + "should allow the sharing opt-out when the policy relaxes it" + ); + assert!( + !allows( + &resolver_policy().allowing_us_opt_out(UsOptOut::TargetedAdvertising), + &sharing + ), + "should keep blocking sharing when only another opt-out is relaxed" + ); + + let targeted = consent_with_gpp(|gpp| gpp.us_targeted_advertising_opt_out = Some(true)); + assert!( + !allows(&strict, &targeted), + "should block on the targeted-advertising opt-out" + ); + assert!( + allows( + &resolver_policy().allowing_us_opt_out(UsOptOut::TargetedAdvertising), + &targeted + ), + "should allow the targeted-advertising opt-out when the policy relaxes it" + ); + + let gpc = consent_with_gpp(|gpp| gpp.us_gpc = Some(true)); + assert!(!allows(&strict, &gpc), "should block on GPP's GPC field"); + assert!( + allows(&resolver_policy().allowing_gpc(), &gpc), + "should allow GPP's GPC field when the policy relaxes GPC" + ); + + let child = consent_with_gpp(|gpp| gpp.us_known_child = Some(true)); + let everything_relaxed = resolver_policy() + .allowing_gpc() + .allowing_dnt() + .allowing_us_opt_out(UsOptOut::Sale) + .allowing_us_opt_out(UsOptOut::Sharing) + .allowing_us_opt_out(UsOptOut::TargetedAdvertising); + assert!(!allows(&strict, &child), "should block a known child"); + assert!( + !allows(&everything_relaxed, &child), + "should block a known child whatever the policy relaxes" + ); + assert!( + allows(&everything_relaxed, &control), + "should allow the control under the same relaxations" + ); + assert!( + allows(&ConsentPolicy::eid_persistence_only(), &child), + "should leave generic pull's parity policy at the global gates only" + ); + } + + #[test] + fn targeted_advertising_opt_out_without_sale_opt_out_blocks_strict() { + let consent = consent_with_gpp(|gpp| { + gpp.us_sale_opt_out = Some(false); + gpp.us_targeted_advertising_opt_out = Some(true); + }); + assert!( + !allows(&resolver_policy(), &consent), + "should block on targeted advertising alone" + ); + let sale_only = consent_with_gpp(|gpp| gpp.us_sale_opt_out = Some(false)); + assert!( + allows(&resolver_policy(), &sale_only), + "should allow when nothing opts out" + ); + } + + #[test] + fn section_6_sale_opt_out_inside_gpp_blocks_strict() { + let opted_out = consent_with_gpp(|gpp| { + gpp.us_privacy_section = Some(decode_us_privacy("1YYN").expect("should decode")); + }); + assert!( + !allows(&resolver_policy(), &opted_out), + "should block on a sale opt-out in GPP section 6" + ); + assert!( + allows( + &resolver_policy().allowing_us_opt_out(UsOptOut::Sale), + &opted_out + ), + "should allow it when the policy relaxes the sale opt-out" + ); + let not_opted_out = consent_with_gpp(|gpp| { + gpp.us_privacy_section = Some(decode_us_privacy("1YNN").expect("should decode")); + }); + assert!( + allows(&resolver_policy(), ¬_opted_out), + "should allow a section 6 string that does not opt out" + ); + } + + #[test] + fn strict_fails_closed_on_raw_gpp_and_raw_us_privacy_without_a_decoded_form() { + let mut raw_gpp = consent_with(Signals::default()); + raw_gpp.raw_gpp_string = Some("DBABTA~1YNN".to_owned()); + assert!( + !allows(&resolver_policy(), &raw_gpp), + "should block a raw GPP string with no decoded form" + ); + raw_gpp.gpp = Some(decode_gpp_string("DBABTA~1YNN").expect("should decode")); + assert!( + allows(&resolver_policy(), &raw_gpp), + "should allow the same string once it decodes" + ); + + let mut raw_usp = consent_with(Signals::default()); + raw_usp.raw_us_privacy = Some("1YNN".to_owned()); + assert!( + !allows(&resolver_policy(), &raw_usp), + "should block a raw US Privacy string with no decoded form" + ); + raw_usp.us_privacy = Some(decode_us_privacy("1YNN").expect("should decode")); + assert!( + allows(&resolver_policy(), &raw_usp), + "should allow the same string once it decodes" + ); + } + + #[test] + fn custom_tcf_purposes_and_gpp_vendor_consent_are_enforced() { + let gdpr = consent_with(Signals { + gdpr: true, + purposes: vec![1, 4], + vendors: vec![9999], + ..Signals::default() + }); + assert!( + allows(&resolver_policy(), &gdpr), + "should allow purposes 1 and 4 under strict" + ); + assert!( + !allows(&resolver_policy().with_tcf_purposes([7]), &gdpr), + "should block when a custom required purpose is not granted" + ); + let with_seven = consent_with(Signals { + gdpr: true, + purposes: vec![1, 4, 7], + ..Signals::default() + }); + assert!( + allows(&resolver_policy().with_tcf_purposes([7]), &with_seven), + "should allow when the custom required purpose is granted" + ); + + let gpp_tcf = consent_with(Signals { + gdpr: true, + purposes: vec![1, 4], + vendors: vec![9999], + tcf_in_gpp: true, + ..Signals::default() + }); + assert!( + allows(&resolver_policy().with_tcf_vendor(9999), &gpp_tcf), + "should allow a vendor granted in the GPP TCF section" + ); + assert!( + !allows(&resolver_policy().with_tcf_vendor(1234), &gpp_tcf), + "should block a vendor the GPP TCF section does not grant" + ); + } + + fn fingerprint(policy: &ConsentPolicy, consent: &ConsentContext) -> String { + fingerprint_bits(policy, consent, PolicySignals::default()) + } + + #[test] + fn fingerprint_is_stable_across_cmp_resaves_and_changes_with_a_purpose() { + let first = { + let mut consent = consent_with(Signals { + gdpr: true, + purposes: vec![1, 4], + ..Signals::default() + }); + consent.raw_tc_string = Some("tc-string-one".to_owned()); + consent + }; + let resaved = { + let mut consent = first.clone(); + consent.raw_tc_string = Some("tc-string-two".to_owned()); + let tcf = consent.tcf.as_mut().expect("should carry TCF"); + tcf.last_updated_ds = 1_700_000_000; + tcf.created_ds = 1_600_000_000; + tcf.cmp_version = 9; + consent + }; + let without_purpose_four = consent_with(Signals { + gdpr: true, + purposes: vec![1], + ..Signals::default() + }); + let strict = resolver_policy(); + + assert_eq!( + fingerprint(&strict, &first), + fingerprint(&strict, &resaved), + "should give the same bits after a CMP resave with the same decisions" + ); + assert_eq!( + fingerprint(&strict, &first), + "10009", + "should set bits 0 and 3 for purposes 1 and 4, and bit 16 for the decoded TC string" + ); + assert_ne!( + fingerprint(&strict, &first), + fingerprint(&strict, &without_purpose_four), + "should change when purpose 4 is dropped" + ); + assert_eq!( + fingerprint(&strict, &without_purpose_four), + "10001", + "should keep only bit 0 and the TCF bit when purpose 4 is dropped" + ); + } + + #[test] + fn fingerprint_bits_name_each_opt_out_separately() { + let strict = resolver_policy(); + let bits = |consent: &ConsentContext| { + u64::from_str_radix(&fingerprint(&strict, consent), 16).expect("should parse as hex") + }; + let neither = consent_with_gpp(|_| {}); + let sharing = consent_with_gpp(|gpp| gpp.us_sharing_opt_out = Some(true)); + let targeted = consent_with_gpp(|gpp| gpp.us_targeted_advertising_opt_out = Some(true)); + + assert_eq!(bits(&neither), 0, "should be zero with no opt-out"); + assert_eq!( + bits(&sharing), + 1 << FP_US_SHARING, + "should set only the sharing bit" + ); + assert_eq!( + bits(&targeted), + 1 << FP_US_TARGETED, + "should set only the targeted-advertising bit" + ); + assert_ne!( + bits(&sharing), + bits(&targeted), + "should tell the two opt-outs apart" + ); + + let relaxed = resolver_policy() + .allowing_us_opt_out(UsOptOut::Sharing) + .allowing_us_opt_out(UsOptOut::TargetedAdvertising); + assert_eq!( + fingerprint(&relaxed, &sharing), + "0", + "should leave a relaxed sharing opt-out at zero" + ); + assert_eq!( + fingerprint(&relaxed, &targeted), + "0", + "should leave a relaxed targeted-advertising opt-out at zero" + ); + } + + #[test] + fn fingerprint_names_gpp_gpc_sale_and_the_mask() { + let strict = resolver_policy(); + let gpp_gpc = consent_with_gpp(|gpp| gpp.us_gpc = Some(true)); + assert_eq!( + fingerprint(&strict, &gpp_gpc), + "1000", + "should set the existing GPC bit for GPP's GPC field" + ); + let section_6 = consent_with_gpp(|gpp| { + gpp.us_privacy_section = Some(decode_us_privacy("1YYN").expect("should decode")); + }); + assert_eq!( + fingerprint(&strict, §ion_6), + "800", + "should set the sale bit for a section 6 opt-out" + ); + assert_eq!( + (FP_US_SALE, FP_GPC, FP_US_SHARING, FP_US_TARGETED), + (11, 12, 14, 15), + "should keep each signal at its named position" + ); + assert_eq!( + FP_US_OPT_OUT_MASK, 0xd800, + "should cover sale, sharing, targeted advertising and GPC" + ); + } + + #[test] + fn fingerprint_keeps_part_1_bit_positions() { + let policy = resolver_policy() + .with_tcf_purposes([2, 10]) + .with_tcf_vendor(5); + let mut consent = consent_with(Signals { + gdpr: true, + purposes: (1..=10).collect(), + vendors: vec![5], + gpc: true, + us_sale_opt_out: true, + ..Signals::default() + }); + consent.gpc = true; + assert_eq!( + fingerprint_bits(&policy, &consent, PolicySignals { dnt: true }), + "13e0b", + "should keep purposes 1, 2, 4 and 10 at bits 0, 1, 3 and 9, vendor 10, sale 11, GPC 12, DNT 13 and TCF 16" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/provider.rs b/crates/trusted-server-core/src/ec/identity/provider.rs new file mode 100644 index 0000000000..923d45ca9f --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/provider.rs @@ -0,0 +1,156 @@ +//! The identity provider trait. + +use error_stack::Report; + +use super::descriptor::ProviderDescriptor; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::input::{CaptureInput, ResolveInput}; +use super::resolution::{IdentityProviderError, Resolution}; +use crate::settings::IntegrationConfig; + +/// A vendor that turns server-side request signals into EIDs. +/// +/// Only [`descriptor`](Self::descriptor) is required. A provider adds a +/// capability by declaring it in its descriptor and overriding the matching +/// methods: +/// +/// - active resolution: [`Resolves`](super::descriptor::Resolves), plus [`request`](Self::request) and +/// [`response`](Self::response) +/// - passive capture: [`Captures`](super::descriptor::Captures), plus [`capture`](Self::capture) +/// +/// Methods added to this trait later get default bodies, so existing +/// providers keep compiling. No method takes generic parameters, so the +/// trait stays object-safe. Every default body returns the +/// [`IdentityProviderError::not_implemented`] marker, so a capability that is +/// declared but not implemented is caught by the harness. +pub trait IdentityProvider: Send + Sync { + /// What this provider is, what it may write, what it needs and which + /// consent it requires. + fn descriptor(&self) -> &ProviderDescriptor; + + /// Builds the call for this request, or `None` to skip it this time. + /// + /// Skipping is not a failure: the coordinator waits `refresh_after_sec` + /// before asking again (section 8). + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::UndeclaredInput`] when it reads + /// something its descriptor did not declare, or another + /// [`IdentityProviderError`] when the request cannot be built. + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _ = input; + Err(IdentityProviderError::not_implemented("request")) + } + + /// Turns the buffered response to [`request`](Self::request) into a + /// resolution. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Rejected`] for a refusal and + /// [`IdentityProviderError::Malformed`] for a body it cannot parse. + fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, + ) -> Result> { + let _ = (response, input); + Err(IdentityProviderError::not_implemented("response")) + } + + /// Reads IDs from a response TS proxied for this provider, with no + /// outbound call. [`Resolution::no_match`] means the response held + /// nothing to capture, and costs nothing: capture never backs off. + /// + /// # Errors + /// + /// Returns an [`IdentityProviderError`] only for a provider bug. A body + /// it does not recognise is `Ok(Resolution::no_match())`, not an error. + fn capture( + &self, + input: &CaptureInput<'_>, + ) -> Result> { + let _ = input; + Err(IdentityProviderError::not_implemented("capture")) + } +} + +/// Implemented by each identity integration's config type. +/// +/// Builds the provider's descriptor from config alone, with no secrets, so +/// `ts config validate` checks ownership against the same descriptor the +/// running provider uses. +pub trait IdentityProviderConfig: IntegrationConfig { + /// Describes the provider this config enables. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Config`] when the config cannot + /// describe a valid provider. + fn build_descriptor(&self) -> Result>; +} + +// Keeps the trait object-safe. Fails to compile if a method gains generics. +const _: Option<&dyn IdentityProvider> = None; +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use http::Method; + + use super::*; + use crate::ec::identity::descriptor::Captures; + use crate::ec::identity::exchange::ProviderResponse; + use crate::ec::identity::ids::UnixTime; + use crate::ec::identity::input::{CaptureInput, ProviderConsent}; + + struct DescriptorOnly(ProviderDescriptor); + + impl IdentityProvider for DescriptorOnly { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } + } + + #[test] + fn default_capability_methods_return_the_not_implemented_marker() { + let provider = DescriptorOnly( + ProviderDescriptor::builder("p".parse().expect("should parse the provider id")) + .with_source("example.com".parse().expect("should parse the source")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .build() + .expect("should build"), + ); + let boxed: Box = Box::new(provider); + let response = ProviderResponse::for_test(http::StatusCode::OK, b"{}"); + let consent = ProviderConsent::default(); + let stored = BTreeMap::new(); + let input = CaptureInput::new( + boxed.descriptor(), + "/integrations/p/api", + &response, + &stored, + &consent, + "www.example.com", + UnixTime::from_secs(1), + ); + + let error = boxed + .capture(&input) + .expect_err("should return the not-implemented marker"); + assert!( + error.current_context().is_not_implemented(), + "should carry the marker" + ); + assert!( + !IdentityProviderError::config("other") + .current_context() + .is_not_implemented(), + "should not treat an ordinary Config error as the marker" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/removal.rs b/crates/trusted-server-core/src/ec/identity/removal.rs new file mode 100644 index 0000000000..b5d52f3c48 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/removal.rs @@ -0,0 +1,575 @@ +//! The shared removal write: a removal-only mutation written with the +//! withdrawal path's CAS loop (spec 5.2 phase 4, 5.3). + +use std::time::Duration; + +use super::ids::SourceDomain; +use super::mutation::{ApplyContext, IdentityMutation, apply_removals_only}; +use crate::ec::kv::{KvIdentityGraph, MAX_CAS_RETRIES}; +use crate::ec::kv_backend::{EcKvRateLimited, EcKvWriteOutcome}; +use crate::ec::{EcKvSnapshot, log_id}; + +/// Writes the removal write makes at most, as the withdrawal path does. +pub(crate) const REMOVAL_WRITE_ATTEMPTS: u32 = MAX_CAS_RETRIES; + +/// How long Fastly's per-key write limit lasts: about one write per second +/// to one key. +pub(crate) const PER_KEY_WRITE_WINDOW: Duration = Duration::from_secs(1); + +/// When a removal write runs, which decides whether it may wait out the +/// per-key write limit (spec 5.2 phase 4, 5.3). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum RemovalTiming { + /// Before the response is sent. Never waits; a per-key limit ends the + /// attempt. + BeforeSend, + /// After the response is sent, right after a write to the same key + /// failed. Waits one per-key window first, and waits again and retries + /// after a per-key limit. + AfterSend, +} + +/// How a removal write ended. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RemovalWriteOutcome { + /// The removals are in KV. + Written, + /// Nothing to write: a tombstone, a missing row, or already applied. + NothingToWrite, + /// Not written. The caller may try once more after send. + Failed, +} + +/// The KV operations one removal write made. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) struct RemovalCost { + pub(crate) reads: u32, + pub(crate) writes: u32, +} + +/// Reads the row, applies only `removals` and writes them with a CAS, +/// retrying a generation mismatch, and after send a per-key limit too. +/// +/// Runs whatever the request's consent: a revocation can arrive when consent +/// is unusable. It does not log a lost removal; see [`log_removal_lost`]. +#[must_use] +pub(crate) fn write_removals( + graph: &KvIdentityGraph, + ec_id: &str, + removals: &IdentityMutation, + context: &ApplyContext<'_>, + timing: RemovalTiming, +) -> RemovalWriteOutcome { + write_removals_counted(graph, ec_id, removals, context, timing).0 +} + +/// As [`write_removals`], also returning the reads and writes it made, for +/// the request's measurement. +#[must_use] +pub(crate) fn write_removals_counted( + graph: &KvIdentityGraph, + ec_id: &str, + removals: &IdentityMutation, + context: &ApplyContext<'_>, + timing: RemovalTiming, +) -> (RemovalWriteOutcome, RemovalCost) { + let mut cost = RemovalCost::default(); + if timing == RemovalTiming::AfterSend { + graph.pause(PER_KEY_WRITE_WINDOW); + } + for _attempt in 0..REMOVAL_WRITE_ATTEMPTS { + cost.reads = cost.reads.saturating_add(1); + let (entry, generation) = match graph.load_snapshot(ec_id) { + EcKvSnapshot::Present { + entry, + generation: Some(generation), + .. + } => (entry, generation), + EcKvSnapshot::Missing { .. } => return (RemovalWriteOutcome::NothingToWrite, cost), + _ => return (RemovalWriteOutcome::Failed, cost), + }; + let Some(applied) = + apply_removals_only(removals, &entry, context).filter(|applied| applied.changed) + else { + return (RemovalWriteOutcome::NothingToWrite, cost); + }; + cost.writes = cost.writes.saturating_add(1); + match graph.write_if_generation(ec_id, &applied.entry, generation) { + Ok(EcKvWriteOutcome::Written) => return (RemovalWriteOutcome::Written, cost), + Ok(EcKvWriteOutcome::PreconditionFailed) => {} + Err(error) + if timing == RemovalTiming::AfterSend && error.contains::() => + { + log::debug!( + "Identity removal write for '{}' hit the per-key limit; waiting", + log_id(ec_id) + ); + graph.pause(PER_KEY_WRITE_WINDOW); + } + Err(error) => { + log::warn!( + "Identity removal write failed for '{}': {error:?}", + log_id(ec_id) + ); + return (RemovalWriteOutcome::Failed, cost); + } + } + } + log::warn!( + "Identity removal write for '{}': no write after {REMOVAL_WRITE_ATTEMPTS} attempts", + log_id(ec_id) + ); + (RemovalWriteOutcome::Failed, cost) +} + +/// Logs a removal that every attempt failed to write. Names providers and +/// sources only, never a UID. +pub(crate) fn log_removal_lost(ec_id: &str, removals: &IdentityMutation) { + for (provider, source) in removals.removal_labels() { + log::error!( + "Identity removal lost for '{}': provider '{provider}', source '{}'", + log_id(ec_id), + source.as_ref().map_or("*", SourceDomain::as_str), + ); + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + use std::sync::Mutex; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; + + use error_stack::Report; + + use super::*; + use crate::ec::identity::catalog::ProviderCatalog; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::mutation::{PartnerIdChange, ProviderStateChange}; + use crate::ec::identity::revision::{RecordRevision, Revision}; + use crate::ec::identity::test_support::fixed_provider; + use crate::ec::kv::KvIdentityGraph; + use crate::ec::kv_backend::test_support::InMemoryEcKv; + use crate::ec::kv_backend::{ + EcKvLookup, EcKvRateLimited, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome, + }; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId}; + use crate::error::TrustedServerError; + + const EC_ID: &str = "0000000000000000000000000000000000000000000000000000000000000000.ABC123"; + + /// Write counts and pauses the store keeps after seeding. + #[derive(Default)] + struct WriteCounts { + all: AtomicUsize, + generation_checked: AtomicUsize, + pauses: Mutex>, + } + + impl WriteCounts { + fn pauses(&self) -> Vec { + self.pauses.lock().expect("should lock the pauses").clone() + } + } + + /// Fails the first `rate_limited` generation-checked writes with a + /// per-key rate-limit error, then the next `conflicts` with + /// `PreconditionFailed`, or every generation-checked write with a plain + /// error when `error` is set. Counts every write and pause it sees. + struct ScriptedEcKv { + inner: InMemoryEcKv, + conflicts: Mutex, + rate_limited: Mutex, + error: bool, + counts: Arc, + } + + impl EcKvStore for ScriptedEcKv { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup(&self, key: &str) -> Result, Report> { + self.inner.lookup(key) + } + + fn key_exists(&self, key: &str) -> Result> { + self.inner.key_exists(key) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + self.counts.all.fetch_add(1, Ordering::SeqCst); + if matches!(write.mode, EcKvWriteMode::IfGenerationMatch(_)) { + self.counts + .generation_checked + .fetch_add(1, Ordering::SeqCst); + if self.error { + return Err(Report::new(TrustedServerError::KvStore { + store_name: self.inner.store_name().to_owned(), + message: "store unavailable".to_owned(), + })); + } + let mut limited = self.rate_limited.lock().expect("should lock the counter"); + if *limited > 0 { + *limited -= 1; + return Err(Report::new(TrustedServerError::KvStore { + store_name: self.inner.store_name().to_owned(), + message: "per-key write limit".to_owned(), + }) + .attach_opaque(EcKvRateLimited)); + } + drop(limited); + let mut remaining = self.conflicts.lock().expect("should lock the counter"); + if *remaining > 0 { + *remaining -= 1; + return Ok(EcKvWriteOutcome::PreconditionFailed); + } + } + self.inner.insert(key, write) + } + + fn list_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result, Report> { + self.inner.list_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + self.inner.delete(key) + } + + fn pause(&self, duration: Duration) { + self.counts + .pauses + .lock() + .expect("should lock the pauses") + .push(duration); + } + } + + /// Seeds `entry` with `create`, then wraps the store and zeroes the counts. + fn graph_with( + entry: Option<&KvEntry>, + conflicts: u32, + error: bool, + ) -> (KvIdentityGraph, Arc) { + graph_limited(entry, 0, conflicts, error) + } + + /// As [`graph_with`], rate-limiting the first `rate_limited` writes. + fn graph_limited( + entry: Option<&KvEntry>, + rate_limited: u32, + conflicts: u32, + error: bool, + ) -> (KvIdentityGraph, Arc) { + let counts = Arc::new(WriteCounts::default()); + let graph = KvIdentityGraph::new(ScriptedEcKv { + inner: InMemoryEcKv::new("removal-store"), + conflicts: Mutex::new(conflicts), + rate_limited: Mutex::new(rate_limited), + error, + counts: Arc::clone(&counts), + }); + if let Some(entry) = entry { + graph.create(EC_ID, entry).expect("should seed the row"); + } + counts.all.store(0, Ordering::SeqCst); + counts.generation_checked.store(0, Ordering::SeqCst); + (graph, counts) + } + + fn removal() -> IdentityMutation { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: "a.example.com".parse().expect("should parse the source"), + provider: "owner".parse().expect("should parse the provider ID"), + expected: Revision::Set(RecordRevision::from_stored("1111111111111111")), + }); + mutation + } + + fn live() -> KvEntry { + let mut entry = KvEntry::minimal("b.example.com", "x", 1); + entry.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + w: Some("owner".to_owned()), + rv: Some("1111111111111111".to_owned()), + ..KvPartnerId::new("u") + }, + ); + entry + } + + fn run(graph: &KvIdentityGraph, removals: &IdentityMutation) -> RemovalWriteOutcome { + run_at(graph, removals, RemovalTiming::BeforeSend) + } + + fn run_at( + graph: &KvIdentityGraph, + removals: &IdentityMutation, + timing: RemovalTiming, + ) -> RemovalWriteOutcome { + let random = SeededRandom::new(1); + let catalog = ProviderCatalog::new( + vec![fixed_provider("owner", &["a.example.com"])], + [( + "a.example.com".parse().expect("should parse the source"), + "owner".parse().expect("should parse the provider ID"), + )] + .into_iter() + .collect(), + ); + write_removals( + graph, + EC_ID, + removals, + &ApplyContext { + now: 1_000, + random: &random, + catalog: &catalog, + }, + timing, + ) + } + + #[test] + fn before_send_neither_waits_nor_retries_a_per_key_limit() { + let (graph, counts) = graph_limited(Some(&live()), 1, 0, false); + + assert_eq!( + run_at(&graph, &removal(), RemovalTiming::BeforeSend), + RemovalWriteOutcome::Failed, + "should end the attempt at the per-key limit" + ); + + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 1, + "should write once" + ); + assert!(counts.pauses().is_empty(), "should not wait before send"); + } + + #[test] + fn after_send_waits_the_per_key_window_before_its_first_write() { + let (graph, counts) = graph_with(Some(&live()), 0, false); + + assert_eq!( + run_at(&graph, &removal(), RemovalTiming::AfterSend), + RemovalWriteOutcome::Written, + "should write" + ); + + assert_eq!( + counts.pauses(), + vec![PER_KEY_WRITE_WINDOW], + "should wait one per-key window first" + ); + } + + #[test] + fn after_send_retries_a_per_key_limit_after_another_window() { + let (graph, counts) = graph_limited(Some(&live()), 1, 0, false); + + assert_eq!( + run_at(&graph, &removal(), RemovalTiming::AfterSend), + RemovalWriteOutcome::Written, + "should win after the rate-limited write" + ); + + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 2, + "should re-read and write again" + ); + assert_eq!( + counts.pauses(), + vec![PER_KEY_WRITE_WINDOW, PER_KEY_WRITE_WINDOW], + "should wait before the first write and after the limit" + ); + } + + #[test] + fn after_send_gives_up_on_a_per_key_limit_after_five_writes() { + let (graph, counts) = graph_limited(Some(&live()), 5, 0, false); + + assert_eq!( + run_at(&graph, &removal(), RemovalTiming::AfterSend), + RemovalWriteOutcome::Failed, + "should give up" + ); + + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 5, + "should stop at five writes" + ); + } + + #[test] + fn after_send_does_not_retry_other_errors() { + let (graph, counts) = graph_with(Some(&live()), 0, true); + + assert_eq!( + run_at(&graph, &removal(), RemovalTiming::AfterSend), + RemovalWriteOutcome::Failed, + "should stop at a non-CAS, non-limit error" + ); + + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 1, + "should write once" + ); + } + + #[test] + fn writes_the_removal_once_with_a_generation_check() { + let (graph, counts) = graph_with(Some(&live()), 0, false); + + assert_eq!( + run(&graph, &removal()), + RemovalWriteOutcome::Written, + "should write" + ); + + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 1, + "should write once, with IfGenerationMatch" + ); + let (entry, _) = graph + .get(EC_ID) + .expect("should read") + .expect("should hold the row"); + assert!( + !entry.ids.contains_key("a.example.com"), + "should remove the record" + ); + assert_eq!( + entry.ps["owner"].iv.len(), + 1, + "should hash the removed UID into iv" + ); + } + + #[test] + fn retries_only_a_generation_mismatch_up_to_five_writes() { + let (graph, counts) = graph_with(Some(&live()), 2, false); + assert_eq!( + run(&graph, &removal()), + RemovalWriteOutcome::Written, + "should win on the third write" + ); + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 3, + "should re-read and retry each conflict" + ); + + let (graph, counts) = graph_with(Some(&live()), 5, false); + assert_eq!( + run(&graph, &removal()), + RemovalWriteOutcome::Failed, + "should give up after five conflicts" + ); + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 5, + "should stop at five writes" + ); + + let (graph, counts) = graph_with(Some(&live()), 0, true); + assert_eq!( + run(&graph, &removal()), + RemovalWriteOutcome::Failed, + "should stop at a non-CAS error" + ); + assert_eq!( + counts.generation_checked.load(Ordering::SeqCst), + 1, + "should not retry a non-CAS error" + ); + } + + #[test] + fn writes_nothing_for_a_tombstone_a_missing_row_or_an_applied_removal() { + let (graph, counts) = graph_with(Some(&KvEntry::tombstone(1)), 0, false); + assert_eq!( + run(&graph, &removal()), + RemovalWriteOutcome::NothingToWrite, + "should skip a tombstone" + ); + assert_eq!( + counts.all.load(Ordering::SeqCst), + 0, + "should not write a tombstone" + ); + + let (graph, counts) = graph_with(None, 0, false); + assert_eq!( + run(&graph, &removal()), + RemovalWriteOutcome::NothingToWrite, + "should skip a missing row" + ); + assert_eq!( + counts.all.load(Ordering::SeqCst), + 0, + "should not write a missing row" + ); + + let (graph, counts) = + graph_with(Some(&KvEntry::minimal("b.example.com", "x", 1)), 0, false); + assert_eq!( + run(&graph, &removal()), + RemovalWriteOutcome::NothingToWrite, + "should skip an applied removal" + ); + assert_eq!( + counts.all.load(Ordering::SeqCst), + 0, + "should not write an applied removal" + ); + } + + #[test] + fn an_opt_out_and_its_clear_are_written_without_positive_changes() { + let (graph, _) = graph_with(Some(&live()), 0, false); + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + "owner".parse().expect("should parse the provider ID"), + ProviderStateChange { + opt_out: Some(KvOptOut { + at: 1_000, + fp: "1".to_owned(), + }), + ..ProviderStateChange::default() + }, + ); + + assert_eq!( + run(&graph, &mutation.removals()), + RemovalWriteOutcome::Written, + "should write the opt-out" + ); + + let (entry, _) = graph + .get(EC_ID) + .expect("should read") + .expect("should hold the row"); + assert!(entry.ps["owner"].o.is_some(), "should store o"); + assert!( + !entry.ids.contains_key("a.example.com"), + "should remove what the provider wrote" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs new file mode 100644 index 0000000000..2ff20fb30b --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -0,0 +1,392 @@ +//! Identity state one request carries on [`crate::ec::EcContext`] (spec 5.1). + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use url::Url; + +use super::ids::{IdentityProviderId, SourceDomain}; +use super::mutation::{IdentityMutation, is_opted_out}; +use super::runtime::IdentityRuntime; +use crate::ec::kv_types::KvEntry; +use crate::ec::rate_limiter::RateLimiter; + +/// The providers with a sticky opt-out (`ps.o`) in `entry`, unless a later +/// `oc` clears it (spec 7.1). +/// +/// KV is the only source: no cookie carries an opt-out. A key that is not a +/// valid provider ID cannot name a configured provider and is skipped. +#[must_use] +pub(crate) fn opted_out(entry: &KvEntry) -> BTreeSet { + entry + .ps + .keys() + .filter(|provider| is_opted_out(entry, provider)) + .filter_map(|provider| provider.parse().ok()) + .collect() +} + +/// What one request decides and collects for identity. +/// +/// Its [`Debug`](core::fmt::Debug) prints counts and flags only: the mutation +/// holds UIDs that must not reach a log (spec 9.3). +#[derive(Default, Clone)] +pub(crate) struct RequestIdentity { + writer: bool, + spa_writer: bool, + is_real_browser: bool, + user_agent: Option, + dnt: bool, + page_url: Option, + runtime: Option>, + rate_limiter: Option>, + runs_auction: bool, + hold_dispatched: bool, + mutation: IdentityMutation, + pending_removals: Option, + capture_writes: IdentityMutation, + removal_lost: u32, + removal_deferred: u32, + cookie_header_bytes: usize, + create_attempts: u32, + capture_unpersisted: BTreeMap<(IdentityProviderId, SourceDomain), u32>, + global_revoke: bool, + capture_kv_reads: u32, + capture_kv_writes: u32, +} + +impl core::fmt::Debug for RequestIdentity { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("RequestIdentity") + .field("writer", &self.writer) + .field("is_real_browser", &self.is_real_browser) + .field("mutation_changes", &self.mutation.change_count()) + .field("has_pending_removals", &self.pending_removals.is_some()) + .field("capture_writes", &self.capture_writes.change_count()) + .field("removal_lost", &self.removal_lost) + .field("removal_deferred", &self.removal_deferred) + .field("has_runtime", &self.runtime.is_some()) + .finish_non_exhaustive() + } +} + +// Temporary: the accessors and setters below are first used by later identity tasks. +#[allow(dead_code)] +impl RequestIdentity { + /// Whether this request is the one browser-driven writer. + #[must_use] + pub(crate) fn writer(&self) -> bool { + self.writer + } + + /// Whether the request passed the real-browser check. + #[must_use] + pub(crate) fn is_real_browser(&self) -> bool { + self.is_real_browser + } + + /// The request's `User-Agent`, captured in phase 1. + #[must_use] + pub(crate) fn user_agent(&self) -> Option<&str> { + self.user_agent.as_deref() + } + + /// Whether the request sent `DNT: 1`. + #[must_use] + pub(crate) fn dnt(&self) -> bool { + self.dnt + } + + /// The page URL without query or fragment. + #[must_use] + pub(crate) fn page_url(&self) -> Option<&Url> { + self.page_url.as_ref() + } + + /// The request's identity runtime, when the adapter set one. + #[must_use] + pub(crate) fn runtime(&self) -> Option<&Arc> { + self.runtime.as_ref() + } + + /// The limiter the provider burst cap and circuit breaker count in, when + /// the adapter set one. + #[must_use] + pub(crate) fn rate_limiter(&self) -> Option> { + self.rate_limiter.clone() + } + + /// Sets the limiter the hold dispatch counts in. + pub(crate) fn set_rate_limiter(&mut self, limiter: Arc) { + self.rate_limiter = Some(limiter); + } + + /// Whether this navigation runs an auction, so hold-mode providers may + /// be dispatched before it. + #[must_use] + pub(crate) fn runs_auction(&self) -> bool { + self.runs_auction + } + + /// Whether the hold dispatch considered every hold-mode provider on a + /// usable phase 1 row, so phase 4 must not run them again. + #[must_use] + pub(crate) fn hold_dispatched(&self) -> bool { + self.hold_dispatched + } + + /// Records that the hold dispatch ran on a usable phase 1 row. + pub(crate) fn set_hold_dispatched(&mut self) { + self.hold_dispatched = true; + } + + /// Records that this navigation runs an auction (spec 4.7 step 7). + pub(crate) fn set_runs_auction(&mut self, runs_auction: bool) { + self.runs_auction = runs_auction; + } + + /// The writer's one flush. + #[must_use] + pub(crate) fn mutation(&self) -> &IdentityMutation { + &self.mutation + } + + /// The writer's one flush, to fill. + pub(crate) fn mutation_mut(&mut self) -> &mut IdentityMutation { + &mut self.mutation + } + + /// A removal write that failed before send, to try once more after send. + #[must_use] + pub(crate) fn pending_removals(&self) -> Option<&IdentityMutation> { + self.pending_removals.as_ref() + } + + /// Keeps a removal write for one retry after send. + pub(crate) fn set_pending_removals(&mut self, removals: Option) { + self.pending_removals = removals; + } + + /// Positive capture writes a request that is not the writer keeps for + /// the pending key (spec 5.3, Pending key). + #[must_use] + pub(crate) fn capture_writes(&self) -> &IdentityMutation { + &self.capture_writes + } + + /// The capture writes, to fill or take. + pub(crate) fn capture_writes_mut(&mut self) -> &mut IdentityMutation { + &mut self.capture_writes + } + + /// Removals the row write missed that went to the pending key. + #[must_use] + pub(crate) fn removal_deferred(&self) -> u32 { + self.removal_deferred + } + + /// Counts one removal that went to the pending key. + pub(crate) fn record_removal_deferred(&mut self) { + self.removal_deferred = self.removal_deferred.saturating_add(1); + } + + /// Removals every attempt failed to write (spec 5.2 phase 4, 5.3). + #[must_use] + pub(crate) fn removal_lost(&self) -> u32 { + self.removal_lost + } + + /// Create writes the new-EC path attempted before routing. + #[must_use] + pub(crate) fn create_attempts(&self) -> u32 { + self.create_attempts + } + + /// Counts one create write of the new-EC path. + pub(crate) fn record_create_attempt(&mut self) { + self.create_attempts = self.create_attempts.saturating_add(1); + } + + /// The inbound `Cookie` header size in bytes. + #[must_use] + pub(crate) fn cookie_header_bytes(&self) -> usize { + self.cookie_header_bytes + } + + /// Records the inbound `Cookie` header size in bytes. + pub(crate) fn set_cookie_header_bytes(&mut self, bytes: usize) { + self.cookie_header_bytes = bytes; + } + + /// Counts one removal that every attempt failed to write. + pub(crate) fn record_removal_lost(&mut self) { + self.removal_lost = self.removal_lost.saturating_add(1); + } + + /// Positive capture IDs counted but never stored (spec 4.7). + #[must_use] + pub(crate) fn capture_unpersisted(&self) -> &BTreeMap<(IdentityProviderId, SourceDomain), u32> { + &self.capture_unpersisted + } + + /// Counts one positive capture ID that was not stored. + pub(crate) fn record_unpersisted( + &mut self, + provider: IdentityProviderId, + source: SourceDomain, + ) { + let count = self + .capture_unpersisted + .entry((provider, source)) + .or_insert(0); + *count = count.saturating_add(1); + } + + /// KV reads the capture made (its view read and the removal write's). + #[must_use] + pub(crate) fn capture_kv_reads(&self) -> u32 { + self.capture_kv_reads + } + + /// KV writes the capture removal write attempted. + #[must_use] + pub(crate) fn capture_kv_writes(&self) -> u32 { + self.capture_kv_writes + } + + /// Counts KV operations the capture made before send. + pub(crate) fn record_capture_kv(&mut self, reads: u32, writes: u32) { + self.capture_kv_reads = self.capture_kv_reads.saturating_add(reads); + self.capture_kv_writes = self.capture_kv_writes.saturating_add(writes); + } + + /// Whether a real browser's capture asked for a global revoke (spec 8). + #[must_use] + pub(crate) fn global_revoke_requested(&self) -> bool { + self.global_revoke + } + + /// Asks finalize to run the consent-withdrawal tombstone path. + pub(crate) fn request_global_revoke(&mut self) { + self.global_revoke = true; + } + + /// Sets the writer decision and the signals captured with it. + pub(crate) fn set_writer_decision(&mut self, decision: WriterDecision) { + self.writer = decision.writer; + self.is_real_browser = decision.is_real_browser; + self.user_agent = decision.user_agent; + self.dnt = decision.dnt; + self.page_url = decision.page_url; + } + + /// Whether the writer is a `/_ts/page-bids` call, which the interval + /// since the row's last writer flush paces and which runs no hold-mode + /// provider (spec 5.1). + #[must_use] + pub(crate) fn spa_writer(&self) -> bool { + self.spa_writer + } + + /// Marks the writer as the SPA writer. + pub(crate) fn set_spa_writer(&mut self, spa_writer: bool) { + self.spa_writer = spa_writer; + } + + /// Records only the real-browser flag, for a route that is never the writer. + pub(crate) fn set_real_browser(&mut self, is_real_browser: bool) { + self.is_real_browser = is_real_browser; + } + + /// Sets the runtime built once per settings load. + pub(crate) fn set_runtime(&mut self, runtime: Arc) { + self.runtime = Some(runtime); + } +} + +/// The writer decision and the signals captured with it. +pub(crate) struct WriterDecision { + pub(crate) writer: bool, + pub(crate) is_real_browser: bool, + pub(crate) user_agent: Option, + pub(crate) dnt: bool, + pub(crate) page_url: Option, +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::mutation::PartnerIdChange; + use crate::ec::identity::revision::Revision; + use crate::ec::identity::test_support::{pid, record_for_test, source}; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvProviderState}; + + #[test] + fn request_identity_debug_prints_no_uid() { + let mut identity = RequestIdentity::default(); + identity + .mutation_mut() + .add_change(PartnerIdChange::FillMissing { + source: source("ssp.example.com"), + record: record_for_test("secret-uid-1", None, 10), + }); + identity + .mutation_mut() + .add_change(PartnerIdChange::OwnerRemove { + source: source("dsp.example.com"), + provider: pid("owner"), + expected: Revision::Unset, + }); + + let printed = format!("{identity:?}"); + + assert!( + !printed.contains("secret-uid-1"), + "should print no UID: {printed}" + ); + assert!( + printed.contains("mutation_changes: 2"), + "should print the change count: {printed}" + ); + } + + #[test] + fn opted_out_reads_ps_o_and_honours_a_later_clear() { + let mut entry = KvEntry::minimal("ssp.example.com", "uid", 1); + let opt_out = || KvOptOut { + at: 100, + fp: String::new(), + }; + entry.ps.insert( + "a".to_owned(), + KvProviderState { + o: Some(opt_out()), + ..KvProviderState::default() + }, + ); + entry.ps.insert( + "b".to_owned(), + KvProviderState { + o: Some(opt_out()), + oc: Some(101), + ..KvProviderState::default() + }, + ); + entry.ps.insert( + "c".to_owned(), + KvProviderState { + oc: Some(50), + ..KvProviderState::default() + }, + ); + + let result = opted_out(&entry); + + assert_eq!( + result, + BTreeSet::from([pid("a")]), + "should keep only the provider whose opt-out is not cleared later" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/resolution.rs b/crates/trusted-server-core/src/ec/identity/resolution.rs new file mode 100644 index 0000000000..26ddea379c --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/resolution.rs @@ -0,0 +1,1286 @@ +//! Provider results and errors. + +use std::collections::BTreeMap; +use std::iter; +use std::time::Duration; + +use error_stack::Report; +use http::StatusCode; +use serde::Serialize; + +use super::descriptor::{Access, Input}; +use super::ids::{SourceDomain, UnixTime}; +use crate::ec::kv_types::{ + KvEid, KvUid, MAX_ENTRIES_PER_SOURCE, MAX_ENTRY_SET_BYTES, MAX_PARTNER_EXT_BYTES, + MAX_PROVIDER_STATE_BYTES, MAX_UID_LENGTH, MAX_UIDS_PER_SOURCE, entry_set_bytes, + is_valid_eid_field, is_valid_partner_ext, +}; + +/// Marker message of [`IdentityProviderError::not_implemented`]. The harness +/// matches on it to tell a missing override from a real failure. +pub(crate) const NOT_IMPLEMENTED_MARKER: &str = "capability declared but not implemented: "; + +/// A provider failure. Built through the constructors below. +#[derive(Debug, derive_more::Display)] +#[non_exhaustive] +pub enum IdentityProviderError { + /// Configuration or provider code makes the call impossible. + #[display("identity provider configuration error: {message}")] + Config { + /// What is wrong. Never contains a secret, UID or URL query. + message: String, + }, + /// The provider read something its descriptor did not declare. + #[display("`{access}` was read but not declared")] + UndeclaredInput { + /// What was read. + access: Access, + }, + /// A declared input is not available for this request. + #[display("input `{input}` is unavailable for this request")] + InputUnavailable { + /// The missing input. + input: Input, + }, + /// The provider refused the request. + #[display("provider rejected the request with status {status}")] + Rejected { + /// The response status. + status: StatusCode, + }, + /// The provider's response could not be parsed. + #[display("provider response could not be parsed")] + Malformed, + /// The stored private state does not decode as the provider's type. + #[display("stored provider state could not be decoded")] + InvalidState, + /// Set by the coordinator only. Providers never see transport failures. + #[display("provider request failed in transport")] + Transport, +} + +impl core::error::Error for IdentityProviderError {} + +impl IdentityProviderError { + /// A [`Self::Config`] error. + pub fn config(message: impl Into) -> Report { + Report::new(Self::Config { + message: message.into(), + }) + } + + /// A [`Self::Rejected`] error. + pub fn rejected(status: StatusCode) -> Report { + Report::new(Self::Rejected { status }) + } + + /// A [`Self::Malformed`] error. + pub fn malformed() -> Report { + Report::new(Self::Malformed) + } + + /// The default body of a capability method: a `Config` error with a + /// fixed marker message, so the harness can tell a missing override from + /// a real failure. + pub fn not_implemented(method: &'static str) -> Report { + Self::config(format!("{NOT_IMPLEMENTED_MARKER}{method}")) + } + + /// Whether this is the [`Self::not_implemented`] marker. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn is_not_implemented(&self) -> bool { + matches!(self, Self::Config { message } if message.starts_with(NOT_IMPLEMENTED_MARKER)) + } +} + +/// A time as a provider gave it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum When { + /// Relative to the request's `now`. + In(Duration), + /// Absolute. + At(UnixTime), +} + +impl When { + /// Resolves against `now`. + #[must_use] + pub(crate) fn resolve(self, now: UnixTime) -> UnixTime { + match self { + Self::In(delay) => now + delay, + Self::At(at) => at, + } + } +} + +fn ext_fits(ext: &serde_json::Value) -> bool { + serde_json::to_vec(ext).is_ok_and(|bytes| bytes.len() <= MAX_PARTNER_EXT_BYTES) +} + +/// One UID within a [`ResolvedEid`]. +#[derive(Clone, PartialEq)] +pub struct ResolvedUid { + id: String, + atype: Option, + ext: Option, +} + +impl core::fmt::Debug for ResolvedUid { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("ResolvedUid") + .field("id", &"[redacted]") + .field("atype", &self.atype) + .finish_non_exhaustive() + } +} + +impl ResolvedUid { + /// A UID with no `atype` or ext. + #[must_use] + pub fn new(id: impl Into) -> Self { + Self { + id: id.into(), + atype: None, + ext: None, + } + } + + /// Sets `atype`. + #[must_use] + pub fn with_atype(mut self, atype: i32) -> Self { + self.atype = Some(atype); + self + } + + /// UID-level ext. Over 2,048 bytes as compact JSON it is dropped and the + /// UID kept. + #[must_use] + pub fn with_ext(mut self, ext: serde_json::Value) -> Self { + let fits = ext_fits(&ext); + if !fits { + log::debug!("Dropping an oversized UID-level ext"); + } + self.ext = fits.then_some(ext); + self + } + + /// The UID. + #[must_use] + pub fn id(&self) -> &str { + &self.id + } + + /// `atype`. + #[must_use] + pub fn atype(&self) -> Option { + self.atype + } + + /// UID-level ext. + #[must_use] + pub fn ext(&self) -> Option<&serde_json::Value> { + self.ext.as_ref() + } +} + +/// One `OpenRTB` EID entry a provider returns: a source, its UIDs and its +/// EID-level fields. +#[derive(Clone, PartialEq)] +pub struct ResolvedEid { + source: SourceDomain, + uids: Vec, + inserter: Option, + matcher: Option, + mm: Option, + ext: Option, + expiry: Option, + refresh: Option, + refresh_until: Option, +} + +impl core::fmt::Debug for ResolvedEid { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("ResolvedEid") + .field("source", &self.source) + .field("uids", &self.uids) + .field("mm", &self.mm) + .field("expiry", &self.expiry) + .finish_non_exhaustive() + } +} + +impl ResolvedEid { + /// An entry with one UID, the common case. + #[must_use] + pub fn new(source: &SourceDomain, uid: ResolvedUid) -> Self { + Self { + source: source.clone(), + uids: vec![uid], + inserter: None, + matcher: None, + mm: None, + ext: None, + expiry: None, + refresh: None, + refresh_until: None, + } + } + + /// Adds another UID to this entry. Repeatable. + #[must_use] + pub fn with_uid(mut self, uid: ResolvedUid) -> Self { + self.uids.push(uid); + self + } + + /// Sets `inserter`. An entry that sets any of `inserter`, `matcher` and + /// `mm` states its EID-level fields completely, so the descriptor's + /// `with_provenance` default no longer applies to it (spec 4.3). + #[must_use] + pub fn with_inserter(mut self, inserter: &str) -> Self { + self.inserter = Some(inserter.to_owned()); + self + } + + /// Sets `matcher`. As [`Self::with_inserter`], it turns off the + /// descriptor's default for this entry. + #[must_use] + pub fn with_matcher(mut self, matcher: &str) -> Self { + self.matcher = Some(matcher.to_owned()); + self + } + + /// Sets `mm`. As [`Self::with_inserter`], it turns off the descriptor's + /// default for this entry. + #[must_use] + pub fn with_mm(mut self, mm: i32) -> Self { + self.mm = Some(mm); + self + } + + /// EID-level ext. Over 2,048 bytes as compact JSON it is dropped and the + /// entry kept. + #[must_use] + pub fn with_ext(mut self, ext: serde_json::Value) -> Self { + let fits = ext_fits(&ext); + if !fits { + log::debug!( + "Dropping an oversized EID-level ext for source {}", + self.source + ); + } + self.ext = fits.then_some(ext); + self + } + + /// Expires after `delay`. Timing applies to the source's whole entry + /// set, and the earliest time across its entries wins. + #[must_use] + pub fn with_expires_in(mut self, delay: Duration) -> Self { + self.expiry = Some(When::In(delay)); + self + } + + /// Expires at `at`. + #[must_use] + pub fn with_expires_at(mut self, at: UnixTime) -> Self { + self.expiry = Some(When::At(at)); + self + } + + /// Due for refresh after `delay`. + #[must_use] + pub fn with_refresh_in(mut self, delay: Duration) -> Self { + self.refresh = Some(When::In(delay)); + self + } + + /// Due for refresh at `at`. + #[must_use] + pub fn with_refresh_at(mut self, at: UnixTime) -> Self { + self.refresh = Some(When::At(at)); + self + } + + /// When this ID stops being refreshable. Only for a provider declared + /// `with_private_state()`; the coordinator ignores it otherwise. + #[must_use] + pub fn with_refresh_until(mut self, at: UnixTime) -> Self { + self.refresh_until = Some(at); + self + } + + /// The source. + #[must_use] + pub fn source(&self) -> &SourceDomain { + &self.source + } + + /// The UIDs, in order. + #[must_use] + pub fn uids(&self) -> &[ResolvedUid] { + &self.uids + } + + /// `inserter`, if set. + #[must_use] + pub fn inserter(&self) -> Option<&str> { + self.inserter.as_deref() + } + + /// `matcher`, if set. + #[must_use] + pub fn matcher(&self) -> Option<&str> { + self.matcher.as_deref() + } + + /// `mm`, if set. + #[must_use] + pub fn mm(&self) -> Option { + self.mm + } + + /// EID-level ext. + #[must_use] + pub fn ext(&self) -> Option<&serde_json::Value> { + self.ext.as_ref() + } + + /// The expiry as given. + #[must_use] + pub fn expiry(&self) -> Option { + self.expiry + } + + /// The refresh hint as given. + #[must_use] + pub fn refresh(&self) -> Option { + self.refresh + } + + /// The refresh limit. + #[must_use] + pub fn refresh_until(&self) -> Option { + self.refresh_until + } +} + +/// How a result changes the provider's private state. +#[derive(Debug, PartialEq, Eq)] +pub enum StateChange<'a> { + /// Leave it. + Keep, + /// Replace it with this JSON. + Replace(&'a str), + /// Remove it. + Clear, +} + +#[derive(Clone, PartialEq)] +enum StateOp { + Keep, + Replace(String), + Clear, + Rejected, +} + +impl core::fmt::Debug for StateOp { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + match self { + Self::Keep => formatter.write_str("Keep"), + Self::Replace(_) => formatter.write_str("Replace([redacted])"), + Self::Clear => formatter.write_str("Clear"), + Self::Rejected => formatter.write_str("Rejected"), + } + } +} + +/// What one provider call produced. Its `Debug` prints entries through +/// [`ResolvedEid`]'s redacting `Debug` and never prints provider state. +#[derive(Debug, Clone, PartialEq)] +pub struct Resolution { + eids: Vec, + invalidated: Vec, + opted_out: bool, + retry_in: Option, + state: StateOp, +} + +impl Resolution { + fn empty() -> Self { + Self { + eids: Vec::new(), + invalidated: Vec::new(), + opted_out: false, + retry_in: None, + state: StateOp::Keep, + } + } + + /// One or more EID entries. A source may appear in several entries, in + /// the order bidders should see them. The entries of one source form its + /// new entry set, which replaces the stored set whole for an owned + /// source. For an unowned source the set only fills an absent or expired + /// record (spec 7.2). A source the result does not name, or whose set + /// the limits discard, is left as it is. An empty set is `no_match`. + #[must_use] + pub fn found(eids: impl IntoIterator) -> Self { + Self { + eids: eids.into_iter().collect(), + ..Self::empty() + } + } + + /// Nothing recognized, or a capture held nothing. + #[must_use] + pub fn no_match() -> Self { + Self::empty() + } + + /// The stored ID for `source` is no longer valid. + #[must_use] + pub fn invalidate(source: &SourceDomain) -> Self { + Self { + invalidated: vec![source.clone()], + ..Self::empty() + } + } + + /// The provider reports the user opted out. + #[must_use] + pub fn opted_out() -> Self { + Self { + opted_out: true, + ..Self::empty() + } + } + + /// Also invalidates `source`. + #[must_use] + pub fn also_invalidate(mut self, source: &SourceDomain) -> Self { + if !self.invalidated.contains(source) { + self.invalidated.push(source.clone()); + } + self + } + + /// The earliest the coordinator may call again. + #[must_use] + pub fn with_retry_in(mut self, delay: Duration) -> Self { + self.retry_in = Some(delay); + self + } + + /// Replaces the provider's private state. Over 1,024 bytes of JSON, or + /// not serializable, the positive part of the result is rejected whole. + #[must_use] + pub fn replacing_state(mut self, state: &impl Serialize) -> Self { + self.state = match serde_json::to_string(state) { + Ok(json) if json.len() <= MAX_PROVIDER_STATE_BYTES => StateOp::Replace(json), + _ => StateOp::Rejected, + }; + self + } + + /// Clears the provider's private state. + #[must_use] + pub fn clearing_state(mut self) -> Self { + self.state = StateOp::Clear; + self + } + + /// The returned entries, as given. + #[must_use] + pub fn eids(&self) -> &[ResolvedEid] { + &self.eids + } + + /// Invalidated sources. + #[must_use] + pub fn invalidated(&self) -> &[SourceDomain] { + &self.invalidated + } + + /// Whether nothing was found, invalidated or opted out. + #[must_use] + pub fn is_no_match(&self) -> bool { + self.eids.is_empty() && self.invalidated.is_empty() && !self.opted_out + } + + /// Whether the provider reported an opt-out. + #[must_use] + pub fn is_opted_out(&self) -> bool { + self.opted_out + } + + /// The retry delay. + #[must_use] + pub fn retry_in(&self) -> Option { + self.retry_in + } + + /// The state change. + #[must_use] + pub fn state_change(&self) -> StateChange<'_> { + match &self.state { + StateOp::Keep | StateOp::Rejected => StateChange::Keep, + StateOp::Replace(json) => StateChange::Replace(json), + StateOp::Clear => StateChange::Clear, + } + } + + /// Whether `replacing_state` was given an oversized or unserializable + /// value. The coordinator then rejects the positive part (spec 4.5, 8). + #[must_use] + pub(crate) fn state_rejected(&self) -> bool { + self.state == StateOp::Rejected + } + + /// Whether this result changes state at all. + #[must_use] + pub(crate) fn changes_state(&self) -> bool { + self.state != StateOp::Keep + } +} + +/// The part of a result that a spec 4.5 limit dropped. +#[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] +pub(crate) enum DroppedPart { + /// A UID that is empty, blank or over `MAX_UID_LENGTH`. + #[display("uid")] + Uid, + /// A negative `atype`. + #[display("atype")] + Atype, + /// A UID-level ext that is not an object or is oversized. + #[display("uid ext")] + UidExt, + /// A blank or oversized `inserter`. + #[display("inserter")] + Inserter, + /// A blank or oversized `matcher`. + #[display("matcher")] + Matcher, + /// A negative `mm`. + #[display("mm")] + Mm, + /// An EID-level ext that is not an object or is oversized. + #[display("eid ext")] + EidExt, + /// An entry left with no UID. + #[display("entry without uid")] + EmptyEntry, + /// An entry equal to an earlier kept entry of its source. + #[display("duplicate entry")] + DuplicateEntry, + /// An entry past the source's first 8, or with no room left for a UID. + #[display("entry over the limit")] + ExtraEntry, + /// A UID past the source's first 32. + #[display("uid over the limit")] + ExtraUid, +} + +/// One dropped part, for the coordinator's log line (provider, source and +/// part only, never a value). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct EntryDrop { + pub(crate) source: SourceDomain, + pub(crate) part: DroppedPart, +} + +/// One source's new entry set, as it is stored, and its timing resolved +/// against `now`. +#[derive(Clone, PartialEq)] +pub(crate) struct EntrySet { + pub(crate) entries: Vec, + pub(crate) expires_at: Option, + pub(crate) refresh_at: Option, + pub(crate) refresh_until: Option, +} + +impl core::fmt::Debug for EntrySet { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("EntrySet") + .field("entries", &self.entries.len()) + .field("expires_at", &self.expires_at) + .field("refresh_at", &self.refresh_at) + .field("refresh_until", &self.refresh_until) + .finish() + } +} + +/// A result's entries, grouped by source and checked against spec 4.5. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct EntrySets { + /// Each source with at least one kept entry, under every limit. + pub(crate) sets: BTreeMap, + /// What the limits dropped, in order. + pub(crate) drops: Vec, + /// Sources whose set was over `MAX_ENTRY_SET_BYTES` and was discarded + /// whole. A stateful provider's positive part is then rejected. + pub(crate) oversized: Vec, +} + +impl EntrySets { + /// Whether the result named at least one source and every set was + /// discarded. Every entry that is not kept records a drop, so a result + /// with entries and no kept set always has a drop or an oversized set. + /// The coordinator handles such a result as `Malformed` (spec 4.5). + #[must_use] + pub(crate) fn all_discarded(&self) -> bool { + self.sets.is_empty() && (!self.drops.is_empty() || !self.oversized.is_empty()) + } +} + +fn earliest(current: Option, candidate: Option) -> Option { + match (current, candidate) { + (Some(current), Some(candidate)) => Some(current.min(candidate)), + (current, candidate) => current.or(candidate), + } +} + +fn clean_text( + value: Option<&str>, + part: DroppedPart, + dropped: &mut Vec, +) -> Option { + let value = value?; + if !is_valid_eid_field(value) { + dropped.push(part); + return None; + } + Some(value.to_owned()) +} + +fn clean_number( + value: Option, + part: DroppedPart, + dropped: &mut Vec, +) -> Option { + let value = value?; + if value < 0 { + dropped.push(part); + return None; + } + Some(value) +} + +fn clean_ext( + value: Option<&serde_json::Value>, + part: DroppedPart, + dropped: &mut Vec, +) -> Option { + let value = value?; + if !is_valid_partner_ext(value) { + dropped.push(part); + return None; + } + Some(value.clone()) +} + +fn clean_uid(uid: &ResolvedUid, dropped: &mut Vec) -> Option { + if uid.id.trim().is_empty() || uid.id.len() > MAX_UID_LENGTH { + dropped.push(DroppedPart::Uid); + return None; + } + Some(KvUid { + id: uid.id.clone(), + atype: clean_number(uid.atype, DroppedPart::Atype, dropped), + ext: clean_ext(uid.ext.as_ref(), DroppedPart::UidExt, dropped), + }) +} + +fn clean_entry(eid: &ResolvedEid, dropped: &mut Vec) -> Option { + let uids: Vec = eid + .uids + .iter() + .filter_map(|uid| clean_uid(uid, dropped)) + .collect(); + if uids.is_empty() { + return None; + } + Some(KvEid { + uids, + inserter: clean_text(eid.inserter.as_deref(), DroppedPart::Inserter, dropped), + matcher: clean_text(eid.matcher.as_deref(), DroppedPart::Matcher, dropped), + mm: clean_number(eid.mm, DroppedPart::Mm, dropped), + ext: clean_ext(eid.ext.as_ref(), DroppedPart::EidExt, dropped), + }) +} + +/// Groups `eids` by source, in order, and applies every limit of spec 4.5: +/// the browser parser's field and UID rules, then per-source caps the +/// browser parser does not have: duplicates, the first 8 entries and 32 UIDs +/// of a source, and `MAX_ENTRY_SET_BYTES` as [`entry_set_bytes`] measures +/// the stored set. A set over that is discarded whole, never trimmed: field +/// caps do not guarantee the set cap. Each set's `exp`, `ra` and `ru` are +/// the earliest its kept entries set; a dropped or discarded entry's times +/// do not count. +/// +/// Source claims and operator narrowing are the coordinator's, which drops +/// a set for an unclaimed source before using the result. +/// +/// # Performance +/// +/// Serializes each set once to measure it, and compares each entry with +/// the at most 8 entries already kept. +#[must_use] +pub(crate) fn entry_sets(eids: &[ResolvedEid], now: UnixTime) -> EntrySets { + let mut result = EntrySets::default(); + for eid in eids { + let mut dropped = Vec::new(); + let cleaned = clean_entry(eid, &mut dropped); + let set = result + .sets + .entry(eid.source.clone()) + .or_insert_with(|| EntrySet { + entries: Vec::new(), + expires_at: None, + refresh_at: None, + refresh_until: None, + }); + match cleaned { + None => dropped.push(DroppedPart::EmptyEntry), + Some(entry) if set.entries.contains(&entry) => { + dropped.push(DroppedPart::DuplicateEntry) + } + Some(_) if set.entries.len() >= MAX_ENTRIES_PER_SOURCE => { + dropped.push(DroppedPart::ExtraEntry) + } + Some(mut entry) => { + let used: usize = set.entries.iter().map(|kept| kept.uids.len()).sum(); + let room = MAX_UIDS_PER_SOURCE.saturating_sub(used); + if entry.uids.len() > room { + dropped.extend(iter::repeat_n( + DroppedPart::ExtraUid, + entry.uids.len() - room, + )); + entry.uids.truncate(room); + } + if entry.uids.is_empty() { + dropped.push(DroppedPart::ExtraEntry); + } else { + set.expires_at = + earliest(set.expires_at, eid.expiry.map(|when| when.resolve(now))); + set.refresh_at = + earliest(set.refresh_at, eid.refresh.map(|when| when.resolve(now))); + set.refresh_until = earliest(set.refresh_until, eid.refresh_until); + set.entries.push(entry); + } + } + } + result + .drops + .extend(dropped.into_iter().map(|part| EntryDrop { + source: eid.source.clone(), + part, + })); + } + result.sets.retain(|source, set| { + if set.entries.is_empty() { + return false; + } + let fits = entry_set_bytes(&set.entries) <= MAX_ENTRY_SET_BYTES; + if !fits { + result.oversized.push(source.clone()); + } + fits + }); + result +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use serde_json::json; + + use super::*; + use crate::ec::kv_types::MAX_EID_FIELD_BYTES; + + fn source(name: &str) -> SourceDomain { + name.parse().expect("should parse the source") + } + + fn uid(id: &str) -> ResolvedUid { + ResolvedUid::new(id) + } + + fn kv_uid(id: &str) -> KvUid { + KvUid { + id: id.to_owned(), + atype: None, + ext: None, + } + } + + fn now() -> UnixTime { + UnixTime::from_secs(1_000) + } + + fn parts(sets: &EntrySets) -> Vec { + sets.drops.iter().map(|drop| drop.part).collect() + } + + #[test] + fn empty_found_is_no_match() { + let resolution = Resolution::found([]); + assert!(resolution.is_no_match(), "should be no_match"); + assert!(resolution.eids().is_empty(), "should hold no entries"); + } + + #[test] + fn a_one_uid_entry_becomes_a_one_entry_set_without_eid_level_fields() { + let a = source("a.example.com"); + let resolution = Resolution::found([ResolvedEid::new(&a, uid("u").with_atype(3))]); + + let sets = entry_sets(resolution.eids(), now()); + + assert_eq!( + sets.sets[&a].entries, + vec![KvEid { + uids: vec![KvUid { + id: "u".to_owned(), + atype: Some(3), + ext: None + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + }], + "should produce the one-UID shape browser ingest stores" + ); + assert!( + sets.drops.is_empty() && sets.oversized.is_empty(), + "should drop nothing" + ); + } + + #[test] + fn entries_group_by_source_and_keep_every_uid_and_field_in_order() { + let a = source("a.example.com"); + let b = source("b.example.com"); + let first = ResolvedEid::new( + &a, + uid("a1") + .with_atype(1) + .with_ext(json!({ "rtiPartner": "TDID" })), + ) + .with_uid(uid("a2")) + .with_inserter("inserter.example.com") + .with_matcher("matcher.example.com") + .with_mm(2) + .with_ext(json!({ "segments": ["s"] })); + let other = ResolvedEid::new(&b, uid("b1")); + let second = ResolvedEid::new(&a, uid("a3")); + + let sets = entry_sets(&[first, other, second], now()); + + assert_eq!( + sets.sets[&a].entries, + vec![ + KvEid { + uids: vec![ + KvUid { + id: "a1".to_owned(), + atype: Some(1), + ext: Some(json!({ "rtiPartner": "TDID" })) + }, + kv_uid("a2"), + ], + inserter: Some("inserter.example.com".to_owned()), + matcher: Some("matcher.example.com".to_owned()), + mm: Some(2), + ext: Some(json!({ "segments": ["s"] })), + }, + KvEid { + uids: vec![kv_uid("a3")], + inserter: None, + matcher: None, + mm: None, + ext: None + }, + ], + "should keep a's entries in the order returned" + ); + assert_eq!(sets.sets[&b].entries.len(), 1, "should group b on its own"); + assert!(sets.drops.is_empty(), "should drop nothing"); + } + + #[test] + fn bad_fields_are_dropped_and_their_entries_kept() { + let a = source("a.example.com"); + let entry = ResolvedEid::new( + &a, + uid("u") + .with_atype(-1) + .with_ext(json!(["not", "an", "object"])), + ) + .with_inserter(" ") + .with_matcher(&"m".repeat(MAX_EID_FIELD_BYTES + 1)) + .with_mm(-2) + .with_ext(json!("not an object")); + + let sets = entry_sets(&[entry], now()); + + assert_eq!( + sets.sets[&a].entries, + vec![KvEid { + uids: vec![kv_uid("u")], + inserter: None, + matcher: None, + mm: None, + ext: None + }], + "should keep the entry without its bad fields" + ); + assert_eq!( + parts(&sets), + vec![ + DroppedPart::Atype, + DroppedPart::UidExt, + DroppedPart::Inserter, + DroppedPart::Matcher, + DroppedPart::Mm, + DroppedPart::EidExt, + ], + "should name each dropped field" + ); + } + + #[test] + fn bad_uids_are_dropped_and_an_entry_without_uids_is_discarded() { + let a = source("a.example.com"); + let mixed = ResolvedEid::new(&a, uid(" ")).with_uid(uid("kept")); + let empty = ResolvedEid::new(&a, uid(&"x".repeat(MAX_UID_LENGTH + 1))) + .with_inserter("inserter.example.com"); + + let sets = entry_sets(&[mixed, empty], now()); + + assert_eq!( + sets.sets[&a].entries.len(), + 1, + "should discard the entry left with no UID" + ); + assert_eq!( + sets.sets[&a].entries[0].uids, + vec![kv_uid("kept")], + "should keep the valid UID" + ); + assert_eq!( + parts(&sets), + vec![DroppedPart::Uid, DroppedPart::Uid, DroppedPart::EmptyEntry], + "should name the dropped UIDs and the discarded entry" + ); + } + + #[test] + fn a_source_with_no_kept_entry_is_not_named() { + let a = source("a.example.com"); + + let sets = entry_sets(&[ResolvedEid::new(&a, uid(""))], now()); + + assert!( + !sets.sets.contains_key(&a), + "should leave the stored set as it is" + ); + assert_eq!( + parts(&sets), + vec![DroppedPart::Uid, DroppedPart::EmptyEntry], + "should record the dropped UID and the discarded entry" + ); + assert!( + sets.all_discarded(), + "should report that every named set was discarded" + ); + } + + #[test] + fn all_discarded_only_when_nothing_was_kept() { + let a = source("a.example.com"); + let b = source("b.example.com"); + + let mixed = entry_sets( + &[ + ResolvedEid::new(&a, uid("")), + ResolvedEid::new(&b, uid("kept")), + ], + now(), + ); + let empty = entry_sets(&[], now()); + + assert!( + !mixed.all_discarded(), + "should not report a result that kept one set" + ); + assert!( + !empty.all_discarded(), + "should not report a result that named nothing" + ); + } + + #[test] + fn duplicates_and_entries_and_uids_over_the_limits_are_dropped() { + let a = source("a.example.com"); + let mut eids = vec![ + ResolvedEid::new(&a, uid("e0")), + ResolvedEid::new(&a, uid("e0")), + ]; + // 7 more entries of 4 UIDs: 1 + 28 = 29 UIDs, then a ninth entry. + for entry in 1..=7 { + let mut eid = ResolvedEid::new(&a, uid(&format!("e{entry}u0"))); + for index in 1..4 { + eid = eid.with_uid(uid(&format!("e{entry}u{index}"))); + } + eids.push(eid); + } + eids.push(ResolvedEid::new(&a, uid("ninth"))); + + let sets = entry_sets(&eids, now()); + + let set = &sets.sets[&a]; + assert_eq!( + set.entries.len(), + MAX_ENTRIES_PER_SOURCE, + "should keep 8 entries" + ); + let uid_count: usize = set.entries.iter().map(|entry| entry.uids.len()).sum(); + assert_eq!( + uid_count, 29, + "should stay under the UID cap in this control" + ); + assert_eq!( + parts(&sets), + vec![DroppedPart::DuplicateEntry, DroppedPart::ExtraEntry], + "should drop the duplicate and the ninth entry" + ); + + let mut wide = ResolvedEid::new(&a, uid("u0")); + for index in 1..=MAX_UIDS_PER_SOURCE { + wide = wide.with_uid(uid(&format!("u{index}"))); + } + let sets = entry_sets(&[wide, ResolvedEid::new(&a, uid("later"))], now()); + assert_eq!( + sets.sets[&a].entries.len(), + 1, + "should discard the entry left with no room" + ); + assert_eq!( + sets.sets[&a].entries[0].uids.len(), + MAX_UIDS_PER_SOURCE, + "should keep 32 UIDs" + ); + assert_eq!( + parts(&sets), + vec![ + DroppedPart::ExtraUid, + DroppedPart::ExtraUid, + DroppedPart::ExtraEntry + ], + "should drop the 33rd UID and the entry with no room" + ); + } + + #[test] + fn an_oversized_set_is_discarded_whole_and_others_are_kept() { + let a = source("a.example.com"); + let b = source("b.example.com"); + + let sets = entry_sets( + &[oversized_entry(&a), ResolvedEid::new(&b, uid("small"))], + now(), + ); + + assert!( + !sets.sets.contains_key(&a), + "should never trim an oversized set" + ); + assert_eq!( + sets.oversized, + vec![a.clone()], + "should report the oversized source" + ); + assert!(sets.sets.contains_key(&b), "should keep the other source"); + assert!( + !sets.all_discarded(), + "should not report the result as all discarded" + ); + + let only = entry_sets(&[oversized_entry(&a)], now()); + assert!( + only.all_discarded(), + "should report a result whose only set is oversized" + ); + } + + fn oversized_entry(source: &SourceDomain) -> ResolvedEid { + let mut big = ResolvedEid::new(source, uid(&"x".repeat(500))); + for index in 0..8 { + big = big.with_uid(uid(&format!("{index}{}", "x".repeat(499)))); + } + big + } + + #[test] + fn a_set_takes_the_earliest_times_its_entries_set() { + let a = source("a.example.com"); + let first = ResolvedEid::new(&a, uid("one")) + .with_expires_in(Duration::from_secs(100)) + .with_refresh_at(UnixTime::from_secs(1_080)) + .with_refresh_until(UnixTime::from_secs(5_000)); + let second = ResolvedEid::new(&a, uid("two")) + .with_expires_at(UnixTime::from_secs(1_050)) + .with_refresh_in(Duration::from_secs(90)) + .with_refresh_until(UnixTime::from_secs(4_000)); + let third = ResolvedEid::new(&a, uid("three")); + + let set = entry_sets(&[first, second, third], now()) + .sets + .remove(&a) + .expect("should keep a"); + + assert_eq!( + set.expires_at, + Some(UnixTime::from_secs(1_050)), + "should take the earliest expiry" + ); + assert_eq!( + set.refresh_at, + Some(UnixTime::from_secs(1_080)), + "should take the earliest refresh" + ); + assert_eq!( + set.refresh_until, + Some(UnixTime::from_secs(4_000)), + "should take the earliest refresh limit" + ); + } + + #[test] + fn dropped_entries_do_not_count_toward_timing() { + let a = source("a.example.com"); + let kept = ResolvedEid::new(&a, uid("one")).with_expires_at(UnixTime::from_secs(2_000)); + let duplicate = + ResolvedEid::new(&a, uid("one")).with_expires_at(UnixTime::from_secs(1_100)); + let empty = ResolvedEid::new(&a, uid("")).with_refresh_at(UnixTime::from_secs(1_010)); + + let set = entry_sets(&[kept, duplicate, empty], now()) + .sets + .remove(&a) + .expect("should keep a"); + + assert_eq!( + set.expires_at, + Some(UnixTime::from_secs(2_000)), + "should ignore the expiry of a dropped duplicate" + ); + assert_eq!( + set.refresh_at, None, + "should ignore the refresh time of a discarded entry" + ); + } + + #[test] + fn invalidations_chain_and_state_changes_are_reported() { + let a = source("a.example.com"); + let b = source("b.example.com"); + let resolution = Resolution::found([ResolvedEid::new(&a, uid("u"))]) + .also_invalidate(&b) + .with_retry_in(Duration::from_secs(60)) + .replacing_state(&json!({ "signature": "s" })); + assert_eq!(resolution.invalidated(), &[b], "should invalidate b"); + assert_eq!( + resolution.retry_in(), + Some(Duration::from_secs(60)), + "should keep the retry delay" + ); + assert_eq!( + resolution.state_change(), + StateChange::Replace(r#"{"signature":"s"}"#), + "should replace the state" + ); + assert_eq!( + Resolution::no_match().clearing_state().state_change(), + StateChange::Clear, + "should clear the state" + ); + assert_eq!( + Resolution::no_match().state_change(), + StateChange::Keep, + "should keep the state by default" + ); + } + + #[test] + fn resolution_debug_redacts_state() { + let resolution = + Resolution::no_match().replacing_state(&json!({ "signature": "secret-state" })); + + let debug = format!("{resolution:?}"); + + assert!( + !debug.contains("secret-state"), + "should not print provider state" + ); + assert!( + debug.contains("Replace([redacted])"), + "should show that state is replaced" + ); + } + + #[test] + fn oversized_state_is_marked_invalid_not_dropped_silently() { + let resolution = Resolution::found([ResolvedEid::new(&source("a.example.com"), uid("u"))]) + .replacing_state(&"x".repeat(2000)); + assert!( + resolution.state_rejected(), + "should flag the state as rejected" + ); + } + + #[test] + fn oversized_ext_is_dropped_and_the_uid_and_entry_kept() { + let pad = json!({ "pad": "x".repeat(MAX_PARTNER_EXT_BYTES) }); + let resolved = uid("u").with_ext(pad.clone()); + assert_eq!(resolved.id(), "u", "should keep the UID"); + assert!(resolved.ext().is_none(), "should drop the UID-level ext"); + let small = uid("u").with_ext(json!({ "linkType": 2 })); + assert_eq!( + small.ext(), + Some(&json!({ "linkType": 2 })), + "should keep a small ext" + ); + + let eid = ResolvedEid::new(&source("a.example.com"), uid("u")).with_ext(pad); + assert_eq!(eid.uids().len(), 1, "should keep the entry"); + assert!(eid.ext().is_none(), "should drop the EID-level ext"); + } + + #[test] + fn debug_redacts_every_uid() { + let eid = ResolvedEid::new(&source("a.example.com"), uid("secret-one")) + .with_uid(uid("secret-two")); + let debug = format!("{eid:?} {:?}", eid.uids()[0]); + assert!(!debug.contains("secret"), "should not print a UID"); + assert!(debug.contains("a.example.com"), "should print the source"); + } + + #[test] + fn opted_out_is_reported() { + assert!( + Resolution::opted_out().is_opted_out(), + "should report the opt-out" + ); + assert!( + !Resolution::no_match().is_opted_out(), + "should not report an opt-out for no_match" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/revision.rs b/crates/trusted-server-core/src/ec/identity/revision.rs new file mode 100644 index 0000000000..f5f37b618e --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/revision.rs @@ -0,0 +1,277 @@ +//! Record revisions, state revisions, UID hashes and the visitor key. + +use hmac::{Hmac, Mac as _}; +use sha2::{Digest as _, Sha256}; + +use super::input::RandomSource; +use crate::ec::kv_types::{KvEid, KvPartnerId, RECORD_REVISION_HEX_LEN}; +use crate::settings::Settings; + +// Temporary: first used by a later identity task. +#[allow(dead_code)] +type HmacSha256 = Hmac; + +/// The visitor key, derived from `ec.passphrase` once per settings load. +/// +/// It is the only keyed value: nothing stored in the row is keyed +/// (spec 9.3). +#[derive(Clone)] +pub(crate) struct IdentityKeys { + visitor: [u8; 32], +} + +impl core::fmt::Debug for IdentityKeys { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter.write_str("IdentityKeys([redacted])") + } +} + +impl IdentityKeys { + /// Derives the visitor key from `ec.passphrase`. + #[must_use] + pub(crate) fn from_settings(settings: &Settings) -> Self { + let passphrase = settings.ec.passphrase.expose().as_bytes(); + let mut mac = + HmacSha256::new_from_slice(passphrase).expect("should accept any HMAC key length"); + mac.update(b"trusted-server/identity/visitor/v1"); + Self { + visitor: mac.finalize().into_bytes().into(), + } + } +} + +fn hex16(digest: &[u8]) -> String { + hex::encode(&digest[..RECORD_REVISION_HEX_LEN / 2]) +} + +/// A record's stored revision: 16 lowercase hex digits. +#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct RecordRevision(String); + +impl RecordRevision { + /// Wraps a stored `rv`. `KvEntry::validate` has checked the format. + #[must_use] + pub(crate) fn from_stored(value: &str) -> Self { + Self(value.to_owned()) + } + + /// The hex digits. + #[must_use] + pub(crate) fn as_str(&self) -> &str { + &self.0 + } +} + +/// A record's revision (spec 5.4). `Unset` for a record without `rv`, which +/// only a schema version 1 row holds, until its next write gives it one. +#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) enum Revision { + /// The record has no `rv`. + Unset, + /// The record's `rv`. + Set(RecordRevision), +} + +/// A fresh random revision, drawn on every write of a record. +#[must_use] +pub(crate) fn new_revision(random: &dyn RandomSource) -> RecordRevision { + let mut bytes = [0u8; RECORD_REVISION_HEX_LEN / 2]; + random.fill(&mut bytes); + RecordRevision(hex::encode(bytes)) +} + +/// The revision of a stored record. One revision covers the source's whole +/// entry set. +#[must_use] +pub(crate) fn record_revision(record: &KvPartnerId) -> Revision { + record.rv.as_deref().map_or(Revision::Unset, |rv| { + Revision::Set(RecordRevision::from_stored(rv)) + }) +} + +/// The hash of a provider's `ps.st` a call read, so a later apply can tell +/// whether another writer changed it since. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct StateRevision(String); + +impl StateRevision { + /// The first 16 hex digits of the SHA-256 of `st` as compact JSON, or of + /// the empty string when there is none. Compact JSON is never empty, so + /// absent and present state never share a revision. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn of(st: Option<&serde_json::Value>) -> Self { + let bytes = st + .and_then(|value| serde_json::to_vec(value).ok()) + .unwrap_or_default(); + Self(hex16(&Sha256::digest(bytes))) + } +} + +/// The hash stored in `ps.iv` or `ps.ov` for a removed UID: the first 16 hex +/// digits of its unkeyed SHA-256 (spec 6.1). +#[must_use] +pub(crate) fn uid_hash(uid: &str) -> String { + hex16(&Sha256::digest(uid.as_bytes())) +} + +/// One hash per UID of a removed set, in set order, without duplicates. +#[must_use] +// Temporary: first used by a later identity task. +#[allow(dead_code)] +pub(crate) fn set_hashes(set: &[KvEid]) -> Vec { + let mut hashes = Vec::new(); + for uid in set.iter().flat_map(|entry| &entry.uids) { + let hash = uid_hash(&uid.id); + if !hashes.contains(&hash) { + hashes.push(hash); + } + } + hashes +} + +/// Appends hashes to a `ps.iv` or `ps.ov` list, oldest first. A hash +/// already present moves to the end. The caller truncates. +// Temporary: first used by a later identity task. +#[allow(dead_code)] +pub(crate) fn push_hashes(list: &mut Vec, hashes: Vec) { + for hash in hashes { + list.retain(|existing| *existing != hash); + list.push(hash); + } +} + +/// The per-visitor rate-limit key component, from the full EC ID. +#[must_use] +pub(crate) fn visitor_key(keys: &IdentityKeys, ec_id: &str) -> String { + let mut mac = HmacSha256::new_from_slice(&keys.visitor).expect("should accept a 32-byte key"); + mac.update(ec_id.as_bytes()); + hex16(&mac.finalize().into_bytes()) +} + +#[cfg(test)] +mod tests { + use serde_json::json; + use sha2::{Digest as _, Sha256}; + + use super::*; + use crate::ec::identity::input::SeededRandom; + use crate::ec::kv_types::{KvEid, KvPartnerId, KvUid}; + use crate::test_support::tests::create_test_settings; + + fn uid(id: &str) -> KvUid { + KvUid { + id: id.to_owned(), + ..KvUid::default() + } + } + + #[test] + fn new_revisions_are_hex_and_differ() { + let random = SeededRandom::new(7); + let first = new_revision(&random); + let second = new_revision(&random); + assert_eq!(first.as_str().len(), 16, "should be 16 hex digits"); + assert!( + first + .as_str() + .bytes() + .all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase()), + "should be lowercase hex" + ); + assert_ne!( + first, second, + "should give two writes of the same UID different revisions" + ); + } + + #[test] + fn record_revision_is_rv_or_unset() { + let with_rv = KvPartnerId { + rv: Some("0123456789abcdef".to_owned()), + ..KvPartnerId::new("u") + }; + assert_eq!( + record_revision(&with_rv), + Revision::Set(RecordRevision::from_stored("0123456789abcdef")), + "should use rv" + ); + assert_eq!( + record_revision(&KvPartnerId::browser("u", Some(10))), + Revision::Unset, + "should be Unset without rv, whatever the UID and at" + ); + } + + #[test] + fn uid_hashes_are_unkeyed_sha256_prefixes() { + let expected = hex::encode(&Sha256::digest(b"example-uid")[..8]); + assert_eq!( + uid_hash("example-uid"), + expected, + "should be the first 16 hex digits of SHA-256" + ); + + let mut other_settings = create_test_settings(); + other_settings.ec.passphrase = + crate::redacted::Redacted::new("another-example-passphrase-value".to_owned()); + assert_ne!( + visitor_key(&IdentityKeys::from_settings(&other_settings), "ec"), + visitor_key(&IdentityKeys::from_settings(&create_test_settings()), "ec"), + "should key only the visitor key by the passphrase" + ); + } + + #[test] + fn a_removal_hashes_every_uid_of_the_set_and_push_moves_repeats_to_the_end() { + let set = vec![ + KvEid { + uids: vec![uid("a"), uid("b")], + ..KvEid::default() + }, + KvEid { + uids: vec![uid("c"), uid("a")], + ..KvEid::default() + }, + ]; + + let hashes = set_hashes(&set); + + let expected: Vec = ["a", "b", "c"].iter().map(|id| uid_hash(id)).collect(); + assert_eq!(hashes, expected, "should hash each UID once, in set order"); + assert_eq!( + set_hashes(&KvPartnerId::new("u").eids), + vec![uid_hash("u")], + "should cost one hash for a one-UID record" + ); + + let mut list = vec![uid_hash("a"), uid_hash("z")]; + push_hashes(&mut list, vec![uid_hash("a"), uid_hash("y")]); + assert_eq!( + list, + vec![uid_hash("z"), uid_hash("a"), uid_hash("y")], + "should move a repeat to the end" + ); + } + + #[test] + fn state_revisions_tell_absent_and_changed_state_apart() { + let state = json!({ "token": "A0" }); + assert_eq!( + StateRevision::of(Some(&state)), + StateRevision::of(Some(&state.clone())), + "should be stable" + ); + assert_ne!( + StateRevision::of(Some(&state)), + StateRevision::of(Some(&json!({ "token": "A1" }))), + "should change with the state" + ); + assert_ne!( + StateRevision::of(None), + StateRevision::of(Some(&json!(""))), + "should tell absent from empty" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/runtime.rs b/crates/trusted-server-core/src/ec/identity/runtime.rs new file mode 100644 index 0000000000..8d673ea874 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/runtime.rs @@ -0,0 +1,261 @@ +//! The identity state built once per settings load (spec 10.3). + +use std::collections::BTreeMap; + +use error_stack::Report; + +use super::catalog::ProviderCatalog; +use super::coordinator::IdentityDeps; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::input::RandomSource; +use super::revision::IdentityKeys; +use crate::ec::rate_limiter::RateLimiter; +use crate::error::TrustedServerError; +use crate::integrations::IntegrationRegistry; +use crate::platform::RuntimeServices; +use crate::settings::{IdentitySettings, ProviderMode, Settings}; + +/// The provider catalog, the keys and the identity policy of one settings load. +/// +/// The adapter builds it once and shares it through an `Arc`. It is the one +/// identity type the adapter names, so the crate-private catalog and keys +/// never cross the crate boundary. +pub struct IdentityRuntime { + catalog: ProviderCatalog, + // Temporary: first read by a later identity task. + #[allow(dead_code)] + keys: IdentityKeys, + // Temporary: first read by a later identity task. + #[allow(dead_code)] + identity: IdentitySettings, +} + +impl core::fmt::Debug for IdentityRuntime { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("IdentityRuntime") + .field("providers", &self.catalog.providers().count()) + .finish_non_exhaustive() + } +} + +impl IdentityRuntime { + /// Builds the runtime from settings and the providers `integrations` + /// registered, without resolving any secret. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::Configuration`] when a provider cannot be + /// described or the catalog rules fail. + pub fn from_settings( + settings: &Settings, + integrations: &IntegrationRegistry, + ) -> Result> { + let catalog = ProviderCatalog::from_settings(settings, &integrations.identity_providers())? + .with_narrowing(narrowing(&settings.ec.identity)); + Ok(Self { + catalog, + keys: IdentityKeys::from_settings(settings), + identity: settings.ec.identity.clone(), + }) + } + + /// Builds the runtime with `providers` and source `owners` in place of + /// the generic pull providers `settings` describes, for tests that drive + /// a request through a provider no configuration can name. + /// + /// # Errors + /// + /// Never fails today; it returns a `Result` to match + /// [`from_settings`](Self::from_settings). + #[cfg(any(test, feature = "test-utils"))] + pub fn from_settings_with_providers( + settings: &Settings, + providers: Vec>, + owners: BTreeMap, + ) -> Result> { + let catalog = ProviderCatalog::new(providers, owners) + .with_narrowing(narrowing(&settings.ec.identity)); + Ok(Self { + catalog, + keys: IdentityKeys::from_settings(settings), + identity: settings.ec.identity.clone(), + }) + } + + /// The enabled providers. + /// A copy of this runtime with the `ts-eids-s` mirror turned on, for + /// tests that exercise the mirror. + #[cfg(test)] + #[must_use] + pub(crate) fn with_mirror_enabled(&self) -> Self { + let mut identity = self.identity.clone(); + identity.mirror_enabled = true; + Self { + catalog: self.catalog.clone(), + keys: self.keys.clone(), + identity, + } + } + + #[must_use] + pub(crate) fn catalog(&self) -> &ProviderCatalog { + &self.catalog + } + + /// Whether a new visitor's first navigation can run a hold-mode provider: + /// one is configured that does not read the EC ID. A publisher navigation + /// then mints the EC with [`EcContext::mint_deferred`] (spec 4.7). + /// + /// [`EcContext::mint_deferred`]: crate::ec::EcContext::mint_deferred + #[must_use] + pub fn holds_on_first_navigation(&self) -> bool { + self.catalog.providers().any(|provider| { + let descriptor = provider.descriptor(); + self.identity.provider(descriptor.id()).mode == ProviderMode::Hold + && !descriptor + .resolves() + .is_some_and(super::descriptor::Resolves::gets_ec_id) + }) + } + + /// The keys derived from `ec.passphrase`. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn keys(&self) -> &IdentityKeys { + &self.keys + } + + /// The `[ec.identity]` policy. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn identity(&self) -> &IdentitySettings { + &self.identity + } + + /// What the coordinator needs besides the request. + #[must_use] + // Temporary: first used by a later identity task. + #[allow(dead_code)] + pub(crate) fn deps<'a>( + &'a self, + settings: &'a Settings, + services: &'a RuntimeServices, + random: &'a dyn RandomSource, + rate_limiter: Option<&'a dyn RateLimiter>, + ) -> IdentityDeps<'a> { + IdentityDeps { + settings, + catalog: &self.catalog, + keys: &self.keys, + random, + services, + rate_limiter, + } + } +} + +/// Each provider table's `sources`, for the providers that list any. +/// +/// Config validation has rejected a malformed source, so one that does not +/// parse is skipped. +fn narrowing(identity: &IdentitySettings) -> BTreeMap> { + identity + .providers + .iter() + .filter_map(|(id, table)| { + let listed = table.sources.as_ref()?; + Some(( + id.parse().ok()?, + listed + .iter() + .filter_map(|source| source.parse().ok()) + .collect(), + )) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::test_support::{ec_partner, pid, source}; + use crate::settings::ProviderSettings; + use crate::test_support::tests::create_test_settings; + + fn settings_with_listed_sources(listed: Option>) -> Settings { + let mut settings = create_test_settings(); + settings.ec.partners = vec![ec_partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + "example-token-value-0123456789abcdef", + )]; + settings.ec.identity.providers.insert( + "pull-ssp-example-com".to_owned(), + ProviderSettings { + sources: listed, + ..ProviderSettings::default() + }, + ); + settings + } + + #[test] + fn runtime_debug_prints_no_key_material() { + let settings = settings_with_listed_sources(None); + let runtime = IdentityRuntime::from_settings(&settings, &IntegrationRegistry::default()) + .expect("should build"); + + let printed = format!("{runtime:?}"); + + assert!( + printed.contains("providers: 1"), + "should print the provider count: {printed}" + ); + assert!( + !printed.contains(settings.ec.passphrase.expose()), + "should not print the passphrase: {printed}" + ); + assert!( + !printed.contains("keys"), + "should not print the keys: {printed}" + ); + } + + #[test] + fn runtime_narrows_providers_by_their_sources_table() { + let id = pid("pull-ssp-example-com"); + + let unlisted = IdentityRuntime::from_settings( + &settings_with_listed_sources(None), + &IntegrationRegistry::default(), + ) + .expect("should build"); + let kept = IdentityRuntime::from_settings( + &settings_with_listed_sources(Some(vec!["ssp.example.com".to_owned()])), + &IntegrationRegistry::default(), + ) + .expect("should build"); + assert_eq!( + unlisted.catalog().sources_of(&id), + [source("ssp.example.com")], + "control: should claim the descriptor's source with no table" + ); + assert_eq!( + kept.catalog().sources_of(&id), + [source("ssp.example.com")], + "should keep a listed source" + ); + let error = IdentityRuntime::from_settings( + &settings_with_listed_sources(Some(vec!["other.example.com".to_owned()])), + &IntegrationRegistry::default(), + ) + .expect_err("should reject a source the descriptor does not declare"); + assert!( + format!("{error}").contains("other.example.com"), + "should name the undeclared source: {error}" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/servable.rs b/crates/trusted-server-core/src/ec/identity/servable.rs new file mode 100644 index 0000000000..6fe18264f0 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/servable.rs @@ -0,0 +1,392 @@ +//! The one servability check shared by the bidstream, the admin view and +//! `/_ts/api/v1/identify` (spec 7.1). + +use std::collections::BTreeSet; + +use super::catalog::ProviderCatalog; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::policy::{PolicySignals, policy_allows}; +use crate::consent::ConsentContext; +use crate::ec::kv_types::KvPartnerId; + +/// What a servability decision needs from the request. +pub(crate) struct ServeContext<'a> { + pub(crate) now: UnixTime, + pub(crate) consent: &'a ConsentContext, + pub(crate) signals: PolicySignals, + pub(crate) catalog: &'a ProviderCatalog, + /// The providers with a `ps.o` in the row the request read. + pub(crate) opted_out: &'a BTreeSet, +} + +/// Whether a stored record, the source's whole entry set, may be sent from +/// the server. +#[must_use] +pub(crate) fn is_servable( + context: &ServeContext<'_>, + source: &SourceDomain, + record: &KvPartnerId, +) -> bool { + if record.eids.is_empty() { + return false; + } + if record.exp.is_some_and(|exp| context.now.as_secs() >= exp) { + return false; + } + let writer = record + .w + .as_deref() + .and_then(|writer| writer.parse::().ok()); + if record.w.is_some() { + let Some(writer) = writer.as_ref() else { + return false; + }; + let Some(provider) = context.catalog.get(writer) else { + return false; + }; + if context.opted_out.contains(writer) + || !policy_allows( + provider.descriptor().consent_policy(), + context.consent, + context.signals, + ) + { + return false; + } + } + if let Some(owner) = context.catalog.owner_of(source) { + // An opted-out owner stops its source whoever wrote the record, so a + // browser-seeded record its opt-out did not remove is not served. + // No stored record matches the owner's `ps.iv`: every invalidation + // removes its record, and nothing refills one (spec 5.4). + if context.opted_out.contains(owner) { + return false; + } + let Some(owner_provider) = context.catalog.get(owner) else { + return false; + }; + let descriptor = owner_provider.descriptor(); + if !policy_allows( + descriptor.consent_policy(), + context.consent, + context.signals, + ) { + return false; + } + if descriptor.is_expiring(source) + && (writer.as_ref() != Some(owner) || record.exp.is_none()) + { + return false; + } + } + true +} + +#[cfg(test)] +mod tests { + use std::collections::{BTreeMap, BTreeSet}; + use std::sync::Arc; + + use super::*; + use crate::ec::identity::descriptor::{Captures, ConsentPolicy, ProviderDescriptor, Resolves}; + use crate::ec::identity::provider::IdentityProvider; + use crate::ec::identity::test_support::resolver_policy; + use crate::ec::identity::test_support::{ + example_entry_set, full_record, test_consent_allowing_everything, + test_consent_gdpr_without_vendor, + }; + use crate::ec::kv_types::KvPartnerId; + + struct Fixed(ProviderDescriptor); + impl IdentityProvider for Fixed { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } + } + + fn provider( + id: &str, + source: &str, + policy: ConsentPolicy, + expiring: bool, + ) -> Arc { + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_source(source.parse().expect("should parse the source")) + .with_resolves(Resolves::new().with_endpoint("https://api.example.com")) + .with_consent_policy(policy); + if expiring { + builder = builder.with_expiring_ids(); + } + Arc::new(Fixed(builder.build().expect("should build"))) + } + + fn record(writer: Option<&str>, exp: Option) -> KvPartnerId { + KvPartnerId { + w: writer.map(str::to_owned), + exp, + ..KvPartnerId::new("u") + } + } + + fn check( + catalog: &ProviderCatalog, + opted_out: &BTreeSet, + source: &str, + record: &KvPartnerId, + ) -> bool { + let consent = test_consent_allowing_everything(); + let context = ServeContext { + now: UnixTime::from_secs(100), + consent: &consent, + signals: PolicySignals::default(), + catalog, + opted_out, + }; + is_servable( + &context, + &source.parse().expect("should parse the source"), + record, + ) + } + + #[test] + fn expiry_writer_presence_and_opt_out() { + let catalog = ProviderCatalog::new( + vec![provider( + "p", + "a.example.com", + ConsentPolicy::eid_persistence_only(), + false, + )], + BTreeMap::new(), + ); + let none = BTreeSet::new(); + assert!( + check( + &catalog, + &none, + "a.example.com", + &record(Some("p"), Some(101)) + ), + "should serve a valid record" + ); + assert!( + !check( + &catalog, + &none, + "a.example.com", + &record(Some("p"), Some(100)) + ), + "should treat now == exp as expired" + ); + assert!( + !check( + &catalog, + &none, + "a.example.com", + &record(Some("gone"), None) + ), + "should not serve an unknown writer" + ); + let opted: BTreeSet<_> = ["p".parse().expect("should parse the provider ID")] + .into_iter() + .collect(); + assert!( + !check(&catalog, &opted, "a.example.com", &record(Some("p"), None)), + "should not serve an opted-out writer" + ); + assert!( + check(&catalog, &none, "b.example.com", &record(None, None)), + "should serve an unowned browser record" + ); + } + + #[test] + fn an_opted_out_owner_stops_every_record_of_its_source() { + let catalog = ProviderCatalog::new( + vec![provider( + "owner", + "a.example.com", + ConsentPolicy::eid_persistence_only(), + false, + )], + [( + "a.example.com".parse().expect("should parse the source"), + "owner".parse().expect("should parse the provider ID"), + )] + .into_iter() + .collect(), + ); + let opted: BTreeSet<_> = ["owner".parse().expect("should parse the provider ID")] + .into_iter() + .collect(); + + assert!( + !check(&catalog, &opted, "a.example.com", &record(None, None)), + "should not serve a browser-seeded record of the owned source" + ); + assert!( + check( + &catalog, + &BTreeSet::new(), + "a.example.com", + &record(None, None) + ), + "control: should serve it while the owner is not opted out" + ); + } + + #[test] + fn a_full_record_is_servable_or_not_as_a_whole() { + let catalog = ProviderCatalog::new( + vec![provider( + "p", + "a.example.com", + ConsentPolicy::eid_persistence_only(), + false, + )], + BTreeMap::new(), + ); + let none = BTreeSet::new(); + let valid = KvPartnerId { + exp: Some(101), + ..full_record(Some("p"), example_entry_set()) + }; + let expired = KvPartnerId { + exp: Some(100), + ..full_record(Some("p"), example_entry_set()) + }; + assert!( + check(&catalog, &none, "a.example.com", &valid), + "should serve the whole valid set" + ); + assert!( + !check(&catalog, &none, "a.example.com", &expired), + "should never serve an expired set" + ); + } + + #[test] + fn rollback_rule_for_expiring_owned_sources() { + let owner: IdentityProviderId = "owner".parse().expect("should parse the provider ID"); + let catalog = ProviderCatalog::new( + vec![provider( + "owner", + "a.example.com", + ConsentPolicy::eid_persistence_only(), + true, + )], + [( + "a.example.com".parse().expect("should parse the source"), + owner, + )] + .into_iter() + .collect(), + ); + let none = BTreeSet::new(); + assert!( + check( + &catalog, + &none, + "a.example.com", + &record(Some("owner"), Some(200)) + ), + "should serve the owner's value" + ); + assert!( + !check(&catalog, &none, "a.example.com", &record(None, None)), + "should never serve a browser-seeded value from the server" + ); + assert!( + !check( + &catalog, + &none, + "a.example.com", + &record(Some("owner"), None) + ), + "should not serve a value with a stripped exp" + ); + } + + #[test] + fn narrowing_limits_claims_and_capture_only_sources() { + let capture_only: Arc = Arc::new(Fixed( + ProviderDescriptor::builder("c".parse().expect("should parse the provider ID")) + .with_source("a.example.com".parse().expect("should parse the source")) + .with_source("b.example.com".parse().expect("should parse the source")) + .with_captures(Captures::new().with_path_prefix("/integrations/c/")) + .build() + .expect("should build"), + )); + let resolver = provider( + "p", + "a.example.com", + ConsentPolicy::eid_persistence_only(), + false, + ); + let a: SourceDomain = "a.example.com".parse().expect("should parse the source"); + let b: SourceDomain = "b.example.com".parse().expect("should parse the source"); + let c: IdentityProviderId = "c".parse().expect("should parse the provider ID"); + let p: IdentityProviderId = "p".parse().expect("should parse the provider ID"); + + let catalog = ProviderCatalog::new(vec![capture_only, resolver], BTreeMap::new()) + .with_narrowing([(c.clone(), vec![b.clone()])].into_iter().collect()); + + assert_eq!( + catalog.sources_of(&c), + std::slice::from_ref(&b), + "should narrow the capture-only provider" + ); + assert!( + catalog.is_capture_only(&c) && !catalog.is_capture_only(&p), + "should tell capture-only apart" + ); + assert_eq!( + catalog.capture_only_sources(&c), + std::slice::from_ref(&b), + "should list its narrowed sources" + ); + assert!( + catalog.capture_only_sources(&p).is_empty(), + "should list none for a resolver" + ); + assert_eq!( + catalog.claimants(&a).collect::>(), + vec![&p], + "should drop a narrowed-away claim" + ); + assert_eq!( + catalog.claimants(&b).collect::>(), + vec![&c], + "should keep the narrowed claim" + ); + } + + #[test] + fn writer_policy_applies_to_unowned_sources() { + let strict = resolver_policy().with_tcf_vendor(9999); + let catalog = ProviderCatalog::new( + vec![provider("p", "a.example.com", strict, false)], + BTreeMap::new(), + ); + let consent = test_consent_gdpr_without_vendor(9999); + let none = BTreeSet::new(); + let context = ServeContext { + now: UnixTime::from_secs(100), + consent: &consent, + signals: PolicySignals::default(), + catalog: &catalog, + opted_out: &none, + }; + let source = "a.example.com".parse().expect("should parse the source"); + assert!( + !is_servable(&context, &source, &record(Some("p"), None)), + "should not serve when the writer's vendor is denied" + ); + assert!( + is_servable(&context, &source, &record(None, None)), + "should leave a browser value unaffected" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs new file mode 100644 index 0000000000..4ada943441 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -0,0 +1,3032 @@ +//! Shared helpers for identity tests. + +#![cfg_attr( + not(test), + allow( + dead_code, + reason = "the test-utils library build compiles the crate-private helpers without their unit-test callers" + ) +)] + +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::net::IpAddr; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use error_stack::Report; +use http::{HeaderMap, HeaderName, HeaderValue, Method, StatusCode}; +use serde::Serialize; +use serde::de::DeserializeOwned; +use serde_json::json; +use url::Url; + +use crate::consent::jurisdiction::Jurisdiction; +use crate::consent::types::RawConsentSignals; +use crate::consent::types::TcfConsent; +use crate::consent::{ConsentContext, build_context_from_signals}; +use crate::ec::identity::catalog::ProviderCatalog; +use crate::ec::identity::coordinator::{OutcomeContext, RequestFacts}; +use crate::ec::identity::descriptor::{ + Access, Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, ResolveKind, + Resolves, +}; +use crate::ec::identity::exchange::{ProviderRequest, ProviderResponse}; +use crate::ec::identity::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use crate::ec::identity::input::{ + CaptureInput, HashedEmail, ProviderConsent, Region, ResolveInput, ResolveSignals, SeededRandom, + StoredId, +}; +use crate::ec::identity::policy::{PolicySignals, policy_allows}; +use crate::ec::identity::provider::{IdentityProvider, IdentityProviderConfig}; +use crate::ec::identity::resolution::{ + IdentityProviderError, Resolution, ResolvedEid, ResolvedUid, StateChange, When, entry_sets, +}; +use crate::ec::identity::revision::{StateRevision, new_revision}; +use crate::ec::kv_backend::in_memory::InMemoryEcKv; +use crate::ec::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome}; +use crate::ec::kv_types::{KvEid, KvPartnerId, KvUid}; +use crate::ec::rate_limiter::RateLimiter; +use crate::error::TrustedServerError; +use crate::openrtb::{Eid, Uid}; +use crate::redacted::Redacted; +use crate::settings::{EcPartner, IdentitySettings, ProviderSettings}; + +/// A request with no consent signal and no regulated jurisdiction. +/// Purposes 1 and 4 with no vendor: the policy test resolvers run under. +#[must_use] +pub fn resolver_policy() -> ConsentPolicy { + ConsentPolicy::strict() + .with_tcf_purposes([4]) + .without_tcf_vendor() +} + +#[must_use] +pub fn test_consent_allowing_everything() -> ConsentContext { + ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + ..ConsentContext::default() + } +} + +/// A GDPR request whose TCF string grants every purpose but not `vendor_id`. +#[must_use] +pub fn test_consent_gdpr_without_vendor(vendor_id: u16) -> ConsentContext { + let vendors: Vec = (1..=20).filter(|vendor| *vendor != vendor_id).collect(); + ConsentContext { + gdpr_applies: true, + jurisdiction: Jurisdiction::Gdpr, + tcf: Some(TcfConsent { + version: 2, + cmp_id: 1, + cmp_version: 1, + consent_screen: 1, + consent_language: "EN".to_owned(), + vendor_list_version: 1, + tcf_policy_version: 4, + created_ds: 0, + last_updated_ds: 0, + purpose_consents: vec![true; 24], + purpose_legitimate_interests: vec![false; 24], + vendor_consents: vendors, + vendor_legitimate_interests: Vec::new(), + special_feature_opt_ins: vec![false; 12], + }), + ..ConsentContext::default() + } +} + +/// Spec 6.1's example set: two entries, three UIDs, with UID-level and +/// EID-level fields. +#[must_use] +pub fn example_entry_set() -> Vec { + vec![ + KvEid { + uids: vec![ + KvUid { + id: "example-a".to_owned(), + atype: Some(1), + ext: None, + }, + KvUid { + id: "example-b".to_owned(), + atype: Some(3), + ext: Some(json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("example.com".to_owned()), + mm: Some(3), + ..KvEid::default() + }, + KvEid { + uids: vec![KvUid { + id: "example-c".to_owned(), + atype: Some(1), + ext: None, + }], + matcher: Some("partner.example.com".to_owned()), + ..KvEid::default() + }, + ] +} + +/// A record holding `set`, written by `writer`. +#[must_use] +pub fn full_record(writer: Option<&str>, set: Vec) -> KvPartnerId { + KvPartnerId { + w: writer.map(str::to_owned), + eids: set, + ..KvPartnerId::default() + } +} + +/// A descriptor-only resolver for `id`, declaring `sources`, with a dummy +/// endpoint and the `eid_persistence_only` policy. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn fixed_provider(id: &str, sources: &[&str]) -> Arc { + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_resolves(Resolves::new().with_endpoint("https://api.example.com")) + .with_consent_policy(ConsentPolicy::eid_persistence_only()); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse the source")); + } + Arc::new(DescriptorOnly( + builder.build().expect("should build the descriptor"), + )) +} + +/// A descriptor-only capture-only provider for `id`, declaring `sources`. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn fixed_capture_provider(id: &str, sources: &[&str]) -> Arc { + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_captures(Captures::new().with_path_prefix(&format!("/integrations/{id}/"))) + .with_consent_policy(ConsentPolicy::eid_persistence_only()); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse the source")); + } + Arc::new(DescriptorOnly( + builder.build().expect("should build the descriptor"), + )) +} + +/// A capture-only provider that answers as the 2xx JSON body says: +/// `{"tokens":[{"source":"a.example.com","value":"x"}],"invalidate":["a.example.com"], +/// "opted_out":true,"state":"s"}`. Every member is optional. `opted_out` wins, +/// then `tokens`, then `invalidate`, and anything else is `no_match`. +struct AnsweringCapture { + descriptor: ProviderDescriptor, +} + +impl IdentityProvider for AnsweringCapture { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn capture( + &self, + input: &CaptureInput<'_>, + ) -> Result> { + let Ok(body) = input.response().json::() else { + return Ok(Resolution::no_match()); + }; + let sources = |name: &str| -> Vec { + body[name] + .as_array() + .into_iter() + .flatten() + .filter_map(|value| value.as_str()?.parse().ok()) + .collect() + }; + let tokens: Vec = body["tokens"] + .as_array() + .into_iter() + .flatten() + .filter_map(|token| { + let source = input.source_named(token["source"].as_str()?)?; + Some(ResolvedEid::new( + source, + ResolvedUid::new(token["value"].as_str()?), + )) + }) + .collect(); + let invalidated = sources("invalidate"); + let mut resolution = if body["opted_out"].as_bool() == Some(true) { + Resolution::opted_out() + } else if !tokens.is_empty() { + Resolution::found(tokens) + } else if let Some(first) = invalidated.first() { + Resolution::invalidate(first) + } else { + Resolution::no_match() + }; + for source in &invalidated { + resolution = resolution.also_invalidate(source); + } + if let Some(state) = body["state"].as_str() { + resolution = resolution.replacing_state(&state); + } + Ok(resolution) + } +} + +/// A capture-only provider `id` over `sources` that answers as the response +/// body says, with a capture rule on `/integrations//`. With `gvl_id` its +/// consent policy names that TCF vendor. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn answering_capture_provider( + id: &str, + sources: &[&str], + gvl_id: Option, +) -> Arc { + answering_capture_provider_at(id, sources, gvl_id, &format!("/integrations/{id}/")) +} + +/// As [`answering_capture_provider`], with the capture rule on `prefix`. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn answering_capture_provider_at( + id: &str, + sources: &[&str], + gvl_id: Option, + prefix: &str, +) -> Arc { + let policy = match gvl_id { + Some(vendor) => resolver_policy().with_tcf_vendor(vendor), + None => ConsentPolicy::eid_persistence_only(), + }; + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_captures(Captures::new().with_path_prefix(prefix)) + .with_consent_policy(policy); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse the source")); + } + Arc::new(AnsweringCapture { + descriptor: builder.build().expect("should build the descriptor"), + }) +} + +/// A [`fixed_provider`] that also declares a `with_provenance` default per +/// listed source. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn provenance_provider( + id: &str, + sources: &[&str], + provenance: Vec<(&str, EidProvenance)>, +) -> Arc { + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_resolves(Resolves::new().with_endpoint("https://api.example.com")) + .with_consent_policy(ConsentPolicy::eid_persistence_only()); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse the source")); + } + for (source, default) in provenance { + builder = + builder.with_provenance(&source.parse().expect("should parse the source"), default); + } + provider_from(builder.build().expect("should build the descriptor")) +} + +/// A descriptor-only provider for `descriptor`, for tests that need more +/// than [`fixed_provider`] declares. +#[must_use] +pub fn provider_from(descriptor: ProviderDescriptor) -> Arc { + Arc::new(DescriptorOnly(descriptor)) +} + +/// The source [`MultiEntryTestProvider`] writes. +pub const MULTI_ENTRY_SOURCE: &str = "exampleid.example.com"; + +/// The endpoint [`MultiEntryTestProvider`] calls. +pub const MULTI_ENTRY_ENDPOINT: &str = "https://api.exampleid.example.com/v1/resolve"; + +/// A resolver for [`MULTI_ENTRY_SOURCE`] that returns a complete set, for +/// request-level tests of a registered provider. +/// +/// The JSON body of the endpoint's reply picks the answer through its +/// `entries` member: +/// +/// - `"full"`, or a body without the member: two entries. The first holds +/// `uid-a` (`atype` 1, a UID-level `ext`) and `uid-b`, with `inserter`, +/// `matcher` and `mm`. The second holds `uid-c` and an EID-level `ext`, and +/// sets none of `inserter`, `matcher` and `mm`. +/// - `"first"`: the first entry alone. +/// - `"single"`: `uid-a` with `atype` 1 and its UID-level `ext`, and no +/// EID-level field. +/// - `"oversized"`: a set over `MAX_ENTRY_SET_BYTES`, which also invalidates +/// the source. +/// - `"none"`: no match. +/// +/// The descriptor declares a `with_provenance` default of an `inserter` and a +/// `matcher`. [`stateful`](Self::stateful) is the same provider declared +/// `with_private_state()`; it keeps `{"seen": true}` in every answer that +/// finds an ID. +pub struct MultiEntryTestProvider { + descriptor: ProviderDescriptor, + stateful: bool, +} + +impl MultiEntryTestProvider { + /// The stateless provider. + /// + /// # Panics + /// + /// Panics when the fixed descriptor does not build. + #[must_use] + pub fn new() -> Self { + Self::build(false) + } + + /// The same provider, declared `with_private_state()`. + /// + /// # Panics + /// + /// Panics when the fixed descriptor does not build. + #[must_use] + pub fn stateful() -> Self { + Self::build(true) + } + + fn build(stateful: bool) -> Self { + let mut resolves = Resolves::new().with_endpoint(MULTI_ENTRY_ENDPOINT); + if stateful { + resolves = resolves.with_private_state(); + } + let descriptor = + ProviderDescriptor::builder("exampleid".parse().expect("should parse the provider ID")) + .with_source(source(MULTI_ENTRY_SOURCE)) + .with_resolves(resolves) + .with_consent_policy(resolver_policy()) + .with_provenance( + &source(MULTI_ENTRY_SOURCE), + EidProvenance::new() + .with_inserter("default-inserter.example.com") + .with_matcher("default-matcher.example.com"), + ) + .build() + .expect("should build the descriptor"); + Self { + descriptor, + stateful, + } + } + + fn first_entry(source: &SourceDomain) -> ResolvedEid { + ResolvedEid::new( + source, + ResolvedUid::new("uid-a") + .with_atype(1) + .with_ext(json!({ "rtiPartner": "example" })), + ) + .with_uid(ResolvedUid::new("uid-b")) + .with_inserter("inserter-a.example.com") + .with_matcher("matcher-a.example.com") + .with_mm(2) + } +} + +impl Default for MultiEntryTestProvider { + fn default() -> Self { + Self::new() + } +} + +impl IdentityProvider for MultiEntryTestProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + Ok(Some(ProviderRequest::get(input.endpoint()?))) + } + + fn response( + &self, + response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let body: serde_json::Value = response.success_json()?; + let source = source(MULTI_ENTRY_SOURCE); + let resolution = match body["entries"].as_str() { + Some("none") => Resolution::no_match(), + Some("first") => Resolution::found([Self::first_entry(&source)]), + Some("single") => Resolution::found([ResolvedEid::new( + &source, + ResolvedUid::new("uid-a") + .with_atype(1) + .with_ext(json!({ "rtiPartner": "example" })), + )]), + Some("oversized") => { + let entries = (0..8).map(|index| { + ResolvedEid::new( + &source, + ResolvedUid::new(format!("uid-{index}")) + .with_ext(json!({ "pad": "x".repeat(1_000) })), + ) + }); + return Ok(self.with_state(Resolution::found(entries).also_invalidate(&source))); + } + _ => Resolution::found([ + Self::first_entry(&source), + ResolvedEid::new(&source, ResolvedUid::new("uid-c")) + .with_ext(json!({ "segments": ["example"] })), + ]), + }; + Ok(self.with_state(resolution)) + } +} + +impl MultiEntryTestProvider { + fn with_state(&self, resolution: Resolution) -> Resolution { + if self.stateful { + resolution.replacing_state(&json!({ "seen": true })) + } else { + resolution + } + } +} + +/// A `fixed_provider` declared `with_private_state()`. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn stateful_fixed_provider(id: &str, sources: &[&str]) -> Arc { + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.example.com") + .with_private_state(), + ) + .with_consent_policy(ConsentPolicy::eid_persistence_only()); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse the source")); + } + provider_from(builder.build().expect("should build the descriptor")) +} + +/// A request with no consent signal and no regulated jurisdiction. +#[must_use] +pub fn allowing_consent() -> ConsentContext { + test_consent_allowing_everything() +} + +/// A source domain. +/// +/// # Panics +/// +/// Panics when `domain` is not valid. +#[must_use] +pub fn source(domain: &str) -> SourceDomain { + domain.parse().expect("should parse the source") +} + +/// A provider ID. +/// +/// # Panics +/// +/// Panics when `id` is not valid. +#[must_use] +pub fn pid(id: &str) -> IdentityProviderId { + id.parse().expect("should parse the provider ID") +} + +/// A one-UID record with a fresh `rv`, written by `writer` at `at`. +#[must_use] +pub fn record_for_test(uid: &str, writer: Option<&str>, at: u64) -> KvPartnerId { + KvPartnerId { + w: writer.map(str::to_owned), + rv: Some(new_revision(&SeededRandom::new(at)).as_str().to_owned()), + at: Some(at), + ..KvPartnerId::new(uid) + } +} + +/// Facts for a returning visitor's writer navigation that runs no auction: +/// a real browser, persistence allowed and every input available. +#[must_use] +pub(crate) fn facts_for_writer() -> RequestFacts { + RequestFacts { + is_real_browser: true, + ec_persistence_allowed: true, + policy_signals: PolicySignals::default(), + writer: true, + new_visitor: false, + runs_auction: false, + now: UnixTime::from_secs(1_000), + available_inputs: BTreeSet::from([ + Input::ClientIp, + Input::UserAgent, + Input::PageUrl, + Input::Region, + Input::HashedEmail, + ]), + } +} + +/// An owned fixture for [`OutcomeContext`], which borrows from it. +/// +/// `new` holds a catalog with provider `p` claiming `a.example.com` and +/// `b.example.com`, no source owners and default settings. +pub(crate) struct OutcomeFixture { + catalog: ProviderCatalog, + settings: ProviderSettings, + identity: IdentitySettings, + expected_state: Option, + fingerprint: String, +} + +impl OutcomeFixture { + #[must_use] + pub(crate) fn new() -> Self { + Self { + catalog: ProviderCatalog::new( + vec![fixed_provider("p", &["a.example.com", "b.example.com"])], + std::collections::BTreeMap::new(), + ), + settings: ProviderSettings::default(), + identity: IdentitySettings::default(), + expected_state: None, + fingerprint: "0".to_owned(), + } + } + + /// Replaces the catalog. + #[must_use] + pub(crate) fn with_catalog(mut self, catalog: ProviderCatalog) -> Self { + self.catalog = catalog; + self + } + + /// Replaces the provider's settings. + #[must_use] + pub(crate) fn with_settings(mut self, settings: ProviderSettings) -> Self { + self.settings = settings; + self + } + + /// Sets the consent fingerprint an opt-out records. + #[must_use] + pub(crate) fn with_fingerprint(mut self, fingerprint: &str) -> Self { + fingerprint.clone_into(&mut self.fingerprint); + self + } + + /// Sets the state revision the call read. + #[must_use] + pub(crate) fn with_expected_state(mut self, expected_state: StateRevision) -> Self { + self.expected_state = Some(expected_state); + self + } + + /// The context for an outcome at `now`. + #[must_use] + pub(crate) fn context_at(&self, now: u64) -> OutcomeContext<'_> { + OutcomeContext { + now: UnixTime::from_secs(now), + catalog: &self.catalog, + settings: &self.settings, + identity: &self.identity, + expected_state: self.expected_state.clone(), + fingerprint: &self.fingerprint, + } + } +} + +/// A valid EC ID in `{64-hex}.{6-alnum}` format. +pub const TEST_EC_ID: &str = + "0000000000000000000000000000000000000000000000000000000000000000.ABC123"; + +/// A partner with pull sync enabled, calling `pull_sync_url` with `token`. +/// +/// The URL's host is the only allowed domain. +/// +/// # Panics +/// +/// Panics when `source` is not a domain or `pull_sync_url` is not a URL with a host. +#[must_use] +pub fn ec_partner(source: &str, pull_sync_url: &str, token: &str) -> EcPartner { + let host = Url::parse(pull_sync_url) + .expect("should parse the pull URL") + .host_str() + .expect("should have a pull URL host") + .to_owned(); + EcPartner { + name: format!("Partner {source}"), + source_domain: source.to_owned(), + openrtb_atype: EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: None, + batch_rate_limit: EcPartner::default_batch_rate_limit(), + pull_sync_enabled: true, + pull_sync_url: Some(pull_sync_url.to_owned()), + pull_sync_allowed_domains: vec![host], + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, + ts_pull_token: Some(Redacted::new(token.to_owned())), + } +} + +/// A stored one-UID record of `source`, as a provider reads it. +/// +/// # Panics +/// +/// Panics when the record is not valid. +#[must_use] +pub(crate) fn stored_for_test(source: &SourceDomain, uid: &str) -> StoredId { + StoredId::for_test(source, uid) +} + +/// Write and lookup counts of a [`WriteCountingEcKv`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct WriteCounts { + /// Writes in [`EcKvWriteMode::Add`] mode. + pub add: usize, + /// Writes in [`EcKvWriteMode::IfGenerationMatch`] mode. + pub if_generation_match: usize, + /// Writes in [`EcKvWriteMode::Overwrite`] mode. + pub overwrite: usize, + /// Point reads of identity rows. + pub lookups: usize, + /// Point reads of pending keys (spec 5.3, Pending key). + pub pending_lookups: usize, +} + +impl WriteCounts { + /// Every write, whatever its mode. + #[must_use] + pub fn total(&self) -> usize { + self.add + self.if_generation_match + self.overwrite + } +} + +/// An in-memory KV store that counts lookups and writes by mode. +/// +/// Every write is applied or refused as the in-memory store does, so a +/// write that loses its precondition is still counted and reports +/// [`EcKvWriteOutcome::PreconditionFailed`]. +pub struct WriteCountingEcKv { + inner: InMemoryEcKv, + add: AtomicUsize, + if_generation_match: AtomicUsize, + overwrite: AtomicUsize, + lookups: AtomicUsize, + pending_lookups: AtomicUsize, + refusals: Mutex>, + failing: Mutex>, + failing_all: Mutex>, + pauses: Mutex>, +} + +impl WriteCountingEcKv { + /// An empty store named `name`. + #[must_use] + pub fn new(name: impl Into) -> Self { + Self { + inner: InMemoryEcKv::new(name), + add: AtomicUsize::new(0), + if_generation_match: AtomicUsize::new(0), + overwrite: AtomicUsize::new(0), + lookups: AtomicUsize::new(0), + pending_lookups: AtomicUsize::new(0), + refusals: Mutex::new(BTreeMap::new()), + failing: Mutex::new(BTreeSet::new()), + failing_all: Mutex::new(BTreeSet::new()), + pauses: Mutex::new(Vec::new()), + } + } + + /// Fails every generation-checked write to `key` with a store error that + /// is never retried. A failed write is still counted. + /// + /// # Panics + /// + /// Panics when the failure table is poisoned. + pub fn fail_conditional_writes(&self, key: &str) { + self.failing + .lock() + .expect("should lock the failures") + .insert(key.to_owned()); + } + + /// Fails every write to `key`, whatever its mode, with a store error. + /// A failed write is still counted. + /// + /// # Panics + /// + /// Panics when the failure table is poisoned. + pub fn fail_all_writes(&self, key: &str) { + self.failing_all + .lock() + .expect("should lock the failures") + .insert(key.to_owned()); + } + + /// Refuses the next `count` generation-checked writes to `key` with + /// [`EcKvWriteOutcome::PreconditionFailed`], the retryable refusal. A + /// refused write is still counted. + /// + /// # Panics + /// + /// Panics when the refusal table is poisoned. + pub fn refuse_conditional_writes(&self, key: &str, count: usize) { + self.refusals + .lock() + .expect("should lock the refusals") + .insert(key.to_owned(), count); + } + + /// Every [`EcKvStore::pause`] so far, in order. + /// + /// # Panics + /// + /// Panics when the pause list is poisoned. + #[must_use] + pub fn pauses(&self) -> Vec { + self.pauses.lock().expect("should lock the pauses").clone() + } + + /// The counts so far, seeding included. + #[must_use] + pub fn writes(&self) -> WriteCounts { + WriteCounts { + add: self.add.load(Ordering::SeqCst), + if_generation_match: self.if_generation_match.load(Ordering::SeqCst), + overwrite: self.overwrite.load(Ordering::SeqCst), + lookups: self.lookups.load(Ordering::SeqCst), + pending_lookups: self.pending_lookups.load(Ordering::SeqCst), + } + } + + /// Zeroes the counts, to count only what follows the seeding. + pub fn reset_counts(&self) { + for count in [ + &self.add, + &self.if_generation_match, + &self.overwrite, + &self.lookups, + &self.pending_lookups, + ] { + count.store(0, Ordering::SeqCst); + } + } +} + +impl EcKvStore for WriteCountingEcKv { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup(&self, key: &str) -> Result, Report> { + let count = if key.starts_with("pending:") { + &self.pending_lookups + } else { + &self.lookups + }; + count.fetch_add(1, Ordering::SeqCst); + self.inner.lookup(key) + } + + fn key_exists(&self, key: &str) -> Result> { + self.inner.key_exists(key) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + let count = match write.mode { + EcKvWriteMode::Add => &self.add, + EcKvWriteMode::IfGenerationMatch(_) => &self.if_generation_match, + EcKvWriteMode::Overwrite => &self.overwrite, + }; + count.fetch_add(1, Ordering::SeqCst); + if self + .failing_all + .lock() + .expect("should lock the failures") + .contains(key) + { + return Err(Report::new(TrustedServerError::KvStore { + store_name: self.inner.store_name().to_owned(), + message: "store unavailable".to_owned(), + })); + } + if matches!(write.mode, EcKvWriteMode::IfGenerationMatch(_)) { + if self + .failing + .lock() + .expect("should lock the failures") + .contains(key) + { + return Err(Report::new(TrustedServerError::KvStore { + store_name: self.inner.store_name().to_owned(), + message: "store unavailable".to_owned(), + })); + } + let mut refusals = self.refusals.lock().expect("should lock the refusals"); + if let Some(remaining) = refusals.get_mut(key) + && *remaining > 0 + { + *remaining -= 1; + return Ok(EcKvWriteOutcome::PreconditionFailed); + } + } + self.inner.insert(key, write) + } + + fn list_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result, Report> { + self.inner.list_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + self.inner.delete(key) + } + + fn pause(&self, duration: Duration) { + self.pauses + .lock() + .expect("should lock the pauses") + .push(duration); + } +} + +/// The harness clock when a test does not set one. +const HARNESS_NOW: u64 = 1_760_000_000; + +/// Inputs for one harness run. The harness's own `with_*` methods delegate +/// to an inner `HarnessSetup`. +#[derive(Debug, Clone, Default)] +pub struct HarnessSetup { + client_ip: Option, + user_agent: Option, + page_url: Option, + region: Option, + hashed_email: Option, + consent: TestConsent, + stored: Vec, + state_json: Option, + now: Option, + random_seed: Option, +} + +impl HarnessSetup { + /// Sets the client IP. + /// + /// # Panics + /// + /// Panics when `ip` is not an IP address. + #[must_use] + pub fn with_client_ip(mut self, ip: &str) -> Self { + self.client_ip = Some(ip.parse().expect("should parse the client IP")); + self + } + + /// Sets the `User-Agent`. + #[must_use] + pub fn with_user_agent(mut self, user_agent: &str) -> Self { + self.user_agent = Some(user_agent.to_owned()); + self + } + + /// Sets the page URL, without its query string and fragment, as a + /// request gives it to a provider. + /// + /// # Panics + /// + /// Panics when `url` is not a URL. + #[must_use] + pub fn with_page_url(mut self, url: &str) -> Self { + let mut url = Url::parse(url).expect("should parse the page URL"); + url.set_query(None); + url.set_fragment(None); + self.page_url = Some(url); + self + } + + /// Sets the region. + /// + /// # Panics + /// + /// Panics when `country` is not a country code. + #[must_use] + pub fn with_region(mut self, country: &str, subdivision: Option<&str>) -> Self { + self.region = + Some(Region::from_geo(country, subdivision).expect("should accept the country")); + self + } + + /// Sets the hashed email. + #[must_use] + pub fn with_hashed_email(mut self, email: HashedEmail) -> Self { + self.hashed_email = Some(email); + self + } + + /// Sets the raw consent strings. Default: GDPR does not apply and there + /// are no opt-outs. + #[must_use] + pub fn with_consent(mut self, consent: TestConsent) -> Self { + self.consent = consent; + self + } + + /// Gives the provider a stored record, valid or expired. Its source is + /// the one its first entry names, or the provider's primary source. + #[must_use] + pub fn with_stored(mut self, stored: StoredId) -> Self { + self.stored.push(stored); + self + } + + /// Gives the provider its private state, as stored. + /// + /// # Panics + /// + /// Panics when `state` does not serialize. + #[must_use] + pub fn with_state(mut self, state: &impl Serialize) -> Self { + self.state_json = Some(serde_json::to_string(state).expect("should serialize the state")); + self + } + + /// Sets the fixed clock. Default: 1,760,000,000. + #[must_use] + pub fn with_now(mut self, now: UnixTime) -> Self { + self.now = Some(now); + self + } + + /// Seeds the random source. Default: 1. + #[must_use] + pub fn with_random_seed(mut self, seed: u64) -> Self { + self.random_seed = Some(seed); + self + } + + fn now(&self) -> UnixTime { + self.now.unwrap_or(UnixTime::from_secs(HARNESS_NOW)) + } + + fn random(&self) -> SeededRandom { + SeededRandom::new(self.random_seed.unwrap_or(1)) + } +} + +/// Raw TCF, GPP and US Privacy strings, as a request would carry them. +#[derive(Debug, Clone, Default)] +pub struct TestConsent { + gdpr_applies: Option, + tc_string: Option, + gpp: Option<(String, Vec)>, + us_privacy: Option, + gpc: bool, + dnt: bool, +} + +impl TestConsent { + /// Sets whether GDPR applies, whatever the strings say. + #[must_use] + pub fn with_gdpr_applies(mut self, applies: bool) -> Self { + self.gdpr_applies = Some(applies); + self + } + + /// Sets the raw TCF string. + #[must_use] + pub fn with_tc_string(mut self, tc: &str) -> Self { + self.tc_string = Some(tc.to_owned()); + self + } + + /// Sets the raw GPP string and its section IDs. + #[must_use] + pub fn with_gpp_string(mut self, gpp: &str, section_ids: &[u16]) -> Self { + self.gpp = Some((gpp.to_owned(), section_ids.to_vec())); + self + } + + /// Sets the raw US Privacy string. + #[must_use] + pub fn with_us_privacy(mut self, us_privacy: &str) -> Self { + self.us_privacy = Some(us_privacy.to_owned()); + self + } + + /// Sets the Global Privacy Control signal. + #[must_use] + pub fn with_gpc(mut self) -> Self { + self.gpc = true; + self + } + + /// Sets the `DNT: 1` signal. + #[must_use] + pub fn with_dnt(mut self) -> Self { + self.dnt = true; + self + } + + /// The consent context a request carrying these signals would build. + fn context(&self) -> ConsentContext { + let signals = RawConsentSignals { + raw_tc_string: self.tc_string.clone(), + raw_gpp_string: self.gpp.as_ref().map(|(gpp, _)| gpp.clone()), + raw_gpp_sid: self + .gpp + .as_ref() + .map(|(_, ids)| ids.iter().map(u16::to_string).collect::>().join(",")), + raw_us_privacy: self.us_privacy.clone(), + gpc: self.gpc, + }; + let mut context = build_context_from_signals(&signals); + // A known non-regulated region, so only the strings given matter. + context.jurisdiction = Jurisdiction::NonRegulated; + if let Some(applies) = self.gdpr_applies { + context.gdpr_applies = applies; + } + context + } +} + +/// One proxied call: the request a capture rule matches, and its response. +#[derive(Debug, Clone)] +pub struct CaptureFixture { + method: Method, + path: String, + status: StatusCode, + body: Vec, +} + +impl CaptureFixture { + /// A proxied call with `method` and `path`, answered 200 with no body. + #[must_use] + pub fn new(method: Method, path: &str) -> Self { + Self { + method, + path: path.to_owned(), + status: StatusCode::OK, + body: Vec::new(), + } + } + + /// Sets the response status and JSON body. + /// + /// # Panics + /// + /// Panics when `body` does not serialize. + #[must_use] + pub fn with_response(mut self, status: StatusCode, body: &serde_json::Value) -> Self { + self.status = status; + self.body = serde_json::to_vec(body).expect("should serialize the response body"); + self + } +} + +#[derive(Debug, Clone)] +struct ResolveCase { + setup: HarnessSetup, + status: StatusCode, + headers: Vec<(String, String)>, + body: Vec, +} + +/// The setups to run `request` under, the responses to answer with, and the +/// proxied calls to capture from, for one conformance run. +#[derive(Debug, Clone, Default)] +pub struct ConformanceCases { + resolve: Vec, + capture: Vec, +} + +impl ConformanceCases { + /// A setup to run `request` under, and the response to answer it with. + /// + /// # Panics + /// + /// Panics when `response` does not serialize. + #[must_use] + pub fn with_resolve_case( + self, + setup: HarnessSetup, + status: StatusCode, + response: &serde_json::Value, + ) -> Self { + let body = serde_json::to_vec(response).expect("should serialize the response body"); + self.with_resolve_case_raw(setup, status, &[], &body) + } + + /// As [`Self::with_resolve_case`], for text, binary or empty replies, + /// such as a base64 body or a 204. + #[must_use] + pub fn with_resolve_case_raw( + mut self, + setup: HarnessSetup, + status: StatusCode, + headers: &[(&str, &str)], + body: &[u8], + ) -> Self { + self.resolve.push(ResolveCase { + setup, + status, + headers: headers + .iter() + .map(|(name, value)| ((*name).to_owned(), (*value).to_owned())) + .collect(), + body: body.to_vec(), + }); + self + } + + /// A proxied call to run `capture` on. + #[must_use] + pub fn with_capture_case(mut self, fixture: CaptureFixture) -> Self { + self.capture.push(fixture); + self + } +} + +/// The first conformance check a provider failed. Each variant names what +/// failed and never a value. +#[derive(Debug, derive_more::Display)] +#[non_exhaustive] +pub enum ConformanceFailure { + /// A read of something the descriptor does not declare. + #[display("the provider read `{access}` without declaring it")] + UndeclaredInput { + /// What was read. + access: Access, + }, + /// A request to a host the descriptor does not declare. + #[display("a request went to a host the descriptor does not declare")] + UndeclaredHost, + /// A request that is not `https`. + #[display("a request did not use https")] + InsecureScheme, + /// A request that sets a `Cookie` header. + #[display("a request set a Cookie header")] + CookieHeader, + /// A request header whose value is not visible ASCII. + #[display("a request header value is not valid")] + InvalidHeader, + /// An entry for a source the descriptor does not declare. + #[display("a result holds an entry for undeclared source `{source}`")] + UndeclaredSource { + /// The source. + source: SourceDomain, + }, + /// A fixture result that hits a limit of spec 4.5. + #[display("a result for `{source}` hit a limit: {part} was dropped")] + LimitHit { + /// The source. + source: SourceDomain, + /// What the limit dropped. + part: String, + }, + /// A fixture set over `MAX_ENTRY_SET_BYTES`. + #[display("the entry set for `{source}` is over the size limit")] + SetTooLarge { + /// The source. + source: SourceDomain, + }, + /// A capability the descriptor declares and the provider does not + /// override. + #[display("`{method}` is declared but not implemented")] + NotImplemented { + /// The missing method. + method: &'static str, + }, + /// A provider that declares `resolves` and builds no request in any case. + #[display("the provider built no request for any listed setup")] + ResolverNeverRequests, + /// A provider that declares `captures` and captures nothing in any case. + #[display("the provider captured nothing for any listed fixture")] + CapturerNeverMatches, + /// A provider without `with_private_state()` that returns state. + #[display("a provider without private state returned a state change")] + StatelessState, + /// A capture that returns a state change. + #[display("a capture returned a state change")] + CaptureStateChange, + /// A fixture set of an expiring source without an expiry. + #[display("the entry set for expiring source `{source}` carries no expiry")] + MissingExpiry { + /// The source. + source: SourceDomain, + }, + /// A stateful provider that accepts a bare-JSON-string stored state. + #[display("the provider accepted a bare-string stored state as valid")] + BareStringStateAccepted, + /// A returned state that does not round-trip through the state type. + #[display("a returned state does not round-trip through the state type")] + StateRoundTrip, + /// A stateful provider checked without a state type. + #[display( + "the provider keeps private state: use `check_conformance_with_state` with its state type" + )] + StatefulNeedsStateType, + /// A provider bug that is not one of the checks above. + #[display("the provider failed in `{stage}` with a configuration error")] + ProviderFailed { + /// The failing method. + stage: &'static str, + }, +} + +impl core::error::Error for ConformanceFailure {} + +/// Checks that a returned state round-trips through the provider's state type. +type StateRoundTrip<'a, 'provider> = dyn Fn(&ProviderHarness<'provider>, &str) -> bool + 'a; + +/// Drives one provider the way the coordinator does, so a vendor's tests +/// stay short. +/// +/// A source with a [`with_stored`](Self::with_stored) record is a `Refresh`, +/// any other an `Initial`. Sources whose kind needs an input the harness +/// does not hold are left out, and with none left +/// [`request`](Self::request) returns `Ok(None)` without calling the +/// provider. So does a consent the provider's policy refuses. +pub struct ProviderHarness<'provider> { + provider: &'provider dyn IdentityProvider, + setup: HarnessSetup, + random: SeededRandom, + ec_id: Option, + last_request: Option, + recorded: Vec, + state_decoded: bool, +} + +impl<'provider> ProviderHarness<'provider> { + /// A harness for `provider` with no inputs, the clock at 1,760,000,000 + /// and random seed 1. + #[must_use] + pub fn new(provider: &'provider dyn IdentityProvider) -> Self { + let setup = HarnessSetup::default(); + Self { + provider, + random: setup.random(), + setup, + ec_id: None, + last_request: None, + recorded: Vec::new(), + state_decoded: false, + } + } + + /// Replaces every input at once, for conformance cases. + #[must_use] + pub fn with_setup(mut self, setup: HarnessSetup) -> Self { + self.random = setup.random(); + self.setup = setup; + self + } + + fn map_setup(mut self, change: impl FnOnce(HarnessSetup) -> HarnessSetup) -> Self { + self.setup = change(core::mem::take(&mut self.setup)); + self + } + + /// Sets the client IP. + /// + /// # Panics + /// + /// Panics when `ip` is not an IP address. + #[must_use] + pub fn with_client_ip(self, ip: &str) -> Self { + self.map_setup(|setup| setup.with_client_ip(ip)) + } + + /// Sets the `User-Agent`. + #[must_use] + pub fn with_user_agent(self, user_agent: &str) -> Self { + self.map_setup(|setup| setup.with_user_agent(user_agent)) + } + + /// Sets the page URL. + /// + /// # Panics + /// + /// Panics when `url` is not a URL. + #[must_use] + pub fn with_page_url(self, url: &str) -> Self { + self.map_setup(|setup| setup.with_page_url(url)) + } + + /// Sets the region. + /// + /// # Panics + /// + /// Panics when `country` is not a country code. + #[must_use] + pub fn with_region(self, country: &str, subdivision: Option<&str>) -> Self { + self.map_setup(|setup| setup.with_region(country, subdivision)) + } + + /// Sets the hashed email. + #[must_use] + pub fn with_hashed_email(self, email: HashedEmail) -> Self { + self.map_setup(|setup| setup.with_hashed_email(email)) + } + + /// Sets the raw consent strings. Default: GDPR does not apply and no + /// opt-outs. + #[must_use] + pub fn with_consent(self, consent: TestConsent) -> Self { + self.map_setup(|setup| setup.with_consent(consent)) + } + + /// Gives the provider a stored record, valid or expired. Its source is + /// the one its first entry names, or the provider's primary source. + #[must_use] + pub fn with_stored(self, stored: StoredId) -> Self { + self.map_setup(|setup| setup.with_stored(stored)) + } + + /// Gives the provider its private state, as stored. + /// + /// # Panics + /// + /// Panics when `state` does not serialize. + #[must_use] + pub fn with_state(self, state: &impl Serialize) -> Self { + self.map_setup(|setup| setup.with_state(state)) + } + + /// Sets the fixed clock. + #[must_use] + pub fn with_now(self, now: UnixTime) -> Self { + self.map_setup(|setup| setup.with_now(now)) + } + + /// Reseeds the random source. + #[must_use] + pub fn with_random_seed(mut self, seed: u64) -> Self { + self.setup = core::mem::take(&mut self.setup).with_random_seed(seed); + self.random = self.setup.random(); + self + } + + /// Gives the provider the EC ID. Only generic pull reads it. + #[must_use] + pub(crate) fn with_ec_id(mut self, ec_id: &str) -> Self { + self.ec_id = Some(ec_id.to_owned()); + self + } + + /// Calls `request` and keeps the built request and its call context. + /// + /// # Errors + /// + /// Returns what the provider returns. + pub fn request(&mut self) -> Result, Report> { + let kinds = self.kinds(); + let (result, recorded, decoded) = if kinds.is_empty() || !self.consent_allows() { + (Ok(None), Vec::new(), false) + } else { + self.call(kinds, |input| self.provider.request(input)) + }; + self.recorded.extend(recorded); + self.state_decoded |= decoded; + let request = result?; + self.last_request.clone_from(&request); + Ok(request) + } + + /// Decodes the last built request's JSON body. + /// + /// # Panics + /// + /// Panics when no request was built or its body is not a `T`. + #[must_use] + pub fn request_json(&self) -> T { + let request = self + .last_request + .as_ref() + .expect("should have built a request"); + serde_json::from_slice(request.body()).expect("should decode the request body") + } + + /// Calls `response` with the kept call context. + /// + /// # Errors + /// + /// Returns what the provider returns. + /// + /// # Panics + /// + /// Panics when `body` does not serialize. + pub fn respond( + &mut self, + status: StatusCode, + body: &serde_json::Value, + ) -> Result> { + let bytes = serde_json::to_vec(body).expect("should serialize the response body"); + self.respond_raw(status, &[], &bytes) + } + + /// Calls `response` with `headers` and a raw `body`, such as an empty 204. + /// + /// # Errors + /// + /// Returns what the provider returns. + /// + /// # Panics + /// + /// Panics when a header is not valid. + pub fn respond_raw( + &mut self, + status: StatusCode, + headers: &[(&str, &str)], + body: &[u8], + ) -> Result> { + let mut header_map = HeaderMap::new(); + for (name, value) in headers { + header_map.insert( + HeaderName::from_bytes(name.as_bytes()).expect("should be a header name"), + HeaderValue::from_str(value).expect("should be a header value"), + ); + } + let call_context = self + .last_request + .as_ref() + .and_then(|request| request.call_context_json().map(str::to_owned)); + let response = ProviderResponse::new(status, header_map, body.to_vec(), call_context); + let (result, recorded, decoded) = self.call(self.kinds(), |input| { + self.provider.response(&response, input) + }); + self.recorded.extend(recorded); + self.state_decoded |= decoded; + result + } + + /// Runs a capture as the coordinator does: the fixture's response is + /// already decoded. A call no capture rule matches is `no_match`, as + /// the coordinator never runs the provider for it. + /// + /// # Errors + /// + /// Returns what the provider returns. + pub fn capture( + &self, + fixture: &CaptureFixture, + ) -> Result> { + let descriptor = self.provider.descriptor(); + if !descriptor + .captures() + .is_some_and(|captures| captures.matches(&fixture.method, &fixture.path)) + { + return Ok(Resolution::no_match()); + } + let response = + ProviderResponse::new(fixture.status, HeaderMap::new(), fixture.body.clone(), None); + let stored = self.stored_by_source(); + let consent = ProviderConsent::from_context(&self.setup.consent.context()); + let input = CaptureInput::new( + descriptor, + &fixture.path, + &response, + &stored, + &consent, + "www.example.com", + self.setup.now(), + ); + self.provider.capture(&input) + } + + /// Resolves an expiry or refresh against the fixed clock. + #[must_use] + pub fn resolve(&self, when: When) -> UnixTime { + when.resolve(self.setup.now()) + } + + /// The replacement state in a result, decoded, for assertions. + #[must_use] + pub fn state_of(&self, resolution: &Resolution) -> Option { + match resolution.state_change() { + StateChange::Replace(json) => serde_json::from_str(json).ok(), + StateChange::Keep | StateChange::Clear => None, + } + } + + /// The EIDs bidders would see for a result: the section 4.5 rules, then + /// the descriptor's provenance default for an entry that sets none of + /// `inserter`, `matcher` and `mm` (section 7.1). + #[must_use] + pub fn output_eids(&self, resolution: &Resolution) -> Vec { + let sets = entry_sets(resolution.eids(), self.setup.now()); + let mut order: Vec<&SourceDomain> = Vec::new(); + for eid in resolution.eids() { + if !order.contains(&eid.source()) { + order.push(eid.source()); + } + } + let descriptor = self.provider.descriptor(); + let mut output = Vec::new(); + for source in order { + let Some(set) = sets.sets.get(source) else { + continue; + }; + for entry in &set.entries { + let mut eid = Eid { + source: source.as_str().to_owned(), + uids: entry + .uids + .iter() + .map(|uid| Uid { + id: uid.id.clone(), + atype: uid.atype, + ext: uid.ext.clone(), + }) + .collect(), + inserter: entry.inserter.clone(), + matcher: entry.matcher.clone(), + mm: entry.mm, + ext: entry.ext.clone(), + }; + if let Some(default) = descriptor.provenance(source) + && eid.inserter.is_none() + && eid.matcher.is_none() + && eid.mm.is_none() + { + eid.inserter = default.inserter().map(str::to_owned); + eid.matcher = default.matcher().map(str::to_owned); + eid.mm = default.mm(); + } + output.push(eid); + } + } + output + } + + /// Runs only the cases passed in, never the harness's own inputs, and + /// fails the test on the first failed check. A provider declared + /// `with_private_state()` fails here with a pointer to + /// [`check_conformance_with_state`](Self::check_conformance_with_state). + /// + /// # Panics + /// + /// Panics when the provider does not conform. + pub fn check_conformance(&self, cases: &ConformanceCases) { + self.try_check_conformance(cases) + .expect("should conform: the provider failed a conformance check"); + } + + /// As [`check_conformance`](Self::check_conformance), for a provider with + /// private state of type `S`. Also checks that `S` round-trips through + /// `replacing_state` and `input.state()`. + /// + /// # Panics + /// + /// Panics when the provider does not conform. + pub fn check_conformance_with_state( + &self, + cases: &ConformanceCases, + ) { + self.try_check_conformance_with_state::(cases) + .expect("should conform: the provider failed a conformance check"); + } + + /// The same checks as [`check_conformance`](Self::check_conformance), + /// returning the first failure instead of failing the test. A panic + /// aborts a `wasm32-wasip1` test run, so the harness's own negative + /// tests use this. + /// + /// # Errors + /// + /// The first failed check. + pub fn try_check_conformance( + &self, + cases: &ConformanceCases, + ) -> Result<(), Report> { + self.conform(cases, None) + } + + /// As [`try_check_conformance`](Self::try_check_conformance), for a + /// provider with private state of type `S`. + /// + /// # Errors + /// + /// The first failed check. + pub fn try_check_conformance_with_state( + &self, + cases: &ConformanceCases, + ) -> Result<(), Report> { + self.conform( + cases, + Some(&|harness, json| harness.state_round_trips::(json)), + ) + } + + fn conform( + &self, + cases: &ConformanceCases, + round_trip: Option<&StateRoundTrip<'_, 'provider>>, + ) -> Result<(), Report> { + let descriptor = self.provider.descriptor(); + let stateful = descriptor + .resolves() + .is_some_and(Resolves::has_private_state); + if stateful && round_trip.is_none() { + return Err(Report::new(ConformanceFailure::StatefulNeedsStateType)); + } + + let mut requested = false; + for case in &cases.resolve { + let mut harness = Self::new(self.provider).with_setup(case.setup.clone()); + let request = harness.request(); + harness.check_recorded()?; + let request = match request { + Ok(Some(request)) => request, + Ok(None) => continue, + Err(error) => { + classify(error.current_context(), "request")?; + continue; + } + }; + requested = true; + check_request(descriptor, &request)?; + let headers: Vec<(&str, &str)> = case + .headers + .iter() + .map(|(name, value)| (name.as_str(), value.as_str())) + .collect(); + let resolution = harness.respond_raw(case.status, &headers, &case.body); + harness.check_recorded()?; + match resolution { + Ok(resolution) => { + if !stateful && resolution.changes_state() { + return Err(Report::new(ConformanceFailure::StatelessState)); + } + self.check_resolution(&resolution)?; + if let (StateChange::Replace(json), Some(round_trip)) = + (resolution.state_change(), round_trip) + && !round_trip(&harness, json) + { + return Err(Report::new(ConformanceFailure::StateRoundTrip)); + } + } + Err(error) => classify(error.current_context(), "response")?, + } + if stateful { + self.check_bare_string_state(case)?; + } + } + if descriptor.resolves().is_some() && !requested { + return Err(Report::new(ConformanceFailure::ResolverNeverRequests)); + } + + let mut captured = false; + for fixture in &cases.capture { + match self.capture(fixture) { + Ok(resolution) => { + if resolution.changes_state() { + return Err(Report::new(ConformanceFailure::CaptureStateChange)); + } + self.check_resolution(&resolution)?; + captured |= !resolution.is_no_match(); + } + Err(error) => classify(error.current_context(), "capture")?, + } + } + if descriptor.captures().is_some() && !captured { + return Err(Report::new(ConformanceFailure::CapturerNeverMatches)); + } + Ok(()) + } + + fn check_recorded(&self) -> Result<(), Report> { + match self.recorded.first() { + Some(access) => Err(Report::new(ConformanceFailure::UndeclaredInput { + access: *access, + })), + None => Ok(()), + } + } + + fn check_resolution(&self, resolution: &Resolution) -> Result<(), Report> { + let descriptor = self.provider.descriptor(); + for eid in resolution.eids() { + if !descriptor.sources().contains(eid.source()) { + return Err(Report::new(ConformanceFailure::UndeclaredSource { + source: eid.source().clone(), + })); + } + } + let sets = entry_sets(resolution.eids(), self.setup.now()); + if let Some(dropped) = sets.drops.first() { + return Err(Report::new(ConformanceFailure::LimitHit { + source: dropped.source.clone(), + part: dropped.part.to_string(), + })); + } + if let Some(source) = sets.oversized.first() { + return Err(Report::new(ConformanceFailure::SetTooLarge { + source: source.clone(), + })); + } + for (source, set) in &sets.sets { + if descriptor.is_expiring(source) && set.expires_at.is_none() { + return Err(Report::new(ConformanceFailure::MissingExpiry { + source: source.clone(), + })); + } + } + Ok(()) + } + + /// Injects a stored state that is a bare JSON string, which no struct + /// decodes, and fails when the provider decodes it as valid state. + fn check_bare_string_state( + &self, + case: &ResolveCase, + ) -> Result<(), Report> { + let mut setup = case.setup.clone(); + setup.state_json = Some("\"bare\"".to_owned()); + let mut probe = Self::new(self.provider).with_setup(setup); + if matches!(probe.request(), Ok(Some(_))) { + let _ignored = probe.respond_raw(case.status, &[], &case.body); + } + if probe.state_decoded { + return Err(Report::new(ConformanceFailure::BareStringStateAccepted)); + } + Ok(()) + } + + /// Whether `json` decodes as `S`, serializes back to the same value and + /// decodes again through `input.state()`. + fn state_round_trips(&self, json: &str) -> bool { + let Ok(state) = serde_json::from_str::(json) else { + return false; + }; + let Ok(again) = serde_json::to_string(&state) else { + return false; + }; + let same = serde_json::from_str::(json).ok() + == serde_json::from_str::(&again).ok(); + let mut setup = self.setup.clone(); + setup.state_json = Some(again); + let probe = Self::new(self.provider).with_setup(setup); + #[allow( + clippy::redundant_closure_for_method_calls, + reason = "the method reference does not satisfy the higher-ranked closure bound" + )] + let (decoded, _, _) = + probe.call(probe.kinds(), |input: &ResolveInput<'_>| input.state::()); + same && matches!(decoded, Ok(Some(_))) + } + + fn call( + &self, + kinds: BTreeMap, + run: impl FnOnce(&ResolveInput<'_>) -> R, + ) -> (R, Vec, bool) { + let signals = self.signals(); + let input = ResolveInput::new(self.provider.descriptor(), kinds, &signals) + .with_state_json(self.setup.state_json.clone()); + let result = run(&input); + (result, input.recorded_errors(), input.state_was_decoded()) + } + + fn consent_allows(&self) -> bool { + policy_allows( + self.provider.descriptor().consent_policy(), + &self.setup.consent.context(), + PolicySignals { + dnt: self.setup.consent.dnt, + }, + ) + } + + fn stored_by_source(&self) -> BTreeMap { + self.setup + .stored + .iter() + .map(|stored| { + let source = stored + .eids() + .first() + .and_then(|eid| eid.source.parse().ok()) + .unwrap_or_else(|| self.provider.descriptor().primary_source().clone()); + (source, stored.clone()) + }) + .collect() + } + + fn available_inputs(&self) -> BTreeSet { + let mut available = BTreeSet::new(); + for (input, present) in [ + (Input::ClientIp, self.setup.client_ip.is_some()), + (Input::UserAgent, self.setup.user_agent.is_some()), + (Input::PageUrl, self.setup.page_url.is_some()), + (Input::Region, self.setup.region.is_some()), + (Input::HashedEmail, self.setup.hashed_email.is_some()), + ] { + if present { + available.insert(input); + } + } + available + } + + fn kinds(&self) -> BTreeMap { + let descriptor = self.provider.descriptor(); + let Some(resolves) = descriptor.resolves() else { + return BTreeMap::new(); + }; + let available = self.available_inputs(); + let stored = self.stored_by_source(); + descriptor + .sources() + .iter() + .filter_map(|source| { + let kind = if stored.contains_key(source) { + ResolveKind::Refresh + } else { + ResolveKind::Initial + }; + (resolves.kinds().contains(&kind) && resolves.needs(kind).is_subset(&available)) + .then(|| (source.clone(), kind)) + }) + .collect() + } + + fn signals(&self) -> ResolveSignals<'_> { + ResolveSignals { + client_ip: self.setup.client_ip, + user_agent: self.setup.user_agent.as_deref(), + page_url: self.setup.page_url.clone(), + region: self.setup.region.clone(), + hashed_email: self.setup.hashed_email.clone(), + consent: ProviderConsent::from_context(&self.setup.consent.context()), + publisher_domain: "www.example.com", + now: self.setup.now(), + random: &self.random, + ec_id: self.ec_id.clone(), + stored: self.stored_by_source(), + } + } +} + +/// Maps a provider error to the conformance check it breaks. Failures the +/// coordinator treats as a vendor outcome, such as a refusal or a body that +/// does not parse, are not bugs. +fn classify( + error: &IdentityProviderError, + stage: &'static str, +) -> Result<(), Report> { + match error { + IdentityProviderError::UndeclaredInput { access } => { + Err(Report::new(ConformanceFailure::UndeclaredInput { + access: *access, + })) + } + error if error.is_not_implemented() => { + Err(Report::new(ConformanceFailure::NotImplemented { + method: stage, + })) + } + IdentityProviderError::Config { .. } => { + Err(Report::new(ConformanceFailure::ProviderFailed { stage })) + } + _ => Ok(()), + } +} + +/// The request rules of the coordinator, each as its own failure. +fn check_request( + descriptor: &ProviderDescriptor, + request: &ProviderRequest, +) -> Result<(), Report> { + if request.url().scheme() != "https" { + return Err(Report::new(ConformanceFailure::InsecureScheme)); + } + let host = request.url().host_str().unwrap_or_default(); + let declared = descriptor.resolves().is_some_and(|resolves| { + resolves + .hosts() + .iter() + .any(|allowed| allowed.eq_ignore_ascii_case(host)) + }); + if !declared { + return Err(Report::new(ConformanceFailure::UndeclaredHost)); + } + if request.has_cookie_header() { + return Err(Report::new(ConformanceFailure::CookieHeader)); + } + if request + .headers() + .values() + .any(|value| value.to_str().is_err()) + { + return Err(Report::new(ConformanceFailure::InvalidHeader)); + } + Ok(()) +} + +/// Builds the descriptor from a config with placeholder secret names and +/// from one with resolved secrets, and fails the test when the two differ +/// (spec 10.2). +/// +/// # Panics +/// +/// Panics when either config cannot describe a provider, or when the +/// descriptors differ. +pub fn check_descriptor_ignores_secrets(placeholder: &C, resolved: &C) { + let from_placeholder = placeholder + .build_descriptor() + .expect("should build the descriptor from the placeholder config"); + let from_resolved = resolved + .build_descriptor() + .expect("should build the descriptor from the resolved config"); + assert_eq!( + from_placeholder, from_resolved, + "should build the same descriptor whatever the secrets" + ); +} + +struct DescriptorOnly(ProviderDescriptor); + +impl IdentityProvider for DescriptorOnly { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } +} + +/// An in-memory [`RateLimiter`] that can be set to fail each method. +/// +/// `exceeded` counts like the Fastly limiter: it increments on every check +/// under the limit. `record` and `count` use the same counts. +#[derive(Debug, Default)] +pub struct FakeRateLimiter { + counts: Mutex>, + fail_exceeded: AtomicBool, + fail_record: AtomicBool, + fail_count: AtomicBool, +} + +#[allow(clippy::missing_panics_doc, reason = "the fake locks its own mutex")] +impl FakeRateLimiter { + /// An empty limiter whose methods all succeed. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Makes `exceeded` and `exceeded_per_minute` fail. + #[must_use] + pub fn failing_exceeded(self) -> Self { + self.fail_exceeded.store(true, Ordering::SeqCst); + self + } + + /// Makes `record` fail. + #[must_use] + pub fn failing_record(self) -> Self { + self.fail_record.store(true, Ordering::SeqCst); + self + } + + /// Makes `count` fail. + #[must_use] + pub fn failing_count(self) -> Self { + self.fail_count.store(true, Ordering::SeqCst); + self + } + + /// Sets the count of `key`. + #[must_use] + pub fn with_count(self, key: &str, count: u32) -> Self { + self.counts + .lock() + .expect("should lock the counts") + .insert(key.to_owned(), count); + self + } + + /// The count of `key`, 0 when it was never touched. + #[must_use] + pub fn count_of(&self, key: &str) -> u32 { + self.counts + .lock() + .expect("should lock the counts") + .get(key) + .copied() + .unwrap_or(0) + } + + /// Every key with a count, sorted. + #[must_use] + pub fn keys(&self) -> Vec { + let mut keys: Vec = self + .counts + .lock() + .expect("should lock the counts") + .keys() + .cloned() + .collect(); + keys.sort(); + keys + } + + fn unavailable() -> Report { + Report::new(TrustedServerError::Proxy { + message: "limiter unavailable".to_owned(), + }) + } +} + +impl RateLimiter for FakeRateLimiter { + fn exceeded(&self, key: &str, hourly_limit: u32) -> Result> { + if self.fail_exceeded.load(Ordering::SeqCst) { + return Err(Self::unavailable()); + } + let limit = hourly_limit.div_ceil(60); + let mut counts = self.counts.lock().expect("should lock the counts"); + let current = counts.entry(key.to_owned()).or_insert(0); + if *current >= limit { + return Ok(true); + } + *current += 1; + Ok(false) + } + + fn record(&self, key: &str) -> Result<(), Report> { + if self.fail_record.load(Ordering::SeqCst) { + return Err(Self::unavailable()); + } + *self + .counts + .lock() + .expect("should lock the counts") + .entry(key.to_owned()) + .or_insert(0) += 1; + Ok(()) + } + + fn count(&self, key: &str) -> Result> { + if self.fail_count.load(Ordering::SeqCst) { + return Err(Self::unavailable()); + } + Ok(self.count_of(key)) + } +} + +/// A fake HTTP client, backend and provider for dispatch tests. +#[cfg(any(test, feature = "test-utils"))] +#[allow( + clippy::missing_panics_doc, + clippy::new_without_default, + reason = "test doubles lock their own mutexes and are built with new" +)] +pub mod fake_http { + use std::collections::BTreeMap; + use std::sync::{Arc, Mutex}; + use std::time::Duration; + + use error_stack::Report; + + use crate::ec::identity::descriptor::EidProvenance; + use crate::ec::identity::descriptor::{ConsentPolicy, ProviderDescriptor, Resolves}; + use crate::ec::identity::exchange::{ProviderRequest, ProviderResponse}; + use crate::ec::identity::ids::SourceDomain; + use crate::ec::identity::input::ResolveInput; + use crate::ec::identity::provider::IdentityProvider; + use crate::ec::identity::resolution::{ + IdentityProviderError, Resolution, ResolvedEid, ResolvedUid, + }; + #[cfg(test)] + use crate::platform::test_support::{NoopConfigStore, NoopGeo, NoopSecretStore}; + use crate::platform::{ + BackendNamingPolicy, PlatformBackend, PlatformBackendSpec, PlatformError, + PlatformHttpClient, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, + PlatformSelectResult, + }; + #[cfg(test)] + use crate::platform::{ClientInfo, RuntimeServices}; + use serde_json::json; + + /// What the fake does for one host. + #[derive(Clone)] + enum Route { + Respond { + status: u16, + headers: Vec<(String, String)>, + body: Vec, + }, + Fail, + } + + struct FakePending { + host: String, + route: Route, + delay: Duration, + /// The first-byte timeout the request's backend was registered with. + timeout: Option, + } + + /// An HTTP client that answers per host, and records what it was sent. + pub struct FakeHttpClient { + routes: Mutex>, + delays: Mutex>, + sent: Mutex>, + uris: Mutex>, + backend: Mutex>>, + earlier: Mutex, + } + + impl FakeHttpClient { + /// A client with no routes. + #[must_use] + pub fn new() -> Self { + Self { + routes: Mutex::new(BTreeMap::new()), + delays: Mutex::new(BTreeMap::new()), + sent: Mutex::new(Vec::new()), + uris: Mutex::new(Vec::new()), + backend: Mutex::new(None), + earlier: Mutex::new(0), + } + } + + /// Counts `count` backend requests as started before any provider + /// call, as the origin or the auction would. + pub fn started_earlier(&self, count: usize) { + *self.earlier.lock().expect("should lock the earlier count") = count; + } + + /// Makes a wait that outlasts the first-byte timeout `backend` last + /// registered for the host fail once that timeout has passed, as a + /// real backend does. + pub fn honour_timeouts_of(&self, backend: &Arc) { + *self.backend.lock().expect("should lock the backend") = Some(Arc::clone(backend)); + } + + /// Answers `host` with `status` and `body`. + pub fn respond(&self, host: &str, status: u16, body: &[u8]) { + self.respond_with_headers(host, status, body, &[]); + } + + /// Answers `host` with `status`, `headers` and `body`. + pub fn respond_with_headers( + &self, + host: &str, + status: u16, + body: &[u8], + headers: &[(&str, &str)], + ) { + let route = Route::Respond { + status, + headers: headers + .iter() + .map(|(name, value)| ((*name).to_owned(), (*value).to_owned())) + .collect(), + body: body.to_vec(), + }; + self.routes + .lock() + .expect("should lock the routes") + .insert(host.to_owned(), route); + } + + /// Makes the wait for `host` fail, as a first-byte timeout does. + pub fn time_out(&self, host: &str) { + self.routes + .lock() + .expect("should lock the routes") + .insert(host.to_owned(), Route::Fail); + } + + /// Makes the wait for `host` take `delay` of wall-clock time. + pub fn delay(&self, host: &str, delay: Duration) { + self.delays + .lock() + .expect("should lock the delays") + .insert(host.to_owned(), delay); + } + + /// The URIs requests were sent to, in order. + pub fn sent_uris(&self) -> Vec { + self.uris.lock().expect("should lock the sent URIs").clone() + } + + /// The hosts requests were sent to, in order. + pub fn sent_hosts(&self) -> Vec { + self.sent + .lock() + .expect("should lock the sent hosts") + .clone() + } + } + + #[async_trait::async_trait(?Send)] + impl PlatformHttpClient for FakeHttpClient { + fn requests_started(&self) -> Option { + let earlier = *self.earlier.lock().expect("should lock the earlier count"); + Some(earlier + self.sent.lock().expect("should lock the sent hosts").len()) + } + + async fn send( + &self, + _request: PlatformHttpRequest, + ) -> Result> { + Err(Report::new(PlatformError::Unsupported)) + } + + async fn send_async( + &self, + request: PlatformHttpRequest, + ) -> Result> { + let host = request.request.uri().host().unwrap_or_default().to_owned(); + self.sent + .lock() + .expect("should lock the sent hosts") + .push(host.clone()); + self.uris + .lock() + .expect("should lock the sent URIs") + .push(request.request.uri().to_string()); + let route = self + .routes + .lock() + .expect("should lock the routes") + .get(&host) + .cloned() + .ok_or_else(|| Report::new(PlatformError::HttpClient))?; + let delay = self + .delays + .lock() + .expect("should lock the delays") + .get(&host) + .copied() + .unwrap_or_default(); + let timeout = self + .backend + .lock() + .expect("should lock the backend") + .as_ref() + .and_then(|backend| backend.last_first_byte_timeout(&host)); + Ok(PlatformPendingRequest::new(FakePending { + host, + route, + delay, + timeout, + })) + } + + async fn select( + &self, + mut pending_requests: Vec, + ) -> Result> { + if pending_requests.is_empty() { + return Err(Report::new(PlatformError::HttpClient)); + } + let first = pending_requests.remove(0); + let ready = self.wait(first).await; + Ok(PlatformSelectResult { + ready, + remaining: pending_requests, + failed_backend_name: None, + }) + } + + async fn wait( + &self, + pending: PlatformPendingRequest, + ) -> Result> { + let pending = pending + .downcast::() + .map_err(|_| Report::new(PlatformError::HttpClient))?; + if let Some(timeout) = pending.timeout + && pending.delay > timeout + { + std::thread::sleep(timeout); + return Err(Report::new(PlatformError::HttpClient)); + } + if !pending.delay.is_zero() { + std::thread::sleep(pending.delay); + } + match pending.route { + Route::Fail => Err(Report::new(PlatformError::HttpClient)), + Route::Respond { + status, + headers, + body, + } => { + let mut builder = edgezero_core::http::response_builder().status(status); + for (name, value) in headers { + builder = builder.header(name, value); + } + let response = builder + .body(edgezero_core::body::Body::from(body)) + .map_err(|_| Report::new(PlatformError::HttpClient))?; + Ok(PlatformResponse::new(response).with_backend_name(pending.host)) + } + } + } + } + + /// A backend named after its host, that records each spec it is given. + pub struct FakeBackend { + specs: Mutex>, + } + + impl FakeBackend { + /// A backend that has registered nothing. + #[must_use] + pub fn new() -> Self { + Self { + specs: Mutex::new(Vec::new()), + } + } + + /// The first-byte timeout `host` was last registered with. + fn last_first_byte_timeout(&self, host: &str) -> Option { + self.specs + .lock() + .expect("should lock the specs") + .iter() + .rev() + .find(|spec| spec.host == host) + .map(|spec| spec.first_byte_timeout) + } + + /// The connect timeouts the backends were registered with, in order. + pub fn connect_timeouts(&self) -> Vec> { + self.specs + .lock() + .expect("should lock the specs") + .iter() + .map(|spec| spec.connect_timeout) + .collect() + } + + /// The first-byte timeouts the backends were registered with, in order. + pub fn first_byte_timeouts(&self) -> Vec { + self.specs + .lock() + .expect("should lock the specs") + .iter() + .map(|spec| spec.first_byte_timeout) + .collect() + } + } + + impl PlatformBackend for FakeBackend { + fn naming_policy(&self) -> BackendNamingPolicy { + BackendNamingPolicy::Axum + } + + fn predict_name( + &self, + spec: &PlatformBackendSpec, + ) -> Result> { + Ok(spec.host.clone()) + } + + fn ensure(&self, spec: &PlatformBackendSpec) -> Result> { + self.specs + .lock() + .expect("should lock the specs") + .push(spec.clone()); + Ok(spec.host.clone()) + } + } + + /// Services that send through `client` and register backends in `backend`. + #[cfg(test)] + pub(crate) fn fake_services( + client: &Arc, + backend: &Arc, + ) -> RuntimeServices { + fake_services_from(client, backend, ClientInfo::default()) + } + + /// [`fake_services`] for a request from `client_info`. + #[cfg(test)] + pub(crate) fn fake_services_from( + client: &Arc, + backend: &Arc, + client_info: ClientInfo, + ) -> RuntimeServices { + client.honour_timeouts_of(backend); + RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore)) + .backend(backend.clone()) + .http_client(client.clone()) + .geo(Arc::new(NoopGeo)) + .client_info(client_info) + .build() + } + + /// A resolver that calls its endpoint and answers `no_match` to a 2xx. + pub(crate) struct ScriptedProvider { + descriptor: ProviderDescriptor, + reads_user_agent: bool, + stateful: bool, + } + + impl ScriptedProvider { + /// A provider `id` calling `host`, claiming `source`. With `generic` + /// it is a generic pull provider. + pub(crate) fn new(id: &str, host: &str, source: &str, generic: bool) -> Self { + let mut resolves = Resolves::new().with_endpoint(&format!("https://{host}/resolve")); + if generic { + resolves = resolves.with_ec_id(); + } + let descriptor = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_source(source.parse().expect("should parse the source")) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::eid_persistence_only()) + .build() + .expect("should build the descriptor"); + Self { + descriptor, + reads_user_agent: false, + stateful: false, + } + } + + /// Reads `user_agent()` without declaring it, and ignores the error. + pub(crate) fn reading_undeclared_user_agent(mut self) -> Self { + self.reads_user_agent = true; + self + } + + /// Declares private state, and sends what it read in a `state` query. + pub(crate) fn stateful(mut self) -> Self { + self.stateful = true; + self.descriptor = ProviderDescriptor::builder(self.descriptor.id().clone()) + .with_source(self.descriptor.primary_source().clone()) + .with_resolves( + Resolves::new() + .with_endpoint( + self.descriptor + .endpoint() + .expect("should have an endpoint") + .as_str(), + ) + .with_private_state(), + ) + .with_consent_policy(ConsentPolicy::eid_persistence_only()) + .build() + .expect("should build the descriptor"); + self + } + + pub(crate) fn into_provider(self) -> Arc { + Arc::new(self) + } + } + + impl IdentityProvider for ScriptedProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + if self.reads_user_agent { + let _ = input.user_agent().ok(); + } + let mut request = ProviderRequest::get(input.endpoint()?); + if self.stateful { + let state = input.state::()?; + request = request.with_query( + "state", + &state.map_or_else(|| "none".to_owned(), |state| state.to_string()), + ); + } + Ok(Some(request)) + } + + fn response( + &self, + response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + response.success_text().map(|_| Resolution::no_match()) + } + } + + /// A resolver that calls its endpoint and answers as the 2xx JSON body + /// says: `{"answer":"found","entries":[{"uids":[{"id":"x","atype":1}], + /// "inserter":"example.com"}]}`, `{"answer":"invalidate"}` or anything + /// else for `no_match`. + pub(crate) struct AnsweringProvider { + descriptor: ProviderDescriptor, + stateful: bool, + } + + impl AnsweringProvider { + /// A provider `id` calling `host`, claiming `source`, with `default` + /// as its provenance default for the source when given. + pub(crate) fn new( + id: &str, + host: &str, + source: &str, + default: Option, + ) -> Self { + Self::build(id, host, source, default, false, false) + } + + /// As [`Self::new`], declaring private state that a `found` replaces. + pub(crate) fn stateful(id: &str, host: &str, source: &str) -> Self { + Self::build(id, host, source, None, true, false) + } + + /// As [`Self::new`], reading the EC ID, as generic pull does. + pub(crate) fn reading_ec_id(id: &str, host: &str, source: &str) -> Self { + Self::build(id, host, source, None, false, true) + } + + fn build( + id: &str, + host: &str, + source: &str, + default: Option, + stateful: bool, + ec_id: bool, + ) -> Self { + let mut resolves = Resolves::new().with_endpoint(&format!("https://{host}/resolve")); + if stateful { + resolves = resolves.with_private_state(); + } + if ec_id { + resolves = resolves.with_ec_id(); + } + let source: SourceDomain = source.parse().expect("should parse the source"); + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_source(source.clone()) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::eid_persistence_only()); + if let Some(default) = default { + builder = builder.with_provenance(&source, default); + } + Self { + descriptor: builder.build().expect("should build the descriptor"), + stateful, + } + } + + pub(crate) fn into_provider(self) -> Arc { + Arc::new(self) + } + } + + impl IdentityProvider for AnsweringProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + if self.stateful { + input.state::()?; + } + Ok(Some(ProviderRequest::get(input.endpoint()?))) + } + + fn response( + &self, + response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let body: serde_json::Value = response.success_json()?; + let source = self.descriptor.primary_source(); + match body["answer"].as_str() { + Some("found") => { + let eids = body["entries"] + .as_array() + .map(Vec::as_slice) + .unwrap_or_default() + .iter() + .map(|entry| { + let mut uids = + entry["uids"].as_array().into_iter().flatten().map(|uid| { + let mut resolved = ResolvedUid::new( + uid["id"].as_str().expect("should have a UID"), + ); + if let Some(atype) = uid["atype"].as_i64() { + resolved = resolved.with_atype( + i32::try_from(atype).expect("should fit an atype"), + ); + } + resolved + }); + let mut eid = ResolvedEid::new( + source, + uids.next().expect("should have a first UID"), + ); + for uid in uids { + eid = eid.with_uid(uid); + } + if let Some(inserter) = entry["inserter"].as_str() { + eid = eid.with_inserter(inserter); + } + eid + }); + let mut resolution = Resolution::found(eids); + if self.stateful { + resolution = resolution.replacing_state(&json!({ "seen": true })); + } + Ok(resolution) + } + Some("invalidate") => Ok(Resolution::invalidate(source)), + _ => Ok(Resolution::no_match()), + } + } + } +} + +/// A returning writer navigation with a live row, hold-mode providers and a +/// fake client, for the tests of hold dispatch and the flush that follows. +#[cfg(test)] +pub(crate) mod hold_fixture { + use std::collections::BTreeMap; + use std::sync::Arc; + + use http::StatusCode; + + use super::fake_http::{FakeBackend, FakeHttpClient, fake_services}; + use super::{FakeRateLimiter, TEST_EC_ID, WriteCountingEcKv, allowing_consent}; + use crate::ec::EcContext; + use crate::ec::finalize::BrowserSeed; + use crate::ec::identity::coordinator::{HoldResults, run_hold}; + use crate::ec::identity::flush::{AfterSendInput, FlushOutcome, run_after_send}; + use crate::ec::identity::ids::{IdentityProviderId, SourceDomain}; + use crate::ec::identity::input::ThreadRandom; + use crate::ec::identity::provider::IdentityProvider; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::runtime::IdentityRuntime; + use crate::ec::kv::KvIdentityGraph; + use crate::ec::kv_types::KvEntry; + use crate::platform::RuntimeServices; + use crate::settings::{ProviderMode, ProviderSettings, Settings}; + use crate::test_support::tests::create_test_settings; + + /// A live, consenting row sized for the flush, with no IDs. + pub(crate) fn live_row() -> KvEntry { + let mut entry = KvEntry::new(&allowing_consent(), None, 1_000, "test-publisher.com"); + entry.network = Some(crate::ec::kv_types::KvNetwork { + cluster_size: Some(1), + }); + entry + } + + /// Everything one hold navigation reads. + pub(crate) struct HoldFixture { + pub(crate) settings: Settings, + pub(crate) runtime: Arc, + pub(crate) client: Arc, + pub(crate) backend: Arc, + pub(crate) services: RuntimeServices, + pub(crate) graph: KvIdentityGraph, + pub(crate) store: Arc, + pub(crate) context: EcContext, + pub(crate) limiter: Arc, + } + + impl HoldFixture { + /// A writer navigation with `row` read in phase 1. Each provider runs + /// in the given mode. With `runs_auction` the navigation runs one. + pub(crate) fn new( + providers: Vec<(Arc, ProviderMode)>, + row: &KvEntry, + runs_auction: bool, + ) -> Self { + let mut settings = create_test_settings(); + for (provider, mode) in &providers { + settings.ec.identity.providers.insert( + provider.descriptor().id().to_string(), + ProviderSettings { + mode: *mode, + ..ProviderSettings::default() + }, + ); + } + let runtime = Arc::new( + IdentityRuntime::from_settings_with_providers( + &settings, + providers + .into_iter() + .map(|(provider, _)| provider) + .collect(), + BTreeMap::::new(), + ) + .expect("should build the runtime"), + ); + let client = Arc::new(FakeHttpClient::new()); + let backend = Arc::new(FakeBackend::new()); + let services = fake_services(&client, &backend); + let (graph, store) = KvIdentityGraph::write_counting("hold-store"); + graph.create(TEST_EC_ID, row).expect("should seed the row"); + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + allowing_consent(), + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(Arc::clone(&runtime)); + context.identity_mut().set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + context.identity_mut().set_runs_auction(runs_auction); + context.set_kv_snapshot(graph.load_snapshot(TEST_EC_ID)); + store.reset_counts(); + Self { + settings, + runtime, + client, + backend, + services, + graph, + store, + context, + limiter: Arc::new(FakeRateLimiter::new()), + } + } + + /// A new visitor's writer navigation: no row, and an EC minted with its + /// create deferred until after the hold dispatch (spec 4.7). + pub(crate) fn new_visitor( + providers: Vec<(Arc, ProviderMode)>, + ) -> Self { + let mut fixture = Self::new(providers, &live_row(), true); + let (graph, store) = KvIdentityGraph::write_counting("hold-store"); + let mut context = EcContext::new_for_test_with_ip( + None, + allowing_consent(), + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(Arc::clone(&fixture.runtime)); + context.identity_mut().set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + context.identity_mut().set_runs_auction(true); + context + .mint_deferred(&fixture.settings, &BrowserSeed::empty()) + .expect("should mint the EC"); + fixture.graph = graph; + fixture.store = store; + fixture.context = context; + fixture + } + + /// Runs the hold dispatch before the auction. + /// + /// It runs inside an executor, as the publisher handler does, so a + /// nested `block_on` anywhere in the hold dispatch panics every hold + /// test instead of only failing in production. + pub(crate) fn run_hold(&mut self) -> HoldResults { + let random = ThreadRandom; + let deps = self.runtime.deps( + &self.settings, + &self.services, + &random, + Some(self.limiter.as_ref()), + ); + futures::executor::block_on(run_hold(&deps, &mut self.context)) + } + + /// Runs phase 4 for a response with `status`. + pub(crate) fn flush(&mut self, status: StatusCode) -> FlushOutcome { + run_after_send(AfterSendInput { + runtime: &self.runtime, + settings: &self.settings, + services: &self.services, + graph: &self.graph, + context: &mut self.context, + status, + rate_limiter: self.limiter.as_ref(), + }) + } + + /// The row as KV holds it. + pub(crate) fn row(&self) -> KvEntry { + self.graph + .get(TEST_EC_ID) + .expect("should read the row") + .expect("should hold the row") + .0 + } + } +} + +/// A proxied API call from a browser with a live row and capture-only +/// providers, for the tests of the capture removal write. +#[cfg(test)] +pub(crate) mod capture_fixture { + use std::collections::BTreeMap; + use std::sync::Arc; + + use http::{HeaderMap, Method, StatusCode}; + + use super::fake_http::{FakeBackend, FakeHttpClient, fake_services}; + use super::{TEST_EC_ID, WriteCountingEcKv, allowing_consent}; + use crate::consent::ConsentContext; + use crate::ec::EcContext; + use crate::ec::identity::coordinator::{CaptureCall, run_capture}; + use crate::ec::identity::flush::retry_pending_removals; + use crate::ec::identity::ids::{IdentityProviderId, SourceDomain}; + use crate::ec::identity::input::ThreadRandom; + use crate::ec::identity::provider::IdentityProvider; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::runtime::IdentityRuntime; + use crate::ec::kv::KvIdentityGraph; + use crate::ec::kv_types::KvEntry; + use crate::platform::RuntimeServices; + use crate::settings::{ProviderSettings, RevokeScope, Settings}; + use crate::test_support::tests::create_test_settings; + + /// The path every fixture capture call goes to. + pub(crate) fn capture_path(provider: &str) -> String { + format!("/integrations/{provider}/api/token") + } + + /// Everything one capture call reads. + pub(crate) struct CaptureFixture { + pub(crate) settings: Settings, + pub(crate) runtime: Arc, + pub(crate) services: RuntimeServices, + pub(crate) graph: KvIdentityGraph, + pub(crate) store: Arc, + pub(crate) context: EcContext, + } + + impl CaptureFixture { + /// A real browser's proxied call with `row` stored, read in phase 1, + /// and every provider's `revoke_scope` at its default. + pub(crate) fn new( + providers: Vec>, + row: Option<&KvEntry>, + ) -> Self { + Self::with_scopes(providers, row, &[]) + } + + /// As [`Self::new`], with `revoke_scope = "global"` for each provider + /// listed in `global`. + pub(crate) fn with_scopes( + providers: Vec>, + row: Option<&KvEntry>, + global: &[&str], + ) -> Self { + let mut settings = create_test_settings(); + for id in global { + settings.ec.identity.providers.insert( + (*id).to_owned(), + ProviderSettings { + revoke_scope: Some(RevokeScope::Global), + ..ProviderSettings::default() + }, + ); + } + let runtime = Arc::new( + IdentityRuntime::from_settings_with_providers( + &settings, + providers, + BTreeMap::::new(), + ) + .expect("should build the runtime"), + ); + let client = Arc::new(FakeHttpClient::new()); + let backend = Arc::new(FakeBackend::new()); + let services = fake_services(&client, &backend); + let (graph, store) = KvIdentityGraph::write_counting("capture-store"); + if let Some(row) = row { + graph.create(TEST_EC_ID, row).expect("should seed the row"); + } + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + allowing_consent(), + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(Arc::clone(&runtime)); + context.identity_mut().set_writer_decision(WriterDecision { + writer: false, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + context.set_kv_snapshot(graph.load_snapshot(TEST_EC_ID)); + store.reset_counts(); + Self { + settings, + runtime, + services, + graph, + store, + context, + } + } + + /// Replaces the request's consent. + pub(crate) fn with_consent(mut self, consent: ConsentContext) -> Self { + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + consent, + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(Arc::clone(&self.runtime)); + context.set_kv_snapshot(self.context.kv_snapshot().clone()); + context.identity_mut().set_real_browser(true); + self.context = context; + self + } + + /// Makes the request a client that does not look like a browser. + pub(crate) fn not_a_browser(mut self) -> Self { + self.context.identity_mut().set_real_browser(false); + self + } + + /// Runs the capture for a 200 response holding `body` at `path`. + pub(crate) fn capture(&mut self, path: &str, body: &serde_json::Value) { + let bytes = serde_json::to_vec(body).expect("should serialize the body"); + self.capture_bytes(path, StatusCode::OK, &bytes); + } + + /// Runs the capture for a response with `status` holding `bytes`. + pub(crate) fn capture_bytes(&mut self, path: &str, status: StatusCode, bytes: &[u8]) { + let random = ThreadRandom; + let deps = self + .runtime + .deps(&self.settings, &self.services, &random, None); + let headers = HeaderMap::new(); + run_capture( + &deps, + Some(&self.graph), + &mut self.context, + &CaptureCall { + method: &Method::POST, + path, + status, + headers: &headers, + body: bytes, + }, + ); + } + + /// Runs the removal write's one retry after send. + pub(crate) fn retry_after_send(&mut self) { + retry_pending_removals(&mut self.context, &self.graph); + } + + /// The row as KV holds it. + pub(crate) fn row(&self) -> KvEntry { + self.graph + .get(TEST_EC_ID) + .expect("should read the row") + .expect("should hold the row") + .0 + } + } +} diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index b0eb83d3fa..c53b4e8160 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -20,12 +20,15 @@ use error_stack::{Report, ResultExt}; use crate::error::TrustedServerError; use super::generation::ec_hash; -use super::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome}; -use super::kv_types::{KvEntry, KvMetadata, KvNetwork}; +use super::kv_backend::{ + EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome, KvOperationCounts, + OperationCountingEcKv, +}; +use super::kv_types::{KvEntry, KvMetadata}; use super::{EcKvSnapshot, checked_current_timestamp, current_timestamp, log_id}; /// Maximum number of CAS retry attempts before giving up. -const MAX_CAS_RETRIES: u32 = 5; +pub(crate) const MAX_CAS_RETRIES: u32 = 5; /// Maximum number of keys to request when counting hash-prefix matches /// for cluster size evaluation. Anything above this is clearly a large @@ -77,137 +80,57 @@ pub(crate) struct PartnerIdUpdate { pub(crate) partner_id: String, /// Partner-scoped user ID value. pub(crate) uid: String, + /// Unix seconds this request saw the ID. Stored as the record's `at`. + pub(crate) acquired_at: Option, } impl PartnerIdUpdate { - /// Creates a partner UID update. + /// Creates a partner UID update with no acquisition time. pub(crate) fn new(partner_id: impl Into, uid: impl Into) -> Self { Self { partner_id: partner_id.into(), uid: uid.into(), + acquired_at: None, } } + + /// Records when this request saw the ID. + #[must_use] + pub(crate) fn with_acquired_at(mut self, at: u64) -> Self { + self.acquired_at = Some(at); + self + } } -/// Terminal result of one browser EID-cookie persistence attempt. +/// Terminal result of the browser EID-cookie persistence the `EID sync +/// measurement` line still reports: a new EC's create. #[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] pub(crate) enum EidCookieSyncOutcome { - /// The stored values already matched without a write. - #[display("already_matched")] - AlreadyMatched, - /// The one conditional write succeeded. + /// The new EC's create stored the browser ingest. #[display("written")] Written, - /// The write added missing IDs while deferring different values of unknown freshness. - #[display("written_with_deferred_freshness")] - WrittenWithDeferredFreshness, - /// A conflicting writer persisted every desired value. - #[display("conflict_matched")] - ConflictMatched, - /// A conflict left at least one desired value absent or different. - #[display("deferred_conflict")] - DeferredConflict, - /// A different stored value had unknown freshness. - #[display("deferred_freshness")] - DeferredFreshness, - /// An eventually consistent read missed a row already proven to exist. - #[display("deferred_stale_read")] - DeferredStaleRead, - /// The identity graph row was missing. - #[display("missing")] - Missing, - /// Consent had been withdrawn in the identity graph. - #[display("consent_withdrawn")] - ConsentWithdrawn, - /// KV or serialization failed. - #[display("failed")] - Failed, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum CookieUpdateApplication { - AlreadyMatched, - Changed { deferred: bool }, - DeferredFreshness, -} - -fn apply_cookie_partner_id_updates( - entry: &mut KvEntry, - updates: &[PartnerIdUpdate], -) -> CookieUpdateApplication { - let mut latest_updates = BTreeMap::new(); - for update in updates { - latest_updates.insert(update.partner_id.as_str(), update.uid.as_str()); - } - - let mut changed = false; - let mut deferred = false; - for (partner_id, uid) in latest_updates { - match entry.ids.get(partner_id) { - Some(existing) if existing.uid == uid => continue, - // Browser EID cookies carry no value-owned sequence or timestamp. - // A different value therefore has unknown freshness and must not - // replace the stored value. - Some(_) => { - deferred = true; - continue; - } - None => {} - } - - entry.ids.insert( - partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, - ); - changed = true; - } - - if changed { - CookieUpdateApplication::Changed { deferred } - } else if deferred { - CookieUpdateApplication::DeferredFreshness - } else { - CookieUpdateApplication::AlreadyMatched - } -} - -fn partner_id_updates_match(entry: &KvEntry, updates: &[PartnerIdUpdate]) -> bool { - let mut latest_updates = BTreeMap::new(); - for update in updates { - latest_updates.insert(update.partner_id.as_str(), update.uid.as_str()); - } - - latest_updates.into_iter().all(|(partner_id, uid)| { - entry - .ids - .get(partner_id) - .is_some_and(|existing| existing.uid == uid) - }) } pub(crate) fn apply_partner_id_updates(entry: &mut KvEntry, updates: &[PartnerIdUpdate]) -> bool { let mut latest_updates = BTreeMap::new(); for update in updates { - latest_updates.insert(update.partner_id.as_str(), update.uid.as_str()); + latest_updates.insert(update.partner_id.as_str(), update); } let mut changed = false; - for (partner_id, uid) in latest_updates { + for (partner_id, update) in latest_updates { + let uid = update.uid.as_str(); if entry .ids .get(partner_id) - .is_some_and(|existing| existing.uid == uid) + .is_some_and(|existing| existing.first_uid() == Some(uid)) { continue; } entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, + super::kv_types::KvPartnerId::browser(uid, update.acquired_at), ); changed = true; } @@ -255,12 +178,55 @@ impl KvIdentityGraph { } } + /// Returns a graph over the same store that also counts its operations + /// into `counts`. + #[must_use] + pub fn with_operation_counts(self, counts: &KvOperationCounts) -> Self { + Self { + store: Arc::new(OperationCountingEcKv::new(self.store, counts.clone())), + } + } + + /// Blocks the request for `duration` through the store, so an + /// after-send write can outlast the platform's per-key write limit. + pub(crate) fn pause(&self, duration: core::time::Duration) { + self.store.pause(duration); + } + /// Returns the configured store name. #[must_use] pub fn store_name(&self) -> &str { self.store.store_name() } + /// Reads any key of the store, such as a pending key, raw. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::KvStore`] on store open or read failure. + pub(crate) fn lookup_key( + &self, + key: &str, + ) -> Result, Report> { + self.store.lookup(key) + } + + /// Writes any key of the store, such as a pending key, raw, with no + /// metadata. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::KvStore`] on store open or write failure. + pub(crate) fn write_key( + &self, + key: &str, + body: &str, + ttl: Duration, + mode: EcKvWriteMode, + ) -> Result> { + self.write_entry(key, body, "", ttl, mode) + } + fn kv_error(&self, message: String) -> Report { Report::new(TrustedServerError::KvStore { store_name: self.store_name().to_owned(), @@ -357,12 +323,28 @@ impl KvIdentityGraph { ec_id: &str, body_bytes: &[u8], ) -> Result> { - let entry: KvEntry = + let mut entry: KvEntry = serde_json::from_slice(body_bytes).change_context(TrustedServerError::KvStore { store_name: store_name.to_owned(), message: format!("Failed to deserialize entry for key '{}'", log_id(ec_id)), })?; + let dropped = entry.upgrade_for_read().map_err(|message| { + Report::new(TrustedServerError::KvStore { + store_name: store_name.to_owned(), + message: format!( + "Loaded invalid entry for key '{}': {message}", + log_id(ec_id) + ), + }) + })?; + for message in dropped { + log::warn!( + "Dropping an invalid record on read for key '{}': {message}", + log_id(ec_id) + ); + } + entry.validate().map_err(|message| { Report::new(TrustedServerError::KvStore { store_name: store_name.to_owned(), @@ -555,154 +537,6 @@ impl KvIdentityGraph { ))) } - /// Persists browser EID cookies with one conditional write and one conflict read. - /// - /// Browser cookies do not carry a value-owned version, so a different - /// existing value has unknown freshness and is always deferred. After a CAS - /// conflict this method never writes again. A live follow-up becomes the - /// authoritative snapshot; a missing or failed follow-up retains the live - /// pre-write snapshot as proof and defers the update. - pub(crate) fn sync_eid_cookie_updates_from_snapshot( - &self, - ec_id: &str, - updates: &[PartnerIdUpdate], - snapshot: EcKvSnapshot, - ) -> (EcKvSnapshot, EidCookieSyncOutcome) { - if updates.is_empty() { - return (snapshot, EidCookieSyncOutcome::AlreadyMatched); - } - - let proven = match snapshot { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - .. - } if snapshot_id == ec_id => Some(snapshot.clone()), - _ => None, - }; - - let current = match snapshot { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - generation: Some(_), - .. - } if snapshot_id == ec_id => snapshot, - EcKvSnapshot::Failed { - ec_id: ref snapshot_id, - } if snapshot_id == ec_id => return (snapshot, EidCookieSyncOutcome::Failed), - _ => self.load_snapshot(ec_id), - }; - - let (mut entry, generation) = match current { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - ref entry, - generation: Some(generation), - } if snapshot_id == ec_id => (entry.as_ref().clone(), generation), - EcKvSnapshot::Missing { .. } => { - let outcome = if proven.is_some() { - EidCookieSyncOutcome::DeferredStaleRead - } else { - EidCookieSyncOutcome::Missing - }; - let kept = Self::keep_proven(ec_id, current, proven.as_ref()); - return (kept, outcome); - } - EcKvSnapshot::Failed { .. } => { - let kept = Self::keep_proven(ec_id, current, proven.as_ref()); - return (kept, EidCookieSyncOutcome::Failed); - } - EcKvSnapshot::Present { .. } | EcKvSnapshot::NotRead => { - return ( - EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }, - EidCookieSyncOutcome::Failed, - ); - } - }; - - if !entry.consent.ok { - return (current, EidCookieSyncOutcome::ConsentWithdrawn); - } - let deferred_freshness = match apply_cookie_partner_id_updates(&mut entry, updates) { - CookieUpdateApplication::AlreadyMatched => { - return (current, EidCookieSyncOutcome::AlreadyMatched); - } - CookieUpdateApplication::DeferredFreshness => { - return (current, EidCookieSyncOutcome::DeferredFreshness); - } - CookieUpdateApplication::Changed { deferred } => deferred, - }; - - let Ok((body, meta_str)) = Self::serialize_entry(&entry, self.store_name()) else { - return ( - EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }, - EidCookieSyncOutcome::Failed, - ); - }; - match self.write_entry( - ec_id, - &body, - &meta_str, - ENTRY_TTL, - EcKvWriteMode::IfGenerationMatch(generation), - ) { - Ok(EcKvWriteOutcome::Written) => ( - EcKvSnapshot::Present { - ec_id: ec_id.to_owned(), - entry: Box::new(entry), - generation: None, - }, - if deferred_freshness { - EidCookieSyncOutcome::WrittenWithDeferredFreshness - } else { - EidCookieSyncOutcome::Written - }, - ), - Ok(EcKvWriteOutcome::PreconditionFailed) => { - let refreshed = self.load_snapshot(ec_id); - let Some(refreshed_entry) = refreshed.entry_for(ec_id) else { - // The failed CAS proved `current`'s generation stale. Keep - // only its existence proof so later writes must reread. - let proof = match current { - EcKvSnapshot::Present { - ec_id: proven_id, - entry, - .. - } => EcKvSnapshot::Present { - ec_id: proven_id, - entry, - generation: None, - }, - other => other, - }; - let kept = Self::keep_proven(ec_id, refreshed, Some(&proof)); - return (kept, EidCookieSyncOutcome::DeferredConflict); - }; - if !refreshed_entry.consent.ok { - return (refreshed, EidCookieSyncOutcome::ConsentWithdrawn); - } - let outcome = if partner_id_updates_match(refreshed_entry, updates) { - EidCookieSyncOutcome::ConflictMatched - } else { - EidCookieSyncOutcome::DeferredConflict - }; - (refreshed, outcome) - } - Err(err) => { - log::warn!("EID cookie sync write failed: {err:?}"); - ( - EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }, - EidCookieSyncOutcome::Failed, - ) - } - } - } - /// Merges partner IDs using request-scoped persisted state as the first CAS input. /// /// A caller-supplied `Present` snapshot is *proof* that the row exists — @@ -720,6 +554,9 @@ impl KvIdentityGraph { /// later chance to retry. /// /// [`generate_if_needed`]: super::generate_if_needed + // No production caller since phase 4 replaced pull sync's own write; its + // tests still pin the merge rules. Remove with them in a cleanup task. + #[allow(dead_code)] pub(crate) fn upsert_partner_ids_from_snapshot( &self, ec_id: &str, @@ -837,6 +674,31 @@ impl KvIdentityGraph { } } + /// Writes `entry` only if the row is still at `generation`. + /// + /// The identity removal write and the phase 4 flush share this path, so + /// every identity write is validated and generation-checked. + /// + /// # Errors + /// + /// Returns an error when `entry` does not serialize or validate, or the + /// store write fails for a reason other than a generation mismatch. + pub(crate) fn write_if_generation( + &self, + ec_id: &str, + entry: &KvEntry, + generation: u64, + ) -> Result> { + let (body, metadata) = Self::serialize_entry(entry, self.store_name())?; + self.write_entry( + ec_id, + &body, + &metadata, + ENTRY_TTL, + EcKvWriteMode::IfGenerationMatch(generation), + ) + } + /// Returns `proven` instead of a read outcome that cannot disprove it. /// /// Partner-ID enrichment is best effort. When the caller already held proof @@ -846,6 +708,9 @@ impl KvIdentityGraph { /// update is logged. Write failures are *not* routed here: they are real /// operation failures and stay [`EcKvSnapshot::Failed`] so callers can /// report them. + // No production caller since phase 4 replaced pull sync's own write; its + // tests still pin the merge rules. Remove with them in a cleanup task. + #[allow(dead_code)] fn keep_proven( ec_id: &str, downgraded: EcKvSnapshot, @@ -913,16 +778,14 @@ impl KvIdentityGraph { if entry .ids .get(partner_id) - .is_some_and(|existing| existing.uid == uid) + .is_some_and(|existing| existing.first_uid() == Some(uid)) { return Ok(()); } entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, + super::kv_types::KvPartnerId::new(uid), ); let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; @@ -971,6 +834,28 @@ impl KvIdentityGraph { ec_id: &str, partner_id: &str, uid: &str, + ) -> Result> { + self.upsert_partner_id_if_exists_policy(ec_id, partner_id, uid, false) + } + + /// Like [`Self::upsert_partner_id_if_exists`], with the replacement rule + /// chosen by the caller. + /// + /// With `only_when_absent`, a source that already holds a value is left + /// untouched and the call returns [`UpsertResult::Unchanged`] without a + /// write. Batch sync sets it for a source an identity provider owns, so a + /// provider's entry set is never replaced. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::KvStore`] on store I/O or CAS + /// exhaustion errors. + pub fn upsert_partner_id_if_exists_policy( + &self, + ec_id: &str, + partner_id: &str, + uid: &str, + only_when_absent: bool, ) -> Result> { for attempt in 0..MAX_CAS_RETRIES { let (mut entry, generation) = match self.get(ec_id)? { @@ -982,19 +867,21 @@ impl KvIdentityGraph { return Ok(UpsertResult::ConsentWithdrawn); } + if only_when_absent && entry.ids.contains_key(partner_id) { + return Ok(UpsertResult::Unchanged); + } + if entry .ids .get(partner_id) - .is_some_and(|existing| existing.uid == uid) + .is_some_and(|existing| existing.first_uid() == Some(uid)) { return Ok(UpsertResult::Unchanged); } entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, + super::kv_types::KvPartnerId::new(uid), ); let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; @@ -1468,84 +1355,16 @@ impl KvIdentityGraph { .count_keys_with_prefix(hash_prefix, CLUSTER_LIST_LIMIT) } - /// Evaluates the cluster size for an EC entry. + /// Counts the entries sharing `ec_id`'s hash prefix, without writing. /// - /// Returns the stored `cluster_size` when it has already been evaluated - /// for a live entry. Tombstone entries return `None` without store I/O so - /// their 24-hour withdrawal TTL is not extended. Otherwise, counts the - /// number of keys sharing the same hash prefix via - /// [`count_hash_prefix_keys`](Self::count_hash_prefix_keys) and writes the - /// result back to the entry. The CAS write is best-effort — on conflict - /// or write failure, the computed value is still returned. + /// The prefix list and count only. A caller that wants the size kept + /// stages it on a write of its own. /// /// # Errors /// /// Returns [`TrustedServerError::KvStore`] on store or list failure. - pub fn evaluate_cluster( - &self, - ec_id: &str, - entry: &KvEntry, - generation: u64, - ) -> Result, Report> { - if !entry.consent.ok { - log::trace!("evaluate_cluster: skipping tombstone entry"); - return Ok(None); - } - - if let Some(cluster_size) = entry - .network - .as_ref() - .and_then(|network| network.cluster_size) - { - log::trace!("evaluate_cluster: using stored cluster_size"); - return Ok(Some(cluster_size)); - } - - // Compute cluster size via prefix list. - let hash_prefix = ec_hash(ec_id); - let cluster_size = self.count_hash_prefix_keys(hash_prefix)?; - - log::debug!( - "evaluate_cluster: computed cluster_size={cluster_size} for '{}'", - log_id(ec_id) - ); - - // Best-effort CAS write-back — update only the cluster size so any - // future `network` fields are preserved across this lazy write. - let mut updated_entry = entry.clone(); - let mut network = updated_entry - .network - .unwrap_or(KvNetwork { cluster_size: None }); - network.cluster_size = Some(cluster_size); - updated_entry.network = Some(network); - - let (body, meta_str) = Self::serialize_entry(&updated_entry, self.store_name())?; - - match self.write_entry( - ec_id, - &body, - &meta_str, - ENTRY_TTL, - EcKvWriteMode::IfGenerationMatch(generation), - ) { - Ok(EcKvWriteOutcome::Written) => {} - Ok(EcKvWriteOutcome::PreconditionFailed) => { - log::debug!( - "evaluate_cluster: CAS conflict writing cluster_size for '{}', \ - returning computed value anyway", - log_id(ec_id), - ); - } - Err(report) => { - // Log but don't fail — the computed value is still valid. - log::warn!( - "evaluate_cluster: failed to write cluster_size for '{}': {report}", - log_id(ec_id) - ); - } - } - - Ok(Some(cluster_size)) + pub fn compute_cluster_size(&self, ec_id: &str) -> Result> { + self.count_hash_prefix_keys(ec_hash(ec_id)) } /// Hard-deletes the entry and any withdrawal completion marker. @@ -1564,6 +1383,31 @@ impl KvIdentityGraph { } } +#[cfg(any(test, feature = "test-utils"))] +impl KvIdentityGraph { + /// Test helper: a graph backed by an in-memory store that counts lookups + /// and writes by mode, with the store to read the counts from. + /// + /// Counts include any seeding, so call + /// [`WriteCountingEcKv::reset_counts`](crate::ec::identity::test_support::WriteCountingEcKv::reset_counts) + /// after seeding to count only what the code under test does. + #[must_use] + pub fn write_counting( + store_name: impl Into, + ) -> ( + Self, + Arc, + ) { + let store = Arc::new(crate::ec::identity::test_support::WriteCountingEcKv::new( + store_name, + )); + let graph = Self { + store: store.clone(), + }; + (graph, store) + } +} + #[cfg(test)] impl KvIdentityGraph { /// Test helper: a graph whose every store operation fails, mimicking a @@ -1733,22 +1577,74 @@ mod tests { } #[test] - fn deserialize_entry_rejects_invalid_legacy_values() { + fn deserialize_entry_drops_an_invalid_record() { let mut entry = KvEntry::tombstone(1000); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), - }, + crate::ec::kv_types::KvPartnerId::new( + "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), + ), + ); + entry.ids.insert( + "ssp_y".to_owned(), + crate::ec::kv_types::KvPartnerId::new("valid-uid"), ); let body = serde_json::to_vec(&entry).expect("should serialize invalid entry payload"); - let err = KvIdentityGraph::deserialize_entry("test-store", "ec-id", &body) - .expect_err("should reject invalid legacy entry values"); - let err_text = format!("{err}"); + let read = KvIdentityGraph::deserialize_entry("test-store", "ec-id", &body) + .expect("should read a row whose one record is invalid"); + + assert!( + !read.ids.contains_key("ssp_x"), + "should drop the invalid record" + ); + assert_eq!( + read.ids["ssp_y"].first_uid(), + Some("valid-uid"), + "should keep the valid record" + ); + + let malformed = serde_json::json!({ + "v": 2, + "created": 1000, + "consent": { "ok": "yes", "updated": 1000 }, + "geo": { "country": "US" } + }); + let err = KvIdentityGraph::deserialize_entry( + "test-store", + "ec-id", + &serde_json::to_vec(&malformed).expect("should serialize"), + ) + .expect_err("should still fail a row-level fault as a whole"); assert!( - err_text.contains("Loaded invalid entry"), - "should report validation failure for loaded entries" + format!("{err}").contains("Failed to deserialize entry"), + "should report the row-level fault" + ); + } + + #[test] + fn deserialize_entry_reads_a_version_1_body() { + let body = serde_json::to_vec(&serde_json::json!({ + "v": 1, + "created": 1_760_000_000, + "consent": { "ok": true, "updated": 1_760_000_000 }, + "geo": { "country": "US" }, + "ids": { "id5-sync.com": { "uid": "ID5*example" } } + })) + .expect("should serialize a row from main"); + + let entry = KvIdentityGraph::deserialize_entry("test-store", "ec-id", &body) + .expect("should read a version 1 row"); + + assert_eq!( + entry.v, + crate::ec::kv_types::SCHEMA_VERSION, + "should upgrade the version" + ); + assert_eq!( + entry.ids["id5-sync.com"], + crate::ec::kv_types::KvPartnerId::new("ID5*example"), + "should read the uid as one entry" ); } @@ -1772,9 +1668,9 @@ mod tests { let mut entry = KvEntry::tombstone(1000); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), - }, + crate::ec::kv_types::KvPartnerId::new( + "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), + ), ); let err = KvIdentityGraph::serialize_entry(&entry, "test-store") @@ -1796,9 +1692,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "concurrent.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "concurrent-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("concurrent-uid"), ); entry } @@ -1943,136 +1837,11 @@ mod tests { } } - /// Store that replaces the row during the first EID CAS write and records - /// the request's reads and conditional writes. - struct EidConflictEcKv { - inner: InMemoryEcKv, - concurrent_entry: KvEntry, - lookups: std::sync::Arc, - conditional_writes: std::sync::Arc, - follow_up_miss: bool, - miss_next_lookup: std::sync::atomic::AtomicBool, - } - - impl EidConflictEcKv { - fn new( - concurrent_entry: KvEntry, - lookups: std::sync::Arc, - conditional_writes: std::sync::Arc, - ) -> Self { - Self { - inner: InMemoryEcKv::new("eid-conflict-store"), - concurrent_entry, - lookups, - conditional_writes, - follow_up_miss: false, - miss_next_lookup: std::sync::atomic::AtomicBool::new(false), - } - } - - fn with_follow_up_miss(mut self) -> Self { - self.follow_up_miss = true; - self - } - - fn seed_live(&self, ec_id: &str) { - let (body, meta) = - KvIdentityGraph::serialize_entry(&live_entry(), self.inner.store_name()) - .expect("should serialize initial entry"); - self.inner - .insert( - ec_id, - EcKvWrite { - body: &body, - metadata: &meta, - ttl: ENTRY_TTL, - mode: EcKvWriteMode::Add, - }, - ) - .expect("should seed initial entry"); - } - } - - impl EcKvStore for EidConflictEcKv { - fn store_name(&self) -> &str { - self.inner.store_name() - } - - fn lookup(&self, key: &str) -> Result, Report> { - self.lookups - .fetch_add(1, std::sync::atomic::Ordering::Relaxed); - if self - .miss_next_lookup - .swap(false, std::sync::atomic::Ordering::Relaxed) - { - return Ok(None); - } - self.inner.lookup(key) - } - - fn key_exists(&self, key: &str) -> Result> { - self.inner.key_exists(key) - } - - fn insert( - &self, - key: &str, - write: EcKvWrite<'_>, - ) -> Result> { - if matches!(write.mode, EcKvWriteMode::IfGenerationMatch(_)) { - self.conditional_writes - .fetch_add(1, std::sync::atomic::Ordering::Relaxed); - let (body, meta) = KvIdentityGraph::serialize_entry( - &self.concurrent_entry, - self.inner.store_name(), - ) - .expect("should serialize concurrent entry"); - self.inner - .insert( - key, - EcKvWrite { - body: &body, - metadata: &meta, - ttl: ENTRY_TTL, - mode: EcKvWriteMode::Overwrite, - }, - ) - .expect("should write concurrent entry"); - if self.follow_up_miss { - self.miss_next_lookup - .store(true, std::sync::atomic::Ordering::Relaxed); - } - return Ok(EcKvWriteOutcome::PreconditionFailed); - } - self.inner.insert(key, write) - } - - fn list_keys_with_prefix( - &self, - prefix: &str, - limit: u32, - ) -> Result, Report> { - self.inner.list_keys_with_prefix(prefix, limit) - } - - fn count_keys_with_prefix( - &self, - prefix: &str, - limit: u32, - ) -> Result> { - self.inner.count_keys_with_prefix(prefix, limit) - } - - fn delete(&self, key: &str) -> Result<(), Report> { - self.inner.delete(key) - } - } - - #[test] - fn create_or_revive_retries_cas_conflict_and_succeeds() { - let store = ConflictInjectingEcKv::new(2, false); - store.seed_tombstone("ec-1"); - let graph = KvIdentityGraph::new(store); + #[test] + fn create_or_revive_retries_cas_conflict_and_succeeds() { + let store = ConflictInjectingEcKv::new(2, false); + store.seed_tombstone("ec-1"); + let graph = KvIdentityGraph::new(store); graph .create_or_revive("ec-1", &live_entry()) @@ -2131,16 +1900,35 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "uid-1".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("uid-1"), ); let updates = vec![PartnerIdUpdate::new("ssp_x", "uid-1")]; let changed = apply_partner_id_updates(&mut entry, &updates); assert!(!changed, "should not change when UID already matches"); - assert_eq!(entry.ids["ssp_x"].uid, "uid-1"); + assert_eq!(entry.ids["ssp_x"].first_uid(), Some("uid-1")); + } + + #[test] + fn browser_ingest_records_acquisition_time() { + let mut entry = KvEntry::minimal("sharedid.org", "existing", 1); + let updates = + [PartnerIdUpdate::new("example.com", "new-uid").with_acquired_at(1_760_000_000)]; + + let changed = apply_partner_id_updates(&mut entry, &updates); + + assert!(changed, "should report a change"); + assert_eq!( + entry.ids["example.com"], + crate::ec::kv_types::KvPartnerId::browser("new-uid", Some(1_760_000_000)), + "should store the UID with at and no writer" + ); + assert_eq!( + entry.ids["sharedid.org"], + crate::ec::kv_types::KvPartnerId::new("existing"), + "should leave other sources alone" + ); } #[test] @@ -2151,7 +1939,7 @@ mod tests { let changed = apply_partner_id_updates(&mut entry, &updates); assert!(changed, "should report changed entry"); - assert_eq!(entry.ids["ssp_x"].uid, "uid-1"); + assert_eq!(entry.ids["ssp_x"].first_uid(), Some("uid-1")); } #[test] @@ -2159,16 +1947,14 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "old-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("old-uid"), ); let updates = vec![PartnerIdUpdate::new("ssp_x", "new-uid")]; let changed = apply_partner_id_updates(&mut entry, &updates); assert!(changed, "should report changed entry"); - assert_eq!(entry.ids["ssp_x"].uid, "new-uid"); + assert_eq!(entry.ids["ssp_x"].first_uid(), Some("new-uid")); } #[test] @@ -2182,8 +1968,8 @@ mod tests { let changed = apply_partner_id_updates(&mut entry, &updates); assert!(changed, "should report changed entry"); - assert_eq!(entry.ids["ssp_x"].uid, "uid-x"); - assert_eq!(entry.ids["ssp_y"].uid, "uid-y"); + assert_eq!(entry.ids["ssp_x"].first_uid(), Some("uid-x")); + assert_eq!(entry.ids["ssp_y"].first_uid(), Some("uid-y")); } #[test] @@ -2191,9 +1977,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "original".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("original"), ); let updates = vec![ PartnerIdUpdate::new("ssp_x", "intermediate"), @@ -2206,324 +1990,7 @@ mod tests { !changed, "should not write when the final duplicate value matches existing state" ); - assert_eq!(entry.ids["ssp_x"].uid, "original"); - } - - #[test] - fn eid_cookie_sync_conflict_matching_value_stops_after_one_write() { - let ec_id = snapshot_ec_id(); - let mut concurrent = live_entry(); - concurrent.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "desired-uid".to_owned(), - }, - ); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - concurrent, - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - let snapshot = graph.load_snapshot(&ec_id); - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - snapshot, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::ConflictMatched); - assert_eq!( - snapshot - .entry_for(&ec_id) - .and_then(|entry| entry.ids.get("ssp_x")) - .map(|id| id.uid.as_str()), - Some("desired-uid"), - "the authoritative follow-up snapshot should replace stale request state" - ); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "a conflict should perform exactly one follow-up read" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "a conflict must not trigger another conditional write" - ); - } - - #[test] - fn eid_cookie_sync_keeps_live_proof_when_conflict_follow_up_misses() { - let ec_id = snapshot_ec_id(); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - live_entry(), - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ) - .with_follow_up_miss(); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::DeferredConflict); - assert!( - snapshot.entry_for(&ec_id).is_some(), - "the live pre-write row should prevent conflict deferral from entering orphan recovery" - ); - assert_eq!( - snapshot.generation_for(&ec_id), - None, - "a failed CAS must not return its rejected generation" - ); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "the initial refresh and conflict follow-up should be the only reads" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "the conflict must remain the request's only conditional write" - ); - } - - #[test] - fn eid_cookie_sync_reports_proven_refresh_miss_as_deferred() { - let ec_id = snapshot_ec_id(); - let graph = KvIdentityGraph::in_memory("empty-store"); - let proven = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: None, - }; - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - proven, - ); - - assert!( - snapshot.entry_for(&ec_id).is_some(), - "the add-confirmed row should survive an eventually consistent miss" - ); - assert_eq!( - outcome, - EidCookieSyncOutcome::DeferredStaleRead, - "a row already proven to exist should report a deferred stale read" - ); - } - - #[test] - fn partner_id_conflict_match_uses_last_duplicate_value() { - let mut entry = live_entry(); - entry.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "latest-uid".to_owned(), - }, - ); - let updates = [ - PartnerIdUpdate::new("ssp_x", "older-uid"), - PartnerIdUpdate::new("ssp_x", "latest-uid"), - ]; - - assert!( - partner_id_updates_match(&entry, &updates), - "conflict matching should use the same last-value-wins rule as writes" - ); - } - - #[test] - fn eid_cookie_sync_conflicting_value_defers_after_one_write() { - let ec_id = snapshot_ec_id(); - let mut concurrent = live_entry(); - concurrent.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "concurrent-uid".to_owned(), - }, - ); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - concurrent, - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - let snapshot = graph.load_snapshot(&ec_id); - - let (_, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "stale-uid")], - snapshot, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::DeferredConflict); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "a conflict should perform exactly one follow-up read" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "a conflict must not trigger another conditional write" - ); - let (stored, _) = graph - .get(&ec_id) - .expect("should read concurrent row") - .expect("row should remain"); - assert_eq!(stored.ids["ssp_x"].uid, "concurrent-uid"); - } - - #[test] - fn eid_cookie_sync_preserves_concurrent_withdrawal_after_conflict() { - let ec_id = snapshot_ec_id(); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - KvEntry::tombstone(2_000), - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - let snapshot = graph.load_snapshot(&ec_id); - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - snapshot, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::ConsentWithdrawn); - assert!( - snapshot - .entry_for(&ec_id) - .is_some_and(|entry| !entry.consent.ok), - "the refreshed withdrawal tombstone must remain authoritative" - ); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "a conflict should perform exactly one follow-up read" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "a conflict must not trigger another conditional write" - ); - } - - #[test] - fn eid_cookie_sync_defers_different_values_without_value_owned_freshness() { - let ec_id = snapshot_ec_id(); - let graph = KvIdentityGraph::in_memory("freshness-store"); - let mut entry = live_entry(); - entry.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "protected-uid".to_owned(), - }, - ); - graph.create(&ec_id, &entry).expect("should seed entry"); - - let (_, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "incoming-uid")], - graph.load_snapshot(&ec_id), - ); - - assert_eq!(outcome, EidCookieSyncOutcome::DeferredFreshness); - let (stored, _) = graph - .get(&ec_id) - .expect("should read protected row") - .expect("row should remain"); - assert_eq!(stored.ids["ssp_x"].uid, "protected-uid"); - } - - #[test] - fn eid_cookie_sync_adds_missing_ids_while_deferring_different_values() { - let ec_id = snapshot_ec_id(); - let graph = KvIdentityGraph::in_memory("mixed-freshness-store"); - let mut entry = live_entry(); - entry.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "protected-uid".to_owned(), - }, - ); - graph.create(&ec_id, &entry).expect("should seed entry"); - - let (_, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[ - PartnerIdUpdate::new("ssp_x", "different-uid"), - PartnerIdUpdate::new("ssp_y", "new-uid"), - ], - graph.load_snapshot(&ec_id), - ); - - assert_eq!(outcome, EidCookieSyncOutcome::WrittenWithDeferredFreshness); - let (stored, _) = graph - .get(&ec_id) - .expect("should read updated row") - .expect("row should remain"); - assert_eq!(stored.ids["ssp_x"].uid, "protected-uid"); - assert_eq!(stored.ids["ssp_y"].uid, "new-uid"); - } - - #[test] - fn evaluate_cluster_returns_stored_value_without_store_io() { - let kv = KvIdentityGraph::failing("nonexistent_store_for_cluster_cache_test"); - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let mut entry = live_entry(); - entry.network = Some(KvNetwork { - cluster_size: Some(5), - }); - - let cluster_size = kv - .evaluate_cluster(&ec_id, &entry, 0) - .expect("should not touch store when cluster_size is already known"); - - assert_eq!( - cluster_size, - Some(5), - "should return stored cluster_size without re-listing keys" - ); - } - - #[test] - fn evaluate_cluster_skips_tombstone_without_store_io() { - let kv = KvIdentityGraph::failing("nonexistent_store_for_tombstone_cluster_test"); - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let entry = KvEntry::tombstone(1000); - - let cluster_size = kv - .evaluate_cluster(&ec_id, &entry, 0) - .expect("should not touch store for tombstone entries"); - - assert_eq!( - cluster_size, None, - "should not evaluate or write cluster_size for tombstones" - ); + assert_eq!(entry.ids["ssp_x"].first_uid(), Some("original")); } #[test] @@ -3139,7 +2606,7 @@ mod tests { .expect("should read persisted entry") .expect("should retain entry after conflict"); assert_eq!( - entry.ids.get("ssp_x").map(|id| id.uid.as_str()), + entry.ids.get("ssp_x").and_then(|id| id.first_uid()), Some("uid-1"), "should persist requested UID after retry" ); @@ -3176,7 +2643,7 @@ mod tests { .entry_for(&ec_id) .expect("should retain persisted entry"); assert_eq!( - entry.ids.get("ssp_x").map(|id| id.uid.as_str()), + entry.ids.get("ssp_x").and_then(|id| id.first_uid()), Some("uid-1") ); assert_eq!( @@ -3607,7 +3074,7 @@ mod tests { outcome .entry_for(&ec_id) .and_then(|entry| entry.ids.get("ssp_x")) - .map(|id| id.uid.as_str()), + .and_then(|id| id.first_uid()), Some("uid-1") ); assert_eq!( @@ -3695,7 +3162,7 @@ mod tests { outcome .entry_for(&ec_id) .and_then(|entry| entry.ids.get("ssp_x")) - .map(|id| id.uid.as_str()), + .and_then(|id| id.first_uid()), Some("uid-1"), "the fifth CAS attempt must still succeed after a refresh and four conflicts" ); @@ -3715,7 +3182,7 @@ mod tests { .entry_for(&ec_id) .expect("should persist re-merged entry"); assert_eq!( - entry.ids.get("ssp_x").map(|id| id.uid.as_str()), + entry.ids.get("ssp_x").and_then(|id| id.first_uid()), Some("uid-1"), "conflict must re-merge our update onto the concurrently revived row" ); @@ -3742,9 +3209,10 @@ mod tests { ); assert_eq!( - outcome - .entry_for(&ec_id) - .and_then(|entry| entry.ids.get("ssp_x").map(|id| id.uid.clone())), + outcome.entry_for(&ec_id).and_then(|entry| entry + .ids + .get("ssp_x") + .and_then(|id| id.first_uid().map(str::to_owned))), Some("uid-1".to_owned()), "a stale miss must be revalidated before the updates are dropped" ); @@ -3753,7 +3221,7 @@ mod tests { .expect("should read store") .expect("row should remain"); assert_eq!( - stored.ids.get("ssp_x").map(|id| id.uid.as_str()), + stored.ids.get("ssp_x").and_then(|id| id.first_uid()), Some("uid-1"), "the revalidated update must reach the store" ); diff --git a/crates/trusted-server-core/src/ec/kv_backend.rs b/crates/trusted-server-core/src/ec/kv_backend.rs index bfe74ca6f3..2a6d1eadbe 100644 --- a/crates/trusted-server-core/src/ec/kv_backend.rs +++ b/crates/trusted-server-core/src/ec/kv_backend.rs @@ -11,6 +11,8 @@ //! expressed through [`EcKvWriteMode`] so compare-and-swap loops stay in //! core while the platform supplies the actual precondition mechanics. +use std::sync::Arc; +use std::sync::atomic::{AtomicU32, Ordering}; use std::time::Duration; use error_stack::Report; @@ -64,13 +66,22 @@ pub enum EcKvWriteOutcome { PreconditionFailed, } +/// Attached to an [`EcKvStore::insert`] error when the platform refused the +/// write under its per-key write rate limit, so a caller that can wait may +/// try again after [`EcKvStore::pause`]. +#[derive(Debug, Clone, Copy)] +pub struct EcKvRateLimited; + /// Raw KV store primitives backing the EC identity graph. /// /// Implementations map these operations onto the platform KV API. /// Infrastructure failures are reported as [`TrustedServerError::KvStore`]; /// write precondition failures are part of the normal control flow and are /// returned as [`EcKvWriteOutcome::PreconditionFailed`] instead of errors. -pub trait EcKvStore { +/// +/// `Send + Sync` so a [`super::kv::KvIdentityGraph`] can be built once and +/// held in shared application state. +pub trait EcKvStore: Send + Sync { /// Returns the platform store name, used in log and error messages. fn store_name(&self) -> &str; @@ -115,8 +126,9 @@ pub trait EcKvStore { /// # Errors /// /// Returns [`TrustedServerError::KvStore`] on store open or write - /// failure. Precondition failures are reported through the - /// [`EcKvWriteOutcome`] instead. + /// failure, with an [`EcKvRateLimited`] attachment when the platform's + /// per-key write limit refused it. Precondition failures are reported + /// through the [`EcKvWriteOutcome`] instead. fn insert( &self, key: &str, @@ -169,13 +181,214 @@ pub trait EcKvStore { /// /// Returns [`TrustedServerError::KvStore`] on store open or delete failure. fn delete(&self, key: &str) -> Result<(), Report>; + + /// Blocks the request for `duration`, so a write made after the response + /// was sent can outlast the platform's per-key write limit. + /// + /// The default returns at once, for platforms and test doubles that have + /// no such limit. + fn pause(&self, duration: Duration) { + let _ = duration; + } +} + +/// Counts of KV operations one request makes, shared by every clone. +/// +/// Reads are row lookups and writes are inserts and deletes. Existence +/// checks and key listings are not counted: they read no row. A graph reports into it through +/// [`KvIdentityGraph::with_operation_counts`](super::kv::KvIdentityGraph::with_operation_counts). +#[derive(Debug, Clone, Default)] +pub struct KvOperationCounts { + reads: Arc, + writes: Arc, +} + +impl KvOperationCounts { + /// Reads counted so far. + #[must_use] + pub fn reads(&self) -> u32 { + self.reads.load(Ordering::Relaxed) + } + + /// Writes counted so far. + #[must_use] + pub fn writes(&self) -> u32 { + self.writes.load(Ordering::Relaxed) + } + + fn record_read(&self) { + self.reads.fetch_add(1, Ordering::Relaxed); + } + + fn record_write(&self) { + self.writes.fetch_add(1, Ordering::Relaxed); + } +} + +/// An [`EcKvStore`] that counts every operation it forwards. +pub(crate) struct OperationCountingEcKv { + inner: Arc, + counts: KvOperationCounts, +} + +impl OperationCountingEcKv { + /// Wraps `inner`, reporting into `counts`. + pub(crate) fn new(inner: Arc, counts: KvOperationCounts) -> Self { + Self { inner, counts } + } +} + +impl EcKvStore for OperationCountingEcKv { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup(&self, key: &str) -> Result, Report> { + self.counts.record_read(); + self.inner.lookup(key) + } + + fn key_exists(&self, key: &str) -> Result> { + self.inner.key_exists(key) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + self.counts.record_write(); + self.inner.insert(key, write) + } + + fn list_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result, Report> { + self.inner.list_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + self.counts.record_write(); + self.inner.delete(key) + } + + fn pause(&self, duration: Duration) { + self.inner.pause(duration); + } +} + +/// An in-memory store for tests, reachable from other crates through the +/// `test-utils` feature. +#[cfg(any(test, feature = "test-utils"))] +pub mod in_memory { + use std::collections::BTreeMap; + use std::sync::Mutex; + + use super::*; + + /// In-memory [`EcKvStore`] with generation tracking for CAS tests. + pub struct InMemoryEcKv { + name: String, + entries: Mutex>, + } + + struct StoredEntry { + body: Vec, + metadata: Option>, + generation: u64, + } + + impl InMemoryEcKv { + /// An empty store named `name`. + pub fn new(name: impl Into) -> Self { + Self { + name: name.into(), + entries: Mutex::new(BTreeMap::new()), + } + } + } + + impl EcKvStore for InMemoryEcKv { + fn store_name(&self) -> &str { + &self.name + } + + fn lookup(&self, key: &str) -> Result, Report> { + let entries = self.entries.lock().expect("should lock in-memory store"); + Ok(entries.get(key).map(|stored| EcKvLookup { + body: stored.body.clone(), + metadata: stored.metadata.clone(), + generation: stored.generation, + })) + } + + fn key_exists(&self, key: &str) -> Result> { + let entries = self.entries.lock().expect("should lock in-memory store"); + Ok(entries.contains_key(key)) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + let mut entries = self.entries.lock().expect("should lock in-memory store"); + let existing_generation = entries.get(key).map(|stored| stored.generation); + + match write.mode { + EcKvWriteMode::Add if existing_generation.is_some() => { + return Ok(EcKvWriteOutcome::PreconditionFailed); + } + EcKvWriteMode::IfGenerationMatch(expected) + if existing_generation != Some(expected) => + { + return Ok(EcKvWriteOutcome::PreconditionFailed); + } + EcKvWriteMode::Add + | EcKvWriteMode::Overwrite + | EcKvWriteMode::IfGenerationMatch(_) => {} + } + + entries.insert( + key.to_owned(), + StoredEntry { + body: write.body.as_bytes().to_vec(), + metadata: Some(write.metadata.as_bytes().to_vec()), + generation: existing_generation.unwrap_or(0) + 1, + }, + ); + Ok(EcKvWriteOutcome::Written) + } + + fn list_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result, Report> { + let entries = self.entries.lock().expect("should lock in-memory store"); + Ok(entries + .keys() + .filter(|key| key.starts_with(prefix)) + .take(limit as usize) + .cloned() + .collect()) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + let mut entries = self.entries.lock().expect("should lock in-memory store"); + entries.remove(key); + Ok(()) + } + } } #[cfg(test)] pub(crate) mod test_support { - use std::collections::BTreeMap; use std::sync::Mutex; + pub(crate) use super::in_memory::InMemoryEcKv; use super::*; /// [`EcKvStore`] wrapper that counts `lookup` calls through a shared counter @@ -312,100 +525,6 @@ pub(crate) mod test_support { } } - /// In-memory [`EcKvStore`] with generation tracking for CAS tests. - pub(crate) struct InMemoryEcKv { - name: String, - entries: Mutex>, - } - - struct StoredEntry { - body: Vec, - metadata: Option>, - generation: u64, - } - - impl InMemoryEcKv { - pub(crate) fn new(name: impl Into) -> Self { - Self { - name: name.into(), - entries: Mutex::new(BTreeMap::new()), - } - } - } - - impl EcKvStore for InMemoryEcKv { - fn store_name(&self) -> &str { - &self.name - } - - fn lookup(&self, key: &str) -> Result, Report> { - let entries = self.entries.lock().expect("should lock in-memory store"); - Ok(entries.get(key).map(|stored| EcKvLookup { - body: stored.body.clone(), - metadata: stored.metadata.clone(), - generation: stored.generation, - })) - } - - fn key_exists(&self, key: &str) -> Result> { - let entries = self.entries.lock().expect("should lock in-memory store"); - Ok(entries.contains_key(key)) - } - - fn insert( - &self, - key: &str, - write: EcKvWrite<'_>, - ) -> Result> { - let mut entries = self.entries.lock().expect("should lock in-memory store"); - let existing_generation = entries.get(key).map(|stored| stored.generation); - - match write.mode { - EcKvWriteMode::Add if existing_generation.is_some() => { - return Ok(EcKvWriteOutcome::PreconditionFailed); - } - EcKvWriteMode::IfGenerationMatch(expected) - if existing_generation != Some(expected) => - { - return Ok(EcKvWriteOutcome::PreconditionFailed); - } - EcKvWriteMode::Add - | EcKvWriteMode::Overwrite - | EcKvWriteMode::IfGenerationMatch(_) => {} - } - - entries.insert( - key.to_owned(), - StoredEntry { - body: write.body.as_bytes().to_vec(), - metadata: Some(write.metadata.as_bytes().to_vec()), - generation: existing_generation.unwrap_or(0) + 1, - }, - ); - Ok(EcKvWriteOutcome::Written) - } - - fn list_keys_with_prefix( - &self, - prefix: &str, - limit: u32, - ) -> Result, Report> { - let entries = self.entries.lock().expect("should lock in-memory store"); - Ok(entries - .keys() - .filter(|key| key.starts_with(prefix)) - .take(limit as usize) - .cloned() - .collect()) - } - - fn delete(&self, key: &str) -> Result<(), Report> { - let mut entries = self.entries.lock().expect("should lock in-memory store"); - entries.remove(key); - Ok(()) - } - } - /// [`EcKvStore`] that fails every operation, mimicking a missing or /// unreachable platform store. pub(crate) struct FailingEcKv { diff --git a/crates/trusted-server-core/src/ec/kv_types.rs b/crates/trusted-server-core/src/ec/kv_types.rs index 589f280c11..86b2766a89 100644 --- a/crates/trusted-server-core/src/ec/kv_types.rs +++ b/crates/trusted-server-core/src/ec/kv_types.rs @@ -4,7 +4,8 @@ //! EC identity graph. Each EC ID (`{64hex}.{6alnum}`) maps to a [`KvEntry`] //! containing consent state, geo location, and accumulated partner IDs. //! -//! The schema is versioned (`v: 1`) to allow future migrations. +//! The schema is versioned (`v: 2`). Version 1 rows, which hold one `uid` per +//! source, still read and are written back as version 2. use std::collections::{BTreeMap, BTreeSet}; @@ -14,10 +15,11 @@ use crate::consent::ConsentContext; use crate::geo::GeoInfo; /// Current schema version for KV entries. -pub const SCHEMA_VERSION: u8 = 1; +pub const SCHEMA_VERSION: u8 = 2; -// Unsupported schema versions fail closed on read. Future schema bumps must -// add an explicit lazy migration or backfill path before changing this value. +// Versions 1 and 2 read (see [`KvEntry::upgrade_for_read`]). Any other +// version fails closed on read. Future schema bumps must add an explicit lazy +// migration or backfill path before changing this value. /// Maximum number of publisher-domain entries accepted in /// [`KvPubProperties::seen_domains`]. @@ -34,11 +36,41 @@ pub const MAX_STORED_DOMAIN_LENGTH: usize = 255; /// validation. pub const MAX_UID_LENGTH: usize = 512; +/// Maximum serialized bytes of a stored UID-level or EID-level `ext`. +pub const MAX_PARTNER_EXT_BYTES: usize = 2048; +/// Maximum bytes of a stored `inserter` or `matcher`. +pub const MAX_EID_FIELD_BYTES: usize = 255; +/// Maximum number of EID entries stored for one source. +pub const MAX_ENTRIES_PER_SOURCE: usize = 8; +/// Maximum number of UIDs stored for one source, across its entries. +pub const MAX_UIDS_PER_SOURCE: usize = 32; +/// Maximum bytes of one source's stored entry set, as measured by +/// [`entry_set_bytes`]. +pub const MAX_ENTRY_SET_BYTES: usize = 4096; +/// Maximum serialized bytes of one provider's private state (`ps.st`). +pub const MAX_PROVIDER_STATE_BYTES: usize = 1024; +/// Maximum number of providers with a `ps` entry in one row. +pub const MAX_PROVIDER_STATES: usize = 32; +/// Maximum number of UID hashes kept in each of a provider's `ps.iv` and +/// `ps.ov` lists, enough for four whole 32-UID sets. +pub const MAX_INVALIDATED_UIDS: usize = 128; +/// Length of a record revision (`rv`) in lowercase hex digits. +pub const RECORD_REVISION_HEX_LEN: usize = 16; +/// Maximum length of an opt-out fingerprint (`ps.o.fp`) in lowercase hex +/// digits. +const OPT_OUT_FINGERPRINT_MAX_HEX_LEN: usize = 16; + /// Full KV entry stored as the body of an EC identity graph record. /// /// **KV key:** Full EC ID (`{64hex}.{6alnum}`). -/// **KV value:** JSON-serialized `KvEntry` (max ~5KB). -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +/// **KV value:** JSON-serialized `KvEntry`. This is a documented estimate, not +/// an enforced limit (spec 6.1): a one-UID record is about 120 bytes, and a +/// record at every cap is about 4.2 KB. With every cap reached, eight owned +/// sources come to about 34 KB, plus 1 KB for each provider with state, and +/// `ps.iv` and `ps.ov` add up to 2.4 KB each per provider. A row with two full +/// sets and two stateful providers at every cap is about 15 KB, and typical +/// sets are far smaller. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct KvEntry { /// Schema version — always [`SCHEMA_VERSION`]. pub v: u8, @@ -62,10 +94,22 @@ pub struct KvEntry { /// Populated by pixel sync, batch sync, and pull sync operations. #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] pub ids: BTreeMap, + /// Per-provider scheduling, opt-out, removal and private state, keyed by + /// identity provider ID. Never sent to bidders. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub ps: BTreeMap, + /// The newest pending-key generation merged into this row (spec 5.3, + /// Pending key). Items at or below it are never merged again. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub pg: Option, + /// Unix seconds of the row's last writer flush, which paces the SPA + /// writer (spec 5.1). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub wa: Option, } /// Consent state within a KV entry. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct KvConsent { /// Raw TCF v2 consent string. #[serde(default, skip_serializing_if = "Option::is_none")] @@ -80,7 +124,7 @@ pub struct KvConsent { } /// Geo location within a KV entry. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct KvGeo { /// ISO 3166-1 alpha-2 country code (e.g. `"US"`). pub country: String, @@ -97,11 +141,225 @@ pub struct KvGeo { pub dma: Option, } -/// A partner user ID within a KV entry. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +/// A source's record within a KV entry (spec 6.1). +/// +/// Every record stores its source's entry set in `eids`. A version 1 record, +/// which holds one `uid`, reads as one entry holding that UID. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +#[serde(from = "KvPartnerIdWire")] pub struct KvPartnerId { - /// The partner's user identifier. - pub uid: String, + /// Writer: an identity provider ID. Absent for browser ingest and + /// version 1 records. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub w: Option, + /// Record revision: 16 lowercase hex digits, random on every write. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rv: Option, + /// Unix seconds when the ID was acquired or last refreshed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub at: Option, + /// Unix seconds at or after which the ID must not be sent. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub exp: Option, + /// Unix seconds after which the ID is due for refresh. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ra: Option, + /// Unix seconds after which the ID can no longer be refreshed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ru: Option, + /// The source's entry set, in order. Never empty in a valid record. + pub eids: Vec, +} + +/// The stored shape of [`KvPartnerId`], which also reads version 1's `uid`. +#[derive(Deserialize)] +struct KvPartnerIdWire { + #[serde(default)] + uid: Option, + #[serde(default)] + w: Option, + #[serde(default)] + rv: Option, + #[serde(default)] + at: Option, + #[serde(default)] + exp: Option, + #[serde(default)] + ra: Option, + #[serde(default)] + ru: Option, + #[serde(default)] + eids: Vec, +} + +impl From for KvPartnerId { + fn from(wire: KvPartnerIdWire) -> Self { + let eids = match (wire.eids.is_empty(), wire.uid) { + (true, Some(uid)) => vec![KvEid::one_uid(uid)], + (_, _) => wire.eids, + }; + Self { + w: wire.w, + rv: wire.rv, + at: wire.at, + exp: wire.exp, + ra: wire.ra, + ru: wire.ru, + eids, + } + } +} + +impl KvPartnerId { + /// A record holding one entry with one UID, and no metadata. + #[must_use] + pub fn new(uid: impl Into) -> Self { + Self { + eids: vec![KvEid::one_uid(uid.into())], + ..Self::default() + } + } + + /// A browser-ingested record: no writer, with its acquisition time. + #[must_use] + pub fn browser(uid: impl Into, acquired_at: Option) -> Self { + Self { + at: acquired_at, + ..Self::new(uid) + } + } + + /// The first UID of the first entry, which `/identify` returns and + /// today's one-UID readers use. + #[must_use] + pub fn first_uid(&self) -> Option<&str> { + self.eids + .first() + .and_then(|entry| entry.uids.first()) + .map(|uid| uid.id.as_str()) + } +} + +/// One EID entry of a record (spec 6.1). +/// +/// The `OpenRTB` EID object without `source`, which the `ids` key holds. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct KvEid { + /// The entry's UIDs, in order. Never empty in a valid record. + pub uids: Vec, + /// `eid.inserter`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub inserter: Option, + /// `eid.matcher`. Absent means `source`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub matcher: Option, + /// `eid.mm`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub mm: Option, + /// EID-level `ext`, at most [`MAX_PARTNER_EXT_BYTES`] serialized bytes. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ext: Option, +} + +impl KvEid { + /// One entry holding one UID and nothing else: the shape browser + /// ingest, generic pull and batch sync write. + fn one_uid(id: String) -> Self { + Self { + uids: vec![KvUid { + id, + atype: None, + ext: None, + }], + ..Self::default() + } + } +} + +/// One UID of a stored entry. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct KvUid { + /// The user identifier. + pub id: String, + /// `uid.atype`. Absent: use the partner's `openrtb_atype`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub atype: Option, + /// UID-level `ext`, at most [`MAX_PARTNER_EXT_BYTES`] serialized bytes. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ext: Option, +} + +/// A provider's sticky opt-out (spec 8). +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct KvOptOut { + /// Unix seconds when the opt-out was reported. + pub at: u64, + /// The consent bits the provider's policy read then, as a lowercase hex + /// bitmask of at most 16 digits. Plain, not keyed (spec 8). + pub fp: String, +} + +/// Per-provider state in a KV row (`ps.`). +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct KvProviderState { + /// Unix seconds before which the provider must not be called. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub n: Option, + /// Consecutive failures, for backoff. + #[serde(default, skip_serializing_if = "is_zero")] + pub f: u32, + /// Sticky opt-out. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub o: Option, + /// When the provider's opt-out was last cleared. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oc: Option, + /// Unkeyed hashes of UIDs the owner invalidated, oldest first, at most + /// [`MAX_INVALIDATED_UIDS`]. Never emptied (spec 5.4). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub iv: Vec, + /// Unkeyed hashes of UIDs the provider's opt-out removed, oldest first, + /// at most [`MAX_INVALIDATED_UIDS`]. Emptied when the opt-out clears. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub ov: Vec, + /// The provider's private state as embedded JSON. KV only. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub st: Option, +} + +fn is_zero(value: &u32) -> bool { + *value == 0 +} + +/// Bytes of a stored entry set as compact JSON: the array of entries without +/// `source`, as `eids` stores it. Both `validate` and the coordinator measure +/// [`MAX_ENTRY_SET_BYTES`] with this function. +pub(crate) fn entry_set_bytes(set: &[KvEid]) -> usize { + serde_json::to_vec(set).map_or(usize::MAX, |bytes| bytes.len()) +} + +/// Whether `value` is non-empty lowercase hex of at most `max_len` digits. +/// +/// Shared by the `rv` and `ps.o.fp` checks. +pub(crate) fn is_lowercase_hex(value: &str, max_len: usize) -> bool { + !value.is_empty() + && value.len() <= max_len + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +/// Whether an `inserter` or `matcher` is stored-valid: not blank and at most +/// [`MAX_EID_FIELD_BYTES`] bytes. +pub(crate) fn is_valid_eid_field(value: &str) -> bool { + !value.trim().is_empty() && value.len() <= MAX_EID_FIELD_BYTES +} + +/// Whether a UID-level or EID-level `ext` is stored-valid: a JSON object of at +/// most [`MAX_PARTNER_EXT_BYTES`] bytes as compact JSON. +pub(crate) fn is_valid_partner_ext(ext: &serde_json::Value) -> bool { + ext.is_object() + && serde_json::to_vec(ext).is_ok_and(|bytes| bytes.len() <= MAX_PARTNER_EXT_BYTES) } /// Publisher property metadata captured when an EC entry is created. @@ -293,6 +551,9 @@ impl KvEntry { device: None, network: None, ids: BTreeMap::new(), + ps: BTreeMap::new(), + pg: None, + wa: None, } } @@ -303,12 +564,7 @@ impl KvEntry { #[must_use] pub fn minimal(partner_id: &str, uid: &str, now: u64) -> Self { let mut ids = BTreeMap::new(); - ids.insert( - partner_id.to_owned(), - KvPartnerId { - uid: uid.to_owned(), - }, - ); + ids.insert(partner_id.to_owned(), KvPartnerId::new(uid)); Self { v: SCHEMA_VERSION, created: now, @@ -328,6 +584,9 @@ impl KvEntry { device: None, network: None, ids, + ps: BTreeMap::new(), + pg: None, + wa: None, } } @@ -360,6 +619,9 @@ impl KvEntry { device: None, network: None, ids: BTreeMap::new(), + ps: BTreeMap::new(), + pg: None, + wa: None, } } @@ -380,11 +642,35 @@ impl KvEntry { )); } - for (partner_id, partner_uid) in &self.ids { - if partner_uid.uid.len() > MAX_UID_LENGTH { + for (partner_id, record) in &self.ids { + validate_record(partner_id, record)?; + } + + if self.ps.len() > MAX_PROVIDER_STATES { + return Err(format!( + "ps exceeds MAX_PROVIDER_STATES ({})", + self.ps.len() + )); + } + for (provider, state) in &self.ps { + for (name, hashes) in [("iv", &state.iv), ("ov", &state.ov)] { + if hashes.len() > MAX_INVALIDATED_UIDS { + return Err(format!( + "ps '{provider}' {name} exceeds {MAX_INVALIDATED_UIDS} hashes" + )); + } + } + if let Some(opt_out) = &state.o + && !is_lowercase_hex(&opt_out.fp, OPT_OUT_FINGERPRINT_MAX_HEX_LEN) + { + return Err(format!("ps '{provider}' o.fp is not a hex bitmask")); + } + if let Some(st) = &state.st + && serde_json::to_vec(st) + .map_or(true, |bytes| bytes.len() > MAX_PROVIDER_STATE_BYTES) + { return Err(format!( - "partner ID '{partner_id}' exceeds MAX_UID_LENGTH ({})", - partner_uid.uid.len() + "ps '{provider}' st exceeds {MAX_PROVIDER_STATE_BYTES} bytes" )); } } @@ -417,6 +703,117 @@ impl KvEntry { Ok(()) } + + /// Brings a row read from KV to the current schema (spec 6.1). + /// + /// Accepts schema versions 1 and 2 and sets the version to 2. Drops each + /// record that breaks a rule of spec 4.5 or 6.1, and returns one message + /// per dropped record, naming the source and the rule, never a UID. + /// Called on read only, before [`Self::validate`], so one bad record + /// never makes the row unreadable. + /// + /// # Errors + /// + /// Returns an error naming the version when the row's version is + /// neither 1 nor 2. + pub(crate) fn upgrade_for_read(&mut self) -> Result, String> { + if !(1..=SCHEMA_VERSION).contains(&self.v) { + return Err(format!( + "unsupported KV entry schema version {} (expected 1 or {})", + self.v, SCHEMA_VERSION + )); + } + self.v = SCHEMA_VERSION; + let mut messages = Vec::new(); + self.ids.retain( + |partner_id, record| match validate_record(partner_id, record) { + Ok(()) => true, + Err(message) => { + messages.push(message); + false + } + }, + ); + Ok(messages) + } +} + +/// Checks one record against the limits of spec 4.5 and 6.1. +/// +/// Messages name the partner ID and the rule and never a UID, so they are safe +/// to log. +fn validate_record(partner_id: &str, record: &KvPartnerId) -> Result<(), String> { + if record.eids.is_empty() { + return Err(format!("partner ID '{partner_id}' has an empty eids")); + } + if let Some(rv) = &record.rv + && (rv.len() != RECORD_REVISION_HEX_LEN || !is_lowercase_hex(rv, RECORD_REVISION_HEX_LEN)) + { + return Err(format!( + "partner ID '{partner_id}' rv is not {RECORD_REVISION_HEX_LEN} hex digits" + )); + } + if record.eids.len() > MAX_ENTRIES_PER_SOURCE { + return Err(format!( + "partner ID '{partner_id}' has more than {MAX_ENTRIES_PER_SOURCE} entries" + )); + } + let mut uid_count = 0; + for entry in &record.eids { + if entry.uids.is_empty() { + return Err(format!( + "partner ID '{partner_id}' has an entry without a uid" + )); + } + for (name, value) in [("inserter", &entry.inserter), ("matcher", &entry.matcher)] { + if value + .as_deref() + .is_some_and(|value| !is_valid_eid_field(value)) + { + return Err(format!( + "partner ID '{partner_id}' {name} is blank or over {MAX_EID_FIELD_BYTES} bytes" + )); + } + } + if entry.mm.is_some_and(|mm| mm < 0) { + return Err(format!("partner ID '{partner_id}' mm is negative")); + } + validate_ext(partner_id, entry.ext.as_ref())?; + for uid in &entry.uids { + if uid.id.trim().is_empty() || uid.id.len() > MAX_UID_LENGTH { + return Err(format!( + "partner ID '{partner_id}' has a blank uid or one over MAX_UID_LENGTH ({MAX_UID_LENGTH})" + )); + } + if uid.atype.is_some_and(|atype| atype < 0) { + return Err(format!("partner ID '{partner_id}' atype is negative")); + } + validate_ext(partner_id, uid.ext.as_ref())?; + } + uid_count += entry.uids.len(); + } + if uid_count > MAX_UIDS_PER_SOURCE { + return Err(format!( + "partner ID '{partner_id}' has more than {MAX_UIDS_PER_SOURCE} uids" + )); + } + if entry_set_bytes(&record.eids) > MAX_ENTRY_SET_BYTES { + return Err(format!( + "partner ID '{partner_id}' entry set exceeds {MAX_ENTRY_SET_BYTES} bytes" + )); + } + Ok(()) +} + +/// Checks an optional UID-level or EID-level `ext` with +/// [`is_valid_partner_ext`]. +fn validate_ext(partner_id: &str, ext: Option<&serde_json::Value>) -> Result<(), String> { + if ext.is_some_and(|ext| !is_valid_partner_ext(ext)) { + return Err(format!( + "partner ID '{partner_id}' ext is not an object of at most {MAX_PARTNER_EXT_BYTES} bytes" + )); + } + Ok(()) } impl KvMetadata { @@ -463,6 +860,7 @@ impl KvGeo { #[cfg(test)] mod tests { use super::*; + use serde_json::json; fn sample_consent_context() -> ConsentContext { ConsentContext { @@ -490,12 +888,9 @@ mod tests { let geo = sample_geo_info(); let consent = sample_consent_context(); let mut entry = KvEntry::new(&consent, Some(&geo), 1_741_824_000, "example.com"); - entry.ids.insert( - "liveramp".to_owned(), - KvPartnerId { - uid: "LR_xyz".to_owned(), - }, - ); + entry + .ids + .insert("liveramp".to_owned(), KvPartnerId::new("LR_xyz")); let json = serde_json::to_string(&entry).expect("should serialize KvEntry"); let deserialized: KvEntry = @@ -514,7 +909,7 @@ mod tests { assert_eq!(deserialized.geo.asn, Some(7922)); assert_eq!(deserialized.geo.dma, Some(807)); assert_eq!( - deserialized.ids.get("liveramp").map(|p| p.uid.as_str()), + deserialized.ids.get("liveramp").and_then(|p| p.first_uid()), Some("LR_xyz"), ); assert!( @@ -560,7 +955,7 @@ mod tests { entry .ids .get("liveramp") - .map(|partner| partner.uid.as_str()), + .and_then(|partner| partner.first_uid()), Some("abc") ); assert!( @@ -737,9 +1132,7 @@ mod tests { let mut entry = KvEntry::tombstone(1000); entry.ids.insert( "ssp_x".to_owned(), - KvPartnerId { - uid: "x".repeat(MAX_UID_LENGTH + 1), - }, + KvPartnerId::new("x".repeat(MAX_UID_LENGTH + 1)), ); let err = entry @@ -853,7 +1246,7 @@ mod tests { ); assert_eq!(entry.ids.len(), 1); let partner = entry.ids.get("ssp_x").expect("should have ssp_x entry"); - assert_eq!(partner.uid, "abc123"); + assert_eq!(partner.first_uid(), Some("abc123")); } #[test] @@ -1206,4 +1599,405 @@ mod tests { "known_browser should default to None" ); } + + fn kv_uid(id: &str, atype: Option) -> KvUid { + KvUid { + id: id.to_owned(), + atype, + ext: None, + } + } + + fn kv_eid(uids: Vec) -> KvEid { + KvEid { + uids, + inserter: None, + matcher: None, + mm: None, + ext: None, + } + } + + /// The set of spec 6.1's `example-ids.example.com` record. + fn example_full_set() -> Vec { + vec![ + KvEid { + uids: vec![ + kv_uid("example-a", Some(1)), + KvUid { + id: "example-b".to_owned(), + atype: Some(3), + ext: Some(json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("example.com".to_owned()), + matcher: None, + mm: Some(3), + ext: None, + }, + KvEid { + uids: vec![kv_uid("example-c", Some(1))], + inserter: None, + matcher: Some("partner.example.com".to_owned()), + mm: None, + ext: None, + }, + ] + } + + fn with_set(set: Vec) -> KvEntry { + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ids.get_mut("example.com").expect("should exist").eids = set; + entry + } + + #[test] + fn golden_version_1_row_reads_as_one_entry_records_and_writes_version_2() { + let main_row = json!({ + "v": 1, + "created": 1_760_000_000, + "consent": { "ok": true, "updated": 1_760_000_000 }, + "geo": { "country": "US" }, + "ids": { + "id5-sync.com": { "uid": "ID5*example" }, + "sharedid.org": { "uid": "example-shared-id" } + } + }); + + let mut entry: KvEntry = + serde_json::from_value(main_row).expect("should read a row from main"); + let messages = entry.upgrade_for_read().expect("should accept version 1"); + entry.validate().expect("should be valid once upgraded"); + + assert!( + messages.is_empty(), + "should drop nothing from a valid version 1 row" + ); + assert_eq!( + entry.ids["id5-sync.com"], + KvPartnerId::new("ID5*example"), + "should read uid as one entry" + ); + assert_eq!( + serde_json::to_value(&entry).expect("should write"), + json!({ + "v": 2, + "created": 1_760_000_000, + "consent": { "ok": true, "updated": 1_760_000_000 }, + "geo": { "country": "US" }, + "ids": { + "id5-sync.com": { "eids": [{ "uids": [{ "id": "ID5*example" }] }] }, + "sharedid.org": { "eids": [{ "uids": [{ "id": "example-shared-id" }] }] } + } + }), + "should write version 2, with no uid key and no ps" + ); + } + + #[test] + fn version_2_row_and_provider_state_round_trip() { + let stored = json!({ + "v": 2, + "created": 1_760_000_000, + "consent": { "ok": true, "updated": 1_760_000_000 }, + "geo": { "country": "US" }, + "ids": { + "id5-sync.com": { + "w": "id5", + "rv": "9c1d04b7e6a83f2a", + "at": 1_760_000_000, + "exp": 1_767_776_000, + "ra": 1_760_086_400, + "ru": 1_770_000_000, + "eids": [{ "uids": [{ "id": "ID5*example", "atype": 1, "ext": { "linkType": 2 } }] }] + }, + "example-ids.example.com": { + "w": "exampleid", + "rv": "4be71f0c2d9a6e35", + "at": 1_760_000_000, + "eids": [ + { + "uids": [ + { "id": "example-a", "atype": 1 }, + { "id": "example-b", "atype": 3, "ext": { "stype": "ppuid" } } + ], + "inserter": "example.com", + "mm": 3 + }, + { + "uids": [{ "id": "example-c", "atype": 1 }], + "matcher": "partner.example.com" + } + ] + } + }, + "ps": { + "id5": { + "n": 1_760_003_600, + "f": 1, + "o": { "at": 1_759_000_000, "fp": "1a" }, + "oc": 1_759_500_000, + "iv": ["aaaaaaaaaaaaaaaa"], + "ov": ["bbbbbbbbbbbbbbbb"], + "st": { "signature": "example-opaque-state" } + } + } + }); + + let entry: KvEntry = serde_json::from_value(stored.clone()).expect("should read"); + entry.validate().expect("should be valid"); + assert_eq!( + serde_json::to_value(&entry).expect("should write"), + stored, + "should round-trip exactly" + ); + assert_eq!( + entry.ids["example-ids.example.com"].eids, + example_full_set(), + "should read the set whole and in order" + ); + } + + #[test] + fn a_record_reads_its_eids_and_its_version_1_uid_only_without_them() { + let both: KvPartnerId = serde_json::from_value(json!({ + "uid": "stale-copy", + "eids": [{ "uids": [{ "id": "example-a" }] }] + })) + .expect("should read a record with both keys"); + assert_eq!( + both.first_uid(), + Some("example-a"), + "should take eids over a uid copy" + ); + + let neither: KvPartnerId = + serde_json::from_value(json!({ "w": "exampleid" })).expect("should read"); + assert!( + neither.eids.is_empty(), + "should read no entry from a record with neither key" + ); + assert_eq!(neither.first_uid(), None, "should have no first UID"); + with_set(Vec::new()) + .validate() + .expect_err("should reject a record with an empty eids on write"); + } + + #[test] + fn upgrade_for_read_drops_bad_records_and_names_the_source_only() { + let mut set = example_full_set(); + set[1].matcher = Some(" ".to_owned()); + let mut entry = with_set(set); + entry.ids.insert( + "sharedid.org".to_owned(), + KvPartnerId::new("example-shared-id"), + ); + entry.v = 1; + + let messages = entry.upgrade_for_read().expect("should accept the row"); + + assert_eq!(entry.v, SCHEMA_VERSION, "should upgrade the version"); + assert_eq!(messages.len(), 1, "should report one dropped record"); + assert!( + messages[0].contains("example.com"), + "should name the source" + ); + assert!( + !messages[0].contains("example-a"), + "should never name a UID" + ); + assert!( + !entry.ids.contains_key("example.com"), + "should drop the bad record" + ); + assert_eq!( + entry.ids["sharedid.org"], + KvPartnerId::new("example-shared-id"), + "should leave a valid record alone" + ); + entry.validate().expect("should validate once upgraded"); + + let mut future = KvEntry::minimal("example.com", "example-uid", 1); + future.v = SCHEMA_VERSION + 1; + future + .upgrade_for_read() + .expect_err("should reject a version it does not know"); + } + + #[test] + fn entry_set_bytes_measures_the_stored_array() { + let set = example_full_set(); + let bytes = serde_json::to_vec(&set).expect("should serialize the set"); + + assert_eq!( + entry_set_bytes(&set), + bytes.len(), + "should measure the stored array" + ); + assert!( + !String::from_utf8(bytes) + .expect("should be UTF-8") + .contains("\"source\""), + "should measure entries without source" + ); + } + + #[test] + fn validate_rejects_entry_set_limits() { + let rejects = |entry: &KvEntry, field: &str| { + let error = entry + .validate() + .expect_err(&format!("should reject {field}")); + assert!(error.contains(field), "should name {field}, got {error}"); + }; + + rejects(&with_set(Vec::new()), "eids"); + + let mut set = example_full_set(); + set[0].ext = Some(json!({ "pad": "x".repeat(MAX_PARTNER_EXT_BYTES) })); + rejects(&with_set(set), "ext"); + + let mut set = example_full_set(); + set[0].uids[0].ext = Some(json!("not an object")); + rejects(&with_set(set), "ext"); + + let mut set = example_full_set(); + set[0].inserter = Some(" ".to_owned()); + rejects(&with_set(set), "inserter"); + + let mut set = example_full_set(); + set[1].matcher = Some("m".repeat(MAX_EID_FIELD_BYTES + 1)); + rejects(&with_set(set), "matcher"); + + let mut set = example_full_set(); + set[0].mm = Some(-1); + rejects(&with_set(set), "mm"); + + let mut set = example_full_set(); + set[0].uids[1].atype = Some(-1); + rejects(&with_set(set), "atype"); + + let mut set = example_full_set(); + set[1].uids[0].id = " ".to_owned(); + rejects(&with_set(set), "uid"); + + let mut set = example_full_set(); + set[1].uids.clear(); + rejects(&with_set(set), "uid"); + + let nine_entries = (0..=MAX_ENTRIES_PER_SOURCE) + .map(|index| kv_eid(vec![kv_uid(&format!("example-{index}"), None)])) + .collect(); + rejects(&with_set(nine_entries), "entries"); + + let thirty_three_uids = (0..=MAX_UIDS_PER_SOURCE) + .map(|index| kv_uid(&format!("example-{index}"), None)) + .collect(); + rejects(&with_set(vec![kv_eid(thirty_three_uids)]), "uids"); + + let oversized = (0..4) + .map(|index| KvEid { + ext: Some(json!({ "pad": "x".repeat(1_100) })), + ..kv_eid(vec![kv_uid(&format!("example-{index}"), None)]) + }) + .collect(); + rejects(&with_set(oversized), "entry set"); + + let mut old = with_set(example_full_set()); + old.v = 1; + rejects(&old, "version"); + + with_set(example_full_set()) + .validate() + .expect("should accept the control set"); + } + + #[test] + fn validate_rejects_oversized_state_and_too_many_providers() { + let with_state = |state: KvProviderState| { + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ps.insert("id5".to_owned(), state); + entry + }; + let hashes = |count: usize| vec!["a".repeat(16); count]; + + let error = with_state(KvProviderState { + st: Some(json!("x".repeat(MAX_PROVIDER_STATE_BYTES))), + ..KvProviderState::default() + }) + .validate() + .expect_err("should reject st"); + assert!(error.contains("st"), "should name st"); + + for (name, state) in [ + ( + "iv", + KvProviderState { + iv: hashes(MAX_INVALIDATED_UIDS + 1), + ..KvProviderState::default() + }, + ), + ( + "ov", + KvProviderState { + ov: hashes(MAX_INVALIDATED_UIDS + 1), + ..KvProviderState::default() + }, + ), + ( + "fp", + KvProviderState { + o: Some(KvOptOut { + at: 1, + fp: "XYZ".to_owned(), + }), + ..KvProviderState::default() + }, + ), + ] { + let error = with_state(state) + .validate() + .expect_err(&format!("should reject {name}")); + assert!(error.contains(name), "should name {name}, got {error}"); + } + with_state(KvProviderState { + iv: hashes(MAX_INVALIDATED_UIDS), + ov: hashes(MAX_INVALIDATED_UIDS), + o: Some(KvOptOut { + at: 1, + fp: "1a".to_owned(), + }), + ..KvProviderState::default() + }) + .validate() + .expect("should accept 128 hashes in each list and a hex fingerprint"); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + for index in 0..=MAX_PROVIDER_STATES { + entry + .ps + .insert(format!("p{index}"), KvProviderState::default()); + } + assert!( + entry + .validate() + .expect_err("should reject ps size") + .contains("ps"), + "should name ps" + ); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ids.get_mut("example.com").expect("should exist").rv = Some("not-hex".to_owned()); + assert!( + entry + .validate() + .expect_err("should reject rv") + .contains("rv"), + "should name rv" + ); + + KvEntry::minimal("example.com", "example-uid", 1) + .validate() + .expect("should accept the control entry"); + } } diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index cc6705eaf8..6b1d4706f7 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -41,13 +41,13 @@ pub mod eids; pub mod finalize; pub mod generation; pub mod identify; +pub mod identity; pub mod kv; pub mod kv_backend; pub mod kv_types; pub mod partner; pub mod prebid_eids; pub mod pull_sync; -pub(crate) mod pull_sync_marker; pub mod rate_limiter; pub mod registry; @@ -68,32 +68,43 @@ pub fn log_id(ec_id: &str) -> String { format!("{prefix}\u{2026}") } +use std::sync::Arc; + use cookie::CookieJar; use edgezero_core::body::Body as EdgeBody; use error_stack::Report; -use http::Request; +use http::{Request, header}; +use url::Url; use crate::consent::{self as consent_mod, ConsentContext, ConsentPipelineInput}; -use crate::constants::{COOKIE_TS_EC, COOKIE_TS_EC_PULL_COMPLETE}; +use crate::constants::{COOKIE_TS_EC, COOKIE_TS_EC_REVOKED}; use crate::cookies::handle_request_cookies; use crate::ec::cookies::ec_id_has_only_allowed_chars; use crate::error::TrustedServerError; use crate::geo::GeoInfo; -use crate::openrtb::Eid; +use crate::http_util::{is_document_navigation, is_prefetch_request}; use crate::platform::RuntimeServices; use crate::settings::Settings; use device::DeviceSignals; -use self::kv::{CreateIfAbsentOutcome, KvIdentityGraph}; +use self::finalize::BrowserSeed; +use self::identity::measurement::{IdentityMeasurement, MeasurementInput, RouteKind}; +use self::identity::request::{RequestIdentity, WriterDecision}; +use self::identity::runtime::IdentityRuntime; +use self::kv::{CreateIfAbsentOutcome, KvIdentityGraph, apply_partner_id_updates}; use self::kv_types::KvEntry; -use self::pull_sync_marker::{PullSyncMarkerState, validate_marker_state}; +use self::rate_limiter::RateLimiter; -/// Bounded request classifications that may persist browser EID cookies. +/// Bounded request classifications that may stage browser EID cookies. /// /// Adapters classify publisher navigations and `POST /auction` only after /// pre-route filters allow dispatch. The shared page-bids handler classifies an /// admitted SPA navigation, while EC finalization classifies new identities. -/// Challenged or blocked requests remain unclassified and cannot persist EIDs. +/// Challenged or blocked requests remain unclassified and cannot stage EIDs. +/// +/// Only [`Self::NewEc`] still reports an `EID sync measurement` line. A +/// returning visitor's request reports the `identity request measurement` +/// line instead, which carries no source label. #[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] pub enum EidSyncSource { /// Publisher top-level document navigation. @@ -175,6 +186,9 @@ impl EcKvSnapshot { } } +/// Create attempts a new EC makes, each with a fresh ID after a collision. +const MAX_CREATE_ATTEMPTS: usize = 5; + pub use generation::{ ec_hash, generate_ec_id, is_valid_ec_hash, is_valid_ec_id, normalize_ec_id_for_kv, }; @@ -183,8 +197,6 @@ pub use generation::{ struct RequestEc { /// EC ID from the `ts-ec` cookie, if present. cookie_ec: Option, - /// Pull-sync completeness marker, if present. - pull_sync_marker: Option, /// The parsed cookie jar (retained for consent pipeline input). jar: Option, } @@ -201,17 +213,7 @@ fn parse_ec_from_request(req: &Request) -> Result Option { @@ -273,16 +275,19 @@ pub struct EcContext { kv_snapshot: EcKvSnapshot, /// Whether this request may rotate an orphaned EC identity. recovery_eligible: bool, - /// EIDs the route resolved from the current request: the `/auction` JSON - /// body, or the `ts-eids` cookie when the body carried none. Carried to - /// response finalization so KV ingestion can use the full set the client - /// sent instead of being limited to whatever fits in the size-capped - /// `ts-eids` cookie. - client_eids: Option>, - /// Browser-carried proof of recent pull-partner completeness. - pull_sync_marker: PullSyncMarkerState, /// Allowed returning-user EID persistence source, assigned only after request filters pass. eid_sync_source: Option, + /// Identity state this request decides and collects. + identity: RequestIdentity, + /// Withdrawn EC IDs whose before-send tombstone returned + /// [`EcKvSnapshot::Failed`], for the after-send retry. + failed_withdrawals: Vec, + /// The `ts-ec-revoked` marker the request carried: the consent + /// fingerprint a global revoke saw (spec 8). + revoke_marker: Option, + /// The row of an EC minted by [`EcContext::mint_deferred`], created after + /// the hold dispatch. + deferred_create: Option>, } impl EcContext { @@ -362,9 +367,25 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, - pull_sync_marker: PullSyncMarkerState::from_cookie(parsed.pull_sync_marker), eid_sync_source: None, + identity: { + let mut identity = RequestIdentity::default(); + identity.set_cookie_header_bytes( + req.headers() + .get_all(header::COOKIE) + .iter() + .map(|value| value.as_bytes().len()) + .sum(), + ); + identity + }, + failed_withdrawals: Vec::new(), + revoke_marker: parsed + .jar + .as_ref() + .and_then(|jar| jar.get(COOKIE_TS_EC_REVOKED)) + .map(|cookie| cookie.value().to_owned()), + deferred_create: None, }) } @@ -377,6 +398,10 @@ impl EcContext { /// If an EC ID already exists (from the request), this is a no-op. /// If consent does not permit EC creation, this is a no-op. /// + /// The new row carries `seed`, the request's browser ingest, so a new + /// visitor's create is the one write (spec 5.2 phase 1). Pass + /// [`BrowserSeed::empty`] when there is nothing to seed. + /// /// # Errors /// /// Returns an error if the client IP is unavailable and generation is @@ -385,9 +410,88 @@ impl EcContext { &mut self, settings: &Settings, kv: Option<&KvIdentityGraph>, + seed: &BrowserSeed, ) -> Result<(), Report> { - if self.ec_value.is_some() { + let Some(entry) = self.new_entry_if_needed(settings, seed) else { + return Ok(()); + }; + self.create_entry(settings, kv, entry, MAX_CREATE_ATTEMPTS) + } + + /// Mints an EC ID as [`generate_if_needed`](Self::generate_if_needed) + /// would, but keeps its row for + /// [`complete_deferred_create`](Self::complete_deferred_create), so a new + /// visitor's hold-mode providers can join the create (spec 4.7). + /// + /// No row exists until the create, so nothing may disclose the ID before + /// it: the hold dispatch calls no provider that reads it. + /// + /// # Errors + /// + /// Returns an error if the client IP is unavailable or HMAC generation + /// fails. + pub fn mint_deferred( + &mut self, + settings: &Settings, + seed: &BrowserSeed, + ) -> Result<(), Report> { + let Some(entry) = self.new_entry_if_needed(settings, seed) else { + return Ok(()); + }; + let ec_id = generation::generate_ec_id(settings, self.require_client_ip()?)?; + log::info!( + "Minted new EC ID with a deferred create: {}", + log_id(&ec_id) + ); + self.ec_value = Some(ec_id); + self.ec_generated = true; + self.deferred_create = Some(Box::new(entry)); + Ok(()) + } + + /// Creates the row [`mint_deferred`](Self::mint_deferred) kept, with + /// whatever the hold dispatch added. Does nothing without one. + /// + /// # Errors + /// + /// Returns an error if the create fails or no unique ID is found. + pub fn complete_deferred_create( + &mut self, + settings: &Settings, + kv: Option<&KvIdentityGraph>, + ) -> Result<(), Report> { + let Some(entry) = self.deferred_create.take() else { return Ok(()); + }; + // One attempt: the auction may already hold the minted ID, so a + // collision fails the create instead of minting another. + self.create_entry(settings, kv, *entry, 1) + } + + /// Whether a minted EC still waits for its create. + #[must_use] + pub fn has_deferred_create(&self) -> bool { + self.deferred_create.is_some() + } + + /// The row a minted EC will be created with. + #[must_use] + pub(crate) fn deferred_entry(&self) -> Option<&KvEntry> { + self.deferred_create.as_deref() + } + + /// Replaces the row a minted EC will be created with. + pub(crate) fn set_deferred_entry(&mut self, entry: KvEntry) { + if self.deferred_create.is_some() { + self.deferred_create = Some(Box::new(entry)); + } + } + + /// The new row for a request without an EC whose consent allows one, or + /// `None` when it needs none. + fn new_entry_if_needed(&self, settings: &Settings, seed: &BrowserSeed) -> Option { + if self.ec_value.is_some() { + return None; } if !consent::ec_consent_granted(&self.consent) { @@ -395,32 +499,61 @@ impl EcContext { "EC generation skipped: consent not granted (jurisdiction={})", self.consent.jurisdiction, ); - return Ok(()); + return None; + } + + if self.revoke_blocks_minting() { + log::info!("EC generation skipped: a global revoke holds until consent changes"); + return None; } - let client_ip = self.client_ip.as_deref().ok_or_else(|| { + let mut entry = KvEntry::new( + &self.consent, + self.geo_info.as_ref(), + current_timestamp(), + &settings.publisher.domain, + ); + entry.device = self + .device_signals + .as_ref() + .map(DeviceSignals::to_kv_device); + apply_partner_id_updates(&mut entry, seed.updates()); + Some(entry) + } + + fn require_client_ip(&self) -> Result<&str, Report> { + self.client_ip.as_deref().ok_or_else(|| { Report::new(TrustedServerError::EdgeCookie { message: "Client IP required for EC generation but unavailable".to_owned(), }) - })?; - - const MAX_CREATE_ATTEMPTS: usize = 5; - for attempt in 0..MAX_CREATE_ATTEMPTS { - let ec_id = generation::generate_ec_id(settings, client_ip)?; - log::info!("Generated new EC ID: {}", log_id(&ec_id)); - let now = current_timestamp(); - let mut entry = KvEntry::new( - &self.consent, - self.geo_info.as_ref(), - now, - &settings.publisher.domain, - ); - entry.device = self - .device_signals - .as_ref() - .map(DeviceSignals::to_kv_device); + }) + } + + /// Creates `entry` under the minted EC ID, or a fresh one, retrying an ID + /// collision with a fresh ID. + fn create_entry( + &mut self, + settings: &Settings, + kv: Option<&KvIdentityGraph>, + entry: KvEntry, + max_attempts: usize, + ) -> Result<(), Report> { + let client_ip = self.require_client_ip()?.to_owned(); + let mut minted = self.ec_value.take(); + self.ec_generated = false; + + for attempt in 0..max_attempts { + let ec_id = match minted.take() { + Some(ec_id) => ec_id, + None => { + let ec_id = generation::generate_ec_id(settings, &client_ip)?; + log::info!("Generated new EC ID: {}", log_id(&ec_id)); + ec_id + } + }; if let Some(graph) = kv { + self.identity.record_create_attempt(); match graph.create_if_absent(&ec_id, &entry) { Ok(CreateIfAbsentOutcome::Written) => { self.kv_snapshot = EcKvSnapshot::Present { @@ -431,7 +564,7 @@ impl EcContext { } Ok(CreateIfAbsentOutcome::AlreadyExists) => { log::warn!( - "Generated EC ID collision on attempt {}/{MAX_CREATE_ATTEMPTS}", + "Generated EC ID collision on attempt {}/{max_attempts}", attempt + 1 ); continue; @@ -451,14 +584,11 @@ impl EcContext { self.ec_value = Some(ec_id); self.ec_generated = true; - self.pull_sync_marker.invalidate_for_replaced_ec(); return Ok(()); } Err(Report::new(TrustedServerError::EdgeCookie { - message: format!( - "Failed to allocate a unique EC ID after {MAX_CREATE_ATTEMPTS} attempts" - ), + message: format!("Failed to allocate a unique EC ID after {max_attempts} attempts"), })) } @@ -514,6 +644,209 @@ impl EcContext { self.device_signals = Some(signals); } + /// Builds the `identity request measurement` for this request after + /// phase 4 (spec 14, step 1). + /// + /// The route kind is `new_ec` for a request that generated an EC, + /// `writer` for the one browser-driven writer, and `stager` otherwise. + #[must_use] + pub fn identity_measurement(&self, input: &MeasurementInput<'_>) -> IdentityMeasurement { + let route = if self.ec_generated { + RouteKind::NewEc + } else if self.identity.writer() { + RouteKind::Writer + } else { + RouteKind::Stager + }; + IdentityMeasurement { + cookie_header_bytes: self.identity.cookie_header_bytes(), + route, + // A snapshot that came from the new-EC create read nothing, and + // each create attempt is one write made before routing. + kv_reads: input + .kv_reads + .saturating_add(u32::from( + input.phase_one_read && self.identity.create_attempts() == 0, + )) + .saturating_add(self.identity.capture_kv_reads()), + // The capture's pre-send reads and writes go through the + // request-state graph, which the adapter's counters do not see. + kv_writes: input + .kv_writes + .saturating_add(self.identity.create_attempts()) + .saturating_add(self.identity.capture_kv_writes()), + flush: input.flush.copied(), + removal_lost: self.identity.removal_lost(), + removal_deferred: self.identity.removal_deferred(), + withdrawal_after_send: input.withdrawal_after_send, + withdrawal_after_send_failed: input.withdrawal_after_send_failed, + capture_unpersisted: self.identity.capture_unpersisted().clone(), + } + } + + /// Asks finalize to run the consent-withdrawal tombstone path for this + /// request, as a `revoke_scope = "global"` capture does from any client + /// (spec 8). + pub(crate) fn request_global_revoke(&mut self) { + self.identity.request_global_revoke(); + } + + /// Whether the request carries a global-revoke marker for its own + /// consent, which stops EC minting (spec 8). + #[must_use] + pub(crate) fn revoke_blocks_minting(&self) -> bool { + self.revoke_marker.as_deref() == Some(cookies::revoke_marker_value(&self.consent).as_str()) + } + + /// Whether the request carries a global-revoke marker at all. + #[must_use] + pub(crate) fn has_revoke_marker(&self) -> bool { + self.revoke_marker.is_some() + } + + /// Sets the request's global-revoke marker, as if it carried the cookie. + #[cfg(test)] + pub(crate) fn set_revoke_marker_for_test(&mut self, marker: &str) { + self.revoke_marker = Some(marker.to_owned()); + } + + /// Whether a capture asked for the consent-withdrawal tombstone path. + /// The adapter gives finalize the identity graph when it did. + #[must_use] + pub fn global_revoke_requested(&self) -> bool { + self.identity.global_revoke_requested() + } + + /// The identity state this request decides and collects. + #[must_use] + pub(crate) fn identity(&self) -> &RequestIdentity { + &self.identity + } + + /// The identity state this request decides and collects, to fill. + pub(crate) fn identity_mut(&mut self) -> &mut RequestIdentity { + &mut self.identity + } + + /// Gives the request the identity runtime built for the settings load. + /// + /// The adapter calls this where it builds the request's EC state. + pub fn set_identity_runtime(&mut self, runtime: Arc) { + self.identity.set_runtime(runtime); + } + + /// Turns the `ts-eids-s` mirror on for this request, keeping its provider + /// catalog, or with no provider when it has no runtime yet. + #[cfg(test)] + pub(crate) fn enable_mirror_for_test(&mut self) { + let runtime = match self.identity_runtime() { + Some(runtime) => runtime.with_mirror_enabled(), + None => IdentityRuntime::from_settings_with_providers( + &crate::test_support::tests::create_test_settings(), + Vec::new(), + std::collections::BTreeMap::new(), + ) + .expect("should build an empty runtime") + .with_mirror_enabled(), + }; + self.set_identity_runtime(Arc::new(runtime)); + } + + /// Gives the request the limiter that counts provider calls for the burst + /// cap and the circuit breaker before the response is sent. + pub fn set_identity_rate_limiter(&mut self, limiter: Arc) { + self.identity.set_rate_limiter(limiter); + } + + /// The identity runtime, when the adapter set one. + #[must_use] + pub(crate) fn identity_runtime(&self) -> Option<&IdentityRuntime> { + self.identity.runtime().map(Arc::as_ref) + } + + /// Records whether the request passed the real-browser check, for a route + /// that is never the writer. + pub fn set_real_browser(&mut self, is_real_browser: bool) { + self.identity.set_real_browser(is_real_browser); + } + + /// Decides whether this request is the one browser-driven identity writer. + /// + /// Only an organic real-browser top-level document navigation that is not + /// a prefetch writes (spec 5.1). Also captures the user agent, `DNT` and + /// the page URL without query or fragment. + /// + /// The page URL is read from the request URI, so it needs an absolute + /// URI. The Fastly adapter gives one (`edgezero-adapter-fastly` builds the + /// request from `get_url_str()`); a request with an origin-form URI has no + /// page URL, and a provider that wants the `PageUrl` input is then not + /// due. + pub fn decide_writer( + &mut self, + req: &Request, + organic_route: bool, + is_real_browser: bool, + ) { + let writer = organic_route + && is_document_navigation(req) + && !is_prefetch_request(req) + && is_real_browser; + let user_agent = req + .headers() + .get(header::USER_AGENT) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + let dnt = req + .headers() + .get("dnt") + .and_then(|value| value.to_str().ok()) + .is_some_and(|value| value.trim() == "1"); + let page_url = Url::parse(&req.uri().to_string()).ok().map(|mut url| { + url.set_query(None); + url.set_fragment(None); + url + }); + self.identity.set_writer_decision(WriterDecision { + writer, + is_real_browser, + user_agent, + dnt, + page_url, + }); + } + + /// Decides whether a `/_ts/page-bids` call is the SPA writer (spec 5.1): + /// a real browser, `Sec-Fetch-Site: same-origin` and an interval above + /// zero. The flush then checks the interval against the row's last + /// writer flush. Reads the real-browser flag set earlier with + /// [`set_real_browser`](Self::set_real_browser). + pub fn decide_spa_writer(&mut self, req: &Request, interval_sec: u64) { + let same_origin = req + .headers() + .get("sec-fetch-site") + .and_then(|value| value.to_str().ok()) + == Some("same-origin"); + if !(same_origin && interval_sec > 0 && self.identity.is_real_browser()) { + return; + } + self.identity.set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: req + .headers() + .get(header::USER_AGENT) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned), + dnt: req + .headers() + .get("dnt") + .and_then(|value| value.to_str().ok()) + .is_some_and(|value| value.trim() == "1"), + page_url: None, + }); + self.identity.set_spa_writer(true); + } + /// Returns the device signals, if set. #[must_use] pub fn device_signals(&self) -> Option<&DeviceSignals> { @@ -538,6 +871,49 @@ impl EcContext { &self.kv_snapshot } + /// Returns the withdrawn EC IDs whose tombstone has failed and not yet + /// succeeded. + /// + /// Before the after-send retry these are the before-send failures. After + /// it, only the IDs that failed a second time remain, which is what the + /// failure counters count. + #[must_use] + pub(crate) fn failed_withdrawals(&self) -> &[String] { + &self.failed_withdrawals + } + + /// Whether any withdrawn EC ID's tombstone is still unwritten. + #[must_use] + pub fn has_failed_withdrawals(&self) -> bool { + !self.failed_withdrawals.is_empty() + } + + /// Whether a capture removal write failed before send and awaits its + /// one retry after send. + #[must_use] + pub fn has_pending_removals(&self) -> bool { + self.identity.pending_removals().is_some() + } + + /// Whether a capture kept positive writes for the pending key, which + /// [`identity::flush::defer_capture_writes`] appends after send. + #[must_use] + pub fn has_capture_writes(&self) -> bool { + !self.identity.capture_writes().is_empty() + } + + /// Forgets a withdrawn EC ID whose retried tombstone was written. + pub(crate) fn clear_failed_withdrawal(&mut self, ec_id: &str) { + self.failed_withdrawals.retain(|id| id != ec_id); + } + + /// Records a withdrawn EC ID whose before-send tombstone failed. + pub(crate) fn record_failed_withdrawal(&mut self, ec_id: &str) { + if !self.failed_withdrawals.iter().any(|id| id == ec_id) { + self.failed_withdrawals.push(ec_id.to_owned()); + } + } + /// Replaces the request-scoped identity-graph snapshot. pub fn set_kv_snapshot(&mut self, snapshot: EcKvSnapshot) { self.kv_snapshot = snapshot; @@ -565,55 +941,11 @@ impl EcContext { self.recovery_eligible } - /// Records the current request's own EIDs (e.g. an `/auction` body) for - /// use by response finalization's KV ingestion. - pub(crate) fn set_client_eids(&mut self, eids: Vec) { - self.client_eids = Some(eids); - } - - /// Returns the current request's own EIDs, if the route captured any. - #[must_use] - pub(crate) fn client_eids(&self) -> Option<&[Eid]> { - self.client_eids.as_deref() - } - - /// Validates a browser completeness marker against the active EC and partner set. - pub(crate) fn validate_pull_sync_marker( - &mut self, - settings: &Settings, - registry: ®istry::PartnerRegistry, - ) { - validate_marker_state( - &mut self.pull_sync_marker, - settings, - registry, - self.ec_value.as_deref(), - ); - } - - /// Returns the current pull-sync marker state. - #[must_use] - pub(crate) fn pull_sync_marker(&self) -> &PullSyncMarkerState { - &self.pull_sync_marker - } - - /// Returns mutable pull-sync marker state for response reconciliation. - pub(crate) fn pull_sync_marker_mut(&mut self) -> &mut PullSyncMarkerState { - &mut self.pull_sync_marker - } - - /// Sets pull-sync marker state in focused unit tests. - #[cfg(test)] - pub(crate) fn set_pull_sync_marker_for_test(&mut self, state: PullSyncMarkerState) { - self.pull_sync_marker = state; - } - /// Replaces an orphaned active ID after its new backing row is persisted. pub(crate) fn replace_with_generated(&mut self, ec_id: String, snapshot: EcKvSnapshot) { self.ec_value = Some(ec_id); self.ec_generated = true; self.kv_snapshot = snapshot; - self.pull_sync_marker.invalidate_for_replaced_ec(); } /// Returns whether EC creation is permitted by consent for this request. @@ -663,12 +995,34 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, - pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, + identity: RequestIdentity::default(), + failed_withdrawals: Vec::new(), + revoke_marker: None, + deferred_create: None, } } + /// Creates a test-only [`EcContext`] for a real browser's writer + /// navigation, with explicit client IP. + #[cfg(test)] + #[must_use] + pub fn new_writer_for_test_with_ip( + ec_value: Option, + consent: ConsentContext, + client_ip: Option, + ) -> Self { + let mut context = Self::new_for_test_with_ip(ec_value, consent, client_ip); + context.identity.set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: None, + dnt: false, + page_url: None, + }); + context + } + /// Creates a test-only [`EcContext`] with explicit client IP. #[cfg(test)] #[must_use] @@ -688,9 +1042,11 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, - pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, + identity: RequestIdentity::default(), + failed_withdrawals: Vec::new(), + revoke_marker: None, + deferred_create: None, } } @@ -716,9 +1072,11 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, - pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, + identity: RequestIdentity::default(), + failed_withdrawals: Vec::new(), + revoke_marker: None, + deferred_create: None, } } } @@ -774,6 +1132,128 @@ mod tests { use crate::platform::test_support::noop_services; use crate::test_support::tests::create_test_settings; + #[test] + fn decide_spa_writer_needs_a_same_origin_browser_and_an_interval() { + let page_bids = |site: Option<&str>| { + let mut builder = Request::builder() + .uri("https://www.example.com/_ts/page-bids?path=/a") + .header("x-tsjs-page-bids", "1"); + if let Some(site) = site { + builder = builder.header("sec-fetch-site", site); + } + builder + .body(EdgeBody::empty()) + .expect("should build the request") + }; + let decide = |request: &Request, browser: bool, interval: u64| { + let mut context = EcContext::default(); + context.set_real_browser(browser); + context.decide_spa_writer(request, interval); + (context.identity().writer(), context.identity().spa_writer()) + }; + + assert_eq!( + decide(&page_bids(Some("same-origin")), true, 300), + (true, true), + "should make a same-origin browser page-bids call the SPA writer" + ); + for (name, request, browser, interval) in [ + ("no Sec-Fetch-Site", page_bids(None), true, 300), + ("same-site", page_bids(Some("same-site")), true, 300), + ("not a browser", page_bids(Some("same-origin")), false, 300), + ("interval off", page_bids(Some("same-origin")), true, 0), + ] { + assert_eq!( + decide(&request, browser, interval), + (false, false), + "should not be the writer: {name}" + ); + } + } + + #[test] + fn decide_writer_needs_every_condition() { + let navigation = || { + Request::builder() + .uri("https://www.example.com/a?b=c#d") + .header("sec-fetch-dest", "document") + .header(header::USER_AGENT, "ExampleBrowser/1.0") + .header("dnt", "1") + .body(EdgeBody::empty()) + .expect("should build the request") + }; + let mut context = EcContext::default(); + context.decide_writer(&navigation(), true, true); + assert!( + context.identity().writer(), + "should make an organic real-browser document navigation the writer" + ); + assert!( + context.identity().is_real_browser(), + "should record the real browser" + ); + assert_eq!( + context.identity().page_url().map(Url::as_str), + Some("https://www.example.com/a"), + "should drop the query and fragment" + ); + assert_eq!( + context.identity().user_agent(), + Some("ExampleBrowser/1.0"), + "should capture the user agent" + ); + assert!(context.identity().dnt(), "should capture DNT"); + + for (name, organic, browser, request) in [ + ("named route", false, true, navigation()), + ("bot", true, false, navigation()), + ( + "prefetch", + true, + true, + Request::builder() + .header("sec-fetch-dest", "document") + .header("sec-purpose", "prefetch") + .body(EdgeBody::empty()) + .expect("should build the request"), + ), + ] { + let mut context = EcContext::default(); + context.decide_writer(&request, organic, browser); + assert!( + !context.identity().writer(), + "should not be a writer: {name}" + ); + assert_eq!( + context.identity().is_real_browser(), + browser, + "should keep the real-browser flag: {name}" + ); + } + } + + #[test] + fn decide_writer_has_no_page_url_for_an_origin_form_uri() { + let request = Request::builder() + .uri("/a?b=c") + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("should build the request"); + let mut context = EcContext::default(); + + context.decide_writer(&request, true, true); + + assert!( + context.identity().writer(), + "control: the request is still the writer" + ); + assert_eq!( + context.identity().page_url(), + None, + "should have no page URL without an absolute URI" + ); + } + /// [`EcKvStore`] wrapper whose first `collisions` `Add` writes report a /// precondition failure, forcing generation to retry with a fresh suffix. struct AddCollidingEcKv { @@ -845,7 +1325,7 @@ mod tests { EcContext::new_for_test_with_ip(None, granting_consent(), Some("192.0.2.5".to_owned())); let graph = KvIdentityGraph::new(AddCollidingEcKv::new(2)); - ec.generate_if_needed(&settings, Some(&graph)) + ec.generate_if_needed(&settings, Some(&graph), &BrowserSeed::empty()) .expect("should generate after bounded collisions"); assert!(ec.ec_value().is_some(), "should allocate a fresh EC ID"); @@ -856,6 +1336,100 @@ mod tests { ); } + #[test] + fn a_revoke_marker_for_this_consent_stops_minting_until_consent_changes() { + let settings = create_test_settings(); + let mint_under = |consent: ConsentContext, marker: &str| { + let mut ec = + EcContext::new_for_test_with_ip(None, consent, Some("192.0.2.5".to_owned())); + ec.set_revoke_marker_for_test(marker); + let graph = KvIdentityGraph::in_memory("ec-store"); + ec.generate_if_needed(&settings, Some(&graph), &BrowserSeed::empty()) + .expect("should not error"); + ec.ec_generated() + }; + let same = cookies::revoke_marker_value(&granting_consent()); + let mut with_gpc = granting_consent(); + with_gpc.gpc = true; + let earlier = cookies::revoke_marker_value(&with_gpc); + + assert_ne!(earlier, same, "precondition: GPC changes the marker value"); + assert!( + !mint_under(granting_consent(), &same), + "should not mint while the consent the revoke saw is unchanged" + ); + assert!( + mint_under(granting_consent(), &earlier), + "should mint once the consent differs from the one the revoke saw" + ); + } + + #[test] + fn read_from_request_reads_the_revoke_marker() { + let settings = create_test_settings(); + let req = create_test_request(&[("cookie", "ts-ec-revoked=10009")]); + + let ec = EcContext::read_from_request(&settings, &req, &noop_services()) + .expect("should read EC context"); + + assert!(ec.has_revoke_marker(), "should read the marker cookie"); + let plain = + EcContext::read_from_request(&settings, &create_test_request(&[]), &noop_services()) + .expect("should read EC context"); + assert!(!plain.has_revoke_marker(), "control: no cookie, no marker"); + } + + #[test] + fn mint_deferred_writes_nothing_until_the_create() { + let settings = create_test_settings(); + let graph = KvIdentityGraph::in_memory("ec-store"); + let mut ec = + EcContext::new_for_test_with_ip(None, granting_consent(), Some("192.0.2.5".to_owned())); + + ec.mint_deferred(&settings, &BrowserSeed::empty()) + .expect("should mint"); + + let ec_id = ec.ec_value().expect("should hold the minted ID").to_owned(); + assert!(ec.has_deferred_create(), "should keep the row for later"); + assert!( + graph.get(&ec_id).expect("should read").is_none(), + "should write nothing yet" + ); + + ec.complete_deferred_create(&settings, Some(&graph)) + .expect("should create"); + + assert_eq!( + ec.ec_value(), + Some(ec_id.as_str()), + "should keep the minted ID" + ); + assert!( + graph.get(&ec_id).expect("should read").is_some(), + "should create the row" + ); + assert!( + matches!(ec.kv_snapshot(), EcKvSnapshot::Present { .. }), + "should hold the created row" + ); + } + + #[test] + fn deferred_create_fails_on_a_collision_instead_of_reminting() { + let settings = create_test_settings(); + let graph = KvIdentityGraph::new(AddCollidingEcKv::new(1)); + let mut ec = + EcContext::new_for_test_with_ip(None, granting_consent(), Some("192.0.2.5".to_owned())); + ec.mint_deferred(&settings, &BrowserSeed::empty()) + .expect("should mint"); + + let result = ec.complete_deferred_create(&settings, Some(&graph)); + + assert!(result.is_err(), "should fail rather than mint another ID"); + assert!(ec.ec_value().is_none(), "should keep no EC"); + assert!(!ec.ec_generated(), "should not count as generated"); + } + #[test] fn generate_if_needed_errors_after_collision_exhaustion() { let settings = create_test_settings(); @@ -864,7 +1438,7 @@ mod tests { // Collide on every attempt so the bounded retry is exhausted. let graph = KvIdentityGraph::new(AddCollidingEcKv::new(u32::MAX)); - let result = ec.generate_if_needed(&settings, Some(&graph)); + let result = ec.generate_if_needed(&settings, Some(&graph), &BrowserSeed::empty()); assert!(result.is_err(), "should fail after exhausting attempts"); assert!( @@ -1085,7 +1659,7 @@ mod tests { let mut ec = EcContext::read_from_request(&settings, &req, &noop_services()) .expect("should read EC context"); - ec.generate_if_needed(&settings, None) + ec.generate_if_needed(&settings, None, &BrowserSeed::empty()) .expect("should not error when EC already exists"); assert_eq!( diff --git a/crates/trusted-server-core/src/ec/prebid_eids.rs b/crates/trusted-server-core/src/ec/prebid_eids.rs index 9d2b587806..c3718cb200 100644 --- a/crates/trusted-server-core/src/ec/prebid_eids.rs +++ b/crates/trusted-server-core/src/ec/prebid_eids.rs @@ -148,11 +148,14 @@ fn raw_diagnostic_sources(raw: &RawValue) -> Vec { /// route the cookie. The `sharedId` cookie is applied last, so it wins over a /// Prebid EID for the same partner. With repeated sources, the later entry's /// first valid UID is the candidate for that partner. +/// +/// When `now` is `Some`, each update carries it as its acquisition time. pub(crate) fn collect_eid_updates( eids_cookie: Option<&str>, sharedid_cookie: Option<&str>, body_eids: Option<&[Eid]>, registry: &PartnerRegistry, + now: Option, ) -> Vec { if registry.is_empty() { return Vec::new(); @@ -170,6 +173,12 @@ pub(crate) fn collect_eid_updates( { updates.push(update); } + if let Some(now) = now { + updates = updates + .into_iter() + .map(|update| update.with_acquired_at(now)) + .collect(); + } dedupe_partner_updates(updates) } @@ -233,13 +242,10 @@ fn collect_prebid_eid_updates_from_eids( pub(crate) fn dedupe_partner_updates(updates: Vec) -> Vec { let mut latest = std::collections::BTreeMap::new(); for update in updates { - latest.insert(update.partner_id, update.uid); + latest.insert(update.partner_id.clone(), update); } - latest - .into_iter() - .map(|(partner_id, uid)| PartnerIdUpdate::new(partner_id, uid)) - .collect() + latest.into_values().collect() } fn first_valid_uid(uids: &[Uid]) -> Option<&Uid> { @@ -313,6 +319,8 @@ mod tests { use base64::engine::general_purpose::STANDARD as BASE64; use serde_json::json; + use crate::ec::kv::apply_partner_id_updates; + use crate::ec::kv_types::KvEntry; use crate::ec::registry::PartnerRegistry; use crate::redacted::Redacted; use crate::settings::EcPartner; @@ -330,8 +338,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -689,7 +698,13 @@ mod tests { {"source": "id5-sync.com", "uids": [{"id": "ID5_abc", "atype": 1}]} ])); - let updates = collect_eid_updates(Some(&eids_cookie), Some(" shared-1 "), None, ®istry); + let updates = collect_eid_updates( + Some(&eids_cookie), + Some(" shared-1 "), + None, + ®istry, + None, + ); assert_eq!( updates.len(), @@ -729,7 +744,8 @@ mod tests { ), ]; - let updates = collect_eid_updates(Some(&eids_cookie), None, Some(&body_eids), ®istry); + let updates = + collect_eid_updates(Some(&eids_cookie), None, Some(&body_eids), ®istry, None); assert_eq!( updates, @@ -750,7 +766,8 @@ mod tests { {"source": "id5-sync.com", "uids": [{"id": "ID5_abc", "atype": 1}]} ])); - let updates = collect_eid_updates(Some(&eids_cookie), Some("shared-1"), None, ®istry); + let updates = + collect_eid_updates(Some(&eids_cookie), Some("shared-1"), None, ®istry, None); assert!( updates.is_empty(), @@ -791,7 +808,13 @@ mod tests { {"source": "liveramp.com", "uids": [{"id": "LR_xyz", "atype": 3}]} ])); - let updates = collect_eid_updates(Some(&cookie), Some("shared-cookie-id"), None, ®istry); + let updates = collect_eid_updates( + Some(&cookie), + Some("shared-cookie-id"), + None, + ®istry, + None, + ); assert_eq!( updates, @@ -814,7 +837,7 @@ mod tests { } ])); - let updates = collect_eid_updates(Some(&cookie), None, None, ®istry); + let updates = collect_eid_updates(Some(&cookie), None, None, ®istry, None); assert_eq!( updates, @@ -830,7 +853,8 @@ mod tests { {"source": "sharedid.org", "uids": [{"id": "prebid-shared", "atype": 3}]} ])); - let updates = collect_eid_updates(Some(&cookie), Some("cookie-shared"), None, ®istry); + let updates = + collect_eid_updates(Some(&cookie), Some("cookie-shared"), None, ®istry, None); assert_eq!( updates, @@ -846,11 +870,53 @@ mod tests { {"source": "unknown.example", "uids": [{"id": "unknown", "atype": 1}]} ])); - let updates = collect_eid_updates(Some(&cookie), None, None, ®istry); + let updates = collect_eid_updates(Some(&cookie), None, None, ®istry, None); assert!( updates.is_empty(), "should ignore unmatched partner updates" ); } + + #[test] + fn golden_browser_ingest_stores_one_uid_per_registered_source() { + let registry = make_registry(vec![("id5", "id5-sync.com"), ("sharedid", "sharedid.org")]); + let eids_cookie = encode_json(&json!([ + { "source": "id5-sync.com", "uids": [{ "id": "ID5_earlier_entry", "atype": 1 }] }, + { + "source": "id5-sync.com", + "uids": [ + { "id": " ", "atype": 1 }, + { "id": "ID5_first", "atype": 1, "ext": { "linkType": 2 } }, + { "id": "ID5_second", "atype": 1 } + ], + "inserter": "example.com", + "matcher": "partner.example.com", + "mm": 3, + "ext": { "segments": ["example-segment"] } + }, + { "source": "sharedid.org", "uids": [{ "id": "prebid-shared-id", "atype": 1 }] }, + { "source": "unregistered.example.com", "uids": [{ "id": "other", "atype": 1 }] } + ])); + let updates = collect_eid_updates( + Some(&eids_cookie), + Some(" cookie-shared-id "), + None, + ®istry, + Some(1_760_000_000), + ); + let mut entry = KvEntry::minimal("example.com", "existing", 1); + apply_partner_id_updates(&mut entry, &updates); + let stored = serde_json::to_value(&entry.ids).expect("should serialize ids"); + + assert_eq!( + stored, + json!({ + "example.com": { "eids": [{ "uids": [{ "id": "existing" }] }] }, + "id5-sync.com": { "at": 1_760_000_000, "eids": [{ "uids": [{ "id": "ID5_first" }] }] }, + "sharedid.org": { "at": 1_760_000_000, "eids": [{ "uids": [{ "id": "cookie-shared-id" }] }] } + }), + "should store one UID per registered source by today's rule, with no EID-level field, atype or ext" + ); + } } diff --git a/crates/trusted-server-core/src/ec/pull_sync.rs b/crates/trusted-server-core/src/ec/pull_sync.rs index 191e6c6d24..a718aa5568 100644 --- a/crates/trusted-server-core/src/ec/pull_sync.rs +++ b/crates/trusted-server-core/src/ec/pull_sync.rs @@ -1,552 +1,69 @@ -//! Pull sync background dispatch. +//! Pull sync URL helpers. //! -//! Launches partner pull-sync requests for organic traffic after the client -//! response has been sent. Dispatch is best-effort and never affects client -//! response status. -//! -//! Pull sync currently fills missing partner UIDs only. Once a partner UID is -//! present in the EC identity entry, it is not periodically refreshed because -//! the entry no longer stores per-partner sync timestamps. +//! Pull partners are called as generic pull identity providers +//! (`identity::generic_pull`), dispatched by the writer's phase 4 +//! flush. These helpers validate and build the partner URL they call. -use edgezero_core::body::Body as EdgeBody; -use http::{Method, StatusCode, header}; -use serde::Deserialize; use url::Url; -use crate::consent::allows_eid_persistence; -use crate::platform::{ - DEFAULT_FIRST_BYTE_TIMEOUT, PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, - PlatformResponse, RuntimeServices, -}; -use crate::settings::Settings; - -use super::generation::{ec_hash, is_valid_ec_id}; -use super::kv::{KvIdentityGraph, PartnerIdUpdate}; -use super::kv_types::KvEntry; -use super::rate_limiter::RateLimiter; -use super::registry::{PartnerConfig, PartnerRegistry}; - -// `current_timestamp` is defined in the parent `ec` module. -use super::EcContext; -use super::EcKvSnapshot; -use super::current_timestamp; - -/// Inputs needed to dispatch pull sync after response flush. -#[derive(Debug, Clone)] -pub struct PullSyncContext { - ec_id: String, - snapshot: EcKvSnapshot, -} - -impl PullSyncContext { - /// Returns the EC ID for the request. - #[must_use] - pub fn ec_id(&self) -> &str { - &self.ec_id - } -} - -struct InFlightPull { - source_domain: String, - pending: PlatformPendingRequest, -} - -#[derive(Debug, Deserialize)] -struct PullSyncResponse { - uid: Option, -} - -/// Builds post-send pull-sync context from the route EC context. +/// Returns the pull URL when it is HTTPS and its host is allowlisted. /// -/// Returns `None` when there are no pull-enabled partners, when consent denies -/// EC, when there is no valid active EC ID, or when the request snapshot holds -/// no consented row that is still missing at least one pull-partner UID. -#[must_use] -pub fn build_pull_sync_context( - ec_context: &EcContext, - registry: &PartnerRegistry, -) -> Option { - // Without EID-persistence consent (TCF Purpose 1 + 4), finalization - // withholds browser-supplied partner UIDs. That leaves partner slots empty, - // which pull sync would otherwise read as eligible and disclose the EC ID - // to partners for exactly the users who opted out. - if registry.pull_enabled_partners().is_empty() - || !ec_context.ec_allowed() - || !allows_eid_persistence(ec_context.consent()) - { - return None; - } - - let ec_id_ref = ec_context.ec_value()?; - if !is_valid_ec_id(ec_id_ref) { - log::debug!("Pull sync: skipping dispatch because active EC ID is invalid format"); - return None; - } - - let entry = ec_context.kv_snapshot().entry_for(ec_id_ref)?; - if !entry.consent.ok || entry_is_pull_complete(entry, registry) { - return None; - } - - let ec_id = ec_id_ref.to_owned(); - let snapshot = ec_context.kv_snapshot().clone(); - Some(PullSyncContext { ec_id, snapshot }) -} - -/// Dispatches partner pull-sync requests in the background. -/// -/// This function is best-effort: all errors are logged and swallowed. -/// -/// # Consent revalidation -/// -/// Pull sync runs *after* the page response has been flushed to the browser and -/// discloses the raw `ec_id` to partners, so it cannot be authorized by the -/// request snapshot alone: a concurrent CMP withdrawal can tombstone the row in -/// the window between snapshot capture and dispatch. The live row is re-read -/// immediately before the first partner request and anything short of a live, -/// consenting entry cancels the dispatch. The read is deferred until after the -/// snapshot-based eligibility filter, so a request with nothing to pull still -/// costs no KV operation, and the fresh snapshot is reused for the write-back -/// instead of the stale one. -/// -/// # Panics -/// -/// Panics if the HTTP request builder produces an invalid request, which -/// cannot happen with the hardcoded method and well-formed URI used here. -pub fn dispatch_pull_sync( - settings: &Settings, - kv: &KvIdentityGraph, - registry: &PartnerRegistry, - rate_limiter: &dyn RateLimiter, - context: &PullSyncContext, - services: &RuntimeServices, -) { - let now = current_timestamp(); - let mut pull_partners = registry.pull_enabled_partners(); - - if pull_partners.is_empty() { - return; - } - - let Some(request_entry) = context.snapshot.entry_for(context.ec_id()) else { - log::debug!( - "Pull sync: skipping dispatch for '{}' because the request captured no usable \ - identity snapshot", - super::log_id(context.ec_id()) - ); - return; - }; - if !request_entry.consent.ok { - return; - } - - // Sort by source domain for deterministic ordering, then apply a rotating - // hourly offset so that different partners get dispatch priority (§10.3). - pull_partners.sort_by(|a, b| a.source_domain.cmp(&b.source_domain)); - - log::debug!( - "Pull sync: {} pull-enabled partners after filtering", - pull_partners.len(), - ); - - // Rotate the partner list so that the starting partner changes each - // hour. This ensures fair distribution when max_concurrency limits - // how many partners are dispatched per request. - let offset = (now / 3600) as usize % pull_partners.len(); - pull_partners.rotate_left(offset); - - // Drop partners the request snapshot already has a UID for before paying - // for the revalidation read: a dispatch with nothing to pull must not add a - // KV operation. - pull_partners.retain(|partner| is_partner_pull_eligible(partner, Some(request_entry))); - if pull_partners.is_empty() { - return; - } - - let live_snapshot = kv.load_snapshot(context.ec_id()); - let Some(live_entry) = live_snapshot.entry_for(context.ec_id()) else { - log::warn!( - "Pull sync: skipping dispatch for '{}' because the live identity row could not be \ - confirmed", - super::log_id(context.ec_id()) - ); - return; - }; - if !live_entry.consent.ok { - log::info!( - "Pull sync: skipping dispatch for '{}' because consent was withdrawn after the \ - request snapshot was captured", - super::log_id(context.ec_id()) - ); - return; - } - - let max_concurrency = settings.ec.pull_sync_concurrency.max(1); - let mut in_flight: Vec = Vec::new(); - let mut updates = Vec::new(); - - for partner in pull_partners { - // Re-checked against the live row: a concurrent request may have filled - // this partner's UID since the request snapshot was captured. - if !is_partner_pull_eligible(partner, Some(live_entry)) { - continue; - } - - let Some(url) = validated_pull_sync_url(partner) else { - continue; - }; - - let rate_key = pull_rate_limit_key(&partner.source_domain, context.ec_id()); - match rate_limiter.exceeded(&rate_key, partner.pull_sync_rate_limit) { - Ok(true) => { - log::debug!( - "Pull sync: rate-limited partner '{}' for ec_id '{}'", - partner.source_domain, - super::log_id(context.ec_id()) - ); - continue; - } - Ok(false) => {} - Err(err) => { - log::warn!( - "Pull sync: failed to read rate limit for partner '{}': {err:?}", - partner.source_domain - ); - continue; - } - } - - let Some(token) = partner.ts_pull_token.as_ref() else { - log::warn!( - "Pull sync: partner '{}' enabled but missing ts_pull_token", - partner.source_domain - ); - continue; - }; - - let request_url = build_pull_request_url(url, context.ec_id()); - let scheme = request_url.scheme().to_string(); - let host = request_url.host_str().unwrap_or_default().to_string(); - let port = request_url.port(); - - let backend_name = match services.backend().ensure(&PlatformBackendSpec { - scheme, - host, - port, - host_header_override: None, - certificate_check: settings.proxy.certificate_check, - first_byte_timeout: DEFAULT_FIRST_BYTE_TIMEOUT, - between_bytes_timeout: DEFAULT_FIRST_BYTE_TIMEOUT, - discriminator: None, - }) { - Ok(name) => name, - Err(err) => { - log::warn!( - "Pull sync: failed to resolve backend for partner '{}': {err:?}", - partner.source_domain - ); - continue; - } - }; - - let request = http::Request::builder() - .method(Method::GET) - .uri(request_url.as_str()) - .header("authorization", format!("Bearer {}", token.expose())) - .body(EdgeBody::empty()) - .expect("should build pull sync request"); - - let pending = match futures::executor::block_on( - services - .http_client() - .send_async(PlatformHttpRequest::new(request, backend_name)), - ) { - Ok(pending) => pending, - Err(err) => { - log::warn!( - "Pull sync: failed to dispatch partner '{}': {err:?}", - partner.source_domain - ); - continue; - } - }; - - in_flight.push(InFlightPull { - source_domain: partner.source_domain.clone(), - pending, - }); - - if in_flight.len() >= max_concurrency { - drain_pull_batch(&mut in_flight, services, &mut updates); - } - } - - drain_pull_batch(&mut in_flight, services, &mut updates); - if !updates.is_empty() { - // Write back from the revalidated snapshot, not the request one: its - // generation is current, so the first CAS attempt is not spent losing a - // conflict against the read that authorized this dispatch. - let outcome = - kv.upsert_partner_ids_from_snapshot(context.ec_id(), &updates, live_snapshot.clone()); - if matches!(outcome, EcKvSnapshot::Failed { .. }) { - log::warn!( - "Pull sync: failed to persist partner updates for '{}'", - super::log_id(context.ec_id()) - ); - } - } -} - -/// Returns whether a consented live entry holds an ID for every pull-enabled -/// partner. Always returns `false` when there are no pull-enabled partners. -#[must_use] -pub(crate) fn entry_is_pull_complete(entry: &KvEntry, registry: &PartnerRegistry) -> bool { - let pull_partners = registry.pull_enabled_partners(); - !pull_partners.is_empty() - && entry.consent.ok - && pull_partners - .iter() - .all(|partner| !is_partner_pull_eligible(partner, Some(entry))) -} - -fn is_partner_pull_eligible(partner: &PartnerConfig, kv_entry: Option<&KvEntry>) -> bool { - kv_entry - .and_then(|entry| entry.ids.get(&partner.source_domain)) - .is_none() -} - -fn validated_pull_sync_url(partner: &PartnerConfig) -> Option { - let pull_sync_url = partner.pull_sync_url.as_deref()?; +/// The host comparison ignores case and a trailing dot on both sides. +/// Returns `None`, with a log line that names no partner, for a missing, +/// invalid or non-HTTPS URL and for a host outside `allowed_domains`. +pub(crate) fn validated_pull_sync_url( + pull_sync_url: Option<&str>, + allowed_domains: &[String], +) -> Option { + let pull_sync_url = pull_sync_url?; let parsed = match Url::parse(pull_sync_url) { Ok(url) => url, Err(err) => { - log::error!( - "Pull sync: partner '{}' has invalid pull_sync_url '{}': {err}", - partner.source_domain, - pull_sync_url - ); + log::error!("Pull sync: invalid pull_sync_url '{pull_sync_url}': {err}"); return None; } }; if parsed.scheme() != "https" { log::error!( - "Pull sync: partner '{}' pull_sync_url must use HTTPS, got scheme '{}'", - partner.source_domain, + "Pull sync: pull_sync_url must use HTTPS, got scheme '{}'", parsed.scheme() ); return None; } let Some(hostname) = parsed.host_str() else { - log::error!( - "Pull sync: partner '{}' pull_sync_url has no hostname: {}", - partner.source_domain, - pull_sync_url - ); + log::error!("Pull sync: pull_sync_url has no hostname: {pull_sync_url}"); return None; }; let hostname = hostname.trim_end_matches('.').to_ascii_lowercase(); - if !partner.pull_sync_allowed_domains.iter().any(|domain| { + if !allowed_domains.iter().any(|domain| { domain .trim() .trim_end_matches('.') .eq_ignore_ascii_case(&hostname) }) { - log::error!( - "Pull sync: partner '{}' URL host '{}' not in pull_sync_allowed_domains", - partner.source_domain, - hostname - ); + log::error!("Pull sync: URL host '{hostname}' not in pull_sync_allowed_domains"); return None; } Some(parsed) } -fn build_pull_request_url(mut base_url: Url, ec_id: &str) -> Url { +/// Appends the `ec_id` query parameter to a validated pull URL. +pub(crate) fn build_pull_request_url(mut base_url: Url, ec_id: &str) -> Url { base_url.query_pairs_mut().append_pair("ec_id", ec_id); base_url } -fn pull_rate_limit_key(source_domain: &str, ec_id: &str) -> String { - format!("pull:{source_domain}:{}", ec_hash(ec_id)) -} - -fn drain_pull_batch( - in_flight: &mut Vec, - services: &RuntimeServices, - updates: &mut Vec, -) { - for pending in in_flight.drain(..) { - let source_domain = pending.source_domain; - // All requests were dispatched up front via send_async, so waiting on - // each in turn does not change concurrency. - let response = - match futures::executor::block_on(services.http_client().wait(pending.pending)) { - Ok(response) => response, - Err(err) => { - log::warn!( - "Pull sync: request failed for partner '{}': {err:?}", - source_domain - ); - continue; - } - }; - - let Some(uid) = extract_pull_uid(response, &source_domain) else { - continue; - }; - - updates.push(PartnerIdUpdate::new(source_domain, uid)); - } -} - -/// Maximum response body size accepted from pull sync partners (64 KiB). -/// -/// The expected response is `{"uid":""}`, so 64 KiB is generous. -/// This prevents a misbehaving partner from exhausting WASM memory. -const MAX_PULL_RESPONSE_BYTES: usize = 64 * 1024; - -fn response_content_length_exceeds_limit(response: &PlatformResponse, source_domain: &str) -> bool { - let Some(value) = response.response.headers().get(header::CONTENT_LENGTH) else { - return false; - }; - - let Some(value) = value.to_str().ok() else { - log::warn!( - "Pull sync: partner '{}' returned invalid Content-Length header, rejecting", - source_domain - ); - return true; - }; - - let Ok(length) = value.parse::() else { - log::warn!( - "Pull sync: partner '{}' returned malformed Content-Length header, rejecting", - source_domain - ); - return true; - }; - - if length > MAX_PULL_RESPONSE_BYTES { - log::warn!( - "Pull sync: partner '{}' returned oversized Content-Length ({} bytes), rejecting", - source_domain, - length - ); - return true; - } - - false -} - -fn extract_pull_uid(response: PlatformResponse, source_domain: &str) -> Option { - let status = response.response.status(); - - if status == StatusCode::NOT_FOUND { - log::debug!( - "Pull sync: partner '{}' returned 404, treating as no-op", - source_domain - ); - return None; - } - - if !status.is_success() { - log::warn!( - "Pull sync: partner '{}' returned non-success status {}", - source_domain, - status - ); - return None; - } - - if response_content_length_exceeds_limit(&response, source_domain) { - return None; - } - - let body = response - .response - .into_body() - .into_bytes() - .unwrap_or_default(); - if body.len() > MAX_PULL_RESPONSE_BYTES { - log::warn!( - "Pull sync: partner '{}' returned oversized response ({} bytes), rejecting", - source_domain, - body.len() - ); - return None; - } - let payload = match serde_json::from_slice::(&body) { - Ok(payload) => payload, - Err(err) => { - log::warn!( - "Pull sync: partner '{}' returned invalid JSON body: {err}", - source_domain - ); - return None; - } - }; - - use super::kv_types::MAX_UID_LENGTH; - - let uid = payload.uid.filter(|value| !value.trim().is_empty()); - match uid { - None => { - log::debug!( - "Pull sync: partner '{}' returned null/empty uid, treating as no-op", - source_domain - ); - None - } - Some(ref value) if value.len() > MAX_UID_LENGTH => { - log::warn!( - "Pull sync: partner '{}' returned uid exceeding {} bytes (got {}), rejecting", - source_domain, - MAX_UID_LENGTH, - value.len() - ); - None - } - _ => uid, - } -} - #[cfg(test)] mod tests { use super::*; - use crate::consent::types::ConsentContext; - use crate::ec::kv_types::KvEntry; - use crate::platform::PlatformResponse; + use crate::ec::registry::PartnerConfig; use crate::redacted::Redacted; - use crate::test_support::tests::purpose_one_only_consent; - fn make_response(status: u16, body: &[u8]) -> PlatformResponse { - PlatformResponse::new( - edgezero_core::http::response_builder() - .status(status) - .body(EdgeBody::from(body.to_vec())) - .expect("should build test response"), - ) - } - - fn make_response_with_content_length( - status: u16, - content_length: usize, - body: &[u8], - ) -> PlatformResponse { - PlatformResponse::new( - edgezero_core::http::response_builder() - .status(status) - .header(header::CONTENT_LENGTH, content_length.to_string()) - .body(EdgeBody::from(body.to_vec())) - .expect("should build test response"), - ) - } - - fn pull_partner(ttl_sec: u64) -> PartnerConfig { + fn pull_partner() -> PartnerConfig { PartnerConfig { name: "SSP X".to_owned(), api_key_hash: Some("deadbeef".to_owned()), @@ -557,168 +74,20 @@ mod tests { pull_sync_enabled: true, pull_sync_url: Some("https://sync.partner.test/pull".to_owned()), pull_sync_allowed_domains: vec!["sync.partner.test".to_owned()], - pull_sync_ttl_sec: ttl_sec, - pull_sync_rate_limit: 20, ts_pull_token: Some(Redacted::new("token".to_owned())), + owner: None, } } - #[test] - fn build_pull_sync_context_returns_context_when_valid() { - let consent = ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, - ..ConsentContext::default() - }; - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); - let graph = KvIdentityGraph::in_memory("pull_store"); - ec_context.set_kv_snapshot(seed_present_snapshot(&graph, &ec_id)); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - - let context = build_pull_sync_context(&ec_context, ®istry) - .expect("should build pull sync context for valid EC"); - assert_eq!( - context.ec_id(), - ec_context.ec_value().expect("ec should be present"), - "should capture the EC ID from context" - ); - } - - #[test] - fn build_pull_sync_context_rejects_invalid_ec_id() { - let consent = ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, - ..ConsentContext::default() - }; - let ec_context = EcContext::new_for_test(Some("invalid-ec".to_owned()), consent); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - - let context = build_pull_sync_context(&ec_context, ®istry); - assert!( - context.is_none(), - "should reject pull sync context when EC ID format is invalid" - ); - } - - #[test] - fn build_pull_sync_context_skips_when_eid_persistence_is_denied() { - // Purpose 1 granted keeps the EC, but Purpose 4 denied withholds EID - // writes and leaves partner slots empty. Pull sync must not read those - // empty slots as eligible and disclose the EC ID to partners. - let consent = purpose_one_only_consent(); - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); - let graph = KvIdentityGraph::in_memory("pull_store"); - ec_context.set_kv_snapshot(seed_present_snapshot(&graph, &ec_id)); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - assert!( - ec_context.ec_allowed(), - "should keep the EC allowed when Purpose 1 is granted" - ); - - let context = build_pull_sync_context(&ec_context, ®istry); - - assert!( - context.is_none(), - "should skip pull sync when TCF Purpose 4 is denied" - ); - } - - #[test] - fn build_pull_sync_context_skips_empty_registry_and_complete_snapshot() { - let consent = ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, - ..ConsentContext::default() - }; - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); - let graph = KvIdentityGraph::in_memory("pull_store"); - let mut snapshot = seed_present_snapshot(&graph, &ec_id); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - - assert!( - build_pull_sync_context(&ec_context, &PartnerRegistry::empty()).is_none(), - "no pull partners should skip before graph construction" - ); - assert!( - build_pull_sync_context(&ec_context, ®istry).is_none(), - "an unread snapshot should skip before graph construction" - ); - - if let EcKvSnapshot::Present { entry, .. } = &mut snapshot { - entry.ids.insert( - "ssp.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "uid".to_owned(), - }, - ); - } - ec_context.set_kv_snapshot(snapshot); - assert!( - build_pull_sync_context(&ec_context, ®istry).is_none(), - "a complete snapshot should skip post-send work" - ); - } - - #[test] - fn partner_is_eligible_when_missing_from_entry() { - let partner = pull_partner(3600); - let entry = KvEntry::minimal("other_partner", "uid-1", 100); - - assert!( - is_partner_pull_eligible(&partner, Some(&entry)), - "should dispatch when partner has no stored UID" - ); - } - - #[test] - fn partner_is_not_eligible_when_already_present() { - let partner = pull_partner(3600); - let entry = KvEntry::minimal("ssp.example.com", "uid-1", 1000); - - assert!( - !is_partner_pull_eligible(&partner, Some(&entry)), - "should skip dispatch when partner already has a stored UID" - ); - } - - #[test] - fn completeness_requires_all_pull_partner_ids() { - let registry = PartnerRegistry::from_config(&[ - pull_enabled_ec_partner("a.example.com"), - pull_enabled_ec_partner("b.example.com"), - ]) - .expect("should build registry"); - let mut entry = KvEntry::minimal("a.example.com", "uid-a", 1_000); - - assert!( - !entry_is_pull_complete(&entry, ®istry), - "entry missing a pull partner ID should be incomplete" - ); - - entry.ids.insert( - "b.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "uid-b".to_owned(), - }, - ); - - assert!( - entry_is_pull_complete(&entry, ®istry), - "entry with every pull partner ID should be complete" - ); - } - #[test] fn validated_pull_sync_url_rejects_http_scheme() { - let mut partner = pull_partner(3600); + let mut partner = pull_partner(); partner.pull_sync_url = Some("http://sync.partner.test/pull".to_owned()); - let validated = validated_pull_sync_url(&partner); + let validated = validated_pull_sync_url( + partner.pull_sync_url.as_deref(), + &partner.pull_sync_allowed_domains, + ); assert!( validated.is_none(), "should reject pull_sync_url with HTTP scheme" @@ -727,10 +96,13 @@ mod tests { #[test] fn validated_pull_sync_url_rejects_non_allowlisted_host() { - let mut partner = pull_partner(3600); + let mut partner = pull_partner(); partner.pull_sync_url = Some("https://evil.test/pull".to_owned()); - let validated = validated_pull_sync_url(&partner); + let validated = validated_pull_sync_url( + partner.pull_sync_url.as_deref(), + &partner.pull_sync_allowed_domains, + ); assert!( validated.is_none(), "should reject runtime pull_sync_url host outside allowlist" @@ -739,11 +111,14 @@ mod tests { #[test] fn validated_pull_sync_url_accepts_normalized_allowlist_match() { - let mut partner = pull_partner(3600); + let mut partner = pull_partner(); partner.pull_sync_url = Some("https://SYNC.PARTNER.TEST./pull".to_owned()); partner.pull_sync_allowed_domains = vec!["sync.partner.test".to_owned()]; - let validated = validated_pull_sync_url(&partner); + let validated = validated_pull_sync_url( + partner.pull_sync_url.as_deref(), + &partner.pull_sync_allowed_domains, + ); assert!( validated.is_some(), "should accept allowlist match after hostname normalization" @@ -763,488 +138,4 @@ mod tests { "should not forward client IP to partners" ); } - - #[test] - fn pull_rate_limit_key_uses_ec_hash_only() { - let first_ec_id = format!("{}.ABC123", "a".repeat(64)); - let second_ec_id = format!("{}.XYZ789", "a".repeat(64)); - - let first_key = pull_rate_limit_key("ssp.example.com", &first_ec_id); - let second_key = pull_rate_limit_key("ssp.example.com", &second_ec_id); - - assert_eq!( - first_key, second_key, - "should bucket different suffixes for the same EC hash together" - ); - assert_eq!( - first_key, - format!("pull:ssp.example.com:{}", "a".repeat(64)), - "should key pull-sync rate limiting by source domain and EC hash" - ); - } - - #[test] - fn extract_pull_uid_treats_404_as_noop() { - let response = make_response(404, b""); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should treat 404 as no-op"); - } - - #[test] - fn extract_pull_uid_treats_uid_null_as_noop() { - let response = make_response(200, b"{\"uid\":null}"); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should treat uid=null as no-op"); - } - - #[test] - fn extract_pull_uid_rejects_oversized_uid() { - let long_uid = "x".repeat(513); - let body = format!("{{\"uid\":\"{long_uid}\"}}"); - let response = make_response(200, body.as_bytes()); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should reject uid exceeding 512 bytes"); - } - - #[test] - fn extract_pull_uid_reads_uid_from_success_body() { - let response = make_response(200, b"{\"uid\":\"abc123\"}"); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert_eq!( - uid.as_deref(), - Some("abc123"), - "should parse uid from 200 body" - ); - } - - #[test] - fn extract_pull_uid_rejects_oversized_content_length_before_body_read() { - let response = make_response_with_content_length( - 200, - MAX_PULL_RESPONSE_BYTES + 1, - b"{\"uid\":\"abc123\"}", - ); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!( - uid.is_none(), - "should reject oversized Content-Length before parsing body" - ); - } - - #[test] - fn extract_pull_uid_accepts_small_body_without_content_length() { - let response = make_response(200, b"{\"uid\":\"abc123\"}"); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert_eq!( - uid.as_deref(), - Some("abc123"), - "should accept small valid response without Content-Length" - ); - } - - #[test] - fn extract_pull_uid_rejects_body_larger_than_limit() { - let body = format!("{{\"uid\":\"{}\"}}", "x".repeat(MAX_PULL_RESPONSE_BYTES)); - let response = make_response(200, body.as_bytes()); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should reject body larger than limit"); - } - - #[test] - fn rotating_offset_distributes_partners_across_hours() { - // Simulate 3 partners sorted by source domain: alpha, beta, gamma. - let ids = vec!["alpha.example.com", "beta.example.com", "gamma.example.com"]; - - // Hour 0: offset = 0 % 3 = 0 → [alpha, beta, gamma] - let ts_h0: u64 = 100; // within hour 0 - let offset_h0 = (ts_h0 / 3600) as usize % ids.len(); - assert_eq!(offset_h0, 0, "hour 0 should start at index 0"); - - // Hour 1: offset = (3600 / 3600) % 3 = 1 → [beta, gamma, alpha] - let offset_h1 = (3600u64 / 3600) as usize % ids.len(); - assert_eq!(offset_h1, 1, "hour 1 should start at index 1"); - - // Hour 2: offset = (7200 / 3600) % 3 = 2 → [gamma, alpha, beta] - let offset_h2 = (7200u64 / 3600) as usize % ids.len(); - assert_eq!(offset_h2, 2, "hour 2 should start at index 2"); - - // Hour 3: offset = (10800 / 3600) % 3 = 0 → wraps back to [alpha, beta, gamma] - let offset_h3 = (10800u64 / 3600) as usize % ids.len(); - assert_eq!(offset_h3, 0, "hour 3 should wrap back to index 0"); - - // Verify rotate_left produces expected ordering - let mut rotated = ids.clone(); - rotated.rotate_left(offset_h1); - assert_eq!( - rotated, - vec!["beta.example.com", "gamma.example.com", "alpha.example.com"], - "hour 1 rotation should move beta to front" - ); - } - - // ----------------------------------------------------------------------- - // Snapshot-driven eligibility and request-wide aggregation - // ----------------------------------------------------------------------- - - use crate::ec::kv::TombstoneOutcome; - use crate::error::TrustedServerError; - use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; - use crate::settings::EcPartner; - use crate::test_support::tests::create_test_settings; - use error_stack::Report; - use std::sync::Arc; - - struct AllowAllRateLimiter; - - impl RateLimiter for AllowAllRateLimiter { - fn exceeded( - &self, - _key: &str, - _hourly_limit: u32, - ) -> Result> { - Ok(false) - } - } - - fn pull_enabled_ec_partner(source_domain: &str) -> EcPartner { - EcPartner { - name: format!("Partner {source_domain}"), - source_domain: source_domain.to_owned(), - openrtb_atype: EcPartner::default_openrtb_atype(), - bidstream_enabled: true, - api_token: Some(Redacted::new(format!( - "{source_domain}-api-token-32-bytes-minimum" - ))), - batch_rate_limit: EcPartner::default_batch_rate_limit(), - pull_sync_enabled: true, - pull_sync_url: Some(format!("https://{source_domain}/sync")), - pull_sync_allowed_domains: vec![source_domain.to_owned()], - pull_sync_ttl_sec: 3600, - pull_sync_rate_limit: 100, - ts_pull_token: Some(Redacted::new("outbound-token".to_owned())), - } - } - - fn snapshot_ec_id() -> String { - format!("{}.ABC123", "a".repeat(64)) - } - - fn seed_present_snapshot(graph: &KvIdentityGraph, ec_id: &str) -> EcKvSnapshot { - let mut entry = KvEntry::tombstone(1000); - entry.consent.ok = true; - graph.create(ec_id, &entry).expect("should seed live entry"); - graph.load_snapshot(ec_id) - } - - #[test] - fn dispatch_pull_sync_aggregates_batches_into_one_bulk_write() { - let mut settings = create_test_settings(); - // Force one partner per concurrency batch so responses span batches. - settings.ec.pull_sync_concurrency = 1; - let registry = PartnerRegistry::from_config(&[ - pull_enabled_ec_partner("alpha.example.com"), - pull_enabled_ec_partner("beta.example.com"), - ]) - .expect("should build pull registry"); - - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let snapshot = seed_present_snapshot(&graph, &ec_id); - - let stub = Arc::new(StubHttpClient::new()); - // One JSON response per partner, drained across two concurrency batches. - stub.push_response(200, br#"{"uid":"synced-uid"}"#.to_vec()); - stub.push_response(200, br#"{"uid":"synced-uid"}"#.to_vec()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - let (entry, generation) = graph - .get(&ec_id) - .expect("should read store") - .expect("entry should exist"); - assert_eq!( - entry.ids.get("alpha.example.com").map(|id| id.uid.as_str()), - Some("synced-uid"), - "first partner UID should persist" - ); - assert_eq!( - entry.ids.get("beta.example.com").map(|id| id.uid.as_str()), - Some("synced-uid"), - "second partner UID should persist" - ); - assert_eq!( - generation, 2, - "two partner responses across batches must persist in exactly one bulk write" - ); - } - - #[test] - fn dispatch_pull_sync_calls_and_persists_only_missing_partner() { - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = PartnerRegistry::from_config(&[ - pull_enabled_ec_partner("alpha.example.com"), - pull_enabled_ec_partner("beta.example.com"), - ]) - .expect("should build pull registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - graph - .create( - &ec_id, - &KvEntry::minimal("alpha.example.com", "existing-alpha", 1_000), - ) - .expect("should seed partial entry"); - let snapshot = graph.load_snapshot(&ec_id); - let stub = Arc::new(StubHttpClient::new()); - stub.push_response(200, br#"{"uid":"new-beta"}"#.to_vec()); - let services = build_services_with_http_client(stub.clone()); - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert_eq!( - stub.recorded_backend_names().len(), - 1, - "should call only the missing partner" - ); - let (entry, generation) = graph - .get(&ec_id) - .expect("should read store") - .expect("entry should exist"); - assert_eq!( - entry - .ids - .get("alpha.example.com") - .map(|partner_id| partner_id.uid.as_str()), - Some("existing-alpha"), - "existing UID should remain unchanged" - ); - assert_eq!( - entry - .ids - .get("beta.example.com") - .map(|partner_id| partner_id.uid.as_str()), - Some("new-beta"), - "missing UID should persist" - ); - assert_eq!(generation, 2, "missing UID should use one bulk CAS write"); - } - - #[test] - fn dispatch_pull_sync_skips_non_present_snapshots() { - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let stub = Arc::new(StubHttpClient::new()); - let services = build_services_with_http_client(stub.clone()); - - for snapshot in [ - EcKvSnapshot::NotRead, - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - EcKvSnapshot::Failed { - ec_id: ec_id.clone(), - }, - ] { - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - } - - assert!( - stub.recorded_backend_names().is_empty(), - "not-read, missing, and failed snapshots must not dispatch pull sync" - ); - assert!( - graph.get(&ec_id).expect("should read store").is_none(), - "no snapshot state should create a missing root" - ); - } - - #[test] - fn dispatch_pull_sync_skips_tombstone_snapshot() { - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(KvEntry::tombstone(1000)), - generation: Some(1), - }; - let stub = Arc::new(StubHttpClient::new()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert!( - stub.recorded_backend_names().is_empty(), - "a tombstone snapshot must not dispatch pull sync" - ); - } - #[test] - fn dispatch_pull_sync_skips_dispatch_when_ec_is_tombstoned_after_snapshot_capture() { - // The request snapshot authorized pull sync, then a concurrent request - // completed a CMP withdrawal while the page response was in flight. - // Pull sync runs post-send and discloses the raw `ec_id` to partners, so - // it must revalidate the live row and cancel rather than leak an - // identity the user just withdrew. - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let snapshot = seed_present_snapshot(&graph, &ec_id); - - // Concurrent withdrawal lands after the snapshot was captured. - assert_eq!( - graph - .write_withdrawal_tombstone(&ec_id, drop) - .expect("should tombstone the row"), - TombstoneOutcome::Written, - "should tombstone the row the snapshot still reports as present" - ); - - let stub = Arc::new(StubHttpClient::new()); - stub.push_response(200, br#"{"uid":"leaked-uid"}"#.to_vec()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert!( - stub.recorded_backend_names().is_empty(), - "a withdrawal that lands after snapshot capture must cancel post-send pull sync" - ); - let (entry, _) = graph - .get(&ec_id) - .expect("should read store") - .expect("tombstone should remain"); - assert!( - entry.ids.is_empty(), - "no partner UID may be written back onto a tombstoned row" - ); - } - - #[test] - fn dispatch_pull_sync_skips_revalidation_read_when_no_partner_is_eligible() { - // Every pull-enabled partner already has a UID in the request snapshot, - // so there is nothing to dispatch. The revalidation read exists to - // authorize outbound calls; with no calls to authorize it must not cost - // a KV operation. - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let mut entry = KvEntry::tombstone(1000); - entry.consent.ok = true; - entry.ids.insert( - "alpha.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "already-known".to_owned(), - }, - ); - graph - .create(&ec_id, &entry) - .expect("should seed live entry"); - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(entry), - generation: Some(1), - }; - - let stub = Arc::new(StubHttpClient::new()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert!( - stub.recorded_backend_names().is_empty(), - "a fully synced entry must not dispatch pull sync" - ); - } } diff --git a/crates/trusted-server-core/src/ec/pull_sync_marker.rs b/crates/trusted-server-core/src/ec/pull_sync_marker.rs deleted file mode 100644 index f2a2fe6013..0000000000 --- a/crates/trusted-server-core/src/ec/pull_sync_marker.rs +++ /dev/null @@ -1,693 +0,0 @@ -//! Short-lived browser marker for complete pull-sync partner state. - -use core::fmt; - -use edgezero_core::body::Body as EdgeBody; -use hmac::{Hmac, Mac as _}; -use http::{HeaderValue, Response, header}; -use sha2::{Digest as _, Sha256}; - -use crate::constants::COOKIE_TS_EC_PULL_COMPLETE; -use crate::redacted::Redacted; -use crate::settings::Settings; - -use super::pull_sync::entry_is_pull_complete; -use super::registry::PartnerRegistry; -use super::{EcKvSnapshot, current_timestamp}; - -type HmacSha256 = Hmac; - -const MARKER_VERSION: &str = "v1"; -const MARKER_KEY_LABEL: &[u8] = b"trusted-server/ec-pull-complete/key/v1"; -const MARKER_MAX_AGE_SECS: u64 = 60 * 60; -const MARKER_CLOCK_SKEW_SECS: u64 = 60; -const MAX_MARKER_LENGTH: usize = 256; - -/// Request-local validation state for the pull-sync completeness marker. -#[derive(Clone, Default)] -pub(crate) enum PullSyncMarkerState { - /// No marker cookie was present. - #[default] - Absent, - /// A marker was present but has not been checked against the active EC and partner set. - Unvalidated(Redacted), - /// A present marker failed validation or was disproved by authoritative KV state. - Invalid, - /// The marker is valid until the given Unix timestamp. - Valid { expires_at: u64 }, -} - -impl fmt::Debug for PullSyncMarkerState { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - Self::Absent => formatter.write_str("Absent"), - Self::Unvalidated(_) => formatter.write_str("Unvalidated()"), - Self::Invalid => formatter.write_str("Invalid"), - Self::Valid { expires_at } => formatter - .debug_struct("Valid") - .field("expires_at", expires_at) - .finish(), - } - } -} - -impl PullSyncMarkerState { - /// Creates marker state from an optional request cookie value. - #[must_use] - pub(crate) fn from_cookie(value: Option) -> Self { - value - .map(Redacted::new) - .map_or(Self::Absent, Self::Unvalidated) - } - - /// Returns whether a marker cookie was present on the request. - #[must_use] - pub(crate) fn was_present(&self) -> bool { - !matches!(self, Self::Absent) - } - - /// Returns whether the marker is currently valid. - #[must_use] - pub(crate) fn is_valid(&self) -> bool { - matches!(self, Self::Valid { .. }) - } - - /// Invalidates state bound to a replaced active EC ID. - pub(crate) fn invalidate_for_replaced_ec(&mut self) { - if self.was_present() { - *self = Self::Invalid; - } - } -} - -/// Validates an unvalidated marker against the active EC and current pull-partner set. -pub(crate) fn validate_marker_state( - state: &mut PullSyncMarkerState, - settings: &Settings, - registry: &PartnerRegistry, - ec_id: Option<&str>, -) { - let PullSyncMarkerState::Unvalidated(value) = state else { - return; - }; - let valid_until = ec_id.and_then(|ec_id| { - validate_marker( - value.expose(), - settings, - registry, - ec_id, - current_timestamp(), - ) - }); - *state = valid_until.map_or(PullSyncMarkerState::Invalid, |expires_at| { - PullSyncMarkerState::Valid { expires_at } - }); -} - -/// Reconciles browser marker state against finalized authoritative KV state. -pub(crate) fn reconcile_marker( - settings: &Settings, - registry: &PartnerRegistry, - ec_id: Option<&str>, - snapshot: &EcKvSnapshot, - state: &mut PullSyncMarkerState, - response: &mut Response, -) { - let pull_partners = sorted_pull_partner_domains(registry); - if pull_partners.is_empty() { - expire_if_present(state, response); - return; - } - - let Some(ec_id) = ec_id else { - expire_if_present(state, response); - return; - }; - - if !matches!(snapshot, EcKvSnapshot::NotRead) && !snapshot.belongs_to(ec_id) { - expire_if_present(state, response); - return; - } - - match snapshot { - EcKvSnapshot::Present { .. } => { - let entry = snapshot - .entry_for(ec_id) - .expect("snapshot binding should be checked before marker reconciliation"); - if !entry.consent.ok { - expire_if_present(state, response); - } else if entry_is_pull_complete(entry, registry) { - if !state.is_valid() { - set_marker(settings, registry, ec_id, state, response); - } - } else { - expire_if_present(state, response); - } - } - EcKvSnapshot::Missing { .. } => { - expire_if_present(state, response); - } - EcKvSnapshot::Failed { .. } | EcKvSnapshot::NotRead => { - if matches!(state, PullSyncMarkerState::Invalid) { - expire_if_present(state, response); - } - } - } -} - -/// Expires the completeness marker regardless of KV state. -pub(crate) fn expire_marker(state: &mut PullSyncMarkerState, response: &mut Response) { - append_cookie(response, &format_marker_cookie("", 0)); - *state = PullSyncMarkerState::Absent; -} - -fn set_marker( - settings: &Settings, - registry: &PartnerRegistry, - ec_id: &str, - state: &mut PullSyncMarkerState, - response: &mut Response, -) { - let now = current_timestamp(); - let expires_at = now.saturating_add(MARKER_MAX_AGE_SECS); - let Some(value) = create_marker(settings, registry, ec_id, expires_at) else { - return; - }; - append_cookie(response, &format_marker_cookie(&value, MARKER_MAX_AGE_SECS)); - *state = PullSyncMarkerState::Valid { expires_at }; -} - -fn expire_if_present(state: &mut PullSyncMarkerState, response: &mut Response) { - if state.was_present() { - expire_marker(state, response); - } -} - -fn append_cookie(response: &mut Response, value: &str) { - match HeaderValue::from_str(value) { - Ok(value) => { - response.headers_mut().append(header::SET_COOKIE, value); - } - Err(err) => { - log::warn!("Skipping pull-sync marker cookie: invalid header value: {err}"); - } - } -} - -fn format_marker_cookie(value: &str, max_age: u64) -> String { - // Keep this cookie host-only. The signed payload does not include the - // request host, so a Domain attribute would allow replay across sibling - // hosts that share the domain-wide EC cookie and passphrase. - format!( - "{COOKIE_TS_EC_PULL_COMPLETE}={value}; Path=/; Secure; SameSite=Lax; Max-Age={max_age}; HttpOnly" - ) -} - -#[cfg(test)] -pub(crate) fn create_marker_for_test( - settings: &Settings, - registry: &PartnerRegistry, - ec_id: &str, -) -> String { - create_marker_for_test_with_expiry( - settings, - registry, - ec_id, - current_timestamp().saturating_add(MARKER_MAX_AGE_SECS), - ) -} - -#[cfg(test)] -pub(crate) fn create_marker_for_test_with_expiry( - settings: &Settings, - registry: &PartnerRegistry, - ec_id: &str, - expires_at: u64, -) -> String { - create_marker(settings, registry, ec_id, expires_at) - .expect("should create marker for non-empty test registry") -} - -fn create_marker( - settings: &Settings, - registry: &PartnerRegistry, - ec_id: &str, - expires_at: u64, -) -> Option { - let fingerprint = partner_set_fingerprint(registry)?; - let payload = marker_payload(ec_id, expires_at, &fingerprint); - let key = marker_key(settings); - let mut mac = HmacSha256::new_from_slice(&key).expect("should create marker HMAC"); - mac.update(payload.as_bytes()); - let tag = hex::encode(mac.finalize().into_bytes()); - Some(format!("{MARKER_VERSION}.{expires_at}.{fingerprint}.{tag}")) -} - -fn validate_marker( - value: &str, - settings: &Settings, - registry: &PartnerRegistry, - ec_id: &str, - now: u64, -) -> Option { - if value.len() > MAX_MARKER_LENGTH { - return None; - } - - let mut segments = value.split('.'); - let version = segments.next()?; - let expires = segments.next()?; - let fingerprint = segments.next()?; - let tag = segments.next()?; - if segments.next().is_some() || version != MARKER_VERSION { - return None; - } - - let expires_at = expires.parse::().ok()?; - let max_future = now - .saturating_add(MARKER_MAX_AGE_SECS) - .saturating_add(MARKER_CLOCK_SKEW_SECS); - if expires_at <= now || expires_at > max_future { - return None; - } - - let expected_fingerprint = partner_set_fingerprint(registry)?; - if fingerprint != expected_fingerprint { - return None; - } - - let tag = hex::decode(tag).ok()?; - if tag.len() != 32 { - return None; - } - - let payload = marker_payload(ec_id, expires_at, fingerprint); - let key = marker_key(settings); - let mut mac = HmacSha256::new_from_slice(&key).expect("should create marker HMAC"); - mac.update(payload.as_bytes()); - mac.verify_slice(&tag).ok()?; - Some(expires_at) -} - -fn marker_key(settings: &Settings) -> [u8; 32] { - let mut mac = HmacSha256::new_from_slice(settings.ec.passphrase.expose().as_bytes()) - .expect("should create marker key HMAC"); - mac.update(MARKER_KEY_LABEL); - mac.finalize().into_bytes().into() -} - -fn marker_payload(ec_id: &str, expires_at: u64, fingerprint: &str) -> String { - format!("{MARKER_VERSION}\0{ec_id}\0{expires_at}\0{fingerprint}") -} - -fn partner_set_fingerprint(registry: &PartnerRegistry) -> Option { - let domains = sorted_pull_partner_domains(registry); - if domains.is_empty() { - return None; - } - - let mut hasher = Sha256::new(); - hasher.update(b"trusted-server/ec-pull-partners/v1\0"); - for domain in domains { - hasher.update((domain.len() as u64).to_be_bytes()); - hasher.update(domain.as_bytes()); - } - Some(hex::encode(hasher.finalize())) -} - -fn sorted_pull_partner_domains(registry: &PartnerRegistry) -> Vec { - let mut domains = registry - .pull_enabled_partners() - .into_iter() - .map(|partner| partner.source_domain.clone()) - .collect::>(); - domains.sort(); - domains -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::ec::kv_types::KvEntry; - use crate::redacted::Redacted; - use crate::settings::{EcPartner, Settings}; - use crate::test_support::tests::create_test_settings; - - const EC_ID: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ABC123"; - - fn pull_partner(source_domain: &str) -> EcPartner { - EcPartner { - name: format!("Partner {source_domain}"), - source_domain: source_domain.to_owned(), - openrtb_atype: EcPartner::default_openrtb_atype(), - bidstream_enabled: true, - api_token: Some(Redacted::new(format!( - "token-{source_domain}-32-bytes-minimum-value" - ))), - batch_rate_limit: EcPartner::default_batch_rate_limit(), - pull_sync_enabled: true, - pull_sync_url: Some(format!("https://sync.{source_domain}/pull")), - pull_sync_allowed_domains: vec![format!("sync.{source_domain}")], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), - ts_pull_token: Some(Redacted::new("pull-token".to_owned())), - } - } - - fn settings_and_registry(domains: &[&str]) -> (Settings, PartnerRegistry) { - let mut settings = create_test_settings(); - settings.ec.partners = domains.iter().map(|domain| pull_partner(domain)).collect(); - let registry = PartnerRegistry::from_config(&settings.ec.partners) - .expect("should build pull partner registry"); - (settings, registry) - } - - fn empty_response() -> Response { - Response::builder() - .status(200) - .body(EdgeBody::empty()) - .expect("should build response") - } - - fn live_snapshot(ec_id: &str, domains: &[&str]) -> EcKvSnapshot { - let mut entry = KvEntry::tombstone(1_000); - entry.consent.ok = true; - for domain in domains { - entry.ids.insert( - (*domain).to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: format!("uid-{domain}"), - }, - ); - } - EcKvSnapshot::Present { - ec_id: ec_id.to_owned(), - entry: Box::new(entry), - generation: Some(1), - } - } - - fn marker_cookies(response: &Response) -> Vec<&str> { - response - .headers() - .get_all(header::SET_COOKIE) - .iter() - .filter_map(|value| value.to_str().ok()) - .collect() - } - - fn assert_expired(state: &PullSyncMarkerState, response: &Response) { - assert!(matches!(state, PullSyncMarkerState::Absent)); - assert_eq!( - marker_cookies(response), - vec!["ts-ec-pull-complete=; Path=/; Secure; SameSite=Lax; Max-Age=0; HttpOnly"], - "should emit the exact host-only expiration cookie" - ); - } - - #[test] - fn marker_round_trip_binds_ec_and_partner_set() { - let (settings, registry) = settings_and_registry(&["a.example.com", "b.example.com"]); - let now = 1_000; - let marker = - create_marker(&settings, ®istry, EC_ID, now + 3_600).expect("should create marker"); - - assert_eq!( - validate_marker(&marker, &settings, ®istry, EC_ID, now), - Some(now + 3_600), - "should validate the issued marker" - ); - assert!( - validate_marker(&marker, &settings, ®istry, "wrong", now).is_none(), - "should reject a marker for another EC" - ); - } - - #[test] - fn marker_fingerprint_is_order_independent_and_set_sensitive() { - let (settings, first) = settings_and_registry(&["a.example.com", "b.example.com"]); - let (_, reordered) = settings_and_registry(&["b.example.com", "a.example.com"]); - let (_, changed) = settings_and_registry(&["a.example.com", "c.example.com"]); - let marker = create_marker(&settings, &first, EC_ID, 4_600).expect("should create marker"); - - assert!( - validate_marker(&marker, &settings, &reordered, EC_ID, 1_000).is_some(), - "config ordering should not change the marker" - ); - assert!( - validate_marker(&marker, &settings, &changed, EC_ID, 1_000).is_none(), - "partner-set changes should invalidate the marker" - ); - } - - #[test] - fn fingerprint_changes_for_enable_disable_add_and_remove() { - let (_, enabled_one) = settings_and_registry(&["a.example.com"]); - let (_, enabled_two) = settings_and_registry(&["a.example.com", "b.example.com"]); - let mut disabled_config = pull_partner("a.example.com"); - disabled_config.pull_sync_enabled = false; - let disabled = PartnerRegistry::from_config(&[disabled_config]) - .expect("should build disabled registry"); - let removed = PartnerRegistry::empty(); - - let base = partner_set_fingerprint(&enabled_one); - assert_ne!( - base, - partner_set_fingerprint(&enabled_two), - "adding a pull partner should change the fingerprint" - ); - assert_ne!( - base, - partner_set_fingerprint(&disabled), - "disabling a pull partner should change the fingerprint" - ); - assert_ne!( - base, - partner_set_fingerprint(&removed), - "removing a pull partner should change the fingerprint" - ); - assert_ne!( - partner_set_fingerprint(&disabled), - partner_set_fingerprint(&enabled_one), - "enabling a partner should change the fingerprint" - ); - } - - #[test] - fn reconcile_rejects_snapshot_bound_to_different_ec() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let snapshot = live_snapshot("different-ec", &["a.example.com"]); - let mut valid_state = PullSyncMarkerState::Valid { expires_at: 4_600 }; - let mut valid_response = empty_response(); - - reconcile_marker( - &settings, - ®istry, - Some(EC_ID), - &snapshot, - &mut valid_state, - &mut valid_response, - ); - assert_expired(&valid_state, &valid_response); - - let mut absent_state = PullSyncMarkerState::Absent; - let mut absent_response = empty_response(); - reconcile_marker( - &settings, - ®istry, - Some(EC_ID), - &snapshot, - &mut absent_state, - &mut absent_response, - ); - assert!(marker_cookies(&absent_response).is_empty()); - assert!(matches!(absent_state, PullSyncMarkerState::Absent)); - } - - #[test] - fn authoritative_incomplete_states_expire_valid_marker() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let mut tombstone = KvEntry::tombstone(1_000); - tombstone.ids.insert( - "a.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "stale".to_owned(), - }, - ); - let snapshots = [ - live_snapshot(EC_ID, &[]), - EcKvSnapshot::Missing { - ec_id: EC_ID.to_owned(), - }, - EcKvSnapshot::Present { - ec_id: EC_ID.to_owned(), - entry: Box::new(tombstone), - generation: Some(1), - }, - ]; - - for snapshot in snapshots { - let mut state = PullSyncMarkerState::Valid { expires_at: 4_600 }; - let mut response = empty_response(); - reconcile_marker( - &settings, - ®istry, - Some(EC_ID), - &snapshot, - &mut state, - &mut response, - ); - assert_expired(&state, &response); - } - } - - #[test] - fn non_authoritative_states_preserve_valid_fixed_expiry_marker() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let snapshots = [ - EcKvSnapshot::Failed { - ec_id: EC_ID.to_owned(), - }, - EcKvSnapshot::NotRead, - ]; - - for snapshot in snapshots { - let mut state = PullSyncMarkerState::Valid { expires_at: 4_600 }; - let mut response = empty_response(); - reconcile_marker( - &settings, - ®istry, - Some(EC_ID), - &snapshot, - &mut state, - &mut response, - ); - assert!(matches!( - state, - PullSyncMarkerState::Valid { expires_at: 4_600 } - )); - assert!(marker_cookies(&response).is_empty()); - } - } - - #[test] - fn steady_states_leave_the_marker_cookie_untouched() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let complete = live_snapshot(EC_ID, &["a.example.com"]); - let incomplete = live_snapshot(EC_ID, &[]); - let cases = [ - (PullSyncMarkerState::Valid { expires_at: 4_600 }, &complete), - (PullSyncMarkerState::Absent, &incomplete), - ]; - - for (mut state, snapshot) in cases { - let mut response = empty_response(); - reconcile_marker( - &settings, - ®istry, - Some(EC_ID), - snapshot, - &mut state, - &mut response, - ); - assert!( - marker_cookies(&response).is_empty(), - "a steady marker state should not add a Set-Cookie header" - ); - } - } - - #[test] - fn invalid_marker_is_cleared_without_authoritative_snapshot() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let mut state = PullSyncMarkerState::Invalid; - let mut response = empty_response(); - - reconcile_marker( - &settings, - ®istry, - Some(EC_ID), - &EcKvSnapshot::NotRead, - &mut state, - &mut response, - ); - - assert_expired(&state, &response); - } - - #[test] - fn marker_allows_sixty_seconds_of_issuer_clock_skew() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let marker = - create_marker(&settings, ®istry, EC_ID, 4_600).expect("should create marker"); - - assert_eq!( - validate_marker(&marker, &settings, ®istry, EC_ID, 940), - Some(4_600), - "a marker from an issuer sixty seconds ahead should remain valid" - ); - } - - #[test] - fn marker_rejects_expired_overlong_and_tampered_values() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let marker = - create_marker(&settings, ®istry, EC_ID, 4_600).expect("should create marker"); - let mut tampered = marker.clone(); - tampered.push('0'); - - assert!( - validate_marker(&marker, &settings, ®istry, EC_ID, 4_600).is_none(), - "expired marker should fail" - ); - assert!( - validate_marker(&marker, &settings, ®istry, EC_ID, 939).is_none(), - "marker beyond the lifetime and clock-skew allowance should fail" - ); - assert!( - validate_marker(&tampered, &settings, ®istry, EC_ID, 1_000).is_none(), - "tampering should fail" - ); - assert!( - validate_marker( - &"x".repeat(MAX_MARKER_LENGTH + 1), - &settings, - ®istry, - EC_ID, - 1_000 - ) - .is_none(), - "overlong marker should fail" - ); - } - - #[test] - fn marker_rejects_wrong_passphrase_and_empty_partner_set() { - let (settings, registry) = settings_and_registry(&["a.example.com"]); - let marker = - create_marker(&settings, ®istry, EC_ID, 4_600).expect("should create marker"); - let mut changed_settings = settings.clone(); - changed_settings.ec.passphrase = - Redacted::new("different-secret-key-32-bytes-minimum".to_owned()); - let empty = PartnerRegistry::empty(); - - assert!( - validate_marker(&marker, &changed_settings, ®istry, EC_ID, 1_000).is_none(), - "passphrase rotation should invalidate the marker" - ); - assert!( - create_marker(&settings, &empty, EC_ID, 4_600).is_none(), - "empty partner sets should not produce a marker" - ); - } - - #[test] - fn marker_cookie_is_host_only_and_secure() { - let cookie = format_marker_cookie("value", MARKER_MAX_AGE_SECS); - assert_eq!( - cookie, - "ts-ec-pull-complete=value; Path=/; Secure; SameSite=Lax; Max-Age=3600; HttpOnly" - ); - assert!(!cookie.contains("Domain="), "marker should be host-only"); - } -} diff --git a/crates/trusted-server-core/src/ec/rate_limiter.rs b/crates/trusted-server-core/src/ec/rate_limiter.rs index 5a5dac27e5..5fa335f20b 100644 --- a/crates/trusted-server-core/src/ec/rate_limiter.rs +++ b/crates/trusted-server-core/src/ec/rate_limiter.rs @@ -45,4 +45,20 @@ pub trait RateLimiter { // hourly API used by pixel sync. self.exceeded(key, per_minute_limit.saturating_mul(60)) } + + /// Adds one to the 60-second count for `key`, without checking a limit. + /// + /// The identity circuit breaker records one provider failure this way. + /// + /// # Errors + /// + /// Returns [`TrustedServerError`] on rate counter I/O failure. + fn record(&self, key: &str) -> Result<(), Report>; + + /// Returns the 60-second count for `key`, without changing it. + /// + /// # Errors + /// + /// Returns [`TrustedServerError`] on rate counter I/O failure. + fn count(&self, key: &str) -> Result>; } diff --git a/crates/trusted-server-core/src/ec/registry.rs b/crates/trusted-server-core/src/ec/registry.rs index c4637b4310..357ba4b53f 100644 --- a/crates/trusted-server-core/src/ec/registry.rs +++ b/crates/trusted-server-core/src/ec/registry.rs @@ -38,15 +38,10 @@ pub struct PartnerConfig { pub pull_sync_url: Option, /// Allowlist of domains TS may call for this partner's pull sync. pub pull_sync_allowed_domains: Vec, - /// Legacy seconds between pull sync refreshes retained for config compatibility. - /// - /// Pull sync currently fills missing partner UIDs only, so this value is - /// not used to refresh existing UIDs. - pub pull_sync_ttl_sec: u64, - /// Max pull sync calls per EC hash per partner per hour. - pub pull_sync_rate_limit: u32, /// Outbound bearer token for pull sync requests. pub ts_pull_token: Option>, + /// The identity provider that owns this source, when one does. + pub owner: Option, } /// In-memory partner registry with O(1) lookups by source domain and API key hash. @@ -103,14 +98,13 @@ impl PartnerRegistry { })); } - validate_rate_limits_values(partner.batch_rate_limit, partner.pull_sync_rate_limit) - .map_err(|error| { - Report::new(TrustedServerError::Configuration { - message: format!( - "ec.partners: invalid rate limits for '{normalized_source}': {error}" - ), - }) - })?; + validate_rate_limits_values(partner.batch_rate_limit).map_err(|error| { + Report::new(TrustedServerError::Configuration { + message: format!( + "ec.partners: invalid rate limits for '{normalized_source}': {error}" + ), + }) + })?; if partner.pull_sync_enabled { validate_pull_sync_fields( @@ -302,34 +296,24 @@ fn build_partner_config( pull_sync_enabled: partner.pull_sync_enabled, pull_sync_url: partner.pull_sync_url.clone(), pull_sync_allowed_domains: partner.pull_sync_allowed_domains.clone(), - pull_sync_ttl_sec: partner.pull_sync_ttl_sec, - pull_sync_rate_limit: partner.pull_sync_rate_limit, ts_pull_token: partner.ts_pull_token.clone(), + owner: partner.owner.clone(), } } fn validate_rate_limits(config: &PartnerConfig) -> Result<(), Report> { - validate_rate_limits_values(config.batch_rate_limit, config.pull_sync_rate_limit).map_err( - |message| { - Report::new(TrustedServerError::Configuration { - message: message.to_owned(), - }) - }, - ) + validate_rate_limits_values(config.batch_rate_limit).map_err(|message| { + Report::new(TrustedServerError::Configuration { + message: message.to_owned(), + }) + }) } -fn validate_rate_limits_values( - batch_rate_limit: u32, - pull_sync_rate_limit: u32, -) -> Result<(), &'static str> { +fn validate_rate_limits_values(batch_rate_limit: u32) -> Result<(), &'static str> { if batch_rate_limit == 0 { return Err("batch_rate_limit must be greater than 0"); } - if pull_sync_rate_limit == 0 { - return Err("pull_sync_rate_limit must be greater than 0"); - } - Ok(()) } @@ -429,8 +413,9 @@ mod tests { pull_sync_enabled: false, pull_sync_url: None, pull_sync_allowed_domains: vec![], - pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), - pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + removed_pull_sync_ttl_sec: None, + removed_pull_sync_rate_limit: None, + owner: None, ts_pull_token: None, } } @@ -669,17 +654,4 @@ mod tests { let result = PartnerRegistry::from_config(&[partner]); assert!(result.is_err(), "should reject zero batch_rate_limit"); } - - #[test] - fn zero_pull_sync_rate_limit_is_rejected() { - let mut partner = make_partner("pull.example.com", &valid_api_token("token-p")); - partner.pull_sync_enabled = true; - partner.pull_sync_url = Some("https://pull.example.com/sync".to_owned()); - partner.pull_sync_allowed_domains = vec!["pull.example.com".to_owned()]; - partner.pull_sync_rate_limit = 0; - partner.ts_pull_token = Some(Redacted::new("outbound-token".to_owned())); - - let result = PartnerRegistry::from_config(&[partner]); - assert!(result.is_err(), "should reject zero pull_sync_rate_limit"); - } } diff --git a/crates/trusted-server-core/src/http_util.rs b/crates/trusted-server-core/src/http_util.rs index 457e503568..beea21e6e4 100644 --- a/crates/trusted-server-core/src/http_util.rs +++ b/crates/trusted-server-core/src/http_util.rs @@ -78,6 +78,31 @@ pub fn sanitize_forwarded_headers(req: &mut Request) { } } +/// Returns whether the request is a top-level document navigation. +/// +/// True only for `Sec-Fetch-Dest: document`. Unlike [`is_navigation_request`] +/// there is no `Accept` fallback, because `fetch()` can set `Accept: text/html`. +#[must_use] +pub fn is_document_navigation(req: &Request) -> bool { + req.headers() + .get("sec-fetch-dest") + .and_then(|v| v.to_str().ok()) + .is_some_and(|dest| dest.trim().eq_ignore_ascii_case("document")) +} + +/// Returns true when the request advertises itself as a prefetch via either +/// the standard `Sec-Purpose` or the legacy `Purpose` header. +#[must_use] +pub fn is_prefetch_request(req: &Request) -> bool { + let header = |name: &str| { + req.headers() + .get(name) + .and_then(|v| v.to_str().ok()) + .is_some_and(|v| v.contains("prefetch")) + }; + header("sec-purpose") || header("purpose") +} + /// Returns `true` when the request looks like a top-level document navigation. /// /// Uses [`Sec-Fetch-Dest`] when available (preferred — it is a forbidden @@ -491,6 +516,38 @@ mod tests { use crate::settings::TrustedClientIpConfig; use http::{HeaderName, HeaderValue, Method}; + #[test] + fn document_navigation_requires_sec_fetch_dest_document() { + let with_dest = Request::builder() + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("should build the request"); + let accept_only = Request::builder() + .header(header::ACCEPT, "text/html") + .body(EdgeBody::empty()) + .expect("should build the request"); + let iframe = Request::builder() + .header("sec-fetch-dest", "iframe") + .body(EdgeBody::empty()) + .expect("should build the request"); + assert!( + is_document_navigation(&with_dest), + "should count a document destination" + ); + assert!( + !is_document_navigation(&accept_only), + "should not count the Accept fallback" + ); + assert!( + is_navigation_request(&accept_only), + "control: the broad check should still accept it" + ); + assert!( + !is_document_navigation(&iframe), + "should not count an iframe" + ); + } + fn build_request(method: Method, uri: &str) -> Request { Request::builder() .method(method) diff --git a/crates/trusted-server-core/src/integrations/datadome/protection.rs b/crates/trusted-server-core/src/integrations/datadome/protection.rs index a7c55bf864..09f374c1ec 100644 --- a/crates/trusted-server-core/src/integrations/datadome/protection.rs +++ b/crates/trusted-server-core/src/integrations/datadome/protection.rs @@ -237,6 +237,7 @@ impl DataDomeIntegration { certificate_check: true, first_byte_timeout: Duration::from_millis(u64::from(self.config.timeout_ms)), between_bytes_timeout: Duration::from_millis(u64::from(self.config.timeout_ms)), + connect_timeout: None, discriminator: None, }; diff --git a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs index 1447a83586..e31890f982 100644 --- a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs +++ b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs @@ -271,7 +271,7 @@ pub fn prepare_request( let cookie_state = console_cookie_state(request); let eligible_navigation = request.method() == Method::GET && is_navigation_request(request) - && !crate::publisher::is_prefetch_request(request) + && !crate::http_util::is_prefetch_request(request) && !crate::publisher::is_bot_user_agent(request); sanitize_console_cookie(request); diff --git a/crates/trusted-server-core/src/integrations/lockr.rs b/crates/trusted-server-core/src/integrations/lockr.rs index 002182bd13..2f86d30e40 100644 --- a/crates/trusted-server-core/src/integrations/lockr.rs +++ b/crates/trusted-server-core/src/integrations/lockr.rs @@ -201,7 +201,8 @@ impl LockrIntegration { .unwrap_or_default(); let target_url = format!("{}{}{}", self.config.api_endpoint, target_path, query); - log::info!("Forwarding to Lockr API: {}", target_url); + // The query string can carry identifiers, so only the path is logged. + log::info!("Forwarding to Lockr API path: {target_path}"); let request_body = if method == Method::POST { let bytes = @@ -227,8 +228,7 @@ impl LockrIntegration { .send(PlatformHttpRequest::new(target_req, backend_name)) .await .change_context(Self::error(format!( - "Failed to forward request to {}", - target_url + "Failed to forward request to Lockr API path {target_path}" )))? .response; @@ -578,6 +578,41 @@ mod tests { ); } + #[test] + fn lockr_proxy_log_omits_query() { + use crate::test_support::logs::{captured_logs, install_capturing_logger}; + + install_capturing_logger(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, b"ok".to_vec()); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let settings = create_test_settings(); + let integration = LockrIntegration::new(test_config()); + let req = http::Request::builder() + .method(HttpMethod::GET) + .uri("https://publisher.example/integrations/lockr/api/logpath/v1/token?uid=query-secret-4471&email=person-4471%40example.com") + .body(EdgeBody::empty()) + .expect("should build request"); + + futures::executor::block_on(integration.handle(&settings, &services, req)) + .expect("should proxy request"); + + let logs = captured_logs(); + assert!( + logs.iter() + .any(|line| line == "Forwarding to Lockr API path: /logpath/v1/token"), + "should log the upstream path: {logs:?}" + ); + assert!( + !logs + .iter() + .any(|line| line.contains("query-secret-4471") || line.contains("person-4471")), + "should log no part of the query string" + ); + } + #[test] fn lockr_proxy_forwards_body_and_strips_publisher_credentials() { // Regression guard for the upstream-rejection / credential-leak causes: diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index 820e2c24b8..ae193625d6 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -156,6 +156,7 @@ fn integration_backend_spec( certificate_check, first_byte_timeout, between_bytes_timeout: first_byte_timeout, + connect_timeout: None, // Distinguish this integration's backend from any other provider that // targets the same origin, so auction response correlation by backend // name cannot cross providers. diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index a7db8624e4..3eaeb57827 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -36,7 +36,7 @@ use crate::auction::types::{AuctionContext, AuctionRequest, MediaType}; use crate::auction::types::{AuctionResponse, Bid as AuctionBid}; use crate::cache_policy::{CacheControlPolicy, EdgeCacheHeader}; use crate::consent_config::ConsentForwardingMode; -use crate::cookies::{CONSENT_COOKIE_NAMES, strip_cookies}; +use crate::cookies::{CONSENT_COOKIE_NAMES, TS_IDENTITY_COOKIE_NAMES, strip_cookie_headers}; use crate::error::TrustedServerError; #[cfg(test)] use crate::http_util::RequestInfo; @@ -973,6 +973,9 @@ fn validate_external_bundle_config( pub struct PrebidIntegration { config: PrebidIntegrationConfig, + /// Every `[[ec.partners]]` source domain, sorted, for the shim's + /// `ts-eids` trim (spec 5.3). + registered_sources: Vec, planned_head_inserts: Option>, #[cfg(test)] legacy_config: Option, @@ -986,6 +989,7 @@ impl PrebidIntegration { let engine = Arc::new(BidParamOverrideEngine::try_from_config(&config)?); Ok(Arc::new(Self { config: PrebidIntegrationConfig::from(&config), + registered_sources: Vec::new(), planned_head_inserts: None, legacy_config: Some(config), engine, @@ -997,9 +1001,14 @@ impl PrebidIntegration { Self::try_new(config).expect("should compile prebid bid param overrides") } - fn for_browser_plan(config: &PrebidIntegrationConfig, plan: &AuctionPlan) -> Arc { + fn for_browser_plan( + config: &PrebidIntegrationConfig, + plan: &AuctionPlan, + registered_sources: Vec, + ) -> Arc { let mut integration = Self { config: config.clone(), + registered_sources, planned_head_inserts: None, #[cfg(test)] legacy_config: None, @@ -1123,6 +1132,7 @@ impl PrebidIntegration { } else { Vec::new() }), + registered_sources: &self.registered_sources, ..InjectedPrebidClientConfig::from(browser_config) }; let config_json = serialize_injected_prebid_config(&payload); @@ -1368,7 +1378,15 @@ pub fn register_for_plan( config.excluded_gam_ad_unit_path_suffixes = canonical; validate_browser_config_for_startup(&config, &settings.proxy.allowed_domains)?; validate_browser_bidder_ownership(&config, plan)?; - let integration = PrebidIntegration::for_browser_plan(&config, plan); + let registered_sources = settings + .ec + .partners + .iter() + .map(|partner| partner.source_domain.clone()) + .collect::>() + .into_iter() + .collect(); + let integration = PrebidIntegration::for_browser_plan(&config, plan, registered_sources); Ok(Some( IntegrationRegistration::builder(PREBID_INTEGRATION_ID) .with_proxy(integration.clone()) @@ -1582,6 +1600,9 @@ struct InjectedPrebidClientConfig<'a> { client_side_bidders: &'a [String], #[serde(skip_serializing_if = "<[String]>::is_empty")] excluded_gam_ad_unit_path_suffixes: &'a [String], + /// The source domains of `[[ec.partners]]`, nothing else from the table. + #[serde(skip_serializing_if = "<[String]>::is_empty")] + registered_sources: &'a [String], } impl<'a> From<&'a PrebidIntegrationConfig> for InjectedPrebidClientConfig<'a> { @@ -1599,6 +1620,7 @@ impl<'a> From<&'a PrebidIntegrationConfig> for InjectedPrebidClientConfig<'a> { server_side_bidders: None, client_side_bidders: &config.client_side_bidders, excluded_gam_ad_unit_path_suffixes: &config.excluded_gam_ad_unit_path_suffixes, + registered_sources: &[], } } } @@ -2026,8 +2048,14 @@ fn copy_request_headers( consent_forwarding: ConsentForwardingMode, client_ip: Option, ) { + let cookies: Vec = from + .headers() + .get_all(header::COOKIE) + .iter() + .cloned() + .collect(); apply_prebid_header_values( - from.headers().get(header::COOKIE), + &cookies, from.headers().get(header::USER_AGENT), from.headers().get(header::REFERER), from.headers().get(header::ACCEPT_LANGUAGE), @@ -2045,7 +2073,7 @@ pub(crate) fn apply_prebid_transport_headers( client_ip: Option, ) { apply_prebid_header_values( - from.cookie(), + from.cookies(), from.user_agent(), from.referer(), from.accept_language(), @@ -2060,7 +2088,7 @@ pub(crate) fn apply_prebid_transport_headers( reason = "the helper preserves four independently optional raw headers plus transport policy" )] fn apply_prebid_header_values( - cookie: Option<&HeaderValue>, + cookies: &[HeaderValue], user_agent: Option<&HeaderValue>, referer: Option<&HeaderValue>, accept_language: Option<&HeaderValue>, @@ -2085,27 +2113,18 @@ fn apply_prebid_header_values( .insert(header::HeaderName::from_static("x-forwarded-for"), value); } - let Some(cookie_value) = cookie else { - return; - }; - if !consent_forwarding.strips_consent_cookies() { - to.headers_mut() - .insert(header::COOKIE, cookie_value.clone()); - return; - } - match cookie_value.to_str() { - Ok(value) => { - let stripped = strip_cookies(value, CONSENT_COOKIE_NAMES); - if !stripped.is_empty() - && let Ok(cookie_header) = HeaderValue::from_str(&stripped) - { - to.headers_mut().insert(header::COOKIE, cookie_header); - } - } - Err(_) => { - to.headers_mut() - .insert(header::COOKIE, cookie_value.clone()); - } + // Identity cookies are stripped in every mode, before the consent mode is + // checked, so no mode forwards the raw header. + let mut cookie_headers = http::HeaderMap::new(); + for value in cookies { + cookie_headers.append(header::COOKIE, value.clone()); + } + strip_cookie_headers(&mut cookie_headers, TS_IDENTITY_COOKIE_NAMES); + if consent_forwarding.strips_consent_cookies() { + strip_cookie_headers(&mut cookie_headers, CONSENT_COOKIE_NAMES); + } + if let Some(cookie_header) = cookie_headers.remove(header::COOKIE) { + to.headers_mut().insert(header::COOKIE, cookie_header); } } @@ -4152,6 +4171,60 @@ server_url = "https://prebid.example/openrtb2/auction" "should inject the canonical suffix list: {config_insert}" ); } + #[test] + fn injected_config_lists_registered_sources() { + let mut settings = make_settings(); + settings + .integrations + .insert_config( + PREBID_INTEGRATION_ID, + &json!({ + "enabled": true, + "external_bundle_url": "https://assets.example/prebid/trusted-prebid.js" + }), + ) + .expect("should replace Prebid test configuration"); + settings.ec.partners = ["zeta.example.com", "alpha.example.com", "mid.example.com"] + .into_iter() + .map(|source| { + crate::ec::identity::test_support::ec_partner( + source, + "https://sync.example.com/pull", + "example-token-value-0123456789abcdef", + ) + }) + .collect(); + let plan = Arc::new( + crate::auction::compile_auction_plan(&settings).expect("should compile auction plan"), + ); + let registry = IntegrationRegistry::with_plan(&settings, plan) + .expect("should build integration registry"); + let document_state = IntegrationDocumentState::default(); + let context = IntegrationHtmlContext { + request_host: "pub.example", + request_scheme: "https", + origin_host: "origin.example", + document_state: &document_state, + }; + + let inserts = registry.head_inserts(&context); + let config_insert = inserts + .iter() + .find(|insert| insert.contains("window.__tsjs_prebid")) + .expect("should inject planned Prebid config"); + + assert!( + config_insert.contains( + r#""registeredSources":["alpha.example.com","mid.example.com","zeta.example.com"]"# + ), + "should list every partner source domain, sorted: {config_insert}" + ); + assert!( + !config_insert.contains("example-token") && !config_insert.contains("sync.example.com"), + "should carry nothing else from the partner table: {config_insert}" + ); + } + #[test] fn managed_user_ids_parse_with_opaque_params() { let config = parse_prebid_toml( @@ -9757,7 +9830,7 @@ set = { networkId = 42 } } #[test] - fn planned_transport_preserves_malformed_cookie_bytes_and_attested_xff() { + fn planned_transport_drops_malformed_cookie_bytes_and_uses_attested_xff() { let inbound = http::Request::builder() .uri("https://publisher.example/auction") .header( @@ -9796,13 +9869,9 @@ set = { networkId = 42 } routed.attested_client_ip(), ); - assert_eq!( - outbound - .headers() - .get(header::COOKIE) - .map(HeaderValue::as_bytes), - Some(b"session=fictional;\xffbroken".as_slice()), - "should forward malformed cookie bytes unchanged" + assert!( + outbound.headers().get(header::COOKIE).is_none(), + "should drop a non-UTF-8 Cookie field instead of forwarding it" ); assert_eq!( outbound @@ -9870,6 +9939,115 @@ set = { networkId = 42 } ); } + fn cookie_values(request: &http::Request) -> Vec { + request + .headers() + .get_all(header::COOKIE) + .iter() + .map(|value| value.to_str().expect("should be ASCII").to_owned()) + .collect() + } + + #[test] + fn prebid_headers_strip_identity_cookies_when_consent_cookies_are_forwarded() { + let cookies = [ + HeaderValue::from_static("ts-ec=abc; euconsent-v2=tcf; other=1"), + HeaderValue::from_static("__Host-ts-eids-s=eyJ2IjoxfQ; partner=p"), + HeaderValue::from_static("ts-eids=W10; second=2"), + ]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::Both, + None, + ); + + assert_eq!( + cookie_values(&to), + vec!["euconsent-v2=tcf; other=1; partner=p; second=2".to_owned()], + "should drop identity cookies, keep consent and unrelated cookies, and read every field" + ); + } + + #[test] + fn prebid_headers_strip_identity_and_consent_cookies_in_openrtb_only_mode() { + let cookies = [HeaderValue::from_static( + "ts-ec=abc; euconsent-v2=tcf; other=1", + )]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::OpenrtbOnly, + None, + ); + + assert_eq!( + cookie_values(&to), + vec!["other=1".to_owned()], + "should keep only other=1" + ); + } + + #[test] + fn prebid_headers_drop_non_utf8_cookie_fields_and_empty_results() { + let cookies = [ + HeaderValue::from_bytes(b"x=\xff").expect("should build an opaque header"), + HeaderValue::from_static("ts-ec=abc"), + ]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::Both, + None, + ); + + assert!( + to.headers().get(header::COOKIE).is_none(), + "should send no Cookie header" + ); + } + + #[test] + fn prebid_headers_drop_non_utf8_field_but_strip_identity_from_the_rest() { + let cookies = [ + HeaderValue::from_bytes(b"ts-ec=\xff; leaked=1") + .expect("should build an opaque header"), + HeaderValue::from_static("ts-ec=abc; ts-eids=W10; keep=1"), + ]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::Both, + None, + ); + + assert_eq!( + cookie_values(&to), + vec!["keep=1".to_owned()], + "should strip identity cookies from valid fields and drop the non-UTF-8 field" + ); + } + fn header_value<'a>(headers: &'a [(String, Vec)], name: &str) -> Option<&'a [u8]> { headers .iter() @@ -9904,7 +10082,7 @@ set = { networkId = 42 } ConsentForwardingMode::OpenrtbOnly, HeaderValue::from_bytes(b"session=fictional;\xffbroken") .expect("should build malformed cookie header"), - Some(b"session=fictional;\xffbroken".as_slice()), + None, ), ] { let stub = Arc::new(StubHttpClient::new()); diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index d3c09cd7d7..5098827c91 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -5,12 +5,17 @@ use std::sync::{Arc, Mutex, MutexGuard, PoisonError}; use async_trait::async_trait; use edgezero_core::body::Body as EdgeBody; use error_stack::Report; +use futures::StreamExt as _; use http::{Method, Request, Response}; use matchit::Router; use crate::auction::AuctionPlan; use crate::constants::HEADER_X_TS_EC; use crate::ec::EcContext; +use crate::ec::finalize::BrowserSeed; +use crate::ec::identity::coordinator::{CaptureCall, decode_body, has_capture_rule, run_capture}; +use crate::ec::identity::input::ThreadRandom; +use crate::ec::identity::{IdentityProvider, IdentityProviderError}; use crate::ec::kv::KvIdentityGraph; use crate::error::TrustedServerError; use crate::geo::GeoInfo; @@ -629,6 +634,7 @@ pub struct IntegrationRegistration { pub html_stream_processors: Vec>, pub head_injectors: Vec>, pub request_filters: Vec>, + pub identity_providers: Vec>, } impl IntegrationRegistration { @@ -655,6 +661,7 @@ impl IntegrationRegistrationBuilder { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters: Vec::new(), + identity_providers: Vec::new(), }, } } @@ -701,6 +708,34 @@ impl IntegrationRegistrationBuilder { self } + /// Registers an identity provider this integration contributes. + #[must_use] + pub fn with_identity_provider(mut self, provider: Arc) -> Self { + self.registration.identity_providers.push(provider); + self + } + + /// Registers a provider built by a fallible constructor. + /// + /// A failure becomes [`TrustedServerError::Integration`], naming the + /// integration the builder already holds. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::Integration`] when `provider` is an error. + pub fn try_with_identity_provider( + self, + provider: Result>, + ) -> Result> { + match provider { + Ok(provider) => Ok(self.with_identity_provider(Arc::new(provider))), + Err(error) => Err(Report::new(TrustedServerError::Integration { + integration: self.registration.integration_id.to_owned(), + message: error.current_context().to_string(), + })), + } + } + /// Mark this integration's JS module for deferred loading via /// `