Add design spec for server-side identity providers

CHANGELOG.md

@@ -9,6 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- Consent decoding for identity providers is stricter: a raw consent string that cannot be decoded blocks a provider whose policy is strict, and the GPP sharing, targeted-advertising and GPC fields of the US sections now block it too. A header-only GPP string with no applicable section does not block where TCF does not apply.
+- The Lockr proxy now logs only the path of the upstream URL, never its query string.
+- Identity KV is written at most once per EC-bearing request. Only a full page navigation from a returning visitor (the writer request) writes KV, once, after the response is sent: browser EIDs, pull partner results and the cluster size all go in that one write, with a generation check and no retry of the whole write. Every other route reads KV and stages but writes nothing. A consent-withdrawal tombstone and the removal-only retry of a failed write are the only exceptions. Mid-branch builds drop browser ingest until the whole change ships.
+- `POST /auction`, `GET /_ts/page-bids` and `GET /_ts/api/v1/identify` no longer write KV, and `/_ts/api/v1/identify` omits expired IDs. The `/identify` response shape is unchanged: one UID per source.
+- Pull partners now run from a visitor's second page load, from the writer request after send, and back off on errors. They are paced by `[ec.identity.providers.pull-<id>]` (`refresh_after_sec`, `max_calls_per_minute`), so the old default of 10 calls per hour becomes an effective 60 per hour. That 60 per hour is a ceiling, not a rate: `refresh_after_sec` and error backoff gate each partner first. Runtime startup now builds each pull partner's descriptor: a partner whose allowed host fails the descriptor host check stops startup. See [Identity Providers](docs/guide/configuration.md#identity-providers-ec-identity).
+- **Breaking:** `ec.pull_sync_concurrency`, and `pull_sync_ttl_sec` and `pull_sync_rate_limit` on `[[ec.partners]]`, are removed, and validation now rejects them with an error that names the replacement. Use `[ec.identity].max_concurrency` (the example config's value moves from 3 to the default 4), `refresh_after_sec`, and `max_calls_per_minute = ceil(N / 60)` for `pull_sync_rate_limit = N`, the last two in `[ec.identity.providers.<pull-id>]`.
+- Identity providers that resolve must now declare the TCF purposes their GVL entry lists (`ConsentPolicy::with_tcf_purposes`) and their GVL ID (`with_tcf_vendor`), or opt out of vendor consent with `without_tcf_vendor`; `ConsentPolicy::strict()` alone now requires only Purpose 1, and `build()` rejects an undeclared resolver.
+- A new visitor's first navigation that runs an auction now runs hold-mode identity providers that do not read the EC ID, and creates its row with their answers in the one create write, so a visitor who leaves after one page can still get a provider ID. It waits up to `hold_budget_ms`; without such a provider the create runs before routing as before.
+- A same-origin `/_ts/page-bids` call from a real browser now writes browser ingest and after-send provider results, at most once per `ec.identity.spa_writer_interval_sec` (default 300, `0` turns it off), so single-page apps do not wait for a full navigation.
+- A removal whose KV writes all fail now waits in a `pending:<ec_id>` KV key (one-day TTL) for the next writer navigation to merge, counted in `removal_deferred`; `removal_lost` counts only a removal that could not reach that key either. Positive IDs a capture sees on a request that is not the writer wait there too, so a capture's full entry set reaches KV. Every returning writer navigation reads that key once, a second KV read.
+- A global revoke (`revoke_scope = "global"`) now also sets an `HttpOnly` `ts-ec-revoked` cookie with a fingerprint of the visitor's consent, and no EC is minted until that consent changes, so a CMP that still grants does not undo the revoke on the next page.
+- `max_calls_per_ip_per_minute` (default 10) on each `[ec.identity.providers.<id>]` caps calls to that provider from one client address across visitors, next to the per-visitor cap.
+- `[ec.identity] max_backend_requests` (default 32, Fastly's per-execution limit) bounds the origin, auction and identity provider calls of one request together. A provider call past it is skipped and is not counted as a failure, so it never opens the provider's circuit breaker. Hold-mode calls also cap their connect timeout at their call timeout.
+- A provider opt-out clears on a TCF re-grant only when the opt-out was recorded under a TCF denial that is now granted; a later CMP re-save that was already granting no longer clears it. A clear records the opt-out it lifted rather than its own time, so it cannot lift an opt-out written after its request read the row.
+- `Resolves::with_refresh_grace` lets an identity provider without private state keep an expired ID it wrote for a source it owns as a refresh anchor until the expiry plus the grace. The ID is never served meanwhile.
+- With `[ec.identity] mirror_enabled = true` (off by default), `/auction` responses carry `ext.ts_eids_s` and `/_ts/page-bids` responses carry `ts_eids_s`: the payload (`value` with `max_age`, `clear`, or absent) for the client-only `__Host-ts-eids-s` mirror cookie, at most 2,048 bytes and one day. The shim now writes only `[[ec.partners]]` sources into `ts-eids`, merging each auction's entries per source instead of replacing the cookie, and no longer clears it when a request carries no EIDs (a TCF block still clears it).
+- The `ts-ec-pull-complete` cookie is removed: Trusted Server no longer sets or reads it, and a copy a browser holds expires within one hour.
+- **Breaking:** The KV identity row moves to schema version 2. A release before this one cannot read a version 2 row, so rolling back past this release is unsupported once it has written one. See [Rolling back](docs/guide/configuration.md#rolling-back). This release should merge after the EID-fidelity changes, whose behaviour its golden tests pin.
+- The `EID sync measurement: ...` log line is no longer logged for returning visitors; only a new EC still logs it. Each request now logs one `identity request measurement: ...` line after the response is sent, with the inbound `Cookie` header size, KV reads and writes by route kind, the flush outcome, lost removals and the after-send withdrawal run. It holds no identifiers. Operator dashboards that read the old line should read the new one.
+- **Breaking:** Prebid Server no longer receives any Trusted Server identity cookie (`ts-ec`, `ts-eids`, and the `__Host-ts-eids-s` mirror that this release's shim writes); today it receives `ts-ec` and `ts-eids`. The publisher origin still receives `ts-ec` and `ts-eids` but never the mirror. `[ec] strip_eids_cookie_from_origin = true` strips `ts-eids` from origin requests too.
 - **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER__GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header.
 - **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias.
 - The S2S `/_ts/api/v1/batch-sync` endpoint now validates the full batch and calls the CAS-protected update path once per distinct normalized EC ID. The last valid UID wins within a group, and infrastructure failures reject the failing and each unprocessed group, so accepted and `kv_unavailable` input indexes may interleave.
@@ -44,6 +64,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Server-side identity provider framework. An integration registers an `IdentityProvider` with `try_with_identity_provider`, and the coordinator handles consent, transport limits, backoff and the single KV write. Provider authors get a test harness (`ProviderHarness`, `check_conformance`, `output_eids`). See [Writing an Identity Provider](docs/guide/identity-providers.md). No vendor ships with the framework.
+- Source ownership: `owner` on an `[[ec.partners]]` entry names the provider that owns a source. An owner's result replaces the source's whole entry set, and batch sync writes an owned source only when the row holds no value for it (an expired owned record counts as present and is not filled). Validation rejects an owner that is not enabled, is capture-only, does not claim the source, or collides with another provider. See [Source ownership](docs/guide/configuration.md#source-ownership-ec-partners-owner).
+- Hold mode: `mode = "hold"` in `[ec.identity.providers.<id>]` runs a provider before the navigation's auction, within the shared `hold_budget_ms`, with `timeout_ms` and `after_send_timeout_ms` as call timeouts.
+- Revoke-only passive capture for providers that proxy a vendor's API: a captured removal or opt-out is written to KV, a positive result clears an opt-out, and positive IDs are only counted. `revoke_scope` (`provider` by default, or `global` to run the consent-withdrawal tombstone) sets how far an opt-out reaches. Removals and opt-outs apply from any client, as consent withdrawals do; only a clear needs a real browser. A capture-only provider must list its `sources`. See the [capture provider checklist](docs/guide/edge-cookies.md#capture-provider-checklist).
+- Complete provider EIDs: a provider can return several entries and UIDs per source, with UID-level `atype` and `ext` and EID-level `inserter`, `matcher`, `mm` and `ext`. KV, the bidstream, `x-ts-eids` and the `ts-eids-s` mirror carry the whole set in order; `/identify` still returns one UID per source. See [Complete provider EIDs](docs/guide/edge-cookies.md#complete-provider-eids).
+- `/_ts/api/v1/identify` returns an ID a server-side provider wrote only to a browser call from the publisher's site (an allowed `Origin` and `Sec-Fetch-Site: same-origin` or `same-site`), because the EC ID is a bearer value partners and the origin hold. Other IDs are returned as before.
+- The `capture_unpersisted` and `removal_lost` counters in the `identity request measurement: ...` log line, and a per-provider circuit breaker (`breaker_failures_per_minute`) that skips a provider for every visitor after repeated transport failures, timeouts, 429s or 5xx responses.
+
 - Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `<base>` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
 - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary.
 - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape.

crates/trusted-server-adapter-axum/src/app.rs

@@ -18,6 +18,7 @@ use trusted_server_core::ec::EcContext;
 use trusted_server_core::ec::admin::{
     admin_ec_lookup_not_supported, deny_admin_diagnostic_fallback, handle_admin_eids_lookup,
 };
+use trusted_server_core::ec::finalize::BrowserSeed;
 use trusted_server_core::ec::registry::PartnerRegistry;
 use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError};
 use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput};
@@ -232,6 +233,8 @@ async fn dispatch_fallback(
                 kv: None,
                 ec_context: &mut ec_context,
                 services,
+                is_real_browser: false,
+                seed: &BrowserSeed::empty(),
                 req,
             })
             .await
@@ -325,7 +328,7 @@ const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[
     Method::DELETE,
 ];
 
-fn named_routes() -> [NamedRoute; 17] {
+fn named_routes() -> [NamedRoute; 18] {
     [
         NamedRoute {
             path: "/.well-known/trusted-server.json",
@@ -371,6 +374,13 @@ fn named_routes() -> [NamedRoute; 17] {
             primary_methods: &[Method::GET],
             handler: NamedRouteHandler::AdminEcNotSupported,
         },
+        // The provider opt-out clear writes the EC identity graph, so it
+        // answers like the lookup: not available on this adapter.
+        NamedRoute {
+            path: "/_ts/admin/ec/{id}/providers/{provider}/opt-out",
+            primary_methods: &[Method::DELETE],
+            handler: NamedRouteHandler::AdminEcNotSupported,
+        },
         // Admin EIDs echo: pure request inspection (no KV), so the dev
         // server serves the real handler.
         NamedRoute {

crates/trusted-server-adapter-axum/src/platform.rs

@@ -666,6 +666,7 @@ mod tests {
             certificate_check: true,
             first_byte_timeout: Duration::from_secs(15),
             between_bytes_timeout: Duration::from_secs(15),
+            connect_timeout: None,
             host_header_override: None,
             discriminator: None,
         };
@@ -687,6 +688,7 @@ mod tests {
             certificate_check: true,
             first_byte_timeout: Duration::from_secs(15),
             between_bytes_timeout: Duration::from_secs(15),
+            connect_timeout: None,
             host_header_override: None,
             discriminator: None,
         };

crates/trusted-server-adapter-axum/tests/routes.rs

@@ -114,6 +114,7 @@ fn all_explicit_routes_are_registered() {
         ("POST", "/_ts/admin/keys/deactivate"),
         ("GET", "/_ts/admin/ec"),
         ("GET", "/_ts/admin/ec/{id}"),
+        ("DELETE", "/_ts/admin/ec/{id}/providers/{provider}/opt-out"),
         ("GET", "/_ts/admin/eids"),
         ("POST", "/admin/keys/rotate"),
         ("POST", "/admin/keys/deactivate"),
@@ -388,6 +389,55 @@ async fn authenticated_admin_ec_routes_return_501() {
     }
 }
 
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn authenticated_admin_opt_out_clear_route_returns_501() {
+    // The clear writes the EC identity graph, which is Fastly KV backed, so
+    // the Axum dev server answers the DELETE locally with 501 like the EC
+    // lookup.
+    let path = format!(
+        "/_ts/admin/ec/{}.abc123/providers/pull-ssp-example-com/opt-out",
+        "a".repeat(64)
+    );
+    let send = |method: &'static str| {
+        let path = path.clone();
+        async move {
+            let req = Request::builder()
+                .method(method)
+                .uri(&path)
+                .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==")
+                .body(AxumBody::empty())
+                .expect("should build request");
+            make_service()
+                .ready()
+                .await
+                .expect("should be ready")
+                .call(req)
+                .await
+                .expect("should respond")
+        }
+    };
+
+    let deleted = send("DELETE").await;
+    assert_eq!(
+        deleted.status().as_u16(),
+        501,
+        "DELETE {path} should report that the opt-out clear is unsupported"
+    );
+    assert_eq!(
+        deleted
+            .headers()
+            .get("content-type")
+            .and_then(|v| v.to_str().ok()),
+        Some("application/json"),
+        "should answer with the same JSON as the EC lookup"
+    );
+    assert_eq!(
+        send("GET").await.status().as_u16(),
+        404,
+        "control: GET {path} should be denied locally as an unknown admin route"
+    );
+}
+
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn admin_ec_route_without_credentials_returns_401() {
     let mut svc = make_service();

crates/trusted-server-adapter-cloudflare/src/app.rs

@@ -22,6 +22,7 @@ use trusted_server_core::ec::admin::{
     admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported,
     deny_admin_diagnostic_fallback, handle_admin_eids_lookup,
 };
+use trusted_server_core::ec::finalize::BrowserSeed;
 use trusted_server_core::ec::registry::PartnerRegistry;
 use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError};
 use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput};
@@ -544,6 +545,8 @@ fn build_router(state: &Arc<AppState>) -> RouterService {
                         kv: None,
                         ec_context: &mut ec_context,
                         services: &services,
+                        is_real_browser: false,
+                        seed: &BrowserSeed::empty(),
                         req,
                     })
                     .await
@@ -643,6 +646,14 @@ fn build_router(state: &Arc<AppState>) -> RouterService {
             .get("/_ts/admin/ec/{id}", |_ctx: RequestContext| async {
                 Ok::<Response, EdgeError>(admin_ec_lookup_not_supported())
             })
+            // The provider opt-out clear writes the EC identity graph, so it
+            // answers like the lookup.
+            .delete(
+                "/_ts/admin/ec/{id}/providers/{provider}/opt-out",
+                |_ctx: RequestContext| async {
+                    Ok::<Response, EdgeError>(admin_ec_lookup_not_supported())
+                },
+            )
             // Admin EIDs echo: pure request inspection (no KV), so this
             // adapter serves the real handler.
             .get(