From 21fd17269d63717e25804c3a24d39466f61b01eb Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:25:06 +0530 Subject: [PATCH 01/33] Add implementation plan for server-side identity providers Signed-off-by: dhruv8sh --- ...26-10-22-server-side-identity-providers.md | 7442 +++++++++++++++++ 1 file changed, 7442 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-22-server-side-identity-providers.md diff --git a/docs/superpowers/plans/2026-10-22-server-side-identity-providers.md b/docs/superpowers/plans/2026-10-22-server-side-identity-providers.md new file mode 100644 index 0000000000..4ce4338879 --- /dev/null +++ b/docs/superpowers/plans/2026-10-22-server-side-identity-providers.md @@ -0,0 +1,7442 @@ +# Server-side Identity Providers Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let Trusted Server call identity providers (ID5, LiveRamp ATS, Lockr, UID2 and generic pull partners) from the edge through one `IdentityProvider` trait and one coordinator. Every page load makes at most one browser-driven KV write. Provider IDs are cached first in a sealed first-party cookie (`__Host-ts-eids-s`) and then in KV, and removals always win. + +**Architecture:** New code lives in `crates/trusted-server-core/src/ec/identity/`. Providers are pure. They describe themselves with a `ProviderDescriptor`, build `ProviderRequest`s and parse buffered `ProviderResponse`s into `Resolution`s. The `IdentityCoordinator` owns eligibility, consent, transport limits, backoff and turning results into changes. Each request is either the **writer** (a real-browser top-level document navigation) or a **stager**. The writer adds every KV change to one request-scoped `IdentityMutation` and applies it with one conditional write after the response is sent. Stagers never write KV. They save positive changes in `__Host-ts-eids-s` and removals in per-batch `__Host-ts-eids-r-` cookies, and the next writer flushes them. Today's pull sync becomes `GenericPullProvider`. + +**Tech Stack:** Rust 1.95 (edition 2024), `trusted-server-core`, `trusted-server-adapter-fastly`, `error-stack`, `derive_more`, `serde`/`serde_json`, `hmac`/`sha2`, `base64`, `rand`, `flate2`/`brotli` (already dependencies), and a new `aes-gcm = "0.10"`. Fastly Compute (Viceroy for tests). + +**Spec:** `docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md` (revision 11, branch `spec/1246-server-side-identity-providers`). Read it before starting. "Spec 5.4" means section 5.4 of that file. Where this plan and the spec disagree, the spec wins. Stop and report the disagreement instead of choosing one. The spec cites `file:line` at commit `c84a73da7`. Line numbers have drifted slightly since then, so find code by the function names given here, not by line number. + +## How this plan is split + +The spec's rollout (spec 14) has three steps, and each step can ship on its own. This plan has one **Part** per step, and each Part is one pull request. Each PR's branch is stacked on the previous one: + +| Part | Spec step | Branch | Base | Tasks | +| ---- | ------------------------ | ---------------------------------- | ------------------------------------------ | -------- | +| 0 | 0. Strips first | `feat/1246-identity-cookie-strips` | `spec/1246-server-side-identity-providers` | 1 to 4 | +| 1 | 1. One write, no vendors | `feat/1246-identity-single-write` | `feat/1246-identity-cookie-strips` | 5 to 31 | +| 2 | 2. Providers | `feat/1246-identity-providers` | `feat/1246-identity-single-write` | 32 to 41 | + +Part 0 is the rollback floor (spec 14). Do not start Part 1 until Part 0's PR is open. Do not start Part 2 until Part 1's one-write invariant tests (Task 30) pass. + +## Global Constraints + +Copied from the spec and `AGENTS.md`. Every task implicitly includes these. + +- Follow `AGENTS.md` for commands, style, error handling and CI gates. Bare `cargo test`, `cargo build` and `cargo check` fail at the workspace root by design. Use the aliases (`cargo test-fastly`, `cargo clippy-fastly`, and the others). +- Errors use `error-stack` `Report` with `derive_more::Display` enums and `impl core::error::Error`. No `anyhow` and no `thiserror`. Use `expect("should ...")`, never `unwrap()`, in production and in tests. +- No local imports inside functions. No wildcard imports except `use super::*` in test modules. Comments go on their own line, above the code. +- Every public item has a doc comment, and every fallible public function has `# Errors`. Every builder method and getter is `#[must_use]`. Spec 4.2, "Conventions for the listings", gives the derive list for each type. Copy it exactly. +- Never log UIDs, envelopes, HEMs, provider state, consent strings, cookie values, or provider URLs with a query string. EC IDs go through `log_id`. Log provider errors with `current_context()` only, never `{:?}` of the report (spec 9.3). +- Use only example or fictional data in code, tests, comments and docs (`example.com` hosts). Public protocol source domains (`id5-sync.com`, `liveramp.com`, `uidapi.com`, `sharedid.org`) may appear where the spec uses them. +- **Fixed values from the spec. Do not change them:** + - `__Host-ts-eids-s`: sealed value at most **2,048** bytes. `__Host-ts-eids-r-<8 hex>`: sealed value at most **512** bytes, normally at most **4** cookies. Both: `Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=31536000`, with no `Domain`. + - Seal: AES-256-GCM, value `v1..`, base64url without padding, 12-byte nonce, the full EC ID as associated data. Key labels: `trusted-server/ts-eids-s/key/v1` and `trusted-server/ts-eids-r/key/v1`, each derived with HMAC-SHA256 from `ec.passphrase`. + - KV: `ext` at most **256** serialized bytes, `ps.st` at most **1,024** bytes of embedded JSON, at most **32** `ps` entries, at most **4** `ps.iv` hashes, `rv` is **16** hex digits (64 random bits). `SCHEMA_VERSION` stays **1**. + - `ProviderRequest` call context: at most **1,024** bytes of JSON. `IdentityProviderId`: `^[a-z][a-z0-9_-]{0,62}$`. + - `[ec.identity]` defaults: `max_concurrency = 4`, `hold_budget_ms = 100`, `hold_retry_sec = 300`, `max_response_bytes = 65536`, `pending_grace_sec = 300`, `staged_max_age_sec = 604800`, `kv_revalidate_sec = 3600`. Per-provider defaults: `mode = "after_send"`, `timeout_ms` = `hold_budget_ms` in hold mode and `1000` after send, `refresh_after_sec = 86400`, `max_calls_per_minute = 1`, `breaker_failures_per_minute = 50`, `cookie_priority = 0`. + - Backoff (spec 8): a `no_match` waits `n = now + retry_in`, default 24 h and at most 7 d. An input-caused failure waits `min(1 h × 2^(f-1), 24 h)`. A transport, 429 or 5xx failure waits `min(5 min × 2^(f-1), 1 h)`. A cut-off hold call waits `hold_retry_sec`. A skip waits `refresh_after_sec`. A marked provider that was not dispatched gets `ps.n = max(ps.n, now + 60 s)`. +- **G4 is strict.** A page load makes at most one browser-driven KV write and never retries it. The only exceptions are the consent-withdrawal tombstone path (unchanged) and the removal overflow write (Task 27). +- **Removals win.** No filter (consent re-gate, staleness, ownership, `cf` mismatch, size cap) may drop a removal, an opt-out or an `iv` hash. Only the EC binding and, when present, the revision check apply to them (spec 5.6). +- **Keep `ts-eids` and `x-ts-eids`.** Do not remove either (product requirement). +- **Feature gate for test support.** The spec names a `test-support` feature. `trusted-server-core` already has a `test-utils` feature for the same purpose (`crates/trusted-server-core/Cargo.toml`). Reuse `test-utils` and gate on `#[cfg(any(test, feature = "test-utils"))]`. This is the only intended deviation from the spec. Mention it in the PR description. +- Test discipline: + 1. **Red first.** Run each new test against unchanged code and watch it fail for the expected reason before you implement. + 2. **Pair absence with presence.** Each "X is gone" assertion sits next to a "control Y is still there" assertion on the same value. + 3. Compare structured outputs whole (full `KvEntry`, full `serde_json::Value`). Do not use `..Default::default()` in exhaustiveness tests. List every field. + 4. Use `json!` for JSON fixtures, except where exact byte lengths matter. +- Commits: sentence case, imperative mood, no `fix:`/`feat:` prefixes, no bracketed tags, `git commit --signoff -S`, one `-m` flag per paragraph, no heredoc, no AI attribution lines. + +## Test commands + +| What | Command | +| --------------------------------------- | --------------------------------------------------------------------------------------------- | +| Core and Fastly adapter (Viceroy) | `cargo test-fastly` | +| One core test by name | `cargo test-fastly -- ` | +| Fastly adapter with reusable sandbox | `cargo test-fastly-reuse` | +| Other adapters (Part 0 strips, Task 31) | `cargo test-axum && cargo test-cloudflare && cargo test-spin` | +| Lint, Fastly target | `cargo clippy-fastly` | +| Docs | `cargo doc --no-deps --all-features -p trusted-server-core` | +| Parity crate | `cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity` | +| CLI (config validation, Task 33) | `./scripts/test-cli.sh` | +| Full gate before each PR | `/check-ci` | + +`cargo test-fastly -- ` runs only the tests whose path contains ``. Every "Run" step below gives the exact filter. + +## File map + +| File | Responsibility | Tasks | +| ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------- | +| `Cargo.toml` (workspace), `crates/trusted-server-core/Cargo.toml` | `aes-gcm` dependency | 5 | +| `crates/trusted-server-core/src/constants.rs`, `src/cookies.rs` | Cookie names, identity cookie lists, name-and-prefix strip | 1 | +| `crates/trusted-server-core/src/auction/routing.rs`, `src/integrations/prebid.rs` | Strip identity cookies from Prebid Server requests | 2 | +| `crates/trusted-server-core/src/publisher.rs` | Origin strip; `is_prefetch_request` move; page-bids stops ingest; navigation merge; hold mode | 3, 23, 26, 28, 36, 37 | +| `crates/trusted-server-core/src/cookies/template_cache_policy.rs` | New cookies refused as cache keys | 4 | +| `crates/trusted-server-core/src/ec/identity/mod.rs`, `ids.rs`, `resolution.rs` (new) | Module, ID newtypes, error type, results | 5, 10 | +| `crates/trusted-server-core/src/ec/kv_types.rs` | `KvPartnerId` metadata, `KvProviderState`, `KvEntry.ps`, validation | 6 | +| `crates/trusted-server-core/src/ec/kv.rs` | Constructors; `PartnerIdUpdate.acquired_at`; `compute_cluster_size`; removal overflow write; counting store | 6, 7, 22, 24, 27 | +| `crates/trusted-server-core/src/ec/admin.rs` | Redact `ps.st`; servability; operator opt-out clear | 6, 14, 36 | +| `crates/trusted-server-core/src/ec/identity/descriptor.rs` (new) | Descriptor, builder, `Resolves`, `Captures`, `ConsentPolicy`, `EidProvenance`, `Input`, `Access`, `ResolveKind`, `UsOptOut` | 5, 8 | +| `crates/trusted-server-core/src/ec/identity/input.rs` (new) | `ResolveInput`, `CaptureInput`, `StoredId`, `Region`, `HashedEmail`, `ProviderConsent`, randomness | 9 | +| `crates/trusted-server-core/src/ec/identity/exchange.rs` (new) | `ProviderRequest`, `ProviderResponse` | 10 | +| `crates/trusted-server-core/src/ec/identity/provider.rs` (new) | `IdentityProvider`, `IdentityProviderConfig` | 11 | +| `crates/trusted-server-core/src/ec/identity/policy.rs` (new), `src/consent/*` | Policy evaluation, GPP opt-outs, `stricter_of`, fingerprint, redacting `Debug` | 12, 34 | +| `crates/trusted-server-core/src/ec/identity/revision.rs` (new) | `IdentityKeys`, `Revision`, keyed hashes | 13 | +| `crates/trusted-server-core/src/ec/identity/catalog.rs`, `servable.rs` (new) | Provider catalog, ownership, `cf`; the servability check | 14, 22, 32, 33 | +| `crates/trusted-server-core/src/ec/eids.rs`, `ec/identify.rs` | Servability for KV IDs; `/identify` stops writing | 14, 24 | +| `crates/trusted-server-core/src/ec/identity/mutation.rs` (new) | `IdentityMutation` and its apply algorithm | 15 | +| `crates/trusted-server-core/src/ec/identity/cookie.rs` (new) | Seal and open, `ts-eids-s`, size cap, KV merge, `ts-eids-r` | 16, 17, 18 | +| `crates/trusted-server-core/src/ec/identity/config.rs` (new), `src/settings.rs`, `src/config.rs` | `[ec.identity]`, `owner`, deploy and runtime validation | 19, 33 | +| `crates/trusted-server-core/src/ec/identity/coordinator.rs` (new) | Eligibility, outcomes, dispatch, breaker, hold, capture, re-grants | 20, 21, 35, 36, 37, 38 | +| `crates/trusted-server-core/src/ec/identity/generic_pull.rs` (new), `ec/pull_sync.rs` | `GenericPullProvider`; pull sync dispatcher removed | 22, 29 | +| `crates/trusted-server-core/src/ec/identity/test_support.rs` (new) | Shared fixtures, `ProviderHarness`, fake HTTP client and limiter | 14, 15, 20, 21, 22, 35, 39 | +| `crates/trusted-server-core/src/ec/identity/request.rs` (new), `src/ec/mod.rs`, `src/http_util.rs` | Writer decision, phase-1 captures, request identity state, read triggers | 23, 29 | +| `crates/trusted-server-core/src/ec/finalize.rs` | Mutation instead of writes; stager cookies; `ts-eids-r` on every path; metrics | 25, 26, 27, 29, 31 | +| `crates/trusted-server-core/src/auction/endpoints.rs` | KV-gap staging; server set in the merge; owner denials; hold mode | 26, 28, 36, 37 | +| `crates/trusted-server-core/src/ec/identity/flush.rs` (new) | Phase 4 `run_after_send` | 29 | +| `crates/trusted-server-adapter-fastly/src/app.rs`, `src/main.rs`, `src/rate_limiter.rs`, `tests/identity_writes.rs` | Writer decision, status capture, phase 4, `record`/`count`, invariant tests | 23, 25, 29, 30, 35 | +| `crates/trusted-server-core/src/integrations/registry.rs` | Provider registration, capture-path check, capture plumbing, `is_real_browser` | 23, 32, 38 | +| `crates/trusted-server-core/src/ec/rate_limiter.rs` | `record` and `count` | 35 | +| `crates/trusted-server-core/src/ec/batch_sync.rs` | Owned sources fill only when absent | 40 | +| `crates/trusted-server-core/src/ec/identity/examples.rs` (new, tests only) | Spec 4.6 reference providers | 39 | +| `crates/trusted-server-core/src/migration_guards.rs` | Every new `ec/identity/*.rs` file | 5 onward | +| `docs/guide/*.md`, `CHANGELOG.md`, `trusted-server.example.toml` | Docs | 4, 19, 31, 41 | + +------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------- | +| `Cargo.toml` (workspace), `crates/trusted-server-core/Cargo.toml` | `aes-gcm` dependency | 5 | +| `crates/trusted-server-core/src/constants.rs` | New cookie-name constants | 1 | +| `crates/trusted-server-core/src/cookies.rs` | Identity-cookie lists and the name-and-prefix strip helper | 1 | +| `crates/trusted-server-core/src/integrations/prebid.rs` | Strip identity cookies from Prebid Server requests | 2 | +| `crates/trusted-server-core/src/publisher.rs` | Origin strip; writer-request hooks; page-bids stops ingest; navigation merge and hold mode | 3, 22, 25, 27, 36 | +| `crates/trusted-server-core/src/cookies/template_cache_policy.rs` | New cookies refused as cache keys | 4 | +| `crates/trusted-server-core/src/ec/identity/mod.rs` (new) | Module docs, re-exports | 5 | +| `crates/trusted-server-core/src/ec/identity/ids.rs` (new) | `IdentityProviderId`, `SourceDomain`, `UnixTime` | 5 | +| `crates/trusted-server-core/src/ec/kv_types.rs` | `KvPartnerId` metadata, `KvProviderState`, `KvEntry.ps`, validation | 6 | +| `crates/trusted-server-core/src/ec/kv.rs` | Literals to constructors; `PartnerIdUpdate.acquired_at`; mutation flush; overflow write | 6, 7, 21, 26 | +| `crates/trusted-server-core/src/ec/admin.rs` | Redact `ps.st`; servability in the admin view | 6, 13 | +| `crates/trusted-server-core/src/ec/identity/descriptor.rs` (new) | `ProviderDescriptor`, builder, `Resolves`, `Captures`, `ConsentPolicy`, `EidProvenance`, `Input`, `Access`, `UsOptOut` | 8 | +| `crates/trusted-server-core/src/ec/identity/input.rs` (new) | `ResolveInput`, `CaptureInput`, `StoredId`, `Region`, `HashedEmail`, `ProviderConsent`, `ResolveKind` | 9 | +| `crates/trusted-server-core/src/ec/identity/exchange.rs` (new) | `ProviderRequest`, `ProviderResponse` | 10 | +| `crates/trusted-server-core/src/ec/identity/resolution.rs` (new) | `Resolution`, `ResolvedId`, `When`, `StateChange`, `IdentityProviderError` | 10 | +| `crates/trusted-server-core/src/ec/identity/provider.rs` (new) | `IdentityProvider`, `IdentityProviderConfig` | 11 | +| `crates/trusted-server-core/src/ec/identity/revision.rs` (new) | `Revision`, `rv` generation, keyed hashes (`pv` fallback, `iv`, visitor key, fingerprint) | 12 | +| `crates/trusted-server-core/src/ec/identity/servable.rs` (new) | The shared servability check | 13 | +| `crates/trusted-server-core/src/ec/eids.rs`, `ec/identify.rs` | Use the servability check; `/identify` stops writing | 13, 23 | +| `crates/trusted-server-core/src/ec/identity/mutation.rs` (new) | `IdentityMutation`, `PartnerIdChange`, `ProviderResult`, apply algorithm | 14 | +| `crates/trusted-server-core/src/ec/identity/cookie.rs` (new) | Seal and open, `ts-eids-s` payload, size cap, `ts-eids-r` batches, merge with KV | 15, 16, 17 | +| `crates/trusted-server-core/src/ec/identity/config.rs` (new) | `[ec.identity]` settings and validation | 18, 32 | +| `crates/trusted-server-core/src/settings.rs` | `Ec.identity`; `EcPartner.owner`; deprecate `pull_sync_concurrency` | 18, 32 | +| `crates/trusted-server-core/src/ec/identity/coordinator.rs` (new) | Eligibility, merged view, dispatch, outcomes, markers, breaker | 19, 20, 34, 35 | +| `crates/trusted-server-core/src/ec/identity/generic_pull.rs` (new) | `GenericPullProvider` | 21 | +| `crates/trusted-server-core/src/ec/pull_sync.rs` | Reduced to helpers `GenericPullProvider` reuses; dispatch removed | 21, 28 | +| `crates/trusted-server-core/src/ec/identity/test_support.rs` (new) | `ProviderHarness`, `WriteCountingEcKv`, fake HTTP client | 11, 21, 40 | +| `crates/trusted-server-core/src/ec/mod.rs` | `EcContext` writer flag, phase-1 captures, identity cookie state, mutation | 22 | +| `crates/trusted-server-core/src/http_util.rs` | `is_prefetch_request` moves here; `is_document_navigation` | 22 | +| `crates/trusted-server-core/src/ec/finalize.rs` | Mutation instead of ingest writes; stager cookie saves; `ts-eids-r` on every path | 24 | +| `crates/trusted-server-core/src/auction/endpoints.rs` | KV-gap staging; server set in `merge_auction_eids`; hold mode | 25, 27, 36 | +| `crates/trusted-server-adapter-fastly/src/app.rs`, `src/main.rs` | Writer decision, status capture, phase 4 entry point | 22, 28 | +| `crates/trusted-server-core/src/integrations/registry.rs` | Identity provider registration; capture plumbing; capture-path check | 31, 37 | +| `crates/trusted-server-core/src/config.rs` | `validate_identity_integration`, ownership validation | 32 | +| `crates/trusted-server-core/src/consent/{gpp.rs,tcf.rs,types.rs,mod.rs}` | Sharing and targeted-advertising opt-outs, section 6, GPC and known-child, `has_vendor_consent`, `stricter_of`, redacting `Debug` | 33 | +| `crates/trusted-server-core/src/ec/identity/policy.rs` (new) | `ConsentPolicy` evaluation and the consent fingerprint | 33 | +| `crates/trusted-server-core/src/ec/rate_limiter.rs`, `adapter-fastly/src/rate_limiter.rs` | `record` and `count` | 34 | +| `crates/trusted-server-core/src/ec/batch_sync.rs` | Owned sources fill only when absent | 39 | +| `docs/guide/edge-cookies.md`, `configuration.md`, `api-reference.md`, `ec-setup-guide.md`, `integrations/prebid.md` | Docs | 4, 30, 41 | + +--- + +# Part 0: Strip the identity cookies first (spec 14, step 0) + +This part only removes cookies from outbound requests. It sets no new cookies and changes no KV behaviour. It ships one release before Part 1, so a rollback after Part 1 still keeps the new one-year cookies away from the origin and from Prebid Server. + +Before you start: create a worktree from the spec branch, `git worktree add -b feat/1246-identity-cookie-strips .claude/worktrees/1246-identity-providers spec/1246-server-side-identity-providers`. The implementation stacks on its spec branch, which already carries the Prebid EID-fidelity work this spec depends on. Build with the toolchain that branch pins in `rust-toolchain.toml`. + +### Task 1: Identity cookie names and a name-and-prefix strip helper + +**Files:** + +- Modify: `crates/trusted-server-core/src/constants.rs` (next to `COOKIE_TS_EIDS`, line 8) +- Modify: `crates/trusted-server-core/src/cookies.rs` (after `CONSENT_COOKIE_NAMES`, and after `strip_cookies`) +- Test: `crates/trusted-server-core/src/cookies.rs` (`mod tests`) + +**Interfaces:** + +- Produces: + - `pub const COOKIE_TS_EIDS_S: &str = "__Host-ts-eids-s";` + - `pub const COOKIE_TS_EIDS_R_PREFIX: &str = "__Host-ts-eids-r-";` + - `pub const TS_IDENTITY_COOKIE_NAMES: &[&str]` = `ts-ec`, `ts-eids`, `__Host-ts-eids-s`, `ts-ec-pull-complete` + - `pub const TS_ORIGIN_STRIPPED_COOKIE_NAMES: &[&str]` = `ts-eids`, `__Host-ts-eids-s` + - `pub const TS_IDENTITY_COOKIE_PREFIXES: &[&str]` = `__Host-ts-eids-r-` + - `pub fn strip_cookie_pairs(cookie_header: &str, names: &[&str], prefixes: &[&str]) -> String` + - `pub fn strip_cookie_headers(headers: &mut http::HeaderMap, names: &[&str], prefixes: &[&str])` + +- [ ] **Step 1: Write the failing tests** + +Append to `mod tests` in `cookies.rs`: + +```rust + #[test] + fn strip_cookie_pairs_removes_names_and_prefixes() { + let header = "ts-ec=abc; ts-eids=W10; __Host-ts-eids-s=v1.x.y; \ + __Host-ts-eids-r-0a1b2c3d=v1.p.q; other=1; ts-eids-extra=keep"; + + let stripped = strip_cookie_pairs( + header, + TS_ORIGIN_STRIPPED_COOKIE_NAMES, + TS_IDENTITY_COOKIE_PREFIXES, + ); + + assert_eq!( + stripped, "ts-ec=abc; other=1; ts-eids-extra=keep", + "should drop the ID cookies and keep ts-ec and unrelated cookies" + ); + } + + #[test] + fn strip_cookie_headers_joins_every_cookie_field_and_drops_non_utf8() { + let mut headers = http::HeaderMap::new(); + headers.append(header::COOKIE, http::HeaderValue::from_static("ts-ec=abc; ts-eids=W10")); + headers.append( + header::COOKIE, + http::HeaderValue::from_bytes(b"bad=\xff").expect("should build an opaque header"), + ); + headers.append(header::COOKIE, http::HeaderValue::from_static("other=1")); + + strip_cookie_headers(&mut headers, TS_IDENTITY_COOKIE_NAMES, TS_IDENTITY_COOKIE_PREFIXES); + + let values: Vec<_> = headers.get_all(header::COOKIE).iter().collect(); + assert_eq!(values.len(), 1, "should collapse into one Cookie header"); + assert_eq!(values[0], "other=1", "should keep only the unrelated cookie"); + } + + #[test] + fn strip_cookie_headers_removes_header_when_nothing_is_left() { + let mut headers = http::HeaderMap::new(); + headers.insert( + header::COOKIE, + http::HeaderValue::from_static("ts-ec=abc; __Host-ts-eids-r-00000000=v1.a.b"), + ); + headers.insert(header::USER_AGENT, http::HeaderValue::from_static("ExampleBrowser/1.0")); + + strip_cookie_headers(&mut headers, TS_IDENTITY_COOKIE_NAMES, TS_IDENTITY_COOKIE_PREFIXES); + + assert!(headers.get(header::COOKIE).is_none(), "should remove the empty Cookie header"); + assert!(headers.get(header::USER_AGENT).is_some(), "should leave other headers alone"); + } +``` + +- [ ] **Step 2: Run the tests and see them fail** + +Run: `cargo test-fastly -- cookies::tests::strip_cookie` +Expected: compile errors, because `strip_cookie_pairs`, `strip_cookie_headers` and the constants do not exist. + +- [ ] **Step 3: Implement** + +In `constants.rs`, under `COOKIE_TS_EIDS`: + +```rust +/// Server-owned, sealed cache of provider-collected EIDs (spec 6.2). +pub const COOKIE_TS_EIDS_S: &str = "__Host-ts-eids-s"; +/// Name prefix of the per-batch removal cookies, `__Host-ts-eids-r-<8 hex>`. +pub const COOKIE_TS_EIDS_R_PREFIX: &str = "__Host-ts-eids-r-"; +``` + +In `cookies.rs`, extend the `crate::constants` import with `COOKIE_SHAREDID` (if you use it later), `COOKIE_TS_EC`, `COOKIE_TS_EC_PULL_COMPLETE`, `COOKIE_TS_EIDS`, `COOKIE_TS_EIDS_R_PREFIX` and `COOKIE_TS_EIDS_S`. Then add, after `CONSENT_COOKIE_NAMES`: + +```rust +/// Trusted Server identity cookies that never reach a third party. +/// +/// Stripped from Prebid Server requests and never added to proxy or provider +/// requests. The `__Host-ts-eids-r-` family is matched by +/// [`TS_IDENTITY_COOKIE_PREFIXES`]. +pub const TS_IDENTITY_COOKIE_NAMES: &[&str] = &[ + COOKIE_TS_EC, + COOKIE_TS_EIDS, + COOKIE_TS_EIDS_S, + COOKIE_TS_EC_PULL_COMPLETE, +]; + +/// Identity cookies removed from publisher-origin requests. +/// +/// `ts-ec` and `ts-ec-pull-complete` still reach the origin. The ID cookies +/// are removed so that they cannot push the origin's `Cookie` header past +/// common 8 KB header limits. +pub const TS_ORIGIN_STRIPPED_COOKIE_NAMES: &[&str] = &[COOKIE_TS_EIDS, COOKIE_TS_EIDS_S]; + +/// Name prefixes of identity cookie families, stripped wherever the named +/// identity cookies are. +pub const TS_IDENTITY_COOKIE_PREFIXES: &[&str] = &[COOKIE_TS_EIDS_R_PREFIX]; +``` + +After `strip_cookies`: + +```rust +/// Strips cookies matching any exact name or any name prefix from a `Cookie` +/// header value. +/// +/// Returns an empty string when nothing is left. +#[must_use] +pub fn strip_cookie_pairs(cookie_header: &str, names: &[&str], prefixes: &[&str]) -> String { + cookie_header + .split(';') + .map(str::trim) + .filter(|pair| !pair.is_empty()) + .filter(|pair| { + let name = pair.split('=').next().unwrap_or_default().trim(); + !names.contains(&name) && !prefixes.iter().any(|prefix| name.starts_with(prefix)) + }) + .collect::>() + .join("; ") +} + +/// Strips matching cookies from every `Cookie` field in `headers`. +/// +/// All `Cookie` fields are read, matching pairs are removed, and the rest is +/// written back as one field. A field that is not valid UTF-8 is dropped +/// rather than forwarded, because it cannot be checked. The header is removed +/// when nothing is left. +pub fn strip_cookie_headers(headers: &mut http::HeaderMap, names: &[&str], prefixes: &[&str]) { + let retained = headers + .get_all(header::COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .map(|value| strip_cookie_pairs(value, names, prefixes)) + .filter(|value| !value.is_empty()) + .collect::>() + .join("; "); + + headers.remove(header::COOKIE); + if retained.is_empty() { + return; + } + match http::HeaderValue::from_str(&retained) { + Ok(value) => { + headers.insert(header::COOKIE, value); + } + Err(_) => log::debug!("Dropping Cookie header that is invalid after stripping"), + } +} +``` + +`unwrap_or_default()` on an `Option<&str>` is not `unwrap()`, and `split` always yields at least one item, so it is safe here. + +- [ ] **Step 4: Run the tests and see them pass** + +Run: `cargo test-fastly -- cookies::tests` +Expected: PASS, including the existing `strip_cookies` tests. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/constants.rs crates/trusted-server-core/src/cookies.rs +git commit --signoff -S -m "Add identity cookie lists and a name-and-prefix cookie strip" +``` + +### Task 2: Strip identity cookies from Prebid Server requests in every mode + +**Files:** + +- Modify: `crates/trusted-server-core/src/auction/routing.rs` (`PrebidTransportHeaders`, around line 205) +- Modify: `crates/trusted-server-core/src/integrations/prebid.rs` (`apply_prebid_header_values` and both callers, around lines 2025-2108) +- Test: `crates/trusted-server-core/src/integrations/prebid.rs` (`mod tests`) + +**Interfaces:** + +- Consumes: `strip_cookie_pairs`, `TS_IDENTITY_COOKIE_NAMES`, `TS_IDENTITY_COOKIE_PREFIXES`, `CONSENT_COOKIE_NAMES` (Task 1) +- Produces: `PrebidTransportHeaders::cookies(&self) -> &[HeaderValue]`, which replaces `cookie()`. `apply_prebid_header_values` takes `cookies: &[HeaderValue]` as its first argument. + +Today `apply_prebid_header_values` forwards the raw `Cookie` header unchanged whenever `consent_forwarding.strips_consent_cookies()` is false, and both callers pass only the first `Cookie` field. After this task, identity cookies are stripped in every mode, every field is read, and a non-UTF-8 field is dropped (spec 6.2, "Prebid Server"). + +- [ ] **Step 1: Write the failing tests** + +Find the existing tests of `apply_prebid_header_values` with `grep -n "apply_prebid_header_values\|strips_consent_cookies" crates/trusted-server-core/src/integrations/prebid.rs`, and add next to them. Build the target request the way those tests do. The `ConsentForwardingMode` variants are listed in `consent_config.rs:137-150`. Use the one that does **not** strip consent cookies for the first test and the one that does for the second. + +```rust + fn cookie_values(request: &http::Request) -> Vec { + request + .headers() + .get_all(header::COOKIE) + .iter() + .map(|value| value.to_str().expect("should be ASCII").to_owned()) + .collect() + } + + #[test] + fn prebid_headers_strip_identity_cookies_when_consent_cookies_are_forwarded() { + let cookies = [ + HeaderValue::from_static("ts-ec=abc; euconsent-v2=tcf; other=1"), + HeaderValue::from_static("__Host-ts-eids-s=v1.a.b; __Host-ts-eids-r-0a1b2c3d=v1.c.d"), + HeaderValue::from_static("ts-eids=W10; ts-ec-pull-complete=v1; second=2"), + ]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::CookiesAndBody, + None, + ); + + assert_eq!( + cookie_values(&to), + vec!["euconsent-v2=tcf; other=1; second=2".to_owned()], + "should drop identity cookies, keep consent and unrelated cookies, and read every field" + ); + } + + #[test] + fn prebid_headers_strip_identity_and_consent_cookies_in_openrtb_only_mode() { + let cookies = [HeaderValue::from_static("ts-ec=abc; euconsent-v2=tcf; other=1")]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::OpenrtbOnly, + None, + ); + + assert_eq!(cookie_values(&to), vec!["other=1".to_owned()], "should keep only other=1"); + } + + #[test] + fn prebid_headers_drop_non_utf8_cookie_fields_and_empty_results() { + let cookies = [ + HeaderValue::from_bytes(b"x=\xff").expect("should build an opaque header"), + HeaderValue::from_static("ts-ec=abc"), + ]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::CookiesAndBody, + None, + ); + + assert!(to.headers().get(header::COOKIE).is_none(), "should send no Cookie header"); + } +``` + +If the mode variants are named differently, use the real names and keep the meaning: one mode that forwards consent cookies and `OpenrtbOnly`. + +- [ ] **Step 2: Run the tests and see them fail** + +Run: `cargo test-fastly -- integrations::prebid::tests::prebid_headers_` +Expected: compile error, because the first argument is still `Option<&HeaderValue>`. + +- [ ] **Step 3: Implement** + +In `auction/routing.rs`, change the field and its accessor: + +```rust +pub(crate) struct PrebidTransportHeaders { + cookies: Vec, + user_agent: Option, + referer: Option, + accept_language: Option, +} + +impl PrebidTransportHeaders { + /// Every inbound `Cookie` field, in order. + pub(crate) fn cookies(&self) -> &[HeaderValue] { + &self.cookies + } + // user_agent, referer and accept_language unchanged +``` + +and in `snapshot`: + +```rust + cookies: request.headers().get_all(header::COOKIE).iter().cloned().collect(), +``` + +Fix every other use of `cookie:` or `.cookie()` in `routing.rs` that the compiler reports, including the tests at about lines 1300-1315 that append two `Cookie` fields. Those tests should now see both fields. + +In `integrations/prebid.rs`, change the import to `use crate::cookies::{CONSENT_COOKIE_NAMES, TS_IDENTITY_COOKIE_NAMES, TS_IDENTITY_COOKIE_PREFIXES, strip_cookie_pairs};`. Then replace the cookie half of `apply_prebid_header_values`: + +```rust +fn apply_prebid_header_values( + cookies: &[HeaderValue], + user_agent: Option<&HeaderValue>, + referer: Option<&HeaderValue>, + accept_language: Option<&HeaderValue>, + to: &mut http::Request, + consent_forwarding: ConsentForwardingMode, + client_ip: Option, +) { + // ... unchanged header loop and x-forwarded-for ... + + // Identity cookies are stripped in every mode, before the consent mode is + // checked, so no mode forwards the raw header (spec 6.2). + let consent_names: &[&str] = if consent_forwarding.strips_consent_cookies() { + CONSENT_COOKIE_NAMES + } else { + &[] + }; + let retained = cookies + .iter() + .filter_map(|value| value.to_str().ok()) + .map(|value| strip_cookie_pairs(value, TS_IDENTITY_COOKIE_NAMES, TS_IDENTITY_COOKIE_PREFIXES)) + .map(|value| strip_cookie_pairs(&value, consent_names, &[])) + .filter(|value| !value.is_empty()) + .collect::>() + .join("; "); + if !retained.is_empty() + && let Ok(cookie_header) = HeaderValue::from_str(&retained) + { + to.headers_mut().insert(header::COOKIE, cookie_header); + } +} +``` + +Update both callers. `copy_request_headers`-style callers pass a local `let cookies: Vec = from.headers().get_all(header::COOKIE).iter().cloned().collect();` and then `&cookies`. `apply_prebid_transport_headers` passes `from.cookies()`. Remove `strip_cookies` from the import if nothing else uses it. + +- [ ] **Step 4: Run the tests and see them pass** + +Run: `cargo test-fastly -- integrations::prebid` and then `cargo test-fastly -- auction::routing` +Expected: PASS. If an existing test asserted that the raw header is forwarded unchanged, update its expectation to the stripped value and say why in the commit body. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/auction/routing.rs crates/trusted-server-core/src/integrations/prebid.rs +git commit --signoff -S -m "Strip identity cookies from Prebid Server requests in every mode" -m "Every Cookie field is now read and checked, and a field that is not valid UTF-8 is dropped instead of forwarded." +``` + +### Task 3: Strip the ID cookies from publisher-origin requests + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (`rewrite_origin_request`, around line 3124) +- Test: `crates/trusted-server-core/src/publisher.rs` (`mod tests`) + +**Interfaces:** + +- Consumes: `strip_cookie_headers`, `TS_ORIGIN_STRIPPED_COOKIE_NAMES`, `TS_IDENTITY_COOKIE_PREFIXES` (Task 1) + +- [ ] **Step 1: Write the failing test** + +Add to `publisher.rs` tests, next to any existing `rewrite_origin_request` test (`grep -n "fn .*rewrite_origin_request" crates/trusted-server-core/src/publisher.rs`): + +```rust + #[test] + fn rewrite_origin_request_strips_id_cookies_and_keeps_ts_ec() { + let mut req = Request::builder() + .uri("https://www.example.com/article") + .header(header::COOKIE, "ts-ec=abc; ts-eids=W10; session=1") + .header( + header::COOKIE, + "__Host-ts-eids-s=v1.a.b; __Host-ts-eids-r-0a1b2c3d=v1.c.d; ts-ec-pull-complete=v1", + ) + .body(EdgeBody::empty()) + .expect("should build request"); + + rewrite_origin_request( + &mut req, + "https://origin.example.com/article".parse().expect("should parse uri"), + "origin.example.com", + ) + .expect("should rewrite request"); + + let cookies: Vec<_> = req.headers().get_all(header::COOKIE).iter().collect(); + assert_eq!(cookies.len(), 1, "should send one Cookie header"); + assert_eq!( + cookies[0], "ts-ec=abc; session=1; ts-ec-pull-complete=v1", + "should keep ts-ec, the pull marker and unrelated cookies, and drop the ID cookies" + ); + } + + #[test] + fn rewrite_origin_request_drops_cookie_header_when_only_id_cookies() { + let mut req = Request::builder() + .uri("https://www.example.com/") + .header(header::COOKIE, "ts-eids=W10; __Host-ts-eids-s=v1.a.b") + .body(EdgeBody::empty()) + .expect("should build request"); + + rewrite_origin_request( + &mut req, + "https://origin.example.com/".parse().expect("should parse uri"), + "origin.example.com", + ) + .expect("should rewrite request"); + + assert!(req.headers().get(header::COOKIE).is_none(), "should drop the empty Cookie header"); + assert_eq!(req.headers()[header::HOST], "origin.example.com", "should still set Host"); + } +``` + +- [ ] **Step 2: Run and see it fail** + +Run: `cargo test-fastly -- publisher::tests::rewrite_origin_request_` +Expected: FAIL. The first assertion sees two `Cookie` fields that still hold `ts-eids`. + +- [ ] **Step 3: Implement** + +In `rewrite_origin_request`, after `req.headers_mut().remove("fastly-ssl");`: + +```rust + // The publisher origin keeps `ts-ec` but never receives the ID cookies, + // which can push its `Cookie` header past common 8 KB limits (spec 6.2). + strip_cookie_headers( + req.headers_mut(), + TS_ORIGIN_STRIPPED_COOKIE_NAMES, + TS_IDENTITY_COOKIE_PREFIXES, + ); +``` + +Add `use crate::cookies::{TS_IDENTITY_COOKIE_PREFIXES, TS_ORIGIN_STRIPPED_COOKIE_NAMES, strip_cookie_headers};` to the imports at the top of `publisher.rs`. + +- [ ] **Step 4: Run and see it pass, then run the other adapters** + +Run: `cargo test-fastly -- publisher::tests` then `cargo test-axum && cargo test-cloudflare && cargo test-spin` +Expected: PASS. The strip lives in core, so every adapter gets it (spec 11). If an adapter test asserted that `ts-eids` reaches the origin, update it and note the behaviour change in the commit body. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs +git commit --signoff -S -m "Strip ID cookies from publisher origin requests" -m "The origin keeps ts-ec and the pull-complete marker. ts-eids, __Host-ts-eids-s and the __Host-ts-eids-r- cookies no longer reach it. An origin that read ts-eids must read x-ts-eids instead." +``` + +### Task 4: Refuse the new cookies as template-cache keys, and document Part 0 + +**Files:** + +- Modify: `crates/trusted-server-core/src/cookies/template_cache_policy.rs` (around line 35) +- Modify: `docs/guide/edge-cookies.md`, `CHANGELOG.md` +- Test: `crates/trusted-server-core/src/cookies/template_cache_policy.rs` (`mod tests`, line 172) + +- [ ] **Step 1: Write the failing tests** + +Find the test that rejects `ts-eids` as a key (`grep -n "identity cookie" crates/trusted-server-core/src/cookies/template_cache_policy.rs`). It shows how the validating function is called. Add two cases next to it with the same call shape: + +```rust + #[test] + fn template_cache_key_rejects_sealed_identity_cookies() { + for name in ["__Host-ts-eids-s", "__Host-ts-eids-r-0a1b2c3d"] { + let error = validate_names_for_test(&[name.to_owned()], &[]) + .expect_err("should reject identity cookie as a cache key"); + assert!( + error.contains("must not key on Trusted Server identity cookie"), + "should explain the rejection for {name}, got {error}" + ); + } + validate_names_for_test(&["session".to_owned()], &[]) + .expect("should still accept an unrelated cookie"); + } +``` + +`validate_names_for_test` stands for the call the existing identity-cookie test makes. Use that exact call, not a new helper. + +- [ ] **Step 2: Run and see it fail** + +Run: `cargo test-fastly -- template_cache_policy::tests::template_cache_key_rejects_sealed` +Expected: FAIL, because both names are accepted today. + +- [ ] **Step 3: Implement** + +Replace the condition at line 35: + +```rust + if field == "template_cache_key_cookies" + && ([COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_TS_EIDS_S, COOKIE_SHAREDID] + .contains(&name.as_str()) + || name.starts_with(COOKIE_TS_EIDS_R_PREFIX)) + { +``` + +and extend the `crate::constants` import with `COOKIE_TS_EIDS_R_PREFIX` and `COOKIE_TS_EIDS_S`. + +- [ ] **Step 4: Run and see it pass** + +Run: `cargo test-fastly -- template_cache_policy` +Expected: PASS. + +- [ ] **Step 5: Docs** + +In `docs/guide/edge-cookies.md`, in the section that lists TS cookies, add a short "Cookies forwarded to the origin" paragraph. Say that `ts-ec` and `ts-ec-pull-complete` still reach the publisher origin, that `ts-eids` and the server-owned `__Host-ts-eids-s` and `__Host-ts-eids-r-*` cookies do not, and that origin code that read `ts-eids` should read the `x-ts-eids` response header instead. Also say that Prebid Server never receives any TS identity cookie. + +In `CHANGELOG.md` under `## [Unreleased]` / `### Changed`, add: + +```markdown +- **Breaking:** The publisher origin no longer receives the `ts-eids` cookie, and Prebid Server no longer receives any Trusted Server identity cookie (`ts-ec`, `ts-eids`, `ts-ec-pull-complete`, and the server-owned `__Host-ts-eids-s` and `__Host-ts-eids-r-*` cookies that a later release introduces). The origin still receives `ts-ec`. Origin code that read `ts-eids` should read the `x-ts-eids` response header instead. This release is the rollback floor for the server-side identity provider work. +``` + +Run: `cd docs && npm run format && cd ..` and `docs/node_modules/.bin/prettier --config docs/.prettierrc --write CHANGELOG.md` + +- [ ] **Step 6: Commit** + +```bash +git add crates/trusted-server-core/src/cookies/template_cache_policy.rs docs/guide/edge-cookies.md CHANGELOG.md +git commit --signoff -S -m "Refuse the sealed identity cookies as template cache keys" +``` + +- [ ] **Step 7: Part 0 gate and PR** + +Run `/check-ci`. Every gate must pass. Open the PR against `spec/1246-server-side-identity-providers` with the `pr-creator` agent. Title: "Strip identity cookies from origin and Prebid Server requests". The description must say this is spec step 0 and the rollback floor for the later parts, and must name the origin `ts-eids` behaviour change. + +--- + +# Part 1: One write per page load, no new vendors (spec 14, step 1) + +Part 1 builds the trait, descriptor, harness and coordinator, moves every browser-driven KV write into the writer request's single flush, adds `__Host-ts-eids-s` and `__Host-ts-eids-r-*`, and replaces pull sync with `GenericPullProvider`. No vendor provider is registered yet, and no provider runs in hold mode or captures. Those come in Part 2. + +Before you start: `git switch feat/1246-identity-cookie-strips && git switch -c feat/1246-identity-single-write`. + +Tasks 5 to 22 build pure, unit-tested modules with no request wiring. Tasks 23 to 29 wire them into requests. Task 30 proves the one-write invariant, and Task 31 ships. + +### Task 5: The `ec::identity` module, its ID newtypes and its error type + +**Files:** + +- Modify: `Cargo.toml` (workspace `[workspace.dependencies]`, alphabetical), `crates/trusted-server-core/Cargo.toml` +- Create: `crates/trusted-server-core/src/ec/identity/mod.rs` +- Create: `crates/trusted-server-core/src/ec/identity/ids.rs` +- Create: `crates/trusted-server-core/src/ec/identity/resolution.rs` (error type only in this task, extended in Task 10) +- Modify: `crates/trusted-server-core/src/ec/mod.rs` (module list, around line 32) +- Modify: `crates/trusted-server-core/src/migration_guards.rs` (`checked_sources`) + +**Interfaces:** + +- Produces: + - `IdentityProviderId`: `FromStr>`, `as_str(&self) -> &str`, serde as a string. Derives `Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display`. + - `SourceDomain`: `FromStr` through `normalize_partner_source_domain`, `as_str`, `matches(&self, name: &str) -> bool`, serde as a string. Same derives. + - `UnixTime`: `from_secs(u64)`, `from_millis(u64)`, `as_secs(self) -> u64`, `Add`. Derives `Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord`. + - `IdentityProviderError` (all seven variants from spec 4.5) with `config`, `rejected`, `malformed` and `not_implemented` constructors. `Access` and `Input` are referenced by two variants, so this task adds a minimal `descriptor.rs` with just those two enums. Task 8 fills in the rest of the file. + +- [ ] **Step 1: Add the dependency** + +In the workspace `Cargo.toml`, under `[workspace.dependencies]` in alphabetical order: + +```toml +aes-gcm = "0.10" +``` + +In `crates/trusted-server-core/Cargo.toml` `[dependencies]`, alphabetical: + +```toml +aes-gcm = { workspace = true } +``` + +Run: `cargo check-fastly && cargo check-cloudflare` +Expected: both build. `aes-gcm` 0.10 depends on `getrandom` only through its optional `getrandom` feature, which stays off. Nonces come from `rand::thread_rng` (Task 16). If `cargo check-cloudflare` fails on `getrandom`, set `default-features = false, features = ["aes", "alloc"]` in the workspace entry and run it again. + +- [ ] **Step 2: Write the failing tests** + +Create `ec/identity/ids.rs` with only this test module for now: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn provider_id_accepts_the_documented_grammar() { + for id in ["id5", "pull-ssp-example-com", "lockr_aim", "a"] { + id.parse::() + .unwrap_or_else(|_| panic!("should accept `{id}`")); + } + let longest = format!("a{}", "b".repeat(62)); + longest.parse::().expect("should accept 63 characters"); + } + + #[test] + fn provider_id_rejects_everything_else() { + let too_long = format!("a{}", "b".repeat(63)); + for id in ["", "1id", "Id5", "id 5", "id.5", too_long.as_str()] { + let error = id.parse::().expect_err("should reject the id"); + assert!( + matches!(error.current_context(), IdentityProviderError::Config { .. }), + "should be a Config error for `{id}`" + ); + } + } + + #[test] + fn source_domain_normalizes_and_matches() { + let source: SourceDomain = "ID5-Sync.com.".parse().expect("should parse"); + assert_eq!(source.as_str(), "id5-sync.com", "should lower-case and drop the trailing dot"); + assert!(source.matches("id5-sync.COM"), "should match another spelling"); + assert!(!source.matches("example.com"), "should not match another domain"); + assert!(!source.matches("https://id5-sync.com"), "should not match a URL"); + } + + #[test] + fn source_domain_round_trips_through_serde() { + let source: SourceDomain = "example.com".parse().expect("should parse"); + let json = serde_json::to_string(&source).expect("should serialize"); + assert_eq!(json, "\"example.com\"", "should serialize as a bare string"); + let back: SourceDomain = serde_json::from_str(&json).expect("should deserialize"); + assert_eq!(back, source, "should round-trip"); + serde_json::from_str::("\"not a domain\"") + .expect_err("should reject an invalid domain on deserialize"); + } + + #[test] + fn unix_time_converts_millis_and_adds_durations() { + assert_eq!(UnixTime::from_millis(1_760_000_000_999).as_secs(), 1_760_000_000, "should floor millis"); + let later = UnixTime::from_secs(100) + Duration::from_secs(50); + assert_eq!(later, UnixTime::from_secs(150), "should add seconds"); + let saturated = UnixTime::from_secs(u64::MAX) + Duration::from_secs(1); + assert_eq!(saturated, UnixTime::from_secs(u64::MAX), "should saturate"); + } +} +``` + +Create `ec/identity/mod.rs`: + +```rust +//! Server-side identity providers. +//! +//! An [`IdentityProvider`](provider::IdentityProvider) turns request signals +//! into OpenRTB EIDs. Providers only build requests and parse buffered +//! responses. The coordinator owns eligibility, consent, transport limits and +//! KV write-back, so every page load makes at most one browser-driven KV +//! write. See `docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md`. + +pub mod descriptor; +pub mod ids; +pub mod resolution; + +pub use descriptor::{Access, Input}; +pub use ids::{IdentityProviderId, SourceDomain, UnixTime}; +pub use resolution::IdentityProviderError; +``` + +Add `pub mod identity;` to the module list in `ec/mod.rs`, in alphabetical order after `pub mod identify;`. + +- [ ] **Step 3: Run and see the tests fail** + +Run: `cargo test-fastly -- ec::identity::ids` +Expected: compile errors, because the types do not exist. + +- [ ] **Step 4: Implement `resolution.rs` (error only) and the minimal `descriptor.rs`** + +`ec/identity/descriptor.rs` for now: + +```rust +//! What a provider is, may write, needs and must respect. + +/// An input a provider can ask for. +/// +/// Consent strings, the provider's stored IDs, the publisher domain, the +/// time and the declared endpoint are always available. Private state is +/// available to providers declared `with_private_state()`. +#[derive( + Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, +)] +#[non_exhaustive] +pub enum Input { + /// The platform's client address. + #[display("client_ip")] + ClientIp, + /// The request's `User-Agent`. + #[display("user_agent")] + UserAgent, + /// The top-level page URL without query string or fragment. + #[display("page_url")] + PageUrl, + /// Country and subdivision only. + #[display("region")] + Region, + /// A SHA-256 of a normalized email address. Never available in v1. + #[display("hashed_email")] + HashedEmail, +} + +/// Something a provider read without declaring it. +#[derive( + Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, +)] +#[non_exhaustive] +pub enum Access { + /// An undeclared [`Input`]. + #[display("{_0}")] + Input(Input), + /// The endpoint, when the descriptor declares no `resolves`. + #[display("endpoint")] + Endpoint, + /// Private state, without `with_private_state()`. + #[display("private_state")] + PrivateState, +} +``` + +`ec/identity/resolution.rs` for now: + +```rust +//! Provider results and errors. + +use error_stack::Report; +use http::StatusCode; + +use super::descriptor::{Access, Input}; + +/// Marker message of [`IdentityProviderError::not_implemented`]. The harness +/// matches on it to tell a missing override from a real failure. +pub(crate) const NOT_IMPLEMENTED_MARKER: &str = "capability declared but not implemented: "; + +/// A provider failure. Built through the constructors below. +#[derive(Debug, derive_more::Display)] +#[non_exhaustive] +pub enum IdentityProviderError { + /// Configuration or provider code makes the call impossible. + #[display("identity provider configuration error: {message}")] + Config { + /// What is wrong. Never contains a secret, UID or URL query. + message: String, + }, + /// The provider read something its descriptor did not declare. + #[display("`{access}` was read but not declared")] + UndeclaredInput { + /// What was read. + access: Access, + }, + /// A declared input is not available for this request. + #[display("input `{input}` is unavailable for this request")] + InputUnavailable { + /// The missing input. + input: Input, + }, + /// The provider refused the request. + #[display("provider rejected the request with status {status}")] + Rejected { + /// The response status. + status: StatusCode, + }, + /// The provider's response could not be parsed. + #[display("provider response could not be parsed")] + Malformed, + /// The stored private state does not decode as the provider's type. + #[display("stored provider state could not be decoded")] + InvalidState, + /// Set by the coordinator only. Providers never see transport failures. + #[display("provider request failed in transport")] + Transport, +} + +impl core::error::Error for IdentityProviderError {} + +impl IdentityProviderError { + /// A [`Self::Config`] error. + #[must_use] + pub fn config(message: impl Into) -> Report { + Report::new(Self::Config { message: message.into() }) + } + + /// A [`Self::Rejected`] error. + #[must_use] + pub fn rejected(status: StatusCode) -> Report { + Report::new(Self::Rejected { status }) + } + + /// A [`Self::Malformed`] error. + #[must_use] + pub fn malformed() -> Report { + Report::new(Self::Malformed) + } + + /// The default body of a capability method: a `Config` error with a + /// fixed marker message, so the harness can tell a missing override from + /// a real failure. + #[must_use] + pub fn not_implemented(method: &'static str) -> Report { + Self::config(format!("{NOT_IMPLEMENTED_MARKER}{method}")) + } + + /// Whether this is the [`Self::not_implemented`] marker. + #[must_use] + pub(crate) fn is_not_implemented(&self) -> bool { + matches!(self, Self::Config { message } if message.starts_with(NOT_IMPLEMENTED_MARKER)) + } +} +``` + +- [ ] **Step 5: Implement `ids.rs`** + +Put this above the test module: + +```rust +//! Identifiers shared by the identity provider API. + +use core::ops::Add; +use core::str::FromStr; +use std::time::Duration; + +use error_stack::Report; +use serde::{Deserialize, Serialize}; + +use super::resolution::IdentityProviderError; +use crate::ec::partner::normalize_partner_source_domain; + +/// An identity provider's ID, such as `id5` or `pull-ssp-example-com`. +/// +/// Matches `^[a-z][a-z0-9_-]{0,62}$`, so it can be a bare TOML key and can +/// equal an integration ID that uses `_`. Separate from +/// [`crate::auction::ProviderId`]. +#[derive( + Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, Serialize, Deserialize, +)] +#[serde(try_from = "String", into = "String")] +pub struct IdentityProviderId(String); + +impl IdentityProviderId { + /// The ID as written in config. + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl FromStr for IdentityProviderId { + type Err = Report; + + fn from_str(value: &str) -> Result { + let mut bytes = value.bytes(); + let valid_first = bytes.next().is_some_and(|byte| byte.is_ascii_lowercase()); + let valid_rest = bytes.all(|byte| { + byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'_' || byte == b'-' + }); + if valid_first && valid_rest && value.len() <= 63 { + Ok(Self(value.to_owned())) + } else { + Err(IdentityProviderError::config(format!( + "identity provider id `{value}` must match ^[a-z][a-z0-9_-]{{0,62}}$" + ))) + } + } +} + +impl TryFrom for IdentityProviderId { + type Error = Report; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(id: IdentityProviderId) -> Self { + id.0 + } +} + +/// An EID `source` domain, normalized. Also the key of the KV `ids` map. +#[derive( + Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, Serialize, Deserialize, +)] +#[serde(try_from = "String", into = "String")] +pub struct SourceDomain(String); + +impl SourceDomain { + /// The normalized domain. + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } + + /// Whether a vendor's spelling of a source names this one, after + /// normalization. + #[must_use] + pub fn matches(&self, name: &str) -> bool { + normalize_partner_source_domain(name).is_ok_and(|normalized| normalized == self.0) + } +} + +impl FromStr for SourceDomain { + type Err = Report; + + fn from_str(value: &str) -> Result { + normalize_partner_source_domain(value) + .map(Self) + .map_err(IdentityProviderError::config) + } +} + +impl TryFrom for SourceDomain { + type Error = Report; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(source: SourceDomain) -> Self { + source.0 + } +} + +/// Unix seconds. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub struct UnixTime(u64); + +impl UnixTime { + /// From Unix seconds. + #[must_use] + pub fn from_secs(secs: u64) -> Self { + Self(secs) + } + + /// From Unix milliseconds, for vendors such as Lockr and UID2. Floors to + /// whole seconds. + #[must_use] + pub fn from_millis(millis: u64) -> Self { + Self(millis / 1000) + } + + /// As Unix seconds. + #[must_use] + pub fn as_secs(self) -> u64 { + self.0 + } +} + +impl Add for UnixTime { + type Output = UnixTime; + + fn add(self, rhs: Duration) -> UnixTime { + UnixTime(self.0.saturating_add(rhs.as_secs())) + } +} +``` + +serde's `try_from` needs the error to implement `Display`, and `Report` does. Check that `normalize_partner_source_domain` rejects `"not a domain"` (spaces fail the label check). If it does not, use `"not/a/domain"` in the test. + +Add both new files to `checked_sources()` in `migration_guards.rs`, in alphabetical order, following the `ec/client_eids.rs` entry: + +```rust + ("ec/identity/descriptor.rs", include_str!("ec/identity/descriptor.rs")), + ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), + ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), + ("ec/identity/resolution.rs", include_str!("ec/identity/resolution.rs")), +``` + +Every later task that creates a file under `ec/identity/` adds it to this list the same way. + +- [ ] **Step 6: Run and see the tests pass** + +Run: `cargo test-fastly -- ec::identity` and `cargo test-fastly -- migration_guards` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add Cargo.toml Cargo.lock crates/trusted-server-core/Cargo.toml crates/trusted-server-core/src/ec/mod.rs crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Add the identity module with provider and source ID newtypes" -m "Adds aes-gcm, which the sealed identity cookies use." +``` + +### Task 6: KV schema: record metadata, provider state, `Default` and constructors + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/kv_types.rs` (`KvEntry`, `KvConsent`, `KvGeo`, `KvPartnerId`, `validate`, constructors) +- Modify: every file with a `KvPartnerId {` literal (find them with `grep -rn "KvPartnerId {" crates/`) +- Modify: `crates/trusted-server-core/src/ec/admin.rs` (raw lookup response, around `admin.rs:394`) +- Test: `crates/trusted-server-core/src/ec/kv_types.rs`, `crates/trusted-server-core/src/ec/admin.rs` + +**Interfaces:** + +- Produces: + - `KvPartnerId { uid, w: Option, rv: Option, at: Option, exp: Option, ra: Option, ru: Option, atype: Option, ext: Option }`, deriving `Default`, with `KvPartnerId::new(uid: impl Into) -> Self` and `KvPartnerId::browser(uid: impl Into, acquired_at: Option) -> Self` + - `KvProviderState { n: Option, f: u32, o: Option, oc: Option, iv: Vec, st: Option }`, deriving `Default` + - `KvOptOut { at: u64, fp: String }` + - `KvEntry.ps: BTreeMap`, and `KvEntry` derives `Default` + - Constants `MAX_PARTNER_EXT_BYTES = 256`, `MAX_PROVIDER_STATE_BYTES = 1024`, `MAX_PROVIDER_STATES = 32`, `MAX_INVALIDATED_UIDS = 4`, `RECORD_REVISION_HEX_LEN = 16` + +- [ ] **Step 1: Write the failing tests** + +Add to `kv_types.rs` tests: + +```rust + #[test] + fn legacy_partner_record_reads_and_writes_unchanged() { + let legacy = json!({ "uid": "example-uid" }); + let record: KvPartnerId = serde_json::from_value(legacy.clone()).expect("should read legacy"); + assert_eq!(record, KvPartnerId::new("example-uid"), "should fill every new field with None"); + assert_eq!(serde_json::to_value(&record).expect("should write"), legacy, "should add no fields"); + } + + #[test] + fn provider_record_and_state_round_trip() { + let stored = json!({ + "v": 1, + "created": 1_760_000_000, + "consent": { "ok": true, "updated": 1_760_000_000 }, + "geo": { "country": "US" }, + "ids": { + "id5-sync.com": { + "uid": "ID5*example", + "w": "id5", + "rv": "9c1d04b7e6a83f2a", + "at": 1_760_000_000, + "exp": 1_767_776_000, + "ra": 1_760_086_400, + "ru": 1_770_000_000, + "atype": 1, + "ext": { "linkType": 2 } + } + }, + "ps": { + "id5": { + "n": 1_760_003_600, + "f": 1, + "o": { "at": 1_759_000_000, "fp": "0123456789abcdef" }, + "oc": 1_759_500_000, + "iv": ["aaaaaaaaaaaaaaaa"], + "st": { "signature": "example-opaque-state" } + } + } + }); + + let entry: KvEntry = serde_json::from_value(stored.clone()).expect("should read"); + entry.validate().expect("should be valid"); + assert_eq!(serde_json::to_value(&entry).expect("should write"), stored, "should round-trip exactly"); + } + + #[test] + fn validate_rejects_oversized_ext_state_and_too_many_providers() { + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ids.get_mut("example.com").expect("should exist").ext = + Some(json!({ "pad": "x".repeat(MAX_PARTNER_EXT_BYTES) })); + assert!(entry.validate().expect_err("should reject ext").contains("ext"), "should name ext"); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ps.insert( + "id5".to_owned(), + KvProviderState { st: Some(json!("x".repeat(MAX_PROVIDER_STATE_BYTES))), ..KvProviderState::default() }, + ); + assert!(entry.validate().expect_err("should reject st").contains("st"), "should name st"); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + for index in 0..=MAX_PROVIDER_STATES { + entry.ps.insert(format!("p{index}"), KvProviderState::default()); + } + assert!(entry.validate().expect_err("should reject ps size").contains("ps"), "should name ps"); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ps.insert( + "id5".to_owned(), + KvProviderState { iv: vec!["a".repeat(16); MAX_INVALIDATED_UIDS + 1], ..KvProviderState::default() }, + ); + assert!(entry.validate().expect_err("should reject iv").contains("iv"), "should name iv"); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ids.get_mut("example.com").expect("should exist").rv = Some("not-hex".to_owned()); + assert!(entry.validate().expect_err("should reject rv").contains("rv"), "should name rv"); + + KvEntry::minimal("example.com", "example-uid", 1).validate().expect("control entry should be valid"); + } +``` + +The validation test builds `KvProviderState` with `..KvProviderState::default()` because it tests one field at a time. It is not an exhaustiveness test. + +Add to `admin.rs` tests, next to the existing raw-lookup test (`grep -n "lookup_raw\|fn .*lookup" crates/trusted-server-core/src/ec/admin.rs`). Use the same setup that test uses to put an entry in the in-memory graph and call the handler: + +```rust + #[test] + fn admin_lookup_redacts_provider_state() { + let mut entry = KvEntry::minimal("example.com", "example-uid", 1_760_000_000); + entry.ps.insert( + "id5".to_owned(), + KvProviderState { n: Some(5), st: Some(json!({ "signature": "secret-state" })), ..KvProviderState::default() }, + ); + let body = admin_lookup_body_for_test(&entry); + + assert_eq!(body["ps"]["id5"]["st"], json!("[redacted]"), "should redact the state"); + assert_eq!(body["ps"]["id5"]["n"], json!(5), "should keep the rest of ps"); + assert_eq!(body["ids"]["example.com"]["uid"], json!("example-uid"), "should keep the ids"); + } +``` + +`admin_lookup_body_for_test` stands for whatever the existing raw-lookup test does to get the JSON body: insert the row, call the handler and parse the body. Write it as a small helper in the test module that reuses that test's setup. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::kv_types::tests` and `cargo test-fastly -- ec::admin::tests::admin_lookup_redacts` +Expected: compile errors (`ps`, `KvProviderState` and `KvPartnerId::new` do not exist). + +- [ ] **Step 3: Implement the schema** + +In `kv_types.rs`, add the constants after `MAX_UID_LENGTH`: + +```rust +/// Maximum serialized bytes of a stored UID-level `ext`. +pub const MAX_PARTNER_EXT_BYTES: usize = 256; +/// Maximum serialized bytes of one provider's private state (`ps.st`). +pub const MAX_PROVIDER_STATE_BYTES: usize = 1024; +/// Maximum number of providers with a `ps` entry in one row. +pub const MAX_PROVIDER_STATES: usize = 32; +/// Maximum number of invalidated-UID hashes kept per provider (`ps.iv`). +pub const MAX_INVALIDATED_UIDS: usize = 4; +/// Length of a record revision (`rv`) in lowercase hex digits. +pub const RECORD_REVISION_HEX_LEN: usize = 16; +``` + +Add `Default` to the derives of `KvEntry`, `KvConsent` and `KvGeo`. Add the `ps` field at the end of `KvEntry`: + +```rust + /// Per-provider scheduling, opt-out and private state, keyed by + /// identity provider ID. Never sent to bidders or cookies. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub ps: BTreeMap, +``` + +Replace `KvPartnerId`: + +```rust +/// A partner user ID within a KV entry. +/// +/// Every field but `uid` is optional and omitted when absent, so legacy rows +/// read and write unchanged and `SCHEMA_VERSION` stays 1 (spec 6.1). +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct KvPartnerId { + /// The partner's user identifier. + pub uid: String, + /// Writer: an identity provider ID. Absent for browser ingest and legacy rows. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub w: Option, + /// Record revision: 16 lowercase hex digits, random on every write. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rv: Option, + /// Unix seconds when the ID was acquired or last refreshed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub at: Option, + /// Unix seconds after which the ID must not be sent. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub exp: Option, + /// Unix seconds after which the ID is due for refresh. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ra: Option, + /// Unix seconds after which the ID can no longer be refreshed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ru: Option, + /// Provider-supplied `atype`. Absent: use the partner's `openrtb_atype`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub atype: Option, + /// UID-level `ext`, at most [`MAX_PARTNER_EXT_BYTES`] serialized bytes. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ext: Option, +} + +impl KvPartnerId { + /// A bare record holding only a UID. + #[must_use] + pub fn new(uid: impl Into) -> Self { + Self { uid: uid.into(), ..Self::default() } + } + + /// A browser-ingested record: no writer, with its acquisition time. + #[must_use] + pub fn browser(uid: impl Into, acquired_at: Option) -> Self { + Self { uid: uid.into(), at: acquired_at, ..Self::default() } + } +} + +/// A provider's sticky opt-out (spec 8). +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct KvOptOut { + /// Unix seconds when the opt-out was reported. + pub at: u64, + /// Consent fingerprint at that time: 16 lowercase hex digits. + pub fp: String, +} + +/// Per-provider state in a KV row (`ps.`). +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct KvProviderState { + /// Unix seconds before which the provider must not be called. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub n: Option, + /// Consecutive failures, for backoff. + #[serde(default, skip_serializing_if = "is_zero")] + pub f: u32, + /// Sticky opt-out. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub o: Option, + /// When the provider's opt-out was last cleared. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oc: Option, + /// Keyed hashes of UIDs the owner invalidated, newest last. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub iv: Vec, + /// The provider's private state as embedded JSON. KV only. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub st: Option, +} + +fn is_zero(value: &u32) -> bool { + *value == 0 +} +``` + +The listed constructors (`KvEntry::new`, `minimal`, `tombstone`) gain `ps: BTreeMap::new()`. `minimal` uses `KvPartnerId::new(uid)`. + +Extend `validate()`. Inside the existing `for (partner_id, partner_uid) in &self.ids` loop, after the UID length check: + +```rust + if let Some(ext) = &partner_uid.ext + && serde_json::to_vec(ext).map_or(true, |bytes| bytes.len() > MAX_PARTNER_EXT_BYTES) + { + return Err(format!("partner ID '{partner_id}' ext exceeds {MAX_PARTNER_EXT_BYTES} bytes")); + } + if let Some(rv) = &partner_uid.rv + && (rv.len() != RECORD_REVISION_HEX_LEN + || !rv.bytes().all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))) + { + return Err(format!("partner ID '{partner_id}' rv is not {RECORD_REVISION_HEX_LEN} hex digits")); + } +``` + +After the loop: + +```rust + if self.ps.len() > MAX_PROVIDER_STATES { + return Err(format!("ps exceeds MAX_PROVIDER_STATES ({})", self.ps.len())); + } + for (provider, state) in &self.ps { + if state.iv.len() > MAX_INVALIDATED_UIDS { + return Err(format!("ps '{provider}' iv exceeds {MAX_INVALIDATED_UIDS} hashes")); + } + if let Some(st) = &state.st + && serde_json::to_vec(st).map_or(true, |bytes| bytes.len() > MAX_PROVIDER_STATE_BYTES) + { + return Err(format!("ps '{provider}' st exceeds {MAX_PROVIDER_STATE_BYTES} bytes")); + } + } +``` + +- [ ] **Step 4: Move every literal to the constructor** + +Run `grep -rn "KvPartnerId {" crates/` and change each `KvPartnerId { uid: X }` to `KvPartnerId::new(X)`. That covers about 30 sites in 8 files, including `ec/kv.rs` (`apply_cookie_partner_id_updates`, `apply_partner_id_updates` and others), the batch sync and pull sync writers, the admin tests and `kv_types.rs`. Run `grep -rn "KvEntry {" crates/` too, and add `ps: BTreeMap::new()` to any full literal the compiler flags. + +- [ ] **Step 5: Redact `ps.st` in the admin lookup** + +In `ec/admin.rs`, find where the raw lookup body is returned verbatim (`grep -n "lookup_raw" crates/trusted-server-core/src/ec/admin.rs`). Before it is written to the response, parse it, redact and re-serialize: + +```rust +/// Replaces every `ps..st` with a marker. Provider state is never +/// returned by the admin lookup (spec 6.1, 9.3). +fn redact_provider_state(body: &[u8]) -> Vec { + let Ok(mut value) = serde_json::from_slice::(body) else { + return body.to_vec(); + }; + if let Some(states) = value.get_mut("ps").and_then(serde_json::Value::as_object_mut) { + for state in states.values_mut() { + if let Some(st) = state.get_mut("st") { + *st = serde_json::Value::String("[redacted]".to_owned()); + } + } + } + serde_json::to_vec(&value).unwrap_or_else(|_| body.to_vec()) +} +``` + +Call it on `lookup.body` where the handler builds its response. A body that is not JSON is returned unchanged, as today, because the lookup exists to inspect corrupt rows. + +- [ ] **Step 6: Run and see the tests pass** + +Run: `cargo test-fastly -- ec::` and then `cargo clippy-fastly` +Expected: PASS. Every existing KV, batch sync and pull sync test must still pass unchanged. + +- [ ] **Step 7: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Add provider metadata and provider state to the KV schema" -m "New fields are optional and omitted when absent, so legacy rows read and write unchanged and the schema version stays 1. KvPartnerId and KvEntry derive Default, and literals move to constructors. The admin lookup redacts provider state." +``` + +### Task 7: Browser ingest records when it acquired an ID + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (`PartnerIdUpdate`, `apply_cookie_partner_id_updates`, `apply_partner_id_updates`) +- Modify: `crates/trusted-server-core/src/ec/prebid_eids.rs` (`collect_eid_updates`, `collect_sharedid_update`, `collect_prebid_eid_updates_from_eids`) +- Modify: every `PartnerIdUpdate::new` caller the compiler reports +- Test: `crates/trusted-server-core/src/ec/kv.rs` + +**Interfaces:** + +- Produces: `PartnerIdUpdate { partner_id, uid, acquired_at: Option }`, `PartnerIdUpdate::new(partner_id, uid)` (`acquired_at = None`), `PartnerIdUpdate::with_acquired_at(self, at: u64) -> Self`. A record that browser ingest inserts now carries `at`. + +- [ ] **Step 1: Write the failing test** + +```rust + #[test] + fn browser_ingest_records_acquisition_time() { + let mut entry = KvEntry::minimal("sharedid.org", "existing", 1); + let updates = [PartnerIdUpdate::new("example.com", "new-uid").with_acquired_at(1_760_000_000)]; + + let changed = apply_partner_id_updates(&mut entry, &updates); + + assert!(changed, "should report a change"); + assert_eq!( + entry.ids["example.com"], + KvPartnerId::browser("new-uid", Some(1_760_000_000)), + "should store the UID with at and no writer" + ); + assert_eq!(entry.ids["sharedid.org"], KvPartnerId::new("existing"), "should leave other sources alone"); + } +``` + +- [ ] **Step 2: Run and see it fail** + +Run: `cargo test-fastly -- ec::kv::tests::browser_ingest_records_acquisition_time` +Expected: compile error, because `with_acquired_at` does not exist. + +- [ ] **Step 3: Implement** + +Add the field and builder: + +```rust +pub(crate) struct PartnerIdUpdate { + pub(crate) partner_id: String, + pub(crate) uid: String, + /// Unix seconds this request saw the ID. Stored as the record's `at`. + pub(crate) acquired_at: Option, +} + +impl PartnerIdUpdate { + pub(crate) fn new(partner_id: impl Into, uid: impl Into) -> Self { + Self { partner_id: partner_id.into(), uid: uid.into(), acquired_at: None } + } + + /// Records when this request saw the ID. + #[must_use] + pub(crate) fn with_acquired_at(mut self, at: u64) -> Self { + self.acquired_at = Some(at); + self + } +} +``` + +In both `apply_*` functions, the de-duplicating map now keeps the whole update: `BTreeMap<&str, &PartnerIdUpdate>`. Insert `KvPartnerId::browser(update.uid.clone(), update.acquired_at)`. Equality checks still compare only `uid`, so an unchanged UID is not rewritten just to bump `at`. + +In `prebid_eids.rs`, give `collect_eid_updates` a `now: Option` parameter. Call `.with_acquired_at(now)` on each update when it is `Some`, and pass `checked_current_timestamp()` from the callers in `finalize.rs`. Leave batch sync and pull sync without `acquired_at`. They get it through the mutation in Tasks 15 and 22. + +- [ ] **Step 4: Run and see it pass** + +Run: `cargo test-fastly -- ec::` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/trusted-server-core/src/ec +git commit --signoff -S -m "Record when browser ingest acquired each ID" +``` + +### Task 8: The provider descriptor + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/identity/descriptor.rs` (Task 5 created it with `Input` and `Access`) +- Modify: `crates/trusted-server-core/src/ec/identity/mod.rs` (re-exports) +- Test: `crates/trusted-server-core/src/ec/identity/descriptor.rs` + +**Interfaces:** + +- Consumes: `IdentityProviderId`, `SourceDomain`, `IdentityProviderError` (Task 5) +- Produces, exactly as spec 4.3 lists them: + - `ProviderDescriptor` and its getters: `builder`, `id`, `sources`, `primary_source`, `source_named`, `resolves`, `endpoint`, `captures`, `consent_policy`, `provenance`, `is_expiring` + - `ProviderDescriptorBuilder`: `with_source`, `with_resolves`, `with_captures`, `with_consent_policy`, `with_expiring_ids`, `with_expiring_source`, `with_provenance`, `build` + - `Resolves`: `new`, `with_endpoint(&str)`, `with_host(&str)`, `with_needs`, `with_needs_for`, `with_wants`, `with_kinds`, `with_private_state`, plus `pub(in crate::ec::identity) fn with_ec_id(self)` and crate-visible getters `hosts()`, `needs(kind)`, `wants()`, `kinds()`, `has_private_state()`, `gets_ec_id()` + - `Captures`: `new`, `with_path(Method, &str)`, `with_path_prefix(&str)`, `with_request_consent(pointers)`, plus crate-visible `matches(&Method, &str) -> bool`, `paths()`, `prefixes()`, `consent_pointers()` + - `EidProvenance`: `new`, `with_inserter`, `with_matcher`, `with_mm`, `inserter`, `matcher`, `mm` + - `ConsentPolicy`: `strict`, `with_tcf_purposes`, `with_tcf_vendor`, `allowing_dnt`, `allowing_gpc`, `allowing_us_opt_out`, plus `pub(in crate::ec::identity) fn eid_persistence_only()` for generic pull and crate-visible getters + - `UsOptOut { Sale, Sharing, TargetedAdvertising }` + - `ResolveKind { Initial, Refresh }` lives here rather than in `input.rs` because `Resolves` keys on it. `input.rs` re-exports it. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use http::Method; + + use super::*; + + fn source(name: &str) -> SourceDomain { + name.parse().expect("should parse source") + } + + fn id(name: &str) -> IdentityProviderId { + name.parse().expect("should parse id") + } + + #[test] + fn builds_the_documented_example() { + let descriptor = ProviderDescriptor::builder(id("exampleid")) + .with_source(source("exampleid.example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.exampleid.example.com/v1/fetch") + .with_needs([Input::ClientIp, Input::UserAgent]) + .with_wants([Input::PageUrl]), + ) + .with_consent_policy(ConsentPolicy::strict().with_tcf_vendor(9999)) + .build() + .expect("should build"); + + assert_eq!(descriptor.id().as_str(), "exampleid", "should keep the id"); + assert_eq!(descriptor.primary_source(), &source("exampleid.example.com"), "should keep the source"); + assert_eq!( + descriptor.endpoint().map(Url::as_str), + Some("https://api.exampleid.example.com/v1/fetch"), + "should parse the endpoint" + ); + let resolves = descriptor.resolves().expect("should resolve"); + assert_eq!( + resolves.hosts(), + &["api.exampleid.example.com".to_owned()], + "should allow the endpoint host" + ); + assert_eq!( + resolves.kinds(), + &[ResolveKind::Initial, ResolveKind::Refresh], + "should default to Initial and Refresh" + ); + assert!(descriptor.captures().is_none(), "should not capture"); + assert!(!descriptor.is_expiring(&source("exampleid.example.com")), "should not expire by default"); + assert_eq!(descriptor.consent_policy().tcf_vendor(), Some(9999), "should keep the vendor"); + assert_eq!( + descriptor.consent_policy().tcf_purposes().iter().copied().collect::>(), + vec![1, 4], + "strict should require Purposes 1 and 4" + ); + } + + #[test] + fn build_rejects_invalid_descriptors() { + let cases: Vec<(&str, ProviderDescriptorBuilder)> = vec![ + ("no source", ProviderDescriptor::builder(id("p")).with_resolves(Resolves::new().with_endpoint("https://a.example.com"))), + ("no capability", ProviderDescriptor::builder(id("p")).with_source(source("example.com"))), + ("no endpoint or host", ProviderDescriptor::builder(id("p")).with_source(source("example.com")).with_resolves(Resolves::new())), + ("http endpoint", ProviderDescriptor::builder(id("p")).with_source(source("example.com")).with_resolves(Resolves::new().with_endpoint("http://a.example.com"))), + ("bad host", ProviderDescriptor::builder(id("p")).with_source(source("example.com")).with_resolves(Resolves::new().with_host("a.example.com/path"))), + ( + "provenance for undeclared source", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_provenance(&source("other.example.com"), EidProvenance::new().with_mm(2)), + ), + ( + "expiring undeclared source", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_expiring_source(&source("other.example.com")), + ), + ]; + for (name, builder) in cases { + let error = builder.build().expect_err(name); + assert!( + matches!(error.current_context(), IdentityProviderError::Config { .. }), + "should be a Config error: {name}" + ); + } + } + + #[test] + fn expiring_and_provenance_are_per_source() { + let a = source("a.example.com"); + let b = source("b.example.com"); + let descriptor = ProviderDescriptor::builder(id("p")) + .with_source(a.clone()) + .with_source(b.clone()) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .with_expiring_source(&b) + .with_provenance(&a, EidProvenance::new().with_inserter("example-inserter").with_mm(2)) + .build() + .expect("should build"); + + assert!(!descriptor.is_expiring(&a), "a should not expire"); + assert!(descriptor.is_expiring(&b), "b should expire"); + let provenance = descriptor.provenance(&a).expect("a should have provenance"); + assert_eq!(provenance.inserter(), Some("example-inserter"), "should keep the inserter"); + assert_eq!(provenance.matcher(), None, "should leave the matcher unset"); + assert_eq!(provenance.mm(), Some(2), "should keep mm"); + assert!(descriptor.provenance(&b).is_none(), "b should have none"); + assert_eq!(descriptor.source_named("A.EXAMPLE.COM"), Some(&a), "should find a by any spelling"); + } + + #[test] + fn expiring_ids_marks_every_source() { + let descriptor = ProviderDescriptor::builder(id("p")) + .with_source(source("a.example.com")) + .with_source(source("b.example.com")) + .with_expiring_ids() + .with_captures(Captures::new().with_path_prefix("/integrations/p/")) + .build() + .expect("should build"); + assert!(descriptor.sources().iter().all(|s| descriptor.is_expiring(s)), "should mark every source"); + } + + #[test] + fn captures_match_exact_paths_and_prefixes() { + let captures = Captures::new() + .with_path(Method::POST, "/integrations/p/api/token") + .with_path_prefix("/integrations/p/stream/"); + assert!(captures.matches(&Method::POST, "/integrations/p/api/token"), "should match the exact path"); + assert!(!captures.matches(&Method::GET, "/integrations/p/api/token"), "should require the method"); + assert!(!captures.matches(&Method::POST, "/integrations/p/api/token/x"), "exact means exact"); + assert!(captures.matches(&Method::GET, "/integrations/p/stream/1"), "prefix matches any method"); + assert!(!captures.matches(&Method::GET, "/integrations/p/settings"), "should not match others"); + } + + #[test] + fn needs_differ_by_kind() { + let resolves = Resolves::new() + .with_endpoint("https://a.example.com") + .with_needs([Input::ClientIp]) + .with_needs_for(ResolveKind::Initial, [Input::HashedEmail]); + assert_eq!( + resolves.needs(ResolveKind::Initial), + [Input::ClientIp, Input::HashedEmail].into_iter().collect(), + "Initial needs both" + ); + assert_eq!( + resolves.needs(ResolveKind::Refresh), + [Input::ClientIp].into_iter().collect(), + "Refresh needs only the IP" + ); + } +} +``` + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::descriptor` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Add above the existing enums in `descriptor.rs`: + +```rust +use std::collections::{BTreeMap, BTreeSet}; + +use error_stack::Report; +use http::Method; +use url::Url; + +use super::ids::{IdentityProviderId, SourceDomain}; +use super::resolution::IdentityProviderError; +use crate::ec::partner::normalize_partner_source_domain; + +/// What a provider is, may write, needs and must respect. +/// +/// Built from config with no secrets, so `ts config validate` and the running +/// coordinator read the same value. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ProviderDescriptor { + id: IdentityProviderId, + sources: Vec, + resolves: Option, + endpoint: Option, + captures: Option, + consent_policy: ConsentPolicy, + provenance: BTreeMap, + expiring: BTreeSet, +} + +impl ProviderDescriptor { + /// Starts a descriptor for `id`. + #[must_use] + pub fn builder(id: IdentityProviderId) -> ProviderDescriptorBuilder { + ProviderDescriptorBuilder { + id, + sources: Vec::new(), + resolves: None, + captures: None, + consent_policy: ConsentPolicy::strict(), + provenance: BTreeMap::new(), + expiring_all: false, + expiring: BTreeSet::new(), + } + } + + /// The provider's ID. + #[must_use] + pub fn id(&self) -> &IdentityProviderId { + &self.id + } + + /// The sources this provider may write, in declaration order. + #[must_use] + pub fn sources(&self) -> &[SourceDomain] { + &self.sources + } + + /// The first source added. Every built descriptor has one. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + &self.sources[0] + } + + /// Finds a declared source by a vendor's spelling of it, normalized. + #[must_use] + pub fn source_named(&self, name: &str) -> Option<&SourceDomain> { + self.sources.iter().find(|source| source.matches(name)) + } + + /// Active resolution, when declared. + #[must_use] + pub fn resolves(&self) -> Option<&Resolves> { + self.resolves.as_ref() + } + + /// The endpoint declared with [`Resolves::with_endpoint`], if any. + #[must_use] + pub fn endpoint(&self) -> Option<&Url> { + self.endpoint.as_ref() + } + + /// Passive capture, when declared. + #[must_use] + pub fn captures(&self) -> Option<&Captures> { + self.captures.as_ref() + } + + /// The consent this provider requires. + #[must_use] + pub fn consent_policy(&self) -> &ConsentPolicy { + &self.consent_policy + } + + /// EID-level fields for `source`, if declared. + #[must_use] + pub fn provenance(&self, source: &SourceDomain) -> Option<&EidProvenance> { + self.provenance.get(source) + } + + /// Whether IDs of `source` are declared as expiring. + #[must_use] + pub fn is_expiring(&self, source: &SourceDomain) -> bool { + self.expiring.contains(source) + } +} + +/// Builds a [`ProviderDescriptor`]. +#[derive(Debug, Clone)] +pub struct ProviderDescriptorBuilder { + id: IdentityProviderId, + sources: Vec, + resolves: Option, + captures: Option, + consent_policy: ConsentPolicy, + provenance: BTreeMap, + expiring_all: bool, + expiring: BTreeSet, +} + +impl ProviderDescriptorBuilder { + /// Adds a source this provider may write. Repeatable. A repeat is ignored. + #[must_use] + pub fn with_source(mut self, source: SourceDomain) -> Self { + if !self.sources.contains(&source) { + self.sources.push(source); + } + self + } + + /// Declares active resolution. + #[must_use] + pub fn with_resolves(mut self, resolves: Resolves) -> Self { + self.resolves = Some(resolves); + self + } + + /// Declares passive capture. + #[must_use] + pub fn with_captures(mut self, captures: Captures) -> Self { + self.captures = Some(captures); + self + } + + /// Replaces the default policy, [`ConsentPolicy::strict`]. + #[must_use] + pub fn with_consent_policy(mut self, policy: ConsentPolicy) -> Self { + self.consent_policy = policy; + self + } + + /// Marks the IDs of every declared source as expiring (spec 6.1). + #[must_use] + pub fn with_expiring_ids(mut self) -> Self { + self.expiring_all = true; + self + } + + /// Marks the IDs of one declared source as expiring. + #[must_use] + pub fn with_expiring_source(mut self, source: &SourceDomain) -> Self { + self.expiring.insert(source.clone()); + self + } + + /// EID-level fields for a declared source, added to every server-side + /// value of that source this provider wrote. No KV field stores them. + #[must_use] + pub fn with_provenance(mut self, source: &SourceDomain, provenance: EidProvenance) -> Self { + self.provenance.insert(source.clone(), provenance); + self + } + + /// Validates and builds the descriptor. + /// + /// # Errors + /// + /// Returns `Config` when the descriptor has no source, no capability, a + /// resolver with no endpoint or host, an endpoint that is not an `https` + /// URL, a host that is not a bare hostname, or provenance or an expiring + /// mark for a source it does not declare. + pub fn build(self) -> Result> { + let id = self.id; + let fail = |what: &str| IdentityProviderError::config(format!("provider `{id}`: {what}")); + + if self.sources.is_empty() { + return Err(fail("declares no source")); + } + if self.resolves.is_none() && self.captures.is_none() { + return Err(fail("declares neither resolves nor captures")); + } + for source in self.provenance.keys().chain(self.expiring.iter()) { + if !self.sources.contains(source) { + return Err(fail(&format!("names undeclared source `{source}`"))); + } + } + + let mut endpoint = None; + let mut resolves = self.resolves; + if let Some(resolves) = resolves.as_mut() { + if let Some(raw) = resolves.raw_endpoint.as_deref() { + let url = Url::parse(raw).map_err(|_| fail("endpoint is not a URL"))?; + if url.scheme() != "https" { + return Err(fail("endpoint must use https")); + } + let host = url.host_str().ok_or_else(|| fail("endpoint has no host"))?.to_owned(); + if !resolves.hosts.contains(&host) { + resolves.hosts.insert(0, host); + } + endpoint = Some(url); + } + if resolves.hosts.is_empty() { + return Err(fail("resolves declares no endpoint or host")); + } + for host in &resolves.hosts { + if normalize_partner_source_domain(host).ok().as_deref() != Some(host.as_str()) { + return Err(fail(&format!("host `{host}` is not a bare lowercase hostname"))); + } + } + } + + let expiring = if self.expiring_all { + self.sources.iter().cloned().collect() + } else { + self.expiring + }; + + Ok(ProviderDescriptor { + id, + sources: self.sources, + resolves, + endpoint, + captures: self.captures, + consent_policy: self.consent_policy, + provenance: self.provenance, + expiring, + }) + } +} + +/// The kind of resolution one source needs. +#[derive( + Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, +)] +#[non_exhaustive] +pub enum ResolveKind { + /// No stored record for the source. + #[display("initial")] + Initial, + /// A stored record exists, valid or expired, and is due. + #[display("refresh")] + Refresh, +} + +/// Active resolution: the provider calls an endpoint. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Resolves { + raw_endpoint: Option, + hosts: Vec, + needs: BTreeSet, + needs_for: BTreeMap>, + wants: BTreeSet, + kinds: Vec, + private_state: bool, + ec_id: bool, +} + +impl Default for Resolves { + fn default() -> Self { + Self::new() + } +} + +impl Resolves { + /// A resolver with no inputs, handling `Initial` and `Refresh`. + #[must_use] + pub fn new() -> Self { + Self { + raw_endpoint: None, + hosts: Vec::new(), + needs: BTreeSet::new(), + needs_for: BTreeMap::new(), + wants: BTreeSet::new(), + kinds: vec![ResolveKind::Initial, ResolveKind::Refresh], + private_state: false, + ec_id: false, + } + } + + /// The endpoint the provider calls, as config spells it. `build()` parses + /// it and requires `https`. Its host becomes an allowed host. + #[must_use] + pub fn with_endpoint(mut self, endpoint: &str) -> Self { + self.raw_endpoint = Some(endpoint.to_owned()); + self + } + + /// Adds another host the provider may call. Repeatable. + #[must_use] + pub fn with_host(mut self, host: &str) -> Self { + if !self.hosts.iter().any(|known| known == host) { + self.hosts.push(host.to_owned()); + } + self + } + + /// Inputs required for every kind. + #[must_use] + pub fn with_needs(mut self, inputs: impl IntoIterator) -> Self { + self.needs.extend(inputs); + self + } + + /// Inputs required for one kind only. + #[must_use] + pub fn with_needs_for(mut self, kind: ResolveKind, inputs: impl IntoIterator) -> Self { + self.needs_for.entry(kind).or_default().extend(inputs); + self + } + + /// Inputs passed when available, never required. + #[must_use] + pub fn with_wants(mut self, inputs: impl IntoIterator) -> Self { + self.wants.extend(inputs); + self + } + + /// The kinds this provider handles. Default: `Initial` and `Refresh`. + #[must_use] + pub fn with_kinds(mut self, kinds: impl IntoIterator) -> Self { + self.kinds = kinds.into_iter().collect(); + self + } + + /// This provider keeps private state, so it runs only on the writer + /// request, where state can be stored (spec 5.3). + #[must_use] + pub fn with_private_state(mut self) -> Self { + self.private_state = true; + self + } + + /// Gives the provider the full EC ID. Only generic pull may use this. + #[must_use] + pub(in crate::ec::identity) fn with_ec_id(mut self) -> Self { + self.ec_id = true; + self + } + + /// Hosts the provider may call. + #[must_use] + pub(crate) fn hosts(&self) -> &[String] { + &self.hosts + } + + /// Inputs required for `kind`: the common needs plus that kind's. + #[must_use] + pub(crate) fn needs(&self, kind: ResolveKind) -> BTreeSet { + let mut needs = self.needs.clone(); + if let Some(extra) = self.needs_for.get(&kind) { + needs.extend(extra.iter().copied()); + } + needs + } + + /// Every input declared for any kind, needed or wanted. + #[must_use] + pub(crate) fn declared_inputs(&self) -> BTreeSet { + let mut all = self.needs.clone(); + all.extend(self.wants.iter().copied()); + for extra in self.needs_for.values() { + all.extend(extra.iter().copied()); + } + all + } + + /// Inputs passed when available. + #[must_use] + pub(crate) fn wants(&self) -> &BTreeSet { + &self.wants + } + + /// The kinds this provider handles. + #[must_use] + pub(crate) fn kinds(&self) -> &[ResolveKind] { + &self.kinds + } + + /// Whether the provider keeps private state. + #[must_use] + pub(crate) fn has_private_state(&self) -> bool { + self.private_state + } + + /// Whether the provider receives the full EC ID. + #[must_use] + pub(crate) fn gets_ec_id(&self) -> bool { + self.ec_id + } +} + +/// Passive capture from responses TS proxies for this provider. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Captures { + paths: Vec<(Method, String)>, + prefixes: Vec, + consent_pointers: Vec<&'static str>, +} + +impl Captures { + /// No capture rules yet. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Offers proxied responses to exactly this method and path. Repeatable. + #[must_use] + pub fn with_path(mut self, method: Method, path: &str) -> Self { + self.paths.push((method, path.to_owned())); + self + } + + /// Offers proxied responses whose path starts with `prefix`, for any + /// method. Repeatable. + #[must_use] + pub fn with_path_prefix(mut self, prefix: &str) -> Self { + self.prefixes.push(prefix.to_owned()); + self + } + + /// JSON pointers to consent strings in the proxied request's JSON body. + /// The coordinator extracts only these fields and gates capture by the + /// stricter of cookie and body consent. The provider never sees the body. + #[must_use] + pub fn with_request_consent(mut self, pointers: impl IntoIterator) -> Self { + self.consent_pointers.extend(pointers); + self + } + + /// Whether a proxied request matches a rule. + #[must_use] + pub(crate) fn matches(&self, method: &Method, path: &str) -> bool { + self.paths.iter().any(|(rule_method, rule_path)| rule_method == method && rule_path == path) + || self.prefixes.iter().any(|prefix| path.starts_with(prefix.as_str())) + } + + /// Exact path rules. + #[must_use] + pub(crate) fn paths(&self) -> &[(Method, String)] { + &self.paths + } + + /// Prefix rules. + #[must_use] + pub(crate) fn prefixes(&self) -> &[String] { + &self.prefixes + } + + /// Declared body consent pointers. + #[must_use] + pub(crate) fn consent_pointers(&self) -> &[&'static str] { + &self.consent_pointers + } +} + +/// EID-level fields (OpenRTB 2.6 §3.2.27) that bidders read. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct EidProvenance { + inserter: Option, + matcher: Option, + mm: Option, +} + +impl EidProvenance { + /// No fields set. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Sets `inserter`. + #[must_use] + pub fn with_inserter(mut self, inserter: &str) -> Self { + self.inserter = Some(inserter.to_owned()); + self + } + + /// Sets `matcher`. + #[must_use] + pub fn with_matcher(mut self, matcher: &str) -> Self { + self.matcher = Some(matcher.to_owned()); + self + } + + /// Sets `mm`. + #[must_use] + pub fn with_mm(mut self, mm: i32) -> Self { + self.mm = Some(mm); + self + } + + /// `inserter`, if set. + #[must_use] + pub fn inserter(&self) -> Option<&str> { + self.inserter.as_deref() + } + + /// `matcher`, if set. + #[must_use] + pub fn matcher(&self) -> Option<&str> { + self.matcher.as_deref() + } + + /// `mm`, if set. + #[must_use] + pub fn mm(&self) -> Option { + self.mm + } +} + +/// A US opt-out a policy can relax. +#[derive( + Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, +)] +#[non_exhaustive] +pub enum UsOptOut { + /// Sale of personal information. + #[display("sale")] + Sale, + /// Sharing for cross-context behavioural advertising. + #[display("sharing")] + Sharing, + /// Targeted advertising. + #[display("targeted_advertising")] + TargetedAdvertising, +} + +/// The consent a provider requires. Evaluated by the coordinator (Task 12). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ConsentPolicy { + tcf_purposes: BTreeSet, + tcf_vendor: Option, + allow_dnt: bool, + allow_gpc: bool, + allowed_us_opt_outs: BTreeSet, + eid_persistence_only: bool, +} + +impl ConsentPolicy { + /// The default. Blocked when any consent signal present cannot be + /// decoded. Where TCF applies, requires Purposes 1 and 4. Blocked by + /// every US opt-out TS decodes, by `Sec-GPC: 1` and by `DNT: 1`. + #[must_use] + pub fn strict() -> Self { + Self { + tcf_purposes: [1, 4].into_iter().collect(), + tcf_vendor: None, + allow_dnt: false, + allow_gpc: false, + allowed_us_opt_outs: BTreeSet::new(), + eid_persistence_only: false, + } + } + + /// Also requires these TCF purposes, where TCF applies. + #[must_use] + pub fn with_tcf_purposes(mut self, purposes: impl IntoIterator) -> Self { + self.tcf_purposes.extend(purposes); + self + } + + /// Requires vendor consent for this IAB Global Vendor List ID. + #[must_use] + pub fn with_tcf_vendor(mut self, vendor_id: u16) -> Self { + self.tcf_vendor = Some(vendor_id); + self + } + + /// Does not block on `DNT: 1`. Needs a documented legal basis. + #[must_use] + pub fn allowing_dnt(mut self) -> Self { + self.allow_dnt = true; + self + } + + /// Does not block on GPC. Needs a documented legal basis. + #[must_use] + pub fn allowing_gpc(mut self) -> Self { + self.allow_gpc = true; + self + } + + /// Does not block on one US opt-out. Needs a documented legal basis. + #[must_use] + pub fn allowing_us_opt_out(mut self, opt_out: UsOptOut) -> Self { + self.allowed_us_opt_outs.insert(opt_out); + self + } + + /// Blocks exactly what pull sync blocks today: only the global EC and + /// EID-persistence gates. For generic pull's parity (spec 10.4). + #[must_use] + pub(in crate::ec::identity) fn eid_persistence_only() -> Self { + Self { eid_persistence_only: true, ..Self::strict() } + } + + /// Required TCF purposes. + #[must_use] + pub fn tcf_purposes(&self) -> &BTreeSet { + &self.tcf_purposes + } + + /// Required GVL vendor, if any. + #[must_use] + pub fn tcf_vendor(&self) -> Option { + self.tcf_vendor + } + + /// Whether `DNT: 1` is allowed. + #[must_use] + pub fn allows_dnt(&self) -> bool { + self.allow_dnt + } + + /// Whether GPC is allowed. + #[must_use] + pub fn allows_gpc(&self) -> bool { + self.allow_gpc + } + + /// Whether a US opt-out is allowed. + #[must_use] + pub fn allows_us_opt_out(&self, opt_out: UsOptOut) -> bool { + self.allowed_us_opt_outs.contains(&opt_out) + } + + /// Whether this is generic pull's parity policy. + #[must_use] + pub(crate) fn is_eid_persistence_only(&self) -> bool { + self.eid_persistence_only + } +} +``` + +`normalize_partner_source_domain` doubles as a hostname check: it accepts exactly a bare, lowercase-normalized hostname. Comparing its output to the input also rejects uppercase. Update `mod.rs`: + +```rust +pub use descriptor::{ + Access, Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, + ProviderDescriptorBuilder, ResolveKind, Resolves, UsOptOut, +}; +``` + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::descriptor` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity +git commit --signoff -S -m "Add the identity provider descriptor and its builder" +``` + +### Task 9: Provider inputs + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/input.rs` +- Modify: `crates/trusted-server-core/src/ec/identity/mod.rs`, `migration_guards.rs` +- Test: `crates/trusted-server-core/src/ec/identity/input.rs` + +**Interfaces:** + +- Consumes: `ProviderDescriptor`, `Resolves`, `Input`, `Access`, `ResolveKind` (Task 8), and `ConsentContext` (`consent/types.rs`) +- Produces: + - `ProviderConsent` with `gdpr_applies`, `raw_tc_string`, `raw_gpp_string`, `gpp_section_ids`, `raw_us_privacy`, and `pub(crate) fn from_context(&ConsentContext) -> Self` + - `Region` with `country`, `subdivision`, `is`, and `pub(crate) fn from_geo(country: &str, region: Option<&str>) -> Option`, which normalizes to upper case and strips an `XX-` prefix + - `HashedEmail` (32 bytes) with `as_bytes`, `to_hex`, `to_base64`, and a redacting `Debug` + - `StoredId` with `uid`, `atype`, `ext`, `writer`, `acquired_at`, `expires_at`, `refresh_at`, `is_expired`, and `pub(crate) fn from_record(&KvPartnerId) -> Self` + - `ResolveSignals<'a>`: a crate-private struct holding every signal the coordinator has (IP, UA, page URL, region, HEM, consent, publisher domain, now, random seed source, EC ID, stored IDs, raw state JSON, dry-run flag) + - `ResolveInput<'a>` with the spec 4.4 accessors, `pub(crate) fn new(descriptor: &'a ProviderDescriptor, kinds: BTreeMap, signals: &'a ResolveSignals<'a>) -> Self`, `pub(crate) fn recorded_errors(&self) -> Vec`, `pub(crate) fn state_decode_failed(&self) -> bool`, and `pub(in crate::ec::identity) fn ec_id(&self) -> Result<&str, Report>` + - `CaptureInput<'a>` with the spec 4.4 accessors and `pub(crate) fn new(...)` + - `RandomSource`: a trait `fn fill(&self, bytes: &mut [u8])`, with `ThreadRandom` for production and `SeededRandom` (a ChaCha-free xorshift from a `u64` seed, test support only) + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + use std::net::IpAddr; + + use super::*; + use crate::ec::identity::descriptor::{Captures, ProviderDescriptor, Resolves}; + + fn descriptor(resolves: Resolves) -> ProviderDescriptor { + ProviderDescriptor::builder("p".parse().expect("should parse")) + .with_source("example.com".parse().expect("should parse")) + .with_resolves(resolves) + .build() + .expect("should build") + } + + fn signals() -> ResolveSignals<'static> { + ResolveSignals { + client_ip: Some("192.0.2.10".parse().expect("should parse ip")), + user_agent: Some("ExampleBrowser/1.0"), + page_url: None, + region: Region::from_geo("us", Some("us-ca")), + hashed_email: None, + consent: ProviderConsent::default(), + publisher_domain: "www.example.com", + now: UnixTime::from_secs(1_760_000_000), + random: &ThreadRandom, + ec_id: Some("a".repeat(64) + ".ABC123"), + stored: BTreeMap::new(), + state_json: None, + dry_run: false, + } + } + + #[test] + fn declared_inputs_are_available_and_wanted_missing_ones_are_unavailable() { + let descriptor = descriptor( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_needs([Input::ClientIp, Input::Region]) + .with_wants([Input::PageUrl]), + ); + let signals = signals(); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Initial), &signals); + + assert_eq!(input.client_ip().expect("should read ip"), "192.0.2.10".parse::().expect("ip"), "ip"); + let region = input.region().expect("should read region"); + assert_eq!((region.country(), region.subdivision()), ("US", Some("CA")), "should normalize the region"); + assert!(region.is("US", "CA"), "should test country and subdivision"); + let error = input.page_url().expect_err("should be unavailable"); + assert!( + matches!(error.current_context(), IdentityProviderError::InputUnavailable { input: Input::PageUrl }), + "a declared but missing input is InputUnavailable" + ); + assert!(input.recorded_errors().is_empty(), "declared reads record nothing"); + } + + #[test] + fn undeclared_reads_are_recorded_even_when_ignored() { + let descriptor = descriptor(Resolves::new().with_endpoint("https://a.example.com")); + let signals = signals(); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Initial), &signals); + + let ignored = input.user_agent().ok(); + let state = input.state::(); + + assert!(ignored.is_none(), "should not return an undeclared input"); + assert!(state.is_err(), "state needs with_private_state"); + assert_eq!( + input.recorded_errors(), + vec![Access::Input(Input::UserAgent), Access::PrivateState], + "should record both undeclared reads" + ); + input.endpoint().expect("the declared endpoint is always available"); + } + + #[test] + fn state_decodes_and_a_bad_state_is_recorded() { + let descriptor = descriptor(Resolves::new().with_endpoint("https://a.example.com").with_private_state()); + let mut good = signals(); + good.state_json = Some(r#"{"signature":"s"}"#.to_owned()); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Refresh), &good); + let state: serde_json::Value = input.state().expect("should decode").expect("should be present"); + assert_eq!(state["signature"], "s", "should decode the state"); + assert!(!input.state_decode_failed(), "a good state is not a failure"); + + #[derive(serde::Deserialize)] + struct Typed { + #[allow(dead_code)] + signature: String, + } + let mut bad = signals(); + bad.state_json = Some(r#""a bare string""#.to_owned()); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Refresh), &bad); + let error = input.state::().expect_err("should not decode"); + assert!(matches!(error.current_context(), IdentityProviderError::InvalidState), "InvalidState"); + assert!(input.state_decode_failed(), "should record the failure"); + } + + #[test] + fn dry_run_is_never_true_for_stateful_providers() { + let stateful = descriptor(Resolves::new().with_endpoint("https://a.example.com").with_private_state()); + let stateless = descriptor(Resolves::new().with_endpoint("https://a.example.com")); + let mut dry = signals(); + dry.dry_run = true; + assert!(!ResolveInput::new(&stateful, kinds(&stateful, ResolveKind::Initial), &dry).is_dry_run(), "stateful"); + assert!(ResolveInput::new(&stateless, kinds(&stateless, ResolveKind::Initial), &dry).is_dry_run(), "stateless"); + } + + #[test] + fn ec_id_is_only_for_providers_declared_with_ec_id() { + let generic = descriptor(Resolves::new().with_host("a.example.com").with_ec_id()); + let other = descriptor(Resolves::new().with_host("a.example.com")); + let signals = signals(); + ResolveInput::new(&generic, kinds(&generic, ResolveKind::Initial), &signals) + .ec_id() + .expect("generic pull reads the EC ID"); + let input = ResolveInput::new(&other, kinds(&other, ResolveKind::Initial), &signals); + input.ec_id().expect_err("others may not"); + assert_eq!(input.recorded_errors().len(), 1, "should record the read"); + } + + #[test] + fn stored_ids_and_expiry() { + let record = crate::ec::kv_types::KvPartnerId { + uid: "u".to_owned(), + w: Some("p".to_owned()), + rv: Some("0123456789abcdef".to_owned()), + at: Some(10), + exp: Some(20), + ra: Some(15), + ru: None, + atype: Some(3), + ext: None, + }; + let stored = StoredId::from_record(&record); + assert_eq!(stored.writer().map(IdentityProviderId::as_str), Some("p"), "writer"); + assert!(!stored.is_expired(UnixTime::from_secs(19)), "not expired before exp"); + assert!(stored.is_expired(UnixTime::from_secs(20)), "expired at exp"); + assert_eq!(stored.refresh_at(), Some(UnixTime::from_secs(15)), "ra"); + } + + #[test] + fn hashed_email_debug_is_redacted() { + let email = HashedEmail::from_bytes([0xab; 32]); + assert_eq!(email.to_hex(), "ab".repeat(32), "hex"); + assert_eq!(format!("{email:?}"), "HashedEmail([redacted])", "should redact Debug"); + } + + fn kinds(descriptor: &ProviderDescriptor, kind: ResolveKind) -> BTreeMap { + descriptor.sources().iter().map(|source| (source.clone(), kind)).collect() + } + + #[test] + fn capture_input_exposes_path_and_never_the_request_body() { + let descriptor = ProviderDescriptor::builder("c".parse().expect("id")) + .with_source("example.com".parse().expect("source")) + .with_captures(Captures::new().with_path(http::Method::POST, "/integrations/c/api")) + .build() + .expect("should build"); + let response = crate::ec::identity::exchange::ProviderResponse::for_test(http::StatusCode::OK, b"{}"); + let consent = ProviderConsent::default(); + let input = CaptureInput::new( + &descriptor, + "/integrations/c/api", + &response, + &BTreeMap::new(), + &consent, + "www.example.com", + UnixTime::from_secs(1), + ); + assert_eq!(input.path(), "/integrations/c/api", "path"); + assert_eq!(input.primary_source().as_str(), "example.com", "source"); + } +} +``` + +`ProviderResponse::for_test` arrives in Task 10. Until then, comment out the last test and enable it in Task 10, Step 4. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::input` +Expected: compile errors. + +- [ ] **Step 3: Implement `input.rs`** + +```rust +//! What a provider may read for one call. + +use core::cell::RefCell; +use core::fmt; +use std::collections::{BTreeMap, BTreeSet}; +use std::net::IpAddr; + +use base64::Engine as _; +use base64::engine::general_purpose::STANDARD; +use error_stack::Report; +use rand::RngCore as _; +use serde::de::DeserializeOwned; +use url::Url; + +use super::descriptor::{Access, Input, ProviderDescriptor, ResolveKind}; +use super::exchange::ProviderResponse; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::resolution::IdentityProviderError; +use crate::consent::ConsentContext; +use crate::ec::kv_types::KvPartnerId; + + +/// The raw consent strings a provider forwards, and `gdpr_applies`. +/// +/// Providers never decide consent. The coordinator has already applied the +/// descriptor's policy. +#[derive(Clone, Default, PartialEq, Eq)] +pub struct ProviderConsent { + gdpr_applies: bool, + tc_string: Option, + gpp_string: Option, + gpp_section_ids: Option>, + us_privacy: Option, +} + +impl fmt::Debug for ProviderConsent { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderConsent") + .field("gdpr_applies", &self.gdpr_applies) + .field("tc_string", &self.tc_string.as_ref().map(|_| "[redacted]")) + .field("gpp_string", &self.gpp_string.as_ref().map(|_| "[redacted]")) + .field("gpp_section_ids", &self.gpp_section_ids) + .field("us_privacy", &self.us_privacy.as_ref().map(|_| "[redacted]")) + .finish() + } +} + +impl ProviderConsent { + /// The view of a request's consent context. + #[must_use] + pub(crate) fn from_context(context: &ConsentContext) -> Self { + Self { + gdpr_applies: context.gdpr_applies, + tc_string: context.raw_tc_string.clone(), + gpp_string: context.raw_gpp_string.clone(), + gpp_section_ids: context.gpp_section_ids.clone(), + us_privacy: context.raw_us_privacy.clone(), + } + } + + /// Whether GDPR applies to this request. + #[must_use] + pub fn gdpr_applies(&self) -> bool { + self.gdpr_applies + } + + /// The raw TCF string. + #[must_use] + pub fn raw_tc_string(&self) -> Option<&str> { + self.tc_string.as_deref() + } + + /// The raw GPP string. + #[must_use] + pub fn raw_gpp_string(&self) -> Option<&str> { + self.gpp_string.as_deref() + } + + /// The applicable GPP section IDs. + #[must_use] + pub fn gpp_section_ids(&self) -> Option<&[u16]> { + self.gpp_section_ids.as_deref() + } + + /// The raw US Privacy string. + #[must_use] + pub fn raw_us_privacy(&self) -> Option<&str> { + self.us_privacy.as_deref() + } +} +``` + +Check the real field names on `ConsentContext` (`consent/types.rs:109-148`): `grep -n "pub " crates/trusted-server-core/src/consent/types.rs | sed -n 1,60p`. Use them in `from_context`. If the section IDs are stored as another type, convert here. The rest of the file: + +```rust +/// Country and subdivision, upper case. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Region { + country: String, + subdivision: Option, +} + +impl Region { + /// Normalizes platform geo: upper case, and an ISO 3166-2 `US-CA` style + /// region loses its country prefix. `None` without a two-letter country. + #[must_use] + pub(crate) fn from_geo(country: &str, region: Option<&str>) -> Option { + let country = country.trim().to_ascii_uppercase(); + if country.len() != 2 || !country.bytes().all(|byte| byte.is_ascii_alphabetic()) { + return None; + } + let subdivision = region + .map(|value| value.trim().to_ascii_uppercase()) + .map(|value| match value.split_once('-') { + Some((prefix, rest)) if prefix == country => rest.to_owned(), + _ => value, + }) + .filter(|value| !value.is_empty() && value.bytes().all(|byte| byte.is_ascii_alphanumeric())); + Some(Self { country, subdivision }) + } + + /// ISO 3166-1 alpha-2, upper case, for example `"US"`. + #[must_use] + pub fn country(&self) -> &str { + &self.country + } + + /// The ISO 3166-2 subdivision without the country prefix, for example + /// `"CA"`. `None` when the platform does not know it. + #[must_use] + pub fn subdivision(&self) -> Option<&str> { + self.subdivision.as_deref() + } + + /// Whether this is `country` and `subdivision`. + #[must_use] + pub fn is(&self, country: &str, subdivision: &str) -> bool { + self.country == country && self.subdivision.as_deref() == Some(subdivision) + } +} + +/// The 32 raw bytes of a SHA-256 of a normalized email. Personal data. +#[derive(Clone, PartialEq, Eq)] +pub struct HashedEmail([u8; 32]); + +impl fmt::Debug for HashedEmail { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("HashedEmail([redacted])") + } +} + +impl HashedEmail { + /// From digest bytes. No code in v1 calls this outside tests. + #[must_use] + pub(crate) fn from_bytes(bytes: [u8; 32]) -> Self { + Self(bytes) + } + + /// The digest. + #[must_use] + pub fn as_bytes(&self) -> &[u8; 32] { + &self.0 + } + + /// Lowercase hex, as LiveRamp takes it. + #[must_use] + pub fn to_hex(&self) -> String { + hex::encode(self.0) + } + + /// Standard base64, as UID2 takes it. + #[must_use] + pub fn to_base64(&self) -> String { + STANDARD.encode(self.0) + } +} + +/// A stored record for one of this provider's sources, valid or expired. +#[derive(Clone, PartialEq)] +pub struct StoredId { + uid: String, + atype: Option, + ext: Option, + writer: Option, + acquired_at: Option, + expires_at: Option, + refresh_at: Option, +} + +impl fmt::Debug for StoredId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("StoredId") + .field("uid", &"[redacted]") + .field("writer", &self.writer) + .field("acquired_at", &self.acquired_at) + .field("expires_at", &self.expires_at) + .field("refresh_at", &self.refresh_at) + .finish_non_exhaustive() + } +} + +impl StoredId { + /// From a KV record. A writer that does not parse is treated as none. + #[must_use] + pub(crate) fn from_record(record: &KvPartnerId) -> Self { + Self { + uid: record.uid.clone(), + atype: record.atype, + ext: record.ext.clone(), + writer: record.w.as_deref().and_then(|writer| writer.parse().ok()), + acquired_at: record.at.map(UnixTime::from_secs), + expires_at: record.exp.map(UnixTime::from_secs), + refresh_at: record.ra.map(UnixTime::from_secs), + } + } + + /// The UID. + #[must_use] + pub fn uid(&self) -> &str { + &self.uid + } + + /// Provider-supplied `atype`. + #[must_use] + pub fn atype(&self) -> Option { + self.atype + } + + /// UID-level ext, as bidders read it in `uids[].ext`. + #[must_use] + pub fn ext(&self) -> Option<&serde_json::Value> { + self.ext.as_ref() + } + + /// The provider that wrote it. `None` for browser ingest and legacy rows. + #[must_use] + pub fn writer(&self) -> Option<&IdentityProviderId> { + self.writer.as_ref() + } + + /// When it was acquired or last refreshed. + #[must_use] + pub fn acquired_at(&self) -> Option { + self.acquired_at + } + + /// When it must stop being sent. + #[must_use] + pub fn expires_at(&self) -> Option { + self.expires_at + } + + /// When it is due for refresh. + #[must_use] + pub fn refresh_at(&self) -> Option { + self.refresh_at + } + + /// Whether `now` is at or past its expiry. + #[must_use] + pub fn is_expired(&self, now: UnixTime) -> bool { + self.expires_at.is_some_and(|expires| now >= expires) + } +} + +/// A source of random bytes. +pub(crate) trait RandomSource: Send + Sync { + /// Fills `bytes`. + fn fill(&self, bytes: &mut [u8]); +} + +/// The platform generator EC generation uses (`rand::thread_rng`). +pub(crate) struct ThreadRandom; + +impl RandomSource for ThreadRandom { + fn fill(&self, bytes: &mut [u8]) { + rand::thread_rng().fill_bytes(bytes); + } +} + +/// Deterministic bytes for tests: xorshift64* from a seed. +#[cfg(any(test, feature = "test-utils"))] +pub(crate) struct SeededRandom(std::sync::Mutex); + +#[cfg(any(test, feature = "test-utils"))] +impl SeededRandom { + pub(crate) fn new(seed: u64) -> Self { + Self(std::sync::Mutex::new(seed.max(1))) + } +} + +#[cfg(any(test, feature = "test-utils"))] +impl RandomSource for SeededRandom { + fn fill(&self, bytes: &mut [u8]) { + let mut state = self.0.lock().expect("should lock the seed"); + for byte in bytes { + *state ^= *state >> 12; + *state ^= *state << 25; + *state ^= *state >> 27; + *byte = (state.wrapping_mul(0x2545_F491_4F6C_DD1D) >> 56) as u8; + } + } +} + +/// Every signal the coordinator holds for one request. Crate-private: a +/// provider only ever sees a [`ResolveInput`] filtered by its descriptor. +pub(crate) struct ResolveSignals<'a> { + pub(crate) client_ip: Option, + pub(crate) user_agent: Option<&'a str>, + pub(crate) page_url: Option, + pub(crate) region: Option, + pub(crate) hashed_email: Option, + pub(crate) consent: ProviderConsent, + pub(crate) publisher_domain: &'a str, + pub(crate) now: UnixTime, + pub(crate) random: &'a dyn RandomSource, + pub(crate) ec_id: Option, + pub(crate) stored: BTreeMap, + pub(crate) state_json: Option, + pub(crate) dry_run: bool, +} + +/// What one provider may read for one call. +pub struct ResolveInput<'a> { + descriptor: &'a ProviderDescriptor, + kinds: BTreeMap, + signals: &'a ResolveSignals<'a>, + available: BTreeSet, + undeclared: RefCell>, + state_failed: RefCell, +} + +impl fmt::Debug for ResolveInput<'_> { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ResolveInput") + .field("provider", self.descriptor.id()) + .field("kinds", &self.kinds) + .finish_non_exhaustive() + } +} + +impl<'a> ResolveInput<'a> { + /// The input for one call. `kinds` holds only the sources to resolve. + #[must_use] + pub(crate) fn new( + descriptor: &'a ProviderDescriptor, + kinds: BTreeMap, + signals: &'a ResolveSignals<'a>, + ) -> Self { + let available = descriptor + .resolves() + .map(|resolves| resolves.declared_inputs()) + .unwrap_or_default(); + Self { + descriptor, + kinds, + signals, + available, + undeclared: RefCell::new(Vec::new()), + state_failed: RefCell::new(false), + } + } + + fn record(&self, access: Access) -> Report { + self.undeclared.borrow_mut().push(access); + Report::new(IdentityProviderError::UndeclaredInput { access }) + } + + fn declared(&self, input: Input, value: Option) -> Result> { + if !self.available.contains(&input) { + return Err(self.record(Access::Input(input))); + } + value.ok_or_else(|| Report::new(IdentityProviderError::InputUnavailable { input })) + } + + /// Undeclared reads recorded so far. The coordinator logs them as + /// `Config`, and the harness fails the test. + #[must_use] + pub(crate) fn recorded_errors(&self) -> Vec { + self.undeclared.borrow().clone() + } + + /// Whether a `state()` call failed to decode. + #[must_use] + pub(crate) fn state_decode_failed(&self) -> bool { + *self.state_failed.borrow() + } + + /// The sources to resolve this time, each with its own kind. + pub fn sources(&self) -> impl Iterator + '_ { + self.kinds.iter().map(|(source, kind)| (source, *kind)) + } + + /// Whether `source` is being resolved for the first time. + #[must_use] + pub fn is_initial(&self, source: &SourceDomain) -> bool { + self.kinds.get(source) == Some(&ResolveKind::Initial) + } + + /// Whether `source` is being refreshed. + #[must_use] + pub fn is_refresh(&self, source: &SourceDomain) -> bool { + self.kinds.get(source) == Some(&ResolveKind::Refresh) + } + + /// The descriptor's primary source. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + self.descriptor.primary_source() + } + + /// The stored record for one of this provider's sources. + #[must_use] + pub fn stored(&self, source: &SourceDomain) -> Option<&StoredId> { + self.signals.stored.get(source) + } + + /// This provider's private state, decoded. `Ok(None)` when there is none. + /// + /// # Errors + /// + /// `InvalidState` when the stored state does not decode as `T`. The + /// failure is recorded, and the coordinator clears the state whatever + /// the provider returns. `UndeclaredInput` without `with_private_state()`. + pub fn state(&self) -> Result, Report> { + if !self.descriptor.resolves().is_some_and(|resolves| resolves.has_private_state()) { + return Err(self.record(Access::PrivateState)); + } + let Some(raw) = self.signals.state_json.as_deref() else { + return Ok(None); + }; + serde_json::from_str(raw).map(Some).map_err(|_| { + *self.state_failed.borrow_mut() = true; + Report::new(IdentityProviderError::InvalidState) + }) + } + + /// True during the dry run (spec 5.2). Always false for a provider + /// declared `with_private_state()`. + #[must_use] + pub fn is_dry_run(&self) -> bool { + self.signals.dry_run + && !self.descriptor.resolves().is_some_and(|resolves| resolves.has_private_state()) + } + + /// The endpoint declared with `Resolves::with_endpoint`. + /// + /// # Errors + /// + /// `UndeclaredInput { access: Endpoint }` when none was declared. + pub fn endpoint(&self) -> Result<&Url, Report> { + self.descriptor.endpoint().ok_or_else(|| self.record(Access::Endpoint)) + } + + /// The consent strings to forward. + #[must_use] + pub fn consent(&self) -> &ProviderConsent { + &self.signals.consent + } + + /// The publisher's domain. + #[must_use] + pub fn publisher_domain(&self) -> &str { + self.signals.publisher_domain + } + + /// The request's time. + #[must_use] + pub fn now(&self) -> UnixTime { + self.signals.now + } + + /// Random bytes from the platform generator. Fixed by the harness. + #[must_use] + pub fn random_bytes(&self) -> [u8; N] { + let mut bytes = [0u8; N]; + self.signals.random.fill(&mut bytes); + bytes + } + + /// The platform's client address. + /// + /// # Errors + /// + /// `UndeclaredInput` when not declared, `InputUnavailable` when missing. + pub fn client_ip(&self) -> Result> { + self.declared(Input::ClientIp, self.signals.client_ip) + } + + /// The `User-Agent`. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn user_agent(&self) -> Result<&str, Report> { + self.declared(Input::UserAgent, self.signals.user_agent) + } + + /// The page URL, without query string or fragment. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn page_url(&self) -> Result<&Url, Report> { + self.declared(Input::PageUrl, self.signals.page_url.as_ref()) + } + + /// Country and subdivision. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn region(&self) -> Result<&Region, Report> { + self.declared(Input::Region, self.signals.region.as_ref()) + } + + /// A hashed email. Never available in v1. + /// + /// # Errors + /// + /// As [`Self::client_ip`]. + pub fn hashed_email(&self) -> Result<&HashedEmail, Report> { + self.declared(Input::HashedEmail, self.signals.hashed_email.as_ref()) + } + + /// The full EC ID. Only for a resolver declared `with_ec_id()`. + /// + /// # Errors + /// + /// `UndeclaredInput` otherwise. `InputUnavailable` (as `ClientIp`, the + /// nearest input) when the request has no EC ID. + pub(in crate::ec::identity) fn ec_id(&self) -> Result<&str, Report> { + if !self.descriptor.resolves().is_some_and(|resolves| resolves.gets_ec_id()) { + return Err(self.record(Access::PrivateState)); + } + self.signals + .ec_id + .as_deref() + .ok_or_else(|| Report::new(IdentityProviderError::InputUnavailable { input: Input::ClientIp })) + } +} + +/// What a capture may read from one proxied response. +pub struct CaptureInput<'a> { + descriptor: &'a ProviderDescriptor, + path: &'a str, + response: &'a ProviderResponse, + stored: &'a BTreeMap, + consent: &'a ProviderConsent, + publisher_domain: &'a str, + now: UnixTime, +} + +impl fmt::Debug for CaptureInput<'_> { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("CaptureInput") + .field("provider", self.descriptor.id()) + .field("path", &self.path) + .finish_non_exhaustive() + } +} + +impl<'a> CaptureInput<'a> { + /// The input for one capture. + #[must_use] + pub(crate) fn new( + descriptor: &'a ProviderDescriptor, + path: &'a str, + response: &'a ProviderResponse, + stored: &'a BTreeMap, + consent: &'a ProviderConsent, + publisher_domain: &'a str, + now: UnixTime, + ) -> Self { + Self { descriptor, path, response, stored, consent, publisher_domain, now } + } + + /// The proxied request path, without query string. + #[must_use] + pub fn path(&self) -> &str { + self.path + } + + /// The proxied response, decoded. + #[must_use] + pub fn response(&self) -> &ProviderResponse { + self.response + } + + /// The descriptor's primary source. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + self.descriptor.primary_source() + } + + /// Finds a declared source by a vendor's spelling. + #[must_use] + pub fn source_named(&self, name: &str) -> Option<&SourceDomain> { + self.descriptor.source_named(name) + } + + /// The stored record for a source. + #[must_use] + pub fn stored(&self, source: &SourceDomain) -> Option<&StoredId> { + self.stored.get(source) + } + + /// The stricter of the cookie consent and the request-body consent. + #[must_use] + pub fn consent(&self) -> &ProviderConsent { + self.consent + } + + /// The publisher's domain. + #[must_use] + pub fn publisher_domain(&self) -> &str { + self.publisher_domain + } + + /// The request's time. + #[must_use] + pub fn now(&self) -> UnixTime { + self.now + } +} +``` + +`CaptureInput::new` takes 7 arguments, which is the maximum `AGENTS.md` allows. Do not add an eighth. Use a struct if you need one. `hex` is already a dependency of core (check with `grep -n '^hex' crates/trusted-server-core/Cargo.toml`). If it is only a build dependency, add `hex = { workspace = true }` to `[dependencies]`. `ResolveKind` stays in `descriptor.rs` and is re-exported from `mod.rs`. Update `mod.rs`: + +```rust +pub mod input; +pub use input::{CaptureInput, HashedEmail, ProviderConsent, Region, ResolveInput, StoredId}; +``` + +Add `ec/identity/input.rs` to `migration_guards.rs`. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::input` +Expected: PASS for every test except the `capture_input_...` test, which waits for Task 10. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Add declared-input enforcement for identity providers" +``` + +### Task 10: Requests, responses and resolutions + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/exchange.rs` +- Modify: `crates/trusted-server-core/src/ec/identity/resolution.rs` (add `Resolution`, `ResolvedId`, `When`, `StateChange`) +- Modify: `mod.rs`, `migration_guards.rs` +- Test: both files + +**Interfaces:** + +- Produces: everything listed in spec 4.5, plus: + - `ProviderRequest::call_context_json(&self) -> Option<&str>` (crate-private) + - `ProviderRequest::has_cookie_header(&self) -> bool` (crate-private) + - `ProviderResponse::new(status: StatusCode, headers: HeaderMap, body: Vec, call_context: Option) -> Self` (crate-private) + - `ProviderResponse::for_test(status, body: &[u8]) -> Self` (test support) + - `Resolution::found_ids_by_source(&self)`: the first ID per source, plus the count of duplicates dropped (crate-private) + - `MAX_CALL_CONTEXT_BYTES = 1024`, `MAX_UID_EXT_BYTES = 256` + +- [ ] **Step 1: Write the failing tests** + +In `exchange.rs`: + +```rust +#[cfg(test)] +mod tests { + use http::{HeaderName, HeaderValue, Method, StatusCode}; + use serde_json::json; + + use super::*; + + fn url() -> Url { + Url::parse("https://api.example.com/v1/fetch").expect("should parse") + } + + #[test] + fn post_json_sets_body_and_content_type() { + let request = ProviderRequest::post_json(&url(), &json!({ "a": 1 })).expect("should build"); + assert_eq!(request.method(), Method::POST, "method"); + assert_eq!(request.headers()["content-type"], "application/json", "content type"); + assert_eq!(request.body(), br#"{"a":1}"#, "body"); + } + + #[test] + fn query_bearer_and_headers() { + let request = ProviderRequest::get(&url()) + .with_query("pid", "1 2") + .with_bearer(&Redacted::new("token".to_owned())) + .try_with_header("x-example", "v") + .expect("should accept a valid header"); + assert_eq!(request.url().as_str(), "https://api.example.com/v1/fetch?pid=1+2", "should encode the query"); + assert_eq!(request.headers()["authorization"], "Bearer token", "bearer"); + assert_eq!(request.headers()["x-example"], "v", "header"); + ProviderRequest::get(&url()).try_with_header("bad header", "v").expect_err("should reject a bad name"); + ProviderRequest::get(&url()).try_with_header("x", "bad\nvalue").expect_err("should reject a bad value"); + } + + #[test] + fn cookie_headers_are_flagged_from_either_helper() { + let typed = ProviderRequest::get(&url()) + .with_header(HeaderName::from_static("cookie"), HeaderValue::from_static("a=1")); + let dynamic = ProviderRequest::get(&url()).try_with_header("Cookie", "a=1").expect("should build"); + assert!(typed.has_cookie_header(), "typed helper"); + assert!(dynamic.has_cookie_header(), "dynamic helper"); + assert!(!ProviderRequest::get(&url()).has_cookie_header(), "control"); + } + + #[test] + fn call_context_round_trips_and_is_capped() { + let request = ProviderRequest::get(&url()).try_with_call_context(&json!({ "nonce": [1, 2] })).expect("ok"); + let response = ProviderResponse::new( + StatusCode::OK, + http::HeaderMap::new(), + Vec::new(), + request.call_context_json().map(str::to_owned), + ); + let context: serde_json::Value = response.call_context().expect("decode").expect("present"); + assert_eq!(context, json!({ "nonce": [1, 2] }), "should round-trip"); + + let error = ProviderRequest::get(&url()) + .try_with_call_context(&"x".repeat(MAX_CALL_CONTEXT_BYTES)) + .expect_err("should reject oversize"); + assert!(matches!(error.current_context(), IdentityProviderError::Config { .. }), "Config"); + } + + #[test] + fn success_helpers_check_status_and_encoding() { + let ok = ProviderResponse::for_test(StatusCode::OK, br#"{"uid":"u"}"#); + let value: serde_json::Value = ok.success_json().expect("should parse"); + assert_eq!(value["uid"], "u", "uid"); + + let rejected = ProviderResponse::for_test(StatusCode::FORBIDDEN, br#"{}"#); + let error = rejected.success_json::().expect_err("should reject"); + assert!( + matches!(error.current_context(), IdentityProviderError::Rejected { status } if *status == StatusCode::FORBIDDEN), + "Rejected with the status" + ); + rejected.json::().expect("json ignores the status"); + + let binary = ProviderResponse::for_test(StatusCode::OK, &[0xff, 0xfe]); + let error = binary.success_text().expect_err("should reject non-UTF-8"); + assert!(matches!(error.current_context(), IdentityProviderError::Malformed), "Malformed"); + } +} +``` + +In `resolution.rs`: + +```rust +#[cfg(test)] +mod tests { + use std::time::Duration; + + use serde_json::json; + + use super::*; + + fn source(name: &str) -> SourceDomain { + name.parse().expect("should parse") + } + + #[test] + fn empty_found_is_no_match() { + let resolution = Resolution::found([]); + assert!(resolution.is_no_match(), "should be no_match"); + assert!(resolution.ids().is_empty(), "no ids"); + } + + #[test] + fn found_keeps_the_first_id_per_source() { + let a = source("a.example.com"); + let resolution = Resolution::found([ResolvedId::new(&a, "first"), ResolvedId::new(&a, "second")]); + let (kept, dropped) = resolution.found_ids_by_source(); + assert_eq!(kept.len(), 1, "one per source"); + assert_eq!(kept[0].uid(), "first", "should keep the first"); + assert_eq!(dropped, 1, "should count the duplicate"); + } + + #[test] + fn invalidations_chain_and_state_changes_are_reported() { + let a = source("a.example.com"); + let b = source("b.example.com"); + let resolution = Resolution::found([ResolvedId::new(&a, "u")]) + .also_invalidate(&b) + .with_retry_in(Duration::from_secs(60)) + .replacing_state(&json!({ "signature": "s" })); + assert_eq!(resolution.invalidated(), &[b], "should invalidate b"); + assert_eq!(resolution.retry_in(), Some(Duration::from_secs(60)), "retry"); + assert_eq!(resolution.state_change(), StateChange::Replace(r#"{"signature":"s"}"#), "state"); + assert_eq!(Resolution::no_match().clearing_state().state_change(), StateChange::Clear, "clear"); + assert_eq!(Resolution::no_match().state_change(), StateChange::Keep, "keep"); + } + + #[test] + fn oversized_state_is_marked_invalid_not_dropped_silently() { + let resolution = Resolution::found([ResolvedId::new(&source("a.example.com"), "u")]) + .replacing_state(&"x".repeat(2000)); + assert!(resolution.state_rejected(), "should flag the state as rejected"); + } + + #[test] + fn oversized_ext_is_dropped_and_the_id_kept() { + let id = ResolvedId::new(&source("a.example.com"), "u").with_ext(json!({ "pad": "x".repeat(300) })); + assert_eq!(id.uid(), "u", "should keep the id"); + assert!(id.ext().is_none(), "should drop the ext"); + let small = ResolvedId::new(&source("a.example.com"), "u").with_ext(json!({ "linkType": 2 })); + assert_eq!(small.ext(), Some(&json!({ "linkType": 2 })), "control: a small ext is kept"); + } + + #[test] + fn opted_out_is_reported() { + assert!(Resolution::opted_out().is_opted_out(), "opted out"); + assert!(!Resolution::no_match().is_opted_out(), "control"); + } +} +``` + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::exchange` and `cargo test-fastly -- ec::identity::resolution` +Expected: compile errors. + +- [ ] **Step 3: Implement `exchange.rs`** + +```rust +//! What a provider sends and receives. The coordinator does all I/O. + +use core::fmt; + +use base64::Engine as _; +use error_stack::Report; +use http::header::{AUTHORIZATION, CONTENT_TYPE, COOKIE}; +use http::{HeaderMap, HeaderName, HeaderValue, Method, StatusCode}; +use serde::Serialize; +use serde::de::DeserializeOwned; +use url::Url; + +use super::resolution::IdentityProviderError; +use crate::redacted::Redacted; + +/// Maximum JSON bytes of a call context. +pub const MAX_CALL_CONTEXT_BYTES: usize = 1024; + +/// One call a provider wants made. +#[derive(Clone)] +pub struct ProviderRequest { + method: Method, + url: Url, + headers: HeaderMap, + body: Vec, + call_context: Option, +} + +impl fmt::Debug for ProviderRequest { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderRequest") + .field("method", &self.method) + .field("host", &self.url.host_str()) + .field("path", &self.url.path()) + .field("headers", &self.headers.keys().collect::>()) + .field("body_len", &self.body.len()) + .finish_non_exhaustive() + } +} + +impl ProviderRequest { + /// Any method, with no body. + #[must_use] + pub fn new(method: Method, url: &Url) -> Self { + Self { method, url: url.clone(), headers: HeaderMap::new(), body: Vec::new(), call_context: None } + } + + /// A `GET`. + #[must_use] + pub fn get(url: &Url) -> Self { + Self::new(Method::GET, url) + } + + /// A `POST` with a JSON body. + /// + /// # Errors + /// + /// `Config` when `body` does not serialize. + pub fn post_json(url: &Url, body: &impl Serialize) -> Result> { + let bytes = serde_json::to_vec(body) + .map_err(|_| IdentityProviderError::config("request body does not serialize"))?; + Self::new(Method::POST, url).with_body("application/json", bytes) + } + + /// A `POST` with a `text/plain` body. + #[must_use] + pub fn post_text(url: &Url, body: &str) -> Self { + let mut request = Self::new(Method::POST, url); + request.headers.insert(CONTENT_TYPE, HeaderValue::from_static("text/plain")); + request.body = body.as_bytes().to_vec(); + request + } + + /// Sets the body and its `Content-Type`. + /// + /// # Errors + /// + /// `Config` when `content_type` is not a valid header value. + pub fn with_body(mut self, content_type: &str, body: Vec) -> Result> { + let value = HeaderValue::from_str(content_type) + .map_err(|_| IdentityProviderError::config("invalid content type"))?; + self.headers.insert(CONTENT_TYPE, value); + self.body = body; + Ok(self) + } + + /// Appends a query parameter, URL-encoded. + #[must_use] + pub fn with_query(mut self, name: &str, value: &str) -> Self { + self.url.query_pairs_mut().append_pair(name, value); + self + } + + /// Sets a header. A `Cookie` header makes the coordinator refuse the call. + #[must_use] + pub fn with_header(mut self, name: HeaderName, value: HeaderValue) -> Self { + self.headers.insert(name, value); + self + } + + /// Sets a header from strings. + /// + /// # Errors + /// + /// `Config` when the name or value is invalid. + pub fn try_with_header(self, name: &str, value: &str) -> Result> { + let name = HeaderName::from_bytes(name.as_bytes()) + .map_err(|_| IdentityProviderError::config("invalid header name"))?; + let value = HeaderValue::from_str(value) + .map_err(|_| IdentityProviderError::config("invalid header value"))?; + Ok(self.with_header(name, value)) + } + + /// Adds `Authorization: Bearer `. + #[must_use] + pub fn with_bearer(mut self, token: &Redacted) -> Self { + if let Ok(mut value) = HeaderValue::from_str(&format!("Bearer {}", token.expose())) { + value.set_sensitive(true); + self.headers.insert(AUTHORIZATION, value); + } + self + } + + /// Data `response` needs from this call. Kept in memory, never sent. + /// + /// # Errors + /// + /// `Config` when it does not serialize or is over 1,024 bytes of JSON. + pub fn try_with_call_context(mut self, context: &impl Serialize) -> Result> { + let json = serde_json::to_string(context) + .map_err(|_| IdentityProviderError::config("call context does not serialize"))?; + if json.len() > MAX_CALL_CONTEXT_BYTES { + return Err(IdentityProviderError::config("call context exceeds 1024 bytes")); + } + self.call_context = Some(json); + Ok(self) + } + + /// The method. + #[must_use] + pub fn method(&self) -> &Method { + &self.method + } + + /// The URL. + #[must_use] + pub fn url(&self) -> &Url { + &self.url + } + + /// The headers. + #[must_use] + pub fn headers(&self) -> &HeaderMap { + &self.headers + } + + /// The body. + #[must_use] + pub fn body(&self) -> &[u8] { + &self.body + } + + /// The call context JSON. + #[must_use] + pub(crate) fn call_context_json(&self) -> Option<&str> { + self.call_context.as_deref() + } + + /// Whether the provider set a `Cookie` header. + #[must_use] + pub(crate) fn has_cookie_header(&self) -> bool { + self.headers.contains_key(COOKIE) + } +} + +/// A buffered, decoded provider response. +#[derive(Clone)] +pub struct ProviderResponse { + status: StatusCode, + headers: HeaderMap, + body: Vec, + call_context: Option, +} + +impl fmt::Debug for ProviderResponse { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderResponse") + .field("status", &self.status) + .field("body_len", &self.body.len()) + .finish_non_exhaustive() + } +} + +impl ProviderResponse { + /// Built by the coordinator after buffering and decoding. + #[must_use] + pub(crate) fn new(status: StatusCode, headers: HeaderMap, body: Vec, call_context: Option) -> Self { + Self { status, headers, body, call_context } + } + + /// A response for tests. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn for_test(status: StatusCode, body: &[u8]) -> Self { + Self::new(status, HeaderMap::new(), body.to_vec(), None) + } + + /// The status. + #[must_use] + pub fn status(&self) -> StatusCode { + self.status + } + + /// A header as a string, if present and visible ASCII. + #[must_use] + pub fn header(&self, name: &str) -> Option<&str> { + self.headers.get(name).and_then(|value| value.to_str().ok()) + } + + /// The decoded body. + #[must_use] + pub fn body(&self) -> &[u8] { + &self.body + } + + /// Parses the body as JSON, whatever the status. + /// + /// # Errors + /// + /// `Malformed` when it does not parse. + pub fn json(&self) -> Result> { + serde_json::from_slice(&self.body).map_err(|_| IdentityProviderError::malformed()) + } + + /// Requires 2xx, then parses JSON. + /// + /// # Errors + /// + /// `Rejected` for a non-2xx status, `Malformed` for a bad body. + pub fn success_json(&self) -> Result> { + self.require_success()?; + self.json() + } + + /// Requires 2xx and a UTF-8 body. + /// + /// # Errors + /// + /// `Rejected` for a non-2xx status, `Malformed` for non-UTF-8. + pub fn success_text(&self) -> Result<&str, Report> { + self.require_success()?; + core::str::from_utf8(&self.body).map_err(|_| IdentityProviderError::malformed()) + } + + /// The context the request carried, decoded. + /// + /// # Errors + /// + /// `Config` when it does not decode as `T`. + pub fn call_context(&self) -> Result, Report> { + self.call_context + .as_deref() + .map(|json| { + serde_json::from_str(json) + .map_err(|_| IdentityProviderError::config("call context does not decode")) + }) + .transpose() + } + + fn require_success(&self) -> Result<(), Report> { + if self.status.is_success() { + Ok(()) + } else { + Err(IdentityProviderError::rejected(self.status)) + } + } +} +``` + +Check where `Redacted` lives (`grep -rn "pub struct Redacted" crates/trusted-server-core/src`) and fix the import. Drop the unused `base64` import if clippy flags it. + +- [ ] **Step 4: Implement the rest of `resolution.rs`** + +Append to the file from Task 5: + +```rust +use std::time::Duration; + +use serde::Serialize; + +use super::ids::{SourceDomain, UnixTime}; +use crate::ec::kv_types::{MAX_PARTNER_EXT_BYTES, MAX_PROVIDER_STATE_BYTES}; + +/// A time as a provider gave it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum When { + /// Relative to the request's `now`. + In(Duration), + /// Absolute. + At(UnixTime), +} + +impl When { + /// Resolves against `now`. + #[must_use] + pub(crate) fn resolve(self, now: UnixTime) -> UnixTime { + match self { + Self::In(delay) => now + delay, + Self::At(at) => at, + } + } +} + +/// One ID a provider returned. +#[derive(Clone, PartialEq)] +pub struct ResolvedId { + source: SourceDomain, + uid: String, + atype: Option, + ext: Option, + expiry: Option, + refresh: Option, + refresh_until: Option, +} + +impl core::fmt::Debug for ResolvedId { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("ResolvedId") + .field("source", &self.source) + .field("uid", &"[redacted]") + .field("expiry", &self.expiry) + .finish_non_exhaustive() + } +} + +impl ResolvedId { + /// An ID for `source`. + #[must_use] + pub fn new(source: &SourceDomain, uid: impl Into) -> Self { + Self { + source: source.clone(), + uid: uid.into(), + atype: None, + ext: None, + expiry: None, + refresh: None, + refresh_until: None, + } + } + + /// Sets `atype`. + #[must_use] + pub fn with_atype(mut self, atype: i32) -> Self { + self.atype = Some(atype); + self + } + + /// UID-level ext. Over 256 serialized bytes it is dropped and the ID kept. + #[must_use] + pub fn with_ext(mut self, ext: serde_json::Value) -> Self { + let fits = serde_json::to_vec(&ext).is_ok_and(|bytes| bytes.len() <= MAX_PARTNER_EXT_BYTES); + self.ext = fits.then_some(ext); + if !fits { + log::debug!("Dropping oversized ext for source {}", self.source); + } + self + } + + /// Expires after `delay`. + #[must_use] + pub fn with_expires_in(mut self, delay: Duration) -> Self { + self.expiry = Some(When::In(delay)); + self + } + + /// Expires at `at`. + #[must_use] + pub fn with_expires_at(mut self, at: UnixTime) -> Self { + self.expiry = Some(When::At(at)); + self + } + + /// Due for refresh after `delay`. + #[must_use] + pub fn with_refresh_in(mut self, delay: Duration) -> Self { + self.refresh = Some(When::In(delay)); + self + } + + /// Due for refresh at `at`. + #[must_use] + pub fn with_refresh_at(mut self, at: UnixTime) -> Self { + self.refresh = Some(When::At(at)); + self + } + + /// When this ID stops being refreshable. Only for a provider declared + /// `with_private_state()`; the coordinator ignores it otherwise. + #[must_use] + pub fn with_refresh_until(mut self, at: UnixTime) -> Self { + self.refresh_until = Some(at); + self + } + + /// The source. + #[must_use] + pub fn source(&self) -> &SourceDomain { + &self.source + } + + /// The UID. + #[must_use] + pub fn uid(&self) -> &str { + &self.uid + } + + /// `atype`. + #[must_use] + pub fn atype(&self) -> Option { + self.atype + } + + /// UID-level ext. + #[must_use] + pub fn ext(&self) -> Option<&serde_json::Value> { + self.ext.as_ref() + } + + /// The expiry as given. + #[must_use] + pub fn expiry(&self) -> Option { + self.expiry + } + + /// The refresh hint as given. + #[must_use] + pub fn refresh(&self) -> Option { + self.refresh + } + + /// The refresh limit. + #[must_use] + pub fn refresh_until(&self) -> Option { + self.refresh_until + } +} + +/// How a result changes the provider's private state. +#[derive(Debug, PartialEq, Eq)] +pub enum StateChange<'a> { + /// Leave it. + Keep, + /// Replace it with this JSON. + Replace(&'a str), + /// Remove it. + Clear, +} + +#[derive(Debug, Clone, PartialEq)] +enum StateOp { + Keep, + Replace(String), + Clear, + Rejected, +} + +/// What one provider call produced. +#[derive(Debug, Clone, PartialEq)] +pub struct Resolution { + ids: Vec, + invalidated: Vec, + opted_out: bool, + retry_in: Option, + state: StateOp, +} + +impl Resolution { + fn empty() -> Self { + Self { ids: Vec::new(), invalidated: Vec::new(), opted_out: false, retry_in: None, state: StateOp::Keep } + } + + /// One or more IDs, at most one per source. An empty set is `no_match`. + #[must_use] + pub fn found(ids: impl IntoIterator) -> Self { + Self { ids: ids.into_iter().collect(), ..Self::empty() } + } + + /// Nothing recognized, or a capture held nothing. + #[must_use] + pub fn no_match() -> Self { + Self::empty() + } + + /// The stored ID for `source` is no longer valid. + #[must_use] + pub fn invalidate(source: &SourceDomain) -> Self { + Self { invalidated: vec![source.clone()], ..Self::empty() } + } + + /// The provider reports the user opted out. + #[must_use] + pub fn opted_out() -> Self { + Self { opted_out: true, ..Self::empty() } + } + + /// Also invalidates `source`. + #[must_use] + pub fn also_invalidate(mut self, source: &SourceDomain) -> Self { + if !self.invalidated.contains(source) { + self.invalidated.push(source.clone()); + } + self + } + + /// The earliest the coordinator may call again. + #[must_use] + pub fn with_retry_in(mut self, delay: Duration) -> Self { + self.retry_in = Some(delay); + self + } + + /// Replaces the provider's private state. Over 1,024 bytes of JSON, or + /// not serializable, the positive part of the result is rejected whole. + #[must_use] + pub fn replacing_state(mut self, state: &impl Serialize) -> Self { + self.state = match serde_json::to_string(state) { + Ok(json) if json.len() <= MAX_PROVIDER_STATE_BYTES => StateOp::Replace(json), + _ => StateOp::Rejected, + }; + self + } + + /// Clears the provider's private state. + #[must_use] + pub fn clearing_state(mut self) -> Self { + self.state = StateOp::Clear; + self + } + + /// The returned IDs, as given. + #[must_use] + pub fn ids(&self) -> &[ResolvedId] { + &self.ids + } + + /// Invalidated sources. + #[must_use] + pub fn invalidated(&self) -> &[SourceDomain] { + &self.invalidated + } + + /// Whether nothing was found, invalidated or opted out. + #[must_use] + pub fn is_no_match(&self) -> bool { + self.ids.is_empty() && self.invalidated.is_empty() && !self.opted_out + } + + /// Whether the provider reported an opt-out. + #[must_use] + pub fn is_opted_out(&self) -> bool { + self.opted_out + } + + /// The retry delay. + #[must_use] + pub fn retry_in(&self) -> Option { + self.retry_in + } + + /// The state change. + #[must_use] + pub fn state_change(&self) -> StateChange<'_> { + match &self.state { + StateOp::Keep | StateOp::Rejected => StateChange::Keep, + StateOp::Replace(json) => StateChange::Replace(json), + StateOp::Clear => StateChange::Clear, + } + } + + /// Whether `replacing_state` was given an oversized or unserializable + /// value. The coordinator then rejects the positive part (spec 4.5, 8). + #[must_use] + pub(crate) fn state_rejected(&self) -> bool { + self.state == StateOp::Rejected + } + + /// Whether this result changes state at all. + #[must_use] + pub(crate) fn changes_state(&self) -> bool { + self.state != StateOp::Keep + } + + /// The first ID per source, in order, and how many duplicates were dropped. + #[must_use] + pub(crate) fn found_ids_by_source(&self) -> (Vec<&ResolvedId>, usize) { + let mut seen = std::collections::BTreeSet::new(); + let mut kept = Vec::new(); + let mut dropped = 0; + for id in &self.ids { + if seen.insert(id.source()) { + kept.push(id); + } else { + dropped += 1; + } + } + (kept, dropped) + } +} +``` + +Add `exchange.rs` to `mod.rs` (`pub mod exchange; pub use exchange::{ProviderRequest, ProviderResponse};`) and `pub use resolution::{Resolution, ResolvedId, StateChange, When};`. Add `exchange.rs` to `migration_guards.rs`. Uncomment the capture test from Task 9. + +- [ ] **Step 5: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Add provider requests, responses and resolutions" +``` + +### Task 11: The `IdentityProvider` trait + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/provider.rs` +- Modify: `mod.rs`, `migration_guards.rs` +- Test: `crates/trusted-server-core/src/ec/identity/provider.rs` + +**Interfaces:** + +- Consumes: Tasks 8 to 10 +- Produces: `pub trait IdentityProvider: Send + Sync` with `descriptor`, `request`, `response` and `capture` exactly as spec 4.2 shows, and `pub trait IdentityProviderConfig: IntegrationConfig { fn build_descriptor(&self) -> Result>; }`. Also the compile-time guard `const _: Option<&dyn IdentityProvider> = None;`. + +- [ ] **Step 1: Write the failing test** + +```rust +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use http::Method; + + use super::*; + use crate::ec::identity::descriptor::Captures; + use crate::ec::identity::exchange::ProviderResponse; + use crate::ec::identity::ids::UnixTime; + use crate::ec::identity::input::{CaptureInput, ProviderConsent}; + + struct DescriptorOnly(ProviderDescriptor); + + impl IdentityProvider for DescriptorOnly { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } + } + + #[test] + fn default_capability_methods_return_the_not_implemented_marker() { + let provider = DescriptorOnly( + ProviderDescriptor::builder("p".parse().expect("id")) + .with_source("example.com".parse().expect("source")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .build() + .expect("should build"), + ); + let boxed: Box = Box::new(provider); + let response = ProviderResponse::for_test(http::StatusCode::OK, b"{}"); + let consent = ProviderConsent::default(); + let stored = BTreeMap::new(); + let input = CaptureInput::new( + boxed.descriptor(), + "/integrations/p/api", + &response, + &stored, + &consent, + "www.example.com", + UnixTime::from_secs(1), + ); + + let error = boxed.capture(&input).expect_err("should be the marker"); + assert!(error.current_context().is_not_implemented(), "should carry the marker"); + assert!( + !IdentityProviderError::config("other").current_context().is_not_implemented(), + "control: an ordinary Config error is not the marker" + ); + } +} +``` + +- [ ] **Step 2: Run and see it fail** + +Run: `cargo test-fastly -- ec::identity::provider` +Expected: compile error. + +- [ ] **Step 3: Implement** + +Copy the trait and config trait verbatim from spec 4.2, with these imports: + +```rust +//! The identity provider trait. + +use error_stack::Report; + +use super::descriptor::ProviderDescriptor; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::input::{CaptureInput, ResolveInput}; +use super::resolution::{IdentityProviderError, Resolution}; +use crate::settings::IntegrationConfig; + +// ... trait IdentityProvider and trait IdentityProviderConfig exactly as spec 4.2 ... + +// Keeps the trait object-safe. Fails to compile if a method gains generics. +const _: Option<&dyn IdentityProvider> = None; +``` + +Check where `IntegrationConfig` lives (`grep -rn "pub trait IntegrationConfig" crates/trusted-server-core/src`) and fix the path. Add `pub mod provider; pub use provider::{IdentityProvider, IdentityProviderConfig};` to `mod.rs`, and the file to `migration_guards.rs`. + +- [ ] **Step 4: Run and see it pass** + +Run: `cargo test-fastly -- ec::identity` and `cargo doc --no-deps --all-features -p trusted-server-core` +Expected: PASS, with no broken intra-doc links. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Add the IdentityProvider trait" +``` + +### Task 12: Consent policy evaluation with the signals TS decodes today + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/policy.rs` +- Modify: `crates/trusted-server-core/src/consent/tcf.rs` (add `has_vendor_consent`) +- Modify: `mod.rs`, `migration_guards.rs` +- Test: `policy.rs`, `consent/tcf.rs` + +**Interfaces:** + +- Consumes: `ConsentPolicy` (Task 8), `ConsentContext`, `allows_eid_persistence`, `ec_consent_granted`, and the TCF and GPP decoded forms in `consent/` +- Produces: + - `pub(crate) struct PolicySignals { dnt: bool }`. DNT is not in `ConsentContext`, so the caller passes it. + - `pub(crate) fn policy_allows(policy: &ConsentPolicy, consent: &ConsentContext, signals: PolicySignals) -> bool` + - `pub(crate) fn tcf_applies(consent: &ConsentContext) -> bool`, the same test `consent_allows_server_side_auction` uses (`consent/mod.rs:298-308`) + - `pub(crate) fn has_vendor_consent(tcf: &DecodedTcf, vendor_id: u16) -> bool` in `consent/tcf.rs` + +Part 2 (Task 34) extends `policy_allows` with the sharing and targeted-advertising opt-outs, GPP section 6, and the GPC and known-child fields. Until then, a strict policy blocks on the signals already decoded: TCF purposes and vendor, the US sale opt-out (GPP and US Privacy), `Sec-GPC`, DNT, and any raw string present without a decoded form. + +- [ ] **Step 1: Write the failing tests** + +Build `ConsentContext` values the way existing consent tests do (`grep -n "fn .*consent_context\|ConsentContext {" crates/trusted-server-core/src/consent/mod.rs | head`). Reuse their helper. Then: + +```rust +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::descriptor::{ConsentPolicy, UsOptOut}; + + #[test] + fn strict_blocks_each_signal_it_decodes() { + let strict = ConsentPolicy::strict(); + let allowed = consent_with(Signals::default()); + assert!(policy_allows(&strict, &allowed, PolicySignals { dnt: false }), "control: nothing set"); + + for (name, signals, dnt) in [ + ("gdpr without purpose 4", Signals { gdpr: true, purposes: vec![1], ..Signals::default() }, false), + ("us sale opt-out", Signals { us_sale_opt_out: true, ..Signals::default() }, false), + ("gpc", Signals { gpc: true, ..Signals::default() }, false), + ("dnt", Signals::default(), true), + ("undecodable tc string", Signals { raw_tc_without_decode: true, ..Signals::default() }, false), + ] { + assert!( + !policy_allows(&strict, &consent_with(signals), PolicySignals { dnt }), + "strict should block: {name}" + ); + } + } + + #[test] + fn relaxations_and_vendor_consent() { + let gpc = consent_with(Signals { gpc: true, ..Signals::default() }); + assert!(policy_allows(&ConsentPolicy::strict().allowing_gpc(), &gpc, PolicySignals { dnt: false }), "gpc relaxed"); + let sale = consent_with(Signals { us_sale_opt_out: true, ..Signals::default() }); + assert!( + policy_allows(&ConsentPolicy::strict().allowing_us_opt_out(UsOptOut::Sale), &sale, PolicySignals { dnt: false }), + "sale relaxed" + ); + let gdpr = consent_with(Signals { gdpr: true, purposes: vec![1, 4], vendors: vec![9999], ..Signals::default() }); + assert!(policy_allows(&ConsentPolicy::strict().with_tcf_vendor(9999), &gdpr, PolicySignals { dnt: false }), "vendor granted"); + assert!(!policy_allows(&ConsentPolicy::strict().with_tcf_vendor(1234), &gdpr, PolicySignals { dnt: false }), "vendor missing"); + } + + #[test] + fn eid_persistence_only_ignores_provider_signals() { + let consent = consent_with(Signals { gpc: true, us_sale_opt_out: true, ..Signals::default() }); + assert!( + policy_allows(&ConsentPolicy::eid_persistence_only(), &consent, PolicySignals { dnt: true }), + "generic pull's parity policy checks only the global gates, which the coordinator applies separately" + ); + } +} +``` + +`Signals` and `consent_with` are a small test helper you write in this module. It builds a `ConsentContext` with GDPR applying, the given TCF purposes and vendor consents decoded, the US sale opt-out decoded from a US Privacy string `1YYN`, `gpc` set, and, for `raw_tc_without_decode`, `raw_tc_string = Some(..)` with no decoded TCF. Build it on whatever constructor the consent tests use. Read `consent/types.rs` first. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::policy` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Add to `consent/tcf.rs`, next to the purpose-consent helpers at about line 145: + +```rust +/// Whether the TC string grants consent to an IAB Global Vendor List vendor. +#[must_use] +pub(crate) fn has_vendor_consent(tcf: &TcfConsent, vendor_id: u16) -> bool { + tcf.vendor_consents.contains(&vendor_id) +} +``` + +Use the real decoded type name and its vendor-consent field. Read the struct at the top of `tcf.rs`. If vendor consents are stored as a bitfield, index into it instead. Add a unit test next to the existing TCF tests that decodes a fixture TC string already used in that file and asserts one vendor present and one absent. + +`policy.rs`: + +```rust +//! Evaluates a provider's [`ConsentPolicy`] against a request's consent. + +use super::descriptor::{ConsentPolicy, UsOptOut}; +use crate::consent::ConsentContext; +use crate::consent::jurisdiction::Jurisdiction; + +/// Signals the consent context does not carry. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) struct PolicySignals { + /// `DNT: 1` was present. + pub(crate) dnt: bool, +} + +/// Where TCF applies: `gdpr_applies`, or a jurisdiction of GDPR or unknown. +#[must_use] +pub(crate) fn tcf_applies(consent: &ConsentContext) -> bool { + consent.gdpr_applies || matches!(consent.jurisdiction, Jurisdiction::Gdpr | Jurisdiction::Unknown) +} + +/// Whether `policy` allows a provider under this request's consent. The +/// global EC and EID-persistence gates are checked separately (eligibility +/// steps 1 and 2) and are not repeated here. +#[must_use] +pub(crate) fn policy_allows(policy: &ConsentPolicy, consent: &ConsentContext, signals: PolicySignals) -> bool { + if policy.is_eid_persistence_only() { + return true; + } + // Fail closed: a raw string with no decoded form blocks. + if (consent.raw_tc_string.is_some() && consent.tcf.is_none()) + || (consent.raw_gpp_string.is_some() && consent.gpp.is_none()) + || (consent.raw_us_privacy.is_some() && consent.us_privacy.is_none()) + { + return false; + } + if tcf_applies(consent) { + let Some(tcf) = consent.tcf.as_ref() else { + return false; + }; + if !policy.tcf_purposes().iter().all(|purpose| tcf.has_purpose_consent(*purpose)) { + return false; + } + if let Some(vendor) = policy.tcf_vendor() + && !crate::consent::tcf::has_vendor_consent(tcf, vendor) + { + return false; + } + } + if !policy.allows_us_opt_out(UsOptOut::Sale) && crate::consent::us_sale_opted_out(consent) { + return false; + } + if !policy.allows_gpc() && consent.gpc { + return false; + } + if !policy.allows_dnt() && signals.dnt { + return false; + } + true +} +``` + +Every `consent.*` field name and helper above (`tcf`, `gpp`, `us_privacy`, `gpc`, `jurisdiction`, `has_purpose_consent`, `us_sale_opted_out`) must be matched to the real API in `consent/`. Run `grep -n "pub fn\|pub(crate) fn\|pub [a-z_]*:" crates/trusted-server-core/src/consent/*.rs` and use what exists. If there is no single US-sale helper, combine `decode_us_sale_opt_out` (`gpp.rs:125`) and the US Privacy check (`us_privacy.rs:50-63`) in a private `fn us_sale_opt_out(consent: &ConsentContext) -> bool` in this file. Do not change any existing consent function's behaviour. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::policy` and `cargo test-fastly -- consent::` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/consent/tcf.rs crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Evaluate provider consent policies against decoded consent" +``` + +### Task 13: Revisions and keyed hashes + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/revision.rs` +- Modify: `mod.rs`, `migration_guards.rs` +- Test: `revision.rs` + +**Interfaces:** + +- Produces: + - `pub(crate) struct IdentityKeys`, derived once per settings load, with `pub(crate) fn from_settings(settings: &Settings) -> Self`. It holds an HMAC key for each of: `revision` (legacy `pv` fallback), `invalidated` (`ps.iv`), `visitor` (rate-limit key), `fingerprint` (consent), and the AES keys `cookie_s` and `cookie_r` (Task 16 uses them). Labels: `trusted-server/identity/revision/v1`, `trusted-server/identity/iv/v1`, `trusted-server/identity/visitor/v1`, `trusted-server/identity/fingerprint/v1`, `trusted-server/ts-eids-s/key/v1`, `trusted-server/ts-eids-r/key/v1`. + - `pub(crate) struct Revision(String)`: 16 lowercase hex digits, with `as_str` + - `pub(crate) fn new_revision(random: &dyn RandomSource) -> Revision`: 8 random bytes as hex + - `pub(crate) fn record_revision(keys: &IdentityKeys, record: &KvPartnerId) -> Revision`: `rv` when present, else the first 16 hex digits of HMAC(`revision` key, `uid \0 at`) + - `pub(crate) fn invalidated_uid_hash(keys: &IdentityKeys, provider: &IdentityProviderId, uid: &str) -> String`: 16 hex + - `pub(crate) fn visitor_key(keys: &IdentityKeys, ec_id: &str) -> String`: 16 hex + - `pub(crate) fn consent_fingerprint(keys: &IdentityKeys, bits: &str) -> String`: 16 hex. Task 34 builds `bits`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::input::SeededRandom; + use crate::ec::kv_types::KvPartnerId; + use crate::test_support::tests::create_test_settings; + + fn keys() -> IdentityKeys { + IdentityKeys::from_settings(&create_test_settings()) + } + + #[test] + fn new_revisions_are_hex_and_differ() { + let random = SeededRandom::new(7); + let first = new_revision(&random); + let second = new_revision(&random); + assert_eq!(first.as_str().len(), 16, "16 hex digits"); + assert!(first.as_str().bytes().all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase()), "lowercase hex"); + assert_ne!(first, second, "two writes of the same UID get different revisions"); + } + + #[test] + fn record_revision_prefers_rv_and_hashes_legacy_records() { + let keys = keys(); + let with_rv = KvPartnerId { rv: Some("0123456789abcdef".to_owned()), ..KvPartnerId::new("u") }; + assert_eq!(record_revision(&keys, &with_rv).as_str(), "0123456789abcdef", "should use rv"); + + let legacy_a = KvPartnerId::browser("u", Some(10)); + let legacy_b = KvPartnerId::browser("u", Some(11)); + assert_eq!(record_revision(&keys, &legacy_a), record_revision(&keys, &legacy_a.clone()), "stable"); + assert_ne!(record_revision(&keys, &legacy_a), record_revision(&keys, &legacy_b), "at changes it"); + } + + #[test] + fn hashes_are_keyed_and_namespaced() { + let keys = keys(); + let provider: IdentityProviderId = "p".parse().expect("id"); + let iv = invalidated_uid_hash(&keys, &provider, "u"); + let visitor = visitor_key(&keys, "u"); + assert_eq!(iv.len(), 16, "iv length"); + assert_ne!(iv, visitor, "different labels give different hashes for the same input"); + + let mut other_settings = create_test_settings(); + other_settings.ec.passphrase = crate::redacted::Redacted::new("another-example-passphrase-value".to_owned()); + let other = IdentityKeys::from_settings(&other_settings); + assert_ne!(invalidated_uid_hash(&other, &provider, "u"), iv, "keyed by the passphrase"); + } +} +``` + +Find the settings test helper with `grep -rn "fn create_test_settings" crates/trusted-server-core/src`, and fix the import path to match. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::revision` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +```rust +//! Record revisions and keyed hashes of identifiers. + +use hmac::{Hmac, Mac as _}; +use sha2::Sha256; + +use super::ids::IdentityProviderId; +use super::input::RandomSource; +use crate::ec::kv_types::KvPartnerId; +use crate::settings::Settings; + +type HmacSha256 = Hmac; + +/// Keys derived from `ec.passphrase`, once per settings load. +#[derive(Clone)] +pub(crate) struct IdentityKeys { + revision: [u8; 32], + invalidated: [u8; 32], + visitor: [u8; 32], + fingerprint: [u8; 32], + pub(crate) cookie_s: [u8; 32], + pub(crate) cookie_r: [u8; 32], +} + +impl core::fmt::Debug for IdentityKeys { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter.write_str("IdentityKeys([redacted])") + } +} + +fn derive(passphrase: &[u8], label: &[u8]) -> [u8; 32] { + let mut mac = HmacSha256::new_from_slice(passphrase).expect("should accept any HMAC key length"); + mac.update(label); + mac.finalize().into_bytes().into() +} + +fn keyed_hex16(key: &[u8; 32], parts: &[&[u8]]) -> String { + let mut mac = HmacSha256::new_from_slice(key).expect("should accept a 32-byte key"); + for (index, part) in parts.iter().enumerate() { + if index > 0 { + mac.update(b"\0"); + } + mac.update(part); + } + hex::encode(&mac.finalize().into_bytes()[..8]) +} + +impl IdentityKeys { + /// Derives every identity key from `ec.passphrase`. + #[must_use] + pub(crate) fn from_settings(settings: &Settings) -> Self { + let passphrase = settings.ec.passphrase.expose().as_bytes(); + Self { + revision: derive(passphrase, b"trusted-server/identity/revision/v1"), + invalidated: derive(passphrase, b"trusted-server/identity/iv/v1"), + visitor: derive(passphrase, b"trusted-server/identity/visitor/v1"), + fingerprint: derive(passphrase, b"trusted-server/identity/fingerprint/v1"), + cookie_s: derive(passphrase, b"trusted-server/ts-eids-s/key/v1"), + cookie_r: derive(passphrase, b"trusted-server/ts-eids-r/key/v1"), + } + } +} + +/// A record's opaque revision: 16 lowercase hex digits. +#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct Revision(String); + +impl Revision { + /// Wraps a stored `rv` or `pv`. Callers have validated the format. + #[must_use] + pub(crate) fn from_stored(value: &str) -> Self { + Self(value.to_owned()) + } + + /// The hex digits. + #[must_use] + pub(crate) fn as_str(&self) -> &str { + &self.0 + } +} + +/// A fresh random revision, drawn on every write of a record. +#[must_use] +pub(crate) fn new_revision(random: &dyn RandomSource) -> Revision { + let mut bytes = [0u8; 8]; + random.fill(&mut bytes); + Revision(hex::encode(bytes)) +} + +/// The revision of a stored record: its `rv`, or for a legacy record without +/// one, the keyed hash of its `(uid, at)` until its next write. +#[must_use] +pub(crate) fn record_revision(keys: &IdentityKeys, record: &KvPartnerId) -> Revision { + if let Some(rv) = &record.rv { + return Revision(rv.clone()); + } + let at = record.at.unwrap_or_default().to_string(); + Revision(keyed_hex16(&keys.revision, &[record.uid.as_bytes(), at.as_bytes()])) +} + +/// The keyed hash stored in `ps.iv` for a UID the owner invalidated. +#[must_use] +pub(crate) fn invalidated_uid_hash(keys: &IdentityKeys, provider: &IdentityProviderId, uid: &str) -> String { + keyed_hex16(&keys.invalidated, &[provider.as_str().as_bytes(), uid.as_bytes()]) +} + +/// The per-visitor rate-limit key component, from the full EC ID. +#[must_use] +pub(crate) fn visitor_key(keys: &IdentityKeys, ec_id: &str) -> String { + keyed_hex16(&keys.visitor, &[ec_id.as_bytes()]) +} + +/// The consent fingerprint over the signals a provider's policy reads. +#[must_use] +pub(crate) fn consent_fingerprint(keys: &IdentityKeys, bits: &str) -> String { + keyed_hex16(&keys.fingerprint, &[bits.as_bytes()]) +} +``` + +Check that `Settings.ec.passphrase` is `Redacted` with `.expose()`. `pull_sync_marker.rs:293` uses exactly that. Register the module in `mod.rs` (`pub(crate) mod revision;`) and in `migration_guards.rs`. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::revision` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Add record revisions and keyed identity hashes" +``` + +--- + +### Task 14: The provider catalog and the shared servability check + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/catalog.rs` +- Create: `crates/trusted-server-core/src/ec/identity/servable.rs` +- Modify: `crates/trusted-server-core/src/ec/eids.rs` (`resolve_partner_ids`, and the stale module comment at lines 1-5) +- Modify: callers of `resolve_partner_ids`: `auction/endpoints.rs:465`, `ec/admin.rs:485` +- Modify: `crates/trusted-server-core/src/ec/identify.rs` (`entry.ids` read at about lines 99-103) +- Modify: `mod.rs`, `migration_guards.rs` +- Test: `servable.rs`, `ec/eids.rs`, `ec/identify.rs` + +**Interfaces:** + +- Produces: + - `pub(crate) struct ProviderCatalog` with `pub(crate) fn new(providers: Vec>, owners: BTreeMap) -> Self`, `get(&IdentityProviderId) -> Option<&Arc>`, `providers() -> impl Iterator>`, `owner_of(&SourceDomain) -> Option<&IdentityProviderId>`, `is_configured(&IdentityProviderId) -> bool`, `configured_sources() -> &[SourceDomain]` (sorted and de-duplicated, the order that `ts-eids-s` indexes use), `fingerprint() -> &str` (`cf`: the first 8 hex digits of SHA-256 over the sorted `provider|source|owner` lines), and `ProviderCatalog::empty()`. Part 1 builds it from generic pull providers only (Task 22), with no owners. + - `pub(crate) struct ServeContext<'a> { now: UnixTime, consent: &'a ConsentContext, signals: PolicySignals, catalog: &'a ProviderCatalog, opted_out: &'a BTreeSet }` + - `pub(crate) fn is_servable(context: &ServeContext<'_>, source: &SourceDomain, record: &KvPartnerId) -> bool` + - `resolve_partner_ids(registry: &PartnerRegistry, entry: &KvEntry, serve: &ServeContext<'_>) -> Vec`. The new third argument is required. `ResolvedPartnerId` gains `atype` from the record (`record.atype.unwrap_or(partner.openrtb_atype)`), `ext: Option`, and `writer: Option`. + +The rule (spec 6.1 and 7.1), applied in this order: + +1. An empty UID is not servable. +2. `exp` at or before `now` is not servable. +3. A record with a writer `w`: + - The writer must be in the catalog. A disabled or unconfigured writer means not servable. + - The writer must not be in `opted_out`, which is the union of KV `ps.o` and the `ts-eids-r` opt-outs. The caller computes it. + - The writer's consent policy must pass under this request's consent (`policy_allows`). +4. An owned source (the catalog has an owner): + - The owner's policy must pass. + - If the owner's descriptor marks the source as expiring, the record must have `w == owner` and a set, unexpired `exp`. This is the rollback rule. +5. Anything else, meaning no writer and an unowned source, is servable. The caller already applied the generic gates. + +- [ ] **Step 1: Write the failing tests** + +In `servable.rs`: + +```rust +#[cfg(test)] +mod tests { + use std::collections::{BTreeMap, BTreeSet}; + use std::sync::Arc; + + use super::*; + use crate::ec::identity::descriptor::{ConsentPolicy, ProviderDescriptor, Resolves}; + use crate::ec::identity::provider::IdentityProvider; + use crate::ec::kv_types::KvPartnerId; + + struct Fixed(ProviderDescriptor); + impl IdentityProvider for Fixed { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } + } + + fn provider(id: &str, source: &str, policy: ConsentPolicy, expiring: bool) -> Arc { + let mut builder = ProviderDescriptor::builder(id.parse().expect("id")) + .with_source(source.parse().expect("source")) + .with_resolves(Resolves::new().with_endpoint("https://api.example.com")) + .with_consent_policy(policy); + if expiring { + builder = builder.with_expiring_ids(); + } + Arc::new(Fixed(builder.build().expect("should build"))) + } + + fn record(writer: Option<&str>, exp: Option) -> KvPartnerId { + KvPartnerId { w: writer.map(str::to_owned), exp, ..KvPartnerId::new("u") } + } + + fn check(catalog: &ProviderCatalog, opted_out: &BTreeSet, source: &str, record: &KvPartnerId) -> bool { + let consent = test_consent_allowing_everything(); + let context = ServeContext { + now: UnixTime::from_secs(100), + consent: &consent, + signals: PolicySignals::default(), + catalog, + opted_out, + }; + is_servable(&context, &source.parse().expect("source"), record) + } + + #[test] + fn expiry_writer_presence_and_opt_out() { + let catalog = ProviderCatalog::new( + vec![provider("p", "a.example.com", ConsentPolicy::eid_persistence_only(), false)], + BTreeMap::new(), + ); + let none = BTreeSet::new(); + assert!(check(&catalog, &none, "a.example.com", &record(Some("p"), Some(101))), "control: valid"); + assert!(!check(&catalog, &none, "a.example.com", &record(Some("p"), Some(100))), "expired at exp"); + assert!(!check(&catalog, &none, "a.example.com", &record(Some("gone"), None)), "unknown writer"); + let opted: BTreeSet<_> = ["p".parse().expect("id")].into_iter().collect(); + assert!(!check(&catalog, &opted, "a.example.com", &record(Some("p"), None)), "opted-out writer"); + assert!(check(&catalog, &none, "b.example.com", &record(None, None)), "browser record, unowned"); + } + + #[test] + fn rollback_rule_for_expiring_owned_sources() { + let owner: IdentityProviderId = "owner".parse().expect("id"); + let catalog = ProviderCatalog::new( + vec![provider("owner", "a.example.com", ConsentPolicy::eid_persistence_only(), true)], + [("a.example.com".parse().expect("source"), owner)].into_iter().collect(), + ); + let none = BTreeSet::new(); + assert!(check(&catalog, &none, "a.example.com", &record(Some("owner"), Some(200))), "control: owner value"); + assert!(!check(&catalog, &none, "a.example.com", &record(None, None)), "browser-seeded, never from server"); + assert!(!check(&catalog, &none, "a.example.com", &record(Some("owner"), None)), "stripped exp"); + } + + #[test] + fn writer_policy_applies_to_unowned_sources() { + let strict = ConsentPolicy::strict().with_tcf_vendor(9999); + let catalog = ProviderCatalog::new(vec![provider("p", "a.example.com", strict, false)], BTreeMap::new()); + let consent = test_consent_gdpr_without_vendor(9999); + let none = BTreeSet::new(); + let context = ServeContext { + now: UnixTime::from_secs(100), + consent: &consent, + signals: PolicySignals::default(), + catalog: &catalog, + opted_out: &none, + }; + let source = "a.example.com".parse().expect("source"); + assert!(!is_servable(&context, &source, &record(Some("p"), None)), "writer's vendor denied"); + assert!(is_servable(&context, &source, &record(None, None)), "control: browser value unaffected"); + } +} +``` + +`test_consent_allowing_everything` and `test_consent_gdpr_without_vendor` reuse the helper you wrote in Task 12. Move it to `ec/identity/test_support.rs` now (create the file with `#![cfg(any(test, feature = "test-utils"))]` items and `pub(crate) mod test_support;` in `mod.rs`) so every identity test can share it. + +In `ec/eids.rs` tests, extend the existing `resolve_partner_ids` test (line 195) to pass a `ServeContext` built from `ProviderCatalog::empty()`, and add: + +```rust + #[test] + fn resolve_partner_ids_skips_expired_records_and_uses_stored_atype() { + let registry = registry_with(&["a.example.com", "b.example.com"]); + let mut entry = KvEntry::minimal("a.example.com", "ua", 1); + entry.ids.insert( + "b.example.com".to_owned(), + KvPartnerId { exp: Some(50), ..KvPartnerId::new("ub") }, + ); + entry.ids.get_mut("a.example.com").expect("a").atype = Some(3); + + let resolved = resolve_partner_ids(®istry, &entry, &serve_context_at(100)); + + assert_eq!(resolved.len(), 1, "should drop the expired b"); + assert_eq!(resolved[0].source_domain, "a.example.com", "should keep a"); + assert_eq!(resolved[0].openrtb_atype, 3, "should prefer the stored atype"); + } +``` + +Write `registry_with` and `serve_context_at` as local helpers. `registry_with` follows the existing test's registry setup, and `serve_context_at` uses `ProviderCatalog::empty()` with permissive consent. + +In `ec/identify.rs` tests, add one that stores a row with an expired record and a valid one and checks that the identify response lists only the valid one. Copy the setup from the nearest existing identify test (`grep -n "fn .*identify" crates/trusted-server-core/src/ec/identify.rs`). + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::servable`, `cargo test-fastly -- ec::eids`, `cargo test-fastly -- ec::identify` +Expected: compile errors, then the identify test fails because it returns the expired ID. + +- [ ] **Step 3: Implement `catalog.rs`** + +```rust +//! The enabled identity providers and source ownership. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use sha2::{Digest as _, Sha256}; + +use super::ids::{IdentityProviderId, SourceDomain}; +use super::provider::IdentityProvider; + +/// Enabled providers, source owners, and the derived source index and `cf`. +/// Built once per settings load. +#[derive(Clone)] +pub(crate) struct ProviderCatalog { + providers: BTreeMap>, + owners: BTreeMap, + sources: Vec, + fingerprint: String, +} + +impl ProviderCatalog { + /// A catalog with no providers. + #[must_use] + pub(crate) fn empty() -> Self { + Self::new(Vec::new(), BTreeMap::new()) + } + + /// Builds the catalog. Validation (ownership, duplicates) happens in + /// config validation before this is called. + #[must_use] + pub(crate) fn new( + providers: Vec>, + owners: BTreeMap, + ) -> Self { + let providers: BTreeMap<_, _> = providers + .into_iter() + .map(|provider| (provider.descriptor().id().clone(), provider)) + .collect(); + let sources: Vec = providers + .values() + .flat_map(|provider| provider.descriptor().sources().iter().cloned()) + .chain(owners.keys().cloned()) + .collect::>() + .into_iter() + .collect(); + let mut hasher = Sha256::new(); + for (id, provider) in &providers { + for source in provider.descriptor().sources() { + let owner = owners.get(source).map(IdentityProviderId::as_str).unwrap_or_default(); + hasher.update(format!("{id}|{source}|{owner}\n").as_bytes()); + } + } + let fingerprint = hex::encode(&hasher.finalize()[..4]); + Self { providers, owners, sources, fingerprint } + } + + /// A provider by ID. + #[must_use] + pub(crate) fn get(&self, id: &IdentityProviderId) -> Option<&Arc> { + self.providers.get(id) + } + + /// Every provider, ordered by ID. + pub(crate) fn providers(&self) -> impl Iterator> { + self.providers.values() + } + + /// Whether `id` is an enabled provider. + #[must_use] + pub(crate) fn is_configured(&self, id: &IdentityProviderId) -> bool { + self.providers.contains_key(id) + } + + /// The owner of `source`, if any. + #[must_use] + pub(crate) fn owner_of(&self, source: &SourceDomain) -> Option<&IdentityProviderId> { + self.owners.get(source) + } + + /// Every configured source, sorted. Indexes into this list are the + /// `s` values of `ts-eids-s`. + #[must_use] + pub(crate) fn configured_sources(&self) -> &[SourceDomain] { + &self.sources + } + + /// `cf`: 8 hex digits over providers, sources and owners. + #[must_use] + pub(crate) fn fingerprint(&self) -> &str { + &self.fingerprint + } +} +``` + +- [ ] **Step 4: Implement `servable.rs`** + +```rust +//! The one servability check shared by the bidstream, the admin view and +//! `/_ts/api/v1/identify` (spec 7.1). + +use std::collections::BTreeSet; + +use super::catalog::ProviderCatalog; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::policy::{PolicySignals, policy_allows}; +use crate::consent::ConsentContext; +use crate::ec::kv_types::KvPartnerId; + +/// What a servability decision needs from the request. +pub(crate) struct ServeContext<'a> { + pub(crate) now: UnixTime, + pub(crate) consent: &'a ConsentContext, + pub(crate) signals: PolicySignals, + pub(crate) catalog: &'a ProviderCatalog, + /// KV `ps.o` plus `ts-eids-r` opt-outs. + pub(crate) opted_out: &'a BTreeSet, +} + +/// Whether a stored record may be sent from the server. +#[must_use] +pub(crate) fn is_servable(context: &ServeContext<'_>, source: &SourceDomain, record: &KvPartnerId) -> bool { + if record.uid.is_empty() { + return false; + } + if record.exp.is_some_and(|exp| context.now.as_secs() >= exp) { + return false; + } + let writer = record.w.as_deref().and_then(|writer| writer.parse::().ok()); + if record.w.is_some() { + let Some(writer) = writer.as_ref() else { + return false; + }; + let Some(provider) = context.catalog.get(writer) else { + return false; + }; + if context.opted_out.contains(writer) + || !policy_allows(provider.descriptor().consent_policy(), context.consent, context.signals) + { + return false; + } + } + if let Some(owner) = context.catalog.owner_of(source) { + let Some(owner_provider) = context.catalog.get(owner) else { + return false; + }; + let descriptor = owner_provider.descriptor(); + if !policy_allows(descriptor.consent_policy(), context.consent, context.signals) { + return false; + } + if descriptor.is_expiring(source) && (writer.as_ref() != Some(owner) || record.exp.is_none()) { + return false; + } + } + true +} +``` + +- [ ] **Step 5: Switch the readers** + +In `ec/eids.rs`, replace the module comment at lines 1-5 with one that says KV IDs are filtered by the shared servability check in `ec::identity::servable`. Add `serve: &ServeContext<'_>` to `resolve_partner_ids`, and inside the loop add `if !is_servable(serve, &source, partner_uid) { continue; }` after parsing `source_domain` into a `SourceDomain`. Skip any key that does not parse. Take `atype` from the record and keep `ext` and the writer on `ResolvedPartnerId` (Task 28 adds provenance in `to_eids`). Update `to_eids` to emit the UID-level `ext` when present. + +At each caller, build a `ServeContext`. Until Task 23 puts the catalog and `ts-eids-r` opt-outs on `EcContext`, pass `ProviderCatalog::empty()` and the KV `ps.o` keys as `opted_out`. Task 23 replaces both. `now` comes from `checked_current_timestamp()`. If that fails, resolve no KV IDs, because a zero time would make every expiry check pass (spec 4.4). `consent` is the request's `ConsentContext`. DNT is read from the request header there (`auction/routing.rs:336` shows how). + +In `ec/identify.rs`, replace the direct `entry.ids` read with `resolve_partner_ids`, using the same context built from the request's consent, never the row's stored `consent` (spec 7.1). + +- [ ] **Step 6: Run and see them pass** + +Run: `cargo test-fastly -- ec::` and `cargo test-fastly -- auction::endpoints` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Filter served KV IDs through one servability check" -m "Expired IDs, IDs from disabled or opted-out writers, and IDs whose writer or owner policy fails are no longer sent to bidders, the admin view or /identify. /identify uses the request's consent, never the row's stored snapshot." +``` + +### Task 15: `IdentityMutation` and its merge into a live row + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/mutation.rs` +- Modify: `mod.rs`, `migration_guards.rs` +- Test: `mutation.rs` + +**Interfaces:** + +- Consumes: `KvEntry`, `KvPartnerId`, `KvProviderState`, `KvOptOut` (Task 6), `Revision`, `new_revision`, `record_revision`, `invalidated_uid_hash`, `IdentityKeys` (Task 13), `ProviderCatalog` (Task 14), `RandomSource` (Task 9) +- Produces: + - `pub(crate) enum PartnerIdChange`: + - `FillMissing { source, record: KvPartnerId, writer: Option, staged_at: Option }` + - `OwnerWrite { source, record, writer, expected: Option, staged_at: Option }` + - `OwnerRemove { source, writer, expected: Revision }` + - `WriterRemove { source, writer, staged_at: Option }` + - `pub(crate) enum ExpectedState { NotRead, Absent, Json(String) }` + - `pub(crate) enum StateUpdate { Keep, Replace(serde_json::Value), Clear }` + - `pub(crate) struct ProviderStateChange { n: Option, f: Option, opt_out: Option, opt_out_clear: Option, iv_add: Vec, st: StateUpdate }`, deriving `Default` + - `pub(crate) struct ProviderResult { provider, ids: Vec, state: ProviderStateChange, expected_state: ExpectedState }` + - `pub(crate) struct IdentityMutation`, deriving `Default`, with `add_change(PartnerIdChange)`, `add_result(ProviderResult)`, `merge_state(IdentityProviderId, ProviderStateChange)`, `set_cluster_size(u32)`, `is_empty()`, `has_removals()` + - `pub(crate) struct ApplyContext<'a> { now: u64, keys: &'a IdentityKeys, random: &'a dyn RandomSource, catalog: &'a ProviderCatalog }` + - `pub(crate) struct Applied { entry: KvEntry, changed: bool, removals_only: bool }` + - `pub(crate) fn apply(mutation: &IdentityMutation, live: &KvEntry, context: &ApplyContext<'_>) -> Option`. `None` means drop everything: the live row is a tombstone, or even the removals-only result failed validation. + +Apply order (spec 5.4). Copy it into the doc comment of `apply`: + +1. A tombstone (`consent.ok == false`) gives `None`. +2. Removals. Every `OwnerRemove` whose `expected` equals the live record's revision. Every `WriterRemove` whose live record has `w == writer` and, when `staged_at` is set, `at <= staged_at` (equal seconds go to the removal). Each new opt-out applies when `at > live.o.at` (or there is no live `o`) and `at >= live.oc` (or there is no live `oc`). Applying it removes every live record that provider wrote, at any revision, and clears its `st`. Then each staged clear (`opt_out_clear`) that is strictly later than the provider's latest opt-out `at`, live or staged, is stored as the live `oc`. A clear that is not later is dropped. `iv_add` hashes are appended, de-duplicated, and trimmed to the newest 4. +3. Positive changes. Skip every change by a provider whose (post-step-2) live `o` is set and not cleared (`oc` strictly later than `o.at`), and skip a staged change whose `staged_at <= o.at` even after a clear. A `FillMissing` applies when the source is absent, or its record is expired, or its writer is not configured. It is skipped when its UID's hash is in the source owner's `ps.iv`. An `OwnerWrite` applies when the live revision equals `expected`, or when `expected` is `None` and the source has no valid record with `w == writer`. A written record gets `w = writer`, a fresh `rv`, and `at = record.at.or(now)`. Each `ProviderResult` applies only when every one of its ID changes would apply on its own and the live `ps.st` matches `expected_state`. Its state change then applies with its IDs. Otherwise the whole result is dropped. +4. `ps` changes: `n` takes the later value, `f` is replaced, `st` is replaced or cleared. Then `cluster_size`. Then prune the `ps` entries of providers the catalog does not hold. Then drop expired records, except refresh anchors: the source's owner holds a `st` and the record's `ru` has not passed. +5. `validate()`. On failure, rebuild from the live entry with only step 2 applied (removals, opt-outs, clears, `iv`) and validate again. If that also fails, give `None` and log. +6. `changed` is whether the result differs from `live`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + use std::sync::Arc; + + use serde_json::json; + + use super::*; + use crate::ec::identity::catalog::ProviderCatalog; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::revision::IdentityKeys; + use crate::ec::identity::test_support::{fixed_provider, test_keys}; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId, KvProviderState}; + + const NOW: u64 = 1_000; + + fn src(name: &str) -> SourceDomain { + name.parse().expect("source") + } + + fn pid(name: &str) -> IdentityProviderId { + name.parse().expect("id") + } + + fn catalog() -> ProviderCatalog { + ProviderCatalog::new( + vec![fixed_provider("owner", &["a.example.com"]), fixed_provider("other", &["b.example.com"])], + [(src("a.example.com"), pid("owner"))].into_iter().collect(), + ) + } + + fn run(mutation: &IdentityMutation, live: &KvEntry) -> Option { + let keys = test_keys(); + let random = SeededRandom::new(1); + let catalog = catalog(); + apply(mutation, live, &ApplyContext { now: NOW, keys: &keys, random: &random, catalog: &catalog }) + } + + fn owned(uid: &str, rv: &str) -> KvPartnerId { + KvPartnerId { w: Some("owner".to_owned()), rv: Some(rv.to_owned()), at: Some(900), ..KvPartnerId::new(uid) } + } + + #[test] + fn tombstone_drops_everything() { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::FillMissing { + source: src("b.example.com"), + record: KvPartnerId::browser("u", Some(NOW)), + writer: None, + staged_at: None, + }); + assert!(run(&mutation, &KvEntry::tombstone(1)).is_none(), "should drop on a tombstone"); + } + + #[test] + fn fill_missing_fills_absent_and_expired_only() { + let mut live = KvEntry::minimal("b.example.com", "existing", 1); + live.ids.insert("c.example.com".to_owned(), KvPartnerId { exp: Some(NOW), ..KvPartnerId::new("old") }); + let mut mutation = IdentityMutation::default(); + for source in ["b.example.com", "c.example.com", "d.example.com"] { + mutation.add_change(PartnerIdChange::FillMissing { + source: src(source), + record: KvPartnerId::browser("new", Some(NOW)), + writer: None, + staged_at: None, + }); + } + + let applied = run(&mutation, &live).expect("should apply"); + + assert!(applied.changed, "should change"); + assert_eq!(applied.entry.ids["b.example.com"].uid, "existing", "should not replace a live value"); + assert_eq!(applied.entry.ids["c.example.com"].uid, "new", "should replace an expired value"); + assert_eq!(applied.entry.ids["d.example.com"].uid, "new", "should fill an absent source"); + } + + #[test] + fn owner_write_checks_revision_and_gets_a_fresh_rv() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + + let mut stale = IdentityMutation::default(); + stale.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("v2"), + writer: pid("owner"), + expected: Some(Revision::from_stored("2222222222222222")), + staged_at: None, + }); + let applied = run(&stale, &live).expect("should apply"); + assert!(!applied.changed, "a stale revision loses to the newer write"); + + let mut fresh = IdentityMutation::default(); + fresh.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("v2"), + writer: pid("owner"), + expected: Some(Revision::from_stored("1111111111111111")), + staged_at: None, + }); + let applied = run(&fresh, &live).expect("should apply"); + let record = &applied.entry.ids["a.example.com"]; + assert_eq!(record.uid, "v2", "should write"); + assert_eq!(record.w.as_deref(), Some("owner"), "should set the writer"); + assert_ne!(record.rv.as_deref(), Some("1111111111111111"), "should draw a new rv"); + assert_eq!(record.at, Some(NOW), "should stamp at"); + } + + #[test] + fn browser_seeded_value_never_blocks_the_owner() { + let mut live = KvEntry::minimal("a.example.com", "seeded", 1); + live.ids.get_mut("a.example.com").expect("a").at = Some(5); + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("owned"), + writer: pid("owner"), + expected: None, + staged_at: None, + }); + let applied = run(&mutation, &live).expect("should apply"); + assert_eq!(applied.entry.ids["a.example.com"].uid, "owned", "owner takes over a seeded value"); + } + + #[test] + fn removals_win_ties_and_opt_out_removes_everything_the_writer_wrote() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + live.ids.insert( + "c.example.com".to_owned(), + KvPartnerId { w: Some("owner".to_owned()), at: Some(950), ..KvPartnerId::new("unowned-by-owner") }, + ); + + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + pid("owner"), + ProviderStateChange { opt_out: Some(KvOptOut { at: NOW, fp: "0".repeat(16) }), ..ProviderStateChange::default() }, + ); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("v2"), + writer: pid("owner"), + expected: None, + staged_at: Some(NOW), + }); + + let applied = run(&mutation, &live).expect("should apply"); + + assert!(!applied.entry.ids.contains_key("a.example.com"), "should remove the owned record"); + assert!(!applied.entry.ids.contains_key("c.example.com"), "should remove the unowned record it wrote"); + assert_eq!(applied.entry.ids["b.example.com"].uid, "x", "control: other writers untouched"); + assert_eq!(applied.entry.ps["owner"].o.as_ref().map(|o| o.at), Some(NOW), "should store the opt-out"); + } + + #[test] + fn writer_remove_without_revision_respects_staged_at() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + let remove = |staged_at| { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::WriterRemove { + source: src("a.example.com"), + writer: pid("owner"), + staged_at: Some(staged_at), + }); + run(&mutation, &live).expect("should apply") + }; + assert!(remove(900).entry.ids.get("a.example.com").is_none(), "same second: the removal wins"); + assert!(remove(899).entry.ids.get("a.example.com").is_some(), "re-acquired after sa: kept"); + } + + #[test] + fn opt_out_and_clear_in_the_same_second_leave_it_opted_out() { + let live = KvEntry::minimal("b.example.com", "x", 1); + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + pid("owner"), + ProviderStateChange { + opt_out: Some(KvOptOut { at: NOW, fp: "0".repeat(16) }), + opt_out_clear: Some(NOW), + ..ProviderStateChange::default() + }, + ); + let applied = run(&mutation, &live).expect("should apply"); + assert!(applied.entry.ps["owner"].o.is_some(), "should stay opted out"); + assert_eq!(applied.entry.ps["owner"].oc, None, "should drop a clear that is not later"); + } + + #[test] + fn invalidated_uid_is_not_reseeded() { + let keys = test_keys(); + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + iv: vec![crate::ec::identity::revision::invalidated_uid_hash(&keys, &pid("owner"), "rejected")], + ..KvProviderState::default() + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::FillMissing { + source: src("a.example.com"), + record: KvPartnerId::browser("rejected", Some(NOW)), + writer: None, + staged_at: None, + }); + let applied = run(&mutation, &live).expect("should apply"); + assert!(!applied.entry.ids.contains_key("a.example.com"), "should not re-seed the rejected UID"); + } + + #[test] + fn provider_result_applies_atomically() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert("a.example.com".to_owned(), owned("newer", "3333333333333333")); + live.ps.insert("owner".to_owned(), KvProviderState { st: Some(json!({ "token": "B" })), ..KvProviderState::default() }); + + let mut mutation = IdentityMutation::default(); + mutation.add_result(ProviderResult { + provider: pid("owner"), + ids: vec![PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("mine"), + writer: pid("owner"), + expected: Some(Revision::from_stored("1111111111111111")), + staged_at: None, + }], + state: ProviderStateChange { st: StateUpdate::Replace(json!({ "token": "A" })), f: Some(0), ..ProviderStateChange::default() }, + expected_state: ExpectedState::Json(r#"{"token":"A0"}"#.to_owned()), + }); + + let applied = run(&mutation, &live).expect("should apply"); + + assert_eq!(applied.entry.ids["a.example.com"].uid, "newer", "should keep the other writer's ID"); + assert_eq!(applied.entry.ps["owner"].st, Some(json!({ "token": "B" })), "with that writer's state"); + assert!(!applied.changed, "nothing else changed"); + } + + #[test] + fn refresh_anchor_survives_until_ru_or_state_loss() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { exp: Some(NOW - 1), ru: Some(NOW + 100), ..owned("expired", "1111111111111111") }, + ); + live.ps.insert("owner".to_owned(), KvProviderState { st: Some(json!({ "r": 1 })), ..KvProviderState::default() }); + let mut unrelated = IdentityMutation::default(); + unrelated.set_cluster_size(1); + + let kept = run(&unrelated, &live).expect("should apply"); + assert!(kept.entry.ids.contains_key("a.example.com"), "anchor kept while state lives"); + + live.ps.get_mut("owner").expect("owner").st = None; + let dropped = run(&unrelated, &live).expect("should apply"); + assert!(!dropped.entry.ids.contains_key("a.example.com"), "anchor dropped without state"); + } + + #[test] + fn unconfigured_provider_state_is_pruned() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert("retired".to_owned(), KvProviderState { n: Some(5), ..KvProviderState::default() }); + live.ps.insert("other".to_owned(), KvProviderState { n: Some(5), ..KvProviderState::default() }); + let mut mutation = IdentityMutation::default(); + mutation.set_cluster_size(1); + let applied = run(&mutation, &live).expect("should apply"); + assert!(!applied.entry.ps.contains_key("retired"), "should prune"); + assert!(applied.entry.ps.contains_key("other"), "control: configured kept"); + } + + #[test] + fn invalid_result_falls_back_to_removals_only() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::FillMissing { + source: src("d.example.com"), + record: KvPartnerId::browser("x".repeat(600), Some(NOW)), + writer: None, + staged_at: None, + }); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + writer: pid("owner"), + expected: Revision::from_stored("1111111111111111"), + }); + let applied = run(&mutation, &live).expect("should apply removals"); + assert!(applied.removals_only, "should fall back"); + assert!(!applied.entry.ids.contains_key("a.example.com"), "removal applied"); + assert!(!applied.entry.ids.contains_key("d.example.com"), "invalid fill dropped"); + } +} +``` + +Add `fixed_provider(id: &str, sources: &[&str]) -> Arc` (a descriptor-only provider with a dummy `https://api.example.com` endpoint and the `eid_persistence_only` policy) and `test_keys() -> IdentityKeys` to `ec/identity/test_support.rs`. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::mutation` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Write `mutation.rs` with the types listed under Interfaces, then implement `apply` as five private steps over a cloned `KvEntry`. Use one function per step so each rule can be read on its own: + +```rust +pub(crate) fn apply(mutation: &IdentityMutation, live: &KvEntry, context: &ApplyContext<'_>) -> Option { + if !live.consent.ok { + return None; + } + + let mut entry = live.clone(); + apply_removals(&mut entry, mutation, context); + let removals_only = entry.clone(); + + apply_positive(&mut entry, mutation, context); + apply_state_and_cleanup(&mut entry, mutation, context); + + let (entry, removals_only_used) = match entry.validate() { + Ok(()) => (entry, false), + Err(message) => { + log::warn!("Identity mutation produced an invalid row, keeping removals only: {message}"); + let mut fallback = removals_only; + prune_and_drop_expired(&mut fallback, context); + if let Err(message) = fallback.validate() { + log::warn!("Identity mutation removals-only row is invalid, dropping mutation: {message}"); + return None; + } + (fallback, true) + } + }; + + let changed = &entry != live; + Some(Applied { entry, changed, removals_only: removals_only_used }) +} +``` + +- `apply_removals`: iterate `OwnerRemove` and `WriterRemove` from `partner_ids` and from every `ProviderResult`'s state-free removals. When building a `ProviderResult`, the coordinator always puts removals in `partner_ids`, never in `ProviderResult.ids` (spec 5.4 comment on `ProviderResult`). Then opt-outs, clears and `iv` from `provider_state` and from each result's `state`. A revision is compared with `record_revision(context.keys, live_record)`. +- `apply_positive`: compute `opted_out(provider)` from the post-removal `ps`. For each plain change, call `fn change_applies(entry, change, context) -> bool` and, when true, `fn write_change(entry, change, context)`. For each `ProviderResult`, first check that every ID change applies and that `expected_state` matches `entry.ps[provider].st` serialized with `serde_json::to_string`. `ExpectedState::NotRead` always matches. Then apply all of them and the result's `state` together. +- `apply_state_and_cleanup`: merge `provider_state` (`n` = max, `f` replaced when `Some`, `st` per `StateUpdate`), set `entry.network`'s `cluster_size` when given (match the existing `KvNetwork` field), then call `prune_and_drop_expired`. +- `prune_and_drop_expired`: `entry.ps.retain(|id, _| parsed id is configured)`. Drop records with `exp <= now` unless they are anchors: the catalog's owner of that source has `entry.ps[owner].st.is_some()` and `record.ru.is_some_and(|ru| now < ru)`. + +`IdentityMutation::merge_state(provider, change)` merges into any existing change for the same provider: later `n`, last `f`, union of `iv_add`, the later `opt_out`, the later `opt_out_clear`, and the last non-`Keep` `st`. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::mutation` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Add IdentityMutation and its merge into a live identity row" -m "Removals apply first and win equal-second ties, a provider opt-out removes every record that provider wrote, stateful results apply atomically, refresh anchors survive while their owner holds state, and an invalid row falls back to removals only." +``` + +### Task 16: Sealing and opening the identity cookies + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/cookie.rs` (seal, open, read and format helpers) +- Modify: `mod.rs`, `migration_guards.rs` +- Test: `cookie.rs` + +**Interfaces:** + +- Consumes: `IdentityKeys.cookie_s`, `.cookie_r` (Task 13), `RandomSource` (Task 9), `COOKIE_TS_EIDS_S`, `COOKIE_TS_EIDS_R_PREFIX` (Task 1) +- Produces: + - `pub(crate) fn seal(key: &[u8; 32], ec_id: &str, payload: &[u8], random: &dyn RandomSource) -> String`, which returns `v1..` + - `pub(crate) fn open(key: &[u8; 32], ec_id: &str, value: &str) -> Option>` + - `pub(crate) enum CookieRead { Absent, Duplicate, Value(String) }` and `pub(crate) fn read_single_cookie(headers: &HeaderMap, name: &str) -> CookieRead`, which treats two cookies of the same name as none + - `pub(crate) fn read_prefixed_cookies(headers: &HeaderMap, prefix: &str) -> Vec<(String, String)>` + - `pub(crate) fn set_cookie_header(name: &str, value: &str) -> String` and `pub(crate) fn delete_cookie_header(name: &str) -> String` + - `pub(crate) const COOKIE_MAX_AGE_SECS: u64 = 31_536_000;` + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use http::{HeaderMap, HeaderValue, header}; + + use super::*; + use crate::ec::identity::input::SeededRandom; + + const EC: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ABC123"; + const OTHER_EC: &str = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ABC123"; + + #[test] + fn seal_and_open_round_trip_bound_to_the_ec_id() { + let key = [7u8; 32]; + let random = SeededRandom::new(3); + let sealed = seal(&key, EC, br#"{"v":1}"#, &random); + + assert!(sealed.starts_with("v1."), "should carry the version prefix"); + assert_eq!(sealed.split('.').count(), 3, "should have three parts"); + assert!(!sealed.contains('='), "should be unpadded base64url"); + assert_eq!(open(&key, EC, &sealed).as_deref(), Some(&br#"{"v":1}"#[..]), "should open for its EC"); + assert!(open(&key, OTHER_EC, &sealed).is_none(), "should not open for another EC"); + assert!(open(&[8u8; 32], EC, &sealed).is_none(), "should not open with another key"); + let mut tampered = sealed.clone(); + tampered.pop(); + tampered.push(if sealed.ends_with('A') { 'B' } else { 'A' }); + assert!(open(&key, EC, &tampered).is_none(), "should reject tampering"); + assert!(open(&key, EC, "v2.x.y").is_none(), "should reject another version"); + } + + #[test] + fn duplicate_cookie_names_count_as_none() { + let mut headers = HeaderMap::new(); + headers.append(header::COOKIE, HeaderValue::from_static("__Host-ts-eids-s=one")); + headers.append(header::COOKIE, HeaderValue::from_static("other=1; __Host-ts-eids-s=two")); + assert_eq!(read_single_cookie(&headers, "__Host-ts-eids-s"), CookieRead::Duplicate, "duplicate"); + + let mut single = HeaderMap::new(); + single.insert(header::COOKIE, HeaderValue::from_static("__Host-ts-eids-s=one; x=2")); + assert_eq!(read_single_cookie(&single, "__Host-ts-eids-s"), CookieRead::Value("one".to_owned()), "single"); + } + + #[test] + fn prefixed_cookies_and_attributes() { + let mut headers = HeaderMap::new(); + headers.insert( + header::COOKIE, + HeaderValue::from_static("__Host-ts-eids-r-0a1b2c3d=a; __Host-ts-eids-r-11111111=b; x=1"), + ); + let found = read_prefixed_cookies(&headers, "__Host-ts-eids-r-"); + assert_eq!(found.len(), 2, "should find both removal cookies"); + assert_eq!( + set_cookie_header("__Host-ts-eids-s", "v1.a.b"), + "__Host-ts-eids-s=v1.a.b; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=31536000", + "should use the spec attributes and no Domain" + ); + assert_eq!( + delete_cookie_header("__Host-ts-eids-s"), + "__Host-ts-eids-s=; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=0", + "should expire with matching attributes" + ); + } +} +``` + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::cookie` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +```rust +//! The server-owned identity cookies: sealing, reading and formatting. + +use aes_gcm::aead::{Aead as _, KeyInit as _, Payload}; +use aes_gcm::{Aes256Gcm, Nonce}; +use base64::Engine as _; +use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use http::{HeaderMap, header}; + +use super::input::RandomSource; + +/// Lifetime of both identity cookie kinds. +pub(crate) const COOKIE_MAX_AGE_SECS: u64 = 31_536_000; + +const SEAL_VERSION: &str = "v1"; + +/// Seals `payload` with AES-256-GCM, the EC ID as associated data. +#[must_use] +pub(crate) fn seal(key: &[u8; 32], ec_id: &str, payload: &[u8], random: &dyn RandomSource) -> String { + let cipher = Aes256Gcm::new(key.into()); + let mut nonce = [0u8; 12]; + random.fill(&mut nonce); + let ciphertext = cipher + .encrypt(Nonce::from_slice(&nonce), Payload { msg: payload, aad: ec_id.as_bytes() }) + .expect("should encrypt a bounded in-memory payload"); + format!( + "{SEAL_VERSION}.{}.{}", + URL_SAFE_NO_PAD.encode(nonce), + URL_SAFE_NO_PAD.encode(ciphertext) + ) +} + +/// Opens a sealed value. `None` for any failure; never log the value. +#[must_use] +pub(crate) fn open(key: &[u8; 32], ec_id: &str, value: &str) -> Option> { + let mut parts = value.split('.'); + if parts.next()? != SEAL_VERSION { + return None; + } + let nonce = URL_SAFE_NO_PAD.decode(parts.next()?).ok()?; + let ciphertext = URL_SAFE_NO_PAD.decode(parts.next()?).ok()?; + if parts.next().is_some() || nonce.len() != 12 { + return None; + } + Aes256Gcm::new(key.into()) + .decrypt(Nonce::from_slice(&nonce), Payload { msg: &ciphertext, aad: ec_id.as_bytes() }) + .ok() +} + +/// One cookie read by exact name. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum CookieRead { + Absent, + /// Two or more cookies share the name; treated as none (spec 6.2). + Duplicate, + Value(String), +} + +fn cookie_pairs(headers: &HeaderMap) -> impl Iterator { + headers + .get_all(header::COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .flat_map(|value| value.split(';')) + .filter_map(|pair| pair.trim().split_once('=')) + .map(|(name, value)| (name.trim(), value.trim())) +} + +/// Reads a cookie that must occur at most once. +#[must_use] +pub(crate) fn read_single_cookie(headers: &HeaderMap, name: &str) -> CookieRead { + let mut found = cookie_pairs(headers).filter(|(cookie, _)| *cookie == name).map(|(_, value)| value); + match (found.next(), found.next()) { + (None, _) => CookieRead::Absent, + (Some(_), Some(_)) => CookieRead::Duplicate, + (Some(value), None) => CookieRead::Value(value.to_owned()), + } +} + +/// Every cookie whose name starts with `prefix`, as `(name, value)`. +#[must_use] +pub(crate) fn read_prefixed_cookies(headers: &HeaderMap, prefix: &str) -> Vec<(String, String)> { + cookie_pairs(headers) + .filter(|(name, _)| name.starts_with(prefix)) + .map(|(name, value)| (name.to_owned(), value.to_owned())) + .collect() +} + +/// A `Set-Cookie` value with the identity cookie attributes. +#[must_use] +pub(crate) fn set_cookie_header(name: &str, value: &str) -> String { + format!("{name}={value}; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age={COOKIE_MAX_AGE_SECS}") +} + +/// A `Set-Cookie` value that deletes an identity cookie. +#[must_use] +pub(crate) fn delete_cookie_header(name: &str) -> String { + format!("{name}=; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=0") +} +``` + +If the `aes-gcm` 0.10 key conversion `key.into()` does not infer, use `Aes256Gcm::new_from_slice(key).expect("should accept a 32-byte key")`. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::cookie` and `cargo check-cloudflare` +Expected: PASS and a clean build. The Cloudflare check proves that `aes-gcm` compiles for `wasm32-unknown-unknown`, because core is shared. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Seal and read the server-owned identity cookies" +``` + +### Task 17: The `ts-eids-s` payload, its size cap and its merge with KV + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/identity/cookie.rs` +- Test: `cookie.rs` + +**Interfaces:** + +- Consumes: Task 16, `ProviderCatalog` (Task 14), `KvEntry` (Task 6), `record_revision` (Task 13), `EcKvSnapshot` (`ec/mod.rs:118`) +- Produces: + - Payload types with the spec 6.2 short field names, and serde `skip_serializing_if` on every optional field: + - `SCookie { v: u8, kc: u64, cf: String, m: bool, ids: Vec, ps: BTreeMap }` + - `SEntry { s: usize, u, w, t, e, r, a, x, rv, pv, sa, b, k, d }`. `b`, `k` and `d` serialize as `1`/`0` through small `bool_as_int` helpers. + - `SProviderState { n, f, o: Option, iv: Vec, oc, p, sg, kn }`. `kn` is the KV `n` that a cookie-only `n` was derived from (spec 6.3). + - `pub(crate) const MAX_S_COOKIE_BYTES: usize = 2048;` + - `pub(crate) fn open_s_cookie(keys, ec_id, catalog, value) -> SOpen`, where `SOpen` is `Missing | Failed | ConfigChanged(SCookie) | Valid(SCookie)`. `ConfigChanged` keeps only markers. + - `pub(crate) fn encode_s_cookie(keys, ec_id, cookie: &mut SCookie, random) -> String`, which applies the drop order until the sealed value fits in 2,048 bytes, else falls back to a minimal cookie with `kc = 0` and the markers only (spec 5.6) + - `pub(crate) fn merge_with_kv(cookie: &SCookie, snapshot: &EcKvSnapshot, context: &CookieMergeContext<'_>) -> MergeOutcome`, where `MergeOutcome { cookie: SCookie, changed: bool, expire: bool }` + - `pub(crate) fn cookie_view(cookie: &SCookie, catalog) -> BTreeMap`: the cookie's entries resolved to sources. `Staging` is `Confirmed | Staged { pv, sa, browser }`. + - `pub(crate) fn minimal_missing_cookie(now) -> SCookie` (`m: true`, no entries) + +Merge rules (spec 6.3, step 3). Each is one test: + +- A tombstone or `consent.ok = false` gives `expire: true`. +- `Missing` gives `minimal_missing_cookie`. Markers are dropped too. +- For an owned source, KV wins, except that a staged provider entry `OwnerWrite` would still apply (its `pv` matches the live revision, or it has no `pv` and KV holds only a value the provider did not write) stays `k: 0`. +- A `k: 1` entry whose source is gone from KV, or present at another revision, is replaced by the KV record (when owned) or dropped. +- A staged entry KV already holds (same UID, same writer) becomes `k: 1` with KV's `rv`. +- A `b: 1` entry is dropped once KV holds any value for its source. +- A stub becomes the KV record. +- Markers (`p`) are cleared when answered: KV `n >= p`, KV `o` set, a claimed source with `at >= p` or a source that lost the cookie's value, or the provider ineligible on the row. The last one takes a callback, `answered_by_eligibility`, which Task 20 supplies. +- `m` is cleared when the row is present. +- `kc = now`. `changed` is true when anything but `kc` changed, or the inbound `kc` is older than `kv_revalidate_sec / 2`. + +The cookie caches values only for owned sources once they are in KV (spec 5.3). Unowned confirmed values are not copied into the cookie by the merge. In Part 1 no source is owned, so the cookie holds only staged entries, markers and mirrored `ps`. + +Size-cap drop order (spec 6.2): + +1. `k: 1` entries become stubs `{s, d: 1}`. +2. `ps` entries with no `p` and no `sg` are dropped. +3. Provider `k: 0` entries, lowest `cookie_priority` first. Task 19 adds the priority. Pass it in as `&dyn Fn(&str) -> i32`. +4. `b: 1` entries, last. + +Markers are never dropped. Log each drop with the provider and source only. + +- [ ] **Step 1: Write the failing tests** + +```rust + mod s_cookie { + use super::super::*; + use crate::ec::identity::test_support::{catalog_with_owner, test_keys}; + use crate::ec::identity::input::SeededRandom; + use crate::ec::kv_types::{KvEntry, KvPartnerId, KvProviderState}; + use crate::ec::EcKvSnapshot; + + const EC: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ABC123"; + + fn present(entry: KvEntry) -> EcKvSnapshot { + EcKvSnapshot::Present { ec_id: EC.to_owned(), entry: Box::new(entry), generation: Some(1) } + } + + #[test] + fn round_trips_and_detects_config_change() { + let keys = test_keys(); + let catalog = catalog_with_owner("owner", "a.example.com"); + let random = SeededRandom::new(1); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ps.insert("owner".to_owned(), SProviderState { p: Some(90), ..SProviderState::default() }); + let sealed = encode_s_cookie(&keys, EC, &mut cookie, &random, &|_| 0); + + match open_s_cookie(&keys, EC, &catalog, &sealed) { + SOpen::Valid(opened) => assert_eq!(opened, cookie, "should round-trip"), + other => panic!("should open, got {other:?}"), + } + let other_catalog = catalog_with_owner("owner", "b.example.com"); + match open_s_cookie(&keys, EC, &other_catalog, &sealed) { + SOpen::ConfigChanged(kept) => { + assert!(kept.ids.is_empty(), "should drop positive entries"); + assert_eq!(kept.ps["owner"].p, Some(90), "should keep markers"); + } + other => panic!("should flag the config change, got {other:?}"), + } + } + + #[test] + fn tombstone_expires_and_missing_row_minimizes() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let cookie = SCookie::new(&catalog, 100); + let context = CookieMergeContext::for_test(&catalog, 200); + assert!(merge_with_kv(&cookie, &present(KvEntry::tombstone(1)), &context).expire, "tombstone"); + let missing = merge_with_kv(&cookie, &EcKvSnapshot::Missing { ec_id: EC.to_owned() }, &context); + assert!(missing.cookie.m, "should set m"); + assert!(missing.cookie.ids.is_empty() && missing.cookie.ps.is_empty(), "no entries, no markers"); + } + + #[test] + fn staged_entry_flips_to_confirmed_once_in_kv() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ids.push(SEntry::staged_browser(0, "u", 100)); + let mut entry = KvEntry::minimal("a.example.com", "u", 1); + entry.ids.get_mut("a.example.com").expect("a").rv = Some("1111111111111111".to_owned()); + + let merged = merge_with_kv(&cookie, &present(entry), &CookieMergeContext::for_test(&catalog, 200)); + + assert!(merged.changed, "should change"); + assert!(merged.cookie.ids.is_empty(), "a b:1 entry is dropped once KV holds the source"); + assert_eq!(merged.cookie.kc, 200, "should stamp kc"); + } + + #[test] + fn marker_is_answered_by_a_newer_n() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ps.insert("owner".to_owned(), SProviderState { p: Some(150), ..SProviderState::default() }); + let mut entry = KvEntry::minimal("b.example.com", "x", 1); + entry.ps.insert("owner".to_owned(), KvProviderState { n: Some(160), ..KvProviderState::default() }); + + let merged = merge_with_kv(&cookie, &present(entry), &CookieMergeContext::for_test(&catalog, 200)); + assert_eq!(merged.cookie.ps.get("owner").and_then(|state| state.p), None, "should clear the marker"); + } + + #[test] + fn size_cap_drops_in_order_and_keeps_markers() { + let keys = test_keys(); + let catalog = catalog_with_owner("owner", "a.example.com"); + let random = SeededRandom::new(1); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ps.insert("owner".to_owned(), SProviderState { p: Some(90), ..SProviderState::default() }); + for _ in 0..40 { + cookie.ids.push(SEntry::staged_browser(0, &"u".repeat(60), 100)); + } + let sealed = encode_s_cookie(&keys, EC, &mut cookie, &random, &|_| 0); + assert!(sealed.len() <= MAX_S_COOKIE_BYTES, "should fit the cap"); + assert_eq!(cookie.ps["owner"].p, Some(90), "should keep the marker"); + } + } +``` + +Add `catalog_with_owner(provider, source)` to `test_support.rs`. It builds a `ProviderCatalog` with one `fixed_provider` that owns `source`. Add `CookieMergeContext::for_test(catalog, now)`, which uses `kv_revalidate_sec = 3600`, test keys, and an `answered_by_eligibility` that always returns `false`. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::cookie::tests::s_cookie` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Write the payload types with serde attributes, then `SCookie::new(catalog, now)` (`v: 1`, `kc: now`, `cf: catalog.fingerprint()`), `SEntry::staged_browser(index, uid, sa)`, `open_s_cookie`, `encode_s_cookie`, `merge_with_kv` and `cookie_view`. Give each merge rule above its own private helper, named after the rule (`fn flip_confirmed`, `fn drop_browser_entries_kv_holds`, `fn answer_markers`, and so on). That way a reviewer can match each helper to its spec bullet. A source index that is out of range for `catalog.configured_sources()` is dropped on open. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::cookie` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity +git commit --signoff -S -m "Encode the ts-eids-s cookie and reconcile it with KV" +``` + +### Task 18: The `ts-eids-r` removal cookies + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/identity/cookie.rs` +- Test: `cookie.rs` + +**Interfaces:** + +- Produces: + - `RBatch { rm: Vec, o: BTreeMap, iv: BTreeMap> }` and `RRemoval { s: String /* the source domain itself */, w: String, pv: Option, sa: u64 }` + - `pub(crate) const MAX_R_COOKIE_BYTES: usize = 512;`, `pub(crate) const R_COOKIE_SOFT_LIMIT: usize = 4;` + - `pub(crate) struct OpenedRemovals { cookies: Vec<(String, RBatch)>, unopenable: Vec }` and `pub(crate) fn open_r_cookies(keys, ec_id, headers) -> OpenedRemovals` + - `pub(crate) fn r_batch_to_changes(batch: &RBatch) -> (Vec, BTreeMap)`, which drops entries whose source or provider does not parse + - `pub(crate) fn r_batch_applied(batch: &RBatch, entry: &KvEntry, keys) -> bool`, the "applied" definition of spec 6.2 + - `pub(crate) enum RPlan { Cookie { name, value }, Overflow }` and `pub(crate) fn plan_r_batch(keys, ec_id, batch, inbound_openable: usize, random) -> RPlan`. It returns `Overflow` when the batch does not fit one 512-byte cookie or the request already carries four openable cookies. + - `pub(crate) fn split_r_batch(keys, ec_id, batch, random) -> Vec<(String, String)>`: the overflow fallback, as many 512-byte cookies as it takes, never coarsened (each removal, opt-out and `iv` hash kept exactly) + - `pub(crate) fn new_r_cookie_name(random) -> String`: `__Host-ts-eids-r-` plus 8 random hex digits + +- [ ] **Step 1: Write the failing tests** + +```rust + mod r_cookie { + use super::super::*; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::test_support::test_keys; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId, KvProviderState}; + + const EC: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ABC123"; + + fn removal(source: &str, writer: &str, sa: u64) -> RRemoval { + RRemoval { s: source.to_owned(), w: writer.to_owned(), pv: None, sa } + } + + #[test] + fn a_typical_batch_fits_one_cookie_and_a_long_domain_fits_alone() { + let keys = test_keys(); + let random = SeededRandom::new(1); + let four = RBatch { rm: (0..4).map(|i| removal(&format!("s{i}.example.com"), "provider", 1)).collect(), ..RBatch::default() }; + assert!(matches!(plan_r_batch(&keys, EC, &four, 0, &random), RPlan::Cookie { .. }), "four typical removals fit"); + + let label = "a".repeat(63); + let long_domain = format!("{label}.{label}.{label}.{}.com", "b".repeat(57)); + assert_eq!(long_domain.len(), 253, "should be a 253-byte domain"); + let one = RBatch { rm: vec![removal(&long_domain, "a-long-provider-id", 1)], ..RBatch::default() }; + assert!(matches!(plan_r_batch(&keys, EC, &one, 0, &random), RPlan::Cookie { .. }), "one long removal fits"); + } + + #[test] + fn overflow_when_too_big_or_four_cookies_present() { + let keys = test_keys(); + let random = SeededRandom::new(1); + let big = RBatch { rm: (0..12).map(|i| removal(&format!("s{i}.example.com"), "provider", 1)).collect(), ..RBatch::default() }; + assert!(matches!(plan_r_batch(&keys, EC, &big, 0, &random), RPlan::Overflow), "too big"); + let small = RBatch { rm: vec![removal("a.example.com", "p", 1)], ..RBatch::default() }; + assert!(matches!(plan_r_batch(&keys, EC, &small, 4, &random), RPlan::Overflow), "four present"); + } + + #[test] + fn split_keeps_every_entry_exactly() { + let keys = test_keys(); + let random = SeededRandom::new(1); + let big = RBatch { + rm: (0..12).map(|i| removal(&format!("s{i}.example.com"), "provider", 1)).collect(), + o: [("provider".to_owned(), 5)].into_iter().collect(), + iv: [("provider".to_owned(), vec!["0123456789abcdef".to_owned()])].into_iter().collect(), + }; + let cookies = split_r_batch(&keys, EC, &big, &random); + assert!(cookies.iter().all(|(_, value)| value.len() <= MAX_R_COOKIE_BYTES), "each fits"); + let mut headers = http::HeaderMap::new(); + let joined = cookies.iter().map(|(name, value)| format!("{name}={value}")).collect::>().join("; "); + headers.insert(http::header::COOKIE, http::HeaderValue::from_str(&joined).expect("header")); + let opened = open_r_cookies(&keys, EC, &headers); + let rebuilt = opened.cookies.iter().fold(RBatch::default(), |mut all, (_, batch)| { + all.rm.extend(batch.rm.clone()); + all.o.extend(batch.o.clone()); + for (provider, hashes) in &batch.iv { + all.iv.entry(provider.clone()).or_default().extend(hashes.clone()); + } + all + }); + assert_eq!(rebuilt.rm.len(), 12, "every removal survives"); + assert_eq!(rebuilt.o, big.o, "the opt-out survives unchanged"); + assert_eq!(rebuilt.iv, big.iv, "the iv hash survives"); + } + + #[test] + fn applied_rules() { + let keys = test_keys(); + let mut entry = KvEntry::minimal("b.example.com", "x", 1); + entry.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { w: Some("p".to_owned()), at: Some(10), ..KvPartnerId::new("u") }, + ); + assert!(!r_batch_applied(&RBatch { rm: vec![removal("a.example.com", "p", 10)], ..RBatch::default() }, &entry, &keys), "pending"); + assert!(r_batch_applied(&RBatch { rm: vec![removal("a.example.com", "p", 9)], ..RBatch::default() }, &entry, &keys), "re-acquired after sa"); + assert!(r_batch_applied(&RBatch { rm: vec![removal("a.example.com", "q", 10)], ..RBatch::default() }, &entry, &keys), "other writer"); + + entry.ps.insert("p".to_owned(), KvProviderState { o: Some(KvOptOut { at: 5, fp: "0".repeat(16) }), ..KvProviderState::default() }); + let opt_out = |at| RBatch { o: [("p".to_owned(), at)].into_iter().collect(), ..RBatch::default() }; + assert!(r_batch_applied(&opt_out(5), &entry, &keys), "KV o.at at or after"); + assert!(!r_batch_applied(&opt_out(6), &entry, &keys), "newer opt-out pending"); + entry.ps.get_mut("p").expect("p").oc = Some(6); + assert!(!r_batch_applied(&opt_out(6), &entry, &keys), "same-second clear does not count"); + entry.ps.get_mut("p").expect("p").oc = Some(7); + assert!(r_batch_applied(&opt_out(6), &entry, &keys), "later clear makes it stale"); + } + + #[test] + fn unopenable_cookies_are_listed_for_deletion() { + let keys = test_keys(); + let mut headers = http::HeaderMap::new(); + headers.insert(http::header::COOKIE, http::HeaderValue::from_static("__Host-ts-eids-r-00000000=v1.bad.value")); + let opened = open_r_cookies(&keys, EC, &headers); + assert!(opened.cookies.is_empty(), "nothing opens"); + assert_eq!(opened.unopenable, vec!["__Host-ts-eids-r-00000000".to_owned()], "listed for deletion"); + } + } +``` + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::cookie::tests::r_cookie` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Serialize `RBatch` as compact JSON and omit empty maps. `plan_r_batch` seals once and measures. `split_r_batch` packs greedily: removals one by one, then each provider's opt-out, then each `iv` hash, sealing after each addition and starting a new cookie when the sealed value would pass 512. A single item that can never fit (impossible within the measured budget) is logged at error and still emitted in its own cookie. Never drop it. `r_batch_applied` implements exactly the three spec 6.2 bullets. `iv` entries never keep a cookie alive. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::cookie` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity +git commit --signoff -S -m "Add per-batch ts-eids-r removal cookies" +``` + +### Task 19: `[ec.identity]` settings + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/config.rs` +- Modify: `crates/trusted-server-core/src/settings.rs` (`Ec`, around line 542) +- Modify: `trusted-server.example.toml` (the `[ec]` section) +- Test: `config.rs`, `settings.rs` + +**Interfaces:** + +- Produces: + - `IdentitySettings { max_concurrency: Option, hold_budget_ms: u64, hold_retry_sec: u64, max_response_bytes: usize, pending_grace_sec: u64, staged_max_age_sec: u64, kv_revalidate_sec: u64, providers: BTreeMap }`, with the defaults from Global Constraints and `Default` + - `ProviderSettings { mode: ProviderMode, timeout_ms: Option, sources: Option>, refresh_after_sec: u64, max_calls_per_minute: u32, breaker_failures_per_minute: u32, cookie_priority: i32 }` + - `enum ProviderMode { Hold, AfterSend }`, serde `snake_case`, default `AfterSend` + - `IdentitySettings::effective_max_concurrency(&self, legacy_pull_concurrency: usize) -> usize` + - `ProviderSettings::effective_timeout_ms(&self, settings: &IdentitySettings) -> u64` + - `IdentitySettings::provider(&self, id: &IdentityProviderId) -> ProviderSettings`, which returns the defaults when there is no table + - `pub(crate) fn validate_identity_settings(settings: &IdentitySettings) -> Result<(), Report>`, covering the numeric rules of spec 10.2: `hold_budget_ms <= 1000`, `max_concurrency != Some(0)`, `refresh_after_sec > 0`, `kv_revalidate_sec < 86_400`, `timeout_ms <= hold_budget_ms` for hold mode. Unknown provider IDs are checked later, in Task 22, when the catalog exists. + - `Ec.identity: IdentitySettings` with `#[serde(default)]` + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn defaults_without_any_table() { + let settings: IdentitySettings = toml::from_str("").expect("should parse empty"); + assert_eq!(settings.hold_budget_ms, 100, "hold budget"); + assert_eq!(settings.hold_retry_sec, 300, "hold retry"); + assert_eq!(settings.max_response_bytes, 65_536, "max bytes"); + assert_eq!(settings.pending_grace_sec, 300, "grace"); + assert_eq!(settings.staged_max_age_sec, 604_800, "staged age"); + assert_eq!(settings.kv_revalidate_sec, 3600, "revalidate"); + assert_eq!(settings.effective_max_concurrency(4), 4, "falls back to the legacy key"); + let provider = settings.provider(&"anything".parse().expect("id")); + assert_eq!(provider.mode, ProviderMode::AfterSend, "mode"); + assert_eq!(provider.effective_timeout_ms(&settings), 1000, "after-send timeout"); + assert_eq!((provider.refresh_after_sec, provider.max_calls_per_minute, provider.breaker_failures_per_minute, provider.cookie_priority), (86_400, 1, 50, 0), "provider defaults"); + validate_identity_settings(&settings).expect("defaults are valid"); + } + + #[test] + fn hold_mode_timeout_defaults_to_the_budget_and_bounds_are_enforced() { + let settings: IdentitySettings = toml::from_str( + "hold_budget_ms = 120\n[providers.id5]\nmode = \"hold\"\n", + ) + .expect("should parse"); + let provider = settings.provider(&"id5".parse().expect("id")); + assert_eq!(provider.effective_timeout_ms(&settings), 120, "hold timeout = budget"); + + for (name, toml_text) in [ + ("budget over 1000", "hold_budget_ms = 1001"), + ("zero concurrency", "max_concurrency = 0"), + ("revalidate a day", "kv_revalidate_sec = 86400"), + ("zero refresh", "[providers.p]\nrefresh_after_sec = 0"), + ("timeout over budget", "[providers.p]\nmode = \"hold\"\ntimeout_ms = 101"), + ] { + let settings: IdentitySettings = toml::from_str(toml_text).expect(name); + validate_identity_settings(&settings).expect_err(name); + } + toml::from_str::("[providers.p]\nmode = \"sometimes\"").expect_err("bad mode"); + } +} +``` + +Use whichever TOML crate core already uses in tests (`grep -n "toml" crates/trusted-server-core/Cargo.toml`). If only `config` is available, build the value through the same path `settings.rs` tests use. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::config` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Write the structs with `#[serde(default = "...")]` per field and `#[serde(deny_unknown_fields)]`, which matches the rest of `settings.rs`. Check the convention there first. Add to `Ec`: + +```rust + /// Server-side identity provider policy (`[ec.identity]`). + #[serde(default)] + pub identity: IdentitySettings, +``` + +Call `validate_identity_settings(&self.ec.identity)` from both `validate_settings_for_deploy` and `validate_settings_for_runtime` in `config.rs`, next to `PartnerRegistry::...`. Mark `pull_sync_concurrency`'s doc comment as deprecated in favour of `[ec.identity].max_concurrency`, and log a `warn!` once at settings load when `identity.max_concurrency` is unset and `pull_sync_concurrency` is not its default. In `trusted-server.example.toml`, add a commented `[ec.identity]` block that copies spec 10.1's comments and values, with `example.com` partner names. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::config`, `cargo test-fastly -- settings`, `cargo test-fastly -- config::` +Expected: PASS. Existing configs with no `[ec.identity]` still parse. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/trusted-server-core/src trusted-server.example.toml +git commit --signoff -S -m "Add [ec.identity] coordinator settings" +``` + +### Task 20: Coordinator eligibility and outcomes + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/coordinator.rs` (pure logic only in this task) +- Modify: `mod.rs`, `migration_guards.rs` +- Test: `coordinator.rs` + +**Interfaces:** + +- Consumes: catalog, servable, mutation, cookie view, config, policy (Tasks 12 to 19) +- Produces: + - `pub(crate) struct EligibilityView`, the merged view of spec 4.7, built by `EligibilityView::merge(kv: Option<&KvEntry>, cookie: Option<&SCookie>, removals: &[RBatch], catalog, now) -> Self`. It holds per-source records (with KV as the base when read, else the cookie's last confirmed copy, cookie `k: 0` entries on top when their `pv` matches, and `ts-eids-r` removals hiding records), per-provider `n` (the later of cookie and KV), `iv` (union), `o` (the staged opt-out counts unless `ts-eids-s` stages a later `oc`, and the mirrored opt-out yields to KV when KV was read), and markers (cookie only). + - `pub(crate) struct RequestFacts { is_real_browser, ec_persistence_allowed, policy_signals: PolicySignals, writer: bool, new_visitor: bool, runs_auction: bool, now: UnixTime }` + - `pub(crate) enum Phase { Hold, AfterSendDryRun, AfterSend }` + - `pub(crate) enum Eligibility { Eligible { kinds: BTreeMap }, Skip(SkipReason) }`, with a `SkipReason` enum naming each of steps 1 to 8 (step 9, the burst cap, is checked at dispatch) + - `pub(crate) fn eligibility(provider: &dyn IdentityProvider, settings: &ProviderSettings, identity: &IdentitySettings, view: &EligibilityView, facts: &RequestFacts, consent: &ConsentContext, phase: Phase, catalog: &ProviderCatalog) -> Eligibility` + - `pub(crate) enum CallOutcome { Resolved(Resolution), Skipped, InputUnavailable, InvalidState(Option), HoldCutOff, InputFailure, TransportFailure, BreakerOpen }` + - `pub(crate) fn outcome_to_changes(provider: &dyn IdentityProvider, outcome: CallOutcome, view: &EligibilityView, context: &OutcomeContext<'_>) -> OutcomeChanges`, where `OutcomeChanges { removals: Vec, result: Option, state: ProviderStateChange }`. This is the section 8 table, row by row. `OutcomeContext` holds `now`, `keys`, `catalog`, provider settings, `identity` settings, `expected_state` and the consent fingerprint bits. + +Eligibility steps, in order (spec 4.7): + +1. `facts.is_real_browser` +2. `facts.ec_persistence_allowed` +3. `policy_allows(descriptor.consent_policy(), consent, facts.policy_signals)` +4. Per-source kinds: + - **Owned by this provider.** `Initial` when there is no record. `Refresh` when there is one and it is due: expired, written by someone else, `ra` passed, or no `ra` and `at + refresh_after_sec` passed or `at` missing. A browser-seeded UID whose hash is in `iv` is never due. + - **Unowned.** `Initial` only, and only when there is no servable record. An expired record, or one whose writer is unconfigured, counts as absent. + - A source whose kind the provider does not list, or whose needs (`resolves.needs(kind)`) are not available this request, is left out. No sources left means skip. +5. `n` has passed, and no sticky `o` is set (`oc` not strictly later than `o.at`). +6. No unanswered marker for the provider. +7. (Breaker: checked at dispatch in Part 2, Task 35. Part 1 has no breaker.) +8. Mode: + - No provider runs when `facts.new_visitor`. + - `Hold` needs `settings.mode == Hold` and `facts.runs_auction`. A stateful provider also needs `facts.writer`. + - `AfterSendDryRun` and `AfterSend` need `facts.writer`, and either `settings.mode == AfterSend`, or a stateful hold-mode provider on a writer navigation that does not run an auction. + +The needs check in step 4 needs to know which inputs are available. Add `available_inputs: BTreeSet` to `RequestFacts`. + +- [ ] **Step 1: Write the failing tests** + +Write one test per eligibility step and one per section 8 row, using `fixed_provider` and `ProviderSettings::default()`. Name them after the rule, for example `skips_when_not_a_real_browser`, `skips_new_visitor_in_every_mode`, `owned_record_written_by_someone_else_is_due_for_refresh`, `browser_seeded_uid_in_iv_is_never_due`, `unowned_source_with_servable_record_is_not_initial`, `expired_unowned_record_counts_as_absent`, `stateful_hold_provider_runs_after_send_without_auction`, `hold_provider_never_runs_on_stager_when_stateful`, `no_match_waits_retry_in_capped_at_seven_days`, `input_failure_backs_off_exponentially_to_a_day`, `transport_failure_backs_off_to_an_hour`, `hold_cut_off_waits_hold_retry_without_counting_a_failure`, `skip_waits_refresh_after`, `invalid_state_clears_state_and_keeps_f`, `oversized_state_rejects_positive_part_but_keeps_removals`, `opted_out_removes_everything_and_clears_state`, `invalidate_records_iv_hash_and_sets_n_now`. Here are three written out in full. Write the rest in the same shape: + +```rust + #[test] + fn skips_new_visitor_in_every_mode() { + let provider = fixed_provider("p", &["a.example.com"]); + let view = EligibilityView::empty(); + let catalog = ProviderCatalog::new(vec![provider.clone()], BTreeMap::new()); + for phase in [Phase::Hold, Phase::AfterSendDryRun, Phase::AfterSend] { + let facts = RequestFacts { new_visitor: true, ..facts_for_writer() }; + assert!( + matches!( + eligibility(provider.as_ref(), &ProviderSettings::default(), &IdentitySettings::default(), &view, &facts, &allowing_consent(), phase, &catalog), + Eligibility::Skip(SkipReason::NewVisitor) + ), + "no provider runs on a new visitor's first navigation ({phase:?})" + ); + } + let returning = facts_for_writer(); + assert!( + matches!( + eligibility(provider.as_ref(), &ProviderSettings::default(), &IdentitySettings::default(), &view, &returning, &allowing_consent(), Phase::AfterSend, &catalog), + Eligibility::Eligible { .. } + ), + "control: a returning writer is eligible" + ); + } + + #[test] + fn no_match_waits_retry_in_capped_at_seven_days() { + let provider = fixed_provider("p", &["a.example.com"]); + let context = outcome_context_at(1_000); + let default = outcome_to_changes(provider.as_ref(), CallOutcome::Resolved(Resolution::no_match()), &EligibilityView::empty(), &context); + assert_eq!(default.state.n, Some(1_000 + 86_400), "24 h default"); + let long = outcome_to_changes( + provider.as_ref(), + CallOutcome::Resolved(Resolution::no_match().with_retry_in(Duration::from_secs(30 * 86_400))), + &EligibilityView::empty(), + &context, + ); + assert_eq!(long.state.n, Some(1_000 + 7 * 86_400), "capped at 7 days"); + } + + #[test] + fn transport_failure_backs_off_to_an_hour() { + let provider = fixed_provider("p", &["a.example.com"]); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 5); + let changes = outcome_to_changes(provider.as_ref(), CallOutcome::TransportFailure, &view, &outcome_context_at(1_000)); + assert_eq!(changes.state.f, Some(6), "should count the failure"); + assert_eq!(changes.state.n, Some(1_000 + 3_600), "min(5 min x 2^5, 1 h) = 1 h"); + } +``` + +Add `facts_for_writer()`, `allowing_consent()` and `outcome_context_at(now)` to `test_support.rs`. `facts_for_writer` returns a real browser, persistence allowed, a writer, a returning visitor, no auction, and every input available. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::coordinator` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Implement `EligibilityView::merge`, `eligibility` and `outcome_to_changes` as plain functions. In `outcome_to_changes`, `Resolved(resolution)` does the following: + +- It drops IDs for sources the provider does not claim (after `sources` narrowing from `ProviderSettings`), IDs with an empty UID or one over `MAX_UID_LENGTH`, and duplicates. It logs each drop with the provider ID and source only. +- An ID of an expiring source without an expiry keeps the stored `exp` when its UID is unchanged and is dropped otherwise. +- `with_refresh_until` from a provider without private state is ignored with a warning. +- Each kept ID becomes `OwnerWrite` with `expected = record_revision(stored)` (or `None` when no record) when this provider owns the source, else `FillMissing { writer: Some(provider) }`. +- `invalidated` becomes `OwnerRemove` with the revision read, plus an `iv` hash of the stored UID, plus `n = now`. +- `opted_out` becomes `WriterRemove` for every source the provider claims (the mutation also removes the unowned records it wrote), plus `opt_out = KvOptOut { at: now, fp }` and `st = Clear`. +- When the provider has private state, the positive IDs plus the state change plus `n` and `f` go into one `ProviderResult`. Otherwise they go into `removals`, a plain change list, and `state`. +- `state_rejected()` drops the positive part and applies the `Malformed` backoff. + +Backoff arithmetic uses `saturating_pow` and `min`, so a large `f` cannot overflow. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::coordinator` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/ec/identity crates/trusted-server-core/src/migration_guards.rs +git commit --signoff -S -m "Add coordinator eligibility and outcome rules" +``` + +### Task 21: Coordinator dispatch + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/identity/coordinator.rs` +- Modify: `crates/trusted-server-core/src/ec/identity/test_support.rs` (fake HTTP client) +- Test: `coordinator.rs` + +**Interfaces:** + +- Consumes: `RuntimeServices::http_client().send_async`, `services.backend().ensure(&PlatformBackendSpec { .. })` exactly as `dispatch_pull_sync` uses them (`ec/pull_sync.rs`, the body of `dispatch_pull_sync`), the bounded decoders in `streaming_processor.rs`, and the rate limiter's `exceeded_per_minute` +- Produces: + - `pub(crate) struct IdentityDeps<'a> { settings: &'a Settings, catalog: &'a ProviderCatalog, keys: &'a IdentityKeys, services: &'a RuntimeServices, rate_limiter: Option<&'a dyn RateLimiter> }`. This one struct keeps `handle_auction` and `handle_publisher_request` within 7 arguments. + - `pub(crate) struct Call { provider: IdentityProviderId, kinds: BTreeMap }` + - `pub(crate) fn dispatch(deps: &IdentityDeps<'_>, calls: Vec, signals: &ResolveSignals<'_>, budget: Option) -> Vec<(IdentityProviderId, CallOutcome)>` + - `pub(crate) fn check_request(request: &ProviderRequest, descriptor: &ProviderDescriptor) -> Result<(), Report>`: `https` only, the host in `resolves.hosts()`, no `Cookie` header + - `pub(crate) fn decode_body(encoding: Option<&str>, body: Vec, cap: usize) -> Option>`: identity, `gzip` or `br`, the cap applied to the decoded size, `None` when over the cap or for an unknown encoding + +Dispatch per call: + +1. Build the `ResolveInput` and call `request`. + - `Ok(None)` is `Skipped`. + - `Err` with `InputUnavailable` is `InputUnavailable`. + - `Err` with `UndeclaredInput` or any other `Config` is `InputFailure`, logged as `Config` with `current_context()` only. + - Any recorded undeclared read is also logged as `Config` and turns the outcome into `InputFailure`. +2. `check_request`. On failure: log `Config`, `InputFailure`. +3. The burst cap: `rate_limiter.exceeded_per_minute(&format!("identity:{provider}:{}", visitor_key(keys, ec_id)), settings.max_calls_per_minute)`. Exceeded means not dispatched (Task 29 adds the 60-second marker backoff). A limiter error is logged and the call is skipped. +4. `ensure` a backend with first-byte timeout `effective_timeout_ms`, or in hold mode `min(timeout, remaining budget)` rounded up to the orchestrator's buckets (`auction/orchestrator.rs`, search for the bucket rounding). Do not follow redirects. Send with `send_async`, keeping at most `effective_max_concurrency` in flight. +5. Collect responses: check `Content-Length` against `max_response_bytes`, buffer, `decode_body`. Over the cap is `TransportFailure`. Build the `ProviderResponse` with the request's call context. Call `response`. + - `Ok(resolution)` is `Resolved`, or `InvalidState(Some(resolution))` when the input recorded a state decode failure. + - `Rejected` with 429 or 5xx is `TransportFailure`. + - Other `Rejected`, `Malformed` and `Config` are `InputFailure`. + - `InvalidState` is `InvalidState(None)`. +6. A hold-mode call still pending when the budget runs out is dropped and becomes `HoldCutOff`. + +- [ ] **Step 1: Write the failing tests** + +Add a `FakeHttpClient` to `test_support.rs`. It implements the platform HTTP client trait that `RuntimeServices` holds (`grep -n "trait .*HttpClient" crates/trusted-server-core/src/platform/*.rs`) and returns canned responses per host, with an optional delay counter for hold-budget tests. Check first whether `auction/test_support.rs` already has one, and reuse it if so. Then test: + +```rust + #[test] + fn check_request_enforces_https_host_and_no_cookie() { + let descriptor = fixed_provider("p", &["a.example.com"]).descriptor().clone(); + let good = ProviderRequest::get(&"https://api.example.com/x".parse().expect("url")); + check_request(&good, &descriptor).expect("control: allowed"); + for (name, request) in [ + ("http", ProviderRequest::get(&"http://api.example.com/x".parse().expect("url"))), + ("host", ProviderRequest::get(&"https://evil.example.net/x".parse().expect("url"))), + ("cookie", good.clone().try_with_header("cookie", "a=1").expect("header")), + ] { + check_request(&request, &descriptor).expect_err(name); + } + } + + #[test] + fn decode_body_caps_decoded_size() { + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + std::io::Write::write_all(&mut encoder, &vec![b'a'; 10_000]).expect("write"); + let gzipped = encoder.finish().expect("finish"); + assert_eq!(decode_body(Some("gzip"), gzipped.clone(), 10_000).map(|b| b.len()), Some(10_000), "fits"); + assert!(decode_body(Some("gzip"), gzipped, 9_999).is_none(), "over the cap after decoding"); + assert!(decode_body(Some("zstd"), vec![1], 10).is_none(), "unknown encoding"); + assert_eq!(decode_body(None, b"ok".to_vec(), 10).as_deref(), Some(&b"ok"[..]), "identity"); + } +``` + +Also write `dispatch_maps_status_codes_to_outcomes` (200 is `Resolved`, 503 is `TransportFailure`, 400 is `InputFailure`, a body over the cap is `TransportFailure`) and `undeclared_read_is_an_input_failure_even_when_ignored`, using a provider that reads `user_agent().ok()` without declaring it, against the fake client. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::coordinator` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Follow the transport code in `dispatch_pull_sync` closely. It already handles backend resolution, `send_async`, draining the in-flight batch (`drain_pull_batch`), the `Content-Length` check (`response_content_length_exceeds_limit`) and buffering (`extract_pull_uid`). Move those three helpers out of `pull_sync.rs` into `coordinator.rs`, generalized to `max_response_bytes`, and make `pull_sync.rs` call the moved versions until Task 29 deletes its dispatcher. For decoding, reuse the bounded decoders in `streaming_processor.rs` if a buffered entry point exists (`grep -n "pub(crate) fn\|pub fn" crates/trusted-server-core/src/streaming_processor.rs`). Otherwise use `flate2::read::MultiGzDecoder` and `brotli::Decompressor` with `Read::take(cap as u64 + 1)` and reject when more than `cap` bytes come out. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity::coordinator` and `cargo test-fastly -- ec::pull_sync` +Expected: PASS. Pull sync behaviour is unchanged so far. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Add coordinator dispatch with request checks and byte caps" +``` + +### Task 22: `GenericPullProvider`, the write-counting store and the basic harness + +**Files:** + +- Create: `crates/trusted-server-core/src/ec/identity/generic_pull.rs` +- Modify: `crates/trusted-server-core/src/ec/identity/test_support.rs` (`ProviderHarness` basics, `WriteCountingEcKv`) +- Modify: `crates/trusted-server-core/src/ec/identity/catalog.rs` (`ProviderCatalog::from_settings`) +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (`KvIdentityGraph::write_counting` constructor, test support) +- Test: `generic_pull.rs` + +**Interfaces:** + +- Consumes: `PartnerConfig` (`ec/registry.rs:22`), `validated_pull_sync_url` and `build_pull_request_url` (`ec/pull_sync.rs`, made `pub(crate)`) +- Produces: + - `pub(crate) struct GenericPullProvider` with `pub(crate) fn from_partner(partner: &PartnerConfig) -> Result>` + - `pub(crate) fn generic_pull_id(source_domain: &str) -> IdentityProviderId`: `pull-` plus the domain with `.` replaced by `-`. Over 63 characters, it is cut to 54 characters, followed by `-` and the first 8 hex digits of SHA-256 of the full domain. + - `ProviderCatalog::from_settings(settings: &Settings, registry: &PartnerRegistry) -> Result>`. In Part 1 it holds generic pull providers only, with no owners, and rejects a duplicate provider ID and more than 32 providers. + - `pub(crate) fn legacy_rate_limit_per_minute(pull_sync_rate_limit: u32) -> u32` = `max(1, ceil(limit / 60))` + - Test support: + - `WriteCountingEcKv` (an in-memory store that counts `Add`, `IfGenerationMatch` and `Overwrite` writes and lookups) with `pub(crate) fn writes(&self) -> WriteCounts` + - `KvIdentityGraph::write_counting(name) -> (KvIdentityGraph, Arc)` + - `ProviderHarness` with `new`, the `with_*` setters, `request`, `request_json`, `respond`, `respond_raw` and `resolve`. Conformance checks come in Task 39. + +Generic pull's descriptor: + +```rust +let id = generic_pull_id(&partner.source_domain); +let mut resolves = Resolves::new().with_ec_id(); +for host in &partner.pull_sync_allowed_domains { + resolves = resolves.with_host(host); +} +ProviderDescriptor::builder(id) + .with_source(partner.source_domain.parse()?) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::eid_persistence_only()) + .build() +``` + +Its `request` is a `GET` of `validated_pull_sync_url(partner)` with `?ec_id=` (via `build_pull_request_url`) and `with_bearer(ts_pull_token)`. Skip (`Ok(None)`) when there is no valid URL or token. Its `response` is `success_json::<{ uid: String }>()`. An empty UID or one over `MAX_UID_LENGTH` (`is_valid_eid_uid`) is `no_match`, and any other UID is `found([ResolvedId::new(source, uid)])`. The provider's settings default to `mode = AfterSend`, `max_calls_per_minute = legacy_rate_limit_per_minute(partner.pull_sync_rate_limit)`, and `refresh_after_sec = partner.pull_sync_ttl_sec` (used only when Part 2's `owner` makes it authoritative). A `[ec.identity.providers.]` table overrides these. Expose `fn default_settings_for(partner) -> ProviderSettings` and make `IdentitySettings::provider` fall back to it for `pull-` IDs. To do that, `provider()` takes an `Option` base. + +- [ ] **Step 1: Port today's pull sync tests** + +List them with `grep -n "fn " crates/trusted-server-core/src/ec/pull_sync.rs | sed -n '/mod tests/,$p'`. For each test of protocol behaviour (URL building, bearer, response parsing, 64 KiB cap, allowlist, empty or oversized UID), write the equivalent against `GenericPullProvider` through `ProviderHarness`. For example: + +```rust + #[test] + fn calls_the_partner_url_with_the_ec_id_and_bearer() { + let partner = partner_config("ssp.example.com", "https://sync.ssp.example.com/pull", "example-token-value-0123456789abcdef"); + let provider = GenericPullProvider::from_partner(&partner).expect("should build"); + let mut harness = ProviderHarness::new(&provider).with_ec_id(TEST_EC_ID); + + let request = harness.request().expect("should build").expect("should not skip"); + + assert_eq!(request.url().host_str(), Some("sync.ssp.example.com"), "host"); + assert_eq!( + request.url().query_pairs().find(|(name, _)| name == "ec_id").map(|(_, value)| value.into_owned()), + Some(TEST_EC_ID.to_owned()), + "should pass the EC ID" + ); + assert_eq!(request.headers()["authorization"], "Bearer example-token-value-0123456789abcdef", "bearer"); + + let resolution = harness.respond(StatusCode::OK, &json!({ "uid": "partner-uid" })).expect("should parse"); + assert_eq!(resolution.ids()[0].uid(), "partner-uid", "uid"); + } + + #[test] + fn provider_ids_are_derived_and_bounded() { + assert_eq!(generic_pull_id("ssp.example.com").as_str(), "pull-ssp-example-com", "simple"); + let long = format!("{}.example.com", "a".repeat(70)); + let id = generic_pull_id(&long); + assert!(id.as_str().len() <= 63, "bounded"); + assert_ne!(id, generic_pull_id(&format!("{}.example.com", "a".repeat(71))), "unique by hash"); + } + + #[test] + fn legacy_hourly_limit_maps_to_per_minute() { + assert_eq!(legacy_rate_limit_per_minute(0), 1, "at least one"); + assert_eq!(legacy_rate_limit_per_minute(59), 1, "rounds up"); + assert_eq!(legacy_rate_limit_per_minute(65), 2, "65/hr is 2/min, as the limiter does today"); + } +``` + +`ProviderHarness::with_ec_id` is crate-private (only generic pull reads the EC ID). Add it next to the public setters with `pub(crate)`. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::identity::generic_pull` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Write `generic_pull.rs`, `ProviderCatalog::from_settings`, the harness basics and the counting store. The harness builds `ResolveSignals` from its setters (`SeededRandom` from `with_random_seed`, default seed `1`, default `now` 1,760,000,000). It computes kinds as spec 13 says: a source with a `with_stored` record is `Refresh`, any other is `Initial`, and sources whose kind's needs are missing are dropped, with `Ok(None)` and no provider call when none are left. It keeps the built request's call context for `respond`. `respond` builds a `ProviderResponse` from `serde_json::to_vec(body)`. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::identity` and `cargo test-fastly -- ec::pull_sync` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Model pull sync partners as GenericPullProvider" -m "Adds the provider harness basics and a KV store that counts writes by mode." +``` + +--- + +### Task 23: The writer decision and request-scoped identity state on `EcContext` + +**Files:** + +- Modify: `crates/trusted-server-core/src/http_util.rs` (add `is_document_navigation`, move `is_prefetch_request` here) +- Modify: `crates/trusted-server-core/src/publisher.rs` (`is_prefetch_request` at about line 3200 becomes a re-export or its callers switch) +- Modify: `crates/trusted-server-core/src/ec/mod.rs` (`EcContext` fields and accessors) +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` (`build_ec_request_state`, the publisher fallback, and the integration-proxy EC generation path) +- Modify: `crates/trusted-server-core/src/integrations/registry.rs` (integration-proxy EC generation, about line 1046, gains the real-browser check) +- Test: `http_util.rs`, `ec/mod.rs`, `adapter-fastly/src/app.rs` + +**Interfaces:** + +- Produces: + - `pub fn is_document_navigation(req: &Request) -> bool`: `Sec-Fetch-Dest: document` only, with no `Accept` fallback + - `pub fn is_prefetch_request(req: &Request) -> bool`, moved unchanged and made public + - `pub(crate) struct RequestIdentity` stored on `EcContext`, with accessors: + - `writer: bool` + - `user_agent: Option`, `dnt: bool`, `page_url: Option` (captured in phase 1) + - `s_cookie: SOpen`, `removals: OpenedRemovals` (opened lazily, at most once, Task 16 and 18 types) + - `mutation: IdentityMutation` + - `staged: StagedCookies`: what this response must set or delete. `s_cookie: Option` to set, `expire_s: bool`, `new_r_batch: Option`, `delete_r: Vec`. + - `EcContext::identity(&self) -> &RequestIdentity` and `identity_mut(&mut self) -> &mut RequestIdentity` + - `EcContext::decide_writer(&mut self, req: &Request, organic_route: bool, is_real_browser: bool)`. It sets `writer = organic_route && is_document_navigation(req) && !is_prefetch_request(req) && is_real_browser` and captures the user agent, DNT and the page URL (the request URL without query or fragment). + - `EcContext::open_identity_cookies(&mut self, keys: &IdentityKeys, catalog: &ProviderCatalog, headers: &HeaderMap)`: idempotent, and a no-op without an EC ID + +The writer decision is made in the adapter where routing already decides EC generation (spec 5.1). That is after pre-route filters allow dispatch: + +- In the publisher fallback branch of `app.rs`, next to `let is_navigation = is_navigation_request(&req);`, call `ec.ec_context.decide_writer(&req, true, ec.is_real_browser)`. +- In the integration-proxy branch, call `decide_writer(&req, true, is_real_browser)` before `handle_proxy`, and make proxy EC generation require `is_real_browser` explicitly, not just a present KV graph (`registry.rs:1048`). `ProxyDispatchInput` gains `is_real_browser: bool` for this. +- Every other route leaves `writer = false`. + +- [ ] **Step 1: Write the failing tests** + +In `http_util.rs`: + +```rust + #[test] + fn document_navigation_requires_sec_fetch_dest_document() { + let with_dest = Request::builder() + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("request"); + let accept_only = Request::builder() + .header(header::ACCEPT, "text/html") + .body(EdgeBody::empty()) + .expect("request"); + let iframe = Request::builder() + .header("sec-fetch-dest", "iframe") + .body(EdgeBody::empty()) + .expect("request"); + assert!(is_document_navigation(&with_dest), "document counts"); + assert!(!is_document_navigation(&accept_only), "the Accept fallback does not"); + assert!(is_navigation_request(&accept_only), "control: the broad check still accepts it"); + assert!(!is_document_navigation(&iframe), "iframes do not"); + } +``` + +In `ec/mod.rs` tests: + +```rust + #[test] + fn decide_writer_needs_every_condition() { + let navigation = || { + Request::builder() + .uri("https://www.example.com/a?b=c#d") + .header("sec-fetch-dest", "document") + .header(header::USER_AGENT, "ExampleBrowser/1.0") + .header("dnt", "1") + .body(EdgeBody::empty()) + .expect("request") + }; + let mut context = EcContext::default(); + context.decide_writer(&navigation(), true, true); + assert!(context.identity().writer(), "organic real-browser document navigation"); + assert_eq!(context.identity().page_url().map(Url::as_str), Some("https://www.example.com/a"), "no query"); + assert_eq!(context.identity().user_agent(), Some("ExampleBrowser/1.0"), "ua captured"); + assert!(context.identity().dnt(), "dnt captured"); + + for (name, organic, browser, request) in [ + ("named route", false, true, navigation()), + ("bot", true, false, navigation()), + ( + "prefetch", + true, + true, + Request::builder() + .header("sec-fetch-dest", "document") + .header("sec-purpose", "prefetch") + .body(EdgeBody::empty()) + .expect("request"), + ), + ] { + let mut context = EcContext::default(); + context.decide_writer(&request, organic, browser); + assert!(!context.identity().writer(), "not a writer: {name}"); + } + } +``` + +Use the prefetch headers `is_prefetch_request` actually checks. Read it before writing the test. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- http_util::tests::document_navigation` and `cargo test-fastly -- ec::tests::decide_writer` +Expected: compile errors. + +- [ ] **Step 3: Implement** + +Add `RequestIdentity` (in `ec/identity/mod.rs` as `pub(crate) mod request;`, file `ec/identity/request.rs`, registered in `migration_guards.rs`) with `Default`, the accessors above, and setters used by later tasks. Add the field `identity: RequestIdentity` to `EcContext` (it already derives `Default`, so this needs `RequestIdentity: Default + Clone`). `IdentityMutation`, `SCookie`, `OpenedRemovals` and `RBatch` need `Clone`, so add `#[derive(Clone)]` where it is missing. `EcFinalizeState` must stay `Clone + Send + Sync`, and every type here is plain data, so it does. + +Move `is_prefetch_request` into `http_util.rs` as `pub`. Replace the body in `publisher.rs` with `use crate::http_util::is_prefetch_request;` at the top, and delete the old definition. + +Wire `decide_writer` in `app.rs` as described above. Add `is_real_browser` to `ProxyDispatchInput` and set it from `EcRequestState`. In `IntegrationRegistry::handle_proxy`, require it before proxy EC generation. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly` (full: the adapter and core) +Expected: PASS. Behaviour is unchanged so far, apart from integration-proxy EC generation now needing a real browser. If an existing proxy test minted an EC for a non-browser request, update it and note the change in the commit body. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/ +git commit --signoff -S -m "Decide the writer request once routing allows dispatch" -m "Only an organic real-browser top-level document navigation that is not a prefetch is the writer. Integration-proxy EC generation now requires a real browser explicitly." +``` + +### Task 24: `/identify` stops writing; cluster size rides on the writer + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/identify.rs` (the `evaluate_cluster` call at about line 107) +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (`evaluate_cluster` at about line 1484 gains a non-writing variant) +- Test: `ec/identify.rs`, `ec/kv.rs` + +**Interfaces:** + +- Produces: + - `KvIdentityGraph::compute_cluster_size(&self, ec_id: &str) -> Result>`: the prefix list and count only, no write + - `/identify` uses a stored `cluster_size` when present and `compute_cluster_size` otherwise, and never writes + - `pub(crate) fn queue_cluster_size(context: &mut EcContext, graph: &KvIdentityGraph, ec_id: &str)`, called by phase 4 (Task 29) when its live row lacks `cluster_size` + +- [ ] **Step 1: Write the failing test** + +```rust + #[test] + fn identify_computes_cluster_size_without_writing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_row_without_cluster_size(&graph); + let response = identify_for_test(&graph, &ec_id); + + assert_eq!(response.status(), StatusCode::OK, "should answer"); + assert!(response_body_json(response)["cluster_size"].is_number(), "should report a size"); + assert_eq!(store.writes().total(), 0, "should not write"); + } +``` + +`seed_row_without_cluster_size`, `identify_for_test` and `response_body_json` are thin wrappers around what the existing identify tests already do. Write them in the test module from that setup. + +- [ ] **Step 2: Run and see it fail** + +Run: `cargo test-fastly -- ec::identify::tests::identify_computes_cluster_size_without_writing` +Expected: FAIL with one write. + +- [ ] **Step 3: Implement** + +Split `evaluate_cluster` into `compute_cluster_size` (list and count) and the existing write. `/identify` calls only `compute_cluster_size`. Keep `evaluate_cluster` for now. Task 29 calls it through the mutation (`IdentityMutation::set_cluster_size`), and once nothing else calls it, remove it in that task. + +- [ ] **Step 4: Run and see it pass** + +Run: `cargo test-fastly -- ec::identify` and `cargo test-fastly -- ec::kv` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/trusted-server-core/src/ec +git commit --signoff -S -m "Stop writing cluster size from /identify" +``` + +### Task 25: Finalize builds the mutation, and stagers save into cookies + +This is the core behaviour change of Part 1. `ec_finalize_response` stops writing KV for returning visitors. On the writer request it adds changes to the mutation. On stagers it saves changes into `__Host-ts-eids-s` and `__Host-ts-eids-r-*`. + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/finalize.rs` (`ec_finalize_response`, `finalize_unusable_consent`, `confirm_then_recover_orphaned_ec`, `recover_orphaned_ec`) +- Modify: `crates/trusted-server-core/src/ec/mod.rs` (`generate_if_needed` takes the browser ingest) +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (`create_if_absent` callers pass the seeded entry) +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` (pass the parsed `ts-eids` and sharedId to `generate_if_needed`) +- Test: `ec/finalize.rs` + +**Interfaces:** + +- Consumes: `RequestIdentity` (Task 23), `IdentityMutation` (Task 15), the cookie functions (Tasks 16 to 18), `ProviderCatalog`, `IdentityKeys` +- Produces: + - `ec_finalize_response` gains one argument, `identity: &IdentityFinalize<'_>`, which bundles `keys`, `catalog`, `identity_settings`, `now` and `random`, so the function stays within 7 arguments + - Writer, returning EC: adds `FillMissing` browser ingest (`acquired_at = now`) for `ts-eids` and sharedId, plus the flush-checked staged entries from `ts-eids-s` and every `ts-eids-r` batch, to `ec_context.identity_mut().mutation`. It writes nothing. + - Stager, returning EC: stages its changes in `staged.s_cookie` (positive) and `staged.new_r_batch` (removals). + - `pub(crate) fn emit_identity_cookies(context: &mut EcContext, identity: &IdentityFinalize<'_>, response: &mut Response)`, called at the end of every finalize path (the unusable-consent path included). It appends the `Set-Cookie` headers for `staged`, and it is the only place `__Host-ts-eids-*` cookies are set. + - `generate_if_needed(settings, kv, seed: &[PartnerIdUpdate])`: the new-visitor create carries the browser ingest, so finalize no longer writes for a new EC (spec 5.2, phase 1) + - Orphan rotation creates the new row carrying this request's browser ingest plus the removals, opt-outs, `oc` and `iv` from `ts-eids-r`, never positive staged IDs. When the create is `Written`, the response deletes the `ts-eids-r` cookies and sets a fresh `ts-eids-s` from the created row. Rotation empties the mutation and the markers. + +What each path does (spec 5.2, 5.3, 5.6): + +| Path | KV | Cookies | +| --------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| New EC (writer or not) | create before routing, carrying browser ingest (`generate_if_needed`) | `ts-ec` as today. `ts-eids-s` from the created row. | +| Returning EC, writer | none (phase 4 flushes the mutation) | `ts-eids-s` only under phase 3's rules: something staged, a marker, a merge change, or a missing, failed or stale cookie | +| Returning EC, stager | none | `ts-eids-s` when it saved an entry or its KV read changed the cookie. A new `ts-eids-r` when it produced a removal. | +| Unusable consent, not withdrawn | none | a new `ts-eids-r` when it produced a removal. `ts-eids-s` expired. | +| Consent withdrawn | today's tombstone path, unchanged | `ts-eids-s` expired. Every `ts-eids-r` deleted when the tombstone write succeeded, kept when it failed. | +| Any KV read that found a tombstone | none | `ts-eids-s` expired, every `ts-eids-r` deleted | +| Any KV read that found the row missing (no recovery here) | none | `ts-eids-s` replaced by the minimal `m: 1` cookie. `ts-eids-r` kept. | + +Flush checks for positive staged entries (spec 5.3). Write each as a predicate with its own test: + +- The cookie opened with this EC's ID (guaranteed by `open`). +- `now - sa <= staged_max_age_sec`. +- The writer `w` is in the catalog and still claims the source. For an owned source it also owns it. +- `gate_eid_updates_by_consent` and the writer's `policy_allows` pass under this request's consent. +- The writer has no sticky opt-out, unless the cookie stages a clear `oc` later than that opt-out's `at`. +- Conversion: `OwnerWrite { expected: pv }` when the writer owns the source, else `FillMissing`. A `b: 1` entry becomes `FillMissing { writer: None }`. + +Removals and staged `o`, `oc` and `iv` skip every check except the EC binding (spec 5.6). + +- [ ] **Step 1: Write the failing tests** + +Add a `mod identity_flow` test module to `finalize.rs`. Build requests and contexts the way the existing finalize tests do (`grep -n "fn finalize_" crates/trusted-server-core/src/ec/finalize.rs | head -40`). Use `KvIdentityGraph::write_counting`. Write these tests. The first two are shown in full: + +```rust + #[test] + fn returning_writer_navigation_adds_ingest_to_the_mutation_and_writes_nothing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, KvEntry::minimal("other.example.com", "x", 1)); + let mut context = returning_context(&ec_id, Writer::Yes); + let mut response = html_response(); + + ec_finalize_response( + &settings_with_partner("sharedid.org"), + &mut context, + Some(&graph), + ®istry_with_partner("sharedid.org"), + None, + Some("example-shared-id"), + &mut response, + &identity_finalize_for_test(), + ); + + assert_eq!(store.writes().total(), 0, "finalize must not write for a returning writer"); + assert!(!context.identity().mutation().is_empty(), "should queue the sharedId fill"); + assert!(set_cookie_names(&response).is_empty(), "no Set-Cookie on an unchanged returning HTML response"); + } + + #[test] + fn auction_stager_saves_the_kv_gap_into_ts_eids_s() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = seed_live_row(&graph, KvEntry::minimal("other.example.com", "x", 1)); + let mut context = returning_context(&ec_id, Writer::No); + context.set_eid_sync_source(EidSyncSource::Auction); + context.set_client_eids(vec![eid("gap.example.com", "body-only-uid")]); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + let mut response = json_response(); + + ec_finalize_response( + &settings_with_partner("gap.example.com"), + &mut context, + Some(&graph), + ®istry_with_partner("gap.example.com"), + None, + None, + &mut response, + &identity_finalize_for_test(), + ); + + assert_eq!(store.writes().total(), 0, "a stager never writes KV"); + let cookie = open_set_s_cookie(&response, &ec_id).expect("should set ts-eids-s"); + assert_eq!(cookie.ids.len(), 1, "should save one entry"); + assert!(cookie.ids[0].is_browser_gap(), "as a b:1 KV-gap entry"); + assert_eq!(cookie.ids[0].uid(), "body-only-uid", "with the body's UID"); + } +``` + +The others: + +- `stager_never_saves_positive_entries_when_consent_denies_persistence` (it may still set a `ts-eids-r`) +- `unusable_consent_path_still_sets_removal_cookie_and_expires_ts_eids_s` +- `withdrawal_expires_ts_eids_s_and_deletes_removals_only_after_tombstone_lands` +- `tombstone_read_expires_ts_eids_s_and_deletes_removal_cookies` +- `missing_row_on_stager_sets_minimal_m_cookie_and_keeps_removals` +- `orphan_rotation_carries_removals_but_never_positive_staged_ids` (with a `ts-eids-s` holding a `k: 0` provider entry and a `ts-eids-r` removal: the created row has the removal's opt-out and none of the staged IDs) +- `new_visitor_create_carries_browser_ingest_in_one_write` (one `Add` write, and the row holds the sharedId) +- `flush_checks_drop_stale_unowned_disabled_and_opted_out_entries_but_never_removals` (one assertion per check, each paired with a control entry that survives) +- `config_change_discards_positive_entries_and_keeps_markers_and_removals` + +`identity_finalize_for_test()`, `open_set_s_cookie`, `set_cookie_names`, `seed_live_row`, `returning_context` and the response builders go in the test module. Build them on `ec/identity/test_support.rs`. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- ec::finalize::tests::identity_flow` +Expected: compile errors, then assertion failures (finalize still writes). + +- [ ] **Step 3: Implement** + +Restructure `ec_finalize_response`: + +1. At the top, call `ec_context.open_identity_cookies(...)` once (it is idempotent and needs the EC ID). +2. Unusable consent: keep today's code, then stage any removals the route produced (Task 38 adds captures, so in Part 1 these come only from `ts-eids-r` carry-over), expire `ts-eids-s`, and call `emit_identity_cookies`. On withdrawal, delete the `ts-eids-r` cookies only when every tombstone write returned success. Check what `apply_withdrawal_tombstones` reports. If it returns nothing, make it return whether every write succeeded. +3. Returning EC with a graph: + - Collect browser ingest as today, then consent-gate it. + - If the request read KV (`kv_snapshot` is not `NotRead`), run `merge_with_kv` on the cookie and handle tombstone, missing and orphan recovery as the table says. Today's `confirm_then_recover_orphaned_ec` stays, with its `updates` argument extended to carry the `ts-eids-r` removals into the created row. + - **Writer:** add ingest, then the flush-checked staged entries, then every `ts-eids-r` batch, to the mutation. Set `staged.s_cookie` only under the phase 3 rules. Record which `ts-eids-r` cookies the phase 1 read shows applied (`r_batch_applied`) or unopenable, in `staged.delete_r`. + - **Stager:** run the KV-gap logic (Task 26 adds the `/auction` part) and save. Never touch `staged.delete_r`. + - Remove the call to `sync_eid_cookie_updates` and the `EidCookieSyncOutcome` measurement for these paths. Keep `record_eid_sync_terminal` for the new-EC create only, or delete it if nothing uses it. +4. New EC: `generate_if_needed` already wrote the seeded row. Finalize sets `ts-ec` as today and stages `ts-eids-s` from the created row. +5. Call `reconcile_pull_sync_marker` as today, then `emit_identity_cookies`. + +In `ec/mod.rs`, give `generate_if_needed` a `seed: &[PartnerIdUpdate]` parameter and apply it to the `KvEntry::new(..)` it creates, via `apply_partner_id_updates`, before `create_if_absent`. In `app.rs`, parse `ec.eids_cookie` and `ec.sharedid_cookie` with `collect_eid_updates` and `gate_eid_updates_by_consent` before calling `generate_if_needed`. Both helpers are `pub(crate)` in core, so add a `pub fn browser_seed_updates(settings, registry, consent, eids_cookie, sharedid_cookie, now) -> Vec` wrapper in `ec/finalize.rs` for the adapter. + +`ec_finalize_response` callers: the Fastly adapter's `finalize_with_ec_state` (`main.rs`, about line 597) and `publisher.rs` tests. Build `IdentityFinalize` from `AppState`, which holds the catalog and keys built once per settings load (Task 29 adds them to `AppState`). Until then, construct them in `finalize_with_ec_state` from settings. + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- ec::finalize` and then the full `cargo test-fastly` +Expected: PASS. Existing finalize tests that asserted a KV write on a returning navigation, `/auction` or `/_ts/page-bids` must now assert no write plus a queued mutation or a staged cookie. Update each one and list them in the commit body. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/ +git commit --signoff -S -m "Queue browser ingest in the writer's mutation and stage it on stagers" -m "Returning-visitor finalize no longer writes KV. The writer adds ingest, staged entries and removal batches to its single mutation. Stagers save positive entries in __Host-ts-eids-s and removals in a new __Host-ts-eids-r cookie. A new visitor's create now carries browser ingest, so it stays one write." +``` + +### Task 26: `/auction` saves the KV gap; `/_ts/page-bids` stops ingesting + +**Files:** + +- Modify: `crates/trusted-server-core/src/auction/endpoints.rs` (after the KV read at about lines 307-311) +- Modify: `crates/trusted-server-core/src/publisher.rs` (the `/_ts/page-bids` ingest at about line 6925) +- Modify: `crates/trusted-server-core/src/ec/finalize.rs` (stager branch) +- Test: `auction/endpoints.rs`, `publisher.rs` + +**Interfaces:** + +- Produces: `pub(crate) fn kv_gap_entries(body_eids: &[Eid], snapshot: &EcKvSnapshot, registry: &PartnerRegistry, consent: &ConsentContext, invalidated: &BTreeSet, now: u64) -> Vec`. For each registered source in the consent-gated body set that the KV row lacks or holds only expired, it picks the UID with `collect_eid_updates`'s rules and skips a UID whose hash is in the owner's `iv`. + +Rules (spec 5.3): + +- Save only when the client set came from the body, not from the `ts-eids` cookie. +- Save nothing when the KV read failed, found the row missing, or found a tombstone. +- `/auction` returns early on consent denial before it reads KV. Then nothing is saved. +- `/_ts/page-bids` no longer ingests `ts-eids` into KV. Its `EidSyncSource::PageBids` still lets finalize run cookie upkeep from its KV read. + +- [ ] **Step 1: Write the failing tests** + +In `auction/endpoints.rs`, next to the existing `/auction` + KV tests: + +- `auction_body_only_source_is_staged_not_written`: a registered source in the body and not in KV. Assert zero writes and a `ts-eids-s` `Set-Cookie` holding it. +- `auction_cookie_sourced_client_set_stages_nothing`: no body EIDs, `ts-eids` cookie present. No `ts-eids-s`. +- `auction_with_missing_row_stages_no_gap_and_sets_m`. +- `auction_skips_source_kv_already_holds_and_fills_expired`. + +In `publisher.rs`: + +- `page_bids_never_writes_kv`: with a `ts-eids` cookie holding a registered source missing from KV, zero writes. + +- [ ] **Step 2: Run and see them fail** + +Run: `cargo test-fastly -- auction::endpoints::tests::auction_` and `cargo test-fastly -- publisher::tests::page_bids_never_writes_kv` +Expected: failures (today `/auction` and page-bids write). + +- [ ] **Step 3: Implement** + +In finalize's stager branch, when `eid_sync_source == Some(EidSyncSource::Auction)` and `client_eids` came from the body (add a `client_eids_from_body: bool` to `EcContext`, set by `handle_auction` where it chooses body or cookie), call `kv_gap_entries` and save each as `SEntry::staged_browser`. In `publisher.rs`, remove the page-bids path that leads to a KV write. With Task 25 in place, finalize no longer writes for a returning EC, so check that page-bids has no other direct write (`grep -n "upsert\|sync_eid\|create" crates/trusted-server-core/src/publisher.rs`). + +- [ ] **Step 4: Run and see them pass** + +Run: `cargo test-fastly -- auction::` and `cargo test-fastly -- publisher::` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Stage the /auction KV gap and stop /_ts/page-bids KV ingest" +``` + +### Task 27: The removal overflow write + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (new `apply_removals_now`) +- Modify: `crates/trusted-server-core/src/ec/finalize.rs` (`emit_identity_cookies`) +- Test: `ec/kv.rs`, `ec/finalize.rs` + +**Interfaces:** + +- Produces: `KvIdentityGraph::apply_removals_now(&self, ec_id: &str, mutation: &IdentityMutation, context: &ApplyContext<'_>) -> RemovalWriteOutcome` with `RemovalWriteOutcome { Written, AlreadyApplied, Failed }`. It reads the row, applies only the batch's removals, opt-outs, clears and `iv` (through `mutation::apply` on a removals-only mutation), and writes with `IfGenerationMatch`. On a generation mismatch it re-reads and retries, up to 5 attempts in total, as the withdrawal path does. Any other error (a per-key rate limit included) stops at once with `Failed`. + +`emit_identity_cookies` calls `plan_r_batch`. On `RPlan::Overflow` it calls `apply_removals_now` before the response is sent. `Written` and `AlreadyApplied` set no removal cookie. `Failed` sets `split_r_batch`'s cookies. This is the second and last page-load exception to G4 (spec 6.2). + +- [ ] **Step 1: Write the failing tests** + +- `overflow_batch_is_written_before_send_with_no_cookie`: a batch of 12 removals, a live row. Exactly one `IfGenerationMatch` write and no `ts-eids-r` `Set-Cookie`. +- `overflow_write_failure_falls_back_to_split_cookies`: a failing store. Every removal is present across the new cookies, and no other cookie is rewritten. +- `overflow_retries_only_on_generation_mismatch`: a store that fails the CAS twice and then succeeds gives 3 attempts. A store that returns a non-CAS error gives 1 attempt. +- `four_openable_cookies_force_overflow`. + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- overflow` +Expected: FAIL, then PASS after implementing. Model the retry loop on `ec/kv.rs`'s withdrawal tombstone loop (`grep -n "fn write_withdrawal_tombstone\|attempt" crates/trusted-server-core/src/ec/kv.rs`). + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Write overflowing removal batches before send" -m "When a removal batch does not fit one cookie, or four removal cookies are already present, it is applied to KV with the withdrawal path's CAS retries. If that write fails it is split across new cookies, never coarsened." +``` + +### Task 28: The server set in the auction merge + +**Files:** + +- Modify: `crates/trusted-server-core/src/auction/endpoints.rs` (`merge_auction_eids`, about line 505) +- Modify: `crates/trusted-server-core/src/publisher.rs` (navigation and page-bids merge, about lines 5547-5568) +- Modify: `crates/trusted-server-core/src/ec/eids.rs` (`to_eids` adds provenance) +- Test: `auction/endpoints.rs`, `publisher.rs` + +**Interfaces:** + +- Produces: `merge_auction_eids(client: &[Eid], server: &ServerSet, registry: &PartnerRegistry) -> Vec`. `ServerSet` holds, per source, the servable record from the `ts-eids-s` cookie view (staged entries included) overlaid on KV, minus stubs, removed entries (`ts-eids-r`) and expired entries. Build it with `pub(crate) fn server_set(context: &EcContext, serve: &ServeContext<'_>) -> ServerSet`. + +Precedence (spec 7.1): + +| Source | Browser sent it | Browser did not send it | +| ------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | +| Owned | the owner's valid value (`w == owner`, not expired), else the browser's | the owner's valid value, else a browser-seeded value if the source does not expire | +| Unowned | the browser's (today) | the server value (today) | + +Part 1 has no owners, so only the unowned row applies. Part 2 (Task 36) adds owner denials of browser candidates. A server value uses its stored `atype`, else the partner's, plus its stored UID-level `ext`. A value a provider wrote also gets that provider's EID-level `inserter`, `matcher` and `mm` from `descriptor.provenance(source)`. Sources with `bidstream_enabled = false` are left out. The merged set goes through `gate_eids_by_consent` and into `x-ts-eids` exactly as today. + +- [ ] **Step 1: Write the failing tests** + +- `server_set_includes_staged_ts_eids_s_entry_for_unowned_source_browser_did_not_send` +- `browser_value_wins_for_unowned_source` (control: today's rule) +- `removed_entry_in_ts_eids_r_is_not_sent_before_it_reaches_kv` +- `expired_entry_is_not_sent` +- `stored_atype_and_ext_are_emitted` +- `x_ts_eids_matches_what_was_sent` + +Compare whole `Vec` values with every field written out, with no `..Default::default()`. + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- merge` and `cargo test-fastly -- auction::endpoints` +Expected: FAIL, then PASS. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Merge the server identity set into auction EIDs" +``` + +### Task 29: Phase 4: the single flush after send, replacing pull sync + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (`run_edgezero_pull_sync_after_send` and its callers at about lines 482-560 and 618-668) +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` (`EcFinalizeState` gains the writer flag and the response status; `AppState` gains `ProviderCatalog` and `IdentityKeys`) +- Create: `crates/trusted-server-core/src/ec/identity/flush.rs` (`run_after_send`) +- Modify: `crates/trusted-server-core/src/ec/pull_sync.rs` (delete `dispatch_pull_sync`, `build_pull_sync_context` and `PullSyncContext`. Keep `entry_is_pull_complete` and the helpers generic pull uses) +- Modify: `crates/trusted-server-core/src/ec/finalize.rs` (writer phase 3: dry runs and pending markers) +- Test: `flush.rs`, `adapter-fastly/src/main.rs` + +**Interfaces:** + +- Produces: `pub fn run_after_send(input: AfterSendInput<'_>) -> FlushOutcome`. `AfterSendInput` holds `settings`, `catalog`, `keys`, `services`, `graph: &KvIdentityGraph`, `context: &EcContext`, `status: StatusCode` and `rate_limiter`. `FlushOutcome` is `Skipped | NoChange | Written | Dropped(&'static str) | WriteFailed`. + +Phase 3 additions in finalize (writer, returning EC, status 2xx or 304; spec 5.2): + +- For each provider that is eligible on the merged view in phase `AfterSendDryRun`: + - **Without private state:** build its `ResolveInput` with `dry_run = true` and call `request`. `Some` sets a pending marker `ps..p = now` in `staged.s_cookie`. `None` sets `ps.n = now + refresh_after_sec` in the mutation. + - **With private state:** no dry run. Set the marker when it is due on the cookie view. +- Unowned dry runs return `Some` only when the phase 1 read showed the source missing and the pull-complete marker is absent (spec 5.6). Enforce this in the coordinator before the dry run. The provider does not decide it. +- Section 6.3's phase-1 read triggers must hold. The adapter reads KV in phase 1 (the existing preload) whenever any trigger is true. Add `pub(crate) fn phase1_read_needed(context: &EcContext, catalog, settings, auction_needs_row: bool) -> bool` in `ec/identity/request.rs`, and make the publisher preload decision (`publisher.rs`, `auction_needs_row` and the preload at about lines 3110-3117 and 4819-4831) consult it in addition to today's rule. + +Phase 4 (`run_after_send`, spec 5.2, steps 1 to 5): + +1. If the request is not a returning-EC writer, or `status` is not 2xx or 304: `Skipped`. Hold-mode answers already live in the cookie. +2. If the mutation is empty and no marker was set this request: `Skipped`. No read. +3. If phase 1 read the row (`Present` with a generation) and no marker is set, use that snapshot. Otherwise call `graph.get(ec_id)`. A tombstone, `consent.ok = false`, a missing row or a failed read means drop and stop. +4. Recompute eligibility steps 4 to 6 from the live row for every marked provider. Rebuild each request from the live row (the live state for stateful providers). Dispatch with `coordinator::dispatch`. Turn the outcomes into changes and add them to a copy of the mutation. A marked provider not dispatched for a reason the live row does not show (the burst cap, or the breaker in Part 2) gets `n = max(n, now + 60)`. +5. If the live row has no `cluster_size`, compute it with `compute_cluster_size` and `set_cluster_size`. +6. `mutation::apply` against the live entry. If `changed`, write once with `IfGenerationMatch(generation)`. Never retry. Log failure at `warn` with `log_id`, and return `WriteFailed`. + +Remove `run_edgezero_pull_sync_after_send` and `prepare_pull_sync_after_send`. Their callers call `run_after_send`. The adapter captures the response status before `send_edgezero_response` consumes the response (`main.rs` around lines 510-517) and passes it in. Build the KV graph after send exactly as today (`require_identity_graph`). + +- [ ] **Step 1: Write the failing tests** + +In `flush.rs`: + +- `writer_flushes_mutation_once_with_generation_match` (1 `IfGenerationMatch` write) +- `writer_with_nothing_to_do_reads_and_writes_nothing` (0 reads, 0 writes) +- `non_2xx_or_304_status_drops_the_mutation` +- `tombstone_on_reread_drops_mutation_and_calls_no_provider` (the fake HTTP client sees zero calls) +- `marked_generic_pull_provider_dispatches_after_reread_and_its_uid_lands_in_the_same_write` +- `cas_conflict_is_not_retried` (1 attempt, `WriteFailed`) +- `burst_capped_marked_provider_gets_sixty_second_backoff` +- `missing_cluster_size_is_added_to_the_one_write` + +In the adapter (`main.rs` tests): replace `pull_sync_noop_states_skip_post_send_graph_factory` with `after_send_noop_states_skip_post_send_graph_factory`, which checks that no graph is built when there is nothing to flush. + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- flush` and `cargo test-fastly` +Expected: FAIL, then PASS. + +- [ ] **Step 3: Delete pull sync's dispatcher** + +Delete `dispatch_pull_sync`, `build_pull_sync_context`, `PullSyncContext`, `InFlightPull` and `pull_rate_limit_key` from `ec/pull_sync.rs`. Port every remaining pull sync test whose behaviour is unchanged to `generic_pull.rs` or `flush.rs` before deleting it. For each behaviour change in spec 10.4, write a test named after it: + +- `generic_pull_runs_only_on_returning_writer` +- `new_visitor_reaches_pull_partners_from_second_page_load` +- `generic_pull_results_join_the_single_flush` +- `failing_pull_partner_backs_off_through_ps` +- `rate_limit_key_is_per_visitor_not_per_ip` + +- [ ] **Step 4: Run everything** + +Run: `cargo test-fastly && cargo test-fastly-reuse && cargo clippy-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add -A crates/ +git commit --signoff -S -m "Flush identity changes once after send and run pull partners there" -m "Phase 4 replaces the pull sync after-send hook. It reuses the phase 1 snapshot when no provider is marked, otherwise re-reads the row, dispatches marked providers, and applies the whole mutation in one IfGenerationMatch write that is never retried. Pull partners now run only on a returning visitor's writer navigation, and back off through ps." +``` + +### Task 30: Prove the one-write invariant, reads and G10 + +**Files:** + +- Create: `crates/trusted-server-adapter-fastly/tests/identity_writes.rs` (or a `#[cfg(test)] mod identity_writes` in `app.rs` if the adapter has no integration-test harness; check how existing adapter tests drive a full request: `grep -rn "fn .*end_to_end\|TrustedServerApp" crates/trusted-server-adapter-fastly/src | head`) +- Test only + +**Interfaces:** + +- Consumes: `WriteCountingEcKv`, the fake HTTP client, and the adapter's request-driving test helpers + +- [ ] **Step 1: Write the scenario table as tests** + +One test per row of spec 13's "One-write invariant" table. Each drives a full request through the adapter (routing, finalize, phase 4) and asserts the exact write counts by mode: + +| Test | Writes | +| ----------------------------------------------------------- | ------------------------------------------------- | +| `new_visitor_navigation_writes_once` | 1 `Add` | +| `new_visitor_collision_remints_and_writes_once` | 1 `Add` that lands (a precondition failure first) | +| `returning_visitor_with_changes_writes_once` | 1 `IfGenerationMatch` | +| `returning_visitor_without_changes_writes_nothing` | 0 | +| `revalidation_304_with_changes_writes_once` | 1 | +| `returning_navigation_answered_3xx_4xx_5xx_writes_nothing` | 0 each | +| `new_visitor_prefetch_writes_once` | 1 `Add` | +| `returning_prefetch_writes_nothing` | 0 | +| `new_visitor_without_sec_fetch_dest_writes_once` | 1 `Add` | +| `returning_iframe_or_no_sec_fetch_dest_writes_nothing` | 0 | +| `integration_proxy_navigation_minting_ec_writes_once` | 1 `Add` | +| `auction_page_bids_identify_proxy_and_assets_write_nothing` | 0 each | +| `consent_withdrawal_writes_tombstone_and_marker` | today's count, per ID | +| `overflowing_removal_batch_writes_before_send` | 1 `IfGenerationMatch` | +| `cas_conflict_on_flush_is_one_attempt` | 1 failed, no retry | + +Also write: + +- **Reads:** `returning_visitor_with_recent_cookie_and_nothing_due_reads_zero_times_on_navigation`, and `auction_and_page_bids_keep_one_read`. +- **G10:** `body_only_source_reaches_kv_on_next_navigation` (an `/auction` with a body-only registered source, then a navigation: the source is in KV after the navigation's single write), and `gap_too_large_for_cookie_is_saved_again_by_next_auction`. +- **Failed flush:** `refused_flush_keeps_staged_entries_for_next_navigation`. +- **Orphans:** `deleted_row_is_found_within_kv_revalidate_sec_and_rotated` and `stager_that_sees_missing_row_sets_m_and_next_writer_recovers`. +- **Withdrawal:** `withdrawal_on_auction_after_navigation_send_makes_phase_four_drop_and_call_nothing`. + +- [ ] **Step 2: Run them** + +Run: `cargo test-fastly -- identity_writes` +Expected: PASS. A failure here is a real bug in Tasks 23 to 29. Fix it in the task that owns the behaviour, and do not loosen the test. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-adapter-fastly +git commit --signoff -S -m "Test the one-write-per-page-load invariant end to end" +``` + +### Task 31: Metrics, docs and the Part 1 PR + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/finalize.rs` or `ec/identity/request.rs` (metrics) +- Modify: `docs/guide/edge-cookies.md`, `docs/guide/configuration.md`, `docs/guide/ec-setup-guide.md`, `docs/guide/api-reference.md`, `CHANGELOG.md`, `trusted-server.example.toml` + +- [ ] **Step 1: Metrics** + +Find how the repo emits request metrics or counters today (`grep -rn "record_eid_sync_terminal\|metric\|counter" crates/trusted-server-core/src/ec/finalize.rs | head`). Emit the same way, with no IDs and no cookie values: + +- the inbound `Cookie` header size in bytes +- the number and total sealed size of `__Host-ts-eids-r-*` cookies per request +- KV reads and writes per request, labelled by route kind (writer, stager, new EC) +- phase 4 outcome (`FlushOutcome` as a label) + +Add one test that asserts the counts are recorded for a writer navigation. + +- [ ] **Step 2: Docs** + +- `edge-cookies.md`: add sections for `__Host-ts-eids-s` and `__Host-ts-eids-r-*` (purpose, sealed, host-only, `HttpOnly`, one-year lifetime, 2,048 and 512 byte caps, never read by the shim, never sent to the origin or Prebid Server). Also describe the writer-request rule in plain words: only a full page navigation writes KV, once, after the page is sent. +- `configuration.md`: the `[ec.identity]` table, with every key, its default and its meaning, from spec 10.1. Note that `ec.pull_sync_concurrency` is deprecated. +- `ec-setup-guide.md`: pull partners now run from a visitor's second page load and back off on errors. +- `api-reference.md`: `/_ts/api/v1/identify` and `/_ts/page-bids` no longer write KV, and `/identify` omits expired IDs. +- `CHANGELOG.md` `### Changed`: one entry each for fewer KV writes (with the writer-request rule), `/auction`, `/_ts/page-bids` and `/identify` no longer writing, pull partners from the second page load, and the two new cookies. Mark the rollback note: after this release, do not roll back past it once providers write metadata (spec 14). + +Run `cd docs && npm run format` and the root markdown prettier command from `AGENTS.md` gate 8. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/ docs/ CHANGELOG.md trusted-server.example.toml +git commit --signoff -S -m "Document the single identity write and the new identity cookies" +``` + +- [ ] **Step 4: Part 1 gate and PR** + +Run `/check-ci`. Every gate must pass, including `scripts/lint-wasm-clock.sh` (new core code must not call `std::time::{Instant,SystemTime}::now`. Use `checked_current_timestamp()` and `web_time` where you need a clock) and the parity crate. Open the PR against `feat/1246-identity-cookie-strips` with `pr-creator`. Title: "Write identity KV once per page load". The description lists the behaviour changes from spec 14 step 1 and the `test-utils` feature reuse. + +--- + +# Part 2: Providers (spec 14, step 2) + +Part 2 lets integrations register providers, adds ownership, hold mode, passive capture, provider state and sticky opt-outs, the circuit breaker, and the consent decoding that strict policies need. No vendor ships here: ID5, LiveRamp, Lockr and UID2 each get their own spec and plan (spec 2.2). This part proves the framework with the fictional `exampleid` and capture providers from spec 4.6, registered only in tests. + +Before you start: `git switch feat/1246-identity-single-write && git switch -c feat/1246-identity-providers`. + +### Task 32: Registering identity providers + +**Files:** + +- Modify: `crates/trusted-server-core/src/integrations/registry.rs` (`IntegrationRegistration`, the builder around line 641, `with_plan` around line 842) +- Modify: `crates/trusted-server-core/src/ec/identity/catalog.rs` (`from_settings` adds registered providers) +- Test: `integrations/registry.rs` + +**Interfaces:** + +- Produces: + - `IntegrationRegistrationBuilder::with_identity_provider(self, provider: Arc) -> Self` + - `IntegrationRegistrationBuilder::try_with_identity_provider(self, provider: Result>) -> Result>`. A failure becomes `TrustedServerError::Integration { integration: , message: }`. + - `IntegrationRegistry::identity_providers(&self) -> Vec>` + - In `with_plan`: every capture path and prefix must lie under one of the registering integration's `IntegrationProxy::routes()`. Otherwise it is a `Configuration` error naming the integration. + - `ProviderCatalog::from_settings(settings, partner_registry, integration_registry)` adds the registered providers after generic pull, and rejects duplicate IDs. + +- [ ] **Step 1: Write the failing tests** + +```rust + #[test] + fn try_with_identity_provider_converts_a_constructor_error_once() { + let failing: Result> = + Err(IdentityProviderError::config("bad partner id")); + let error = IntegrationRegistration::builder("exampleid") + .try_with_identity_provider(failing) + .expect_err("should fail"); + assert!( + matches!(error.current_context(), TrustedServerError::Integration { integration, .. } if integration == "exampleid"), + "should name the integration" + ); + + let built = IntegrationRegistration::builder("exampleid") + .try_with_identity_provider(Ok(ExampleTestProvider::new())) + .expect("should register") + .build(); + assert_eq!(built.identity_providers.len(), 1, "control: one provider registered"); + } + + #[test] + fn capture_rule_outside_the_integrations_routes_is_rejected_at_startup() { + let settings = settings_with_capture_integration("/integrations/other/api"); + let error = IntegrationRegistry::with_plan(&settings, empty_plan()).expect_err("should reject"); + assert!(format!("{error}").contains("examplecapture"), "should name the integration"); + let ok = settings_with_capture_integration("/integrations/examplecapture/api/token"); + IntegrationRegistry::with_plan(&ok, empty_plan()).expect("control: a rule under its own routes"); + } +``` + +`ExampleTestProvider` and the capture integration are test-only types defined in this module's tests. The capture integration registers an `IntegrationProxy` with routes under `/integrations/examplecapture/` and a capture provider whose path comes from the helper's argument. Follow how existing registry tests register a stub integration (`grep -n "from_request_filters\|fn stub_" crates/trusted-server-core/src/integrations/registry.rs | head`). + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- integrations::registry::tests::try_with_identity_provider` and `cargo test-fastly -- integrations::registry::tests::capture_rule` +Expected: FAIL, then PASS. Add `identity_providers: Vec>` to `IntegrationRegistration` and merge it in `with_plan`, as proxies are merged. A route matches a capture path when the path equals the route's path or, for a wildcard route, starts with its prefix. Mirror how `handle_proxy` matches routes today. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Let integrations register identity providers" +``` + +### Task 33: Ownership and deploy-time validation + +**Files:** + +- Modify: `crates/trusted-server-core/src/settings.rs` (`EcPartner` gains `owner`) +- Modify: `crates/trusted-server-core/src/ec/registry.rs` (`PartnerConfig.owner`) +- Modify: `crates/trusted-server-core/src/config.rs` (`validate_identity_integration`, the identity validation pass in deploy and runtime validation) +- Modify: `crates/trusted-server-core/src/ec/identity/config.rs` (`validate_identity_catalog`) +- Modify: `crates/trusted-server-core/src/ec/identity/catalog.rs` (owners from `[[ec.partners]].owner`, plus `sources` narrowing) +- Modify: `crates/trusted-server-core/src/ec/identity/generic_pull.rs` (an owned pull partner uses `pull_sync_ttl_sec` as `refresh_after_sec`) +- Test: `config.rs`, `ec/identity/config.rs` + +**Interfaces:** + +- Produces: + - `EcPartner.owner: Option`, `#[serde(default, skip_serializing_if = "Option::is_none")]` + - `fn validate_identity_integration(settings: &Settings, id: &str) -> Result, Report>`. It parses `[integrations.]` as `T` and, when enabled, calls `build_descriptor`. It never builds a provider and never reads secrets. + - `pub(crate) fn validate_identity_catalog(descriptors: &[ProviderDescriptor], partners: &[EcPartner], identity: &IdentitySettings) -> Result<(), Report>`, with every error of spec 10.2: + - `owner` names a provider that is not enabled + - `owner` names a provider whose (narrowed) descriptor lacks that source + - an enabled provider claims a source with no `[[ec.partners]]` entry + - two enabled providers claim a source one of them owns (`sources` narrowing applied first) + - `sources` names an undeclared source, or narrows to nothing + - two providers have the same ID + - `[ec.identity.providers.]` names an unknown provider + - more than 32 enabled providers, generic pull included + - The validation runs in `validate_settings_for_deploy` (next to `PartnerRegistry::validate_config_for_deploy`) and in `validate_settings_for_runtime`. The deploy path collects descriptors from a new `identity_descriptors_for_deploy(settings)` list, which has one `validate_identity_integration::` line per identity integration. It is empty in this PR except under `#[cfg(test)]`, where the test `ExampleIdConfig` is added, so the list's shape is exercised. + +- [ ] **Step 1: Write the failing tests** + +One test per error above, each paired with a control config that passes. Also: + +- `config_without_ec_identity_is_accepted` +- `build_descriptor_runs_in_deploy_validation_without_secrets`: a config whose secret field holds a key name, not a value, still validates + +For example: + +```rust + #[test] + fn owner_must_name_an_enabled_provider_that_declares_the_source() { + let descriptors = vec![descriptor("exampleid", &["exampleid.example.com"])]; + let ok = vec![partner("exampleid.example.com", Some("exampleid"))]; + validate_identity_catalog(&descriptors, &ok, &IdentitySettings::default()).expect("control"); + + let unknown = vec![partner("exampleid.example.com", Some("missing"))]; + let error = validate_identity_catalog(&descriptors, &unknown, &IdentitySettings::default()).expect_err("unknown owner"); + assert!(format!("{error}").contains("missing"), "should name the owner"); + + let wrong_source = vec![partner("exampleid.example.com", None), partner("other.example.com", Some("exampleid"))]; + validate_identity_catalog(&descriptors, &wrong_source, &IdentitySettings::default()).expect_err("owner lacks source"); + } +``` + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- ec::identity::config` and `cargo test-fastly -- config::` +Expected: FAIL, then PASS. + +- [ ] **Step 3: CLI check** + +Run: `./scripts/test-cli.sh` +Expected: PASS. `ts config validate` and `ts config push` call the deploy validation, so the CLI tests exercise the new pass. If a CLI fixture config now fails, it is a real config error by the new rules. Fix the fixture and say so in the commit. + +- [ ] **Step 4: Commit** + +```bash +git add -A crates/ +git commit --signoff -S -m "Add source ownership and validate identity provider config at deploy" +``` + +### Task 34: Consent decoding for strict policies, and the consent fingerprint + +**Files:** + +- Modify: `crates/trusted-server-core/src/consent/gpp.rs` (around `decode_us_sale_opt_out` at line 125, and the section-6 skip at line 112) +- Modify: `crates/trusted-server-core/src/consent/types.rs` (decoded fields, and a redacting `Debug` for `ConsentContext`) +- Modify: `crates/trusted-server-core/src/consent/mod.rs` (`stricter_of`) +- Modify: `crates/trusted-server-core/src/ec/identity/policy.rs` (new blocks, and the fingerprint bits) +- Test: `consent/gpp.rs`, `consent/mod.rs`, `policy.rs` + +**Interfaces:** + +- Produces: + - Decoded on the GPP form: `us_sharing_opt_out: Option`, `us_targeted_advertising_opt_out: Option`, `us_gpc: Option` and `us_known_child: Option` (US National section), and the section 6 US Privacy string read through the existing US Privacy decoder. Use the `iab_gpp` fields the spec names: `sharing_opt_out` (US National, California) and `targeted_advertising_opt_out` (US National and the state sections that define it). Check the `iab_gpp` version in `Cargo.lock` and read its section structs before writing the extraction. + - `policy_allows` additionally blocks a strict policy on the sharing opt-out, the targeted-advertising opt-out, GPP's GPC field, and a known child without consent (a known child blocks every provider, whatever the relaxations). + - `pub(crate) fn fingerprint_bits(policy: &ConsentPolicy, consent: &ConsentContext, signals: PolicySignals) -> String`: a canonical string of the TCF purpose bits and vendor bit the policy reads, the US opt-outs it does not relax, and GPC and DNT unless relaxed. Never the raw consent string. + - `pub(crate) fn stricter_of(cookie: &ConsentContext, body: &ConsentContext) -> ConsentContext`: field by field, any opt-out, missing purpose or missing vendor consent in either wins + - A hand-written `impl fmt::Debug for ConsentContext` that prints `[redacted]` for every raw string + +- [ ] **Step 1: Write the failing tests** + +- In `gpp.rs`: one decode test per new field, per section that defines it, using GPP strings built the way existing tests build them. Each asserts the decoded value, next to a control string where the field is not set. +- In `policy.rs`: `strict_blocks_sharing_targeted_gpc_field_and_known_child`, and `targeted_advertising_opt_out_without_sale_opt_out_blocks_strict`. +- In `consent/mod.rs`: `stricter_of_takes_any_opt_out_or_missing_consent_from_either_side`. +- `consent_context_debug_redacts_raw_strings`: `format!("{context:?}")` does not contain the TC string. +- `fingerprint_is_stable_across_cmp_resaves_and_changes_with_a_purpose`: two TC strings with identical decisions but different `LastUpdated` give the same bits, and dropping purpose 4 changes them. + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- consent::` and `cargo test-fastly -- ec::identity::policy` +Expected: FAIL, then PASS. Do not change any existing auction consent decision. Run `cargo test-fastly -- auction::` and `cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity` to confirm. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src/consent crates/trusted-server-core/src/ec/identity +git commit --signoff -S -m "Decode the GPP opt-outs strict identity policies need" -m "Adds the US sharing and targeted-advertising opt-outs, GPP section 6, and the US National GPC and known-child fields. ConsentContext's Debug output no longer prints raw consent strings." +``` + +### Task 35: Rate-limiter `record` and `count`, and the circuit breaker + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/rate_limiter.rs` (trait) +- Modify: `crates/trusted-server-adapter-fastly/src/rate_limiter.rs` (`FastlyRateLimiter`) +- Modify: `crates/trusted-server-core/src/ec/identity/coordinator.rs` (breaker in dispatch, step 7) +- Test: all three + +**Interfaces:** + +- Produces: + - `RateLimiter::record(&self, key: &str) -> Result<(), Report>` and `RateLimiter::count(&self, key: &str) -> Result>`. Both are trait methods without defaults, so every implementation must decide. Fastly uses `counter.increment(key, 1)` and `counter.lookup_count(key, CounterDuration::SixtySecs)`. + - Breaker: before dispatch, `count("identity-fail:") >= breaker_failures_per_minute` gives `CallOutcome::BreakerOpen` (not dispatched, `ps` unchanged). After dispatch, only `TransportFailure` outcomes (transport errors, timeouts, 429 and 5xx) call `record("identity-fail:")`. `HoldCutOff` is a budget cut, not a provider failure, and never records (spec 4.7). + - A limiter error is logged, and the breaker is treated as closed. Skip the burst cap check on that request too, failing closed for disclosure: no dispatch. + +- [ ] **Step 1: Write the failing tests** + +In core, with a `FakeRateLimiter` in `test_support.rs` that stores counts in a `Mutex`: + +- `breaker_opens_at_threshold_and_skips_every_visitor` +- `four_xx_and_malformed_never_open_the_breaker`: 1,000 visitors with 400 responses leave `count` at 0 +- `hold_cut_off_does_not_record_a_failure` +- `burst_cap_is_checked_after_every_other_eligibility_step`: an ineligible provider never increments the burst counter + +In the adapter: a unit test for `record` and `count` against Viceroy's rate counter, following the existing `mod tests` in `rate_limiter.rs`. + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- rate_limiter` and `cargo test-fastly -- ec::identity::coordinator` +Expected: FAIL, then PASS. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/ +git commit --signoff -S -m "Add a per-provider circuit breaker on transport failures" +``` + +### Task 36: Provider state, sticky opt-outs, re-grants and owner denials + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/identity/coordinator.rs` (opt-out clearing on re-grant) +- Modify: `crates/trusted-server-core/src/auction/endpoints.rs` and `publisher.rs` (owner denials filter browser candidates in the merge) +- Modify: `crates/trusted-server-core/src/ec/admin.rs` (an operator clear of `ps.o`) +- Test: `coordinator.rs`, `auction/endpoints.rs`, `ec/admin.rs` + +**Interfaces:** + +- Produces: + - `pub(crate) fn regrant_clears_opt_out(descriptor: &ProviderDescriptor, opt_out: &KvOptOut, consent: &ConsentContext, keys: &IdentityKeys, signals: PolicySignals) -> bool` (spec 8): + - **GVL vendor:** a TC string whose `LastUpdated` is after `o.at` and that grants the vendor and purposes. + - **Any provider, under GPP:** the applicable US section shows none of the opt-outs the policy reads, where the stored `fp` showed one. Compare the current `fingerprint_bits`-derived fingerprint's opt-out part. + - A jurisdiction change, a newly present GPP string, or a relaxed signal never clears. + - When it clears, the writer adds `opt_out_clear = Some(now)` to that provider's `ProviderStateChange`. Clears are writer-only, so they ride the one flush. + - Merge (spec 7.1, "Owner denials apply to browser candidates"): before the precedence table and before `gate_eids_by_consent`, drop the browser's candidate for an owned source when its UID's keyed hash is in the owner's `iv` (KV `ps.iv` ∪ `ts-eids-r` `iv`), or when the owner has a sticky opt-out (KV ∪ `ts-eids-r`). Unowned sources are unaffected. + - The owned rows of the precedence table become active: the owner's valid value wins over the browser's. When the browser did not send the source, a browser-seeded value is used only if the source does not expire. + - Admin: `DELETE /_ts/admin/ec/{id}/providers/{provider}/opt-out`, which clears `ps.o` by writing `oc = now` through the existing admin write path. Add it to the admin route table next to the EC lookup, behind the same Basic-auth coverage, and add the route to the admin coverage test. + +- [ ] **Step 1: Write the failing tests** + +- `tcf_regrant_after_opt_out_clears_it` and `jurisdiction_change_never_clears_it` +- `gpp_section_without_opt_out_clears_when_fingerprint_had_one` +- `invalidated_uid_in_auction_body_is_not_sent_before_or_after_removal_reaches_kv` +- `owner_opt_out_drops_browser_value_for_every_owned_source` +- `owner_valid_value_beats_browser_value` +- `browser_seeded_value_never_beats_browser_current_value` +- `expiring_owned_source_never_serves_browser_seeded_value` +- `unowned_source_unaffected_by_owner_denials` (control) +- `admin_clears_provider_opt_out` + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- ec::identity` and `cargo test-fastly -- auction::endpoints` and `cargo test-fastly -- ec::admin` +Expected: FAIL, then PASS. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Apply owner precedence, owner denials and opt-out re-grants" +``` + +### Task 37: Hold mode + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (before the navigation auction dispatch, about lines 4886-4889) +- Modify: `crates/trusted-server-core/src/auction/endpoints.rs` (`handle_auction`, before the merge) +- Modify: `crates/trusted-server-core/src/publisher.rs` (the `/_ts/page-bids` auction, about lines 7077-7090) +- Modify: `crates/trusted-server-core/src/ec/identity/coordinator.rs` (`run_hold`) +- Test: `coordinator.rs`, `auction/endpoints.rs`, `publisher.rs` + +**Interfaces:** + +- Produces: `pub(crate) fn run_hold(deps: &IdentityDeps<'_>, context: &mut EcContext) -> HoldResults`. It runs eligible `Hold`-phase providers within `hold_budget_ms` from the first dispatch, after recomputing eligibility from the KV row this request read (pre-disclosure revalidation, spec 4.7). It returns the answers as server-set additions for this auction and stages them: + - A stateless provider's IDs go into `staged.s_cookie` as `k: 0` with `pv`, its `ps` changes are marked `sg: 1`, and its removals and opt-outs go into `staged.new_r_batch`. On the writer they also join the mutation. + - A stateful provider (writer only) puts its whole `ProviderResult` in the mutation only. Never in the cookie. + - A cut-off call sets `n = now + hold_retry_sec` and counts no failure. +- `handle_auction` and the navigation take `IdentityDeps` inside the existing deps or context struct so neither passes 7 arguments. +- `/_ts/page-bids` runs hold mode only when its auction runs, because only then does it read KV. + +- [ ] **Step 1: Write the failing tests** + +With the fake HTTP client's latency control: + +- `auction_starts_at_the_hold_budget_when_a_provider_is_slow` (the elapsed fake time equals the budget, and the auction ran) +- `hold_answer_feeds_this_auction_and_is_saved_with_k0` +- `cut_off_call_sets_hold_retry_without_counting_a_failure` +- `stateful_hold_provider_never_runs_on_a_stager` +- `stateful_hold_answer_stays_out_of_the_cookie` +- `writer_navigation_with_non_2xx_status_keeps_hold_answers_in_its_cookie` +- `cookie_only_no_match_stops_a_recall_on_the_next_auction` (the merged view) +- `kv_opt_out_blocks_dispatch_even_when_the_cookie_lacks_it` + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- hold` +Expected: FAIL, then PASS. Use the platform's time source the auction orchestrator uses for its own timeouts. Do not use `std::time::Instant::now` (`scripts/lint-wasm-clock.sh` rejects it in shared crates). + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Run hold-mode identity providers before the auction" +``` + +### Task 38: Passive capture + +**Files:** + +- Modify: `crates/trusted-server-core/src/integrations/registry.rs` (`handle_proxy`, about lines 1031-1061) +- Modify: `crates/trusted-server-core/src/ec/identity/coordinator.rs` (`run_capture`) +- Test: `integrations/registry.rs`, `coordinator.rs` + +**Interfaces:** + +- Produces: `pub(crate) fn run_capture(deps: &IdentityDeps<'_>, context: &mut EcContext, method: &Method, path: &str, request_consent: Option, response_status: StatusCode, response_headers: &HeaderMap, response_body: &[u8])` + +`handle_proxy` per request, only when some provider's `Captures::matches(method, path)`: + +1. **Before `handle`:** if a matching descriptor declares `with_request_consent` and the request has a body, buffer it with `collect_body_bounded` (`integrations/mod.rs:182`, 256 KiB). Extract only the declared JSON pointers, decode them into a `ConsentContext` with the same decoders the cookie path uses, and re-wrap the bytes with `EdgeBody::from(bytes)` before calling `handle`. A body over the cap or not valid JSON gives `request_consent = None` and allows no positive IDs. +2. **After `handle`,** on 2xx only: buffer the response up to `max_response_bytes`. A larger body is passed through uncaptured. Decode a copy with `decode_body`, call `capture` per matching provider, and re-wrap the original bytes so the browser receives the response unchanged. +3. **Apply:** + - Removals and opt-outs always apply, whatever the consent. + - Positive IDs apply only when eligibility steps 1 to 3 pass under `stricter_of(cookie, body)` and the provider has no sticky opt-out, unless this result clears it. A capture-only provider without a GVL ID that returns IDs under consenting cookie and body consent stages `ps.oc = now` beside them. + - Capture never touches `n`, `f` or the breaker. + - A state change in a capture result is ignored with a warning. + - Results are staged on `EcContext` and written by finalize: positive into `ts-eids-s`, removals into `ts-eids-r`. On a writer request they also join the mutation. + +- [ ] **Step 1: Write the failing tests** + +Using a test capture integration (from Task 32) and its provider: + +- `capture_runs_only_for_matching_rules` +- `gzip_and_brotli_responses_are_decoded_for_capture_and_forwarded_unchanged` (byte-equal body to the browser) +- `only_declared_consent_pointers_are_read_from_the_request_body` (a body with an email field. The provider sees no body, and a log capture shows no email) +- `stricter_body_consent_blocks_positive_ids_but_not_removals` +- `revoke_style_opt_out_removes_every_record_the_provider_wrote_including_stubbed_sources` +- `capture_removal_survives_a_concurrent_auction_rewriting_ts_eids_s` (two responses built from the same inbound cookies: the `ts-eids-r` cookie from the capture is still present after the `/auction` response's `ts-eids-s` is applied) +- `revoke_then_reconsent_capture_ends_with_new_ids_and_oc_in_kv` (across a navigation, and again after a failed flush) +- `capture_never_touches_n_f_or_the_breaker` +- `capture_state_change_is_ignored` +- `oversized_proxied_body_passes_through_uncaptured` + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- capture` +Expected: FAIL, then PASS. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Capture identity IDs from proxied provider responses" +``` + +### Task 39: The full provider harness and conformance checks + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/identity/test_support.rs` +- Create: `crates/trusted-server-core/src/ec/identity/examples.rs` (`#[cfg(test)]` only): the spec 4.6 `ExampleIdProvider` and the capture example, as reference providers +- Test: `examples.rs` + +**Interfaces:** + +- Produces everything in spec 13's harness listing: `ProviderHarness::{with_setup, with_client_ip, with_user_agent, with_page_url, with_region, with_hashed_email, with_consent, with_stored, with_state, with_now, with_random_seed, as_dry_run, request, request_json, respond, respond_raw, capture, resolve, state_of, check_conformance, check_conformance_with_state}`, `HarnessSetup`, `TestConsent`, `CaptureFixture`, `ConformanceCases`, `StoredId::{for_test, written_by, expiring_at}`, `HashedEmail::for_test`, and `check_descriptor_ignores_secrets`. All of them sit behind `#[cfg(any(test, feature = "test-utils"))]` and are reachable from outside the crate as `trusted_server_core::ec::identity::test_support::*`, so the module is `pub` under the gate. + +`check_conformance` asserts every bullet of spec 13's conformance list: + +- inputs hold only what the descriptor declares +- a resolver returns `Some` for at least one listed setup, and its `response` never returns the `not_implemented` marker +- a capturer returns a non-`no_match` result for at least one fixture, and never the marker +- no undeclared input or state read is recorded +- no state is returned without `with_private_state()` +- requests go only to declared hosts, over HTTPS, with no `Cookie` header and with valid headers +- IDs are only for declared sources, at most one per source +- `expiry()` and `refresh()` report what was set, and a fixture ID of an expiring source carries an expiry +- `capture` never returns a state change +- a bare-JSON-string stored state is never treated as valid +- the dry run is discarded unsent, and `as_dry_run()` fails for a stateful provider + +`check_conformance` fails with a pointer to `check_conformance_with_state` for a stateful provider. `check_conformance_with_state::` also round-trips `S` through `replacing_state` and `input.state()`. + +- [ ] **Step 1: Write the tests (the spec's examples are the first harness cases)** + +In `examples.rs`, copy `ExampleIdConfig`, `ExampleIdProvider` and the capture-only example verbatim from spec 4.6, then: + +```rust + #[test] + fn example_provider_round_trip() { + let provider = ExampleIdProvider::new(&example_config()).expect("should build the provider"); + let mut harness = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0"); + + let request = harness.request().expect("should build a request").expect("should not skip the request"); + assert_eq!(request.url().host_str(), Some("api.exampleid.example.com"), "should call the declared host"); + + let resolution = harness + .respond(StatusCode::OK, &json!({ "uid": "example-uid", "signature": "example-signature" })) + .expect("should parse the response"); + assert_eq!(resolution.ids()[0].uid(), "example-uid", "should return the uid"); + let state: ExampleIdState = harness.state_of(&resolution).expect("should replace state"); + assert_eq!(state.signature, "example-signature", "should keep the signature"); + } + + #[test] + fn example_provider_conforms() { + let provider = ExampleIdProvider::new(&example_config()).expect("should build"); + ProviderHarness::new(&provider).check_conformance_with_state::( + &ConformanceCases::default().with_resolve_case( + HarnessSetup::default().with_client_ip("192.0.2.10").with_user_agent("ExampleBrowser/1.0"), + StatusCode::OK, + &json!({ "uid": "example-uid", "signature": "s" }), + ), + ); + } + + #[test] + fn conformance_catches_an_undeclared_read() { + let provider = UndeclaredUserAgentReader::new(); + let result = std::panic::catch_unwind(|| { + ProviderHarness::new(&provider).check_conformance( + &ConformanceCases::default().with_resolve_case(HarnessSetup::default(), StatusCode::OK, &json!({})), + ); + }); + assert!(result.is_err(), "should fail a provider that reads an undeclared input through .ok()"); + } + + #[test] + fn descriptor_ignores_secrets() { + check_descriptor_ignores_secrets(&example_config(), &example_config()); + } +``` + +Add negative conformance tests the same way for: a `Cookie` header, a request to an undeclared host, two IDs for one source, a capability declared but not overridden (the `not_implemented` marker), a stateless provider returning state, a capture returning a state change, and an expiring source without an expiry. Each uses a tiny test provider written in the test module, and each asserts the harness panics. + +- [ ] **Step 2: Run, see them fail, implement, see them pass** + +Run: `cargo test-fastly -- ec::identity::examples` +Expected: FAIL, then PASS. + +- [ ] **Step 3: Check the feature gate from outside the crate** + +Run: `cargo check -p trusted-server-adapter-fastly --target wasm32-wasip1 --tests --features trusted-server-core/test-utils` (or whichever adapter already enables `test-utils` in its dev-dependencies) +Expected: builds. This proves a provider crate can use the harness. + +- [ ] **Step 4: Commit** + +```bash +git add -A crates/trusted-server-core/src +git commit --signoff -S -m "Add the provider harness and conformance checks" +``` + +### Task 40: Batch sync never replaces an owned value + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/batch_sync.rs` (around line 247) +- Test: `ec/batch_sync.rs` + +- [ ] **Step 1: Write the failing test** + +`batch_sync_update_to_owned_source_applies_only_when_absent`: with the source owned by a provider, a batch update applies when the source is absent, and is `Unchanged` (no write) when a value exists. A control update to an unowned source keeps today's behaviour. + +- [ ] **Step 2: Run, see it fail, implement, see it pass** + +Run: `cargo test-fastly -- ec::batch_sync` +Expected: FAIL, then PASS. The batch sync handler needs to know the owners. Pass `&ProviderCatalog` (or just the owned source set) through its existing deps. + +- [ ] **Step 3: Commit** + +```bash +git add -A crates/trusted-server-core/src/ec/batch_sync.rs +git commit --signoff -S -m "Keep batch sync from replacing an owned source's value" +``` + +### Task 41: Docs, the provider author guide, and final verification + +**Files:** + +- Create: `docs/guide/identity-providers.md` (provider author guide) +- Modify: `docs/guide/configuration.md` (`[[ec.partners]].owner`, per-provider tables), `docs/guide/edge-cookies.md`, `docs/.vitepress` sidebar config (find it with `grep -rn "edge-cookies" docs/.vitepress`), `CHANGELOG.md` + +- [ ] **Step 1: Write the author guide** + +Base it on spec 4.2 to 4.6 and 13, using only the fictional `exampleid` provider: + +- what a provider is +- the descriptor +- declaring inputs and reading them with `?` or `.ok()` +- building requests +- parsing responses +- returning a `Resolution` +- private state +- capture +- registration with `try_with_identity_provider` +- testing with `ProviderHarness` and `check_conformance` +- what the coordinator does for you (consent, transport, backoff, KV) + +Link to the spec for the protocol. + +- [ ] **Step 2: Config and changelog** + +Document `owner`, `mode`, `timeout_ms`, `sources` narrowing, `refresh_after_sec`, `max_calls_per_minute`, `breaker_failures_per_minute` and `cookie_priority`, each with its default and a validation error example. In `CHANGELOG.md` under `### Added`, add the provider framework, ownership, hold mode, passive capture and the breaker. Under `### Changed`, add stricter consent decoding for identity providers. + +Run the docs and markdown format commands. + +- [ ] **Step 3: Commit** + +```bash +git add docs/ CHANGELOG.md +git commit --signoff -S -m "Document the identity provider framework for operators and authors" +``` + +- [ ] **Step 4: Full verification** + +Run `/check-ci`, then `cargo test-fastly-reuse`, `./scripts/test-cli.sh`, the parity crate, `scripts/lint-wasm-clock.sh`, and `cargo doc --no-deps --all-features -p trusted-server-core`. Every one must pass. Then use the `verify-app` agent for an independent pass. + +- [ ] **Step 5: PR** + +Open the PR against `feat/1246-identity-single-write` with `pr-creator`. Title: "Add server-side identity providers". The description lists spec 14 step 2's contents, says no vendor ships in it, and names the follow-up vendor specs (ID5, LiveRamp, Lockr, UID2). + +--- + +## Spec coverage + +| Spec section | Tasks | +| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| 2.1 G1, G2, G11 (trait, coordinator, ergonomics) | 8 to 11, 20, 21, 39 | +| 2.1 G3 (hold or after-send) | 19, 20, 29, 37 | +| 2.1 G4 (one write) | 25, 26, 27, 29, 30 | +| 2.1 G5 (cookie first, then KV) | 16, 17, 25, 28 | +| 2.1 G6 (ownership) | 33, 36 | +| 2.1 G7 (expiry, refresh) | 14, 15, 20, 36 | +| 2.1 G8 (pull sync on the trait) | 22, 29 | +| 2.1 G9 (HEM slot) | 9 (always empty in v1) | +| 2.1 G10 (no lost EIDs) | 25, 26, 30 | +| 2.1 G12 (removals win) | 15, 18, 25, 27, 38 | +| 3.4 platform facts (status capture, real-browser gate, clock failure) | 14, 23, 29 | +| 4.2 trait and additivity rules | 11, 39 | +| 4.3 descriptor | 8, 32 (capture-path check) | +| 4.4 inputs | 9, 23 (phase-1 captures) | +| 4.5 requests, responses, results | 10, 20 | +| 4.6 example provider and registration | 32, 39 | +| 4.7 eligibility, dispatch, hold timing, breaker, capture | 20, 21, 35, 37, 38 | +| 5.1 writer and stagers; withdrawal exception | 23, 25 | +| 5.2 phases 1 to 4 | 23, 25, 29 | +| 5.3 stagers, KV gap, flush checks | 25, 26 | +| 5.4 `IdentityMutation` | 15 | +| 5.5 orphans, collisions, deletion | 25, 30 (the deletion framework itself is deferred; its tombstone rule is a note for that work) | +| 5.6 removal, marker and cookie rules | 15, 17, 18, 25, 29 | +| 6.1 KV record, rollback rule, admin redaction | 6, 14 | +| 6.2 cookies, sealing, size caps, overflow, outbound strips | 1 to 4, 16, 17, 18, 27 | +| 6.3 read order | 17, 29 | +| 7.1 merge, servability, `/identify` consent | 14, 28, 36 | +| 7.2 ownership semantics, batch sync | 33, 36, 40 | +| 8 errors and outcomes, sticky opt-outs, fingerprint | 13, 20, 34, 36 | +| 9.1 consent | 12, 34 | +| 9.2 outbound requests | 1 to 3, 21 | +| 9.3 logging and redaction | 9, 10, 34, every task's logging | +| 9.4 caching | 25 (cookies only when needed; the privacy rule already applies) | +| 10.1 to 10.3 configuration, validation, registration | 19, 32, 33 | +| 10.4 pull sync on the trait | 22, 29 | +| 11 adapters | 3 (strips on every adapter), 31 | +| 12 vendor examples | Out of scope (spec 2.2). The fictional examples in 39 prove the API | +| 13 testing | each task, 30, 39 | +| 14 rollout | Parts 0, 1, 2 | +| 15 open questions | Not implemented. Track as issues after Part 1 ships | From 91b45d45003818f4c2d7dbb8056c298626898ce4 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:27:33 +0530 Subject: [PATCH 02/33] Add identity cookie lists and a name-and-prefix cookie strip Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/constants.rs | 4 + crates/trusted-server-core/src/cookies.rs | 146 +++++++++++++++++++- 2 files changed, 149 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index 1fd3c2382c..882c0311ff 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -6,6 +6,10 @@ pub const COOKIE_TS_EC_PULL_COMPLETE: &str = "ts-ec-pull-complete"; /// Cookie written by the Trusted Server JS SDK containing a standard-base64-encoded /// JSON array of Extended User IDs (`[{ source, uids }]`) from identity providers. pub const COOKIE_TS_EIDS: &str = "ts-eids"; +/// Server-owned, sealed cache of provider-collected EIDs. +pub const COOKIE_TS_EIDS_S: &str = "__Host-ts-eids-s"; +/// Name prefix of the per-batch removal cookies, `__Host-ts-eids-r-<8 hex>`. +pub const COOKIE_TS_EIDS_R_PREFIX: &str = "__Host-ts-eids-r-"; pub const COOKIE_TS_TESTER: &str = "ts-tester"; pub const COOKIE_SHAREDID: &str = "sharedId"; diff --git a/crates/trusted-server-core/src/cookies.rs b/crates/trusted-server-core/src/cookies.rs index 3fe6719ef7..2eef51314f 100644 --- a/crates/trusted-server-core/src/cookies.rs +++ b/crates/trusted-server-core/src/cookies.rs @@ -11,7 +11,10 @@ use error_stack::{Report, ResultExt as _}; use http::Request; use http::header; -use crate::constants::{COOKIE_EUCONSENT_V2, COOKIE_GPP, COOKIE_GPP_SID, COOKIE_US_PRIVACY}; +use crate::constants::{ + COOKIE_EUCONSENT_V2, COOKIE_GPP, COOKIE_GPP_SID, COOKIE_TS_EC, COOKIE_TS_EC_PULL_COMPLETE, + COOKIE_TS_EIDS, COOKIE_TS_EIDS_R_PREFIX, COOKIE_TS_EIDS_S, COOKIE_US_PRIVACY, +}; use crate::error::TrustedServerError; /// Cookie names carrying privacy consent signals. @@ -26,6 +29,29 @@ pub const CONSENT_COOKIE_NAMES: &[&str] = &[ COOKIE_US_PRIVACY, ]; +/// Trusted Server identity cookies that never reach a third party. +/// +/// Stripped from Prebid Server requests and never added to proxy or provider +/// requests. The `__Host-ts-eids-r-` family is matched by +/// [`TS_IDENTITY_COOKIE_PREFIXES`]. +pub const TS_IDENTITY_COOKIE_NAMES: &[&str] = &[ + COOKIE_TS_EC, + COOKIE_TS_EIDS, + COOKIE_TS_EIDS_S, + COOKIE_TS_EC_PULL_COMPLETE, +]; + +/// Identity cookies removed from publisher-origin requests. +/// +/// `ts-ec` and `ts-ec-pull-complete` still reach the origin. The ID cookies +/// are removed so they cannot push the origin's `Cookie` header past common +/// 8 KB header limits. +pub const TS_ORIGIN_STRIPPED_COOKIE_NAMES: &[&str] = &[COOKIE_TS_EIDS, COOKIE_TS_EIDS_S]; + +/// Name prefixes of identity cookie families, stripped wherever the named +/// identity cookies are. +pub const TS_IDENTITY_COOKIE_PREFIXES: &[&str] = &[COOKIE_TS_EIDS_R_PREFIX]; + /// Parses a cookie string into a [`CookieJar`]. /// /// Returns an empty jar if the cookie string is unparseable. @@ -104,6 +130,52 @@ pub fn strip_cookies(cookie_header: &str, cookie_names: &[&str]) -> String { .join("; ") } +/// Strips cookies matching any exact name or name prefix from a `Cookie` +/// header value. +/// +/// Returns an empty string when nothing is left. +#[must_use] +pub fn strip_cookie_pairs(cookie_header: &str, names: &[&str], prefixes: &[&str]) -> String { + cookie_header + .split(';') + .map(str::trim) + .filter(|pair| !pair.is_empty()) + .filter(|pair| { + let name = pair.split('=').next().unwrap_or_default().trim(); + !names.contains(&name) && !prefixes.iter().any(|prefix| name.starts_with(prefix)) + }) + .collect::>() + .join("; ") +} + +/// Strips matching cookies from every `Cookie` field in `headers`. +/// +/// All `Cookie` fields are read, matching pairs are removed, and the rest is +/// written back as one field. A field that is not valid UTF-8 is dropped +/// rather than forwarded, because it cannot be checked. The header is removed +/// when nothing is left. +pub fn strip_cookie_headers(headers: &mut http::HeaderMap, names: &[&str], prefixes: &[&str]) { + let retained = headers + .get_all(header::COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .map(|value| strip_cookie_pairs(value, names, prefixes)) + .filter(|value| !value.is_empty()) + .collect::>() + .join("; "); + + headers.remove(header::COOKIE); + if retained.is_empty() { + return; + } + match http::HeaderValue::from_str(&retained) { + Ok(value) => { + headers.insert(header::COOKIE, value); + } + Err(_) => log::debug!("Dropping Cookie header that is invalid after stripping"), + } +} + /// Copies the `Cookie` header from one request to another, optionally /// stripping consent cookies. /// @@ -150,6 +222,78 @@ mod tests { use super::*; + #[test] + fn strip_cookie_pairs_removes_names_and_prefixes() { + let header = "ts-ec=abc; ts-eids=W10; __Host-ts-eids-s=v1.x.y; \ + __Host-ts-eids-r-0a1b2c3d=v1.p.q; other=1; ts-eids-extra=keep"; + + let stripped = strip_cookie_pairs( + header, + TS_ORIGIN_STRIPPED_COOKIE_NAMES, + TS_IDENTITY_COOKIE_PREFIXES, + ); + + assert_eq!( + stripped, "ts-ec=abc; other=1; ts-eids-extra=keep", + "should drop the ID cookies and keep ts-ec and unrelated cookies" + ); + } + + #[test] + fn strip_cookie_headers_joins_every_cookie_field_and_drops_non_utf8() { + let mut headers = http::HeaderMap::new(); + headers.append( + header::COOKIE, + HeaderValue::from_static("ts-ec=abc; ts-eids=W10"), + ); + headers.append( + header::COOKIE, + HeaderValue::from_bytes(b"bad=\xff").expect("should build an opaque header"), + ); + headers.append(header::COOKIE, HeaderValue::from_static("other=1")); + + strip_cookie_headers( + &mut headers, + TS_IDENTITY_COOKIE_NAMES, + TS_IDENTITY_COOKIE_PREFIXES, + ); + + let values: Vec<_> = headers.get_all(header::COOKIE).iter().collect(); + assert_eq!(values.len(), 1, "should collapse into one Cookie header"); + assert_eq!( + values[0], "other=1", + "should keep only the unrelated cookie" + ); + } + + #[test] + fn strip_cookie_headers_removes_header_when_nothing_is_left() { + let mut headers = http::HeaderMap::new(); + headers.insert( + header::COOKIE, + HeaderValue::from_static("ts-ec=abc; __Host-ts-eids-r-00000000=v1.a.b"), + ); + headers.insert( + header::USER_AGENT, + HeaderValue::from_static("ExampleBrowser/1.0"), + ); + + strip_cookie_headers( + &mut headers, + TS_IDENTITY_COOKIE_NAMES, + TS_IDENTITY_COOKIE_PREFIXES, + ); + + assert!( + headers.get(header::COOKIE).is_none(), + "should remove the empty Cookie header" + ); + assert!( + headers.get(header::USER_AGENT).is_some(), + "should leave other headers alone" + ); + } + fn build_request(cookie_header: Option<&str>) -> Request { let mut builder = Request::builder().method("GET").uri("http://example.com"); if let Some(cookie_header) = cookie_header { From 1028e042078e15ab0df380d402a92c30c423275c Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:31:04 +0530 Subject: [PATCH 03/33] Strip identity cookies from Prebid Server requests in every mode Every Cookie field is now read and checked, and a field that is not valid UTF-8 is dropped instead of forwarded. Two transport tests that pinned raw forwarding of malformed cookie bytes now expect the field to be dropped. Signed-off-by: dhruv8sh --- .../src/auction/routing.rs | 23 ++- .../src/integrations/prebid.rs | 156 ++++++++++++++---- 2 files changed, 146 insertions(+), 33 deletions(-) diff --git a/crates/trusted-server-core/src/auction/routing.rs b/crates/trusted-server-core/src/auction/routing.rs index 2b8135da3c..dc9dffd012 100644 --- a/crates/trusted-server-core/src/auction/routing.rs +++ b/crates/trusted-server-core/src/auction/routing.rs @@ -203,15 +203,16 @@ impl ProviderSlotInput { /// the same legacy handling. Client-supplied `X-Forwarded-For` is never read. #[derive(Debug, Clone, Default)] pub(crate) struct PrebidTransportHeaders { - cookie: Option, + cookies: Vec, user_agent: Option, referer: Option, accept_language: Option, } impl PrebidTransportHeaders { - pub(crate) fn cookie(&self) -> Option<&HeaderValue> { - self.cookie.as_ref() + /// Every inbound `Cookie` field, in order. + pub(crate) fn cookies(&self) -> &[HeaderValue] { + &self.cookies } pub(crate) fn user_agent(&self) -> Option<&HeaderValue> { @@ -228,7 +229,12 @@ impl PrebidTransportHeaders { fn snapshot(request: &Request) -> Self { Self { - cookie: request.headers().get(header::COOKIE).cloned(), + cookies: request + .headers() + .get_all(header::COOKIE) + .iter() + .cloned() + .collect(), user_agent: request.headers().get(header::USER_AGENT).cloned(), referer: request.headers().get(header::REFERER).cloned(), accept_language: request.headers().get(header::ACCEPT_LANGUAGE).cloned(), @@ -1336,7 +1342,14 @@ mod tests { Some(attested), ); let headers = routed.prebid_transport_headers(); - assert_eq!(headers.cookie(), Some(&HeaderValue::from_static("first=1"))); + assert_eq!( + headers.cookies(), + &[ + HeaderValue::from_static("first=1"), + HeaderValue::from_static("second=2") + ], + "should retain every Cookie field so identity cookies can be stripped from each" + ); assert_eq!( headers.user_agent().expect("should retain UA").as_bytes(), b"agent-\x80" diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index a7db8624e4..0ff38a1171 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -36,7 +36,9 @@ use crate::auction::types::{AuctionContext, AuctionRequest, MediaType}; use crate::auction::types::{AuctionResponse, Bid as AuctionBid}; use crate::cache_policy::{CacheControlPolicy, EdgeCacheHeader}; use crate::consent_config::ConsentForwardingMode; -use crate::cookies::{CONSENT_COOKIE_NAMES, strip_cookies}; +use crate::cookies::{ + CONSENT_COOKIE_NAMES, TS_IDENTITY_COOKIE_NAMES, TS_IDENTITY_COOKIE_PREFIXES, strip_cookie_pairs, +}; use crate::error::TrustedServerError; #[cfg(test)] use crate::http_util::RequestInfo; @@ -2026,8 +2028,14 @@ fn copy_request_headers( consent_forwarding: ConsentForwardingMode, client_ip: Option, ) { + let cookies: Vec = from + .headers() + .get_all(header::COOKIE) + .iter() + .cloned() + .collect(); apply_prebid_header_values( - from.headers().get(header::COOKIE), + &cookies, from.headers().get(header::USER_AGENT), from.headers().get(header::REFERER), from.headers().get(header::ACCEPT_LANGUAGE), @@ -2045,7 +2053,7 @@ pub(crate) fn apply_prebid_transport_headers( client_ip: Option, ) { apply_prebid_header_values( - from.cookie(), + from.cookies(), from.user_agent(), from.referer(), from.accept_language(), @@ -2060,7 +2068,7 @@ pub(crate) fn apply_prebid_transport_headers( reason = "the helper preserves four independently optional raw headers plus transport policy" )] fn apply_prebid_header_values( - cookie: Option<&HeaderValue>, + cookies: &[HeaderValue], user_agent: Option<&HeaderValue>, referer: Option<&HeaderValue>, accept_language: Option<&HeaderValue>, @@ -2085,27 +2093,27 @@ fn apply_prebid_header_values( .insert(header::HeaderName::from_static("x-forwarded-for"), value); } - let Some(cookie_value) = cookie else { - return; + // Identity cookies are stripped in every mode, before the consent mode is + // checked, so no mode forwards the raw header. + let consent_names: &[&str] = if consent_forwarding.strips_consent_cookies() { + CONSENT_COOKIE_NAMES + } else { + &[] }; - if !consent_forwarding.strips_consent_cookies() { - to.headers_mut() - .insert(header::COOKIE, cookie_value.clone()); - return; - } - match cookie_value.to_str() { - Ok(value) => { - let stripped = strip_cookies(value, CONSENT_COOKIE_NAMES); - if !stripped.is_empty() - && let Ok(cookie_header) = HeaderValue::from_str(&stripped) - { - to.headers_mut().insert(header::COOKIE, cookie_header); - } - } - Err(_) => { - to.headers_mut() - .insert(header::COOKIE, cookie_value.clone()); - } + let retained = cookies + .iter() + .filter_map(|value| value.to_str().ok()) + .map(|value| { + strip_cookie_pairs(value, TS_IDENTITY_COOKIE_NAMES, TS_IDENTITY_COOKIE_PREFIXES) + }) + .map(|value| strip_cookie_pairs(&value, consent_names, &[])) + .filter(|value| !value.is_empty()) + .collect::>() + .join("; "); + if !retained.is_empty() + && let Ok(cookie_header) = HeaderValue::from_str(&retained) + { + to.headers_mut().insert(header::COOKIE, cookie_header); } } @@ -9757,7 +9765,7 @@ set = { networkId = 42 } } #[test] - fn planned_transport_preserves_malformed_cookie_bytes_and_attested_xff() { + fn planned_transport_drops_malformed_cookie_bytes_and_uses_attested_xff() { let inbound = http::Request::builder() .uri("https://publisher.example/auction") .header( @@ -9801,8 +9809,8 @@ set = { networkId = 42 } .headers() .get(header::COOKIE) .map(HeaderValue::as_bytes), - Some(b"session=fictional;\xffbroken".as_slice()), - "should forward malformed cookie bytes unchanged" + None, + "should drop a non-UTF-8 Cookie field, which cannot be checked for identity cookies" ); assert_eq!( outbound @@ -9814,6 +9822,89 @@ set = { networkId = 42 } ); } + fn cookie_values(request: &http::Request) -> Vec { + request + .headers() + .get_all(header::COOKIE) + .iter() + .map(|value| value.to_str().expect("should be ASCII").to_owned()) + .collect() + } + + #[test] + fn prebid_headers_strip_identity_cookies_when_consent_cookies_are_forwarded() { + let cookies = [ + HeaderValue::from_static("ts-ec=abc; euconsent-v2=tcf; other=1"), + HeaderValue::from_static("__Host-ts-eids-s=v1.a.b; __Host-ts-eids-r-0a1b2c3d=v1.c.d"), + HeaderValue::from_static("ts-eids=W10; ts-ec-pull-complete=v1; second=2"), + ]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::Both, + None, + ); + + assert_eq!( + cookie_values(&to), + vec!["euconsent-v2=tcf; other=1; second=2".to_owned()], + "should drop identity cookies, keep consent and unrelated cookies, and read every field" + ); + } + + #[test] + fn prebid_headers_strip_identity_and_consent_cookies_in_openrtb_only_mode() { + let cookies = [HeaderValue::from_static( + "ts-ec=abc; euconsent-v2=tcf; other=1", + )]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::OpenrtbOnly, + None, + ); + + assert_eq!( + cookie_values(&to), + vec!["other=1".to_owned()], + "should keep only other=1" + ); + } + + #[test] + fn prebid_headers_drop_non_utf8_cookie_fields_and_empty_results() { + let cookies = [ + HeaderValue::from_bytes(b"x=\xff").expect("should build an opaque header"), + HeaderValue::from_static("ts-ec=abc"), + ]; + let mut to = http::Request::new(EdgeBody::empty()); + + apply_prebid_header_values( + &cookies, + None, + None, + None, + &mut to, + ConsentForwardingMode::Both, + None, + ); + + assert!( + to.headers().get(header::COOKIE).is_none(), + "should send no Cookie header" + ); + } + #[test] fn copy_request_headers_replaces_client_supplied_xff_with_attested_ip() { let from = http::Request::builder() @@ -9904,7 +9995,16 @@ set = { networkId = 42 } ConsentForwardingMode::OpenrtbOnly, HeaderValue::from_bytes(b"session=fictional;\xffbroken") .expect("should build malformed cookie header"), - Some(b"session=fictional;\xffbroken".as_slice()), + // A non-UTF-8 Cookie field cannot be checked for identity + // cookies, so it is dropped rather than forwarded. + None, + ), + ( + ConsentForwardingMode::Both, + HeaderValue::from_static( + "ts-ec=fictional; session=fictional; __Host-ts-eids-s=v1.a.b; ts-eids=W10", + ), + Some(b"session=fictional".as_slice()), ), ] { let stub = Arc::new(StubHttpClient::new()); From cf708fb783fa58eac5cdf57ffe1e8b073c65a4fe Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:34:18 +0530 Subject: [PATCH 04/33] Strip ID cookies from publisher origin requests The origin keeps ts-ec and the pull-complete marker. ts-eids, __Host-ts-eids-s and the __Host-ts-eids-r- cookies no longer reach it. An origin that read ts-eids must read x-ts-eids instead. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/publisher.rs | 74 ++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index e6761efdca..7a327ec372 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -58,8 +58,11 @@ use crate::cache_policy::{ }; use crate::consent::{consent_allows_server_side_auction, gate_eids_by_consent}; use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS, HEADER_X_COMPRESS_HINT}; -use crate::cookies::handle_request_cookies; use crate::cookies::template_cache_policy::{TemplateCookieDecision, evaluate_cookie_policy}; +use crate::cookies::{ + TS_IDENTITY_COOKIE_PREFIXES, TS_ORIGIN_STRIPPED_COOKIE_NAMES, handle_request_cookies, + strip_cookie_headers, +}; use crate::creative_opportunities::{ AdStackGateInput, AssemblyMode, CreativeOpportunitiesConfig, RuntimeAdStackExpected, evaluate_ad_stack_gate, @@ -3136,6 +3139,13 @@ fn rewrite_origin_request( // separate change. Until then we strip it here so the edge signal never // reaches publisher backends. req.headers_mut().remove("fastly-ssl"); + // The publisher origin keeps `ts-ec` but never receives the ID cookies, + // which can push its `Cookie` header past common 8 KB limits. + strip_cookie_headers( + req.headers_mut(), + TS_ORIGIN_STRIPPED_COOKIE_NAMES, + TS_IDENTITY_COOKIE_PREFIXES, + ); *req.uri_mut() = target_uri; req.headers_mut().insert( header::HOST, @@ -25324,3 +25334,65 @@ mod tests { } } } + +#[cfg(test)] +mod origin_cookie_strip_tests { + use super::*; + + #[test] + fn rewrite_origin_request_strips_id_cookies_and_keeps_ts_ec() { + let mut req = Request::builder() + .uri("https://www.example.com/article") + .header(header::COOKIE, "ts-ec=abc; ts-eids=W10; session=1") + .header( + header::COOKIE, + "__Host-ts-eids-s=v1.a.b; __Host-ts-eids-r-0a1b2c3d=v1.c.d; ts-ec-pull-complete=v1", + ) + .body(EdgeBody::empty()) + .expect("should build request"); + + rewrite_origin_request( + &mut req, + "https://origin.example.com/article" + .parse() + .expect("should parse uri"), + "origin.example.com", + ) + .expect("should rewrite request"); + + let cookies: Vec<_> = req.headers().get_all(header::COOKIE).iter().collect(); + assert_eq!(cookies.len(), 1, "should send one Cookie header"); + assert_eq!( + cookies[0], "ts-ec=abc; session=1; ts-ec-pull-complete=v1", + "should keep ts-ec, the pull marker and unrelated cookies, and drop the ID cookies" + ); + } + + #[test] + fn rewrite_origin_request_drops_cookie_header_when_only_id_cookies() { + let mut req = Request::builder() + .uri("https://www.example.com/") + .header(header::COOKIE, "ts-eids=W10; __Host-ts-eids-s=v1.a.b") + .body(EdgeBody::empty()) + .expect("should build request"); + + rewrite_origin_request( + &mut req, + "https://origin.example.com/" + .parse() + .expect("should parse uri"), + "origin.example.com", + ) + .expect("should rewrite request"); + + assert!( + req.headers().get(header::COOKIE).is_none(), + "should drop the empty Cookie header" + ); + assert_eq!( + req.headers()[header::HOST], + "origin.example.com", + "should still set Host" + ); + } +} From 24780c05cec07ea2b3bc958bc8d84c9d1ed0c7e8 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:35:40 +0530 Subject: [PATCH 05/33] Refuse the sealed identity cookies as template cache keys Documents which identity cookies reach the publisher origin and Prebid Server. Signed-off-by: dhruv8sh --- CHANGELOG.md | 1 + .../src/cookies/template_cache_policy.rs | 29 +++++++++++++++++-- docs/guide/edge-cookies.md | 15 ++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f691c6a005..9829af8b32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** The publisher origin no longer receives the `ts-eids` cookie, and Prebid Server no longer receives any Trusted Server identity cookie (`ts-ec`, `ts-eids`, `ts-ec-pull-complete`, and the server-owned `__Host-ts-eids-s` and `__Host-ts-eids-r-*` cookies that a later release introduces). The origin still receives `ts-ec`. Origin code that read `ts-eids` should read the `x-ts-eids` response header instead. A Prebid Server `Cookie` header that is not valid UTF-8 is now dropped instead of forwarded. This release is the rollback floor for the server-side identity provider work. - **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER__GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header. - **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias. - The S2S `/_ts/api/v1/batch-sync` endpoint now validates the full batch and calls the CAS-protected update path once per distinct normalized EC ID. The last valid UID wins within a group, and infrastructure failures reject the failing and each unprocessed group, so accepted and `kv_unavailable` input indexes may interleave. diff --git a/crates/trusted-server-core/src/cookies/template_cache_policy.rs b/crates/trusted-server-core/src/cookies/template_cache_policy.rs index 61c3b9139c..7e7f4f5319 100644 --- a/crates/trusted-server-core/src/cookies/template_cache_policy.rs +++ b/crates/trusted-server-core/src/cookies/template_cache_policy.rs @@ -2,7 +2,9 @@ use std::collections::{HashMap, HashSet}; -use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EC, COOKIE_TS_EIDS}; +use crate::constants::{ + COOKIE_SHAREDID, COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_TS_EIDS_R_PREFIX, COOKIE_TS_EIDS_S, +}; use crate::platform::TemplateCookieValue; /// Whether the request can share a template and its explicit cookie dimensions. @@ -32,7 +34,14 @@ pub(crate) fn validate_cookie_names( return Err(format!("{field} contains invalid cookie name `{name}`")); } if field == "template_cache_key_cookies" - && [COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_SHAREDID].contains(&name.as_str()) + && ([ + COOKIE_TS_EC, + COOKIE_TS_EIDS, + COOKIE_TS_EIDS_S, + COOKIE_SHAREDID, + ] + .contains(&name.as_str()) + || name.starts_with(COOKIE_TS_EIDS_R_PREFIX)) { return Err(format!( "{field} must not key on Trusted Server identity cookie `{name}`" @@ -246,6 +255,22 @@ mod tests { } } + #[test] + fn template_cookie_policy_rejects_sealed_identity_cookie_keys() { + for name in ["__Host-ts-eids-s", "__Host-ts-eids-r-0a1b2c3d"] { + let error = validate_cookie_names(&names(&[name]), &[]) + .expect_err("should reject identity cookie as a cache key"); + assert!( + error.contains("must not key on Trusted Server identity cookie"), + "should explain the rejection for {name}, got {error}" + ); + } + assert!( + validate_cookie_names(&names(&["session"]), &[]).is_ok(), + "should still accept an unrelated cookie" + ); + } + #[test] fn template_cookie_policy_allows_only_independent_unlisted_duplicates() { for fields in [ diff --git a/docs/guide/edge-cookies.md b/docs/guide/edge-cookies.md index 979e268b70..61acae11fa 100644 --- a/docs/guide/edge-cookies.md +++ b/docs/guide/edge-cookies.md @@ -304,6 +304,21 @@ sets `Path=/`, `Secure`, `HttpOnly`, `SameSite=Lax`, and a `Max-Age`. `Domain` is computed as `.{publisher.domain}`. The separate `cookie_domain` setting applies only to non-EC cookies. +## Cookies Forwarded Upstream + +Trusted Server keeps its identity cookies away from third parties and out of +the publisher origin's `Cookie` header: + +- **Publisher origin.** `ts-ec` and `ts-ec-pull-complete` are still forwarded, + because publisher servers may read the EC. `ts-eids` and the server-owned + `__Host-ts-eids-s` and `__Host-ts-eids-r-*` cookies are removed, so they + cannot push the origin's `Cookie` header past common 8 KB limits. Origin + code that read `ts-eids` should read the `x-ts-eids` response header + instead. +- **Prebid Server.** No Trusted Server identity cookie is forwarded, in any + consent-forwarding mode. A `Cookie` header that is not valid UTF-8 is + dropped rather than forwarded. + ## Operational Notes - Rotate the secret periodically. Rotation produces a new 64-hex base From 7a3fe16e2e88078eba653e871ebf8ec4a8b35b0b Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:44:02 +0530 Subject: [PATCH 06/33] Add the identity module with provider and source ID newtypes Adds aes-gcm, which the sealed identity cookies use. Signed-off-by: dhruv8sh --- Cargo.lock | 63 ++++- Cargo.toml | 1 + crates/trusted-server-core/Cargo.toml | 1 + .../src/ec/identity/descriptor.rs | 41 ++++ .../src/ec/identity/ids.rs | 229 ++++++++++++++++++ .../src/ec/identity/mod.rs | 15 ++ .../src/ec/identity/resolution.rs | 103 ++++++++ crates/trusted-server-core/src/ec/mod.rs | 1 + .../src/migration_guards.rs | 10 + 9 files changed, 461 insertions(+), 3 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity/descriptor.rs create mode 100644 crates/trusted-server-core/src/ec/identity/ids.rs create mode 100644 crates/trusted-server-core/src/ec/identity/mod.rs create mode 100644 crates/trusted-server-core/src/ec/identity/resolution.rs diff --git a/Cargo.lock b/Cargo.lock index 00dff9fc05..1fc4ab6ed5 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -18,6 +18,31 @@ dependencies = [ "generic-array", ] +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures 0.2.17", +] + +[[package]] +name = "aes-gcm" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" +dependencies = [ + "aead", + "aes", + "cipher", + "ctr", + "ghash", + "subtle", +] + [[package]] name = "ahash" version = "0.8.12" @@ -1114,6 +1139,15 @@ dependencies = [ "linktime-proc-macro", ] +[[package]] +name = "ctr" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" +dependencies = [ + "cipher", +] + [[package]] name = "curve25519-dalek" version = "4.1.3" @@ -2181,6 +2215,16 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "ghash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" +dependencies = [ + "opaque-debug", + "polyval", +] + [[package]] name = "gix" version = "0.83.0" @@ -4600,6 +4644,18 @@ dependencies = [ "universal-hash", ] +[[package]] +name = "polyval" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "opaque-debug", + "universal-hash", +] + [[package]] name = "portable-atomic" version = "1.13.1" @@ -4761,7 +4817,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" dependencies = [ "heck", - "itertools 0.10.5", + "itertools 0.13.0", "log", "multimap", "once_cell", @@ -4781,7 +4837,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" dependencies = [ "anyhow", - "itertools 0.10.5", + "itertools 0.13.0", "proc-macro2", "quote", "syn 2.0.118", @@ -4794,7 +4850,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" dependencies = [ "anyhow", - "itertools 0.10.5", + "itertools 0.13.0", "proc-macro2", "quote", "syn 2.0.118", @@ -6636,6 +6692,7 @@ dependencies = [ name = "trusted-server-core" version = "0.1.0" dependencies = [ + "aes-gcm", "async-stream", "async-trait", "base64", diff --git a/Cargo.toml b/Cargo.toml index c0d79bc033..0d494f9ac7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -36,6 +36,7 @@ publish = false version = "0.1.0" [workspace.dependencies] +aes-gcm = "0.10" anyhow = "1" assert_cmd = "2" async-stream = "0.3" diff --git a/crates/trusted-server-core/Cargo.toml b/crates/trusted-server-core/Cargo.toml index 0a8d052805..98a7031533 100644 --- a/crates/trusted-server-core/Cargo.toml +++ b/crates/trusted-server-core/Cargo.toml @@ -11,6 +11,7 @@ version = { workspace = true } workspace = true [dependencies] +aes-gcm = { workspace = true } async-trait = { workspace = true } async-stream = { workspace = true } base64 = { workspace = true } diff --git a/crates/trusted-server-core/src/ec/identity/descriptor.rs b/crates/trusted-server-core/src/ec/identity/descriptor.rs new file mode 100644 index 0000000000..d1e92afa73 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/descriptor.rs @@ -0,0 +1,41 @@ +//! What a provider is, may write, needs and must respect. + +/// An input a provider can ask for. +/// +/// Consent strings, the provider's stored IDs, the publisher domain, the +/// time and the declared endpoint are always available. Private state is +/// available to providers declared `with_private_state()`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum Input { + /// The platform's client address. + #[display("client_ip")] + ClientIp, + /// The request's `User-Agent`. + #[display("user_agent")] + UserAgent, + /// The top-level page URL without query string or fragment. + #[display("page_url")] + PageUrl, + /// Country and subdivision only. + #[display("region")] + Region, + /// A SHA-256 of a normalized email address. Never available in v1. + #[display("hashed_email")] + HashedEmail, +} + +/// Something a provider read without declaring it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum Access { + /// An undeclared [`Input`]. + #[display("{_0}")] + Input(Input), + /// The endpoint, when the descriptor declares none. + #[display("endpoint")] + Endpoint, + /// Private state, without `with_private_state()`. + #[display("private_state")] + PrivateState, +} diff --git a/crates/trusted-server-core/src/ec/identity/ids.rs b/crates/trusted-server-core/src/ec/identity/ids.rs new file mode 100644 index 0000000000..8037f40029 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/ids.rs @@ -0,0 +1,229 @@ +//! Identifiers shared by the identity provider API. + +use core::ops::Add; +use core::str::FromStr; +use std::time::Duration; + +use error_stack::Report; +use serde::{Deserialize, Serialize}; + +use super::resolution::IdentityProviderError; +use crate::ec::partner::normalize_partner_source_domain; + +/// Longest identity provider ID, in bytes. +const MAX_PROVIDER_ID_LENGTH: usize = 63; + +/// An identity provider's ID, such as `id5` or `pull-ssp-example-com`. +/// +/// Matches `^[a-z][a-z0-9_-]{0,62}$`, so it can be a bare TOML key and can +/// equal an integration ID that uses `_`. Separate from +/// [`crate::auction::ProviderId`]. +#[derive( + Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, Serialize, Deserialize, +)] +#[serde(try_from = "String", into = "String")] +pub struct IdentityProviderId(String); + +impl IdentityProviderId { + /// The ID as written in config. + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl FromStr for IdentityProviderId { + type Err = Report; + + fn from_str(value: &str) -> Result { + let mut bytes = value.bytes(); + let valid_first = bytes.next().is_some_and(|byte| byte.is_ascii_lowercase()); + let valid_rest = bytes.all(|byte| { + byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'_' || byte == b'-' + }); + if valid_first && valid_rest && value.len() <= MAX_PROVIDER_ID_LENGTH { + Ok(Self(value.to_owned())) + } else { + Err(IdentityProviderError::config(format!( + "identity provider id `{value}` must match ^[a-z][a-z0-9_-]{{0,62}}$" + ))) + } + } +} + +impl TryFrom for IdentityProviderId { + type Error = Report; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(id: IdentityProviderId) -> Self { + id.0 + } +} + +/// An EID `source` domain, normalized. Also the key of the KV `ids` map. +#[derive( + Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display, Serialize, Deserialize, +)] +#[serde(try_from = "String", into = "String")] +pub struct SourceDomain(String); + +impl SourceDomain { + /// The normalized domain. + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } + + /// Whether a vendor's spelling of a source names this one, after + /// normalization. + #[must_use] + pub fn matches(&self, name: &str) -> bool { + normalize_partner_source_domain(name).is_ok_and(|normalized| normalized == self.0) + } +} + +impl FromStr for SourceDomain { + type Err = Report; + + fn from_str(value: &str) -> Result { + normalize_partner_source_domain(value) + .map(Self) + .map_err(IdentityProviderError::config) + } +} + +impl TryFrom for SourceDomain { + type Error = Report; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(source: SourceDomain) -> Self { + source.0 + } +} + +/// Unix seconds. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub struct UnixTime(u64); + +impl UnixTime { + /// From Unix seconds. + #[must_use] + pub fn from_secs(secs: u64) -> Self { + Self(secs) + } + + /// From Unix milliseconds, for vendors such as Lockr and UID2. Floors to + /// whole seconds. + #[must_use] + pub fn from_millis(millis: u64) -> Self { + Self(millis / 1000) + } + + /// As Unix seconds. + #[must_use] + pub fn as_secs(self) -> u64 { + self.0 + } +} + +impl Add for UnixTime { + type Output = UnixTime; + + fn add(self, rhs: Duration) -> UnixTime { + UnixTime(self.0.saturating_add(rhs.as_secs())) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn provider_id_accepts_the_documented_grammar() { + for id in ["id5", "pull-ssp-example-com", "lockr_aim", "a"] { + assert!( + id.parse::().is_ok(), + "should accept `{id}`" + ); + } + let longest = format!("a{}", "b".repeat(62)); + longest + .parse::() + .expect("should accept 63 characters"); + } + + #[test] + fn provider_id_rejects_everything_else() { + let too_long = format!("a{}", "b".repeat(63)); + for id in ["", "1id", "Id5", "id 5", "id.5", too_long.as_str()] { + let error = id + .parse::() + .expect_err("should reject the id"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ), + "should be a Config error for `{id}`" + ); + } + } + + #[test] + fn source_domain_normalizes_and_matches() { + let source: SourceDomain = "ID5-Sync.com.".parse().expect("should parse"); + assert_eq!( + source.as_str(), + "id5-sync.com", + "should lower-case and drop the trailing dot" + ); + assert!( + source.matches("id5-sync.COM"), + "should match another spelling" + ); + assert!( + !source.matches("example.com"), + "should not match another domain" + ); + assert!( + !source.matches("https://id5-sync.com"), + "should not match a URL" + ); + } + + #[test] + fn source_domain_round_trips_through_serde() { + let source: SourceDomain = "example.com".parse().expect("should parse"); + let json = serde_json::to_string(&source).expect("should serialize"); + assert_eq!(json, "\"example.com\"", "should serialize as a bare string"); + let back: SourceDomain = serde_json::from_str(&json).expect("should deserialize"); + assert_eq!(back, source, "should round-trip"); + assert!( + serde_json::from_str::("\"not/a/domain\"").is_err(), + "should reject an invalid domain on deserialize" + ); + } + + #[test] + fn unix_time_converts_millis_and_adds_durations() { + assert_eq!( + UnixTime::from_millis(1_760_000_000_999).as_secs(), + 1_760_000_000, + "should floor millis" + ); + let later = UnixTime::from_secs(100) + Duration::from_secs(50); + assert_eq!(later, UnixTime::from_secs(150), "should add seconds"); + let saturated = UnixTime::from_secs(u64::MAX) + Duration::from_secs(1); + assert_eq!(saturated, UnixTime::from_secs(u64::MAX), "should saturate"); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs new file mode 100644 index 0000000000..398b542b43 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -0,0 +1,15 @@ +//! Server-side identity providers. +//! +//! An [`IdentityProvider`](provider::IdentityProvider) turns request signals +//! into `OpenRTB` EIDs. Providers only build requests and parse buffered +//! responses. The coordinator owns eligibility, consent, transport limits and +//! KV write-back, so every page load makes at most one browser-driven KV +//! write. + +pub mod descriptor; +pub mod ids; +pub mod resolution; + +pub use descriptor::{Access, Input}; +pub use ids::{IdentityProviderId, SourceDomain, UnixTime}; +pub use resolution::IdentityProviderError; diff --git a/crates/trusted-server-core/src/ec/identity/resolution.rs b/crates/trusted-server-core/src/ec/identity/resolution.rs new file mode 100644 index 0000000000..08360f87f0 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/resolution.rs @@ -0,0 +1,103 @@ +//! Provider results and errors. + +use error_stack::Report; +use http::StatusCode; + +use super::descriptor::{Access, Input}; + +/// Marker message of [`IdentityProviderError::not_implemented`]. The harness +/// matches on it to tell a missing override from a real failure. +pub(crate) const NOT_IMPLEMENTED_MARKER: &str = "capability declared but not implemented: "; + +/// A provider failure. Built through the constructors below. +#[derive(Debug, derive_more::Display)] +#[non_exhaustive] +pub enum IdentityProviderError { + /// Configuration or provider code makes the call impossible. + #[display("identity provider configuration error: {message}")] + Config { + /// What is wrong. Never contains a secret, UID or URL query. + message: String, + }, + /// The provider read something its descriptor did not declare. + #[display("`{access}` was read but not declared")] + UndeclaredInput { + /// What was read. + access: Access, + }, + /// A declared input is not available for this request. + #[display("input `{input}` is unavailable for this request")] + InputUnavailable { + /// The missing input. + input: Input, + }, + /// The provider refused the request. + #[display("provider rejected the request with status {status}")] + Rejected { + /// The response status. + status: StatusCode, + }, + /// The provider's response could not be parsed. + #[display("provider response could not be parsed")] + Malformed, + /// The stored private state does not decode as the provider's type. + #[display("stored provider state could not be decoded")] + InvalidState, + /// Set by the coordinator only. Providers never see transport failures. + #[display("provider request failed in transport")] + Transport, +} + +impl core::error::Error for IdentityProviderError {} + +impl IdentityProviderError { + /// A [`Self::Config`] error. + pub fn config(message: impl Into) -> Report { + Report::new(Self::Config { + message: message.into(), + }) + } + + /// A [`Self::Rejected`] error. + pub fn rejected(status: StatusCode) -> Report { + Report::new(Self::Rejected { status }) + } + + /// A [`Self::Malformed`] error. + pub fn malformed() -> Report { + Report::new(Self::Malformed) + } + + /// The default body of a capability method: a `Config` error with a + /// fixed marker message, so the harness can tell a missing override from + /// a real failure. + pub fn not_implemented(method: &'static str) -> Report { + Self::config(format!("{NOT_IMPLEMENTED_MARKER}{method}")) + } + + /// Whether this is the [`Self::not_implemented`] marker. + #[cfg(test)] + #[must_use] + pub(crate) fn is_not_implemented(&self) -> bool { + matches!(self, Self::Config { message } if message.starts_with(NOT_IMPLEMENTED_MARKER)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn not_implemented_marker_is_recognised() { + let marker = IdentityProviderError::not_implemented("capture"); + assert!( + marker.current_context().is_not_implemented(), + "should recognise the marker" + ); + let other = IdentityProviderError::config("bad partner id"); + assert!( + !other.current_context().is_not_implemented(), + "an ordinary Config error is not the marker" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index cc6705eaf8..ce599c0c36 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -41,6 +41,7 @@ pub mod eids; pub mod finalize; pub mod generation; pub mod identify; +pub mod identity; pub mod kv; pub mod kv_backend; pub mod kv_types; diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 683fbbc138..d22b47c1b8 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -83,6 +83,16 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { ("ec/finalize.rs", include_str!("ec/finalize.rs")), ("ec/generation.rs", include_str!("ec/generation.rs")), ("ec/identify.rs", include_str!("ec/identify.rs")), + ( + "ec/identity/descriptor.rs", + include_str!("ec/identity/descriptor.rs"), + ), + ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), + ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), + ( + "ec/identity/resolution.rs", + include_str!("ec/identity/resolution.rs"), + ), ("ec/kv.rs", include_str!("ec/kv.rs")), ("ec/kv_types.rs", include_str!("ec/kv_types.rs")), ("ec/mod.rs", include_str!("ec/mod.rs")), From 1dde88037cbfb33be1169b533c353e71b3bb3544 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:47:23 +0530 Subject: [PATCH 07/33] Add provider metadata and provider state to the KV schema New fields are optional and omitted when absent, so legacy rows read and write unchanged and the schema version stays 1. KvPartnerId and KvEntry derive Default, and literals move to constructors. The admin lookup redacts provider state. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/ec/admin.rs | 60 +++- crates/trusted-server-core/src/ec/eids.rs | 8 +- crates/trusted-server-core/src/ec/finalize.rs | 4 +- crates/trusted-server-core/src/ec/kv.rs | 64 ++-- crates/trusted-server-core/src/ec/kv_types.rs | 315 ++++++++++++++++-- .../trusted-server-core/src/ec/pull_sync.rs | 12 +- .../src/ec/pull_sync_marker.rs | 8 +- crates/trusted-server-core/src/publisher.rs | 4 +- 8 files changed, 378 insertions(+), 97 deletions(-) diff --git a/crates/trusted-server-core/src/ec/admin.rs b/crates/trusted-server-core/src/ec/admin.rs index 18b5563307..ef0f018e3d 100644 --- a/crates/trusted-server-core/src/ec/admin.rs +++ b/crates/trusted-server-core/src/ec/admin.rs @@ -368,6 +368,19 @@ fn requested_ec_id(req: &Request) -> Result.st` with a marker. Provider state is never +/// returned by the admin lookup. +fn redact_provider_state(entry_json: &mut JsonValue) { + let Some(states) = entry_json.get_mut("ps").and_then(JsonValue::as_object_mut) else { + return; + }; + for state in states.values_mut() { + if let Some(st) = state.get_mut("st") { + *st = JsonValue::String("[redacted]".to_owned()); + } + } +} + /// Builds the success payload from a raw KV lookup. /// /// Parse failures are reported in the payload rather than propagated, so @@ -393,6 +406,7 @@ fn build_lookup_response( match serde_json::from_slice::(&lookup.body) { Ok(mut entry_json) => { + redact_provider_state(&mut entry_json); add_iso_timestamp_companions(&mut entry_json); payload.entry = Some(entry_json); @@ -778,15 +792,11 @@ mod tests { let mut entry = KvEntry::minimal("bidstream.example", "uid-live", 1_741_824_000); entry.ids.insert( "disabled.example".to_owned(), - KvPartnerId { - uid: "uid-disabled".to_owned(), - }, + KvPartnerId::new("uid-disabled".to_owned()), ); entry.ids.insert( "unknown.example".to_owned(), - KvPartnerId { - uid: "uid-unknown".to_owned(), - }, + KvPartnerId::new("uid-unknown".to_owned()), ); entry } @@ -1307,6 +1317,44 @@ mod tests { assert_eq!(response.status(), StatusCode::NOT_IMPLEMENTED); } + #[test] + fn admin_lookup_redacts_provider_state() { + let mut entry = KvEntry::minimal("example.com", "example-uid", 1_760_000_000); + entry.ps.insert( + "id5".to_owned(), + crate::ec::kv_types::KvProviderState { + n: Some(5), + st: Some(serde_json::json!({ "signature": "secret-state" })), + ..crate::ec::kv_types::KvProviderState::default() + }, + ); + let lookup = EcKvLookup { + body: serde_json::to_vec(&entry).expect("should serialize entry"), + metadata: None, + generation: 1, + }; + let registry = PartnerRegistry::from_config(&[]).expect("should build empty registry"); + + let payload = build_lookup_response(®istry, "ec-store", "ec".to_owned(), &lookup); + let body = payload.entry.expect("should include the entry"); + + assert_eq!( + body["ps"]["id5"]["st"], + serde_json::json!("[redacted]"), + "should redact the state" + ); + assert_eq!( + body["ps"]["id5"]["n"], + serde_json::json!(5), + "should keep the rest of ps" + ); + assert_eq!( + body["ids"]["example.com"]["uid"], + serde_json::json!("example-uid"), + "should keep the ids" + ); + } + #[test] fn unsupported_ec_lookup_response_is_json_and_no_store() { let response = admin_ec_lookup_not_supported(); diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index 0493f5dec1..fe03c47d33 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -181,15 +181,11 @@ mod tests { entry.consent.ok = true; entry.ids.insert( "zeta.example.com".to_owned(), - super::super::kv_types::KvPartnerId { - uid: "uid-z".to_owned(), - }, + super::super::kv_types::KvPartnerId::new("uid-z".to_owned()), ); entry.ids.insert( "alpha.example.com".to_owned(), - super::super::kv_types::KvPartnerId { - uid: "uid-a".to_owned(), - }, + super::super::kv_types::KvPartnerId::new("uid-a".to_owned()), ); let resolved = resolve_partner_ids(®istry, &entry); diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index e41a8180d4..5612d821c7 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -2376,9 +2376,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "partner-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("partner-uid".to_owned()), ); ec_context.set_kv_snapshot(EcKvSnapshot::Present { ec_id, diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index b0eb83d3fa..9d0ff4811a 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -157,9 +157,7 @@ fn apply_cookie_partner_id_updates( entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, + super::kv_types::KvPartnerId::new(uid.to_owned()), ); changed = true; } @@ -205,9 +203,7 @@ pub(crate) fn apply_partner_id_updates(entry: &mut KvEntry, updates: &[PartnerId entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, + super::kv_types::KvPartnerId::new(uid.to_owned()), ); changed = true; } @@ -920,9 +916,7 @@ impl KvIdentityGraph { entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, + super::kv_types::KvPartnerId::new(uid.to_owned()), ); let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; @@ -992,9 +986,7 @@ impl KvIdentityGraph { entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, + super::kv_types::KvPartnerId::new(uid.to_owned()), ); let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; @@ -1737,9 +1729,9 @@ mod tests { let mut entry = KvEntry::tombstone(1000); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), - }, + crate::ec::kv_types::KvPartnerId::new( + "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), + ), ); let body = serde_json::to_vec(&entry).expect("should serialize invalid entry payload"); @@ -1772,9 +1764,9 @@ mod tests { let mut entry = KvEntry::tombstone(1000); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), - }, + crate::ec::kv_types::KvPartnerId::new( + "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), + ), ); let err = KvIdentityGraph::serialize_entry(&entry, "test-store") @@ -1796,9 +1788,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "concurrent.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "concurrent-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("concurrent-uid".to_owned()), ); entry } @@ -2131,9 +2121,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "uid-1".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("uid-1".to_owned()), ); let updates = vec![PartnerIdUpdate::new("ssp_x", "uid-1")]; @@ -2159,9 +2147,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "old-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("old-uid".to_owned()), ); let updates = vec![PartnerIdUpdate::new("ssp_x", "new-uid")]; @@ -2191,9 +2177,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "original".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("original".to_owned()), ); let updates = vec![ PartnerIdUpdate::new("ssp_x", "intermediate"), @@ -2215,9 +2199,7 @@ mod tests { let mut concurrent = live_entry(); concurrent.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "desired-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("desired-uid".to_owned()), ); let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -2333,9 +2315,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "latest-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("latest-uid".to_owned()), ); let updates = [ PartnerIdUpdate::new("ssp_x", "older-uid"), @@ -2354,9 +2334,7 @@ mod tests { let mut concurrent = live_entry(); concurrent.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "concurrent-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("concurrent-uid".to_owned()), ); let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -2439,9 +2417,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "protected-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("protected-uid".to_owned()), ); graph.create(&ec_id, &entry).expect("should seed entry"); @@ -2466,9 +2442,7 @@ mod tests { let mut entry = live_entry(); entry.ids.insert( "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "protected-uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("protected-uid".to_owned()), ); graph.create(&ec_id, &entry).expect("should seed entry"); diff --git a/crates/trusted-server-core/src/ec/kv_types.rs b/crates/trusted-server-core/src/ec/kv_types.rs index 589f280c11..d223f2c418 100644 --- a/crates/trusted-server-core/src/ec/kv_types.rs +++ b/crates/trusted-server-core/src/ec/kv_types.rs @@ -34,11 +34,22 @@ pub const MAX_STORED_DOMAIN_LENGTH: usize = 255; /// validation. pub const MAX_UID_LENGTH: usize = 512; +/// Maximum serialized bytes of a stored UID-level `ext`. +pub const MAX_PARTNER_EXT_BYTES: usize = 256; +/// Maximum serialized bytes of one provider's private state (`ps.st`). +pub const MAX_PROVIDER_STATE_BYTES: usize = 1024; +/// Maximum number of providers with a `ps` entry in one row. +pub const MAX_PROVIDER_STATES: usize = 32; +/// Maximum number of invalidated-UID hashes kept per provider (`ps.iv`). +pub const MAX_INVALIDATED_UIDS: usize = 4; +/// Length of a record revision (`rv`) in lowercase hex digits. +pub const RECORD_REVISION_HEX_LEN: usize = 16; + /// Full KV entry stored as the body of an EC identity graph record. /// /// **KV key:** Full EC ID (`{64hex}.{6alnum}`). /// **KV value:** JSON-serialized `KvEntry` (max ~5KB). -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct KvEntry { /// Schema version — always [`SCHEMA_VERSION`]. pub v: u8, @@ -62,10 +73,14 @@ pub struct KvEntry { /// Populated by pixel sync, batch sync, and pull sync operations. #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] pub ids: BTreeMap, + /// Per-provider scheduling, opt-out and private state, keyed by identity + /// provider ID. Never sent to bidders or cookies. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub ps: BTreeMap, } /// Consent state within a KV entry. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct KvConsent { /// Raw TCF v2 consent string. #[serde(default, skip_serializing_if = "Option::is_none")] @@ -80,7 +95,7 @@ pub struct KvConsent { } /// Geo location within a KV entry. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct KvGeo { /// ISO 3166-1 alpha-2 country code (e.g. `"US"`). pub country: String, @@ -98,10 +113,98 @@ pub struct KvGeo { } /// A partner user ID within a KV entry. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +/// +/// Every field but `uid` is optional and omitted when absent, so legacy rows +/// read and write unchanged and [`SCHEMA_VERSION`] stays 1. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct KvPartnerId { /// The partner's user identifier. pub uid: String, + /// Writer: an identity provider ID. Absent for browser ingest and legacy rows. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub w: Option, + /// Record revision: 16 lowercase hex digits, random on every write. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rv: Option, + /// Unix seconds when the ID was acquired or last refreshed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub at: Option, + /// Unix seconds after which the ID must not be sent. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub exp: Option, + /// Unix seconds after which the ID is due for refresh. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ra: Option, + /// Unix seconds after which the ID can no longer be refreshed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ru: Option, + /// Provider-supplied `atype`. Absent: use the partner's `openrtb_atype`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub atype: Option, + /// UID-level `ext`, at most [`MAX_PARTNER_EXT_BYTES`] serialized bytes. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ext: Option, +} + +impl KvPartnerId { + /// A bare record holding only a UID. + #[must_use] + pub fn new(uid: impl Into) -> Self { + Self { + uid: uid.into(), + ..Self::default() + } + } + + /// A browser-ingested record: no writer, with its acquisition time. + #[must_use] + pub fn browser(uid: impl Into, acquired_at: Option) -> Self { + Self { + uid: uid.into(), + at: acquired_at, + ..Self::default() + } + } +} + +/// A provider's sticky opt-out. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct KvOptOut { + /// Unix seconds when the opt-out was reported. + pub at: u64, + /// Consent fingerprint at that time: 16 lowercase hex digits. + pub fp: String, +} + +/// Per-provider state in a KV row (`ps.`). +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct KvProviderState { + /// Unix seconds before which the provider must not be called. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub n: Option, + /// Consecutive failures, for backoff. + #[serde(default, skip_serializing_if = "is_zero")] + pub f: u32, + /// Sticky opt-out. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub o: Option, + /// When the provider's opt-out was last cleared. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oc: Option, + /// Keyed hashes of UIDs the owner invalidated, newest last. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub iv: Vec, + /// The provider's private state as embedded JSON. KV only. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub st: Option, +} + +#[allow( + clippy::trivially_copy_pass_by_ref, + reason = "serde's skip_serializing_if passes the field by reference" +)] +fn is_zero(value: &u32) -> bool { + *value == 0 } /// Publisher property metadata captured when an EC entry is created. @@ -293,6 +396,7 @@ impl KvEntry { device: None, network: None, ids: BTreeMap::new(), + ps: BTreeMap::new(), } } @@ -303,12 +407,7 @@ impl KvEntry { #[must_use] pub fn minimal(partner_id: &str, uid: &str, now: u64) -> Self { let mut ids = BTreeMap::new(); - ids.insert( - partner_id.to_owned(), - KvPartnerId { - uid: uid.to_owned(), - }, - ); + ids.insert(partner_id.to_owned(), KvPartnerId::new(uid.to_owned())); Self { v: SCHEMA_VERSION, created: now, @@ -328,6 +427,7 @@ impl KvEntry { device: None, network: None, ids, + ps: BTreeMap::new(), } } @@ -360,6 +460,7 @@ impl KvEntry { device: None, network: None, ids: BTreeMap::new(), + ps: BTreeMap::new(), } } @@ -387,6 +488,41 @@ impl KvEntry { partner_uid.uid.len() )); } + if let Some(ext) = &partner_uid.ext + && serialized_len(ext) > MAX_PARTNER_EXT_BYTES + { + return Err(format!( + "partner ID '{partner_id}' ext exceeds {MAX_PARTNER_EXT_BYTES} bytes" + )); + } + if let Some(rv) = &partner_uid.rv + && !is_record_revision(rv) + { + return Err(format!( + "partner ID '{partner_id}' rv is not {RECORD_REVISION_HEX_LEN} lowercase hex digits" + )); + } + } + + if self.ps.len() > MAX_PROVIDER_STATES { + return Err(format!( + "ps exceeds MAX_PROVIDER_STATES ({})", + self.ps.len() + )); + } + for (provider, state) in &self.ps { + if state.iv.len() > MAX_INVALIDATED_UIDS { + return Err(format!( + "ps '{provider}' iv exceeds {MAX_INVALIDATED_UIDS} hashes" + )); + } + if let Some(st) = &state.st + && serialized_len(st) > MAX_PROVIDER_STATE_BYTES + { + return Err(format!( + "ps '{provider}' st exceeds {MAX_PROVIDER_STATE_BYTES} bytes" + )); + } } if let Some(pub_properties) = &self.pub_properties { @@ -460,10 +596,156 @@ impl KvGeo { } } +/// Serialized JSON length of a value. A value that cannot serialize counts +/// as unbounded, so it always fails a size check. +fn serialized_len(value: &serde_json::Value) -> usize { + serde_json::to_vec(value).map_or(usize::MAX, |bytes| bytes.len()) +} + +/// Whether `value` is a record revision: 16 lowercase hex digits. +fn is_record_revision(value: &str) -> bool { + value.len() == RECORD_REVISION_HEX_LEN + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + #[cfg(test)] mod tests { use super::*; + #[test] + fn legacy_partner_record_reads_and_writes_unchanged() { + let legacy = serde_json::json!({ "uid": "example-uid" }); + let record: KvPartnerId = + serde_json::from_value(legacy.clone()).expect("should read legacy"); + assert_eq!( + record, + KvPartnerId::new("example-uid"), + "should fill every new field with None" + ); + assert_eq!( + serde_json::to_value(&record).expect("should write"), + legacy, + "should add no fields" + ); + } + + #[test] + fn provider_record_and_state_round_trip() { + let stored = serde_json::json!({ + "v": 1, + "created": 1_760_000_000, + "consent": { "ok": true, "updated": 1_760_000_000 }, + "geo": { "country": "US" }, + "ids": { + "id5-sync.com": { + "uid": "ID5*example", + "w": "id5", + "rv": "9c1d04b7e6a83f2a", + "at": 1_760_000_000, + "exp": 1_767_776_000, + "ra": 1_760_086_400, + "ru": 1_770_000_000, + "atype": 1, + "ext": { "linkType": 2 } + } + }, + "ps": { + "id5": { + "n": 1_760_003_600, + "f": 1, + "o": { "at": 1_759_000_000, "fp": "0123456789abcdef" }, + "oc": 1_759_500_000, + "iv": ["aaaaaaaaaaaaaaaa"], + "st": { "signature": "example-opaque-state" } + } + } + }); + + let entry: KvEntry = serde_json::from_value(stored.clone()).expect("should read"); + entry.validate().expect("should be valid"); + assert_eq!( + serde_json::to_value(&entry).expect("should write"), + stored, + "should round-trip exactly" + ); + } + + #[test] + fn validate_rejects_oversized_ext_state_rv_and_too_many_providers() { + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ids.get_mut("example.com").expect("should exist").ext = + Some(serde_json::json!({ "pad": "x".repeat(MAX_PARTNER_EXT_BYTES) })); + assert!( + entry + .validate() + .expect_err("should reject ext") + .contains("ext"), + "should name ext" + ); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ps.insert( + "id5".to_owned(), + KvProviderState { + st: Some(serde_json::json!("x".repeat(MAX_PROVIDER_STATE_BYTES))), + ..KvProviderState::default() + }, + ); + assert!( + entry + .validate() + .expect_err("should reject st") + .contains("st"), + "should name st" + ); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + for index in 0..=MAX_PROVIDER_STATES { + entry + .ps + .insert(format!("p{index}"), KvProviderState::default()); + } + assert!( + entry + .validate() + .expect_err("should reject ps size") + .contains("ps"), + "should name ps" + ); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ps.insert( + "id5".to_owned(), + KvProviderState { + iv: vec!["a".repeat(16); MAX_INVALIDATED_UIDS + 1], + ..KvProviderState::default() + }, + ); + assert!( + entry + .validate() + .expect_err("should reject iv") + .contains("iv"), + "should name iv" + ); + + let mut entry = KvEntry::minimal("example.com", "example-uid", 1); + entry.ids.get_mut("example.com").expect("should exist").rv = Some("not-hex".to_owned()); + assert!( + entry + .validate() + .expect_err("should reject rv") + .contains("rv"), + "should name rv" + ); + + KvEntry::minimal("example.com", "example-uid", 1) + .validate() + .expect("control entry should be valid"); + } + fn sample_consent_context() -> ConsentContext { ConsentContext { raw_tc_string: Some("CP_test_tc_string".to_owned()), @@ -490,12 +772,9 @@ mod tests { let geo = sample_geo_info(); let consent = sample_consent_context(); let mut entry = KvEntry::new(&consent, Some(&geo), 1_741_824_000, "example.com"); - entry.ids.insert( - "liveramp".to_owned(), - KvPartnerId { - uid: "LR_xyz".to_owned(), - }, - ); + entry + .ids + .insert("liveramp".to_owned(), KvPartnerId::new("LR_xyz".to_owned())); let json = serde_json::to_string(&entry).expect("should serialize KvEntry"); let deserialized: KvEntry = @@ -737,9 +1016,7 @@ mod tests { let mut entry = KvEntry::tombstone(1000); entry.ids.insert( "ssp_x".to_owned(), - KvPartnerId { - uid: "x".repeat(MAX_UID_LENGTH + 1), - }, + KvPartnerId::new("x".repeat(MAX_UID_LENGTH + 1)), ); let err = entry diff --git a/crates/trusted-server-core/src/ec/pull_sync.rs b/crates/trusted-server-core/src/ec/pull_sync.rs index 191e6c6d24..911600df9b 100644 --- a/crates/trusted-server-core/src/ec/pull_sync.rs +++ b/crates/trusted-server-core/src/ec/pull_sync.rs @@ -652,9 +652,7 @@ mod tests { if let EcKvSnapshot::Present { entry, .. } = &mut snapshot { entry.ids.insert( "ssp.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "uid".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("uid".to_owned()), ); } ec_context.set_kv_snapshot(snapshot); @@ -702,9 +700,7 @@ mod tests { entry.ids.insert( "b.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "uid-b".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("uid-b".to_owned()), ); assert!( @@ -1213,9 +1209,7 @@ mod tests { entry.consent.ok = true; entry.ids.insert( "alpha.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "already-known".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("already-known".to_owned()), ); graph .create(&ec_id, &entry) diff --git a/crates/trusted-server-core/src/ec/pull_sync_marker.rs b/crates/trusted-server-core/src/ec/pull_sync_marker.rs index f2a2fe6013..32ce1dfb00 100644 --- a/crates/trusted-server-core/src/ec/pull_sync_marker.rs +++ b/crates/trusted-server-core/src/ec/pull_sync_marker.rs @@ -375,9 +375,7 @@ mod tests { for domain in domains { entry.ids.insert( (*domain).to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: format!("uid-{domain}"), - }, + crate::ec::kv_types::KvPartnerId::new(format!("uid-{domain}")), ); } EcKvSnapshot::Present { @@ -510,9 +508,7 @@ mod tests { let mut tombstone = KvEntry::tombstone(1_000); tombstone.ids.insert( "a.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId { - uid: "stale".to_owned(), - }, + crate::ec::kv_types::KvPartnerId::new("stale".to_owned()), ); let snapshots = [ live_snapshot(EC_ID, &[]), diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 7a327ec372..ff8e8ec29c 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -7929,9 +7929,7 @@ mod tests { let mut entry = KvEntry::minimal("ids.example.com", "kv-stale", 1000); entry.ids.insert( "other.example.com".to_owned(), - KvPartnerId { - uid: "kv-other".to_owned(), - }, + KvPartnerId::new("kv-other".to_owned()), ); let snapshot = EcKvSnapshot::Present { ec_id: ec_id.clone(), From 4c3103bad5d62c62a77bb9083012715bb5f7fe9a Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:49:44 +0530 Subject: [PATCH 08/33] Record when browser ingest acquired each ID Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/ec/finalize.rs | 21 ++++- crates/trusted-server-core/src/ec/kv.rs | 84 +++++++++++++++---- 2 files changed, 85 insertions(+), 20 deletions(-) diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index 5612d821c7..c058afcee5 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -88,7 +88,8 @@ pub fn ec_finalize_response( }) .unwrap_or_default(); let had_eid_updates = !collected.is_empty(); - let updates = gate_eid_updates_by_consent(collected, ec_context.consent()); + let updates = + stamp_acquired_at(gate_eid_updates_by_consent(collected, ec_context.consent())); // `sync_eid_cookie_updates` early-returns without refreshing the // snapshot when `updates` is empty, which is correct when there // was never anything to write (an unconfigured registry, no sync @@ -137,10 +138,10 @@ pub fn ec_finalize_response( }; let (cookie_input, client_input) = browser_eid_input(ec_context, eids_cookie); - let updates = gate_eid_updates_by_consent( + let updates = stamp_acquired_at(gate_eid_updates_by_consent( collect_eid_updates(cookie_input, sharedid_cookie, client_input, registry), ec_context.consent(), - ); + )); sync_eid_cookie_updates(graph, ec_context, &ec_id, &updates, EidSyncSource::NewEc); if ec_context.kv_snapshot().entry_for(&ec_id).is_some() { set_ec_cookie_on_response(settings, ec_context, response); @@ -152,6 +153,20 @@ pub fn ec_finalize_response( reconcile_pull_sync_marker(settings, registry, ec_context, response); } +/// Records when this request saw each browser ID, so ingest stores `at`. +/// +/// Without a trustworthy clock the updates are left unstamped rather than +/// stamped with zero. +fn stamp_acquired_at(updates: Vec) -> Vec { + let Some(now) = super::checked_current_timestamp() else { + return updates; + }; + updates + .into_iter() + .map(|update| update.with_acquired_at(now)) + .collect() +} + /// Chooses the browser EID input KV ingestion reads for this request. /// /// `/auction` ingests only the client set the auction itself used (its body, diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index 9d0ff4811a..fe8b252683 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -21,7 +21,7 @@ use crate::error::TrustedServerError; use super::generation::ec_hash; use super::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome}; -use super::kv_types::{KvEntry, KvMetadata, KvNetwork}; +use super::kv_types::{KvEntry, KvMetadata, KvNetwork, KvPartnerId}; use super::{EcKvSnapshot, checked_current_timestamp, current_timestamp, log_id}; /// Maximum number of CAS retry attempts before giving up. @@ -77,6 +77,8 @@ pub(crate) struct PartnerIdUpdate { pub(crate) partner_id: String, /// Partner-scoped user ID value. pub(crate) uid: String, + /// Unix seconds this request saw the ID. Stored as the record's `at`. + pub(crate) acquired_at: Option, } impl PartnerIdUpdate { @@ -85,8 +87,16 @@ impl PartnerIdUpdate { Self { partner_id: partner_id.into(), uid: uid.into(), + acquired_at: None, } } + + /// Records when this request saw the ID. + #[must_use] + pub(crate) fn with_acquired_at(mut self, at: u64) -> Self { + self.acquired_at = Some(at); + self + } } /// Terminal result of one browser EID-cookie persistence attempt. @@ -137,14 +147,14 @@ fn apply_cookie_partner_id_updates( ) -> CookieUpdateApplication { let mut latest_updates = BTreeMap::new(); for update in updates { - latest_updates.insert(update.partner_id.as_str(), update.uid.as_str()); + latest_updates.insert(update.partner_id.as_str(), update); } let mut changed = false; let mut deferred = false; - for (partner_id, uid) in latest_updates { + for (partner_id, update) in latest_updates { match entry.ids.get(partner_id) { - Some(existing) if existing.uid == uid => continue, + Some(existing) if existing.uid == update.uid => continue, // Browser EID cookies carry no value-owned sequence or timestamp. // A different value therefore has unknown freshness and must not // replace the stored value. @@ -157,7 +167,7 @@ fn apply_cookie_partner_id_updates( entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId::new(uid.to_owned()), + KvPartnerId::browser(update.uid.clone(), update.acquired_at), ); changed = true; } @@ -188,22 +198,22 @@ fn partner_id_updates_match(entry: &KvEntry, updates: &[PartnerIdUpdate]) -> boo pub(crate) fn apply_partner_id_updates(entry: &mut KvEntry, updates: &[PartnerIdUpdate]) -> bool { let mut latest_updates = BTreeMap::new(); for update in updates { - latest_updates.insert(update.partner_id.as_str(), update.uid.as_str()); + latest_updates.insert(update.partner_id.as_str(), update); } let mut changed = false; - for (partner_id, uid) in latest_updates { + for (partner_id, update) in latest_updates { if entry .ids .get(partner_id) - .is_some_and(|existing| existing.uid == uid) + .is_some_and(|existing| existing.uid == update.uid) { continue; } entry.ids.insert( partner_id.to_owned(), - super::kv_types::KvPartnerId::new(uid.to_owned()), + KvPartnerId::browser(update.uid.clone(), update.acquired_at), ); changed = true; } @@ -914,10 +924,9 @@ impl KvIdentityGraph { return Ok(()); } - entry.ids.insert( - partner_id.to_owned(), - super::kv_types::KvPartnerId::new(uid.to_owned()), - ); + entry + .ids + .insert(partner_id.to_owned(), KvPartnerId::new(uid.to_owned())); let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; @@ -984,10 +993,9 @@ impl KvIdentityGraph { return Ok(UpsertResult::Unchanged); } - entry.ids.insert( - partner_id.to_owned(), - super::kv_types::KvPartnerId::new(uid.to_owned()), - ); + entry + .ids + .insert(partner_id.to_owned(), KvPartnerId::new(uid.to_owned())); let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; @@ -1612,6 +1620,48 @@ mod tests { use super::*; use crate::ec::kv_backend::test_support::InMemoryEcKv; + #[test] + fn browser_ingest_records_acquisition_time() { + let mut entry = KvEntry::minimal("sharedid.org", "existing", 1); + let updates = [ + PartnerIdUpdate::new("example.com", "new-uid").with_acquired_at(1_760_000_000), + PartnerIdUpdate::new("sharedid.org", "existing").with_acquired_at(1_760_000_000), + ]; + + let changed = apply_partner_id_updates(&mut entry, &updates); + + assert!(changed, "should report a change"); + assert_eq!( + entry.ids["example.com"], + KvPartnerId::browser("new-uid", Some(1_760_000_000)), + "should store the UID with at and no writer" + ); + assert_eq!( + entry.ids["sharedid.org"], + KvPartnerId::new("existing"), + "should not rewrite an unchanged UID just to bump at" + ); + } + + #[test] + fn cookie_ingest_records_acquisition_time() { + let mut entry = KvEntry::minimal("sharedid.org", "existing", 1); + let updates = [PartnerIdUpdate::new("example.com", "new-uid").with_acquired_at(7)]; + + let outcome = apply_cookie_partner_id_updates(&mut entry, &updates); + + assert_eq!( + outcome, + CookieUpdateApplication::Changed { deferred: false }, + "should change" + ); + assert_eq!( + entry.ids["example.com"], + KvPartnerId::browser("new-uid", Some(7)), + "should store at" + ); + } + /// [`EcKvStore`] wrapper whose first CAS write both fails the precondition /// and deletes the key, simulating a concurrent withdrawal that removes the /// row between this writer's read and its write. From a9de29180e06630d1ae6fcb158813e0a19820df7 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:52:23 +0530 Subject: [PATCH 09/33] Add the identity provider descriptor and its builder Signed-off-by: dhruv8sh --- .../src/ec/identity/descriptor.rs | 917 ++++++++++++++++++ .../src/ec/identity/mod.rs | 12 +- 2 files changed, 928 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-core/src/ec/identity/descriptor.rs b/crates/trusted-server-core/src/ec/identity/descriptor.rs index d1e92afa73..145c898a86 100644 --- a/crates/trusted-server-core/src/ec/identity/descriptor.rs +++ b/crates/trusted-server-core/src/ec/identity/descriptor.rs @@ -1,5 +1,667 @@ //! What a provider is, may write, needs and must respect. +use std::collections::{BTreeMap, BTreeSet}; + +use error_stack::Report; +use http::Method; +use url::Url; + +use super::ids::{IdentityProviderId, SourceDomain}; +use super::resolution::IdentityProviderError; +use crate::ec::partner::normalize_partner_source_domain; + +/// What a provider is, may write, needs and must respect. +/// +/// Built from config with no secrets, so `ts config validate` and the running +/// coordinator read the same value. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ProviderDescriptor { + id: IdentityProviderId, + sources: Vec, + resolves: Option, + endpoint: Option, + captures: Option, + consent_policy: ConsentPolicy, + provenance: BTreeMap, + expiring: BTreeSet, +} + +impl ProviderDescriptor { + /// Starts a descriptor for `id`. + #[must_use] + pub fn builder(id: IdentityProviderId) -> ProviderDescriptorBuilder { + ProviderDescriptorBuilder { + id, + sources: Vec::new(), + resolves: None, + captures: None, + consent_policy: ConsentPolicy::strict(), + provenance: BTreeMap::new(), + expiring_all: false, + expiring: BTreeSet::new(), + } + } + + /// The provider's ID. + #[must_use] + pub fn id(&self) -> &IdentityProviderId { + &self.id + } + + /// The sources this provider may write, in declaration order. + #[must_use] + pub fn sources(&self) -> &[SourceDomain] { + &self.sources + } + + /// The first source added. Every built descriptor has one. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + &self.sources[0] + } + + /// Finds a declared source by a vendor's spelling of it, normalized. + #[must_use] + pub fn source_named(&self, name: &str) -> Option<&SourceDomain> { + self.sources.iter().find(|source| source.matches(name)) + } + + /// Active resolution, when declared. + #[must_use] + pub fn resolves(&self) -> Option<&Resolves> { + self.resolves.as_ref() + } + + /// The endpoint declared with [`Resolves::with_endpoint`], if any. + #[must_use] + pub fn endpoint(&self) -> Option<&Url> { + self.endpoint.as_ref() + } + + /// Passive capture, when declared. + #[must_use] + pub fn captures(&self) -> Option<&Captures> { + self.captures.as_ref() + } + + /// The consent this provider requires. + #[must_use] + pub fn consent_policy(&self) -> &ConsentPolicy { + &self.consent_policy + } + + /// EID-level fields for `source`, if declared. + #[must_use] + pub fn provenance(&self, source: &SourceDomain) -> Option<&EidProvenance> { + self.provenance.get(source) + } + + /// Whether IDs of `source` are declared as expiring. + #[must_use] + pub fn is_expiring(&self, source: &SourceDomain) -> bool { + self.expiring.contains(source) + } +} + +/// Builds a [`ProviderDescriptor`]. +#[derive(Debug, Clone)] +pub struct ProviderDescriptorBuilder { + id: IdentityProviderId, + sources: Vec, + resolves: Option, + captures: Option, + consent_policy: ConsentPolicy, + provenance: BTreeMap, + expiring_all: bool, + expiring: BTreeSet, +} + +impl ProviderDescriptorBuilder { + /// Adds a source this provider may write. Repeatable. A repeat is ignored. + #[must_use] + pub fn with_source(mut self, source: SourceDomain) -> Self { + if !self.sources.contains(&source) { + self.sources.push(source); + } + self + } + + /// Declares active resolution. + #[must_use] + pub fn with_resolves(mut self, resolves: Resolves) -> Self { + self.resolves = Some(resolves); + self + } + + /// Declares passive capture. + #[must_use] + pub fn with_captures(mut self, captures: Captures) -> Self { + self.captures = Some(captures); + self + } + + /// Replaces the default policy, [`ConsentPolicy::strict`]. + #[must_use] + pub fn with_consent_policy(mut self, policy: ConsentPolicy) -> Self { + self.consent_policy = policy; + self + } + + /// Marks the IDs of every declared source as expiring. + #[must_use] + pub fn with_expiring_ids(mut self) -> Self { + self.expiring_all = true; + self + } + + /// Marks the IDs of one declared source as expiring, for a provider whose + /// sources differ. + #[must_use] + pub fn with_expiring_source(mut self, source: &SourceDomain) -> Self { + self.expiring.insert(source.clone()); + self + } + + /// EID-level fields for a declared source. The coordinator adds them to + /// every server-side value of that source this provider wrote. No KV + /// field stores them. + #[must_use] + pub fn with_provenance(mut self, source: &SourceDomain, provenance: EidProvenance) -> Self { + self.provenance.insert(source.clone(), provenance); + self + } + + /// Validates and builds the descriptor. + /// + /// # Errors + /// + /// Returns `Config` when the descriptor has no source, no capability, a + /// resolver with no endpoint or host, an endpoint that is not an `https` + /// URL, a host that is not a bare lowercase hostname, or provenance or an + /// expiring mark for a source it does not declare. + pub fn build(self) -> Result> { + let id = self.id; + let fail = |what: &str| IdentityProviderError::config(format!("provider `{id}`: {what}")); + + if self.sources.is_empty() { + return Err(fail("declares no source")); + } + if self.resolves.is_none() && self.captures.is_none() { + return Err(fail("declares neither resolves nor captures")); + } + for source in self.provenance.keys().chain(self.expiring.iter()) { + if !self.sources.contains(source) { + return Err(fail(&format!("names undeclared source `{source}`"))); + } + } + + let mut endpoint = None; + let mut resolves = self.resolves; + if let Some(resolves) = resolves.as_mut() { + if let Some(raw) = resolves.raw_endpoint.as_deref() { + let url = Url::parse(raw).map_err(|_| fail("endpoint is not a URL"))?; + if url.scheme() != "https" { + return Err(fail("endpoint must use https")); + } + let host = url + .host_str() + .ok_or_else(|| fail("endpoint has no host"))? + .to_owned(); + if !resolves.hosts.contains(&host) { + resolves.hosts.insert(0, host); + } + endpoint = Some(url); + } + if resolves.hosts.is_empty() { + return Err(fail("resolves declares no endpoint or host")); + } + for host in &resolves.hosts { + // The partner-source normalizer accepts exactly a bare + // hostname; comparing its output also rejects upper case. + if normalize_partner_source_domain(host).ok().as_deref() != Some(host.as_str()) { + return Err(fail(&format!( + "host `{host}` is not a bare lowercase hostname" + ))); + } + } + } + + let expiring = if self.expiring_all { + self.sources.iter().cloned().collect() + } else { + self.expiring + }; + + Ok(ProviderDescriptor { + id, + sources: self.sources, + resolves, + endpoint, + captures: self.captures, + consent_policy: self.consent_policy, + provenance: self.provenance, + expiring, + }) + } +} + +/// The kind of resolution one source needs. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum ResolveKind { + /// No stored record for the source. + #[display("initial")] + Initial, + /// A stored record exists, valid or expired, and is due. + #[display("refresh")] + Refresh, +} + +/// Active resolution: the provider calls an endpoint. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Resolves { + raw_endpoint: Option, + hosts: Vec, + needs: BTreeSet, + needs_for: BTreeMap>, + wants: BTreeSet, + kinds: Vec, + private_state: bool, + ec_id: bool, +} + +impl Default for Resolves { + fn default() -> Self { + Self::new() + } +} + +impl Resolves { + /// A resolver with no inputs, handling `Initial` and `Refresh`. + #[must_use] + pub fn new() -> Self { + Self { + raw_endpoint: None, + hosts: Vec::new(), + needs: BTreeSet::new(), + needs_for: BTreeMap::new(), + wants: BTreeSet::new(), + kinds: vec![ResolveKind::Initial, ResolveKind::Refresh], + private_state: false, + ec_id: false, + } + } + + /// The endpoint the provider calls, as config spells it. `build()` parses + /// it and requires `https`. Its host becomes an allowed host. + #[must_use] + pub fn with_endpoint(mut self, endpoint: &str) -> Self { + self.raw_endpoint = Some(endpoint.to_owned()); + self + } + + /// Adds another host the provider may call. Repeatable. + #[must_use] + pub fn with_host(mut self, host: &str) -> Self { + if !self.hosts.iter().any(|known| known == host) { + self.hosts.push(host.to_owned()); + } + self + } + + /// Inputs required for every kind. Without them the provider is skipped. + #[must_use] + pub fn with_needs(mut self, inputs: impl IntoIterator) -> Self { + self.needs.extend(inputs); + self + } + + /// Inputs required for one kind only. + #[must_use] + pub fn with_needs_for( + mut self, + kind: ResolveKind, + inputs: impl IntoIterator, + ) -> Self { + self.needs_for.entry(kind).or_default().extend(inputs); + self + } + + /// Inputs passed when available, never required. + #[must_use] + pub fn with_wants(mut self, inputs: impl IntoIterator) -> Self { + self.wants.extend(inputs); + self + } + + /// The kinds this provider handles. Default: `Initial` and `Refresh`. + #[must_use] + pub fn with_kinds(mut self, kinds: impl IntoIterator) -> Self { + self.kinds = kinds.into_iter().collect(); + self + } + + /// This provider keeps private state, so it runs only on the writer + /// request, where state can be stored. + #[must_use] + pub fn with_private_state(mut self) -> Self { + self.private_state = true; + self + } + + /// Gives the provider the full EC ID. Only generic pull may use this. + #[must_use] + pub(in crate::ec::identity) fn with_ec_id(mut self) -> Self { + self.ec_id = true; + self + } + + /// Hosts the provider may call. + #[must_use] + pub(crate) fn hosts(&self) -> &[String] { + &self.hosts + } + + /// Inputs required for `kind`: the common needs plus that kind's. + #[must_use] + pub(crate) fn needs(&self, kind: ResolveKind) -> BTreeSet { + let mut needs = self.needs.clone(); + if let Some(extra) = self.needs_for.get(&kind) { + needs.extend(extra.iter().copied()); + } + needs + } + + /// Every input declared for any kind, needed or wanted. + #[must_use] + pub(crate) fn declared_inputs(&self) -> BTreeSet { + let mut all = self.needs.clone(); + all.extend(self.wants.iter().copied()); + for extra in self.needs_for.values() { + all.extend(extra.iter().copied()); + } + all + } + + /// Inputs passed when available. + #[must_use] + pub(crate) fn wants(&self) -> &BTreeSet { + &self.wants + } + + /// The kinds this provider handles. + #[must_use] + pub(crate) fn kinds(&self) -> &[ResolveKind] { + &self.kinds + } + + /// Whether the provider keeps private state. + #[must_use] + pub(crate) fn has_private_state(&self) -> bool { + self.private_state + } + + /// Whether the provider receives the full EC ID. + #[must_use] + pub(crate) fn gets_ec_id(&self) -> bool { + self.ec_id + } +} + +/// Passive capture from responses TS proxies for this provider. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Captures { + paths: Vec<(Method, String)>, + prefixes: Vec, + consent_pointers: Vec<&'static str>, +} + +impl Captures { + /// No capture rules yet. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Offers proxied responses to exactly this method and path. Repeatable. + #[must_use] + pub fn with_path(mut self, method: Method, path: &str) -> Self { + self.paths.push((method, path.to_owned())); + self + } + + /// Offers proxied responses whose path starts with `prefix`, for any + /// method. Repeatable. + #[must_use] + pub fn with_path_prefix(mut self, prefix: &str) -> Self { + self.prefixes.push(prefix.to_owned()); + self + } + + /// JSON pointers to consent strings in the proxied request's JSON body + /// (TCF, GPP or US Privacy, detected by format). The coordinator extracts + /// only these fields and gates capture by the stricter of the cookie + /// consent and the body consent. The provider never sees the body. + #[must_use] + pub fn with_request_consent( + mut self, + pointers: impl IntoIterator, + ) -> Self { + self.consent_pointers.extend(pointers); + self + } + + /// Whether a proxied request matches a rule. + #[must_use] + pub(crate) fn matches(&self, method: &Method, path: &str) -> bool { + self.paths + .iter() + .any(|(rule_method, rule_path)| rule_method == method && rule_path == path) + || self + .prefixes + .iter() + .any(|prefix| path.starts_with(prefix.as_str())) + } + + /// Exact path rules. + #[must_use] + pub(crate) fn paths(&self) -> &[(Method, String)] { + &self.paths + } + + /// Prefix rules. + #[must_use] + pub(crate) fn prefixes(&self) -> &[String] { + &self.prefixes + } + + /// Declared body consent pointers. + #[must_use] + pub(crate) fn consent_pointers(&self) -> &[&'static str] { + &self.consent_pointers + } +} + +/// EID-level fields (`OpenRTB` 2.6 §3.2.27) that bidders read, such as who +/// inserted an ID and how it was matched. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct EidProvenance { + inserter: Option, + matcher: Option, + mm: Option, +} + +impl EidProvenance { + /// No fields set. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Sets `inserter`. + #[must_use] + pub fn with_inserter(mut self, inserter: &str) -> Self { + self.inserter = Some(inserter.to_owned()); + self + } + + /// Sets `matcher`. + #[must_use] + pub fn with_matcher(mut self, matcher: &str) -> Self { + self.matcher = Some(matcher.to_owned()); + self + } + + /// Sets `mm`. + #[must_use] + pub fn with_mm(mut self, mm: i32) -> Self { + self.mm = Some(mm); + self + } + + /// `inserter`, if set. + #[must_use] + pub fn inserter(&self) -> Option<&str> { + self.inserter.as_deref() + } + + /// `matcher`, if set. + #[must_use] + pub fn matcher(&self) -> Option<&str> { + self.matcher.as_deref() + } + + /// `mm`, if set. + #[must_use] + pub fn mm(&self) -> Option { + self.mm + } +} + +/// A US opt-out a policy can relax. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, derive_more::Display)] +#[non_exhaustive] +pub enum UsOptOut { + /// Sale of personal information. + #[display("sale")] + Sale, + /// Sharing for cross-context behavioural advertising. + #[display("sharing")] + Sharing, + /// Targeted advertising. + #[display("targeted_advertising")] + TargetedAdvertising, +} + +/// The consent a provider requires. The coordinator evaluates it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ConsentPolicy { + tcf_purposes: BTreeSet, + tcf_vendor: Option, + allow_dnt: bool, + allow_gpc: bool, + allowed_us_opt_outs: BTreeSet, + eid_persistence_only: bool, +} + +impl ConsentPolicy { + /// The default. Blocked when any consent signal present cannot be + /// decoded. Where TCF applies, requires Purposes 1 and 4. Blocked by + /// every US opt-out TS decodes, by `Sec-GPC: 1` and by `DNT: 1`. + #[must_use] + pub fn strict() -> Self { + Self { + tcf_purposes: [1, 4].into_iter().collect(), + tcf_vendor: None, + allow_dnt: false, + allow_gpc: false, + allowed_us_opt_outs: BTreeSet::new(), + eid_persistence_only: false, + } + } + + /// Also requires these TCF purposes, where TCF applies. + #[must_use] + pub fn with_tcf_purposes(mut self, purposes: impl IntoIterator) -> Self { + self.tcf_purposes.extend(purposes); + self + } + + /// Requires vendor consent for this IAB Global Vendor List ID. + #[must_use] + pub fn with_tcf_vendor(mut self, vendor_id: u16) -> Self { + self.tcf_vendor = Some(vendor_id); + self + } + + /// Does not block on `DNT: 1`. Needs a documented legal basis. + #[must_use] + pub fn allowing_dnt(mut self) -> Self { + self.allow_dnt = true; + self + } + + /// Does not block on GPC. Needs a documented legal basis. + #[must_use] + pub fn allowing_gpc(mut self) -> Self { + self.allow_gpc = true; + self + } + + /// Does not block on one US opt-out. Needs a documented legal basis. + #[must_use] + pub fn allowing_us_opt_out(mut self, opt_out: UsOptOut) -> Self { + self.allowed_us_opt_outs.insert(opt_out); + self + } + + /// Blocks exactly what pull sync blocks today: only the global EC and + /// EID-persistence gates. Generic pull's parity policy. + #[must_use] + pub(in crate::ec::identity) fn eid_persistence_only() -> Self { + Self { + eid_persistence_only: true, + ..Self::strict() + } + } + + /// Required TCF purposes. + #[must_use] + pub fn tcf_purposes(&self) -> &BTreeSet { + &self.tcf_purposes + } + + /// Required GVL vendor, if any. + #[must_use] + pub fn tcf_vendor(&self) -> Option { + self.tcf_vendor + } + + /// Whether `DNT: 1` is allowed. + #[must_use] + pub fn allows_dnt(&self) -> bool { + self.allow_dnt + } + + /// Whether GPC is allowed. + #[must_use] + pub fn allows_gpc(&self) -> bool { + self.allow_gpc + } + + /// Whether a US opt-out is allowed. + #[must_use] + pub fn allows_us_opt_out(&self, opt_out: UsOptOut) -> bool { + self.allowed_us_opt_outs.contains(&opt_out) + } + + /// Whether this is generic pull's parity policy. + #[must_use] + pub(crate) fn is_eid_persistence_only(&self) -> bool { + self.eid_persistence_only + } +} + /// An input a provider can ask for. /// /// Consent strings, the provider's stored IDs, the publisher domain, the @@ -39,3 +701,258 @@ pub enum Access { #[display("private_state")] PrivateState, } + +#[cfg(test)] +mod tests { + use http::Method; + use url::Url; + + use super::*; + + fn source(name: &str) -> SourceDomain { + name.parse().expect("should parse source") + } + + fn id(name: &str) -> IdentityProviderId { + name.parse().expect("should parse id") + } + + fn capture_rule() -> Captures { + Captures::new().with_path(Method::POST, "/integrations/p/api") + } + + #[test] + fn builds_the_documented_example() { + let descriptor = ProviderDescriptor::builder(id("exampleid")) + .with_source(source("exampleid.example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.exampleid.example.com/v1/fetch") + .with_needs([Input::ClientIp, Input::UserAgent]) + .with_wants([Input::PageUrl]), + ) + .with_consent_policy(ConsentPolicy::strict().with_tcf_vendor(9999)) + .build() + .expect("should build"); + + assert_eq!(descriptor.id().as_str(), "exampleid", "should keep the id"); + assert_eq!( + descriptor.primary_source(), + &source("exampleid.example.com"), + "should keep the source" + ); + assert_eq!( + descriptor.endpoint().map(Url::as_str), + Some("https://api.exampleid.example.com/v1/fetch"), + "should parse the endpoint" + ); + let resolves = descriptor.resolves().expect("should resolve"); + assert_eq!( + resolves.hosts(), + &["api.exampleid.example.com".to_owned()], + "should allow the endpoint host" + ); + assert_eq!( + resolves.kinds(), + &[ResolveKind::Initial, ResolveKind::Refresh], + "should default to Initial and Refresh" + ); + assert!(descriptor.captures().is_none(), "should not capture"); + assert!( + !descriptor.is_expiring(&source("exampleid.example.com")), + "should not expire by default" + ); + assert_eq!( + descriptor.consent_policy().tcf_vendor(), + Some(9999), + "should keep the vendor" + ); + assert_eq!( + descriptor + .consent_policy() + .tcf_purposes() + .iter() + .copied() + .collect::>(), + vec![1, 4], + "strict should require Purposes 1 and 4" + ); + } + + #[test] + fn build_rejects_invalid_descriptors() { + let cases: Vec<(&str, ProviderDescriptorBuilder)> = vec![ + ( + "no source", + ProviderDescriptor::builder(id("p")) + .with_resolves(Resolves::new().with_endpoint("https://a.example.com")), + ), + ( + "no capability", + ProviderDescriptor::builder(id("p")).with_source(source("example.com")), + ), + ( + "no endpoint or host", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new()), + ), + ( + "http endpoint", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new().with_endpoint("http://a.example.com")), + ), + ( + "bad host", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new().with_host("a.example.com/path")), + ), + ( + "upper-case host", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_resolves(Resolves::new().with_host("A.example.com")), + ), + ( + "provenance for undeclared source", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(capture_rule()) + .with_provenance( + &source("other.example.com"), + EidProvenance::new().with_mm(2), + ), + ), + ( + "expiring undeclared source", + ProviderDescriptor::builder(id("p")) + .with_source(source("example.com")) + .with_captures(capture_rule()) + .with_expiring_source(&source("other.example.com")), + ), + ]; + for (name, builder) in cases { + let error = builder.build().expect_err(name); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ), + "should be a Config error: {name}" + ); + } + } + + #[test] + fn expiring_and_provenance_are_per_source() { + let a = source("a.example.com"); + let b = source("b.example.com"); + let descriptor = ProviderDescriptor::builder(id("p")) + .with_source(a.clone()) + .with_source(b.clone()) + .with_captures(capture_rule()) + .with_expiring_source(&b) + .with_provenance( + &a, + EidProvenance::new() + .with_inserter("example-inserter") + .with_mm(2), + ) + .build() + .expect("should build"); + + assert!(!descriptor.is_expiring(&a), "a should not expire"); + assert!(descriptor.is_expiring(&b), "b should expire"); + let provenance = descriptor.provenance(&a).expect("a should have provenance"); + assert_eq!( + provenance.inserter(), + Some("example-inserter"), + "should keep the inserter" + ); + assert_eq!(provenance.matcher(), None, "should leave the matcher unset"); + assert_eq!(provenance.mm(), Some(2), "should keep mm"); + assert!(descriptor.provenance(&b).is_none(), "b should have none"); + assert_eq!( + descriptor.source_named("A.EXAMPLE.COM"), + Some(&a), + "should find a by any spelling" + ); + } + + #[test] + fn expiring_ids_marks_every_source() { + let descriptor = ProviderDescriptor::builder(id("p")) + .with_source(source("a.example.com")) + .with_source(source("b.example.com")) + .with_expiring_ids() + .with_captures(Captures::new().with_path_prefix("/integrations/p/")) + .build() + .expect("should build"); + assert!( + descriptor + .sources() + .iter() + .all(|source| descriptor.is_expiring(source)), + "should mark every source" + ); + } + + #[test] + fn captures_match_exact_paths_and_prefixes() { + let captures = Captures::new() + .with_path(Method::POST, "/integrations/p/api/token") + .with_path_prefix("/integrations/p/stream/"); + assert!( + captures.matches(&Method::POST, "/integrations/p/api/token"), + "should match the exact path" + ); + assert!( + !captures.matches(&Method::GET, "/integrations/p/api/token"), + "should require the method" + ); + assert!( + !captures.matches(&Method::POST, "/integrations/p/api/token/x"), + "exact means exact" + ); + assert!( + captures.matches(&Method::GET, "/integrations/p/stream/1"), + "prefix matches any method" + ); + assert!( + !captures.matches(&Method::GET, "/integrations/p/settings"), + "should not match others" + ); + } + + #[test] + fn needs_differ_by_kind() { + let resolves = Resolves::new() + .with_endpoint("https://a.example.com") + .with_needs([Input::ClientIp]) + .with_needs_for(ResolveKind::Initial, [Input::HashedEmail]); + assert_eq!( + resolves.needs(ResolveKind::Initial), + [Input::ClientIp, Input::HashedEmail].into_iter().collect(), + "Initial needs both" + ); + assert_eq!( + resolves.needs(ResolveKind::Refresh), + [Input::ClientIp].into_iter().collect(), + "Refresh needs only the IP" + ); + } + + #[test] + fn eid_persistence_only_is_crate_private_parity_policy() { + assert!( + ConsentPolicy::eid_persistence_only().is_eid_persistence_only(), + "parity policy" + ); + assert!( + !ConsentPolicy::strict().is_eid_persistence_only(), + "control: strict is not" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 398b542b43..532d41dd83 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -6,10 +6,20 @@ //! KV write-back, so every page load makes at most one browser-driven KV //! write. +// Several crate-private helpers are only called once the coordinator is wired +// into requests later in this series. Remove this allow when that lands. +#![allow( + dead_code, + reason = "crate-private identity helpers are wired into requests by later commits" +)] + pub mod descriptor; pub mod ids; pub mod resolution; -pub use descriptor::{Access, Input}; +pub use descriptor::{ + Access, Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, + ProviderDescriptorBuilder, ResolveKind, Resolves, UsOptOut, +}; pub use ids::{IdentityProviderId, SourceDomain, UnixTime}; pub use resolution::IdentityProviderError; From 587fb9ea03b08c2d4a56b94817dc928399ea7a80 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:55:38 +0530 Subject: [PATCH 10/33] Add declared-input enforcement for identity providers Signed-off-by: dhruv8sh --- .../src/ec/identity/input.rs | 845 ++++++++++++++++++ .../src/ec/identity/mod.rs | 2 + .../src/migration_guards.rs | 1 + 3 files changed, 848 insertions(+) create mode 100644 crates/trusted-server-core/src/ec/identity/input.rs diff --git a/crates/trusted-server-core/src/ec/identity/input.rs b/crates/trusted-server-core/src/ec/identity/input.rs new file mode 100644 index 0000000000..80dddc47ac --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/input.rs @@ -0,0 +1,845 @@ +//! What a provider may read for one call. + +use core::cell::RefCell; +use core::fmt; +use std::collections::{BTreeMap, BTreeSet}; +use std::net::IpAddr; + +use base64::Engine as _; +use base64::engine::general_purpose::STANDARD; +use error_stack::Report; +use rand::RngCore as _; +use serde::de::DeserializeOwned; +use url::Url; + +use super::descriptor::{Access, Input, ProviderDescriptor, ResolveKind}; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::resolution::IdentityProviderError; +use crate::consent::ConsentContext; +use crate::ec::kv_types::KvPartnerId; + +/// The raw consent strings a provider forwards, and `gdpr_applies`. +/// +/// Providers never decide consent. The coordinator has already applied the +/// descriptor's policy. +#[derive(Clone, Default, PartialEq, Eq)] +pub struct ProviderConsent { + gdpr_applies: bool, + tc_string: Option, + gpp_string: Option, + gpp_section_ids: Option>, + us_privacy: Option, +} + +impl fmt::Debug for ProviderConsent { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderConsent") + .field("gdpr_applies", &self.gdpr_applies) + .field("tc_string", &self.tc_string.as_ref().map(|_| "[redacted]")) + .field( + "gpp_string", + &self.gpp_string.as_ref().map(|_| "[redacted]"), + ) + .field("gpp_section_ids", &self.gpp_section_ids) + .field( + "us_privacy", + &self.us_privacy.as_ref().map(|_| "[redacted]"), + ) + .finish() + } +} + +impl ProviderConsent { + /// The view of a request's consent context. + #[must_use] + pub(crate) fn from_context(context: &ConsentContext) -> Self { + Self { + gdpr_applies: context.gdpr_applies, + tc_string: context.raw_tc_string.clone(), + gpp_string: context.raw_gpp_string.clone(), + gpp_section_ids: context.gpp_section_ids.clone(), + us_privacy: context.raw_us_privacy.clone(), + } + } + + /// Whether GDPR applies to this request. + #[must_use] + pub fn gdpr_applies(&self) -> bool { + self.gdpr_applies + } + + /// The raw TCF string. + #[must_use] + pub fn raw_tc_string(&self) -> Option<&str> { + self.tc_string.as_deref() + } + + /// The raw GPP string. + #[must_use] + pub fn raw_gpp_string(&self) -> Option<&str> { + self.gpp_string.as_deref() + } + + /// The applicable GPP section IDs. + #[must_use] + pub fn gpp_section_ids(&self) -> Option<&[u16]> { + self.gpp_section_ids.as_deref() + } + + /// The raw US Privacy string. + #[must_use] + pub fn raw_us_privacy(&self) -> Option<&str> { + self.us_privacy.as_deref() + } +} + +/// Country and subdivision, upper case. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Region { + country: String, + subdivision: Option, +} + +impl Region { + /// Normalizes platform geo: upper case, and an ISO 3166-2 `US-CA` style + /// region loses its country prefix. `None` without a two-letter country. + #[must_use] + pub(crate) fn from_geo(country: &str, region: Option<&str>) -> Option { + let country = country.trim().to_ascii_uppercase(); + if country.len() != 2 || !country.bytes().all(|byte| byte.is_ascii_alphabetic()) { + return None; + } + let subdivision = region + .map(|value| value.trim().to_ascii_uppercase()) + .map(|value| match value.split_once('-') { + Some((prefix, rest)) if prefix == country => rest.to_owned(), + _ => value, + }) + .filter(|value| { + !value.is_empty() && value.bytes().all(|byte| byte.is_ascii_alphanumeric()) + }); + Some(Self { + country, + subdivision, + }) + } + + /// ISO 3166-1 alpha-2, upper case, for example `"US"`. + #[must_use] + pub fn country(&self) -> &str { + &self.country + } + + /// The ISO 3166-2 subdivision without the country prefix, upper case, for + /// example `"CA"`. `None` when the platform does not know it. + #[must_use] + pub fn subdivision(&self) -> Option<&str> { + self.subdivision.as_deref() + } + + /// Whether this is `country` and `subdivision`. + #[must_use] + pub fn is(&self, country: &str, subdivision: &str) -> bool { + self.country == country && self.subdivision.as_deref() == Some(subdivision) + } +} + +/// The 32 raw bytes of a SHA-256 of a normalized email address. Personal +/// data: its `Debug` output is redacted. +#[derive(Clone, PartialEq, Eq)] +pub struct HashedEmail([u8; 32]); + +impl fmt::Debug for HashedEmail { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("HashedEmail([redacted])") + } +} + +impl HashedEmail { + /// From digest bytes. No code in v1 constructs one outside tests. + #[must_use] + pub(crate) fn from_bytes(bytes: [u8; 32]) -> Self { + Self(bytes) + } + + /// The digest. + #[must_use] + pub fn as_bytes(&self) -> &[u8; 32] { + &self.0 + } + + /// Lowercase hex, as `LiveRamp` takes it. + #[must_use] + pub fn to_hex(&self) -> String { + hex::encode(self.0) + } + + /// Standard base64, as UID2 takes it. + #[must_use] + pub fn to_base64(&self) -> String { + STANDARD.encode(self.0) + } +} + +/// A stored record for one of this provider's sources, valid or expired. +#[derive(Clone, PartialEq)] +pub struct StoredId { + uid: String, + atype: Option, + ext: Option, + writer: Option, + acquired_at: Option, + expires_at: Option, + refresh_at: Option, +} + +impl fmt::Debug for StoredId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("StoredId") + .field("uid", &"[redacted]") + .field("writer", &self.writer) + .field("acquired_at", &self.acquired_at) + .field("expires_at", &self.expires_at) + .field("refresh_at", &self.refresh_at) + .finish_non_exhaustive() + } +} + +impl StoredId { + /// From a KV record. A writer that does not parse is treated as none. + #[must_use] + pub(crate) fn from_record(record: &KvPartnerId) -> Self { + Self { + uid: record.uid.clone(), + atype: record.atype, + ext: record.ext.clone(), + writer: record.w.as_deref().and_then(|writer| writer.parse().ok()), + acquired_at: record.at.map(UnixTime::from_secs), + expires_at: record.exp.map(UnixTime::from_secs), + refresh_at: record.ra.map(UnixTime::from_secs), + } + } + + /// The UID. + #[must_use] + pub fn uid(&self) -> &str { + &self.uid + } + + /// Provider-supplied `atype`. + #[must_use] + pub fn atype(&self) -> Option { + self.atype + } + + /// UID-level ext, as bidders read it in `uids[].ext`. + #[must_use] + pub fn ext(&self) -> Option<&serde_json::Value> { + self.ext.as_ref() + } + + /// The provider that wrote it. `None` for browser ingest and legacy rows. + #[must_use] + pub fn writer(&self) -> Option<&IdentityProviderId> { + self.writer.as_ref() + } + + /// When it was acquired or last refreshed. + #[must_use] + pub fn acquired_at(&self) -> Option { + self.acquired_at + } + + /// When it must stop being sent. + #[must_use] + pub fn expires_at(&self) -> Option { + self.expires_at + } + + /// When it is due for refresh. + #[must_use] + pub fn refresh_at(&self) -> Option { + self.refresh_at + } + + /// Whether `now` is at or past its expiry. + #[must_use] + pub fn is_expired(&self, now: UnixTime) -> bool { + self.expires_at.is_some_and(|expires| now >= expires) + } +} + +/// A source of random bytes. +pub(crate) trait RandomSource: Send + Sync { + /// Fills `bytes`. + fn fill(&self, bytes: &mut [u8]); +} + +/// The platform generator EC generation uses (`rand::thread_rng`). +pub(crate) struct ThreadRandom; + +impl RandomSource for ThreadRandom { + fn fill(&self, bytes: &mut [u8]) { + rand::thread_rng().fill_bytes(bytes); + } +} + +/// Deterministic bytes for tests: xorshift64* from a seed. +#[cfg(any(test, feature = "test-utils"))] +pub(crate) struct SeededRandom(std::sync::Mutex); + +#[cfg(any(test, feature = "test-utils"))] +impl SeededRandom { + /// A generator seeded with `seed`. A zero seed is replaced by one, + /// because xorshift never leaves zero. + pub(crate) fn new(seed: u64) -> Self { + Self(std::sync::Mutex::new(seed.max(1))) + } +} + +#[cfg(any(test, feature = "test-utils"))] +impl RandomSource for SeededRandom { + fn fill(&self, bytes: &mut [u8]) { + let mut state = self.0.lock().expect("should lock the seed"); + for byte in bytes { + *state ^= *state >> 12; + *state ^= *state << 25; + *state ^= *state >> 27; + let [top, ..] = state.wrapping_mul(0x2545_F491_4F6C_DD1D).to_be_bytes(); + *byte = top; + } + } +} + +/// Every signal the coordinator holds for one request. Crate-private: a +/// provider only ever sees a [`ResolveInput`] filtered by its descriptor. +pub(crate) struct ResolveSignals<'a> { + pub(crate) client_ip: Option, + pub(crate) user_agent: Option<&'a str>, + pub(crate) page_url: Option, + pub(crate) region: Option, + pub(crate) hashed_email: Option, + pub(crate) consent: ProviderConsent, + pub(crate) publisher_domain: &'a str, + pub(crate) now: UnixTime, + pub(crate) random: &'a dyn RandomSource, + pub(crate) ec_id: Option, + pub(crate) stored: BTreeMap, + pub(crate) state_json: Option, + pub(crate) dry_run: bool, +} + +/// What one provider may read for one call. +/// +/// Every accessor checks the descriptor. Reading something undeclared fails +/// with `UndeclaredInput` and is also recorded, so the coordinator and the +/// harness catch it even when the provider ignores the error with `.ok()`. +pub struct ResolveInput<'a> { + descriptor: &'a ProviderDescriptor, + kinds: BTreeMap, + signals: &'a ResolveSignals<'a>, + available: BTreeSet, + undeclared: RefCell>, + state_failed: RefCell, +} + +impl fmt::Debug for ResolveInput<'_> { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ResolveInput") + .field("provider", self.descriptor.id()) + .field("kinds", &self.kinds) + .finish_non_exhaustive() + } +} + +impl<'a> ResolveInput<'a> { + /// The input for one call. `kinds` holds only the sources to resolve. + #[must_use] + pub(crate) fn new( + descriptor: &'a ProviderDescriptor, + kinds: BTreeMap, + signals: &'a ResolveSignals<'a>, + ) -> Self { + let available = descriptor + .resolves() + .map(super::descriptor::Resolves::declared_inputs) + .unwrap_or_default(); + Self { + descriptor, + kinds, + signals, + available, + undeclared: RefCell::new(Vec::new()), + state_failed: RefCell::new(false), + } + } + + fn record(&self, access: Access) -> Report { + self.undeclared.borrow_mut().push(access); + Report::new(IdentityProviderError::UndeclaredInput { access }) + } + + fn declared( + &self, + input: Input, + value: Option, + ) -> Result> { + if !self.available.contains(&input) { + return Err(self.record(Access::Input(input))); + } + value.ok_or_else(|| Report::new(IdentityProviderError::InputUnavailable { input })) + } + + fn has_private_state(&self) -> bool { + self.descriptor + .resolves() + .is_some_and(super::descriptor::Resolves::has_private_state) + } + + /// Undeclared reads recorded so far. + #[must_use] + pub(crate) fn recorded_errors(&self) -> Vec { + self.undeclared.borrow().clone() + } + + /// Whether a `state()` call failed to decode. + #[must_use] + pub(crate) fn state_decode_failed(&self) -> bool { + *self.state_failed.borrow() + } + + /// The sources to resolve this time, each with its own kind. + pub fn sources(&self) -> impl Iterator + '_ { + self.kinds.iter().map(|(source, kind)| (source, *kind)) + } + + /// Whether `source` is being resolved for the first time. + #[must_use] + pub fn is_initial(&self, source: &SourceDomain) -> bool { + self.kinds.get(source) == Some(&ResolveKind::Initial) + } + + /// Whether `source` is being refreshed. + #[must_use] + pub fn is_refresh(&self, source: &SourceDomain) -> bool { + self.kinds.get(source) == Some(&ResolveKind::Refresh) + } + + /// The descriptor's primary source, for single-source providers. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + self.descriptor.primary_source() + } + + /// The stored record for one of this provider's sources, valid or + /// expired. + #[must_use] + pub fn stored(&self, source: &SourceDomain) -> Option<&StoredId> { + self.signals.stored.get(source) + } + + /// This provider's private state, decoded. `Ok(None)` when there is none. + /// + /// # Errors + /// + /// `InvalidState` when the stored state does not decode as `T`. The + /// failure is recorded, and the coordinator clears the stored state + /// whatever the provider returns. Without `with_private_state()` it is + /// `UndeclaredInput`. + pub fn state(&self) -> Result, Report> { + if !self.has_private_state() { + return Err(self.record(Access::PrivateState)); + } + let Some(raw) = self.signals.state_json.as_deref() else { + return Ok(None); + }; + serde_json::from_str(raw).map(Some).map_err(|_| { + *self.state_failed.borrow_mut() = true; + Report::new(IdentityProviderError::InvalidState) + }) + } + + /// True during the dry run, when the request is discarded unsent. Always + /// false for a provider declared `with_private_state()`, which gets no + /// dry run. + #[must_use] + pub fn is_dry_run(&self) -> bool { + self.signals.dry_run && !self.has_private_state() + } + + /// The endpoint declared with `Resolves::with_endpoint`. + /// + /// # Errors + /// + /// `UndeclaredInput` with [`Access::Endpoint`] when none was declared. + pub fn endpoint(&self) -> Result<&Url, Report> { + self.descriptor + .endpoint() + .ok_or_else(|| self.record(Access::Endpoint)) + } + + /// The consent strings to forward. + #[must_use] + pub fn consent(&self) -> &ProviderConsent { + &self.signals.consent + } + + /// The publisher's domain. + #[must_use] + pub fn publisher_domain(&self) -> &str { + self.signals.publisher_domain + } + + /// The request's time. + #[must_use] + pub fn now(&self) -> UnixTime { + self.signals.now + } + + /// Random bytes from the platform's generator. Fixed by the harness, so + /// tests are deterministic. + #[must_use] + pub fn random_bytes(&self) -> [u8; N] { + let mut bytes = [0u8; N]; + self.signals.random.fill(&mut bytes); + bytes + } + + /// The platform's client address. + /// + /// # Errors + /// + /// `UndeclaredInput` when not declared, `InputUnavailable` when missing. + pub fn client_ip(&self) -> Result> { + self.declared(Input::ClientIp, self.signals.client_ip) + } + + /// The `User-Agent`. + /// + /// # Errors + /// + /// `UndeclaredInput` when not declared, `InputUnavailable` when missing. + pub fn user_agent(&self) -> Result<&str, Report> { + self.declared(Input::UserAgent, self.signals.user_agent) + } + + /// The page URL, without query string or fragment. + /// + /// # Errors + /// + /// `UndeclaredInput` when not declared, `InputUnavailable` when missing. + pub fn page_url(&self) -> Result<&Url, Report> { + self.declared(Input::PageUrl, self.signals.page_url.as_ref()) + } + + /// Country and subdivision. + /// + /// # Errors + /// + /// `UndeclaredInput` when not declared, `InputUnavailable` when missing. + pub fn region(&self) -> Result<&Region, Report> { + self.declared(Input::Region, self.signals.region.as_ref()) + } + + /// A hashed email. Never available in v1. + /// + /// # Errors + /// + /// `UndeclaredInput` when not declared, `InputUnavailable` when missing. + pub fn hashed_email(&self) -> Result<&HashedEmail, Report> { + self.declared(Input::HashedEmail, self.signals.hashed_email.as_ref()) + } + + /// The full EC ID. Only for a resolver declared `with_ec_id()`. + /// + /// # Errors + /// + /// `UndeclaredInput` otherwise. `InputUnavailable` when the request has + /// no EC ID. + pub(in crate::ec::identity) fn ec_id(&self) -> Result<&str, Report> { + if !self + .descriptor + .resolves() + .is_some_and(super::descriptor::Resolves::gets_ec_id) + { + return Err(self.record(Access::PrivateState)); + } + self.signals.ec_id.as_deref().ok_or_else(|| { + Report::new(IdentityProviderError::InputUnavailable { + input: Input::ClientIp, + }) + }) + } +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + use std::net::IpAddr; + + use super::*; + use crate::ec::identity::descriptor::{ProviderDescriptor, Resolves}; + use crate::ec::kv_types::KvPartnerId; + + fn descriptor(resolves: Resolves) -> ProviderDescriptor { + ProviderDescriptor::builder("p".parse().expect("should parse")) + .with_source("example.com".parse().expect("should parse")) + .with_resolves(resolves) + .build() + .expect("should build") + } + + fn signals() -> ResolveSignals<'static> { + ResolveSignals { + client_ip: Some("192.0.2.10".parse().expect("should parse ip")), + user_agent: Some("ExampleBrowser/1.0"), + page_url: None, + region: Region::from_geo("us", Some("us-ca")), + hashed_email: None, + consent: ProviderConsent::default(), + publisher_domain: "www.example.com", + now: UnixTime::from_secs(1_760_000_000), + random: &ThreadRandom, + ec_id: Some(format!("{}.ABC123", "a".repeat(64))), + stored: BTreeMap::new(), + state_json: None, + dry_run: false, + } + } + + fn kinds( + descriptor: &ProviderDescriptor, + kind: ResolveKind, + ) -> BTreeMap { + descriptor + .sources() + .iter() + .map(|source| (source.clone(), kind)) + .collect() + } + + #[test] + fn declared_inputs_are_available_and_wanted_missing_ones_are_unavailable() { + let descriptor = descriptor( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_needs([Input::ClientIp, Input::Region]) + .with_wants([Input::PageUrl]), + ); + let signals = signals(); + let input = ResolveInput::new( + &descriptor, + kinds(&descriptor, ResolveKind::Initial), + &signals, + ); + + assert_eq!( + input.client_ip().expect("should read ip"), + "192.0.2.10".parse::().expect("should parse ip"), + "should return the client ip" + ); + let region = input.region().expect("should read region"); + assert_eq!( + (region.country(), region.subdivision()), + ("US", Some("CA")), + "should normalize the region" + ); + assert!(region.is("US", "CA"), "should test country and subdivision"); + let error = input.page_url().expect_err("should be unavailable"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::InputUnavailable { + input: Input::PageUrl + } + ), + "a declared but missing input is InputUnavailable" + ); + assert!( + input.recorded_errors().is_empty(), + "declared reads record nothing" + ); + assert!( + input.is_initial(input.primary_source()), + "should be initial" + ); + assert!( + !input.is_refresh(input.primary_source()), + "should not be refresh" + ); + } + + #[test] + fn undeclared_reads_are_recorded_even_when_ignored() { + let descriptor = descriptor(Resolves::new().with_endpoint("https://a.example.com")); + let signals = signals(); + let input = ResolveInput::new( + &descriptor, + kinds(&descriptor, ResolveKind::Initial), + &signals, + ); + + let ignored = input.user_agent().ok(); + let state = input.state::(); + + assert!(ignored.is_none(), "should not return an undeclared input"); + assert!(state.is_err(), "state needs with_private_state"); + assert_eq!( + input.recorded_errors(), + vec![Access::Input(Input::UserAgent), Access::PrivateState], + "should record both undeclared reads" + ); + input + .endpoint() + .expect("the declared endpoint is always available"); + } + + #[test] + fn state_decodes_and_a_bad_state_is_recorded() { + #[derive(serde::Deserialize)] + struct Typed { + signature: String, + } + + let descriptor = descriptor( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_private_state(), + ); + let mut good = signals(); + good.state_json = Some(r#"{"signature":"s"}"#.to_owned()); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Refresh), &good); + let state: Typed = input + .state() + .expect("should decode") + .expect("should be present"); + assert_eq!(state.signature, "s", "should decode the state"); + assert!( + !input.state_decode_failed(), + "a good state is not a failure" + ); + + let mut bad = signals(); + bad.state_json = Some(r#""a bare string""#.to_owned()); + let input = ResolveInput::new(&descriptor, kinds(&descriptor, ResolveKind::Refresh), &bad); + let error = input.state::().err().expect("should not decode"); + assert!( + matches!(error.current_context(), IdentityProviderError::InvalidState), + "should be InvalidState" + ); + assert!(input.state_decode_failed(), "should record the failure"); + } + + #[test] + fn dry_run_is_never_true_for_stateful_providers() { + let stateful = descriptor( + Resolves::new() + .with_endpoint("https://a.example.com") + .with_private_state(), + ); + let stateless = descriptor(Resolves::new().with_endpoint("https://a.example.com")); + let mut dry = signals(); + dry.dry_run = true; + assert!( + !ResolveInput::new(&stateful, kinds(&stateful, ResolveKind::Initial), &dry) + .is_dry_run(), + "stateful providers get no dry run" + ); + assert!( + ResolveInput::new(&stateless, kinds(&stateless, ResolveKind::Initial), &dry) + .is_dry_run(), + "stateless providers see the dry run" + ); + } + + #[test] + fn ec_id_is_only_for_providers_declared_with_ec_id() { + let generic = descriptor(Resolves::new().with_host("a.example.com").with_ec_id()); + let other = descriptor(Resolves::new().with_host("a.example.com")); + let signals = signals(); + ResolveInput::new(&generic, kinds(&generic, ResolveKind::Initial), &signals) + .ec_id() + .expect("generic pull reads the EC ID"); + let input = ResolveInput::new(&other, kinds(&other, ResolveKind::Initial), &signals); + assert!(input.ec_id().is_err(), "others may not"); + assert_eq!(input.recorded_errors().len(), 1, "should record the read"); + } + + #[test] + fn stored_ids_and_expiry() { + let record = KvPartnerId { + uid: "u".to_owned(), + w: Some("p".to_owned()), + rv: Some("0123456789abcdef".to_owned()), + at: Some(10), + exp: Some(20), + ra: Some(15), + ru: None, + atype: Some(3), + ext: None, + }; + let stored = StoredId::from_record(&record); + assert_eq!( + stored.writer().map(IdentityProviderId::as_str), + Some("p"), + "should parse the writer" + ); + assert!( + !stored.is_expired(UnixTime::from_secs(19)), + "not expired before exp" + ); + assert!(stored.is_expired(UnixTime::from_secs(20)), "expired at exp"); + assert_eq!( + stored.refresh_at(), + Some(UnixTime::from_secs(15)), + "should expose ra" + ); + assert_eq!(stored.atype(), Some(3), "should expose atype"); + assert!( + !format!("{stored:?}").contains("\"u\""), + "Debug should not print the UID" + ); + } + + #[test] + fn hashed_email_debug_is_redacted() { + let email = HashedEmail::from_bytes([0xab; 32]); + assert_eq!(email.to_hex(), "ab".repeat(32), "should hex-encode"); + assert_eq!( + format!("{email:?}"), + "HashedEmail([redacted])", + "should redact Debug" + ); + } + + #[test] + fn region_rejects_bad_countries_and_strips_prefix_only_for_own_country() { + assert!(Region::from_geo("USA", None).is_none(), "three letters"); + assert!(Region::from_geo("", None).is_none(), "empty"); + let foreign = Region::from_geo("US", Some("GB-ENG")).expect("should parse"); + assert_eq!( + foreign.subdivision(), + None, + "should drop a subdivision with another country's prefix" + ); + let plain = Region::from_geo("us", Some("ca")).expect("should parse"); + assert_eq!(plain.subdivision(), Some("CA"), "should upper-case"); + } + + #[test] + fn seeded_random_is_deterministic() { + let first = SeededRandom::new(9); + let second = SeededRandom::new(9); + let mut a = [0u8; 16]; + let mut b = [0u8; 16]; + first.fill(&mut a); + second.fill(&mut b); + assert_eq!(a, b, "same seed, same bytes"); + let mut c = [0u8; 16]; + first.fill(&mut c); + assert_ne!(a, c, "the stream advances"); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 532d41dd83..2a37fea23c 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -15,6 +15,7 @@ pub mod descriptor; pub mod ids; +pub mod input; pub mod resolution; pub use descriptor::{ @@ -22,4 +23,5 @@ pub use descriptor::{ ProviderDescriptorBuilder, ResolveKind, Resolves, UsOptOut, }; pub use ids::{IdentityProviderId, SourceDomain, UnixTime}; +pub use input::{HashedEmail, ProviderConsent, Region, ResolveInput, StoredId}; pub use resolution::IdentityProviderError; diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index d22b47c1b8..2c4c14698c 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -88,6 +88,7 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { include_str!("ec/identity/descriptor.rs"), ), ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), + ("ec/identity/input.rs", include_str!("ec/identity/input.rs")), ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), ( "ec/identity/resolution.rs", From 8fcc5d9fee75dade3ae5a95c9f0ad5b6fbdb9c33 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 16:59:10 +0530 Subject: [PATCH 11/33] Add provider requests, responses and resolutions Signed-off-by: dhruv8sh --- .../src/ec/identity/exchange.rs | 479 ++++++++++++++++++ .../src/ec/identity/input.rs | 135 +++++ .../src/ec/identity/mod.rs | 6 +- .../src/ec/identity/resolution.rs | 472 +++++++++++++++++ .../src/migration_guards.rs | 4 + 5 files changed, 1094 insertions(+), 2 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity/exchange.rs diff --git a/crates/trusted-server-core/src/ec/identity/exchange.rs b/crates/trusted-server-core/src/ec/identity/exchange.rs new file mode 100644 index 0000000000..705f8c867b --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/exchange.rs @@ -0,0 +1,479 @@ +//! What a provider sends and receives. The coordinator does all I/O. + +use core::fmt; + +use error_stack::Report; +use http::header::{AUTHORIZATION, CONTENT_TYPE, COOKIE}; +use http::{HeaderMap, HeaderName, HeaderValue, Method, StatusCode}; +use serde::Serialize; +use serde::de::DeserializeOwned; +use url::Url; + +use super::resolution::IdentityProviderError; +use crate::redacted::Redacted; + +/// Maximum JSON bytes of a call context. +pub const MAX_CALL_CONTEXT_BYTES: usize = 1024; + +/// One call a provider wants made. The coordinator checks and sends it. +#[derive(Clone)] +pub struct ProviderRequest { + method: Method, + url: Url, + headers: HeaderMap, + body: Vec, + call_context: Option, +} + +impl fmt::Debug for ProviderRequest { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderRequest") + .field("method", &self.method) + .field("host", &self.url.host_str()) + .field("path", &self.url.path()) + .field("headers", &self.headers.keys().collect::>()) + .field("body_len", &self.body.len()) + .finish_non_exhaustive() + } +} + +impl ProviderRequest { + /// Any method, with no body. `get`, `post_json` and `post_text` are + /// shorthands for this. + #[must_use] + pub fn new(method: Method, url: &Url) -> Self { + Self { + method, + url: url.clone(), + headers: HeaderMap::new(), + body: Vec::new(), + call_context: None, + } + } + + /// A `GET`. + #[must_use] + pub fn get(url: &Url) -> Self { + Self::new(Method::GET, url) + } + + /// A `POST` with a JSON body. + /// + /// # Errors + /// + /// `Config` when `body` does not serialize. + pub fn post_json( + url: &Url, + body: &impl Serialize, + ) -> Result> { + let bytes = serde_json::to_vec(body) + .map_err(|_| IdentityProviderError::config("request body does not serialize"))?; + Self::new(Method::POST, url).with_body("application/json", bytes) + } + + /// A `POST` with a `text/plain` body, for APIs that send base64 text. + #[must_use] + pub fn post_text(url: &Url, body: &str) -> Self { + let mut request = Self::new(Method::POST, url); + request + .headers + .insert(CONTENT_TYPE, HeaderValue::from_static("text/plain")); + request.body = body.as_bytes().to_vec(); + request + } + + /// Sets the body and its `Content-Type`. + /// + /// # Errors + /// + /// `Config` when `content_type` is not a valid header value. + pub fn with_body( + mut self, + content_type: &str, + body: Vec, + ) -> Result> { + let value = HeaderValue::from_str(content_type) + .map_err(|_| IdentityProviderError::config("invalid content type"))?; + self.headers.insert(CONTENT_TYPE, value); + self.body = body; + Ok(self) + } + + /// Appends a query parameter, URL-encoded. + #[must_use] + pub fn with_query(mut self, name: &str, value: &str) -> Self { + self.url.query_pairs_mut().append_pair(name, value); + self + } + + /// Sets a header. A `Cookie` header makes the coordinator refuse to send + /// the request. + #[must_use] + pub fn with_header(mut self, name: HeaderName, value: HeaderValue) -> Self { + self.headers.insert(name, value); + self + } + + /// Sets a header from strings, for dynamic values. A `Cookie` header + /// makes the coordinator refuse to send the request. + /// + /// # Errors + /// + /// `Config` when the name or value is invalid. + pub fn try_with_header( + self, + name: &str, + value: &str, + ) -> Result> { + let name = HeaderName::from_bytes(name.as_bytes()) + .map_err(|_| IdentityProviderError::config("invalid header name"))?; + let value = HeaderValue::from_str(value) + .map_err(|_| IdentityProviderError::config("invalid header value"))?; + Ok(self.with_header(name, value)) + } + + /// Adds `Authorization: Bearer `, marked sensitive. + #[must_use] + pub fn with_bearer(mut self, token: &Redacted) -> Self { + match HeaderValue::from_str(&format!("Bearer {}", token.expose())) { + Ok(mut value) => { + value.set_sensitive(true); + self.headers.insert(AUTHORIZATION, value); + } + Err(_) => log::warn!("Identity provider bearer token is not a valid header value"), + } + self + } + + /// Data `response` needs from this call, such as a nonce or which + /// endpoint was used. Kept in memory with the call, never sent. + /// + /// # Errors + /// + /// `Config` when it does not serialize, or is over 1,024 bytes of JSON. + pub fn try_with_call_context( + mut self, + context: &impl Serialize, + ) -> Result> { + let json = serde_json::to_string(context) + .map_err(|_| IdentityProviderError::config("call context does not serialize"))?; + if json.len() > MAX_CALL_CONTEXT_BYTES { + return Err(IdentityProviderError::config(format!( + "call context exceeds {MAX_CALL_CONTEXT_BYTES} bytes" + ))); + } + self.call_context = Some(json); + Ok(self) + } + + /// The method. + #[must_use] + pub fn method(&self) -> &Method { + &self.method + } + + /// The URL. + #[must_use] + pub fn url(&self) -> &Url { + &self.url + } + + /// The headers. + #[must_use] + pub fn headers(&self) -> &HeaderMap { + &self.headers + } + + /// The body. + #[must_use] + pub fn body(&self) -> &[u8] { + &self.body + } + + /// The call context JSON. + #[must_use] + pub(crate) fn call_context_json(&self) -> Option<&str> { + self.call_context.as_deref() + } + + /// Whether the provider set a `Cookie` header. + #[must_use] + pub(crate) fn has_cookie_header(&self) -> bool { + self.headers.contains_key(COOKIE) + } +} + +/// A buffered provider response, with `Content-Encoding` already undone. +#[derive(Clone)] +pub struct ProviderResponse { + status: StatusCode, + headers: HeaderMap, + body: Vec, + call_context: Option, +} + +impl fmt::Debug for ProviderResponse { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProviderResponse") + .field("status", &self.status) + .field("body_len", &self.body.len()) + .finish_non_exhaustive() + } +} + +impl ProviderResponse { + /// Built by the coordinator after buffering and decoding. + #[must_use] + pub(crate) fn new( + status: StatusCode, + headers: HeaderMap, + body: Vec, + call_context: Option, + ) -> Self { + Self { + status, + headers, + body, + call_context, + } + } + + /// A response for tests. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn for_test(status: StatusCode, body: &[u8]) -> Self { + Self::new(status, HeaderMap::new(), body.to_vec(), None) + } + + /// The status. + #[must_use] + pub fn status(&self) -> StatusCode { + self.status + } + + /// A header as a string, if present and visible ASCII. + #[must_use] + pub fn header(&self, name: &str) -> Option<&str> { + self.headers.get(name).and_then(|value| value.to_str().ok()) + } + + /// The decoded body. + #[must_use] + pub fn body(&self) -> &[u8] { + &self.body + } + + /// Parses the body as JSON, whatever the status. + /// + /// # Errors + /// + /// `Malformed` when it does not parse. + pub fn json(&self) -> Result> { + serde_json::from_slice(&self.body).map_err(|_| IdentityProviderError::malformed()) + } + + /// Requires a 2xx status, then parses the body as JSON. + /// + /// # Errors + /// + /// `Rejected` for a non-2xx status, `Malformed` for a bad body. + pub fn success_json(&self) -> Result> { + self.require_success()?; + self.json() + } + + /// Requires a 2xx status and a UTF-8 body. + /// + /// # Errors + /// + /// `Rejected` for a non-2xx status, `Malformed` for non-UTF-8. + pub fn success_text(&self) -> Result<&str, Report> { + self.require_success()?; + core::str::from_utf8(&self.body).map_err(|_| IdentityProviderError::malformed()) + } + + /// The context the request carried, decoded. `Ok(None)` when there was + /// none. + /// + /// # Errors + /// + /// `Config` when it does not decode as `T`. + pub fn call_context( + &self, + ) -> Result, Report> { + self.call_context + .as_deref() + .map(|json| { + serde_json::from_str(json) + .map_err(|_| IdentityProviderError::config("call context does not decode")) + }) + .transpose() + } + + fn require_success(&self) -> Result<(), Report> { + if self.status.is_success() { + Ok(()) + } else { + Err(IdentityProviderError::rejected(self.status)) + } + } +} + +#[cfg(test)] +mod tests { + use http::{HeaderName, HeaderValue, Method, StatusCode}; + use serde_json::json; + + use super::*; + + fn url() -> Url { + Url::parse("https://api.example.com/v1/fetch").expect("should parse") + } + + #[test] + fn post_json_sets_body_and_content_type() { + let request = ProviderRequest::post_json(&url(), &json!({ "a": 1 })).expect("should build"); + assert_eq!(request.method(), Method::POST, "should be a POST"); + assert_eq!( + request.headers()["content-type"], + "application/json", + "should set the content type" + ); + assert_eq!(request.body(), br#"{"a":1}"#, "should serialize the body"); + } + + #[test] + fn post_text_and_with_body() { + let text = ProviderRequest::post_text(&url(), "abc"); + assert_eq!(text.headers()["content-type"], "text/plain", "text type"); + assert_eq!(text.body(), b"abc", "text body"); + let custom = ProviderRequest::new(Method::PUT, &url()) + .with_body("application/octet-stream", vec![1, 2]) + .expect("should accept a valid content type"); + assert_eq!(custom.method(), Method::PUT, "any method"); + assert_eq!(custom.body(), &[1, 2], "raw body"); + assert!( + ProviderRequest::new(Method::PUT, &url()) + .with_body("bad\ntype", Vec::new()) + .is_err(), + "should reject an invalid content type" + ); + } + + #[test] + fn query_bearer_and_headers() { + let request = ProviderRequest::get(&url()) + .with_query("pid", "1 2") + .with_bearer(&Redacted::new("token".to_owned())) + .try_with_header("x-example", "v") + .expect("should accept a valid header"); + assert_eq!( + request.url().as_str(), + "https://api.example.com/v1/fetch?pid=1+2", + "should encode the query" + ); + assert_eq!( + request.headers()["authorization"], + "Bearer token", + "should set the bearer" + ); + assert_eq!(request.headers()["x-example"], "v", "should set the header"); + assert!( + ProviderRequest::get(&url()) + .try_with_header("bad header", "v") + .is_err(), + "should reject a bad name" + ); + assert!( + ProviderRequest::get(&url()) + .try_with_header("x", "bad\nvalue") + .is_err(), + "should reject a bad value" + ); + } + + #[test] + fn cookie_headers_are_flagged_from_either_helper() { + let typed = ProviderRequest::get(&url()).with_header( + HeaderName::from_static("cookie"), + HeaderValue::from_static("a=1"), + ); + let dynamic = ProviderRequest::get(&url()) + .try_with_header("Cookie", "a=1") + .expect("should build"); + assert!(typed.has_cookie_header(), "typed helper"); + assert!(dynamic.has_cookie_header(), "dynamic helper"); + assert!(!ProviderRequest::get(&url()).has_cookie_header(), "control"); + } + + #[test] + fn call_context_round_trips_and_is_capped() { + let request = ProviderRequest::get(&url()) + .try_with_call_context(&json!({ "nonce": [1, 2] })) + .expect("should accept a small context"); + let response = ProviderResponse::new( + StatusCode::OK, + http::HeaderMap::new(), + Vec::new(), + request.call_context_json().map(str::to_owned), + ); + let context: serde_json::Value = response + .call_context() + .expect("should decode") + .expect("should be present"); + assert_eq!(context, json!({ "nonce": [1, 2] }), "should round-trip"); + + let error = ProviderRequest::get(&url()) + .try_with_call_context(&"x".repeat(MAX_CALL_CONTEXT_BYTES)) + .expect_err("should reject an oversized context"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Config { .. } + ), + "should be Config" + ); + assert!( + ProviderResponse::for_test(StatusCode::OK, b"") + .call_context::() + .expect("should decode nothing") + .is_none(), + "no context is None" + ); + } + + #[test] + fn success_helpers_check_status_and_encoding() { + let ok = ProviderResponse::for_test(StatusCode::OK, br#"{"uid":"u"}"#); + let value: serde_json::Value = ok.success_json().expect("should parse"); + assert_eq!(value["uid"], "u", "should parse the uid"); + + let rejected = ProviderResponse::for_test(StatusCode::FORBIDDEN, br#"{}"#); + let error = rejected + .success_json::() + .expect_err("should reject"); + assert!( + matches!( + error.current_context(), + IdentityProviderError::Rejected { status } if *status == StatusCode::FORBIDDEN + ), + "should be Rejected with the status" + ); + rejected + .json::() + .expect("json ignores the status"); + + let binary = ProviderResponse::for_test(StatusCode::OK, &[0xff, 0xfe]); + let error = binary.success_text().expect_err("should reject non-UTF-8"); + assert!( + matches!(error.current_context(), IdentityProviderError::Malformed), + "should be Malformed" + ); + assert!( + !format!("{ok:?}").contains("uid"), + "Debug should not print the body" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/input.rs b/crates/trusted-server-core/src/ec/identity/input.rs index 80dddc47ac..2899a99979 100644 --- a/crates/trusted-server-core/src/ec/identity/input.rs +++ b/crates/trusted-server-core/src/ec/identity/input.rs @@ -13,6 +13,7 @@ use serde::de::DeserializeOwned; use url::Url; use super::descriptor::{Access, Input, ProviderDescriptor, ResolveKind}; +use super::exchange::ProviderResponse; use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; use super::resolution::IdentityProviderError; use crate::consent::ConsentContext; @@ -575,6 +576,89 @@ impl<'a> ResolveInput<'a> { } } +/// The parts a [`CaptureInput`] is built from. +pub(crate) struct CaptureParts<'a> { + pub(crate) descriptor: &'a ProviderDescriptor, + pub(crate) path: &'a str, + pub(crate) response: &'a ProviderResponse, + pub(crate) stored: &'a BTreeMap, + pub(crate) consent: &'a ProviderConsent, + pub(crate) publisher_domain: &'a str, + pub(crate) now: UnixTime, +} + +/// What a capture may read from one proxied response. It never exposes the +/// proxied request's body or headers. +pub struct CaptureInput<'a> { + parts: CaptureParts<'a>, +} + +impl fmt::Debug for CaptureInput<'_> { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("CaptureInput") + .field("provider", self.parts.descriptor.id()) + .field("path", &self.parts.path) + .finish_non_exhaustive() + } +} + +impl<'a> CaptureInput<'a> { + /// The input for one capture. + #[must_use] + pub(crate) fn new(parts: CaptureParts<'a>) -> Self { + Self { parts } + } + + /// The proxied request path, without query string. + #[must_use] + pub fn path(&self) -> &str { + self.parts.path + } + + /// The proxied response, decoded. + #[must_use] + pub fn response(&self) -> &ProviderResponse { + self.parts.response + } + + /// The descriptor's primary source. + #[must_use] + pub fn primary_source(&self) -> &SourceDomain { + self.parts.descriptor.primary_source() + } + + /// Finds a declared source by a vendor's spelling of it. + #[must_use] + pub fn source_named(&self, name: &str) -> Option<&SourceDomain> { + self.parts.descriptor.source_named(name) + } + + /// The stored record for a source. + #[must_use] + pub fn stored(&self, source: &SourceDomain) -> Option<&StoredId> { + self.parts.stored.get(source) + } + + /// The stricter of the cookie consent and the request-body consent. + #[must_use] + pub fn consent(&self) -> &ProviderConsent { + self.parts.consent + } + + /// The publisher's domain. + #[must_use] + pub fn publisher_domain(&self) -> &str { + self.parts.publisher_domain + } + + /// The request's time. + #[must_use] + pub fn now(&self) -> UnixTime { + self.parts.now + } +} + #[cfg(test)] mod tests { use std::collections::BTreeMap; @@ -843,3 +927,54 @@ mod tests { assert_ne!(a, c, "the stream advances"); } } + +#[cfg(test)] +mod capture_tests { + use std::collections::BTreeMap; + + use http::{Method, StatusCode}; + + use super::*; + use crate::ec::identity::descriptor::Captures; + + #[test] + fn capture_input_exposes_path_and_sources() { + let descriptor = ProviderDescriptor::builder("c".parse().expect("should parse id")) + .with_source("example.com".parse().expect("should parse source")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/c/api")) + .build() + .expect("should build"); + let response = ProviderResponse::for_test(StatusCode::OK, b"{}"); + let consent = ProviderConsent::default(); + let stored = BTreeMap::new(); + let input = CaptureInput::new(CaptureParts { + descriptor: &descriptor, + path: "/integrations/c/api", + response: &response, + stored: &stored, + consent: &consent, + publisher_domain: "www.example.com", + now: UnixTime::from_secs(1), + }); + assert_eq!( + input.path(), + "/integrations/c/api", + "should expose the path" + ); + assert_eq!( + input.primary_source().as_str(), + "example.com", + "should expose the source" + ); + assert_eq!( + input.source_named("EXAMPLE.com").map(SourceDomain::as_str), + Some("example.com"), + "should normalize vendor spellings" + ); + assert_eq!( + input.response().status(), + StatusCode::OK, + "should expose the response" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 2a37fea23c..a03a3cc066 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -14,6 +14,7 @@ )] pub mod descriptor; +pub mod exchange; pub mod ids; pub mod input; pub mod resolution; @@ -22,6 +23,7 @@ pub use descriptor::{ Access, Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, ProviderDescriptorBuilder, ResolveKind, Resolves, UsOptOut, }; +pub use exchange::{ProviderRequest, ProviderResponse}; pub use ids::{IdentityProviderId, SourceDomain, UnixTime}; -pub use input::{HashedEmail, ProviderConsent, Region, ResolveInput, StoredId}; -pub use resolution::IdentityProviderError; +pub use input::{CaptureInput, HashedEmail, ProviderConsent, Region, ResolveInput, StoredId}; +pub use resolution::{IdentityProviderError, Resolution, ResolvedId, StateChange, When}; diff --git a/crates/trusted-server-core/src/ec/identity/resolution.rs b/crates/trusted-server-core/src/ec/identity/resolution.rs index 08360f87f0..6d07b91d17 100644 --- a/crates/trusted-server-core/src/ec/identity/resolution.rs +++ b/crates/trusted-server-core/src/ec/identity/resolution.rs @@ -1,9 +1,15 @@ //! Provider results and errors. +use std::collections::BTreeSet; +use std::time::Duration; + use error_stack::Report; use http::StatusCode; +use serde::Serialize; use super::descriptor::{Access, Input}; +use super::ids::{SourceDomain, UnixTime}; +use crate::ec::kv_types::{MAX_PARTNER_EXT_BYTES, MAX_PROVIDER_STATE_BYTES}; /// Marker message of [`IdentityProviderError::not_implemented`]. The harness /// matches on it to tell a missing override from a real failure. @@ -83,6 +89,352 @@ impl IdentityProviderError { } } +/// A time as a provider gave it: relative to `now`, or absolute. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum When { + /// Relative to the request's `now`. + In(Duration), + /// Absolute. + At(UnixTime), +} + +impl When { + /// Resolves against `now`. + #[must_use] + pub(crate) fn resolve(self, now: UnixTime) -> UnixTime { + match self { + Self::In(delay) => now + delay, + Self::At(at) => at, + } + } +} + +/// One ID a provider returned. +#[derive(Clone, PartialEq)] +pub struct ResolvedId { + source: SourceDomain, + uid: String, + atype: Option, + ext: Option, + expiry: Option, + refresh: Option, + refresh_until: Option, +} + +impl core::fmt::Debug for ResolvedId { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("ResolvedId") + .field("source", &self.source) + .field("uid", &"[redacted]") + .field("expiry", &self.expiry) + .field("refresh", &self.refresh) + .finish_non_exhaustive() + } +} + +impl ResolvedId { + /// An ID for `source`. + #[must_use] + pub fn new(source: &SourceDomain, uid: impl Into) -> Self { + Self { + source: source.clone(), + uid: uid.into(), + atype: None, + ext: None, + expiry: None, + refresh: None, + refresh_until: None, + } + } + + /// Sets `atype`. + #[must_use] + pub fn with_atype(mut self, atype: i32) -> Self { + self.atype = Some(atype); + self + } + + /// UID-level ext. Over 256 serialized bytes it is dropped and the ID kept. + #[must_use] + pub fn with_ext(mut self, ext: serde_json::Value) -> Self { + let fits = serde_json::to_vec(&ext).is_ok_and(|bytes| bytes.len() <= MAX_PARTNER_EXT_BYTES); + if fits { + self.ext = Some(ext); + } else { + log::debug!("Dropping oversized ext for source {}", self.source); + self.ext = None; + } + self + } + + /// Expires after `delay`. + #[must_use] + pub fn with_expires_in(mut self, delay: Duration) -> Self { + self.expiry = Some(When::In(delay)); + self + } + + /// Expires at `at`. + #[must_use] + pub fn with_expires_at(mut self, at: UnixTime) -> Self { + self.expiry = Some(When::At(at)); + self + } + + /// Due for refresh after `delay`. + #[must_use] + pub fn with_refresh_in(mut self, delay: Duration) -> Self { + self.refresh = Some(When::In(delay)); + self + } + + /// Due for refresh at `at`. + #[must_use] + pub fn with_refresh_at(mut self, at: UnixTime) -> Self { + self.refresh = Some(When::At(at)); + self + } + + /// When this ID stops being refreshable, such as a refresh token's + /// expiry. Only for a provider declared `with_private_state()`; from any + /// other it is ignored with a warning. Until then an expired record is + /// kept as a refresh anchor. + #[must_use] + pub fn with_refresh_until(mut self, at: UnixTime) -> Self { + self.refresh_until = Some(at); + self + } + + /// The source. + #[must_use] + pub fn source(&self) -> &SourceDomain { + &self.source + } + + /// The UID. + #[must_use] + pub fn uid(&self) -> &str { + &self.uid + } + + /// `atype`. + #[must_use] + pub fn atype(&self) -> Option { + self.atype + } + + /// UID-level ext. + #[must_use] + pub fn ext(&self) -> Option<&serde_json::Value> { + self.ext.as_ref() + } + + /// The expiry as given. + #[must_use] + pub fn expiry(&self) -> Option { + self.expiry + } + + /// The refresh hint as given. + #[must_use] + pub fn refresh(&self) -> Option { + self.refresh + } + + /// The refresh limit. + #[must_use] + pub fn refresh_until(&self) -> Option { + self.refresh_until + } +} + +/// How a result changes the provider's private state. +#[derive(Debug, PartialEq, Eq)] +pub enum StateChange<'a> { + /// Leave it. + Keep, + /// Replace it with this JSON. + Replace(&'a str), + /// Remove it. + Clear, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum StateOp { + Keep, + Replace(String), + Clear, + Rejected, +} + +/// What one provider call produced. +#[derive(Debug, Clone, PartialEq)] +pub struct Resolution { + ids: Vec, + invalidated: Vec, + opted_out: bool, + retry_in: Option, + state: StateOp, +} + +impl Resolution { + fn empty() -> Self { + Self { + ids: Vec::new(), + invalidated: Vec::new(), + opted_out: false, + retry_in: None, + state: StateOp::Keep, + } + } + + /// One or more IDs, at most one per source: KV holds one UID per + /// source. The coordinator keeps the first ID for a source and discards + /// the rest. An empty set is `no_match`. + #[must_use] + pub fn found(ids: impl IntoIterator) -> Self { + Self { + ids: ids.into_iter().collect(), + ..Self::empty() + } + } + + /// The provider recognized no one, or a capture held nothing. + #[must_use] + pub fn no_match() -> Self { + Self::empty() + } + + /// The stored ID for `source` is no longer valid. + #[must_use] + pub fn invalidate(source: &SourceDomain) -> Self { + Self { + invalidated: vec![source.clone()], + ..Self::empty() + } + } + + /// The provider reports the user opted out. + #[must_use] + pub fn opted_out() -> Self { + Self { + opted_out: true, + ..Self::empty() + } + } + + /// Also invalidates `source`. Chainable after `found`, so one response + /// can return some sources and drop others. + #[must_use] + pub fn also_invalidate(mut self, source: &SourceDomain) -> Self { + if !self.invalidated.contains(source) { + self.invalidated.push(source.clone()); + } + self + } + + /// The earliest the coordinator may call again. Default: 24 hours after + /// `no_match`, none otherwise. + #[must_use] + pub fn with_retry_in(mut self, delay: Duration) -> Self { + self.retry_in = Some(delay); + self + } + + /// Replaces the provider's private state. + /// + /// The state is stored as embedded JSON of at most 1,024 bytes. A larger + /// or unserializable value rejects the positive part of the result whole: + /// no ID, state or scheduling change is stored, and the old state stays. + /// The result's removals still apply. + #[must_use] + pub fn replacing_state(mut self, state: &impl Serialize) -> Self { + self.state = match serde_json::to_string(state) { + Ok(json) if json.len() <= MAX_PROVIDER_STATE_BYTES => StateOp::Replace(json), + _ => StateOp::Rejected, + }; + self + } + + /// Clears the provider's private state. + #[must_use] + pub fn clearing_state(mut self) -> Self { + self.state = StateOp::Clear; + self + } + + /// The returned IDs, as given. + #[must_use] + pub fn ids(&self) -> &[ResolvedId] { + &self.ids + } + + /// Invalidated sources. + #[must_use] + pub fn invalidated(&self) -> &[SourceDomain] { + &self.invalidated + } + + /// Whether nothing was found, invalidated or opted out. + #[must_use] + pub fn is_no_match(&self) -> bool { + self.ids.is_empty() && self.invalidated.is_empty() && !self.opted_out + } + + /// Whether the provider reported an opt-out. + #[must_use] + pub fn is_opted_out(&self) -> bool { + self.opted_out + } + + /// The retry delay. + #[must_use] + pub fn retry_in(&self) -> Option { + self.retry_in + } + + /// The state change. A rejected replacement reports `Keep`. + #[must_use] + pub fn state_change(&self) -> StateChange<'_> { + match &self.state { + StateOp::Keep | StateOp::Rejected => StateChange::Keep, + StateOp::Replace(json) => StateChange::Replace(json), + StateOp::Clear => StateChange::Clear, + } + } + + /// Whether `replacing_state` was given an oversized or unserializable + /// value. The coordinator then rejects the positive part. + #[must_use] + pub(crate) fn state_rejected(&self) -> bool { + self.state == StateOp::Rejected + } + + /// Whether this result asks for any state change. + #[must_use] + pub(crate) fn changes_state(&self) -> bool { + self.state != StateOp::Keep + } + + /// The first ID per source, in order, and how many duplicates were + /// dropped. + #[must_use] + pub(crate) fn found_ids_by_source(&self) -> (Vec<&ResolvedId>, usize) { + let mut seen = BTreeSet::new(); + let mut kept = Vec::new(); + let mut dropped = 0; + for id in &self.ids { + if seen.insert(id.source()) { + kept.push(id); + } else { + dropped += 1; + } + } + (kept, dropped) + } +} + #[cfg(test)] mod tests { use super::*; @@ -101,3 +453,123 @@ mod tests { ); } } + +#[cfg(test)] +mod resolution_tests { + use std::time::Duration; + + use serde_json::json; + + use super::*; + + fn source(name: &str) -> SourceDomain { + name.parse().expect("should parse") + } + + #[test] + fn empty_found_is_no_match() { + let resolution = Resolution::found([]); + assert!(resolution.is_no_match(), "should be no_match"); + assert!(resolution.ids().is_empty(), "no ids"); + } + + #[test] + fn found_keeps_the_first_id_per_source() { + let a = source("a.example.com"); + let resolution = + Resolution::found([ResolvedId::new(&a, "first"), ResolvedId::new(&a, "second")]); + let (kept, dropped) = resolution.found_ids_by_source(); + assert_eq!(kept.len(), 1, "one per source"); + assert_eq!(kept[0].uid(), "first", "should keep the first"); + assert_eq!(dropped, 1, "should count the duplicate"); + } + + #[test] + fn invalidations_chain_and_state_changes_are_reported() { + let a = source("a.example.com"); + let b = source("b.example.com"); + let resolution = Resolution::found([ResolvedId::new(&a, "u")]) + .also_invalidate(&b) + .with_retry_in(Duration::from_secs(60)) + .replacing_state(&json!({ "signature": "s" })); + assert_eq!(resolution.invalidated(), &[b], "should invalidate b"); + assert_eq!( + resolution.retry_in(), + Some(Duration::from_secs(60)), + "should keep the retry" + ); + assert_eq!( + resolution.state_change(), + StateChange::Replace(r#"{"signature":"s"}"#), + "should replace the state" + ); + assert_eq!( + Resolution::no_match().clearing_state().state_change(), + StateChange::Clear, + "should clear" + ); + assert_eq!( + Resolution::no_match().state_change(), + StateChange::Keep, + "should keep by default" + ); + assert!( + !Resolution::invalidate(&source("c.example.com")).is_no_match(), + "an invalidation is not no_match" + ); + } + + #[test] + fn oversized_state_is_flagged_not_dropped_silently() { + let resolution = Resolution::found([ResolvedId::new(&source("a.example.com"), "u")]) + .replacing_state(&"x".repeat(2000)); + assert!( + resolution.state_rejected(), + "should flag the state as rejected" + ); + assert_eq!( + resolution.state_change(), + StateChange::Keep, + "a rejected state keeps the old one" + ); + } + + #[test] + fn oversized_ext_is_dropped_and_the_id_kept() { + let id = ResolvedId::new(&source("a.example.com"), "u") + .with_ext(json!({ "pad": "x".repeat(300) })); + assert_eq!(id.uid(), "u", "should keep the id"); + assert!(id.ext().is_none(), "should drop the ext"); + let small = + ResolvedId::new(&source("a.example.com"), "u").with_ext(json!({ "linkType": 2 })); + assert_eq!( + small.ext(), + Some(&json!({ "linkType": 2 })), + "control: a small ext is kept" + ); + } + + #[test] + fn times_resolve_against_now() { + let id = ResolvedId::new(&source("a.example.com"), "u") + .with_expires_in(Duration::from_secs(10)) + .with_refresh_at(UnixTime::from_secs(5)); + let now = UnixTime::from_secs(100); + assert_eq!( + id.expiry().map(|when| when.resolve(now)), + Some(UnixTime::from_secs(110)), + "relative expiry" + ); + assert_eq!( + id.refresh().map(|when| when.resolve(now)), + Some(UnixTime::from_secs(5)), + "absolute refresh" + ); + } + + #[test] + fn opted_out_is_reported() { + assert!(Resolution::opted_out().is_opted_out(), "opted out"); + assert!(!Resolution::no_match().is_opted_out(), "control"); + } +} diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 2c4c14698c..b835018bac 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -87,6 +87,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/descriptor.rs", include_str!("ec/identity/descriptor.rs"), ), + ( + "ec/identity/exchange.rs", + include_str!("ec/identity/exchange.rs"), + ), ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), ("ec/identity/input.rs", include_str!("ec/identity/input.rs")), ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), From b957cad3e4ec8a7af066a01fa422662173e7b27b Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:00:17 +0530 Subject: [PATCH 12/33] Add the IdentityProvider trait Signed-off-by: dhruv8sh --- .../src/ec/identity/mod.rs | 4 +- .../src/ec/identity/provider.rs | 149 ++++++++++++++++++ .../src/migration_guards.rs | 4 + 3 files changed, 156 insertions(+), 1 deletion(-) create mode 100644 crates/trusted-server-core/src/ec/identity/provider.rs diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index a03a3cc066..e076e54cb8 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -1,6 +1,6 @@ //! Server-side identity providers. //! -//! An [`IdentityProvider`](provider::IdentityProvider) turns request signals +//! An [`IdentityProvider`] turns request signals //! into `OpenRTB` EIDs. Providers only build requests and parse buffered //! responses. The coordinator owns eligibility, consent, transport limits and //! KV write-back, so every page load makes at most one browser-driven KV @@ -17,6 +17,7 @@ pub mod descriptor; pub mod exchange; pub mod ids; pub mod input; +pub mod provider; pub mod resolution; pub use descriptor::{ @@ -26,4 +27,5 @@ pub use descriptor::{ pub use exchange::{ProviderRequest, ProviderResponse}; pub use ids::{IdentityProviderId, SourceDomain, UnixTime}; pub use input::{CaptureInput, HashedEmail, ProviderConsent, Region, ResolveInput, StoredId}; +pub use provider::{IdentityProvider, IdentityProviderConfig}; pub use resolution::{IdentityProviderError, Resolution, ResolvedId, StateChange, When}; diff --git a/crates/trusted-server-core/src/ec/identity/provider.rs b/crates/trusted-server-core/src/ec/identity/provider.rs new file mode 100644 index 0000000000..d4c74f7a12 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/provider.rs @@ -0,0 +1,149 @@ +//! The identity provider trait. + +use error_stack::Report; + +use super::descriptor::ProviderDescriptor; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::input::{CaptureInput, ResolveInput}; +use super::resolution::{IdentityProviderError, Resolution}; +use crate::settings::IntegrationConfig; + +/// A vendor that turns server-side request signals into EIDs. +/// +/// Only [`descriptor`](Self::descriptor) is required. A provider adds a +/// capability by declaring it in its descriptor and overriding the matching +/// methods: +/// +/// - active resolution: [`Resolves`](super::descriptor::Resolves), plus +/// [`request`](Self::request) and [`response`](Self::response) +/// - passive capture: [`Captures`](super::descriptor::Captures), plus +/// [`capture`](Self::capture) +/// +/// Methods added to this trait later get default bodies, so existing +/// providers keep compiling. No method takes generic parameters, so the trait +/// stays object-safe. Every default body returns the +/// [`IdentityProviderError::not_implemented`] marker, so a capability that is +/// declared but not implemented is caught by the harness. +pub trait IdentityProvider: Send + Sync { + /// What this provider is, what it may write, what it needs and which + /// consent it requires. + fn descriptor(&self) -> &ProviderDescriptor; + + /// Builds the call for this request, or `None` to skip it this time. + /// + /// Skipping is not a failure: the coordinator waits `refresh_after_sec` + /// before asking again. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::UndeclaredInput`] when it reads + /// something its descriptor did not declare, or another + /// [`IdentityProviderError`] when the request cannot be built. + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _ = input; + Err(IdentityProviderError::not_implemented("request")) + } + + /// Turns the buffered response to [`request`](Self::request) into a + /// resolution. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Rejected`] for a refusal and + /// [`IdentityProviderError::Malformed`] for a body it cannot parse. + fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, + ) -> Result> { + let _ = (response, input); + Err(IdentityProviderError::not_implemented("response")) + } + + /// Reads IDs from a response TS proxied for this provider, with no + /// outbound call. [`Resolution::no_match`] means the response held + /// nothing to capture, and costs nothing: capture never backs off. + /// + /// # Errors + /// + /// Returns an [`IdentityProviderError`] only for a provider bug. A body + /// it does not recognise is `Ok(Resolution::no_match())`, not an error. + fn capture( + &self, + input: &CaptureInput<'_>, + ) -> Result> { + let _ = input; + Err(IdentityProviderError::not_implemented("capture")) + } +} + +/// Implemented by each identity integration's config type. +/// +/// Builds the provider's descriptor from config alone, with no secrets, so +/// `ts config validate` checks ownership against the same descriptor the +/// running provider uses. +pub trait IdentityProviderConfig: IntegrationConfig { + /// Describes the provider this config enables. + /// + /// # Errors + /// + /// Returns [`IdentityProviderError::Config`] when the config cannot + /// describe a valid provider. + fn build_descriptor(&self) -> Result>; +} + +// Keeps the trait object-safe. Fails to compile if a method gains generics. +const _: Option<&dyn IdentityProvider> = None; + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use http::{Method, StatusCode}; + + use super::*; + use crate::ec::identity::descriptor::Captures; + use crate::ec::identity::ids::UnixTime; + use crate::ec::identity::input::{CaptureParts, ProviderConsent}; + + struct DescriptorOnly(ProviderDescriptor); + + impl IdentityProvider for DescriptorOnly { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } + } + + #[test] + fn default_capability_methods_return_the_not_implemented_marker() { + let provider = DescriptorOnly( + ProviderDescriptor::builder("p".parse().expect("should parse id")) + .with_source("example.com".parse().expect("should parse source")) + .with_captures(Captures::new().with_path(Method::POST, "/integrations/p/api")) + .build() + .expect("should build"), + ); + let boxed: Box = Box::new(provider); + let response = ProviderResponse::for_test(StatusCode::OK, b"{}"); + let consent = ProviderConsent::default(); + let stored = BTreeMap::new(); + let input = CaptureInput::new(CaptureParts { + descriptor: boxed.descriptor(), + path: "/integrations/p/api", + response: &response, + stored: &stored, + consent: &consent, + publisher_domain: "www.example.com", + now: UnixTime::from_secs(1), + }); + + let error = boxed.capture(&input).expect_err("should be the marker"); + assert!( + error.current_context().is_not_implemented(), + "should carry the marker" + ); + } +} diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index b835018bac..140290f82c 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -94,6 +94,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), ("ec/identity/input.rs", include_str!("ec/identity/input.rs")), ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), + ( + "ec/identity/provider.rs", + include_str!("ec/identity/provider.rs"), + ), ( "ec/identity/resolution.rs", include_str!("ec/identity/resolution.rs"), From 3d56fbb5cebe323f9f8df4aa00a8b667dd57a21e Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:02:48 +0530 Subject: [PATCH 13/33] Evaluate provider consent policies against decoded consent Reuses TcfConsent::has_vendor_consent and the effective TCF lookup the consent module already uses. Adds shared consent fixtures for identity tests. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/consent/mod.rs | 2 +- .../src/ec/identity/mod.rs | 3 + .../src/ec/identity/policy.rs | 244 ++++++++++++++++++ .../src/ec/identity/test_support.rs | 83 ++++++ .../src/migration_guards.rs | 8 + 5 files changed, 339 insertions(+), 1 deletion(-) create mode 100644 crates/trusted-server-core/src/ec/identity/policy.rs create mode 100644 crates/trusted-server-core/src/ec/identity/test_support.rs diff --git a/crates/trusted-server-core/src/consent/mod.rs b/crates/trusted-server-core/src/consent/mod.rs index 76aa6ad859..184222b926 100644 --- a/crates/trusted-server-core/src/consent/mod.rs +++ b/crates/trusted-server-core/src/consent/mod.rs @@ -280,7 +280,7 @@ fn has_eu_tcf_signal(raw_tc_present: bool, gpp_section_ids: Option<&[u16]>) -> b /// Returns the effective decoded TCF consent for enforcement decisions. #[must_use] -fn effective_tcf(ctx: &ConsentContext) -> Option<&types::TcfConsent> { +pub(crate) fn effective_tcf(ctx: &ConsentContext) -> Option<&types::TcfConsent> { ctx.tcf.as_ref().or_else(|| { let g = ctx.gpp.as_ref()?; g.eu_tcf.as_ref() diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index e076e54cb8..1f3e67a8fe 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -17,8 +17,11 @@ pub mod descriptor; pub mod exchange; pub mod ids; pub mod input; +pub(crate) mod policy; pub mod provider; pub mod resolution; +#[cfg(test)] +pub(crate) mod test_support; pub use descriptor::{ Access, Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, diff --git a/crates/trusted-server-core/src/ec/identity/policy.rs b/crates/trusted-server-core/src/ec/identity/policy.rs new file mode 100644 index 0000000000..428d461a7a --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/policy.rs @@ -0,0 +1,244 @@ +//! Evaluates a provider's [`ConsentPolicy`] against a request's consent. + +use super::descriptor::{ConsentPolicy, UsOptOut}; +use crate::consent::jurisdiction::Jurisdiction; +use crate::consent::types::PrivacyFlag; +use crate::consent::{ConsentContext, effective_tcf}; + +/// Signals a policy reads that the consent context does not carry. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) struct PolicySignals { + /// `DNT: 1` was present. + pub(crate) dnt: bool, +} + +/// Where TCF applies: `gdpr_applies`, or a jurisdiction of GDPR or unknown. +/// The same test [`crate::consent::consent_allows_server_side_auction`] uses. +#[must_use] +pub(crate) fn tcf_applies(consent: &ConsentContext) -> bool { + consent.gdpr_applies + || matches!( + consent.jurisdiction, + Jurisdiction::Gdpr | Jurisdiction::Unknown + ) +} + +/// Whether the request carries a US sale opt-out, from GPP or US Privacy. +fn us_sale_opted_out(consent: &ConsentContext) -> bool { + consent.gpp.as_ref().and_then(|gpp| gpp.us_sale_opt_out) == Some(true) + || consent + .us_privacy + .as_ref() + .is_some_and(|usp| usp.opt_out_sale == PrivacyFlag::Yes) +} + +/// Whether a raw consent string is present without its decoded form, for +/// example in proxy mode or after a decode failure. A strict policy fails +/// closed on it. +fn has_undecoded_signal(consent: &ConsentContext) -> bool { + (consent.raw_tc_string.is_some() && consent.tcf.is_none()) + || (consent.raw_gpp_string.is_some() && consent.gpp.is_none()) + || (consent.raw_us_privacy.is_some() && consent.us_privacy.is_none()) +} + +/// Whether `policy` allows a provider under this request's consent. +/// +/// The global EC and EID-persistence gates are checked separately by the +/// coordinator and are not repeated here. +#[must_use] +pub(crate) fn policy_allows( + policy: &ConsentPolicy, + consent: &ConsentContext, + signals: PolicySignals, +) -> bool { + if policy.is_eid_persistence_only() { + return true; + } + if has_undecoded_signal(consent) { + return false; + } + if tcf_applies(consent) { + let Some(tcf) = effective_tcf(consent) else { + return false; + }; + let purposes_granted = policy + .tcf_purposes() + .iter() + .all(|purpose| tcf.has_purpose_consent(usize::from(*purpose))); + let vendor_granted = policy + .tcf_vendor() + .is_none_or(|vendor| tcf.has_vendor_consent(vendor)); + if !purposes_granted || !vendor_granted { + return false; + } + } + if !policy.allows_us_opt_out(UsOptOut::Sale) && us_sale_opted_out(consent) { + return false; + } + if !policy.allows_gpc() && consent.gpc { + return false; + } + if !policy.allows_dnt() && signals.dnt { + return false; + } + true +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::descriptor::{ConsentPolicy, UsOptOut}; + use crate::ec::identity::test_support::{TestSignals, consent_with}; + + const NO_DNT: PolicySignals = PolicySignals { dnt: false }; + + #[test] + fn strict_blocks_each_signal_it_decodes() { + let strict = ConsentPolicy::strict(); + assert!( + policy_allows(&strict, &consent_with(&TestSignals::default()), NO_DNT), + "control: nothing set" + ); + let gdpr_ok = TestSignals { + gdpr: true, + purposes: vec![1, 4], + ..TestSignals::default() + }; + assert!( + policy_allows(&strict, &consent_with(&gdpr_ok), NO_DNT), + "control: GDPR with purposes 1 and 4" + ); + + for (name, signals, dnt) in [ + ( + "gdpr without purpose 4", + TestSignals { + gdpr: true, + purposes: vec![1], + ..TestSignals::default() + }, + false, + ), + ( + "us sale opt-out", + TestSignals { + us_sale_opt_out: true, + ..TestSignals::default() + }, + false, + ), + ( + "gpc", + TestSignals { + gpc: true, + ..TestSignals::default() + }, + false, + ), + ("dnt", TestSignals::default(), true), + ( + "undecodable tc string", + TestSignals { + raw_tc_without_decode: true, + ..TestSignals::default() + }, + false, + ), + ] { + assert!( + !policy_allows(&strict, &consent_with(&signals), PolicySignals { dnt }), + "strict should block: {name}" + ); + } + } + + #[test] + fn relaxations_and_vendor_consent() { + let gpc = consent_with(&TestSignals { + gpc: true, + ..TestSignals::default() + }); + assert!( + policy_allows(&ConsentPolicy::strict().allowing_gpc(), &gpc, NO_DNT), + "gpc relaxed" + ); + let sale = consent_with(&TestSignals { + us_sale_opt_out: true, + ..TestSignals::default() + }); + assert!( + policy_allows( + &ConsentPolicy::strict().allowing_us_opt_out(UsOptOut::Sale), + &sale, + NO_DNT + ), + "sale relaxed" + ); + assert!( + policy_allows( + &ConsentPolicy::strict().allowing_dnt(), + &consent_with(&TestSignals::default()), + PolicySignals { dnt: true } + ), + "dnt relaxed" + ); + let gdpr = consent_with(&TestSignals { + gdpr: true, + purposes: vec![1, 4], + vendors: vec![9999], + ..TestSignals::default() + }); + assert!( + policy_allows( + &ConsentPolicy::strict().with_tcf_vendor(9999), + &gdpr, + NO_DNT + ), + "vendor granted" + ); + assert!( + !policy_allows( + &ConsentPolicy::strict().with_tcf_vendor(1234), + &gdpr, + NO_DNT + ), + "vendor missing" + ); + assert!( + !policy_allows( + &ConsentPolicy::strict().with_tcf_purposes([7]), + &gdpr, + NO_DNT + ), + "extra purpose missing" + ); + } + + #[test] + fn eid_persistence_only_ignores_provider_signals() { + let consent = consent_with(&TestSignals { + gpc: true, + us_sale_opt_out: true, + ..TestSignals::default() + }); + assert!( + policy_allows( + &ConsentPolicy::eid_persistence_only(), + &consent, + PolicySignals { dnt: true } + ), + "generic pull's parity policy checks only the global gates" + ); + } + + #[test] + fn tcf_applies_in_unknown_jurisdiction() { + let mut consent = consent_with(&TestSignals::default()); + consent.jurisdiction = crate::consent::jurisdiction::Jurisdiction::Unknown; + assert!(tcf_applies(&consent), "unknown jurisdiction needs TCF"); + assert!( + !policy_allows(&ConsentPolicy::strict(), &consent, NO_DNT), + "strict blocks without a TC string where TCF applies" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs new file mode 100644 index 0000000000..8603dffde9 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -0,0 +1,83 @@ +//! Shared fixtures for identity tests. + +use crate::consent::ConsentContext; +use crate::consent::jurisdiction::Jurisdiction; +use crate::consent::types::{PrivacyFlag, TcfConsent, UsPrivacy}; + +/// Consent signals for building a [`ConsentContext`] in tests. +#[derive(Debug, Clone, Default)] +pub(crate) struct TestSignals { + /// GDPR applies, with a decoded TC string granting these purposes. + pub(crate) gdpr: bool, + /// Granted TCF purposes (1-indexed). + pub(crate) purposes: Vec, + /// Granted GVL vendors. + pub(crate) vendors: Vec, + /// A decoded US Privacy string opting out of sale. + pub(crate) us_sale_opt_out: bool, + /// `Sec-GPC: 1`. + pub(crate) gpc: bool, + /// A raw TC string with no decoded form. + pub(crate) raw_tc_without_decode: bool, +} + +/// A decoded TC string granting `purposes` and `vendors`. +pub(crate) fn test_tcf(purposes: &[usize], vendors: &[u16]) -> TcfConsent { + let max = purposes.iter().copied().max().unwrap_or(0).max(11); + TcfConsent { + version: 2, + cmp_id: 1, + cmp_version: 1, + consent_screen: 1, + consent_language: "EN".to_owned(), + vendor_list_version: 1, + tcf_policy_version: 4, + created_ds: 1, + last_updated_ds: 1, + purpose_consents: (1..=max) + .map(|purpose| purposes.contains(&purpose)) + .collect(), + purpose_legitimate_interests: vec![false; max], + vendor_consents: vendors.to_vec(), + vendor_legitimate_interests: Vec::new(), + special_feature_opt_ins: Vec::new(), + } +} + +/// Builds a consent context from `signals`. Without GDPR the jurisdiction is +/// non-regulated. +pub(crate) fn consent_with(signals: &TestSignals) -> ConsentContext { + let mut context = ConsentContext { + gdpr_applies: signals.gdpr, + gpc: signals.gpc, + jurisdiction: if signals.gdpr { + Jurisdiction::Gdpr + } else { + Jurisdiction::NonRegulated + }, + ..ConsentContext::default() + }; + if signals.gdpr { + context.raw_tc_string = Some("example-tc-string".to_owned()); + context.tcf = Some(test_tcf(&signals.purposes, &signals.vendors)); + } + if signals.us_sale_opt_out { + context.raw_us_privacy = Some("1YYN".to_owned()); + context.us_privacy = Some(UsPrivacy { + version: 1, + notice_given: PrivacyFlag::Yes, + opt_out_sale: PrivacyFlag::Yes, + lspa_covered: PrivacyFlag::No, + }); + } + if signals.raw_tc_without_decode { + context.raw_tc_string = Some("undecodable".to_owned()); + context.tcf = None; + } + context +} + +/// A consent context that every policy allows. +pub(crate) fn allowing_consent() -> ConsentContext { + consent_with(&TestSignals::default()) +} diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 140290f82c..ee5626888c 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -94,6 +94,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), ("ec/identity/input.rs", include_str!("ec/identity/input.rs")), ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), + ( + "ec/identity/policy.rs", + include_str!("ec/identity/policy.rs"), + ), ( "ec/identity/provider.rs", include_str!("ec/identity/provider.rs"), @@ -102,6 +106,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/resolution.rs", include_str!("ec/identity/resolution.rs"), ), + ( + "ec/identity/test_support.rs", + include_str!("ec/identity/test_support.rs"), + ), ("ec/kv.rs", include_str!("ec/kv.rs")), ("ec/kv_types.rs", include_str!("ec/kv_types.rs")), ("ec/mod.rs", include_str!("ec/mod.rs")), From f642f85d75e80331d23d30876c2ea0f8ff26f901 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:04:17 +0530 Subject: [PATCH 14/33] Add record revisions and keyed identity hashes Signed-off-by: dhruv8sh --- .../src/ec/identity/mod.rs | 1 + .../src/ec/identity/revision.rs | 247 ++++++++++++++++++ .../src/migration_guards.rs | 4 + 3 files changed, 252 insertions(+) create mode 100644 crates/trusted-server-core/src/ec/identity/revision.rs diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 1f3e67a8fe..6da909864d 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -20,6 +20,7 @@ pub mod input; pub(crate) mod policy; pub mod provider; pub mod resolution; +pub(crate) mod revision; #[cfg(test)] pub(crate) mod test_support; diff --git a/crates/trusted-server-core/src/ec/identity/revision.rs b/crates/trusted-server-core/src/ec/identity/revision.rs new file mode 100644 index 0000000000..b8923453bb --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/revision.rs @@ -0,0 +1,247 @@ +//! Record revisions and keyed hashes of identifiers. +//! +//! Every hash here is an HMAC with a key derived from `ec.passphrase` under +//! its own label, so a hash cannot be linked to an EC ID or UID without the +//! key, and hashes from different uses never collide. + +use core::fmt; + +use hmac::{Hmac, Mac as _}; +use sha2::Sha256; + +use super::ids::IdentityProviderId; +use super::input::RandomSource; +use crate::ec::kv_types::KvPartnerId; +use crate::settings::Settings; + +type HmacSha256 = Hmac; + +/// Keys derived from `ec.passphrase`, once per settings load. +#[derive(Clone)] +pub(crate) struct IdentityKeys { + revision: [u8; 32], + invalidated: [u8; 32], + visitor: [u8; 32], + fingerprint: [u8; 32], + /// AES-256 key for `__Host-ts-eids-s`. + pub(crate) cookie_s: [u8; 32], + /// AES-256 key for `__Host-ts-eids-r-*`. + pub(crate) cookie_r: [u8; 32], +} + +impl fmt::Debug for IdentityKeys { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("IdentityKeys([redacted])") + } +} + +fn derive(passphrase: &[u8], label: &[u8]) -> [u8; 32] { + let mut mac = + HmacSha256::new_from_slice(passphrase).expect("should accept any HMAC key length"); + mac.update(label); + mac.finalize().into_bytes().into() +} + +/// The first 16 hex digits of an HMAC over `parts`, NUL-separated. +fn keyed_hex16(key: &[u8; 32], parts: &[&[u8]]) -> String { + let mut mac = HmacSha256::new_from_slice(key).expect("should accept a 32-byte key"); + for (index, part) in parts.iter().enumerate() { + if index > 0 { + mac.update(b"\0"); + } + mac.update(part); + } + let digest = mac.finalize().into_bytes(); + hex::encode(&digest[..8]) +} + +impl IdentityKeys { + /// Derives every identity key from `ec.passphrase`. + #[must_use] + pub(crate) fn from_settings(settings: &Settings) -> Self { + let passphrase = settings.ec.passphrase.expose().as_bytes(); + Self { + revision: derive(passphrase, b"trusted-server/identity/revision/v1"), + invalidated: derive(passphrase, b"trusted-server/identity/iv/v1"), + visitor: derive(passphrase, b"trusted-server/identity/visitor/v1"), + fingerprint: derive(passphrase, b"trusted-server/identity/fingerprint/v1"), + cookie_s: derive(passphrase, b"trusted-server/ts-eids-s/key/v1"), + cookie_r: derive(passphrase, b"trusted-server/ts-eids-r/key/v1"), + } + } +} + +/// A record's opaque revision: 16 lowercase hex digits. +/// +/// Drawn at random on every write of a record, including a refresh that +/// keeps the UID, because neither the UID nor `at` can tell two refreshes in +/// one second apart. +#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct Revision(String); + +impl Revision { + /// Wraps a stored `rv` or `pv`. Callers have validated the format. + #[must_use] + pub(crate) fn from_stored(value: &str) -> Self { + Self(value.to_owned()) + } + + /// The hex digits. + #[must_use] + pub(crate) fn as_str(&self) -> &str { + &self.0 + } +} + +/// A fresh random revision. +#[must_use] +pub(crate) fn new_revision(random: &dyn RandomSource) -> Revision { + let mut bytes = [0u8; 8]; + random.fill(&mut bytes); + Revision(hex::encode(bytes)) +} + +/// The revision of a stored record: its `rv`, or for a legacy record without +/// one, the keyed hash of its `(uid, at)` until its next write. +#[must_use] +pub(crate) fn record_revision(keys: &IdentityKeys, record: &KvPartnerId) -> Revision { + if let Some(rv) = &record.rv { + return Revision(rv.clone()); + } + let at = record.at.unwrap_or_default().to_string(); + Revision(keyed_hex16( + &keys.revision, + &[record.uid.as_bytes(), at.as_bytes()], + )) +} + +/// The keyed hash stored in `ps.iv` for a UID the owner invalidated. +#[must_use] +pub(crate) fn invalidated_uid_hash( + keys: &IdentityKeys, + provider: &IdentityProviderId, + uid: &str, +) -> String { + keyed_hex16( + &keys.invalidated, + &[provider.as_str().as_bytes(), uid.as_bytes()], + ) +} + +/// The per-visitor rate-limit key component, from the full EC ID. +#[must_use] +pub(crate) fn visitor_key(keys: &IdentityKeys, ec_id: &str) -> String { + keyed_hex16(&keys.visitor, &[ec_id.as_bytes()]) +} + +/// The consent fingerprint over the signals a provider's policy reads. +#[must_use] +pub(crate) fn consent_fingerprint(keys: &IdentityKeys, bits: &str) -> String { + keyed_hex16(&keys.fingerprint, &[bits.as_bytes()]) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ec::identity::input::SeededRandom; + use crate::ec::kv_types::KvPartnerId; + use crate::redacted::Redacted; + use crate::test_support::tests::create_test_settings; + + fn keys() -> IdentityKeys { + IdentityKeys::from_settings(&create_test_settings()) + } + + #[test] + fn new_revisions_are_lowercase_hex_and_differ() { + let random = SeededRandom::new(7); + let first = new_revision(&random); + let second = new_revision(&random); + assert_eq!(first.as_str().len(), 16, "should be 16 hex digits"); + assert!( + first + .as_str() + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)), + "should be lowercase hex" + ); + assert_ne!( + first, second, + "two writes of the same UID get different revisions" + ); + } + + #[test] + fn record_revision_prefers_rv_and_hashes_legacy_records() { + let keys = keys(); + let with_rv = KvPartnerId { + rv: Some("0123456789abcdef".to_owned()), + ..KvPartnerId::new("u") + }; + assert_eq!( + record_revision(&keys, &with_rv).as_str(), + "0123456789abcdef", + "should use rv" + ); + + let legacy_a = KvPartnerId::browser("u", Some(10)); + let legacy_b = KvPartnerId::browser("u", Some(11)); + assert_eq!( + record_revision(&keys, &legacy_a), + record_revision(&keys, &legacy_a.clone()), + "should be stable" + ); + assert_ne!( + record_revision(&keys, &legacy_a), + record_revision(&keys, &legacy_b), + "at changes it" + ); + assert_eq!( + record_revision(&keys, &legacy_a).as_str().len(), + 16, + "legacy revisions are 16 hex digits too" + ); + } + + #[test] + fn hashes_are_keyed_and_namespaced() { + let keys = keys(); + let provider: IdentityProviderId = "p".parse().expect("should parse id"); + let iv = invalidated_uid_hash(&keys, &provider, "u"); + let visitor = visitor_key(&keys, "u"); + assert_eq!(iv.len(), 16, "iv length"); + assert_ne!( + iv, visitor, + "different labels give different hashes for the same input" + ); + assert_ne!( + invalidated_uid_hash(&keys, &"q".parse().expect("should parse id"), "u"), + iv, + "the provider is part of the iv hash" + ); + + let mut other_settings = create_test_settings(); + other_settings.ec.passphrase = Redacted::new("another-example-passphrase-value".to_owned()); + let other = IdentityKeys::from_settings(&other_settings); + assert_ne!( + invalidated_uid_hash(&other, &provider, "u"), + iv, + "keyed by the passphrase" + ); + assert_ne!( + other.cookie_s, keys.cookie_s, + "cookie keys follow the passphrase" + ); + assert_ne!(keys.cookie_s, keys.cookie_r, "each cookie has its own key"); + assert_eq!( + consent_fingerprint(&keys, "bits").len(), + 16, + "fingerprint length" + ); + assert_eq!( + format!("{keys:?}"), + "IdentityKeys([redacted])", + "Debug never prints keys" + ); + } +} diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index ee5626888c..b3a360f43f 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -106,6 +106,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/resolution.rs", include_str!("ec/identity/resolution.rs"), ), + ( + "ec/identity/revision.rs", + include_str!("ec/identity/revision.rs"), + ), ( "ec/identity/test_support.rs", include_str!("ec/identity/test_support.rs"), From aca8ae5defd453cdb66f4e7222d089365c36a85e Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:08:49 +0530 Subject: [PATCH 15/33] Filter served KV IDs through one servability check Expired IDs, IDs from disabled or opted-out writers, and IDs whose writer or owner policy fails are no longer sent to bidders, the admin view or /identify. /identify uses the request's consent, never the row's stored snapshot. Stored atype and UID-level ext are emitted. The provider catalog is empty until generic pull is wired in. Signed-off-by: dhruv8sh --- .../src/auction/endpoints.rs | 21 +- crates/trusted-server-core/src/ec/admin.rs | 20 +- crates/trusted-server-core/src/ec/eids.rs | 101 +++++- crates/trusted-server-core/src/ec/identify.rs | 83 ++++- .../src/ec/identity/catalog.rs | 178 +++++++++++ .../src/ec/identity/mod.rs | 2 + .../src/ec/identity/servable.rs | 301 ++++++++++++++++++ .../src/ec/identity/test_support.rs | 48 +++ .../src/migration_guards.rs | 8 + 9 files changed, 742 insertions(+), 20 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity/catalog.rs create mode 100644 crates/trusted-server-core/src/ec/identity/servable.rs diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index c21caa1b0a..461c623a2e 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -16,6 +16,10 @@ use crate::ec::EcContext; use crate::ec::EcKvSnapshot; use crate::ec::client_eids::parse_client_eids; use crate::ec::eids::{resolve_partner_ids, to_eids}; +use crate::ec::identity::UnixTime; +use crate::ec::identity::catalog::ProviderCatalog; +use crate::ec::identity::policy::PolicySignals; +use crate::ec::identity::servable::{ServeContext, kv_opted_out}; use crate::ec::kv::KvIdentityGraph; use crate::ec::partner::normalize_partner_source_domain; use crate::ec::prebid_eids::parse_prebid_eids_cookie; @@ -462,7 +466,22 @@ pub(crate) fn resolve_auction_eids( return Some(Vec::new()); } - let resolved = resolve_partner_ids(registry, entry); + // Without a trustworthy clock every expiry check would pass, so no KV ID + // is served. + let Some(now) = crate::ec::checked_current_timestamp() else { + log::warn!("Skipping KV EIDs because the clock is unavailable"); + return Some(Vec::new()); + }; + let catalog = ProviderCatalog::empty(); + let opted_out = kv_opted_out(entry); + let serve = ServeContext { + now: UnixTime::from_secs(now), + consent: ec_context.consent(), + signals: PolicySignals::default(), + catalog: &catalog, + opted_out: &opted_out, + }; + let resolved = resolve_partner_ids(registry, entry, &serve); Some(to_eids(&resolved)) } diff --git a/crates/trusted-server-core/src/ec/admin.rs b/crates/trusted-server-core/src/ec/admin.rs index ef0f018e3d..05320e24d4 100644 --- a/crates/trusted-server-core/src/ec/admin.rs +++ b/crates/trusted-server-core/src/ec/admin.rs @@ -26,6 +26,7 @@ use serde_json::Value as JsonValue; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt as _}; +use crate::consent::ConsentContext; use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; use crate::cookies::extract_cookie_value; use crate::error::TrustedServerError; @@ -33,6 +34,10 @@ use crate::openrtb::Eid; use super::eids::{resolve_partner_ids, to_eids}; use super::generation::is_valid_ec_id; +use super::identity::UnixTime; +use super::identity::catalog::ProviderCatalog; +use super::identity::policy::PolicySignals; +use super::identity::servable::{ServeContext, kv_opted_out}; use super::kv::KvIdentityGraph; use super::kv_backend::EcKvLookup; use super::kv_types::{KvEntry, KvMetadata}; @@ -495,8 +500,21 @@ fn iso_timestamp(unix_seconds: u64) -> Option { /// Derives the auction EID view for a valid entry, mirroring the filters in /// [`resolve_partner_ids`] and reporting why each stored ID was skipped. +/// +/// The admin request carries no visitor consent, so the view applies expiry, +/// writer presence and opt-outs with a default consent context. fn build_auction_view(registry: &PartnerRegistry, entry: &KvEntry) -> AuctionEidsView { - let resolved = resolve_partner_ids(registry, entry); + let consent = ConsentContext::default(); + let catalog = ProviderCatalog::empty(); + let opted_out = kv_opted_out(entry); + let serve = ServeContext { + now: UnixTime::from_secs(super::current_timestamp()), + consent: &consent, + signals: PolicySignals::default(), + catalog: &catalog, + opted_out: &opted_out, + }; + let resolved = resolve_partner_ids(registry, entry, &serve); let eids = to_eids(&resolved); let mut skipped = Vec::new(); diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index fe03c47d33..b45f4dd2ed 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -1,8 +1,10 @@ //! Shared EID resolution and formatting helpers. //! -//! Used by both `/_ts/api/v1/identify` and `/auction` to resolve source-domain -//! keyed IDs from KV entries, convert them to `OpenRTB` EID structures, and -//! build base64-encoded response headers. +//! Used by `/auction`, the navigation auction and the admin view to resolve +//! source-domain keyed IDs from KV entries, convert them to `OpenRTB` EID +//! structures, and build base64-encoded response headers. Every KV ID passes +//! the shared servability check in [`crate::ec::identity::servable`], so +//! expired IDs and IDs a writer or owner may not disclose are never sent. use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; use error_stack::{Report, ResultExt as _}; @@ -10,6 +12,8 @@ use error_stack::{Report, ResultExt as _}; use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; +use super::identity::SourceDomain; +use super::identity::servable::{ServeContext, is_servable}; use super::kv_types::KvEntry; use super::registry::PartnerRegistry; @@ -24,8 +28,10 @@ pub struct ResolvedPartnerId { pub source_domain: String, /// The synced user ID value. pub uid: String, - /// `OpenRTB` agent type for this partner's identifiers. + /// `OpenRTB` agent type: the stored provider `atype`, else the partner's. pub openrtb_atype: i32, + /// Stored UID-level `ext`, when a provider supplied one. + pub ext: Option, } /// Resolves source-domain keyed IDs from a KV entry against the partner registry. @@ -33,25 +39,32 @@ pub struct ResolvedPartnerId { /// Filters to partners with `bidstream_enabled = true` and non-empty UIDs, /// sorted deterministically by source domain. #[must_use] -pub fn resolve_partner_ids(registry: &PartnerRegistry, entry: &KvEntry) -> Vec { +pub(crate) fn resolve_partner_ids( + registry: &PartnerRegistry, + entry: &KvEntry, + serve: &ServeContext<'_>, +) -> Vec { let mut resolved = Vec::new(); for (source_domain, partner_uid) in &entry.ids { - if partner_uid.uid.is_empty() { - continue; - } - let Some(partner) = registry.get(source_domain) else { continue; }; if !partner.bidstream_enabled { continue; } + let Ok(source) = source_domain.parse::() else { + continue; + }; + if !is_servable(serve, &source, partner_uid) { + continue; + } resolved.push(ResolvedPartnerId { source_domain: partner.source_domain.clone(), uid: partner_uid.uid.clone(), - openrtb_atype: partner.openrtb_atype, + openrtb_atype: partner_uid.atype.unwrap_or(partner.openrtb_atype), + ext: partner_uid.ext.clone(), }); } @@ -70,7 +83,7 @@ pub fn to_eids(resolved: &[ResolvedPartnerId]) -> Vec { vec![Uid { id: item.uid.clone(), atype: Some(item.openrtb_atype), - ext: None, + ext: item.ext.clone(), }], ) }) @@ -188,7 +201,7 @@ mod tests { super::super::kv_types::KvPartnerId::new("uid-a".to_owned()), ); - let resolved = resolve_partner_ids(®istry, &entry); + let resolved = resolve_partner_ids(®istry, &entry, &serve_context(&serve_parts())); let source_domains: Vec<&str> = resolved .iter() .map(|item| item.source_domain.as_str()) @@ -201,6 +214,65 @@ mod tests { ); } + struct ServeParts { + consent: crate::consent::ConsentContext, + catalog: crate::ec::identity::catalog::ProviderCatalog, + opted_out: std::collections::BTreeSet, + } + + fn serve_parts() -> ServeParts { + ServeParts { + consent: crate::ec::identity::test_support::allowing_consent(), + catalog: crate::ec::identity::catalog::ProviderCatalog::empty(), + opted_out: std::collections::BTreeSet::new(), + } + } + + fn serve_context(parts: &ServeParts) -> ServeContext<'_> { + ServeContext { + now: crate::ec::identity::UnixTime::from_secs(100), + consent: &parts.consent, + signals: crate::ec::identity::policy::PolicySignals::default(), + catalog: &parts.catalog, + opted_out: &parts.opted_out, + } + } + + #[test] + fn resolve_partner_ids_skips_expired_records_and_uses_stored_atype_and_ext() { + let partners = vec![ + make_test_partner("a.example.com"), + make_test_partner("b.example.com"), + ]; + let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); + let mut entry = KvEntry::minimal("a.example.com", "ua", 1); + entry.ids.insert( + "b.example.com".to_owned(), + super::super::kv_types::KvPartnerId { + exp: Some(50), + ..super::super::kv_types::KvPartnerId::new("ub") + }, + ); + let stored = entry.ids.get_mut("a.example.com").expect("should hold a"); + stored.atype = Some(3); + stored.ext = Some(serde_json::json!({ "linkType": 2 })); + + let resolved = resolve_partner_ids(®istry, &entry, &serve_context(&serve_parts())); + + assert_eq!(resolved.len(), 1, "should drop the expired b"); + assert_eq!(resolved[0].source_domain, "a.example.com", "should keep a"); + assert_eq!( + resolved[0].openrtb_atype, 3, + "should prefer the stored atype" + ); + let eids = to_eids(&resolved); + assert_eq!( + eids[0].uids[0].ext, + Some(serde_json::json!({ "linkType": 2 })), + "should emit the stored UID-level ext" + ); + } + #[test] fn to_eids_maps_resolved_ids_correctly() { let resolved = vec![ @@ -208,16 +280,19 @@ mod tests { uid: "LR_xyz".to_owned(), source_domain: "liveramp.com".to_owned(), openrtb_atype: 3, + ext: None, }, ResolvedPartnerId { uid: "ID5_abc".to_owned(), source_domain: "id5-sync.com".to_owned(), openrtb_atype: 1, + ext: None, }, ResolvedPartnerId { uid: "pair-id".to_owned(), source_domain: "google.com".to_owned(), openrtb_atype: 571187, + ext: None, }, ]; @@ -247,6 +322,7 @@ mod tests { uid: format!("uid_{}", "x".repeat(100)), source_domain: format!("partner-{idx}.example.com"), openrtb_atype: 3, + ext: None, }); } @@ -266,6 +342,7 @@ mod tests { uid: "u1".to_owned(), source_domain: "ssp.com".to_owned(), openrtb_atype: 3, + ext: None, }]; let (encoded, truncated) = diff --git a/crates/trusted-server-core/src/ec/identify.rs b/crates/trusted-server-core/src/ec/identify.rs index c22a592461..e9983945d5 100644 --- a/crates/trusted-server-core/src/ec/identify.rs +++ b/crates/trusted-server-core/src/ec/identify.rs @@ -16,7 +16,12 @@ use crate::openrtb::{Eid, Uid}; use crate::settings::Settings; use super::EcContext; +use super::identity::catalog::ProviderCatalog; +use super::identity::policy::PolicySignals; +use super::identity::servable::{ServeContext, is_servable, kv_opted_out}; +use super::identity::{SourceDomain, UnixTime}; use super::kv::KvIdentityGraph; +use super::kv_types::KvEntry; use super::log_id; use super::registry::PartnerRegistry; @@ -95,12 +100,10 @@ pub fn handle_identify( // write back with the live-entry TTL. log::trace!("Identify found tombstone for '{}'", log_id(ec_id)); } else { - // Extract only this partner's UID. - if let Some(partner_uid) = entry.ids.get(&partner.source_domain) - && !partner_uid.uid.is_empty() - { - uid = Some(partner_uid.uid.clone()); - } + // Extract only this partner's UID, through the shared + // servability check, against this request's visitor consent + // and never the row's stored consent snapshot. + uid = servable_partner_uid(&entry, &partner.source_domain, ec_context); // Evaluate cluster size lazily for identify responses. Existing // stored cluster_size values are reused without a prefix-list call. @@ -331,6 +334,28 @@ fn apply_cors_headers(response: &mut Response, origin: &str) { ); } +/// The partner's stored UID, when the shared servability check lets this +/// request see it. +fn servable_partner_uid( + entry: &KvEntry, + source_domain: &str, + ec_context: &EcContext, +) -> Option { + let record = entry.ids.get(source_domain)?; + let source = source_domain.parse::().ok()?; + let now = super::checked_current_timestamp()?; + let catalog = ProviderCatalog::empty(); + let opted_out = kv_opted_out(entry); + let serve = ServeContext { + now: UnixTime::from_secs(now), + consent: ec_context.consent(), + signals: PolicySignals::default(), + catalog: &catalog, + opted_out: &opted_out, + }; + is_servable(&serve, &source, record).then(|| record.uid.clone()) +} + #[cfg(test)] mod tests { use super::*; @@ -641,6 +666,52 @@ mod tests { ); } + fn identify_uid_for_record(record: crate::ec::kv_types::KvPartnerId) -> Option { + let settings = create_test_settings(); + let kv = KvIdentityGraph::in_memory("identify_store"); + let partners = vec![make_test_partner("ssp.example.com", VALID_API_TOKEN)]; + let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = crate::ec::kv_types::KvEntry::minimal("other.example.com", "x", 1); + entry.ids.insert("ssp.example.com".to_owned(), record); + kv.create(&ec_id, &entry).expect("should seed the row"); + let req = Request::builder() + .method("GET") + .uri("https://edge.test-publisher.com/identify") + .header("authorization", format!("Bearer {VALID_API_TOKEN}")) + .body(EdgeBody::empty()) + .expect("should build test request"); + let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + + let response = handle_identify(&settings, &kv, ®istry, &req, &ec_context) + .expect("should answer identify"); + let body = serde_json::from_slice::( + &response.into_body().into_bytes().unwrap_or_default(), + ) + .expect("should decode identify response JSON"); + body.get("uid") + .and_then(serde_json::Value::as_str) + .map(str::to_owned) + } + + #[test] + fn handle_identify_omits_expired_ids() { + let expired = crate::ec::kv_types::KvPartnerId { + exp: Some(1), + ..crate::ec::kv_types::KvPartnerId::new("expired-uid") + }; + assert_eq!( + identify_uid_for_record(expired), + None, + "should not return an expired ID" + ); + assert_eq!( + identify_uid_for_record(crate::ec::kv_types::KvPartnerId::new("live-uid")), + Some("live-uid".to_owned()), + "control: should return a live ID" + ); + } + #[test] fn handle_identify_denies_mismatched_browser_origin() { let settings = create_test_settings(); diff --git a/crates/trusted-server-core/src/ec/identity/catalog.rs b/crates/trusted-server-core/src/ec/identity/catalog.rs new file mode 100644 index 0000000000..deffe97725 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/catalog.rs @@ -0,0 +1,178 @@ +//! The enabled identity providers and source ownership. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use sha2::{Digest as _, Sha256}; + +use super::ids::{IdentityProviderId, SourceDomain}; +use super::provider::IdentityProvider; + +/// Enabled providers, source owners, and the derived source index and `cf`. +/// Built once per settings load. +#[derive(Clone)] +pub(crate) struct ProviderCatalog { + providers: BTreeMap>, + owners: BTreeMap, + sources: Vec, + fingerprint: String, +} + +impl core::fmt::Debug for ProviderCatalog { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + formatter + .debug_struct("ProviderCatalog") + .field("providers", &self.providers.keys().collect::>()) + .field("owners", &self.owners) + .field("fingerprint", &self.fingerprint) + .finish() + } +} + +impl ProviderCatalog { + /// A catalog with no providers. + #[must_use] + pub(crate) fn empty() -> Self { + Self::new(Vec::new(), BTreeMap::new()) + } + + /// Builds the catalog. Config validation (ownership, duplicates) runs + /// before this is called. + #[must_use] + pub(crate) fn new( + providers: Vec>, + owners: BTreeMap, + ) -> Self { + let providers: BTreeMap<_, _> = providers + .into_iter() + .map(|provider| (provider.descriptor().id().clone(), provider)) + .collect(); + let sources: Vec = providers + .values() + .flat_map(|provider| provider.descriptor().sources().iter().cloned()) + .chain(owners.keys().cloned()) + .collect::>() + .into_iter() + .collect(); + let mut hasher = Sha256::new(); + for (id, provider) in &providers { + for source in provider.descriptor().sources() { + let owner = owners + .get(source) + .map(IdentityProviderId::as_str) + .unwrap_or_default(); + hasher.update(format!("{id}|{source}|{owner}\n").as_bytes()); + } + } + let fingerprint = hex::encode(&hasher.finalize()[..4]); + Self { + providers, + owners, + sources, + fingerprint, + } + } + + /// A provider by ID. + #[must_use] + pub(crate) fn get(&self, id: &IdentityProviderId) -> Option<&Arc> { + self.providers.get(id) + } + + /// Every provider, ordered by ID. + pub(crate) fn providers(&self) -> impl Iterator> { + self.providers.values() + } + + /// Whether `id` is an enabled provider. + #[must_use] + pub(crate) fn is_configured(&self, id: &IdentityProviderId) -> bool { + self.providers.contains_key(id) + } + + /// The owner of `source`, if any. + #[must_use] + pub(crate) fn owner_of(&self, source: &SourceDomain) -> Option<&IdentityProviderId> { + self.owners.get(source) + } + + /// Every configured source, sorted. Indexes into this list are the `s` + /// values of `ts-eids-s`. + #[must_use] + pub(crate) fn configured_sources(&self) -> &[SourceDomain] { + &self.sources + } + + /// `cf`: 8 hex digits over providers, sources and owners. + #[must_use] + pub(crate) fn fingerprint(&self) -> &str { + &self.fingerprint + } +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use super::*; + use crate::ec::identity::test_support::fixed_provider; + + #[test] + fn indexes_sources_sorted_and_fingerprints_ownership() { + let catalog = ProviderCatalog::new( + vec![ + fixed_provider("p", &["b.example.com", "a.example.com"]), + fixed_provider("q", &["c.example.com"]), + ], + BTreeMap::new(), + ); + assert_eq!( + catalog + .configured_sources() + .iter() + .map(SourceDomain::as_str) + .collect::>(), + vec!["a.example.com", "b.example.com", "c.example.com"], + "should sort and de-duplicate sources" + ); + assert_eq!(catalog.fingerprint().len(), 8, "cf is 8 hex digits"); + assert!( + catalog.is_configured(&"p".parse().expect("should parse id")), + "p is configured" + ); + assert!( + !catalog.is_configured(&"z".parse().expect("should parse id")), + "z is not" + ); + + let owned = ProviderCatalog::new( + vec![ + fixed_provider("p", &["b.example.com", "a.example.com"]), + fixed_provider("q", &["c.example.com"]), + ], + [( + "a.example.com".parse().expect("should parse source"), + "p".parse().expect("should parse id"), + )] + .into_iter() + .collect(), + ); + assert_ne!( + owned.fingerprint(), + catalog.fingerprint(), + "ownership changes cf" + ); + assert_eq!( + owned + .owner_of(&"a.example.com".parse().expect("should parse source")) + .map(IdentityProviderId::as_str), + Some("p"), + "should report the owner" + ); + assert_eq!( + ProviderCatalog::empty().configured_sources().len(), + 0, + "empty catalog" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 6da909864d..d327e63efa 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -13,6 +13,7 @@ reason = "crate-private identity helpers are wired into requests by later commits" )] +pub(crate) mod catalog; pub mod descriptor; pub mod exchange; pub mod ids; @@ -21,6 +22,7 @@ pub(crate) mod policy; pub mod provider; pub mod resolution; pub(crate) mod revision; +pub(crate) mod servable; #[cfg(test)] pub(crate) mod test_support; diff --git a/crates/trusted-server-core/src/ec/identity/servable.rs b/crates/trusted-server-core/src/ec/identity/servable.rs new file mode 100644 index 0000000000..8edc4df090 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/servable.rs @@ -0,0 +1,301 @@ +//! The one servability check shared by the bidstream, the admin view and +//! `/_ts/api/v1/identify`. + +use std::collections::BTreeSet; + +use super::catalog::ProviderCatalog; +use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; +use super::policy::{PolicySignals, policy_allows}; +use crate::consent::ConsentContext; +use crate::ec::kv_types::{KvEntry, KvPartnerId}; + +/// What a servability decision needs from the request. +pub(crate) struct ServeContext<'a> { + /// The request's time. + pub(crate) now: UnixTime, + /// The visitor's consent on this request, never the row's stored copy. + pub(crate) consent: &'a ConsentContext, + /// Signals the consent context does not carry. + pub(crate) signals: PolicySignals, + /// Enabled providers and owners. + pub(crate) catalog: &'a ProviderCatalog, + /// Providers with a sticky opt-out: KV `ps.o` plus `ts-eids-r` opt-outs. + pub(crate) opted_out: &'a BTreeSet, +} + +/// Providers whose KV `ps.o` is set and not cleared by a strictly later +/// `ps.oc`. An opt-out and a clear in the same second leave it opted out. +#[must_use] +pub(crate) fn kv_opted_out(entry: &KvEntry) -> BTreeSet { + entry + .ps + .iter() + .filter(|(_, state)| { + state + .o + .as_ref() + .is_some_and(|opt_out| state.oc.is_none_or(|cleared| cleared <= opt_out.at)) + }) + .filter_map(|(provider, _)| provider.parse().ok()) + .collect() +} + +/// Whether a stored record may be sent from the server. +/// +/// In order: an empty UID or an expired record is never served. A record +/// with a writer needs that writer to be enabled, not opted out, and allowed +/// by its consent policy. An owned source also needs the owner's policy, and +/// when the owner declares the source as expiring, only a record the owner +/// wrote with an unexpired `exp` is served (the rollback rule). Anything else +/// is servable: the caller already applied the generic gates. +#[must_use] +pub(crate) fn is_servable( + context: &ServeContext<'_>, + source: &SourceDomain, + record: &KvPartnerId, +) -> bool { + if record.uid.is_empty() { + return false; + } + if record.exp.is_some_and(|exp| context.now.as_secs() >= exp) { + return false; + } + let writer = record + .w + .as_deref() + .and_then(|writer| writer.parse::().ok()); + if record.w.is_some() { + let Some(writer) = writer.as_ref() else { + return false; + }; + let Some(provider) = context.catalog.get(writer) else { + return false; + }; + if context.opted_out.contains(writer) + || !policy_allows( + provider.descriptor().consent_policy(), + context.consent, + context.signals, + ) + { + return false; + } + } + if let Some(owner) = context.catalog.owner_of(source) { + let Some(owner_provider) = context.catalog.get(owner) else { + return false; + }; + let descriptor = owner_provider.descriptor(); + if !policy_allows( + descriptor.consent_policy(), + context.consent, + context.signals, + ) { + return false; + } + if descriptor.is_expiring(source) + && (writer.as_ref() != Some(owner) || record.exp.is_none()) + { + return false; + } + } + true +} + +#[cfg(test)] +mod tests { + use std::collections::{BTreeMap, BTreeSet}; + + use super::*; + use crate::ec::identity::descriptor::ConsentPolicy; + use crate::ec::identity::test_support::{ + TestSignals, allowing_consent, consent_with, fixed_provider, provider_with_policy, + }; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId, KvProviderState}; + + fn record(writer: Option<&str>, exp: Option) -> KvPartnerId { + KvPartnerId { + w: writer.map(str::to_owned), + exp, + ..KvPartnerId::new("u") + } + } + + fn check( + catalog: &ProviderCatalog, + opted_out: &BTreeSet, + source: &str, + record: &KvPartnerId, + ) -> bool { + let consent = allowing_consent(); + let context = ServeContext { + now: UnixTime::from_secs(100), + consent: &consent, + signals: PolicySignals::default(), + catalog, + opted_out, + }; + is_servable( + &context, + &source.parse().expect("should parse source"), + record, + ) + } + + #[test] + fn expiry_writer_presence_and_opt_out() { + let catalog = ProviderCatalog::new( + vec![fixed_provider("p", &["a.example.com"])], + BTreeMap::new(), + ); + let none = BTreeSet::new(); + assert!( + check( + &catalog, + &none, + "a.example.com", + &record(Some("p"), Some(101)) + ), + "control: valid" + ); + assert!( + !check( + &catalog, + &none, + "a.example.com", + &record(Some("p"), Some(100)) + ), + "expired at exp" + ); + assert!( + !check( + &catalog, + &none, + "a.example.com", + &record(Some("gone"), None) + ), + "unknown writer" + ); + let opted: BTreeSet<_> = ["p".parse().expect("should parse id")] + .into_iter() + .collect(); + assert!( + !check(&catalog, &opted, "a.example.com", &record(Some("p"), None)), + "opted-out writer" + ); + assert!( + check(&catalog, &none, "b.example.com", &record(None, None)), + "browser record, unowned" + ); + assert!( + !check(&catalog, &none, "b.example.com", &KvPartnerId::new("")), + "empty uid" + ); + } + + #[test] + fn rollback_rule_for_expiring_owned_sources() { + let catalog = ProviderCatalog::new( + vec![provider_with_policy( + "owner", + &["a.example.com"], + ConsentPolicy::eid_persistence_only(), + true, + )], + [( + "a.example.com".parse().expect("should parse source"), + "owner".parse().expect("should parse id"), + )] + .into_iter() + .collect(), + ); + let none = BTreeSet::new(); + assert!( + check( + &catalog, + &none, + "a.example.com", + &record(Some("owner"), Some(200)) + ), + "control: owner value" + ); + assert!( + !check(&catalog, &none, "a.example.com", &record(None, None)), + "browser-seeded value is never served from the server" + ); + assert!( + !check( + &catalog, + &none, + "a.example.com", + &record(Some("owner"), None) + ), + "stripped exp" + ); + } + + #[test] + fn writer_policy_applies_to_unowned_sources() { + let strict = ConsentPolicy::strict().with_tcf_vendor(9999); + let catalog = ProviderCatalog::new( + vec![provider_with_policy("p", &["a.example.com"], strict, false)], + BTreeMap::new(), + ); + let consent = consent_with(&TestSignals { + gdpr: true, + purposes: vec![1, 4], + ..TestSignals::default() + }); + let none = BTreeSet::new(); + let context = ServeContext { + now: UnixTime::from_secs(100), + consent: &consent, + signals: PolicySignals::default(), + catalog: &catalog, + opted_out: &none, + }; + let source = "a.example.com".parse().expect("should parse source"); + assert!( + !is_servable(&context, &source, &record(Some("p"), None)), + "writer's vendor denied" + ); + assert!( + is_servable(&context, &source, &record(None, None)), + "control: browser value unaffected" + ); + } + + #[test] + fn kv_opt_outs_respect_later_clears() { + let mut entry = KvEntry::minimal("a.example.com", "u", 1); + let opt_out = KvOptOut { + at: 10, + fp: "0".repeat(16), + }; + entry.ps.insert( + "active".to_owned(), + KvProviderState { + o: Some(opt_out.clone()), + oc: Some(10), + ..KvProviderState::default() + }, + ); + entry.ps.insert( + "cleared".to_owned(), + KvProviderState { + o: Some(opt_out), + oc: Some(11), + ..KvProviderState::default() + }, + ); + let opted = kv_opted_out(&entry); + assert!( + opted.contains(&"active".parse().expect("should parse id")), + "a same-second clear leaves it opted out" + ); + assert!( + !opted.contains(&"cleared".parse().expect("should parse id")), + "a later clear clears it" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index 8603dffde9..9b50241d7c 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -81,3 +81,51 @@ pub(crate) fn consent_with(signals: &TestSignals) -> ConsentContext { pub(crate) fn allowing_consent() -> ConsentContext { consent_with(&TestSignals::default()) } + +/// A descriptor-only provider for tests. +pub(crate) struct FixedProvider(pub(crate) crate::ec::identity::ProviderDescriptor); + +impl crate::ec::identity::IdentityProvider for FixedProvider { + fn descriptor(&self) -> &crate::ec::identity::ProviderDescriptor { + &self.0 + } +} + +/// A resolving provider over `sources` with the given policy. +pub(crate) fn provider_with_policy( + id: &str, + sources: &[&str], + policy: crate::ec::identity::ConsentPolicy, + expiring: bool, +) -> std::sync::Arc { + let mut builder = crate::ec::identity::ProviderDescriptor::builder( + id.parse().expect("should parse provider id"), + ); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse source")); + } + builder = builder + .with_resolves( + crate::ec::identity::Resolves::new().with_endpoint("https://api.example.com/v1"), + ) + .with_consent_policy(policy); + if expiring { + builder = builder.with_expiring_ids(); + } + std::sync::Arc::new(FixedProvider( + builder.build().expect("should build provider"), + )) +} + +/// A resolving provider with generic pull's permissive policy. +pub(crate) fn fixed_provider( + id: &str, + sources: &[&str], +) -> std::sync::Arc { + provider_with_policy( + id, + sources, + crate::ec::identity::ConsentPolicy::eid_persistence_only(), + false, + ) +} diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index b3a360f43f..d0d5f17f2a 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -83,6 +83,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { ("ec/finalize.rs", include_str!("ec/finalize.rs")), ("ec/generation.rs", include_str!("ec/generation.rs")), ("ec/identify.rs", include_str!("ec/identify.rs")), + ( + "ec/identity/catalog.rs", + include_str!("ec/identity/catalog.rs"), + ), ( "ec/identity/descriptor.rs", include_str!("ec/identity/descriptor.rs"), @@ -110,6 +114,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/revision.rs", include_str!("ec/identity/revision.rs"), ), + ( + "ec/identity/servable.rs", + include_str!("ec/identity/servable.rs"), + ), ( "ec/identity/test_support.rs", include_str!("ec/identity/test_support.rs"), From 2a62a1d19801623ddfbf9765892c140327fa6e8f Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:12:29 +0530 Subject: [PATCH 16/33] Add IdentityMutation and its merge into a live identity row Removals apply first and win equal-second ties, a provider opt-out removes every record that provider wrote, stateful results apply atomically, refresh anchors survive while their owner holds state, and an invalid row falls back to removals only. Signed-off-by: dhruv8sh --- .../src/ec/identity/mod.rs | 1 + .../src/ec/identity/mutation.rs | 1311 +++++++++++++++++ .../src/ec/identity/test_support.rs | 7 + .../src/migration_guards.rs | 4 + 4 files changed, 1323 insertions(+) create mode 100644 crates/trusted-server-core/src/ec/identity/mutation.rs diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index d327e63efa..a52f511924 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -18,6 +18,7 @@ pub mod descriptor; pub mod exchange; pub mod ids; pub mod input; +pub(crate) mod mutation; pub(crate) mod policy; pub mod provider; pub mod resolution; diff --git a/crates/trusted-server-core/src/ec/identity/mutation.rs b/crates/trusted-server-core/src/ec/identity/mutation.rs new file mode 100644 index 0000000000..2d56c63e10 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/mutation.rs @@ -0,0 +1,1311 @@ +//! `IdentityMutation`: every KV change of the writer request, applied to the +//! live row with one write. +//! +//! Changes are checked against the live entry, not the snapshot they were +//! derived from, so a later writer can re-apply the same changes, re-derived +//! or still staged, without losing a concurrent tab's newer data. + +use std::collections::BTreeMap; + +use super::catalog::ProviderCatalog; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::input::RandomSource; +use super::revision::{ + IdentityKeys, Revision, invalidated_uid_hash, new_revision, record_revision, +}; +use crate::ec::kv_types::{ + KvEntry, KvNetwork, KvOptOut, KvPartnerId, KvProviderState, MAX_INVALIDATED_UIDS, +}; + +/// One change to a source's record. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum PartnerIdChange { + /// From browser ingest, a KV-gap entry or an unowned provider. Applies + /// only if the source is absent, its record is expired, or its writer is + /// no longer configured. + FillMissing { + source: SourceDomain, + record: KvPartnerId, + writer: Option, + staged_at: Option, + }, + /// From the owning provider. Applies if the live record's revision equals + /// `expected`, or if `expected` is `None` and the source has no valid ID + /// this provider wrote. A browser-seeded or legacy value never blocks the + /// owner. + OwnerWrite { + source: SourceDomain, + record: KvPartnerId, + writer: IdentityProviderId, + expected: Option, + staged_at: Option, + }, + /// From `Resolution::invalidate` or `also_invalidate`. Applies only if the + /// revision matches. + OwnerRemove { + source: SourceDomain, + writer: IdentityProviderId, + expected: Revision, + }, + /// From `Resolution::opted_out`, or a staged removal without a revision. + /// Removes the source's record if this writer wrote it and, when + /// `staged_at` is set, acquired it at or before then. A tie goes to the + /// removal. + WriterRemove { + source: SourceDomain, + writer: IdentityProviderId, + staged_at: Option, + }, +} + +impl PartnerIdChange { + fn is_removal(&self) -> bool { + matches!(self, Self::OwnerRemove { .. } | Self::WriterRemove { .. }) + } +} + +/// The provider state a call read, for the atomic check of its result. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ExpectedState { + /// The call read no state: no check. + NotRead, + /// The call read no stored state. + Absent, + /// The call read this state JSON. + Json(String), +} + +/// How a change affects `ps.st`. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) enum StateUpdate { + #[default] + Keep, + Replace(serde_json::Value), + Clear, +} + +/// Changes to one provider's `ps` entry. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct ProviderStateChange { + /// Next eligible time. Merged as the later value. + pub(crate) n: Option, + /// Consecutive failures. Replaced when set. + pub(crate) f: Option, + /// A newly reported opt-out. + pub(crate) opt_out: Option, + /// A re-consent clear of the opt-out, at this time. + pub(crate) opt_out_clear: Option, + /// Keyed hashes of newly invalidated UIDs. + pub(crate) iv_add: Vec, + /// Private state. + pub(crate) st: StateUpdate, +} + +impl ProviderStateChange { + fn merge(&mut self, other: Self) { + self.n = later(self.n, other.n); + if other.f.is_some() { + self.f = other.f; + } + if let Some(opt_out) = other.opt_out + && self + .opt_out + .as_ref() + .is_none_or(|current| opt_out.at > current.at) + { + self.opt_out = Some(opt_out); + } + self.opt_out_clear = later(self.opt_out_clear, other.opt_out_clear); + for hash in other.iv_add { + if !self.iv_add.contains(&hash) { + self.iv_add.push(hash); + } + } + if other.st != StateUpdate::Keep { + self.st = other.st; + } + } + + fn has_removal(&self) -> bool { + self.opt_out.is_some() || self.opt_out_clear.is_some() || !self.iv_add.is_empty() + } +} + +fn later(current: Option, other: Option) -> Option { + match (current, other) { + (Some(current), Some(other)) => Some(current.max(other)), + (current, other) => current.or(other), + } +} + +/// The positive part of one stateful provider's result, applied whole or not +/// at all. Its removals, opt-out and `iv` hashes are not part of it: they go +/// into the mutation's plain changes and apply on their own. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct ProviderResult { + pub(crate) provider: IdentityProviderId, + /// `OwnerWrite` and `FillMissing` changes for the sources it returned. + pub(crate) ids: Vec, + /// The state replacement or clear, and the new `n` and `f`. + pub(crate) state: ProviderStateChange, + /// The `ps.st` the call read. + pub(crate) expected_state: ExpectedState, +} + +/// Every KV change of one writer request. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct IdentityMutation { + partner_ids: Vec, + provider_results: Vec, + provider_state: BTreeMap, + cluster_size: Option, +} + +impl IdentityMutation { + /// Adds a change. + pub(crate) fn add_change(&mut self, change: PartnerIdChange) { + self.partner_ids.push(change); + } + + /// Adds a stateful provider's positive result. + pub(crate) fn add_result(&mut self, result: ProviderResult) { + self.provider_results.push(result); + } + + /// Merges a change into the provider's pending state change. + pub(crate) fn merge_state( + &mut self, + provider: IdentityProviderId, + change: ProviderStateChange, + ) { + self.provider_state + .entry(provider) + .or_default() + .merge(change); + } + + /// Stores an evaluated cluster size. + pub(crate) fn set_cluster_size(&mut self, cluster_size: u32) { + self.cluster_size = Some(cluster_size); + } + + /// Whether there is nothing to apply. + #[must_use] + pub(crate) fn is_empty(&self) -> bool { + self.partner_ids.is_empty() + && self.provider_results.is_empty() + && self.provider_state.is_empty() + && self.cluster_size.is_none() + } + + /// Whether it holds a removal, opt-out, clear or `iv` hash. + #[must_use] + pub(crate) fn has_removals(&self) -> bool { + self.partner_ids.iter().any(PartnerIdChange::is_removal) + || self + .provider_state + .values() + .any(ProviderStateChange::has_removal) + } + + /// Only the removals, opt-outs, clears and `iv` hashes. + #[must_use] + pub(crate) fn removals_only(&self) -> Self { + Self { + partner_ids: self + .partner_ids + .iter() + .filter(|change| change.is_removal()) + .cloned() + .collect(), + provider_results: Vec::new(), + provider_state: self + .provider_state + .iter() + .map(|(provider, change)| { + ( + provider.clone(), + ProviderStateChange { + opt_out: change.opt_out.clone(), + opt_out_clear: change.opt_out_clear, + iv_add: change.iv_add.clone(), + ..ProviderStateChange::default() + }, + ) + }) + .filter(|(_, change)| change.has_removal()) + .collect(), + cluster_size: None, + } + } +} + +/// What applying a mutation needs besides the live entry. +pub(crate) struct ApplyContext<'a> { + /// Unix seconds. + pub(crate) now: u64, + pub(crate) keys: &'a IdentityKeys, + pub(crate) random: &'a dyn RandomSource, + pub(crate) catalog: &'a ProviderCatalog, +} + +/// The result of applying a mutation. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct Applied { + /// The entry to write. + pub(crate) entry: KvEntry, + /// Whether it differs from the live entry. + pub(crate) changed: bool, + /// Whether the full result failed validation and only removals applied. + pub(crate) removals_only: bool, +} + +/// Applies `mutation` to the live entry. +/// +/// In order: +/// +/// 1. A tombstone drops everything: `None`. +/// 2. Removals: revision-checked `OwnerRemove`, `WriterRemove`, then new +/// opt-outs (each removes every record its provider wrote and clears its +/// state), then clears strictly later than the provider's latest opt-out, +/// then `iv` hashes (newest four kept). +/// 3. Positive changes by their own rules. Changes from an opted-out +/// provider, and staged changes at or before its latest opt-out, are +/// skipped. Each stateful result applies whole, or not at all when any of +/// its IDs would not apply or the live state is not the one it read. +/// 4. `ps` changes and `cluster_size`, then pruning of unconfigured +/// providers' `ps` and of expired records that are not refresh anchors. +/// 5. Validation. On failure only the removals are kept, and if that fails +/// too the mutation is dropped: `None`. +/// 6. `changed` reports whether anything differs from the live entry. +#[must_use] +pub(crate) fn apply( + mutation: &IdentityMutation, + live: &KvEntry, + context: &ApplyContext<'_>, +) -> Option { + if !live.consent.ok { + return None; + } + + let mut entry = live.clone(); + apply_removals(&mut entry, mutation, context); + let mut removals_entry = entry.clone(); + + apply_positive(&mut entry, mutation, context); + apply_state(&mut entry, mutation); + prune(&mut entry, context); + + let (entry, removals_only) = match entry.validate() { + Ok(()) => (entry, false), + Err(message) => { + log::warn!( + "Identity mutation produced an invalid row, keeping removals only: {message}" + ); + prune(&mut removals_entry, context); + if let Err(message) = removals_entry.validate() { + log::warn!( + "Identity mutation removals-only row is invalid, dropping it: {message}" + ); + return None; + } + (removals_entry, true) + } + }; + + let changed = &entry != live; + Some(Applied { + entry, + changed, + removals_only, + }) +} + +fn apply_removals(entry: &mut KvEntry, mutation: &IdentityMutation, context: &ApplyContext<'_>) { + for change in &mutation.partner_ids { + match change { + PartnerIdChange::OwnerRemove { + source, expected, .. + } => { + let matches = entry + .ids + .get(source.as_str()) + .is_some_and(|record| record_revision(context.keys, record) == *expected); + if matches { + entry.ids.remove(source.as_str()); + } + } + PartnerIdChange::WriterRemove { + source, + writer, + staged_at, + } => { + let matches = entry.ids.get(source.as_str()).is_some_and(|record| { + record.w.as_deref() == Some(writer.as_str()) + && staged_at.is_none_or(|staged| record.at.unwrap_or(0) <= staged) + }); + if matches { + entry.ids.remove(source.as_str()); + } + } + PartnerIdChange::FillMissing { .. } | PartnerIdChange::OwnerWrite { .. } => {} + } + } + + for (provider, change) in &mutation.provider_state { + if let Some(opt_out) = &change.opt_out { + apply_opt_out(entry, provider, opt_out); + } + } + for (provider, change) in &mutation.provider_state { + if let Some(clear) = change.opt_out_clear { + apply_opt_out_clear(entry, provider, change.opt_out.as_ref(), clear); + } + if !change.iv_add.is_empty() { + let state = entry.ps.entry(provider.to_string()).or_default(); + for hash in &change.iv_add { + if !state.iv.contains(hash) { + state.iv.push(hash.clone()); + } + } + let excess = state.iv.len().saturating_sub(MAX_INVALIDATED_UIDS); + state.iv.drain(..excess); + } + } +} + +/// Applies an opt-out reported after the live one and at or after the live +/// clear. It removes every record the provider wrote, at any revision, and +/// clears its state. +fn apply_opt_out(entry: &mut KvEntry, provider: &IdentityProviderId, opt_out: &KvOptOut) { + let state = entry.ps.entry(provider.to_string()).or_default(); + let newer = state + .o + .as_ref() + .is_none_or(|current| opt_out.at > current.at); + let not_stale = state.oc.is_none_or(|cleared| opt_out.at >= cleared); + if !newer || !not_stale { + return; + } + state.o = Some(opt_out.clone()); + state.st = None; + entry + .ids + .retain(|_, record| record.w.as_deref() != Some(provider.as_str())); +} + +/// Stores a clear only when it is strictly later than the provider's latest +/// opt-out, live or staged. An opt-out and a clear in the same second leave +/// the provider opted out. +fn apply_opt_out_clear( + entry: &mut KvEntry, + provider: &IdentityProviderId, + staged_opt_out: Option<&KvOptOut>, + clear: u64, +) { + let state = entry.ps.entry(provider.to_string()).or_default(); + let latest_opt_out = state + .o + .as_ref() + .map(|opt_out| opt_out.at) + .max(staged_opt_out.map(|opt_out| opt_out.at)); + let Some(latest_opt_out) = latest_opt_out else { + return; + }; + if clear > latest_opt_out { + state.oc = later(state.oc, Some(clear)); + } +} + +fn opt_out_of(entry: &KvEntry, provider: &IdentityProviderId) -> Option<(u64, bool)> { + let state = entry.ps.get(provider.as_str())?; + let opt_out = state.o.as_ref()?; + let active = state.oc.is_none_or(|cleared| cleared <= opt_out.at); + Some((opt_out.at, active)) +} + +/// Whether a provider's opt-out blocks a change staged at `staged_at`. +fn blocked_by_opt_out( + entry: &KvEntry, + provider: &IdentityProviderId, + staged_at: Option, +) -> bool { + match opt_out_of(entry, provider) { + Some((_, true)) => true, + Some((at, false)) => staged_at.is_some_and(|staged| staged <= at), + None => false, + } +} + +fn record_is_expired(record: &KvPartnerId, now: u64) -> bool { + record.exp.is_some_and(|exp| now >= exp) +} + +fn writer_is_configured(record: &KvPartnerId, catalog: &ProviderCatalog) -> bool { + record.w.as_deref().is_none_or(|writer| { + writer + .parse::() + .is_ok_and(|writer| catalog.is_configured(&writer)) + }) +} + +fn invalidated_by_owner( + entry: &KvEntry, + source: &SourceDomain, + uid: &str, + context: &ApplyContext<'_>, +) -> bool { + let Some(owner) = context.catalog.owner_of(source) else { + return false; + }; + let Some(state) = entry.ps.get(owner.as_str()) else { + return false; + }; + let hash = invalidated_uid_hash(context.keys, owner, uid); + state.iv.contains(&hash) +} + +fn change_applies(entry: &KvEntry, change: &PartnerIdChange, context: &ApplyContext<'_>) -> bool { + match change { + PartnerIdChange::FillMissing { + source, + record, + writer, + staged_at, + } => { + if writer + .as_ref() + .is_some_and(|writer| blocked_by_opt_out(entry, writer, *staged_at)) + { + return false; + } + if invalidated_by_owner(entry, source, &record.uid, context) { + return false; + } + entry.ids.get(source.as_str()).is_none_or(|existing| { + record_is_expired(existing, context.now) + || !writer_is_configured(existing, context.catalog) + }) + } + PartnerIdChange::OwnerWrite { + source, + writer, + expected, + staged_at, + .. + } => { + if blocked_by_opt_out(entry, writer, *staged_at) { + return false; + } + let existing = entry.ids.get(source.as_str()); + match expected { + Some(expected) => existing + .is_some_and(|record| record_revision(context.keys, record) == *expected), + None => !existing.is_some_and(|record| { + record.w.as_deref() == Some(writer.as_str()) + && !record_is_expired(record, context.now) + }), + } + } + PartnerIdChange::OwnerRemove { .. } | PartnerIdChange::WriterRemove { .. } => false, + } +} + +fn write_change(entry: &mut KvEntry, change: &PartnerIdChange, context: &ApplyContext<'_>) { + let (source, record, writer) = match change { + PartnerIdChange::FillMissing { + source, + record, + writer, + .. + } => (source, record, writer.as_ref()), + PartnerIdChange::OwnerWrite { + source, + record, + writer, + .. + } => (source, record, Some(writer)), + PartnerIdChange::OwnerRemove { .. } | PartnerIdChange::WriterRemove { .. } => return, + }; + let mut written = record.clone(); + written.w = writer.map(ToString::to_string); + written.rv = Some(new_revision(context.random).as_str().to_owned()); + written.at = record.at.or(Some(context.now)); + entry.ids.insert(source.to_string(), written); +} + +fn state_matches(entry: &KvEntry, provider: &IdentityProviderId, expected: &ExpectedState) -> bool { + let live = entry + .ps + .get(provider.as_str()) + .and_then(|state| state.st.as_ref()); + match expected { + ExpectedState::NotRead => true, + ExpectedState::Absent => live.is_none(), + ExpectedState::Json(json) => serde_json::from_str::(json) + .ok() + .is_some_and(|read| live == Some(&read)), + } +} + +fn apply_positive(entry: &mut KvEntry, mutation: &IdentityMutation, context: &ApplyContext<'_>) { + for change in &mutation.partner_ids { + if change_applies(entry, change, context) { + write_change(entry, change, context); + } + } + for result in &mutation.provider_results { + let whole = result + .ids + .iter() + .all(|change| change_applies(entry, change, context)) + && state_matches(entry, &result.provider, &result.expected_state); + if !whole { + log::debug!( + "Dropping identity result for {} that lost to a newer write", + result.provider + ); + continue; + } + for change in &result.ids { + write_change(entry, change, context); + } + apply_provider_state(entry, &result.provider, &result.state); + } +} + +fn apply_provider_state( + entry: &mut KvEntry, + provider: &IdentityProviderId, + change: &ProviderStateChange, +) { + let state: &mut KvProviderState = entry.ps.entry(provider.to_string()).or_default(); + state.n = later(state.n, change.n); + if let Some(failures) = change.f { + state.f = failures; + } + match &change.st { + StateUpdate::Keep => {} + StateUpdate::Replace(value) => state.st = Some(value.clone()), + StateUpdate::Clear => state.st = None, + } +} + +fn apply_state(entry: &mut KvEntry, mutation: &IdentityMutation) { + for (provider, change) in &mutation.provider_state { + let opted_out = opt_out_of(entry, provider).is_some_and(|(_, active)| active); + let mut positive = change.clone(); + // An opted-out provider keeps its state cleared. + if opted_out { + positive.st = StateUpdate::Keep; + } + apply_provider_state(entry, provider, &positive); + } + if let Some(cluster_size) = mutation.cluster_size { + entry + .network + .get_or_insert(KvNetwork { cluster_size: None }) + .cluster_size = Some(cluster_size); + } +} + +/// Drops the `ps` of unconfigured providers and expired records, except +/// refresh anchors: an expired record of an owned source whose owner still +/// holds state and whose `ru` has not passed. +fn prune(entry: &mut KvEntry, context: &ApplyContext<'_>) { + entry.ps.retain(|provider, _| { + provider + .parse::() + .is_ok_and(|provider| context.catalog.is_configured(&provider)) + }); + let states = entry.ps.clone(); + entry.ids.retain(|source, record| { + if !record_is_expired(record, context.now) { + return true; + } + let owner_holds_state = source + .parse::() + .ok() + .and_then(|source| context.catalog.owner_of(&source).cloned()) + .and_then(|owner| states.get(owner.as_str())) + .is_some_and(|state| state.st.is_some()); + owner_holds_state && record.ru.is_some_and(|until| context.now < until) + }); +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::*; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::revision::invalidated_uid_hash; + use crate::ec::identity::test_support::{fixed_provider, test_keys}; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId, KvProviderState}; + + const NOW: u64 = 1_000; + + fn src(name: &str) -> SourceDomain { + name.parse().expect("should parse source") + } + + fn pid(name: &str) -> IdentityProviderId { + name.parse().expect("should parse id") + } + + fn catalog() -> ProviderCatalog { + ProviderCatalog::new( + vec![ + fixed_provider("owner", &["a.example.com", "c.example.com"]), + fixed_provider("other", &["b.example.com"]), + ], + [(src("a.example.com"), pid("owner"))].into_iter().collect(), + ) + } + + fn run(mutation: &IdentityMutation, live: &KvEntry) -> Option { + let keys = test_keys(); + let random = SeededRandom::new(1); + let catalog = catalog(); + apply( + mutation, + live, + &ApplyContext { + now: NOW, + keys: &keys, + random: &random, + catalog: &catalog, + }, + ) + } + + fn owned(uid: &str, rv: &str) -> KvPartnerId { + KvPartnerId { + w: Some("owner".to_owned()), + rv: Some(rv.to_owned()), + at: Some(900), + ..KvPartnerId::new(uid) + } + } + + fn fill(source: &str, uid: &str) -> PartnerIdChange { + PartnerIdChange::FillMissing { + source: src(source), + record: KvPartnerId::browser(uid, Some(NOW)), + writer: None, + staged_at: None, + } + } + + #[test] + fn tombstone_drops_everything() { + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill("b.example.com", "u")); + assert!( + run(&mutation, &KvEntry::tombstone(1)).is_none(), + "should drop on a tombstone" + ); + } + + #[test] + fn empty_mutation_changes_nothing() { + let live = KvEntry::minimal("b.example.com", "x", 1); + let applied = run(&IdentityMutation::default(), &live).expect("should apply"); + assert!(!applied.changed, "nothing to change"); + assert!(IdentityMutation::default().is_empty(), "default is empty"); + } + + #[test] + fn fill_missing_fills_absent_and_expired_only() { + let mut live = KvEntry::minimal("b.example.com", "existing", 1); + live.ids.insert( + "c.example.com".to_owned(), + KvPartnerId { + exp: Some(NOW), + ..KvPartnerId::new("old") + }, + ); + let mut mutation = IdentityMutation::default(); + for source in ["b.example.com", "c.example.com", "d.example.com"] { + mutation.add_change(fill(source, "new")); + } + + let applied = run(&mutation, &live).expect("should apply"); + + assert!(applied.changed, "should change"); + assert_eq!( + applied.entry.ids["b.example.com"].uid, "existing", + "should not replace a live value" + ); + assert_eq!( + applied.entry.ids["c.example.com"].uid, "new", + "should replace an expired value" + ); + assert_eq!( + applied.entry.ids["d.example.com"].uid, "new", + "should fill an absent source" + ); + } + + #[test] + fn fill_missing_replaces_a_record_whose_writer_is_gone() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "d.example.com".to_owned(), + KvPartnerId { + w: Some("retired".to_owned()), + ..KvPartnerId::new("orphaned") + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill("d.example.com", "fresh")); + let applied = run(&mutation, &live).expect("should apply"); + assert_eq!( + applied.entry.ids["d.example.com"].uid, "fresh", + "a record from an unconfigured writer counts as absent" + ); + } + + #[test] + fn owner_write_checks_revision_and_gets_a_fresh_rv() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids + .insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + + let mut stale = IdentityMutation::default(); + stale.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("v2"), + writer: pid("owner"), + expected: Some(Revision::from_stored("2222222222222222")), + staged_at: None, + }); + let applied = run(&stale, &live).expect("should apply"); + assert!( + !applied.changed, + "a stale revision loses to the newer write" + ); + + let mut fresh = IdentityMutation::default(); + fresh.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("v2"), + writer: pid("owner"), + expected: Some(Revision::from_stored("1111111111111111")), + staged_at: None, + }); + let applied = run(&fresh, &live).expect("should apply"); + let record = &applied.entry.ids["a.example.com"]; + assert_eq!(record.uid, "v2", "should write"); + assert_eq!(record.w.as_deref(), Some("owner"), "should set the writer"); + assert!( + record + .rv + .as_deref() + .is_some_and(|rv| rv != "1111111111111111" && rv.len() == 16), + "should draw a new rv" + ); + assert_eq!(record.at, Some(NOW), "should stamp at"); + } + + #[test] + fn browser_seeded_value_never_blocks_the_owner() { + let mut live = KvEntry::minimal("a.example.com", "seeded", 1); + live.ids.get_mut("a.example.com").expect("should hold a").at = Some(5); + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("owned"), + writer: pid("owner"), + expected: None, + staged_at: None, + }); + let applied = run(&mutation, &live).expect("should apply"); + assert_eq!( + applied.entry.ids["a.example.com"].uid, "owned", + "owner takes over a seeded value" + ); + } + + #[test] + fn owner_write_without_revision_loses_to_its_own_valid_value() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids + .insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("v2"), + writer: pid("owner"), + expected: None, + staged_at: None, + }); + let applied = run(&mutation, &live).expect("should apply"); + assert_eq!( + applied.entry.ids["a.example.com"].uid, "v1", + "a write without a revision cannot overwrite the owner's own valid value" + ); + } + + #[test] + fn owner_remove_needs_the_matching_revision() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids + .insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + let remove = |rv: &str| { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + writer: pid("owner"), + expected: Revision::from_stored(rv), + }); + run(&mutation, &live).expect("should apply") + }; + assert!( + !remove("1111111111111111") + .entry + .ids + .contains_key("a.example.com"), + "matching revision removes" + ); + assert!( + remove("2222222222222222") + .entry + .ids + .contains_key("a.example.com"), + "a newer write survives" + ); + } + + #[test] + fn removals_win_ties_and_opt_out_removes_everything_the_writer_wrote() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids + .insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + live.ids.insert( + "c.example.com".to_owned(), + KvPartnerId { + w: Some("owner".to_owned()), + at: Some(950), + ..KvPartnerId::new("unowned-by-owner") + }, + ); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + st: Some(json!({ "token": "t" })), + ..KvProviderState::default() + }, + ); + + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + pid("owner"), + ProviderStateChange { + opt_out: Some(KvOptOut { + at: NOW, + fp: "0".repeat(16), + }), + ..ProviderStateChange::default() + }, + ); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("v2"), + writer: pid("owner"), + expected: None, + staged_at: Some(NOW), + }); + + let applied = run(&mutation, &live).expect("should apply"); + + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should remove the owned record" + ); + assert!( + !applied.entry.ids.contains_key("c.example.com"), + "should remove the unowned record it wrote" + ); + assert_eq!( + applied.entry.ids["b.example.com"].uid, "x", + "control: other records untouched" + ); + assert_eq!( + applied.entry.ps["owner"] + .o + .as_ref() + .map(|opt_out| opt_out.at), + Some(NOW), + "should store the opt-out" + ); + assert_eq!(applied.entry.ps["owner"].st, None, "should clear the state"); + } + + #[test] + fn writer_remove_without_revision_respects_staged_at() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids + .insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + let remove = |staged_at| { + let mut mutation = IdentityMutation::default(); + mutation.add_change(PartnerIdChange::WriterRemove { + source: src("a.example.com"), + writer: pid("owner"), + staged_at: Some(staged_at), + }); + run(&mutation, &live).expect("should apply") + }; + assert!( + !remove(900).entry.ids.contains_key("a.example.com"), + "same second: the removal wins" + ); + assert!( + remove(899).entry.ids.contains_key("a.example.com"), + "re-acquired after sa: kept" + ); + } + + #[test] + fn opt_out_and_clear_in_the_same_second_leave_it_opted_out() { + let live = KvEntry::minimal("b.example.com", "x", 1); + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + pid("owner"), + ProviderStateChange { + opt_out: Some(KvOptOut { + at: NOW, + fp: "0".repeat(16), + }), + opt_out_clear: Some(NOW), + ..ProviderStateChange::default() + }, + ); + let applied = run(&mutation, &live).expect("should apply"); + assert!( + applied.entry.ps["owner"].o.is_some(), + "should stay opted out" + ); + assert_eq!( + applied.entry.ps["owner"].oc, None, + "should drop a clear that is not later" + ); + } + + #[test] + fn later_clear_lifts_the_opt_out_but_older_staged_entries_stay_out() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 500, + fp: "0".repeat(16), + }), + ..KvProviderState::default() + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + pid("owner"), + ProviderStateChange { + opt_out_clear: Some(600), + ..ProviderStateChange::default() + }, + ); + mutation.add_change(PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("before-opt-out"), + writer: pid("owner"), + expected: None, + staged_at: Some(400), + }); + mutation.add_change(PartnerIdChange::FillMissing { + source: src("c.example.com"), + record: KvPartnerId::new("after-clear"), + writer: Some(pid("owner")), + staged_at: Some(700), + }); + + let applied = run(&mutation, &live).expect("should apply"); + + assert_eq!( + applied.entry.ps["owner"].oc, + Some(600), + "should store the clear" + ); + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "an entry staged before the opt-out stays out" + ); + assert_eq!( + applied.entry.ids["c.example.com"].uid, "after-clear", + "an entry staged after the clear lands" + ); + } + + #[test] + fn invalidated_uid_is_not_reseeded() { + let keys = test_keys(); + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + iv: vec![invalidated_uid_hash(&keys, &pid("owner"), "rejected")], + ..KvProviderState::default() + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill("a.example.com", "rejected")); + mutation.add_change(fill("b.example.com", "other")); + let applied = run(&mutation, &live).expect("should apply"); + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "should not re-seed the rejected UID" + ); + assert_eq!( + applied.entry.ids["b.example.com"].uid, "x", + "control: the unowned source is untouched by iv" + ); + } + + #[test] + fn iv_hashes_keep_the_newest_four() { + let live = KvEntry::minimal("b.example.com", "x", 1); + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + pid("owner"), + ProviderStateChange { + iv_add: (0..6).map(|index| format!("{index:016x}")).collect(), + ..ProviderStateChange::default() + }, + ); + let applied = run(&mutation, &live).expect("should apply"); + assert_eq!( + applied.entry.ps["owner"].iv, + (2..6) + .map(|index| format!("{index:016x}")) + .collect::>(), + "should keep the four newest hashes" + ); + } + + #[test] + fn provider_result_applies_atomically() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + owned("newer", "3333333333333333"), + ); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + st: Some(json!({ "token": "B" })), + ..KvProviderState::default() + }, + ); + + let mut mutation = IdentityMutation::default(); + mutation.add_result(ProviderResult { + provider: pid("owner"), + ids: vec![PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("mine"), + writer: pid("owner"), + expected: Some(Revision::from_stored("1111111111111111")), + staged_at: None, + }], + state: ProviderStateChange { + st: StateUpdate::Replace(json!({ "token": "A" })), + f: Some(0), + n: Some(5_000), + ..ProviderStateChange::default() + }, + expected_state: ExpectedState::Json(r#"{"token":"A0"}"#.to_owned()), + }); + + let applied = run(&mutation, &live).expect("should apply"); + + assert_eq!( + applied.entry.ids["a.example.com"].uid, "newer", + "should keep the other writer's ID" + ); + assert_eq!( + applied.entry.ps["owner"].st, + Some(json!({ "token": "B" })), + "with that writer's state" + ); + assert_eq!(applied.entry.ps["owner"].n, None, "n is part of the result"); + assert!(!applied.changed, "nothing else changed"); + } + + #[test] + fn provider_result_lands_whole_when_everything_matches() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + st: Some(json!({ "token": "A0" })), + f: 2, + ..KvProviderState::default() + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.add_result(ProviderResult { + provider: pid("owner"), + ids: vec![PartnerIdChange::OwnerWrite { + source: src("a.example.com"), + record: KvPartnerId::new("mine"), + writer: pid("owner"), + expected: None, + staged_at: None, + }], + state: ProviderStateChange { + st: StateUpdate::Replace(json!({ "token": "A1" })), + f: Some(0), + ..ProviderStateChange::default() + }, + expected_state: ExpectedState::Json(r#"{"token":"A0"}"#.to_owned()), + }); + let applied = run(&mutation, &live).expect("should apply"); + assert_eq!(applied.entry.ids["a.example.com"].uid, "mine", "id lands"); + assert_eq!( + applied.entry.ps["owner"].st, + Some(json!({ "token": "A1" })), + "state lands" + ); + assert_eq!(applied.entry.ps["owner"].f, 0, "f lands"); + } + + #[test] + fn refresh_anchor_survives_until_ru_or_state_loss() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + exp: Some(NOW - 1), + ru: Some(NOW + 100), + ..owned("expired", "1111111111111111") + }, + ); + live.ps.insert( + "owner".to_owned(), + KvProviderState { + st: Some(json!({ "r": 1 })), + ..KvProviderState::default() + }, + ); + let mut unrelated = IdentityMutation::default(); + unrelated.set_cluster_size(1); + + let kept = run(&unrelated, &live).expect("should apply"); + assert!( + kept.entry.ids.contains_key("a.example.com"), + "anchor kept while state lives" + ); + assert_eq!( + kept.entry + .network + .as_ref() + .and_then(|network| network.cluster_size), + Some(1), + "cluster size stored" + ); + + live.ps.get_mut("owner").expect("should hold owner").st = None; + let dropped = run(&unrelated, &live).expect("should apply"); + assert!( + !dropped.entry.ids.contains_key("a.example.com"), + "anchor dropped without state" + ); + + live.ps.get_mut("owner").expect("should hold owner").st = Some(json!({ "r": 1 })); + live.ids.get_mut("a.example.com").expect("should hold a").ru = Some(NOW); + let past = run(&unrelated, &live).expect("should apply"); + assert!( + !past.entry.ids.contains_key("a.example.com"), + "anchor dropped once ru passes" + ); + } + + #[test] + fn unconfigured_provider_state_is_pruned() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ps.insert( + "retired".to_owned(), + KvProviderState { + n: Some(5), + ..KvProviderState::default() + }, + ); + live.ps.insert( + "other".to_owned(), + KvProviderState { + n: Some(5), + ..KvProviderState::default() + }, + ); + let mut mutation = IdentityMutation::default(); + mutation.set_cluster_size(1); + let applied = run(&mutation, &live).expect("should apply"); + assert!(!applied.entry.ps.contains_key("retired"), "should prune"); + assert!( + applied.entry.ps.contains_key("other"), + "control: configured kept" + ); + } + + #[test] + fn invalid_result_falls_back_to_removals_only() { + let mut live = KvEntry::minimal("b.example.com", "x", 1); + live.ids + .insert("a.example.com".to_owned(), owned("v1", "1111111111111111")); + let mut mutation = IdentityMutation::default(); + mutation.add_change(fill("d.example.com", &"x".repeat(600))); + mutation.add_change(PartnerIdChange::OwnerRemove { + source: src("a.example.com"), + writer: pid("owner"), + expected: Revision::from_stored("1111111111111111"), + }); + let applied = run(&mutation, &live).expect("should apply removals"); + assert!(applied.removals_only, "should fall back"); + assert!( + !applied.entry.ids.contains_key("a.example.com"), + "removal applied" + ); + assert!( + !applied.entry.ids.contains_key("d.example.com"), + "invalid fill dropped" + ); + } + + #[test] + fn merge_state_combines_changes_for_one_provider() { + let mut mutation = IdentityMutation::default(); + mutation.merge_state( + pid("owner"), + ProviderStateChange { + n: Some(10), + iv_add: vec!["a".repeat(16)], + ..ProviderStateChange::default() + }, + ); + mutation.merge_state( + pid("owner"), + ProviderStateChange { + n: Some(5), + f: Some(3), + iv_add: vec!["b".repeat(16)], + st: StateUpdate::Clear, + ..ProviderStateChange::default() + }, + ); + let live = KvEntry::minimal("b.example.com", "x", 1); + let applied = run(&mutation, &live).expect("should apply"); + let state = &applied.entry.ps["owner"]; + assert_eq!(state.n, Some(10), "n takes the later value"); + assert_eq!(state.f, 3, "f is the last one given"); + assert_eq!(state.iv.len(), 2, "iv is the union"); + assert!(mutation.has_removals(), "iv additions count as removals"); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index 9b50241d7c..adabbf64a3 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -129,3 +129,10 @@ pub(crate) fn fixed_provider( false, ) } + +/// Identity keys derived from the shared test settings. +pub(crate) fn test_keys() -> crate::ec::identity::revision::IdentityKeys { + crate::ec::identity::revision::IdentityKeys::from_settings( + &crate::test_support::tests::create_test_settings(), + ) +} diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index d0d5f17f2a..8119fb831e 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -98,6 +98,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), ("ec/identity/input.rs", include_str!("ec/identity/input.rs")), ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), + ( + "ec/identity/mutation.rs", + include_str!("ec/identity/mutation.rs"), + ), ( "ec/identity/policy.rs", include_str!("ec/identity/policy.rs"), From 2114969281a2aabfd7dcfdba1e093c216157f601 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:14:28 +0530 Subject: [PATCH 17/33] Seal and read the server-owned identity cookies Signed-off-by: dhruv8sh --- .../src/ec/identity/cookie.rs | 248 ++++++++++++++++++ .../src/ec/identity/mod.rs | 1 + .../src/migration_guards.rs | 4 + 3 files changed, 253 insertions(+) create mode 100644 crates/trusted-server-core/src/ec/identity/cookie.rs diff --git a/crates/trusted-server-core/src/ec/identity/cookie.rs b/crates/trusted-server-core/src/ec/identity/cookie.rs new file mode 100644 index 0000000000..512075eacc --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/cookie.rs @@ -0,0 +1,248 @@ +//! The server-owned identity cookies: sealing, reading and formatting. +//! +//! `__Host-ts-eids-s` and every `__Host-ts-eids-r-*` cookie is sealed with +//! AES-256-GCM as `v1..`, base64url without +//! padding, with the full EC ID as associated data. A cookie copied from +//! another visitor, or left over after an EC rotation, fails to open and is +//! ignored. Values are never logged. + +use aes_gcm::aead::{Aead as _, KeyInit as _, Payload}; +use aes_gcm::{Aes256Gcm, Nonce}; +use base64::Engine as _; +use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use http::{HeaderMap, header}; + +use super::input::RandomSource; + +/// Lifetime of both identity cookie kinds, in seconds. +pub(crate) const COOKIE_MAX_AGE_SECS: u64 = 31_536_000; + +const SEAL_VERSION: &str = "v1"; +const NONCE_BYTES: usize = 12; + +/// Seals `payload` with AES-256-GCM, the EC ID as associated data. +#[must_use] +pub(crate) fn seal( + key: &[u8; 32], + ec_id: &str, + payload: &[u8], + random: &dyn RandomSource, +) -> String { + let cipher = Aes256Gcm::new_from_slice(key).expect("should accept a 32-byte key"); + let mut nonce = [0u8; NONCE_BYTES]; + random.fill(&mut nonce); + let ciphertext = cipher + .encrypt( + Nonce::from_slice(&nonce), + Payload { + msg: payload, + aad: ec_id.as_bytes(), + }, + ) + .expect("should encrypt a bounded in-memory payload"); + format!( + "{SEAL_VERSION}.{}.{}", + URL_SAFE_NO_PAD.encode(nonce), + URL_SAFE_NO_PAD.encode(ciphertext) + ) +} + +/// Opens a sealed value. `None` for any failure. +#[must_use] +pub(crate) fn open(key: &[u8; 32], ec_id: &str, value: &str) -> Option> { + let mut parts = value.split('.'); + if parts.next()? != SEAL_VERSION { + return None; + } + let nonce = URL_SAFE_NO_PAD.decode(parts.next()?).ok()?; + let ciphertext = URL_SAFE_NO_PAD.decode(parts.next()?).ok()?; + if parts.next().is_some() || nonce.len() != NONCE_BYTES { + return None; + } + Aes256Gcm::new_from_slice(key) + .ok()? + .decrypt( + Nonce::from_slice(&nonce), + Payload { + msg: &ciphertext, + aad: ec_id.as_bytes(), + }, + ) + .ok() +} + +/// One cookie read by exact name. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum CookieRead { + /// Not present. + Absent, + /// Two or more cookies share the name; treated as none. + Duplicate, + /// Present once. + Value(String), +} + +fn cookie_pairs(headers: &HeaderMap) -> impl Iterator { + headers + .get_all(header::COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .flat_map(|value| value.split(';')) + .filter_map(|pair| pair.trim().split_once('=')) + .map(|(name, value)| (name.trim(), value.trim())) +} + +/// Reads a cookie that must occur at most once. +#[must_use] +pub(crate) fn read_single_cookie(headers: &HeaderMap, name: &str) -> CookieRead { + let mut found = cookie_pairs(headers) + .filter(|(cookie, _)| *cookie == name) + .map(|(_, value)| value); + match (found.next(), found.next()) { + (None, _) => CookieRead::Absent, + (Some(_), Some(_)) => CookieRead::Duplicate, + (Some(value), None) => CookieRead::Value(value.to_owned()), + } +} + +/// Every cookie whose name starts with `prefix`, as `(name, value)`. +#[must_use] +pub(crate) fn read_prefixed_cookies(headers: &HeaderMap, prefix: &str) -> Vec<(String, String)> { + cookie_pairs(headers) + .filter(|(name, _)| name.starts_with(prefix)) + .map(|(name, value)| (name.to_owned(), value.to_owned())) + .collect() +} + +/// A `Set-Cookie` value with the identity cookie attributes. Host-only: no +/// `Domain`. +#[must_use] +pub(crate) fn set_cookie_header(name: &str, value: &str) -> String { + format!("{name}={value}; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age={COOKIE_MAX_AGE_SECS}") +} + +/// A `Set-Cookie` value that deletes an identity cookie. +#[must_use] +pub(crate) fn delete_cookie_header(name: &str) -> String { + format!("{name}=; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=0") +} + +#[cfg(test)] +mod tests { + use http::{HeaderMap, HeaderValue, header}; + + use super::*; + use crate::ec::identity::input::SeededRandom; + + const EC: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ABC123"; + const OTHER_EC: &str = + "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ABC123"; + + #[test] + fn seal_and_open_round_trip_bound_to_the_ec_id() { + let key = [7u8; 32]; + let random = SeededRandom::new(3); + let sealed = seal(&key, EC, br#"{"v":1}"#, &random); + + assert!(sealed.starts_with("v1."), "should carry the version prefix"); + assert_eq!(sealed.split('.').count(), 3, "should have three parts"); + assert!(!sealed.contains('='), "should be unpadded base64url"); + assert_eq!( + open(&key, EC, &sealed).as_deref(), + Some(&br#"{"v":1}"#[..]), + "should open for its EC" + ); + assert!( + open(&key, OTHER_EC, &sealed).is_none(), + "should not open for another EC" + ); + assert!( + open(&[8u8; 32], EC, &sealed).is_none(), + "should not open with another key" + ); + let mut tampered = sealed.clone(); + let last = tampered.pop().expect("should have a last character"); + tampered.push(if last == 'A' { 'B' } else { 'A' }); + assert!( + open(&key, EC, &tampered).is_none(), + "should reject tampering" + ); + assert!( + open(&key, EC, "v2.x.y").is_none(), + "should reject another version" + ); + assert!( + open(&key, EC, "v1.x").is_none(), + "should reject a short value" + ); + assert_ne!( + seal(&key, EC, b"same", &random), + seal(&key, EC, b"same", &random), + "every seal draws a fresh nonce" + ); + } + + #[test] + fn duplicate_cookie_names_count_as_none() { + let mut headers = HeaderMap::new(); + headers.append( + header::COOKIE, + HeaderValue::from_static("__Host-ts-eids-s=one"), + ); + headers.append( + header::COOKIE, + HeaderValue::from_static("other=1; __Host-ts-eids-s=two"), + ); + assert_eq!( + read_single_cookie(&headers, "__Host-ts-eids-s"), + CookieRead::Duplicate, + "a duplicate counts as none" + ); + + let mut single = HeaderMap::new(); + single.insert( + header::COOKIE, + HeaderValue::from_static("__Host-ts-eids-s=one; x=2"), + ); + assert_eq!( + read_single_cookie(&single, "__Host-ts-eids-s"), + CookieRead::Value("one".to_owned()), + "a single cookie is read" + ); + assert_eq!( + read_single_cookie(&HeaderMap::new(), "__Host-ts-eids-s"), + CookieRead::Absent, + "absent" + ); + } + + #[test] + fn prefixed_cookies_and_attributes() { + let mut headers = HeaderMap::new(); + headers.insert( + header::COOKIE, + HeaderValue::from_static( + "__Host-ts-eids-r-0a1b2c3d=a; __Host-ts-eids-r-11111111=b; x=1", + ), + ); + let found = read_prefixed_cookies(&headers, "__Host-ts-eids-r-"); + assert_eq!( + found, + vec![ + ("__Host-ts-eids-r-0a1b2c3d".to_owned(), "a".to_owned()), + ("__Host-ts-eids-r-11111111".to_owned(), "b".to_owned()), + ], + "should find both removal cookies" + ); + assert_eq!( + set_cookie_header("__Host-ts-eids-s", "v1.a.b"), + "__Host-ts-eids-s=v1.a.b; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=31536000", + "should use the identity cookie attributes and no Domain" + ); + assert_eq!( + delete_cookie_header("__Host-ts-eids-s"), + "__Host-ts-eids-s=; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=0", + "should expire with matching attributes" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index a52f511924..7bbf56630e 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -14,6 +14,7 @@ )] pub(crate) mod catalog; +pub(crate) mod cookie; pub mod descriptor; pub mod exchange; pub mod ids; diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 8119fb831e..2d029fe476 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -87,6 +87,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/catalog.rs", include_str!("ec/identity/catalog.rs"), ), + ( + "ec/identity/cookie.rs", + include_str!("ec/identity/cookie.rs"), + ), ( "ec/identity/descriptor.rs", include_str!("ec/identity/descriptor.rs"), From 43b6e595f5f8cecef1f6a2c70fb49fa432dcaaa9 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:18:32 +0530 Subject: [PATCH 18/33] Encode the ts-eids-s cookie and reconcile it with KV Signed-off-by: dhruv8sh --- .../src/ec/identity/cookie.rs | 1085 +++++++++++++++++ .../src/ec/identity/test_support.rs | 16 + 2 files changed, 1101 insertions(+) diff --git a/crates/trusted-server-core/src/ec/identity/cookie.rs b/crates/trusted-server-core/src/ec/identity/cookie.rs index 512075eacc..5c1b7ede21 100644 --- a/crates/trusted-server-core/src/ec/identity/cookie.rs +++ b/crates/trusted-server-core/src/ec/identity/cookie.rs @@ -6,13 +6,23 @@ //! another visitor, or left over after an EC rotation, fails to open and is //! ignored. Values are never logged. +use core::fmt; +use std::collections::{BTreeMap, BTreeSet}; + use aes_gcm::aead::{Aead as _, KeyInit as _, Payload}; use aes_gcm::{Aes256Gcm, Nonce}; use base64::Engine as _; use base64::engine::general_purpose::URL_SAFE_NO_PAD; use http::{HeaderMap, header}; +use serde::{Deserialize, Serialize}; +use super::catalog::ProviderCatalog; +use super::ids::SourceDomain; use super::input::RandomSource; +use super::revision::{IdentityKeys, record_revision}; +use crate::constants::COOKIE_TS_EIDS_S as COOKIE_S; +use crate::ec::EcKvSnapshot; +use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId}; /// Lifetime of both identity cookie kinds, in seconds. pub(crate) const COOKIE_MAX_AGE_SECS: u64 = 31_536_000; @@ -127,6 +137,750 @@ pub(crate) fn delete_cookie_header(name: &str) -> String { format!("{name}=; Path=/; Secure; SameSite=Lax; HttpOnly; Max-Age=0") } +/// Maximum sealed bytes of `__Host-ts-eids-s`. +pub(crate) const MAX_S_COOKIE_BYTES: usize = 2048; + +/// Serializes a flag as `1`, and reads `0` or `1`. +mod int_bool { + use serde::{Deserialize as _, Deserializer, Serializer}; + + #[allow( + clippy::trivially_copy_pass_by_ref, + reason = "serde's `with` passes the field by reference" + )] + pub(super) fn serialize(value: &bool, serializer: S) -> Result { + serializer.serialize_u8(u8::from(*value)) + } + + pub(super) fn deserialize<'de, D: Deserializer<'de>>( + deserializer: D, + ) -> Result { + Ok(u8::deserialize(deserializer)? != 0) + } +} + +#[allow( + clippy::trivially_copy_pass_by_ref, + reason = "serde's skip_serializing_if passes the field by reference" +)] +fn is_false(value: &bool) -> bool { + !*value +} + +/// The `__Host-ts-eids-s` payload. +#[derive(Clone, Default, PartialEq, Serialize, Deserialize)] +pub(crate) struct SCookie { + /// Payload version, always 1. + pub(crate) v: u8, + /// When a request last confirmed this cookie against the KV row. + pub(crate) kc: u64, + /// Fingerprint of the identity config (providers, sources, owners). + pub(crate) cf: String, + /// A request saw this EC's row missing; the next writer reads KV. + #[serde(default, skip_serializing_if = "is_false", with = "int_bool")] + pub(crate) m: bool, + /// Cached and staged records. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub(crate) ids: Vec, + /// Mirrored and staged provider state, and pending markers. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub(crate) ps: BTreeMap, +} + +impl fmt::Debug for SCookie { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SCookie") + .field("kc", &self.kc) + .field("cf", &self.cf) + .field("m", &self.m) + .field("ids", &self.ids) + .field("ps", &self.ps.keys().collect::>()) + .finish() + } +} + +impl SCookie { + /// An empty cookie for the current config, confirmed at `now`. + #[must_use] + pub(crate) fn new(catalog: &ProviderCatalog, now: u64) -> Self { + Self { + v: 1, + kc: now, + cf: catalog.fingerprint().to_owned(), + ..Self::default() + } + } + + /// Only the pending markers, for a config change or the size cap's last + /// resort. `kc = 0` forces a KV read at the next navigation. + #[must_use] + fn markers_only(&self, cf: &str) -> Self { + Self { + v: 1, + kc: 0, + cf: cf.to_owned(), + m: self.m, + ids: Vec::new(), + ps: self + .ps + .iter() + .filter_map(|(provider, state)| { + state.p.map(|marker| { + ( + provider.clone(), + SProviderState { + p: Some(marker), + ..SProviderState::default() + }, + ) + }) + }) + .collect(), + } + } +} + +/// One record in `__Host-ts-eids-s`. +#[derive(Clone, Default, PartialEq, Serialize, Deserialize)] +pub(crate) struct SEntry { + /// The source, as an index into the catalog's configured sources. + pub(crate) s: usize, + /// The UID. Absent on a stub. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) u: Option, + /// Writer. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) w: Option, + /// KV `at`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) t: Option, + /// KV `exp`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) e: Option, + /// KV `ra`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) r: Option, + /// KV `atype`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) a: Option, + /// KV `ext`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) x: Option, + /// KV `rv` of a confirmed entry. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) rv: Option, + /// The revision of the record a staged entry replaces. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) pv: Option, + /// When the entry was staged. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) sa: Option, + /// A browser KV-gap entry. + #[serde(default, skip_serializing_if = "is_false", with = "int_bool")] + pub(crate) b: bool, + /// Known to be in KV. + #[serde(default, skip_serializing_if = "is_false", with = "int_bool")] + pub(crate) k: bool, + /// A stub: dropped for size and still in KV. + #[serde(default, skip_serializing_if = "is_false", with = "int_bool")] + pub(crate) d: bool, +} + +impl fmt::Debug for SEntry { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SEntry") + .field("s", &self.s) + .field("w", &self.w) + .field("b", &self.b) + .field("k", &self.k) + .field("d", &self.d) + .finish_non_exhaustive() + } +} + +impl SEntry { + /// A browser KV-gap entry staged at `sa`. + #[must_use] + pub(crate) fn staged_browser(s: usize, uid: &str, sa: u64) -> Self { + Self { + s, + u: Some(uid.to_owned()), + t: Some(sa), + sa: Some(sa), + b: true, + ..Self::default() + } + } + + /// A provider entry staged at `sa`, replacing revision `pv`. + #[must_use] + pub(crate) fn staged_provider( + s: usize, + record: &KvPartnerId, + pv: Option, + sa: u64, + ) -> Self { + Self { + rv: None, + pv, + sa: Some(sa), + k: false, + ..Self::confirmed(s, record) + } + } + + /// A record known to be in KV. + #[must_use] + pub(crate) fn confirmed(s: usize, record: &KvPartnerId) -> Self { + Self { + s, + u: Some(record.uid.clone()), + w: record.w.clone(), + t: record.at, + e: record.exp, + r: record.ra, + a: record.atype, + x: record.ext.clone(), + rv: record.rv.clone(), + k: true, + ..Self::default() + } + } + + /// A stub for a source that is still in KV. + #[must_use] + pub(crate) fn stub(s: usize) -> Self { + Self { + s, + d: true, + ..Self::default() + } + } + + /// The UID, absent on a stub. + #[must_use] + pub(crate) fn uid(&self) -> Option<&str> { + self.u.as_deref() + } + + /// Whether this is a browser KV-gap entry. + #[must_use] + pub(crate) fn is_browser_gap(&self) -> bool { + self.b && !self.d + } + + /// Whether this is a staged provider entry. + #[must_use] + pub(crate) fn is_staged_provider(&self) -> bool { + !self.k && !self.b && !self.d && self.w.is_some() + } + + /// The record this entry carries. `None` for a stub. + #[must_use] + pub(crate) fn to_record(&self) -> Option { + Some(KvPartnerId { + uid: self.u.clone()?, + w: self.w.clone(), + rv: self.rv.clone(), + at: self.t, + exp: self.e, + ra: self.r, + ru: None, + atype: self.a, + ext: self.x.clone(), + }) + } +} + +/// Mirrored and staged state of one provider in `__Host-ts-eids-s`. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct SProviderState { + /// Next eligible time. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) n: Option, + /// Consecutive failures. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) f: Option, + /// Mirrored opt-out. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) o: Option, + /// Mirrored invalidated-UID hashes. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub(crate) iv: Vec, + /// A staged opt-out clear. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) oc: Option, + /// Pending marker: when an after-send dispatch happened. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) p: Option, + /// `n` and `f` are staged and not yet written. + #[serde(default, skip_serializing_if = "is_false", with = "int_bool")] + pub(crate) sg: bool, + /// The KV `n` a cookie-only `n` was derived from. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) kn: Option, +} + +impl SProviderState { + /// Still in KV, with no marker and no staged change: droppable for size. + fn is_droppable(&self) -> bool { + self.p.is_none() && !self.sg && self.oc.is_none() + } +} + +/// The result of opening `__Host-ts-eids-s`. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum SOpen { + /// No cookie. + Missing, + /// Duplicated, failed to open or decode. Rebuilt by the next KV read. + Failed, + /// The identity config changed. Only the markers were kept. + ConfigChanged(SCookie), + /// Opened for this EC and config. + Valid(SCookie), +} + +impl SOpen { + /// The usable cookie, if any. + #[must_use] + pub(crate) fn cookie(&self) -> Option<&SCookie> { + match self { + Self::Valid(cookie) | Self::ConfigChanged(cookie) => Some(cookie), + Self::Missing | Self::Failed => None, + } + } +} + +/// Opens `__Host-ts-eids-s` for this EC and config. +#[must_use] +pub(crate) fn open_s_cookie( + keys: &IdentityKeys, + ec_id: &str, + catalog: &ProviderCatalog, + headers: &HeaderMap, +) -> SOpen { + let value = match read_single_cookie(headers, COOKIE_S) { + CookieRead::Absent => return SOpen::Missing, + CookieRead::Duplicate => { + log::debug!("Ignoring duplicate {COOKIE_S} cookies"); + return SOpen::Failed; + } + CookieRead::Value(value) => value, + }; + let Some(cookie) = open(&keys.cookie_s, ec_id, &value) + .and_then(|payload| serde_json::from_slice::(&payload).ok()) + .filter(|cookie| cookie.v == 1) + else { + log::debug!("Ignoring {COOKIE_S} cookie that failed to open"); + return SOpen::Failed; + }; + if cookie.cf != catalog.fingerprint() { + return SOpen::ConfigChanged(cookie.markers_only(catalog.fingerprint())); + } + let source_count = catalog.configured_sources().len(); + let mut cookie = cookie; + cookie.ids.retain(|entry| entry.s < source_count); + SOpen::Valid(cookie) +} + +fn seal_s_cookie( + keys: &IdentityKeys, + ec_id: &str, + cookie: &SCookie, + random: &dyn RandomSource, +) -> String { + let payload = serde_json::to_vec(cookie).expect("should serialize the identity cookie"); + seal(&keys.cookie_s, ec_id, &payload, random) +} + +/// Seals `cookie`, dropping entries until it fits [`MAX_S_COOKIE_BYTES`]. +/// +/// The drop order: confirmed entries become stubs; `ps` entries with no +/// marker or staged change go; staged provider entries go, lowest +/// `cookie_priority` first; browser KV-gap entries go last. If it still does +/// not fit, only the markers are kept, with `kc = 0`. Markers are never +/// dropped. `cookie` is updated to what was sealed. +#[must_use] +pub(crate) fn encode_s_cookie( + keys: &IdentityKeys, + ec_id: &str, + cookie: &mut SCookie, + random: &dyn RandomSource, + priority: &dyn Fn(&str) -> i32, +) -> String { + let mut sealed = seal_s_cookie(keys, ec_id, cookie, random); + if sealed.len() <= MAX_S_COOKIE_BYTES { + return sealed; + } + + for entry in cookie.ids.iter_mut().filter(|entry| entry.k && !entry.d) { + log::debug!( + "Identity cookie over its cap: stubbing source index {}", + entry.s + ); + *entry = SEntry::stub(entry.s); + } + sealed = seal_s_cookie(keys, ec_id, cookie, random); + if sealed.len() <= MAX_S_COOKIE_BYTES { + return sealed; + } + + cookie.ps.retain(|_, state| !state.is_droppable()); + sealed = seal_s_cookie(keys, ec_id, cookie, random); + if sealed.len() <= MAX_S_COOKIE_BYTES { + return sealed; + } + + loop { + let lowest = cookie + .ids + .iter() + .enumerate() + .filter(|(_, entry)| entry.is_staged_provider()) + .min_by_key(|(_, entry)| priority(entry.w.as_deref().unwrap_or_default())) + .map(|(index, _)| index); + let Some(index) = lowest else { + break; + }; + let dropped = cookie.ids.remove(index); + log::debug!( + "Identity cookie over its cap: dropping staged entry of {} for source index {}", + dropped.w.as_deref().unwrap_or_default(), + dropped.s + ); + sealed = seal_s_cookie(keys, ec_id, cookie, random); + if sealed.len() <= MAX_S_COOKIE_BYTES { + return sealed; + } + } + + while let Some(index) = cookie.ids.iter().rposition(SEntry::is_browser_gap) { + let dropped = cookie.ids.remove(index); + log::debug!( + "Identity cookie over its cap: dropping browser entry for source index {}", + dropped.s + ); + sealed = seal_s_cookie(keys, ec_id, cookie, random); + if sealed.len() <= MAX_S_COOKIE_BYTES { + return sealed; + } + } + + log::warn!("Identity cookie over its cap after every drop, keeping markers only"); + *cookie = cookie.markers_only(&cookie.cf.clone()); + seal_s_cookie(keys, ec_id, cookie, random) +} + +/// A minimal cookie for a row seen missing: `m` only, which forces the next +/// writer to read KV and recover. +#[must_use] +pub(crate) fn minimal_missing_cookie(catalog: &ProviderCatalog) -> SCookie { + SCookie { + v: 1, + kc: 0, + cf: catalog.fingerprint().to_owned(), + m: true, + ids: Vec::new(), + ps: BTreeMap::new(), + } +} + +/// What reconciling the cookie with a KV read needs. +pub(crate) struct CookieMergeContext<'a> { + pub(crate) catalog: &'a ProviderCatalog, + pub(crate) keys: IdentityKeys, + /// Unix seconds. + pub(crate) now: u64, + pub(crate) kv_revalidate_sec: u64, + /// Whether the live row makes a provider ineligible, which answers its + /// marker (spec 5.6). + pub(crate) answered_by_eligibility: &'a dyn Fn(&str, &KvEntry) -> bool, +} + +#[cfg(test)] +fn never_answered(_: &str, _: &KvEntry) -> bool { + false +} + +#[cfg(test)] +impl<'a> CookieMergeContext<'a> { + /// A context with test keys, a one-hour revalidation and no eligibility + /// answers. + pub(crate) fn for_test(catalog: &'a ProviderCatalog, now: u64) -> Self { + Self { + catalog, + keys: super::test_support::test_keys(), + now, + kv_revalidate_sec: 3600, + answered_by_eligibility: &never_answered, + } + } +} + +/// The cookie after a KV read. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct MergeOutcome { + /// The reconciled cookie. + pub(crate) cookie: SCookie, + /// Whether it must be re-emitted. + pub(crate) changed: bool, + /// Whether the row is a tombstone: expire the cookie. + pub(crate) expire: bool, +} + +/// Reconciles the cookie with what a KV read found (spec 6.3, step 3). +#[must_use] +pub(crate) fn merge_with_kv( + cookie: &SCookie, + snapshot: &EcKvSnapshot, + context: &CookieMergeContext<'_>, +) -> MergeOutcome { + match snapshot { + EcKvSnapshot::NotRead | EcKvSnapshot::Failed { .. } => MergeOutcome { + cookie: cookie.clone(), + changed: false, + expire: false, + }, + EcKvSnapshot::Missing { .. } => MergeOutcome { + cookie: minimal_missing_cookie(context.catalog), + changed: true, + expire: false, + }, + EcKvSnapshot::Present { entry, .. } if !entry.consent.ok => MergeOutcome { + cookie: SCookie::new(context.catalog, context.now), + changed: true, + expire: true, + }, + EcKvSnapshot::Present { entry, .. } => { + let merged = SCookie { + v: 1, + kc: context.now, + cf: context.catalog.fingerprint().to_owned(), + m: false, + ids: merge_ids(cookie, entry, context), + ps: merge_provider_state(cookie, entry, context), + }; + let content_changed = merged.ids != cookie.ids + || merged.ps != cookie.ps + || merged.m != cookie.m + || merged.cf != cookie.cf; + let stale = context.now.saturating_sub(cookie.kc) > context.kv_revalidate_sec / 2; + MergeOutcome { + cookie: merged, + changed: content_changed || stale, + expire: false, + } + } + } +} + +/// Whether a staged provider entry would still apply as an `OwnerWrite`. +fn staged_would_apply(entry: &SEntry, kv: Option<&KvPartnerId>, keys: &IdentityKeys) -> bool { + match (&entry.pv, kv) { + (Some(pv), Some(record)) => record_revision(keys, record).as_str() == pv, + (Some(_), None) => false, + (None, None) => true, + (None, Some(record)) => record.w != entry.w, + } +} + +fn merge_ids(cookie: &SCookie, entry: &KvEntry, context: &CookieMergeContext<'_>) -> Vec { + let sources = context.catalog.configured_sources(); + let owned = |source: &SourceDomain| context.catalog.owner_of(source).is_some(); + let mut ids = Vec::new(); + let mut covered = BTreeSet::new(); + + for cookie_entry in &cookie.ids { + let Some(source) = sources.get(cookie_entry.s) else { + continue; + }; + let kv = entry.ids.get(source.as_str()); + if cookie_entry.d || cookie_entry.k { + // A stub or confirmed copy: the KV record wins, and the cookie + // caches only owned sources. + if let Some(record) = kv + && owned(source) + { + ids.push(SEntry::confirmed(cookie_entry.s, record)); + covered.insert(cookie_entry.s); + } + continue; + } + if cookie_entry.b { + if kv.is_none() { + ids.push(cookie_entry.clone()); + } + continue; + } + let flushed = kv.is_some_and(|record| { + Some(&record.uid) == cookie_entry.u.as_ref() && record.w == cookie_entry.w + }); + if flushed { + if let Some(record) = kv + && owned(source) + { + ids.push(SEntry::confirmed(cookie_entry.s, record)); + covered.insert(cookie_entry.s); + } + } else if staged_would_apply(cookie_entry, kv, &context.keys) { + ids.push(cookie_entry.clone()); + covered.insert(cookie_entry.s); + } + } + + for (index, source) in sources.iter().enumerate() { + if covered.contains(&index) || !owned(source) { + continue; + } + if let Some(record) = entry.ids.get(source.as_str()) { + ids.push(SEntry::confirmed(index, record)); + } + } + ids +} + +/// Whether a KV read answers a provider's pending marker set at `marker`. +fn marker_answered( + provider: &str, + marker: u64, + cookie: &SCookie, + entry: &KvEntry, + context: &CookieMergeContext<'_>, +) -> bool { + let kv_state = entry.ps.get(provider); + if kv_state.is_some_and(|state| state.n.is_some_and(|n| n >= marker) || state.o.is_some()) { + return true; + } + let claimed: Vec<&SourceDomain> = provider + .parse() + .ok() + .and_then(|id| context.catalog.get(&id)) + .map(|provider| provider.descriptor().sources().iter().collect()) + .unwrap_or_default(); + let sources = context.catalog.configured_sources(); + let acquired_since = claimed.iter().any(|source| { + entry + .ids + .get(source.as_str()) + .is_some_and(|record| record.at.is_some_and(|at| at >= marker)) + }); + let lost_cookie_value = cookie.ids.iter().any(|cookie_entry| { + cookie_entry.u.is_some() + && sources.get(cookie_entry.s).is_some_and(|source| { + claimed.contains(&source) && !entry.ids.contains_key(source.as_str()) + }) + }); + acquired_since || lost_cookie_value || (context.answered_by_eligibility)(provider, entry) +} + +fn merge_provider_state( + cookie: &SCookie, + entry: &KvEntry, + context: &CookieMergeContext<'_>, +) -> BTreeMap { + let mut ps: BTreeMap = BTreeMap::new(); + for (provider, kv_state) in &entry.ps { + let configured = provider + .parse() + .is_ok_and(|id| context.catalog.is_configured(&id)); + if !configured { + continue; + } + ps.insert( + provider.clone(), + SProviderState { + n: kv_state.n, + f: (kv_state.f > 0).then_some(kv_state.f), + o: kv_state.o.clone(), + iv: kv_state.iv.clone(), + ..SProviderState::default() + }, + ); + } + + for (provider, state) in &cookie.ps { + let kv_state = entry.ps.get(provider); + let kv_n = kv_state.and_then(|kv_state| kv_state.n); + let target = ps.entry(provider.clone()).or_default(); + if let Some(marker) = state.p + && !marker_answered(provider, marker, cookie, entry, context) + { + target.p = Some(marker); + } + if state.sg + && state + .n + .is_some_and(|staged| kv_n.is_none_or(|kv| kv < staged)) + { + target.n = state.n; + target.f = state.f; + target.sg = true; + } + if let Some(clear) = state.oc + && kv_state + .and_then(|kv_state| kv_state.oc) + .is_none_or(|kv_clear| kv_clear < clear) + { + target.oc = Some(clear); + } + if !state.sg + && let (Some(n), Some(seen)) = (state.n, state.kn) + && kv_n == Some(seen) + { + target.n = target.n.max(Some(n)); + target.kn = Some(seen); + } + } + ps.retain(|_, state| *state != SProviderState::default()); + ps +} + +/// A record the cookie view holds for one source. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum Staging { + /// Known to be in KV. + Confirmed, + /// Staged by a stager and not yet written. + Staged { + /// The revision it replaces. + pv: Option, + /// When it was staged. + sa: Option, + /// A browser KV-gap entry. + browser: bool, + }, +} + +/// The cookie's records resolved to sources, stubs excluded. +#[must_use] +pub(crate) fn cookie_view( + cookie: &SCookie, + catalog: &ProviderCatalog, +) -> Vec<(SourceDomain, KvPartnerId, Staging)> { + let sources = catalog.configured_sources(); + cookie + .ids + .iter() + .filter_map(|entry| { + let source = sources.get(entry.s)?.clone(); + let record = entry.to_record()?; + let staging = if entry.k { + Staging::Confirmed + } else { + Staging::Staged { + pv: entry.pv.clone(), + sa: entry.sa, + browser: entry.b, + } + }; + Some((source, record, staging)) + }) + .collect() +} + #[cfg(test)] mod tests { use http::{HeaderMap, HeaderValue, header}; @@ -246,3 +1000,334 @@ mod tests { ); } } + +#[cfg(test)] +mod s_cookie_tests { + use super::*; + use crate::ec::EcKvSnapshot; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::test_support::{catalog_with_owner, test_keys}; + use crate::ec::kv_types::{KvEntry, KvPartnerId, KvProviderState}; + + const EC: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ABC123"; + + fn present(entry: KvEntry) -> EcKvSnapshot { + EcKvSnapshot::Present { + ec_id: EC.to_owned(), + entry: Box::new(entry), + generation: Some(1), + } + } + + fn headers_with(value: &str) -> http::HeaderMap { + let mut headers = http::HeaderMap::new(); + headers.insert( + http::header::COOKIE, + http::HeaderValue::from_str(&format!("{COOKIE_S}={value}")) + .expect("should build cookie header"), + ); + headers + } + + fn owner_record(uid: &str, rv: &str) -> KvPartnerId { + KvPartnerId { + w: Some("owner".to_owned()), + rv: Some(rv.to_owned()), + at: Some(90), + ..KvPartnerId::new(uid) + } + } + + #[test] + fn round_trips_and_detects_config_change() { + let keys = test_keys(); + let catalog = catalog_with_owner("owner", "a.example.com"); + let random = SeededRandom::new(1); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ps.insert( + "owner".to_owned(), + SProviderState { + p: Some(90), + ..SProviderState::default() + }, + ); + cookie.ids.push(SEntry::staged_browser(0, "u", 100)); + let sealed = encode_s_cookie(&keys, EC, &mut cookie, &random, &|_| 0); + + match open_s_cookie(&keys, EC, &catalog, &headers_with(&sealed)) { + SOpen::Valid(opened) => assert_eq!(opened, cookie, "should round-trip"), + other => panic!("should open, got {other:?}"), + } + let other_catalog = catalog_with_owner("owner", "b.example.com"); + match open_s_cookie(&keys, EC, &other_catalog, &headers_with(&sealed)) { + SOpen::ConfigChanged(kept) => { + assert!(kept.ids.is_empty(), "should drop positive entries"); + assert_eq!( + kept.ps.get("owner").and_then(|state| state.p), + Some(90), + "should keep markers" + ); + } + other => panic!("should flag the config change, got {other:?}"), + } + assert!( + matches!( + open_s_cookie(&keys, EC, &catalog, &http::HeaderMap::new()), + SOpen::Missing + ), + "no cookie is Missing" + ); + assert!( + matches!( + open_s_cookie(&keys, EC, &catalog, &headers_with("v1.bad.value")), + SOpen::Failed + ), + "garbage fails to open" + ); + } + + #[test] + fn tombstone_expires_and_missing_row_minimizes() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ps.insert( + "owner".to_owned(), + SProviderState { + p: Some(90), + ..SProviderState::default() + }, + ); + cookie.ids.push(SEntry::staged_browser(0, "u", 100)); + let context = CookieMergeContext::for_test(&catalog, 200); + assert!( + merge_with_kv(&cookie, &present(KvEntry::tombstone(1)), &context).expire, + "tombstone expires the cookie" + ); + let missing = merge_with_kv( + &cookie, + &EcKvSnapshot::Missing { + ec_id: EC.to_owned(), + }, + &context, + ); + assert!(missing.cookie.m, "should set m"); + assert!( + missing.cookie.ids.is_empty() && missing.cookie.ps.is_empty(), + "no entries, no markers" + ); + assert!(missing.changed, "should be rewritten"); + let unread = merge_with_kv(&cookie, &EcKvSnapshot::NotRead, &context); + assert!(!unread.changed, "no read, no change"); + assert_eq!(unread.cookie, cookie, "no read keeps the cookie"); + } + + #[test] + fn browser_gap_entry_is_dropped_once_kv_holds_the_source() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ids.push(SEntry::staged_browser(0, "u", 100)); + + let held = merge_with_kv( + &cookie, + &present(KvEntry::minimal("a.example.com", "other", 1)), + &CookieMergeContext::for_test(&catalog, 200), + ); + assert!(held.changed, "should change"); + assert!( + held.cookie.ids.iter().all(|entry| !entry.is_browser_gap()), + "the b:1 entry is dropped once KV holds the source" + ); + assert_eq!(held.cookie.kc, 200, "should stamp kc"); + + let absent = merge_with_kv( + &cookie, + &present(KvEntry::minimal("z.example.com", "x", 1)), + &CookieMergeContext::for_test(&catalog, 200), + ); + assert!( + absent.cookie.ids.iter().any(SEntry::is_browser_gap), + "control: kept while KV lacks the source" + ); + } + + #[test] + fn owned_kv_record_is_cached_and_staged_owner_entry_follows_revision() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.ids.insert( + "a.example.com".to_owned(), + owner_record("in-kv", "1111111111111111"), + ); + let context = CookieMergeContext::for_test(&catalog, 200); + + let cached = merge_with_kv( + &SCookie::new(&catalog, 100), + &present(entry.clone()), + &context, + ); + let confirmed = cached + .cookie + .ids + .iter() + .find(|cookie_entry| cookie_entry.s == 0) + .expect("should cache the owned record"); + assert!(confirmed.k, "cached as known in KV"); + assert_eq!(confirmed.uid(), Some("in-kv"), "with the KV UID"); + + let staged = |pv: &str| { + let mut cookie = SCookie::new(&catalog, 100); + cookie.ids.push(SEntry::staged_provider( + 0, + &owner_record("staged", "ffffffffffffffff"), + Some(pv.to_owned()), + 150, + )); + merge_with_kv(&cookie, &present(entry.clone()), &context).cookie + }; + let kept = staged("1111111111111111"); + assert!( + kept.ids + .iter() + .any(|cookie_entry| cookie_entry.uid() == Some("staged") && !cookie_entry.k), + "a staged refresh OwnerWrite would apply stays staged" + ); + let lost = staged("2222222222222222"); + assert!( + lost.ids + .iter() + .all(|cookie_entry| cookie_entry.uid() != Some("staged")), + "a staged refresh against another revision is dropped" + ); + assert!( + lost.ids + .iter() + .any(|cookie_entry| cookie_entry.uid() == Some("in-kv") && cookie_entry.k), + "KV wins" + ); + } + + #[test] + fn marker_is_answered_by_a_newer_n_or_an_opt_out() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ps.insert( + "owner".to_owned(), + SProviderState { + p: Some(150), + ..SProviderState::default() + }, + ); + let answered = |state: KvProviderState| { + let mut entry = KvEntry::minimal("b.example.com", "x", 1); + entry.ps.insert("owner".to_owned(), state); + merge_with_kv( + &cookie, + &present(entry), + &CookieMergeContext::for_test(&catalog, 200), + ) + .cookie + .ps + .get("owner") + .and_then(|state| state.p) + }; + assert_eq!( + answered(KvProviderState { + n: Some(160), + ..KvProviderState::default() + }), + None, + "a newer n answers the marker" + ); + assert_eq!( + answered(KvProviderState { + n: Some(140), + ..KvProviderState::default() + }), + Some(150), + "an older n does not" + ); + } + + #[test] + fn unchanged_recent_cookie_is_not_rewritten() { + let catalog = catalog_with_owner("owner", "a.example.com"); + let cookie = SCookie::new(&catalog, 1_000); + let entry = KvEntry::minimal("z.example.com", "x", 1); + let recent = merge_with_kv( + &cookie, + &present(entry.clone()), + &CookieMergeContext::for_test(&catalog, 1_100), + ); + assert!(!recent.changed, "nothing changed and kc is recent"); + let stale = merge_with_kv( + &cookie, + &present(entry), + &CookieMergeContext::for_test(&catalog, 1_000 + 1_801), + ); + assert!(stale.changed, "kc older than half of kv_revalidate_sec"); + } + + #[test] + fn size_cap_drops_in_order_and_keeps_markers() { + let keys = test_keys(); + let catalog = catalog_with_owner("owner", "a.example.com"); + let random = SeededRandom::new(1); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ps.insert( + "owner".to_owned(), + SProviderState { + p: Some(90), + ..SProviderState::default() + }, + ); + for _ in 0..40 { + cookie + .ids + .push(SEntry::staged_browser(0, &"u".repeat(60), 100)); + } + let sealed = encode_s_cookie(&keys, EC, &mut cookie, &random, &|_| 0); + assert!(sealed.len() <= MAX_S_COOKIE_BYTES, "should fit the cap"); + assert_eq!( + cookie.ps.get("owner").and_then(|state| state.p), + Some(90), + "should keep the marker" + ); + assert!( + !cookie.ids.is_empty(), + "should keep as many entries as fit, not drop them all" + ); + } + + #[test] + fn size_cap_stubs_confirmed_entries_before_dropping_staged_ones() { + let keys = test_keys(); + let catalog = catalog_with_owner("owner", "a.example.com"); + let random = SeededRandom::new(1); + let mut cookie = SCookie::new(&catalog, 100); + cookie.ids.push(SEntry::confirmed( + 0, + &owner_record(&"c".repeat(500), "1111111111111111"), + )); + cookie + .ids + .push(SEntry::staged_browser(0, &"b".repeat(400), 100)); + for _ in 0..3 { + cookie.ids.push(SEntry::staged_provider( + 0, + &owner_record(&"p".repeat(300), "ffffffffffffffff"), + None, + 100, + )); + } + let sealed = encode_s_cookie(&keys, EC, &mut cookie, &random, &|_| 0); + assert!(sealed.len() <= MAX_S_COOKIE_BYTES, "should fit the cap"); + assert!( + cookie.ids.iter().any(|entry| entry.d), + "the confirmed entry becomes a stub first" + ); + assert!( + cookie.ids.iter().any(SEntry::is_browser_gap), + "the browser gap entry is dropped last" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index adabbf64a3..3946a16bf6 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -136,3 +136,19 @@ pub(crate) fn test_keys() -> crate::ec::identity::revision::IdentityKeys { &crate::test_support::tests::create_test_settings(), ) } + +/// A catalog with one provider that owns `source`. +pub(crate) fn catalog_with_owner( + provider: &str, + source: &str, +) -> crate::ec::identity::catalog::ProviderCatalog { + crate::ec::identity::catalog::ProviderCatalog::new( + vec![fixed_provider(provider, &[source])], + [( + source.parse().expect("should parse source"), + provider.parse().expect("should parse provider id"), + )] + .into_iter() + .collect(), + ) +} From 9a1dd40f1e28fb50776dbfddc5f9e9712fe668cb Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:20:44 +0530 Subject: [PATCH 19/33] Add per-batch ts-eids-r removal cookies Each removal-producing response gets its own sealed cookie. A batch is never coarsened: an oversized one is planned as an overflow write, or split across cookies when that write fails. Signed-off-by: dhruv8sh --- .../src/ec/identity/cookie.rs | 581 +++++++++++++++++- 1 file changed, 578 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-core/src/ec/identity/cookie.rs b/crates/trusted-server-core/src/ec/identity/cookie.rs index 5c1b7ede21..20ff04f082 100644 --- a/crates/trusted-server-core/src/ec/identity/cookie.rs +++ b/crates/trusted-server-core/src/ec/identity/cookie.rs @@ -17,10 +17,11 @@ use http::{HeaderMap, header}; use serde::{Deserialize, Serialize}; use super::catalog::ProviderCatalog; -use super::ids::SourceDomain; +use super::ids::{IdentityProviderId, SourceDomain}; use super::input::RandomSource; -use super::revision::{IdentityKeys, record_revision}; -use crate::constants::COOKIE_TS_EIDS_S as COOKIE_S; +use super::mutation::{PartnerIdChange, ProviderStateChange}; +use super::revision::{IdentityKeys, Revision, record_revision}; +use crate::constants::{COOKIE_TS_EIDS_R_PREFIX, COOKIE_TS_EIDS_S as COOKIE_S}; use crate::ec::EcKvSnapshot; use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId}; @@ -881,6 +882,287 @@ pub(crate) fn cookie_view( .collect() } +/// Maximum sealed bytes of one `__Host-ts-eids-r-*` cookie. +pub(crate) const MAX_R_COOKIE_BYTES: usize = 512; +/// Openable removal cookies at which TS writes overflow batches to KV +/// instead of adding another cookie. +pub(crate) const R_COOKIE_SOFT_LIMIT: usize = 4; + +/// One staged removal. `s` is the source domain itself, never an index, +/// because removal cookies carry no `cf`. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RRemoval { + /// The source domain. + pub(crate) s: String, + /// The writer whose record is removed. + pub(crate) w: String, + /// The revision removed. Absent: whatever `w` acquired at or before `sa`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) pv: Option, + /// When the removal was staged. + pub(crate) sa: u64, +} + +/// One response's batch of removals, opt-outs and invalidated-UID hashes. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RBatch { + /// Removals. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub(crate) rm: Vec, + /// Opt-outs: provider to the time it was reported. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub(crate) o: BTreeMap, + /// Invalidated-UID hashes per provider. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub(crate) iv: BTreeMap>, +} + +impl RBatch { + /// Whether the batch holds nothing. + #[must_use] + pub(crate) fn is_empty(&self) -> bool { + self.rm.is_empty() && self.o.is_empty() && self.iv.is_empty() + } + + fn item_count(&self) -> usize { + self.rm.len() + self.o.len() + self.iv.values().map(Vec::len).sum::() + } +} + +/// The removal cookies a request carried. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct OpenedRemovals { + /// Cookies that opened, by name. + pub(crate) cookies: Vec<(String, RBatch)>, + /// Names of cookies that failed to open. The writer deletes them. + pub(crate) unopenable: Vec, +} + +/// Opens every `__Host-ts-eids-r-*` cookie for this EC. +#[must_use] +pub(crate) fn open_r_cookies( + keys: &IdentityKeys, + ec_id: &str, + headers: &HeaderMap, +) -> OpenedRemovals { + let mut opened = OpenedRemovals::default(); + for (name, value) in read_prefixed_cookies(headers, COOKIE_TS_EIDS_R_PREFIX) { + match open(&keys.cookie_r, ec_id, &value) + .and_then(|payload| serde_json::from_slice::(&payload).ok()) + { + Some(batch) => opened.cookies.push((name, batch)), + None => { + log::debug!("Ignoring removal cookie that failed to open"); + opened.unopenable.push(name); + } + } + } + opened +} + +/// Turns a batch into mutation changes. Entries whose source or provider +/// does not parse are dropped. +#[must_use] +pub(crate) fn r_batch_to_changes( + batch: &RBatch, +) -> ( + Vec, + BTreeMap, +) { + let mut changes = Vec::new(); + for removal in &batch.rm { + let (Ok(source), Ok(writer)) = ( + removal.s.parse::(), + removal.w.parse::(), + ) else { + continue; + }; + changes.push(match &removal.pv { + Some(pv) => PartnerIdChange::OwnerRemove { + source, + writer, + expected: Revision::from_stored(pv), + }, + None => PartnerIdChange::WriterRemove { + source, + writer, + staged_at: Some(removal.sa), + }, + }); + } + + let mut states: BTreeMap = BTreeMap::new(); + for (provider, at) in &batch.o { + let Ok(provider) = provider.parse::() else { + continue; + }; + // The batch carries no fingerprint: a re-grant by GPP section values + // cannot be compared against it, so only the other clears apply. + states.entry(provider).or_default().opt_out = Some(KvOptOut { + at: *at, + fp: String::new(), + }); + } + for (provider, hashes) in &batch.iv { + let Ok(provider) = provider.parse::() else { + continue; + }; + states + .entry(provider) + .or_default() + .iv_add + .extend(hashes.iter().cloned()); + } + (changes, states) +} + +/// Whether the live row shows every removal and opt-out of the batch +/// applied. `iv` hashes never keep a cookie alive. +#[must_use] +pub(crate) fn r_batch_applied(batch: &RBatch, entry: &KvEntry, keys: &IdentityKeys) -> bool { + let removals_applied = batch.rm.iter().all(|removal| { + let Some(record) = entry.ids.get(&removal.s) else { + return true; + }; + match &removal.pv { + Some(pv) => record_revision(keys, record).as_str() != pv, + None => { + record.w.as_deref() != Some(removal.w.as_str()) + || record.at.is_some_and(|at| at > removal.sa) + } + } + }); + let opt_outs_applied = batch.o.iter().all(|(provider, at)| { + entry.ps.get(provider).is_some_and(|state| { + state.o.as_ref().is_some_and(|opt_out| opt_out.at >= *at) + || state.oc.is_some_and(|cleared| cleared > *at) + }) + }); + removals_applied && opt_outs_applied +} + +/// A new removal cookie name: the prefix and 8 random hex digits. +#[must_use] +pub(crate) fn new_r_cookie_name(random: &dyn RandomSource) -> String { + let mut bytes = [0u8; 4]; + random.fill(&mut bytes); + format!("{COOKIE_TS_EIDS_R_PREFIX}{}", hex::encode(bytes)) +} + +fn seal_r_batch( + keys: &IdentityKeys, + ec_id: &str, + batch: &RBatch, + random: &dyn RandomSource, +) -> String { + let payload = serde_json::to_vec(batch).expect("should serialize a removal batch"); + seal(&keys.cookie_r, ec_id, &payload, random) +} + +/// Where a response's removal batch goes. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum RPlan { + /// A new cookie of its own. + Cookie { + /// The cookie name. + name: String, + /// The sealed value. + value: String, + }, + /// Written to KV before send (spec 6.2, overflow). + Overflow, +} + +/// Plans a batch: one new cookie when it fits and fewer than four openable +/// removal cookies are present, else an overflow write. +#[must_use] +pub(crate) fn plan_r_batch( + keys: &IdentityKeys, + ec_id: &str, + batch: &RBatch, + inbound_openable: usize, + random: &dyn RandomSource, +) -> RPlan { + if inbound_openable >= R_COOKIE_SOFT_LIMIT { + return RPlan::Overflow; + } + let value = seal_r_batch(keys, ec_id, batch, random); + if value.len() > MAX_R_COOKIE_BYTES { + return RPlan::Overflow; + } + RPlan::Cookie { + name: new_r_cookie_name(random), + value, + } +} + +/// Splits a batch across as many removal cookies as it needs, without +/// coarsening it: every removal, opt-out and hash is kept exactly. +#[must_use] +pub(crate) fn split_r_batch( + keys: &IdentityKeys, + ec_id: &str, + batch: &RBatch, + random: &dyn RandomSource, +) -> Vec<(String, String)> { + let mut items: Vec = Vec::with_capacity(batch.item_count()); + items.extend(batch.rm.iter().map(|removal| RBatch { + rm: vec![removal.clone()], + ..RBatch::default() + })); + items.extend(batch.o.iter().map(|(provider, at)| RBatch { + o: [(provider.clone(), *at)].into_iter().collect(), + ..RBatch::default() + })); + for (provider, hashes) in &batch.iv { + items.extend(hashes.iter().map(|hash| { + RBatch { + iv: [(provider.clone(), vec![hash.clone()])] + .into_iter() + .collect(), + ..RBatch::default() + } + })); + } + + let mut cookies = Vec::new(); + let mut current = RBatch::default(); + for item in items { + let mut candidate = current.clone(); + candidate.rm.extend(item.rm.clone()); + candidate.o.extend(item.o.clone()); + for (provider, hashes) in &item.iv { + candidate + .iv + .entry(provider.clone()) + .or_default() + .extend(hashes.iter().cloned()); + } + if current.is_empty() + || seal_r_batch(keys, ec_id, &candidate, random).len() <= MAX_R_COOKIE_BYTES + { + current = candidate; + continue; + } + cookies.push(current); + current = item; + } + if !current.is_empty() { + cookies.push(current); + } + + cookies + .into_iter() + .map(|batch| { + let value = seal_r_batch(keys, ec_id, &batch, random); + if value.len() > MAX_R_COOKIE_BYTES { + log::error!("A single removal does not fit a removal cookie; setting it anyway"); + } + (new_r_cookie_name(random), value) + }) + .collect() +} + #[cfg(test)] mod tests { use http::{HeaderMap, HeaderValue, header}; @@ -1331,3 +1613,296 @@ mod s_cookie_tests { ); } } + +#[cfg(test)] +mod r_cookie_tests { + use super::*; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::mutation::PartnerIdChange; + use crate::ec::identity::test_support::test_keys; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId, KvProviderState}; + + const EC: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ABC123"; + + fn removal(source: &str, writer: &str, sa: u64) -> RRemoval { + RRemoval { + s: source.to_owned(), + w: writer.to_owned(), + pv: None, + sa, + } + } + + fn batch_of(count: usize) -> RBatch { + RBatch { + rm: (0..count) + .map(|index| removal(&format!("s{index}.example.com"), "provider", 1_760_000_000)) + .collect(), + ..RBatch::default() + } + } + + fn headers_for(cookies: &[(String, String)]) -> http::HeaderMap { + let joined = cookies + .iter() + .map(|(name, value)| format!("{name}={value}")) + .collect::>() + .join("; "); + let mut headers = http::HeaderMap::new(); + headers.insert( + http::header::COOKIE, + http::HeaderValue::from_str(&joined).expect("should build header"), + ); + headers + } + + #[test] + fn a_typical_batch_fits_one_cookie_and_a_long_domain_fits_alone() { + let keys = test_keys(); + let random = SeededRandom::new(1); + let RPlan::Cookie { name, value } = plan_r_batch(&keys, EC, &batch_of(4), 0, &random) + else { + panic!("four typical removals should fit one cookie"); + }; + assert!( + name.starts_with("__Host-ts-eids-r-"), + "should use the prefix" + ); + assert_eq!( + name.len(), + "__Host-ts-eids-r-".len() + 8, + "should add 8 hex digits" + ); + assert!(value.len() <= MAX_R_COOKIE_BYTES, "should fit 512 bytes"); + + let label = "a".repeat(63); + let long_domain = format!("{label}.{label}.{label}.{}.com", "b".repeat(57)); + assert_eq!(long_domain.len(), 253, "should be a 253-byte domain"); + let one = RBatch { + rm: vec![removal(&long_domain, "a-long-provider-id", 1_760_000_000)], + ..RBatch::default() + }; + assert!( + matches!( + plan_r_batch(&keys, EC, &one, 0, &random), + RPlan::Cookie { .. } + ), + "one long removal fits" + ); + } + + #[test] + fn overflow_when_too_big_or_four_cookies_present() { + let keys = test_keys(); + let random = SeededRandom::new(1); + assert!( + matches!( + plan_r_batch(&keys, EC, &batch_of(12), 0, &random), + RPlan::Overflow + ), + "too big" + ); + assert!( + matches!( + plan_r_batch(&keys, EC, &batch_of(1), 4, &random), + RPlan::Overflow + ), + "four present" + ); + assert!( + matches!( + plan_r_batch(&keys, EC, &batch_of(1), 3, &random), + RPlan::Cookie { .. } + ), + "control: three present still allows a cookie" + ); + } + + #[test] + fn split_keeps_every_entry_exactly() { + let keys = test_keys(); + let random = SeededRandom::new(1); + let big = RBatch { + o: [("provider".to_owned(), 5)].into_iter().collect(), + iv: [("provider".to_owned(), vec!["0123456789abcdef".to_owned()])] + .into_iter() + .collect(), + ..batch_of(12) + }; + let cookies = split_r_batch(&keys, EC, &big, &random); + assert!(cookies.len() > 1, "should need several cookies"); + assert!( + cookies + .iter() + .all(|(_, value)| value.len() <= MAX_R_COOKIE_BYTES), + "each fits" + ); + let opened = open_r_cookies(&keys, EC, &headers_for(&cookies)); + assert!(opened.unopenable.is_empty(), "every cookie opens"); + let mut rebuilt = RBatch::default(); + for (_, batch) in &opened.cookies { + rebuilt.rm.extend(batch.rm.clone()); + rebuilt.o.extend(batch.o.clone()); + for (provider, hashes) in &batch.iv { + rebuilt + .iv + .entry(provider.clone()) + .or_default() + .extend(hashes.clone()); + } + } + rebuilt.rm.sort_by(|a, b| a.s.cmp(&b.s)); + let mut expected = big.rm.clone(); + expected.sort_by(|a, b| a.s.cmp(&b.s)); + assert_eq!(rebuilt.rm, expected, "every removal survives unchanged"); + assert_eq!(rebuilt.o, big.o, "the opt-out survives unchanged"); + assert_eq!(rebuilt.iv, big.iv, "the iv hash survives unchanged"); + } + + #[test] + fn applied_rules() { + let keys = test_keys(); + let mut entry = KvEntry::minimal("b.example.com", "x", 1); + entry.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + w: Some("p".to_owned()), + at: Some(10), + rv: Some("1111111111111111".to_owned()), + ..KvPartnerId::new("u") + }, + ); + let without_pv = |writer: &str, sa| RBatch { + rm: vec![removal("a.example.com", writer, sa)], + ..RBatch::default() + }; + assert!( + !r_batch_applied(&without_pv("p", 10), &entry, &keys), + "pending" + ); + assert!( + r_batch_applied(&without_pv("p", 9), &entry, &keys), + "re-acquired after sa" + ); + assert!( + r_batch_applied(&without_pv("q", 10), &entry, &keys), + "other writer" + ); + let with_pv = |pv: &str| RBatch { + rm: vec![RRemoval { + pv: Some(pv.to_owned()), + ..removal("a.example.com", "p", 10) + }], + ..RBatch::default() + }; + assert!( + !r_batch_applied(&with_pv("1111111111111111"), &entry, &keys), + "same revision" + ); + assert!( + r_batch_applied(&with_pv("2222222222222222"), &entry, &keys), + "other revision" + ); + + entry.ps.insert( + "p".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 5, + fp: "0".repeat(16), + }), + ..KvProviderState::default() + }, + ); + let opt_out = |at| RBatch { + o: [("p".to_owned(), at)].into_iter().collect(), + ..RBatch::default() + }; + assert!( + r_batch_applied(&opt_out(5), &entry, &keys), + "KV o.at at or after" + ); + assert!( + !r_batch_applied(&opt_out(6), &entry, &keys), + "newer opt-out pending" + ); + entry.ps.get_mut("p").expect("should hold p").oc = Some(6); + assert!( + !r_batch_applied(&opt_out(6), &entry, &keys), + "a same-second clear does not count" + ); + entry.ps.get_mut("p").expect("should hold p").oc = Some(7); + assert!( + r_batch_applied(&opt_out(6), &entry, &keys), + "a later clear makes it stale" + ); + let iv_only = RBatch { + iv: [("p".to_owned(), vec!["0".repeat(16)])] + .into_iter() + .collect(), + ..RBatch::default() + }; + assert!( + r_batch_applied(&iv_only, &entry, &keys), + "iv never keeps a cookie alive" + ); + } + + #[test] + fn batches_become_changes_by_source_domain() { + let batch = RBatch { + rm: vec![ + removal("a.example.com", "p", 10), + RRemoval { + pv: Some("1111111111111111".to_owned()), + ..removal("b.example.com", "p", 10) + }, + removal("not a domain", "p", 10), + ], + o: [("p".to_owned(), 7)].into_iter().collect(), + iv: [("p".to_owned(), vec!["0".repeat(16)])] + .into_iter() + .collect(), + }; + let (changes, states) = r_batch_to_changes(&batch); + assert_eq!( + changes.len(), + 2, + "should drop the entry that does not parse" + ); + assert!( + matches!( + &changes[0], + PartnerIdChange::WriterRemove { + staged_at: Some(10), + .. + } + ), + "a removal without pv is a WriterRemove" + ); + assert!( + matches!(&changes[1], PartnerIdChange::OwnerRemove { .. }), + "a removal with pv is an OwnerRemove" + ); + let state = &states[&"p".parse().expect("should parse id")]; + assert_eq!(state.opt_out.as_ref().map(|o| o.at), Some(7), "opt-out at"); + assert_eq!(state.iv_add.len(), 1, "iv hash"); + } + + #[test] + fn unopenable_cookies_are_listed_for_deletion() { + let keys = test_keys(); + let mut headers = http::HeaderMap::new(); + headers.insert( + http::header::COOKIE, + http::HeaderValue::from_static("__Host-ts-eids-r-00000000=v1.bad.value"), + ); + let opened = open_r_cookies(&keys, EC, &headers); + assert!(opened.cookies.is_empty(), "nothing opens"); + assert_eq!( + opened.unopenable, + vec!["__Host-ts-eids-r-00000000".to_owned()], + "listed for deletion" + ); + } +} From bb36616a32311ab8f0688055519a4c286501d169 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:26:33 +0530 Subject: [PATCH 20/33] Add [ec.identity] coordinator settings Validated at deploy and at startup. ec.pull_sync_concurrency is deprecated in favour of [ec.identity].max_concurrency and is still honoured when changed from its default. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/config.rs | 12 +- .../src/ec/identity/config.rs | 361 ++++++++++++++++++ .../src/ec/identity/mod.rs | 1 + .../src/migration_guards.rs | 4 + crates/trusted-server-core/src/settings.rs | 7 + trusted-server.example.toml | 29 ++ 6 files changed, 413 insertions(+), 1 deletion(-) create mode 100644 crates/trusted-server-core/src/ec/identity/config.rs diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index e48f5c2f03..e24dd3961e 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -13,6 +13,7 @@ use error_stack::Report; use serde::{Deserialize, Deserializer, Serialize, Serializer}; use validator::{Validate, ValidationError, ValidationErrors}; +use crate::ec::identity::config::validate_identity_settings; use crate::ec::registry::PartnerRegistry; use crate::error::TrustedServerError; use crate::integrations::{ @@ -32,7 +33,7 @@ use crate::integrations::{ sourcepoint::SourcepointConfig, testlight::TestlightConfig, }; -use crate::settings::{AssetOriginAuth, IntegrationConfig, Settings}; +use crate::settings::{AssetOriginAuth, Ec, IntegrationConfig, Settings}; const DEPLOY_VALIDATION_FIELD: &str = "trusted_server"; #[cfg(test)] @@ -258,6 +259,7 @@ pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report, + /// Shared wall-clock budget for all hold-mode providers, milliseconds. + #[serde(default = "IdentitySettings::default_hold_budget_ms")] + pub hold_budget_ms: u64, + /// Wait after a call is cut off by the hold budget, seconds. + #[serde(default = "IdentitySettings::default_hold_retry_sec")] + pub hold_retry_sec: u64, + /// Per provider response, before and after decoding. + #[serde(default = "IdentitySettings::default_max_response_bytes")] + pub max_response_bytes: usize, + /// Seconds a pending marker outlives its provider's timeout. + #[serde(default = "IdentitySettings::default_pending_grace_sec")] + pub pending_grace_sec: u64, + /// Staged cookie entries older than this are not written. + #[serde(default = "IdentitySettings::default_staged_max_age_sec")] + pub staged_max_age_sec: u64, + /// A cookie not confirmed against KV for this long forces a KV read. + #[serde(default = "IdentitySettings::default_kv_revalidate_sec")] + pub kv_revalidate_sec: u64, + /// Per-provider operator settings, keyed by provider ID. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub providers: BTreeMap, +} + +impl Default for IdentitySettings { + fn default() -> Self { + Self { + max_concurrency: None, + hold_budget_ms: Self::default_hold_budget_ms(), + hold_retry_sec: Self::default_hold_retry_sec(), + max_response_bytes: Self::default_max_response_bytes(), + pending_grace_sec: Self::default_pending_grace_sec(), + staged_max_age_sec: Self::default_staged_max_age_sec(), + kv_revalidate_sec: Self::default_kv_revalidate_sec(), + providers: BTreeMap::new(), + } + } +} + +impl IdentitySettings { + const fn default_hold_budget_ms() -> u64 { + 100 + } + + const fn default_hold_retry_sec() -> u64 { + 300 + } + + const fn default_max_response_bytes() -> usize { + 65_536 + } + + const fn default_pending_grace_sec() -> u64 { + 300 + } + + const fn default_staged_max_age_sec() -> u64 { + 604_800 + } + + const fn default_kv_revalidate_sec() -> u64 { + 3600 + } + + /// `max_concurrency`, else an explicitly changed legacy + /// `ec.pull_sync_concurrency`, else 4. + #[must_use] + pub fn effective_max_concurrency(&self, legacy_pull_concurrency: usize) -> usize { + self.max_concurrency + .unwrap_or(if legacy_pull_concurrency == LEGACY_PULL_SYNC_CONCURRENCY { + DEFAULT_MAX_CONCURRENCY + } else { + legacy_pull_concurrency + }) + } + + /// The settings for `id`: its table if there is one, else `base` (a + /// provider's own defaults, such as a pull partner's legacy rate limit), + /// else the defaults. + #[must_use] + pub fn provider( + &self, + id: &IdentityProviderId, + base: Option, + ) -> ProviderSettings { + self.providers + .get(id.as_str()) + .cloned() + .or(base) + .unwrap_or_default() + } +} + +/// When a provider is called. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum ProviderMode { + /// Before the auction, which waits within the hold budget. + Hold, + /// After the response has been sent. + #[default] + AfterSend, +} + +/// Operator settings for one provider (`[ec.identity.providers.]`). +#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct ProviderSettings { + /// When the provider is called. + #[serde(default)] + pub mode: ProviderMode, + /// Per-call timeout. Default: `hold_budget_ms` in hold mode, 1000 after + /// send. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub timeout_ms: Option, + /// Narrows the sources the provider may write. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub sources: Option>, + /// Refresh interval without a provider hint, and the wait after a skip. + #[serde(default = "ProviderSettings::default_refresh_after_sec")] + pub refresh_after_sec: u64, + /// Burst cap per visitor, in 60-second windows. + #[serde(default = "ProviderSettings::default_max_calls_per_minute")] + pub max_calls_per_minute: u32, + /// Transport failures per 60-second window that open the breaker. + #[serde(default = "ProviderSettings::default_breaker_failures_per_minute")] + pub breaker_failures_per_minute: u32, + /// Lower is dropped first when `ts-eids-s` exceeds its cap. + #[serde(default)] + pub cookie_priority: i32, +} + +impl Default for ProviderSettings { + fn default() -> Self { + Self { + mode: ProviderMode::default(), + timeout_ms: None, + sources: None, + refresh_after_sec: Self::default_refresh_after_sec(), + max_calls_per_minute: Self::default_max_calls_per_minute(), + breaker_failures_per_minute: Self::default_breaker_failures_per_minute(), + cookie_priority: 0, + } + } +} + +impl ProviderSettings { + const fn default_refresh_after_sec() -> u64 { + 86_400 + } + + const fn default_max_calls_per_minute() -> u32 { + 1 + } + + const fn default_breaker_failures_per_minute() -> u32 { + 50 + } + + /// The per-call timeout in milliseconds. + #[must_use] + pub fn effective_timeout_ms(&self, identity: &IdentitySettings) -> u64 { + self.timeout_ms.unwrap_or(match self.mode { + ProviderMode::Hold => identity.hold_budget_ms, + ProviderMode::AfterSend => 1000, + }) + } +} + +fn config_error(message: String) -> Report { + Report::new(TrustedServerError::Configuration { message }) +} + +/// Validates the numeric and per-provider rules of `[ec.identity]`. +/// +/// Rules that need the enabled providers (unknown IDs, ownership) run where +/// the provider catalog is built. +/// +/// # Errors +/// +/// Returns [`TrustedServerError::Configuration`] naming the first violation. +pub fn validate_identity_settings( + settings: &IdentitySettings, +) -> Result<(), Report> { + if settings.hold_budget_ms > MAX_HOLD_BUDGET_MS { + return Err(config_error(format!( + "ec.identity.hold_budget_ms must be at most {MAX_HOLD_BUDGET_MS}" + ))); + } + if settings.max_concurrency == Some(0) { + return Err(config_error( + "ec.identity.max_concurrency must be at least 1".to_owned(), + )); + } + if settings.kv_revalidate_sec >= TOMBSTONE_TTL_SECS { + return Err(config_error(format!( + "ec.identity.kv_revalidate_sec must be below {TOMBSTONE_TTL_SECS}" + ))); + } + for (id, provider) in &settings.providers { + if id.parse::().is_err() { + return Err(config_error(format!( + "ec.identity.providers.{id}: invalid provider id" + ))); + } + if provider.refresh_after_sec == 0 { + return Err(config_error(format!( + "ec.identity.providers.{id}.refresh_after_sec must be at least 1" + ))); + } + if provider.mode == ProviderMode::Hold + && provider.effective_timeout_ms(settings) > settings.hold_budget_ms + { + return Err(config_error(format!( + "ec.identity.providers.{id}.timeout_ms must not exceed hold_budget_ms in hold mode" + ))); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn parse(text: &str) -> IdentitySettings { + toml::from_str(text).expect("should parse identity settings") + } + + #[test] + fn defaults_without_any_table() { + let settings = parse(""); + assert_eq!(settings.hold_budget_ms, 100, "hold budget"); + assert_eq!(settings.hold_retry_sec, 300, "hold retry"); + assert_eq!(settings.max_response_bytes, 65_536, "max bytes"); + assert_eq!(settings.pending_grace_sec, 300, "grace"); + assert_eq!(settings.staged_max_age_sec, 604_800, "staged age"); + assert_eq!(settings.kv_revalidate_sec, 3600, "revalidate"); + assert_eq!( + settings.effective_max_concurrency(3), + 4, + "the default legacy value gives the new default" + ); + assert_eq!( + settings.effective_max_concurrency(6), + 6, + "an explicit legacy value is still honoured" + ); + assert_eq!( + parse("max_concurrency = 2").effective_max_concurrency(6), + 2, + "the new key wins" + ); + let provider = settings.provider(&"anything".parse().expect("should parse id"), None); + assert_eq!(provider.mode, ProviderMode::AfterSend, "mode"); + assert_eq!( + provider.effective_timeout_ms(&settings), + 1000, + "after-send timeout" + ); + assert_eq!( + ( + provider.refresh_after_sec, + provider.max_calls_per_minute, + provider.breaker_failures_per_minute, + provider.cookie_priority + ), + (86_400, 1, 50, 0), + "provider defaults" + ); + validate_identity_settings(&settings).expect("defaults are valid"); + } + + #[test] + fn provider_table_overrides_and_base_defaults() { + let settings = parse( + "hold_budget_ms = 120\n[providers.id5]\nmode = \"hold\"\nsources = [\"id5-sync.com\"]\ncookie_priority = 10\n", + ); + let provider = settings.provider(&"id5".parse().expect("should parse id"), None); + assert_eq!(provider.mode, ProviderMode::Hold, "hold mode"); + assert_eq!( + provider.effective_timeout_ms(&settings), + 120, + "hold timeout defaults to the budget" + ); + assert_eq!( + provider.sources, + Some(vec!["id5-sync.com".to_owned()]), + "narrowing" + ); + assert_eq!(provider.cookie_priority, 10, "priority"); + + let base = ProviderSettings { + max_calls_per_minute: 3, + ..ProviderSettings::default() + }; + assert_eq!( + settings + .provider(&"pull-x".parse().expect("should parse id"), Some(base)) + .max_calls_per_minute, + 3, + "a provider without a table uses its base" + ); + } + + #[test] + fn bounds_are_enforced() { + for (name, text) in [ + ("budget over 1000", "hold_budget_ms = 1001"), + ("zero concurrency", "max_concurrency = 0"), + ("revalidate a day", "kv_revalidate_sec = 86400"), + ("zero refresh", "[providers.p]\nrefresh_after_sec = 0"), + ( + "timeout over budget", + "[providers.p]\nmode = \"hold\"\ntimeout_ms = 101", + ), + ("bad provider id", "[providers.Bad]\nmode = \"hold\""), + ] { + assert!( + validate_identity_settings(&parse(text)).is_err(), + "should reject: {name}" + ); + } + assert!( + toml::from_str::("[providers.p]\nmode = \"sometimes\"").is_err(), + "should reject an unknown mode" + ); + assert!( + toml::from_str::("unknown_key = 1").is_err(), + "should reject unknown keys" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 7bbf56630e..63a1ea1cc8 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -14,6 +14,7 @@ )] pub(crate) mod catalog; +pub mod config; pub(crate) mod cookie; pub mod descriptor; pub mod exchange; diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 2d029fe476..007c0ddae4 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -87,6 +87,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/catalog.rs", include_str!("ec/identity/catalog.rs"), ), + ( + "ec/identity/config.rs", + include_str!("ec/identity/config.rs"), + ), ( "ec/identity/cookie.rs", include_str!("ec/identity/cookie.rs"), diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 3efc157701..407a3993bf 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -549,9 +549,16 @@ pub struct Ec { pub ec_store: Option, /// Maximum number of concurrent pull-sync requests. + /// + /// Deprecated: use `[ec.identity].max_concurrency`. A value other than + /// the default is still honoured when that key is unset. #[serde(default = "Ec::default_pull_sync_concurrency")] pub pull_sync_concurrency: usize, + /// Server-side identity provider policy (`[ec.identity]`). + #[serde(default)] + pub identity: crate::ec::identity::config::IdentitySettings, + /// Entries with `cluster_size` at or below this value are treated as /// individual users for identity resolution. B2B publishers should /// raise this to 50+ since readers are frequently on office networks. diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 066ad659e3..9ea30cb51f 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -85,6 +85,35 @@ pull_sync_concurrency = 3 # cluster_trust_threshold = 10 # entries with cluster_size <= this are individual users # cluster_recheck_secs = 3600 # re-evaluate cluster_size after this many seconds +# Optional server-side identity provider policy (defaults shown). Every +# table is optional. +# [ec.identity] +# Replaces ec.pull_sync_concurrency. +# max_concurrency = 4 +# Shared wall-clock budget for all hold-mode providers, in milliseconds. +# hold_budget_ms = 100 +# After a call is cut off by the hold budget, wait this long before the next. +# hold_retry_sec = 300 +# Per provider response, before and after decoding. +# max_response_bytes = 65536 +# Seconds a pending marker outlives its provider's timeout. +# pending_grace_sec = 300 +# Staged cookie entries older than this are not written. +# staged_max_age_sec = 604800 +# A cookie not confirmed against KV for this long forces a KV read. Must be +# below the 24-hour tombstone TTL. +# kv_revalidate_sec = 3600 +# +# Per-provider settings, keyed by provider ID. A pull partner's ID is +# "pull-" plus its source domain with dots replaced by "-". +# [ec.identity.providers.pull-partner-example-com] +# mode = "after_send" # or "hold" +# timeout_ms = 1000 # hold mode: at most hold_budget_ms +# refresh_after_sec = 86400 +# max_calls_per_minute = 1 +# breaker_failures_per_minute = 50 +# cookie_priority = 0 + # Example partner configuration. Provision referenced keys in # trusted_server_secrets before validating/pushing. # [[ec.partners]] From c0ee7acb872d8dce6bd78e1e9f366d465021ce72 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:32:25 +0530 Subject: [PATCH 21/33] Add coordinator eligibility and outcome rules Adds the merged eligibility view of KV, the identity cookies and removal batches, the eligibility steps, and the mapping from each call outcome to KV changes and backoff. Also adds the consent fingerprint bits a provider opt-out records. Signed-off-by: dhruv8sh --- .../src/ec/identity/coordinator.rs | 1608 +++++++++++++++++ .../src/ec/identity/mod.rs | 1 + .../src/ec/identity/policy.rs | 76 + .../src/ec/identity/test_support.rs | 22 + .../src/migration_guards.rs | 4 + 5 files changed, 1711 insertions(+) create mode 100644 crates/trusted-server-core/src/ec/identity/coordinator.rs diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs new file mode 100644 index 0000000000..b5e835c3da --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -0,0 +1,1608 @@ +//! The identity coordinator: which providers run, and what their results +//! change. +//! +//! This module holds the pure rules: the merged eligibility view, the +//! eligibility steps, and the mapping from a call's outcome to KV changes. + +use std::collections::{BTreeMap, BTreeSet}; +use std::time::Duration; + +use super::catalog::ProviderCatalog; +use super::config::{IdentitySettings, ProviderMode, ProviderSettings}; +use super::cookie::{RBatch, SCookie, Staging, cookie_view}; +use super::descriptor::{Input, ProviderDescriptor, ResolveKind}; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::mutation::{ + ExpectedState, PartnerIdChange, ProviderResult, ProviderStateChange, StateUpdate, +}; +use super::policy::{PolicySignals, policy_allows}; +use super::provider::IdentityProvider; +use super::resolution::{Resolution, StateChange}; +use super::revision::{IdentityKeys, invalidated_uid_hash, record_revision}; +use crate::consent::ConsentContext; +use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId, MAX_UID_LENGTH}; + +/// Default wait after a `no_match`. +const NO_MATCH_RETRY: Duration = Duration::from_secs(24 * 60 * 60); +/// Longest wait a provider may ask for. +const MAX_RETRY: Duration = Duration::from_secs(7 * 24 * 60 * 60); +/// First wait after an input-caused failure, doubled per failure. +const INPUT_FAILURE_BASE_SECS: u64 = 60 * 60; +/// Longest wait after input-caused failures. +const INPUT_FAILURE_CAP_SECS: u64 = 24 * 60 * 60; +/// First wait after a transport failure, doubled per failure. +const TRANSPORT_FAILURE_BASE_SECS: u64 = 5 * 60; +/// Longest wait after transport failures. +const TRANSPORT_FAILURE_CAP_SECS: u64 = 60 * 60; + +/// One provider's state in the merged view. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct ViewState { + /// Next eligible time: the later of cookie and KV. + pub(crate) n: Option, + /// Consecutive failures. + pub(crate) f: u32, + /// Sticky opt-out. + pub(crate) o: Option, + /// Latest clear of the opt-out. + pub(crate) oc: Option, + /// Invalidated-UID hashes: the union of cookie, KV and removal cookies. + pub(crate) iv: Vec, + /// Pending marker, from the cookie only. + pub(crate) marker: Option, + /// Private state, from KV only. + pub(crate) st: Option, +} + +impl ViewState { + fn opted_out(&self) -> bool { + self.o + .as_ref() + .is_some_and(|opt_out| self.oc.is_none_or(|cleared| cleared <= opt_out.at)) + } +} + +/// The merged view of stored state that eligibility reads, on every route. +/// +/// KV is the base when the request read it, else the cookie's last confirmed +/// copy. Staged cookie entries overlay it when their revision still matches. +/// Removal cookies hide the records they name and add their opt-outs. +/// Pending markers come from the cookie only. +#[derive(Debug, Clone, Default)] +pub(crate) struct EligibilityView { + records: BTreeMap, + states: BTreeMap, +} + +impl EligibilityView { + /// A view with nothing stored. + #[must_use] + pub(crate) fn empty() -> Self { + Self::default() + } + + /// Builds the merged view. + #[must_use] + pub(crate) fn merge( + kv: Option<&KvEntry>, + cookie: Option<&SCookie>, + removals: &[RBatch], + catalog: &ProviderCatalog, + keys: &IdentityKeys, + ) -> Self { + let mut view = Self::default(); + let view_entries = cookie.map(|cookie| cookie_view(cookie, catalog)); + + if let Some(entry) = kv { + for (source, record) in &entry.ids { + if let Ok(source) = source.parse::() { + view.records.insert(source, record.clone()); + } + } + for (provider, state) in &entry.ps { + view.states.insert( + provider.clone(), + ViewState { + n: state.n, + f: state.f, + o: state.o.clone(), + oc: state.oc, + iv: state.iv.clone(), + marker: None, + st: state.st.clone(), + }, + ); + } + } else if let Some(entries) = &view_entries { + for (source, record, staging) in entries { + if *staging == Staging::Confirmed { + view.records.insert(source.clone(), record.clone()); + } + } + } + + if let Some(entries) = &view_entries { + for (source, record, staging) in entries { + let Staging::Staged { pv, browser, .. } = staging else { + continue; + }; + let base = view.records.get(source); + let overlay = if *browser { + base.is_none() + } else { + match (pv, base) { + (Some(pv), Some(base)) => record_revision(keys, base).as_str() == pv, + (Some(_), None) => false, + (None, base) => base.is_none_or(|base| base.w != record.w), + } + }; + if overlay { + view.records.insert(source.clone(), record.clone()); + } + } + } + + if let Some(cookie) = cookie { + for (provider, cookie_state) in &cookie.ps { + let state = view.states.entry(provider.clone()).or_default(); + state.n = state.n.max(cookie_state.n); + state.marker = cookie_state.p; + for hash in &cookie_state.iv { + if !state.iv.contains(hash) { + state.iv.push(hash.clone()); + } + } + // A mirrored opt-out yields to KV when this request read it. + if kv.is_none() && state.o.is_none() { + state.o.clone_from(&cookie_state.o); + } + state.oc = state.oc.max(cookie_state.oc); + if kv.is_none() && cookie_state.f.is_some() { + state.f = cookie_state.f.unwrap_or_default(); + } + } + } + + for batch in removals { + for removal in &batch.rm { + let Ok(source) = removal.s.parse::() else { + continue; + }; + let hidden = view + .records + .get(&source) + .is_some_and(|record| match &removal.pv { + Some(pv) => record_revision(keys, record).as_str() == pv, + None => { + record.w.as_deref() == Some(removal.w.as_str()) + && record.at.unwrap_or(0) <= removal.sa + } + }); + if hidden { + view.records.remove(&source); + } + } + for (provider, at) in &batch.o { + let state = view.states.entry(provider.clone()).or_default(); + if state.o.as_ref().is_none_or(|current| *at > current.at) { + state.o = Some(KvOptOut { + at: *at, + fp: String::new(), + }); + } + } + for (provider, hashes) in &batch.iv { + let state = view.states.entry(provider.clone()).or_default(); + for hash in hashes { + if !state.iv.contains(hash) { + state.iv.push(hash.clone()); + } + } + } + } + view + } + + /// The merged record for `source`. + #[must_use] + pub(crate) fn record(&self, source: &SourceDomain) -> Option<&KvPartnerId> { + self.records.get(source) + } + + /// The merged state of `provider`. + #[must_use] + pub(crate) fn state(&self, provider: &str) -> Option<&ViewState> { + self.states.get(provider) + } + + /// Whether `provider` has a sticky opt-out that is not cleared. + #[must_use] + pub(crate) fn is_opted_out(&self, provider: &IdentityProviderId) -> bool { + self.state(provider.as_str()) + .is_some_and(ViewState::opted_out) + } + + #[cfg(test)] + pub(crate) fn set_failures_for_test(&mut self, provider: &str, failures: u32) { + self.states.entry(provider.to_owned()).or_default().f = failures; + } + + #[cfg(test)] + pub(crate) fn set_marker_for_test(&mut self, provider: &str, marker: u64) { + self.states.entry(provider.to_owned()).or_default().marker = Some(marker); + } +} + +/// When a provider would run in this request. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Phase { + /// Before the auction. + Hold, + /// The phase-2 dry run of an after-send provider. + AfterSendDryRun, + /// After the response has been sent. + AfterSend, +} + +/// What the request is. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct RequestFacts { + pub(crate) is_real_browser: bool, + /// Global EC and EID-persistence consent. + pub(crate) ec_persistence_allowed: bool, + pub(crate) policy_signals: PolicySignals, + /// The writer request of the page load. + pub(crate) writer: bool, + /// A new visitor's first navigation. + pub(crate) new_visitor: bool, + /// The request runs an auction. + pub(crate) runs_auction: bool, + pub(crate) now: super::ids::UnixTime, + /// Inputs this request can supply. + pub(crate) available_inputs: BTreeSet, +} + +/// Why a provider is not eligible. +#[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] +pub(crate) enum SkipReason { + #[display("not_browser")] + NotBrowser, + #[display("persistence_denied")] + PersistenceDenied, + #[display("policy_denied")] + PolicyDenied, + #[display("not_resolving")] + NotResolving, + #[display("new_visitor")] + NewVisitor, + #[display("mode")] + Mode, + #[display("nothing_due")] + NothingDue, + #[display("backoff")] + Backoff, + #[display("opted_out")] + OptedOut, + #[display("pending_marker")] + PendingMarker, +} + +/// Whether a provider runs, and for which sources and kinds. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum Eligibility { + Eligible { + kinds: BTreeMap, + }, + Skip(SkipReason), +} + +/// The request-wide inputs of an eligibility decision. +pub(crate) struct EligibilityRequest<'a> { + pub(crate) identity: &'a IdentitySettings, + pub(crate) view: &'a EligibilityView, + pub(crate) facts: &'a RequestFacts, + pub(crate) consent: &'a ConsentContext, + pub(crate) catalog: &'a ProviderCatalog, + pub(crate) keys: &'a IdentityKeys, +} + +/// The sources a provider may write after the operator's `sources` +/// narrowing. +#[must_use] +pub(crate) fn claimed_sources<'a>( + descriptor: &'a ProviderDescriptor, + settings: &ProviderSettings, +) -> Vec<&'a SourceDomain> { + descriptor + .sources() + .iter() + .filter(|source| { + settings + .sources + .as_ref() + .is_none_or(|narrowed| narrowed.iter().any(|name| source.matches(name))) + }) + .collect() +} + +fn writer_configured(record: &KvPartnerId, catalog: &ProviderCatalog) -> bool { + record.w.as_deref().is_none_or(|writer| { + writer + .parse::() + .is_ok_and(|writer| catalog.is_configured(&writer)) + }) +} + +fn expired(record: &KvPartnerId, now: u64) -> bool { + record.exp.is_some_and(|exp| now >= exp) +} + +/// Whether the owner's stored record is due for refresh. +fn owned_record_due( + record: &KvPartnerId, + owner: &IdentityProviderId, + settings: &ProviderSettings, + request: &EligibilityRequest<'_>, +) -> bool { + let now = request.facts.now.as_secs(); + if record.w.is_none() { + let hash = invalidated_uid_hash(request.keys, owner, &record.uid); + let rejected = request + .view + .state(owner.as_str()) + .is_some_and(|state| state.iv.contains(&hash)); + if rejected { + return false; + } + } + if expired(record, now) || record.w.as_deref() != Some(owner.as_str()) { + return true; + } + match record.ra { + Some(refresh_at) => now >= refresh_at, + None => record + .at + .is_none_or(|at| now >= at.saturating_add(settings.refresh_after_sec)), + } +} + +fn mode_allows( + resolves_state: bool, + settings: &ProviderSettings, + phase: Phase, + facts: &RequestFacts, +) -> bool { + match phase { + Phase::Hold => { + settings.mode == ProviderMode::Hold + && facts.runs_auction + && (!resolves_state || facts.writer) + } + Phase::AfterSendDryRun | Phase::AfterSend => { + facts.writer + && (settings.mode == ProviderMode::AfterSend + || (resolves_state + && settings.mode == ProviderMode::Hold + && !facts.runs_auction)) + } + } +} + +/// Decides whether `provider` runs in `phase` (spec 4.7, steps 1 to 8). The +/// burst cap, step 9, is checked at dispatch, because the limiter counts on +/// every check. +#[must_use] +pub(crate) fn eligibility( + provider: &dyn IdentityProvider, + settings: &ProviderSettings, + phase: Phase, + request: &EligibilityRequest<'_>, +) -> Eligibility { + let descriptor = provider.descriptor(); + let facts = request.facts; + if !facts.is_real_browser { + return Eligibility::Skip(SkipReason::NotBrowser); + } + if !facts.ec_persistence_allowed { + return Eligibility::Skip(SkipReason::PersistenceDenied); + } + if !policy_allows( + descriptor.consent_policy(), + request.consent, + facts.policy_signals, + ) { + return Eligibility::Skip(SkipReason::PolicyDenied); + } + let Some(resolves) = descriptor.resolves() else { + return Eligibility::Skip(SkipReason::NotResolving); + }; + if facts.new_visitor { + return Eligibility::Skip(SkipReason::NewVisitor); + } + if !mode_allows(resolves.has_private_state(), settings, phase, facts) { + return Eligibility::Skip(SkipReason::Mode); + } + + let id = descriptor.id(); + let now = facts.now.as_secs(); + let mut kinds = BTreeMap::new(); + for source in claimed_sources(descriptor, settings) { + let record = request.view.record(source); + let kind = if request.catalog.owner_of(source) == Some(id) { + match record { + None => Some(ResolveKind::Initial), + Some(record) => { + owned_record_due(record, id, settings, request).then_some(ResolveKind::Refresh) + } + } + } else { + let absent = record.is_none_or(|record| { + expired(record, now) || !writer_configured(record, request.catalog) + }); + absent.then_some(ResolveKind::Initial) + }; + let Some(kind) = kind else { + continue; + }; + let needs_met = resolves + .needs(kind) + .iter() + .all(|input| facts.available_inputs.contains(input)); + if resolves.kinds().contains(&kind) && needs_met { + kinds.insert(source.clone(), kind); + } + } + if kinds.is_empty() { + log::debug!("Identity provider {id} skipped: nothing due"); + return Eligibility::Skip(SkipReason::NothingDue); + } + + let state = request.view.state(id.as_str()); + if state.is_some_and(|state| state.n.is_some_and(|next| next > now)) { + return Eligibility::Skip(SkipReason::Backoff); + } + if state.is_some_and(ViewState::opted_out) { + return Eligibility::Skip(SkipReason::OptedOut); + } + if state.is_some_and(|state| state.marker.is_some()) { + return Eligibility::Skip(SkipReason::PendingMarker); + } + Eligibility::Eligible { kinds } +} + +/// How one provider call ended. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum CallOutcome { + /// The provider answered. + Resolved(Resolution), + /// `request` returned `None`. + Skipped, + /// A declared input was missing. + InputUnavailable, + /// Stored state did not decode. Carries the result, if the provider + /// still returned one. + InvalidState(Option), + /// The hold budget ran out first. + HoldCutOff, + /// A 4xx other than 429, `Config`, `UndeclaredInput` or `Malformed`. + InputFailure, + /// A transport error, timeout, 429 or 5xx. + TransportFailure, + /// The circuit breaker was open. + BreakerOpen, +} + +/// What turning an outcome into changes needs. +pub(crate) struct OutcomeContext<'a> { + /// Unix seconds. + pub(crate) now: u64, + pub(crate) keys: &'a IdentityKeys, + pub(crate) catalog: &'a ProviderCatalog, + pub(crate) provider_settings: &'a ProviderSettings, + pub(crate) identity: &'a IdentitySettings, + /// The `ps.st` the call read. + pub(crate) expected_state: ExpectedState, + /// The consent fingerprint an opt-out records. + pub(crate) fingerprint: String, +} + +/// The KV changes one call produced. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct OutcomeChanges { + /// Removals: they always apply on their own. + pub(crate) removals: Vec, + /// Positive changes of a stateless provider. + pub(crate) positive: Vec, + /// The atomic positive result of a stateful provider. + pub(crate) result: Option, + /// Scheduling, opt-out, `iv` and state changes outside the result. + pub(crate) state: ProviderStateChange, +} + +fn exponential_backoff(failures: u32, base: u64, cap: u64) -> u64 { + let doublings = failures.saturating_sub(1).min(32); + base.saturating_mul(1u64 << doublings).min(cap) +} + +/// Turns one call's outcome into KV changes (spec 8). +#[must_use] +pub(crate) fn outcome_to_changes( + provider: &dyn IdentityProvider, + outcome: CallOutcome, + view: &EligibilityView, + context: &OutcomeContext<'_>, +) -> OutcomeChanges { + let id = provider.descriptor().id(); + let failures = view.state(id.as_str()).map_or(0, |state| state.f); + let now = context.now; + let mut changes = OutcomeChanges::default(); + match outcome { + CallOutcome::BreakerOpen => {} + CallOutcome::Skipped | CallOutcome::InputUnavailable => { + changes.state.n = Some(now + context.provider_settings.refresh_after_sec); + } + CallOutcome::HoldCutOff => { + changes.state.n = Some(now + context.identity.hold_retry_sec); + } + CallOutcome::InputFailure => { + input_failure(&mut changes.state, failures, now); + } + CallOutcome::TransportFailure => { + let failures = failures.saturating_add(1); + changes.state.f = Some(failures); + changes.state.n = Some( + now + exponential_backoff( + failures, + TRANSPORT_FAILURE_BASE_SECS, + TRANSPORT_FAILURE_CAP_SECS, + ), + ); + } + CallOutcome::InvalidState(resolution) => { + if let Some(resolution) = resolution { + changes = resolved(provider, &resolution, view, context, failures); + } + changes.state.st = StateUpdate::Clear; + changes.state.f = None; + if let Some(result) = changes.result.as_mut() { + result.state.st = StateUpdate::Keep; + result.state.f = None; + } + let result_sets_n = changes + .result + .as_ref() + .is_some_and(|result| result.state.n.is_some()); + if changes.state.n.is_none() && !result_sets_n { + changes.state.n = Some(now + context.provider_settings.refresh_after_sec); + } + } + CallOutcome::Resolved(resolution) => { + changes = resolved(provider, &resolution, view, context, failures); + } + } + changes +} + +fn input_failure(state: &mut ProviderStateChange, failures: u32, now: u64) { + let failures = failures.saturating_add(1); + state.f = Some(failures); + state.n = + Some(now + exponential_backoff(failures, INPUT_FAILURE_BASE_SECS, INPUT_FAILURE_CAP_SECS)); +} + +fn resolved( + provider: &dyn IdentityProvider, + resolution: &Resolution, + view: &EligibilityView, + context: &OutcomeContext<'_>, + failures: u32, +) -> OutcomeChanges { + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let now = context.now; + let stateful = descriptor + .resolves() + .is_some_and(super::descriptor::Resolves::has_private_state); + let claimed = claimed_sources(descriptor, context.provider_settings); + let mut changes = OutcomeChanges::default(); + + if !stateful && resolution.changes_state() { + log::warn!("Identity provider {id} returned state without declaring it"); + input_failure(&mut changes.state, failures, now); + return changes; + } + + if resolution.is_opted_out() { + for source in &claimed { + changes.removals.push(PartnerIdChange::WriterRemove { + source: (*source).clone(), + writer: id.clone(), + staged_at: None, + }); + } + changes.state.opt_out = Some(KvOptOut { + at: now, + fp: context.fingerprint.clone(), + }); + changes.state.st = StateUpdate::Clear; + log::info!("Identity provider {id} reported an opt-out"); + return changes; + } + + for source in resolution.invalidated() { + if !claimed.contains(&source) { + continue; + } + if let Some(record) = view.record(source) { + changes.removals.push(PartnerIdChange::OwnerRemove { + source: source.clone(), + writer: id.clone(), + expected: record_revision(context.keys, record), + }); + changes + .state + .iv_add + .push(invalidated_uid_hash(context.keys, id, &record.uid)); + } + changes.state.n = Some(now); + log::info!("Identity provider {id} invalidated source {source}"); + } + + if resolution.state_rejected() { + log::warn!("Identity provider {id} returned an oversized or unserializable state"); + input_failure(&mut changes.state, failures, now); + return changes; + } + + let (kept, dropped) = resolution.found_ids_by_source(); + if dropped > 0 { + log::debug!("Identity provider {id} returned {dropped} duplicate IDs"); + } + let mut positive = Vec::new(); + for resolved_id in kept { + let source = resolved_id.source(); + if !claimed.contains(&source) { + log::debug!("Identity provider {id} returned unclaimed source {source}"); + continue; + } + if resolved_id.uid().is_empty() || resolved_id.uid().len() > MAX_UID_LENGTH { + log::debug!("Identity provider {id} returned an invalid UID for {source}"); + continue; + } + let stored = view.record(source); + let mut exp = resolved_id + .expiry() + .map(|when| when.resolve(super::ids::UnixTime::from_secs(now)).as_secs()); + if exp.is_none() && descriptor.is_expiring(source) { + match stored { + Some(stored) if stored.uid == resolved_id.uid() && stored.exp.is_some() => { + exp = stored.exp; + } + _ => { + log::debug!( + "Identity provider {id} returned an expiring ID without an expiry for {source}" + ); + continue; + } + } + } + let refresh_until = match resolved_id.refresh_until() { + Some(until) if stateful => Some(until.as_secs()), + Some(_) => { + log::warn!("Identity provider {id} set refresh_until without private state"); + None + } + None => None, + }; + let record = KvPartnerId { + uid: resolved_id.uid().to_owned(), + w: None, + rv: None, + at: Some(now), + exp, + ra: resolved_id + .refresh() + .map(|when| when.resolve(super::ids::UnixTime::from_secs(now)).as_secs()), + ru: refresh_until, + atype: resolved_id.atype(), + ext: resolved_id.ext().cloned(), + }; + positive.push(if context.catalog.owner_of(source) == Some(id) { + PartnerIdChange::OwnerWrite { + source: source.clone(), + record, + writer: id.clone(), + expected: stored.map(|stored| record_revision(context.keys, stored)), + staged_at: None, + } + } else { + PartnerIdChange::FillMissing { + source: source.clone(), + record, + writer: Some(id.clone()), + staged_at: None, + } + }); + } + + let mut scheduling = ProviderStateChange { + f: Some(0), + ..ProviderStateChange::default() + }; + if resolution.is_no_match() { + let retry = resolution + .retry_in() + .unwrap_or(NO_MATCH_RETRY) + .min(MAX_RETRY); + scheduling.n = Some(now + retry.as_secs()); + } else if let Some(retry) = resolution.retry_in() { + scheduling.n = Some(now + retry.min(MAX_RETRY).as_secs()); + } + scheduling.st = match resolution.state_change() { + StateChange::Keep => StateUpdate::Keep, + StateChange::Replace(json) => serde_json::from_str(json) + .map(StateUpdate::Replace) + .unwrap_or(StateUpdate::Keep), + StateChange::Clear => StateUpdate::Clear, + }; + + if stateful && (!positive.is_empty() || scheduling.st != StateUpdate::Keep) { + changes.result = Some(ProviderResult { + provider: id.clone(), + ids: positive, + state: scheduling, + expected_state: context.expected_state.clone(), + }); + } else { + changes.positive = positive; + changes.state.n = changes.state.n.max(scheduling.n); + changes.state.f = scheduling.f; + changes.state.st = scheduling.st; + } + changes +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + use std::sync::Arc; + use std::time::Duration; + + use serde_json::json; + + use super::*; + use crate::ec::identity::descriptor::{ConsentPolicy, ProviderDescriptor, Resolves}; + use crate::ec::identity::mutation::ExpectedState; + use crate::ec::identity::resolution::{Resolution, ResolvedId}; + use crate::ec::identity::revision::{IdentityKeys, Revision, invalidated_uid_hash}; + use crate::ec::identity::test_support::{ + FixedProvider, TestSignals, allowing_consent, consent_with, facts_for_writer, + fixed_provider, test_keys, + }; + use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId, KvProviderState}; + + const NOW: u64 = 1_000; + + fn src(name: &str) -> SourceDomain { + name.parse().expect("should parse source") + } + + fn pid(name: &str) -> IdentityProviderId { + name.parse().expect("should parse id") + } + + struct Fixture { + keys: IdentityKeys, + catalog: ProviderCatalog, + identity: IdentitySettings, + consent: crate::consent::ConsentContext, + } + + impl Fixture { + fn new(catalog: ProviderCatalog) -> Self { + Self { + keys: test_keys(), + catalog, + identity: IdentitySettings::default(), + consent: allowing_consent(), + } + } + + fn unowned(id: &str, sources: &[&str]) -> Self { + Self::new(ProviderCatalog::new( + vec![fixed_provider(id, sources)], + BTreeMap::new(), + )) + } + + fn owned(id: &str, source: &str) -> Self { + Self::new(ProviderCatalog::new( + vec![fixed_provider(id, &[source])], + [(src(source), pid(id))].into_iter().collect(), + )) + } + + fn provider(&self, id: &str) -> &dyn IdentityProvider { + self.catalog + .get(&pid(id)) + .expect("should hold the provider") + .as_ref() + } + + fn eligibility( + &self, + id: &str, + view: &EligibilityView, + facts: &RequestFacts, + phase: Phase, + settings: &ProviderSettings, + ) -> Eligibility { + eligibility( + self.provider(id), + settings, + phase, + &EligibilityRequest { + identity: &self.identity, + view, + facts, + consent: &self.consent, + catalog: &self.catalog, + keys: &self.keys, + }, + ) + } + + fn outcome( + &self, + id: &str, + outcome: CallOutcome, + view: &EligibilityView, + ) -> OutcomeChanges { + let settings = ProviderSettings::default(); + outcome_to_changes( + self.provider(id), + outcome, + view, + &OutcomeContext { + now: NOW, + keys: &self.keys, + catalog: &self.catalog, + provider_settings: &settings, + identity: &self.identity, + expected_state: ExpectedState::NotRead, + fingerprint: "0123456789abcdef".to_owned(), + }, + ) + } + } + + fn view_with(entry: &KvEntry, fixture: &Fixture) -> EligibilityView { + EligibilityView::merge(Some(entry), None, &[], &fixture.catalog, &fixture.keys) + } + + fn after_send() -> ProviderSettings { + ProviderSettings::default() + } + + fn kinds_of(eligibility: &Eligibility) -> Vec<(String, ResolveKind)> { + match eligibility { + Eligibility::Eligible { kinds } => kinds + .iter() + .map(|(source, kind)| (source.to_string(), *kind)) + .collect(), + Eligibility::Skip(reason) => panic!("should be eligible, skipped for {reason:?}"), + } + } + + fn skip_of(eligibility: &Eligibility) -> SkipReason { + match eligibility { + Eligibility::Skip(reason) => *reason, + Eligibility::Eligible { .. } => panic!("should be skipped"), + } + } + + #[test] + fn global_gates_come_first() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let view = EligibilityView::empty(); + let bot = RequestFacts { + is_real_browser: false, + ..facts_for_writer() + }; + assert_eq!( + skip_of(&fixture.eligibility("p", &view, &bot, Phase::AfterSend, &after_send())), + SkipReason::NotBrowser, + "a bot is never eligible" + ); + let denied = RequestFacts { + ec_persistence_allowed: false, + ..facts_for_writer() + }; + assert_eq!( + skip_of(&fixture.eligibility("p", &view, &denied, Phase::AfterSend, &after_send())), + SkipReason::PersistenceDenied, + "EID persistence consent is required" + ); + assert_eq!( + kinds_of(&fixture.eligibility( + "p", + &view, + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + vec![("a.example.com".to_owned(), ResolveKind::Initial)], + "control: a returning writer is eligible for an absent source" + ); + } + + #[test] + fn provider_policy_is_checked() { + let strict = ConsentPolicy::strict().with_tcf_vendor(9999); + let mut fixture = Fixture::new(ProviderCatalog::new( + vec![crate::ec::identity::test_support::provider_with_policy( + "p", + &["a.example.com"], + strict, + false, + )], + BTreeMap::new(), + )); + fixture.consent = consent_with(&TestSignals { + gdpr: true, + purposes: vec![1, 4], + ..TestSignals::default() + }); + assert_eq!( + skip_of(&fixture.eligibility( + "p", + &EligibilityView::empty(), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::PolicyDenied, + "missing vendor consent blocks the provider" + ); + } + + #[test] + fn skips_new_visitor_in_every_mode() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let view = EligibilityView::empty(); + let hold = ProviderSettings { + mode: ProviderMode::Hold, + ..ProviderSettings::default() + }; + for (phase, settings) in [ + (Phase::Hold, &hold), + (Phase::AfterSendDryRun, &after_send()), + (Phase::AfterSend, &after_send()), + ] { + let facts = RequestFacts { + new_visitor: true, + runs_auction: true, + ..facts_for_writer() + }; + assert_eq!( + skip_of(&fixture.eligibility("p", &view, &facts, phase, settings)), + SkipReason::NewVisitor, + "no provider runs on a new visitor's first navigation ({phase:?})" + ); + } + } + + #[test] + fn owned_record_written_by_someone_else_is_due_for_refresh() { + let fixture = Fixture::owned("owner", "a.example.com"); + let mut entry = KvEntry::minimal("a.example.com", "seeded", 1); + entry + .ids + .get_mut("a.example.com") + .expect("should hold a") + .at = Some(NOW - 10); + assert_eq!( + kinds_of(&fixture.eligibility( + "owner", + &view_with(&entry, &fixture), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + vec![("a.example.com".to_owned(), ResolveKind::Refresh)], + "a browser-seeded record is due at once" + ); + + let mut own = KvEntry::minimal("z.example.com", "x", 1); + own.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + w: Some("owner".to_owned()), + at: Some(NOW - 10), + ..KvPartnerId::new("fresh") + }, + ); + assert_eq!( + skip_of(&fixture.eligibility( + "owner", + &view_with(&own, &fixture), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::NothingDue, + "its own recent record is not due" + ); + + own.ids.get_mut("a.example.com").expect("should hold a").ra = Some(NOW); + assert_eq!( + kinds_of(&fixture.eligibility( + "owner", + &view_with(&own, &fixture), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + vec![("a.example.com".to_owned(), ResolveKind::Refresh)], + "ra passed makes it due" + ); + } + + #[test] + fn browser_seeded_uid_in_iv_is_never_due() { + let fixture = Fixture::owned("owner", "a.example.com"); + let mut entry = KvEntry::minimal("a.example.com", "rejected", 1); + entry.ps.insert( + "owner".to_owned(), + KvProviderState { + iv: vec![invalidated_uid_hash( + &fixture.keys, + &pid("owner"), + "rejected", + )], + ..KvProviderState::default() + }, + ); + assert_eq!( + skip_of(&fixture.eligibility( + "owner", + &view_with(&entry, &fixture), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::NothingDue, + "the owner does not refresh a UID it rejected" + ); + } + + #[test] + fn unowned_source_with_servable_record_is_not_initial_but_expired_counts_as_absent() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let held = KvEntry::minimal("a.example.com", "x", 1); + assert_eq!( + skip_of(&fixture.eligibility( + "p", + &view_with(&held, &fixture), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::NothingDue, + "unowned sources are never refreshed" + ); + let mut expired = KvEntry::minimal("z.example.com", "x", 1); + expired.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + exp: Some(NOW), + ..KvPartnerId::new("old") + }, + ); + assert_eq!( + kinds_of(&fixture.eligibility( + "p", + &view_with(&expired, &fixture), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + vec![("a.example.com".to_owned(), ResolveKind::Initial)], + "an expired record counts as absent" + ); + } + + #[test] + fn missing_needs_and_narrowing_leave_sources_out() { + let descriptor = ProviderDescriptor::builder(pid("p")) + .with_source(src("a.example.com")) + .with_source(src("b.example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.example.com") + .with_needs_for(ResolveKind::Initial, [Input::HashedEmail]), + ) + .build() + .expect("should build"); + let fixture = Fixture::new(ProviderCatalog::new( + vec![Arc::new(FixedProvider(descriptor))], + BTreeMap::new(), + )); + assert_eq!( + skip_of(&fixture.eligibility( + "p", + &EligibilityView::empty(), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::NothingDue, + "no HEM, no Initial" + ); + + let fixture = Fixture::unowned("q", &["a.example.com", "b.example.com"]); + let narrowed = ProviderSettings { + sources: Some(vec!["b.example.com".to_owned()]), + ..ProviderSettings::default() + }; + assert_eq!( + kinds_of(&fixture.eligibility( + "q", + &EligibilityView::empty(), + &facts_for_writer(), + Phase::AfterSend, + &narrowed + )), + vec![("b.example.com".to_owned(), ResolveKind::Initial)], + "sources narrowing applies" + ); + } + + #[test] + fn backoff_opt_out_and_markers_block() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let state = |state: KvProviderState| { + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.ps.insert("p".to_owned(), state); + view_with(&entry, &fixture) + }; + assert_eq!( + skip_of(&fixture.eligibility( + "p", + &state(KvProviderState { + n: Some(NOW + 1), + ..KvProviderState::default() + }), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::Backoff, + "n in the future" + ); + assert_eq!( + skip_of(&fixture.eligibility( + "p", + &state(KvProviderState { + o: Some(KvOptOut { + at: 5, + fp: "0".repeat(16), + }), + ..KvProviderState::default() + }), + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::OptedOut, + "a sticky opt-out" + ); + let mut marked = EligibilityView::empty(); + marked.set_marker_for_test("p", NOW - 1); + assert_eq!( + skip_of(&fixture.eligibility( + "p", + &marked, + &facts_for_writer(), + Phase::AfterSend, + &after_send() + )), + SkipReason::PendingMarker, + "an unanswered marker" + ); + } + + #[test] + fn mode_rules_for_hold_after_send_and_stateful_providers() { + let stateful = ProviderDescriptor::builder(pid("s")) + .with_source(src("a.example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.example.com") + .with_private_state(), + ) + .build() + .expect("should build"); + let mut fixture = Fixture::unowned("p", &["a.example.com"]); + fixture.catalog = ProviderCatalog::new( + vec![ + fixed_provider("p", &["a.example.com"]), + Arc::new(FixedProvider(stateful)), + ], + BTreeMap::new(), + ); + let view = EligibilityView::empty(); + let hold = ProviderSettings { + mode: ProviderMode::Hold, + ..ProviderSettings::default() + }; + let auction_stager = RequestFacts { + writer: false, + runs_auction: true, + ..facts_for_writer() + }; + assert!( + matches!( + fixture.eligibility("p", &view, &auction_stager, Phase::Hold, &hold), + Eligibility::Eligible { .. } + ), + "a stateless hold provider runs on an auction stager" + ); + assert_eq!( + skip_of(&fixture.eligibility("s", &view, &auction_stager, Phase::Hold, &hold)), + SkipReason::Mode, + "a stateful hold provider never runs on a stager" + ); + assert_eq!( + skip_of(&fixture.eligibility( + "p", + &view, + &auction_stager, + Phase::AfterSend, + &after_send() + )), + SkipReason::Mode, + "after-send providers run only on the writer" + ); + let writer_without_auction = facts_for_writer(); + assert!( + matches!( + fixture.eligibility("s", &view, &writer_without_auction, Phase::AfterSend, &hold), + Eligibility::Eligible { .. } + ), + "a stateful hold provider runs after send on a writer navigation without an auction" + ); + assert_eq!( + skip_of(&fixture.eligibility("p", &view, &writer_without_auction, Phase::Hold, &hold)), + SkipReason::Mode, + "hold mode needs an auction" + ); + } + + #[test] + fn no_match_waits_retry_in_capped_at_seven_days() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let view = EligibilityView::empty(); + let default = fixture.outcome("p", CallOutcome::Resolved(Resolution::no_match()), &view); + assert_eq!(default.state.n, Some(NOW + 86_400), "24 h default"); + assert_eq!(default.state.f, Some(0), "a no_match resets f"); + let long = fixture.outcome( + "p", + CallOutcome::Resolved( + Resolution::no_match().with_retry_in(Duration::from_secs(30 * 86_400)), + ), + &view, + ); + assert_eq!(long.state.n, Some(NOW + 7 * 86_400), "capped at 7 days"); + } + + #[test] + fn found_ids_become_fill_missing_for_unowned_and_owner_write_for_owned() { + let fixture = Fixture::owned("owner", "a.example.com"); + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + rv: Some("1111111111111111".to_owned()), + ..KvPartnerId::new("seeded") + }, + ); + let changes = fixture.outcome( + "owner", + CallOutcome::Resolved(Resolution::found([ResolvedId::new( + &src("a.example.com"), + "fresh", + ) + .with_atype(3) + .with_ext(json!({ "linkType": 2 })) + .with_expires_in(Duration::from_secs(60))])), + &view_with(&entry, &fixture), + ); + assert_eq!(changes.positive.len(), 1, "one change"); + match &changes.positive[0] { + PartnerIdChange::OwnerWrite { + record, expected, .. + } => { + assert_eq!(record.uid, "fresh", "uid"); + assert_eq!(record.atype, Some(3), "atype"); + assert_eq!(record.exp, Some(NOW + 60), "relative expiry resolved"); + assert_eq!( + expected.as_ref().map(Revision::as_str), + Some("1111111111111111"), + "expects the revision it read" + ); + } + other => panic!("should be an OwnerWrite, got {other:?}"), + } + + let unowned = Fixture::unowned("p", &["a.example.com"]); + let changes = unowned.outcome( + "p", + CallOutcome::Resolved(Resolution::found([ + ResolvedId::new(&src("a.example.com"), "u"), + ResolvedId::new(&src("other.example.com"), "not-claimed"), + ResolvedId::new(&src("a.example.com"), "duplicate"), + ])), + &EligibilityView::empty(), + ); + assert_eq!( + changes.positive.len(), + 1, + "unclaimed and duplicate IDs are dropped" + ); + assert!( + matches!(&changes.positive[0], PartnerIdChange::FillMissing { writer: Some(writer), .. } if writer.as_str() == "p"), + "an unowned source gets FillMissing with the writer" + ); + } + + #[test] + fn invalidate_records_iv_hash_and_sets_n_now() { + let fixture = Fixture::owned("owner", "a.example.com"); + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + w: Some("owner".to_owned()), + rv: Some("1111111111111111".to_owned()), + ..KvPartnerId::new("stale") + }, + ); + let changes = fixture.outcome( + "owner", + CallOutcome::Resolved(Resolution::invalidate(&src("a.example.com"))), + &view_with(&entry, &fixture), + ); + assert!( + matches!(&changes.removals[0], PartnerIdChange::OwnerRemove { expected, .. } if expected.as_str() == "1111111111111111"), + "removes by the revision it read" + ); + assert_eq!( + changes.state.iv_add, + vec![invalidated_uid_hash(&fixture.keys, &pid("owner"), "stale")], + "records the rejected UID" + ); + assert_eq!( + changes.state.n, + Some(NOW), + "eligible again at once for Initial" + ); + } + + #[test] + fn opted_out_removes_everything_and_clears_state() { + let fixture = Fixture::unowned("p", &["a.example.com", "b.example.com"]); + let changes = fixture.outcome( + "p", + CallOutcome::Resolved(Resolution::opted_out()), + &EligibilityView::empty(), + ); + assert_eq!( + changes.removals.len(), + 2, + "a WriterRemove per claimed source" + ); + assert_eq!( + changes.state.opt_out, + Some(KvOptOut { + at: NOW, + fp: "0123456789abcdef".to_owned() + }), + "records the opt-out with the fingerprint" + ); + assert_eq!(changes.state.st, StateUpdate::Clear, "clears state"); + assert!(changes.positive.is_empty(), "no positive change"); + } + + #[test] + fn backoff_rows() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 5); + let transport = fixture.outcome("p", CallOutcome::TransportFailure, &view); + assert_eq!(transport.state.f, Some(6), "should count the failure"); + assert_eq!( + transport.state.n, + Some(NOW + 3_600), + "min(5 min x 2^5, 1 h) = 1 h" + ); + + let first = fixture.outcome( + "p", + CallOutcome::TransportFailure, + &EligibilityView::empty(), + ); + assert_eq!( + first.state.n, + Some(NOW + 300), + "first transport failure waits 5 min" + ); + + let input = fixture.outcome("p", CallOutcome::InputFailure, &view); + assert_eq!( + input.state.n, + Some(NOW + 86_400), + "min(1 h x 2^5, 24 h) = 24 h" + ); + let first_input = + fixture.outcome("p", CallOutcome::InputFailure, &EligibilityView::empty()); + assert_eq!( + first_input.state.n, + Some(NOW + 3_600), + "first input failure waits 1 h" + ); + + let cut = fixture.outcome("p", CallOutcome::HoldCutOff, &view); + assert_eq!(cut.state.n, Some(NOW + 300), "hold_retry_sec"); + assert_eq!(cut.state.f, None, "a cut-off is not a failure"); + + let skipped = fixture.outcome("p", CallOutcome::Skipped, &view); + assert_eq!(skipped.state.n, Some(NOW + 86_400), "refresh_after_sec"); + assert_eq!(skipped.state.f, None, "a skip is not a failure"); + + let breaker = fixture.outcome("p", CallOutcome::BreakerOpen, &view); + assert_eq!( + breaker.state, + ProviderStateChange::default(), + "breaker changes nothing" + ); + } + + #[test] + fn invalid_state_clears_state_and_keeps_f() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let mut view = EligibilityView::empty(); + view.set_failures_for_test("p", 2); + let changes = fixture.outcome("p", CallOutcome::InvalidState(None), &view); + assert_eq!(changes.state.st, StateUpdate::Clear, "clears state"); + assert_eq!(changes.state.f, None, "f unchanged"); + assert_eq!( + changes.state.n, + Some(NOW + 86_400), + "waits refresh_after_sec" + ); + } + + #[test] + fn oversized_state_rejects_positive_part_but_keeps_removals() { + let stateful = ProviderDescriptor::builder(pid("s")) + .with_source(src("a.example.com")) + .with_source(src("b.example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.example.com") + .with_private_state(), + ) + .build() + .expect("should build"); + let fixture = Fixture::new(ProviderCatalog::new( + vec![Arc::new(FixedProvider(stateful))], + [ + (src("a.example.com"), pid("s")), + (src("b.example.com"), pid("s")), + ] + .into_iter() + .collect(), + )); + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.ids.insert( + "b.example.com".to_owned(), + KvPartnerId { + w: Some("s".to_owned()), + rv: Some("2222222222222222".to_owned()), + ..KvPartnerId::new("dropped") + }, + ); + let changes = fixture.outcome( + "s", + CallOutcome::Resolved( + Resolution::found([ResolvedId::new(&src("a.example.com"), "u")]) + .also_invalidate(&src("b.example.com")) + .replacing_state(&"x".repeat(2_000)), + ), + &view_with(&entry, &fixture), + ); + assert!(changes.result.is_none(), "no positive result"); + assert!(changes.positive.is_empty(), "no positive change"); + assert_eq!(changes.removals.len(), 1, "the invalidation still applies"); + assert_eq!(changes.state.f, Some(1), "backs off as Malformed"); + } + + #[test] + fn stateful_found_becomes_one_atomic_result() { + let stateful = ProviderDescriptor::builder(pid("s")) + .with_source(src("a.example.com")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.example.com") + .with_private_state(), + ) + .build() + .expect("should build"); + let fixture = Fixture::new(ProviderCatalog::new( + vec![Arc::new(FixedProvider(stateful))], + [(src("a.example.com"), pid("s"))].into_iter().collect(), + )); + let changes = fixture.outcome( + "s", + CallOutcome::Resolved( + Resolution::found([ResolvedId::new(&src("a.example.com"), "u") + .with_refresh_until(crate::ec::identity::UnixTime::from_secs(NOW + 99))]) + .replacing_state(&json!({ "token": "t" })), + ), + &EligibilityView::empty(), + ); + let result = changes.result.expect("should build a result"); + assert_eq!(result.ids.len(), 1, "the ID is in the result"); + assert_eq!( + result.state.st, + StateUpdate::Replace(json!({ "token": "t" })), + "the state is in the result" + ); + assert!(changes.positive.is_empty(), "nothing outside the result"); + assert!( + matches!(&result.ids[0], PartnerIdChange::OwnerWrite { record, .. } if record.ru == Some(NOW + 99)), + "refresh_until is kept for a stateful provider" + ); + } + + #[test] + fn eligibility_view_applies_removal_cookies_and_staged_entries() { + let fixture = Fixture::unowned("p", &["a.example.com"]); + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.ids.insert( + "a.example.com".to_owned(), + KvPartnerId { + w: Some("p".to_owned()), + at: Some(10), + ..KvPartnerId::new("written") + }, + ); + let removals = [crate::ec::identity::cookie::RBatch { + rm: vec![crate::ec::identity::cookie::RRemoval { + s: "a.example.com".to_owned(), + w: "p".to_owned(), + pv: None, + sa: 10, + }], + o: [("p".to_owned(), 20)].into_iter().collect(), + ..crate::ec::identity::cookie::RBatch::default() + }]; + let view = EligibilityView::merge( + Some(&entry), + None, + &removals, + &fixture.catalog, + &fixture.keys, + ); + assert!( + view.record(&src("a.example.com")).is_none(), + "a staged removal hides the record" + ); + assert!(view.is_opted_out(&pid("p")), "a staged opt-out counts"); + assert!( + view.record(&src("z.example.com")).is_some(), + "control: other records stay" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 63a1ea1cc8..f72769654d 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -16,6 +16,7 @@ pub(crate) mod catalog; pub mod config; pub(crate) mod cookie; +pub(crate) mod coordinator; pub mod descriptor; pub mod exchange; pub mod ids; diff --git a/crates/trusted-server-core/src/ec/identity/policy.rs b/crates/trusted-server-core/src/ec/identity/policy.rs index 428d461a7a..66b71a5013 100644 --- a/crates/trusted-server-core/src/ec/identity/policy.rs +++ b/crates/trusted-server-core/src/ec/identity/policy.rs @@ -84,6 +84,40 @@ pub(crate) fn policy_allows( true } +/// The canonical consent signals `policy` reads, for the opt-out +/// fingerprint. Built from decoded decisions, never from the raw consent +/// string, which changes whenever a CMP re-saves it. +#[must_use] +pub(crate) fn fingerprint_bits( + policy: &ConsentPolicy, + consent: &ConsentContext, + signals: PolicySignals, +) -> String { + let tcf = effective_tcf(consent); + let mut bits: Vec = policy + .tcf_purposes() + .iter() + .map(|purpose| { + let granted = tcf.is_some_and(|tcf| tcf.has_purpose_consent(usize::from(*purpose))); + format!("p{purpose}={}", u8::from(granted)) + }) + .collect(); + if let Some(vendor) = policy.tcf_vendor() { + let granted = tcf.is_some_and(|tcf| tcf.has_vendor_consent(vendor)); + bits.push(format!("v{vendor}={}", u8::from(granted))); + } + if !policy.allows_us_opt_out(UsOptOut::Sale) { + bits.push(format!("sale={}", u8::from(us_sale_opted_out(consent)))); + } + if !policy.allows_gpc() { + bits.push(format!("gpc={}", u8::from(consent.gpc))); + } + if !policy.allows_dnt() { + bits.push(format!("dnt={}", u8::from(signals.dnt))); + } + bits.join(";") +} + #[cfg(test)] mod tests { use super::*; @@ -231,6 +265,48 @@ mod tests { ); } + #[test] + fn fingerprint_bits_follow_decisions_not_raw_strings() { + let policy = ConsentPolicy::strict().with_tcf_vendor(9999); + let mut granted = consent_with(&TestSignals { + gdpr: true, + purposes: vec![1, 4], + vendors: vec![9999], + ..TestSignals::default() + }); + let before = fingerprint_bits(&policy, &granted, NO_DNT); + assert_eq!( + before, "p1=1;p4=1;v9999=1;sale=0;gpc=0;dnt=0", + "should list each decision the policy reads" + ); + granted.raw_tc_string = Some("re-saved-by-the-cmp".to_owned()); + assert_eq!( + fingerprint_bits(&policy, &granted, NO_DNT), + before, + "a re-saved string with the same decisions keeps the bits" + ); + let denied = consent_with(&TestSignals { + gdpr: true, + purposes: vec![1], + vendors: vec![9999], + ..TestSignals::default() + }); + assert_ne!( + fingerprint_bits(&policy, &denied, NO_DNT), + before, + "dropping purpose 4 changes the bits" + ); + assert_eq!( + fingerprint_bits( + &ConsentPolicy::strict().allowing_gpc().allowing_dnt(), + &granted, + NO_DNT + ), + "p1=1;p4=1;sale=0", + "relaxed signals are left out" + ); + } + #[test] fn tcf_applies_in_unknown_jurisdiction() { let mut consent = consent_with(&TestSignals::default()); diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index 3946a16bf6..5a64db67eb 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -152,3 +152,25 @@ pub(crate) fn catalog_with_owner( .collect(), ) } + +/// Facts for a returning visitor's writer navigation with no auction and +/// every input available. +pub(crate) fn facts_for_writer() -> crate::ec::identity::coordinator::RequestFacts { + crate::ec::identity::coordinator::RequestFacts { + is_real_browser: true, + ec_persistence_allowed: true, + policy_signals: crate::ec::identity::policy::PolicySignals::default(), + writer: true, + new_visitor: false, + runs_auction: false, + now: crate::ec::identity::UnixTime::from_secs(1_000), + available_inputs: [ + crate::ec::identity::Input::ClientIp, + crate::ec::identity::Input::UserAgent, + crate::ec::identity::Input::PageUrl, + crate::ec::identity::Input::Region, + ] + .into_iter() + .collect(), + } +} diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 007c0ddae4..28604b7b31 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -95,6 +95,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/cookie.rs", include_str!("ec/identity/cookie.rs"), ), + ( + "ec/identity/coordinator.rs", + include_str!("ec/identity/coordinator.rs"), + ), ( "ec/identity/descriptor.rs", include_str!("ec/identity/descriptor.rs"), From 1de5e2c6976eeb9255b7ba0738e79589217a04bb Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:36:47 +0530 Subject: [PATCH 22/33] Add coordinator dispatch with request checks and byte caps Each call is checked for https, a declared host and no Cookie header, capped per visitor, sent with the provider's timeout, and decoded under max_response_bytes before the provider sees it. An undeclared input read fails the call and nothing is sent. Signed-off-by: dhruv8sh --- .../src/ec/identity/coordinator.rs | 5 +- .../src/ec/identity/dispatch.rs | 695 ++++++++++++++++++ .../src/ec/identity/input.rs | 27 + .../src/ec/identity/mod.rs | 1 + .../src/migration_guards.rs | 4 + 5 files changed, 731 insertions(+), 1 deletion(-) create mode 100644 crates/trusted-server-core/src/ec/identity/dispatch.rs diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index b5e835c3da..98c54a09a0 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -490,6 +490,9 @@ pub(crate) enum CallOutcome { TransportFailure, /// The circuit breaker was open. BreakerOpen, + /// Not sent for a reason the stored row does not show, such as the + /// per-visitor burst cap. Phase 4 backs a marked provider off by a minute. + NotDispatched, } /// What turning an outcome into changes needs. @@ -537,7 +540,7 @@ pub(crate) fn outcome_to_changes( let now = context.now; let mut changes = OutcomeChanges::default(); match outcome { - CallOutcome::BreakerOpen => {} + CallOutcome::BreakerOpen | CallOutcome::NotDispatched => {} CallOutcome::Skipped | CallOutcome::InputUnavailable => { changes.state.n = Some(now + context.provider_settings.refresh_after_sec); } diff --git a/crates/trusted-server-core/src/ec/identity/dispatch.rs b/crates/trusted-server-core/src/ec/identity/dispatch.rs new file mode 100644 index 0000000000..dcebbd0611 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/dispatch.rs @@ -0,0 +1,695 @@ +//! Sends provider requests: request checks, transport limits, decoding. +//! +//! Providers never do I/O. For each call this module builds the provider's +//! request, checks it (HTTPS, a declared host, no `Cookie`), applies the +//! per-visitor burst cap, sends it with the provider's timeout, buffers and +//! decodes the response under `max_response_bytes`, and hands the bytes back +//! to the provider. Provider errors are logged with their context only. + +use std::collections::BTreeMap; +use std::io::Read; +use std::time::Duration; + +use error_stack::Report; +use http::{StatusCode, header}; +use web_time::Instant; + +use super::catalog::ProviderCatalog; +use super::config::ProviderSettings; +use super::coordinator::CallOutcome; +use super::descriptor::{ProviderDescriptor, ResolveKind}; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::input::{ResolveInput, ResolveSignals, StoredId}; +use super::mutation::ExpectedState; +use super::provider::IdentityProvider; +use super::resolution::IdentityProviderError; +use super::revision::{IdentityKeys, visitor_key}; +use crate::ec::rate_limiter::RateLimiter; +use crate::platform::{ + PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, + RuntimeServices, +}; +use crate::settings::Settings; + +/// Shared dependencies of identity work, in one struct so request handlers +/// stay within their argument limit. +pub(crate) struct IdentityDeps<'a> { + pub(crate) settings: &'a Settings, + pub(crate) catalog: &'a ProviderCatalog, + pub(crate) keys: &'a IdentityKeys, + pub(crate) services: &'a RuntimeServices, + pub(crate) rate_limiter: Option<&'a dyn RateLimiter>, +} + +/// One provider call to make. +#[derive(Debug, Clone)] +pub(crate) struct Call { + pub(crate) settings: ProviderSettings, + pub(crate) provider: IdentityProviderId, + /// The sources to resolve, each with its kind. + pub(crate) kinds: BTreeMap, + /// The provider's stored records. + pub(crate) stored: BTreeMap, + /// The provider's private state JSON. + pub(crate) state_json: Option, +} + +/// How one call ended. +#[derive(Debug, Clone)] +pub(crate) struct CallReport { + pub(crate) provider: IdentityProviderId, + pub(crate) outcome: CallOutcome, + /// The state the call read, for the atomic check of its result. + pub(crate) expected_state: ExpectedState, +} + +/// One dispatch round. +pub(crate) struct DispatchRequest<'a> { + pub(crate) deps: &'a IdentityDeps<'a>, + /// The request's signals; each call gets its own stored records. + pub(crate) signals: &'a ResolveSignals<'a>, + /// Hold mode: the shared wall-clock budget from the first dispatch. + pub(crate) hold_budget: Option, + /// The full EC ID, for the per-visitor burst cap key. + pub(crate) visitor: Option<&'a str>, +} + +/// Checks a provider's request before it is sent: `https`, a host the +/// descriptor declares, and no `Cookie` header. +/// +/// # Errors +/// +/// `Config` naming the failed check. +pub(crate) fn check_request( + request: &ProviderRequest, + descriptor: &ProviderDescriptor, +) -> Result<(), Report> { + if request.url().scheme() != "https" { + return Err(IdentityProviderError::config("request is not https")); + } + let host = request.url().host_str().unwrap_or_default(); + let allowed = descriptor + .resolves() + .is_some_and(|resolves| resolves.hosts().iter().any(|known| known == host)); + if !allowed { + return Err(IdentityProviderError::config( + "request host is not declared", + )); + } + if request.has_cookie_header() { + return Err(IdentityProviderError::config( + "request carries a Cookie header", + )); + } + Ok(()) +} + +fn read_capped(reader: impl Read, cap: usize) -> Option> { + let mut out = Vec::new(); + let limit = u64::try_from(cap).unwrap_or(u64::MAX).saturating_add(1); + reader.take(limit).read_to_end(&mut out).ok()?; + (out.len() <= cap).then_some(out) +} + +/// Undoes `Content-Encoding`, capping the decoded size. `None` when over the +/// cap, corrupt, or an encoding TS does not decode. +#[must_use] +pub(crate) fn decode_body(encoding: Option<&str>, body: Vec, cap: usize) -> Option> { + let encoding = encoding.map(|value| value.trim().to_ascii_lowercase()); + match encoding.as_deref() { + None | Some("" | "identity") => (body.len() <= cap).then_some(body), + Some("gzip" | "x-gzip") => read_capped(flate2::read::MultiGzDecoder::new(&body[..]), cap), + Some("br") => read_capped(brotli::Decompressor::new(&body[..], 4096), cap), + Some(_) => None, + } +} + +/// A sent call: what finishing it needs, beside its pending request. +struct InFlight<'a> { + provider: std::sync::Arc, + call: Call, + signals: ResolveSignals<'a>, + call_context: Option, + state_failed: bool, + expected_state: ExpectedState, +} + +fn expected_state_for(provider: &dyn IdentityProvider, call: &Call) -> ExpectedState { + let stateful = provider + .descriptor() + .resolves() + .is_some_and(super::descriptor::Resolves::has_private_state); + match (stateful, &call.state_json) { + (false, _) => ExpectedState::NotRead, + (true, Some(json)) => ExpectedState::Json(json.clone()), + (true, None) => ExpectedState::Absent, + } +} + +fn log_provider_error(id: &IdentityProviderId, stage: &str, error: &Report) { + log::warn!( + "Identity provider {id} {stage} failed: {}", + error.current_context() + ); +} + +/// Calls `request` and maps its result. `Ok` holds the request and whether +/// the state failed to decode. +fn build_request( + provider: &dyn IdentityProvider, + call: &Call, + signals: &ResolveSignals<'_>, +) -> Result<(ProviderRequest, bool), CallOutcome> { + let id = provider.descriptor().id(); + let input = ResolveInput::new(provider.descriptor(), call.kinds.clone(), signals); + let result = provider.request(&input); + let undeclared = input.recorded_errors(); + if !undeclared.is_empty() { + log::warn!("Identity provider {id} read undeclared inputs: {undeclared:?}"); + return Err(CallOutcome::InputFailure); + } + match result { + Ok(Some(request)) => Ok((request, input.state_decode_failed())), + Ok(None) => Err(CallOutcome::Skipped), + Err(error) => match error.current_context() { + IdentityProviderError::InputUnavailable { .. } => Err(CallOutcome::InputUnavailable), + IdentityProviderError::InvalidState => Err(CallOutcome::InvalidState(None)), + _ => { + log_provider_error(id, "request", &error); + Err(CallOutcome::InputFailure) + } + }, + } +} + +fn burst_capped(request: &DispatchRequest<'_>, call: &Call) -> bool { + let (Some(limiter), Some(visitor)) = (request.deps.rate_limiter, request.visitor) else { + return false; + }; + let key = format!( + "identity:{}:{}", + call.provider, + visitor_key(request.deps.keys, visitor) + ); + match limiter.exceeded_per_minute(&key, call.settings.max_calls_per_minute) { + Ok(exceeded) => exceeded, + Err(error) => { + log::warn!( + "Identity provider {} burst cap check failed: {}", + call.provider, + error.current_context() + ); + true + } + } +} + +fn send( + request: &DispatchRequest<'_>, + provider_request: &ProviderRequest, + timeout_ms: u64, +) -> Option { + let services = request.deps.services; + let url = provider_request.url(); + let configured = u32::try_from(timeout_ms).unwrap_or(u32::MAX); + let transport_ms = services + .backend() + .canonicalize_transport_timeout_ms(configured, configured); + let timeout = Duration::from_millis(u64::from(transport_ms.max(1))); + let backend_name = services + .backend() + .ensure(&PlatformBackendSpec { + scheme: url.scheme().to_owned(), + host: url.host_str().unwrap_or_default().to_owned(), + port: url.port(), + host_header_override: None, + certificate_check: request.deps.settings.proxy.certificate_check, + first_byte_timeout: timeout, + between_bytes_timeout: timeout, + discriminator: None, + }) + .ok()?; + + let mut builder = http::Request::builder() + .method(provider_request.method().clone()) + .uri(url.as_str()); + for (name, value) in provider_request.headers() { + builder = builder.header(name, value); + } + let outbound = builder + .body(edgezero_core::body::Body::from( + provider_request.body().to_vec(), + )) + .ok()?; + futures::executor::block_on( + services + .http_client() + .send_async(PlatformHttpRequest::new(outbound, backend_name)), + ) + .ok() +} + +fn content_length_exceeds(response: &PlatformResponse, cap: usize) -> bool { + response + .response + .headers() + .get(header::CONTENT_LENGTH) + .is_some_and(|value| { + value + .to_str() + .ok() + .and_then(|value| value.parse::().ok()) + .is_none_or(|length| length > cap) + }) +} + +fn finish(item: &InFlight<'_>, response: PlatformResponse, cap: usize) -> CallOutcome { + let descriptor = item.provider.descriptor(); + let id = descriptor.id(); + if content_length_exceeds(&response, cap) { + log::warn!("Identity provider {id} declared an oversized response"); + return CallOutcome::TransportFailure; + } + let status = response.response.status(); + let headers = response.response.headers().clone(); + let raw = response + .response + .into_body() + .into_bytes() + .unwrap_or_default() + .to_vec(); + if raw.len() > cap { + log::warn!("Identity provider {id} returned an oversized response"); + return CallOutcome::TransportFailure; + } + let encoding = headers + .get(header::CONTENT_ENCODING) + .and_then(|value| value.to_str().ok()); + let Some(body) = decode_body(encoding, raw, cap) else { + log::warn!("Identity provider {id} response could not be decoded under the cap"); + return CallOutcome::TransportFailure; + }; + + let provider_response = ProviderResponse::new(status, headers, body, item.call_context.clone()); + let input = ResolveInput::new(descriptor, item.call.kinds.clone(), &item.signals); + let result = item.provider.response(&provider_response, &input); + let undeclared = input.recorded_errors(); + if !undeclared.is_empty() { + log::warn!("Identity provider {id} read undeclared inputs: {undeclared:?}"); + return CallOutcome::InputFailure; + } + match result { + Ok(resolution) if item.state_failed || input.state_decode_failed() => { + CallOutcome::InvalidState(Some(resolution)) + } + Ok(resolution) => CallOutcome::Resolved(resolution), + Err(error) => { + log_provider_error(id, "response", &error); + match error.current_context() { + IdentityProviderError::Rejected { status } + if *status == StatusCode::TOO_MANY_REQUESTS || status.is_server_error() => + { + CallOutcome::TransportFailure + } + IdentityProviderError::InvalidState => CallOutcome::InvalidState(None), + _ => CallOutcome::InputFailure, + } + } + } +} + +fn drain( + request: &DispatchRequest<'_>, + in_flight: &mut Vec<(PlatformPendingRequest, InFlight<'_>)>, + started: Instant, + reports: &mut Vec, +) { + let cap = request.deps.settings.ec.identity.max_response_bytes; + for (pending, item) in in_flight.drain(..) { + let result = futures::executor::block_on(request.deps.services.http_client().wait(pending)); + let elapsed = started.elapsed(); + let id = item.call.provider.clone(); + log::debug!( + "Identity provider {id} answered after {} ms", + elapsed.as_millis() + ); + let outcome = match result { + Ok(response) => finish(&item, response, cap), + Err(_) if request.hold_budget.is_some_and(|budget| elapsed >= budget) => { + CallOutcome::HoldCutOff + } + Err(error) => { + log::warn!( + "Identity provider {id} transport failed: {}", + error.current_context() + ); + CallOutcome::TransportFailure + } + }; + reports.push(CallReport { + provider: id, + outcome, + expected_state: item.expected_state, + }); + } +} + +/// Builds, checks, sends and finishes every call, at most +/// `max_concurrency` at once. +#[must_use] +pub(crate) fn dispatch(request: &DispatchRequest<'_>, calls: Vec) -> Vec { + let settings = request.deps.settings; + let max_concurrency = settings + .ec + .identity + .effective_max_concurrency(settings.ec.pull_sync_concurrency) + .max(1); + let started = Instant::now(); + let mut reports = Vec::new(); + let mut in_flight: Vec<(PlatformPendingRequest, InFlight<'_>)> = Vec::new(); + + for call in calls { + let Some(provider) = request.deps.catalog.get(&call.provider).cloned() else { + continue; + }; + let expected_state = expected_state_for(provider.as_ref(), &call); + let report = |outcome| CallReport { + provider: call.provider.clone(), + outcome, + expected_state: expected_state.clone(), + }; + let signals = + request + .signals + .for_provider(call.stored.clone(), call.state_json.clone(), false); + let (provider_request, state_failed) = + match build_request(provider.as_ref(), &call, &signals) { + Ok(built) => built, + Err(outcome) => { + reports.push(report(outcome)); + continue; + } + }; + if let Err(error) = check_request(&provider_request, provider.descriptor()) { + log_provider_error(&call.provider, "request check", &error); + reports.push(report(CallOutcome::InputFailure)); + continue; + } + if burst_capped(request, &call) { + log::debug!("Identity provider {} hit its burst cap", call.provider); + reports.push(report(CallOutcome::NotDispatched)); + continue; + } + + let mut timeout_ms = call.settings.effective_timeout_ms(&settings.ec.identity); + if let Some(budget) = request.hold_budget { + let remaining = budget.saturating_sub(started.elapsed()); + if remaining.is_zero() { + reports.push(report(CallOutcome::HoldCutOff)); + continue; + } + timeout_ms = timeout_ms.min(u64::try_from(remaining.as_millis()).unwrap_or(u64::MAX)); + } + + let Some(pending) = send(request, &provider_request, timeout_ms) else { + log::warn!("Identity provider {} could not be sent", call.provider); + reports.push(report(CallOutcome::TransportFailure)); + continue; + }; + in_flight.push(( + pending, + InFlight { + provider, + call_context: provider_request.call_context_json().map(str::to_owned), + state_failed, + expected_state, + signals, + call, + }, + )); + if in_flight.len() >= max_concurrency { + drain(request, &mut in_flight, started, &mut reports); + } + } + drain(request, &mut in_flight, started, &mut reports); + reports +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + use std::io::Write as _; + use std::sync::Arc; + + use error_stack::Report; + use serde::Deserialize; + use url::Url; + + use super::*; + use crate::ec::identity::descriptor::{Input, ProviderDescriptor, Resolves}; + use crate::ec::identity::exchange::{ProviderRequest, ProviderResponse}; + use crate::ec::identity::ids::UnixTime; + use crate::ec::identity::input::{ProviderConsent, ResolveInput, ThreadRandom}; + use crate::ec::identity::resolution::{IdentityProviderError, Resolution, ResolvedId}; + use crate::ec::identity::test_support::{fixed_provider, test_keys}; + use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; + use crate::test_support::tests::create_test_settings; + + struct TestPull { + descriptor: ProviderDescriptor, + read_user_agent: bool, + } + + impl TestPull { + fn new(read_user_agent: bool) -> Self { + Self { + descriptor: ProviderDescriptor::builder("tp".parse().expect("should parse id")) + .with_source("a.example.com".parse().expect("should parse source")) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.example.com/v1/fetch") + .with_needs([Input::ClientIp]), + ) + .build() + .expect("should build"), + read_user_agent, + } + } + } + + #[derive(Deserialize)] + struct Reply { + uid: String, + } + + impl IdentityProvider for TestPull { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + if self.read_user_agent { + let _ignored = input.user_agent().ok(); + } + let _ip = input.client_ip()?; + Ok(Some(ProviderRequest::get(input.endpoint()?))) + } + + fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, + ) -> Result> { + let reply: Reply = response.success_json()?; + Ok(Resolution::found([ResolvedId::new( + input.primary_source(), + reply.uid, + )])) + } + } + + fn signals() -> ResolveSignals<'static> { + ResolveSignals { + client_ip: Some("192.0.2.10".parse().expect("should parse ip")), + user_agent: Some("ExampleBrowser/1.0"), + page_url: None, + region: None, + hashed_email: None, + consent: ProviderConsent::default(), + publisher_domain: "www.example.com", + now: UnixTime::from_secs(1_000), + random: &ThreadRandom, + ec_id: None, + stored: BTreeMap::new(), + state_json: None, + dry_run: false, + } + } + + fn call() -> Call { + Call { + settings: ProviderSettings::default(), + provider: "tp".parse().expect("should parse id"), + kinds: [( + "a.example.com".parse().expect("should parse source"), + ResolveKind::Initial, + )] + .into_iter() + .collect(), + stored: BTreeMap::new(), + state_json: None, + } + } + + fn run(provider: TestPull, stub: &Arc) -> CallOutcome { + let settings = create_test_settings(); + let catalog = ProviderCatalog::new(vec![Arc::new(provider)], BTreeMap::new()); + let keys = test_keys(); + let services = build_services_with_http_client( + Arc::clone(stub) as Arc + ); + let deps = IdentityDeps { + settings: &settings, + catalog: &catalog, + keys: &keys, + services: &services, + rate_limiter: None, + }; + let signals = signals(); + let mut reports = dispatch( + &DispatchRequest { + deps: &deps, + signals: &signals, + hold_budget: None, + visitor: None, + }, + vec![call()], + ); + assert_eq!(reports.len(), 1, "one report per call"); + reports.remove(0).outcome + } + + #[test] + fn check_request_enforces_https_host_and_no_cookie() { + let provider = fixed_provider("p", &["a.example.com"]); + let descriptor = provider.descriptor(); + let good = ProviderRequest::get(&Url::parse("https://api.example.com/x").expect("url")); + check_request(&good, descriptor).expect("control: allowed"); + for (name, request) in [ + ( + "http", + ProviderRequest::get(&Url::parse("http://api.example.com/x").expect("url")), + ), + ( + "host", + ProviderRequest::get(&Url::parse("https://evil.example.net/x").expect("url")), + ), + ( + "cookie", + good.clone() + .try_with_header("cookie", "a=1") + .expect("should build header"), + ), + ] { + assert!( + check_request(&request, descriptor).is_err(), + "should reject: {name}" + ); + } + } + + #[test] + fn decode_body_caps_decoded_size() { + let mut gzip = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + gzip.write_all(&vec![b'a'; 10_000]).expect("should write"); + let gzipped = gzip.finish().expect("should finish"); + assert_eq!( + decode_body(Some("gzip"), gzipped.clone(), 10_000).map(|body| body.len()), + Some(10_000), + "fits" + ); + assert!( + decode_body(Some("gzip"), gzipped, 9_999).is_none(), + "over the cap after decoding" + ); + + let mut brotli = brotli::CompressorWriter::new(Vec::new(), 4096, 5, 22); + brotli.write_all(b"hello").expect("should write"); + let compressed = brotli.into_inner(); + assert_eq!( + decode_body(Some("br"), compressed, 10).as_deref(), + Some(&b"hello"[..]), + "brotli" + ); + assert!( + decode_body(Some("zstd"), vec![1], 10).is_none(), + "unknown encoding" + ); + assert_eq!( + decode_body(None, b"ok".to_vec(), 10).as_deref(), + Some(&b"ok"[..]), + "identity" + ); + assert!( + decode_body(None, vec![0; 11], 10).is_none(), + "identity over the cap" + ); + } + + #[test] + fn dispatch_maps_status_codes_to_outcomes() { + let ok = Arc::new(StubHttpClient::new()); + ok.push_response(200, br#"{"uid":"u"}"#.to_vec()); + assert!( + matches!(run(TestPull::new(false), &ok), CallOutcome::Resolved(resolution) if resolution.ids()[0].uid() == "u"), + "200 resolves" + ); + assert_eq!(ok.recorded_request_uris().len(), 1, "one request sent"); + + for (status, expected) in [(503, "transport"), (429, "transport"), (400, "input")] { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(status, br#"{}"#.to_vec()); + let outcome = run(TestPull::new(false), &stub); + match expected { + "transport" => assert_eq!(outcome, CallOutcome::TransportFailure, "{status}"), + _ => assert_eq!(outcome, CallOutcome::InputFailure, "{status}"), + } + } + + let oversized = Arc::new(StubHttpClient::new()); + oversized.push_response(200, vec![b' '; 70_000]); + assert_eq!( + run(TestPull::new(false), &oversized), + CallOutcome::TransportFailure, + "a body over max_response_bytes is a transport failure" + ); + + let failed = Arc::new(StubHttpClient::new()); + failed.push_response(200, br#"{"uid":"u"}"#.to_vec()); + failed.push_wait_error(); + assert_eq!( + run(TestPull::new(false), &failed), + CallOutcome::TransportFailure, + "a transport error" + ); + } + + #[test] + fn undeclared_read_is_an_input_failure_and_nothing_is_sent() { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, br#"{"uid":"u"}"#.to_vec()); + assert_eq!( + run(TestPull::new(true), &stub), + CallOutcome::InputFailure, + "an undeclared read even through .ok() fails the call" + ); + assert!( + stub.recorded_request_uris().is_empty(), + "no request is sent" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/input.rs b/crates/trusted-server-core/src/ec/identity/input.rs index 2899a99979..b9aeae367c 100644 --- a/crates/trusted-server-core/src/ec/identity/input.rs +++ b/crates/trusted-server-core/src/ec/identity/input.rs @@ -332,6 +332,33 @@ pub(crate) struct ResolveSignals<'a> { pub(crate) dry_run: bool, } +impl<'a> ResolveSignals<'a> { + /// The signals for one provider: its own stored records and state. + #[must_use] + pub(crate) fn for_provider( + &self, + stored: BTreeMap, + state_json: Option, + dry_run: bool, + ) -> ResolveSignals<'a> { + ResolveSignals { + client_ip: self.client_ip, + user_agent: self.user_agent, + page_url: self.page_url.clone(), + region: self.region.clone(), + hashed_email: self.hashed_email.clone(), + consent: self.consent.clone(), + publisher_domain: self.publisher_domain, + now: self.now, + random: self.random, + ec_id: self.ec_id.clone(), + stored, + state_json, + dry_run, + } + } +} + /// What one provider may read for one call. /// /// Every accessor checks the descriptor. Reading something undeclared fails diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index f72769654d..e63902d5dc 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -18,6 +18,7 @@ pub mod config; pub(crate) mod cookie; pub(crate) mod coordinator; pub mod descriptor; +pub(crate) mod dispatch; pub mod exchange; pub mod ids; pub mod input; diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 28604b7b31..49143ecddf 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -103,6 +103,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/descriptor.rs", include_str!("ec/identity/descriptor.rs"), ), + ( + "ec/identity/dispatch.rs", + include_str!("ec/identity/dispatch.rs"), + ), ( "ec/identity/exchange.rs", include_str!("ec/identity/exchange.rs"), From b43af79586b8087153e7663a2bdda17d29b7cf4f Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:41:05 +0530 Subject: [PATCH 23/33] Model pull sync partners as GenericPullProvider Each pull-enabled partner becomes a provider with an unowned source, its allowed domains as hosts, and the parity consent policy. Its legacy hourly rate limit maps to the per-minute cap the limiter already enforces. Adds the basic provider harness and a KV test store that counts writes by mode. Signed-off-by: dhruv8sh --- .../src/ec/identity/catalog.rs | 69 +++ .../src/ec/identity/generic_pull.rs | 396 ++++++++++++++++++ .../src/ec/identity/mod.rs | 1 + .../src/ec/identity/test_support.rs | 199 +++++++++ crates/trusted-server-core/src/ec/kv.rs | 41 ++ .../trusted-server-core/src/ec/kv_backend.rs | 123 ++++++ .../trusted-server-core/src/ec/pull_sync.rs | 4 +- .../src/migration_guards.rs | 4 + 8 files changed, 835 insertions(+), 2 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity/generic_pull.rs diff --git a/crates/trusted-server-core/src/ec/identity/catalog.rs b/crates/trusted-server-core/src/ec/identity/catalog.rs index deffe97725..f74b636701 100644 --- a/crates/trusted-server-core/src/ec/identity/catalog.rs +++ b/crates/trusted-server-core/src/ec/identity/catalog.rs @@ -3,10 +3,16 @@ use std::collections::{BTreeMap, BTreeSet}; use std::sync::Arc; +use error_stack::Report; use sha2::{Digest as _, Sha256}; +use super::config::{IdentitySettings, ProviderSettings}; +use super::generic_pull::{GenericPullProvider, generic_pull_settings}; use super::ids::{IdentityProviderId, SourceDomain}; use super::provider::IdentityProvider; +use crate::ec::kv_types::MAX_PROVIDER_STATES; +use crate::ec::registry::PartnerRegistry; +use crate::error::TrustedServerError; /// Enabled providers, source owners, and the derived source index and `cf`. /// Built once per settings load. @@ -16,6 +22,9 @@ pub(crate) struct ProviderCatalog { owners: BTreeMap, sources: Vec, fingerprint: String, + /// Provider defaults that an operator table replaces, such as a pull + /// partner's legacy rate limit. + bases: BTreeMap, } impl core::fmt::Debug for ProviderCatalog { @@ -70,9 +79,69 @@ impl ProviderCatalog { owners, sources, fingerprint, + bases: BTreeMap::new(), } } + /// The catalog for the partner registry: one generic pull provider per + /// pull-enabled partner. A partner that cannot form a provider is logged + /// and left out. + /// + /// # Errors + /// + /// [`TrustedServerError::Configuration`] for two providers with one ID or + /// more than 32 enabled providers. + pub(crate) fn from_partners( + registry: &PartnerRegistry, + ) -> Result> { + let mut providers: Vec> = Vec::new(); + let mut bases = BTreeMap::new(); + let mut seen = BTreeSet::new(); + for partner in registry.pull_enabled_partners() { + let provider = match GenericPullProvider::from_partner(partner) { + Ok(provider) => provider, + Err(error) => { + log::warn!( + "Pull partner {} cannot run as an identity provider: {}", + partner.source_domain, + error.current_context() + ); + continue; + } + }; + let id = provider.descriptor().id().clone(); + if !seen.insert(id.clone()) { + return Err(Report::new(TrustedServerError::Configuration { + message: format!("two identity providers share the id `{id}`"), + })); + } + bases.insert(id, generic_pull_settings(partner)); + providers.push(Arc::new(provider)); + } + if providers.len() > MAX_PROVIDER_STATES { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "{} identity providers are enabled; at most {MAX_PROVIDER_STATES} are allowed", + providers.len() + ), + })); + } + let mut catalog = Self::new(providers, BTreeMap::new()); + catalog.bases = bases; + Ok(catalog) + } + + /// The operator settings for `id`: its `[ec.identity.providers.]` + /// table, else the provider's own defaults, else the global defaults. + #[must_use] + pub(crate) fn provider_settings( + &self, + identity: &IdentitySettings, + id: &IdentityProviderId, + ) -> ProviderSettings { + identity.provider(id, self.bases.get(id).cloned()) + } + /// A provider by ID. #[must_use] pub(crate) fn get(&self, id: &IdentityProviderId) -> Option<&Arc> { diff --git a/crates/trusted-server-core/src/ec/identity/generic_pull.rs b/crates/trusted-server-core/src/ec/identity/generic_pull.rs new file mode 100644 index 0000000000..672f5dc738 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/generic_pull.rs @@ -0,0 +1,396 @@ +//! `GenericPullProvider`: today's pull sync partners on the provider trait. +//! +//! The protocol is unchanged: `GET ?ec_id=` with the +//! partner's bearer token, answered with `{"uid": "..."}`. Its sources are +//! unowned, so it only fills missing IDs, as pull sync always has. + +use error_stack::Report; +use http::StatusCode; +use serde::Deserialize; +use sha2::{Digest as _, Sha256}; + +use super::config::{ProviderMode, ProviderSettings}; +use super::descriptor::{ConsentPolicy, ProviderDescriptor, Resolves}; +use super::exchange::{ProviderRequest, ProviderResponse}; +use super::ids::IdentityProviderId; +use super::input::ResolveInput; +use super::provider::IdentityProvider; +use super::resolution::{IdentityProviderError, Resolution, ResolvedId}; +use crate::ec::kv_types::MAX_UID_LENGTH; +use crate::ec::pull_sync::{build_pull_request_url, validated_pull_sync_url}; +use crate::ec::registry::PartnerConfig; +use crate::redacted::Redacted; + +/// Longest provider ID. +const MAX_ID_LENGTH: usize = 63; +/// Hex digits of the hash suffix a cut ID gets. +const ID_HASH_SUFFIX_LENGTH: usize = 8; + +/// `pull-` plus the source domain with dots replaced by `-`. An ID that +/// would exceed 63 characters is cut and given an 8-digit hash suffix of the +/// full domain, so IDs stay unique and valid. +#[must_use] +pub(crate) fn generic_pull_id(source_domain: &str) -> IdentityProviderId { + let full = format!("pull-{}", source_domain.replace('.', "-")); + let id = if full.len() <= MAX_ID_LENGTH { + full + } else { + let digest = Sha256::digest(source_domain.as_bytes()); + let suffix = &hex::encode(digest)[..ID_HASH_SUFFIX_LENGTH]; + let keep = MAX_ID_LENGTH - ID_HASH_SUFFIX_LENGTH - 1; + format!("{}-{suffix}", full[..keep].trim_end_matches('-')) + }; + id.parse() + .expect("should derive a valid provider id from a normalized source domain") +} + +/// The per-minute burst cap the limiter enforces for an hourly legacy limit: +/// `max(1, ceil(limit / 60))`. +#[must_use] +pub(crate) fn legacy_rate_limit_per_minute(pull_sync_rate_limit: u32) -> u32 { + pull_sync_rate_limit.div_ceil(60).max(1) +} + +/// A pull partner's settings when no `[ec.identity.providers.]` table +/// overrides them. +#[must_use] +pub(crate) fn generic_pull_settings(partner: &PartnerConfig) -> ProviderSettings { + ProviderSettings { + mode: ProviderMode::AfterSend, + max_calls_per_minute: legacy_rate_limit_per_minute(partner.pull_sync_rate_limit), + refresh_after_sec: partner.pull_sync_ttl_sec.max(1), + ..ProviderSettings::default() + } +} + +/// One `[[ec.partners]]` entry with `pull_sync_enabled = true`. +pub(crate) struct GenericPullProvider { + descriptor: ProviderDescriptor, + partner: PartnerConfig, + token: Option>, +} + +#[derive(Deserialize)] +struct PullReply { + uid: Option, +} + +impl GenericPullProvider { + /// Builds the provider from a partner. + /// + /// # Errors + /// + /// `Config` when the partner's source or allowed domains cannot form a + /// valid descriptor. + pub(crate) fn from_partner( + partner: &PartnerConfig, + ) -> Result> { + let mut resolves = Resolves::new().with_ec_id(); + for host in &partner.pull_sync_allowed_domains { + resolves = resolves.with_host(&host.trim().trim_end_matches('.').to_ascii_lowercase()); + } + let descriptor = ProviderDescriptor::builder(generic_pull_id(&partner.source_domain)) + .with_source(partner.source_domain.parse()?) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::eid_persistence_only()) + .build()?; + Ok(Self { + descriptor, + partner: partner.clone(), + token: partner.ts_pull_token.clone(), + }) + } +} + +impl IdentityProvider for GenericPullProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + let (Some(url), Some(token)) = + (validated_pull_sync_url(&self.partner), self.token.as_ref()) + else { + return Ok(None); + }; + let url = build_pull_request_url(url, input.ec_id()?); + Ok(Some(ProviderRequest::get(&url).with_bearer(token))) + } + + fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, + ) -> Result> { + if response.status() == StatusCode::NOT_FOUND { + return Ok(Resolution::no_match()); + } + let reply: PullReply = response.success_json()?; + let uid = reply + .uid + .filter(|uid| !uid.trim().is_empty() && uid.len() <= MAX_UID_LENGTH); + Ok(match uid { + Some(uid) => Resolution::found([ResolvedId::new(input.primary_source(), uid)]), + None => Resolution::no_match(), + }) + } +} + +#[cfg(test)] +mod tests { + use http::StatusCode; + use serde_json::json; + + use super::*; + use crate::ec::identity::catalog::ProviderCatalog; + use crate::ec::identity::resolution::IdentityProviderError; + use crate::ec::identity::test_support::ProviderHarness; + use crate::ec::registry::PartnerRegistry; + use crate::redacted::Redacted; + use crate::settings::EcPartner; + + const TOKEN: &str = "example-token-value-0123456789abcdef"; + + fn ec_id() -> String { + format!("{}.ABC123", "a".repeat(64)) + } + + fn partner(source: &str, url: &str, allowed: &[&str]) -> EcPartner { + EcPartner { + name: format!("Partner {source}"), + source_domain: source.to_owned(), + openrtb_atype: EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: None, + batch_rate_limit: EcPartner::default_batch_rate_limit(), + pull_sync_enabled: true, + pull_sync_url: Some(url.to_owned()), + pull_sync_allowed_domains: allowed.iter().map(|domain| (*domain).to_owned()).collect(), + pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: 65, + ts_pull_token: Some(Redacted::new(TOKEN.to_owned())), + } + } + + fn provider_for(partner_config: &EcPartner) -> GenericPullProvider { + let registry = PartnerRegistry::from_config(std::slice::from_ref(partner_config)) + .expect("should build registry"); + GenericPullProvider::from_partner( + registry + .get(&partner_config.source_domain) + .expect("should hold the partner"), + ) + .expect("should build the provider") + } + + fn standard() -> GenericPullProvider { + provider_for(&partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + &["sync.ssp.example.com"], + )) + } + + #[test] + fn calls_the_partner_url_with_the_ec_id_and_bearer() { + let provider = standard(); + let mut harness = ProviderHarness::new(&provider).with_ec_id(&ec_id()); + + let request = harness + .request() + .expect("should build") + .expect("should not skip"); + + assert_eq!(request.method(), http::Method::GET, "GET"); + assert_eq!( + request.url().host_str(), + Some("sync.ssp.example.com"), + "should call the partner host" + ); + assert_eq!( + request + .url() + .query_pairs() + .find(|(name, _)| name == "ec_id") + .map(|(_, value)| value.into_owned()), + Some(ec_id()), + "should pass the EC ID" + ); + assert_eq!( + request.headers()["authorization"], + format!("Bearer {TOKEN}").as_str(), + "should send the bearer token" + ); + + let resolution = harness + .respond(StatusCode::OK, &json!({ "uid": "partner-uid" })) + .expect("should parse"); + assert_eq!(resolution.ids().len(), 1, "one ID"); + assert_eq!(resolution.ids()[0].uid(), "partner-uid", "uid"); + assert_eq!( + resolution.ids()[0].source().as_str(), + "ssp.example.com", + "for the partner source" + ); + } + + #[test] + fn response_mapping_matches_pull_sync() { + let provider = standard(); + let mut harness = ProviderHarness::new(&provider).with_ec_id(&ec_id()); + harness.request().expect("should build"); + + assert!( + harness + .respond_raw(StatusCode::NOT_FOUND, &[], b"") + .expect("404 is not an error") + .is_no_match(), + "404 is no_match" + ); + for body in [json!({ "uid": null }), json!({ "uid": " " }), json!({})] { + assert!( + harness + .respond(StatusCode::OK, &body) + .expect("should parse") + .is_no_match(), + "a null, blank or missing uid is no_match: {body}" + ); + } + assert!( + harness + .respond(StatusCode::OK, &json!({ "uid": "x".repeat(513) })) + .expect("should parse") + .is_no_match(), + "an oversized uid is no_match" + ); + let error = harness + .respond(StatusCode::INTERNAL_SERVER_ERROR, &json!({})) + .expect_err("should reject"); + assert!( + matches!(error.current_context(), IdentityProviderError::Rejected { status } if *status == StatusCode::INTERNAL_SERVER_ERROR), + "5xx is Rejected" + ); + let error = harness + .respond_raw(StatusCode::OK, &[], b"not json") + .expect_err("should reject"); + assert!( + matches!(error.current_context(), IdentityProviderError::Malformed), + "bad JSON is Malformed" + ); + } + + #[test] + fn descriptor_matches_the_partner() { + let provider = standard(); + let descriptor = provider.descriptor(); + assert_eq!(descriptor.id().as_str(), "pull-ssp-example-com", "id"); + assert_eq!( + descriptor.primary_source().as_str(), + "ssp.example.com", + "one source" + ); + assert_eq!( + descriptor + .resolves() + .map(|resolves| resolves.hosts().to_vec()), + Some(vec!["sync.ssp.example.com".to_owned()]), + "allowed hosts" + ); + assert!( + descriptor.consent_policy().is_eid_persistence_only(), + "blocks exactly what pull sync blocks" + ); + assert!( + !descriptor.is_expiring(descriptor.primary_source()), + "IDs do not expire" + ); + } + + #[test] + fn provider_ids_are_derived_and_bounded() { + assert_eq!( + generic_pull_id("ssp.example.com").as_str(), + "pull-ssp-example-com", + "simple" + ); + let long = format!("{}.example.com", "a".repeat(60)); + let id = generic_pull_id(&long); + assert!(id.as_str().len() <= 63, "bounded"); + assert_ne!( + id, + generic_pull_id(&format!("{}.example.com", "a".repeat(61))), + "unique by hash" + ); + } + + #[test] + fn legacy_hourly_limit_maps_to_per_minute() { + assert_eq!(legacy_rate_limit_per_minute(0), 1, "at least one"); + assert_eq!(legacy_rate_limit_per_minute(59), 1, "rounds up"); + assert_eq!( + legacy_rate_limit_per_minute(65), + 2, + "65/hr is 2/min, as the limiter does today" + ); + let base = generic_pull_settings( + PartnerRegistry::from_config(&[partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + &["sync.ssp.example.com"], + )]) + .expect("should build registry") + .get("ssp.example.com") + .expect("should hold the partner"), + ); + assert_eq!( + base.max_calls_per_minute, 2, + "base settings carry the legacy limit" + ); + assert_eq!( + base.mode, + crate::ec::identity::config::ProviderMode::AfterSend, + "after send" + ); + } + + #[test] + fn catalog_holds_only_pull_enabled_partners() { + let settings = crate::test_support::tests::create_test_settings(); + let mut no_pull = partner("dsp.example.com", "https://sync.dsp.example.com/pull", &[]); + no_pull.pull_sync_enabled = false; + no_pull.pull_sync_url = None; + no_pull.ts_pull_token = None; + let partners = vec![ + partner( + "ssp.example.com", + "https://sync.ssp.example.com/pull", + &["sync.ssp.example.com"], + ), + no_pull, + ]; + let registry = PartnerRegistry::from_config(&partners).expect("should build"); + let catalog = ProviderCatalog::from_partners(®istry).expect("should build"); + let ids: Vec<_> = catalog + .providers() + .map(|provider| provider.descriptor().id().to_string()) + .collect(); + assert_eq!( + ids, + vec!["pull-ssp-example-com".to_owned()], + "pull partners only" + ); + assert_eq!( + catalog + .provider_settings( + &settings.ec.identity, + &"pull-ssp-example-com".parse().expect("should parse id") + ) + .max_calls_per_minute, + 2, + "base settings without an operator table" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index e63902d5dc..d23bcd187b 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -20,6 +20,7 @@ pub(crate) mod coordinator; pub mod descriptor; pub(crate) mod dispatch; pub mod exchange; +pub(crate) mod generic_pull; pub mod ids; pub mod input; pub(crate) mod mutation; diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index 5a64db67eb..d1d7c7f65c 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -174,3 +174,202 @@ pub(crate) fn facts_for_writer() -> crate::ec::identity::coordinator::RequestFac .collect(), } } + +/// Drives one provider the way the coordinator does, for provider tests. +pub(crate) struct ProviderHarness<'p> { + provider: &'p dyn crate::ec::identity::IdentityProvider, + client_ip: Option, + user_agent: Option, + ec_id: Option, + stored: std::collections::BTreeMap< + crate::ec::identity::SourceDomain, + crate::ec::identity::StoredId, + >, + state_json: Option, + now: crate::ec::identity::UnixTime, + random: crate::ec::identity::input::SeededRandom, + last_request: Option, +} + +impl<'p> ProviderHarness<'p> { + /// A harness with no inputs, at a fixed test time. + pub(crate) fn new(provider: &'p dyn crate::ec::identity::IdentityProvider) -> Self { + Self { + provider, + client_ip: None, + user_agent: None, + ec_id: None, + stored: std::collections::BTreeMap::new(), + state_json: None, + now: crate::ec::identity::UnixTime::from_secs(1_760_000_000), + random: crate::ec::identity::input::SeededRandom::new(1), + last_request: None, + } + } + + /// Sets the client IP. + #[must_use] + pub(crate) fn with_client_ip(mut self, ip: &str) -> Self { + self.client_ip = Some(ip.parse().expect("should parse the client IP")); + self + } + + /// Sets the user agent. + #[must_use] + pub(crate) fn with_user_agent(mut self, user_agent: &str) -> Self { + self.user_agent = Some(user_agent.to_owned()); + self + } + + /// Sets the EC ID, which only generic pull may read. + #[must_use] + pub(crate) fn with_ec_id(mut self, ec_id: &str) -> Self { + self.ec_id = Some(ec_id.to_owned()); + self + } + + /// Stores a record for `source`, which makes its kind `Refresh`. + #[must_use] + pub(crate) fn with_stored( + mut self, + source: &str, + record: &crate::ec::kv_types::KvPartnerId, + ) -> Self { + self.stored.insert( + source.parse().expect("should parse source"), + crate::ec::identity::StoredId::from_record(record), + ); + self + } + + fn available_inputs(&self) -> std::collections::BTreeSet { + let mut available = std::collections::BTreeSet::new(); + if self.client_ip.is_some() { + available.insert(crate::ec::identity::Input::ClientIp); + } + if self.user_agent.is_some() { + available.insert(crate::ec::identity::Input::UserAgent); + } + available + } + + fn kinds( + &self, + ) -> std::collections::BTreeMap< + crate::ec::identity::SourceDomain, + crate::ec::identity::ResolveKind, + > { + let descriptor = self.provider.descriptor(); + let Some(resolves) = descriptor.resolves() else { + return std::collections::BTreeMap::new(); + }; + let available = self.available_inputs(); + descriptor + .sources() + .iter() + .filter_map(|source| { + let kind = if self.stored.contains_key(source) { + crate::ec::identity::ResolveKind::Refresh + } else { + crate::ec::identity::ResolveKind::Initial + }; + let needs_met = resolves + .needs(kind) + .iter() + .all(|input| available.contains(input)); + (resolves.kinds().contains(&kind) && needs_met).then(|| (source.clone(), kind)) + }) + .collect() + } + + fn signals(&self) -> crate::ec::identity::input::ResolveSignals<'_> { + crate::ec::identity::input::ResolveSignals { + client_ip: self.client_ip, + user_agent: self.user_agent.as_deref(), + page_url: None, + region: None, + hashed_email: None, + consent: crate::ec::identity::ProviderConsent::default(), + publisher_domain: "www.example.com", + now: self.now, + random: &self.random, + ec_id: self.ec_id.clone(), + stored: self.stored.clone(), + state_json: self.state_json.clone(), + dry_run: false, + } + } + + /// Calls `request` as the coordinator would. With no source left after + /// the needs check it returns `Ok(None)` without calling the provider. + pub(crate) fn request( + &mut self, + ) -> Result< + Option, + error_stack::Report, + > { + let kinds = self.kinds(); + if kinds.is_empty() { + return Ok(None); + } + let result = { + let signals = self.signals(); + let input = + crate::ec::identity::ResolveInput::new(self.provider.descriptor(), kinds, &signals); + let result = self.provider.request(&input); + assert!( + input.recorded_errors().is_empty(), + "provider read undeclared inputs: {:?}", + input.recorded_errors() + ); + result + }; + self.last_request = result.as_ref().ok().cloned().flatten(); + result + } + + /// Calls `response` with a JSON body and the kept call context. + pub(crate) fn respond( + &mut self, + status: http::StatusCode, + body: &serde_json::Value, + ) -> Result< + crate::ec::identity::Resolution, + error_stack::Report, + > { + let bytes = serde_json::to_vec(body).expect("should serialize the response body"); + self.respond_raw(status, &[], &bytes) + } + + /// Calls `response` with raw headers and body. + pub(crate) fn respond_raw( + &mut self, + status: http::StatusCode, + headers: &[(&str, &str)], + body: &[u8], + ) -> Result< + crate::ec::identity::Resolution, + error_stack::Report, + > { + let mut header_map = http::HeaderMap::new(); + for (name, value) in headers { + header_map.insert( + http::HeaderName::from_bytes(name.as_bytes()).expect("should be a header name"), + http::HeaderValue::from_str(value).expect("should be a header value"), + ); + } + let context = self + .last_request + .as_ref() + .and_then(|request| request.call_context_json().map(str::to_owned)); + let response = + crate::ec::identity::ProviderResponse::new(status, header_map, body.to_vec(), context); + let signals = self.signals(); + let input = crate::ec::identity::ResolveInput::new( + self.provider.descriptor(), + self.kinds(), + &signals, + ); + self.provider.response(&response, &input) + } +} diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index fe8b252683..4359d586b0 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -1604,6 +1604,14 @@ impl KvIdentityGraph { )) } + /// Test helper: an in-memory graph that counts reads and writes by mode. + pub(crate) fn write_counting( + store_name: impl Into, + ) -> (Self, super::kv_backend::test_support::WriteCounter) { + let (store, counter) = super::kv_backend::test_support::WriteCountingEcKv::new(store_name); + (Self::new(store), counter) + } + /// Test helper: a graph whose point reads always miss and whose list API /// errors, so absence can neither be observed nor proved. pub(crate) fn unprovable_absence(store_name: impl Into) -> Self { @@ -1620,6 +1628,39 @@ mod tests { use super::*; use crate::ec::kv_backend::test_support::InMemoryEcKv; + #[test] + fn write_counting_graph_counts_by_mode_and_injects_conflicts() { + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + graph + .create(&ec_id, &KvEntry::minimal("a.example.com", "u", 1)) + .expect("should create"); + let (_, generation) = graph + .get(&ec_id) + .expect("should read") + .expect("should exist"); + counter.conflict_next(1); + let first = graph.store.insert( + &ec_id, + EcKvWrite { + body: "{}", + metadata: "{}", + ttl: ENTRY_TTL, + mode: EcKvWriteMode::IfGenerationMatch(generation), + }, + ); + assert_eq!( + first.expect("should answer"), + EcKvWriteOutcome::PreconditionFailed, + "the armed conflict fails the first CAS" + ); + let counts = counter.writes(); + assert_eq!(counts.add, 1, "one create"); + assert_eq!(counts.if_generation_match, 1, "one CAS attempt"); + assert_eq!(counts.lookups, 1, "one read"); + assert_eq!(counts.total(), 2, "two writes in total"); + } + #[test] fn browser_ingest_records_acquisition_time() { let mut entry = KvEntry::minimal("sharedid.org", "existing", 1); diff --git a/crates/trusted-server-core/src/ec/kv_backend.rs b/crates/trusted-server-core/src/ec/kv_backend.rs index bfe74ca6f3..5d85a42e78 100644 --- a/crates/trusted-server-core/src/ec/kv_backend.rs +++ b/crates/trusted-server-core/src/ec/kv_backend.rs @@ -233,6 +233,129 @@ pub(crate) mod test_support { } } + /// Writes and reads a [`WriteCountingEcKv`] performed, by mode. + #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] + pub(crate) struct WriteCounts { + /// `lookup` calls. + pub(crate) lookups: usize, + /// `Add` writes attempted. + pub(crate) add: usize, + /// `IfGenerationMatch` writes attempted. + pub(crate) if_generation_match: usize, + /// `Overwrite` writes attempted. + pub(crate) overwrite: usize, + } + + impl WriteCounts { + /// Every write attempted, whatever its mode or outcome. + pub(crate) fn total(&self) -> usize { + self.add + self.if_generation_match + self.overwrite + } + } + + /// [`EcKvStore`] that counts reads and writes by mode, and can fail the + /// next conditional writes with a precondition failure. + pub(crate) struct WriteCountingEcKv { + inner: InMemoryEcKv, + counts: std::sync::Arc>, + conflicts: std::sync::Arc, + } + + /// A handle that reads a [`WriteCountingEcKv`]'s counts and arms its + /// conflicts after the store moved into a graph. + #[derive(Clone)] + pub(crate) struct WriteCounter { + counts: std::sync::Arc>, + conflicts: std::sync::Arc, + } + + impl WriteCounter { + /// The counts so far. + pub(crate) fn writes(&self) -> WriteCounts { + *self.counts.lock().expect("should lock write counts") + } + + /// Fails the next `count` `IfGenerationMatch` writes with a + /// precondition failure, as a concurrent writer would. + pub(crate) fn conflict_next(&self, count: usize) { + self.conflicts + .store(count, std::sync::atomic::Ordering::Relaxed); + } + } + + impl WriteCountingEcKv { + pub(crate) fn new(name: impl Into) -> (Self, WriteCounter) { + let counts = std::sync::Arc::new(Mutex::new(WriteCounts::default())); + let conflicts = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); + ( + Self { + inner: InMemoryEcKv::new(name), + counts: std::sync::Arc::clone(&counts), + conflicts: std::sync::Arc::clone(&conflicts), + }, + WriteCounter { counts, conflicts }, + ) + } + } + + impl EcKvStore for WriteCountingEcKv { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup(&self, key: &str) -> Result, Report> { + self.counts + .lock() + .expect("should lock write counts") + .lookups += 1; + self.inner.lookup(key) + } + + fn key_exists(&self, key: &str) -> Result> { + self.inner.key_exists(key) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + { + let mut counts = self.counts.lock().expect("should lock write counts"); + match write.mode { + EcKvWriteMode::Add => counts.add += 1, + EcKvWriteMode::IfGenerationMatch(_) => counts.if_generation_match += 1, + EcKvWriteMode::Overwrite => counts.overwrite += 1, + } + } + if matches!(write.mode, EcKvWriteMode::IfGenerationMatch(_)) + && self + .conflicts + .fetch_update( + std::sync::atomic::Ordering::Relaxed, + std::sync::atomic::Ordering::Relaxed, + |remaining| remaining.checked_sub(1), + ) + .is_ok() + { + return Ok(EcKvWriteOutcome::PreconditionFailed); + } + self.inner.insert(key, write) + } + + fn list_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result, Report> { + self.inner.list_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + self.inner.delete(key) + } + } + /// [`EcKvStore`] wrapper that models an eventually-consistent point read. /// /// The first `stale_lookups` calls to [`EcKvStore::lookup`] report the key diff --git a/crates/trusted-server-core/src/ec/pull_sync.rs b/crates/trusted-server-core/src/ec/pull_sync.rs index 911600df9b..9e8ad8e547 100644 --- a/crates/trusted-server-core/src/ec/pull_sync.rs +++ b/crates/trusted-server-core/src/ec/pull_sync.rs @@ -315,7 +315,7 @@ fn is_partner_pull_eligible(partner: &PartnerConfig, kv_entry: Option<&KvEntry>) .is_none() } -fn validated_pull_sync_url(partner: &PartnerConfig) -> Option { +pub(crate) fn validated_pull_sync_url(partner: &PartnerConfig) -> Option { let pull_sync_url = partner.pull_sync_url.as_deref()?; let parsed = match Url::parse(pull_sync_url) { Ok(url) => url, @@ -365,7 +365,7 @@ fn validated_pull_sync_url(partner: &PartnerConfig) -> Option { Some(parsed) } -fn build_pull_request_url(mut base_url: Url, ec_id: &str) -> Url { +pub(crate) fn build_pull_request_url(mut base_url: Url, ec_id: &str) -> Url { base_url.query_pairs_mut().append_pair("ec_id", ec_id); base_url } diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 49143ecddf..f66f2e3a82 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -111,6 +111,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/exchange.rs", include_str!("ec/identity/exchange.rs"), ), + ( + "ec/identity/generic_pull.rs", + include_str!("ec/identity/generic_pull.rs"), + ), ("ec/identity/ids.rs", include_str!("ec/identity/ids.rs")), ("ec/identity/input.rs", include_str!("ec/identity/input.rs")), ("ec/identity/mod.rs", include_str!("ec/identity/mod.rs")), From cd6165e7631c8b0dc50bc73513c09892dcae1b94 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:46:11 +0530 Subject: [PATCH 24/33] Decide the writer request once routing allows dispatch Only an organic real-browser top-level document navigation that is not a prefetch is the writer. EcContext now captures the user agent, DNT and the raw identity cookies when the request is read, because after-send code no longer has the request. Integration-proxy EC generation now requires a real browser explicitly, and is_prefetch_request moves to http_util. Signed-off-by: dhruv8sh --- crates/trusted-server-adapter-axum/src/app.rs | 1 + .../src/app.rs | 1 + .../trusted-server-adapter-fastly/src/app.rs | 2 + crates/trusted-server-adapter-spin/src/app.rs | 1 + .../src/ec/identity/mod.rs | 1 + .../src/ec/identity/request.rs | 197 ++++++++++++++++++ crates/trusted-server-core/src/ec/mod.rs | 119 +++++++++++ crates/trusted-server-core/src/http_util.rs | 70 +++++++ .../src/integrations/gpt_diagnostics.rs | 2 +- .../src/integrations/registry.rs | 67 +++++- .../src/migration_guards.rs | 4 + crates/trusted-server-core/src/publisher.rs | 16 +- 12 files changed, 465 insertions(+), 16 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity/request.rs diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index caba714d68..97de0be5e8 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -233,6 +233,7 @@ async fn dispatch_fallback( ec_context: &mut ec_context, services, req, + is_real_browser: false, }) .await .unwrap_or_else(|| { diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index 87b9567e7f..4ce119e414 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -545,6 +545,7 @@ fn build_router(state: &Arc) -> RouterService { ec_context: &mut ec_context, services: &services, req, + is_real_browser: false, }) .await .unwrap_or_else(|| { diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index bf09c2086e..588e47cb75 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -820,6 +820,7 @@ async fn dispatch_fallback( ec_context: &mut ec.ec_context, services, req, + is_real_browser: ec.is_real_browser, }) .await .unwrap_or_else(|| { @@ -847,6 +848,7 @@ async fn dispatch_fallback( if is_navigation { ec.ec_context.set_eid_sync_source(EidSyncSource::Navigation); } + ec.ec_context.decide_writer(&req, true, ec.is_real_browser); let is_publisher_navigation = ec.is_real_browser && is_navigation; if is_publisher_navigation && let Err(err) = ec diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 804a23b620..535ef5f37f 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -826,6 +826,7 @@ fn build_router(state: &Arc) -> RouterService { ec_context: &mut ec_context, services: &services, req, + is_real_browser: false, }) .await .unwrap_or_else(|| { diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index d23bcd187b..21a913bb79 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -26,6 +26,7 @@ pub mod input; pub(crate) mod mutation; pub(crate) mod policy; pub mod provider; +pub(crate) mod request; pub mod resolution; pub(crate) mod revision; pub(crate) mod servable; diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs new file mode 100644 index 0000000000..a26741c3e7 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -0,0 +1,197 @@ +//! Request-scoped identity state, carried on `EcContext` to finalize and to +//! the after-send flush. + +use http::HeaderMap; +use url::Url; + +use super::catalog::ProviderCatalog; +use super::cookie::{ + CookieRead, OpenedRemovals, RBatch, SCookie, SOpen, open, read_prefixed_cookies, + read_single_cookie, +}; +use super::mutation::IdentityMutation; +use super::revision::IdentityKeys; +use crate::constants::{COOKIE_TS_EIDS_R_PREFIX, COOKIE_TS_EIDS_S}; + +/// The identity cookies' raw values, captured when the request is read and +/// opened only by the routes that use identity. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(crate) struct RawIdentityCookies { + s: Option, + r: Vec<(String, String)>, +} + +impl RawIdentityCookies { + /// Captures both cookie kinds from request headers. + #[must_use] + pub(crate) fn capture(headers: &HeaderMap) -> Self { + let s = match read_single_cookie(headers, COOKIE_TS_EIDS_S) { + CookieRead::Absent => None, + read => Some(read), + }; + Self { + s, + r: read_prefixed_cookies(headers, COOKIE_TS_EIDS_R_PREFIX), + } + } + + /// Whether the request carried either cookie. + #[must_use] + pub(crate) fn has_any(&self) -> bool { + self.s.is_some() || !self.r.is_empty() + } +} + +/// The identity cookies after opening them for this EC. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct OpenedIdentity { + pub(crate) s: SOpen, + pub(crate) removals: OpenedRemovals, +} + +/// What this response must do with the identity cookies. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct StagedCookies { + /// A `__Host-ts-eids-s` to set. + pub(crate) s_cookie: Option, + /// Expire `__Host-ts-eids-s`. + pub(crate) expire_s: bool, + /// This response's removal batch. + pub(crate) new_r_batch: RBatch, + /// Removal cookies to delete, by name. + pub(crate) delete_r: Vec, +} + +/// Request-scoped identity state. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct RequestIdentity { + writer: bool, + user_agent: Option, + dnt: bool, + page_url: Option, + raw: RawIdentityCookies, + opened: Option, + mutation: IdentityMutation, + staged: StagedCookies, +} + +impl RequestIdentity { + /// Captures the signals phase 1 needs, because after-send code no longer + /// has the request. + #[must_use] + pub(crate) fn capture(headers: &HeaderMap) -> Self { + let header = |name: &str| headers.get(name).and_then(|value| value.to_str().ok()); + Self { + user_agent: header("user-agent").map(str::to_owned), + dnt: header("dnt").is_some_and(|value| value.trim() == "1"), + raw: RawIdentityCookies::capture(headers), + ..Self::default() + } + } + + /// Whether this request is the page load's writer. + #[must_use] + pub(crate) fn writer(&self) -> bool { + self.writer + } + + pub(crate) fn set_writer(&mut self, writer: bool) { + self.writer = writer; + } + + /// The request's `User-Agent`. + #[must_use] + pub(crate) fn user_agent(&self) -> Option<&str> { + self.user_agent.as_deref() + } + + /// Whether `DNT: 1` was sent. + #[must_use] + pub(crate) fn dnt(&self) -> bool { + self.dnt + } + + /// The page URL, without query or fragment. + #[must_use] + pub(crate) fn page_url(&self) -> Option<&Url> { + self.page_url.as_ref() + } + + /// Records the page URL, dropping its query and fragment. + pub(crate) fn set_page_url(&mut self, mut url: Url) { + url.set_query(None); + url.set_fragment(None); + self.page_url = Some(url); + } + + /// The raw identity cookies. + #[must_use] + pub(crate) fn raw_cookies(&self) -> &RawIdentityCookies { + &self.raw + } + + /// Opens the identity cookies for `ec_id`, once. Later calls do nothing. + pub(crate) fn open_cookies( + &mut self, + keys: &IdentityKeys, + catalog: &ProviderCatalog, + ec_id: &str, + ) { + if self.opened.is_some() { + return; + } + let s = match &self.raw.s { + None => SOpen::Missing, + Some(raw) => { + let mut headers = HeaderMap::new(); + if let CookieRead::Value(value) = raw + && let Ok(header) = + http::HeaderValue::from_str(&format!("{COOKIE_TS_EIDS_S}={value}")) + { + headers.insert(http::header::COOKIE, header); + super::cookie::open_s_cookie(keys, ec_id, catalog, &headers) + } else { + SOpen::Failed + } + } + }; + let mut removals = OpenedRemovals::default(); + for (name, value) in &self.raw.r { + match open(&keys.cookie_r, ec_id, value) + .and_then(|payload| serde_json::from_slice::(&payload).ok()) + { + Some(batch) => removals.cookies.push((name.clone(), batch)), + None => removals.unopenable.push(name.clone()), + } + } + self.opened = Some(OpenedIdentity { s, removals }); + } + + /// The opened cookies, once [`Self::open_cookies`] ran. + #[must_use] + pub(crate) fn opened(&self) -> Option<&OpenedIdentity> { + self.opened.as_ref() + } + + /// The writer's single mutation. + #[must_use] + pub(crate) fn mutation(&self) -> &IdentityMutation { + &self.mutation + } + + /// The writer's single mutation, to add changes. + pub(crate) fn mutation_mut(&mut self) -> &mut IdentityMutation { + &mut self.mutation + } + + /// What the response must do with the identity cookies. + #[must_use] + pub(crate) fn staged(&self) -> &StagedCookies { + &self.staged + } + + /// What the response must do with the identity cookies, to change it. + pub(crate) fn staged_mut(&mut self) -> &mut StagedCookies { + &mut self.staged + } +} diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index ce599c0c36..a10880dd62 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -284,6 +284,10 @@ pub struct EcContext { pull_sync_marker: PullSyncMarkerState, /// Allowed returning-user EID persistence source, assigned only after request filters pass. eid_sync_source: Option, + /// Request-scoped identity provider state: the writer decision, phase-1 + /// signals, the identity cookies, the single mutation and the cookies + /// this response sets. + identity: identity::request::RequestIdentity, } impl EcContext { @@ -366,6 +370,7 @@ impl EcContext { client_eids: None, pull_sync_marker: PullSyncMarkerState::from_cookie(parsed.pull_sync_marker), eid_sync_source: None, + identity: identity::request::RequestIdentity::capture(req.headers()), }) } @@ -649,6 +654,48 @@ impl EcContext { self.ec_value.as_deref().map(generation::ec_hash) } + /// Request-scoped identity provider state. + #[allow( + dead_code, + reason = "identity finalize, which reads this, lands in a following commit" + )] + #[must_use] + pub(crate) fn identity(&self) -> &identity::request::RequestIdentity { + &self.identity + } + + /// Request-scoped identity provider state, to change it. + #[allow( + dead_code, + reason = "identity finalize, which writes this, lands in a following commit" + )] + pub(crate) fn identity_mut(&mut self) -> &mut identity::request::RequestIdentity { + &mut self.identity + } + + /// Decides whether this request is the page load's identity writer. + /// + /// Call once pre-route filters have allowed dispatch, where routing + /// decides EC generation. The writer is an organic route (the publisher + /// fallback or an integration proxy), a `Sec-Fetch-Dest: document` + /// navigation that is not a prefetch, from a real browser. It also records + /// the page URL for providers. + pub fn decide_writer( + &mut self, + req: &Request, + organic_route: bool, + is_real_browser: bool, + ) { + let writer = organic_route + && is_real_browser + && crate::http_util::is_document_navigation(req) + && !crate::http_util::is_prefetch_request(req); + self.identity.set_writer(writer); + if writer && let Ok(url) = url::Url::parse(&req.uri().to_string()) { + self.identity.set_page_url(url); + } + } + /// Creates a test-only `EcContext` with explicit field values. #[cfg(test)] #[must_use] @@ -667,6 +714,7 @@ impl EcContext { client_eids: None, pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, + identity: identity::request::RequestIdentity::default(), } } @@ -692,6 +740,7 @@ impl EcContext { client_eids: None, pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, + identity: identity::request::RequestIdentity::default(), } } @@ -720,6 +769,7 @@ impl EcContext { client_eids: None, pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, + identity: identity::request::RequestIdentity::default(), } } } @@ -775,6 +825,75 @@ mod tests { use crate::platform::test_support::noop_services; use crate::test_support::tests::create_test_settings; + fn navigation_request() -> Request { + Request::builder() + .uri("https://www.example.com/a?b=c#d") + .header("sec-fetch-dest", "document") + .header(http::header::USER_AGENT, "ExampleBrowser/1.0") + .header("dnt", "1") + .header( + http::header::COOKIE, + "__Host-ts-eids-s=v1.a.b; __Host-ts-eids-r-0a1b2c3d=v1.c.d", + ) + .body(EdgeBody::empty()) + .expect("should build request") + } + + #[test] + fn decide_writer_needs_every_condition() { + let mut context = EcContext::default(); + context.decide_writer(&navigation_request(), true, true); + assert!( + context.identity().writer(), + "organic real-browser document navigation" + ); + assert_eq!( + context.identity().page_url().map(url::Url::as_str), + Some("https://www.example.com/a"), + "the page URL has no query or fragment" + ); + + let prefetch = Request::builder() + .header("sec-fetch-dest", "document") + .header("sec-purpose", "prefetch") + .body(EdgeBody::empty()) + .expect("should build request"); + let accept_only = Request::builder() + .header(http::header::ACCEPT, "text/html") + .body(EdgeBody::empty()) + .expect("should build request"); + for (name, organic, browser, request) in [ + ("named route", false, true, navigation_request()), + ("bot", true, false, navigation_request()), + ("prefetch", true, true, prefetch), + ("no Sec-Fetch-Dest", true, true, accept_only), + ] { + let mut context = EcContext::default(); + context.decide_writer(&request, organic, browser); + assert!(!context.identity().writer(), "not a writer: {name}"); + } + } + + #[test] + fn read_from_request_captures_identity_signals() { + let settings = create_test_settings(); + let context = + EcContext::read_from_request(&settings, &navigation_request(), &noop_services()) + .expect("should read the request"); + let identity = context.identity(); + assert_eq!( + identity.user_agent(), + Some("ExampleBrowser/1.0"), + "captures the user agent" + ); + assert!(identity.dnt(), "captures DNT"); + assert!( + identity.raw_cookies().has_any(), + "captures the raw identity cookies" + ); + assert!(!identity.writer(), "reading alone never makes a writer"); + } + /// [`EcKvStore`] wrapper whose first `collisions` `Add` writes report a /// precondition failure, forcing generation to retry with a fresh suffix. struct AddCollidingEcKv { diff --git a/crates/trusted-server-core/src/http_util.rs b/crates/trusted-server-core/src/http_util.rs index 457e503568..a0a9804497 100644 --- a/crates/trusted-server-core/src/http_util.rs +++ b/crates/trusted-server-core/src/http_util.rs @@ -117,6 +117,34 @@ pub fn is_navigation_request(req: &Request) -> bool { }) } +/// Returns whether this is a top-level document navigation by +/// `Sec-Fetch-Dest: document` alone. +/// +/// Unlike [`is_navigation_request`], the `Accept: text/html` fallback does not +/// count, because a page script's `fetch()` can send it. Only such a request +/// can be a page load's identity writer. +#[must_use] +pub fn is_document_navigation(req: &Request) -> bool { + req.headers() + .get("sec-fetch-dest") + .and_then(|value| value.to_str().ok()) + .is_some_and(|dest| dest.trim().eq_ignore_ascii_case("document")) +} + +/// Returns true when the request advertises itself as a prefetch or +/// prerender via either the standard `Sec-Purpose` or the legacy `Purpose` +/// header. +#[must_use] +pub fn is_prefetch_request(req: &Request) -> bool { + let header = |name: &str| { + req.headers() + .get(name) + .and_then(|v| v.to_str().ok()) + .is_some_and(|v| v.contains("prefetch")) + }; + header("sec-purpose") || header("purpose") +} + /// Extracted request information for host rewriting. /// /// This struct captures the effective host and scheme from an incoming request. @@ -487,6 +515,48 @@ pub fn enforce_max_body_size( mod tests { use super::*; use crate::platform::ClientInfo; + + #[test] + fn document_navigation_requires_sec_fetch_dest_document() { + let with_dest = Request::builder() + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("should build request"); + let accept_only = Request::builder() + .header(header::ACCEPT, "text/html") + .body(EdgeBody::empty()) + .expect("should build request"); + let iframe = Request::builder() + .header("sec-fetch-dest", "iframe") + .body(EdgeBody::empty()) + .expect("should build request"); + assert!(is_document_navigation(&with_dest), "document counts"); + assert!( + !is_document_navigation(&accept_only), + "the Accept fallback does not" + ); + assert!( + is_navigation_request(&accept_only), + "control: the broad check still accepts it" + ); + assert!(!is_document_navigation(&iframe), "iframes do not"); + } + + #[test] + fn prefetch_is_detected_from_either_header() { + for header_name in ["sec-purpose", "purpose"] { + let request = Request::builder() + .header(header_name, "prefetch;prerender") + .body(EdgeBody::empty()) + .expect("should build request"); + assert!(is_prefetch_request(&request), "{header_name} prefetch"); + } + let plain = Request::builder() + .body(EdgeBody::empty()) + .expect("should build request"); + assert!(!is_prefetch_request(&plain), "control: no header"); + } + use crate::redacted::Redacted; use crate::settings::TrustedClientIpConfig; use http::{HeaderName, HeaderValue, Method}; diff --git a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs index 1447a83586..e31890f982 100644 --- a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs +++ b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs @@ -271,7 +271,7 @@ pub fn prepare_request( let cookie_state = console_cookie_state(request); let eligible_navigation = request.method() == Method::GET && is_navigation_request(request) - && !crate::publisher::is_prefetch_request(request) + && !crate::http_util::is_prefetch_request(request) && !crate::publisher::is_bot_user_agent(request); sanitize_console_cookie(request); diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index d3c09cd7d7..87285118e3 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -807,6 +807,9 @@ pub struct ProxyDispatchInput<'a> { pub ec_context: &'a mut EcContext, pub services: &'a RuntimeServices, pub req: Request, + /// Whether the bot gate recognised a real browser. Proxy EC generation + /// and the identity writer decision both require it. + pub is_real_browser: bool, } /// In-memory registry of integrations discovered from settings. @@ -1040,18 +1043,20 @@ impl IntegrationRegistry { ec_context, services, mut req, + is_real_browser, } = input; if let Some((proxy, _)) = self.find_route(method, path) { // Organic proxy handler: generate if needed (best effort). // Only generate for document navigations — subresource requests // may lack consent signals such as the Sec-GPC header. - if is_navigation_request(&req) { + ec_context.decide_writer(&req, true, is_real_browser); + if is_real_browser && is_navigation_request(&req) { if let Err(err) = ec_context.generate_if_needed(settings, kv) { log::warn!("EC generation failed for integration proxy: {err:?}"); } } else { log::debug!( - "EC generation skipped for integration proxy: non-document request (path={path})", + "EC generation skipped for integration proxy: not a real-browser document request (path={path})", ); } @@ -1691,6 +1696,7 @@ mod tests { ec_context: &mut ec_context, services: &noop_services(), req, + is_real_browser: true, })) .expect("should match route") .expect("proxy should succeed"); @@ -2048,6 +2054,7 @@ mod tests { ec_context: &mut ec_context, services: &services, req, + is_real_browser: true, })); // Should have matched and returned a response @@ -2063,6 +2070,59 @@ mod tests { ); } + fn proxy_navigation(is_real_browser: bool) -> EcContext { + let settings = create_test_settings(); + let routes = vec![( + Method::GET, + "/integrations/test/ec", + ( + Arc::new(EcTestProxy) as Arc, + "ec_test", + ), + )]; + let registry = IntegrationRegistry::from_routes(routes); + let req = Request::builder() + .method(Method::GET) + .uri("https://test-publisher.com/integrations/test/ec") + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("should build request"); + let consent = crate::consent::ConsentContext { + jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, + ..crate::consent::ConsentContext::default() + }; + let mut ec_context = + EcContext::new_for_test_with_ip(None, consent, Some("192.0.2.10".to_owned())); + let kv = crate::ec::kv::KvIdentityGraph::in_memory("proxy_store"); + let services = noop_services(); + let result = futures::executor::block_on(registry.handle_proxy(ProxyDispatchInput { + method: &Method::GET, + path: "/integrations/test/ec", + settings: &settings, + kv: Some(&kv), + ec_context: &mut ec_context, + services: &services, + req, + is_real_browser, + })); + assert!(result.is_some(), "should find the route"); + ec_context + } + + #[test] + fn handle_proxy_mints_ec_and_decides_writer_only_for_real_browsers() { + let browser = proxy_navigation(true); + assert!( + browser.ec_generated(), + "a real browser's navigation mints an EC" + ); + assert!(browser.identity().writer(), "and is the writer"); + + let bot = proxy_navigation(false); + assert!(!bot.ec_generated(), "a bot's navigation mints no EC"); + assert!(!bot.identity().writer(), "and is not the writer"); + } + #[test] fn handle_proxy_rejects_invalid_ec_request_header() { let settings = create_test_settings(); @@ -2098,6 +2158,7 @@ mod tests { ec_context: &mut ec_context, services: &services, req, + is_real_browser: true, })) .expect("should handle proxy request"); @@ -2147,6 +2208,7 @@ mod tests { ec_context: &mut ec_context, services: &services, req, + is_real_browser: true, })) .expect("should handle proxy request"); @@ -2193,6 +2255,7 @@ mod tests { ec_context: &mut ec_context, services: &services, req, + is_real_browser: true, })); assert!(result.is_some(), "Should find POST route"); diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index f66f2e3a82..e69b644597 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -130,6 +130,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/provider.rs", include_str!("ec/identity/provider.rs"), ), + ( + "ec/identity/request.rs", + include_str!("ec/identity/request.rs"), + ), ( "ec/identity/resolution.rs", include_str!("ec/identity/resolution.rs"), diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index ff8e8ec29c..14949fd0d2 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -72,7 +72,9 @@ use crate::ec::registry::PartnerRegistry; use crate::ec::{EcContext, EidSyncSource}; use crate::error::TrustedServerError; use crate::html_processor::BodyCloseInjection; -use crate::http_util::{RequestInfo, is_navigation_request, serve_static_with_etag}; +use crate::http_util::{ + RequestInfo, is_navigation_request, is_prefetch_request, serve_static_with_etag, +}; use crate::integrations::IntegrationRegistry; use crate::platform::{ GeoInfo, PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, @@ -3205,18 +3207,6 @@ pub(crate) fn is_bot_user_agent(req: &Request) -> bool { .any(|frag| ua.contains(frag)) } -/// Returns true when the request advertises itself as a prefetch via either -/// the standard `Sec-Purpose` or the legacy `Purpose` header. -pub(crate) fn is_prefetch_request(req: &Request) -> bool { - let header = |name: &str| { - req.headers() - .get(name) - .and_then(|v| v.to_str().ok()) - .is_some_and(|v| v.contains("prefetch")) - }; - header("sec-purpose") || header("purpose") -} - /// Returns whether request-scoped signals permit an ad-eligible navigation. /// /// This is the request half of the shared ad-stack gate: the configuration From c9625a37abdd40fdaeca4eb1bfba7e2ff0f0eec1 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 17:48:25 +0530 Subject: [PATCH 25/33] Stop writing cluster size from /identify /identify uses a stored cluster size, or computes one without writing it. The writer navigation stores it in its single flush. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/ec/identify.rs | 48 ++++++++++- crates/trusted-server-core/src/ec/kv.rs | 79 +++++-------------- 2 files changed, 63 insertions(+), 64 deletions(-) diff --git a/crates/trusted-server-core/src/ec/identify.rs b/crates/trusted-server-core/src/ec/identify.rs index e9983945d5..7654dcb953 100644 --- a/crates/trusted-server-core/src/ec/identify.rs +++ b/crates/trusted-server-core/src/ec/identify.rs @@ -93,7 +93,7 @@ pub fn handle_identify( let mut cluster_size: Option = None; match kv.get(ec_id) { - Ok(Some((entry, generation))) => { + Ok(Some((entry, _generation))) => { if !entry.consent.ok { // Tombstone entries preserve the withdrawal signal for 24 hours. // Do not extract IDs or evaluate cluster size because that would @@ -105,9 +105,9 @@ pub fn handle_identify( // and never the row's stored consent snapshot. uid = servable_partner_uid(&entry, &partner.source_domain, ec_context); - // Evaluate cluster size lazily for identify responses. Existing - // stored cluster_size values are reused without a prefix-list call. - match kv.evaluate_cluster(ec_id, &entry, generation) { + // Use a stored cluster size, or compute one without writing it. + // The writer navigation stores it in its single mutation. + match kv.cluster_size(ec_id, &entry) { Ok(size) => { cluster_size = size; } @@ -694,6 +694,46 @@ mod tests { .map(str::to_owned) } + #[test] + fn handle_identify_computes_cluster_size_without_writing() { + let settings = create_test_settings(); + let (kv, counter) = KvIdentityGraph::write_counting("identify_store"); + let partners = vec![make_test_partner("ssp.example.com", VALID_API_TOKEN)]; + let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + kv.create( + &ec_id, + &crate::ec::kv_types::KvEntry::minimal("ssp.example.com", "u", 1), + ) + .expect("should seed the row"); + let writes_before = counter.writes().total(); + let req = Request::builder() + .method("GET") + .uri("https://edge.test-publisher.com/identify") + .header("authorization", format!("Bearer {VALID_API_TOKEN}")) + .body(EdgeBody::empty()) + .expect("should build test request"); + let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + + let response = handle_identify(&settings, &kv, ®istry, &req, &ec_context) + .expect("should answer identify"); + let body = serde_json::from_slice::( + &response.into_body().into_bytes().unwrap_or_default(), + ) + .expect("should decode identify response JSON"); + + assert_eq!( + body["cluster_size"], + serde_json::json!(1), + "should report a size" + ); + assert_eq!( + counter.writes().total(), + writes_before, + "should not write the computed size" + ); + } + #[test] fn handle_identify_omits_expired_ids() { let expired = crate::ec::kv_types::KvPartnerId { diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index 4359d586b0..a1b2028659 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -21,7 +21,7 @@ use crate::error::TrustedServerError; use super::generation::ec_hash; use super::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome}; -use super::kv_types::{KvEntry, KvMetadata, KvNetwork, KvPartnerId}; +use super::kv_types::{KvEntry, KvMetadata, KvPartnerId}; use super::{EcKvSnapshot, checked_current_timestamp, current_timestamp, log_id}; /// Maximum number of CAS retry attempts before giving up. @@ -1468,27 +1468,25 @@ impl KvIdentityGraph { .count_keys_with_prefix(hash_prefix, CLUSTER_LIST_LIMIT) } - /// Evaluates the cluster size for an EC entry. + /// Returns the cluster size for an entry, without writing it. /// - /// Returns the stored `cluster_size` when it has already been evaluated - /// for a live entry. Tombstone entries return `None` without store I/O so - /// their 24-hour withdrawal TTL is not extended. Otherwise, counts the - /// number of keys sharing the same hash prefix via - /// [`count_hash_prefix_keys`](Self::count_hash_prefix_keys) and writes the - /// result back to the entry. The CAS write is best-effort — on conflict - /// or write failure, the computed value is still returned. + /// Uses the stored `cluster_size` when the entry has one. Tombstone + /// entries return `None` without store I/O. Otherwise counts the keys + /// that share the EC's hash prefix with + /// [`count_hash_prefix_keys`](Self::count_hash_prefix_keys). The writer + /// request stores a computed size in its single mutation; `/identify` + /// never writes. /// /// # Errors /// - /// Returns [`TrustedServerError::KvStore`] on store or list failure. - pub fn evaluate_cluster( + /// Returns [`TrustedServerError::KvStore`] on list failure. + pub fn cluster_size( &self, ec_id: &str, entry: &KvEntry, - generation: u64, ) -> Result, Report> { if !entry.consent.ok { - log::trace!("evaluate_cluster: skipping tombstone entry"); + log::trace!("cluster_size: skipping tombstone entry"); return Ok(None); } @@ -1497,54 +1495,15 @@ impl KvIdentityGraph { .as_ref() .and_then(|network| network.cluster_size) { - log::trace!("evaluate_cluster: using stored cluster_size"); + log::trace!("cluster_size: using stored cluster_size"); return Ok(Some(cluster_size)); } - // Compute cluster size via prefix list. - let hash_prefix = ec_hash(ec_id); - let cluster_size = self.count_hash_prefix_keys(hash_prefix)?; - + let cluster_size = self.count_hash_prefix_keys(ec_hash(ec_id))?; log::debug!( - "evaluate_cluster: computed cluster_size={cluster_size} for '{}'", + "cluster_size: computed cluster_size={cluster_size} for '{}'", log_id(ec_id) ); - - // Best-effort CAS write-back — update only the cluster size so any - // future `network` fields are preserved across this lazy write. - let mut updated_entry = entry.clone(); - let mut network = updated_entry - .network - .unwrap_or(KvNetwork { cluster_size: None }); - network.cluster_size = Some(cluster_size); - updated_entry.network = Some(network); - - let (body, meta_str) = Self::serialize_entry(&updated_entry, self.store_name())?; - - match self.write_entry( - ec_id, - &body, - &meta_str, - ENTRY_TTL, - EcKvWriteMode::IfGenerationMatch(generation), - ) { - Ok(EcKvWriteOutcome::Written) => {} - Ok(EcKvWriteOutcome::PreconditionFailed) => { - log::debug!( - "evaluate_cluster: CAS conflict writing cluster_size for '{}', \ - returning computed value anyway", - log_id(ec_id), - ); - } - Err(report) => { - // Log but don't fail — the computed value is still valid. - log::warn!( - "evaluate_cluster: failed to write cluster_size for '{}': {report}", - log_id(ec_id) - ); - } - } - Ok(Some(cluster_size)) } @@ -2556,16 +2515,16 @@ mod tests { } #[test] - fn evaluate_cluster_returns_stored_value_without_store_io() { + fn cluster_size_returns_stored_value_without_store_io() { let kv = KvIdentityGraph::failing("nonexistent_store_for_cluster_cache_test"); let ec_id = format!("{}.ABC123", "a".repeat(64)); let mut entry = live_entry(); - entry.network = Some(KvNetwork { + entry.network = Some(crate::ec::kv_types::KvNetwork { cluster_size: Some(5), }); let cluster_size = kv - .evaluate_cluster(&ec_id, &entry, 0) + .cluster_size(&ec_id, &entry) .expect("should not touch store when cluster_size is already known"); assert_eq!( @@ -2576,13 +2535,13 @@ mod tests { } #[test] - fn evaluate_cluster_skips_tombstone_without_store_io() { + fn cluster_size_skips_tombstone_without_store_io() { let kv = KvIdentityGraph::failing("nonexistent_store_for_tombstone_cluster_test"); let ec_id = format!("{}.ABC123", "a".repeat(64)); let entry = KvEntry::tombstone(1000); let cluster_size = kv - .evaluate_cluster(&ec_id, &entry, 0) + .cluster_size(&ec_id, &entry) .expect("should not touch store for tombstone entries"); assert_eq!( From 0d4fba3d02931f79264b62c3dbda5702cc0d76a0 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:07:08 +0530 Subject: [PATCH 26/33] Queue browser ingest in the writer's mutation and stage it on stagers Returning-visitor finalize no longer writes KV. The writer navigation adds browser ingest, flush-checked staged entries and removal batches to its single mutation. /auction saves the KV gap of its body EIDs in __Host-ts-eids-s, and /_ts/page-bids no longer ingests. A new visitor's create now carries its browser EIDs, so it stays one write. Orphan rotation carries removals into the new row and never positive staged IDs. Consent loss and tombstone reads expire __Host-ts-eids-s. The provider catalog now indexes every registered partner source, so a KV-gap entry can be staged for any of them. The old ingest-sync path in kv.rs is removed with its tests. The queued mutation is written by the after-send flush added later in this series. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/app.rs | 19 +- .../src/auction/endpoints.rs | 76 +- crates/trusted-server-core/src/ec/finalize.rs | 459 ++++----- .../src/ec/identity/catalog.rs | 58 +- .../src/ec/identity/finalize.rs | 946 ++++++++++++++++++ .../src/ec/identity/mod.rs | 1 + .../src/ec/identity/mutation.rs | 7 + .../src/ec/identity/request.rs | 43 + .../src/ec/identity/test_support.rs | 64 ++ crates/trusted-server-core/src/ec/kv.rs | 659 ------------ crates/trusted-server-core/src/ec/mod.rs | 80 +- .../src/migration_guards.rs | 4 + 12 files changed, 1416 insertions(+), 1000 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity/finalize.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 588e47cb75..85dfc22e06 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -850,12 +850,23 @@ async fn dispatch_fallback( } ec.ec_context.decide_writer(&req, true, ec.is_real_browser); let is_publisher_navigation = ec.is_real_browser && is_navigation; - if is_publisher_navigation - && let Err(err) = ec + if is_publisher_navigation { + // A new visitor's create carries the browser EIDs, so it stays + // the page load's one KV write. + if let Ok(partner_registry) = PartnerRegistry::from_config(&state.settings.ec.partners) + { + ec.ec_context.seed_generation_from_cookies( + &partner_registry, + ec.eids_cookie.as_deref(), + ec.sharedid_cookie.as_deref(), + ); + } + if let Err(err) = ec .ec_context .generate_if_needed(&state.settings, ec.kv_graph.as_ref()) - { - log::warn!("EC generation failed for publisher proxy: {err:?}"); + { + log::warn!("EC generation failed for publisher proxy: {err:?}"); + } } // Run the server-side auction with the configured creative- diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index 461c623a2e..fb34cf8a27 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -291,14 +291,22 @@ pub async fn handle_auction( // only strips on TCF/GDPR signals. This matches the publisher and // `/_ts/page-bids` paths, which also resolve client EIDs only when // `ec_id.is_some()`. + let body_eids = ec_id + .as_ref() + .and(body.eids.as_deref()) + .map(parse_client_eids) + .filter(|eids| !eids.is_empty()); + let client_eids_from_body = body_eids.is_some(); let client_eids = if ec_id.is_some() { - resolve_client_auction_eids( - body.eids.as_deref(), - extract_cookie_value(&http_req, COOKIE_TS_EIDS).as_deref(), - ) + body_eids.or_else(|| { + parse_cookie_auction_eids(extract_cookie_value(&http_req, COOKIE_TS_EIDS).as_deref()) + }) } else { None }; + ec_context + .identity_mut() + .set_client_eids_from_body(client_eids_from_body); // Carry the full request-local EID set to response finalization so KV // ingestion uses what this request actually sent, not just whatever fits // in the size-capped `ts-eids` cookie (see `ec::finalize::ec_finalize_response`). @@ -640,7 +648,7 @@ mod tests { } #[tokio::test] - async fn auction_endpoint_snapshot_is_reused_by_response_finalization() { + async fn auction_endpoint_snapshot_is_reused_and_finalize_writes_nothing() { // `/auction` loads the identity-graph row to resolve server-side EIDs. // Finalization runs afterwards on the same EC context and ingests // `ts-eids`/`sharedId` updates. Both must be served by a single billable @@ -729,10 +737,9 @@ mod tests { .get(&ec_id) .expect("should read store") .expect("row should exist"); - assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some("shared-cookie-id"), - "the sharedId update must still be ingested from the shared snapshot" + assert!( + stored.ids.is_empty(), + "/auction is a stager: the sharedId cookie waits for the next navigation" ); } @@ -776,7 +783,7 @@ mod tests { } #[tokio::test] - async fn auction_ingests_its_body_eids_and_not_a_cookie_only_source() { + async fn auction_stages_its_body_eids_and_not_a_cookie_only_source() { let settings = create_test_settings(); let mut orchestrator = AuctionOrchestrator::new(AuctionConfig { enabled: true, @@ -829,14 +836,18 @@ mod tests { ); assert_eq!( - stored_ids(&graph, &ec_id), + crate::ec::identity::test_support::staged_browser_entries(&response, &ec_id, ®istry), vec![("body.example.com".to_owned(), "body-uid".to_owned())], - "KV should learn the body's source and not the cookie-only source" + "the KV gap holds the body's source and not the cookie-only source" + ); + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "/auction writes nothing" ); } #[tokio::test] - async fn disabled_auction_ingests_nothing_from_the_cookie_but_keeps_shared_id() { + async fn disabled_auction_stages_and_writes_nothing() { let settings = create_test_settings(); let orchestrator = AuctionOrchestrator::new(AuctionConfig { enabled: false, @@ -876,15 +887,19 @@ mod tests { &mut response, ); - assert_eq!( - stored_ids(&graph, &ec_id), - vec![("sharedid.org".to_owned(), "shared-cookie-id".to_owned())], - "an /auction that resolved no client set should ingest only the sharedId cookie" + assert!( + crate::ec::identity::test_support::staged_browser_entries(&response, &ec_id, ®istry) + .is_empty(), + "an /auction that resolved no body set stages no KV gap" + ); + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "and writes nothing: the cookies wait for the next navigation" ); } #[test] - fn navigation_ingests_the_cookie_only_source() { + fn writer_navigation_queues_the_cookie_only_source() { // Control for the two tests above: the registry does match the // cookie-only source when the route reads the cookie. let settings = create_test_settings(); @@ -893,6 +908,7 @@ mod tests { let graph = seeded_graph(&ec_id); let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); ec_context.set_eid_sync_source(crate::ec::EidSyncSource::Navigation); + ec_context.identity_mut().set_writer(true); let cookie = cookie_only_ts_eids(); let mut response = http::Response::new(EdgeBody::empty()); @@ -907,14 +923,18 @@ mod tests { ); assert_eq!( - stored_ids(&graph, &ec_id), + crate::ec::identity::test_support::queued_fills(&ec_context), vec![(COOKIE_ONLY_SOURCE.to_owned(), "cookie-uid".to_owned())], - "a navigation should ingest the cookie's registered source" + "the writer navigation queues the cookie's registered source" + ); + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "the single flush runs after send, not in finalize" ); } #[tokio::test] - async fn auction_body_eids_reach_kv_even_when_the_ts_eids_cookie_is_absent() { + async fn auction_body_eids_are_staged_as_a_kv_gap_without_a_ts_eids_cookie() { // Regression test for #1184: `/auction` sends every EID it has in the // request body, but response finalization used to ingest identity // graph updates only from the `ts-eids` cookie — which the browser @@ -995,14 +1015,14 @@ mod tests { &mut response, ); - let (stored, _) = graph - .get(&ec_id) - .expect("should read store") - .expect("row should exist"); assert_eq!( - stored.ids.get("id5-sync.com").map(|id| id.uid.as_str()), - Some("ID5_from_body"), - "the body's EID must be ingested into KV without a ts-eids cookie" + crate::ec::identity::test_support::staged_browser_entries(&response, &ec_id, ®istry), + vec![("id5-sync.com".to_owned(), "ID5_from_body".to_owned())], + "the body's EID must be staged as a KV gap without a ts-eids cookie" + ); + assert!( + stored_ids(&graph, &ec_id).is_empty(), + "/auction never writes KV; the next navigation flushes the gap" ); } diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index c058afcee5..4611dc5201 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -16,9 +16,14 @@ use crate::settings::Settings; use super::EcContext; use super::cookies::{expire_ec_cookie, set_ec_cookie}; use super::generation::{generate_ec_id, is_valid_ec_id}; +use super::identity::finalize::{ + IdentityFinalize, after_rotation, emit_identity_cookies, open_identity_cookies, + phase1_read_needed, queue_writer_changes, reconcile_with_snapshot, rotation_removals, + stage_consent_loss, stage_kv_gap, stage_new_ec_cookie, +}; +use super::identity::mutation::{ApplyContext, apply as apply_mutation}; use super::kv::{ - CreateIfAbsentOutcome, EidCookieSyncOutcome, KvIdentityGraph, PartnerIdUpdate, - apply_partner_id_updates, + CreateIfAbsentOutcome, KvIdentityGraph, PartnerIdUpdate, apply_partner_id_updates, }; use super::kv_types::KvEntry; use super::prebid_eids::collect_eid_updates; @@ -56,6 +61,10 @@ pub fn ec_finalize_response( response: &mut Response, ) { ec_context.validate_pull_sync_marker(settings, registry); + let identity = IdentityFinalize::from_settings(settings, registry); + if let Some(identity) = &identity { + open_identity_cookies(ec_context, identity); + } let consent_allows_ec = ec_consent_granted(ec_context.consent()); let consent_withdrawn = ec_consent_withdrawn(ec_context.consent()); @@ -66,7 +75,7 @@ pub fn ec_finalize_response( expire_marker(ec_context.pull_sync_marker_mut(), response); } - finalize_unusable_consent( + let tombstoned = finalize_unusable_consent( settings, ec_context, kv, @@ -74,6 +83,12 @@ pub fn ec_finalize_response( consent_withdrawn, response, ); + if let Some(identity) = &identity { + // Removals stay in their cookies unless a tombstone now covers + // them; `__Host-ts-eids-s` never outlives usable consent. + stage_consent_loss(ec_context, tombstoned); + emit_identity_cookies(ec_context, identity, response); + } return; } @@ -90,37 +105,52 @@ pub fn ec_finalize_response( let had_eid_updates = !collected.is_empty(); let updates = stamp_acquired_at(gate_eid_updates_by_consent(collected, ec_context.consent())); - // `sync_eid_cookie_updates` early-returns without refreshing the - // snapshot when `updates` is empty, which is correct when there - // was never anything to write (an unconfigured registry, no sync - // source, or no EID cookies/body this request — the case - // `finalize_not_read_snapshot_does_not_rotate` covers). But when - // consent gating (e.g. TCF Purpose 4 denial) is what emptied - // `updates`, the request *did* have EID data, and orphan recovery - // below still needs an actual `Missing` read to detect an - // orphaned cookie. An unread `NotRead` snapshot proves nothing, so - // consent gating must not also skip that one read on a - // recovery-eligible request. - if updates.is_empty() - && had_eid_updates - && ec_context.recovery_eligible() - && matches!(ec_context.kv_snapshot(), EcKvSnapshot::NotRead) - { + let writer = ec_context.identity().writer(); + // The writer reads KV before send when its cookie needs + // reconciling or it has browser ingest to compare. A request whose + // consent gating emptied its updates still reads on a + // recovery-eligible navigation, because orphan recovery needs an + // actual `Missing` read: an unread snapshot proves nothing. + let needs_read = (writer + && (!updates.is_empty() + || identity + .as_ref() + .is_some_and(|identity| phase1_read_needed(ec_context, identity)))) + || (updates.is_empty() && had_eid_updates && ec_context.recovery_eligible()); + if needs_read && matches!(ec_context.kv_snapshot(), EcKvSnapshot::NotRead) { let snapshot = graph.load_snapshot(&ec_id); ec_context.set_kv_snapshot(snapshot); - } else if let Some(source) = source { - sync_eid_cookie_updates(graph, ec_context, &ec_id, &updates, source); } if matches!(ec_context.kv_snapshot(), EcKvSnapshot::Missing { .. }) && ec_context.recovery_eligible() { confirm_then_recover_orphaned_ec( - settings, ec_context, graph, &ec_id, &updates, response, + settings, + ec_context, + graph, + &ec_id, + &updates, + identity.as_ref(), + response, ); } + let rotated = ec_context.ec_value() != Some(ec_id.as_str()); + if let Some(identity) = &identity + && !rotated + { + reconcile_with_snapshot(ec_context, identity); + if writer { + queue_writer_changes(ec_context, identity, &updates); + } else { + stage_kv_gap(ec_context, identity, registry); + } + } } reconcile_pull_sync_marker(settings, registry, ec_context, response); + if let Some(identity) = &identity { + emit_identity_cookies(ec_context, identity, response); + } // Ordinary returning-user page views no longer refresh the browser // cookie, emit the EC header, or update KV TTL. @@ -129,28 +159,44 @@ pub fn ec_finalize_response( // Newly generated EC in this request. Do not emit a generated EC when // there is no KV graph: that would mint a browser cookie with no backing - // identity-graph row, producing a phantom ID on later requests. + // identity-graph row, producing a phantom ID on later requests. The + // create already carried this request's browser ingest, so finalize + // writes nothing more. if ec_context.ec_generated() { - let (Some(graph), Some(ec_id)) = (kv, ec_context.ec_value().map(str::to_owned)) else { + let (Some(_graph), Some(ec_id)) = (kv, ec_context.ec_value().map(str::to_owned)) else { log::info!("Skipping generated EC response write because KV graph is unavailable"); reconcile_pull_sync_marker(settings, registry, ec_context, response); return; }; - let (cookie_input, client_input) = browser_eid_input(ec_context, eids_cookie); - let updates = stamp_acquired_at(gate_eid_updates_by_consent( - collect_eid_updates(cookie_input, sharedid_cookie, client_input, registry), - ec_context.consent(), - )); - sync_eid_cookie_updates(graph, ec_context, &ec_id, &updates, EidSyncSource::NewEc); if ec_context.kv_snapshot().entry_for(&ec_id).is_some() { set_ec_cookie_on_response(settings, ec_context, response); + if let Some(identity) = &identity { + stage_new_ec_cookie(ec_context, identity); + } } else { log::warn!("Skipping generated EC cookie because backing row is not authoritative"); } } reconcile_pull_sync_marker(settings, registry, ec_context, response); + if let Some(identity) = &identity { + emit_identity_cookies(ec_context, identity, response); + } +} + +/// Browser EID updates for a new visitor's create, consent-gated and stamped +/// with the time they were seen. +pub(crate) fn browser_seed_updates( + registry: &PartnerRegistry, + consent: &ConsentContext, + eids_cookie: Option<&str>, + sharedid_cookie: Option<&str>, +) -> Vec { + stamp_acquired_at(gate_eid_updates_by_consent( + collect_eid_updates(eids_cookie, sharedid_cookie, None, registry), + consent, + )) } /// Records when this request saw each browser ID, so ingest stores `at`. @@ -184,72 +230,6 @@ fn browser_eid_input<'a>( } } -fn sync_eid_cookie_updates( - graph: &KvIdentityGraph, - ec_context: &mut EcContext, - ec_id: &str, - updates: &[PartnerIdUpdate], - source: EidSyncSource, -) { - if updates.is_empty() { - return; - } - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - ec_id, - updates, - ec_context.kv_snapshot().clone(), - ); - ec_context.set_kv_snapshot(snapshot); - record_eid_sync_terminal(source, outcome); -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -struct EidSyncMeasurement { - source: EidSyncSource, - outcome: EidCookieSyncOutcome, - already_matched: u8, - written: u8, - conflict_duplicate: u8, - deferred: u8, -} - -impl EidSyncMeasurement { - fn new(source: EidSyncSource, outcome: EidCookieSyncOutcome) -> Self { - Self { - source, - outcome, - already_matched: u8::from(matches!(outcome, EidCookieSyncOutcome::AlreadyMatched)), - written: u8::from(matches!( - outcome, - EidCookieSyncOutcome::Written | EidCookieSyncOutcome::WrittenWithDeferredFreshness - )), - conflict_duplicate: u8::from(matches!(outcome, EidCookieSyncOutcome::ConflictMatched)), - deferred: u8::from(matches!( - outcome, - EidCookieSyncOutcome::WrittenWithDeferredFreshness - | EidCookieSyncOutcome::DeferredConflict - | EidCookieSyncOutcome::DeferredFreshness - | EidCookieSyncOutcome::DeferredStaleRead - )), - } - } -} - -fn record_eid_sync_terminal(source: EidSyncSource, outcome: EidCookieSyncOutcome) { - let measurement = EidSyncMeasurement::new(source, outcome); - log::info!( - "EID sync measurement: source={} outcome={} attempted=1 already_matched={} written={} \ - conflict_duplicate={} deferred={}", - measurement.source, - measurement.outcome, - measurement.already_matched, - measurement.written, - measurement.conflict_duplicate, - measurement.deferred, - ); -} - fn reconcile_pull_sync_marker( settings: &Settings, registry: &PartnerRegistry, @@ -299,6 +279,7 @@ fn recover_orphaned_ec( ec_context: &mut EcContext, graph: &KvIdentityGraph, updates: &[super::kv::PartnerIdUpdate], + identity: Option<&IdentityFinalize>, response: &mut Response, ) { // Snapshot the orphaned ID once so every fail-closed exit binds the failed @@ -336,6 +317,12 @@ fn recover_orphaned_ec( .device_signals() .map(super::device::DeviceSignals::to_kv_device); apply_partner_id_updates(&mut entry, updates); + // The new row carries the request's removals, opt-outs, clears and + // invalidated-UID hashes, never positive staged IDs, so a deletion + // is never undone through the cookie. + if let Some(identity) = identity { + entry = carry_removals(ec_context, identity, entry); + } match graph.create_if_absent(&ec_id, &entry) { Ok(CreateIfAbsentOutcome::Written) => { @@ -346,6 +333,9 @@ fn recover_orphaned_ec( }; ec_context.replace_with_generated(ec_id, snapshot); set_ec_cookie_on_response(settings, ec_context, response); + if let Some(identity) = identity { + after_rotation(ec_context, identity); + } return; } Ok(CreateIfAbsentOutcome::AlreadyExists) => continue, @@ -365,6 +355,25 @@ fn recover_orphaned_ec( }); } +/// Applies the request's removal cookies to a rotation's new row. +fn carry_removals(ec_context: &EcContext, identity: &IdentityFinalize, entry: KvEntry) -> KvEntry { + let removals = rotation_removals(ec_context); + if removals.is_empty() { + return entry; + } + let applied = apply_mutation( + &removals, + &entry, + &ApplyContext { + now: identity.now, + keys: &identity.keys, + random: identity.random(), + catalog: &identity.catalog, + }, + ); + applied.map_or(entry, |applied| applied.entry) +} + /// Proves an orphaned cookie is genuinely absent before rotating it. /// /// Rotation abandons a year-lived identity graph and its accumulated EIDs, so @@ -389,18 +398,21 @@ fn confirm_then_recover_orphaned_ec( graph: &KvIdentityGraph, ec_id: &str, updates: &[super::kv::PartnerIdUpdate], + identity: Option<&IdentityFinalize>, response: &mut Response, ) { let confirmed = graph.load_snapshot(ec_id); match confirmed { EcKvSnapshot::Present { .. } => { - // The row became visible after the origin round trip: adopt it and - // merge any pending updates rather than rotating a valid identity. + // The row became visible after the origin round trip: adopt it + // rather than rotating a valid identity. Its pending updates join + // the writer's single mutation. ec_context.set_kv_snapshot(confirmed); - sync_eid_cookie_updates(graph, ec_context, ec_id, updates, EidSyncSource::Navigation); } EcKvSnapshot::Missing { .. } => match graph.key_exists_confirmed(ec_id) { - Ok(false) => recover_orphaned_ec(settings, ec_context, graph, updates, response), + Ok(false) => { + recover_orphaned_ec(settings, ec_context, graph, updates, identity, response); + } Ok(true) => { log::warn!( "Orphan EC recovery skipped: both point reads missed a row the store still \ @@ -476,11 +488,11 @@ fn finalize_unusable_consent( registry: &PartnerRegistry, consent_withdrawn: bool, response: &mut Response, -) { +) -> bool { clear_ec_headers_on_response(response, Some(registry)); if !(consent_withdrawn && ec_context.cookie_was_present()) { - return; + return false; } expire_ec_cookie(settings, response); @@ -490,6 +502,7 @@ fn finalize_unusable_consent( // The identity-graph tombstone is the authoritative withdrawal marker // for subsequent EC behavior. + let mut all_tombstoned = kv.is_some() && !ids_to_withdraw.is_empty(); if let Some(graph) = kv { apply_withdrawal_tombstones(&ids_to_withdraw, |ec_id| { // The graph hands back the post-withdrawal snapshot rather than @@ -508,6 +521,7 @@ fn finalize_unusable_consent( // consented. Report every failure, including the non-active cookie // ID whose outcome is not retained on the request context. if matches!(outcome, EcKvSnapshot::Failed { .. }) { + all_tombstoned = false; log::warn!( "EC withdrawal tombstone failed for '{}': the identity-graph row may \ still be live with consent granted", @@ -519,6 +533,7 @@ fn finalize_unusable_consent( } }); } + all_tombstoned } fn withdrawal_ec_ids(ec_context: &EcContext) -> HashSet { @@ -1272,7 +1287,7 @@ mod tests { } #[test] - fn finalize_navigation_routes_persist_returning_user_eid_updates() { + fn finalize_writer_navigation_queues_ingest_and_page_bids_does_not() { for (source, suffix, cookie_id) in [ (EidSyncSource::Navigation, "naveid", "navigation-cookie-id"), (EidSyncSource::PageBids, "spaeid", "page-bids-cookie-id"), @@ -1291,6 +1306,8 @@ mod tests { .expect("should seed the live row"); let mut ec_context = returning_user_context(&ec_id, graph.load_snapshot(&ec_id), true); ec_context.set_eid_sync_source(source); + let writer = source == EidSyncSource::Navigation; + ec_context.identity_mut().set_writer(writer); let partners = vec![make_partner("sharedid.org")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); let mut response = empty_response(); @@ -1309,68 +1326,22 @@ mod tests { .get(&ec_id) .expect("should read store") .expect("row should remain"); + assert!(stored.ids.is_empty(), "{source}: finalize never writes KV"); + let expected = if writer { + vec![("sharedid.org".to_owned(), cookie_id.to_owned())] + } else { + Vec::new() + }; assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some(cookie_id), - "{source} should persist the returning-user EID cookie" + crate::ec::identity::test_support::queued_fills(&ec_context), + expected, + "{source}: only the writer queues the returning-user EID cookie" ); } } #[test] - fn finalize_generated_ec_without_sync_source_ingests_the_ts_eids_cookie() { - let settings = create_test_settings(); - let ec_id = sample_ec_id("genck1"); - let graph = KvIdentityGraph::in_memory("test_store"); - let live = KvEntry::new( - &granting_consent(), - None, - current_timestamp(), - &settings.publisher.domain, - ); - graph - .create(&ec_id, &live) - .expect("should seed generated row"); - let mut ec_context = - make_context(Some(&ec_id), None, false, true, Jurisdiction::NonRegulated); - ec_context.set_kv_snapshot(graph.load_snapshot(&ec_id)); - let registry = PartnerRegistry::from_config(&[make_partner("ids.example.com")]) - .expect("should build registry"); - let eids_cookie = base64::engine::general_purpose::STANDARD.encode( - serde_json::to_vec(&serde_json::json!([ - { "source": "ids.example.com", "uids": [{ "id": "cookie-uid" }] } - ])) - .expect("should serialize test cookie payload"), - ); - let mut response = empty_response(); - - ec_finalize_response( - &settings, - &mut ec_context, - Some(&graph), - ®istry, - Some(&eids_cookie), - None, - &mut response, - ); - - let (stored, _) = graph - .get(&ec_id) - .expect("should read store") - .expect("row should remain"); - assert_eq!( - stored - .ids - .iter() - .map(|(source, id)| (source.as_str(), id.uid.as_str())) - .collect::>(), - vec![("ids.example.com", "cookie-uid")], - "a new EC with no sync source should ingest the cookie" - ); - } - - #[test] - fn finalize_generated_ec_persists_eid_updates() { + fn finalize_generated_ec_writes_nothing_more() { let settings = create_test_settings(); let ec_id = sample_ec_id("geneid"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1404,113 +1375,22 @@ mod tests { .get(&ec_id) .expect("should read store") .expect("row should remain"); - assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some("generated-cookie-id") - ); - } - - #[test] - fn eid_sync_measurement_dimensions_are_bounded_and_identity_free() { - let sources = [ - EidSyncSource::Navigation, - EidSyncSource::Auction, - EidSyncSource::PageBids, - EidSyncSource::NewEc, - ]; - let outcomes = [ - EidCookieSyncOutcome::AlreadyMatched, - EidCookieSyncOutcome::Written, - EidCookieSyncOutcome::WrittenWithDeferredFreshness, - EidCookieSyncOutcome::ConflictMatched, - EidCookieSyncOutcome::DeferredConflict, - EidCookieSyncOutcome::DeferredFreshness, - EidCookieSyncOutcome::DeferredStaleRead, - EidCookieSyncOutcome::Missing, - EidCookieSyncOutcome::ConsentWithdrawn, - EidCookieSyncOutcome::Failed, - ]; - - assert_eq!( - sources.map(|source| source.to_string()), - ["navigation", "auction", "page_bids", "new_ec"] - ); - assert_eq!( - outcomes.map(|outcome| outcome.to_string()), - [ - "already_matched", - "written", - "written_with_deferred_freshness", - "conflict_matched", - "deferred_conflict", - "deferred_freshness", - "deferred_stale_read", - "missing", - "consent_withdrawn", - "failed", - ] - ); - - assert_eq!( - EidSyncMeasurement::new( - EidSyncSource::Navigation, - EidCookieSyncOutcome::AlreadyMatched, - ), - EidSyncMeasurement { - source: EidSyncSource::Navigation, - outcome: EidCookieSyncOutcome::AlreadyMatched, - already_matched: 1, - written: 0, - conflict_duplicate: 0, - deferred: 0, - } - ); - assert_eq!( - EidSyncMeasurement::new( - EidSyncSource::Auction, - EidCookieSyncOutcome::WrittenWithDeferredFreshness, - ), - EidSyncMeasurement { - source: EidSyncSource::Auction, - outcome: EidCookieSyncOutcome::WrittenWithDeferredFreshness, - already_matched: 0, - written: 1, - conflict_duplicate: 0, - deferred: 1, - } - ); - assert_eq!( - EidSyncMeasurement::new(EidSyncSource::NewEc, EidCookieSyncOutcome::ConflictMatched,), - EidSyncMeasurement { - source: EidSyncSource::NewEc, - outcome: EidCookieSyncOutcome::ConflictMatched, - already_matched: 0, - written: 0, - conflict_duplicate: 1, - deferred: 0, - } + assert!( + stored.ids.is_empty(), + "the create carried any seed; finalize writes nothing more" ); - assert_eq!( - EidSyncMeasurement::new( - EidSyncSource::PageBids, - EidCookieSyncOutcome::DeferredStaleRead, - ), - EidSyncMeasurement { - source: EidSyncSource::PageBids, - outcome: EidCookieSyncOutcome::DeferredStaleRead, - already_matched: 0, - written: 0, - conflict_duplicate: 0, - deferred: 1, - } + assert!( + get_header_str(&response, "set-cookie") + .is_some_and(|value| value.starts_with("ts-ec=")), + "the new EC cookie is still set" ); } #[test] - fn finalize_auction_transient_miss_still_persists_eid_updates() { + fn finalize_auction_stale_miss_writes_nothing_and_marks_the_cookie() { // `/auction` saves its first lookup into the context and is never - // recovery eligible, so a stale miss there has no later chance to - // retry. Finalization must revalidate before dropping collected IDs. + // recovery eligible. A stale miss there writes nothing and marks the + // identity cookie, so the next writer navigation reads KV. let settings = create_test_settings(); let ec_id = sample_ec_id("named1"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1550,20 +1430,21 @@ mod tests { .get(&ec_id) .expect("should read store") .expect("row should remain"); - assert_eq!( - stored.ids.get("sharedid.org").map(|id| id.uid.as_str()), - Some("shared-cookie-id"), - "a stale endpoint miss must not suppress EID persistence" + assert!(stored.ids.is_empty(), "a stager never writes KV"); + assert!( + crate::ec::identity::test_support::staged_s_cookie(&response, &ec_id) + .is_some_and(|cookie| cookie.m && cookie.ids.is_empty()), + "a stale miss sets the minimal m cookie, so the next writer reads KV and finds the row" ); } #[test] - fn finalize_persists_every_configured_partner_from_client_eids_over_a_trimmed_cookie() { + fn finalize_stages_every_configured_partner_from_the_body_over_a_trimmed_cookie() { // Regression test for #1184: the `ts-eids` cookie only carries what // the browser could fit under its size cap, but `/auction` also // hands finalization the full EID set from the request body via - // `EcContext::set_client_eids`. That full set must land in KV even - // when the cookie alone would have dropped a configured partner. + // `EcContext::set_client_eids`. That full set must be staged for KV + // even when the cookie alone would have dropped a configured partner. let settings = create_test_settings(); let ec_id = sample_ec_id("cleids1"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1576,14 +1457,9 @@ mod tests { graph .create(&ec_id, &live) .expect("should seed the live row this request updates"); - let mut ec_context = returning_user_context( - &ec_id, - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - false, - ); + let mut ec_context = returning_user_context(&ec_id, graph.load_snapshot(&ec_id), false); ec_context.set_eid_sync_source(EidSyncSource::Auction); + ec_context.identity_mut().set_client_eids_from_body(true); // Only `id5-sync.com` "fit" in the (simulated) trimmed cookie; // `liveramp.com` was dropped by the browser's size cap. let eids_cookie = base64::engine::general_purpose::STANDARD.encode( @@ -1624,19 +1500,24 @@ mod tests { &mut response, ); + let mut staged = + crate::ec::identity::test_support::staged_browser_entries(&response, &ec_id, ®istry); + staged.sort(); + assert_eq!( + staged, + vec![ + ("id5-sync.com".to_owned(), "ID5_from_body".to_owned()), + ("liveramp.com".to_owned(), "LR_from_body".to_owned()), + ], + "the body's EIDs, including a partner the cookie trimmed, are staged as a KV gap" + ); let (stored, _) = graph .get(&ec_id) .expect("should read store") - .expect("row should exist after ingestion"); - assert_eq!( - stored.ids.get("id5-sync.com").map(|id| id.uid.as_str()), - Some("ID5_from_body"), - "the request body's EID should win over the cookie's stale value" - ); - assert_eq!( - stored.ids.get("liveramp.com").map(|id| id.uid.as_str()), - Some("LR_from_body"), - "a partner the cookie trimmed must still be ingested from the body" + .expect("row should exist"); + assert!( + stored.ids.is_empty(), + "the next writer navigation writes them" ); } @@ -2066,8 +1947,14 @@ mod tests { ); assert!(!ec_context.ec_generated(), "must not mark a rotated EC"); assert!( - get_header(response, "set-cookie").is_none(), - "must not emit a replacement cookie" + !response + .headers() + .get_all("set-cookie") + .iter() + .any(|value| value + .to_str() + .is_ok_and(|value| value.starts_with("ts-ec="))), + "must not emit a replacement EC cookie" ); } diff --git a/crates/trusted-server-core/src/ec/identity/catalog.rs b/crates/trusted-server-core/src/ec/identity/catalog.rs index f74b636701..2231f6af82 100644 --- a/crates/trusted-server-core/src/ec/identity/catalog.rs +++ b/crates/trusted-server-core/src/ec/identity/catalog.rs @@ -56,31 +56,55 @@ impl ProviderCatalog { .into_iter() .map(|provider| (provider.descriptor().id().clone(), provider)) .collect(); - let sources: Vec = providers + let mut catalog = Self { + providers, + owners, + sources: Vec::new(), + fingerprint: String::new(), + bases: BTreeMap::new(), + }; + catalog.index(BTreeSet::new()); + catalog + } + + /// Adds registered `[[ec.partners]]` sources to the source index, so a + /// browser KV-gap entry for any registered source can be staged. + #[must_use] + pub(crate) fn with_registered_sources( + mut self, + sources: impl IntoIterator, + ) -> Self { + let registered = sources.into_iter().collect(); + self.index(registered); + self + } + + /// Recomputes the sorted source index and `cf`. + fn index(&mut self, registered: BTreeSet) { + self.sources = self + .providers .values() .flat_map(|provider| provider.descriptor().sources().iter().cloned()) - .chain(owners.keys().cloned()) + .chain(self.owners.keys().cloned()) + .chain(registered) .collect::>() .into_iter() .collect(); let mut hasher = Sha256::new(); - for (id, provider) in &providers { + for source in &self.sources { + hasher.update(format!("source|{source}\n").as_bytes()); + } + for (id, provider) in &self.providers { for source in provider.descriptor().sources() { - let owner = owners + let owner = self + .owners .get(source) .map(IdentityProviderId::as_str) .unwrap_or_default(); hasher.update(format!("{id}|{source}|{owner}\n").as_bytes()); } } - let fingerprint = hex::encode(&hasher.finalize()[..4]); - Self { - providers, - owners, - sources, - fingerprint, - bases: BTreeMap::new(), - } + self.fingerprint = hex::encode(&hasher.finalize()[..4]); } /// The catalog for the partner registry: one generic pull provider per @@ -126,7 +150,10 @@ impl ProviderCatalog { ), })); } - let mut catalog = Self::new(providers, BTreeMap::new()); + let registered = registry + .all() + .filter_map(|partner| partner.source_domain.parse::().ok()); + let mut catalog = Self::new(providers, BTreeMap::new()).with_registered_sources(registered); catalog.bases = bases; Ok(catalog) } @@ -165,8 +192,9 @@ impl ProviderCatalog { self.owners.get(source) } - /// Every configured source, sorted. Indexes into this list are the `s` - /// values of `ts-eids-s`. + /// Every configured source, sorted: provider-claimed, owned and + /// registered partner sources. Indexes into this list are the `s` values + /// of `ts-eids-s`. #[must_use] pub(crate) fn configured_sources(&self) -> &[SourceDomain] { &self.sources diff --git a/crates/trusted-server-core/src/ec/identity/finalize.rs b/crates/trusted-server-core/src/ec/identity/finalize.rs new file mode 100644 index 0000000000..e22b711aeb --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/finalize.rs @@ -0,0 +1,946 @@ +//! Identity work at response finalize: reconciling the identity cookies with +//! a KV read, queueing the writer's changes, and setting the cookies. +//! +//! Returning visitors never write KV here. The writer request adds every +//! change to its single mutation, which the after-send flush applies. +//! Stagers save positive changes in `__Host-ts-eids-s` and removals in a new +//! `__Host-ts-eids-r-*` cookie. + +use edgezero_core::body::Body as EdgeBody; +use http::{HeaderValue, Response, header}; + +use super::catalog::ProviderCatalog; +use super::config::IdentitySettings; +use super::cookie::{ + CookieMergeContext, RBatch, RPlan, SCookie, SEntry, SOpen, Staging, cookie_view, + delete_cookie_header, encode_s_cookie, merge_with_kv, plan_r_batch, r_batch_applied, + r_batch_to_changes, set_cookie_header, split_r_batch, +}; +use super::coordinator::{EligibilityView, claimed_sources}; +use super::ids::{IdentityProviderId, SourceDomain}; +use super::input::{RandomSource, ThreadRandom}; +use super::mutation::{IdentityMutation, PartnerIdChange, ProviderStateChange}; +use super::policy::{PolicySignals, policy_allows}; +use super::revision::{IdentityKeys, Revision, invalidated_uid_hash}; +use crate::consent::allows_eid_persistence; +use crate::constants::COOKIE_TS_EIDS_S; +use crate::ec::EidSyncSource; +use crate::ec::kv::PartnerIdUpdate; +use crate::ec::kv_types::{KvEntry, KvPartnerId}; +use crate::ec::prebid_eids::collect_eid_updates; +use crate::ec::registry::PartnerRegistry; +use crate::ec::{EcContext, EcKvSnapshot}; +use crate::settings::Settings; + +/// What identity finalize needs besides the request: keys, the provider +/// catalog, the `[ec.identity]` settings, the time and a random source. +pub(crate) struct IdentityFinalize { + pub(crate) keys: IdentityKeys, + pub(crate) catalog: ProviderCatalog, + pub(crate) identity: IdentitySettings, + /// Unix seconds. + pub(crate) now: u64, + random: Box, +} + +impl IdentityFinalize { + /// Built from settings and the partner registry. `None` when the clock is + /// unavailable, because a zero time would make every expiry check pass. + #[must_use] + pub(crate) fn from_settings(settings: &Settings, registry: &PartnerRegistry) -> Option { + let now = crate::ec::checked_current_timestamp()?; + let catalog = ProviderCatalog::from_partners(registry).unwrap_or_else(|error| { + log::warn!( + "Identity providers unavailable: {}", + error.current_context() + ); + ProviderCatalog::empty() + }); + Some(Self { + keys: IdentityKeys::from_settings(settings), + catalog, + identity: settings.ec.identity.clone(), + now, + random: Box::new(ThreadRandom), + }) + } + + /// A deterministic context for tests. + #[cfg(test)] + #[must_use] + pub(crate) fn for_test(catalog: ProviderCatalog, now: u64) -> Self { + Self { + keys: super::test_support::test_keys(), + catalog, + identity: IdentitySettings::default(), + now, + random: Box::new(super::input::SeededRandom::new(1)), + } + } + + /// The random source. + #[must_use] + pub(crate) fn random(&self) -> &dyn RandomSource { + self.random.as_ref() + } +} + +/// Opens the identity cookies for the request's EC, once. +pub(crate) fn open_identity_cookies(context: &mut EcContext, finalize: &IdentityFinalize) { + let Some(ec_id) = context.ec_value().map(str::to_owned) else { + return; + }; + context + .identity_mut() + .open_cookies(&finalize.keys, &finalize.catalog, &ec_id); +} + +/// Whether the writer must read KV before send (spec 6.3, step 2): the +/// cookie is missing, unreadable, from another config, stale, marked +/// missing, holds staged entries or markers, or removal cookies are present. +#[must_use] +pub(crate) fn phase1_read_needed(context: &EcContext, finalize: &IdentityFinalize) -> bool { + let Some(opened) = context.identity().opened() else { + return false; + }; + if !opened.removals.cookies.is_empty() || !opened.removals.unopenable.is_empty() { + return true; + } + match &opened.s { + SOpen::Missing | SOpen::Failed | SOpen::ConfigChanged(_) => true, + SOpen::Valid(cookie) => { + cookie.m + || finalize.now.saturating_sub(cookie.kc) > finalize.identity.kv_revalidate_sec + || cookie.ids.iter().any(|entry| !entry.k && !entry.d) + || cookie.ps.values().any(|state| state.p.is_some()) + } + } +} + +/// A KV read answers a marker when every unowned candidate source of the +/// provider is now present, because the provider is then ineligible. +fn answered_by_present_sources(catalog: &ProviderCatalog) -> impl Fn(&str, &KvEntry) -> bool + '_ { + move |provider, entry| { + provider + .parse::() + .ok() + .and_then(|id| catalog.get(&id).map(|found| (id, found))) + .is_some_and(|(id, found)| { + found.descriptor().sources().iter().all(|source| { + catalog.owner_of(source) != Some(&id) && entry.ids.contains_key(source.as_str()) + }) + }) + } +} + +/// Reconciles the identity cookies with the request's KV read (spec 6.3, +/// step 3) and stages the cookie changes. A request that did not read KV +/// changes nothing. +pub(crate) fn reconcile_with_snapshot(context: &mut EcContext, finalize: &IdentityFinalize) { + let snapshot = context.kv_snapshot().clone(); + if matches!( + snapshot, + EcKvSnapshot::NotRead | EcKvSnapshot::Failed { .. } + ) { + return; + } + let Some(opened) = context.identity().opened().cloned() else { + return; + }; + let writer = context.identity().writer(); + let cookie = opened + .s + .cookie() + .cloned() + .unwrap_or_else(|| SCookie::new(&finalize.catalog, 0)); + let answered = answered_by_present_sources(&finalize.catalog); + let merge_context = CookieMergeContext { + catalog: &finalize.catalog, + keys: finalize.keys.clone(), + now: finalize.now, + kv_revalidate_sec: finalize.identity.kv_revalidate_sec, + answered_by_eligibility: &answered, + }; + let outcome = merge_with_kv(&cookie, &snapshot, &merge_context); + let staged = context.identity_mut().staged_mut(); + + if outcome.expire { + staged.expire_s = matches!( + opened.s, + SOpen::Valid(_) | SOpen::ConfigChanged(_) | SOpen::Failed + ); + staged.s_cookie = None; + staged.delete_r = opened + .removals + .cookies + .iter() + .map(|(name, _)| name.clone()) + .chain(opened.removals.unopenable.iter().cloned()) + .collect(); + return; + } + if outcome.changed || !matches!(opened.s, SOpen::Valid(_)) { + staged.s_cookie = Some(outcome.cookie); + } + if writer && let EcKvSnapshot::Present { entry, .. } = &snapshot { + for (name, batch) in &opened.removals.cookies { + if r_batch_applied(batch, entry, &finalize.keys) && !staged.delete_r.contains(name) { + staged.delete_r.push(name.clone()); + } + } + for name in &opened.removals.unopenable { + if !staged.delete_r.contains(name) { + staged.delete_r.push(name.clone()); + } + } + } +} + +/// Saves the `/auction` KV gap into `__Host-ts-eids-s` (spec 5.3): each +/// registered source in the consent-gated body set that the live row lacks +/// or holds only expired, unless its owner invalidated that UID. Nothing is +/// saved for a cookie-sourced client set, a failed or missing read, or a +/// tombstone. +pub(crate) fn stage_kv_gap( + context: &mut EcContext, + finalize: &IdentityFinalize, + registry: &PartnerRegistry, +) { + if context.eid_sync_source() != Some(EidSyncSource::Auction) + || !context.identity().client_eids_from_body() + || !allows_eid_persistence(context.consent()) + { + return; + } + let Some(ec_id) = context.ec_value() else { + return; + }; + let Some(entry) = context.kv_snapshot().entry_for(ec_id) else { + return; + }; + if !entry.consent.ok { + return; + } + let Some(eids) = context.client_eids() else { + return; + }; + let sources = finalize.catalog.configured_sources(); + let mut gap = Vec::new(); + for update in collect_eid_updates(None, None, Some(eids), registry) { + let Ok(source) = update.partner_id.parse::() else { + continue; + }; + let Some(index) = sources.iter().position(|known| *known == source) else { + continue; + }; + let filled = entry + .ids + .get(source.as_str()) + .is_some_and(|record| record.exp.is_none_or(|exp| finalize.now < exp)); + if filled { + continue; + } + let rejected = finalize.catalog.owner_of(&source).is_some_and(|owner| { + entry.ps.get(owner.as_str()).is_some_and(|state| { + state + .iv + .contains(&invalidated_uid_hash(&finalize.keys, owner, &update.uid)) + }) + }); + if !rejected { + gap.push(SEntry::staged_browser(index, &update.uid, finalize.now)); + } + } + if gap.is_empty() { + return; + } + + let base = context + .identity() + .staged() + .s_cookie + .clone() + .or_else(|| { + context + .identity() + .opened() + .and_then(|opened| opened.s.cookie().cloned()) + }) + .unwrap_or_else(|| SCookie::new(&finalize.catalog, finalize.now)); + let mut cookie = base; + for entry in gap { + let already = cookie + .ids + .iter() + .any(|existing| existing.s == entry.s && existing.is_browser_gap()); + if !already { + cookie.ids.push(entry); + } + } + context.identity_mut().staged_mut().s_cookie = Some(cookie); +} + +fn fill_missing_from_update(update: &PartnerIdUpdate) -> Option { + Some(PartnerIdChange::FillMissing { + source: update.partner_id.parse::().ok()?, + record: KvPartnerId::browser(update.uid.clone(), update.acquired_at), + writer: None, + staged_at: None, + }) +} + +/// The flush checks for one positive staged entry (spec 5.3). `Some` is the +/// change it becomes. +fn flush_checked_change( + context: &EcContext, + finalize: &IdentityFinalize, + view: &EligibilityView, + entry: (&SourceDomain, &KvPartnerId, &Staging), +) -> Option { + let (source, record, staging) = entry; + let Staging::Staged { pv, sa, browser } = staging else { + return None; + }; + let staged_at = sa.unwrap_or(0); + if finalize.now.saturating_sub(staged_at) > finalize.identity.staged_max_age_sec { + return None; + } + if !allows_eid_persistence(context.consent()) { + return None; + } + if *browser { + return Some(PartnerIdChange::FillMissing { + source: source.clone(), + record: record.clone(), + writer: None, + staged_at: *sa, + }); + } + + let writer = record.w.as_deref()?.parse::().ok()?; + let provider = finalize.catalog.get(&writer)?; + let settings = finalize + .catalog + .provider_settings(&finalize.identity, &writer); + if !claimed_sources(provider.descriptor(), &settings).contains(&source) { + return None; + } + let owner = finalize.catalog.owner_of(source); + if owner.is_some_and(|owner| owner != &writer) { + return None; + } + let signals = PolicySignals { + dnt: context.identity().dnt(), + }; + if !policy_allows( + provider.descriptor().consent_policy(), + context.consent(), + signals, + ) { + return None; + } + if view.is_opted_out(&writer) { + return None; + } + let mut record = record.clone(); + record.w = None; + record.rv = None; + Some(if owner == Some(&writer) { + PartnerIdChange::OwnerWrite { + source: source.clone(), + record, + writer, + expected: pv.as_deref().map(Revision::from_stored), + staged_at: *sa, + } + } else { + PartnerIdChange::FillMissing { + source: source.clone(), + record, + writer: Some(writer), + staged_at: *sa, + } + }) +} + +/// Adds the writer's changes to its single mutation: browser ingest, the +/// flush-checked staged entries of `__Host-ts-eids-s`, and every removal +/// batch. Removals skip every check but the EC binding (spec 5.6). +pub(crate) fn queue_writer_changes( + context: &mut EcContext, + finalize: &IdentityFinalize, + ingest: &[PartnerIdUpdate], +) { + let mut changes: Vec = + ingest.iter().filter_map(fill_missing_from_update).collect(); + let mut states: Vec<(IdentityProviderId, ProviderStateChange)> = Vec::new(); + + if let Some(opened) = context.identity().opened().cloned() { + let batches: Vec = opened + .removals + .cookies + .iter() + .map(|(_, batch)| batch.clone()) + .collect(); + let kv_entry = context + .ec_value() + .and_then(|ec_id| context.kv_snapshot().entry_for(ec_id)); + if let Some(cookie) = opened.s.cookie() { + let view = EligibilityView::merge( + kv_entry, + Some(cookie), + &batches, + &finalize.catalog, + &finalize.keys, + ); + for (source, record, staging) in cookie_view(cookie, &finalize.catalog) { + if let Some(change) = + flush_checked_change(context, finalize, &view, (&source, &record, &staging)) + { + changes.push(change); + } + } + for (provider, state) in &cookie.ps { + let Ok(provider) = provider.parse::() else { + continue; + }; + let staged = ProviderStateChange { + n: state.n.filter(|_| state.sg), + f: state.f.filter(|_| state.sg), + opt_out_clear: state.oc, + ..ProviderStateChange::default() + }; + if staged != ProviderStateChange::default() { + states.push((provider, staged)); + } + } + } + for batch in &batches { + let (removals, batch_states) = r_batch_to_changes(batch); + changes.extend(removals); + states.extend(batch_states); + } + } + + let mutation = context.identity_mut().mutation_mut(); + for change in changes { + mutation.add_change(change); + } + for (provider, state) in states { + mutation.merge_state(provider, state); + } +} + +/// The removals, opt-outs, clears and `iv` hashes of every removal cookie, +/// for an orphan rotation to carry into the new row. +#[must_use] +pub(crate) fn rotation_removals(context: &EcContext) -> IdentityMutation { + let mut mutation = IdentityMutation::default(); + if let Some(opened) = context.identity().opened() { + for (_, batch) in &opened.removals.cookies { + let (removals, states) = r_batch_to_changes(batch); + for change in removals { + mutation.add_change(change); + } + for (provider, state) in states { + mutation.merge_state(provider, state); + } + } + } + mutation.removals_only() +} + +/// After a rotation's create was written: the removal cookies are deleted, +/// because the new row carries them, and `__Host-ts-eids-s` is rebuilt from +/// the created row. The rest of the mutation and every marker are dropped. +pub(crate) fn after_rotation(context: &mut EcContext, finalize: &IdentityFinalize) { + let names: Vec = context + .identity() + .opened() + .map(|opened| { + opened + .removals + .cookies + .iter() + .map(|(name, _)| name.clone()) + .chain(opened.removals.unopenable.iter().cloned()) + .collect() + }) + .unwrap_or_default(); + let snapshot = context.kv_snapshot().clone(); + let never = |_: &str, _: &KvEntry| false; + let merge_context = CookieMergeContext { + catalog: &finalize.catalog, + keys: finalize.keys.clone(), + now: finalize.now, + kv_revalidate_sec: finalize.identity.kv_revalidate_sec, + answered_by_eligibility: &never, + }; + let rebuilt = merge_with_kv( + &SCookie::new(&finalize.catalog, finalize.now), + &snapshot, + &merge_context, + ) + .cookie; + let identity = context.identity_mut(); + *identity.mutation_mut() = IdentityMutation::default(); + let staged = identity.staged_mut(); + staged.s_cookie = Some(rebuilt); + staged.expire_s = false; + staged.delete_r = names; +} + +/// Stages `__Host-ts-eids-s` for a newly created row. +pub(crate) fn stage_new_ec_cookie(context: &mut EcContext, finalize: &IdentityFinalize) { + let snapshot = context.kv_snapshot().clone(); + let never = |_: &str, _: &KvEntry| false; + let merge_context = CookieMergeContext { + catalog: &finalize.catalog, + keys: finalize.keys.clone(), + now: finalize.now, + kv_revalidate_sec: finalize.identity.kv_revalidate_sec, + answered_by_eligibility: &never, + }; + let cookie = merge_with_kv( + &SCookie::new(&finalize.catalog, finalize.now), + &snapshot, + &merge_context, + ) + .cookie; + context.identity_mut().staged_mut().s_cookie = Some(cookie); +} + +/// Expires `__Host-ts-eids-s` when the request carried one, and on request +/// deletes every removal cookie, for a withdrawal or unusable consent. +pub(crate) fn stage_consent_loss(context: &mut EcContext, delete_removals: bool) { + let has_s = context.identity().raw_cookies().has_s(); + let names = context.identity().raw_cookies().r_names(); + let staged = context.identity_mut().staged_mut(); + staged.s_cookie = None; + staged.expire_s = has_s; + if delete_removals { + staged.delete_r = names; + } +} + +fn append_set_cookie(response: &mut Response, value: &str) { + match HeaderValue::from_str(value) { + Ok(value) => { + response.headers_mut().append(header::SET_COOKIE, value); + } + Err(_) => log::warn!("Dropping identity cookie that is not a valid header value"), + } +} + +/// Sets and deletes the identity cookies this response staged. The only +/// place `__Host-ts-eids-*` cookies are written. +pub(crate) fn emit_identity_cookies( + context: &mut EcContext, + finalize: &IdentityFinalize, + response: &mut Response, +) { + let ec_id = context.ec_value().map(str::to_owned); + let inbound_openable = context + .identity() + .opened() + .map_or(0, |opened| opened.removals.cookies.len()); + let staged = context.identity().staged().clone(); + + if staged.expire_s { + append_set_cookie(response, &delete_cookie_header(COOKIE_TS_EIDS_S)); + } else if let (Some(mut cookie), Some(ec_id)) = (staged.s_cookie, ec_id.as_deref()) { + let priority = |provider: &str| { + provider + .parse::() + .map(|id| { + finalize + .catalog + .provider_settings(&finalize.identity, &id) + .cookie_priority + }) + .unwrap_or_default() + }; + let value = encode_s_cookie( + &finalize.keys, + ec_id, + &mut cookie, + finalize.random(), + &priority, + ); + append_set_cookie(response, &set_cookie_header(COOKIE_TS_EIDS_S, &value)); + } + + if !staged.new_r_batch.is_empty() + && let Some(ec_id) = ec_id.as_deref() + { + let cookies = match plan_r_batch( + &finalize.keys, + ec_id, + &staged.new_r_batch, + inbound_openable, + finalize.random(), + ) { + RPlan::Cookie { name, value } => vec![(name, value)], + RPlan::Overflow => split_r_batch( + &finalize.keys, + ec_id, + &staged.new_r_batch, + finalize.random(), + ), + }; + for (name, value) in cookies { + append_set_cookie(response, &set_cookie_header(&name, &value)); + } + } + + for name in &staged.delete_r { + append_set_cookie(response, &delete_cookie_header(name)); + } +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use http::{HeaderMap, HeaderValue, header}; + + use super::*; + use crate::consent::jurisdiction::Jurisdiction; + use crate::consent::types::{ConsentContext, ConsentSource}; + use crate::ec::current_timestamp; + use crate::ec::finalize::ec_finalize_response; + use crate::ec::identity::cookie::{RRemoval, seal}; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::test_support::test_keys; + use crate::ec::kv::KvIdentityGraph; + use crate::settings::EcPartner; + use crate::test_support::tests::create_test_settings; + + fn ec_id() -> String { + format!("{}.ABC123", "a".repeat(64)) + } + + fn partner(source: &str) -> EcPartner { + EcPartner { + name: format!("Partner {source}"), + source_domain: source.to_owned(), + openrtb_atype: EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: None, + batch_rate_limit: EcPartner::default_batch_rate_limit(), + pull_sync_enabled: false, + pull_sync_url: None, + pull_sync_allowed_domains: vec![], + pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + ts_pull_token: None, + } + } + + fn registry() -> PartnerRegistry { + PartnerRegistry::from_config(&[partner("a.example.com"), partner("b.example.com")]) + .expect("should build registry") + } + + fn consent(jurisdiction: Jurisdiction) -> ConsentContext { + ConsentContext { + jurisdiction, + source: ConsentSource::Cookie, + ..ConsentContext::default() + } + } + + fn sealed_s(cookie: &SCookie) -> String { + seal( + &test_keys().cookie_s, + &ec_id(), + &serde_json::to_vec(cookie).expect("should serialize"), + &SeededRandom::new(5), + ) + } + + fn sealed_r(batch: &RBatch) -> String { + seal( + &test_keys().cookie_r, + &ec_id(), + &serde_json::to_vec(batch).expect("should serialize"), + &SeededRandom::new(6), + ) + } + + fn context_with(cookies: &str, writer: bool, jurisdiction: Jurisdiction) -> EcContext { + let mut context = EcContext::new_for_test(Some(ec_id()), consent(jurisdiction)); + let mut headers = HeaderMap::new(); + headers.insert( + header::COOKIE, + HeaderValue::from_str(cookies).expect("should build cookie header"), + ); + context.identity_mut().set_raw_cookies_for_test(&headers); + context.identity_mut().set_writer(writer); + context + } + + fn set_cookies(response: &Response) -> Vec { + response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok().map(str::to_owned)) + .collect() + } + + fn removal_batch(writer: &str) -> RBatch { + RBatch { + rm: vec![RRemoval { + s: "a.example.com".to_owned(), + w: writer.to_owned(), + pv: None, + sa: current_timestamp(), + }], + o: [("pull-b-example-com".to_owned(), current_timestamp())] + .into_iter() + .collect(), + ..RBatch::default() + } + } + + #[test] + fn flush_checks_drop_stale_and_unknown_writer_entries_but_never_removals() { + let settings = create_test_settings(); + let registry = registry(); + let catalog = ProviderCatalog::from_partners(®istry).expect("should build catalog"); + let now = current_timestamp(); + let mut cookie = SCookie::new(&catalog, now); + cookie.ids.push(SEntry::staged_browser(0, "fresh", now)); + cookie.ids.push(SEntry::staged_browser( + 1, + "stale", + now.saturating_sub(8 * 86_400), + )); + cookie.ids.push(SEntry::staged_provider( + 1, + &KvPartnerId { + w: Some("gone".to_owned()), + ..KvPartnerId::new("from-a-disabled-provider") + }, + None, + now, + )); + let cookies = format!( + "__Host-ts-eids-s={}; __Host-ts-eids-r-0a1b2c3d={}", + sealed_s(&cookie), + sealed_r(&removal_batch("gone")) + ); + let graph = KvIdentityGraph::in_memory("ec-store"); + graph + .create(&ec_id(), &KvEntry::minimal("z.example.com", "x", 1)) + .expect("should seed"); + let mut context = context_with(&cookies, true, Jurisdiction::NonRegulated); + context.set_kv_snapshot(graph.load_snapshot(&ec_id())); + let mut response = Response::new(EdgeBody::empty()); + + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + None, + None, + &mut response, + ); + + let changes = context.identity().mutation().changes(); + assert!( + changes.iter().any(|change| matches!( + change, + PartnerIdChange::FillMissing { record, .. } if record.uid == "fresh" + )), + "a fresh browser entry is flushed" + ); + assert!( + !changes.iter().any(|change| matches!( + change, + PartnerIdChange::FillMissing { record, .. } + if record.uid == "stale" || record.uid == "from-a-disabled-provider" + )), + "a stale entry and one from an unknown writer are dropped" + ); + assert!( + changes.iter().any(|change| matches!( + change, + PartnerIdChange::WriterRemove { writer, .. } if writer.as_str() == "gone" + )), + "a removal survives even for a writer that is no longer configured" + ); + assert!( + context.identity().mutation().has_removals(), + "the staged opt-out joins the mutation too" + ); + } + + #[test] + fn tombstone_read_expires_the_cookie_and_deletes_removal_cookies() { + let settings = create_test_settings(); + let registry = registry(); + let catalog = ProviderCatalog::from_partners(®istry).expect("should build catalog"); + let cookies = format!( + "__Host-ts-eids-s={}; __Host-ts-eids-r-0a1b2c3d={}", + sealed_s(&SCookie::new(&catalog, current_timestamp())), + sealed_r(&removal_batch("pull-a-example-com")) + ); + let mut context = context_with(&cookies, false, Jurisdiction::NonRegulated); + context.set_kv_snapshot(EcKvSnapshot::Present { + ec_id: ec_id(), + entry: Box::new(KvEntry::tombstone(1)), + generation: Some(1), + }); + let graph = KvIdentityGraph::in_memory("ec-store"); + let mut response = Response::new(EdgeBody::empty()); + + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + None, + None, + &mut response, + ); + + let cookies = set_cookies(&response); + assert!( + cookies + .iter() + .any(|value| value.starts_with("__Host-ts-eids-s=;") && value.contains("Max-Age=0")), + "expires ts-eids-s: {cookies:?}" + ); + assert!( + cookies + .iter() + .any(|value| value.starts_with("__Host-ts-eids-r-0a1b2c3d=;")), + "deletes the removal cookie the tombstone covers" + ); + } + + #[test] + fn stager_keeps_removal_cookies_and_sets_none_of_its_own_without_removals() { + let settings = create_test_settings(); + let registry = registry(); + let cookies = format!( + "__Host-ts-eids-r-0a1b2c3d={}", + sealed_r(&removal_batch("pull-a-example-com")) + ); + let graph = KvIdentityGraph::in_memory("ec-store"); + graph + .create(&ec_id(), &KvEntry::minimal("z.example.com", "x", 1)) + .expect("should seed"); + let mut context = context_with(&cookies, false, Jurisdiction::NonRegulated); + context.set_kv_snapshot(graph.load_snapshot(&ec_id())); + let mut response = Response::new(EdgeBody::empty()); + + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + None, + None, + &mut response, + ); + + assert!( + !set_cookies(&response) + .iter() + .any(|value| value.starts_with("__Host-ts-eids-r-")), + "a stager never deletes or adds a removal cookie without removals" + ); + assert!( + context.identity().mutation().is_empty(), + "a stager queues nothing" + ); + } + + #[test] + fn unusable_consent_expires_ts_eids_s_and_keeps_removals() { + let settings = create_test_settings(); + let registry = registry(); + let catalog = ProviderCatalog::from_partners(®istry).expect("should build catalog"); + let cookies = format!( + "__Host-ts-eids-s={}; __Host-ts-eids-r-0a1b2c3d={}", + sealed_s(&SCookie::new(&catalog, current_timestamp())), + sealed_r(&removal_batch("pull-a-example-com")) + ); + let mut context = context_with(&cookies, true, Jurisdiction::Unknown); + let graph = KvIdentityGraph::in_memory("ec-store"); + let mut response = Response::new(EdgeBody::empty()); + + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + None, + None, + &mut response, + ); + + let cookies = set_cookies(&response); + assert!( + cookies + .iter() + .any(|value| value.starts_with("__Host-ts-eids-s=;")), + "expires ts-eids-s: {cookies:?}" + ); + assert!( + !cookies + .iter() + .any(|value| value.starts_with("__Host-ts-eids-r-")), + "keeps the removal cookie until a writer applies it" + ); + } + + #[test] + fn orphan_rotation_carries_removals_but_never_positive_staged_ids() { + let registry = registry(); + let finalize = IdentityFinalize::for_test( + ProviderCatalog::from_partners(®istry).expect("should build catalog"), + current_timestamp(), + ); + let mut context = context_with( + &format!( + "__Host-ts-eids-r-0a1b2c3d={}", + sealed_r(&removal_batch("pull-a-example-com")) + ), + true, + Jurisdiction::NonRegulated, + ); + open_identity_cookies(&mut context, &finalize); + let removals = rotation_removals(&context); + assert!(removals.has_removals(), "carries the staged opt-out"); + assert!( + removals.changes().iter().all(|change| matches!( + change, + PartnerIdChange::WriterRemove { .. } | PartnerIdChange::OwnerRemove { .. } + )), + "and only removals" + ); + + let entry = KvEntry::minimal("z.example.com", "x", 1); + let applied = crate::ec::identity::mutation::apply( + &removals, + &entry, + &crate::ec::identity::mutation::ApplyContext { + now: finalize.now, + keys: &finalize.keys, + random: finalize.random(), + catalog: &finalize.catalog, + }, + ) + .expect("should apply"); + assert_eq!( + applied.entry.ps, + BTreeMap::new(), + "an opt-out of an unconfigured provider is pruned with its state" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 21a913bb79..7951f982e7 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -20,6 +20,7 @@ pub(crate) mod coordinator; pub mod descriptor; pub(crate) mod dispatch; pub mod exchange; +pub(crate) mod finalize; pub(crate) mod generic_pull; pub mod ids; pub mod input; diff --git a/crates/trusted-server-core/src/ec/identity/mutation.rs b/crates/trusted-server-core/src/ec/identity/mutation.rs index 2d56c63e10..85a1a53631 100644 --- a/crates/trusted-server-core/src/ec/identity/mutation.rs +++ b/crates/trusted-server-core/src/ec/identity/mutation.rs @@ -208,6 +208,13 @@ impl IdentityMutation { .any(ProviderStateChange::has_removal) } + /// The plain changes, for tests. + #[cfg(test)] + #[must_use] + pub(crate) fn changes(&self) -> &[PartnerIdChange] { + &self.partner_ids + } + /// Only the removals, opt-outs, clears and `iv` hashes. #[must_use] pub(crate) fn removals_only(&self) -> Self { diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index a26741c3e7..17c93bf42b 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -12,6 +12,7 @@ use super::cookie::{ use super::mutation::IdentityMutation; use super::revision::IdentityKeys; use crate::constants::{COOKIE_TS_EIDS_R_PREFIX, COOKIE_TS_EIDS_S}; +use crate::ec::kv::PartnerIdUpdate; /// The identity cookies' raw values, captured when the request is read and /// opened only by the routes that use identity. @@ -40,6 +41,18 @@ impl RawIdentityCookies { pub(crate) fn has_any(&self) -> bool { self.s.is_some() || !self.r.is_empty() } + + /// Whether the request carried `__Host-ts-eids-s`. + #[must_use] + pub(crate) fn has_s(&self) -> bool { + self.s.is_some() + } + + /// The names of every removal cookie the request carried. + #[must_use] + pub(crate) fn r_names(&self) -> Vec { + self.r.iter().map(|(name, _)| name.clone()).collect() + } } /// The identity cookies after opening them for this EC. @@ -73,6 +86,8 @@ pub(crate) struct RequestIdentity { opened: Option, mutation: IdentityMutation, staged: StagedCookies, + seed: Vec, + client_eids_from_body: bool, } impl RequestIdentity { @@ -89,6 +104,13 @@ impl RequestIdentity { } } + /// Replaces the captured cookies, for tests that build a context by hand. + #[cfg(test)] + pub(crate) fn set_raw_cookies_for_test(&mut self, headers: &HeaderMap) { + self.raw = RawIdentityCookies::capture(headers); + self.opened = None; + } + /// Whether this request is the page load's writer. #[must_use] pub(crate) fn writer(&self) -> bool { @@ -190,6 +212,27 @@ impl RequestIdentity { &self.staged } + /// Whether `/auction`'s client set came from its JSON body rather than + /// the `ts-eids` cookie. Only a body set is saved as a KV gap. + #[must_use] + pub(crate) fn client_eids_from_body(&self) -> bool { + self.client_eids_from_body + } + + pub(crate) fn set_client_eids_from_body(&mut self, from_body: bool) { + self.client_eids_from_body = from_body; + } + + /// Browser ingest a new visitor's create carries. + #[must_use] + pub(crate) fn seed(&self) -> &[PartnerIdUpdate] { + &self.seed + } + + pub(crate) fn set_seed(&mut self, seed: Vec) { + self.seed = seed; + } + /// What the response must do with the identity cookies, to change it. pub(crate) fn staged_mut(&mut self) -> &mut StagedCookies { &mut self.staged diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index d1d7c7f65c..8c49be0ac1 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -373,3 +373,67 @@ impl<'p> ProviderHarness<'p> { self.provider.response(&response, &input) } } + +/// The `__Host-ts-eids-s` cookie a response set, opened for `ec_id`. +pub(crate) fn staged_s_cookie( + response: &http::Response, + ec_id: &str, +) -> Option { + let prefix = format!("{}=", crate::constants::COOKIE_TS_EIDS_S); + let value = response + .headers() + .get_all(http::header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .find_map(|value| value.strip_prefix(&prefix)) + .and_then(|rest| rest.split(';').next()) + .filter(|value| !value.is_empty())?; + let payload = crate::ec::identity::cookie::open(&test_keys().cookie_s, ec_id, value) + .expect("should open the identity cookie"); + Some(serde_json::from_slice(&payload).expect("should decode the identity cookie")) +} + +/// The `(source, uid)` of every browser KV-gap entry in the +/// `__Host-ts-eids-s` cookie a response set, opened for `ec_id`. +pub(crate) fn staged_browser_entries( + response: &http::Response, + ec_id: &str, + registry: &crate::ec::registry::PartnerRegistry, +) -> Vec<(String, String)> { + let catalog = crate::ec::identity::catalog::ProviderCatalog::from_partners(registry) + .expect("should build the catalog"); + staged_s_cookie(response, ec_id) + .map(|cookie| { + cookie + .ids + .iter() + .filter(|entry| entry.is_browser_gap()) + .map(|entry| { + ( + catalog.configured_sources()[entry.s].to_string(), + entry.uid().unwrap_or_default().to_owned(), + ) + }) + .collect() + }) + .unwrap_or_default() +} + +/// The `(source, uid)` of every browser fill the writer queued. +pub(crate) fn queued_fills(context: &crate::ec::EcContext) -> Vec<(String, String)> { + context + .identity() + .mutation() + .changes() + .iter() + .filter_map(|change| match change { + crate::ec::identity::mutation::PartnerIdChange::FillMissing { + source, + record, + writer: None, + .. + } => Some((source.to_string(), record.uid.clone())), + _ => None, + }) + .collect() +} diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index a1b2028659..ef12bcbe3c 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -99,102 +99,6 @@ impl PartnerIdUpdate { } } -/// Terminal result of one browser EID-cookie persistence attempt. -#[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] -pub(crate) enum EidCookieSyncOutcome { - /// The stored values already matched without a write. - #[display("already_matched")] - AlreadyMatched, - /// The one conditional write succeeded. - #[display("written")] - Written, - /// The write added missing IDs while deferring different values of unknown freshness. - #[display("written_with_deferred_freshness")] - WrittenWithDeferredFreshness, - /// A conflicting writer persisted every desired value. - #[display("conflict_matched")] - ConflictMatched, - /// A conflict left at least one desired value absent or different. - #[display("deferred_conflict")] - DeferredConflict, - /// A different stored value had unknown freshness. - #[display("deferred_freshness")] - DeferredFreshness, - /// An eventually consistent read missed a row already proven to exist. - #[display("deferred_stale_read")] - DeferredStaleRead, - /// The identity graph row was missing. - #[display("missing")] - Missing, - /// Consent had been withdrawn in the identity graph. - #[display("consent_withdrawn")] - ConsentWithdrawn, - /// KV or serialization failed. - #[display("failed")] - Failed, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum CookieUpdateApplication { - AlreadyMatched, - Changed { deferred: bool }, - DeferredFreshness, -} - -fn apply_cookie_partner_id_updates( - entry: &mut KvEntry, - updates: &[PartnerIdUpdate], -) -> CookieUpdateApplication { - let mut latest_updates = BTreeMap::new(); - for update in updates { - latest_updates.insert(update.partner_id.as_str(), update); - } - - let mut changed = false; - let mut deferred = false; - for (partner_id, update) in latest_updates { - match entry.ids.get(partner_id) { - Some(existing) if existing.uid == update.uid => continue, - // Browser EID cookies carry no value-owned sequence or timestamp. - // A different value therefore has unknown freshness and must not - // replace the stored value. - Some(_) => { - deferred = true; - continue; - } - None => {} - } - - entry.ids.insert( - partner_id.to_owned(), - KvPartnerId::browser(update.uid.clone(), update.acquired_at), - ); - changed = true; - } - - if changed { - CookieUpdateApplication::Changed { deferred } - } else if deferred { - CookieUpdateApplication::DeferredFreshness - } else { - CookieUpdateApplication::AlreadyMatched - } -} - -fn partner_id_updates_match(entry: &KvEntry, updates: &[PartnerIdUpdate]) -> bool { - let mut latest_updates = BTreeMap::new(); - for update in updates { - latest_updates.insert(update.partner_id.as_str(), update.uid.as_str()); - } - - latest_updates.into_iter().all(|(partner_id, uid)| { - entry - .ids - .get(partner_id) - .is_some_and(|existing| existing.uid == uid) - }) -} - pub(crate) fn apply_partner_id_updates(entry: &mut KvEntry, updates: &[PartnerIdUpdate]) -> bool { let mut latest_updates = BTreeMap::new(); for update in updates { @@ -561,154 +465,6 @@ impl KvIdentityGraph { ))) } - /// Persists browser EID cookies with one conditional write and one conflict read. - /// - /// Browser cookies do not carry a value-owned version, so a different - /// existing value has unknown freshness and is always deferred. After a CAS - /// conflict this method never writes again. A live follow-up becomes the - /// authoritative snapshot; a missing or failed follow-up retains the live - /// pre-write snapshot as proof and defers the update. - pub(crate) fn sync_eid_cookie_updates_from_snapshot( - &self, - ec_id: &str, - updates: &[PartnerIdUpdate], - snapshot: EcKvSnapshot, - ) -> (EcKvSnapshot, EidCookieSyncOutcome) { - if updates.is_empty() { - return (snapshot, EidCookieSyncOutcome::AlreadyMatched); - } - - let proven = match snapshot { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - .. - } if snapshot_id == ec_id => Some(snapshot.clone()), - _ => None, - }; - - let current = match snapshot { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - generation: Some(_), - .. - } if snapshot_id == ec_id => snapshot, - EcKvSnapshot::Failed { - ec_id: ref snapshot_id, - } if snapshot_id == ec_id => return (snapshot, EidCookieSyncOutcome::Failed), - _ => self.load_snapshot(ec_id), - }; - - let (mut entry, generation) = match current { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - ref entry, - generation: Some(generation), - } if snapshot_id == ec_id => (entry.as_ref().clone(), generation), - EcKvSnapshot::Missing { .. } => { - let outcome = if proven.is_some() { - EidCookieSyncOutcome::DeferredStaleRead - } else { - EidCookieSyncOutcome::Missing - }; - let kept = Self::keep_proven(ec_id, current, proven.as_ref()); - return (kept, outcome); - } - EcKvSnapshot::Failed { .. } => { - let kept = Self::keep_proven(ec_id, current, proven.as_ref()); - return (kept, EidCookieSyncOutcome::Failed); - } - EcKvSnapshot::Present { .. } | EcKvSnapshot::NotRead => { - return ( - EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }, - EidCookieSyncOutcome::Failed, - ); - } - }; - - if !entry.consent.ok { - return (current, EidCookieSyncOutcome::ConsentWithdrawn); - } - let deferred_freshness = match apply_cookie_partner_id_updates(&mut entry, updates) { - CookieUpdateApplication::AlreadyMatched => { - return (current, EidCookieSyncOutcome::AlreadyMatched); - } - CookieUpdateApplication::DeferredFreshness => { - return (current, EidCookieSyncOutcome::DeferredFreshness); - } - CookieUpdateApplication::Changed { deferred } => deferred, - }; - - let Ok((body, meta_str)) = Self::serialize_entry(&entry, self.store_name()) else { - return ( - EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }, - EidCookieSyncOutcome::Failed, - ); - }; - match self.write_entry( - ec_id, - &body, - &meta_str, - ENTRY_TTL, - EcKvWriteMode::IfGenerationMatch(generation), - ) { - Ok(EcKvWriteOutcome::Written) => ( - EcKvSnapshot::Present { - ec_id: ec_id.to_owned(), - entry: Box::new(entry), - generation: None, - }, - if deferred_freshness { - EidCookieSyncOutcome::WrittenWithDeferredFreshness - } else { - EidCookieSyncOutcome::Written - }, - ), - Ok(EcKvWriteOutcome::PreconditionFailed) => { - let refreshed = self.load_snapshot(ec_id); - let Some(refreshed_entry) = refreshed.entry_for(ec_id) else { - // The failed CAS proved `current`'s generation stale. Keep - // only its existence proof so later writes must reread. - let proof = match current { - EcKvSnapshot::Present { - ec_id: proven_id, - entry, - .. - } => EcKvSnapshot::Present { - ec_id: proven_id, - entry, - generation: None, - }, - other => other, - }; - let kept = Self::keep_proven(ec_id, refreshed, Some(&proof)); - return (kept, EidCookieSyncOutcome::DeferredConflict); - }; - if !refreshed_entry.consent.ok { - return (refreshed, EidCookieSyncOutcome::ConsentWithdrawn); - } - let outcome = if partner_id_updates_match(refreshed_entry, updates) { - EidCookieSyncOutcome::ConflictMatched - } else { - EidCookieSyncOutcome::DeferredConflict - }; - (refreshed, outcome) - } - Err(err) => { - log::warn!("EID cookie sync write failed: {err:?}"); - ( - EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }, - EidCookieSyncOutcome::Failed, - ) - } - } - } - /// Merges partner IDs using request-scoped persisted state as the first CAS input. /// /// A caller-supplied `Present` snapshot is *proof* that the row exists — @@ -1643,25 +1399,6 @@ mod tests { ); } - #[test] - fn cookie_ingest_records_acquisition_time() { - let mut entry = KvEntry::minimal("sharedid.org", "existing", 1); - let updates = [PartnerIdUpdate::new("example.com", "new-uid").with_acquired_at(7)]; - - let outcome = apply_cookie_partner_id_updates(&mut entry, &updates); - - assert_eq!( - outcome, - CookieUpdateApplication::Changed { deferred: false }, - "should change" - ); - assert_eq!( - entry.ids["example.com"], - KvPartnerId::browser("new-uid", Some(7)), - "should store at" - ); - } - /// [`EcKvStore`] wrapper whose first CAS write both fails the precondition /// and deletes the key, simulating a concurrent withdrawal that removes the /// row between this writer's read and its write. @@ -1983,131 +1720,6 @@ mod tests { } } - /// Store that replaces the row during the first EID CAS write and records - /// the request's reads and conditional writes. - struct EidConflictEcKv { - inner: InMemoryEcKv, - concurrent_entry: KvEntry, - lookups: std::sync::Arc, - conditional_writes: std::sync::Arc, - follow_up_miss: bool, - miss_next_lookup: std::sync::atomic::AtomicBool, - } - - impl EidConflictEcKv { - fn new( - concurrent_entry: KvEntry, - lookups: std::sync::Arc, - conditional_writes: std::sync::Arc, - ) -> Self { - Self { - inner: InMemoryEcKv::new("eid-conflict-store"), - concurrent_entry, - lookups, - conditional_writes, - follow_up_miss: false, - miss_next_lookup: std::sync::atomic::AtomicBool::new(false), - } - } - - fn with_follow_up_miss(mut self) -> Self { - self.follow_up_miss = true; - self - } - - fn seed_live(&self, ec_id: &str) { - let (body, meta) = - KvIdentityGraph::serialize_entry(&live_entry(), self.inner.store_name()) - .expect("should serialize initial entry"); - self.inner - .insert( - ec_id, - EcKvWrite { - body: &body, - metadata: &meta, - ttl: ENTRY_TTL, - mode: EcKvWriteMode::Add, - }, - ) - .expect("should seed initial entry"); - } - } - - impl EcKvStore for EidConflictEcKv { - fn store_name(&self) -> &str { - self.inner.store_name() - } - - fn lookup(&self, key: &str) -> Result, Report> { - self.lookups - .fetch_add(1, std::sync::atomic::Ordering::Relaxed); - if self - .miss_next_lookup - .swap(false, std::sync::atomic::Ordering::Relaxed) - { - return Ok(None); - } - self.inner.lookup(key) - } - - fn key_exists(&self, key: &str) -> Result> { - self.inner.key_exists(key) - } - - fn insert( - &self, - key: &str, - write: EcKvWrite<'_>, - ) -> Result> { - if matches!(write.mode, EcKvWriteMode::IfGenerationMatch(_)) { - self.conditional_writes - .fetch_add(1, std::sync::atomic::Ordering::Relaxed); - let (body, meta) = KvIdentityGraph::serialize_entry( - &self.concurrent_entry, - self.inner.store_name(), - ) - .expect("should serialize concurrent entry"); - self.inner - .insert( - key, - EcKvWrite { - body: &body, - metadata: &meta, - ttl: ENTRY_TTL, - mode: EcKvWriteMode::Overwrite, - }, - ) - .expect("should write concurrent entry"); - if self.follow_up_miss { - self.miss_next_lookup - .store(true, std::sync::atomic::Ordering::Relaxed); - } - return Ok(EcKvWriteOutcome::PreconditionFailed); - } - self.inner.insert(key, write) - } - - fn list_keys_with_prefix( - &self, - prefix: &str, - limit: u32, - ) -> Result, Report> { - self.inner.list_keys_with_prefix(prefix, limit) - } - - fn count_keys_with_prefix( - &self, - prefix: &str, - limit: u32, - ) -> Result> { - self.inner.count_keys_with_prefix(prefix, limit) - } - - fn delete(&self, key: &str) -> Result<(), Report> { - self.inner.delete(key) - } - } - #[test] fn create_or_revive_retries_cas_conflict_and_succeeds() { let store = ConflictInjectingEcKv::new(2, false); @@ -2243,277 +1855,6 @@ mod tests { assert_eq!(entry.ids["ssp_x"].uid, "original"); } - #[test] - fn eid_cookie_sync_conflict_matching_value_stops_after_one_write() { - let ec_id = snapshot_ec_id(); - let mut concurrent = live_entry(); - concurrent.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId::new("desired-uid".to_owned()), - ); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - concurrent, - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - let snapshot = graph.load_snapshot(&ec_id); - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - snapshot, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::ConflictMatched); - assert_eq!( - snapshot - .entry_for(&ec_id) - .and_then(|entry| entry.ids.get("ssp_x")) - .map(|id| id.uid.as_str()), - Some("desired-uid"), - "the authoritative follow-up snapshot should replace stale request state" - ); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "a conflict should perform exactly one follow-up read" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "a conflict must not trigger another conditional write" - ); - } - - #[test] - fn eid_cookie_sync_keeps_live_proof_when_conflict_follow_up_misses() { - let ec_id = snapshot_ec_id(); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - live_entry(), - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ) - .with_follow_up_miss(); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::DeferredConflict); - assert!( - snapshot.entry_for(&ec_id).is_some(), - "the live pre-write row should prevent conflict deferral from entering orphan recovery" - ); - assert_eq!( - snapshot.generation_for(&ec_id), - None, - "a failed CAS must not return its rejected generation" - ); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "the initial refresh and conflict follow-up should be the only reads" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "the conflict must remain the request's only conditional write" - ); - } - - #[test] - fn eid_cookie_sync_reports_proven_refresh_miss_as_deferred() { - let ec_id = snapshot_ec_id(); - let graph = KvIdentityGraph::in_memory("empty-store"); - let proven = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: None, - }; - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - proven, - ); - - assert!( - snapshot.entry_for(&ec_id).is_some(), - "the add-confirmed row should survive an eventually consistent miss" - ); - assert_eq!( - outcome, - EidCookieSyncOutcome::DeferredStaleRead, - "a row already proven to exist should report a deferred stale read" - ); - } - - #[test] - fn partner_id_conflict_match_uses_last_duplicate_value() { - let mut entry = live_entry(); - entry.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId::new("latest-uid".to_owned()), - ); - let updates = [ - PartnerIdUpdate::new("ssp_x", "older-uid"), - PartnerIdUpdate::new("ssp_x", "latest-uid"), - ]; - - assert!( - partner_id_updates_match(&entry, &updates), - "conflict matching should use the same last-value-wins rule as writes" - ); - } - - #[test] - fn eid_cookie_sync_conflicting_value_defers_after_one_write() { - let ec_id = snapshot_ec_id(); - let mut concurrent = live_entry(); - concurrent.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId::new("concurrent-uid".to_owned()), - ); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - concurrent, - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - let snapshot = graph.load_snapshot(&ec_id); - - let (_, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "stale-uid")], - snapshot, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::DeferredConflict); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "a conflict should perform exactly one follow-up read" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "a conflict must not trigger another conditional write" - ); - let (stored, _) = graph - .get(&ec_id) - .expect("should read concurrent row") - .expect("row should remain"); - assert_eq!(stored.ids["ssp_x"].uid, "concurrent-uid"); - } - - #[test] - fn eid_cookie_sync_preserves_concurrent_withdrawal_after_conflict() { - let ec_id = snapshot_ec_id(); - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let writes = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let store = EidConflictEcKv::new( - KvEntry::tombstone(2_000), - std::sync::Arc::clone(&lookups), - std::sync::Arc::clone(&writes), - ); - store.seed_live(&ec_id); - let graph = KvIdentityGraph::new(store); - let snapshot = graph.load_snapshot(&ec_id); - - let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "desired-uid")], - snapshot, - ); - - assert_eq!(outcome, EidCookieSyncOutcome::ConsentWithdrawn); - assert!( - snapshot - .entry_for(&ec_id) - .is_some_and(|entry| !entry.consent.ok), - "the refreshed withdrawal tombstone must remain authoritative" - ); - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 2, - "a conflict should perform exactly one follow-up read" - ); - assert_eq!( - writes.load(std::sync::atomic::Ordering::Relaxed), - 1, - "a conflict must not trigger another conditional write" - ); - } - - #[test] - fn eid_cookie_sync_defers_different_values_without_value_owned_freshness() { - let ec_id = snapshot_ec_id(); - let graph = KvIdentityGraph::in_memory("freshness-store"); - let mut entry = live_entry(); - entry.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId::new("protected-uid".to_owned()), - ); - graph.create(&ec_id, &entry).expect("should seed entry"); - - let (_, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "incoming-uid")], - graph.load_snapshot(&ec_id), - ); - - assert_eq!(outcome, EidCookieSyncOutcome::DeferredFreshness); - let (stored, _) = graph - .get(&ec_id) - .expect("should read protected row") - .expect("row should remain"); - assert_eq!(stored.ids["ssp_x"].uid, "protected-uid"); - } - - #[test] - fn eid_cookie_sync_adds_missing_ids_while_deferring_different_values() { - let ec_id = snapshot_ec_id(); - let graph = KvIdentityGraph::in_memory("mixed-freshness-store"); - let mut entry = live_entry(); - entry.ids.insert( - "ssp_x".to_owned(), - crate::ec::kv_types::KvPartnerId::new("protected-uid".to_owned()), - ); - graph.create(&ec_id, &entry).expect("should seed entry"); - - let (_, outcome) = graph.sync_eid_cookie_updates_from_snapshot( - &ec_id, - &[ - PartnerIdUpdate::new("ssp_x", "different-uid"), - PartnerIdUpdate::new("ssp_y", "new-uid"), - ], - graph.load_snapshot(&ec_id), - ); - - assert_eq!(outcome, EidCookieSyncOutcome::WrittenWithDeferredFreshness); - let (stored, _) = graph - .get(&ec_id) - .expect("should read updated row") - .expect("row should remain"); - assert_eq!(stored.ids["ssp_x"].uid, "protected-uid"); - assert_eq!(stored.ids["ssp_y"].uid, "new-uid"); - } - #[test] fn cluster_size_returns_stored_value_without_store_io() { let kv = KvIdentityGraph::failing("nonexistent_store_for_cluster_cache_test"); diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index a10880dd62..ae03b3516d 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -425,6 +425,9 @@ impl EcContext { .device_signals .as_ref() .map(DeviceSignals::to_kv_device); + // The create carries the request's browser ingest, so it is the + // page load's one write. + kv::apply_partner_id_updates(&mut entry, self.identity.seed()); if let Some(graph) = kv { match graph.create_if_absent(&ec_id, &entry) { @@ -655,24 +658,33 @@ impl EcContext { } /// Request-scoped identity provider state. - #[allow( - dead_code, - reason = "identity finalize, which reads this, lands in a following commit" - )] #[must_use] pub(crate) fn identity(&self) -> &identity::request::RequestIdentity { &self.identity } /// Request-scoped identity provider state, to change it. - #[allow( - dead_code, - reason = "identity finalize, which writes this, lands in a following commit" - )] pub(crate) fn identity_mut(&mut self) -> &mut identity::request::RequestIdentity { &mut self.identity } + /// Prepares a new visitor's create to carry the browser EIDs of the + /// `ts-eids` and `sharedId` cookies, consent-gated. Call before + /// [`Self::generate_if_needed`]. + pub fn seed_generation_from_cookies( + &mut self, + registry: ®istry::PartnerRegistry, + eids_cookie: Option<&str>, + sharedid_cookie: Option<&str>, + ) { + if self.ec_value.is_some() { + return; + } + let seed = + finalize::browser_seed_updates(registry, &self.consent, eids_cookie, sharedid_cookie); + self.identity.set_seed(seed); + } + /// Decides whether this request is the page load's identity writer. /// /// Call once pre-route filters have allowed dispatch, where routing @@ -839,6 +851,58 @@ mod tests { .expect("should build request") } + #[test] + fn new_visitor_create_carries_the_browser_seed_in_one_write() { + let settings = create_test_settings(); + let (graph, counter) = kv::KvIdentityGraph::write_counting("ec-store"); + let consent = ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + source: ConsentSource::Cookie, + ..ConsentContext::default() + }; + let mut context = + EcContext::new_for_test_with_ip(None, consent, Some("192.0.2.10".to_owned())); + let registry = registry::PartnerRegistry::from_config(&[crate::settings::EcPartner { + name: "Shared".to_owned(), + source_domain: "sharedid.org".to_owned(), + openrtb_atype: crate::settings::EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: None, + batch_rate_limit: crate::settings::EcPartner::default_batch_rate_limit(), + pull_sync_enabled: false, + pull_sync_url: None, + pull_sync_allowed_domains: vec![], + pull_sync_ttl_sec: crate::settings::EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: crate::settings::EcPartner::default_pull_sync_rate_limit(), + ts_pull_token: None, + }]) + .expect("should build registry"); + + context.seed_generation_from_cookies(®istry, None, Some("shared-cookie-id")); + context + .generate_if_needed(&settings, Some(&graph)) + .expect("should generate"); + + let ec_id = context + .ec_value() + .expect("should hold the new EC") + .to_owned(); + let (stored, _) = graph + .get(&ec_id) + .expect("should read") + .expect("should exist"); + assert_eq!( + stored + .ids + .get("sharedid.org") + .map(|record| record.uid.as_str()), + Some("shared-cookie-id"), + "the create carries the sharedId cookie" + ); + assert_eq!(counter.writes().total(), 1, "one write: the create"); + assert_eq!(counter.writes().add, 1, "an Add"); + } + #[test] fn decide_writer_needs_every_condition() { let mut context = EcContext::default(); diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index e69b644597..537ef89457 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -111,6 +111,10 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/exchange.rs", include_str!("ec/identity/exchange.rs"), ), + ( + "ec/identity/finalize.rs", + include_str!("ec/identity/finalize.rs"), + ), ( "ec/identity/generic_pull.rs", include_str!("ec/identity/generic_pull.rs"), From 90d6663fdf38c11856f2c04135e4a06c58447a05 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:10:20 +0530 Subject: [PATCH 27/33] Write overflowing removal batches before send When a removal batch does not fit one cookie, or four removal cookies are already present, it is applied to KV with the withdrawal path's CAS retries, which retry only on a generation mismatch. If that write fails it is split across new cookies, never coarsened. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/ec/finalize.rs | 6 +- .../src/ec/identity/finalize.rs | 217 +++++++++++++++++- crates/trusted-server-core/src/ec/kv.rs | 74 ++++++ 3 files changed, 286 insertions(+), 11 deletions(-) diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index 4611dc5201..fc75527fce 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -87,7 +87,7 @@ pub fn ec_finalize_response( // Removals stay in their cookies unless a tombstone now covers // them; `__Host-ts-eids-s` never outlives usable consent. stage_consent_loss(ec_context, tombstoned); - emit_identity_cookies(ec_context, identity, response); + emit_identity_cookies(ec_context, identity, kv, response); } return; } @@ -149,7 +149,7 @@ pub fn ec_finalize_response( reconcile_pull_sync_marker(settings, registry, ec_context, response); if let Some(identity) = &identity { - emit_identity_cookies(ec_context, identity, response); + emit_identity_cookies(ec_context, identity, kv, response); } // Ordinary returning-user page views no longer refresh the browser @@ -181,7 +181,7 @@ pub fn ec_finalize_response( reconcile_pull_sync_marker(settings, registry, ec_context, response); if let Some(identity) = &identity { - emit_identity_cookies(ec_context, identity, response); + emit_identity_cookies(ec_context, identity, kv, response); } } diff --git a/crates/trusted-server-core/src/ec/identity/finalize.rs b/crates/trusted-server-core/src/ec/identity/finalize.rs index e22b711aeb..f299692854 100644 --- a/crates/trusted-server-core/src/ec/identity/finalize.rs +++ b/crates/trusted-server-core/src/ec/identity/finalize.rs @@ -19,13 +19,13 @@ use super::cookie::{ use super::coordinator::{EligibilityView, claimed_sources}; use super::ids::{IdentityProviderId, SourceDomain}; use super::input::{RandomSource, ThreadRandom}; -use super::mutation::{IdentityMutation, PartnerIdChange, ProviderStateChange}; +use super::mutation::{ApplyContext, IdentityMutation, PartnerIdChange, ProviderStateChange}; use super::policy::{PolicySignals, policy_allows}; use super::revision::{IdentityKeys, Revision, invalidated_uid_hash}; use crate::consent::allows_eid_persistence; use crate::constants::COOKIE_TS_EIDS_S; use crate::ec::EidSyncSource; -use crate::ec::kv::PartnerIdUpdate; +use crate::ec::kv::{KvIdentityGraph, PartnerIdUpdate, RemovalWriteOutcome}; use crate::ec::kv_types::{KvEntry, KvPartnerId}; use crate::ec::prebid_eids::collect_eid_updates; use crate::ec::registry::PartnerRegistry; @@ -532,11 +532,30 @@ fn append_set_cookie(response: &mut Response, value: &str) { } } +/// A removal batch as a removals-only mutation. +fn batch_mutation(batch: &RBatch) -> IdentityMutation { + let (removals, states) = r_batch_to_changes(batch); + let mut mutation = IdentityMutation::default(); + for change in removals { + mutation.add_change(change); + } + for (provider, state) in states { + mutation.merge_state(provider, state); + } + mutation.removals_only() +} + /// Sets and deletes the identity cookies this response staged. The only /// place `__Host-ts-eids-*` cookies are written. +/// +/// A removal batch that does not fit one cookie, or meets four removal +/// cookies already present, is written to KV before send. Only when that +/// write fails is it set in new cookies of its own, split but never +/// coarsened. pub(crate) fn emit_identity_cookies( context: &mut EcContext, finalize: &IdentityFinalize, + graph: Option<&KvIdentityGraph>, response: &mut Response, ) { let ec_id = context.ec_value().map(str::to_owned); @@ -581,12 +600,31 @@ pub(crate) fn emit_identity_cookies( finalize.random(), ) { RPlan::Cookie { name, value } => vec![(name, value)], - RPlan::Overflow => split_r_batch( - &finalize.keys, - ec_id, - &staged.new_r_batch, - finalize.random(), - ), + RPlan::Overflow => { + let outcome = graph.map_or(RemovalWriteOutcome::Failed, |graph| { + graph.apply_removals_now( + ec_id, + &batch_mutation(&staged.new_r_batch), + &ApplyContext { + now: finalize.now, + keys: &finalize.keys, + random: finalize.random(), + catalog: &finalize.catalog, + }, + ) + }); + match outcome { + RemovalWriteOutcome::Written | RemovalWriteOutcome::AlreadyApplied => { + Vec::new() + } + RemovalWriteOutcome::Failed => split_r_batch( + &finalize.keys, + ec_id, + &staged.new_r_batch, + finalize.random(), + ), + } + } }; for (name, value) in cookies { append_set_cookie(response, &set_cookie_header(&name, &value)); @@ -899,6 +937,169 @@ mod tests { ); } + fn big_batch(count: usize) -> RBatch { + RBatch { + rm: (0..count) + .map(|index| RRemoval { + s: format!("s{index}.example.com"), + w: "p".to_owned(), + pv: None, + sa: current_timestamp(), + }) + .collect(), + ..RBatch::default() + } + } + + fn row_with_records(count: usize) -> KvEntry { + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + for index in 0..count { + entry.ids.insert( + format!("s{index}.example.com"), + KvPartnerId { + w: Some("p".to_owned()), + at: Some(1), + ..KvPartnerId::new("u") + }, + ); + } + entry + } + + fn emit_with(graph: &KvIdentityGraph, cookies: &str, batch: RBatch) -> Response { + let registry = registry(); + let finalize = IdentityFinalize::for_test( + ProviderCatalog::from_partners(®istry).expect("should build catalog"), + current_timestamp(), + ); + let mut context = context_with(cookies, false, Jurisdiction::NonRegulated); + open_identity_cookies(&mut context, &finalize); + context.identity_mut().staged_mut().new_r_batch = batch; + let mut response = Response::new(EdgeBody::empty()); + emit_identity_cookies(&mut context, &finalize, Some(graph), &mut response); + response + } + + fn removal_cookie_count(response: &Response) -> usize { + set_cookies(response) + .iter() + .filter(|value| value.starts_with("__Host-ts-eids-r-") && !value.contains("Max-Age=0")) + .count() + } + + #[test] + fn overflow_batch_is_written_before_send_with_no_cookie() { + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph + .create(&ec_id(), &row_with_records(12)) + .expect("should seed"); + let before = counter.writes(); + + let response = emit_with(&graph, "x=1", big_batch(12)); + + assert_eq!( + counter.writes().if_generation_match - before.if_generation_match, + 1, + "one conditional write before send" + ); + assert_eq!( + removal_cookie_count(&response), + 0, + "no removal cookie is set" + ); + let (stored, _) = graph + .get(&ec_id()) + .expect("should read") + .expect("should exist"); + assert_eq!( + stored.ids.len(), + 1, + "every named record is removed, the control kept" + ); + } + + #[test] + fn overflow_retries_only_on_generation_mismatch() { + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph + .create(&ec_id(), &row_with_records(12)) + .expect("should seed"); + counter.conflict_next(2); + let before = counter.writes(); + + let response = emit_with(&graph, "x=1", big_batch(12)); + + assert_eq!( + counter.writes().if_generation_match - before.if_generation_match, + 3, + "two conflicts, then the write lands" + ); + assert_eq!( + removal_cookie_count(&response), + 0, + "no removal cookie is set" + ); + } + + #[test] + fn overflow_write_failure_falls_back_to_split_cookies() { + let graph = KvIdentityGraph::failing("ec-store"); + + let response = emit_with(&graph, "x=1", big_batch(12)); + + assert!( + removal_cookie_count(&response) >= 2, + "the batch is split across new cookies when the write fails" + ); + assert!( + !set_cookies(&response) + .iter() + .any(|value| value.starts_with("__Host-ts-eids-s=")), + "no other cookie is rewritten" + ); + } + + #[test] + fn four_openable_removal_cookies_force_overflow() { + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph + .create(&ec_id(), &row_with_records(1)) + .expect("should seed"); + let existing = sealed_r(&big_batch(1)); + let cookies = (0..4) + .map(|index| format!("__Host-ts-eids-r-0000000{index}={existing}")) + .collect::>() + .join("; "); + let before = counter.writes(); + + let response = emit_with(&graph, &cookies, big_batch(1)); + + assert_eq!( + counter.writes().if_generation_match - before.if_generation_match, + 1, + "a small batch is written when four cookies are present" + ); + assert_eq!( + removal_cookie_count(&response), + 0, + "and adds no fifth cookie" + ); + } + + #[test] + fn a_small_batch_gets_a_cookie_of_its_own() { + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph + .create(&ec_id(), &row_with_records(1)) + .expect("should seed"); + let before = counter.writes(); + + let response = emit_with(&graph, "x=1", big_batch(1)); + + assert_eq!(counter.writes().total() - before.total(), 0, "no write"); + assert_eq!(removal_cookie_count(&response), 1, "one removal cookie"); + } + #[test] fn orphan_rotation_carries_removals_but_never_positive_staged_ids() { let registry = registry(); diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index ef12bcbe3c..458ec84783 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -125,6 +125,17 @@ pub(crate) fn apply_partner_id_updates(entry: &mut KvEntry, updates: &[PartnerId changed } +/// Outcome of [`KvIdentityGraph::apply_removals_now`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum RemovalWriteOutcome { + /// The removals were written. + Written, + /// The row already shows them applied, or holds nothing to remove. + AlreadyApplied, + /// The write failed; the caller keeps the batch in cookies. + Failed, +} + /// EC identity graph on top of the platform KV store primitives. /// /// Each EC ID (`{64hex}.{6alnum}`) maps to a JSON-encoded [`KvEntry`] @@ -1224,6 +1235,69 @@ impl KvIdentityGraph { .count_keys_with_prefix(hash_prefix, CLUSTER_LIST_LIMIT) } + /// Writes a removal batch before send: the removal overflow write, the + /// second page-load exception to the one-write rule. + /// + /// Reads the row and applies only the batch's removals, opt-outs, clears + /// and invalidated-UID hashes. Like the withdrawal path it retries up to + /// five times, and only when the CAS generation no longer matches. Any + /// other failure, the per-key write limit included, ends at once. + pub(crate) fn apply_removals_now( + &self, + ec_id: &str, + removals: &super::identity::mutation::IdentityMutation, + context: &super::identity::mutation::ApplyContext<'_>, + ) -> RemovalWriteOutcome { + for attempt in 0..MAX_CAS_RETRIES { + let (entry, generation) = match self.get(ec_id) { + Ok(Some(found)) => found, + Ok(None) => return RemovalWriteOutcome::AlreadyApplied, + Err(error) => { + log::warn!( + "Removal overflow read failed for '{}': {}", + log_id(ec_id), + error.current_context() + ); + return RemovalWriteOutcome::Failed; + } + }; + let Some(applied) = super::identity::mutation::apply(removals, &entry, context) else { + return RemovalWriteOutcome::AlreadyApplied; + }; + if !applied.changed { + return RemovalWriteOutcome::AlreadyApplied; + } + let Ok((body, meta)) = Self::serialize_entry(&applied.entry, self.store_name()) else { + return RemovalWriteOutcome::Failed; + }; + match self.write_entry( + ec_id, + &body, + &meta, + ENTRY_TTL, + EcKvWriteMode::IfGenerationMatch(generation), + ) { + Ok(EcKvWriteOutcome::Written) => return RemovalWriteOutcome::Written, + Ok(EcKvWriteOutcome::PreconditionFailed) => { + log::debug!( + "Removal overflow CAS conflict on attempt {}/{MAX_CAS_RETRIES} for '{}'", + attempt + 1, + log_id(ec_id) + ); + } + Err(error) => { + log::warn!( + "Removal overflow write failed for '{}': {}", + log_id(ec_id), + error.current_context() + ); + return RemovalWriteOutcome::Failed; + } + } + } + RemovalWriteOutcome::Failed + } + /// Returns the cluster size for an entry, without writing it. /// /// Uses the stored `cluster_size` when the entry has one. Tombstone From be2a35bc9fd32a968062df96b157a99f37d6b319 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:14:22 +0530 Subject: [PATCH 28/33] Merge the server identity set into auction EIDs The server set is KV plus the __Host-ts-eids-s entries, staged ones included, with removed, expired and unservable records left out. A removal cookie hides its record before the removal reaches KV. Server values keep their stored atype and UID-level ext and gain the writing provider's EID-level provenance. Signed-off-by: dhruv8sh --- .../src/auction/endpoints.rs | 277 ++++++++++++++++-- crates/trusted-server-core/src/ec/eids.rs | 63 +++- .../src/ec/identity/coordinator.rs | 15 + .../src/ec/identity/request.rs | 76 +++-- crates/trusted-server-core/src/publisher.rs | 6 + 5 files changed, 373 insertions(+), 64 deletions(-) diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index fb34cf8a27..faf3baa3eb 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -15,11 +15,14 @@ use crate::cookies::extract_cookie_value; use crate::ec::EcContext; use crate::ec::EcKvSnapshot; use crate::ec::client_eids::parse_client_eids; -use crate::ec::eids::{resolve_partner_ids, to_eids}; +use crate::ec::eids::{resolve_partner_records, to_eids}; use crate::ec::identity::UnixTime; use crate::ec::identity::catalog::ProviderCatalog; +use crate::ec::identity::cookie::RBatch; +use crate::ec::identity::coordinator::EligibilityView; use crate::ec::identity::policy::PolicySignals; -use crate::ec::identity::servable::{ServeContext, kv_opted_out}; +use crate::ec::identity::revision::IdentityKeys; +use crate::ec::identity::servable::ServeContext; use crate::ec::kv::KvIdentityGraph; use crate::ec::partner::normalize_partner_source_domain; use crate::ec::prebid_eids::parse_prebid_eids_cookie; @@ -329,7 +332,7 @@ pub async fn handle_auction( if !matches!(auction_kv_snapshot, EcKvSnapshot::NotRead) { ec_context.set_kv_snapshot(auction_kv_snapshot.clone()); } - let eids = resolve_auction_eids(&auction_kv_snapshot, registry, ec_context); + let eids = resolve_auction_eids(settings, &auction_kv_snapshot, registry, ec_context); // Look up geo for device info. let geo = services @@ -454,6 +457,7 @@ pub async fn handle_auction( /// store, no EC, consent denied). On KV or partner-resolution errors, logs a /// warning and returns empty EIDs so the auction can proceed in degraded mode. pub(crate) fn resolve_auction_eids( + settings: &Settings, snapshot: &EcKvSnapshot, registry: Option<&PartnerRegistry>, ec_context: &EcContext, @@ -466,30 +470,49 @@ pub(crate) fn resolve_auction_eids( let ec_id = ec_context.ec_value()?; - let Some(entry) = snapshot.entry_for(ec_id) else { - return Some(Vec::new()); + // A read that found the row missing or failed yields no server set; a + // tombstone yields none either. Without a read the cookie view stands in. + let entry = match snapshot { + EcKvSnapshot::NotRead => None, + _ => match snapshot.entry_for(ec_id) { + Some(entry) if entry.consent.ok => Some(entry), + _ => return Some(Vec::new()), + }, }; - if !entry.consent.ok { - return Some(Vec::new()); - } - - // Without a trustworthy clock every expiry check would pass, so no KV ID - // is served. + // Without a trustworthy clock every expiry check would pass, so no + // server-side ID is served. let Some(now) = crate::ec::checked_current_timestamp() else { - log::warn!("Skipping KV EIDs because the clock is unavailable"); + log::warn!("Skipping server-side EIDs because the clock is unavailable"); return Some(Vec::new()); }; - let catalog = ProviderCatalog::empty(); - let opted_out = kv_opted_out(entry); + let keys = IdentityKeys::from_settings(settings); + let catalog = + ProviderCatalog::from_partners(registry).unwrap_or_else(|_| ProviderCatalog::empty()); + let opened = ec_context.identity().opened_or_open(&keys, &catalog, ec_id); + let batches: Vec = opened + .removals + .cookies + .iter() + .map(|(_, batch)| batch.clone()) + .collect(); + let view = EligibilityView::merge(entry, opened.s.cookie(), &batches, &catalog, &keys); + let opted_out = view.opted_out_providers(); let serve = ServeContext { now: UnixTime::from_secs(now), consent: ec_context.consent(), - signals: PolicySignals::default(), + signals: PolicySignals { + dnt: ec_context.identity().dnt(), + }, catalog: &catalog, opted_out: &opted_out, }; - let resolved = resolve_partner_ids(registry, entry, &serve); + let resolved = resolve_partner_records( + registry, + view.records() + .map(|(source, record)| (source.clone(), record)), + &serve, + ); Some(to_eids(&resolved)) } @@ -647,6 +670,193 @@ mod tests { } } + mod server_set { + use super::*; + use crate::ec::identity::cookie::{RRemoval, SCookie, SEntry, seal}; + use crate::ec::identity::input::SeededRandom; + use crate::ec::identity::test_support::test_keys; + use crate::ec::kv_types::{KvEntry, KvPartnerId}; + + const PULL_SOURCE: &str = "pull.example.com"; + + fn ec_id() -> String { + format!("{}.ABC123", "a".repeat(64)) + } + + fn registry() -> PartnerRegistry { + let mut pull = counting_test_partner(PULL_SOURCE); + pull.pull_sync_enabled = true; + pull.pull_sync_url = Some("https://sync.pull.example.com/pull".to_owned()); + pull.pull_sync_allowed_domains = vec!["sync.pull.example.com".to_owned()]; + pull.ts_pull_token = Some(crate::redacted::Redacted::new( + "example-token-value-0123456789abcdef".to_owned(), + )); + PartnerRegistry::from_config(&[ + counting_test_partner("a.example.com"), + counting_test_partner("b.example.com"), + pull, + ]) + .expect("should build registry") + } + + fn context_with_cookies(cookies: &str) -> EcContext { + let mut context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id())); + let mut headers = http::HeaderMap::new(); + if !cookies.is_empty() { + headers.insert( + header::COOKIE, + http::HeaderValue::from_str(cookies).expect("should build header"), + ); + } + context.identity_mut().set_raw_cookies_for_test(&headers); + context + } + + fn present(entry: KvEntry) -> EcKvSnapshot { + EcKvSnapshot::Present { + ec_id: ec_id(), + entry: Box::new(entry), + generation: Some(1), + } + } + + fn sources(eids: Option>) -> Vec<(String, String)> { + eids.unwrap_or_default() + .into_iter() + .map(|eid| (eid.source, eid.uids[0].id.clone())) + .collect() + } + + #[test] + fn staged_cookie_entry_fills_a_source_the_browser_did_not_send() { + let registry = registry(); + let catalog = crate::ec::identity::catalog::ProviderCatalog::from_partners(®istry) + .expect("should build catalog"); + let index = catalog + .configured_sources() + .iter() + .position(|source| source.as_str() == "b.example.com") + .expect("should index b"); + let mut cookie = SCookie::new(&catalog, crate::ec::current_timestamp()); + cookie + .ids + .push(SEntry::staged_browser(index, "staged-uid", 1)); + let sealed = seal( + &test_keys().cookie_s, + &ec_id(), + &serde_json::to_vec(&cookie).expect("should serialize"), + &SeededRandom::new(1), + ); + let context = context_with_cookies(&format!("__Host-ts-eids-s={sealed}")); + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry + .ids + .insert("a.example.com".to_owned(), KvPartnerId::new("server-a")); + + let server = resolve_auction_eids( + &create_test_settings(), + &present(entry), + Some(®istry), + &context, + ); + let client = Some(vec![Eid::new( + "a.example.com".to_owned(), + vec![Uid { + id: "browser-a".to_owned(), + atype: Some(1), + ext: None, + }], + )]); + let merged = merge_auction_eids(client, server); + + assert_eq!( + sources(merged), + vec![ + ("a.example.com".to_owned(), "browser-a".to_owned()), + ("b.example.com".to_owned(), "staged-uid".to_owned()), + ], + "the browser wins an unowned source, and a staged entry fills one it did not send" + ); + } + + #[test] + fn removal_cookie_hides_a_record_before_the_removal_reaches_kv() { + let registry = registry(); + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.ids.insert( + PULL_SOURCE.to_owned(), + KvPartnerId { + w: Some("pull-pull-example-com".to_owned()), + at: Some(5), + ..KvPartnerId::new("pull-uid") + }, + ); + let control = resolve_auction_eids( + &create_test_settings(), + &present(entry.clone()), + Some(®istry), + &context_with_cookies(""), + ); + assert_eq!( + sources(control), + vec![(PULL_SOURCE.to_owned(), "pull-uid".to_owned())], + "control: the pull partner's record is served" + ); + + let batch = crate::ec::identity::cookie::RBatch { + rm: vec![RRemoval { + s: PULL_SOURCE.to_owned(), + w: "pull-pull-example-com".to_owned(), + pv: None, + sa: 5, + }], + ..crate::ec::identity::cookie::RBatch::default() + }; + let sealed = seal( + &test_keys().cookie_r, + &ec_id(), + &serde_json::to_vec(&batch).expect("should serialize"), + &SeededRandom::new(2), + ); + let hidden = resolve_auction_eids( + &create_test_settings(), + &present(entry), + Some(®istry), + &context_with_cookies(&format!("__Host-ts-eids-r-0a1b2c3d={sealed}")), + ); + assert!( + sources(hidden).is_empty(), + "a staged removal stops the record at once" + ); + } + + #[test] + fn missing_or_tombstoned_rows_yield_no_server_set() { + let registry = registry(); + let context = context_with_cookies(""); + assert_eq!( + resolve_auction_eids( + &create_test_settings(), + &EcKvSnapshot::Missing { ec_id: ec_id() }, + Some(®istry), + &context + ), + Some(Vec::new()), + "a missing row" + ); + assert_eq!( + resolve_auction_eids( + &create_test_settings(), + &present(KvEntry::tombstone(1)), + Some(®istry), + &context + ), + Some(Vec::new()), + "a tombstone" + ); + } + } + #[tokio::test] async fn auction_endpoint_snapshot_is_reused_and_finalize_writes_nothing() { // `/auction` loads the identity-graph row to resolve server-side EIDs. @@ -1708,7 +1918,12 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &create_test_settings(), + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + ); assert!( result.is_some_and(|eids| eids.is_empty()), "should degrade to empty EIDs without a snapshot" @@ -1720,7 +1935,12 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, None, &ec_context); + let result = resolve_auction_eids( + &create_test_settings(), + &EcKvSnapshot::NotRead, + None, + &ec_context, + ); assert!( result.is_none(), "should return None when registry is missing" @@ -1733,7 +1953,12 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::Unknown, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &create_test_settings(), + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + ); assert!( result.is_none(), "should return None when consent is denied" @@ -1745,7 +1970,12 @@ mod tests { let registry = PartnerRegistry::empty(); let ec_context = make_ec_context(Jurisdiction::NonRegulated, None); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &create_test_settings(), + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + ); assert!( result.is_none(), "should return None when no EC value is present" @@ -1761,7 +1991,12 @@ mod tests { let snapshot = EcKvSnapshot::Failed { ec_id: ec_id.clone(), }; - let result = resolve_auction_eids(&snapshot, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &create_test_settings(), + &snapshot, + Some(®istry), + &ec_context, + ); let eids = result.expect("should return Some on KV error (degraded mode)"); assert!( eids.is_empty(), diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index b45f4dd2ed..98623b4a26 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -12,9 +12,9 @@ use error_stack::{Report, ResultExt as _}; use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; -use super::identity::SourceDomain; use super::identity::servable::{ServeContext, is_servable}; -use super::kv_types::KvEntry; +use super::identity::{EidProvenance, SourceDomain}; +use super::kv_types::{KvEntry, KvPartnerId}; use super::registry::PartnerRegistry; /// Maximum size (in bytes) for the base64-encoded `x-ts-eids` header value. @@ -32,6 +32,8 @@ pub struct ResolvedPartnerId { pub openrtb_atype: i32, /// Stored UID-level `ext`, when a provider supplied one. pub ext: Option, + /// EID-level fields the writing provider declared for this source. + pub provenance: Option, } /// Resolves source-domain keyed IDs from a KV entry against the partner registry. @@ -43,28 +45,46 @@ pub(crate) fn resolve_partner_ids( registry: &PartnerRegistry, entry: &KvEntry, serve: &ServeContext<'_>, +) -> Vec { + resolve_partner_records( + registry, + entry + .ids + .iter() + .filter_map(|(source, record)| Some((source.parse::().ok()?, record))), + serve, + ) +} + +/// Resolves any set of records, such as the merged server set of KV and the +/// identity cookies, the way [`resolve_partner_ids`] resolves a KV entry. +pub(crate) fn resolve_partner_records<'r>( + registry: &PartnerRegistry, + records: impl IntoIterator, + serve: &ServeContext<'_>, ) -> Vec { let mut resolved = Vec::new(); - for (source_domain, partner_uid) in &entry.ids { - let Some(partner) = registry.get(source_domain) else { - continue; - }; - if !partner.bidstream_enabled { - continue; - } - let Ok(source) = source_domain.parse::() else { + for (source, record) in records { + let Some(partner) = registry.get(source.as_str()) else { continue; }; - if !is_servable(serve, &source, partner_uid) { + if !partner.bidstream_enabled || !is_servable(serve, &source, record) { continue; } + let provenance = record + .w + .as_deref() + .and_then(|writer| writer.parse().ok()) + .and_then(|writer| serve.catalog.get(&writer)) + .and_then(|provider| provider.descriptor().provenance(&source).cloned()); resolved.push(ResolvedPartnerId { source_domain: partner.source_domain.clone(), - uid: partner_uid.uid.clone(), - openrtb_atype: partner_uid.atype.unwrap_or(partner.openrtb_atype), - ext: partner_uid.ext.clone(), + uid: record.uid.clone(), + openrtb_atype: record.atype.unwrap_or(partner.openrtb_atype), + ext: record.ext.clone(), + provenance, }); } @@ -78,14 +98,20 @@ pub fn to_eids(resolved: &[ResolvedPartnerId]) -> Vec { resolved .iter() .map(|item| { - Eid::new( + let mut eid = Eid::new( item.source_domain.clone(), vec![Uid { id: item.uid.clone(), atype: Some(item.openrtb_atype), ext: item.ext.clone(), }], - ) + ); + if let Some(provenance) = &item.provenance { + eid.inserter = provenance.inserter().map(str::to_owned); + eid.matcher = provenance.matcher().map(str::to_owned); + eid.mm = provenance.mm(); + } + eid }) .collect() } @@ -281,18 +307,21 @@ mod tests { source_domain: "liveramp.com".to_owned(), openrtb_atype: 3, ext: None, + provenance: None, }, ResolvedPartnerId { uid: "ID5_abc".to_owned(), source_domain: "id5-sync.com".to_owned(), openrtb_atype: 1, ext: None, + provenance: None, }, ResolvedPartnerId { uid: "pair-id".to_owned(), source_domain: "google.com".to_owned(), openrtb_atype: 571187, ext: None, + provenance: None, }, ]; @@ -323,6 +352,7 @@ mod tests { source_domain: format!("partner-{idx}.example.com"), openrtb_atype: 3, ext: None, + provenance: None, }); } @@ -343,6 +373,7 @@ mod tests { source_domain: "ssp.com".to_owned(), openrtb_atype: 3, ext: None, + provenance: None, }]; let (encoded, truncated) = diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index 98c54a09a0..749b0f282a 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -215,6 +215,21 @@ impl EligibilityView { self.states.get(provider) } + /// Every merged record, by source. + pub(crate) fn records(&self) -> impl Iterator { + self.records.iter() + } + + /// Providers with a sticky opt-out that is not cleared. + #[must_use] + pub(crate) fn opted_out_providers(&self) -> BTreeSet { + self.states + .iter() + .filter(|(_, state)| state.opted_out()) + .filter_map(|(provider, _)| provider.parse().ok()) + .collect() + } + /// Whether `provider` has a sticky opt-out that is not cleared. #[must_use] pub(crate) fn is_opted_out(&self, provider: &IdentityProviderId) -> bool { diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index 17c93bf42b..d06447957c 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -42,6 +42,40 @@ impl RawIdentityCookies { self.s.is_some() || !self.r.is_empty() } + /// Opens both cookie kinds for `ec_id`. + #[must_use] + pub(crate) fn open( + &self, + keys: &IdentityKeys, + catalog: &ProviderCatalog, + ec_id: &str, + ) -> OpenedIdentity { + let s = match &self.s { + None => SOpen::Missing, + Some(CookieRead::Value(value)) => { + let mut headers = HeaderMap::new(); + match http::HeaderValue::from_str(&format!("{COOKIE_TS_EIDS_S}={value}")) { + Ok(header) => { + headers.insert(http::header::COOKIE, header); + super::cookie::open_s_cookie(keys, ec_id, catalog, &headers) + } + Err(_) => SOpen::Failed, + } + } + Some(_) => SOpen::Failed, + }; + let mut removals = OpenedRemovals::default(); + for (name, value) in &self.r { + match open(&keys.cookie_r, ec_id, value) + .and_then(|payload| serde_json::from_slice::(&payload).ok()) + { + Some(batch) => removals.cookies.push((name.clone(), batch)), + None => removals.unopenable.push(name.clone()), + } + } + OpenedIdentity { s, removals } + } + /// Whether the request carried `__Host-ts-eids-s`. #[must_use] pub(crate) fn has_s(&self) -> bool { @@ -159,34 +193,22 @@ impl RequestIdentity { catalog: &ProviderCatalog, ec_id: &str, ) { - if self.opened.is_some() { - return; + if self.opened.is_none() { + self.opened = Some(self.raw.open(keys, catalog, ec_id)); } - let s = match &self.raw.s { - None => SOpen::Missing, - Some(raw) => { - let mut headers = HeaderMap::new(); - if let CookieRead::Value(value) = raw - && let Ok(header) = - http::HeaderValue::from_str(&format!("{COOKIE_TS_EIDS_S}={value}")) - { - headers.insert(http::header::COOKIE, header); - super::cookie::open_s_cookie(keys, ec_id, catalog, &headers) - } else { - SOpen::Failed - } - } - }; - let mut removals = OpenedRemovals::default(); - for (name, value) in &self.raw.r { - match open(&keys.cookie_r, ec_id, value) - .and_then(|payload| serde_json::from_slice::(&payload).ok()) - { - Some(batch) => removals.cookies.push((name.clone(), batch)), - None => removals.unopenable.push(name.clone()), - } - } - self.opened = Some(OpenedIdentity { s, removals }); + } + + /// The opened cookies, or the raw ones opened now without keeping them. + #[must_use] + pub(crate) fn opened_or_open( + &self, + keys: &IdentityKeys, + catalog: &ProviderCatalog, + ec_id: &str, + ) -> OpenedIdentity { + self.opened + .clone() + .unwrap_or_else(|| self.raw.open(keys, catalog, ec_id)) } /// The opened cookies, once [`Self::open_cookies`] ran. diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 14949fd0d2..4969ccc3e5 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -4931,6 +4931,7 @@ pub async fn handle_publisher_request( apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { + settings, cookie_jar: cookie_jar.as_ref(), ec_id, kv_snapshot: ec_context.kv_snapshot(), @@ -5527,6 +5528,7 @@ pub(crate) struct MatchedSlotsContext<'a> { /// Borrowed inputs for [`apply_auction_eids_and_device`], bundled to keep the /// helper within the project's 7-argument cap. struct AuctionEidTargeting<'a> { + settings: &'a Settings, cookie_jar: Option<&'a CookieJar>, ec_id: Option<&'a str>, kv_snapshot: &'a crate::ec::EcKvSnapshot, @@ -5558,6 +5560,7 @@ fn apply_auction_eids_and_device( None }; let kv_eids = resolve_auction_eids( + targeting.settings, targeting.kv_snapshot, targeting.partner_registry, targeting.ec_context, @@ -7106,6 +7109,7 @@ pub async fn handle_page_bids( apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { + settings, cookie_jar: cookie_jar.as_ref(), ec_id: ec_id.as_deref(), kv_snapshot: &page_bids_kv_snapshot, @@ -7948,9 +7952,11 @@ mod tests { auction_request.user.consent = None; auction_request.user.eids = None; + let settings = create_test_settings(); apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { + settings: &settings, cookie_jar: Some(&jar), ec_id: Some(ec_id.as_str()), kv_snapshot: &snapshot, From 57a545a366e26fd9265bfe42d8bd9b600a4e7803 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:26:00 +0530 Subject: [PATCH 29/33] Flush identity changes once after send and run pull partners there Phase 4 replaces the pull sync after-send hook. It reuses the phase 1 snapshot when no provider is marked, otherwise re-reads the row, dispatches the marked providers in an hourly rotation, and applies the whole mutation in one IfGenerationMatch write that is never retried. Pull partners now run only on a returning visitor's writer navigation, are marked pending in phase 3 after a read shows their source missing, and back off through ps. A read that shows a pull partner's sources present records a cookie-only next-due time, so the unowned read trigger does not fire on every navigation. The pull sync dispatcher, its bulk upsert and their tests are removed; the adapter builds a KV graph after send only when there is something to flush. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/app.rs | 19 +- .../trusted-server-adapter-fastly/src/main.rs | 90 +- crates/trusted-server-core/src/ec/finalize.rs | 12 +- .../src/ec/identity/finalize.rs | 491 +++++++- .../src/ec/identity/flush.rs | 714 +++++++++++ .../src/ec/identity/mod.rs | 1 + .../src/ec/identity/request.rs | 13 + crates/trusted-server-core/src/ec/kv.rs | 558 +-------- .../trusted-server-core/src/ec/pull_sync.rs | 1064 +---------------- .../src/migration_guards.rs | 1 + 10 files changed, 1321 insertions(+), 1642 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity/flush.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 85dfc22e06..bb31a56f60 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -325,9 +325,8 @@ pub(crate) struct EcFinalizeState { pub(crate) use_finalize_kv: bool, pub(crate) eids_cookie: Option, pub(crate) sharedid_cookie: Option, - pub(crate) is_real_browser: bool, - /// Per-request services carried to the entry point so the pull-sync - /// dispatcher can reuse the same platform HTTP client. + /// Per-request services carried to the entry point so the after-send + /// identity flush can reuse the same platform HTTP client. pub(crate) services: RuntimeServices, } @@ -361,7 +360,6 @@ impl EcRequestState { use_finalize_kv: self.finalize_kv_graph.is_some(), eids_cookie: self.eids_cookie, sharedid_cookie: self.sharedid_cookie, - is_real_browser: self.is_real_browser, services: self.services, } } @@ -735,8 +733,6 @@ async fn run_named_route( /// graph + partner registry + rate limiter, with a default EC context for /// response finalization. fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> Response { - let device_signals = device_signals_for(&req); - let is_real_browser = device_signals.looks_like_browser(); let eids_cookie = crate::extract_cookie_value(&req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(&req, COOKIE_SHAREDID); @@ -754,7 +750,6 @@ fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> use_finalize_kv: false, eids_cookie, sharedid_cookie, - is_real_browser, services: services.clone(), }); response @@ -2430,7 +2425,10 @@ mod tests { .get::() .expect("fallback response should carry EcFinalizeState"); assert!( - finalize.is_real_browser, + finalize + .ec_context + .device_signals() + .is_some_and(DeviceSignals::looks_like_browser), "browser-shaped JA4/H2 signals from the request extension must mark the request as a real browser" ); } @@ -2449,7 +2447,10 @@ mod tests { .get::() .expect("fallback response should carry EcFinalizeState"); assert!( - !finalize.is_real_browser, + !finalize + .ec_context + .device_signals() + .is_some_and(DeviceSignals::looks_like_browser), "a request without captured device signals should not be classified as a real browser" ); } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index b8f814a01c..8b7e2a80cf 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -13,12 +13,11 @@ use fastly::{Request as FastlyRequest, Response as FastlyResponse}; use trusted_server_core::cache_policy::{EdgeCacheHeader, cache_control_headers_have_directive}; use trusted_server_core::constants::HEADER_X_TS_VERSION; +use trusted_server_core::ec::EcContext; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; +use trusted_server_core::ec::identity::flush::{AfterSendInput, after_send_needed, run_after_send}; use trusted_server_core::ec::kv::KvIdentityGraph; -use trusted_server_core::ec::pull_sync::{ - PullSyncContext, build_pull_sync_context, dispatch_pull_sync, -}; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::RequestFilterEffects; @@ -509,12 +508,13 @@ fn edgezero_main(mut req: FastlyRequest, sandbox: &mut Sandbox, ordinal: u64, re if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &mut ec_state, &mut response) { Ok(partner_registry) => { + let status = response.status(); send_edgezero_response( response, request_filter_effects.as_ref(), counters.as_ref(), ); - run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); + run_identity_after_send(settings, &partner_registry, &ec_state, status); return; } Err(e) => { @@ -528,15 +528,17 @@ fn edgezero_main(mut req: FastlyRequest, sandbox: &mut Sandbox, ordinal: u64, re Ok(settings) => { match apply_edgezero_ec_finalize(&settings, &mut ec_state, &mut response) { Ok(partner_registry) => { + let status = response.status(); send_edgezero_response( response, request_filter_effects.as_ref(), counters.as_ref(), ); - run_edgezero_pull_sync_after_send( + run_identity_after_send( &settings, &partner_registry, &ec_state, + status, ); return; } @@ -615,49 +617,50 @@ fn apply_edgezero_ec_finalize( Ok(partner_registry) } -fn run_edgezero_pull_sync_after_send( +/// Phase 4: the writer request's single identity flush, after the response +/// is sent. The KV graph is built only when there is something to flush. +fn run_identity_after_send( settings: &Settings, partner_registry: &PartnerRegistry, ec_state: &EcFinalizeState, + status: edgezero_core::http::StatusCode, ) { - if !ec_state.is_real_browser { - return; - } - - let prepared_context = build_pull_sync_context(&ec_state.ec_context, partner_registry); - let Some((context, kv)) = - prepare_pull_sync_after_send(prepared_context, || require_identity_graph(settings)) - else { + let Some(kv) = prepare_identity_after_send(&ec_state.ec_context, status, || { + require_identity_graph(settings) + }) else { return; }; - let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); - dispatch_pull_sync( + let outcome = run_after_send(&AfterSendInput { settings, - &kv, - partner_registry, - &limiter, - &context, - &ec_state.services, - ); + registry: partner_registry, + services: &ec_state.services, + graph: &kv, + context: &ec_state.ec_context, + status, + rate_limiter: Some(&limiter), + }); + log::debug!("Identity flush outcome: {outcome}"); } -fn prepare_pull_sync_after_send( - context: Option, +fn prepare_identity_after_send( + context: &EcContext, + status: edgezero_core::http::StatusCode, graph_factory: F, -) -> Option<(PullSyncContext, KvIdentityGraph)> +) -> Option where F: FnOnce() -> Result>, { - let context = context?; - let kv = match graph_factory() { - Ok(kv) => kv, + if !after_send_needed(context, status) { + return None; + } + match graph_factory() { + Ok(kv) => Some(kv), Err(err) => { - log::debug!("Pull sync: identity graph unavailable, skipping: {err:?}"); - return None; + log::debug!("Identity flush: identity graph unavailable, skipping: {err:?}"); + None } - }; - Some((context, kv)) + } } /// Sends a finalized `EdgeZero` response to the client. @@ -878,19 +881,18 @@ mod tests { } #[test] - fn pull_sync_noop_states_skip_post_send_graph_factory() { + fn identity_noop_states_skip_post_send_graph_factory() { let calls = std::cell::Cell::new(0); - let result = prepare_pull_sync_after_send(None, || { - calls.set(calls.get() + 1); - Err(Report::new(TrustedServerError::KvStore { - store_name: "unexpected".to_owned(), - message: "graph factory should not run".to_owned(), - })) - }); - assert!( - result.is_none(), - "a skipped pull-sync plan should return none" - ); + let context = EcContext::default(); + let result = + prepare_identity_after_send(&context, edgezero_core::http::StatusCode::OK, || { + calls.set(calls.get() + 1); + Err(Report::new(TrustedServerError::KvStore { + store_name: "unexpected".to_owned(), + message: "graph factory should not run".to_owned(), + })) + }); + assert!(result.is_none(), "nothing to flush returns none"); assert_eq!(calls.get(), 0, "should not invoke the graph factory"); } diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index fc75527fce..91e21a15e9 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -17,9 +17,9 @@ use super::EcContext; use super::cookies::{expire_ec_cookie, set_ec_cookie}; use super::generation::{generate_ec_id, is_valid_ec_id}; use super::identity::finalize::{ - IdentityFinalize, after_rotation, emit_identity_cookies, open_identity_cookies, - phase1_read_needed, queue_writer_changes, reconcile_with_snapshot, rotation_removals, - stage_consent_loss, stage_kv_gap, stage_new_ec_cookie, + IdentityFinalize, after_rotation, emit_identity_cookies, mark_after_send, + open_identity_cookies, phase1_read_needed, queue_writer_changes, reconcile_with_snapshot, + rotation_removals, stage_consent_loss, stage_kv_gap, stage_new_ec_cookie, }; use super::identity::mutation::{ApplyContext, apply as apply_mutation}; use super::kv::{ @@ -141,6 +141,12 @@ pub fn ec_finalize_response( reconcile_with_snapshot(ec_context, identity); if writer { queue_writer_changes(ec_context, identity, &updates); + mark_after_send( + ec_context, + identity, + &settings.publisher.domain, + response.status(), + ); } else { stage_kv_gap(ec_context, identity, registry); } diff --git a/crates/trusted-server-core/src/ec/identity/finalize.rs b/crates/trusted-server-core/src/ec/identity/finalize.rs index f299692854..39dd9c0fc9 100644 --- a/crates/trusted-server-core/src/ec/identity/finalize.rs +++ b/crates/trusted-server-core/src/ec/identity/finalize.rs @@ -16,21 +16,32 @@ use super::cookie::{ delete_cookie_header, encode_s_cookie, merge_with_kv, plan_r_batch, r_batch_applied, r_batch_to_changes, set_cookie_header, split_r_batch, }; -use super::coordinator::{EligibilityView, claimed_sources}; +use super::coordinator::{ + Eligibility, EligibilityRequest, EligibilityView, Phase, RequestFacts, claimed_sources, + eligibility, +}; +use super::descriptor::{Input, ResolveKind}; +use super::ids::UnixTime; use super::ids::{IdentityProviderId, SourceDomain}; +use super::input::{ProviderConsent, Region, ResolveInput, ResolveSignals, StoredId}; use super::input::{RandomSource, ThreadRandom}; use super::mutation::{ApplyContext, IdentityMutation, PartnerIdChange, ProviderStateChange}; use super::policy::{PolicySignals, policy_allows}; +use super::provider::IdentityProvider; +use super::resolution::IdentityProviderError; use super::revision::{IdentityKeys, Revision, invalidated_uid_hash}; use crate::consent::allows_eid_persistence; use crate::constants::COOKIE_TS_EIDS_S; use crate::ec::EidSyncSource; +use crate::ec::consent::ec_consent_granted; use crate::ec::kv::{KvIdentityGraph, PartnerIdUpdate, RemovalWriteOutcome}; use crate::ec::kv_types::{KvEntry, KvPartnerId}; use crate::ec::prebid_eids::collect_eid_updates; use crate::ec::registry::PartnerRegistry; use crate::ec::{EcContext, EcKvSnapshot}; use crate::settings::Settings; +use std::collections::BTreeMap; +use std::sync::Arc; /// What identity finalize needs besides the request: keys, the provider /// catalog, the `[ec.identity]` settings, the time and a random source. @@ -95,6 +106,129 @@ pub(crate) fn open_identity_cookies(context: &mut EcContext, finalize: &Identity .open_cookies(&finalize.keys, &finalize.catalog, &ec_id); } +/// The inputs this request can supply to providers. +fn available_inputs(context: &EcContext) -> std::collections::BTreeSet { + let identity = context.identity(); + let mut inputs = std::collections::BTreeSet::new(); + if context.client_ip().is_some() { + inputs.insert(Input::ClientIp); + } + if identity.user_agent().is_some() { + inputs.insert(Input::UserAgent); + } + if identity.page_url().is_some() { + inputs.insert(Input::PageUrl); + } + if region_of(context).is_some() { + inputs.insert(Input::Region); + } + inputs +} + +fn region_of(context: &EcContext) -> Option { + let geo = context.geo_info()?; + Region::from_geo(&geo.country, geo.region.as_deref()) +} + +/// The facts of the writer request, for after-send eligibility. +#[must_use] +pub(crate) fn writer_facts(context: &EcContext, finalize: &IdentityFinalize) -> RequestFacts { + RequestFacts { + is_real_browser: context.identity().writer(), + ec_persistence_allowed: ec_consent_granted(context.consent()) + && allows_eid_persistence(context.consent()), + policy_signals: PolicySignals { + dnt: context.identity().dnt(), + }, + writer: context.identity().writer(), + new_visitor: context.ec_generated(), + runs_auction: false, + now: UnixTime::from_secs(finalize.now), + available_inputs: available_inputs(context), + } +} + +/// An after-send provider eligible in `phase`, with the sources to resolve. +pub(crate) struct Candidate { + pub(crate) provider: Arc, + pub(crate) kinds: BTreeMap, +} + +impl Candidate { + fn stateful(&self) -> bool { + self.provider + .descriptor() + .resolves() + .is_some_and(super::descriptor::Resolves::has_private_state) + } + + fn generic_pull(&self) -> bool { + self.provider + .descriptor() + .resolves() + .is_some_and(super::descriptor::Resolves::gets_ec_id) + } + + fn has_unowned_source(&self, catalog: &ProviderCatalog) -> bool { + let id = self.provider.descriptor().id(); + self.kinds + .keys() + .any(|source| catalog.owner_of(source) != Some(id)) + } +} + +/// Every provider eligible in `phase` on `view`. A valid pull-complete +/// marker leaves generic pull providers out, as it always has. +pub(crate) fn candidates( + context: &EcContext, + finalize: &IdentityFinalize, + view: &EligibilityView, + phase: Phase, +) -> Vec { + let facts = writer_facts(context, finalize); + let request = EligibilityRequest { + identity: &finalize.identity, + view, + facts: &facts, + consent: context.consent(), + catalog: &finalize.catalog, + keys: &finalize.keys, + }; + let pull_complete = context.pull_sync_marker().is_valid(); + finalize + .catalog + .providers() + .filter_map(|provider| { + let settings = finalize + .catalog + .provider_settings(&finalize.identity, provider.descriptor().id()); + match eligibility(provider.as_ref(), &settings, phase, &request) { + Eligibility::Eligible { kinds } => Some(Candidate { + provider: Arc::clone(provider), + kinds, + }), + Eligibility::Skip(_) => None, + } + }) + .filter(|candidate| !(pull_complete && candidate.generic_pull())) + .collect() +} + +fn opened_batches(context: &EcContext) -> Vec { + context + .identity() + .opened() + .map(|opened| { + opened + .removals + .cookies + .iter() + .map(|(_, batch)| batch.clone()) + .collect() + }) + .unwrap_or_default() +} + /// Whether the writer must read KV before send (spec 6.3, step 2): the /// cookie is missing, unreadable, from another config, stale, marked /// missing, holds staged entries or markers, or removal cookies are present. @@ -106,15 +240,74 @@ pub(crate) fn phase1_read_needed(context: &EcContext, finalize: &IdentityFinaliz if !opened.removals.cookies.is_empty() || !opened.removals.unopenable.is_empty() { return true; } - match &opened.s { - SOpen::Missing | SOpen::Failed | SOpen::ConfigChanged(_) => true, - SOpen::Valid(cookie) => { - cookie.m - || finalize.now.saturating_sub(cookie.kc) > finalize.identity.kv_revalidate_sec - || cookie.ids.iter().any(|entry| !entry.k && !entry.d) - || cookie.ps.values().any(|state| state.p.is_some()) + let cookie = match &opened.s { + SOpen::Missing | SOpen::Failed | SOpen::ConfigChanged(_) => return true, + SOpen::Valid(cookie) => cookie, + }; + if cookie.m + || finalize.now.saturating_sub(cookie.kc) > finalize.identity.kv_revalidate_sec + || cookie.ids.iter().any(|entry| !entry.k && !entry.d) + || cookie.ps.values().any(|state| state.p.is_some()) + { + return true; + } + // An after-send provider with an unowned candidate source needs KV for + // its dry run, because the cookie caches no unowned values. + let view = EligibilityView::merge( + None, + Some(cookie), + &opened_batches(context), + &finalize.catalog, + &finalize.keys, + ); + candidates(context, finalize, &view, Phase::AfterSendDryRun) + .iter() + .any(|candidate| !candidate.stateful() && candidate.has_unowned_source(&finalize.catalog)) +} + +/// Records, in the cookie only, that a read showed every unowned candidate +/// source of a provider present: `ps.n = now + refresh_after_sec`, with the +/// KV `n` it saw. Stops the unowned read trigger firing on every navigation. +fn apply_cookie_only_backoff( + cookie: &mut SCookie, + entry: &KvEntry, + finalize: &IdentityFinalize, +) -> bool { + let mut changed = false; + for provider in finalize.catalog.providers() { + let descriptor = provider.descriptor(); + let Some(resolves) = descriptor.resolves() else { + continue; + }; + if resolves.has_private_state() { + continue; } + let id = descriptor.id(); + let settings = finalize.catalog.provider_settings(&finalize.identity, id); + let unowned: Vec<&SourceDomain> = claimed_sources(descriptor, &settings) + .into_iter() + .filter(|source| finalize.catalog.owner_of(source) != Some(id)) + .collect(); + let all_present = !unowned.is_empty() + && unowned.iter().all(|source| { + entry + .ids + .get(source.as_str()) + .is_some_and(|record| record.exp.is_none_or(|exp| finalize.now < exp)) + }); + if !all_present { + continue; + } + let kv_n = entry.ps.get(id.as_str()).and_then(|state| state.n); + let state = cookie.ps.entry(id.to_string()).or_default(); + if state.n.is_some_and(|n| n > finalize.now) && state.kn == kv_n { + continue; + } + state.n = Some(finalize.now + settings.refresh_after_sec); + state.kn = kv_n; + changed = true; } + changed } /// A KV read answers a marker when every unowned candidate source of the @@ -179,8 +372,15 @@ pub(crate) fn reconcile_with_snapshot(context: &mut EcContext, finalize: &Identi .collect(); return; } - if outcome.changed || !matches!(opened.s, SOpen::Valid(_)) { - staged.s_cookie = Some(outcome.cookie); + let mut cookie = outcome.cookie; + let backed_off = match &snapshot { + EcKvSnapshot::Present { entry, .. } => { + apply_cookie_only_backoff(&mut cookie, entry, finalize) + } + _ => false, + }; + if outcome.changed || backed_off || !matches!(opened.s, SOpen::Valid(_)) { + staged.s_cookie = Some(cookie); } if writer && let EcKvSnapshot::Present { entry, .. } = &snapshot { for (name, batch) in &opened.removals.cookies { @@ -431,6 +631,148 @@ pub(crate) fn queue_writer_changes( } } +/// The signals a provider call may use, from the request context. +pub(crate) fn request_signals<'a>( + context: &'a EcContext, + finalize: &'a IdentityFinalize, + publisher_domain: &'a str, + client_ip: Option, +) -> ResolveSignals<'a> { + ResolveSignals { + client_ip, + user_agent: context.identity().user_agent(), + page_url: context.identity().page_url().cloned(), + region: region_of(context), + hashed_email: None, + consent: ProviderConsent::from_context(context.consent()), + publisher_domain, + now: UnixTime::from_secs(finalize.now), + random: finalize.random(), + ec_id: context.ec_value().map(str::to_owned), + stored: BTreeMap::new(), + state_json: None, + dry_run: false, + } +} + +/// A provider's stored records and private state in a row or view. +pub(crate) fn provider_records( + provider: &dyn IdentityProvider, + view: &EligibilityView, +) -> (BTreeMap, Option) { + let stored = provider + .descriptor() + .sources() + .iter() + .filter_map(|source| { + view.record(source) + .map(|record| (source.clone(), StoredId::from_record(record))) + }) + .collect(); + let state = view + .state(provider.descriptor().id().as_str()) + .and_then(|state| state.st.as_ref()) + .and_then(|st| serde_json::to_string(st).ok()); + (stored, state) +} + +/// Phase 3 on the writer (spec 5.2): dry-runs each eligible after-send +/// provider and marks the ones that would call, so phase 4 knows to re-read +/// the row. A provider with private state gets no dry run and is marked when +/// due. An unowned dry run needs a KV read that showed the source missing. +pub(crate) fn mark_after_send( + context: &mut EcContext, + finalize: &IdentityFinalize, + publisher_domain: &str, + status: http::StatusCode, +) { + let returning = context.ec_was_present() && !context.ec_generated(); + if !context.identity().writer() + || !returning + || !(status.is_success() || status == http::StatusCode::NOT_MODIFIED) + { + return; + } + let kv_read = context + .ec_value() + .and_then(|ec_id| context.kv_snapshot().entry_for(ec_id)) + .is_some(); + let cookie = context.identity().staged().s_cookie.clone().or_else(|| { + context + .identity() + .opened() + .and_then(|opened| opened.s.cookie().cloned()) + }); + let view = EligibilityView::merge( + context + .ec_value() + .and_then(|ec_id| context.kv_snapshot().entry_for(ec_id)), + cookie.as_ref(), + &opened_batches(context), + &finalize.catalog, + &finalize.keys, + ); + + let mut marked = Vec::new(); + let mut skipped = Vec::new(); + for candidate in candidates(context, finalize, &view, Phase::AfterSendDryRun) { + let id = candidate.provider.descriptor().id().clone(); + if candidate.stateful() { + marked.push(id); + continue; + } + if candidate.has_unowned_source(&finalize.catalog) && !kv_read { + continue; + } + let (stored, state) = provider_records(candidate.provider.as_ref(), &view); + let base = request_signals(context, finalize, publisher_domain, None); + let client_ip = context + .client_ip() + .and_then(|ip| ip.parse::().ok()); + let signals = ResolveSignals { + client_ip, + ..base.for_provider(stored, state, true) + }; + let input = ResolveInput::new(candidate.provider.descriptor(), candidate.kinds, &signals); + match candidate.provider.request(&input) { + Ok(Some(_)) => marked.push(id), + Ok(None) => skipped.push(id), + Err(error) + if matches!( + error.current_context(), + IdentityProviderError::InputUnavailable { .. } + ) => + { + skipped.push(id); + } + Err(error) => log::warn!( + "Identity provider {id} dry run failed: {}", + error.current_context() + ), + } + } + + for id in skipped { + let settings = finalize.catalog.provider_settings(&finalize.identity, &id); + context.identity_mut().mutation_mut().merge_state( + id, + ProviderStateChange { + n: Some(finalize.now + settings.refresh_after_sec), + ..ProviderStateChange::default() + }, + ); + } + if marked.is_empty() { + return; + } + let mut cookie = cookie.unwrap_or_else(|| SCookie::new(&finalize.catalog, finalize.now)); + for id in marked { + cookie.ps.entry(id.to_string()).or_default().p = Some(finalize.now); + context.identity_mut().mark(id); + } + context.identity_mut().staged_mut().s_cookie = Some(cookie); +} + /// The removals, opt-outs, clears and `iv` hashes of every removal cookie, /// for an orphan rotation to carry into the new row. #[must_use] @@ -1100,6 +1442,135 @@ mod tests { assert_eq!(removal_cookie_count(&response), 1, "one removal cookie"); } + fn pull_registry() -> PartnerRegistry { + let mut pull = partner("ssp.example.com"); + pull.pull_sync_enabled = true; + pull.pull_sync_url = Some("https://sync.ssp.example.com/pull".to_owned()); + pull.pull_sync_allowed_domains = vec!["sync.ssp.example.com".to_owned()]; + pull.ts_pull_token = Some(crate::redacted::Redacted::new( + "example-token-value-0123456789abcdef".to_owned(), + )); + PartnerRegistry::from_config(&[pull]).expect("should build registry") + } + + fn pull_finalize() -> IdentityFinalize { + IdentityFinalize::for_test( + ProviderCatalog::from_partners(&pull_registry()).expect("should build catalog"), + current_timestamp(), + ) + } + + #[test] + fn unowned_pull_partner_is_marked_only_after_a_read_shows_it_missing() { + let finalize = pull_finalize(); + let graph = KvIdentityGraph::in_memory("ec-store"); + graph + .create(&ec_id(), &KvEntry::minimal("z.example.com", "x", 1)) + .expect("should seed"); + + let mut unread = context_with("x=1", true, Jurisdiction::NonRegulated); + open_identity_cookies(&mut unread, &finalize); + mark_after_send( + &mut unread, + &finalize, + "www.example.com", + http::StatusCode::OK, + ); + assert!( + unread.identity().marked().is_empty(), + "without a read the cookie cannot know the source is missing" + ); + + let mut read = context_with("x=1", true, Jurisdiction::NonRegulated); + read.set_kv_snapshot(graph.load_snapshot(&ec_id())); + open_identity_cookies(&mut read, &finalize); + mark_after_send( + &mut read, + &finalize, + "www.example.com", + http::StatusCode::OK, + ); + assert_eq!( + read.identity() + .marked() + .iter() + .map(IdentityProviderId::as_str) + .collect::>(), + vec!["pull-ssp-example-com"], + "a read showing the source missing marks the pull partner" + ); + assert!( + read.identity() + .staged() + .s_cookie + .as_ref() + .is_some_and(|cookie| cookie.ps["pull-ssp-example-com"].p.is_some()), + "the marker goes into ts-eids-s" + ); + + let mut redirect = context_with("x=1", true, Jurisdiction::NonRegulated); + redirect.set_kv_snapshot(graph.load_snapshot(&ec_id())); + open_identity_cookies(&mut redirect, &finalize); + mark_after_send( + &mut redirect, + &finalize, + "www.example.com", + http::StatusCode::FOUND, + ); + assert!( + redirect.identity().marked().is_empty(), + "a 302 marks nothing" + ); + } + + #[test] + fn present_unowned_source_backs_off_the_read_trigger_in_the_cookie() { + let finalize = pull_finalize(); + let mut entry = KvEntry::minimal("ssp.example.com", "held", 1); + entry + .ids + .get_mut("ssp.example.com") + .expect("should hold it") + .at = Some(1); + let mut context = context_with("x=1", true, Jurisdiction::NonRegulated); + context.set_kv_snapshot(EcKvSnapshot::Present { + ec_id: ec_id(), + entry: Box::new(entry), + generation: Some(1), + }); + open_identity_cookies(&mut context, &finalize); + assert!( + phase1_read_needed(&context, &finalize), + "control: with no cookie the writer reads" + ); + + reconcile_with_snapshot(&mut context, &finalize); + let cookie = context + .identity() + .staged() + .s_cookie + .clone() + .expect("should stage the cookie"); + assert!( + cookie.ps["pull-ssp-example-com"] + .n + .is_some_and(|next| next > finalize.now), + "the cookie records when the partner is next due" + ); + + let sealed = sealed_s(&cookie); + let mut next = context_with( + &format!("__Host-ts-eids-s={sealed}"), + true, + Jurisdiction::NonRegulated, + ); + open_identity_cookies(&mut next, &finalize); + assert!( + !phase1_read_needed(&next, &finalize), + "the next navigation skips the read" + ); + } + #[test] fn orphan_rotation_carries_removals_but_never_positive_staged_ids() { let registry = registry(); diff --git a/crates/trusted-server-core/src/ec/identity/flush.rs b/crates/trusted-server-core/src/ec/identity/flush.rs new file mode 100644 index 0000000000..aed7d5c610 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/flush.rs @@ -0,0 +1,714 @@ +//! Phase 4: the writer request's single KV write, after the response is +//! sent. + +use std::collections::BTreeSet; + +use http::StatusCode; + +use super::coordinator::{ + CallOutcome, Eligibility, EligibilityRequest, EligibilityView, OutcomeContext, Phase, + eligibility, outcome_to_changes, +}; +use super::dispatch::{Call, DispatchRequest, IdentityDeps, dispatch}; +use super::finalize::{IdentityFinalize, provider_records, request_signals, writer_facts}; +use super::mutation::{ApplyContext, ProviderStateChange, apply}; +use super::policy::fingerprint_bits; +use super::revision::consent_fingerprint; +use crate::ec::kv::KvIdentityGraph; +use crate::ec::kv_backend::EcKvWriteMode; +use crate::ec::rate_limiter::RateLimiter; +use crate::ec::registry::PartnerRegistry; +use crate::ec::{EcContext, EcKvSnapshot, log_id}; +use crate::platform::RuntimeServices; +use crate::settings::Settings; + +/// Wait added for a marked provider that was not dispatched for a reason the +/// live row does not show, such as the burst cap. +const NOT_DISPATCHED_BACKOFF_SECS: u64 = 60; + +/// What the after-send flush needs. +pub struct AfterSendInput<'a> { + pub settings: &'a Settings, + pub registry: &'a PartnerRegistry, + pub services: &'a RuntimeServices, + pub graph: &'a KvIdentityGraph, + pub context: &'a EcContext, + /// The response status, captured before the response was sent. + pub status: StatusCode, + pub rate_limiter: Option<&'a dyn RateLimiter>, +} + +/// How the flush ended. +#[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] +pub enum FlushOutcome { + /// Nothing to do: not a returning writer, or nothing queued or marked. + #[display("skipped")] + Skipped, + /// The live row already held every change. + #[display("no_change")] + NoChange, + /// The one conditional write landed. + #[display("written")] + Written, + /// The mutation was dropped: the reason is the label. + #[display("dropped_{_0}")] + Dropped(&'static str), + /// The one write failed and is not retried. + #[display("write_failed")] + WriteFailed, +} + +fn returning_writer(context: &EcContext) -> bool { + context.identity().writer() && context.ec_was_present() && !context.ec_generated() +} + +fn status_allows_flush(status: StatusCode) -> bool { + status.is_success() || status == StatusCode::NOT_MODIFIED +} + +/// Whether the flush has anything to do. The adapter checks this before it +/// builds a KV graph, so most responses pay nothing after send. +#[must_use] +pub fn after_send_needed(context: &EcContext, status: StatusCode) -> bool { + returning_writer(context) + && status_allows_flush(status) + && (!context.identity().mutation().is_empty() || !context.identity().marked().is_empty()) +} + +/// Phase 4 (spec 5.2): reuse or re-read the row, dispatch the marked +/// after-send providers, and apply the whole mutation with one +/// `IfGenerationMatch` write that is never retried. +#[must_use] +pub fn run_after_send(input: &AfterSendInput<'_>) -> FlushOutcome { + let context = input.context; + if !returning_writer(context) { + return FlushOutcome::Skipped; + } + if !status_allows_flush(input.status) { + return FlushOutcome::Dropped("status"); + } + let marked = context.identity().marked(); + if context.identity().mutation().is_empty() && marked.is_empty() { + return FlushOutcome::Skipped; + } + let Some(ec_id) = context.ec_value() else { + return FlushOutcome::Skipped; + }; + let Some(finalize) = IdentityFinalize::from_settings(input.settings, input.registry) else { + return FlushOutcome::Dropped("clock"); + }; + + let reused = match context.kv_snapshot() { + EcKvSnapshot::Present { + entry, + generation: Some(generation), + .. + } if marked.is_empty() && context.kv_snapshot().belongs_to(ec_id) => { + Some(((**entry).clone(), *generation)) + } + _ => None, + }; + let (live, generation) = match reused { + Some(found) => found, + None => match input.graph.get(ec_id) { + Ok(Some(found)) => found, + Ok(None) => return FlushOutcome::Dropped("missing"), + Err(error) => { + log::warn!( + "Identity flush read failed for '{}': {}", + log_id(ec_id), + error.current_context() + ); + return FlushOutcome::Dropped("read_failed"); + } + }, + }; + if !live.consent.ok { + return FlushOutcome::Dropped("tombstone"); + } + + let mut mutation = context.identity().mutation().clone(); + if !marked.is_empty() { + dispatch_marked(input, &finalize, &live, &mut mutation); + } + if live + .network + .as_ref() + .and_then(|network| network.cluster_size) + .is_none() + { + match input.graph.cluster_size(ec_id, &live) { + Ok(Some(size)) => mutation.set_cluster_size(size), + Ok(None) => {} + Err(error) => log::debug!( + "Identity flush skipped cluster size: {}", + error.current_context() + ), + } + } + + let Some(applied) = apply( + &mutation, + &live, + &ApplyContext { + now: finalize.now, + keys: &finalize.keys, + random: finalize.random(), + catalog: &finalize.catalog, + }, + ) else { + return FlushOutcome::Dropped("invalid"); + }; + if !applied.changed { + return FlushOutcome::NoChange; + } + match input.graph.write_identity( + ec_id, + &applied.entry, + EcKvWriteMode::IfGenerationMatch(generation), + ) { + Ok(true) => FlushOutcome::Written, + Ok(false) => { + log::warn!( + "Identity flush lost a CAS race for '{}'; staged changes stay in the cookie", + log_id(ec_id) + ); + FlushOutcome::WriteFailed + } + Err(error) => { + log::warn!( + "Identity flush write failed for '{}': {}", + log_id(ec_id), + error.current_context() + ); + FlushOutcome::WriteFailed + } + } +} + +/// Rotates a sorted list by an hourly offset, so a different provider leads +/// each hour and `max_concurrency` stays fair, as pull sync always has. +fn rotate_hourly(items: &mut [T], now: u64) { + if items.is_empty() { + return; + } + let hours = usize::try_from(now / 3600).unwrap_or(usize::MAX); + items.rotate_left(hours % items.len()); +} + +/// Re-checks eligibility for each marked provider on the live row, so no EC +/// ID is disclosed before consent is confirmed, then dispatches them and adds +/// their results to the mutation. +fn dispatch_marked( + input: &AfterSendInput<'_>, + finalize: &IdentityFinalize, + live: &crate::ec::kv_types::KvEntry, + mutation: &mut super::mutation::IdentityMutation, +) { + let context = input.context; + let mut marked: Vec<_> = context + .identity() + .marked() + .iter() + .cloned() + .collect::>() + .into_iter() + .collect(); + rotate_hourly(&mut marked, finalize.now); + let batches: Vec<_> = context + .identity() + .opened() + .map(|opened| { + opened + .removals + .cookies + .iter() + .map(|(_, batch)| batch.clone()) + .collect() + }) + .unwrap_or_default(); + let view = EligibilityView::merge( + Some(live), + None, + &batches, + &finalize.catalog, + &finalize.keys, + ); + let facts = writer_facts(context, finalize); + let request = EligibilityRequest { + identity: &finalize.identity, + view: &view, + facts: &facts, + consent: context.consent(), + catalog: &finalize.catalog, + keys: &finalize.keys, + }; + + let mut calls = Vec::new(); + for id in &marked { + let Some(provider) = finalize.catalog.get(id) else { + continue; + }; + let settings = finalize.catalog.provider_settings(&finalize.identity, id); + let Eligibility::Eligible { kinds } = + eligibility(provider.as_ref(), &settings, Phase::AfterSend, &request) + else { + continue; + }; + let (stored, state_json) = provider_records(provider.as_ref(), &view); + calls.push(Call { + settings, + provider: id.clone(), + kinds, + stored, + state_json, + }); + } + if calls.is_empty() { + return; + } + + let deps = IdentityDeps { + settings: input.settings, + catalog: &finalize.catalog, + keys: &finalize.keys, + services: input.services, + rate_limiter: input.rate_limiter, + }; + let signals = request_signals( + context, + finalize, + &input.settings.publisher.domain, + input.services.client_info().client_ip, + ); + let reports = dispatch( + &DispatchRequest { + deps: &deps, + signals: &signals, + hold_budget: None, + visitor: context.ec_value(), + }, + calls, + ); + + for report in reports { + let Some(provider) = finalize.catalog.get(&report.provider) else { + continue; + }; + if report.outcome == CallOutcome::NotDispatched { + mutation.merge_state( + report.provider.clone(), + ProviderStateChange { + n: Some(finalize.now + NOT_DISPATCHED_BACKOFF_SECS), + ..ProviderStateChange::default() + }, + ); + continue; + } + let settings = finalize + .catalog + .provider_settings(&finalize.identity, &report.provider); + let bits = fingerprint_bits( + provider.descriptor().consent_policy(), + context.consent(), + facts.policy_signals, + ); + let changes = outcome_to_changes( + provider.as_ref(), + report.outcome, + &view, + &OutcomeContext { + now: finalize.now, + keys: &finalize.keys, + catalog: &finalize.catalog, + provider_settings: &settings, + identity: &finalize.identity, + expected_state: report.expected_state, + fingerprint: consent_fingerprint(&finalize.keys, &bits), + }, + ); + for change in changes.removals.into_iter().chain(changes.positive) { + mutation.add_change(change); + } + if let Some(result) = changes.result { + mutation.add_result(result); + } + mutation.merge_state(report.provider, changes.state); + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use error_stack::Report; + + use super::*; + use crate::consent::ConsentContext; + use crate::consent::jurisdiction::Jurisdiction; + use crate::consent::types::ConsentSource; + use crate::ec::identity::mutation::PartnerIdChange; + use crate::ec::identity::{IdentityProviderId, SourceDomain}; + use crate::ec::kv_types::{KvEntry, KvNetwork, KvPartnerId}; + use crate::error::TrustedServerError; + use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; + use crate::redacted::Redacted; + use crate::settings::EcPartner; + use crate::test_support::tests::create_test_settings; + + const PULL_SOURCE: &str = "ssp.example.com"; + const PULL_ID: &str = "pull-ssp-example-com"; + + fn ec_id() -> String { + format!("{}.ABC123", "a".repeat(64)) + } + + fn registry(pull: bool) -> PartnerRegistry { + let partner = EcPartner { + name: "SSP".to_owned(), + source_domain: PULL_SOURCE.to_owned(), + openrtb_atype: EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: None, + batch_rate_limit: EcPartner::default_batch_rate_limit(), + pull_sync_enabled: pull, + pull_sync_url: pull.then(|| "https://sync.ssp.example.com/pull".to_owned()), + pull_sync_allowed_domains: if pull { + vec!["sync.ssp.example.com".to_owned()] + } else { + Vec::new() + }, + pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + ts_pull_token: pull + .then(|| Redacted::new("example-token-value-0123456789abcdef".to_owned())), + }; + PartnerRegistry::from_config(&[partner]).expect("should build registry") + } + + fn row() -> KvEntry { + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.network = Some(KvNetwork { + cluster_size: Some(1), + }); + entry + } + + fn writer_context(graph: &KvIdentityGraph, reuse_snapshot: bool) -> EcContext { + let consent = ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + source: ConsentSource::Cookie, + ..ConsentContext::default() + }; + let mut context = EcContext::new_for_test(Some(ec_id()), consent); + context.identity_mut().set_writer(true); + if reuse_snapshot { + context.set_kv_snapshot(graph.load_snapshot(&ec_id())); + } + context + } + + fn queue_fill(context: &mut EcContext) { + context + .identity_mut() + .mutation_mut() + .add_change(PartnerIdChange::FillMissing { + source: PULL_SOURCE + .parse::() + .expect("should parse source"), + record: KvPartnerId::browser("browser-uid", Some(1)), + writer: None, + staged_at: None, + }); + } + + struct Fixture { + settings: Settings, + registry: PartnerRegistry, + stub: Arc, + services: RuntimeServices, + } + + impl Fixture { + fn new(pull: bool) -> Self { + let stub = Arc::new(StubHttpClient::new()); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + Self { + settings: create_test_settings(), + registry: registry(pull), + stub, + services, + } + } + + fn run( + &self, + graph: &KvIdentityGraph, + context: &EcContext, + status: StatusCode, + limiter: Option<&dyn RateLimiter>, + ) -> FlushOutcome { + run_after_send(&AfterSendInput { + settings: &self.settings, + registry: &self.registry, + services: &self.services, + graph, + context, + status, + rate_limiter: limiter, + }) + } + } + + struct AlwaysExceeded; + + impl RateLimiter for AlwaysExceeded { + fn exceeded(&self, _key: &str, _limit: u32) -> Result> { + Ok(true) + } + } + + #[test] + fn writer_flushes_its_mutation_once_reusing_the_phase_one_read() { + let fixture = Fixture::new(false); + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph.create(&ec_id(), &row()).expect("should seed"); + let mut context = writer_context(&graph, true); + queue_fill(&mut context); + let before = counter.writes(); + + let outcome = fixture.run(&graph, &context, StatusCode::OK, None); + + assert_eq!(outcome, FlushOutcome::Written, "should write"); + let after = counter.writes(); + assert_eq!( + after.if_generation_match - before.if_generation_match, + 1, + "one CAS write" + ); + assert_eq!(after.lookups, before.lookups, "reuses the phase 1 read"); + let (stored, _) = graph + .get(&ec_id()) + .expect("should read") + .expect("should exist"); + assert_eq!( + stored.ids[PULL_SOURCE].uid, "browser-uid", + "the fill landed" + ); + } + + #[test] + fn nothing_queued_reads_and_writes_nothing() { + let fixture = Fixture::new(false); + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph.create(&ec_id(), &row()).expect("should seed"); + let context = writer_context(&graph, false); + let before = counter.writes(); + + assert!( + !after_send_needed(&context, StatusCode::OK), + "nothing needed" + ); + assert_eq!( + fixture.run(&graph, &context, StatusCode::OK, None), + FlushOutcome::Skipped, + "skipped" + ); + assert_eq!(counter.writes(), before, "no read and no write"); + } + + #[test] + fn non_success_status_drops_the_mutation() { + let fixture = Fixture::new(false); + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph.create(&ec_id(), &row()).expect("should seed"); + let mut context = writer_context(&graph, true); + queue_fill(&mut context); + let before = counter.writes(); + + for status in [ + StatusCode::FOUND, + StatusCode::NOT_FOUND, + StatusCode::BAD_GATEWAY, + ] { + assert!( + !after_send_needed(&context, status), + "{status} needs nothing" + ); + assert_eq!( + fixture.run(&graph, &context, status, None), + FlushOutcome::Dropped("status"), + "{status} drops the mutation" + ); + } + assert!( + after_send_needed(&context, StatusCode::NOT_MODIFIED), + "control: a 304 still flushes" + ); + assert_eq!(counter.writes(), before, "nothing read or written"); + } + + #[test] + fn tombstone_on_reread_drops_and_calls_no_provider() { + let fixture = Fixture::new(true); + let (graph, _counter) = KvIdentityGraph::write_counting("ec-store"); + graph + .create(&ec_id(), &KvEntry::tombstone(1)) + .expect("should seed"); + let mut context = writer_context(&graph, false); + context + .identity_mut() + .mark(PULL_ID.parse().expect("should parse id")); + + let outcome = fixture.run(&graph, &context, StatusCode::OK, None); + + assert_eq!(outcome, FlushOutcome::Dropped("tombstone"), "dropped"); + assert!( + fixture.stub.recorded_request_uris().is_empty(), + "no provider was called" + ); + } + + #[test] + fn marked_pull_partner_dispatches_after_reread_and_its_uid_lands_in_the_one_write() { + let fixture = Fixture::new(true); + fixture + .stub + .push_response(200, br#"{"uid":"partner-uid"}"#.to_vec()); + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph.create(&ec_id(), &row()).expect("should seed"); + let mut context = writer_context(&graph, true); + context + .identity_mut() + .mark(PULL_ID.parse().expect("should parse id")); + let before = counter.writes(); + + let outcome = fixture.run(&graph, &context, StatusCode::OK, None); + + assert_eq!(outcome, FlushOutcome::Written, "written"); + assert_eq!( + counter.writes().lookups - before.lookups, + 1, + "re-reads once because a provider is marked" + ); + assert_eq!( + counter.writes().if_generation_match - before.if_generation_match, + 1, + "one write" + ); + assert_eq!( + fixture.stub.recorded_request_uris().len(), + 1, + "one partner call" + ); + let (stored, _) = graph + .get(&ec_id()) + .expect("should read") + .expect("should exist"); + let record = &stored.ids[PULL_SOURCE]; + assert_eq!(record.uid, "partner-uid", "the partner UID landed"); + assert_eq!( + record.w.as_deref(), + Some(PULL_ID), + "written by the pull provider" + ); + } + + #[test] + fn cas_conflict_is_not_retried() { + let fixture = Fixture::new(false); + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + graph.create(&ec_id(), &row()).expect("should seed"); + let mut context = writer_context(&graph, true); + queue_fill(&mut context); + counter.conflict_next(1); + let before = counter.writes(); + + assert_eq!( + fixture.run(&graph, &context, StatusCode::OK, None), + FlushOutcome::WriteFailed, + "a lost race fails" + ); + assert_eq!( + counter.writes().if_generation_match - before.if_generation_match, + 1, + "one attempt, never retried" + ); + } + + #[test] + fn burst_capped_marked_provider_backs_off_a_minute() { + let fixture = Fixture::new(true); + let (graph, _counter) = KvIdentityGraph::write_counting("ec-store"); + graph.create(&ec_id(), &row()).expect("should seed"); + let mut context = writer_context(&graph, false); + context + .identity_mut() + .mark(PULL_ID.parse().expect("should parse id")); + let now = crate::ec::current_timestamp(); + + let outcome = fixture.run(&graph, &context, StatusCode::OK, Some(&AlwaysExceeded)); + + assert_eq!(outcome, FlushOutcome::Written, "the backoff is written"); + assert!( + fixture.stub.recorded_request_uris().is_empty(), + "not dispatched" + ); + let (stored, _) = graph + .get(&ec_id()) + .expect("should read") + .expect("should exist"); + let next = stored.ps[PULL_ID].n.expect("should set n"); + assert!(next >= now + 60, "waits at least a minute"); + } + + #[test] + fn missing_cluster_size_rides_on_the_one_write() { + let fixture = Fixture::new(false); + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + let mut entry = row(); + entry.network = None; + graph.create(&ec_id(), &entry).expect("should seed"); + let mut context = writer_context(&graph, true); + queue_fill(&mut context); + let before = counter.writes(); + + assert_eq!( + fixture.run(&graph, &context, StatusCode::OK, None), + FlushOutcome::Written, + "written" + ); + assert_eq!( + counter.writes().total() - before.total(), + 1, + "still one write" + ); + let (stored, _) = graph + .get(&ec_id()) + .expect("should read") + .expect("should exist"); + assert!( + stored + .network + .and_then(|network| network.cluster_size) + .is_some(), + "cluster size stored" + ); + } + + #[test] + fn hourly_rotation_changes_the_leading_provider() { + let ids: Vec = ["a", "b", "c"] + .iter() + .map(|id| id.parse().expect("should parse id")) + .collect(); + let mut hour_one = ids.clone(); + rotate_hourly(&mut hour_one, 3600); + assert_eq!(hour_one[0].as_str(), "b", "hour 1 starts at b"); + let mut hour_three = ids.clone(); + rotate_hourly(&mut hour_three, 3 * 3600); + assert_eq!(hour_three, ids, "hour 3 wraps"); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 7951f982e7..57dfb48eee 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -21,6 +21,7 @@ pub mod descriptor; pub(crate) mod dispatch; pub mod exchange; pub(crate) mod finalize; +pub mod flush; pub(crate) mod generic_pull; pub mod ids; pub mod input; diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index d06447957c..f6b45fc579 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -122,6 +122,7 @@ pub(crate) struct RequestIdentity { staged: StagedCookies, seed: Vec, client_eids_from_body: bool, + marked: Vec, } impl RequestIdentity { @@ -245,6 +246,18 @@ impl RequestIdentity { self.client_eids_from_body = from_body; } + /// After-send providers this request marked pending in phase 3. + #[must_use] + pub(crate) fn marked(&self) -> &[super::ids::IdentityProviderId] { + &self.marked + } + + pub(crate) fn mark(&mut self, provider: super::ids::IdentityProviderId) { + if !self.marked.contains(&provider) { + self.marked.push(provider); + } + } + /// Browser ingest a new visitor's create carries. #[must_use] pub(crate) fn seed(&self) -> &[PartnerIdUpdate] { diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index 458ec84783..d916a25dbd 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -476,167 +476,6 @@ impl KvIdentityGraph { ))) } - /// Merges partner IDs using request-scoped persisted state as the first CAS input. - /// - /// A caller-supplied `Present` snapshot is *proof* that the row exists — - /// either an `Add`-confirmed create from [`generate_if_needed`] or an - /// earlier authoritative read in the same request. Partner-ID enrichment is - /// best effort, so a refresh that reads absent or unreadable never - /// downgrades that proof: the update is skipped and logged, and the proven - /// snapshot is returned so cookie issuance continues from the confirmed - /// write. A failed *write* still reports [`EcKvSnapshot::Failed`] — that is - /// an operation failure, not an ambiguous read. - /// - /// A caller-supplied `Missing` snapshot is revalidated once before the - /// updates are dropped, because a point read on an eventually-consistent - /// store cannot prove absence and routes without orphan recovery have no - /// later chance to retry. - /// - /// [`generate_if_needed`]: super::generate_if_needed - pub(crate) fn upsert_partner_ids_from_snapshot( - &self, - ec_id: &str, - updates: &[PartnerIdUpdate], - snapshot: EcKvSnapshot, - ) -> EcKvSnapshot { - if updates.is_empty() { - return snapshot; - } - - // Existence proof carried by the incoming snapshot. Retained across - // every refresh so a best-effort enrichment read can never retract it. - let proven = match snapshot { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - .. - } if snapshot_id == ec_id => Some(snapshot.clone()), - _ => None, - }; - - // Resolve the initial usable snapshot without spending a CAS attempt. - // Every snapshot except a usable `Present` for this EC ID and a read - // that already failed is refreshed once. A `Missing` recorded earlier - // in the request is not proof of absence: edge KV point reads are - // eventually consistent, and named routes such as `/auction` and - // `/_ts/page-bids` never run orphan recovery, so short-circuiting on a - // stale miss there would drop the request's collected partner IDs - // outright. A refresh that still misses keeps the no-create behavior. - // A `Failed` read is returned as-is — the hot path never retries a - // lookup that already errored. Resolving here keeps all - // `MAX_CAS_RETRIES` iterations available for actual writes. - let mut current = match snapshot { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - generation: Some(_), - .. - } if snapshot_id == ec_id => snapshot, - EcKvSnapshot::Failed { - ec_id: ref snapshot_id, - } if snapshot_id == ec_id => return snapshot, - _ => self.load_snapshot(ec_id), - }; - - for _attempt in 0..MAX_CAS_RETRIES { - let (mut entry, generation) = match current { - EcKvSnapshot::Present { - ec_id: ref snapshot_id, - ref entry, - generation: Some(generation), - } if snapshot_id == ec_id => (entry.as_ref().clone(), generation), - // A refreshed read that is absent or unreadable is authoritative - // for this write: never create or overwrite a missing root. It - // is not authoritative for *existence* though, so a snapshot - // that already proved the row exists survives the refresh. - EcKvSnapshot::Missing { .. } | EcKvSnapshot::Failed { .. } => { - return Self::keep_proven(ec_id, current, proven.as_ref()); - } - // `load_snapshot` never yields `NotRead` or a generation-less - // `Present`; fail closed if that invariant is ever violated. - EcKvSnapshot::Present { .. } | EcKvSnapshot::NotRead => { - return EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }; - } - }; - - if !entry.consent.ok { - return current; - } - if !apply_partner_id_updates(&mut entry, updates) { - return current; - } - let Ok((body, meta_str)) = Self::serialize_entry(&entry, self.store_name()) else { - return EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }; - }; - match self.write_entry( - ec_id, - &body, - &meta_str, - ENTRY_TTL, - EcKvWriteMode::IfGenerationMatch(generation), - ) { - Ok(EcKvWriteOutcome::Written) => { - return EcKvSnapshot::Present { - ec_id: ec_id.to_owned(), - entry: Box::new(entry), - generation: None, - }; - } - Ok(EcKvWriteOutcome::PreconditionFailed) => { - current = self.load_snapshot(ec_id); - } - Err(err) => { - log::warn!( - "snapshot partner upsert failed for '{}': {err:?}", - log_id(ec_id) - ); - return EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - }; - } - } - } - - log::warn!( - "snapshot partner upsert for '{}': CAS conflict after {MAX_CAS_RETRIES} retries; \ - {} partner updates were not persisted", - log_id(ec_id), - updates.len(), - ); - EcKvSnapshot::Failed { - ec_id: ec_id.to_owned(), - } - } - - /// Returns `proven` instead of a read outcome that cannot disprove it. - /// - /// Partner-ID enrichment is best effort. When the caller already held proof - /// that the row exists — an `Add`-confirmed create or an earlier - /// authoritative read in the same request — a refresh that misses or fails - /// says nothing about existence, so the proof is kept and the skipped - /// update is logged. Write failures are *not* routed here: they are real - /// operation failures and stay [`EcKvSnapshot::Failed`] so callers can - /// report them. - fn keep_proven( - ec_id: &str, - downgraded: EcKvSnapshot, - proven: Option<&EcKvSnapshot>, - ) -> EcKvSnapshot { - match proven { - Some(proven) => { - log::warn!( - "snapshot partner upsert skipped for '{}': refresh was not authoritative; \ - keeping the confirmed row", - log_id(ec_id) - ); - proven.clone() - } - None => downgraded, - } - } - /// Atomically merges a partner ID into the existing entry. /// /// Uses CAS (generation markers) to avoid clobbering concurrent writes @@ -1298,6 +1137,26 @@ impl KvIdentityGraph { RemovalWriteOutcome::Failed } + /// Writes a whole identity row with the given precondition. `Ok(false)` + /// is a precondition failure. Never retried here. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::KvStore`] on serialization or store + /// failure. + pub(crate) fn write_identity( + &self, + ec_id: &str, + entry: &KvEntry, + mode: EcKvWriteMode, + ) -> Result> { + let (body, meta) = Self::serialize_entry(entry, self.store_name())?; + Ok(matches!( + self.write_entry(ec_id, &body, &meta, ENTRY_TTL, mode)?, + EcKvWriteOutcome::Written + )) + } + /// Returns the cluster size for an entry, without writing it. /// /// Uses the stored `cluster_size` when the entry has one. Tombstone @@ -2601,51 +2460,6 @@ mod tests { ); } - #[test] - fn snapshot_bulk_upsert_returns_persisted_entry_without_stale_generation() { - let kv = KvIdentityGraph::in_memory("test_store"); - let ec_id = format!("{}.ABC123", "a".repeat(64)); - kv.create(&ec_id, &live_entry()).expect("should create"); - let snapshot = kv.load_snapshot(&ec_id); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = kv.upsert_partner_ids_from_snapshot(&ec_id, &updates, snapshot); - - let entry = outcome - .entry_for(&ec_id) - .expect("should retain persisted entry"); - assert_eq!( - entry.ids.get("ssp_x").map(|id| id.uid.as_str()), - Some("uid-1") - ); - assert_eq!( - outcome.generation_for(&ec_id), - None, - "backend does not return the post-write generation" - ); - } - - #[test] - fn snapshot_bulk_upsert_does_not_create_missing_root() { - let kv = KvIdentityGraph::in_memory("test_store"); - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = kv.upsert_partner_ids_from_snapshot( - &ec_id, - &updates, - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - ); - - assert!( - matches!(outcome, EcKvSnapshot::Missing { .. }), - "missing snapshot should remain missing" - ); - assert!(kv.get(&ec_id).expect("should read store").is_none()); - } - #[test] fn tombstone_existing_from_snapshot_never_creates_missing_key() { let kv = KvIdentityGraph::in_memory("test_store"); @@ -3022,274 +2836,6 @@ mod tests { } } - #[test] - fn snapshot_upsert_with_generation_writes_without_reading() { - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let graph = KvIdentityGraph::counting("counting-store", lookups.clone()); - let ec_id = snapshot_ec_id(); - graph.create(&ec_id, &live_entry()).expect("should seed"); - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: Some(1), - }; - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, snapshot); - - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 0, - "a usable generation must avoid the initial read" - ); - assert_eq!( - outcome - .entry_for(&ec_id) - .and_then(|entry| entry.ids.get("ssp_x")) - .map(|id| id.uid.as_str()), - Some("uid-1") - ); - assert_eq!( - outcome.generation_for(&ec_id), - None, - "a successful write should clear the stale generation" - ); - } - - #[test] - fn snapshot_upsert_unchanged_updates_preserve_generation() { - let kv = KvIdentityGraph::in_memory("test_store"); - let ec_id = snapshot_ec_id(); - let mut seeded = live_entry(); - apply_partner_id_updates(&mut seeded, &[PartnerIdUpdate::new("ssp_x", "uid-1")]); - kv.create(&ec_id, &seeded).expect("should seed"); - let snapshot = kv.load_snapshot(&ec_id); - assert_eq!( - snapshot.generation_for(&ec_id), - Some(1), - "seeded snapshot should carry its stored generation" - ); - - let outcome = kv.upsert_partner_ids_from_snapshot( - &ec_id, - &[PartnerIdUpdate::new("ssp_x", "uid-1")], - snapshot, - ); - - assert_eq!( - outcome.generation_for(&ec_id), - Some(1), - "an unchanged merge preserves the usable generation and performs no write" - ); - } - - #[test] - fn snapshot_upsert_refreshes_unavailable_generation_exactly_once() { - let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let graph = KvIdentityGraph::counting("counting-store", lookups.clone()); - let ec_id = snapshot_ec_id(); - graph.create(&ec_id, &live_entry()).expect("should seed"); - // Finalize-written style snapshot: entry known, generation unavailable. - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: None, - }; - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, snapshot); - - assert_eq!( - lookups.load(std::sync::atomic::Ordering::Relaxed), - 1, - "an unavailable generation refreshes exactly once before CAS" - ); - assert!( - outcome - .entry_for(&ec_id) - .is_some_and(|e| e.ids.contains_key("ssp_x")), - "refreshing an unavailable generation should persist the partner update" - ); - } - - #[test] - fn snapshot_upsert_gen_unavailable_survives_four_conflicts_then_writes() { - // A generation-unavailable snapshot (finalize-written style) refreshes - // once to obtain a usable generation. That refresh must not consume a - // CAS attempt, so all five write attempts remain: four conflicts - // followed by a successful fifth write still persist the update. - let graph = KvIdentityGraph::new(ConflictInjectingEcKv::new(4, false)); - let ec_id = snapshot_ec_id(); - graph.create(&ec_id, &live_entry()).expect("should seed"); - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: None, - }; - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, snapshot); - - assert_eq!( - outcome - .entry_for(&ec_id) - .and_then(|entry| entry.ids.get("ssp_x")) - .map(|id| id.uid.as_str()), - Some("uid-1"), - "the fifth CAS attempt must still succeed after a refresh and four conflicts" - ); - } - - #[test] - fn snapshot_upsert_cas_conflict_remerges_concurrent_data() { - let graph = KvIdentityGraph::new(ConflictInjectingEcKv::new(1, true)); - let ec_id = snapshot_ec_id(); - graph.create(&ec_id, &live_entry()).expect("should seed"); - let snapshot = graph.load_snapshot(&ec_id); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, snapshot); - - let entry = outcome - .entry_for(&ec_id) - .expect("should persist re-merged entry"); - assert_eq!( - entry.ids.get("ssp_x").map(|id| id.uid.as_str()), - Some("uid-1"), - "conflict must re-merge our update onto the concurrently revived row" - ); - assert!(entry.consent.ok, "concurrent revive keeps the row live"); - } - - #[test] - fn snapshot_upsert_revalidates_transient_missing_and_persists() { - // An eventually-consistent point read earlier in the request missed a - // row that exists. Named routes such as `/auction` never run orphan - // recovery, so this refresh is the request's only chance to persist the - // collected partner IDs. - let kv = KvIdentityGraph::in_memory("test_store"); - let ec_id = snapshot_ec_id(); - kv.create(&ec_id, &live_entry()).expect("should seed live"); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = kv.upsert_partner_ids_from_snapshot( - &ec_id, - &updates, - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - ); - - assert_eq!( - outcome - .entry_for(&ec_id) - .and_then(|entry| entry.ids.get("ssp_x").map(|id| id.uid.clone())), - Some("uid-1".to_owned()), - "a stale miss must be revalidated before the updates are dropped" - ); - let (stored, _) = kv - .get(&ec_id) - .expect("should read store") - .expect("row should remain"); - assert_eq!( - stored.ids.get("ssp_x").map(|id| id.uid.as_str()), - Some("uid-1"), - "the revalidated update must reach the store" - ); - } - - #[test] - fn snapshot_upsert_confirmed_missing_still_never_creates() { - // Revalidation only changes what a *stale* miss does. A row that is - // genuinely absent on the refresh must stay absent. - let kv = KvIdentityGraph::in_memory("test_store"); - let ec_id = snapshot_ec_id(); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = kv.upsert_partner_ids_from_snapshot( - &ec_id, - &updates, - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - ); - - assert!( - matches!(outcome, EcKvSnapshot::Missing { .. }), - "a confirmed miss must stay missing" - ); - assert!( - kv.get(&ec_id).expect("should read store").is_none(), - "must not create a root entry for a missing key" - ); - } - - #[test] - fn snapshot_upsert_failed_snapshot_is_not_revalidated() { - // A lookup that already errored is not retried on the hot path, even - // though the row exists. - let kv = KvIdentityGraph::in_memory("test_store"); - let ec_id = snapshot_ec_id(); - kv.create(&ec_id, &live_entry()).expect("should seed live"); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = kv.upsert_partner_ids_from_snapshot( - &ec_id, - &updates, - EcKvSnapshot::Failed { - ec_id: ec_id.clone(), - }, - ); - - assert!( - matches!(outcome, EcKvSnapshot::Failed { .. }), - "a failed lookup must not be retried by partner enrichment" - ); - } - - #[test] - fn snapshot_upsert_rejects_tombstone() { - let kv = KvIdentityGraph::in_memory("test_store"); - let ec_id = snapshot_ec_id(); - kv.create(&ec_id, &KvEntry::tombstone(1000)) - .expect("should seed tombstone"); - let snapshot = kv.load_snapshot(&ec_id); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = kv.upsert_partner_ids_from_snapshot(&ec_id, &updates, snapshot); - - assert!( - outcome - .entry_for(&ec_id) - .is_some_and(|entry| entry.ids.is_empty()), - "a tombstone must reject partner enrichment" - ); - let (stored, _) = kv - .get(&ec_id) - .expect("should read store") - .expect("tombstone should remain"); - assert!(stored.ids.is_empty(), "no update should reach the store"); - } - - #[test] - fn snapshot_upsert_store_failure_returns_failed_not_request_local() { - let graph = KvIdentityGraph::new(WriteFailingEcKv::new()); - let ec_id = snapshot_ec_id(); - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: Some(1), - }; - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, snapshot); - - assert!( - matches!(outcome, EcKvSnapshot::Failed { .. }), - "a store write failure must not claim request-local IDs were persisted" - ); - } - #[test] fn tombstone_existing_from_snapshot_skips_backend_for_authoritative_tombstone() { for generation in [Some(7), None] { @@ -3882,70 +3428,6 @@ mod tests { ); } - #[test] - fn snapshot_upsert_keeps_add_confirmed_present_when_refresh_misses() { - // `generate_if_needed` records a successful `Add` as `Present` without a - // generation. EID ingestion refreshes that snapshot to obtain one; on an - // eventually-consistent store the refresh can miss. Enrichment is best - // effort, so the miss must not retract the confirmed create — otherwise - // finalization suppresses the `ts-ec` cookie for a root that was written. - let graph = KvIdentityGraph::in_memory("empty-store"); - let ec_id = snapshot_ec_id(); - let add_confirmed = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: None, - }; - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, add_confirmed); - - assert!( - outcome.entry_for(&ec_id).is_some(), - "an Add-confirmed row must survive a non-authoritative refresh miss" - ); - assert!( - graph.get(&ec_id).expect("should read store").is_none(), - "a missed refresh must not create or overwrite a root" - ); - } - - #[test] - fn snapshot_upsert_keeps_add_confirmed_present_when_refresh_fails() { - let graph = KvIdentityGraph::failing("failing-store"); - let ec_id = snapshot_ec_id(); - let add_confirmed = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(live_entry()), - generation: None, - }; - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, add_confirmed); - - assert!( - outcome.entry_for(&ec_id).is_some(), - "a read failure is not evidence of absence and must not retract the create" - ); - } - - #[test] - fn snapshot_upsert_without_proof_still_reports_a_refresh_miss() { - // No prior proof of existence: a `NotRead` snapshot that refreshes into - // a miss must stay `Missing` so finalization can run orphan recovery. - let graph = KvIdentityGraph::in_memory("empty-store"); - let ec_id = snapshot_ec_id(); - let updates = [PartnerIdUpdate::new("ssp_x", "uid-1")]; - - let outcome = - graph.upsert_partner_ids_from_snapshot(&ec_id, &updates, EcKvSnapshot::NotRead); - - assert!( - matches!(outcome, EcKvSnapshot::Missing { .. }), - "an unproven refresh miss must remain a miss" - ); - } - #[test] fn a_store_error_never_carries_the_whole_identifier() { // Every message in this module goes through `log_id`, so a report that diff --git a/crates/trusted-server-core/src/ec/pull_sync.rs b/crates/trusted-server-core/src/ec/pull_sync.rs index 9e8ad8e547..7a6418709f 100644 --- a/crates/trusted-server-core/src/ec/pull_sync.rs +++ b/crates/trusted-server-core/src/ec/pull_sync.rs @@ -1,302 +1,18 @@ -//! Pull sync background dispatch. +//! Pull sync partner helpers. //! -//! Launches partner pull-sync requests for organic traffic after the client -//! response has been sent. Dispatch is best-effort and never affects client -//! response status. +//! Pull partners run as [`GenericPullProvider`](super::identity::generic_pull) +//! identity providers, dispatched by the after-send identity flush. This +//! module keeps the partner URL checks they use and the completeness test +//! behind the pull-complete marker. //! -//! Pull sync currently fills missing partner UIDs only. Once a partner UID is -//! present in the EC identity entry, it is not periodically refreshed because -//! the entry no longer stores per-partner sync timestamps. +//! Pull partners fill missing partner UIDs only, unless an operator makes one +//! a source owner. -use edgezero_core::body::Body as EdgeBody; -use http::{Method, StatusCode, header}; -use serde::Deserialize; use url::Url; -use crate::consent::allows_eid_persistence; -use crate::platform::{ - DEFAULT_FIRST_BYTE_TIMEOUT, PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, - PlatformResponse, RuntimeServices, -}; -use crate::settings::Settings; - -use super::generation::{ec_hash, is_valid_ec_id}; -use super::kv::{KvIdentityGraph, PartnerIdUpdate}; use super::kv_types::KvEntry; -use super::rate_limiter::RateLimiter; use super::registry::{PartnerConfig, PartnerRegistry}; -// `current_timestamp` is defined in the parent `ec` module. -use super::EcContext; -use super::EcKvSnapshot; -use super::current_timestamp; - -/// Inputs needed to dispatch pull sync after response flush. -#[derive(Debug, Clone)] -pub struct PullSyncContext { - ec_id: String, - snapshot: EcKvSnapshot, -} - -impl PullSyncContext { - /// Returns the EC ID for the request. - #[must_use] - pub fn ec_id(&self) -> &str { - &self.ec_id - } -} - -struct InFlightPull { - source_domain: String, - pending: PlatformPendingRequest, -} - -#[derive(Debug, Deserialize)] -struct PullSyncResponse { - uid: Option, -} - -/// Builds post-send pull-sync context from the route EC context. -/// -/// Returns `None` when there are no pull-enabled partners, when consent denies -/// EC, when there is no valid active EC ID, or when the request snapshot holds -/// no consented row that is still missing at least one pull-partner UID. -#[must_use] -pub fn build_pull_sync_context( - ec_context: &EcContext, - registry: &PartnerRegistry, -) -> Option { - // Without EID-persistence consent (TCF Purpose 1 + 4), finalization - // withholds browser-supplied partner UIDs. That leaves partner slots empty, - // which pull sync would otherwise read as eligible and disclose the EC ID - // to partners for exactly the users who opted out. - if registry.pull_enabled_partners().is_empty() - || !ec_context.ec_allowed() - || !allows_eid_persistence(ec_context.consent()) - { - return None; - } - - let ec_id_ref = ec_context.ec_value()?; - if !is_valid_ec_id(ec_id_ref) { - log::debug!("Pull sync: skipping dispatch because active EC ID is invalid format"); - return None; - } - - let entry = ec_context.kv_snapshot().entry_for(ec_id_ref)?; - if !entry.consent.ok || entry_is_pull_complete(entry, registry) { - return None; - } - - let ec_id = ec_id_ref.to_owned(); - let snapshot = ec_context.kv_snapshot().clone(); - Some(PullSyncContext { ec_id, snapshot }) -} - -/// Dispatches partner pull-sync requests in the background. -/// -/// This function is best-effort: all errors are logged and swallowed. -/// -/// # Consent revalidation -/// -/// Pull sync runs *after* the page response has been flushed to the browser and -/// discloses the raw `ec_id` to partners, so it cannot be authorized by the -/// request snapshot alone: a concurrent CMP withdrawal can tombstone the row in -/// the window between snapshot capture and dispatch. The live row is re-read -/// immediately before the first partner request and anything short of a live, -/// consenting entry cancels the dispatch. The read is deferred until after the -/// snapshot-based eligibility filter, so a request with nothing to pull still -/// costs no KV operation, and the fresh snapshot is reused for the write-back -/// instead of the stale one. -/// -/// # Panics -/// -/// Panics if the HTTP request builder produces an invalid request, which -/// cannot happen with the hardcoded method and well-formed URI used here. -pub fn dispatch_pull_sync( - settings: &Settings, - kv: &KvIdentityGraph, - registry: &PartnerRegistry, - rate_limiter: &dyn RateLimiter, - context: &PullSyncContext, - services: &RuntimeServices, -) { - let now = current_timestamp(); - let mut pull_partners = registry.pull_enabled_partners(); - - if pull_partners.is_empty() { - return; - } - - let Some(request_entry) = context.snapshot.entry_for(context.ec_id()) else { - log::debug!( - "Pull sync: skipping dispatch for '{}' because the request captured no usable \ - identity snapshot", - super::log_id(context.ec_id()) - ); - return; - }; - if !request_entry.consent.ok { - return; - } - - // Sort by source domain for deterministic ordering, then apply a rotating - // hourly offset so that different partners get dispatch priority (§10.3). - pull_partners.sort_by(|a, b| a.source_domain.cmp(&b.source_domain)); - - log::debug!( - "Pull sync: {} pull-enabled partners after filtering", - pull_partners.len(), - ); - - // Rotate the partner list so that the starting partner changes each - // hour. This ensures fair distribution when max_concurrency limits - // how many partners are dispatched per request. - let offset = (now / 3600) as usize % pull_partners.len(); - pull_partners.rotate_left(offset); - - // Drop partners the request snapshot already has a UID for before paying - // for the revalidation read: a dispatch with nothing to pull must not add a - // KV operation. - pull_partners.retain(|partner| is_partner_pull_eligible(partner, Some(request_entry))); - if pull_partners.is_empty() { - return; - } - - let live_snapshot = kv.load_snapshot(context.ec_id()); - let Some(live_entry) = live_snapshot.entry_for(context.ec_id()) else { - log::warn!( - "Pull sync: skipping dispatch for '{}' because the live identity row could not be \ - confirmed", - super::log_id(context.ec_id()) - ); - return; - }; - if !live_entry.consent.ok { - log::info!( - "Pull sync: skipping dispatch for '{}' because consent was withdrawn after the \ - request snapshot was captured", - super::log_id(context.ec_id()) - ); - return; - } - - let max_concurrency = settings.ec.pull_sync_concurrency.max(1); - let mut in_flight: Vec = Vec::new(); - let mut updates = Vec::new(); - - for partner in pull_partners { - // Re-checked against the live row: a concurrent request may have filled - // this partner's UID since the request snapshot was captured. - if !is_partner_pull_eligible(partner, Some(live_entry)) { - continue; - } - - let Some(url) = validated_pull_sync_url(partner) else { - continue; - }; - - let rate_key = pull_rate_limit_key(&partner.source_domain, context.ec_id()); - match rate_limiter.exceeded(&rate_key, partner.pull_sync_rate_limit) { - Ok(true) => { - log::debug!( - "Pull sync: rate-limited partner '{}' for ec_id '{}'", - partner.source_domain, - super::log_id(context.ec_id()) - ); - continue; - } - Ok(false) => {} - Err(err) => { - log::warn!( - "Pull sync: failed to read rate limit for partner '{}': {err:?}", - partner.source_domain - ); - continue; - } - } - - let Some(token) = partner.ts_pull_token.as_ref() else { - log::warn!( - "Pull sync: partner '{}' enabled but missing ts_pull_token", - partner.source_domain - ); - continue; - }; - - let request_url = build_pull_request_url(url, context.ec_id()); - let scheme = request_url.scheme().to_string(); - let host = request_url.host_str().unwrap_or_default().to_string(); - let port = request_url.port(); - - let backend_name = match services.backend().ensure(&PlatformBackendSpec { - scheme, - host, - port, - host_header_override: None, - certificate_check: settings.proxy.certificate_check, - first_byte_timeout: DEFAULT_FIRST_BYTE_TIMEOUT, - between_bytes_timeout: DEFAULT_FIRST_BYTE_TIMEOUT, - discriminator: None, - }) { - Ok(name) => name, - Err(err) => { - log::warn!( - "Pull sync: failed to resolve backend for partner '{}': {err:?}", - partner.source_domain - ); - continue; - } - }; - - let request = http::Request::builder() - .method(Method::GET) - .uri(request_url.as_str()) - .header("authorization", format!("Bearer {}", token.expose())) - .body(EdgeBody::empty()) - .expect("should build pull sync request"); - - let pending = match futures::executor::block_on( - services - .http_client() - .send_async(PlatformHttpRequest::new(request, backend_name)), - ) { - Ok(pending) => pending, - Err(err) => { - log::warn!( - "Pull sync: failed to dispatch partner '{}': {err:?}", - partner.source_domain - ); - continue; - } - }; - - in_flight.push(InFlightPull { - source_domain: partner.source_domain.clone(), - pending, - }); - - if in_flight.len() >= max_concurrency { - drain_pull_batch(&mut in_flight, services, &mut updates); - } - } - - drain_pull_batch(&mut in_flight, services, &mut updates); - if !updates.is_empty() { - // Write back from the revalidated snapshot, not the request one: its - // generation is current, so the first CAS attempt is not spent losing a - // conflict against the read that authorized this dispatch. - let outcome = - kv.upsert_partner_ids_from_snapshot(context.ec_id(), &updates, live_snapshot.clone()); - if matches!(outcome, EcKvSnapshot::Failed { .. }) { - log::warn!( - "Pull sync: failed to persist partner updates for '{}'", - super::log_id(context.ec_id()) - ); - } - } -} - /// Returns whether a consented live entry holds an ID for every pull-enabled /// partner. Always returns `false` when there are no pull-enabled partners. #[must_use] @@ -370,181 +86,11 @@ pub(crate) fn build_pull_request_url(mut base_url: Url, ec_id: &str) -> Url { base_url } -fn pull_rate_limit_key(source_domain: &str, ec_id: &str) -> String { - format!("pull:{source_domain}:{}", ec_hash(ec_id)) -} - -fn drain_pull_batch( - in_flight: &mut Vec, - services: &RuntimeServices, - updates: &mut Vec, -) { - for pending in in_flight.drain(..) { - let source_domain = pending.source_domain; - // All requests were dispatched up front via send_async, so waiting on - // each in turn does not change concurrency. - let response = - match futures::executor::block_on(services.http_client().wait(pending.pending)) { - Ok(response) => response, - Err(err) => { - log::warn!( - "Pull sync: request failed for partner '{}': {err:?}", - source_domain - ); - continue; - } - }; - - let Some(uid) = extract_pull_uid(response, &source_domain) else { - continue; - }; - - updates.push(PartnerIdUpdate::new(source_domain, uid)); - } -} - -/// Maximum response body size accepted from pull sync partners (64 KiB). -/// -/// The expected response is `{"uid":""}`, so 64 KiB is generous. -/// This prevents a misbehaving partner from exhausting WASM memory. -const MAX_PULL_RESPONSE_BYTES: usize = 64 * 1024; - -fn response_content_length_exceeds_limit(response: &PlatformResponse, source_domain: &str) -> bool { - let Some(value) = response.response.headers().get(header::CONTENT_LENGTH) else { - return false; - }; - - let Some(value) = value.to_str().ok() else { - log::warn!( - "Pull sync: partner '{}' returned invalid Content-Length header, rejecting", - source_domain - ); - return true; - }; - - let Ok(length) = value.parse::() else { - log::warn!( - "Pull sync: partner '{}' returned malformed Content-Length header, rejecting", - source_domain - ); - return true; - }; - - if length > MAX_PULL_RESPONSE_BYTES { - log::warn!( - "Pull sync: partner '{}' returned oversized Content-Length ({} bytes), rejecting", - source_domain, - length - ); - return true; - } - - false -} - -fn extract_pull_uid(response: PlatformResponse, source_domain: &str) -> Option { - let status = response.response.status(); - - if status == StatusCode::NOT_FOUND { - log::debug!( - "Pull sync: partner '{}' returned 404, treating as no-op", - source_domain - ); - return None; - } - - if !status.is_success() { - log::warn!( - "Pull sync: partner '{}' returned non-success status {}", - source_domain, - status - ); - return None; - } - - if response_content_length_exceeds_limit(&response, source_domain) { - return None; - } - - let body = response - .response - .into_body() - .into_bytes() - .unwrap_or_default(); - if body.len() > MAX_PULL_RESPONSE_BYTES { - log::warn!( - "Pull sync: partner '{}' returned oversized response ({} bytes), rejecting", - source_domain, - body.len() - ); - return None; - } - let payload = match serde_json::from_slice::(&body) { - Ok(payload) => payload, - Err(err) => { - log::warn!( - "Pull sync: partner '{}' returned invalid JSON body: {err}", - source_domain - ); - return None; - } - }; - - use super::kv_types::MAX_UID_LENGTH; - - let uid = payload.uid.filter(|value| !value.trim().is_empty()); - match uid { - None => { - log::debug!( - "Pull sync: partner '{}' returned null/empty uid, treating as no-op", - source_domain - ); - None - } - Some(ref value) if value.len() > MAX_UID_LENGTH => { - log::warn!( - "Pull sync: partner '{}' returned uid exceeding {} bytes (got {}), rejecting", - source_domain, - MAX_UID_LENGTH, - value.len() - ); - None - } - _ => uid, - } -} - #[cfg(test)] mod tests { use super::*; - use crate::consent::types::ConsentContext; use crate::ec::kv_types::KvEntry; - use crate::platform::PlatformResponse; use crate::redacted::Redacted; - use crate::test_support::tests::purpose_one_only_consent; - - fn make_response(status: u16, body: &[u8]) -> PlatformResponse { - PlatformResponse::new( - edgezero_core::http::response_builder() - .status(status) - .body(EdgeBody::from(body.to_vec())) - .expect("should build test response"), - ) - } - - fn make_response_with_content_length( - status: u16, - content_length: usize, - body: &[u8], - ) -> PlatformResponse { - PlatformResponse::new( - edgezero_core::http::response_builder() - .status(status) - .header(header::CONTENT_LENGTH, content_length.to_string()) - .body(EdgeBody::from(body.to_vec())) - .expect("should build test response"), - ) - } fn pull_partner(ttl_sec: u64) -> PartnerConfig { PartnerConfig { @@ -563,105 +109,6 @@ mod tests { } } - #[test] - fn build_pull_sync_context_returns_context_when_valid() { - let consent = ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, - ..ConsentContext::default() - }; - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); - let graph = KvIdentityGraph::in_memory("pull_store"); - ec_context.set_kv_snapshot(seed_present_snapshot(&graph, &ec_id)); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - - let context = build_pull_sync_context(&ec_context, ®istry) - .expect("should build pull sync context for valid EC"); - assert_eq!( - context.ec_id(), - ec_context.ec_value().expect("ec should be present"), - "should capture the EC ID from context" - ); - } - - #[test] - fn build_pull_sync_context_rejects_invalid_ec_id() { - let consent = ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, - ..ConsentContext::default() - }; - let ec_context = EcContext::new_for_test(Some("invalid-ec".to_owned()), consent); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - - let context = build_pull_sync_context(&ec_context, ®istry); - assert!( - context.is_none(), - "should reject pull sync context when EC ID format is invalid" - ); - } - - #[test] - fn build_pull_sync_context_skips_when_eid_persistence_is_denied() { - // Purpose 1 granted keeps the EC, but Purpose 4 denied withholds EID - // writes and leaves partner slots empty. Pull sync must not read those - // empty slots as eligible and disclose the EC ID to partners. - let consent = purpose_one_only_consent(); - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); - let graph = KvIdentityGraph::in_memory("pull_store"); - ec_context.set_kv_snapshot(seed_present_snapshot(&graph, &ec_id)); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - assert!( - ec_context.ec_allowed(), - "should keep the EC allowed when Purpose 1 is granted" - ); - - let context = build_pull_sync_context(&ec_context, ®istry); - - assert!( - context.is_none(), - "should skip pull sync when TCF Purpose 4 is denied" - ); - } - - #[test] - fn build_pull_sync_context_skips_empty_registry_and_complete_snapshot() { - let consent = ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, - ..ConsentContext::default() - }; - let ec_id = format!("{}.ABC123", "a".repeat(64)); - let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); - let graph = KvIdentityGraph::in_memory("pull_store"); - let mut snapshot = seed_present_snapshot(&graph, &ec_id); - let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) - .expect("should build registry"); - - assert!( - build_pull_sync_context(&ec_context, &PartnerRegistry::empty()).is_none(), - "no pull partners should skip before graph construction" - ); - assert!( - build_pull_sync_context(&ec_context, ®istry).is_none(), - "an unread snapshot should skip before graph construction" - ); - - if let EcKvSnapshot::Present { entry, .. } = &mut snapshot { - entry.ids.insert( - "ssp.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId::new("uid".to_owned()), - ); - } - ec_context.set_kv_snapshot(snapshot); - assert!( - build_pull_sync_context(&ec_context, ®istry).is_none(), - "a complete snapshot should skip post-send work" - ); - } - #[test] fn partner_is_eligible_when_missing_from_entry() { let partner = pull_partner(3600); @@ -684,6 +131,25 @@ mod tests { ); } + fn pull_enabled_ec_partner(source_domain: &str) -> crate::settings::EcPartner { + crate::settings::EcPartner { + name: format!("Partner {source_domain}"), + source_domain: source_domain.to_owned(), + openrtb_atype: crate::settings::EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: None, + batch_rate_limit: crate::settings::EcPartner::default_batch_rate_limit(), + pull_sync_enabled: true, + pull_sync_url: Some(format!("https://sync.{source_domain}/pull")), + pull_sync_allowed_domains: vec![format!("sync.{source_domain}")], + pull_sync_ttl_sec: crate::settings::EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: crate::settings::EcPartner::default_pull_sync_rate_limit(), + ts_pull_token: Some(Redacted::new( + "example-token-value-0123456789abcdef".to_owned(), + )), + } + } + #[test] fn completeness_requires_all_pull_partner_ids() { let registry = PartnerRegistry::from_config(&[ @@ -760,485 +226,7 @@ mod tests { ); } - #[test] - fn pull_rate_limit_key_uses_ec_hash_only() { - let first_ec_id = format!("{}.ABC123", "a".repeat(64)); - let second_ec_id = format!("{}.XYZ789", "a".repeat(64)); - - let first_key = pull_rate_limit_key("ssp.example.com", &first_ec_id); - let second_key = pull_rate_limit_key("ssp.example.com", &second_ec_id); - - assert_eq!( - first_key, second_key, - "should bucket different suffixes for the same EC hash together" - ); - assert_eq!( - first_key, - format!("pull:ssp.example.com:{}", "a".repeat(64)), - "should key pull-sync rate limiting by source domain and EC hash" - ); - } - - #[test] - fn extract_pull_uid_treats_404_as_noop() { - let response = make_response(404, b""); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should treat 404 as no-op"); - } - - #[test] - fn extract_pull_uid_treats_uid_null_as_noop() { - let response = make_response(200, b"{\"uid\":null}"); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should treat uid=null as no-op"); - } - - #[test] - fn extract_pull_uid_rejects_oversized_uid() { - let long_uid = "x".repeat(513); - let body = format!("{{\"uid\":\"{long_uid}\"}}"); - let response = make_response(200, body.as_bytes()); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should reject uid exceeding 512 bytes"); - } - - #[test] - fn extract_pull_uid_reads_uid_from_success_body() { - let response = make_response(200, b"{\"uid\":\"abc123\"}"); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert_eq!( - uid.as_deref(), - Some("abc123"), - "should parse uid from 200 body" - ); - } - - #[test] - fn extract_pull_uid_rejects_oversized_content_length_before_body_read() { - let response = make_response_with_content_length( - 200, - MAX_PULL_RESPONSE_BYTES + 1, - b"{\"uid\":\"abc123\"}", - ); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!( - uid.is_none(), - "should reject oversized Content-Length before parsing body" - ); - } - - #[test] - fn extract_pull_uid_accepts_small_body_without_content_length() { - let response = make_response(200, b"{\"uid\":\"abc123\"}"); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert_eq!( - uid.as_deref(), - Some("abc123"), - "should accept small valid response without Content-Length" - ); - } - - #[test] - fn extract_pull_uid_rejects_body_larger_than_limit() { - let body = format!("{{\"uid\":\"{}\"}}", "x".repeat(MAX_PULL_RESPONSE_BYTES)); - let response = make_response(200, body.as_bytes()); - - let uid = extract_pull_uid(response, "ssp.example.com"); - assert!(uid.is_none(), "should reject body larger than limit"); - } - - #[test] - fn rotating_offset_distributes_partners_across_hours() { - // Simulate 3 partners sorted by source domain: alpha, beta, gamma. - let ids = vec!["alpha.example.com", "beta.example.com", "gamma.example.com"]; - - // Hour 0: offset = 0 % 3 = 0 → [alpha, beta, gamma] - let ts_h0: u64 = 100; // within hour 0 - let offset_h0 = (ts_h0 / 3600) as usize % ids.len(); - assert_eq!(offset_h0, 0, "hour 0 should start at index 0"); - - // Hour 1: offset = (3600 / 3600) % 3 = 1 → [beta, gamma, alpha] - let offset_h1 = (3600u64 / 3600) as usize % ids.len(); - assert_eq!(offset_h1, 1, "hour 1 should start at index 1"); - - // Hour 2: offset = (7200 / 3600) % 3 = 2 → [gamma, alpha, beta] - let offset_h2 = (7200u64 / 3600) as usize % ids.len(); - assert_eq!(offset_h2, 2, "hour 2 should start at index 2"); - - // Hour 3: offset = (10800 / 3600) % 3 = 0 → wraps back to [alpha, beta, gamma] - let offset_h3 = (10800u64 / 3600) as usize % ids.len(); - assert_eq!(offset_h3, 0, "hour 3 should wrap back to index 0"); - - // Verify rotate_left produces expected ordering - let mut rotated = ids.clone(); - rotated.rotate_left(offset_h1); - assert_eq!( - rotated, - vec!["beta.example.com", "gamma.example.com", "alpha.example.com"], - "hour 1 rotation should move beta to front" - ); - } - // ----------------------------------------------------------------------- // Snapshot-driven eligibility and request-wide aggregation // ----------------------------------------------------------------------- - - use crate::ec::kv::TombstoneOutcome; - use crate::error::TrustedServerError; - use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; - use crate::settings::EcPartner; - use crate::test_support::tests::create_test_settings; - use error_stack::Report; - use std::sync::Arc; - - struct AllowAllRateLimiter; - - impl RateLimiter for AllowAllRateLimiter { - fn exceeded( - &self, - _key: &str, - _hourly_limit: u32, - ) -> Result> { - Ok(false) - } - } - - fn pull_enabled_ec_partner(source_domain: &str) -> EcPartner { - EcPartner { - name: format!("Partner {source_domain}"), - source_domain: source_domain.to_owned(), - openrtb_atype: EcPartner::default_openrtb_atype(), - bidstream_enabled: true, - api_token: Some(Redacted::new(format!( - "{source_domain}-api-token-32-bytes-minimum" - ))), - batch_rate_limit: EcPartner::default_batch_rate_limit(), - pull_sync_enabled: true, - pull_sync_url: Some(format!("https://{source_domain}/sync")), - pull_sync_allowed_domains: vec![source_domain.to_owned()], - pull_sync_ttl_sec: 3600, - pull_sync_rate_limit: 100, - ts_pull_token: Some(Redacted::new("outbound-token".to_owned())), - } - } - - fn snapshot_ec_id() -> String { - format!("{}.ABC123", "a".repeat(64)) - } - - fn seed_present_snapshot(graph: &KvIdentityGraph, ec_id: &str) -> EcKvSnapshot { - let mut entry = KvEntry::tombstone(1000); - entry.consent.ok = true; - graph.create(ec_id, &entry).expect("should seed live entry"); - graph.load_snapshot(ec_id) - } - - #[test] - fn dispatch_pull_sync_aggregates_batches_into_one_bulk_write() { - let mut settings = create_test_settings(); - // Force one partner per concurrency batch so responses span batches. - settings.ec.pull_sync_concurrency = 1; - let registry = PartnerRegistry::from_config(&[ - pull_enabled_ec_partner("alpha.example.com"), - pull_enabled_ec_partner("beta.example.com"), - ]) - .expect("should build pull registry"); - - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let snapshot = seed_present_snapshot(&graph, &ec_id); - - let stub = Arc::new(StubHttpClient::new()); - // One JSON response per partner, drained across two concurrency batches. - stub.push_response(200, br#"{"uid":"synced-uid"}"#.to_vec()); - stub.push_response(200, br#"{"uid":"synced-uid"}"#.to_vec()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - let (entry, generation) = graph - .get(&ec_id) - .expect("should read store") - .expect("entry should exist"); - assert_eq!( - entry.ids.get("alpha.example.com").map(|id| id.uid.as_str()), - Some("synced-uid"), - "first partner UID should persist" - ); - assert_eq!( - entry.ids.get("beta.example.com").map(|id| id.uid.as_str()), - Some("synced-uid"), - "second partner UID should persist" - ); - assert_eq!( - generation, 2, - "two partner responses across batches must persist in exactly one bulk write" - ); - } - - #[test] - fn dispatch_pull_sync_calls_and_persists_only_missing_partner() { - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = PartnerRegistry::from_config(&[ - pull_enabled_ec_partner("alpha.example.com"), - pull_enabled_ec_partner("beta.example.com"), - ]) - .expect("should build pull registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - graph - .create( - &ec_id, - &KvEntry::minimal("alpha.example.com", "existing-alpha", 1_000), - ) - .expect("should seed partial entry"); - let snapshot = graph.load_snapshot(&ec_id); - let stub = Arc::new(StubHttpClient::new()); - stub.push_response(200, br#"{"uid":"new-beta"}"#.to_vec()); - let services = build_services_with_http_client(stub.clone()); - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert_eq!( - stub.recorded_backend_names().len(), - 1, - "should call only the missing partner" - ); - let (entry, generation) = graph - .get(&ec_id) - .expect("should read store") - .expect("entry should exist"); - assert_eq!( - entry - .ids - .get("alpha.example.com") - .map(|partner_id| partner_id.uid.as_str()), - Some("existing-alpha"), - "existing UID should remain unchanged" - ); - assert_eq!( - entry - .ids - .get("beta.example.com") - .map(|partner_id| partner_id.uid.as_str()), - Some("new-beta"), - "missing UID should persist" - ); - assert_eq!(generation, 2, "missing UID should use one bulk CAS write"); - } - - #[test] - fn dispatch_pull_sync_skips_non_present_snapshots() { - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let stub = Arc::new(StubHttpClient::new()); - let services = build_services_with_http_client(stub.clone()); - - for snapshot in [ - EcKvSnapshot::NotRead, - EcKvSnapshot::Missing { - ec_id: ec_id.clone(), - }, - EcKvSnapshot::Failed { - ec_id: ec_id.clone(), - }, - ] { - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - } - - assert!( - stub.recorded_backend_names().is_empty(), - "not-read, missing, and failed snapshots must not dispatch pull sync" - ); - assert!( - graph.get(&ec_id).expect("should read store").is_none(), - "no snapshot state should create a missing root" - ); - } - - #[test] - fn dispatch_pull_sync_skips_tombstone_snapshot() { - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(KvEntry::tombstone(1000)), - generation: Some(1), - }; - let stub = Arc::new(StubHttpClient::new()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert!( - stub.recorded_backend_names().is_empty(), - "a tombstone snapshot must not dispatch pull sync" - ); - } - #[test] - fn dispatch_pull_sync_skips_dispatch_when_ec_is_tombstoned_after_snapshot_capture() { - // The request snapshot authorized pull sync, then a concurrent request - // completed a CMP withdrawal while the page response was in flight. - // Pull sync runs post-send and discloses the raw `ec_id` to partners, so - // it must revalidate the live row and cancel rather than leak an - // identity the user just withdrew. - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let snapshot = seed_present_snapshot(&graph, &ec_id); - - // Concurrent withdrawal lands after the snapshot was captured. - assert_eq!( - graph - .write_withdrawal_tombstone(&ec_id, drop) - .expect("should tombstone the row"), - TombstoneOutcome::Written, - "should tombstone the row the snapshot still reports as present" - ); - - let stub = Arc::new(StubHttpClient::new()); - stub.push_response(200, br#"{"uid":"leaked-uid"}"#.to_vec()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert!( - stub.recorded_backend_names().is_empty(), - "a withdrawal that lands after snapshot capture must cancel post-send pull sync" - ); - let (entry, _) = graph - .get(&ec_id) - .expect("should read store") - .expect("tombstone should remain"); - assert!( - entry.ids.is_empty(), - "no partner UID may be written back onto a tombstoned row" - ); - } - - #[test] - fn dispatch_pull_sync_skips_revalidation_read_when_no_partner_is_eligible() { - // Every pull-enabled partner already has a UID in the request snapshot, - // so there is nothing to dispatch. The revalidation read exists to - // authorize outbound calls; with no calls to authorize it must not cost - // a KV operation. - let mut settings = create_test_settings(); - settings.ec.pull_sync_concurrency = 4; - let registry = - PartnerRegistry::from_config(&[pull_enabled_ec_partner("alpha.example.com")]) - .expect("should build registry"); - let graph = KvIdentityGraph::in_memory("pull_store"); - let ec_id = snapshot_ec_id(); - let mut entry = KvEntry::tombstone(1000); - entry.consent.ok = true; - entry.ids.insert( - "alpha.example.com".to_owned(), - crate::ec::kv_types::KvPartnerId::new("already-known".to_owned()), - ); - graph - .create(&ec_id, &entry) - .expect("should seed live entry"); - let snapshot = EcKvSnapshot::Present { - ec_id: ec_id.clone(), - entry: Box::new(entry), - generation: Some(1), - }; - - let stub = Arc::new(StubHttpClient::new()); - let services = build_services_with_http_client(stub.clone()); - - let context = PullSyncContext { - ec_id: ec_id.clone(), - snapshot, - }; - dispatch_pull_sync( - &settings, - &graph, - ®istry, - &AllowAllRateLimiter, - &context, - &services, - ); - - assert!( - stub.recorded_backend_names().is_empty(), - "a fully synced entry must not dispatch pull sync" - ); - } } diff --git a/crates/trusted-server-core/src/migration_guards.rs b/crates/trusted-server-core/src/migration_guards.rs index 537ef89457..118e06bdc5 100644 --- a/crates/trusted-server-core/src/migration_guards.rs +++ b/crates/trusted-server-core/src/migration_guards.rs @@ -115,6 +115,7 @@ fn checked_sources() -> &'static [(&'static str, &'static str)] { "ec/identity/finalize.rs", include_str!("ec/identity/finalize.rs"), ), + ("ec/identity/flush.rs", include_str!("ec/identity/flush.rs")), ( "ec/identity/generic_pull.rs", include_str!("ec/identity/generic_pull.rs"), From b3ced253570530d86b159765875bb7c45724369f Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:28:23 +0530 Subject: [PATCH 30/33] Test the one-write-per-page-load invariant end to end Drives the create before routing, finalize before send and the after-send flush, as the adapter runs them, against a write-counting store: one write per page load, none on stagers or non-2xx/304 navigations, no retry on a CAS conflict, zero reads for a recent cookie with nothing due, and an /auction body-only source reaching KV on the next navigation even after a refused flush. Signed-off-by: dhruv8sh --- .../src/ec/identity/invariant_tests.rs | 479 ++++++++++++++++++ .../src/ec/identity/mod.rs | 2 + 2 files changed, 481 insertions(+) create mode 100644 crates/trusted-server-core/src/ec/identity/invariant_tests.rs diff --git a/crates/trusted-server-core/src/ec/identity/invariant_tests.rs b/crates/trusted-server-core/src/ec/identity/invariant_tests.rs new file mode 100644 index 0000000000..c4eb0ec864 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity/invariant_tests.rs @@ -0,0 +1,479 @@ +//! The one-write-per-page-load invariant, end to end through the same core +//! steps the adapter runs: the create before routing, `ec_finalize_response` +//! before send, and `run_after_send` after it. + +use std::sync::Arc; + +use edgezero_core::body::Body as EdgeBody; +use http::{HeaderMap, HeaderValue, Response, StatusCode, header}; + +use super::flush::{AfterSendInput, FlushOutcome, run_after_send}; +use crate::consent::ConsentContext; +use crate::consent::jurisdiction::Jurisdiction; +use crate::consent::types::ConsentSource; +use crate::ec::finalize::ec_finalize_response; +use crate::ec::kv::KvIdentityGraph; +use crate::ec::kv_backend::test_support::{WriteCounter, WriteCounts}; +use crate::ec::kv_types::{KvEntry, KvNetwork, KvPartnerId}; +use crate::ec::registry::PartnerRegistry; +use crate::ec::{EcContext, EidSyncSource}; +use crate::platform::RuntimeServices; +use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; +use crate::settings::{EcPartner, Settings}; +use crate::test_support::tests::create_test_settings; + +const SHARED: &str = "sharedid.org"; +const GAP: &str = "gap.example.com"; + +fn partner(source: &str) -> EcPartner { + EcPartner { + name: format!("Partner {source}"), + source_domain: source.to_owned(), + openrtb_atype: EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: None, + batch_rate_limit: EcPartner::default_batch_rate_limit(), + pull_sync_enabled: false, + pull_sync_url: None, + pull_sync_allowed_domains: vec![], + pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + ts_pull_token: None, + } +} + +fn consent() -> ConsentContext { + ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + source: ConsentSource::Cookie, + ..ConsentContext::default() + } +} + +struct World { + settings: Settings, + registry: PartnerRegistry, + services: RuntimeServices, + graph: KvIdentityGraph, + counter: WriteCounter, +} + +/// One request: its role, cookies and response status. +struct Visit<'a> { + writer: bool, + source: Option, + cookies: &'a str, + sharedid: Option<&'a str>, + status: StatusCode, +} + +impl World { + fn new() -> Self { + let (graph, counter) = KvIdentityGraph::write_counting("ec-store"); + Self { + settings: create_test_settings(), + registry: PartnerRegistry::from_config(&[partner(SHARED), partner(GAP)]) + .expect("should build registry"), + services: build_services_with_http_client(Arc::new(StubHttpClient::new())), + graph, + counter, + } + } + + fn ec_id() -> String { + format!("{}.ABC123", "a".repeat(64)) + } + + fn seed_returning(&self, entry: &KvEntry) { + self.graph + .create(&Self::ec_id(), entry) + .expect("should seed the row"); + } + + fn returning_context(&self, visit: &Visit<'_>) -> EcContext { + let mut context = EcContext::new_for_test_with_ip( + Some(Self::ec_id()), + consent(), + Some("192.0.2.10".to_owned()), + ); + let mut headers = HeaderMap::new(); + if !visit.cookies.is_empty() { + headers.insert( + header::COOKIE, + HeaderValue::from_str(visit.cookies).expect("should build cookie header"), + ); + } + context.identity_mut().set_raw_cookies_for_test(&headers); + context.identity_mut().set_writer(visit.writer); + if let Some(source) = visit.source { + context.set_eid_sync_source(source); + } + context + } + + /// Runs finalize and the after-send flush, as the adapter does, and + /// returns the response, the flush outcome and the writes this visit made. + fn finish( + &self, + mut context: EcContext, + visit: &Visit<'_>, + ) -> (Response, FlushOutcome, WriteCounts) { + let before = self.counter.writes(); + let mut response = Response::builder() + .status(visit.status) + .body(EdgeBody::empty()) + .expect("should build response"); + ec_finalize_response( + &self.settings, + &mut context, + Some(&self.graph), + &self.registry, + None, + visit.sharedid, + &mut response, + ); + let outcome = run_after_send(&AfterSendInput { + settings: &self.settings, + registry: &self.registry, + services: &self.services, + graph: &self.graph, + context: &context, + status: visit.status, + rate_limiter: None, + }); + let after = self.counter.writes(); + let delta = WriteCounts { + lookups: after.lookups - before.lookups, + add: after.add - before.add, + if_generation_match: after.if_generation_match - before.if_generation_match, + overwrite: after.overwrite - before.overwrite, + }; + (response, outcome, delta) + } + + fn visit(&self, visit: &Visit<'_>) -> (Response, FlushOutcome, WriteCounts) { + let context = self.returning_context(visit); + self.finish(context, visit) + } +} + +fn navigation<'a>(cookies: &'a str, sharedid: Option<&'a str>) -> Visit<'a> { + Visit { + writer: true, + source: Some(EidSyncSource::Navigation), + cookies, + sharedid, + status: StatusCode::OK, + } +} + +fn live_row() -> KvEntry { + let mut entry = KvEntry::minimal("z.example.com", "x", 1); + entry.network = Some(KvNetwork { + cluster_size: Some(1), + }); + entry +} + +fn s_cookie_header(response: &Response) -> String { + response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .find(|value| value.starts_with("__Host-ts-eids-s=") && !value.contains("Max-Age=0")) + .and_then(|value| value.split(';').next()) + .expect("should set ts-eids-s") + .to_owned() +} + +#[test] +fn new_visitor_navigation_writes_once() { + let world = World::new(); + let mut context = + EcContext::new_for_test_with_ip(None, consent(), Some("192.0.2.10".to_owned())); + context.identity_mut().set_writer(true); + context.seed_generation_from_cookies(&world.registry, None, Some("shared-cookie-id")); + let before = world.counter.writes(); + context + .generate_if_needed(&world.settings, Some(&world.graph)) + .expect("should create the row"); + let visit = navigation("", Some("shared-cookie-id")); + let (response, outcome, finish) = world.finish(context, &visit); + + assert_eq!( + world.counter.writes().add - before.add, + 1, + "one Add: the create" + ); + assert_eq!( + finish.total(), + 0, + "finalize and the flush write nothing more" + ); + assert_eq!( + outcome, + FlushOutcome::Skipped, + "a new visitor's flush is skipped" + ); + assert!( + response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .any(|value| value + .to_str() + .is_ok_and(|value| value.starts_with("ts-ec="))), + "the EC cookie is set" + ); +} + +#[test] +fn returning_visitor_with_changes_writes_once() { + let world = World::new(); + world.seed_returning(&live_row()); + + let (_, outcome, writes) = world.visit(&navigation("", Some("new-shared-id"))); + + assert_eq!(outcome, FlushOutcome::Written, "flushed"); + assert_eq!(writes.if_generation_match, 1, "one conditional write"); + assert_eq!(writes.total(), 1, "and nothing else"); + assert_eq!(writes.lookups, 1, "one read, reused after send"); +} + +#[test] +fn returning_visitor_with_nothing_changed_writes_nothing() { + let world = World::new(); + let mut entry = live_row(); + entry + .ids + .insert(SHARED.to_owned(), KvPartnerId::browser("same-id", Some(1))); + world.seed_returning(&entry); + + let (_, outcome, writes) = world.visit(&navigation("", Some("same-id"))); + + assert_eq!(writes.total(), 0, "no write"); + assert!( + matches!(outcome, FlushOutcome::NoChange | FlushOutcome::Skipped), + "nothing to flush: {outcome}" + ); +} + +#[test] +fn revalidation_304_with_changes_writes_once() { + let world = World::new(); + world.seed_returning(&live_row()); + let visit = Visit { + status: StatusCode::NOT_MODIFIED, + ..navigation("", Some("new-shared-id")) + }; + + let (_, outcome, writes) = world.visit(&visit); + + assert_eq!(outcome, FlushOutcome::Written, "a 304 flushes"); + assert_eq!(writes.total(), 1, "one write"); +} + +#[test] +fn returning_navigation_answered_3xx_4xx_or_5xx_writes_nothing() { + for status in [ + StatusCode::FOUND, + StatusCode::NOT_FOUND, + StatusCode::INTERNAL_SERVER_ERROR, + ] { + let world = World::new(); + world.seed_returning(&live_row()); + let visit = Visit { + status, + ..navigation("", Some("new-shared-id")) + }; + let (_, _, writes) = world.visit(&visit); + assert_eq!(writes.total(), 0, "{status} writes nothing"); + } +} + +#[test] +fn returning_non_writer_navigations_write_nothing() { + for (name, source) in [ + ( + "prefetch, iframe or no Sec-Fetch-Dest", + Some(EidSyncSource::Navigation), + ), + ("auction", Some(EidSyncSource::Auction)), + ("page-bids", Some(EidSyncSource::PageBids)), + ("identify, proxy or assets", None), + ] { + let world = World::new(); + world.seed_returning(&live_row()); + let visit = Visit { + writer: false, + source, + cookies: "", + sharedid: Some("new-shared-id"), + status: StatusCode::OK, + }; + let (_, outcome, writes) = world.visit(&visit); + assert_eq!(writes.total(), 0, "{name}: no write"); + assert_eq!(outcome, FlushOutcome::Skipped, "{name}: no flush"); + } +} + +#[test] +fn cas_conflict_on_the_flush_is_one_attempt() { + let world = World::new(); + world.seed_returning(&live_row()); + world.counter.conflict_next(1); + + let (_, outcome, writes) = world.visit(&navigation("", Some("new-shared-id"))); + + assert_eq!(outcome, FlushOutcome::WriteFailed, "the race is lost"); + assert_eq!(writes.if_generation_match, 1, "one attempt, not retried"); +} + +#[test] +fn recent_cookie_with_nothing_due_reads_zero_times() { + let world = World::new(); + world.seed_returning(&live_row()); + let (first, _, _) = world.visit(&navigation("", None)); + let cookie = s_cookie_header(&first); + + let (_, outcome, writes) = world.visit(&navigation(&cookie, None)); + + assert_eq!(writes.lookups, 0, "no KV read"); + assert_eq!(writes.total(), 0, "no write"); + assert_eq!(outcome, FlushOutcome::Skipped, "nothing to flush"); +} + +#[test] +fn body_only_source_reaches_kv_on_the_next_navigation() { + let world = World::new(); + world.seed_returning(&live_row()); + let auction = Visit { + writer: false, + source: Some(EidSyncSource::Auction), + cookies: "", + sharedid: None, + status: StatusCode::OK, + }; + let mut context = world.returning_context(&auction); + context.set_kv_snapshot(world.graph.load_snapshot(&World::ec_id())); + context.set_client_eids(vec![crate::openrtb::Eid::new( + GAP.to_owned(), + vec![crate::openrtb::Uid { + id: "body-only".to_owned(), + atype: Some(1), + ext: None, + }], + )]); + context.identity_mut().set_client_eids_from_body(true); + let (auction_response, _, auction_writes) = world.finish(context, &auction); + assert_eq!(auction_writes.total(), 0, "/auction writes nothing"); + let cookie = s_cookie_header(&auction_response); + + let (_, outcome, writes) = world.visit(&navigation(&cookie, None)); + + assert_eq!( + outcome, + FlushOutcome::Written, + "the next navigation flushes" + ); + assert_eq!(writes.total(), 1, "in one write"); + let (stored, _) = world + .graph + .get(&World::ec_id()) + .expect("should read") + .expect("should exist"); + assert_eq!( + stored.ids[GAP].uid, "body-only", + "the body-only source reached KV" + ); +} + +#[test] +fn refused_flush_keeps_staged_entries_for_the_next_navigation() { + let world = World::new(); + world.seed_returning(&live_row()); + let mut context = world.returning_context(&Visit { + writer: false, + source: Some(EidSyncSource::Auction), + cookies: "", + sharedid: None, + status: StatusCode::OK, + }); + context.set_kv_snapshot(world.graph.load_snapshot(&World::ec_id())); + context.set_client_eids(vec![crate::openrtb::Eid::new( + GAP.to_owned(), + vec![crate::openrtb::Uid { + id: "body-only".to_owned(), + atype: Some(1), + ext: None, + }], + )]); + context.identity_mut().set_client_eids_from_body(true); + let auction = Visit { + writer: false, + source: Some(EidSyncSource::Auction), + cookies: "", + sharedid: None, + status: StatusCode::OK, + }; + let (auction_response, _, _) = world.finish(context, &auction); + let cookie = s_cookie_header(&auction_response); + + world.counter.conflict_next(1); + let (refused_response, refused, _) = world.visit(&navigation(&cookie, None)); + assert_eq!(refused, FlushOutcome::WriteFailed, "the flush is refused"); + let still_staged = refused_response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .any(|value| { + value + .to_str() + .is_ok_and(|value| value.starts_with("__Host-ts-eids-s=")) + }); + let next_cookie = if still_staged { + s_cookie_header(&refused_response) + } else { + cookie + }; + + let (_, outcome, _) = world.visit(&navigation(&next_cookie, None)); + assert_eq!( + outcome, + FlushOutcome::Written, + "the next navigation writes it" + ); + let (stored, _) = world + .graph + .get(&World::ec_id()) + .expect("should read") + .expect("should exist"); + assert_eq!(stored.ids[GAP].uid, "body-only", "nothing was lost"); +} + +#[test] +fn deleted_row_is_found_and_rotated_before_send() { + let world = World::new(); + let mut context = world.returning_context(&navigation("", None)); + context.set_recovery_eligible(true); + let visit = navigation("", None); + let before = world.counter.writes(); + let (response, _, _) = world.finish(context, &visit); + + assert_eq!( + world.counter.writes().add - before.add, + 1, + "one create for the rotation" + ); + assert!( + response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .any(|value| value + .to_str() + .is_ok_and(|value| value.starts_with("ts-ec="))), + "a new EC cookie is set" + ); +} diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 57dfb48eee..0fd206630d 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -25,6 +25,8 @@ pub mod flush; pub(crate) mod generic_pull; pub mod ids; pub mod input; +#[cfg(test)] +mod invariant_tests; pub(crate) mod mutation; pub(crate) mod policy; pub mod provider; From 9080fb80333d17001d812877ac47bdc2f821fa3c Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:31:23 +0530 Subject: [PATCH 31/33] Remove the temporary dead-code allow from the identity module Test-only helpers are gated on cfg(test). The capture, hold-mode and breaker hooks the providers change uses carry a targeted dead_code expectation, which fails once that change starts using them. Signed-off-by: dhruv8sh --- .../src/ec/identity/cookie.rs | 3 ++ .../src/ec/identity/coordinator.rs | 17 +++++++++-- .../src/ec/identity/descriptor.rs | 23 +++++++++++++++ .../src/ec/identity/finalize.rs | 1 - .../src/ec/identity/flush.rs | 1 - .../src/ec/identity/input.rs | 14 ++++++++-- .../src/ec/identity/mod.rs | 7 ----- .../src/ec/identity/mutation.rs | 1 + .../src/ec/identity/request.rs | 1 + .../src/ec/identity/test_support.rs | 28 ------------------- 10 files changed, 54 insertions(+), 42 deletions(-) diff --git a/crates/trusted-server-core/src/ec/identity/cookie.rs b/crates/trusted-server-core/src/ec/identity/cookie.rs index 20ff04f082..f2f9d356f7 100644 --- a/crates/trusted-server-core/src/ec/identity/cookie.rs +++ b/crates/trusted-server-core/src/ec/identity/cookie.rs @@ -316,6 +316,7 @@ impl SEntry { } /// A provider entry staged at `sa`, replacing revision `pv`. + #[cfg(test)] #[must_use] pub(crate) fn staged_provider( s: usize, @@ -361,6 +362,7 @@ impl SEntry { } /// The UID, absent on a stub. + #[cfg(test)] #[must_use] pub(crate) fn uid(&self) -> Option<&str> { self.u.as_deref() @@ -939,6 +941,7 @@ pub(crate) struct OpenedRemovals { } /// Opens every `__Host-ts-eids-r-*` cookie for this EC. +#[cfg(test)] #[must_use] pub(crate) fn open_r_cookies( keys: &IdentityKeys, diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index 749b0f282a..2470771e05 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -76,6 +76,7 @@ pub(crate) struct EligibilityView { impl EligibilityView { /// A view with nothing stored. + #[cfg(test)] #[must_use] pub(crate) fn empty() -> Self { Self::default() @@ -252,6 +253,13 @@ impl EligibilityView { #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum Phase { /// Before the auction. + #[cfg_attr( + not(test), + expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + ) + )] Hold, /// The phase-2 dry run of an after-send provider. AfterSendDryRun, @@ -313,7 +321,6 @@ pub(crate) enum Eligibility { /// The request-wide inputs of an eligibility decision. pub(crate) struct EligibilityRequest<'a> { - pub(crate) identity: &'a IdentitySettings, pub(crate) view: &'a EligibilityView, pub(crate) facts: &'a RequestFacts, pub(crate) consent: &'a ConsentContext, @@ -504,6 +511,13 @@ pub(crate) enum CallOutcome { /// A transport error, timeout, 429 or 5xx. TransportFailure, /// The circuit breaker was open. + #[cfg_attr( + not(test), + expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + ) + )] BreakerOpen, /// Not sent for a reason the stored row does not show, such as the /// per-visitor burst cap. Phase 4 backs a marked provider off by a minute. @@ -860,7 +874,6 @@ mod tests { settings, phase, &EligibilityRequest { - identity: &self.identity, view, facts, consent: &self.consent, diff --git a/crates/trusted-server-core/src/ec/identity/descriptor.rs b/crates/trusted-server-core/src/ec/identity/descriptor.rs index 145c898a86..82a1f1d105 100644 --- a/crates/trusted-server-core/src/ec/identity/descriptor.rs +++ b/crates/trusted-server-core/src/ec/identity/descriptor.rs @@ -384,6 +384,10 @@ impl Resolves { } /// Inputs passed when available. + #[expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + )] #[must_use] pub(crate) fn wants(&self) -> &BTreeSet { &self.wants @@ -452,6 +456,13 @@ impl Captures { } /// Whether a proxied request matches a rule. + #[cfg_attr( + not(test), + expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + ) + )] #[must_use] pub(crate) fn matches(&self, method: &Method, path: &str) -> bool { self.paths @@ -464,18 +475,30 @@ impl Captures { } /// Exact path rules. + #[expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + )] #[must_use] pub(crate) fn paths(&self) -> &[(Method, String)] { &self.paths } /// Prefix rules. + #[expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + )] #[must_use] pub(crate) fn prefixes(&self) -> &[String] { &self.prefixes } /// Declared body consent pointers. + #[expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + )] #[must_use] pub(crate) fn consent_pointers(&self) -> &[&'static str] { &self.consent_pointers diff --git a/crates/trusted-server-core/src/ec/identity/finalize.rs b/crates/trusted-server-core/src/ec/identity/finalize.rs index 39dd9c0fc9..6cab943bd5 100644 --- a/crates/trusted-server-core/src/ec/identity/finalize.rs +++ b/crates/trusted-server-core/src/ec/identity/finalize.rs @@ -187,7 +187,6 @@ pub(crate) fn candidates( ) -> Vec { let facts = writer_facts(context, finalize); let request = EligibilityRequest { - identity: &finalize.identity, view, facts: &facts, consent: context.consent(), diff --git a/crates/trusted-server-core/src/ec/identity/flush.rs b/crates/trusted-server-core/src/ec/identity/flush.rs index aed7d5c610..04efb0e5f1 100644 --- a/crates/trusted-server-core/src/ec/identity/flush.rs +++ b/crates/trusted-server-core/src/ec/identity/flush.rs @@ -236,7 +236,6 @@ fn dispatch_marked( ); let facts = writer_facts(context, finalize); let request = EligibilityRequest { - identity: &finalize.identity, view: &view, facts: &facts, consent: context.consent(), diff --git a/crates/trusted-server-core/src/ec/identity/input.rs b/crates/trusted-server-core/src/ec/identity/input.rs index b9aeae367c..180b3ba005 100644 --- a/crates/trusted-server-core/src/ec/identity/input.rs +++ b/crates/trusted-server-core/src/ec/identity/input.rs @@ -159,6 +159,7 @@ impl fmt::Debug for HashedEmail { impl HashedEmail { /// From digest bytes. No code in v1 constructs one outside tests. + #[cfg(test)] #[must_use] pub(crate) fn from_bytes(bytes: [u8; 32]) -> Self { Self(bytes) @@ -288,10 +289,10 @@ impl RandomSource for ThreadRandom { } /// Deterministic bytes for tests: xorshift64* from a seed. -#[cfg(any(test, feature = "test-utils"))] +#[cfg(test)] pub(crate) struct SeededRandom(std::sync::Mutex); -#[cfg(any(test, feature = "test-utils"))] +#[cfg(test)] impl SeededRandom { /// A generator seeded with `seed`. A zero seed is replaced by one, /// because xorshift never leaves zero. @@ -300,7 +301,7 @@ impl SeededRandom { } } -#[cfg(any(test, feature = "test-utils"))] +#[cfg(test)] impl RandomSource for SeededRandom { fn fill(&self, bytes: &mut [u8]) { let mut state = self.0.lock().expect("should lock the seed"); @@ -632,6 +633,13 @@ impl fmt::Debug for CaptureInput<'_> { impl<'a> CaptureInput<'a> { /// The input for one capture. + #[cfg_attr( + not(test), + expect( + dead_code, + reason = "the providers change (passive capture, hold mode, the breaker) uses this" + ) + )] #[must_use] pub(crate) fn new(parts: CaptureParts<'a>) -> Self { Self { parts } diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 0fd206630d..6ec31380e6 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -6,13 +6,6 @@ //! KV write-back, so every page load makes at most one browser-driven KV //! write. -// Several crate-private helpers are only called once the coordinator is wired -// into requests later in this series. Remove this allow when that lands. -#![allow( - dead_code, - reason = "crate-private identity helpers are wired into requests by later commits" -)] - pub(crate) mod catalog; pub mod config; pub(crate) mod cookie; diff --git a/crates/trusted-server-core/src/ec/identity/mutation.rs b/crates/trusted-server-core/src/ec/identity/mutation.rs index 85a1a53631..c9747256a4 100644 --- a/crates/trusted-server-core/src/ec/identity/mutation.rs +++ b/crates/trusted-server-core/src/ec/identity/mutation.rs @@ -199,6 +199,7 @@ impl IdentityMutation { } /// Whether it holds a removal, opt-out, clear or `iv` hash. + #[cfg(test)] #[must_use] pub(crate) fn has_removals(&self) -> bool { self.partner_ids.iter().any(PartnerIdChange::is_removal) diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index f6b45fc579..b27f0b994d 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -37,6 +37,7 @@ impl RawIdentityCookies { } /// Whether the request carried either cookie. + #[cfg(test)] #[must_use] pub(crate) fn has_any(&self) -> bool { self.s.is_some() || !self.r.is_empty() diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index 8c49be0ac1..0018d31e9e 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -207,20 +207,6 @@ impl<'p> ProviderHarness<'p> { } } - /// Sets the client IP. - #[must_use] - pub(crate) fn with_client_ip(mut self, ip: &str) -> Self { - self.client_ip = Some(ip.parse().expect("should parse the client IP")); - self - } - - /// Sets the user agent. - #[must_use] - pub(crate) fn with_user_agent(mut self, user_agent: &str) -> Self { - self.user_agent = Some(user_agent.to_owned()); - self - } - /// Sets the EC ID, which only generic pull may read. #[must_use] pub(crate) fn with_ec_id(mut self, ec_id: &str) -> Self { @@ -228,20 +214,6 @@ impl<'p> ProviderHarness<'p> { self } - /// Stores a record for `source`, which makes its kind `Refresh`. - #[must_use] - pub(crate) fn with_stored( - mut self, - source: &str, - record: &crate::ec::kv_types::KvPartnerId, - ) -> Self { - self.stored.insert( - source.parse().expect("should parse source"), - crate::ec::identity::StoredId::from_record(record), - ); - self - } - fn available_inputs(&self) -> std::collections::BTreeSet { let mut available = std::collections::BTreeSet::new(); if self.client_ip.is_some() { From 67ae2f0aa1dc0077e3eaed7fb643574489884ce1 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:32:27 +0530 Subject: [PATCH 32/33] Log identity cookie sizes and the flush outcome per request One measurement line per identity request reports the inbound Cookie header size, the number and total size of removal cookies, and whether the writer read KV and queued changes. The adapter logs each flush outcome. No ID or cookie value is logged. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/main.rs | 2 +- crates/trusted-server-core/src/ec/finalize.rs | 2 + .../src/ec/identity/request.rs | 58 +++++++++++++++++++ 3 files changed, 61 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 8b7e2a80cf..a870d161c1 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -640,7 +640,7 @@ fn run_identity_after_send( status, rate_limiter: Some(&limiter), }); - log::debug!("Identity flush outcome: {outcome}"); + log::info!("Identity flush measurement: outcome={outcome}"); } fn prepare_identity_after_send( diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index 91e21a15e9..de5034e919 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -156,6 +156,8 @@ pub fn ec_finalize_response( reconcile_pull_sync_marker(settings, registry, ec_context, response); if let Some(identity) = &identity { emit_identity_cookies(ec_context, identity, kv, response); + let kv_read = !matches!(ec_context.kv_snapshot(), EcKvSnapshot::NotRead); + log::info!("{}", ec_context.identity().measurement(kv_read)); } // Ordinary returning-user page views no longer refresh the browser diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index b27f0b994d..0f206143cb 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -124,6 +124,7 @@ pub(crate) struct RequestIdentity { seed: Vec, client_eids_from_body: bool, marked: Vec, + cookie_header_bytes: usize, } impl RequestIdentity { @@ -136,10 +137,35 @@ impl RequestIdentity { user_agent: header("user-agent").map(str::to_owned), dnt: header("dnt").is_some_and(|value| value.trim() == "1"), raw: RawIdentityCookies::capture(headers), + cookie_header_bytes: headers + .get_all(http::header::COOKIE) + .iter() + .map(|value| value.as_bytes().len()) + .sum(), ..Self::default() } } + /// One log line per identity request, with no IDs or cookie values: the + /// inbound `Cookie` header size, the removal cookies' count and total + /// size, and whether this request is the writer and read KV. + pub(crate) fn measurement(&self, kv_read: bool) -> String { + format!( + "Identity measurement: writer={} kv_read={} cookie_header_bytes={} \ + removal_cookies={} removal_cookie_bytes={} queued_changes={}", + self.writer, + kv_read, + self.cookie_header_bytes, + self.raw.r.len(), + self.raw + .r + .iter() + .map(|(name, value)| name.len() + value.len() + 1) + .sum::(), + !self.mutation.is_empty(), + ) + } + /// Replaces the captured cookies, for tests that build a context by hand. #[cfg(test)] pub(crate) fn set_raw_cookies_for_test(&mut self, headers: &HeaderMap) { @@ -274,3 +300,35 @@ impl RequestIdentity { &mut self.staged } } + +#[cfg(test)] +mod tests { + use http::{HeaderMap, HeaderValue, header}; + + use super::*; + + #[test] + fn measurement_reports_sizes_without_values() { + let mut headers = HeaderMap::new(); + headers.append(header::COOKIE, HeaderValue::from_static("a=1")); + headers.append( + header::COOKIE, + HeaderValue::from_static("__Host-ts-eids-r-0a1b2c3d=secret-value"), + ); + let identity = RequestIdentity::capture(&headers); + let line = identity.measurement(true); + assert!( + line.contains("cookie_header_bytes=41"), + "sums every Cookie field: {line}" + ); + assert!( + line.contains("removal_cookies=1"), + "counts removal cookies: {line}" + ); + assert!( + line.contains("removal_cookie_bytes=38"), + "sizes them: {line}" + ); + assert!(!line.contains("secret-value"), "never logs a value"); + } +} From 56ddd07c1e5d93ec179a79b2373c05b065e91812 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 22 Oct 2026 18:34:04 +0530 Subject: [PATCH 33/33] Document the single identity write and the new identity cookies Signed-off-by: dhruv8sh --- CHANGELOG.md | 3 +++ docs/guide/api-reference.md | 8 ++++++++ docs/guide/configuration.md | 40 ++++++++++++++++++++++++++++++++++++- docs/guide/edge-cookies.md | 31 +++++++++++++++++++++++++++- 4 files changed, 80 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9829af8b32..6882b7689e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- A page load now writes the EC identity row at most once. Only the writer request, a real browser's top-level document navigation, writes, after the page has been sent; a failed write is not retried. `/auction`, `/_ts/page-bids` and `/_ts/api/v1/identify` no longer write KV. `/auction` stages the body EIDs that KV lacks in the new sealed `__Host-ts-eids-s` cookie, and the next navigation writes them; `/_ts/page-bids` EIDs reach KV through `ts-eids` at the next navigation. Removals and opt-outs not yet in KV travel in per-batch `__Host-ts-eids-r-*` cookies. A new visitor's row creation now carries its browser EIDs, so it stays one write. +- Pull sync partners now run as server-side identity providers. They are called after send on a returning visitor's writer navigation, from the visitor's second page load, back off per visitor on errors instead of retrying on every request, and are rate-limited per visitor instead of per client IP. `ec.pull_sync_concurrency` is deprecated in favour of `[ec.identity].max_concurrency`. Rollback: once providers write metadata, do not roll back past this release; an older reader drops the new KV fields on its next write. +- KV IDs sent to bidders, the admin view and `/_ts/api/v1/identify` now pass one servability check: expired IDs, and IDs whose writing provider is disabled, opted out or blocked by its consent policy, are never served. `/_ts/api/v1/identify` uses the request's consent, never the row's stored consent snapshot. Stored provider `atype` and UID-level `ext` are emitted. - **Breaking:** The publisher origin no longer receives the `ts-eids` cookie, and Prebid Server no longer receives any Trusted Server identity cookie (`ts-ec`, `ts-eids`, `ts-ec-pull-complete`, and the server-owned `__Host-ts-eids-s` and `__Host-ts-eids-r-*` cookies that a later release introduces). The origin still receives `ts-ec`. Origin code that read `ts-eids` should read the `x-ts-eids` response header instead. A Prebid Server `Cookie` header that is not valid UTF-8 is now dropped instead of forwarded. This release is the rollback floor for the server-side identity provider work. - **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER__GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header. - **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias. diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md index 721226f38b..0937eb5b79 100644 --- a/docs/guide/api-reference.md +++ b/docs/guide/api-reference.md @@ -325,6 +325,9 @@ optional and must be `json` when present. A request is admitted only when request carries `X-TSJS-Page-Bids`. Every `OPTIONS` request is denied so a cross-origin caller cannot acquire permission to send that non-simple header. +`/_ts/page-bids` never writes KV. Browser EIDs in `ts-eids` reach KV at the +visitor's next full page load. + **Response schema:** `200 application/json` with `{ "slots": [...], "bids": {...} }`. The ad-stack kill switch, an auction kill switch, or denied consent returns empty collections. Unknown `format` returns `400`; a rejected GET or @@ -369,6 +372,11 @@ below `[publisher].domain` receive reflected credentialed CORS for The endpoint is Fastly-only and requires the EC store and a statically configured `[[ec.partners]]` token. +The endpoint never writes KV. It reports a stored `cluster_size`, or computes +one without storing it. It returns only a UID the shared servability check +allows for this request: an expired ID, or one whose writing provider's +consent policy fails under the request's consent, is omitted. + **Request:** - Uses `ts-ec` cookie and consent signals diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index eeed15c2c4..5302de83ad 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -643,7 +643,7 @@ be at least 32 bytes. Keep it stable to preserve EC identifier continuity. | ------------------------- | -------------- | -------- | ----------------------------------------------------------------------- | | `passphrase` | String | Yes | Publisher passphrase used as HMAC key | | `ec_store` | String or null | No | Fastly KV store name for EC identity graph and withdrawal state | -| `pull_sync_concurrency` | Integer | No | Maximum concurrent pull-sync requests per organic response | +| `pull_sync_concurrency` | Integer | No | Deprecated: use `[ec.identity].max_concurrency` | | `cluster_trust_threshold` | Integer | No | Cluster size threshold for identity trust decisions | | `cluster_recheck_secs` | Integer | No | Legacy compatibility setting; cluster rechecks no longer use timestamps | | `partners` | Array | No | Static partner registry entries | @@ -684,6 +684,44 @@ TRUSTED_SERVER__EC__PASSPHRASE=ec_passphrase TRUSTED_SERVER__EC__EC_STORE=ec_identity_store ``` +### `[ec.identity]` + +Coordinator policy for server-side identity providers. Pull partners +(`[[ec.partners]]` with `pull_sync_enabled = true`) run as providers. Every +table is optional; the defaults below apply when it is absent. + +| Field | Default | Description | +| -------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | +| `max_concurrency` | `4` | Providers dispatched at once. Replaces `ec.pull_sync_concurrency`, which is still honoured when changed and unset here | +| `hold_budget_ms` | `100` | Shared wall-clock budget for hold-mode providers. At most 1,000 | +| `hold_retry_sec` | `300` | Wait after a hold-mode call is cut off by the budget | +| `max_response_bytes` | `65536` | Largest provider response, before and after decoding | +| `pending_grace_sec` | `300` | How long a pending marker outlives its provider's timeout | +| `staged_max_age_sec` | `604800` | Staged cookie entries older than this are not written | +| `kv_revalidate_sec` | `3600` | A cookie not confirmed against KV for this long forces a KV read. Must be below 86,400 | + +Per-provider tables are keyed by provider ID. A pull partner's ID is `pull-` +plus its source domain with dots replaced by `-`, for example +`pull-ssp-example-com`. + +| Field | Default | Description | +| ----------------------------- | ------------------------------------------ | ------------------------------------------------------------------- | +| `mode` | `"after_send"` | `"hold"` (before the auction) or `"after_send"` | +| `timeout_ms` | `hold_budget_ms` in hold mode, else `1000` | Per-call timeout. In hold mode it must not exceed `hold_budget_ms` | +| `sources` | every declared source | Narrows the sources the provider may write | +| `refresh_after_sec` | `86400` | Refresh interval without a provider hint, and the wait after a skip | +| `max_calls_per_minute` | `1`; a pull partner's legacy hourly limit | Burst cap per visitor | +| `breaker_failures_per_minute` | `50` | Transport failures per minute that pause the provider | +| `cookie_priority` | `0` | Lower is dropped first when `__Host-ts-eids-s` is over its cap | + +```toml +[ec.identity] +kv_revalidate_sec = 3600 + +[ec.identity.providers.pull-ssp-example-com] +max_calls_per_minute = 2 +``` + ### Field Details #### `passphrase` diff --git a/docs/guide/edge-cookies.md b/docs/guide/edge-cookies.md index 61acae11fa..6caf5b1773 100644 --- a/docs/guide/edge-cookies.md +++ b/docs/guide/edge-cookies.md @@ -319,6 +319,35 @@ the publisher origin's `Cookie` header: consent-forwarding mode. A `Cookie` header that is not valid UTF-8 is dropped rather than forwarded. +## Identity Cookies and the Single KV Write + +Only a full page load writes the EC identity row, once, after the page has +been sent. That request is the **writer**: a real browser's top-level +document navigation (`Sec-Fetch-Dest: document`, not a prefetch) to the +publisher or an integration proxy. Every other request, including `/auction`, +`/_ts/page-bids`, `/_ts/api/v1/identify`, iframes and prefetches, never writes +KV. It saves what it learned in two server-owned cookies instead, and the next +writer writes it: + +- **`__Host-ts-eids-s`** caches provider-collected IDs and holds entries other + requests staged, such as browser EIDs that `/auction` saw in its request + body and KV does not have yet. At most 2,048 bytes. +- **`__Host-ts-eids-r-`** holds one response's removals and opt-outs not + yet in KV. Each removal batch gets its own cookie, so a concurrent request + cannot lose it. At most 512 bytes each. A batch that does not fit, or meets + four such cookies, is written to KV before send instead. + +Both are sealed with AES-256-GCM, bound to the visitor's EC ID, host-only, +`HttpOnly`, `Secure` and `SameSite=Lax`, and last a year. The TSJS shim never +reads or writes them, and neither reaches the publisher origin or Prebid +Server. A cookie copied from another visitor, or left after an EC rotation, +does not open and is ignored. A consent withdrawal expires `__Host-ts-eids-s`. + +A new visitor's row is still created before the EC cookie is set, carrying the +browser's EIDs, so a new visitor's page load is also one write. Two writes +outside that rule remain: a consent withdrawal's tombstone, and an overflowing +removal batch. + ## Operational Notes - Rotate the secret periodically. Rotation produces a new 64-hex base @@ -333,7 +362,7 @@ the publisher origin's `Cookie` header: - When consent is blocked but not explicitly withdrawn, Trusted Server strips EC response headers for that request but leaves any existing `ts-ec` cookie intact; cookie expiry and tombstones happen only on explicit withdrawal. - `/_ts/api/v1/identify` is read-oriented and returns identity enrichment for the authenticated partner. It computes `cluster_size` only when the EC entry does not already store one. - `/_ts/api/v1/batch-sync` validates every input, groups valid mappings by normalized EC ID, and applies the last valid UID for each group once. Group outcomes still account for every original input; infrastructure failures reject the failing and remaining groups. Mapping timestamps remain required for API compatibility but do not order writes. See the [API Reference](/guide/api-reference) for the complete contract. -- Pull sync fills missing partner UIDs only. Existing partner UIDs are not periodically refreshed because EC entries no longer store per-partner sync timestamps. +- Pull partners fill missing partner UIDs only, and run after a returning visitor's page load has been sent, from the visitor's second page load. A failing partner backs off per visitor: a few minutes after a transport failure, up to an hour, and an hour after a rejected request, up to a day. ## Next Steps