Skip to content

Add design spec for server-side identity providers - #1428

Open
dhruv8sh wants to merge 70 commits into
spec/1246-prebid-eid-fidelityfrom
spec/1246-server-side-identity-providers
Open

dhruv8sh wants to merge 70 commits into
spec/1246-prebid-eid-fidelityfrom
spec/1246-server-side-identity-providers

Conversation

@dhruv8sh

@dhruv8sh dhruv8sh commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Adds a design spec, for approval before any implementation. It lets Trusted Server collect EIDs server-side through an IdentityProvider trait, so LiveRamp, Lockr, ID5, UID2 and other vendors can plug in without changing the core. It builds on the EID fidelity spec in Preserve Prebid EID fidelity from browser capture to bidder request #1416 and is stacked on that branch.
  • Holds every page load to one KV write. Only the navigation writes; /auction, /_ts/page-bids and proxy calls save their changes into a sealed server cookie (ts-eids-s), and removals into per-batch removal cookies (ts-eids-r). Fastly's write limit for one key is about one per second and can be stricter, so a failed write is never retried.
  • The trait is synchronous and object-safe. Its one required method is descriptor(); request, response and capture are default methods, so new capabilities are additive. The LiveRamp, Lockr and UID2 providers in section 12 illustrate the API only. Each vendor's real code and behaviour are settled in its implementation plan.

Changes

File Change
docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md New spec: the trait and coordinator, the one-write protocol, storage and cookies, merge and ownership, errors, privacy, configuration, adapters, example providers, testing, rollout, open questions, decisions log and a review log of nine rounds

Closes

Part of #1246. #1416 closes it; this spec adds the server-side provider design on top.

Spec

Spec: docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md (this PR)

Test plan

  • cargo test-fastly && cargo test-axum
  • cargo clippy-fastly && cargo clippy-axum
  • cargo fmt --all -- --check
  • JS tests: cd crates/trusted-server-js/lib && npx vitest run
  • JS format: cd crates/trusted-server-js/lib && npm run format
  • Docs format: cd docs && npm run format
  • WASM build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
  • Manual testing via fastly compute serve
  • Other: docs-only change, so the Rust and JS gates do not apply

Checklist

  • Feature PRs: spec linked above, ending in a review log of two or more agent review rounds
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses tracing macros (not println!)
  • New code has tests
  • No secrets or credentials committed

Define an IdentityProvider trait and coordinator so vendors such as LiveRamp, Lockr, UID2 and ID5 can resolve or capture EIDs server-side, with owner-configured precedence per source.

Hold every page load to one KV write: only the navigation writes, stagers save changes into the sealed ts-eids-s cookie and removals into per-batch ts-eids-r cookies, and a failed write is never retried.

Include illustrative LiveRamp, Lockr and UID2 providers, the test harness, configuration, rollout steps and the log of nine review rounds.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh dhruv8sh self-assigned this Oct 21, 2026
…de-identity-providers

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh dhruv8sh linked an issue Oct 21, 2026 that may be closed by this pull request

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The capability-based IdentityProvider, descriptor, coordinator, and existing integration registration are the right architecture. Requesting changes because the privacy, refresh-state, and removal-persistence contracts have unresolved correctness gaps before implementation.

All seven inline findings are prose-only: their fixes span multiple specification rules and tests, so none carries a one-click suggestion.

Blocking

  • 🔧 [P1] Browser fallback can resend an ID the provider rejected — see inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:2341.
  • 🔧 [P1] Apply writer consent policy to unowned provider-written IDs — see inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:2333.
  • 🔧 [P1] Commit provider IDs and private refresh state atomically — see inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:1805.
  • 🔧 [P1] Preserve refresh eligibility after advertising-ID expiry — see inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:1806.
  • 🔧 [P1] Make removal-cookie overflow durable under concurrency — see inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:2149.
  • 🔧 [P2] Use record versions and explicit privacy-event ordering — see inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:1777.
  • ❓ [P2] Define the visitor-consent context used by /identify — see inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:2367.

Design coverage

No additional umbrella integration trait is needed. The proposed trait already attaches identity capabilities to the integration registry while keeping vendor protocol logic separate from coordinator-owned eligibility, consent, transport, and persistence.

The contract supports single-exchange resolvers, multiple output sources, per-source expiry, private refresh state, and passive response capture. It should be described as extensible for those declared capabilities rather than universal identity-provider support:

  • HEM intake and local publisher-cookie derivation are explicitly outside v1.
  • Multiple sequential exchanges, credential-refresh lifecycles, and multiple UIDs per source are not demonstrated or supported by the current contract.
  • Capture cannot update private state.
  • Stateful providers run only on writer navigations; new visitors run no provider on their first navigation. A new visitor staying in an SPA may therefore get no stateful server-side resolution during the session.
  • Native ownership requires narrowing overlapping capture providers such as Lockr. Parallel passive fallback would require separating collection permission from precedence.

Non-blocking recommendations

  • Retain the trait/coordinator structure.
  • Separate the one-write storage redesign from vendor extensibility in implementation milestones.
  • Validate illustrative vendor examples with primary-protocol request/response fixtures before treating them as conformance proof.
  • Add adversarial state-machine tests for the interleavings in the inline comments.
  • Keep the accepted x-ts-eids page-script exposure explicit; sealed HttpOnly cookies do not hide IDs exposed through that header.

CI Status

  • PASS — integration tests (Fastly EC lifecycle)
  • PASS — browser integration tests
  • PASS — integration tests
  • PASS — cargo test (ts CLI, native) (macos-latest)
  • PASS — cargo check (cloudflare native + wasm32-unknown-unknown)
  • PASS — cargo fmt
  • PASS — format-typescript
  • PASS — prepare integration artifacts
  • PASS — cargo test
  • PASS — cargo check/build/test (spin native + wasm32-wasip1)
  • PASS — docs-proposal scripts
  • PASS — cargo test (cross-adapter parity)
  • PASS — CLAUDE.md symlink guard
  • PASS — vitest
  • PASS — cargo test (axum native)
  • PASS — format-docs
  • PASS — cargo test (ts CLI, native) (ubuntu-latest)

No required checks were reported by the required-check query. All 17 reported checks passed.

Verification scope

Reviewed the full changed specification and relevant existing registration, consent, lookup, and EID paths in an isolated worktree at f0f7a17ddf633d45b1a172c7fe8df0f84d172732. Small executable models confirmed counterexamples to the specified ordering and concurrency rules. This PR contains no implementation; no runtime correctness claim is made and no code was changed.

Comment thread docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md Outdated
Comment thread docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md Outdated
Comment thread docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md Outdated
Comment thread docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md Outdated
Comment thread docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md Outdated
Comment thread docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md Outdated
Filter browser candidates by the owner's invalidations and opt-outs, check the writer's consent policy for every provider-written ID, apply a stateful provider's IDs and state atomically, keep expired records as refresh anchors while refresh credentials last, replace the removal-cookie merge with a synchronous overflow write, give records a random revision with removal-wins ties, and define the consent /identify checks.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh
dhruv8sh requested a review from prk-Jr October 22, 2026 07:34
State that the overflow write, like the withdrawal tombstone, retries only a CAS generation conflict and stops at any other failure, Fastly's per-key limit included, so it never repeats a rate-limited write. Move both round 10 decisions out of the pending table.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This review found no blocking issues in the shape of the provider API: the capability-based IdentityProvider trait, the declarative descriptor, the coordinator and registration through the existing integration builder. The Rust listings type-check in a scratch crate, the trait is object-safe, and the derive claims hold. Requesting changes for four reasons:

  • The cookie-staging protocol does not yet keep "removals win" (G12).
  • Several premises the design leans on are contradicted by the code or by vendor and platform documentation.
  • The plan would not build or test as written.
  • The spec needs splitting into phases and a clean review round under CONTRIBUTING.md.

The ordering and concurrency counterexamples below were reproduced in a small executable model of the 5.4 and 6.2 rules. It uses real AES-GCM sealing with the EC ID as associated data, a KV store whose point reads can lag, and CAS with Fastly's 1 write/sec/item limit.

7 of the inline comments below carry a one-click GitHub suggestion. Use Commit suggestion (or Add suggestion to batch for several at once) to apply them as commits on the PR branch. The remaining comments describe the fix in prose, because the change spans several sections, or the spec and the plan together, and can't be auto-applied.

Blocking

🔧 wrench

  • Split this into phase specs, one PR each (inline at spec 2026-10-20-server-side-identity-providers-design.md:3891)
  • The review log ends on a blocking round, and decisions are still pending (inline at spec 2026-10-20-server-side-identity-providers-design.md:3984)
  • The apply order lets a mutation undo its own removals (inline at spec 2026-10-20-server-side-identity-providers-design.md:1830)
  • "Applied" is inferred from absence, so removal cookies are deleted before they take effect (inline at spec 2026-10-20-server-side-identity-providers-design.md:2179)
  • Host-only removal cookies do not fit a domain-wide EC (inline at spec 2026-10-20-server-side-identity-providers-design.md:2072)
  • Rotating ec.passphrase silently drops every pending removal; reuse the existing XChaCha20-Poly1305 dependency (inline at spec 2026-10-20-server-side-identity-providers-design.md:2087)
  • An after-send opt-out lost to a failed flush may never be repeated (inline at spec 2026-10-20-server-side-identity-providers-design.md:1635)
  • The missing-row rules act on reads the code says are not proof (inline at spec 2026-10-20-server-side-identity-providers-design.md:1909)
  • The removal-cookie budget fails at maximum lengths (inline at spec 2026-10-20-server-side-identity-providers-design.md:2199)
  • Expiry rules leave gaps in G7 and in refresh (inline at spec 2026-10-20-server-side-identity-providers-design.md:1845)
  • The EC ID is a bearer value that partners and the origin hold (inline at spec 2026-10-20-server-side-identity-providers-design.md:2477)
  • strict() is not the strictest policy, and generic pull is exempt from it (inline at spec 2026-10-20-server-side-identity-providers-design.md:652)
  • The opt-out clear rules do not match TCF timestamps or the stored fingerprint (inline at spec 2026-10-20-server-side-identity-providers-design.md:2542)
  • Stripping ts-eids from the origin breaks origins with no migration path, and is not needed (inline at spec 2026-10-20-server-side-identity-providers-design.md:2304)
  • Clients that send no Sec-Fetch-Dest never become writers (inline at spec 2026-10-20-server-side-identity-providers-design.md:1468)
  • Provider calls have no per-request or per-IP bound (inline at spec 2026-10-20-server-side-identity-providers-design.md:1322)
  • The cost and latency model rests on assumptions the vendors and the code contradict (inline at spec 2026-10-20-server-side-identity-providers-design.md:1387)
  • The ts-eids-s source index cannot encode most KV-gap entries (inline at spec 2026-10-20-server-side-identity-providers-design.md:2116)
  • Part 1 never wires the real catalog into the servability check (inline at plan 2026-10-22-server-side-identity-providers.md:4893)
  • Plan code that fails clippy or panics at runtime (inline at plan 2026-10-22-server-side-identity-providers.md:6434)
  • The tests that gate Part 2 cannot run as written (inline at plan 2026-10-22-server-side-identity-providers.md:6783)
  • Phase 1 and phase 4 wiring is missing steps the spec requires (inline at plan 2026-10-22-server-side-identity-providers.md:6724)
  • Part 1 is one 27-task PR with broken commits in the middle (inline at plan 2026-10-22-server-side-identity-providers.md:23)
  • Remove the stale duplicate file map (inline at plan 2026-10-22-server-side-identity-providers.md:148)
  • A derived Debug cannot redact (inline at spec 2026-10-20-server-side-identity-providers-design.md:504)
  • Fastly's KV metadata cap is 2,000 bytes (inline at spec 2026-10-20-server-side-identity-providers-design.md:1979)
  • has_vendor_consent already exists (inline at spec 2026-10-20-server-side-identity-providers-design.md:2593)
  • The Lockr proxy logs the query today (inline at spec 2026-10-20-server-side-identity-providers-design.md:2658)
  • Core's test feature is test-utils (inline at spec 2026-10-20-server-side-identity-providers-design.md:3490)

❓ question

  • How does this relate to open draft #1260? (inline at spec 2026-10-20-server-side-identity-providers-design.md:24)
  • Have the vendor protocols v1 depends on been confirmed? (inline at spec 2026-10-20-server-side-identity-providers-design.md:77)

Cross-cutting / body-level findings

  • 🔧 Prior round, re-checked at 21fd172. prk-Jr's seven threads are all marked resolved. Status of each fix:

    Finding Status
    Browser fallback resends a rejected ID Partly closed. Holds on /auction and while ts-eids-r exists, but is bypassed through KV ingest and cookie-only navigation auctions (inline at spec 1830)
    Writer policy for unowned provider-written IDs Closed. Side effect: disabling a writer now hides the UIDs it wrote, which 10.4 does not list
    Atomic stateful ID and state Closed for the stated race. expected_state: None means both "did not read" and "absent", so a state-only result from a call that read no state can overwrite another writer's state. Record Absent or a hash at read time
    Refresh eligibility after expiry Partly closed. Covers stateful owners only (inline at spec 1845)
    Removal-cookie overflow durability Closed for the merge race. The byte budget fails at maximum lengths (spec 2199), and "applied" is inferred from absence (spec 2179)
    Revisions and tie ordering Closed for rv and the stated ties. Clear timestamps can still dismiss a revoke (spec 2542)
    /identify consent context The answer relies on the call carrying the visitor's own cookies, which a partner server can forge (spec 2477)
  • 🔧 The PR description and the branch are stale.

    • The description's Changes table omits the 7,442-line plan, and it says the review log has nine rounds (it has ten). The spec says "Issue: none filed yet" (line 6).
    • The branch is 9 commits behind its base (spec/1246-prebid-eid-fidelity at 9b901f2):
      • The PR head pins Rust 1.95.0, while the base pins 1.98.1.
      • At the base, the shim writes ts-eids only with TCF Purposes 1 and 4 (hasEidPersistenceConsent). Spec 57-58, 213-214 and 294-296 still describe that write as unconditional.
    • Please merge the base and update the description.
  • 🔧 Section 10.4's change lists are incomplete.

    • The pull-sync first-byte timeout drops from 15 s (platform/mod.rs:82, used at ec/pull_sync.rs:238) to the new 1,000 ms after-send default (2708-2710).
    • 10.1 gives max_concurrency = 4; today's default is 3 (settings.rs:587, trusted-server.example.toml:83).
    • "Fair rotation under max_concurrency" (2826-2827) is not what the code does. Every eligible partner is dispatched on each run, and max_concurrency only sizes the batches (ec/pull_sync.rs:279-284). The code comment at :152-154 is wrong too.
    • Today's gates are understated (136-137, 2850). Pull sync also requires pull partners, ec_allowed, allows_eid_persistence (ec/pull_sync.rs:69-78), no valid pull-complete marker (:87) and is_real_browser (adapter-fastly/src/main.rs:623-625).
    • Disabling a pull partner now hides its previously pulled UIDs from the bidstream and /identify, because disabled writers are not served (2410-2413).
    • Partner call volume behind a shared IP rises with the number of visitors once the limiter key becomes per-EC.
    • Generic pull IDs can collide. Replacing dots with - maps a-b.example.com and a.b-example.com to the same pull-a-b-example-com, which 10.2 then rejects as a duplicate. Escape -, or always add the hash suffix.
  • 🔧 Smaller accuracy fixes in the spec.

    • Citations:
      • 2405: the entry.consent.ok check is at auction/endpoints.rs:461-463; cite 449-463.
      • 2792 and 3382: secret_fields() spans config.rs:136-236.
      • 474: settings.rs:235 is :241 at the base.
    • Claims that don't match the code:
      • 2270-2271: only /_ts/page-bids is no-store (publisher.rs:7243). /auction sets no Cache-Control and becomes private only through the Set-Cookie guard.
      • 1885-1886: a deletion tombstone is two writes, the tombstone and its completion marker (ec/kv.rs:1415, :1115-1124).
      • 1806-1809: today's KvPartnerId has no at, so for every existing row the legacy revision hash is a hash of the UID alone.
      • 1352-1355: on Fastly a non-streaming response is read whole, up to 10 MiB (adapter-fastly/src/platform.rs:413-429), before max_response_bytes is checked. That makes it an acceptance check, not a read bound.
      • 2073: ts-ec and the GPT diagnostics cookie are also server-set, so the pull marker is not "the repo's other server-set cookie".
    • Write counts and wording:
      • 3744: each collision re-mint is another Add, so a new visitor whose first ID collides makes two creates.
      • 1648 ("Writes are at most one, and never retried") and 1933/4095 ("always seen") need to account for the two retrying exceptions and for eventually consistent reads.
      • 2061-2062 ("Stagers never delete one") contradicts 2064, under which any request whose KV read confirms a tombstone deletes them.
    • Behaviour gaps:
      • 2208-2211: a writer's own pre-send overflow write puts its phase 4 write inside the 1 write/sec/item limit, so the flush then fails.
      • 1483-1487 and 1593-1595: a stateful hold answer is dropped on a non-2xx origin status. A redirecting navigation can then consume a rotating refresh token at the vendor and lose the new one. Commit stateful results whatever the origin status.
      • 2068-2069: ts-eids-s is revalidated at least hourly but lives a year; staged_max_age_sec would do. Document the legal basis for setting one-year ts-eids-r cookies on the unusable-consent path.
  • 🔧 Smaller fixes in the plan.

    • Wrong names and signatures:
      • ConsentForwardingMode::CookiesAndBody does not exist (382). The variants are OpenrtbOnly, CookiesOnly and Both (consent_config.rs:132-140).
      • finalize_with_ec_state does not exist (6557). The function is apply_edgezero_ec_finalize (adapter-fastly/src/main.rs:595).
      • encode_s_cookie has 4 arguments in its interface (5526) but 5 in its test (5578).
      • IdentitySettings::provider changes signature between 5829 and 6169.
    • Missing checks and tasks:
      • The unknown-provider-ID check is deferred to Task 22 (5831) but lands only in Task 33, so Part 1 lacks it.
      • Spec items with no task:
        • the Lockr logging change;
        • per-provider response-time logging (spec 1380);
        • PageUrl from the Referer on /auction, and from origin plus path on /_ts/page-bids (spec 858-863);
        • ProxyDispatchInput gaining the writer decision and IdentityDeps (spec 1965-1966);
        • the /identify jurisdiction and no-consent tests (spec 3868-3872).
    • Debug, lint and input handling:
      • RequestIdentity needs a redacting Debug, because EcContext derives Debug (ec/mod.rs:249) and the mutation holds UIDs.
      • Resolution's derived Debug prints the state JSON (3912), which spec 9.3 forbids logging.
      • The harness compiles under feature = "test-utils" without cfg(test), where clippy::panic is denied (Cargo.toml:139). Add a scoped allow.
      • ResolveInput exposes the inputs of every kind (2957-2960), while spec 728-730 makes an input declared only for another kind InputUnavailable.
      • ec_id() reports a missing EC ID as InputUnavailable { ClientIp } and an undeclared read as PrivateState (3139-3147).
    • Process:
      • Tasks 25-29 and 32-38 merge "run, fail, implement, pass" into one step with no code, exactly where the risk is highest. Task 20 says "Write the rest in the same shape" (5954).
      • --signoff -S (51) is not required by CONTRIBUTING.md, and -S needs a GPG key the implementer may not have.
      • The Tech Stack says Rust 1.95 (9); the base pins 1.98.1.

CI Status

  • CLAUDE.md symlink guard: PASS
  • browser integration tests: PASS
  • cargo check (cloudflare native + wasm32-unknown-unknown): PASS
  • cargo check/build/test (spin native + wasm32-wasip1): PASS
  • cargo fmt: PASS
  • cargo test (axum native): PASS
  • cargo test (cross-adapter parity): PASS
  • cargo test (ts CLI, native) (macos-latest): PASS
  • cargo test (ts CLI, native) (ubuntu-latest): PASS
  • cargo test: PASS
  • docs-proposal scripts: PASS
  • format-docs: PASS
  • format-typescript: PASS
  • integration tests (Fastly EC lifecycle): PASS
  • integration tests: PASS
  • prepare integration artifacts: PASS
  • vitest: PASS

All 17 reported checks passed. No check is marked required on this branch.

client EID parser and the `ts-eids` cookie carrying the same set the shim
sends.

This spec does not depend on PR #1260. Its `IdentityCapture` trait,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ How does this relate to open draft #1260?

#1260 (draft, open) carries its own server-side identity foundation spec (2026-10-09-server-side-identity-foundation-design.md), a Lockr capture spec, a plan and an implementation. It defines an IdentityCapture trait, an identity_owner field and a ts-eids-s cookie with different semantics, and it retires ts-eids and x-ts-eids, which this spec keeps (2.2, 7.1). This line treats the two as independent, but they target the same subsystem, the same files (ec/kv_types.rs, ec/finalize.rs, integrations/lockr.rs, publisher.rs) and the same cookie name.

Which design is the plan of record? Please state in the spec whether this one supersedes #1260 (and coordinate closing or rebasing it with its authors), or how the two reconcile, and pick a cookie name that cannot be confused with #1260's.


## 14. Rollout

Three steps, each shippable on its own:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Split this into phase specs, one PR each

CONTRIBUTING.md:67-70: "Don't force a large feature into one huge spec. If the work splits into natural phases, write one spec per phase, each with its own review rounds and pull request". This section already names three phases that each ship on their own, and the spec plus the plan are 11,711 lines in one PR, which makes a complete review impractical.

Suggested split, each with its own review log:

  1. Outbound cookie strips (step 0). They are independent of the identity work and can target main rather than this stack.
  2. The one-write storage protocol: writer and stager rules, IdentityMutation, ts-eids-s, ts-eids-r, the KV schema and the servability check, with today's pull sync feeding the mutation unchanged.
  3. The provider API: trait, descriptor, coordinator, harness, GenericPullProvider, ownership, hold mode and capture.

That is also the separation the previous review asked for (storage redesign separate from vendor extensibility). As written, step 1 freezes the public provider API before any vendor uses it. Please land the implementation plan only after the matching phase spec is approved: it already encodes decisions that are still pending (see the comment on the decisions log).

| Deletion | A tombstone with the 24-hour `TOMBSTONE_TTL` replaces the hard delete. No extra writes. |
| GPP opt-outs | Sharing and targeted-advertising opt-outs are decoded, and the strict policy blocks them. |
| `x-ts-eids` | Kept as a full troubleshooting header. Its exposure of server-only IDs to page scripts is accepted. |
| Consent withdrawal | Synchronous tombstone on any request, one of two G4 exceptions with the removal overflow write. Proposed in review; not yet confirmed. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 The review log ends on a blocking round, and decisions are still pending

CONTRIBUTING.md:95 asks to "keep going until a round finds nothing blocking". Round 10 (4251-4253) found 6 blocking issues and a question, and revision 11 has not had a clean round since. This review is round 11 and finds more.

The decisions log also records choices nobody has confirmed:

  • 3984: the consent-withdrawal exception is "Proposed in review; not yet confirmed".
  • 3988-3989: the round-7 table (per-batch ts-eids-r, opt-out semantics, provider cap, KV read cost) is "applied in revision 8 pending the user's review of the spec".
  • 4002-4003: the round-10 table (random rv, atomic stateful results, refresh anchors, disclosure policy, browser-candidate filtering, /identify consent) is "pending the user's review".

The plan treats all of these as fixed values (plan lines 35-43). Please resolve each one, or move it to section 15 as an open question, before the spec merges. "With the user" (3961) should also name the decision-maker, since a reader of the committed spec cannot tell who that is.

live `ps.oc`, so the next step sees the provider as not opted out and a
later replay of the old opt-out is stale. A clear that is not later is
dropped.
3. Apply the positive changes by their own rules. Every change written by

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 The apply order lets a mutation undo its own removals

Step 3 checks FillMissing against the live ps.iv and skips only changes written by an opted-out provider, while the mutation's own iv hashes and o land in step 4 (1843). I reproduced three interleavings in an executable model of the 5.4 and 6.2 rules:

  1. KV holds S = A from owner P. An /auction invalidates A, staging {pv: rv1} plus iv[h(A)] in ts-eids-r. The next writer's ts-eids still carries A. Step 2 removes A, step 3 applies the browser's FillMissing(A) because the live iv is still empty, and step 4 then adds h(A). A is now stored browser-seeded, never due (1290-1291), and served whenever the browser omits it (2421). The same happens when an after-send call invalidates A after phase 3 already queued the ingest.
  2. A provider opt-out lands in step 2. Browser ingest, a KV-gap entry or batch sync then refills that owner's source in step 3, because none of them is "written by" the opted-out provider. The live browser candidate is filtered (2444-2445), but the stored copy is served from KV.
  3. A staged OwnerWrite is never checked against iv, so a stale staged entry for an invalidated UID applies once the removal has emptied the source.

Related gap: owner denials are read "from KV and from ts-eids-r" (2436-2438). A navigation auction that reads neither (a cookie-only view, 1655 and 2352-2355) forwards the denied A, and the size cap can drop the mirrored o and iv (2256).

Proposed: apply iv and opt-outs in step 2. For an owned source, skip every positive change (FillMissing of any origin, and OwnerWrite) while the owner has a sticky o or the UID is in the post-step-2 iv. Make the servability check apply the owner's o and iv to browser-seeded and legacy records too, and name the ts-eids-s mirror as a denial source the size cap never drops. Add the three interleavings as tests.

removed too and resolved again later. Opt-outs as `o: {<provider>: <at>}`, and invalidated UIDs as
`iv: {<provider>: [<hash>]}`.

An entry is **applied**, and is skipped if the mutation meets it again,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 "Applied" is inferred from absence, so removal cookies are deleted before they take effect

A removal counts as applied when the record is "gone" or at another revision, and the writer then deletes its cookie (2190-2191). Two ways that loses a removal, both reproduced in a model of these rules:

  1. Removal of a staged entry. KV has nothing for owned source S. /auction X1 resolves A1 and stages {u: A1, k: 0} with no pv. /auction X2 sees A1 due and gets invalidate. A staged entry has no rv (2160), so the batch is {s, w, sa} plus iv[h(A1)], and nothing drops A1 from ts-eids-s. Writer N reads KV, finds S absent, treats the batch as applied and deletes it, then flushes A1 as OwnerWrite(None). A1 is written and served, and no removal remains.
  2. Eventually consistent reads. N1's phase 4 writes A at rv1. /auction X stages an invalidation with pv: rv1. N2's phase 1 point read lands on a replica one write behind, sees no record, deletes the cookie, and N2's CAS then fails. After convergence A is served. The codebase says such a read "is not proof of absence: edge KV point reads are eventually consistent" (ec/kv.rs:744-751). The overflow write's "already applied" shortcut (2215-2217) has the same problem.

The opt-out test is safe, because ps.o.at and ps.oc only grow. The removal tests are not.

Proposed:

  • Delete a removal cookie only on positive evidence. For example, the mutation that applies a batch records its <id> in the row (ab: {<id>: at}, pruned after 2 × kv_revalidate_sec), and a cookie is deleted only when a read shows that id. Never infer "applied" from absence or from a different revision.
  • At flush, drop a staged positive entry when an unapplied removal names its source and writer with sa at or after the entry's sa, or when its UID hash is in the staged or live iv.
  • A stager that stages a removal also drops the matching k: 0 entry when it rewrites ts-eids-s.
  • Define "the revision the request read" (2524) for staged entries and stubs.

Comment on lines +503 to +504
has no `Display`. `ProviderRequest`, `StoredId` and `HashedEmail` derive
`Debug, Clone`; `HashedEmail`'s `Debug` prints a redacted marker.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 A derived Debug cannot redact

These types are listed as deriving Debug, but section 9.3 (2652-2656) requires redacting Debug implementations for ProviderRequest, StoredId and HashedEmail, and a derived Debug on HashedEmail prints all 32 bytes. The plan already writes them by hand.

Suggested change
has no `Display`. `ProviderRequest`, `StoredId` and `HashedEmail` derive
`Debug, Clone`; `HashedEmail`'s `Debug` prints a redacted marker.
has no `Display`. `ProviderRequest`, `StoredId` and `HashedEmail` derive
`Clone` and implement a redacting `Debug` by hand (section 9.3).

`KvPartnerId` (`ec/kv_types.rs:102`) gains optional fields. `KvEntry` gains
`ps`. All new fields use `#[serde(default, skip_serializing_if = ...)]`, so
`SCHEMA_VERSION` stays 1 and old rows read unchanged. KV item metadata, which
Fastly caps at 2,048 bytes, holds no IDs (`ec/kv_types.rs:199-219`) and is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Fastly's KV metadata cap is 2,000 bytes

The Fastly limits page and the SDK both say 2,000: pub const METADATA_MAX_BYTES: usize = 2000; (fastly-0.12.1/src/kv_store/handle.rs:19). The repo comment at ec/kv_types.rs:199 and its test at :642-656 repeat 2,048. No impact today, since the metadata is about 100 bytes.

Suggested change
Fastly caps at 2,048 bytes, holds no IDs (`ec/kv_types.rs:199-219`) and is
Fastly caps at 2,000 bytes, holds no IDs (`ec/kv_types.rs:199-219`) and is

Comment on lines +2592 to +2593
- TCF purposes and vendor consents (`consent/tcf.rs:145`). A small
`has_vendor_consent` helper is new.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 has_vendor_consent already exists

TcfConsent::has_vendor_consent(&self, vendor_id: u16) is at consent/types.rs:251-253. The plan's Task 12 would add a second one in consent/tcf.rs (plan 4264-4269); it should call the existing method instead.

Suggested change
- TCF purposes and vendor consents (`consent/tcf.rs:145`). A small
`has_vendor_consent` helper is new.
- TCF purposes and vendor consents (`consent/tcf.rs:145`). Vendor consent
uses the existing `TcfConsent::has_vendor_consent` (`consent/types.rs:251`).

Comment on lines +2657 to +2658
- The Lockr proxy logs only the path of the upstream URL, not its query
(`integrations/lockr.rs:202-204`).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 The Lockr proxy logs the query today

integrations/lockr.rs:204 logs target_url, query included, at info level, and the error context at :228-231 carries it too. As written, this reads as current behaviour, so the change has no task in the plan.

Suggested change
- The Lockr proxy logs only the path of the upstream URL, not its query
(`integrations/lockr.rs:202-204`).
- The Lockr proxy changes to log only the path of the upstream URL. Today it
logs the full URL, query included, at info level and in its error context
(`integrations/lockr.rs:204`, `:228-231`).

Comment on lines +3487 to +3490
The harness API, in the test-support module, gated by
`#[cfg(any(test, feature = "test-support"))]`. A dependent crate never sees
`cfg(test)`, so a provider in its own crate enables `test-support` as a
dev-dependency feature of `trusted-server-core`:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Core's test feature is test-utils

There is no test-support feature. Core already has test-utils (crates/trusted-server-core/Cargo.toml:69), every adapter enables it, and the plan reuses it as its one declared deviation (plan line 45). Line 3737 ("shared through a test-support feature") needs the same change.

Suggested change
The harness API, in the test-support module, gated by
`#[cfg(any(test, feature = "test-support"))]`. A dependent crate never sees
`cfg(test)`, so a provider in its own crate enables `test-support` as a
dev-dependency feature of `trusted-server-core`:
The harness API, in the test-support module, gated by
`#[cfg(any(test, feature = "test-utils"))]`, the feature core already has
for this. A dependent crate never sees `cfg(test)`, so a provider in its own
crate enables `test-utils` as a dev-dependency feature of
`trusted-server-core`:

Fold in findings from #1246 and the #1260 review: the ORTB2 EID dependency, the first-auction race metric, the empty GPP string, capture off by default, a capture rollout checklist and the cookie lifetime decision.

Propose, pending review, a real-browser gate on capture effects, an operator-gated global revoke and dedicated rotating cookie keys.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Revision 15 drops the sealed identity cookies, the keyring and the removal overflow write. Removals reach KV on the request that carries them, KV moves to schema version 2, ts-eids-s becomes a client-only mirror written by the shim, and capture is revoke-only.

The implementation plan is rewritten to match: tasks for the cookie keyring and the race metric are retired, Task 27 now holds the withdrawal tombstone retry, and the remaining tasks follow the new KV-only eligibility, mirror payload and capture removal write.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Revision 15 is a real simplification. Dropping the sealed cookies, the ts-eids-r removal cookies, the keyring and the overflow write makes about half of the round-11 blockers moot, and most of the round-11 plan findings are fixed. Requesting changes because two new defects, two plan defects and several unresolved round-11 spec findings remain. None of the round-11 inline threads has a reply at this head.

Blocking

🔧 wrench

  • The review log is linked but not committed. See inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:8.
  • Removal retries give up at once on Fastly's per-key write limit. See inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:1805.
  • Capture removals from clients that fail the bot check are dropped, against the codebase's revocation policy. See inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:1624.
  • Axum, Cloudflare and Spin stop compiling from Task 23 on. See inline at docs/superpowers/plans/2026-10-22-server-side-identity-providers.md:8420.
  • Hold mode would nest futures::executor::block_on. See inline at docs/superpowers/plans/2026-10-22-server-side-identity-providers.md:8206.

❓ question

  • Rollout step 1 cannot be rolled back. See inline at docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md:4241.

Non-blocking

  • 🤔 cluster_size is never stored when it is the only change, and rotation doesn't clearly skip phase 4. See inline at docs/superpowers/plans/2026-10-22-server-side-identity-providers.md:9003.

Cross-cutting / body-level findings

  • 🔧 Round-11 findings still present at e49916f. Each is unchanged in the text, and none of their threads has a reply:

    Finding Status
    /identify described as a cross-site bearer endpoint that cannot rely on TS cookies (spec 2079-2081) Still present, and now contradicts 7.1 (2513-2518). 7.1's "a partner's server cannot ask on a visitor's behalf" is also still contradicted by ec/identify.rs:43-84: a call with no Origin header is accepted, and the EC is just the ts-ec value in the Cookie header
    Stripping ts-eids from origin requests (2341-2350) Still present. The suggested replacement, x-ts-eids, is in INTERNAL_HEADERS (constants.rs:67-71) and never reaches the origin
    "A small has_vendor_consent helper is new" (2665) Still present. It exists at consent/types.rs:251 (the plan already reuses it)
    The Lockr proxy described as logging only the path (2736-2737) Still present. integrations/lockr.rs:204 logs the full URL, query included, at info level
    KV metadata cap given as 2,048 bytes (2104) Still present. Fastly's cap is 2,000
    Clients without Sec-Fetch-Dest never become writers (1658, 1690) Still present. G10, pull sync and orphan recovery never run for them
    Owner opt-out vs a browser-seeded record Partly fixed. A browser-seeded record of an opted-out resolving owner's source is still stored (step 3 skips this only for capture-only providers) and served from KV when the browser omits the source. The servability check keys on the writer's ps.o, not the owner's
    Deletion re-seeded through the browser Still present. 5.5 says "a deletion is never undone through a cookie" (2056-2058), but the rotated row's create carries ts-eids and sharedId ingest (1735-1736)
    strict() needs vendor consent only with with_tcf_vendor; generic pull is exempt from it Still present. 7.1's "the sharing and targeted-advertising opt-outs and GPC stop it" does not hold for generic pull
    Generic pull provider IDs can collide Still present. Replacing dots with - maps a-b.example.com and a.b-example.com to the same ID (10.4), and 10.2 rejects duplicates
    10.4 "rotation keeps dispatch fair under max_concurrency" Still inaccurate. Every eligible partner is dispatched, and max_concurrency only sizes the batches (pull_sync.rs:279-284). The example's max_concurrency = 4 also differs from today's default of 3 (settings.rs:587)
    Pull sync first-byte timeout drops from 15 s to 1 s Not listed in 10.4 "What changes"
    LiveRamp example: initial 204 mapped to no_match, consent sent as gdpr_consent Still present, and 13 makes these the first harness fixtures
    One spec per phase (CONTRIBUTING.md:67-70) Not addressed. Spec and plan are 14,265 lines, Part 1 is 31 tasks in one PR, and there is no runtime switch
    Branch behind its base, stale PR description Still present. The branch is 11 commits behind spec/1246-prebid-eid-fidelity (Rust 1.95.0 here, 1.98.1 at the base). The description still says nine review rounds and omits the plan

    Closed since round 11: the sealed and removal cookie findings (host-only scope, passphrase rotation, the byte budget, "applied" inferred from absence, overflow durability) are moot. Plan findings 1, 2, 4, 5, 7, 9, 12 and 14 and the wrong names are fixed.

  • 🤔 A positive Lockr capture costs an uncounted KV read. Deciding a clear needs the live ps.o, so every consenting positive capture goes through the removal write's read. 6.3 (2387-2389) says a proxied Lockr call with no revocation reads no KV, and the cost table omits it.

  • 📝 Smaller plan items:

    • pub enum FlushOutcome wraps pub(crate) enum RemovalWriteOutcome (plan 8993, 7619), which trips private_interfaces under -D warnings.
    • test_support items compiled under test-utils by clippy-fastly --all-features need # Panics docs and #[must_use], as the existing gated code has (test_support.rs:126-131).
    • The only ResponseExt literal is in auction/formats.rs:505, which Task 26 does not list.
    • The per-provider response-time log (spec 1567) has no task.
    • Task 29, the core flush, and several later tasks still collapse "write the test, fail, implement, pass" into one step with no code.

CI Status

All 17 reported checks pass at e49916f, including:

  • format-docs
  • cargo fmt
  • cargo test
  • the axum, cloudflare, spin and parity jobs
  • vitest
  • integration tests
  • browser integration tests

No check is marked required. This PR is docs-only, so passing CI establishes formatting only.

removes the `ts-eids-r` removal cookies, runs providers only on writer
navigations, makes capture revoke-only and moves KV to schema version 2,
accepting breaking changes. Section 16 lists the decisions; the
[review log](2026-10-20-server-side-identity-providers-review-log.md) maps

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 The review log is linked but not committed

This line and line 4339 link 2026-10-20-server-side-identity-providers-review-log.md, but the PR adds only the spec and the plan, so the link is broken. CONTRIBUTING.md:95-107 asks the spec to end with a ## Review log section, one entry per round, so reviewers can see which questions have already been answered. Revision 15 moved rounds 1 to 14 out of the spec, so that record is now missing from the PR.

The decisions log also still has an unconfirmed entry ("Consent withdrawal … Proposed in review; not yet confirmed"). Its "with the user" and "made by the user" still don't name the decision-maker.

Please commit the log, or restore the section, and resolve or name each pending decision.

Comment on lines +1805 to +1813
5. If the write fails, on a precondition failure, Fastly's per-key rate
limit or any other error, log. The flush is not retried as a whole.
Fastly limits writes to one key to about one per second, and the limit
can be stricter, so an immediate retry after a conflict would usually
fail too. When the mutation holds removals, opt-outs or clears, they are
then written alone, with the CAS loop of the capture removal write
(section 5.3): a re-read, the removal rules of section 5.4 and up to five
attempts. When those fail too, they are lost and counted in
`removal_lost`. Nothing else backs the flush, so:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Removal retries give up at once on Fastly's per-key write limit

The Fastly adapter maps every insert error other than ItemPreconditionFailed to Err, including KVStoreError::TooManyRequests (adapter-fastly/src/ec_kv.rs:149-159). The plan's write_removals returns Failed on any Err at once (plan 7448: "an immediate retry after a per-key limit would fail too").

Both retries run right after the failure they recover from:

  • Phase 4. The removal-only retry starts immediately after a failed flush. The usual cause of that failure is a write by another tab within about the last second, especially after an after-send dispatch, whose re-read is at most after_send_timeout_ms before the write. So the retry's write is likely rate-limited too, and the removal is counted as lost.
  • Capture. The "once more after send" attempt follows a small proxied JSON response by milliseconds, so it is usually inside the same window. The withdrawal path's reasoning (1716, "by which time Fastly's per-key limit has usually passed") fits a large navigation body, not this.

Example: tab A's after-send UID2 call returns optout while tab B writes the row. A's flush fails on the conflict, the retry is rate-limited, and the opt-out is lost. If the source is filled meanwhile, A's provider may never be eligible to report it again. This is the case raised in the round-11 comment at 1635, whose proposal ("after the per-key window") was not adopted.

Proposed: after send, wait at least the per-key window (for example 1 s since the failed attempt) before the removal-only attempt, and treat TooManyRequests as retryable after that wait. Latency is not at stake after send. The pre-send capture attempt can stay immediate. Add a test with a store that rate-limits the first write after a conflict.

Source inference: the plan has not been run.

Comment on lines +1624 to +1631
- **The real-browser gate.** A client that fails `looks_like_browser()`
makes no capture removal write and no clear. Its removals, opt-outs and
global revokes are dropped and logged with the provider ID and source
only. Earlier revisions staged them in that client's own cookie, where
they reached KV only if the same client later made a real-browser
navigation, which in practice never happens. So a script replaying a
copied `ts-ec` cookie through the proxy can neither strip nor tombstone a
row. Consent withdrawal keeps its own exception (section 5.1).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Dropping capture removals from clients that fail the bot check goes against the codebase's revocation policy

build_ec_request_state keeps the KV write path for withdrawals from clients that fail looks_like_browser(): "Revocations keep the write path so tombstones stay authoritative even for privacy-extension-heavy clients" (adapter-fastly/src/app.rs:445-452). looks_like_browser() fails whenever the JA4 class or the UA platform is missing (ec/device.rs:84-86), for example with a UA-spoofing extension.

The rationale here, stopping a replayed ts-ec from stripping a row, does not hold. The same replay with a denying consent cookie can already tombstone the row through the withdrawal path (5.1).

Consequence: a Lockr revoke-consent from a visitor whose extension fails the check is dropped. Their previously ingested Lockr IDs stay in KV and are served again as soon as a later request passes the check. That contradicts G12 ("Every removal … is written to KV on the request that carries it").

Proposed: apply provider-scoped removals and opt-outs whatever the classification (they only remove), and keep the gate on clears, which re-enable IDs. Decide "global" revokes explicitly, consistent with the withdrawal exception.

and `ps.st` do not: invalidated values can be re-seeded once, and
stateful providers start over when rolled forward.

**Rolling back past step 1 is unsupported.** Step 1 writes schema version 2,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Was a non-rollbackable step 1 accepted knowingly, and why not ship the reader first?

Step 1 is the largest step: strips, the writer rule, the single flush, schema v2, the GenericPullProvider port and the mirror. It is also the one that cannot be rolled back.

An older build rejects a v2 row in KvEntry::validate (ec/kv_types.rs:376-381), and load_snapshot maps that to Failed. Old instances therefore serve no stored IDs and skip ingest and pull sync for every row step 1 has rewritten. A withdrawal tombstone on such a row also fails closed (ec/kv.rs:1384-1392), so a withdrawal made during the rollback is not recorded. The same applies while a deploy is still rolling out and old and new instances run side by side, which the spec doesn't cover.

The usual mitigation costs one release: ship a reader that accepts v1 and v2 but still writes v1, then switch writes to v2. Each step can then be rolled back to the previous one. Please confirm who accepted the current trade-off, or add the reader-first step to section 14.

}
```

`effective_tcf` (`consent/mod.rs:283-288`) prefers `tcf` and falls back to `gpp.eu_tcf`, as the auction's consent checks do. Change only its visibility to `pub(crate)`. The fail-closed check needs no special case for `DBAA`: it decodes, so `consent.gpp` is `Some` and nothing blocks. The field names above (`raw_tc_string`, `tcf`, `raw_gpp_string`, `gpp`, `raw_us_privacy`, `us_privacy`, `gpc`, `gdpr_applies`, `jurisdiction`) are `ConsentContext`'s (`consent/types.rs:109-148`). `has_purpose_consent` takes a `usize`, so the `u8` purposes convert with `usize::from`. `us_sale_opt_out` reads the decoded `GppConsent

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Axum, Cloudflare and Spin stop compiling from this task on

ProxyDispatchInput is built with struct literals in every adapter:

  • trusted-server-adapter-axum/src/app.rs:228
  • trusted-server-adapter-cloudflare/src/app.rs:540
  • trusted-server-adapter-spin/src/app.rs:821
  • the Fastly adapter (app.rs:815)

This task adds is_real_browser and Task 25 adds seed, but both tasks list and test only the Fastly adapter (cargo test-fastly). Every commit from here to the full CI run in Task 31 fails check-axum, check-cloudflare and check-spin. Spec 11 also promises those adapters "change nothing" except the cookie strips.

Proposed: add the three adapter files to this task's file list with the field set to their current behaviour, and run cargo check-axum && cargo check-cloudflare && cargo check-spin in its verification step. Or give the struct a constructor with defaults so new fields don't touch every adapter.

}
```

`effective_tcf` (`consent/mod.rs:283-288`) prefers `tcf` and falls back to `gpp.eu_tcf`, as the auction's consent checks do. Change only its visibility to `pub(crate)`. The fail-closed check needs no special case for `DBAA`: it decodes, so `consent.gpp` is `Some` and nothing blocks. The field names above (`raw_tc_string`, `tcf`, `raw_gpp_string`, `gpp`, `raw_us_privacy`, `us_privacy`, `gpc`, `gdpr_applies`, `jurisdiction`) are `ConsentContext`'s (`consent/types.rs:109-148`). `has_purpose_consent` takes a `usize`, so the `u8` purposes convert with `usize::from`. `us_sale_opt_out` reads the decoded `GppConsent

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Hold mode would nest futures::executor::block_on

dispatch (8142) is synchronous, and this step says to follow dispatch_pull_sync, which waits with futures::executor::block_on (ec/pull_sync.rs:259, :387). That is fine after send. But run_hold (Task 37) calls dispatch from inside the async handle_publisher_request, which the Fastly adapter already drives with futures::executor::block_on(app.router().oneshot(..)) (adapter-fastly/src/main.rs:467). A nested block_on on the futures executor panics, so every hold-mode writer navigation that runs an auction would fail. This was raised in round 11 at plan 6434 and is still present.

Proposed: make dispatch async (async fn dispatch(..)), and await the in-flight calls with select/wait as the auction orchestrator does. Wrap it in block_on only at the phase 4 call site, which runs outside the router.

}
```

`effective_tcf` (`consent/mod.rs:283-288`) prefers `tcf` and falls back to `gpp.eu_tcf`, as the auction's consent checks do. Change only its visibility to `pub(crate)`. The fail-closed check needs no special case for `DBAA`: it decodes, so `consent.gpp` is `Some` and nothing blocks. The field names above (`raw_tc_string`, `tcf`, `raw_gpp_string`, `gpp`, `raw_us_privacy`, `us_privacy`, `gpc`, `gdpr_applies`, `jurisdiction`) are `ConsentContext`'s (`consent/types.rs:109-148`). `has_purpose_consent` takes a `usize`, so the `u8` purposes convert with `usize::from`. `us_sale_opt_out` reads the decoded `GppConsent

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 cluster_size is never stored when it is the only change, and rotation doesn't clearly skip phase 4

  1. cluster_size. This step returns Skipped when the mutation is empty and no provider is due, and step 5 is the only place cluster_size is added. So a returning visitor with nothing else to write never stores it. Spec 2081-2085 says a writer whose read finds it missing adds it to the mutation. Proposed: when the phase 1 view lacks cluster_size, count that as a pending change in step 2. Add a test where it is the only change.
  2. Rotation. Line 8632 says rotation empties the mutation and phase 4 skips, but step 1 checks only "returning-EC writer". Nothing marks a rotated request, so a due after-send provider could still dispatch and make a second write on the new row. Proposed: a rotated flag that makes step 1 return Skipped, with a test that a rotation plus a due after-send provider makes exactly one write.

A found result now resets the failure count, and with_retry_in raises n on every resolution, not only no_match. Burst-capped calls get their own outcome with a 60-second wait, while a limiter error leaves ps unchanged. RemovalWriteOutcome becomes pub because FlushOutcome carries it, and the capture consent rules cite spec 4.7 instead of 5.5.

The spec gains the 60-second burst-cap wait as a section 8 row and in section 4.7 step 8. The generic pull provider ID now carries a hash suffix when the domain already contains a hyphen, matching the plan, and the operator docs explain the rule.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Every Cookie field is now read and checked, and a field that is not valid UTF-8 is dropped instead of forwarded. Tests that asserted a malformed Cookie field is forwarded unchanged now expect it to be dropped.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The origin keeps ts-ec and every unrelated cookie. ts-eids and __Host-ts-eids-s no longer reach it. An origin that read ts-eids must read x-ts-eids instead.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Every record stores its source's entry set in eids, with optional writer, revision and timing fields, and the row gains per-provider state. Version 1 rows read as one-entry records and are written as version 2. A build without this change cannot read a version 2 row, so rolling back past it is unsupported.

A record that breaks a limit is dropped on read and logged with its source only, so one bad record never makes a row unreadable; only the write path rejects it. Blank UIDs are now rejected on validation, so a version 1 record with a blank uid is dropped on read.

KvPartnerId and KvEntry derive Default, and literals move to constructors. The admin lookup redacts provider state.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The estimate cites spec 6.1 and uses its numbers.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Log one identity request measurement line per request after phase 4, with the Cookie header size, KV reads and writes by route kind, the flush outcome, lost removals and the after-send withdrawal run. It holds no identifier.

Document the writer-request rule, the mirror payload, the [ec.identity] table, the removed config keys, the rollback limit and the removed pull-complete cookie. Trim the EID sync outcome enum to the one value still logged.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Record each create attempt of the new-EC path on the request and add it to kv_writes, and skip the inferred phase 1 read when the snapshot came from that create. Keep the flush outcome as a value and format its label on display.

Word the single-write docs as per EC-bearing request, and note in the changelog that the example config moves max_concurrency from 3 to 4 and that the pull rate is a ceiling.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Integrations can add identity providers to their registration, and the identity runtime and catalog now take the registered providers. Capture rules must lie under the registering integration's own routes.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Log the registry failure and build the identity runtime without registered providers, so generic pull still runs. Name the rejected path or prefix in capture-rule errors and cover duplicate provider IDs and prefixes under exact routes.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add EcPartner.owner and the ownership, sources-narrowing and capture-only rules to validate_identity_catalog. Add validate_identity_integration for deploy-time descriptor checks without secrets, and record source owners in the provider catalog.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Adds the US sharing and targeted-advertising opt-outs, GPP section 6, and the US National GPC and known-child fields. A header-only GPP string with no applicable section still does not block a strict policy. ConsentContext's Debug output no longer prints raw consent strings.

The consent fingerprint moves to policy.rs and appends the sharing and targeted-advertising bits after the existing ones.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A US section or section 6 that fails to decode now sets a flag on the decoded GPP form, and every policy except generic pull's parity policy blocks on it. The spec names the parity exemption, the bit 11 doc mentions GPP section 6, and the coordinator test import is regrouped.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add record and count to the rate limiter trait, backed by Fastly counter increment and 60-second lookup. The identity dispatch skips a provider whose identity-fail count has reached breaker_failures_per_minute, and records only transport failures, never hold cut-offs or input failures.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add the owned row of the precedence table, the re-grant clear on the writer's flush, and an admin DELETE route that clears a provider's opt-out on every adapter.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Answers feed that auction and join the writer's single flush. Stagers run no provider, and a navigation without an auction runs hold-mode providers after send instead.

The navigation records that it ran an auction, so phase 4 skips hold-mode providers it already dispatched. An error status drops a found outcome's scheduling with its positive answer.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…iter

The fake provider client now fails a wait that outlasts the first-byte timeout its backend was given, so the slow-provider test pins a real cut-off.

Phase 4 treats hold-mode providers as already handled only when the hold dispatch ran on a usable phase 1 row. Without a rate limiter on the request the hold dispatch sends nothing.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Capture is revoke-only: a real browser's captured removals, opt-outs and clears are written to KV on the proxied request with a CAS loop, retried once after send, and counted as lost if both fail. Positive IDs are counted, never stored. A non-browser client's capture writes nothing. revoke_scope = "global" runs the consent-withdrawal tombstone from a real browser.

The Lockr proxy now logs only the upstream path, not its query.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…operations

A capture whose row cannot be read now logs and counts its removals lost instead of dropping them with a warning. The capture's view read and removal-write reads and writes join the measurement line's KV counts, and a requested global revoke skips the other providers' removal writes.

Adds a Fastly adapter test for a global-scope capture that tombstones the row and expires the EC cookie.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
An unreadable view now loses only the source invalidations that need the record revision. The opt-out is built without a view and goes through the removal write, with the after-send retry, and counts as lost only if that write and its retry fail.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A batch update to a source an identity provider owns now applies only when the row holds no value for it, and is otherwise unchanged without a write. Unowned sources keep today's replace behaviour.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add HarnessSetup, TestConsent, CaptureFixture, ConformanceCases and ConformanceFailure, with try_check_conformance and check_conformance in plain and stateful forms, output_eids, capture and check_descriptor_ignores_secrets. Add the gated StoredId and HashedEmail test constructors, and cover the harness with the spec examples and one negative case per conformance check.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add MultiEntryTestProvider to the core test support, and request-level tests that register it through an integration registration and follow its two-entry result to KV, the mirror payload, the bidstream and identify, with refresh, oversized, rollback, capture and 404 cases. Run the Part 1 golden tests again with the provider registered.

Fix the Fastly adapter test driver from Task 29: it drove the router with futures::executor::block_on, which panicked when the hold dispatch of an auction navigation blocked on its own futures. It now polls the future directly. No production code changed.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add the provider author guide, the owner and per-provider configuration keys with their validation errors, the capture provider checklist, revoke scope, complete provider EIDs and rollback steps, the commented capture keys in the example config, and the changelog entries.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Hold mode dispatched providers with futures::executor::block_on from inside the publisher handler, which already runs on an executor, and nested block_on panics there. The dispatch path is now async: the hold run awaits it, and phase 4, which runs after send outside any executor, blocks on it once.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The shim used to replace ts-eids with each auction's EIDs, trimmed with registered sources first. An auction missing one ID module then dropped that module's ID from the cookie, and unregistered sources still used cookie space the writer never ingests.

The shim now writes only [[ec.partners]] sources, replaces the cookie's entries per source with the request's, keeps sources the request lacks, and leaves the cookie as it is when a request carries no EIDs. A TCF block still clears it. The spec, guides and fixture follow.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Batch sync writes an owned source only when the row holds no value for it, and an expired owned record counts as present. The author guide now names the harness helpers' real signatures and notes that ExampleIdProvider is a crate-private fixture.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…erge

The spec and plan now describe the provider dispatch as async, awaited before the auction and blocked on only after send, and the plan describes the shim's registered-only, per-source ts-eids merge in place of the registered-first trim.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The spec now uses the existing TcfConsent::has_vendor_consent instead of announcing a new helper, gives Fastly's KV metadata cap as 2,000 bytes, and states that #1260 will not merge, so this spec supersedes it and keeps ts-eids and x-ts-eids.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Pin the merge order, repeated cookie sources, source spelling and malformed entries, exact-domain matching, and that a size trim drops kept cookie entries before fresh request ones. At the buildRequests level, a corrupt cookie is replaced and a source that is no longer registered leaves the cookie on the next write.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A request gives providers the page URL without its query string or fragment, but the harness passed the URL through as written, so a provider test could not catch a provider that forwards a query it never receives in production.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The spec linked a review log file that was never added to the repository, so both links were broken.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Write Lockr IDs to KV store

4 participants