Move OpenRTB into an auction protocol module

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Breaking for crates outside core:** core's auction names no protocol. A demand source implements `CompiledDemand`, which builds its own request from the routed input (`build_request`, answering the body, the headers, the slot ids and any state kept for the answer) and reads its own answer, and core sends what it built. OpenRTB 2.6 is the protocol module `crates/auction-protocol/openrtb`, where the request builder, the response reader, the wire types with their extension objects, and `OpenRtbDemand`, the hook trait a demand implementation that speaks OpenRTB fills in, now live, run through core's seam by `OpenRtbDriver`. `auction::openrtb` and core's `openrtb` module are gone, and core's own manifest keeps the generated types only for the response `POST /auction` returns. The dimension admission helpers are `auction::admission`, `apply_notification_policy` and `ignored_bidder_params_count` are `auction::provider`, and `Eid` and `Uid` are `ec::eids`. A deployment's settings do not change.
 - **Breaking:** The ad server this repository ships is `ad-server.iabtl`, from `crates/ad-server/iabtl` as the package `trusted-server-ad-server-iabtl`, where it was `ad-server.mock` from `crates/ad-server/mock`. A deployment that selected it writes `[ad-server] module = "iabtl"` and `[ad-server.iabtl]` where it wrote `module = "mock"` and `[ad-server.mock]`, and an `implementation = "ad-server.mock"` line names `ad-server.iabtl`. The seam probe, the test fixture that proves the integration seam from outside core, is `crates/testing/iabtl` as the package `trusted-server-testing-iabtl`, selected as `testing.iabtl`, where it was `crates/testing/seam-probe` selected as `testing.seam-probe`, with its shared table `[shared.iabtl]`, its loading cache `loading-cache.iabtl` and its declared cache `iabtl-answers` named the same way. The module that adds sellers to `ads.txt` is `ads-txt.iabtl`, from `crates/ads-txt/iabtl` as the package `trusted-server-ads-txt-iabtl`, where it was `ads-txt.sellers` from `crates/ads-txt/sellers`. A deployment that selected it writes `[ads-txt] modules = ["iabtl"]` and `[ads-txt.iabtl]` where it wrote `modules = ["sellers"]` and `[ads-txt.sellers]`, and the `[[fetch]]` entry on `/ads.txt` names the middleware `ads-txt.iabtl` where it named `ads-txt.sellers`. Each module describes the middleware it supplies on its builder with `IntegrationBuilder::with_middleware`, as `SuppliedMiddleware` values naming the middleware, the phase an entry names it in and the media type the entry covers, and `trusted_server_core::integrations::supplied_middleware(builders)` lists what a build supplies in the order the registry registers them, core's robots.txt and service version middleware included. The registry refuses a registration whose middleware contradicts its builder's description, in a phase it does not run in or for a media type it does not work on. **Breaking for crates outside core:** `crates/trusted-server-modules` is gone. Each adapter lists the module builders it links in its own `app.rs`, as `shipped_module_builders()`, with `module_builders_with(extra)` putting a deployment's builders after them, and the `ts` tool lists the builders it links and writes its entries and a module's section from what they describe. A deployment that took its list from `trusted_server_modules::builders()` or `builders_with(...)` names the crates it links instead, or takes an adapter's list.
 - **Breaking for crates outside core:** the consent decoders live in the crates named for their standards. The TCF string decoder is in `trusted-server-permission-signal-tcf`, the GPP string decoder in `trusted-server-permission-signal-gpp`, and the US Privacy string decoder in `trusted-server-permission-signal-us-privacy`, and each module reads its own string from the request through `PermissionSignalModule::read`, once, and reports a `SignalRecord`. `ConsentContext` is the records the configured modules reported and the request's jurisdiction, read by scheme name, with nothing of any standard in it, and a reader that needs a string in its standard's own form asks that standard's crate (`tc_string`, `applies`, `gpp_string`, `section_ids`, `us_privacy_string`, `signaled`). A record also carries the permissions its signal consents to, and the server-side auction gate keeps its rule by reading them, so in a GDPR or unknown jurisdiction, or where a record says its regulation applies, the auction needs consent to device storage (TCF Purpose 1) from a record that applies. Without `tcf` in the modules no TC String is read, and one that cannot be read, standalone or in a GPP string, is no consent to any purpose TCF maps. `[consent]` keeps its keys, with `mode` parsed by core and the rest read by the modules, and `signals.<scheme>.authoritative` in `permissions.yaml` is read for any scheme. The identity graph record writes every vouched signal under its scheme, in the shape it wrote before for the two it stored. `CONSENT_COOKIE_NAMES`, `forward_cookie_header`, the consent cookie and header constants, `ConsentForwardingMode` (now the Prebid Server crate's), the `GdprConsent` error and the never-filled Additional Consent field and `additional_consent` policy flag are removed.
 - **Breaking for crates outside core:** the identifiers a request carries come from identifier modules. Core declares `IdentifierModule` in `trusted_server_core::ec::identifiers`, a registration offers one with `IntegrationRegistrationBuilder::with_identifier_module`, and core asks every running module once per request, when it reads the Edge Cookie state, then syncs the answers to the identity graph and carries them to the auction as it did the `ts-eids` cookie. The Prebid module owns that cookie, its two shapes and its size limit, and is the first identifier module. `COOKIE_EIDS` is removed from core, `ec_finalize_response` no longer takes the cookie's value, `EcContext::module_eids` holds what the modules learned, and `RuntimeServices` carries the modules through `with_identifier_modules`. A `template_cache_key_cookies` entry naming any cookie under the public prefix is refused, where the Edge Cookie and the EIDs cookie were refused by name. A deployment's settings do not change.

crates/ad-server/iabtl/src/lib.rs

@@ -27,12 +27,12 @@ use std::sync::Arc;
 use std::time::Duration;
 use validator::Validate;
 
-use trusted_server_core::auction::context::{ContextQueryParams, build_url_with_context_params};
-use trusted_server_core::auction::demand::AdServerImplementation;
-use trusted_server_core::auction::openrtb::{
+use trusted_server_core::auction::admission::{
     BidDimensionIndex, BidRejectionReason, build_bid_dimension_index_from_slots,
     parse_optional_bid_dimension, resolve_bid_dimensions,
 };
+use trusted_server_core::auction::context::{ContextQueryParams, build_url_with_context_params};
+use trusted_server_core::auction::demand::AdServerImplementation;
 use trusted_server_core::auction::provider::{AuctionProvider, ProviderRequestOutcome};
 use trusted_server_core::auction::types::{
     AuctionContext, AuctionRequest, AuctionResponse, Bid, BidStatus, MediaType,

crates/auction-protocol/openrtb/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "trusted-server-auction-protocol-openrtb"
-description = "The plain OpenRTB 2.6 demand implementation, for an exchange that needs no code of its own."
+description = "OpenRTB 2.6 as an auction protocol: the request builder, the response reader and the hook trait a demand implementation that speaks it fills in, with the plain implementation for an exchange that needs no code of its own."
 authors = { workspace = true }
 edition = { workspace = true }
 license = { workspace = true }
@@ -15,17 +15,28 @@ workspace = true
 
 [dependencies]
 async-trait = { workspace = true }
+edgezero-core = { workspace = true }
 error-stack = { workspace = true }
+http = { workspace = true }
+log = { workspace = true }
 serde = { workspace = true }
 serde_json = { workspace = true }
 trusted-server-core = { workspace = true }
+trusted-server-openrtb = { workspace = true }
+
+[features]
+# Exposes the test helpers, so a demand implementation's own tests read the
+# request the driver builds for it.
+test-utils = ["trusted-server-core/test-utils"]
 
 [dev-dependencies]
+futures = { workspace = true }
 trusted-server-core = { workspace = true, features = ["test-utils"] }
+url = { workspace = true }
 
 # The visible owner of this crate, the way Prebid.js requires a named
-# maintainer of every adapter. The plain OpenRTB implementation is the
-# project's own, so the Trusted Server maintainers own it.
+# maintainer of every adapter. The protocol and its plain implementation are
+# the project's own, so the Trusted Server maintainers own them.
 [package.metadata.maintainers]
 owner = "Trusted Server maintainers"
 status = "project owned"

crates/auction-protocol/openrtb/src/demand.rs

@@ -0,0 +1,264 @@
+//! What a demand source that speaks `OpenRTB` decides, and the driver that
+//! runs it through core's demand seam.
+//!
+//! The driver builds every standard request field the same way for all of
+//! them and reads a response the ordinary way. A source decides only what
+//! [`OpenRtbDemand`] exposes, which is its field policy, its own extensions,
+//! its outbound headers and how its responses are read.
+
+use core::any::Any;
+use std::collections::HashSet;
+
+use async_trait::async_trait;
+use error_stack::{Report, ResultExt as _};
+use http::{HeaderMap, HeaderValue, header};
+use serde_json::{Map, Value};
+use trusted_server_core::auction::demand::{
+    CompiledDemand, DemandResponse, DemandTransport, OutboundRequest, ProviderAuctionInput,
+    ProviderSlotInput, RequestFinalization,
+};
+use trusted_server_core::auction::types::AuctionResponse;
+use trusted_server_core::consent::ConsentContext;
+use trusted_server_core::error::TrustedServerError;
+use trusted_server_core::platform::PlatformResponse;
+
+/// The longest primary `Accept-Language` tag a conservative source receives.
+///
+/// A source that sets no limit receives the whole primary tag.
+pub const CONSERVATIVE_LANGUAGE_MAX_BYTES: usize = 8;
+
+/// How the `regs` object is derived from the consent the auction admitted.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub enum RegsPolicy {
+    /// No `regs` without consent data, and GDPR set from the applicability bit
+    /// or from a GDPR jurisdiction.
+    #[default]
+    Jurisdiction,
+    /// `regs` for any admitted consent, and GDPR set from the applicability bit
+    /// alone.
+    ApplicabilityBit,
+}
+
+/// The standard `OpenRTB` field choices one demand source makes.
+///
+/// The driver constructs every standard field and reads this policy while it
+/// does. A policy can leave data out of what central privacy enforcement
+/// approved, and it cannot add data that enforcement removed.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct DemandFieldPolicy {
+    /// Send the slot id as `imp.tagid`.
+    pub imp_tagid: bool,
+    /// Copy the first banner format into `banner.w` and `banner.h`, and send
+    /// `banner.topframe` as not in the top frame.
+    pub primary_banner_size: bool,
+    /// Send the accepted `Referer` header as `site.ref`.
+    pub site_ref: bool,
+    /// Send precise latitude and longitude with the coarse geo.
+    pub precise_geo: bool,
+    /// The longest primary `Accept-Language` tag sent, or `None` for no limit.
+    pub language_max_bytes: Option<usize>,
+    /// How `regs` is derived from consent.
+    pub regs: RegsPolicy,
+    /// Mark the request as a test request.
+    pub test: bool,
+    /// Keep the Trusted Server host and scheme in `ext.trusted_server` when the
+    /// request is not signed.
+    pub unsigned_request_identity: bool,
+    /// Send `Accept: application/json`.
+    pub accept_json: bool,
+    /// Treat routed bidder parameters as request data rather than reporting
+    /// them as parameters the source ignored.
+    pub consumes_bidder_params: bool,
+}
+
+/// The extension objects a source may add to a request the driver built, one
+/// for the request and one for each impression.
+///
+/// This is the only part of the request a source writes. Every standard field
+/// is the driver's, chosen through [`DemandFieldPolicy`], so a source can add
+/// what its exchange needs and cannot change what privacy enforcement decided.
+/// The `trusted_server` request extension is the driver's too, and a request
+/// that claims it is refused.
+pub struct RequestExtensions<'a> {
+    /// The request-level `ext` object.
+    pub request: &'a mut Option<Map<String, Value>>,
+    /// The impressions the driver built, each beside the slot it came from.
+    pub impressions: Vec<ImpressionExtension<'a>>,
+}
+
+/// One impression's `ext` object beside the routed slot it was built from.
+pub struct ImpressionExtension<'a> {
+    /// The routed slot, with its bidder parameters and zone.
+    pub slot: &'a ProviderSlotInput,
+    /// The impression's `ext` object.
+    pub ext: &'a mut Option<Map<String, Value>>,
+    /// Set by a source that has no demand to send for this slot. The driver
+    /// leaves the impression out of the request, and makes no request at all
+    /// when every impression is left out.
+    pub omitted: bool,
+}
+
+/// What one demand source that speaks `OpenRTB` decides, compiled from its
+/// `[demand.<name>]` table at startup and run by [`OpenRtbDriver`].
+///
+/// Every method except [`field_policy`](Self::field_policy),
+/// [`augment_request`](Self::augment_request),
+/// [`parse_response`](Self::parse_response) and [`as_any`](Self::as_any) has
+/// a default that suits a source with no special handling.
+#[async_trait(?Send)]
+pub trait OpenRtbDemand: Send + Sync {
+    /// The standard field choices this source makes.
+    fn field_policy(&self) -> DemandFieldPolicy;
+
+    /// The domain sent as `site.domain` and `site.publisher.domain`.
+    fn site_domain(&self, publisher_domain: &str) -> String {
+        publisher_domain.to_owned()
+    }
+
+    /// The page sent as `site.page`, given the publisher's page and the site
+    /// domain [`site_domain`](Self::site_domain) chose.
+    fn site_page(&self, publisher_page: Option<&str>, site_domain: &str) -> Option<String> {
+        let _ = site_domain;
+        publisher_page.map(str::to_owned)
+    }
+
+    /// The consent that may travel in the request body.
+    fn body_consent<'c>(&self, consent: Option<&'c ConsentContext>) -> Option<&'c ConsentContext> {
+        consent
+    }
+
+    /// Adds this source's own request and impression extensions.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error when an extension cannot be built for this request.
+    fn augment_request(
+        &self,
+        extensions: &mut RequestExtensions<'_>,
+        input: &ProviderAuctionInput,
+    ) -> Result<(), Report<TrustedServerError>>;
+
+    /// Adds to the outbound headers before the request is sent, for example to
+    /// forward admitted headers. The driver has already set `Content-Type`,
+    /// and `Accept` when the field policy asks for it.
+    fn prepare_outbound(&self, headers: &mut HeaderMap, transport: DemandTransport<'_>) {
+        let _ = (headers, transport);
+    }
+
+    /// Keeps request-local state the response parser needs, taken from the
+    /// serialized body and the outbound headers as they will be sent, after
+    /// [`prepare_outbound`](Self::prepare_outbound) has run.
+    fn capture_request(
+        &self,
+        body: &[u8],
+        headers: &HeaderMap,
+    ) -> Option<Box<dyn Any + Send + Sync>> {
+        let _ = (body, headers);
+        None
+    }
+
+    /// Reads the upstream response into an auction response.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error when the response cannot be read at all. A response
+    /// that arrived but is not usable is reported as an error response rather
+    /// than as an `Err`.
+    async fn parse_response(
+        &self,
+        context: DemandResponse<'_>,
+        response: PlatformResponse,
+    ) -> Result<AuctionResponse, Report<TrustedServerError>>;
+
+    /// This source as [`Any`], so the implementation that created it can read
+    /// its own settings back, for example while it registers routes.
+    fn as_any(&self) -> &dyn Any;
+}
+
+/// Runs one [`OpenRtbDemand`] through core's demand seam.
+///
+/// An implementation's `compile` wraps the source it compiled in this, so
+/// core sends the request the driver builds and hands the answer back to the
+/// source's own reader. Core's [`CompiledDemand::as_any`] answers with the
+/// wrapped source, so the implementation reads its own settings back as if
+/// nothing stood between.
+pub struct OpenRtbDriver {
+    demand: Box<dyn OpenRtbDemand>,
+}
+
+impl OpenRtbDriver {
+    /// The driver over one compiled source.
+    #[must_use]
+    pub fn new(demand: impl OpenRtbDemand + 'static) -> Self {
+        Self {
+            demand: Box::new(demand),
+        }
+    }
+
+    /// The source the driver runs.
+    #[must_use]
+    pub fn demand(&self) -> &dyn OpenRtbDemand {
+        self.demand.as_ref()
+    }
+}
+
+#[async_trait(?Send)]
+impl CompiledDemand for OpenRtbDriver {
+    fn build_request(
+        &self,
+        input: &ProviderAuctionInput,
+        transport: DemandTransport<'_>,
+        timeout_ms: u32,
+        finalization: &RequestFinalization<'_>,
+    ) -> Result<Option<OutboundRequest>, Report<TrustedServerError>> {
+        let demand = self.demand.as_ref();
+        let Some(request) =
+            crate::request::build_request(demand, input, transport, timeout_ms, finalization)?
+        else {
+            return Ok(None);
+        };
+        let slot_ids = request
+            .imp
+            .iter()
+            .filter_map(|impression| impression.id.clone())
+            .collect::<HashSet<_>>();
+        let body = serde_json::to_vec(&request).change_context(TrustedServerError::Auction {
+            message: format!(
+                "Provider {} request serialization failed",
+                input.provider_id().as_str()
+            ),
+        })?;
+        let mut headers = HeaderMap::new();
+        headers.insert(
+            header::CONTENT_TYPE,
+            HeaderValue::from_static("application/json"),
+        );
+        if demand.field_policy().accept_json {
+            headers.insert(header::ACCEPT, HeaderValue::from_static("application/json"));
+        }
+        demand.prepare_outbound(&mut headers, transport);
+        let captured = demand.capture_request(&body, &headers);
+        Ok(Some(OutboundRequest {
+            body,
+            headers,
+            slot_ids,
+            captured,
+        }))
+    }
+
+    async fn parse_response(
+        &self,
+        context: DemandResponse<'_>,
+        response: PlatformResponse,
+    ) -> Result<AuctionResponse, Report<TrustedServerError>> {
+        self.demand.parse_response(context, response).await
+    }
+
+    fn consumes_bidder_params(&self) -> bool {
+        self.demand.field_policy().consumes_bidder_params
+    }
+
+    fn as_any(&self) -> &dyn Any {
+        self.demand.as_any()
+    }
+}