From 1e20c8352487e13952b473166861f853c824cd9d Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Sat, 10 Oct 2026 03:31:29 +0100 Subject: [PATCH 1/3] The sellers a deployment adds to the publisher's ads.txt are a module `[ads-txt] modules = ["sellers"]` selects it, with the lines to add in `[ads-txt.sellers] lines`. The module is a middleware for `text/plain`, named in a `[[fetch]]` entry on the file's path, which passes the origin's file through and writes the lines after its last one, on a line of their own, so a seller the deployment introduces is declared beside the publisher's own. The publisher's file stays authoritative, and a publisher whose origin has no file gets the origin's answer, because the module adds to the publisher's file and never invents one. Every buyer's crawler reads the file and a typo there costs the publisher revenue silently, so each line is checked as the settings load against the file's grammar: a data record is a domain, an account id, DIRECT or RESELLER and an optional certification authority id, separated by commas, and a variable is one of CONTACT, SUBDOMAIN, INVENTORYPARTNERDOMAIN, OWNERDOMAIN or MANAGERDOMAIN with a value. A comment, an empty line and an empty list are refused, naming the line and what is wrong with it, and so is a document that selects the module but names it in no entry, where the lines would never be written. The stock list offers the module after the page modules and lists its middleware in the fetch phase, the example configuration shows its tables and entry, the cargo aliases build and test it with the other modules, and the crate is the project's own, which its manifest states. The tests cover each refusal, the lines the grammar accepts, the file with and without a final line break, an empty file, a file read in pieces, and the module's place in a registry. --- .cargo/config.toml | 12 +- Cargo.lock | 11 + Cargo.toml | 2 + crates/ads-txt/sellers/Cargo.toml | 30 + .../src/fixtures/sellers.settings.toml | 10 + crates/ads-txt/sellers/src/lib.rs | 525 ++++++++++++++++++ crates/trusted-server-modules/Cargo.toml | 1 + crates/trusted-server-modules/src/lib.rs | 81 ++- trusted-server.example.toml | 15 + 9 files changed, 657 insertions(+), 30 deletions(-) create mode 100644 crates/ads-txt/sellers/Cargo.toml create mode 100644 crates/ads-txt/sellers/src/fixtures/sellers.settings.toml create mode 100644 crates/ads-txt/sellers/src/lib.rs diff --git a/.cargo/config.toml b/.cargo/config.toml index a0f90ba645..211c7a899b 100644 --- a/.cargo/config.toml +++ b/.cargo/config.toml @@ -26,10 +26,10 @@ test_details = "test --target aarch64-apple-darwin" # native crate needs no change here. Axum (native), Cloudflare # (wasm32-unknown-unknown), Spin, the CLI (native), and integration-tests # (native) are simply not listed. -build-fastly = "build -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1" -check-fastly = "check -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1" -clippy-fastly = "clippy -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --all-targets --all-features --target wasm32-wasip1 -- -D warnings" -test-fastly = "test -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1" +build-fastly = "build -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-ads-txt-sellers -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1" +check-fastly = "check -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-ads-txt-sellers -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1" +clippy-fastly = "clippy -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-ads-txt-sellers -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --all-targets --all-features --target wasm32-wasip1 -- -D warnings" +test-fastly = "test -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-device-fastly -p trusted-server-geo-fastly -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-ads-txt-sellers -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1" # Feature-on counterpart of `test-fastly`. `test-fastly` does NOT pass # --all-features, so without this the reusable-sandbox code is linted by # `clippy-fastly` but its tests never execute. @@ -40,8 +40,8 @@ test-fastly-reuse = "test -p trusted-server-adapter-fastly --target wasm32-wasip # run with the Axum adapter in CI. build-axum = "build -p trusted-server-adapter-axum" check-axum = "check -p trusted-server-adapter-axum" -clippy-axum = "clippy -p trusted-server-adapter-axum -p trusted-server-permission-signal-gpc -p trusted-server-permission-signal-gpp -p trusted-server-permission-signal-mtm -p trusted-server-permission-signal-tcf -p trusted-server-permission-signal-us-privacy -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb --all-targets --all-features -- -D warnings" -test-axum = "test -p trusted-server-adapter-axum -p trusted-server-permission-signal-gpc -p trusted-server-permission-signal-gpp -p trusted-server-permission-signal-mtm -p trusted-server-permission-signal-tcf -p trusted-server-permission-signal-us-privacy -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb" +clippy-axum = "clippy -p trusted-server-adapter-axum -p trusted-server-permission-signal-gpc -p trusted-server-permission-signal-gpp -p trusted-server-permission-signal-mtm -p trusted-server-permission-signal-tcf -p trusted-server-permission-signal-us-privacy -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-ads-txt-sellers -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb --all-targets --all-features -- -D warnings" +test-axum = "test -p trusted-server-adapter-axum -p trusted-server-permission-signal-gpc -p trusted-server-permission-signal-gpp -p trusted-server-permission-signal-mtm -p trusted-server-permission-signal-tcf -p trusted-server-permission-signal-us-privacy -p trusted-server-cmp-osano -p trusted-server-modules -p trusted-server-identity-lockr -p trusted-server-audience-permutive -p trusted-server-cmp-sourcepoint -p trusted-server-cmp-didomi -p trusted-server-cmp-inmobi -p trusted-server-ads-txt-sellers -p trusted-server-tag-google-tag-manager -p trusted-server-testing-testlight -p trusted-server-framework-nextjs -p trusted-server-ad-tag-google -p trusted-server-bot-protection-datadome -p trusted-server-ad-server-mock -p trusted-server-auction-aps -p trusted-server-auction-prebid -p trusted-server-auction-prebid-server -p trusted-server-auction-protocol-openrtb" # --- Cloudflare adapter (native host + wasm32-unknown-unknown) --- # Build/check target the WASM runtime (requires the `cloudflare` feature); diff --git a/Cargo.lock b/Cargo.lock index f964df6e8a..ff8aa0428e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6664,6 +6664,16 @@ dependencies = [ "trusted-server-permission-signal-us-privacy", ] +[[package]] +name = "trusted-server-ads-txt-sellers" +version = "0.1.0" +dependencies = [ + "error-stack", + "serde", + "trusted-server-core", + "validator", +] + [[package]] name = "trusted-server-auction-aps" version = "0.1.0" @@ -7056,6 +7066,7 @@ dependencies = [ "tokio", "trusted-server-ad-server-mock", "trusted-server-ad-tag-google", + "trusted-server-ads-txt-sellers", "trusted-server-auction-aps", "trusted-server-auction-prebid", "trusted-server-auction-prebid-server", diff --git a/Cargo.toml b/Cargo.toml index 665192d19a..1ce94a0328 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -19,6 +19,7 @@ members = [ "crates/cmp/sourcepoint", "crates/cmp/didomi", "crates/cmp/inmobi", + "crates/ads-txt/sellers", "crates/tag/google-tag-manager", "crates/testing/testlight", "crates/framework/nextjs", @@ -163,6 +164,7 @@ trusted-server-audience-permutive = { path = "crates/audience/permutive" } trusted-server-cmp-sourcepoint = { path = "crates/cmp/sourcepoint" } trusted-server-cmp-didomi = { path = "crates/cmp/didomi" } trusted-server-cmp-inmobi = { path = "crates/cmp/inmobi" } +trusted-server-ads-txt-sellers = { path = "crates/ads-txt/sellers" } trusted-server-tag-google-tag-manager = { path = "crates/tag/google-tag-manager" } trusted-server-testing-testlight = { path = "crates/testing/testlight" } trusted-server-framework-nextjs = { path = "crates/framework/nextjs" } diff --git a/crates/ads-txt/sellers/Cargo.toml b/crates/ads-txt/sellers/Cargo.toml new file mode 100644 index 0000000000..aa4cdf188a --- /dev/null +++ b/crates/ads-txt/sellers/Cargo.toml @@ -0,0 +1,30 @@ +[package] +name = "trusted-server-ads-txt-sellers" +description = "Adds a publisher's sellers to the origin's ads.txt as it passes through, from the lines the settings give." +authors = { workspace = true } +edition = { workspace = true } +license = { workspace = true } +publish = { workspace = true } +version = { workspace = true } + +[lib] +doctest = false + +[lints] +workspace = true + +[dependencies] +error-stack = { workspace = true } +serde = { workspace = true } +trusted-server-core = { workspace = true } +validator = { workspace = true } + +[dev-dependencies] +trusted-server-core = { workspace = true, features = ["test-utils"] } + +# The visible owner of this crate, the way Prebid.js requires a named +# maintainer of every adapter. The module is the project's own, because it +# writes what the settings say and names no vendor. +[package.metadata.maintainers] +owner = "Trusted Server maintainers" +status = "project owned" diff --git a/crates/ads-txt/sellers/src/fixtures/sellers.settings.toml b/crates/ads-txt/sellers/src/fixtures/sellers.settings.toml new file mode 100644 index 0000000000..1d4b71bcd8 --- /dev/null +++ b/crates/ads-txt/sellers/src/fixtures/sellers.settings.toml @@ -0,0 +1,10 @@ +[ads-txt] +modules = ["sellers"] + +[ads-txt.sellers] +lines = ["ssp.example, 12345, DIRECT, f08c47fec0942fa0", "CONTACT=ads@publisher.example"] + +[[fetch]] +media_type = "text/plain" +path = "/ads.txt" +middleware = ["ads-txt.sellers"] diff --git a/crates/ads-txt/sellers/src/lib.rs b/crates/ads-txt/sellers/src/lib.rs new file mode 100644 index 0000000000..a01ba90441 --- /dev/null +++ b/crates/ads-txt/sellers/src/lib.rs @@ -0,0 +1,525 @@ +//! The sellers a deployment adds to the publisher's `ads.txt`. +//! +//! The publisher's own file at the origin stays authoritative. Where a +//! `[[fetch]]` entry for `text/plain` names [`MODULE`] on the file's path, +//! this module's middleware passes the file through and writes the lines +//! `[ads-txt.sellers] lines` gives at its end, so a seller the deployment +//! introduces is declared beside the publisher's own. A publisher whose +//! origin has no file gets the origin's answer, because the module adds to +//! the publisher's file and never invents one. +//! +//! Each line is checked as the settings load against the file's grammar, +//! because every buyer's crawler reads the file and a typo there costs the +//! publisher revenue silently: a data record is a domain, an account id, a +//! relationship of `DIRECT` or `RESELLER` and an optional certification +//! authority id, separated by commas, and a variable is one of the five the +//! specification names, with its value. + +use std::sync::Arc; + +use error_stack::Report; +use serde::Deserialize; +use validator::Validate; + +use trusted_server_core::error::TrustedServerError; +use trusted_server_core::integrations::{IntegrationBuilder, IntegrationRegistration}; +use trusted_server_core::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, +}; +use trusted_server_core::settings::{IntegrationConfig, Settings}; +use trusted_server_core::streaming_processor::StreamProcessor; + +/// The id the module's registration carries. +const SELLERS_INTEGRATION_ID: &str = "ads_txt_sellers"; + +/// The name `[ads-txt] modules` selects the module by and a `[[fetch]]` entry +/// names its middleware by, from this crate's folder. Its settings are the +/// table `[ads-txt.sellers]`. +pub const MODULE: &str = "ads-txt.sellers"; + +/// The media type an `ads.txt` is served as, and the one the middleware +/// works on. +pub const MEDIA_TYPE: &str = "text/plain"; + +/// The variables the specification names, each written as `NAME=value`. +const VARIABLES: [&str; 5] = [ + "CONTACT", + "SUBDOMAIN", + "INVENTORYPARTNERDOMAIN", + "OWNERDOMAIN", + "MANAGERDOMAIN", +]; + +/// Configuration for the sellers added to `ads.txt`. +#[derive(Debug, Clone, Deserialize, Validate)] +#[serde(deny_unknown_fields)] +pub struct SellersConfig { + /// The lines written at the end of the publisher's file, each a data + /// record or a variable, as the file's grammar has them. + pub lines: Vec, +} + +impl IntegrationConfig for SellersConfig {} + +/// Refuses a line that is not a data record or a variable, saying which part +/// is wrong. +/// +/// A comment or an empty line is refused too, because the settings carry +/// what the deployment declares and a comment declares nothing. +fn check_line(line: &str) -> Result<(), String> { + let trimmed = line.trim(); + if trimmed.is_empty() { + return Err("is empty".to_owned()); + } + if trimmed.contains('#') { + return Err("carries a comment, and a line in the settings declares something".to_owned()); + } + if trimmed.contains(['\n', '\r']) { + return Err("is more than one line".to_owned()); + } + if let Some((name, value)) = trimmed.split_once('=') { + let name = name.trim(); + if VARIABLES.contains(&name) { + if value.trim().is_empty() { + return Err(format!("gives `{name}` no value")); + } + return Ok(()); + } + return Err(format!( + "names the variable `{name}`, which is not one of {}", + VARIABLES.join(", ") + )); + } + let fields: Vec<&str> = trimmed.split(',').map(str::trim).collect(); + if fields.len() < 3 || fields.len() > 4 { + return Err(format!( + "has {} fields, and a data record has three or four: the domain, the account id, \ + DIRECT or RESELLER, and the certification authority id", + fields.len() + )); + } + if fields[0].is_empty() || fields[0].contains(char::is_whitespace) { + return Err("has no domain where its first field should be one".to_owned()); + } + if fields[1].is_empty() { + return Err("has no account id in its second field".to_owned()); + } + if !fields[2].eq_ignore_ascii_case("DIRECT") && !fields[2].eq_ignore_ascii_case("RESELLER") { + return Err(format!( + "says the relationship is `{}`, and it is DIRECT or RESELLER", + fields[2] + )); + } + if fields.len() == 4 && fields[3].is_empty() { + return Err("ends with a comma and no certification authority id".to_owned()); + } + Ok(()) +} + +/// Refuses settings whose lines are not each a record or a variable, or that +/// give no line. +fn validate_lines(config: &SellersConfig) -> Result<(), Report> { + if config.lines.is_empty() { + return Err(Report::new(TrustedServerError::Settings { + message: "`lines` in [ads-txt.sellers] is empty, so the module would add nothing. \ + Give the lines to add, or do not select the module" + .to_owned(), + })); + } + for (index, line) in config.lines.iter().enumerate() { + if let Err(problem) = check_line(line) { + return Err(Report::new(TrustedServerError::Settings { + message: format!( + "`lines` in [ads-txt.sellers], line {}, `{}`, {problem}", + index + 1, + line.trim() + ), + })); + } + } + Ok(()) +} + +/// Refuses a document that selects the module but names it in no entry, where +/// the lines would never be written. +fn require_an_entry(settings: &Settings) -> Result<(), Report> { + let placed = settings.fetch.entries().iter().any(|entry| { + entry.media_type == MEDIA_TYPE && entry.middleware.iter().any(|name| name == MODULE) + }); + if placed { + return Ok(()); + } + Err(Report::new(TrustedServerError::Settings { + message: format!( + "[ads-txt] selects `sellers`, but no [[fetch]] entry for `{MEDIA_TYPE}` names \ + `{MODULE}`, so the lines would never be written. Add an entry with media_type = \ + `{MEDIA_TYPE}`, path = `/ads.txt` and the middleware" + ), + })) +} + +/// Validates the sellers' configuration and reports whether a section selects +/// the module. +/// +/// # Errors +/// +/// Returns an error when the table cannot be read, a line is not a record or +/// a variable, no line is given, or no entry places the middleware. +pub fn validate(settings: &Settings) -> Result> { + let Some(config) = settings.module_config::(MODULE)? else { + return Ok(false); + }; + validate_lines(&config)?; + require_an_entry(settings)?; + Ok(true) +} + +/// Registers the sellers' middleware when a section selects the module. +/// +/// # Errors +/// +/// Returns the errors [`validate`] returns. +pub fn register( + settings: &Settings, +) -> Result, Report> { + let Some(config) = settings.module_config::(MODULE)? else { + return Ok(None); + }; + validate_lines(&config)?; + require_an_entry(settings)?; + Ok(Some( + IntegrationRegistration::builder(SELLERS_INTEGRATION_ID) + .without_js() + .with_middleware(Arc::new(Sellers::new(&config.lines))) + .build(), + )) +} + +/// The builder a deployment installs to offer this module. +#[must_use] +pub fn builder() -> IntegrationBuilder { + IntegrationBuilder::new( + SELLERS_INTEGRATION_ID, + env!("CARGO_PKG_NAME"), + register, + validate, + ) + .with_module_name(MODULE) +} + +/// The lines written at the end of the file, as the middleware adds them. +#[derive(Debug)] +pub struct Sellers { + /// The lines, each ending in a line break. + tail: Arc, +} + +impl Sellers { + /// The middleware for `lines`, which the caller has checked. + #[must_use] + pub fn new(lines: &[String]) -> Self { + let mut tail = String::new(); + for line in lines { + tail.push_str(line.trim()); + tail.push('\n'); + } + Self { + tail: Arc::from(tail), + } + } +} + +impl Middleware for Sellers { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn handles_media_type(&self, media_type: &str) -> bool { + media_type == MEDIA_TYPE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + stream: Some(Box::new(AppendLines { + tail: Arc::clone(&self.tail), + ended_with_break: true, + })), + ..MiddlewareAction::pass() + } + } +} + +/// Passes the file through and writes the lines after its last byte, on a +/// line of their own. +struct AppendLines { + tail: Arc, + /// Whether the file so far ends with a line break, true for an empty + /// file, so the lines never join the publisher's last one. + ended_with_break: bool, +} + +impl StreamProcessor for AppendLines { + fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> Result, std::io::Error> { + let mut output = chunk.to_vec(); + if let Some(last) = chunk.last() { + self.ended_with_break = *last == b'\n'; + } + if is_last { + if !self.ended_with_break { + output.push(b'\n'); + } + output.extend_from_slice(self.tail.as_bytes()); + } + Ok(output) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use trusted_server_core::html_processor::test_support::page_settings; + use trusted_server_core::integrations::{IntegrationDocumentState, IntegrationRegistry}; + use trusted_server_core::middleware::MiddlewareChain; + use trusted_server_core::middleware::test_support::context; + + const RECORD: &str = "ssp.example, 12345, DIRECT, f08c47fec0942fa0"; + + /// A document selecting the module with `lines`, placed on the file. + fn settings_with(lines: &[&str]) -> Settings { + let quoted: Vec = lines.iter().map(|line| format!("\"{line}\"")).collect(); + page_settings(&format!( + "[ads-txt]\nmodules = [\"sellers\"]\n\n[ads-txt.sellers]\nlines = [{}]\n\n\ + [[fetch]]\nmedia_type = \"text/plain\"\npath = \"/ads.txt\"\n\ + middleware = [\"ads-txt.sellers\"]\n", + quoted.join(", ") + )) + } + + /// What `body` becomes through the middleware, read `chunk` bytes at a + /// time. + fn through(lines: &[&str], body: &str, chunk: usize) -> String { + let lines: Vec = lines.iter().map(|line| (*line).to_owned()).collect(); + let document_state = IntegrationDocumentState::default(); + let plan = MiddlewareChain::new( + MiddlewarePhase::Fetch, + MEDIA_TYPE, + vec![Arc::new(Sellers::new(&lines))], + ) + .plan(&context(MiddlewarePhase::Fetch, &document_state)) + .expect("the chain should plan"); + let mut processors = plan.processors; + assert_eq!(processors.len(), 1, "the middleware adds one processor"); + let processor = &mut processors[0]; + let bytes = body.as_bytes(); + let mut output = Vec::new(); + if bytes.is_empty() { + output.extend(processor.process_chunk(&[], true).expect("should process")); + } else { + let chunks: Vec<&[u8]> = bytes.chunks(chunk).collect(); + for (index, piece) in chunks.iter().enumerate() { + let is_last = index + 1 == chunks.len(); + output.extend( + processor + .process_chunk(piece, is_last) + .expect("should process"), + ); + } + } + String::from_utf8(output).expect("text") + } + + #[test] + fn module_constant_is_the_crate_folder() { + assert_eq!( + MODULE, + trusted_server_core::module_name!(), + "should be named by the folder this crate lives in" + ); + } + + #[test] + fn an_unselected_module_does_not_register() { + let settings = page_settings(""); + assert!(register(&settings).expect("should read").is_none()); + assert!(!validate(&settings).expect("should read")); + } + + #[test] + fn the_lines_are_offered_as_the_modules_middleware_on_text() { + let settings = settings_with(&[RECORD]); + let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) + .expect("should build the registry"); + + assert_eq!( + registry + .middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + MEDIA_TYPE, + "/ads.txt" + ) + .ids(), + [MODULE], + "should run on the file" + ); + assert!( + registry + .middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + MEDIA_TYPE, + "/robots.txt" + ) + .is_empty(), + "should run on no other path" + ); + let registration = register(&settings) + .expect("should parse") + .expect("should register"); + assert!( + registration.js_disabled, + "nothing of this module runs in the browser" + ); + } + + #[test] + fn a_document_naming_the_module_in_no_entry_is_refused() { + let settings = page_settings(&format!( + "[ads-txt]\nmodules = [\"sellers\"]\n\n[ads-txt.sellers]\nlines = [\"{RECORD}\"]\n" + )); + + let refused = validate(&settings).expect_err("should refuse a module no entry places"); + assert!( + format!("{refused:?}") + .contains("no [[fetch]] entry for `text/plain` names `ads-txt.sellers`"), + "{refused:?}" + ); + assert!(register(&settings).is_err()); + } + + #[test] + fn a_line_that_is_not_a_record_or_a_variable_is_refused_for_its_reason() { + let cases = [ + ("", "line 2, ``, is empty"), + ("# a comment", "carries a comment"), + ("ssp.example, 12345", "has 2 fields"), + ("ssp.example, 12345, DIRECT, abc, extra", "has 5 fields"), + (", 12345, DIRECT", "has no domain"), + ("ssp example, 12345, DIRECT", "has no domain"), + ("ssp.example, , DIRECT", "has no account id"), + ( + "ssp.example, 12345, PARTNER", + "says the relationship is `PARTNER`", + ), + ("ssp.example, 12345, DIRECT, ", "ends with a comma"), + ("CONTACT=", "gives `CONTACT` no value"), + ("EMAIL=ads@publisher.example", "names the variable `EMAIL`"), + ]; + for (line, reason) in cases { + let settings = settings_with(&[RECORD, line]); + for (call, refusal) in [ + ("validate", validate(&settings).err()), + ("register", register(&settings).err()), + ] { + let message = refusal + .unwrap_or_else(|| panic!("{call} should refuse {line:?}")) + .current_context() + .to_string(); + assert!( + message.contains("`lines` in [ads-txt.sellers], line 2") + && message.contains(reason), + "{call} should say {reason:?} of {line:?}, and said {message:?}" + ); + } + } + } + + #[test] + fn no_line_at_all_is_refused() { + let settings = settings_with(&[]); + let message = validate(&settings) + .expect_err("should refuse an empty list") + .current_context() + .to_string(); + assert!( + message.contains("is empty, so the module would add nothing"), + "{message}" + ); + } + + #[test] + fn records_and_variables_are_accepted_as_the_grammar_has_them() { + for line in [ + "ssp.example, 12345, DIRECT", + "ssp.example,12345,reseller,f08c47fec0942fa0", + " ssp.example , 12345 , Direct ", + "CONTACT=ads@publisher.example", + "SUBDOMAIN=news.publisher.example", + "INVENTORYPARTNERDOMAIN=partner.example", + "OWNERDOMAIN=publisher.example", + "MANAGERDOMAIN=manager.example, GB", + ] { + let settings = settings_with(&[line]); + assert!(validate(&settings).unwrap_or_else(|error| panic!("{line}: {error:?}"))); + } + } + + #[test] + fn the_lines_follow_the_publishers_file_on_a_line_of_their_own() { + let lines = [RECORD, "CONTACT=ads@publisher.example"]; + let added = format!("{RECORD}\nCONTACT=ads@publisher.example\n"); + + assert_eq!( + through(&lines, "own.example, 1, DIRECT\n", 8192), + format!("own.example, 1, DIRECT\n{added}"), + "a file ending with a line break takes the lines straight after it" + ); + assert_eq!( + through(&lines, "own.example, 1, DIRECT", 8192), + format!("own.example, 1, DIRECT\n{added}"), + "a file without a final line break gets one before the lines" + ); + assert_eq!( + through(&lines, "", 8192), + added, + "an empty file is the lines" + ); + } + + #[test] + fn a_file_read_in_pieces_is_the_same_file() { + let lines = [RECORD]; + let body = "a.example, 1, DIRECT\nb.example, 2, RESELLER"; + let whole = through(&lines, body, 8192); + + for chunk in [1, 3, 7, 20] { + assert_eq!( + through(&lines, body, chunk), + whole, + "read {chunk} bytes at a time" + ); + } + assert_eq!(whole, format!("{body}\n{RECORD}\n")); + } + + #[test] + fn the_lines_are_written_as_given_and_trimmed() { + assert_eq!( + through(&[" ssp.example, 1, direct "], "x\n", 8192), + "x\nssp.example, 1, direct\n", + "the relationship is written as the publisher wrote it" + ); + } + + #[test] + fn the_middleware_works_on_text_alone_in_the_fetch_phase() { + let sellers = Sellers::new(&[RECORD.to_owned()]); + assert!(sellers.handles_media_type(MEDIA_TYPE)); + assert!(!sellers.handles_media_type("text/html")); + assert_eq!(sellers.phases(), [MiddlewarePhase::Fetch]); + } +} diff --git a/crates/trusted-server-modules/Cargo.toml b/crates/trusted-server-modules/Cargo.toml index 488c14fbde..9ba7a7b20e 100644 --- a/crates/trusted-server-modules/Cargo.toml +++ b/crates/trusted-server-modules/Cargo.toml @@ -24,6 +24,7 @@ trusted-server-audience-permutive = { workspace = true } trusted-server-bot-protection-datadome = { workspace = true } trusted-server-cmp-didomi = { workspace = true } trusted-server-cmp-inmobi = { workspace = true } +trusted-server-ads-txt-sellers = { workspace = true } trusted-server-cmp-osano = { workspace = true } trusted-server-cmp-sourcepoint = { workspace = true } trusted-server-core = { workspace = true } diff --git a/crates/trusted-server-modules/src/lib.rs b/crates/trusted-server-modules/src/lib.rs index 04eabf82b2..57aac2679b 100644 --- a/crates/trusted-server-modules/src/lib.rs +++ b/crates/trusted-server-modules/src/lib.rs @@ -18,7 +18,7 @@ )] use trusted_server_core::integrations::IntegrationBuilder; -use trusted_server_core::middleware::MiddlewarePhase; +use trusted_server_core::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; /// The builders of the modules a stock build ships from crates of their own, /// in hook order. @@ -40,6 +40,7 @@ pub fn builders() -> Vec { trusted_server_bot_protection_datadome::builder(), trusted_server_ad_tag_google::builder(), trusted_server_ad_tag_google::diagnostics::builder(), + trusted_server_ads_txt_sellers::builder(), // Implementations `[demand]` and `[ad-server]` can name, which no // section selects. trusted_server_auction_protocol_openrtb::builder(), @@ -81,6 +82,8 @@ pub struct StockMiddleware { pub integration: &'static str, /// The phase it runs in. pub phase: MiddlewarePhase, + /// The media type it works on, which an entry naming it covers. + pub media_type: &'static str, /// The name an entry writes. pub name: &'static str, } @@ -96,6 +99,7 @@ pub struct StockMiddleware { pub fn middleware() -> Vec { use MiddlewarePhase::{Fetch, Serve}; use trusted_server_ad_tag_google as google; + use trusted_server_ads_txt_sellers as ads_txt_sellers; use trusted_server_auction_prebid as prebid; use trusted_server_audience_permutive as permutive; use trusted_server_bot_protection_datadome as datadome; @@ -111,6 +115,7 @@ pub fn middleware() -> Vec { let row = |integration, phase, name| StockMiddleware { integration, phase, + media_type: HTML_MEDIA_TYPE, name, }; vec![ @@ -137,11 +142,19 @@ pub fn middleware() -> Vec { Serve, google::diagnostics::MODULE, ), + StockMiddleware { + integration: ads_txt_sellers::builder().id(), + phase: Fetch, + media_type: ads_txt_sellers::MEDIA_TYPE, + name: ads_txt_sellers::MODULE, + }, ] } -/// The names of the middleware that the stock modules with the integration -/// ids `ids` supply for `phase`, in the order an entry names them. +/// The names of the page middleware that the stock modules with the +/// integration ids `ids` supply for `phase`, in the order an entry names +/// them. A middleware for a media type other than HTML is left out, because +/// the entry a tool writes covers every page. /// /// A tool that knows a module by its integration id, as the `ts` audit does /// from what it finds on a page, writes the entry that places the module's @@ -150,7 +163,11 @@ pub fn middleware() -> Vec { pub fn middleware_for(ids: &[&str], phase: MiddlewarePhase) -> Vec<&'static str> { middleware() .into_iter() - .filter(|row| row.phase == phase && ids.contains(&row.integration)) + .filter(|row| { + row.phase == phase + && row.media_type == HTML_MEDIA_TYPE + && ids.contains(&row.integration) + }) .map(|row| row.name) .collect() } @@ -158,7 +175,7 @@ pub fn middleware_for(ids: &[&str], phase: MiddlewarePhase) -> Vec<&'static str> #[cfg(test)] mod tests { use trusted_server_core::integrations::{IntegrationBuilder, IntegrationRegistry}; - use trusted_server_core::middleware::MiddlewarePhase; + use trusted_server_core::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; use super::{builders, builders_with, middleware, middleware_for, selection_of}; @@ -179,6 +196,7 @@ mod tests { include_str!("../../tag/google-tag-manager/src/fixtures/page-change.settings.toml"), include_str!("../../bot-protection/datadome/src/fixtures/page-change.settings.toml"), include_str!("../../ad-tag/google/src/fixtures/page-change.settings.toml"), + include_str!("../../ads-txt/sellers/src/fixtures/sellers.settings.toml"), ]; #[test] @@ -272,26 +290,40 @@ mod tests { Settings::from_toml(&template).expect("should load the template with its entries"); for phase in MiddlewarePhase::ALL { - let listed: Vec<&str> = middleware() - .iter() - .filter(|row| row.phase == phase) - .map(|row| row.name) - .collect(); let entries = settings.phase_entries(phase).entries(); - assert_eq!( - entries.len(), - 1, - "should document one [[{phase}]] entry for every page" - ); - assert!( - entries[0].path.is_none(), - "should document the [[{phase}]] entry that covers every page" - ); - assert_eq!( - entries[0].middleware, listed, - "should document every {phase} middleware the stock modules supply, in the \ - listed order" - ); + let mut media_types: Vec<&str> = Vec::new(); + for row in middleware().iter().filter(|row| row.phase == phase) { + if !media_types.contains(&row.media_type) { + media_types.push(row.media_type); + } + } + for media_type in media_types { + let listed: Vec<&str> = middleware() + .iter() + .filter(|row| row.phase == phase && row.media_type == media_type) + .map(|row| row.name) + .collect(); + let covering: Vec<_> = entries + .iter() + .filter(|entry| entry.media_type == media_type) + .collect(); + assert_eq!( + covering.len(), + 1, + "should document one [[{phase}]] entry for {media_type}" + ); + if media_type == HTML_MEDIA_TYPE { + assert!( + covering[0].path.is_none(), + "should document the [[{phase}]] entry that covers every page" + ); + } + assert_eq!( + covering[0].middleware, listed, + "should document every {phase} middleware the stock modules supply for \ + {media_type}, in the listed order" + ); + } } } @@ -379,6 +411,7 @@ mod tests { "bot-protection.datadome", "ad-tag.google", "ad-tag.google.diagnostics", + "ads-txt.sellers", "auction-protocol.openrtb", "auction.prebid-server", "auction.aps", diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 64a66daca6..bd44b9d45f 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -205,6 +205,21 @@ pull_sync_concurrency = 3 # always_allow = ["/ads.txt"] # sitemap = "https://publisher.example.com/sitemap.xml" +# The sellers added to the publisher's ads.txt. The origin's file stays +# authoritative and the lines are written at its end, where a [[fetch]] entry +# for text/plain on the file's path names the middleware. Each line is a data +# record or a variable of the ads.txt grammar, checked as the settings load. +# [ads-txt] +# modules = ["sellers"] + +# [ads-txt.sellers] +# lines = ["ssp.example, 12345, DIRECT, f08c47fec0942fa0"] + +# [[fetch]] +# media_type = "text/plain" +# path = "/ads.txt" +# middleware = ["ads-txt.sellers"] + # Sign outbound OpenRTB/API requests. The runtime reads its keys from fixed # store names (`jwks_store` / `signing_keys`), and `ts keys` writes them. To keep # signing OFF, leave this whole block commented out. From e660d2cb9c8a394b0f517ed1be4bba060951d445 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Sat, 10 Oct 2026 03:31:30 +0100 Subject: [PATCH 2/3] The guides describe [ads-txt] The configuration guide gains the section, its module's table, how the file is made and what is refused, with the entry that places the middleware, and lists the module in its key sections and placing tables. The rules guide names the section among the types and in its table. The changelog records the module. --- AGENTS.md | 1 + CHANGELOG.md | 1 + docs/guide/configuration-rules.md | 4 ++- docs/guide/configuration.md | 50 ++++++++++++++++++++++++++++++- 4 files changed, 54 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 58df514094..fe9e071677 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,6 +23,7 @@ crates/ trusted-server-cli/ # Host-target `ts` operator CLI trusted-server-modules/ # the modules a stock build ships, in the order their hooks run ad-server/ # ad server implementation crates (mock) + ads-txt/ # ads.txt line module crates (sellers) ad-tag/ # ad tag module crates (google, which holds its diagnostics module too) auction/ # auction module crates (aps, prebid, prebid-server) audience/ # audience module crates (permutive) diff --git a/CHANGELOG.md b/CHANGELOG.md index f95bd21a5d..f480d2a597 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,6 +78,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the publisher's `ads.txt`, from `crates/ads-txt/sellers`. The module is a middleware for `text/plain`, named in a `[[fetch]]` entry on the file's path, which passes the origin's file through and writes the lines after its last one, so the publisher's file stays authoritative and a publisher whose origin has no file gets the origin's answer. Each line is checked as the settings load as a data record or a variable of the ads.txt grammar, and a selected module no entry places is refused. - A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript`, `application/json` or `image/svg+xml`, written as a bare `type/subtype` in lower case. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then, for a response that is not stored, the serve phase, and an action that decides about HTML there is refused by name. - The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, and a selected prompt that no entry places is refused. 51Degrees maintain the crate until InMobi adopt it. - `[robots-txt]` serves a publisher's `/robots.txt` from the settings. `modules` names what makes the file, in order. `refuse_all` and `allow_all` are Trusted Server's own, and any other name is a module that contributes rules through the new `RobotsTxtContributor` extension point, declared with `with_robots_txt_contributor`. Trusted Server writes the file, so a contributor cannot produce a malformed one, a refusal cannot be loosened, and a path in `always_allow` stays open to every crawler refused the whole site. With `refuse_all` selected every response an adapter finalizes carries `X-Robots-Tag: noindex, nofollow`. A module's answer is held in the key-value store until its refresh is due, a module that cannot answer has its last answer served, and with none held the answer is `503` and never an empty file. Without the section `/robots.txt` reaches the publisher's origin as before. diff --git a/docs/guide/configuration-rules.md b/docs/guide/configuration-rules.md index 236f06e1d4..1baa6a97bc 100644 --- a/docs/guide/configuration-rules.md +++ b/docs/guide/configuration-rules.md @@ -23,7 +23,8 @@ setting = "value" 1. **The type is the job.** Each type is one top-level table, named for what its modules do: `ec`, `geo`, `device`, `permission-signal`, `demand`, - `ad-server`, `robots-txt` and `analytics`, and the section of each module + `ad-server`, `robots-txt`, `ads-txt` and `analytics`, and the section of each + module type, such as `cmp` or `tag`. 2. **The selector chooses what runs.** A type that runs one module takes `module`, a string. A type that runs several takes `modules`, a list. @@ -54,6 +55,7 @@ setting = "value" | `demand` | several | `modules`, a list | `auction-protocol.openrtb`, `auction.prebid-server`, `auction.aps` | | `ad-server` | one | `module`, a string | `mock` | | `robots-txt` | several, in order | `modules`, a list | `refuse_all`, `allow_all`, or an integration that supplies robots.txt rules | +| `ads-txt` | several, in order | `modules`, a list | `sellers` | | `analytics` | one | `module`, a string | `tinybird` | | `cmp` | one | `module`, a string | `didomi`, `sourcepoint`, `osano` | | `tag` | several | `modules`, a list | `google-tag-manager` | diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 7640bf55cd..951ad93826 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -212,7 +212,7 @@ fail and the service will return its startup-error response. ## Key Sections -11 of these sections select what runs, with `module` where one runs and +12 of these sections select what runs, with `module` where one runs and `modules` where several run, and each gives every selected name its own `[.]` settings table, as [Configuration Rules](/guide/configuration-rules) describes. @@ -220,6 +220,7 @@ fail and the service will return its startup-error response. | Section | Selects | Purpose | | ----------------------------------------------------------------------------------------------------------------- | --------------------- | --------------------------------------------------------------------------------------------- | | `[ad-server]` | one module | The ad server that picks the winner | +| `[ads-txt]` | several modules | The sellers added to the publisher's `ads.txt`, see ads.txt Configuration | | `[analytics]` | one module | The module that receives auction telemetry | | `[auction]` | several modules | Auction orchestration, bidder routes, and the modules the auction runs, `prebid` among them | | `[cache]` | nothing | Static and rehosted asset cache policy | @@ -1074,6 +1075,52 @@ modules = ["refuse_all"] always_allow = ["/ads.txt"] ``` +## ads.txt Configuration + +The sellers a deployment adds to the publisher's `ads.txt`. The publisher's +own file at the origin stays authoritative, and with no `[ads-txt]` section +it passes through untouched. + +### `[ads-txt]` + +| Field | Type | Required | Description | +| --------- | ----------------------- | -------- | ------------------------------------------------------------------------------------------- | +| `modules` | String or Array[String] | Yes | What adds lines to the file, in order. `sellers` writes the lines `[ads-txt.sellers]` gives | + +### `[ads-txt.sellers]` + +| Field | Type | Required | Description | +| ------- | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `lines` | Array[String] | Yes | The lines written at the end of the file, each a data record (domain, account id, `DIRECT` or `RESELLER`, and an optional certification authority id) or a variable | + +**How the file is made.** The module is a middleware for `text/plain`, and +runs where a `[[fetch]]` entry for that media type on the file's path names +it. It passes the origin's file through and writes the lines after its last +one, on a line of their own, so a seller the deployment introduces is +declared beside the publisher's own. A publisher whose origin has no file +gets the origin's answer, because the module adds to the publisher's file +and never invents one. + +**Refused when the settings load**: a line that is not a data record or a +variable, naming which part is wrong, a comment, an empty list, and a +document that selects the module but names it in no `[[fetch]]` entry for +`text/plain`, where the lines would never be written. + +**Example**: + +```toml +[ads-txt] +modules = ["sellers"] + +[ads-txt.sellers] +lines = ["ssp.example, 12345, DIRECT, f08c47fec0942fa0", "CONTACT=ads@publisher.example"] + +[[fetch]] +media_type = "text/plain" +path = "/ads.txt" +middleware = ["ads-txt.sellers"] +``` + ## Request Signing Configuration for Ed25519 request signing. @@ -1784,6 +1831,7 @@ the names in and the order `trusted-server.example.toml` shows them in. | `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | | `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | | `ad-tag.google.diagnostics` | `[[serve]]` | For a request that activated diagnostics, writes the bootstrap ahead of the script bundle and the tag that loads the diagnostics module straight after it | +| `ads-txt.sellers` | `[[fetch]]` | Writes the lines `[ads-txt.sellers]` gives at the end of the publisher's `ads.txt`, on `text/plain` alone | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type From 097b5f2c5c60a3801abe82a457ed59bf0958b4cb Mon Sep 17 00:00:00 2001 From: James Rogers Date: Mon, 12 Oct 2026 15:18:25 +0100 Subject: [PATCH 3/3] The sellers module reads a variable name in either case A variable name is matched in either case, as the relationship already was, and a record whose account id holds an equals sign is read as a record rather than refused as a variable. A domain with a space has its own message. The refusal and the guide say that an entry's path is a prefix, because PhaseEntry::covers is shared by every entry. A file whose lines end with a carriage return is tested, and the module doc, the manifest comment, the guide and the changelog are cut. --- CHANGELOG.md | 6 +-- crates/ads-txt/sellers/Cargo.toml | 4 +- crates/ads-txt/sellers/src/lib.rs | 62 ++++++++++++++++++++----------- docs/guide/configuration-rules.md | 4 +- docs/guide/configuration.md | 16 +++----- 5 files changed, 52 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2f93d905ff..ae4a9ca9dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,9 +78,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the publisher's `ads.txt`, from `crates/ads-txt/sellers`. The module is a middleware for `text/plain`, named in a `[[fetch]]` entry on the file's path, which passes the origin's file through and writes the lines after its last one, so the publisher's file stays authoritative and a publisher whose origin has no file gets the origin's answer. Each line is checked as the settings load as a data record or a variable of the ads.txt grammar, and a selected module no entry places is refused. -- A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript` or `application/json`, written as a bare `type/subtype` in lower case. A response that declares a charset other than UTF-8 passes through untouched. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then the serve phase, and an action that decides about HTML there is refused by name. -- The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, and a selected prompt that no entry places is refused. The Trusted Server maintainers hold the crate until InMobi adopt it. +- `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the end of the publisher's `ads.txt`, from `crates/ads-txt/sellers`, where a `[[fetch]]` entry for `text/plain` names it. Each line is checked against the ads.txt grammar as the settings load, and a selected module no entry places is refused. +- A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript`, `application/json` or `image/svg+xml`, written as a bare `type/subtype` in lower case. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then, for a response that is not stored, the serve phase, and an action that decides about HTML there is refused by name. +- The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, and a selected prompt that no entry places is refused. 51Degrees maintain the crate until InMobi adopt it. - `[robots-txt]` serves a publisher's `/robots.txt` from the settings. `modules` names what makes the file, in order. `refuse_all` and `allow_all` are Trusted Server's own, and any other name is a module that contributes rules through the new `RobotsTxtContributor` extension point, declared with `with_robots_txt_contributor`. Trusted Server writes the file, so a contributor cannot produce a malformed one, a refusal cannot be loosened, and a path in `always_allow` stays open to every crawler refused the whole site. With `refuse_all` selected every response an adapter finalizes carries `X-Robots-Tag: noindex, nofollow`. A module's answer is held in the key-value store until its refresh is due, a module that cannot answer has its last answer served, and with none held the answer is `503` and never an empty file. Without the section `/robots.txt` reaches the publisher's origin as before. - `ts keys rotate` and `ts keys deactivate` rotate and retire the request signing keys from an operator's machine. They write the two stores the service links as `jwks_store` and `signing_keys` through the `fastly` CLI, with the token from `FASTLY_API_TOKEN`. A private key reaches `fastly` on its standard input and never in its arguments, and a key id no platform can store is refused before anything is written. A command that cannot read the config store writes nothing, and a delete that failed part way can be run again. - Attestation. With `[attestation]` in the settings a deployment serves evidence, in the sense of RFC 9334, of who operates it and which build it runs, as a page and as JSON at `[attestation] endpoint`, which defaults to `/_ts/attestation`. The evidence names the host the request arrived for, the publisher, the platform service and the build, with the time and the caller's nonce, and is signed with ECDSA P-256 under a configured context. The signing keys are a schedule compiled in from the build input `TRUSTED_SERVER_ATTESTATION_KEYS`, and a build given none answers `503` at the endpoint. A build also reports the commit, the build run and the build time it was given, from `TRUSTED_SERVER_COMMIT`, `TRUSTED_SERVER_BUILD_RUN` and `SOURCE_DATE_EPOCH`. See `docs/guide/attestation.md`. diff --git a/crates/ads-txt/sellers/Cargo.toml b/crates/ads-txt/sellers/Cargo.toml index aa4cdf188a..1bab501e44 100644 --- a/crates/ads-txt/sellers/Cargo.toml +++ b/crates/ads-txt/sellers/Cargo.toml @@ -22,9 +22,7 @@ validator = { workspace = true } [dev-dependencies] trusted-server-core = { workspace = true, features = ["test-utils"] } -# The visible owner of this crate, the way Prebid.js requires a named -# maintainer of every adapter. The module is the project's own, because it -# writes what the settings say and names no vendor. +# The visible owner of this crate. [package.metadata.maintainers] owner = "Trusted Server maintainers" status = "project owned" diff --git a/crates/ads-txt/sellers/src/lib.rs b/crates/ads-txt/sellers/src/lib.rs index a01ba90441..9f79039526 100644 --- a/crates/ads-txt/sellers/src/lib.rs +++ b/crates/ads-txt/sellers/src/lib.rs @@ -1,19 +1,6 @@ //! The sellers a deployment adds to the publisher's `ads.txt`. //! -//! The publisher's own file at the origin stays authoritative. Where a -//! `[[fetch]]` entry for `text/plain` names [`MODULE`] on the file's path, -//! this module's middleware passes the file through and writes the lines -//! `[ads-txt.sellers] lines` gives at its end, so a seller the deployment -//! introduces is declared beside the publisher's own. A publisher whose -//! origin has no file gets the origin's answer, because the module adds to -//! the publisher's file and never invents one. -//! -//! Each line is checked as the settings load against the file's grammar, -//! because every buyer's crawler reads the file and a typo there costs the -//! publisher revenue silently: a data record is a domain, an account id, a -//! relationship of `DIRECT` or `RESELLER` and an optional certification -//! authority id, separated by commas, and a variable is one of the five the -//! specification names, with its value. +//! The publisher's own file at the origin stays authoritative. use std::sync::Arc; @@ -77,9 +64,14 @@ fn check_line(line: &str) -> Result<(), String> { if trimmed.contains(['\n', '\r']) { return Err("is more than one line".to_owned()); } - if let Some((name, value)) = trimmed.split_once('=') { + if let Some((name, value)) = trimmed.split_once('=') + && !name.contains(',') + { let name = name.trim(); - if VARIABLES.contains(&name) { + if VARIABLES + .iter() + .any(|variable| variable.eq_ignore_ascii_case(name)) + { if value.trim().is_empty() { return Err(format!("gives `{name}` no value")); } @@ -98,9 +90,15 @@ fn check_line(line: &str) -> Result<(), String> { fields.len() )); } - if fields[0].is_empty() || fields[0].contains(char::is_whitespace) { + if fields[0].is_empty() { return Err("has no domain where its first field should be one".to_owned()); } + if fields[0].contains(char::is_whitespace) { + return Err(format!( + "has `{}` as its domain, and a domain holds no space", + fields[0] + )); + } if fields[1].is_empty() { return Err("has no account id in its second field".to_owned()); } @@ -153,7 +151,8 @@ fn require_an_entry(settings: &Settings) -> Result<(), Report, - /// Whether the file so far ends with a line break, true for an empty - /// file, so the lines never join the publisher's last one. + /// Whether the file so far ends with a line break. ended_with_break: bool, } @@ -409,7 +407,10 @@ mod tests { ("ssp.example, 12345", "has 2 fields"), ("ssp.example, 12345, DIRECT, abc, extra", "has 5 fields"), (", 12345, DIRECT", "has no domain"), - ("ssp example, 12345, DIRECT", "has no domain"), + ( + "ssp example, 12345, DIRECT", + "has `ssp example` as its domain, and a domain holds no space", + ), ("ssp.example, , DIRECT", "has no account id"), ( "ssp.example, 12345, PARTNER", @@ -458,6 +459,9 @@ mod tests { "ssp.example,12345,reseller,f08c47fec0942fa0", " ssp.example , 12345 , Direct ", "CONTACT=ads@publisher.example", + "contact=ads@publisher.example", + "Subdomain=news.publisher.example", + "ssp.example, pub=12345, DIRECT", "SUBDOMAIN=news.publisher.example", "INVENTORYPARTNERDOMAIN=partner.example", "OWNERDOMAIN=publisher.example", @@ -490,6 +494,20 @@ mod tests { ); } + #[test] + fn a_file_whose_lines_end_with_a_carriage_return_takes_the_lines_after_its_last_one() { + assert_eq!( + through(&[RECORD], "own.example, 1, DIRECT\r\n", 8192), + format!("own.example, 1, DIRECT\r\n{RECORD}\n"), + "a file ending with a carriage return and a line feed takes no second break" + ); + assert_eq!( + through(&[RECORD], "a.example, 1, DIRECT\r\nb.example, 2, DIRECT", 3), + format!("a.example, 1, DIRECT\r\nb.example, 2, DIRECT\n{RECORD}\n"), + "a file without a final line break gets one before the lines" + ); + } + #[test] fn a_file_read_in_pieces_is_the_same_file() { let lines = [RECORD]; @@ -511,7 +529,7 @@ mod tests { assert_eq!( through(&[" ssp.example, 1, direct "], "x\n", 8192), "x\nssp.example, 1, direct\n", - "the relationship is written as the publisher wrote it" + "the relationship is written as the deployment wrote it" ); } diff --git a/docs/guide/configuration-rules.md b/docs/guide/configuration-rules.md index 39cca64eec..abb932fba9 100644 --- a/docs/guide/configuration-rules.md +++ b/docs/guide/configuration-rules.md @@ -54,8 +54,8 @@ setting = "value" | `permission-signal` | several, in order | `modules`, a list | `gpc`, `gpp`, `us-privacy`, `tcf` | | `demand` | several | `modules`, a list | `auction-protocol.openrtb`, `auction.prebid-server`, `auction.aps` | | `ad-server` | one | `module`, a string | `mock` | -| `robots-txt` | several, in order | `modules`, a list | `refuse_all`, `allow_all`, or a module that supplies robots.txt rules | -| `ads-txt` | several, in order | `modules`, a list | `sellers` | +| `robots-txt` | several, in order | `modules`, a list | `refuse_all`, `allow_all`, or an integration that supplies robots.txt rules | +| `ads-txt` | several | `modules`, a list | `sellers` | | `analytics` | one | `module`, a string | `tinybird` | | `cmp` | one | `module`, a string | `didomi`, `sourcepoint`, `osano` | | `tag` | several | `modules`, a list | `google-tag-manager` | diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 3397adc3f7..16e6db7b15 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1083,9 +1083,9 @@ it passes through untouched. ### `[ads-txt]` -| Field | Type | Required | Description | -| --------- | ----------------------- | -------- | ------------------------------------------------------------------------------------------- | -| `modules` | String or Array[String] | Yes | What adds lines to the file, in order. `sellers` writes the lines `[ads-txt.sellers]` gives | +| Field | Type | Required | Description | +| --------- | ------------- | -------- | --------------------------------------------------------------------------------- | +| `modules` | Array[String] | Yes | What adds lines to the file. `sellers` writes the lines `[ads-txt.sellers]` gives | ### `[ads-txt.sellers]` @@ -1093,13 +1093,9 @@ it passes through untouched. | ------- | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `lines` | Array[String] | Yes | The lines written at the end of the file, each a data record (domain, account id, `DIRECT` or `RESELLER`, and an optional certification authority id) or a variable | -**How the file is made.** The module is a middleware for `text/plain`, and -runs where a `[[fetch]]` entry for that media type on the file's path names -it. It passes the origin's file through and writes the lines after its last -one, on a line of their own, so a seller the deployment introduces is -declared beside the publisher's own. A publisher whose origin has no file -gets the origin's answer, because the module adds to the publisher's file -and never invents one. +**How the file is made.** Where a `[[fetch]]` entry for `text/plain` names +the module, it writes the lines after the origin's last one. The entry's +`path` is a prefix, so `/ads.txt` also covers `/ads.txt.bak`. **Refused when the settings load**: a line that is not a data record or a variable, naming which part is wrong, a comment, an empty list, and a