diff --git a/CHANGELOG.md b/CHANGELOG.md index c008a174a5..892bcf4e4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -130,6 +130,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `ts config migrate --adapter ` rewrites a settings document written for an earlier release, such as one written before the breaking changes above, into the shape this release reads, and prints the result. The host's location lookup the earlier release ran becomes `[geo] module = "platform"` on Fastly and Cloudflare and no geo module on Axum and Spin, which have no lookup. On stderr it lists each change it made, each setting it left for the operator because it needs a decision or a value the document does not hold, such as `[geo] assume_single_jurisdiction` for an Edge Cookie module with no geo module, and each change in behavior no setting expresses, then checks the result with the settings validation `ts config validate` runs. It keeps the comments and the order of the tables, comments out a module that was not enabled under its new name, never selects `headers.geo`, which writes the reader's location, never prints a value from the document, rewrites the file in place with `--write`, and with `--check` changes nothing and exits 1 when the document needs migrating. Migrating a migrated document changes nothing. - A 51Degrees client hints middleware, `51degrees.client_hints`, from `crates/middleware/51degrees`, run by `[shared.51degrees]` and placed by a `[[serve]]` entry. On the Fastly adapter, the only one that runs request filters, it asks the 51Degrees service which client hints the reader's browser can send for each page navigation an entry covers, sending the `User-Agent` and `Sec-CH-UA` hints alone, and leaves the hint names in the request's state for the other 51Degrees modules, which keeps a page it names hints for out of the stored templates and sends it `private, no-store`. On each page an entry covers it writes `Accept-CH` and `Critical-CH` on the response unless the browser already sends every one of those hints, `Permissions-Policy` and the `Delegate-CH` tag that delegate those hints to the service's origin, each keeping every line the response already carries. `critical` in `[middleware.51degrees.client_hints]` turns the retry of the first navigation off. The service's answers load through the cache `51degrees-answers`, selected and sized in `[cache.51degrees-answers]`, and are held only where the build links a loading cache module, so a stock build asks the service for each page navigation an entry covers. What the 51Degrees modules share, being the table, the one call to the service, the evidence it sends and the cache its answer is held in, is the new `crates/shared/51degrees`. - `[middleware.]`, a middleware's own settings table, written with the middleware's full name and read through `Settings::middleware_config`. A table holds values and arrays, a table within one is the longer name's table and never a setting, and a table at a name no running module supplies is refused at startup with the names that could be written. - A serve middleware may set headers on a response the publisher path sends for a request path its entry covers, through `Middleware::response_headers`, which is given the response's status and headers and runs whatever the response's media type and whether or not it has a body, so a redirect with no body reaches a middleware that rewrites its `Location`. A header it sets replaces one of the same name. The method is not called for a refusal made before the origin is asked, such as the `405` for a method a signed origin does not take, for a page served from the template cache or the refusal of such a page, or for an answer made from an error. diff --git a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs index 90c403341d..1503b09f0d 100644 --- a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs +++ b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs @@ -7,7 +7,7 @@ mod gpt_slots; mod page_patterns; mod slot_toml; mod unit_template; -mod validate; +pub(crate) mod validate; use std::collections::BTreeSet; use std::fs; diff --git a/crates/trusted-server-cli/src/commands/audit/generate/validate.rs b/crates/trusted-server-cli/src/commands/audit/generate/validate.rs index 9ea724f08f..731fa231cf 100644 --- a/crates/trusted-server-cli/src/commands/audit/generate/validate.rs +++ b/crates/trusted-server-cli/src/commands/audit/generate/validate.rs @@ -52,7 +52,7 @@ pub(super) fn check_candidate(candidate: &str, baseline: &str) -> CliResult CliResult<()> { +pub(crate) fn validate_source_config(source: &str) -> CliResult<()> { crate::app_config::register_linked_modules(); let config: TrustedServerAppConfig = toml::from_str(source).map_err(|error| { error.span().map_or_else( diff --git a/crates/trusted-server-cli/src/commands/config/migrate/layout.rs b/crates/trusted-server-cli/src/commands/config/migrate/layout.rs new file mode 100644 index 0000000000..ae4a0830c8 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/config/migrate/layout.rs @@ -0,0 +1,516 @@ +//! Places tables and comments in a `toml_edit` document, so that a rewrite +//! keeps the order and the comments the operator wrote. +//! +//! `toml_edit` writes each table at its position, and holds the comment +//! lines written between two tables with the second, so a table put between +//! two others is put before the comment lines that follow the first one's +//! last setting, as they belong with it. A comment paragraph is a run of +//! comment lines ended by a blank line. + +use std::collections::BTreeSet; + +use toml_edit::{DocumentMut, Item, RawString, Table}; + +/// The gap left between the positions of the tables the document was read +/// with, so new tables fit between two of them. +const STRIDE: isize = 1 << 12; + +/// The start of the root key that holds a table while it waits to be +/// written as a comment block. +const HELD_PREFIX: &str = "__ts_config_migrate_held_"; + +/// One step of the path to a table, a key or the index of an entry in an +/// array of tables. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(super) enum Step { + Key(String), + Index(usize), +} + +/// The path from the root to a table. +pub(super) type TablePath = Vec; + +/// The path of the table at `keys`, below the root. +pub(super) fn path(keys: &[&str]) -> TablePath { + keys.iter() + .map(|key| Step::Key((*key).to_owned())) + .collect() +} + +/// A table taken out of the document to be written as a comment block at +/// its place, under the name the stack reads it by. +struct Held { + key: String, + header: Option>, +} + +/// The document being rewritten, with the tables that wait to be written as +/// comments. +pub(super) struct Layout { + pub(super) doc: DocumentMut, + held: Vec, +} + +impl Layout { + /// The document read from `doc`, with room left between its tables. + pub(super) fn new(mut doc: DocumentMut) -> Self { + spread_positions(doc.as_table_mut()); + Self { + doc, + held: Vec::new(), + } + } + + /// The table at `path`. + pub(super) fn table_mut(&mut self, path: &[Step]) -> Option<&mut Table> { + table_at_mut(self.doc.as_table_mut(), path) + } + + /// The position of the table at `path`. + pub(super) fn position_of(&self, path: &[Step]) -> Option { + table_at(self.doc.as_table(), path).and_then(Table::position) + } + + /// The last position the document writes a table at. + pub(super) fn last_position(&self) -> isize { + render_order(self.doc.as_table()) + .last() + .map_or(0, |(position, _)| *position) + } + + /// The largest position of the table at `path` and every table below it. + pub(super) fn subtree_end(&self, path: &[Step]) -> Option { + let anchor = self.position_of(path)?; + let end = render_order(self.doc.as_table()) + .into_iter() + .filter(|(_, table)| table.starts_with(path)) + .map(|(position, _)| position) + .max(); + Some(end.map_or(anchor, |end| end.max(anchor))) + } + + /// Writes the tables at `new`, in order, directly after the table at + /// position `anchor` and before the next one, and moves the comment lines + /// that follow the anchor's last setting ahead of the first of them. + /// + /// Each path in `new` names a table already in the document, a parent + /// ahead of its children. + pub(super) fn place_after(&mut self, anchor: isize, new: &[TablePath]) { + let order = render_order(self.doc.as_table()); + let used: BTreeSet = order.iter().map(|(position, _)| *position).collect(); + let next = order + .iter() + .filter(|(position, table)| *position > anchor && !new.contains(table)) + .find(|(_, table)| !table.is_empty() && self.is_visible(table)) + .map(|(_, table)| table.clone()); + + let mut position = anchor; + for table in new { + position += 1; + while used.contains(&position) { + position += 1; + } + if let Some(table) = self.table_mut(table) { + table.set_position(position); + } + } + + let Some(first) = new.first() else { + return; + }; + let leading = match &next { + Some(next) => self.table_mut(next).map_or_else(String::new, |table| { + let prefix = prefix_of(table); + let (leading, rest) = split_leading_paragraph(&prefix); + let leading = leading.to_owned(); + let rest = rest.to_owned(); + if !leading.is_empty() { + set_prefix(table, &rest); + } + leading + }), + None => { + let trailing = self.doc.trailing().as_str().unwrap_or("").to_owned(); + let (leading, rest) = split_leading_paragraph(&trailing); + let leading = leading.to_owned(); + if !leading.is_empty() { + self.doc.set_trailing(rest.to_owned()); + } + leading + } + }; + if leading.is_empty() { + return; + } + if let Some(table) = self.table_mut(first) { + let own = prefix_or_blank_line(table); + set_prefix(table, &format!("{leading}{own}")); + } + } + + /// Takes the table at `path` out of the document, with its comments and + /// its position, to be put back elsewhere. + pub(super) fn take(&mut self, path: &[Step]) -> Option { + take_table(self.doc.as_table_mut(), path) + } + + /// Turns an inline table at `path` into a table with a header of its + /// own, so the rules move every table the same way. + pub(super) fn make_table(&mut self, path: &[Step]) { + let Some((Step::Key(key), parent)) = path.split_last() else { + return; + }; + let Some(item) = + table_at_mut(self.doc.as_table_mut(), parent).and_then(|parent| parent.get_mut(key)) + else { + return; + }; + if item.is_inline_table() { + let inline = std::mem::take(item); + *item = inline.into_table().map_or_else(|kept| kept, Item::Table); + } + } + + /// Removes the table at `path`, with the tables below it, and gives the + /// comment paragraphs written above its own to the table written next, + /// so the table takes only the comments that describe it. + pub(super) fn remove(&mut self, path: &[Step]) -> Option
{ + let position = self.position_of(path); + let table = take_table(self.doc.as_table_mut(), path)?; + let prefix = prefix_of(&table); + let (head, _own) = split_trailing_paragraph(&prefix); + if head.lines().any(|line| !line.trim().is_empty()) { + self.prepend_to_next(position.unwrap_or(0), head); + } + Some(table) + } + + /// Takes the table at `path` out of the document to be written in its + /// place as a comment block, headed `[
]` when `header` is set + /// and the comments above the table alone otherwise. + /// + /// The table waits at the root, at its position, until + /// [`Self::finish`], so a table placed after it still finds the comment + /// lines above it. + pub(super) fn hold(&mut self, path: &[Step], header: Option<&[&str]>) { + let Some(table) = take_table(self.doc.as_table_mut(), path) else { + return; + }; + let key = format!("{HELD_PREFIX}{}", self.held.len()); + self.doc.as_table_mut().insert(&key, Item::Table(table)); + self.held.push(Held { + key, + header: header.map(|parts| parts.iter().map(|part| (*part).to_owned()).collect()), + }); + } + + /// The document as text, with each held table written as comments in + /// its place. + pub(super) fn finish(mut self) -> String { + let mut blocks = Vec::new(); + for held in &self.held { + let Some(Item::Table(table)) = self.doc.as_table_mut().remove(&held.key) else { + continue; + }; + let mut marker = Table::new(); + if let Some(position) = table.position() { + marker.set_position(position); + } + marker.decor_mut().set_prefix(""); + marker.decor_mut().set_suffix(""); + self.doc + .as_table_mut() + .insert(&held.key, Item::Table(marker)); + blocks.push(( + format!("[{}]\n", held.key), + commented_block(table, held.header.as_deref()), + )); + } + let mut text = self.doc.to_string(); + for (marker, block) in blocks { + text = text.replacen(&marker, &block, 1); + } + text + } + + /// Whether the table at `path` writes a header, which an implicit table + /// with no settings of its own does not. + fn is_visible(&self, path: &[Step]) -> bool { + table_at(self.doc.as_table(), path) + .is_some_and(|table| !(table.is_implicit() && table.get_values().is_empty())) + } + + /// Puts `text` ahead of the prefix of the first table written after + /// `position`, or ahead of the document's trailing text when none is. + fn prepend_to_next(&mut self, position: isize, text: &str) { + let next = render_order(self.doc.as_table()) + .into_iter() + .filter(|(at, table)| *at >= position && !table.is_empty()) + .find(|(_, table)| self.is_visible(table)) + .map(|(_, table)| table); + match next.and_then(|next| self.table_mut(&next)) { + Some(table) => { + let prefix = prefix_of(table); + set_prefix(table, &format!("{text}{prefix}")); + } + None => { + let trailing = self.doc.trailing().as_str().unwrap_or("").to_owned(); + self.doc.set_trailing(format!("{text}{trailing}")); + } + } + } +} + +/// The text written ahead of a table's header. +pub(super) fn prefix_of(table: &Table) -> String { + table + .decor() + .prefix() + .and_then(RawString::as_str) + .unwrap_or("") + .to_owned() +} + +/// Sets the text written ahead of a table's header. +pub(super) fn set_prefix(table: &mut Table, text: &str) { + table.decor_mut().set_prefix(text.to_owned()); +} + +/// The text ahead of a table's header, or a blank line where none was +/// written, which is what `toml_edit` writes ahead of a table it made. +fn prefix_or_blank_line(table: &Table) -> String { + table + .decor() + .prefix() + .and_then(RawString::as_str) + .map_or_else(|| "\n".to_owned(), str::to_owned) +} + +/// Splits off the comment lines at the start of `text` that a blank line +/// separates from the rest. Those lines follow the setting written above +/// them, so they belong with it. Text that starts with a blank line, or that +/// has none, has no such lines. +pub(super) fn split_leading_paragraph(text: &str) -> (&str, &str) { + let mut offset = 0; + for line in text.split_inclusive('\n') { + if line.trim().is_empty() { + return text.split_at(offset); + } + offset += line.len(); + } + ("", text) +} + +/// Splits off the comment lines at the end of `text`, directly above what +/// follows it, from the text ahead of them. +pub(super) fn split_trailing_paragraph(text: &str) -> (&str, &str) { + let mut start = 0; + let mut offset = 0; + for line in text.split_inclusive('\n') { + offset += line.len(); + if line.trim().is_empty() { + start = offset; + } + } + text.split_at(start) +} + +/// Multiplies the position of every table by [`STRIDE`]. +fn spread_positions(table: &mut Table) { + if let Some(position) = table.position() { + table.set_position(position.saturating_mul(STRIDE)); + } + for (_, item) in table.iter_mut() { + match item { + Item::Table(child) => spread_positions(child), + Item::ArrayOfTables(array) => { + for child in array.iter_mut() { + spread_positions(child); + } + } + Item::None | Item::Value(_) => {} + } + } +} + +/// Every table below `root`, with the position `toml_edit` sorts it by, in +/// the order it writes them. A table with no position takes the position of +/// the table visited before it, as `toml_edit` does. +fn render_order(root: &Table) -> Vec<(isize, TablePath)> { + fn visit( + table: &Table, + path: &mut TablePath, + last: &mut isize, + out: &mut Vec<(isize, TablePath)>, + ) { + if !table.is_dotted() { + if let Some(position) = table.position() { + *last = position; + } + out.push((*last, path.clone())); + } + for (key, item) in table.iter() { + match item { + Item::Table(child) => { + path.push(Step::Key(key.to_owned())); + visit(child, path, last, out); + path.pop(); + } + Item::ArrayOfTables(array) => { + for (index, child) in array.iter().enumerate() { + path.push(Step::Key(key.to_owned())); + path.push(Step::Index(index)); + visit(child, path, last, out); + path.pop(); + path.pop(); + } + } + Item::None | Item::Value(_) => {} + } + } + } + let mut out = Vec::new(); + visit(root, &mut Vec::new(), &mut 0, &mut out); + out.sort_by_key(|(position, _)| *position); + out +} + +fn table_at<'a>(root: &'a Table, path: &[Step]) -> Option<&'a Table> { + let mut table = root; + let mut steps = path.iter(); + while let Some(step) = steps.next() { + let Step::Key(key) = step else { + return None; + }; + table = match table.get(key)? { + Item::Table(child) => child, + Item::ArrayOfTables(array) => { + let Some(Step::Index(index)) = steps.next() else { + return None; + }; + array.get(*index)? + } + Item::None | Item::Value(_) => return None, + }; + } + Some(table) +} + +fn table_at_mut<'a>(root: &'a mut Table, path: &[Step]) -> Option<&'a mut Table> { + let mut table = root; + let mut steps = path.iter(); + while let Some(step) = steps.next() { + let Step::Key(key) = step else { + return None; + }; + table = match table.get_mut(key)? { + Item::Table(child) => child, + Item::ArrayOfTables(array) => { + let Some(Step::Index(index)) = steps.next() else { + return None; + }; + array.get_mut(*index)? + } + Item::None | Item::Value(_) => return None, + }; + } + Some(table) +} + +/// Takes the table at `path` out of its parent, and an entry out of its +/// array of tables, dropping the array when it is left empty. +fn take_table(root: &mut Table, path: &[Step]) -> Option
{ + match path.split_last()? { + (Step::Key(key), parent) => { + let parent = table_at_mut(root, parent)?; + if !parent.get(key)?.is_table() { + return None; + } + parent.remove(key).and_then(|item| item.into_table().ok()) + } + (Step::Index(index), parent) => { + let (Step::Key(key), grandparent) = parent.split_last()? else { + return None; + }; + let grandparent = table_at_mut(root, grandparent)?; + let array = grandparent.get_mut(key)?.as_array_of_tables_mut()?; + if *index >= array.len() { + return None; + } + let table = array.remove(*index); + if array.is_empty() { + grandparent.remove(key); + } + Some(table) + } + } +} + +/// `table` written as comments: the comments above it, and, under +/// `header`, its settings and the tables below it with each line commented +/// out. Lines that are already comments, and blank lines, are kept as they +/// are. +fn commented_block(table: Table, header: Option<&[String]>) -> String { + let prefix = prefix_of(&table); + let Some(header) = header else { + return prefix; + }; + let mut table = table; + set_prefix(&mut table, ""); + table.set_implicit(false); + let mut mini = DocumentMut::new(); + let mut parent = mini.as_table_mut(); + let Some((last, parents)) = header.split_last() else { + return prefix; + }; + for part in parents { + let entry = parent.entry(part).or_insert_with(|| { + let mut implicit = Table::new(); + implicit.set_implicit(true); + Item::Table(implicit) + }); + let Some(next) = entry.as_table_mut() else { + return prefix; + }; + parent = next; + } + parent.insert(last, Item::Table(table)); + let body = mini.to_string(); + let mut block = prefix; + for line in body.split_inclusive('\n') { + let trimmed = line.trim_start(); + if trimmed.trim().is_empty() || trimmed.starts_with('#') { + block.push_str(line); + } else { + block.push_str("# "); + block.push_str(line); + } + } + block +} + +/// Whether `text` ends its first line with a carriage return and a line +/// feed. +pub(super) fn uses_crlf(text: &str) -> bool { + text.find('\n').is_some_and(|index| { + index + .checked_sub(1) + .and_then(|before| text.as_bytes().get(before)) + == Some(&b'\r') + }) +} + +/// `text` with every line feed not already after a carriage return given +/// one, for a document read with Windows line endings. +pub(super) fn to_crlf(text: &str) -> String { + let mut out = String::with_capacity(text.len() + text.len() / 16); + let mut previous = '\0'; + for character in text.chars() { + if character == '\n' && previous != '\r' { + out.push('\r'); + } + out.push(character); + previous = character; + } + out +} diff --git a/crates/trusted-server-cli/src/commands/config/migrate/mod.rs b/crates/trusted-server-cli/src/commands/config/migrate/mod.rs new file mode 100644 index 0000000000..de03b9a5a2 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/config/migrate/mod.rs @@ -0,0 +1,181 @@ +//! `ts config migrate`, which rewrites a settings document written for an +//! earlier release into the shape this release reads. +//! +//! The rewrite works on the file alone, for the adapter the deployment runs +//! on, keeps the comments and the order the operator wrote, and leaves in +//! place, with directions, every setting that needs a decision or a value +//! the document does not hold. See +//! `docs/superpowers/specs/2026-10-17-config-migrate-design.md`. + +mod layout; +mod rules; +#[cfg(test)] +mod tests; + +use std::fs; +use std::io::Write; +use std::path::PathBuf; + +use crate::RunOutcome; +use crate::commands::audit::generate::validate::validate_source_config; + +use rules::{Migration, migrate}; + +/// Rewrite a settings document written for an earlier release into the shape +/// this release reads. +#[derive(Debug, clap::Args)] +pub struct ConfigMigrateArgs { + /// The settings document to migrate, such as `trusted-server.toml`. + pub file: PathBuf, + /// The adapter the deployment runs on, which decides what main's location + /// lookup becomes. + #[arg(long, value_enum)] + pub adapter: Adapter, + /// Rewrite the file in place, rather than print the migrated document. + #[arg(long, conflicts_with = "check")] + pub write: bool, + /// Change nothing, and exit 1 when the document needs migrating. + #[arg(long)] + pub check: bool, +} + +/// The adapter a deployment runs on. +/// +/// Main's location lookup and location headers differed by adapter, and +/// the Axum and Spin adapters refuse `[geo] module = "platform"` at startup +/// because neither host has a location lookup. +#[derive(Debug, Clone, Copy, PartialEq, Eq, clap::ValueEnum)] +pub enum Adapter { + /// Fastly Compute. + Fastly, + /// Cloudflare Workers. + Cloudflare, + /// The Axum dev server. + Axum, + /// Spin. + Spin, +} + +impl Adapter { + /// Whether the host has a location lookup of its own, which `[geo] + /// module = "platform"` selects. + pub(crate) const fn has_location_lookup(self) -> bool { + matches!(self, Self::Fastly | Self::Cloudflare) + } + + /// The adapter as a line on stderr names it. + pub(crate) const fn name(self) -> &'static str { + match self { + Self::Fastly => "Fastly", + Self::Cloudflare => "Cloudflare", + Self::Axum => "the Axum dev server", + Self::Spin => "Spin", + } + } +} + +/// Migrates the document `args` names, printing the result on stdout and +/// what was done and what is left on stderr. +/// +/// # Errors +/// +/// When the file cannot be read or written, or is not TOML. +pub fn run_config_migrate(args: &ConfigMigrateArgs) -> Result { + let stdout = std::io::stdout(); + let stderr = std::io::stderr(); + run_with_writers(args, &mut stdout.lock(), &mut stderr.lock()) +} + +/// [`run_config_migrate`], writing to `out` and `report`. +/// +/// The outcome is a success when nothing is left to do and the migrated +/// document passes validation, and with `--check` when the document needs +/// no migrating. +/// +/// # Errors +/// +/// When the file cannot be read or written, or is not TOML. +pub(crate) fn run_with_writers( + args: &ConfigMigrateArgs, + out: &mut dyn Write, + report: &mut dyn Write, +) -> Result { + let input = fs::read_to_string(&args.file) + .map_err(|error| format!("failed to read {}: {error}", args.file.display()))?; + let migration = migrate(&input, args.adapter) + .map_err(|error| format!("failed to read {}: {error}", args.file.display()))?; + write_report(report, &migration)?; + + if args.check { + let needs = needs_migrating(&migration); + line( + report, + if needs { + "the document needs migrating" + } else { + "the document needs no migrating" + }, + )?; + return Ok(outcome(!needs)); + } + + if args.write { + if migration.text != input { + fs::write(&args.file, &migration.text) + .map_err(|error| format!("failed to write {}: {error}", args.file.display()))?; + } + } else { + out.write_all(migration.text.as_bytes()) + .map_err(|error| format!("failed to write the migrated document: {error}"))?; + } + + let valid = validate_source_config(&migration.text); + match &valid { + Ok(()) => line(report, "the migrated document passes validation")?, + Err(reason) => line( + report, + &format!( + "the migrated document does not pass validation ({reason}). `ts config \ + validate` prints the whole refusal" + ), + )?, + } + Ok(outcome(valid.is_ok() && migration.report.to_do.is_empty())) +} + +/// Whether the document needs migrating, being a rewrite to make or an item +/// left for the operator. +fn needs_migrating(migration: &Migration) -> bool { + !migration.report.changed.is_empty() || !migration.report.to_do.is_empty() +} + +const fn outcome(success: bool) -> RunOutcome { + if success { + RunOutcome::Success + } else { + RunOutcome::AssertionFailed + } +} + +/// One line on stderr for each change, each item left to do and each +/// change in behavior to check. +fn write_report(report: &mut dyn Write, migration: &Migration) -> Result<(), String> { + let lines = &migration.report; + if lines.changed.is_empty() && lines.to_do.is_empty() { + line(report, "nothing to migrate")?; + } + for (label, items) in [ + ("changed", &lines.changed), + ("to do", &lines.to_do), + ("check", &lines.check), + ] { + for item in items { + line(report, &format!("{label}: {item}"))?; + } + } + Ok(()) +} + +fn line(report: &mut dyn Write, text: &str) -> Result<(), String> { + writeln!(report, "{text}").map_err(|error| format!("failed to write command output: {error}")) +} diff --git a/crates/trusted-server-cli/src/commands/config/migrate/rules.rs b/crates/trusted-server-cli/src/commands/config/migrate/rules.rs new file mode 100644 index 0000000000..855627c36a --- /dev/null +++ b/crates/trusted-server-cli/src/commands/config/migrate/rules.rs @@ -0,0 +1,1993 @@ +//! The rewrites `ts config migrate` makes, one for each change the stack +//! made to the settings document. +//! +//! Every message names a setting and never holds a value from the +//! document, because a document can hold secrets. + +use toml_edit::{Array, DocumentMut, Item, Key, RawString, Table, Value, value}; +use trusted_server_core::integrations::supplied_middleware; +use trusted_server_core::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; +use trusted_server_core::module_name::short_form; +use trusted_server_core::prefix::PublicPrefix; + +use super::Adapter; +use super::layout::{Layout, Step, TablePath, path, prefix_of, set_prefix, to_crlf, uses_crlf}; + +/// The tables main read at the top level of a document, and `[permissions]`, +/// which main did not read and which an operator may add before migrating to +/// name the policy file this release requires. Main refused any other +/// table, so a document with `[publisher]` and only these tables was written +/// for main. +const MAIN_TABLES: &[&str] = &[ + "publisher", + "permissions", + "tester_cookie", + "trusted_client_ip", + "ec", + "integrations", + "handlers", + "response_headers", + "request_signing", + "rewrite", + "auction", + "consent", + "cache", + "proxy", + "creative_opportunities", + "image_optimizer", + "tinybird", + "debug", +]; + +/// An `[integrations.]` table main read, the value main gave `enabled` +/// when the table left it out, and whether the module that replaces it has +/// settings of its own. +struct MainIntegration { + id: &'static str, + enabled_by_default: bool, + has_settings: bool, +} + +/// Main's integrations that each became one module. `aps` and +/// `adserver_mock` became parts of the auction and are rewritten apart. +const MAIN_INTEGRATIONS: &[MainIntegration] = &[ + MainIntegration::new("prebid", true, true), + MainIntegration::new("testlight", false, true), + MainIntegration::new("nextjs", false, true), + MainIntegration::new("permutive", true, true), + MainIntegration::new("lockr", true, true), + MainIntegration::new("didomi", true, true), + MainIntegration::new("sourcepoint", false, true), + MainIntegration::new("osano", false, false), + MainIntegration::new("google_tag_manager", false, true), + MainIntegration::new("datadome", false, true), + MainIntegration::new("gpt", true, true), + MainIntegration::new("gpt_diagnostics", false, false), + MainIntegration::new("js_asset_proxy", false, true), +]; + +impl MainIntegration { + const fn new(id: &'static str, enabled_by_default: bool, has_settings: bool) -> Self { + Self { + id, + enabled_by_default, + has_settings, + } + } +} + +/// The item left to do in a document that names no permission policy file. +pub(crate) const POLICY_TO_DO: &str = "`[permissions] policy` names no permission policy \ + file, and a domain whose settings name none does not start. Name the publisher's file \ + there by its stem, as `policy = \"standard\"` names `standard.yaml`, and supply the \ + file with `ts config push --permissions `"; + +/// The sections that select several modules, written with `modules` even +/// when one is selected. +const LIST_SECTIONS: &[&str] = &["tag", "ad-tag", "auction", "proxy", "ads-txt", "headers"]; + +/// Core's script proxy, which no linked builder describes. +const JS_ASSET_PROXY: &str = "js_asset_proxy"; + +/// The file paths of the middleware a `[[fetch]]` entry places on one file. +const FILE_MIDDLEWARE: &[(&str, &str)] = + &[("ads-txt.iabtl", "/ads.txt"), ("robots_txt", "/robots.txt")]; + +/// The comment written above the first page entry the command adds. +const ENTRIES_COMMENT: &str = concat!( + "\n# A module changes a page only where one of these entries names its\n", + "# middleware, and a page takes the first entry that covers it.\n", +); + +/// What a migration did and what it left. +#[derive(Debug, Default)] +pub(crate) struct Report { + /// A line for each rewrite made. + pub(crate) changed: Vec, + /// A line for each item the document cannot load without, or setting the + /// stack ignores whose value the operator carries elsewhere, which needs + /// a decision or a value the document does not hold. + pub(crate) to_do: Vec, + /// A line for each change in behavior no setting expresses. + pub(crate) check: Vec, +} + +/// A migrated document and its report. +#[derive(Debug)] +pub(crate) struct Migration { + /// The document in the shape the stack reads, the input itself when + /// nothing changed. + pub(crate) text: String, + /// What the migration did and what it left. + pub(crate) report: Report, +} + +/// Rewrites `input`, a settings document, into the shape the stack reads on +/// `adapter`. +/// +/// # Errors +/// +/// When `input` is not TOML, naming the line reading stopped at and nothing +/// from the line. +pub(crate) fn migrate(input: &str, adapter: Adapter) -> Result { + let doc = input + .parse::() + .map_err(|error| match error.span() { + Some(span) => format!( + "the document is not TOML, from line {}", + input + .get(..span.start) + .map_or(1, |head| head.matches('\n').count() + 1) + ), + None => "the document is not TOML".to_owned(), + })?; + let mut migrator = Migrator::new(doc, adapter); + migrator.run(); + let Migrator { layout, report, .. } = migrator; + let text = if report.changed.is_empty() { + input.to_owned() + } else if uses_crlf(input) { + to_crlf(&layout.finish()) + } else { + layout.finish() + }; + Ok(Migration { text, report }) +} + +/// A module a section can select, as the builders the tool links describe +/// it. +struct KnownModule { + id: &'static str, + section: &'static str, + written: &'static str, + full: &'static str, +} + +/// A `[[fetch]]` or `[[serve]]` entry the migration adds. +struct PageEntry { + phase: &'static str, + media_type: &'static str, + path: Option<&'static str>, + middleware: Vec<&'static str>, +} + +/// The document being migrated, the adapter it is for, what the migration +/// has reported, and what the tool's builders say of the modules. +struct Migrator { + layout: Layout, + adapter: Adapter, + report: Report, + modules: Vec, + shared: Vec<(&'static str, &'static str)>, + for_main: bool, +} + +impl Migrator { + fn new(doc: DocumentMut, adapter: Adapter) -> Self { + let builders = crate::app_config::linked_builders(); + let mut modules = vec![KnownModule { + id: JS_ASSET_PROXY, + section: "proxy", + written: JS_ASSET_PROXY, + full: JS_ASSET_PROXY, + }]; + let mut shared = Vec::new(); + for builder in &builders { + if let (Some(section), Some(full)) = (builder.section(), builder.module_name()) { + modules.push(KnownModule { + id: builder.id(), + section, + written: short_form(section, full), + full, + }); + } + if let Some(vendor) = builder.shared_settings() { + shared.push((builder.id(), vendor)); + } + } + let for_main = written_for_main(doc.as_table()); + Self { + layout: Layout::new(doc), + adapter, + report: Report::default(), + modules, + shared, + for_main, + } + } + + fn run(&mut self) { + self.publisher_source(); + self.asset_routes(); + self.handlers(); + self.request_signing(); + self.edge_cookie(); + self.permission_signal(); + self.device(); + self.demand(); + self.integrations(); + self.mediator_left(); + self.ad_server_names(); + self.ads_txt_names(); + self.tinybird(); + self.debug_options(); + self.consent_lists(); + self.permission_policy(); + self.template_cookies(); + if self.for_main { + self.main_defaults(); + } + self.location_at_startup(); + self.page_entries(); + if self.for_main { + self.main_checks(); + } + } + + fn root(&self) -> &Table { + self.layout.doc.as_table() + } + + fn root_mut(&mut self) -> &mut Table { + self.layout.doc.as_table_mut() + } + + fn push_changed(&mut self, line: impl Into) { + self.report.changed.push(line.into()); + } + + fn push_to_do(&mut self, line: impl Into) { + self.report.to_do.push(line.into()); + } + + fn push_check(&mut self, line: impl Into) { + self.report.check.push(line.into()); + } + + /// `[publisher]`'s origin settings into `[source]`, which selects core's + /// `origin` source and holds them in `[source.origin]`. + fn publisher_source(&mut self) { + const MOVES: &[(&str, &str)] = &[ + ("origin_url", "url"), + ("origin_host_header_override", "host_header"), + ("directory_index", "directory_index"), + ("origin_auth", "auth"), + ]; + let Some(publisher) = self.root().get("publisher").and_then(Item::as_table) else { + return; + }; + let moving = keys_in_order(publisher, MOVES.iter().map(|(old, _)| *old)); + if moving.is_empty() { + return; + } + if self.root().contains_key("source") { + self.push_to_do( + "`[publisher]` holds origin settings and `[source]` is written too. Keep one of \ + the two, with the origin's settings in `[source.origin]`", + ); + return; + } + let anchor = self.layout.position_of(&path(&["publisher"])); + let Some(publisher) = self.layout.table_mut(&path(&["publisher"])) else { + return; + }; + let mut origin = Table::new(); + let mut lines = Vec::new(); + for old in &moving { + let Some(new) = MOVES + .iter() + .find(|(from, _)| from == old) + .map(|(_, to)| *to) + else { + continue; + }; + if let Some((key, item)) = publisher.remove_entry(old) { + origin.insert_formatted(&renamed(&key, new), item); + lines.push(if lines.is_empty() { + format!( + "moved `[publisher] {old}` to `[source.origin] {new}`, with `[source] \ + module = \"origin\"`" + ) + } else { + format!("moved `[publisher] {old}` to `[source.origin] {new}`") + }); + } + } + for line in lines { + self.push_changed(line); + } + self.rename_signer(&mut origin, "`[source.origin.auth] type`"); + if origin + .get("url") + .and_then(Item::as_str) + .and_then(|url| url::Url::parse(url).ok()) + .is_some_and(|url| !url.path().is_empty() && url.path() != "/") + { + self.push_check( + "`[source.origin] url` has a path, which main dropped and this release keeps, so \ + each request now goes to the folder the path names. Check the origin serves the \ + site there, or remove the path", + ); + } + if origin.contains_key("host_header") { + self.push_check( + "`[source.origin] host_header` is set, so the addresses a page gives under that \ + host are now rewritten to the request's host, where main rewrote those under the \ + host of `[source.origin] url`. Check which host the origin's pages name", + ); + } + let auth_is_table = origin.get("auth").is_some_and(Item::is_table); + let mut source = Table::new(); + source.insert("module", value("origin")); + source.insert("origin", Item::Table(origin)); + self.root_mut().insert("source", Item::Table(source)); + let mut new = vec![path(&["source"]), path(&["source", "origin"])]; + if auth_is_table { + new.push(path(&["source", "origin", "auth"])); + } + let anchor = anchor.unwrap_or_else(|| self.layout.last_position()); + self.layout.place_after(anchor, &new); + } + + /// Each `[[proxy.asset_routes]]` entry's `origin_url` and `auth` into its + /// `source` table. + fn asset_routes(&mut self) { + const MOVES: &[(&str, &str)] = &[("origin_url", "url"), ("auth", "auth")]; + let Some(routes) = self + .root() + .get("proxy") + .and_then(|proxy| proxy.get("asset_routes")) + .and_then(Item::as_array_of_tables) + else { + return; + }; + let count = routes.len(); + for index in 0..count { + let number = index + 1; + let route_path = vec![ + Step::Key("proxy".to_owned()), + Step::Key("asset_routes".to_owned()), + Step::Index(index), + ]; + let anchor = self.layout.position_of(&route_path); + let Some(route) = self.layout.table_mut(&route_path) else { + continue; + }; + let moving = keys_in_order(route, MOVES.iter().map(|(old, _)| *old)); + if moving.is_empty() { + continue; + } + if route.contains_key("source") { + self.push_to_do(format!( + "`[[proxy.asset_routes]]` entry {number} holds origin settings and a \ + `source` table too. Keep one of the two" + )); + continue; + } + let mut origin = Table::new(); + for old in &moving { + if let Some((key, item)) = route.remove_entry(old) { + let new = MOVES + .iter() + .find(|(from, _)| from == old) + .map_or(*old, |(_, to)| *to); + origin.insert_formatted(&renamed(&key, new), item); + } + } + for old in &moving { + let new = MOVES + .iter() + .find(|(from, _)| from == old) + .map_or(*old, |(_, to)| *to); + self.push_changed(format!( + "moved `{old}` of `[[proxy.asset_routes]]` entry {number} to \ + `[proxy.asset_routes.source.origin] {new}`, with `module = \"origin\"`" + )); + } + self.rename_signer( + &mut origin, + &format!("The `auth` type of `[[proxy.asset_routes]]` entry {number}"), + ); + let auth_is_table = origin.get("auth").is_some_and(Item::is_table); + let mut source = Table::new(); + source.insert("module", value("origin")); + source.insert("origin", Item::Table(origin)); + let Some(route) = self.layout.table_mut(&route_path) else { + continue; + }; + route.insert("source", Item::Table(source)); + let mut source_path = route_path.clone(); + source_path.push(Step::Key("source".to_owned())); + let mut origin_path = source_path.clone(); + origin_path.push(Step::Key("origin".to_owned())); + let mut new = vec![source_path, origin_path.clone()]; + if auth_is_table { + let mut auth_path = origin_path; + auth_path.push(Step::Key("auth".to_owned())); + new.push(auth_path); + } + let anchor = anchor.unwrap_or_else(|| self.layout.last_position()); + self.layout.place_after(anchor, &new); + } + } + + /// The spelling `s3_sig_v4` of the S3 signer, which the stack no longer + /// reads, in the `auth` table of `origin`. + fn rename_signer(&mut self, origin: &mut Table, setting: &str) { + let Some(auth) = origin.get_mut("auth").and_then(Item::as_table_like_mut) else { + return; + }; + let Some(signer) = auth.get_mut("type") else { + return; + }; + if signer.as_str() == Some("s3_sig_v4") { + set_string(signer, "s3_sigv4"); + self.push_changed(format!( + "{setting} uses the old spelling of the S3 signer, now `s3_sigv4`" + )); + } + } + + /// `[[handlers]]`, removed where an entry guards only paths this release + /// closes, and left for the operator otherwise. + fn handlers(&mut self) { + let Some(item) = self.root().get("handlers") else { + return; + }; + let Some(entries) = item.as_array_of_tables() else { + self.push_to_do( + "`handlers` is not read, and no request is asked for a password. Put a password \ + on the paths it guards at the origin or in the hosting platform, then remove it", + ); + return; + }; + let patterns: Vec> = entries + .iter() + .map(|entry| entry.get("path").and_then(Item::as_str).map(str::to_owned)) + .collect(); + let mut changed = Vec::new(); + let mut to_do = Vec::new(); + for (index, pattern) in patterns.iter().enumerate().rev() { + let number = index + 1; + if pattern.as_deref().is_some_and(guards_only_closed_paths) { + self.layout + .remove(&[Step::Key("handlers".to_owned()), Step::Index(index)]); + changed.push(format!( + "removed `[[handlers]]` entry {number}, which guarded only paths this release \ + closes, where nothing is served now" + )); + } else { + to_do.push(format!( + "`[[handlers]]` entry {number} can guard paths beyond the `/_ts/admin` and \ + `/admin/keys` paths this release closes, and no request is asked for a \ + password now. Put a password on those paths at the origin or in the hosting \ + platform, then remove the entry" + )); + } + } + self.report.changed.extend(changed.into_iter().rev()); + self.report.to_do.extend(to_do.into_iter().rev()); + } + + /// `[request_signing]`'s store ids, which the service does not read. + fn request_signing(&mut self) { + let Some(signing) = self + .root_mut() + .get_mut("request_signing") + .and_then(Item::as_table_like_mut) + else { + return; + }; + let removed: Vec<&str> = ["config_store_id", "secret_store_id"] + .into_iter() + .filter(|key| signing.remove(key).is_some()) + .collect(); + if removed.is_empty() { + return; + } + for key in removed { + self.push_changed(format!( + "removed `[request_signing] {key}`, which the service does not read" + )); + } + self.push_check( + "give the store ids `[request_signing]` held to `ts keys rotate` and `ts keys \ + deactivate`, as `--config-store-id` and `--secret-store-id`", + ); + } + + /// The Edge Cookie section: the deprecated `[ec] passphrase` into the + /// `hmac` module's table, and the old table and module names. + fn edge_cookie(&mut self) { + const RENAMES: &[(&str, &str)] = &[ + ("host-signals", "host_signals"), + ("client-fixed", "client_fixed"), + ]; + let ec_path = path(&["ec"]); + let anchor = self.layout.position_of(&ec_path); + let Some(ec) = self.layout.table_mut(&ec_path) else { + return; + }; + let mut changed = Vec::new(); + let mut to_do = Vec::new(); + + let renamed_block = |name: &str| { + RENAMES + .iter() + .find(|(old, _)| *old == name) + .map_or(name.to_owned(), |(_, new)| (*new).to_owned()) + }; + if let Some(providers) = ec.get("providers").and_then(Item::as_table) { + let names = keys_of(providers); + if let Some(taken) = names + .iter() + .find(|name| ec.contains_key(&renamed_block(name))) + { + to_do.push(format!( + "`[ec.providers.{taken}]` and `[ec.{}]` are both written. Keep one", + renamed_block(taken) + )); + } else if let Some(Ok(mut providers)) = ec.remove("providers").map(Item::into_table) { + for name in names { + let Some((key, item)) = providers.remove_entry(&name) else { + continue; + }; + let new = renamed_block(&name); + ec.insert_formatted(&renamed(&key, &new), item); + changed.push(format!("moved `[ec.providers.{name}]` to `[ec.{new}]`")); + } + } + } + for (old, new) in RENAMES { + if ec.get(old).is_some_and(Item::is_table) && !ec.contains_key(new) { + rename_key(ec, old, new); + changed.push(format!("renamed `[ec.{old}]` to `[ec.{new}]`")); + } + } + if let Some(module) = ec.get_mut("module") + && let Some((_, new)) = RENAMES + .iter() + .find(|(old, _)| module.as_str() == Some(*old)) + { + set_string(module, new); + changed.push(format!( + "renamed the module `[ec] module` selects to `{new}`" + )); + } + for name in keys_of(ec) { + let Some(block) = ec.get_mut(&name).and_then(Item::as_table_like_mut) else { + continue; + }; + let Some(implementation) = block.get_mut("implementation") else { + continue; + }; + if let Some((_, new)) = RENAMES + .iter() + .find(|(old, _)| implementation.as_str() == Some(*old)) + { + set_string(implementation, new); + changed.push(format!( + "renamed the module `[ec.{name}] implementation` names to `{new}`" + )); + } + } + + let mut moved_passphrase = false; + if ec.contains_key("passphrase") { + let selects = ec.contains_key("module") + || ec + .iter() + .any(|(_, item)| item.is_table() || item.is_inline_table()); + if selects { + to_do.push( + "`[ec] passphrase` sits beside an Edge Cookie module of the new form. Keep \ + one form, with the passphrase in the selected module's table, as \ + `[ec.hmac] passphrase`" + .to_owned(), + ); + } else if let Some((key, item)) = ec.remove_entry("passphrase") { + let mut hmac = Table::new(); + hmac.insert_formatted(&key, item); + ec.insert("hmac", Item::Table(hmac)); + insert_first(ec, "module", value("hmac")); + changed.push( + "moved `[ec] passphrase` to `[ec.hmac] passphrase`, with `[ec] module = \ + \"hmac\"`" + .to_owned(), + ); + moved_passphrase = true; + } + } + self.report.changed.extend(changed); + self.report.to_do.extend(to_do); + if moved_passphrase { + let anchor = anchor.unwrap_or_else(|| self.layout.last_position()); + self.layout.place_after(anchor, &[path(&["ec", "hmac"])]); + } + } + + /// `[permission_signal] sources` into `[permission-signal] modules`, with + /// the module names the stack renamed. + fn permission_signal(&mut self) { + const RENAMES: &[(&str, &str)] = &[ + ("gpp-sale-opt-out", "gpp"), + ("gpp_sale_opt_out", "gpp"), + ("us_privacy", "us-privacy"), + ]; + if self.root().contains_key("permission_signal") { + if self.root().contains_key("permission-signal") { + self.push_to_do( + "`[permission_signal]` and `[permission-signal]` are both written. Keep \ + `[permission-signal]`", + ); + return; + } + rename_key(self.root_mut(), "permission_signal", "permission-signal"); + self.push_changed("renamed `[permission_signal]` to `[permission-signal]`"); + } + let Some(section) = self + .root_mut() + .get_mut("permission-signal") + .and_then(Item::as_table_mut) + else { + return; + }; + let mut changed = Vec::new(); + if section.contains_key("sources") && !section.contains_key("modules") { + rename_key(section, "sources", "modules"); + changed.push("renamed `[permission-signal] sources` to `modules`".to_owned()); + } + if let Some(modules) = section.get_mut("modules").and_then(Item::as_array_mut) { + for (old, new) in RENAMES { + let Some(index) = modules.iter().position(|name| name.as_str() == Some(*old)) + else { + continue; + }; + if modules.iter().any(|name| name.as_str() == Some(*new)) { + modules.remove(index); + } else if let Some(name) = modules.get_mut(index) { + set_value_string(name, new); + } + changed.push(format!( + "renamed a module `[permission-signal] modules` names to `{new}`" + )); + } + } + self.report.changed.extend(changed); + } + + /// `[device] module = "fastly"`, which the stack spells `platform`. + fn device(&mut self) { + let Some(module) = self + .root_mut() + .get_mut("device") + .and_then(Item::as_table_like_mut) + .and_then(|device| device.get_mut("module")) + else { + return; + }; + if module.as_str() == Some("fastly") { + set_string(module, "platform"); + self.push_changed( + "`[device] module` now selects `platform` for the platform's own device module", + ); + } + } + + /// `[auction.providers.]` into `[demand.]`, selected in + /// `[demand] modules`, and each bidder route's `provider` into `module`. + fn demand(&mut self) { + let providers_path = path(&["auction", "providers"]); + let Some(providers) = self.layout.table_mut(&providers_path) else { + return; + }; + let ids: Vec = providers + .iter() + .filter(|(_, item)| item.is_table()) + .map(|(key, _)| key.to_owned()) + .collect(); + if ids.is_empty() { + return; + } + if self.root().contains_key("demand") { + self.push_to_do( + "`[auction.providers]` and `[demand]` are both written. Move each provider to \ + `[demand.]`, selected in `[demand] modules`", + ); + return; + } + let mut names: Vec<(String, String)> = Vec::new(); + let mut tables = Vec::new(); + for id in &ids { + let mut provider_path = providers_path.clone(); + provider_path.push(Step::Key(id.clone())); + let profile = self.layout.table_mut(&provider_path).and_then(|table| { + table + .get("profile") + .map(|item| item.as_str().map(str::to_owned)) + }); + let implementation = match profile { + None => Some("auction-protocol.openrtb"), + Some(Some(profile)) => implementation_of(&profile), + Some(None) => None, + }; + let Some(implementation) = implementation else { + self.push_to_do(format!( + "`[auction.providers.{id}] profile` names no implementation this release \ + has, so the provider stays where it is. Move it to `[demand.]` with \ + an `implementation` line by hand" + )); + continue; + }; + let collides = self.layout.table_mut(&provider_path).is_some_and(|table| { + table + .get("profile_config") + .and_then(Item::as_table_like) + .is_some_and(|config| { + config + .iter() + .any(|(key, _)| key == "implementation" || table.contains_key(key)) + }) + }); + if collides { + self.push_to_do(format!( + "`[auction.providers.{id}.profile_config]` holds a setting named as one of \ + the provider's own, so the provider stays where it is. Move it to \ + `[demand.]` by hand" + )); + continue; + } + let Some(mut table) = self.layout.take(&provider_path) else { + continue; + }; + table.remove("protocol"); + let profile_key = table.remove_entry("profile").map(|(key, _)| key); + if let Some(config) = table.remove("profile_config") { + flatten_into(&mut table, config); + } + let key = profile_key.map_or_else( + || Key::new("implementation"), + |key| renamed(&key, "implementation"), + ); + insert_first_formatted(&mut table, &key, value(implementation)); + let name = id.replace('-', "_"); + self.push_changed(format!( + "moved `[auction.providers.{id}]` to `[demand.{name}]`, with `implementation = \ + \"{implementation}\"` and its `profile_config` settings in the table" + )); + names.push((id.clone(), name.clone())); + tables.push((name, table)); + } + if self + .layout + .table_mut(&providers_path) + .is_some_and(|providers| providers.is_empty()) + { + self.layout.remove(&providers_path); + } + if tables.is_empty() { + return; + } + + let mut demand = Table::new(); + let selected: Array = tables.iter().map(|(name, _)| name.as_str()).collect(); + demand.insert("modules", value(selected)); + for (index, (name, mut table)) in tables.into_iter().enumerate() { + if index == 0 { + if let Some(position) = table.position() { + demand.set_position(position); + } + set_prefix(&mut demand, &prefix_of(&table)); + set_prefix(&mut table, "\n"); + } + demand.insert(&name, Item::Table(table)); + } + self.root_mut().insert("demand", Item::Table(demand)); + + let Some(bidders) = self + .root_mut() + .get_mut("auction") + .and_then(|auction| auction.get_mut("bidders")) + .and_then(Item::as_table_mut) + else { + return; + }; + let mut changed = Vec::new(); + for code in keys_of(bidders) { + let Some(route) = bidders.get_mut(&code).and_then(Item::as_table_mut) else { + continue; + }; + if !route.contains_key("provider") || route.contains_key("module") { + continue; + } + rename_key(route, "provider", "module"); + if let Some(module) = route.get_mut("module") + && let Some((_, name)) = names + .iter() + .find(|(id, _)| module.as_str() == Some(id.as_str())) + { + set_string(module, name); + } + changed.push(format!( + "renamed `[auction.bidders.{code}] provider` to `module`, naming the demand \ + source by its new name" + )); + } + self.report.changed.extend(changed); + } + + /// Every `[integrations.]` table, each into the module that replaced + /// it, and `[integration]`, which is left for the operator. + fn integrations(&mut self) { + if self.root().contains_key("integration") { + self.push_to_do( + "`[integration]` is not read. Select each module in the section of its type, \ + with its settings in `[
.]`", + ); + } + let Some(integrations) = self.root().get("integrations") else { + return; + }; + let Some(integrations) = integrations.as_table() else { + self.push_to_do( + "`integrations` is not read. Select each module in the section of its type, \ + with its settings in `[
.]`", + ); + return; + }; + let ids = keys_of(integrations); + for id in ids { + self.layout + .make_table(&path(&["integrations", id.as_str()])); + match id.as_str() { + "aps" => self.aps(), + "adserver_mock" => self.adserver_mock(), + _ => match MAIN_INTEGRATIONS.iter().find(|entry| entry.id == id) { + Some(entry) => self.integration(entry), + None => self.push_to_do(format!( + "`[integrations.{id}]` names no module this release has. Select the \ + module that does its job in the section of its type, or remove the \ + table" + )), + }, + } + } + if self + .root() + .get("integrations") + .and_then(Item::as_table) + .is_some_and(Table::is_empty) + { + self.layout.remove(&path(&["integrations"])); + } + } + + /// One `[integrations.]` table into its module. + fn integration(&mut self, entry: &MainIntegration) { + let id = entry.id; + let table_path = path(&["integrations", id]); + let Some((section, written)) = self.target(id) else { + self.push_to_do(format!( + "`[integrations.{id}]` names a module this build of the tool does not link" + )); + return; + }; + let Some(table) = self.layout.table_mut(&table_path) else { + return; + }; + let enabled = match table.get("enabled") { + None => entry.enabled_by_default, + Some(item) => { + if let Some(enabled) = item.as_bool() { + enabled + } else { + self.push_to_do(format!( + "`[integrations.{id}] enabled` is not true or false, so it cannot say \ + whether the module runs" + )); + return; + } + } + }; + table.remove("enabled"); + let empty = table.is_empty(); + let mut header = vec![section]; + header.extend(written.split('.')); + + if !enabled { + if entry.has_settings { + self.layout.hold(&table_path, Some(&header[..])); + self.push_changed(format!( + "commented out `[integrations.{id}]` as `[{}]`, because it was not enabled", + header.join(".") + )); + } else { + self.layout.hold(&table_path, None); + self.push_changed(format!( + "removed `[integrations.{id}]`, which was not enabled and has no settings, \ + keeping the comments above it" + )); + } + return; + } + + if !entry.has_settings || empty { + let position = self.layout.position_of(&table_path); + if self.root().contains_key(section) { + self.layout.remove(&table_path); + } else if let Some(table) = self.layout.take(&table_path) { + self.ensure_section(section, position, &prefix_of(&table)); + } + let key = self.select(section, written); + self.push_changed(format!( + "selected `{written}` in `[{section}] {key}` for `[integrations.{id}]`, which \ + sets nothing else" + )); + return; + } + + let Some(mut table) = self.layout.take(&table_path) else { + return; + }; + if self.ensure_section(section, table.position(), &prefix_of(&table)) { + set_prefix(&mut table, "\n"); + } + insert_at(self.root_mut(), &header, table); + let key = self.select(section, written); + self.push_changed(format!( + "moved `[integrations.{id}]` to `[{}]`, selected in `[{section}] {key}`", + header.join(".") + )); + } + + /// `[integrations.aps] rendering_mode` into each `auction.aps` demand + /// source. Main read it only from an enabled table. + fn aps(&mut self) { + let table_path = path(&["integrations", "aps"]); + let Some(table) = self.layout.table_mut(&table_path) else { + return; + }; + let enabled = match table.get("enabled") { + None => false, + Some(item) => { + if let Some(enabled) = item.as_bool() { + enabled + } else { + self.push_to_do( + "`[integrations.aps] enabled` is not true or false, so it cannot say \ + whether `rendering_mode` applies", + ); + return; + } + } + }; + let mode = table.get("rendering_mode").cloned(); + let sources: Vec = self + .root() + .get("demand") + .and_then(Item::as_table) + .map(|demand| { + demand + .iter() + .filter(|(_, item)| { + item.get("implementation").and_then(Item::as_str) == Some("auction.aps") + }) + .map(|(name, _)| name.to_owned()) + .collect() + }) + .unwrap_or_default(); + match mode { + Some(mode) if enabled => { + if sources.is_empty() { + self.push_to_do( + "`[integrations.aps] rendering_mode` belongs in the table of an \ + `auction.aps` demand source, and the document has none. Add it to that \ + source's `[demand.]` table, then remove `[integrations.aps]`", + ); + return; + } + for name in &sources { + if let Some(source) = self + .root_mut() + .get_mut("demand") + .and_then(|demand| demand.get_mut(name)) + .and_then(Item::as_table_mut) + && !source.contains_key("rendering_mode") + { + source.insert("rendering_mode", mode.clone()); + } + } + self.layout.remove(&table_path); + self.push_changed( + "moved `[integrations.aps] rendering_mode` into the table of each \ + `auction.aps` demand source", + ); + } + mode => { + self.layout.remove(&table_path); + self.push_changed( + "removed `[integrations.aps]`, which sets nothing this release reads", + ); + if mode.is_some() { + self.push_check( + "main did not apply `[integrations.aps] rendering_mode`, because the \ + table was not enabled. Set `rendering_mode` in an `auction.aps` demand \ + source's table to use it", + ); + } + } + } + } + + /// `[integrations.adserver_mock]` into `[ad-server.iabtl]`, selected in + /// place of `[auction] mediator`, which named it. + fn adserver_mock(&mut self) { + let table_path = path(&["integrations", "adserver_mock"]); + let mediator = self + .root() + .get("auction") + .and_then(|auction| auction.get("mediator")) + .and_then(Item::as_str) + .map(str::to_owned); + let names_it = mediator.as_deref() == Some("adserver_mock"); + if names_it + && (self.root().contains_key("ad-server") || self.root().contains_key("adserver")) + { + self.push_to_do( + "`[auction] mediator` and `[ad-server]` are both written. Keep `[ad-server]`", + ); + return; + } + let Some(table) = self.layout.table_mut(&table_path) else { + return; + }; + table.remove("enabled"); + let header = ["ad-server", "iabtl"]; + if !names_it { + self.layout.hold(&table_path, Some(&header[..])); + self.push_changed( + "commented out `[integrations.adserver_mock]` as `[ad-server.iabtl]`, because \ + `[auction] mediator` does not name it", + ); + return; + } + let Some(mut table) = self.layout.take(&table_path) else { + return; + }; + if self.ensure_section("ad-server", table.position(), &prefix_of(&table)) { + set_prefix(&mut table, "\n"); + } + insert_at(self.root_mut(), &header, table); + self.select("ad-server", "iabtl"); + if let Some(auction) = self + .root_mut() + .get_mut("auction") + .and_then(Item::as_table_like_mut) + { + auction.remove("mediator"); + } + self.push_changed( + "moved `[integrations.adserver_mock]` to `[ad-server.iabtl]`, selected with \ + `[ad-server] module = \"iabtl\"` in place of `[auction] mediator`", + ); + } + + /// An `[auction] mediator` the mock ad server's table did not come with. + /// A mediator left beside that table is reported where the table is. + fn mediator_left(&mut self) { + let mock_table = self + .root() + .get("integrations") + .is_some_and(|integrations| integrations.get("adserver_mock").is_some()); + if !mock_table + && self + .root() + .get("auction") + .is_some_and(|auction| auction.get("mediator").is_some()) + { + self.push_to_do( + "`[auction] mediator` names an ad server whose settings the document does not \ + hold. Select the ad server with `[ad-server] module = \"iabtl\"`, give it an \ + `[ad-server.iabtl]` table, then remove the line", + ); + } + } + + /// The ad server the repository ships, `ad-server.iabtl`, where it was + /// `ad-server.mock`, and `[ad-server]`, where it was `[adserver]`. + fn ad_server_names(&mut self) { + if self.root().contains_key("adserver") { + if self.root().contains_key("ad-server") { + self.push_to_do( + "`[adserver]` and `[ad-server]` are both written. Keep `[ad-server]`", + ); + return; + } + rename_key(self.root_mut(), "adserver", "ad-server"); + self.push_changed("renamed `[adserver]` to `[ad-server]`"); + } + let Some(section) = self + .root_mut() + .get_mut("ad-server") + .and_then(Item::as_table_mut) + else { + return; + }; + let mut changed = Vec::new(); + if let Some(module) = section.get_mut("module") { + let renamed = match module.as_str() { + Some("mock") => Some("iabtl"), + Some("ad-server.mock") => Some("ad-server.iabtl"), + _ => None, + }; + if let Some(new) = renamed { + set_string(module, new); + changed.push(format!( + "renamed the ad server `[ad-server] module` selects to `{new}`" + )); + } + } + if section.get("mock").is_some_and(Item::is_table) && !section.contains_key("iabtl") { + rename_key(section, "mock", "iabtl"); + changed.push("renamed `[ad-server.mock]` to `[ad-server.iabtl]`".to_owned()); + } + for name in keys_of(section) { + let Some(implementation) = section + .get_mut(&name) + .and_then(Item::as_table_like_mut) + .and_then(|table| table.get_mut("implementation")) + else { + continue; + }; + if implementation.as_str() == Some("ad-server.mock") { + set_string(implementation, "ad-server.iabtl"); + changed.push(format!( + "renamed the ad server `[ad-server.{name}] implementation` names to \ + `ad-server.iabtl`" + )); + } + } + self.report.changed.extend(changed); + } + + /// The `ads.txt` module `ads-txt.iabtl`, where it was `ads-txt.sellers`, in + /// its section, its table and the `[[fetch]]` entries. + fn ads_txt_names(&mut self) { + let mut changed = Vec::new(); + if let Some(section) = self + .root_mut() + .get_mut("ads-txt") + .and_then(Item::as_table_mut) + { + for key in ["module", "modules"] { + let Some(item) = section.get_mut(key) else { + continue; + }; + if rename_names( + item, + &[("sellers", "iabtl"), ("ads-txt.sellers", "ads-txt.iabtl")], + ) { + changed.push(format!( + "renamed the ads.txt module `[ads-txt] {key}` selects to `iabtl`" + )); + } + } + if section.get("sellers").is_some_and(Item::is_table) && !section.contains_key("iabtl") + { + rename_key(section, "sellers", "iabtl"); + changed.push("renamed `[ads-txt.sellers]` to `[ads-txt.iabtl]`".to_owned()); + } + } + for phase in ["fetch", "serve"] { + let Some(entries) = self + .root_mut() + .get_mut(phase) + .and_then(Item::as_array_of_tables_mut) + else { + continue; + }; + for (index, entry) in entries.iter_mut().enumerate() { + let Some(names) = entry.get_mut("middleware") else { + continue; + }; + if rename_names(names, &[("ads-txt.sellers", "ads-txt.iabtl")]) { + changed.push(format!( + "renamed the middleware `ads-txt.sellers` to `ads-txt.iabtl` in \ + `[[{phase}]]` entry {}", + index + 1 + )); + } + } + } + self.report.changed.extend(changed); + } + + /// `[tinybird]` into `[analytics.tinybird]`, selected by `[analytics] + /// module` in place of its `enabled` line. + fn tinybird(&mut self) { + let table_path = path(&["tinybird"]); + self.layout.make_table(&table_path); + let Some(table) = self.layout.table_mut(&table_path) else { + return; + }; + let enabled = match table.get("enabled") { + None => false, + Some(item) => { + if let Some(enabled) = item.as_bool() { + enabled + } else { + self.push_to_do( + "`[tinybird] enabled` is not true or false, so it cannot say whether \ + telemetry is sent", + ); + return; + } + } + }; + if self.root().contains_key("analytics") { + self.push_to_do("`[tinybird]` and `[analytics]` are both written. Keep `[analytics]`"); + return; + } + if let Some(table) = self.layout.table_mut(&table_path) { + table.remove("enabled"); + } + let header = ["analytics", "tinybird"]; + if !enabled { + self.layout.hold(&table_path, Some(&header[..])); + self.push_changed( + "commented out `[tinybird]` as `[analytics.tinybird]`, because it was not enabled", + ); + return; + } + let Some(mut table) = self.layout.take(&table_path) else { + return; + }; + if self.ensure_section("analytics", table.position(), &prefix_of(&table)) { + set_prefix(&mut table, "\n"); + } + insert_at(self.root_mut(), &header, table); + self.select("analytics", "tinybird"); + self.push_changed( + "moved `[tinybird]` to `[analytics.tinybird]`, selected with `[analytics] module = \ + \"tinybird\"` in place of its `enabled` line", + ); + self.push_check( + "a telemetry row is now sent only for a reader whose permissions include \ + `analytics.ad_reporting.measure_ad_performance`", + ); + } + + /// `include_mediator_response`, which the stack spells + /// `include_adserver_response`. + fn debug_options(&mut self) { + let Some(options) = self + .root_mut() + .get_mut("debug") + .and_then(|debug| debug.get_mut("auction_html_comment_options")) + .and_then(Item::as_table_mut) + else { + return; + }; + if options.contains_key("include_mediator_response") + && !options.contains_key("include_adserver_response") + { + rename_key( + options, + "include_mediator_response", + "include_adserver_response", + ); + self.push_changed( + "renamed `[debug.auction_html_comment_options] include_mediator_response` to \ + `include_adserver_response`", + ); + } + } + + /// The lists of places `[consent]` held, which the stack reads from the + /// publisher's permission policy file. + fn consent_lists(&mut self) { + let Some(consent) = self.root().get("consent") else { + return; + }; + let held: Vec<&str> = ["gdpr", "us_states"] + .into_iter() + .filter(|key| consent.get(key).is_some()) + .collect(); + for key in held { + self.push_to_do(format!( + "`[consent.{key}]` is not read. The places it covers are the `jurisdiction` \ + values in the publisher's permission policy file, the one `[permissions] \ + policy` names, so carry any change of yours there, then remove the table" + )); + } + } + + /// A document that names no permission policy file, which no domain + /// starts without. No policy ships with Trusted Server, so the command + /// chooses none and leaves naming one to do. + fn permission_policy(&mut self) { + let named = self + .root() + .get("permissions") + .and_then(|permissions| permissions.get("policy")) + .and_then(Item::as_str) + .is_some_and(|policy| !policy.is_empty()); + if !named { + self.push_to_do(POLICY_TO_DO); + } + } + + /// A `template_cache_key_cookies` entry under the public prefix, which + /// is one reader's own state. + fn template_cookies(&mut self) { + let prefix = PublicPrefix::default(); + let refused = self + .root() + .get("creative_opportunities") + .and_then(|config| config.get("template_cache_key_cookies")) + .and_then(Item::as_array) + .is_some_and(|names| { + names + .iter() + .filter_map(Value::as_str) + .any(|name| prefix.owns_cookie_name(name.as_bytes())) + }); + if refused { + self.push_to_do(format!( + "`[creative_opportunities] template_cache_key_cookies` names a cookie under the \ + `{prefix}` prefix, which is one reader's own state and is refused now. Remove \ + it" + )); + } + } + + /// The host's location lookup main ran on every request, for a document + /// written for main, on an adapter whose host has one. The Axum and Spin + /// adapters refuse the selection, and main's lookup there located no + /// request, so the document gets no geo module there. Cloudflare's + /// lookup now reads the region, which main's did not, so the region + /// rules of `permissions.yaml` can apply there, and a check line says so. + fn main_defaults(&mut self) { + if !self.adapter.has_location_lookup() || self.root().contains_key("geo") { + return; + } + let mut geo = Table::new(); + geo.insert("module", value("platform")); + self.root_mut().insert("geo", Item::Table(geo)); + self.push_changed( + "selected `[geo] module = \"platform\"`, the host's location lookup main ran on \ + every request", + ); + if self.adapter == Adapter::Cloudflare { + self.push_check( + "on Cloudflare the location lookup now reads the reader's region from \ + `cf-region-code`, where Cloudflare sends it, and main's read none, so the \ + region rules of `permissions.yaml`, such as a US state's `us-state` \ + jurisdiction, can apply to a request now", + ); + } + let anchor = self + .layout + .subtree_end(&path(&["source"])) + .or_else(|| self.layout.position_of(&path(&["publisher"]))) + .unwrap_or_else(|| self.layout.last_position()); + self.layout.place_after(anchor, &[path(&["geo"])]); + } + + /// The location settings startup refuses and `ts config validate` does + /// not check, being the host's own lookup on an adapter whose host has + /// none, and an Edge Cookie module with no geo module unless `[geo] + /// assume_single_jurisdiction` accepts that every request then takes the + /// top of the `permissions.yaml` rules. Which way out to take is the + /// operator's decision, so each is left to do. + fn location_at_startup(&mut self) { + let geo = self.root().get("geo"); + let module = geo + .and_then(|geo| geo.get("module")) + .and_then(Item::as_str) + .map(str::to_owned); + let acknowledged = geo + .and_then(|geo| geo.get("assume_single_jurisdiction")) + .and_then(Item::as_bool) + == Some(true); + let edge_cookie = self + .root() + .get("ec") + .and_then(|ec| ec.get("module")) + .and_then(Item::as_str) + .is_some_and(|module| module != "none"); + let name = self.adapter.name(); + if module.as_deref() == Some("platform") && !self.adapter.has_location_lookup() { + let acknowledge = if edge_cookie { + " and set `[geo] assume_single_jurisdiction = true`, which accepts that every \ + request takes the top of the `permissions.yaml` rules" + } else { + "" + }; + self.push_to_do(format!( + "`[geo] module = \"platform\"` selects the host's location lookup, and {name} has \ + none, so startup refuses it. Select a geo module a linked crate offers, or remove \ + the line{acknowledge}" + )); + return; + } + if edge_cookie && matches!(module.as_deref(), None | Some("none")) && !acknowledged { + self.push_to_do(if self.adapter.has_location_lookup() { + "`[ec]` selects an Edge Cookie module and `[geo]` selects no geo module, so every \ + request takes the top of the `permissions.yaml` rules, which startup refuses \ + unless `[geo] assume_single_jurisdiction = true` accepts it. Select a geo \ + module, such as `platform`, or set that line" + .to_owned() + } else { + format!( + "`[ec]` selects an Edge Cookie module and no geo module is selected, because \ + {name} has no location lookup of its own, so every request takes the top of \ + the `permissions.yaml` rules, which startup refuses unless `[geo] \ + assume_single_jurisdiction = true` accepts it. Set that line, or select a geo \ + module a linked crate offers" + ) + }); + } + } + + /// The `[[fetch]]` and `[[serve]]` entries that place the page changes + /// of the selected modules, for a document written for main, and the + /// entry each file a module makes needs. + fn page_entries(&mut self) { + let entries = self.wanted_entries(); + let mut new: Vec = Vec::new(); + for wanted in entries { + let PageEntry { + phase, + media_type, + path: file, + middleware, + } = wanted; + let mut entry = Table::new(); + entry.insert("media_type", value(media_type)); + if let Some(file) = file { + entry.insert("path", value(file)); + } + entry.insert("middleware", value(name_list(&middleware))); + set_prefix( + &mut entry, + if new.is_empty() { + ENTRIES_COMMENT + } else { + "\n" + }, + ); + let array = self + .root_mut() + .entry(phase) + .or_insert_with(|| Item::ArrayOfTables(toml_edit::ArrayOfTables::new())); + let Some(array) = array.as_array_of_tables_mut() else { + continue; + }; + array.push(entry); + new.push(vec![ + Step::Key(phase.to_owned()), + Step::Index(array.len() - 1), + ]); + self.report.changed.push(match file { + Some(file) => format!( + "added a `[[{phase}]]` entry for `{media_type}` on `{file}` that names `{}`", + middleware.join("`, `") + ), + None => format!( + "added a `[[{phase}]]` entry for `{media_type}` that names the middleware of \ + the selected modules" + ), + }); + } + if new.is_empty() { + return; + } + let anchor = self.layout.last_position(); + self.layout.place_after(anchor, &new); + } + + /// The page entries the document lacks: one for each phase and media + /// type the selected modules supply middleware for, in the order the + /// builders supply it, for a document written for main, whose modules + /// changed every page, and one for each file a selected module makes + /// that no entry places, which startup refuses. A document in the + /// stack's shape loads without page entries, so they are left to the + /// operator there. + fn wanted_entries(&self) -> Vec { + let selected = self.selected_ids(); + let supplied = supplied_middleware(&crate::app_config::linked_builders()); + let mut entries = Vec::new(); + for phase in MiddlewarePhase::ALL { + let mut media_types: Vec<&'static str> = Vec::new(); + for (id, middleware) in &supplied { + if middleware.phase == phase + && selected.contains(id) + && !media_types.contains(&middleware.media_type) + { + media_types.push(middleware.media_type); + } + } + media_types.sort_by_key(|media_type| *media_type != HTML_MEDIA_TYPE); + for media_type in media_types { + let names: Vec<&'static str> = supplied + .iter() + .filter(|(id, middleware)| { + middleware.phase == phase + && middleware.media_type == media_type + && selected.contains(id) + }) + .map(|(_, middleware)| middleware.name) + .collect(); + let mut rest = Vec::new(); + for name in names { + match FILE_MIDDLEWARE.iter().find(|(file, _)| *file == name) { + Some((_, file)) => { + if !self.names_middleware(name) { + entries.push(PageEntry { + phase: phase.as_str(), + media_type, + path: Some(*file), + middleware: vec![name], + }); + } + } + None => rest.push(name), + } + } + if self.for_main && !rest.is_empty() { + entries.push(PageEntry { + phase: phase.as_str(), + media_type, + path: None, + middleware: rest, + }); + } + } + } + entries.sort_by_key(|entry| entry.phase != MiddlewarePhase::Fetch.as_str()); + entries + } + + /// What main did on every request that no setting expresses now, for a + /// document written for main. + fn main_checks(&mut self) { + self.push_check( + "no response carries `x-ts-version` or the platform's version now. Name \ + `service_version` in the `[[serve]]` entry to write `x-ts-version` and \ + `x-ts-service-version` on the pages it covers", + ); + if !self.selected_ids().contains(&"nextjs") { + self.push_check( + "core no longer moves the origin's address in a React Server Components Flight \ + response (`text/x-component`). A Next.js App Router site selects `[framework] \ + module = \"nextjs\"`, which adds its page changes, and names \ + `framework.nextjs.flight` in a `[[fetch]]` entry for `text/x-component`", + ); + } + self.push_check(self.geo_headers_check()); + } + + /// What became of the location headers main wrote on every response on + /// the adapter. `headers.geo` is never selected for the operator, + /// because it writes the reader's location. + fn geo_headers_check(&self) -> String { + match self.adapter { + Adapter::Fastly => "no response carries an `x-geo-*` header now, where main wrote the \ + reader's location in them on every response. To keep them, \ + select `[headers] modules = [\"geo\"]` and name `headers.geo` in a \ + `[[serve]]` entry for the pages that are to carry them" + .to_owned(), + Adapter::Cloudflare => "no response carries `x-geo-info-available` now, the one \ + `x-geo-*` header main wrote on Cloudflare. `headers.geo` \ + would write it again, and with it the reader's location in \ + the other `x-geo-*` headers, which main never sent on \ + Cloudflare" + .to_owned(), + Adapter::Axum | Adapter::Spin => format!( + "no response carries `x-geo-info-available` now, which main wrote as `false` on \ + every response on {}, where it located no request", + self.adapter.name() + ), + } + } + + /// The section and the name a section selects the module that replaced + /// main's integration `id` by. + fn target(&self, id: &str) -> Option<(&'static str, &'static str)> { + self.modules + .iter() + .find(|module| module.id == id) + .map(|module| (module.section, module.written)) + } + + /// The ids of the modules the document selects, core's `robots.txt` + /// middleware with `[robots-txt]` among them. + fn selected_ids(&self) -> Vec<&'static str> { + let root = self.root(); + let mut ids: Vec<&'static str> = self + .modules + .iter() + .filter(|module| { + root.get(module.section).is_some_and(|section| { + selection(section) + .iter() + .any(|name| name == module.written || name == module.full) + }) + }) + .map(|module| module.id) + .collect(); + for &(id, vendor) in &self.shared { + if root + .get("shared") + .is_some_and(|shared| shared.get(vendor).is_some()) + { + ids.push(id); + } + } + if root.contains_key("robots-txt") { + ids.push(trusted_server_core::robots_txt::middleware::MODULE); + } + ids + } + + /// Whether a `[[fetch]]` or `[[serve]]` entry names `middleware`. + fn names_middleware(&self, middleware: &str) -> bool { + ["fetch", "serve"].into_iter().any(|phase| { + self.root() + .get(phase) + .and_then(Item::as_array_of_tables) + .is_some_and(|entries| { + entries.iter().any(|entry| { + entry + .get("middleware") + .and_then(Item::as_array) + .is_some_and(|names| { + names.iter().any(|name| name.as_str() == Some(middleware)) + }) + }) + }) + }) + } + + /// Makes `section` a table that writes its header, at `position` and + /// with `prefix` ahead of it when it is made now, and says whether it + /// took them. + fn ensure_section(&mut self, section: &str, position: Option, prefix: &str) -> bool { + let root = self.root_mut(); + if let Some(existing) = root.get_mut(section) { + let Some(table) = existing.as_table_mut() else { + return false; + }; + if !table.is_implicit() { + return false; + } + table.set_implicit(false); + if let Some(position) = position { + table.set_position(position); + } + set_prefix(table, prefix); + return true; + } + let mut table = Table::new(); + if let Some(position) = position { + table.set_position(position); + } + set_prefix(&mut table, prefix); + root.insert(section, Item::Table(table)); + true + } + + /// Selects `written` in `section`, in the order the builders list the + /// modules, and gives the key the selection is written under. + fn select(&mut self, section: &str, written: &'static str) -> &'static str { + let ranks: Vec<(&'static str, usize)> = self + .modules + .iter() + .enumerate() + .filter(|(_, module)| module.section == section) + .map(|(rank, module)| (module.written, rank)) + .collect(); + let rank = |name: &str| { + ranks + .iter() + .find(|(written, _)| *written == name) + .map_or(usize::MAX, |(_, rank)| *rank) + }; + let Some(table) = self + .root_mut() + .get_mut(section) + .and_then(Item::as_table_mut) + else { + return "module"; + }; + if let Some(modules) = table.get_mut("modules").and_then(Item::as_array_mut) { + if !modules.iter().any(|name| name.as_str() == Some(written)) { + let index = modules + .iter() + .position(|name| name.as_str().is_some_and(|name| rank(name) > rank(written))) + .unwrap_or(modules.len()); + modules.insert(index, written); + modules.fmt(); + } + return "modules"; + } + if let Some(existing) = table + .get("module") + .and_then(Item::as_str) + .map(str::to_owned) + { + if existing == written { + return "module"; + } + let mut names = vec![existing.as_str(), written]; + names.sort_by_key(|name| rank(name)); + rename_key(table, "module", "modules"); + if let Some(item) = table.get_mut("modules") { + let names: Array = names.into_iter().collect(); + *item = value(names); + } + return "modules"; + } + if LIST_SECTIONS.contains(§ion) { + let names: Array = std::iter::once(written).collect(); + insert_first(table, "modules", value(names)); + "modules" + } else { + insert_first(table, "module", value(written)); + "module" + } + } +} + +/// Whether `root` was written for main: it has `[publisher]`, and every +/// table at its top level is one main read. +fn written_for_main(root: &Table) -> bool { + root.contains_key("publisher") && root.iter().all(|(key, _)| MAIN_TABLES.contains(&key)) +} + +/// The implementation a main `profile` names, by module path. +fn implementation_of(profile: &str) -> Option<&'static str> { + match profile { + "standard" => Some("auction-protocol.openrtb"), + "prebid-server" => Some("auction.prebid-server"), + "aps" => Some("auction.aps"), + _ => None, + } +} + +/// Whether every path a `[[handlers]]` pattern matches is one this release +/// closes, with no alternative outside a group that could match another. +/// +/// Main matched a pattern anywhere in the path, so the pattern must be +/// anchored. Every path that starts with `/_ts/admin` is closed, but of +/// `/admin/keys` only the address itself and the paths below `/admin/keys/` +/// are, so that prefix must be followed by a `/` or the end of the path, and +/// `^/admin/keys` alone, which also matches `/admin/keystore`, is not. +/// `?` and `*` let what comes before them match no times, and so can a +/// count in `{}`, so none of the three may follow `^/_ts/admin` or that +/// bound, as `^/_ts/admin?` also matches `/_ts/admit` and `^/admin/keys/?` +/// matches `/admin/keystore`. The text after the prefix or the bound may +/// hold no `[` and no `#` either, because a character class, or a comment +/// the `(?x)` flag allows, can hide a `|` from +/// [`has_top_level_alternation`], as the `(` in `[](]` is a character and +/// not a group. +fn guards_only_closed_paths(pattern: &str) -> bool { + const KEYS_BOUNDS: &[&str] = &["/", "$", "(?:/|$)", "(/|$)"]; + let rest = match pattern.strip_prefix("^/admin/keys") { + Some(rest) => KEYS_BOUNDS + .iter() + .find_map(|bound| rest.strip_prefix(bound)), + None => pattern.strip_prefix("^/_ts/admin"), + }; + rest.is_some_and(|rest| !rest.starts_with(['?', '*', '{']) && !rest.contains(['[', '#'])) + && !has_top_level_alternation(pattern) +} + +/// Whether `pattern` has a `|` outside every group. +/// +/// It reads escapes and groups, and no character class or comment, which +/// its caller refuses. +fn has_top_level_alternation(pattern: &str) -> bool { + let mut depth = 0_usize; + let mut characters = pattern.chars(); + while let Some(character) = characters.next() { + match character { + '\\' => { + characters.next(); + } + '(' => depth += 1, + ')' => depth = depth.saturating_sub(1), + '|' if depth == 0 => return true, + _ => {} + } + } + false +} + +/// The names a section selects, from `module` or `modules`. +fn selection(section: &Item) -> Vec { + if let Some(module) = section.get("module").and_then(Item::as_str) { + return vec![module.to_owned()]; + } + section + .get("modules") + .and_then(Item::as_array) + .map(|names| { + names + .iter() + .filter_map(Value::as_str) + .map(str::to_owned) + .collect() + }) + .unwrap_or_default() +} + +/// The keys of `table`, in order. +fn keys_of(table: &Table) -> Vec { + table.iter().map(|(key, _)| key.to_owned()).collect() +} + +/// The keys of `table` among `wanted`, in the order the table holds them. +fn keys_in_order<'a>(table: &Table, wanted: impl Iterator) -> Vec<&'a str> { + let wanted: Vec<&str> = wanted.collect(); + table + .iter() + .filter_map(|(key, _)| wanted.iter().find(|name| **name == key).copied()) + .collect() +} + +/// `key` under the name `name`, with the comments and spacing written +/// around it. +fn renamed(key: &Key, name: &str) -> Key { + Key::new(name).with_leaf_decor(key.leaf_decor().clone()) +} + +/// Renames `old` to `new` in `table`, keeping its place and its comments. +fn rename_key(table: &mut Table, old: &str, new: &str) { + let entries: Vec<(Key, Item)> = keys_of(table) + .iter() + .filter_map(|key| table.remove_entry(key)) + .collect(); + for (key, item) in entries { + if key.get() == old { + table.insert_formatted(&renamed(&key, new), item); + } else { + table.insert_formatted(&key, item); + } + } +} + +/// Puts `item` under `key` ahead of every other entry of `table`. +fn insert_first(table: &mut Table, key: &str, item: Item) { + insert_first_formatted(table, &Key::new(key), item); +} + +/// Puts `item` under `key`, as formatted, ahead of every other entry. +fn insert_first_formatted(table: &mut Table, key: &Key, item: Item) { + let entries: Vec<(Key, Item)> = keys_of(table) + .iter() + .filter_map(|name| table.remove_entry(name)) + .collect(); + table.insert_formatted(key, item); + for (key, item) in entries { + table.insert_formatted(&key, item); + } +} + +/// Puts `table` at `parts` below `root`, making the tables on the way as +/// tables that write no header. +fn insert_at(root: &mut Table, parts: &[&str], table: Table) { + let Some((last, parents)) = parts.split_last() else { + return; + }; + let mut parent = root; + for part in parents { + let entry = parent.entry(part).or_insert_with(|| { + let mut implicit = Table::new(); + implicit.set_implicit(true); + Item::Table(implicit) + }); + let Some(next) = entry.as_table_mut() else { + return; + }; + parent = next; + } + parent.insert(last, Item::Table(table)); +} + +/// Moves the settings of `config`, a provider's `profile_config`, into the +/// provider's own table, after its own, with the comments written above +/// `config` ahead of the first of them. +fn flatten_into(table: &mut Table, config: Item) { + let Ok(mut config) = config.into_table() else { + return; + }; + let mut prefix = Some(prefix_of(&config)); + for name in keys_of(&config) { + let Some((mut key, item)) = config.remove_entry(&name) else { + continue; + }; + if item.is_value() + && let Some(prefix) = prefix.take() + { + let own = key + .leaf_decor() + .prefix() + .and_then(RawString::as_str) + .unwrap_or("") + .to_owned(); + key.leaf_decor_mut().set_prefix(format!("{prefix}{own}")); + } + table.insert_formatted(&key, item); + } +} + +/// Sets the string `item` holds to `text`, keeping its comments. +fn set_string(item: &mut Item, text: &str) { + if let Some(found) = item.as_value_mut() { + set_value_string(found, text); + } +} + +/// Sets the string `found` holds to `text`, keeping its comments. +fn set_value_string(found: &mut Value, text: &str) { + let decor = found.decor().clone(); + *found = Value::from(text); + *found.decor_mut() = decor; +} + +/// Renames the names `item` holds, a name or a list of names, and says +/// whether any changed. +fn rename_names(item: &mut Item, renames: &[(&str, &str)]) -> bool { + let mut changed = false; + if let Some(name) = item.as_value_mut() { + if let Some(array) = name.as_array_mut() { + for name in array.iter_mut() { + if let Some((_, new)) = renames.iter().find(|(old, _)| name.as_str() == Some(*old)) + { + set_value_string(name, new); + changed = true; + } + } + } else if let Some((_, new)) = renames.iter().find(|(old, _)| name.as_str() == Some(*old)) { + set_value_string(name, new); + changed = true; + } + } + changed +} + +/// The names of a page entry's `middleware`, one to a line where there are +/// several. +fn name_list(names: &[&str]) -> Array { + let mut array: Array = names.iter().copied().collect(); + if names.len() > 1 { + for name in array.iter_mut() { + name.decor_mut().set_prefix("\n "); + } + array.set_trailing_comma(true); + array.set_trailing("\n"); + } + array +} diff --git a/crates/trusted-server-cli/src/commands/config/migrate/tests.rs b/crates/trusted-server-cli/src/commands/config/migrate/tests.rs new file mode 100644 index 0000000000..ed0cbdaa59 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/config/migrate/tests.rs @@ -0,0 +1,2089 @@ +use std::fs; + +use edgezero_core::body::Body as EdgeBody; +use http::Request; +use regex::Regex; +use tempfile::TempDir; +use trusted_server_core::closed_paths::closed_path_response; +use trusted_server_core::settings::Settings; + +use super::rules::{Migration, POLICY_TO_DO}; +use super::{Adapter, ConfigMigrateArgs, migrate, run_with_writers}; +use crate::RunOutcome; +use crate::commands::audit::generate::validate::validate_source_config; + +const ADAPTERS: [Adapter; 4] = [ + Adapter::Fastly, + Adapter::Cloudflare, + Adapter::Axum, + Adapter::Spin, +]; + +/// Main's own example settings document. +const MAIN_EXAMPLE: &str = include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/tests/fixtures/main-trusted-server.example.toml" +)); + +/// The stack's example settings document, which shows the page entries in +/// the order the modules' page changes run. +const STACK_EXAMPLE: &str = include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../trusted-server.example.toml" +)); + +/// The settings every document written for main carries, with hosts that +/// are not the template's placeholders. +const MAIN_BASE: &str = r#"[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +origin_url = "https://origin.publisher.example.com" +proxy_secret = "publisher_proxy_secret" + +[ec] +passphrase = "ec_passphrase" +"#; + +/// `input` migrated for Fastly, checked to change nothing when migrated +/// again. +fn migrated(input: &str) -> Migration { + migrated_on(Adapter::Fastly, input) +} + +/// `input` migrated for `adapter`, checked to change nothing when migrated +/// again. +fn migrated_on(adapter: Adapter, input: &str) -> Migration { + let migration = migrate(input, adapter).expect("should migrate the document"); + let again = migrate(&migration.text, adapter).expect("should migrate the migrated document"); + assert!( + again.report.changed.is_empty(), + "migrating a migrated document should change nothing, got {:?}", + again.report.changed + ); + assert_eq!( + again.text, migration.text, + "migrating a migrated document should leave its text as it was" + ); + migration +} + +fn parsed(text: &str) -> toml::Value { + toml::from_str(text).unwrap_or_else(|error| panic!("should be TOML: {error}\n{text}")) +} + +/// `text` read as core reads its settings, before the checks startup makes. +fn settings_of(text: &str) -> Settings { + toml::from_str(text).unwrap_or_else(|error| panic!("should read the settings: {error}\n{text}")) +} + +fn main_document(rest: &str) -> String { + format!("{MAIN_BASE}\n{rest}") +} + +fn reported(lines: &[String], fragment: &str) -> bool { + lines.iter().any(|line| line.contains(fragment)) +} + +/// The items `migration` leaves to do besides naming a permission policy, +/// which a document written for main always leaves. +fn left_to_do(migration: &Migration) -> Vec<&str> { + migration + .report + .to_do + .iter() + .map(String::as_str) + .filter(|line| *line != POLICY_TO_DO) + .collect() +} + +/// A document written for main, migrated, with the publisher's permission +/// policy file named, as the guide says to do after migrating. It is +/// migrated once more, so it is a document the command changes nothing in. +fn migrated_main_naming_a_policy() -> String { + let migrated = migrate(MAIN_BASE, Adapter::Fastly) + .expect("should migrate") + .text; + migrate( + &format!("{migrated}\n[permissions]\npolicy = \"standard\"\n"), + Adapter::Fastly, + ) + .expect("should migrate the named document") + .text +} + +#[test] +fn publisher_origin_settings_move_to_the_origin_source() { + let input = r#"[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +# Where the pages come from. +origin_url = "https://origin.publisher.example.com" +origin_host_header_override = "www.publisher.example.com" +proxy_secret = "publisher_proxy_secret" +"#; + let migration = migrated(input); + let value = parsed(&migration.text); + + assert_eq!(value["source"]["module"].as_str(), Some("origin")); + assert_eq!( + value["source"]["origin"]["url"].as_str(), + Some("https://origin.publisher.example.com") + ); + assert_eq!( + value["source"]["origin"]["host_header"].as_str(), + Some("www.publisher.example.com") + ); + let publisher = value["publisher"] + .as_table() + .expect("should keep [publisher]"); + assert!(!publisher.contains_key("origin_url")); + assert!(!publisher.contains_key("origin_host_header_override")); + assert_eq!( + publisher["proxy_secret"].as_str(), + Some("publisher_proxy_secret") + ); + assert!( + migration.text.contains( + "# Where the pages come from.\nurl = \"https://origin.publisher.example.com\"" + ), + "should keep the comment above the moved setting:\n{}", + migration.text + ); + assert!(reported( + &migration.report.changed, + "moved `[publisher] origin_url` to `[source.origin] url`" + )); +} + +#[test] +fn a_path_in_the_origin_address_is_reported_to_check() { + let migration = + migrated(&MAIN_BASE.replace("origin.publisher.example.com", "storage.example.com/site")); + + assert!(reported( + &migration.report.check, + "`[source.origin] url` has a path" + )); + let plain = migrated(MAIN_BASE); + assert!(!reported(&plain.report.check, "has a path")); +} + +#[test] +fn a_host_header_override_is_reported_to_check() { + let migration = migrated(&MAIN_BASE.replacen( + "proxy_secret =", + "origin_host_header_override = \"www.publisher.example.com\"\nproxy_secret =", + 1, + )); + + assert!( + reported( + &migration.report.check, + "`[source.origin] host_header` is set, so the addresses a page gives under that host" + ), + "{:?}", + migration.report.check + ); + let plain = migrated(MAIN_BASE); + assert!(!reported(&plain.report.check, "host_header")); +} + +#[test] +fn an_asset_route_origin_moves_to_its_source_with_the_signer_renamed() { + let input = main_document( + r#"[proxy] + +[[proxy.asset_routes]] +prefix = "/.image/" +origin_url = "https://assets.publisher.example.com" +path_pattern = "^/.image/(.*)$" +target_path = "/image/$1" + +[proxy.asset_routes.auth] +type = "s3_sig_v4" +region = "us-east-1" +access_key_id = "access_key_id" +secret_access_key = "secret_access_key" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + let route = &value["proxy"]["asset_routes"][0]; + + assert_eq!(route["prefix"].as_str(), Some("/.image/")); + assert!(route.get("origin_url").is_none()); + assert!(route.get("auth").is_none()); + assert_eq!(route["source"]["module"].as_str(), Some("origin")); + assert_eq!( + route["source"]["origin"]["url"].as_str(), + Some("https://assets.publisher.example.com") + ); + assert_eq!( + route["source"]["origin"]["auth"]["type"].as_str(), + Some("s3_sigv4") + ); + assert_eq!( + route["source"]["origin"]["auth"]["region"].as_str(), + Some("us-east-1") + ); + assert!(reported( + &migration.report.changed, + "old spelling of the S3 signer" + )); +} + +#[test] +fn a_publisher_origin_auth_table_moves_with_the_origin() { + let input = r#"[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +origin_url = "https://storage.publisher.example.com/bucket" +proxy_secret = "publisher_proxy_secret" +origin_auth = { type = "s3_sig_v4", region = "us-east-1" } +"#; + let migration = migrated(input); + let value = parsed(&migration.text); + + assert_eq!( + value["source"]["origin"]["auth"]["type"].as_str(), + Some("s3_sigv4") + ); + assert!(value["publisher"].get("origin_auth").is_none()); +} + +#[test] +fn admin_handlers_are_removed_and_any_other_is_left_to_do() { + let input = main_document( + r#"[[handlers]] +path = "^/_ts/admin" +username = "admin" +password = "handler_password" + +[[handlers]] +path = "^/secure" +username = "user" +password = "user_password" + +[[handlers]] +path = "^/_ts/admin|^/members" +username = "user" +password = "user_password" + +[[handlers]] +path = "^/admin/keys(?:/|$)" +username = "admin" +password = "handler_password" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + let handlers = value["handlers"] + .as_array() + .expect("should keep two handlers"); + + assert_eq!(handlers.len(), 2); + assert_eq!(handlers[0]["path"].as_str(), Some("^/secure")); + assert_eq!(handlers[1]["path"].as_str(), Some("^/_ts/admin|^/members")); + assert!(reported( + &migration.report.changed, + "removed `[[handlers]]` entry 1" + )); + assert!(reported( + &migration.report.changed, + "removed `[[handlers]]` entry 4" + )); + assert!(reported(&migration.report.to_do, "`[[handlers]]` entry 2")); + assert!(reported(&migration.report.to_do, "`[[handlers]]` entry 3")); +} + +/// A document written for main with one handler guarding `pattern`. +fn guarded(pattern: &str) -> String { + let path = toml::Value::String(pattern.to_owned()); + main_document(&format!( + "[[handlers]]\npath = {path}\nusername = \"admin\"\npassword = \"handler_password\"\n" + )) +} + +/// Request paths that tell apart what a handler pattern guarded on main. +const PROBE_PATHS: [&str; 16] = [ + "/", + "/admin", + "/admin/keys", + "/admin/keys/", + "/admin/keys/rotate", + "/admin/keys-old", + "/admin/keystore", + "/admin/keysx", + "/members", + "/members/_ts/admin", + "/_ts/admi", + "/_ts/admit", + "/_ts/admin", + "/_ts/admin/", + "/_ts/admin/ec", + "/_ts/adminx", +]; + +/// The probe paths a handler on `pattern` guarded on main that this release +/// forwards to the origin. Main searched the path with the `regex` crate, so +/// a match anywhere in it counted. +fn forwarded_paths_main_guarded(pattern: &str) -> Vec<&'static str> { + let regex = Regex::new(pattern).expect("should compile as main compiled it"); + PROBE_PATHS + .into_iter() + .filter(|path| regex.is_match(path)) + .filter(|path| { + let request = Request::builder() + .uri(format!("https://publisher.example.com{path}")) + .body(EdgeBody::empty()) + .expect("should build a request"); + closed_path_response(&request).is_none() + }) + .collect() +} + +/// Checks that the handler on `pattern` is kept and left to do. +fn assert_left_to_do(pattern: &str) { + let migration = migrated(&guarded(pattern)); + let value = parsed(&migration.text); + let kept = value + .get("handlers") + .and_then(|handlers| handlers.get(0)) + .and_then(|handler| handler.get("path")) + .and_then(toml::Value::as_str); + assert_eq!( + kept, + Some(pattern), + "should keep the handler on `{pattern}`:\n{}", + migration.text + ); + assert!( + reported( + &migration.report.to_do, + "`[[handlers]]` entry 1 can guard paths beyond the `/_ts/admin` and \ + `/admin/keys` paths this release closes" + ), + "`{pattern}`: {:?}", + migration.report.to_do + ); +} + +#[test] +fn a_handler_is_removed_only_where_every_path_it_matches_is_closed() { + for pattern in [ + "^/_ts/admin", + "^/_ts/admin/", + "^/_ts/admin+", + "^/admin/keys/", + "^/admin/keys/+", + "^/admin/keys$", + "^/admin/keys(?:/|$)", + "^/admin/keys(/|$)", + ] { + let forwarded = forwarded_paths_main_guarded(pattern); + assert!( + forwarded.is_empty(), + "`{pattern}` guarded {forwarded:?} on main, which this release forwards" + ); + let migration = migrated(&guarded(pattern)); + assert!( + parsed(&migration.text).get("handlers").is_none(), + "should remove the handler on `{pattern}`:\n{}", + migration.text + ); + assert!( + left_to_do(&migration).is_empty(), + "`{pattern}`: {:?}", + migration.report.to_do + ); + } + // Main matched a pattern anywhere in the path, so `^/admin/keys` also + // guarded `/admin/keystore`, which the stack forwards to the origin, and + // so does `^/admin/keys/?`, whose quantifier lets the bound match + // nothing. `^/_ts/admin?` matches `/_ts/admit` the same way. + for pattern in [ + "^/admin/keys", + "^/admin/keys.*", + "^/admin/keysx", + "^/admin", + "/_ts/admin", + "^/admin/keys(?:/|$)|^/members", + "^/admin/keys/?", + "^/admin/keys/*", + "^/admin/keys/{0}", + "^/admin/keys$?", + "^/admin/keys(?:/|$)?", + "^/admin/keys(/|$)??", + "^/_ts/admin?", + "^/_ts/admin*", + "^/_ts/admin{0,1}", + ] { + assert!( + !forwarded_paths_main_guarded(pattern).is_empty(), + "`{pattern}` should guard a path this release forwards" + ); + assert_left_to_do(pattern); + } + // The command reads what follows the bound and not what the whole + // pattern means, so a quantifier there leaves the handler to do even + // where every path it matches is closed. + assert!(forwarded_paths_main_guarded("^/admin/keys/?$").is_empty()); + assert_left_to_do("^/admin/keys/?$"); +} + +#[test] +fn a_handler_with_a_class_or_comment_after_its_prefix_is_left_to_do() { + // The `(` in a class, or in a comment the `(?x)` flag allows, opens no + // group, so each pattern ends in an alternative that guarded `/members`. + for pattern in [ + "^/_ts/admin[](]|^/members", + "^/_ts/admin[[:alpha:](]|^/members", + "^/_ts/admin(?x)#(\n|^/members", + ] { + assert!( + forwarded_paths_main_guarded(pattern).contains(&"/members"), + "`{pattern}` should guard `/members`" + ); + assert_left_to_do(pattern); + } + // The command does not read classes, so it leaves one to do even where + // every path the pattern matches is closed. + assert!(forwarded_paths_main_guarded("^/_ts/admin/[a-z]+").is_empty()); + assert_left_to_do("^/_ts/admin/[a-z]+"); +} + +#[test] +fn request_signing_store_ids_are_removed() { + let input = main_document( + r#"[request_signing] +enabled = true +config_store_id = "config-store-id" +secret_store_id = "secret-store-id" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert_eq!(value["request_signing"]["enabled"].as_bool(), Some(true)); + assert!(value["request_signing"].get("config_store_id").is_none()); + assert!(value["request_signing"].get("secret_store_id").is_none()); + assert!(reported(&migration.report.check, "ts keys rotate")); +} + +#[test] +fn the_ec_passphrase_moves_to_the_hmac_table() { + let input = r#"[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +origin_url = "https://origin.publisher.example.com" +proxy_secret = "publisher_proxy_secret" + +[ec] +# The secret the Edge Cookie is made with. +passphrase = "ec_passphrase" +ec_store = "ec_identity_store" +"#; + let migration = migrated(input); + let value = parsed(&migration.text); + + assert_eq!(value["ec"]["module"].as_str(), Some("hmac")); + assert_eq!( + value["ec"]["hmac"]["passphrase"].as_str(), + Some("ec_passphrase") + ); + assert!(value["ec"].get("passphrase").is_none()); + assert_eq!(value["ec"]["ec_store"].as_str(), Some("ec_identity_store")); + assert!( + migration + .text + .contains("[ec.hmac]\n# The secret the Edge Cookie is made with.\npassphrase"), + "should keep the comment above the passphrase:\n{}", + migration.text + ); +} + +#[test] +fn ec_provider_tables_and_module_names_take_their_new_names() { + let input = r#"[ec] +module = "host-signals" + +[ec.providers.host-signals] +passphrase = "ec_host_signals_passphrase" +"#; + let migration = migrated(input); + let value = parsed(&migration.text); + + assert_eq!(value["ec"]["module"].as_str(), Some("host_signals")); + assert_eq!( + value["ec"]["host_signals"]["passphrase"].as_str(), + Some("ec_host_signals_passphrase") + ); + assert!(value["ec"].get("providers").is_none()); +} + +#[test] +fn permission_signal_sources_become_modules_with_the_new_names() { + let input = r#"[permission_signal] +sources = ["gpc", "gpp_sale_opt_out", "us_privacy", "tcf"] +"#; + let migration = migrated(input); + let value = parsed(&migration.text); + + assert!(value.get("permission_signal").is_none()); + let modules: Vec<&str> = value["permission-signal"]["modules"] + .as_array() + .expect("should write modules") + .iter() + .filter_map(toml::Value::as_str) + .collect(); + assert_eq!(modules, ["gpc", "gpp", "us-privacy", "tcf"]); +} + +#[test] +fn the_platform_device_module_is_spelled_platform() { + let migration = migrated("[device]\nmodule = \"fastly\"\n"); + let value = parsed(&migration.text); + + assert_eq!(value["device"]["module"].as_str(), Some("platform")); +} + +#[test] +fn auction_providers_move_to_demand_with_their_bidder_routes() { + let input = main_document( + r#"[auction] +enabled = false + +# The server-side demand. +[auction.providers.pbs-main] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://prebid.example.com/openrtb2/auction" +routing = "explicit" + +[auction.providers.pbs-main.profile_config] +debug = false +consent_forwarding = "both" + +[auction.providers.pbs-main.notifications] +suppress_all = false + +[auction.providers.aps-main] +protocol = "openrtb-2.6" +profile = "aps" +endpoint = "https://aps.example.com/e/pb/bid" +routing = "all_eligible" + +[auction.providers.aps-main.profile_config] +account_id = "example-aps-account-id" + +[auction.providers.plain] +protocol = "openrtb-2.6" +endpoint = "https://ssp.example.com/bid" + +[auction.bidders.example-bidder] +provider = "pbs-main" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + let demand = &value["demand"]; + + let modules: Vec<&str> = demand["modules"] + .as_array() + .expect("should select the demand sources") + .iter() + .filter_map(toml::Value::as_str) + .collect(); + assert_eq!(modules, ["pbs_main", "aps_main", "plain"]); + assert_eq!( + demand["pbs_main"]["implementation"].as_str(), + Some("auction.prebid-server") + ); + assert_eq!(demand["pbs_main"]["debug"].as_bool(), Some(false)); + assert_eq!( + demand["pbs_main"]["consent_forwarding"].as_str(), + Some("both") + ); + assert_eq!( + demand["pbs_main"]["notifications"]["suppress_all"].as_bool(), + Some(false) + ); + assert!(demand["pbs_main"].get("protocol").is_none()); + assert!(demand["pbs_main"].get("profile_config").is_none()); + assert_eq!( + demand["aps_main"]["implementation"].as_str(), + Some("auction.aps") + ); + assert_eq!( + demand["aps_main"]["account_id"].as_str(), + Some("example-aps-account-id") + ); + assert_eq!( + demand["plain"]["implementation"].as_str(), + Some("auction-protocol.openrtb") + ); + assert!(value["auction"].get("providers").is_none()); + let route = &value["auction"]["bidders"]["example-bidder"]; + assert_eq!(route["module"].as_str(), Some("pbs_main")); + assert!(route.get("provider").is_none()); + assert!( + migration + .text + .contains("# The server-side demand.\n[demand]"), + "should keep the comment above the first provider with [demand]:\n{}", + migration.text + ); +} + +#[test] +fn a_provider_profile_no_implementation_matches_is_left_to_do() { + let input = main_document( + r#"[auction.providers.odd] +protocol = "openrtb-2.6" +profile = "something-else" +endpoint = "https://ssp.example.com/bid" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert!(value["auction"]["providers"].get("odd").is_some()); + assert!(reported( + &migration.report.to_do, + "`[auction.providers.odd] profile`" + )); +} + +#[test] +fn a_profile_setting_named_as_one_of_the_providers_own_is_left_to_do() { + let input = main_document( + r#"[auction.providers.pbs-main] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://prebid.example.com/openrtb2/auction" + +[auction.providers.pbs-main.profile_config] +endpoint = "https://other.example.com/openrtb2/auction" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert!(value["auction"]["providers"].get("pbs-main").is_some()); + assert!(value.get("demand").is_none()); + assert!(reported( + &migration.report.to_do, + "`[auction.providers.pbs-main.profile_config]` holds a setting named as one of the \ + provider's own" + )); +} + +#[test] +fn both_forms_of_a_setting_are_left_to_do_as_written() { + let input = r#"[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +origin_url = "https://origin.publisher.example.com" +proxy_secret = "publisher_proxy_secret" + +[source] +module = "origin" + +[source.origin] +url = "https://origin.publisher.example.com" + +[ec] +module = "hmac" +passphrase = "ec_passphrase" + +[ec.hmac] +passphrase = "ec_passphrase" + +[ec.providers.hmac] +passphrase = "ec_passphrase" + +[permission_signal] +sources = ["gpc"] + +[permission-signal] +modules = ["gpc"] + +[auction] +mediator = "adserver_mock" + +[auction.providers.pbs-main] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://prebid.example.com/openrtb2/auction" + +[demand] +modules = [] + +[integrations.adserver_mock] +enabled = true +endpoint = "https://adserver.example.com/mediate" + +[adserver] +module = "iabtl" + +[ad-server] +module = "iabtl" + +[tinybird] +enabled = true +api_host = "api.us-east.tinybird.example" + +[analytics] +module = "tinybird" + +[proxy] + +[[proxy.asset_routes]] +prefix = "/.image/" +origin_url = "https://assets.publisher.example.com" + +[proxy.asset_routes.source] +module = "origin" +"#; + let migration = migrated(input); + let to_do = &migration.report.to_do; + + for fragment in [ + "`[publisher]` holds origin settings and `[source]` is written too", + "`[ec] passphrase` sits beside an Edge Cookie module of the new form", + "`[ec.providers.hmac]` and `[ec.hmac]` are both written", + "`[permission_signal]` and `[permission-signal]` are both written", + "`[auction.providers]` and `[demand]` are both written", + "`[auction] mediator` and `[ad-server]` are both written", + "`[adserver]` and `[ad-server]` are both written", + "`[tinybird]` and `[analytics]` are both written", + "`[[proxy.asset_routes]]` entry 1 holds origin settings and a `source` table too", + ] { + assert!( + reported(to_do, fragment), + "should leave `{fragment}`: {to_do:?}" + ); + } + assert!( + !reported(to_do, "names an ad server whose settings"), + "should report the mediator once, beside the ad server it names: {to_do:?}" + ); + assert!( + migration.report.changed.is_empty(), + "should change nothing it left: {:?}", + migration.report.changed + ); + assert_eq!(migration.text, input); +} + +#[test] +fn the_integration_table_is_left_to_do() { + let migration = migrated("[integration]\nprovider = [\"didomi\"]\n"); + + assert!(reported( + &migration.report.to_do, + "`[integration]` is not read" + )); + assert_eq!(migration.text, "[integration]\nprovider = [\"didomi\"]\n"); +} + +#[test] +fn an_enabled_integration_moves_to_the_section_of_its_type() { + let input = main_document( + r#"# Google's ad tag. +[integrations.gpt] +enabled = true +# Attribution. +gam_attribution_enabled = false +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert!(value.get("integrations").is_none()); + let modules = value["ad-tag"]["modules"] + .as_array() + .expect("should select"); + assert_eq!(modules[0].as_str(), Some("google")); + assert_eq!( + value["ad-tag"]["google"]["gam_attribution_enabled"].as_bool(), + Some(false) + ); + assert!(value["ad-tag"]["google"].get("enabled").is_none()); + assert!( + migration + .text + .contains("# Google's ad tag.\n[ad-tag]\nmodules = [\"google\"]"), + "should write the section where the table was, under its comment:\n{}", + migration.text + ); + assert!( + migration + .text + .contains("# Attribution.\ngam_attribution_enabled = false") + ); +} + +#[test] +fn an_integration_that_was_not_enabled_is_commented_out_under_its_new_name() { + let input = main_document( + r#"# The consent prompt. +[integrations.didomi] +enabled = false +geo_query_parameters = false +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert!(value.get("integrations").is_none()); + assert!(value.get("cmp").is_none()); + assert!( + migration + .text + .contains("# The consent prompt.\n# [cmp.didomi]\n# geo_query_parameters = false\n"), + "should comment the table out under its new name:\n{}", + migration.text + ); + assert!(reported( + &migration.report.changed, + "commented out `[integrations.didomi]` as `[cmp.didomi]`" + )); +} + +#[test] +fn an_integration_main_ran_by_default_runs_without_an_enabled_line() { + let input = main_document( + r#"[integrations.lockr] +app_id = "lockr-app" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert_eq!(value["identity"]["module"].as_str(), Some("lockr")); + assert_eq!( + value["identity"]["lockr"]["app_id"].as_str(), + Some("lockr-app") + ); +} + +#[test] +fn a_module_with_no_settings_is_selected_with_no_table() { + let input = main_document( + r#"[integrations.osano] +enabled = true + +[integrations.gpt_diagnostics] +enabled = true +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert_eq!(value["cmp"]["module"].as_str(), Some("osano")); + assert!(value["cmp"].get("osano").is_none()); + let modules = value["ad-tag"]["modules"] + .as_array() + .expect("should select"); + assert_eq!(modules[0].as_str(), Some("google.diagnostics")); + assert!(value["ad-tag"].get("google").is_none()); +} + +#[test] +fn integrations_of_one_type_are_selected_together_in_the_builders_order() { + let input = main_document( + r#"[integrations.gpt_diagnostics] +enabled = true + +[integrations.sourcepoint] +enabled = true +rewrite_sdk = true + +[integrations.gpt] +enabled = true + +[integrations.didomi] +enabled = true +geo_query_parameters = false +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + let names = |section: &str| -> Vec { + value[section]["modules"] + .as_array() + .expect("should select several") + .iter() + .filter_map(toml::Value::as_str) + .map(str::to_owned) + .collect() + }; + assert_eq!(names("ad-tag"), ["google", "google.diagnostics"]); + assert_eq!(names("cmp"), ["didomi", "sourcepoint"]); + assert_eq!( + value["cmp"]["sourcepoint"]["rewrite_sdk"].as_bool(), + Some(true) + ); +} + +#[test] +fn prebid_and_testlight_are_selected_in_the_auction_section() { + let input = main_document( + r#"[auction] +enabled = true + +[integrations.testlight] +enabled = true +endpoint = "https://testlight.example.com/openrtb2/auction" + +[integrations.prebid] +timeout_ms = 1000 + +[integrations.prebid.bundle.modules] +bidder = ["rubiconBidAdapter"] +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + let auction = &value["auction"]; + + let modules: Vec<&str> = auction["modules"] + .as_array() + .expect("should select") + .iter() + .filter_map(toml::Value::as_str) + .collect(); + assert_eq!(modules, ["prebid", "testing.testlight"]); + assert_eq!( + auction["testing"]["testlight"]["endpoint"].as_str(), + Some("https://testlight.example.com/openrtb2/auction") + ); + assert_eq!(auction["prebid"]["timeout_ms"].as_integer(), Some(1000)); + assert_eq!( + auction["prebid"]["bundle"]["modules"]["bidder"][0].as_str(), + Some("rubiconBidAdapter") + ); + assert_eq!(auction["enabled"].as_bool(), Some(true)); +} + +#[test] +fn the_script_proxy_is_selected_in_the_proxy_section_with_its_assets() { + let input = main_document( + r#"[integrations.js_asset_proxy] +enabled = true + +[[integrations.js_asset_proxy.assets]] +path = "/assets/vendor.js" +origin_url = "https://cdn.example.com/vendor.js" +proxy = "enabled" + +[proxy] +certificate_check = true +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert_eq!( + value["proxy"]["modules"][0].as_str(), + Some("js_asset_proxy") + ); + assert_eq!(value["proxy"]["certificate_check"].as_bool(), Some(true)); + assert_eq!( + value["proxy"]["js_asset_proxy"]["assets"][0]["path"].as_str(), + Some("/assets/vendor.js") + ); +} + +#[test] +fn an_integration_no_module_replaces_is_left_to_do() { + let input = main_document("[integrations.unknown]\nenabled = true\n"); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert!(value["integrations"].get("unknown").is_some()); + assert!(reported( + &migration.report.to_do, + "`[integrations.unknown]` names no module" + )); +} + +#[test] +fn an_enabled_line_that_is_not_a_boolean_is_left_to_do() { + let input = main_document("[integrations.lockr]\nenabled = \"yes\"\napp_id = \"lockr\"\n"); + let migration = migrated(&input); + + assert!(reported( + &migration.report.to_do, + "`[integrations.lockr] enabled` is not true or false" + )); + assert!( + parsed(&migration.text)["integrations"] + .get("lockr") + .is_some() + ); +} + +#[test] +fn the_aps_rendering_mode_moves_into_each_aps_demand_source() { + let input = main_document( + r#"[auction.providers.aps-main] +protocol = "openrtb-2.6" +profile = "aps" +endpoint = "https://aps.example.com/e/pb/bid" + +[auction.providers.aps-main.profile_config] +account_id = "example-aps-account-id" + +[integrations.aps] +enabled = true +rendering_mode = "publisher_native" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert_eq!( + value["demand"]["aps_main"]["rendering_mode"].as_str(), + Some("publisher_native") + ); + assert!(value.get("integrations").is_none()); +} + +#[test] +fn an_aps_rendering_mode_with_no_aps_source_is_left_to_do() { + let input = main_document( + "[integrations.aps]\nenabled = true\nrendering_mode = \"publisher_native\"\n", + ); + let migration = migrated(&input); + + assert!(reported( + &migration.report.to_do, + "`[integrations.aps] rendering_mode` belongs in the table" + )); + assert!(parsed(&migration.text)["integrations"].get("aps").is_some()); + + let disabled = migrated(&main_document( + "[integrations.aps]\nenabled = false\nrendering_mode = \"publisher_native\"\n", + )); + assert!(parsed(&disabled.text).get("integrations").is_none()); + assert!(reported( + &disabled.report.check, + "main did not apply `[integrations.aps] rendering_mode`" + )); +} + +#[test] +fn the_mediator_and_the_mock_ad_server_become_the_ad_server() { + let input = main_document( + r#"[auction] +enabled = true +mediator = "adserver_mock" + +[integrations.adserver_mock] +enabled = true +endpoint = "https://adserver.example.com/mediate" + +[integrations.adserver_mock.context_query_params] +example_segments = "segments" +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert_eq!(value["ad-server"]["module"].as_str(), Some("iabtl")); + assert_eq!( + value["ad-server"]["iabtl"]["endpoint"].as_str(), + Some("https://adserver.example.com/mediate") + ); + assert_eq!( + value["ad-server"]["iabtl"]["context_query_params"]["example_segments"].as_str(), + Some("segments") + ); + assert!(value["auction"].get("mediator").is_none()); + assert!(value.get("integrations").is_none()); +} + +#[test] +fn a_mediator_with_no_mock_ad_server_is_left_to_do_and_an_unnamed_one_is_commented_out() { + let lone = migrated(&main_document("[auction]\nmediator = \"adserver_mock\"\n")); + assert!(reported( + &lone.report.to_do, + "`[auction] mediator` names an ad server" + )); + assert!(parsed(&lone.text)["auction"].get("mediator").is_some()); + + let unnamed = migrated(&main_document( + "[integrations.adserver_mock]\nenabled = true\nendpoint = \"https://adserver.example.com/mediate\"\n", + )); + assert!(parsed(&unnamed.text).get("ad-server").is_none()); + assert!(unnamed.text.contains("# [ad-server.iabtl]\n")); +} + +#[test] +fn the_ad_server_and_the_ads_txt_module_take_their_new_names() { + let input = r#"[adserver] +module = "mock" + +[adserver.mock] +endpoint = "https://adserver.example.com/decide" + +[ads-txt] +modules = ["sellers"] + +[ads-txt.sellers] +lines = ["ssp.example, 12345, DIRECT"] + +[[fetch]] +media_type = "text/plain" +path = "/ads.txt" +middleware = ["ads-txt.sellers"] +"#; + let migration = migrated(input); + let value = parsed(&migration.text); + + assert!(value.get("adserver").is_none()); + assert_eq!(value["ad-server"]["module"].as_str(), Some("iabtl")); + assert!(value["ad-server"].get("iabtl").is_some()); + assert_eq!(value["ads-txt"]["modules"][0].as_str(), Some("iabtl")); + assert!(value["ads-txt"].get("iabtl").is_some()); + assert_eq!( + value["fetch"][0]["middleware"][0].as_str(), + Some("ads-txt.iabtl") + ); + assert_eq!( + value["fetch"].as_array().map(Vec::len), + Some(1), + "should add no second entry for the file" + ); +} + +#[test] +fn a_selected_ads_txt_module_and_robots_txt_get_their_file_entries() { + let input = r#"[ads-txt] +modules = ["iabtl"] + +[ads-txt.iabtl] +lines = ["ssp.example, 12345, DIRECT"] + +[robots-txt] +modules = ["refuse_all"] +"#; + let migration = migrated(input); + let value = parsed(&migration.text); + let entries = value["fetch"].as_array().expect("should add the entries"); + + assert_eq!(entries.len(), 2); + assert_eq!(entries[0]["media_type"].as_str(), Some("text/plain")); + assert_eq!(entries[0]["path"].as_str(), Some("/ads.txt")); + assert_eq!(entries[0]["middleware"][0].as_str(), Some("ads-txt.iabtl")); + assert_eq!(entries[1]["path"].as_str(), Some("/robots.txt")); + assert_eq!(entries[1]["middleware"][0].as_str(), Some("robots_txt")); +} + +#[test] +fn tinybird_moves_to_analytics_and_is_commented_out_when_not_enabled() { + let enabled = migrated(&main_document( + r#"[tinybird] +enabled = true +api_host = "api.us-east.tinybird.example" +auction_token_secret = "tinybird_auction_append_token" +"#, + )); + let value = parsed(&enabled.text); + assert_eq!(value["analytics"]["module"].as_str(), Some("tinybird")); + assert_eq!( + value["analytics"]["tinybird"]["api_host"].as_str(), + Some("api.us-east.tinybird.example") + ); + assert!(value["analytics"]["tinybird"].get("enabled").is_none()); + assert!(value.get("tinybird").is_none()); + assert!(reported( + &enabled.report.check, + "analytics.ad_reporting.measure_ad_performance" + )); + + let disabled = migrated(&main_document( + "[tinybird]\nenabled = false\napi_host = \"api.us-east.tinybird.example\"\n", + )); + let value = parsed(&disabled.text); + assert!(value.get("analytics").is_none()); + assert!(value.get("tinybird").is_none()); + assert!( + disabled + .text + .contains("# [analytics.tinybird]\n# api_host = \"api.us-east.tinybird.example\"\n") + ); +} + +#[test] +fn the_mediator_response_option_is_renamed() { + let input = main_document( + r#"[debug.auction_html_comment_options] +include_mediator_response = false +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + let options = &value["debug"]["auction_html_comment_options"]; + + assert_eq!(options["include_adserver_response"].as_bool(), Some(false)); + assert!(options.get("include_mediator_response").is_none()); +} + +#[test] +fn the_consent_place_lists_are_left_to_do() { + let input = main_document( + r#"[consent] +mode = "interpreter" + +[consent.gdpr] +applies_in = ["FR"] + +[consent.us_states] +privacy_states = ["CA"] +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + + assert!(value["consent"].get("gdpr").is_some()); + assert!(reported( + &migration.report.to_do, + "`[consent.gdpr]` is not read" + )); + assert!(reported( + &migration.report.to_do, + "`[consent.us_states]` is not read" + )); +} + +#[test] +fn a_document_written_for_main_that_names_its_policy_migrates_as_main() { + let without = migrated(MAIN_BASE); + let with = migrated(&main_document("[permissions]\npolicy = \"standard\"\n")); + + assert_eq!( + with.report.changed, without.report.changed, + "should make the same rewrites" + ); + assert_eq!( + left_to_do(&with), + left_to_do(&without), + "should leave the same items apart from naming a policy" + ); + assert!( + without.report.to_do.iter().any(|line| line == POLICY_TO_DO) + && !with.report.to_do.iter().any(|line| line == POLICY_TO_DO), + "only the document naming no policy should leave naming one to do" + ); + assert_eq!(with.report.check, without.report.check); + + let mut named = parsed(&with.text); + let permissions = named + .as_table_mut() + .and_then(|root| root.remove("permissions")) + .expect("should keep the policy's table"); + assert_eq!( + permissions.get("policy").and_then(toml::Value::as_str), + Some("standard") + ); + assert_eq!( + named, + parsed(&without.text), + "should migrate the rest as the same document without the policy" + ); + // The migrated document holds `[source]`, which main did not read, so + // it is not taken as written for main again. + assert!(named.get("source").is_some()); + assert!(validate_source_config(&with.text).is_ok()); +} + +#[test] +fn a_document_that_names_no_policy_leaves_naming_one_to_do() { + let migration = migrated(MAIN_BASE); + + assert!( + migration + .report + .to_do + .iter() + .any(|line| line == POLICY_TO_DO), + "should leave naming a policy to do: {:?}", + migration.report.to_do + ); + assert!( + parsed(&migration.text).get("permissions").is_none(), + "should name no policy itself" + ); + assert!( + validate_source_config(&migration.text).is_ok(), + "the command's validation should run no policy check" + ); + + let named = migrated(&migrated_main_naming_a_policy()); + assert!( + !named.report.to_do.iter().any(|line| line == POLICY_TO_DO), + "a document naming a policy should leave nothing about it: {:?}", + named.report.to_do + ); +} + +#[test] +fn a_template_key_cookie_under_the_public_prefix_is_left_to_do() { + let input = main_document( + r#"[creative_opportunities] +gam_network_id = "123456789" +template_cache_key_cookies = ["ab_bucket", "ts-tester"] +"#, + ); + let migration = migrated(&input); + + assert!(reported( + &migration.report.to_do, + "`[creative_opportunities] template_cache_key_cookies` names a cookie under the `ts` prefix" + )); + let clean = migrated(&main_document( + "[creative_opportunities]\ngam_network_id = \"123456789\"\ntemplate_cache_key_cookies = [\"ab_bucket\"]\n", + )); + assert!(left_to_do(&clean).is_empty()); +} + +#[test] +fn a_document_for_main_keeps_the_location_lookup() { + let migration = migrated(MAIN_BASE); + let value = parsed(&migration.text); + + assert_eq!(value["geo"]["module"].as_str(), Some("platform")); + assert!(value.get("fetch").is_none()); + assert!(value.get("serve").is_none()); + assert!(reported(&migration.report.check, "`x-ts-version`")); + assert!(reported(&migration.report.check, "Flight response")); + assert!( + left_to_do(&migration).is_empty(), + "{:?}", + migration.report.to_do + ); +} + +#[test] +fn the_location_lookup_is_the_one_each_adapter_accepts() { + for adapter in [Adapter::Fastly, Adapter::Cloudflare] { + let migration = migrated_on(adapter, MAIN_BASE); + assert_eq!( + parsed(&migration.text)["geo"]["module"].as_str(), + Some("platform"), + "{adapter:?} should keep the host's lookup:\n{}", + migration.text + ); + assert!( + left_to_do(&migration).is_empty(), + "{adapter:?}: {:?}", + migration.report.to_do + ); + let settings = settings_of(&migration.text); + settings + .geo + .validate_jurisdiction_acknowledgment(&settings.ec) + .unwrap_or_else(|error| panic!("{adapter:?} should pass the startup check: {error:?}")); + } + // Neither host has a lookup, so each refuses `platform` at startup, and + // main's lookup there located no request. + for adapter in [Adapter::Axum, Adapter::Spin] { + let migration = migrated_on(adapter, MAIN_BASE); + assert!( + parsed(&migration.text).get("geo").is_none(), + "{adapter:?} has no lookup to select:\n{}", + migration.text + ); + assert!( + reported( + &migration.report.to_do, + "no geo module is selected, because" + ), + "{adapter:?}: {:?}", + migration.report.to_do + ); + let settings = settings_of(&migration.text); + assert!( + settings + .geo + .validate_jurisdiction_acknowledgment(&settings.ec) + .is_err(), + "{adapter:?}: startup should refuse what the item left to do names" + ); + + let done = format!( + "{}\n[geo]\nassume_single_jurisdiction = true\n", + migration.text + ); + let settings = settings_of(&done); + settings + .geo + .validate_jurisdiction_acknowledgment(&settings.ec) + .unwrap_or_else(|error| panic!("{adapter:?} should pass the startup check: {error:?}")); + let again = migrate(&done, adapter).expect("should migrate the document"); + assert!( + left_to_do(&again).is_empty() && again.report.changed.is_empty(), + "{adapter:?}: {:?}", + again.report + ); + } +} + +#[test] +fn the_location_settings_startup_refuses_are_left_to_do_in_any_document() { + let platform = "[geo]\nmodule = \"platform\"\n"; + for adapter in [Adapter::Axum, Adapter::Spin] { + let migration = migrated_on(adapter, platform); + assert!( + reported( + &migration.report.to_do, + "`[geo] module = \"platform\"` selects the host's location lookup" + ), + "{adapter:?}: {:?}", + migration.report.to_do + ); + assert_eq!(migration.text, platform); + } + for adapter in [Adapter::Fastly, Adapter::Cloudflare] { + let migration = migrated_on(adapter, platform); + assert!( + left_to_do(&migration).is_empty(), + "{adapter:?}: {:?}", + migration.report.to_do + ); + } + + let unlocated = "[ec]\nmodule = \"hmac\"\n\n[ec.hmac]\npassphrase = \"ec_passphrase\"\n"; + let acknowledged = format!("{unlocated}\n[geo]\nassume_single_jurisdiction = true\n"); + for adapter in ADAPTERS { + let migration = migrated_on(adapter, unlocated); + assert!( + reported( + &migration.report.to_do, + "`[ec]` selects an Edge Cookie module and" + ), + "{adapter:?}: {:?}", + migration.report.to_do + ); + let migration = migrated_on(adapter, &acknowledged); + assert!( + left_to_do(&migration).is_empty(), + "{adapter:?}: {:?}", + migration.report.to_do + ); + } +} + +#[test] +fn no_location_header_is_selected_for_the_operator() { + for adapter in ADAPTERS { + let migration = migrated_on(adapter, MAIN_BASE); + let value = parsed(&migration.text); + assert!( + value.get("headers").is_none() && value.get("serve").is_none(), + "{adapter:?} should select no location header:\n{}", + migration.text + ); + } + assert!(reported( + &migrated_on(Adapter::Fastly, MAIN_BASE).report.check, + "To keep them, select `[headers] modules = [\"geo\"]`" + )); + assert!(reported( + &migrated_on(Adapter::Cloudflare, MAIN_BASE).report.check, + "the one `x-geo-*` header main wrote on Cloudflare" + )); + for adapter in [Adapter::Axum, Adapter::Spin] { + assert!(reported( + &migrated_on(adapter, MAIN_BASE).report.check, + "which main wrote as `false`" + )); + } +} + +#[test] +fn the_region_the_cloudflare_lookup_now_reads_is_reported_to_check() { + // Main's Cloudflare lookup set no region, so its US state rules never + // applied there. + for adapter in ADAPTERS { + let migration = migrated_on(adapter, MAIN_BASE); + assert_eq!( + reported(&migration.report.check, "`cf-region-code`"), + adapter == Adapter::Cloudflare, + "{adapter:?}: {:?}", + migration.report.check + ); + } +} + +#[test] +fn a_document_already_in_the_new_shape_gets_no_location_lookup() { + let input = r#"[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +proxy_secret = "publisher_proxy_secret" + +[source] +module = "origin" + +[source.origin] +url = "https://origin.publisher.example.com" +"#; + let migration = migrated(input); + + assert!( + migration.report.changed.is_empty(), + "{:?}", + migration.report.changed + ); + assert!( + migration.report.check.is_empty(), + "{:?}", + migration.report.check + ); + assert_eq!(migration.text, input); +} + +/// The middleware names the stack's example lists in the commented entry for +/// `phase` and `media_type`. +fn example_entry(phase: &str, media_type: &str) -> Vec { + let start = format!("# [[{phase}]]\n# media_type = \"{media_type}\"\n# middleware = [\n"); + let at = STACK_EXAMPLE + .find(&start) + .unwrap_or_else(|| panic!("the example should list a {phase} entry for {media_type}")); + STACK_EXAMPLE[at + start.len()..] + .lines() + .take_while(|line| *line != "# ]") + .map(|line| { + line.trim_start_matches('#') + .trim() + .trim_end_matches(',') + .trim_matches('"') + .to_owned() + }) + .collect() +} + +#[test] +fn page_entries_name_the_middleware_in_the_order_the_example_shows() { + let input = main_document( + r#"[integrations.prebid] +enabled = true + +[integrations.js_asset_proxy] +enabled = true + +[integrations.testlight] +enabled = true +endpoint = "https://testlight.example.com/openrtb2/auction" + +[integrations.nextjs] +enabled = true + +[integrations.permutive] +organization_id = "organization" +workspace_id = "workspace" + +[integrations.lockr] +app_id = "lockr" + +[integrations.didomi] + +[integrations.sourcepoint] +enabled = true + +[integrations.google_tag_manager] +enabled = true +container_id = "GTM-ABCDEF" + +[integrations.datadome] +enabled = true + +[integrations.gpt] + +[integrations.gpt_diagnostics] +enabled = true +"#, + ); + let migration = migrated(&input); + let value = parsed(&migration.text); + let names = |entry: &toml::Value| -> Vec { + entry["middleware"] + .as_array() + .expect("should name middleware") + .iter() + .filter_map(toml::Value::as_str) + .map(str::to_owned) + .collect() + }; + let entry = |phase: &str, media_type: &str| -> Vec { + value[phase] + .as_array() + .expect("should add entries") + .iter() + .find(|entry| entry["media_type"].as_str() == Some(media_type)) + .map(names) + .unwrap_or_default() + }; + + let fetch = entry("fetch", "text/html"); + let expected: Vec = example_entry("fetch", "text/html") + .into_iter() + .filter(|name| name != "cmp.inmobi") + .collect(); + assert_eq!(fetch, expected); + + let serve = entry("serve", "text/html"); + let expected: Vec = example_entry("serve", "text/html") + .into_iter() + .filter(|name| { + ![ + "tag.google-tag", + "headers.geo", + "51degrees.client_hints", + "service_version", + ] + .contains(&name.as_str()) + }) + .collect(); + assert_eq!(serve, expected); + + assert_eq!( + entry("fetch", "text/x-component"), + ["framework.nextjs.flight"], + "the Next.js module should place its Flight middleware" + ); + assert!(!reported(&migration.report.check, "Flight response")); + assert!( + !reported(&migration.report.to_do, "does not link"), + "every main integration should name a module the tool links: {:?}", + migration.report.to_do + ); +} + +#[test] +fn a_document_in_the_stack_shape_with_no_page_entries_gets_none_added() { + let input = r#"[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +proxy_secret = "publisher_proxy_secret" + +[source] +module = "origin" + +[source.origin] +url = "https://origin.publisher.example.com" + +[cmp] +module = "didomi" +"#; + let migration = migrated(input); + + assert!( + migration.report.changed.is_empty(), + "should leave the page changes to the operator: {:?}", + migration.report.changed + ); + assert_eq!(migration.text, input); +} + +#[test] +fn a_document_with_page_entries_gets_none_added() { + let input = r#"[cmp] +module = "didomi" + +[[fetch]] +media_type = "text/html" +middleware = [] +"#; + let migration = migrated(input); + + assert!( + migration.report.changed.is_empty(), + "{:?}", + migration.report.changed + ); +} + +/// Main's example with its placeholder hosts replaced, as an operator's +/// document would have them. +fn main_example_with_hosts() -> String { + MAIN_EXAMPLE + .replace("\"example.com\"", "\"publisher.example.com\"") + .replace("\".example.com\"", "\".publisher.example.com\"") + .replace( + "https://origin.example.com", + "https://origin.publisher.example.com", + ) +} + +#[test] +fn mains_example_migrates_into_a_document_that_validates() { + let input = main_example_with_hosts(); + let migration = migrated(&input); + + validate_source_config(&migration.text).unwrap_or_else(|error| { + panic!( + "the migrated example should validate: {error}\n{}", + migration.text + ) + }); + assert!( + left_to_do(&migration).is_empty(), + "{:?}", + migration.report.to_do + ); + + let value = parsed(&migration.text); + assert!(value.get("integrations").is_none()); + assert!(value.get("handlers").is_none()); + assert_eq!(value["source"]["module"].as_str(), Some("origin")); + assert_eq!(value["ec"]["module"].as_str(), Some("hmac")); + assert_eq!(value["demand"]["modules"][0].as_str(), Some("pbs_main")); + assert_eq!( + value["auction"]["bidders"]["example-bidder"]["module"].as_str(), + Some("pbs_main") + ); + for block in [ + "# [proxy.js_asset_proxy]\n", + "# [[proxy.js_asset_proxy.assets]]\n", + "# [auction.prebid]\n# timeout_ms = 1000\n", + "# [cmp.didomi]\n# geo_query_parameters = false\n", + "# [bot-protection.datadome]\n", + "# [ad-tag.google]\n", + "# [tag.google-tag-manager]\n", + ] { + assert!( + migration.text.contains(block), + "should comment out `{block}`:\n{}", + migration.text + ); + } + assert!( + migration + .text + .starts_with("# trusted-server.example.toml from IABTechLab/trusted-server main"), + "should keep the comments that open the file" + ); +} + +#[test] +fn mains_example_validates_only_once_migrated() { + assert!( + validate_source_config(&main_example_with_hosts()).is_err(), + "main's example should not load as written" + ); +} + +#[test] +fn comments_and_table_order_survive() { + let input = r#"# The opening comment. + +# About the publisher. +[publisher] +# The domain. +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +# The origin. +origin_url = "https://origin.publisher.example.com" +proxy_secret = "publisher_proxy_secret" +# A trailing comment of the publisher. + +# About the Edge Cookie. +[ec] +# The passphrase. +passphrase = "ec_passphrase" +ec_store = "ec_identity_store" # the store + +# About the auction. +[auction] +enabled = false + +# About the templates. +[creative_opportunities] +gam_network_id = "123456789" + +# The closing comment. +"#; + let migration = migrated(input); + + for line in input.lines().filter(|line| line.contains('#')) { + assert!( + migration.text.contains(line), + "should keep `{line}`:\n{}", + migration.text + ); + } + let order: Vec = [ + "[publisher]", + "[source]", + "[ec]", + "[auction]", + "[creative_opportunities]", + ] + .iter() + .map(|header| { + migration + .text + .find(&format!("\n{header}\n")) + .unwrap_or_else(|| panic!("should write {header}:\n{}", migration.text)) + }) + .collect(); + assert!( + order.windows(2).all(|pair| pair[0] < pair[1]), + "should keep the tables in order:\n{}", + migration.text + ); + assert!( + migration.text.contains( + "proxy_secret = \"publisher_proxy_secret\"\n# A trailing comment of the publisher.\n" + ), + "should keep the trailing comment with the publisher:\n{}", + migration.text + ); + assert!(migration.text.ends_with("# The closing comment.\n")); +} + +#[test] +fn windows_line_endings_are_kept() { + let input = MAIN_BASE.replace('\n', "\r\n"); + let migration = migrated(&input); + + assert!(!migration.report.changed.is_empty()); + assert!( + !migration + .text + .bytes() + .zip(migration.text.bytes().skip(1)) + .any(|(before, after)| before != b'\r' && after == b'\n'), + "every line should end with a carriage return and a line feed:\n{:?}", + migration.text + ); + assert!(!migration.text.starts_with('\n')); +} + +/// A document every rule rewrites or reports, whose every value is a marker +/// no line may print. +const SECRET_DOCUMENT: &str = r#"[publisher] +domain = "SECRET-domain" +cookie_domain = "SECRET-cookie-domain" +origin_url = "https://SECRET-origin.example.com/SECRET-path" +origin_host_header_override = "SECRET-host" +proxy_secret = "SECRET-proxy" + +[ec] +passphrase = "SECRET-passphrase" + +[[ec.partners]] +name = "SECRET-partner" +source_domain = "SECRET-source" +api_token = "SECRET-token" + +[[handlers]] +path = "^/_ts/admin" +username = "SECRET-admin" +password = "SECRET-password" + +[[handlers]] +path = "^/SECRET-secure" +username = "SECRET-user" +password = "SECRET-user-password" + +[request_signing] +config_store_id = "SECRET-config-store" +secret_store_id = "SECRET-secret-store" + +[auction] +mediator = "SECRET-mediator" + +[auction.providers.pbs-main] +protocol = "openrtb-2.6" +profile = "SECRET-profile" +endpoint = "https://SECRET-pbs.example.com" + +[auction.providers.pbs-other] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://SECRET-pbs.example.com" + +[auction.providers.pbs-other.profile_config] +debug_query_params = "SECRET-debug" + +[auction.bidders.bidder] +provider = "pbs-other" + +[integrations.gpt] +script_url = "https://SECRET-gpt.example.com/gpt.js" + +[integrations.didomi] +enabled = false +sdk_origin = "https://SECRET-didomi.example.com" + +[integrations.aps] +enabled = true +rendering_mode = "SECRET-rendering" + +[integrations.SECRET-integration] +enabled = true + +[tinybird] +enabled = true +api_host = "SECRET-tinybird" +auction_token_secret = "SECRET-tinybird-token" + +[consent.gdpr] +applies_in = ["SECRET-country"] + +[creative_opportunities] +gam_network_id = "SECRET-network" +template_cache_key_cookies = ["ts-SECRET-cookie"] +"#; + +#[test] +fn no_line_carries_a_value_from_the_document() { + for adapter in ADAPTERS { + let migration = migrate(SECRET_DOCUMENT, adapter).expect("should migrate the document"); + let lines: Vec<&String> = migration + .report + .changed + .iter() + .chain(&migration.report.to_do) + .chain(&migration.report.check) + .collect(); + assert!(!lines.is_empty()); + for line in lines { + let shown = line.replace("SECRET-integration", ""); + assert!( + !shown.contains("SECRET"), + "a line printed a value from the document on {adapter:?}: {line}" + ); + } + } + + let temp = TempDir::new().expect("should make a folder"); + let file = temp.path().join("trusted-server.toml"); + fs::write(&file, SECRET_DOCUMENT).expect("should write the document"); + let mut out = Vec::new(); + let mut report = Vec::new(); + run_with_writers(&args(file.clone(), false, false), &mut out, &mut report).expect("should run"); + let report = String::from_utf8(report).expect("should report text"); + assert!( + !report.replace("SECRET-integration", "").contains("SECRET"), + "the report printed a value from the document:\n{report}" + ); + + fs::write(&file, "token = \"SECRET-token\"\n[broken\n").expect("should write the document"); + let error = run_with_writers(&args(file, false, false), &mut Vec::new(), &mut Vec::new()) + .expect_err("should refuse a document that is not TOML"); + assert!(error.contains("line 2"), "should name the line: {error}"); + assert!(!error.contains("SECRET"), "should quote nothing: {error}"); +} + +/// The command's arguments for `file` on Fastly. +fn args(file: std::path::PathBuf, write: bool, check: bool) -> ConfigMigrateArgs { + ConfigMigrateArgs { + file, + adapter: Adapter::Fastly, + write, + check, + } +} + +#[test] +fn check_changes_nothing_and_says_whether_the_document_needs_migrating() { + let temp = TempDir::new().expect("should make a folder"); + let file = temp.path().join("trusted-server.toml"); + fs::write(&file, MAIN_BASE).expect("should write the document"); + + let mut out = Vec::new(); + let mut report = Vec::new(); + let outcome = run_with_writers(&args(file.clone(), false, true), &mut out, &mut report) + .expect("should check"); + assert_eq!(outcome, RunOutcome::AssertionFailed); + assert!(out.is_empty(), "should print no document"); + assert_eq!( + fs::read_to_string(&file).expect("should read"), + MAIN_BASE, + "should leave the file as it was" + ); + let report = String::from_utf8(report).expect("should report text"); + assert!(report.contains("changed: "), "{report}"); + assert!( + report.ends_with("the document needs migrating\n"), + "{report}" + ); + + fs::write(&file, migrated_main_naming_a_policy()).expect("should write the migrated document"); + let mut report = Vec::new(); + let outcome = run_with_writers(&args(file, false, true), &mut Vec::new(), &mut report) + .expect("should check"); + assert_eq!(outcome, RunOutcome::Success); + let report = String::from_utf8(report).expect("should report text"); + assert!(report.contains("nothing to migrate"), "{report}"); + assert!( + report.ends_with("the document needs no migrating\n"), + "{report}" + ); +} + +#[test] +fn write_rewrites_the_file_and_the_default_prints_the_document() { + let temp = TempDir::new().expect("should make a folder"); + let file = temp.path().join("trusted-server.toml"); + let input = MAIN_BASE; + fs::write(&file, input).expect("should write the document"); + let expected = migrate(input, Adapter::Fastly) + .expect("should migrate") + .text; + + let mut out = Vec::new(); + let mut report = Vec::new(); + let outcome = run_with_writers(&args(file.clone(), false, false), &mut out, &mut report) + .expect("should migrate"); + assert_eq!(String::from_utf8(out).expect("should print text"), expected); + assert_eq!(fs::read_to_string(&file).expect("should read"), input); + // A document written for main names no policy, so naming one is left + // to do and the run fails, though the migrated document validates. + assert_eq!(outcome, RunOutcome::AssertionFailed); + let report = String::from_utf8(report).expect("should report text"); + assert!( + report.contains(&format!("to do: {POLICY_TO_DO}\n")), + "{report}" + ); + assert!( + report.ends_with("the migrated document passes validation\n"), + "{report}" + ); + + let mut out = Vec::new(); + run_with_writers(&args(file.clone(), true, false), &mut out, &mut Vec::new()) + .expect("should migrate"); + assert!(out.is_empty(), "should print no document with --write"); + assert_eq!(fs::read_to_string(&file).expect("should read"), expected); +} + +#[test] +fn an_item_left_to_do_fails_the_run() { + let temp = TempDir::new().expect("should make a folder"); + let file = temp.path().join("trusted-server.toml"); + fs::write( + &file, + main_document("[[handlers]]\npath = \"^/secure\"\nusername = \"u\"\npassword = \"p\"\n"), + ) + .expect("should write the document"); + + let mut report = Vec::new(); + let outcome = run_with_writers(&args(file, false, false), &mut Vec::new(), &mut report) + .expect("should migrate"); + assert_eq!(outcome, RunOutcome::AssertionFailed); + let report = String::from_utf8(report).expect("should report text"); + assert!(report.contains("to do: `[[handlers]]` entry 1"), "{report}"); + assert!( + report.contains("the migrated document does not pass validation"), + "{report}" + ); +} diff --git a/crates/trusted-server-cli/src/commands/config/mod.rs b/crates/trusted-server-cli/src/commands/config/mod.rs index f8bfce214e..8e062bb469 100644 --- a/crates/trusted-server-cli/src/commands/config/mod.rs +++ b/crates/trusted-server-cli/src/commands/config/mod.rs @@ -1,4 +1,5 @@ pub mod ad_templates; pub mod init; pub mod keyed_push; +pub mod migrate; pub mod policy; diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index 7b3a0265ea..3baaddb5c4 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -10,6 +10,7 @@ use crate::commands::audit::{AuditArgs, run_audit}; use crate::commands::config::ad_templates::{AdTemplatesCommand, run_ad_templates}; use crate::commands::config::init::{ConfigInitArgs, run_config_init}; use crate::commands::config::keyed_push; +use crate::commands::config::migrate::{ConfigMigrateArgs, run_config_migrate}; use crate::commands::config::policy; use crate::commands::pbs::{self, PbsArgs}; use crate::prebid_bundle::{NpmPrebidBundleGenerator, PrebidBundleArgs, run_bundle}; @@ -71,6 +72,9 @@ enum ConfigCommand { Diff(policy::DiffArgs), /// Reclaim orphaned chunk entries leaked from prior oversized pushes. Gc(ConfigGcArgs), + /// Rewrite a config written for an earlier release into the shape this + /// release reads. + Migrate(ConfigMigrateArgs), /// Push `trusted-server.toml`, with the permission policy file it names, /// as a blob envelope through `EdgeZero`, or under a Fastly `--key` of /// its own. @@ -148,6 +152,7 @@ fn dispatch(args: Args) -> Result { Command::Config(ConfigCommand::Gc(args)) => { edgezero_cli::run_config_gc(&args).map(|()| RunOutcome::Success) } + Command::Config(ConfigCommand::Migrate(args)) => run_config_migrate(&args), Command::Config(ConfigCommand::Push(args)) => match keyed_push::own_key(&args.edgezero) { Some(key) => { let stdout = std::io::stdout(); @@ -205,6 +210,7 @@ mod tests { use edgezero_cli::args::{AuthSub, ConfigDiffArgs, ConfigPushArgs, ConfigValidateArgs}; use super::*; + use crate::commands::config::migrate::Adapter; fn parse(args: &[&str]) -> Args { Args::try_parse_from(args).expect("should parse args") @@ -242,6 +248,51 @@ mod tests { .expect_err("should refuse both scopes at once"); } + #[test] + fn parses_config_migrate_with_its_file_its_adapter_and_one_mode() { + let args = parse(&[ + "ts", + "config", + "migrate", + "--adapter", + "spin", + "--check", + "trusted-server.toml", + ]); + let Command::Config(ConfigCommand::Migrate(migrate)) = args.command else { + panic!("expected config migrate command"); + }; + assert_eq!(migrate.file, PathBuf::from("trusted-server.toml")); + assert_eq!(migrate.adapter, Adapter::Spin); + assert!(migrate.check, "should check"); + assert!(!migrate.write, "should not write"); + + Args::try_parse_from([ + "ts", + "config", + "migrate", + "--adapter", + "fastly", + "--write", + "--check", + "trusted-server.toml", + ]) + .expect_err("should refuse writing and checking at once"); + Args::try_parse_from(["ts", "config", "migrate", "--adapter", "fastly"]) + .expect_err("should need the file to migrate"); + Args::try_parse_from(["ts", "config", "migrate", "trusted-server.toml"]) + .expect_err("should need the adapter the deployment runs on"); + Args::try_parse_from([ + "ts", + "config", + "migrate", + "--adapter", + "other", + "trusted-server.toml", + ]) + .expect_err("should refuse an adapter the stack does not have"); + } + #[test] fn parses_keys_rotate_with_its_stores_and_kid() { let args = parse(&[ diff --git a/crates/trusted-server-cli/tests/fixtures/main-trusted-server.example.toml b/crates/trusted-server-cli/tests/fixtures/main-trusted-server.example.toml new file mode 100644 index 0000000000..6b252d6024 --- /dev/null +++ b/crates/trusted-server-cli/tests/fixtures/main-trusted-server.example.toml @@ -0,0 +1,680 @@ +# trusted-server.example.toml from IABTechLab/trusted-server main at commit +# 5bc38862512c5c71085b5052bc3e528ee9b9987d, the input the ts config migrate +# golden test rewrites. +# ============================================================================= +# Trusted Server — application configuration template +# ============================================================================= +# +# This is the source-controlled starting point for an operator-owned +# `trusted-server.toml`. Copy it (`ts config init`), fill in the required +# values, and push it (`ts config push`) as an EdgeZero app-config blob. +# +# Only three sections are REQUIRED for the server to start and pass validation: +# 1. [[handlers]] covering /_ts/admin (admin authentication) +# 2. [publisher] (domain + origin) +# 3. [ec] passphrase (Edge Cookie identity secret) +# +# Everything below those is OPTIONAL. Most optional blocks are commented out — +# uncomment and edit one to enable it — but a few integrations are kept as active +# `enabled = false` stubs (see the Integrations section for why). All example +# hosts use `example.com`; replace them with your real endpoints in your private +# config, not in this template. +# +# `TRUSTED_SERVER__` env vars can override values when the `ts` CLI builds and +# validates this config for push (e.g. TRUSTED_SERVER__PUBLISHER__DOMAIN=...; +# nested keys use `__`). The overlay only replaces SCALAR leaves that already +# exist in the parsed TOML: a commented-out or absent key is silently ignored, +# and arrays or tables must be edited in TOML directly, not via env vars. The +# deployed runtime reads the pushed config blob, so these env vars do not change +# live behavior on their own. +# ============================================================================= + + +# ----------------------------------------------------------------------------- +# REQUIRED — Admin authentication +# ----------------------------------------------------------------------------- +# HTTP Basic-auth handler(s). At least one handler whose `path` regex covers +# the /_ts/admin endpoints is mandatory; startup fails without it. Each handler +# needs a non-placeholder username/password (deploy validation rejects the +# sample password below). +[[handlers]] +# Regex matched against the request path. This one guards the admin surface. +path = "^/_ts/admin" +username = "admin" +password = "handler_password" + +# You can add more handlers to basic-auth-protect other path prefixes. The +# sample password below is a known placeholder that deploy validation rejects, +# so it forces a real secret before push: +# [[handlers]] +# path = "^/secure" +# username = "user" +# password = "replace-with-admin-password" + + +# ----------------------------------------------------------------------------- +# REQUIRED — Publisher / origin +# ----------------------------------------------------------------------------- +[publisher] +# Public domain Trusted Server is fronting. +domain = "example.com" +# Cookie scope for first-party identity cookies (leading dot = include subdomains). +cookie_domain = ".example.com" +# Upstream origin to proxy publisher content from. No trailing slash. +origin_url = "https://origin.example.com" +# Secret Store key for the HMAC secret used to sign first-party proxy URLs. +proxy_secret = "publisher_proxy_secret" +# Optional: override the outbound Host header sent to origin_url. +# origin_host_header_override = "www.example.com" +# Optional: max bytes buffered when a response is post-processed in full (HTML +# rewriting/injection) instead of streamed. Default 16 MiB; larger responses +# return 502. Raise for deployments serving very large publisher pages. +# max_buffered_body_bytes = 16777216 # 16 MiB + + +# ----------------------------------------------------------------------------- +# REQUIRED — Edge Cookie (EC) identity +# ----------------------------------------------------------------------------- +[ec] +# Secret Store key for the secret used to derive EC identifiers. +passphrase = "ec_passphrase" +# KV store that persists EC identity state. This is the physical store name +# bound per adapter (e.g. `ec_identity_store` in fastly.toml); edgezero.toml's +# logical KV id is `trusted_server_kv`. +ec_store = "ec_identity_store" +# Max concurrent partner pull-sync requests. +pull_sync_concurrency = 3 +# Optional cluster-heuristic tuning (defaults shown): +# cluster_trust_threshold = 10 # entries with cluster_size <= this are individual users +# cluster_recheck_secs = 3600 # re-evaluate cluster_size after this many seconds + +# Example partner configuration. Provision referenced keys in +# trusted_server_secrets before validating/pushing. +# [[ec.partners]] +# name = "Example Partner" +# source_domain = "partner.example.com" +# OpenRTB source.atype (default 3); vendor-specific values are supported +# (PAIR uses 571187). +# openrtb_atype = 3 +# include this partner's UIDs in auction user.eids +# bidstream_enabled = true +# Only for inbound identify or batch-sync API access: +# api_token = "partner_api_token" +# Required when pull sync is enabled: +# ts_pull_token = "partner_ts_pull_token" +# batch_rate_limit = 60 # max batch-sync requests/min (default 60) +# pull_sync_enabled = false # default false + + +# ============================================================================= +# OPTIONAL — Core features (disabled/omitted by default) +# ============================================================================= + +# Custom headers added to every response (e.g. X-Robots-Tag: noindex). +# [response_headers] +# X-Robots-Tag = "noindex" + +# Sign outbound OpenRTB/API requests. These are platform management-API store +# IDs used for key-rotation WRITES; the runtime reads keys from fixed store names +# (`jwks_store` / `signing_keys`), not these. To keep signing OFF, leave this +# whole block commented out. If you uncomment it - even with `enabled = false` - +# both store IDs are required and must be REAL values: the key rotate/deactivate +# admin routes read them regardless of `enabled`, so deploy validation rejects +# the placeholders below (replace them before pushing). +# [request_signing] +# enabled = false +# config_store_id = "" +# secret_store_id = "" + +# First-party HTML/CSS rewriting controls. +# [rewrite] +# Domains left as-is (not proxied/rewritten). Supports "*.example.com" wildcards. +# exclude_domains = ["*.cdn.example.com"] + +# Trust a fronting CDN's reader IP only when the same request also supplies +# the matching shared secret. The front door must set both headers on requests +# it sends here and remove client-supplied copies on its other routes. +# `shared_secret` is a key in trusted_server_secrets, not the credential itself. +# Fastly-only; see docs/guide/fastly.md for the required front-door setup. +# [trusted_client_ip] +# ip_header = "x-ts-client-ip" +# auth_header = "x-ts-client-ip-auth" +# shared_secret = "trusted_client_ip_shared_secret" + +# Tester-cookie endpoints: GET /_ts/set-tester sets ts-tester=true and +# GET /_ts/clear-tester clears it on publisher.cookie_domain. +# max_age_seconds is the cookie lifetime; it defaults to 30 days. +# [tester_cookie] +# enabled = false +# max_age_seconds = 2592000 + +# Consent forwarding. All values below are the defaults — uncomment to override. +# [consent] +# mode = "interpreter" # "interpreter" (decode + forward) or "proxy" (raw passthrough) +# check_expiration = true # check TCF consent freshness +# max_consent_age_days = 395 # max age before consent is treated as expired (~13 months) +# +# [consent.gdpr] +# applies_in = ["AT","BE","BG","HR","CY","CZ","DK","EE","FI","FR","DE","GR","HU","IE","IT","LV","LT","LU","MT","NL","PL","PT","RO","SK","SI","ES","SE","IS","LI","NO","GB"] +# +# [consent.us_states] +# privacy_states = ["CA","VA","CO","CT","UT","MT","OR","TX","FL","DE","IA","NE","NH","NJ","TN","MN","MD","IN","KY","RI"] +# +# [consent.us_privacy_defaults] +# notice_given = true # has the publisher shown CCPA notice? +# lspa_covered = false # is the publisher subject to LSPA? +# gpc_implies_optout = true # should Sec-GPC: 1 trigger opt-out? +# +# [consent.conflict_resolution] +# mode = "restrictive" # "restrictive" | "newest" | "permissive" +# freshness_threshold_days = 30 + +[integrations.js_asset_proxy] +enabled = false +# Uncomment to override upstream cache headers for every asset below. +# This replaces upstream directives, including private and no-store. +# Use only when each asset's bytes are identical for every visitor. +# cache_ttl_seconds = 3600 +# Asset fetches use a fixed TrustedServer/1.0 User-Agent. Do not proxy assets +# that vary by browser User-Agent or use integrity hashes for UA-specific bytes. + +[[integrations.js_asset_proxy.assets]] +path = "/assets/example-vendor-loader.js" +origin_url = "https://cdn.example.com/vendor-loader.js" +proxy = "enabled" + +# Proxy behavior and first-party asset routing. Kept active with defaults. +[proxy] +# Verify TLS certs when proxying to HTTPS origins (default true; false only for +# local self-signed dev). +# certificate_check = true +# Allowlist for first-party proxy signing, initial fetch targets, and redirect +# targets. Supports exact ("example.com") and wildcard ("*.example.com", also +# matches apex). +# REQUIRED to include the Prebid external_bundle_url host when Prebid is enabled. +# allowed_domains = ["ads.example.com", "assets.example.com", "*.cdn.example.com"] +# +# Route first-party asset paths to a different backend origin. Longest matching +# prefix wins; only GET/HEAD participate; query string is preserved. +# [[proxy.asset_routes]] +# prefix = "/.image/" +# origin_url = "https://assets.example.com" +# Optional path rewrite before sending upstream (the prefix must match what the +# pattern expects — here both use `/.image/`): +# path_pattern = "^/\\.image/(.*)/[^/]+\\.([^/.]+)$" +# target_path = "/image/upload/$1.$2" +# +# Optional S3 SigV4 auth for a private origin: +# [proxy.asset_routes.auth] +# type = "s3_sigv4" +# region = "us-east-1" +# origin_query = "strip" +# secret_store = "s3-auth" +# access_key_id = "access_key_id" +# secret_access_key = "secret_access_key" +# +# Optional Fastly Image Optimizer for the route (references a profile_set below): +# [proxy.asset_routes.image_optimizer] +# enabled = true +# region = "us_east" +# profile_set = "default_images" + +# Reusable Fastly Image Optimizer profile sets referenced by asset routes. +# Supported IO params are a strict subset: quality, resize-filter, format, +# width, height, crop. Keep production profile tables in private config. +# [image_optimizer.profile_sets.default_images] +# base_params = "quality=70&resize-filter=bicubic" +# default_profile = "default" +# unknown_profile = "use_default" # "use_default" or "reject" +# profile_param = "profile" +# +# [image_optimizer.profile_sets.default_images.profiles] +# default = "width=1920" +# thumbnail = "width=150&crop=1:1,smart" +# medium = "format=auto&width=828" + +# Operator-controlled cache header policies for static/rehosted assets. Disabled +# rules never match, and matcher/policy validation is deferred until a rule is +# enabled; IDs must be non-empty and unique. Keep rules disabled unless the +# matched publisher paths are known to be content-addressed. +# [[cache.asset_rules]] +# id = "nextjs-static" +# enabled = false +# preset = "nextjs-static" # built-in matcher/policy preset +# visibility = "public" +# browser_ttl_seconds = 31536000 +# edge_ttl_seconds = 31536000 +# immutable = true +# +# [[cache.asset_rules]] +# id = "publisher-fingerprinted-assets" +# enabled = false +# path_globs = ["/assets/**/*.js", "/assets/**/*.css", "/assets/**/*.png", "/assets/**/*.webp"] +# Immutable custom rules require an unambiguous fingerprint_style, either "hex" +# or "esbuild-base32". "vite-base64-url" is allowed only for non-immutable rules +# because ordinary filenames can match its shape. +# fingerprint_style = "hex" +# visibility = "public" +# browser_ttl_seconds = 31536000 +# edge_ttl_seconds = 31536000 +# immutable = true + +# Server-side auction. Providers are declared below; mediator names must match +# enabled integrations. Kept active with the creative-processing leaves present +# so the EdgeZero environment override can apply to them. +[auction] +# Keep disabled until provider endpoints, routes, and profile values below are +# replaced with deployment-specific settings. +enabled = false +# Rewrite winning-bid creative HTML to first-party endpoints (default true). Set +# false to skip proxy/click-URL conversion and creative TSJS injection. +# Sanitization is controlled separately by `sanitize_creatives` below. Restore +# true before rolling back to an older binary that rejects unknown fields. +rewrite_creatives = true +# Strip executable markup (script/object/embed/form/...) from winning-bid adm, +# removing those elements together with their inner content. Defaults to false +# (executable markup preserved). Note that with `rewrite_creatives = true` the +# adm is still not untouched: eligible URLs are rewritten, bidder `` +# elements are removed, and the creative TSJS runtime is injected. Enable it +# whenever creatives can render in a context that shares the publisher origin +# (its primary defence there); leave it off when creatives render in a +# foreign-origin frame (e.g. the Prebid Universal Creative inside the ad +# server's iframe), since it removes script-based creatives entirely and would +# blank slots on a script-heavy demand stack. +sanitize_creatives = false +timeout_ms = 2000 +# mediator = "adserver_mock" # optional mediator integration +# Context keys the JS client may forward into auction requests (allowlist; +# empty blocks all). +allowed_context_keys = [] + +# Example server-side provider declarations. Remove or customize before enabling auctions. +[auction.providers.pbs-main] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://prebid.example.com/openrtb2/auction" +# `prebid-server` defaults to 1000 ms when omitted. +routing = "explicit" + +[auction.providers.pbs-main.profile_config] +debug = false +test_mode = false +consent_forwarding = "both" + +[auction.providers.pbs-main.notifications] +suppress_all = false +suppress_seats = [] + +[auction.bidders.example-bidder] +provider = "pbs-main" + +# APS server behavior is also provider/profile-owned. Use routing = "all_eligible" +# when every banner-compatible slot should be eligible. The APS profile timeout +# defaults to 800 ms; debug and script creatives default to false. +# [auction.providers.aps-main] +# protocol = "openrtb-2.6" +# profile = "aps" +# endpoint = "https://aps.example.com/e/pb/bid" +# routing = "all_eligible" +# +# [auction.providers.aps-main.profile_config] +# account_id = "example-aps-account-id" +# debug = false +# allow_script_creatives = false +# inventory_domain = "publisher.example" +# inventory_page_origin = "https://www.publisher.example" + +# Server-side ad slot templates + creative-opportunity auction. Kept active. +[creative_opportunities] +# Set false to disable server-side ad templates while keeping slot definitions +# and direct POST /auction callers. Structurally inactive templates fall back to +# the browser-only `Cache-Control: private, max-age=60` policy (this is one +# cause of it). Origin private/no-store policies, and bot/prefetch/consent-denied +# requests, retain their origin policy; CDN-specific cache headers are unchanged. +# Restore true, re-push, and finalize before rolling back to a binary that +# predates this setting. +enabled = true +gam_network_id = "123456789" +price_granularity = "dense" +# FCP is not affected by this value — body content above has already +# streamed and painted before the hold begins. What this caps is the slip on +# DOMContentLoaded and window.load. 500 ms is the recommended default; raise +# only if your SSPs need more headroom and analytics confirm the DCL slip is OK. +auction_timeout_ms = 500 +# +# Initial-page delivery mode. `inline` is the default and current production +# behaviour; `esi` is opt-in per deployment and stores an inert comment in a +# shared, reader-neutral template cache. See docs/guide/configuration.md and run +# scripts/template-cache-local-test.sh esi before enabling. This and the +# cache keys below belong in this [creative_opportunities] table. +# assembly_mode = "inline" +# Request headers (besides Accept-Encoding) the origin may name in Vary. Every +# emitted Vary name must be covered here or template storage is refused. Never +# include Cookie — shared templates must be reader-neutral. +# template_cache_vary = [ +# "rsc", +# "next-router-state-tree", +# "next-router-prefetch", +# "next-router-segment-prefetch", +# ] +# Safety ceiling (seconds) for one shared template; TS uses the smaller of this +# and the origin-authorized remaining edge freshness. Default 60; range 1-86400. +# template_cache_max_age_seconds = 1200 +# Optional exact, case-sensitive cookie names; omitted and empty lists are equal. +# Key bounded variants only, never account, session, or TS identity IDs. +# Absent and empty values select different variants. No cardinality limit exists. +# template_cache_key_cookies = ["ab_bucket"] +# Presence, including session=, bypasses template lookup and storage. +# template_cache_bypass_cookies = ["session"] +# With either list nonempty, false bypasses unlisted cookies; true asserts those +# cookies do not change origin HTML. No identity or consent cookie is exempt. +# With both lists empty, false bypasses all Cookie headers, as before. +# On the template path, origin `Vary: Cookie` is always refused. Readthrough +# has no response-side guard; run `ts origin probe-shareability` before enabling. +# Configured key-cookie dimensions disable readthrough, preserving template variants. +# origin_is_cookie_independent = false +# Invalid names, duplicates, cross-list overlap, and TS identity cookies in the +# key list reject configuration. Named policies bypass ambiguous/nonconforming +# Cookie fields, including duplicate key cookies. +# If a downstream CDN maps ab_bucket to X-Exp-Variant, also add x-exp-variant to +# template_cache_vary; that header alone cannot separate arms at TS. +# Rollback: older binaries reject both new fields even as empty lists. Remove +# both fields AND restore independence=false or disable ESI when HTML uses cookies. +# +# Applies shareability checks to the platform's origin cache. Default false keeps +# the existing policy: ad-serving requests bypass; other publisher requests use +# the platform default. True admits shareable ad requests and bypasses unshareable +# non-ad requests. A DIFFERENT cache from the template cache +# above, and a weaker one: the decision is made before the origin replies, so +# none of the template cache's response-side refusals (`Set-Cookie`, CSP nonce, +# missing freshness) apply. `ts origin probe-shareability` must pass on every +# axis and verdict first. Tagged readthrough objects can be purged with +# ts cache purge (--all or the exact reader-facing --page URL). Older untagged +# objects must expire on the origin TTL; see the configuration guide's rollback note. +# origin_readthrough_enabled = false +# +# `gam_unit_path` may be a template. Supported placeholders: +# {network_id} -> gam_network_id +# {slot_id} -> the slot's id +# {section} -> a path segment of the request, sanitized to [A-Za-z0-9_-]; +# `section_segment` picks which one, `section_root` covers +# paths that have no such segment. +# A template with no placeholders (or an absent gam_unit_path) keeps the old +# behavior: verbatim path, or the default `//`. +# +# `section_root` is REQUIRED when any slot's template uses {section}. There is no +# default - the home-section name is publisher-specific. Must be [A-Za-z0-9_-]+. +# +# `section_segment` is the 0-based index of the segment that names the section; +# it defaults to 0 (the first segment). Set it to 1 for locale-prefixed URLs, so +# "/en/news/article" resolves to "news" instead of "en". +# +# Both are left commented out: no slot below uses {section}, and an unused key +# still ships in the pushed config blob. +# section_root = "home" +# section_segment = 0 +# +# Slot templates. Add one block per slot below. (This is an array of tables, so +# the env overlay cannot set it - edit the `[[creative_opportunities.slot]]` +# blocks in TOML directly.) +# [[creative_opportunities.slot]] +# id = "leaderboard" +# gam_unit_path = "/123456789/leaderboard" +# div_id = "div-gpt-ad-leaderboard" +# page_patterns = ["/"] # glob syntax (not regex); "/" matches only the homepage +# formats = [{ width = 728, height = 90 }] +# +# Example templated slot (one rule serves every section). Uncomment +# `section_root` above when enabling it. Note that "/news/*" does not match +# "/news" - list the section landing page separately. +# [[creative_opportunities.slot]] +# id = "ad-header" +# gam_unit_path = "/{network_id}/example/{section}" +# page_patterns = ["/", "/news", "/news/*", "/reviews", "/reviews/*"] +# formats = [{ width = 728, height = 90 }] +# "/" -> /123456789/example/home +# "/news" -> /123456789/example/news +# "/news/x" -> /123456789/example/news +# "/reviews/y" -> /123456789/example/reviews + +# Direct Tinybird auction telemetry (off by default). When enabled, `api_host` +# is required and must be a bare host (no scheme or path). +# [tinybird] +# enabled = true +# api_host = "api.us-east.tinybird.example" # required when enabled; host only +# auction_dataset = "auction_events" # Events API datasource name +# auction_token_secret = "tinybird_auction_append_token" # Key in trusted_server_secrets + +# Debug endpoints (all default false — never enable in production). +# [debug] +# Exposes GET /_ts/debug/ja4 returning TLS/JA4 fingerprint details. Disable +# after investigation; it reflects data browser JS cannot normally read. +# ja4_endpoint_enabled = false +# Injects a `` auction dump before ``. NEVER enable +# in production: even the default "redacted" mode still includes bid-level fields +# and creative previews — it is not a fully anonymized dump. +# auction_html_comment = false +# +# Fine-grained control over the dump (all optional; defaults shown). +# [debug.auction_html_comment_options] +# include_provider_responses = true +# include_mediator_response = true +# include_bids = true +# Subset of the fixed validated metadata keys shown in "redacted" (and, for these +# three keys, "upstream") mode. Any other key fails config load. +# metadata_keys = ["error_type", "http_status", "message"] +# "redacted" (default), "upstream", or "full". "upstream" exposes six untyped +# provider diagnostic values that may contain request or identity data; "full" +# additionally exposes all response metadata and untruncated creatives. Never use +# either sensitive mode in production. +# verbosity = "redacted" +# "compact" (default) or "pretty"; pretty formats only the outer dump, JSON +# request/response bodies remain strings exactly as captured. +# format = "compact" + + +# ============================================================================= +# OPTIONAL — Integrations +# ============================================================================= +# Every integration is off by default. Most are fully commented out; uncomment a +# block and set `enabled = true` to activate it. Four (gpt, didomi, datadome, +# google_tag_manager) are kept as active `enabled = false` stubs so the `ts audit` +# CLI can flip them in place — leave those sections present. Integrations whose +# `enabled` defaults to true (prebid, permutive, lockr, ...) still stay OFF while +# their section is commented out. Required fields are noted per block. +# ============================================================================= + +# Browser-side Prebid.js integration. Server-side bidder routing belongs under +# [auction.providers] and [auction.bidders]. When enabled, +# `external_bundle_url` is required and its host must be listed in +# [proxy].allowed_domains. Kept active but disabled. +[integrations.prebid] +enabled = false +timeout_ms = 1000 +debug = false +client_side_bidders = [] # bidders running via native Prebid.js adapters +# Keep selected GAM inventory out of Trusted Server's Prebid refresh auctions. +# Matching slots still refresh through GAM. +# excluded_gam_ad_unit_path_suffixes = ["/trackingonly"] +# Runtime bundle metadata — set after running `ts prebid client` and uploading: +# external_bundle_url = "https://assets.example.com/prebid/trusted-prebid-.js" +# external_bundle_sha256 = "" +# external_bundle_sri = "" +# Bundle build inputs consumed by `ts prebid client`, not by the edge runtime. +# Values are exact upstream module stems without `.js`. +# [integrations.prebid.bundle.modules] +# bidder = ["rubiconBidAdapter"] +# Omit user_id for the curated default preset; use [] to select none. +# user_id = ["sharedIdSystem"] +# Omitted or empty analytics selects no analytics adapters. +# analytics = ["atsAnalyticsAdapter"] + +# Prebid User ID modules that Trusted Server installs and keeps installed, so +# operators can manage identity centrally without publisher JavaScript changes. +# Each entry is forwarded to Prebid.js verbatim: `name` is a +# `userSync.userIds` entry name, `params` and `storage` are whatever that module +# documents. Trusted Server does not interpret them; supported names come from +# the checked-in User ID registry. Each `name` must appear only once. +# +# The module must be present in the built bundle: name it under +# [integrations.prebid.bundle.modules].user_id, or omit that list to take the +# generator's default preset. `ts prebid client` resolves every managed name +# through the checked-in User ID registry and fails if the generated manifest +# omits its required module without updating the configured hash or SRI. +# +# Persisting a resolved ID into the EC identity graph additionally needs a +# matching partner under [[ec.partners]] whose source_domain equals the module's +# OpenRTB EID source; without one the ID still reaches the auction but is never +# written to KV. +# +# [[integrations.prebid.managed_user_ids]] +# name = "sharedId" +# +# [integrations.prebid.managed_user_ids.storage] +# type = "cookie" # or "html5" +# name = "_sharedid" +# expires = 15 # days; omit to keep Prebid's default +# refresh_in_seconds = 1800 # omit to keep Prebid's default + +# Next.js first-party rewriting for App Router / RSC payloads. +# [integrations.nextjs] +# enabled = true +# rewrite_attributes = ["href", "link", "siteBaseUrl", "siteProductionDomain", "url"] +# max_combined_payload_bytes = 10485760 # 10 MiB + +# Testlight OpenRTB test integration. `endpoint` required when enabled. +# [integrations.testlight] +# enabled = true +# endpoint = "https://testlight.example.com/openrtb2/auction" +# timeout_ms = 1200 +# rewrite_scripts = true + +# Didomi CMP SDK/API first-party proxy. Kept active but disabled so `ts audit` +# can flip `enabled` to true when Didomi is detected on the audited page. +[integrations.didomi] +enabled = false +geo_query_parameters = false +# sdk_origin = "https://sdk.example.com" +# api_origin = "https://api.example.com" + +# Sourcepoint CMP first-party proxy. +# `cdn_origin` is intentionally omitted: the upstream CDN origin is pinned to +# Sourcepoint's own host and is not operator-selectable. Leave it unset so the +# validated default applies — overriding it with any other host fails config +# validation. +# [integrations.sourcepoint] +# enabled = true +# rewrite_sdk = true +# cache_ttl_seconds = 3600 +# auth_cookie_name = "sp_auth" # optional: forward a custom authCookie upstream + +# Osano consent management (proxy toggle only). +# [integrations.osano] +# enabled = true + +# Permutive DMP. `organization_id` and `workspace_id` required when enabled. +# [integrations.permutive] +# enabled = true +# organization_id = "your-permutive-organization-id" # required (non-empty) +# workspace_id = "your-permutive-workspace-id" # required (non-empty) +# project_id = "your-permutive-project-id" +# api_endpoint = "https://api.example.com" +# secure_signals_endpoint = "https://secure-signals.example.com" + +# lockr identity SDK. `app_id` required when enabled. +# [integrations.lockr] +# enabled = true +# app_id = "your-lockr-app-id" # required (non-empty) +# api_endpoint = "https://identity.example.com" +# sdk_url = "https://identity.example.com/trusted-server.js" +# cache_ttl_seconds = 3600 +# rewrite_sdk = true + +# DataDome bot protection. Proxies tags.js + signal-collection API first-party. +# Kept active but disabled so `ts audit` can flip `enabled` when detected. +[integrations.datadome] +enabled = false +# sdk_origin = "https://sdk.example.com" +# api_origin = "https://api.example.com" +# cache_ttl_seconds = 3600 +# rewrite_sdk = true +# Server-side Protection API validation (fails open on timeout/error): +# enable_protection = false +# Required when enable_protection = true. The value is a key in +# trusted_server_secrets, not the DataDome credential itself. +# server_side_key_secret_name = "datadome_server_side_key" +# protection_api_origin = "https://api.example.com" +# timeout_ms = 1500 +# protection_excluded_methods = ["OPTIONS"] +# Client-side tag auto-injection (emits only when client_side_key is non-empty): +# client_side_key = "" +# inject_client_side_tag = true +# client_side_tag_url = "/integrations/datadome/tags.js" +# Temporary static-header bypass for an access-controlled STAGING environment +# only. A matching `x-ts-datadome-bypass` header skips server-side Protection +# API validation and is stripped before the origin sees it. The credential is +# loaded from the Secret Store at runtime (>= 32 bytes of high-entropy material). +# Never enable in production. +# [integrations.datadome.protection_test_bypass] +# enabled = true +# credential_secret_store = "ts_secrets" +# credential_secret_name = "datadome_protection_test_bypass" + +# Google Publisher Tag (GPT) first-party proxy. Kept active but disabled so +# `ts audit` can flip `enabled` to true when GPT is detected. +[integrations.gpt] +enabled = false +# Kept as an active leaf so the environment override can apply; the overlay +# cannot create a missing configuration leaf. GAM attribution stays off until +# this is set true. +gam_attribution_enabled = false +# script_url = "https://ads.example.com/gpt.js" +# cache_ttl_seconds = 3600 +# rewrite_script = true + +# GPT runtime diagnostics browser overlay. Optional and enabled manually (not +# flipped by `ts audit`); serves a diagnostics module gated behind an activation +# query param + session cookie. +# [integrations.gpt_diagnostics] +# enabled = true + +# APS browser renderer ownership. Server-side APS behavior belongs under an +# [auction.providers] entry with profile = "aps". +# [integrations.aps] +# enabled = true +# Default `trusted_server` uses TS's opaque static renderer route. Set +# `publisher_native` only for a controlled publisher-origin friendly-frame cohort. +# rendering_mode = "trusted_server" + +# Google Tag Manager first-party proxy. Kept active but disabled so `ts audit` +# can fill container_id and flip `enabled` when GTM is detected. `container_id` +# is required when this integration is actually enabled. +[integrations.google_tag_manager] +enabled = false +# Invalid placeholder on purpose: enabling GTM without a real GTM-XXXXXX id +# fails validation. `ts audit` overwrites this when it detects a real container. +container_id = "GTM-REPLACE-ME" +# Must be a credential-free https bare origin with a literal host: no path, +# query, fragment, `user:pass@` or wildcard. Targets are built by appending a +# rooted path to this value, so anything beyond an origin corrupts them. +# upstream_url = "https://tags.example.com" +# Tag ids the `gtag/js` path may serve from your own origin, in addition to +# `container_id`. A request naming any other tag id is redirected to the +# upstream instead, so this origin never serves a container you did not +# choose. The visitor loads that tag from the upstream directly, which a +# `script-src` that does not list the upstream origin will block. +# +# Set this if your pages load gtag with a measurement id separate from the +# container above (a GA4 `G-`, Ads `AW-` or Merchant Center `MC-` id). A tag +# that is missing here also loses the ad-blocker resilience this integration +# provides. Find the ids by grepping your rendered HTML for `gtag/js?id=`. +# allowed_tag_ids = ["G-EXAMPLE1", "AW-000000", "MC-EXAMPLE1"] + +# Mock ad server used for auction mediation in dev/testing. +# [integrations.adserver_mock] +# enabled = true +# endpoint = "https://adserver.example.com/mediate" +# timeout_ms = 1000 +# Map auction context keys to mediation URL query params: +# [integrations.adserver_mock.context_query_params] +# example_segments = "segments" diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 3ce991e5ef..ccfa5c1a08 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -34,6 +34,7 @@ commands are shown explicitly. | `ts config diff` | Linux + macOS | Diff `trusted-server.toml` against the live `EdgeZero` config | `ts config diff [OPTIONS] --adapter ` | | `ts config gc` | Linux + macOS | Reclaim orphaned chunk entries leaked from prior oversized pushes | `ts config gc [OPTIONS] --adapter ` | | `ts config init` | Linux + macOS | Initialize a Trusted Server config file from the example template | `ts config init [OPTIONS]` | +| `ts config migrate` | Linux + macOS | Rewrite a config written for an earlier release into the shape this release reads | `ts config migrate [OPTIONS] --adapter ` | | `ts config push` | Linux + macOS | Push `trusted-server.toml`, with the permission policy file it names, as a blob envelope through `EdgeZero` | `ts config push [OPTIONS] --adapter ` | | `ts config validate` | Linux + macOS | Validate `edgezero.toml`, the typed Trusted Server config and, with `--permissions`, the permission policy file it names | `ts config validate [OPTIONS]` | | `ts deploy` | Linux + macOS | Deploy the project through a target adapter | `ts deploy [OPTIONS] --adapter [-- ...]` | @@ -184,6 +185,88 @@ sweep, and `--store ` overrides the manifest's config-store id outright. Both change which store gets swept, so on a destructive run check the store id `gc` reports before passing `--yes`. +### Upgrade a configuration written for an earlier release + +Run `ts config migrate` first when you upgrade. A configuration written for an +earlier release does not load until its settings take the shape this release +reads, and the command rewrites the ones it can. Keep a copy of the original +before you use `--write`, because a rollback restores it with the old build, +and it holds the `[request_signing]` store ids that `ts keys rotate` and +`ts keys deactivate` take. + +```bash +# Print the migrated configuration, with what changed on stderr. +ts config migrate --adapter fastly trusted-server.toml > trusted-server.migrated.toml + +# Rewrite the file in place. +ts config migrate --adapter fastly --write trusted-server.toml + +# Change nothing, and exit 1 when the file needs migrating, for CI. +ts config migrate --adapter fastly --check trusted-server.toml +``` + +`--adapter` names the adapter the deployment runs on, being `fastly`, +`cloudflare`, `axum` or `spin`. On Fastly and Cloudflare the host's location +lookup, which an earlier release ran on every request, becomes +`[geo] module = "platform"`. The Axum dev server and Spin have no lookup and +refuse that line at startup, so there the command selects no geo module and +leaves a `to do:` item for the Edge Cookie module, which then needs +`[geo] assume_single_jurisdiction = true` or a geo module of your choice. A +configuration you push to more than one adapter, such as the Axum dev server +and Fastly, is migrated once for each adapter into a file of its own. + +`config migrate ` prints the migrated configuration on stdout. On stderr it +prints a line for each change it made, starting `changed:`, a line for each item +left for you, starting `to do:`, and a line for each change in behavior no +setting expresses, starting `check:`. Its last line says whether the migrated +configuration passes the settings validation `ts config validate` runs, except +with `--check`, which does not validate. A `to do:` item is a setting the +configuration cannot load without, such as a `[[handlers]]` entry that can guard +a path of the publisher's own, or one this release ignores whose value you carry +elsewhere, such as `[consent.gdpr] applies_in`. Each needs a decision or a value +the configuration does not hold, so the command leaves the setting where it is +and reports it on every run until it is done. A configuration that names no +permission policy file leaves one more item, because no policy ships with +Trusted Server and the command chooses none, so name the publisher's file in +`[permissions] policy` and supply it with `ts config push --permissions `. + +The command reads the file alone, with no manifest and no environment overlay. +It keeps the comments and the order of the tables you wrote. A module that was +not enabled is written under its new name as a comment block, and to run it you +uncomment the block and select the module in the section of its type. The +command never selects `headers.geo`, because the `x-geo-*` headers carry the +reader's location, and a `check:` line says what the adapter's responses carried +before. It never prints a value from the configuration, because a configuration +can hold secrets. Each line names the setting, a failed validation gives only +the byte offset or the phase that refused, and `ts config validate` prints the +whole refusal. Migrating a migrated configuration changes nothing. + +It exits 0 when nothing is left to do and the migrated configuration passes +validation, 1 when an item is left to do or the configuration does not pass, +and 2 when the file cannot be read, written or parsed as TOML. With `--check` +it exits 0 when the configuration needs no migrating and 1 otherwise. `--write` +cannot be used with `--check`, and `--write` writes the result even when an item +is left to do. + +After migrating: + +1. Do each `to do:` item, and read each `check:` item. +2. Rename each environment variable that set a moved setting, such as + `TRUSTED_SERVER__PUBLISHER__ORIGIN_URL`, which is now + `TRUSTED_SERVER__SOURCE__ORIGIN__URL`, because `ts config push` ignores a + variable for a key the configuration does not hold. +3. Run `ts config validate --permissions ` with the publisher's policy + file. +4. Push the configuration and the policy file with the new build, because a + build from before this release refuses the migrated configuration and this + release refuses the old one. On Fastly, `ts deploy --staging` and + `ts config push --staging --permissions ` let you check the pair + together first. On Cloudflare and Spin, keep the key-value store this + release requires, the Workers KV binding `TRUSTED_SERVER_KV` and Spin's + `default` store, or every request fails. +5. Expect the stored pages to be fetched again once after the upgrade, because + their keys hold a code worked out from the settings. + ### Diagnose ad-template configuration The static `ts config ad-templates` commands evaluate local configuration diff --git a/docs/superpowers/specs/2026-10-17-config-migrate-design.md b/docs/superpowers/specs/2026-10-17-config-migrate-design.md new file mode 100644 index 0000000000..6e832125e6 --- /dev/null +++ b/docs/superpowers/specs/2026-10-17-config-migrate-design.md @@ -0,0 +1,213 @@ +# Design Spec: `ts config migrate` rewrites a settings document written for main into the shape the stack reads + +**Status:** Proposed, 2026-10-17, revised 2026-10-18 after review rounds 1 and 2, and revised 2026-10-27 for the permission policy, which is selected when a service is deployed. Raised as IABTechLab/trusted-server#1409, one of the stack's pull requests, row 51 of the stack in IABTechLab/trusted-server#777, opened on 2026-10-17, for Tech Lab review. References to the stack's code are to the code of IABTechLab/trusted-server#1409, by file and function, with a line number where one helps. +**Author:** 51Degrees (contributed), for Tech Lab review +**Related specs:** `2026-10-14-source-modules-design.md`, `2026-10-13-middleware-settings-design.md`, `2026-10-14-rsc-flight-middleware-design.md`, `2026-10-15-geo-module-request-design.md`, and the design documents of the stack in IABTechLab/trusted-server#1342 +**Related PRs:** IABTechLab/trusted-server#1409 +**Issues:** none +**Last updated:** 2026-10-27 + +> **Why this spec exists.** The changelog of the stack lists 41 breaking entries under Unreleased, where main lists 10, and a settings document written for Tech Lab main does not load under the stack. Every operator would rewrite their document by hand from the changelog. `ts config migrate` makes the rewrite automatic where the document holds everything the rewrite needs, and says what is left where it does not. + +## 1. Problem + +A document written for Tech Lab main (`upstream/main` at `5bc388625`) is refused at startup by the stack. `[integrations]` is gone, the origin settings moved to `[source]`, the auction providers moved to `[demand]`, `[[handlers]]` is refused, and a module changes a page only where a `[[fetch]]` or `[[serve]]` entry names its middleware. The changelog lists each change under Unreleased, with 41 entries marked breaking, and the settings code holds a few more that no changelog line names. An operator who upgrades has to read all of them, work out which apply to their document, and rewrite the document by hand without losing the comments they wrote. + +Most of the rewrite is mechanical, because the old shape says everything the new one needs. A few changes need a decision only the operator can make, or a value the document does not hold, a few change behavior in a way no setting can express, and the location settings depend on the adapter the deployment runs on. + +## 2. Design + +### 2.1 What the command does + +`ts config migrate --adapter ` reads the document, rewrites every shape the stack refuses or reads differently, and prints the result on stdout. On stderr it prints one line for each change it made, one line for each item left for the operator and one line for each change in behavior to check, and then, unless `--check` is given, the result of validating the migrated document. + +- `--adapter` names the adapter the deployment runs on, being `fastly`, `cloudflare`, `axum` or `spin`, as `ts config push` and `ts config diff` take one. It is required, because main's location lookup and location headers differed by adapter, and what each adapter accepts in `[geo]` differs too (§3). +- `--write` rewrites the file in place, and leaves the file untouched when nothing changes. It writes the result even when an item is left to do or the result does not pass validation, because the document did not load before either, and the exit status says so. +- `--check` changes nothing, prints nothing on stdout and does not validate. It exits 1 when the document needs migrating, being a rewrite to make or an item left to do, so CI can hold a document to the current shape. +- `--write` and `--check` cannot be used together. + +The command works on the file alone. It reads no `edgezero.toml`, applies no `TRUSTED_SERVER__` overlay and asks no platform anything, because the rewrite is a property of the document and the adapter. It validates the result as `ts audit generate` validates a document it is about to write, by parsing the text into `TrustedServerAppConfig` with no manifest and running `validate_excluding_secrets` with the modules the tool links (`crates/trusted-server-cli/src/commands/audit/generate/validate.rs` lines 55 to 66). That is the push-time validation `ts config validate` runs, and `ts config validate` also applies the overlay, checks secrets and runs EdgeZero's adapter checks (EdgeZero v0.0.10, `crates/edgezero-cli/src/config.rs` lines 245 to 277). Neither runs the checks made only when a deployment starts, being the checks core makes once the settings load that push-time validation does not repeat. Those are the checks of the old Edge Cookie layout, the `[ec]`, `[device]` and `[geo]` module selections and the jurisdiction acknowledgment (`Settings::finalize_deserialized` in core `settings.rs`, where `validate_resolve_allowed_origins`, `validate_module_sections` and `validate_phase_entries` run at push time too), the runtime loader's check of the permission policy file against the digest the push wrote (`load_domain_policy` in core `config_payload.rs`), and each adapter's checks as it builds its state. The command reports the location settings among those as items left to do, because every document written for main gains an Edge Cookie module from its passphrase, and whether that document then starts depends on its geo settings and its adapter (§3). The command's validation leaves the policy check out, because the command takes no `--permissions` (`ConfigMigrateArgs` in `crates/trusted-server-cli/src/commands/config/migrate/mod.rs`) and a policy file is the publisher's to supply, so a missing policy is a `to do:` line (§2.2) and a document can still reach exit status 0 once the operator has named a policy. `ts audit generate`, which shares the validation (`check_candidate` in the same `validate.rs`), leaves it out as well. `ts config validate` checks the policy against the files `--permissions` supplies, and without `--permissions` says the policy was not checked (`run_validate_with` in `crates/trusted-server-cli/src/commands/config/policy.rs`), as row 4 of the stack (IABTechLab/trusted-server#1345) builds §3.1 of `2026-07-30-permission-model-design.md`. The command's validation, like that of `ts audit generate`, counts `permissions.sha256` as present and refuses an `[inspect]` pattern under `permissions.text`, as every pattern check does (`validate_patterns` in core `inspect/config.rs`, and `2026-10-12-inspection-endpoints-design.md`). + +Each line on stderr starts with one of three words. + +- `changed:` a rewrite the command made. +- `to do:` an item the document cannot load without, or a setting the stack ignores whose value the operator has to carry elsewhere. It needs a decision or a value the document does not hold, so the command leaves the setting where it is, says what to do, and reports the item on every run until the operator has done it. +- `check:` a behavior change no setting expresses, printed with the rewrite it follows from, or for a document written for main. Nothing in the document has to change. + +The exit status is 0 when nothing is left to do and the migrated document passes validation, 1 when an item is left to do or the result does not pass, and 2 when the file cannot be read, written or parsed as TOML, or the arguments are refused, as `--write` with `--check` is. With `--check` it is 0 when the document needs no migrating and 1 otherwise. + +### 2.2 Every change and how it is handled + +The table lists every breaking entry in `CHANGELOG.md` under Unreleased that concerns the document or how a deployment upgrades, and the changes in the settings code that no changelog line names, marked "settings code". Entries for crates outside core, for the `ts` tool and for request handling with no setting are grouped in the last rows. + +| Change | Handling | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `[integrations]` is gone, and each integration is a module selected in the section of its type | Rewritten. An enabled `[integrations.]` becomes the module in its section, `module` or `modules`, with its settings in `[
.]` and `enabled` dropped. One that was not enabled is commented out under its new name. `enabled` follows main's default for each id. | +| `[auction.providers.]` moved to `[demand] modules` and `[demand.]` | Rewritten. The name takes `_` for `-`, `profile` becomes `implementation` by module path, `profile_config` moves into the table, and `protocol`, which main only took as `openrtb-2.6`, is dropped. | +| `[auction.bidders.] provider` is `module`, naming the demand source | Rewritten, with the demand source's new name. | +| `[auction] mediator = "adserver_mock"` and `[integrations.adserver_mock]` moved to `[ad-server]` | Rewritten to `[ad-server] module = "iabtl"` and `[ad-server.iabtl]`. A mock ad server no mediator names is commented out. | +| `[integrations.aps] rendering_mode` moved to the `auction.aps` demand table | Rewritten into each `auction.aps` demand table. Main read it only from an enabled table. | +| `include_mediator_response` is `include_adserver_response` | Rewritten. | +| The origin settings moved from `[publisher]` to `[source]` | Rewritten to `[source] module = "origin"` and `[source.origin]`, with `origin_url` as `url`, `origin_host_header_override` as `host_header`, `origin_auth` as `auth` and `directory_index` unchanged. Check, below, where `host_header` is set. | +| An asset route's `origin_url` and `auth` moved to its `source` table | Rewritten to `[proxy.asset_routes.source] module = "origin"` and `[proxy.asset_routes.source.origin]`. | +| The signer spelling `s3_sig_v4` is no longer read | Rewritten to `s3_sigv4` in every `auth` table. | +| `[[handlers]]` and HTTP Basic authentication are removed | Rewritten by removing an entry where every path its pattern can match is one the stack closes, where nothing is served now, as the paragraph below the table defines. Main matched a pattern anywhere in the path (main core `settings.rs` line 734), so `^/admin/keys` alone also guarded `/admin/keystore`, which the stack forwards to the origin, and so did `^/admin/keys/?`, whose `?` lets the `/` match nothing, and such an entry is to do, below. | +| `[request_signing]` does not take `config_store_id` or `secret_store_id` | Rewritten by removing both. Check, below, for where the ids go. | +| `[ec] passphrase` is read as deprecated (settings code) | Rewritten to `[ec] module = "hmac"` and `[ec.hmac] passphrase`, where the stack documents it. The deprecated key still loads, with a warning at startup, for one release. | +| `[ec.providers.]` is `[ec.]`, and the module names `host-signals` and `client-fixed` are `host_signals` and `client_fixed` | Rewritten, in the tables, the selection and an `implementation` line. | +| `[permission_signal] sources` is `[permission-signal] modules`, with `gpp-sale-opt-out` and `gpp_sale_opt_out` as `gpp` and `us_privacy` as `us-privacy` | Rewritten. | +| `[device] module = "fastly"` is `"platform"` | Rewritten. | +| The host's location lookup runs only where `[geo]` selects it (settings code) | Rewritten for a document written for main on Fastly and Cloudflare by selecting `[geo] module = "platform"`, the lookup main ran on every request there. Check, on Cloudflare, that the lookup now reads the reader's region, which main's did not (§4). On Axum and Spin main's lookup located no request, and those adapters refuse the selection at startup, so the command selects no geo module there, and the Edge Cookie module that follows is to do, below. | +| No response carries `x-geo-*` unless `headers.geo` is selected and placed | Check, for a document written for main, saying what main's responses carried on the adapter and, on Fastly, how to keep the headers. The command never selects `headers.geo`, because the headers carry the reader's location (§4). | +| `[tinybird]` is `[analytics.tinybird]`, selected by `[analytics] module` | Rewritten, with `enabled` dropped. A table that was not enabled is commented out. Check, below, for the consent gate. | +| A module's page change runs only where a `[[fetch]]` or `[[serve]]` entry names its middleware | Rewritten for a document written for main by adding one entry for each phase and media type the selected modules supply middleware for, naming that middleware in the order the linked builders supply it, which is the order `trusted-server.example.toml` shows. A document whose modules supply no fetch middleware gets no `[[fetch]]` entry. A document in the stack's shape is left as written, because it loads with no entry, with a warning naming each middleware no entry names (CHANGELOG line 51), and whether its modules change pages is the operator's choice. | +| The Flight rewriter is the middleware `framework.nextjs.flight` | Rewritten by adding a `[[fetch]]` entry for `text/x-component`, among the page entries above, where the Next.js module is selected. Check, below, where it is not. | +| The `/robots.txt` route is the `robots_txt` middleware | Rewritten by adding a `[[fetch]]` entry for `text/plain` on `/robots.txt` to a document with `[robots-txt]` and no entry naming `robots_txt`, which the stack refuses. | +| `ad-server.mock` is `ad-server.iabtl`, `[adserver]` is `[ad-server]` (settings code for the table), and `ads-txt.sellers` is `ads-txt.iabtl` | Rewritten in the selection, the table, an `implementation` line and a `[[fetch]]` middleware name. A selected `[ads-txt]` with no entry naming its middleware gets a `[[fetch]]` entry for `text/plain` on `/ads.txt`, because the module refuses that document (`crates/ads-txt/iabtl/src/lib.rs` lines 154 to 175). | +| A `[[handlers]]` entry whose pattern can match a path the stack does not close | To do. Nothing asks for a password now, so the operator puts one on those paths at the origin or in the hosting platform and then removes the entry. The command never removes it, because the stack refuses the list so that no deployment believes a path is protected. | +| An Edge Cookie module with no geo module, and `[geo] module = "platform"` on Axum or Spin (settings code and the adapters) | To do, for any document. Startup on every adapter refuses the first unless `[geo] assume_single_jurisdiction = true` accepts that every request takes the top of the `permissions.yaml` rules (`GeoConfig::validate_jurisdiction_acknowledgment` in core `settings.rs`), and the Axum and Spin adapters refuse the second, because neither host has a location lookup. `ts config validate` checks neither. The operator chooses between a geo module and the acknowledgment, because the stack asks for that choice to be made rather than assumed. | +| The permission policy is a file the settings name in `[permissions] policy` and the push supplies (`2026-07-30-permission-model-design.md` §3.1, built by row 4, IABTechLab/trusted-server#1345) | To do, for any document whose `[permissions] policy` is missing or empty (`permission_policy` in `rules.rs`). No policy ships with Trusted Server, so the command never chooses one, adds nothing to `[permissions]` and writes no policy file. The line is the same on every adapter (`POLICY_TO_DO`). It says that a domain whose settings name no policy does not start, to name the publisher's file there by its stem, as `policy = "standard"` names `standard.yaml`, and to supply the file with `ts config push --permissions `. It does not say that on Axum the file goes in the folder `TRUSTED_SERVER_PERMISSIONS_DIR` names (`2026-07-30-permission-model-design.md` §3.1, item 4). So a run on a document written for main that names no policy exits 1, though the migrated document passes validation. The operator may name the policy before or after migrating, because a document written for main that names its policy is still taken as written for main (below the table). | +| `[consent.gdpr] applies_in` and `[consent.us_states] privacy_states` are not read (settings code) | To do, as a setting the stack ignores whose value the operator has to carry elsewhere. The stack loads both tables and reads neither (core `consent_config.rs` lines 15 to 28). The places each covers are the `jurisdiction` values in the policy file the document names, so the operator carries any change of their own into that file and removes the table. The command keeps the table, because the list is the value the operator has to carry, so `--check` fails on a document that still carries one, which is intended, as a table nothing reads invites edits that change nothing. | +| A `template_cache_key_cookies` entry under the public prefix is refused | To do, naming the setting. | +| A setting the command cannot place | To do, leaving it where it is. These are an `[integrations.]` naming no module, an `enabled` that is not a boolean, `[integration]`, `rendering_mode` with no `auction.aps` source, a mediator with no mock ad server table, an unknown `profile`, a `profile_config` setting named as one of its provider's own, and both forms of a setting. | +| A key in a module table that main ignored and the module now refuses, in the tables of `gpt`, `didomi`, `lockr`, `permutive`, `sourcepoint`, `testlight`, `nextjs`, `js_asset_proxy`, `google_tag_manager`, `adserver_mock` and `[tinybird]` (settings code) | Reported through the validation result, and `ts config validate` names the key. | +| No response carries `x-ts-version` or the platform's version unless `service_version` is named | Check. The command does not name the middleware, because the changelog lists the version's removal under Security. | +| A path in `origin_url` is kept, and the request path is added after it | Check, where `[source.origin] url` has a path, which main dropped. | +| Where `host_header` is set, the page's own addresses under that host are rewritten, where those under the `url`'s host were (CHANGELOG line 26) | Check, where `origin_host_header_override` moved to `[source.origin] host_header`. | +| Core no longer moves the origin's address in a Flight response | Check, where the Next.js module is not selected. Selecting it changes every HTML page, so the command does not. | +| A telemetry row is sent only for a reader whose permissions include `analytics.ad_reporting.measure_ad_performance` | Check, where the table was enabled. | +| The signing store ids go to `ts keys rotate` and `ts keys deactivate` | Check, where the command removed them. | +| The Cloudflare and Spin adapters require their key-value store | Out of scope. The binding lives in `wrangler.toml` and the Spin manifest, outside the document. Main's `wrangler.toml` declares `TRUSTED_SERVER_KV` and its `spin.toml` grants `default`, and §6 says to keep both. | +| The Cloudflare adapter reads its settings from its config store, the Workers KV namespace `ts config push` writes, and no longer reads the `TRUSTED_SERVER_CONFIG` variable | Out of scope. The binding lives in `wrangler.toml`, outside the document, and §6 says to bind the namespace and push into it. | +| Breaking entries for crates outside core: platform headers, staging, config store chunks, backend naming, the auction protocol, consent decoders, identifier modules, geo modules, cookie constants, the page traits, the modules crate and origin signers | Out of scope. They change code, not the document. | +| The seam probe is `testing.iabtl`, where it was `testing.seam-probe` | Out of scope. It is a test fixture that no production build links (the crate comment of `crates/testing/iabtl/src/lib.rs`), so no deployment's document selects it. | +| Breaking entries for the `ts` tool: `ts cache purge` and `ts prebid client` | Out of scope. | +| Breaking entries for requests with no setting: the admin path check and the removed admin, purge and key rotation routes | Out of scope. | +| Environment variable names, such as `TRUSTED_SERVER__SOURCE__ORIGIN__URL` | Out of scope. They live outside the document, and §6 says to rename them. | +| Main's own breaking entries the stack keeps: the auction maps, APS OpenRTB, `site.page`, `bid_param_zone_overrides`, string scalars, Sourcepoint and creative sanitization | Nothing to do. A document written for main already meets them, and the maps move again under the second row. | +| Earlier shapes of the stack that neither the changelog nor the settings code names, such as `[ec] provider` | Out of scope. They never reached a release, and nothing tells an operator about them. | + +Where the table and §4 say the places are the `jurisdiction` values in the policy file the document names, they describe §3.1 of `2026-07-30-permission-model-design.md`, and the command's line for each list says the same (`consent_lists` in `rules.rs`). + +A `[[handlers]]` pattern guards only paths the stack closes when it starts `^/_ts/admin`, because every path that begins `/_ts/admin` is closed, or starts `^/admin/keys` followed by `/`, `$`, `(?:/|$)` or `(/|$)`, because of `/admin/keys` only the address itself and the paths below `/admin/keys/` are closed (core `closed_paths.rs` lines 18, 25 and 75 to 81), and when nothing after that prefix or bound widens what it matches, which rules out the following. + +- A `?`, `*` or `{` straight after the prefix or the bound, which starts a quantifier that can let the last character of `^/_ts/admin`, or the whole bound, match nothing, so that `^/_ts/admin?` matched `/_ts/admit` and `^/admin/keys/?` matched `/admin/keystore` under main's search. A `+` is accepted, because it matches what comes before it at least once. +- A `[` or a `#` anywhere after the prefix or the bound, because a character class, or a comment the `(?x)` flag allows, can hide a `|` from the command, as the `(` in `^/_ts/admin[](]|^/members` is a character rather than a group, and that pattern matched `/members`. +- A `|` outside a group, which would let the pattern match another path. + +The command reads the pattern's text and not what the pattern means, so where the text alone cannot show that every path is closed it leaves the entry to do, as it does for `^/admin/keys/?$` and `^/_ts/admin/[a-z]+`, although every path either matches is closed. + +A document is taken as written for main when it has `[publisher]` and every table at its top level is one main read, which are `publisher`, `tester_cookie`, `trusted_client_ip`, `ec`, `integrations`, `handlers`, `response_headers`, `request_signing`, `rewrite`, `auction`, `consent`, `cache`, `proxy`, `creative_opportunities`, `image_optimizer`, `tinybird` and `debug`, or is `permissions`, which main did not read and an operator may add before migrating to name the policy file (`written_for_main` and `MAIN_TABLES` in `rules.rs`). Main refused any other table, and every migrated document has `[source]`, so a second run never takes it as written for main. The geo row, the page entry rows, and the checks about `x-ts-version`, the location headers and Flight responses apply to such a document alone, so a document already in the stack's shape keeps its own geo settings and page entries, or the lack of them. A document written for main that already names its policy migrates as the same document without that table does, with the same rewrites, the same items apart from the one that names the policy, and the same check lines, and the result keeps its `[permissions]` table (`a_document_written_for_main_that_names_its_policy_migrates_as_main`). + +The rules for `[permission_signal]`, `[ec.providers]`, `[adserver]`, `ad-server.mock`, `ads-txt.sellers`, `[device] module = "fastly"` and `[robots-txt]` with no entry rewrite shapes no document written for main holds. Each was the shape of an earlier pull request of this stack and never of a release, and each is named by a changelog entry or refused with directions by the settings code, so a deployment that ran a build of one of those pull requests holds a document in that shape. The rules save its operator the rewrite by hand, and an earlier shape that nothing names stays out of scope. + +### 2.3 How the rewrite keeps what the operator wrote + +The command edits the document with `toml_edit`, the editor the tool already uses, so values, comments and formatting it does not touch come out as they went in. + +- A key that moves keeps its line and the comment written directly above it, and a table that moves keeps its place in the file and its comments. A new section such as `[cmp]` takes the place of the table it replaces, with that table's comments above it. +- A new table that follows another, such as `[source]` after `[publisher]`, comes after the comment lines that directly follow that table's last setting, because `toml_edit` holds those lines with the next table. A comment paragraph is the run of comment lines up to a blank line. +- A table removed with nothing to replace it, such as an admin handler, takes the paragraph directly above it and leaves the earlier ones, such as the file's opening comments, to the next table. +- A module that was not enabled is written as a comment block in its place, under its new name, so the operator runs it by uncommenting the block and selecting the module in its section. A module that has no settings, such as Osano, keeps only the comments above its old table. +- The new sections that replace nothing, `[source]` and `[geo]`, follow `[publisher]`, and `[ec.hmac]` follows `[ec]`. +- A setting the stack does not read goes with the comment written directly above it. +- The page entries go after the last table, after the comment lines written directly below its last setting, and ahead of any comment paragraph a blank line separates from that table, such as the comments that close the file. +- Comment text is kept as written, so a comment may still name an old setting. +- A document with Windows line endings keeps them. + +Migrating a migrated document changes nothing, because each rule matches only a shape the stack refuses, a spelling it reads only as deprecated, such as `[ec] passphrase`, or a document written for main, and the migrated document is none of those. An item left to do changes nothing in the document, so a second run reports it again and changes nothing. + +### 2.4 What the command never does + +- It never prints a value from the document. Every line names the setting. The validation result gives the byte offset where reading failed or the phase that refused, as `ts audit generate` does, because a refusal can quote a value, and `ts config validate` prints the refusal whole. +- It never removes a setting the operator still has to decide on, such as a handler whose pattern can match a path the stack does not close. +- It never sets `[geo] assume_single_jurisdiction`, because the stack asks the operator to accept that every request takes the top of the `permissions.yaml` rules rather than have that assumed. +- It never names `service_version` or selects `headers.geo`, and it never selects the Next.js module for a document that did not enable `[integrations.nextjs]`. It changes a value the document holds only where the stack renamed what the value names, which is a module, a middleware, a signer, an implementation or a demand source. +- It never writes the file without `--write`, and never writes it when nothing changes. +- It never chooses a permission policy and never writes a policy file. No policy ships with Trusted Server, and the tool carries none. + +## 3. The adapters + +The command runs in the tool and does the same on every machine, but the document it writes runs on one adapter, and main's location lookup and location headers differed by adapter. Only the location settings depend on `--adapter`. + +| Adapter | Main's location lookup | What main's responses carried | What a document written for main gets | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Fastly | The platform's lookup, on every request | The `x-geo-*` headers with the reader's location, or `x-geo-info-available: false` where none was found (main `crates/trusted-server-adapter-fastly/src/middleware.rs` lines 202 to 214) | `[geo] module = "platform"`, and a check line saying how to keep the headers | +| Cloudflare | Cloudflare's request headers, on every request (main `crates/trusted-server-adapter-cloudflare/src/platform.rs` lines 668 to 684) | `x-geo-info-available` alone, `true` where `cf-ipcountry` was present and not `XX` (main `crates/trusted-server-adapter-cloudflare/src/middleware.rs` lines 139 to 147) | `[geo] module = "platform"`, a check line saying that the lookup now reads the reader's region, and one saying that `headers.geo` would add the reader's location | +| Axum | None, as the lookup answered nothing (main `crates/trusted-server-adapter-axum/src/platform.rs` lines 183 to 187) | `x-geo-info-available: false` (main `crates/trusted-server-adapter-axum/src/middleware.rs` lines 133 to 137) | No geo module, an item left to do for the Edge Cookie module, and a check line on the header | +| Spin | None, as the lookup answered nothing (main `crates/trusted-server-adapter-spin/src/platform.rs` lines 757 to 761) | `x-geo-info-available: false` (main `crates/trusted-server-adapter-spin/src/middleware.rs` lines 167 to 175) | As on Axum | + +The stack's Axum and Spin adapters refuse `[geo] module = "platform"` at startup naming the setting, because neither host has a lookup (`refuse_platform_geo` in `crates/trusted-server-adapter-axum/src/app.rs` and in `crates/trusted-server-adapter-spin/src/app.rs`, CHANGELOG line 42). Every document main ran with has an Edge Cookie passphrase (main core `settings.rs` lines 545 and 625 to 633), which becomes `[ec] module = "hmac"`, and startup on every adapter refuses an Edge Cookie module with no geo module unless `[geo] assume_single_jurisdiction = true` accepts that every request takes the top of the `permissions.yaml` rules. On Axum and Spin the operator chooses between that acceptance and a geo module a linked crate offers, so the command leaves the choice to do. Writing `platform` there would report a document as passing that the adapter refuses, and writing the acknowledgment would make the choice the stack asks the operator to make. + +The rest of the document is the same on every adapter. Only the Fastly adapter sends the telemetry an `[analytics]` selection names, as only Fastly's did on main (CHANGELOG lines 20 and 48), and the Cloudflare and Spin adapters need their key-value store (§6). + +A deployment that ran one document on two adapters, as main's quick start pushes one `trusted-server.toml` to the Axum dev server and to Fastly (main `docs/guide/getting-started.md` lines 46 to 88 and 197), runs the command once for each adapter and keeps a document for each. The Axum dev server refuses the `[geo] module = "platform"` the Fastly document gets, and the Axum document, with no geo module, would leave Fastly without the lookup main ran there. + +## 4. Consent and privacy + +- **No value is printed.** Every line on stderr names a setting and holds no value from the document, because a document can hold secrets, and the validation result gives only the byte offset or the phase that refused. Lines do carry table names the operator chose, such as an integration id or a bidder code (the test `no_line_carries_a_value_from_the_document`). +- **No location header is added.** The command never selects `headers.geo`, which writes the reader's location in the `x-geo-*` headers on the responses a `[[serve]]` entry covers. Main wrote those headers on Fastly alone, and on Cloudflare wrote only `x-geo-info-available`, so selecting the middleware for every document written for main would send the reader's location on Cloudflare, where main sent none. Whether pages carry the reader's location is the operator's decision, as the stack made it one (CHANGELOG line 41), so the check line for each adapter says what main's responses carried and, on Fastly, how to keep the headers. +- **The location the consent rules see.** On Fastly and Cloudflare the document keeps the lookup main ran, so the `permissions.yaml` rules see the location main's jurisdiction check saw, where without a geo module every request would take the top of those rules. Main had no `permissions.yaml`. Its check compared the country with `[consent.gdpr] applies_in` and the region with `[consent.us_states] privacy_states` (main core `consent/jurisdiction.rs` lines 41 to 79), and the jurisdiction it found fed the Edge Cookie consent check and the OpenRTB `regs.gdpr` value (main core `ec/consent.rs` lines 11 to 22, `auction/openrtb.rs` lines 652 to 665). On Cloudflare the stack's lookup also reads the region, from `cf-region-code` where Cloudflare sends it, where main's lookup read none (`CloudflareGeo` and `build_geo` in `crates/trusted-server-adapter-cloudflare/src/platform.rs`), so the region rules of `permissions.yaml` can apply there now, such as the `us-state` jurisdiction the repository's sample gives each US state it lists where the country gives `non-regulated` (`config/permissions/sample.yaml` lines 421 to 427), and a check line printed with the geo rewrite says so. On Axum and Spin main located no request and the document locates none, and an Edge Cookie module runs there only once the operator accepts that with `[geo] assume_single_jurisdiction = true` or selects a geo module. +- **Passwords.** The command removes a `[[handlers]]` entry only where every path its pattern can match is one the stack closes, read as the paragraph below the §2.2 table says, so a publisher path such as `/admin/keystore` never loses its password without an item left to do, whether the pattern is `^/admin/keys`, `^/admin/keys/?` or one whose character class hides another alternative. The tests check each pattern they name against main's matching and the stack's closed paths (§8). +- **Telemetry.** A telemetry row is sent only for a reader whose permissions include `analytics.ad_reporting.measure_ad_performance`, and a check line says so where `[tinybird]` was enabled. +- **The consent lists.** The places the GDPR and US state rules apply are the `jurisdiction` values in the policy file the document names, so a change the operator made to `applies_in` or `privacy_states` is an item left to do until they carry it into that file and remove the table. + +## 5. Failure + +- **A file that cannot be read, written or parsed.** The command exits 2. A file that is not TOML is refused naming the line reading stopped at and nothing from it (`migrate` in `rules.rs`, and the test `no_line_carries_a_value_from_the_document`). `--write` rewrites the file in place with one write, so a write that fails part of the way can leave the file damaged, which is one reason §6 asks the operator to keep the original. +- **An item left to do, or a result that does not pass validation.** The command exits 1, and `--write` writes the result anyway, because the document did not load before either and the items say what is left. +- **The checks made only at startup.** The command's validation, like `ts config validate`, leaves out the checks core makes only once the settings load, which §2.1 lists, and those each adapter makes as it builds its state. Of those it reports the location settings, and a document that names no policy, as items left to do, for any document. The policy file itself is checked by `ts config validate --permissions` and by `ts config push`, which always checks it (§2.1). The others, such as a module name that no linked crate supplies, show only when the deployment starts. + +## 6. Compatibility and rollout + +1. **The migrated document goes live with the new build.** A build from before this change refuses it, as main refuses a table it does not know (main core `settings.rs` line 2958), and this build refuses the old one, so no one document starts both builds (CHANGELOG line 72, and the changelog entry that removes HTTP Basic authentication). +2. **On Fastly, stage the pair first**, with `ts deploy --staging` and `ts config push --staging --permissions `. Under EdgeZero `v0.0.10` a staged push writes the key its target's `__KEY` names, which on Fastly must be the logical store ID (`resolve_config_store_and_key` in its `config.rs`), so the staged pair is kept apart by the store the staging target's `__NAME` selects, which holds the staged blob with its policy file inside it. A service that chooses its blob by host pushes each host's staged blob with `--key ` and `__NAME` naming the staging store, never with `--staging`, which EdgeZero refuses beside `--key` (`2026-10-15-fastly-per-host-selector-design.md` section 2.5). That push writes the host's policy entries into the same store before the blob (`run` in `crates/trusted-server-cli/src/commands/config/keyed_push.rs`). +3. **Keep the original document before `--write`.** A rollback restores it with the old build (CHANGELOG line 72), and it holds the `[request_signing]` store ids the command removes and never prints, which `ts keys rotate` and `ts keys deactivate` take as `--config-store-id` and `--secret-store-id`. +4. **Rename the overlay variables.** `ts config push` ignores a variable for a key the document does not hold, so `TRUSTED_SERVER__PUBLISHER__ORIGIN_URL` and `TRUSTED_SERVER__INTEGRATIONS____*` set nothing once the document is migrated. They become `TRUSTED_SERVER__SOURCE__ORIGIN__URL` and `TRUSTED_SERVER__
____*` (CHANGELOG lines 26, 56 and 72). +5. **Keep the key-value store on Cloudflare and Spin.** A Worker without the `TRUSTED_SERVER_KV` binding, or a Spin component that cannot open `default`, now fails every request before any route runs (CHANGELOG line 46). Main's `wrangler.toml` declares the binding and main's `spin.toml` grants the store, so a deployment that kept those files has both. +6. **On Cloudflare, push the blob to the config namespace.** The Worker reads its blob from the KV namespace that `EDGEZERO__STORES__CONFIG__TRUSTED_SERVER_CONFIG__NAME` names, by default `trusted_server_config`, which `ts config push` writes, and no longer reads the `TRUSTED_SERVER_CONFIG` variable (`load_startup_settings` in the Cloudflare adapter's `app.rs`, as row 4, IABTechLab/trusted-server#1345, builds it). Declare that namespace in `wrangler.toml` as a namespace of its own beside `TRUSTED_SERVER_KV`, push the migrated document with `ts config push --adapter cloudflare --permissions `, and remove the variable (`2026-07-30-permission-model-design.md` §3.1, item 5). +7. **Expect the stored pages to be fetched again** once after the upgrade, because their keys hold a code worked out from the settings (CHANGELOG line 26). Core holds no copy of a file, and the asset cache module's keys hold no such code. + +The CLI guide's section "Upgrade a configuration written for an earlier release" gives every step but step 6, in the order an operator takes them. It runs the command first, and then has the operator do each `to do:` item, rename the overlay variables, run `ts config validate --permissions ` and push with the new build. The Cloudflare guide (`docs/guide/cloudflare.md`) says how to bind the config namespace and push into it. + +## 7. What is not in this change + +- The breaking changes for crates outside core, for the `ts` tool and for request handling, as the table says. +- Pushing the result. The operator runs `ts config validate` and then `ts config push`, each with `--permissions` naming the publisher's policy file, as the CLI guide's steps say (§6). +- A write through a temporary file, which would keep the original whole when a write fails part of the way. §6 asks the operator to keep the original instead. +- Rewriting comment text, which would mean guessing what a comment refers to. +- Reading what a `[[handlers]]` pattern means rather than its text, which would also remove a pattern such as `^/admin/keys/?$` whose every path is closed. The command leaves such a pattern to do, and the operator removes it. +- Earlier shapes of the stack that neither the changelog nor the settings code names. + +## 8. Tests + +- One test for each rewrite rule in the table, each on a small document, checking the result and the line it reports. +- One test for each item left to do, checking that the setting is left in place and that the line names it. Whether a run fails follows from whether any item is left, so one test, `an_item_left_to_do_fails_the_run`, runs the command on a document with an item left and checks that the run fails. Naming the policy has `a_document_that_names_no_policy_leaves_naming_one_to_do`, which also checks that the command adds no `[permissions]` and that the migrated document passes the command's validation, and `write_rewrites_the_file_and_the_default_prints_the_document` checks that a default run on a document written for main therefore fails, with that item reported and the migrated document passing validation. `a_document_written_for_main_that_names_its_policy_migrates_as_main` checks that such a document with `[permissions] policy` added makes the same rewrites, leaves the same items apart from naming the policy, prints the same check lines and gives the same document apart from the policy's table, and that the result holds `[source]` and passes validation. The other tests that check nothing is left to do leave that item aside (`left_to_do` in the tests). +- A golden test that migrates main's own `trusted-server.example.toml`, copied from `upstream/main` byte for byte into `crates/trusted-server-cli/tests/fixtures/main-trusted-server.example.toml` under a header that names the commit. The test replaces its placeholder domain and origin, as an operator's document would have them. The result loads and passes validation under the stack's settings code, and migrating it again changes nothing. The result names no policy, so the run also reports naming the policy file as an item left to do, which the test leaves aside, and the result passes the command's validation, which leaves the policy check out (§2.1). +- `the_location_lookup_is_the_one_each_adapter_accepts`, which migrates a document written for main for each adapter. On Fastly and Cloudflare the result selects `platform` and passes core's startup check of the jurisdiction acknowledgment. On Axum and Spin it selects no geo module, that check refuses it and an item left to do says so, and once `[geo] assume_single_jurisdiction = true` is added the check passes and a second run leaves nothing to do but naming the policy file. +- `the_location_settings_startup_refuses_are_left_to_do_in_any_document`, for `[geo] module = "platform"` on Axum and Spin and for an Edge Cookie module with no geo module on every adapter. +- `no_location_header_is_selected_for_the_operator`, on every adapter, with the check line each adapter gets, and `the_region_the_cloudflare_lookup_now_reads_is_reported_to_check`, on every adapter. +- `a_handler_is_removed_only_where_every_path_it_matches_is_closed`, with the bounded and the unbounded `/admin/keys` patterns and those a quantifier after the prefix or the bound widens, and `a_handler_with_a_class_or_comment_after_its_prefix_is_left_to_do`. Both run each pattern through the `regex` crate main matched with and through core's `closed_path_response` on a set of request paths, so each pattern the command removes is shown to guard no path the stack forwards, each it keeps as unsafe is shown to guard one, and `^/admin/keys/?$` and `^/_ts/admin/[a-z]+` are shown kept although they guard none. +- `a_host_header_override_is_reported_to_check`, and `a_document_in_the_stack_shape_with_no_page_entries_gets_none_added`. +- A test that the comments of a document survive and its tables keep their order, and one that a document with Windows line endings keeps them. +- A test that no line the command prints carries a value from the document, on every adapter, run on a document whose every value is a marker string. +- Tests of the command itself, that `--check` writes nothing and exits 1 when the document needs migrating and 0 for a migrated document that names a policy, that `--write` writes the file, and that a file that is not TOML exits with an error that quotes nothing from it, and a test that the arguments need `--adapter` and the file and refuse an unknown adapter and `--write` with `--check`. +- A test that the page entries name the middleware in the order `trusted-server.example.toml` shows them. + +No test starts the Axum or Spin adapter with a migrated document, because the tool links no adapter crate. Their refusal of `platform` is covered by the migrated document selecting no geo module on either. + +## Revision record + +| Date | Change | +| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-10-17 | First draft. | +| 2026-10-18 | Fixes from review round 1. `--adapter`, with the location lookup each adapter accepts and the Edge Cookie and geo settings startup refuses left to do. No location header is selected, a handler is removed only where every path its pattern matches is closed, a host header is reported to check, and page entries are added to a document written for main alone. The flags, the validation's reach, the consent lists, the adapters, consent and privacy, failure, compatibility and rollout, the tests, and 39 breaking entries. Fixes from review round 2. A handler pattern with a quantifier straight after its prefix or bound, or a character class or comment after them, is left to do, and a check line names the region the Cloudflare lookup now reads. The checks made only at startup, the location main's consent saw, a document run on two adapters, the stored assets, the passwords, the tests and the status line. | +| 2026-10-27 | The permission policy is a file the settings name and the push supplies. A document that names none is an item left to do, whose line names the file's stem and is the same on every adapter, a document written for main may name its policy before or after it is migrated, the command never chooses a policy or writes a policy file, and its validation leaves the policy check out and follows the pattern rule for `permissions.sha256` and `permissions.text`. The consent lists are carried into the publisher's file, as the command's line says. The count of breaking entries is 41, and the Cloudflare config store has a row. The status names IABTechLab/trusted-server#1409, whose code the references cite, the EdgeZero version is `v0.0.10`, the settings lines are brought up to date, the staged push is described as `v0.0.10` does it, a host's blob is staged with `--key`, and §6 gains the step for Cloudflare's config namespace. Step 7 of §6 says only the stored pages are fetched again, because core holds no copy of a file once the asset cache is a module. | + +## Review log + +1. **Round 1, 2026-10-18.** A review in a fresh session, against `split/51-config-migrate` at `50c68be1b`. Issues found, four of them blocking. The four were that the Axum and Spin adapters refuse the `[geo] module = "platform"` the command wrote into every document written for main, that the `[headers]` rewrite started sending the reader's location on Cloudflare, where main sent none, that the `[[handlers]]` rule removed a handler on `^/admin/keys`, which also guarded publisher paths such as `/admin/keystore` that the stack forwards to the origin, and that the spec had no compatibility and rollout section. Fixed in `18b7c7d16`, where the command selects `platform` only where the host has a lookup and leaves to do, for any document, an Edge Cookie module with no geo module and `platform` on an adapter with none, in `4465e2ad0`, where it takes `--adapter`, never selects `headers.geo` and prints a check line saying what main's responses carried, in `65bb09b14`, where it removes a handler only where every path its pattern can match is closed, and in this revision, which adds the adapters, consent and privacy, failure, and compatibility and rollout. The non-blocking findings are fixed too, with a host header reported to check in `65814d561` and page entries added to a document written for main alone in `f8dffebdc`. In this revision the consent lists stay items left to do with the definition of `to do:` widened to say so, the seam probe and the key-value store have rows of their own, and the count of breaking entries is 39. The review's statement that a selected `[ads-txt]` with no entry loads with a warning is not borne out, because the module refuses that document (`crates/ads-txt/iabtl/src/lib.rs` lines 154 to 175), so the `/ads.txt` entry is still added to any document. +2. **Round 2, 2026-10-18.** A review in a fresh session, against `split/51-config-migrate` at `6e3c09200`, of the round 1 fixes and the new text. Issues found, one of them blocking. The one was that a quantifier after the `^/admin/keys` bound or the `^/_ts/admin` prefix still let the command remove a handler that guarded publisher paths, with a `changed:` line and no `to do:`, as `^/admin/keys/?`, `^/admin/keys/*` and `^/admin/keys(?:/|$)?` matched `/admin/keystore` and `^/_ts/admin?` matched `/_ts/admit` under main's search. Fixed in `83dd4f1db`, where the command leaves such a pattern to do when `?`, `*` or `{` follows the prefix or the bound, and in `074d454e3`, where it also leaves to do a pattern with a `[` or a `#` after the prefix or the bound, because a character class or a comment the `(?x)` flag allows could hide an alternative such as `|^/members` from its scan, a fault the review did not name, found while checking which other text widens the match. The tests run each pattern through the `regex` crate and core's closed paths. The non-blocking findings are fixed too, with a check line on Cloudflare for the region its lookup now reads in `eb25e6d16`, and in this revision, which names the checks made only at startup, says the consent rules see the location main's jurisdiction check saw through `applies_in` and `privacy_states`, says a document run on two adapters is migrated once for each, adds the step on stored pages to the CLI guide, gives the comment on items left to do the spec's definition, and drops "the top pull request" from the status line. The round 1 commit hashes stay as they are, because `split/51-config-migrate` is merged forward and never rebased, so they resolve once it is pushed. +3. **Round 3.** A review in a fresh session, against `split/51-config-migrate`. Issues found, two of them blocking. The two were that the design had the command's validation check the policy against the files `--permissions` supplies, when the command takes no `--permissions`, so that every document, and every run of `ts audit generate`, which shares that validation, would fail, and that the header said the pull request was not yet raised, when IABTechLab/trusted-server#1409 was open. This revision keeps the policy checks out of the command's validation, has the command write a `to do:` line for a document that names no policy, naming the policy by the file's stem and, on Axum, the folder the file goes in, and never choose a policy or write a policy file, and names IABTechLab/trusted-server#1409 in the header. It also marks the new wording of the consent lists' `to do:` line as the design that row 4 builds, describes the staged push as EdgeZero `v0.0.10` makes it, and brings the line references up to date. +4. **Round 4.** A review in a fresh session, against `split/51-config-migrate`, of the round 3 fixes and the new text. Approved, none blocking. This revision has the command's validation, like that of `ts audit generate`, count `permissions.sha256` as present and refuse a pattern under `permissions.text` once row 4 builds the policy design, as every pattern check then does, drops a citation of a changelog entry that says a staged deploy reads a configuration key of its own, which EdgeZero `v0.0.10` does not do, has a service that chooses its blob by host stage each host's blob with `--key`, and adds the step in section 6 that pushes Cloudflare's blob to its config namespace. +5. **Round 5.** A review in a fresh session, against `split/51-config-migrate`, with main read at `5bc388625`. No issues found. The round found every citation by file and function at the row's head, the forty-one entries marked breaking in the changelog and each line it cites, the lines it cites on main, and that the command's validation reaches the one pattern check through the deploy validation it shares with `ts audit generate`, while the policy check stays out of it because the command takes no `--permissions`. Before this round the text was brought into line with the row's code, so the line left to do for the policy file is the same on every adapter and names no folder on Axum, as round 3 read it, and a document written for main may name its policy before or after it is migrated.