diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d2a58986b..b3d17a3ec5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -94,6 +94,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `[shared.]`, one table per vendor holding what every module of that vendor shares, read by the vendor's crates through `Settings::shared_config` and by nothing in core. A builder marked with `IntegrationBuilder::with_shared_settings` runs when its vendor's table is present and is selected by no section, a table no builder reads is refused when the registry is built, a table whose name is not lower case letters, digits, `_` or `-` is refused when the settings load, a refusal of a module nothing supplies names the absent table, and a secret in the table is declared, resolved and checked at push to name a key as a module's own are. A shared table, or a module's own, that cannot be read or fails its validation is refused naming the setting at fault and never its value. - `robots_txt`, core's own middleware for the file `[robots-txt]` describes, named in a `[[fetch]]` entry for `text/plain` on `/robots.txt`. Its preparation makes the file from the selected modules through their holding, and its action replaces the origin's file with it. The file is made in place of any other answer the origin gives to a `GET` or `HEAD`, such as a `404`, a page, an error or a redirect. The origin is asked for its whole file, without a crawler's conditional or range headers, and the middleware works on `/robots.txt` alone. A module that cannot answer with nothing held refuses the request with `503` and `Retry-After` before any document is processed. A section with no such entry, or with an entry naming the middleware on another path or media type, is refused when the settings load. - A middleware may say which request paths it works on, with `Middleware::handles_path`, that it writes the whole document, with `writes_whole_document`, so core makes the document in place of any answer it would not hand the middleware as a `200` of the entry's media type, and that it needs the origin's whole body, with `needs_whole_body`, so the origin is asked without the reader's conditional and range headers. By default a middleware works on every path its entry covers and says neither of the other two. - A middleware may prepare, and may refuse. `Middleware::prepare` runs in the request's handler before the document is processed, as the middleware's module with the request's services, and what it answers is kept under the phase and the middleware's name for `create` in that phase, so a decision that needs a store or the network is made there while `create` stays synchronous. The fetch phase of a document that may be stored is prepared with no client IP, TLS details or client signals in the services. A refusal answers the request with its status and `Retry-After`, with no document, and nothing is stored. Core logs a refusal as a warning naming the middleware, and answers a status that is not a client or server error with `500`. diff --git a/crates/testing/seam-probe/src/lib.rs b/crates/testing/seam-probe/src/lib.rs index 7b564c537d..10b3d97230 100644 --- a/crates/testing/seam-probe/src/lib.rs +++ b/crates/testing/seam-probe/src/lib.rs @@ -824,6 +824,120 @@ pub fn builder() -> IntegrationBuilder { .with_adserver(&SEAM_PROBE_ADSERVER) } +/// The vendor's name, which names the table `[shared.seam-probe]` that runs +/// [`shared_builder`]. +pub const SEAM_PROBE_SHARED_VENDOR: &str = "seam-probe"; + +/// The id of the builder the shared table runs. +pub const SEAM_PROBE_SHARED_ID: &str = "seam_probe_shared"; + +/// The route the shared builder answers with the table's marker. +pub const SEAM_PROBE_SHARED_PATH: &str = "/integrations/seam_probe_shared/marker"; + +/// The shared route's suffix under the builder's proxy prefix. +const SHARED_ROUTE_SUFFIX: &str = "/marker"; + +/// What `[shared.seam-probe]` carries. +#[derive(Debug, Clone, Deserialize, Validate)] +#[serde(deny_unknown_fields)] +pub struct SeamProbeShared { + /// The text the shared route answers with. + #[validate(length(min = 1))] + pub marker: String, +} + +/// A route answering the shared table's marker, which proves the table ran +/// the builder and reached what it registered. +#[derive(Debug)] +struct SharedMarkerRoute { + marker: String, +} + +impl SharedMarkerRoute { + async fn route( + &self, + _req: Request, + ) -> Result, Report> { + Response::builder() + .status(200) + .header( + header::CONTENT_TYPE, + HeaderValue::from_static("text/plain; charset=utf-8"), + ) + .body(EdgeBody::from(self.marker.clone().into_bytes())) + .map_err(|error| { + Report::new(TrustedServerError::Integration { + integration: SEAM_PROBE_SHARED_ID.to_string(), + message: error.to_string(), + }) + }) + } +} + +#[async_trait(?Send)] +impl IntegrationProxy for SharedMarkerRoute { + fn integration_name(&self) -> &'static str { + SEAM_PROBE_SHARED_ID + } + + fn routes(&self) -> Vec { + vec![self.get(SHARED_ROUTE_SUFFIX)] + } + + async fn handle( + &self, + call: ModuleCall<'_>, + req: Request, + ) -> Result, Report> { + call.inject_with(self, req, Self::route)?.await + } +} + +/// Registers the shared route when `[shared.seam-probe]` is present. +/// +/// # Errors +/// +/// Returns an error when the table cannot be read or fails its validation. +pub fn register_shared( + settings: &Settings, +) -> Result, Report> { + let Some(shared) = settings.shared_config::(SEAM_PROBE_SHARED_VENDOR)? else { + return Ok(None); + }; + Ok(Some( + IntegrationRegistration::builder(SEAM_PROBE_SHARED_ID) + .without_js() + .with_proxy(Arc::new(SharedMarkerRoute { + marker: shared.marker, + })) + .build(), + )) +} + +/// Reports whether the shared table is present, validating it. +/// +/// # Errors +/// +/// Returns an error when the table cannot be read or fails its validation. +pub fn validate_shared(settings: &Settings) -> Result> { + Ok(settings + .shared_config::(SEAM_PROBE_SHARED_VENDOR)? + .is_some()) +} + +/// The builder `[shared.seam-probe]` runs, which proves a vendor's shared +/// table from outside core. +#[must_use] +pub fn shared_builder() -> IntegrationBuilder { + IntegrationBuilder::new( + SEAM_PROBE_SHARED_ID, + SEAM_PROBE_SOURCE, + register_shared, + validate_shared, + ) + .with_shared_settings(SEAM_PROBE_SHARED_VENDOR) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/trusted-server-adapter-axum/tests/seam_probe.rs b/crates/trusted-server-adapter-axum/tests/seam_probe.rs index 760a6aa6aa..3765eae991 100644 --- a/crates/trusted-server-adapter-axum/tests/seam_probe.rs +++ b/crates/trusted-server-adapter-axum/tests/seam_probe.rs @@ -258,6 +258,48 @@ async fn proxy_route_reports_the_modules_geo_and_that_the_preparer_ran() { ); } +/// `[shared.seam-probe]` runs the builder the probe shares, whose route +/// answers the table's marker. Without the table the builder is not built, +/// and the table with no builder reading it is refused at startup. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_vendors_shared_table_runs_its_builder() { + let mut with = service_with( + settings_with(&format!( + "[shared.seam-probe]\nmarker = \"from the shared table\"\n{PROBE_BLOCK}" + )), + &[seam_probe::builder(), seam_probe::shared_builder()], + ); + let response = get(&mut with, seam_probe::SEAM_PROBE_SHARED_PATH).await; + assert_eq!( + response.status().as_u16(), + 200, + "the table runs the builder" + ); + assert_eq!(body_text(response).await, "from the shared table"); + + let mut without = service_with( + settings_with(PROBE_BLOCK), + &[seam_probe::builder(), seam_probe::shared_builder()], + ); + let response = get(&mut without, seam_probe::SEAM_PROBE_SHARED_PATH).await; + assert_ne!( + response.status().as_u16(), + 200, + "without the table the builder is not built, so the route is not this server's" + ); + + let refused = TrustedServerApp::routes_with_registrations( + settings_with("[shared.seam-probe]\nmarker = \"x\"\n"), + &[seam_probe::builder()], + ) + .err() + .expect("nothing reads the table"); + assert!( + format!("{refused:?}").contains("[shared.seam-probe] is read by no module"), + "{refused:?}" + ); +} + /// With `refuse_all` selected every response this adapter finalizes says not /// to index, and the file itself is the publisher path's, made by the /// `robots_txt` middleware the entry names, so no route of this server names diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index ff65ed7c2a..9519836c14 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -509,20 +509,36 @@ fn validate_secret_key_references( )?; } - // Each setting a selected module declares as naming a secret, where the - // module's table puts it to use. + // Each setting a running builder declares as naming a secret, where the + // builder's table puts it to use. A vendor's shared table runs the + // builders that carry its mark, and a section runs the module it selects. for builder in crate::integrations::all_builders(extra_integrations) { let secrets = builder.secret_settings(); - let Some((section, written)) = builder - .module_name() - .filter(|_| !secrets.is_empty()) - .and_then(|name| settings.module_selection(name)) - else { + if secrets.is_empty() { continue; + } + let (place, table) = if let Some(vendor) = builder.shared_settings() { + let Some(table) = settings.shared.table(vendor) else { + continue; + }; + ( + format!("{}.{vendor}", crate::settings::SHARED_TYPE), + table.clone(), + ) + } else { + let Some((section, written)) = builder + .module_name() + .and_then(|name| settings.module_selection(name)) + else { + continue; + }; + ( + format!("{section}.{written}"), + settings.section_table(section, written), + ) }; - let table = settings.section_table(section, written); for secret in secrets.iter().filter(|secret| (secret.in_use)(&table)) { - let path = format!("{section}.{written}.{}", secret.path.join(".")); + let path = format!("{place}.{}", secret.path.join(".")); let reference = crate::module_secrets::value_at(&table, secret.path) .and_then(serde_json::Value::as_str) .ok_or_else(|| missing_secret_key_reference(&path))?; @@ -1736,6 +1752,63 @@ passphrase = "production-secret-key-32-bytes-min" } } + /// A builder its vendor's shared table runs, which declares a secret in + /// that table. + fn shared_builder_with_a_secret() -> IntegrationBuilder { + IntegrationBuilder::new( + "probe", + "example-crate", + crate::integrations::registry_test_support::probe_registration, + crate::integrations::registry_test_support::validate_nothing, + ) + .with_shared_settings("probe") + .with_secret_settings(MODULE_SECRETS) + } + + #[test] + fn deploy_validation_refuses_a_shared_table_s_secret_setting_in_use_that_names_no_key() { + for (table, expected) in [ + ( + Some(serde_json::json!({ "lock": true })), + Some("shared.probe.spare.key_name"), + ), + ( + Some(serde_json::json!({ "lock": true, "spare": { "key_name": " " } })), + Some("shared.probe.spare.key_name"), + ), + ( + Some(serde_json::json!({ "lock": true, "spare": { "key_name": "vendor_key" } })), + None, + ), + // A setting the table does not put to use is not checked. + (Some(serde_json::json!({ "lock": false })), None), + // Without its table the builder does not run. + (None, None), + ] { + let mut settings = valid_settings(); + if let Some(table) = &table { + settings.shared = serde_json::from_value(serde_json::json!({ "probe": table })) + .expect("should read the shared table"); + } + + let result = + validate_settings_for_deploy_with(&settings, &[shared_builder_with_a_secret()]); + + match expected { + Some(path) => { + let error = result.expect_err("should refuse a setting that names no key"); + assert!( + format!("{error:?}").contains(path), + "should name the setting for {table:?}: {error:?}" + ); + } + None => result.unwrap_or_else(|error| { + panic!("should accept {table:?}: {error:?}"); + }), + } + } + } + const PLAN_RULE_MESSAGE: &str = "probe module refuses an enabled auction"; fn refuse_an_enabled_auction( diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index 8795c29505..0bc11f499e 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -424,6 +424,7 @@ pub struct IntegrationBuilder { module: Option<&'static str>, section: Option<&'static str>, loading_cache: Option, + shared: Option<&'static str>, } /// Gives the loading cache a module supplies, see @@ -469,6 +470,7 @@ impl IntegrationBuilder { module: None, section: None, loading_cache: None, + shared: None, } } @@ -497,6 +499,7 @@ impl IntegrationBuilder { module: None, section: None, loading_cache: None, + shared: None, } } @@ -639,6 +642,23 @@ impl IntegrationBuilder { self } + /// Runs this builder when `[shared.]` is present, for a vendor + /// whose modules all read that one table and cannot be built without it. + /// A builder the table runs is selected by no section, so the registry + /// refuses one that also has a module name. + #[must_use] + pub const fn with_shared_settings(mut self, vendor: &'static str) -> Self { + self.shared = Some(vendor); + self + } + + /// The vendor whose `[shared.]` table runs this builder, when one + /// does. + #[must_use] + pub const fn shared_settings(&self) -> Option<&'static str> { + self.shared + } + /// The loading cache this builder offers, when it offers one. #[must_use] pub const fn loading_cache(&self) -> Option { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 5bb11d0601..e422b4a66c 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -908,6 +908,8 @@ struct IntegrationRegistryInner { // Every builder's module name, with whether a section selects it, so the // refusal of a name a selector wrote can say what that name is. builder_modules: Vec<(&'static str, bool)>, + // Vendors whose shared table is absent, for the refusal note. + absent_shared: Vec<&'static str>, // The module `[geo] module` resolved to, or `None` when the selector // is `platform` and the adapter's own host lookup stands. Unset and `none` // both resolve the disabled module. @@ -944,6 +946,7 @@ impl Default for IntegrationRegistryInner { device_modules: Vec::new(), robots_txt_contributors: Vec::new(), builder_modules: Vec::new(), + absent_shared: Vec::new(), geo_module: None, ec_module: None, device_module: None, @@ -1027,6 +1030,27 @@ impl IntegrationRegistryInner { ) } } + + /// A sentence naming the shared tables whose absence kept their vendors' + /// modules from running, to follow a refusal of a module nothing supplies, + /// or nothing when every builder a shared table runs is running. + fn absent_shared_note(&self) -> String { + if self.absent_shared.is_empty() { + return String::new(); + } + let tables = self + .absent_shared + .iter() + .map(|vendor| format!("[shared.{vendor}]")) + .collect::>() + .join(" or "); + let those = if self.absent_shared.len() == 1 { + "that table runs" + } else { + "those tables run" + }; + format!(". There is no {tables}, so the modules {those} are not running") + } } /// The module a name written in a section means, among those the running @@ -1092,8 +1116,9 @@ fn named_module( Err(Report::new(TrustedServerError::Configuration { message: format!( "`[{section}] module` names `{written}`, which no module this deployment runs \ - supplies. {runs}{}", - inner.builder_note(type_folder, section, written) + supplies. {runs}{}{}", + inner.builder_note(type_folder, section, written), + inner.absent_shared_note() ), })) } @@ -1461,6 +1486,7 @@ pub struct IntegrationRegistry { fn check_section_selections( settings: &Settings, extra: &[crate::integrations::IntegrationBuilder], + inner: &IntegrationRegistryInner, ) -> Result<(), Report> { let offered: Vec<&'static str> = crate::integrations::all_builders(extra) .filter_map(|builder| builder.module_name()) @@ -1495,7 +1521,8 @@ fn check_section_selections( return Err(Report::new(TrustedServerError::Configuration { message: format!( "[{section}] selects `{written}`, which no module this deployment \ - supplies. {supplied}" + supplies. {supplied}{}", + inner.absent_shared_note() ), })); } @@ -1599,6 +1626,33 @@ impl IntegrationRegistry { let mut plan_registrations: Vec = Vec::new(); let mut registrations: Vec = Vec::new(); + // A vendor's shared table is read by that vendor's builder alone, so + // one no builder reads is a table the deployment cannot use. + let unread: Vec<&str> = settings + .shared + .vendors() + .filter(|vendor| { + !crate::integrations::all_builders(extra) + .any(|builder| builder.shared_settings() == Some(*vendor)) + }) + .collect(); + if !unread.is_empty() { + let read: Vec<&str> = crate::integrations::all_builders(extra) + .filter_map(|builder| builder.shared_settings()) + .collect(); + let reads = if read.is_empty() { + "It reads none".to_owned() + } else { + format!("The shared tables it reads are [{}]", read.join(", ")) + }; + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "[shared.{}] is read by no module this deployment supplies. {reads}", + unread.join("] and [shared."), + ), + })); + } + for builder in crate::integrations::all_builders(extra) { if let Some((_, first_source)) = inner.builder_ids.iter().find(|(id, _)| *id == builder.id()) @@ -1612,6 +1666,19 @@ impl IntegrationRegistry { })); } inner.builder_ids.push((builder.id(), builder.source())); + // A builder its shared table runs answers to no section, so a + // module name would let a section select what it does not run. + if let (Some(vendor), Some(name)) = (builder.shared_settings(), builder.module_name()) { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "integration `{}` from `{}` is run by [shared.{vendor}] and also names \ + the module `{name}`. A builder its shared table runs is selected by \ + no section, so it takes no module name", + builder.id(), + builder.source() + ), + })); + } // Preparers are collected before the selection check, so an // integration can sanitize its own reserved query or cookie in a // deployment that does not run it. @@ -1644,7 +1711,19 @@ impl IntegrationRegistry { if let Some(name) = builder.module_name() { inner.builder_modules.push((name, selected)); } - if !selected { + // A builder a vendor's shared table runs is built when the table + // is present, whatever any section selects. + let runs = match builder.shared_settings() { + Some(vendor) => { + let present = settings.shared.contains(vendor); + if !present && !inner.absent_shared.contains(&vendor) { + inner.absent_shared.push(vendor); + } + present + } + None => selected, + }; + if !runs { continue; } @@ -1663,7 +1742,7 @@ impl IntegrationRegistry { // the selections are checked against them rather than in the // settings. Deploy validation deliberately does not make this check, // because a vendor crate the CLI never links may supply the name. - check_section_selections(settings, extra)?; + check_section_selections(settings, extra, &inner)?; // The geo, Edge Cookie and device module names taken so far, with the // integration that supplies each. @@ -2436,6 +2515,7 @@ impl IntegrationRegistry { device_modules: Vec::new(), robots_txt_contributors: Vec::new(), builder_modules: Vec::new(), + absent_shared: Vec::new(), geo_module: None, ec_module: None, device_module: None, @@ -2515,6 +2595,7 @@ impl IntegrationRegistry { device_modules: Vec::new(), robots_txt_contributors: Vec::new(), builder_modules: Vec::new(), + absent_shared: Vec::new(), geo_module: None, ec_module: None, device_module: None, @@ -4828,6 +4909,134 @@ mod tests { ); } + const SHARED_VENDOR: &str = "example"; + + fn shared_probe_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("shared_probe").build(), + )) + } + + /// The `shared_probe` builder, run by `[shared.example]`. + fn shared_probe_builder() -> crate::integrations::IntegrationBuilder { + crate::integrations::IntegrationBuilder::new( + "shared_probe", + "seam-probe", + shared_probe_registration, + validate_nothing, + ) + .with_shared_settings(SHARED_VENDOR) + } + + fn with_shared_table(mut settings: Settings) -> Settings { + settings.shared = + serde_json::from_value(serde_json::json!({ SHARED_VENDOR: { "key": "value" } })) + .expect("should read the shared tables"); + settings + } + + #[test] + fn a_builder_its_shared_table_runs_is_built_only_when_the_table_is_present() { + let extra = [shared_probe_builder()]; + + let without = IntegrationRegistry::with_registrations( + &crate::test_support::tests::create_test_settings(), + &extra, + ) + .expect("nothing is built, which is not an error"); + assert!(!without.integration_runs("shared_probe")); + + let with = IntegrationRegistry::with_registrations( + &with_shared_table(crate::test_support::tests::create_test_settings()), + &extra, + ) + .expect("should build the builder its table runs"); + assert!(with.integration_runs("shared_probe")); + } + + #[test] + fn a_shared_table_no_builder_reads_is_refused() { + let refused = IntegrationRegistry::with_registrations( + &with_shared_table(crate::test_support::tests::create_test_settings()), + &[], + ) + .err() + .expect("nothing reads the table") + .to_string(); + + assert!( + refused.contains("[shared.example] is read by no module"), + "{refused}" + ); + } + + #[test] + fn a_builder_its_shared_table_runs_answers_to_no_selection() { + let refused = IntegrationRegistry::with_registrations( + &with_shared_table(settings_naming("shared_probe")), + &[shared_probe_builder()], + ) + .err() + .expect("the table runs the builder, so no section can select it") + .to_string(); + + assert!(refused.contains("selects `shared_probe`"), "{refused}"); + } + + #[test] + fn a_selection_nothing_supplies_names_the_shared_table_that_is_absent() { + let mut settings = crate::test_support::tests::create_test_settings(); + settings.geo.module = Some("example".to_owned()); + + let refused = IntegrationRegistry::with_registrations(&settings, &[shared_probe_builder()]) + .err() + .expect("no geo module is supplied") + .to_string(); + + assert!( + refused.contains("There is no [shared.example]"), + "the message has to name the table to add: {refused}" + ); + } + + #[test] + fn a_list_selection_nothing_supplies_names_the_shared_table_that_is_absent() { + let mut settings = crate::test_support::tests::create_test_settings(); + settings.select_module("identity", "example.id"); + + let refused = IntegrationRegistry::with_registrations(&settings, &[shared_probe_builder()]) + .err() + .expect("no identity module is supplied") + .to_string(); + + assert!( + refused.contains("[identity] selects `example.id`") + && refused.contains("There is no [shared.example]"), + "the message has to name the table to add: {refused}" + ); + } + + #[test] + fn a_builder_its_shared_table_runs_is_refused_a_module_name() { + let mut settings = with_shared_table(crate::test_support::tests::create_test_settings()); + settings.geo.module = Some("example".to_owned()); + + let refused = IntegrationRegistry::with_registrations( + &settings, + &[shared_probe_builder().with_module_name("geo.example")], + ) + .err() + .expect("a section could select what the table runs") + .to_string(); + + assert!( + refused.contains("is run by [shared.example] and also names the module `geo.example`"), + "{refused}" + ); + } + #[test] fn with_registrations_rejects_a_duplicate_integration_id_naming_both_sources() { let settings = crate::test_support::tests::create_test_settings(); diff --git a/crates/trusted-server-core/src/module_secrets.rs b/crates/trusted-server-core/src/module_secrets.rs index b075d3fa98..24e9653fc6 100644 --- a/crates/trusted-server-core/src/module_secrets.rs +++ b/crates/trusted-server-core/src/module_secrets.rs @@ -15,10 +15,14 @@ use serde_json::{Map, Value}; use crate::integrations::IntegrationBuilder; use crate::module_name::short_form; -/// The section a module is selected in and the names its table can sit -/// under there: the name with the section's type folder left off, and the -/// name in full where the two differ. +/// The section a module's table sits under and the names it can be written +/// at there, being the name with the section's type folder left off and the +/// name in full where the two differ, or a vendor's shared table under +/// `shared`, at the vendor's name alone. fn written_forms(builder: &IntegrationBuilder) -> Option<(&'static str, Vec<&'static str>)> { + if let Some(vendor) = builder.shared_settings() { + return Some((crate::settings::SHARED_TYPE, vec![vendor])); + } let name = builder.module_name()?; let section = builder.section()?; let short = short_form(section, name); @@ -100,7 +104,12 @@ pub(crate) fn clear_unused(data: &mut Value, builders: &[IntegrationBuilder]) { let Some((section, forms)) = written_forms(builder) else { continue; }; - let selected = selected_as(data, section, &forms); + // A vendor's table runs its builder, so a secret in the table counts + // as in use by the table's own rules. + let selected = match builder.shared_settings() { + Some(vendor) => Some(vendor), + None => selected_as(data, section, &forms), + }; for form in &forms { let Some(table) = table_mut(data, section, form) else { continue; @@ -189,6 +198,58 @@ mod tests { .with_secret_settings(SECRETS) } + const VENDOR: &str = "example"; + + fn shared_builder() -> IntegrationBuilder { + IntegrationBuilder::new( + "shared_probe", + "seam-probe", + probe_registration, + validate_nothing, + ) + .with_shared_settings(VENDOR) + .with_secret_settings(SECRETS) + } + + #[test] + fn a_secret_in_a_shared_table_is_kept_when_the_table_is_present_and_in_use() { + let mut data = json!({ "shared": { VENDOR: table(true, false) } }); + + clear_unused(&mut data, &[shared_builder()]); + + assert_eq!( + data.pointer("/shared/example/key_name"), + Some(&json!("main-key")), + "the table runs the builder, so its setting in use stays" + ); + assert_eq!(data.pointer("/shared/example/spare/key_name"), None); + } + + #[test] + fn a_secret_in_a_shared_table_is_declared_under_the_vendors_name() { + let paths: Vec = secret_fields(&[shared_builder()]) + .iter() + .map(|field| { + field + .path + .iter() + .map(|segment| match segment { + SecretPathSegment::Field(name) | SecretPathSegment::OptionalField(name) => { + name.to_string() + } + _ => "*".to_owned(), + }) + .collect::>() + .join(".") + }) + .collect(); + + assert_eq!( + paths, + vec!["shared.example.key_name", "shared.example.spare.key_name"] + ); + } + fn table(lock: bool, spare: bool) -> Value { json!({ "lock": lock, diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index e84f299d2e..807cbf5fac 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -326,6 +326,91 @@ impl TypeSections { } } +/// The section that holds what one vendor's modules share. +pub const SHARED_TYPE: &str = "shared"; + +/// The `[shared]` section, one `[shared.]` table per vendor, holding +/// the settings every module of that vendor reads, such as an endpoint and a +/// license key, read through [`Settings::shared_config`]. +#[derive(Clone, Default, Deserialize, Serialize)] +#[serde(transparent)] +pub struct SharedTables(BTreeMap>); + +impl std::fmt::Debug for SharedTables { + // The tables hold secrets, so only the vendors are shown. + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("SharedTables") + .field("vendors", &self.vendors().collect::>()) + .finish() + } +} + +impl SharedTables { + /// Whether no vendor has a table. + #[must_use] + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } + + /// The vendors with a table, in name order. + pub fn vendors(&self) -> impl Iterator { + self.0.keys().map(String::as_str) + } + + /// Whether `[shared.]` is present. + #[must_use] + pub fn contains(&self, vendor: &str) -> bool { + self.0.contains_key(vendor) + } + + /// `[shared.]` as written, for a check that reads a setting by its + /// path, such as that a secret setting names a key. + #[must_use] + pub(crate) fn table(&self, vendor: &str) -> Option<&serde_json::Map> { + self.0.get(vendor) + } + + /// Reads and validates `[shared.]` as `T`, or `None` when the table + /// is absent. + /// + /// # Errors + /// + /// When the table cannot be read as `T` or fails its validation, naming + /// the table and the setting at fault but never the setting's value. + pub fn get_typed(&self, vendor: &str) -> Result, Report> + where + T: DeserializeOwned + Validate, + { + let Some(table) = self.0.get(vendor) else { + return Ok(None); + }; + read_table(&format!("{SHARED_TYPE}.{vendor}"), table.clone()).map(Some) + } + + /// Refuses a table whose name is not a vendor's name, being lower case + /// letters, digits, `_` or `-`. + /// + /// # Errors + /// + /// Naming the first table at fault. + pub fn validate_names(&self) -> Result<(), Report> { + match self + .vendors() + .find(|vendor| vendor.contains('.') || !crate::module_name::is_valid(vendor)) + { + Some(vendor) => Err(Report::new(TrustedServerError::Configuration { + message: format!( + "[{SHARED_TYPE}] has a table named `{vendor}`, which is not a vendor's \ + name. A vendor's shared table is named in lower case letters, \ + digits, `_` or `-`" + ), + })), + None => Ok(()), + } + } +} + /// Reads and validates a table as `T`, where `place` is the table's name as /// written, such as `shared.example`. /// @@ -4195,6 +4280,9 @@ pub struct Settings { pub consent: ConsentConfig, #[serde(default)] pub cache: CacheSettings, + /// What one vendor's modules share, a table per vendor, `[shared.]`. + #[serde(default, skip_serializing_if = "SharedTables::is_empty")] + pub shared: SharedTables, #[serde(default)] pub proxy: Proxy, #[serde(default)] @@ -4479,6 +4567,7 @@ impl Settings { .auction_html_comment_options .validate_metadata_keys()?; self.validate_asset_image_optimizer_profile_sets()?; + self.shared.validate_names()?; if let Some(robots_txt) = &self.robots_txt { robots_txt.prepare_runtime()?; self.require_robots_txt_placed()?; @@ -4770,6 +4859,20 @@ impl Settings { read_table(&format!("{section}.{written}"), table).map(Some) } + /// Reads `[shared.]`, the settings every module of `vendor` + /// shares, or `None` when the table is absent. + /// + /// # Errors + /// + /// When the table cannot be read as `T` or fails its validation, naming + /// the table. + pub fn shared_config(&self, vendor: &str) -> Result, Report> + where + T: DeserializeOwned + Validate, + { + self.shared.get_typed(vendor) + } + /// The modules `section` selects, to change, with the section created /// empty when it is a module type's and absent. fn section_modules_mut(&mut self, section: &str) -> &mut SectionModules { @@ -11140,6 +11243,178 @@ mod permission_signal_config_tests { "the refusal names the block and says why it is refused: {message}" ); } + + mod shared_tables { + use super::*; + + #[derive(Debug, Deserialize, Validate)] + #[serde(deny_unknown_fields)] + struct ExampleShared { + #[validate(url)] + endpoint: String, + } + + fn with(toml: &str) -> Result> { + Settings::from_toml(&format!("{}\n{toml}", crate_test_settings_str())) + } + + #[test] + fn a_vendor_reads_its_own_shared_table_and_nothing_else() { + let settings = with( + r#" + [shared.example] + endpoint = "https://api.example.com/v1" + "#, + ) + .expect("should load"); + + let read: ExampleShared = settings + .shared_config("example") + .expect("should read") + .expect("the table is present"); + assert_eq!(read.endpoint, "https://api.example.com/v1"); + assert!( + settings + .shared_config::("another") + .expect("an absent table reads as nothing") + .is_none() + ); + } + + /// A vendor's settings with a secret in them, as the table holds them + /// at startup, once the secret is resolved from the store. + #[derive(Debug, Deserialize, Validate)] + #[serde(deny_unknown_fields)] + struct SecretShared { + #[validate(length(max = 8))] + license_key: String, + #[validate(range(max = 10))] + retries: Option, + } + + const SECRET: &str = "resolved-secret-value-4f7a"; + + #[test] + fn a_shared_table_that_is_refused_names_the_setting_and_never_its_value() { + for (table, setting) in [ + // Fails a rule, whose parameters carry the value checked. + (format!("license_key = '{SECRET}'"), "license_key"), + // Cannot be read, where `serde` quotes the value it found. + ( + format!("license_key = 'short'\nretries = '{SECRET}'"), + "retries", + ), + ] { + let settings = with(&format!("[shared.example]\n{table}")) + .expect("core reads none of the table, so it loads"); + + let refused = settings + .shared_config::("example") + .expect_err("the table breaks the vendor's rules"); + + let shown = format!("{refused}\n{refused:?}"); + assert!( + !shown.contains(SECRET), + "printed a setting's value: {shown}" + ); + assert!( + shown.contains("[shared.example]") && shown.contains(setting), + "should name the table and the setting: {shown}" + ); + } + } + + #[test] + fn a_shared_table_its_vendor_cannot_read_is_refused_naming_the_table() { + let settings = with( + r#" + [shared.example] + endpoint = "https://api.example.com/v1" + unknown = true + "#, + ) + .expect("core reads none of the table, so it loads"); + + let refused = format!( + "{:?}", + settings + .shared_config::("example") + .expect_err("the vendor does not know the key") + ); + assert!( + refused.contains("[shared.example] could not be read") + && refused.contains("unknown"), + "{refused}" + ); + } + + #[test] + fn a_shared_table_failing_its_validation_is_refused_naming_the_table() { + let settings = with( + r#" + [shared.example] + endpoint = "not-a-url" + "#, + ) + .expect("core reads none of the table, so it loads"); + + let refused = format!( + "{:?}", + settings + .shared_config::("example") + .expect_err("the endpoint is not a URL") + ); + assert!( + refused.contains("[shared.example] failed validation"), + "{refused}" + ); + } + + #[test] + fn a_shared_table_not_named_as_a_vendor_is_refused() { + for table in ["[shared.Example]", "[shared.\"an.example\"]"] { + let refused = with(&format!("{table}\nendpoint = \"https://api.example.com\"")) + .expect_err("a vendor's table is named by its folder"); + + assert!( + format!("{refused:?}").contains("which is not a vendor's name"), + "{table}: {refused:?}" + ); + } + } + + /// The tables hold secrets, so a settings dump shows the vendors alone, + /// and the serialized settings carry the tables whole. + #[test] + fn a_shared_table_is_shown_by_its_vendor_and_serialized_whole() { + let settings = with( + r#" + [shared.example] + endpoint = "https://api.example.com/v1" + license_key = "a-secret-value" + "#, + ) + .expect("should load"); + + let shown = format!("{:?}", settings.shared); + assert!(shown.contains("example"), "{shown}"); + assert!(!shown.contains("a-secret-value"), "{shown}"); + + let serialized = serde_json::to_value(&settings).expect("should serialize"); + assert_eq!( + serialized.pointer("/shared/example/endpoint"), + Some(&serde_json::json!("https://api.example.com/v1")) + ); + } + + #[test] + fn settings_without_a_shared_section_are_written_without_one() { + let settings = with("").expect("should load"); + + let serialized = serde_json::to_value(&settings).expect("should serialize"); + assert!(serialized.get("shared").is_none()); + } + } } #[cfg(test)] diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 6a9a260337..3ab5f8530e 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -252,6 +252,7 @@ fail and the service will return its startup-error response. | `[rewrite]` | nothing | First-party URL rewrite exclusions | | `[robots-txt]` | several modules | What `/robots.txt` answers, and `X-Robots-Tag` on its responses when every crawler is refused | | `[[serve]]` | nothing | Which page changes run on each reader's copy of a page, and in what order | +| `[shared]` | nothing | What one vendor's modules share, a table per vendor, which runs that vendor's builder | | `[tester_cookie]` | nothing | Optional tester-cookie endpoints | | `[trusted_client_ip]` | nothing | Authenticated front-door client-IP forwarding | | `[]`, such as `[cmp]`, `[tag]`, `[ad-tag]`, `[bot-protection]`, `[identity]`, `[audience]` or `[framework]` | one module or several | The section of a module type, selecting the modules of that type that run | @@ -1239,6 +1240,20 @@ path = "/ads.txt" middleware = ["ads-txt.sellers"] ``` +## Shared Settings + +### `[shared]` + +What every module of one vendor shares, in one `[shared.]` table per vendor. Core reads none of a table. The vendor's modules read their own through `Settings::shared_config`, and the table's presence is what runs the builder that registers them, so settings without it run none of that vendor's modules and a selection naming one is refused with the table to add. A builder a table runs is selected by no section. A table that no module in the deployment reads is refused at startup, and so is a table whose name is not lower case letters, digits, `_` or `-`. + +A secret in a shared table is written as the name of its entry in the secret store, declared by the vendor's builder as a module's own secret settings are, checked at push to name a key, and resolved when the table is present. + +```toml +[shared.example] +endpoint = "https://api.example.com/v1" +license_key = "example_license_key" +``` + ## Request Signing Configuration for Ed25519 request signing. diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index 60385ac191..82bfd1caea 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -423,6 +423,7 @@ points at the functions that do the work. | `.with_plan_validator(...)` | Optional. Checks the module's settings against the compiled auction plan, when a deployment is validated and as the settings load | | `.with_demand(...)` and `.with_adserver(...)` | Optional. Registers an auction implementation that `[demand]` or `[ad-server]` can name. A crate that supplies nothing else starts from `IntegrationBuilder::implementations(...)` | | `.with_loading_cache(...)` | Optional. Offers the loading cache the crate supplies, which a feature selects by the module's name in its `[cache.]` table. An adapter takes the selections with `loading_caches_from` | +| `.with_shared_settings(...)` | Optional. Runs the builder when its vendor's `[shared.]` table is present, for a vendor whose modules all read that one table. A builder the table runs is selected by no section | ```rust pub fn module_name() -> &'static str { @@ -848,6 +849,20 @@ the list does not order page changes, which run in the order an entry names them. Offering a module does not run it, because the registry builds a module only when a section of the settings selects it. +### A vendor's shared table + +What every module of one vendor needs, such as an endpoint and a license key, +goes in one `[shared.]` table. Core reads none of it. The vendor's +crates read it as their own typed settings with +`Settings::shared_config::(vendor)`, which refuses a table that cannot be +read as `T` or fails its validation, naming the table and the setting at fault +and never the setting's value. The vendor's builder marks itself with +`.with_shared_settings(vendor)`, and the registry builds it when the table is +present and refuses a table that no builder reads. A secret in the table is +declared with `.with_secret_settings(...)` at its path inside the table, as a +module's own secrets are, is checked at push to name a key, and is resolved +from the secret store when the table is present. + ### Supplying a loading cache Some answers cost a call to make and are the same for many requests, such as diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 4f19d98441..627773b48d 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -33,6 +33,10 @@ # complete rules, and what is checked when, are in # docs/guide/configuration-rules.md. # +# [shared.] holds what one vendor's modules share, such as an endpoint +# and a license key. That vendor's crates alone read it, its presence runs the +# builder they share, and a table no module reads is refused. +# # `TRUSTED_SERVER__` env vars can override values when the `ts` CLI builds and # validates this config for push (e.g. TRUSTED_SERVER__PUBLISHER__DOMAIN=...; # nested keys use `__`). The overlay only replaces SCALAR leaves that already