From 9ee13c5355ffa2bdd7b22a942bc15c75d0b9ea00 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 00:20:01 +0100 Subject: [PATCH 01/36] Record the page each module's hooks produce, before the hooks change form A module changes a page through four hook traits today. The integration seam design replaces them with one middleware contract (section 3.7), and asks that each converted implementation reproduces its recorded output. This is the recording, made by the hooks as they are. `html_processor::test_support::assert_page_is_recorded`, behind `test-utils`, runs a page through the document pipeline of a deployment that writes a module's settings and links its builder, and compares what a reader receives with a file, byte for byte. With `TS_RECORD_PAGES` set it writes the file instead. The script bundle's `?v=` hash is replaced by a fixed word, so a recording changes when a module's page change does and not when a browser module is rebuilt. Eleven module crates and core's own script proxy each gain one test and three files under `src/fixtures`, being the settings a deployment writes, the page the origin sends and the page a reader receives. A `.gitattributes` line keeps the two pages' line endings as committed, as the seam probe does for its script. Between them the recordings pass through all twenty hook implementations in this tree, being nine attribute rewriters, three script rewriters, one stream processor and seven head injectors. Two of the head injectors act on a request their module marked, the Google Publisher Tag diagnostics and DataDome's tag, and the recordings are of a request with no mark, so the first writes nothing and the second writes its tag. Two things the recordings show as they are. The Google Tag Manager page holds `'&l='+l` where the origin's snippet holds `'&l='+l`, because a rewritten script is written back escaped. And Prebid's script removal takes out a link whose address matches a script pattern, as it takes out a script. --- crates/ad-tag/google/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 17 + .../src/fixtures/page-change.recorded.html | 947 ++++++++++++++++++ .../src/fixtures/page-change.settings.toml | 5 + crates/ad-tag/google/src/lib.rs | 17 + crates/auction/aps/.gitattributes | 4 + .../aps/src/fixtures/page-change.input.html | 13 + .../src/fixtures/page-change.recorded.html | 13 + .../src/fixtures/page-change.settings.toml | 9 + crates/auction/aps/src/lib.rs | 17 + crates/auction/prebid/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 17 + .../src/fixtures/page-change.recorded.html | 17 + .../src/fixtures/page-change.settings.toml | 11 + crates/auction/prebid/src/lib.rs | 17 + crates/audience/permutive/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 17 + .../src/fixtures/page-change.recorded.html | 17 + .../src/fixtures/page-change.settings.toml | 6 + crates/audience/permutive/src/lib.rs | 17 + crates/bot-protection/datadome/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 16 + .../src/fixtures/page-change.recorded.html | 16 + .../src/fixtures/page-change.settings.toml | 5 + crates/bot-protection/datadome/src/lib.rs | 17 + crates/cmp/didomi/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 13 + .../src/fixtures/page-change.recorded.html | 13 + .../src/fixtures/page-change.settings.toml | 7 + crates/cmp/didomi/src/lib.rs | 17 + crates/cmp/sourcepoint/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 17 + .../src/fixtures/page-change.recorded.html | 17 + .../src/fixtures/page-change.settings.toml | 5 + crates/cmp/sourcepoint/src/lib.rs | 17 + crates/framework/nextjs/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 15 + .../src/fixtures/page-change.recorded.html | 15 + .../src/fixtures/page-change.settings.toml | 5 + crates/framework/nextjs/src/lib.rs | 17 + crates/identity/lockr/.gitattributes | 4 + .../lockr/src/fixtures/page-change.input.html | 17 + .../src/fixtures/page-change.recorded.html | 17 + .../src/fixtures/page-change.settings.toml | 5 + crates/identity/lockr/src/lib.rs | 17 + crates/tag/google-tag-manager/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 21 + .../src/fixtures/page-change.recorded.html | 21 + .../src/fixtures/page-change.settings.toml | 6 + crates/tag/google-tag-manager/src/lib.rs | 17 + crates/testing/testlight/.gitattributes | 4 + .../src/fixtures/page-change.input.html | 16 + .../src/fixtures/page-change.recorded.html | 16 + .../src/fixtures/page-change.settings.toml | 7 + crates/testing/testlight/src/lib.rs | 17 + crates/trusted-server-core/.gitattributes | 4 + .../trusted-server-core/src/html_processor.rs | 201 ++++ .../fixtures/js-asset-proxy.input.html | 17 + .../fixtures/js-asset-proxy.recorded.html | 17 + .../fixtures/js-asset-proxy.settings.toml | 9 + .../src/integrations/js_asset_proxy.rs | 17 + 61 files changed, 1855 insertions(+) create mode 100644 crates/ad-tag/google/.gitattributes create mode 100644 crates/ad-tag/google/src/fixtures/page-change.input.html create mode 100644 crates/ad-tag/google/src/fixtures/page-change.recorded.html create mode 100644 crates/ad-tag/google/src/fixtures/page-change.settings.toml create mode 100644 crates/auction/aps/.gitattributes create mode 100644 crates/auction/aps/src/fixtures/page-change.input.html create mode 100644 crates/auction/aps/src/fixtures/page-change.recorded.html create mode 100644 crates/auction/aps/src/fixtures/page-change.settings.toml create mode 100644 crates/auction/prebid/.gitattributes create mode 100644 crates/auction/prebid/src/fixtures/page-change.input.html create mode 100644 crates/auction/prebid/src/fixtures/page-change.recorded.html create mode 100644 crates/auction/prebid/src/fixtures/page-change.settings.toml create mode 100644 crates/audience/permutive/.gitattributes create mode 100644 crates/audience/permutive/src/fixtures/page-change.input.html create mode 100644 crates/audience/permutive/src/fixtures/page-change.recorded.html create mode 100644 crates/audience/permutive/src/fixtures/page-change.settings.toml create mode 100644 crates/bot-protection/datadome/.gitattributes create mode 100644 crates/bot-protection/datadome/src/fixtures/page-change.input.html create mode 100644 crates/bot-protection/datadome/src/fixtures/page-change.recorded.html create mode 100644 crates/bot-protection/datadome/src/fixtures/page-change.settings.toml create mode 100644 crates/cmp/didomi/.gitattributes create mode 100644 crates/cmp/didomi/src/fixtures/page-change.input.html create mode 100644 crates/cmp/didomi/src/fixtures/page-change.recorded.html create mode 100644 crates/cmp/didomi/src/fixtures/page-change.settings.toml create mode 100644 crates/cmp/sourcepoint/.gitattributes create mode 100644 crates/cmp/sourcepoint/src/fixtures/page-change.input.html create mode 100644 crates/cmp/sourcepoint/src/fixtures/page-change.recorded.html create mode 100644 crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml create mode 100644 crates/framework/nextjs/.gitattributes create mode 100644 crates/framework/nextjs/src/fixtures/page-change.input.html create mode 100644 crates/framework/nextjs/src/fixtures/page-change.recorded.html create mode 100644 crates/framework/nextjs/src/fixtures/page-change.settings.toml create mode 100644 crates/identity/lockr/.gitattributes create mode 100644 crates/identity/lockr/src/fixtures/page-change.input.html create mode 100644 crates/identity/lockr/src/fixtures/page-change.recorded.html create mode 100644 crates/identity/lockr/src/fixtures/page-change.settings.toml create mode 100644 crates/tag/google-tag-manager/.gitattributes create mode 100644 crates/tag/google-tag-manager/src/fixtures/page-change.input.html create mode 100644 crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html create mode 100644 crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml create mode 100644 crates/testing/testlight/.gitattributes create mode 100644 crates/testing/testlight/src/fixtures/page-change.input.html create mode 100644 crates/testing/testlight/src/fixtures/page-change.recorded.html create mode 100644 crates/testing/testlight/src/fixtures/page-change.settings.toml create mode 100644 crates/trusted-server-core/.gitattributes create mode 100644 crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.input.html create mode 100644 crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.recorded.html create mode 100644 crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml diff --git a/crates/ad-tag/google/.gitattributes b/crates/ad-tag/google/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/ad-tag/google/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/ad-tag/google/src/fixtures/page-change.input.html b/crates/ad-tag/google/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..6a9f667691 --- /dev/null +++ b/crates/ad-tag/google/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Google Publisher Tag + + + + + + + +An article on the origin + +
+ + diff --git a/crates/ad-tag/google/src/fixtures/page-change.recorded.html b/crates/ad-tag/google/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..3d3a347ffd --- /dev/null +++ b/crates/ad-tag/google/src/fixtures/page-change.recorded.html @@ -0,0 +1,947 @@ + + + + +Google Publisher Tag + + + + + + + +An article on the origin + +
+ + diff --git a/crates/ad-tag/google/src/fixtures/page-change.settings.toml b/crates/ad-tag/google/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..4d358367e3 --- /dev/null +++ b/crates/ad-tag/google/src/fixtures/page-change.settings.toml @@ -0,0 +1,5 @@ +[ad-tag] +modules = ["google", "google.diagnostics"] + +[ad-tag.google] +gam_attribution_enabled = false diff --git a/crates/ad-tag/google/src/lib.rs b/crates/ad-tag/google/src/lib.rs index 403f40cf1c..6ce0256a15 100644 --- a/crates/ad-tag/google/src/lib.rs +++ b/crates/ad-tag/google/src/lib.rs @@ -1609,4 +1609,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder(), super::diagnostics::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/auction/aps/.gitattributes b/crates/auction/aps/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/auction/aps/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/auction/aps/src/fixtures/page-change.input.html b/crates/auction/aps/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..6a58b86ede --- /dev/null +++ b/crates/auction/aps/src/fixtures/page-change.input.html @@ -0,0 +1,13 @@ + + + + +APS + + + +An article on the origin + +
+ + diff --git a/crates/auction/aps/src/fixtures/page-change.recorded.html b/crates/auction/aps/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..c33ea015d7 --- /dev/null +++ b/crates/auction/aps/src/fixtures/page-change.recorded.html @@ -0,0 +1,13 @@ + + + + +APS + + + +An article on the origin + +
+ + diff --git a/crates/auction/aps/src/fixtures/page-change.settings.toml b/crates/auction/aps/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..0bc023ecf3 --- /dev/null +++ b/crates/auction/aps/src/fixtures/page-change.settings.toml @@ -0,0 +1,9 @@ +[demand] +modules = ["aps_main"] + +[demand.aps_main] +implementation = "auction.aps" +endpoint = "https://aps.example.com/e/pb/bid" +routing = "all_eligible" +account_id = "example-aps-account-id" +rendering_mode = "publisher_native" diff --git a/crates/auction/aps/src/lib.rs b/crates/auction/aps/src/lib.rs index 7d8692d2ff..139c39c99b 100644 --- a/crates/auction/aps/src/lib.rs +++ b/crates/auction/aps/src/lib.rs @@ -4639,4 +4639,21 @@ mod engine_tests { .expect("should serialize routing metadata"); assert!(!routing.contains("example-bidder") && !routing.contains("private")); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/auction/prebid/.gitattributes b/crates/auction/prebid/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/auction/prebid/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/auction/prebid/src/fixtures/page-change.input.html b/crates/auction/prebid/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..c7454e5cfc --- /dev/null +++ b/crates/auction/prebid/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Prebid + + + + + + + +An article on the origin + +A link whose path matches + + diff --git a/crates/auction/prebid/src/fixtures/page-change.recorded.html b/crates/auction/prebid/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..b8d0b8e4eb --- /dev/null +++ b/crates/auction/prebid/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Prebid + + + + + + + +An article on the origin + + + + diff --git a/crates/auction/prebid/src/fixtures/page-change.settings.toml b/crates/auction/prebid/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..ef33b648bb --- /dev/null +++ b/crates/auction/prebid/src/fixtures/page-change.settings.toml @@ -0,0 +1,11 @@ +[proxy] +allowed_domains = ["assets.example"] + +[auction] +modules = ["prebid"] + +[auction.prebid] +external_bundle_url = "https://assets.example/prebid/trusted-prebid.js" +timeout_ms = 1000 +script_patterns = ["/prebid.js", "/prebid.min.js"] +debug = false diff --git a/crates/auction/prebid/src/lib.rs b/crates/auction/prebid/src/lib.rs index 4a4e161499..27a6d72fd1 100644 --- a/crates/auction/prebid/src/lib.rs +++ b/crates/auction/prebid/src/lib.rs @@ -3473,4 +3473,21 @@ server_url = "https://prebid.example" "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/audience/permutive/.gitattributes b/crates/audience/permutive/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/audience/permutive/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/audience/permutive/src/fixtures/page-change.input.html b/crates/audience/permutive/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..cd143b0737 --- /dev/null +++ b/crates/audience/permutive/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Permutive + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/audience/permutive/src/fixtures/page-change.recorded.html b/crates/audience/permutive/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..e2d9fdfbc7 --- /dev/null +++ b/crates/audience/permutive/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Permutive + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/audience/permutive/src/fixtures/page-change.settings.toml b/crates/audience/permutive/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..067a874e21 --- /dev/null +++ b/crates/audience/permutive/src/fixtures/page-change.settings.toml @@ -0,0 +1,6 @@ +[audience] +modules = ["permutive"] + +[audience.permutive] +organization_id = "myorg" +workspace_id = "abc123" diff --git a/crates/audience/permutive/src/lib.rs b/crates/audience/permutive/src/lib.rs index 16679f1fe0..9458149c24 100644 --- a/crates/audience/permutive/src/lib.rs +++ b/crates/audience/permutive/src/lib.rs @@ -763,4 +763,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/bot-protection/datadome/.gitattributes b/crates/bot-protection/datadome/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/bot-protection/datadome/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/bot-protection/datadome/src/fixtures/page-change.input.html b/crates/bot-protection/datadome/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..add3e2e96b --- /dev/null +++ b/crates/bot-protection/datadome/src/fixtures/page-change.input.html @@ -0,0 +1,16 @@ + + + + +DataDome + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/bot-protection/datadome/src/fixtures/page-change.recorded.html b/crates/bot-protection/datadome/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..3c0e736fae --- /dev/null +++ b/crates/bot-protection/datadome/src/fixtures/page-change.recorded.html @@ -0,0 +1,16 @@ + + + + +DataDome + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/bot-protection/datadome/src/fixtures/page-change.settings.toml b/crates/bot-protection/datadome/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..4d185c624e --- /dev/null +++ b/crates/bot-protection/datadome/src/fixtures/page-change.settings.toml @@ -0,0 +1,5 @@ +[bot-protection] +modules = ["datadome"] + +[bot-protection.datadome] +client_side_key = "test-client-key" diff --git a/crates/bot-protection/datadome/src/lib.rs b/crates/bot-protection/datadome/src/lib.rs index 88ed844994..801ee7d219 100644 --- a/crates/bot-protection/datadome/src/lib.rs +++ b/crates/bot-protection/datadome/src/lib.rs @@ -2042,4 +2042,21 @@ mod tests { "should identify the removed field: {error:?}" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/cmp/didomi/.gitattributes b/crates/cmp/didomi/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/cmp/didomi/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/cmp/didomi/src/fixtures/page-change.input.html b/crates/cmp/didomi/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..e6a40bef11 --- /dev/null +++ b/crates/cmp/didomi/src/fixtures/page-change.input.html @@ -0,0 +1,13 @@ + + + + +Didomi + + + + +An article on the origin + + + diff --git a/crates/cmp/didomi/src/fixtures/page-change.recorded.html b/crates/cmp/didomi/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..c7e21c1ab9 --- /dev/null +++ b/crates/cmp/didomi/src/fixtures/page-change.recorded.html @@ -0,0 +1,13 @@ + + + + +Didomi + + + + +An article on the origin + + + diff --git a/crates/cmp/didomi/src/fixtures/page-change.settings.toml b/crates/cmp/didomi/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..378eb1e26d --- /dev/null +++ b/crates/cmp/didomi/src/fixtures/page-change.settings.toml @@ -0,0 +1,7 @@ +[cmp] +modules = ["didomi"] + +[cmp.didomi] +geo_query_parameters = false +sdk_origin = "https://sdk.example.com" +api_origin = "https://api.example.com" diff --git a/crates/cmp/didomi/src/lib.rs b/crates/cmp/didomi/src/lib.rs index 8b753a39c7..a5ba6aba15 100644 --- a/crates/cmp/didomi/src/lib.rs +++ b/crates/cmp/didomi/src/lib.rs @@ -1460,4 +1460,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/cmp/sourcepoint/.gitattributes b/crates/cmp/sourcepoint/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/cmp/sourcepoint/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/cmp/sourcepoint/src/fixtures/page-change.input.html b/crates/cmp/sourcepoint/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..66137ff8cb --- /dev/null +++ b/crates/cmp/sourcepoint/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Sourcepoint + + + + + + + +An article on the origin + +A link to another host + + diff --git a/crates/cmp/sourcepoint/src/fixtures/page-change.recorded.html b/crates/cmp/sourcepoint/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..925a486535 --- /dev/null +++ b/crates/cmp/sourcepoint/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Sourcepoint + + + + + + + +An article on the origin + +A link to another host + + diff --git a/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml b/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..ddb1a43041 --- /dev/null +++ b/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml @@ -0,0 +1,5 @@ +[cmp] +modules = ["sourcepoint"] + +[cmp.sourcepoint] +rewrite_sdk = true diff --git a/crates/cmp/sourcepoint/src/lib.rs b/crates/cmp/sourcepoint/src/lib.rs index b9cd3e90a8..00702d55da 100644 --- a/crates/cmp/sourcepoint/src/lib.rs +++ b/crates/cmp/sourcepoint/src/lib.rs @@ -2213,4 +2213,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/framework/nextjs/.gitattributes b/crates/framework/nextjs/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/framework/nextjs/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/framework/nextjs/src/fixtures/page-change.input.html b/crates/framework/nextjs/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..66a36e3ca4 --- /dev/null +++ b/crates/framework/nextjs/src/fixtures/page-change.input.html @@ -0,0 +1,15 @@ + + + + +Next.js + + +An article on the origin + + + + + + + diff --git a/crates/framework/nextjs/src/fixtures/page-change.recorded.html b/crates/framework/nextjs/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..11f71f6cc0 --- /dev/null +++ b/crates/framework/nextjs/src/fixtures/page-change.recorded.html @@ -0,0 +1,15 @@ + + + + +Next.js + + +An article on the origin + + + + + + + diff --git a/crates/framework/nextjs/src/fixtures/page-change.settings.toml b/crates/framework/nextjs/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..0811497114 --- /dev/null +++ b/crates/framework/nextjs/src/fixtures/page-change.settings.toml @@ -0,0 +1,5 @@ +[framework] +modules = ["nextjs"] + +[framework.nextjs] +rewrite_attributes = ["href", "link", "url"] diff --git a/crates/framework/nextjs/src/lib.rs b/crates/framework/nextjs/src/lib.rs index 4c06663909..4742701853 100644 --- a/crates/framework/nextjs/src/lib.rs +++ b/crates/framework/nextjs/src/lib.rs @@ -945,4 +945,21 @@ mod tests { ); } } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/identity/lockr/.gitattributes b/crates/identity/lockr/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/identity/lockr/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/identity/lockr/src/fixtures/page-change.input.html b/crates/identity/lockr/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..4e816c68d6 --- /dev/null +++ b/crates/identity/lockr/src/fixtures/page-change.input.html @@ -0,0 +1,17 @@ + + + + +Lockr + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/identity/lockr/src/fixtures/page-change.recorded.html b/crates/identity/lockr/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..199fbc145e --- /dev/null +++ b/crates/identity/lockr/src/fixtures/page-change.recorded.html @@ -0,0 +1,17 @@ + + + + +Lockr + + + + + + + +An article on the origin + +A link that is not the SDK + + diff --git a/crates/identity/lockr/src/fixtures/page-change.settings.toml b/crates/identity/lockr/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..33c2319acc --- /dev/null +++ b/crates/identity/lockr/src/fixtures/page-change.settings.toml @@ -0,0 +1,5 @@ +[identity] +modules = ["lockr"] + +[identity.lockr] +app_id = "test-app-id" diff --git a/crates/identity/lockr/src/lib.rs b/crates/identity/lockr/src/lib.rs index 5b79563d94..f7c9aefe62 100644 --- a/crates/identity/lockr/src/lib.rs +++ b/crates/identity/lockr/src/lib.rs @@ -761,4 +761,21 @@ mod tests { "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/tag/google-tag-manager/.gitattributes b/crates/tag/google-tag-manager/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/tag/google-tag-manager/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.input.html b/crates/tag/google-tag-manager/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..134e4914c3 --- /dev/null +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.input.html @@ -0,0 +1,21 @@ + + + + +Google Tag Manager + + + + + + + +An article on the origin + + + + diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html b/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..dc8fd8d3bd --- /dev/null +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html @@ -0,0 +1,21 @@ + + + + +Google Tag Manager + + + + + + + +An article on the origin + + + + diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml b/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..abcdc009bc --- /dev/null +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml @@ -0,0 +1,6 @@ +[tag] +modules = ["google-tag-manager"] + +[tag.google-tag-manager] +container_id = "GTM-TEST1234" +upstream_url = "https://www.googletagmanager.com" diff --git a/crates/tag/google-tag-manager/src/lib.rs b/crates/tag/google-tag-manager/src/lib.rs index 35c8069f30..301bc92da1 100644 --- a/crates/tag/google-tag-manager/src/lib.rs +++ b/crates/tag/google-tag-manager/src/lib.rs @@ -3949,4 +3949,21 @@ assume_single_jurisdiction = true "should be named by the folder this crate lives in" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[super::builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/testing/testlight/.gitattributes b/crates/testing/testlight/.gitattributes new file mode 100644 index 0000000000..fad78ffdb5 --- /dev/null +++ b/crates/testing/testlight/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/fixtures/page-change.*.html text eol=lf diff --git a/crates/testing/testlight/src/fixtures/page-change.input.html b/crates/testing/testlight/src/fixtures/page-change.input.html new file mode 100644 index 0000000000..74055212dc --- /dev/null +++ b/crates/testing/testlight/src/fixtures/page-change.input.html @@ -0,0 +1,16 @@ + + + + +Testlight + + + + + + +An article on the origin + +

testlight.js is named here in text, which is left as it is.

+ + diff --git a/crates/testing/testlight/src/fixtures/page-change.recorded.html b/crates/testing/testlight/src/fixtures/page-change.recorded.html new file mode 100644 index 0000000000..386f2effe4 --- /dev/null +++ b/crates/testing/testlight/src/fixtures/page-change.recorded.html @@ -0,0 +1,16 @@ + + + + +Testlight + + + + + + +An article on the origin + +

testlight.js is named here in text, which is left as it is.

+ + diff --git a/crates/testing/testlight/src/fixtures/page-change.settings.toml b/crates/testing/testlight/src/fixtures/page-change.settings.toml new file mode 100644 index 0000000000..a8f539e0d0 --- /dev/null +++ b/crates/testing/testlight/src/fixtures/page-change.settings.toml @@ -0,0 +1,7 @@ +[auction] +modules = ["testing.testlight"] + +[auction.testing.testlight] +endpoint = "https://example.com/openrtb2/auction" +rewrite_scripts = true +shim_src = "https://edge.example.com/static/testlight.js" diff --git a/crates/testing/testlight/src/lib.rs b/crates/testing/testlight/src/lib.rs index 118c72e641..ff1112e33b 100644 --- a/crates/testing/testlight/src/lib.rs +++ b/crates/testing/testlight/src/lib.rs @@ -672,4 +672,21 @@ mod tests { "the vendor's own URL should be gone from the page" ); } + + /// The page a reader receives with this module running, kept as a file + /// so that changing how the page change is made can be shown to leave + /// the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + trusted_server_core::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/page-change.settings.toml"), + &[builder()], + include_str!("fixtures/page-change.input.html"), + include_str!("fixtures/page-change.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/fixtures/page-change.recorded.html" + ), + ); + } } diff --git a/crates/trusted-server-core/.gitattributes b/crates/trusted-server-core/.gitattributes new file mode 100644 index 0000000000..6e7a419fbc --- /dev/null +++ b/crates/trusted-server-core/.gitattributes @@ -0,0 +1,4 @@ +# The recording is compared byte for byte with the page made from the input +# beside it, so both must arrive with the line endings they were committed +# with on every platform. +src/integrations/fixtures/*.html text eol=lf diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index c77b678b10..3035ab5035 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -725,6 +725,207 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso } } +/// Running a page through the document pipeline and comparing what comes out +/// with a recording, for a module crate's own tests. +/// +/// A recording is the page a reader receives, kept as a file beside the test. +/// It is how a change to the way a module makes its page change is shown to +/// leave the page as it was. +#[cfg(any(test, feature = "test-utils"))] +pub mod test_support { + use std::io::Cursor; + + use super::{HtmlProcessorConfig, create_html_processor}; + use crate::integrations::IntegrationRegistry; + use crate::settings::Settings; + use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; + + /// The host a test page is fetched from. + pub const ORIGIN_HOST: &str = "origin.example.com"; + + /// The host a test page is served on, over HTTPS. + pub const REQUEST_HOST: &str = "publisher.example.com"; + + /// Set in the environment to write recordings in place of comparing with + /// them. + pub const RECORD_ENV: &str = "TS_RECORD_PAGES"; + + /// What stands in a recording for the hash of the script bundle, which + /// changes whenever any browser module does. + const BUNDLE_HASH: &str = "BUNDLE-HASH"; + + /// The page a reader of `html` receives from a deployment running + /// `settings` with `registry`'s modules, read from the origin `chunk_size` + /// bytes at a time. + /// + /// The script bundle's `?v=` hash is replaced by a fixed word, so a + /// recording changes when a module's page change does and not when a + /// browser module is rebuilt. + /// + /// # Panics + /// + /// When the pipeline refuses the page, or leaves something that is not + /// UTF-8. + #[must_use] + pub fn processed_page( + settings: &Settings, + registry: &IntegrationRegistry, + html: &str, + chunk_size: usize, + ) -> String { + let config = HtmlProcessorConfig::from_settings( + settings, + registry, + ORIGIN_HOST, + REQUEST_HOST, + "https", + ); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + create_html_processor(config), + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should process the page"); + without_bundle_hashes(&String::from_utf8(output).expect("should leave the page UTF-8")) + } + + /// `page` with each `?v=` hash of a script bundle address replaced. + fn without_bundle_hashes(page: &str) -> String { + const MARK: &str = "/static/tsjs="; + let mut out = String::with_capacity(page.len()); + let mut rest = page; + while let Some(at) = rest.find(MARK) { + let (before, address) = rest.split_at(at + MARK.len()); + out.push_str(before); + let end = address + .find(['"', '\'', ' ', '>', '<']) + .unwrap_or(address.len()); + let (address, after) = address.split_at(end); + match address.split_once("?v=") { + Some((file, hash)) + if !hash.is_empty() && hash.bytes().all(|b| b.is_ascii_hexdigit()) => + { + out.push_str(file); + out.push_str("?v="); + out.push_str(BUNDLE_HASH); + } + _ => out.push_str(address), + } + rest = after; + } + out.push_str(rest); + out + } + + /// The least a deployment's settings must say, which a module's own + /// settings are written after. + const BASE_SETTINGS: &str = r#" +[[handlers]] +path = "^/_ts/admin" +username = "admin" +password = "page-recording-password" + +[publisher] +domain = "publisher.example.com" +cookie_domain = ".publisher.example.com" +origin_url = "https://origin.example.com" +proxy_secret = "page-recording-proxy-secret" + +[geo] +assume_single_jurisdiction = true + +[ec] +module = "hmac" + +[ec.hmac] +passphrase = "page-recording-passphrase-32-bytes" +"#; + + /// The settings of a deployment that writes `module_settings`, a TOML + /// document selecting a module and holding its table, after the least a + /// deployment must say. + /// + /// # Panics + /// + /// When the two together are not settings a deployment could load. + #[must_use] + pub fn page_settings(module_settings: &str) -> Settings { + Settings::from_toml(&format!("{BASE_SETTINGS}\n{module_settings}")) + .expect("should read the page's settings") + } + + /// The size of the chunks a recorded page is read from the origin in, + /// large enough that a test page arrives whole. + const RECORDING_CHUNK_SIZE: usize = 8192; + + /// Checks the page `html` becomes is the one in `recorded`, byte for + /// byte. + /// + /// The deployment writes `module_settings`, see [`page_settings`], and + /// links `builders` beside core's own modules. `recorded` is the recording + /// as the test embeds it and `path` is where it is kept. With + /// [`RECORD_ENV`] set the recording is written to `path` instead, which is + /// how a recording is made. A recording is made again only when the page + /// is meant to change for every reader. + /// + /// # Panics + /// + /// When the page is not the recorded one, or the recording cannot be + /// written. + pub fn assert_page_is_recorded( + module_settings: &str, + builders: &[crate::integrations::IntegrationBuilder], + html: &str, + recorded: &str, + path: &str, + ) { + let settings = &page_settings(module_settings); + let registry = &IntegrationRegistry::with_registrations(settings, builders) + .expect("should build the registry"); + let page = processed_page(settings, registry, html, RECORDING_CHUNK_SIZE); + #[cfg(not(target_arch = "wasm32"))] + if std::env::var_os(RECORD_ENV).is_some() { + std::fs::write(path, &page).expect("should write the recording"); + return; + } + assert!( + recorded == page, + "the page is not the one recorded at {path}.\n--- recorded\n{recorded}\n--- now\n{page}\n\ + ---\nA recording changes only when the page is meant to change for every reader, \ + by running the test with {RECORD_ENV} set." + ); + } + + #[cfg(test)] + mod tests { + use super::without_bundle_hashes; + + #[test] + fn a_bundle_hash_is_replaced_and_nothing_else_is() { + let page = "\ + \ + kept\ + kept"; + + let replaced = without_bundle_hashes(page); + + assert_eq!( + replaced, + "\ + \ + kept\ + kept" + ); + } + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.input.html b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.input.html new file mode 100644 index 0000000000..14c7aa4842 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.input.html @@ -0,0 +1,17 @@ + + + + +JS asset proxy + + + + + + + +An article on the origin + +A link to the same address + + diff --git a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.recorded.html b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.recorded.html new file mode 100644 index 0000000000..4168885df0 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.recorded.html @@ -0,0 +1,17 @@ + + + + +JS asset proxy + + + + + + + +An article on the origin + +A link to the same address + + diff --git a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml new file mode 100644 index 0000000000..c582079376 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml @@ -0,0 +1,9 @@ +[proxy] +modules = ["js_asset_proxy"] + +[proxy.js_asset_proxy] + +[[proxy.js_asset_proxy.assets]] +path = "/assets/example-vendor-loader.js" +origin_url = "https://cdn.example.com/vendor-loader.js" +proxy = "enabled" diff --git a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs index 6677417024..65eee4e0ed 100644 --- a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs +++ b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs @@ -1696,4 +1696,21 @@ mod tests { "should identify the unknown asset field" ); } + + /// The page a reader receives with the script proxy running, kept as a + /// file so that changing how the page change is made can be shown to + /// leave the page as it was. + #[test] + fn the_page_a_reader_receives_is_the_recorded_one() { + crate::html_processor::test_support::assert_page_is_recorded( + include_str!("fixtures/js-asset-proxy.settings.toml"), + &[], + include_str!("fixtures/js-asset-proxy.input.html"), + include_str!("fixtures/js-asset-proxy.recorded.html"), + concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/integrations/fixtures/js-asset-proxy.recorded.html" + ), + ); + } } From 68cb6c372c5017c9f27bc8a879f699ac5e99fd2c Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 01:00:42 +0100 Subject: [PATCH 02/36] Write a rewritten script back as the rewriter gave it When a script rewriter replaces a script's text, the HTML processor wrote the replacement with `ContentType::Text`, which escapes `&`, `<` and `>`. A browser decodes no entity inside a `"#; + + let mut settings = create_test_settings(); + settings + .insert_module_config( + "framework", + "framework.nextjs", + &json!({ + "rewrite_attributes": ["href", "link", "url"], + }), + ) + .expect("should update nextjs config"); + let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) + .expect("should create registry"); + let config = config_from_settings(&settings, ®istry); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size: 8192, + }, + create_html_processor(config), + ); + + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("pipeline should process HTML"); + let processed = String::from_utf8_lossy(&output); + + assert!( + processed.contains(r#""href":"https://test.example.com/search?q=shoes&page=2""#), + "should move the address and keep its query string as written: {processed}" + ); + assert!( + processed.contains(r#""note":"a < b && c > d""#), + "should keep a value it did not rewrite as written: {processed}" + ); + } + #[test] fn html_processor_rewrites_next_data_fixture_like_payload() { let html = r#" diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html b/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html index dc8fd8d3bd..cbb6bebe36 100644 --- a/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.recorded.html @@ -6,7 +6,7 @@ diff --git a/crates/tag/google-tag-manager/src/lib.rs b/crates/tag/google-tag-manager/src/lib.rs index 301bc92da1..d176129440 100644 --- a/crates/tag/google-tag-manager/src/lib.rs +++ b/crates/tag/google-tag-manager/src/lib.rs @@ -3338,6 +3338,84 @@ assume_single_jurisdiction = true ); } + /// The page a reader receives when `html` arrives from the origin + /// `chunk_size` bytes at a time, with Google Tag Manager running. + fn page_with_gtm(html: &str, chunk_size: usize) -> String { + let mut settings = make_settings(); + settings + .insert_module_config( + "tag", + MODULE, + &serde_json::json!({ + "container_id": "GTM-12345", + "upstream_url": "https://www.googletagmanager.com" + }), + ) + .expect("should update config"); + let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) + .expect("should create registry"); + let config = config_from_settings(&settings, ®istry); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + create_html_processor(config), + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should process"); + String::from_utf8(output).expect("should stay UTF-8") + } + + #[test] + fn the_rewritten_snippet_keeps_its_data_layer_parameter() { + // The standard snippet adds `'&l='+l` to the container's address when + // a page names its own data layer. A browser decodes no entity inside + // a script, so `&l=` would send a parameter called `amp;l`, and + // the container would read the default data layer instead. + let processed = page_with_gtm( + r#""#, + 8192, + ); + + assert!( + processed.contains("'/integrations/google_tag_manager/gtm.js?id='+i+dl"), + "should point the snippet at the first-party path: {processed}" + ); + assert!( + processed.contains("dl=l!='dataLayer'?'&l='+l:''"), + "should keep the data layer parameter as the snippet wrote it: {processed}" + ); + assert!( + !processed.contains("&"), + "should escape nothing inside the script: {processed}" + ); + } + + #[test] + fn a_loader_of_the_page_s_own_keeps_its_operators() { + let processed = page_with_gtm( + r#""#, + 8192, + ); + + assert!( + processed.contains( + "" + ), + "should change the address and leave the script's operators as they were: \ + {processed}" + ); + } + #[test] fn test_container_id_validation_accepts_valid_ids() { // Valid container IDs with different lengths diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 3035ab5035..9f0bc886aa 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -690,7 +690,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso match rewriter.rewrite(text.as_str(), &ctx) { ScriptRewriteAction::Keep => {} ScriptRewriteAction::Replace(rewritten) => { - text.replace(&rewritten, ContentType::Text); + // Written back as markup, because the rewriter was + // handed the text as the page wrote it. Writing it as + // text would escape `&`, `<` and `>`, and a browser + // decodes none of those inside a script. + text.replace(&rewritten, ContentType::Html); } ScriptRewriteAction::RemoveNode => { text.remove(); @@ -2098,6 +2102,54 @@ mod tests { ); } + #[test] + fn a_rewritten_script_is_written_back_as_the_rewriter_gave_it() { + // A script's text is not HTML text. A browser decodes no entity inside + // a script, so `&&` written back as `&&` is a syntax error. + struct HostRewriter; + + impl crate::integrations::IntegrationScriptRewriter for HostRewriter { + fn integration_id(&self) -> &'static str { + "host-rewriter" + } + + fn selector(&self) -> &'static str { + "script" + } + + fn rewrite( + &self, + content: &str, + _ctx: &IntegrationScriptContext<'_>, + ) -> ScriptRewriteAction { + ScriptRewriteAction::replace( + content.replace("cdn.vendor.example", "test.example.com"), + ) + } + } + + let mut config = create_test_config(); + config.integrations = + IntegrationRegistry::from_rewriters(Vec::new(), vec![Arc::new(HostRewriter)]); + let mut processor = create_html_processor(config); + + let output = processor + .process_chunk( + br#""#, + true, + ) + .expect("should process the document"); + let processed = String::from_utf8(output).expect("should stay UTF-8"); + + assert!( + processed.contains( + r#""# + ), + "should write the rewritten script back with its operators and its query string \ + as they were: {processed}" + ); + } + #[test] fn bodyless_marker_mode_emits_an_owned_terminal_seam_even_after_source_bytes() { const MARKER: &str = ""; From 950003760887b2718a9011151d1dde5c578f71b0 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 01:11:11 +0100 Subject: [PATCH 03/36] Write script text held for a marker in the order it came The Google Tag Manager module holds a script's text back while a marker might still arrive in a later part, and writes it with the script's last part. A part that arrived after the held text had stopped looking like a marker was kept as it came, so it went out at once, ahead of the text held before it, and the script reached the page out of order. `main` does the same. The parser hands a script's text over in parts split where a `<` might close the script, so this needs no help from the network. Before this change, footer.innerHTML = 'Search with google or another engine'; came out as footer.innerHTML = '<> or another engine';b>Search with google"); + let first_chunk_ends = html.find("google").expect("should find the word") + "google".len(); + + let processed = page_with_gtm(&html, first_chunk_ends); + + assert!( + processed.contains(&format!("")), + "should write a script it held and did not change back as the page wrote it: \ + {processed}" + ); + } + + #[test] + fn a_script_held_and_then_let_go_keeps_its_order() { + // The first chunk ends on `google`, so the script's text is held. Two + // chunks later nothing held can become a marker any more, and what was + // held has to go out ahead of the text that follows it. + let script = r#"var engine = "google"; var one = 1; var two = 2; var three = 3; var four = 4; var five = 5; var six = 6;"#; + let html = format!(""); + let first_chunk_ends = html.find("google").expect("should find the word") + "google".len(); + + let processed = page_with_gtm(&html, first_chunk_ends); + + assert!( + processed.contains(&format!("")), + "should write the script back in the order the page wrote it: {processed}" + ); + } + + #[test] + fn a_script_that_writes_markup_keeps_its_order() { + // The page arrives in one piece. The parser still hands a script's + // text over in parts, split where a `<` might close the script, so the + // part ending on `google` is held without any help from the network. + let script = + r#"footer.innerHTML = 'Search with google or another engine';"#; + let html = format!(""); + + let processed = page_with_gtm(&html, 8192); + + assert!( + processed.contains(&format!("")), + "should write the script back in the order the page wrote it: {processed}" + ); + } + #[test] fn test_container_id_validation_accepts_valid_ids() { // Valid container IDs with different lengths From 0b21fe3577024be37354acf1abdd385ffc341113 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 02:03:37 +0100 Subject: [PATCH 04/36] A page change can be a middleware that an entry places A module changes a page through four hook traits, and every hook of a selected module runs on every page. This adds the other way to make a page change, which the modules convert to in the commits that follow. A middleware is a page change a module supplies and the operator's settings place. It runs only on the pages a `[[fetch]]` entry names it for, in the order that entry gives. [[fetch]] media_type = "text/html" path = "/news/" middleware = ["tag.example.cleanup", "tag.example"] [[fetch]] media_type = "text/html" middleware = ["tag.example"] A page takes the first entry that covers it. An entry places and orders, and switches nothing on. The section that selects a module still decides whether it runs, and a middleware reads its settings from its module's own table. `trusted_server_core::middleware` holds the contract. A middleware is asked once for each document, and answers with any of head markup, markup for straight after the script bundle, element handlers, text handlers and a stream processor. What core does with it. 1. The registry takes `.with_middleware(...)` from a registration. At startup it refuses two registrations that supply one name, a name no entry could write, and an entry naming a middleware that no running module supplies, listing the names that could be written. A middleware no entry names is logged as a warning, because a module may be selected for what else it does. 2. The settings read `[[fetch]]` and refuse an entry that could not do what it says, when they load and at deploy validation. That is a media type other than `text/html`, a path that is not a request path prefix, an entry that names nothing or names one middleware twice, a key an entry does not read, and an entry an earlier one already covers. 3. The publisher path carries the request path to where a document's processor is made, and the chain for that path is planned there. 4. The HTML processor carries the plan out after the hooks of the same kind. Every text handler, a hook's or a middleware's, is now handed a script as the one before it left it. Before, a second rewriter that matched the same script was handed the origin's text, and its replacement overwrote the first's. A fetch middleware is told nothing about the reader, because what it leaves is what a shared template stores. The entries are part of the settings a template's fingerprint covers, so changing them selects a template of its own. A middleware chooses its selectors for each document, so one that does not parse cannot be refused at startup. The response fails with a 500 that names the middleware, through both finalizers, where the parser would otherwise panic. The seam probe supplies a middleware from outside core, with one of each handler, and the Axum and Fastly adapters' probe tests cover an entry at startup. Tests cover the entries, the registry's checks, what the document pipeline does with each part of a plan, a whole request through both finalizers at a covered path and at another, and a stored shared template. Eleven of them were watched failing with the code broken in six places, being the chain dropped from the publisher path, the chain planned for the root path whatever was asked for, text handlers handed the origin's text, the registry's check removed, and each of the two settings checks removed. The guides describe `[[fetch]]` and the contract, and the documentation test compiles the guide's middleware fixture as it compiles the existing one. No stock module supplies a middleware yet, so no page changes. --- crates/testing/seam-probe/Cargo.toml | 1 + crates/testing/seam-probe/src/lib.rs | 182 ++- .../tests/seam_probe.rs | 68 + .../src/app/seam_probe_tests.rs | 30 + crates/trusted-server-core/src/config.rs | 1 + .../trusted-server-core/src/html_processor.rs | 197 ++- .../src/integrations/mod.rs | 4 + .../src/integrations/registry.rs | 649 ++++++++++ crates/trusted-server-core/src/lib.rs | 1 + crates/trusted-server-core/src/middleware.rs | 584 +++++++++ .../src/middleware/tests.rs | 1103 +++++++++++++++++ crates/trusted-server-core/src/publisher.rs | 396 +++++- crates/trusted-server-core/src/settings.rs | 31 + .../README.md | 2 +- .../tests/documentation_snippets.rs | 29 +- docs/guide/configuration.md | 56 + docs/guide/integration-guide.md | 112 ++ 17 files changed, 3408 insertions(+), 38 deletions(-) create mode 100644 crates/trusted-server-core/src/middleware.rs create mode 100644 crates/trusted-server-core/src/middleware/tests.rs diff --git a/crates/testing/seam-probe/Cargo.toml b/crates/testing/seam-probe/Cargo.toml index 8577ebd359..a3221b34d4 100644 --- a/crates/testing/seam-probe/Cargo.toml +++ b/crates/testing/seam-probe/Cargo.toml @@ -23,6 +23,7 @@ validator = { workspace = true } [dev-dependencies] hex = { workspace = true } +trusted-server-core = { workspace = true, features = ["test-utils"] } sha2 = { workspace = true } tokio = { workspace = true } diff --git a/crates/testing/seam-probe/src/lib.rs b/crates/testing/seam-probe/src/lib.rs index 9a1720803a..211c2e56e2 100644 --- a/crates/testing/seam-probe/src/lib.rs +++ b/crates/testing/seam-probe/src/lib.rs @@ -2,10 +2,10 @@ //! part of the integration seam from a vendor crate's position. //! //! One registration carries a browser module, a proxy route, a geo module, -//! an Edge Cookie identity module and a device module, alongside its own -//! configuration block, and the builder adds a request preparer, a demand -//! implementation `[demand]` can name and an ad server implementation -//! `[ad-server]` can name. The round-trip +//! an Edge Cookie identity module, a device module and a middleware that +//! changes a page, alongside its own configuration block, and the builder +//! adds a request preparer, a demand implementation `[demand]` can name and +//! an ad server implementation `[ad-server]` can name. The round-trip //! tests in `crates/trusted-server-adapter-axum/tests/seam_probe.rs` and //! `crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs` drive //! each of those through a real adapter, so the seam is proven by a caller that @@ -20,6 +20,7 @@ use std::collections::BTreeMap; use std::net::IpAddr; +use std::rc::Rc; use std::sync::{Arc, LazyLock, Mutex}; use async_trait::async_trait; @@ -40,8 +41,11 @@ use trusted_server_core::ec::module::{EdgeCookieModule, GeneratedEdgeCookie, Mod use trusted_server_core::error::TrustedServerError; use trusted_server_core::evidence::RequestInfo; use trusted_server_core::integrations::{ - CarriedJsModule, IntegrationBuilder, IntegrationEndpoint, IntegrationProxy, - IntegrationRegistration, + AttributeRewriteAction, CarriedJsModule, IntegrationBuilder, IntegrationEndpoint, + IntegrationProxy, IntegrationRegistration, ScriptRewriteAction, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, TextHandler, }; use trusted_server_core::module_context::ModuleCall; use trusted_server_core::platform::{ @@ -139,6 +143,76 @@ fn count_prepare_run(request: &Request) { *counts.entry(token.to_string()).or_insert(0) += 1; } +/// What the probe's middleware writes at the start of `` when the probe +/// is set to `country`. +#[must_use] +pub fn seam_probe_head_marker(country: &str) -> String { + format!("") +} + +/// The word the probe's middleware swaps for the country, in the text of a +/// script that carries `data-seam-probe`. +pub const SEAM_PROBE_SCRIPT_WORD: &str = "SEAM_PROBE_COUNTRY"; + +/// The probe's page change, which an entry names as the module is named. +/// +/// It marks the head with the country the probe is set to, points every link +/// that carries `data-seam-probe` at the probe's report route, and writes the +/// country in place of [`SEAM_PROBE_SCRIPT_WORD`] in the text of a script +/// that carries `data-seam-probe`. +pub struct SeamProbeMiddleware { + country: String, +} + +impl Middleware for SeamProbeMiddleware { + fn middleware_id(&self) -> &'static str { + module_name() + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + head_inserts: vec![seam_probe_head_marker(&self.country)], + element_handlers: vec![Box::new(AttributeRewrite::matching( + "a[data-seam-probe]", + "href", + Rc::new(|_matched| AttributeRewriteAction::replace(SEAM_PROBE_REPORT_PATH)), + ))], + text_handlers: vec![Box::new(CountryInScript { + country: self.country.clone(), + held: String::new(), + })], + ..MiddlewareAction::pass() + } + } +} + +/// Holds a script's text until its last chunk arrives, then writes the whole +/// of it back with the word swapped, because the word may lie across chunks. +struct CountryInScript { + country: String, + held: String, +} + +impl TextHandler for CountryInScript { + fn selector(&self) -> &str { + "script[data-seam-probe]" + } + + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + self.held.push_str(text); + if !is_last { + return ScriptRewriteAction::remove_node(); + } + ScriptRewriteAction::replace( + std::mem::take(&mut self.held).replace(SEAM_PROBE_SCRIPT_WORD, &self.country), + ) + } +} + /// Marker the request preparer inserts into the request extensions. /// /// The proxy route reports [`Self::runs`], so a test observes both that the @@ -520,7 +594,10 @@ pub fn register( .with_js_module(CarriedJsModule { source: PROBE_JS, sha256: PROBE_JS_SHA256, - }); + }) + .with_middleware(Arc::new(SeamProbeMiddleware { + country: config.country.clone(), + })); if config.declares_geo { registration = registration .with_geo_module(module_name(), Arc::new(SeamProbeGeo::new(config.country))); @@ -730,6 +807,97 @@ mod tests { .expect("should parse seam probe test settings") } + /// A page with one link and one script the probe's middleware changes, + /// and one of each that it leaves alone. + const PROBE_PAGE: &str = concat!( + "Page", + "probe", + "other", + "", + "", + "", + ); + + /// [`PROBE_PAGE`] as a reader receives it from a deployment running + /// `settings` with the probe registered, read `chunk_size` bytes at a + /// time. + fn probe_page(settings: &Settings, chunk_size: usize) -> String { + use trusted_server_core::html_processor::test_support::processed_page; + use trusted_server_core::integrations::IntegrationRegistry; + + let registry = IntegrationRegistry::with_registrations(settings, &[builder()]) + .expect("should build a registry with the probe registered"); + processed_page(settings, ®istry, PROBE_PAGE, chunk_size) + } + + #[test] + fn the_middleware_changes_a_page_where_an_entry_names_it() { + use trusted_server_core::html_processor::test_support::REQUEST_HOST; + + let settings = settings_with_probe( + r#"country = "NZ" + + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] + "#, + ); + + // Read whole, and then seven bytes at a time so the script's text + // reaches the handler in pieces. + for chunk_size in [8192, 7] { + let page = probe_page(&settings, chunk_size); + + assert_eq!( + page.matches(&seam_probe_head_marker("NZ")).count(), + 1, + "should mark the head once, with the country the probe is set to: {page}" + ); + assert!( + page.contains(&format!( + "probe" + )), + "should point the probe's link at its report route: {page}" + ); + assert!( + page.contains(&format!( + "other" + )), + "should leave another link to core, which moves it to the reader's host: {page}" + ); + assert!( + page.contains(""), + "should write the country into the probe's script: {page}" + ); + assert!( + page.contains(""), + "should leave another script as it was: {page}" + ); + } + } + + #[test] + fn the_middleware_changes_nothing_where_no_entry_names_it() { + use trusted_server_core::html_processor::test_support::REQUEST_HOST; + + let page = probe_page(&settings_with_probe(r#"country = "NZ""#), 8192); + + assert!( + !page.contains("seam-probe-country"), + "should not mark the head of a page no entry places it on: {page}" + ); + assert!( + page.contains(&format!( + "probe" + )), + "should leave the probe's link to core: {page}" + ); + assert!( + page.contains(""), + "should leave the probe's script as it was: {page}" + ); + } + /// The registry rejects a carried module whose declared hash is wrong, so /// this keeps the fixture's literal honest rather than letting every other /// test fail with a startup error. diff --git a/crates/trusted-server-adapter-axum/tests/seam_probe.rs b/crates/trusted-server-adapter-axum/tests/seam_probe.rs index 82e1db37cd..0bef24a2ae 100644 --- a/crates/trusted-server-adapter-axum/tests/seam_probe.rs +++ b/crates/trusted-server-adapter-axum/tests/seam_probe.rs @@ -378,6 +378,74 @@ fn geo_selector_naming_a_module_without_a_geo_module_fails_at_startup() { ); } +/// A `[[fetch]]` entry naming the probe's middleware starts the adapter, and +/// the registry runs that middleware on the paths the entry covers. +#[test] +fn fetch_entry_naming_a_modules_middleware_starts_the_adapter() { + use trusted_server_core::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; + + let settings = || { + settings_with(&format!( + r#" + {PROBE_BLOCK} + + [[fetch]] + media_type = "text/html" + path = "/news/" + middleware = ["testing.seam-probe"] + "# + )) + }; + + let registry = registry_with_probe(&settings()); + let chain_for = |path: &str| { + registry + .middleware_chain( + &settings().fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + path, + ) + .ids() + }; + assert_eq!( + chain_for("/news/today"), + [seam_probe::module_name()], + "should run the module's middleware under the entry's path" + ); + assert!( + chain_for("/sport/today").is_empty(), + "should run nothing on a path no entry covers" + ); + + TrustedServerApp::routes_with_registrations(settings(), &[seam_probe::builder()]) + .expect("should start with an entry naming a middleware the module supplies"); +} + +/// A `[[fetch]]` entry naming the middleware of a module no section selects +/// is a startup error, raised where the adapter builds its routes. +#[test] +fn fetch_entry_naming_the_middleware_of_an_unselected_module_fails_at_startup() { + let settings = settings_with( + r#" + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] + "#, + ); + + let error = TrustedServerApp::routes_with_registrations(settings, &[seam_probe::builder()]) + .err() + .expect("should refuse to start when an entry names a middleware that does not run"); + + let message = error.to_string(); + assert!( + message.contains("[[fetch]] entry 1 names `testing.seam-probe`") + && message.contains("`testing.seam-probe` is a module no section selects"), + "should name the entry and say why the middleware is not running: {message}" + ); +} + // --------------------------------------------------------------------------- // 5. Two builders claiming one id are rejected, naming both sources // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs index 251a9b62f7..70a9d0fa47 100644 --- a/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs +++ b/crates/trusted-server-adapter-fastly/src/app/seam_probe_tests.rs @@ -256,6 +256,36 @@ fn a_named_route_and_the_fallback_each_prepare_the_request_exactly_once() { ); } +/// A `[[fetch]]` entry naming the probe's middleware builds this adapter's +/// state, and one naming the middleware of a module no section selects is a +/// startup error. +#[test] +fn fetch_entry_builds_state_only_when_its_middleware_is_running() { + const ENTRY: &str = r#" + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] +"#; + + let selected = settings_with(&format!("{HMAC_BLOCK}{PROBE_BLOCK}{ENTRY}")); + assert!( + build_state_with_registrations(selected, &[seam_probe::builder()]).is_ok(), + "should build state with an entry naming a middleware the module supplies" + ); + + let unselected = settings_with(&format!("{HMAC_BLOCK}{ENTRY}")); + let error = build_state_with_registrations(unselected, &[seam_probe::builder()]) + .err() + .expect("should refuse to start when an entry names a middleware that does not run"); + + let message = error.to_string(); + assert!( + message.contains("[[fetch]] entry 1 names `testing.seam-probe`") + && message.contains("`testing.seam-probe` is a module no section selects"), + "should name the entry and say why the middleware is not running: {message}" + ); +} + /// `[geo] module` naming a selected module that supplies no geo module is a /// startup error, raised where this adapter builds its state. #[test] diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index 38fd987d40..73e3fcb1ad 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -354,6 +354,7 @@ pub fn validate_settings_for_deploy_with( // The selection is checked first, so a block nothing runs is reported as // that rather than as whatever its unread settings fail next. settings.validate_module_sections()?; + settings.validate_phase_entries()?; validate_secret_key_references(settings, extra_integrations)?; validate_non_secret_deploy_placeholders(settings)?; diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 9f0bc886aa..b99808c76f 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -1,7 +1,7 @@ //! Simplified HTML processor that combines URL replacement and integration injection //! //! This module provides a `StreamProcessor` implementation for HTML content. -use std::cell::Cell; +use std::cell::{Cell, RefCell}; use std::io; use std::rc::Rc; use std::sync::Arc; @@ -9,15 +9,16 @@ use std::sync::atomic::{AtomicBool, Ordering}; use lol_html::{ EndTagHandler, Settings as RewriterSettings, element, end, - html_content::{ContentType, EndTag}, + html_content::{ContentType, EndTag, TextChunk}, text, }; use crate::integrations::{ - AttributeRewriteOutcome, IntegrationAttributeContext, IntegrationDocumentState, - IntegrationHtmlContext, IntegrationRegistry, IntegrationRequestState, IntegrationScriptContext, - ScriptRewriteAction, + AttributeRewriteAction, AttributeRewriteOutcome, IntegrationAttributeContext, + IntegrationDocumentState, IntegrationHtmlContext, IntegrationRegistry, IntegrationRequestState, + IntegrationScriptContext, ScriptRewriteAction, }; +use crate::middleware::{MatchedAttribute, MiddlewareChain, MiddlewareContext, MiddlewarePlan}; use crate::publisher::build_empty_bids_script; use crate::settings::Settings; use crate::streaming_processor::{HtmlRewriterAdapter, StreamProcessor}; @@ -194,6 +195,42 @@ impl HtmlProcessorConfig { } } +/// The chunk of text being rewritten, as the last handler to change it left +/// it, shared by every text handler of one document. +/// +/// The parser hands every handler a chunk as the origin sent it and writes +/// only the last replacement, so without this a handler would undo the change +/// of each handler before it. +#[derive(Default)] +struct TextEdits(RefCell); + +impl TextEdits { + /// The chunk as the handlers before this one left it. + fn current(&self, chunk: &TextChunk<'_>) -> String { + if chunk.removed() { + self.0.borrow().clone() + } else { + chunk.as_str().to_owned() + } + } + + /// Writes `text` in the chunk's place, as it is. + /// + /// Written as markup, because a handler is handed the text as the page + /// wrote it. Writing it as text would escape `&`, `<` and `>`, and a + /// browser decodes none of those inside a script. + fn replace(&self, chunk: &mut TextChunk<'_>, text: &str) { + chunk.replace(text, ContentType::Html); + text.clone_into(&mut self.0.borrow_mut()); + } + + /// Leaves the chunk out, along with anything an earlier handler wrote in + /// its place. + fn remove(&self, chunk: &mut TextChunk<'_>) { + self.replace(chunk, ""); + } +} + /// Create an HTML processor with URL replacement and integration hooks. /// /// # Panics @@ -201,11 +238,67 @@ impl HtmlProcessorConfig { /// Panics if the `ad_bids_state` `Mutex` is poisoned. This cannot happen in /// normal operation since no code holds the lock across a panic boundary. #[must_use] +pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcessor { + build_html_processor(config, MiddlewarePlan::default()) +} + +/// [`create_html_processor`], with the middleware of `chain` run on the +/// document as well. +/// +/// Each middleware is asked once, here, what it will do with this document. +/// The state the chain shares is the document's own and holds nothing a +/// request left, because what this processor writes may be stored and served +/// to other readers. +/// +/// A middleware's head markup follows the hooks' own. Its element handlers +/// are asked after core has moved the origin's address in an attribute and +/// after the hooks. Its text handlers are handed a chunk as the hooks and the +/// handlers before them left it, and its stream processors run after the +/// hooks' own. +/// +/// # Errors +/// +/// When a handler's selector does not parse, naming the selector. +/// +/// # Panics +/// +/// As [`create_html_processor`]. +pub fn create_html_processor_with_middleware( + config: HtmlProcessorConfig, + chain: &MiddlewareChain, +) -> Result, String> { + let document_state = IntegrationDocumentState::default(); + let plan = chain.plan(&MiddlewareContext { + phase: chain.phase(), + request_host: &config.request_host, + request_scheme: &config.request_scheme, + origin_host: &config.origin_host, + document_state: &document_state, + max_buffered_script_bytes: config.max_buffered_body_bytes, + })?; + Ok(build_html_processor(config, plan)) +} + +/// The processor behind [`create_html_processor`] and +/// [`create_html_processor_with_middleware`], carrying out `plan`. +/// +/// A handler in `plan` whose selector does not parse panics here, which +/// [`MiddlewareChain::plan`] rules out by refusing to plan one. #[allow( clippy::needless_pass_by_value, reason = "the returned processor owns request configuration captured by its handlers" )] -pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcessor { +fn build_html_processor( + config: HtmlProcessorConfig, + plan: MiddlewarePlan, +) -> impl StreamProcessor + use<> { + let MiddlewarePlan { + head_inserts: middleware_head_inserts, + after_bundle_inserts: middleware_after_bundle_inserts, + element_handlers: middleware_element_handlers, + text_handlers: middleware_text_handlers, + processors: middleware_processors, + } = plan; let stream_processor_factories = config.integrations.html_stream_processor_factories(); let document_state = IntegrationDocumentState::default(); config.request_state.seed(&document_state); @@ -335,6 +428,9 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso for insert in integrations.head_inserts(&ctx) { snippet.push_str(&insert); } + for insert in &middleware_head_inserts { + snippet.push_str(insert); + } // Main bundle: core + non-deferred integrations (synchronous). let immediate_parts = integrations.js_parts_immediate(); let script_attributes = integrations.tsjs_script_tag_attributes(); @@ -347,6 +443,9 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso for insert in integrations.after_bundle_inserts(&ctx) { snippet.push_str(&insert); } + for insert in &middleware_after_bundle_inserts { + snippet.push_str(insert); + } // Deferred bundles: large modules like prebid loaded after // HTML parsing completes. Empty when none are enabled. let deferred_parts = integrations.js_parts_deferred(); @@ -668,15 +767,51 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso })); } + // Asked after core has moved the origin's address in an attribute and + // after the hooks, so a handler judges an address as it will be served. + for mut handler in middleware_element_handlers { + let selector = handler.selector().to_owned(); + let attribute = handler.attribute().to_owned(); + element_content_handlers.push(element!(selector, move |el| { + if el.removed() { + return Ok(()); + } + let Some(value) = el.get_attribute(&attribute) else { + return Ok(()); + }; + let element_name = el.tag_name(); + match handler.decide(&MatchedAttribute { + element_name: &element_name, + attribute_name: &attribute, + value: &value, + }) { + AttributeRewriteAction::Keep => {} + AttributeRewriteAction::Replace(replacement) => { + if replacement != value { + el.set_attribute(&attribute, &replacement)?; + } + } + AttributeRewriteAction::RemoveElement => el.remove(), + } + Ok(()) + })); + } + + // Shared by every text handler below, the hooks' and the middleware's, + // so each is handed a chunk as the one before it left it. + let edits = Rc::new(TextEdits::default()); + for script_rewriter in script_rewriters { let selector = script_rewriter.selector(); let rewriter = script_rewriter.clone(); let patterns = patterns.clone(); let document_state = document_state.clone(); + let edits = Rc::clone(&edits); element_content_handlers.push(text!(selector, { let rewriter = rewriter.clone(); let patterns = patterns.clone(); let document_state = document_state.clone(); + let edits = Rc::clone(&edits); move |text| { let ctx = IntegrationScriptContext { selector, @@ -687,24 +822,31 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso max_buffered_script_bytes: config.max_buffered_body_bytes, document_state: &document_state, }; - match rewriter.rewrite(text.as_str(), &ctx) { + let current = edits.current(text); + match rewriter.rewrite(¤t, &ctx) { ScriptRewriteAction::Keep => {} - ScriptRewriteAction::Replace(rewritten) => { - // Written back as markup, because the rewriter was - // handed the text as the page wrote it. Writing it as - // text would escape `&`, `<` and `>`, and a browser - // decodes none of those inside a script. - text.replace(&rewritten, ContentType::Html); - } - ScriptRewriteAction::RemoveNode => { - text.remove(); - } + ScriptRewriteAction::Replace(rewritten) => edits.replace(text, &rewritten), + ScriptRewriteAction::RemoveNode => edits.remove(text), } Ok(()) } })); } + for mut handler in middleware_text_handlers { + let selector = handler.selector().to_owned(); + let edits = Rc::clone(&edits); + element_content_handlers.push(text!(selector, move |text| { + let current = edits.current(text); + match handler.decide(¤t, text.last_in_text_node()) { + ScriptRewriteAction::Keep => {} + ScriptRewriteAction::Replace(rewritten) => edits.replace(text, &rewritten), + ScriptRewriteAction::RemoveNode => edits.remove(text), + } + Ok(()) + })); + } + let rewriter_settings = RewriterSettings { document_content_handlers, element_content_handlers, @@ -719,10 +861,11 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso origin_host: config.origin_host.clone(), document_state: document_state.clone(), }; - let processors = stream_processor_factories + let mut processors: Vec> = stream_processor_factories .into_iter() .map(|factory| factory.create(stream_context.clone())) .collect(); + processors.extend(middleware_processors); HtmlWithStreamingProcessors { inner: Box::new(inner), processors, @@ -739,8 +882,9 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso pub mod test_support { use std::io::Cursor; - use super::{HtmlProcessorConfig, create_html_processor}; + use super::{HtmlProcessorConfig, create_html_processor_with_middleware}; use crate::integrations::IntegrationRegistry; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; use crate::settings::Settings; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; @@ -758,9 +902,13 @@ pub mod test_support { /// changes whenever any browser module does. const BUNDLE_HASH: &str = "BUNDLE-HASH"; + /// The path a test page is asked for at, which decides the entry of the + /// settings that covers it. + pub const REQUEST_PATH: &str = "/"; + /// The page a reader of `html` receives from a deployment running /// `settings` with `registry`'s modules, read from the origin `chunk_size` - /// bytes at a time. + /// bytes at a time and asked for at [`REQUEST_PATH`]. /// /// The script bundle's `?v=` hash is replaced by a fixed word, so a /// recording changes when a module's page change does and not when a @@ -784,13 +932,20 @@ pub mod test_support { REQUEST_HOST, "https", ); + let chain = registry.middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + REQUEST_PATH, + ); let mut pipeline = StreamingPipeline::new( PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, chunk_size, }, - create_html_processor(config), + create_html_processor_with_middleware(config, &chain) + .expect("should plan the page's middleware"), ); let mut output = Vec::new(); pipeline diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index f12f4671b5..4466be1b5f 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -717,6 +717,10 @@ const BUILT_IN_BUILDERS: &[IntegrationBuilder] = &[ // the same tests. #[cfg(test)] registry_test_support::deferred_fixture::BUILDER, + // A stand-in for a module that changes a page through middleware, for + // the same tests. + #[cfg(test)] + registry_test_support::middleware_fixture::BUILDER, // A stand-in for the plainest demand implementation there can be, which // core's own tests name where they need a source. #[cfg(test)] diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index b3cb49a253..d5ec794447 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -18,6 +18,7 @@ use crate::ec::module::{EcModuleSelection, EdgeCookieModule}; use crate::error::TrustedServerError; use crate::geo::GeoInfo; use crate::http_util::is_navigation_request; +use crate::middleware::{Middleware, MiddlewareChain, MiddlewarePhase, PhaseEntries}; use crate::module_context::{ModuleCall, ModuleContext, ResolvedRequest}; use crate::platform::{DisabledGeo, PlatformGeo, RuntimeServices}; use crate::settings::Settings; @@ -782,6 +783,11 @@ pub struct IntegrationRegistration { pub html_stream_processors: Vec>, pub head_injectors: Vec>, pub request_filters: Vec>, + /// The page changes this registration supplies, see [`crate::middleware`]. + /// + /// Declaring one does not make it run, because a middleware changes a + /// page only where an entry of the settings names it. + pub middleware: Vec>, /// Geo module this registration supplies, with the name `[geo] module` /// selects it by. /// @@ -830,6 +836,7 @@ impl IntegrationRegistrationBuilder { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters: Vec::new(), + middleware: Vec::new(), geo_module: None, ec_module: None, device_module: None, @@ -879,6 +886,18 @@ impl IntegrationRegistrationBuilder { self } + /// Declare a page change this registration supplies, see + /// [`crate::middleware`]. Called once for each one. + /// + /// The middleware changes a page only where an entry of the settings + /// names it, and one no entry names is logged as a warning when the + /// registry is built. + #[must_use] + pub fn with_middleware(mut self, middleware: Arc) -> Self { + self.registration.middleware.push(middleware); + self + } + /// Declare the geo module this registration supplies, under its `name`. /// /// The name is the path under `crates` of the crate the module lives in, @@ -998,6 +1017,9 @@ struct IntegrationRegistryInner { html_stream_processors: Vec>, head_injectors: Vec>, request_filters: Vec>, + // The middleware the running registrations supply, each against the + // integration that supplied it, in registration order. + middleware: Vec<(&'static str, Arc)>, /// JS module IDs to include in the bundle that come from a source other than /// a registered integration, for example a module tied to the selected Edge /// Cookie module. Populated in [`IntegrationRegistry::new`] from settings. @@ -1049,6 +1071,7 @@ impl Default for IntegrationRegistryInner { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters: Vec::new(), + middleware: Vec::new(), extra_js_module_ids: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), @@ -1087,6 +1110,35 @@ const DEVICE_TYPE: &str = "device"; const GEO_TYPE: &str = "geo"; impl IntegrationRegistryInner { + /// The registered middleware an entry means by `name`. + fn middleware_named(&self, name: &str) -> Option<&Arc> { + self.middleware + .iter() + .map(|(_, middleware)| middleware) + .find(|middleware| middleware.middleware_id() == name) + } + + /// A sentence to follow the refusal of a middleware name an entry wrote, + /// when the name is that of a module no section selects, or begins with + /// one. Empty otherwise. + fn unselected_module_note(&self, name: &str) -> String { + self.builder_modules + .iter() + .find(|(module, selected)| { + !*selected + && name + .strip_prefix(*module) + .is_some_and(|rest| rest.is_empty() || rest.starts_with('.')) + }) + .map(|(module, _)| { + format!( + ". `{module}` is a module no section selects, so nothing it supplies is \ + running" + ) + }) + .unwrap_or_default() + } + /// A sentence to follow the refusal of a name a selector wrote, when the /// name is a builder's module: one no section selects, which is why /// nothing it supplies is running, or one that is selected and supplies no @@ -1373,6 +1425,113 @@ pub struct ProxyDispatchInput<'a> { pub req: Request, } +/// Refuses a registered middleware no entry could name, and an entry this +/// deployment cannot run. +/// +/// Only knowable here, where every running module's middleware have been +/// handed over. A name no module supplies, or one named in a phase it does not +/// run in, would otherwise do nothing on every request and say so nowhere. +/// +/// A middleware no entry names is left alone and logged, because a module +/// may be selected for what else it does. +/// +/// # Errors +/// +/// Naming the integration or the entry at fault. +fn check_middleware( + settings: &Settings, + inner: &IntegrationRegistryInner, +) -> Result<(), Report> { + let refuse = |message: String| Report::new(TrustedServerError::Configuration { message }); + let phases_of = |middleware: &Arc| -> Vec { + middleware + .phases() + .iter() + .map(|phase| format!("[[{phase}]]")) + .collect() + }; + for (position, (integration, middleware)) in inner.middleware.iter().enumerate() { + let id = middleware.middleware_id(); + if !crate::module_name::is_valid(id) { + return Err(refuse(format!( + "integration `{integration}` supplies a middleware named `{id}`, which is not a \ + name an entry can write. A name is parts joined by `.`, each in lower case \ + letters, digits, `_` or `-`" + ))); + } + if middleware.phases().is_empty() { + return Err(refuse(format!( + "integration `{integration}` supplies the middleware `{id}`, which says it runs \ + in no phase, so no entry could name it" + ))); + } + if let Some((earlier, _)) = inner.middleware[..position] + .iter() + .find(|(_, earlier)| earlier.middleware_id() == id) + { + return Err(refuse(format!( + "integrations `{earlier}` and `{integration}` both supply a middleware named \ + `{id}`, so an entry could not say which one it means" + ))); + } + } + for phase in MiddlewarePhase::ALL { + for (index, entry) in settings.phase_entries(phase).entries().iter().enumerate() { + let at = format!("[[{phase}]] entry {}", index + 1); + for name in &entry.middleware { + let Some(middleware) = inner.middleware_named(name) else { + let supplied: Vec<&str> = inner + .middleware + .iter() + .map(|(_, middleware)| middleware.middleware_id()) + .collect(); + return Err(refuse(format!( + "{at} names `{name}`, which no module that runs supplies. The \ + middleware the running modules supply is [{}]{}", + supplied.join(", "), + inner.unselected_module_note(name) + ))); + }; + if !middleware.phases().contains(&phase) { + return Err(refuse(format!( + "{at} names `{name}`, which does not run in that phase. It runs in {}", + phases_of(middleware).join(" and ") + ))); + } + } + } + } + for (integration, middleware) in unnamed_middleware(settings, inner) { + log::warn!( + "Integration `{integration}` supplies the middleware `{}` and no {} entry names \ + it, so it changes no page", + middleware.middleware_id(), + phases_of(middleware).join(" or ") + ); + } + Ok(()) +} + +/// The middleware no entry names in a phase it runs in, each with the +/// integration that supplies it, in registration order. +fn unnamed_middleware<'a>( + settings: &Settings, + inner: &'a IntegrationRegistryInner, +) -> Vec<(&'static str, &'a Arc)> { + inner + .middleware + .iter() + .filter(|(_, middleware)| { + let id = middleware.middleware_id(); + !middleware + .phases() + .iter() + .any(|phase| settings.phase_entries(*phase).names().contains(&id)) + }) + .map(|(integration, middleware)| (*integration, middleware)) + .collect() +} + /// In-memory registry of integrations discovered from settings. #[derive(Clone, Default)] pub struct IntegrationRegistry { @@ -1644,6 +1803,11 @@ impl IntegrationRegistry { .extend(registration.html_stream_processors); inner.head_injectors.extend(registration.head_injectors); inner.request_filters.extend(registration.request_filters); + for middleware in registration.middleware { + inner + .middleware + .push((registration.integration_id, middleware)); + } if let Some((name, module)) = registration.geo_module { claim_module_name(&mut claimed, "geo", name, registration.integration_id)?; inner.geo_modules.push((name, module)); @@ -1723,6 +1887,7 @@ impl IntegrationRegistry { inner.ec_module = resolve_ec_module(settings, &inner); inner.device_module = resolve_device_module(settings, &inner)?; inner.geo_module = geo_module; + check_middleware(settings, &inner)?; Ok(Self { inner: Arc::new(inner), @@ -1760,6 +1925,36 @@ impl IntegrationRegistry { self.inner.device_module.clone() } + /// The name of every middleware a running module supplies, in + /// registration order. + #[must_use] + pub fn middleware_ids(&self) -> Vec<&'static str> { + self.inner + .middleware + .iter() + .map(|(_, middleware)| middleware.middleware_id()) + .collect() + } + + /// The middleware the first entry covering a response selects, in the + /// order they run, for a response of `media_type` to a request for `path`. + /// A response no entry covers gives an empty chain. + #[must_use] + pub fn middleware_chain( + &self, + entries: &PhaseEntries, + phase: MiddlewarePhase, + media_type: &str, + path: &str, + ) -> MiddlewareChain { + let middleware = entries + .for_response(media_type, path) + .iter() + .filter_map(|name| self.inner.middleware_named(name).map(Arc::clone)) + .collect(); + MiddlewareChain::new(phase, middleware) + } + /// Every integration id the registry was built from, named or not, in /// registration order. A registry test enumerates this to check every /// builder was considered, while the deploy validation test iterates the @@ -2305,6 +2500,7 @@ impl IntegrationRegistry { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters: Vec::new(), + middleware: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2348,6 +2544,7 @@ impl IntegrationRegistry { html_stream_processors: Vec::new(), head_injectors, request_filters: Vec::new(), + middleware: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2387,6 +2584,7 @@ impl IntegrationRegistry { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters, + middleware: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2466,6 +2664,7 @@ impl IntegrationRegistry { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters: Vec::new(), + middleware: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2867,6 +3066,153 @@ pub(crate) mod test_support { } } + /// A stand-in for a module that changes a page through middleware, for + /// core's own tests of the entries and of the page path. + /// + /// Selected, it supplies three fetch middleware. The one under the + /// module's own name writes a marker carrying the module's `label` + /// setting at the start of ``. The one named `.links` points every + /// link that carries `data-fixture` at a fixed path. The one named + /// `.broken` asks for a selector that does not parse. + pub(crate) mod middleware_fixture { + use std::rc::Rc; + use std::sync::Arc; + + use error_stack::Report; + + use crate::error::TrustedServerError; + use crate::integrations::registry::{AttributeRewriteAction, IntegrationRegistration}; + use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{ + AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, + }; + use crate::settings::Settings; + + /// The integration id the stand-in registers under. + pub(crate) const ID: &str = "middleware_fixture"; + /// The name a test's settings select the stand-in by, in `[testing]`. + pub(crate) const MODULE: &str = "testing.middleware-fixture"; + /// The name of the middleware that marks the head. + pub(crate) const HEAD: &str = MODULE; + /// The name of the middleware that moves links. + pub(crate) const LINKS: &str = "testing.middleware-fixture.links"; + /// The name of the middleware whose selector does not parse. + pub(crate) const BROKEN: &str = "testing.middleware-fixture.broken"; + /// Where the links middleware points a link. + pub(crate) const LINK_TARGET: &str = "/fixture/link"; + /// The selector the broken middleware asks for. + pub(crate) const BROKEN_SELECTOR: &str = "a["; + + /// The builder core's test build lists beside its own. + pub(crate) const BUILDER: IntegrationBuilder = + IntegrationBuilder::new(ID, CORE_SOURCE, register, validate).with_module_name(MODULE); + + #[derive(Debug, serde::Deserialize, validator::Validate)] + #[serde(deny_unknown_fields)] + struct FixtureSettings { + #[serde(default)] + label: String, + } + + impl crate::settings::IntegrationConfig for FixtureSettings {} + + /// The marker the head middleware writes for `label`. + pub(crate) fn head_marker(label: &str) -> String { + format!("") + } + + struct Head { + label: String, + } + + impl Middleware for Head { + fn middleware_id(&self) -> &'static str { + HEAD + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + head_inserts: vec![head_marker(&self.label)], + ..MiddlewareAction::pass() + } + } + } + + struct Links; + + impl Middleware for Links { + fn middleware_id(&self) -> &'static str { + LINKS + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + "a[data-fixture]", + "href", + Rc::new(|_matched| AttributeRewriteAction::replace(LINK_TARGET)), + ))], + ..MiddlewareAction::pass() + } + } + } + + struct Broken; + + impl Middleware for Broken { + fn middleware_id(&self) -> &'static str { + BROKEN + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + BROKEN_SELECTOR, + "href", + Rc::new(|_matched| AttributeRewriteAction::keep()), + ))], + ..MiddlewareAction::pass() + } + } + } + + fn register( + settings: &Settings, + ) -> Result, Report> { + let Some(config) = settings.module_config::(MODULE)? else { + return Ok(None); + }; + Ok(Some( + IntegrationRegistration::builder(ID) + .without_js() + .with_middleware(Arc::new(Head { + label: config.label, + })) + .with_middleware(Arc::new(Links)) + .with_middleware(Arc::new(Broken)) + .build(), + )) + } + + fn validate(settings: &Settings) -> Result> { + settings + .module_config::(MODULE) + .map(|config| config.is_some()) + } + } + /// A stand-in for an integration that tags a page, for core's own tests /// of the HTML processor and the JavaScript asset proxy. /// @@ -3260,11 +3606,13 @@ pub(crate) mod test_support { #[cfg(test)] mod tests { + use super::test_support::middleware_fixture as fixture; use super::test_support::{ PROBE_JS, PROBE_JS_SHA256, carried_probe_registration, probe_registration, validate_nothing, }; use super::*; use crate::constants::COOKIE_TS_EC; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewareAction, MiddlewareContext, PhaseEntry}; use crate::permissions::{Permission, PermissionSet, PermissionState}; use crate::platform::test_support::noop_services; use http::{HeaderValue, StatusCode, header}; @@ -5894,4 +6242,305 @@ mod tests { "error should name the module and both integrations: {message}" ); } + + fn html_entry(path: Option<&str>, names: &[&str]) -> PhaseEntry { + PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + } + } + + /// Settings that select core's middleware stand-in and hold `entries`. + fn fixture_settings_with_entries(entries: Vec) -> Settings { + let mut settings = settings_naming(fixture::MODULE); + settings.fetch = PhaseEntries::new(entries); + settings + } + + #[test] + fn a_selected_module_s_middleware_run_where_an_entry_names_them() { + let settings = fixture_settings_with_entries(vec![ + html_entry(Some("/news/"), &[fixture::LINKS, fixture::HEAD]), + html_entry(None, &[fixture::HEAD]), + ]); + + let registry = IntegrationRegistry::new(&settings) + .expect("should build a registry whose entries name registered middleware"); + + assert_eq!( + registry.middleware_ids(), + [fixture::HEAD, fixture::LINKS, fixture::BROKEN], + "should list the middleware the selected module supplies" + ); + let chain_for = |media_type: &str, path: &str| { + registry + .middleware_chain(&settings.fetch, MiddlewarePhase::Fetch, media_type, path) + .ids() + }; + assert_eq!( + chain_for(HTML_MEDIA_TYPE, "/news/today"), + [fixture::LINKS, fixture::HEAD], + "should run the first covering entry's middleware in the order it names them" + ); + assert_eq!(chain_for(HTML_MEDIA_TYPE, "/"), [fixture::HEAD]); + assert!( + chain_for("text/css", "/news/site.css").is_empty(), + "should run nothing on a media type no entry covers" + ); + } + + #[test] + fn a_module_no_section_selects_supplies_no_middleware() { + let settings = crate::test_support::tests::create_test_settings(); + + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + + assert!( + !registry.middleware_ids().contains(&fixture::HEAD), + "should register nothing of a module that does not run" + ); + } + + #[test] + fn an_entry_naming_a_middleware_nothing_supplies_is_refused_with_what_is_supplied() { + let settings = fixture_settings_with_entries(vec![ + html_entry(Some("/news/"), &[fixture::HEAD]), + html_entry(None, &[fixture::HEAD, "testing.nothing"]), + ]); + + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse an entry naming a middleware nothing supplies"); + + let message = error.to_string(); + assert!( + message.contains( + "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies" + ) && message.contains(&format!( + "[{}, {}, {}]", + fixture::HEAD, + fixture::LINKS, + fixture::BROKEN + )), + "should name the entry and list what could be named: {message}" + ); + assert!( + !message.contains("no section selects"), + "should not blame a selection when the name is no module's: {message}" + ); + } + + #[test] + fn an_entry_naming_the_middleware_of_a_module_no_section_selects_says_so() { + let mut settings = crate::test_support::tests::create_test_settings(); + settings.fetch = PhaseEntries::new(vec![html_entry(None, &[fixture::LINKS])]); + + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse an entry naming the middleware of a module that does not run"); + + let message = error.to_string(); + assert!( + message.contains(&format!("[[fetch]] entry 1 names `{}`", fixture::LINKS)) + && message.contains(&format!( + "`{}` is a module no section selects, so nothing it supplies is running", + fixture::MODULE + )), + "should say the module is not selected: {message}" + ); + } + + #[test] + fn a_middleware_no_entry_names_is_listed_for_the_startup_warning() { + let unnamed = |entries: Vec| -> Vec<&'static str> { + let settings = fixture_settings_with_entries(entries); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + unnamed_middleware(&settings, ®istry.inner) + .into_iter() + .map(|(integration, middleware)| { + assert_eq!(integration, fixture::ID, "should name the supplier"); + middleware.middleware_id() + }) + .collect() + }; + + assert_eq!( + unnamed(Vec::new()), + [fixture::HEAD, fixture::LINKS, fixture::BROKEN], + "should list every middleware when there are no entries" + ); + assert_eq!( + unnamed(vec![ + html_entry(Some("/news/"), &[fixture::LINKS]), + html_entry(None, &[fixture::HEAD]), + ]), + [fixture::BROKEN], + "should leave out a middleware any entry names" + ); + } + + /// A middleware under a name of the test's choosing, running in the + /// phases given. + struct Named { + id: &'static str, + phases: &'static [MiddlewarePhase], + } + + impl Middleware for Named { + fn middleware_id(&self) -> &'static str { + self.id + } + + fn phases(&self) -> &[MiddlewarePhase] { + self.phases + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction::pass() + } + } + + const FETCH_ONLY: &[MiddlewarePhase] = &[MiddlewarePhase::Fetch]; + + fn first_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("first_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "testing.shared-name", + phases: FETCH_ONLY, + })) + .build(), + )) + } + + fn second_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("second_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "testing.shared-name", + phases: FETCH_ONLY, + })) + .build(), + )) + } + + fn misnamed_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("misnamed_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "Not A Name", + phases: FETCH_ONLY, + })) + .build(), + )) + } + + fn phaseless_supplier( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("phaseless_supplier") + .without_js() + .with_middleware(Arc::new(Named { + id: "testing.phaseless", + phases: &[], + })) + .build(), + )) + } + + /// A builder for `register`, selected in `settings` under a module name + /// made from its id. + fn selected_supplier( + settings: &mut Settings, + id: &'static str, + module: &'static str, + register: crate::integrations::IntegrationBuilderFn, + ) -> crate::integrations::IntegrationBuilder { + settings.select_module("testing", module); + crate::integrations::IntegrationBuilder::new( + id, + "a-vendor-crate", + register, + validate_nothing, + ) + .with_module_name(module) + } + + #[test] + fn two_modules_supplying_one_middleware_name_are_refused() { + let mut settings = crate::test_support::tests::create_test_settings(); + let extra = [ + selected_supplier( + &mut settings, + "first_supplier", + "testing.first-supplier", + first_supplier, + ), + selected_supplier( + &mut settings, + "second_supplier", + "testing.second-supplier", + second_supplier, + ), + ]; + + let error = IntegrationRegistry::with_registrations(&settings, &extra) + .err() + .expect("should refuse one middleware name supplied twice"); + + let message = error.to_string(); + assert!( + message.contains("`first_supplier` and `second_supplier` both supply a middleware") + && message.contains("`testing.shared-name`"), + "should name both suppliers and the name: {message}" + ); + } + + #[test] + fn a_middleware_no_entry_could_name_is_refused() { + let mut settings = crate::test_support::tests::create_test_settings(); + let extra = [selected_supplier( + &mut settings, + "misnamed_supplier", + "testing.misnamed-supplier", + misnamed_supplier, + )]; + let error = IntegrationRegistry::with_registrations(&settings, &extra) + .err() + .expect("should refuse a middleware whose name no entry can write"); + let message = error.to_string(); + assert!( + message.contains("`misnamed_supplier` supplies a middleware named `Not A Name`") + && message.contains("not a name an entry can write"), + "should name the supplier and the name: {message}" + ); + + let mut settings = crate::test_support::tests::create_test_settings(); + let extra = [selected_supplier( + &mut settings, + "phaseless_supplier", + "testing.phaseless-supplier", + phaseless_supplier, + )]; + let error = IntegrationRegistry::with_registrations(&settings, &extra) + .err() + .expect("should refuse a middleware that runs in no phase"); + let message = error.to_string(); + assert!( + message.contains("`phaseless_supplier` supplies the middleware `testing.phaseless`") + && message.contains("runs in no phase"), + "should name the supplier and the middleware: {message}" + ); + } } diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 3fdab2e7d1..21b3961ae5 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -54,6 +54,7 @@ pub mod html_processor; pub mod http_util; pub mod inspect; pub mod integrations; +pub mod middleware; pub mod module_context; pub mod module_name; pub(crate) mod module_secrets; diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs new file mode 100644 index 0000000000..1deb630232 --- /dev/null +++ b/crates/trusted-server-core/src/middleware.rs @@ -0,0 +1,584 @@ +//! Page changes, which a module makes to a document on its way to a reader. +//! +//! A middleware is a page change the settings place. A module registers the +//! ones it supplies, and one runs only where an entry in the settings names +//! it. The entries are an ordered list. An entry covers one media type, +//! optionally only the requests under a path prefix, and names the middleware +//! to run in order. A response takes the first entry that covers it, and each +//! middleware is handed the document as the ones before it left it. +//! +//! ```toml +//! [[fetch]] +//! media_type = "text/html" +//! path = "/news/" +//! middleware = ["example.strip", "example.tag"] +//! +//! [[fetch]] +//! media_type = "text/html" +//! middleware = ["example.strip"] +//! ``` +//! +//! An entry carries no settings and switches nothing on. Whether a module +//! runs is decided by the section that selects it, and an entry places and +//! orders the page changes of the modules that run. A middleware reads its +//! settings from its module's own table. +//! +//! Fetch is obtaining the document from the origin. A fetch middleware is +//! handed nothing about the reader, so what it leaves is what a shared +//! template stores for every reader. + +use std::fmt; +use std::rc::Rc; +use std::sync::Arc; + +use serde::{Deserialize, Deserializer, Serialize, Serializer, de}; + +use crate::integrations::{AttributeRewriteAction, IntegrationDocumentState, ScriptRewriteAction}; +use crate::streaming_processor::StreamProcessor; + +/// The media type a document is matched as, and the only one a middleware +/// runs on. +pub const HTML_MEDIA_TYPE: &str = "text/html"; + +/// When a middleware runs, relative to the store of shared templates. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum MiddlewarePhase { + /// On the document as the origin sent it. What is left is what a shared + /// template stores, so nothing here may depend on the reader. + Fetch, +} + +impl MiddlewarePhase { + /// Every phase, in the order they happen. + pub const ALL: [Self; 1] = [Self::Fetch]; + + /// The name the phase's entries are written under in the settings. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::Fetch => "fetch", + } + } +} + +impl fmt::Display for MiddlewarePhase { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +/// What a middleware is told about the document it is asked to change. +/// +/// In [`MiddlewarePhase::Fetch`] the document's state holds nothing a +/// request left, because the document may be stored and served to other +/// readers. +#[derive(Debug, Clone, Copy)] +pub struct MiddlewareContext<'a> { + /// Which phase is running. + pub phase: MiddlewarePhase, + /// Publisher-facing host the reader asked for. + pub request_host: &'a str, + /// Publisher-facing scheme the reader asked for. + pub request_scheme: &'a str, + /// Host the document was fetched from. + pub origin_host: &'a str, + /// State shared by the middleware working on this document in this + /// phase. + pub document_state: &'a IntegrationDocumentState, + /// The most a middleware may hold of one script while it decides. + pub max_buffered_script_bytes: usize, +} + +/// One attribute of one matched element, as a handler is asked about it. +#[derive(Debug, Clone, Copy)] +pub struct MatchedAttribute<'a> { + /// The element's tag name, in lower case. + pub element_name: &'a str, + /// The attribute the handler judges. + pub attribute_name: &'a str, + /// The attribute's value, as the handlers before this one left it. Core + /// has already moved the origin's address in it to the reader's host. + pub value: &'a str, +} + +/// A middleware's decision about one attribute of the elements a selector +/// matches, being to keep the element, replace the attribute's value or +/// remove the element. +pub trait ElementHandler { + /// The CSS selector of the elements this handler is asked about. + fn selector(&self) -> &str; + + /// The attribute whose value the handler judges. An element the selector + /// matches that carries no such attribute is left as it is. + fn attribute(&self) -> &str; + + /// Decides what happens to one matched element. + fn decide(&mut self, matched: &MatchedAttribute<'_>) -> AttributeRewriteAction; +} + +/// The question an [`AttributeRewrite`] asks about each value. +pub type AttributeRewriteFn = dyn Fn(&MatchedAttribute<'_>) -> AttributeRewriteAction; + +/// An [`ElementHandler`] made from a function, such as one that points a +/// vendor's script at a first-party path. +pub struct AttributeRewrite { + selector: String, + attribute: &'static str, + decide: Rc, +} + +impl AttributeRewrite { + /// Asks `decide` about `attribute` on every element that carries it. + #[must_use] + pub fn new(attribute: &'static str, decide: Rc) -> Self { + Self::matching(&format!("[{attribute}]"), attribute, decide) + } + + /// Asks `decide` about `attribute` on the elements `selector` matches. + #[must_use] + pub fn matching( + selector: &str, + attribute: &'static str, + decide: Rc, + ) -> Self { + Self { + selector: selector.to_owned(), + attribute, + decide, + } + } +} + +impl ElementHandler for AttributeRewrite { + fn selector(&self) -> &str { + &self.selector + } + + fn attribute(&self) -> &str { + self.attribute + } + + fn decide(&mut self, matched: &MatchedAttribute<'_>) -> AttributeRewriteAction { + (self.decide)(matched) + } +} + +/// A middleware's decision about the text inside the elements a selector +/// matches, being to keep it, replace it or remove it. +/// +/// Text arrives in chunks. A handler that judges a whole script removes each +/// chunk as it arrives while keeping a copy, and writes back what it kept, as +/// it wants it, with the last chunk. +pub trait TextHandler { + /// The CSS selector of the elements whose text this handler is asked + /// about. + fn selector(&self) -> &str; + + /// Decides what happens to one chunk of text. `text` is the chunk as the + /// handlers before this one left it, and `is_last` says whether it is the + /// last chunk of its text node. + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction; +} + +/// What a middleware does to one document. A middleware may do several of +/// these at once, and an action with nothing set leaves the document as it +/// arrived. +#[derive(Default)] +pub struct MiddlewareAction { + /// Markup to write at the start of ``, in the order given. + pub head_inserts: Vec, + /// Markup to write straight after the main script bundle and before any + /// deferred one, for a script that needs the bundle to have run and has + /// to run before the page's own scripts. + pub after_bundle_inserts: Vec, + /// Decisions about elements. + pub element_handlers: Vec>, + /// Decisions about the text inside elements. + pub text_handlers: Vec>, + /// A processor over the document the handlers left. + pub stream: Option>, +} + +impl MiddlewareAction { + /// Nothing to do for this document. + #[must_use] + pub fn pass() -> Self { + Self::default() + } + + /// Whether the action leaves the document as it arrived. + #[must_use] + pub fn is_pass(&self) -> bool { + self.head_inserts.is_empty() + && self.after_bundle_inserts.is_empty() + && self.element_handlers.is_empty() + && self.text_handlers.is_empty() + && self.stream.is_none() + } + + /// Checks every handler's selector parses, before a rewriter is built + /// from it. + fn validate_selectors(&self) -> Result<(), String> { + let check = |selector: &str| { + selector + .parse::() + .map(|_| ()) + .map_err(|error| format!("`{selector}` is not a usable selector: {error}")) + }; + for handler in &self.element_handlers { + check(handler.selector())?; + } + for handler in &self.text_handlers { + check(handler.selector())?; + } + Ok(()) + } +} + +impl fmt::Debug for MiddlewareAction { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("MiddlewareAction") + .field("head_inserts", &self.head_inserts) + .field("after_bundle_inserts", &self.after_bundle_inserts) + .field( + "element_handlers", + &selectors_of_elements(&self.element_handlers), + ) + .field("text_handlers", &selectors_of_text(&self.text_handlers)) + .field("stream", &self.stream.is_some()) + .finish() + } +} + +fn selectors_of_elements(handlers: &[Box]) -> Vec { + handlers + .iter() + .map(|handler| format!("{}@{}", handler.selector(), handler.attribute())) + .collect() +} + +fn selectors_of_text(handlers: &[Box]) -> Vec { + handlers + .iter() + .map(|handler| handler.selector().to_owned()) + .collect() +} + +/// A page change a module supplies. +/// +/// A registered middleware holds no request state. For each document it is +/// asked once what it will do, and what it answers is used for that document +/// alone. +pub trait Middleware: Send + Sync { + /// The name an entry selects this middleware by. A module names its + /// middleware by its own name, as [`crate::module_name!`] gives it, with a + /// part of its own after it when the module supplies several, and core + /// names its own by a bare word. + fn middleware_id(&self) -> &'static str; + + /// The phases an entry may name this middleware in. + fn phases(&self) -> &[MiddlewarePhase]; + + /// Decides what to do with one document. + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction; +} + +/// One entry of a phase, such as `[[fetch]]`, being which responses it +/// covers and the middleware run on them, in the order written. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PhaseEntry { + /// The media type the entry covers, written as a bare `type/subtype`. + pub media_type: String, + /// A prefix of the request path the entry covers, compared as written, + /// or every path when absent. A folder is written with its closing + /// slash, because `/news` also begins `/newsletter`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub path: Option, + /// The middleware run on a response the entry covers, in the order + /// written. + pub middleware: Vec, +} + +impl PhaseEntry { + /// Whether the entry covers a response of `media_type` to a request for + /// `path`. + #[must_use] + pub fn covers(&self, media_type: &str, path: &str) -> bool { + self.media_type == media_type + && self + .path + .as_deref() + .is_none_or(|prefix| path.starts_with(prefix)) + } +} + +/// One phase's entries, in the order written. +/// +/// A response takes the first entry that covers it and runs that entry's +/// middleware alone, so an entry with a path goes before the entry covering +/// every path. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct PhaseEntries(Vec); + +impl PhaseEntries { + /// Entries, in the order they are matched. + #[must_use] + pub fn new(entries: Vec) -> Self { + Self(entries) + } + + /// The entries, in the order they are matched. + #[must_use] + pub fn entries(&self) -> &[PhaseEntry] { + &self.0 + } + + /// Whether the phase has no entries. + #[must_use] + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } + + /// The names the first entry covering a response selects, in the order + /// they run, or none when no entry covers it. + #[must_use] + pub fn for_response(&self, media_type: &str, path: &str) -> &[String] { + self.0 + .iter() + .find(|entry| entry.covers(media_type, path)) + .map_or(&[][..], |entry| entry.middleware.as_slice()) + } + + /// Every name the entries select, each once, in the order first written. + #[must_use] + pub fn names(&self) -> Vec<&str> { + let mut names: Vec<&str> = Vec::new(); + for entry in &self.0 { + for name in &entry.middleware { + if !names.contains(&name.as_str()) { + names.push(name); + } + } + } + names + } + + /// Checks each entry's media type, path and names, and that no entry is + /// hidden behind an earlier one. + /// + /// # Errors + /// + /// A message naming the entry by its position, and what is wrong with it. + pub fn validate(&self, phase: MiddlewarePhase) -> Result<(), String> { + for (index, entry) in self.0.iter().enumerate() { + let at = format!("[[{phase}]] entry {}", index + 1); + if entry.media_type != HTML_MEDIA_TYPE { + return Err(format!( + "{at} covers `{}`, and a middleware runs on `{HTML_MEDIA_TYPE}` alone. Write \ + media_type = \"{HTML_MEDIA_TYPE}\", in lower case with no parameters", + entry.media_type + )); + } + if let Some(path) = &entry.path { + if !path.starts_with('/') { + return Err(format!( + "{at}: path `{path}` must start with `/`, being a prefix of the request \ + path" + )); + } + if path.contains(['*', '?', '#']) { + return Err(format!( + "{at}: path `{path}` is a prefix of the request path, compared as \ + written, so it holds no pattern, query or fragment. Write `/news/` to \ + cover everything under that folder" + )); + } + } + if entry.middleware.is_empty() { + return Err(format!( + "{at} names no middleware. Remove the entry rather than leaving an empty list" + )); + } + for (position, name) in entry.middleware.iter().enumerate() { + if !crate::module_name::is_valid(name) { + return Err(format!( + "{at} names `{name}`, which is not a middleware name. A name is parts \ + joined by `.`, each in lower case letters, digits, `_` or `-`, such as \ + `cmp.example`" + )); + } + if entry.middleware[..position].contains(name) { + return Err(format!("{at} names `{name}` more than once")); + } + } + if let Some(earlier) = self.0[..index].iter().position(|earlier| { + earlier.media_type == entry.media_type + && match (&earlier.path, &entry.path) { + (None, _) => true, + (Some(covering), Some(path)) => path.starts_with(covering.as_str()), + (Some(_), None) => false, + } + }) { + return Err(format!( + "{at} is never reached, because [[{phase}]] entry {} covers every response \ + it covers and a response takes the first entry that covers it. Put the \ + entry with the longer path first", + earlier + 1 + )); + } + } + Ok(()) + } +} + +impl<'de> Deserialize<'de> for PhaseEntries { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let entries = Vec::::deserialize(deserializer).map_err(|error| { + de::Error::custom(format!( + "{error}. A phase is a list of entries, each written as [[fetch]] with \ + media_type and middleware, and path where it covers part of the site" + )) + })?; + Ok(Self(entries)) + } +} + +impl Serialize for PhaseEntries { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + self.0.serialize(serializer) + } +} + +/// The middleware one entry selects, in the order they run. +#[derive(Clone)] +pub struct MiddlewareChain { + phase: MiddlewarePhase, + middleware: Vec>, +} + +impl fmt::Debug for MiddlewareChain { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("MiddlewareChain") + .field("phase", &self.phase) + .field("middleware", &self.ids()) + .finish() + } +} + +impl MiddlewareChain { + /// A chain of middleware already in the order they run. + #[must_use] + pub fn new(phase: MiddlewarePhase, middleware: Vec>) -> Self { + Self { phase, middleware } + } + + /// The phase this chain runs in. + #[must_use] + pub fn phase(&self) -> MiddlewarePhase { + self.phase + } + + /// The middleware names, in the order they run. + #[must_use] + pub fn ids(&self) -> Vec<&'static str> { + self.middleware + .iter() + .map(|middleware| middleware.middleware_id()) + .collect() + } + + /// Whether the chain would do nothing at all. + #[must_use] + pub fn is_empty(&self) -> bool { + self.middleware.is_empty() + } + + /// Asks every middleware in turn what it will do with one document. + /// + /// A selector cannot be checked when the settings load, because a + /// middleware chooses its handlers for each document, so each is checked + /// here. The caller refuses the response when one does not parse, because + /// a change that silently does not happen is worse than a response that + /// fails. + /// + /// # Errors + /// + /// A message naming the middleware and the selector that does not parse. + pub fn plan(&self, context: &MiddlewareContext<'_>) -> Result { + let mut plan = MiddlewarePlan::default(); + for middleware in &self.middleware { + let action = middleware.create(context); + action.validate_selectors().map_err(|problem| { + format!( + "the {} middleware `{}` cannot run, because {problem}", + self.phase, + middleware.middleware_id() + ) + })?; + plan.head_inserts.extend(action.head_inserts); + plan.after_bundle_inserts + .extend(action.after_bundle_inserts); + plan.element_handlers.extend(action.element_handlers); + plan.text_handlers.extend(action.text_handlers); + plan.processors.extend(action.stream); + } + Ok(plan) + } +} + +/// What one document's chain will do, in the order it happens. +#[derive(Default)] +pub struct MiddlewarePlan { + /// Markup for the start of ``, in the order it is written. + pub head_inserts: Vec, + /// Markup for straight after the main script bundle, in the order it is + /// written. + pub after_bundle_inserts: Vec, + /// Element handlers, in the order the middleware were named. Two that + /// match one element are both asked, in this order. + pub element_handlers: Vec>, + /// Text handlers, in the order the middleware were named. + pub text_handlers: Vec>, + /// Stream processors, each handed what the one before it produced. + pub processors: Vec>, +} + +impl MiddlewarePlan { + /// Whether the chain decided to do nothing to this document. + #[must_use] + pub fn is_empty(&self) -> bool { + self.head_inserts.is_empty() + && self.after_bundle_inserts.is_empty() + && self.element_handlers.is_empty() + && self.text_handlers.is_empty() + && self.processors.is_empty() + } +} + +impl fmt::Debug for MiddlewarePlan { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("MiddlewarePlan") + .field("head_inserts", &self.head_inserts) + .field("after_bundle_inserts", &self.after_bundle_inserts) + .field( + "element_handlers", + &selectors_of_elements(&self.element_handlers), + ) + .field("text_handlers", &selectors_of_text(&self.text_handlers)) + .field("processors", &self.processors.len()) + .finish() + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs new file mode 100644 index 0000000000..e004a25cca --- /dev/null +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -0,0 +1,1103 @@ +use std::io::{self, Cursor}; +use std::rc::Rc; +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use serde_json::json; + +use super::*; +use crate::html_processor::{ + HtmlProcessorConfig, create_html_processor, create_html_processor_with_middleware, +}; +use crate::integrations::IntegrationRegistry; +use crate::integrations::registry_test_support::request_fixture; +use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; +use crate::test_support::tests::create_test_settings; + +const ORIGIN_HOST: &str = "origin.example.com"; +const REQUEST_HOST: &str = "test.example.com"; +const BUNDLE_TAG: &str = "id=\"trustedserver-js\""; + +type CreateFn = dyn Fn(&MiddlewareContext<'_>) -> MiddlewareAction + Send + Sync; + +/// A middleware made from a function, under a name of the test's choosing. +struct FromFn { + id: &'static str, + create: Box, +} + +impl Middleware for FromFn { + fn middleware_id(&self) -> &'static str { + self.id + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + (self.create)(context) + } +} + +fn middleware( + id: &'static str, + create: impl Fn(&MiddlewareContext<'_>) -> MiddlewareAction + Send + Sync + 'static, +) -> Arc { + Arc::new(FromFn { + id, + create: Box::new(create), + }) +} + +fn chain_of(middleware: Vec>) -> MiddlewareChain { + MiddlewareChain::new(MiddlewarePhase::Fetch, middleware) +} + +/// A middleware that writes `markup` at the start of ``. +fn head(id: &'static str, markup: &'static str) -> Arc { + middleware(id, move |_| MiddlewareAction { + head_inserts: vec![markup.to_owned()], + ..MiddlewareAction::pass() + }) +} + +/// A middleware with one element handler. +fn element( + id: &'static str, + selector: &'static str, + attribute: &'static str, + decide: impl Fn(&MatchedAttribute<'_>) -> AttributeRewriteAction + Send + Sync + Clone + 'static, +) -> Arc { + middleware(id, move |_| MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + selector, + attribute, + Rc::new(decide.clone()), + ))], + ..MiddlewareAction::pass() + }) +} + +struct TextFn { + selector: &'static str, + decide: F, +} + +impl TextHandler for TextFn +where + F: FnMut(&str, bool) -> ScriptRewriteAction, +{ + fn selector(&self) -> &str { + self.selector + } + + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + (self.decide)(text, is_last) + } +} + +/// A middleware with one text handler, made afresh for each document. +fn text( + id: &'static str, + selector: &'static str, + make: impl Fn() -> F + Send + Sync + 'static, +) -> Arc +where + F: FnMut(&str, bool) -> ScriptRewriteAction + 'static, +{ + middleware(id, move |_| MiddlewareAction { + text_handlers: vec![Box::new(TextFn { + selector, + decide: make(), + })], + ..MiddlewareAction::pass() + }) +} + +fn config() -> HtmlProcessorConfig { + HtmlProcessorConfig::from_settings( + &create_test_settings(), + &IntegrationRegistry::default(), + ORIGIN_HOST, + REQUEST_HOST, + "https", + ) +} + +/// `html` as a reader receives it, read from the origin `chunk_size` bytes +/// at a time. +fn page_with( + config: HtmlProcessorConfig, + chain: &MiddlewareChain, + html: &str, + chunk_size: usize, +) -> String { + let processor = create_html_processor_with_middleware(config, chain) + .expect("should plan the document's middleware"); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + processor, + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should process the document"); + String::from_utf8(output).expect("should leave the document UTF-8") +} + +fn page(chain: &MiddlewareChain, html: &str) -> String { + page_with(config(), chain, html, 8192) +} + +fn index_of(page: &str, needle: &str) -> usize { + page.find(needle) + .unwrap_or_else(|| panic!("should find `{needle}` in: {page}")) +} + +fn entry(path: Option<&str>, names: &[&str]) -> PhaseEntry { + PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + } +} + +fn refusal(entries: Vec) -> String { + PhaseEntries::new(entries) + .validate(MiddlewarePhase::Fetch) + .expect_err("should refuse the entries") +} + +// --------------------------------------------------------------------------- +// Entries +// --------------------------------------------------------------------------- + +#[test] +fn a_response_takes_the_first_entry_that_covers_it() { + let entries = PhaseEntries::new(vec![ + entry(Some("/news/"), &["example.news", "example.all"]), + entry(None, &["example.all"]), + ]); + + assert_eq!( + entries.for_response(HTML_MEDIA_TYPE, "/news/today"), + ["example.news", "example.all"], + "should take the entry for its path, which is written first" + ); + assert_eq!( + entries.for_response(HTML_MEDIA_TYPE, "/sport/today"), + ["example.all"], + "should fall to the entry covering every path" + ); +} + +#[test] +fn a_response_no_entry_covers_runs_nothing() { + let entries = PhaseEntries::new(vec![entry(Some("/news/"), &["example.news"])]); + + assert!( + entries + .for_response(HTML_MEDIA_TYPE, "/sport/today") + .is_empty(), + "should run nothing on a path no entry covers" + ); + assert!( + entries + .for_response("text/css", "/news/site.css") + .is_empty(), + "should run nothing on a media type no entry covers" + ); + assert!( + PhaseEntries::default() + .for_response(HTML_MEDIA_TYPE, "/") + .is_empty(), + "should run nothing when there are no entries" + ); +} + +#[test] +fn a_path_is_a_prefix_compared_as_written() { + let entries = PhaseEntries::new(vec![entry(Some("/news"), &["example.news"])]); + + assert_eq!( + entries.for_response(HTML_MEDIA_TYPE, "/newsletter"), + ["example.news"], + "should cover every path that begins with the prefix, which is why a folder is \ + written with its closing slash" + ); + assert!( + entries.for_response(HTML_MEDIA_TYPE, "/News").is_empty(), + "should compare the path as written, upper and lower case apart" + ); +} + +#[test] +fn names_lists_each_middleware_once_in_the_order_first_written() { + let entries = PhaseEntries::new(vec![ + entry(Some("/news/"), &["example.b", "example.a"]), + entry(None, &["example.a", "example.c"]), + ]); + + assert_eq!(entries.names(), ["example.b", "example.a", "example.c"]); +} + +#[test] +fn entries_from_the_longest_path_to_every_path_are_accepted() { + PhaseEntries::new(vec![ + entry(Some("/news/sport/"), &["example.a"]), + entry(Some("/news/"), &["example.b"]), + entry(Some("/weather/"), &["example.c"]), + entry(None, &["example.d"]), + ]) + .validate(MiddlewarePhase::Fetch) + .expect("should accept entries no earlier entry hides"); +} + +#[test] +fn an_entry_for_a_media_type_other_than_html_is_refused() { + for media_type in ["text/css", "text/html; charset=utf-8", "TEXT/HTML", "*/*"] { + let mut css = entry(None, &["example.a"]); + css.media_type = media_type.to_owned(); + + let message = refusal(vec![css]); + + assert!( + message.contains("[[fetch]] entry 1") + && message.contains(&format!("covers `{media_type}`")) + && message.contains("media_type = \"text/html\""), + "should name the entry and the media type to write: {message}" + ); + } +} + +#[test] +fn an_entry_whose_path_could_cover_nothing_is_refused() { + let message = refusal(vec![entry(Some("news/"), &["example.a"])]); + assert!( + message.contains("[[fetch]] entry 1") && message.contains("must start with `/`"), + "should refuse a path that is not a request path: {message}" + ); + + for path in ["/news/*", "/news/?page=2", "/news/#top"] { + let message = refusal(vec![entry(Some(path), &["example.a"])]); + assert!( + message.contains(&format!("path `{path}`")) && message.contains("compared as written"), + "should refuse a pattern, a query and a fragment, none of which a request path \ + holds: {message}" + ); + } +} + +#[test] +fn an_entry_that_names_nothing_is_refused() { + let message = refusal(vec![entry(None, &[])]); + + assert!( + message.contains("[[fetch]] entry 1 names no middleware"), + "should refuse an entry that would run nothing: {message}" + ); +} + +#[test] +fn an_entry_naming_something_that_is_no_name_is_refused() { + let message = refusal(vec![entry(None, &["example.a", "Not A Name"])]); + assert!( + message.contains("names `Not A Name`, which is not a middleware name"), + "should refuse a name no middleware could have: {message}" + ); + + let message = refusal(vec![entry(None, &["example.a", "example.b", "example.a"])]); + assert!( + message.contains("names `example.a` more than once"), + "should refuse a middleware named twice in one entry: {message}" + ); +} + +#[test] +fn an_entry_hidden_behind_an_earlier_one_is_refused() { + let cases = [ + // The same path twice. + vec![ + entry(Some("/news/"), &["example.a"]), + entry(Some("/news/"), &["example.b"]), + ], + // A longer path after the shorter one that covers it. + vec![ + entry(Some("/news/"), &["example.a"]), + entry(Some("/news/sport/"), &["example.b"]), + ], + // Anything after the entry covering every path. + vec![ + entry(None, &["example.a"]), + entry(Some("/news/"), &["example.b"]), + ], + vec![entry(None, &["example.a"]), entry(None, &["example.b"])], + ]; + + for entries in cases { + let message = refusal(entries); + assert!( + message.contains("[[fetch]] entry 2 is never reached") + && message.contains("[[fetch]] entry 1 covers every response it covers"), + "should name the hidden entry and the one hiding it: {message}" + ); + } +} + +#[test] +fn entries_are_read_from_a_list_and_written_back_as_they_were() { + let written = json!([ + { "media_type": "text/html", "path": "/news/", "middleware": ["example.a", "example.b"] }, + { "media_type": "text/html", "middleware": ["example.a"] }, + ]); + + let entries: PhaseEntries = + serde_json::from_value(written.clone()).expect("should read the entries"); + + assert_eq!( + entries, + PhaseEntries::new(vec![ + entry(Some("/news/"), &["example.a", "example.b"]), + entry(None, &["example.a"]), + ]) + ); + assert_eq!( + serde_json::to_value(&entries).expect("should write the entries"), + written, + "should write an entry with no path without one" + ); +} + +#[test] +fn an_entry_with_a_key_it_does_not_read_is_refused() { + let error = serde_json::from_value::(json!([ + { "media_type": "text/html", "middleware": ["example.a"], "settings": {} }, + ])) + .expect_err("should refuse a key an entry does not read"); + + assert!( + error.to_string().contains("unknown field `settings`"), + "should name the key: {error}" + ); +} + +#[test] +fn a_phase_written_as_a_table_is_told_it_is_a_list() { + let error = serde_json::from_value::(json!({ + "media_type": "text/html", + "middleware": ["example.a"], + })) + .expect_err("should refuse a phase that is not a list"); + + assert!( + error + .to_string() + .contains("A phase is a list of entries, each written as [[fetch]]"), + "should say how a phase is written: {error}" + ); +} + +// --------------------------------------------------------------------------- +// What the document pipeline does with a chain +// --------------------------------------------------------------------------- + +const PAGE: &str = "Page"; + +#[test] +fn a_chain_that_names_nothing_leaves_the_page_as_it_would_be_without_one() { + let html = r#"Page + one + + "#; + + let mut plain = create_html_processor(config()); + let without = String::from_utf8( + crate::streaming_processor::StreamProcessor::process_chunk( + &mut plain, + html.as_bytes(), + true, + ) + .expect("should process the document"), + ) + .expect("should leave the document UTF-8"); + + assert_eq!(page(&chain_of(Vec::new()), html), without); +} + +#[test] +fn head_markup_goes_ahead_of_the_bundle_in_the_order_the_middleware_are_named() { + let chain = chain_of(vec![ + head("example.first", ""), + head("example.second", ""), + ]); + + let page = page(&chain, PAGE); + + let first = index_of(&page, "name=\"first\""); + let second = index_of(&page, "name=\"second\""); + let bundle = index_of(&page, BUNDLE_TAG); + let title = index_of(&page, ""); + assert!( + first < second && second < bundle && bundle < title, + "should write head markup in the order named, ahead of the bundle and of the \ + page's own head: {page}" + ); + assert_eq!( + page.matches("name=\"first\"").count(), + 1, + "should write head markup once" + ); +} + +#[test] +fn markup_for_after_the_bundle_goes_between_the_bundle_and_the_page_s_own_head() { + let chain = chain_of(vec![middleware("example.after", |_| MiddlewareAction { + after_bundle_inserts: vec!["<script id=\"after\"></script>".to_owned()], + ..MiddlewareAction::pass() + })]); + + let page = page(&chain, PAGE); + + let bundle = index_of(&page, BUNDLE_TAG); + let after = index_of(&page, "id=\"after\""); + let title = index_of(&page, "<title>"); + assert!( + bundle < after && after < title, + "should write the markup straight after the bundle: {page}" + ); +} + +#[test] +fn a_page_with_no_head_gets_no_head_markup() { + let chain = chain_of(vec![head("example.first", "<meta name=\"first\">")]); + + let page = page(&chain, "<p>a fragment</p>"); + + assert_eq!( + page, "<p>a fragment</p>", + "should leave a document with no head as it arrived" + ); +} + +#[test] +fn an_element_handler_is_asked_about_an_address_as_it_will_be_served() { + let chain = chain_of(vec![element("example.seen", "a.x", "href", |matched| { + AttributeRewriteAction::replace(format!( + "seen:{}:{}:{}", + matched.element_name, matched.attribute_name, matched.value + )) + })]); + + let page = page( + &chain, + r#"<html><body><a class="x" href="https://origin.example.com/page">x</a><a href="https://origin.example.com/other">y</a></body></html>"#, + ); + + assert!( + page.contains(r#"<a class="x" href="seen:a:href:https://test.example.com/page">"#), + "should hand the handler the address after core moved it to the reader's host: {page}" + ); + assert!( + page.contains(r#"<a href="https://test.example.com/other">"#), + "should leave an element the selector does not match to core: {page}" + ); +} + +#[test] +fn an_element_handler_can_remove_the_element() { + let chain = chain_of(vec![element("example.remove", "a.gone", "href", |_| { + AttributeRewriteAction::remove_element() + })]); + + let page = page( + &chain, + r#"<html><body><a class="gone" href="/x">removed</a><a class="gone">no address</a><a href="/y">kept</a></body></html>"#, + ); + + assert!( + !page.contains("removed"), + "should remove the element and what it holds: {page}" + ); + assert!( + page.contains(r#"<a class="gone">no address</a>"#), + "should leave an element that carries no such attribute as it is: {page}" + ); + assert!(page.contains(r#"<a href="/y">kept</a>"#)); +} + +#[test] +fn handlers_on_one_element_are_asked_in_order_each_seeing_the_last_value() { + let chain = chain_of(vec![ + element("example.one", "a", "href", |_| { + AttributeRewriteAction::replace("/one") + }), + element("example.two", "a[href]", "href", |matched| { + AttributeRewriteAction::replace(format!("{}/two", matched.value)) + }), + ]); + + let page = page( + &chain, + r#"<html><body><a href="/start">x</a></body></html>"#, + ); + + assert!( + page.contains(r#"<a href="/one/two">"#), + "should hand the second handler the value the first left: {page}" + ); +} + +#[test] +fn an_element_an_earlier_handler_removed_is_not_asked_about_again() { + let asked = Arc::new(AtomicUsize::new(0)); + let counted = Arc::clone(&asked); + let chain = chain_of(vec![ + element("example.remove", "a", "href", |_| { + AttributeRewriteAction::remove_element() + }), + element("example.count", "a", "href", move |_| { + counted.fetch_add(1, Ordering::SeqCst); + AttributeRewriteAction::keep() + }), + ]); + + let page = page(&chain, r#"<html><body><a href="/x">x</a></body></html>"#); + + assert!(!page.contains("<a "), "should remove the element: {page}"); + assert_eq!( + asked.load(Ordering::SeqCst), + 0, + "should not ask a later handler about an element that is already gone" + ); +} + +#[test] +fn a_text_handler_is_handed_the_text_as_the_one_before_it_left_it() { + let chain = chain_of(vec![ + text("example.one", "script.x", || { + |text: &str, _is_last: bool| { + if text.contains("alpha") { + ScriptRewriteAction::replace(text.replace("alpha", "beta")) + } else { + ScriptRewriteAction::keep() + } + } + }), + text("example.two", "script", || { + |text: &str, _is_last: bool| { + if text.contains("beta") { + ScriptRewriteAction::replace(text.replace("beta", "gamma")) + } else { + ScriptRewriteAction::keep() + } + } + }), + ]); + + let page = page( + &chain, + r#"<html><body><script class="x">var a = "alpha";</script><script>var b = "alpha";</script></body></html>"#, + ); + + assert!( + page.contains(r#"<script class="x">var a = "gamma";</script>"#), + "should hand the second handler what the first wrote: {page}" + ); + assert!( + page.contains(r#"<script>var b = "alpha";</script>"#), + "should leave a script only the second matches, which holds nothing it changes: {page}" + ); +} + +#[test] +fn text_a_handler_removed_is_empty_for_the_next_which_may_write_in_its_place() { + let chain = chain_of(vec![ + text("example.remove", "script", || { + |_text: &str, _is_last: bool| ScriptRewriteAction::remove_node() + }), + text("example.write", "script", || { + |text: &str, is_last: bool| { + assert_eq!(text, "", "should hand over nothing where text was removed"); + if is_last { + ScriptRewriteAction::replace("written()") + } else { + ScriptRewriteAction::keep() + } + } + }), + ]); + + let page = page( + &chain, + "<html><body><script>removed()</script></body></html>", + ); + + assert!( + page.contains("<script>written()</script>"), + "should write the later handler's text in place of what was removed: {page}" + ); +} + +#[test] +fn a_text_handler_s_replacement_is_written_as_it_is() { + let chain = chain_of(vec![text("example.script", "script", || { + |text: &str, _is_last: bool| { + if text.is_empty() { + ScriptRewriteAction::keep() + } else { + ScriptRewriteAction::replace("if (a && b < c || d > e) { go(\"?x=1&y=2\"); }") + } + } + })]); + + let page = page(&chain, "<html><body><script>old()</script></body></html>"); + + assert!( + page.contains("<script>if (a && b < c || d > e) { go(\"?x=1&y=2\"); }</script>"), + "should write a script's text with no character escaped: {page}" + ); +} + +#[test] +fn a_handler_that_holds_a_script_writes_it_back_whole() { + // The script reaches the handler in pieces, because the document is read + // seven bytes at a time, and the word to change lies across pieces. + let chain = chain_of(vec![text("example.hold", "script", || { + let mut held = String::new(); + move |text: &str, is_last: bool| { + held.push_str(text); + if !is_last { + return ScriptRewriteAction::remove_node(); + } + ScriptRewriteAction::replace(std::mem::take(&mut held).replace("ORIGINAL", "CHANGED")) + } + })]); + + let page = page_with( + config(), + &chain, + "<html><body><script>one(); two(\"ORIGINAL\"); three();</script><p>after</p></body></html>", + 7, + ); + + assert!( + page.contains("<script>one(); two(\"CHANGED\"); three();</script><p>after</p>"), + "should write the whole script once, in order, with the word changed: {page}" + ); +} + +/// Appends a comment to the document, and swaps one word for another. +struct Rewording { + from: &'static str, + to: &'static str, + comment: &'static str, +} + +impl crate::streaming_processor::StreamProcessor for Rewording { + fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> io::Result<Vec<u8>> { + let mut text = String::from_utf8_lossy(chunk).replace(self.from, self.to); + if is_last { + text.push_str(self.comment); + } + Ok(text.into_bytes()) + } +} + +#[test] +fn a_stream_processor_is_handed_the_document_the_handlers_left() { + let chain = chain_of(vec![ + middleware("example.first", |_| MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + "a", + "href", + Rc::new(|_| AttributeRewriteAction::replace("/from-handler")), + ))], + stream: Some(Box::new(Rewording { + from: "/from-handler", + to: "/from-first", + comment: "<!--first-->", + })), + ..MiddlewareAction::pass() + }), + middleware("example.second", |_| MiddlewareAction { + stream: Some(Box::new(Rewording { + from: "/from-first", + to: "/from-second", + comment: "<!--second-->", + })), + ..MiddlewareAction::pass() + }), + ]); + + let page = page( + &chain, + r#"<html><body><a href="/start">x</a></body></html>"#, + ); + + assert!( + page.contains(r#"<a href="/from-second">"#), + "should hand each processor what the handlers and the processor before it left: {page}" + ); + assert!( + page.ends_with("<!--first--><!--second-->"), + "should run the processors in the order their middleware are named: {page}" + ); +} + +/// What one middleware leaves in the document's state for another. +struct Shared(&'static str); + +#[test] +fn the_middleware_of_one_document_share_its_state_and_see_nothing_a_request_left() { + const KEY: &str = "example"; + let chain = chain_of(vec![ + middleware("example.leaves", |context| { + context + .document_state + .get_or_insert_with(KEY, || Shared("left by the first")); + MiddlewareAction::pass() + }), + middleware("example.reads", |context| { + let shared = context + .document_state + .get::<Shared>(KEY) + .map_or("nothing", |shared| shared.0); + let request_mark = context + .document_state + .get::<request_fixture::Mark>(request_fixture::ID) + .is_some(); + MiddlewareAction { + head_inserts: vec![format!( + "<meta name=\"state\" content=\"{shared}; request mark {request_mark}\">" + )], + ..MiddlewareAction::pass() + } + }), + ]); + // The request left a mark, which the hooks of an unstored page may read. + let config = config().with_request_state(request_fixture::marked()); + + let page = page_with(config, &chain, PAGE, 8192); + + assert!( + page.contains("content=\"left by the first; request mark false\""), + "should share the document's state along the chain, with nothing of the request in \ + it: {page}" + ); +} + +#[test] +fn each_document_is_planned_afresh() { + let created = Arc::new(AtomicUsize::new(0)); + let counted = Arc::clone(&created); + let chain = chain_of(vec![middleware("example.counts", move |context| { + let nth = counted.fetch_add(1, Ordering::SeqCst) + 1; + let earlier = context.document_state.get::<Shared>("example").is_some(); + context + .document_state + .get_or_insert_with("example", || Shared("this document")); + MiddlewareAction { + head_inserts: vec![format!( + "<meta name=\"nth\" content=\"{nth}; earlier state {earlier}\">" + )], + ..MiddlewareAction::pass() + } + })]); + + let first = page(&chain, PAGE); + let second = page(&chain, PAGE); + + assert!(first.contains("content=\"1; earlier state false\"")); + assert!( + second.contains("content=\"2; earlier state false\""), + "should ask the middleware again for the next document, with a state of its own: \ + {second}" + ); + assert_eq!(created.load(Ordering::SeqCst), 2); +} + +#[test] +fn a_middleware_is_told_the_phase_the_hosts_and_how_much_it_may_hold() { + let chain = chain_of(vec![middleware("example.context", |context| { + MiddlewareAction { + head_inserts: vec![format!( + "<meta name=\"context\" content=\"{} {}://{} from {} holding {}\">", + context.phase, + context.request_scheme, + context.request_host, + context.origin_host, + context.max_buffered_script_bytes + )], + ..MiddlewareAction::pass() + } + })]); + let most = create_test_settings().publisher.max_buffered_body_bytes; + + let page = page(&chain, PAGE); + + assert!( + page.contains(&format!( + "content=\"fetch https://{REQUEST_HOST} from {ORIGIN_HOST} holding {most}\"" + )), + "should tell the middleware about the document: {page}" + ); +} + +#[test] +fn a_selector_that_does_not_parse_refuses_the_document_naming_the_middleware() { + let chains = [ + chain_of(vec![ + head("example.fine", "<meta name=\"fine\">"), + element("example.broken", "a[", "href", |_| { + AttributeRewriteAction::keep() + }), + ]), + chain_of(vec![text("example.broken", "a[", || { + |_text: &str, _is_last: bool| ScriptRewriteAction::keep() + })]), + ]; + + for chain in chains { + let message = create_html_processor_with_middleware(config(), &chain) + .err() + .expect("should refuse to plan the document"); + + assert!( + message.contains("the fetch middleware `example.broken` cannot run") + && message.contains("`a[` is not a usable selector"), + "should name the middleware and the selector: {message}" + ); + } +} + +#[test] +fn a_chain_reports_its_phase_and_the_names_in_the_order_they_run() { + let chain = chain_of(vec![ + head("example.b", "<meta name=\"b\">"), + head("example.a", "<meta name=\"a\">"), + ]); + + assert_eq!(chain.phase(), MiddlewarePhase::Fetch); + assert_eq!(chain.ids(), ["example.b", "example.a"]); + assert!(!chain.is_empty()); + assert!(chain_of(Vec::new()).is_empty()); +} + +/// A module's hooks, each of which marks what it was handed. +struct Hooks; + +impl crate::integrations::IntegrationHeadInjector for Hooks { + fn integration_id(&self) -> &'static str { + "hooks" + } + + fn head_inserts(&self, _ctx: &crate::integrations::IntegrationHtmlContext<'_>) -> Vec<String> { + vec!["<meta name=\"hook\">".to_owned()] + } +} + +impl crate::integrations::IntegrationAttributeRewriter for Hooks { + fn integration_id(&self) -> &'static str { + "hooks" + } + + fn handles_attribute(&self, attribute: &str) -> bool { + attribute == "href" + } + + fn rewrite( + &self, + _attr_name: &str, + attr_value: &str, + _ctx: &crate::integrations::IntegrationAttributeContext<'_>, + ) -> AttributeRewriteAction { + AttributeRewriteAction::replace(format!("{attr_value}/hook")) + } +} + +impl crate::integrations::IntegrationScriptRewriter for Hooks { + fn integration_id(&self) -> &'static str { + "hooks" + } + + fn selector(&self) -> &'static str { + "script" + } + + fn rewrite( + &self, + content: &str, + _ctx: &crate::integrations::IntegrationScriptContext<'_>, + ) -> ScriptRewriteAction { + if content.contains("origin()") { + ScriptRewriteAction::replace(content.replace("origin()", "hook()")) + } else { + ScriptRewriteAction::keep() + } + } +} + +#[test] +fn a_middleware_runs_after_the_hooks_of_the_same_kind() { + let chain = chain_of(vec![middleware("example.after", |_| MiddlewareAction { + head_inserts: vec!["<meta name=\"middleware\">".to_owned()], + element_handlers: vec![Box::new(AttributeRewrite::matching( + "a", + "href", + Rc::new(|matched| { + AttributeRewriteAction::replace(format!("{}/middleware", matched.value)) + }), + ))], + text_handlers: vec![Box::new(TextFn { + selector: "script", + decide: |text: &str, _is_last: bool| { + if text.contains("hook()") { + ScriptRewriteAction::replace(text.replace("hook()", "middleware()")) + } else { + ScriptRewriteAction::keep() + } + }, + })], + ..MiddlewareAction::pass() + })]); + let mut config = config(); + config.integrations = IntegrationRegistry::from_rewriters_with_head_injectors( + vec![Arc::new(Hooks)], + vec![Arc::new(Hooks)], + vec![Arc::new(Hooks)], + ); + + let page = page_with( + config, + &chain, + r#"<html><head></head><body><a href="/start">x</a><script>origin()</script></body></html>"#, + 8192, + ); + + let hook = index_of(&page, "name=\"hook\""); + let after = index_of(&page, "name=\"middleware\""); + assert!( + hook < after && after < index_of(&page, BUNDLE_TAG), + "should write a middleware's head markup after the hooks' own: {page}" + ); + assert!( + page.contains(r#"<a href="/start/hook/middleware">"#), + "should ask an element handler about the value the hooks left: {page}" + ); + assert!( + page.contains("<script>middleware()</script>"), + "should hand a text handler the script as the hooks left it: {page}" + ); +} + +// --------------------------------------------------------------------------- +// The settings +// --------------------------------------------------------------------------- + +fn settings_with_document(extra: &str) -> Result<crate::settings::Settings, String> { + let document = format!( + "{}\n{extra}", + crate::test_support::tests::crate_test_settings_str() + ); + crate::settings::Settings::from_toml(&document).map_err(|error| format!("{error:?}")) +} + +#[test] +fn fetch_entries_are_read_from_the_settings_in_the_order_written() { + let settings = settings_with_document( + r#" +[[fetch]] +media_type = "text/html" +path = "/news/" +middleware = ["example.news", "example.all"] + +[[fetch]] +media_type = "text/html" +middleware = ["example.all"] +"#, + ) + .expect("should read settings with fetch entries"); + + assert_eq!( + settings.phase_entries(MiddlewarePhase::Fetch), + &PhaseEntries::new(vec![ + entry(Some("/news/"), &["example.news", "example.all"]), + entry(None, &["example.all"]), + ]) + ); +} + +#[test] +fn settings_without_entries_write_no_fetch_key() { + let settings = settings_with_document("").expect("should read settings with no entries"); + let written = serde_json::to_value(&settings).expect("should write the settings"); + assert!( + written.get("fetch").is_none(), + "should leave the key out of a stored document that wrote no entries" + ); + + let settings = settings_with_document( + "\n[[fetch]]\nmedia_type = \"text/html\"\nmiddleware = [\"example.all\"]\n", + ) + .expect("should read settings with one entry"); + let written = serde_json::to_value(&settings).expect("should write the settings"); + assert_eq!( + written.get("fetch"), + Some(&json!([{ "media_type": "text/html", "middleware": ["example.all"] }])), + "should write the entries a document holds" + ); +} + +#[test] +fn settings_with_a_misshapen_entry_are_refused() { + let cases = [ + ( + "\n[[fetch]]\nmedia_type = \"text/css\"\nmiddleware = [\"example.all\"]\n", + "[[fetch]] entry 1 covers `text/css`", + ), + ( + "\n[fetch]\nmedia_type = \"text/html\"\nmiddleware = [\"example.all\"]\n", + "A phase is a list of entries, each written as [[fetch]]", + ), + ( + "\n[[fetch]]\nmedia_type = \"text/html\"\nmiddleware = [\"example.all\"]\nenabled = true\n", + "unknown field `enabled`", + ), + ( + "\n[[fetch]]\nmedia_type = \"text/html\"\n", + "missing field `middleware`", + ), + ]; + + for (document, expected) in cases { + let message = settings_with_document(document).expect_err("should refuse the settings"); + assert!( + message.contains(expected), + "should say `{expected}`: {message}" + ); + } +} + +#[test] +fn deploy_validation_refuses_a_misshapen_entry() { + let mut settings = create_test_settings(); + settings.fetch = PhaseEntries::new(vec![ + entry(None, &["example.all"]), + entry(Some("/news/"), &["example.news"]), + ]); + + let error = crate::config::validate_settings_for_deploy(&settings) + .expect_err("should refuse entries with one that is never reached"); + + assert!( + error + .to_string() + .contains("[[fetch]] entry 2 is never reached"), + "should name the entry: {error}" + ); +} diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 576183723d..b3bb60f9af 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -630,6 +630,8 @@ struct ProcessResponseParams<'a> { origin_url: &'a str, request_host: &'a str, request_scheme: &'a str, + /// See [`OwnedProcessResponseParams::request_path`]. + request_path: &'a str, settings: &'a Settings, content_type: &'a str, integration_registry: &'a IntegrationRegistry, @@ -676,6 +678,7 @@ impl PublisherBodyProcessor { origin_host: ¶ms.origin_host, request_host: ¶ms.request_host, request_scheme: ¶ms.request_scheme, + request_path: ¶ms.request_path, settings, integration_registry, permissions_script: permissions_script_for(params, settings), @@ -760,6 +763,7 @@ fn process_response_streaming<W: Write>( origin_host: params.origin_host, request_host: params.request_host, request_scheme: params.request_scheme, + request_path: params.request_path, settings: params.settings, integration_registry: params.integration_registry, permissions_script: params.permissions_script.map(str::to_string), @@ -1317,6 +1321,8 @@ struct HtmlStreamProcessorParams<'a> { origin_host: &'a str, request_host: &'a str, request_scheme: &'a str, + /// See [`OwnedProcessResponseParams::request_path`]. + request_path: &'a str, settings: &'a Settings, integration_registry: &'a IntegrationRegistry, /// Head script carrying this request's permission state, or [`None`] under a @@ -1513,7 +1519,8 @@ fn deferred_inline_seam_token( fn create_html_stream_processor( params: HtmlStreamProcessorParams<'_>, ) -> Result<impl StreamProcessor + use<>, Report<TrustedServerError>> { - use crate::html_processor::{HtmlProcessorConfig, create_html_processor}; + use crate::html_processor::{HtmlProcessorConfig, create_html_processor_with_middleware}; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; let config = HtmlProcessorConfig::from_settings( params.settings, @@ -1547,7 +1554,16 @@ fn create_html_stream_processor( .with_body_close(body_close) .with_csp_nonce_observer(csp_nonce_observed); - Ok(create_html_processor(config)) + // The middleware of the first [[fetch]] entry covering this path. What + // they leave is what a shared template stores. + let chain = params.integration_registry.middleware_chain( + ¶ms.settings.fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + params.request_path, + ); + create_html_processor_with_middleware(config, &chain) + .map_err(|message| Report::new(TrustedServerError::Configuration { message })) } /// Result of publisher request handling, indicating whether the response body @@ -1731,6 +1747,9 @@ pub struct OwnedProcessResponseParams { pub(crate) origin_url: String, pub(crate) request_host: String, pub(crate) request_scheme: String, + /// The path the reader asked for, which decides the `[[fetch]]` entry + /// that covers the document. + pub(crate) request_path: String, pub(crate) content_type: String, pub(crate) ad_slots_script: Option<String>, pub(crate) ad_bids_state: AdBidsState, @@ -2182,6 +2201,8 @@ fn build_template_assembly_params( origin_url: settings.publisher.origin_url.clone(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), + // Set by the caller, with the rest of what belongs to the request. + request_path: String::new(), content_type: entry.metadata.content_type.clone(), // The template already carries the head seam; re-injecting would duplicate it. permissions_json, @@ -2951,6 +2972,7 @@ pub fn stream_publisher_body<W: Write>( origin_url: ¶ms.origin_url, request_host: ¶ms.request_host, request_scheme: ¶ms.request_scheme, + request_path: ¶ms.request_path, settings, content_type: ¶ms.content_type, integration_registry, @@ -3057,6 +3079,7 @@ pub async fn stream_publisher_body_async<W: Write>( origin_host: ¶ms.origin_host, request_host: ¶ms.request_host, request_scheme: ¶ms.request_scheme, + request_path: ¶ms.request_path, settings, integration_registry, permissions_script: permissions_script_for(params, settings), @@ -5198,6 +5221,7 @@ pub async fn handle_publisher_request( ad_bids_state.clone(), permissions_json.clone(), ); + params.request_path.clone_from(&request_path); params.seam_ad_slots = seam_ad_slots.clone(); params.dispatched_auction = dispatched_auction.take(); params.auction_observation = auction_observation.take(); @@ -5571,6 +5595,7 @@ pub async fn handle_publisher_request( origin_url: settings.publisher.origin_url.clone(), request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), + request_path: request_path.clone(), content_type, permissions_json, ad_slots_script: ad_slots_script.clone(), @@ -9215,6 +9240,7 @@ mod tests { origin_url: settings.publisher.origin_url.clone(), request_host: settings.publisher.domain.clone(), request_scheme: "https".to_owned(), + request_path: "/".to_owned(), content_type: "application/json".to_owned(), permissions_json: String::new(), ad_slots_script: None, @@ -9832,6 +9858,50 @@ mod tests { settings } + #[test] + fn the_fetch_entries_change_the_fingerprint() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + // What a fetch middleware writes is stored with the template, so + // which run, in what order and on which paths all select one. + let fingerprint_with = |entries: &[(Option<&str>, &[&str])]| { + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.fetch = PhaseEntries::new( + entries + .iter() + .map(|(path, names)| PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + }) + .collect(), + ); + fingerprint(&settings) + }; + + let fingerprints = [ + fingerprint_with(&[]), + fingerprint_with(&[(None, &[fixture::HEAD])]), + fingerprint_with(&[(None, &[fixture::HEAD, fixture::LINKS])]), + fingerprint_with(&[(None, &[fixture::LINKS, fixture::HEAD])]), + fingerprint_with(&[(Some("/news/"), &[fixture::HEAD])]), + ]; + + for (index, fingerprint) in fingerprints.iter().enumerate() { + assert!( + !fingerprints[..index].contains(fingerprint), + "entries {index} must select a template of their own" + ); + } + assert_eq!( + fingerprint_with(&[(None, &[fixture::HEAD])]), + fingerprints[1], + "the same entries must select the same template" + ); + } + #[test] fn disabling_an_integration_changes_the_fingerprint() { // The fingerprint was `concatenated_hash(all_module_ids())` — every module @@ -11048,6 +11118,145 @@ mod tests { ); } + /// [`settings_with_mode`], with core's middleware stand-in selected + /// under `label` and one `[[fetch]]` entry naming `names` for `path`. + fn settings_with_fetch_entry( + mode: &str, + label: &str, + path: Option<&str>, + names: &[&str], + ) -> Settings { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_mode(mode); + settings + .insert_module_config( + "testing", + fixture::MODULE, + &serde_json::json!({ "label": label }), + ) + .expect("should select the middleware stand-in"); + settings.fetch = + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + }]); + settings + } + + /// [`navigation_request`], for another path of the same site. + fn navigation_request_for(path: &str) -> Request<EdgeBody> { + let mut request = navigation_request(); + *request.uri_mut() = format!("https://ts.example.com{path}") + .parse() + .expect("should build the address"); + request + } + + async fn page_of(response: Response<EdgeBody>) -> String { + String::from_utf8(body_of(response).await).expect("should serve a UTF-8 page") + } + + #[tokio::test] + async fn a_fetch_middleware_changes_a_page_under_the_path_its_entry_covers() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_fetch_entry( + "inline", + label, + Some("/article"), + &[fixture::HEAD], + )); + let marker = fixture::head_marker(label); + + queue_shareable_html(&stub); + let covered = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + assert_eq!( + covered.matches(&marker).count(), + 1, + "should run the entry's middleware once on a {label} page under its path, \ + with the module's own setting: {covered}" + ); + + queue_shareable_html(&stub); + let elsewhere = page_of( + run_via( + &settings, + &services, + navigation_request_for("/elsewhere"), + finalizer, + ) + .await, + ) + .await; + assert!( + elsewhere.contains("origin") && !elsewhere.contains(&marker), + "should run nothing on a {label} page no entry covers: {elsewhere}" + ); + } + } + + #[tokio::test] + async fn what_a_fetch_middleware_writes_is_stored_with_the_shared_template() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_fetch_entry( + "esi", + label, + None, + &[fixture::HEAD], + )); + let marker = fixture::head_marker(label); + // One answer from the origin, so the second reader can only + // be served from the store. + queue_shareable_html(&stub); + + let first = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + let second = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + + assert_eq!( + stub.recorded_cache_intents().len(), + 1, + "should serve the second {label} reader from the stored template" + ); + for (reader, page) in [("first", &first), ("second", &second)] { + assert_eq!( + page.matches(&marker).count(), + 1, + "should serve the {reader} {label} reader what the middleware wrote, \ + once: {page}" + ); + } + } + } + /// Drive `handle_publisher_request` through the **EC-preload** fetch path. /// /// `run_with_orchestrator` passes `kv: None` and no EC id, so @@ -20294,6 +20503,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20328,6 +20538,169 @@ mod tests { ); } + /// A selector is chosen for each document, so one that does not parse + /// cannot be refused when the settings load. The response is refused + /// instead of the parser panicking. + #[test] + fn a_middleware_whose_selector_does_not_parse_refuses_the_response() { + use crate::error::IntoHttpResponse as _; + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::BROKEN.to_owned()], + }]); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/".to_owned(), + content_type: "text/html; charset=utf-8".to_string(), + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }; + + let mut output = Vec::new(); + let error = stream_publisher_body( + EdgeBody::from(b"<html><head></head><body><a href=\"/x\">x</a></body></html>".to_vec()), + &mut output, + ¶ms, + &settings, + ®istry, + ) + .expect_err("should refuse a page whose middleware cannot run"); + + let message = error.to_string(); + assert!( + message.contains(&format!( + "the fetch middleware `{}` cannot run", + fixture::BROKEN + )) && message.contains(&format!( + "`{}` is not a usable selector", + fixture::BROKEN_SELECTOR + )), + "should name the middleware and the selector: {message}" + ); + assert_eq!( + error.current_context().status_code(), + StatusCode::INTERNAL_SERVER_ERROR, + "should answer as a fault of the deployment's own" + ); + assert!(output.is_empty(), "should write nothing of the page"); + } + + /// The same refusal through each finalizer, so an adapter answers with an + /// error where it would otherwise send the headers of a page it cannot + /// deliver. + #[tokio::test] + async fn a_middleware_whose_selector_does_not_parse_fails_the_response_through_each_finalizer() + { + use crate::error::IntoHttpResponse as _; + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::BROKEN.to_owned()], + }]); + let settings = Arc::new(settings); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + )); + let services = noop_services(); + let page = || PublisherResponse::Stream { + response: Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build the response"), + body: EdgeBody::from( + b"<html><head></head><body><a href=\"/x\">x</a></body></html>".to_vec(), + ), + params: Box::new(OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/".to_owned(), + content_type: "text/html; charset=utf-8".to_string(), + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }), + }; + + let streamed = publisher_response_into_streaming_response( + page(), + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + services.clone(), + ) + .await; + let buffered = buffer_publisher_response_async( + page(), + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await; + + for (finalizer, result) in [("streaming", streamed), ("buffered", buffered)] { + let error = result + .err() + .unwrap_or_else(|| panic!("should fail the response in the {finalizer} finalizer")); + assert!( + error.to_string().contains(&format!( + "the fetch middleware `{}` cannot run", + fixture::BROKEN + )), + "should name the middleware in the {finalizer} finalizer: {error}" + ); + assert_eq!( + error.current_context().status_code(), + StatusCode::INTERNAL_SERVER_ERROR, + "should answer as a fault of the deployment's own in the {finalizer} finalizer" + ); + } + } + /// Empty origin body on the streaming route must produce no output /// without erroring. Exercises the `Ok(0)` branch of `process_chunks` /// plus the processor's `is_last=true, chunk=[]` terminal call. @@ -20353,6 +20726,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20401,6 +20775,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20529,6 +20904,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20595,6 +20971,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20664,6 +21041,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20733,6 +21111,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20802,6 +21181,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -20851,6 +21231,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21076,6 +21457,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -21153,6 +21535,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -21232,6 +21615,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21305,6 +21689,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/css".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -21460,6 +21845,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -22132,6 +22518,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -22329,6 +22716,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -22410,6 +22798,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "Text/HTML; Charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: Some( @@ -22474,6 +22863,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -22586,6 +22976,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html; charset=utf-8".to_string(), permissions_json: String::new(), ad_slots_script: None, @@ -22647,6 +23038,7 @@ mod tests { origin_url: "https://origin.example.com".to_string(), request_host: "proxy.example.com".to_string(), request_scheme: "https".to_string(), + request_path: "/".to_owned(), content_type: "text/html".to_string(), permissions_json: String::new(), ad_slots_script: None, diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index a6484bf933..5a500e3801 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -26,6 +26,7 @@ use crate::ec::module::{ }; use crate::error::TrustedServerError; use crate::host_header::validate_host_header_override_value; +use crate::middleware::{MiddlewarePhase, PhaseEntries}; use crate::platform::PlatformImageOptimizerRegion; use crate::provider_table::{ProviderChoice, ProviderList, SectionModules}; use crate::redacted::Redacted; @@ -3917,6 +3918,10 @@ pub struct Settings { skip_serializing_if = "crate::inspect::config::InspectConfig::is_default" )] pub inspect: crate::inspect::config::InspectConfig, + /// The page changes run on a document fetched from the origin, written + /// as `[[fetch]]` entries. See [`crate::middleware`]. + #[serde(default, skip_serializing_if = "PhaseEntries::is_empty")] + pub fetch: PhaseEntries, /// Where the loader wrote secrets, which the configuration view masks. /// Never read from a document and never written to one. #[serde(skip)] @@ -4033,6 +4038,7 @@ impl Settings { settings.ec.validate_module_selection()?; settings.ec.validate_resolve_allowed_origins()?; settings.validate_module_sections()?; + settings.validate_phase_entries()?; settings.device.validate_module_selection()?; settings.geo.validate_module_selection()?; GeoConfig::validate_permission_policy()?; @@ -4556,6 +4562,31 @@ impl Settings { Ok(()) } + /// The entries of `phase`, in the order written. + #[must_use] + pub fn phase_entries(&self, phase: MiddlewarePhase) -> &PhaseEntries { + match phase { + MiddlewarePhase::Fetch => &self.fetch, + } + } + + /// Checks the shape of each phase's entries, see + /// [`PhaseEntries::validate`]. Whether a name is one a running module + /// supplies is only knowable where the registry is built, so it is + /// checked there. + /// + /// # Errors + /// + /// Naming the entry at fault. + pub fn validate_phase_entries(&self) -> Result<(), Report<TrustedServerError>> { + for phase in MiddlewarePhase::ALL { + self.phase_entries(phase) + .validate(phase) + .map_err(|message| Report::new(TrustedServerError::Configuration { message }))?; + } + Ok(()) + } + /// Checks every section's selection and tables, and that no module is /// selected in two sections. /// diff --git a/crates/trusted-server-integration-tests/README.md b/crates/trusted-server-integration-tests/README.md index 2d3981721e..8ca3655a32 100644 --- a/crates/trusted-server-integration-tests/README.md +++ b/crates/trusted-server-integration-tests/README.md @@ -7,7 +7,7 @@ end-to-end publisher behavior. It is not linked into an adapter artifact. - `tests/parity.rs` calls Axum, Cloudflare, and Spin routers in process and compares their shared route behavior. -- `tests/documentation_snippets.rs` extracts the checked integration-guide +- `tests/documentation_snippets.rs` extracts each checked integration-guide fixture and compiles it in an isolated offline crate. - `tests/integration.rs` exercises the Fastly/Viceroy and Axum paths against the WordPress and Next.js fixture containers. diff --git a/crates/trusted-server-integration-tests/tests/documentation_snippets.rs b/crates/trusted-server-integration-tests/tests/documentation_snippets.rs index 0e56dbc119..b215d47388 100644 --- a/crates/trusted-server-integration-tests/tests/documentation_snippets.rs +++ b/crates/trusted-server-integration-tests/tests/documentation_snippets.rs @@ -3,15 +3,30 @@ use std::process::Command; use std::time::{SystemTime, UNIX_EPOCH}; const GUIDE: &str = include_str!("../../../docs/guide/integration-guide.md"); -const START: &str = "<!-- documentation-snippet:runtime-services:start -->"; -const END: &str = "<!-- documentation-snippet:runtime-services:end -->"; #[test] fn integration_guide_runtime_services_fixture_compiles() { + assert_documented_fixture_compiles("runtime-services"); +} + +#[test] +fn integration_guide_middleware_fixture_compiles() { + assert_documented_fixture_compiles("middleware"); +} + +/// Compiles the fence the guide marks as the snippet `name`, exactly as +/// written, as an isolated crate that depends on core alone. +#[allow( + clippy::panic, + reason = "a helper of the tests above, which fails the test that called it" +)] +fn assert_documented_fixture_compiles(name: &str) { + let start = format!("<!-- documentation-snippet:{name}:start -->"); + let end = format!("<!-- documentation-snippet:{name}:end -->"); let marked = GUIDE - .split_once(START) - .and_then(|(_, tail)| tail.split_once(END).map(|(body, _)| body)) - .expect("should contain one bounded runtime-services snippet"); + .split_once(&start) + .and_then(|(_, tail)| tail.split_once(&end).map(|(body, _)| body)) + .unwrap_or_else(|| panic!("should contain one bounded {name} snippet")); let source = marked .trim() .strip_prefix("```rust\n") @@ -23,7 +38,7 @@ fn integration_guide_runtime_services_fixture_compiles() { .expect("should have a current system clock") .as_nanos(); let fixture_root = std::env::temp_dir().join(format!( - "trusted-server-documentation-snippet-{}-{nonce}", + "trusted-server-documentation-snippet-{name}-{}-{nonce}", std::process::id() )); let source_dir = fixture_root.join("src"); @@ -82,7 +97,7 @@ fn integration_guide_runtime_services_fixture_compiles() { assert!( output.status.success(), - "documented integration fixture must compile:\nstdout:\n{}\nstderr:\n{}", + "the documented {name} fixture must compile:\nstdout:\n{}\nstderr:\n{}", String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) ); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 456e7fea93..e32cb8de4e 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -228,6 +228,7 @@ fail and the service will return its startup-error response. | `[demand]` | several modules | The auction's demand sources | | `[device]` | one module | Device classification | | `[ec]` | one module | Edge Cookie identity, persistence, and partner sync | +| `[[fetch]]` | nothing | Which page changes run on which pages, and in what order | | `[geo]` | one module | Which module resolves location, if any | | `[[handlers]]` | nothing | Ordered HTTP Basic-auth rules | | `[image_optimizer]` | nothing | Reusable Fastly Image Optimizer profiles | @@ -1724,6 +1725,53 @@ not page integrations, so a section that names one refuses startup like any name no module supplies. Their settings live in `[demand.<name>]` and `[ad-server.<name>]`. +### Placing page changes + +A module can supply a change it makes to a page as a middleware, and a +middleware runs only on the pages where an entry names it. Selecting the +module makes its middleware available, and the `[[fetch]]` entries say which +pages each one runs on and in what order. + +| Field | Type | Required | Description | +| ------------ | ---------------- | -------- | -------------------------------------------------------------------------------------- | +| `media_type` | String | Yes | The media type the entry covers. Only `text/html` is accepted | +| `path` | String | No | A prefix of the request path, compared as written. Absent, the entry covers every path | +| `middleware` | Array of strings | Yes | The middleware to run, in order, each handed the page as the one before it left it | + +A page takes the first entry that covers it and runs that entry's middleware +alone. Entries are therefore written from the longest path to the entry with +no path, and a folder is written with its closing slash, because `/news` also +begins `/newsletter`. + +```toml +[[fetch]] +media_type = "text/html" +path = "/news/" +middleware = ["tag.example.cleanup", "tag.example"] + +[[fetch]] +media_type = "text/html" +middleware = ["tag.example"] +``` + +A fetch middleware runs on the page as the origin sent it. It is told nothing +about the reader, because what it leaves is what a +[shared template](#shared-template-assembly-assembly-mode-esi) stores for +every reader, and the entries are part of what selects a stored template, so +changing them never serves a page built under the old ones. + +A middleware takes its settings from its module's own `[<section>.<name>]` +table. An entry carries none and switches nothing on. + +An entry that could not do what it says refuses the configuration, both when +a deployment is validated and when the settings load. That is a media type +other than `text/html`, a path that does not start with `/` or that holds a +`*`, a `?` or a `#`, an entry that names no middleware or names one twice, a +key an entry does not read, and an entry that an earlier one already covers. +An entry naming a middleware that no running module supplies refuses startup, +and the message lists the names that could be written. A middleware that no +entry names changes no page, and startup logs a warning that names it. + The sections below give each module's settings, and the integration guides describe what each one does. @@ -3229,6 +3277,11 @@ from the file, and startup checks the rest and runs the first set again. - Each integration validates its own block, selected or not, so a typo in a block that is switched off is still caught +**Page changes**: + +- Every `[[fetch]]` entry covers `text/html`, has a path no earlier entry + already covers, and names at least one middleware, each once + ### Checked when the service starts Everything above runs again on the loaded configuration, and these join it: @@ -3237,6 +3290,9 @@ Everything above runs again on the loaded configuration, and these join it: this build does not have, a missing settings table, or a table the selector does not name, stops the service on its next start. A passing `ts config validate` is not proof that a change to those four will start +- The middleware each `[[fetch]]` entry names. A name no running module + supplies stops the service on its next start, because which middleware a + build has is only known where its modules are registered - Resolved secret values, so a passphrase shorter than 32 bytes, a placeholder or a weak handler password fails here - The compiled `permissions.yaml` policy, and the `assume_single_jurisdiction` diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index 038324ae01..aefdc74a94 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -17,6 +17,8 @@ Axum, or Spin SDK types. - `IntegrationHtmlPostProcessor` is for bounded whole-document work that cannot be performed during streaming. - `IntegrationRequestFilter` makes an early request decision. +- `Middleware` changes a page, on the pages where the operator's settings + place it. See [Changing a page](#changing-a-page). Build one `IntegrationRegistration` with only the hooks the feature needs. Use `with_deferred_js()` only for a separately loaded integration module and @@ -306,6 +308,7 @@ builder every integration uses. | `.with_script_rewriter(...)` | Rewrites inline script contents | | `.with_html_stream_processor(...)` | Works on the document as it streams | | `.with_request_filter(...)` | Inspects a request and can turn it back before it reaches the origin | +| `.with_middleware(...)` | Offers a page change that a `[[fetch]]` entry may place by the middleware's name | | `.with_js_module(CarriedJsModule { source, sha256 })` | Carries the integration's own browser script, built outside `trusted-server-js` | | `.with_deferred_js()` | Serves the script as its own `<script defer>` tag instead of in the main bundle | | `.with_standalone_js()` | Serves the script only on its own path, for an integration that injects its own tag | @@ -342,6 +345,115 @@ startup. The message lists the modules of that type the deployment runs, and says when the name is a module no section selects or one that supplies no module of that type. +### Changing a page + +A middleware is a page change that the operator's settings place. The module +registers each one with `.with_middleware(...)`, and it runs only on the +pages where an entry names it, in the order that entry gives. See +[Placing page changes](/guide/configuration#placing-page-changes). + +The fixture below loads a vendor's tag from a first-party path and marks the +head. The documentation test extracts this exact fence and compiles it as an +isolated crate. + +<!-- documentation-snippet:middleware:start --> + +```rust +use std::rc::Rc; +use std::sync::Arc; + +use trusted_server_core::integrations::{AttributeRewriteAction, IntegrationRegistration}; +use trusted_server_core::middleware::{ + AttributeRewrite, Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, +}; + +/// The tag the vendor serves, and the first-party path it is loaded from +/// instead. +const VENDOR_TAG: &str = "https://cdn.vendor.example/tag.js"; +const FIRST_PARTY_TAG: &str = "/integrations/example_tag/tag.js"; + +/// Loads the vendor's tag from a first-party path, and says so in the head. +pub struct ExampleTag; + +impl Middleware for ExampleTag { + fn middleware_id(&self) -> &'static str { + "tag.example" + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + head_inserts: vec!["<meta name=\"example-tag\" content=\"first-party\">".to_owned()], + element_handlers: vec![Box::new(AttributeRewrite::matching( + "script[src]", + "src", + Rc::new(|matched| { + if matched.value == VENDOR_TAG { + AttributeRewriteAction::replace(FIRST_PARTY_TAG) + } else { + AttributeRewriteAction::keep() + } + }), + ))], + ..MiddlewareAction::pass() + } + } +} + +/// What the module's build function returns. +pub fn registration() -> IntegrationRegistration { + IntegrationRegistration::builder("example_tag") + .without_js() + .with_middleware(Arc::new(ExampleTag)) + .build() +} +``` + +<!-- documentation-snippet:middleware:end --> + +A middleware is asked once for each document, by `create`, and what it +answers is used for that document alone, so a handler may hold state between +the chunks of one script. The answer is a `MiddlewareAction`, which may set +any of five things. + +| Field | What it does | +| ---------------------- | -------------------------------------------------------------------------------------------------------- | +| `head_inserts` | Writes markup at the start of `<head>`, ahead of the script bundle | +| `after_bundle_inserts` | Writes markup straight after the script bundle, for a script that needs the bundle to have run | +| `element_handlers` | For the elements a CSS selector matches, keeps the element, replaces one attribute's value or removes it | +| `text_handlers` | For the text inside the elements a CSS selector matches, keeps a chunk, replaces it or removes it | +| `stream` | Runs a stream processor over the document the handlers left | + +A middleware can rely on these rules. + +- The middleware of one entry run in the order the entry names them. Two + handlers that match one element, or one script, are both asked, and each is + handed what the one before it left. +- An element handler is asked after core has moved the origin's address in + the attribute to the reader's host, and an element an earlier handler + removed is not asked about again. +- A script's text arrives in chunks. A handler that needs the whole script + removes each chunk while keeping a copy, and writes back what it kept with + the last chunk. A replacement is written as it is, with no character + escaped. +- A fetch middleware is told nothing about the reader, because what it leaves + is what a shared template stores. `MiddlewareContext::document_state` is + shared by the middleware working on one document, and holds nothing a + request left. +- A selector that does not parse fails the response with a `500` and a + message naming the middleware. A middleware chooses its selectors for each + document, so they cannot be checked when the service starts. + +A middleware's name is the module's own name, as `module_name!()` gives it, +with a part of its own after it when the module supplies several, such as +`tag.example` and `tag.example.cleanup`. Startup refuses two registrations +that supply one name, and a name an entry could not write. A middleware reads +its settings from the module's own table, which the build function is +handed. + ### What a module is handed Core builds one module context for a request and hands each module it calls From 4600b1cdd6c96de8540e5d8a53617833daf97919 Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 9 Oct 2026 02:27:11 +0100 Subject: [PATCH 05/36] A page change can run for each reader, as a serve middleware The commit before this one places a middleware on the page as the origin sent it, where what it writes is stored with a shared template. Some page changes belong to one reader. This adds the second phase. A `[[serve]]` entry names the middleware that run on each reader's copy, whether the page came from the store or from the origin, and nothing they write is stored. [[serve]] media_type = "text/html" middleware = ["tag.example.reader"] A middleware says which phase it runs in. Startup refuses an entry that names one in the other phase, and the message says which phase it runs in. A serve middleware is handed what its module's request hooks left on the reader's request. A request that carries any was already kept off the shared template path, so a page from the store has none to hand over. A page leaves from four places, and the serve entry runs at each. 1. A page that is not stored runs it on the same pass as the fetch entry, after it, so a streamed page is still streamed. 2. A page transformed as a shared template runs it once the template is stored and this reader's copy is assembled. 3. A page from the store runs it after assembly in the buffered finalizer. 4. A page from the store runs it piece by piece in the streaming finalizer, so the head still goes out before the auction is awaited. A serve middleware works on a page core has already rewritten, and nothing of core's own is written twice. Its head markup goes straight before the script bundle's tag, after the fetch entry's, and its markup for after the bundle goes straight after the tag. A page from the store keeps no record of its origin's host. A serve middleware is told the host from the settings, so it is told the same on a hit as on a miss. A serve entry that cannot run fails the response with a 500 at each of the four places, before a byte of the page is sent. The seam probe supplies a serve middleware from outside core, and the Axum adapter's probe tests cover a `[[serve]]` entry at startup, one that names a serve middleware and one that names a fetch middleware. Tests drive a serve middleware through a whole request at each of the four places and through both finalizers, with an auction and without, and read the stored template to see it holds what the fetch entry wrote and nothing the serve entry wrote. Each of the four places was then broken in turn, and tests failed for each. Tests also failed with the host read from the page's own record, and with the serve entry run inside the transform of a page that is stored. The guides describe both phases. --- crates/testing/seam-probe/src/lib.rs | 84 ++- .../tests/seam_probe.rs | 38 ++ .../trusted-server-core/src/html_processor.rs | 275 +++++++-- .../src/integrations/registry.rs | 159 ++++- crates/trusted-server-core/src/middleware.rs | 49 +- .../src/middleware/tests.rs | 362 ++++++++++- crates/trusted-server-core/src/publisher.rs | 568 +++++++++++++++++- crates/trusted-server-core/src/settings.rs | 5 + docs/guide/configuration.md | 43 +- docs/guide/integration-guide.md | 31 +- 10 files changed, 1495 insertions(+), 119 deletions(-) diff --git a/crates/testing/seam-probe/src/lib.rs b/crates/testing/seam-probe/src/lib.rs index 211c2e56e2..d00258f05d 100644 --- a/crates/testing/seam-probe/src/lib.rs +++ b/crates/testing/seam-probe/src/lib.rs @@ -2,8 +2,8 @@ //! part of the integration seam from a vendor crate's position. //! //! One registration carries a browser module, a proxy route, a geo module, -//! an Edge Cookie identity module, a device module and a middleware that -//! changes a page, alongside its own configuration block, and the builder +//! an Edge Cookie identity module, a device module and a middleware for +//! each phase of a page, alongside its own configuration block, and the builder //! adds a request preparer, a demand implementation `[demand]` can name and //! an ad server implementation `[ad-server]` can name. The round-trip //! tests in `crates/trusted-server-adapter-axum/tests/seam_probe.rs` and @@ -190,6 +190,41 @@ impl Middleware for SeamProbeMiddleware { } } +/// The name an entry gives the probe's serve middleware, which is the +/// module's name with a part of its own after it. +#[must_use] +pub fn reader_middleware_name() -> &'static str { + trusted_server_core::module_name!("reader") +} + +/// What the probe's serve middleware writes ahead of the script bundle for a +/// reader who asked for `request_host`. +#[must_use] +pub fn seam_probe_reader_marker(request_host: &str) -> String { + format!("<meta name=\"seam-probe-reader\" content=\"{request_host}\">") +} + +/// The probe's change to one reader's copy of a page, which marks the head +/// with the host that reader asked for. +pub struct SeamProbeReaderMiddleware; + +impl Middleware for SeamProbeReaderMiddleware { + fn middleware_id(&self) -> &'static str { + reader_middleware_name() + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] + } + + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + head_inserts: vec![seam_probe_reader_marker(context.request_host)], + ..MiddlewareAction::pass() + } + } +} + /// Holds a script's text until its last chunk arrives, then writes the whole /// of it back with the word swapped, because the word may lie across chunks. struct CountryInScript { @@ -597,7 +632,8 @@ pub fn register( }) .with_middleware(Arc::new(SeamProbeMiddleware { country: config.country.clone(), - })); + })) + .with_middleware(Arc::new(SeamProbeReaderMiddleware)); if config.declares_geo { registration = registration .with_geo_module(module_name(), Arc::new(SeamProbeGeo::new(config.country))); @@ -876,6 +912,46 @@ mod tests { } } + #[test] + fn the_serve_middleware_marks_a_reader_s_copy_where_an_entry_names_it() { + use trusted_server_core::html_processor::test_support::REQUEST_HOST; + + let settings = settings_with_probe( + r#"country = "NZ" + + [[fetch]] + media_type = "text/html" + middleware = ["testing.seam-probe"] + + [[serve]] + media_type = "text/html" + middleware = ["testing.seam-probe.reader"] + "#, + ); + + let page = probe_page(&settings, 8192); + + let country = page + .find(&seam_probe_head_marker("NZ")) + .expect("should mark the head with the country"); + let reader = page + .find(&seam_probe_reader_marker(REQUEST_HOST)) + .expect("should mark the head with the host the reader asked for"); + let bundle = page + .find("id=\"trustedserver-js\"") + .expect("should carry the script bundle"); + assert!( + country < reader && reader < bundle, + "should write the serve middleware's mark after the fetch one's and ahead of the \ + bundle: {page}" + ); + assert_eq!( + reader_middleware_name(), + "testing.seam-probe.reader", + "should name the serve middleware after the module" + ); + } + #[test] fn the_middleware_changes_nothing_where_no_entry_names_it() { use trusted_server_core::html_processor::test_support::REQUEST_HOST; @@ -883,7 +959,7 @@ mod tests { let page = probe_page(&settings_with_probe(r#"country = "NZ""#), 8192); assert!( - !page.contains("seam-probe-country"), + !page.contains("seam-probe-country") && !page.contains("seam-probe-reader"), "should not mark the head of a page no entry places it on: {page}" ); assert!( diff --git a/crates/trusted-server-adapter-axum/tests/seam_probe.rs b/crates/trusted-server-adapter-axum/tests/seam_probe.rs index 0bef24a2ae..7b4de0c8e9 100644 --- a/crates/trusted-server-adapter-axum/tests/seam_probe.rs +++ b/crates/trusted-server-adapter-axum/tests/seam_probe.rs @@ -422,6 +422,44 @@ fn fetch_entry_naming_a_modules_middleware_starts_the_adapter() { .expect("should start with an entry naming a middleware the module supplies"); } +/// A `[[serve]]` entry starts the adapter when it names a middleware that +/// runs in that phase, and is a startup error when it names one that does +/// not. +#[test] +fn serve_entry_starts_the_adapter_only_with_a_middleware_of_that_phase() { + let settings = |name: &str| { + settings_with(&format!( + r#" + {PROBE_BLOCK} + + [[serve]] + media_type = "text/html" + middleware = ["{name}"] + "# + )) + }; + + TrustedServerApp::routes_with_registrations( + settings(seam_probe::reader_middleware_name()), + &[seam_probe::builder()], + ) + .expect("should start with a serve entry naming the module's serve middleware"); + + let error = TrustedServerApp::routes_with_registrations( + settings(seam_probe::module_name()), + &[seam_probe::builder()], + ) + .err() + .expect("should refuse to start when a serve entry names a fetch middleware"); + + let message = error.to_string(); + assert!( + message.contains("[[serve]] entry 1 names `testing.seam-probe`") + && message.contains("does not run in that phase. It runs in [[fetch]]"), + "should say which phase the middleware runs in: {message}" + ); +} + /// A `[[fetch]]` entry naming the middleware of a module no section selects /// is a startup error, raised where the adapter builds its routes. #[test] diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index b99808c76f..3297a90d40 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -18,7 +18,10 @@ use crate::integrations::{ IntegrationDocumentState, IntegrationHtmlContext, IntegrationRegistry, IntegrationRequestState, IntegrationScriptContext, ScriptRewriteAction, }; -use crate::middleware::{MatchedAttribute, MiddlewareChain, MiddlewareContext, MiddlewarePlan}; +use crate::middleware::{ + ElementHandler, MatchedAttribute, MiddlewareChain, MiddlewareContext, MiddlewarePlan, + TextHandler, +}; use crate::publisher::build_empty_bids_script; use crate::settings::Settings; use crate::streaming_processor::{HtmlRewriterAdapter, StreamProcessor}; @@ -242,19 +245,23 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso build_html_processor(config, MiddlewarePlan::default()) } -/// [`create_html_processor`], with the middleware of `chain` run on the -/// document as well. +/// [`create_html_processor`], with the middleware of the `fetch` chain run +/// on the document as well, and then those of the `serve` chain where the +/// document goes to one reader and is not stored. +/// +/// `serve` is `None` for a document that may be stored, whose serve chain +/// runs on each reader's copy instead, through [`create_serve_processor`]. /// /// Each middleware is asked once, here, what it will do with this document. -/// The state the chain shares is the document's own and holds nothing a -/// request left, because what this processor writes may be stored and served -/// to other readers. +/// The state the fetch chain shares is the document's own and holds nothing a +/// request left, because what that chain writes may be stored and served to +/// other readers. /// -/// A middleware's head markup follows the hooks' own. Its element handlers -/// are asked after core has moved the origin's address in an attribute and -/// after the hooks. Its text handlers are handed a chunk as the hooks and the -/// handlers before them left it, and its stream processors run after the -/// hooks' own. +/// A fetch middleware's head markup follows the hooks' own. Its element +/// handlers are asked after core has moved the origin's address in an +/// attribute and after the hooks. Its text handlers are handed a chunk as the +/// hooks and the handlers before them left it, and its stream processors run +/// after the hooks' own. /// /// # Errors /// @@ -265,18 +272,190 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso /// As [`create_html_processor`]. pub fn create_html_processor_with_middleware( config: HtmlProcessorConfig, - chain: &MiddlewareChain, + fetch: &MiddlewareChain, + serve: Option<&MiddlewareChain>, ) -> Result<impl StreamProcessor + use<>, String> { let document_state = IntegrationDocumentState::default(); - let plan = chain.plan(&MiddlewareContext { - phase: chain.phase(), + let plan = fetch.plan(&MiddlewareContext { + phase: fetch.phase(), request_host: &config.request_host, request_scheme: &config.request_scheme, origin_host: &config.origin_host, document_state: &document_state, max_buffered_script_bytes: config.max_buffered_body_bytes, })?; - Ok(build_html_processor(config, plan)) + let reader = match serve { + Some(serve) => create_serve_processor( + serve, + &ReaderDocument { + request_host: &config.request_host, + request_scheme: &config.request_scheme, + origin_host: &config.origin_host, + request_state: &config.request_state, + max_buffered_script_bytes: config.max_buffered_body_bytes, + }, + )?, + None => None, + }; + let mut processor = build_html_processor(config, plan); + processor.processors.extend(reader); + Ok(processor) +} + +/// What a serve chain is told about the reader's copy it works on. +#[derive(Debug, Clone, Copy)] +pub struct ReaderDocument<'a> { + /// Publisher-facing host the reader asked for. + pub request_host: &'a str, + /// Publisher-facing scheme the reader asked for. + pub request_scheme: &'a str, + /// Host the document was fetched from. + pub origin_host: &'a str, + /// What modules left on this reader's request for their page changes. + pub request_state: &'a IntegrationRequestState, + /// The most a middleware may hold of one script while it decides. + pub max_buffered_script_bytes: usize, +} + +/// A processor that carries out a serve chain alone, on one reader's copy of +/// a document core has already rewritten, or `None` when the chain has +/// nothing to do. +/// +/// Nothing of core's own is written or rewritten a second time. Head markup +/// goes straight before the script bundle's tag and markup for after the +/// bundle straight after it, so a document core wrote no bundle into gets +/// neither. +/// +/// # Errors +/// +/// When a handler's selector does not parse, naming the selector. +pub fn create_serve_processor( + chain: &MiddlewareChain, + document: &ReaderDocument<'_>, +) -> Result<Option<Box<dyn StreamProcessor>>, String> { + if chain.is_empty() { + return Ok(None); + } + let document_state = IntegrationDocumentState::default(); + document.request_state.seed(&document_state); + let plan = chain.plan(&MiddlewareContext { + phase: chain.phase(), + request_host: document.request_host, + request_scheme: document.request_scheme, + origin_host: document.origin_host, + document_state: &document_state, + max_buffered_script_bytes: document.max_buffered_script_bytes, + })?; + if plan.is_empty() { + return Ok(None); + } + Ok(Some(Box::new(build_serve_processor(plan)))) +} + +/// The tag of the script bundle, which core writes once at the start of +/// `<head>`. +const BUNDLE_SELECTOR: &str = "script#trustedserver-js"; + +/// One selector and what a rewriter does with what it matches. +type ContentHandler = ( + std::borrow::Cow<'static, lol_html::Selector>, + lol_html::ElementContentHandlers<'static>, +); + +/// The rewriter handlers that carry out a plan's element and text handlers. +/// +/// The text handlers share `edits` with every text handler registered before +/// them, so each is handed a chunk as the one before it left it. +/// +/// A selector that does not parse panics here, which +/// [`MiddlewareChain::plan`] rules out by refusing to plan one. +fn middleware_content_handlers( + element_handlers: Vec<Box<dyn ElementHandler>>, + text_handlers: Vec<Box<dyn TextHandler>>, + edits: &Rc<TextEdits>, +) -> Vec<ContentHandler> { + let mut handlers: Vec<ContentHandler> = Vec::new(); + for mut handler in element_handlers { + let selector = handler.selector().to_owned(); + let attribute = handler.attribute().to_owned(); + handlers.push(element!(selector, move |el| { + if el.removed() { + return Ok(()); + } + let Some(value) = el.get_attribute(&attribute) else { + return Ok(()); + }; + let element_name = el.tag_name(); + match handler.decide(&MatchedAttribute { + element_name: &element_name, + attribute_name: &attribute, + value: &value, + }) { + AttributeRewriteAction::Keep => {} + AttributeRewriteAction::Replace(replacement) => { + if replacement != value { + el.set_attribute(&attribute, &replacement)?; + } + } + AttributeRewriteAction::RemoveElement => el.remove(), + } + Ok(()) + })); + } + for mut handler in text_handlers { + let selector = handler.selector().to_owned(); + let edits = Rc::clone(edits); + handlers.push(text!(selector, move |text| { + let current = edits.current(text); + match handler.decide(¤t, text.last_in_text_node()) { + ScriptRewriteAction::Keep => {} + ScriptRewriteAction::Replace(rewritten) => edits.replace(text, &rewritten), + ScriptRewriteAction::RemoveNode => edits.remove(text), + } + Ok(()) + })); + } + handlers +} + +/// The processor behind [`create_serve_processor`], carrying out `plan` and +/// nothing else. +fn build_serve_processor(plan: MiddlewarePlan) -> HtmlWithStreamingProcessors { + let MiddlewarePlan { + head_inserts, + after_bundle_inserts, + element_handlers, + text_handlers, + processors, + } = plan; + let mut element_content_handlers: Vec<ContentHandler> = Vec::new(); + if !head_inserts.is_empty() || !after_bundle_inserts.is_empty() { + let before = head_inserts.concat(); + let after = after_bundle_inserts.concat(); + let written = Cell::new(false); + element_content_handlers.push(element!(BUNDLE_SELECTOR, move |el| { + // Core's tag is the first in the document. An element of the + // page's own under the same id is left alone. + if !written.replace(true) { + el.before(&before, ContentType::Html); + el.after(&after, ContentType::Html); + } + Ok(()) + })); + } + let edits = Rc::new(TextEdits::default()); + element_content_handlers.extend(middleware_content_handlers( + element_handlers, + text_handlers, + &edits, + )); + HtmlWithStreamingProcessors { + inner: Box::new(HtmlRewriterAdapter::new(RewriterSettings { + element_content_handlers, + ..RewriterSettings::default() + })), + processors, + } } /// The processor behind [`create_html_processor`] and @@ -291,7 +470,7 @@ pub fn create_html_processor_with_middleware( fn build_html_processor( config: HtmlProcessorConfig, plan: MiddlewarePlan, -) -> impl StreamProcessor + use<> { +) -> HtmlWithStreamingProcessors { let MiddlewarePlan { head_inserts: middleware_head_inserts, after_bundle_inserts: middleware_after_bundle_inserts, @@ -767,36 +946,6 @@ fn build_html_processor( })); } - // Asked after core has moved the origin's address in an attribute and - // after the hooks, so a handler judges an address as it will be served. - for mut handler in middleware_element_handlers { - let selector = handler.selector().to_owned(); - let attribute = handler.attribute().to_owned(); - element_content_handlers.push(element!(selector, move |el| { - if el.removed() { - return Ok(()); - } - let Some(value) = el.get_attribute(&attribute) else { - return Ok(()); - }; - let element_name = el.tag_name(); - match handler.decide(&MatchedAttribute { - element_name: &element_name, - attribute_name: &attribute, - value: &value, - }) { - AttributeRewriteAction::Keep => {} - AttributeRewriteAction::Replace(replacement) => { - if replacement != value { - el.set_attribute(&attribute, &replacement)?; - } - } - AttributeRewriteAction::RemoveElement => el.remove(), - } - Ok(()) - })); - } - // Shared by every text handler below, the hooks' and the middleware's, // so each is handed a chunk as the one before it left it. let edits = Rc::new(TextEdits::default()); @@ -833,19 +982,14 @@ fn build_html_processor( })); } - for mut handler in middleware_text_handlers { - let selector = handler.selector().to_owned(); - let edits = Rc::clone(&edits); - element_content_handlers.push(text!(selector, move |text| { - let current = edits.current(text); - match handler.decide(¤t, text.last_in_text_node()) { - ScriptRewriteAction::Keep => {} - ScriptRewriteAction::Replace(rewritten) => edits.replace(text, &rewritten), - ScriptRewriteAction::RemoveNode => edits.remove(text), - } - Ok(()) - })); - } + // Registered after core's own handlers and the hooks', so an element + // handler judges an address as it will be served and a text handler is + // handed a script as the hooks left it. + element_content_handlers.extend(middleware_content_handlers( + middleware_element_handlers, + middleware_text_handlers, + &edits, + )); let rewriter_settings = RewriterSettings { document_content_handlers, @@ -908,7 +1052,8 @@ pub mod test_support { /// The page a reader of `html` receives from a deployment running /// `settings` with `registry`'s modules, read from the origin `chunk_size` - /// bytes at a time and asked for at [`REQUEST_PATH`]. + /// bytes at a time and asked for at [`REQUEST_PATH`]. The page is not + /// stored, so both phases' middleware run on the one pass. /// /// The script bundle's `?v=` hash is replaced by a fixed word, so a /// recording changes when a module's page change does and not when a @@ -932,19 +1077,25 @@ pub mod test_support { REQUEST_HOST, "https", ); - let chain = registry.middleware_chain( + let fetch = registry.middleware_chain( &settings.fetch, MiddlewarePhase::Fetch, HTML_MEDIA_TYPE, REQUEST_PATH, ); + let serve = registry.middleware_chain( + &settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + REQUEST_PATH, + ); let mut pipeline = StreamingPipeline::new( PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, chunk_size, }, - create_html_processor_with_middleware(config, &chain) + create_html_processor_with_middleware(config, &fetch, Some(&serve)) .expect("should plan the page's middleware"), ); let mut output = Vec::new(); diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index d5ec794447..7e55eaa955 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -3074,6 +3074,12 @@ pub(crate) mod test_support { /// setting at the start of `<head>`. The one named `.links` points every /// link that carries `data-fixture` at a fixed path. The one named /// `.broken` asks for a selector that does not parse. + /// + /// It supplies two serve middleware as well. The one named `.reader` + /// writes a marker ahead of the script bundle, saying which origin it was + /// told of and whether the request stand-in left its mark, and a script + /// straight after the bundle. The one named `.broken-reader` asks for a + /// selector that does not parse. pub(crate) mod middleware_fixture { use std::rc::Rc; use std::sync::Arc; @@ -3098,6 +3104,12 @@ pub(crate) mod test_support { pub(crate) const LINKS: &str = "testing.middleware-fixture.links"; /// The name of the middleware whose selector does not parse. pub(crate) const BROKEN: &str = "testing.middleware-fixture.broken"; + /// The name of the serve middleware that marks a reader's copy. + pub(crate) const READER: &str = "testing.middleware-fixture.reader"; + /// The name of the serve middleware whose selector does not parse. + pub(crate) const BROKEN_READER: &str = "testing.middleware-fixture.broken-reader"; + /// What the reader middleware writes straight after the bundle. + pub(crate) const READER_SCRIPT: &str = "<script data-fixture-reader></script>"; /// Where the links middleware points a link. pub(crate) const LINK_TARGET: &str = "/fixture/link"; /// The selector the broken middleware asks for. @@ -3142,6 +3154,64 @@ pub(crate) mod test_support { } } + /// The marker the reader middleware writes when told of `origin_host`, + /// for a request the request stand-in marked or did not. + pub(crate) fn reader_marker(origin_host: &str, marked: bool) -> String { + format!( + "<meta name=\"middleware-fixture-reader\" content=\"origin {origin_host}; marked \ + {marked}\">" + ) + } + + struct Reader; + + impl Middleware for Reader { + fn middleware_id(&self) -> &'static str { + READER + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] + } + + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + use super::request_fixture; + + let marked = context + .document_state + .get::<request_fixture::Mark>(request_fixture::ID) + .is_some(); + MiddlewareAction { + head_inserts: vec![reader_marker(context.origin_host, marked)], + after_bundle_inserts: vec![READER_SCRIPT.to_owned()], + ..MiddlewareAction::pass() + } + } + } + + struct BrokenReader; + + impl Middleware for BrokenReader { + fn middleware_id(&self) -> &'static str { + BROKEN_READER + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + BROKEN_SELECTOR, + "href", + Rc::new(|_matched| AttributeRewriteAction::keep()), + ))], + ..MiddlewareAction::pass() + } + } + } + struct Links; impl Middleware for Links { @@ -3202,6 +3272,8 @@ pub(crate) mod test_support { })) .with_middleware(Arc::new(Links)) .with_middleware(Arc::new(Broken)) + .with_middleware(Arc::new(Reader)) + .with_middleware(Arc::new(BrokenReader)) .build(), )) } @@ -6270,7 +6342,13 @@ mod tests { assert_eq!( registry.middleware_ids(), - [fixture::HEAD, fixture::LINKS, fixture::BROKEN], + [ + fixture::HEAD, + fixture::LINKS, + fixture::BROKEN, + fixture::READER, + fixture::BROKEN_READER, + ], "should list the middleware the selected module supplies" ); let chain_for = |media_type: &str, path: &str| { @@ -6318,10 +6396,12 @@ mod tests { message.contains( "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies" ) && message.contains(&format!( - "[{}, {}, {}]", + "[{}, {}, {}, {}, {}]", fixture::HEAD, fixture::LINKS, - fixture::BROKEN + fixture::BROKEN, + fixture::READER, + fixture::BROKEN_READER )), "should name the entry and list what could be named: {message}" ); @@ -6367,7 +6447,13 @@ mod tests { assert_eq!( unnamed(Vec::new()), - [fixture::HEAD, fixture::LINKS, fixture::BROKEN], + [ + fixture::HEAD, + fixture::LINKS, + fixture::BROKEN, + fixture::READER, + fixture::BROKEN_READER, + ], "should list every middleware when there are no entries" ); assert_eq!( @@ -6375,11 +6461,74 @@ mod tests { html_entry(Some("/news/"), &[fixture::LINKS]), html_entry(None, &[fixture::HEAD]), ]), - [fixture::BROKEN], + [fixture::BROKEN, fixture::READER, fixture::BROKEN_READER], "should leave out a middleware any entry names" ); } + #[test] + fn an_entry_naming_a_middleware_in_a_phase_it_does_not_run_in_is_refused() { + let mut settings = settings_naming(fixture::MODULE); + settings.serve = PhaseEntries::new(vec![html_entry(None, &[fixture::HEAD])]); + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse a fetch middleware named in a serve entry"); + let message = error.to_string(); + assert!( + message.contains(&format!("[[serve]] entry 1 names `{}`", fixture::HEAD)) + && message.contains("does not run in that phase. It runs in [[fetch]]"), + "should say which phase the middleware runs in: {message}" + ); + + let mut settings = settings_naming(fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![html_entry(None, &[fixture::READER])]); + let error = IntegrationRegistry::new(&settings) + .err() + .expect("should refuse a serve middleware named in a fetch entry"); + let message = error.to_string(); + assert!( + message.contains(&format!("[[fetch]] entry 1 names `{}`", fixture::READER)) + && message.contains("does not run in that phase. It runs in [[serve]]"), + "should say which phase the middleware runs in: {message}" + ); + } + + #[test] + fn each_phase_has_entries_of_its_own() { + let mut settings = settings_naming(fixture::MODULE); + settings.fetch = PhaseEntries::new(vec![html_entry(None, &[fixture::HEAD])]); + settings.serve = PhaseEntries::new(vec![html_entry(Some("/news/"), &[fixture::READER])]); + let registry = IntegrationRegistry::new(&settings) + .expect("should build a registry with an entry in each phase"); + + let chain_for = |phase: MiddlewarePhase, path: &str| { + registry + .middleware_chain(settings.phase_entries(phase), phase, HTML_MEDIA_TYPE, path) + .ids() + }; + assert_eq!( + chain_for(MiddlewarePhase::Fetch, "/news/today"), + [fixture::HEAD] + ); + assert_eq!( + chain_for(MiddlewarePhase::Serve, "/news/today"), + [fixture::READER] + ); + assert!( + chain_for(MiddlewarePhase::Serve, "/sport/today").is_empty(), + "should run no serve middleware on a path only the fetch entries cover" + ); + let unnamed: Vec<&str> = unnamed_middleware(&settings, ®istry.inner) + .into_iter() + .map(|(_, middleware)| middleware.middleware_id()) + .collect(); + assert_eq!( + unnamed, + [fixture::LINKS, fixture::BROKEN, fixture::BROKEN_READER], + "should count a middleware as named only by an entry of a phase it runs in" + ); + } + /// A middleware under a name of the test's choosing, running in the /// phases given. struct Named { diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index 1deb630232..48c131ef95 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -2,10 +2,11 @@ //! //! A middleware is a page change the settings place. A module registers the //! ones it supplies, and one runs only where an entry in the settings names -//! it. The entries are an ordered list. An entry covers one media type, -//! optionally only the requests under a path prefix, and names the middleware -//! to run in order. A response takes the first entry that covers it, and each -//! middleware is handed the document as the ones before it left it. +//! it. Each phase has its own ordered list of entries. An entry covers one +//! media type, optionally only the requests under a path prefix, and names +//! the middleware to run in order. A response takes the first entry that +//! covers it, and each middleware is handed the document as the ones before +//! it left it. //! //! ```toml //! [[fetch]] @@ -16,6 +17,10 @@ //! [[fetch]] //! media_type = "text/html" //! middleware = ["example.strip"] +//! +//! [[serve]] +//! media_type = "text/html" +//! middleware = ["example.reader"] //! ``` //! //! An entry carries no settings and switches nothing on. Whether a module @@ -25,7 +30,10 @@ //! //! Fetch is obtaining the document from the origin. A fetch middleware is //! handed nothing about the reader, so what it leaves is what a shared -//! template stores for every reader. +//! template stores for every reader. Serve is preparing the document for one +//! response. A serve middleware runs for each reader, on that reader's copy, +//! whether the page came from the store or from the origin, and nothing it +//! writes is stored. use std::fmt; use std::rc::Rc; @@ -46,17 +54,21 @@ pub enum MiddlewarePhase { /// On the document as the origin sent it. What is left is what a shared /// template stores, so nothing here may depend on the reader. Fetch, + /// On one reader's copy, on its way out, whether it came from the store + /// or from the origin. Nothing written here is stored. + Serve, } impl MiddlewarePhase { - /// Every phase, in the order they happen. - pub const ALL: [Self; 1] = [Self::Fetch]; + /// Both phases, in the order they happen. + pub const ALL: [Self; 2] = [Self::Fetch, Self::Serve]; /// The name the phase's entries are written under in the settings. #[must_use] pub const fn as_str(self) -> &'static str { match self { Self::Fetch => "fetch", + Self::Serve => "serve", } } } @@ -71,7 +83,8 @@ impl fmt::Display for MiddlewarePhase { /// /// In [`MiddlewarePhase::Fetch`] the document's state holds nothing a /// request left, because the document may be stored and served to other -/// readers. +/// readers. In [`MiddlewarePhase::Serve`] it holds what the modules' request +/// hooks left for this reader's request. #[derive(Debug, Clone, Copy)] pub struct MiddlewareContext<'a> { /// Which phase is running. @@ -80,7 +93,8 @@ pub struct MiddlewareContext<'a> { pub request_host: &'a str, /// Publisher-facing scheme the reader asked for. pub request_scheme: &'a str, - /// Host the document was fetched from. + /// Host the document was fetched from, the same for a page served from + /// the store as for one fetched for this request. pub origin_host: &'a str, /// State shared by the middleware working on this document in this /// phase. @@ -97,7 +111,8 @@ pub struct MatchedAttribute<'a> { /// The attribute the handler judges. pub attribute_name: &'a str, /// The attribute's value, as the handlers before this one left it. Core - /// has already moved the origin's address in it to the reader's host. + /// has already moved the origin's address in it to the reader's host, in + /// either phase. pub value: &'a str, } @@ -185,7 +200,8 @@ pub trait TextHandler { /// arrived. #[derive(Default)] pub struct MiddlewareAction { - /// Markup to write at the start of `<head>`, in the order given. + /// Markup to write in `<head>` ahead of the script bundle, in the order + /// given. A document with no `<head>` gets none. pub head_inserts: Vec<String>, /// Markup to write straight after the main script bundle and before any /// deferred one, for a script that needs the bundle to have run and has @@ -284,8 +300,8 @@ pub trait Middleware: Send + Sync { fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction; } -/// One entry of a phase, such as `[[fetch]]`, being which responses it -/// covers and the middleware run on them, in the order written. +/// One entry of `[[fetch]]` or `[[serve]]`, being which responses it covers +/// and the middleware run on them, in the order written. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct PhaseEntry { @@ -440,8 +456,8 @@ impl<'de> Deserialize<'de> for PhaseEntries { { let entries = Vec::<PhaseEntry>::deserialize(deserializer).map_err(|error| { de::Error::custom(format!( - "{error}. A phase is a list of entries, each written as [[fetch]] with \ - media_type and middleware, and path where it covers part of the site" + "{error}. A phase is a list of entries, each written as [[fetch]] or [[serve]] \ + with media_type and middleware, and path where it covers part of the site" )) })?; Ok(Self(entries)) @@ -538,7 +554,8 @@ impl MiddlewareChain { /// What one document's chain will do, in the order it happens. #[derive(Default)] pub struct MiddlewarePlan { - /// Markup for the start of `<head>`, in the order it is written. + /// Markup for `<head>`, ahead of the script bundle, in the order it is + /// written. pub head_inserts: Vec<String>, /// Markup for straight after the main script bundle, in the order it is /// written. diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index e004a25cca..c79392fde5 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -7,10 +7,11 @@ use serde_json::json; use super::*; use crate::html_processor::{ - HtmlProcessorConfig, create_html_processor, create_html_processor_with_middleware, + HtmlProcessorConfig, ReaderDocument, create_html_processor, + create_html_processor_with_middleware, create_serve_processor, }; -use crate::integrations::IntegrationRegistry; use crate::integrations::registry_test_support::request_fixture; +use crate::integrations::{IntegrationRegistry, IntegrationRequestState}; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use crate::test_support::tests::create_test_settings; @@ -21,6 +22,8 @@ const BUNDLE_TAG: &str = "id=\"trustedserver-js\""; type CreateFn = dyn Fn(&MiddlewareContext<'_>) -> MiddlewareAction + Send + Sync; /// A middleware made from a function, under a name of the test's choosing. +/// It says it runs in both phases, and a test runs it in the one its chain +/// names. struct FromFn { id: &'static str, create: Box<CreateFn>, @@ -32,7 +35,7 @@ impl Middleware for FromFn { } fn phases(&self) -> &[MiddlewarePhase] { - &[MiddlewarePhase::Fetch] + &MiddlewarePhase::ALL } fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { @@ -54,6 +57,55 @@ fn chain_of(middleware: Vec<Arc<dyn Middleware>>) -> MiddlewareChain { MiddlewareChain::new(MiddlewarePhase::Fetch, middleware) } +fn serve_chain_of(middleware: Vec<Arc<dyn Middleware>>) -> MiddlewareChain { + MiddlewareChain::new(MiddlewarePhase::Serve, middleware) +} + +/// What a serve chain is told of a reader whose request left `request_state`. +fn reader(request_state: &IntegrationRequestState) -> ReaderDocument<'_> { + ReaderDocument { + request_host: REQUEST_HOST, + request_scheme: "https", + origin_host: ORIGIN_HOST, + request_state, + max_buffered_script_bytes: 16 * 1024 * 1024, + } +} + +/// `html`, a document core has already rewritten, as a reader whose request +/// left `request_state` receives it after `chain` has run, read `chunk_size` +/// bytes at a time. A chain with nothing to do leaves it as it was. +fn served_with( + chain: &MiddlewareChain, + request_state: &IntegrationRequestState, + html: &str, + chunk_size: usize, +) -> String { + let Some(mut processor) = create_serve_processor(chain, &reader(request_state)) + .expect("should plan the reader's middleware") + else { + return html.to_owned(); + }; + let mut output = Vec::new(); + for chunk in html.as_bytes().chunks(chunk_size) { + output.extend( + processor + .process_chunk(chunk, false) + .expect("should process a piece of the reader's copy"), + ); + } + output.extend( + processor + .process_chunk(&[], true) + .expect("should finish the reader's copy"), + ); + String::from_utf8(output).expect("should leave the reader's copy UTF-8") +} + +fn served(chain: &MiddlewareChain, html: &str) -> String { + served_with(chain, &IntegrationRequestState::default(), html, 8192) +} + /// A middleware that writes `markup` at the start of `<head>`. fn head(id: &'static str, markup: &'static str) -> Arc<dyn Middleware> { middleware(id, move |_| MiddlewareAction { @@ -133,7 +185,7 @@ fn page_with( html: &str, chunk_size: usize, ) -> String { - let processor = create_html_processor_with_middleware(config, chain) + let processor = create_html_processor_with_middleware(config, chain, None) .expect("should plan the document's middleware"); let mut pipeline = StreamingPipeline::new( PipelineConfig { @@ -863,7 +915,7 @@ fn a_selector_that_does_not_parse_refuses_the_document_naming_the_middleware() { ]; for chain in chains { - let message = create_html_processor_with_middleware(config(), &chain) + let message = create_html_processor_with_middleware(config(), &chain, None) .err() .expect("should refuse to plan the document"); @@ -995,6 +1047,256 @@ fn a_middleware_runs_after_the_hooks_of_the_same_kind() { ); } +// --------------------------------------------------------------------------- +// What the document pipeline does with a serve chain +// --------------------------------------------------------------------------- + +/// [`PAGE`] as core leaves it for a store, with the script bundle in its +/// head and one link. +fn rewritten_page() -> String { + page( + &chain_of(Vec::new()), + r#"<html><head><title>Pagex"#, + ) +} + +#[test] +fn a_serve_chain_writes_head_markup_straight_before_the_bundle_and_the_rest_straight_after() { + let chain = serve_chain_of(vec![ + middleware("example.first", |_| MiddlewareAction { + head_inserts: vec!["".to_owned()], + after_bundle_inserts: vec!["".to_owned()], + ..MiddlewareAction::pass() + }), + middleware("example.second", |_| MiddlewareAction { + head_inserts: vec!["".to_owned()], + after_bundle_inserts: vec!["".to_owned()], + ..MiddlewareAction::pass() + }), + ]); + let rewritten = rewritten_page(); + + let page = served(&chain, &rewritten); + + assert!( + page.contains("" + ), + "should write the markup for after the bundle straight after it: {page}" + ); + assert_eq!( + page.matches(BUNDLE_TAG).count(), + 1, + "should write no second bundle: {page}" + ); + assert_eq!( + page.replace("", "") + .replace( + "", + "" + ), + rewritten, + "should change nothing else of a document core has already rewritten" + ); +} + +#[test] +fn a_serve_chain_writes_head_markup_once_where_the_page_repeats_the_bundle_s_id() { + let chain = serve_chain_of(vec![head("example.first", "")]); + let rewritten = rewritten_page().replace( + "", + "", + ); + + let page = served(&chain, &rewritten); + + assert_eq!( + page.matches("name=\"first\"").count(), + 1, + "should write head markup at core's tag alone: {page}" + ); + assert!( + index_of(&page, "name=\"first\"") < index_of(&page, ""), + "should write it in the head: {page}" + ); +} + +#[test] +fn a_reader_s_copy_with_no_bundle_gets_no_head_markup() { + let chain = serve_chain_of(vec![head("example.first", "<meta name=\"first\">")]); + + let page = served(&chain, "<p>a fragment</p>"); + + assert_eq!(page, "<p>a fragment</p>"); +} + +#[test] +fn a_serve_chain_s_handlers_work_on_the_page_as_core_left_it() { + let chain = serve_chain_of(vec![ + element("example.link", "a", "href", |matched| { + AttributeRewriteAction::replace(format!("{}?reader=1", matched.value)) + }), + text("example.script", "script", || { + |text: &str, _is_last: bool| { + if text.contains("var a = 1;") { + ScriptRewriteAction::replace(text.replace("var a = 1;", "var a = 2 && 3;")) + } else { + ScriptRewriteAction::keep() + } + } + }), + ]); + + let page = served(&chain, &rewritten_page()); + + assert!( + page.contains(r#"<a href="https://test.example.com/x?reader=1">"#), + "should hand an element handler the address core already moved: {page}" + ); + assert!( + page.contains("<script>var a = 2 && 3;</script>"), + "should write a text handler's replacement as it is: {page}" + ); +} + +#[test] +fn a_serve_chain_s_stream_processor_runs_over_the_reader_s_copy() { + let chain = serve_chain_of(vec![middleware("example.stream", |_| MiddlewareAction { + stream: Some(Box::new(Rewording { + from: "<title>Page", + to: "Reader", + comment: "", + })), + ..MiddlewareAction::pass() + })]); + + let page = served(&chain, &rewritten_page()); + + assert!(page.contains("Reader") && page.ends_with("")); +} + +#[test] +fn a_serve_middleware_reads_what_the_request_left() { + let chain = serve_chain_of(vec![middleware("example.reads", |context| { + let marked = context + .document_state + .get::(request_fixture::ID) + .is_some(); + MiddlewareAction { + head_inserts: vec![format!( + "", + context.phase, context.request_scheme, context.request_host, context.origin_host + )], + ..MiddlewareAction::pass() + } + })]); + let rewritten = rewritten_page(); + + let marked = served_with(&chain, &request_fixture::marked(), &rewritten, 8192); + let unmarked = served(&chain, &rewritten); + + let told = format!("serve https://{REQUEST_HOST} from {ORIGIN_HOST}"); + assert!( + marked.contains(&format!("content=\"{told}; marked true\"")), + "should hand a serve middleware what the request left: {marked}" + ); + assert!( + unmarked.contains(&format!("content=\"{told}; marked false\"")), + "should hand it nothing where the request left nothing: {unmarked}" + ); +} + +#[test] +fn a_serve_chain_with_nothing_to_do_makes_no_processor() { + let state = IntegrationRequestState::default(); + + let none = create_serve_processor(&serve_chain_of(Vec::new()), &reader(&state)) + .expect("should plan an empty chain"); + assert!(none.is_none(), "should make no processor for no middleware"); + + let passes = serve_chain_of(vec![middleware("example.pass", |_| { + MiddlewareAction::pass() + })]); + let none = + create_serve_processor(&passes, &reader(&state)).expect("should plan a chain that passes"); + assert!( + none.is_none(), + "should make no processor where every middleware leaves the page alone" + ); +} + +#[test] +fn a_serve_selector_that_does_not_parse_refuses_the_reader_s_copy() { + let chain = serve_chain_of(vec![element("example.broken", "a[", "href", |_| { + AttributeRewriteAction::keep() + })]); + + let message = create_serve_processor(&chain, &reader(&IntegrationRequestState::default())) + .err() + .expect("should refuse to plan the reader's copy"); + + assert!( + message.contains("the serve middleware `example.broken` cannot run") + && message.contains("`a[` is not a usable selector"), + "should name the phase, the middleware and the selector: {message}" + ); +} + +#[test] +fn on_a_page_that_is_not_stored_the_serve_chain_follows_the_fetch_chain_on_the_one_pass() { + let fetch = chain_of(vec![ + head("example.fetch-head", ""), + element("example.fetch-link", "a", "href", |_| { + AttributeRewriteAction::replace("/fetch") + }), + ]); + let serve = serve_chain_of(vec![ + head("example.serve-head", ""), + element("example.serve-link", "a", "href", |matched| { + AttributeRewriteAction::replace(format!("{}/serve", matched.value)) + }), + ]); + let html = + r#"Pagex"#; + + // Read whole, and then seven bytes at a time, so the second rewriter is + // fed the first one's output in pieces. + for chunk_size in [8192, 7] { + let processor = create_html_processor_with_middleware(config(), &fetch, Some(&serve)) + .expect("should plan both chains"); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + processor, + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should process the document"); + let page = String::from_utf8(output).expect("should leave the document UTF-8"); + + assert!( + page.contains(r#""#), + "should hand the serve chain the page as the fetch chain left it: {page}" + ); + let fetch_head = index_of(&page, "name=\"fetch\""); + let serve_head = index_of(&page, "name=\"serve\""); + assert!( + fetch_head < serve_head && serve_head < index_of(&page, BUNDLE_TAG), + "should write the serve chain's head markup after the fetch chain's and ahead of \ + the bundle: {page}" + ); + } +} + // --------------------------------------------------------------------------- // The settings // --------------------------------------------------------------------------- @@ -1032,13 +1334,59 @@ middleware = ["example.all"] ); } +#[test] +fn serve_entries_are_read_from_the_settings_beside_the_fetch_entries() { + let settings = settings_with_document( + r#" +[[fetch]] +media_type = "text/html" +middleware = ["example.all"] + +[[serve]] +media_type = "text/html" +path = "/news/" +middleware = ["example.reader"] +"#, + ) + .expect("should read settings with an entry in each phase"); + + assert_eq!( + settings.phase_entries(MiddlewarePhase::Fetch), + &PhaseEntries::new(vec![entry(None, &["example.all"])]) + ); + assert_eq!( + settings.phase_entries(MiddlewarePhase::Serve), + &PhaseEntries::new(vec![entry(Some("/news/"), &["example.reader"])]) + ); + let written = serde_json::to_value(&settings).expect("should write the settings"); + assert_eq!( + written.get("serve"), + Some(&json!([ + { "media_type": "text/html", "path": "/news/", "middleware": ["example.reader"] } + ])), + "should write the serve entries a document holds" + ); +} + +#[test] +fn settings_with_a_misshapen_serve_entry_are_refused_naming_the_phase() { + let message = + settings_with_document("\n[[serve]]\nmedia_type = \"text/html\"\nmiddleware = []\n") + .expect_err("should refuse the settings"); + + assert!( + message.contains("[[serve]] entry 1 names no middleware"), + "should name the phase and the entry: {message}" + ); +} + #[test] fn settings_without_entries_write_no_fetch_key() { let settings = settings_with_document("").expect("should read settings with no entries"); let written = serde_json::to_value(&settings).expect("should write the settings"); assert!( - written.get("fetch").is_none(), - "should leave the key out of a stored document that wrote no entries" + written.get("fetch").is_none() && written.get("serve").is_none(), + "should leave both keys out of a stored document that wrote no entries" ); let settings = settings_with_document( diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index b3bb60f9af..412f9d6718 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -1556,16 +1556,97 @@ fn create_html_stream_processor( // The middleware of the first [[fetch]] entry covering this path. What // they leave is what a shared template stores. - let chain = params.integration_registry.middleware_chain( + let fetch = params.integration_registry.middleware_chain( ¶ms.settings.fetch, MiddlewarePhase::Fetch, HTML_MEDIA_TYPE, params.request_path, ); - create_html_processor_with_middleware(config, &chain) + // A response that may be stored leaves this processor as the template + // every reader shares, so its [[serve]] entry runs on each reader's copy + // once that copy is assembled. See `run_serve_pass`. + let serve = (!params.shared_template_authorized).then(|| { + params.integration_registry.middleware_chain( + ¶ms.settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + params.request_path, + ) + }); + create_html_processor_with_middleware(config, &fetch, serve.as_ref()) .map_err(|message| Report::new(TrustedServerError::Configuration { message })) } +/// The processor that carries out the `[[serve]]` entry covering this request +/// on a reader's copy that is assembled and not yet encoded, or `None` when +/// no entry covers the request or its middleware have nothing to do. +/// +/// The origin host is read from the settings. A page served from the store +/// keeps no record of it, and a serve middleware is told the same about a +/// page from the store as about one fetched for this request. +fn reader_serve_processor( + params: &OwnedProcessResponseParams, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) -> Result>, Report> { + use crate::html_processor::{ReaderDocument, create_serve_processor}; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; + + if !is_html_content_type(¶ms.content_type) { + return Ok(None); + } + let chain = integration_registry.middleware_chain( + &settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + ¶ms.request_path, + ); + let origin_host = settings.publisher.origin_host(); + create_serve_processor( + &chain, + &ReaderDocument { + request_host: ¶ms.request_host, + request_scheme: ¶ms.request_scheme, + origin_host: &origin_host, + request_state: ¶ms.request_state, + max_buffered_script_bytes: settings.publisher.max_buffered_body_bytes, + }, + ) + .map_err(|message| Report::new(TrustedServerError::Configuration { message })) +} + +/// Runs the `[[serve]]` entry covering this request over a reader's whole +/// copy, assembled and not yet encoded. +fn run_serve_pass( + bytes: Vec, + params: &OwnedProcessResponseParams, + settings: &Settings, + integration_registry: &IntegrationRegistry, +) -> Result, Report> { + let Some(mut processor) = reader_serve_processor(params, settings, integration_registry)? + else { + return Ok(bytes); + }; + processor + .process_chunk(&bytes, true) + .change_context(TrustedServerError::Proxy { + message: "a serve middleware could not rewrite the page".to_string(), + }) +} + +/// One piece of a streamed reader's copy through the `[[serve]]` entry, when +/// one covers the request. +fn serve_chunk( + serve: &mut Option>, + bytes: Vec, + is_last: bool, +) -> Result, std::io::Error> { + match serve { + Some(processor) => processor.process_chunk(&bytes, is_last), + None => Ok(bytes), + } +} + /// Result of publisher request handling, indicating whether the response body /// should be streamed or has already been buffered. pub enum PublisherResponse { @@ -1747,8 +1828,8 @@ pub struct OwnedProcessResponseParams { pub(crate) origin_url: String, pub(crate) request_host: String, pub(crate) request_scheme: String, - /// The path the reader asked for, which decides the `[[fetch]]` entry - /// that covers the document. + /// The path the reader asked for, which decides the `[[fetch]]` and + /// `[[serve]]` entries that cover the document. pub(crate) request_path: String, pub(crate) content_type: String, pub(crate) ad_slots_script: Option, @@ -1979,6 +2060,11 @@ pub async fn buffer_publisher_response_async( set_assembly_response_state(&mut response, state); } let bytes = if was_authorized { + // The transform wrote a template every reader shares, so the + // [[serve]] entry runs here, on this reader's assembled copy. + // A page that was never authorized ran it inside the + // transform. + let bytes = run_serve_pass(bytes, ¶ms, settings, integration_registry)?; encode_complete_body(bytes, response_compression(&response))? } else { bytes @@ -2028,6 +2114,7 @@ pub async fn buffer_publisher_response_async( assembled.extend_from_slice(head); assembled.extend_from_slice(seam.as_bytes()); assembled.extend_from_slice(tail); + let assembled = run_serve_pass(assembled, ¶ms, settings, integration_registry)?; let assembled = encode_complete_body(assembled, response_compression(&response))?; response.headers_mut().insert( http::header::CONTENT_LENGTH, @@ -2511,6 +2598,23 @@ pub async fn publisher_response_into_streaming_response( return Ok(response); } + // Planned before the lazy body is built, so a serve entry that + // cannot run fails the response while it still can be failed. + let mut serve = match reader_serve_processor(¶ms, &settings, integration_registry) { + Ok(serve) => serve, + Err(err) => { + if let Some(dispatched) = params.dispatched_auction.take() { + emit_abandoned_auction( + &services, + params.auction_observation.take(), + dispatched, + "processor_init_error", + ) + .await; + } + return Err(err); + } + }; let services = services.clone(); let settings = Arc::clone(&settings); let orchestrator = Arc::clone(&orchestrator); @@ -2534,8 +2638,10 @@ pub async fn publisher_response_into_streaming_response( let (head, tail) = split_template_at_seam(&template) .map_err(|e| std::io::Error::other(e.to_string()))?; let mut encoder = BodyStreamEncoder::new(compression); + // Each piece goes through the serve entry as it leaves, so + // the head is still written before the auction is awaited. let encoded_head = encoder - .encode_chunk(head.to_vec()) + .encode_chunk(serve_chunk(&mut serve, head.to_vec(), false)?) .map_err(publisher_stream_error)?; if !encoded_head.is_empty() { yield bytes::Bytes::from(encoded_head); @@ -2566,14 +2672,14 @@ pub async fn publisher_response_into_streaming_response( let seam = seam_script_for(¶ms); if !seam.is_empty() { let encoded_seam = encoder - .encode_chunk(seam.into_bytes()) + .encode_chunk(serve_chunk(&mut serve, seam.into_bytes(), false)?) .map_err(publisher_stream_error)?; if !encoded_seam.is_empty() { yield bytes::Bytes::from(encoded_seam); } } let encoded_tail = encoder - .encode_chunk(tail.to_vec()) + .encode_chunk(serve_chunk(&mut serve, tail.to_vec(), true)?) .map_err(publisher_stream_error)?; if !encoded_tail.is_empty() { yield bytes::Bytes::from(encoded_tail); @@ -11257,6 +11363,329 @@ mod tests { } } + /// One `[[serve]]` entry naming `names` for `path`. + fn serve_entry(path: Option<&str>, names: &[&str]) -> crate::middleware::PhaseEntries { + crate::middleware::PhaseEntries::new(vec![crate::middleware::PhaseEntry { + media_type: crate::middleware::HTML_MEDIA_TYPE.to_owned(), + path: path.map(str::to_owned), + middleware: names.iter().map(|name| (*name).to_owned()).collect(), + }]) + } + + /// Settings in `mode` whose fetch entry runs the stand-in's head + /// middleware on every page and whose serve entry runs its reader + /// middleware under `serve_path`. + fn settings_with_both_phases( + mode: &str, + label: &str, + serve_path: Option<&str>, + ) -> Settings { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let mut settings = settings_with_fetch_entry(mode, label, None, &[fixture::HEAD]); + settings.serve = serve_entry(serve_path, &[fixture::READER]); + settings + } + + /// What the stand-in's reader middleware writes for a deployment + /// running `settings`, for a request that left a mark or did not. + fn reader_marker_for(settings: &Settings, marked: bool) -> String { + crate::integrations::registry_test_support::middleware_fixture::reader_marker( + &settings.publisher.origin_host(), + marked, + ) + } + + fn index_in(page: &str, needle: &str) -> usize { + page.find(needle) + .unwrap_or_else(|| panic!("should find `{needle}` in: {page}")) + } + + #[tokio::test] + async fn a_reader_of_a_page_that_is_not_stored_gets_the_serve_entry() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_both_phases("inline", label, None)); + let reader = reader_marker_for(&settings, false); + queue_shareable_html(&stub); + + let page = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + + assert_eq!( + page.matches(&reader).count(), + 1, + "should run the serve entry once on a {label} page: {page}" + ); + assert_eq!(page.matches(fixture::READER_SCRIPT).count(), 1); + let fetch_at = index_in(&page, &fixture::head_marker(label)); + let reader_at = index_in(&page, &reader); + let bundle_at = index_in(&page, "id=\"trustedserver-js\""); + let after_at = index_in(&page, fixture::READER_SCRIPT); + assert!( + fetch_at < reader_at && reader_at < bundle_at && bundle_at < after_at, + "should write the serve entry's markup around the bundle, after the fetch \ + entry's: {page}" + ); + } + } + + #[tokio::test] + async fn a_reader_of_a_shared_template_gets_the_serve_entry_and_the_store_never_does() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::clone(&cache), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = Arc::new(settings_with_both_phases("esi", label, None)); + // Told the same origin on the miss as on the hit, where the + // page's own record of it is empty. + let reader = reader_marker_for(&settings, false); + // One answer from the origin, so the second reader can only + // be served from the store. + queue_shareable_html(&stub); + + let miss = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + let hit = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + + assert_eq!( + stub.recorded_cache_intents().len(), + 1, + "should serve the second {label} reader from the stored template" + ); + for (reader_of, page) in [("miss", &miss), ("hit", &hit)] { + assert_eq!( + page.matches(&reader).count(), + 1, + "should run the serve entry once for the {label} reader of a \ + {reader_of}: {page}" + ); + assert_eq!( + page.matches(fixture::READER_SCRIPT).count(), + 1, + "should write its markup after the bundle once on a {reader_of}: {page}" + ); + assert_eq!( + page.matches(&fixture::head_marker(label)).count(), + 1, + "should serve what the fetch entry wrote once on a {reader_of}: {page}" + ); + } + let stored: Vec = cache + .entries + .lock() + .expect("should lock the stored templates") + .values() + .map(|entry| String::from_utf8_lossy(&entry.body).into_owned()) + .collect(); + assert_eq!(stored.len(), 1, "should store one template"); + assert!( + stored[0].contains(&fixture::head_marker(label)), + "should store what the fetch entry wrote: {}", + stored[0] + ); + assert!( + !stored[0].contains("middleware-fixture-reader") + && !stored[0].contains("data-fixture-reader"), + "should store nothing the serve entry wrote: {}", + stored[0] + ); + } + } + + #[tokio::test] + async fn a_page_from_the_store_sends_its_head_through_the_serve_entry_before_the_auction() { + // The streaming finalizer writes a stored page's head at once and + // waits for the auction only at the seam. The serve entry is a + // rewriter in that stream, so this checks it hands the head on + // instead of holding it until the page ends. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_both_phases("esi", "streamed", None)); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, navigation_request()).await; + let hit = run(&settings, &services, navigation_request()).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 1, + "the second request must be a hit, or this asserts nothing" + ); + let EdgeBody::Stream(mut stream) = hit.into_body() else { + panic!("a page from the store must stream, not buffer"); + }; + let first = stream + .next() + .await + .expect("the stream should yield a first chunk") + .expect("the first chunk should read"); + let first = String::from_utf8_lossy(&first); + + assert!( + first.contains(&reader_marker_for(&settings, false)) + && first.contains("id=\"trustedserver-js\""), + "the first chunk must be the head, with what the serve entry wrote in it: \ + {first}" + ); + assert!( + !first.contains("window.tsjs"), + "the first chunk must come before the bids script: {first}" + ); + } + + #[tokio::test] + async fn the_request_path_picks_the_serve_entry() { + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + Arc::new(RecordingTelemetrySink::default()), + ); + let settings = + Arc::new(settings_with_both_phases("inline", label, Some("/article"))); + let reader = reader_marker_for(&settings, false); + + queue_shareable_html(&stub); + let covered = + page_of(run_via(&settings, &services, navigation_request(), finalizer).await) + .await; + queue_shareable_html(&stub); + let elsewhere = page_of( + run_via( + &settings, + &services, + navigation_request_for("/elsewhere"), + finalizer, + ) + .await, + ) + .await; + + assert_eq!( + covered.matches(&reader).count(), + 1, + "should run the serve entry on a {label} page under its path: {covered}" + ); + assert!( + !elsewhere.contains("middleware-fixture-reader"), + "should run no serve entry on a {label} page no entry covers: {elsewhere}" + ); + } + } + + #[tokio::test] + async fn a_serve_middleware_reads_what_a_request_left_and_that_page_is_not_stored() { + for (finalizer, label) in [ + (Finalizer::Streaming, "streamed"), + (Finalizer::Buffered, "buffered"), + ] { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let services = services_with_cache_and_telemetry( + Arc::clone(&stub), + Arc::clone(&cache), + Arc::new(RecordingTelemetrySink::default()), + ); + // A shared mode, which a request that carries state of its + // own never takes. + let settings = Arc::new(settings_with_both_phases("esi", label, None)); + queue_shareable_html(&stub); + + let page = page_of( + run_via(&settings, &services, marked_navigation_request(), finalizer).await, + ) + .await; + + assert_eq!( + page.matches(&reader_marker_for(&settings, true)).count(), + 1, + "should hand the serve middleware what the {label} request left: {page}" + ); + assert!( + cache + .entries + .lock() + .expect("should lock the stored templates") + .is_empty(), + "should store nothing of a page made for one request" + ); + } + } + + #[tokio::test] + async fn a_reader_with_an_auction_gets_the_serve_entry_and_its_bids() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + for mode in ["inline", "esi"] { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let mut settings = settings_with_bidder(mode); + settings.select_module("testing", fixture::MODULE); + settings.serve = serve_entry(None, &[fixture::READER]); + let settings = Arc::new(settings); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + let reader = reader_marker_for(&settings, false); + + // Bid first, page second: one queue, and the auction is + // dispatched before the origin fetch. + queue_bid_response(&stub); + queue_shareable_html(&stub); + let first = + page_of(run_bidding(&settings, &services, navigation_request()).await).await; + // A second reader, whose page comes from the store under a + // shared mode and from the origin otherwise. + queue_bid_response(&stub); + if mode == "inline" { + queue_shareable_html(&stub); + } + let second = + page_of(run_bidding(&settings, &services, navigation_request()).await).await; + + for (which, page) in [("first", &first), ("second", &second)] { + assert_eq!( + page.matches(&reader).count(), + 1, + "should run the serve entry once for the {which} {mode} reader: {page}" + ); + let bids_at = index_in(page, "3.50"); + assert!( + bids_at < index_in(page, ""), + "should write the {which} {mode} reader's bid ahead of the end of the \ + body: {page}" + ); + } + } + } + /// Drive `handle_publisher_request` through the **EC-preload** fetch path. /// /// `run_with_orchestrator` passes `kv: None` and no EC id, so @@ -20701,6 +21130,131 @@ mod tests { } } + /// A serve entry that cannot run fails the response at each place a page + /// leaves from, being a page that is not stored and a page from the + /// store, through each finalizer. + #[tokio::test] + async fn a_serve_middleware_whose_selector_does_not_parse_fails_the_response_wherever_it_leaves() + { + use crate::error::IntoHttpResponse as _; + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; + + let mut settings = create_test_settings(); + settings.select_module("testing", fixture::MODULE); + settings.serve = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::BROKEN_READER.to_owned()], + }]); + let settings = Arc::new(settings); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + )); + let services = noop_services(); + let response = || { + Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build the response") + }; + let params = || { + Box::new(OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/".to_owned(), + content_type: "text/html; charset=utf-8".to_string(), + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }) + }; + let from_the_origin = || PublisherResponse::Stream { + response: response(), + body: EdgeBody::from( + b"x".to_vec(), + ), + params: params(), + }; + let from_the_store = || PublisherResponse::AssembleTemplate { + response: response(), + template: format!( + "x{AD_ASSEMBLY_SEAM}" + ) + .into_bytes(), + params: params(), + }; + + let mut results = Vec::new(); + let pages: [(&str, &dyn Fn() -> PublisherResponse); 2] = [ + ("from the origin", &from_the_origin), + ("from the store", &from_the_store), + ]; + for (page, publisher_response) in pages { + results.push(( + page, + "streaming", + publisher_response_into_streaming_response( + publisher_response(), + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + services.clone(), + ) + .await, + )); + results.push(( + page, + "buffered", + buffer_publisher_response_async( + publisher_response(), + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await, + )); + } + + for (page, finalizer, result) in results { + let error = result.err().unwrap_or_else(|| { + panic!("should fail a page {page} in the {finalizer} finalizer") + }); + assert!( + error.to_string().contains(&format!( + "the serve middleware `{}` cannot run", + fixture::BROKEN_READER + )), + "should name the middleware for a page {page} in the {finalizer} finalizer: \ + {error}" + ); + assert_eq!( + error.current_context().status_code(), + StatusCode::INTERNAL_SERVER_ERROR, + "should answer as a fault of the deployment's own for a page {page} in the \ + {finalizer} finalizer" + ); + } + } + /// Empty origin body on the streaming route must produce no output /// without erroring. Exercises the `Ok(0)` branch of `process_chunks` /// plus the processor's `is_last=true, chunk=[]` terminal call. diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 5a500e3801..48587f2641 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -3922,6 +3922,10 @@ pub struct Settings { /// as `[[fetch]]` entries. See [`crate::middleware`]. #[serde(default, skip_serializing_if = "PhaseEntries::is_empty")] pub fetch: PhaseEntries, + /// The page changes run on each reader's copy of a document, written as + /// `[[serve]]` entries. + #[serde(default, skip_serializing_if = "PhaseEntries::is_empty")] + pub serve: PhaseEntries, /// Where the loader wrote secrets, which the configuration view masks. /// Never read from a document and never written to one. #[serde(skip)] @@ -4567,6 +4571,7 @@ impl Settings { pub fn phase_entries(&self, phase: MiddlewarePhase) -> &PhaseEntries { match phase { MiddlewarePhase::Fetch => &self.fetch, + MiddlewarePhase::Serve => &self.serve, } } diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index e32cb8de4e..e304256b9d 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -228,7 +228,7 @@ fail and the service will return its startup-error response. | `[demand]` | several modules | The auction's demand sources | | `[device]` | one module | Device classification | | `[ec]` | one module | Edge Cookie identity, persistence, and partner sync | -| `[[fetch]]` | nothing | Which page changes run on which pages, and in what order | +| `[[fetch]]` | nothing | Which page changes run on a page as it is fetched, and in what order | | `[geo]` | one module | Which module resolves location, if any | | `[[handlers]]` | nothing | Ordered HTTP Basic-auth rules | | `[image_optimizer]` | nothing | Reusable Fastly Image Optimizer profiles | @@ -239,6 +239,7 @@ fail and the service will return its startup-error response. | `[request_signing]` | nothing | Outbound Ed25519 request signing and management-store IDs | | `[response_headers]` | nothing | Headers added to Trusted Server responses | | `[rewrite]` | nothing | First-party URL rewrite exclusions | +| `[[serve]]` | nothing | Which page changes run on each reader's copy of a page, and in what order | | `[tester_cookie]` | nothing | Optional tester-cookie endpoints | | `[tinybird]` | nothing | Direct Tinybird auction telemetry | | `[trusted_client_ip]` | nothing | Authenticated front-door client-IP forwarding | @@ -1729,8 +1730,19 @@ name no module supplies. Their settings live in `[demand.]` and A module can supply a change it makes to a page as a middleware, and a middleware runs only on the pages where an entry names it. Selecting the -module makes its middleware available, and the `[[fetch]]` entries say which -pages each one runs on and in what order. +module makes its middleware available, and the `[[fetch]]` and `[[serve]]` +entries say which pages each one runs on and in what order. + +The two lists are the two phases of a page. A middleware runs in the phase +its module built it for, and an entry of the other list that names it refuses +startup. + +| Phase | Runs | Told about the reader | Stored in a shared template | +| ----------- | ------------------------------------------------------------------------------ | --------------------- | --------------------------- | +| `[[fetch]]` | On the page as the origin sent it, once for each fetch | No | Yes | +| `[[serve]]` | On each reader's copy, whether the page came from the store or from the origin | Yes | No | + +An entry of either list has the same three fields. | Field | Type | Required | Description | | ------------ | ---------------- | -------- | -------------------------------------------------------------------------------------- | @@ -1752,6 +1764,10 @@ middleware = ["tag.example.cleanup", "tag.example"] [[fetch]] media_type = "text/html" middleware = ["tag.example"] + +[[serve]] +media_type = "text/html" +middleware = ["tag.example.reader"] ``` A fetch middleware runs on the page as the origin sent it. It is told nothing @@ -1760,6 +1776,12 @@ about the reader, because what it leaves is what a every reader, and the entries are part of what selects a stored template, so changing them never serves a page built under the old ones. +A serve middleware runs on one reader's copy, after the fetch middleware and +after a stored page is assembled for that reader. It is told what the +module's own request hooks left on that reader's request, and nothing it +writes is stored. The markup it adds to the head goes straight before the +script bundle, after the fetch middleware's. + A middleware takes its settings from its module's own `[
.]` table. An entry carries none and switches nothing on. @@ -1770,7 +1792,8 @@ other than `text/html`, a path that does not start with `/` or that holds a key an entry does not read, and an entry that an earlier one already covers. An entry naming a middleware that no running module supplies refuses startup, and the message lists the names that could be written. A middleware that no -entry names changes no page, and startup logs a warning that names it. +entry of its phase names changes no page, and startup logs a warning that +names it. The sections below give each module's settings, and the integration guides describe what each one does. @@ -3279,8 +3302,9 @@ from the file, and startup checks the rest and runs the first set again. **Page changes**: -- Every `[[fetch]]` entry covers `text/html`, has a path no earlier entry - already covers, and names at least one middleware, each once +- Every `[[fetch]]` and `[[serve]]` entry covers `text/html`, has a path no + earlier entry of its list already covers, and names at least one + middleware, each once ### Checked when the service starts @@ -3290,9 +3314,10 @@ Everything above runs again on the loaded configuration, and these join it: this build does not have, a missing settings table, or a table the selector does not name, stops the service on its next start. A passing `ts config validate` is not proof that a change to those four will start -- The middleware each `[[fetch]]` entry names. A name no running module - supplies stops the service on its next start, because which middleware a - build has is only known where its modules are registered +- The middleware each `[[fetch]]` and `[[serve]]` entry names. A name no + running module supplies, or one named in the phase it does not run in, + stops the service on its next start, because which middleware a build has + is only known where its modules are registered - Resolved secret values, so a passphrase shorter than 32 bytes, a placeholder or a weak handler password fails here - The compiled `permissions.yaml` policy, and the `assume_single_jurisdiction` diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index aefdc74a94..e95d52608d 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -308,7 +308,7 @@ builder every integration uses. | `.with_script_rewriter(...)` | Rewrites inline script contents | | `.with_html_stream_processor(...)` | Works on the document as it streams | | `.with_request_filter(...)` | Inspects a request and can turn it back before it reaches the origin | -| `.with_middleware(...)` | Offers a page change that a `[[fetch]]` entry may place by the middleware's name | +| `.with_middleware(...)` | Offers a page change that an entry may place by the middleware's name | | `.with_js_module(CarriedJsModule { source, sha256 })` | Carries the integration's own browser script, built outside `trusted-server-js` | | `.with_deferred_js()` | Serves the script as its own `x"#, + ); + + assert!( + page.contains(r#""#) + && page.contains(r#""#), + "should ask the one decision about each attribute named: {page}" + ); + assert!( + page.contains(r#""#) && page.contains(r#""#), + "should leave another value and another attribute alone: {page}" + ); +} + +#[test] +fn a_module_s_tests_can_call_a_middleware_with_a_test_page_s_context() { + use crate::html_processor::test_support::{ORIGIN_HOST, REQUEST_HOST}; + + let document_state = crate::integrations::IntegrationDocumentState::default(); + let context = test_support::context(MiddlewarePhase::Serve, &document_state); + + assert_eq!(context.phase, MiddlewarePhase::Serve); + assert_eq!( + ( + context.request_scheme, + context.request_host, + context.origin_host + ), + ("https", REQUEST_HOST, ORIGIN_HOST), + "should describe the page the recording helpers serve" + ); +} + #[test] fn an_element_handler_can_remove_the_element() { let chain = chain_of(vec![element("example.remove", "a.gone", "href", |_| { From 47ab129b74e831804066f082f7d80f71e50f04fb Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:11:41 +0100 Subject: [PATCH 09/36] The GPT bootstrap script reaches a page with one set of line endings The GPT module embeds `gpt_bootstrap.js` as it is on disk and writes it into the head of every page. A checkout that rewrites line endings, as a Windows checkout does under `core.autocrlf`, changed the bytes of the script that is served, so a deployment built there served a different page from one built anywhere else. The recorded page showed it. The Axum tests failed on the Windows runner in `the_page_a_reader_receives_is_the_recorded_one`, and giving the file CRLF line endings on a machine where the test passes makes the same test fail. The file is pinned to LF in the crate's `.gitattributes`, as the seam probe pins the script it embeds. --- crates/ad-tag/google/.gitattributes | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/crates/ad-tag/google/.gitattributes b/crates/ad-tag/google/.gitattributes index fad78ffdb5..edbb160800 100644 --- a/crates/ad-tag/google/.gitattributes +++ b/crates/ad-tag/google/.gitattributes @@ -2,3 +2,8 @@ # beside it, so both must arrive with the line endings they were committed # with on every platform. src/fixtures/page-change.*.html text eol=lf + +# The bootstrap script is embedded as it is on disk and written into the head +# of every page, so it must arrive with the line endings it was committed +# with on every platform. +src/gpt_bootstrap.js text eol=lf From 04c7bf304c04195a5685e5d582696ee4ddc0a1c8 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:14:05 +0100 Subject: [PATCH 10/36] Testlight's page change is a middleware Testlight pointed a `src` or `href` that names `testlight.js` at its shim through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `testing.testlight`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_scripts` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware. This is the first module to convert, so the changelog gains the entry for the whole change, which is breaking for a deployment that selects a converted module and writes no entry. The configuration guide gains the table of the middleware the stock modules supply, with this one row. --- CHANGELOG.md | 2 + .../src/fixtures/page-change.settings.toml | 4 + crates/testing/testlight/src/lib.rs | 102 ++++++++++-------- docs/guide/configuration.md | 8 ++ docs/guide/integrations/testlight.md | 4 + 5 files changed, 75 insertions(+), 45 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b2e334d03..f0e6727be1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware, and no longer on every page of a deployment that selects the module. A deployment adds the entries when it upgrades. Until it does, a converted module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. - A pushed configuration keeps DataDome's two deprecated secret store selectors, `server_side_key_secret_store` and `protection_test_bypass.credential_secret_store`, where they were removed as the settings were read. DataDome still accepts and discards both, so a configuration that carries one loads as before. - Integration modules have moved out of `trusted-server-core` into crates of their own under `crates///`, each named by that folder, so `crates/cmp/osano` is the module `cmp.osano`. The settings that select and configure a module do not change. `crates/trusted-server-modules` lists the modules a stock build ships from crates of their own, in the order their hooks run. Every adapter builds its state and loads its settings with that list, and `ts config validate` and `ts config push` run each stock module's own rules. Code that named a moved module under `trusted_server_core::integrations` names its crate instead, such as `trusted_server_cmp_osano`. Moved: Osano, Lockr, Permutive, Sourcepoint, Didomi, Google Tag Manager, Testlight, Next.js, Google Publisher Tags and its diagnostics, DataDome, the mock ad server, APS, Prebid, the Prebid Server demand and the plain OpenRTB demand. Core keeps the JavaScript asset proxy, which is its own. - The Fastly adapter is a library with a thin binary over it. A deployment that ships a vendor crate writes a binary of its own that calls `trusted_server_adapter_fastly::run_with(vec![...])` with that crate's integration builder, in place of editing the adapter. The builders are composed with the built-in ones, and a builder runs only when the settings select its module. The Wasm artifact keeps its name, `trusted-server-adapter-fastly.wasm`. @@ -64,6 +65,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Page changes as middleware. A module registers a change it makes to a page with `.with_middleware(...)`, and the ordered `[[fetch]]` and `[[serve]]` entries of the settings say which pages each one runs on and in what order. A fetch middleware runs on the page as the origin sent it and what it writes is stored with a shared template. A serve middleware runs on each reader's copy, whether the page came from the store or from the origin, and nothing it writes is stored. - A builder registers its module from the compiled auction plan with `with_plan_registration` and checks its settings against the plan with `with_plan_validator`, so a module whose page support follows what the plan selects is not named in `trusted-server-core`. Prebid and APS register this way. A renderer descriptor carries its builder's statement of which payload key holds the identifier the renderer picks its bid by, made with `BidRenderer::picking_bid_by`, and the page's `hb_adid` is read through that statement. - A module declares the settings in its own table that hold the name of a secret, with `with_secret_settings` on its builder, each with a rule for when the table puts it to use. A setting in use is looked up in the default secret store as the settings load and has to name a key for deploy validation to pass, and one not in use is cleared and never looked up. DataDome's server-side key and its test bypass credential are declared this way, and `trusted-server-core` no longer names them. - A module can act on one request from start to finish without `trusted-server-core` naming it. A request preparer or a request filter leaves a value in the request's `IntegrationRequestState`, which is copied into the document's state for the module's page hooks and handed to the response finalizer the builder declares with `with_response_finalizer`. A request that carries any value keeps to the origin path, so its document is never read from or stored as a shared template, and its HTML response is sent `private, no-store`. A head injector can also write straight after the script bundle with `after_bundle_inserts`, and a builder declares with `with_auction_token` that its browser script reads the token an auction publishes with its winning bids. The registry runs the request preparers once for a request. diff --git a/crates/testing/testlight/src/fixtures/page-change.settings.toml b/crates/testing/testlight/src/fixtures/page-change.settings.toml index a8f539e0d0..aa1a4e0c53 100644 --- a/crates/testing/testlight/src/fixtures/page-change.settings.toml +++ b/crates/testing/testlight/src/fixtures/page-change.settings.toml @@ -5,3 +5,7 @@ modules = ["testing.testlight"] endpoint = "https://example.com/openrtb2/auction" rewrite_scripts = true shim_src = "https://edge.example.com/static/testlight.js" + +[[fetch]] +media_type = "text/html" +middleware = ["testing.testlight"] diff --git a/crates/testing/testlight/src/lib.rs b/crates/testing/testlight/src/lib.rs index ff1112e33b..e737f0ff96 100644 --- a/crates/testing/testlight/src/lib.rs +++ b/crates/testing/testlight/src/lib.rs @@ -9,6 +9,7 @@ reason = "tests use direct diagnostics and panic-on-failure helpers" ) )] +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -23,9 +24,13 @@ use validator::Validate; use trusted_server_core::edge_cookie::recognized_ec_id; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, - UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_body_bounded, collect_response_bounded, + AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, + IntegrationRegistration, UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_body_bounded, + collect_response_bounded, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{ProxyRequestConfig, proxy_request}; @@ -192,7 +197,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(TESTLIGHT_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(ShimScript(integration))) .build(), )) } @@ -282,30 +287,43 @@ impl IntegrationProxy for TestlightIntegration { } } -impl IntegrationAttributeRewriter for TestlightIntegration { - fn integration_id(&self) -> &'static str { - TESTLIGHT_INTEGRATION_ID +impl TestlightIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the shim + /// when it is the vendor's script. + fn rewrite_script_address(&self, value: &str) -> AttributeRewriteAction { + let lowered = value.to_ascii_lowercase(); + if lowered.contains("testlight.js") { + AttributeRewriteAction::replace(self.config.shim_src.clone()) + } else { + AttributeRewriteAction::keep() + } } +} + +/// Loads the shim in place of the vendor's script, on the pages a `[[fetch]]` +/// entry names [`MODULE`] for. It does nothing unless `rewrite_scripts` is +/// set. +struct ShimScript(Arc); - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_scripts && matches!(attribute, "src" | "href") +impl Middleware for ShimScript { + fn middleware_id(&self) -> &'static str { + MODULE } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if !self.config.rewrite_scripts { - return AttributeRewriteAction::keep(); - } + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } - let lowered = attr_value.to_ascii_lowercase(); - if lowered.contains("testlight.js") { - AttributeRewriteAction::replace(self.config.shim_src.clone()) - } else { - AttributeRewriteAction::keep() + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + if !self.0.config.rewrite_scripts { + return MiddlewareAction::pass(); + } + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_script_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() } } } @@ -355,16 +373,8 @@ mod tests { }; let integration = TestlightIntegration::new(config); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - let rewritten = - integration.rewrite("src", "https://cdn.testlight.net/v1/testlight.js", &ctx); + integration.rewrite_script_address("https://cdn.testlight.net/v1/testlight.js"); assert!( matches!( rewritten, @@ -384,18 +394,16 @@ mod tests { rewrite_scripts: false, }; let integration = TestlightIntegration::new(config); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); - assert!(matches!( - integration.rewrite("src", "https://cdn.testlight.net/script.js", &ctx), - AttributeRewriteAction::Keep - )); + assert!( + ShimScript(integration).create(&context).is_pass(), + "should leave every page alone when script rewriting is off" + ); } #[test] @@ -615,7 +623,10 @@ mod tests { fn a_served_page_loads_the_shim_in_place_of_the_vendor_script() { use std::io::Cursor; - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::streaming_processor::{ Compression, PipelineConfig, StreamingPipeline, @@ -638,6 +649,7 @@ mod tests { }), ) .expect("should insert testlight config"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); @@ -654,7 +666,7 @@ mod tests { output_compression: Compression::None, chunk_size: 8192, }, - create_html_processor(config), + create_page_processor(&settings, ®istry, config), ); let mut output = Vec::new(); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index e304256b9d..a72815c918 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1785,6 +1785,14 @@ script bundle, after the fetch middleware's. A middleware takes its settings from its module's own `[
.]` table. An entry carries none and switches nothing on. +The modules this repository ships supply the middleware below. A deployment +that runs several of them names them all in one entry's list, because a page +takes one entry, and the order written is the order they run in. + +| Middleware | Phase | What it changes | +| ------------------- | ----------- | --------------------------------------------------------------------------------------------- | +| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | + An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type other than `text/html`, a path that does not start with `/` or that holds a diff --git a/docs/guide/integrations/testlight.md b/docs/guide/integrations/testlight.md index b7c23cca70..5d333a8742 100644 --- a/docs/guide/integrations/testlight.md +++ b/docs/guide/integrations/testlight.md @@ -23,6 +23,10 @@ When `rewrite_scripts = true`, `src` and `href` values containing are unchanged. The default shim source is the registry-free unified `tsjs` path because Testlight is an immediate bundled module. +The rewrite is the middleware `testing.testlight`, which runs on the pages a +`[[fetch]]` entry names it for. See +[Placing page changes](/guide/configuration#placing-page-changes). + This integration is for controlled development validation, not a production auction provider. See [Configuration](/guide/configuration#testlight-integration) for exact fields and [Integration Guide](/guide/integration-guide) for the From 1e36ef207a3c4e6a5a4607084d5170e834320f2c Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:14:52 +0100 Subject: [PATCH 11/36] Permutive's page change is a middleware Permutive pointed a `src` or `href` that is its SDK's address at the first-party endpoint through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `audience.permutive`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_sdk` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware. --- .../src/fixtures/page-change.settings.toml | 4 + crates/audience/permutive/src/lib.rs | 103 +++++++++--------- docs/guide/configuration.md | 7 +- docs/guide/integrations/permutive.md | 8 ++ 4 files changed, 67 insertions(+), 55 deletions(-) diff --git a/crates/audience/permutive/src/fixtures/page-change.settings.toml b/crates/audience/permutive/src/fixtures/page-change.settings.toml index 067a874e21..3463a0e97d 100644 --- a/crates/audience/permutive/src/fixtures/page-change.settings.toml +++ b/crates/audience/permutive/src/fixtures/page-change.settings.toml @@ -4,3 +4,7 @@ modules = ["permutive"] [audience.permutive] organization_id = "myorg" workspace_id = "abc123" + +[[fetch]] +media_type = "text/html" +middleware = ["audience.permutive"] diff --git a/crates/audience/permutive/src/lib.rs b/crates/audience/permutive/src/lib.rs index 9458149c24..933548d91d 100644 --- a/crates/audience/permutive/src/lib.rs +++ b/crates/audience/permutive/src/lib.rs @@ -15,6 +15,7 @@ ) )] +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -28,10 +29,13 @@ use validator::Validate; use trusted_server_core::constants::INTERNAL_HEADERS; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, - UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, collect_response_bounded, - ensure_integration_backend, + AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, + IntegrationRegistration, UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, + collect_response_bounded, ensure_integration_backend, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::{PlatformHttpRequest, RuntimeServices}; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -352,7 +356,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(PERMUTIVE_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(SdkAddress(integration))) .build(), )) } @@ -460,26 +464,11 @@ impl IntegrationProxy for PermutiveIntegration { } } -impl IntegrationAttributeRewriter for PermutiveIntegration { - fn integration_id(&self) -> &'static str { - PERMUTIVE_INTEGRATION_ID - } - - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_sdk && matches!(attribute, "src" | "href") - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if !self.config.rewrite_sdk { - return AttributeRewriteAction::keep(); - } - - if self.is_permutive_sdk_url(attr_value) { +impl PermutiveIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party endpoint when it is the SDK's address. + fn rewrite_sdk_address(&self, value: &str) -> AttributeRewriteAction { + if self.is_permutive_sdk_url(value) { // Rewrite to first-party SDK endpoint. // Root-relative so the browser resolves it against the page host. // Note: a page-level `` participates in this resolution, @@ -493,6 +482,33 @@ impl IntegrationAttributeRewriter for PermutiveIntegration { } } +/// Loads the SDK from the first-party endpoint, on the pages a `[[fetch]]` +/// entry names [`MODULE`] for. It does nothing unless `rewrite_sdk` is set. +struct SdkAddress(Arc); + +impl Middleware for SdkAddress { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + if !self.0.config.rewrite_sdk { + return MiddlewareAction::pass(); + } + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_sdk_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } + } +} + // Default value functions fn default_api_endpoint() -> String { "https://api.permutive.com".to_string() @@ -579,19 +595,8 @@ mod tests { }; let integration = PermutiveIntegration::new(config); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - - let rewritten = integration.rewrite( - "src", - "https://myorg.edge.permutive.app/workspace-123-web.js", - &ctx, - ); + let rewritten = integration + .rewrite_sdk_address("https://myorg.edge.permutive.app/workspace-123-web.js"); assert!(matches!(rewritten, AttributeRewriteAction::Replace(_))); if let AttributeRewriteAction::Replace(url) = rewritten { @@ -611,22 +616,16 @@ mod tests { rewrite_sdk: false, // Disabled }; let integration = PermutiveIntegration::new(config); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - - let rewritten = integration.rewrite( - "src", - "https://myorg.edge.permutive.app/workspace-123-web.js", - &ctx, + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, ); - assert!(matches!(rewritten, AttributeRewriteAction::Keep)); + assert!( + SdkAddress(integration).create(&context).is_pass(), + "should leave every page alone when SDK rewriting is off" + ); } #[test] diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index a72815c918..8108f2c5ae 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1789,9 +1789,10 @@ The modules this repository ships supply the middleware below. A deployment that runs several of them names them all in one entry's list, because a page takes one entry, and the order written is the order they run in. -| Middleware | Phase | What it changes | -| ------------------- | ----------- | --------------------------------------------------------------------------------------------- | -| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | +| Middleware | Phase | What it changes | +| -------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ | +| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | +| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/integrations/permutive.md b/docs/guide/integrations/permutive.md index d418e40202..d05103d7d4 100644 --- a/docs/guide/integrations/permutive.md +++ b/docs/guide/integrations/permutive.md @@ -26,8 +26,16 @@ api_endpoint = "https://api.permutive.com" secure_signals_endpoint = "https://secure-signals.permutive.app" cache_ttl_seconds = 3600 rewrite_sdk = true + +[[fetch]] +media_type = "text/html" +middleware = ["audience.permutive"] ``` +The SDK rewrite is the middleware `audience.permutive`, which runs on the +pages a `[[fetch]]` entry names it for. See +[Placing page changes](/guide/configuration#placing-page-changes). + ## Endpoints - `GET /integrations/permutive/sdk` - SDK serving From b37c64b70bdba01f328768ed22d6533a8d9161a6 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:15:25 +0100 Subject: [PATCH 12/36] lockr's page change is a middleware lockr pointed a `src` or `href` that is its SDK's address at the first-party endpoint through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `identity.lockr`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_sdk` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware. --- .../src/fixtures/page-change.settings.toml | 4 + crates/identity/lockr/src/lib.rs | 97 ++++++++++--------- docs/guide/configuration.md | 6 +- 3 files changed, 60 insertions(+), 47 deletions(-) diff --git a/crates/identity/lockr/src/fixtures/page-change.settings.toml b/crates/identity/lockr/src/fixtures/page-change.settings.toml index 33c2319acc..f1878e4b19 100644 --- a/crates/identity/lockr/src/fixtures/page-change.settings.toml +++ b/crates/identity/lockr/src/fixtures/page-change.settings.toml @@ -3,3 +3,7 @@ modules = ["lockr"] [identity.lockr] app_id = "test-app-id" + +[[fetch]] +media_type = "text/html" +middleware = ["identity.lockr"] diff --git a/crates/identity/lockr/src/lib.rs b/crates/identity/lockr/src/lib.rs index f7c9aefe62..11577890b2 100644 --- a/crates/identity/lockr/src/lib.rs +++ b/crates/identity/lockr/src/lib.rs @@ -19,6 +19,7 @@ ) )] +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -32,10 +33,13 @@ use validator::Validate; use trusted_server_core::constants::INTERNAL_HEADERS; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, - UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, collect_response_bounded, - ensure_integration_backend, + AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, + IntegrationRegistration, UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, + collect_response_bounded, ensure_integration_backend, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::{PlatformHttpRequest, RuntimeServices}; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -373,7 +377,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(LOCKR_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(SdkAddress(integration))) .build(), )) } @@ -421,26 +425,11 @@ impl IntegrationProxy for LockrIntegration { } } -impl IntegrationAttributeRewriter for LockrIntegration { - fn integration_id(&self) -> &'static str { - LOCKR_INTEGRATION_ID - } - - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_sdk && matches!(attribute, "src" | "href") - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if !self.config.rewrite_sdk { - return AttributeRewriteAction::Keep; - } - - if self.is_lockr_sdk_url(attr_value) { +impl LockrIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party endpoint when it is the SDK's address. + fn rewrite_sdk_address(&self, value: &str) -> AttributeRewriteAction { + if self.is_lockr_sdk_url(value) { // Root-relative so the browser resolves it against the page host. // Note: a page-level `` participates in this resolution, // so on pages that set an external base URL these resolve against @@ -455,6 +444,33 @@ impl IntegrationAttributeRewriter for LockrIntegration { } } +/// Loads the SDK from the first-party endpoint, on the pages a `[[fetch]]` +/// entry names [`MODULE`] for. It does nothing unless `rewrite_sdk` is set. +struct SdkAddress(Arc); + +impl Middleware for SdkAddress { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + if !self.0.config.rewrite_sdk { + return MiddlewareAction::pass(); + } + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_sdk_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } + } +} + fn default_api_endpoint() -> String { "https://identity.loc.kr".to_string() } @@ -495,16 +511,6 @@ mod tests { } } - fn test_context() -> IntegrationAttributeContext<'static> { - IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - } - } - #[test] fn test_lockr_sdk_url_detection() { let integration = LockrIntegration::new(test_config()); @@ -547,9 +553,8 @@ mod tests { #[test] fn test_attribute_rewriter_rewrites_sdk_urls() { let integration = LockrIntegration::new(test_config()); - let ctx = test_context(); - let result = integration.rewrite("src", "https://aim.loc.kr/identity-lockr-v1.0.js", &ctx); + let result = integration.rewrite_sdk_address("https://aim.loc.kr/identity-lockr-v1.0.js"); assert_eq!( result, @@ -561,9 +566,8 @@ mod tests { #[test] fn test_attribute_rewriter_keeps_non_lockr_urls() { let integration = LockrIntegration::new(test_config()); - let ctx = test_context(); - let result = integration.rewrite("src", "https://example.com/other.js", &ctx); + let result = integration.rewrite_sdk_address("https://example.com/other.js"); assert_eq!( result, @@ -579,14 +583,15 @@ mod tests { ..test_config() }; let integration = LockrIntegration::new(config); - let ctx = test_context(); - - let result = integration.rewrite("src", "https://aim.loc.kr/identity-lockr-v1.0.js", &ctx); + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); - assert_eq!( - result, - AttributeRewriteAction::Keep, - "should keep all URLs when rewrite_sdk is disabled" + assert!( + SdkAddress(integration).create(&context).is_pass(), + "should leave every page alone when rewrite_sdk is disabled" ); } diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 8108f2c5ae..454fa1dc70 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1793,6 +1793,7 @@ takes one entry, and the order written is the order they run in. | -------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ | | `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | | `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | +| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type @@ -1941,7 +1942,10 @@ Each `[[proxy.js_asset_proxy.assets]]` entry: | `rewrite_sdk_host` | Boolean or null | `null` | Deprecated compatibility input | | `origin_override` | URL or null | `null` | Optional upstream `Origin` override | -See [lockr](/guide/integrations/lockr). +The SDK rewrite is the middleware `identity.lockr`, which runs on the pages +a `[[fetch]]` entry names it for. See +[Placing page changes](#placing-page-changes) and +[lockr](/guide/integrations/lockr). ### Prebid Integration From 33d679081802d3f4dfdfaaa0b380e808c44d5717 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:15:59 +0100 Subject: [PATCH 13/36] Didomi's page change is a middleware Didomi wrote `window.__tsjs_didomi` into the head through a head injector, on every page of a deployment that selected it. The script hands the browser module the path Didomi is served under. It is now the middleware `cmp.didomi`, which runs on the pages a `[[fetch]]` entry names it for, and writes the same script. The recorded page is unchanged. Its settings gain the entry that places the middleware. --- .../src/fixtures/page-change.settings.toml | 4 + crates/cmp/didomi/src/lib.rs | 87 +++++++++++-------- docs/guide/configuration.md | 1 + docs/guide/integrations/didomi.md | 5 +- 4 files changed, 62 insertions(+), 35 deletions(-) diff --git a/crates/cmp/didomi/src/fixtures/page-change.settings.toml b/crates/cmp/didomi/src/fixtures/page-change.settings.toml index 378eb1e26d..7540bfd8a3 100644 --- a/crates/cmp/didomi/src/fixtures/page-change.settings.toml +++ b/crates/cmp/didomi/src/fixtures/page-change.settings.toml @@ -5,3 +5,7 @@ modules = ["didomi"] geo_query_parameters = false sdk_origin = "https://sdk.example.com" api_origin = "https://api.example.com" + +[[fetch]] +media_type = "text/html" +middleware = ["cmp.didomi"] diff --git a/crates/cmp/didomi/src/lib.rs b/crates/cmp/didomi/src/lib.rs index a5ba6aba15..d0d4204251 100644 --- a/crates/cmp/didomi/src/lib.rs +++ b/crates/cmp/didomi/src/lib.rs @@ -22,9 +22,11 @@ use validator::{Validate, ValidationError}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationHeadInjector, - IntegrationHtmlContext, IntegrationProxy, IntegrationRegistration, collect_body_bounded, - ensure_integration_backend, + INTEGRATION_MAX_BODY_BYTES, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, + collect_body_bounded, ensure_integration_backend, +}; +use trusted_server_core::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, }; use trusted_server_core::platform::{GeoInfo, PlatformHttpRequest, RuntimeServices}; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -437,7 +439,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(DIDOMI_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_head_injector(integration) + .with_middleware(Arc::new(ClientConfig(integration))) .build(), )) } @@ -567,12 +569,10 @@ impl IntegrationProxy for DidomiIntegration { } } -impl IntegrationHeadInjector for DidomiIntegration { - fn integration_id(&self) -> &'static str { - DIDOMI_INTEGRATION_ID - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { +impl DidomiIntegration { + /// The script that hands the browser module the path this deployment + /// serves Didomi under. + fn client_config_script(&self) -> String { #[derive(Serialize)] #[serde(rename_all = "camelCase")] struct InjectedDidomiClientConfig { @@ -591,9 +591,28 @@ impl IntegrationHeadInjector for DidomiIntegration { }) .replace("window.__tsjs_didomi={config_json};"# - )] + format!(r#""#) + } +} + +/// Writes the browser module's settings into the head, on the pages a +/// `[[fetch]]` entry names [`MODULE`] for. +struct ClientConfig(Arc); + +impl Middleware for ClientConfig { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + head_inserts: vec![self.0.client_config_script()], + ..MiddlewareAction::pass() + } } } @@ -604,7 +623,7 @@ mod tests { use super::*; use http::Method; - use trusted_server_core::integrations::{IntegrationDocumentState, IntegrationRegistry}; + use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::platform::PlatformCacheIntent; use trusted_server_core::platform::test_support::{ NoopConfigStore, NoopSecretStore, StubBackend, StubHttpClient, @@ -1362,18 +1381,25 @@ mod tests { api_origin: default_api_origin(), }; let integration = DidomiIntegration::new(Arc::new(custom_config)); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let inserts = integration.head_inserts(&ctx); - assert_eq!(inserts.len(), 1); + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + + let action = ClientConfig(integration).create(&context); + assert_eq!( - inserts[0], - r#""# + action.head_inserts, + [r#""#], + "should write the one script, with the path this deployment serves Didomi under" + ); + assert!( + action.after_bundle_inserts.is_empty() + && action.element_handlers.is_empty() + && action.text_handlers.is_empty() + && action.stream.is_none(), + "should change nothing else of the page" ); } @@ -1438,16 +1464,9 @@ mod tests { #[test] fn head_injector_default_path() { let integration = DidomiIntegration::new(Arc::new(config())); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - let inserts = integration.head_inserts(&ctx); + assert_eq!( - inserts[0], + integration.client_config_script(), r#""# ); } diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 454fa1dc70..311b9fe995 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1794,6 +1794,7 @@ takes one entry, and the order written is the order they run in. | `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | | `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | | `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | +| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/integrations/didomi.md b/docs/guide/integrations/didomi.md index c07b3b846d..e4669de5bf 100644 --- a/docs/guide/integrations/didomi.md +++ b/docs/guide/integrations/didomi.md @@ -79,7 +79,10 @@ This serves Didomi at `/my-custom-consent/*`. The path: - may start with `/`; Trusted Server normalizes the leading slash. Trusted Server passes the resolved path to its browser bundle through -`window.__tsjs_didomi.proxyPath`. +`window.__tsjs_didomi.proxyPath`. The script that sets it is the middleware +`cmp.didomi`, which runs on the pages a `[[fetch]]` entry names it for, so +name it for every page that loads Didomi. See +[Placing page changes](/guide/configuration#placing-page-changes). ## Notice-loader geo flow From b97c5b4a0ba2781d519817e3c421dcc1840b2958 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:16:31 +0100 Subject: [PATCH 14/36] Sourcepoint's page changes are one middleware Sourcepoint changed a page through two hooks, on every page of a deployment that selected it. A head injector wrote the browser module's settings and, with `rewrite_sdk` set, the trap on `window._sp_`. An attribute hook pointed a `src` or `href` on Sourcepoint's CDN at the first-party path, also only with `rewrite_sdk` set. Both are now the one middleware `cmp.sourcepoint`, which runs on the pages a `[[fetch]]` entry names it for and makes the same two changes under the same setting. The recorded page is unchanged. Its settings gain the entry that places the middleware. --- .../src/fixtures/page-change.settings.toml | 4 + crates/cmp/sourcepoint/src/lib.rs | 133 +++++++++--------- docs/guide/configuration.md | 13 +- docs/guide/integrations/sourcepoint.md | 2 + 4 files changed, 76 insertions(+), 76 deletions(-) diff --git a/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml b/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml index ddb1a43041..7c05107150 100644 --- a/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml +++ b/crates/cmp/sourcepoint/src/fixtures/page-change.settings.toml @@ -3,3 +3,7 @@ modules = ["sourcepoint"] [cmp.sourcepoint] rewrite_sdk = true + +[[fetch]] +media_type = "text/html" +middleware = ["cmp.sourcepoint"] diff --git a/crates/cmp/sourcepoint/src/lib.rs b/crates/cmp/sourcepoint/src/lib.rs index 00702d55da..64f1223f8d 100644 --- a/crates/cmp/sourcepoint/src/lib.rs +++ b/crates/cmp/sourcepoint/src/lib.rs @@ -7,10 +7,10 @@ //! //! | Layer | Mechanism | What it catches | //! |-------|-----------|-----------------| -//! | HTML attributes | `IntegrationAttributeRewriter` | Static `", self.config.rewrite_sdk @@ -1128,7 +1144,7 @@ impl IntegrationHeadInjector for SourcepointIntegration { #[cfg(test)] mod tests { use super::*; - use trusted_server_core::integrations::{IntegrationDocumentState, IntegrationRegistry}; + use trusted_server_core::integrations::IntegrationRegistry; use trusted_server_core::test_support::tests::create_test_settings; fn config() -> SourcepointConfig { @@ -1161,18 +1177,8 @@ mod tests { #[test] fn rewrites_cdn_urls_to_first_party_paths() { let integration = SourcepointIntegration::new(Arc::new(config())); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - - let rewritten = integration.rewrite( - "src", + let rewritten = integration.rewrite_sdk_address( "https://cdn.privacy-mgmt.com/mms/v2/get_site_data?account_id=821", - &ctx, ); assert_eq!( @@ -1186,16 +1192,9 @@ mod tests { #[test] fn leaves_non_sourcepoint_urls_unchanged() { let integration = SourcepointIntegration::new(Arc::new(config())); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; assert_eq!( - integration.rewrite("src", "https://example.com/script.js", &ctx), + integration.rewrite_sdk_address("https://example.com/script.js"), AttributeRewriteAction::keep() ); } @@ -1465,14 +1464,22 @@ mod tests { let mut cfg = config(); cfg.rewrite_sdk = false; let integration = SourcepointIntegration::new(Arc::new(cfg)); + let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + + let action = PageChange(integration).create(&context); assert!( - !integration.handles_attribute("src"), - "should not handle src when rewrite_sdk is false" + action.element_handlers.is_empty(), + "should judge no address when rewrite_sdk is false" ); - assert!( - !integration.handles_attribute("href"), - "should not handle href when rewrite_sdk is false" + assert_eq!( + action.head_inserts.len(), + 1, + "should still write the browser module's settings" ); } @@ -1501,15 +1508,8 @@ mod tests { #[test] fn head_injector_emits_config_script_plus_trap_when_enabled() { let integration = SourcepointIntegration::new(Arc::new(config())); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "ts.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - document_state: &document_state, - }; - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!( inserts.len(), 2, @@ -1564,15 +1564,8 @@ mod tests { let mut cfg = config(); cfg.rewrite_sdk = false; let integration = SourcepointIntegration::new(Arc::new(cfg)); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "ts.prospecta.com", - request_scheme: "https", - origin_host: "origin.prospecta.com", - document_state: &document_state, - }; - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!( inserts.len(), 1, diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 311b9fe995..68f7d2e0a1 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1789,12 +1789,13 @@ The modules this repository ships supply the middleware below. A deployment that runs several of them names them all in one entry's list, because a page takes one entry, and the order written is the order they run in. -| Middleware | Phase | What it changes | -| -------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ | -| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | -| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | -| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | -| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | +| Middleware | Phase | What it changes | +| -------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | +| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | +| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | +| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | +| `cmp.sourcepoint` | `[[fetch]]` | Writes `window.__tsjs_sourcepoint` into the head and, when `rewrite_sdk` is set, the trap on `window._sp_`, and points a `src` or `href` on Sourcepoint's CDN at `/integrations/sourcepoint/cdn` | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/integrations/sourcepoint.md b/docs/guide/integrations/sourcepoint.md index 8d5bce98cc..9028c582fe 100644 --- a/docs/guide/integrations/sourcepoint.md +++ b/docs/guide/integrations/sourcepoint.md @@ -49,6 +49,8 @@ The Sourcepoint browser module is now opt-in through `[cmp] module`. Existing de ## HTML Rewriting +Sourcepoint's changes to a page are the middleware `cmp.sourcepoint`, which runs on the pages a `[[fetch]]` entry names it for. See [Placing page changes](/guide/configuration#placing-page-changes). + When `rewrite_sdk = true`, Trusted Server rewrites matching Sourcepoint URLs in HTML responses: ```html From c64c9c4d0b84787efc149a0f41eaf22aef3a5458 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:22:10 +0100 Subject: [PATCH 15/36] Google Tag Manager's page changes are one middleware Google Tag Manager changed a page through two hooks, on every page of a deployment that selected it. An attribute hook pointed Google's tag addresses in `src` and `href` at the first-party path, and a script hook did the same in the text of inline scripts. Both are now the one middleware `tag.google-tag-manager`, which runs on the pages a `[[fetch]]` entry names it for and makes the same decisions. The script hook kept the text it was holding in the document's state, under its own id, so that one document's held text could not reach the next. The handler that replaces it is made for one document and holds the text itself. The test that an interrupted document leaves nothing for the next one now asks the registered middleware for a handler twice. The recorded page is unchanged. Its settings gain the entry that places the middleware. --- .../src/fixtures/page-change.settings.toml | 4 + crates/tag/google-tag-manager/src/lib.rs | 393 ++++++------------ docs/guide/configuration.md | 15 +- docs/guide/integrations/google_tag_manager.md | 2 +- 4 files changed, 149 insertions(+), 265 deletions(-) diff --git a/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml b/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml index abcdc009bc..8520eb5692 100644 --- a/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml +++ b/crates/tag/google-tag-manager/src/fixtures/page-change.settings.toml @@ -4,3 +4,7 @@ modules = ["google-tag-manager"] [tag.google-tag-manager] container_id = "GTM-TEST1234" upstream_url = "https://www.googletagmanager.com" + +[[fetch]] +media_type = "text/html" +middleware = ["tag.google-tag-manager"] diff --git a/crates/tag/google-tag-manager/src/lib.rs b/crates/tag/google-tag-manager/src/lib.rs index 4f5f8c84f6..add23f1024 100644 --- a/crates/tag/google-tag-manager/src/lib.rs +++ b/crates/tag/google-tag-manager/src/lib.rs @@ -24,6 +24,7 @@ ) )] +use std::rc::Rc; use std::sync::{Arc, LazyLock}; use async_trait::async_trait; @@ -36,12 +37,13 @@ use serde::{Deserialize, Serialize}; use validator::{Validate, ValidationError}; use trusted_server_core::error::TrustedServerError; -use trusted_server_core::integrations::ScriptTextAccumulator; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, IntegrationScriptContext, - IntegrationScriptRewriter, ScriptRewriteAction, UPSTREAM_SDK_MAX_RESPONSE_BYTES, - collect_response_bounded, + AttributeRewriteAction, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, + ScriptRewriteAction, UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_response_bounded, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, TextHandler, }; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{ @@ -312,9 +314,9 @@ fn default_max_beacon_body_size() -> usize { 65536 // 64KB - prevents memory pressure from oversized payloads } -/// GTM domain markers the script rewriter looks for. Kept in one place so the +/// GTM domain markers the script handler looks for. Kept in one place so the /// boundary-safe prefix check in [`might_contain_gtm_prefix`] and the full-match -/// check in [`GoogleTagManagerIntegration::rewrite`] cannot drift apart. +/// check in [`InlineSnippet`] cannot drift apart. const GTM_SCRIPT_MARKERS: &[&str] = &["googletagmanager.com", "google-analytics.com"]; /// Minimum trailing-prefix length required to engage the accumulation path. @@ -331,13 +333,11 @@ const GTM_MIN_PREFIX_LEN: usize = 6; /// Return true if `text` either already contains a GTM marker or ends with a /// proper prefix of one of length ≥ [`GTM_MIN_PREFIX_LEN`] — i.e., the next -/// fragment could still complete a match. Used to gate whether the GTM -/// rewriter should claim ownership of a script's output via -/// `RemoveNode`/`Replace`. +/// fragment could still complete a match. It gates whether the script +/// handler holds a script's text via `RemoveNode`/`Replace`. /// -/// Returning false means the rewriter can safely return `Keep` and leave the -/// script untouched, preserving any replacement a more-specific rewriter -/// (e.g., `NextJsNextDataRewriter`) made on the same element. +/// Returning false means the handler can safely return `Keep` and leave the +/// script untouched. fn might_contain_gtm_prefix(text: &str) -> bool { for marker in GTM_SCRIPT_MARKERS { if text.contains(marker) { @@ -941,8 +941,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(GTM_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_script_rewriter(integration) + .with_middleware(integration) .build(), )) } @@ -1034,58 +1033,68 @@ impl IntegrationProxy for GoogleTagManagerIntegration { } } -impl IntegrationAttributeRewriter for GoogleTagManagerIntegration { - fn integration_id(&self) -> &'static str { - GTM_INTEGRATION_ID +impl GoogleTagManagerIntegration { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party path when it is one of Google's tag addresses. + fn rewrite_address(value: &str) -> AttributeRewriteAction { + if Self::is_rewritable_url(value) { + AttributeRewriteAction::replace(Self::rewrite_gtm_urls(value)) + } else { + AttributeRewriteAction::keep() + } } +} - fn handles_attribute(&self, attribute: &str) -> bool { - matches!(attribute, "src" | "href") +/// Google Tag Manager's change to a page, on the pages a `[[fetch]]` entry +/// names [`MODULE`] for. It points Google's tag addresses at the first-party +/// path, in `src` and `href` attributes and in the text of inline scripts. +impl Middleware for GoogleTagManagerIntegration { + fn middleware_id(&self) -> &'static str { + MODULE } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if Self::is_rewritable_url(attr_value) { - AttributeRewriteAction::replace(Self::rewrite_gtm_urls(attr_value)) - } else { - AttributeRewriteAction::keep() + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let decide: Rc = + Rc::new(|matched| Self::rewrite_address(matched.value)); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + text_handlers: vec![Box::new(InlineSnippet::default())], + ..MiddlewareAction::pass() } } } -impl IntegrationScriptRewriter for GoogleTagManagerIntegration { - fn integration_id(&self) -> &'static str { - GTM_INTEGRATION_ID - } +/// Rewrites Google's tag addresses in the text of inline scripts. +/// +/// Made for one document, so text held for a script that never ends cannot +/// reach another document. +#[derive(Default)] +struct InlineSnippet { + /// Text held back while it might still turn out to hold a marker. + held: String, +} - fn selector(&self) -> &'static str { +impl TextHandler for InlineSnippet { + fn selector(&self) -> &str { "script" // Match all scripts to find inline GTM snippets } - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { - // Per document, never per registry: a registry-lifetime buffer would - // carry one document's partial script into the next. - let accumulator = ctx - .document_state - .get_or_insert_with(GTM_INTEGRATION_ID, ScriptTextAccumulator::default); - let mut buf = accumulator.buffer(); - - // Cheap gate: only engage the accumulation path for scripts whose - // running text could plausibly contain a GTM/GA domain. Unrelated - // scripts (the vast majority) return Keep on every fragment so they - // stay visible in `lol_html`'s output unchanged — and, critically, - // so a Replace from this rewriter does not clobber a Replace from - // another rewriter on the same element (e.g., `NextJsNextDataRewriter` - // on `script#__NEXT_DATA__`). See `might_contain_gtm_prefix` for the - // boundary-safe substring check. + fn decide(&mut self, content: &str, is_last: bool) -> ScriptRewriteAction { + let buf = &mut self.held; + + // Cheap gate: only hold text for scripts whose running text could + // plausibly contain a GTM/GA domain. Unrelated scripts (the vast + // majority) are kept on every fragment, so they pass through + // untouched. See `might_contain_gtm_prefix` for the boundary-safe + // substring check. let prior_and_current_might_match = - might_contain_gtm_prefix(&buf) || might_contain_gtm_prefix(content); + might_contain_gtm_prefix(buf) || might_contain_gtm_prefix(content); - if !ctx.is_last_in_text_node { + if !is_last { // Intermediate fragment. Only accumulate + suppress when the script // might still resolve to a GTM match. Otherwise return Keep so the // fragment is emitted as-is by `lol_html` and untouched by us. @@ -1098,7 +1107,7 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { // written after text that came later in the script. if !buf.is_empty() { buf.push_str(content); - return ScriptRewriteAction::replace(std::mem::take(&mut *buf)); + return ScriptRewriteAction::replace(std::mem::take(buf)); } return ScriptRewriteAction::keep(); } @@ -1108,7 +1117,7 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { None } else { buf.push_str(content); - Some(std::mem::take(&mut *buf)) + Some(std::mem::take(buf)) }; let text = full_content.as_deref().unwrap_or(content); @@ -1121,7 +1130,9 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { .iter() .any(|marker| text.contains(marker)) { - return ScriptRewriteAction::replace(Self::rewrite_gtm_urls(text)); + return ScriptRewriteAction::replace(GoogleTagManagerIntegration::rewrite_gtm_urls( + text, + )); } // No GTM content — if we accumulated fragments, emit them unchanged. @@ -1137,11 +1148,12 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { #[cfg(test)] mod tests { use super::*; - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationDocumentState, IntegrationRegistry, IntegrationScriptContext, - IntegrationScriptRewriter, ScriptRewriteAction, + AttributeRewriteAction, IntegrationDocumentState, IntegrationRegistry, ScriptRewriteAction, }; use trusted_server_core::platform::test_support::{ StubHttpClient, build_services_with_http_client, @@ -2375,23 +2387,9 @@ mod tests { #[test] fn test_attribute_rewriter() { - let config = tag_config("GTM-TEST1234", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - // Case 1: Standard HTTPS URL - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/gtm.js?id=GTM-TEST1234", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!( @@ -2403,11 +2401,8 @@ mod tests { } // Case 2: Protocol-relative URL - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", + let action = GoogleTagManagerIntegration::rewrite_address( "//www.googletagmanager.com/gtm.js?id=GTM-TEST1234", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!( @@ -2422,11 +2417,8 @@ mod tests { } // Case 3: gtag/js URL in href (preload link) - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/gtag/js?id=G-DQMZGMPHXN", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!( @@ -2441,11 +2433,8 @@ mod tests { } // Case 4: google-analytics.com URL in href - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.google-analytics.com/g/collect", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!(val, "/integrations/google_tag_manager/g/collect"); @@ -2457,11 +2446,8 @@ mod tests { } // Case 5: analytics.google.com URL in href - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://analytics.google.com/g/collect?v=2", - &ctx, ); if let AttributeRewriteAction::Replace(val) = action { assert_eq!(val, "/integrations/google_tag_manager/g/collect?v=2"); @@ -2473,12 +2459,7 @@ mod tests { } // Case 6: Other URL (should be kept) - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", - "https://other.com/script.js", - &ctx, - ); + let action = GoogleTagManagerIntegration::rewrite_address("https://other.com/script.js"); assert!(matches!(action, AttributeRewriteAction::Keep)); } @@ -2486,23 +2467,9 @@ mod tests { fn test_attribute_rewriter_rejects_false_positives() { // Test that URLs with GTM domains in query parameters or paths are NOT rewritten // This verifies the fix for P2: proper URL parsing instead of substring matching - let config = tag_config("GTM-TEST1234", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let ctx = IntegrationAttributeContext { - attribute_name: "href", - element_name: "a", - request_host: "example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - // Case 1: GTM domain in query parameter - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://evil.com/?redirect=https://www.google-analytics.com/collect", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2510,11 +2477,8 @@ mod tests { ); // Case 2: GTM domain in path component - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://example.com/www.googletagmanager.com/gtm.js", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2522,11 +2486,8 @@ mod tests { ); // Case 3: Unsupported path on valid GTM domain - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/ns.html", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2534,11 +2495,8 @@ mod tests { ); // Case 4: Fragment with GTM domain - should NOT be rewritten - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "href", + let action = GoogleTagManagerIntegration::rewrite_address( "https://example.com/page#https://www.googletagmanager.com/gtm.js", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Keep), @@ -2546,11 +2504,8 @@ mod tests { ); // Case 5: Valid GTM URL should STILL be rewritten (sanity check) - let action = IntegrationAttributeRewriter::rewrite( - &*integration, - "src", + let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/gtm.js?id=GTM-TEST", - &ctx, ); assert!( matches!(action, AttributeRewriteAction::Replace(_)), @@ -2560,19 +2515,7 @@ mod tests { #[test] fn test_script_rewriter() { - let config = tag_config("GTM-TEST1234", &[]); - let integration = GoogleTagManagerIntegration::new(config); - let doc_state = IntegrationDocumentState::default(); - - let ctx = IntegrationScriptContext { - selector: "script", - request_host: "example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: true, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &doc_state, - }; + let mut script = InlineSnippet::default(); // Case 1: Inline GTM snippet let snippet = r#"(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': @@ -2581,7 +2524,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-XXXX');"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, snippet, &ctx); + let action = script.decide(snippet, true); if let ScriptRewriteAction::Replace(val) = action { assert!(val.contains("/integrations/google_tag_manager/gtm.js")); assert!(!val.contains("https://www.googletagmanager.com/gtm.js")); @@ -2591,7 +2534,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= // Case 2: Protocol relative let snippet_proto = r#"j.src='//www.googletagmanager.com/gtm.js?id='+i+dl;"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, snippet_proto, &ctx); + let action = script.decide(snippet_proto, true); if let ScriptRewriteAction::Replace(val) = action { assert!(val.contains("/integrations/google_tag_manager/gtm.js")); assert!(!val.contains("//www.googletagmanager.com/gtm.js")); @@ -2604,7 +2547,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= // Case 3: Irrelevant script let other_script = "console.log('hello');"; - let action = IntegrationScriptRewriter::rewrite(&*integration, other_script, &ctx); + let action = script.decide(other_script, true); assert!(matches!(action, ScriptRewriteAction::Keep)); } @@ -3085,8 +3028,12 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= assert!(!rewritten.contains("https://www.google-analytics.com")); } + /// Settings that run this module's middleware on every page, for the + /// tests that select the module. fn make_settings() -> Settings { - create_test_settings() + let mut settings = create_test_settings(); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); + settings } fn config_from_settings( @@ -3199,7 +3146,7 @@ assume_single_jurisdiction = true let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -3239,7 +3186,7 @@ assume_single_jurisdiction = true let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -3304,7 +3251,7 @@ assume_single_jurisdiction = true let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -3368,7 +3315,7 @@ assume_single_jurisdiction = true output_compression: Compression::None, chunk_size, }, - create_html_processor(config), + create_page_processor(&settings, ®istry, config), ); let mut output = Vec::new(); pipeline @@ -3573,32 +3520,14 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= #[test] fn fragmented_gtm_snippet_is_accumulated_and_rewritten() { - let config = tag_config("GTM-FRAG1", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let document_state = IntegrationDocumentState::default(); - // Simulate lol_html splitting the GTM snippet mid-domain. let fragment1 = r#"(function(w,d,s,l,i){j.src='https://www.google"#; let fragment2 = r#"tagmanager.com/gtm.js?id='+i;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-FRAG1');"#; - let ctx_intermediate = IntegrationScriptContext { - selector: "script", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: false, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &document_state, - }; - let ctx_last = IntegrationScriptContext { - is_last_in_text_node: true, - ..ctx_intermediate - }; + let mut script = InlineSnippet::default(); // Intermediate fragment: should be suppressed. - let action1 = - IntegrationScriptRewriter::rewrite(&*integration, fragment1, &ctx_intermediate); + let action1 = script.decide(fragment1, false); assert_eq!( action1, ScriptRewriteAction::RemoveNode, @@ -3606,7 +3535,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= ); // Last fragment: should emit full rewritten content. - let action2 = IntegrationScriptRewriter::rewrite(&*integration, fragment2, &ctx_last); + let action2 = script.decide(fragment2, true); match action2 { ScriptRewriteAction::Replace(rewritten) => { assert!( @@ -3624,32 +3553,31 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= #[test] fn an_interrupted_document_leaves_no_residue_for_the_next_document() { - // The registry holds one `Arc` for the - // lifetime of the application, so both documents below go through the - // SAME rewriter instance. Only the document state differs. With the - // buffer owned by the rewriter this test fails: document two emits - // document one's secret. + // The registry holds one middleware for the lifetime of the + // application, so both documents below go through the SAME instance. + // Each is given a handler of its own, which is what holds the text. + // With the text held by the middleware this test fails: document two + // emits document one's secret. let integration = GoogleTagManagerIntegration::new(tag_config("GTM-LEAK01", &[])); + let handler_for = |document_state: &IntegrationDocumentState| { + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + document_state, + ); + integration.create(&context).text_handlers.remove(0) + }; // Document one: a GTM snippet that is cut off before its final // fragment ever arrives, as a client disconnect or truncated origin // body would do. let first_document = IntegrationDocumentState::default(); - let interrupted = IntegrationScriptContext { - selector: "script", - request_host: "first.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: false, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &first_document, - }; + let mut interrupted = handler_for(&first_document); // Must end mid-domain: that is what makes the cheap prefix gate // accumulate rather than pass the fragment through untouched. let secret = r#"(function(w,d,s,l,i){var token='SESSION-ONE-SECRET'; j.src='https://www.google"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, secret, &interrupted); + let action = interrupted.decide(secret, false); assert_eq!( action, ScriptRewriteAction::RemoveNode, @@ -3657,20 +3585,12 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= ); // Document two: a different request, a fresh document state, the same - // registry and the same rewriter. + // registry and the same middleware. let second_document = IntegrationDocumentState::default(); - let fresh = IntegrationScriptContext { - selector: "script", - request_host: "second.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: true, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &second_document, - }; + let mut fresh = handler_for(&second_document); let benign = r#"(function(w,d,s,l,i){j.src='https://www.googletagmanager.com/gtm.js?id='+i;})(window,document,'script','dataLayer','GTM-LEAK01');"#; - let action = IntegrationScriptRewriter::rewrite(&*integration, benign, &fresh); + let action = fresh.decide(benign, true); let emitted = match action { ScriptRewriteAction::Replace(rewritten) => rewritten, @@ -3695,40 +3615,21 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= #[test] fn non_gtm_fragmented_script_returns_keep_on_every_fragment() { // A script with no GTM marker in flight (and no plausible prefix) must - // return `Keep` on every fragment. Returning `RemoveNode` + - // `Replace(unchanged)` would stomp on other rewriters' replacements - // when selectors overlap (see `GoogleTagManagerIntegration::rewrite`). - let config = tag_config("GTM-PASS1", &[]); - let integration = GoogleTagManagerIntegration::new(config); - - let document_state = IntegrationDocumentState::default(); - + // return `Keep` on every fragment, so a script that is none of this + // module's business is never held or written again. let fragment1 = "console.log('hel"; let fragment2 = "lo world');"; - let ctx_intermediate = IntegrationScriptContext { - selector: "script", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - is_last_in_text_node: false, - max_buffered_script_bytes: 16 * 1024 * 1024, - document_state: &document_state, - }; - let ctx_last = IntegrationScriptContext { - is_last_in_text_node: true, - ..ctx_intermediate - }; + let mut script = InlineSnippet::default(); - let action1 = - IntegrationScriptRewriter::rewrite(&*integration, fragment1, &ctx_intermediate); + let action1 = script.decide(fragment1, false); assert_eq!( action1, ScriptRewriteAction::Keep, "non-GTM intermediate fragment should not be claimed" ); - let action2 = IntegrationScriptRewriter::rewrite(&*integration, fragment2, &ctx_last); + let action2 = script.decide(fragment2, true); assert_eq!( action2, ScriptRewriteAction::Keep, @@ -3741,33 +3642,16 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= /// is a fragmented non-GTM script. Both must produce correct output. #[test] fn accumulation_buffer_drains_between_consecutive_script_elements() { - let config = tag_config("GTM-MULTI1", &[]); - let integration = GoogleTagManagerIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - // --- First " )] } -} - -impl IntegrationAttributeRewriter for DataDomeIntegration { - fn integration_id(&self) -> &'static str { - DATADOME_INTEGRATION_ID - } - - fn handles_attribute(&self, attribute: &str) -> bool { - self.config.rewrite_sdk && matches!(attribute, "src" | "href") - } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { + /// What becomes of a `src` or an `href`, which is pointed at the + /// first-party path when it is a `DataDome` script's address. + fn rewrite_sdk_address(attr_value: &str) -> AttributeRewriteAction { // Check if this is a DataDome script URL let is_datadome = attr_value.contains("js.datadome.co") || attr_value.contains("datadome.co/tags.js"); @@ -1086,8 +1131,8 @@ pub fn register( let mut builder = IntegrationRegistration::builder(DATADOME_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_head_injector(integration.clone()); + .with_middleware(Arc::new(SdkAddress(integration.clone()))) + .with_middleware(Arc::new(ClientTag(integration.clone()))); if integration.config.enable_protection { builder = builder.with_request_filter(integration); @@ -1288,17 +1333,6 @@ mod tests { ); } - fn html_context_for_tests( - document_state: &trusted_server_core::integrations::IntegrationDocumentState, - ) -> IntegrationHtmlContext<'_> { - IntegrationHtmlContext { - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - document_state, - } - } - #[test] fn protection_secrets_are_absent_by_default() { let config = DataDomeConfig::default(); @@ -1524,10 +1558,8 @@ mod tests { config.client_side_key = "test-client-key".to_string(); config.client_side_tag_url = "/integrations/datadome/tags.js?one=1&two=2".to_string(); let integration = DataDomeIntegration::new(config); - let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); - let ctx = html_context_for_tests(&document_state); - - let inserts = integration.head_inserts(&ctx); + let document_state = IntegrationDocumentState::default(); + let inserts = integration.client_tag(&document_state); assert!( inserts[0].contains( @@ -1543,10 +1575,8 @@ mod tests { config.client_side_key = "test-client-key".to_string(); config.client_side_configuration = serde_json::json!({ "ajaxListenerPath": true }); let integration = DataDomeIntegration::new(config); - let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); - let ctx = html_context_for_tests(&document_state); - - let inserts = integration.head_inserts(&ctx); + let document_state = IntegrationDocumentState::default(); + let inserts = integration.client_tag(&document_state); assert_eq!(inserts.len(), 1, "should emit one combined DataDome insert"); assert!( @@ -1568,14 +1598,12 @@ mod tests { let mut suppressed = test_config(); suppressed.client_side_key = "test-client-key".to_string(); let suppressed_integration = DataDomeIntegration::new(suppressed); - let suppressed_state = - trusted_server_core::integrations::IntegrationDocumentState::default(); + let suppressed_state = IntegrationDocumentState::default(); suppressed_state .get_or_insert_with(DATADOME_INTEGRATION_ID, || DataDomeClientTagSuppressed); - let suppressed_ctx = html_context_for_tests(&suppressed_state); assert!( suppressed_integration - .head_inserts(&suppressed_ctx) + .client_tag(&suppressed_state) .is_empty(), "should omit the tag when the request is IP-excluded" ); @@ -1583,10 +1611,9 @@ mod tests { let mut blank_key = test_config(); blank_key.client_side_key = " ".to_string(); let integration = DataDomeIntegration::new(blank_key); - let document_state = trusted_server_core::integrations::IntegrationDocumentState::default(); - let ctx = html_context_for_tests(&document_state); + let document_state = IntegrationDocumentState::default(); assert!( - integration.head_inserts(&ctx).is_empty(), + integration.client_tag(&document_state).is_empty(), "should not inject a tag without a client-side key" ); @@ -1595,7 +1622,7 @@ mod tests { disabled.inject_client_side_tag = false; let integration = DataDomeIntegration::new(disabled); assert!( - integration.head_inserts(&ctx).is_empty(), + integration.client_tag(&document_state).is_empty(), "should not inject a tag when injection is disabled" ); } @@ -1643,24 +1670,50 @@ mod tests { } #[test] - fn attribute_rewriter_matches_datadome() { - let integration = DataDomeIntegration::new(test_config()); + fn the_tag_middleware_is_named_after_the_module() { + assert_eq!( + TAG_MIDDLEWARE, + trusted_server_core::module_name!("tag"), + "should be the module's name with a part of its own" + ); + } - // Should handle both src and href attributes - assert!(integration.handles_attribute("src")); - assert!(integration.handles_attribute("href")); - assert!(!integration.handles_attribute("data-src")); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "publisher.com", - request_scheme: "https", - origin_host: "origin.publisher.com", - }; + #[test] + fn the_sdk_rewrite_is_off_without_rewrite_sdk() { + let mut config = test_config(); + config.rewrite_sdk = false; + let document_state = IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + + assert!( + SdkAddress(DataDomeIntegration::new(config)) + .create(&context) + .is_pass(), + "should leave every page alone when rewrite_sdk is false" + ); + } + + #[test] + fn attribute_rewriter_matches_datadome() { + // Should judge both src and href attributes, and no other + let document_state = IntegrationDocumentState::default(); + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); + let action = SdkAddress(DataDomeIntegration::new(test_config())).create(&context); + let judged: Vec<&str> = action + .element_handlers + .iter() + .map(|handler| handler.attribute()) + .collect(); + assert_eq!(judged, ["src", "href"]); // Should rewrite DataDome URLs in src - let action = integration.rewrite("src", "https://js.datadome.co/tags.js", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co/tags.js"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/tags.js"); @@ -1669,7 +1722,7 @@ mod tests { } // Should rewrite DataDome URLs in href (for link preload/prefetch) - let action = integration.rewrite("href", "https://js.datadome.co/tags.js", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co/tags.js"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/tags.js"); @@ -1678,24 +1731,14 @@ mod tests { } // Should not rewrite other URLs - let action = integration.rewrite("src", "https://example.com/script.js", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://example.com/script.js"); assert!(matches!(action, AttributeRewriteAction::Keep)); } #[test] fn attribute_rewriter_preserves_path() { - let integration = DataDomeIntegration::new(test_config()); - - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "publisher.com", - request_scheme: "https", - origin_host: "origin.publisher.com", - }; - // Should preserve /js/... paths for signal collection API - let action = integration.rewrite("src", "https://js.datadome.co/js/check", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co/js/check"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/js/check"); @@ -1704,7 +1747,7 @@ mod tests { } // Should handle protocol-relative URLs - let action = integration.rewrite("href", "//js.datadome.co/js/signal", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("//js.datadome.co/js/signal"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/js/signal"); @@ -1713,7 +1756,7 @@ mod tests { } // Bare domain without path should default to /tags.js - let action = integration.rewrite("src", "https://js.datadome.co", &ctx); + let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co"); match action { AttributeRewriteAction::Replace(new_url) => { assert_eq!(new_url, "/integrations/datadome/tags.js"); @@ -1765,7 +1808,10 @@ mod tests { const PUBLISHER_TAGGED_DOCUMENT: &[u8] = br#"content"#; fn process_document(suppress: bool) -> String { - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::{IntegrationRegistry, IntegrationRequestState}; use trusted_server_core::streaming_processor::StreamProcessor as _; @@ -1777,6 +1823,8 @@ mod tests { &serde_json::json!({ "client_side_key": "test-client-key" }), ) .expect("should configure DataDome integration"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); + place_on_every_page(&mut settings, MiddlewarePhase::Serve, &[TAG_MIDDLEWARE]); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create integration registry with DataDome"); let mut request = http::Request::builder() @@ -1794,7 +1842,7 @@ mod tests { "https", ) .with_request_state(IntegrationRequestState::of(&request)); - let mut processor = create_html_processor(config); + let mut processor = create_page_processor(&settings, ®istry, config); let output = processor .process_chunk(PUBLISHER_TAGGED_DOCUMENT, true) diff --git a/crates/bot-protection/datadome/src/protection.rs b/crates/bot-protection/datadome/src/protection.rs index 17c8760a2f..eb6a0a8936 100644 --- a/crates/bot-protection/datadome/src/protection.rs +++ b/crates/bot-protection/datadome/src/protection.rs @@ -44,9 +44,9 @@ impl DataDomeIntegration { let test_bypass_matched = self.take_protection_test_bypass_header(input.request); if test_bypass_matched { // Both markers travel together. The first is DataDome's own - // tag-suppression signal, left for its head injector. The second tells - // core the response is personalized to this request and cannot be - // shared through a cache or a template. + // tag-suppression signal, left for its serve middleware. The second + // tells core the response is personalized to this request and cannot + // be shared through a cache or a template. super::suppress_client_tag(input.request); input .request @@ -165,9 +165,9 @@ impl DataDomeIntegration { } => { if suppress_client_tag { // Both markers travel together. The first is DataDome's own - // tag-suppression signal, left for its head injector. The second - // tells core the response is personalized to this request and - // cannot be shared through a cache or a template. + // tag-suppression signal, left for its serve middleware. The + // second tells core the response is personalized to this request + // and cannot be shared through a cache or a template. super::suppress_client_tag(input.request); input .request diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index b49fd9fbb5..cc7abf1656 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1789,14 +1789,16 @@ The modules this repository ships supply the middleware below. A deployment that runs several of them names them all in one entry's list, because a page takes one entry, and the order written is the order they run in. -| Middleware | Phase | What it changes | -| ------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | -| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | -| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | -| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | -| `cmp.sourcepoint` | `[[fetch]]` | Writes `window.__tsjs_sourcepoint` into the head and, when `rewrite_sdk` is set, the trap on `window._sp_`, and points a `src` or `href` on Sourcepoint's CDN at `/integrations/sourcepoint/cdn` | -| `tag.google-tag-manager` | `[[fetch]]` | Points Google Tag Manager and Google Analytics addresses at `/integrations/google_tag_manager`, in `src` and `href` attributes and in the text of inline scripts | +| Middleware | Phase | What it changes | +| ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | +| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | +| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | +| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | +| `cmp.sourcepoint` | `[[fetch]]` | Writes `window.__tsjs_sourcepoint` into the head and, when `rewrite_sdk` is set, the trap on `window._sp_`, and points a `src` or `href` on Sourcepoint's CDN at `/integrations/sourcepoint/cdn` | +| `tag.google-tag-manager` | `[[fetch]]` | Points Google Tag Manager and Google Analytics addresses at `/integrations/google_tag_manager`, in `src` and `href` attributes and in the text of inline scripts | +| `bot-protection.datadome` | `[[fetch]]` | Points a `src` or `href` that is a DataDome script's address at `/integrations/datadome`, when `rewrite_sdk` is set | +| `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/integrations/datadome.md b/docs/guide/integrations/datadome.md index 179bd59bcc..4114ab90ac 100644 --- a/docs/guide/integrations/datadome.md +++ b/docs/guide/integrations/datadome.md @@ -108,7 +108,7 @@ client_side_key = "YOUR_DATADOME_JS_KEY" inject_client_side_tag = true ``` -Trusted Server emits the DataDome configuration before the Trusted Server JavaScript bundle: +The tag is the serve middleware `bot-protection.datadome.tag`, which runs on each reader's copy of the pages a `[[serve]]` entry names it for. See [Placing page changes](/guide/configuration#placing-page-changes). It writes the DataDome configuration straight before the Trusted Server JavaScript bundle: ```html ``` -If `rewrite_sdk` is enabled, Trusted Server rewrites existing DataDome script tags in HTML: +If `rewrite_sdk` is enabled, Trusted Server rewrites existing DataDome script tags in HTML, on the pages a `[[fetch]]` entry names the middleware `bot-protection.datadome` for: ```html @@ -441,7 +441,7 @@ Also verify the request is not excluded by the default internal/static route exc ### HTML rewriting not working -Ensure `rewrite_sdk = true` and that your pages are being proxied through Trusted Server's HTML processing pipeline. +Ensure `rewrite_sdk = true`, that a `[[fetch]]` entry names `bot-protection.datadome` for the page, and that your pages are being proxied through Trusted Server's HTML processing pipeline. ## See also From 4d6e7b234305cde096b4f38eed43487f9e9044eb Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:28:24 +0100 Subject: [PATCH 17/36] Prebid's page changes are one middleware Prebid changed a page through two hooks, on every page of a deployment that ran it. A head injector wrote the configuration Prebid.js reads and the tag that loads the bundle. An attribute hook removed an element whose `src` or `href` matches `script_patterns`, which is the publisher's own Prebid script. Both are now the one middleware `auction.prebid`, which runs on the pages a `[[fetch]]` entry names it for and makes the same two changes. Neither depends on the request, so both stay in the fetch phase and what they write is stored with a shared template, as it was. The recorded page is unchanged. Its settings gain the entry that places the middleware. --- .../src/fixtures/page-change.settings.toml | 4 + crates/auction/prebid/src/lib.rs | 250 ++++++------------ docs/guide/configuration.md | 1 + docs/guide/integrations/prebid.md | 4 +- 4 files changed, 89 insertions(+), 170 deletions(-) diff --git a/crates/auction/prebid/src/fixtures/page-change.settings.toml b/crates/auction/prebid/src/fixtures/page-change.settings.toml index ef33b648bb..9e65dba815 100644 --- a/crates/auction/prebid/src/fixtures/page-change.settings.toml +++ b/crates/auction/prebid/src/fixtures/page-change.settings.toml @@ -9,3 +9,7 @@ external_bundle_url = "https://assets.example/prebid/trusted-prebid.js" timeout_ms = 1000 script_patterns = ["/prebid.js", "/prebid.min.js"] debug = false + +[[fetch]] +media_type = "text/html" +middleware = ["auction.prebid"] diff --git a/crates/auction/prebid/src/lib.rs b/crates/auction/prebid/src/lib.rs index 27a6d72fd1..5080d02a7d 100644 --- a/crates/auction/prebid/src/lib.rs +++ b/crates/auction/prebid/src/lib.rs @@ -18,6 +18,7 @@ )] use std::collections::HashSet; +use std::rc::Rc; use std::sync::{Arc, LazyLock}; use async_trait::async_trait; @@ -41,9 +42,11 @@ use trusted_server_core::auction::plan::AuctionPlan; use trusted_server_core::cache_policy::{CacheControlPolicy, EdgeCacheHeader}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationEndpoint, IntegrationHeadInjector, IntegrationHtmlContext, IntegrationProxy, - IntegrationRegistration, + AttributeRewriteAction, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{ProxyRequestConfig, is_host_allowed, proxy_request}; @@ -1100,8 +1103,7 @@ pub fn register_for_plan( Ok(Some( IntegrationRegistration::builder(PREBID_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_head_injector(integration) + .with_middleware(Arc::new(PageChange(integration))) .with_deferred_js() .build(), )) @@ -1119,8 +1121,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(PREBID_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration.clone()) - .with_head_injector(integration) + .with_middleware(Arc::new(PageChange(integration))) .with_deferred_js() .build(), )) @@ -1188,22 +1189,38 @@ impl IntegrationProxy for PrebidIntegration { } } -impl IntegrationAttributeRewriter for PrebidIntegration { - fn integration_id(&self) -> &'static str { - PREBID_INTEGRATION_ID +/// Prebid's change to a page, on the pages a `[[fetch]]` entry names +/// [`MODULE`] for. It writes the configuration Prebid.js reads and the tag +/// that loads the bundle into the head, and removes the publisher's own +/// Prebid script. +struct PageChange(Arc); + +impl Middleware for PageChange { + fn middleware_id(&self) -> &'static str { + MODULE } - fn handles_attribute(&self, attribute: &str) -> bool { - matches!(attribute, "src" | "href") + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if self.matches_script_url(attr_value) { + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let integration = Arc::clone(&self.0); + let decide: Rc = + Rc::new(move |matched| integration.rewrite_script_address(matched.value)); + MiddlewareAction { + head_inserts: self.0.head_markup(), + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } + } +} + +impl PrebidIntegration { + /// What becomes of a `src` or an `href`, whose element is removed when it + /// loads the publisher's own Prebid script. + fn rewrite_script_address(&self, value: &str) -> AttributeRewriteAction { + if self.matches_script_url(value) { AttributeRewriteAction::remove_element() } else { AttributeRewriteAction::keep() @@ -1227,12 +1244,10 @@ fn injected_prebid_config_script(config_json: &str) -> String { ) } -impl IntegrationHeadInjector for PrebidIntegration { - fn integration_id(&self) -> &'static str { - PREBID_INTEGRATION_ID - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { +impl PrebidIntegration { + /// The scripts written into the head, being the configuration Prebid.js + /// reads and the tag that loads the bundle. + fn head_markup(&self) -> Vec { if let Some(inserts) = &self.planned_head_inserts { return inserts.clone(); } @@ -1352,7 +1367,10 @@ mod tests { use trusted_server_core::auction::plan::{BidderId, BidderRouteConfig, ProviderId}; - use trusted_server_core::html_processor::{HtmlProcessorConfig, create_html_processor}; + use trusted_server_core::html_processor::HtmlProcessorConfig; + use trusted_server_core::html_processor::test_support::{ + create_page_processor, place_on_every_page, + }; use trusted_server_core::integrations::{ AttributeRewriteAction, IntegrationDocumentState, IntegrationRegistry, }; @@ -1419,6 +1437,7 @@ mod tests { }), ) .expect("should select Prebid"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); settings } @@ -1856,14 +1875,21 @@ server_url = "https://prebid.example/openrtb2/auction" IntegrationRegistry::with_plan_and_registrations(&settings, plan, &[builder()]) .expect("should build integration registry"); let document_state = IntegrationDocumentState::default(); - let context = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); - let inserts = registry.head_inserts(&context); + let inserts = registry + .middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + trusted_server_core::middleware::HTML_MEDIA_TYPE, + "/", + ) + .plan(&context) + .expect("should plan the page's middleware") + .head_inserts; let config_insert = inserts .iter() .find(|insert| insert.contains("window.__tsjs_prebid")) @@ -2140,34 +2166,18 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] #[test] fn attribute_rewriter_removes_prebid_scripts() { let integration = PrebidIntegration::new(base_config()); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - }; - - let rewritten = integration.rewrite("src", "https://cdn.prebid.org/prebid.min.js", &ctx); + let rewritten = integration.rewrite_script_address("https://cdn.prebid.org/prebid.min.js"); assert!(matches!(rewritten, AttributeRewriteAction::RemoveElement)); - let untouched = integration.rewrite("src", "https://cdn.example.com/app.js", &ctx); + let untouched = integration.rewrite_script_address("https://cdn.example.com/app.js"); assert!(matches!(untouched, AttributeRewriteAction::Keep)); } #[test] fn attribute_rewriter_handles_query_strings_and_links() { let integration = PrebidIntegration::new(base_config()); - let ctx = IntegrationAttributeContext { - attribute_name: "href", - element_name: "a", - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - }; - let rewritten = - integration.rewrite("href", "https://cdn.prebid.org/prebid.js?v=1.2.3", &ctx); + integration.rewrite_script_address("https://cdn.prebid.org/prebid.js?v=1.2.3"); assert!(matches!(rewritten, AttributeRewriteAction::RemoveElement)); } @@ -2194,7 +2204,7 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -2243,7 +2253,7 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -2967,15 +2977,7 @@ external_bundle_sri = "sha384-AAAA" #[test] fn head_injector_emits_config_script() { let integration = PrebidIntegration::new(base_config()); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should produce config and bundle inserts"); let script = &inserts[0]; @@ -3030,17 +3032,15 @@ external_bundle_sri = "sha384-AAAA" let plan = trusted_server_core::auction::compile_auction_plan(&settings) .expect("should compile auction plan"); let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; + let context = trusted_server_core::middleware::test_support::context( + MiddlewarePhase::Fetch, + &document_state, + ); for enabled in [true, false] { let registration = register_for_plan(&settings, &plan.clone().with_enabled(enabled)) .expect("should register prebid") .expect("should enable prebid"); - let inserts = registration.head_injectors[0].head_inserts(&ctx); + let inserts = registration.middleware[0].create(&context).head_inserts; let script = &inserts[0]; assert!( script.contains(r#""managedUserIds":[{"name":"exampleId""#), @@ -3078,15 +3078,7 @@ external_bundle_sri = "sha384-AAAA" }), }]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3106,15 +3098,7 @@ external_bundle_sri = "sha384-AAAA" storage: None, }]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3126,15 +3110,7 @@ external_bundle_sri = "sha384-AAAA" #[test] fn head_injector_omits_managed_user_ids_when_none_configured() { let integration = PrebidIntegration::new(base_config()); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3219,15 +3195,7 @@ external_bundle_sri = "sha384-AAAA" ..valid_managed_user_id() }]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( @@ -3255,15 +3223,7 @@ external_bundle_sri = "sha384-AAAA" config.excluded_gam_ad_unit_path_suffixes = vec!["/trackingonly".to_string(), "/measurement-only".to_string()]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains( @@ -3279,15 +3239,7 @@ external_bundle_sri = "sha384-AAAA" let mut config = base_config(); config.account_id = None; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains(r#""accountId":"""#), @@ -3305,15 +3257,7 @@ external_bundle_sri = "sha384-AAAA" config.external_bundle_sha256 = Some(sha256.clone()); config.external_bundle_sri = Some(test_sri("sha384", &[0; 48])); let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should emit config and bundle scripts"); assert!( @@ -3339,15 +3283,7 @@ external_bundle_sri = "sha384-AAAA" config.external_bundle_url = Some("https://assets.example/prebid/trusted-prebid.js".to_string()); let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should emit config and bundle scripts"); assert!( @@ -3367,15 +3303,7 @@ external_bundle_sri = "sha384-AAAA" let mut config = base_config(); config.account_id = Some("".to_string()); let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains(r#""accountId":"\u003c/script>\u003cscript>alert(1)\u003c/script>""#), @@ -3387,15 +3315,7 @@ external_bundle_sri = "sha384-AAAA" #[test] fn head_injector_omits_client_side_bidders_when_empty() { let integration = PrebidIntegration::new(base_config()); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( !script.contains("clientSideBidders"), @@ -3409,15 +3329,7 @@ external_bundle_sri = "sha384-AAAA" let mut config = base_config(); config.client_side_bidders = vec!["rubicon".to_string(), "magnite".to_string()]; let integration = PrebidIntegration::new(config); - let document_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "pub.example", - request_scheme: "https", - origin_host: "origin.example", - document_state: &document_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); let script = &inserts[0]; assert!( script.contains(r#""clientSideBidders":["rubicon","magnite"]"#), diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index cc7abf1656..72d9e371ec 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1799,6 +1799,7 @@ takes one entry, and the order written is the order they run in. | `tag.google-tag-manager` | `[[fetch]]` | Points Google Tag Manager and Google Analytics addresses at `/integrations/google_tag_manager`, in `src` and `href` attributes and in the text of inline scripts | | `bot-protection.datadome` | `[[fetch]]` | Points a `src` or `href` that is a DataDome script's address at `/integrations/datadome`, when `rewrite_sdk` is set | | `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | +| `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/integrations/prebid.md b/docs/guide/integrations/prebid.md index 4edec5108f..3526849ab0 100644 --- a/docs/guide/integrations/prebid.md +++ b/docs/guide/integrations/prebid.md @@ -455,7 +455,9 @@ Optional Ed25519 request signing for bid request authentication and fraud preven ### Script Interception -The `script_patterns` configuration controls which publisher-provided Prebid scripts are intercepted and replaced with empty JavaScript. Trusted Server always injects its managed first-party `/integrations/prebid/bundle.js` script for the configured external bundle, so interception prevents duplicate Prebid instances. +The `script_patterns` configuration controls which publisher-provided Prebid scripts are intercepted and replaced with empty JavaScript. Trusted Server injects its managed first-party `/integrations/prebid/bundle.js` script for the configured external bundle, so interception prevents duplicate Prebid instances. + +Both changes to a page, the removal of the publisher's own script tags and the configuration and bundle tag written into the head, are the middleware `auction.prebid`. It runs on the pages a `[[fetch]]` entry names it for, so name it for every page that runs Prebid. See [Placing page changes](/guide/configuration#placing-page-changes). **Pattern Matching**: From 60732a75be0e06b0d9f4534f7c71fa81d0e51643 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 03:31:19 +0100 Subject: [PATCH 18/36] A registration states the attributes it puts on the bundle's tag A browser module can read a setting from the tag the script bundle was loaded by. Two modules use that today, and each sets its attribute through a method of the head injector trait, which goes when the page hooks have converted. The attribute is not a change to a page that an entry places. It belongs to the module's browser code and is the same on every page the bundle is written into. A registration now states it with `.with_bundle_tag_attribute(name, value)`. The registry puts those after the attributes the head injectors still ask for, in registration order, and keeps the first value a name is given as it did. The tag is written as markup with no escaping, so a name or a value that could end the attribute refuses startup, where the tag's writer only asserted it in a debug build. Core's tagging stand-in states its attribute this way, so the test that a page's bundle tag carries it runs through the new path. --- .../src/integrations/registry.rs | 233 +++++++++++++++--- docs/guide/integration-guide.md | 33 +-- 2 files changed, 213 insertions(+), 53 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 7e55eaa955..9fe5c4de02 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -788,6 +788,10 @@ pub struct IntegrationRegistration { /// Declaring one does not make it run, because a middleware changes a /// page only where an entry of the settings names it. pub middleware: Vec>, + /// Attributes this registration puts on the script bundle's tag, each a + /// name and a value, for a browser module that reads a setting from the + /// tag it was loaded by. + pub bundle_tag_attributes: Vec<(&'static str, &'static str)>, /// Geo module this registration supplies, with the name `[geo] module` /// selects it by. /// @@ -837,6 +841,7 @@ impl IntegrationRegistrationBuilder { head_injectors: Vec::new(), request_filters: Vec::new(), middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), geo_module: None, ec_module: None, device_module: None, @@ -898,6 +903,19 @@ impl IntegrationRegistrationBuilder { self } + /// Put an attribute on the script bundle's tag, on every page the bundle + /// is written into. + /// + /// The name is lower case letters, digits and hyphens, and the value + /// holds none of `"`, `&`, `<` and `>`, or startup is refused. Where two + /// registrations give one name different values the first is kept, and + /// the other is logged as a warning. + #[must_use] + pub fn with_bundle_tag_attribute(mut self, name: &'static str, value: &'static str) -> Self { + self.registration.bundle_tag_attributes.push((name, value)); + self + } + /// Declare the geo module this registration supplies, under its `name`. /// /// The name is the path under `crates` of the crate the module lives in, @@ -1020,6 +1038,10 @@ struct IntegrationRegistryInner { // The middleware the running registrations supply, each against the // integration that supplied it, in registration order. middleware: Vec<(&'static str, Arc)>, + // The attributes the running registrations put on the script bundle's + // tag, each against the integration that asked for it, in registration + // order. + bundle_tag_attributes: Vec<(&'static str, (&'static str, &'static str))>, /// JS module IDs to include in the bundle that come from a source other than /// a registered integration, for example a module tied to the selected Edge /// Cookie module. Populated in [`IntegrationRegistry::new`] from settings. @@ -1072,6 +1094,7 @@ impl Default for IntegrationRegistryInner { head_injectors: Vec::new(), request_filters: Vec::new(), middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), extra_js_module_ids: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), @@ -1808,6 +1831,31 @@ impl IntegrationRegistry { .middleware .push((registration.integration_id, middleware)); } + for (name, value) in registration.bundle_tag_attributes { + // The tag is written as markup with no escaping, so a name + // or a value that could end the attribute is refused. + let name_is_plain = !name.is_empty() + && name.bytes().all(|byte| { + byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-' + }); + let value_is_plain = !value + .bytes() + .any(|byte| matches!(byte, b'"' | b'&' | b'<' | b'>')); + if !name_is_plain || !value_is_plain { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "integration `{}` puts the attribute `{name}` on the script \ + bundle's tag with the value `{value}`. A name is lower case \ + letters, digits and hyphens, and a value holds none of `\"`, `&`, \ + `<` and `>`", + registration.integration_id + ), + })); + } + inner + .bundle_tag_attributes + .push((registration.integration_id, (name, value))); + } if let Some((name, module)) = registration.geo_module { claim_module_name(&mut claimed, "geo", name, registration.integration_id)?; inner.geo_modules.push((name, module)); @@ -2218,25 +2266,32 @@ impl IntegrationRegistry { .collect() } - /// Collect static attributes for the publisher TSJS bundle tag. + /// Collect static attributes for the publisher TSJS bundle tag, being + /// those the head injectors ask for and then those the registrations + /// state, each keeping the first value a name is given. #[must_use] pub fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { + let from_injectors = self.inner.head_injectors.iter().flat_map(|injector| { + injector + .tsjs_script_tag_attributes() + .into_iter() + .map(|attribute| (injector.integration_id(), attribute)) + }); let mut attributes: Vec<(&'static str, &'static str)> = Vec::new(); - for injector in &self.inner.head_injectors { - for attribute in injector.tsjs_script_tag_attributes() { - let existing = attributes - .iter() - .find(|(name, _)| *name == attribute.0) - .copied(); - match existing { - None => attributes.push(attribute), - Some((_, kept_value)) if kept_value != attribute.1 => log::warn!( - "Integration `{}` emits conflicting value for publisher tag attribute `{}`; keeping the first", - injector.integration_id(), - attribute.0 - ), - Some(_) => {} - } + for (integration, attribute) in + from_injectors.chain(self.inner.bundle_tag_attributes.iter().copied()) + { + let existing = attributes + .iter() + .find(|(name, _)| *name == attribute.0) + .copied(); + match existing { + None => attributes.push(attribute), + Some((_, kept_value)) if kept_value != attribute.1 => log::warn!( + "Integration `{integration}` emits conflicting value for publisher tag attribute `{}`; keeping the first", + attribute.0 + ), + Some(_) => {} } } attributes @@ -2501,6 +2556,7 @@ impl IntegrationRegistry { head_injectors: Vec::new(), request_filters: Vec::new(), middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2545,6 +2601,7 @@ impl IntegrationRegistry { head_injectors, request_filters: Vec::new(), middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2585,6 +2642,7 @@ impl IntegrationRegistry { head_injectors: Vec::new(), request_filters, middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -2665,6 +2723,7 @@ impl IntegrationRegistry { head_injectors: Vec::new(), request_filters: Vec::new(), middleware: Vec::new(), + bundle_tag_attributes: Vec::new(), request_preparers: Vec::new(), response_finalizers: Vec::new(), deferred_js_ids: Vec::new(), @@ -3331,9 +3390,7 @@ pub(crate) mod test_support { impl crate::settings::IntegrationConfig for FixtureSettings {} - struct Tag { - mark_bundle: bool, - } + struct Tag; impl IntegrationHeadInjector for Tag { fn integration_id(&self) -> &'static str { @@ -3343,14 +3400,6 @@ pub(crate) mod test_support { fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { vec![format!("")] } - - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - if self.mark_bundle { - vec![(BUNDLE_ATTRIBUTE, "true")] - } else { - Vec::new() - } - } } impl IntegrationAttributeRewriter for Tag { @@ -3382,16 +3431,15 @@ pub(crate) mod test_support { let Some(config) = settings.module_config::(MODULE)? else { return Ok(None); }; - let tag = Arc::new(Tag { - mark_bundle: config.mark_bundle, - }); - Ok(Some( - IntegrationRegistration::builder(ID) - .without_js() - .with_attribute_rewriter(tag.clone()) - .with_head_injector(tag) - .build(), - )) + let tag = Arc::new(Tag); + let mut registration = IntegrationRegistration::builder(ID) + .without_js() + .with_attribute_rewriter(tag.clone()) + .with_head_injector(tag); + if config.mark_bundle { + registration = registration.with_bundle_tag_attribute(BUNDLE_ATTRIBUTE, "true"); + } + Ok(Some(registration.build())) } fn validate(settings: &Settings) -> Result> { @@ -6656,6 +6704,117 @@ mod tests { ); } + fn marking_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("marking") + .without_js() + .with_bundle_tag_attribute("data-example-mode", "first") + .with_bundle_tag_attribute("data-example-flag", "true") + .build(), + )) + } + + fn disagreeing_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("disagreeing") + .without_js() + .with_bundle_tag_attribute("data-example-mode", "second") + .with_bundle_tag_attribute("data-example-other", "kept") + .build(), + )) + } + + fn unsafe_name_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("unsafe_name") + .without_js() + .with_bundle_tag_attribute("onload=alert(1) data-x", "true") + .build(), + )) + } + + fn unsafe_value_registration( + _settings: &Settings, + ) -> Result, Report> { + Ok(Some( + IntegrationRegistration::builder("unsafe_value") + .without_js() + .with_bundle_tag_attribute("data-example-mode", "\">` must be neutralised: the only // `` left is the tag's own legitimate closer. @@ -1578,15 +1527,7 @@ mod tests { #[test] fn head_inserts_omits_slim_prebid_url_when_not_configured() { let integration = GptIntegration::new(test_config()); - let doc_state = IntegrationDocumentState::default(); - let ctx = IntegrationHtmlContext { - request_host: "edge.example.com", - request_scheme: "https", - origin_host: "example.com", - document_state: &doc_state, - }; - - let inserts = integration.head_inserts(&ctx); + let inserts = integration.head_markup(); assert_eq!( inserts.len(), diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 72d9e371ec..77e48289af 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1800,6 +1800,7 @@ takes one entry, and the order written is the order they run in. | `bot-protection.datadome` | `[[fetch]]` | Points a `src` or `href` that is a DataDome script's address at `/integrations/datadome`, when `rewrite_sdk` is set | | `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | | `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | +| `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/integrations/gpt.md b/docs/guide/integrations/gpt.md index e707dedd2c..afbae5f12c 100644 --- a/docs/guide/integrations/gpt.md +++ b/docs/guide/integrations/gpt.md @@ -40,7 +40,7 @@ All of these are served from `securepubads.g.doubleclick.net`. There are three layers: -1. **HTML attribute rewriting** (server-side) -- Rewrites `src`/`href` attributes on the initial `gpt.js` ` "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -186,7 +275,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -233,7 +322,7 @@ mod tests { // of it has to come back as the page wrote it. let html = r#""#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -252,7 +341,7 @@ mod tests { output_compression: Compression::None, chunk_size: 8192, }, - create_html_processor(config), + create_page_processor(&settings, ®istry, config), ); let mut output = Vec::new(); @@ -297,7 +386,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -316,7 +405,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -373,7 +462,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -386,7 +475,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -419,7 +508,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -432,7 +521,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -467,7 +556,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -481,7 +570,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -530,7 +619,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -544,7 +633,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -600,7 +689,7 @@ mod tests { "#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -613,7 +702,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); // Use small chunk size to force fragmentation let pipeline_config = PipelineConfig { input_compression: Compression::None, @@ -661,7 +750,7 @@ mod tests { // Build a __NEXT_DATA__ payload large enough to cross a 32-byte chunk boundary. let html = r#""#; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -674,7 +763,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); // Use a very small chunk size to force fragmentation. let pipeline_config = PipelineConfig { @@ -716,7 +805,7 @@ mod tests { "x".repeat(400), // pad to guarantee chunk-boundary fragmentation ); - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -729,7 +818,7 @@ mod tests { let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); let config = config_from_settings(&settings, ®istry); - let processor = create_html_processor(config); + let processor = create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, @@ -773,7 +862,7 @@ mod tests { /// Build the production HTML processor for the Next.js integration and run /// `html` through it at `chunk_size`, returning the streamed output. fn stream_nextjs_html(html: &str, chunk_size: usize) -> String { - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -785,7 +874,11 @@ mod tests { .expect("should update nextjs config"); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); - let processor = create_html_processor(config_from_settings(&settings, ®istry)); + let processor = create_page_processor( + &settings, + ®istry, + config_from_settings(&settings, ®istry), + ); let mut pipeline = StreamingPipeline::new( PipelineConfig { input_compression: Compression::None, @@ -878,7 +971,7 @@ mod tests { " ); - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -892,7 +985,11 @@ mod tests { .expect("should update nextjs config"); let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) .expect("should create registry"); - let processor = create_html_processor(config_from_settings(&settings, ®istry)); + let processor = create_page_processor( + &settings, + ®istry, + config_from_settings(&settings, ®istry), + ); let mut pipeline = StreamingPipeline::new( PipelineConfig { input_compression: Compression::None, @@ -950,7 +1047,7 @@ mod tests { fn output_overflow_restores_an_in_progress_script_at_every_split() { use trusted_server_core::streaming_processor::StreamProcessor as _; - let mut settings = create_test_settings(); + let mut settings = settings_placing_the_module(); settings .insert_module_config( "framework", @@ -966,7 +1063,11 @@ mod tests { let expected = format!("{first}{padding}"); for split in 1..script.len() { - let mut processor = create_html_processor(config_from_settings(&settings, ®istry)); + let mut processor = create_page_processor( + &settings, + ®istry, + config_from_settings(&settings, ®istry), + ); let mut output = processor .process_chunk(first.as_bytes(), false) .expect("should process unresolved RSC group"); diff --git a/crates/framework/nextjs/src/rsc_placeholders.rs b/crates/framework/nextjs/src/rsc_placeholders.rs index 08d81b3932..a79a1474ab 100644 --- a/crates/framework/nextjs/src/rsc_placeholders.rs +++ b/crates/framework/nextjs/src/rsc_placeholders.rs @@ -1,9 +1,8 @@ use std::sync::Arc; -use trusted_server_core::integrations::{ - IntegrationScriptContext, IntegrationScriptRewriter, ScriptRewriteAction, -}; +use trusted_server_core::integrations::ScriptRewriteAction; +use super::NextJsIntegrationConfig; use super::rsc::DEFAULT_MAX_COMBINED_PAYLOAD_BYTES; #[cfg(test)] pub(super) use super::rsc_stream::RSC_PAYLOAD_PLACEHOLDER_PREFIX; @@ -12,10 +11,9 @@ use super::rsc_stream::{ capture_fragment, classify_rsc_group, document_state, rsc_payload_placeholder, }; use super::shared::{ - RSC_RECEIVER_CONTEXT_BYTES, find_rsc_push_payload_range, find_trimmed_rsc_push_payload_range, - receiver_context_is_flight_push, + RSC_RECEIVER_CONTEXT_BYTES, ScriptContext, ScriptRewriter, find_rsc_push_payload_range, + find_trimmed_rsc_push_payload_range, receiver_context_is_flight_push, }; -use super::{NEXTJS_INTEGRATION_ID, NextJsIntegrationConfig}; pub(super) struct NextJsRscPlaceholderRewriter { config: Arc, @@ -169,16 +167,12 @@ impl NextJsRscPlaceholderRewriter { } } -impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { - fn integration_id(&self) -> &'static str { - NEXTJS_INTEGRATION_ID - } - +impl ScriptRewriter for NextJsRscPlaceholderRewriter { fn selector(&self) -> &'static str { "script" } - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { + fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if self.config.rewrite_attributes.is_empty() { return ScriptRewriteAction::keep(); } @@ -344,15 +338,15 @@ fn remember_released(context: &mut String, released: &str) { #[cfg(test)] mod tests { use super::*; + use crate::NEXTJS_INTEGRATION_ID; use crate::rsc_stream::NextJsDocumentState; use trusted_server_core::integrations::IntegrationDocumentState; fn ctx( is_last_in_text_node: bool, document_state: &IntegrationDocumentState, - ) -> IntegrationScriptContext<'_> { - IntegrationScriptContext { - selector: "script", + ) -> ScriptContext<'_> { + ScriptContext { request_host: "proxy.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -732,7 +726,7 @@ mod tests { let state = IntegrationDocumentState::default(); let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); - let context = IntegrationScriptContext { + let context = ScriptContext { max_buffered_script_bytes: budget, ..ctx(true, &state) }; diff --git a/crates/framework/nextjs/src/rsc_stream.rs b/crates/framework/nextjs/src/rsc_stream.rs index c927e70585..b8e77e441d 100644 --- a/crates/framework/nextjs/src/rsc_stream.rs +++ b/crates/framework/nextjs/src/rsc_stream.rs @@ -2,9 +2,8 @@ use std::collections::VecDeque; use std::io; use std::sync::{Arc, Mutex}; -use trusted_server_core::integrations::{ - IntegrationDocumentState, IntegrationHtmlStreamContext, IntegrationHtmlStreamProcessorFactory, -}; +use trusted_server_core::integrations::IntegrationDocumentState; +use trusted_server_core::middleware::MiddlewareContext; use trusted_server_core::streaming_processor::StreamProcessor; use super::rsc::{ @@ -151,24 +150,19 @@ impl NextJsRscStreamProcessorFactory { pub(super) fn new(config: Arc) -> Self { Self { config } } -} - -impl IntegrationHtmlStreamProcessorFactory for NextJsRscStreamProcessorFactory { - fn integration_id(&self) -> &'static str { - NEXTJS_INTEGRATION_ID - } - fn create(&self, context: IntegrationHtmlStreamContext) -> Box { + /// The processor of the one document `context` describes. + pub(super) fn create(&self, context: &MiddlewareContext<'_>) -> Box { let limit = if self.config.max_combined_payload_bytes == 0 { DEFAULT_MAX_COMBINED_PAYLOAD_BYTES } else { self.config.max_combined_payload_bytes }; Box::new(NextJsRscStreamProcessor::new( - document_state(&context.document_state), - context.origin_host, - context.request_host, - context.request_scheme, + document_state(context.document_state), + context.origin_host.to_owned(), + context.request_host.to_owned(), + context.request_scheme.to_owned(), limit, )) } diff --git a/crates/framework/nextjs/src/script_rewriter.rs b/crates/framework/nextjs/src/script_rewriter.rs index 37de438f84..b1d50de3fe 100644 --- a/crates/framework/nextjs/src/script_rewriter.rs +++ b/crates/framework/nextjs/src/script_rewriter.rs @@ -4,13 +4,11 @@ use error_stack::Report; use regex::{Regex, escape}; use trusted_server_core::error::TrustedServerError; -use trusted_server_core::integrations::{ - IntegrationScriptContext, IntegrationScriptRewriter, ScriptRewriteAction, -}; +use trusted_server_core::integrations::ScriptRewriteAction; +use super::NextJsIntegrationConfig; use super::rsc_stream::{FragmentCapture, capture_fragment, document_state}; -use super::shared::strip_origin_host_with_optional_port; -use super::{NEXTJS_INTEGRATION_ID, NextJsIntegrationConfig}; +use super::shared::{ScriptContext, ScriptRewriter, strip_origin_host_with_optional_port}; pub(super) struct NextJsNextDataRewriter { config: Arc, @@ -27,11 +25,7 @@ impl NextJsNextDataRewriter { }) } - fn rewrite_structured( - &self, - content: &str, - ctx: &IntegrationScriptContext<'_>, - ) -> ScriptRewriteAction { + fn rewrite_structured(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if ctx.origin_host.is_empty() || ctx.request_host.is_empty() || self.config.rewrite_attributes.is_empty() @@ -52,16 +46,12 @@ impl NextJsNextDataRewriter { } } -impl IntegrationScriptRewriter for NextJsNextDataRewriter { - fn integration_id(&self) -> &'static str { - NEXTJS_INTEGRATION_ID - } - +impl ScriptRewriter for NextJsNextDataRewriter { fn selector(&self) -> &'static str { "script#__NEXT_DATA__" } - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { + fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if self.config.rewrite_attributes.is_empty() { return ScriptRewriteAction::keep(); } @@ -218,12 +208,8 @@ mod tests { }) } - fn ctx<'a>( - selector: &'static str, - document_state: &'a IntegrationDocumentState, - ) -> IntegrationScriptContext<'a> { - IntegrationScriptContext { - selector, + fn ctx(document_state: &IntegrationDocumentState) -> ScriptContext<'_> { + ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -244,9 +230,9 @@ mod tests { // Document one: cut off before its final fragment arrives. let first_document = IntegrationDocumentState::default(); - let interrupted = IntegrationScriptContext { + let interrupted = ScriptContext { is_last_in_text_node: false, - ..ctx("script#__NEXT_DATA__", &first_document) + ..ctx(&first_document) }; let secret = r#"{"props":{"pageProps":{"sessionToken":"SESSION-ONE-SECRET","href":"#; @@ -259,7 +245,7 @@ mod tests { // Document two: fresh document state, same rewriter. let second_document = IntegrationDocumentState::default(); - let fresh = ctx("script#__NEXT_DATA__", &second_document); + let fresh = ctx(&second_document); let benign = r#"{"props":{"pageProps":{"href":"https://origin.example.com/reviews"}}}"#; let action = rewriter.rewrite(benign, &fresh); @@ -286,7 +272,7 @@ mod tests { let rewriter = NextJsNextDataRewriter::new(test_config()) .expect("should build Next.js structured rewriter"); let document_state = IntegrationDocumentState::default(); - let result = rewriter.rewrite(payload, &ctx("script#__NEXT_DATA__", &document_state)); + let result = rewriter.rewrite(payload, &ctx(&document_state)); match result { ScriptRewriteAction::Replace(value) => { @@ -544,8 +530,7 @@ mod tests { let fragment1 = r#"{"props":{"pageProps":{"href":"https://origin."#; let fragment2 = r#"example.com/reviews"}}}"#; - let ctx_intermediate = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let ctx_intermediate = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -553,7 +538,7 @@ mod tests { max_buffered_script_bytes: 16 * 1024 * 1024, document_state: &document_state, }; - let ctx_last = IntegrationScriptContext { + let ctx_last = ScriptContext { is_last_in_text_node: true, ..ctx_intermediate }; @@ -591,8 +576,7 @@ mod tests { let document_state = IntegrationDocumentState::default(); let payload = r#"{"props":{"pageProps":{"href":"https://origin.example.com/page"}}}"#; - let ctx_single = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let ctx_single = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -622,8 +606,7 @@ mod tests { let fragment1 = r#"{"props":{"pageProps":{"title":"Hello"#; let fragment2 = r#" World","count":42}}}"#; - let ctx_intermediate = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let ctx_intermediate = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -631,7 +614,7 @@ mod tests { max_buffered_script_bytes: 16 * 1024 * 1024, document_state: &document_state, }; - let ctx_last = IntegrationScriptContext { + let ctx_last = ScriptContext { is_last_in_text_node: true, ..ctx_intermediate }; @@ -658,8 +641,7 @@ mod tests { fn fragmented_next_data_releases_suppressed_prefix_on_overflow_and_resets() { let rewriter = NextJsNextDataRewriter::new(test_config()).expect("should build rewriter"); let document_state = IntegrationDocumentState::default(); - let first = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let first = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -681,7 +663,7 @@ mod tests { assert_eq!( rewriter.rewrite( "-tail", - &IntegrationScriptContext { + &ScriptContext { is_last_in_text_node: true, ..first }, @@ -701,8 +683,7 @@ mod tests { let rewriter = NextJsNextDataRewriter::new(test_config()).expect("should build rewriter"); let first_state = IntegrationDocumentState::default(); let second_state = IntegrationDocumentState::default(); - let first_context = IntegrationScriptContext { - selector: "script#__NEXT_DATA__", + let first_context = ScriptContext { request_host: "ts.example.com", request_scheme: "https", origin_host: "origin.example.com", @@ -710,7 +691,7 @@ mod tests { max_buffered_script_bytes: 64, document_state: &first_state, }; - let second_context = IntegrationScriptContext { + let second_context = ScriptContext { document_state: &second_state, ..first_context }; @@ -728,7 +709,7 @@ mod tests { let ScriptRewriteAction::Replace(first_output) = rewriter.rewrite( "done", - &IntegrationScriptContext { + &ScriptContext { is_last_in_text_node: true, ..first_context }, diff --git a/crates/framework/nextjs/src/shared.rs b/crates/framework/nextjs/src/shared.rs index 1ae78a7cab..70d7606eed 100644 --- a/crates/framework/nextjs/src/shared.rs +++ b/crates/framework/nextjs/src/shared.rs @@ -8,6 +8,34 @@ use std::sync::LazyLock; use regex::Regex; use trusted_server_core::host_rewrite::rewrite_bare_host_at_boundaries; +use trusted_server_core::integrations::{IntegrationDocumentState, ScriptRewriteAction}; + +/// What a script rewriter is told about the chunk of text in front of it. +#[derive(Debug)] +pub(crate) struct ScriptContext<'a> { + /// The host the page is served on. + pub(crate) request_host: &'a str, + /// The scheme the page is served over. + pub(crate) request_scheme: &'a str, + /// The host the page was fetched from. + pub(crate) origin_host: &'a str, + /// Whether the chunk is the last of its text node. + pub(crate) is_last_in_text_node: bool, + /// The most script text a rewriter may hold while it waits for the rest. + pub(crate) max_buffered_script_bytes: usize, + /// The state kept for the document the script is in. + pub(crate) document_state: &'a IntegrationDocumentState, +} + +/// A decision about the text of the scripts a selector matches, made a chunk +/// at a time. +pub(crate) trait ScriptRewriter: Send + Sync { + /// The CSS selector of the scripts this rewriter is asked about. + fn selector(&self) -> &'static str; + + /// Decides what happens to one chunk of a script's text. + fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction; +} // These are static code-defined literals, not config-derived patterns, so they // intentionally remain lazy statics instead of participating in diff --git a/crates/tag/google-tag-manager/src/lib.rs b/crates/tag/google-tag-manager/src/lib.rs index add23f1024..55bbabc485 100644 --- a/crates/tag/google-tag-manager/src/lib.rs +++ b/crates/tag/google-tag-manager/src/lib.rs @@ -3807,7 +3807,12 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= ) .expect("should update nextjs config"); - // Next.js first, as a stock build orders the two. + // Both modules change the page, Next.js first. + place_on_every_page( + &mut settings, + MiddlewarePhase::Fetch, + &[trusted_server_framework_nextjs::MODULE, MODULE], + ); let registry = IntegrationRegistry::with_registrations( &settings, &[trusted_server_framework_nextjs::builder(), builder()], @@ -3878,7 +3883,12 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= ) .expect("should update nextjs config"); - // Next.js first, as a stock build orders the two. + // Both modules change the page, Next.js first. + place_on_every_page( + &mut settings, + MiddlewarePhase::Fetch, + &[trusted_server_framework_nextjs::MODULE, MODULE], + ); let registry = IntegrationRegistry::with_registrations( &settings, &[trusted_server_framework_nextjs::builder(), builder()], diff --git a/crates/trusted-server-core/src/test_support.rs b/crates/trusted-server-core/src/test_support.rs index 6671dc3584..14d0372cd3 100644 --- a/crates/trusted-server-core/src/test_support.rs +++ b/crates/trusted-server-core/src/test_support.rs @@ -280,6 +280,12 @@ pub mod nextjs_auction { # fails closed on an unknown jurisdiction. [geo] module = "platform" + + # What Next.js changes in a page is a middleware, which runs on + # the pages an entry names it for. + [[fetch]] + media_type = "text/html" + middleware = ["framework.nextjs"] "#, ) .expect("should parse Next.js auction fixture settings"); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 2a158fc4dc..e61e36f623 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1802,6 +1802,7 @@ takes one entry, and the order written is the order they run in. | `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | | `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | | `ad-tag.google.diagnostics` | `[[serve]]` | For a request that activated diagnostics, writes the bootstrap ahead of the script bundle and the tag that loads the diagnostics module straight after it | +| `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/integrations/nextjs.md b/docs/guide/integrations/nextjs.md index 7536de2d40..cc4b8d8dfe 100644 --- a/docs/guide/integrations/nextjs.md +++ b/docs/guide/integrations/nextjs.md @@ -34,6 +34,17 @@ rewrite_attributes = ["href", "link", "url"] max_combined_payload_bytes = 10485760 ``` +What the integration changes in a page is the middleware `framework.nextjs`, which runs +on the pages a `[[fetch]]` entry names it for: + +```toml +[[fetch]] +media_type = "text/html" +middleware = ["framework.nextjs"] +``` + +See [Placing page changes](/guide/configuration#placing-page-changes). + ### Configuration Options | Field | Type | Default | Description | From d4d5fe1428dc05c0f749cab42e17096f3e1654b7 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 04:52:34 +0100 Subject: [PATCH 23/36] The JavaScript asset proxy changes a page as a middleware Core's JavaScript asset proxy pointed a configured script at its first-party path, or removed one that is blocked, through an attribute hook. The hook ran on every page of a deployment that selected the proxy, and ahead of every module's own. It is now the middleware `js_asset_proxy`, which runs on the pages a `[[fetch]]` entry names it for. Its selector asks about the `src` of `script` elements alone, which the hook checked for itself. Its place among the page changes is the place the entry gives it. A middleware is asked about an address as the one named before it left it, so the proxy is named first where it has to see the address the page wrote. One test names it ahead of a module and another names the module ahead of it, and the guide says to name it first. The tag stand-in that core's tests run beside the proxy becomes a fetch middleware in the same change, because the two were tested together through the order of their hooks. The test that asked the hook about an element that is not a script is removed. The selector decides that now, and the page test that holds an image with a configured address covers it. A test is added for a script tag written in upper case. The recorded page is unchanged. Its settings gain the entry that places the middleware. --- .../trusted-server-core/src/html_processor.rs | 27 +- .../fixtures/js-asset-proxy.settings.toml | 4 + .../src/integrations/js_asset_proxy.rs | 340 +++++++++--------- .../src/integrations/registry.rs | 64 ++-- crates/trusted-server-core/src/publisher.rs | 5 + docs/guide/configuration.md | 7 + 6 files changed, 227 insertions(+), 220 deletions(-) diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 9cc4e957ca..1d849ecdcb 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -1457,12 +1457,12 @@ mod tests { } #[test] - fn integration_head_injector_marks_only_the_bundle_it_asks_to_mark() { + fn a_registration_marks_only_the_bundle_it_asks_to_mark() { use crate::integrations::registry_test_support::tag_fixture as tag; fn process(mark_bundle: Option) -> String { - let integrations = if let Some(mark_bundle) = mark_bundle { - let mut settings = create_test_settings(); + let mut settings = create_test_settings(); + if let Some(mark_bundle) = mark_bundle { settings .insert_module_config( "testing", @@ -1472,13 +1472,16 @@ mod tests { }), ) .expect("should insert the stand-in's settings"); - IntegrationRegistry::new(&settings).expect("should build the registry") - } else { - IntegrationRegistry::empty_for_tests() - }; + test_support::place_on_every_page( + &mut settings, + crate::middleware::MiddlewarePhase::Fetch, + &[tag::MODULE], + ); + } + let registry = IntegrationRegistry::new(&settings).expect("should build the registry"); let mut config = create_test_config(); - config.integrations = integrations; - let mut processor = create_html_processor(config); + config.integrations = registry.clone(); + let mut processor = test_support::create_page_processor(&settings, ®istry, config); let output = processor .process_chunk(b"", true) .expect("should process HTML"); @@ -1500,11 +1503,11 @@ mod tests { let attribute = format!("{}=\"true\"", tag::BUNDLE_ATTRIBUTE); assert!( marked.contains(&attribute), - "should mark the bundle an injector asks to mark" + "should mark the bundle a registration asks to mark" ); assert!( !unmarked.contains(tag::BUNDLE_ATTRIBUTE), - "should leave the bundle unmarked when the injector asks for nothing" + "should leave the bundle unmarked when the registration asks for nothing" ); assert!( !without_the_integration.contains(tag::BUNDLE_ATTRIBUTE), @@ -1519,7 +1522,7 @@ mod tests { .expect("should include the publisher bundle"); assert!( head_insert_index < publisher_bundle_index, - "should keep integration head inserts before the publisher bundle" + "should keep a middleware's head markup before the publisher bundle" ); } diff --git a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml index c582079376..2cedf868f5 100644 --- a/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml +++ b/crates/trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml @@ -7,3 +7,7 @@ modules = ["js_asset_proxy"] path = "/assets/example-vendor-loader.js" origin_url = "https://cdn.example.com/vendor-loader.js" proxy = "enabled" + +[[fetch]] +media_type = "text/html" +middleware = ["js_asset_proxy"] diff --git a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs index 65eee4e0ed..2cd5a87592 100644 --- a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs +++ b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs @@ -6,6 +6,7 @@ //! proxying, or block matching script tags from publisher HTML. use std::collections::HashSet; +use std::rc::Rc; use std::sync::Arc; use async_trait::async_trait; @@ -22,8 +23,11 @@ use crate::constants::{ }; use crate::error::TrustedServerError; use crate::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, + AttributeRewriteAction, IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, +}; +use crate::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, }; use crate::platform::RuntimeServices; use crate::proxy::{ProxyRequestConfig, proxy_request}; @@ -278,14 +282,32 @@ impl JsAssetProxyIntegration { fn asset_for_script_src( &self, script_src: &str, - ctx: &IntegrationAttributeContext<'_>, + request_scheme: &str, ) -> Option<&JsAssetProxyAsset> { self.asset_for_origin_url(script_src).or_else(|| { - let normalized_src = normalize_script_src(script_src, ctx.request_scheme)?; + let normalized_src = normalize_script_src(script_src, request_scheme)?; self.asset_for_origin_url(&normalized_src) }) } + /// What happens to a script whose address is `script_src`, on a page + /// served over `request_scheme`. + fn rewrite_script_address( + &self, + script_src: &str, + request_scheme: &str, + ) -> AttributeRewriteAction { + let Some(asset) = self.asset_for_script_src(script_src, request_scheme) else { + return AttributeRewriteAction::keep(); + }; + + match asset.proxy { + JsAssetProxyMode::Enabled => AttributeRewriteAction::replace(asset.path.clone()), + JsAssetProxyMode::Disabled => AttributeRewriteAction::keep(), + JsAssetProxyMode::Blocked => AttributeRewriteAction::remove_element(), + } + } + fn build_proxy_config<'a>( origin_url: &'a str, req: &Request, @@ -492,7 +514,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(JS_ASSET_PROXY_INTEGRATION_ID) .with_proxy(integration.clone()) - .with_attribute_rewriter(integration) + .with_middleware(Arc::new(ScriptAddress(integration))) .build(), )) } @@ -566,33 +588,36 @@ impl IntegrationProxy for JsAssetProxyIntegration { } } -impl IntegrationAttributeRewriter for JsAssetProxyIntegration { - fn integration_id(&self) -> &'static str { - JS_ASSET_PROXY_INTEGRATION_ID +/// Points a configured script at its first-party path, or removes one that +/// is blocked, on the pages a `[[fetch]]` entry names [`MODULE`] for. +/// +/// It is asked about a script's address as the middleware named before it +/// left it, so it is named first where it has to see the address the page +/// wrote. +struct ScriptAddress(Arc); + +impl Middleware for ScriptAddress { + fn middleware_id(&self) -> &'static str { + MODULE } - fn handles_attribute(&self, attribute: &str) -> bool { - attribute == "src" + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - attr_name: &str, - attr_value: &str, - ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if attr_name != "src" || !ctx.element_name.eq_ignore_ascii_case("script") { - return AttributeRewriteAction::keep(); - } - - let Some(asset) = self.asset_for_script_src(attr_value, ctx) else { - return AttributeRewriteAction::keep(); - }; - - match asset.proxy { - JsAssetProxyMode::Enabled => AttributeRewriteAction::replace(asset.path.clone()), - JsAssetProxyMode::Disabled => AttributeRewriteAction::keep(), - JsAssetProxyMode::Blocked => AttributeRewriteAction::remove_element(), + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + let integration = Arc::clone(&self.0); + let request_scheme = context.request_scheme.to_owned(); + let decide: Rc = Rc::new(move |matched| { + integration.rewrite_script_address(matched.value, &request_scheme) + }); + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::matching( + "script[src]", + "src", + decide, + ))], + ..MiddlewareAction::pass() } } } @@ -604,10 +629,13 @@ mod tests { use std::sync::Arc; use crate::constants::{HEADER_REFERER, HEADER_X_FORWARDED_FOR, HEADER_X_TS_EC}; - use crate::html_processor::{BodyCloseInjection, HtmlProcessorConfig, create_html_processor}; - use crate::integrations::{ - AttributeRewriteAction, IntegrationAttributeRewriter, IntegrationRegistry, + use crate::html_processor::test_support::{place_on_every_page, processed_page}; + use crate::html_processor::{ + BodyCloseInjection, HtmlProcessorConfig, create_html_processor_with_middleware, }; + use crate::integrations::registry_test_support::tag_fixture as tag; + use crate::integrations::{AttributeRewriteAction, IntegrationRegistry}; + use crate::middleware::MiddlewareChain; use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use crate::test_support::tests::create_test_settings; @@ -638,41 +666,31 @@ mod tests { } } - fn rewrite_context() -> IntegrationAttributeContext<'static> { - IntegrationAttributeContext { - attribute_name: "src", - element_name: "script", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - } - } - + /// The page a reader of `html` receives when `integration`'s middleware + /// alone runs on it. fn process_html_with_integration( html: &str, integration: Arc, ) -> String { - let rewriter: Arc = integration; - process_html_with_registry( - html, - IntegrationRegistry::from_rewriters(vec![rewriter], Vec::new()), - ) - } - - fn process_html_with_registry(html: &str, integrations: IntegrationRegistry) -> String { - let processor = create_html_processor(HtmlProcessorConfig { + let chain = MiddlewareChain::new( + MiddlewarePhase::Fetch, + vec![Arc::new(ScriptAddress(integration))], + ); + let config = HtmlProcessorConfig { csp_nonce_observed: None, body_close: BodyCloseInjection::None, origin_host: "origin.example.com".to_string(), request_host: "publisher.example.com".to_string(), request_scheme: "https".to_string(), - integrations, + integrations: IntegrationRegistry::default(), ad_slots_script: None, ad_bids_state: Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, request_state: crate::integrations::IntegrationRequestState::default(), permissions_script: None, - }); + }; + let processor = create_html_processor_with_middleware(config, &chain, None) + .expect("should plan the asset proxy's middleware"); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, @@ -687,6 +705,35 @@ mod tests { String::from_utf8(output).expect("should produce UTF-8 HTML") } + /// The page a reader of `html` receives from a deployment running + /// `settings`. + fn process_html_with_settings(html: &str, settings: &Settings) -> String { + let registry = IntegrationRegistry::new(settings).expect("should build registry"); + processed_page(settings, ®istry, html, 8192) + } + + /// Settings that run the tag stand-in and proxy its script in `mode`, + /// with the two middleware named in the order given. + fn settings_proxying_the_tag(mode: &str, middleware: &[&str]) -> Settings { + let mut settings = create_test_settings(); + settings.select_module("testing", tag::MODULE); + settings + .insert_module_config( + "proxy", + "js_asset_proxy", + &json!({ + "assets": [{ + "path": "/assets/sdk.js", + "origin_url": tag::SCRIPT_URL, + "proxy": mode + }] + }), + ) + .expect("should insert JS asset proxy config"); + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, middleware); + settings + } + #[test] fn an_integration_that_is_not_named_registers_no_routes() { let settings = create_test_settings(); @@ -741,17 +788,16 @@ mod tests { assert_eq!(routes[0].method, Method::GET); assert_eq!(routes[0].path, "/assets/enabled.js"); - let ctx = rewrite_context(); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/enabled.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/enabled.js", "https"), AttributeRewriteAction::Replace(ref value) if value == "/assets/enabled.js" )); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/disabled.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/disabled.js", "https"), AttributeRewriteAction::Keep )); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/blocked.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/blocked.js", "https"), AttributeRewriteAction::RemoveElement )); } @@ -763,46 +809,13 @@ mod tests { "https://cdn.example.com/vendor.js", JsAssetProxyMode::Enabled, )])); - let ctx = rewrite_context(); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/vendor.js?v=1", &ctx), + integration.rewrite_script_address("https://cdn.example.com/vendor.js?v=1", "https"), AttributeRewriteAction::Keep )); assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/other.js", &ctx), - AttributeRewriteAction::Keep - )); - } - - #[test] - fn non_script_src_matches_are_not_rewritten_or_blocked() { - let integration = JsAssetProxyIntegration::new(config_with_assets(vec![ - asset( - "/assets/enabled.js", - "https://cdn.example.com/enabled.js", - JsAssetProxyMode::Enabled, - ), - asset( - "/assets/blocked.js", - "https://cdn.example.com/blocked.js", - JsAssetProxyMode::Blocked, - ), - ])); - let ctx = IntegrationAttributeContext { - attribute_name: "src", - element_name: "img", - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - }; - - assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/enabled.js", &ctx), - AttributeRewriteAction::Keep - )); - assert!(matches!( - integration.rewrite("src", "https://cdn.example.com/blocked.js", &ctx), + integration.rewrite_script_address("https://cdn.example.com/other.js", "https"), AttributeRewriteAction::Keep )); } @@ -836,6 +849,25 @@ mod tests { assert!(processed.contains(r#""#)); } + #[test] + fn a_script_tag_written_in_upper_case_is_a_script() { + let integration = JsAssetProxyIntegration::new(config_with_assets(vec![asset( + "/assets/enabled.js", + "https://cdn.example.com/enabled.js", + JsAssetProxyMode::Enabled, + )])); + + let processed = process_html_with_integration( + r#""#, + integration, + ); + + assert!( + processed.contains(r#"SRC="/assets/enabled.js""#), + "should rewrite a script whatever case its tag is written in: {processed}" + ); + } + #[test] fn script_src_matching_normalizes_common_browser_url_forms() { let integration = JsAssetProxyIntegration::new(config_with_assets(vec![asset( @@ -843,7 +875,6 @@ mod tests { "https://cdn.example.com/vendor.js", JsAssetProxyMode::Enabled, )])); - let ctx = rewrite_context(); for script_src in [ "//cdn.example.com/vendor.js", @@ -852,7 +883,7 @@ mod tests { ] { assert!( matches!( - integration.rewrite("src", script_src, &ctx), + integration.rewrite_script_address(script_src, "https"), AttributeRewriteAction::Replace(ref value) if value == "/assets/vendor.js" ), "script src {script_src} should normalize to the configured origin URL" @@ -861,119 +892,84 @@ mod tests { } #[test] - fn js_asset_proxy_rewriter_takes_precedence_over_native_rewriters() { - let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::tag_fixture::MODULE, - ); - settings - .insert_module_config( - "proxy", - "js_asset_proxy", - &json!({ - "assets": [{ - "path": "/assets/sdk.js", - "origin_url": crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL, - "proxy": "enabled" - }] - }), - ) - .expect("should insert JS asset proxy config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); + fn the_asset_proxy_named_first_rewrites_a_script_ahead_of_a_module() { + let settings = settings_proxying_the_tag("enabled", &[MODULE, tag::MODULE]); let html = format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL + tag::SCRIPT_URL ); - let processed = process_html_with_registry(&html, registry); + let processed = process_html_with_settings(&html, &settings); assert!( processed.contains(r#""#), - "JS asset proxy should rewrite before the integration's own rewriter: {processed}" + "JS asset proxy should rewrite before the module named after it: {processed}" ); assert!( - !processed.contains( - crate::integrations::registry_test_support::tag_fixture::FIRST_PARTY_SCRIPT - ), - "the integration's own rewrite should not override JS asset proxy" + !processed.contains(tag::FIRST_PARTY_SCRIPT), + "the module's own rewrite should not override JS asset proxy" ); } #[test] - fn js_asset_proxy_blocking_takes_precedence_over_native_rewriters() { - let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::tag_fixture::MODULE, + fn a_module_named_ahead_of_the_asset_proxy_rewrites_the_script_first() { + let settings = settings_proxying_the_tag("enabled", &[tag::MODULE, MODULE]); + let html = format!( + r#""#, + tag::SCRIPT_URL ); - settings - .insert_module_config( - "proxy", - "js_asset_proxy", - &json!({ - "assets": [{ - "path": "/assets/sdk.js", - "origin_url": crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL, - "proxy": "blocked" - }] - }), - ) - .expect("should insert JS asset proxy config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); + + let processed = process_html_with_settings(&html, &settings); + + assert!( + processed.contains(&format!( + r#""#, + tag::FIRST_PARTY_SCRIPT + )), + "the module named first should rewrite the script: {processed}" + ); + assert!( + !processed.contains("/assets/sdk.js"), + "JS asset proxy should not know an address the module has already moved" + ); + } + + #[test] + fn the_asset_proxy_named_first_blocks_a_script_ahead_of_a_module() { + let settings = settings_proxying_the_tag("blocked", &[MODULE, tag::MODULE]); let html = format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL + tag::SCRIPT_URL ); - let processed = process_html_with_registry(&html, registry); + let processed = process_html_with_settings(&html, &settings); assert!( !processed.contains("vendor.cmd"), - "blocked JS asset should remove the script element before the integration can rewrite it" + "blocked JS asset should remove the script element before the module can rewrite it" ); assert!( - !processed.contains( - crate::integrations::registry_test_support::tag_fixture::FIRST_PARTY_SCRIPT - ), - "the integration's own rewrite should not keep a blocked script" + !processed.contains(tag::FIRST_PARTY_SCRIPT), + "the module's own rewrite should not keep a blocked script" ); } #[test] fn disabled_js_asset_proxy_candidate_allows_native_rewriters() { - let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::tag_fixture::MODULE, - ); - settings - .insert_module_config( - "proxy", - "js_asset_proxy", - &json!({ - "assets": [{ - "path": "/assets/sdk.js", - "origin_url": crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL, - "proxy": "disabled" - }] - }), - ) - .expect("should insert JS asset proxy config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); + let settings = settings_proxying_the_tag("disabled", &[MODULE, tag::MODULE]); let html = format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::SCRIPT_URL + tag::SCRIPT_URL ); - let processed = process_html_with_registry(&html, registry); + let processed = process_html_with_settings(&html, &settings); assert!( processed.contains(&format!( r#""#, - crate::integrations::registry_test_support::tag_fixture::FIRST_PARTY_SCRIPT + tag::FIRST_PARTY_SCRIPT )), - "disabled JS asset proxy entries should not suppress an integration's own rewrite" + "disabled JS asset proxy entries should not suppress a module's own rewrite" ); } @@ -1322,10 +1318,10 @@ mod tests { }), ) .expect("should insert integration config"); - let registry = IntegrationRegistry::new(&settings).expect("should build registry"); - let processed = process_html_with_registry( + place_on_every_page(&mut settings, MiddlewarePhase::Fetch, &[MODULE]); + let processed = process_html_with_settings( r#""#, - registry, + &settings, ); assert!(processed.contains(r#""#)); diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 9fe5c4de02..8c8dc35629 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -3347,21 +3347,23 @@ pub(crate) mod test_support { /// A stand-in for an integration that tags a page, for core's own tests /// of the HTML processor and the JavaScript asset proxy. /// - /// Selected, it inserts one script at the start of `` and rewrites - /// the address of its own script to a first-party path. With - /// `mark_bundle` set it also asks for an attribute on the publisher - /// bundle tag. + /// Selected, it supplies one fetch middleware under its own name, which + /// writes one script in `` ahead of the bundle and rewrites the + /// address of its own script to a first-party path. With `mark_bundle` + /// set it also asks for an attribute on the publisher bundle tag. pub(crate) mod tag_fixture { + use std::rc::Rc; use std::sync::Arc; use error_stack::Report; use crate::error::TrustedServerError; - use crate::integrations::registry::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationHeadInjector, IntegrationHtmlContext, IntegrationRegistration, - }; + use crate::integrations::registry::{AttributeRewriteAction, IntegrationRegistration}; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, MiddlewareContext, + MiddlewarePhase, + }; use crate::settings::Settings; /// The integration id the stand-in registers under. @@ -3392,35 +3394,27 @@ pub(crate) mod test_support { struct Tag; - impl IntegrationHeadInjector for Tag { - fn integration_id(&self) -> &'static str { - ID - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - vec![format!("")] - } - } - - impl IntegrationAttributeRewriter for Tag { - fn integration_id(&self) -> &'static str { - ID + impl Middleware for Tag { + fn middleware_id(&self) -> &'static str { + MODULE } - fn handles_attribute(&self, attribute: &str) -> bool { - attribute == "src" + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if attr_value == SCRIPT_URL { - AttributeRewriteAction::Replace(FIRST_PARTY_SCRIPT.to_owned()) - } else { - AttributeRewriteAction::Keep + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let decide: Rc = Rc::new(|matched| { + if matched.value == SCRIPT_URL { + AttributeRewriteAction::Replace(FIRST_PARTY_SCRIPT.to_owned()) + } else { + AttributeRewriteAction::Keep + } + }); + MiddlewareAction { + head_inserts: vec![format!("")], + element_handlers: vec![Box::new(AttributeRewrite::new("src", decide))], + ..MiddlewareAction::pass() } } } @@ -3431,11 +3425,9 @@ pub(crate) mod test_support { let Some(config) = settings.module_config::(MODULE)? else { return Ok(None); }; - let tag = Arc::new(Tag); let mut registration = IntegrationRegistration::builder(ID) .without_js() - .with_attribute_rewriter(tag.clone()) - .with_head_injector(tag); + .with_middleware(Arc::new(Tag)); if config.mark_bundle { registration = registration.with_bundle_tag_attribute(BUNDLE_ATTRIBUTE, "true"); } diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 412f9d6718..35eb124438 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -22429,6 +22429,11 @@ mod tests { }), ) .expect("should insert the stand-in's settings"); + crate::html_processor::test_support::place_on_every_page( + &mut settings, + crate::middleware::MiddlewarePhase::Fetch, + &[tag::MODULE], + ); let body = streaming_finalize_response_with_settings( html_stream_params("", None), diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index e61e36f623..67fd258eca 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1803,6 +1803,7 @@ takes one entry, and the order written is the order they run in. | `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | | `ad-tag.google.diagnostics` | `[[serve]]` | For a request that activated diagnostics, writes the bootstrap ahead of the script bundle and the tag that loads the diagnostics module straight after it | | `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | +| `js_asset_proxy` | `[[fetch]]` | Points each configured script at its first-party path and removes one that is blocked. It is asked about an address as the middleware named before it left it, so name it first | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type @@ -1937,6 +1938,12 @@ Each `[[proxy.js_asset_proxy.assets]]` entry: | `proxy` | String | `enabled` | `enabled`, `disabled`, or `blocked` (removes script tags) | | `cache_ttl_seconds` | Integer | None | Optional per-asset downstream cache TTL override | +Rewriting and removing script tags is the middleware `js_asset_proxy`, which +runs on the pages a `[[fetch]]` entry names it for. It knows a script by the +address it is handed, which is the address as the middleware named before it +left it, so name it first in the entry. See +[Placing page changes](#placing-page-changes). + ### lockr Integration **Section**: `[identity.lockr]` From 0b6c1e319ed4481a370b0842d417de773f150af7 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 05:15:09 +0100 Subject: [PATCH 24/36] Core's deferred and payload stand-ins change a page as middleware Core's tests stand two modules in for a vendor's. One carries a deferred browser module and writes its settings into the head, and the other rewrites script payloads in two passes. Both made their page change through the hooks. Each now supplies one fetch middleware under its own name, and a test that runs one on a page places it with an entry. The payload stand-in kept its captured payloads and its held script text in the document's state, because one rewriter served every document. Its text handler and its stream processor are now made for one document, and share what the handler captures between the two of them. Two tests checked that the payload stand-in had registered a stream processor before going on. They check that its middleware runs on the page. The test that two configurations of a module never share a template now also checks that each page carries the markup its own configuration writes. Nothing looked at what the deferred stand-in wrote before. --- .../src/integrations/registry.rs | 172 ++++++++++-------- crates/trusted-server-core/src/publisher.rs | 70 ++++--- 2 files changed, 146 insertions(+), 96 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 8c8dc35629..65826cb4f9 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -3035,19 +3035,18 @@ pub(crate) mod test_support { /// core's own tests of how modules are divided between the bundle and /// deferred tags, and of how a module's settings reach a page template. /// - /// Selected, it carries a deferred browser module and writes its two - /// settings into ``. + /// Selected, it carries a deferred browser module and supplies one fetch + /// middleware under its own name, which writes its two settings into + /// ``. pub(crate) mod deferred_fixture { use std::sync::Arc; use error_stack::Report; use crate::error::TrustedServerError; - use crate::integrations::registry::{ - CarriedJsModule, IntegrationHeadInjector, IntegrationHtmlContext, - IntegrationRegistration, - }; + use crate::integrations::registry::{CarriedJsModule, IntegrationRegistration}; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase}; use crate::settings::Settings; /// The integration id the stand-in registers under. @@ -3081,22 +3080,39 @@ pub(crate) mod test_support { timeout_ms: u32, } - impl IntegrationHeadInjector for Head { - fn integration_id(&self) -> &'static str { - ID + impl Middleware for Head { + fn middleware_id(&self) -> &'static str { + MODULE + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { let config = serde_json::json!({ "label": self.label, "timeoutMs": self.timeout_ms, }); - vec![format!( - "" - )] + MiddlewareAction { + head_inserts: vec![format!( + "" + )], + ..MiddlewareAction::pass() + } } } + /// Has `settings` run the stand-in's middleware, and no other, on + /// every page. + pub(crate) fn place(settings: &mut Settings) { + crate::html_processor::test_support::place_on_every_page( + settings, + MiddlewarePhase::Fetch, + &[MODULE], + ); + } + fn register( settings: &Settings, ) -> Result, Report> { @@ -3110,7 +3126,7 @@ pub(crate) mod test_support { sha256: JS_SHA256, }) .with_deferred_js() - .with_head_injector(Arc::new(Head { + .with_middleware(Arc::new(Head { label: config.label, timeout_ms: config.timeout_ms, })) @@ -3444,13 +3460,14 @@ pub(crate) mod test_support { /// A stand-in for an integration that rewrites script payloads in two /// passes, for core's own tests of the page pipeline. /// - /// The script rewriter swaps the payload of each `fixture_payload("...")` - /// call for a placeholder that carries a namespace made per document, and - /// keeps the payload in the document state. The stream processor swaps - /// each placeholder back for its payload with the origin host rewritten. - /// A payload pushed with `fixture_payload_open` leaves its group - /// unresolved, and the processor holds its output from that placeholder - /// on until a `fixture_payload_close` arrives. + /// Selected, it supplies one fetch middleware under its own name. Its + /// text handler swaps the payload of each `fixture_payload("...")` call + /// for a placeholder that carries a namespace made per document, and + /// keeps the payload. Its stream processor swaps each placeholder back + /// for its payload with the origin host rewritten. A payload pushed with + /// `fixture_payload_open` leaves its group unresolved, and the processor + /// holds its output from that placeholder on until a + /// `fixture_payload_close` arrives. pub(crate) mod payload_fixture { use std::io; use std::sync::{Arc, Mutex, PoisonError}; @@ -3458,12 +3475,11 @@ pub(crate) mod test_support { use error_stack::Report; use crate::error::TrustedServerError; - use crate::integrations::registry::{ - IntegrationHtmlStreamContext, IntegrationHtmlStreamProcessorFactory, - IntegrationRegistration, IntegrationScriptContext, IntegrationScriptRewriter, - ScriptRewriteAction, ScriptTextAccumulator, - }; + use crate::integrations::registry::{IntegrationRegistration, ScriptRewriteAction}; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, TextHandler, + }; use crate::settings::Settings; use crate::streaming_processor::StreamProcessor; @@ -3488,7 +3504,7 @@ pub(crate) mod test_support { unresolved: bool, } - /// What one document's script rewriter has captured so far. + /// What one document's text handler has captured so far. struct Captured { namespace: String, payloads: Vec, @@ -3503,18 +3519,16 @@ pub(crate) mod test_support { } } - fn captured( - state: &crate::integrations::registry::IntegrationDocumentState, - ) -> Arc> { - state.get_or_insert_with(ID, || Mutex::new(Captured::default())) + /// The scripts of one document, each held until its last chunk. + struct Scripts { + captured: Arc>, + held: String, } - struct ScriptRewriter; - - impl ScriptRewriter { + impl Scripts { /// Swaps the payload of a whole script for a placeholder, or /// leaves a script that pushes no payload as it is. - fn rewrite_whole(script: &str, ctx: &IntegrationScriptContext<'_>) -> Option { + fn rewrite_whole(&self, script: &str) -> Option { let (call, opens, closes) = [ ("fixture_payload_open(\"", true, false), ("fixture_payload_close(\"", false, true), @@ -3528,8 +3542,7 @@ pub(crate) mod test_support { return None; } - let shared = captured(ctx.document_state); - let mut captured = shared.lock().unwrap_or_else(PoisonError::into_inner); + let mut captured = self.captured.lock().unwrap_or_else(PoisonError::into_inner); let placeholder = format!( "{PLACEHOLDER_PREFIX}{}_{}{PLACEHOLDER_END}", captured.namespace, @@ -3552,55 +3565,61 @@ pub(crate) mod test_support { } } - impl IntegrationScriptRewriter for ScriptRewriter { - fn integration_id(&self) -> &'static str { - ID - } - - fn selector(&self) -> &'static str { + impl TextHandler for Scripts { + fn selector(&self) -> &str { "script" } - fn rewrite( - &self, - content: &str, - ctx: &IntegrationScriptContext<'_>, - ) -> ScriptRewriteAction { - let accumulator = ctx - .document_state - .get_or_insert_with(ID, ScriptTextAccumulator::default); - let mut buffer = accumulator.buffer(); - let claimed = !buffer.is_empty() || content.contains("fixture_payload"); + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + let claimed = !self.held.is_empty() || text.contains("fixture_payload"); if !claimed { return ScriptRewriteAction::Keep; } - buffer.push_str(content); - if !ctx.is_last_in_text_node { + self.held.push_str(text); + if !is_last { return ScriptRewriteAction::RemoveNode; } - let script = std::mem::take(&mut *buffer); - let rewritten = Self::rewrite_whole(&script, ctx).unwrap_or(script); + let script = std::mem::take(&mut self.held); + let rewritten = self.rewrite_whole(&script).unwrap_or(script); ScriptRewriteAction::replace(rewritten) } } - struct StreamFactory; + struct Payloads; - impl IntegrationHtmlStreamProcessorFactory for StreamFactory { - fn integration_id(&self) -> &'static str { - ID + impl Middleware for Payloads { + fn middleware_id(&self) -> &'static str { + MODULE } - fn create(&self, context: IntegrationHtmlStreamContext) -> Box { - Box::new(Processor { - context, - held: Vec::new(), - }) + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + /// The text handler and the stream processor of one document + /// share what the handler captures. + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + let captured = Arc::new(Mutex::new(Captured::default())); + MiddlewareAction { + text_handlers: vec![Box::new(Scripts { + captured: Arc::clone(&captured), + held: String::new(), + })], + stream: Some(Box::new(Processor { + captured, + origin_host: context.origin_host.to_owned(), + request_host: context.request_host.to_owned(), + held: Vec::new(), + })), + ..MiddlewareAction::pass() + } } } struct Processor { - context: IntegrationHtmlStreamContext, + captured: Arc>, + origin_host: String, + request_host: String, /// Output not yet released, because it ends inside a placeholder /// or starts at one whose group is unresolved. held: Vec, @@ -3617,8 +3636,7 @@ pub(crate) mod test_support { impl StreamProcessor for Processor { fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> Result, io::Error> { self.held.extend_from_slice(chunk); - let shared = captured(&self.context.document_state); - let captured = shared.lock().unwrap_or_else(PoisonError::into_inner); + let captured = self.captured.lock().unwrap_or_else(PoisonError::into_inner); let prefix = PLACEHOLDER_PREFIX.as_bytes(); let mut out = Vec::with_capacity(self.held.len()); let mut at = 0; @@ -3652,7 +3670,7 @@ pub(crate) mod test_support { out.extend_from_slice( payload .original - .replace(&self.context.origin_host, &self.context.request_host) + .replace(&self.origin_host, &self.request_host) .as_bytes(), ); at = end; @@ -3686,6 +3704,17 @@ pub(crate) mod test_support { } } + /// Has `settings` select the stand-in and run its middleware, and no + /// other, on every page. + pub(crate) fn select_and_place(settings: &mut Settings) { + settings.select_module("testing", MODULE); + crate::html_processor::test_support::place_on_every_page( + settings, + MiddlewarePhase::Fetch, + &[MODULE], + ); + } + fn register( settings: &Settings, ) -> Result, Report> { @@ -3694,8 +3723,7 @@ pub(crate) mod test_support { } Ok(Some( IntegrationRegistration::builder(ID) - .with_script_rewriter(Arc::new(ScriptRewriter)) - .with_html_stream_processor(Arc::new(StreamFactory)) + .with_middleware(Arc::new(Payloads)) .build(), )) } diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 35eb124438..61116708f7 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -10023,8 +10023,8 @@ mod tests { #[test] fn reconfiguring_an_integration_changes_the_fingerprint() { - // Config reaches the template directly: an integration's head insert - // carries its settings into bytes shared between readers. + // Config reaches the template directly: a middleware's head markup + // carries its module's settings into bytes shared between readers. assert_ne!( fingerprint(&settings_with_integration(true, 1000)), fingerprint(&settings_with_integration(true, 2500)), @@ -13687,14 +13687,20 @@ mod tests { async fn two_pass_integration_templates_are_identical_across_request_shapes() { let mut settings = settings_with_mode("esi"); settings.publisher.origin_url = "https://origin.example.com".to_owned(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::new(&settings).expect("should create registry"); assert!( - !registry.html_stream_processor_factories().is_empty(), - "should exercise the stand-in's stream processor" + !registry + .middleware_chain( + &settings.fetch, + crate::middleware::MiddlewarePhase::Fetch, + crate::middleware::HTML_MEDIA_TYPE, + "/", + ) + .is_empty(), + "should exercise the stand-in's middleware, which supplies a stream processor" ); let settings = Arc::new(settings); let html = br#"
"#; @@ -14086,6 +14092,7 @@ mod tests { }), ) .expect("should insert the stand-in's table"); + crate::integrations::registry_test_support::deferred_fixture::place(&mut settings); settings } @@ -14128,9 +14135,20 @@ mod tests { queue_shareable_html(&stub); queue_shareable_html(&stub); - let _ = run(&first, &services, navigation_request()).await; - let _ = run(&second, &services, navigation_request()).await; + let first_page = String::from_utf8( + body_of(run(&first, &services, navigation_request()).await).await, + ) + .expect("should serve the first configuration's page as UTF-8"); + let second_page = String::from_utf8( + body_of(run(&second, &services, navigation_request()).await).await, + ) + .expect("should serve the second configuration's page as UTF-8"); + assert!( + first_page.contains("\"timeoutMs\":1000") + && second_page.contains("\"timeoutMs\":2500"), + "each page should carry the markup its own configuration writes" + ); let stored = stored_cache_keys(&cache); assert_eq!( stored.len(), @@ -22501,9 +22519,8 @@ mod tests { // request token emitted by lol_html at the structural end is a seam. let page = br#"
still streaming
"#; let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let params = html_stream_params( "", @@ -22678,9 +22695,8 @@ mod tests { settings.auction.enabled = true; settings.auction.provider_names = vec!["seam_test".to_owned()]; settings.auction.timeout_ms = 60_000; - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let client = Arc::new(GatedAuctionHttpClient { inner: StubHttpClient::new(), @@ -23489,11 +23505,11 @@ mod tests { /// routes through `Stream`, and the shared processor pipeline applies it. #[test] fn streaming_html_with_stream_processors_rewrites_body() { - // Select the stand-in so a stream processor is registered. + // Select the stand-in and place its middleware, so the page has a + // stream processor. let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::with_plan( @@ -23506,8 +23522,15 @@ mod tests { .expect("should create integration registry"); assert!( - !registry.html_stream_processor_factories().is_empty(), - "the stand-in must register an HTML stream processor" + !registry + .middleware_chain( + &settings.fetch, + crate::middleware::MiddlewarePhase::Fetch, + crate::middleware::HTML_MEDIA_TYPE, + "/", + ) + .is_empty(), + "the stand-in's middleware, which supplies an HTML stream processor, must run on the page" ); assert_eq!( classify_response_route( @@ -23572,9 +23595,8 @@ mod tests { #[test] fn document_state_placeholders_substitute_through_streaming_path() { let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::payload_fixture::MODULE, + crate::integrations::registry_test_support::payload_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::with_plan( &settings, From 072e3620edd9a97fd44d310900c95d51e9673ec3 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 05:38:07 +0100 Subject: [PATCH 25/36] Core's request stand-in writes into a reader's copy as a serve middleware Core's tests stand one module in for a vendor's that acts on one request. Its preparer or its filter marks the request, and its head injector read the mark and wrote a script ahead of the bundle and its module's tag after it. What it writes is one reader's, which the page path kept out of a shared template by handing a shared mode no request state. It now supplies one serve middleware under its own name, which reads the mark from the state the request left. A serve middleware runs on a reader's copy and nothing it writes is stored, so the same output is kept out of a shared template by where it runs. The tests of a request a module acts on place the middleware with a `[[serve]]` entry. The tests that render whole documents compose the serve entry as the production function does, which runs none on a template every reader shares. They also run a serve middleware that writes for every reader, marked or not, and refuse its output in a shared template. Without it a template that ran the serve entry would still pass, because a shared mode is handed no request state and the stand-in writes nothing without a mark. --- .../trusted-server-core/src/html_processor.rs | 9 +- .../src/integrations/registry.rs | 65 +++++++------ crates/trusted-server-core/src/publisher.rs | 95 ++++++++++++------- 3 files changed, 101 insertions(+), 68 deletions(-) diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 1d849ecdcb..5bef859fb5 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -1532,13 +1532,14 @@ mod tests { let html = "Test"; let mut settings = create_test_settings(); - settings.select_module("testing", request_fixture::MODULE); - let mut config = create_test_config(); - config.integrations = + request_fixture::select_and_place(&mut settings); + let registry = IntegrationRegistry::new(&settings).expect("should build integration registry"); + let mut config = create_test_config(); + config.integrations = registry.clone(); config.request_state = request_fixture::marked(); - let processor = create_html_processor(config); + let processor = test_support::create_page_processor(&settings, ®istry, config); let pipeline_config = PipelineConfig { input_compression: Compression::None, output_compression: Compression::None, diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 65826cb4f9..df93389573 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -2798,10 +2798,10 @@ pub(crate) mod test_support { /// does, the way a module that reserves a cookie does. Selected, its /// request filter leaves the same mark on a request that carries its /// header, which is the route a module that decides in its filter takes. - /// Selected, the - /// stand-in's head injector reads the mark from the document's state and - /// writes one script at the start of `` and the tag of its - /// standalone module after the bundle. Its response finalizer sets a + /// Selected, it supplies one serve middleware under its own name, which + /// reads the mark from the state the request left and writes one script + /// ahead of the bundle and the tag of its standalone module after it, + /// into that reader's copy alone. Its response finalizer sets a /// cookie on the response to a marked request, and its builder declares /// that it reads the auction token. pub(crate) mod request_fixture { @@ -2813,11 +2813,12 @@ pub(crate) mod test_support { use crate::error::TrustedServerError; use crate::integrations::registry::{ - CarriedJsModule, IntegrationHeadInjector, IntegrationHtmlContext, - IntegrationRegistration, IntegrationRequestFilter, IntegrationRequestState, - RequestFilterDecision, RequestFilterEffects, RequestFilterInput, + CarriedJsModule, IntegrationRegistration, IntegrationRequestFilter, + IntegrationRequestState, RequestFilterDecision, RequestFilterEffects, + RequestFilterInput, }; use crate::integrations::{CORE_SOURCE, IntegrationBuilder}; + use crate::middleware::{Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase}; use crate::settings::Settings; use crate::tsjs_bundle::JsModulePart; @@ -2971,40 +2972,46 @@ pub(crate) mod test_support { struct Head; - impl Head { - fn marked(ctx: &IntegrationHtmlContext<'_>) -> bool { - ctx.document_state.get::(ID).is_some() - } - } - - impl IntegrationHeadInjector for Head { - fn integration_id(&self) -> &'static str { - ID + impl Middleware for Head { + fn middleware_id(&self) -> &'static str { + MODULE } - fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - if !Self::marked(ctx) { - return Vec::new(); - } - vec![format!("")] + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Serve] } - fn after_bundle_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec { - if !Self::marked(ctx) { - return Vec::new(); + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + if context.document_state.get::(ID).is_none() { + return MiddlewareAction::pass(); } let module = JsModulePart { id: ID, source: JS, sha256: JS_SHA256, }; - vec![format!( - "", - crate::tsjs::tsjs_single_module_script_src(&module) - )] + MiddlewareAction { + head_inserts: vec![format!("")], + after_bundle_inserts: vec![format!( + "", + crate::tsjs::tsjs_single_module_script_src(&module) + )], + ..MiddlewareAction::pass() + } } } + /// Has `settings` select the stand-in and run its middleware, and no + /// other, on every reader's copy of a page. + pub(crate) fn select_and_place(settings: &mut Settings) { + settings.select_module("testing", MODULE); + crate::html_processor::test_support::place_on_every_page( + settings, + MiddlewarePhase::Serve, + &[MODULE], + ); + } + fn register( settings: &Settings, ) -> Result, Report> { @@ -3018,7 +3025,7 @@ pub(crate) mod test_support { sha256: JS_SHA256, }) .with_standalone_js() - .with_head_injector(Arc::new(Head)) + .with_middleware(Arc::new(Head)) .with_request_filter(Arc::new(Filter)) .build(), )) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 61116708f7..d79bc239d9 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -8017,9 +8017,8 @@ mod tests { &[SCHEDULING_PROVIDER], ); settings.proxy.allowed_domains = vec!["*.example".to_owned(), "*.example.com".to_owned()]; - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); settings } @@ -9399,9 +9398,8 @@ mod tests { #[test] fn stream_publisher_body_writes_what_a_module_left_on_the_request_into_materialized_html() { let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); let integration_registry = IntegrationRegistry::new(&settings).expect("should create integration registry"); @@ -9579,17 +9577,19 @@ mod tests { //! passing code: the decisions were right and nothing checked what the //! composition of them *renders*. //! - //! These tests render whole documents through `create_html_processor`, - //! composing the same three decisions `create_html_stream_processor` uses, - //! and compare bytes. A future request-dependent injection added at either - //! seam fails here even if every decision function is left untouched. + //! These tests render whole documents through the processor + //! `create_html_stream_processor` builds, composing the same decisions it + //! uses, and compare bytes. A future request-dependent injection added at + //! either seam fails here even if every decision function is left untouched. use super::template_neutrality_tests::{settings_with_slots, slot}; use super::*; use crate::creative_opportunities::AssemblyMode; - use crate::html_processor::{HtmlProcessorConfig, create_html_processor}; + use crate::html_processor::{HtmlProcessorConfig, create_html_processor_with_middleware}; use crate::integrations::IntegrationRegistry; + use crate::integrations::registry_test_support::middleware_fixture as fixture; use crate::integrations::registry_test_support::request_fixture; + use crate::middleware::{HTML_MEDIA_TYPE, MiddlewarePhase}; const DOCUMENT: &[u8] = b"t

content

"; @@ -9600,7 +9600,7 @@ mod tests { struct RequestShape { /// Folds in consent, bot classification, prefetch and the kill switch. ad_stack_ran: bool, - /// A module left state on the request for its page hooks. + /// A module left state on the request for its serve middleware. module_state_left: bool, /// A resolved auction, present only when one was dispatched. bids_available: bool, @@ -9612,9 +9612,19 @@ mod tests { let mut settings = settings_with_slots(); settings.proxy.allowed_domains = vec!["*.example".to_string(), "*.example.com".to_string()]; - // The stand-in is what writes a marked request's state into a - // document, so it runs for every shape. + // The request stand-in is what writes a marked request's state + // into a document, so it runs for every shape. The reader + // middleware beside it writes into every reader's copy, marked or + // not, so a template that ran the serve entry would show it. settings.select_module("testing", request_fixture::MODULE); + settings.select_module("testing", fixture::MODULE); + crate::html_processor::test_support::place_on_every_page( + &mut settings, + MiddlewarePhase::Serve, + &[request_fixture::MODULE, fixture::READER], + ); + let registry = IntegrationRegistry::new(&settings) + .expect("should build a registry that runs the stand-in"); let slots = [slot()]; let ad_slots_script = @@ -9639,8 +9649,7 @@ mod tests { origin_host: "origin.example.com".to_string(), request_host: "example.com".to_string(), request_scheme: "https".to_string(), - integrations: IntegrationRegistry::new(&settings) - .expect("should build a registry that runs the stand-in"), + integrations: registry.clone(), permissions_script: template_permissions_script(mode, &permissions_json_fixture()), ad_slots_script, ad_bids_state, @@ -9649,7 +9658,27 @@ mod tests { body_close, }; - let mut processor = create_html_processor(config); + let fetch = registry.middleware_chain( + &settings.fetch, + MiddlewarePhase::Fetch, + HTML_MEDIA_TYPE, + "/", + ); + // A shared mode builds the template every reader shares, which is + // the response `create_html_stream_processor` runs no serve entry + // on. + let shared_template_authorized = matches!(mode, AssemblyMode::Esi); + let serve = (!shared_template_authorized).then(|| { + registry.middleware_chain( + &settings.serve, + MiddlewarePhase::Serve, + HTML_MEDIA_TYPE, + "/", + ) + }); + let mut processor = + create_html_processor_with_middleware(config, &fetch, serve.as_ref()) + .expect("should plan the document's middleware"); let out = processor .process_chunk(DOCUMENT, true) .expect("should process the document"); @@ -9731,6 +9760,7 @@ mod tests { "permissions", request_fixture::HEAD_FLAG, request_fixture::MODULE_FILE, + fixture::READER_SCRIPT, ] { assert!( !rendered.contains(forbidden), @@ -9813,6 +9843,12 @@ mod tests { && !unmarked.contains(request_fixture::MODULE_FILE), "a request no module left state on should get neither:\n{unmarked}" ); + assert!( + rendered.contains(fixture::READER_SCRIPT) + && unmarked.contains(fixture::READER_SCRIPT), + "an inline document is one reader's copy, so the serve entry should run on \ + it whatever the request left:\n{unmarked}" + ); } #[test] @@ -15459,10 +15495,7 @@ mod tests { let stub = Arc::new(StubHttpClient::new()); let cache = Arc::new(MemoryTemplateCache::default()); let mut raw = settings_with_mode("esi"); - raw.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, - ); + crate::integrations::registry_test_support::request_fixture::select_and_place(&mut raw); let settings = Arc::new(raw); let services = services(Arc::clone(&stub), Arc::clone(&cache)); queue_shareable_html(&stub); @@ -15551,10 +15584,7 @@ mod tests { let stub = Arc::new(StubHttpClient::new()); let cache = Arc::new(MemoryTemplateCache::default()); let mut raw = settings_with_mode("esi"); - raw.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, - ); + crate::integrations::registry_test_support::request_fixture::select_and_place(&mut raw); let settings = Arc::new(raw); let services = services(Arc::clone(&stub), Arc::clone(&cache)); queue_shareable_html(&stub); @@ -15621,10 +15651,7 @@ mod tests { .as_mut() .expect("fixture configures creative opportunities") .origin_is_cookie_independent = Some(true); - raw.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, - ); + crate::integrations::registry_test_support::request_fixture::select_and_place(&mut raw); let settings = Arc::new(raw); let services = services(Arc::clone(&stub), Arc::clone(&cache)); queue_shareable_html(&stub); @@ -18493,9 +18520,8 @@ mod tests { async fn inactive_ad_stack_preserves_the_cache_privacy_of_a_request_a_module_acts_on() { // Arrange let mut settings = settings_with_disabled_ad_templates(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); let stub = Arc::new(StubHttpClient::new()); queue_html_response_with_cache_control(&stub, "no-cache"); @@ -19110,9 +19136,8 @@ mod tests { #[tokio::test] async fn what_a_request_filter_leaves_survives_into_the_publisher_html_pipeline() { let mut settings = create_test_settings(); - settings.select_module( - "testing", - crate::integrations::registry_test_support::request_fixture::MODULE, + crate::integrations::registry_test_support::request_fixture::select_and_place( + &mut settings, ); let registry = IntegrationRegistry::new(&settings) .expect("should create integration registry with the stand-in"); From a00b0052ae4e0883b386f40f66d5fba833ec87ca Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 06:01:58 +0100 Subject: [PATCH 26/36] Test the two attributes core rewrites that nothing asserted on Core moves the origin's address in `srcset` and `imagesrcset`, as it does in `href`, `src` and `action`, and no test said so. Taking either rewrite out left every test passing. This test runs both through the document processor, with a bare host among the candidates of the `srcset`, and a `srcset` that names another host, which is left as it is. --- .../trusted-server-core/src/html_processor.rs | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 5bef859fb5..4cace67ad4 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -1629,6 +1629,37 @@ mod tests { assert!(!result.contains("origin.example.com")); } + #[test] + fn the_origin_s_address_is_moved_in_srcset_and_imagesrcset() { + let mut processor = create_html_processor(create_test_config()); + + let output = processor + .process_chunk( + br#""#, + true, + ) + .expect("should process the document"); + let processed = String::from_utf8(output).expect("should stay UTF-8"); + + assert!( + processed.contains( + r#"imagesrcset="https://test.example.com/a.png 1x, https://test.example.com/b.png 2x, //test.example.com/c.png 3x""# + ), + "should move the origin's address in every candidate of an imagesrcset: {processed}" + ); + assert!( + processed.contains( + r#"srcset="https://test.example.com/a.png 1x, https://test.example.com/b.png 2x, //test.example.com/c.png 3x, test.example.com/d.png 4x""# + ), + "should move the origin's address in every candidate of a srcset, a bare host \ + included: {processed}" + ); + assert!( + processed.contains(r#""#), + "should leave a srcset that names another host as it is: {processed}" + ); + } + #[test] fn test_html_processor_config_from_settings() { let settings = create_test_settings(); From 7e039ae73e08e88f5229e60ac38e977665d4c035 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 9 Oct 2026 06:18:31 +0100 Subject: [PATCH 27/36] The four page hook traits are removed Every module that changed a page through a hook now supplies a middleware, core's JavaScript asset proxy and its test stand-ins included, so nothing implements the hooks any longer. Removed from `trusted-server-core`: - `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector` - their context types, `IntegrationAttributeContext`, `IntegrationScriptContext`, `IntegrationHtmlContext` and `IntegrationHtmlStreamContext` - the registration methods `with_attribute_rewriter`, `with_script_rewriter`, `with_html_stream_processor` and `with_head_injector`, and the registry's four lists behind them - `AttributeRewriteOutcome`, which was the registry's answer after asking every attribute rewriter - `ScriptTextAccumulator`, which kept a script's text in the document's state because one rewriter served every document. A text handler is made for one document and keeps its own. The document processor asks nothing of the registry about a page any more. It moves the origin's address in the five attributes it always has, and carries out the plan of the middleware an entry names. A request's state reached a document through the hooks, and a function handed a shared mode none of it, so that nothing made for one request could be stored. That function is removed with its reader. What a request left now reaches a document through the `[[serve]]` entry alone, which does not run on the pass that builds a template every reader shares, and the fetch chain is handed none of it. The tests that render whole documents fail if the serve entry is run on a shared template. Tests of a hook are removed where a middleware test already asserts the same thing. The test that a registration from the auction plan goes ahead of a section's module reads the order from the middleware names the registry lists. The guides describe a middleware as the one way a module changes a page, and the changelog lists what is removed. --- AGENTS.md | 3 +- CHANGELOG.md | 5 +- crates/cmp/osano/src/lib.rs | 4 +- .../trusted-server-core/src/html_processor.rs | 463 ++----------- .../src/integrations/mod.rs | 19 +- .../src/integrations/registry.rs | 642 ++---------------- crates/trusted-server-core/src/middleware.rs | 14 +- .../src/middleware/tests.rs | 109 +-- .../src/platform/template_cache.rs | 2 +- crates/trusted-server-core/src/publisher.rs | 55 +- .../lib/src/integrations/prebid/index.ts | 4 +- docs/guide/api-reference.md | 2 +- docs/guide/configuration.md | 42 +- docs/guide/creative-processing.md | 103 ++- docs/guide/integration-guide.md | 102 +-- docs/guide/integrations-overview.md | 5 +- docs/guide/integrations/gpt.md | 2 +- 17 files changed, 251 insertions(+), 1325 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 477bb1e31e..29bd3b1de3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -430,8 +430,7 @@ Integrations register in Rust via: ```rust IntegrationRegistration::builder(ID) .with_proxy() - .with_attribute_rewriter() - .with_head_injector() + .with_middleware() .build() ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index f0e6727be1..a03e219acc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware, and no longer on every page of a deployment that selects the module. A deployment adds the entries when it upgrades. Until it does, a converted module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. +- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware, and no longer on every page of a deployment that selects the module. A deployment adds the entries when it upgrades. Until it does, a module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. +- **Breaking:** The four page hook traits are removed, being `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector`, with their context types, `ScriptTextAccumulator`, `AttributeRewriteOutcome` and the registration methods that took them. A module that implemented one supplies a `Middleware` instead, whose action carries the same decisions. A browser module's attribute on the script bundle's tag is stated with `with_bundle_tag_attribute`. - A pushed configuration keeps DataDome's two deprecated secret store selectors, `server_side_key_secret_store` and `protection_test_bypass.credential_secret_store`, where they were removed as the settings were read. DataDome still accepts and discards both, so a configuration that carries one loads as before. - Integration modules have moved out of `trusted-server-core` into crates of their own under `crates///`, each named by that folder, so `crates/cmp/osano` is the module `cmp.osano`. The settings that select and configure a module do not change. `crates/trusted-server-modules` lists the modules a stock build ships from crates of their own, in the order their hooks run. Every adapter builds its state and loads its settings with that list, and `ts config validate` and `ts config push` run each stock module's own rules. Code that named a moved module under `trusted_server_core::integrations` names its crate instead, such as `trusted_server_cmp_osano`. Moved: Osano, Lockr, Permutive, Sourcepoint, Didomi, Google Tag Manager, Testlight, Next.js, Google Publisher Tags and its diagnostics, DataDome, the mock ad server, APS, Prebid, the Prebid Server demand and the plain OpenRTB demand. Core keeps the JavaScript asset proxy, which is its own. - The Fastly adapter is a library with a thin binary over it. A deployment that ships a vendor crate writes a binary of its own that calls `trusted_server_adapter_fastly::run_with(vec![...])` with that crate's integration builder, in place of editing the adapter. The builders are composed with the built-in ones, and a builder runs only when the settings select its module. The Wasm artifact keeps its name, `trusted-server-adapter-fastly.wasm`. @@ -68,7 +69,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Page changes as middleware. A module registers a change it makes to a page with `.with_middleware(...)`, and the ordered `[[fetch]]` and `[[serve]]` entries of the settings say which pages each one runs on and in what order. A fetch middleware runs on the page as the origin sent it and what it writes is stored with a shared template. A serve middleware runs on each reader's copy, whether the page came from the store or from the origin, and nothing it writes is stored. - A builder registers its module from the compiled auction plan with `with_plan_registration` and checks its settings against the plan with `with_plan_validator`, so a module whose page support follows what the plan selects is not named in `trusted-server-core`. Prebid and APS register this way. A renderer descriptor carries its builder's statement of which payload key holds the identifier the renderer picks its bid by, made with `BidRenderer::picking_bid_by`, and the page's `hb_adid` is read through that statement. - A module declares the settings in its own table that hold the name of a secret, with `with_secret_settings` on its builder, each with a rule for when the table puts it to use. A setting in use is looked up in the default secret store as the settings load and has to name a key for deploy validation to pass, and one not in use is cleared and never looked up. DataDome's server-side key and its test bypass credential are declared this way, and `trusted-server-core` no longer names them. -- A module can act on one request from start to finish without `trusted-server-core` naming it. A request preparer or a request filter leaves a value in the request's `IntegrationRequestState`, which is copied into the document's state for the module's page hooks and handed to the response finalizer the builder declares with `with_response_finalizer`. A request that carries any value keeps to the origin path, so its document is never read from or stored as a shared template, and its HTML response is sent `private, no-store`. A head injector can also write straight after the script bundle with `after_bundle_inserts`, and a builder declares with `with_auction_token` that its browser script reads the token an auction publishes with its winning bids. The registry runs the request preparers once for a request. +- A module can act on one request from start to finish without `trusted-server-core` naming it. A request preparer or a request filter leaves a value in the request's `IntegrationRequestState`, which the module's serve middleware reads from the state of the reader's copy and which is handed to the response finalizer the builder declares with `with_response_finalizer`. A request that carries any value keeps to the origin path, so its document is never read from or stored as a shared template, and its HTML response is sent `private, no-store`. A middleware can also write straight after the script bundle with `after_bundle_inserts`, and a builder declares with `with_auction_token` that its browser script reads the token an auction publishes with its winning bids. The registry runs the request preparers once for a request. - Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery. - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary. - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape. diff --git a/crates/cmp/osano/src/lib.rs b/crates/cmp/osano/src/lib.rs index ec8fdbc83c..6e236b316e 100644 --- a/crates/cmp/osano/src/lib.rs +++ b/crates/cmp/osano/src/lib.rs @@ -120,8 +120,8 @@ mod tests { "Osano v1 should not register Rust proxy routes" ); assert!( - registration.head_injectors.is_empty(), - "Osano v1 should not inject HTML from Rust" + registration.middleware.is_empty(), + "Osano v1 should not change a page from Rust" ); } diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 4cace67ad4..b7e5d6ff6d 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -14,9 +14,8 @@ use lol_html::{ }; use crate::integrations::{ - AttributeRewriteAction, AttributeRewriteOutcome, IntegrationAttributeContext, - IntegrationDocumentState, IntegrationHtmlContext, IntegrationRegistry, IntegrationRequestState, - IntegrationScriptContext, ScriptRewriteAction, + AttributeRewriteAction, IntegrationDocumentState, IntegrationRegistry, IntegrationRequestState, + ScriptRewriteAction, }; use crate::middleware::{ ElementHandler, MatchedAttribute, MiddlewareChain, MiddlewareContext, MiddlewarePlan, @@ -100,10 +99,10 @@ pub struct HtmlProcessorConfig { /// Maximum bytes an integration may retain while processing one script or /// unresolved streaming group. pub max_buffered_body_bytes: usize, - /// What modules left on the request for their page hooks, copied into the - /// document's state before parsing starts. Empty for a document that may - /// be stored and served to other readers, where nothing made for one - /// request may appear. + /// What modules left on the request, which the serve chain is handed + /// where it runs on the same pass as the fetch chain. The fetch chain is + /// handed none of it, because what that chain writes may be stored and + /// served to other readers. pub request_state: IntegrationRequestState, /// What the `` seam injects. Decided by the caller rather than inferred /// from [`Self::ad_slots_script`]. @@ -180,7 +179,7 @@ impl HtmlProcessorConfig { self } - /// Attach what modules left on the request for their page hooks. + /// Attach what modules left on the request, for the serve chain. #[must_use] pub fn with_request_state(mut self, request_state: IntegrationRequestState) -> Self { self.request_state = request_state; @@ -234,7 +233,8 @@ impl TextEdits { } } -/// Create an HTML processor with URL replacement and integration hooks. +/// Create an HTML processor that moves the origin's address to the +/// publisher's and writes the script bundle, with no middleware. /// /// # Panics /// @@ -257,11 +257,10 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso /// request left, because what that chain writes may be stored and served to /// other readers. /// -/// A fetch middleware's head markup follows the hooks' own. Its element -/// handlers are asked after core has moved the origin's address in an -/// attribute and after the hooks. Its text handlers are handed a chunk as the -/// hooks and the handlers before them left it, and its stream processors run -/// after the hooks' own. +/// A fetch middleware's element handlers are asked after core has moved +/// the origin's address in an attribute. Its text handlers are handed a chunk +/// as the handlers before them left it, and its stream processor is handed +/// what the one before it produced. /// /// # Errors /// @@ -478,10 +477,6 @@ fn build_html_processor( text_handlers: middleware_text_handlers, processors: middleware_processors, } = plan; - let stream_processor_factories = config.integrations.html_stream_processor_factories(); - let document_state = IntegrationDocumentState::default(); - config.request_state.seed(&document_state); - // Simplified URL patterns structure - stores only core data and generates variants on-demand struct UrlPatterns { origin_host: String, @@ -548,7 +543,6 @@ fn build_html_processor( let injected_tsjs = Rc::new(Cell::new(false)); let injected_bids = Arc::new(AtomicBool::new(false)); let integration_registry = config.integrations.clone(); - let script_rewriters = integration_registry.script_rewriters(); let ad_slots_script = config.ad_slots_script.clone(); let permissions_script = config.permissions_script.clone(); let body_close = config.body_close.clone(); @@ -579,8 +573,6 @@ fn build_html_processor( element!("head", { let injected_tsjs = injected_tsjs.clone(); let integrations = integration_registry.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); let ad_slots_script = ad_slots_script.clone(); let permissions_script = permissions_script.clone(); move |el| { @@ -596,17 +588,9 @@ fn build_html_processor( if let Some(ref slots_script) = ad_slots_script { snippet.push_str(slots_script); } - let ctx = IntegrationHtmlContext { - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - document_state: &document_state, - }; - // First inject integration-specific config (e.g., window.__tsjs_prebid) - // so it's available when the bundle's auto-init code reads it. - for insert in integrations.head_inserts(&ctx) { - snippet.push_str(&insert); - } + // What the middleware write ahead of the bundle, such as + // a module's configuration, which is then in place when + // the bundle's own start-up code reads it. for insert in &middleware_head_inserts { snippet.push_str(insert); } @@ -617,11 +601,8 @@ fn build_html_processor( &immediate_parts, &script_attributes, )); - // What an integration loads after the bundle and ahead of + // What a middleware loads after the bundle and ahead of // the page's own scripts in the origin head. - for insert in integrations.after_bundle_inserts(&ctx) { - snippet.push_str(&insert); - } for insert in &middleware_after_bundle_inserts { snippet.push_str(insert); } @@ -702,39 +683,11 @@ fn build_html_processor( // Replace URLs in href attributes element!("[href]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut href) = el.get_attribute("href") { - let original_href = href.clone(); - let element_name = el.tag_name(); - if let Some(rewritten) = patterns.rewrite_url_value(&href) { - href = rewritten; - } - - match integrations.rewrite_attribute( - "href", - &href, - &IntegrationAttributeContext { - attribute_name: "href", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_href) => { - href = integration_href; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if href != original_href { - el.set_attribute("href", &href)?; - } + if let Some(href) = el.get_attribute("href") + && let Some(rewritten) = patterns.rewrite_url_value(&href) + { + el.set_attribute("href", &rewritten)?; } Ok(()) } @@ -742,38 +695,11 @@ fn build_html_processor( // Replace URLs in src attributes element!("[src]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut src) = el.get_attribute("src") { - let original_src = src.clone(); - let element_name = el.tag_name(); - if let Some(rewritten) = patterns.rewrite_url_value(&src) { - src = rewritten; - } - match integrations.rewrite_attribute( - "src", - &src, - &IntegrationAttributeContext { - attribute_name: "src", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_src) => { - src = integration_src; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if src != original_src { - el.set_attribute("src", &src)?; - } + if let Some(src) = el.get_attribute("src") + && let Some(rewritten) = patterns.rewrite_url_value(&src) + { + el.set_attribute("src", &rewritten)?; } Ok(()) } @@ -781,39 +707,11 @@ fn build_html_processor( // Replace URLs in action attributes element!("[action]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut action) = el.get_attribute("action") { - let original_action = action.clone(); - let element_name = el.tag_name(); - if let Some(rewritten) = patterns.rewrite_url_value(&action) { - action = rewritten; - } - - match integrations.rewrite_attribute( - "action", - &action, - &IntegrationAttributeContext { - attribute_name: "action", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_action) => { - action = integration_action; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if action != original_action { - el.set_attribute("action", &action)?; - } + if let Some(action) = el.get_attribute("action") + && let Some(rewritten) = patterns.rewrite_url_value(&action) + { + el.set_attribute("action", &rewritten)?; } Ok(()) } @@ -821,12 +719,9 @@ fn build_html_processor( // Replace URLs in srcset attributes (for responsive images) element!("[srcset]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut srcset) = el.get_attribute("srcset") { - let original_srcset = srcset.clone(); - let element_name = el.tag_name(); - let new_srcset = srcset + if let Some(srcset) = el.get_attribute("srcset") { + let rewritten = srcset .replace(&patterns.https_origin(), &patterns.replacement_url()) .replace(&patterns.http_origin(), &patterns.replacement_url()) .replace( @@ -834,33 +729,8 @@ fn build_html_processor( &patterns.protocol_relative_replacement(), ) .replace(&patterns.origin_host, &patterns.request_host); - if new_srcset != srcset { - srcset = new_srcset; - } - - match integrations.rewrite_attribute( - "srcset", - &srcset, - &IntegrationAttributeContext { - attribute_name: "srcset", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_srcset) => { - srcset = integration_srcset; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if srcset != original_srcset { - el.set_attribute("srcset", &srcset)?; + if rewritten != srcset { + el.set_attribute("srcset", &rewritten)?; } } Ok(()) @@ -869,45 +739,17 @@ fn build_html_processor( // Replace URLs in imagesrcset attributes (for link preload) element!("[imagesrcset]", { let patterns = patterns.clone(); - let integrations = integration_registry.clone(); move |el| { - if let Some(mut imagesrcset) = el.get_attribute("imagesrcset") { - let original_imagesrcset = imagesrcset.clone(); - let element_name = el.tag_name(); - let new_imagesrcset = imagesrcset + if let Some(imagesrcset) = el.get_attribute("imagesrcset") { + let rewritten = imagesrcset .replace(&patterns.https_origin(), &patterns.replacement_url()) .replace(&patterns.http_origin(), &patterns.replacement_url()) .replace( &patterns.protocol_relative_origin(), &patterns.protocol_relative_replacement(), ); - if new_imagesrcset != imagesrcset { - imagesrcset = new_imagesrcset; - } - - match integrations.rewrite_attribute( - "imagesrcset", - &imagesrcset, - &IntegrationAttributeContext { - attribute_name: "imagesrcset", - element_name: &element_name, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - }, - ) { - AttributeRewriteOutcome::Unchanged => {} - AttributeRewriteOutcome::Replaced(integration_imagesrcset) => { - imagesrcset = integration_imagesrcset; - } - AttributeRewriteOutcome::RemoveElement => { - el.remove(); - return Ok(()); - } - } - - if imagesrcset != original_imagesrcset { - el.set_attribute("imagesrcset", &imagesrcset)?; + if rewritten != imagesrcset { + el.set_attribute("imagesrcset", &rewritten)?; } } Ok(()) @@ -946,45 +788,12 @@ fn build_html_processor( })); } - // Shared by every text handler below, the hooks' and the middleware's, - // so each is handed a chunk as the one before it left it. + // Shared by every text handler, so each is handed a chunk as the one + // before it left it. let edits = Rc::new(TextEdits::default()); - for script_rewriter in script_rewriters { - let selector = script_rewriter.selector(); - let rewriter = script_rewriter.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); - let edits = Rc::clone(&edits); - element_content_handlers.push(text!(selector, { - let rewriter = rewriter.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); - let edits = Rc::clone(&edits); - move |text| { - let ctx = IntegrationScriptContext { - selector, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - is_last_in_text_node: text.last_in_text_node(), - max_buffered_script_bytes: config.max_buffered_body_bytes, - document_state: &document_state, - }; - let current = edits.current(text); - match rewriter.rewrite(¤t, &ctx) { - ScriptRewriteAction::Keep => {} - ScriptRewriteAction::Replace(rewritten) => edits.replace(text, &rewritten), - ScriptRewriteAction::RemoveNode => edits.remove(text), - } - Ok(()) - } - })); - } - - // Registered after core's own handlers and the hooks', so an element - // handler judges an address as it will be served and a text handler is - // handed a script as the hooks left it. + // Registered after core's own handlers, so an element handler judges an + // address as it will be served. element_content_handlers.extend(middleware_content_handlers( middleware_element_handlers, middleware_text_handlers, @@ -999,20 +808,9 @@ fn build_html_processor( let inner = HtmlRewriterAdapter::new(rewriter_settings); - let stream_context = crate::integrations::IntegrationHtmlStreamContext { - request_host: config.request_host.clone(), - request_scheme: config.request_scheme.clone(), - origin_host: config.origin_host.clone(), - document_state: document_state.clone(), - }; - let mut processors: Vec> = stream_processor_factories - .into_iter() - .map(|factory| factory.create(stream_context.clone())) - .collect(); - processors.extend(middleware_processors); HtmlWithStreamingProcessors { inner: Box::new(inner), - processors, + processors: middleware_processors, } } @@ -1300,10 +1098,6 @@ passphrase = "page-recording-passphrase-32-bytes" #[cfg(test)] mod tests { use super::*; - use crate::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationHeadInjector, IntegrationHtmlContext, - }; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use crate::test_support::tests::create_test_settings; use serde_json::json; @@ -1331,131 +1125,6 @@ mod tests { } } - #[test] - fn integration_attribute_rewriter_can_remove_elements() { - struct RemovingLinkRewriter; - - impl IntegrationAttributeRewriter for RemovingLinkRewriter { - fn integration_id(&self) -> &'static str { - "removing" - } - - fn handles_attribute(&self, attribute: &str) -> bool { - attribute == "href" - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - if attr_value.contains("remove-me") { - AttributeRewriteAction::remove_element() - } else { - AttributeRewriteAction::keep() - } - } - } - - let html = r#" -
remove - keep - "#; - - let mut config = create_test_config(); - config.integrations = - IntegrationRegistry::from_rewriters(vec![Arc::new(RemovingLinkRewriter)], Vec::new()); - - let processor = create_html_processor(config); - let pipeline_config = PipelineConfig { - input_compression: Compression::None, - output_compression: Compression::None, - chunk_size: 8192, - }; - let mut pipeline = StreamingPipeline::new(pipeline_config, processor); - - let mut output = Vec::new(); - pipeline - .process(Cursor::new(html.as_bytes()), &mut output) - .expect("pipeline should process HTML"); - let processed = String::from_utf8(output).expect("output should be valid UTF-8"); - - assert!(processed.contains("keep-me")); - assert!(!processed.contains("remove-me")); - } - - #[test] - fn integration_head_injector_prepends_after_tsjs_once() { - struct TestHeadInjector; - - impl IntegrationHeadInjector for TestHeadInjector { - fn integration_id(&self) -> &'static str { - "test" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { - vec!["".to_owned()] - } - } - - let html = "Test"; - - let mut config = create_test_config(); - config.integrations = IntegrationRegistry::from_rewriters_with_head_injectors( - Vec::new(), - Vec::new(), - vec![Arc::new(TestHeadInjector)], - ); - - let processor = create_html_processor(config); - let pipeline_config = PipelineConfig { - input_compression: Compression::None, - output_compression: Compression::None, - chunk_size: 8192, - }; - let mut pipeline = StreamingPipeline::new(pipeline_config, processor); - - let mut output = Vec::new(); - pipeline - .process(Cursor::new(html.as_bytes()), &mut output) - .expect("pipeline should process HTML"); - let processed = String::from_utf8(output).expect("output should be valid UTF-8"); - - let tsjs_marker = "id=\"trustedserver-js\""; - let head_marker = "window.__testHeadInjector=true"; - - assert_eq!( - processed.matches(tsjs_marker).count(), - 1, - "should inject unified tsjs tag once" - ); - assert_eq!( - processed.matches(head_marker).count(), - 1, - "should inject head snippet once" - ); - - let tsjs_index = processed - .find(tsjs_marker) - .expect("should include unified tsjs tag"); - let head_index = processed - .find(head_marker) - .expect("should include head snippet"); - let title_index = processed - .find("") - .expect("should keep existing head content"); - - assert!( - head_index < tsjs_index, - "should inject config before tsjs bundle so auto-init can read it" - ); - assert!( - tsjs_index < title_index, - "should prepend all injected content before existing head content" - ); - } - #[test] fn a_registration_marks_only_the_bundle_it_asks_to_mark() { use crate::integrations::registry_test_support::tag_fixture as tag; @@ -2504,54 +2173,6 @@ mod tests { ); } - #[test] - fn a_rewritten_script_is_written_back_as_the_rewriter_gave_it() { - // A script's text is not HTML text. A browser decodes no entity inside - // a script, so `&&` written back as `&&` is a syntax error. - struct HostRewriter; - - impl crate::integrations::IntegrationScriptRewriter for HostRewriter { - fn integration_id(&self) -> &'static str { - "host-rewriter" - } - - fn selector(&self) -> &'static str { - "script" - } - - fn rewrite( - &self, - content: &str, - _ctx: &IntegrationScriptContext<'_>, - ) -> ScriptRewriteAction { - ScriptRewriteAction::replace( - content.replace("cdn.vendor.example", "test.example.com"), - ) - } - } - - let mut config = create_test_config(); - config.integrations = - IntegrationRegistry::from_rewriters(Vec::new(), vec![Arc::new(HostRewriter)]); - let mut processor = create_html_processor(config); - - let output = processor - .process_chunk( - br#"<html><head><script>if (a && b < c || d > e) { load("https://cdn.vendor.example/tag.js?x=1&y=2"); }</script></head><body></body></html>"#, - true, - ) - .expect("should process the document"); - let processed = String::from_utf8(output).expect("should stay UTF-8"); - - assert!( - processed.contains( - r#"<script>if (a && b < c || d > e) { load("https://test.example.com/tag.js?x=1&y=2"); }</script>"# - ), - "should write the rewritten script back with its operators and its query string \ - as they were: {processed}" - ); - } - #[test] fn bodyless_marker_mode_emits_an_owned_terminal_seam_even_after_source_bytes() { const MARKER: &str = "<!--reserved-template-cache-seam-->"; diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index 4466be1b5f..6e342339e3 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -20,15 +20,12 @@ mod registry; #[cfg(test)] pub(crate) use registry::test_support as registry_test_support; pub use registry::{ - AttributeRewriteAction, AttributeRewriteOutcome, CarriedJsModule, HeaderMutation, - HeaderMutationMode, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationDocumentState, IntegrationEndpoint, IntegrationHeadInjector, IntegrationHtmlContext, - IntegrationHtmlStreamContext, IntegrationHtmlStreamProcessorFactory, IntegrationMetadata, - IntegrationProxy, IntegrationRegistration, IntegrationRegistrationBuilder, IntegrationRegistry, - IntegrationRequestFilter, IntegrationRequestState, IntegrationScriptContext, - IntegrationScriptRewriter, ProxyDispatchInput, RequestFilterDecision, RequestFilterEffects, - RequestFilterInput, RequestFilterRegistryInput, RequestFilterRegistryOutcome, - ScriptRewriteAction, ScriptTextAccumulator, + AttributeRewriteAction, CarriedJsModule, HeaderMutation, HeaderMutationMode, + IntegrationDocumentState, IntegrationEndpoint, IntegrationMetadata, IntegrationProxy, + IntegrationRegistration, IntegrationRegistrationBuilder, IntegrationRegistry, + IntegrationRequestFilter, IntegrationRequestState, ProxyDispatchInput, RequestFilterDecision, + RequestFilterEffects, RequestFilterInput, RequestFilterRegistryInput, + RequestFilterRegistryOutcome, ScriptRewriteAction, }; /// Registers or retrieves a platform backend for the given URL. @@ -699,8 +696,8 @@ impl IntegrationBuilder { /// The built-in integrations, in hook order. const BUILT_IN_BUILDERS: &[IntegrationBuilder] = &[ - // This must remain the first module a section selects: attribute - // rewriters chain replacements and short-circuit removals. + // This stays the first module a section selects, so its middleware is + // the first of theirs in the order the registry lists them. js_asset_proxy::BUILDER, // A stand-in for an integration that streams, which core's own tests // select where they need one. diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index df93389573..8dd11ae9ec 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -1,6 +1,6 @@ use std::any::{Any, TypeId}; use std::collections::BTreeMap; -use std::sync::{Arc, Mutex, MutexGuard, PoisonError}; +use std::sync::{Arc, Mutex}; use async_trait::async_trait; use edgezero_core::body::Body as EdgeBody; @@ -22,9 +22,9 @@ use crate::middleware::{Middleware, MiddlewareChain, MiddlewarePhase, PhaseEntri use crate::module_context::{ModuleCall, ModuleContext, ResolvedRequest}; use crate::platform::{DisabledGeo, PlatformGeo, RuntimeServices}; use crate::settings::Settings; -use crate::streaming_processor::StreamProcessor; -/// Action returned by attribute rewriters to describe how the runtime should mutate the element. +/// What a middleware's element handler decides about one attribute of an +/// element. #[derive(Debug, Clone, PartialEq, Eq)] pub enum AttributeRewriteAction { /// Leave the attribute and element untouched. @@ -52,15 +52,8 @@ impl AttributeRewriteAction { } } -/// Outcome returned by the registry after running every matching attribute rewriter. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum AttributeRewriteOutcome { - Unchanged, - Replaced(String), - RemoveElement, -} - -/// Action returned by inline script rewriters to describe how to mutate the node. +/// What a middleware's text handler decides about one chunk of the text +/// inside an element. #[derive(Debug, Clone, PartialEq, Eq)] pub enum ScriptRewriteAction { Keep, @@ -85,28 +78,6 @@ impl ScriptRewriteAction { } } -/// Context provided to integration HTML attribute rewriters. -#[derive(Debug)] -pub struct IntegrationAttributeContext<'a> { - pub attribute_name: &'a str, - pub element_name: &'a str, - pub request_host: &'a str, - pub request_scheme: &'a str, - pub origin_host: &'a str, -} - -/// Context passed to script/text rewriters for inline HTML handling. -#[derive(Debug)] -pub struct IntegrationScriptContext<'a> { - pub selector: &'a str, - pub request_host: &'a str, - pub request_scheme: &'a str, - pub origin_host: &'a str, - pub is_last_in_text_node: bool, - pub max_buffered_script_bytes: usize, - pub document_state: &'a IntegrationDocumentState, -} - type IntegrationDocumentStateMap = BTreeMap<(&'static str, TypeId), Arc<dyn Any + Send + Sync>>; /// Per-document state shared between HTML/script rewriters and post-processors. @@ -196,20 +167,20 @@ impl IntegrationDocumentState { } } -/// Values a module's request hooks leave for its own page hooks, for one -/// request. +/// Values a module's request hooks leave for its own serve middleware, for +/// one request. /// /// A request preparer or a request filter leaves a value under its -/// integration id with [`IntegrationRequestState::insert`]. When the request -/// produces an HTML document for that one reader, every value is copied into -/// the document's [`IntegrationDocumentState`] before parsing starts, so the -/// module's head injector, rewriters and stream processors read it there, and -/// the same values are handed to the module's response finalizer. +/// integration id with [`IntegrationRequestState::insert`]. Where the +/// request's reader is served an HTML document, every value is copied into +/// the [`IntegrationDocumentState`] the `[[serve]]` entry's middleware share, +/// so the module's serve middleware reads it there, and the same values are +/// handed to the module's response finalizer. /// /// A request that carries any value keeps to the origin path. Its document is /// never read from a shared template and never stored as one, and an HTML -/// response with a body is sent `private, no-store`. A document built to be -/// shared starts with none of these values. +/// response with a body is sent `private, no-store`. A fetch middleware is +/// handed none of these values. #[derive(Clone, Default)] pub struct IntegrationRequestState { values: IntegrationDocumentStateMap, @@ -224,8 +195,8 @@ impl std::fmt::Debug for IntegrationRequestState { } impl IntegrationRequestState { - /// Leaves `value` on `request` for the page hooks of `integration_id`, - /// in place of a value of the same type left earlier. + /// Leaves `value` on `request` for the serve middleware of + /// `integration_id`, in place of a value of the same type left earlier. pub fn insert<T>(request: &mut Request<EdgeBody>, integration_id: &'static str, value: T) where T: Any + Send + Sync + 'static, @@ -238,8 +209,8 @@ impl IntegrationRequestState { request.extensions_mut().insert(state); } - /// Holds `value` for the page hooks of `integration_id`, in place of a - /// value of the same type held earlier. + /// Holds `value` for the serve middleware of `integration_id`, in place + /// of a value of the same type held earlier. pub fn set<T>(&mut self, integration_id: &'static str, value: T) where T: Any + Send + Sync + 'static, @@ -294,38 +265,6 @@ impl IntegrationRequestState { } } -/// Per-document buffer for script text fragments split across chunks. -/// -/// `lol_html` can deliver one text node as several chunks, so a rewriter that -/// needs the whole script must accumulate until `is_last_in_text_node`. -/// -/// This lives in [`IntegrationDocumentState`] rather than on the rewriter. -/// Rewriters are registered once as `Arc<dyn IntegrationScriptRewriter>` and -/// live as long as the [`IntegrationRegistry`], so a buffer owned by a -/// rewriter is shared by every document that registry serves. A document whose -/// stream ends before the final fragment — client disconnect, origin error, -/// truncated body — leaves its partial script in that buffer, and the next -/// document prepends the residue to its own accumulation. That corrupts the -/// response and can disclose the previous document's content. -/// -/// Keyed per integration id, so each integration gets its own buffer, and -/// dropped with the document state at end of document. -#[derive(Debug, Default)] -pub struct ScriptTextAccumulator { - buffer: Mutex<String>, -} - -impl ScriptTextAccumulator { - /// Locks the buffer. - /// - /// Recovers from poisoning rather than panicking: a poisoned buffer holds - /// at worst a partial script, and the caller's `is_last_in_text_node` - /// handling already tolerates unexpected contents. - pub fn buffer(&self) -> MutexGuard<'_, String> { - self.buffer.lock().unwrap_or_else(PoisonError::into_inner) - } -} - /// Describes an HTTP endpoint exposed by an integration. #[derive(Clone, Debug)] pub struct IntegrationEndpoint { @@ -676,83 +615,6 @@ fn apply_header_mutation_to_response(response: &mut Response<EdgeBody>, mutation } } -/// Trait for integration-provided HTML attribute rewrite hooks. -pub trait IntegrationAttributeRewriter: Send + Sync { - /// Identifier for logging/diagnostics. - fn integration_id(&self) -> &'static str; - /// Return true when this rewriter wants to inspect a given attribute. - fn handles_attribute(&self, attribute: &str) -> bool; - /// Attempt to rewrite the attribute value. Return `AttributeRewriteAction::Replace` - /// to update the attribute, `Keep` to leave it untouched, or `RemoveElement` to drop the node. - fn rewrite( - &self, - attr_name: &str, - attr_value: &str, - ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction; -} - -/// Trait for integration-provided inline script/text rewrite hooks. -pub trait IntegrationScriptRewriter: Send + Sync { - /// Identifier for logging/diagnostics. - fn integration_id(&self) -> &'static str; - /// CSS selector (e.g. `script#__NEXT_DATA__`) that should trigger this rewriter. - fn selector(&self) -> &'static str; - /// Attempt to rewrite the inline text content for the selector. - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction; -} - -/// Context for HTML post-processors. -#[derive(Debug)] -pub struct IntegrationHtmlContext<'a> { - pub request_host: &'a str, - pub request_scheme: &'a str, - pub origin_host: &'a str, - pub document_state: &'a IntegrationDocumentState, -} - -/// Owned request data supplied when an integration creates an HTML stream processor. -#[derive(Clone)] -pub struct IntegrationHtmlStreamContext { - /// Publisher-facing host used for rewritten URLs. - pub request_host: String, - /// Publisher-facing scheme used for rewritten URLs. - pub request_scheme: String, - /// Origin host whose URLs may be rewritten. - pub origin_host: String, - /// Request-local state shared with the document's integration rewriters. - pub document_state: IntegrationDocumentState, -} - -/// Creates one mutable HTML output processor for each document. -pub trait IntegrationHtmlStreamProcessorFactory: Send + Sync { - /// Identifier for logging and diagnostics. - fn integration_id(&self) -> &'static str; - - /// Create a request-local streaming processor. - fn create(&self, context: IntegrationHtmlStreamContext) -> Box<dyn StreamProcessor>; -} - -/// Trait for integration-provided HTML head injections. -pub trait IntegrationHeadInjector: Send + Sync { - /// Identifier for logging/diagnostics. - fn integration_id(&self) -> &'static str; - /// Return HTML snippets to insert at the start of `<head>`. - fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec<String>; - - /// Return HTML snippets to insert straight after the main script bundle - /// and before any deferred one, for a script that needs the bundle to - /// have run and has to run before the page's own scripts. - fn after_bundle_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - Vec::new() - } - - /// Return attributes to add to the publisher TSJS bundle tag. - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - Vec::new() - } -} - /// A browser module a registration carries, for a module built outside /// `trusted-server-js`. The crate embeds its built IIFE with `include_str!` /// and states its SHA-256 as a literal next to it; the registry verifies the @@ -778,10 +640,6 @@ pub struct IntegrationRegistration { /// integration injects the tag itself when it decides to. pub js_standalone: bool, pub proxies: Vec<Arc<dyn IntegrationProxy>>, - pub attribute_rewriters: Vec<Arc<dyn IntegrationAttributeRewriter>>, - pub script_rewriters: Vec<Arc<dyn IntegrationScriptRewriter>>, - pub html_stream_processors: Vec<Arc<dyn IntegrationHtmlStreamProcessorFactory>>, - pub head_injectors: Vec<Arc<dyn IntegrationHeadInjector>>, pub request_filters: Vec<Arc<dyn IntegrationRequestFilter>>, /// The page changes this registration supplies, see [`crate::middleware`]. /// @@ -835,10 +693,6 @@ impl IntegrationRegistrationBuilder { js_module: None, js_standalone: false, proxies: Vec::new(), - attribute_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters: Vec::new(), middleware: Vec::new(), bundle_tag_attributes: Vec::new(), @@ -855,36 +709,6 @@ impl IntegrationRegistrationBuilder { self } - #[must_use] - pub fn with_attribute_rewriter( - mut self, - rewriter: Arc<dyn IntegrationAttributeRewriter>, - ) -> Self { - self.registration.attribute_rewriters.push(rewriter); - self - } - - #[must_use] - pub fn with_script_rewriter(mut self, rewriter: Arc<dyn IntegrationScriptRewriter>) -> Self { - self.registration.script_rewriters.push(rewriter); - self - } - - #[must_use] - pub fn with_html_stream_processor( - mut self, - processor: Arc<dyn IntegrationHtmlStreamProcessorFactory>, - ) -> Self { - self.registration.html_stream_processors.push(processor); - self - } - - #[must_use] - pub fn with_head_injector(mut self, injector: Arc<dyn IntegrationHeadInjector>) -> Self { - self.registration.head_injectors.push(injector); - self - } - #[must_use] pub fn with_request_filter(mut self, filter: Arc<dyn IntegrationRequestFilter>) -> Self { self.registration.request_filters.push(filter); @@ -1030,10 +854,6 @@ struct IntegrationRegistryInner { // Modules carried by their registrations, verified against their // declared hash at construction. carried_js: Vec<(&'static str, CarriedJsModule)>, - html_rewriters: Vec<Arc<dyn IntegrationAttributeRewriter>>, - script_rewriters: Vec<Arc<dyn IntegrationScriptRewriter>>, - html_stream_processors: Vec<Arc<dyn IntegrationHtmlStreamProcessorFactory>>, - head_injectors: Vec<Arc<dyn IntegrationHeadInjector>>, request_filters: Vec<Arc<dyn IntegrationRequestFilter>>, // The middleware the running registrations supply, each against the // integration that supplied it, in registration order. @@ -1088,10 +908,6 @@ impl Default for IntegrationRegistryInner { disabled_js_ids: Vec::new(), standalone_js_ids: Vec::new(), carried_js: Vec::new(), - html_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters: Vec::new(), middleware: Vec::new(), bundle_tag_attributes: Vec::new(), @@ -1414,9 +1230,8 @@ fn resolve_geo_module( pub struct IntegrationMetadata { pub id: &'static str, pub routes: Vec<IntegrationEndpoint>, - pub attribute_rewriters: usize, - pub script_selectors: Vec<&'static str>, - pub head_injectors: usize, + /// The names of the middleware the integration supplies. + pub middleware: Vec<&'static str>, pub request_filters: usize, } @@ -1425,9 +1240,7 @@ impl IntegrationMetadata { Self { id, routes: Vec::new(), - attribute_rewriters: 0, - script_selectors: Vec::new(), - head_injectors: 0, + middleware: Vec::new(), request_filters: 0, } } @@ -1817,14 +1630,6 @@ impl IntegrationRegistry { inner.routes.push((route, registration.integration_id)); } } - inner - .html_rewriters - .extend(registration.attribute_rewriters); - inner.script_rewriters.extend(registration.script_rewriters); - inner - .html_stream_processors - .extend(registration.html_stream_processors); - inner.head_injectors.extend(registration.head_injectors); inner.request_filters.extend(registration.request_filters); for middleware in registration.middleware { inner @@ -2195,92 +2000,12 @@ impl IntegrationRegistry { } } - /// Give integrations a chance to rewrite HTML attributes. - #[must_use] - pub fn rewrite_attribute( - &self, - attr_name: &str, - attr_value: &str, - ctx: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteOutcome { - let mut current = attr_value.to_owned(); - let mut changed = false; - for rewriter in &self.inner.html_rewriters { - if !rewriter.handles_attribute(attr_name) { - continue; - } - match rewriter.rewrite(attr_name, ¤t, ctx) { - AttributeRewriteAction::Keep => {} - AttributeRewriteAction::Replace(next_value) => { - current = next_value; - changed = true; - } - AttributeRewriteAction::RemoveElement => { - return AttributeRewriteOutcome::RemoveElement; - } - } - } - - if changed { - AttributeRewriteOutcome::Replaced(current) - } else { - AttributeRewriteOutcome::Unchanged - } - } - - /// Expose registered script/text rewriters for HTML processing. - #[must_use] - pub fn script_rewriters(&self) -> Vec<Arc<dyn IntegrationScriptRewriter>> { - self.inner.script_rewriters.clone() - } - - /// Expose registered per-document HTML stream processor factories. - #[must_use] - pub fn html_stream_processor_factories( - &self, - ) -> Vec<Arc<dyn IntegrationHtmlStreamProcessorFactory>> { - self.inner.html_stream_processors.clone() - } - - /// Collect HTML snippets for insertion at the start of `<head>`. - #[must_use] - pub fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - let mut inserts = Vec::new(); - for injector in &self.inner.head_injectors { - let mut next = injector.head_inserts(ctx); - if !next.is_empty() { - inserts.append(&mut next); - } - } - inserts - } - - /// Collect HTML snippets for insertion straight after the main script - /// bundle. - #[must_use] - pub fn after_bundle_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - self.inner - .head_injectors - .iter() - .flat_map(|injector| injector.after_bundle_inserts(ctx)) - .collect() - } - - /// Collect static attributes for the publisher TSJS bundle tag, being - /// those the head injectors ask for and then those the registrations - /// state, each keeping the first value a name is given. + /// Collect the attributes the registrations put on the publisher TSJS + /// bundle tag, each keeping the first value a name is given. #[must_use] pub fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - let from_injectors = self.inner.head_injectors.iter().flat_map(|injector| { - injector - .tsjs_script_tag_attributes() - .into_iter() - .map(|attribute| (injector.integration_id(), attribute)) - }); let mut attributes: Vec<(&'static str, &'static str)> = Vec::new(); - for (integration, attribute) in - from_injectors.chain(self.inner.bundle_tag_attributes.iter().copied()) - { + for (integration, attribute) in self.inner.bundle_tag_attributes.iter().copied() { let existing = attributes .iter() .find(|(name, _)| *name == attribute.0) @@ -2297,7 +2022,7 @@ impl IntegrationRegistry { attributes } - /// Provide a snapshot of registered integrations and their hooks. + /// Provide a snapshot of registered integrations and what each supplies. #[must_use] pub fn registered_integrations(&self) -> Vec<IntegrationMetadata> { let mut map: BTreeMap<&'static str, IntegrationMetadata> = BTreeMap::new(); @@ -2317,25 +2042,11 @@ impl IntegrationRegistry { )); } - for rewriter in &self.inner.html_rewriters { + for (integration_id, middleware) in &self.inner.middleware { let entry = map - .entry(rewriter.integration_id()) - .or_insert_with(|| IntegrationMetadata::new(rewriter.integration_id())); - entry.attribute_rewriters += 1; - } - - for rewriter in &self.inner.script_rewriters { - let entry = map - .entry(rewriter.integration_id()) - .or_insert_with(|| IntegrationMetadata::new(rewriter.integration_id())); - entry.script_selectors.push(rewriter.selector()); - } - - for injector in &self.inner.head_injectors { - let entry = map - .entry(injector.integration_id()) - .or_insert_with(|| IntegrationMetadata::new(injector.integration_id())); - entry.head_injectors += 1; + .entry(*integration_id) + .or_insert_with(|| IntegrationMetadata::new(integration_id)); + entry.middleware.push(middleware.middleware_id()); } for filter in &self.inner.request_filters { @@ -2532,95 +2243,6 @@ impl IntegrationRegistry { } } - #[cfg(test)] - #[must_use] - pub fn from_rewriters( - attribute_rewriters: Vec<Arc<dyn IntegrationAttributeRewriter>>, - script_rewriters: Vec<Arc<dyn IntegrationScriptRewriter>>, - ) -> Self { - Self { - inner: Arc::new(IntegrationRegistryInner { - get_router: Router::new(), - post_router: Router::new(), - put_router: Router::new(), - delete_router: Router::new(), - patch_router: Router::new(), - head_router: Router::new(), - options_router: Router::new(), - routes: Vec::new(), - builder_ids: Vec::new(), - running_integration_ids: Vec::new(), - html_rewriters: attribute_rewriters, - script_rewriters, - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), - request_filters: Vec::new(), - middleware: Vec::new(), - bundle_tag_attributes: Vec::new(), - request_preparers: Vec::new(), - response_finalizers: Vec::new(), - deferred_js_ids: Vec::new(), - disabled_js_ids: Vec::new(), - extra_js_module_ids: Vec::new(), - standalone_js_ids: Vec::new(), - carried_js: Vec::new(), - geo_modules: Vec::new(), - ec_modules: Vec::new(), - device_modules: Vec::new(), - builder_modules: Vec::new(), - geo_module: None, - ec_module: None, - device_module: None, - }), - plan: None, - } - } - - #[cfg(test)] - #[must_use] - pub fn from_rewriters_with_head_injectors( - attribute_rewriters: Vec<Arc<dyn IntegrationAttributeRewriter>>, - script_rewriters: Vec<Arc<dyn IntegrationScriptRewriter>>, - head_injectors: Vec<Arc<dyn IntegrationHeadInjector>>, - ) -> Self { - Self { - inner: Arc::new(IntegrationRegistryInner { - get_router: Router::new(), - post_router: Router::new(), - put_router: Router::new(), - delete_router: Router::new(), - patch_router: Router::new(), - head_router: Router::new(), - options_router: Router::new(), - routes: Vec::new(), - builder_ids: Vec::new(), - running_integration_ids: Vec::new(), - html_rewriters: attribute_rewriters, - script_rewriters, - html_stream_processors: Vec::new(), - head_injectors, - request_filters: Vec::new(), - middleware: Vec::new(), - bundle_tag_attributes: Vec::new(), - request_preparers: Vec::new(), - response_finalizers: Vec::new(), - deferred_js_ids: Vec::new(), - disabled_js_ids: Vec::new(), - extra_js_module_ids: Vec::new(), - standalone_js_ids: Vec::new(), - carried_js: Vec::new(), - geo_modules: Vec::new(), - ec_modules: Vec::new(), - device_modules: Vec::new(), - builder_modules: Vec::new(), - geo_module: None, - ec_module: None, - device_module: None, - }), - plan: None, - } - } - #[cfg(any(test, feature = "test-utils"))] #[must_use] pub fn from_request_filters(request_filters: Vec<Arc<dyn IntegrationRequestFilter>>) -> Self { @@ -2636,10 +2258,6 @@ impl IntegrationRegistry { routes: Vec::new(), builder_ids: Vec::new(), running_integration_ids: Vec::new(), - html_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters, middleware: Vec::new(), bundle_tag_attributes: Vec::new(), @@ -2717,10 +2335,6 @@ impl IntegrationRegistry { routes: Vec::new(), builder_ids: Vec::new(), running_integration_ids: Vec::new(), - html_rewriters: Vec::new(), - script_rewriters: Vec::new(), - html_stream_processors: Vec::new(), - head_injectors: Vec::new(), request_filters: Vec::new(), middleware: Vec::new(), bundle_tag_attributes: Vec::new(), @@ -3764,79 +3378,6 @@ mod tests { use crate::platform::test_support::noop_services; use http::{HeaderValue, StatusCode, header}; - struct DefaultMetadataHeadInjector; - - impl IntegrationHeadInjector for DefaultMetadataHeadInjector { - fn integration_id(&self) -> &'static str { - "default-metadata" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - Vec::new() - } - } - - struct StaticMetadataHeadInjector; - - impl IntegrationHeadInjector for StaticMetadataHeadInjector { - fn integration_id(&self) -> &'static str { - "static-metadata" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - Vec::new() - } - - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - vec![ - ("data-ts-gam-attribution", "true"), - ("data-test-order", "second"), - ] - } - } - - struct ConflictingMetadataHeadInjector; - - impl IntegrationHeadInjector for ConflictingMetadataHeadInjector { - fn integration_id(&self) -> &'static str { - "conflicting-metadata" - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - Vec::new() - } - - fn tsjs_script_tag_attributes(&self) -> Vec<(&'static str, &'static str)> { - vec![ - ("data-ts-gam-attribution", "false"), - ("data-third-attribute", "third"), - ] - } - } - - #[test] - fn tsjs_script_tag_attributes_preserve_registration_order_and_default_empty() { - let registry = IntegrationRegistry::from_rewriters_with_head_injectors( - Vec::new(), - Vec::new(), - vec![ - Arc::new(DefaultMetadataHeadInjector), - Arc::new(StaticMetadataHeadInjector), - Arc::new(ConflictingMetadataHeadInjector), - ], - ); - - assert_eq!( - registry.tsjs_script_tag_attributes(), - vec![ - ("data-ts-gam-attribution", "true"), - ("data-test-order", "second"), - ("data-third-attribute", "third"), - ], - "should keep the first value for duplicate names and preserve attribute order" - ); - } - // Mock integration proxy for testing struct MockProxy; @@ -3977,76 +3518,6 @@ mod tests { ); } - struct CountingStreamFactory(&'static str); - - impl IntegrationHtmlStreamProcessorFactory for CountingStreamFactory { - fn integration_id(&self) -> &'static str { - self.0 - } - - fn create(&self, _context: IntegrationHtmlStreamContext) -> Box<dyn StreamProcessor> { - struct CountingStreamProcessor(usize); - - impl StreamProcessor for CountingStreamProcessor { - fn process_chunk( - &mut self, - chunk: &[u8], - _is_last: bool, - ) -> std::io::Result<Vec<u8>> { - self.0 += 1; - let mut output = self.0.to_string().into_bytes(); - output.extend_from_slice(chunk); - Ok(output) - } - } - - Box::new(CountingStreamProcessor(0)) - } - } - - #[test] - fn html_stream_factories_preserve_order_and_create_isolated_sessions() { - let registration = IntegrationRegistration::builder("test") - .with_html_stream_processor(Arc::new(CountingStreamFactory("first"))) - .with_html_stream_processor(Arc::new(CountingStreamFactory("second"))) - .build(); - let identifiers: Vec<_> = registration - .html_stream_processors - .iter() - .map(|factory| factory.integration_id()) - .collect(); - assert_eq!( - identifiers, - ["first", "second"], - "should preserve factory registration order", - ); - - let context = IntegrationHtmlStreamContext { - request_host: "proxy.example.com".to_owned(), - request_scheme: "https".to_owned(), - origin_host: "origin.example.com".to_owned(), - document_state: IntegrationDocumentState::default(), - }; - let factory = ®istration.html_stream_processors[0]; - let mut first = factory.create(context.clone()); - let mut second = factory.create(context); - - assert_eq!( - first - .process_chunk(b"a", false) - .expect("should process first session"), - b"1a", - "should initialize the first session counter", - ); - assert_eq!( - second - .process_chunk(b"b", true) - .expect("should process second session"), - b"1b", - "should initialize an independent second session counter", - ); - } - #[test] fn handle_proxy_passes_http_request_without_fastly_round_trip() { let settings = create_test_settings(); @@ -5433,33 +4904,20 @@ mod tests { ); } - /// Writes one fixed insert, so a test can read the order hooks ran in. - struct FixedHeadInsert { - id: &'static str, - insert: &'static str, - } - - impl IntegrationHeadInjector for FixedHeadInsert { - fn integration_id(&self) -> &'static str { - self.id - } - - fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - vec![self.insert.to_owned()] - } - } - - const SECTION_INSERT: &str = "<!--from the section-->"; - const PLAN_INSERT: &str = "<!--from the plan-->"; + // Each probe supplies a middleware that changes nothing, so a test can + // read the order registrations were made in from the names the registry + // lists. + const SECTION_MIDDLEWARE: &str = "testing.probe-section"; + const PLAN_MIDDLEWARE: &str = "testing.probe-plan"; fn section_probe_registration( _settings: &Settings, ) -> Result<Option<IntegrationRegistration>, Report<TrustedServerError>> { Ok(Some( IntegrationRegistration::builder("probe-section") - .with_head_injector(Arc::new(FixedHeadInsert { - id: "probe-section", - insert: SECTION_INSERT, + .with_middleware(Arc::new(Named { + id: SECTION_MIDDLEWARE, + phases: FETCH_ONLY, })) .build(), )) @@ -5473,9 +4931,9 @@ mod tests { ) -> Result<Option<IntegrationRegistration>, Report<TrustedServerError>> { Ok(plan.enabled().then(|| { IntegrationRegistration::builder("probe-plan") - .with_head_injector(Arc::new(FixedHeadInsert { - id: "probe-plan", - insert: PLAN_INSERT, + .with_middleware(Arc::new(Named { + id: PLAN_MIDDLEWARE, + phases: FETCH_ONLY, })) .build() })) @@ -5503,17 +4961,11 @@ mod tests { .with_module_name("testing.probe-plan") .with_plan_registration(plan_probe_registration), ]; - let probe_inserts = |registry: &IntegrationRegistry| { - let document_state = IntegrationDocumentState::default(); + let probe_middleware = |registry: &IntegrationRegistry| { registry - .head_inserts(&IntegrationHtmlContext { - request_host: "publisher.example.com", - request_scheme: "https", - origin_host: "origin.example.com", - document_state: &document_state, - }) + .middleware_ids() .into_iter() - .filter(|insert| insert == SECTION_INSERT || insert == PLAN_INSERT) + .filter(|name| *name == SECTION_MIDDLEWARE || *name == PLAN_MIDDLEWARE) .collect::<Vec<_>>() }; @@ -5524,7 +4976,7 @@ mod tests { !registry.integration_runs("probe-plan"), "should register nothing where the function finds nothing in the plan" ); - assert_eq!(probe_inserts(®istry), vec![SECTION_INSERT]); + assert_eq!(probe_middleware(®istry), vec![SECTION_MIDDLEWARE]); settings.auction.enabled = true; let registry = IntegrationRegistry::with_registrations(&settings, &extra) @@ -5534,9 +4986,9 @@ mod tests { "should register from the plan a module no section selects" ); assert_eq!( - probe_inserts(®istry), - vec![PLAN_INSERT, SECTION_INSERT], - "should run the hooks of what the plan registered ahead of a section's module" + probe_middleware(®istry), + vec![PLAN_MIDDLEWARE, SECTION_MIDDLEWARE], + "should register what the plan supplies ahead of a section's module" ); } diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index 32117f8393..bd9c07867e 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -1,12 +1,12 @@ //! Page changes, which a module makes to a document on its way to a reader. //! -//! A middleware is a page change the settings place. A module registers the -//! ones it supplies, and one runs only where an entry in the settings names -//! it. Each phase has its own ordered list of entries. An entry covers one -//! media type, optionally only the requests under a path prefix, and names -//! the middleware to run in order. A response takes the first entry that -//! covers it, and each middleware is handed the document as the ones before -//! it left it. +//! A module changes a page through a middleware and in no other way. It +//! registers the ones it supplies, and one runs only where an entry in the +//! settings names it. Each phase has its own ordered list of entries. An +//! entry covers one media type, optionally only the requests under a path +//! prefix, and names the middleware to run in order. A response takes the +//! first entry that covers it, and each middleware is handed the document as +//! the ones before it left it. //! //! ```toml //! [[fetch]] diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index 503d58f48b..d084dcca2e 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -901,7 +901,7 @@ fn the_middleware_of_one_document_share_its_state_and_see_nothing_a_request_left } }), ]); - // The request left a mark, which the hooks of an unstored page may read. + // The request left a mark, which a serve middleware may read. let config = config().with_request_state(request_fixture::marked()); let page = page_with(config, &chain, PAGE, 8192); @@ -1010,113 +1010,6 @@ fn a_chain_reports_its_phase_and_the_names_in_the_order_they_run() { assert!(chain_of(Vec::new()).is_empty()); } -/// A module's hooks, each of which marks what it was handed. -struct Hooks; - -impl crate::integrations::IntegrationHeadInjector for Hooks { - fn integration_id(&self) -> &'static str { - "hooks" - } - - fn head_inserts(&self, _ctx: &crate::integrations::IntegrationHtmlContext<'_>) -> Vec<String> { - vec!["<meta name=\"hook\">".to_owned()] - } -} - -impl crate::integrations::IntegrationAttributeRewriter for Hooks { - fn integration_id(&self) -> &'static str { - "hooks" - } - - fn handles_attribute(&self, attribute: &str) -> bool { - attribute == "href" - } - - fn rewrite( - &self, - _attr_name: &str, - attr_value: &str, - _ctx: &crate::integrations::IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - AttributeRewriteAction::replace(format!("{attr_value}/hook")) - } -} - -impl crate::integrations::IntegrationScriptRewriter for Hooks { - fn integration_id(&self) -> &'static str { - "hooks" - } - - fn selector(&self) -> &'static str { - "script" - } - - fn rewrite( - &self, - content: &str, - _ctx: &crate::integrations::IntegrationScriptContext<'_>, - ) -> ScriptRewriteAction { - if content.contains("origin()") { - ScriptRewriteAction::replace(content.replace("origin()", "hook()")) - } else { - ScriptRewriteAction::keep() - } - } -} - -#[test] -fn a_middleware_runs_after_the_hooks_of_the_same_kind() { - let chain = chain_of(vec![middleware("example.after", |_| MiddlewareAction { - head_inserts: vec!["<meta name=\"middleware\">".to_owned()], - element_handlers: vec![Box::new(AttributeRewrite::matching( - "a", - "href", - Rc::new(|matched| { - AttributeRewriteAction::replace(format!("{}/middleware", matched.value)) - }), - ))], - text_handlers: vec![Box::new(TextFn { - selector: "script", - decide: |text: &str, _is_last: bool| { - if text.contains("hook()") { - ScriptRewriteAction::replace(text.replace("hook()", "middleware()")) - } else { - ScriptRewriteAction::keep() - } - }, - })], - ..MiddlewareAction::pass() - })]); - let mut config = config(); - config.integrations = IntegrationRegistry::from_rewriters_with_head_injectors( - vec![Arc::new(Hooks)], - vec![Arc::new(Hooks)], - vec![Arc::new(Hooks)], - ); - - let page = page_with( - config, - &chain, - r#"<html><head></head><body><a href="/start">x</a><script>origin()</script></body></html>"#, - 8192, - ); - - let hook = index_of(&page, "name=\"hook\""); - let after = index_of(&page, "name=\"middleware\""); - assert!( - hook < after && after < index_of(&page, BUNDLE_TAG), - "should write a middleware's head markup after the hooks' own: {page}" - ); - assert!( - page.contains(r#"<a href="/start/hook/middleware">"#), - "should ask an element handler about the value the hooks left: {page}" - ); - assert!( - page.contains("<script>middleware()</script>"), - "should hand a text handler the script as the hooks left it: {page}" - ); -} - // --------------------------------------------------------------------------- // What the document pipeline does with a serve chain // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-core/src/platform/template_cache.rs b/crates/trusted-server-core/src/platform/template_cache.rs index 440e5a7b86..efa3f170f3 100644 --- a/crates/trusted-server-core/src/platform/template_cache.rs +++ b/crates/trusted-server-core/src/platform/template_cache.rs @@ -56,7 +56,7 @@ pub struct TemplateCacheKey { /// mutate first. pub url: String, /// Host and scheme. The post-processed output is host-dependent by construction: - /// both reach `IntegrationHtmlContext` and drive URL rewriting. + /// both are handed to every middleware and drive URL rewriting. pub request_host: String, /// See [`Self::request_host`]. pub request_scheme: String, diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index d79bc239d9..bae0f460d9 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -1330,8 +1330,8 @@ struct HtmlStreamProcessorParams<'a> { permissions_script: Option<String>, ad_slots_script: Option<String>, ad_bids_state: Arc<Mutex<Option<String>>>, - /// What modules left on the request for their page hooks. See - /// [`template_request_state`]. + /// What modules left on the request, which the `[[serve]]` entry's + /// middleware read where that entry runs on this pass. request_state: IntegrationRequestState, /// Whether a shared template was authorized for this response. /// @@ -1344,25 +1344,6 @@ struct HtmlStreamProcessorParams<'a> { deferred_inline_marker: Option<String>, } -/// The request state a document's page hooks may read. -/// -/// What a module leaves on a request is made for that one request, so it -/// must not reach a template that is stored and served to other readers. -/// -/// [`handle_publisher_request`] already keeps a request that carries any off -/// the shared template path, so a shared mode never sees one today. This -/// makes the same guarantee where the document is built, so that neither -/// place depends on the other. -pub(crate) fn template_request_state( - mode: AssemblyMode, - request_state: IntegrationRequestState, -) -> IntegrationRequestState { - match mode { - AssemblyMode::Inline => request_state, - AssemblyMode::Esi => IntegrationRequestState::default(), - } -} - /// The marker emitted at the `</body>` seam under [`AssemblyMode::Esi`], reserving the /// place this reader's slots and bids are spliced into. /// @@ -1538,8 +1519,6 @@ fn create_html_stream_processor( _ => body_close_injection(assembly_mode, params.ad_slots_script.is_some()), }; - let request_state = template_request_state(assembly_mode, params.request_state); - // Only a response that can be stored has a consumer for the observation, so the // handlers are not registered for ordinary inline traffic. let csp_nonce_observed = params @@ -1550,12 +1529,13 @@ fn create_html_stream_processor( let config = config .with_permissions_script(params.permissions_script) .with_ad_state(params.ad_slots_script, params.ad_bids_state) - .with_request_state(request_state) + .with_request_state(params.request_state) .with_body_close(body_close) .with_csp_nonce_observer(csp_nonce_observed); // The middleware of the first [[fetch]] entry covering this path. What - // they leave is what a shared template stores. + // they leave is what a shared template stores, and they are handed + // nothing the request left. let fetch = params.integration_registry.middleware_chain( ¶ms.settings.fetch, MiddlewarePhase::Fetch, @@ -1564,7 +1544,9 @@ fn create_html_stream_processor( ); // A response that may be stored leaves this processor as the template // every reader shares, so its [[serve]] entry runs on each reader's copy - // once that copy is assembled. See `run_serve_pass`. + // once that copy is assembled. See `run_serve_pass`. What the request + // left reaches a document through that entry alone, so none of it reaches + // a template. let serve = (!params.shared_template_authorized).then(|| { params.integration_registry.middleware_chain( ¶ms.settings.serve, @@ -1844,7 +1826,8 @@ pub struct OwnedProcessResponseParams { pub(crate) dispatched_auction: Option<DispatchedAuction>, /// Price granularity used to bucket bids when building `tsjs.bids`. pub(crate) price_granularity: PriceGranularity, - /// What modules left on the request for their page hooks. + /// What modules left on the request, for their serve middleware and + /// their response finalizers. pub(crate) request_state: IntegrationRequestState, /// Set by the transform when the document carries a response-bound CSP nonce. /// @@ -9630,14 +9613,11 @@ mod tests { let ad_slots_script = template_ad_slots_script(mode, shape.ad_stack_ran, &settings, &slots, "/"); let body_close = body_close_injection(mode, ad_slots_script.is_some()); - let request_state = template_request_state( - mode, - if shape.module_state_left { - request_fixture::marked() - } else { - IntegrationRequestState::default() - }, - ); + let request_state = if shape.module_state_left { + request_fixture::marked() + } else { + IntegrationRequestState::default() + }; let ad_bids_state = std::sync::Arc::new(std::sync::Mutex::new(shape.bids_available.then(|| { @@ -10914,8 +10894,9 @@ mod tests { request } - /// A navigation a module left state on for its page hooks, the way a - /// real one arrives: on the request, left by a preparer or a filter. + /// A navigation a module left state on for its serve middleware, the + /// way a real one arrives: on the request, left by a preparer or a + /// filter. fn marked_navigation_request() -> Request<EdgeBody> { let mut request = navigation_request(); crate::integrations::registry_test_support::request_fixture::mark(&mut request); diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index e9a457af4f..b2a36bf727 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -47,7 +47,7 @@ type PbjsGlobal = typeof _pbjsDefault & { // Prebid.js itself is NOT bundled into this module. It is served as the // external bundle configured via `auction.prebid.external_bundle_url` // (required whenever the prebid integration runs) and owns the -// `window.pbjs` global. The Rust head injector emits a stub +// `window.pbjs` global. The Rust module's middleware emits a stub // (`window.pbjs = window.pbjs || {que:[],cmd:[]}`) before any script runs and // Prebid.js installs its API onto that same object, so capturing the reference // at module scope is safe regardless of evaluation order. @@ -179,7 +179,7 @@ export interface PrebidNpmConfig { /** * Shape of the server-injected config at `window.__tsjs_prebid`. - * Set by the Rust IntegrationHeadInjector from trusted-server.toml values. + * Set by the Rust module's middleware from trusted-server.toml values. */ interface InjectedPrebidConfig { accountId?: string; diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md index ef6501d061..f70254031c 100644 --- a/docs/guide/api-reference.md +++ b/docs/guide/api-reference.md @@ -1275,7 +1275,7 @@ All integration modules are built at compile time. At runtime, the server concat Every row below records a compiled integration registration predicate. An integration with no HTTP route can still contribute a browser module, -rewriter, injector, post-processor, request filter, or ad server. In the +middleware, request filter, or ad server. In the predicates below, `named` means a section selects the integration's module. | Integration | Registration predicate | HTTP routes | diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 67fd258eca..7c74ae8b5a 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1728,10 +1728,10 @@ name no module supplies. Their settings live in `[demand.<name>]` and ### Placing page changes -A module can supply a change it makes to a page as a middleware, and a -middleware runs only on the pages where an entry names it. Selecting the -module makes its middleware available, and the `[[fetch]]` and `[[serve]]` -entries say which pages each one runs on and in what order. +A module changes a page through a middleware, and a middleware runs only on +the pages where an entry names it. Selecting the module makes its middleware +available, and the `[[fetch]]` and `[[serve]]` entries say which pages each +one runs on and in what order. The two lists are the two phases of a page. A middleware runs in the phase its module built it for, and an entry of the other list that names it refuses @@ -1789,21 +1789,21 @@ The modules this repository ships supply the middleware below. A deployment that runs several of them names them all in one entry's list, because a page takes one entry, and the order written is the order they run in. -| Middleware | Phase | What it changes | -| ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | -| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | -| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | -| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | -| `cmp.sourcepoint` | `[[fetch]]` | Writes `window.__tsjs_sourcepoint` into the head and, when `rewrite_sdk` is set, the trap on `window._sp_`, and points a `src` or `href` on Sourcepoint's CDN at `/integrations/sourcepoint/cdn` | -| `tag.google-tag-manager` | `[[fetch]]` | Points Google Tag Manager and Google Analytics addresses at `/integrations/google_tag_manager`, in `src` and `href` attributes and in the text of inline scripts | -| `bot-protection.datadome` | `[[fetch]]` | Points a `src` or `href` that is a DataDome script's address at `/integrations/datadome`, when `rewrite_sdk` is set | -| `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | -| `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | -| `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | -| `ad-tag.google.diagnostics` | `[[serve]]` | For a request that activated diagnostics, writes the bootstrap ahead of the script bundle and the tag that loads the diagnostics module straight after it | -| `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | -| `js_asset_proxy` | `[[fetch]]` | Points each configured script at its first-party path and removes one that is blocked. It is asked about an address as the middleware named before it left it, so name it first | +| Middleware | Phase | What it changes | +| ----------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | +| `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | +| `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | +| `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | +| `cmp.sourcepoint` | `[[fetch]]` | Writes `window.__tsjs_sourcepoint` into the head and, when `rewrite_sdk` is set, the trap on `window._sp_`, and points a `src` or `href` on Sourcepoint's CDN at `/integrations/sourcepoint/cdn` | +| `tag.google-tag-manager` | `[[fetch]]` | Points Google Tag Manager and Google Analytics addresses at `/integrations/google_tag_manager`, in `src` and `href` attributes and in the text of inline scripts | +| `bot-protection.datadome` | `[[fetch]]` | Points a `src` or `href` that is a DataDome script's address at `/integrations/datadome`, when `rewrite_sdk` is set | +| `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | +| `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | +| `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | +| `ad-tag.google.diagnostics` | `[[serve]]` | For a request that activated diagnostics, writes the bootstrap ahead of the script bundle and the tag that loads the diagnostics module straight after it | +| `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | +| `js_asset_proxy` | `[[fetch]]` | Points each configured script at its first-party path and removes one that is blocked. It is asked about an address as the middleware named before it left it, so name it ahead of one that moves the same script | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type @@ -1941,8 +1941,8 @@ Each `[[proxy.js_asset_proxy.assets]]` entry: Rewriting and removing script tags is the middleware `js_asset_proxy`, which runs on the pages a `[[fetch]]` entry names it for. It knows a script by the address it is handed, which is the address as the middleware named before it -left it, so name it first in the entry. See -[Placing page changes](#placing-page-changes). +left it, so name it ahead of a middleware that moves the address of a script +it is to proxy or block. See [Placing page changes](#placing-page-changes). ### lockr Integration diff --git a/docs/guide/creative-processing.md b/docs/guide/creative-processing.md index d491e852b2..5d35e5ccbf 100644 --- a/docs/guide/creative-processing.md +++ b/docs/guide/creative-processing.md @@ -698,86 +698,67 @@ exclude_domains = ["assets.publisher.com"] Skip resources already on your domain. -## Integration Hooks +## Page Changes by Modules -### Attribute Rewriters +A module changes a page through a middleware, which runs on the pages a +`[[fetch]]` or `[[serve]]` entry names it for. A middleware may do any of +these to a document. -Integrations can override attribute rewriting: - -**Example**: Next.js integration rewrites origin URLs - -```rust -impl IntegrationAttributeRewriter for NextJsIntegration { - fn rewrite(&self, attr_name: &str, attr_value: &str, ctx: &Context) - -> AttributeRewriteAction - { - if attr_name == "href" && attr_value.contains(&ctx.origin_host) { - let rewritten = attr_value.replace(&ctx.origin_host, &ctx.request_host); - return AttributeRewriteAction::replace(rewritten); - } - AttributeRewriteAction::keep() - } -} -``` - -**Actions**: +**Decide about an attribute** of the elements a CSS selector matches: - `keep()` - Leave attribute unchanged - `replace(value)` - Change attribute value - `remove_element()` - Delete entire element -### Script Rewriters +**Decide about the text** inside the elements a selector matches, such as a +script's: + +- `keep()` - Leave the text unchanged +- `replace(content)` - Replace the text +- `remove_node()` - Remove the text -Integrations can modify `<script>` content: +**Write markup into `<head>`**, ahead of the unified TSJS bundle or straight +after it. -**Example**: Next.js rewrites `__NEXT_DATA__` JSON +**Run a stream processor** over the document the handlers left. + +**Example**: a middleware that moves the origin's address in an attribute +core does not rewrite ```rust -impl IntegrationScriptRewriter for NextJsIntegration { - fn selector(&self) -> &'static str { - "script#__NEXT_DATA__" +impl Middleware for ExampleLinks { + fn middleware_id(&self) -> &'static str { + "framework.example" } - fn rewrite(&self, content: &str, ctx: &Context) -> ScriptRewriteAction { - let rewritten = rewrite_next_data_urls(content, ctx); - ScriptRewriteAction::replace(rewritten) + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } -} -``` - -**Actions**: - -- `keep()` - Leave script unchanged -- `replace(content)` - Replace script content -- `remove_node()` - Delete script element - -### Head Injectors - -Integrations can inject HTML snippets at the start of `<head>`, immediately after the unified TSJS bundle: -**Example**: An integration injects configuration that runs after the TSJS API is available - -```rust -impl IntegrationHeadInjector for MyIntegration { - fn integration_id(&self) -> &'static str { "my_integration" } - - fn head_inserts(&self, ctx: &IntegrationHtmlContext<'_>) -> Vec<String> { - vec![format!( - r#"<script>tsjs.setConfig({{ host: "{}" }});</script>"#, - ctx.request_host - )] + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + let origin = context.origin_host.to_owned(); + let host = context.request_host.to_owned(); + MiddlewareAction { + element_handlers: vec![Box::new(AttributeRewrite::new( + "data-link", + Rc::new(move |matched| { + if matched.value.contains(&origin) { + AttributeRewriteAction::replace(matched.value.replace(&origin, &host)) + } else { + AttributeRewriteAction::keep() + } + }), + ))], + ..MiddlewareAction::pass() + } } } ``` -**Behavior**: - -- Snippets are prepended into `<head>` after the TSJS bundle tag -- Called once per HTML response -- Multiple integrations can each contribute snippets -- If no snippets are returned, no extra markup is added - -See [Integration Guide](/guide/integration-guide) for creating custom rewriters. +See [Changing a page](/guide/integration-guide#changing-a-page) for what a +middleware can rely on, and +[Placing page changes](/guide/configuration#placing-page-changes) for the +entries that place one. ## TSJS Injection diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index d6b3378d60..3cc955aaf4 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -11,14 +11,10 @@ Axum, or Spin SDK types. EdgeZero-neutral request and its call into the request's module context, from which it names the `Settings`, the `RuntimeServices` and whatever else it needs. -- `IntegrationAttributeRewriter` inspects selected HTML attributes. -- `IntegrationScriptRewriter` handles one declared selector. -- `IntegrationHeadInjector` inserts deterministic head markup. -- `IntegrationHtmlPostProcessor` is for bounded whole-document work that - cannot be performed during streaming. - `IntegrationRequestFilter` makes an early request decision. - `Middleware` changes a page, on the pages where the operator's settings - place it. See [Changing a page](#changing-a-page). + place it, and is the one way a module changes a page. See + [Changing a page](#changing-a-page). Build one `IntegrationRegistration` with only the hooks the feature needs. Use `with_deferred_js()` only for a separately loaded integration module and @@ -27,7 +23,7 @@ namespaced and bounded; do not introduce a general outbound proxy. ## Compiling core-neutral fixture -The fixture below registers an attribute rewriter and constructs every +The fixture below registers a middleware and constructs every required `RuntimeServices` service without an adapter dependency. The documentation test extracts this exact fence and compiles it as an isolated crate. @@ -36,12 +32,16 @@ crate. ```rust use std::net::IpAddr; +use std::rc::Rc; use std::sync::Arc; use error_stack::Report; use trusted_server_core::integrations::{ - AttributeRewriteAction, IntegrationAttributeContext, - IntegrationAttributeRewriter, IntegrationRegistration, + AttributeRewriteAction, IntegrationRegistration, +}; +use trusted_server_core::middleware::{ + AttributeRewrite, AttributeRewriteFn, Middleware, MiddlewareAction, + MiddlewareContext, MiddlewarePhase, }; use trusted_server_core::platform::{ BackendNamingPolicy, ClientInfo, GeoInfo, PlatformBackend, @@ -145,33 +145,35 @@ impl PlatformGeo for FixtureGeo { } } -struct AssetRewriter; +struct AssetAddress; -impl IntegrationAttributeRewriter for AssetRewriter { - fn integration_id(&self) -> &'static str { - "example" +impl Middleware for AssetAddress { + fn middleware_id(&self) -> &'static str { + "testing.example" } - fn handles_attribute(&self, attribute: &str) -> bool { - matches!(attribute, "src" | "href") + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] } - fn rewrite( - &self, - _attribute: &str, - value: &str, - _context: &IntegrationAttributeContext<'_>, - ) -> AttributeRewriteAction { - value - .strip_prefix("https://assets.example/") - .map(|path| AttributeRewriteAction::replace(format!("/assets/{path}"))) - .unwrap_or_else(AttributeRewriteAction::keep) + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + let decide: Rc<AttributeRewriteFn> = Rc::new(|matched| { + matched + .value + .strip_prefix("https://assets.example/") + .map(|path| AttributeRewriteAction::replace(format!("/assets/{path}"))) + .unwrap_or_else(AttributeRewriteAction::keep) + }); + MiddlewareAction { + element_handlers: AttributeRewrite::each(&["src", "href"], &decide), + ..MiddlewareAction::pass() + } } } pub fn registration() -> IntegrationRegistration { IntegrationRegistration::builder("example") - .with_attribute_rewriter(Arc::new(AssetRewriter)) + .with_middleware(Arc::new(AssetAddress)) .build() } @@ -303,10 +305,6 @@ builder every integration uses. | Declaration | What it does | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `.with_proxy(...)` | Routes the paths the proxy declares | -| `.with_head_injector(...)` | Emits markup at the start of `<head>` and, if it chooses, after the script bundle | -| `.with_attribute_rewriter(...)` | Rewrites attribute values in publisher HTML | -| `.with_script_rewriter(...)` | Rewrites inline script contents | -| `.with_html_stream_processor(...)` | Works on the document as it streams | | `.with_request_filter(...)` | Inspects a request and can turn it back before it reaches the origin | | `.with_middleware(...)` | Offers a page change that an entry may place by the middleware's name | | `.with_bundle_tag_attribute(name, value)` | Puts an attribute on the script bundle's tag, for a browser module that reads a setting from it | @@ -348,10 +346,10 @@ module of that type. ### Changing a page -A middleware is a page change that the operator's settings place. The module -registers each one with `.with_middleware(...)`, and it runs only on the -pages where an entry names it, in the order that entry gives. See -[Placing page changes](/guide/configuration#placing-page-changes). +A module changes a page through a middleware, which the operator's settings +place. The module registers each one with `.with_middleware(...)`, and it +runs only on the pages where an entry names it, in the order that entry +gives. See [Placing page changes](/guide/configuration#placing-page-changes). The fixture below loads a vendor's tag from a first-party path and marks the head. The documentation test extracts this exact fence and compiles it as an @@ -564,12 +562,13 @@ preparer or a request filter leaves a value under the integration's id. IntegrationRequestState::insert(request, EXAMPLE_ID, ExampleDecision::default()); ``` -When the request produces an HTML document, each value is copied into the -document's state before parsing starts, so the head injector, the rewriters -and the stream processors read it with -`ctx.document_state.get::<ExampleDecision>(EXAMPLE_ID)`. The same values are -handed to the response finalizer the builder declared, which runs on the -response the page path returns. +Where the request's reader is served an HTML document, each value is copied +into the state the `[[serve]]` entry's middleware share, so the module's +serve middleware reads it with +`context.document_state.get::<ExampleDecision>(EXAMPLE_ID)`. A fetch +middleware is handed none of it. The same values are handed to the response +finalizer the builder declared, which runs on the response the page path +returns. A request that carries any value gets a document made for it alone. The document is fetched from the origin, never read from a shared template and @@ -577,10 +576,10 @@ never stored as one, and an HTML response with a body is sent `private, no-store`. So a module leaves a value only for a request it will act on, and an ordinary request stays on the shared path. -A head injector has two places to write. `head_inserts` writes at the start -of `<head>`, before the script bundle, and `after_bundle_inserts` writes -straight after the bundle, for a script that needs the bundle to have run and -has to run before the page's own scripts. +A serve middleware has two places to write markup. `head_inserts` writes +straight before the script bundle, and `after_bundle_inserts` writes straight +after it, for a script that needs the bundle to have run and has to run +before the page's own scripts. ### A module's own secrets @@ -673,13 +672,14 @@ probe through the Fastly adapter under Viceroy. ### The modules a stock build ships `crates/trusted-server-modules` lists the modules a stock build ships from -crates of their own, in the order their hooks run. Every adapter builds its -state with that list followed by the builders a deployment added, and loads -its settings against the same list. The `ts` tool registers it for deploy -validation. A module joins a stock build by adding its crate and its builder -to that list, in the place its hooks should run. Offering a module does not -run it, because the registry builds a module only when a section of the -settings selects it. +crates of their own, in the order their request hooks run. Every adapter +builds its state with that list followed by the builders a deployment added, +and loads its settings against the same list. The `ts` tool registers it for +deploy validation. A module joins a stock build by adding its crate and its +builder to that list, in the place its request hooks should run. The order of +the list does not order page changes, which run in the order an entry names +them. Offering a module does not run it, because the registry builds a module +only when a section of the settings selects it. ### Two traps a vendor will hit diff --git a/docs/guide/integrations-overview.md b/docs/guide/integrations-overview.md index 8d95f6bec6..e546db809a 100644 --- a/docs/guide/integrations-overview.md +++ b/docs/guide/integrations-overview.md @@ -67,8 +67,9 @@ families and adapter availability. Selected integrations share ordered request, HTML, and browser pipelines. They are not guaranteed to be independent. Registration predicates determine which -proxy routes, attribute or script rewriters, head injectors, request filters, -post-processors, auction providers, and browser modules are active. Validate +proxy routes, middleware, request filters, auction providers, and browser +modules are active, and a middleware changes a page only where a `[[fetch]]` +or `[[serve]]` entry names it. Validate the complete configuration and test the resulting page rather than assuming that any arbitrary combination is conflict-free. diff --git a/docs/guide/integrations/gpt.md b/docs/guide/integrations/gpt.md index afbae5f12c..2196c3da49 100644 --- a/docs/guide/integrations/gpt.md +++ b/docs/guide/integrations/gpt.md @@ -25,7 +25,7 @@ All of these are served from `securepubads.g.doubleclick.net`. Publisher HTML │ ├─ <script src="securepubads.g.doubleclick.net/tag/js/gpt.js"> - │ ↓ (attribute rewriter) + │ ↓ (middleware) │ <script src="publisher.com/integrations/gpt/script"> │ ├─ Server fetches gpt.js from Google, serves it verbatim From 3352f8de60362940ce7d34332d72d701cc35166e Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 9 Oct 2026 06:37:45 +0100 Subject: [PATCH 28/36] The stock list says which middleware its modules supply A tool that writes a configuration, as `ts audit` does, selects the modules it finds on a page. A module's page changes run only where an entry names its middleware, and a tool had no way to learn those names without naming a module itself. `trusted-server-modules` now lists the middleware the stock modules supply, core's script proxy among them, each with the integration that supplies it and the phase it runs in, in the order the registry registers the modules. `middleware_for` gives the names for a set of integration ids and a phase. The list is held to what the modules register. One test builds a registry from the settings each module's recorded page is made with and compares what it registers for each phase, and another holds the order to the stock builders'. The registry gains `middleware_in`, which gives the names that run in a phase. --- .../src/integrations/registry.rs | 22 ++ crates/trusted-server-modules/src/lib.rs | 207 +++++++++++++++++- 2 files changed, 227 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 8dd11ae9ec..fc8284fb8f 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -1789,6 +1789,18 @@ impl IntegrationRegistry { .collect() } + /// The names of the middleware that run in `phase`, in registration + /// order. + #[must_use] + pub fn middleware_in(&self, phase: MiddlewarePhase) -> Vec<&'static str> { + self.inner + .middleware + .iter() + .filter(|(_, middleware)| middleware.phases().contains(&phase)) + .map(|(_, middleware)| middleware.middleware_id()) + .collect() + } + /// The middleware the first entry covering a response selects, in the /// order they run, for a response of `media_type` to a request for `path`. /// A response no entry covers gives an empty chain. @@ -5878,6 +5890,16 @@ mod tests { ], "should list the middleware the selected module supplies" ); + assert_eq!( + registry.middleware_in(MiddlewarePhase::Fetch), + [fixture::HEAD, fixture::LINKS, fixture::BROKEN], + "should list the middleware that run in the fetch phase" + ); + assert_eq!( + registry.middleware_in(MiddlewarePhase::Serve), + [fixture::READER, fixture::BROKEN_READER], + "should list the middleware that run in the serve phase" + ); let chain_for = |media_type: &str, path: &str| { registry .middleware_chain(&settings.fetch, MiddlewarePhase::Fetch, media_type, path) diff --git a/crates/trusted-server-modules/src/lib.rs b/crates/trusted-server-modules/src/lib.rs index 2951f12b33..623d098ba2 100644 --- a/crates/trusted-server-modules/src/lib.rs +++ b/crates/trusted-server-modules/src/lib.rs @@ -5,6 +5,10 @@ //! hooks run. Offering a module does not run it, because the registry builds a //! module only when a section of the settings selects it. //! +//! It also says which middleware those modules supply, so a tool that writes +//! a configuration can place a module's page changes without naming one +//! itself. +//! //! A deployment that ships a module of its own hands that module's builder to //! its adapter, which runs it after these. @@ -14,6 +18,7 @@ )] use trusted_server_core::integrations::IntegrationBuilder; +use trusted_server_core::middleware::MiddlewarePhase; /// The builders of the modules a stock build ships from crates of their own, /// in hook order. @@ -68,11 +73,209 @@ pub fn selection_of(id: &str) -> Option<(&'static str, &'static str)> { )) } +/// A middleware a stock module supplies. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct StockMiddleware { + /// The id of the integration that supplies it. + pub integration: &'static str, + /// The phase it runs in. + pub phase: MiddlewarePhase, + /// The name an entry writes. + pub name: &'static str, +} + +/// The middleware a stock build's modules supply, core's script proxy among +/// them, in the order a deployment that runs them all names them in an +/// entry. +/// +/// That is the order the registry registers the modules in. Prebid registers +/// from the auction plan and so comes first, core's script proxy is the first +/// module a section selects, and the rest follow [`builders`]. +#[must_use] +pub fn middleware() -> Vec<StockMiddleware> { + use MiddlewarePhase::{Fetch, Serve}; + use trusted_server_ad_tag_google as google; + use trusted_server_auction_prebid as prebid; + use trusted_server_audience_permutive as permutive; + use trusted_server_bot_protection_datadome as datadome; + use trusted_server_cmp_didomi as didomi; + use trusted_server_cmp_sourcepoint as sourcepoint; + use trusted_server_core::integrations::js_asset_proxy; + use trusted_server_framework_nextjs as nextjs; + use trusted_server_identity_lockr as lockr; + use trusted_server_tag_google_tag_manager as google_tag_manager; + use trusted_server_testing_testlight as testlight; + + let row = |integration, phase, name| StockMiddleware { + integration, + phase, + name, + }; + vec![ + row(prebid::builder().id(), Fetch, prebid::MODULE), + // Core registers its script proxy under the module's own name. + row(js_asset_proxy::MODULE, Fetch, js_asset_proxy::MODULE), + row(testlight::builder().id(), Fetch, testlight::MODULE), + row(nextjs::builder().id(), Fetch, nextjs::MODULE), + row(permutive::builder().id(), Fetch, permutive::MODULE), + row(lockr::builder().id(), Fetch, lockr::MODULE), + row(didomi::builder().id(), Fetch, didomi::MODULE), + row(sourcepoint::builder().id(), Fetch, sourcepoint::MODULE), + row( + google_tag_manager::builder().id(), + Fetch, + google_tag_manager::MODULE, + ), + row(datadome::builder().id(), Fetch, datadome::MODULE), + row(datadome::builder().id(), Serve, datadome::TAG_MIDDLEWARE), + row(google::builder().id(), Fetch, google::MODULE), + row( + google::diagnostics::builder().id(), + Serve, + google::diagnostics::MODULE, + ), + ] +} + +/// The names of the middleware that the stock modules with the integration +/// ids `ids` supply for `phase`, in the order an entry names them. +/// +/// A tool that knows a module by its integration id, as the `ts` audit does +/// from what it finds on a page, writes the entry that places the module's +/// page changes with this and names no middleware itself. +#[must_use] +pub fn middleware_for(ids: &[&str], phase: MiddlewarePhase) -> Vec<&'static str> { + middleware() + .into_iter() + .filter(|row| row.phase == phase && ids.contains(&row.integration)) + .map(|row| row.name) + .collect() +} + #[cfg(test)] mod tests { - use trusted_server_core::integrations::IntegrationBuilder; + use trusted_server_core::integrations::{IntegrationBuilder, IntegrationRegistry}; + use trusted_server_core::middleware::MiddlewarePhase; + + use super::{builders, builders_with, middleware, middleware_for, selection_of}; + + /// The settings each stock module's recorded page is made with, which + /// select the module and place every middleware it supplies. + const PAGE_SETTINGS: &[&str] = &[ + include_str!("../../auction/prebid/src/fixtures/page-change.settings.toml"), + include_str!( + "../../trusted-server-core/src/integrations/fixtures/js-asset-proxy.settings.toml" + ), + include_str!("../../testing/testlight/src/fixtures/page-change.settings.toml"), + include_str!("../../framework/nextjs/src/fixtures/page-change.settings.toml"), + include_str!("../../audience/permutive/src/fixtures/page-change.settings.toml"), + include_str!("../../identity/lockr/src/fixtures/page-change.settings.toml"), + include_str!("../../cmp/didomi/src/fixtures/page-change.settings.toml"), + include_str!("../../cmp/sourcepoint/src/fixtures/page-change.settings.toml"), + include_str!("../../tag/google-tag-manager/src/fixtures/page-change.settings.toml"), + include_str!("../../bot-protection/datadome/src/fixtures/page-change.settings.toml"), + include_str!("../../ad-tag/google/src/fixtures/page-change.settings.toml"), + ]; + + #[test] + fn the_listed_middleware_are_the_ones_the_modules_register() { + let listed = middleware(); + let mut checked: Vec<&str> = Vec::new(); + + for module_settings in PAGE_SETTINGS { + let settings = + trusted_server_core::html_processor::test_support::page_settings(module_settings); + let registry = IntegrationRegistry::with_registrations(&settings, &builders()) + .expect("should build a registry from a recorded page's settings"); + let running: Vec<&str> = registry + .registered_integrations() + .iter() + .map(|integration| integration.id) + .collect(); + + for phase in MiddlewarePhase::ALL { + let expected: Vec<&str> = listed + .iter() + .filter(|row| row.phase == phase && running.contains(&row.integration)) + .map(|row| row.name) + .collect(); + assert_eq!( + registry.middleware_in(phase), + expected, + "should list for {phase} what the modules {running:?} register for it" + ); + } + checked.extend(running); + } + + for row in &listed { + assert!( + checked.contains(&row.integration), + "should check `{}` against a module that registers it, and none of the \ + recorded pages runs `{}`", + row.name, + row.integration + ); + } + } + + #[test] + fn the_listed_middleware_follow_the_order_the_modules_register_in() { + // What registers from the auction plan goes first, core's own module + // is the first a section selects, and the stock modules follow. + let mut order = vec![ + trusted_server_auction_prebid::builder().id(), + trusted_server_core::integrations::js_asset_proxy::MODULE, + ]; + for builder in builders() { + if !order.contains(&builder.id()) { + order.push(builder.id()); + } + } + let place = |integration: &str| { + order + .iter() + .position(|id| *id == integration) + .unwrap_or_else(|| panic!("`{integration}` should be a stock module")) + }; + + for phase in MiddlewarePhase::ALL { + let places: Vec<usize> = middleware() + .iter() + .filter(|row| row.phase == phase) + .map(|row| place(row.integration)) + .collect(); + assert!( + places.is_sorted(), + "should list the {phase} middleware in the order their modules register: \ + {places:?}" + ); + } + } - use super::{builders, builders_with, selection_of}; + #[test] + fn a_tool_is_handed_a_module_s_middleware_by_its_integration_id() { + use trusted_server_bot_protection_datadome as datadome; + use trusted_server_cmp_didomi as didomi; + + let ids = [datadome::builder().id(), didomi::builder().id()]; + + assert_eq!( + middleware_for(&ids, MiddlewarePhase::Fetch), + [didomi::MODULE, datadome::MODULE], + "should name the modules' fetch middleware in the order an entry runs them, \ + whatever order the ids came in" + ); + assert_eq!( + middleware_for(&ids, MiddlewarePhase::Serve), + [datadome::TAG_MIDDLEWARE], + "should name the one serve middleware the two supply" + ); + assert!( + middleware_for(&["no_such_module"], MiddlewarePhase::Fetch).is_empty(), + "should name nothing for an id no stock module has" + ); + } #[test] fn stock_modules_are_offered_in_hook_order() { From 21623bb5b069153128fce5ade90d8ce1c834c970 Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 9 Oct 2026 07:06:06 +0100 Subject: [PATCH 29/36] `ts audit` places the page changes of the modules it selects `ts audit generate` writes a draft configuration that selects the modules it can configure from what it finds on a page. A module changes a page only where an entry names its middleware, so a draft with no entry selected modules that changed nothing. The draft now ends with one `[[fetch]]` entry, and one `[[serve]]` entry where a selected module supplies a serve middleware, that run those modules' middleware on every HTML page. The names and their order come from the stock list, so the tool names no middleware itself. The script proxy is placed with the rest, so an asset an operator enables after review is rewritten as well as served. `trusted-server.example.toml` documents the two entries with every middleware the stock modules supply. The stock list's tests hold the template's entries and the configuration guide's table to the list, in order, so neither can fall behind a module. The guide's table is put in that order, and the changelog says how a deployment keeps a page as it was when it upgrades. The lockr guide and the getting started guide say a page change needs an entry, as the other module guides do. --- CHANGELOG.md | 3 +- .../src/commands/audit/generate/mod.rs | 186 +++++++++++++++++- crates/trusted-server-modules/src/lib.rs | 67 +++++++ docs/guide/cli.md | 5 +- docs/guide/configuration.md | 10 +- docs/guide/getting-started.md | 1 + docs/guide/integrations/lockr.md | 11 ++ trusted-server.example.toml | 28 +++ 8 files changed, 304 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a03e219acc..0277463cd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware, and no longer on every page of a deployment that selects the module. A deployment adds the entries when it upgrades. Until it does, a module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. +- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware, and no longer on every page of a deployment that selects the module. A deployment adds the entries when it upgrades. Until it does, a module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. To keep a page as it was, name the middleware of the modules a deployment selects in one `[[fetch]]` entry and one `[[serve]]` entry for `text/html`, in the order `trusted-server.example.toml` shows them, which is the order the modules' hooks ran in. - **Breaking:** The four page hook traits are removed, being `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector`, with their context types, `ScriptTextAccumulator`, `AttributeRewriteOutcome` and the registration methods that took them. A module that implemented one supplies a `Middleware` instead, whose action carries the same decisions. A browser module's attribute on the script bundle's tag is stated with `with_bundle_tag_attribute`. - A pushed configuration keeps DataDome's two deprecated secret store selectors, `server_side_key_secret_store` and `protection_test_bypass.credential_secret_store`, where they were removed as the settings were read. DataDome still accepts and discards both, so a configuration that carries one loads as before. - Integration modules have moved out of `trusted-server-core` into crates of their own under `crates/<type>/<vendor>/`, each named by that folder, so `crates/cmp/osano` is the module `cmp.osano`. The settings that select and configure a module do not change. `crates/trusted-server-modules` lists the modules a stock build ships from crates of their own, in the order their hooks run. Every adapter builds its state and loads its settings with that list, and `ts config validate` and `ts config push` run each stock module's own rules. Code that named a moved module under `trusted_server_core::integrations` names its crate instead, such as `trusted_server_cmp_osano`. Moved: Osano, Lockr, Permutive, Sourcepoint, Didomi, Google Tag Manager, Testlight, Next.js, Google Publisher Tags and its diagnostics, DataDome, the mock ad server, APS, Prebid, the Prebid Server demand and the plain OpenRTB demand. Core keeps the JavaScript asset proxy, which is its own. @@ -66,6 +66,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `ts audit generate` writes the `[[fetch]]` and `[[serve]]` entries that run the page changes of the modules it selects on every HTML page, and `trusted-server.example.toml` documents the entries with every middleware the stock modules supply. `trusted-server-modules` lists those middleware with `middleware()` and `middleware_for(...)`. - Page changes as middleware. A module registers a change it makes to a page with `.with_middleware(...)`, and the ordered `[[fetch]]` and `[[serve]]` entries of the settings say which pages each one runs on and in what order. A fetch middleware runs on the page as the origin sent it and what it writes is stored with a shared template. A serve middleware runs on each reader's copy, whether the page came from the store or from the origin, and nothing it writes is stored. - A builder registers its module from the compiled auction plan with `with_plan_registration` and checks its settings against the plan with `with_plan_validator`, so a module whose page support follows what the plan selects is not named in `trusted-server-core`. Prebid and APS register this way. A renderer descriptor carries its builder's statement of which payload key holds the identifier the renderer picks its bid by, made with `BidRenderer::picking_bid_by`, and the page's `hb_adid` is read through that statement. - A module declares the settings in its own table that hold the name of a secret, with `with_secret_settings` on its builder, each with a rule for when the table puts it to use. A setting in use is looked up in the default secret store as the settings load and has to name a key for deploy validation to pass, and one not in use is cleared and never looked up. DataDome's server-side key and its test bypass credential are declared this way, and `trusted-server-core` no longer names them. diff --git a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs index 1415123e35..858ca05eeb 100644 --- a/crates/trusted-server-cli/src/commands/audit/generate/mod.rs +++ b/crates/trusted-server-cli/src/commands/audit/generate/mod.rs @@ -19,6 +19,7 @@ use serde::Serialize; use trusted_server_core::creative_opportunities::{ CreativeOpportunitiesConfig, validate_page_pattern, }; +use trusted_server_core::middleware::MiddlewarePhase; use url::Url; use crate::commands::audit::ad_templates::{origin_changed, without_fragment}; @@ -160,6 +161,7 @@ struct DraftConfig { /// Id of the integration the audit writes its discovered assets for. const JS_ASSET_PROXY_ID: &str = "js_asset_proxy"; +const GOOGLE_TAG_MANAGER_ID: &str = "google_tag_manager"; #[derive(Debug, Clone)] struct JsAssetProxySection { @@ -480,7 +482,9 @@ fn build_draft_config_with_generator( } // The template documents every module as a commented example, so the - // sections the audit fills in are appended after it. + // sections the audit fills in are appended after it. The integrations + // whose modules it selects are kept, so their page changes can be placed. + let mut selected: Vec<&str> = Vec::new(); for id in ["datadome", "didomi", "gpt"] { if detected.contains(id) && let Some((section, written)) = module_section(id) @@ -493,16 +497,22 @@ fn build_draft_config_with_generator( modules = [\"{written}\"]\n" ), ); + selected.push(id); } } if let Some(container_id) = >m_container_id { append_section(&mut draft, &build_google_tag_manager_section(container_id)); + selected.push(GOOGLE_TAG_MANAGER_ID); } if asset_proxy_section.candidate_count > 0 { append_section(&mut draft, &asset_proxy_section.toml); + selected.push(JS_ASSET_PROXY_ID); } else { append_section(&mut draft, &asset_proxy_section.notes); } + if let Some(entries) = build_page_entries(&selected) { + append_section(&mut draft, &entries); + } let mut manual_review = Vec::new(); if detected.contains("google_tag_manager") && gtm_container_id.is_none() { @@ -652,6 +662,40 @@ fn build_google_tag_manager_section(container_id: &str) -> String { ) } +/// The entries that run the middleware of the selected modules on every +/// HTML page, or `None` when none of them changes a page. +/// +/// A module changes a page only where an entry names its middleware, so a +/// draft that selected a module and wrote no entry would leave the module +/// changing nothing. The names and their order come from the stock list. +fn build_page_entries(integration_ids: &[&str]) -> Option<String> { + let mut entries = String::new(); + for phase in MiddlewarePhase::ALL { + let names = trusted_server_modules::middleware_for(integration_ids, phase); + if names.is_empty() { + continue; + } + if entries.is_empty() { + entries.push_str( + "# Written by `ts audit`. A module changes a page only where an entry names\n\ + # its middleware, so these run what the modules selected above supply on\n\ + # every HTML page, in the order written.\n", + ); + } else { + entries.push('\n'); + } + let listed = names + .iter() + .map(|name| toml_string(name)) + .collect::<Vec<_>>() + .join(", "); + entries.push_str(&format!( + "[[{phase}]]\nmedia_type = \"text/html\"\nmiddleware = [{listed}]\n" + )); + } + (!entries.is_empty()).then_some(entries) +} + /// Selects the JavaScript asset proxy in the template's `[proxy]` section by /// uncommenting the selection the template documents there. fn select_js_asset_proxy(draft: &str) -> CliResult<String> { @@ -2741,6 +2785,146 @@ mod tests { ); } + /// What the draft's entries of `phase` hold, each as its media type, + /// whether it has a path, and the middleware it names. + fn drafted_entries(parsed: &toml::Value, phase: &str) -> Vec<(String, bool, Vec<String>)> { + parsed + .get(phase) + .and_then(toml::Value::as_array) + .map(|entries| { + entries + .iter() + .map(|entry| { + ( + entry["media_type"] + .as_str() + .expect("should write a media type") + .to_owned(), + entry.get("path").is_some(), + entry["middleware"] + .as_array() + .expect("should write the middleware") + .iter() + .filter_map(toml::Value::as_str) + .map(str::to_owned) + .collect(), + ) + }) + .collect() + }) + .unwrap_or_default() + } + + #[test] + fn build_draft_config_places_the_page_changes_of_the_modules_it_selects() { + let url = Url::parse("https://publisher.example.com/page").expect("should parse URL"); + let mut detected_integrations: Vec<DetectedIntegration> = ["gpt", "datadome", "didomi"] + .into_iter() + .map(|id| DetectedIntegration { + id: id.to_string(), + evidence: format!("https://{id}.example.com/tag.js"), + }) + .collect(); + detected_integrations.push(DetectedIntegration { + id: "google_tag_manager".to_string(), + evidence: "GTM-ABC123".to_string(), + }); + let artifact = AuditArtifact { + audited_url: url.to_string(), + page_title: Some("Example".to_string()), + js_asset_count: 1, + third_party_asset_count: 1, + detected_integrations, + assets: vec![audited_asset( + "https://cdn.vendor.example.com/sdk.js", + AssetParty::ThirdParty, + None, + )], + warnings: Vec::new(), + }; + let mut generator = FixedPathGenerator::new(&["/assets/aaaaaaaaaaaaaaaaaaaaaaaa.js"]); + + let draft = build_draft_config_with_generator( + &url, + &artifact, + &gpt_slots::DiscoveredSlots::default(), + &mut generator, + ) + .expect("should build draft config"); + + let parsed = + toml::from_str::<toml::Value>(&draft.toml).expect("draft should parse as TOML"); + assert_eq!( + drafted_entries(&parsed, "fetch"), + vec![( + "text/html".to_owned(), + false, + vec![ + "js_asset_proxy".to_owned(), + "cmp.didomi".to_owned(), + "tag.google-tag-manager".to_owned(), + "bot-protection.datadome".to_owned(), + "ad-tag.google".to_owned(), + ], + )], + "should run the fetch middleware of every module it selected on every HTML \ + page, in the order the stock modules register" + ); + assert_eq!( + drafted_entries(&parsed, "serve"), + vec![( + "text/html".to_owned(), + false, + vec!["bot-protection.datadome.tag".to_owned()], + )], + "should run the one serve middleware the selected modules supply" + ); + trusted_server_core::settings::Settings::from_toml( + &draft + .toml + .replace( + "password = \"handler_password\"", + "password = \"test-admin-password-32-bytes-minimum\"", + ) + .replace( + "passphrase = \"ec_passphrase\"", + "passphrase = \"test-ec-passphrase-32-bytes-minimum\"", + ) + .replace( + "proxy_secret = \"publisher_proxy_secret\"", + "proxy_secret = \"test-proxy-secret-32-bytes-minimum\"", + ), + ) + .expect("should load the draft, its entries included, as settings"); + } + + #[test] + fn build_draft_config_writes_no_entry_when_it_selects_no_module() { + let url = Url::parse("https://publisher.example/path").expect("should parse URL"); + let artifact = AuditArtifact { + audited_url: url.to_string(), + page_title: None, + js_asset_count: 0, + third_party_asset_count: 0, + detected_integrations: vec![DetectedIntegration { + id: "prebid".to_string(), + evidence: "inline script matched `prebid`".to_string(), + }], + assets: Vec::new(), + warnings: Vec::new(), + }; + + let draft = build_draft_config(&url, &artifact, &gpt_slots::DiscoveredSlots::default()) + .expect("should build draft config"); + + let parsed = toml::from_str::<toml::Value>(&draft).expect("draft should parse as TOML"); + assert!( + drafted_entries(&parsed, "fetch").is_empty() + && drafted_entries(&parsed, "serve").is_empty(), + "should place nothing for a module it leaves to manual review" + ); + } + #[test] fn build_draft_config_does_not_name_gtm_without_container_id() { let url = Url::parse("https://publisher.example/path").expect("should parse URL"); diff --git a/crates/trusted-server-modules/src/lib.rs b/crates/trusted-server-modules/src/lib.rs index 623d098ba2..c55f0ba443 100644 --- a/crates/trusted-server-modules/src/lib.rs +++ b/crates/trusted-server-modules/src/lib.rs @@ -253,6 +253,73 @@ mod tests { } } + #[test] + fn the_template_s_entries_name_the_listed_middleware() { + use trusted_server_core::settings::Settings; + use trusted_server_core::test_support::template::{ + template_with_resolved_required_secrets, uncomment_block, + }; + + let template = uncomment_block( + &uncomment_block(&template_with_resolved_required_secrets(), "[[fetch]]"), + "[[serve]]", + ); + let settings = + Settings::from_toml(&template).expect("should load the template with its entries"); + + for phase in MiddlewarePhase::ALL { + let listed: Vec<&str> = middleware() + .iter() + .filter(|row| row.phase == phase) + .map(|row| row.name) + .collect(); + let entries = settings.phase_entries(phase).entries(); + assert_eq!( + entries.len(), + 1, + "should document one [[{phase}]] entry for every page" + ); + assert!( + entries[0].path.is_none(), + "should document the [[{phase}]] entry that covers every page" + ); + assert_eq!( + entries[0].middleware, listed, + "should document every {phase} middleware the stock modules supply, in the \ + listed order" + ); + } + } + + #[test] + fn the_guide_s_table_lists_the_same_middleware() { + const GUIDE: &str = include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../docs/guide/configuration.md" + )); + + // A row of the table is a name and then the list it is written in. + let documented: Vec<(String, String)> = GUIDE + .lines() + .filter_map(|line| { + let mut cells = line.split('|').map(str::trim).skip(1); + let name = cells.next()?.strip_prefix('`')?.strip_suffix('`')?; + let list = cells.next()?.strip_prefix("`[[")?.strip_suffix("]]`")?; + Some((name.to_owned(), list.to_owned())) + }) + .collect(); + let listed: Vec<(String, String)> = middleware() + .iter() + .map(|row| (row.name.to_owned(), row.phase.to_string())) + .collect(); + + assert_eq!( + documented, listed, + "should document the middleware the stock modules supply, each with its phase, \ + in the listed order" + ); + } + #[test] fn a_tool_is_handed_a_module_s_middleware_by_its_integration_id() { use trusted_server_bot_protection_datadome as datadome; diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 364f9847cf..4f4d7e4aef 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -305,7 +305,10 @@ ts config validate ``` The draft also selects, in the section of each module's type, the modules it -can configure from what it found, and writes their tables. Where it found +can configure from what it found, and writes their tables. It writes the +`[[fetch]]` and `[[serve]]` entries that run those modules' page changes on +every HTML page, because a module changes a page only where an entry names +its middleware. Where it found third-party scripts it writes `[proxy.js_asset_proxy]` with each one `proxy = "disabled"`, so they are inventory only, and nothing is served or rewritten until you review a candidate and change its `proxy` value to diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 7c74ae8b5a..594d5b6074 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1787,11 +1787,16 @@ table. An entry carries none and switches nothing on. The modules this repository ships supply the middleware below. A deployment that runs several of them names them all in one entry's list, because a page -takes one entry, and the order written is the order they run in. +takes one entry, and the order written is the order they run in. The table +is in the order the modules register, which is the order `ts audit` writes +the names in and the order `trusted-server.example.toml` shows them in. | Middleware | Phase | What it changes | | ----------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | +| `js_asset_proxy` | `[[fetch]]` | Points each configured script at its first-party path and removes one that is blocked. It is asked about an address as the middleware named before it left it, so name it ahead of one that moves the same script | | `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | +| `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | | `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | | `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | | `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | @@ -1799,11 +1804,8 @@ takes one entry, and the order written is the order they run in. | `tag.google-tag-manager` | `[[fetch]]` | Points Google Tag Manager and Google Analytics addresses at `/integrations/google_tag_manager`, in `src` and `href` attributes and in the text of inline scripts | | `bot-protection.datadome` | `[[fetch]]` | Points a `src` or `href` that is a DataDome script's address at `/integrations/datadome`, when `rewrite_sdk` is set | | `bot-protection.datadome.tag` | `[[serve]]` | Writes DataDome's client tag into the head, unless the request filter marked the request, `inject_client_side_tag` is off or no `client_side_key` is set | -| `auction.prebid` | `[[fetch]]` | Writes `window.__tsjs_prebid` and the tag that loads the bundle into the head, and removes an element whose `src` or `href` matches `script_patterns` | | `ad-tag.google` | `[[fetch]]` | Writes the `tsjs.adInit` bootstrap into the head and, when `rewrite_script` is set, points a `src` or `href` that is the GPT script's address at `/integrations/gpt/script` | | `ad-tag.google.diagnostics` | `[[serve]]` | For a request that activated diagnostics, writes the bootstrap ahead of the script bundle and the tag that loads the diagnostics module straight after it | -| `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | -| `js_asset_proxy` | `[[fetch]]` | Points each configured script at its first-party path and removes one that is blocked. It is asked about an address as the middleware named before it left it, so name it ahead of one that moves the same script | An entry that could not do what it says refuses the configuration, both when a deployment is validated and when the settings load. That is a media type diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index b49dd98969..dda6414416 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -175,6 +175,7 @@ only the asset proxy entries you want to serve or block. Then validate it. Edit `trusted-server.toml` to configure: - the page modules that run, each selected in the section of its type (`[cmp]`, `[tag]`, `[ad-tag]`, `[bot-protection]`, `[identity]`, `[audience]`, `[framework]`, and `[auction]` for Prebid), with their settings under `[<section>.<name>]` +- the `[[fetch]]` and `[[serve]]` entries that say which pages each selected module's page changes run on, which the audit's draft writes for the modules it selects - the demand sources, in `[demand] modules`, each with its settings under `[demand.<name>]` - the ad server, if one runs, in `[ad-server] module`, with its settings under `[ad-server.<name>]` - server bidder routes under `[auction.bidders.<code>]` diff --git a/docs/guide/integrations/lockr.md b/docs/guide/integrations/lockr.md index d334f6060d..458e28e89f 100644 --- a/docs/guide/integrations/lockr.md +++ b/docs/guide/integrations/lockr.md @@ -58,6 +58,17 @@ project_id = "your-project-id" | `organization_id` | string | Yes | Your Lockr organization ID | | `project_id` | string | Yes | Your Lockr project ID | +Pointing the lockr SDK's address at the first-party route is the middleware +`identity.lockr`, which runs on the pages a `[[fetch]]` entry names it for: + +```toml +[[fetch]] +media_type = "text/html" +middleware = ["identity.lockr"] +``` + +See [Placing page changes](/guide/configuration#placing-page-changes). + ### Environment Variables ```bash diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 4d5d852718..77385dbac1 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -320,6 +320,34 @@ pull_sync_concurrency = 3 # startup. Nothing is selected by default. The sections and tables are # further down, commented out. +# A module changes a page through a middleware, and a middleware runs only on +# the pages where a [[fetch]] or [[serve]] entry names it. A page takes the +# first entry that covers it and runs that entry's middleware in the order +# written. The two entries below name every middleware the modules further +# down supply, in the order `ts audit` writes them. Keep the names of the +# modules that are selected and remove the rest, because an entry that names +# the middleware of a module no section selects refuses startup. See "Placing +# page changes" in the configuration guide. +# [[fetch]] +# media_type = "text/html" +# middleware = [ +# "auction.prebid", +# "js_asset_proxy", +# "testing.testlight", +# "framework.nextjs", +# "audience.permutive", +# "identity.lockr", +# "cmp.didomi", +# "cmp.sourcepoint", +# "tag.google-tag-manager", +# "bot-protection.datadome", +# "ad-tag.google", +# ] + +# [[serve]] +# media_type = "text/html" +# middleware = ["bot-protection.datadome.tag", "ad-tag.google.diagnostics"] + # First-party proxying of named third-party scripts. Uncomment the # `modules` line under [proxy] below to switch it on. # [proxy.js_asset_proxy] From c4c40764fcce4c66cd420b8b57d24a4817d21c01 Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 9 Oct 2026 07:20:20 +0100 Subject: [PATCH 30/36] What was named for a hook is named for the middleware The tests of a module's page change kept the names of the hooks they once called, such as `head_injector_emits_config_script` and `attribute_rewriter_rewrites_gpt_urls`, which name traits that no longer exist. They are renamed for the parts of a middleware they now test, `head_markup_`, `element_handler_` and `text_handler`. Thirty-nine names and section comments change in eight module crates, and no test's body does. Four comments in core still spoke of script rewriters. They speak of text handlers, and the description of the state a document's middleware share says what it holds in each phase. The test of the payload stand-in's placeholders is renamed, because the stand-in no longer keeps them in that state. --- crates/ad-tag/google/src/lib.rs | 12 ++++---- crates/auction/prebid/src/lib.rs | 28 +++++++++---------- crates/audience/permutive/src/lib.rs | 4 +-- crates/bot-protection/datadome/src/lib.rs | 10 +++---- crates/cmp/didomi/src/lib.rs | 4 +-- crates/cmp/sourcepoint/src/lib.rs | 6 ++-- crates/identity/lockr/src/lib.rs | 6 ++-- crates/tag/google-tag-manager/src/lib.rs | 8 +++--- .../src/integrations/registry.rs | 6 ++-- crates/trusted-server-core/src/publisher.rs | 7 ++--- .../src/streaming_processor.rs | 8 +++--- 11 files changed, 50 insertions(+), 49 deletions(-) diff --git a/crates/ad-tag/google/src/lib.rs b/crates/ad-tag/google/src/lib.rs index 4e39ffae80..523eb2c55f 100644 --- a/crates/ad-tag/google/src/lib.rs +++ b/crates/ad-tag/google/src/lib.rs @@ -726,10 +726,10 @@ mod tests { ); } - // -- Attribute rewriter -- + // -- Element handler -- #[test] - fn attribute_rewriter_rewrites_gpt_urls() { + fn element_handler_rewrites_gpt_urls() { let result = GptIntegration::rewrite_script_address( "https://securepubads.g.doubleclick.net/tag/js/gpt.js", ); @@ -746,7 +746,7 @@ mod tests { } #[test] - fn attribute_rewriter_keeps_non_gpt_urls() { + fn element_handler_keeps_non_gpt_urls() { let result = GptIntegration::rewrite_script_address("https://cdn.example.com/analytics.js"); assert_eq!( @@ -757,7 +757,7 @@ mod tests { } #[test] - fn attribute_rewriter_noop_when_disabled() { + fn element_handler_noop_when_disabled() { let config = GptConfig { rewrite_script: false, ..test_config() @@ -1205,10 +1205,10 @@ mod tests { ); } - // -- Head injector -- + // -- Head markup -- #[test] - fn head_injector_emits_enable_flag() { + fn head_markup_emits_enable_flag() { let integration = GptIntegration::new(test_config()); let inserts = integration.head_markup(); diff --git a/crates/auction/prebid/src/lib.rs b/crates/auction/prebid/src/lib.rs index 5080d02a7d..a44b995f73 100644 --- a/crates/auction/prebid/src/lib.rs +++ b/crates/auction/prebid/src/lib.rs @@ -2164,7 +2164,7 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] } #[test] - fn attribute_rewriter_removes_prebid_scripts() { + fn element_handler_removes_prebid_scripts() { let integration = PrebidIntegration::new(base_config()); let rewritten = integration.rewrite_script_address("https://cdn.prebid.org/prebid.min.js"); assert!(matches!(rewritten, AttributeRewriteAction::RemoveElement)); @@ -2174,7 +2174,7 @@ excluded_gam_ad_unit_path_suffixes = ["{suffix}"] } #[test] - fn attribute_rewriter_handles_query_strings_and_links() { + fn element_handler_handles_query_strings_and_links() { let integration = PrebidIntegration::new(base_config()); let rewritten = integration.rewrite_script_address("https://cdn.prebid.org/prebid.js?v=1.2.3"); @@ -2975,7 +2975,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_emits_config_script() { + fn head_markup_emits_config_script() { let integration = PrebidIntegration::new(base_config()); let inserts = integration.head_markup(); assert_eq!(inserts.len(), 2, "should produce config and bundle inserts"); @@ -3062,7 +3062,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_includes_managed_user_ids() { + fn head_markup_includes_managed_user_ids() { let mut config = base_config(); config.managed_user_ids = vec![PrebidManagedUserIdConfig { name: "exampleId".to_string(), @@ -3090,7 +3090,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_omits_optional_managed_user_id_fields_when_unset() { + fn head_markup_omits_optional_managed_user_id_fields_when_unset() { let mut config = base_config(); config.managed_user_ids = vec![PrebidManagedUserIdConfig { name: "exampleId".to_string(), @@ -3108,7 +3108,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_omits_managed_user_ids_when_none_configured() { + fn head_markup_omits_managed_user_ids_when_none_configured() { let integration = PrebidIntegration::new(base_config()); let inserts = integration.head_markup(); let script = &inserts[0]; @@ -3185,7 +3185,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_escapes_script_breakout_in_managed_user_ids() { + fn head_markup_escapes_script_breakout_in_managed_user_ids() { let mut config = base_config(); config.managed_user_ids = vec![PrebidManagedUserIdConfig { params: serde_json::Map::from_iter([( @@ -3218,7 +3218,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_includes_excluded_gam_ad_unit_path_suffixes() { + fn head_markup_includes_excluded_gam_ad_unit_path_suffixes() { let mut config = base_config(); config.excluded_gam_ad_unit_path_suffixes = vec!["/trackingonly".to_string(), "/measurement-only".to_string()]; @@ -3235,7 +3235,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_handles_missing_account_id() { + fn head_markup_handles_missing_account_id() { let mut config = base_config(); config.account_id = None; let integration = PrebidIntegration::new(config); @@ -3249,7 +3249,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_emits_external_bundle_script_with_hash_and_integrity() { + fn head_markup_emits_external_bundle_script_with_hash_and_integrity() { let sha256 = "a".repeat(64); let mut config = base_config(); config.external_bundle_url = @@ -3278,7 +3278,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_emits_external_bundle_script_without_hash_query_when_unhashed() { + fn head_markup_emits_external_bundle_script_without_hash_query_when_unhashed() { let mut config = base_config(); config.external_bundle_url = Some("https://assets.example/prebid/trusted-prebid.js".to_string()); @@ -3299,7 +3299,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_escapes_less_than_signs_in_values() { + fn head_markup_escapes_less_than_signs_in_values() { let mut config = base_config(); config.account_id = Some("</script><script>alert(1)</script>".to_string()); let integration = PrebidIntegration::new(config); @@ -3313,7 +3313,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_omits_client_side_bidders_when_empty() { + fn head_markup_omits_client_side_bidders_when_empty() { let integration = PrebidIntegration::new(base_config()); let inserts = integration.head_markup(); let script = &inserts[0]; @@ -3325,7 +3325,7 @@ external_bundle_sri = "sha384-AAAA" } #[test] - fn head_injector_includes_client_side_bidders_when_configured() { + fn head_markup_includes_client_side_bidders_when_configured() { let mut config = base_config(); config.client_side_bidders = vec!["rubicon".to_string(), "magnite".to_string()]; let integration = PrebidIntegration::new(config); diff --git a/crates/audience/permutive/src/lib.rs b/crates/audience/permutive/src/lib.rs index 933548d91d..ba7719cc3a 100644 --- a/crates/audience/permutive/src/lib.rs +++ b/crates/audience/permutive/src/lib.rs @@ -583,7 +583,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_rewrites_sdk_urls() { + fn test_element_handler_rewrites_sdk_urls() { let config = PermutiveConfig { organization_id: "myorg".to_string(), workspace_id: "workspace-123".to_string(), @@ -605,7 +605,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_noop_when_disabled() { + fn test_element_handler_noop_when_disabled() { let config = PermutiveConfig { organization_id: "myorg".to_string(), workspace_id: "workspace-123".to_string(), diff --git a/crates/bot-protection/datadome/src/lib.rs b/crates/bot-protection/datadome/src/lib.rs index 2726edad6f..67469cd44f 100644 --- a/crates/bot-protection/datadome/src/lib.rs +++ b/crates/bot-protection/datadome/src/lib.rs @@ -1553,7 +1553,7 @@ mod tests { } #[test] - fn head_injector_escapes_client_side_tag_url_attribute() { + fn head_markup_escapes_client_side_tag_url_attribute() { let mut config = test_config(); config.client_side_key = "test-client-key".to_string(); config.client_side_tag_url = "/integrations/datadome/tags.js?one=1&two=2".to_string(); @@ -1570,7 +1570,7 @@ mod tests { } #[test] - fn head_injector_emits_client_side_tag_when_key_configured() { + fn head_markup_emits_client_side_tag_when_key_configured() { let mut config = test_config(); config.client_side_key = "test-client-key".to_string(); config.client_side_configuration = serde_json::json!({ "ajaxListenerPath": true }); @@ -1594,7 +1594,7 @@ mod tests { } #[test] - fn head_injector_omits_client_side_tag_when_disabled_or_blank() { + fn head_markup_omits_client_side_tag_when_disabled_or_blank() { let mut suppressed = test_config(); suppressed.client_side_key = "test-client-key".to_string(); let suppressed_integration = DataDomeIntegration::new(suppressed); @@ -1697,7 +1697,7 @@ mod tests { } #[test] - fn attribute_rewriter_matches_datadome() { + fn element_handler_matches_datadome() { // Should judge both src and href attributes, and no other let document_state = IntegrationDocumentState::default(); let context = trusted_server_core::middleware::test_support::context( @@ -1736,7 +1736,7 @@ mod tests { } #[test] - fn attribute_rewriter_preserves_path() { + fn element_handler_preserves_path() { // Should preserve /js/... paths for signal collection API let action = DataDomeIntegration::rewrite_sdk_address("https://js.datadome.co/js/check"); match action { diff --git a/crates/cmp/didomi/src/lib.rs b/crates/cmp/didomi/src/lib.rs index d0d4204251..907b67ed6d 100644 --- a/crates/cmp/didomi/src/lib.rs +++ b/crates/cmp/didomi/src/lib.rs @@ -1373,7 +1373,7 @@ mod tests { } #[test] - fn head_injector_emits_proxy_path() { + fn head_markup_emits_proxy_path() { let custom_config = DidomiIntegrationConfig { geo_query_parameters: false, proxy_path: Some("my-consent".to_string()), @@ -1462,7 +1462,7 @@ mod tests { } #[test] - fn head_injector_default_path() { + fn head_markup_default_path() { let integration = DidomiIntegration::new(Arc::new(config())); assert_eq!( diff --git a/crates/cmp/sourcepoint/src/lib.rs b/crates/cmp/sourcepoint/src/lib.rs index 64f1223f8d..b13e9dc96d 100644 --- a/crates/cmp/sourcepoint/src/lib.rs +++ b/crates/cmp/sourcepoint/src/lib.rs @@ -1460,7 +1460,7 @@ mod tests { } #[test] - fn attribute_rewriter_skips_when_rewrite_disabled() { + fn element_handler_skips_when_rewrite_disabled() { let mut cfg = config(); cfg.rewrite_sdk = false; let integration = SourcepointIntegration::new(Arc::new(cfg)); @@ -1506,7 +1506,7 @@ mod tests { } #[test] - fn head_injector_emits_config_script_plus_trap_when_enabled() { + fn head_markup_emits_config_script_plus_trap_when_enabled() { let integration = SourcepointIntegration::new(Arc::new(config())); let inserts = integration.head_markup(); @@ -1560,7 +1560,7 @@ mod tests { } #[test] - fn head_injector_returns_config_when_rewrite_disabled() { + fn head_markup_returns_config_when_rewrite_disabled() { let mut cfg = config(); cfg.rewrite_sdk = false; let integration = SourcepointIntegration::new(Arc::new(cfg)); diff --git a/crates/identity/lockr/src/lib.rs b/crates/identity/lockr/src/lib.rs index 11577890b2..abff2f96ec 100644 --- a/crates/identity/lockr/src/lib.rs +++ b/crates/identity/lockr/src/lib.rs @@ -551,7 +551,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_rewrites_sdk_urls() { + fn test_element_handler_rewrites_sdk_urls() { let integration = LockrIntegration::new(test_config()); let result = integration.rewrite_sdk_address("https://aim.loc.kr/identity-lockr-v1.0.js"); @@ -564,7 +564,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_keeps_non_lockr_urls() { + fn test_element_handler_keeps_non_lockr_urls() { let integration = LockrIntegration::new(test_config()); let result = integration.rewrite_sdk_address("https://example.com/other.js"); @@ -577,7 +577,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_noop_when_disabled() { + fn test_element_handler_noop_when_disabled() { let config = LockrConfig { rewrite_sdk: false, ..test_config() diff --git a/crates/tag/google-tag-manager/src/lib.rs b/crates/tag/google-tag-manager/src/lib.rs index 55bbabc485..67da461a2a 100644 --- a/crates/tag/google-tag-manager/src/lib.rs +++ b/crates/tag/google-tag-manager/src/lib.rs @@ -2386,7 +2386,7 @@ mod tests { } #[test] - fn test_attribute_rewriter() { + fn test_element_handler() { // Case 1: Standard HTTPS URL let action = GoogleTagManagerIntegration::rewrite_address( "https://www.googletagmanager.com/gtm.js?id=GTM-TEST1234", @@ -2464,7 +2464,7 @@ mod tests { } #[test] - fn test_attribute_rewriter_rejects_false_positives() { + fn test_element_handler_rejects_false_positives() { // Test that URLs with GTM domains in query parameters or paths are NOT rewritten // This verifies the fix for P2: proper URL parsing instead of substring matching // Case 1: GTM domain in query parameter - should NOT be rewritten @@ -2514,7 +2514,7 @@ mod tests { } #[test] - fn test_script_rewriter() { + fn test_text_handler() { let mut script = InlineSnippet::default(); // Case 1: Inline GTM snippet @@ -3945,7 +3945,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= // English or minified tokens and must NOT engage GTM accumulation. // Previously these returned true because any non-empty prefix of a // marker was accepted, which let GTM claim and clobber fragments - // from overlapping script rewriters (see PR #618 P1). + // from overlapping text handlers (see PR #618 P1). for text in [ "x", // "g"-less "img", // ends in 'g' diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index fc8284fb8f..86f10cc8e2 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -80,9 +80,11 @@ impl ScriptRewriteAction { type IntegrationDocumentStateMap = BTreeMap<(&'static str, TypeId), Arc<dyn Any + Send + Sync>>; -/// Per-document state shared between HTML/script rewriters and post-processors. +/// State shared by the middleware working on one document in one phase. /// -/// This exists to support multi-phase HTML processing without requiring a second HTML parse. +/// What one handler leaves here a later handler or a stream processor reads, +/// so the document is parsed once. In the serve phase it starts with what +/// the request's hooks left, see [`IntegrationRequestState`]. #[derive(Clone, Default)] pub struct IntegrationDocumentState { inner: Arc<Mutex<IntegrationDocumentStateMap>>, diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index bae0f460d9..6a9a68c233 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -23594,12 +23594,11 @@ mod tests { ); } - /// Document-state survives from the parser pass into the stream processor. - /// A script rewriter writes into `IntegrationDocumentState` during - /// parsing, and the request-local stream processor reads it and substitutes. + /// What a text handler captures during parsing reaches the stream + /// processor of the same document, which substitutes it. /// Regression test: placeholders must be inserted and removed from final output. #[test] - fn document_state_placeholders_substitute_through_streaming_path() { + fn a_handler_s_placeholders_are_substituted_through_the_streaming_path() { let mut settings = create_test_settings(); crate::integrations::registry_test_support::payload_fixture::select_and_place( &mut settings, diff --git a/crates/trusted-server-core/src/streaming_processor.rs b/crates/trusted-server-core/src/streaming_processor.rs index d95e6ebf85..e3d9272e6b 100644 --- a/crates/trusted-server-core/src/streaming_processor.rs +++ b/crates/trusted-server-core/src/streaming_processor.rs @@ -310,8 +310,8 @@ impl lol_html::OutputSink for RcVecSink { /// Adapter to use `lol_html` [`HtmlRewriter`](lol_html::HtmlRewriter) as a [`StreamProcessor`]. /// /// Output is emitted incrementally on every [`process_chunk`](StreamProcessor::process_chunk) -/// call. Script rewriters that receive text from `lol_html` must be fragment-safe — -/// they accumulate text fragments internally until `is_last_in_text_node` is true. +/// call. A text handler that receives text from `lol_html` must be fragment-safe, +/// so it accumulates text fragments until the last chunk of the text node arrives. /// /// The adapter is single-use: one adapter per request. Calling [`StreamProcessor::reset`] /// is a no-op because the rewriter consumes its settings on construction. @@ -1731,8 +1731,8 @@ mod tests { } /// Verify that `lol_html` fragments text nodes when input chunks split - /// mid-text-node. Script rewriters must be fragment-safe — they accumulate - /// text fragments internally until `is_last_in_text_node` is true. + /// mid-text-node. A text handler must be fragment-safe, so it accumulates + /// text fragments until the last chunk of the text node arrives. #[test] fn lol_html_fragments_text_across_chunk_boundaries() { use std::cell::RefCell; From 5ec3fb4f5e8613cb348aba6dc746fd94fca5279d Mon Sep 17 00:00:00 2001 From: James Rogers <JamesR@51degrees.com> Date: Mon, 12 Oct 2026 15:17:43 +0100 Subject: [PATCH 31/36] Next.js handles text directly and messages say module Next.js implements TextHandler on a per-document struct for each of its two script rewriters, and the private ScriptRewriter trait is gone. The registry's middleware messages say module. The fetch and serve rule is stated once, on the MiddlewarePhase variants, and the remaining comments, the changelog and the Next.js guide drop hook and rewriter wording. --- CHANGELOG.md | 4 +- crates/framework/nextjs/src/lib.rs | 66 +++++-------------- .../framework/nextjs/src/rsc_placeholders.rs | 20 +++++- .../framework/nextjs/src/script_rewriter.rs | 20 +++++- crates/framework/nextjs/src/shared.rs | 42 +++++++++--- .../trusted-server-core/src/html_processor.rs | 14 +--- .../src/integrations/mod.rs | 7 +- .../src/integrations/registry.rs | 17 ++--- crates/trusted-server-core/src/middleware.rs | 28 +++----- crates/trusted-server-core/src/publisher.rs | 14 ++-- docs/guide/integrations/nextjs.md | 2 +- 11 files changed, 111 insertions(+), 123 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0277463cd4..0954f1076e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware, and no longer on every page of a deployment that selects the module. A deployment adds the entries when it upgrades. Until it does, a module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. To keep a page as it was, name the middleware of the modules a deployment selects in one `[[fetch]]` entry and one `[[serve]]` entry for `text/html`, in the order `trusted-server.example.toml` shows them, which is the order the modules' hooks ran in. -- **Breaking:** The four page hook traits are removed, being `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector`, with their context types, `ScriptTextAccumulator`, `AttributeRewriteOutcome` and the registration methods that took them. A module that implemented one supplies a `Middleware` instead, whose action carries the same decisions. A browser module's attribute on the script bundle's tag is stated with `with_bundle_tag_attribute`. +- **Breaking:** A module's change to a page runs only on the pages where a `[[fetch]]` or `[[serve]]` entry names its middleware. A deployment adds the entries when it upgrades. Until it does, a module changes no page, and startup logs a warning naming each middleware that no entry names. "Placing page changes" in the configuration guide lists the middleware each module supplies and the phase each runs in. To keep a page as it was, name the middleware of the modules a deployment selects in one `[[fetch]]` entry and one `[[serve]]` entry for `text/html`, in the order `trusted-server.example.toml` shows them, which is the order the modules' hooks ran in. +- **Breaking:** The four page traits are removed, being `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector`, with their context types, `ScriptTextAccumulator`, `AttributeRewriteOutcome` and the registration methods that took them. A module that implemented one supplies a `Middleware` instead, whose action carries the same decisions. A browser module's attribute on the script bundle's tag is stated with `with_bundle_tag_attribute`. - A pushed configuration keeps DataDome's two deprecated secret store selectors, `server_side_key_secret_store` and `protection_test_bypass.credential_secret_store`, where they were removed as the settings were read. DataDome still accepts and discards both, so a configuration that carries one loads as before. - Integration modules have moved out of `trusted-server-core` into crates of their own under `crates/<type>/<vendor>/`, each named by that folder, so `crates/cmp/osano` is the module `cmp.osano`. The settings that select and configure a module do not change. `crates/trusted-server-modules` lists the modules a stock build ships from crates of their own, in the order their hooks run. Every adapter builds its state and loads its settings with that list, and `ts config validate` and `ts config push` run each stock module's own rules. Code that named a moved module under `trusted_server_core::integrations` names its crate instead, such as `trusted_server_cmp_osano`. Moved: Osano, Lockr, Permutive, Sourcepoint, Didomi, Google Tag Manager, Testlight, Next.js, Google Publisher Tags and its diagnostics, DataDome, the mock ad server, APS, Prebid, the Prebid Server demand and the plain OpenRTB demand. Core keeps the JavaScript asset proxy, which is its own. - The Fastly adapter is a library with a thin binary over it. A deployment that ships a vendor crate writes a binary of its own that calls `trusted_server_adapter_fastly::run_with(vec![...])` with that crate's integration builder, in place of editing the adapter. The builders are composed with the built-in ones, and a builder runs only when the settings select its module. The Wasm artifact keeps its name, `trusted-server-adapter-fastly.wasm`. diff --git a/crates/framework/nextjs/src/lib.rs b/crates/framework/nextjs/src/lib.rs index f0ed442555..4d31ebbd5d 100644 --- a/crates/framework/nextjs/src/lib.rs +++ b/crates/framework/nextjs/src/lib.rs @@ -16,11 +16,9 @@ use serde::{Deserialize, Serialize}; use validator::Validate; use trusted_server_core::error::TrustedServerError; -use trusted_server_core::integrations::{ - IntegrationDocumentState, IntegrationRegistration, ScriptRewriteAction, -}; +use trusted_server_core::integrations::IntegrationRegistration; use trusted_server_core::middleware::{ - Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, TextHandler, + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, }; use trusted_server_core::settings::{IntegrationConfig, Settings}; @@ -50,10 +48,10 @@ mod shared; pub use rsc::rewrite_rsc_scripts_combined; -use rsc_placeholders::NextJsRscPlaceholderRewriter; +use rsc_placeholders::{NextJsRscPlaceholderRewriter, RscPlaceholderText}; use rsc_stream::NextJsRscStreamProcessorFactory; -use script_rewriter::NextJsNextDataRewriter; -use shared::{ScriptContext, ScriptRewriter}; +use script_rewriter::{NextDataText, NextJsNextDataRewriter}; +use shared::ScriptDocument; #[derive(Debug, Clone, Deserialize, Serialize, Validate)] #[serde(deny_unknown_fields)] @@ -140,8 +138,8 @@ pub fn register( /// Moves the origin's address to the publisher's in the data Next.js writes /// into a page, on the pages a `[[fetch]]` entry names [`MODULE`] for. struct PageChange { - structured: Arc<dyn ScriptRewriter>, - placeholders: Arc<dyn ScriptRewriter>, + structured: Arc<NextJsNextDataRewriter>, + placeholders: Arc<NextJsRscPlaceholderRewriter>, stream: NextJsRscStreamProcessorFactory, } @@ -158,20 +156,16 @@ impl Middleware for PageChange { /// placeholder rewriter, which is asked about every script. The stream /// processor then works on the document the two left. fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { - let script_text = |rewriter: &Arc<dyn ScriptRewriter>| -> Box<dyn TextHandler> { - Box::new(ScriptText { - rewriter: Arc::clone(rewriter), - request_host: context.request_host.to_owned(), - request_scheme: context.request_scheme.to_owned(), - origin_host: context.origin_host.to_owned(), - max_buffered_script_bytes: context.max_buffered_script_bytes, - document_state: context.document_state.clone(), - }) - }; MiddlewareAction { text_handlers: vec![ - script_text(&self.structured), - script_text(&self.placeholders), + Box::new(NextDataText { + rewriter: Arc::clone(&self.structured), + document: ScriptDocument::new(context), + }), + Box::new(RscPlaceholderText { + rewriter: Arc::clone(&self.placeholders), + document: ScriptDocument::new(context), + }), ], stream: Some(self.stream.create(context)), ..MiddlewareAction::pass() @@ -179,36 +173,6 @@ impl Middleware for PageChange { } } -/// One of the module's script rewriters at work on one document. -struct ScriptText { - rewriter: Arc<dyn ScriptRewriter>, - request_host: String, - request_scheme: String, - origin_host: String, - max_buffered_script_bytes: usize, - document_state: IntegrationDocumentState, -} - -impl TextHandler for ScriptText { - fn selector(&self) -> &str { - self.rewriter.selector() - } - - fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { - self.rewriter.rewrite( - text, - &ScriptContext { - request_host: &self.request_host, - request_scheme: &self.request_scheme, - origin_host: &self.origin_host, - is_last_in_text_node: is_last, - max_buffered_script_bytes: self.max_buffered_script_bytes, - document_state: &self.document_state, - }, - ) - } -} - fn build( settings: &Settings, ) -> Result<Option<Arc<NextJsIntegrationConfig>>, Report<TrustedServerError>> { diff --git a/crates/framework/nextjs/src/rsc_placeholders.rs b/crates/framework/nextjs/src/rsc_placeholders.rs index a79a1474ab..6a542270ea 100644 --- a/crates/framework/nextjs/src/rsc_placeholders.rs +++ b/crates/framework/nextjs/src/rsc_placeholders.rs @@ -1,6 +1,7 @@ use std::sync::Arc; use trusted_server_core::integrations::ScriptRewriteAction; +use trusted_server_core::middleware::TextHandler; use super::NextJsIntegrationConfig; use super::rsc::DEFAULT_MAX_COMBINED_PAYLOAD_BYTES; @@ -11,7 +12,7 @@ use super::rsc_stream::{ capture_fragment, classify_rsc_group, document_state, rsc_payload_placeholder, }; use super::shared::{ - RSC_RECEIVER_CONTEXT_BYTES, ScriptContext, ScriptRewriter, find_rsc_push_payload_range, + RSC_RECEIVER_CONTEXT_BYTES, ScriptContext, ScriptDocument, find_rsc_push_payload_range, find_trimmed_rsc_push_payload_range, receiver_context_is_flight_push, }; @@ -167,11 +168,24 @@ impl NextJsRscPlaceholderRewriter { } } -impl ScriptRewriter for NextJsRscPlaceholderRewriter { - fn selector(&self) -> &'static str { +/// The RSC placeholder rewriter at work on one document. +pub(super) struct RscPlaceholderText { + pub(super) rewriter: Arc<NextJsRscPlaceholderRewriter>, + pub(super) document: ScriptDocument, +} + +impl TextHandler for RscPlaceholderText { + fn selector(&self) -> &str { "script" } + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + self.rewriter.rewrite(text, &self.document.chunk(is_last)) + } +} + +impl NextJsRscPlaceholderRewriter { + /// Decides what happens to one chunk of a script's text. fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if self.config.rewrite_attributes.is_empty() { return ScriptRewriteAction::keep(); diff --git a/crates/framework/nextjs/src/script_rewriter.rs b/crates/framework/nextjs/src/script_rewriter.rs index b1d50de3fe..9243cb3cc4 100644 --- a/crates/framework/nextjs/src/script_rewriter.rs +++ b/crates/framework/nextjs/src/script_rewriter.rs @@ -5,10 +5,11 @@ use regex::{Regex, escape}; use trusted_server_core::error::TrustedServerError; use trusted_server_core::integrations::ScriptRewriteAction; +use trusted_server_core::middleware::TextHandler; use super::NextJsIntegrationConfig; use super::rsc_stream::{FragmentCapture, capture_fragment, document_state}; -use super::shared::{ScriptContext, ScriptRewriter, strip_origin_host_with_optional_port}; +use super::shared::{ScriptContext, ScriptDocument, strip_origin_host_with_optional_port}; pub(super) struct NextJsNextDataRewriter { config: Arc<NextJsIntegrationConfig>, @@ -46,11 +47,24 @@ impl NextJsNextDataRewriter { } } -impl ScriptRewriter for NextJsNextDataRewriter { - fn selector(&self) -> &'static str { +/// The `__NEXT_DATA__` rewriter at work on one document. +pub(super) struct NextDataText { + pub(super) rewriter: Arc<NextJsNextDataRewriter>, + pub(super) document: ScriptDocument, +} + +impl TextHandler for NextDataText { + fn selector(&self) -> &str { "script#__NEXT_DATA__" } + fn decide(&mut self, text: &str, is_last: bool) -> ScriptRewriteAction { + self.rewriter.rewrite(text, &self.document.chunk(is_last)) + } +} + +impl NextJsNextDataRewriter { + /// Decides what happens to one chunk of the `__NEXT_DATA__` script. fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction { if self.config.rewrite_attributes.is_empty() { return ScriptRewriteAction::keep(); diff --git a/crates/framework/nextjs/src/shared.rs b/crates/framework/nextjs/src/shared.rs index 70d7606eed..a9483099d0 100644 --- a/crates/framework/nextjs/src/shared.rs +++ b/crates/framework/nextjs/src/shared.rs @@ -8,7 +8,8 @@ use std::sync::LazyLock; use regex::Regex; use trusted_server_core::host_rewrite::rewrite_bare_host_at_boundaries; -use trusted_server_core::integrations::{IntegrationDocumentState, ScriptRewriteAction}; +use trusted_server_core::integrations::IntegrationDocumentState; +use trusted_server_core::middleware::MiddlewareContext; /// What a script rewriter is told about the chunk of text in front of it. #[derive(Debug)] @@ -27,14 +28,39 @@ pub(crate) struct ScriptContext<'a> { pub(crate) document_state: &'a IntegrationDocumentState, } -/// A decision about the text of the scripts a selector matches, made a chunk -/// at a time. -pub(crate) trait ScriptRewriter: Send + Sync { - /// The CSS selector of the scripts this rewriter is asked about. - fn selector(&self) -> &'static str; +/// What a script rewriter keeps of the document it works on, to tell it about +/// each chunk. +pub(crate) struct ScriptDocument { + request_host: String, + request_scheme: String, + origin_host: String, + max_buffered_script_bytes: usize, + document_state: IntegrationDocumentState, +} + +impl ScriptDocument { + /// What `context` says of the document. + pub(crate) fn new(context: &MiddlewareContext<'_>) -> Self { + Self { + request_host: context.request_host.to_owned(), + request_scheme: context.request_scheme.to_owned(), + origin_host: context.origin_host.to_owned(), + max_buffered_script_bytes: context.max_buffered_script_bytes, + document_state: context.document_state.clone(), + } + } - /// Decides what happens to one chunk of a script's text. - fn rewrite(&self, content: &str, ctx: &ScriptContext<'_>) -> ScriptRewriteAction; + /// The context of one chunk of a script's text. + pub(crate) fn chunk(&self, is_last_in_text_node: bool) -> ScriptContext<'_> { + ScriptContext { + request_host: &self.request_host, + request_scheme: &self.request_scheme, + origin_host: &self.origin_host, + is_last_in_text_node, + max_buffered_script_bytes: self.max_buffered_script_bytes, + document_state: &self.document_state, + } + } } // These are static code-defined literals, not config-derived patterns, so they diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index b7e5d6ff6d..d3961d09e5 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -252,16 +252,6 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso /// `serve` is `None` for a document that may be stored, whose serve chain /// runs on each reader's copy instead, through [`create_serve_processor`]. /// -/// Each middleware is asked once, here, what it will do with this document. -/// The state the fetch chain shares is the document's own and holds nothing a -/// request left, because what that chain writes may be stored and served to -/// other readers. -/// -/// A fetch middleware's element handlers are asked after core has moved -/// the origin's address in an attribute. Its text handlers are handed a chunk -/// as the handlers before them left it, and its stream processor is handed -/// what the one before it produced. -/// /// # Errors /// /// When a handler's selector does not parse, naming the selector. @@ -355,13 +345,13 @@ pub fn create_serve_processor( /// `<head>`. const BUNDLE_SELECTOR: &str = "script#trustedserver-js"; -/// One selector and what a rewriter does with what it matches. +/// One selector and what a handler does with what it matches. type ContentHandler = ( std::borrow::Cow<'static, lol_html::Selector>, lol_html::ElementContentHandlers<'static>, ); -/// The rewriter handlers that carry out a plan's element and text handlers. +/// The handlers that carry out a plan's element and text handlers. /// /// The text handlers share `edits` with every text handler registered before /// them, so each is handed a chunk as the one before it left it. diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index 6e342339e3..c100a74336 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -694,7 +694,7 @@ impl IntegrationBuilder { } } -/// The built-in integrations, in hook order. +/// The built-in integrations, in the order `trusted-server-modules` lists them. const BUILT_IN_BUILDERS: &[IntegrationBuilder] = &[ // This stays the first module a section selects, so its middleware is // the first of theirs in the order the registry lists them. @@ -744,13 +744,14 @@ const BUILT_IN_BUILDERS: &[IntegrationBuilder] = &[ .with_adserver(&crate::auction::test_support::adserver_fixture::ADSERVER), ]; -/// The built-in integration builders, in hook order. +/// The built-in integration builders, in the order `trusted-server-modules` +/// lists them. pub(crate) fn builders() -> &'static [IntegrationBuilder] { BUILT_IN_BUILDERS } /// Every builder the registry will consider: the built-in set followed by -/// `extra`, in that order, so hook order for the built-ins never changes. +/// `extra`, in that order, so the built-ins keep their order. pub(crate) fn all_builders( extra: &[IntegrationBuilder], ) -> impl Iterator<Item = IntegrationBuilder> + '_ { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index d214474fad..f904b9fc35 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -1260,16 +1260,11 @@ pub struct ProxyDispatchInput<'a> { /// Refuses a registered middleware no entry could name, and an entry this /// deployment cannot run. /// -/// Only knowable here, where every running module's middleware have been -/// handed over. A name no module supplies, or one named in a phase it does not -/// run in, would otherwise do nothing on every request and say so nowhere. -/// -/// A middleware no entry names is left alone and logged, because a module -/// may be selected for what else it does. +/// A middleware no entry names is left alone and logged. /// /// # Errors /// -/// Naming the integration or the entry at fault. +/// Naming the module or the entry at fault. fn check_middleware( settings: &Settings, inner: &IntegrationRegistryInner, @@ -1286,14 +1281,14 @@ fn check_middleware( let id = middleware.middleware_id(); if !crate::module_name::is_valid(id) { return Err(refuse(format!( - "integration `{integration}` supplies a middleware named `{id}`, which is not a \ + "module `{integration}` supplies a middleware named `{id}`, which is not a \ name an entry can write. A name is parts joined by `.`, each in lower case \ letters, digits, `_` or `-`" ))); } if middleware.phases().is_empty() { return Err(refuse(format!( - "integration `{integration}` supplies the middleware `{id}`, which says it runs \ + "module `{integration}` supplies the middleware `{id}`, which says it runs \ in no phase, so no entry could name it" ))); } @@ -1302,7 +1297,7 @@ fn check_middleware( .find(|(_, earlier)| earlier.middleware_id() == id) { return Err(refuse(format!( - "integrations `{earlier}` and `{integration}` both supply a middleware named \ + "modules `{earlier}` and `{integration}` both supply a middleware named \ `{id}`, so an entry could not say which one it means" ))); } @@ -1335,7 +1330,7 @@ fn check_middleware( } for (integration, middleware) in unnamed_middleware(settings, inner) { log::warn!( - "Integration `{integration}` supplies the middleware `{}` and no {} entry names \ + "Module `{integration}` supplies the middleware `{}` and no {} entry names \ it, so it changes no page", middleware.middleware_id(), phases_of(middleware).join(" or ") diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index bd9c07867e..e032548913 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -28,12 +28,7 @@ //! orders the page changes of the modules that run. A middleware reads its //! settings from its module's own table. //! -//! Fetch is obtaining the document from the origin. A fetch middleware is -//! handed nothing about the reader, so what it leaves is what a shared -//! template stores for every reader. Serve is preparing the document for one -//! response. A serve middleware runs for each reader, on that reader's copy, -//! whether the page came from the store or from the origin, and nothing it -//! writes is stored. +//! [`MiddlewarePhase`] says when each phase runs. use std::fmt; use std::rc::Rc; @@ -52,10 +47,13 @@ pub const HTML_MEDIA_TYPE: &str = "text/html"; #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum MiddlewarePhase { /// On the document as the origin sent it. What is left is what a shared - /// template stores, so nothing here may depend on the reader. + /// template stores, so a fetch middleware is handed nothing a request + /// left. Fetch, /// On one reader's copy, on its way out, whether it came from the store - /// or from the origin. Nothing written here is stored. + /// or from the origin. A serve middleware is handed what the modules' + /// request preparers left for this reader's request, and nothing written + /// here is stored. Serve, } @@ -80,11 +78,6 @@ impl fmt::Display for MiddlewarePhase { } /// What a middleware is told about the document it is asked to change. -/// -/// In [`MiddlewarePhase::Fetch`] the document's state holds nothing a -/// request left, because the document may be stored and served to other -/// readers. In [`MiddlewarePhase::Serve`] it holds what the modules' request -/// hooks left for this reader's request. #[derive(Debug, Clone, Copy)] pub struct MiddlewareContext<'a> { /// Which phase is running. @@ -248,7 +241,7 @@ impl MiddlewareAction { && self.stream.is_none() } - /// Checks every handler's selector parses, before a rewriter is built + /// Checks every handler's selector parses, before a handler is built /// from it. fn validate_selectors(&self) -> Result<(), String> { let check = |selector: &str| { @@ -545,11 +538,8 @@ impl MiddlewareChain { /// Asks every middleware in turn what it will do with one document. /// - /// A selector cannot be checked when the settings load, because a - /// middleware chooses its handlers for each document, so each is checked - /// here. The caller refuses the response when one does not parse, because - /// a change that silently does not happen is worse than a response that - /// fails. + /// Checks each handler's selector, refusing the document when one does + /// not parse. /// /// # Errors /// diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 6a9a68c233..e0210da8da 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -1533,9 +1533,7 @@ fn create_html_stream_processor( .with_body_close(body_close) .with_csp_nonce_observer(csp_nonce_observed); - // The middleware of the first [[fetch]] entry covering this path. What - // they leave is what a shared template stores, and they are handed - // nothing the request left. + // The middleware of the first [[fetch]] entry covering this path. let fetch = params.integration_registry.middleware_chain( ¶ms.settings.fetch, MiddlewarePhase::Fetch, @@ -1544,9 +1542,7 @@ fn create_html_stream_processor( ); // A response that may be stored leaves this processor as the template // every reader shares, so its [[serve]] entry runs on each reader's copy - // once that copy is assembled. See `run_serve_pass`. What the request - // left reaches a document through that entry alone, so none of it reaches - // a template. + // once that copy is assembled. See `run_serve_pass`. let serve = (!params.shared_template_authorized).then(|| { params.integration_registry.middleware_chain( ¶ms.settings.serve, @@ -2043,10 +2039,8 @@ pub async fn buffer_publisher_response_async( set_assembly_response_state(&mut response, state); } let bytes = if was_authorized { - // The transform wrote a template every reader shares, so the - // [[serve]] entry runs here, on this reader's assembled copy. - // A page that was never authorized ran it inside the - // transform. + // The [[serve]] entry runs here on a shared template, and + // inside the transform on a page that was never authorized. let bytes = run_serve_pass(bytes, ¶ms, settings, integration_registry)?; encode_complete_body(bytes, response_compression(&response))? } else { diff --git a/docs/guide/integrations/nextjs.md b/docs/guide/integrations/nextjs.md index cc4b8d8dfe..2a89ef4d14 100644 --- a/docs/guide/integrations/nextjs.md +++ b/docs/guide/integrations/nextjs.md @@ -2,7 +2,7 @@ **Category**: Framework Support **Status**: Production -**Type**: HTML/Script Rewriter +**Type**: Middleware ## Overview From 68c67ed3b2733f91eb49654875c6955da27866af Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 16 Oct 2026 09:07:01 +0100 Subject: [PATCH 32/36] Refuse an entry that follows one for the path / The path of a request for a page begins with `/`, so an entry with `path = "/"` covers every page, as an entry with no path does. An entry with no path written after it was accepted and never reached. The check for an entry hidden behind an earlier one now treats `/` and no path as the same prefix, so either order is refused when the settings load and at deploy validation, with the message that the two entries cover the same pages. --- crates/trusted-server-core/src/middleware.rs | 15 +++--- .../src/middleware/tests.rs | 47 +++++++++++++++++++ 2 files changed, 56 insertions(+), 6 deletions(-) diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index e032548913..ed67c1a533 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -337,6 +337,13 @@ impl PhaseEntry { .as_deref() .is_none_or(|prefix| path.starts_with(prefix)) } + + /// The prefix the paths the entry covers begin with. The path of a + /// request for a page begins with `/`, so `/` covers the same pages as + /// no path at all. + fn prefix(&self) -> &str { + self.path.as_deref().unwrap_or("/") + } } /// One phase's entries, in the order written. @@ -440,13 +447,9 @@ impl PhaseEntries { } if let Some(earlier) = self.0[..index].iter().position(|earlier| { earlier.media_type == entry.media_type - && match (&earlier.path, &entry.path) { - (None, _) => true, - (Some(covering), Some(path)) => path.starts_with(covering.as_str()), - (Some(_), None) => false, - } + && entry.prefix().starts_with(earlier.prefix()) }) { - return Err(if self.0[earlier].path == entry.path { + return Err(if self.0[earlier].prefix() == entry.prefix() { format!( "{at} covers the same pages as [[{phase}]] entry {}, and a response \ takes the first entry that covers it, so this one is never reached. \ diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index d084dcca2e..efff381e22 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -417,6 +417,31 @@ fn an_entry_for_the_same_pages_as_an_earlier_one_is_told_to_share_its_list() { } } +#[test] +fn a_path_of_slash_covers_the_same_pages_as_no_path() { + // The path of a request for a page begins with `/`, so whichever of the + // two entries comes second is never reached. + let cases = [ + vec![ + entry(Some("/"), &["example.a"]), + entry(None, &["example.b"]), + ], + vec![ + entry(None, &["example.a"]), + entry(Some("/"), &["example.b"]), + ], + ]; + + for entries in cases { + let message = refusal(entries); + assert!( + message.contains("[[fetch]] entry 2 covers the same pages as [[fetch]] entry 1") + && message.contains("Name every middleware for those pages in one entry's list"), + "should refuse the entry that is never reached: {message}" + ); + } +} + #[test] fn entries_are_read_from_a_list_and_written_back_as_they_were() { let written = json!([ @@ -1394,6 +1419,28 @@ fn settings_with_a_misshapen_entry_are_refused() { } } +#[test] +fn settings_with_an_entry_after_one_for_the_path_slash_are_refused() { + let message = settings_with_document( + r#" +[[fetch]] +media_type = "text/html" +path = "/" +middleware = ["example.root"] + +[[fetch]] +media_type = "text/html" +middleware = ["example.all"] +"#, + ) + .expect_err("should refuse settings with an entry that is never reached"); + + assert!( + message.contains("[[fetch]] entry 2 covers the same pages as [[fetch]] entry 1"), + "should name the entry that is never reached: {message}" + ); +} + #[test] fn deploy_validation_refuses_a_misshapen_entry() { let mut settings = create_test_settings(); From 8340e693d7289f3c782fb53b481dac9406b1e2d2 Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 16 Oct 2026 09:08:18 +0100 Subject: [PATCH 33/36] Keep the script text a middleware writes inside its script A text handler's replacement is written as markup, because a browser decodes no entity inside a script. Text holding `</script` followed by whitespace, `/` or `>`, in any letter case, therefore ended the script element, and what followed it was read as markup. After `<!--`, a `<script` followed by one of those kept the element's own end tag from closing it, so the rest of the page was read as script. lol_html leaves this to the caller, whose text in a script must not look like a closing tag. A guard registered after every middleware text handler, in both phases, now follows the text written into each script the way a browser's tokenizer reads it. Where text a middleware changed would end the element or start `<script` inside `<!--`, the `<` that begins it is written as `\u003c`, which a JavaScript string, template literal or regular expression and a JSON string all read as `<`. A sequence split across chunks is held back until the next chunk shows how it ends. Text the origin sent is written as it was, and `<!--` alone is left as written, because the end tag still closes the element after it. --- .../trusted-server-core/src/html_processor.rs | 26 +- .../src/html_processor/script_text.rs | 357 ++++++++++++++++++ .../src/middleware/tests.rs | 171 +++++++++ 3 files changed, 552 insertions(+), 2 deletions(-) create mode 100644 crates/trusted-server-core/src/html_processor/script_text.rs diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index d3961d09e5..281be64345 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -9,7 +9,7 @@ use std::sync::atomic::{AtomicBool, Ordering}; use lol_html::{ EndTagHandler, Settings as RewriterSettings, element, end, - html_content::{ContentType, EndTag, TextChunk}, + html_content::{ContentType, EndTag, TextChunk, TextType}, text, }; @@ -26,6 +26,10 @@ use crate::settings::Settings; use crate::streaming_processor::{HtmlRewriterAdapter, StreamProcessor}; use crate::tsjs; +mod script_text; + +use script_text::ScriptTextGuard; + struct HtmlWithStreamingProcessors { inner: Box<dyn StreamProcessor>, processors: Vec<Box<dyn StreamProcessor>>, @@ -354,7 +358,10 @@ type ContentHandler = ( /// The handlers that carry out a plan's element and text handlers. /// /// The text handlers share `edits` with every text handler registered before -/// them, so each is handed a chunk as the one before it left it. +/// them, so each is handed a chunk as the one before it left it. A +/// [`ScriptTextGuard`] registered after them writes what they left in a +/// script, so that their text cannot end the script or keep its end tag from +/// closing it. /// /// A selector that does not parse panics here, which /// [`MiddlewareChain::plan`] rules out by refusing to plan one. @@ -391,6 +398,7 @@ fn middleware_content_handlers( Ok(()) })); } + let guarded = !text_handlers.is_empty(); for mut handler in text_handlers { let selector = handler.selector().to_owned(); let edits = Rc::clone(edits); @@ -404,6 +412,20 @@ fn middleware_content_handlers( Ok(()) })); } + if guarded { + let edits = Rc::clone(edits); + let mut guard = ScriptTextGuard::default(); + handlers.push(text!("script", move |text| { + if text.text_type() == TextType::ScriptData { + let written = edits.current(text); + if let Some(safe) = guard.write(&written, text.removed(), text.last_in_text_node()) + { + edits.replace(text, &safe); + } + } + Ok(()) + })); + } handlers } diff --git a/crates/trusted-server-core/src/html_processor/script_text.rs b/crates/trusted-server-core/src/html_processor/script_text.rs new file mode 100644 index 0000000000..507673ad5c --- /dev/null +++ b/crates/trusted-server-core/src/html_processor/script_text.rs @@ -0,0 +1,357 @@ +//! Script text a middleware writes, kept inside its `<script>` element. +//! +//! A browser decodes no entity inside a script, so text written there cannot +//! be escaped as markup. The element ends at the first `</script` followed by +//! whitespace, `/` or `>`, in any letter case, and after `<!--` a `<script` +//! followed by one of those stops the element's own end tag from closing it. +//! [`ScriptTextGuard`] reads the text written into a script as a browser's +//! tokenizer does, following the script data states of the HTML standard. +//! Where text a middleware changed would do either, the `<` that begins the +//! sequence is written as `\u003c`, which a JavaScript string, template +//! literal or regular expression and a JSON string all read as `<`. + +use std::mem; + +/// The tag name both sequences spell, matched in any letter case. +const NAME: &[u8] = b"script"; + +/// How many letters of [`NAME`] a whole name matches. +const WHOLE: u8 = 6; + +/// What `<` is written as where it would let the text out of its script. +const ESCAPED_LESS_THAN: &str = "\\u003c"; + +/// Where a browser's tokenizer is in the text of a script. A number counts +/// the letters of `script` matched so far. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +enum State { + #[default] + Data, + LessThan, + EscapeStart, + EscapeStartDash, + EndTag(u8), + /// After `<!--`, where `</script` still ends the element. + Escaped, + EscapedDash, + EscapedDashDash, + EscapedLessThan, + EscapedEndTag(u8), + DoubleEscapeStart(u8), + /// After `<!--` and `<script`, where `</script` no longer ends the + /// element. + DoubleEscaped, + DoubleEscapedDash, + DoubleEscapedDashDash, + DoubleEscapedLessThan, + DoubleEscapeEnd(u8), +} + +/// What one character of the text does. +enum Step { + /// Moves to a state, with the character read. + Next(State), + /// Moves to a state, which reads the same character again. + Again(State), + /// Ends a sequence that lets the text out of its script, unless its `<` + /// is escaped, leading to the first state when it is and the second + /// when it is not. + Escape(State, State), +} + +impl State { + /// Whether the state is inside a sequence whose `<` may yet have to be + /// escaped. + fn is_open(self) -> bool { + matches!( + self, + Self::LessThan + | Self::EndTag(_) + | Self::EscapedLessThan + | Self::EscapedEndTag(_) + | Self::DoubleEscapeStart(_) + ) + } + + fn step(self, byte: u8) -> Step { + match self { + Self::Data if byte == b'<' => Step::Next(Self::LessThan), + Self::Data => Step::Next(Self::Data), + Self::LessThan => match byte { + b'/' => Step::Next(Self::EndTag(0)), + b'!' => Step::Next(Self::EscapeStart), + _ => Step::Again(Self::Data), + }, + Self::EscapeStart if byte == b'-' => Step::Next(Self::EscapeStartDash), + Self::EscapeStartDash if byte == b'-' => Step::Next(Self::EscapedDashDash), + Self::EscapeStart | Self::EscapeStartDash => Step::Again(Self::Data), + // `</script` would end the element. + Self::EndTag(WHOLE) if ends_name(byte) => Step::Escape(Self::Data, Self::Data), + Self::EndTag(matched) => name(matched, byte, Self::EndTag, Self::Data), + Self::Escaped | Self::EscapedDash | Self::EscapedDashDash => { + Step::Next(self.escaped(byte)) + } + Self::EscapedLessThan => match byte { + b'/' => Step::Next(Self::EscapedEndTag(0)), + _ if byte.is_ascii_alphabetic() => Step::Again(Self::DoubleEscapeStart(0)), + _ => Step::Again(Self::Escaped), + }, + Self::EscapedEndTag(WHOLE) if ends_name(byte) => { + Step::Escape(Self::Escaped, Self::Escaped) + } + Self::EscapedEndTag(matched) => name(matched, byte, Self::EscapedEndTag, Self::Escaped), + // `<script` inside `<!--` would keep the element's own end tag + // from closing it. + Self::DoubleEscapeStart(WHOLE) if ends_name(byte) => { + Step::Escape(Self::Escaped, Self::DoubleEscaped) + } + Self::DoubleEscapeStart(matched) => { + name(matched, byte, Self::DoubleEscapeStart, Self::Escaped) + } + Self::DoubleEscaped | Self::DoubleEscapedDash | Self::DoubleEscapedDashDash => { + Step::Next(self.double_escaped(byte)) + } + Self::DoubleEscapedLessThan if byte == b'/' => Step::Next(Self::DoubleEscapeEnd(0)), + Self::DoubleEscapedLessThan => Step::Again(Self::DoubleEscaped), + Self::DoubleEscapeEnd(WHOLE) if ends_name(byte) => Step::Next(Self::Escaped), + Self::DoubleEscapeEnd(matched) => { + name(matched, byte, Self::DoubleEscapeEnd, Self::DoubleEscaped) + } + } + } + + /// The state after `byte` in one of the states after `<!--`. + fn escaped(self, byte: u8) -> Self { + match (self, byte) { + (_, b'<') => Self::EscapedLessThan, + (Self::Escaped, b'-') => Self::EscapedDash, + (Self::EscapedDash | Self::EscapedDashDash, b'-') => Self::EscapedDashDash, + (Self::EscapedDashDash, b'>') => Self::Data, + _ => Self::Escaped, + } + } + + /// The state after `byte` in one of the states after `<!--` and + /// `<script`. + fn double_escaped(self, byte: u8) -> Self { + match (self, byte) { + (_, b'<') => Self::DoubleEscapedLessThan, + (Self::DoubleEscaped, b'-') => Self::DoubleEscapedDash, + (Self::DoubleEscapedDash | Self::DoubleEscapedDashDash, b'-') => { + Self::DoubleEscapedDashDash + } + (Self::DoubleEscapedDashDash, b'>') => Self::Data, + _ => Self::DoubleEscaped, + } + } +} + +/// Whether `byte` ends a tag name, as whitespace, `/` or `>` do. +fn ends_name(byte: u8) -> bool { + matches!(byte, b'\t' | b'\n' | b'\x0C' | b'\r' | b' ' | b'/' | b'>') +} + +/// Reads `byte` in a state that has matched `matched` letters of `script`. +/// A letter that continues `script` moves to `reading`, and a name that can +/// no longer be `script` returns to `other`. +fn name(matched: u8, byte: u8, reading: fn(u8) -> State, other: State) -> Step { + if !byte.is_ascii_alphabetic() { + return Step::Again(other); + } + match NAME.get(usize::from(matched)) { + Some(&letter) if byte.to_ascii_lowercase() == letter => Step::Next(reading(matched + 1)), + _ => Step::Next(other), + } +} + +/// Follows the text written into one script after another, a chunk at a +/// time, and says what to write so that text a middleware changed stays +/// inside its script. +/// +/// Text the origin sent is written as it is, because the parser already +/// read all of it as the script's text. +#[derive(Debug, Default)] +pub(super) struct ScriptTextGuard { + /// Where the tokenizer is after the text written so far. + state: State, + /// The end of the last chunk, held back from it because it begins a + /// sequence that only the next chunk shows the end of. + held: String, + /// Whether a middleware changed a chunk of this script so far. + changed: bool, +} + +impl ScriptTextGuard { + /// What to write in place of one chunk of a script's text, being + /// `written`, which the handlers left there, after anything held back + /// from the chunk before. `changed` says whether a handler changed this + /// chunk, and `is_last` whether it is the last of the script. `None` + /// leaves the chunk as the handlers left it. + pub(super) fn write(&mut self, written: &str, changed: bool, is_last: bool) -> Option<String> { + self.changed |= changed; + if self.held.is_empty() && self.state == State::Data && !written.contains('<') { + self.finish(is_last); + return None; + } + + let text = format!("{}{written}", mem::take(&mut self.held)); + let bytes = text.as_bytes(); + let mut state = self.state; + // The `<` that began the sequence being read, and the state before it. + let mut open: Option<(usize, State)> = None; + let mut escapes: Vec<usize> = Vec::new(); + let mut index = 0; + while let Some(&byte) = bytes.get(index) { + let before = state; + let read = match state.step(byte) { + Step::Next(next) => { + state = next; + true + } + Step::Again(next) => { + state = next; + false + } + Step::Escape(escaped, as_written) => { + match open { + Some((at, _)) if self.changed => { + escapes.push(at); + state = escaped; + } + _ => state = as_written, + } + true + } + }; + if !state.is_open() { + open = None; + } else if !before.is_open() { + open = Some((index, before)); + } + if read { + index += 1; + } + } + + let end = match open { + Some((at, before)) if !is_last => { + self.state = before; + at + } + _ => { + self.state = state; + text.len() + } + }; + self.held = text[end..].to_owned(); + self.finish(is_last); + if escapes.is_empty() && text.len() == written.len() && end == text.len() { + return None; + } + + let mut safe = String::with_capacity(end + escapes.len() * ESCAPED_LESS_THAN.len()); + let mut from = 0; + for at in escapes { + safe.push_str(&text[from..at]); + safe.push_str(ESCAPED_LESS_THAN); + from = at + 1; + } + safe.push_str(&text[from..end]); + Some(safe) + } + + /// Starts afresh for the next script once the last chunk of this one is + /// written. + fn finish(&mut self, is_last: bool) { + if is_last { + *self = Self::default(); + } + } +} + +#[cfg(test)] +mod tests { + use super::ScriptTextGuard; + + /// What `guard` writes for a script whose chunks are `chunks`, each + /// changed by a middleware when `changed`. + fn written(chunks: &[&str], changed: bool) -> String { + let mut guard = ScriptTextGuard::default(); + let mut page = String::new(); + for (index, chunk) in chunks.iter().enumerate() { + let is_last = index + 1 == chunks.len(); + match guard.write(chunk, changed, is_last) { + Some(safe) => page.push_str(&safe), + None => page.push_str(chunk), + } + } + page + } + + #[test] + fn a_closing_tag_a_middleware_writes_is_escaped_in_any_letter_case() { + for (text, expected) in [ + ("a('</script>')", "a('\\u003c/script>')"), + ("a('</SCRIPT >')", "a('\\u003c/SCRIPT >')"), + ("a('</Script/')", "a('\\u003c/Script/')"), + ("a('</script\n')", "a('\\u003c/script\n')"), + ("a('<!--</script>')", "a('<!--\\u003c/script>')"), + ] { + assert_eq!(written(&[text], true), expected, "should escape `{text}`"); + } + } + + #[test] + fn what_cannot_end_the_script_is_written_as_it_is() { + for text in [ + "if (a < b && c > d) {}", + "a('</scripts>')", + "a('</scrip>')", + "a('<script>')", + "a('<!--')", + "<!-- legacy(); //-->", + "a('</script')", + ] { + assert_eq!( + written(&[text], true), + text, + "should leave `{text}` as it is" + ); + } + } + + #[test] + fn a_script_tag_inside_a_comment_opening_a_middleware_writes_is_escaped() { + assert_eq!( + written(&["a('<!--<script>')"], true), + "a('<!--\\u003cscript>')", + "should keep the script's own end tag closing it" + ); + assert_eq!( + written(&["a('<!--<script>'); b('</script>')"], true), + "a('<!--\\u003cscript>'); b('\\u003c/script>')", + "should escape the closing tag that the escaped tag leaves able to end the script" + ); + } + + #[test] + fn a_sequence_split_across_chunks_is_escaped_where_it_begins() { + let chunks = ["a('<", "/scr", "ipt", ">');"]; + assert_eq!(written(&chunks, true), "a('\\u003c/script>');"); + + let chunks = ["a('", "<", "/", "s", "c", "r", "i", "p", "t", " ", "');"]; + assert_eq!(written(&chunks, true), "a('\\u003c/script ');"); + } + + #[test] + fn text_the_origin_sent_is_written_as_it_is() { + let legacy = "<!-- document.write('<script src=\"a.js\"></script>'); //-->"; + assert_eq!(written(&[legacy], false), legacy); + let chunks = [ + "<!-- document.write('<scr", + "ipt src=\"a.js\"></scr", + "ipt>'); //-->", + ]; + assert_eq!(written(&chunks, false), chunks.concat()); + } +} diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index efff381e22..c3a10dc485 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -838,6 +838,177 @@ fn a_handler_that_holds_a_script_writes_it_back_whole() { ); } +const SCRIPT_PAGE: &str = "<html><body><script>one(); two();</script><p>after</p></body></html>"; + +/// A middleware that writes `script` as the text of every script. +fn writes_script(script: &str) -> Arc<dyn Middleware> { + let script = script.to_owned(); + text("example.script", "script", move || { + let script = script.clone(); + move |_text: &str, is_last: bool| { + if is_last { + ScriptRewriteAction::replace(script.clone()) + } else { + ScriptRewriteAction::remove_node() + } + } + }) +} + +/// The text of each script in `page`, and the name of each element, as a +/// browser's tokenizer reads the page. +fn as_a_browser_reads(page: &str) -> (Vec<String>, Vec<String>) { + let scripts = std::cell::RefCell::new((String::new(), Vec::new())); + let elements = std::cell::RefCell::new(Vec::new()); + lol_html::rewrite_str( + page, + lol_html::RewriteStrSettings { + element_content_handlers: vec![ + lol_html::element!("*", |element| { + elements.borrow_mut().push(element.tag_name()); + Ok(()) + }), + lol_html::text!("script", |text| { + let mut scripts = scripts.borrow_mut(); + let (script, read) = &mut *scripts; + script.push_str(text.as_str()); + if text.last_in_text_node() { + read.push(std::mem::take(script)); + } + Ok(()) + }), + ], + ..lol_html::RewriteStrSettings::new() + }, + ) + .expect("should read the page"); + (scripts.into_inner().1, elements.into_inner()) +} + +/// Asserts that a browser reads `page` as one script holding `script`, +/// followed by the paragraph, and no other element. +fn assert_script_stays_whole(page: &str, script: &str) { + let (scripts, elements) = as_a_browser_reads(page); + assert_eq!( + scripts, + [script], + "should read the script's text as written, up to its own end tag: {page}" + ); + assert_eq!( + elements, + ["html", "body", "script", "p"], + "should read what follows the script as markup, and nothing in it: {page}" + ); +} + +#[test] +fn script_text_that_would_end_the_script_is_written_so_the_script_stays_whole() { + for (written, expected) in [ + ( + "var s = \"</script><img src=x onerror=alert(1)>\";", + "var s = \"\\u003c/script><img src=x onerror=alert(1)>\";", + ), + ( + "var s = \"</SCRIPT ><img src=x onerror=alert(1)>\";", + "var s = \"\\u003c/SCRIPT ><img src=x onerror=alert(1)>\";", + ), + ] { + let page = page(&chain_of(vec![writes_script(written)]), SCRIPT_PAGE); + + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should write the `<` of the closing tag as `\\u003c`: {page}" + ); + assert_script_stays_whole(&page, expected); + } +} + +#[test] +fn script_text_with_a_comment_opening_leaves_the_end_tag_closing_the_script() { + // After `<!--`, `<script` would keep the script's own end tag from + // closing it, and the rest of the page would be read as script. + for (written, expected) in [ + ("var s = \"<!--\";", "var s = \"<!--\";"), + ( + "var s = \"<!--<script>\";", + "var s = \"<!--\\u003cscript>\";", + ), + ] { + let page = page(&chain_of(vec![writes_script(written)]), SCRIPT_PAGE); + + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should write `<!--` as it is, and the `<` of a script tag after it as \ + `\\u003c`: {page}" + ); + assert_script_stays_whole(&page, expected); + } +} + +#[test] +fn script_text_written_in_pieces_cannot_end_the_script_between_them() { + // The page is read a byte at a time, so the script's text arrives in + // several chunks. The handler writes the `<` of a closing tag in place + // of the first chunk and the rest of the tag in place of the last. + let chain = chain_of(vec![text("example.pieces", "script", || { + let mut first = true; + move |_text: &str, is_last: bool| { + if is_last { + ScriptRewriteAction::replace("/script><img src=x onerror=alert(1)>") + } else if std::mem::take(&mut first) { + ScriptRewriteAction::replace("<") + } else { + ScriptRewriteAction::remove_node() + } + } + })]); + + let page = page_with(config(), &chain, SCRIPT_PAGE, 1); + + let expected = "\\u003c/script><img src=x onerror=alert(1)>"; + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should escape a closing tag whose `<` was written with an earlier chunk: {page}" + ); + assert_script_stays_whole(&page, expected); +} + +#[test] +fn script_text_a_serve_middleware_writes_cannot_end_the_script_either() { + let chain = serve_chain_of(vec![writes_script( + "var s = \"</script><img src=x onerror=alert(1)>\";", + )]); + + let page = served(&chain, SCRIPT_PAGE); + + let expected = "var s = \"\\u003c/script><img src=x onerror=alert(1)>\";"; + assert!( + page.contains(&format!("<script>{expected}</script><p>after</p>")), + "should keep the text a serve middleware writes inside its script: {page}" + ); + assert_script_stays_whole(&page, expected); +} + +#[test] +fn a_script_no_middleware_changes_is_written_as_the_page_wrote_it() { + // An old page hides a script inside `<!--` and writes a script tag from + // within it, which the parser reads as the script's own text. + let html = "<html><body><script><!--\ndocument.write('<script src=\"a.js\"></script>');\n\ + //--></script><p>after</p></body></html>"; + let chain = chain_of(vec![text("example.keep", "script", || { + |_text: &str, _is_last: bool| ScriptRewriteAction::keep() + })]); + + for chunk_size in [1, 7, 8192] { + assert_eq!( + page_with(config(), &chain, html, chunk_size), + html, + "should leave a script no middleware changed byte for byte, read {chunk_size} \ + bytes at a time" + ); + } +} + /// Appends a comment to the document, and swaps one word for another. struct Rewording { from: &'static str, From ae9f360cafdf3be90a6402edbc39a662f8e4e43e Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 16 Oct 2026 09:15:42 +0100 Subject: [PATCH 34/36] Say how script text and a path of / are handled in the guides The integration guide said a text handler's replacement is written with no character escaped. It now says that a `<` that would end the script, or begin `<script` after `<!--`, is written as `\u003c`. The configuration guide's field table says `/` covers every path, as no path does, and the changelog entry for page changes as middleware states both. --- CHANGELOG.md | 2 +- docs/guide/configuration.md | 10 +++++----- docs/guide/integration-guide.md | 5 +++-- 3 files changed, 9 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54059b201e..92584215fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,7 +70,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - `ts audit generate` writes the `[[fetch]]` and `[[serve]]` entries that run the page changes of the modules it selects on every HTML page, and `trusted-server.example.toml` documents the entries with every middleware the stock modules supply. `trusted-server-modules` lists those middleware with `middleware()` and `middleware_for(...)`. -- Page changes as middleware. A module registers a change it makes to a page with `.with_middleware(...)`, and the ordered `[[fetch]]` and `[[serve]]` entries of the settings say which pages each one runs on and in what order. A fetch middleware runs on the page as the origin sent it and what it writes is stored with a shared template. A serve middleware runs on each reader's copy, whether the page came from the store or from the origin, and nothing it writes is stored. +- Page changes as middleware. A module registers a change it makes to a page with `.with_middleware(...)`, and the ordered `[[fetch]]` and `[[serve]]` entries of the settings say which pages each one runs on and in what order. A fetch middleware runs on the page as the origin sent it and what it writes is stored with a shared template. A serve middleware runs on each reader's copy, whether the page came from the store or from the origin, and nothing it writes is stored. Where text a middleware writes into a script would end the script, or begin `<script` after `<!--`, its `<` is written as `\u003c`. An entry with no path after one with `path = "/"` is refused, because both cover every page. - A builder registers its module from the compiled auction plan with `with_plan_registration` and checks its settings against the plan with `with_plan_validator`, so a module whose page support follows what the plan selects is not named in `trusted-server-core`. Prebid and APS register this way. A renderer descriptor carries its builder's statement of which payload key holds the identifier the renderer picks its bid by, made with `BidRenderer::picking_bid_by`, and the page's `hb_adid` is read through that statement. - A module declares the settings in its own table that hold the name of a secret, with `with_secret_settings` on its builder, each with a rule for when the table puts it to use. A setting in use is looked up in the default secret store as the settings load and has to name a key for deploy validation to pass, and one not in use is cleared and never looked up. DataDome's server-side key and its test bypass credential are declared this way, and `trusted-server-core` no longer names them. - A module can act on one request from start to finish without `trusted-server-core` naming it. A request preparer or a request filter leaves a value in the request's `IntegrationRequestState`, which the module's serve middleware reads from the state of the reader's copy and which is handed to the response finalizer the builder declares with `with_response_finalizer`. A request that carries any value keeps to the origin path, so its document is never read from or stored as a shared template, and its HTML response is sent `private, no-store`. A middleware can also write straight after the script bundle with `after_bundle_inserts`, and a builder declares with `with_auction_token` that its browser script reads the token an auction publishes with its winning bids. The registry runs the request preparers once for a request. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index f3ddb1999a..71a593ef81 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1752,11 +1752,11 @@ startup. An entry of either list has the same three fields. -| Field | Type | Required | Description | -| ------------ | ---------------- | -------- | -------------------------------------------------------------------------------------- | -| `media_type` | String | Yes | The media type the entry covers. Only `text/html` is accepted | -| `path` | String | No | A prefix of the request path, compared as written. Absent, the entry covers every path | -| `middleware` | Array of strings | Yes | The middleware to run, in order, each handed the page as the one before it left it | +| Field | Type | Required | Description | +| ------------ | ---------------- | -------- | ---------------------------------------------------------------------------------------------- | +| `media_type` | String | Yes | The media type the entry covers. Only `text/html` is accepted | +| `path` | String | No | A prefix of the request path, compared as written. Absent, or `/`, the entry covers every path | +| `middleware` | Array of strings | Yes | The middleware to run, in order, each handed the page as the one before it left it | A page takes the first entry that covers it and runs that entry's middleware alone. Entries are therefore written from the longest path to the entry with diff --git a/docs/guide/integration-guide.md b/docs/guide/integration-guide.md index 1c5db23e2d..5b47769c6d 100644 --- a/docs/guide/integration-guide.md +++ b/docs/guide/integration-guide.md @@ -444,8 +444,9 @@ A middleware can rely on these rules. removed is not asked about again. - A script's text arrives in chunks. A handler that needs the whole script removes each chunk while keeping a copy, and writes back what it kept with - the last chunk. A replacement is written as it is, with no character - escaped. + the last chunk. A replacement is written as it is, except that a `<` that + would end the script, or begin `<script` after `<!--`, is written as + `\u003c`. - A fetch middleware is told nothing about the reader, because what it leaves is what a shared template stores. `MiddlewareContext::document_state` is shared by the middleware working on one document in one phase. From e3707750b46e987f3f6fdd1ee23487fe03006d40 Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Mon, 12 Oct 2026 18:40:35 +0100 Subject: [PATCH 35/36] Move the origin's addresses in one pass, never reading a replacement again Core moved the origin's address in `href`, `src` and `action` with three replacements in a row, for the `https://`, `http://` and protocol-relative spellings, in `imagesrcset` with the same three, and in `srcset` with a fourth for the bare host. Each read the previous one's output. When the request host begins with the origin host, a later spelling matched inside an earlier replacement and the request host's tail was written again. With `publisher.example` served as `publisher.example.localhost` over `http`, every `href` came out as `http://publisher.example.localhost.localhost.localhost/`, and over `https` with one extra `.localhost`. The bare-host replacement in `srcset` did the same to a `www.` host in front of an apex origin, writing `https://example.com/a.png` as `https://www.www.example.com/a.png`. `replace_each_once` scans a value once from the start. It takes the earliest match of any spelling, the longer of two that start at the same place, writes that spelling's replacement and carries on after the matched text, so no replacement is read again. `UrlPatterns` builds the spellings once per document. A value that begins with the bare host is still moved only when the host ends there. The tests check that a request host which begins with the origin host is written once in `href`, `src`, `action`, `srcset` and `imagesrcset` over both schemes, that a hyphenated host and a `www.` host over an apex origin are written as before, and `replace_each_once` on its own. --- .../trusted-server-core/src/html_processor.rs | 241 +++++++++++++----- 1 file changed, 183 insertions(+), 58 deletions(-) diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index d3961d09e5..cb85e9f1bf 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -447,6 +447,34 @@ fn build_serve_processor(plan: MiddlewarePlan) -> HtmlWithStreamingProcessors { } } +/// `value` with each occurrence of a form's text replaced by that form's +/// replacement, in one pass from the start, so no replacement is read again. +/// +/// The earliest match is taken each time, and the longer of two that start at +/// the same place. A form with empty text is ignored. +fn replace_each_once(value: &str, forms: &[(String, String)]) -> String { + let mut out = String::with_capacity(value.len()); + let mut rest = value; + loop { + let next = forms + .iter() + .filter(|(text, _)| !text.is_empty()) + .filter_map(|(text, replacement)| { + rest.find(text.as_str()) + .map(|at| (at, text.len(), replacement)) + }) + .min_by(|a, b| a.0.cmp(&b.0).then_with(|| b.1.cmp(&a.1))); + let Some((at, len, replacement)) = next else { + break; + }; + out.push_str(&rest[..at]); + out.push_str(replacement); + rest = &rest[at + len..]; + } + out.push_str(rest); + out +} + /// The processor behind [`create_html_processor`] and /// [`create_html_processor_with_middleware`], carrying out `plan`. /// @@ -467,32 +495,36 @@ fn build_html_processor( text_handlers: middleware_text_handlers, processors: middleware_processors, } = plan; - // Simplified URL patterns structure - stores only core data and generates variants on-demand + // The spellings of the origin's address in a URL attribute, each with its + // replacement, built once per document. They are replaced in one pass and a + // replacement is never read again, because a request host that begins with + // the origin host would otherwise match a later spelling inside an earlier + // spelling's replacement. struct UrlPatterns { origin_host: String, request_host: String, - request_scheme: String, + /// The `https://`, `http://` and protocol-relative spellings. + url_forms: Vec<(String, String)>, + /// The same and the bare host, which a `srcset` candidate may name. + srcset_forms: Vec<(String, String)>, } impl UrlPatterns { - fn https_origin(&self) -> String { - format!("https://{}", self.origin_host) - } - - fn http_origin(&self) -> String { - format!("http://{}", self.origin_host) - } - - fn protocol_relative_origin(&self) -> String { - format!("//{}", self.origin_host) - } - - fn replacement_url(&self) -> String { - format!("{}://{}", self.request_scheme, self.request_host) - } - - fn protocol_relative_replacement(&self) -> String { - format!("//{}", self.request_host) + fn new(origin_host: &str, request_host: &str, request_scheme: &str) -> Self { + let replacement_url = format!("{request_scheme}://{request_host}"); + let url_forms = vec![ + (format!("https://{origin_host}"), replacement_url.clone()), + (format!("http://{origin_host}"), replacement_url), + (format!("//{origin_host}"), format!("//{request_host}")), + ]; + let mut srcset_forms = url_forms.clone(); + srcset_forms.push((origin_host.to_owned(), request_host.to_owned())); + Self { + origin_host: origin_host.to_owned(), + request_host: request_host.to_owned(), + url_forms, + srcset_forms, + } } fn rewrite_url_value(&self, value: &str) -> Option<String> { @@ -500,35 +532,32 @@ fn build_html_processor( return None; } - let https_origin = self.https_origin(); - let http_origin = self.http_origin(); - let protocol_relative_origin = self.protocol_relative_origin(); - let replacement_url = self.replacement_url(); - let protocol_relative_replacement = self.protocol_relative_replacement(); - - let mut rewritten = value - .replace(&https_origin, &replacement_url) - .replace(&http_origin, &replacement_url) - .replace(&protocol_relative_origin, &protocol_relative_replacement); - - if rewritten.starts_with(&self.origin_host) { - let suffix = &rewritten[self.origin_host.len()..]; - let boundary_ok = suffix.is_empty() - || matches!(suffix.as_bytes().first(), Some(b'/' | b'?' | b'#')); - if boundary_ok { - rewritten = format!("{}{}", self.request_host, suffix); + // A value that begins with the bare host is moved when the host + // ends there, so `origin.example.com/news` moves and + // `origin.example.com.mirror.example` does not. + let rewritten = match value.strip_prefix(self.origin_host.as_str()) { + Some(rest) + if rest.is_empty() + || matches!(rest.as_bytes().first(), Some(b'/' | b'?' | b'#')) => + { + format!( + "{}{}", + self.request_host, + replace_each_once(rest, &self.url_forms) + ) } - } + _ => replace_each_once(value, &self.url_forms), + }; (rewritten != value).then_some(rewritten) } } - let patterns = Rc::new(UrlPatterns { - origin_host: config.origin_host.clone(), - request_host: config.request_host.clone(), - request_scheme: config.request_scheme.clone(), - }); + let patterns = Rc::new(UrlPatterns::new( + &config.origin_host, + &config.request_host, + &config.request_scheme, + )); let injected_tsjs = Rc::new(Cell::new(false)); let injected_bids = Arc::new(AtomicBool::new(false)); @@ -711,14 +740,7 @@ fn build_html_processor( let patterns = patterns.clone(); move |el| { if let Some(srcset) = el.get_attribute("srcset") { - let rewritten = srcset - .replace(&patterns.https_origin(), &patterns.replacement_url()) - .replace(&patterns.http_origin(), &patterns.replacement_url()) - .replace( - &patterns.protocol_relative_origin(), - &patterns.protocol_relative_replacement(), - ) - .replace(&patterns.origin_host, &patterns.request_host); + let rewritten = replace_each_once(&srcset, &patterns.srcset_forms); if rewritten != srcset { el.set_attribute("srcset", &rewritten)?; } @@ -731,13 +753,7 @@ fn build_html_processor( let patterns = patterns.clone(); move |el| { if let Some(imagesrcset) = el.get_attribute("imagesrcset") { - let rewritten = imagesrcset - .replace(&patterns.https_origin(), &patterns.replacement_url()) - .replace(&patterns.http_origin(), &patterns.replacement_url()) - .replace( - &patterns.protocol_relative_origin(), - &patterns.protocol_relative_replacement(), - ); + let rewritten = replace_each_once(&imagesrcset, &patterns.url_forms); if rewritten != imagesrcset { el.set_attribute("imagesrcset", &rewritten)?; } @@ -1319,6 +1335,115 @@ mod tests { ); } + /// A request host that begins with the origin host, as a local host named + /// after the publisher does, is written once in every address over either + /// scheme, rather than matched again inside its own replacement. + #[test] + fn a_request_host_that_begins_with_the_origin_host_is_written_once() { + for scheme in ["http", "https"] { + let mut config = create_test_config(); + config.origin_host = "publisher.example".to_owned(); + config.request_host = "publisher.example.localhost".to_owned(); + config.request_scheme = scheme.to_owned(); + let mut processor = create_html_processor(config); + + let output = processor + .process_chunk( + br#"<html><head><link rel="preload" as="image" imagesrcset="https://publisher.example/a.png 1x, //publisher.example/b.png 2x"></head><body><a href="https://publisher.example/news">n</a><a href="http://publisher.example/old">o</a><a href="publisher.example/bare">b</a><img src="//publisher.example/c.png" srcset="https://publisher.example/d.png 1x, publisher.example/e.png 2x"><form action="https://publisher.example/search"></form></body></html>"#, + true, + ) + .expect("should process the document"); + let processed = String::from_utf8(output).expect("should stay UTF-8"); + + assert!( + !processed.contains(".localhost.localhost"), + "should write the request host once over {scheme}: {processed}" + ); + for expected in [ + format!(r#"href="{scheme}://publisher.example.localhost/news""#), + format!(r#"href="{scheme}://publisher.example.localhost/old""#), + r#"href="publisher.example.localhost/bare""#.to_owned(), + r#"src="//publisher.example.localhost/c.png""#.to_owned(), + format!( + r#"srcset="{scheme}://publisher.example.localhost/d.png 1x, publisher.example.localhost/e.png 2x""# + ), + format!(r#"action="{scheme}://publisher.example.localhost/search""#), + format!( + r#"imagesrcset="{scheme}://publisher.example.localhost/a.png 1x, //publisher.example.localhost/b.png 2x""# + ), + ] { + assert!( + processed.contains(&expected), + "should contain {expected} over {scheme}: {processed}" + ); + } + } + } + + /// Hosts that do not begin with the origin host are written as before, + /// both one with no part of the origin host in it and a `www.` host in + /// front of an apex origin, whose `srcset` must not become `www.www.`. + #[test] + fn a_request_host_that_does_not_begin_with_the_origin_host_is_written_as_before() { + for (origin, request) in [ + ("publisher.example", "publisher-example.localhost"), + ("example.com", "www.example.com"), + ] { + let mut config = create_test_config(); + config.origin_host = origin.to_owned(); + config.request_host = request.to_owned(); + config.request_scheme = "https".to_owned(); + let mut processor = create_html_processor(config); + + let input = format!( + r#"<html><head></head><body><a href="https://{origin}/a">a</a><a href="http://{origin}/b">b</a><img src="//{origin}/c.png" srcset="https://{origin}/d.png 1x, {origin}/e.png 2x"></body></html>"# + ); + let output = processor + .process_chunk(input.as_bytes(), true) + .expect("should process the document"); + let processed = String::from_utf8(output).expect("should stay UTF-8"); + + for expected in [ + format!(r#"href="https://{request}/a""#), + format!(r#"href="https://{request}/b""#), + format!(r#"src="//{request}/c.png""#), + format!(r#"srcset="https://{request}/d.png 1x, {request}/e.png 2x""#), + ] { + assert!( + processed.contains(&expected), + "should contain {expected} for {origin} served as {request}: {processed}" + ); + } + } + } + + #[test] + fn each_form_is_replaced_once_from_the_start() { + let forms = vec![ + ( + "https://a.example".to_owned(), + "http://a.example.localhost".to_owned(), + ), + ( + "http://a.example".to_owned(), + "http://a.example.localhost".to_owned(), + ), + ("//a.example".to_owned(), "//a.example.localhost".to_owned()), + ("a.example".to_owned(), "a.example.localhost".to_owned()), + (String::new(), "never".to_owned()), + ]; + + assert_eq!( + replace_each_once( + "https://a.example/1 http://a.example/2 //a.example/3 a.example/4", + &forms + ), + "http://a.example.localhost/1 http://a.example.localhost/2 //a.example.localhost/3 a.example.localhost/4" + ); + assert_eq!(replace_each_once("nothing here", &forms), "nothing here"); + assert_eq!(replace_each_once("", &forms), ""); + } + #[test] fn test_html_processor_config_from_settings() { let settings = create_test_settings(); From d31af8d5f9d7778f4677f7e28233fc7a6ac61629 Mon Sep 17 00:00:00 2001 From: James Rosewell <james@51degrees.com> Date: Fri, 16 Oct 2026 17:26:16 +0100 Subject: [PATCH 36/36] Say in the changelog that the origin's address is written once --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54059b201e..723b22391a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -60,6 +60,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- The origin's address in a page's `href`, `src`, `action`, `srcset` and `imagesrcset` is now replaced in one pass. A request host that begins with the origin host, such as `publisher.example.localhost` for `publisher.example`, was matched again inside its own replacement, so `https://publisher.example/` served over `http` came out as `http://publisher.example.localhost.localhost.localhost/`. A `srcset` served on a `www.` host in front of an apex origin came out with `www.www.`. - Image Optimizer configuration now rejects profile and profile-set names within the same map that collide after trimming, even when their values match, instead of silently overwriting an entry. Empty names continue to be discarded. - Browser integration bundles now share context providers and the configured log level with TSJS core (#1196). With Permutive enabled and cohorts available, `tsjs.requestAds()` now sends `config.permutive_segments` to `/auction`; the server retains that key only when `auction.allowed_context_keys` includes `"permutive_segments"`. This restores a previously inactive data flow, not a consent gate: review publisher consent handling before relying on it. Integration logs now follow `tsjs.setConfig` and `tsjs.log.setLevel`, and creative no longer raises the default `warn` level to `info`. The `?tsdebug=1` query parameter or `localStorage['tsdebug'] = '1'` intentionally enables `debug` logging across all bundles when creative installs, overriding any log level set earlier through `tsjs.setConfig` or `tsjs.log.setLevel`. Later calls to either API can change the level again. - `[auction].allowed_context_keys` now serializes in sorted, deduplicated order, so ESI template-cache fingerprints and `ts config diff`/`push` envelope hashes are stable across loads. Template fingerprints also sort object keys independently of `serde_json/preserve_order`. Existing envelopes may show a one-time allowlist reorder after upgrading; push once to settle it. The updated fingerprint format causes one template-cache miss per cached page after deployment.