From 7b0c211389b6b72c32c71a8369f9afec64f45d95 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Wed, 14 Oct 2026 23:01:04 +0100 Subject: [PATCH 1/6] The Flight rewriter is a middleware of the Next.js module Core rewrote the addresses in a React Server Components Flight stream by its media type, in a module of its own, and the publisher path named text/x-component to run it in place of the plain replacement it makes in any other text body. That was the one piece of a framework's handling left in core. The rewriter moves to crates/framework/nextjs as the fetch middleware framework.nextjs.flight, registered with the module and run where a [[fetch]] entry for text/x-component names it. Core knows nothing of Flight, and text/x-component is a media type an entry may cover, as every other text type is. A plain replacement in a length prefixed row leaves the old length, so the middleware says with Middleware::moves_origin_address that it moves the origin's address itself, and core leaves the addresses in a body of the media types such a middleware handles to it, whether or not an entry names it. With the module selected and no entry, a Flight response passes through as the origin sent it. The registry answers that question for a media type, and the publisher path's text pass skips its replacer on that answer. The middleware works from the origin host and the reader's host and scheme its context carries, as the module's HTML middleware does, so a folder in the origin's address stays in a rewritten address. The stock middleware list, the configuration guide, the Next.js guide, the RSC hydration guide, the template and the module's recorded page settings carry the entry, and the design page records the change. --- CHANGELOG.md | 3 +- .../src/fixtures/page-change.settings.toml | 4 + .../nextjs/src/flight.rs} | 262 +++++++++++------ crates/framework/nextjs/src/lib.rs | 107 +++++++ crates/trusted-server-core/README.md | 5 +- .../trusted-server-core/src/host_rewrite.rs | 4 +- .../src/integrations/registry.rs | 95 ++++++- crates/trusted-server-core/src/lib.rs | 1 - crates/trusted-server-core/src/middleware.rs | 20 +- .../src/middleware/tests.rs | 6 +- .../trusted-server-core/src/platform/mod.rs | 2 +- crates/trusted-server-core/src/publisher.rs | 268 +++++++++++++----- crates/trusted-server-modules/src/lib.rs | 14 +- docs/guide/configuration.md | 28 +- docs/guide/integrations/nextjs.md | 14 +- docs/guide/rsc-hydration.md | 11 +- ...2026-10-14-rsc-flight-middleware-design.md | 57 ++++ trusted-server.example.toml | 9 + 18 files changed, 706 insertions(+), 204 deletions(-) rename crates/{trusted-server-core/src/rsc_flight.rs => framework/nextjs/src/flight.rs} (57%) create mode 100644 docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 6373a277bd..e3eeea58f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** The React Server Components Flight rewriter is the middleware `framework.nextjs.flight` of the Next.js module, which runs on a `text/x-component` response where a `[[fetch]]` entry names it, where core rewrote every Flight response by media type. A deployment that fronts a Next.js site selects `[framework] module = "nextjs"` and adds the entry, as `trusted-server.example.toml` shows it. With the module selected and no entry, a Flight response passes through as the origin sent it, and an entry may now cover `text/x-component`. A middleware says with `Middleware::moves_origin_address` that it moves the origin's address itself, and core then leaves the addresses in a body of the media types it handles to it, whether or not an entry names it, because a plain replacement in a length prefixed row leaves the old length. **For crates outside core**, `trusted_server_core::rsc_flight` is removed. - **Breaking:** Where content comes from is a source module. `[source]` selects the publisher's, and each `[[proxy.asset_routes]]` entry selects its own in a `source` table of the same shape, with `module` naming the source and its settings in the table beneath. Core supplies `origin`, an HTTP origin, and a module a build links can offer another through `IntegrationRegistrationBuilder::with_source_module`. A source answers for each request where that request's content comes from, and core fetches, caches, rewrites the origin host and follows the redirect rule from that answer. `[source]` is required. The origin settings move into the `origin` table, and a document that still writes one where it was is refused at startup naming the new place. An environment variable that set the address is now `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The old name is refused where the build reads variables, and `ts config push` ignores it, as it ignores any variable for a key the document does not hold. A path in an asset route's `url` names a folder the target path is added under, where a path was refused, a target path with a `.` or `..` segment under a folder is answered `400`, and a route applies `directory_index` as the publisher does. An existing deployment's stored pages and assets keep their keys. **For crates outside core**, `handle_asset_proxy_request` takes the route's source, which `IntegrationRegistry::route_source` gives, and `Publisher::origin_host`, `origin_host_header` and `origin_query_is_stripped` are replaced by `Settings::origin_source` and the `SourceLocation` a source answers. | Previous | Now | @@ -120,7 +121,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `PlatformValueCache`, the host's shared cache of small values with lifetimes, which `RuntimeServices` carries where the host supplies one. `lookup_or_reserve` obliges the caller that finds a key missing to fill it where the host can make other callers wait, and `remove` drops an entry. The Fastly adapter supplies one over Fastly's cache, answering the reservation with a transaction lookup and the removal with a purge of the entry's own surrogate key. - A file the origin does not have is made by the middleware a `[[fetch]]` entry with a `path`, for a media type other than HTML, names for it. When the origin answers `404 Not Found` for a path under it, the middleware runs on an empty document of the entry's media type and its output is the file, so `ads.txt` is made from the sellers lines alone where the publisher has none. With no such entry the origin's answer stands. - `[ads-txt] modules = ["sellers"]` adds the lines `[ads-txt.sellers] lines` gives to the publisher's `ads.txt`, from `crates/ads-txt/sellers`. The module is a middleware for `text/plain`, named in a `[[fetch]]` entry on the file's path, which passes the origin's file through and writes the lines after its last one, so the publisher's file stays authoritative, and a publisher whose origin has no file gets one made of the lines. Each line is checked as the settings load as a data record or a variable of the ads.txt grammar, and a selected module no entry places is refused. -- A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type except `text/x-component`, `application/javascript`, `application/json` or `image/svg+xml`, written as a bare `type/subtype` in lower case. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then, for a response that is not stored, the serve phase, and an action that decides about HTML there is refused by name. +- A `[[fetch]]` or `[[serve]]` entry may cover a media type other than `text/html`: another `text/` type, `application/javascript`, `application/json` or `image/svg+xml`, written as a bare `type/subtype` in lower case. A middleware says which media types it works on with `handles_media_type`, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built. On a media type other than HTML a middleware's stream processor alone runs on the body, after core has moved the origin's addresses, in the fetch phase and then, for a response that is not stored, the serve phase, and an action that decides about HTML there is refused by name. - The InMobi Choice consent prompt as a module, `[cmp] module = "inmobi"` with `[cmp.inmobi] script_url`, from `crates/cmp/inmobi`. Where a `[[fetch]]` entry names `cmp.inmobi`, its middleware writes the IAB TCF stub, InMobi's GPP stub and the deferred Choice loader into the head, in that order, and the two stubs' behavior is tested with Node's test runner, which CI now runs for every module's `tests/*.test.mjs`. The script URL is refused unless it is `https` on an InMobi Choice host with no user name or password, and a selected prompt that no entry places is refused. 51Degrees maintain the crate until InMobi adopt it. - `[robots-txt]` serves a publisher's `/robots.txt` from the settings. `modules` names what makes the file, in order. `refuse_all` and `allow_all` are Trusted Server's own, and any other name is a module that contributes rules through the new `RobotsTxtContributor` extension point, declared with `with_robots_txt_contributor`. Trusted Server writes the file, so a contributor cannot produce a malformed one, a refusal cannot be loosened, and a path in `always_allow` stays open to every crawler refused the whole site. With `refuse_all` selected every response an adapter finalizes carries `X-Robots-Tag: noindex, nofollow`. A module's answer is held in the key-value store until its refresh is due, a module that cannot answer has its last answer served, and with none held the answer is `503` and never an empty file. Without the section `/robots.txt` reaches the publisher's origin as before. - `ts keys rotate` and `ts keys deactivate` rotate and retire the request signing keys from an operator's machine. They write the two stores the service links as `jwks_store` and `signing_keys` through the `fastly` CLI, with the token from `FASTLY_API_TOKEN`. A private key reaches `fastly` on its standard input and never in its arguments, and a key id no platform can store is refused before anything is written. A command that cannot read the config store writes nothing, and a delete that failed part way can be run again. diff --git a/crates/framework/nextjs/src/fixtures/page-change.settings.toml b/crates/framework/nextjs/src/fixtures/page-change.settings.toml index b5fc71ebf8..5c2ee600a6 100644 --- a/crates/framework/nextjs/src/fixtures/page-change.settings.toml +++ b/crates/framework/nextjs/src/fixtures/page-change.settings.toml @@ -7,3 +7,7 @@ rewrite_attributes = ["href", "link", "url"] [[fetch]] media_type = "text/html" middleware = ["framework.nextjs"] + +[[fetch]] +media_type = "text/x-component" +middleware = ["framework.nextjs.flight"] diff --git a/crates/trusted-server-core/src/rsc_flight.rs b/crates/framework/nextjs/src/flight.rs similarity index 57% rename from crates/trusted-server-core/src/rsc_flight.rs rename to crates/framework/nextjs/src/flight.rs index dac4a4b6ae..82e4fa60da 100644 --- a/crates/trusted-server-core/src/rsc_flight.rs +++ b/crates/framework/nextjs/src/flight.rs @@ -1,7 +1,64 @@ +//! The Flight response the App Router answers a navigation with, as a fetch +//! middleware on `text/x-component`. +//! +//! A client side navigation fetches a React Server Components Flight stream, +//! the row format the `__next_f` scripts carry inside a page, here with no +//! script wrapper and no string escaping. The rows are: +//! - `:\n`, JSON ended by a newline +//! - `:\n`, tagged, ended by a newline +//! - `:T,`, tagged `T`, length delimited, no newline +//! +//! A `T` row's length is the byte length of its content, so moving an address +//! inside it changes the length and the header is written again with the new +//! one. Core's plain replacement would leave the old length, which is why the +//! middleware moves the addresses itself and core leaves the body to it. + use std::io; -use crate::host_rewrite::rewrite_bare_host_at_boundaries; -use crate::streaming_processor::StreamProcessor; +use trusted_server_core::host_rewrite::rewrite_bare_host_at_boundaries; +use trusted_server_core::middleware::{ + Middleware, MiddlewareAction, MiddlewareContext, MiddlewarePhase, +}; +use trusted_server_core::streaming_processor::StreamProcessor; + +/// The name a `[[fetch]]` entry for [`FLIGHT_MEDIA_TYPE`] writes. +pub const FLIGHT_MIDDLEWARE: &str = "framework.nextjs.flight"; + +/// The media type of a Flight response, which the middleware works on. +pub const FLIGHT_MEDIA_TYPE: &str = "text/x-component"; + +/// Moves the origin's address to the publisher's in a Flight response, on +/// the responses a `[[fetch]]` entry names [`FLIGHT_MIDDLEWARE`] for. +pub(crate) struct Flight; + +impl Middleware for Flight { + fn middleware_id(&self) -> &'static str { + FLIGHT_MIDDLEWARE + } + + fn handles_media_type(&self, media_type: &str) -> bool { + media_type == FLIGHT_MEDIA_TYPE + } + + fn moves_origin_address(&self) -> bool { + true + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + stream: Some(Box::new(RscFlightUrlRewriter::new( + context.origin_host, + context.request_host, + context.request_scheme, + ))), + ..MiddlewareAction::pass() + } + } +} #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum RowState { @@ -12,27 +69,18 @@ enum RowState { ChunkByLength, } -/// Rewrites URLs inside a React Server Components (RSC) Flight stream. +/// Rewrites the addresses inside a Flight stream row by row, writing a `T` +/// row's length again after its content changes. /// -/// Next.js (App Router) uses `react-server-dom-webpack` ("Flight") for navigation responses -/// and for inlined `__next_f` data. The wire format is a sequence of rows: -/// - `:\n` (JSON terminated by `\n`) -/// - `:\n` (tagged, terminated by `\n`) -/// - `:T,` (tagged by `T`, length-delimited, **no trailing newline**) -/// -/// For `T` rows, the length prefix is the UTF-8 byte length of the content bytes. If we rewrite -/// URLs inside the content, we must recompute the length and rewrite the header. -/// -/// ## Limitations -/// -/// This rewriter performs simple string replacement and does NOT handle JSON escape sequences. -/// URLs like `\/\/origin.example.com` (JSON-escaped slashes) will not be rewritten. This is -/// acceptable because Flight responses from client-side navigation typically contain plain URLs, -/// not doubly-escaped JSON-in-JS content. For inlined `__next_f` data in HTML (which can have -/// escape sequences), the HTML post-processor in `crates/framework/nextjs` handles those cases. -pub struct RscFlightUrlRewriter { +/// The replacement is the plain one core makes in any other text: the +/// origin's `https://` and `http://` addresses to the publisher's, then +/// protocol relative ones, then the bare host at hostname boundaries. It does +/// not read JSON escapes, so `\/\/origin.example.com` is left alone, which a +/// navigation response does not carry. The escaped form inside a page's +/// `__next_f` scripts is handled by the HTML middleware. +pub(crate) struct RscFlightUrlRewriter { origin_url: String, - origin_http_url: Option, + origin_http_url: String, origin_host: String, origin_protocol_relative: String, request_url: String, @@ -50,28 +98,15 @@ pub struct RscFlightUrlRewriter { impl RscFlightUrlRewriter { #[must_use] - pub fn new( - origin_host: &str, - origin_url: &str, - request_host: &str, - request_scheme: &str, - ) -> Self { - let request_url = format!("{request_scheme}://{request_host}"); - let origin_protocol_relative = format!("//{origin_host}"); - let request_protocol_relative = format!("//{request_host}"); - - let origin_http_url = origin_url - .strip_prefix("https://") - .map(|rest| format!("http://{rest}")); - + pub(crate) fn new(origin_host: &str, request_host: &str, request_scheme: &str) -> Self { Self { - origin_url: origin_url.to_owned(), - origin_http_url, + origin_url: format!("https://{origin_host}"), + origin_http_url: format!("http://{origin_host}"), origin_host: origin_host.to_owned(), - origin_protocol_relative, - request_url, + origin_protocol_relative: format!("//{origin_host}"), + request_url: format!("{request_scheme}://{request_host}"), request_host: request_host.to_owned(), - request_protocol_relative, + request_protocol_relative: format!("//{request_host}"), state: RowState::Id, row_id: Vec::new(), row_tag: None, @@ -97,21 +132,12 @@ impl RscFlightUrlRewriter { return bytes.to_vec(); }; - if !text.contains(&self.origin_host) && !text.contains(&self.origin_url) { - if let Some(http_url) = &self.origin_http_url { - if !text.contains(http_url) { - return bytes.to_vec(); - } - } else { - return bytes.to_vec(); - } + if !text.contains(&self.origin_host) { + return bytes.to_vec(); } - // Keep replacement semantics consistent with `create_url_replacer`. let mut rewritten = text.replace(&self.origin_url, &self.request_url); - if let Some(http_url) = &self.origin_http_url { - rewritten = rewritten.replace(http_url, &self.request_url); - } + rewritten = rewritten.replace(&self.origin_http_url, &self.request_url); rewritten = rewritten.replace( &self.origin_protocol_relative, &self.request_protocol_relative, @@ -137,7 +163,8 @@ impl RscFlightUrlRewriter { fn finalize_length_row(&mut self, out: &mut Vec) { let Some(tag) = self.row_tag else { - // Should never happen for length-delimited rows; fall back to passthrough. + // A length delimited row always has a tag, so this passes the + // bytes through as they came. out.extend_from_slice(&self.raw_header); out.extend_from_slice(&self.row_content); self.reset_row(); @@ -155,7 +182,7 @@ impl RscFlightUrlRewriter { out.push(b','); out.extend_from_slice(&rewritten); } else { - // Length-delimited row type we don't transform (e.g., future/binary Flight types). + // A length delimited row of another tag is not text to rewrite. out.extend_from_slice(format!("{:x}", self.declared_length).as_bytes()); out.push(b','); out.extend_from_slice(&self.row_content); @@ -206,7 +233,7 @@ impl StreamProcessor for RscFlightUrlRewriter { self.raw_header.push(b); self.state = RowState::ChunkByNewline; } else { - // Not a recognized tag; treat as first byte of a JSON row. + // Not a tag, so the first byte of a JSON row. self.row_tag = None; self.row_content.push(b); self.state = RowState::ChunkByNewline; @@ -237,7 +264,7 @@ impl StreamProcessor for RscFlightUrlRewriter { }; let end = i + pos; self.row_content.extend_from_slice(&chunk[i..end]); - i = end + 1; // Skip '\n' + i = end + 1; self.finalize_newline_row(&mut out); } RowState::ChunkByLength => { @@ -268,6 +295,13 @@ impl StreamProcessor for RscFlightUrlRewriter { #[cfg(test)] mod tests { + use std::sync::Arc; + + use trusted_server_core::html_processor::test_support::{ORIGIN_HOST, REQUEST_HOST}; + use trusted_server_core::integrations::IntegrationDocumentState; + use trusted_server_core::middleware::test_support::context; + use trusted_server_core::middleware::{HTML_MEDIA_TYPE, MiddlewareChain}; + use super::*; fn run_rewriter( @@ -294,18 +328,15 @@ mod tests { output } + fn rewriter() -> RscFlightUrlRewriter { + RscFlightUrlRewriter::new("origin.example.com", "proxy.example.com", "https") + } + #[test] fn rewrites_newline_rows() { let input = b"0:[\"https://origin.example.com/page\"]\n"; - let mut rewriter = RscFlightUrlRewriter::new( - "origin.example.com", - "https://origin.example.com", - "proxy.example.com", - "https", - ); - - let output = run_rewriter(&mut rewriter, input, 8); + let output = run_rewriter(&mut rewriter(), input, 8); let output_str = String::from_utf8(output).expect("should be valid UTF-8"); assert_eq!( output_str, "0:[\"https://proxy.example.com/page\"]\n", @@ -313,20 +344,25 @@ mod tests { ); } + #[test] + fn rewrites_the_http_address_to_the_request_scheme() { + let input = b"0:[\"http://origin.example.com/page\",\"//origin.example.com/a\"]\n"; + + let output = run_rewriter(&mut rewriter(), input, 8192); + let output_str = String::from_utf8(output).expect("should be valid UTF-8"); + assert_eq!( + output_str, "0:[\"https://proxy.example.com/page\",\"//proxy.example.com/a\"]\n", + "should move an http address to the request's scheme and a protocol relative one as it is" + ); + } + #[test] fn rewrites_t_rows_and_updates_length() { let t_content = r#"{"url":"https://origin.example.com/page"}"#; let json_row = "2:[\"ok\"]\n"; let input = format!("1:T{:x},{}{}", t_content.len(), t_content, json_row); - let mut rewriter = RscFlightUrlRewriter::new( - "origin.example.com", - "https://origin.example.com", - "proxy.example.com", - "https", - ); - - let output = run_rewriter(&mut rewriter, input.as_bytes(), 7); + let output = run_rewriter(&mut rewriter(), input.as_bytes(), 7); let output_str = String::from_utf8(output).expect("should be valid UTF-8"); let rewritten_t_content = r#"{"url":"https://proxy.example.com/page"}"#; @@ -348,15 +384,8 @@ mod tests { let t_content = r#"{"url":"https://origin.example.com/page"}"#; let input = format!("1:T{:x},{}", t_content.len(), t_content); - let mut rewriter = RscFlightUrlRewriter::new( - "origin.example.com", - "https://origin.example.com", - "proxy.example.com", - "https", - ); - // Split such that the header ends before the comma and content begins in a later chunk. - let output = run_rewriter(&mut rewriter, input.as_bytes(), 3); + let output = run_rewriter(&mut rewriter(), input.as_bytes(), 3); let output_str = String::from_utf8(output).expect("should be valid UTF-8"); let rewritten_t_content = r#"{"url":"https://proxy.example.com/page"}"#; @@ -372,14 +401,7 @@ mod tests { fn bare_host_rewrite_respects_hostname_boundaries() { let input = b"0:[\"cdn.origin.example.com\",\"notorigin.example.com\",\"origin.example.com.uk\",\"origin.example.com/news\",\"origin.example.com\"]\n"; - let mut rewriter = RscFlightUrlRewriter::new( - "origin.example.com", - "https://origin.example.com", - "proxy.example.com", - "https", - ); - - let output = run_rewriter(&mut rewriter, input, 5); + let output = run_rewriter(&mut rewriter(), input, 5); let output_str = String::from_utf8(output).expect("should be valid UTF-8"); assert_eq!( @@ -388,4 +410,68 @@ mod tests { "Output should only rewrite bare host occurrences" ); } + + #[test] + fn the_middleware_runs_on_flight_alone_in_the_fetch_phase_and_moves_the_address_itself() { + let middleware = Flight; + + assert_eq!(middleware.middleware_id(), FLIGHT_MIDDLEWARE); + assert!( + middleware.handles_media_type(FLIGHT_MEDIA_TYPE), + "should work on a Flight response" + ); + assert!( + !middleware.handles_media_type(HTML_MEDIA_TYPE) + && !middleware.handles_media_type("text/plain"), + "should work on no other media type" + ); + assert!( + middleware.moves_origin_address(), + "should tell core to leave the origin's address to it" + ); + assert_eq!(middleware.phases(), &[MiddlewarePhase::Fetch]); + } + + #[test] + fn the_middleware_s_plan_rewrites_a_flight_response_with_its_lengths() { + let chain = MiddlewareChain::new( + MiddlewarePhase::Fetch, + FLIGHT_MEDIA_TYPE, + vec![Arc::new(Flight) as Arc], + ); + let state = IntegrationDocumentState::default(); + let mut plan = chain + .plan(&context(MiddlewarePhase::Fetch, &state)) + .expect("should plan a stream processor alone"); + assert_eq!( + plan.processors.len(), + 1, + "should carry one stream processor and nothing about HTML: {plan:?}" + ); + let t_content = format!(r#"{{"url":"https://{ORIGIN_HOST}/page"}}"#); + let input = format!( + "1:T{:x},{t_content}2:[\"//{ORIGIN_HOST}/a\"]\n", + t_content.len() + ); + + let mut processor = plan.processors.remove(0); + let mut output = processor + .process_chunk(input.as_bytes(), false) + .expect("should process the stream"); + output.extend( + processor + .process_chunk(&[], true) + .expect("should finish the stream"), + ); + + let rewritten = format!(r#"{{"url":"https://{REQUEST_HOST}/page"}}"#); + assert_eq!( + String::from_utf8(output).expect("should be valid UTF-8"), + format!( + "1:T{:x},{rewritten}2:[\"//{REQUEST_HOST}/a\"]\n", + rewritten.len() + ), + "should move the origin's address and write the T row's length again" + ); + } } diff --git a/crates/framework/nextjs/src/lib.rs b/crates/framework/nextjs/src/lib.rs index 4d31ebbd5d..b9d07ecdcc 100644 --- a/crates/framework/nextjs/src/lib.rs +++ b/crates/framework/nextjs/src/lib.rs @@ -40,12 +40,14 @@ pub fn builder() -> trusted_server_core::integrations::IntegrationBuilder { .with_module_name(MODULE) } +mod flight; mod rsc; mod rsc_placeholders; mod rsc_stream; mod script_rewriter; mod shared; +pub use flight::{FLIGHT_MEDIA_TYPE, FLIGHT_MIDDLEWARE}; pub use rsc::rewrite_rsc_scripts_combined; use rsc_placeholders::{NextJsRscPlaceholderRewriter, RscPlaceholderText}; @@ -131,6 +133,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(NEXTJS_INTEGRATION_ID) .with_middleware(Arc::new(page_change)) + .with_middleware(Arc::new(flight::Flight)) .build(), )) } @@ -192,6 +195,7 @@ mod tests { create_page_processor, place_on_every_page, }; use trusted_server_core::integrations::IntegrationRegistry; + use trusted_server_core::middleware::{HTML_MEDIA_TYPE, PhaseEntries, PhaseEntry}; use trusted_server_core::streaming_processor::{ Compression, PipelineConfig, StreamingPipeline, }; @@ -205,6 +209,20 @@ mod tests { settings } + /// Test settings that select the module and place its HTML middleware on + /// every page. + fn settings_selecting_and_placing_the_module() -> Settings { + let mut settings = settings_placing_the_module(); + settings + .insert_module_config( + "framework", + MODULE, + &json!({"rewrite_attributes": ["href", "link", "url"]}), + ) + .expect("should select the nextjs integration"); + settings + } + fn config_from_settings( settings: &Settings, registry: &IntegrationRegistry, @@ -1007,6 +1025,95 @@ mod tests { ); } + /// The Flight middleware is registered with the module, in the fetch + /// phase, and tells core to leave a Flight response's addresses to it + /// whether or not an entry names it. + #[test] + fn the_flight_middleware_is_registered_with_the_module() { + let settings = settings_selecting_and_placing_the_module(); + let registry = IntegrationRegistry::with_registrations(&settings, &[builder()]) + .expect("should create registry"); + + assert!( + registry + .middleware_in(MiddlewarePhase::Fetch) + .contains(&FLIGHT_MIDDLEWARE), + "should register the Flight middleware for the fetch phase" + ); + assert!( + registry.middleware_moves_origin_address(FLIGHT_MEDIA_TYPE), + "core should leave a Flight response's addresses to the module" + ); + assert!( + !registry.middleware_moves_origin_address(HTML_MEDIA_TYPE) + && !registry.middleware_moves_origin_address("text/plain"), + "core should move the addresses in every other body" + ); + } + + /// A Flight response takes the entry for its media type, and runs + /// nothing where no entry covers it. + #[test] + fn a_flight_response_takes_the_entry_naming_the_flight_middleware() { + let fetch = |entries: Vec| { + let mut settings = settings_selecting_and_placing_the_module(); + settings.fetch = PhaseEntries::new(entries); + settings + }; + let html = PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![MODULE.to_owned()], + }; + let flight = PhaseEntry { + media_type: FLIGHT_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![FLIGHT_MIDDLEWARE.to_owned()], + }; + let chain_for = |settings: &Settings| { + IntegrationRegistry::with_registrations(settings, &[builder()]) + .expect("should create registry") + .middleware_chain( + settings.phase_entries(MiddlewarePhase::Fetch), + MiddlewarePhase::Fetch, + FLIGHT_MEDIA_TYPE, + "/news", + ) + .ids() + }; + + assert_eq!( + chain_for(&fetch(vec![html.clone(), flight])), + [FLIGHT_MIDDLEWARE], + "should run the Flight middleware on a Flight response its entry covers" + ); + assert!( + chain_for(&fetch(vec![html])).is_empty(), + "should run nothing on a Flight response no entry covers" + ); + } + + #[test] + fn an_entry_for_html_naming_the_flight_middleware_is_refused() { + let mut settings = settings_selecting_and_placing_the_module(); + settings.fetch = PhaseEntries::new(vec![PhaseEntry { + media_type: HTML_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![MODULE.to_owned(), FLIGHT_MIDDLEWARE.to_owned()], + }]); + + let error = IntegrationRegistry::with_registrations(&settings, &[builder()]) + .err() + .expect("should refuse the entry"); + + let message = format!("{error:?}"); + assert!( + message.contains(FLIGHT_MIDDLEWARE) + && message.contains("does not work on that media type"), + "should name the middleware and say it does not work on HTML: {message}" + ); + } + #[test] fn output_overflow_restores_an_in_progress_script_at_every_split() { use trusted_server_core::streaming_processor::StreamProcessor as _; diff --git a/crates/trusted-server-core/README.md b/crates/trusted-server-core/README.md index 067f9785a0..f6a37bf9b1 100644 --- a/crates/trusted-server-core/README.md +++ b/crates/trusted-server-core/README.md @@ -18,9 +18,8 @@ on an edge SDK, Tokio runtime, filesystem, socket, or host-only process API. post-processing, provider, and browser-module capabilities. - `ec` owns edge-cookie generation, consent decisions, identity graph access, and partner synchronization. -- `html_processor`, `host_rewrite`, `streaming_processor`, and `rsc_flight` - transform eligible publisher responses without corrupting non-HTML or RSC - payloads. +- `html_processor`, `host_rewrite`, and `streaming_processor` transform + eligible publisher responses without corrupting non-HTML payloads. - `proxy`, `creative`, `image_optimizer`, and `asset_routes` implement bounded first-party asset and creative handling. - `auth`, `request_signing`, `key_manager`, and `jwk` implement authentication diff --git a/crates/trusted-server-core/src/host_rewrite.rs b/crates/trusted-server-core/src/host_rewrite.rs index 2324a8a3d2..535f29854e 100644 --- a/crates/trusted-server-core/src/host_rewrite.rs +++ b/crates/trusted-server-core/src/host_rewrite.rs @@ -4,8 +4,8 @@ /// A numeric `:port` immediately after the host is treated as part of a standalone authority and /// is preserved when rewriting the host. /// -/// This is used by both HTML (`__next_f` payloads) and Flight (`text/x-component`) rewriting to -/// avoid corrupting unrelated hostnames. +/// Used where an address is written as a bare host, in a page's data and in a +/// text response, so an unrelated hostname is left alone. #[must_use] pub fn rewrite_bare_host_at_boundaries( text: &str, diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 56d0fd9737..04c03a06c1 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -2275,6 +2275,16 @@ impl IntegrationRegistry { .collect() } + /// Whether a registered middleware moves the origin's address itself in + /// a body of `media_type`, so core leaves the addresses in such a body + /// alone, see [`Middleware::moves_origin_address`]. + #[must_use] + pub fn middleware_moves_origin_address(&self, media_type: &str) -> bool { + self.inner.middleware.iter().any(|(_, middleware)| { + middleware.moves_origin_address() && middleware.handles_media_type(media_type) + }) + } + /// The middleware the first entry covering a response selects, in the /// order they run, for a response of `media_type` to a request for `path`. /// A response no entry covers gives an empty chain. @@ -3335,6 +3345,14 @@ pub(crate) mod test_support { pub(crate) const TEXT_MEDIA_TYPE: &str = "text/plain"; /// The line the text middleware adds at the end of a text body. pub(crate) const TEXT_LINE: &str = "# added by the middleware fixture\n"; + /// The name of the fetch middleware that moves the origin's address + /// itself, on a media type of its own. + pub(crate) const FRAMED: &str = "testing.middleware-fixture.framed"; + /// The media type the framed middleware works on, whose body core + /// leaves to it. + pub(crate) const FRAMED_MEDIA_TYPE: &str = "text/x-fixture-rows"; + /// The line the framed middleware adds at the end of a body. + pub(crate) const FRAMED_LINE: &str = "# framed by the middleware fixture\n"; /// The name of the fetch middleware that prepares. pub(crate) const PREPARING: &str = "testing.middleware-fixture.preparing"; /// The name of the fetch middleware that refuses every request. @@ -3640,8 +3658,8 @@ pub(crate) mod test_support { } } - /// Passes a text body through and adds a line at its end. - struct AppendLine; + /// Passes a text body through and adds the line at its end. + struct AppendLine(&'static str); impl crate::streaming_processor::StreamProcessor for AppendLine { fn process_chunk( @@ -3651,7 +3669,7 @@ pub(crate) mod test_support { ) -> Result, std::io::Error> { let mut output = chunk.to_vec(); if is_last { - output.extend_from_slice(TEXT_LINE.as_bytes()); + output.extend_from_slice(self.0.as_bytes()); } Ok(output) } @@ -3674,7 +3692,7 @@ pub(crate) mod test_support { fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { MiddlewareAction { - stream: Some(Box::new(AppendLine)), + stream: Some(Box::new(AppendLine(TEXT_LINE))), ..MiddlewareAction::pass() } } @@ -3706,6 +3724,36 @@ pub(crate) mod test_support { } } + /// Says it moves the origin's address itself, on its own media type, + /// and then leaves the body as it came with a line added, so a test + /// sees what core left to it. + struct Framed; + + impl Middleware for Framed { + fn middleware_id(&self) -> &'static str { + FRAMED + } + + fn handles_media_type(&self, media_type: &str) -> bool { + media_type == FRAMED_MEDIA_TYPE + } + + fn moves_origin_address(&self) -> bool { + true + } + + fn phases(&self) -> &[MiddlewarePhase] { + &[MiddlewarePhase::Fetch] + } + + fn create(&self, _context: &MiddlewareContext<'_>) -> MiddlewareAction { + MiddlewareAction { + stream: Some(Box::new(AppendLine(FRAMED_LINE))), + ..MiddlewareAction::pass() + } + } + } + struct Broken; impl Middleware for Broken { @@ -3754,6 +3802,7 @@ pub(crate) mod test_support { .with_middleware(Arc::new(TextHeader)) .with_middleware(Arc::new(Locating)) .with_middleware(Arc::new(Labeled { text: labeled })) + .with_middleware(Arc::new(Framed)) .build(), )) } @@ -7014,6 +7063,7 @@ mod tests { fixture::TEXT_HEADER, fixture::LOCATING, fixture::LABELED, + fixture::FRAMED, crate::service_version::MODULE, ], "should list the middleware the selected module supplies, and core's own after them" @@ -7027,7 +7077,8 @@ mod tests { fixture::TEXT, fixture::PREPARING, fixture::REFUSING, - fixture::LABELED + fixture::LABELED, + fixture::FRAMED ], "should list the middleware that run in the fetch phase" ); @@ -7060,6 +7111,30 @@ mod tests { ); } + #[test] + fn a_registered_middleware_that_moves_the_address_itself_says_so_for_its_media_type() { + let registry = IntegrationRegistry::new(&fixture_settings_with_entries(Vec::new())) + .expect("should build a registry"); + + assert!( + registry.middleware_moves_origin_address(fixture::FRAMED_MEDIA_TYPE), + "should leave the framed media type to the middleware, which no entry names" + ); + assert!( + !registry.middleware_moves_origin_address(fixture::TEXT_MEDIA_TYPE) + && !registry.middleware_moves_origin_address(HTML_MEDIA_TYPE), + "should move the addresses in a body of every other media type" + ); + + let unselected = + IntegrationRegistry::new(&crate::test_support::tests::create_test_settings()) + .expect("should build a registry without the fixture"); + assert!( + !unselected.middleware_moves_origin_address(fixture::FRAMED_MEDIA_TYPE), + "should move the addresses where no selected module's middleware claims the body" + ); + } + #[test] fn middleware_for_path_gives_every_covering_entry_s_middleware_once_in_entry_order() { let registry = IntegrationRegistry::new(&fixture_settings_with_entries(Vec::new())) @@ -7186,7 +7261,7 @@ mod tests { message.contains( "[[fetch]] entry 2 names `testing.nothing`, which no module that runs supplies" ) && message.contains(&format!( - "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}]", + "[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}]", fixture::HEAD, fixture::LINKS, fixture::BROKEN, @@ -7198,6 +7273,7 @@ mod tests { fixture::TEXT_HEADER, fixture::LOCATING, fixture::LABELED, + fixture::FRAMED, crate::service_version::MODULE )), "should name the entry and list what could be named: {message}" @@ -7377,6 +7453,7 @@ mod tests { fixture::TEXT_HEADER, fixture::LOCATING, fixture::LABELED, + fixture::FRAMED, ], "should list every middleware when there are no entries" ); @@ -7394,7 +7471,8 @@ mod tests { fixture::REFUSING, fixture::TEXT_HEADER, fixture::LOCATING, - fixture::LABELED + fixture::LABELED, + fixture::FRAMED ], "should leave out a middleware any entry names" ); @@ -7467,7 +7545,8 @@ mod tests { fixture::REFUSING, fixture::TEXT_HEADER, fixture::LOCATING, - fixture::LABELED + fixture::LABELED, + fixture::FRAMED ], "should count a middleware as named only by an entry of a phase it runs in" ); diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 3a84b26c25..aaca6e934d 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -73,7 +73,6 @@ pub mod redacted; pub mod request_signing; pub mod response_privacy; pub mod robots_txt; -pub mod rsc_flight; pub(crate) mod s3_sigv4; pub mod secret_resolution; pub mod service_version; diff --git a/crates/trusted-server-core/src/middleware.rs b/crates/trusted-server-core/src/middleware.rs index f9e3390f96..fea569efd8 100644 --- a/crates/trusted-server-core/src/middleware.rs +++ b/crates/trusted-server-core/src/middleware.rs @@ -62,12 +62,11 @@ pub const HTML_MEDIA_TYPE: &str = "text/html"; /// bare `type/subtype` in lower case. /// /// HTML and the text assets that stream through core as text: every `text/` -/// type except `text/x-component`, and JavaScript and JSON. +/// type, and JavaScript and JSON. #[must_use] pub fn is_dispatched_media_type(media_type: &str) -> bool { - media_type != "text/x-component" - && (media_type.starts_with("text/") - || matches!(media_type, "application/javascript" | "application/json")) + media_type.starts_with("text/") + || matches!(media_type, "application/javascript" | "application/json") } /// The type and subtype of a content type header, in lower case and without @@ -505,6 +504,16 @@ pub trait Middleware: Send + Sync { media_type == HTML_MEDIA_TYPE } + /// Whether this middleware moves the origin's address to the reader's + /// host itself, in a body of a media type it handles other than HTML. + /// Core then leaves the addresses in such a body alone, whether or not + /// an entry names the middleware, for a format whose rows count their + /// bytes and that core's plain replacement would break. False by + /// default, where core moves the addresses before the middleware runs. + fn moves_origin_address(&self) -> bool { + false + } + /// The phases an entry may name this middleware in. fn phases(&self) -> &[MiddlewarePhase]; @@ -698,8 +707,7 @@ impl PhaseEntries { return Err(format!( "{at} covers `{}`, which is not a media type a middleware runs on. Write a \ bare `type/subtype` in lower case with no parameters, being `text/html`, \ - another `text/` type other than `text/x-component`, \ - `application/javascript` or `application/json`", + another `text/` type, `application/javascript` or `application/json`", entry.media_type )); } diff --git a/crates/trusted-server-core/src/middleware/tests.rs b/crates/trusted-server-core/src/middleware/tests.rs index 37c9d94364..76e0ab38a7 100644 --- a/crates/trusted-server-core/src/middleware/tests.rs +++ b/crates/trusted-server-core/src/middleware/tests.rs @@ -361,7 +361,6 @@ fn an_entry_for_a_media_type_a_middleware_does_not_run_on_is_refused() { "text/*", "image/png", "application/octet-stream", - "text/x-component", "image/svg+xml", "text/", "text", @@ -388,6 +387,7 @@ fn an_entry_may_cover_a_text_media_type_that_is_not_html() { for media_type in [ "text/plain", "text/css", + "text/x-component", "application/javascript", "application/json", ] { @@ -409,8 +409,8 @@ fn a_media_type_is_the_type_and_subtype_of_a_content_type_in_lower_case() { assert!(is_dispatched_media_type("text/plain")); assert!(!is_dispatched_media_type("image/svg+xml")); assert!( - !is_dispatched_media_type("text/x-component"), - "a length prefixed stream is not changed" + is_dispatched_media_type("text/x-component"), + "a text type whose rows count their bytes is one a middleware may move the addresses of" ); assert!(!is_dispatched_media_type("image/png")); } diff --git a/crates/trusted-server-core/src/platform/mod.rs b/crates/trusted-server-core/src/platform/mod.rs index e00331317f..cac51e7b40 100644 --- a/crates/trusted-server-core/src/platform/mod.rs +++ b/crates/trusted-server-core/src/platform/mod.rs @@ -26,7 +26,7 @@ //! any content-rewriting implementation: //! //! - **Content rewriting** — `html_processor`, `streaming_processor`, -//! `streaming_replacer`, and `rsc_flight` modules use only standard Rust +//! and `streaming_replacer` modules use only standard Rust //! (`std::io::Read`/`Write`, `lol_html`, `flate2`, `brotli`). The pipeline //! is accessed via [`StreamingPipeline::process`](crate::streaming_processor::StreamingPipeline::process) which //! accepts any reader that implements `std::io::Read`. diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 890351127a..0a9f086804 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -89,7 +89,6 @@ use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, }; -use crate::rsc_flight::RscFlightUrlRewriter; use crate::settings::{ AUCTION_DEBUG_METADATA_ALLOWLIST, AUCTION_DEBUG_UPSTREAM_METADATA_KEYS, AssetOriginAuth, AuctionDebugCommentFormat, AuctionDebugCommentOptions, AuctionDebugCommentVerbosity, Settings, @@ -693,7 +692,10 @@ struct TextResponse<'a> { /// headers the serve entry's middleware write on the response, or none /// when its media type is not one a middleware runs on, its charset is not /// UTF-8 or no entry covers it. Both phases run here because the template -/// cache stores HTML alone. +/// cache stores HTML alone. The pass also says whether a registered +/// middleware moves the origin's address in a body of this media type +/// itself, which holds whatever the charset and whether or not an entry +/// names that middleware. /// /// # Errors /// @@ -709,11 +711,17 @@ fn middleware_stream_processors( }; let media_type = media_type_of(response.content_type); - if !is_dispatched_media_type(&media_type) - || !is_utf8_or_undeclared_charset(response.content_type) - { + if !is_dispatched_media_type(&media_type) { return Ok(TextPass::default()); } + let addresses_left_to_middleware = + integration_registry.middleware_moves_origin_address(&media_type); + if !is_utf8_or_undeclared_charset(response.content_type) { + return Ok(TextPass { + addresses_left_to_middleware, + ..TextPass::default() + }); + } let mut processors = Vec::new(); let mut response_headers = Vec::new(); for phase in &MiddlewarePhase::ALL { @@ -748,15 +756,49 @@ fn middleware_stream_processors( Ok(TextPass { processors, response_headers, + addresses_left_to_middleware, }) } /// What the entries covering a text response do to it: the processors over -/// its body and the headers the serve entry's middleware write. +/// its body, the headers the serve entry's middleware write, and whether +/// the origin's address in the body is left to a middleware. #[derive(Default)] struct TextPass { processors: Vec>, response_headers: Vec, + /// Whether a registered middleware moves the origin's address in the + /// body itself, so core's plain replacement does not run on it. + addresses_left_to_middleware: bool, +} + +impl TextPass { + /// The processor over the body, being core's move of the origin's + /// address unless a middleware moves it itself, and then the entries' + /// processors, with the headers the serve entry's middleware write. + fn into_processor( + self, + origin_host: &str, + origin_url: &str, + request_host: &str, + request_scheme: &str, + ) -> (WithStreamProcessors, Vec) { + let inner = (!self.addresses_left_to_middleware).then(|| { + Box::new(create_url_replacer( + origin_host, + origin_url, + request_host, + request_scheme, + )) as Box + }); + ( + WithStreamProcessors { + inner, + processors: self.processors, + }, + self.response_headers, + ) + } } /// Runs the preparation of the middleware the entries of each phase in @@ -871,16 +913,20 @@ fn refusal_response(refusal: &Refusal) -> Result, Report, + inner: Option>, processors: Vec>, } impl StreamProcessor for WithStreamProcessors { fn process_chunk(&mut self, chunk: &[u8], is_last: bool) -> Result, std::io::Error> { - let mut output = self.inner.process_chunk(chunk, is_last)?; + let mut output = match &mut self.inner { + Some(inner) => inner.process_chunk(chunk, is_last)?, + None => chunk.to_vec(), + }; for processor in &mut self.processors { output = processor.process_chunk(&output, is_last)?; } @@ -888,7 +934,9 @@ impl StreamProcessor for WithStreamProcessors { } fn reset(&mut self) { - self.inner.reset(); + if let Some(inner) = &mut self.inner { + inner.reset(); + } for processor in &mut self.processors { processor.reset(); } @@ -908,16 +956,14 @@ impl PublisherBodyProcessor { integration_registry: &IntegrationRegistry, ) -> Result<(Self, Option>), Report> { let is_html = is_html_content_type(¶ms.content_type); - let is_rsc_flight = - content_type_contains_ascii_case_insensitive(¶ms.content_type, "text/x-component"); let inline_seam_token = deferred_inline_seam_token( settings, params.template_cache_key.is_some(), params.ad_slots_script.is_some(), is_html && params.dispatched_auction.is_some(), ); - let mut response_headers = Vec::new(); - let inner: Box = if is_html { + let (inner, response_headers): (Box, Vec) = if is_html + { let (processor, headers) = create_html_stream_processor(HtmlStreamProcessorParams { origin_host: ¶ms.origin_host, request_host: ¶ms.request_host, @@ -937,17 +983,9 @@ impl PublisherBodyProcessor { .as_ref() .map(|token| String::from_utf8_lossy(token).into_owned()), })?; - response_headers = headers; - Box::new(processor) - } else if is_rsc_flight { - Box::new(RscFlightUrlRewriter::new( - ¶ms.origin_host, - ¶ms.origin_url, - ¶ms.request_host, - ¶ms.request_scheme, - )) + (Box::new(processor), headers) } else { - let pass = middleware_stream_processors( + let (processor, headers) = middleware_stream_processors( &TextResponse { content_type: ¶ms.content_type, request_path: ¶ms.request_path, @@ -960,17 +998,14 @@ impl PublisherBodyProcessor { }, settings, integration_registry, - )?; - response_headers = pass.response_headers; - Box::new(WithStreamProcessors { - inner: Box::new(create_url_replacer( - ¶ms.origin_host, - ¶ms.origin_url, - ¶ms.request_host, - ¶ms.request_scheme, - )), - processors: pass.processors, - }) + )? + .into_processor( + ¶ms.origin_host, + ¶ms.origin_url, + ¶ms.request_host, + ¶ms.request_scheme, + ); + (Box::new(processor), headers) }; Ok(( @@ -998,10 +1033,10 @@ impl StreamProcessor for PublisherBodyProcessor { /// Process response body through the streaming pipeline. /// -/// Selects the appropriate processor based on content type (HTML rewriter, -/// RSC Flight rewriter, or URL replacer) and pipes chunks from `body` -/// through it into `output`. The caller decides what `output` is — a -/// `Vec` for buffered responses, or a `StreamingBody` for streaming. +/// Selects the processor by content type, the HTML rewriter or the text +/// pass of the entries' middleware, and pipes chunks from `body` through it +/// into `output`. The caller decides what `output` is, a `Vec` for +/// buffered responses or a `StreamingBody` for streaming. /// /// # Errors /// @@ -1013,14 +1048,11 @@ fn process_response_streaming( output_compression: Compression, ) -> Result, Report> { let is_html = is_html_content_type(params.content_type); - let is_rsc_flight = - content_type_contains_ascii_case_insensitive(params.content_type, "text/x-component"); log::debug!( - "process_response_streaming: content_type={}, content_encoding={}, is_html={}, is_rsc_flight={}", + "process_response_streaming: content_type={}, content_encoding={}, is_html={}", params.content_type, params.content_encoding, - is_html, - is_rsc_flight + is_html ); let compression = Compression::from_content_encoding(params.content_encoding); @@ -1037,8 +1069,7 @@ fn process_response_streaming( // gzip would reject bodies the identity, deflate and brotli paths accept. let max_pending_decoded_bytes = params.settings.publisher.max_buffered_body_bytes; - let mut response_headers = Vec::new(); - if is_html { + let response_headers = if is_html { let (processor, headers) = create_html_stream_processor(HtmlStreamProcessorParams { origin_host: params.origin_host, request_host: params.request_host, @@ -1056,24 +1087,12 @@ fn process_response_streaming( public_prefix: params.public_prefix, deferred_inline_marker: None, })?; - response_headers = headers; - StreamingPipeline::new(config, processor) - .with_max_pending_decoded_bytes(max_pending_decoded_bytes) - .process(body_as_reader(body)?, output)?; - } else if is_rsc_flight { - // RSC Flight responses are length-prefixed (T rows). A naive string replacement will - // corrupt the stream by changing byte lengths without updating the prefixes. - let processor = RscFlightUrlRewriter::new( - params.origin_host, - params.origin_url, - params.request_host, - params.request_scheme, - ); StreamingPipeline::new(config, processor) .with_max_pending_decoded_bytes(max_pending_decoded_bytes) .process(body_as_reader(body)?, output)?; + headers } else { - let pass = middleware_stream_processors( + let (processor, headers) = middleware_stream_processors( &TextResponse { content_type: params.content_type, request_path: params.request_path, @@ -1086,21 +1105,18 @@ fn process_response_streaming( }, params.settings, params.integration_registry, - )?; - response_headers = pass.response_headers; - let processor = WithStreamProcessors { - inner: Box::new(create_url_replacer( - params.origin_host, - params.origin_url, - params.request_host, - params.request_scheme, - )), - processors: pass.processors, - }; + )? + .into_processor( + params.origin_host, + params.origin_url, + params.request_host, + params.request_scheme, + ); StreamingPipeline::new(config, processor) .with_max_pending_decoded_bytes(max_pending_decoded_bytes) .process(body_as_reader(body)?, output)?; - } + headers + }; Ok(response_headers) } @@ -23056,6 +23072,114 @@ mod tests { ); } + /// A body whose registered middleware moves the origin's address itself + /// is handed to that middleware as the origin sent it, so core's plain + /// replacement cannot break its rows. + #[test] + fn a_body_whose_middleware_moves_the_address_itself_is_handed_to_it_as_sent() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::FRAMED_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::FRAMED.to_owned()], + }]); + let body = "See https://origin.example.com/more\n"; + + let output = text_body_through_the_publisher(&settings, fixture::FRAMED_MEDIA_TYPE, body); + + assert_eq!( + output, + format!("{body}{}", fixture::FRAMED_LINE), + "should leave the origin's address to the middleware and run it" + ); + } + + /// With no entry naming that middleware, the body passes through as the + /// origin sent it, neither core nor the middleware touching it. + #[test] + fn a_body_whose_middleware_moves_the_address_itself_passes_through_with_no_entry() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + + let settings = settings_running_the_middleware_fixture(Vec::new()); + let body = "See https://origin.example.com/more\n"; + + let output = text_body_through_the_publisher(&settings, fixture::FRAMED_MEDIA_TYPE, body); + + assert_eq!( + output, body, + "should pass the body through as the origin sent it" + ); + } + + /// The same through the response path a streamed body takes, which + /// builds its processor apart from the buffered path's. + #[tokio::test] + async fn a_streamed_body_whose_middleware_moves_the_address_itself_is_handed_to_it_as_sent() { + use crate::integrations::registry_test_support::middleware_fixture as fixture; + use crate::platform::test_support::noop_services; + + let settings = settings_running_the_middleware_fixture(vec![PhaseEntry { + media_type: fixture::FRAMED_MEDIA_TYPE.to_owned(), + path: None, + middleware: vec![fixture::FRAMED.to_owned()], + }]); + let registry = IntegrationRegistry::new(&settings).expect("should build a registry"); + let orchestrator = AuctionOrchestrator::new( + crate::auction::test_support::legacy_auction_config(&settings), + ); + let services = noop_services(); + let mut params = OwnedProcessResponseParams { + public_prefix: crate::prefix::PublicPrefix::default(), + csp_nonce_observed: None, + prepared: Arc::new(Prepared::default()), + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), + content_encoding: String::new(), + origin_host: "origin.example.com".to_string(), + origin_url: "https://origin.example.com".to_string(), + request_host: "proxy.example.com".to_string(), + request_scheme: "https".to_string(), + request_path: "/a.txt".to_owned(), + content_type: fixture::FRAMED_MEDIA_TYPE.to_string(), + permissions_json: String::new(), + ad_slots_script: None, + ad_bids_state: AdBidsState::default(), + auction_observation: None, + auction_request: None, + dispatched_auction: None, + price_granularity: crate::price_bucket::PriceGranularity::default(), + request_state: IntegrationRequestState::default(), + }; + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"See https://origin.example.com/"), + bytes::Bytes::from_static(b"more\n"), + ])); + let mut output = Vec::new(); + + stream_publisher_body_async( + body, + &mut output, + &mut params, + &settings, + ®istry, + &orchestrator, + &services, + ) + .await + .expect("should stream a text body"); + + assert_eq!( + String::from_utf8(output).expect("text"), + format!( + "See https://origin.example.com/more\n{}", + fixture::FRAMED_LINE + ), + "should leave the origin's address to the middleware on the streamed path too" + ); + } + /// The same through the response path a streamed body takes, which /// builds its processor apart from the buffered path's. #[tokio::test] diff --git a/crates/trusted-server-modules/src/lib.rs b/crates/trusted-server-modules/src/lib.rs index 0b6da8f243..4daef38b4b 100644 --- a/crates/trusted-server-modules/src/lib.rs +++ b/crates/trusted-server-modules/src/lib.rs @@ -131,6 +131,12 @@ pub fn middleware() -> Vec { row(js_asset_proxy::MODULE, Fetch, js_asset_proxy::MODULE), row(testlight::builder().id(), Fetch, testlight::MODULE), row(nextjs::builder().id(), Fetch, nextjs::MODULE), + StockMiddleware { + integration: nextjs::builder().id(), + phase: Fetch, + media_type: nextjs::FLIGHT_MEDIA_TYPE, + name: nextjs::FLIGHT_MIDDLEWARE, + }, row(permutive::builder().id(), Fetch, permutive::MODULE), row(lockr::builder().id(), Fetch, lockr::MODULE), row(didomi::builder().id(), Fetch, didomi::MODULE), @@ -382,7 +388,11 @@ middleware = [\"robots_txt\"] .iter() .filter(|entry| entry.media_type == media_type) .collect(); - if media_type == HTML_MEDIA_TYPE { + if media_type == HTML_MEDIA_TYPE + || covering.iter().any(|entry| entry.path.is_none()) + { + // One entry covers every page, for HTML and for a media + // type a page's own responses come in. assert_eq!( covering.len(), 1, @@ -396,7 +406,7 @@ middleware = [\"robots_txt\"] // A file's middleware is placed on the file's own path, one // entry for each file. assert!( - !covering.is_empty() && covering.iter().all(|entry| entry.path.is_some()), + !covering.is_empty(), "should document a [[{phase}]] entry on a path for each {media_type} file" ); } diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 0ff62313a1..6ba7dde805 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -2164,24 +2164,27 @@ do. An entry carries none and switches nothing on. ### Entries for a media type other than HTML -An entry covers `text/html`, another `text/` type other than -`text/x-component`, `application/javascript` or `application/json`, written -as a bare `type/subtype` in lower case with no parameters. A response is -matched by the type and subtype of its `Content-Type`, without the -parameters, so `text/plain; charset=utf-8` takes the entry for -`text/plain`. Anything else, an image or a length prefixed stream among -them, passes through untouched and no entry can cover it. So does a -response that declares a charset other than UTF-8. +An entry covers `text/html`, another `text/` type, `application/javascript` +or `application/json`, written as a bare `type/subtype` in lower case with +no parameters. A response is matched by the type and subtype of its +`Content-Type`, without the parameters, so `text/plain; charset=utf-8` takes +the entry for `text/plain`. Anything else, an image among them, passes +through untouched and no entry can cover it. So does a response that +declares a charset other than UTF-8. A middleware says which media types it works on, HTML alone unless it says otherwise, and an entry naming one under a media type it does not handle is refused when the registry is built, with the entry and the middleware named. On a media type other than HTML a middleware changes the body as a stream, after core has moved the origin's addresses in it, in the order the entry -names them. Head markup and the element and text handlers are decisions -about HTML, and a middleware that makes one there is refused by name. The -template cache stores HTML alone, so the fetch entry's middleware and then -the serve entry's run on each response as it is fetched. +names them. A middleware may say that it moves the origin's addresses +itself, for a format whose rows count their bytes, and core then leaves the +addresses in a body of that media type to it whether or not an entry names +it, so with no entry such a body passes through as the origin sent it. Head +markup and the element and text handlers are decisions about HTML, and a +middleware that makes one there is refused by name. The template cache +stores HTML alone, so the fetch entry's middleware and then the serve +entry's run on each response as it is fetched. **A file the origin does not have.** When the origin answers `404 Not Found` for a path under the `path` of a `[[fetch]]` entry for a media type other @@ -2215,6 +2218,7 @@ generated configuration places it in. | `js_asset_proxy` | `[[fetch]]` | Points each configured script at its first-party path and removes one that is blocked. It is asked about an address as the middleware named before it left it, so name it ahead of one that moves the same script | | `testing.testlight` | `[[fetch]]` | Points a `src` or `href` that names `testlight.js` at the shim, when `rewrite_scripts` is set | | `framework.nextjs` | `[[fetch]]` | Moves the origin's address to the publisher's in the data Next.js writes into a page, being the `__NEXT_DATA__` script and the React Server Components payload scripts | +| `framework.nextjs.flight` | `[[fetch]]` | Moves the origin's address to the publisher's in a React Server Components Flight response and writes each length prefixed row's length again, on `text/x-component` alone. It moves the addresses itself, so core leaves a Flight response as sent where no entry names it | | `audience.permutive` | `[[fetch]]` | Points a `src` or `href` that is the Permutive SDK's address at `/integrations/permutive/sdk`, when `rewrite_sdk` is set | | `identity.lockr` | `[[fetch]]` | Points a `src` or `href` that is the lockr SDK's address at `/integrations/lockr/sdk`, when `rewrite_sdk` is set | | `cmp.didomi` | `[[fetch]]` | Writes `window.__tsjs_didomi` into the head, which hands the browser module the path Didomi is served under | diff --git a/docs/guide/integrations/nextjs.md b/docs/guide/integrations/nextjs.md index 2a89ef4d14..e5bbdcc689 100644 --- a/docs/guide/integrations/nextjs.md +++ b/docs/guide/integrations/nextjs.md @@ -35,15 +35,24 @@ max_combined_payload_bytes = 10485760 ``` What the integration changes in a page is the middleware `framework.nextjs`, which runs -on the pages a `[[fetch]]` entry names it for: +on the pages a `[[fetch]]` entry names it for. A navigation in the App Router +is answered with a React Server Components Flight response, `text/x-component`, +and what the integration changes in one is the middleware +`framework.nextjs.flight`, which runs on the responses a `[[fetch]]` entry for +that media type names it for: ```toml [[fetch]] media_type = "text/html" middleware = ["framework.nextjs"] + +[[fetch]] +media_type = "text/x-component" +middleware = ["framework.nextjs.flight"] ``` -See [Placing page changes](/guide/configuration#placing-page-changes). +Without the second entry a Flight response passes through as the origin sent +it. See [Placing page changes](/guide/configuration#placing-page-changes). ### Configuration Options @@ -107,6 +116,7 @@ The Next.js integration is implemented across multiple files: - [crates/framework/nextjs/src/lib.rs](https://github.com/IABTechLab/trusted-server/blob/main/crates/framework/nextjs/src/lib.rs) - Main integration - [crates/framework/nextjs/src/rsc.rs](https://github.com/IABTechLab/trusted-server/blob/main/crates/framework/nextjs/src/rsc.rs) - RSC parsing +- [crates/framework/nextjs/src/flight.rs](https://github.com/IABTechLab/trusted-server/blob/main/crates/framework/nextjs/src/flight.rs) - The Flight response middleware - [crates/framework/nextjs/src/script_rewriter.rs](https://github.com/IABTechLab/trusted-server/blob/main/crates/framework/nextjs/src/script_rewriter.rs) - Script rewriting ### Key Components diff --git a/docs/guide/rsc-hydration.md b/docs/guide/rsc-hydration.md index 07a8dd444e..f0a34fb82a 100644 --- a/docs/guide/rsc-hydration.md +++ b/docs/guide/rsc-hydration.md @@ -25,8 +25,13 @@ Trusted Server uses a two-phase pipeline: ## Where This Runs -- HTML responses: streaming rewrite with optional post-processing. -- RSC Flight responses (`text/x-component`): direct Flight parsing and rewrite. +- HTML responses: streaming rewrite with optional post-processing, by the + middleware `framework.nextjs` where a `[[fetch]]` entry for `text/html` + names it. +- RSC Flight responses (`text/x-component`): direct Flight parsing and + rewrite, by the middleware `framework.nextjs.flight` where a `[[fetch]]` + entry for `text/x-component` names it. Without the entry a Flight response + passes through as the origin sent it. ## Internals @@ -178,7 +183,7 @@ The rewrite logic handles multiple URL formats in RSC content: The current implementation lives in: - `crates/framework/nextjs/src/rsc.rs` -- `crates/trusted-server-core/src/rsc_flight.rs` +- `crates/framework/nextjs/src/flight.rs` ## Related Docs diff --git a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md new file mode 100644 index 0000000000..5f35e8131d --- /dev/null +++ b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md @@ -0,0 +1,57 @@ +# Design Spec: The Flight rewriter is a middleware of the Next.js module + +**Status:** Proposed, 2026-10-14. Implemented on a working branch of the fork, with no pull request raised yet. +**Author:** 51Degrees (contributed), for Tech Lab review +**Related specs:** `2026-09-07-850-parser-aware-body-hold-nextjs-streaming-design.md`, `2026-10-14-source-modules-design.md` +**Related PRs:** none yet +**Issues:** IABTechLab/trusted-server#1340 +**Last updated:** 2026-10-14 + +> **Why this spec exists.** Core rewrites the addresses inside a React Server Components Flight stream by its media type, in a module of its own, and the publisher path names that media type to call it. Every other piece of a framework's handling lives in the framework's crate. + +## 1. Problem + +The Next.js App Router answers a client side navigation with a Flight stream, `text/x-component`, the row format its `__next_f` scripts carry inside a page. A `T` row declares the byte length of its content, so moving an address inside one changes the length and the header has to be written again. Core's plain replacement of the origin's address, which runs on every other text body, would leave the old length and break the stream. + +That is why core carried `rsc_flight`, a rewriter that reads the rows and writes each `T` row's length again, and why the publisher path checked for `text/x-component` to run it in place of the plain replacement. The entries for a media type other than HTML could not cover `text/x-component`, because the plain replacement would have run ahead of any middleware named there. + +The Next.js crate carries the rest of the framework's handling. This piece stayed in core because it ran before a middleware could run on a media type other than HTML. + +## 2. Design + +**The rewriter is a middleware of the Next.js module.** `framework.nextjs.flight` is registered beside `framework.nextjs` whenever `[framework] module = "nextjs"` selects the module. It handles `text/x-component` alone, runs in the fetch phase, and its action carries the stream processor that reads the rows and rewrites them, so an entry naming it for HTML is refused when the registry is built, as any middleware named under a media type it does not handle is. + +```toml +[framework] +module = "nextjs" + +[[fetch]] +media_type = "text/html" +middleware = ["framework.nextjs"] + +[[fetch]] +media_type = "text/x-component" +middleware = ["framework.nextjs.flight"] +``` + +**Core names no framework.** The publisher path runs the entries the configuration lists on a text response and knows nothing of Flight. `text/x-component` is a media type an entry may cover, as every other `text/` type is. + +**A middleware may move the origin's address itself.** `Middleware::moves_origin_address` says that a middleware moves the origin's address to the reader's host itself in a body of the media types it handles, for a format whose rows count their bytes. Where a registered middleware says so for a media type, core leaves the addresses in a body of that type alone, whether or not an entry names the middleware. So with the module selected and no entry, a Flight response passes through as the origin sent it, and with the entry it is rewritten as it was before, row by row with each `T` row's length written again. The signal comes from the registered middleware rather than from the entry, because an entry that is missing must leave the stream whole rather than hand it to the plain replacement. + +**What a deployment does.** A deployment that fronts a Next.js site selects the module and names the middleware in a `[[fetch]]` entry for `text/x-component`, as it names the rest of the framework's middleware for `text/html`. `trusted-server.example.toml` shows the entry, the module's recorded page settings carry it, and the stock middleware list in `trusted-server-modules` lists it with its media type, so the configuration guide's table and the template are checked against it. + +**What the middleware is told.** The origin host, the reader's host and the reader's scheme from its context, which is what the module's HTML middleware works from too. The origin's `https://` and `http://` addresses go to the reader's scheme and host, a protocol relative address to the reader's host, and a bare host at hostname boundaries to the reader's host, as core's plain replacement does them. + +## 3. What is not in this change + +- A deployment that serves Flight responses without selecting the Next.js module. Core treats `text/x-component` as it treats any other text body, so such a deployment selects the module. +- Reading JSON escapes inside a Flight row. A navigation response carries plain addresses, and the escaped form inside a page's `__next_f` scripts is handled by the HTML middleware. +- Running the middleware in the serve phase. A Flight response is not stored, and the fetch phase runs on each one as it is fetched. + +## 4. Tests + +- The rewriter moves an address in a newline row, moves an `http://` address to the reader's scheme, writes a `T` row's length again, handles a `T` row split across chunks, and leaves an unrelated hostname alone. +- The middleware handles `text/x-component` alone, runs in the fetch phase, says it moves the origin's address itself, and its plan carries one stream processor that rewrites a Flight body. +- The module registers the middleware, core leaves `text/x-component` to it and moves the addresses in every other body, a Flight response takes the entry naming it and runs nothing where no entry covers it, and an entry for HTML naming it is refused by name. +- In core, a body whose registered middleware moves the address itself is handed to that middleware as the origin sent it, on the buffered path and the streamed path, and passes through untouched where no entry names the middleware, while a `text/plain` body still has its addresses moved before the entry's middleware runs. +- `text/x-component` is accepted as an entry's media type, and the stock middleware list, the configuration guide's table and the template's entries agree on the new middleware. diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 54e9a6996d..d55c1f4b4b 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -397,6 +397,15 @@ pull_sync_concurrency = 3 # "ad-tag.google", # ] +# A navigation in the Next.js App Router is answered with a React Server +# Components Flight response, text/x-component, and the framework module's +# second middleware moves the origin's address in one where this entry names +# it. Without the entry a Flight response passes through as the origin sent +# it. Remove the entry with the module. +# [[fetch]] +# media_type = "text/x-component" +# middleware = ["framework.nextjs.flight"] + # [[serve]] # media_type = "text/html" # middleware = ["tag.google-tag", "bot-protection.datadome.tag", "ad-tag.google.diagnostics", "service_version"] From 303981e494a3ae84c18a653e2a372df0cd527cb9 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 17:20:12 +0100 Subject: [PATCH 2/6] The Flight middleware design states how it differs from core's replacement and its rollout Review round 1 found that the design called the middleware's rewriting the same as before and the same as core's plain replacement, when the middleware matches the origin host alone. The design now states the four differences and why, the alternatives to the new method, the adapters, the refusals, consent, failure and the rollout order. It also asks the reviewer what a deployment that does not select the module should get, lists the cases no test covers and records the review round. The template comment counts the three entries it now shows. --- ...2026-10-14-rsc-flight-middleware-design.md | 66 ++++++++++++++++--- trusted-server.example.toml | 2 +- 2 files changed, 58 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md index 5f35e8131d..b64275109b 100644 --- a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md +++ b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md @@ -5,9 +5,9 @@ **Related specs:** `2026-09-07-850-parser-aware-body-hold-nextjs-streaming-design.md`, `2026-10-14-source-modules-design.md` **Related PRs:** none yet **Issues:** IABTechLab/trusted-server#1340 -**Last updated:** 2026-10-14 +**Last updated:** 2026-10-15 -> **Why this spec exists.** Core rewrites the addresses inside a React Server Components Flight stream by its media type, in a module of its own, and the publisher path names that media type to call it. Every other piece of a framework's handling lives in the framework's crate. +> **Why this spec exists.** Core rewrote the addresses inside a React Server Components Flight stream by its media type, in a module of its own, and the publisher path named that media type to call it. This change moves the rewriter into the Next.js module as the middleware `framework.nextjs.flight`, so the publisher path no longer names the media type. A deployment that fronts a Next.js site selects the module and names the middleware in a `[[fetch]]` entry, and section 2 says what happens to a deployment that does not. ## 1. Problem @@ -19,7 +19,7 @@ The Next.js crate carries the rest of the framework's handling. This piece staye ## 2. Design -**The rewriter is a middleware of the Next.js module.** `framework.nextjs.flight` is registered beside `framework.nextjs` whenever `[framework] module = "nextjs"` selects the module. It handles `text/x-component` alone, runs in the fetch phase, and its action carries the stream processor that reads the rows and rewrites them, so an entry naming it for HTML is refused when the registry is built, as any middleware named under a media type it does not handle is. +**The rewriter is a middleware of the Next.js module.** `framework.nextjs.flight` is registered beside `framework.nextjs` whenever `[framework] module = "nextjs"` selects the module. It handles `text/x-component` alone, so an entry that names it under any other media type, HTML included, is refused when the registry is built, as any middleware named under a media type it does not handle is. It runs in the fetch phase, and its action carries one stream processor that reads the rows and rewrites them. ```toml [framework] @@ -34,17 +34,44 @@ media_type = "text/x-component" middleware = ["framework.nextjs.flight"] ``` -**Core names no framework.** The publisher path runs the entries the configuration lists on a text response and knows nothing of Flight. `text/x-component` is a media type an entry may cover, as every other `text/` type is. +**The publisher path names no framework.** The publisher path runs the entries the configuration lists on a text response and knows nothing of Flight. `text/x-component` is a media type an entry may cover, as every other `text/` type is. -**A middleware may move the origin's address itself.** `Middleware::moves_origin_address` says that a middleware moves the origin's address to the reader's host itself in a body of the media types it handles, for a format whose rows count their bytes. Where a registered middleware says so for a media type, core leaves the addresses in a body of that type alone, whether or not an entry names the middleware. So with the module selected and no entry, a Flight response passes through as the origin sent it, and with the entry it is rewritten as it was before, row by row with each `T` row's length written again. The signal comes from the registered middleware rather than from the entry, because an entry that is missing must leave the stream whole rather than hand it to the plain replacement. +**A middleware may move the origin's address itself.** `Middleware::moves_origin_address` is a new method of the public `Middleware` trait that answers `false` unless a middleware overrides it. A middleware that answers `true` says that it moves the origin's address to the reader's host itself, in a body of the media types it handles other than HTML, for a format whose rows count their bytes. `IntegrationRegistry::middleware_moves_origin_address` is true for a media type when any registered middleware both says so and handles that type, and a module registers its middleware only when a section selects the module. Where the answer is true for a media type, core leaves the addresses in a body of that type alone, whether or not an entry names the middleware, for every response of that type whatever path the request was for and whatever `path` an entry carries. So with the module selected and no entry, a Flight response passes through as the origin sent it, and with the entry the middleware rewrites the response row by row, writing each `T` row's length again. -**What a deployment does.** A deployment that fronts a Next.js site selects the module and names the middleware in a `[[fetch]]` entry for `text/x-component`, as it names the rest of the framework's middleware for `text/html`. `trusted-server.example.toml` shows the entry, the module's recorded page settings carry it, and the stock middleware list in `trusted-server-modules` lists it with its media type, so the configuration guide's table and the template are checked against it. +**Why the middleware says so, and what else was possible.** The signal comes from the registered middleware rather than from the entry, because an entry that is missing must leave the stream whole rather than hand it to the plain replacement. Two other routes were not taken. Leaving the rewriter in core and naming its media type in the publisher path keeps in core the one piece of a framework's handling that this change moves out. Allowing an entry for `text/x-component` and adding no signal leaves core's plain replacement to run ahead of the middleware, which is the fault in section 1. -**What the middleware is told.** The origin host, the reader's host and the reader's scheme from its context, which is what the module's HTML middleware works from too. The origin's `https://` and `http://` addresses go to the reader's scheme and host, a protocol relative address to the reader's host, and a bare host at hostname boundaries to the reader's host, as core's plain replacement does them. +**What a deployment does.** A deployment that fronts a Next.js site selects the module and names the middleware in a `[[fetch]]` entry for `text/x-component`, as it names the rest of the framework's middleware for `text/html`. `trusted-server.example.toml` shows the entry, the module's recorded page settings carry it, and the stock middleware list in `trusted-server-modules` lists it with its media type, so the configuration guide's table and the template are checked against it. `ts audit` writes entries for `text/html` alone, because `trusted_server_modules::middleware_for` leaves out every other media type. `ts audit` also does not detect Next.js and so does not select the module, which means a configuration it writes carries no Flight entry, and a deployment adds the entry by hand. + +**A `path` on the Flight entry.** The registry accepts a `path` on this entry, as it does on any entry, and the entry then covers only the Flight responses fetched for a request path under it. Core still leaves the addresses of every Flight response to the middleware, so a Flight response for a path no entry covers passes through as the origin sent it. An entry for a media type other than HTML that carries a `path` and names a middleware is also the entry that makes a file the origin does not have. With a `path` on the Flight entry, an origin `404` to a `GET` or `HEAD` request for any path under it, an HTML page's included, is therefore answered `200` with an empty Flight document, and the middleware runs on that document. The template's entry carries no `path`, and no test covers an entry with one. + +**Configuration.** The change adds no key and no default. It adds one middleware name, `framework.nextjs.flight`, and lets an entry cover `text/x-component`. The registry refuses an entry that names a middleware no running module supplies, one that names a middleware in a phase it does not run in, and one that names a middleware under a media type it does not handle, each by naming the entry and the middleware. The Flight middleware is therefore refused in a `[[serve]]` entry, under any media type but `text/x-component`, and where no section selects the module. + +**What the middleware is told.** The origin host, the reader's host and the reader's scheme from its context, which is what the module's HTML middleware works from too. The origin host is the `Host` the source sends, being the source's host and port unless the source sets another `Host`. A page names itself by that `Host`, so core tells every middleware that name as the origin host, and a middleware is told no source address. + +**What the middleware moves.** From the origin host, the middleware moves an `https://` or `http://` address to the reader's scheme and host, a protocol relative address to the reader's host, and a bare host at hostname boundaries to the reader's host. What the middleware moves is not exactly what core's plain replacement moves, and not exactly what `rsc_flight` moved, the rewriter in core that this change moves out. Core's replacement and `rsc_flight` build the `https://` and `http://` forms from the source's address as written, where the middleware builds them from the host alone, and `rsc_flight` already respected hostname boundaries as the middleware does. The middleware differs from core's replacement in four ways and from `rsc_flight` in the first three. + +1. **A folder in the source's address.** For a source at `https://origin.example.com/shop`, core's replacement and `rsc_flight` turn `https://origin.example.com/shop/item` into `https://proxy.example.com/item`. The middleware turns the same text into `https://proxy.example.com/shop/item`, so the folder stays in a rewritten address, as it does in the links of a page. +2. **A `Host` that differs from the address.** Where the source sends another `Host` than its address names, core's replacement and `rsc_flight` look for the address as written and for the `Host`, and the middleware looks for the `Host` alone. An address in a Flight response that names only the source's own address is therefore no longer moved. +3. **The scheme of the address.** Core's replacement and `rsc_flight` add the `http://` form only where the source's address is written `https://`. The middleware always moves both forms to the reader's scheme, so for a source written `http://`, an `https://` address in the body takes the reader's scheme, where core's replacement left the `https` the body had. +4. **A bare host inside a longer name.** Core's replacement treats the bare host as plain text, so `notorigin.example.com` becomes `notproxy.example.com`. The middleware moves a bare host only at hostname boundaries, as `rsc_flight` did, and leaves `notorigin.example.com`, `cdn.origin.example.com` and `origin.example.com.uk` alone. + +The middleware builds from the host because it works from what its context carries, as the module's HTML middleware does, and its context carries the origin host and no address. Core's own rewriting of the links in a page also builds from the host. + +**Adapters.** This change touches no adapter. The Axum, Cloudflare, Fastly and Spin adapters each build their registry from the stock builders of `trusted_server_modules`, which include the Next.js module, and each reaches the body path through core's publisher functions, so the middleware, and core leaving a Flight body to it, apply on all four. + +**Consent and privacy.** The middleware reads no consent state and no identifier and sets no header, so the change has no effect on either. Where no entry names the middleware, the origin's own addresses reach the reader inside the Flight response, as the origin sent them. + +**When it fails.** The row reader is the code core had, moved without change. Content that is not valid UTF-8 is written as it came, and so is a row that the end of the stream cuts short. A byte in a row's length that is not a hexadecimal digit counts as zero. A row is held in memory until its newline or its whole length has arrived, and no limit applies to the size held, so a row that never ends is held to the end of the response. A body that declares a charset other than UTF-8, on a media type whose addresses a middleware moves itself, passes through whole, because core skips its replacement for the type and no middleware runs on a charset other than UTF-8. Before this change core rewrote a Flight response whatever charset the response declared. + +**Rollout.** The `CHANGELOG.md` entry for this change is marked breaking. It says that a deployment that fronts a Next.js site selects the module and adds the entry, and that `trusted_server_core::rsc_flight` is removed for crates outside core. The build goes first and the entry second. A build from before this change refuses a configuration that carries the entry, and so does `ts config validate` from before this change, because neither accepts `text/x-component` as the media type of an entry. A build with this change, with the module selected and no entry, starts and logs a warning that no entry names the middleware, and it sends a Flight response as the origin sent it, so the origin's address reaches the reader in Flight responses until the entry is added. Going back to a build from before this change needs the entry removed first, because that build refuses a configuration that carries the entry. + +**Open question for the reviewer.** A deployment that fronts a Next.js site and does not select `[framework] module = "nextjs"` now gets core's plain replacement on a Flight response, because no registered middleware says it moves the addresses of `text/x-component`. Before this change core rewrote Flight for every deployment. The plain replacement changes the text inside a `T` row and leaves the row's length as it was, which is the fault in section 1, so the stream breaks wherever the replacement changes a row's length. Nothing in the code warns about this case at startup or on a request, and the `CHANGELOG.md` entry says only that a deployment that fronts a Next.js site selects the module and adds the entry. The question for the reviewer is whether core should keep a default rewrite of Flight for such a deployment or whether the breaking note is enough. This spec makes no decision on it. ## 3. What is not in this change -- A deployment that serves Flight responses without selecting the Next.js module. Core treats `text/x-component` as it treats any other text body, so such a deployment selects the module. +- A default for a deployment that serves Flight responses without selecting the Next.js module. Core treats `text/x-component` as it treats any other text body for such a deployment, so its Flight responses meet the plain replacement, as the open question in section 2 says. +- An adapter change. The four adapters take the middleware through the stock builders and need none. +- Writing the Flight entry from `ts audit`, which writes `text/html` entries alone and does not detect Next.js. - Reading JSON escapes inside a Flight row. A navigation response carries plain addresses, and the escaped form inside a page's `__next_f` scripts is handled by the HTML middleware. - Running the middleware in the serve phase. A Flight response is not stored, and the fetch phase runs on each one as it is fetched. @@ -53,5 +80,26 @@ middleware = ["framework.nextjs.flight"] - The rewriter moves an address in a newline row, moves an `http://` address to the reader's scheme, writes a `T` row's length again, handles a `T` row split across chunks, and leaves an unrelated hostname alone. - The middleware handles `text/x-component` alone, runs in the fetch phase, says it moves the origin's address itself, and its plan carries one stream processor that rewrites a Flight body. - The module registers the middleware, core leaves `text/x-component` to it and moves the addresses in every other body, a Flight response takes the entry naming it and runs nothing where no entry covers it, and an entry for HTML naming it is refused by name. -- In core, a body whose registered middleware moves the address itself is handed to that middleware as the origin sent it, on the buffered path and the streamed path, and passes through untouched where no entry names the middleware, while a `text/plain` body still has its addresses moved before the entry's middleware runs. +- In core, a body whose registered middleware moves the address itself is handed to that middleware as the origin sent it, on the buffered path and the streamed path. The same body passes through untouched where no entry names the middleware, tested on the buffered path alone. A `text/plain` body still has its addresses moved before the entry's middleware runs. - `text/x-component` is accepted as an entry's media type, and the stock middleware list, the configuration guide's table and the template's entries agree on the new middleware. + +Seven cases have no test. + +1. The real middleware through the publisher path. The tests in core use a middleware of their own, on a media type of its own, and the tests in the Next.js crate stop at the registry and the middleware's own plan, so no test sends a `text/x-component` body through the publisher path with `framework.nextjs.flight` registered. +2. The streamed path where no entry names a middleware that moves the address itself. +3. A body that declares a charset other than UTF-8, on a media type whose addresses a middleware moves itself. The existing charset test uses `text/plain` and a middleware that does not move addresses. +4. A source address with a folder, and a source that sends another `Host`, through the middleware. The rewriter's tests give it a host and nothing else. +5. A Flight entry with a `path`. +6. Any adapter. No adapter route test and no parity test sends a `text/x-component` response, and the shared Next.js fixture they use (`nextjs_auction` in `crates/trusted-server-core/src/test_support.rs`) selects the module with an entry for `text/html` alone. +7. The integration scenario `NextJsRscFlight` (`crates/trusted-server-integration-tests/tests/frameworks/scenarios.rs`), which sends a Flight request to the Next.js test origin. Before this change core rewrote every Flight response. The integration test configuration selects no `[framework]` module and has no `[[fetch]]` entry, so a Flight response in that scenario now meets core's plain replacement and not the middleware. The scenario checks only that the content type is not HTML and that the body holds no injected script, so it would pass with a broken `T` row. + +## Review log + +1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/40-rsc-flight` at `b2124875a`. Issues found, one of them blocking. Section 2 said the middleware rewrites a Flight response as it was rewritten before and as core's plain replacement does, when the middleware builds its addresses from the origin host and so differs from both for a folder in the source's address, for another `Host` and for the `http://` form, and from core's replacement for a bare host inside a longer name. This revision states each difference and why the middleware builds from the host, and it adds the adapters, the configuration refusals, consent and privacy, failure, the rollout order with an open question for deployments that do not select the module, the cases no test covers and the advice the round gave. + +## Revision record + +| Date | Change | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-10-14 | First draft. | +| 2026-10-15 | Fixes from review round 1. The differences from core's plain replacement and why, the alternatives to the new method, adapters, configuration, consent, failure, rollout with its open question, the cases no test covers and the review log. | diff --git a/trusted-server.example.toml b/trusted-server.example.toml index d55c1f4b4b..19ab2b9bd5 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -374,7 +374,7 @@ pull_sync_concurrency = 3 # A module changes a page through a middleware, and a middleware runs only on # the pages where a [[fetch]] or [[serve]] entry names it. A page takes the # first entry that covers it and runs that entry's middleware in the order -# written. The two entries below name every middleware the modules further +# written. The three entries below name every middleware the modules further # down supply, in the order `ts audit` writes them, and core's own # `service_version` last. Keep the names of the modules that are selected and # remove the rest, because an entry that names the middleware of a module no From 01b64f242b72b10c2a7c2fc95861d19be0c5790b Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Thu, 15 Oct 2026 17:46:51 +0100 Subject: [PATCH 3/6] The RSC flight design takes its round 2 corrections Review round 2 approved the design with no blocking issues and found four small errors. The hostname boundary rule holds only for a bare host, because a longer name that begins with the origin host after https://, http:// or // is changed by core's replacement, the old rewriter and the middleware alike. The path paragraph now says that settings validation checks a path and that the entry with the longest path makes an empty document where several cover a request. The template comment limits the order that ts audit writes to the HTML entries, and the review log and revision record carry the round. --- .../specs/2026-10-14-rsc-flight-middleware-design.md | 10 ++++++---- trusted-server.example.toml | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md index b64275109b..db922e927b 100644 --- a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md +++ b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md @@ -42,18 +42,18 @@ middleware = ["framework.nextjs.flight"] **What a deployment does.** A deployment that fronts a Next.js site selects the module and names the middleware in a `[[fetch]]` entry for `text/x-component`, as it names the rest of the framework's middleware for `text/html`. `trusted-server.example.toml` shows the entry, the module's recorded page settings carry it, and the stock middleware list in `trusted-server-modules` lists it with its media type, so the configuration guide's table and the template are checked against it. `ts audit` writes entries for `text/html` alone, because `trusted_server_modules::middleware_for` leaves out every other media type. `ts audit` also does not detect Next.js and so does not select the module, which means a configuration it writes carries no Flight entry, and a deployment adds the entry by hand. -**A `path` on the Flight entry.** The registry accepts a `path` on this entry, as it does on any entry, and the entry then covers only the Flight responses fetched for a request path under it. Core still leaves the addresses of every Flight response to the middleware, so a Flight response for a path no entry covers passes through as the origin sent it. An entry for a media type other than HTML that carries a `path` and names a middleware is also the entry that makes a file the origin does not have. With a `path` on the Flight entry, an origin `404` to a `GET` or `HEAD` request for any path under it, an HTML page's included, is therefore answered `200` with an empty Flight document, and the middleware runs on that document. The template's entry carries no `path`, and no test covers an entry with one. +**A `path` on the Flight entry.** Settings validation accepts a `path` on this entry, as it does on any entry, and the entry then covers only the Flight responses fetched for a request path under it. Core still leaves the addresses of every Flight response to the middleware, so a Flight response for a path no entry covers passes through as the origin sent it. A `[[fetch]]` entry for a media type other than HTML that carries a `path` and names a middleware can also make a file the origin does not have. Where more than one such entry covers the request, the one with the longest `path` makes it, and the first written wins between two as long. With a `path` on the Flight entry, an origin `404` to a `GET` or `HEAD` request for a path under it, an HTML page's included, is therefore answered `200` with an empty Flight document, and the middleware runs on that document, unless another such entry wins under that rule. That entry makes an empty document of its own media type instead. The template's entry carries no `path`, and no test covers an entry with one. **Configuration.** The change adds no key and no default. It adds one middleware name, `framework.nextjs.flight`, and lets an entry cover `text/x-component`. The registry refuses an entry that names a middleware no running module supplies, one that names a middleware in a phase it does not run in, and one that names a middleware under a media type it does not handle, each by naming the entry and the middleware. The Flight middleware is therefore refused in a `[[serve]]` entry, under any media type but `text/x-component`, and where no section selects the module. **What the middleware is told.** The origin host, the reader's host and the reader's scheme from its context, which is what the module's HTML middleware works from too. The origin host is the `Host` the source sends, being the source's host and port unless the source sets another `Host`. A page names itself by that `Host`, so core tells every middleware that name as the origin host, and a middleware is told no source address. -**What the middleware moves.** From the origin host, the middleware moves an `https://` or `http://` address to the reader's scheme and host, a protocol relative address to the reader's host, and a bare host at hostname boundaries to the reader's host. What the middleware moves is not exactly what core's plain replacement moves, and not exactly what `rsc_flight` moved, the rewriter in core that this change moves out. Core's replacement and `rsc_flight` build the `https://` and `http://` forms from the source's address as written, where the middleware builds them from the host alone, and `rsc_flight` already respected hostname boundaries as the middleware does. The middleware differs from core's replacement in four ways and from `rsc_flight` in the first three. +**What the middleware moves.** From the origin host, the middleware moves an `https://` or `http://` address to the reader's scheme and host, a protocol relative address to the reader's host, and a bare host at hostname boundaries to the reader's host. What the middleware moves is not exactly what core's plain replacement moves, and not exactly what `rsc_flight` moved, the rewriter in core that this change moves out. Core's replacement and `rsc_flight` build the `https://` and `http://` forms from the source's address as written, where the middleware builds them from the host alone, and `rsc_flight` already respected hostname boundaries for a bare host, as the middleware does. The middleware differs from core's replacement in four ways and from `rsc_flight` in the first three. 1. **A folder in the source's address.** For a source at `https://origin.example.com/shop`, core's replacement and `rsc_flight` turn `https://origin.example.com/shop/item` into `https://proxy.example.com/item`. The middleware turns the same text into `https://proxy.example.com/shop/item`, so the folder stays in a rewritten address, as it does in the links of a page. 2. **A `Host` that differs from the address.** Where the source sends another `Host` than its address names, core's replacement and `rsc_flight` look for the address as written and for the `Host`, and the middleware looks for the `Host` alone. An address in a Flight response that names only the source's own address is therefore no longer moved. 3. **The scheme of the address.** Core's replacement and `rsc_flight` add the `http://` form only where the source's address is written `https://`. The middleware always moves both forms to the reader's scheme, so for a source written `http://`, an `https://` address in the body takes the reader's scheme, where core's replacement left the `https` the body had. -4. **A bare host inside a longer name.** Core's replacement treats the bare host as plain text, so `notorigin.example.com` becomes `notproxy.example.com`. The middleware moves a bare host only at hostname boundaries, as `rsc_flight` did, and leaves `notorigin.example.com`, `cdn.origin.example.com` and `origin.example.com.uk` alone. +4. **A bare host inside a longer name.** Core's replacement treats the bare host as plain text, so `notorigin.example.com` becomes `notproxy.example.com`. The middleware moves a bare host only at hostname boundaries, as `rsc_flight` did, and leaves a bare `notorigin.example.com`, `cdn.origin.example.com` and `origin.example.com.uk` alone. Core's replacement, `rsc_flight` and the middleware all match the `https://`, `http://` and `//` forms as plain text, so a longer name that begins with the origin host after one of them, such as `https://origin.example.com.uk`, becomes `https://proxy.example.com.uk` in each. The middleware builds from the host because it works from what its context carries, as the module's HTML middleware does, and its context carries the origin host and no address. Core's own rewriting of the links in a page also builds from the host. @@ -77,7 +77,7 @@ The middleware builds from the host because it works from what its context carri ## 4. Tests -- The rewriter moves an address in a newline row, moves an `http://` address to the reader's scheme, writes a `T` row's length again, handles a `T` row split across chunks, and leaves an unrelated hostname alone. +- The rewriter moves an address in a newline row, moves an `http://` address to the reader's scheme, writes a `T` row's length again, handles a `T` row split across chunks, and leaves an unrelated bare hostname alone. - The middleware handles `text/x-component` alone, runs in the fetch phase, says it moves the origin's address itself, and its plan carries one stream processor that rewrites a Flight body. - The module registers the middleware, core leaves `text/x-component` to it and moves the addresses in every other body, a Flight response takes the entry naming it and runs nothing where no entry covers it, and an entry for HTML naming it is refused by name. - In core, a body whose registered middleware moves the address itself is handed to that middleware as the origin sent it, on the buffered path and the streamed path. The same body passes through untouched where no entry names the middleware, tested on the buffered path alone. A `text/plain` body still has its addresses moved before the entry's middleware runs. @@ -96,6 +96,7 @@ Seven cases have no test. ## Review log 1. **Round 1, 2026-10-15.** A review in a fresh session, against `split/40-rsc-flight` at `b2124875a`. Issues found, one of them blocking. Section 2 said the middleware rewrites a Flight response as it was rewritten before and as core's plain replacement does, when the middleware builds its addresses from the origin host and so differs from both for a folder in the source's address, for another `Host` and for the `http://` form, and from core's replacement for a bare host inside a longer name. This revision states each difference and why the middleware builds from the host, and it adds the adapters, the configuration refusals, consent and privacy, failure, the rollout order with an open question for deployments that do not select the module, the cases no test covers and the advice the round gave. +2. **Round 2, 2026-10-15.** A lighter review in a fresh session, against this page at `303981e49`, of the round 1 fixes and the new text. Approved, with no blocking issues. This revision corrects the hostname boundary difference, which holds only for a bare host, the `path` paragraph, which now names the validation that checks a `path` and the rule that picks the entry making an empty document, and the template comment, whose order now covers the HTML entries alone. ## Revision record @@ -103,3 +104,4 @@ Seven cases have no test. | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-10-14 | First draft. | | 2026-10-15 | Fixes from review round 1. The differences from core's plain replacement and why, the alternatives to the new method, adapters, configuration, consent, failure, rollout with its open question, the cases no test covers and the review log. | +| 2026-10-15 | Fixes from review round 2. The hostname boundary difference for a bare host alone, the `path` paragraph, the template comment and the review log. | diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 19ab2b9bd5..9deffacb11 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -375,7 +375,7 @@ pull_sync_concurrency = 3 # the pages where a [[fetch]] or [[serve]] entry names it. A page takes the # first entry that covers it and runs that entry's middleware in the order # written. The three entries below name every middleware the modules further -# down supply, in the order `ts audit` writes them, and core's own +# down supply, in the order `ts audit` writes them for HTML, and core's own # `service_version` last. Keep the names of the modules that are selected and # remove the rest, because an entry that names the middleware of a module no # section selects refuses startup. See "Placing page changes" in the From 6fa5ad9b5d3088af652d0cdf00fa0e26a1e8fac1 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 18:35:28 +0100 Subject: [PATCH 4/6] Move a Flight row's addresses in one pass, never reading a replacement again The Flight middleware moved the origin's address in a row with three replacements in a row, for the `https://`, `http://` and protocol-relative spellings, and then looked for a bare host at hostname boundaries. Each read the previous one's output. When the request host begins with the origin host, a later spelling matched inside an earlier replacement and the rest of the request host was written again. With `publisher.example` served as `publisher.example.localhost` over `http`, `https://publisher.example/news` came out as `http://publisher.example.localhost.localhost.localhost/news`, and served as `publisher.example:8443` over `https` it came out as `https://publisher.example:8443:8443/news`. `rewrite_utf8_bytes` now finds each occurrence of the origin host once, from the start. One that follows `https://`, `http://` or `//` is moved together with that spelling, and the text between those addresses is given to `rewrite_bare_host_at_boundaries` for a bare host, so nothing written is read again. The three origin spellings the rewriter kept are removed, and an empty origin host leaves the row as it came. The tests check that a request host which begins with the origin host, as a `.localhost` name or with a port, is written once in a `T` row and a newline row over both schemes, that a hyphenated host and a `www.` host over an apex origin are written as before with a longer name left alone, and the rewrite on its own, with an empty origin host. --- crates/framework/nextjs/src/flight.rs | 148 ++++++++++++++++++++++---- 1 file changed, 128 insertions(+), 20 deletions(-) diff --git a/crates/framework/nextjs/src/flight.rs b/crates/framework/nextjs/src/flight.rs index 82e4fa60da..a91caae108 100644 --- a/crates/framework/nextjs/src/flight.rs +++ b/crates/framework/nextjs/src/flight.rs @@ -72,17 +72,14 @@ enum RowState { /// Rewrites the addresses inside a Flight stream row by row, writing a `T` /// row's length again after its content changes. /// -/// The replacement is the plain one core makes in any other text: the -/// origin's `https://` and `http://` addresses to the publisher's, then -/// protocol relative ones, then the bare host at hostname boundaries. It does +/// The origin's `https://` and `http://` addresses move to the publisher's, +/// and a protocol relative one and a bare host at hostname boundaries to the +/// publisher's host, in one pass that never reads what it wrote. It does /// not read JSON escapes, so `\/\/origin.example.com` is left alone, which a /// navigation response does not carry. The escaped form inside a page's /// `__next_f` scripts is handled by the HTML middleware. pub(crate) struct RscFlightUrlRewriter { - origin_url: String, - origin_http_url: String, origin_host: String, - origin_protocol_relative: String, request_url: String, request_host: String, request_protocol_relative: String, @@ -100,10 +97,7 @@ impl RscFlightUrlRewriter { #[must_use] pub(crate) fn new(origin_host: &str, request_host: &str, request_scheme: &str) -> Self { Self { - origin_url: format!("https://{origin_host}"), - origin_http_url: format!("http://{origin_host}"), origin_host: origin_host.to_owned(), - origin_protocol_relative: format!("//{origin_host}"), request_url: format!("{request_scheme}://{request_host}"), request_host: request_host.to_owned(), request_protocol_relative: format!("//{request_host}"), @@ -127,26 +121,55 @@ impl RscFlightUrlRewriter { self.raw_header.clear(); } + /// The row's text with each of the origin's addresses moved once, in one + /// pass from the start, so nothing written is read again. + /// + /// An occurrence of the origin host after `https://`, `http://` or `//` + /// is moved with what comes before it, and the text between those + /// addresses is searched for a bare host at hostname boundaries. fn rewrite_utf8_bytes(&self, bytes: &[u8]) -> Vec { let Ok(text) = std::str::from_utf8(bytes) else { return bytes.to_vec(); }; - if !text.contains(&self.origin_host) { + if self.origin_host.is_empty() || !text.contains(&self.origin_host) { return bytes.to_vec(); } - let mut rewritten = text.replace(&self.origin_url, &self.request_url); - rewritten = rewritten.replace(&self.origin_http_url, &self.request_url); - rewritten = rewritten.replace( - &self.origin_protocol_relative, - &self.request_protocol_relative, - ); - rewritten = - rewrite_bare_host_at_boundaries(&rewritten, &self.origin_host, &self.request_host) - .unwrap_or(rewritten); + let mut out = String::with_capacity(text.len()); + // The end of the last address moved, and where the next search starts. + let mut done = 0; + let mut search = 0; + while let Some(found) = text[search..].find(self.origin_host.as_str()) { + let host = search + found; + search = host + self.origin_host.len(); + let before = &text[done..host]; + let Some((gap, replacement)) = [ + ("https://", &self.request_url), + ("http://", &self.request_url), + ("//", &self.request_protocol_relative), + ] + .into_iter() + .find_map(|(prefix, replacement)| { + before.strip_suffix(prefix).map(|gap| (gap, replacement)) + }) else { + continue; + }; + self.push_bare_hosts(&mut out, gap); + out.push_str(replacement); + done = search; + } + self.push_bare_hosts(&mut out, &text[done..]); + out.into_bytes() + } - rewritten.into_bytes() + /// Appends `text` to `out` with a bare origin host at hostname boundaries + /// moved to the publisher's host. + fn push_bare_hosts(&self, out: &mut String, text: &str) { + match rewrite_bare_host_at_boundaries(text, &self.origin_host, &self.request_host) { + Some(rewritten) => out.push_str(&rewritten), + None => out.push_str(text), + } } fn finalize_newline_row(&mut self, out: &mut Vec) { @@ -411,6 +434,91 @@ mod tests { ); } + /// A request host that begins with the origin host, as a local host named + /// after the publisher or the origin host with a port does, is written + /// once in every address over either scheme, in a `T` row whose length + /// follows it and in a newline row, rather than matched again inside its + /// own replacement. + #[test] + fn a_request_host_that_begins_with_the_origin_host_is_written_once() { + let t_content = + r#"{"a":"https://publisher.example/news","b":"http://publisher.example/old"}"#; + let json = r#"["//publisher.example/c.png","publisher.example/bare"]"#; + let input = format!("1:T{:x},{t_content}2:{json}\n", t_content.len()); + + for request in ["publisher.example.localhost", "publisher.example:8443"] { + for scheme in ["http", "https"] { + let mut rewriter = RscFlightUrlRewriter::new("publisher.example", request, scheme); + + let output = run_rewriter(&mut rewriter, input.as_bytes(), 5); + + let t_rewritten = format!( + r#"{{"a":"{scheme}://{request}/news","b":"{scheme}://{request}/old"}}"# + ); + let json_rewritten = format!(r#"["//{request}/c.png","{request}/bare"]"#); + assert_eq!( + String::from_utf8(output).expect("should be valid UTF-8"), + format!( + "1:T{:x},{t_rewritten}2:{json_rewritten}\n", + t_rewritten.len() + ), + "should write {request} once over {scheme}" + ); + } + } + } + + /// Hosts that do not begin with the origin host are written as before, + /// both one with no part of the origin host in it and a `www.` host in + /// front of an apex origin, and a longer name that holds the origin host + /// is left alone. + #[test] + fn a_request_host_that_does_not_begin_with_the_origin_host_is_written_as_before() { + for (origin, request) in [ + ("publisher.example", "publisher-example.localhost"), + ("example.com", "www.example.com"), + ] { + let mut rewriter = RscFlightUrlRewriter::new(origin, request, "https"); + let row = format!( + r#"0:["https://{origin}/a","http://{origin}/b","//{origin}/c.png","{origin}/d","cdn.{origin}/e"]"# + ); + + let output = run_rewriter(&mut rewriter, format!("{row}\n").as_bytes(), 8192); + + let expected = format!( + r#"0:["https://{request}/a","https://{request}/b","//{request}/c.png","{request}/d","cdn.{origin}/e"]"# + ); + assert_eq!( + String::from_utf8(output).expect("should be valid UTF-8"), + format!("{expected}\n"), + "should write {request} once for {origin}" + ); + } + } + + #[test] + fn each_address_is_moved_once_from_the_start() { + let rewriter = RscFlightUrlRewriter::new("a.example", "a.example.localhost", "http"); + let rewrite = |text: &str| { + String::from_utf8(rewriter.rewrite_utf8_bytes(text.as_bytes())) + .expect("should be valid UTF-8") + }; + + assert_eq!( + rewrite("https://a.example/1 http://a.example/2 //a.example/3 a.example/4"), + "http://a.example.localhost/1 http://a.example.localhost/2 //a.example.localhost/3 a.example.localhost/4" + ); + assert_eq!(rewrite("nothing here"), "nothing here"); + assert_eq!(rewrite(""), ""); + + let no_origin = RscFlightUrlRewriter::new("", "a.example.localhost", "http"); + assert_eq!( + no_origin.rewrite_utf8_bytes(b"https://a.example/1"), + b"https://a.example/1".to_vec(), + "should leave the text alone with no origin host" + ); + } + #[test] fn the_middleware_runs_on_flight_alone_in_the_fetch_phase_and_moves_the_address_itself() { let middleware = Flight; From 338a8564421ff1adfd5e2d19a494fb171a188d7f Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 18:36:58 +0100 Subject: [PATCH 5/6] The changelog and the design say a Flight address is written once --- CHANGELOG.md | 1 + .../specs/2026-10-14-rsc-flight-middleware-design.md | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9bec320553..6a4317ac6b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -100,6 +100,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `[auction].allowed_context_keys` now serializes in sorted, deduplicated order, so ESI template-cache fingerprints and `ts config diff`/`push` envelope hashes are stable across loads. Template fingerprints also sort object keys independently of `serde_json/preserve_order`. Existing envelopes may show a one-time allowlist reorder after upgrading; push once to settle it. The updated fingerprint format causes one template-cache miss per cached page after deployment. - TSJS-generated envelopes now send `trustedServer.params.storedRequest: false`, preventing accidental PBS stored lookups without suppressing eligible non-PBS demand. PBS filters unusable impressions after overrides; explicit `true` and omission in valid envelopes retain inline-first stored fallback. A malformed envelope disables stored fallback for the entire slot, including independent direct demand left unusable after overrides. Publisher intent survives repeated and refresh auctions. Deploy compatible server admission everywhere before serving the new JS, and retain it during rollback while cached clients remain. See the Prebid deployment guide. - Protocol-relative creative URLs now honor `rewrite.exclude_domains`, so excluded creative assets stay direct and excluded absolute or protocol-relative URLs submitted to `/first-party/sign` are rejected. +- The origin's address in a React Server Components Flight response is now moved in one pass. A request host that begins with the origin host was matched again inside its own replacement, so `https://publisher.example/` served on `publisher.example.localhost` over `http` came out as `http://publisher.example.localhost.localhost.localhost/`, and on `publisher.example:8443` over `https` as `https://publisher.example:8443:8443/`. - Server-side ad template bids now always carry `hb_adid` in `window.tsjs.bids`. Bidders that return neither a Prebid Cache UUID nor an `adid` previously produced no `hb_adid` at all, so no `hb_adid` GPT targeting key was set and the Universal Creative render bridge had nothing to match — the winning creative never rendered. The OpenRTB bid `id`, which is mandatory per spec, is now the last-resort source; `cache_id` and `adid` still take priority where present. Blank `cacheId`/`adid` values no longer win that precedence and emit an unusable empty `hb_adid`, and `hb_cache_host`/`hb_cache_path` are now emitted only alongside a real Prebid Cache UUID — without one they pointed the Universal Creative at a guaranteed cache miss instead of letting it fall through to the inline creative. ### Added diff --git a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md index db922e927b..00931dabdc 100644 --- a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md +++ b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md @@ -48,7 +48,7 @@ middleware = ["framework.nextjs.flight"] **What the middleware is told.** The origin host, the reader's host and the reader's scheme from its context, which is what the module's HTML middleware works from too. The origin host is the `Host` the source sends, being the source's host and port unless the source sets another `Host`. A page names itself by that `Host`, so core tells every middleware that name as the origin host, and a middleware is told no source address. -**What the middleware moves.** From the origin host, the middleware moves an `https://` or `http://` address to the reader's scheme and host, a protocol relative address to the reader's host, and a bare host at hostname boundaries to the reader's host. What the middleware moves is not exactly what core's plain replacement moves, and not exactly what `rsc_flight` moved, the rewriter in core that this change moves out. Core's replacement and `rsc_flight` build the `https://` and `http://` forms from the source's address as written, where the middleware builds them from the host alone, and `rsc_flight` already respected hostname boundaries for a bare host, as the middleware does. The middleware differs from core's replacement in four ways and from `rsc_flight` in the first three. +**What the middleware moves.** From the origin host, the middleware moves an `https://` or `http://` address to the reader's scheme and host, a protocol relative address to the reader's host, and a bare host at hostname boundaries to the reader's host. It moves each address once, in one pass from the start, where `rsc_flight` read each replacement's output again and wrote a request host that begins with the origin host more than once. What the middleware moves is not exactly what core's plain replacement moves, and not exactly what `rsc_flight` moved, the rewriter in core that this change moves out. Core's replacement and `rsc_flight` build the `https://` and `http://` forms from the source's address as written, where the middleware builds them from the host alone, and `rsc_flight` already respected hostname boundaries for a bare host, as the middleware does. The middleware differs from core's replacement in four ways and from `rsc_flight` in the first three. 1. **A folder in the source's address.** For a source at `https://origin.example.com/shop`, core's replacement and `rsc_flight` turn `https://origin.example.com/shop/item` into `https://proxy.example.com/item`. The middleware turns the same text into `https://proxy.example.com/shop/item`, so the folder stays in a rewritten address, as it does in the links of a page. 2. **A `Host` that differs from the address.** Where the source sends another `Host` than its address names, core's replacement and `rsc_flight` look for the address as written and for the `Host`, and the middleware looks for the `Host` alone. An address in a Flight response that names only the source's own address is therefore no longer moved. @@ -78,6 +78,7 @@ The middleware builds from the host because it works from what its context carri ## 4. Tests - The rewriter moves an address in a newline row, moves an `http://` address to the reader's scheme, writes a `T` row's length again, handles a `T` row split across chunks, and leaves an unrelated bare hostname alone. +- The rewriter writes a request host that begins with the origin host once, as a `.localhost` name or with a port, in a `T` row and a newline row over both schemes, writes a hyphenated host and a `www.` host over an apex origin as before, and moves each address once from the start, leaving a row alone when the origin host is empty. - The middleware handles `text/x-component` alone, runs in the fetch phase, says it moves the origin's address itself, and its plan carries one stream processor that rewrites a Flight body. - The module registers the middleware, core leaves `text/x-component` to it and moves the addresses in every other body, a Flight response takes the entry naming it and runs nothing where no entry covers it, and an entry for HTML naming it is refused by name. - In core, a body whose registered middleware moves the address itself is handed to that middleware as the origin sent it, on the buffered path and the streamed path. The same body passes through untouched where no entry names the middleware, tested on the buffered path alone. A `text/plain` body still has its addresses moved before the entry's middleware runs. @@ -105,3 +106,4 @@ Seven cases have no test. | 2026-10-14 | First draft. | | 2026-10-15 | Fixes from review round 1. The differences from core's plain replacement and why, the alternatives to the new method, adapters, configuration, consent, failure, rollout with its open question, the cases no test covers and the review log. | | 2026-10-15 | Fixes from review round 2. The hostname boundary difference for a bare host alone, the `path` paragraph, the template comment and the review log. | +| 2026-10-16 | The rewriter moves each address once, in one pass, and the tests that show it. | From 1f07d4adfd253a2e9e20403f41dfaf6fc78872ca Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Fri, 16 Oct 2026 20:56:25 +0100 Subject: [PATCH 6/6] The design header names the pull request that carries it --- .../specs/2026-10-14-rsc-flight-middleware-design.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md index 00931dabdc..d7a79b6c61 100644 --- a/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md +++ b/docs/superpowers/specs/2026-10-14-rsc-flight-middleware-design.md @@ -1,11 +1,11 @@ # Design Spec: The Flight rewriter is a middleware of the Next.js module -**Status:** Proposed, 2026-10-14. Implemented on a working branch of the fork, with no pull request raised yet. +**Status:** Proposed, 2026-10-14. Raised as draft pull request IABTechLab/trusted-server#1381 (from jwrosewell/trusted-server#35), for Tech Lab review. **Author:** 51Degrees (contributed), for Tech Lab review **Related specs:** `2026-09-07-850-parser-aware-body-hold-nextjs-streaming-design.md`, `2026-10-14-source-modules-design.md` -**Related PRs:** none yet +**Related PRs:** IABTechLab/trusted-server#1381, jwrosewell/trusted-server#35 **Issues:** IABTechLab/trusted-server#1340 -**Last updated:** 2026-10-15 +**Last updated:** 2026-10-16 > **Why this spec exists.** Core rewrote the addresses inside a React Server Components Flight stream by its media type, in a module of its own, and the publisher path named that media type to call it. This change moves the rewriter into the Next.js module as the middleware `framework.nextjs.flight`, so the publisher path no longer names the media type. A deployment that fronts a Next.js site selects the module and names the middleware in a `[[fetch]]` entry, and section 2 says what happens to a deployment that does not.