Skip to content

Validate ESI approach to cache fragments on pages to separate cachable content for per-user events #1009

Open
Task
#1013
Open
Validate ESI approach to cache fragments on pages to separate cachable content for per-user events#1009
Task
#1013
Assignees
aram356prk-Jr
Milestone
202608

Description

@jevansnyc
jevansnyc
opened on Aug 7, 2026

Description

TS cacheable-root architecture: ESI and the alternatives

Date: 2026-08-06
Context: Trusted Server on www.prospect-a.com (prospect-a), live behind a ts-tester=true cookie + basic-auth gate. TS v394, Next.js 16.2.9 App Router, Fastly Compute.
Status: Design exploration (brainstorm). Not a committed plan.
Redaction: Publisher redacted to prospect-a / prospect-a.com. Safe for internal sharing; still confirm before any public posting (GitHub, Slack outside the deal channel, Artifacts).

Numbers labeled measured come from the live A/B below (N=4 per side, unthrottled, single US edge, one session). Numbers labeled modeled are estimates, not observations.

Part 1. Findings: TS-on vs TS-off (shareable)

We ran a tester-cookie A/B on the same URLs in isolated browser contexts: cookie on = TS/SSAT path, cookie off = plain cached origin. Same URL and auth both sides. Measured on the homepage and on a news article that was new to the tester.

Timing (measured, ms)

Metric homepage OFF homepage ON article OFF article ON
TTFB median ~123 ~1085 ~177 warm (650 first) ~806
TTFB range 70–150 569–1965 65–650 529–1229
First creative painted ~130–190 647–2013 128–717 636–1397
All creatives painted 3.4–8.6 s 3.4–5.7 s 1.4–13.7 s 2.1–6.3 s
Slot fill 6/8 every load 6/8 every load 7/10 every load 7/10 every load

Cache behavior (measured, from document response headers)

Same article, two paths:

  • TS-off: hit-state: MISS-CLUSTER but x-cache: HIT, MISS with age: 14110 → the edge PoP missed, the Fastly shield had it cached (~4h old, from other traffic). Server time 165 ms. Cacheable. First load in a fresh context was 650 ms (shield fetch), then the PoP cached it and subsequent loads dropped to 65–230 ms.
  • TS-on: hit-state: PASS, x-cache: MISS, MISS, cache-control: private, max-age=0, x-ts-version: 394. Server time 773 ms. Never shared-cached; pays the server-side auction on every load.

What it means

TS's measurable cost is TTFB, roughly +600 to +800 ms, and that is the server-side auction (SSAT) running inline before the first byte. Time-to-fully-painted ads is comparable on and off, because TS shifts the auction cost from client-side (post-load) to server-side (pre-first-byte) rather than adding new work. Fill was reliable in all 16 measured loads.

Two secondary findings:

  • A URL new to the tester does not defeat the cache. The Fastly cache is shared; a live article is already warm at the shield from other traffic. Only a cache-buster query would force a cold origin build.
  • TS's googletag shim discards event listeners queued before it loads, so third-party tooling that hooks googletag slot events before TS (viewability, some analytics) may not receive them on the TS build. Flagged for follow-up; not yet filed.

The optimization target this doc addresses: the root HTML is uncacheable under TS (PASS) while the same page off-TS is a shared HIT. If we can make the root cacheable again and keep ad slots per-user, we recover most of the ~600–800 ms.

Part 2. The goal

Cache the static root as a shared Fastly HIT (like the origin page), and deliver the per-user parts (bids, identity) out of band. That recovers the TTFB while preserving the first-party server-side auction.

It is reachable, and not by accident. Four enablers already exist in the codebase:

  • The auction is already decoupled from first byte: it is dispatched early (async) and only the </body> tail is held for bids (publisher.rs:2723, :2214).
  • The per-user payload is already isolated to two injection seams, tsjs.adSlots at <head> open and tsjs.bids/adInit() before </body> (html_processor.rs:300, :344), not smeared through the DOM.
  • A fragment endpoint already exists: /_ts/page-bids returns bids as JSON out-of-band (publisher.rs:3550).
  • A placeholder/token-substitution engine already exists: the Next.js RSC placeholder rewriter swaps payloads for __ts_rsc_payload_N__ tokens during streaming and restores them in post-processing (rsc_placeholders.rs, html_post_process.rs).

Two blockers stand in the way:

  1. The </body> hold welds the bid tail onto the same HTTP response, so the root and the per-user bids are one cache entry. The root can't cache until bids leave the response body.
  2. The cookie-privacy net force-privatizes any Set-Cookie response (response_privacy.rs:31, re-applied at send in the Fastly adapter main.rs:338), and the EC identity cookie rides the first navigation. A cacheable root must be served without setting that cookie inline; identity has to move to a subrequest or edge-KV path.

Both blockers are prerequisites for any of the approaches below, ESI included.

Part 3. Approaches (three ways to the same goal)

All three serve the Part 2 goal. They sit at different layers and are not mutually exclusive.

Lead: ESI via Fastly's esi crate

Templatize the root once (on cache MISS): TS's lol_html pass emits <esi:include src="/_ts/page-bids?..."> (and per-slot includes) at the two seams instead of inlining per-user data. The resulting ESI template carries no per-user data and is shared-cacheable. On every request, Fastly's esi crate processes the cached template, fetches the per-user fragments (the bids fragment is the SSAT auction), and streams the assembled page. The fragment fetch overlaps streaming the cached root.

Alternative A: cache-root + client-fill (no ESI)

Same cacheable templatized root, but the holes are filled by a small inline script that fetches /_ts/page-bids (which already exists) and resolves identity, client-side, after the root loads. No edge assembly, no new dependency, portable across all adapters.

Alternative B: shimmed React (correctness layer, not a transport)

Intercept React hydration to reconcile TS-rewritten URLs and to render ad slots from a late-arriving bids payload without a full re-render. This is the correctness layer that makes a static/dynamic split hydration-safe on the App Router, and it fixes the existing hydration mismatch (issues doc #1). It complements, it does not replace, whichever transport we pick.

Comparison

Dimension ESI (Fastly crate) Client-fill (/_ts/page-bids) React shim
Recovers root TTFB Yes (shared HIT) Yes (shared HIT) No (correctness only)
Where assembly happens Edge (in Compute) Client Client
Extra client round-trip for bids No (edge-assembled) Yes (visible fetch) n/a
Portable across adapters No (Fastly only) Yes Yes
New dependency risk esi 0.7.x none React-internal coupling
Fixes hydration (#1) No No Yes
Relative effort (modeled) Medium-High Low-Medium Medium

Recommendation

  • Spine on Fastly is ESI, scoped to the bids/ad holes only (see Part 5, option 2). It has the highest ceiling on the perf win because assembly stays at the edge with no client round-trip.
  • The portable fallback is client-fill via /_ts/page-bids for the non-Fastly adapters, and the baseline if the portability decision goes against ESI.
  • Layer the React shim as the correctness fix on top of whichever transport, since the hydration mismatch has to be solved regardless.
  • Defer full RSC/flight partitioning. The T-chunk length-preservation logic (rsc.rs) has no static/dynamic split today; carving cacheable vs per-user flight chunks is a separate, larger effort.

Part 4. ESI deep dive (developer view)

The crate (esi 0.7.1, June 2026, MIT)

A streaming ESI parser and executor built for Fastly Compute, implementing a subset of Akamai ESI 5.0.

  • Two modes: Processor::process_response() (buffered; allows response-manipulation functions like $add_header and auto-emits Cache-Control) and Processor::process_response_streaming() (streams body, commits headers early, low TTFB, drops the response-manipulation functions).
  • Config via Configuration: with_chunk_size(usize), with_caching(CacheConfig) (caches rendered output and fetched fragments), with_function_recursion_depth(usize).
  • Fragment sub-requests go through a caller-provided closure (req.with_ttl(120).send_async("backend")), so we control routing and can issue them concurrently. An optional second callback processes fragment responses before streaming.
  • Tag support relevant to us: esi:include, esi:vars, esi:assign, esi:choose/when/otherwise, esi:try/attempt/except, alt and onerror="continue", $(VAR) interpolation, request-control attributes (ttl, no-store, maxwait, method).
  • Requirements: the fastly crate (~0.12), so it is Fastly-Compute-only. Viceroy for tests.

The two-stage mechanism

Today TS does one streaming pass: fetch origin → lol_html rewrite → inline per-user data → hold </body> for bids → stream. That is one HTTP response and one (private) cache entry.

Under ESI it becomes two stages with a cache boundary between them:

  1. Templatize (cache MISS only). Fetch origin, lol_html-rewrite to emit <esi:include> at the two seams instead of inlining. Cache the resulting ESI template (shared, no per-user data).
  2. Assemble (every request). Run the esi crate over the cached template; it fetches the per-user fragments (bids = the auction) and streams. Because the include sits low in the document, the fragment fetch overlaps streaming the cached root.

Steady-state per-request cost drops to ESI assembly + one small auction fragment. The origin fetch and full-document rewrite happen only on MISS. That is where the modeled TTFB recovery comes from.

What it touches

  • Generalize the placeholder rewriter (rsc_placeholders.rs, html_post_process.rs) to emit esi:include markers at the two seams (html_processor.rs:300, :344).
  • Make the templatized root cacheable: relax the should_run_ad_stack privatization (publisher.rs:2882) for the templated path, and stop setting the EC cookie inline.
  • Add the esi crate as a second-stage processor on the Fastly adapter, with the current inline path retained as the non-Fastly fallback.
  • Move identity resolution to a fragment or edge-KV path (ec/ module).

Hard parts

  • Identity off-inline is a hard prerequisite. The privacy net will force-privatize the root otherwise, gate be damned (main.rs:338).
  • Two streaming rewriters in series (lol_html then esi), both !Send and chunk-oriented. Sequencing plus the cache boundary between them is the real integration cost.
  • Cache-key / vary discipline. The origin varies on rsc, next-router-*, prospect-a-Exp (A/B experiments). The template cache key must honor those or serve the wrong variant.
  • Streaming mode drops ESI response-manipulation, so TS keeps setting its own headers around the processor. Minor.

Feasibility verdict: feasible and architecturally aligned, not a moonshot. The seams already exist.

Part 5. ESI options and the tradeoff (VP of product view)

Three injection models

  1. Publisher-authored ESI (prospect-a emits the tags). Rejected: pushes work onto the publisher and breaks the "drop TS in front, no origin changes" promise.
  2. TS-injected ESI, bids/ad holes only (TS templatizes the two seams, caches the root, ESI-fills bids). This is the recommended scope: the smallest change that captures the TTFB win.
  3. TS-injected ESI, full dynamic partitioning (also carve RSC/flight, identity, consent into fragments). Defer; much larger, new work on rsc.rs.

Positive view

This kills the single biggest objection we measured against TS: the ~600–800 ms TTFB tax from running the auction inline. It converts "TS makes the page slower" into "the TS root caches like origin, ads stay per-user," which is exactly the publisher's mental model: cache the article, personalize the ads. It does so with Fastly's own, now-improved, supported crate rather than bespoke assembly code, which is on-platform and easier to defend in review. And it strengthens the core story: keep the first-party server-side auction and identity, lose the perf cost. For a performance-sensitive publisher like prospect-a that is a materially better pitch.

Negative view

  • Portability. The esi crate is Fastly-Compute-only. TS is deliberately multi-adapter (Fastly, Cloudflare, Spin/Akamai, Axum), and the EdgeZero portability story is a stated strategic asset. An ESI assembly path is Fastly-exclusive, so we would maintain two rendering architectures and the flagship perf number would exist only on Fastly. That cuts against "run anywhere."
  • Dependency risk. This puts the critical render path on a 0.7.x crate (young, ~80% documented). If Fastly deprioritizes it, we own a fork.
  • Privacy surface. Caching a root that must be per-user-identical is a deliberate exception to the cookie-privacy net that exists to prevent per-user leakage into shared cache. It has to be airtight and continuously tested.
  • Priority. This is a perf optimization, not a correctness fix. The inline path already fills reliably (6/8, 7/10 measured). It competes for eng time against open correctness issues (hydration Initialize repo #1, config Init Github actions #3, the SSAT 100x price bug).
  • Operational weight. Cache invalidation on article edits, consent changes, and experiment variants is now our problem.

The gating decision

Is Fastly-first acceptable for the flagship perf path, with the inline/client-fill path as the portable fallback?

  • If yes: scope 2 is a strong, aligned bet; green-light a spike.
  • If portability parity is non-negotiable: ESI stays a Fastly-only accelerator layered on a portable inline baseline. Fine, but it can't be the architecture, only a per-platform boost, and the client-fill approach (Alternative A) becomes the primary way to recover TTFB everywhere.

Part 6. Open questions and next steps

  1. Portability decision (blocking). Fastly-first-with-fallback, or portability parity? Everything downstream depends on this.
  2. Identity re-architecture (prerequisite for all approaches). Move EC cookie issuance off the inline navigation response to a fragment/edge-KV path. Needs its own design.
  3. Cache-key/vary strategy. Enumerate what the templated root must vary on (rsc, next-router-*, prospect-a-Exp, geo, consent state) and confirm none of it is per-user.
  4. Spike (modeled).
Step Modeled effort Confidence Why
Templatize one page type to emit esi:include at the two seams ~1 week Medium Placeholder engine exists; new emit path
Wire esi crate as second-stage on Fastly, bids fragment only ~1–2 weeks Low-Medium Two !Send pipelines in series; unproven in this codebase
Identity off-inline (throwaway spike version) ~1 week Low Touches EC module + privacy net
Measure cached-root TTFB vs today ~2 days High Reuse the A/B harness from Part 1
  1. Validation. The ~600–800 ms recovery is modeled. Confirm it with the same tester-cookie A/B harness once a spike serves a cached templated root: expect TS-on TTFB to approach the TS-off shield-HIT numbers (~65–230 ms warm).

Future-state options (not recommended now)

  • Full RSC/flight partitioning (option 3) once the bids-fragment ESI path is proven.
  • Fastly Varnish shielding / classic ESI if the topology ever puts Varnish in front of Compute, which would change the portability calculus.

Appendix: code-grounded seams (file:line)

Concern Location Note
Request entry / route decision publisher.rs:2519, :2723 handle_publisher_request; auction dispatched early
</body> hold for bids publisher.rs:2214, :2088, :772 one-behind buffer; awaits collect_dispatched_auction
Uncacheable stamp publisher.rs:2882-2888 private, max-age=0; gated on should_run_ad_stack
Cookie-privacy net response_privacy.rs:31, :72-89; adapter main.rs:338 force-privatizes Set-Cookie; forbids operator re-enable
Injection seams html_processor.rs:300-333 (head), :344-376 (body-close) the two dynamic holes
Placeholder engine rsc_placeholders.rs, html_post_process.rs token substitution to generalize for ESI markers
RSC URL rewrite nextjs/shared.rs:100-105, rsc.rs:168-334 per-origin regex; T-chunk length preservation
Bids fragment endpoint publisher.rs:3550 /_ts/page-bids JSON, already out-of-band
Fastly streaming send adapter main.rs:327-365; platform.rs:544 Compute streams; header-only cache control, no VCL/ESI today
Cacheable-asset precedent http_util.rs:293-309; proxy.rs TS already emits public/s-maxage/surrogate-control for assets

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Task

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions