Skip to content

Snippet: read accepted roots from a precalculated list on another hostname - #67

Merged
robstradling merged 1 commit into
mainfrom
roots-url
Nov 16, 2026
Merged

robstradling merged 1 commit into
mainfrom
roots-url

Conversation

@robstradling

Copy link
Copy Markdown
Member

Summary

Fixes the Cloudflare error 1201 outage: v1.5.0 fetched get-roots from the log's own hostname and then forwarded the submission, which Cloudflare fails as "Snippet tried to continue to origin multiple times". Requests that reached a copy of the snippet without cached roots (including every one of Google's log monitoring submissions) failed with a 503.

Roots URL per log

  • CONFIG.logEnvironments entries can now be [regular expression, environment, roots URL]; the roots URL is optional, so existing entries are unchanged.
  • The accepted roots are only checked for logs with a roots URL: not when it's omitted, "" or null, nor for logs that use CONFIG.environment. Sectigo's default entries have none yet, so as configured, no roots are checked.
  • The Snippet fails to deploy if a roots URL isn't a URL, or if the log's own regular expression matches it (so it's probably on the log's hostname). At runtime, one on the requested hostname is ignored, so the snippet never fetches roots from the log itself.
  • Roots are cached by URL, so logs that share one fetch it once.

Precalculated roots list

  • A roots URL serves a list of the hashes of the roots' subject names: the header CF_CTreq Accepted Roots v1, then one root per line, as the 8-hex-digit FNV-1a hash of its DER subject name, optionally followed by the subject name for people to read; blank lines and # comments are ignored.
  • npm run roots-hashes -- <source> (tools/roots-hashes.mjs) generates it from a log's get-roots URL, a get-roots response or a PEM bundle, sorted and annotated.
  • Reading precalculated hashes keeps the Snippet well within its 5 ms CPU time limit: a whole request that reads a list takes about 1.5 ms in a fresh snippet (mostly its own start-up) and 0.3 ms warmed up, even for 700 roots, where reading the roots themselves as JSON took about 5 ms.
  • A list that can't be read is skipped, never rejecting for want of roots: an error response, a missing header, a malformed line, more than 256 KiB, or no hashes at all (so a blank or truncated file can't reject everything).

Tests and docs

  • The roots tests give each test log a roots URL on another hostname, and build their lists with the generator, so they also check that its hashes match the snippet's. New tests cover when roots are checked, refusing a roots URL on the log's hostname, shared roots URLs, the list format, and the generator's output from JSON and PEM. The forwarding test's local check uses a generated list.
  • README: the new entry format, why roots can't come from the log itself, the list format and generator, and the new timings.

Validation

  • All tests pass in all four environments, for both the readable and minified snippets; differential tests report no false rejections.
  • With Tiger2027h1's real roots (232 roots, a 19 KB list) served from another hostname, Google's monitoring chain is forwarded and a chain to an unknown root is rejected.

Before re-enabling in production

Deploy the new cf_ctreq.min.js to the ctlabs zone and run npm run test:resubmit against Standee and Dumbo; then configure a roots URL (ideally outside the zone, as whether a hostname in the same zone also counts as the origin isn't documented) and run it again.

Fetching get-roots from the log's own hostname and then forwarding the submission made Cloudflare fail it with error 1201 ("Snippet tried to continue to origin multiple times"). Instead, CONFIG.logEnvironments entries can give a log a roots URL, which must be on another hostname (the Snippet fails to deploy if the log's own pattern matches it, and one on the requested hostname is ignored); roots aren't checked for logs without one, or that use CONFIG.environment.

The roots URL serves a list of the hashes of the roots' subject names ("CF_CTreq Accepted Roots v1"), annotated with the names, which tools/roots-hashes.mjs generates from a log's get-roots endpoint, a get-roots response or a PEM bundle. Reading precalculated hashes, rather than the roots themselves, keeps the Snippet well within its 5 ms CPU time limit (about 0.1 ms for 700 roots, rather than about 5 ms). A list that can't be read, including an empty one, skips the check.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@robstradling
robstradling merged commit 6120016 into main Nov 16, 2026
21 checks passed
@robstradling
robstradling deleted the roots-url branch November 16, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant