diff --git a/README.md b/README.md index ff53c01..11332b4 100644 --- a/README.md +++ b/README.md @@ -60,16 +60,16 @@ The origins differ slightly in what they reject, so `CONFIG.environment` selects | `sunlight` | What [Sunlight](https://github.com/FiloSottile/sunlight) would reject (it uses Go's `encoding/json`, and `certificate-transparency-go`'s `x509` package). | | `trillian` | What the [Trillian](https://github.com/google/certificate-transparency-go/tree/master/trillian) CT front end (CTFE) would reject (it uses Go's `encoding/json`, and `certificate-transparency-go`'s `x509` package). | -If one Snippet serves several logs that run different software, `CONFIG.logEnvironments` selects the environment for each log: it is a list of `[regular expression, environment]` pairs, and a request uses the environment of the first pair whose regular expression matches its hostname followed by its path (e.g. `tiger2027h1.ct.sectigo.com/ct/v1/add-chain`), or else `CONFIG.environment`. So a log can be identified by its hostname or by a path prefix, e.g.: +If one Snippet serves several logs that run different software, `CONFIG.logEnvironments` selects the environment for each log: it is a list of `[regular expression, environment, roots URL]` entries, and a request uses the environment of the first entry whose regular expression matches its hostname followed by its path (e.g. `tiger2027h1.ct.sectigo.com/ct/v1/add-chain`), or else `CONFIG.environment`. So a log can be identified by its hostname or by a path prefix. The optional roots URL serves a list of the hashes of the log's accepted roots' subject names (see [`root-not-accepted`](#root-not-accepted)), which are only checked for logs that have one, e.g.: ```js logEnvironments: [ - [/^(tiger|elephant)\d{4}h[12]\.ct\.example\.com\//, "trillian"], + [/^(tiger|elephant)\d{4}h[12]\.ct\.example\.com\//, "trillian", "https://roots.example.net/rfc6962.txt"], [/^ct\.example\.com\/2027h1\//, "sunlight"], ], ``` -It lists Sectigo's [logs](https://github.com/sectigo/CTLogs) by default (TesseraCT for Monument, Statue, Billboard and Standee, and Trillian for Elephant, Tiger, Dumbo and Tigger), which doesn't affect other logs. A misspelt environment, here or in `CONFIG.environment`, makes the Snippet fail as soon as it is deployed. +It lists Sectigo's [logs](https://github.com/sectigo/CTLogs) by default (TesseraCT for Monument, Statue, Billboard and Standee, and Trillian for Elephant, Tiger, Dumbo and Tigger), without roots URLs, which doesn't affect other logs. A misspelt environment, here or in `CONFIG.environment`, or a roots URL that isn't a URL or that the log's own regular expression matches, makes the Snippet fail as soon as it is deployed. All the origins parse the request body in the same way, but they differ as follows, so these checks depend on the environment: @@ -164,7 +164,27 @@ A `chain` of more than 10 entries (`CONFIG.maxChainLength`) is forwarded to the `400 Bad Request` if `chain` doesn't lead to one of the log's accepted roots: no certificate in `chain` has an issuer name that is byte-identical to the subject name of an accepted root, and nor is `chain` just one certificate with such a subject name. The origins find a chain's root by name in the same way (TesseraCT, Sunlight and Trillian all require the root's subject name to match the issuer name of the certificate that it signs), and only accept the chain if it is the submitted chain, perhaps followed by the root. Only names are compared, so a chain may still be rejected by the origin, e.g. if a signature is invalid. The lookup stops at the first certificate whose issuer is an accepted root, so the certificates after it aren't checked against the roots (but must still meet [`issuer-mismatch`](#issuer-mismatch)). -The accepted roots are read from the origin's `get-roots` endpoint (the same path as the request, with `get-roots` in place of `add-chain` or `add-pre-chain`). The fetch starts while the request body is read, but the snippet doesn't wait for it if every origin already rejects the chain for another reason. Only a 32-bit [FNV-1a](https://datatracker.ietf.org/doc/html/draft-eastlake-fnv) hash of each subject name is kept, in a sorted `Uint32Array` (4 bytes per root), which is searched by binary search (about 10 comparisons for 700 roots). A name whose hash matches a root's, but which isn't the root's name, only makes the snippet forward the chain, for the origin to reject as it would without the snippet, so hash collisions (even contrived ones) can never cause a false rejection. The hashes are cached for `CONFIG.rootsCacheSeconds` (60 by default), both in Cloudflare's cache (via the `fetch()` options `cacheEverything` and `cacheTtlByStatus`, so that error responses aren't cached) and in the snippet's memory (for up to 100 logs). The response is streamed, and only enough of each root is decoded to find its subject name, so even a response of 1 MB (some logs accept about 700 roots) is read in about 1 ms and needs little memory. If the roots can't be read (e.g. the response is an error, isn't a list of certificates, or is larger than 2 MiB, which might take too long to read), this check is skipped, so the snippet never rejects a chain for want of them; such a response is cached too, but a failed fetch isn't. +This check is only made for logs that `CONFIG.logEnvironments` gives a roots URL (see [Origin environment](#origin-environment)), on another hostname. It isn't made for logs whose entry has no roots URL (or `""` or `null`), or that use `CONFIG.environment`. The snippet can't read the log's own `get-roots` endpoint, as Cloudflare fails a request with [error 1201](https://developers.cloudflare.com/rules/snippets/errors/) ("Snippet tried to continue to origin multiple times") if a Snippet makes more than one request to the origin: forwarding the submission after fetching `get-roots` from the log's hostname would always fail. So a roots URL that the log's own regular expression matches makes the Snippet fail as soon as it is deployed, and one on the requested hostname is ignored. + +The roots URL serves a list of the hashes of the log's accepted roots' subject names, precalculated so that reading it takes little CPU time even for hundreds of roots, rather than the roots themselves. [`tools/roots-hashes.mjs`](tools/roots-hashes.mjs) generates it from the log's `get-roots` endpoint, or from a file containing a `get-roots` response or a bundle of PEM certificates: + +``` +npm run roots-hashes -- https://tiger2027h1.ct.sectigo.com/ct/v1/get-roots > tiger2027h1-roots.txt +``` + +The list starts with the line `CF_CTreq Accepted Roots v1`, followed by one line per root: the 32-bit [FNV-1a](https://datatracker.ietf.org/doc/html/draft-eastlake-fnv) hash of its DER-encoded subject name (including the tag and length), as 8 hex digits, then (optionally) whitespace and its subject name, for people to read; blank lines, and lines starting with `#`, are ignored. For example: + +``` +CF_CTreq Accepted Roots v1 +# 232 roots, from https://tiger2027h1.ct.sectigo.com/ct/v1/get-roots +016305c1 OU=GlobalSign Root CA - R6, O=GlobalSign, CN=GlobalSign +01803395 C=US, ST=Texas, L=Houston, O=SSL Corporation, CN=SSL.com EV Root Certification Authority RSA R2 +... +``` + +Whoever runs the log must regenerate the list when the log's accepted roots change: if a root is missing from it, chains that lead to that root are rejected, although the log would accept them. Several logs that accept the same roots can share a roots URL, and its list is then fetched and cached once. + +The fetch starts while the request body is read, but the snippet doesn't wait for it if every origin already rejects the chain for another reason. The hashes are kept in a sorted `Uint32Array` (4 bytes per root), which is searched by binary search (about 10 comparisons for 700 roots). A name whose hash matches a root's, but which isn't the root's name, only makes the snippet forward the chain, for the origin to reject as it would without the snippet, so hash collisions (even contrived ones) can never cause a false rejection. The hashes are cached for `CONFIG.rootsCacheSeconds` (60 by default), both in Cloudflare's cache (via the `fetch()` options `cacheEverything` and `cacheTtlByStatus`, so that error responses aren't cached) and in the snippet's memory (for up to 100 logs). If the list can't be read (e.g. the response is an error, doesn't start with the expected line, has a line that doesn't start with a hash, has no hashes at all, or is larger than 256 KiB), this check is skipped, so the snippet never rejects a chain for want of it (nor rejects every chain because of an empty list); such a response is cached too, but a failed fetch isn't. ### Checks that are not done @@ -181,7 +201,7 @@ The snippet's body size limit is the origin's (see [`body-too-large`](#body-too- ## Deployment -Set `CONFIG.environment` in [`cf_ctreq.js`](cf_ctreq.js) to `tesseract`, `sunlight` or `trillian` if the log runs that software (or leave it as `common`), or `CONFIG.logEnvironments` if the Snippet serves logs that run different software (see [Origin environment](#origin-environment)), and `CONFIG.maxBodyBytes` if the origin's body size limit isn't its default (see [`body-too-large`](#body-too-large)). Then generate the deployment file: +Set `CONFIG.environment` in [`cf_ctreq.js`](cf_ctreq.js) to `tesseract`, `sunlight` or `trillian` if the log runs that software (or leave it as `common`), or `CONFIG.logEnvironments` if the Snippet serves logs that run different software or their accepted roots are to be checked (see [Origin environment](#origin-environment) and [`root-not-accepted`](#root-not-accepted)), and `CONFIG.maxBodyBytes` if the origin's body size limit isn't its default (see [`body-too-large`](#body-too-large)). Then generate the deployment file: ``` npm ci @@ -200,13 +220,13 @@ The snippet forwards every request that isn't for an `add-chain` or `add-pre-cha Keep **Normalize incoming URLs** turned on (under **Rules** > **Settings** > **URL Normalization**; it is on by default). The origin also routes percent-encoded forms of these paths, such as `/ct/v1/add%2Dchain`, to the same endpoints, and [URL normalization](https://developers.cloudflare.com/rules/normalization/how-it-works/) decodes them before the rule above is evaluated. If it were turned off, the rule wouldn't match them, so they would reach the origin without being checked. -The snippet makes up to two subrequests per request: one to forward it, and, at most once per `CONFIG.rootsCacheSeconds` for each log, one to the log's `get-roots` endpoint. That is within every plan's [subrequest limit](https://developers.cloudflare.com/rules/snippets/#availability) (2 on Pro), provided that `get-roots` isn't redirected. +The snippet makes up to two subrequests per request: one to forward it, and, for a log with a roots URL, at most once per `CONFIG.rootsCacheSeconds` for each copy of the snippet, one to the roots URL. That is within every plan's [subrequest limit](https://developers.cloudflare.com/rules/snippets/#availability) (2 on Pro), provided that the roots URL isn't redirected. Only one of them may go to the origin, or else Cloudflare fails the request with error 1201, so the roots URL must be on another hostname (see [`root-not-accepted`](#root-not-accepted)); whether one on another hostname in the same Cloudflare zone counts as the origin too isn't documented, so check a new roots URL with the [resubmission](#resubmission) check before relying on it. ### Limits Cloudflare [limits](https://developers.cloudflare.com/rules/snippets/#limits) a Snippet to 5 ms of CPU time (time spent waiting for subrequests doesn't count), 2 MB of memory and 32 KB of code. Deploy [`cf_ctreq.min.js`](cf_ctreq.min.js), which is tested against that size limit, rather than the larger readable source. Measured with Node.js on a typical machine (where CPU time is only an approximation of Cloudflare's), a typical request takes about 0.05 ms, and a real certificate as large as the body size limit allows (e.g. one with 20,000 DNS names in 512 KiB) takes about 2 ms. Only crafted certificates take longer: the slowest that the 128 KiB limit allows (one with about 10,000 extensions) takes about 1 ms in the `sunlight` environment, but the slowest that the 512 KiB limit allows (about 43,000 extensions) takes about 4 ms in the `trillian` environment, 10 ms in the `tesseract` environment (which also checks for duplicate extensions), and 12 ms in the `common` environment (which parses each certificate with both X.509 parsers). A request that exceeds the CPU time limit fails with Cloudflare [error 1203](https://developers.cloudflare.com/rules/snippets/errors/), rather than being forwarded or rejected. No CA would issue such a certificate, so a log could only receive one from a client that was testing it; set `CONFIG.maxBodyBytes` to `131072` to keep even those within the limit, at the cost of rejecting larger chains that the origin would accept. -Checking the chain against the cached accepted roots adds no measurable time. Reading the roots, when they aren't cached in memory, adds about 0.4 ms for a log with 232 roots (350 KB) and 0.9 ms for one with 670 roots (1 MB), and each log's cached hashes take about 4 KB of memory, including their cache entry. As the response is streamed, it is never all in memory, whereas parsing it with `JSON.parse()` would need over 2 MB for the larger one. +Checking the chain against the cached accepted roots adds no measurable time. Reading a list of roots, when it isn't cached in memory, adds about 0.1 ms or less, even for 700 roots (about 94 KB): a whole request that reads one takes about 1.5 ms in a fresh copy of the snippet (most of it the snippet's own start-up), and 0.3 ms otherwise. Reading the roots themselves instead (about 1 MB of certificates for 700 roots) would take about 5 ms, the whole of the time limit. Each log's cached hashes take about 4 KB of memory, including their cache entry. ## Monitoring rejections @@ -294,7 +314,7 @@ Whichever environment is selected, one test also checks the differences between ### Differential tests -The differential tests check that, in each environment, the snippet never rejects a request that one of that environment's origins would accept. They generate a corpus of request bodies (10,000 by default) with many structural variants of certificates and chains, JSON and base64, and compare the snippet's response to each, on both endpoints, with the verdicts of [models of the origins](test/differential/origins/main.go) that use the origins' own parsers: Go's `encoding/json`, and Go's `crypto/x509` (TesseraCT) or `certificate-transparency-go`'s `x509` package (Sunlight and Trillian, at the versions in [`versions.json`](test/differential/versions.json)). The models check names, precertificate status and body size as the origins do, but (like the snippet) don't verify signatures, and nor do they check that the chain leads to an accepted root, so the snippet is given no roots (it skips [`root-not-accepted`](#root-not-accepted) when it can't read them). They require Go: +The differential tests check that, in each environment, the snippet never rejects a request that one of that environment's origins would accept. They generate a corpus of request bodies (10,000 by default) with many structural variants of certificates and chains, JSON and base64, and compare the snippet's response to each, on both endpoints, with the verdicts of [models of the origins](test/differential/origins/main.go) that use the origins' own parsers: Go's `encoding/json`, and Go's `crypto/x509` (TesseraCT) or `certificate-transparency-go`'s `x509` package (Sunlight and Trillian, at the versions in [`versions.json`](test/differential/versions.json)). The models check names, precertificate status and body size as the origins do, but (like the snippet) don't verify signatures, and nor do they check that the chain leads to an accepted root, so the snippet is given no roots URLs (and so skips [`root-not-accepted`](#root-not-accepted)). They require Go: ``` npm run test:differential diff --git a/cf_ctreq.js b/cf_ctreq.js index 9c45b6f..672deeb 100644 --- a/cf_ctreq.js +++ b/cf_ctreq.js @@ -25,25 +25,31 @@ // Certificate. // - Each certificate's issuer name must be byte-identical to the subject name of // the next certificate in the chain. Signatures are not verified. -// - The chain must lead to one of the log's accepted roots, as listed by the origin's -// get-roots endpoint: the issuer name of one of its certificates (or the subject -// name of the last) must have the same hash as a root's subject name. The list is -// cached for CONFIG.rootsCacheSeconds; if it can't be fetched or read, this check is -// skipped. +// - The chain must lead to one of the log's accepted roots, if CONFIG.logEnvironments +// gives the log a roots URL (on another hostname, listing the hashes of the roots' +// subject names, as generated by tools/roots-hashes.mjs): the issuer name of one of its +// certificates (or the subject name of the last) must have the same hash as a root's. +// The list is cached for CONFIG.rootsCacheSeconds; if it can't be fetched or read, this +// check is skipped. // CONFIG.environment selects whose criteria apply: "sunlight", "tesseract", or // "trillian" rejects what that origin would reject, and "common" only rejects what -// all of them would reject. CONFIG.logEnvironments can select a different one for -// particular logs, identified by their hostnames and paths. +// all of them would reject. CONFIG.logEnvironments can select a different one (and a +// roots URL) for particular logs, identified by their hostnames and paths. // Requests that pass these checks are forwarded to the origin unchanged. const CONFIG = { // Default environment: "sunlight", "tesseract", "trillian", or "common" (only - // reject what all of them would reject). + // reject what all of them would reject). The accepted roots aren't checked for logs + // that use it. environment: "common", - // Environments for particular logs, as [regular expression, environment] pairs: a - // request uses the environment of the first pair whose regular expression matches - // its hostname followed by its path (e.g. "tiger2027h1.ct.sectigo.com/ct/v1/add-chain"), - // or else the default environment. + // Environments for particular logs, as [regular expression, environment, roots URL]: a + // request uses the environment of the first entry whose regular expression matches its + // hostname followed by its path (e.g. "tiger2027h1.ct.sectigo.com/ct/v1/add-chain"), or + // else the default environment. The optional roots URL lists the hashes of the log's + // accepted roots' subject names (see ROOTS_HEADER; tools/roots-hashes.mjs generates it + // from the log's get-roots response or a PEM bundle), and must be on another hostname, as + // Cloudflare fails a Snippet that makes more than one request to the origin (error 1201); + // the accepted roots aren't checked if it's omitted, "" or null. logEnvironments: [ // Sectigo Static CT logs. [/^(monument|statue)\d{4}h[12]\.sub\.ct\.sectigo\.com\//, "tesseract"], @@ -60,7 +66,7 @@ const CONFIG = { // without being checked, so as not to spend the Snippet's limited CPU time on them. maxChainLength: 10, minCertBytes: 128, - // How long each log's accepted roots (from its get-roots endpoint) are cached, both in + // How long each log's accepted roots (from its roots URL) are cached, both in // Cloudflare's cache and in the snippet's memory. rootsCacheSeconds: 60, // Prefix of the problem details "type" URI; the rejection reason code is appended. @@ -88,15 +94,30 @@ const ENVIRONMENTS = { trillian: [ORIGINS.trillian], common: Object.values(ORIGINS), }; -// Resolved when the snippet is loaded, so that a wrong environment fails at -// deployment rather than on every request. -const DEFAULT_ORIGINS = originsFor(CONFIG.environment, "CONFIG.environment"); -const LOG_ORIGINS = CONFIG.logEnvironments.map((pair, i) => { +// Resolved when the snippet is loaded, so that a wrong setting fails at deployment +// rather than on every request. +const DEFAULT_LOG = { origins: originsFor(CONFIG.environment, "CONFIG.environment"), rootsUrl: null }; +const LOG_SETTINGS = CONFIG.logEnvironments.map((entry, i) => { const setting = `CONFIG.logEnvironments[${i}]`; - if (!Array.isArray(pair) || pair.length !== 2 || !(pair[0] instanceof RegExp)) { - throw new Error(`${setting} must be a [regular expression, environment] pair`); + if (!Array.isArray(entry) || entry.length < 2 || entry.length > 3 || !(entry[0] instanceof RegExp)) { + throw new Error(`${setting} must be [regular expression, environment] or [regular expression, environment, roots URL]`); } - return [pair[0], originsFor(pair[1], `${setting}'s environment`)]; + const [pattern, environment, roots] = entry; + let rootsUrl = null; + if (roots !== undefined && roots !== null && roots !== "") { + let parsed; + try { + parsed = new URL(roots); + } catch { + throw new Error(`${setting}'s roots URL must be a URL, "" or null, not ${JSON.stringify(roots)}`); + } + // A roots URL that the log's own pattern matches is probably on the log's hostname. + if ((parsed.hostname + parsed.pathname).search(pattern) !== -1) { + throw new Error(`${setting}'s roots URL must be on another hostname than the log's, as Cloudflare fails a Snippet that makes more than one request to the origin: ${roots}`); + } + rootsUrl = parsed.href; + } + return { pattern, origins: originsFor(environment, `${setting}'s environment`), rootsUrl }; }); function originsFor(environment, setting) { @@ -107,16 +128,16 @@ function originsFor(environment, setting) { return ENVIRONMENTS[environment]; } -// Returns the origins of the environment for a request's URL. -function originsForLog(url) { +// Returns the settings ({ origins, rootsUrl }) for a request's URL. +function logSettings(url) { const log = url.hostname + url.pathname; - for (const [pattern, origins] of LOG_ORIGINS) { + for (const settings of LOG_SETTINGS) { // search() ignores lastIndex, so a pattern with the g or y flag matches every time. - if (log.search(pattern) !== -1) { - return origins; + if (log.search(settings.pattern) !== -1) { + return settings; } } - return DEFAULT_ORIGINS; + return DEFAULT_LOG; } // OIDs as keys from oidKey(), i.e. their minimal encodings (without tag and length). @@ -615,24 +636,33 @@ function isRoot(roots, name) { return roots[lo] === h; } -// Each log's accepted roots, by get-roots URL: { roots, expires }, roots being the +// Each log's accepted roots, by roots URL: { roots, expires }, roots being the // sorted hashes of their subject names (4 bytes per root), or null if they couldn't be // read. const ROOTS_CACHE = new Map(); const MAX_CACHED_LOGS = 100; -// Roots' subjects are mostly within their first 513 bytes (684 base64 characters). -const ROOT_PREFIX_CHARS = 684; -// The largest get-roots response that is read (the largest real ones are about 1 MB), -// so that reading one stays well within the time limit. -const MAX_ROOTS_BYTES = 2 * 1024 * 1024; +// A roots URL's response lists the hashes of a log's accepted roots' subject names (as +// generated by tools/roots-hashes.mjs): this line, then one hash per line, as 8 hex digits +// (nameHash() of the DER subject name), optionally followed by whitespace and anything else +// (such as the subject name, for people to read). Blank lines, and lines starting with "#", +// are ignored. Precalculated hashes, rather than certificates, are read so that a log with +// hundreds of roots stays well within the Snippet's CPU time limit. +const ROOTS_HEADER = "CF_CTreq Accepted Roots v1"; +const ROOTS_LINE = /^([0-9a-fA-F]{8})(?:[\t ]|$)/; +// The largest response that is read (about 3,000 annotated roots). +const MAX_ROOTS_BYTES = 256 * 1024; const ASCII = new TextDecoder(); -// Returns the hashes of the accepted root names of the log that a request (to add-chain -// or add-pre-chain) is for, or null if they are unknown, from the in-memory cache, or else -// the get-roots endpoint, whose response Cloudflare caches too. Only data is shared -// between requests, never a promise, as I/O can't be. -async function rootHashes(url) { - const rootsUrl = url.origin + url.pathname.slice(0, url.pathname.lastIndexOf("/") + 1) + "get-roots"; +// Returns the hashes of the accepted root names of a log, or null if they are unknown or +// aren't to be checked, from the in-memory cache, or else the log's roots URL, whose +// response Cloudflare caches too. Only data is shared between requests, never a promise, +// as I/O can't be. +async function rootHashes(rootsUrl, url) { + // Fetching from the log's own hostname would make forwarding fail (error 1201), so a + // roots URL there is ignored. + if (rootsUrl === null || new URL(rootsUrl).hostname === url.hostname) { + return null; + } const cached = ROOTS_CACHE.get(rootsUrl); if (cached !== undefined && cached.expires > Date.now()) { return cached.roots; @@ -653,95 +683,30 @@ async function rootHashes(url) { return roots; } -// Reads a get-roots response, {"certificates":["",...]}, returning the -// sorted hashes of the roots' subject names, or null if it isn't one. It's streamed, -// and only enough of each root is decoded to find its subject, so that a response of a -// megabyte (hundreds of roots) needs little memory and CPU time. Go's encoding/json escapes no base64 -// characters, so each string ends at the next quote. A response larger than -// MAX_ROOTS_BYTES isn't read. +// Reads a roots URL's response (see ROOTS_HEADER), returning the sorted hashes, or null if +// it isn't a list of at least one hash, or is larger than MAX_ROOTS_BYTES. async function readRootHashes(response) { - if (!response.ok || response.body === null) { + if (!response.ok) { await response.body?.cancel(); return null; } - const reader = response.body.getReader(); + const body = await readBody(response, MAX_ROOTS_BYTES); + const lines = body === null ? [] : ASCII.decode(body).split("\n"); + if (lines[0]?.trimEnd() !== ROOTS_HEADER) { + return null; + } const hashes = []; - let key; - let pending = new Uint8Array(0); - let total = 0; - for (let r = await reader.read(); !r.done; r = await reader.read()) { - total += r.value.length; - if (total > MAX_ROOTS_BYTES) { - await reader.cancel(); - return null; - } - let buf = r.value; - if (pending.length) { - buf = new Uint8Array(pending.length + r.value.length); - buf.set(pending); - buf.set(r.value, pending.length); - } - let i = 0; - for (let open, close; (open = buf.indexOf(0x22, i)) !== -1; i = close + 1) { - close = buf.indexOf(0x22, open + 1); - if (close === -1) { - break; - } - const string = buf.subarray(open + 1, close); - // The first string must be the key, and the rest roots. - const hash = key === undefined ? ((key = ASCII.decode(string)) === "certificates" ? -1 : null) : rootHash(string); - if (hash === null) { - await reader.cancel(); + for (let i = 1; i < lines.length; i++) { + const line = lines[i].trim(); + if (line !== "" && !line.startsWith("#")) { + const hex = ROOTS_LINE.exec(line)?.[1]; + if (hex === undefined) { return null; } - if (hash >= 0) { - hashes.push(hash); - } - } - pending = buf.subarray(buf.indexOf(0x22, i) === -1 ? buf.length : i); - } - return key === undefined ? null : new Uint32Array(hashes).sort(); -} - -// Returns the hash of a root's subject name from the bytes of its base64 DER, or null -// if it can't be found. Usually, only a prefix needs decoding. -function rootHash(bytes) { - try { - let der = atob(ASCII.decode(bytes.subarray(0, ROOT_PREFIX_CHARS))); - let subject = subjectOf(der); - if (subject === null && bytes.length > ROOT_PREFIX_CHARS) { - der = atob(ASCII.decode(bytes)); - subject = subjectOf(der); - } - return subject === null ? null : nameHash(der, subject.start, subject.end); - } catch { - return null; // Not base64. - } -} - -// Returns the location of the subject name in a DER certificate, or a prefix of one, -// or null if it isn't found there. Roots are trusted to be well-formed, so the -// Certificate and TBSCertificate headers, whose values extend past a prefix, are -// simply skipped. -function subjectOf(der) { - let i = 0; - for (let n = 0; n < 2; n++) { - const length = der.charCodeAt(i + 1); - i += 2 + (length > 0x80 ? length & 0x7f : 0); - } - if (der.charCodeAt(i) === 0xa0) { - i = derDecode(der, i)?.end ?? der.length; // version - } - let field; - // serialNumber, signature, issuer, validity, subject. - for (const tag of [0x02, 0x30, 0x30, 0x30, 0x30]) { - field = derDecode(der, i); - if (field?.tag !== tag) { - return null; + hashes.push(parseInt(hex, 16)); } - i = field.end; } - return field; + return hashes.length ? new Uint32Array(hashes).sort() : null; } // Returns true for an add-pre-chain path, false for an add-chain path, or null for any @@ -762,13 +727,13 @@ function isPreChainPath(pathname) { return endpoint === "add-pre-chain" ? true : endpoint === "add-chain" ? false : null; } -// Reads the request body as bytes, or returns null as soon as it exceeds maxBytes, so that -// a body without a Content-Length (e.g. chunked) is bounded too. -async function readBody(request, maxBytes) { +// Reads a request's (or response's) body as bytes, or returns null as soon as it exceeds +// maxBytes, so that a body without a Content-Length (e.g. chunked) is bounded too. +async function readBody(message, maxBytes) { const chunks = []; let total = 0; - if (request.body !== null) { - const reader = request.body.getReader(); + if (message.body !== null) { + const reader = message.body.getReader(); for (let r = await reader.read(); !r.done; r = await reader.read()) { total += r.value.byteLength; if (total > maxBytes) { @@ -866,7 +831,7 @@ function bodyRejectionOrChain(body) { export default { async fetch(request) { const url = new URL(request.url); - const origins = originsForLog(url); + const { origins, rootsUrl } = logSettings(url); const isPreChainAPI = isPreChainPath(url.pathname); if (isPreChainAPI === null) { return fetch(request); @@ -886,7 +851,7 @@ export default { return errorResponse(rejection("body-too-large", "request body too large", 413), origins); } // Started now, so that any fetch of the roots overlaps reading and parsing the body. - const rootsPromise = rootHashes(url); + const rootsPromise = rootHashes(rootsUrl, url); const body = await readBody(request, maxBodyBytes); if (body === null) { return errorResponse(rejection("body-too-large", "request body too large", 413), origins); diff --git a/cf_ctreq.min.js b/cf_ctreq.min.js index 78cc3ff..a757938 100644 --- a/cf_ctreq.min.js +++ b/cf_ctreq.min.js @@ -1 +1 @@ -const e={environment:"common",logEnvironments:[[/^(monument|statue)\d{4}h[12]\.sub\.ct\.sectigo\.com\//,"tesseract"],[/^billboard\.sub\.ct\.sectigo\.com\//,"tesseract"],[/^standee\.sub\.ctlabs\.sectigo\.com\//,"tesseract"],[/^(elephant|tiger)\d{4}h[12]\.ct\.sectigo\.com\//,"trillian"],[/^(dumbo|tigger\d{4})\.ctlabs\.sectigo\.com\//,"trillian"]],maxBodyBytes:null,maxChainLength:10,minCertBytes:128,rootsCacheSeconds:60,problemTypeBase:"https://github.com/sectigo/CF_CTreq#"},t={sunlight:{x509:"ctgo",allowsOptions:!0,rejectsPrecertSigningCerts:!0,allowsAnyOrigin:!0,maxBodyBytes:131072},tesseract:{x509:"go",allowsOptions:!1,rejectsPrecertSigningCerts:!1,allowsAnyOrigin:!1,maxBodyBytes:524288},trillian:{x509:"ctgo",allowsOptions:!1,rejectsPrecertSigningCerts:!1,allowsAnyOrigin:!1,maxBodyBytes:512e3}},n={sunlight:[t.sunlight],tesseract:[t.tesseract],trillian:[t.trillian],common:Object.values(t)},r=o(e.environment,"CONFIG.environment"),l=e.logEnvironments.map((e,t)=>{const n=`CONFIG.logEnvironments[${t}]`;if(!(Array.isArray(e)&&2===e.length&&e[0]instanceof RegExp))throw new Error(`${n} must be a [regular expression, environment] pair`);return[e[0],o(e[1],`${n}'s environment`)]});function o(e,t){if(!Object.hasOwn(n,e)){const r=Object.keys(n).map(e=>JSON.stringify(e));throw new Error(`${t} must be ${r.slice(0,-1).join(", ")}, or ${r.at(-1)}, not ${JSON.stringify(e)}`)}return n[e]}const s="+Öy",i=/^[A-Za-z0-9+/]*={0,2}$/,a=/[\r\n]/g,u=/^chain$/i;function c(e,t,n=e.length){if(t+2>n)return null;const r=e.charCodeAt(t);if(!(31&~r))return null;let l=e.charCodeAt(t+1),o=t+2;if(l>=128){const t=127&l;if(0===t||t>4||o+t>n||0===e.charCodeAt(o))return null;l=0;for(let n=0;nn?null:{tag:r,length:l,offset:o,end:o+l,start:t}}function f(e,t,n){if(t>=n)return null;const r=e.charCodeAt(t++);let l=31&r;if(31===l){l=0;for(let r=0;;r++){if(t>=n||5===r)return null;const o=e.charCodeAt(t++);if(l=128*l+(127&o),o<128)break}if(l<31||l>2147483647)return null}if(t>=n)return null;let o=e.charCodeAt(t++);if(o>=128){const r=127&o;if(0===r)return null;o=0;for(let l=0;l=n||o>=1<<23)return null;if(o=256*o+e.charCodeAt(t++),0===o)return null}if(o<128)return null}return{tag:r,number:l,length:o,offset:t}}function d(e,t,n,r){if(t===n)return"go"===r.x509?null:"";let l=!1,o=0,s=0;for(let i=t;i5)return null;if(o=128*o+(127&t),t<128){if(o>2147483647)return null;o=0,s=0}}if(0!==s)return null;if(!l)return e.slice(t,n);let i="",a=!0;for(let r=t;rl{const n=o(t)?c(e,l,r.end):null;return null!==n&&(l=n.end),n},i=e=>null!==e&&0!==e.length&&e.offset!==r.end&&(l=e.offset,!0);let a=0;if(o(160))if("go"===t.x509){const t=s(160),n=t&&c(e,t.offset,t.end);if(null===n||2!==n.tag||1!==n.length||n.end!==t.end)return null;if(a=e.charCodeAt(n.offset),a>2)return null}else if(!i(f(e,l,r.end))||null===s(2))return null;const u=[2,48,48,48,48,48].map(s);if(u.includes(null))return null;const[,d,g,,y]=u,m=c(e,r.end,n.end);if(48!==m?.tag||e.slice(m.start,m.end)!==e.slice(d.start,d.end))return null;let p=[];if("go"===t.x509){if(a>=1){if(o(129)&&null===s(129)||o(130)&&null===s(130))return null;if(2===a&&o(163)){const n=s(163),r=n&&c(e,n.offset,n.end);p=null!==r&&48===r.tag?h(e,r,t):null}}}else{for(const t of[129,130]){const n=l{if("U%"!==t.oid)return!1;const r=c(e,t.value.offset,t.value.end),l=null!==r&&48===r.tag&&r.end===t.value.end?function(e,t){const n=[];for(let r=t.offset;r6===t.tag?d(e,t.offset,t.end,n):null);return void 0!==o&&!o.includes(null)&&o.includes("+Öy")})}const m=(e,t,n=400,r={},l=void 0)=>({reason:e,message:t,status:n,headers:r,detail:l}),p=(e,t,n,r)=>m(e,t,400,{},`chain[${n}] ${r}`);function b(e,t){const n=[];for(const[r,l]of e.entries()){const e=g(l,t);if(null===e)return r;n.push(e)}return n}function w(e,t){if(null===t)return null;for(const n of e)if(x(t,n.issuer))return null;return x(t,e.at(-1).subject)?null:m("root-not-accepted","chain does not lead to a root that this log accepts")}function C(e,t=0,n=e.length){let r=2166136261;for(let l=t;l>>0}function x(e,t){const n=C(t);let r=0;for(let t=e.length;r>>1;e[l]Date.now())return r.roots;let l;try{const t=await fetch(n,{cf:{cacheEverything:!0,cacheTtlByStatus:{"200-299":e.rootsCacheSeconds}}});l=await async function(e){if(!e.ok||null===e.body)return await(e.body?.cancel()),null;const t=e.body.getReader(),n=[];let r,l=new Uint8Array(0),o=0;for(let e=await t.read();!e.done;e=await t.read()){if(o+=e.value.length,o>v)return await t.cancel(),null;let s=e.value;l.length&&(s=new Uint8Array(l.length+e.value.length),s.set(l),s.set(e.value,l.length));let i=0;for(let e,l;-1!==(e=s.indexOf(34,i))&&(l=s.indexOf(34,e+1),-1!==l);i=l+1){const o=s.subarray(e+1,l),i=void 0===r?"certificates"===(r=O.decode(o))?-1:null:j(o);if(null===i)return await t.cancel(),null;i>=0&&n.push(i)}l=s.subarray(-1===s.indexOf(34,i)?s.length:i)}return void 0===r?null:new Uint32Array(n).sort()}(t)}catch{return null}return A.delete(n),A.size>=100&&A.delete(A.keys().next().value),A.set(n,{roots:l,expires:Date.now()+1e3*e.rootsCacheSeconds}),l}function j(e){try{let t=atob(O.decode(e.subarray(0,684))),n=B(t);return null===n&&e.length>684&&(t=atob(O.decode(e)),n=B(t)),null===n?null:C(t,n.start,n.end)}catch{return null}}function B(e){let t,n=0;for(let t=0;t<2;t++){const t=e.charCodeAt(n+1);n+=2+(t>128?127&t:0)}160===e.charCodeAt(n)&&(n=c(e,n)?.end??e.length);for(const r of[2,48,48,48,48]){if(t=c(e,n),t?.tag!==r)return null;n=t.end}return t}function E({reason:t,message:n,status:r,headers:l,detail:o},s){const i={type:e.problemTypeBase+t,title:n,status:r,...void 0!==o&&{detail:o}},a=405!==r&&s.some(e=>e.allowsAnyOrigin)?{"Access-Control-Allow-Origin":"*"}:{};return new Response(JSON.stringify(i)+"\n",{status:r,headers:{"Content-Type":"application/problem+json","X-CTreq-Reject":t,...a,...l}})}function $(t){const n=new TextDecoder("utf-8",{ignoreBOM:!0}).decode(t);let r,l;try{r=JSON.parse(n)}catch{return m("invalid-json","unable to parse request body")}if(null!==r&&"object"==typeof r&&!Array.isArray(r)){const e=Object.keys(r).filter(e=>5===e.length&&u.test(e));l=e.length>1?function(e){let t,n=0;for(let r=0;re.maxChainLength)return null;const s=[];for(const[t,n]of o.entries()){let r="string"==typeof n?n:null;if(null!==r&&(r.includes("\n")||r.includes("\r"))&&(r=r.replace(a,"")),null===r||r.length%4!=0||!i.test(r))return p("entry-not-base64","chain entries must be base64 strings",t,"is not a string of standard, padded base64");const l=atob(r);if(l.lengthe.allowsOptions)?["OPTIONS","POST"]:["POST"];return n.includes(t)?null:m("method-not-allowed","method not allowed; use POST",405,{Allow:n.join(", ")})}(i,n.method);if(null!==u)return E(u,i);if("POST"!==n.method)return fetch(n);const c=e.maxBodyBytes??Math.max(...i.map(e=>e.maxBodyBytes)),f=n.headers.get("Content-Length");if(null!==f&&Number(f)>c)return E(m("body-too-large","request body too large",413),i);const d=S(o),h=await async function(e,t){const n=[];let r=0;if(null!==e.body){const l=e.body.getReader();for(let e=await l.read();!e.done;e=await l.read()){if(r+=e.value.byteLength,r>t)return await l.cancel(),null;n.push(e.value)}}const l=new Uint8Array(r);let o=0;for(const e of n)l.set(e,o),o+=e.byteLength;return l}(n,c);if(null===h)return E(m("body-too-large","request body too large",413),i);const g=$(h);if(null!==g&&!Array.isArray(g))return E(g,i);if(null!==g){const e=new Map;for(const t of i)e.has(t.x509)||e.set(t.x509,b(g,t));const n=i.map(t=>function(e,t,n){if(!Array.isArray(t))return p("cert-unparseable","chain entry cannot be parsed as an X.509 certificate",t,"cannot be parsed as an X.509 certificate");const{der:r,extensions:l}=t[0],o=l.find(e=>e.oid===s);if(void 0!==o&&(!o.critical||"\0"!==r.slice(o.value.offset,o.value.end)))return m("poison-invalid","CT poison extension must be critical and contain NULL");const i=void 0!==o;if(i&&!n)return m("precert-on-add-chain","precertificate submitted to add-chain; use add-pre-chain");if(!i&&n)return m("cert-on-add-pre-chain","certificate submitted to add-pre-chain; use add-chain");if(e.rejectsPrecertSigningCerts&&i&&t.length>1&&y(t[1],e))return m("psc-not-supported","precertificate signing certificates are not supported");for(let e=0;enull===e)){const t=await d;for(const[r,l]of i.entries())null===n[r]&&(n[r]=w(e.get(l.x509),t))}if(n.every(e=>null!==e))return E(n[Math.max(0,i.indexOf(t.tesseract))],i)}return fetch(new Request(n,{body:h}))}}; \ No newline at end of file +const t={environment:"common",logEnvironments:[[/^(monument|statue)\d{4}h[12]\.sub\.ct\.sectigo\.com\//,"tesseract"],[/^billboard\.sub\.ct\.sectigo\.com\//,"tesseract"],[/^standee\.sub\.ctlabs\.sectigo\.com\//,"tesseract"],[/^(elephant|tiger)\d{4}h[12]\.ct\.sectigo\.com\//,"trillian"],[/^(dumbo|tigger\d{4})\.ctlabs\.sectigo\.com\//,"trillian"]],maxBodyBytes:null,maxChainLength:10,minCertBytes:128,rootsCacheSeconds:60,problemTypeBase:"https://github.com/sectigo/CF_CTreq#"},e={sunlight:{x509:"ctgo",allowsOptions:!0,rejectsPrecertSigningCerts:!0,allowsAnyOrigin:!0,maxBodyBytes:131072},tesseract:{x509:"go",allowsOptions:!1,rejectsPrecertSigningCerts:!1,allowsAnyOrigin:!1,maxBodyBytes:524288},trillian:{x509:"ctgo",allowsOptions:!1,rejectsPrecertSigningCerts:!1,allowsAnyOrigin:!1,maxBodyBytes:512e3}},n={sunlight:[e.sunlight],tesseract:[e.tesseract],trillian:[e.trillian],common:Object.values(e)},r={origins:l(t.environment,"CONFIG.environment"),rootsUrl:null},o=t.logEnvironments.map((t,e)=>{const n=`CONFIG.logEnvironments[${e}]`;if(!Array.isArray(t)||t.length<2||t.length>3||!(t[0]instanceof RegExp))throw new Error(`${n} must be [regular expression, environment] or [regular expression, environment, roots URL]`);const[r,o,s]=t;let i=null;if(null!=s&&""!==s){let t;try{t=new URL(s)}catch{throw new Error(`${n}'s roots URL must be a URL, "" or null, not ${JSON.stringify(s)}`)}if(-1!==(t.hostname+t.pathname).search(r))throw new Error(`${n}'s roots URL must be on another hostname than the log's, as Cloudflare fails a Snippet that makes more than one request to the origin: ${s}`);i=t.href}return{pattern:r,origins:l(o,`${n}'s environment`),rootsUrl:i}});function l(t,e){if(!Object.hasOwn(n,t)){const r=Object.keys(n).map(t=>JSON.stringify(t));throw new Error(`${e} must be ${r.slice(0,-1).join(", ")}, or ${r.at(-1)}, not ${JSON.stringify(t)}`)}return n[t]}const s="+Öy",i=/^[A-Za-z0-9+/]*={0,2}$/,a=/[\r\n]/g,u=/^chain$/i;function c(t,e,n=t.length){if(e+2>n)return null;const r=t.charCodeAt(e);if(!(31&~r))return null;let o=t.charCodeAt(e+1),l=e+2;if(o>=128){const e=127&o;if(0===e||e>4||l+e>n||0===t.charCodeAt(l))return null;o=0;for(let n=0;nn?null:{tag:r,length:o,offset:l,end:l+o,start:e}}function f(t,e,n){if(e>=n)return null;const r=t.charCodeAt(e++);let o=31&r;if(31===o){o=0;for(let r=0;;r++){if(e>=n||5===r)return null;const l=t.charCodeAt(e++);if(o=128*o+(127&l),l<128)break}if(o<31||o>2147483647)return null}if(e>=n)return null;let l=t.charCodeAt(e++);if(l>=128){const r=127&l;if(0===r)return null;l=0;for(let o=0;o=n||l>=1<<23)return null;if(l=256*l+t.charCodeAt(e++),0===l)return null}if(l<128)return null}return{tag:r,number:o,length:l,offset:e}}function d(t,e,n,r){if(e===n)return"go"===r.x509?null:"";let o=!1,l=0,s=0;for(let i=e;i5)return null;if(l=128*l+(127&e),e<128){if(l>2147483647)return null;l=0,s=0}}if(0!==s)return null;if(!o)return t.slice(e,n);let i="",a=!0;for(let r=e;ro{const n=l(e)?c(t,o,r.end):null;return null!==n&&(o=n.end),n},i=t=>null!==t&&0!==t.length&&t.offset!==r.end&&(o=t.offset,!0);let a=0;if(l(160))if("go"===e.x509){const e=s(160),n=e&&c(t,e.offset,e.end);if(null===n||2!==n.tag||1!==n.length||n.end!==e.end)return null;if(a=t.charCodeAt(n.offset),a>2)return null}else if(!i(f(t,o,r.end))||null===s(2))return null;const u=[2,48,48,48,48,48].map(s);if(u.includes(null))return null;const[,d,g,,m]=u,p=c(t,r.end,n.end);if(48!==p?.tag||t.slice(p.start,p.end)!==t.slice(d.start,d.end))return null;let y=[];if("go"===e.x509){if(a>=1){if(l(129)&&null===s(129)||l(130)&&null===s(130))return null;if(2===a&&l(163)){const n=s(163),r=n&&c(t,n.offset,n.end);y=null!==r&&48===r.tag?h(t,r,e):null}}}else{for(const e of[129,130]){const n=o{if("U%"!==e.oid)return!1;const r=c(t,e.value.offset,e.value.end),o=null!==r&&48===r.tag&&r.end===e.value.end?function(t,e){const n=[];for(let r=e.offset;r6===e.tag?d(t,e.offset,e.end,n):null);return void 0!==l&&!l.includes(null)&&l.includes("+Öy")})}const p=(t,e,n=400,r={},o=void 0)=>({reason:t,message:e,status:n,headers:r,detail:o}),y=(t,e,n,r)=>p(t,e,400,{},`chain[${n}] ${r}`);function b(t,e){const n=[];for(const[r,o]of t.entries()){const t=g(o,e);if(null===t)return r;n.push(t)}return n}function w(t,e){if(null===e)return null;for(const n of t)if(C(e,n.issuer))return null;return C(e,t.at(-1).subject)?null:p("root-not-accepted","chain does not lead to a root that this log accepts")}function C(t,e){const n=function(t,e=0,n=t.length){let r=2166136261;for(let o=e;o>>0}(e);let r=0;for(let e=t.length;r>>1;t[o]Date.now())return r.roots;let o;try{const n=await fetch(e,{cf:{cacheEverything:!0,cacheTtlByStatus:{"200-299":t.rootsCacheSeconds}}});o=await async function(t){if(!t.ok)return await(t.body?.cancel()),null;const e=await j(t,O),n=null===e?[]:S.decode(e).split("\n");if(n[0]?.trimEnd()!==A)return null;const r=[];for(let t=1;t=100&&x.delete(x.keys().next().value),x.set(e,{roots:o,expires:Date.now()+1e3*t.rootsCacheSeconds}),o}async function j(t,e){const n=[];let r=0;if(null!==t.body){const o=t.body.getReader();for(let t=await o.read();!t.done;t=await o.read()){if(r+=t.value.byteLength,r>e)return await o.cancel(),null;n.push(t.value)}}const o=new Uint8Array(r);let l=0;for(const t of n)o.set(t,l),l+=t.byteLength;return o}function B({reason:e,message:n,status:r,headers:o,detail:l},s){const i={type:t.problemTypeBase+e,title:n,status:r,...void 0!==l&&{detail:l}},a=405!==r&&s.some(t=>t.allowsAnyOrigin)?{"Access-Control-Allow-Origin":"*"}:{};return new Response(JSON.stringify(i)+"\n",{status:r,headers:{"Content-Type":"application/problem+json","X-CTreq-Reject":e,...a,...o}})}function E(e){const n=new TextDecoder("utf-8",{ignoreBOM:!0}).decode(e);let r,o;try{r=JSON.parse(n)}catch{return p("invalid-json","unable to parse request body")}if(null!==r&&"object"==typeof r&&!Array.isArray(r)){const t=Object.keys(r).filter(t=>5===t.length&&u.test(t));o=t.length>1?function(t){let e,n=0;for(let r=0;rt.maxChainLength)return null;const s=[];for(const[e,n]of l.entries()){let r="string"==typeof n?n:null;if(null!==r&&(r.includes("\n")||r.includes("\r"))&&(r=r.replace(a,"")),null===r||r.length%4!=0||!i.test(r))return y("entry-not-base64","chain entries must be base64 strings",e,"is not a string of standard, padded base64");const o=atob(r);if(o.lengtht.allowsOptions)?["OPTIONS","POST"]:["POST"];return n.includes(e)?null:p("method-not-allowed","method not allowed; use POST",405,{Allow:n.join(", ")})}(i,n.method);if(null!==c)return B(c,i);if("POST"!==n.method)return fetch(n);const f=t.maxBodyBytes??Math.max(...i.map(t=>t.maxBodyBytes)),d=n.headers.get("Content-Length");if(null!==d&&Number(d)>f)return B(p("body-too-large","request body too large",413),i);const h=$(a,l),g=await j(n,f);if(null===g)return B(p("body-too-large","request body too large",413),i);const C=E(g);if(null!==C&&!Array.isArray(C))return B(C,i);if(null!==C){const t=new Map;for(const e of i)t.has(e.x509)||t.set(e.x509,b(C,e));const n=i.map(e=>function(t,e,n){if(!Array.isArray(e))return y("cert-unparseable","chain entry cannot be parsed as an X.509 certificate",e,"cannot be parsed as an X.509 certificate");const{der:r,extensions:o}=e[0],l=o.find(t=>t.oid===s);if(void 0!==l&&(!l.critical||"\0"!==r.slice(l.value.offset,l.value.end)))return p("poison-invalid","CT poison extension must be critical and contain NULL");const i=void 0!==l;if(i&&!n)return p("precert-on-add-chain","precertificate submitted to add-chain; use add-pre-chain");if(!i&&n)return p("cert-on-add-pre-chain","certificate submitted to add-pre-chain; use add-chain");if(t.rejectsPrecertSigningCerts&&i&&e.length>1&&m(e[1],t))return p("psc-not-supported","precertificate signing certificates are not supported");for(let t=0;tnull===t)){const e=await h;for(const[r,o]of i.entries())null===n[r]&&(n[r]=w(t.get(o.x509),e))}if(n.every(t=>null!==t))return B(n[Math.max(0,i.indexOf(e.tesseract))],i)}return fetch(new Request(n,{body:g}))}}; \ No newline at end of file diff --git a/package.json b/package.json index cbc8ed7..b15ed43 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,7 @@ "test:minified": "CTREQ_MINIFIED=1 node --test test/cf_ctreq.test.js", "test:forwarding": "node test/forwarding.mjs", "test:resubmit": "node test/resubmit.mjs", + "roots-hashes": "node tools/roots-hashes.mjs", "test:differential": "node test/differential/run.mjs", "test:replay": "node test/differential/run.mjs --replay" }, diff --git a/test/cf_ctreq.test.js b/test/cf_ctreq.test.js index ce196d7..0f88dd1 100644 --- a/test/cf_ctreq.test.js +++ b/test/cf_ctreq.test.js @@ -3,6 +3,7 @@ import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; import { readFileSync } from "node:fs"; import { tlv, seq, ctOid, OID_EKU, ext, POISON_EXT, makeCert, b64, makeCA, der, CERT, PRECERT, NONCRITICAL, ISSUER } from "./der.mjs"; +import { ROOTS_HEADER, certificatesIn, rootsHashList } from "../tools/roots-hashes.mjs"; // CTREQ_ENVIRONMENT selects CONFIG.environment ("sunlight", "tesseract", "trillian", or "common") // for the snippet under test; by default, the snippet's own setting is tested. Another @@ -148,11 +149,14 @@ if (process.env.CTREQ_VERBOSE || ["verbose", "silly"].includes(process.env.npm_c snippet = { fetch: verbose }; } -// The snippet's get-roots requests ({ url, init }) are answered by getRoots(url), which -// by default lists ISSUER, the issuer of all the other fixtures, as the only root. +// The snippet's requests for roots ({ url, init }) are answered by getRoots(url), which by +// default lists ISSUER, the issuer of all the other fixtures, as the only root. Lists of +// roots are made by tools/roots-hashes.mjs, so these tests also check that its hashes match +// the snippet's. let forwarded; let rootsFetches; -const rootsResponse = (...roots) => Response.json({ certificates: roots }); +const rootsList = (...roots) => rootsHashList(roots.map((root) => Buffer.from(root, "base64"))); +const rootsResponse = (...roots) => new Response(rootsList(...roots)); let getRoots; beforeEach(() => { forwarded = null; @@ -207,10 +211,25 @@ const postBytes = (path, bytes) => const ca = (subject, issuer = subject) => b64(makeCert({ subject: Buffer.from(subject), issuer })); // The DER subject name of makeCert({ subject }), as the snippet compares it. const nameDer = (cn) => seq(tlv(0x31, seq(tlv(0x06, [0x55, 0x04, 0x03]), tlv(0x0c, Buffer.from(cn))))).toString("latin1"); -// These tests use a fresh snippet, with an empty cache of roots, and are never sent to a deployed log. -const freshSnippet = () => loadSnippet(ENVIRONMENT); const NULL = [0x05, 0x00]; +// Each test log's roots URL, a copy of its get-roots response on another hostname, and the +// log that a roots URL is for. +const rootsUrlFor = (log) => `https://roots.example.net/${log}/get-roots`; +const logOf = (rootsUrl) => decodeURIComponent(new URL(rootsUrl).pathname.split("/")[1]); +const LOG_ENVIRONMENTS = /^(\s*logEnvironments: )\[[\s\S]*?\n \],$/m; +assert.match(SOURCE, LOG_ENVIRONMENTS, "cf_ctreq.js must set CONFIG.logEnvironments to a multi-line array"); +// A snippet with the given CONFIG.logEnvironments (as source code), in the given environment. +const logEnvironmentsSnippet = (logEnvironments, environment = ENVIRONMENT) => + loadSource(SOURCE.replace(CONFIG_ENVIRONMENT, `$1${JSON.stringify(environment)}`).replace(LOG_ENVIRONMENTS, `$1${logEnvironments},`)); +// A fresh snippet whose CONFIG.logEnvironments gives each of the given logs (hostnames, +// optionally followed by a path) logEnvironment and its roots URL, rootsUrlFor(log), with +// environment as the default. These tests are never sent to a deployed log. +const rootsSnippet = (logs = ["log.example.com"], { environment = ENVIRONMENT, logEnvironment = environment } = {}) => { + const escape = (s) => s.replace(/[.*+?^${}()|[\]\\/]/g, "\\$&"); + return logEnvironmentsSnippet(`[${logs.map((log) => `[/^${escape(log)}\\//, ${JSON.stringify(logEnvironment)}, ${JSON.stringify(rootsUrlFor(log))}]`).join(", ")}]`, environment); +}; + // Cloudflare limits a Snippet's CPU time, not its elapsed time, so this returns the least // CPU time, in ms, that Node's main thread takes for run(prepare(i)) over several runs, so // that garbage collection and other processes don't decide the result. prepare() isn't timed. @@ -303,7 +322,7 @@ describe("Body size and forwarding", () => { }); test("oversized Content-Length is rejected without fetching roots", async () => { - const s = await freshSnippet(); + const s = await rootsSnippet(); for (const endpoint of ["add-chain", "add-pre-chain"]) { const req = post(`/ct/v1/${endpoint}`, { chain: [CERT, ISSUER] }); req.headers.set("Content-Length", String(MAX_BODY + 1)); @@ -324,17 +343,17 @@ describe("Body size and forwarding", () => { }); test("Content-Length at the limit still fetches roots", async () => { - const s = await freshSnippet(); + const s = await rootsSnippet(); const req = post("/ct/v1/add-chain", { chain: [CERT, ISSUER] }); req.headers.set("Content-Length", String(MAX_BODY)); const res = await s.fetch(req); assert.equal(res.status, 200); assert.equal(rootsFetches.length, 1); - assert.equal(rootsFetches[0].url, "https://log.example.com/ct/v1/get-roots"); + assert.equal(rootsFetches[0].url, rootsUrlFor("log.example.com")); }); test("a log-specific environment determines the Content-Length limit before fetching roots", async () => { - const s = await loadSnippet("sunlight"); + const s = await rootsSnippet(["tiger2027h1.ct.sectigo.com"], { environment: "sunlight", logEnvironment: "trillian" }); const certs = JSON.stringify({ chain: [CERT, ISSUER] }); const maxLength = 128 * 1024; for (const [url, accepted] of [ @@ -347,11 +366,11 @@ describe("Body size and forwarding", () => { assert.equal(req.bodyUsed, accepted); assert.equal(rootsFetches.length, accepted ? 1 : 0); } - assert.equal(rootsFetches[0].url, "https://tiger2027h1.ct.sectigo.com/ct/v1/get-roots"); + assert.equal(rootsFetches[0].url, rootsUrlFor("tiger2027h1.ct.sectigo.com")); }); test("fetching roots still overlaps reading a permitted body", async () => { - const s = await freshSnippet(); + const s = await rootsSnippet(); let startReading; const reading = new Promise((resolve) => { startReading = resolve; }); let releaseBody; @@ -505,13 +524,14 @@ describe("JSON and chain entries", () => { test("chains of more than 10 entries are forwarded without being checked", async () => { // Rejected with 10 entries, for an entry that isn't base64 and for not leading to a root. for (const [chain, reason] of [[[...Array(9).fill(CERT), "!!!"], "entry-not-base64"], [Array(10).fill(b64(makeCert({ issuer: "Unknown CA", subject: Buffer.from("Unknown CA") }))), "root-not-accepted"]]) { - let res = await snippet.fetch(post("/ct/v1/add-chain", { chain })); + const s = reason === "root-not-accepted" ? await rootsSnippet() : snippet; + let res = await s.fetch(post("/ct/v1/add-chain", { chain })); assert.equal(outcome(res), reason); // But forwarded byte for byte with 11, or many more. for (const extra of [[CERT], Array(1000).fill("!!!")]) { const body = JSON.stringify({ chain: [...chain, ...extra] }); forwarded = null; - res = await snippet.fetch(post("/ct/v1/add-pre-chain", body)); + res = await s.fetch(post("/ct/v1/add-pre-chain", body)); assert.equal(outcome(res), "forwarded"); assert.equal(forwarded.body, body); } @@ -861,6 +881,50 @@ describe("Certificate and chain validation", () => { }); describe("Accepted roots and caching", () => { + test("the accepted roots are only checked for logs that CONFIG.logEnvironments gives a roots URL", async () => { + const s = await logEnvironmentsSnippet(`[ + [/^checked\\.example\\.com\\//, ${JSON.stringify(ENVIRONMENT)}, ${JSON.stringify(rootsUrlFor("checked.example.com"))}], + [/^empty\\.example\\.com\\//, ${JSON.stringify(ENVIRONMENT)}, ""], + [/^null\\.example\\.com\\//, ${JSON.stringify(ENVIRONMENT)}, null], + [/^omitted\\.example\\.com\\//, ${JSON.stringify(ENVIRONMENT)}], + ]`); + const unknownRoot = JSON.stringify({ chain: [b64(makeCert({ issuer: "Unknown CA" }))] }); + // Logs that use CONFIG.environment (as no entry matches them) aren't checked either. + for (const log of ["empty", "null", "omitted", "default"]) { + const res = await s.fetch(new Request(`https://${log}.example.com/ct/v1/add-chain`, { method: "POST", body: unknownRoot })); + assert.equal(outcome(res), "forwarded", log); + } + assert.equal(rootsFetches.length, 0); + const res = await s.fetch(new Request("https://checked.example.com/ct/v1/add-chain", { method: "POST", body: unknownRoot })); + assert.equal(outcome(res), "root-not-accepted"); + assert.deepEqual(rootsFetches.map((f) => f.url), [rootsUrlFor("checked.example.com")]); + }); + + test("roots are never fetched from the log's own hostname, as Cloudflare would fail the forwarded request", async () => { + // Cloudflare fails a Snippet that makes more than one request to the origin (error 1201). + // A roots URL that the log's own pattern matches fails at deployment. + await assert.rejects( + logEnvironmentsSnippet('[[/^ct\\.example\\.com\\//, "trillian", "https://ct.example.com/2027h1/ct/v1/get-roots"]]'), + /CONFIG.logEnvironments\[0\]'s roots URL must be on another hostname than the log's/, + ); + await assert.rejects(logEnvironmentsSnippet('[[/^ct\\.example\\.com\\//, "trillian", "roots.json"]]'), /CONFIG.logEnvironments\[0\]'s roots URL must be a URL, "" or null, not "roots.json"/); + // Otherwise, one on the requested hostname is ignored, and the roots aren't checked. + const s = await logEnvironmentsSnippet('[[/^ct\\.example\\.com\\/2027h1\\//, "trillian", "https://ct.example.com/roots.json"]]'); + const res = await s.fetch(new Request("https://ct.example.com/2027h1/ct/v1/add-chain", { method: "POST", body: JSON.stringify({ chain: [b64(makeCert({ issuer: "Unknown CA" }))] }) })); + assert.equal(outcome(res), "forwarded"); + assert.equal(rootsFetches.length, 0); + }); + + test("logs that share a roots URL share its cached roots", async () => { + const url = rootsUrlFor("shared"); + const s = await logEnvironmentsSnippet(`[[/^ct\\.example\\.com\\//, ${JSON.stringify(ENVIRONMENT)}, ${JSON.stringify(url)}]]`); + for (const shard of ["2027h1", "2027h2"]) { + const res = await s.fetch(new Request(`https://ct.example.com/${shard}/ct/v1/add-chain`, { method: "POST", body: JSON.stringify({ chain: [CERT, ISSUER] }) })); + assert.equal(outcome(res), "forwarded", shard); + } + assert.deepEqual(rootsFetches.map((f) => f.url), [url]); + }); + test("chains rejected by every origin do not wait for roots", async () => { const badCert = b64(der(Array(160).fill(0xab))); const otherCA = b64(makeCert({ subject: Buffer.from("Other CA") })); @@ -879,7 +943,7 @@ describe("Accepted roots and caching", () => { cases.push(["/ct/v1/add-pre-chain", [PRECERT, psc, ISSUER], "psc-not-supported"]); } for (const [path, chain, reason] of cases) { - const s = await freshSnippet(); + const s = await rootsSnippet(); let releaseRoots; getRoots = () => new Promise((resolve) => { releaseRoots = resolve; }); const request = s.fetch(post(path, { chain })); @@ -901,12 +965,12 @@ describe("Accepted roots and caching", () => { }); test("common waits for roots when only some origins reject", async () => { - const s = await loadSnippet("common"); + const s = await rootsSnippet(["accepted.example.com", "unaccepted.example.com", "unavailable.example.com"], { environment: "common" }); // TesseraCT rejects the version; Sunlight and Trillian accept it. const chain = [b64(makeCert({ version: [3] })), ISSUER]; for (const [host, roots, want] of [ ["accepted", rootsResponse(ISSUER), "forwarded"], - ["unaccepted", rootsResponse(), "cert-unparseable"], + ["unaccepted", rootsResponse(ca("Other CA")), "cert-unparseable"], ["unavailable", new Response(null, { status: 503 }), "forwarded"], ]) { let releaseRoots; @@ -931,12 +995,12 @@ describe("Accepted roots and caching", () => { }); test("the chain must lead to an accepted root, whose subject is the issuer of any certificate in it", async () => { - const s = await freshSnippet(); + const s = await rootsSnippet(["a", "b", "c", "x"].map((log) => `${log}.example.com`)); // leaf <- A <- B, issued by C. const leaf = b64(makeCert({ issuer: "A" })); const chain = [leaf, ca("A", "B"), ca("B", "C")]; const roots = { a: ca("A"), b: ca("B"), c: ca("C"), x: ca("X") }; - getRoots = (url) => rootsResponse(roots[new URL(url).hostname.split(".")[0]]); + getRoots = (url) => rootsResponse(roots[logOf(url).split(".")[0]]); const check = async (log, chain) => outcome(await s.fetch(new Request(`https://${log}.example.com/ct/v1/add-chain`, { method: "POST", body: JSON.stringify({ chain }) }))); for (const [log, want] of Object.entries({ a: "forwarded", b: "forwarded", c: "forwarded", x: "root-not-accepted" })) { assert.equal(await check(log, chain), want, log); @@ -956,7 +1020,7 @@ describe("Accepted roots and caching", () => { }); test("once a certificate's issuer is an accepted root, no more are looked up", async () => { - const s = await freshSnippet(); + const s = await rootsSnippet(); const chain = [b64(makeCert({ issuer: "A" })), ca("A", "B"), ca("B", "C")]; getRoots = () => rootsResponse(ca("A"), ca("B"), ca("C")); const submit = () => s.fetch(new Request("https://log.example.com/ct/v1/add-chain", { method: "POST", body: JSON.stringify({ chain }) })); @@ -977,12 +1041,12 @@ describe("Accepted roots and caching", () => { assert.equal(bytes, nameDer("A").length * expected({ sunlight: 1, tesseract: 1, trillian: 1, common: 3 })); }); - test("each log's roots are fetched from its get-roots endpoint and cached for CONFIG.rootsCacheSeconds", async () => { - const s = await freshSnippet(); + test("each log's roots are fetched from its roots URL and cached for CONFIG.rootsCacheSeconds", async () => { + const s = await rootsSnippet(["ct.example.com/2027h1", "ct.example.com/2027h2", "other.example.com/2027h1"]); const submit = (url, chain = [CERT, ISSUER]) => s.fetch(new Request(url, { method: "POST", body: JSON.stringify({ chain }) })); await submit("https://ct.example.com/2027h1/ct/v1/add-chain"); assert.equal(rootsFetches.length, 1); - assert.equal(rootsFetches[0].url, "https://ct.example.com/2027h1/ct/v1/get-roots"); + assert.equal(rootsFetches[0].url, rootsUrlFor("ct.example.com/2027h1")); // Cloudflare caches it too, for at least a minute, unless it's an error. assert.deepEqual(rootsFetches[0].init, { cf: { cacheEverything: true, cacheTtlByStatus: { "200-299": 60 } } }); // The same log, by either endpoint, uses the cached roots. @@ -992,7 +1056,7 @@ describe("Accepted roots and caching", () => { // Other logs have their own. await submit("https://ct.example.com/2027h2/ct/v1/add-chain"); await submit("https://other.example.com/2027h1/ct/v1/add-chain"); - assert.deepEqual(rootsFetches.slice(1).map((f) => f.url), ["https://ct.example.com/2027h2/ct/v1/get-roots", "https://other.example.com/2027h1/ct/v1/get-roots"]); + assert.deepEqual(rootsFetches.slice(1).map((f) => f.url), [rootsUrlFor("ct.example.com/2027h2"), rootsUrlFor("other.example.com/2027h1")]); // Fetched again once the cached roots expire. const now = Date.now; try { @@ -1009,21 +1073,26 @@ describe("Accepted roots and caching", () => { }); test("if the roots can't be fetched or read, chains aren't checked against them", async () => { - const s = await freshSnippet(); const unknownRoot = [b64(makeCert({ issuer: "Unknown CA" }))]; - const submit = (log) => s.fetch(new Request(`https://${log}.example.com/ct/v1/add-chain`, { method: "POST", body: JSON.stringify({ chain: unknownRoot }) })); + const list = rootsList(ISSUER); const bad = { "not-found": () => new Response("404 page not found", { status: 404 }), - "server-error": () => Response.json({ certificates: [ISSUER] }, { status: 500 }), - "not-json": () => new Response("origin"), + "server-error": () => new Response(list, { status: 500 }), + "not-a-list": () => new Response("origin"), "empty-body": () => new Response(null), - "other-key": () => Response.json({ roots: [ISSUER] }), - "not-base64": () => Response.json({ certificates: [ISSUER, "!!!!"] }), - "escaped": () => new Response(`{"certificates":["\\u00${ISSUER.charCodeAt(0).toString(16)}${ISSUER.slice(1)}"]}`), - "not-a-cert": () => Response.json({ certificates: [ISSUER, b64(der(Array(160).fill(0xab)))] }), - // Valid, but more than 2 MiB, so not read in case it would take too long. - "too-large": () => new Response(JSON.stringify({ certificates: [ISSUER] }) + " ".repeat(2 * 1024 * 1024)), + "no-header": () => new Response(list.slice(list.indexOf("\n") + 1)), + "other-version": () => new Response(list.replace(ROOTS_HEADER, ROOTS_HEADER.replace("v1", "v2"))), + "get-roots-response": () => Response.json({ certificates: [ISSUER] }), + "short-hash": () => new Response(`${ROOTS_HEADER}\n1a2b3c4\n`), + "long-hash": () => new Response(`${ROOTS_HEADER}\n1a2b3c4d5\n`), + "not-hex": () => new Response(`${list}zzzzzzzz CN=Z\n`), + // A list with no hashes (perhaps a truncated or blank file) would otherwise reject everything. + "no-hashes": () => new Response(`${ROOTS_HEADER}\n# No roots.\n`), + // Valid, but more than 256 KiB, so not read in case it would take too long. + "too-large": () => new Response(list + "#".repeat(256 * 1024) + "\n"), }; + const s = await rootsSnippet([...Object.keys(bad), "unreachable"].map((log) => `${log}.example.com`)); + const submit = (log) => s.fetch(new Request(`https://${log}.example.com/ct/v1/add-chain`, { method: "POST", body: JSON.stringify({ chain: unknownRoot }) })); for (const [log, response] of Object.entries(bad)) { getRoots = response; assert.equal(outcome(await submit(log)), "forwarded", log); @@ -1040,51 +1109,67 @@ describe("Accepted roots and caching", () => { getRoots = () => rootsResponse(ISSUER); assert.equal(outcome(await submit("unreachable")), "root-not-accepted"); assert.equal(rootsFetches.length, Object.keys(bad).length + 2); + }); - // A log with no roots accepts nothing. - getRoots = () => rootsResponse(); - const res = await s.fetch(new Request("https://no-roots.example.com/ct/v1/add-chain", { method: "POST", body: JSON.stringify({ chain: [CERT, ISSUER] }) })); - assert.equal(outcome(res), "root-not-accepted"); + test("the list of roots is read in any chunks, in any order, with annotations, comments and either line ending", async () => { + const roots = [ca("A"), b64(makeCert({ version: null, subject: Buffer.from("V1 CA"), issuer: "V1 CA" })), ISSUER]; + const [header, ...lines] = rootsList(...roots).trimEnd().split("\n"); + assert.equal(header, ROOTS_HEADER); + const lists = { + generated: rootsList(...roots), + // Unsorted, with upper-case hashes, comments, blank lines, and either line ending. + edited: [`${header} `, "# Accepted roots", "", ...lines.reverse().map((line, i) => (i % 2 ? line.toUpperCase() : line))].join("\r\n") + "\r\n", + // Hashes alone, without annotations or a final line break. + "hashes only": [header, ...lines.map((line) => line.slice(0, 8))].join("\n"), + }; + for (const [name, text] of Object.entries(lists)) { + const s = await rootsSnippet(); + rootsFetches = []; + // Sent one byte at a time. + getRoots = () => new Response(new ReadableStream({ + start(controller) { + for (const byte of new TextEncoder().encode(text)) controller.enqueue(new Uint8Array([byte])); + controller.close(); + }, + })); + const check = async (chain) => outcome(await s.fetch(new Request("https://log.example.com/ct/v1/add-chain", { method: "POST", body: JSON.stringify({ chain }) }))); + assert.equal(await check([b64(makeCert({ issuer: "A" }))]), "forwarded", name); + assert.equal(await check([b64(makeCert({ issuer: "V1 CA" }))]), "forwarded", name); + assert.equal(await check([CERT]), "forwarded", name); + assert.equal(await check([b64(makeCert({ issuer: "B" }))]), "root-not-accepted", name); + assert.equal(rootsFetches.length, 1, name); + } }); - test("get-roots responses are read in any chunks, and roots in any form", async () => { - const s = await freshSnippet(); - // A root whose subject ends past the prefix that is decoded first, and v1 roots. - const longName = "L".repeat(600); - const roots = [ca(longName), b64(makeCert({ version: null, subject: Buffer.from("V1 CA"), issuer: "V1 CA" })), ISSUER]; - // As Go's json.Encoder writes it, with a final newline, sent one byte at a time. - const text = JSON.stringify({ certificates: roots }) + "\n"; - getRoots = () => new Response(new ReadableStream({ - start(controller) { - for (const byte of new TextEncoder().encode(text)) controller.enqueue(new Uint8Array([byte])); - controller.close(); - }, - })); - const check = async (chain) => outcome(await s.fetch(new Request("https://log.example.com/ct/v1/add-chain", { method: "POST", body: JSON.stringify({ chain }) }))); - assert.equal(await check([b64(makeCert({ issuer: longName }))]), "forwarded"); - assert.equal(await check([b64(makeCert({ issuer: "V1 CA" }))]), "forwarded"); - assert.equal(await check([CERT]), "forwarded"); - assert.equal(await check([b64(makeCert({ issuer: "L".repeat(599) }))]), "root-not-accepted"); - assert.equal(rootsFetches.length, 1); + test("tools/roots-hashes.mjs lists the same roots from a get-roots response or a PEM bundle", () => { + const roots = [ca("A"), ca("B", "A"), ISSUER]; + const pem = roots.map((root) => `# A comment\n-----BEGIN CERTIFICATE-----\n${root.match(/.{1,64}/g).join("\n")}\n-----END CERTIFICATE-----\n`).join(""); + const fromJson = certificatesIn(JSON.stringify({ certificates: roots })); + const fromPem = certificatesIn(pem); + assert.deepEqual(fromPem, fromJson); + const list = rootsHashList(fromJson, ["3 roots"]); + const [header, comment, ...lines] = list.trimEnd().split("\n"); + assert.deepEqual([header, comment], [ROOTS_HEADER, "# 3 roots"]); + // Sorted by hash, each followed by the subject name. + assert.deepEqual(lines, [...lines].sort()); + assert.deepEqual(lines.map((line) => line.slice(10)).sort(), ["CN=A", "CN=B", "CN=Test CA"]); + for (const line of lines) assert.match(line, /^[0-9a-f]{8} CN=/); }); test("hundreds of roots are read, and looked up, well within a Snippet's 5 ms limit", async () => { - const s = await freshSnippet(); - // 700 roots of about 1,500 bytes, like the largest real get-roots responses (~1 MB). - const roots = Array.from({ length: 700 }, (_, i) => - b64(makeCert({ subject: Buffer.from(`Root CA ${i} `.padEnd(120, "x")), issuer: `Root CA ${i} `.padEnd(120, "x"), extensions: [ext([0x2a, 0x03], false, Buffer.alloc(1000, i))] }))); - const body = new TextEncoder().encode(JSON.stringify({ certificates: roots })); - assert.ok(body.length > 1_000_000, `${body.length} bytes`); - getRoots = () => new Response(body); - const chain = JSON.stringify({ chain: [b64(makeCert({ issuer: `Root CA 699 `.padEnd(120, "x") }))] }); + const s = await rootsSnippet(["warm-up", ...Array.from({ length: 5 }, (_, i) => `miss${i}`)].map((log) => `${log}.example.com`)); + // 700 roots, more than any real log accepts. + const names = Array.from({ length: 700 }, (_, i) => `Root CA ${i} `.padEnd(120, "x")); + const list = new TextEncoder().encode(rootsList(...names.map((name) => ca(name)))); + getRoots = () => new Response(list); + const chain = JSON.stringify({ chain: [b64(makeCert({ issuer: names.at(-1) }))] }); const request = (log) => new Request(`https://${log}.example.com/ct/v1/add-chain`, { method: "POST", body: chain }); const submit = async (req) => assert.equal(outcome(await s.fetch(req)), "forwarded"); await submit(request("warm-up")); // Each log's roots are read once, then cached. const miss = await fastest((i) => request(`miss${i}`), submit); const hit = await fastest(() => request("miss0"), submit); - // Generous, so as not to fail on a slow machine. - assert.ok(miss < 40, `reading the roots took ${miss.toFixed(1)} ms of CPU time`); + assert.ok(miss < 5, `reading the roots took ${miss.toFixed(1)} ms of CPU time`); assert.ok(hit < 5, `with cached roots, took ${hit.toFixed(1)} ms of CPU time`); }); }); @@ -1116,12 +1201,7 @@ describe("Origin environments and rejection responses", () => { }); test("CONFIG.logEnvironments selects the environment by hostname and path", async () => { - const LOG_ENVIRONMENTS = /^(\s*logEnvironments: )\[[\s\S]*?\n \],$/m; - assert.match(SOURCE, LOG_ENVIRONMENTS, "cf_ctreq.js must set CONFIG.logEnvironments to a multi-line array"); - const load = async (logEnvironments) => { - const source = SOURCE.replace(CONFIG_ENVIRONMENT, `$1${JSON.stringify(ENVIRONMENT)}`).replace(LOG_ENVIRONMENTS, `$1${logEnvironments},`); - return loadSource(source); - }; + const load = (logEnvironments) => logEnvironmentsSnippet(logEnvironments); // OPTIONS is only answered by Sunlight, and an unknown version only rejected by TesseraCT. const v4 = b64(makeCert({ version: [3] })); const behaviour = async (s, base) => { @@ -1166,7 +1246,7 @@ describe("Origin environments and rejection responses", () => { "https://other.example.com/2026h2/": null, }); await assert.rejects(load('[[/tiger/, "ctfe"]]'), /CONFIG.logEnvironments\[0\]'s environment must be "sunlight", "tesseract", "trillian", or "common", not "ctfe"/); - await assert.rejects(load('[[/tiger/, "trillian"], ["tuscolo", "sunlight"]]'), /CONFIG.logEnvironments\[1\] must be a \[regular expression, environment\] pair/); + await assert.rejects(load('[[/tiger/, "trillian"], ["tuscolo", "sunlight"]]'), /CONFIG.logEnvironments\[1\] must be \[regular expression, environment\] or \[regular expression, environment, roots URL\]/); }); test("every rejection carries an X-CTreq-Reject reason", async () => { @@ -1195,7 +1275,7 @@ describe("Origin environments and rejection responses", () => { cases["psc-not-supported"] = post(PRE, { chain: [PRECERT, psc, ISSUER] }); } for (const [reason, req] of Object.entries(cases)) { - const res = await snippet.fetch(req); + const res = await (reason === "root-not-accepted" ? await rootsSnippet() : snippet).fetch(req); assert.notEqual(res.status, 200, reason); assert.equal(res.headers.get("X-CTreq-Reject"), reason); assert.equal(res.headers.get("Content-Type"), "application/problem+json", reason); diff --git a/test/differential/run.mjs b/test/differential/run.mjs index 41dea4d..ba4b51b 100644 --- a/test/differential/run.mjs +++ b/test/differential/run.mjs @@ -99,7 +99,7 @@ if (sunlightVersion === trillianVersion) { runModels(trillianVersion, ["trillian"], corpusFile, verdicts); } -// The snippet's outcomes: its rejection reason, or "forwarded". Its get-roots requests get +// The snippet's outcomes: its rejection reason, or "forwarded". Any requests for roots get // this too, which isn't a list of roots, so it doesn't check chains against roots (which // the models don't model). globalThis.fetch = async () => new Response("origin"); diff --git a/test/forwarding.mjs b/test/forwarding.mjs index 6c64cfa..a9ba631 100644 --- a/test/forwarding.mjs +++ b/test/forwarding.mjs @@ -14,6 +14,7 @@ import assert from "node:assert/strict"; import { readFileSync } from "node:fs"; import { GOOGLE_MONITOR_LEAF, GOOGLE_MONITOR_INTERMEDIATE, GOOGLE_MONITOR_ROOT } from "./google-monitor.mjs"; import { cloudflareError, describeBody } from "./cloudflare.mjs"; +import { certificatesIn, rootsHashList } from "../tools/roots-hashes.mjs"; if (!process.env.CTREQ_LOG_URL) { console.error("Usage: CTREQ_LOG_URL= npm run test:forwarding, e.g. CTREQ_LOG_URL=https://tiger2027h1.ct.sectigo.com"); @@ -32,13 +33,15 @@ test(`a request that the snippet forwards reaches ${log.href}`, async (t) => { leaf[leaf.length - 1] ^= 0xff; const body = JSON.stringify({ chain: [leaf.toString("base64"), GOOGLE_MONITOR_INTERMEDIATE, GOOGLE_MONITOR_ROOT] }); - // The local snippet, with its own configuration for this log and the log's roots, must forward it. + // The local snippet, with its own configuration for this log and the log's roots (if it + // has a roots URL), must forward it. + const rootsList = rootsHashList(certificatesIn(rootsText)); const source = readFileSync(new URL("../cf_ctreq.js", import.meta.url), "utf8"); const local = (await import("data:text/javascript;base64," + Buffer.from(source).toString("base64"))).default; const networkFetch = globalThis.fetch; let forwarded = false; globalThis.fetch = async (req) => { - if (!(req instanceof Request)) return new Response(rootsText); + if (!(req instanceof Request)) return new Response(rootsList); forwarded = true; return new Response(null, { status: 200 }); }; diff --git a/tools/roots-hashes.mjs b/tools/roots-hashes.mjs new file mode 100644 index 0000000..113ce56 --- /dev/null +++ b/tools/roots-hashes.mjs @@ -0,0 +1,81 @@ +// Generates the list of a log's accepted roots that a roots URL in CONFIG.logEnvironments +// serves (see ROOTS_HEADER in cf_ctreq.js): the hashes of the roots' subject names, each +// followed by the subject name, for people to read. +// +// Usage: npm run roots-hashes -- > roots.txt +// where is the URL of a log's get-roots endpoint, or a file containing a get-roots +// response ({"certificates": [...]}) or a bundle of PEM certificates. +import { readFileSync } from "node:fs"; +import { X509Certificate } from "node:crypto"; + +export const ROOTS_HEADER = "CF_CTreq Accepted Roots v1"; + +// Returns the DER certificates in a get-roots response or a bundle of PEM certificates. +export function certificatesIn(text) { + if (text.trimStart().startsWith("{")) { + return JSON.parse(text).certificates.map((cert) => Buffer.from(cert, "base64")); + } + return [...text.matchAll(/-----BEGIN CERTIFICATE-----([\s\S]*?)-----END CERTIFICATE-----/g)].map((m) => Buffer.from(m[1].replace(/\s+/g, ""), "base64")); +} + +// Returns the 32-bit FNV-1a hash of a DER certificate's subject name (including its tag and +// length), as cf_ctreq.js's nameHash() hashes the names in a chain. +export function subjectHash(der) { + const tlv = (i) => { + let length = der[i + 1]; + let offset = i + 2; + if (length & 0x80) { + const n = length & 0x7f; + length = 0; + for (let j = 0; j < n; j++) length = length * 256 + der[offset++]; + } + if (offset + length > der.length) throw new Error("truncated certificate"); + return { tag: der[i], start: i, offset, end: offset + length }; + }; + const tbs = tlv(tlv(0).offset); + let i = tbs.offset; + if (der[i] === 0xa0) i = tlv(i).end; // version + // serialNumber, signature, issuer, validity, subject. + let field; + for (const tag of [0x02, 0x30, 0x30, 0x30, 0x30]) { + field = tlv(i); + if (field.tag !== tag) throw new Error("not a certificate"); + i = field.end; + } + let h = 0x811c9dc5; + for (let j = field.start; j < field.end; j++) h = Math.imul(h ^ der[j], 0x01000193); + return h >>> 0; +} + +// Returns the list for the given DER certificates, sorted by hash, with any comment lines. +export function rootsHashList(certs, comments = []) { + const lines = certs.map((der) => { + let name; + try { + name = new X509Certificate(der).subject.split("\n").join(", "); + } catch { + name = "(unreadable subject)"; + } + return `${subjectHash(der).toString(16).padStart(8, "0")} ${name}`; + }); + return [ROOTS_HEADER, ...comments.map((c) => `# ${c}`), ...[...new Set(lines)].sort()].join("\n") + "\n"; +} + +if (import.meta.url === `file://${process.argv[1]}`) { + const [source] = process.argv.slice(2); + if (!source) { + console.error("Usage: npm run roots-hashes -- > roots.txt"); + process.exit(2); + } + let text; + if (/^https?:\/\//.test(source)) { + const res = await fetch(source); + if (!res.ok) throw new Error(`${source}: HTTP ${res.status}`); + text = await res.text(); + } else { + text = readFileSync(source, "utf8"); + } + const certs = certificatesIn(text); + if (!certs.length) throw new Error(`${source}: no certificates found`); + process.stdout.write(rootsHashList(certs, [`${certs.length} roots, from ${source}`])); +}