diff --git a/CHANGELOG.md b/CHANGELOG.md index 46cca95..17e2010 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- **ALPN in `tls_info` now reaches the server.** The SDK documented the client's ALPN list as `tls_info.alpn_protocols`, but the detection service reads `tls_info.alpn` and ignored the other name. The field is now `alpn`. `alpn_protocols` still works, is marked deprecated, and is sent as `alpn` when `alpn` is not set. +- **Web Bot Auth default directories match the platform's.** `DEFAULT_SIGNED_AGENT_DIRECTORIES` no longer lists `https://operator.openai.com`, which no longer resolves and so never supplied keys. It now lists ChatGPT (`https://chatgpt.com`), Google Agent (`https://agent.bot.goog`, Google's AI browsing agent, not Googlebot) and WebDecoyBot (`https://bot.webdecoy.com`, category `monitoring`), the same set the edge validator trusts. If you pass your own `directories`, nothing changes. +- **`captcha.verifyToken()` example awaits the result.** The `@webdecoy/node` README called it without `await`, so `result.valid` was always `undefined` and the check always failed. + +### Changed + +- **`trustedJA4Headers` is documented as it behaves.** The self-hosted captcha reads a JA4 fingerprint from the trusted headers you list, but the built-in JA4 table is empty, so the value does not change the score and is not reported anywhere. The README's TLS fingerprinting description now says that server-side fingerprinting is JA3, from `tls_info` you supply. + ## [0.18.0] - 2026-09-27 ### Added diff --git a/README.md b/README.md index 3ec84a9..901a66b 100644 --- a/README.md +++ b/README.md @@ -194,7 +194,7 @@ The honest rule of thumb: enforce tripwires you control the surface of, and use Everything above runs locally and free, forever. Add an API key to turn on the hosted platform when you want deeper detection and visibility: - **`protect()`** — full server-side analysis (a threat score + allow/block/challenge decision), not just local rules. -- **TLS fingerprinting** — JA3/JA4 hashing and matching against known automation (curl, wget, Selenium, …) and spoofed-browser (TLS↔UA mismatch) detection. +- **TLS fingerprinting** — when you pass the client's TLS ClientHello details in `tls_info`, the server computes a JA3 fingerprint, matches it against known automation (curl, wget, Selenium, …) and checks it against the claimed browser (TLS↔UA mismatch). The adapters do not fill `tls_info`, because Node does not expose the client's ClientHello; this applies when your own TLS terminator can supply it. - **IP enrichment** — reputation, geo, and Tor/VPN/proxy/hosting detection that powers `filter()` expressions. - **Dashboard & analytics** — every tripwire hit and violation, tracked over time. @@ -231,7 +231,7 @@ if (!result.allowed) { |------|------|:---:| | **0 — Tripwires** | Requests for hidden honeypot paths are blocked immediately, before any scoring. Deterministic, zero-FP. | No | | **1 — Local analysis** | Fast on-server heuristics: suspicious/missing headers, datacenter IPs, known bot user-agents, missing `Sec-CH-UA`. | No | -| **2 — Server verification** | JA3/JA4 TLS fingerprinting, known-bot database, TLS↔UA mismatch, IP reputation, GeoIP (Tor/VPN/proxy). | Yes | +| **2 — Server verification** | Known-bot database, IP reputation, GeoIP (Tor/VPN/proxy), and JA3 TLS fingerprinting with TLS↔UA mismatch when `tls_info` is supplied. | Yes | ## Packages diff --git a/docs/verify-ai-agents-web-bot-auth.md b/docs/verify-ai-agents-web-bot-auth.md index 8237cb9..3c2d1e5 100644 --- a/docs/verify-ai-agents-web-bot-auth.md +++ b/docs/verify-ai-agents-web-bot-auth.md @@ -129,15 +129,18 @@ of agents on an allowlist — never a URL taken from the incoming request's middleware fetch an arbitrary origin (no SSRF), and the warm path stays on in-memory keys. -The default list tracks the agents that sign production traffic today (OpenAI -Operator, ChatGPT). Override it — for example to add your own signed crawlers — -via the constructor: +The default list, `DEFAULT_SIGNED_AGENT_DIRECTORIES`, tracks the agents that +sign production traffic today (ChatGPT, Google Agent, WebDecoyBot) and matches +the list the WebDecoy edge validator trusts. Passing `directories` replaces it, +so spread the defaults in to add your own signed crawlers: ```typescript +import { WebDecoy, DEFAULT_SIGNED_AGENT_DIRECTORIES } from '@webdecoy/node'; + const wd = new WebDecoy({ webBotAuth: { directories: [ - { name: 'OpenAI', category: 'ai_crawlers', directory: 'https://operator.openai.com' }, + ...DEFAULT_SIGNED_AGENT_DIRECTORIES, { name: 'Acme Crawler', category: 'monitoring', directory: 'https://crawler.acme.example' }, ], cacheTtlMs: 6 * 60 * 60 * 1000, // stale-while-revalidate; default 6h diff --git a/openwiki/concepts/bot-identity-policy-and-web-bot-auth.md b/openwiki/concepts/bot-identity-policy-and-web-bot-auth.md index 5847f6b..50f1a72 100644 --- a/openwiki/concepts/bot-identity-policy-and-web-bot-auth.md +++ b/openwiki/concepts/bot-identity-policy-and-web-bot-auth.md @@ -161,7 +161,7 @@ Configure additional trusted agents only when their directory is expected and co const wd = new WebDecoy({ webBotAuth: { directories: [ - { name: 'OpenAI', category: 'ai_crawlers', directory: 'https://operator.openai.com' }, + ...DEFAULT_SIGNED_AGENT_DIRECTORIES, // passing `directories` replaces the defaults { name: 'Acme Crawler', category: 'monitoring', directory: 'https://crawler.acme.example' }, ], cacheTtlMs: 6 * 60 * 60 * 1000, diff --git a/packages/webdecoy/README.md b/packages/webdecoy/README.md index 94b9901..9b51e8a 100644 --- a/packages/webdecoy/README.md +++ b/packages/webdecoy/README.md @@ -170,8 +170,8 @@ import { Captcha } from '@webdecoy/node'; const captcha = new Captcha({ secret: process.env.WEBDECOY_SECRET }); -app.post('/login', (req, res) => { - const result = captcha.verifyToken(req.body.webdecoy_token, req.ip); +app.post('/login', async (req, res) => { + const result = await captcha.verifyToken(req.body.webdecoy_token, req.ip); if (!result.valid) return res.status(403).json({ error: 'captcha failed' }); // ...proceed }); diff --git a/packages/webdecoy/src/agent/directory.ts b/packages/webdecoy/src/agent/directory.ts index 8457b46..aec8794 100644 --- a/packages/webdecoy/src/agent/directory.ts +++ b/packages/webdecoy/src/agent/directory.ts @@ -29,10 +29,18 @@ const COLD_RETRY_BACKOFF_MS = 30_000; /** * The default curated allowlist. These are the agents that sign production * traffic today; kept in lock-step with the backend's curated list. + * + * `https://operator.openai.com` is not listed: that host no longer resolves, + * so it contributed no keys. OpenAI's signed traffic verifies against the + * `https://chatgpt.com` directory. + * + * Google Agent is Google's AI browsing agent, not Googlebot. Googlebot does not + * sign requests and is verified by IP range and reverse DNS instead. */ export const DEFAULT_SIGNED_AGENT_DIRECTORIES: SignedAgentDirectory[] = [ - { name: 'OpenAI', category: 'ai_crawlers', directory: 'https://operator.openai.com' }, { name: 'OpenAI ChatGPT', category: 'ai_crawlers', directory: 'https://chatgpt.com' }, + { name: 'Google Agent', category: 'ai_crawlers', directory: 'https://agent.bot.goog' }, + { name: 'WebDecoyBot', category: 'monitoring', directory: 'https://bot.webdecoy.com' }, ]; interface DirectoryCacheOptions { diff --git a/packages/webdecoy/src/agent/types.ts b/packages/webdecoy/src/agent/types.ts index 2dee4a2..875af8e 100644 --- a/packages/webdecoy/src/agent/types.ts +++ b/packages/webdecoy/src/agent/types.ts @@ -68,7 +68,7 @@ export interface SignedAgentDirectory { category: AgentCategory; /** * Origin (scheme + host) whose well-known HTTP Message Signatures directory - * publishes the agent's keys, e.g. `https://operator.openai.com`. The + * publishes the agent's keys, e.g. `https://chatgpt.com`. The * `/.well-known/http-message-signatures-directory` path is appended * automatically. A full URL ending in a path is also accepted verbatim. */ diff --git a/packages/webdecoy/src/agent/web-bot-auth.test.ts b/packages/webdecoy/src/agent/web-bot-auth.test.ts index 8a021cf..17de1ad 100644 --- a/packages/webdecoy/src/agent/web-bot-auth.test.ts +++ b/packages/webdecoy/src/agent/web-bot-auth.test.ts @@ -10,6 +10,7 @@ */ import { AgentVerifier } from './verifier'; +import { DEFAULT_SIGNED_AGENT_DIRECTORIES } from './directory'; import { jwkThumbprint, rsaThumbprint } from './thumbprint'; import type { SignedAgentDirectory } from './types'; @@ -280,3 +281,45 @@ describe('JWK thumbprint', () => { expect(await rsaThumbprint(n, 'AQAB')).toBe('NzbLsXh8uDCcd-6MNwXF4W_7noWXFZAfHkxZsRGC9Xs'); }); }); + +describe('DEFAULT_SIGNED_AGENT_DIRECTORIES', () => { + // Mirrors the backend's curated signed-agent list, which the edge validator + // is fed from. The SDK and the edge must trust the same directories, or a + // request verified at one is "claimed" at the other. + it('matches the curated directories the platform trusts', () => { + expect(DEFAULT_SIGNED_AGENT_DIRECTORIES.map((d) => [d.directory, d.category])).toEqual([ + ['https://chatgpt.com', 'ai_crawlers'], + ['https://agent.bot.goog', 'ai_crawlers'], + ['https://bot.webdecoy.com', 'monitoring'], + ]); + }); + + it('does not list operator.openai.com, which no longer resolves', () => { + expect(DEFAULT_SIGNED_AGENT_DIRECTORIES.map((d) => d.directory)).not.toContain( + 'https://operator.openai.com', + ); + }); + + it('is what a verifier fetches when no directories are configured', async () => { + const fetched: string[] = []; + const verifier = new AgentVerifier({ + fetchImpl: (async (url: string) => { + fetched.push(String(url)); + return new Response(JSON.stringify({ keys: [] }), { status: 200 }); + }) as unknown as typeof fetch, + }); + const kp = (await crypto.subtle.generateKey({ name: 'Ed25519' }, true, [ + 'sign', + 'verify', + ])) as CryptoKeyPair; + const jwk = (await crypto.subtle.exportKey('jwk', kp.publicKey)) as { kty: string; crv: string; x: string }; + const keyid = await jwkThumbprint({ kty: jwk.kty, crv: jwk.crv, x: jwk.x }); + const headers = await signHeaders({ privateKey: kp.privateKey, keyid, url: 'https://site.example/' }); + await verifier.verify(new Request('https://site.example/', { headers })); + expect(fetched.sort()).toEqual( + DEFAULT_SIGNED_AGENT_DIRECTORIES.map( + (d) => `${d.directory}/.well-known/http-message-signatures-directory`, + ).sort(), + ); + }); +}); diff --git a/packages/webdecoy/src/captcha/service.ts b/packages/webdecoy/src/captcha/service.ts index b67bfb8..8573b99 100644 --- a/packages/webdecoy/src/captcha/service.ts +++ b/packages/webdecoy/src/captcha/service.ts @@ -33,7 +33,15 @@ export interface CaptchaOptions { secret?: string; /** Override per-category detection weights. */ weights?: Record; - /** Trusted reverse-proxy header names carrying a JA4 fingerprint. */ + /** + * Trusted reverse-proxy header names carrying a JA4 fingerprint, matched + * against the lower-cased request header keys (e.g. `['x-ja4']`). + * + * Today this has no effect on the score: the value is read and compared with + * a built-in table of known automation JA4 fingerprints that ships empty, so + * nothing matches. It is not recorded on the verdict and not sent anywhere. + * Only list headers your own proxy sets, since a client can send any header. + */ trustedJA4Headers?: string[]; /** Pluggable stores (default to in-memory; swap for Redis in production). */ challengeStore?: ChallengeStore; diff --git a/packages/webdecoy/src/client.ts b/packages/webdecoy/src/client.ts index f7169a5..1ddd219 100644 --- a/packages/webdecoy/src/client.ts +++ b/packages/webdecoy/src/client.ts @@ -113,7 +113,7 @@ export class WebDecoyClient { response = await this.request( 'POST', '/api/v1/sdk/detect', - request, + toWireDetectionRequest(request), ); } catch (error) { if (error instanceof Error && (error.name === 'AbortError' || error.name === 'TimeoutError')) { @@ -230,3 +230,23 @@ export class WebDecoyClient { } } } + +/** + * Shape a detection request for the wire. The detection service reads the + * client's ALPN list from `tls_info.alpn`; the SDK once documented it as + * `alpn_protocols`, which the service ignores. Callers still passing the old + * name get it sent under the name the service reads, and the old key is never + * sent. + */ +export function toWireDetectionRequest(request: SDKDetectionRequest): SDKDetectionRequest { + const tls = request.request_metadata.tls_info; + if (!tls || tls.alpn_protocols === undefined) return request; + const { alpn_protocols, ...rest } = tls; + return { + ...request, + request_metadata: { + ...request.request_metadata, + tls_info: { ...rest, alpn: rest.alpn ?? alpn_protocols }, + }, + }; +} diff --git a/packages/webdecoy/src/detection/detectors/tls.ts b/packages/webdecoy/src/detection/detectors/tls.ts index b260754..b457d80 100644 --- a/packages/webdecoy/src/detection/detectors/tls.ts +++ b/packages/webdecoy/src/detection/detectors/tls.ts @@ -21,7 +21,12 @@ export const KNOWN_BOT_JA3_HASHES: Record = { '5d7974c9fe7862e0f9a3eb35a6a5d9c8': 'Puppeteer default', }; -/** Populate with observed automation JA4 fingerprints per deployment. */ +/** + * Known automation JA4 fingerprints. Ships empty: the SDK has no curated JA4 + * data, so {@link checkJA4Fingerprint} never matches and a JA4 read from + * `trustedJA4Headers` does not change the score. The platform computes JA4 + * only where it terminates the TLS handshake itself. + */ export const KNOWN_BOT_JA4_HASHES: Record = {}; export function checkJA3Fingerprint(ja3Hash?: string | null): Detection[] { diff --git a/packages/webdecoy/src/detection/engine.test.ts b/packages/webdecoy/src/detection/engine.test.ts index 702d621..7503f11 100644 --- a/packages/webdecoy/src/detection/engine.test.ts +++ b/packages/webdecoy/src/detection/engine.test.ts @@ -513,3 +513,22 @@ describe('JA3 fingerprint matching', () => { expect(hasReasonIncluding(v, 'TLS fingerprint matches')).toBe(true); }); }); + +describe('JA4 from trusted headers', () => { + // The documented behaviour of `trustedJA4Headers` is that it does not change + // the score, because the built-in JA4 table is empty. If JA4 data is ever + // added, this fails so the option's documentation is updated with it. + it('reads the header but does not change the score', () => { + const engine = new DetectionEngine(); + const base: DetectionContext = { + ip: '73.15.22.100', + siteKey: 'test', + userAgent: UA_CHROME_WIN, + headers: { 'user-agent': UA_CHROME_WIN, ...GOOD_HEADERS, 'x-ja4': 't13d1516h2_8daaf6152771_02713d6af862' }, + }; + const without = engine.score({}, base); + const withJA4 = engine.score({}, { ...base, trustedJA4Headers: ['x-ja4'] }); + expect(withJA4.score).toBe(without.score); + expect(hasReasonIncluding(withJA4, 'JA4')).toBe(false); + }); +}); diff --git a/packages/webdecoy/src/detection/types.ts b/packages/webdecoy/src/detection/types.ts index fc54c2a..44e81b0 100644 --- a/packages/webdecoy/src/detection/types.ts +++ b/packages/webdecoy/src/detection/types.ts @@ -273,7 +273,12 @@ export interface DetectionContext { headers?: Record; /** Client-supplied JA3 hash (spoofable). */ ja3Hash?: string | null; - /** Trusted reverse-proxy header names carrying a JA4 fingerprint. */ + /** + * Trusted reverse-proxy header names carrying a JA4 fingerprint, matched + * against the lower-cased `headers` keys. The value is checked against + * `KNOWN_BOT_JA4_HASHES`, which ships empty, so it does not currently + * change the score. + */ trustedJA4Headers?: string[]; /** Proof-of-work verification outcome (supplied by the PoW subsystem). */ pow?: PoWOutcome; diff --git a/packages/webdecoy/src/tls-info-wire.test.ts b/packages/webdecoy/src/tls-info-wire.test.ts new file mode 100644 index 0000000..dd7fb52 --- /dev/null +++ b/packages/webdecoy/src/tls-info-wire.test.ts @@ -0,0 +1,86 @@ +/** + * Pins the TLS info field names to what the detection service parses. + * The service's struct tag is `alpn`; a differently named key is silently + * dropped by its JSON decoder, so a rename here would lose the signal with no + * error anywhere. + */ + +import { WebDecoyClient, toWireDetectionRequest } from './client'; +import type { SDKDetectionRequest, TLSInfo } from './types'; + +function request(tls: TLSInfo): SDKDetectionRequest { + return { + request_metadata: { + method: 'GET', + path: '/', + ip: '203.0.113.7', + headers: {}, + tls_info: tls, + timestamp: 0, + }, + local_analysis: { + suspicious_headers: false, + missing_sec_ch_ua: false, + datacenter_ip: false, + local_score: 0, + needs_verification: true, + flags: [], + }, + }; +} + +describe('tls_info on the wire', () => { + const realFetch = global.fetch; + afterEach(() => { + global.fetch = realFetch; + }); + + async function sent(tls: TLSInfo): Promise> { + let body: any; + global.fetch = jest.fn(async (_url: any, init: any) => { + body = JSON.parse(init.body); + return new Response( + JSON.stringify({ + decision: 'allow', + confidence: 0, + threat_level: 'MINIMAL', + bot_detected: false, + detection_id: 'd-1', + rule_enforced: false, + }), + { status: 200 }, + ); + }) as any; + const client = new WebDecoyClient({ + apiKey: 'sk_test_key', + apiUrl: 'https://ingest.example', + timeout: 1000, + debug: false, + tlsRejectUnauthorized: true, + }); + await client.detect(request(tls)); + return body.request_metadata.tls_info; + } + + it('sends ALPN as `alpn`, the name the service reads', async () => { + const tls = await sent({ cipher_suites: [4865], alpn: ['h2', 'http/1.1'] }); + expect(tls.alpn).toEqual(['h2', 'http/1.1']); + expect(tls).not.toHaveProperty('alpn_protocols'); + }); + + it('sends the deprecated `alpn_protocols` input as `alpn`', async () => { + const tls = await sent({ cipher_suites: [4865], alpn_protocols: ['h2'] }); + expect(tls.alpn).toEqual(['h2']); + expect(tls).not.toHaveProperty('alpn_protocols'); + }); + + it('prefers `alpn` when both names are given', () => { + const wire = toWireDetectionRequest(request({ alpn: ['h2'], alpn_protocols: ['http/1.1'] })); + expect(wire.request_metadata.tls_info).toEqual({ alpn: ['h2'] }); + }); + + it('leaves a request without the deprecated name untouched', () => { + const req = request({ cipher_suites: [4865], alpn: ['h2'] }); + expect(toWireDetectionRequest(req)).toBe(req); + }); +}); diff --git a/packages/webdecoy/src/types.ts b/packages/webdecoy/src/types.ts index 5101c94..14c4713 100644 --- a/packages/webdecoy/src/types.ts +++ b/packages/webdecoy/src/types.ts @@ -164,7 +164,14 @@ export interface TLSInfo { /** Server name indication */ server_name?: string; - /** ALPN protocols */ + /** ALPN protocols offered by the client, in wire order (e.g. `['h2', 'http/1.1']`). */ + alpn?: string[]; + + /** + * @deprecated Use {@link TLSInfo.alpn}. The detection service reads `alpn` + * and ignores this name. It is still accepted here and is sent as `alpn` + * when `alpn` itself is not set. + */ alpn_protocols?: string[]; }