Repository navigation
Reuse HTTPS connections, negotiate HTTP/2, add new response fields - #55
Merged
Merged
Conversation
Every lookup used to call the global fetch with its default dispatcher, so a client had no connection pool of its own to control or close. IPData now creates an undici Agent together with the instance (allowH2 for HTTP/2 via ALPN with HTTP/1.1 fallback, 60 s idle timeout, 30 s HTTP/2 PING and TCP keepalive probes) and passes it to every fetch call, so TCP and TLS are negotiated once per client instead of once per request. A trailing, optional httpConfig constructor argument exposes http2, keepAliveTimeout and dispatcher (to inject a ProxyAgent, EnvHttpProxyAgent or MockAgent). close() releases an Agent the instance created and leaves an injected dispatcher alone. A request that fails before any response byte arrived because a pooled connection was reset or closed is retried exactly once; HTTP statuses are never retried. The bulk POST now also sends Accept: application/json. Tests replace the global.fetch patch with an undici MockAgent injected through httpConfig.dispatcher, and a loopback HTTP server proves that consecutive requests reuse one connection and that close() releases it. CI runs on pull requests and on Node 20, 22 and 24. undici ^7.22.0 needs Node >= 20.18.1 (lru-cache 11 already excluded Node 18), the version becomes 3.1.0 and the lock file is regenerated. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API now reports two more top-level booleans next to is_eu, is_satellite and is_anycast, each included only when it is true, and each threat.blocklists entry may carry first_seen and last_seen (YYYY-MM-DD strings, UTC) plus days_seen. Some feeds omit the three blocklist fields, so all of them are optional. LookupResponse declares the new keys as optional (undefined reads as false for the flags), the dates stay strings, and both flag names are accepted by the selectField and fields validation of lookup() and bulkLookup(). Responses are parsed JSON, so unknown keys already pass through unchanged. Tests add fixtures built from the API samples (a satellite ISP, an anycast address with dated blocklist entries, a VPN exit with an undated entry) for single and bulk lookups, and the README documents the five names. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IPDatanow creates an undiciAgenttogether with the instance and passes it as thedispatcherof everyfetchcall: one connection pool per client for its whole lifetime, HTTP/2 negotiated via ALPN (allowH2: true) with automatic HTTP/1.1 fallback, 60 s idle timeout for pooled connections (keepAliveTimeout,keepAliveMaxTimeout600 s), HTTP/2PINGevery 30 s (pingInterval: 30000; undici's default is 60 s) and TCP keepalive probes every 30 s (connect: { keepAliveInitialDelay: 30000 }; undici enables TCP keepalive by default with a 60 s initial delay).httpConfig?: HttpConfigwithdispatcher(inject aProxyAgent,EnvHttpProxyAgent,MockAgent, ...),http2(defaulttrue) andkeepAliveTimeout(milliseconds, default60000); newasync close()that closes anAgentthe instance created and leaves an injected dispatcher alone.ipdata.dispatcheris exposed read-only.ECONNRESET,EPIPE, undiciUND_ERR_SOCKET"other side closed" / "closed" / HTTP/2 GOAWAY) is retried exactly once on another connection; undici itself fails the head-of-line request without retrying. The retry lives in a shared privaterequest()helper, so it covers the non-idempotentPOST /bulkofbulkLookup()and the GET oflookup()alike. Whether the socket was reused cannot be observed throughfetch, so the retry keys on those error codes regardless of whether the connection was fresh. HTTP statuses are never retried, nor is anything after a response started.Accept: application/jsonis added to the bulk POST, the one request that already set headers.is_satellite?: booleanandis_anycast?: booleanonLookupResponsenext tois_eu;threat.blocklists[]entries gain optionalfirst_seen,last_seen(YYYY-MM-DDstrings, kept as strings) anddays_seen(number). Both flag names are accepted by theselectField/fieldsvalidation oflookup()andbulkLookup().global.fetch = jest.fn()patch is replaced by an undiciMockAgent(disableNetConnect, path/method/body interceptors,assertNoPendingInterceptorsafter every test,closeinafterAll) injected throughhttpConfig.dispatcher; a loopbackhttp.createServerproves that consecutive requests reuse one connection and thatclose()releases it; fixtures built from the three API sample responses cover the new fields for single and bulk lookups. 36 tests become 58.default.ymlgains apull_requesttrigger and a Node 20 / 22 / 24 matrix for the test job (lint still runs once on Node 22);release.ymlis unchanged.package.json: dependencyundici ^7.22.0(resolves to 7.30.1),engines.node>=20.18.1, version 3.1.0;package-lock.jsonregenerated withnpm installand verified withnpm cifrom a cleannode_modules.What changes for users
lru-cache11 already excluded Node 18). The package now depends onundiciand no longer uses the globalfetch.httpConfigis a trailing optional argument and the default is zero configuration. After the first lookup, further lookups on the same instance skip the TCP + TLS handshake and are multiplexed over HTTP/2 when api.ipdata.co negotiates it (HTTP/1.1 keep-alive otherwise).IPDataper process (or per API key or endpoint), keep it and reuse it for every call. Node runs JavaScript on one thread, so one instance can serve every request handler in the process;worker_threadsandclusterworkers each need their own instance. Idle pooled connections never keep the event loop alive, so scripts still exit on their own;await ipdata.close()releases the pool immediately at shutdown.httpConfig.dispatcherplugs in aProxyAgentorEnvHttpProxyAgentfor proxies and aMockAgentfor tests;http2: falseforces HTTP/1.1;keepAliveTimeouttunes the idle timeout. The last two only apply to the pool the library creates itself.lookup()andbulkLookup()alike and never to HTTP error statuses.Accept: application/jsonnext to itsContent-Type; GET requests set no headers, as before.is_satelliteandis_anycastaretruewhen present andundefinedwhen the API omits them (it includes each key only when it is true), so read them asinfo.is_satellite ?? false. Blocklist entries may carryfirst_seen,last_seenanddays_seen; some feeds, such as ipdata's own vpn list, omit all three.{ ...body, status }, and transport failures reject with undici'sTypeError: fetch failed(withcause) exactly as they did with Node's global fetch, which is undici too.Release notes
For tag
v3.1.0, which publishesipdata@3.1.0throughrelease.yml(git tag v3.1.0 && git push origin v3.1.0after merge):Verification
Local, Node v22.23.4, npm 10.9.9; everything below was run before each push.
rm -rf node_modules && npm ciundici@7.30.1from the regenerated lock filenpm run build(del lib && tsc)npm run lint(eslint --fix "src/**")import/no-named-as-defaulton the test file's import line, unchanged frommain)npm test(jest)Tests: 58 passed, 58 total(baseline onmain: 36 passed); the full suite was run three times in a row to check the loopback-server tests for flakinessTwo sequential lookups from the built library (
lib/ipdata.js) against api.ipdata.co negotiated HTTP/2 (ALPNh2) and the second reused the connection.CI: the
Lint and Testworkflow now runs onpull_request(trigger added in this PR). Both runs for headca9d09asucceeded at the first attempt, each with the jobslint,test (20),test (22)andtest (24): push run #31 https://github.com/ipdata/node/actions/runs/41621777396 (success) and pull_request run #32 https://github.com/ipdata/node/actions/runs/41621931406 (success). No follow-up commits were needed.Noticed, not changed
awaited lookups issued strictly back-to-back within one turn alternate between two pooled connections (never more), while any turn between them (the normal case in a server) reuses a single connection. Reproduced with plainundici.fetch/undici.requestand anAgenton undici 7.30.1; not caused by this library, and irrelevant over HTTP/2 where one connection is multiplexed. The loopback test asserts at most two connections for back-to-back calls and exactly one with a turn in between.ProxyAgent/EnvHttpProxyAgent, the ALPN list of the tunnelled TLS connection comes fromrequestTlsonly, so HTTP/2 through a proxy needsrequestTls: { allowH2: true };allowH2on theAgentitself is not enough. The README states thathttp2only applies to the library's own pool.CI(workflows/CI/badge.svg), but the workflow is calledLint and Test, so the badge is broken.LookupResponselags the API in places unrelated to this change:threatlacksis_vpnandscores,countis typednumberwhile the API returns a string ("0"),region_typeis missing, and nullable fields (city,time_zone.*, ...) are typed non-null.lookup(ip, selectField)orlookup(ip, undefined, fields)result is cached under the IP and returned by a later full lookup of the same IP, and vice versa. Pre-existing.ipdata-node/<version>User-Agent; undici's defaultuser-agent: undicigoes out.bulkLookup()callsisValidFieldsfor its side effect inside an emptyif(// validation only). Pre-existing.npm auditlists 16 pre-existing advisories, all in dev dependencies (eslint / jest tooling such as minimatch and picomatch); none in runtime dependencies.npm run lintrunseslint --fix, so lint rewrites files in place, also in CI.tsconfig.base.jsontargets ES2015 withlib: ["es2015"], so the source cannot useArray.prototype.includes,Object.entriesand similar; fine for Node 20+, but worth modernising separately.🤖 Generated with Claude Code