Skip to content

Reuse HTTPS connections, negotiate HTTP/2, add new response fields - #55

Merged
jonathan-kosgei merged 2 commits into
mainfrom
http-reuse-and-new-fields
Nov 3, 2026
Merged

jonathan-kosgei merged 2 commits into
mainfrom
http-reuse-and-new-fields

Conversation

@jonathan-kosgei

@jonathan-kosgei jonathan-kosgei commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • IPData now creates an undici Agent together with the instance and passes it as the dispatcher of every fetch call: one connection pool per client for its whole lifetime, HTTP/2 negotiated via ALPN (allowH2: true) with automatic HTTP/1.1 fallback, 60 s idle timeout for pooled connections (keepAliveTimeout, keepAliveMaxTimeout 600 s), HTTP/2 PING every 30 s (pingInterval: 30000; undici's default is 60 s) and TCP keepalive probes every 30 s (connect: { keepAliveInitialDelay: 30000 }; undici enables TCP keepalive by default with a 60 s initial delay).
  • New optional trailing constructor argument httpConfig?: HttpConfig with dispatcher (inject a ProxyAgent, EnvHttpProxyAgent, MockAgent, ...), http2 (default true) and keepAliveTimeout (milliseconds, default 60000); new async close() that closes an Agent the instance created and leaves an injected dispatcher alone. ipdata.dispatcher is exposed read-only.
  • A request that fails before any response byte arrived because a pooled connection was reset or closed (ECONNRESET, EPIPE, undici UND_ERR_SOCKET "other side closed" / "closed" / HTTP/2 GOAWAY) is retried exactly once on another connection; undici itself fails the head-of-line request without retrying. The retry lives in a shared private request() helper, so it covers the non-idempotent POST /bulk of bulkLookup() and the GET of lookup() alike. Whether the socket was reused cannot be observed through fetch, so the retry keys on those error codes regardless of whether the connection was fresh. HTTP statuses are never retried, nor is anything after a response started.
  • Accept: application/json is added to the bulk POST, the one request that already set headers.
  • New response fields: is_satellite?: boolean and is_anycast?: boolean on LookupResponse next to is_eu; threat.blocklists[] entries gain optional first_seen, last_seen (YYYY-MM-DD strings, kept as strings) and days_seen (number). Both flag names are accepted by the selectField / fields validation of lookup() and bulkLookup().
  • Tests: the global.fetch = jest.fn() patch is replaced by an undici MockAgent (disableNetConnect, path/method/body interceptors, assertNoPendingInterceptors after every test, close in afterAll) injected through httpConfig.dispatcher; a loopback http.createServer proves that consecutive requests reuse one connection and that close() releases it; fixtures built from the three API sample responses cover the new fields for single and bulk lookups. 36 tests become 58.
  • CI: default.yml gains a pull_request trigger and a Node 20 / 22 / 24 matrix for the test job (lint still runs once on Node 22); release.yml is unchanged.
  • package.json: dependency undici ^7.22.0 (resolves to 7.30.1), engines.node >=20.18.1, version 3.1.0; package-lock.json regenerated with npm install and verified with npm ci from a clean node_modules.

What changes for users

  • Node.js 20.18.1 or later is required (undici 7 needs it; lru-cache 11 already excluded Node 18). The package now depends on undici and no longer uses the global fetch.
  • Every existing constructor and call signature keeps working; httpConfig is a trailing optional argument and the default is zero configuration. After the first lookup, further lookups on the same instance skip the TCP + TLS handshake and are multiplexed over HTTP/2 when api.ipdata.co negotiates it (HTTP/1.1 keep-alive otherwise).
  • Create one IPData per process (or per API key or endpoint), keep it and reuse it for every call. Node runs JavaScript on one thread, so one instance can serve every request handler in the process; worker_threads and cluster workers each need their own instance. Idle pooled connections never keep the event loop alive, so scripts still exit on their own; await ipdata.close() releases the pool immediately at shutdown.
  • httpConfig.dispatcher plugs in a ProxyAgent or EnvHttpProxyAgent for proxies and a MockAgent for tests; http2: false forces HTTP/1.1; keepAliveTimeout tunes the idle timeout. The last two only apply to the pool the library creates itself.
  • A request that fails because a pooled connection was reset or closed before any response byte arrived is retried once; this applies to lookup() and bulkLookup() alike and never to HTTP error statuses.
  • The bulk POST now sends Accept: application/json next to its Content-Type; GET requests set no headers, as before.
  • is_satellite and is_anycast are true when present and undefined when the API omits them (it includes each key only when it is true), so read them as info.is_satellite ?? false. Blocklist entries may carry first_seen, last_seen and days_seen; some feeds, such as ipdata's own vpn list, omit all three.
  • Error behaviour is unchanged: validation errors are thrown as before, HTTP errors still come back as { ...body, status }, and transport failures reject with undici's TypeError: fetch failed (with cause) exactly as they did with Node's global fetch, which is undici too.

Release notes

For tag v3.1.0, which publishes ipdata@3.1.0 through release.yml (git tag v3.1.0 && git push origin v3.1.0 after merge):

ipdata 3.1.0

Connection reuse and HTTP/2
- An IPData instance now keeps a persistent HTTPS connection pool (an undici Agent) for its whole
  lifetime: TCP and TLS are negotiated once per client instead of once per request, requests are
  multiplexed over HTTP/2 when api.ipdata.co negotiates it and fall back to HTTP/1.1 keep-alive
  otherwise. Idle connections are closed after 60 s; HTTP/2 PING and TCP keepalive probes run every 30 s.
- New optional fourth constructor argument httpConfig: { dispatcher, http2, keepAliveTimeout }.
  Inject your own undici dispatcher (ProxyAgent, EnvHttpProxyAgent, MockAgent), disable HTTP/2,
  or change the idle timeout.
- New close() method releases the pool at shutdown; a dispatcher you injected is left alone.
- A request that fails because the server dropped a pooled connection before any response byte
  arrived is retried once on a fresh connection (this covers the bulk POST); HTTP error statuses
  are never retried.
- Requires Node.js 20.18.1 or later; adds the undici dependency; the global fetch is no longer used.

New response fields
- is_satellite and is_anycast: top-level booleans next to is_eu, sent by the API only when true.
  TypeScript declares them optional, so read `info.is_satellite ?? false`.
- threat.blocklists entries may include first_seen and last_seen (YYYY-MM-DD, UTC) and days_seen;
  all three are optional because some feeds omit them.
- is_satellite and is_anycast are accepted by selectField and fields.

All existing constructor and method signatures are unchanged.

Verification

Local, Node v22.23.4, npm 10.9.9; everything below was run before each push.

Command Result
rm -rf node_modules && npm ci exit 0; installs undici@7.30.1 from the regenerated lock file
npm run build (del lib && tsc) exit 0
npm run lint (eslint --fix "src/**") 0 errors, 1 warning (pre-existing import/no-named-as-default on the test file's import line, unchanged from main)
npm test (jest) Tests: 58 passed, 58 total (baseline on main: 36 passed); the full suite was run three times in a row to check the loopback-server tests for flakiness

Two sequential lookups from the built library (lib/ipdata.js) against api.ipdata.co negotiated HTTP/2 (ALPN h2) and the second reused the connection.

CI: the Lint and Test workflow now runs on pull_request (trigger added in this PR). Both runs for head ca9d09a succeeded at the first attempt, each with the jobs lint, test (20), test (22) and test (24): push run #31 https://github.com/ipdata/node/actions/runs/41621777396 (success) and pull_request run #32 https://github.com/ipdata/node/actions/runs/41621931406 (success). No follow-up commits were needed.

Noticed, not changed

  • undici marks a kept-alive HTTP/1.1 connection reusable only on the event-loop turn after its response completed, so two awaited lookups issued strictly back-to-back within one turn alternate between two pooled connections (never more), while any turn between them (the normal case in a server) reuses a single connection. Reproduced with plain undici.fetch / undici.request and an Agent on undici 7.30.1; not caused by this library, and irrelevant over HTTP/2 where one connection is multiplexed. The loopback test asserts at most two connections for back-to-back calls and exactly one with a turn in between.
  • With undici 7.30.1's ProxyAgent / EnvHttpProxyAgent, the ALPN list of the tunnelled TLS connection comes from requestTls only, so HTTP/2 through a proxy needs requestTls: { allowH2: true }; allowH2 on the Agent itself is not enough. The README states that http2 only applies to the library's own pool.
  • The README badge points at a workflow named CI (workflows/CI/badge.svg), but the workflow is called Lint and Test, so the badge is broken.
  • LookupResponse lags the API in places unrelated to this change: threat lacks is_vpn and scores, count is typed number while the API returns a string ("0"), region_type is missing, and nullable fields (city, time_zone.*, ...) are typed non-null.
  • The cache key is the IP alone: a lookup(ip, selectField) or lookup(ip, undefined, fields) result is cached under the IP and returned by a later full lookup of the same IP, and vice versa. Pre-existing.
  • The two "should not cache the information" tests inspect the shared instance's cache instead of the custom-ttl instance's, so they pass trivially. Pre-existing, kept as is.
  • The library sends no ipdata-node/<version> User-Agent; undici's default user-agent: undici goes out.
  • bulkLookup() calls isValidFields for its side effect inside an empty if (// validation only). Pre-existing.
  • npm audit lists 16 pre-existing advisories, all in dev dependencies (eslint / jest tooling such as minimatch and picomatch); none in runtime dependencies.
  • npm run lint runs eslint --fix, so lint rewrites files in place, also in CI.
  • tsconfig.base.json targets ES2015 with lib: ["es2015"], so the source cannot use Array.prototype.includes, Object.entries and similar; fine for Node 20+, but worth modernising separately.

🤖 Generated with Claude Code

Every lookup used to call the global fetch with its default dispatcher,
so a client had no connection pool of its own to control or close. IPData
now creates an undici Agent together with the instance (allowH2 for
HTTP/2 via ALPN with HTTP/1.1 fallback, 60 s idle timeout, 30 s HTTP/2
PING and TCP keepalive probes) and passes it to every fetch call, so TCP
and TLS are negotiated once per client instead of once per request.

A trailing, optional httpConfig constructor argument exposes http2,
keepAliveTimeout and dispatcher (to inject a ProxyAgent,
EnvHttpProxyAgent or MockAgent). close() releases an Agent the instance
created and leaves an injected dispatcher alone. A request that fails
before any response byte arrived because a pooled connection was reset
or closed is retried exactly once; HTTP statuses are never retried. The
bulk POST now also sends Accept: application/json.

Tests replace the global.fetch patch with an undici MockAgent injected
through httpConfig.dispatcher, and a loopback HTTP server proves that
consecutive requests reuse one connection and that close() releases it.
CI runs on pull requests and on Node 20, 22 and 24. undici ^7.22.0 needs
Node >= 20.18.1 (lru-cache 11 already excluded Node 18), the version
becomes 3.1.0 and the lock file is regenerated.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API now reports two more top-level booleans next to is_eu,
is_satellite and is_anycast, each included only when it is true, and
each threat.blocklists entry may carry first_seen and last_seen
(YYYY-MM-DD strings, UTC) plus days_seen. Some feeds omit the three
blocklist fields, so all of them are optional.

LookupResponse declares the new keys as optional (undefined reads as
false for the flags), the dates stay strings, and both flag names are
accepted by the selectField and fields validation of lookup() and
bulkLookup(). Responses are parsed JSON, so unknown keys already pass
through unchanged. Tests add fixtures built from the API samples (a
satellite ISP, an anycast address with dated blocklist entries, a VPN
exit with an undated entry) for single and bulk lookups, and the README
documents the five names.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jonathan-kosgei
jonathan-kosgei merged commit 1aefdf3 into main Nov 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants