Skip to content

Reuse HTTPS connections, add new response fields - #12

Merged
jonathan-kosgei merged 3 commits into
mainfrom
http-reuse-and-new-fields
Nov 3, 2026
Merged

jonathan-kosgei merged 3 commits into
mainfrom
http-reuse-and-new-fields

Conversation

@jonathan-kosgei

@jonathan-kosgei jonathan-kosgei commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Every request now goes through an httpc profile named ipdata that the application owns and that keeps persistent HTTP/1.1 keep-alive connections to the API, so a TCP and TLS handshake is paid once per connection instead of once per request. ipdata_app starts the profile before the supervisor (an already running profile is accepted), applies ipdata_http:profile_options/0 ([{max_sessions, 16}, {max_keep_alive_length, 4}, {keep_alive_timeout, 60000}], each key overridable through the new httpc_options application environment key) and stops it with the application. Callers that never started the application keep working: a request on a profile that is not running starts it and retries once. TLS 1.3 resumption tickets are enabled so that a reconnect can resume the previous TLS state, the bulk POST is retried exactly once on {error, socket_closed_remotely}, the User-Agent reports the application version instead of a stale hard-coded 1.0.0, and Accept: application/json is sent. ALPN is deliberately not advertised: httpc only speaks HTTP/1.1 and the server would otherwise select h2. HTTP/2 would need gun with a connection-owner process and a new client lifecycle, which is out of scope (the README says so).

Implementation notes:

  • The start sequence (inets:start(httpc, [{profile, ipdata}]) accepting already_started, then httpc:set_options/2) lives in the new exported ipdata_http:start_profile/0, shared by ipdata_app:start/2 and the on-demand path. A failure to start or configure the profile makes application:start(ipdata) return {error, {httpc_profile, Reason}}.
  • The not-started path handles both {error, {not_started, ipdata}} and the exit:{noproc, {gen_server, call, _}} that httpc:request/5 actually raises (see "Noticed, not changed").
  • profile_options/0 merges httpc_options over the defaults with lists:ukeymerge/3, so the list comes back sorted by key.
  • The lifecycle tests are a separate ipdata_app_SUITE (no httpc mocking); ipdata_http_SUITE covers the request path.
  • README Installation now shows {ipdata, "~> 1.1"}.

The second commit documents and tests the API's new response fields: top-level is_satellite and is_anycast (present only when true) and the optional first_seen, last_seen and days_seen on blocklist entries. The library is untyped and passes the decoded JSON through unchanged (jsx tolerates unknown keys, and no field-name validation exists), so no source change was needed for them.

CI now runs the test suite on pull requests and on pushes to main; the publish job is gated on v* tags. A follow-up commit, d0556cc, runs the OTP 25 matrix job on ubuntu-24.04 (see Verification).

What changes for users

  • Lookups from one node reuse established HTTPS connections. Idle connections close after 60 s; TCP keepalive stays at the OTP defaults, as httpc exposes no setting for it. HTTP/2 is not available with httpc, so the saving is the handshake, not multiplexing.
  • application:ensure_all_started(ipdata) also starts the ipdata httpc profile; application:stop(ipdata) closes its connections. If the profile cannot be started or configured, application:start(ipdata) returns {error, {httpc_profile, Reason}}. Client maps stay plain immutable data that can be shared freely between processes; there is nothing to close per client. A request made before the application is started brings the profile up on demand (inets and ssl must be running, as before).
  • New optional tuning through {ipdata, [{httpc_options, [...]}]} in sys.config: any httpc:set_options/2 option, for example https_proxy.
  • Two new exported functions, ipdata_http:start_profile/0 and ipdata_http:profile_options/0. No existing function, type or error shape changed: ipdata:new/1,2, lookup/1,2,3 and bulk/2,3 keep their signatures and still return {error, {request_failed | http_error | json_error | invalid_input, ...}}.
  • Bulk lookups are retried once when the server closed the connection before answering; HTTP error statuses are never retried.
  • User-Agent is erlang-ipdata/<application vsn> (erlang-ipdata/1.1.0 for this release) and Accept: application/json is sent.
  • Responses may now contain is_satellite, is_anycast and dated blocklist entries; read the flags with maps:get(Key, Result, false). The README documents all five fields.

Release notes

Version 1.1.0. After merge, tag v1.1.0 on main (git tag v1.1.0 && git push origin v1.1.0); the publish workflow runs on that tag.

  • Persistent HTTPS connections: the application owns a dedicated httpc profile (ipdata) with HTTP/1.1 keep-alive, 16 persistent connections per host, a 60 s idle timeout and TLS 1.3 resumption tickets. Tune it with the httpc_options application environment key. Starting the application starts the pool; stopping it closes the connections. A request made before the application is started brings the profile up on demand.
  • Bulk lookups (POST) are retried exactly once when the server closed the connection before answering.
  • The User-Agent carries the real application version and Accept: application/json is sent.
  • New response fields, passed through unchanged: is_satellite and is_anycast at the top level (present only when true), and first_seen, last_seen (YYYY-MM-DD, UTC) and days_seen on threat.blocklists entries (optional, absent for some feeds).
  • CI runs on pull requests and pushes to main.

Verification

OTP 25.3.2.8, rebar3 3.19.0; run before each commit and before the push:

Command Result
rebar3 compile ok
rebar3 ct baseline on main: All 24 tests passed; after commit 1: All 40 tests passed (ipdata_SUITE 16, ipdata_app_SUITE 4, ipdata_http_SUITE 20); after commit 2: All 47 tests passed (19 / 4 / 24)
rebar3 xref no issues
rebar3 dialyzer (error_handling, underspecs, unmatched_returns) 0 warnings

There is no eunit suite and no formatter configured; the four commands above are exactly what the workflow runs.

Two ipdata:lookup/2 calls on the ipdata profile against api.ipdata.co were served over the same HTTP/1.1 keep-alive connection (httpc does not negotiate HTTP/2).

CI, workflow "Publish to Hex.pm" (which now also runs the tests; the pull_request trigger is added by this PR, so these are the first runs of the workflow on a pull request):

  • Run 1 on 3cfd4b3 (41621703427): failure. Test (26) and Test (27) passed compile, ct, dialyzer and xref but were marked cancelled by the matrix fail-fast, because Test (25) failed inside erlef/setup-beam@v1 before any step of ours ran: ubuntu-latest now resolves to Ubuntu 26.04 and hex.pm publishes no OTP 25 builds for it (Requested loose Erlang/OTP version (25) not found in version list, at /builds/otp/amd64/ubuntu-26.04/builds.txt). The publish job was skipped, as intended on a non-tag event.
  • Fix in the follow-up commit d0556cc: the OTP 25 matrix entry runs on ubuntu-24.04 (setup-beam supports OTP 24.3+ there, 26+ on ubuntu-26.04); OTP 26 and 27 stay on ubuntu-latest. The tested OTP set is unchanged.
  • Run 2 on d0556cc (41622294991): success. Test (25, ubuntu-24.04), Test (26, ubuntu-latest) and Test (27, ubuntu-latest) each passed compile, rebar3 ct, rebar3 dialyzer and rebar3 xref; the publish job was skipped (not a tag).

Noticed, not changed

  • OTP bug: httpc:request/5 is documented to return {error, {not_started, Profile}} for a profile that is not running, but httpc:handle_request/9 catches error:{noproc, _} while gen_server:call raises exit:{noproc, _}, so the exit reaches the caller. Confirmed on OTP 25 and present in the OTP 26.0, 27.0 and 28.0 sources. The library handles both shapes; worth reporting upstream.
  • httpc already re-dispatches a request itself when a reused keep-alive connection is closed by the server before the response headers arrive (POSTs included); socket_closed_remotely, which this PR retries on, is only handed back for a connection that had not yet served a response, after httpc discarded it. Documented in ipdata_http:post/4, nothing to change.
  • The workflow is still named "Publish to Hex.pm" although it now also serves as CI, and its rebar3-version: '3.24' pin and OTP 25/26/27 matrix were not touched.
  • Commits 97f4c4a and 880f61f once added push/pull_request triggers and then moved the publish guard to step level on the belief that a job-level if did not skip the job; e6a2021 reverted to tag-only triggers as a side effect of a hex auth fix. The job-level if: startsWith(github.ref, 'refs/tags/v') used here is the standard form: the publish job shows as skipped on non-tag events.
  • The README's "Look Up a Specific IP" example still illustrates 8.8.8.8 with Ashburn/Virginia values while the API's current sample for 8.8.8.8 has city: null; illustrative only, left as is.
  • The test matrix keeps GitHub's default fail-fast, so one failing OTP job cancels the others (that is why run 1 shows 26 and 27 as cancelled although every step had passed). fail-fast: false would report all three; not changed.
  • The runner warns that actions/checkout@v4 targets the deprecated Node.js 20 and is forced onto Node.js 24; still works, not updated.

🤖 Generated with Claude Code

Requests went through the default httpc profile with OTP's defaults (two
persistent connections per host, no explicit idle policy) and nothing in
the library owned their lifecycle. The ipdata application now owns an
httpc profile named `ipdata`: ipdata_app starts it before the supervisor
(accepting a profile that is already running), applies
ipdata_http:profile_options/0 ([{max_sessions, 16},
{max_keep_alive_length, 4}, {keep_alive_timeout, 60000}], overridable key
by key through the new `httpc_options` application environment key) and
stops it when the application stops. Both call sites use httpc:request/5
with that profile; when the profile is not running because the
application was never started, the request starts it and retries once,
so existing callers keep working. httpc only speaks HTTP/1.1, so the
saving is the TCP and TLS handshake, not multiplexing; ALPN is
deliberately not advertised.

Also: TLS 1.3 session tickets ({session_tickets, auto}) so a reconnect
after the idle timeout can resume the TLS session; the bulk POST is
retried exactly once on {error, socket_closed_remotely}, which httpc
hands back only when the server closed a connection that had not yet
served a response and after discarding that connection; the User-Agent
version is derived from the application vsn instead of a stale
hard-coded 1.0.0, and Accept: application/json is sent; the supervisor
doc no longer calls the library stateless.

httpc:request/5 is documented to return {error, {not_started, Profile}}
when the profile is down, but OTP releases up to and including 28 let the
gen_server noproc exit through (the catch clause names the wrong
exception class), so both shapes are handled.

Tests: the meck expectations on httpc:request are arity 5 with ipdata as
the profile; new cases cover the profile argument for get and post, the
not_started path in both shapes, the stale-connection retry (once, never
on an HTTP status), the User-Agent version, the TLS options and
profile_options/0 merging; a new ipdata_app_SUITE checks that the profile
follows the application lifecycle and that httpc_options are applied. CI
now runs on pull requests and pushes to main, with the publish job gated
on v* tags. Version 1.1.0.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API now returns two more top-level booleans, is_satellite and
is_anycast (siblings of is_eu), and each threat.blocklists entry may
carry first_seen and last_seen (YYYY-MM-DD, UTC) and days_seen. The API
includes the two flags only when they are true, and some feeds (for
example ipdata's own vpn feed) omit the three dated fields, so this
untyped library keeps passing the decoded jsx maps through unchanged:
callers read the flags with maps:get(Key, Result, false) and treat the
dates as optional binaries that are not parsed. jsx tolerates unknown
keys, so nothing in the decoder changes. The README documents the five
fields with the present-only-when-true behaviour, and the fields
parameter accepts the two new names like any other (the library does
not validate field names).

Tests: fixtures built from the API samples (a satellite ISP, an anycast
address with dated blocklist entries, a VPN exit whose third blocklist
entry has no dates) are decoded through ipdata_http:get/3 and post/4,
asserting the values, the absent-means-false reading, that unknown keys
and non-ASCII strings survive decoding and that bulk items carry the
same shape; ipdata_SUITE checks that the fields pass through lookup
unchanged and that lookup/3 and bulk/3 accept the two new field names.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first pull_request run of the workflow failed in erlef/setup-beam
for OTP 25 before any step of ours ran: ubuntu-latest now resolves to
Ubuntu 26.04, for which hex.pm publishes no OTP 25 builds (setup-beam's
compatibility table lists OTP 26 and later for that image and 24.3 and
later for ubuntu-24.04). OTP 26 and 27 passed compile, ct, dialyzer and
xref on the new image and were only cancelled by the matrix fail-fast.
Pin the OTP 25 matrix entry to ubuntu-24.04 and keep the other two on
ubuntu-latest; the set of tested OTP versions is unchanged.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jonathan-kosgei
jonathan-kosgei merged commit 3380d4f into main Nov 3, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants