Repository navigation
Reuse HTTPS connections, add new response fields - #12
Merged
Merged
Conversation
Requests went through the default httpc profile with OTP's defaults (two
persistent connections per host, no explicit idle policy) and nothing in
the library owned their lifecycle. The ipdata application now owns an
httpc profile named `ipdata`: ipdata_app starts it before the supervisor
(accepting a profile that is already running), applies
ipdata_http:profile_options/0 ([{max_sessions, 16},
{max_keep_alive_length, 4}, {keep_alive_timeout, 60000}], overridable key
by key through the new `httpc_options` application environment key) and
stops it when the application stops. Both call sites use httpc:request/5
with that profile; when the profile is not running because the
application was never started, the request starts it and retries once,
so existing callers keep working. httpc only speaks HTTP/1.1, so the
saving is the TCP and TLS handshake, not multiplexing; ALPN is
deliberately not advertised.
Also: TLS 1.3 session tickets ({session_tickets, auto}) so a reconnect
after the idle timeout can resume the TLS session; the bulk POST is
retried exactly once on {error, socket_closed_remotely}, which httpc
hands back only when the server closed a connection that had not yet
served a response and after discarding that connection; the User-Agent
version is derived from the application vsn instead of a stale
hard-coded 1.0.0, and Accept: application/json is sent; the supervisor
doc no longer calls the library stateless.
httpc:request/5 is documented to return {error, {not_started, Profile}}
when the profile is down, but OTP releases up to and including 28 let the
gen_server noproc exit through (the catch clause names the wrong
exception class), so both shapes are handled.
Tests: the meck expectations on httpc:request are arity 5 with ipdata as
the profile; new cases cover the profile argument for get and post, the
not_started path in both shapes, the stale-connection retry (once, never
on an HTTP status), the User-Agent version, the TLS options and
profile_options/0 merging; a new ipdata_app_SUITE checks that the profile
follows the application lifecycle and that httpc_options are applied. CI
now runs on pull requests and pushes to main, with the publish job gated
on v* tags. Version 1.1.0.
Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API now returns two more top-level booleans, is_satellite and is_anycast (siblings of is_eu), and each threat.blocklists entry may carry first_seen and last_seen (YYYY-MM-DD, UTC) and days_seen. The API includes the two flags only when they are true, and some feeds (for example ipdata's own vpn feed) omit the three dated fields, so this untyped library keeps passing the decoded jsx maps through unchanged: callers read the flags with maps:get(Key, Result, false) and treat the dates as optional binaries that are not parsed. jsx tolerates unknown keys, so nothing in the decoder changes. The README documents the five fields with the present-only-when-true behaviour, and the fields parameter accepts the two new names like any other (the library does not validate field names). Tests: fixtures built from the API samples (a satellite ISP, an anycast address with dated blocklist entries, a VPN exit whose third blocklist entry has no dates) are decoded through ipdata_http:get/3 and post/4, asserting the values, the absent-means-false reading, that unknown keys and non-ASCII strings survive decoding and that bulk items carry the same shape; ipdata_SUITE checks that the fields pass through lookup unchanged and that lookup/3 and bulk/3 accept the two new field names. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first pull_request run of the workflow failed in erlef/setup-beam for OTP 25 before any step of ours ran: ubuntu-latest now resolves to Ubuntu 26.04, for which hex.pm publishes no OTP 25 builds (setup-beam's compatibility table lists OTP 26 and later for that image and 24.3 and later for ubuntu-24.04). OTP 26 and 27 passed compile, ct, dialyzer and xref on the new image and were only cancelled by the matrix fail-fast. Pin the OTP 25 matrix entry to ubuntu-24.04 and keep the other two on ubuntu-latest; the set of tested OTP versions is unchanged. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every request now goes through an
httpcprofile namedipdatathat the application owns and that keeps persistent HTTP/1.1 keep-alive connections to the API, so a TCP and TLS handshake is paid once per connection instead of once per request.ipdata_appstarts the profile before the supervisor (an already running profile is accepted), appliesipdata_http:profile_options/0([{max_sessions, 16}, {max_keep_alive_length, 4}, {keep_alive_timeout, 60000}], each key overridable through the newhttpc_optionsapplication environment key) and stops it with the application. Callers that never started the application keep working: a request on a profile that is not running starts it and retries once. TLS 1.3 resumption tickets are enabled so that a reconnect can resume the previous TLS state, the bulk POST is retried exactly once on{error, socket_closed_remotely}, the User-Agent reports the application version instead of a stale hard-coded1.0.0, andAccept: application/jsonis sent. ALPN is deliberately not advertised:httpconly speaks HTTP/1.1 and the server would otherwise select h2. HTTP/2 would needgunwith a connection-owner process and a new client lifecycle, which is out of scope (the README says so).Implementation notes:
inets:start(httpc, [{profile, ipdata}])acceptingalready_started, thenhttpc:set_options/2) lives in the new exportedipdata_http:start_profile/0, shared byipdata_app:start/2and the on-demand path. A failure to start or configure the profile makesapplication:start(ipdata)return{error, {httpc_profile, Reason}}.{error, {not_started, ipdata}}and theexit:{noproc, {gen_server, call, _}}thathttpc:request/5actually raises (see "Noticed, not changed").profile_options/0mergeshttpc_optionsover the defaults withlists:ukeymerge/3, so the list comes back sorted by key.ipdata_app_SUITE(no httpc mocking);ipdata_http_SUITEcovers the request path.Installationnow shows{ipdata, "~> 1.1"}.The second commit documents and tests the API's new response fields: top-level
is_satelliteandis_anycast(present only whentrue) and the optionalfirst_seen,last_seenanddays_seenon blocklist entries. The library is untyped and passes the decoded JSON through unchanged (jsx tolerates unknown keys, and no field-name validation exists), so no source change was needed for them.CI now runs the test suite on pull requests and on pushes to
main; the publish job is gated onv*tags. A follow-up commit,d0556cc, runs the OTP 25 matrix job onubuntu-24.04(see Verification).What changes for users
httpcexposes no setting for it. HTTP/2 is not available withhttpc, so the saving is the handshake, not multiplexing.application:ensure_all_started(ipdata)also starts theipdatahttpc profile;application:stop(ipdata)closes its connections. If the profile cannot be started or configured,application:start(ipdata)returns{error, {httpc_profile, Reason}}. Client maps stay plain immutable data that can be shared freely between processes; there is nothing to close per client. A request made before the application is started brings the profile up on demand (inetsandsslmust be running, as before).{ipdata, [{httpc_options, [...]}]}insys.config: anyhttpc:set_options/2option, for examplehttps_proxy.ipdata_http:start_profile/0andipdata_http:profile_options/0. No existing function, type or error shape changed:ipdata:new/1,2,lookup/1,2,3andbulk/2,3keep their signatures and still return{error, {request_failed | http_error | json_error | invalid_input, ...}}.erlang-ipdata/<application vsn>(erlang-ipdata/1.1.0for this release) andAccept: application/jsonis sent.is_satellite,is_anycastand dated blocklist entries; read the flags withmaps:get(Key, Result, false). The README documents all five fields.Release notes
Version 1.1.0. After merge, tag
v1.1.0onmain(git tag v1.1.0 && git push origin v1.1.0); the publish workflow runs on that tag.httpcprofile (ipdata) with HTTP/1.1 keep-alive, 16 persistent connections per host, a 60 s idle timeout and TLS 1.3 resumption tickets. Tune it with thehttpc_optionsapplication environment key. Starting the application starts the pool; stopping it closes the connections. A request made before the application is started brings the profile up on demand.Accept: application/jsonis sent.is_satelliteandis_anycastat the top level (present only whentrue), andfirst_seen,last_seen(YYYY-MM-DD, UTC) anddays_seenonthreat.blocklistsentries (optional, absent for some feeds).main.Verification
OTP 25.3.2.8, rebar3 3.19.0; run before each commit and before the push:
rebar3 compilerebar3 ctmain:All 24 tests passed; after commit 1:All 40 tests passed(ipdata_SUITE 16, ipdata_app_SUITE 4, ipdata_http_SUITE 20); after commit 2:All 47 tests passed(19 / 4 / 24)rebar3 xrefrebar3 dialyzer(error_handling,underspecs,unmatched_returns)There is no eunit suite and no formatter configured; the four commands above are exactly what the workflow runs.
Two
ipdata:lookup/2calls on theipdataprofile against api.ipdata.co were served over the same HTTP/1.1 keep-alive connection (httpcdoes not negotiate HTTP/2).CI, workflow "Publish to Hex.pm" (which now also runs the tests; the
pull_requesttrigger is added by this PR, so these are the first runs of the workflow on a pull request):3cfd4b3(41621703427): failure.Test (26)andTest (27)passed compile, ct, dialyzer and xref but were marked cancelled by the matrix fail-fast, becauseTest (25)failed insideerlef/setup-beam@v1before any step of ours ran:ubuntu-latestnow resolves to Ubuntu 26.04 and hex.pm publishes no OTP 25 builds for it (Requested loose Erlang/OTP version (25) not found in version list, at /builds/otp/amd64/ubuntu-26.04/builds.txt). The publish job was skipped, as intended on a non-tag event.d0556cc: the OTP 25 matrix entry runs onubuntu-24.04(setup-beam supports OTP 24.3+ there, 26+ on ubuntu-26.04); OTP 26 and 27 stay onubuntu-latest. The tested OTP set is unchanged.d0556cc(41622294991): success.Test (25, ubuntu-24.04),Test (26, ubuntu-latest)andTest (27, ubuntu-latest)each passed compile,rebar3 ct,rebar3 dialyzerandrebar3 xref; the publish job was skipped (not a tag).Noticed, not changed
httpc:request/5is documented to return{error, {not_started, Profile}}for a profile that is not running, buthttpc:handle_request/9catcheserror:{noproc, _}whilegen_server:callraisesexit:{noproc, _}, so the exit reaches the caller. Confirmed on OTP 25 and present in the OTP 26.0, 27.0 and 28.0 sources. The library handles both shapes; worth reporting upstream.httpcalready re-dispatches a request itself when a reused keep-alive connection is closed by the server before the response headers arrive (POSTs included);socket_closed_remotely, which this PR retries on, is only handed back for a connection that had not yet served a response, after httpc discarded it. Documented inipdata_http:post/4, nothing to change.rebar3-version: '3.24'pin and OTP 25/26/27 matrix were not touched.ifdid not skip the job; e6a2021 reverted to tag-only triggers as a side effect of a hex auth fix. The job-levelif: startsWith(github.ref, 'refs/tags/v')used here is the standard form: the publish job shows as skipped on non-tag events.city: null; illustrative only, left as is.fail-fast, so one failing OTP job cancels the others (that is why run 1 shows 26 and 27 as cancelled although every step had passed).fail-fast: falsewould report all three; not changed.actions/checkout@v4targets the deprecated Node.js 20 and is forced onto Node.js 24; still works, not updated.🤖 Generated with Claude Code