From f64e0961420aa77ab301d6221be24a18c9deb6cc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 10:37:25 +0000 Subject: [PATCH 1/3] Reuse HTTPS connections through a dedicated httpc profile Requests went through the default httpc profile with OTP's defaults (two persistent connections per host, no explicit idle policy) and nothing in the library owned their lifecycle. The ipdata application now owns an httpc profile named `ipdata`: ipdata_app starts it before the supervisor (accepting a profile that is already running), applies ipdata_http:profile_options/0 ([{max_sessions, 16}, {max_keep_alive_length, 4}, {keep_alive_timeout, 60000}], overridable key by key through the new `httpc_options` application environment key) and stops it when the application stops. Both call sites use httpc:request/5 with that profile; when the profile is not running because the application was never started, the request starts it and retries once, so existing callers keep working. httpc only speaks HTTP/1.1, so the saving is the TCP and TLS handshake, not multiplexing; ALPN is deliberately not advertised. Also: TLS 1.3 session tickets ({session_tickets, auto}) so a reconnect after the idle timeout can resume the TLS session; the bulk POST is retried exactly once on {error, socket_closed_remotely}, which httpc hands back only when the server closed a connection that had not yet served a response and after discarding that connection; the User-Agent version is derived from the application vsn instead of a stale hard-coded 1.0.0, and Accept: application/json is sent; the supervisor doc no longer calls the library stateless. httpc:request/5 is documented to return {error, {not_started, Profile}} when the profile is down, but OTP releases up to and including 28 let the gen_server noproc exit through (the catch clause names the wrong exception class), so both shapes are handled. Tests: the meck expectations on httpc:request are arity 5 with ipdata as the profile; new cases cover the profile argument for get and post, the not_started path in both shapes, the stale-connection retry (once, never on an HTTP status), the User-Agent version, the TLS options and profile_options/0 merging; a new ipdata_app_SUITE checks that the profile follows the application lifecycle and that httpc_options are applied. CI now runs on pull requests and pushes to main, with the publish job gated on v* tags. Version 1.1.0. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 --- .github/workflows/publish.yml | 4 + README.md | 27 ++++- src/ipdata.app.src | 4 +- src/ipdata_app.erl | 22 +++- src/ipdata_http.erl | 174 +++++++++++++++++++++++++-- src/ipdata_sup.erl | 10 +- test/ipdata_app_SUITE.erl | 91 ++++++++++++++ test/ipdata_http_SUITE.erl | 220 ++++++++++++++++++++++++++++++++-- 8 files changed, 519 insertions(+), 33 deletions(-) create mode 100644 test/ipdata_app_SUITE.erl diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 7dbe49f..34fe9ef 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2,8 +2,11 @@ name: Publish to Hex.pm on: push: + branches: + - main tags: - 'v*' + pull_request: jobs: test: @@ -42,6 +45,7 @@ jobs: publish: name: Publish to Hex.pm needs: test + if: startsWith(github.ref, 'refs/tags/v') runs-on: ubuntu-latest permissions: contents: read diff --git a/README.md b/README.md index 31224dc..a2e8944 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ Official Erlang client library for the [ipdata.co](https://ipdata.co) IP geoloca - Bulk lookups (up to 100 IPs per request) - Field filtering to reduce response size - EU endpoint support for data residency +- Persistent HTTPS connections (HTTP/1.1 keep-alive) through a dedicated `httpc` profile - Production-ready with proper TLS verification and error handling ## Requirements @@ -27,14 +28,14 @@ Add `ipdata` to your `rebar.config` dependencies: ```erlang {deps, [ - {ipdata, "~> 1.0"} + {ipdata, "~> 1.1"} ]}. ``` ## Quick Start ```erlang -%% Start the application +%% Start the application (it owns the pool of persistent connections) application:ensure_all_started(ipdata). %% Create a client with your API key @@ -137,6 +138,28 @@ case ipdata:lookup(Client, <<"8.8.8.8">>) of end. ``` +## Connection reuse + +The `ipdata` application owns a dedicated `httpc` profile, also named `ipdata`, that keeps persistent HTTPS connections (HTTP/1.1 keep-alive) to the API for as long as the application runs. Start the application once per node and keep it running: every lookup from every process then goes out on an already established connection instead of paying a new TCP and TLS handshake (roughly 660 to 980 ms for a cold connection to api.ipdata.co against 220 to 300 ms for a request on a reused one). Idle connections are closed after 60 seconds. HTTP/2 is not available with `httpc`, so the saving is the TLS handshake, not multiplexing; an HTTP/2 client such as `gun` would need a connection-owner process and a new client lifecycle, which is out of scope for this library. TLS 1.3 session tickets are enabled so that a connection opened after the idle timeout can resume the previous TLS session. + +A client map returned by `ipdata:new/1,2` is plain immutable data: it holds no process, socket or other resource, so it can be shared freely between processes, kept in application state or rebuilt at any time, and there is nothing to close. `application:stop(ipdata)` stops the profile and closes its connections. If a request is made before the application has been started, the profile is started on demand (`inets` and `ssl` must be running). + +The profile is tuned through the `httpc_options` key of the `ipdata` application environment, for example in `sys.config`: + +```erlang +[{ipdata, [ + {httpc_options, [ + {max_sessions, 16}, %% persistent connections per host (default 16) + {max_keep_alive_length, 4}, %% outstanding requests per connection (default 4) + {keep_alive_timeout, 60000} %% close a connection idle for this long, in ms (default 60000) + ]} +]}]. +``` + +Each entry overrides the default of the same name, and any other `httpc:set_options/2` option (for example `https_proxy`) can be added. The options are applied when the application starts. + +A bulk lookup is a POST, so it is retried exactly once if the server closed the connection before answering (`socket_closed_remotely`); `httpc` has already discarded that connection, so the retry goes out on a fresh one. HTTP error statuses are never retried. + ## API Reference ### `ipdata:new/1,2` diff --git a/src/ipdata.app.src b/src/ipdata.app.src index b59c6e7..d4f2076 100644 --- a/src/ipdata.app.src +++ b/src/ipdata.app.src @@ -1,6 +1,6 @@ {application, ipdata, [ {description, "Erlang client library for the ipdata.co IP geolocation API"}, - {vsn, "1.0.1"}, + {vsn, "1.1.0"}, {registered, [ipdata_sup]}, {mod, {ipdata_app, []}}, {applications, [ @@ -11,7 +11,7 @@ public_key, jsx ]}, - {env, []}, + {env, [{httpc_options, []}]}, {modules, []}, {licenses, ["MIT"]}, {links, [ diff --git a/src/ipdata_app.erl b/src/ipdata_app.erl index 39180ab..41b0b4b 100644 --- a/src/ipdata_app.erl +++ b/src/ipdata_app.erl @@ -1,8 +1,12 @@ %%%------------------------------------------------------------------- %%% @doc Application behaviour callback for the ipdata library. %%% -%%% Ensures that required OTP applications (`inets', `ssl') are -%%% started and launches the top-level supervisor. +%%% The application depends on `inets', `ssl' and `public_key', which +%%% `application:ensure_all_started(ipdata)' starts first. On start it +%%% brings up the `ipdata' httpc profile that holds the persistent +%%% connections used by every request (see +%%% `ipdata_http:start_profile/0'), then launches the top-level +%%% supervisor. On stop the profile and its connections are closed. %%% @end %%%------------------------------------------------------------------- -module(ipdata_app). @@ -13,17 +17,29 @@ %%-------------------------------------------------------------------- %% @doc Start the ipdata application. +%% +%% Starts the `ipdata' httpc profile (accepting one that is already +%% running) and applies `ipdata_http:profile_options/0' before the +%% supervisor is started. %% @end %%-------------------------------------------------------------------- -spec start(application:start_type(), term()) -> {ok, pid()} | {error, term()}. start(_StartType, _StartArgs) -> - ipdata_sup:start_link(). + case ipdata_http:start_profile() of + ok -> + ipdata_sup:start_link(); + {error, Reason} -> + {error, {httpc_profile, Reason}} + end. %%-------------------------------------------------------------------- %% @doc Stop the ipdata application. +%% +%% Stops the `ipdata' httpc profile, closing its connections. %% @end %%-------------------------------------------------------------------- -spec stop(term()) -> ok. stop(_State) -> + _ = inets:stop(httpc, ipdata), ok. diff --git a/src/ipdata_http.erl b/src/ipdata_http.erl index 5608d35..dab97c5 100644 --- a/src/ipdata_http.erl +++ b/src/ipdata_http.erl @@ -3,15 +3,44 @@ %%% %%% Wraps OTP's `httpc' module to provide GET and POST requests %%% with proper TLS configuration, JSON parsing, and error handling. +%%% +%%% Every request goes through a dedicated `httpc' profile named +%%% `ipdata' that keeps persistent HTTP/1.1 keep-alive connections to +%%% the API, so the TCP and TLS handshake is paid once per connection +%%% instead of once per request. The profile is started by +%%% `ipdata_app' when the application starts; if the application was +%%% never started, the first request starts it on demand. %%% @end %%%------------------------------------------------------------------- -module(ipdata_http). -export([get/3, post/4]). +-export([start_profile/0, profile_options/0]). + +%% Name of the httpc profile owned by the ipdata application. +-define(PROFILE, ipdata). + +%% Default profile options, overridable through the `httpc_options' +%% application environment key (see profile_options/0). +-define(DEFAULT_PROFILE_OPTIONS, + [{max_sessions, 16}, + {max_keep_alive_length, 4}, + {keep_alive_timeout, 60000}]). --define(USER_AGENT, "erlang-ipdata/1.0.0"). +-define(USER_AGENT_PREFIX, "erlang-ipdata/"). +-define(UNKNOWN_VSN, "unknown"). -define(CONTENT_TYPE_JSON, "application/json"). +-type method() :: get | post. +-type headers() :: [{string(), string()}]. +%% The two httpc request shapes used here: without and with a body. +-type request() :: {string(), headers()} + | {string(), headers(), string(), binary()}. +-type http_options() :: [{timeout, pos_integer()} + | {connect_timeout, pos_integer()} + | {ssl, [ssl:tls_client_option()]}]. +-type request_options() :: [{body_format, binary}]. + %%-------------------------------------------------------------------- %% @doc Perform an HTTP GET request. %% @@ -23,12 +52,11 @@ Timeout :: pos_integer()) -> {ok, jsx:json_term()} | {error, term()}. get(URL, Headers, Timeout) -> - AllHeaders = [{"User-Agent", ?USER_AGENT} | Headers], + AllHeaders = [{"User-Agent", user_agent()}, + {"Accept", ?CONTENT_TYPE_JSON} | Headers], HttpOpts = http_options(Timeout), Opts = [{body_format, binary}], - handle_response( - httpc:request(get, {URL, AllHeaders}, HttpOpts, Opts) - ). + handle_response(request(get, {URL, AllHeaders}, HttpOpts, Opts)). %%-------------------------------------------------------------------- %% @doc Perform an HTTP POST request with a JSON body. @@ -36,39 +64,161 @@ get(URL, Headers, Timeout) -> %% The body term is encoded to JSON before sending. Returns the %% decoded JSON response body on success, or an error tuple on %% failure. +%% +%% A POST is not idempotent, so `httpc' hands a request back with +%% `{error, socket_closed_remotely}' when the server closed the +%% connection before answering it. That request is retried exactly +%% once; `httpc' has already discarded the closed connection, so the +%% retry goes out on a fresh one. HTTP error statuses are never +%% retried. %% @end %%-------------------------------------------------------------------- -spec post(URL :: string(), Headers :: [{string(), string()}], Body :: jsx:json_term(), Timeout :: pos_integer()) -> {ok, jsx:json_term()} | {error, term()}. post(URL, Headers, Body, Timeout) -> - AllHeaders = [{"User-Agent", ?USER_AGENT} | Headers], + AllHeaders = [{"User-Agent", user_agent()}, + {"Accept", ?CONTENT_TYPE_JSON} | Headers], HttpOpts = http_options(Timeout), Opts = [{body_format, binary}], EncodedBody = jsx:encode(Body), - handle_response( - httpc:request(post, - {URL, AllHeaders, ?CONTENT_TYPE_JSON, EncodedBody}, - HttpOpts, Opts) - ). + Request = {URL, AllHeaders, ?CONTENT_TYPE_JSON, EncodedBody}, + handle_response(post_with_stale_retry(Request, HttpOpts, Opts)). + +%%-------------------------------------------------------------------- +%% @doc Start the `ipdata' httpc profile and apply profile_options/0. +%% +%% The profile is started under the `inets' supervision tree, so +%% `inets' must be running. An already running profile is accepted +%% and its options are (re)applied, which makes the function safe to +%% call more than once. +%% @end +%%-------------------------------------------------------------------- +-spec start_profile() -> ok | {error, term()}. +start_profile() -> + case inets:start(httpc, [{profile, ?PROFILE}]) of + {ok, _Pid} -> + httpc:set_options(profile_options(), ?PROFILE); + {error, {already_started, _Pid}} -> + httpc:set_options(profile_options(), ?PROFILE); + {error, _Reason} = Error -> + Error + end. + +%%-------------------------------------------------------------------- +%% @doc Options applied to the `ipdata' httpc profile. +%% +%% The defaults keep up to 16 persistent connections per host, queue +%% at most 4 requests on each of them, and close a connection after +%% 60 seconds idle: +%% +%% ``` +%% [{max_sessions, 16}, {max_keep_alive_length, 4}, {keep_alive_timeout, 60000}] +%% ''' +%% +%% Each entry can be overridden, and further `httpc:set_options/2' +%% options added, through the `httpc_options' key of the `ipdata' +%% application environment: +%% +%% ``` +%% {ipdata, [{httpc_options, [{max_sessions, 4}, +%% {https_proxy, {{"proxy.example.com", 3128}, []}}]}]} +%% ''' +%% @end +%%-------------------------------------------------------------------- +-spec profile_options() -> [{atom(), term()}]. +profile_options() -> + Overrides = application:get_env(ipdata, httpc_options, []), + lists:ukeymerge(1, lists:ukeysort(1, Overrides), + lists:ukeysort(1, ?DEFAULT_PROFILE_OPTIONS)). %%-------------------------------------------------------------------- %% Internal functions %%-------------------------------------------------------------------- +%% Issue a request through the profile, starting the profile on demand +%% when the application was never started (or has been stopped) and +%% retrying the request once. +-spec request(method(), request(), http_options(), request_options()) -> + {ok, term()} | {error, term()}. +request(Method, Request, HttpOpts, Opts) -> + case profile_request(Method, Request, HttpOpts, Opts) of + {error, {not_started, ?PROFILE}} -> + case start_profile() of + ok -> + profile_request(Method, Request, HttpOpts, Opts); + {error, _Reason} = Error -> + Error + end; + Result -> + Result + end. + +%% Retry a POST exactly once when the server closed the connection +%% before any response arrived (see post/4). +-spec post_with_stale_retry(request(), http_options(), request_options()) -> + {ok, term()} | {error, term()}. +post_with_stale_retry(Request, HttpOpts, Opts) -> + case request(post, Request, HttpOpts, Opts) of + {error, socket_closed_remotely} -> + request(post, Request, HttpOpts, Opts); + Result -> + Result + end. + +%% httpc:request/5 is documented to return {error, {not_started, Profile}} +%% when the profile is not running, but OTP releases up to and including +%% 28 catch the wrong exception class (error instead of exit) and let the +%% gen_server noproc exit through instead. Normalise both shapes. +-spec profile_request(method(), request(), http_options(), request_options()) -> + {ok, term()} | {error, term()}. +profile_request(Method, Request, HttpOpts, Opts) -> + try + httpc:request(Method, Request, HttpOpts, Opts, ?PROFILE) + catch + exit:{noproc, {gen_server, call, _}} -> + {error, {not_started, ?PROFILE}} + end. + +%% The User-Agent carries the version of the running ipdata application +%% rather than a hard-coded string, so it cannot go stale. +-spec user_agent() -> nonempty_string(). +user_agent() -> + ?USER_AGENT_PREFIX ++ app_vsn(). + +-spec app_vsn() -> string(). +app_vsn() -> + case application:get_key(ipdata, vsn) of + {ok, Vsn} when is_list(Vsn) -> + Vsn; + _ -> + %% Not loaded yet (the application was never started): + %% loading it is cheap and makes the version available. + _ = application:load(ipdata), + case application:get_key(ipdata, vsn) of + {ok, Vsn} when is_list(Vsn) -> Vsn; + _ -> ?UNKNOWN_VSN + end + end. + -spec http_options(Timeout :: pos_integer()) -> list(). http_options(Timeout) -> [{timeout, Timeout}, {connect_timeout, Timeout}, {ssl, ssl_options()}]. +%% TLS 1.3 session tickets let a connection opened after the idle +%% timeout resume the previous TLS session instead of doing a full +%% handshake. ALPN is deliberately not advertised: httpc only speaks +%% HTTP/1.1, and offering h2 would make the server select it. -spec ssl_options() -> [ssl:tls_client_option()]. ssl_options() -> [{verify, verify_peer}, {depth, 100}, {cacerts, public_key:cacerts_get()}, {customize_hostname_check, - [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}]. + [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}, + {session_tickets, auto}]. -spec handle_response({ok, term()} | {error, term()}) -> {ok, jsx:json_term()} | {error, term()}. diff --git a/src/ipdata_sup.erl b/src/ipdata_sup.erl index f1084e1..2f4a984 100644 --- a/src/ipdata_sup.erl +++ b/src/ipdata_sup.erl @@ -1,10 +1,12 @@ %%%------------------------------------------------------------------- %%% @doc Top-level supervisor for the ipdata application. %%% -%%% This is a minimal supervisor with no children. The ipdata library -%%% is stateless β€” all state is held in the client map returned by -%%% `ipdata:new/1,2'. The supervisor exists to satisfy OTP application -%%% conventions. +%%% This is a minimal supervisor with no children. The client map +%%% returned by `ipdata:new/1,2' is plain immutable data, and the only +%%% process state the library relies on is the `ipdata' httpc profile +%%% holding the persistent connections, which lives in the `inets' +%%% supervision tree and is started and stopped by `ipdata_app'. The +%%% supervisor exists to satisfy OTP application conventions. %%% @end %%%------------------------------------------------------------------- -module(ipdata_sup). diff --git a/test/ipdata_app_SUITE.erl b/test/ipdata_app_SUITE.erl new file mode 100644 index 0000000..0d6040d --- /dev/null +++ b/test/ipdata_app_SUITE.erl @@ -0,0 +1,91 @@ +%%%------------------------------------------------------------------- +%%% @doc Common Test suite for the ipdata application lifecycle. +%%% +%%% Starts and stops the real application (no network access is +%%% needed: no request is made) and checks that the `ipdata' httpc +%%% profile holding the persistent connections follows it. +%%% @end +%%%------------------------------------------------------------------- +-module(ipdata_app_SUITE). + +-include_lib("common_test/include/ct.hrl"). +-include_lib("stdlib/include/assert.hrl"). + +-compile(export_all). + +%%-------------------------------------------------------------------- +%% Suite setup +%%-------------------------------------------------------------------- + +all() -> + [start_creates_profile_with_options, + start_accepts_already_started_profile, + httpc_options_env_applied_on_start, + stop_removes_profile]. + +init_per_suite(Config) -> + {ok, _} = application:ensure_all_started(inets), + _ = application:load(ipdata), + Config. + +end_per_suite(_Config) -> + ok. + +init_per_testcase(_TestCase, Config) -> + reset(), + Config. + +end_per_testcase(_TestCase, _Config) -> + reset(), + ok. + +%%-------------------------------------------------------------------- +%% Test cases +%%-------------------------------------------------------------------- + +start_creates_profile_with_options(_Config) -> + ?assertEqual({error, {not_started, ipdata}}, httpc:info(ipdata)), + {ok, Started} = application:ensure_all_started(ipdata), + ?assert(lists:member(ipdata, Started)), + {ok, Opts} = httpc:get_options([max_sessions, max_keep_alive_length, + keep_alive_timeout], ipdata), + ?assertEqual(16, proplists:get_value(max_sessions, Opts)), + ?assertEqual(4, proplists:get_value(max_keep_alive_length, Opts)), + ?assertEqual(60000, proplists:get_value(keep_alive_timeout, Opts)), + ?assertEqual(lists:sort(ipdata_http:profile_options()), lists:sort(Opts)), + %% nothing has been requested yet: no connection + ?assertEqual({[], [], []}, httpc:which_sessions(ipdata)). + +start_accepts_already_started_profile(_Config) -> + {ok, _} = inets:start(httpc, [{profile, ipdata}]), + {ok, _} = application:ensure_all_started(ipdata), + ?assertEqual({ok, [{max_sessions, 16}]}, + httpc:get_options([max_sessions], ipdata)). + +httpc_options_env_applied_on_start(_Config) -> + ok = application:set_env(ipdata, httpc_options, + [{max_sessions, 3}, {keep_alive_timeout, 1000}]), + {ok, _} = application:ensure_all_started(ipdata), + {ok, Opts} = httpc:get_options([max_sessions, max_keep_alive_length, + keep_alive_timeout], ipdata), + ?assertEqual(3, proplists:get_value(max_sessions, Opts)), + ?assertEqual(1000, proplists:get_value(keep_alive_timeout, Opts)), + ?assertEqual(4, proplists:get_value(max_keep_alive_length, Opts)). + +stop_removes_profile(_Config) -> + {ok, _} = application:ensure_all_started(ipdata), + ?assertMatch({ok, _}, httpc:get_options([max_sessions], ipdata)), + ok = application:stop(ipdata), + ?assertEqual({error, {not_started, ipdata}}, httpc:info(ipdata)), + %% and the application can be started again + {ok, _} = application:ensure_all_started(ipdata), + ?assertMatch({ok, _}, httpc:get_options([max_sessions], ipdata)). + +%%-------------------------------------------------------------------- +%% Helpers +%%-------------------------------------------------------------------- + +reset() -> + _ = application:stop(ipdata), + _ = inets:stop(httpc, ipdata), + ok = application:unset_env(ipdata, httpc_options). diff --git a/test/ipdata_http_SUITE.erl b/test/ipdata_http_SUITE.erl index 51aea91..6d1e504 100644 --- a/test/ipdata_http_SUITE.erl +++ b/test/ipdata_http_SUITE.erl @@ -22,12 +22,28 @@ all() -> get_api_error_with_message, get_api_error_without_json, get_network_error, + get_uses_ipdata_profile, + get_sends_user_agent_with_app_vsn, + get_tls_options, post_success, post_api_error, - post_network_error]. + post_network_error, + post_uses_ipdata_profile, + post_retries_once_on_socket_closed_remotely, + post_retries_only_once, + post_does_not_retry_http_errors, + not_started_starts_profile_and_retries, + not_started_noproc_exit_starts_profile_and_retries, + not_started_profile_start_failure_is_request_failed, + profile_options_defaults, + profile_options_env_overrides]. init_per_suite(Config) -> - application:ensure_all_started(meck), + {ok, _} = application:ensure_all_started(inets), + {ok, _} = application:ensure_all_started(meck), + %% Load (not start) the application so that its environment and + %% version are available to the code under test. + _ = application:load(ipdata), Config. end_per_suite(_Config) -> @@ -48,7 +64,7 @@ end_per_testcase(_TestCase, _Config) -> get_success(_Config) -> Body = jsx:encode(#{<<"ip">> => <<"1.2.3.4">>}), meck:expect(httpc, request, - fun(get, {_URL, _Headers}, _HttpOpts, _Opts) -> + fun(get, {_URL, _Headers}, _HttpOpts, _Opts, ipdata) -> {ok, {{"HTTP/1.1", 200, "OK"}, [], Body}} end), {ok, Result} = ipdata_http:get("https://api.ipdata.co?api-key=test", [], 5000), @@ -57,7 +73,7 @@ get_success(_Config) -> get_json_error(_Config) -> meck:expect(httpc, request, - fun(get, {_URL, _Headers}, _HttpOpts, _Opts) -> + fun(get, {_URL, _Headers}, _HttpOpts, _Opts, ipdata) -> {ok, {{"HTTP/1.1", 200, "OK"}, [], <<"not json">>}} end), ?assertMatch({error, {json_error, _}}, ipdata_http:get("https://example.com", [], 5000)), @@ -66,7 +82,7 @@ get_json_error(_Config) -> get_api_error_with_message(_Config) -> Body = jsx:encode(#{<<"message">> => <<"Invalid API key">>}), meck:expect(httpc, request, - fun(get, {_URL, _Headers}, _HttpOpts, _Opts) -> + fun(get, {_URL, _Headers}, _HttpOpts, _Opts, ipdata) -> {ok, {{"HTTP/1.1", 401, "Unauthorized"}, [], Body}} end), ?assertMatch({error, {http_error, 401, <<"Invalid API key">>}}, @@ -75,7 +91,7 @@ get_api_error_with_message(_Config) -> get_api_error_without_json(_Config) -> meck:expect(httpc, request, - fun(get, {_URL, _Headers}, _HttpOpts, _Opts) -> + fun(get, {_URL, _Headers}, _HttpOpts, _Opts, ipdata) -> {ok, {{"HTTP/1.1", 500, "Server Error"}, [], <<"Internal Server Error">>}} end), @@ -85,7 +101,7 @@ get_api_error_without_json(_Config) -> get_network_error(_Config) -> meck:expect(httpc, request, - fun(get, {_URL, _Headers}, _HttpOpts, _Opts) -> + fun(get, {_URL, _Headers}, _HttpOpts, _Opts, ipdata) -> {error, {failed_connect, [{to_address, {"api.ipdata.co", 443}}, {inet, [inet], timeout}]}} @@ -94,6 +110,42 @@ get_network_error(_Config) -> ipdata_http:get("https://example.com", [], 5000)), ?assert(meck:validate(httpc)). +%% Every request goes through httpc:request/5 with the ipdata profile. +get_uses_ipdata_profile(_Config) -> + expect_response(get, 200, jsx:encode(#{})), + {ok, _} = ipdata_http:get("https://api.ipdata.co?api-key=test", [], 5000), + ?assertEqual(1, meck:num_calls(httpc, request, '_')), + ?assert(meck:called(httpc, request, [get, '_', '_', '_', ipdata])), + ?assert(meck:validate(httpc)). + +%% The User-Agent carries the application version from the .app file. +get_sends_user_agent_with_app_vsn(_Config) -> + {ok, Vsn} = application:get_key(ipdata, vsn), + ?assert(is_list(Vsn) andalso Vsn =/= []), + expect_response(get, 200, jsx:encode(#{})), + {ok, _} = ipdata_http:get("https://api.ipdata.co?api-key=test", + [{"X-Extra", "1"}], 5000), + {_URL, Headers} = meck:capture(first, httpc, request, '_', 2), + ?assertEqual("erlang-ipdata/" ++ Vsn, proplists:get_value("User-Agent", Headers)), + ?assertEqual("application/json", proplists:get_value("Accept", Headers)), + %% caller-supplied headers are kept + ?assertEqual("1", proplists:get_value("X-Extra", Headers)), + ?assert(meck:validate(httpc)). + +%% TLS verification stays on, TLS 1.3 session tickets are enabled and +%% ALPN is never advertised (httpc only speaks HTTP/1.1). +get_tls_options(_Config) -> + expect_response(get, 200, jsx:encode(#{})), + {ok, _} = ipdata_http:get("https://api.ipdata.co?api-key=test", [], 7000), + HttpOpts = meck:capture(first, httpc, request, '_', 3), + ?assertEqual(7000, proplists:get_value(timeout, HttpOpts)), + ?assertEqual(7000, proplists:get_value(connect_timeout, HttpOpts)), + {ssl, SslOpts} = lists:keyfind(ssl, 1, HttpOpts), + ?assertEqual(verify_peer, proplists:get_value(verify, SslOpts)), + ?assertEqual(auto, proplists:get_value(session_tickets, SslOpts)), + ?assertNot(lists:keymember(alpn_advertised_protocols, 1, SslOpts)), + ?assert(meck:validate(httpc)). + %%-------------------------------------------------------------------- %% POST tests %%-------------------------------------------------------------------- @@ -103,7 +155,7 @@ post_success(_Config) -> #{<<"ip">> => <<"1.1.1.1">>}]), meck:expect(httpc, request, fun(post, {_URL, _Headers, "application/json", ReqBody}, - _HttpOpts, _Opts) -> + _HttpOpts, _Opts, ipdata) -> %% Verify the request body is valid JSON _Decoded = jsx:decode(ReqBody, [return_maps]), {ok, {{"HTTP/1.1", 200, "OK"}, [], ResponseBody}} @@ -116,7 +168,7 @@ post_success(_Config) -> post_api_error(_Config) -> Body = jsx:encode(#{<<"message">> => <<"Forbidden">>}), meck:expect(httpc, request, - fun(post, {_URL, _Headers, _CT, _Body}, _HttpOpts, _Opts) -> + fun(post, {_URL, _Headers, _CT, _Body}, _HttpOpts, _Opts, ipdata) -> {ok, {{"HTTP/1.1", 403, "Forbidden"}, [], Body}} end), ?assertMatch({error, {http_error, 403, <<"Forbidden">>}}, @@ -126,10 +178,158 @@ post_api_error(_Config) -> post_network_error(_Config) -> meck:expect(httpc, request, - fun(post, {_URL, _Headers, _CT, _Body}, _HttpOpts, _Opts) -> + fun(post, {_URL, _Headers, _CT, _Body}, _HttpOpts, _Opts, ipdata) -> {error, timeout} end), ?assertMatch({error, {request_failed, timeout}}, ipdata_http:post("https://example.com/bulk", [], [<<"8.8.8.8">>], 5000)), + %% a timeout is not a stale connection: no retry + ?assertEqual(1, meck:num_calls(httpc, request, '_')), ?assert(meck:validate(httpc)). + +post_uses_ipdata_profile(_Config) -> + expect_response(post, 200, jsx:encode([#{<<"ip">> => <<"8.8.8.8">>}])), + {ok, _} = ipdata_http:post("https://api.ipdata.co/bulk?api-key=test", + [], [<<"8.8.8.8">>], 5000), + ?assertEqual(1, meck:num_calls(httpc, request, '_')), + ?assert(meck:called(httpc, request, [post, '_', '_', '_', ipdata])), + ?assert(meck:validate(httpc)). + +%% A POST that httpc hands back because the server closed the connection +%% before answering is retried exactly once, with the same request. +post_retries_once_on_socket_closed_remotely(_Config) -> + ResponseBody = jsx:encode([#{<<"ip">> => <<"8.8.8.8">>}]), + meck:expect(httpc, request, 5, + meck:seq([{error, socket_closed_remotely}, + {ok, {{"HTTP/1.1", 200, "OK"}, [], ResponseBody}}])), + {ok, Results} = ipdata_http:post("https://api.ipdata.co/bulk?api-key=test", + [], [<<"8.8.8.8">>], 5000), + ?assertEqual(1, length(Results)), + ?assertEqual(2, meck:num_calls(httpc, request, [post, '_', '_', '_', ipdata])), + [First, Second] = request_args(), + ?assertEqual(First, Second), + ?assert(meck:validate(httpc)). + +post_retries_only_once(_Config) -> + meck:expect(httpc, request, 5, {error, socket_closed_remotely}), + ?assertEqual({error, {request_failed, socket_closed_remotely}}, + ipdata_http:post("https://example.com/bulk", [], + [<<"8.8.8.8">>], 5000)), + ?assertEqual(2, meck:num_calls(httpc, request, '_')), + ?assert(meck:validate(httpc)). + +post_does_not_retry_http_errors(_Config) -> + expect_response(post, 502, <<"Bad Gateway">>), + ?assertEqual({error, {http_error, 502, <<"Bad Gateway">>}}, + ipdata_http:post("https://example.com/bulk", [], + [<<"8.8.8.8">>], 5000)), + ?assertEqual(1, meck:num_calls(httpc, request, '_')), + ?assert(meck:validate(httpc)). + +%%-------------------------------------------------------------------- +%% Profile lifecycle from the request path +%%-------------------------------------------------------------------- + +%% When the application was never started, the first request starts +%% the profile, applies its options and is retried once. +not_started_starts_profile_and_retries(_Config) -> + ok = stop_profile(), + Body = jsx:encode(#{<<"ip">> => <<"1.2.3.4">>}), + meck:expect(httpc, request, 5, + meck:seq([{error, {not_started, ipdata}}, + {ok, {{"HTTP/1.1", 200, "OK"}, [], Body}}])), + {ok, Result} = ipdata_http:get("https://api.ipdata.co?api-key=test", [], 5000), + ?assertEqual(<<"1.2.3.4">>, maps:get(<<"ip">>, Result)), + ?assertEqual(2, meck:num_calls(httpc, request, [get, '_', '_', '_', ipdata])), + assert_profile_running(), + ok = stop_profile(), + ?assert(meck:validate(httpc)). + +%% OTP's httpc:request/5 lets the gen_server noproc exit through instead +%% of returning {error, {not_started, Profile}}; both shapes are handled. +not_started_noproc_exit_starts_profile_and_retries(_Config) -> + ok = stop_profile(), + Body = jsx:encode([#{<<"ip">> => <<"8.8.8.8">>}]), + Exit = {noproc, {gen_server, call, [httpc_ipdata, request, infinity]}}, + meck:expect(httpc, request, 5, + meck:seq([meck:raise(exit, Exit), + {ok, {{"HTTP/1.1", 200, "OK"}, [], Body}}])), + {ok, Results} = ipdata_http:post("https://api.ipdata.co/bulk?api-key=test", + [], [<<"8.8.8.8">>], 5000), + ?assertEqual(1, length(Results)), + ?assertEqual(2, meck:num_calls(httpc, request, [post, '_', '_', '_', ipdata])), + assert_profile_running(), + ok = stop_profile(), + ?assert(meck:validate(httpc)). + +%% If the profile cannot be started (inets is down) the request fails +%% with the usual request_failed error instead of crashing. +not_started_profile_start_failure_is_request_failed(_Config) -> + ok = stop_profile(), + meck:new(inets, [unstick, passthrough]), + try + meck:expect(inets, start, + fun(httpc, [{profile, ipdata}]) -> {error, inets_not_started} end), + meck:expect(httpc, request, 5, {error, {not_started, ipdata}}), + ?assertEqual({error, {request_failed, inets_not_started}}, + ipdata_http:get("https://api.ipdata.co?api-key=test", [], 5000)), + ?assertEqual(1, meck:num_calls(httpc, request, '_')), + ?assert(meck:validate(inets)), + ?assert(meck:validate(httpc)) + after + meck:unload(inets) + end. + +%%-------------------------------------------------------------------- +%% Profile options +%%-------------------------------------------------------------------- + +profile_options_defaults(_Config) -> + ok = application:unset_env(ipdata, httpc_options), + ?assertEqual([{keep_alive_timeout, 60000}, + {max_keep_alive_length, 4}, + {max_sessions, 16}], + lists:sort(ipdata_http:profile_options())). + +%% httpc_options entries override the defaults key by key and may add +%% further httpc options. +profile_options_env_overrides(_Config) -> + Proxy = {{"proxy.example.com", 3128}, []}, + ok = application:set_env(ipdata, httpc_options, + [{max_sessions, 2}, {https_proxy, Proxy}]), + try + Opts = ipdata_http:profile_options(), + ?assertEqual(4, length(Opts)), + ?assertEqual(2, proplists:get_value(max_sessions, Opts)), + ?assertEqual(4, proplists:get_value(max_keep_alive_length, Opts)), + ?assertEqual(60000, proplists:get_value(keep_alive_timeout, Opts)), + ?assertEqual(Proxy, proplists:get_value(https_proxy, Opts)) + after + ok = application:unset_env(ipdata, httpc_options) + end. + +%%-------------------------------------------------------------------- +%% Helpers +%%-------------------------------------------------------------------- + +expect_response(Method, Status, Body) -> + meck:expect(httpc, request, + fun(M, _Request, _HttpOpts, _Opts, ipdata) when M =:= Method -> + {ok, {{"HTTP/1.1", Status, "Reason"}, [], Body}} + end). + +%% Argument lists of every recorded httpc:request/5 call, in order. +request_args() -> + [Args || {_Pid, {httpc, request, Args}, _Result} <- meck:history(httpc)]. + +stop_profile() -> + _ = inets:stop(httpc, ipdata), + ok. + +assert_profile_running() -> + ?assertMatch({error, {already_started, _}}, + inets:start(httpc, [{profile, ipdata}])), + {ok, Opts} = httpc:get_options([max_sessions, max_keep_alive_length, + keep_alive_timeout], ipdata), + ?assertEqual(lists:sort(ipdata_http:profile_options()), lists:sort(Opts)). From 3cfd4b37c5623a7537e56fbda53f71a6df46f8af Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 10:43:03 +0000 Subject: [PATCH 2/3] Document and test the new response fields The API now returns two more top-level booleans, is_satellite and is_anycast (siblings of is_eu), and each threat.blocklists entry may carry first_seen and last_seen (YYYY-MM-DD, UTC) and days_seen. The API includes the two flags only when they are true, and some feeds (for example ipdata's own vpn feed) omit the three dated fields, so this untyped library keeps passing the decoded jsx maps through unchanged: callers read the flags with maps:get(Key, Result, false) and treat the dates as optional binaries that are not parsed. jsx tolerates unknown keys, so nothing in the decoder changes. The README documents the five fields with the present-only-when-true behaviour, and the fields parameter accepts the two new names like any other (the library does not validate field names). Tests: fixtures built from the API samples (a satellite ISP, an anycast address with dated blocklist entries, a VPN exit whose third blocklist entry has no dates) are decoded through ipdata_http:get/3 and post/4, asserting the values, the absent-means-false reading, that unknown keys and non-ASCII strings survive decoding and that bulk items carry the same shape; ipdata_SUITE checks that the fields pass through lookup unchanged and that lookup/3 and bulk/3 accept the two new field names. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 --- README.md | 25 +++++- test/ipdata_SUITE.erl | 88 +++++++++++++++++++- test/ipdata_http_SUITE.erl | 94 ++++++++++++++++++++++ test/ipdata_http_SUITE_data/anycast.json | 35 ++++++++ test/ipdata_http_SUITE_data/satellite.json | 10 +++ test/ipdata_http_SUITE_data/vpn_exit.json | 18 +++++ 6 files changed, 268 insertions(+), 2 deletions(-) create mode 100644 test/ipdata_http_SUITE_data/anycast.json create mode 100644 test/ipdata_http_SUITE_data/satellite.json create mode 100644 test/ipdata_http_SUITE_data/vpn_exit.json diff --git a/README.md b/README.md index a2e8944..41f25b6 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,10 @@ maps:get(<<"name">>, ASN). %% => <<"Google LLC">> Threat = maps:get(<<"threat">>, Result). maps:get(<<"is_vpn">>, Threat). %% => false +%% Flags the API includes only when they are true: read them with a default +maps:get(<<"is_anycast">>, Result, false). %% => true +maps:get(<<"is_satellite">>, Result, false). %% => false + TimeZone = maps:get(<<"time_zone">>, Result). maps:get(<<"name">>, TimeZone). %% => <<"America/New_York">> ``` @@ -201,6 +205,8 @@ Look up multiple IP addresses in a single request. Maximum 100 IPs. Requires a p |-------|------|-------------| | `ip` | binary | IP address | | `is_eu` | boolean | Whether the country is in the EU | +| `is_satellite` | boolean | Satellite internet provider; present only when `true` | +| `is_anycast` | boolean | Anycast address; present only when `true` | | `city` | binary | City name | | `region` | binary | Region/state name | | `region_code` | binary | Region code | @@ -226,6 +232,8 @@ Look up multiple IP addresses in a single request. Maximum 100 IPs. Requires a p | `threat` | map | Threat data (see below) | | `count` | binary | API request count | +The API includes `is_satellite` and `is_anycast` only when they are `true` and omits them otherwise. The library returns the decoded JSON unchanged, so read them with a default: `maps:get(<<"is_satellite">>, Result, false)`. Both names can be requested through the fields parameter like any other field. + ### Threat Fields | Field | Type | Description | @@ -240,9 +248,24 @@ Look up multiple IP addresses in a single request. Maximum 100 IPs. Requires a p | `is_known_abuser` | boolean | Known abuser | | `is_threat` | boolean | Any threat detected | | `is_bogon` | boolean | Bogon IP | -| `blocklists` | list | Blocklist entries (`name`, `site`, `type`) | +| `blocklists` | list | Blocklist entries (see below) | | `scores` | map | Reputation scores (`vpn_score`, `proxy_score`, `threat_score`, `trust_score`) | +### Blocklist Entry Fields + +Each element of `blocklists` is a map: + +| Field | Type | Description | +|-------|------|-------------| +| `name` | binary | Blocklist name | +| `site` | binary | Blocklist website | +| `type` | binary | Blocklist type (for example `vpn`, `cloud`, `general`) | +| `first_seen` | binary | First day the IP was listed, `YYYY-MM-DD` (UTC); optional | +| `last_seen` | binary | Last day the IP was listed, `YYYY-MM-DD` (UTC); optional | +| `days_seen` | integer | Number of days the IP was listed; optional | + +Some feeds, for example ipdata's own `vpn` feed, carry none of `first_seen`, `last_seen` and `days_seen`, so read them with a default such as `maps:get(<<"days_seen">>, Entry, undefined)`. The dates are returned as binaries and are not parsed. + ## Testing ```bash diff --git a/test/ipdata_SUITE.erl b/test/ipdata_SUITE.erl index 936573b..12ddf3c 100644 --- a/test/ipdata_SUITE.erl +++ b/test/ipdata_SUITE.erl @@ -32,7 +32,10 @@ all() -> bulk_lookup_with_fields, bulk_empty_list_rejected, bulk_too_many_ips_rejected, - bulk_api_error]. + bulk_api_error, + lookup_returns_new_fields_unchanged, + lookup_with_new_field_names, + bulk_with_new_field_names]. init_per_suite(Config) -> application:ensure_all_started(meck), @@ -191,10 +194,93 @@ bulk_api_error(_Config) -> ipdata:bulk(Client, [<<"8.8.8.8">>])), ?assert(meck:validate(ipdata_http)). +%%-------------------------------------------------------------------- +%% New response fields +%%-------------------------------------------------------------------- + +%% is_satellite, is_anycast and the dated blocklist entries reach the +%% caller exactly as decoded: the API omits the two flags when false and +%% some feeds omit the dates, so absent keys stay absent. +lookup_returns_new_fields_unchanged(_Config) -> + MockResponse = satellite_ip_response(), + meck:expect(ipdata_http, get, + fun("https://api.ipdata.co/98.97.34.0?api-key=test-key", [], 5000) -> + {ok, MockResponse} + end), + {ok, Client} = ipdata:new(<<"test-key">>), + {ok, Result} = ipdata:lookup(Client, <<"98.97.34.0">>), + ?assertEqual(MockResponse, Result), + ?assertEqual(true, maps:get(<<"is_satellite">>, Result)), + ?assertNot(maps:is_key(<<"is_anycast">>, Result)), + ?assertEqual(false, maps:get(<<"is_anycast">>, Result, false)), + [Dated, Undated] = maps:get(<<"blocklists">>, maps:get(<<"threat">>, Result)), + ?assertEqual(<<"2026-10-31">>, maps:get(<<"first_seen">>, Dated)), + ?assertEqual(2, maps:get(<<"days_seen">>, Dated)), + ?assertEqual(undefined, maps:get(<<"first_seen">>, Undated, undefined)), + ?assert(meck:validate(ipdata_http)). + +%% The new names are valid field names for the fields parameter. +lookup_with_new_field_names(_Config) -> + MockResponse = #{<<"ip">> => <<"8.8.8.8">>, <<"is_anycast">> => true}, + meck:expect(ipdata_http, get, + fun("https://api.ipdata.co/8.8.8.8?api-key=test-key&fields=ip,is_satellite,is_anycast", + [], 5000) -> + {ok, MockResponse} + end), + {ok, Client} = ipdata:new(<<"test-key">>), + {ok, Result} = ipdata:lookup(Client, <<"8.8.8.8">>, + [<<"ip">>, <<"is_satellite">>, <<"is_anycast">>]), + ?assertEqual(true, maps:get(<<"is_anycast">>, Result)), + ?assertEqual(false, maps:get(<<"is_satellite">>, Result, false)), + ?assert(meck:validate(ipdata_http)). + +bulk_with_new_field_names(_Config) -> + MockResponse = [#{<<"ip">> => <<"98.97.34.0">>, <<"is_satellite">> => true}, + #{<<"ip">> => <<"8.8.8.8">>, <<"is_anycast">> => true}], + meck:expect(ipdata_http, post, + fun("https://api.ipdata.co/bulk?api-key=test-key&fields=ip,is_satellite,is_anycast", + [], [<<"98.97.34.0">>, <<"8.8.8.8">>], 5000) -> + {ok, MockResponse} + end), + {ok, Client} = ipdata:new(<<"test-key">>), + {ok, [Satellite, Anycast]} = + ipdata:bulk(Client, [<<"98.97.34.0">>, <<"8.8.8.8">>], + [<<"ip">>, <<"is_satellite">>, <<"is_anycast">>]), + ?assertEqual(true, maps:get(<<"is_satellite">>, Satellite)), + ?assertEqual(false, maps:get(<<"is_anycast">>, Satellite, false)), + ?assertEqual(true, maps:get(<<"is_anycast">>, Anycast)), + ?assertEqual(false, maps:get(<<"is_satellite">>, Anycast, false)), + ?assert(meck:validate(ipdata_http)). + %%-------------------------------------------------------------------- %% Helpers %%-------------------------------------------------------------------- +%% Decoded shape of a satellite ISP response (is_anycast omitted) whose +%% blocklist has one dated entry and one from a feed without dates. +satellite_ip_response() -> + #{<<"ip">> => <<"98.97.34.0">>, + <<"is_satellite">> => true, + <<"is_eu">> => false, + <<"city">> => <<"Seattle">>, + <<"country_code">> => <<"US">>, + <<"threat">> => #{ + <<"is_vpn">> => false, + <<"is_threat">> => false, + <<"blocklists">> => [ + #{<<"name">> => <<"Gluetun">>, + <<"site">> => <<"https://github.com/qdm12/gluetun">>, + <<"type">> => <<"vpn">>, + <<"days_seen">> => 2, + <<"first_seen">> => <<"2026-10-31">>, + <<"last_seen">> => <<"2026-11-02">>}, + #{<<"name">> => <<"ipdata">>, + <<"site">> => <<"https://ipdata.co">>, + <<"type">> => <<"vpn">>} + ] + }, + <<"count">> => <<"0">>}. + sample_ip_response() -> #{<<"ip">> => <<"8.8.8.8">>, <<"is_eu">> => false, diff --git a/test/ipdata_http_SUITE.erl b/test/ipdata_http_SUITE.erl index 6d1e504..a442e22 100644 --- a/test/ipdata_http_SUITE.erl +++ b/test/ipdata_http_SUITE.erl @@ -32,6 +32,10 @@ all() -> post_retries_once_on_socket_closed_remotely, post_retries_only_once, post_does_not_retry_http_errors, + decode_satellite_response, + decode_anycast_response_with_dated_blocklists, + decode_blocklist_entry_without_dates, + decode_bulk_items_with_new_fields, not_started_starts_profile_and_retries, not_started_noproc_exit_starts_profile_and_retries, not_started_profile_start_failure_is_request_failed, @@ -227,6 +231,91 @@ post_does_not_retry_http_errors(_Config) -> ?assertEqual(1, meck:num_calls(httpc, request, '_')), ?assert(meck:validate(httpc)). +%%-------------------------------------------------------------------- +%% New response fields: is_satellite, is_anycast and dated blocklist +%% entries. The fixtures are the API's own samples; the library passes +%% the decoded JSON through unchanged, and the API includes the two +%% flags only when they are true. +%%-------------------------------------------------------------------- + +decode_satellite_response(Config) -> + expect_response(get, 200, fixture(Config, "satellite.json")), + {ok, R} = ipdata_http:get("https://api.ipdata.co/98.97.34.0?api-key=test", [], 5000), + ?assertEqual(<<"98.97.34.0">>, maps:get(<<"ip">>, R)), + ?assertEqual(true, maps:get(<<"is_satellite">>, R)), + ?assertEqual(false, maps:get(<<"is_eu">>, R)), + %% is_anycast is omitted by the API when false: read it with a default + ?assertNot(maps:is_key(<<"is_anycast">>, R)), + ?assertEqual(false, maps:get(<<"is_anycast">>, R, false)), + ?assertEqual(<<"AS14593">>, maps:get(<<"asn">>, maps:get(<<"asn">>, R))), + ?assertEqual([], maps:get(<<"blocklists">>, maps:get(<<"threat">>, R))), + ?assert(meck:validate(httpc)). + +decode_anycast_response_with_dated_blocklists(Config) -> + expect_response(get, 200, fixture(Config, "anycast.json")), + {ok, R} = ipdata_http:get("https://api.ipdata.co/8.8.8.8?api-key=test", [], 5000), + ?assertEqual(true, maps:get(<<"is_anycast">>, R)), + ?assertNot(maps:is_key(<<"is_satellite">>, R)), + ?assertEqual(false, maps:get(<<"is_satellite">>, R, false)), + ?assertEqual(null, maps:get(<<"city">>, R)), + %% non-ASCII strings survive decoding (U+1F1FA U+1F1F8 as UTF-8) + ?assertEqual(<<240, 159, 135, 186, 240, 159, 135, 184>>, + maps:get(<<"emoji_flag">>, R)), + Threat = maps:get(<<"threat">>, R), + ?assertEqual(true, maps:get(<<"is_datacenter">>, Threat)), + [Cloud, Resolvers] = maps:get(<<"blocklists">>, Threat), + ?assertEqual(#{<<"name">> => <<"ipdata">>, + <<"site">> => <<"https://ipdata.co">>, + <<"type">> => <<"cloud">>, + <<"days_seen">> => 1, + <<"first_seen">> => <<"2026-11-02">>, + <<"last_seen">> => <<"2026-11-02">>}, Cloud), + ?assertEqual(<<"open_resolvers">>, maps:get(<<"type">>, Resolvers)), + %% the dates stay YYYY-MM-DD binaries; they are not parsed + ?assertEqual(<<"2026-11-02">>, maps:get(<<"last_seen">>, Resolvers)), + ?assertEqual(2, maps:get(<<"trust_score">>, maps:get(<<"scores">>, Threat))), + ?assert(meck:validate(httpc)). + +decode_blocklist_entry_without_dates(Config) -> + expect_response(get, 200, fixture(Config, "vpn_exit.json")), + {ok, R} = ipdata_http:get("https://api.ipdata.co/187.13.250.1?api-key=test", [], 5000), + ?assertNot(maps:is_key(<<"is_satellite">>, R)), + ?assertNot(maps:is_key(<<"is_anycast">>, R)), + Threat = maps:get(<<"threat">>, R), + ?assertEqual(true, maps:get(<<"is_vpn">>, Threat)), + Blocklists = maps:get(<<"blocklists">>, Threat), + ?assertEqual(6, length(Blocklists)), + [Gluetun, _Cloud, IpdataVpn | _] = Blocklists, + ?assertMatch(#{<<"name">> := <<"Gluetun">>, <<"type">> := <<"vpn">>, + <<"days_seen">> := 2, + <<"first_seen">> := <<"2026-10-31">>, + <<"last_seen">> := <<"2026-11-02">>}, Gluetun), + %% ipdata's own vpn feed carries none of the three dated fields + ?assertEqual(#{<<"name">> => <<"ipdata">>, + <<"site">> => <<"https://ipdata.co">>, + <<"type">> => <<"vpn">>}, IpdataVpn), + ?assertEqual(undefined, maps:get(<<"days_seen">>, IpdataVpn, undefined)), + ?assert(meck:validate(httpc)). + +%% Bulk responses carry the same shape per item. +decode_bulk_items_with_new_fields(Config) -> + Body = <<"[", (fixture(Config, "satellite.json"))/binary, ",", + (fixture(Config, "anycast.json"))/binary, ",", + (fixture(Config, "vpn_exit.json"))/binary, "]">>, + expect_response(post, 200, Body), + {ok, [Satellite, Anycast, Vpn]} = + ipdata_http:post("https://api.ipdata.co/bulk?api-key=test", [], + [<<"98.97.34.0">>, <<"8.8.8.8">>, <<"187.13.250.1">>], 5000), + ?assertEqual(true, maps:get(<<"is_satellite">>, Satellite)), + ?assertEqual(false, maps:get(<<"is_anycast">>, Satellite, false)), + ?assertEqual(true, maps:get(<<"is_anycast">>, Anycast)), + ?assertEqual(false, maps:get(<<"is_satellite">>, Anycast, false)), + ?assertEqual(false, maps:get(<<"is_satellite">>, Vpn, false)), + ?assertEqual(false, maps:get(<<"is_anycast">>, Vpn, false)), + [_, _, IpdataVpn | _] = maps:get(<<"blocklists">>, maps:get(<<"threat">>, Vpn)), + ?assertNot(maps:is_key(<<"first_seen">>, IpdataVpn)), + ?assert(meck:validate(httpc)). + %%-------------------------------------------------------------------- %% Profile lifecycle from the request path %%-------------------------------------------------------------------- @@ -327,6 +416,11 @@ stop_profile() -> _ = inets:stop(httpc, ipdata), ok. +%% Raw JSON of an API sample from the suite's data directory. +fixture(Config, Name) -> + {ok, Body} = file:read_file(filename:join(?config(data_dir, Config), Name)), + Body. + assert_profile_running() -> ?assertMatch({error, {already_started, _}}, inets:start(httpc, [{profile, ipdata}])), diff --git a/test/ipdata_http_SUITE_data/anycast.json b/test/ipdata_http_SUITE_data/anycast.json new file mode 100644 index 0000000..67424bb --- /dev/null +++ b/test/ipdata_http_SUITE_data/anycast.json @@ -0,0 +1,35 @@ +{ + "ip": "8.8.8.8", + "is_anycast": true, + "is_eu": false, + "city": null, + "region": null, + "region_code": null, + "region_type": null, + "country_name": "United States", + "country_code": "US", + "continent_name": "North America", + "continent_code": "NA", + "latitude": 39.7599983215332, + "longitude": -98.5, + "postal": null, + "calling_code": "1", + "flag": "https://ipdata.co/flags/us.png", + "emoji_flag": "πŸ‡ΊπŸ‡Έ", + "emoji_unicode": "U+1F1FA U+1F1F8", + "asn": { "asn": "AS15169", "name": "Google LLC", "domain": "about.google", "route": "8.8.8.0/24", "type": "hosting" }, + "company": { "name": "Google LLC", "domain": "about.google", "network": "8.8.8.0/24", "type": "hosting" }, + "languages": [ { "name": "English", "native": "English", "code": "en" } ], + "currency": { "name": "US Dollar", "code": "USD", "symbol": "$", "native": "$", "plural": "US dollars" }, + "time_zone": { "name": null, "abbr": null, "offset": null, "is_dst": null, "current_time": null }, + "threat": { + "is_tor": false, "is_vpn": false, "is_icloud_relay": false, "is_proxy": false, "is_datacenter": true, + "is_anonymous": false, "is_known_attacker": false, "is_known_abuser": false, "is_threat": false, "is_bogon": false, + "blocklists": [ + { "days_seen": 1, "first_seen": "2026-11-02", "last_seen": "2026-11-02", "name": "ipdata", "site": "https://ipdata.co", "type": "cloud" }, + { "days_seen": 1, "first_seen": "2026-11-02", "last_seen": "2026-11-02", "name": "ipdata", "site": "https://ipdata.co", "type": "open_resolvers" } + ], + "scores": { "vpn_score": 100, "proxy_score": 93, "threat_score": 100, "trust_score": 2 } + }, + "count": "0" +} diff --git a/test/ipdata_http_SUITE_data/satellite.json b/test/ipdata_http_SUITE_data/satellite.json new file mode 100644 index 0000000..c0abd6f --- /dev/null +++ b/test/ipdata_http_SUITE_data/satellite.json @@ -0,0 +1,10 @@ +{ + "ip": "98.97.34.0", + "is_satellite": true, + "is_eu": false, + "city": "Seattle", + "country_code": "US", + "asn": { "asn": "AS14593", "name": "Space Exploration Technologies Corporation", "domain": "spacex.com", "route": "98.97.32.0/21", "type": "business" }, + "threat": { "is_tor": false, "is_vpn": false, "is_icloud_relay": false, "is_proxy": false, "is_datacenter": false, "is_anonymous": false, "is_known_attacker": false, "is_known_abuser": false, "is_threat": false, "is_bogon": false, "blocklists": [], "scores": { "vpn_score": 0, "proxy_score": 1, "threat_score": 8, "trust_score": 97 } }, + "count": "0" +} diff --git a/test/ipdata_http_SUITE_data/vpn_exit.json b/test/ipdata_http_SUITE_data/vpn_exit.json new file mode 100644 index 0000000..f4d1485 --- /dev/null +++ b/test/ipdata_http_SUITE_data/vpn_exit.json @@ -0,0 +1,18 @@ +{ + "ip": "187.13.250.1", + "is_eu": false, + "threat": { + "is_tor": false, "is_vpn": true, "is_icloud_relay": false, "is_proxy": false, "is_datacenter": true, + "is_anonymous": true, "is_known_attacker": true, "is_known_abuser": true, "is_threat": true, "is_bogon": false, + "blocklists": [ + { "days_seen": 2, "first_seen": "2026-10-31", "last_seen": "2026-11-02", "name": "Gluetun", "site": "https://github.com/qdm12/gluetun", "type": "vpn" }, + { "days_seen": 1, "first_seen": "2026-11-02", "last_seen": "2026-11-02", "name": "ipdata", "site": "https://ipdata.co", "type": "cloud" }, + { "name": "ipdata", "site": "https://ipdata.co", "type": "vpn" }, + { "days_seen": 2, "first_seen": "2026-10-31", "last_seen": "2026-11-02", "name": "NordVPN", "site": "https://nordvpn.com", "type": "vpn" }, + { "days_seen": 2, "first_seen": "2026-10-31", "last_seen": "2026-11-02", "name": "Wikimedia Global Blocklist", "site": "https://wikimedia.org", "type": "general" }, + { "days_seen": 2, "first_seen": "2026-10-31", "last_seen": "2026-11-02", "name": "X4BNet", "site": "https://github.com/X4BNet/lists_vpn", "type": "vpn" } + ], + "scores": { "vpn_score": 100, "proxy_score": 99, "threat_score": 100, "trust_score": 0 } + }, + "count": "0" +} From d0556cc705e0e162b86b969215a8ecabde962036 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 10:48:12 +0000 Subject: [PATCH 3/3] Run the OTP 25 CI job on ubuntu-24.04 The first pull_request run of the workflow failed in erlef/setup-beam for OTP 25 before any step of ours ran: ubuntu-latest now resolves to Ubuntu 26.04, for which hex.pm publishes no OTP 25 builds (setup-beam's compatibility table lists OTP 26 and later for that image and 24.3 and later for ubuntu-24.04). OTP 26 and 27 passed compile, ct, dialyzer and xref on the new image and were only cancelled by the matrix fail-fast. Pin the OTP 25 matrix entry to ubuntu-24.04 and keep the other two on ubuntu-latest; the set of tested OTP versions is unchanged. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 --- .github/workflows/publish.yml | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 34fe9ef..fd98658 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -11,10 +11,19 @@ on: jobs: test: name: Test - runs-on: ubuntu-latest + runs-on: ${{ matrix.os }} strategy: matrix: - otp: ['25', '26', '27'] + # ubuntu-latest resolves to Ubuntu 26.04, for which no OTP 25 builds + # are published (erlef/setup-beam supports OTP 26+ there and 24.3+ on + # ubuntu-24.04), so the OTP 25 job runs on ubuntu-24.04. + include: + - otp: '25' + os: ubuntu-24.04 + - otp: '26' + os: ubuntu-latest + - otp: '27' + os: ubuntu-latest steps: - uses: actions/checkout@v4