Skip to content

Reuse HTTPS connections, add new response fields - #7

Merged
jonathan-kosgei merged 2 commits into
mainfrom
http-reuse-and-new-fields
Nov 3, 2026
Merged

jonathan-kosgei merged 2 commits into
mainfrom
http-reuse-and-new-fields

Conversation

@jonathan-kosgei

@jonathan-kosgei jonathan-kosgei commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

IPData::Client used to build a new Net::HTTP for every request, paying a TCP + TLS handshake each time. Two logical commits:

  1. Reuse one HTTPS connection per client. A client now holds one Net::HTTP, created lazily on the first request, started and kept: HTTP/1.1 keep-alive with a 60 s client-side idle timeout (new keyword keep_alive_timeout:; Net::HTTP's own default is 2 s), open/read/write timeouts from timeout:, max_retries = 1. Requests are serialised with a Mutex. A connection inherited across fork is detected by process id and dropped without finish (no TLS close_notify on the socket shared with the parent) before reconnecting. POST /bulk is retried once on a fresh connection when a reused socket dies before any response byte arrives (EOFError, IOError, Errno::ECONNRESET, Errno::EPIPE, OpenSSL::SSL::SSLError); GETs are retried by Net::HTTP itself; nothing is retried on an HTTP status. New public Client#close. Fix while there: use_ssl now follows the base_url scheme instead of being unconditionally true. Net::HTTP has no HTTP/2, so this is the keep-alive-only variant: the saving is the TLS handshake, not multiplexing.
  2. New response fields. is_satellite and is_anycast (top-level booleans the API sends only when true) and first_seen / last_seen / days_seen on threat.blocklists entries (optional; some feeds omit them). The JSON passes through unchanged; Response gains explicit is_satellite / is_anycast readers that return false when the key is absent (dot-notation would otherwise raise NoMethodError for every non-satellite address), the raw hash is untouched, and the dates stay strings.

No new runtime dependencies (openssl is stdlib and was already loaded by Net::HTTP for HTTPS). CI already runs on pull requests and is unchanged. lib/ipdata/version.rb is not edited (see Release notes).

What changes for users

  • Consecutive calls on one IPData::Client reuse a single TLS connection. Create one client per process (or per API key / endpoint), keep it, and reuse it; each new client pays a handshake.
  • New constructor keyword keep_alive_timeout: (seconds, default 60): how long an idle connection is kept before the next request opens a new one.
  • timeout: now also sets the write timeout (previously only open and read; Net::HTTP's write default was 60 s).
  • New method Client#close: finishes the connection; the client stays usable and reconnects on the next request. Call it at shutdown.
  • A client is safe to share between threads but serialises requests on its one connection; use one client per thread for parallel lookups. Create clients after forking; an inherited client reconnects in the child instead of touching the parent's socket.
  • bulk_lookup is retried once when a reused connection turns out to be dead before any response byte arrived. lookup was already retried once by Net::HTTP in that case.
  • Response#is_satellite and Response#is_anycast read false when the API omitted the key; response["is_satellite"], key? and to_h reflect the raw response. Blocklist entries expose first_seen, last_seen ("YYYY-MM-DD" strings, UTC) and days_seen when present; entry["first_seen"] is nil when a feed omits them.
  • base_url: with an http:// scheme now connects without TLS (it always used TLS before, which could not work for plain-HTTP URLs).
  • Everything else is unchanged: constructor arguments, lookup, bulk_lookup, Client.eu, caching, error classes, User-Agent format.

Release notes

Version: the Release workflow (workflow_dispatch, bump = minor) bumps lib/ipdata/version.rb from 0.1.5 to 0.2.0, commits, tags v0.2.0 and publishes to RubyGems. version.rb is deliberately not edited in this PR.

ipdata 0.2.0

Connection reuse
- An IPData::Client now keeps one persistent HTTPS connection (HTTP/1.1 keep-alive) for its
  lifetime instead of opening a new TLS connection per request. Create one client and reuse it.
- New option keep_alive_timeout: (seconds, default 60) for the client-side idle timeout.
- New method Client#close to finish the connection at shutdown; the client reconnects on demand.
- timeout: now also applies to writes.
- Safe to share between threads (requests are serialised on the connection) and fork-safe: an
  inherited client reconnects in the child without closing the parent's socket.
- bulk_lookup (POST /bulk) is retried once when a reused connection died before any response
  byte arrived; lookups (GET) were already retried once by Net::HTTP. Nothing is retried on an
  HTTP error status.
- use_ssl follows the base_url scheme (plain http:// base URLs now work).

New response fields
- is_satellite and is_anycast: top-level booleans sent by the API only when true;
  response.is_satellite / response.is_anycast read false when absent. Both can be requested
  through fields: and select_field:.
- Blocklist entries (threat.blocklists) carry first_seen, last_seen ("YYYY-MM-DD", UTC) and
  days_seen when the feed reports them; some feeds omit them, so entry["first_seen"] is nil then.

No new dependencies. Ruby >= 3.0 as before.

Verification

Local (Ruby 3.3.6, Bundler 4.0.18), run before each push, exactly as CI does:

  • ruby -Ilib -Itest -e "Dir['test/test_*.rb'].each { |f| require File.expand_path(f) }"
    • baseline on main: 79 runs, 210 assertions, 0 failures, 0 errors
    • after commit 1: 98 runs, 282 assertions, 0 failures, 0 errors
    • after commit 2 (pushed tree): 115 runs, 400 assertions, 0 failures, 0 errors
  • gem build ipdata.gemspec: "Successfully built RubyGem" (ipdata-0.1.5) after each commit. A pre-existing gemspec warning about identical homepage_uri/source_code_uri is untouched.
  • ruby -w -Ilib -e 'require "ipdata"': no warnings from the library. (-w on the test suite reports one pre-existing unused local uri in assert_request, kept as is.)
  • The repository has no rubocop configuration or CI step, so none was run.

Two IPData::Client#lookup calls on one client against api.ipdata.co were served over the same HTTP/1.1 keep-alive connection (Net::HTTP does not negotiate HTTP/2).

CI: the CI workflow already has a pull_request trigger on main (Ruby 3.0, 3.1, 3.2 and 3.3: tests + gem build), so no workflow change was needed. CI run 13 (https://github.com/ipdata/ruby/actions/runs/41620699728) on head 21f8e43 completed with conclusion success. All four matrix jobs passed, each running the test suite and gem build: test (3.0), test (3.1), test (3.2), test (3.3). No follow-up commits were needed.

Noticed, not changed

  • ipdata.gemspec changelog_uri points to CHANGELOG.md, which does not exist in the repository (the release notes for this change are above).
  • gem build warns that homepage_uri and source_code_uri are identical ("Only the first one will be shown on rubygems.org").
  • .github/workflows/ci.yml starts with a Ruby magic comment (# frozen_string_literal: true) that means nothing in YAML.
  • release.yml runs its release job on every push to main (github.event_name == 'push' matches branch pushes as well as tags), so a plain merge to main attempts a RubyGems publish of the current version.
  • lookup(nil, select_field: "city") builds the path //city (double slash) because ip is nil.
  • build_uri replaces the whole path of base_url:, so a base URL with a path prefix loses it.
  • The User-Agent keeps its existing ruby-ipdata/<version> ... format, which differs from the ipdata-<language>/<version> shape used by the other ipdata client libraries.
  • test/test_client.rb assert_request assigns an unused local uri (pre-existing -w warning).
  • README "Development" says bundle exec rake test, while CI runs ruby -Ilib -Itest -e ...; both work in a normal setup.

🤖 Generated with Claude Code

Every request built a fresh Net::HTTP object, so each lookup paid a TCP and
TLS handshake (about 660 to 980 ms against api.ipdata.co, versus 220 to
300 ms on a reused connection). A Client now holds one Net::HTTP that is
created lazily on the first request, started and kept: HTTP/1.1 keep-alive
(Net::HTTP has no HTTP/2, so the saving is the handshake, not
multiplexing), open/read/write timeouts from timeout:, max_retries = 1, and
a client-side idle timeout from the new keyword keep_alive_timeout: (default
60 s; Net::HTTP's own default is 2 s). While there, use_ssl now follows the
base_url scheme instead of being unconditionally true.

Requests are serialised with a Mutex, so a client can be shared between
threads. The process id is recorded when connecting; a connection inherited
across fork is dropped without finish (no TLS close_notify on the socket
shared with the parent) and a new one opened. POST /bulk, which Net::HTTP
never retries, is retried exactly once on a fresh connection when a reused
socket dies before any response byte arrives (EOFError, IOError,
Errno::ECONNRESET, Errno::EPIPE, OpenSSL::SSL::SSLError); GETs are retried
by Net::HTTP itself. Nothing is retried on an HTTP status.

New public method Client#close finishes the connection; the client stays
usable and reconnects on demand. get, post and the error mapping are
unchanged. The Net::HTTP stand-in in the tests now provides start,
started?, finish and the new setters, the tests that counted Net::HTTP.new
calls count requests instead, and new tests cover one connection across
calls, close, reconnect after a pid change, the POST retry (once, not after
response bytes, not on a status) and GET not being retried by the client.
README gains a "Connection reuse" section documenting thread and fork
safety, close and keep_alive_timeout:.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API now returns two more top-level booleans, is_satellite and
is_anycast (siblings of is_eu), and each threat.blocklists entry may carry
first_seen and last_seen (calendar dates as "YYYY-MM-DD" strings, UTC) and
days_seen (an integer). Both flags are included in a response only when
they are true, and some feeds (ipdata's own vpn feed, for example) report
no dates.

The library passes the JSON through unchanged. Because Response resolves
dot-notation through method_missing, response.is_satellite would have
raised NoMethodError for every non-satellite address, so Response gains
explicit is_satellite and is_anycast readers that return false when the
key is absent; the raw hash (to_h, [], key?, each, to_json) is untouched.
The client does not validate fields= names, so the new names already pass
through to the API; a test pins that down. Blocklist dates stay strings
(no parsing, no new dependency) and are reached as before through the
nested Response objects, with [] giving nil when a feed omits them.

Tests add fixtures built from the API samples (a satellite ISP, an anycast
address with dated blocklist entries, and a VPN exit whose third entry has
no dates) and cover the typed values, false-when-absent, the untouched raw
hash, unknown keys passing through, the new names in fields= and
select_field, and the same shape per item in bulk responses. The README
fields documentation gains the five names and the "present only when
true" rule.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jonathan-kosgei
jonathan-kosgei merged commit 9800b9b into main Nov 3, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants