Skip to content

Reuse HTTPS connections, negotiate HTTP/2, add new response fields - #23

Merged
jonathan-kosgei merged 2 commits into
masterfrom
http-reuse-and-new-fields
Nov 3, 2026
Merged

jonathan-kosgei merged 2 commits into
masterfrom
http-reuse-and-new-fields

Conversation

@jonathan-kosgei

@jonathan-kosgei jonathan-kosgei commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Two logical commits, additive only (no public symbol renamed or removed; every constructor, option and method signature unchanged; the Error type users see is unchanged):

  1. Reuse HTTPS connections and negotiate HTTP/2. A Client already kept one *http.Client for its lifetime, but the custom DialContext in newHTTPClient() switches off net/http's automatic HTTP/2 setup, so every request went over HTTP/1.1. The transport now sets ForceAttemptHTTP2: true (h2 through ALPN, with net/http's HTTP/1.1 fallback) and HTTP2: &http.HTTP2Config{SendPingTimeout: 30 * time.Second, PingTimeout: 15 * time.Second} (Go 1.24 API; go.mod already requires 1.24.7); the 60 s idle timeout and 30 s TCP keepalive stay. New func (c Client) CloseIdleConnections() (nil-guarded). The bulk POST is replayed exactly once when a pooled connection died before any response byte arrived (details below). While there: RawLookup/RawBulkLookup close the body of non-200 responses, Version is 0.8.2 (the const said 0.8.0 while the last tag is v0.8.1), the User-Agent names github.com/ipdata/go instead of theckman/go-ipdata, and the stale doc.go example is fixed (import path, two return values, non-empty key). README gains a "Connection reuse and HTTP/2" section after Basic Lookup.
  2. New response fields. IP.IsSatellite and IP.IsAnycast (bool, siblings of IsEU; the API sends each key only when true, so they read false when absent), and on Blocklist the optional FirstSeen, LastSeen (string, YYYY-MM-DD, UTC, deliberately kept as strings) and DaysSeen (*int, nil when a feed such as ipdata's own vpn feed does not report it). Bulk responses carry the same shape per item. encoding/json ignores unknown keys and the package has no strict decoder, so further additions keep decoding. gofmt also normalised Threat, Blocklist and Scores in types.go, which were indented with spaces.

Tests added: the negotiated protocol against an httptest TLS server with EnableHTTP2, using the transport from newHTTPClient() given the test CA (resp.Proto == "HTTP/2.0", ALPN h2); connection reuse across 11 sequential lookups, bulk lookups and error responses counted through ConnState (1 connection) over both plain HTTP/1.1 and HTTPS+h2, then CloseIdleConnections forcing a second dial; nil-guard and custom-transport cases of CloseIdleConnections; the bulk replay against a scripted raw keep-alive server (replayed once, at most once, never after response bytes, never on an HTTP status, never on a fresh connection); the three API sample responses decoded through decodeIP, Lookup and BulkLookup with the typed values, false-when-absent and unknown-key tolerance (region_type) asserted; bulkToIP and the omitempty encoding of a Blocklist without dates.

What changes for users

  • Requests to api.ipdata.co are multiplexed over HTTP/2 when the server negotiates it, HTTP/1.1 keep-alive otherwise. Build one Client per process (or per API key or endpoint) and reuse it for every lookup; it is safe for concurrent use from multiple goroutines. Idle HTTP/2 connections are health-checked with a PING every 30 s and closed if unanswered within 15 s; pooled connections are closed after 60 s idle.
  • client.CloseIdleConnections() releases the pooled connections at shutdown (or before a long idle period); the client stays usable afterwards. It is a no-op for a zero Client or a custom transport without that method.
  • WithHTTPClient keeps working unchanged. Connection reuse then follows that transport's settings; a transport with a custom DialContext or TLSClientConfig must set ForceAttemptHTTP2: true itself to speak HTTP/2 (documented in the README).
  • Bulk lookups: net/http only replays a POST when nothing was written to the connection yet. When a pooled connection that the server had silently dropped is picked, the request bytes go out and the read fails (EOF, reset) before any response byte; RawBulkLookup/BulkLookup now replay the request exactly once in that case, detected with net/http/httptrace (connection reused, no response byte received, error is neither a cancellation nor a timeout). Never on an HTTP status, never after response bytes arrived, never on a connection that was not reused, and at most one extra attempt.
  • RawLookup/RawBulkLookup close the body of non-200 responses instead of leaving it to the garbage collector.
  • New fields IP.IsSatellite, IP.IsAnycast, Blocklist.FirstSeen, Blocklist.LastSeen, Blocklist.DaysSeen; the README field tables document them. LookupFields has no field-name validation, so there was no list to extend.
  • The User-Agent is now go-ipdata/0.8.2 (github.com/ipdata/go) Go-http-client/<goversion> (<os> <arch>).

Release notes

Version in the repository: 0.8.2 (Version const). Tag to push after merge: v0.8.2.

v0.8.2

Connection reuse and HTTP/2
- A Client keeps a pool of persistent HTTPS connections for its lifetime and now
  negotiates HTTP/2 to api.ipdata.co (ALPN, automatic HTTP/1.1 fallback), with
  HTTP/2 keepalive pings every 30 s, a 60 s idle timeout and 30 s TCP keepalive.
- New Client.CloseIdleConnections() releases the pooled connections at shutdown.
- Bulk lookups (POST /bulk) are replayed exactly once when a pooled connection
  died before any response byte arrived; never on an HTTP status or after
  response bytes were received.
- RawLookup and RawBulkLookup close the body of non-200 responses.
- Version now matches the tag, the User-Agent names github.com/ipdata/go, and
  the package example in doc.go is fixed.
- Custom *http.Client via WithHTTPClient: a transport with its own DialContext
  or TLSClientConfig needs ForceAttemptHTTP2: true to speak HTTP/2.

New response fields
- IP.IsSatellite and IP.IsAnycast (bool): the API sends is_satellite and
  is_anycast only when true, so both read false when the key is absent.
- Blocklist.FirstSeen and Blocklist.LastSeen (YYYY-MM-DD strings, UTC) and
  Blocklist.DaysSeen (*int): optional, nil/empty when a feed does not report
  them (for example ipdata's own vpn feed).

Verification

Local, go1.24.7 linux/amd64, all run on the final tree before the push:

  • gofmt -l . prints nothing.
  • go vet ./... clean.
  • go test -race -covermode atomic -cover -v ./...: ok github.com/ipdata/go with coverage 88.8% of statements (baseline 88.3%); 25 top-level tests and 54 subtests PASS, 0 FAIL.
  • go run honnef.co/go/tools/cmd/staticcheck@latest ./... (staticcheck v0.8.2): no findings.
  • The new network tests (Test_newHTTPClient_http2, Test_client_connectionReuse, TestClient_CloseIdleConnections, TestClient_BulkLookup_staleConnectionRetry, TestClient_BulkLookup_freshConnectionNotRetried) were additionally run with -race -count=15: ok, no flakes.

Three sequential LookupWithContext calls on one Client to api.ipdata.co negotiated HTTP/2 (ALPN h2) and the second and third reused the connection.

CI: the workflow already has a pull_request trigger. Run https://github.com/ipdata/go/actions/runs/41621314653 (CI #5, head 1ce679c) completed with conclusion success: Test (Go 1.21), Test (Go 1.22), Test (Go 1.23), Test (Go 1.24) and Static Analysis all succeeded (the 1.21–1.23 jobs switch to the go1.24.7 toolchain, see "Noticed, not changed").

Noticed, not changed

  • LookupFields drops the fields query. LookupFieldsWithContext appends ?fields=... to the URL, but newGetRequestWithContext then assigns req.URL.RawQuery = q.Encode() with only api-key, which overwrites it, so the API always returns the full response. Test_client_LookupFields passes only because the test handler's full response also satisfies the assertions. Fix: put fields into the same url.Values (out of scope for this PR).
  • The CI matrix lists Go 1.21, 1.22 and 1.23 next to 1.24, but go.mod says go 1.24.7, so every job switches to the 1.24.7 toolchain through GOTOOLCHAIN=auto (which is also why http.HTTP2Config builds in all four jobs). The matrix could be reduced to 1.24 and later.
  • doc.go still sends issue reports to github.com/theckman/go-ipdata/issues.
  • Threat and Scores have no doc comments, and Scores is missing from the README's "Nested Types".
  • newHTTPClient() sets no overall request timeout (http.Client.Timeout or ResponseHeaderTimeout); callers rely on contexts.
  • errors.Wrapf(err, "failed to read body ...: %s", resp.Status, err) formats the error into the message it also wraps, so it appears twice in Error().

🤖 Generated with Claude Code

A Client already kept one *http.Client for its lifetime, but the custom
DialContext in newHTTPClient() switches off the automatic HTTP/2 setup of
net/http, so every request to api.ipdata.co went over HTTP/1.1. Set
ForceAttemptHTTP2 so h2 is offered through ALPN (with the HTTP/1.1 fallback
net/http provides) and configure HTTP/2 keepalive pings every 30 seconds
with a 15 second answer timeout, so idle multiplexed connections are health
checked instead of dying silently. The 60 second idle timeout and the 30
second TCP keepalive stay as they were.

Add Client.CloseIdleConnections (nil-guarded) so programs can release the
pooled connections at shutdown, and document the lifecycle: one Client per
process (or per API key or endpoint), safe for concurrent use, reused for
every lookup.

The bulk endpoint is a non-idempotent POST that net/http only replays when
nothing was written to the connection yet. When a pooled connection the
server had already dropped is picked, the request bytes go out and the read
fails before any response byte arrives; the request is now replayed exactly
once in that situation (detected with net/http/httptrace), never after
response bytes arrived, on an HTTP status, or after a cancellation or
timeout.

While there: close the response body of non-200 answers in RawLookup and
RawBulkLookup, set Version to 0.8.2 (the const still said 0.8.0 while the
last tag is v0.8.1), name github.com/ipdata/go instead of the old
theckman/go-ipdata path in the User-Agent, and fix the stale package example
in doc.go (import path, two return values, non-empty key).

Tests cover the negotiated protocol against an httptest TLS server with
HTTP/2 enabled, connection reuse across sequential lookups counted through
ConnState over both HTTP/1.1 and HTTP/2, CloseIdleConnections, and the bulk
replay against a scripted raw server (retried once, at most once, never
after response bytes, never on an HTTP status, never on a fresh connection).

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API now reports two more top-level booleans next to is_eu, and each
element of threat.blocklists may carry the dates on which the address was
seen.

IP and bulkIP gain IsSatellite and IsAnycast, copied through bulkToIP. The
API includes each key only when it is true, so a plain bool reads false
whenever the key is absent, which is the documented meaning. Blocklist
gains FirstSeen and LastSeen (calendar dates as YYYY-MM-DD strings, UTC,
deliberately not parsed into time.Time) and DaysSeen as *int: some feeds,
for example ipdata's own vpn feed, report none of the three, and nil keeps
"not reported" apart from a zero count. encoding/json ignores unknown keys,
so responses carrying further additions keep decoding.

The package has no field-name validation for LookupFields, so there is no
list to extend; the README field tables document the five new names.

Tests decode the three sample responses (satellite ISP, anycast address
with dated blocklist entries, VPN exit whose third entry has no dates)
through decodeIP, Lookup and BulkLookup, assert the false-when-absent
behaviour and the unknown-key tolerance, extend the bulkToIP test, and
check the omitempty encoding of a Blocklist without dates. gofmt also
normalises the whitespace of Threat, Blocklist and Scores in types.go,
which were indented with spaces.

Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jonathan-kosgei
jonathan-kosgei merged commit c9ee7d2 into master Nov 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants