Conversation
….request Add urllib.request.HTTPGzipHandler, an opt-in handler that requests gzip-compressed responses and transparently decodes them. The handler is not one of the default build_opener() handlers, so urlopen()'s behaviour is unchanged unless it is added explicitly. When the request has no Accept-Encoding header, the handler adds "Accept-Encoding: gzip"; if the response then carries "Content-Encoding: gzip", the body is decoded incrementally as it is read and the Content-Encoding and Content-Length headers, which describe the compressed body, are removed. A request that already carries an Accept-Encoding header is left untouched, so a caller that means to handle the compressed response itself still receives it. Only gzip is handled, and the zlib module is required.
Documentation build overview
406 files changed ·
|
2 of 20 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
urllib.request.HTTPGzipHandler, an opt-in handler that sendsAccept-Encoding: gzipand transparently decodes aContent-Encoding: gzipresponse body. It's the long-standing request in this issue (originally bpo-1508475). Ref #43521.Design — the thing that kept this open for 20 years wasn't code, it was one unmade design call: an opt-in handler versus default-on decoding down at
http.client. This PR is a concrete answer to that call, not a claim on it — the call stays the maintainers'. I went opt-in: the handler is not added tobuild_opener()'s default handler list, sourlopen()behaves identically for everyone who doesn't ask for it; you get it withbuild_opener(HTTPGzipHandler). I chose opt-in over on-by-default because the strongest unanswered objection in the thread was Martin Panter's (2014): transparent decoding at this layer would silently decompress a deliberately-gzipped download such as a.tar.gz, and would break partial Range requests. Making the decode the caller's explicit choice sidesteps that. Senthil Kumaran (2012) preferred on-by-default with a flag to switch it off — that's a reasonable alternative; I can flip the default to that if you'd rather — the mechanism is identical either way.No double-decode: the handler decodes only when it added the
Accept-Encodingheader itself. If the request already carries a caller-suppliedAccept-Encoding, the response is returned untouched — this answers the double-decode objection from Serhiy Storchaka (2012) and Martin Panter (2014), whose own code already decodes gzip fromurlopen(). When it does decode, it removesContent-EncodingandContent-Length, which describe the compressed body (Serhiy Storchaka and Senthil Kumaran, 2012, matching RFC guidance).Streaming: the body is decompressed incrementally with
zlib.decompressobj(16 + zlib.MAX_WBITS)as the caller reads, so aread()with no size limit can't bypass the decode (Martin Panter, 2014) and there's no full-buffer decompression bomb — memory is bounded per read. I deliberately did not reusexmlrpc.client's gzip helper (Martin Panter, 2015): it reads the whole body into aBytesIOand caps it at 20 MiB, which suits XML-RPC payloads but not arbitrary-size urllib downloads. A response made of several concatenated gzip members is decoded in full — not just the first — and trailing bytes that aren't another member are ignored, matching the reader's existing leniency. A truncated gzip stream raisesEOFError; a truncated transfer still surfaceshttp.client.IncompleteRead.Scope: gzip only. I left
deflateout on purpose — its raw-vs-zlib-wrapper ambiguity is a known interop trap, so requesting it is easy but decoding it reliably across servers isn't; it can be a follow-up.zlibis required, and constructing the handler without it raisesImportError, which for an opt-in handler is clearer than a silent no-op.Tests cover opt-in-ness (the handler is not in
build_opener()'s defaults but is accepted explicitly), header injection versus non-override, round-trip decode, chunked/streaming reads,Content-Encoding/Content-Lengthremoval, full pass-through when the caller set their ownAccept-Encoding, non-gzip pass-through, concatenated-member decode, trailing-junk tolerance, and the truncated-stream error.test_urllib2,test_urllibandtest_urllib2_localnetpass. Docs, a What's New entry and a NEWS blurb are included.