🐛 Bug Report: http --download misinterprets Content-Length when Content-Encoding: gzip is set

🐛 Bug Report: `http --download` misinterprets `Content-Length` when `Content-Encoding: gzip` is set

Summary

When using httpie with --download to retrieve a file with Content-Encoding: gzip, the tool reports an "Incomplete download" even though the Content-Length is correctly set to the size of the compressed payload.

This behavior contradicts the HTTP/1.1 spec, which states that Content-Length must reflect the encoded (compressed) size when Content-Encoding is used.

Steps to Reproduce

Serve a GZIP-compressed JSON file with these headers:

Content-Type: application/json
Content-Encoding: gzip
Content-Disposition: attachment; filename="data.json.gz"
Content-Length: 5084527  # actual size of compressed content

Use HTTPie to download the file:

http --download GET http://localhost:3000/download

Observe this error:

Incomplete download: size=5084527; downloaded=42846965

Note: The server is not using chunked encoding and Content-Length is accurate.

Expected Behavior

httpie should:

Trust Content-Length as the size of the compressed body.
Simply save the file without decompressing or misinterpreting the length.

This is what browsers, curl, and wget all do correctly.

Actual Behavior

httpie seems to assume Content-Length is the uncompressed size, leading it to flag the download as incomplete when it receives more data than expected.

Environment

HTTPie version: httpie 3.2.4
OS: macOS / Linux (reproduced in both)
Server: Custom Go server (Fiber), but same issue with static gzipped files

References

[RFC 9110 § 8.6](https://www.rfc-editor.org/rfc/rfc9110#section-8.6):

“The Content-Length field indicates the number of octets in the body of the message after any content codings have been applied.”

Debug Info

http get "localhost:3000/metrics/admin/00:11:00:00:22:00/20250101T000000Z/download?format=json" Authorization:"Bearer ey....." --download --debug
HTTPie 3.2.4
Requests 2.32.4
Pygments 2.19.1
Python 3.13.5 (main, Jun 11 2025, 15:36:57) [Clang 17.0.0 (clang-1700.0.13.3)]
/opt/homebrew/Cellar/httpie/3.2.4_3/libexec/bin/python
Darwin 24.5.0

<Environment {'apply_warnings_filter': <function Environment.apply_warnings_filter at 0x10665cd60>,
 'args': Namespace(),
 'as_silent': <function Environment.as_silent at 0x10665cc20>,
 'colors': 256,
 'config': {'default_options': []},
 'config_dir': PosixPath('/Users/alex/.config/httpie'),
 'devnull': <property object at 0x106642110>,
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x10665ccc0>,
 'program_name': 'http',
 'quiet': 0,
 'rich_console': <functools.cached_property object at 0x105a0e690>,
 'rich_error_console': <functools.cached_property object at 0x106630c00>,
 'show_displays': True,
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': True,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

<PluginManager {'adapters': [],
 'auth': [<class 'httpie.plugins.builtin.BasicAuthPlugin'>,
          <class 'httpie.plugins.builtin.DigestAuthPlugin'>,
          <class 'httpie.plugins.builtin.BearerAuthPlugin'>],
 'converters': [],
 'formatters': [<class 'httpie.output.formatters.headers.HeadersFormatter'>,
                <class 'httpie.output.formatters.json.JSONFormatter'>,
                <class 'httpie.output.formatters.xml.XMLFormatter'>,
                <class 'httpie.output.formatters.colors.ColorFormatter'>]}>

>>> requests.request(**{'auth': None,
 'data': RequestJSONDataDict(),
 'headers': <HTTPHeadersDict('User-Agent': b'HTTPie/3.2.4', 'Authorization': b'Bearer eyJh...', 'Accept-Encoding': b'identity')>,
 'method': 'get',
 'params': <generator object MultiValueOrderedDict.items at 0x106a7d030>,
 'url': 'http://localhost:3000/metrics/admin/00:11:00:00:22:00/20250101T000000Z/download?format=json'})

HTTP/1.1 200 OK
Content-Disposition: attachment; filename="001100002200-20250101T000000Z.alog.json.gzip"
Content-Encoding: gzip
Content-Length: 5084527
Content-Type: application/json
Date: Thu, 10 Jul 2025 16:22:21 GMT

Downloading to 001100002200-20250101T000000Z.alog.json.gzip
Done. 42.8 MB in 00:0.07174 (597.2 MB/s)

http: error: Incomplete download: size=5084527; downloaded=42846965

Fix: Skip Content-Length for progress tracking when Content-Encoding …

Fix false "Incomplete download" error with Content-Encoding

Fix httpie#1642: Don't flag download as interrupted when Content-Enco…

Fix false 'Incomplete download' error when Content-Encoding is set

downloads: don't use Content-Length as progress total when Content-En…

Opened #1816 for this.

Root Cause Analysis

The bug is in httpie/downloads.py, specifically the Downloader.start() method.

The Problem

File: httpie/downloads.py, line 219-224 and 271-275

There is even an existing FIXME comment at line 219:

# FIXME: some servers still might sent Content-Encoding: gzip
# <https://github.com/httpie/cli/issues/423>

The chain of events:

pre_request() (line 193) sets Accept-Encoding: identity to prevent server-side compression
Some servers ignore this and respond with Content-Encoding: gzip anyway
The server correctly sets Content-Length: 5084527 (the compressed size per RFC 9110 §8.6)
requests library auto-decompresses via iter_content() — chunks yielded are decompressed
chunk_downloaded() (line 278) tracks decompressed bytes in status.downloaded

The interrupted property (line 271-275) compares:

return (
    self.finished
    and self.status.total_size  # 5,084,527 (compressed)
    and self.status.total_size != self.status.downloaded  # 42,846,965 (decompressed!)
)

This is always True when compression is active → false "Incomplete download" error.

The Fix

In Downloader.start(), after reading Content-Length, check if Content-Encoding indicates the response is compressed. If so, skip the size comparison by setting total_size = None:

# In httpie/downloads.py, replace lines 219-224 with:

# If Content-Encoding indicates compression, skip size validation.
# requests auto-decompresses, so Content-Length (compressed) != downloaded (decompressed).
# See: https://github.com/httpie/cli/issues/423
content_encoding = final_response.headers.get("Content-Encoding", "identity")
if content_encoding != "identity":
    total_size = None
else:
    try:
        total_size = int(final_response.headers["Content-Length"])
    except (KeyError, ValueError, TypeError):
        total_size = None

This is safe because:

The progress display already handles total_size = None gracefully (shows a spinner instead of progress bar)
The interrupted check on line 274 short-circuits when total_size is falsy
curl and wget handle this the same way — they skip size validation when content is encoded

I can submit a PR. Want me to go ahead?

Navigation Menu

Repository navigation

🐛 Bug Report: http --download misinterprets Content-Length when Content-Encoding: gzip is set #1642