Repository navigation
feat!: output limit reaches the decoder, streaming rar2/rar3, multi-frame zstd/lz4, safer CLI - #172
Merged
Merged
Conversation
LimitedDecoder bounds the bytes handed to the caller, but a decoder that builds a block or the whole result in memory first has committed that memory before the wrapper sees a byte. Decoder gains set_output_limit (default: ignore), bridged to RawDecoder::raw_set_output_limit, and LimitedDecoder::new passes its budget down. Decoders that buffer will override it to fail with OutputLimitExceeded up front. BREAKING CHANGE: Decoder has a new provided method; LimitedDecoder::new now calls it on the wrapped decoder. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rint The Snappy decoder buffers the whole block on `decode` and builds the whole result in `finish`, so a `LimitedDecoder` budget used to fire only after both the compressed input and the decoded output were in memory (v0.7.8 audit: 532 MiB committed from a 25 MB block under a 4 KiB cap). The decoder now overrides `raw_set_output_limit`. The uncompressed length is the varint that opens the block, so: - `raw_decode` checks it as soon as the varint is complete, looking at the already-buffered bytes plus the incoming slice *before* buffering that slice. An oversized block fails with `OutputLimitExceeded` from the call that delivers the last varint byte; nothing after it is ever copied. The refusal is sticky until `reset`. - `decompress_block` (the `finish` path) makes the same check before any allocation, which covers a limit set after input was buffered. The limit is configuration and survives `reset`. A block that decodes to exactly the limit succeeds; with no limit (`u64::MAX`, the default) nothing changes, and a malformed varint is still reported by `finish` as `Corrupt`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…header The standalone Huffman decoder buffers the whole stream on `decode` and decodes it in one go in `finish`, so a `LimitedDecoder` budget used to fire only after the compressed input and the decoded output were both in memory (v0.7.8 audit: 590 MiB committed from a 67 MB stream under a 4 KiB cap). The decoder now overrides `raw_set_output_limit`. The original length is the LEB128 varint that opens the stream, so: - `raw_decode` checks it as soon as the varint is complete, looking at the already-buffered bytes plus the incoming slice *before* buffering that slice. An oversized stream fails with `OutputLimitExceeded` from the call that delivers the last varint byte; nothing after it is ever copied. The refusal is sticky until `reset`. - `decode_stream` (the `finish` path) makes the same check right after parsing the varint, ahead of the table and of both output paths (the single-symbol `resize` and the general decode loop). This covers a limit set after input was buffered. The limit is configuration and survives `reset`. A stream that decodes to exactly the limit succeeds; with no limit (`u64::MAX`, the default) nothing changes, and a malformed varint is still reported by `finish` as `Corrupt`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The BWT decoder buffers the whole block stream on `decode` and inverts every block in `finish`, so a `LimitedDecoder` budget used to fire only after the input, the decoded output and the per-block LF-mapping vector were all in memory (v0.7.8 audit: 1 GiB committed from a 537 MB stream under a 4 KiB cap). The transform is a permutation, so each block's `len` header field is its decoded size. The decoder now overrides `raw_set_output_limit` and: - `raw_decode` follows the block headers as input arrives, reading only the 4 `len` bytes of each and skipping the bodies by offset. The scan runs over the already-buffered bytes plus the incoming slice *before* buffering that slice, so when the running total passes the limit the call fails with `OutputLimitExceeded` without copying the block that crosses it (or anything else in that slice), and before any block has been inverted. The refusal is sticky until `reset`. - `decode_all` (the `finish` path) keeps the same running total and refuses before inverting the block that crosses, which covers a limit set after input was buffered. Repeated `finish` calls after that do not build more output. A zero `len` stops the scan and is left for `finish` to report as `Corrupt`, as before. The limit is configuration and survives `reset`. A stream that decodes to exactly the limit succeeds; with no limit (`u64::MAX`, the default) the scan is skipped and nothing changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The CLI never used `compcol::limit::LimitedDecoder`, so `compcol -d` on a 1 KiB zstd/xz bomb wrote until the disk was full: the library's documented defence against decompression bombs was unreachable from its own binary. `--max-output SIZE` (also `--max-output=SIZE`) wraps the decoder in `LimitedDecoder`, which also passes the budget down to the codec via `Decoder::set_output_limit`. SIZE is a plain byte count or one with a K/M/G/T suffix (powers of 1024, case-insensitive, `KiB` spellings accepted); malformed or overflowing values are usage errors. Going over the limit is a runtime error (exit 1) that names the flag and the limit, and takes the same cleanup path as any other decode failure: the partial output file is removed and the input is kept. Output of exactly SIZE bytes is fine. There is no limit by default, and the flag is a usage error when compressing. Codec errors are now carried inside the `io::Error` as the typed `compcol::Error` (their printed text is unchanged) so the limit error can be recognised. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
After the decoder reported end-of-stream the CLI kept reading and threw
the rest of the input away, so
cp a.gz g.gz; echo garbage >> g.gz; compcol -d g.gz
exited 0 and deleted g.gz. Worse, for formats whose decoder stops at the
first frame (zstd, lz4-frame, zlib, brotli, ...) a file made of two
concatenated streams silently decoded to the first one only, and in-place
mode then removed the input holding the rest.
`stream_decode` now stops at `StreamEnd` and reports whether any input was
left: the unconsumed part of the current buffer, or else a single probing
one-byte read. Nothing behind it is read or buffered. `run` turns that
into "trailing data after the end of the compressed stream", exit 1, via
the normal failure path (output removed, input kept).
The rule is the same in every output mode: in-place, `-o`, `-c` and
stdin-to-stdout all fail. With stdout the decoded bytes have already been
written, but the exit status must not depend on where the output went.
`--ignore-trailing` restores the old leniency, with a warning on stderr,
and still reads the input to its end so an upstream writer is not killed
by a closed pipe. It is a usage error without `-d`.
Not trailing data: further gzip members, xz streams and bzip2 streams,
which those decoders already continue into; and whatever a decoder with
no end marker consumes itself.
BREAKING CHANGE: `compcol -d` now exits 1 (and keeps the input, removes
the output) when bytes follow the end of the compressed stream, where it
used to succeed and drop them. This includes concatenated frames in
formats whose decoder handles a single frame (zstd, lz4-frame), which
were previously truncated to the first frame without notice. Pass
`--ignore-trailing` for the previous behaviour.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ng it
`-f` opened the destination with create+truncate, which
(a) destroyed an existing file even when the run then failed (corrupt
input, --max-output, disk full): the old content was gone and an
empty file was left in its place;
(b) followed a symlink at the output path and overwrote its target:
`ln -s tgt link.gz; compcol -t gzip -f link` clobbered `tgt`;
(c) decided whether the file was "ours to clean up" with an exists()
check racing the open.
A `--force` run now writes to a new file created with `create_new` in the
destination's directory (`.compcol-<pid>-<random>.tmp`, the random part
from std's randomly seeded hasher, retried on collision) and renames it
over the destination only after the stream has finished and flushed. On
any failure the temporary file is removed and an existing destination is
left exactly as it was. `rename` replaces a symlink rather than following
it, which is what gzip -f does. The destination's existence is no longer
consulted at all.
Permissions follow the v0.7.9 rules, applied to the temporary file before
the rename: created 0600 on unix when a mode will be applied, then given
the input file's mode. With no input file (stdin) a replaced regular file
keeps its own mode, as overwriting in place used to, and a new one gets
the umask default.
Two cases are deliberately not routed through a rename:
- Without `-f` the destination is still created with `create_new` and
written in place. That is already race-free and never follows a symlink,
and nothing can pre-exist there, so there is nothing to preserve. The
alternative no-replace finish (temporary file + `hard_link`) would fail
on file systems without hard links (FAT/exFAT, some network mounts).
- A `-f` destination that exists and is neither a regular file nor a
directory (`-f -o /dev/null`, a FIFO, also through a symlink) is opened
and written directly, without create or truncate: it cannot be replaced
by a rename and holds nothing to protect. If the opened file turns out
to be a regular file after all (swapped in between) the run refuses.
`-f` onto a directory, or onto a symlink to one, fails up front.
The existing regression test for a failing `--force` run only asserted
that the target still existed (truncated); it now asserts its content is
intact and that no temporary file is left.
BREAKING CHANGE: `compcol -f` no longer writes through a symlink at the
output path (the link is replaced by the output file) and no longer
rewrites the existing file in place: the destination becomes a new file,
so other hard links to the old one keep the old content, its owner/group
and extended attributes are not carried over, and the output directory
must be writable (a writable file in a read-only directory can no longer
be overwritten).
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`compcol -d` without `-t` picks the codec with `factory::detect`. For
archive magics that function returns a member codec's name purely as a
format hint: `rar5` / `rar3` (or `rar2` / `rar1`, depending on features)
for "Rar!\x1A\x07", and `arsenic` for StuffIt ("SIT!"…"rLau" and
"StuffIt"). The CLI took the hint literally and fed the whole archive —
its headers and directory included — to a decoder for one member's raw
stream, ending in a baffling "container header is malformed" or, worse,
in whatever the codec made of the bytes.
Auto-detection now stops on those with a usage error (exit 2) before any
output is created:
looks like a RAR archive; compcol decodes single compressed streams,
not archives — pass -t to force
The container magics are matched in the CLI itself, so the message does
not depend on which rar/arsenic features are compiled in (without them
the answer used to be "could not detect format"); the hint names `detect`
returns for them are checked as well. Every other detected format is
dispatched as before, and an explicit `-t` is always taken at its word.
BREAKING CHANGE: `compcol -d` without `-t` on input starting with a RAR
or StuffIt archive magic is now a usage error (exit 2) instead of running
the rar*/arsenic member decoder over it. Pass `-t ALGO` to run a codec on
such input.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Trigger: a member whose declared unpack size is large (it comes from the container header, so it is attacker-chosen) decoded through LimitedDecoder or any caller that reads only part of the output. Effect: `decode` only buffered input, and the first `finish` decoded the entire member into an internal Vec before handing out a byte. The output limit never saw anything until the whole declared size had been built: 256 KiB of input under a 1 KiB LimitedDecoder grew RSS by 400-516 MiB. Fix: decode incrementally. The 1 MiB sliding window doubles as the pending-output queue: bytes are written to the window only, a `pending` count marks how many of them the caller has not taken yet, and symbols are decoded only while fewer than 64 KiB are pending (one match of at most 260 bytes may overshoot), so held output is bounded independently of the declared size. `decode` now hands bytes out as they are produced and reports StreamEnd once the declared size is reached; `finish` drains and turns "out of bits" into a truncation error. Input is still buffered (`decode` accepts everything, so a call with an empty output slice keeps queueing as before), but the prefix the bit reader has moved past is dropped, so the buffer follows the compressed size actually outstanding. A symbol or block header that needs more bits than are buffered is retried from its saved bit position: every step mutates decoder state only after its last bit read, and the block header now works on a copy of the length table that is committed once the whole header is in. LZ blocks, audio blocks, "keep lengths" delta tables and mid-stream table re-reads all go through the same step function. The decoder has no cross-member solid mode, before or after. A decode error is reported after the bytes decoded before it have been handed out. The `unpack_size > usize::MAX` refusal on 32-bit hosts is gone, since nothing is sized from it any more. Tests: a token-level stream builder with its own reference model (all LZ symbol kinds, both table shapes, 1.3 MiB so the window wraps), an audio -> LZ -> audio stream pinned to the previous decoder's output, a chunking differential (one-shot / 1-byte input / 1-byte output / both) over every fixture including the two real RAR 2.60 ones, bounded-buffer assertions via a doc(hidden) `buffered_output_len`, the LimitedDecoder audit scenario, exact stop at the declared size, and truncation. BREAKING CHANGE: `rar2::Decoder::decode` now writes output and can return `Status::StreamEnd`; it no longer consumes input once the declared size has been produced. Callers that relied on `decode` writing nothing and on `finish` producing the whole member must handle output from both. On a corrupt or truncated member the bytes decoded before the error are now delivered before the error is returned. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Trigger: `Rar2::decoder()`, `rar2::Decoder::new()` or the factory's
"rar2" decoder, fed anything at all.
Effect: `new()` left `unpack_size = 0` with a `have_unpack_size` flag
that nothing ever read, so every stream - valid RAR 2.x data or garbage
- "decoded" as a complete zero-length stream: StreamEnd, 0 bytes, no
error. Data was silently dropped.
Fix: without a declared size the decoder returns `Error::Unsupported`
as soon as it is given a non-empty input, and keeps returning it from
`decode` and `finish` until `reset`. A stream with no input at all is
still an empty success.
Decoding "until the input runs out", as the RAR3 default does, was
considered and rejected: a RAR 2.x member has no end marker (unrar's
Unpack20 loop is driven purely by the size from the file header), and
the data ends on a byte boundary, so up to 7 padding bits - plus
whatever a container reader passes after the member - would be decoded
as extra symbols. With a short code for a literal that is spurious
trailing output on a perfectly valid member; there is no point at which
the decoder could stop soundly.
BREAKING CHANGE: `rar2::Decoder::new()`, `Rar2::decoder()` and
`factory::decoder_by_name("rar2")` used to accept any input and report
an empty stream; they now fail with `Error::Unsupported` on non-empty
input. Construct with `Decoder::with_unpack_size`, or call
`set_unpack_size` before the first `decode`.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder stopped after the first frame: `cat a.zst b.zst` decoded to the content of `a` with `StreamEnd`, and `vec::decompress_to_vec` returned Ok, silently dropping everything after it. A Skippable_Frame was rejected as `Unsupported`, which made every `pzstd` output (it starts with one) undecodable. The decoder is now multi-frame by default, following the gzip decoder's handling of multi-member files: - After a frame, the next byte decides. The first byte of a frame magic (0x28, or 0x50..=0x5F for a Skippable_Frame) starts another frame, which then has to be valid: a wrong magic is `BadHeader`, an input that stops inside it `UnexpectedEnd`. Any other byte ends the stream, and the trailing bytes stay unconsumed (gzip's precedent; `zstd -d` rejects such input). - `decode` cannot report `StreamEnd` at a frame boundary with no byte to look at; it returns `InputEmpty` and `finish` reports the end. - Each frame starts from scratch: empty window, no Huffman / FSE tables to repeat, initial repeat offsets, its own Frame_Content_Size and content checksum. - Skippable_Frames are skipped wherever a frame may start. The payload is counted down and discarded as it arrives, never buffered. A stream of skippable frames alone decodes to nothing, as with `zstd -d`; an empty input is still `UnexpectedEnd`. `DecoderConfig::multi_frame` (default `true`, builder `with_multi_frame`) turns this off for containers that embed one frame followed by other data: `decode` then reports `StreamEnd` right after the frame's last byte and consumes nothing past it. Leading Skippable_Frames are skipped in that mode too. `DecoderConfig` becomes `#[non_exhaustive]` with `with_*` builders, like bzip2's, and is now what `Zstd::decoder_with` takes. Tests: `skippable_frame_magic_is_unsupported` encoded the old behaviour and is replaced by `skippable_frame_magic_is_skipped`; two struct literals of `DecoderConfig` now use the builder. BREAKING CHANGE: the zstd decoder decodes every frame of a stream instead of stopping after the first, and skips Skippable_Frames instead of failing with `Unsupported`; at a frame boundary `decode` returns `InputEmpty` and only `finish` (or a following non-frame byte) yields `StreamEnd`. Use `DecoderConfig::default().with_multi_frame(false)` for the old one-frame behaviour. `<Zstd as Algorithm>::DecoderConfig` is now `zstd::DecoderConfig` instead of `()`, and `zstd::DecoderConfig` is `#[non_exhaustive]` with a new `multi_frame` field: build it with `DecoderConfig::default().with_max_window_size(..)` rather than a struct literal. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The LZ4 frame decoder stopped after the first frame: `cat a.lz4 b.lz4` decoded to the content of `a` with `StreamEnd`, silently dropping the rest, and a skippable frame was a `BadHeader`. The decoder is now multi-frame by default, with the same rules as the gzip decoder for multi-member files (and as the zstd decoder): - After a frame, the next byte decides. The first byte of a frame magic (0x04, or 0x50..=0x5F for a skippable frame) starts another frame, which then has to be valid: a wrong magic is `BadHeader`, an input that stops inside it `UnexpectedEnd`. Any other byte ends the stream, and the trailing bytes stay unconsumed (gzip's precedent; `lz4 -d` rejects such input). - `decode` cannot report `StreamEnd` at a frame boundary with no byte to look at; it returns `InputEmpty` and `finish` reports the end. - Each frame has its own descriptor, window, content size and checksums; a linked block cannot reach into the previous frame. - Skippable frames (magic 0x184D2A50..=0x184D2A5F) are skipped wherever a frame may start, their payload counted down and discarded as it arrives. An empty input still decodes to nothing. `DecoderConfig` gains `multi_frame` (default `true`, builder `with_multi_frame`) and `Decoder::with_config`. With `false` the decoder stops at the end of the first frame for containers that embed one frame followed by other data: `decode` reports `StreamEnd` right after the EndMark (and content checksum) and consumes nothing past it. Leading skippable frames are skipped in that mode too. BREAKING CHANGE: the LZ4 frame decoder decodes every frame of a stream instead of stopping after the first, and skips skippable frames instead of failing with `BadHeader`; at a frame boundary `decode` returns `InputEmpty` and only `finish` (or a following non-frame byte) yields `StreamEnd`. Use `DecoderConfig::default().with_multi_frame(false)` for the old one-frame behaviour. `lz4::frame::DecoderConfig` is no longer a unit struct: it is `#[non_exhaustive]` with a `multi_frame` field, so write `DecoderConfig::default()` instead of `DecoderConfig`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
QpackDecoder::feed_encoder_stream applied instructions as it parsed them and returned UnexpectedEnd, with no consumed count, when a chunk ended inside an instruction. The encoder stream is a QUIC byte stream, so that is the normal case, and a caller that buffered the chunk and retried had the earlier inserts applied twice: the dynamic table silently went out of sync with the peer's. The decoder now keeps the unparsed tail itself. feed_encoder_stream is called with each chunk as it arrives and returns Ok when the chunk ends mid-instruction; every instruction is applied exactly once however the bytes are split. encoder_stream_pending() reports how many bytes of a partial instruction are held, for the caller that learns the stream has ended. An instruction is first measured from its integers alone and only decoded once complete, so feeding a long instruction a byte at a time stays linear. What is held is bounded by the current table capacity: an insert whose strings cannot fit (by raw length, or by the least a Huffman string of that length can decode to) is Corrupt as soon as the length prefix is read, instead of being buffered. An error on the encoder stream is a connection error and is now final: the partial instruction is dropped and later calls return the same error. QpackEncoder has no decoder-stream feeder, so nothing changes there. BREAKING CHANGE: QpackDecoder::feed_encoder_stream no longer returns Error::UnexpectedEnd for a chunk that ends inside an instruction; it buffers the partial instruction and returns Ok(()). Callers that kept their own reassembly buffer must drop it and feed each byte once; use encoder_stream_pending() to detect a truncated stream. After any error, further feed_encoder_stream calls return that error again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The .lzma (LZMA-alone) and ZIP method-14 decoder had no configuration
and a hard-coded 64 MiB dictionary cap. The cap was a silent clamp: a
header declaring more was decoded with a 64 MiB window and failed with
Corrupt at the first match reaching further back, so files written with
a larger dictionary (xz --format=lzma --lzma1=dict=128MiB, 7-Zip's
larger settings) could not be decoded and the caller could neither opt
in nor ask for a tighter bound. The dictionary was also allocated in
full from the header, and again on every call while the five range
coder init bytes were still missing.
lzma::DecoderConfig { max_dict_size } now bounds the dictionary a header
may declare: 64 MiB by default (DEFAULT_MAX_DICT_SIZE), up to the
format's 4 GiB - 1 (MAX_DICT_SIZE; 2 GiB - 1 on 32-bit targets), with
values under 4 KiB meaning 4 KiB. A header over the limit is
Error::Unsupported as soon as it has been read, as zstd does for its
window, and stays so until reset. Constructors: DecoderConfig::new /
with_max_dict_size, Decoder::with_config, Decoder::zip_with_config, and
a chainable Decoder::with_max_dict_size that also serves zip() and
zip_from_gp_flags(); Decoder::max_dict_size() reads the limit back.
Lzma's Algorithm::DecoderConfig is the new type.
The dictionary now grows with the bytes produced, like the LZMA2 core
since v0.7.9, so a stream costs the smaller of its declared dictionary
and its decoded length: a stream declaring 1 GiB and producing a few
bytes allocates 4 KiB.
Checked against xz 5.8.3 (--format=lzma, dict 4 KiB to 1 GiB).
BREAKING CHANGE: <lzma::Lzma as Algorithm>::DecoderConfig is now
lzma::DecoderConfig instead of (). A .lzma stream or ZIP method-14
entry whose header declares a dictionary larger than 64 MiB is now
rejected with Error::Unsupported by default; before, it was decoded
with a 64 MiB window and only failed (Corrupt) if it actually used a
longer distance. Raise DecoderConfig::max_dict_size to decode such
streams.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder builds the whole fork in memory before handing any of it out, bounded only by the out-of-band length or a 256 MiB internal cap. A caller's LimitedDecoder budget could not reach that: an 18 KB payload under a 1 KiB cap still committed ~254 MiB before the wrapper saw the first byte. Override raw_set_output_limit, keep the limit across reset, and check it before every literal and before every match copy, returning Error::OutputLimitExceeded instead of growing past it. A stream that decodes to exactly the limit still succeeds, and with no limit set the behaviour is unchanged. The in-module bit writer now also emits matches, so the tests can build the audit's bomb (a literal followed by maximum-length distance-1 runs). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder buffers the fork and decodes it in one shot into an internal Vec capped only at 256 MiB, so a LimitedDecoder budget was enforced after the memory had been committed, and every geometric NeedMore retry rebuilt the same output from scratch. Override raw_set_output_limit (kept across reset) and thread the limit into pipeline::decode_stream: - every output byte is checked against the limit before it is appended, returning Error::OutputLimitExceeded; - a block is abandoned as soon as it holds more BWT bytes than the remaining budget can absorb. The final RLE drops at most one byte in five, so a block longer than 5/4 of the remaining budget is certain to overrun; refusing it in the token loop avoids the block scratch (up to 16 MiB) and its inverse-BWT table (4 bytes per block byte). A stream decoding to exactly the limit still succeeds; with no limit set nothing changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The standalone .ppmd decoder is buffer-then-drain: the whole output (up to its 64 MiB ceiling) is decoded into memory at finish, so a LimitedDecoder budget only took effect after that. A 24 KB stream under a 1 KiB cap still grew the process by ~61 MiB. The framing header declares the decoded length and decoding runs for exactly that many symbols, so the limit can be applied to the header: override raw_set_output_limit (kept across reset) and reject a declared length above it with Error::OutputLimitExceeded as soon as the 11 header bytes have arrived, before the model arena or any output is allocated. A stream declaring exactly the limit still decodes; with no limit set nothing changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A PPMd header names the size of the model arena and the decoder
allocates and zero-fills all of it as soon as the header is read: up to
255 MiB from byte 1 of the standalone framing, up to 256 MiB from the
2-byte ZIP method 98 header. ZipDecoder::new(1).decode(&[0xF5, 0x0F], ..)
committed ~265 MB, and a caller had no way to refuse.
Add a cap, checked before the arena is allocated (or reused); a header
asking for more is Error::Unsupported, the variant these decoders
already return when the arena cannot be allocated:
- ppmd::DecoderConfig { max_memory } (non_exhaustive, with_max_memory
builder) as Ppmd's Algorithm::DecoderConfig, and
Decoder::with_config. The standalone decoder rejects at the 11-byte
header, in decode, without waiting for finish.
- ZipDecoder::with_max_memory, a builder usable after new() or
until_end_marker(); the setting survives reset.
- ppmd::MAX_MEMORY (256 MiB), the default for both, which accepts every
valid header, so behaviour is unchanged unless a cap is set.
The module docs gain a Memory section stating the default and what a
hostile header costs without a cap. Ppmd7/Ppmd8 and the RAR3 path that
drives Ppmd7 directly are untouched.
BREAKING CHANGE: <Ppmd as Algorithm>::DecoderConfig is now
ppmd::DecoderConfig instead of (). Ppmd::decoder() and Decoder::new()
are unchanged; code calling Ppmd::decoder_with(()) must pass
DecoderConfig::default().
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A frame block is decoded whole into an internal buffer before any of it is handed out, and the only cap on it was the frame's own block size, which a BD code can put at 256 MiB. A 138-byte frame under a 1 KiB LimitedDecoder budget still grew the process by ~239 MiB. Override raw_set_output_limit (kept across reset) and track the budget left after the blocks decoded so far: - a compressed block is decoded with cap = min(frame block size, remaining budget). block::decode_compressed_block already checks its cap before every append, so an over-long block fails before it is built; - a stored block longer than the remaining budget is refused at its size word, before it is gathered. To tell the two caps apart, decode_compressed_block now reports cap overflow as OutputLimitExceeded (it is the only source of that error in the block decoder) and the frame decoder maps it back to Corrupt when the cap that was hit is the frame's block size, so a malformed block still reads as Corrupt with or without a generous limit. The in-module block test for that case follows the internal contract; a new frame- level test pins the public behaviour. A frame decoding to exactly the limit still succeeds; with no limit set nothing changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The frame header's BD code decides how large a block the decoder will
build in memory, up to 256 MiB, and a block of long matches reaches that
from almost no input. An output limit now bounds it for callers who have
one, but a caller could not simply say 'no frame with blocks over N'.
Add lz5::DecoderConfig { max_block_size } (non_exhaustive, with a
with_max_block_size builder) as Lz5's Algorithm::DecoderConfig, plus
Decoder::with_config. A frame whose BD code names a larger block is
rejected with Error::Unsupported when its header is read, after the
header has been validated (a bad header checksum still reports as
such) and before any block is gathered. The setting survives reset.
The default is lz5::MAX_BLOCK_SIZE (256 MiB, now public), which accepts
every frame, so behaviour is unchanged unless a cap is set. The module
docs gain a Memory section with the default and what a hostile header
costs without it.
BREAKING CHANGE: <Lz5 as Algorithm>::DecoderConfig is now
lz5::DecoderConfig instead of (). Lz5::decoder() and Decoder::new() are
unchanged; code calling Lz5::decoder_with(()) must pass
DecoderConfig::default().
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e fork The decoder decoded the whole fork into a Vec at finish and then drained it, so its memory followed the decoded size (up to the 256 MiB open- ended cap, or the supplied length) even though method 13 only ever looks back 64 KiB. Decode on demand instead. The token loop becomes a resumable Stream (the three codes, a bit position in the buffered payload, and the tail of the output); each finish/decode call decodes whole tokens until about 32 KiB is pending, copies what the caller's buffer takes, and drops history that no match can reach, always leaving a full window so window::emit_match's bounds checks keep their meaning. Beyond the compressed payload the decoder now holds about 230 KiB at most, whatever the stream expands to. BitReader gains at()/bit_pos() to resume. Unchanged: the payload is still buffered until finish (the stream's end is only known then), the supplied length is still where decoding stops, the open-ended 256 MiB cap still rejects longer streams with Corrupt, the output limit is still enforced token by token, and every error variant is as before. Changed, as for any streaming decoder: an error that lies part way into a stream (truncation, a bad distance, a length mismatch, the limit) is now reported after the bytes decoded before it have been handed out, rather than before any output. Tests: an 8 MiB bomb drained in 4 KiB chunks with the internal buffer checked to stay under 512 KiB; random literal/match streams (600 KiB, history trimmed several times) against a reference model at five chunk sizes, with and without a known length; truncation after a prefix. The test writer learns general matches for that. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ligning empty metadata block The brotli encoder inherited the default no-op raw_flush, and it keeps up to 64 KiB of input pending until finish, so a peer could not decode what had been fed until the stream ended. flush now writes the pending input as complete ISLAST=0 meta-blocks and then an empty metadata meta-block (RFC 7932 section 9.2: ISLAST 0, MNIBBLES 3, reserved 0, MSKIPBYTES 0, zero padding to the byte boundary), which is what the reference encoder's BROTLI_OPERATION_FLUSH writes. A decoder holding the output so far can produce every byte fed so far; later encode calls continue the stream and finish ends it as before (with the empty last meta-block when nothing is pending). All qualities go through the same path. Whether a flush owes anything is derived from the encoder state (no header yet, input pending, or bitstream not byte-aligned) and not remembered between calls, so a flush cut short by OutputFull and followed by encode is not taken for a finished one: the input accepted since is flushed by the next flush. A flush with nothing pending on an aligned stream writes nothing, like the reference encoder. Flushing after finish is an error, as for deflate. Flush::Full writes the same bytes as Flush::Sync: this encoder compresses each meta-block on its own, no match reaches into an earlier one, so there is no history to drop. Streams that are never flushed are byte-for-byte what they were; the encode and finish paths are untouched. Tests: pieces of several sizes with a flush after each, at every quality, decoding the output so far without finish; first-call flush, repeated flush, one-byte output buffer, interrupted flush, flush then finish. tests/sync_flush.rs holds brotli to the interrupted-flush contract of the deflate family (the helper now takes the format's sync-point check; deflate's is unchanged). Streams and flush-point prefixes were also checked against libbrotli 1.1.0 during development. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder used to run the whole member into one buffer on the first `finish` and drain that afterwards, so nothing came out until everything had been decoded, the full declared size was materialised whatever cap the caller had set (hundreds of MiB under a 1 KiB `LimitedDecoder`), and memory was O(compressed + decompressed). Decoding is now resumable between symbols. The member's state lives in the context (`Phase`, the bit reader, the undelivered output with its member offset), `finish` decodes until bytes are ready, hands them out, and picks up at the next symbol on the following call. Output is released at "release passes", which run at stream positions that depend only on the stream, never on the caller's buffer sizes: once a dictionary's worth (4 MiB - 260, the reference decoder's own lag between unpacking into its window and writing it out) is held, and at the end of the member. A pass runs the in-band filters whose windows are complete, in declaration order as before, and makes final every byte in front of the earliest window still pending. Members smaller than the dictionary are therefore still released in one piece at their end, with the same filter/reset semantics and the same "error before any output" behaviour as before; larger ones stream with about 4.3 MiB held at most. The standalone E8 pass (`with_e8_filter`) runs incrementally over the released bytes and gives exactly the one-shot result. To make that bound hold, a filter declaration whose window starts a whole dictionary or more ahead is now refused as corrupt. The reference decoder addresses filter blocks inside its window, so no encoder can declare one; taken at face value (up to 4 GiB ahead) a single declaration at the front of the queue pinned every later filter and all the output behind them. `raw_set_output_limit` is implemented: with a limit set the decoder fails with `OutputLimitExceeded` at the symbol that would cross it, before producing the bytes, in LZ and PPMd blocks alike and across the members of a solid group. Not done, and documented as such: - PPMd blocks still decode in one go and hold their whole output until drained. `ppmd::RangeDec` borrows the input for its lifetime and exposes no way to save and restore its `(range, code, low, pos)`, so the decode loop cannot return to the caller mid-block. The output limit is what bounds them. - Input is still buffered in full; output starts at `finish`. Tests build their own multi-MiB LZ streams (hand-assembled code tables, Delta declarations) since every fixture is far smaller than the dictionary: buffered output stays under 4.5 MiB while a 13 MiB stream is drained through small buffers; a 13 MiB bomb under a 1 KiB limit fails with at most 1 KiB buffered; overlapping filter windows across release points and the E8 pass match an independent model; one-shot, 1-byte input and 1-byte output decodes of every fixture (solid groups, truncated and corrupted variants included) agree byte for byte. BREAKING CHANGE: `rar3::Decoder::finish` returns output in pieces as it is decoded instead of all at once; for members larger than the 4 MiB dictionary an error can now follow output that was already handed out, and a mid-stream filter reset no longer cancels filters whose windows were released before it. Filter declarations starting 4 MiB or more ahead of the current position are rejected as `Corrupt`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A filter declaration that reuses a declared program costs a few input bytes and schedules a pass over a window of up to 128 KiB (Delta) or 256 KiB (x86). Nothing tied the number of passes to the amount of output: up to 8192 Delta filters could be stacked on the same window, drained, and stacked again, so ~680 KB of input bought about 35 s of `delta_decode` (v0.7.8 audit). The bytes filters process are now budgeted per member against the output decoded so far: once the total would exceed 4x that, the stream is refused as `Corrupt` before the pass runs. Filter cost is thereby linear in the output, like the rest of the decoder. The bound is safe for real archives. RAR's compressor filters each region of a file once, by detected content type; the reference decoder can chain filters only over an identical block, and of the standard programs only three run here (Delta, E8, E8E9; the others are refused outright), the two x86 ones being alternatives. An honest stream thus stays at or below 1x - a filter is only applied once its whole window has been produced, so the first one can never trip the check - and 4x leaves room for stacking that no observed encoder does. Every fixture (rar 6.24 output, solid and filtered) decodes unchanged. BREAKING CHANGE: streams whose in-band filters would process more than four times the member's decoded output are rejected as `Corrupt`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
SECURITY.md promises no panic and a bomb-bounded decode when the caller
supplies a limit, but no fuzz target reached the layer that implements
that, and every decoder target ended its loop on a call that made no
progress, which hides stalls instead of reporting them.
New shared driver (fuzz_targets/common/mod.rs): a decode call that
returns OutputFull with consumed == 0 and written == 0 for a non-empty
input and output is repeated once with identical arguments and panics if
it stalls again; same for a finish call that writes nothing without
reaching StreamEnd. A run can only end in StreamEnd, an Err, or the
driver's own step bound.
New targets:
- limited_dispatch: LimitedDecoder over every factory decoder with a
fuzz-chosen cap (0, 1, around 64 KiB, up to 400 kB) and fuzz-chosen
input chunk / output buffer sizes. Asserts the bytes handed out never
exceed the cap and match bytes_written(). No decoder is excluded: the
header-sized arenas of ppmd / rar3 stayed under the 2 GiB rss limit.
- chunked_vs_oneshot: decodes the same bytes one-shot and in small
pieces and requires the same outcome class (both StreamEnd with
identical bytes, or both Err). Two decoders diverge and are excluded
with the reproducer in a comment, pending triage: squeeze (depends on
the output buffer size) and zip-shrink (depends on the input chunk
size).
- decoder_{snappy,lzw,lzo,lzx,quantum,lzfse,adc,lzma2,huffman,range,bwt}:
dedicated targets for parsers that only decoder_dispatch reached,
using the strict driver.
- hpack_qpack: HpackDecoder over two consecutive blocks and QpackDecoder
over an encoder-stream part plus a field section, with fuzz-chosen
table and header-list limits; a decoded list must fit the limit.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Adds limited_dispatch, chunked_vs_oneshot, hpack_qpack and the eleven new decoder_* targets to the matrix, which now lists exactly the 56 [[bin]] targets of fuzz/Cargo.toml. Also replaces the header comment's stale "9 targets, about 5 min" with what the workflow does: one 30 s job per target. No change to permissions, pins, or the fuzzer flags. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The CLI wrapped the library's Error in an io::Error directly, which needs compcol's std feature for the std::error::Error impl; the binary only requires factory. Carry it in a local newtype instead. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…er size A stream whose last symbol before the end-of-stream symbol is a bare 0x90 run flag (no count byte) is malformed, and `finish` said so with `UnexpectedEnd`. `decode` did not check: when the end symbol was reached and the output fully drained inside a `decode` call it reported `StreamEnd`. Which call reaches the end depends only on the caller's output buffer, so the same bytes decoded cleanly through a 64 KiB buffer and failed through a 1-byte or 255-byte one. Found by the `chunked_vs_oneshot` fuzz target. `decode` now applies the same check as `finish`, so the stream is rejected for every chunking. Both paths also stop losing output to the error. An `Err` carries no progress, so bytes a call had already copied into the caller's buffer were dropped with it -- and how many depended on the buffer size (the reproducer delivered 1100 bytes through a 1-byte buffer and 1020 through a 255-byte one, out of 1101). A call that wrote something now returns that progress and the following call reports the error. This covers the dangling flag and the existing truncated-stream `UnexpectedEnd` in `finish`: every chunking gets the same bytes, then the same error. Removes `squeeze` from the fuzz target's `EXCLUDED` list. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder ends once the declared (or, for `Decoder::raw`, the given) uncompressed length has been produced. `pump` only noticed that through `completed`, which is set when the bytes are *delivered*, so in the meantime it went on decoding every code already sitting in the input buffer. Bytes after the end of the stream were therefore parsed as codes when they arrived in the same `decode` call as the last real code, and not when they arrived later: a stream followed by bytes that are not valid codes failed with `Corrupt` in one shot yet decoded cleanly when fed 1, 2, 3 or 7 bytes at a time. Found by the `chunked_vs_oneshot` fuzz target (`01 00 00 00 00 00 d3 07`: one literal, then garbage). `pump` now stops as soon as the bytes delivered plus the bytes waiting in `emit_buf` reach the declared length, which depends on neither the input chunking nor how fast the caller drains. Nothing after the code that produces the last byte is decoded or validated, as the module docs already described for both the framed and the raw entry point. The `consumed` accounting is unchanged (input is still buffered whole). Removes `zip-shrink` from the fuzz target's `EXCLUDED` list, which is now empty. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`decode` copied every input slice into an internal buffer and reported
all of it consumed. Bytes after the end of a Shrink stream were
swallowed with it -- often in an earlier call than the one that reached
the declared length -- so a caller could not tell where the stream ended
(a ZIP reader looking for whatever follows the entry, say), and the
buffer grew with the input.
The decoder no longer buffers input. Codes are read straight from the
caller's slice through the bit accumulator, and a byte is taken only
while the code being read is still short of bits, so after each code
fewer than 8 bits are held, all from the byte that code ended in. A code
cut off by the end of a slice stays in the accumulator (at most 12 bits)
until more input arrives. Together with stopping at the code that
reaches the declared length, this gives, for every slicing of input and
output, in the framed decoder and in `Decoder::raw` alike:
* `decode` reports `StreamEnd` with the `consumed` of all calls adding
up to exactly the stream: the 4-byte header (framed) plus the payload
through the byte holding the last bit of the last code decoded;
* bytes after that are never taken;
* memory no longer depends on the input size.
`finish` is unchanged: it drains what has been decoded and reports
`UnexpectedEnd` if the declared length was not reached.
One existing test asserted the old contract (a whole 34 KiB payload
reported consumed by a single `decode` into a 4 KiB buffer, then decoded
by `finish`) and is rewritten for the new one.
BREAKING CHANGE: `zip_shrink::Decoder::decode` no longer swallows input.
It leaves bytes after the end of the stream unconsumed, and when the
output buffer fills it returns `OutputFull` having consumed only the
input it decoded, instead of taking the whole slice. Callers must offer
the unconsumed remainder again (the documented decode loop already
does); calling `finish` straight after an `OutputFull` that left input
behind now yields `UnexpectedEnd`.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`RangeDec<'a>` borrows its input for as long as it lives and kept its state `(range, code, low, pos)` to itself, so a caller that owns the input could not return to its own caller in the middle of a stream: the borrow pins the decoder to one stack frame. The RAR3 decoder is such a caller, and it is why its PPMd blocks still decode in one go. Add a crate-private `RangeDecState` — those four fields as plain data — with `RangeDec::save()` to take one between two symbols and `RangeDec::resume(mode, input, state)` to rebuild the decoder over the same input. The error flag is deliberately not part of the state: callers check `err()`/`overran()` after every symbol and stop for good when either is set, so a state is only ever saved from a healthy decoder. Nothing uses it yet outside the tests; the standalone PPMd and ZIP method 98 decoders are untouched. A unit test rebuilds the decoder from its saved state before every symbol of a fixture, in both the 7z and the RAR flavour, and checks symbols and states against an uninterrupted run. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder already streamed its output for LZ blocks — decode until a release pass makes bytes final, hand them out, resume at the next symbol on the following call — but a PPMd block still decoded in one go, because its range decoder borrows the input and could not be suspended. A PPMd member therefore held its whole output until drained, bounded only by the caller's output limit. With `ppmd::RangeDecState` the block can now pause: - `run_ppmd_inner` returns `Segment::Paused` at the same place `expand` does: at the top of the symbol loop, right after a release pass that left bytes ready. Pauses therefore fall between two symbols of the escape layer. A literal, or a whole escape sequence (new-table marker, end of data, an in-band filter declaration with all its payload bytes, a match with its distance and length), is decoded within one call and is never split; so is the solid end-of-member marker, which is read in the same call as the member's last byte. - `run_ppmd` saves the coder in `RunCtx::ppmd_coder` on a pause and rebuilds it from there on the next call; without a saved state it primes a fresh coder from the bytes after the block header, as before. The state is cleared by every PPMd block header and at every member start, so it never outlives its block or its payload. The model and the escape byte already lived in the context. - Release passes happen at the same stream positions as before, so the bytes produced, the output-limit check, the filter-work cap and the truncation policy are unchanged. What changes is that output of a PPMd block longer than the dictionary is handed out as it is decoded, so an error past that point is now reported after the bytes released before it (as for LZ blocks) instead of before any output. That position depends only on the stream, never on the caller's buffer sizes. Held output is now bounded by one dictionary plus one filter window for every block kind; the module docs drop the PPMd exception. Tests. The real PPMd fixtures are smaller than the dictionary and never pause on their own, so unit tests drive the context with the release point pulled forward, suspending the solid notes/prose group every 1, 7 and 4099 bytes (around the match escapes rar codes notes.txt with, and in a block continuing the previous member's model) and comparing with an uninterrupted run, also on damaged and truncated members. For a multi-MiB PPMd stream there is no encoder to build one with, and arbitrary bytes are not a valid payload; the one payload valid at any length is all zeros, which decodes to zeros (what an encoder writes for a run of zeros). Integration tests use it to check that a 6 MiB PPMd block keeps `buffered_output()` under the dictionary bound, that a truncated payload, the output limit and a missing solid end marker fail with the same bytes delivered first for different chunkings, and that an LZ block switching to PPMd mid-member with a Delta window straddling both the switch and the first release point decodes to the independently computed output. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…t does not The limit is passed down to buffering decoders, so the table of decoders that built their output regardless of it no longer applies. What is left is memory that is not output: header-sized model arenas, dictionaries, and compressed input held until finish. List those with the option that bounds each. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The API- and behaviour-changing work deferred from the v0.7.8 audit (#169), now that v0.7.9 carries the compatible fixes. Breaking commits are marked
!with aBREAKING CHANGE:footer.Output limit reaches the decoder
Decoder::set_output_limit(provided method, default: ignore), bridged toRawDecoder::raw_set_output_limit;LimitedDecoder::newand the_cappedhelpers pass their budget down.OutputLimitExceededinstead of building it:snappy,huffman,bwt(from their length headers, before buffering the input),arsenic, standaloneppmd,lz5,rar3.sit13,rar2andrar3(LZ and PPMd blocks) now stream their output; held output no longer scales with the declared size.New decoder configuration (config types that were
()are now structs)lzma::DecoderConfig::max_dict_size(default 64 MiB, up to 4 GiB − 1), also on the ZIP method-14 entry points; the dictionary grows with the output instead of being allocated from the header. A header declaring more than the limit is nowUnsupportedup front — previously it was silently clamped to 64 MiB and failed only if a match reached further.ppmd::DecoderConfig::max_memory/ZipDecoder::with_max_memoryfor the model arena.lz5::DecoderConfig::max_block_size.zstd::DecoderConfig/lz4::frame::DecoderConfig:multi_frame(see below).Behaviour changes
with_multi_frame(false)stops after one frame with exact consumption. Trailing bytes that cannot start a frame end the stream and are left unconsumed (the crate's gzip precedent; the reference CLIs reject them).feed_encoder_streamaccepts the stream in arbitrary pieces, keeps a bounded partial instruction, applies each instruction once;encoder_stream_pending()added; errors are final.decodeemits output and can end the stream; a decoder with no declared unpack size rejects input withUnsupportedinstead of returning empty success (sodecoder_by_name("rar2")cannot decode anything — the factory has no way to pass a size).Corrupt; filter work per member is capped at 4× the output decoded.flush(pending input as meta-blocks plus a byte-aligning empty metadata block), all qualities. No history is carried across meta-blocks, so frequent flushes cost ratio;FullandSyncwrite the same bytes.--max-output SIZE; trailing data after the stream is an error (--ignore-trailingto accept);--forcewrites a temp file and renames it over the target (failed runs leave the old file, symlinks are replaced not followed); archives (RAR, StuffIt) are not auto-dispatched to member codecs.Fuzzing
14 new targets:
limited_dispatch(cap never exceeded, stalls are failures),chunked_vs_oneshot,hpack_qpack, and dedicated targets for eleven decoders previously reached only throughdecoder_dispatch. The differential target found thearc_squeezeandzip_shrinkchunking bugs fixed here; no decoder is excluded from it.Known gaps
finish. Its PPMd pause/resume is tested by forced pauses on the real fixtures and an all-zero multi-MiB stream; filter declarations inside PPMd blocks and PPMd→LZ switches across a pause have no test (no fixture, no encoder).-f -o NULon Windows is unverified; the CLI/factory cannot raisemax_dict_size; auto-detect does not recognise a stream starting with a skippable frame (pzstdoutput) without-t.finish(listed inSECURITY.md).Verification
On the final tree, all passing: fmt; clippy
-D warningsforall,all-features, four narrow feature sets, Windows and macOS targets;cargo testwith--features alland--all-features; everyno_stdbuild incl. thumbv7em and the alloc-without-std set; embed no-alloc tests; MSRV 1.95; docs on stable and nightly--cfg docsrs; i686 check; build of all 56 fuzz targets; embed footprint and cross-check.🤖 Generated with Claude Code