Repository navigation
feat: ContiguousDecoder (lzma, xz, lzma2, zstd) and mmap output in the CLI - #174
Merged
Merged
Conversation
…history Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An LZ decoder keeps the last dictionary-size bytes of its output to copy matches from: up to 1 GiB and more for xz/lzma or `zstd --long`. When the output is a file, that history is already there. With the new `mmap` feature, `compcol -d` maps the output file and drives a `ContiguousDecoder` straight into the mapping, so the page cache backs the dictionary and the decoder holds no window. When it is used. On `-d`, when `factory::contiguous_decoder_by_name` has a decoder for the format and the output is a regular file the CLI has just created: the `create_new` destination, or the temporary file of a `--force` run. Never stdout, a device, a FIFO or a pre-existing file; and on unix only if nobody but the owner can write the new file (always so with a regular input, where it is created 0600; from stdin it depends on the umask). `--no-mmap` forces the streaming path, and so does, silently, any failure to set the mapping up, which happens before anything is decoded. The flag is accepted without effect in builds without the feature. How. The output size is unknown, so the file starts at clamp(4 x input, 4 MiB, 1 GiB), at most doubles (by at most 1 GiB) when the decoder reports `OutputFull`, and is mapped again; the decoder carries on with the same `filled`, as the trait allows. At the end the decoded range is flushed, the mapping dropped and the file cut to the decoded length (in that order: Windows cannot shorten a mapped file), and the usual commit follows: permissions, rename for `--force`, input removal. A zero length is never mapped. Disk full. A write through a mapping that cannot get a block is a SIGBUS, not an error. On unix the blocks are therefore reserved ahead of the decoder by writing zeros through the file handle (256 KiB, doubling up to 16 MiB at a time), and the decoder is only handed the reserved prefix of the mapping, so ENOSPC is an ordinary error with the ordinary cleanup. Measured on a memcpy-speed mock decoder this costs about 0.1 s per GiB of output (4 GiB: 1.83 s -> 2.37 s on btrfs, 1.19 s -> 1.55 s on tmpfs), against 2.3 s/GiB for this crate's zstd decoder and 13 s/GiB for xz. It does not help on copy-on-write or network file systems; `--help` and the README say so. Windows allocates on `set_len`, so nothing is written there; a growth step that fails is retried at half the size. `--max-output N`. The mapping is never grown past N + 1 bytes and the run fails with the same message as the streaming path once the decoder has written, or needs, more than N. The one spare byte is what tells a decoder that stopped exactly at N (and only has a trailer left to read) from one that has more to write, as `LimitedDecoder` does with its probe byte. `set_output_limit(N)` is passed to the decoder as well. Everything else is shared with the streaming path rather than repeated: `mapped::decode` returns the same "input was left over" answer as `stream_decode`, and trailing-data handling, `--ignore-trailing`, the `--max-output` message, cleanup on error and the commit run on that result unchanged. A decoder that keeps asking for room without making progress is an error once it has 256 MiB to write into, not an ever-growing file. `unsafe`. One function, `mapped::map_file`, holds the `MmapMut::map_mut` call and the only `#[allow(unsafe_code)]`; its SAFETY comment states the hazard (a foreign writer or truncation of the file) and why mapping only our own fresh files narrows it. The binary is `#![deny(unsafe_code)]`. For that single `allow` to be possible, the package-wide `[lints.rust] unsafe_code` goes from "forbid" to "deny"; the library keeps its own `#![forbid(unsafe_code)]` in `src/lib.rs`. Cargo. `mmap = ["dep:memmap2"]`, optional `memmap2 = "0.9"` (MSRV 1.65), used by the binary only. Not in `default`, and not in `all`: CI derives its no_std build lists from `all`. Release binaries need `--features all,mmap`. `COMPCOL_DEBUG_IO=1` prints `io: mmap` or `io: stream` on stderr, for the tests. Tests. Unit tests drive the mapped path with a mock `ContiguousDecoder` that copies matches out of `output[..filled]` and checks that prefix on every call: growth over many re-maps (with and without reservation), exact final length, empty output, output produced only by `finish`, the limit (exact, one short, zero), trailing data, a stalled decoder, and cleanup / commit through `FileOutput` with and without `--force`. `tests/cli.rs` runs xz, lzma and zstd through the real binary in every output mode with and without `--no-mmap` and requires the same exit status and the same files (clean, concatenated, trailing, truncated, corrupted, limited). `contiguous_decoder_by_name` has no arms yet, so until it does both runs take the streaming path and these tests assert equality, not the path. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…indow Implement `compcol::ContiguousDecoder` for `zstd::Decoder`. Given the whole output as one buffer, the decoder keeps no history and stages no block output: Raw and RLE blocks are written straight to `output[filled..]`, and a Compressed_Block's sequences are executed there, literals copied in and matches copied from `output[..pos]`. How a block that does not fit is handled: the block's literals and sequence list are decoded into per-block scratch (bounded by the 128 KiB block size, never by the window) and validated by a new `measure_sequences`, which makes the checks `execute_sequences` makes while copying — literal bounds, Block_Maximum_Size, every offset — and returns the regenerated size. Only then, and only if `output` has that much room, is the block executed (`execute_sequences_into`); otherwise the call returns `OutputFull` with nothing of the block written and the block stays pending. Nothing is decoded twice and nothing is copied twice; no index into the caller's slice survives a call, so the buffer may be a different, longer one next time. Both modes share the frame state machine (`Decoder::run`); only the sink differs. So multi-frame and skippable frames, `multi_frame(false)`, trailing bytes, the content checksum (over the bytes written to `output`), Frame_Content_Size, Block_Maximum_Size, exact bitstream consumption and the 4-stream literal minimum behave identically and fail with the same error variants. The offset rule is the windowed one, restated without a history: a match may reach back at most min(bytes this frame has produced, Window_Size) — not into an earlier frame, though its bytes sit right there in the buffer. `DecoderConfig::max_window_size` is not applied in contiguous mode: it is a memory budget and no window is allocated, so a `zstd --long=31` stream decodes with the same small footprint. Documented on the config, the trait impl and the module. Mixing `decode`/`finish` with the contiguous calls on one stream, or a `filled` that is not what the decoder has written since the last reset, is `Error::Unsupported` and leaves the stream state untouched; `reset` allows either mode again. The streaming path is byte-for-byte unchanged and as fast as before (100 MB decode: 0.225 s before, 0.224-0.236 s after, within noise). Tests: contiguous output equals windowed output for encoder output at levels from MIN_LEVEL to 22, with Sync/Full flushes, multi-frame streams with skippable and empty frames, RLE/Raw-heavy data and the reference fixtures — with the buffer grown in small steps into a fresh allocation each time, 1-byte input feeding and exact-size buffers; every malformed or truncated stream of the existing tests gives the same error in both modes (`decode_all_chunkings` now checks it too); a frame declaring a 3.75 TiB window decodes without a history allocation; mode mixing is refused. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… no dictionary The raw LZMA2 decoder now implements `ContiguousDecoder`: when the whole decoded stream goes into one buffer, that buffer is the history. Matches are copied within it, uncompressed chunks are copied into it, and no window is allocated whatever dictionary size is configured. The LZMA kernel is still written once. `LzmaCore` is split into the model (probabilities, LZ state, range coder) and the window, and the decoding loop is generic over a small private `History` trait with two implementations it is monomorphised for: * `WindowSink` — the existing lazily grown ring plus the chunk's output buffer; the streaming path, with its bulk match copies, unchanged; * `ContiguousSink` — the caller's buffer and an absolute position. Both answer "is this distance valid" the same way: below the number of bytes decoded since the last dictionary reset and below the dictionary size. In contiguous mode the reset point is an absolute position in the output (the floor), so a match reaching before a dictionary reset, or further back than the dictionary size, is `Corrupt` in both modes even though the bytes are sitting in the buffer. A match or an uncompressed chunk cut by the end of the buffer resumes on the next call. What is kept across calls is a remaining length and positions relative to the start of the output, never an index into a particular slice, so the buffer may be a longer, relocated one each time. `decode`/`finish` and the contiguous calls cannot be mixed on one stream, and `filled` must be what the decoder has written: both are refused with `Error::Unsupported` without disturbing the decoder (shared `OutputMode`, which the xz decoder will use too). `reset()` allows either mode again. An error from a contiguous call is final. Tests: `decode_all` in tests/lzma2.rs now also drives the contiguous calls (buffer grown from nothing into fresh allocations, one input byte at a time, exact-size buffer) and requires the same output or the same error, so every existing accept/reject test covers both modes. New ones mirror the back-reference-across-a-reset test with the output cut at every length, bound distances by the dictionary size, compare every truncation, check that mixing errors, and assert via the allocation counters that a stream decoded with a 128 MiB dictionary configured allocates zero dictionary bytes. Decoding 50 MB through the streaming path: 355 ms before, 360 ms after (run-to-run noise is larger); contiguous: 346 ms. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…onary The xz decoder now implements `ContiguousDecoder`. A Block whose filter chain is LZMA2 alone (what `xz` writes unless told otherwise) is decoded straight into the caller's buffer, which is its dictionary: no window is allocated, whatever size the Block Header declares. The container state machine is shared, not copied: `raw_decode` and the contiguous calls both run `Decoder::run`, which writes from `output[start..]`. Only the compressed-chunk step and the uncompressed chunk's dictionary feed differ, and only for a Block decoded directly. Headers, declared Block sizes, the Block Check (computed over the bytes written to the buffer), Index verification, concatenated Streams, Stream Padding and exact consumption are the same code in both modes. History floor: a new Block, a new Stream and an LZMA2 dictionary reset each start a fresh history at the current output position. A match reaching before it, or further back than the Block's dictionary size, is `Corrupt` exactly as in the streaming mode. Filter chains: with Delta/BCJ filters before LZMA2, what LZMA2 produces is not what the caller gets, so it cannot be history in the caller's buffer. Such a Block falls back to the windowed core and the filter pipeline, whose output is written to the buffer. This is the one case where a window is still allocated, and it is documented in the module docs. A file may mix both kinds of Block; the window is released when a plain Block follows (`LzmaCore::release_window`). Pending state survives `OutputFull`: a match cut by the end of the buffer, a partly decoded chunk or a partly copied uncompressed chunk resumes on the next call with a longer, possibly relocated buffer. `LzmaCore::can_continue` tells "out of room" from "out of input" for a chunk still arriving. Mixing `decode`/`finish` with the contiguous calls, or a `filled` that is not what the decoder wrote, is `Error::Unsupported`, as for lzma2. One documented difference. An input with two defects may be failed for either. A Block that both outgrows the Uncompressed Size its header declares and has damaged LZMA data further on in the same chunk gives `UnexpectedEnd` when streamed in one piece (the chunk is decoded ahead of being handed out) and `Corrupt` in contiguous mode with a buffer that stops at the declared size. The streaming mode already answers either way depending on how the input is cut. Found by differential fuzzing; pinned by `contiguous_may_meet_another_defect_first`. Tests: `decode_chunked` and `decode_tracking` in tests/xz.rs now also drive the contiguous calls and require the same output or the same error, so the existing corruption, truncation, size, Index, padding, check-type and filter-chain tests cover both modes. New ones: all ten encoder levels with no / Sync / Full flushes, buffer grown a byte at a time into fresh allocations, 1-byte input, exact-size buffers; hand-built multi-Block and multi-Stream files mixing plain and Delta Blocks; a match reaching across a dictionary reset, a Block boundary and a Stream boundary; every truncation point and every byte with a bit flipped; dictionary-size bound; mode mixing; and, in-module with the allocation counters, zero dictionary bytes for a Block declaring 128 MiB, and a window only for the filtered Block of a mixed file. Decoding 50 MB through the streaming path: 425 ms before, 436 ms after (within run-to-run noise, the two alternate); contiguous: 419 ms. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…dictionary `lzma::Decoder` now implements `ContiguousDecoder`: when the whole result goes into one buffer, `decode_contiguous` / `finish_contiguous` read the literal coder's previous byte and match byte, and every match, straight from `output[..filled]` and write to `output[filled..]`. No dictionary is allocated whatever the header declares (`dict_allocated()` stays 0). Both framings are covered: LZMA-alone and the ZIP method-14 modes. One copy of the LZMA logic. The decode loop, header parsing, bounded input absorption and the finish logic are generic over a small `Sink` (where bytes go and where history is read from), monomorphised for: * `WindowSink` — the existing circular dictionary plus the caller's output slice (the dictionary moved out of `LzmaCore` into `Window`); * `ContigSink` — the caller's buffer and a position, rebuilt from the arguments on every call, so nothing refers to a previous slice and the buffer may be reallocated between calls. A match cut by the end of the buffer is kept as (distance, remaining) and resumes in the next one. Which distances are valid is decided in one place, from the stream position and the header's dictionary size (`LzmaCore::dict_has`), not by the sink: the two paths accept and reject the same streams with the same errors, and a match may not reach past the declared dictionary even though the contiguous buffer still holds the bytes. Input handling — exact consumption, StreamEnd, end marker / declared size rules — is the shared code. Choices, documented on the impl: * `max_dict_size` is not applied on the contiguous path. It is the dictionary's memory budget and there is no dictionary; a header declaring 4 GiB - 1 decodes at no extra cost. * A stream is on one path from its first call until `reset`. The other path's calls (`decode`, `finish`, `discard_output` on a contiguous stream, or the reverse), and a `filled` that exceeds `output.len()` or is not what the decoder has written since the reset, are `Error::Unsupported` and change nothing. * `OutputFull` is reported only when a decoded byte has nowhere to go, not as soon as the buffer is full, so a buffer of exactly the decoded size reaches `StreamEnd` (end marker included) without being grown. The streaming path's output is unchanged. To keep its speed where it was after the restructuring, the literal bits are decoded on a local copy of the range decoder and the match half of a packet sits in its own function, which keeps the literal path of the main loop small. Decode speed before/after, best of several alternating runs on a busy machine: `examples/micro lzma source 50000000` 152 -> 150 MB/s, `lorem` 2573 -> 2613 MB/s; xz-made streams (text -0 / -9e, incompressible, 4 KiB dictionary) 87 -> 87, 100 -> 100, 25.8 -> 26.2, 42 -> 44.5 MB/s. Tests (tests/lzma.rs, tests/lzma_zip.rs): contiguous output equals windowed output for the crate encoder at several levels, sizes 0 to 3 MB, compressible and not, with and without end marker / declared size, all ZIP modes and the 7-Zip / Python fixtures — with exact-size buffers, 1-byte input, and a buffer that starts tiny and is replaced by a new, longer allocation on every OutputFull; every truncation, header error, size/marker disagreement and single-byte corruption gives the same error on both paths; a too-small declared dictionary is Corrupt on both; a 1 GiB / 4 GiB - 1 header decodes with no dictionary under any limit; mixing the paths and wrong `filled` values are refused. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
With the arms in place the CLI's mapped output path is live for those formats; the CLI test now requires it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Decodes the same bytes through the windowed path and into one buffer that doubles as the history, and requires the same bytes and the same outcome class. The contiguous buffer is moved to another address and the old one scribbled over every time the decoder asks for room. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…igurable window limit
A stream declaring an LZMA2 dictionary above 128 MiB was decoded with the
size clamped to 128 MiB, and the clamped value was also the bound on
match distances. A valid match reaching further back was therefore
reported as `Corrupt`, in the streaming mode and in the new contiguous
mode alike. Repro: 150 MB of noise, 150 MB of zeros, the same again,
`xz -T1 -0 --lzma2=preset=0,dict=1GiB` (matches 300 MB back).
The declared dictionary size is now always the distance bound, up to the
format's 4 GiB − 1. What used to be a silent clamp is a memory budget,
applied only where a window is allocated:
* Streaming mode: new `max_dict_size` on `xz::DecoderConfig` (new type;
`Xz`'s `DecoderConfig` was `()`) and on `lzma2::DecoderConfig`, default
128 MiB (`DEFAULT_MAX_DICT_SIZE`), up to `MAX_DICT_SIZE` (4 GiB − 1, or
`isize::MAX` on 32-bit targets); out-of-range values are brought into
4 KiB..=MAX. A dictionary over the limit is `Error::Unsupported`, as
for `.lzma`: in xz when the Block Header's filter flags are read, in
raw lzma2 (whose dictionary size is configuration) from the first
`decode`/`finish` call. With the limit raised the stream decodes; the
window still grows with the output, so only streams that really
produce that much pay for it.
* Contiguous mode does not apply the limit (same decision as `.lzma` and
zstd's `max_window_size`): no window exists. The exception is an xz
Block with Delta/BCJ filters, which does keep a window there and is
checked against the limit.
API: `xz::Decoder::with_config`, `xz::Decoder::max_dict_size()`,
`xz::DecoderConfig::{new, with_max_dict_size}`;
`lzma2::DecoderConfig::{new, with_max_dict_size}` (chainable after
`with_dict_prop` / `with_dict_size`, which are kept). Both configs are
`#[non_exhaustive]`. `reset` keeps the limit. lzma2's explicit
`dict_size` is no longer clamped to 128 MiB.
Tests (no large allocations): streams patched to declare 192 MiB, 1 GiB
and 4 GiB − 1 are `Unsupported` by default and with a limit one byte
short; decode with the limit raised, the allocation counters showing a
window of the output's size; decode in contiguous mode with zero
dictionary bytes; and in each case the core's distance bound equals the
declared size. 128 MiB exactly is still accepted by default. A filtered
Block over the limit is refused in contiguous mode too.
Verified on the 600 MB repro against `xz -dc` (sha256 equal): xz
streaming with the limit raised 1.5 s, contiguous 1.3 s; raw lzma2 0.58 s
and 0.36 s; default streaming config: `Unsupported` at once.
BREAKING CHANGE: xz and raw LZMA2 streams declaring a dictionary above
128 MiB are now `Error::Unsupported` by default in the streaming mode,
instead of being decoded with a 128 MiB window (and failing as corrupt on
the first far match); raise `DecoderConfig::max_dict_size` to decode
them. `<Xz as Algorithm>::DecoderConfig` is now `xz::DecoderConfig`
instead of `()`, and `lzma2::DecoderConfig` is `#[non_exhaustive]` with a
new `max_dict_size` field, so it can no longer be built with a struct
literal.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decoding into one contiguous buffer that doubles as the match history, so large-dictionary streams need no window in memory — and a CLI path that uses it with a memory-mapped output file.
Library
ContiguousDecoder(new trait, safe, no dependency): each call gets the whole output buffer plus how much is filled; the decoder reads history from the buffer and appends to it. The buffer may be a different, longer slice on every call, so a caller can grow aVecor re-map a file.lzma(incl. ZIP method 14),lzma2,xzandzstd;factory::contiguous_decoder_by_namereturns them. No dictionary / history is allocated in this mode. Each shares its state machine with the normal path, which is byte-for-byte and speed-wise unchanged (measured per codec, within noise).max_dict_size,max_window_size) are not applied in contiguous mode, since nothing is allocated — except for those filtered xz Blocks.Fix found on the way (breaking)
xz and raw LZMA2 clamped the declared dictionary to 128 MiB and used the clamp as the match-distance bound, so a valid file with a larger dictionary and a match reaching past 128 MiB was reported as corrupt (reference
xzdecodes it). Now:xz::DecoderConfig(was()) andlzma2::DecoderConfiggainmax_dict_size(default 128 MiB, up to 4 GiB − 1); a stream declaring more isUnsupportedup front unless the limit is raised.CLI (
mmapfeature, off by default, not inall)compcol -dto a regular file decodes into a memory-mapped output when the format has a contiguous decoder;--no-mmapforces streaming, and any failure to set the mapping up falls back to streaming silently.--force,create_newotherwise) — never an existing file, device, FIFO or stdout.--max-output, trailing-data detection,--ignore-trailing, cleanup and the--forcerename behave as in the streaming path (shared code).unsafecall (MmapMut::map_mut) and an optional dependency,memmap2, both confined to the binary behind the feature. The library stays#![forbid(unsafe_code)]with no required dependencies; the package-level lint goes fromforbidtodenyso the binary can carry its singleallow.--features all,mmap(own commit; touchesrelease-binaries.yml).Verification
zstd --long=30: the mmap path decodes both to the right hash; streaming refuses both as unsupported (window over the default limit).xz5.8.3 presets/filters/block sizes/dictionaries to 1 GiB;zstdincl.--long=31), every truncation point and bit flips, with the buffer relocated on every growth step.contiguous_vs_windowed(differential, buffer moved and old one scribbled on each growth): clean for 82k + 15k executions.--features all,mmapand clippy for Windows and macOS targets with it.Known gaps
--no-mmap).mmap, or to stdout, the CLI has no flag to raise the dictionary limits, so such files cannot be decoded that way.OutputFullwith up to 128 KiB of room left (a block is written whole or not at all).🤖 Generated with Claude Code