Repository navigation
lz5: decode everything the Lizard CLI writes (sub-blocks, -BD, LIZv1, Huffman) - #171
Merged
Merged
Conversation
Checked against files written by the reference Lizard CLI (2.1.0), the decoder rejected most real input over 128 KiB: - A compressed frame block is one level byte followed by one sub-block per 128 KiB of input, all sharing one window. Only the first sub-block was decoded and the rest of the block was reported as corrupt, so no file compressed with the CLI's default 4 MiB frame block decoded unless it fit a single sub-block. - Frames written with -BD have blocks that refer back into the blocks before them. Every block was decoded as independent, so such matches failed with InvalidDistance. The last 64 KiB of output is now kept as history across the blocks of a dependent frame (the LZ4-codeword path has 16-bit offsets). Levels 10..=19 now decode for every block size and with -BD, on inputs up to 20 MB. LIZv1 levels and Huffman sub-streams stay Unsupported. The two fixtures carry the content checksum the CLI writes by default, which also confirms the xxh32 (seed 0) reading of that field that the decoder has verified since #169. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… module The reverse bit reader, the FSE table decoder and the Huff0 tree decoder move from src/zstd/ to src/huff0/, compiled when either zstd or lz5 is enabled, so that lz5 can decode Lizard's Huffman-coded sub-streams without turning the zstd codec on. No code change: zstd imports the three modules under their old names. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder now reads everything the reference Lizard CLI writes, at every level from 10 to 49: - LIZv1 codewords (levels 20..=29 and 40..=49): tokens below 32 take a 24-bit offset and a match of token + 16 bytes; 0_MMMM_LLL takes a 16-bit offset; 1_MMMM_LLL repeats the last offset, with MMMM = 0 meaning literals only. Length extensions come from the lengths stream or, when a sub-block has none, inline from the literals stream. The last offset carries from one sub-block to the next. - Huffman-coded sub-streams (levels 30 and up): a 3-byte regenerated size, a 3-byte compressed size, then a Huff0 block as HUF_compress writes it (tree description, jump table, four bitstreams), decoded by the entropy decoder shared with zstd. Whether a decoder builds a single- or double-symbol table for it does not change the result, so the earlier reason for leaving this out did not hold. - Block-dependent frames keep 16 MiB of history, the reach of a 24-bit offset, trimmed only once twice that has accumulated. Written from doc/lizard_Block_format.md and from the CLI's output used as a black box; where the document is ambiguous (which flag bit means 'repeat offset', where length extensions live at levels 20..=29) the CLI's streams decided. Validated against Lizard CLI 2.1.0: 2640 files (levels 10..=49, block sizes -B1/-B3/-B7/default, with and without -BD, eleven inputs from empty to 8 MB) plus skewed, run-heavy and 26 MB far-repeat inputs, all decoding to the original with the content checksum verified. Three CLI-written fixtures are added (LIZv1 with a 24-bit offset, LZ4 codewords with Huffman, LIZv1 with Huffman in a dependent frame), and the decoder_lz5 fuzz target ran 2.7M executions seeded with them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #169. I built the reference Lizard CLI (2.1.0, inikep/lizard) and used it as a black box — its CLI and the two format documents in
doc/, not its source — first to check the lz5 decoder, then to complete it.1. Fix: real files over 128 KiB did not decode (predates #169)
-BDframes (blocks referring back into earlier blocks) were decoded as independent and failed withInvalidDistance.2. Feature: every level the CLI writes now decodes
lz5depend onzstd, the bit reader, FSE and Huff0 decoders move to a crate-privatesrc/huff0/(first commit, no code change).3. The content checksum added in #169 is confirmed
Files the CLI writes with its default checksum decode and verify, and flipping a trailer bit fails with
ChecksumMismatch. That was the one unconfirmed item from #169.Verification
-B1/-B3/-B7/default × with and without-BD× eleven inputs from empty to 8 MB — plus 1,008 more on skewed, run-heavy and 26 MB far-repeat inputs. All decode to the original, checksum verified.lizard -dpasses.decoder_lz5for 2.7M executions seeded with real Lizard streams at eight levels.all,all-features, lz5-only, zstd-only, lz4-only, Windows and macOS targets; full test suite; lz5-only tests;no_stdbuilds; MSRV; docs on stable and nightly; i686 check.Notes
lizard_Block_format.mdis ambiguous the CLI's output decided: which flag value means "repeat the last offset" (the document states both), and that at levels 20–29 length extensions sit inline in the literals stream rather than in the lengths stream.Unsupported: block checksums and external dictionaries. The encoder is unchanged (store-only).🤖 Generated with Claude Code