Skip to content

lz5: decode everything the Lizard CLI writes (sub-blocks, -BD, LIZv1, Huffman) - #171

Merged
MagicalTux merged 3 commits into
masterfrom
test/lz5-lizard-fixtures
Nov 10, 2026
Merged

MagicalTux merged 3 commits into
masterfrom
test/lz5-lizard-fixtures

Conversation

@MagicalTux

@MagicalTux MagicalTux commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #169. I built the reference Lizard CLI (2.1.0, inikep/lizard) and used it as a black box — its CLI and the two format documents in doc/, not its source — first to check the lz5 decoder, then to complete it.

1. Fix: real files over 128 KiB did not decode (predates #169)

  • A compressed frame block is a level byte followed by one sub-block per 128 KiB of input, sharing one window. Only the first sub-block was decoded; the rest was reported as corrupt. With the CLI's default 4 MiB frame block, nothing larger than one sub-block decoded.
  • -BD frames (blocks referring back into earlier blocks) were decoded as independent and failed with InvalidDistance.

2. Feature: every level the CLI writes now decodes

  • LIZv1 codewords (levels 20–29, 40–49), including 24-bit offsets and repeat-offset tokens.
  • Huffman-coded sub-streams (levels 30+). These are plain Huff0 blocks, the same coding zstd uses for literals. The module docs had left this out on the grounds that the reference picks between two decode-table strategies without recording the choice; that choice does not affect the decoded bytes, so the existing single-symbol decoder is enough.
  • To share that decoder without making lz5 depend on zstd, the bit reader, FSE and Huff0 decoders move to a crate-private src/huff0/ (first commit, no code change).
  • Dependent frames keep 16 MiB of history (the reach of a 24-bit offset).

3. The content checksum added in #169 is confirmed

Files the CLI writes with its default checksum decode and verify, and flipping a trailer bit fails with ChecksumMismatch. That was the one unconfirmed item from #169.

Verification

  • Against the CLI: 2,640 files — every level 10–49 × -B1/-B3/-B7/default × with and without -BD × eleven inputs from empty to 8 MB — plus 1,008 more on skewed, run-heavy and 26 MB far-repeat inputs. All decode to the original, checksum verified.
  • Other direction: this crate's (store-only) encoder → lizard -d passes.
  • Fixtures: five CLI-written files (13–42 KiB), each decoded under four chunkings, plus truncation and single-byte-corruption sweeps over them.
  • Fuzz: decoder_lz5 for 2.7M executions seeded with real Lizard streams at eight levels.
  • Gates: fmt; clippy for all, all-features, lz5-only, zstd-only, lz4-only, Windows and macOS targets; full test suite; lz5-only tests; no_std builds; MSRV; docs on stable and nightly; i686 check.

Notes

  • Where lizard_Block_format.md is ambiguous the CLI's output decided: which flag value means "repeat the last offset" (the document states both), and that at levels 20–29 length extensions sit inline in the literals stream rather than in the lengths stream.
  • None of the CLI's output in these runs used a Huffman-coded lengths or offset stream, or a non-empty lengths stream. Those paths are implemented from the document and covered only by hand-built unit tests.
  • Still Unsupported: block checksums and external dictionaries. The encoder is unchanged (store-only).

🤖 Generated with Claude Code

MagicalTux and others added 3 commits November 11, 2026 01:25
Checked against files written by the reference Lizard CLI (2.1.0), the
decoder rejected most real input over 128 KiB:

- A compressed frame block is one level byte followed by one sub-block
  per 128 KiB of input, all sharing one window. Only the first sub-block
  was decoded and the rest of the block was reported as corrupt, so no
  file compressed with the CLI's default 4 MiB frame block decoded
  unless it fit a single sub-block.
- Frames written with -BD have blocks that refer back into the blocks
  before them. Every block was decoded as independent, so such matches
  failed with InvalidDistance. The last 64 KiB of output is now kept as
  history across the blocks of a dependent frame (the LZ4-codeword path
  has 16-bit offsets).

Levels 10..=19 now decode for every block size and with -BD, on inputs
up to 20 MB. LIZv1 levels and Huffman sub-streams stay Unsupported.

The two fixtures carry the content checksum the CLI writes by default,
which also confirms the xxh32 (seed 0) reading of that field that the
decoder has verified since #169.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… module

The reverse bit reader, the FSE table decoder and the Huff0 tree decoder
move from src/zstd/ to src/huff0/, compiled when either zstd or lz5 is
enabled, so that lz5 can decode Lizard's Huffman-coded sub-streams
without turning the zstd codec on. No code change: zstd imports the
three modules under their old names.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The decoder now reads everything the reference Lizard CLI writes, at
every level from 10 to 49:

- LIZv1 codewords (levels 20..=29 and 40..=49): tokens below 32 take a
  24-bit offset and a match of token + 16 bytes; 0_MMMM_LLL takes a
  16-bit offset; 1_MMMM_LLL repeats the last offset, with MMMM = 0
  meaning literals only. Length extensions come from the lengths stream
  or, when a sub-block has none, inline from the literals stream. The
  last offset carries from one sub-block to the next.
- Huffman-coded sub-streams (levels 30 and up): a 3-byte regenerated
  size, a 3-byte compressed size, then a Huff0 block as HUF_compress
  writes it (tree description, jump table, four bitstreams), decoded by
  the entropy decoder shared with zstd. Whether a decoder builds a
  single- or double-symbol table for it does not change the result, so
  the earlier reason for leaving this out did not hold.
- Block-dependent frames keep 16 MiB of history, the reach of a 24-bit
  offset, trimmed only once twice that has accumulated.

Written from doc/lizard_Block_format.md and from the CLI's output used
as a black box; where the document is ambiguous (which flag bit means
'repeat offset', where length extensions live at levels 20..=29) the
CLI's streams decided.

Validated against Lizard CLI 2.1.0: 2640 files (levels 10..=49, block
sizes -B1/-B3/-B7/default, with and without -BD, eleven inputs from
empty to 8 MB) plus skewed, run-heavy and 26 MB far-repeat inputs, all
decoding to the original with the content checksum verified. Three
CLI-written fixtures are added (LIZv1 with a 24-bit offset, LZ4
codewords with Huffman, LIZv1 with Huffman in a dependent frame), and
the decoder_lz5 fuzz target ran 2.7M executions seeded with them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MagicalTux MagicalTux changed the title fix(lz5): decode multi-sub-block frame blocks and block-dependent frames lz5: decode everything the Lizard CLI writes (sub-blocks, -BD, LIZv1, Huffman) Nov 10, 2026
@MagicalTux
MagicalTux merged commit 4fafc80 into master Nov 10, 2026
52 checks passed
@MagicalTux MagicalTux mentioned this pull request Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant