Repository navigation
apt uploads: read a .deb's control member within a few MiB - #831
Merged
Merged
Conversation
Reading the control member of an uploaded or imported .deb bounded the time it took but not the memory: `tar` read a GNU long name, long link or PAX header whole before the entry it describes, up to the 256 MiB the member could inflate to, and the xz decoder's window grew alongside. A 39 KB package held over 500 MiB before it was refused, so a few uploads at once could get the instance OOM-killed. - Walk the control tar's entries raw and read a long name, long link or PAX header only when it is at most 64 KiB (the npm reader's cap); apply them as `tar` did, refuse a PAX size other than the header's and GNU sparse entries. - Let the control member inflate to 8 MiB (was 256 MiB) up to the end of its control file, which also bounds the xz window; name the bound when it is reached, and refuse a control file it cuts short. - Give the xz decoder a memory limit (a 64 MiB dictionary, `xz -9`'s) and the zstd decoder an 8 MiB window (level 19's): ruzstd hands nothing out until it has decoded a window more, so it holds the whole window. dpkg-deb -Zzstd -z20 and higher are therefore refused for control members over about 128 KiB, documented as a limitation. The crafted packages of the issue are refused holding 147 KiB (xz) and 78 KiB (gzip), measured by a counting allocator in a test of its own; the reader before held 393 MiB on the xz one. Fixes #815 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJXwVipsHzkYYgSs779
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #815
Problem
Reading the control member of an uploaded or imported
.deb(crates/formats/apt/src/deb.rs) bounded the time it took but not the memory.tarreads a GNU long name, long link or PAX header whole before it returns the entry the header describes, up to the 256 MiB the member could inflate to. The xz decoder's window grew alongside. A 39 KB package held over 500 MiB before it was refused, so a few uploads at once could get the pod OOM-killed.Changes
control_in_tarreads the entries withraw(true), like the npm tarball reader. A long name, long link or PAX header is read only when its header says 64 KiB or less (MAX_EXTENSION_BYTES, the npm reader's cap); a larger one is refused from the header alone. They are applied astarapplied them: at most one of each kind per entry, and the path comes from the long name, then the PAXpath, then the header. A PAXsizeother than the header's and GNU sparse entries are refused, because a raw walk would lay those archives out differently. Control members never contain either.MAX_CONTROL_TAR_BYTES), counted up to the end of thecontrolfile. dpkg-deb sorts the member, so onlyconffilesandconfigcome beforecontrol(checked with dpkg-deb 1.22.6). A builder that putsmd5sumsfirst stays under the bound up to about 70,000 files. When the bound is reached, the error names it, and acontrolfile the bound cuts short is refused instead of being read in part.XzReader::new_mem_limitwithxz -9's 64 MiB dictionary. dpkg-deb compresses the control member like the data, so-Zxz -z9declares 64 MiB on it. lzma-rust2 grows its window only as data is decoded, so what it holds follows the 8 MiB bound.Limitation this introduces
dpkg-deb streams control members larger than about 128 KiB. At
-Zzstd -z20to-z22those declare windows of 32 to 128 MiB, so such packages are now refused withthe control member's zstd window is larger than 8 MiB. Levels up to 19 (Ubuntu's zstd packages included) and xz at any level are read. This is documented indocs/apt.md(upload section and Limitations) and indocs/changes/2.2.0/815.limitation.md. apt is new in 2.2.0, so the bug itself never shipped and gets nofixedfragment.Tests
Unit tests in
deb.rs:crates/formats/apt/tests/control_memory.rs: the issue's acceptance test. A counting global allocator, in a test binary of its own, measures whatcontrol_fileholds on the issue's crafted packages (a 256 MiB GNU long name), built cheaply from concatenated xz streams, a hand-built deflate stream and zstd RLE blocks:The test fails against the old reader.
Real packages built with dpkg-deb 1.22.6 and an 830 KB
md5sums: gzip-z9, xz-z6/-z9, zstd-z3/-z19are read; zstd-z22is refused.cargo test -p forge-format-apt,cargo clippy -p forge-format-apt --all-targets -D warnings,cargo fmt --check,cargo xtask changelog --checkandcargo xtask docs --checkpass.No admin API changes, so no web UI screen applies.
Generated by Claude Code