Repository navigation
refactor(wheel): port wheelmaker to Rust and add it to shrubbery - #4504
Conversation
Port `wheelmaker` to Rust under `shrubbery` so `py_wheel` builds avoid spawning a Python interpreter when `shrubbery` is enabled. The Rust `wheelmaker` subcommand supports the `tools/wheelmaker.py` flags used by `py_wheel`, `@flagfile` parsing, PEP 427/440 normalization, workspace status stamping, and deterministic zip and `RECORD` generation. `py_wheel` selects it through `get_shrubbery_tool` and falls back to `tools/wheelmaker.py` when `shrubbery` is disabled or unavailable. In `-c opt` benchmarks against `tools/wheelmaker.py`: - Small wheels: 28x-34x faster (~6 ms vs ~183-206 ms), ~3x lower peak RSS (~7 MiB vs ~21-24 MiB) - Many files (1,000-10,000): 1.5x-4.2x faster - Uncompressed (`--no_compress`): 1.4x-33.6x faster up to 256 MiB - Compressed (DEFLATE): faster up to ~80-100 MiB at default level 6 (and across library levels 1-9 on multi-file source wheels), ~14% slower at 256 MiB (`0.86x`) once C `zlib` amortizes Python startup Work towards bazel-contrib#4250
|
This pull request was prepared with the assistance of an AI coding This is an agent-generated comment reviewed and approved by the user. |
…mments Explain why wheelmaker uses a 256 KiB buffer (and make BLOCK_SIZE private since nothing outside wheelmaker_lib imports it), and copy over the remaining inline comments from tools/wheelmaker.py covering deterministic entry sorting, streaming hash/size calculation, and minimal CSV quoting in RECORD.
Add Rust unit test coverage for repeated --strip_path_prefix flags in priority order (including prefixes without trailing slashes so RECORD and zip entries strip leading slashes), out-of-order --data_files, --extra_distinfo_file, and directory children, single-semicolon inline Requires-Dist markers, and omitted --description_file fallback to UNKNOWN.
Use zlib-rs/flate2, which is on-par or faster than C zlib. No Rust implementation should be significantly slower than a C implementation. (At least, if both are compiled with LLVM.) |
Use `zlib-rs` with `runtime_detection` instead of `miniz_oxide` as `zip`'s pure-Rust DEFLATE backend so compressed large wheels outpace C `zlib` while keeping cross-compiled static `musl` builds free of C dependencies.
Iterate lines directly with `str::lines` instead of pre-allocating `\r\n`-normalized strings before line splitting, express requirements file filtering as an iterator chain, and document `# Errors` conditions and `Path::parent` behavior.
I switched On the 256 MiB wheel at default level 6 ( This is an automatically generated agent comment. |
Solumin
left a comment
There was a problem hiding this comment.
Overall fine. Since performance came up, I feel obliged to mention that there are a bunch of things in this implementation that could be tightened up, but:
- I expect the performance is going to be dominated by I/O and zipping, and
- I'd want profiling etc. to prove it first
So it depends on how concerned you are.
Fold `get_files_to_package` directly into `collect_sorted_inputs` so entries go straight into the deduplicating `HashMap` before sorting, and normalize `\r\n` while appending the description into `METADATA` in `format_metadata` instead of allocating an extra string in `build_wheel`.
`std::fs::create_dir_all("")` already succeeds as a no-op, so `create_parent_dir`
does not need to guard against `!parent.as_os_str().is_empty()` when `--out` or
`--name_file` is a bare relative filename. Add a subprocess test exercising
bare relative output paths in the child working directory.
|
@Solumin re: performance: Yeah, io is probably the dominate factor. The main case is going to be large wheels. Something like pytorch is one of the extreme cases: tens of thousands of files, with a handful of large binaries. Most wheels are capped by PyPI limits (100MB I think? Unless you get exceptions). I wonder if we could eke out some better performance with some concurrency... |
Profile first, then try rayon, then std::threads, and then reconsider what you really need, and then try async rust. |
|
I am thinking that the wheelmaker could be useful to for making the wheels after patching them. Since this is just a binary, we could in theory use them in repo rules. |
Yes |
Port
wheelmakerto Rust undershrubberysopy_wheelbuilds avoidspawning a Python interpreter when
shrubberyis enabled.The Rust
wheelmakersubcommand supports thetools/wheelmaker.pyflags used by
py_wheel,@flagfileparsing, PEP 427/440normalization, workspace status stamping, and deterministic zip and
RECORDgeneration.py_wheelselects it throughget_shrubbery_tooland falls back to
tools/wheelmaker.pywhenshrubberyis disabled orunavailable.
In
-c optbenchmarks againsttools/wheelmaker.py:peak RSS (~5.2 MiB vs ~21-24 MiB)
--no_compress): 1.3x-43.2x faster up to 256 MiBzlib-rs): 1.6x-36.9x faster up to 256 MiBat default level 6 (1.71 s vs 2.79 s at 256 MiB), and faster across
all levels 0-9
Work towards #4250