Skip to content

refactor(wheel): port wheelmaker to Rust and add it to shrubbery - #4504

Merged
rickeylev merged 8 commits into
bazel-contrib:mainfrom
rickeylev:rust_wheelmaker_conversion_pr
Nov 15, 2026
Merged

rickeylev merged 8 commits into
bazel-contrib:mainfrom
rickeylev:rust_wheelmaker_conversion_pr

Conversation

@rickeylev

@rickeylev rickeylev commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Port wheelmaker to Rust under shrubbery so py_wheel builds avoid
spawning a Python interpreter when shrubbery is enabled.

The Rust wheelmaker subcommand supports the tools/wheelmaker.py
flags used by py_wheel, @flagfile parsing, PEP 427/440
normalization, workspace status stamping, and deterministic zip and
RECORD generation. py_wheel selects it through get_shrubbery_tool
and falls back to tools/wheelmaker.py when shrubbery is disabled or
unavailable.

In -c opt benchmarks against tools/wheelmaker.py:

  • Small wheels: 37x-43x faster (~4.4-5.7 ms vs ~167-203 ms), ~4x lower
    peak RSS (~5.2 MiB vs ~21-24 MiB)
  • Many files (1,000-10,000): 1.6x-3.8x faster
  • Uncompressed (--no_compress): 1.3x-43.2x faster up to 256 MiB
  • Compressed (DEFLATE via zlib-rs): 1.6x-36.9x faster up to 256 MiB
    at default level 6 (1.71 s vs 2.79 s at 256 MiB), and faster across
    all levels 0-9

Work towards #4250

Port `wheelmaker` to Rust under `shrubbery` so `py_wheel` builds avoid
spawning a Python interpreter when `shrubbery` is enabled.

The Rust `wheelmaker` subcommand supports the `tools/wheelmaker.py`
flags used by `py_wheel`, `@flagfile` parsing, PEP 427/440
normalization, workspace status stamping, and deterministic zip and
`RECORD` generation. `py_wheel` selects it through `get_shrubbery_tool`
and falls back to `tools/wheelmaker.py` when `shrubbery` is disabled or
unavailable.

In `-c opt` benchmarks against `tools/wheelmaker.py`:
- Small wheels: 28x-34x faster (~6 ms vs ~183-206 ms), ~3x lower peak
  RSS (~7 MiB vs ~21-24 MiB)
- Many files (1,000-10,000): 1.5x-4.2x faster
- Uncompressed (`--no_compress`): 1.4x-33.6x faster up to 256 MiB
- Compressed (DEFLATE): faster up to ~80-100 MiB at default level 6
  (and across library levels 1-9 on multi-file source wheels), ~14%
  slower at 256 MiB (`0.86x`) once C `zlib` amortizes Python startup

Work towards bazel-contrib#4250
@rickeylev
rickeylev requested a review from aignas as a code owner November 14, 2026 18:58
@rickeylev

Copy link
Copy Markdown
Collaborator Author

This pull request was prepared with the assistance of an AI coding
agent. The human author (@rickeylev) reviews and approves any
agent-generated responses or changes before they are posted, or responds
directly; no agent automatically replies on this PR without human
approval.

This is an agent-generated comment reviewed and approved by the user.

…mments

Explain why wheelmaker uses a 256 KiB buffer (and make BLOCK_SIZE
private since nothing outside wheelmaker_lib imports it), and copy over
the remaining inline comments from tools/wheelmaker.py covering
deterministic entry sorting, streaming hash/size calculation, and
minimal CSV quoting in RECORD.
@rickeylev

Copy link
Copy Markdown
Collaborator Author

@Solumin and @arrdem borrow your rust eyes again?

Something I'm not entirely happy with is that this is 14% slower on larger inputs. AI says its because rust's implementation is slower than C zlib, though, so I'm not sure much can be done.

Add Rust unit test coverage for repeated --strip_path_prefix flags in
priority order (including prefixes without trailing slashes so RECORD
and zip entries strip leading slashes), out-of-order --data_files,
--extra_distinfo_file, and directory children, single-semicolon inline
Requires-Dist markers, and omitted --description_file fallback to
UNKNOWN.
@Solumin

Solumin commented Nov 14, 2026 •

Copy link
Copy Markdown

@Solumin and @arrdem borrow your rust eyes again?

Something I'm not entirely happy with is that this is 14% slower on larger inputs. AI says its because rust's implementation is slower than C zlib, though, so I'm not sure much can be done.

Use zlib-rs/flate2, which is on-par or faster than C zlib.

No Rust implementation should be significantly slower than a C implementation. (At least, if both are compiled with LLVM.)

Comment thread crates/wheelmaker/src/lib.rs Resolved
Use `zlib-rs` with `runtime_detection` instead of `miniz_oxide` as
`zip`'s pure-Rust DEFLATE backend so compressed large wheels outpace C
`zlib` while keeping cross-compiled static `musl` builds free of C
dependencies.
Comment thread crates/wheelmaker/src/lib.rs Resolved
Comment thread crates/wheelmaker/src/lib.rs Resolved
Comment thread crates/wheelmaker/src/lib.rs Resolved
Comment thread crates/wheelmaker/src/lib.rs Resolved
Iterate lines directly with `str::lines` instead of pre-allocating
`\r\n`-normalized strings before line splitting, express requirements
file filtering as an iterator chain, and document `# Errors` conditions
and `Path::parent` behavior.
@rickeylev

Copy link
Copy Markdown
Collaborator Author

Use zlib-rs/flate2, which is on-par or faster than C zlib.

I switched zip/flate2 from miniz_oxide to zlib-rs (with runtime_detection) in d1a8c60.

On the 256 MiB wheel at default level 6 (-c opt), runtime dropped from 3,262 ms (0.86x) to 1,710 ms (149.7 MiB/s), vs. 2,793 ms (91.7 MiB/s) in Python (1.63x faster). The Rust binary is now faster than Python at every tested size and compression level (0..9), and all static musl targets still cross-compile cleanly.

This is an automatically generated agent comment.

@Solumin Solumin left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall fine. Since performance came up, I feel obliged to mention that there are a bunch of things in this implementation that could be tightened up, but:

  1. I expect the performance is going to be dominated by I/O and zipping, and
  2. I'd want profiling etc. to prove it first

So it depends on how concerned you are.

Comment thread crates/wheelmaker/src/lib.rs Resolved
Comment thread crates/wheelmaker/src/lib.rs Resolved
Fold `get_files_to_package` directly into `collect_sorted_inputs` so
entries go straight into the deduplicating `HashMap` before sorting, and
normalize `\r\n` while appending the description into `METADATA` in
`format_metadata` instead of allocating an extra string in `build_wheel`.
`std::fs::create_dir_all("")` already succeeds as a no-op, so `create_parent_dir`
does not need to guard against `!parent.as_os_str().is_empty()` when `--out` or
`--name_file` is a bare relative filename. Add a subprocess test exercising
bare relative output paths in the child working directory.
@rickeylev
rickeylev enabled auto-merge November 14, 2026 23:40
@rickeylev

Copy link
Copy Markdown
Collaborator Author

@Solumin re: performance: Yeah, io is probably the dominate factor. The main case is going to be large wheels. Something like pytorch is one of the extreme cases: tens of thousands of files, with a handful of large binaries. Most wheels are capped by PyPI limits (100MB I think? Unless you get exceptions). I wonder if we could eke out some better performance with some concurrency...

@Solumin

Solumin commented Nov 14, 2026 •

Copy link
Copy Markdown

@Solumin re: performance: Yeah, io is probably the dominate factor. The main case is going to be large wheels. Something like pytorch is one of the extreme cases: tens of thousands of files, with a handful of large binaries. Most wheels are capped by PyPI limits (100MB I think? Unless you get exceptions). I wonder if we could eke out some better performance with some concurrency...

Profile first, then try rayon, then std::threads, and then reconsider what you really need, and then try async rust.

@aignas

aignas commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

I am thinking that the wheelmaker could be useful to for making the wheels after patching them. Since this is just a binary, we could in theory use them in repo rules.

@aignas aignas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rickeylev
rickeylev added this pull request to the merge queue Nov 15, 2026
Merged via the queue into bazel-contrib:main with commit 7dd7797 Nov 15, 2026
17 checks passed
@rickeylev
rickeylev deleted the rust_wheelmaker_conversion_pr branch November 15, 2026 05:39
@rickeylev

Copy link
Copy Markdown
Collaborator Author

I am thinking that the wheelmaker could be useful to for making the wheels after patching them. Since this is just a binary, we could in theory use them in repo rules.

Yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants