Headstart: start dependents on early metadata, in cargo check and cargo build

[zmaril]

Headstart starts dependent crates as soon as a dependency's item interfaces are checked, instead of after all of its function bodies are, for both cargo check and cargo build.

On rustc's default front end, it makes clean builds of 13 real projects up to 54% faster for cargo check and up to 42% for cargo build, and none slower. Failing builds print the same diagnostics as today.

What changes

• rustc -Zearly-metadata (patch):

  • A new analysis_interfaces query splits analysis into item interfaces and function bodies.
  • Between the two, the driver writes libfoo.early-rmeta and announces it with an early-metadata artifact notification.
  • Crate loading accepts early metadata. Before code generation, each dependency loaded from it is swapped for its full metadata; before linking, rustc waits for the rlibs.
  • While waiting, rustc announces wait-metadata, polls, then announces resume. It never touches the jobserver.
  • The crate hash is upstream's metadata-bytes hash, computed over the early metadata plus a supplement that includes the HIR hash of every body. Full metadata carries the same hash.
  • If the crate fails, through linking, the early file is deleted, and anything waiting on it stops.
  • -Zearly-metadata-verify reports two things:
    • any left-out definition whose body was type-checked during the early write anyway;
    • any query that only full metadata can answer, asked of a crate still loaded from early metadata.

• cargo CARGO_HEADSTART=1 (patch):

  • Passes -Zearly-metadata to every compile.
  • Starts every rustc unit on its rlib dependencies' early-metadata notification, in check and build. That covers libraries, binaries, tests, proc macros and build scripts.
  • Doesn't count paused compilations as running, so their job slots go to other work.
  • Reports a unit's output only once its dependencies have succeeded.

  With the variable unset, cargo behaves like upstream.

docs/design.md covers how it works, what early metadata leaves out, the risks, and prior art (#64112).

Correctness

• Sweeps (scripts/sweep.sh):
  • What they build: all 53 rustc-perf compile benchmarks, headstart off and on, with -Zearly-metadata-verify.
  • Six sweeps: cargo check and cargo build, each in debug, release and debug with -Zthreads=8.
  • Result: every sweep is clean. 52 benchmarks pass in both modes with identical diagnostics and no verify reports. stm32f4 fails in both modes (it needs a device feature).
• Real projects: 312 builds, all succeeded with identical diagnostics. The only difference is lemmy under -Zthreads, whose overflow warnings move between runs even without headstart.
• scripts/check-errors.sh: an error in a dependency's body and an error in the binary, under check and build, in human and JSON formats. Diagnostics, exit status and the binary's output are identical.
• scripts/check-incremental.sh check|build: ten edit steps. Each matches headstart-off, and the final state matches a clean build.
• scripts/check-swap.sh: a library that starts on early metadata and swaps in the full metadata while paused, at opt-levels 0–3, s and z. It fails at 2 and 3 without the first fix below.
• Test suites, run with the patches applied and headstart off:
  • rustc's UI suite: 22129 passed, 0 failed.
  • cargo's test suite: 4034 passed. The one failure only flags the missing cross-compile target.

Timing (Linux, 16 cores, median of 3 runs)

project	cargo check	cargo build	on top of -Zthreads=8 (check / build)
rust-analyzer	90 → 42 s (54%)	138 → 80 s (42%)	25% / 22%
polars	172 → 87 s (49%)	341 → 241 s (29%)	13% / 9%
wasmtime	151 → 80 s (47%)	245 → 156 s (36%)	14% / 13%
bevy	121 → 85 s (29%)	196 → 137 s (30%)	4% / 7%
zed	280 → 204 s (27%)	425 → 318 s (25%)	6% / 9%
lemmy	449 → 331 s (26%)	532 → 372 s (30%)	13% / 14%
typst	86 → 63 s (26%)	142 → 123 s (13%)	22% / 2%
atuin	125 → 109 s (13%)	163 → 127 s (22%)	−1% / 5%
vaultwarden	116 → 107 s (8%)	165 → 150 s (9%)	3% / 4%
zola	117 → 115 s (2%)	126 → 126 s (0%)	−2% / 4%

The full tables are in docs/results.md. They also cover helix, lldap, nushell and 21 rustc-perf benchmarks, where headstart saves 5–46% and, on top of -Zthreads=8, 0–32%.

• Deep workspaces gain most. They chain their own crates one after another.
• Wide builds that end in one big crate gain least. zola ends with 81 s of a single crate compiling alone.
• The parallel front end covers some of the same ground. On top of it, headstart adds up to 25%, and three results are within noise of even.

Bugs found by real projects, all fixed

Until this round, the sweeps built only in debug mode, so none of these paths had been exercised:

1. Optimized dependencies (typst, zed): "missing optimized MIR". The early write optimized MIR, and the MIR inliner cached "no MIR" answers from dependencies still loaded from early metadata.
2. Statics evaluated on one thread before the early write. Under -Zthreads, zola was 80% slower with headstart. regex-automata's long-standing small loss had the same cause.
3. cargo check --release: full metadata computed the reachable set. It asked early dependencies for MIR, and the result was never read.

A review of this PR also found that rustc withdrew early metadata only until code generation started. Its guard now lasts through linking.

Open

The next PR in the stack has the full list of what's left, and how to move this work to Claude Code containers. The main items:

• Scheduling when the machine is saturated.
• Fewer cores.
• Peak memory under cargo build.
• Tests that could live upstream.
• A rebase onto a newer rustc.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Tet2pfGBx9bM6ge28w8rgM