From 6a2d0e6daadc36bae0e735a918bbdaac3b210722 Mon Sep 17 00:00:00 2001 From: Zack Maril Date: Sat, 10 Oct 2026 11:54:32 +0000 Subject: [PATCH] docs/coverage-plan.md: coverage beyond functions and blocks, ranked by bugs per effort (branch arms, configuration branches, keyed per-query engine coverage, diagnostics, feature gates consulted, incremental transitions, ...), with hooks, denominators, costs, fuzzer feedback, reports, a build order and log budgets Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01QXiEXbESemwqMLYKaWLDbT --- README.md | 1 + docs/coverage-plan.md | 356 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 357 insertions(+) create mode 100644 docs/coverage-plan.md diff --git a/README.md b/README.md index 75f672c..d21ff7c 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ fixture, the third from fuzzing edits and replaying ten crates' git histories. - [`docs/untracked-reads.md`](docs/untracked-reads.md): reporting reads of untracked state inside rustc - [`docs/grammar.md`](docs/grammar.md): measuring the fixture against Ur's Rust grammar, and filling the gaps - [`docs/coverage.md`](docs/coverage.md): which of the compiler's functions compiling the fixture reaches +- [`docs/coverage-plan.md`](docs/coverage-plan.md): coverage beyond functions and blocks: branch arms, per-query engine paths, diagnostics, feature gates, incremental transitions; ranked, with a build order - [`docs/solver.md`](docs/solver.md): the UI suite under nightly's default trait solver, which the suite does not test - [`docs/props.md`](docs/props.md): MIR validation, optimization levels and the new trait solver on the same corpus - [`docs/plan.md`](docs/plan.md): the plan the work followed, with the properties diff --git a/docs/coverage-plan.md b/docs/coverage-plan.md new file mode 100644 index 0000000..bb41612 --- /dev/null +++ b/docs/coverage-plan.md @@ -0,0 +1,356 @@ +# Coverage beyond blocks: a plan + +Function and block coverage say which code of the compiler ran. They do not say which way a +branch went into a block that other paths also reach, which query a generic engine function +ran for, which kind of type reached a function, which diagnostics and feature combinations were +exercised, or what happened outside the compiler's own crates (the standard library, LLVM). +This plan proposes those dimensions, ranks them by bugs caught per unit of effort, and gives a +build order. Nothing here is built yet. + +## Where coverage stands + +| measure | denominator | covered | source | +|---|---|---|---| +| functions | 62,516 reachable by the call graph (of 71,257 instrumented; 8,741 cannot run) | 50,762 (81.2%); 82.3% without the 911 that only panic | [`coverage.md`](coverage.md), all suites | +| basic blocks | about 696,000 in reachable functions, panic-only blocks aside; 100,208 blocks of logging macros counted apart | 80.7% (2026-10-09 block run, all suites) | `mirth-lab callgraph --block-gaps` | +| grammar alternatives | Ur's Rust grammar, per labeled alternative and literal | per fixture | `mirth-lab grammar-coverage` | +| options | accepted `-C`/`-Z` values and pairs | covering arrays, code reached per row | `flag-universe`, `coverage-flags` | +| UI tests | functions each test adds over a baseline | a greedy pick of tests | `ui-coverage` | + +The checks of the third batch (rustdoc-diff, lint-check, gate-mutate, abi-diff, ...) are being +measured as suites now (`rustc/coverage-checks.sh`), and coverage-guided gate-mutate is being +built on `mirth/covfuzz`. Both use the site set described next. + +### How the current measure works + +`mirth-watch` (`crates/mirth-watch/src/sites.rs`, `instrument`) rewrites each body in scope +(`rustc/coverage.toml`: `rustc_*`) before code generation: under `[coverage] functions` it +inserts a call to the runtime's `cover(site)` at entry, under `blocks` at the start of every +other non-cleanup block, tagging blocks every path from which panics (`panic_only_blocks`) and +blocks that are all logging-macro code (`log_only`). A site is a 64-bit identity of +`"{caller}|cover|{block}"`. The runtime (`crates/mirth-runtime/src/coverage.rs`) keeps a +lock-free open-addressed set of 4M slots (32 MiB); a site's first hit inserts it, later hits +cost one atomic load. At exit the set is written to the process's log under `MIRTH_OUT`; +`coverage-compact` folds logs into a suite's `union.txt`; `mirth-lab callgraph` divides by what +the call graph (`rustc/callgraph.toml`, `[diagnostics] callgraph`) says can run. + +Two limits matter for everything below: + +- **A generic body has one site per block, not per instance.** The query engine + (`rustc_query_impl::execution::try_execute_query`), the dependency graph + (`rustc_middle::dep_graph::graph::try_mark_green`), folders and visitors are each one body + for hundreds of queries and types. A block in `try_execute_query` counts as covered once any + query takes it. +- **A block reached from several predecessors hides which predecessor.** The two arms of an + `if` that join, or a `match` whose arms fall into a shared continuation, are one site. + +The other mechanisms mirth already has: frames (`[[frames]]`: enter, exit, captured +arguments through the `Capture` trait, which takes strings, paths and integers, or `Debug`), +calls (`[[calls]]`, before a call to a matching function, with captured operands), statics +(`[statics]`, each touch of mutable or interior-mutable state), the call graph (`edge`, `body`, +`demand`, `specbound` lines), and the patches (`verify-reuse.patch`, `report-untracked.patch`, +which also turns every option field read into a `read_