Skip to content

feat(coverage): add a flag to use the coverage package from the requirements - #4433

Draft
aignas wants to merge 34 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.coverage-tool-py-library
Draft

aignas wants to merge 34 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.coverage-tool-py-library

Conversation

@aignas

@aignas aignas commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Add the --@rules_python//python/config_settings:coverage label flag to use
the version of coverage that is locked in the requirements, e.g.
@pypi//coverage, for bazel coverage without configuring the Python
toolchains:

coverage --@rules_python//python/config_settings:coverage=@pypi//coverage

Previously, using coverage from a pip.parse hub required pointing
py_runtime.coverage_tool at it for every Python version and platform, and
using a py_library there creates a dependency cycle with the toolchain.

When coverage is collected, the flag target is added to the dependencies of
py_binary and py_test, similar to the debugger flag, so its imports,
runfiles and venv symlinks are handled like any other dependency, including
with --venvs_site_packages=yes. The flag is only resolved when collecting
coverage for the target config, so the coverage wheel isn't fetched for
regular builds. It is added to the transition settings, so it can be set for a
single target via config_settings. If the flag isn't set, the coverage_tool
of the Python runtime is used as before.

The flag target can be a py_library that bundles coverage with plugins and
.coveragerc files in its data. The bootstrap merges those files into the
coverage config it writes: options that are lists, such as omit or
exclude_also, are combined, the other options are overridden, and the
options that Bazel needs, such as relative_files and source, are always set
by rules_python.

Work towards #3902

…ge tool

Before this change the coverage collection was only exercised by running
`bazel coverage` on a few targets in CI and the collected data was not
checked.

This adds a test that runs as part of `bazel test`. A `py_reconfig_test` is
built with `--collect_code_coverage` for a Python 3.12 toolchain that has the
bundled coverage tool configured, and `coverage_collection_test` runs it with
the same environment variables that `bazel coverage` sets and checks the line
coverage in the produced lcov file.

* Adds the `collect_code_coverage` attribute to `py_reconfig_*` rules.
* Configures the bundled coverage tool for the Python 3.12 dev toolchain.

Stacked on bazel-contrib#4415.

Work towards bazel-contrib#3902
Registering the toolchain with the `3.12` minor version made it take
precedence over the `3.12.16` toolchain that //tests/toolchains expects.
…ests

Run the instrumented program once and check the line and branch coverage
of particular lines in separate tests.
Allow py_runtime.coverage_tool to be a py_library of coverage.py, such as
@pypi//coverage, by using its coverage/__main__.py file as the entry
point. This allows users to use the coverage version that is locked in
their requirements.
@aignas

aignas commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator Author

This PR was drafted by an AI coding agent (Claude Code) working for @aignas as part of the plan in #3902. It stays a draft until @aignas has reviewed it. Review comments will be handled by @aignas, possibly with the help of the agent; any agent-written replies will be marked as such.

Agent-generated comment, posted automatically on behalf of the user.

A py_library depends on the Python toolchain, so using @pypi//coverage
as the coverage_tool of the toolchain creates a dependency cycle. Instead
find the coverage/__main__.py entry point in the files of the target, so
that @pypi//coverage:extracted_whl_files can be used.
@aignas aignas changed the title feat(toolchain): allow using a py_library as the coverage_tool feat(toolchain): allow using the extracted coverage wheel as the coverage_tool Nov 8, 2026
aignas added 12 commits November 8, 2026 12:54
It is used to test the coverage collection with the coverage tool from a
PyPI hub.
…p hub

Configure a 3.13 toolchain with the extracted coverage wheel from the
dev_pip hub as the coverage tool and run the same coverage collection
checks as for the bundled coverage tool.
Use TEST_TMPDIR directly and rely on the test working directory being the
workspace runfiles directory.
Also drop the lib.py data dependency, which is already in the runfiles via
the instrumented target.
Also test the error for a coverage_tool without coverage/__main__.py and
name the news entry after the PR.
@aignas
aignas marked this pull request as ready for review November 9, 2026 01:17
@aignas
aignas requested a review from rickeylev as a code owner November 9, 2026 01:17
…e tool

Using a `py_library`, such as `@pypi//coverage`, as the `coverage_tool`
created a dependency cycle: the `py_library` resolves the Python toolchain,
whose `py_runtime` depends on the `coverage_tool` again.

Similarly to how `rules_go` builds `nogo`, the `coverage_tool` attribute
now transitions to a configuration where a private bootstrap flag is set
and `collect_code_coverage` is disabled. The `py_runtime` macro drops the
`coverage_tool` dependency in that configuration, so the cycle is broken.
The hermetic toolchains already select on `collect_code_coverage`.
The bootstrap never executes the coverage tool file: it only adds the
grandparent directory of it to `sys.path` and imports `coverage`. Instead
of searching for `coverage/__main__.py` within the files of the target,
use the import paths of a `coverage_tool` that provides `PyInfo`, e.g.
`@pypi//coverage`.

* Adds `PyRuntimeInfo.coverage_imports`, which is mutually exclusive with
  `PyRuntimeInfo.coverage_tool`.
* Adds the `%coverage_imports%` substitution to the site init template.
@aignas
aignas marked this pull request as draft November 9, 2026 10:23
@aignas aignas changed the title feat(toolchain): allow using the extracted coverage wheel as the coverage_tool feat(toolchain): allow using a py_library such as @pypi//coverage as the coverage_tool Nov 9, 2026
@aignas
aignas marked this pull request as ready for review November 9, 2026 10:24
@aignas

aignas commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

OK, this looks in a good shape. I'm glad I remembered that rules_go had some dependency cycle breaking transitions, so I asked claude to first analyse the code there and do something similar here.

Now this is hopefully compatible with what @jvolkman is doing with TreeArtifacts in #4396 and reworking the code to do whl extraction in build phase.

@rickeylev, PTAL.

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clever! I forgot about this trick. I think the Java rules also do this trick to deal with some bootstrapping issues like this.

Doing a transition here is unfortunate, but is pretty clean and a net improvement.

Bot comments incoming in a bit.

Comment thread python/private/BUILD.bazel Resolved
Comment thread python/private/py_executable.bzl Resolved
Comment thread python/private/py_executable.bzl Resolved
Comment thread python/private/py_runtime_macro.bzl Resolved
Comment thread python/private/py_runtime_rule.bzl Resolved
Comment thread tests/coverage/BUILD.bazel Resolved
Comment thread python/private/BUILD.bazel Resolved
Comment thread python/private/common_labels.bzl Resolved
Comment thread python/private/py_runtime_rule.bzl Resolved
Comment thread python/private/py_runtime_rule.bzl Resolved
Comment thread python/private/py_runtime_info.bzl Resolved
Comment thread docs/coverage.md Resolved
Comment thread python/private/py_runtime_info.bzl Resolved
Comment thread python/private/site_init_template.py Resolved
Comment thread MODULE.bazel Resolved
Comment thread python/private/py_executable.bzl Resolved
rickeylev and others added 5 commits November 14, 2026 11:07
Address mechanical review comments and refresh the bzlmod integration
lockfile after merging main:
- Rename `coverage_tool_active` to `is_bootstrap_coverage_tool_false`
  and hoist its label constant into `common_labels.bzl`.
- Compose caller `select()` expressions on `py_runtime.coverage_tool`
  with the bootstrap guard via a private helper rule.
- Simplify `coverage_tool_missing_message` to take `has_coverage_tool`.
- Trim internal transition details from `docs/coverage.md`, add
  `keep-sorted` markers in `MODULE.bazel`, wrap overlong Starlark lines,
  and regenerate `tests/integration/bzlmod_lockfile/MODULE.bazel.lock`.
…trap_runtime

The flag is set when building a dependency of the `py_runtime` itself, so
use a generic name that is not specific to the coverage tool.
…o.coverage_dep

Instead of passing the import paths of a `PyInfo` coverage tool separately
to the site init, the `coverage_tool` target is exposed as
`PyRuntimeInfo.coverage_dep`. When coverage is enabled, executables add it to
their dependencies, so its imports, runfiles and venv symlinks are handled
like any other dependency.

This makes it work with `--venvs_site_packages=yes` and with a `py_library`
without `imports`.
@aignas
aignas marked this pull request as draft November 15, 2026 07:39
`py_library` targets that have a `.dist-info/METADATA` file in their `data`,
such as the ones `pip.parse` creates from wheels, record their normalized
package name in the new `PyInfo.packages` depset, which is propagated
transitively. Previously the name was only available via `venv_symlinks`
when `--venvs_site_packages=yes` was set.

This allows rules to check whether a target depends on a given package, e.g.
so that a `py_test` can tell whether it already depends on `coverage`.
… py_runtime

Instead of accepting a `PyInfo` target as `py_runtime.coverage_tool`, which
needed a transition and a bootstrap flag to break the dependency cycle with
the toolchain, add the `--@rules_python//python/config_settings:coverage`
label flag. When coverage is collected, the flag target is added to the
dependencies of `py_binary` and `py_test`, so its imports, runfiles and venv
symlinks are handled like any other dependency, and it can be set for a
single target via `config_settings`. The flag is only resolved when
collecting coverage, so e.g. `@pypi//coverage` isn't fetched for regular
builds.

If the flag isn't set, a `coverage` package that the executable already
depends on, as reported by `PyInfo.packages`, is used before falling back to
the `coverage_tool` of the Python runtime.
The `.coveragerc` files in the runfiles of the `coverage` flag target, e.g.
from the `data` of a `py_library` that wraps `@pypi//coverage`, are merged
into the coverage config that the bootstrap writes: options that are lists,
such as `omit` or `exclude_also`, are combined and the other options are
overridden. The options that Bazel needs, such as `relative_files` and
`source`, are always set by `rules_python`.
@aignas aignas changed the title feat(toolchain): allow using a py_library such as @pypi//coverage as the coverage_tool feat(coverage): add a flag to use the coverage package from the requirements Nov 15, 2026
…he caller

The values of label flags, such as `--debugger` and `--coverage`, set via
`config_settings` were resolved by the transition relative to `rules_python`,
so e.g. `":my_coverage"` or `"//tools:my_coverage"` referred to targets in
`rules_python` instead of the caller's repository. The `py_binary` and
`py_test` macros now resolve them relative to the package of the caller, like
label attributes.
…kages

Keep this change to the opt-in `--coverage` flag. Using the `coverage`
package that a test already depends on is better handled by a default for the
flag that resolves `@pypi//coverage` to the active venv, which will be done
separately, so drop the detection and the dependency on `PyInfo.packages`.
`PyInfo.packages` is no longer used by the `--coverage` flag, and it is
proposed separately in bazel-contrib#4506, so remove it from this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants