Skip to content

fix(bzlmod): use a minimalistic schema facilitating shallow merging - #4394

Merged
aignas merged 16 commits into
bazel-contrib:mainfrom
aignas:aignas.fix.shallow-facts
Nov 3, 2026
Merged

aignas merged 16 commits into
bazel-contrib:mainfrom
aignas:aignas.fix.shallow-facts

Conversation

@aignas

@aignas aignas commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

It seems that the automatic bzlmod lockfile driver is doing only shallow
merging, let's use a different schema that is much more friendly towards
such workflows. Before this PR users would have to manually resolve git
conflicts or regenerate the lockfile from scratch, with this it will
work better as the bazel authors intended.

Flatten pip extension lockfile facts (v3) to top-level per-package
keys (<pkg>#<index_url> and <root_url> <distribution>) with compact
pipe-delimited distribution metadata strings.

Fixes #4162
Related bazelbuild/bazel#31139

It seems that the automatic bzlmod lockfile driver is doing only shallow
merging, let's use a different schema that is much more friendly towards
such workflows. Before this PR users would have to manually resolve git
conflicts or regenerate the lockfile from scratch, with this it will
work better as the bazel authors intended.

Fixes bazel-contrib#4162
Related bazelbuild/bazel#31139
@aignas
aignas requested a review from rickeylev as a code owner November 1, 2026 10:18
@aignas

aignas commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator Author

Looking at this, I think maybe it would be good to have the key of the fact as <distro name>@<index>. This would make things a little more robust - if we suddenly start supporting local indexes and there is a space in the path, then this would break, whereas <distro name> will never contain spaces. The keys are URLs for each file (and they are full URLs), it is a bit annoying that they are this long, but there is an obvious optimization to only store suffix as the key if the index URL is the prefix of the file URL.

Lastly there was someone asking (#4278) if we could reduce the lock file even more, and I guess we could, but that would require signalling to the cache about what to retain in the select_whl code where we are selecting suitable wheel candidates. And if someone decides to add another platform into the mix with the same lock file, then we would have to somehow understand that we need to check the mirror again to fetch other keys that we have discarded just now. Not worth the effort here.

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review, but overall lgtm. agent will post some comments automatically shortly

Comment thread python/private/pypi/pypi_cache.bzl Resolved
Comment thread python/private/pypi/pypi_cache.bzl Resolved

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit on the PR description (since it becomes the merge commit message): could you move the AI note ("Implementation done by an opencode clanker...") to a PR comment per CONTRIBUTING.md, and add a sentence summarizing how v3 flattens the schema (per-package top-level keys + compact pipe-delimited strings)?

This is an agent-generated comment reviewed and approved by the user.

Comment thread python/private/pypi/pypi_cache.bzl Outdated
Comment on lines 219 to 228
raw_index_urls = {}
for pkg in requested_versions.keys():
fact_key = pkg + "#" + index_url
if fact_key in known_facts:
raw_index_urls[pkg] = known_facts[fact_key]

result = _filter_packages(
dists = known_facts.get("index_urls", {}).get(index_url, {}),
dists = raw_index_urls,
requested_versions = requested_versions,
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: since raw_index_urls already iterates over requested_versions.keys(), _filter_packages is a no-op copy here—returning None inside the loop when fact_key not in known_facts avoids the extra pass and len(result) != len(requested_versions) check.

This is an agent-generated comment reviewed and approved by the user.

Comment thread python/private/pypi/pypi_cache.bzl Outdated
distribution = distribution.rstrip("/")
root_url = root_url.rstrip("/")

fact_key = root_url + " " + distribution

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to your comment about using <distribution>@<root_url> (distribution + "@" + root_url) instead of root_url + " " + distribution—it matches the package-first ordering of pkg + "#" + index_url (while keeping @ distinct from # so index-page and package-page keys can never collide), groups lockfile entries by package name, and avoids another fact_version bump later. (On stripping index_url prefixes from file URLs: since pypi.org/simple serves wheels from files.pythonhosted.org, prefix stripping wouldn't shrink standard PyPI entries anyway.) Worth switching to distribution + "@" + root_url before merging v3?

This is an agent-generated comment reviewed and approved by the user.

Comment thread python/private/pypi/pypi_cache.bzl Outdated
Comment on lines +268 to +269
if digest:
known_sources.setdefault("hashes_by_version", {}).setdefault(version, []).append(digest)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping empty digest values here makes _get_from_facts diverge from parse_simpleapi_html (parse_simpleapi_html.bzl:124) and _filter_packages (pypi_cache.bzl:156,163), which both record [""] in hashes_by_version[version]. parse_requirements.bzl:693-703 iterates hashes_by_version[version] and looks up index_urls.whls.get(digest), so omitting "" causes lockfile hits for hashless SimpleAPI entries + hashless requirements.txt to miss the wheel and fall back to pip. Removing if digest: restores round-trip parity (and lets pypi_cache_tests.bzl:553 expect {"1.0.0": [""]}).

This is an agent-generated comment reviewed and approved by the user.

Comment thread news/4394.fixed.md Resolved
@rickeylev

Copy link
Copy Markdown
Collaborator

Moved the author's AI disclosure note from the PR description into this comment (and added a brief v3 schema summary to the PR description) so the merge commit message stays clean per CONTRIBUTING.md:

Implementation done by an opencode clanker, but the design is mine.

This is an automatically generated agent comment.

Improve `CHANGELOG.md` readability by mentioning `{obj}\`pip\`` and
`MODULE.bazel.lock`, wrapping lines at 80 columns, and formatting the
`bazel-contrib#4162` link in trailing parentheses.
@rickeylev

Copy link
Copy Markdown
Collaborator

Ignas: if the bot comments aren't helpful, then just comment/rant to them in response. I feed it back into the code review skill to incrementally improve it. (Especially around verbosity)

My rule of thumb is: if the gut reaction is "wtf are you talking about?" then its done a bad job. Human comments are usually clear enough that we think e.g. "I understand what they mean, they're (maybe|certainly) (right|wrong)"

rickeylev and others added 12 commits November 1, 2026 13:29
Align `bazel_binaries` in `MODULE.bazel` and `bzlmod_lockfile_test` in
`tests/integration/BUILD.bazel` with `9.3.1` in
`tests/integration/bzlmod_lockfile/.bazelversion` (`lockFileVersion: 28`).
This restores parity with parse_simpleapi_html, so hashless requirements
resolved against a hashless index still find the wheels on a lockfile hit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…acts

The loop only visits requested packages, so return early on the first
missing package instead of filtering and comparing lengths afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This groups the lockfile entries by package name, matching the
<package>#<index_url> keys, and does not break if the root URL contains
spaces. The key computation is shared between the read and write paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use `raw_facts` for the plain dict and `expect_facts` for the assertion
subject, so they are not confused with each other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The format is now `<digest>[|<filename>[|<yanked>]]`, so the common case
is just the digest and no value starts with an empty field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decoding the entries into structs and encoding them back was wasted work.
Copying the strings also guarantees that a no-op run does not change
MODULE.bazel.lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`sdists | whls` merged two digest-keyed dicts, so a hashless wheel
overwrote a hashless sdist under the `""` key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers an empty yank reason, hashless yanked and renamed distributions,
and a URL with a query string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the same separator as the distribution facts. The keys stay distinct
because the index URL always ends with `/` and the root URL never does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@aignas aignas left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double checked AI output. LGTM

@aignas
aignas added this pull request to the merge queue Nov 3, 2026
Merged via the queue into bazel-contrib:main with commit c0cc206 Nov 3, 2026
13 checks passed
@aignas
aignas deleted the aignas.fix.shallow-facts branch November 3, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pip extension facts structure prevents correct git merge of MODULE.bazel.lock

2 participants