workflows, ci_scripts: add automation logic from wheel_builder#100
Open
threexc wants to merge 12 commits into
Open
workflows, ci_scripts: add automation logic from wheel_builder#100threexc wants to merge 12 commits into
threexc wants to merge 12 commits into
Conversation
Some of the scripts we have ported from the wheel_builder repository are unchanged, and the originals were licensed under Apache-2.0 terms. Add a copy of that LICENSE file as LICENSE-Apache-2.0 for compliance purposes. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Collaborator
Author
threexc
force-pushed
the
tgamblin/migrate_automation
branch
from
62ee69f to
6f8ae60
Compare
Collaborator
Author
|
Everything seems to work - latest run passed everything but the |
justeph
reviewed
Jul 9, 2026
|
|
||
| def find_upstream_issue(package: str) -> Optional[str]: | ||
| """ | ||
| Find issue for a package in the wheel_builder's Upstream milestone. |
Collaborator
There was a problem hiding this comment.
use the old name, should be python-wheels
| run: | | ||
| sed -Ei 's/^scipy.*$//' packages.txt | ||
| sed -Ei 's/^patchelf.*$//' packages.txt | ||
| docker run --platform=linux/riscv64 -w /root --rm \ |
Collaborator
There was a problem hiding this comment.
any reason to run inside docker and not just inside a virtual env since we are on a riscv runner?
Collaborator
Author
There was a problem hiding this comment.
I'll modify it and give it a try.
Collaborator
I am wondering if it worth opening an issue.. Most of the time, we won't fix the issue ourself, but we will wait for a new release upstream fixing the issue.. |
Migrate the script from wheel_builder, with the following changes: - Add new RISE copyright - Include a short description - Make it output a json report for further analysis Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
threexc
force-pushed
the
tgamblin/migrate_automation
branch
2 times, most recently
from
dfb5868 to
27dc8a1
Compare
Migrate the check_versions.py script from wheel_builder. Much of the script has been refactored with the following goals: 1. Use GitHub conventions (instead of GitLab) 2. Create GitHub-specific replacement functions where necessary 3. Simplify other functions where possible 4. Use more Pythonic styling 5. Remove comments where the code is self-documenting More specifically: - Replace usage of `glab` with the corresponding `gh` CLI - Create a REPO variable pointing to "riseproject-dev/python-wheels", and use it wherever the wheel_builder was referenced (in calls to `glab`) - Change configure_git_identity() to use a "github-actions[bot]" user and corresponding noreply email - Change generated branch names' prefix from "gitlab-ci" to "github-actions" - Remove some auth/login-related logic using tokens to generate automated PRs, since the GitHub project allows us to enable this in the repository settings (not programatically) - Add dispatch_workflow() and create_upgrade_pr() functions for triggering a desired workflow (e.g. for a "matplotlib" upgrade, the script should trigger the "build-matplotlib.yml" and "test-matplotlib.yml" workflows for the target version(s) if the workflow scripts exist, or indicate otherwise if not possible - Allow updating of default versions in workflow scripts when auto-generating upgrade PRs. - Add "Trigger" labels to auto-generated PRs for the new versions detected. - Importantly, limit the maximum number of versions to be targeted by an auto-generated PR to three, so as not to create a runaway build queue. - Remove most comments, since the code objectives are obvious after the refactor AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Migrate check_patch.py from the wheel_builder repository, adding a new RISE copyright and updating the description. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Create a new script for reviewing reports created by ci_scripts/audit.sh, and opening/commenting on issues in GitHub to track vulnerabilities identified by nightly pip-audit runs. The issues are opened on a per-vulnerability basis with affected versions, but only if an existing issue matching the same vulnerability and with the "pip-audit" label isn't found. Previously-closed issues are ignored when determining if a new one should be opened. Also use GHA warning notations so that the nightly CI pipeline can report a pass (instead of a failure) while still reporting a vulnerability in the issue list. AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Migrate the deprecated packages file from wheel_builder, adding a new RISE copyright. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Migrate the supported packages file from wheel_builder, adding a new RISE copyright in the process. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Migrate the script from wheel_builder, adding a new RISE copyright and a short description in the process. Also make sure that it only tries to install binary wheels, so that we don't try and build them from scratch during simple checks. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Add a workflow for triggering workflows named "build-<package>.yml" (and if present, "test-<package>.yml") with desired release tags for a given package. For automated upgrade checks and manual changes, this means that the tags only need to be specified in the PR description when opened. This is based on the process in wheel_builder where a child pipeline would be generated automatically when a merge request was submitted with one or more package version tags listed as e.g. "Trigger: polars:py-1.42.1" in the MR description. AI-Generated: Uses Claude Code Sonnet 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Provide a workflow for verifying that content added/modified in commits meets basic requirements, i.e. no debug patches with "revertme", "revert me", or "DO NOT MERGE" at the beginning. Also ensure that an "Upstream-Status" line is provided in any carried patches (by running check_patch.py), so that developers understand the patches' purpose. This is based on a combination of the "check_commit_messages" and "check_patches" jobs in wheel_builder's .gitlab-ci.yml script. AI-Generated: Uses Claude Code Sonnet 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Provide a nightly CI workflow for checking: 1. Latest upstream package versions versus those available in the RISE registry 2. Packages which have upstream riscv64 wheels (ready for deprecation in our CI) 3. Check that packages install correctly in a riscv64 container 4. Run pip-audit to scan for vulnerabilities This is based on the following jobs from wheel_builder's .gitlab-ci.yml script: - check_deprecated_packages - check_installation - check_versions - pip_audit Two notable differences compared to the original logic (in addition to the GitLab CI -> GHA rework): 1. We use RISE's RISC-V Runners for the steps which would otherwise run the setup-qemu action. 2. A step is added using the new audit_report.py script to file issues when vulnerabilities are detected. AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
threexc
force-pushed
the
tgamblin/migrate_automation
branch
from
27dc8a1 to
ccbf97d
Compare
Collaborator
Author
For now I've gone ahead with making it open issues, but I've designed it so that it'll look for an open one and comment on it if the issue's vulnerability ID matches, so we don't get flooded with too many. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrate the automated CI scripts we've been using to maintain the RISE Python Packaging efforts to GitHub. This is a fairly large effort, and I've used Claude to help accelerate the conversion process from wheel_builder's GitLab CI pipelines to Actions workflows, including corresponding changes to the Bash and Python scripts which the workflows still need to use. Compared to the wheel_builder versions, some of these pipelines have been simplified as a result of GitHub's UI-based enablement of automated PR creation/handling (so we don't need to provide and manage a unique token for that or consume it).
Some of the scripts needed no obvious editing (testing will indicate if additional changes are needed). According to the Apache-2.0 terms that they were originally licensed under, those copyright notices must be retained and a copy of the license provided. The first commit in this series does the latter.
Trigger: numpy:v2.5.1