From 69e3d7fef0541c903728460c4e3c114855767f80 Mon Sep 17 00:00:00 2001 From: Richard Levasseur Date: Sun, 1 Nov 2026 01:48:59 -0700 Subject: [PATCH] agents: add merge-queue and soft-failure CI lessons Shepherding #4262 surfaced two agent mistakes: assuming merge-queue builds skip Buildkite's CI config approval step, and reporting soft-failed jobs as passed. Raw `bk job log` output also defeated simple greps. The CI skills now cover finding and unblocking merge-queue builds, which changed files trigger the approval step, checking soft-failed jobs against the latest main build, and cleaning job logs before grepping. --- .agents/skills/buildkite-get-results/SKILL.md | 6 ++++++ .agents/skills/merge-pr/SKILL.md | 15 ++++++++++++--- .agents/skills/monitor-ci-results/SKILL.md | 7 ++++++- 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/.agents/skills/buildkite-get-results/SKILL.md b/.agents/skills/buildkite-get-results/SKILL.md index 801e51c02d..f8cb2c8cdb 100644 --- a/.agents/skills/buildkite-get-results/SKILL.md +++ b/.agents/skills/buildkite-get-results/SKILL.md @@ -8,3 +8,9 @@ Pass the PR number, Build URL, or Build ID to the `scripts/get_buildkite_results The `--jobs` flag can do glob-style filtering of jobs. The `--download` flag will download job logs. + +For quick checks with the `bk` CLI, `bk job log --no-timestamps` +prints a job's log (it doesn't take `--build`). The output keeps a stray +`ESC _` prefix, cursor and color codes, and carriage returns; strip them +before grepping: +`sed 's/\x1b\[[0-9;]*[A-Za-z]//g; s/\x1b_//g; s/\r//g'`. diff --git a/.agents/skills/merge-pr/SKILL.md b/.agents/skills/merge-pr/SKILL.md index 1939d711ee..056e51c09c 100644 --- a/.agents/skills/merge-pr/SKILL.md +++ b/.agents/skills/merge-pr/SKILL.md @@ -31,9 +31,9 @@ this PR", or monitor its merge): transient network flakes (e.g., HTTP 504 gateway timeouts, downloader errors, `setup-bazel` `Unable to find Bazelisk version` errors) and unblock manual approval steps. - - **Blocked Builds (`.bazelci/presubmit.yml` Approval)**: When - `.bazelci/presubmit.yml` is modified, Buildkite blocks the build until - manually approved. Approve and unblock it immediately via + - **Blocked Builds (CI Config Approval)**: When the PR changes a file under + `.bazelci/` or a `.yml` file outside `.github/`, Buildkite blocks the + build until manually approved. Approve and unblock it immediately via `./.agents/skills/retry-ci-job/scripts/retry_ci_jobs.py --pr --unblock` (or `bk job unblock --no-input`) so CI does not sit blocked indefinitely. @@ -44,6 +44,15 @@ this PR", or monitor its merge): `'.[].name | select(test("gh-readonly-queue/.*/pr--"))'` and monitor commit statuses/Buildkite builds running on that temporary branch. + - **Blocked Merge Queue Builds**: If the PR changes a file under + `.bazelci/` or a `.yml` file outside `.github/`, the merge queue build + is blocked again, even with the `presubmit-auto-run` label, and + `monitor_remote_ci.py` doesn't watch it. Find it with `bk build list + --pipeline bazel/rules-python-python --branch --json | + jq '.[] | {number, state, blocked}'`, then unblock it with + `./.agents/skills/retry-ci-job/scripts/retry_ci_jobs.py + --buildkite-build bazel/rules-python-python/ --unblock`. + `--pr ` resolves to the PR's build, not the queue build. 4. **Queue Shepherding**: Periodically check `gh pr view --repo bazel-contrib/rules_python --json state,autoMergeRequest,mergeStateStatus,mergeable`. While `state` is diff --git a/.agents/skills/monitor-ci-results/SKILL.md b/.agents/skills/monitor-ci-results/SKILL.md index c73295ed70..f0b6922f15 100644 --- a/.agents/skills/monitor-ci-results/SKILL.md +++ b/.agents/skills/monitor-ci-results/SKILL.md @@ -43,7 +43,12 @@ or when monitoring CI after PR updates: 3. **Failure Reporting**: When any GitHub check or Buildkite job completes with errors, `monitor_remote_ci.py` dispatches a high-priority notification message reporting the failed check back to your conversation. - - Soft failures (e.g. `*rolling*` Bazel) are non-blocking warnings; ignore. + - Soft failures (e.g. `*rolling*` Bazel) don't block merging, but the + monitor counts them as passed. Before reporting that everything + passed, list them with `bk build view --pipeline + bazel/rules-python-python --json | jq '.jobs[] | select(.soft_failed) + | {id, name}'` and compare their failing tests with the latest `main` + build's. - If `blocks-do-not-merge` fails while the PR intentionally carries the `"do not merge"` label, treat it as an expected merge gate rather than a code failure.