Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/claude-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,15 @@ on:
judgement of what needs a person.
type: string
default: ''
human-review-team:
description: >-
Team slug in the calling repository's org to request a review from
when a human review is needed, once per PR. The team's code review
auto-assignment decides who gets it. Empty disables the request.
Skipped without a github-token: the job token cannot request team
reviewers (GitHub answers 422, unable to resolve the team).
type: string
default: 'ux-team'
tooling-ref:
description: >-
Ref of this repository to take the schema, rubric and scripts from.
Expand Down Expand Up @@ -577,4 +586,6 @@ jobs:
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
APPROVE_WHEN_CLEAN: ${{ inputs.approve-when-clean }}
HUMAN_REVIEW_PATHS: ${{ inputs.human-review-paths }}
HUMAN_REVIEW_TEAM: ${{ inputs.human-review-team }}
HAS_GITHUB_TOKEN: ${{ secrets.github-token != '' }}
run: node .claude-review/tooling/review/submit-verdict.mjs
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,7 @@
| `extra-allowed-tools` | `''` | Comma-separated permission rules appended to the reviewer's `--allowedTools`, e.g. `Bash(go vet:*)`. Empty by default on purpose: anything that executes repo code runs PR-controlled code next to the job's write token, so each repo opts in as its own recorded decision |
| `approve-when-clean` | `true` | Submit an APPROVE review when nothing blocks and no human review is needed. Set `false` to get a COMMENT saying it would have approved instead, to watch the calls before they count |
| `human-review-paths` | `''` | Newline-separated globs, gitignore rules: `*`, `**`, `?`; a name with no slash (trailing one aside) matches at any depth, one with a slash is root-anchored, a directory match covers everything beneath it; no `!`, leading `/`, brackets or braces; `#` lines ignored. A PR touching a match always gets the needs-a-human COMMENT, whatever the reviewer decided |
| `human-review-team` | `ux-team` | Team in the calling repo's org asked for a review when a human review is needed, once per PR. Needs `github-token`; skipped with the job token. Empty disables it |
| `tooling-ref` | `master` | Ref this repo's `review/` assets come from; see below |

The secret is named, not inherited, because the repos call it different things
Expand Down Expand Up @@ -284,7 +285,7 @@
| Result | Review submitted | Job |
|---|---|---|
| Anything at MEDIUM or above | REQUEST_CHANGES | fails |
| Only LOW or none, a human must look | COMMENT listing the reasons | passes, no approval |
| Only LOW or none, a human must look | COMMENT listing the reasons, and with a `github-token` a review request to `human-review-team` | passes, no approval |
| Only LOW or none, `approve-when-clean` off | COMMENT "would approve" | passes |
| Only LOW or none, `approve-when-clean` on | APPROVE | passes |
| Only LOW or none, but GitHub refuses the APPROVE | COMMENT quoting the refusal | passes |
Expand Down Expand Up @@ -325,6 +326,17 @@

Making the review count is branch protection, per repo:

- **Turn on code review auto-assignment for the team** named by

Check notice on line 329 in README.md

View workflow job for this annotation

GitHub Actions / claude-review / Automatic PR review

LOW: The code review auto-assignment bullet is an org team setting but sits under the branch-protection, per-repo heading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW — This bullet sits under "Making the review count is branch protection, per repo:" (line 327), but it is neither: code review auto-assignment is an org team setting, configured once per org rather than per repo, and it does not make the review count towards anything branch protection checks. Every other bullet in the list is a repository branch-protection or Actions setting. A reader following the section heading to their repo's protection rules will not find it there.

It reads as its own paragraph before the list — something like "The team request needs one setting outside this repo:" ahead of it, with the branch-protection list starting at Require 1 approval.

`human-review-team`, in each org's team settings. The workflow requests the
team; auto-assignment is what swaps the team for one member. Without it the
whole team stays requested. The request is made once per PR and skipped if
any reviewer is already requested, so pushes do not reassign it. That
includes code owners: in a repo whose CODEOWNERS already names the team for
every file, GitHub requests it on every PR and this adds nothing. It needs
`github-token`, because the job token cannot request team reviewers; without
one the request is skipped with a log line. A fine-grained PAT may also need
the organisation Members read permission. A refused request is a warning,
never a change to the check.
- **Require 1 approval** and mark `Automatic PR review` required. The
workflow's approval satisfies the first, which is the whole mechanism and the
whole risk: a PR the reviewer misjudges as routine merges with no person
Expand Down
53 changes: 51 additions & 2 deletions review/submit-verdict.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@
* `gh pr review`, so the review state on the PR cannot disagree with the check.
*
* any finding >= MEDIUM REQUEST_CHANGES, job fails
* clean, but a human must look COMMENT naming why, job passes
* clean, but a human must look COMMENT naming why, team review

Check notice on line 12 in review/submit-verdict.mjs

View workflow job for this annotation

GitHub Actions / claude-review / Automatic PR review

LOW: Header outcome table says the team review is requested once per PR without the github-token condition, which is the default for every consumer
* requested once per PR, job passes
Comment on lines +12 to +13

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW — This outcome table now says the human-review path requests a team review "once per PR", unqualified, but requestTeamReview returns early without a github-token (line 269) — which is the default for every consumer, including this repo's own claude-review-self.yml, which maps no github-token. The README's equivalent table row was qualified with "and with a github-token" in d291001; this header comment is the same claim left unqualified.

Suggested change
* clean, but a human must look COMMENT naming why, team review
* requested once per PR, job passes
* clean, but a human must look COMMENT naming why, and with a
* github-token a team review
* requested once per PR, job passes

* clean, approve-when-clean off COMMENT "would approve", job passes
* clean, approve-when-clean on APPROVE, job passes
* clean, but GitHub refuses the APPROVE COMMENT saying so, job passes
Expand Down Expand Up @@ -44,6 +45,9 @@
const number = Number(env('PR_NUMBER'));
const head = env('HEAD_SHA');
const approveWhenClean = env('APPROVE_WHEN_CLEAN') === 'true';
const humanReviewTeam = env('HUMAN_REVIEW_TEAM');
const hasGithubToken = env('HAS_GITHUB_TOKEN') === 'true';
const REQUESTED_MARKER = '<!-- claude-review-requested-team -->';
const humanReviewPaths = env('HUMAN_REVIEW_PATHS')
.split('\n')
.map((p) => p.trim())
Expand Down Expand Up @@ -83,6 +87,10 @@
return all;
};

// May be filled before the verdict is posted; callers that must not see it exclude it by id.
let reviewsCache;
const listReviews = async () => (reviewsCache ??= await paginate(`/pulls/${number}/reviews`));

/** The reviewer's terminal API error, when it died before producing output. */
const terminalApiError = () => {
const file = env('EXECUTION_FILE');
Expand Down Expand Up @@ -226,7 +234,7 @@
*/
const dismissOwnStateReviews = async (own) => {
try {
const reviews = await paginate(`/pulls/${number}/reviews`);
const reviews = await listReviews();
const stale = reviews.filter(
(r) =>
['CHANGES_REQUESTED', 'APPROVED'].includes(r.state) &&
Expand All @@ -247,6 +255,45 @@
}
};

/**
* Asks the caller's org team for a review; the team's auto-assignment picks
* the person. Once per PR: skipped when any reviewer is already requested
* (CODEOWNERS included — after auto-assignment the team is often replaced by
* a user, so checking for the team alone would assign a second person), or
* when an earlier verdict recorded a request. Best effort. Returns the lines
* for the review body.
*/
const requestTeamReview = async () => {
if (!humanReviewTeam) return [];
const team = `${repo.split('/')[0]}/${humanReviewTeam}`;
if (!hasGithubToken) {
console.log(`Not requesting ${team}: the job token cannot request team reviewers. Pass a github-token.`);
return [];
}
try {
const requested = await api(`/pulls/${number}/requested_reviewers`);
if (requested.users?.length || requested.teams?.length) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW — This guard also disables the feature outright in any consumer with a CODEOWNERS file. GitHub auto-requests code owners when the PR opens, so requested.users/requested.teams is already non-empty on the first verdict run and the team is never asked. The comment above frames the guard as being about pushes not reassigning ("so each push does not assign someone new"), and README's own CODEOWNERS bullet a few lines below the new one documents that configuration as supported — so a reader of either would not expect the request to be skipped from the start.

Nothing produces a wrong result (someone is already requested, which is arguably the point), so this is LOW rather than a bug. Worth a sentence in the README bullet, or narrowing the guard to requested.teams plus reviewers the workflow itself requested, if the intent is that ux-team gets asked even when code owners already were.

console.log('A reviewer is already requested; not requesting the team.');
return [];
}
const reviews = await listReviews();
if (reviews.some((r) => (r.body ?? '').includes(MARKER) && r.body.includes(REQUESTED_MARKER))) {
console.log(`${team} was requested on an earlier run; not requesting again.`);
return [];
}
await api(`/pulls/${number}/requested_reviewers`, {
method: 'POST',
body: JSON.stringify({ team_reviewers: [humanReviewTeam] }),
});
console.log(`Requested a review from ${team}.`);
// Backticks, not an @-mention: a mention would notify the whole team.
return ['', `Requested a review from \`${team}\`.`, REQUESTED_MARKER];
} catch (error) {
console.log(`::warning::could not request a review from ${team}: ${escapeData(error.message)}`);
return [];
}
};

const approveHint = (error) => {
if (!/HTTP 422/.test(error.message)) return;
console.log(
Expand Down Expand Up @@ -284,12 +331,14 @@
// On the flag, not the reason count: `required: true` with no reasons is
// schema-valid and must not approve.
if (humanReview.required || reasons.length > 0) {
const requestLines = await requestTeamReview();

Check notice on line 334 in review/submit-verdict.mjs

View workflow job for this annotation

GitHub Actions / claude-review / Automatic PR review

LOW: The once-per-PR marker is recorded by the review posted after the request, so a request that lands while the review post fails can assign a second reviewer on a later run

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW — The "once per PR" record is written by the review posted after the request, so a request that lands while the review that would record it does not leaves no trace of itself.

Sequence: run A requests the team, then submit('COMMENT') throws (a dropped connection, a 5xx, the 15s AbortSignal.timeout) and the outer catch at line 383 turns it into a warning — no review body carries REQUESTED_MARKER. Auto-assignment swaps the team for Alice, Alice submits a review, and GitHub clears her from requested_reviewers. The next push reaches run B with both guards open: the requested list is empty and no review carries the marker, so it requests the team again and a second person is assigned.

Narrow, and the feature is best effort by design, so this is not worth much — but it is the one case the marker exists to cover and does not. Requesting after the review is posted, with the marker in the body either way, would close it.

const own = await submit('COMMENT', [
`**Needs a human review.** ${countLine}`,
'',
'Nothing blocks, but this change is one a person should decide on:',
'',
...(reasons.length ? reasons : ['The reviewer flagged this as needing a person but gave no reason.']).map((r) => `- ${r}`),
...requestLines,
]);
await dismissOwnStateReviews(own);
console.log(`Human review required: ${reasons.length} reason(s).`);
Expand Down
Loading