Skip to content

Narrow the human-review rubric to judgement calls in the diff - #15

Merged
william-gr merged 1 commit into
masterfrom
narrow-human-review
Sep 12, 2026
Merged

william-gr merged 1 commit into
masterfrom
narrow-human-review

Conversation

@william-gr

Copy link
Copy Markdown
Contributor

Summary

The Does this need a human? section of review/rubric.md was asking for a person on nearly every PR. Three of its criteria covered most changes ("touches CI", "changes user-facing behaviour", "a choice the PR description presents as a decision"), and the last one fired on work a description merely deferred, e.g.:

PR description's "Noted, not fixed" defers WrapAsQuotaError misclassifying read errors as quota errors; the deferral is recorded nowhere in the diff

The section now:

  • defaults to false, and says what a wrong true costs;
  • requires every reason to anchor to a line in the diff;
  • narrows each criterion to removing or changing something (an external contract, a default, a test, a workflow's permissions or what it publishes) rather than adding alongside;
  • lists explicit non-reasons: the PR description or deferred work, LOW findings, size, refactors, new tests, wording, CI changes that do not touch permissions or publishing;
  • points repo-specific always-review paths at human-review-paths instead of the model guessing.

README paragraph updated to match. No code changes.

No Jira ticket on hand for the title; add one so PR ticket passes.

It was firing on nearly every PR: "touches CI", "user-facing behaviour"
and "a choice the PR description presents as a decision" each cover
most changes, and the last one flagged deferred work mentioned only in
a description. The section now defaults to no, requires each reason to
anchor to a line in the diff, narrows every criterion to removal or
change rather than addition, and lists what does not count.
@william-gr
william-gr merged commit fece22b into master Sep 12, 2026
6 of 7 checks passed
@william-gr
william-gr deleted the narrow-human-review branch September 12, 2026 15:34
Comment thread README.md
Comment on lines +293 to +297
human?* section of `review/rubric.md`, which defaults to no and says yes only
for a line in the diff that removes or changes something external callers
depend on, adds or major-bumps a dependency, changes a workflow's permissions,
secrets or what it publishes, changes a visible default or removes a feature,
deletes or loosens a test, or migrates persisted data. The PR description,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW — This paragraph says the rubric "says yes only for" the list that follows, but the list is narrower than the section it summarises in the same commit:

  • review/rubric.md says "adds a runtime dependency"; here it is "adds or major-bumps a dependency", which also covers a dev dependency the rubric deliberately leaves out;
  • the bullet on workflows covers "permissions, the secrets or tokens it holds, what it publishes, or who or what can merge" — the last clause is dropped here, which is the clause that decides whether a merge-gating CI change counts;
  • the seventh criterion — "does something the repository's own review guidelines explicitly say a person decides" — is not mentioned at all, so a consumer reading only the README would not know its own guidelines can force the answer.

An "only for" list is a claim a reader will act on when deciding whether to configure human-review-paths, so it is worth either completing it or making it explicitly a summary.

Comment thread review/rubric.md
- size, or the number of files touched;
- a refactor, rename, or move that keeps behaviour;
- new or extended tests;
- comments, docs, or wording changes;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW — "wording changes" as a flat non-reason reads wider than the rest of the list intends. The criterion it replaces was "changes user-facing behaviour, wording, defaults, or whether a feature is available", and the biggest consumer of this rubric is truenas/webui, where wording is product copy: a PR relabelling a destructive action, or rewording an error a user acts on, now matches "comments, docs, or wording changes" and falls through every remaining criterion (it is not a default, not a feature removal, not an external contract) to an automatic APPROVE.

The other bullets in this list are all qualified so they cannot swallow a user-visible change; this one is not. Something like "comments, docs, or wording that a user does not see" would keep the intent — stop flagging every comment and README tweak — without covering product copy.

@github-actions

Copy link
Copy Markdown

Two findings: two LOW. A human must look — this diff changes when the workflow submits an automatic APPROVE.

Both findings are on their lines. Neither blocks: the check fails at MEDIUM and above.

  • LOW — README.md:293 presents its list as what the rubric "says yes only for", but drops "who or what can merge" from the workflow bullet, widens "adds a runtime dependency" to "adds … a dependency", and omits the repo-own-guidelines criterion entirely.
  • LOW — review/rubric.md:138 lists "comments, docs, or wording changes" as a non-reason without the qualifier the neighbouring bullets all carry. In truenas/webui wording is product copy, so a relabelled destructive action or a reworded user-facing error now matches this bullet and misses every remaining criterion.

Why a person should look

review/rubric.md:108 is the line that decides when submit-verdict.mjs reaches its APPROVE branch rather than the "needs a human review" COMMENT. Narrowing the criteria therefore moves a set of PRs — CI changes that do not touch permissions, user-facing wording changes, additions alongside existing behaviour — from "a person decides" to "the bot approves", and where branch protection counts the bot's approval that changes what can satisfy a required review. Consumers reference this repo at @master, so it lands in all four of them on merge, with no human-review-paths configured here or in claude-review-self.yml to sit under it.

That is the change itself, not a defect in it: the PR argues the old criteria fired on nearly every PR, and the specific cases it names (a deferral mentioned only in a description, "touches CI") were genuinely not judgement calls. The narrowing looks right in direction. It is the kind of loosening whose blast radius someone should agree to deliberately, which is what this flag is for.

The rest holds up: the ## Does this need a human? heading that review/schema.json names is unchanged, dropping "answer required: false with an empty reasons" costs nothing because the schema still says it and submit-verdict.mjs:269 ignores reasons on false, and human-review-paths is described accurately as the floor — submit-verdict.mjs:274 ORs it over the reviewer's answer.

check-ticket is red for the missing ticket reference the PR description already calls out; it is not related to the content of the change.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a human review. 2 finding(s): 2 LOW.

Nothing blocks, but this change is one a person should decide on:

  • review/rubric.md:108 narrows the flag submit-verdict.mjs:274 checks before APPROVE, changing what can satisfy a required review in every consumer at @master
  • review/rubric.md:139 makes a CI change not touching permissions or publishing a non-reason, dropping "touches CI" for a repo whose product is workflows

william-gr added a commit that referenced this pull request Sep 12, 2026
…summary (#16)

## Summary

Follow-up to #15, which merged before this landed. Two LOW findings from
its own review:

- `review/rubric.md`: "comments, docs, or wording changes" as a blanket
non-reason would auto-approve product copy in `truenas/webui` (a
relabelled destructive action, a reworded error). Now "wording a user
does not see".
- `README.md`: the summary of the section claimed to be the full list
but omitted the "what can merge" clause, the runtime-only qualifier on
dependencies, and the repo-guidelines criterion. It now carries all
three and says the rubric is the full list.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant