Narrow the human-review rubric to judgement calls in the diff - #15
Conversation
It was firing on nearly every PR: "touches CI", "user-facing behaviour" and "a choice the PR description presents as a decision" each cover most changes, and the last one flagged deferred work mentioned only in a description. The section now defaults to no, requires each reason to anchor to a line in the diff, narrows every criterion to removal or change rather than addition, and lists what does not count.
| human?* section of `review/rubric.md`, which defaults to no and says yes only | ||
| for a line in the diff that removes or changes something external callers | ||
| depend on, adds or major-bumps a dependency, changes a workflow's permissions, | ||
| secrets or what it publishes, changes a visible default or removes a feature, | ||
| deletes or loosens a test, or migrates persisted data. The PR description, |
There was a problem hiding this comment.
LOW — This paragraph says the rubric "says yes only for" the list that follows, but the list is narrower than the section it summarises in the same commit:
review/rubric.mdsays "adds a runtime dependency"; here it is "adds or major-bumps a dependency", which also covers a dev dependency the rubric deliberately leaves out;- the bullet on workflows covers "permissions, the secrets or tokens it holds, what it publishes, or who or what can merge" — the last clause is dropped here, which is the clause that decides whether a merge-gating CI change counts;
- the seventh criterion — "does something the repository's own review guidelines explicitly say a person decides" — is not mentioned at all, so a consumer reading only the README would not know its own guidelines can force the answer.
An "only for" list is a claim a reader will act on when deciding whether to configure human-review-paths, so it is worth either completing it or making it explicitly a summary.
| - size, or the number of files touched; | ||
| - a refactor, rename, or move that keeps behaviour; | ||
| - new or extended tests; | ||
| - comments, docs, or wording changes; |
There was a problem hiding this comment.
LOW — "wording changes" as a flat non-reason reads wider than the rest of the list intends. The criterion it replaces was "changes user-facing behaviour, wording, defaults, or whether a feature is available", and the biggest consumer of this rubric is truenas/webui, where wording is product copy: a PR relabelling a destructive action, or rewording an error a user acts on, now matches "comments, docs, or wording changes" and falls through every remaining criterion (it is not a default, not a feature removal, not an external contract) to an automatic APPROVE.
The other bullets in this list are all qualified so they cannot swallow a user-visible change; this one is not. Something like "comments, docs, or wording that a user does not see" would keep the intent — stop flagging every comment and README tweak — without covering product copy.
|
Two findings: two LOW. A human must look — this diff changes when the workflow submits an automatic APPROVE. Both findings are on their lines. Neither blocks: the check fails at MEDIUM and above.
Why a person should look
That is the change itself, not a defect in it: the PR argues the old criteria fired on nearly every PR, and the specific cases it names (a deferral mentioned only in a description, "touches CI") were genuinely not judgement calls. The narrowing looks right in direction. It is the kind of loosening whose blast radius someone should agree to deliberately, which is what this flag is for. The rest holds up: the
|
There was a problem hiding this comment.
Needs a human review. 2 finding(s): 2 LOW.
Nothing blocks, but this change is one a person should decide on:
- review/rubric.md:108 narrows the flag submit-verdict.mjs:274 checks before APPROVE, changing what can satisfy a required review in every consumer at @master
- review/rubric.md:139 makes a CI change not touching permissions or publishing a non-reason, dropping "touches CI" for a repo whose product is workflows
…summary (#16) ## Summary Follow-up to #15, which merged before this landed. Two LOW findings from its own review: - `review/rubric.md`: "comments, docs, or wording changes" as a blanket non-reason would auto-approve product copy in `truenas/webui` (a relabelled destructive action, a reworded error). Now "wording a user does not see". - `README.md`: the summary of the section claimed to be the full list but omitted the "what can merge" clause, the runtime-only qualifier on dependencies, and the repo-guidelines criterion. It now carries all three and says the rubric is the full list.
Summary
The Does this need a human? section of
review/rubric.mdwas asking for a person on nearly every PR. Three of its criteria covered most changes ("touches CI", "changes user-facing behaviour", "a choice the PR description presents as a decision"), and the last one fired on work a description merely deferred, e.g.:The section now:
false, and says what a wrongtruecosts;human-review-pathsinstead of the model guessing.README paragraph updated to match. No code changes.
No Jira ticket on hand for the title; add one so
PR ticketpasses.