Follow-up: strengthen test contracts, CI enforcement, release delivery, and quality triage #554

Is your feature request related to a problem? Please describe.

A review of the 100 most recently created issues and PRs identified recurring gaps in test effectiveness, CI enforcement, release delivery, and automated triage.

Review snapshot: October 21, 2026, against main at e3119adfdf11b3b9491f7a3ed807f48687e18733. The sample is 100 combined records, #454 through #553, created October 12-21 UTC: 78 PRs and 22 issues. Of these, 56 PRs changed only test files and 20 issues were quality findings.

This is a heavily automated-quality-focused sample, not evidence of 100 distinct production defects. The historical assertion defects cited below have been fixed. This issue tracks remaining test/process gaps and decisions, not newly reproduced production failures.

Describe the solution you'd like

P1: Make CI enforcement explicit

At review time, main branch protection had empty required-status-check lists, and the effective organization/repository rulesets added no required checks. The jobs run, but GitHub does not require them to pass before merging. The e2e-coverage job was intentionally introduced as reporting-only in #488; .github/workflows/test.yml:65-79 still documents that policy.

• Require build-test for normal merges, with an explicitly documented exception/bypass policy.
• Decide whether e2e-coverage should also be required; record the rationale and keep workflow documentation consistent with the decision.

This is a repository-settings/policy task, not a request to bypass existing protections or silently change the original reporting-only decision.

P1: Deliver merged features to release users

The latest stable release at review time was v3.0.1, published September 23, 2026. The floating v3 tag resolved to the same commit, 187c5e3cd95a329c43448e1bdb3b1f5249232e44; main was 218 commits ahead.

OWNERS groups from #486 and delayed reviewer assignment from #530 are documented on main but are not available through those release tags. Documentation examples still use @v3.0.1.

• Prepare a release plan with an appropriate version, compatibility review, and curated upgrade notes.
• Clearly mark newer functionality as unreleased until it is published.
• Verify the release candidate's committed action bundle and reusable workflow, then align documentation/version references using the existing release process.

Do not treat an issue's implementation being merged as proof that a supported release contains it.

P1: Strengthen assertions and the test harness

Near-100% coverage repeatedly failed to distinguish correct behavior from behavior-changing mutations:

• test(remove): assert every DELETE the /remove happy paths issue; enable test/expect-expect #512 repaired assertion-free label-removal tests.
• test(label-sync): pin the case-insensitive name and color folds, the null-description fold, the dry-run counts and per_page #531 and test(pullRequestOwners): omit draft/state/labels for real, pin the rename dedupe and the repository-keyed pull and tip memos #533 repaired fixtures that did not actually represent the cases their test names described.
• test(tide): pin the merge_queue:off no-GraphQL rule, the draft-without-field refusal, and the fork-403 path boundaries #518 showed that an unexpected GraphQL request could be caught by application fallback handling without failing a test.
• [quality] pullRequestOwners.ts: the draft/labels defaults test is vacuous, and the file dedupe and repository-keyed memos survive deletion under both suites #532 documented mutations that survived both unit tests and the rebuilt bundle suite.

The specific examples are resolved, but no repeatable mutation runner is configured in the repository. __tests__/testUtils.ts:16-22 rejects unhandled requests without a general end-of-test assertion that one occurred. __tests__/bundle/fakeGithub.ts:94-100 records unmatched requests but answers them with a normal-looking 404, leaving each test responsible for detecting unexpected traffic.

• Establish bounded, repeatable mutation checks for selected high-risk behavior, with reproducible commands and reviewed equivalent-mutant exclusions. A repository-wide mutation gate is not required.
• Record unexpected API traffic and assert on it outside application error handling, with explicit allowances for intentional probes.
• Require affected tests to assert observable effects: exact writes, prohibited writes, meaningful outcomes, and fixture shapes where absent/null/case/order differences are the point.

When a unit-only mutation survives, check whether the rebuilt bundle already detects it before declaring a repository-wide gap or adding duplicate coverage.

P2: Add cross-request contracts for the new features

The following are recommended regression scenarios, not claims that the implementation currently fails them:

• OWNERS groups: serve a paginated team with the relevant member only on page two; exercise /approve and automatic reviewer selection, not just /lgtm. Assert that page-two membership changes the actual approval/reviewer outcome.
• OWNERS pagination failure: fail a subsequent membership page and assert the warning and fail-closed behavior rather than using incomplete membership.
• Delayed-review timeline: put the latest ready_for_review event on a subsequent page and assert that it determines whether assignment is due.
• Delayed-review boundaries: cover both exact shifted-window boundaries and timestamps immediately outside them.
• Successive sweeps: run again against state updated by the first run or a manual reviewer assignment and assert no duplicate or unwanted reviewer requests.
• Scheduling outage recovery: explicitly document and test the chosen recovery expectation when missed runs exceed the lookback window. The current implementation intentionally excludes sufficiently old PRs; do not silently change this into unlimited catch-up.

Current evidence:

• __tests__/bundle/ownersGroups.test.ts contains four single-page /lgtm scenarios; __tests__/utils/owners.test.ts stubs paginate.
• __tests__/cronJobTest/sweep.test.ts:497-572 and __tests__/bundle/blunderbussDelay.test.ts serve single-page timelines.
• The shared FakeResponse in __tests__/bundle/fakeGithub.ts:11-14 has no response-header support. Add the small capability needed to serve real Link headers before writing bundle pagination scenarios. Existing manually paged pull-list tests do not establish these new endpoints' pagination contracts.

P2: Judge tests by behavior, not percentage changes

#465 was closed because its mid-retry gate-change scenario added no line/branch coverage. That alone does not establish behavioral redundancy. The current direct unit case at __tests__/plugins/tide.test.ts:134-140 supplies an always-false callback; it does not establish the complete caller's response to a gate changing during a retry.

• Reconsider a focused integration scenario where mergeability is initially unknown and LGTM disappears on the next read. Assert that additional retries stop and no merge occurs, or identify an existing scenario that proves the same contract.
• Use a failing regression/mutation or a concrete behavioral contract to justify adding/removing tests, rather than coverage percentage alone.
• Record the coverage-floor decision at the existing [quality] Re-evaluate the e2e branch coverage floor after pending coverage work lands #551; do not create another threshold tracker.
• Resolve the existing auth follow-up through test(bundle): drive getErrorDetails' status-less and message-less object folds (auth.ts:15-17) through dist/index.js #553 or an explicitly linked replacement, rather than duplicating it.

Keep unit and bundle coverage reports separate. A whole-percentage global floor does not catch every single-branch regression. The known negative-count V8 reporting artifact is already documented and linked upstream by #552; do not add ignores, remove useful behavior, or demand 100% to hide it. The replacement of #510 by #528 also illustrates why claims of unreachable behavior need evidence.

P2: Prevent unrelated findings from being consolidated again

#466 accumulated 48 comments and required reopening after partial closure. Findings on #521 and #523 were also folded together using broad shared filenames such as test.ts or dist/index.js. The recent cleanup repaired the records; it did not verify or change the external Hive routing implementation.

• Have the Hive operator review deduplication/routing so concrete behavior and source symbols determine ownership, not shared generated files or basenames.
• Require a canonical owner and explicit acceptance criteria for each independent finding.
• Batch genuinely cohesive work, without treating unrelated findings as duplicates or opening one tracker per uncovered line.

Describe alternatives you've considered

Simply raising coverage thresholds, pursuing 100% coverage, or opening more single-branch test PRs would not address missing assertions, unenforced checks, unreleased features, or incorrect consolidation. Conversely, requiring every behavior to have both a unit and bundle test would create unnecessary duplication. Prefer a small set of behavior-driven scenarios and targeted mutation checks.

Additional context

This is an intentional umbrella follow-up, not a replacement for existing scoped issues. Each section may be delegated to a focused child issue/PR, which should reference this issue without automatically closing it.

Close this tracker only when every section has either shipped or has an explicit recorded disposition (including a linked canonical owner for deferred work). A single test PR must not close the whole tracker. Repository settings, release policy, and external Hive routing require maintainer/operator decisions; creating this issue does not authorize those changes.

Existing canonical follow-ups:

• [quality] Re-evaluate the e2e branch coverage floor after pending coverage work lands #551: e2e branch-floor decision.
• test(bundle): drive getErrorDetails' status-less and message-less object folds (auth.ts:15-17) through dist/index.js #553: auth object-rejection bundle scenarios, open at review time.
• 🐝 Hive Advisory Report #170: living advisory register; keep independently owned findings separate.

The review used issue/PR descriptions and dispositions, changed-file inventories, current source and tests, live branch-protection/ruleset settings, and release/tag comparisons. All counts and settings above describe the recorded review snapshot and should be refreshed before implementation.