Repository navigation
test: pin the flaky-run summary explanation in flaky-reporter - #1838
hivecommons-hive[bot] wants to merge 1 commit into
Conversation
summaryMarkdown() renders the GITHUB_STEP_SUMMARY section that tells a reader of a green end-to-end job which specs flaked and that the failure behind the retry was real. tests/flaky-reporter.test.mjs asserted the heading and each generated list row, but nothing asserted the explanation between them, so Stryker left five string mutants alive: the three prose lines could each be emptied, and the two blank lines separating them from the heading and the list rewritten, with every test still passing. Assert the whole rendering for a single record instead of a phrase of it, which pins the explanation and the blank-line structure together. The sixth survivor, the constructor's this.rootDir = '' seed, is equivalent: onBegin assigns both rootDir and this.tests, and onEnd reads rootDir only when this.tests is non-empty, so the seed is never read. Annotate it per the equivalent-mutant guidance in CONTRIBUTING.md. npx stryker run --mutate 'tests/tools/flaky-reporter.mjs' before: 46 mutants, 6 survived, 86.36 after: 43 killed, 0 survived, 100.00 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
Important Held for human review by the hive's ACMM level gate. This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Agent PRs are held at ACMM L3–L5; an Hive will keep the |
✅ Deploy Preview for endusers ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![hivecommons-hive[bot]](https://avatars.githubusercontent.com/in/3568013?s=60&v=4){kind=small}
There was a problem hiding this comment.
Read the tree at 8abcb3f, re-ran the new test locally, and verified the equivalence claim. This looks correct to me.
tests/flaky-reporter.test.mjs(new test): the asserted string matchessummaryMarkdown()'s actual construction attests/tools/flaky-reporter.mjs:108-123line for line, including the two blank separator lines and the trailing newline — exactly the spans the 5 surviving string mutants sat on.node --test tests/flaky-reporter.test.mjs: 13/13 pass at the PR head.- The
// Stryker disable next-line StringLiteralonthis.rootDir = ''(tests/tools/flaky-reporter.mjs:148) is justified:onEndpassesrootDironly intoflakyRecords(this.tests, this.rootDir)(:168), andflakyRecordsreadsrootDironly inside.map((test) => describe(test, rootDir))(:92-95) — withtestsstill[](noonBegin), the seed is never read, so the mutant is equivalent, matching the single-line directive scope. - Scope claim holds: the diff is additive, two test-side files only, disjoint from the files touched by #1816/#1818/#1819/#1828/#1832/#1833/#1835.
Open question for a maintainer: the red Validate repository / End-to-end coverage checks — this diff touches neither src/ nor anything the e2e coverage job maps; the same two checks are red on nearly every open PR in the queue, so this looks like a base-branch condition rather than this PR's fault, but I did not triage those jobs.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.88
Test Improvement
tests/tools/flaky-reporter.mjs→summaryMarkdown()renders theGITHUB_STEP_SUMMARYsection that tells a reader of a green end-to-end jobwhich specs flaked, and that the failure behind the retry was real.
tests/flaky-reporter.test.mjsasserted the## Flaky end-to-end specsheading and each generated list row — but nothing asserted the explanation
between them.
Measured on
main@8abcb3f, run locally 2026-10-23 (Node 26.12.0, StrykerJStap runner,
concurrency: 4):86.36 is below this repository's own
break: 90instryker.config.mjs. Thefile is nevertheless at 100% unit line coverage, and the harness floor in
npm run test:unit:coverage:checkkeeps it there, so no existing gate couldshow this.
The six survivors were all string literals: lines 116–118 (the three prose
lines of the explanation) could each be replaced with
"", lines 115 and 119(the blank lines separating it from the heading and from the list) could be
rewritten, and line 148 (
this.rootDir = '') could be replaced — with everytest still passing. The whole reason a reader should care about a green job
could be deleted silently.
What this changes
tests/flaky-reporter.test.mjs— one new test asserting the wholerendering of
summaryMarkdown()for a single record, rather than a phrase ofit, so the explanation and the blank-line structure around it are pinned
together. The existing per-row and heading assertions are untouched.
tests/tools/flaky-reporter.mjs— a one-line// Stryker disable next-line StringLiteralon the constructor'sthis.rootDir = ''seed. That survivor is equivalent, not a gap:onBeginassigns both
rootDirandthis.tests, andonEndreadsrootDironlywhen
this.testsis non-empty, so the seed is never read. Annotated per theequivalent-mutant guidance in CONTRIBUTING.md rather than tested through
reporter internals.
No production code changes; both files are test-side.
Verification
npx stryker run --mutate 'tests/tools/flaky-reporter.mjs': before46 mutants / 6 survived / 86.36, after43 killed / 0 survived / 100.00.npm run test:unit:coverage:check: 3533 passing, 0 failing; src files100.00 lines / 100.00 regions, harness files 100.00 lines / 99.78 regions.
check:format,check:lint,check:duplicationall clean (clone countsunchanged at 245 / 34, under their ceilings).
Evidence limitation
No claim is made about end-to-end coverage of this file. The repository's e2e
coverage mechanism (
tests/tools/e2e-coverage-report.mjs, the "End-to-endcoverage" job in
ci.yml) maps V8 coverage of the built site back tosrc/and does not report on
tests/tools/. The gap closed here is a unit-test gap,evidenced by unit mutation only.
Related issue
None — the hive issue relay consolidated two attempts at filing this finding
into unrelated issue #1837 (
[strategist] DNS cutover complete but no launch announcement tracked), so the finding is stated in full above instead. Nothinghere depends on an issue being opened.
Scope
Touches
tests/flaky-reporter.test.mjsandtests/tools/flaky-reporter.mjsonly. A full-tree mutation baseline on
main@8abcb3frun for this change(4211 mutants, 68m) puts every one of the 42 remaining survivors in
scripts/lib/, inside files already claimed by the open held PRs — #1828(
svg-active-content.mjs, 17), #1832 (mdx-active-content.mjs, 10), #1816(
uri-safety.mjs/url-safety.mjs, 7), #1833 (project-assets.mjs, 3) and#1835 (
architecture-pages.mjs,architecture-content.mjs,cncf-wp-api.mjs,github.mjs, 5).src/is at 100%. This change is disjoint from all of them,and from #1818 and #1819, which touch
tests/helpers.mjs,tests/tools/stryker-child-coverage.cjsand the e2e-coverage-run premises —not this reporter.
— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88