Skip to content

fix(commands): stop /gh-issue-troubleshoot picking suites by entry count - #37610

Open
fabrizzio-dotCMS wants to merge 1 commit into
mainfrom
issue-37576-slash-command-suite-rule
Open

fabrizzio-dotCMS wants to merge 1 commit into
mainfrom
issue-37576-slash-command-suite-rule

Conversation

@fabrizzio-dotCMS

Copy link
Copy Markdown
Member

What

Follow-up to #37590, auditing .claude/commands/ — the other half of Scout's slice that no milestone in M0–M3 has read.

A slash command was inventing the suite-selection rule

/gh-issue-troubleshoot writes fixes for GitHub issues, and step 6c registers any new integration test it creates. It said:

Read each suite file to count @SuiteClasses entries. Add the new class to the suite with the fewest entries to keep load balanced.

That rule is not real. #37583 established the actual one the day before: pick whichever MainSuite* has the shortest current CI runtime, measured across at least three recent PR runs, because the suites run in parallel and CI balances them on wall-clock time.

Entry count is not a proxy for runtime. MainSuite2b currently holds 302 entries against MainSuite2a's 52 — following the old text, every new JUnit 4 test would have been funnelled into whichever suite happened to be shortest on paper, regardless of what CI actually does.

Step 6c now defers to docs/testing/INTEGRATION_TESTS.md → "Registering Tests in a MainSuite (CI gate)" and reads the current suite set from .github/test-matrix.yml, rather than restating a list that #37583 explicitly says isn't fixed. Same principle as the .cursor/rules/ fix in #37590: state the consequence, defer the mechanics to the doc that owns them.

The validation script missed a whole suite, and passed by luck

scripts/validate-integration-test-registration.sh (added in #37590) matched @SuiteClasses and @Suite.SuiteClasses, but not @SelectClasses — the JUnit 5 form Junit5Suite1 uses.

Its 15 classes were being counted anyway, by accident: a javadoc line in that file mentioning {@code @SuiteClasses} opened the extraction block early and the real @SelectClasses array fell inside it. The totals were right for the wrong reason, and would have broken silently the moment anyone reworded that comment.

Javadoc lines are now skipped, all three annotation forms are matched, and the per-suite counts are what you'd expect by hand:

MainSuite1a  77    MainSuite3a             64
MainSuite1b  87    Junit5Suite1            15
MainSuite2a  52    OpenSearchUpgradeSuite  18
MainSuite2b 302

Totals against current main: 608 registered, 672 concrete test classes, 87 never run in CI, 0 of the 28 added in the last 60 days unregistered — unchanged from #37590 apart from one class removed upstream since.

The rest of the directory checks out

The four dot-issue-manage wrappers (create-issue, find-issues, query-issue, update-issue) point at a skill that exists at .claude/skills/dot-issue-manage. triage-pr-review.md and check-release-rollback.md make no claims verifiable against source.

Testing

Documentation and tooling only — no production code. The script was run against main at 17fb9cd; per-suite extraction counts were verified by hand against each suite file, and the @SelectClasses / javadoc interaction reproduced before and after the fix.

Refs #37576, #37577

🤖 Generated with Claude Code

Audit of .claude/commands/ (the other never-reviewed half of Scout's slice)
against real source.

Step 6c told the model to register a new integration test in the MainSuite with
the fewest @SuiteClasses entries. That rule is not real. #37583 established the
actual one: pick whichever MainSuite* has the shortest current CI runtime, since
the suites run in parallel and CI balances on wall-clock time. Entry count is not
a proxy for it — MainSuite2b holds 302 entries against MainSuite2a's 52. The step
now defers to docs/testing/INTEGRATION_TESTS.md and reads the current suite set
from .github/test-matrix.yml rather than a hardcoded list.

Also fixes the suite-class extraction in the validation script. It matched
@SuiteClasses and @Suite.SuiteClasses but not @SelectClasses, which Junit5Suite1
uses. That suite's 15 classes were being captured only by accident: a javadoc
line mentioning {@code @SuiteClasses} opened the block early. Javadoc lines are
now skipped and all three annotation forms matched. Totals are unchanged (609
registered, 0 of 28 in the window) — the accidental match happened to produce the
right answer, but would have broken the moment that comment was reworded.

The other six commands check out: the four dot-issue-manage wrappers point at a
skill that exists (.claude/skills/dot-issue-manage), and triage-pr-review.md
makes no verifiable source claims.

Refs #37576, #37577

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Claude finished @fabrizzio-dotCMS's task in 1m 17s —— View job


Review

I read both changed files and cross-checked the claims against the actual suite sources.

Verification against source:

  • Junit5Suite1.java:26 is indeed a javadoc line ( * ... {@code @SuiteClasses} ...) and the real annotation is @SelectClasses({ at line 34 — confirming the "old awk opened the block early on the javadoc mention and swept up @SelectClasses entries by accident" story.
  • All CI suite annotations sit on a single line with ({ attached: @SuiteClasses({ (MainSuite1a/1b/2a/2b, OpenSearchUpgradeSuite), @Suite.SuiteClasses({ (MainSuite3a), @SelectClasses({ (Junit5Suite1). So the new open-regex requirement [[:space:]]*\( is satisfied for every current suite — no suite is accidentally excluded.

New Issues

No issues found.

Notes (non-blocking, no action needed):

  • scripts/validate-integration-test-registration.sh:63 — the awk change is sound. Ordering is correct: inblock{print} runs before the /\}\)/{inblock=0} close check, so a single-line @SuiteClasses({A.class}) form is still printed before the block closes. The javadoc skip /^[[:space:]]*\*/{next} only drops lines beginning with *, which never carry a .class entry, so it can't under-count real registrations.
  • The added \( requirement makes the open-match stricter than before (annotation must be @...Classes( on one line). If a future suite ever splits the annotation from its ( across lines, that suite would extract 0 classes — but the existing fail-loud guard at lines 69–72 turns that into a hard exit 1 rather than a silent miscount, which is the correct outcome. Worth a mental note for whoever adds a suite later.
  • .claude/commands/gh-issue-troubleshoot.md — the doc change correctly removes the invented "fewest @SuiteClasses entries" rule and defers to docs/testing/INTEGRATION_TESTS.md + .github/test-matrix.yml, consistent with docs(testing): fix MainSuite registration rule and audit INTEGRATION_TESTS.md #37583. Reads cleanly.

Documentation and tooling only — no production code, no security/transaction/replay surface. LGTM.

· issue-37576-slash-command-suite-rule

@github-actions github-actions Bot removed the Area : Documentation PR changes documentation files label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant