redgate: point hooks at the handlers that exist; sort runs in C locale - #136
Conversation
Two fixes lifted out of #115 so they do not wait on 76k lines. hooks.json referenced ${CLAUDE_PLUGIN_ROOT}/hooks-handlers/<h>.sh, but both handlers live under hooks/hooks-handlers/, and ${CLAUDE_PLUGIN_ROOT} is the plugin dir on an installed plugin (agent-compiler's own hooks.json resolves the same way). No plugins/redgate/hooks-handlers/ directory exists. So on every installed copy the PreToolUse write guard — the hook that stops a write reaching a ratified run's contract while its round is in MIDDLE — and the SessionStart announcer silently never ran. The cheap tier was green the whole time because nothing resolved handler paths against the filesystem. criteria-index.sh sorted run dirs with the host locale. Its own header promises deterministic output and --check compares against it, so an en_US host can order slugs differently from CI and report drift that is not there. LC_ALL=C pins it. New cheap-tier section 2b resolves every ${CLAUDE_PLUGIN_ROOT}/<rel> in every plugins/*/hooks/hooks.json to a file under plugins/<p>/<rel>, and fails closed if the walk finds nothing. It is resolved against the filesystem, not pattern-matched: voice's hooks.json uses the identical-looking hooks-handlers/ path and is CORRECT, because voice keeps its handler directly under the plugin root. Verified: cheap tier 1295 passed / 0 failed (1290 on main + 5 handler paths). Against main's pre-fix hooks.json the new section reports both redgate hooks as missing (1293/2). Two further mutations — an agent-compiler handler renamed away, and the walk pointed at an empty glob — each go red. Not verified here: the locale ordering difference. No en_US locale is installed in this container; LC_ALL=en_US.UTF-8 falls back to C and sorts identically, so the one-line fix rests on sort(1) semantics and #115's report of a red cheap tier on en_US hosts, not on a reproduction. The deep tier's path filter matches criteria-index.sh, but redgate ships no pier pack, so that check will report green because its leg is skipped, not because it ran. No SKILL.md, command, or skill references/ touched; demonstration discipline does not apply. No eval tier, job, or pack added, removed, or re-scoped, so docs/testing.md is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟡 Changes recommended
The handler-path validation needs canonical containment checks and reliable walker error propagation.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR fixes Redgate hook paths, makes criteria indexing locale-independent, and adds cheap-tier handler validation.
Changes:
- Corrects both Redgate hook handler paths.
- Forces criteria sorting with
LC_ALL=C. - Adds filesystem-based hook path checks.
File summaries
| File | Summary |
|---|---|
plugins/redgate/skills/criteria-contract/scripts/criteria-index.sh |
Makes run ordering deterministic across locales. |
plugins/redgate/hooks/hooks.json |
Corrects handler locations. |
evals/cheap/run.sh |
Validates resolved hook handler paths. |
Review details
Suppressed comments (1)
evals/cheap/run.sh:82
- The Python walker runs in process substitution, so its non-zero exit status is not propagated by
set -o pipefailand the loop only checks whether it saw any output. If an earlier plugin emits valid paths and a later syntactically valid but structurally malformedhooks.jsonraises inside this walker,hook_paths_seenremains nonzero and section 2b reports no failure instead of failing closed. Capture and check the walker's exit status (or run it as a normal command) before accepting the results.
)
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9145b8edd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… is nothing-to-check Two corrections to the section added in a9145b8, both found by running it against something other than the tree it was written on. 1. The counterfeit corpus's synthetic marketplace ships no hooks.json at all, and the section's "zero paths resolved" fail-closed turned that root red: the corpus's own calibration guard reported "baseline plugin is NOT green — corpus is miscalibrated, every rejection below is meaningless" and the required counterfeit tier failed on CI. Reproduced locally (24 passed / 1 failed) before touching anything. The section now counts hooks.json FILES separately from extracted PATHS: no files is a legitimate "nothing to check", the same PASS shape every other section uses for an absent surface; files present but zero paths extracted still fails closed, because that is the walk breaking, not the plugins being clean. 2. The resolver checked isfile() only, while the section's name promises containment. `${CLAUDE_PLUGIN_ROOT}/../shared/x.sh`, or a symlink out of the plugin, would report OK when the target exists even though the installed command resolves outside the plugin root. Copilot's finding on #136. Both paths are now realpath'd and the target must share the plugin dir as its commonpath — the same idiom the relative-link resolver further down already uses. Verified, in order: counterfeit harness on the fixed file: 25 passed / 0 failed, baseline green, all 18 counterfeits rejected by their expected gate (was 24 / 1). cheap tier on the repo: 1295 passed / 0 failed, unchanged. Five mutations against the final section, each restored after: main's pre-fix redgate hooks.json -> 2 FAIL (1293 / 2) agent-compiler handler renamed away -> 1 FAIL (1294 / 1) ../ escape to a REAL file in redgate -> 1 FAIL (1294 / 1) isfile alone said OK extraction regex broken, 3 files kept -> 1 FAIL "3 file(s) present but the walk extracted ZERO handler paths" glob pointed at a root with no hooks -> PASS "nothing to check" (1291 / 0) The regex mutation's first attempt did not apply (a sed pattern that missed the heredoc's escaping left the file untouched and the tier at 1295 / 0); that was a null test, not evidence, and is recorded here so the second, applied attempt is the one that counts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
|
Pushed Counterfeit tier was red, and it was this PR's fault. The corpus's synthetic marketplace ships no Copilot's containment finding was correct and is fixed — both paths are Five mutations against the final section, cheap tier 1295 / 0 unchanged. One null result recorded honestly: the regex-break mutation's first attempt didn't apply (a Two things this run revealed that are outside this diff:
Generated by Claude Code |
Codex's finding on #136, confirmed by reproduction before touching anything: the handler-path scanner ran in process substitution, so its exit status was discarded. A manifest that is valid JSON but not the shape the walk expects — `"command": null` — raised TypeError, the remaining manifests were never scanned, and because an EARLIER manifest had already emitted records the zero-paths guard stayed quiet. On the unmodified file, nulling voice's command made the `voice hooks.json SessionStart` line simply vanish; section 2b reported no failure and the required tier exited 0 (1294 / 0), while stderr carried the traceback nobody reads. Nulling agent-compiler's instead went red only incidentally, via the zero-paths guard, and misreported "1 file(s)" when three exist — the scan had been silently truncated. Three layers now, each verified on its own: * Every shape the walk touches is checked, not assumed. Nonsense in a valid-JSON manifest emits a MALFORMED record naming the plugin and event instead of raising, and the walk continues to the next manifest. * The scanner writes to a temp file and its exit status is captured; a nonzero one is a recorded FAIL saying an unknown number of manifests were never checked. * The temp file itself fails closed if it cannot be created, and the "nothing to check" PASS moved inside the scanner-ran branch, so no silent-scanner path can produce that green line. Verified against the final file: null command, last manifest -> FAIL "MALFORMED hook entry (a hook has no string "command")" 1294 / 1, exit 1 null command, first manifest -> same FAIL, and redgate + voice still scanned after it 1294 / 1 crash on every manifest -> FAIL "scanner exited 1" + zero-paths FAIL crash on the LAST manifest -> FAIL "scanner exited 1" 1294 / 1 (the exact earlier-records shape that used to pass) "hooks" not an object / event not a list / entry's hooks not a list -> one MALFORMED FAIL each TMPDIR unwritable -> FAIL, and no "nothing to check" green The five prior mutations unchanged: main's pre-fix redgate (2 FAIL), handler renamed (1), ../ escape to a real file (1), regex broken (now reports "3 file(s)", not 1 — the walk no longer truncates), no-hooks root (PASS). cheap tier 1295 / 0 unchanged; counterfeit harness 25 / 0, baseline green. The pre-fix reproduction and the full mutation sweep were run by a delegated agent; the fixed-file cases above, both roots, and the hooks.json files being byte-identical to HEAD were re-verified independently before this commit. Survey, not fixed here: sections 10 (per-plugin safety-pack discovery) and 12 (install-smoke coverage) feed `while read` from the same `< <(python3 …)` shape and neither asserts the enumerated count against marketplace.json, so a scanner crash after N plugins would silently run only N packs and stay green. Section 10 is the one this repo's safety story rests on. Follow-up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
The merge brought in #136, which added checks. The figure this branch introduced (~19s / 1290 checks, measured 2026-09-14) was already stale against its own instruction to re-measure rather than trust it. Measured on this checkout at 2026-09-18: 13.6s, 1296 checks, 25 plugins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb
Two fixes lifted out of #115 so they do not wait on 76k lines. Three changed lines, plus a cheap-tier section that would have caught the first one.
The shipped defect
plugins/redgate/hooks/hooks.jsonreferenced${CLAUDE_PLUGIN_ROOT}/hooks-handlers/<handler>.sh. Both handlers live underhooks/hooks-handlers/, and${CLAUDE_PLUGIN_ROOT}is the plugin directory on an installed plugin —agent-compiler's ownhooks.jsonresolves the same way. There is noplugins/redgate/hooks-handlers/directory.So on every installed copy of redgate, the
PreToolUsewrite guard — the hook that stops a write reaching a ratified run's contract while its round is inMIDDLE— and theSessionStartannouncer silently never ran. The cheap tier stayed green throughout, because nothing resolved handler paths against the filesystem.The locale sort
criteria-index.shsorted run directories with the host locale. Its own header promises deterministic output and--checkcompares against it, so anen_UShost can order slugs differently from CI and report drift that is not there.The guard
New cheap-tier section 2b resolves every
${CLAUDE_PLUGIN_ROOT}/<rel>in everyplugins/*/hooks/hooks.jsonto a file underplugins/<p>/<rel>, onePASS/FAILper handler, and fails closed if the walk finds nothing. It is resolved against the filesystem rather than pattern-matched on purpose:voice'shooks.jsonuses the identical-lookinghooks-handlers/path and is correct, because voice keeps its handler directly under the plugin root. A pattern rule would have flagged the healthy plugin and could not have told the two apart.Verification
main+ 5 handler paths)main's pre-fixhooks.jsonNot verified here, stated rather than implied:
en_USlocale is installed in this container;LC_ALL=en_US.UTF-8falls back toCand sorts identically. The one-line fix rests onsort(1)semantics and feat(evals): marketplace-wide agentic test framework (T01–T52) with red-team lane #115's report of a red cheap tier onen_UShosts, not on a reproduction.criteria-index.sh(plugins/*/skills/**/scripts/**), but redgate ships no pier pack, sodeep tier (pier)will report green because its leg is skipped, not because anything ran. That is the existing gate design, named here so a green is not read as proof.No
SKILL.md, command, or skillreferences/touched, so demonstration discipline does not apply. No eval tier, workflow job, or eval pack added, removed, renamed, or re-scoped, sodocs/testing.mdis unchanged and section 20 stays green.Independent of the OpenRouter credit outage blocking #131 and #133 — nothing here calls a model.
🤖 Generated with Claude Code
https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Generated by Claude Code