Skip to content

redgate: point hooks at the handlers that exist; sort runs in C locale - #136

Merged
JRichlen merged 3 commits into
mainfrom
claude/redgate-hooks-path-fix
Sep 15, 2026
Merged

JRichlen merged 3 commits into
mainfrom
claude/redgate-hooks-path-fix

Conversation

@JRichlen

Copy link
Copy Markdown
Owner

Two fixes lifted out of #115 so they do not wait on 76k lines. Three changed lines, plus a cheap-tier section that would have caught the first one.

The shipped defect

plugins/redgate/hooks/hooks.json referenced ${CLAUDE_PLUGIN_ROOT}/hooks-handlers/<handler>.sh. Both handlers live under hooks/hooks-handlers/, and ${CLAUDE_PLUGIN_ROOT} is the plugin directory on an installed plugin — agent-compiler's own hooks.json resolves the same way. There is no plugins/redgate/hooks-handlers/ directory.

So on every installed copy of redgate, the PreToolUse write guard — the hook that stops a write reaching a ratified run's contract while its round is in MIDDLE — and the SessionStart announcer silently never ran. The cheap tier stayed green throughout, because nothing resolved handler paths against the filesystem.

-  "command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks-handlers/guard-redgate-paths.sh\""
+  "command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/hooks-handlers/guard-redgate-paths.sh\""
-  "command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks-handlers/session-start.sh\""
+  "command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/hooks-handlers/session-start.sh\""

The locale sort

criteria-index.sh sorted run directories with the host locale. Its own header promises deterministic output and --check compares against it, so an en_US host can order slugs differently from CI and report drift that is not there.

-  for run in $(ls -d "$RG"/*/ 2>/dev/null | sort); do
+  for run in $(ls -d "$RG"/*/ 2>/dev/null | LC_ALL=C sort); do

The guard

New cheap-tier section 2b resolves every ${CLAUDE_PLUGIN_ROOT}/<rel> in every plugins/*/hooks/hooks.json to a file under plugins/<p>/<rel>, one PASS/FAIL per handler, and fails closed if the walk finds nothing. It is resolved against the filesystem rather than pattern-matched on purpose: voice's hooks.json uses the identical-looking hooks-handlers/ path and is correct, because voice keeps its handler directly under the plugin root. A pattern rule would have flagged the healthy plugin and could not have told the two apart.

Verification

Check Result
cheap tier, this branch 1295 passed, 0 failed (1290 on main + 5 handler paths)
section 2b against main's pre-fix hooks.json both redgate hooks reported missing — 1293 / 2 failed
mutation: agent-compiler handler renamed away red (1294 / 1)
mutation: walk pointed at an empty glob red — "resolved ZERO handler paths" (1290 / 1)
all five current handler paths resolve: agent-compiler ×2, redgate ×2, voice ×1

Not verified here, stated rather than implied:

  • The locale ordering difference. No en_US locale is installed in this container; LC_ALL=en_US.UTF-8 falls back to C and sorts identically. The one-line fix rests on sort(1) semantics and feat(evals): marketplace-wide agentic test framework (T01–T52) with red-team lane #115's report of a red cheap tier on en_US hosts, not on a reproduction.
  • The deep tier. Its path filter matches criteria-index.sh (plugins/*/skills/**/scripts/**), but redgate ships no pier pack, so deep tier (pier) will report green because its leg is skipped, not because anything ran. That is the existing gate design, named here so a green is not read as proof.

No SKILL.md, command, or skill references/ touched, so demonstration discipline does not apply. No eval tier, workflow job, or eval pack added, removed, renamed, or re-scoped, so docs/testing.md is unchanged and section 20 stays green.

Independent of the OpenRouter credit outage blocking #131 and #133 — nothing here calls a model.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd


Generated by Claude Code

Two fixes lifted out of #115 so they do not wait on 76k lines.

hooks.json referenced ${CLAUDE_PLUGIN_ROOT}/hooks-handlers/<h>.sh, but both
handlers live under hooks/hooks-handlers/, and ${CLAUDE_PLUGIN_ROOT} is the
plugin dir on an installed plugin (agent-compiler's own hooks.json resolves
the same way). No plugins/redgate/hooks-handlers/ directory exists. So on
every installed copy the PreToolUse write guard — the hook that stops a write
reaching a ratified run's contract while its round is in MIDDLE — and the
SessionStart announcer silently never ran. The cheap tier was green the whole
time because nothing resolved handler paths against the filesystem.

criteria-index.sh sorted run dirs with the host locale. Its own header
promises deterministic output and --check compares against it, so an en_US
host can order slugs differently from CI and report drift that is not there.
LC_ALL=C pins it.

New cheap-tier section 2b resolves every ${CLAUDE_PLUGIN_ROOT}/<rel> in every
plugins/*/hooks/hooks.json to a file under plugins/<p>/<rel>, and fails
closed if the walk finds nothing. It is resolved against the filesystem, not
pattern-matched: voice's hooks.json uses the identical-looking
hooks-handlers/ path and is CORRECT, because voice keeps its handler directly
under the plugin root.

Verified: cheap tier 1295 passed / 0 failed (1290 on main + 5 handler paths).
Against main's pre-fix hooks.json the new section reports both redgate hooks
as missing (1293/2). Two further mutations — an agent-compiler handler
renamed away, and the walk pointed at an empty glob — each go red.

Not verified here: the locale ordering difference. No en_US locale is
installed in this container; LC_ALL=en_US.UTF-8 falls back to C and sorts
identically, so the one-line fix rests on sort(1) semantics and #115's report
of a red cheap tier on en_US hosts, not on a reproduction. The deep tier's
path filter matches criteria-index.sh, but redgate ships no pier pack, so
that check will report green because its leg is skipped, not because it ran.

No SKILL.md, command, or skill references/ touched; demonstration discipline
does not apply. No eval tier, job, or pack added, removed, or re-scoped, so
docs/testing.md is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Copilot AI lite review requested due to automatic review settings September 15, 2026 02:14
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-15T02:20:52.997686Z a9145b8 PR opened
🔒 Security Review Completed 2026-09-15T02:32:38.805357Z a9145b8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The handler-path validation needs canonical containment checks and reliable walker error propagation.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR fixes Redgate hook paths, makes criteria indexing locale-independent, and adds cheap-tier handler validation.

Changes:

  • Corrects both Redgate hook handler paths.
  • Forces criteria sorting with LC_ALL=C.
  • Adds filesystem-based hook path checks.
File summaries
File Summary
plugins/redgate/skills/criteria-contract/scripts/criteria-index.sh Makes run ordering deterministic across locales.
plugins/redgate/hooks/hooks.json Corrects handler locations.
evals/cheap/run.sh Validates resolved hook handler paths.
Review details

Suppressed comments (1)

evals/cheap/run.sh:82

  • The Python walker runs in process substitution, so its non-zero exit status is not propagated by set -o pipefail and the loop only checks whether it saw any output. If an earlier plugin emits valid paths and a later syntactically valid but structurally malformed hooks.json raises inside this walker, hook_paths_seen remains nonzero and section 2b reports no failure instead of failing closed. Capture and check the walker's exit status (or run it as a normal command) before accepting the results.
)
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread evals/cheap/run.sh Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a9145b8edd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread evals/cheap/run.sh Outdated
… is nothing-to-check

Two corrections to the section added in a9145b8, both found by running it
against something other than the tree it was written on.

1. The counterfeit corpus's synthetic marketplace ships no hooks.json at all,
   and the section's "zero paths resolved" fail-closed turned that root red:
   the corpus's own calibration guard reported "baseline plugin is NOT green —
   corpus is miscalibrated, every rejection below is meaningless" and the
   required counterfeit tier failed on CI. Reproduced locally (24 passed / 1
   failed) before touching anything. The section now counts hooks.json FILES
   separately from extracted PATHS: no files is a legitimate "nothing to
   check", the same PASS shape every other section uses for an absent surface;
   files present but zero paths extracted still fails closed, because that is
   the walk breaking, not the plugins being clean.

2. The resolver checked isfile() only, while the section's name promises
   containment. `${CLAUDE_PLUGIN_ROOT}/../shared/x.sh`, or a symlink out of the
   plugin, would report OK when the target exists even though the installed
   command resolves outside the plugin root. Copilot's finding on #136. Both
   paths are now realpath'd and the target must share the plugin dir as its
   commonpath — the same idiom the relative-link resolver further down already
   uses.

Verified, in order:
  counterfeit harness on the fixed file: 25 passed / 0 failed, baseline green,
    all 18 counterfeits rejected by their expected gate (was 24 / 1).
  cheap tier on the repo: 1295 passed / 0 failed, unchanged.
  Five mutations against the final section, each restored after:
    main's pre-fix redgate hooks.json      -> 2 FAIL (1293 / 2)
    agent-compiler handler renamed away    -> 1 FAIL (1294 / 1)
    ../ escape to a REAL file in redgate   -> 1 FAIL (1294 / 1)  isfile alone said OK
    extraction regex broken, 3 files kept  -> 1 FAIL "3 file(s) present but the
                                              walk extracted ZERO handler paths"
    glob pointed at a root with no hooks   -> PASS "nothing to check" (1291 / 0)
  The regex mutation's first attempt did not apply (a sed pattern that missed
  the heredoc's escaping left the file untouched and the tier at 1295 / 0);
  that was a null test, not evidence, and is recorded here so the second,
  applied attempt is the one that counts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd

Copy link
Copy Markdown
Owner Author

Pushed 2af3101. Two corrections to section 2b, one of them mine to own.

Counterfeit tier was red, and it was this PR's fault. The corpus's synthetic marketplace ships no hooks.json at all, and the section's "zero paths resolved" fail-closed turned that root red — the corpus's own calibration guard said "baseline plugin is NOT green — corpus is miscalibrated, every rejection below is meaningless". Reproduced locally at 24 / 1 before touching anything. The section now counts hooks.json files separately from extracted paths: no files is a legitimate "nothing to check" (the same PASS shape every other section uses for an absent surface); files present but zero paths extracted still fails closed, because that is the walk breaking, not the plugins being clean. Counterfeit harness now 25 / 0, baseline green, all 18 counterfeits rejected by their expected gate.

Copilot's containment finding was correct and is fixed — both paths are realpath'd and the target must share the plugin dir as its commonpath, the same idiom the link resolver further down already uses. The ../ case is now a mutation in the verification set: pointed at a real file in redgate from agent-compiler's hooks.json, isfile alone says OK and the section correctly fails.

Five mutations against the final section, cheap tier 1295 / 0 unchanged. One null result recorded honestly: the regex-break mutation's first attempt didn't apply (a sed pattern missed the heredoc's escaping and left the file untouched at 1295 / 0) — that was no evidence at all, so it was redone with an applied edit, which fires "3 file(s) present but the walk extracted ZERO handler paths".

Two things this run revealed that are outside this diff:

  • The behavioral tier — promptfoo (redgate) leg ran for real (by design — plugins/<p>/hooks/** is behavior surface) and passed 9 / 9 with 0 errors on the live subject. That is the first paid-tier evidence in six days that the OpenRouter account is funded again; evals: confirm the SUBJECT model resolves, and repin it to qwen/qwen3.8-flash #131 and Portal/shunt delegation research, and three skills it sharpens #133 are no longer blocked on credit.
  • That same leg logs evals/paid/capture-example.sh: Permission denied — the script isn't executable, so the gallery capture step is a silent no-op in CI (guarded by || true; the 4 KB "snapshot" artifact is the checked-in docs/examples/data/redgate.json, not a fresh capture). Pre-existing on main, untouched here; noting it rather than widening this PR.

Generated by Claude Code

Codex's finding on #136, confirmed by reproduction before touching anything:
the handler-path scanner ran in process substitution, so its exit status was
discarded. A manifest that is valid JSON but not the shape the walk expects —
`"command": null` — raised TypeError, the remaining manifests were never
scanned, and because an EARLIER manifest had already emitted records the
zero-paths guard stayed quiet. On the unmodified file, nulling voice's command
made the `voice hooks.json SessionStart` line simply vanish; section 2b
reported no failure and the required tier exited 0 (1294 / 0), while stderr
carried the traceback nobody reads. Nulling agent-compiler's instead went red
only incidentally, via the zero-paths guard, and misreported "1 file(s)" when
three exist — the scan had been silently truncated.

Three layers now, each verified on its own:
  * Every shape the walk touches is checked, not assumed. Nonsense in a
    valid-JSON manifest emits a MALFORMED record naming the plugin and event
    instead of raising, and the walk continues to the next manifest.
  * The scanner writes to a temp file and its exit status is captured; a
    nonzero one is a recorded FAIL saying an unknown number of manifests were
    never checked.
  * The temp file itself fails closed if it cannot be created, and the
    "nothing to check" PASS moved inside the scanner-ran branch, so no
    silent-scanner path can produce that green line.

Verified against the final file:
  null command, last manifest   -> FAIL "MALFORMED hook entry (a hook has no
                                   string "command")"          1294 / 1, exit 1
  null command, first manifest  -> same FAIL, and redgate + voice still
                                   scanned after it              1294 / 1
  crash on every manifest       -> FAIL "scanner exited 1" + zero-paths FAIL
  crash on the LAST manifest    -> FAIL "scanner exited 1"      1294 / 1
    (the exact earlier-records shape that used to pass)
  "hooks" not an object / event not a list / entry's hooks not a list
                                -> one MALFORMED FAIL each
  TMPDIR unwritable             -> FAIL, and no "nothing to check" green
  The five prior mutations unchanged: main's pre-fix redgate (2 FAIL), handler
  renamed (1), ../ escape to a real file (1), regex broken (now reports
  "3 file(s)", not 1 — the walk no longer truncates), no-hooks root (PASS).
  cheap tier 1295 / 0 unchanged; counterfeit harness 25 / 0, baseline green.

The pre-fix reproduction and the full mutation sweep were run by a delegated
agent; the fixed-file cases above, both roots, and the hooks.json files being
byte-identical to HEAD were re-verified independently before this commit.

Survey, not fixed here: sections 10 (per-plugin safety-pack discovery) and 12
(install-smoke coverage) feed `while read` from the same `< <(python3 …)`
shape and neither asserts the enumerated count against marketplace.json, so a
scanner crash after N plugins would silently run only N packs and stay green.
Section 10 is the one this repo's safety story rests on. Follow-up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
@JRichlen
JRichlen merged commit 9ace33a into main Sep 15, 2026
56 checks passed
JRichlen pushed a commit that referenced this pull request Sep 18, 2026
The merge brought in #136, which added checks. The figure this branch
introduced (~19s / 1290 checks, measured 2026-09-14) was already stale
against its own instruction to re-measure rather than trust it.

Measured on this checkout at 2026-09-18: 13.6s, 1296 checks, 25 plugins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants