fix(static): analyze description activation intent - #541
Conversation
|
Ran it here since your worktree cannot: The fallback itself is fine. The mismatch is shape. TR3's three baiting patterns are all anchored Since a spec-compliant description is prose by definition, TR1 to TR3 as written will almost never fire on one. Adapt the patterns for sentences rather than the fixture? |
fd55298 to
ab2c9dd
Compare
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Reviewed current head b04c76ca75fa17ebab8ffa0a182cefa909f2e128. The metadata fallback is wired correctly and all hosted checks now pass, but the revised regression only changes the fixture to the one exact description shape the legacy trigger grammar already accepts. It does not establish the advertised analysis of spec-style prose descriptions.
TR1 requires a one-word value, TR2 accepts only a built-in command (or at most two words), and every TR3 expression is anchored to the whole value. A normal description such as Use this skill whenever the user sends any message therefore produces no finding even though it states the same activation intent as all messages. Adapt the description path to extract/match activation clauses without weakening the legacy trigger path, and add positive prose plus benign-description negative regressions. The inline finding identifies the behavior boundary.
bdf32dd to
66790c8
Compare
yashrajp22
left a comment
There was a problem hiding this comment.
The description fallback still needs calibration. The existing prose-detection finding remains reproducible; I am referencing it here to avoid a duplicate inline comment. The additional comment covers ordinary capability descriptions being reported as command shadowing.
Verified head 66790c8b8a2b77001a93c5b76adf73b0373fd477 against current main c13f70ebf14905912c616a58c9a8cb8112ef94a4: fresh wheels and installed-package identities verified, 31 selected tests and two trigger concurrency checks passed, and all 110 offline scans completed. The 48 complete-directory corpus runs agree across source/wheel and base/head. Six focused assertions reproduce the two defects; one separate location assertion exposes the existing list-index location issue. Baseline partial-coverage/reference warnings remain. Greptile’s legacy-precedence suggestion was reviewed and excluded because this PR explicitly promises fallback behavior. These are scoped offline checks, not global detection-accuracy or live-provider coverage.
66790c8 to
e378657
Compare
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Re-reviewed current head e3786579d64ebd87ac578ba0b6efea7f0524a111 against both current review threads, the complete two-file diff, trigger grammar, surrounding analyzer behavior, and exact-head checks.
The requested description-aware behavior remains unimplemented. The current code still passes the entire description to legacy exact/whole-string trigger rules, and the only added test still uses the legacy-shaped value all messages. Realistic activation prose remains undetected, while ordinary two-word capability descriptions beginning with a built-in command can still produce false command-shadowing findings. Implement bounded activation-intent extraction (or an equivalent description-specific matcher) and add realistic positive and benign negative regressions. I have not duplicated the existing inline findings.
All six exact-head checks pass, but the unresolved correctness findings and active change requests block merging.
Priority: P1 — trigger-intent detection materially affects analyzer precision and coverage.
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
e378657 to
ecbe9b3
Compare
|
All review comments are addressed on the current head
CI is green on this head: https://github.com/NVIDIA/SkillSpector/actions/runs/35404559781 (lint, test-unit, docker-smoke, DCO). Per your 2026-09-18 note, this PR is ready for re-review. |
Address rng1995 P1 and yashrajp22 reviews on NVIDIA#541: descriptions are no longer passed whole to the legacy whole-string trigger grammar. Bounded clause extraction keeps only clauses with activation-intent signals; TR3 uses unanchored baiting patterns so realistic prose (e.g. 'whenever the user sends any message') is detected, and TR2 requires invocation/shadowing intent so ordinary capability prose ('Build projects', 'Deploy infrastructure') stays out of the trigger path. Legacy triggers field behavior unchanged. Adds positive and benign-negative regressions. Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
Repair pushed with commit 598f6a0, addressing both reviews. Descriptions are no longer passed whole to the legacy whole-string trigger grammar: bounded clause extraction keeps only clauses with activation-intent signals, TR3 now uses unanchored baiting patterns (catches "description: Use this skill whenever the user sends any message"), and TR2 requires invocation/shadowing intent so benign capability prose ("Build projects", "Deploy infrastructure") does not trip it. Legacy |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Re-reviewed current head 598f6a0627b6ab2beb19c0891a340d68b1f7e2c9 against all prior reviews, both review threads and author replies, the complete two-file diff, surrounding trigger rules and manifest/analyzer integration, the linked issue #458, existing and added regression cases, and current CI.
The exact cases from the earlier comments are now addressed: Use this skill whenever the user sends any message reaches TR3, Build projects and Deploy infrastructure no longer produce TR2, and explicit legacy triggers still use the unchanged legacy rules. Both old inline threads are resolved. I am not repeating those resolved examples.
The new description matcher still requires changes before approval. The inline findings cover: benign capability/limited-domain prose being classified as catch-all activation, direct command-shadowing descriptions discarded before TR2, silent omission of long/later activation clauses, and TR1 remaining unreachable from descriptions despite the stated TR1-TR3 scope. Please address these behavior boundaries and add positive, negative, and limit regressions, including a test starting from actual SKILL.md frontmatter.
All six checks on this commit have completed successfully: changes, lint, test-unit, OpenCode TypeScript Tests, DCO Check, and docker-smoke. These findings are based on tracing the current source and inspecting its tests; contributor code was not executed locally. Green CI currently lacks the cases described below.
Require a bounded activation condition plus an unconditional scope for description TR3, so bare behavior prose (Always preserves file permissions when copying files) and subject-qualified scopes (any questions about PostgreSQL) stay negative while the catch-all positive still fires. Open the TR2 gate to invocation/shadowing clauses on their own, so Intercepts the /build command is analyzed without broad-activation wording. Analyze overlong clauses through bounded head/tail windows and scan every clause for intent signals, so padding cannot silently push a trigger clause out of the analysis. Add the TR1 description path by extracting the trigger phrase from activation prose (whenever the user says hello), proven reachable from SKILL.md frontmatter. Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
Both earlier cases are fixed on 3300434 — |
Address MohammedAlkindi's follow-up on the repaired head: the invocation-intent slash branch now requires a token-start slash, so a slash embedded in a larger token (CI/CD) no longer reads as slash-command invocation intent for TR2. Trigger-phrase extraction skips filler words between the verb and the phrase, so 'asks to create a poster' no longer flags TR1 on 'to'. Regression tests: test_description_embedded_slash_not_invocation_intent and test_description_trigger_phrase_skips_filler_words; test_patterns_new.py 501 passed, ruff lint and format clean. Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
Addressed both in 180dc27.
New regression tests: test_description_embedded_slash_not_invocation_intent, test_description_trigger_phrase_skips_filler_words. test_patterns_new.py 501 passed, ruff lint and format clean. The 49-slash/35-to corpus ratios should collapse on this head. |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Re-reviewed 180dc27203dff1b0ebc3f1eab2689c2b204879a0, including the complete current diff, surrounding manifest parsing and analyzer/coverage integration, every prior review and inline reply, thread states, the added and existing tests, and current CI.
Previous findings checked individually:
- The original realistic catch-all description is detected, and the original
Build projects/Deploy infrastructurenegatives remain clean. Legacy triggers retain their existing behavior. - The specific TR3 false-positive examples (
Always preserves file permissions when copying filesandAnswers any questions about PostgreSQL) are fixed and covered by tests. Broader precision still needs the new correction below. - Direct
Intercepts the /build commandnow reaches TR2 without catch-all wording; that earlier reachability finding is addressed. - The analysis-limit finding is only partially addressed: the exact 120/121-character and inert-padding examples now work, but the updated windowing/candidate limits still silently discard relevant text. See the continuation below.
- TR1 is now reachable from a parsed SKILL.md description, with the requested
helloregression. This resolves the earlier reachability finding; the new extraction introduces the multiword-phrase false positive below. - MohammedAlkindi's exact CI/CD and
asks to/for ... posterexamples are fixed with regression tests.
Approval is still blocked by the remaining silent-omission problem and the concrete precision regressions in the inline comments. Please extend the fixes to these behavior boundaries rather than adding only the earlier literal examples.
All six checks on this commit have passed: changes, lint, test-unit, OpenCode TypeScript Tests, DCO Check, and docker-smoke. The four threads from my last review remain open in GitHub; the two older threads are resolved. Source and tests were inspected, and hosted CI verified; contributor code was not executed locally.
Anchor overlong-clause windows at intent-signal matches so a trigger
sentence in the middle of padding is still inspected; raise the
signal-clause budget and report budget truncation as explicit incomplete
coverage via the inspection ledger. Extract the complete bounded trigger
phrase for TR1 so multiword task descriptions ('code review') are not
judged as single-word triggers. Require an actual invocation,
interception, or override claim (or a slash-command token) for
description TR2, so command documentation prose no longer reads as
shadowing. Bind the TR3 universal scope to the activation condition's own
span, so a scope in a separate instruction cannot establish keyword
baiting.
Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
All 25 words in A comma protects it, which is why the suite misses this. Every description fixture ends on the trigger phrase. |
Address MohammedAlkindi's precision finding on NVIDIA#541: the widened trigger-phrase group swallowed the word after a broad trigger, so 'says hello there' captured 'hello there' and the single-word TR1 guard never fired. Bare articles were also filler-skipped, dropping the broad word from the analysis entirely ('says the zone' captured 'zone'). - Bare the/a/an are no longer filler words, so the captured phrase keeps the broad word ('the zone', not 'zone'). - TR1 fires when the phrase is a broad single word followed only by trailing discourse words ('hello there'), since the skill names the broad word and the rest is trailing prose. A content word after the broad word ('code review', 'the zone', 'hello world') still names a multiword phrase and stays TR1 negative, matching the legacy trigger grammar and rng1995's P2 negatives. Add regression tests for the exact reported examples and for fixtures that do not end on the trigger phrase. Signed-off-by: deepujain <deepujain@gmail.com>
|
Addressed in d623587, with one deliberate boundary worth flagging. Fixed as reported:
Deliberate boundary: a content word after the broad word still names a multiword phrase, so 515 unit tests pass, ruff lint and format clean; CI is running on the new head. |
Fixes #458.
TR1-TR3 now analyze a spec-compliant SKILL.md description when legacy triggers metadata is absent, while retaining legacy triggers behavior.
Validation: git diff --check passed. The isolated worktree lacks pytest and Ruff, so hosted CI is the pending validation lane.