feat(skills): detect ambiguous-trigger verbs in bundled skill descriptions

[ricardo-leiva]

Problem

Bundled PostHog analytics skills (101 of them) are injected into every agent session context regardless of whether the user is doing analytics work or coding. Skills whose descriptions contain coding-overlap verbs like investigate, debug, or explore — with no trigger guard — give the model no basis to distinguish a PostHog analytics intent from an unrelated coding request. The result is false-positive skill invocations during coding sessions (~10,920 tokens of context bias).

Changes

analyzeSkills.ts — extended with:

• AMBIGUOUS_TRIGGER_VERBS const: the 8 coding-overlap verbs that cause false invocations (investigate, diagnose, debug, explore, analyze, audit, instrument, set up)
• hasAmbiguousTrigger(skill): returns true for bundled skills that contain one of those verbs without an accepted trigger guard (TRIGGER when, Use when, or Read when)
• New "ambiguous-trigger" issue type in analyzeSkills() reporting which verbs matched

analyzeSkills.test.ts — 11 new unit tests covering the ambiguous-trigger detection path: bundled vs. non-bundled sources, each guard pattern variant, matched-verb reporting, and a sentinel on the verb list.

bundled-skill-corpus.test.ts (new) — integration tests that load the real built skill files from .vite/build/plugins/posthog/skills/ and assert corpus-level properties:

1. Sanity check: >50 skills loaded
2. No skill has an empty description
3. Every ambiguous-verb skill has a TRIGGER guard
4. No single unguarded verb dominates >8 descriptions (intent-collision check)
5. analyzeSkills reports zero issues on the full corpus

The corpus tests skip automatically on fresh checkouts (before pnpm build) using describe.skipIf.

biome.jsonc — added one override: core *.test.ts files are exempt from the node:* import restriction, since Vitest always runs in Node.js. Production source files in @posthog/core remain fully restricted.

How did you test this?

Agent-run automated tests only — I'm not a human tester and won't claim manual verification.

• pnpm --filter @posthog/core test — 1774 tests pass including all 11 new unit tests and all 5 corpus integration tests
• pnpm --filter @posthog/core typecheck — no errors
• pnpm lint — no errors on changed files

The corpus tests run against the actual built skill files, so they catch regressions whenever a new skill is added or an existing description is changed without a trigger guard.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Authored with Claude Code (claude-sonnet-4-6) in a session driven by @ricardo-leiva. The user identified that 101 bundled PostHog analytics skills loaded into every agent session context without trigger guards could bias the model toward analytics skill invocations on coding tasks. I analyzed the skill corpus, designed the static analysis approach (deterministic, no model calls), implemented the hasAmbiguousTrigger detector, wrote the corpus integration test, and fixed the 5 skills in the PostHog repo that were genuinely missing guards (see companion PR there). No skills were added or removed — the fix is purely descriptor-quality enforcement. The biome.jsonc override for core test files was required because the corpus test reads from disk using node:fs, which the core purity rule otherwise blocks for production code.

[greptile-apps]

_{Reviews (1): Last reviewed commit: "feat(skills): detect ambiguous-trigger v..." | Re-trigger Greptile}