Harden agent compatibility scoring

.github/workflows/validate-plugins.yml

@@ -6,10 +6,19 @@ on:
       - ".cursor-plugin/marketplace.json"
       - "**/plugin.json"
       - "schemas/**"
+      - "agent-compatibility/agents/**"
+      - "agent-compatibility/skills/**"
+      - "agent-compatibility/test/**"
+      - "agent-compatibility/README.md"
+      - "agent-compatibility/CHANGELOG.md"
+      - ".github/workflows/validate-plugins.yml"
 
 jobs:
   validate:
-    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
 
@@ -22,3 +31,6 @@ jobs:
 
       - name: Validate plugin definitions
         run: node scripts/validate-plugins.mjs
+
+      - name: Test agent compatibility contracts
+        run: node --test agent-compatibility/test/*.test.mjs

agent-compatibility/.cursor-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "agent-compatibility",
   "displayName": "Agent Compatibility",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "CLI-backed repo compatibility scans plus agents that audit startup, validation, and docs against reality.",
   "author": {
     "name": "Cursor",
@@ -21,12 +21,7 @@
     "validation"
   ],
   "category": "developer-tools",
-  "tags": [
-    "agents",
-    "compatibility",
-    "quality",
-    "workflow"
-  ],
+  "tags": ["agents", "compatibility", "quality", "workflow"],
   "skills": "./skills/",
   "agents": "./agents/"
 }

agent-compatibility/CHANGELOG.md

@@ -1,11 +1,23 @@
 # Changelog
 
-All notable changes to this plugin will be documented here.
+All notable changes to this plugin are documented here.
 
 ## Unreleased
 
-- Renamed the full-pass skill to `check-agent-compatibility`.
-- Renamed `deterministic-scan-review` to `compatibility-scan-review`.
-- Renamed `docs-reality-review` to `docs-reliability-review`.
-- Clarified the score model so `Agent Compatibility Score` is the final blended score and `Deterministic Compatibility Score` is the raw CLI score.
-- Tightened the README, marketplace copy, and agent wording for public release.
+## 1.1.0 - 2026-08-19
+
+- Pinned the deterministic scanner to `agent-compatibility@0.1.7`.
+- Added an executable scanner guard that validates the pinned version, scanned path, output shape, and Cloudflare Worker classification signals.
+- Made pinned scanner execution shell-free on Windows by resolving npm's `npx-cli.js` entrypoint through Node.
+- Added a fail-closed result synthesizer as the sole owner of score validation, degraded states, and 70/30 arithmetic.
+- Made the synthesizer reject missing evidence, mismatched targets, and stateful results without isolated execution provenance.
+- Made startup and validation writable only inside isolated copies, with deploy, migration, credential, and paid-test boundaries.
+- Added explicit target, budget, mutation, and evidence handoffs for every subagent.
+- Changed specialist output to structured JSON with command outcomes and evidence.
+- Added contract and fixture tests plus CI coverage for agent, skill, and helper-script changes.
+
+## 1.0.0 - 2026-03-25
+
+- Added the full compatibility pass with deterministic, startup, validation, and docs-reliability reviews.
+- Added the 70/30 deterministic and workflow score model.
+- Added marketplace metadata and usage documentation.

agent-compatibility/README.md

@@ -1,86 +1,113 @@
 # Agent Compatibility
 
-Cursor plugin for checking how well a repo holds up under agent workflows. It pairs the published `agent-compatibility` CLI with focused reviews for startup, validation, and docs reliability.
+Cursor plugin for checking how well a repository holds up under agent workflows. It combines a pinned deterministic scanner with observed startup, validation, and docs-reliability reviews.
 
-By default, the full pass returns one overall score and one short list of the highest-leverage fixes. If the user wants the full breakdown, the agents can expose the component scores and the reasoning behind them.
+The default result is one score and a short list of evidence-backed fixes. Ask for a breakdown to see component scores, commands, scanner version, and file references.
 
 ## What it includes
 
-- `check-agent-compatibility`: full compatibility pass
-- `compatibility-scan-review`: raw CLI-backed scan
-- `startup-review`: cold-start and bootstrap review
-- `validation-review`: small-change verification review
-- `docs-reliability-review`: docs reliability review
+- `check-agent-compatibility`: orchestrates the full pass
+- `compatibility-scan-review`: runs and validates the deterministic scan
+- `startup-review`: tests cold bootstrap and startup in an isolated copy
+- `validation-review`: tests the narrowest credible verification loop in an isolated copy
+- `docs-reliability-review`: checks docs against repository interfaces without changing state
+
+## Reliability model
+
+The full pass uses one target root throughout:
+
+1. Run the pinned scanner through an executable guard that verifies its version, scanned path, output shape, and repository classification.
+2. Run startup, validation, and docs reviews in parallel with the scanner result as context.
+3. Require structured results with a matching canonical target, command outcomes, file evidence, and isolated execution provenance from every stateful specialist.
+4. Retry malformed specialist output once instead of guessing missing values.
+5. Pass the four lane results through an executable synthesizer that validates states and owns all score arithmetic.
+6. Refuse to compute the blended score when the deterministic scan is unavailable or its classification conflicts with obvious repository signals.
+
+Startup and validation use writable isolated copies because install, build, test, and start commands often create state. They must not deploy, publish, migrate data, use production credentials, or run paid or live tests. The docs review remains read-only.
 
 ## Score model
 
-- `Agent Compatibility Score`: final blended score shown to the user
-- `Deterministic Compatibility Score`: raw score from the published CLI
-- `Startup Compatibility Score`: how much guesswork it takes to boot the repo
-- `Validation Loop Score`: how practical it is to verify a small change
-- `Docs Reliability Score`: how closely the docs match the real setup path
+- `Agent Compatibility Score`: final blended score
+- `Deterministic Compatibility Score`: raw score from the pinned CLI
+- `Startup Compatibility Score`: observed cold-start friction
+- `Validation Loop Score`: observed small-change verification quality
+- `Docs Reliability Score`: documented paths compared with real repository interfaces
 
-The final score blends the deterministic scan with the workflow checks:
+When every component is usable:
 
 ```text
+workflow = round((startup + validation + docs) / 3)
 Agent Compatibility Score = round((deterministic * 0.7) + (workflow * 0.3))
 ```
 
-The CLI also reports an accelerator layer for committed agent tooling. That extra context informs recommendations, but it does not inflate the deterministic compatibility score itself.
+If the deterministic scan is unavailable or its classification is unreliable, the plugin reports `Agent Compatibility Score: unavailable` and shows a clearly labeled workflow-only score. It never invents the missing deterministic value or silently changes the weighting.
+
+The scanner's accelerator layer informs recommendations but does not inflate the deterministic score.
 
 ## How to use it
 
-Use `check-agent-compatibility` when you want the full pass. That skill fans out to the four review agents above, then returns a compact result:
+Use `check-agent-compatibility` for the full pass. A successful result stays compact:
 
 ```md
 ## Agent Compatibility Score: 72/100
 
 Top fixes
+
 - First issue
 - Second issue
 ```
 
-Ask for a breakdown if you want the component scores or the weighting.
-
 ## CLI notes
 
-The plugin does not bundle the scanner. It runs the published npm package when needed.
+Plugin version 1.1.0 pins scanner version 0.1.7. Updating the scanner requires a plugin version change and contract-test update.
 
-Default scan (compact terminal dashboard):
+The plugin invokes the scanner through `skills/check-agent-compatibility/scripts/run-deterministic-scan.mjs`; direct commands below are for manual inspection. On Windows, the helper runs npm's JavaScript entrypoint through `node.exe` instead of invoking the `npx.cmd` shell shim. It computes final scores with `scripts/synthesize-results.mjs`, which fails closed on malformed or inconsistent lane results.
+
+Default scan:
 
 ```bash
-npx -y agent-compatibility@latest .
+npx -y agent-compatibility@0.1.7 .
 ```
 
 JSON output:
 
 ```bash
-npx -y agent-compatibility@latest --json .
+npx -y agent-compatibility@0.1.7 --json .
 ```
 
 Markdown output:
 
 ```bash
-npx -y agent-compatibility@latest --md .
+npx -y agent-compatibility@0.1.7 --md .
 ```
 
-Plain text output:
+Plain-text output:
 
 ```bash
-npx -y agent-compatibility@latest --text .
+npx -y agent-compatibility@0.1.7 --text .
 ```
 
-Config override for ignored paths or weight overrides:
+Config override for ignored paths or check weights:
+
+```bash
+npx -y agent-compatibility@0.1.7 . --config ./agent-compatibility.config.json
+```
+
+The scanner is heuristic. It scores repository signals and surfaces likely friction; it is not a general code-quality verdict.
+
+## Validate the plugin
+
+Run the contract suite from the marketplace repository root:
 
 ```bash
-npx -y agent-compatibility@latest . --config ./agent-compatibility.config.json
+node --test agent-compatibility/test/*.test.mjs
 ```
 
-The scanner is heuristic. It scores repo signals and surfaces likely friction, but it is not a full quality verdict on the codebase.
+The suite checks execution permissions, scanner pinning, executable guard fixtures, fail-closed score synthesis, degraded scoring, subagent handoffs, structured output, classification safeguards, side-effect boundaries, and CI coverage.
 
 ## Local install
 
-If you want to use this plugin directly, symlink this directory into:
+Symlink this directory into:
 
 ```bash
 ~/.cursor/plugins/local/agent-compatibility

agent-compatibility/agents/compatibility-scan-review.md

@@ -1,40 +1,58 @@
 ---
 name: compatibility-scan-review
-description: Run the agent-compatibility CLI and return the raw repository score with its main problems
+description: Run the pinned agent-compatibility CLI, validate its repository classification, and return structured evidence.
 model: fast
-readonly: true
+readonly: false
 ---
 
 # Compatibility scan review
 
-Runs the published scanner and reports the raw repository score.
-
-## Trigger
-
-Use when the task is specifically to run the published `agent-compatibility` scanner and report the raw compatibility result.
+Run the deterministic scanner without modifying the target repository.
 
 ## Workflow
 
-1. Try the published scanner first with `npx -y agent-compatibility@latest --json "<path>"`.
-2. If you are clearly working inside the scanner source repo and the published package path fails for an environment reason, fall back to the local scanner entrypoint.
-3. Only say the scanner is unavailable after you have actually tried the published package, and the local fallback when it is clearly available.
-4. Prefer JSON when you need structured reasoning. Prefer Markdown when the user wants a direct report.
-5. Keep the scanner's real score, summary direction, and problem ordering.
-6. Do not bundle in startup, validation, or docs-reliability judgments. Those belong to separate agents.
-
-## Output
+1. Require canonical absolute `Target root` and `Scanner helper` paths from the parent task.
+2. Confirm `Scanner helper` ends in `scripts/run-deterministic-scan.mjs`, then run `node "<Scanner helper>" "<Target root>"` exactly once. The helper runs `agent-compatibility@0.1.7`, validates the scanned path, and checks the scanner classification against obvious repository signals.
+3. Return the helper's JSON exactly. Do not reinterpret its score, classification, reliability, command outcomes, or failure status.
+4. If the helper process exits nonzero but emits valid JSON, return that JSON; `unavailable` is evidence about the tool or environment, not a repository score of zero.
+5. If the helper cannot be read or emits invalid JSON, return `unavailable` using the schema below. Do not run the scanner directly or substitute a different package version.
+6. Do not make startup, validation-loop, or docs-reliability judgments.
 
-Reply in **plain text only** (no markdown fences, no `#` headings, no emphasis syntax). Use this layout:
+Npm cache writes are allowed. Do not change files in the target repository.
 
-First line: `Deterministic Compatibility Score: <score>/100`
-
-Then a short summary paragraph.
-
-Then the line `Problems` followed by one bullet per line using `- `.
+## Output
 
-- Use the compatibility scan's real score.
-- Keep accelerator context separate from the deterministic compatibility score itself.
-- Include both rubric issues and accelerator issues when they matter.
-- If there are no meaningful problems, under Problems write `- None.`
-- Do not treat scanner availability as a defect in the target repo.
-- If the scanner truly cannot be run, say that the deterministic scan is unavailable because of the tool environment, not because the repo lacks a compatibility CLI.
+Return JSON only, with no markdown fence:
+
+Allowed `status` values are `complete`, `unreliable`, and `unavailable`. Allowed command `outcome` values are `passed`, `failed`, and `blocked`.
+An obviously wrong classification is returned as `"unreliable"`, with the scanner's evidence preserved.
+
+```json
+{
+  "status": "complete",
+  "scoreName": "Deterministic Compatibility Score",
+  "score": 84,
+  "scannerVersion": "0.1.7",
+  "targetRoot": "/absolute/path",
+  "scannedPath": "/absolute/path",
+  "classification": "application",
+  "classificationReliable": true,
+  "summary": "Short evidence-based summary.",
+  "evidence": ["scannerVersion: 0.1.7", "scannedPath: /absolute/path"],
+  "problems": [
+    {
+      "title": "Problem title",
+      "evidence": ["file:line or scanner evidence"],
+      "remediation": "Concrete fix"
+    }
+  ],
+  "commands": [
+    {
+      "command": "npx -y agent-compatibility@0.1.7 --json <target>",
+      "outcome": "passed"
+    }
+  ]
+}
+```
+
+Always return `targetRoot`, `summary`, and at least one `evidence` string. Use `null` for `score`, `scannerVersion`, or `classification` when unavailable. Use an empty `problems` array when no meaningful deterministic problem exists.

agent-compatibility/agents/docs-reliability-review.md

@@ -1,44 +1,57 @@
 ---
 name: docs-reliability-review
-description: Check whether the documented setup and run paths reliably lead to the real working path
+description: Check whether documented setup, run, and validation paths match the repository's real interfaces.
 model: fast
 readonly: true
 ---
 
 # Docs reliability review
 
-Follows the written setup path and reports where the docs drift from reality.
-
-## Trigger
-
-Use when the user wants to know whether the repo documentation is actually trustworthy for an agent starting fresh.
+Measure whether a cold agent can trust the written setup and workflow guidance.
 
 ## Workflow
 
-1. If a compatibility scan result is already available from the parent task, use it as context. Otherwise run the compatibility scan once.
-2. Read the obvious documentation surfaces: `README`, setup docs, env docs, and contribution or agent guidance.
-3. Follow the documented setup and run path as literally as practical.
-4. Note where docs are accurate, stale, incomplete, or misleading.
-5. Pick a specific score instead of a round bucket. Start from these anchors and move a few points if the evidence clearly warrants it:
-   - around `93/100` if the docs lead to the working path with little or no correction.
-   - around `84/100` if the docs drift in places but an agent can still get to the right setup or run path without much guesswork.
-   - around `68/100` if the docs are stale enough that the agent has to reconstruct important steps from the tree or CI.
-   - around `27/100` if the docs point the agent down the wrong path or omit key steps you need to proceed.
-   - around `12/100` if the real path depends on private docs or internal context that is not available in the repo.
-6. Prefer a specific score such as `81`, `85`, or `92` over a multiple of ten when that is the more honest read.
+1. Require `Target root`, `Deterministic scan result`, `Time budget`, `Allowed mutations`, and `Required evidence` from the parent.
+2. Read the README, setup and environment docs, contribution guidance, agent instructions, manifests, and root task definitions.
+3. Trace documented install, run, and validation commands to their real scripts or targets without changing repository state.
+4. Use the passed deterministic result as context. Do not install or rerun the scanner.
+5. Record exact mismatches, missing prerequisites, stale names, unsupported claims, and commands with no real target.
+6. Score the damage caused by drift. Minor wording differences should not drag an otherwise reliable path into the midrange.
+7. Return status `complete` when the review produced a defensible score. Use `unavailable` only when the target or required files cannot be inspected.
 
-## Output
+## Scoring anchors
 
-Reply in **plain text only** (no markdown fences, no `#` headings, no emphasis syntax). Use this layout:
+- About `93`: docs lead to the working path with little or no correction.
+- About `84`: limited drift exists, but recovery takes little guesswork.
+- About `68`: important steps must be reconstructed from the tree or CI.
+- About `27`: docs point down the wrong path or omit required steps.
+- About `12`: the real path depends on unavailable private context.
 
-First line: `Docs Reliability Score: <score>/100`
+Choose a specific score supported by file references.
 
-Then a short summary paragraph.
-
-Then the line `Problems` followed by one bullet per line using `- `.
+## Output
 
-- Base the score on what happened when you followed the docs.
-- Build Problems from real mismatches, omissions, or misleading guidance.
-- If the repo is blocked on secrets or infrastructure, say so plainly and still use the same output shape.
-- Minor drift or stale references should not drag a good repo into the mid-60s if the real path is still easy to recover.
-- Score the damage from the drift, not the mere existence of drift.
+Return JSON only, with no markdown fence:
+
+Allowed `status` values are `complete` and `unavailable`.
+
+```json
+{
+  "status": "complete",
+  "scoreName": "Docs Reliability Score",
+  "score": 84,
+  "targetRoot": "/absolute/target/path",
+  "summary": "Short evidence-based summary.",
+  "evidence": ["README.md:20 maps to package.json#scripts.test"],
+  "problems": [
+    {
+      "title": "Problem title",
+      "evidence": ["file:line"],
+      "remediation": "Concrete fix"
+    }
+  ],
+  "commands": []
+}
+```
+
+Always return `targetRoot`, `summary`, and at least one `evidence` string. Use `null` for `score` only when status is `unavailable`.