From a05ead037e129f54d3ebb50f6e6376ed86aa9d58 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 17:06:22 +0000 Subject: [PATCH] Add CLAUDE.md and docs/PLAN.md with roadmap course of action CLAUDE.md gives future sessions repo context and condensed priorities; docs/PLAN.md sequences the 6 open issues into phases with the agent roles needed to execute each one. --- CLAUDE.md | 60 +++++++++++++++++++++++++++ docs/PLAN.md | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 172 insertions(+) create mode 100644 CLAUDE.md create mode 100644 docs/PLAN.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..2881e12 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,60 @@ +# CLAUDE.md + +Guidance for Claude Code sessions working in this repo. Keep this file short; +the full roadmap and agent breakdown lives in [docs/PLAN.md](docs/PLAN.md). + +## What this repo is + +**tool-semantics** is a Python CLI/library that catches breaking changes in +AI-agent tool interfaces (MCP servers, tool APIs) before they ship. It +snapshots a tool interface, diffs baseline vs. candidate, and reports risk +across five compatibility layers: + +1. Protocol — can the client still speak to the server? +2. Schema — are parameters/types still valid? +3. Tool selection — will models still pick the right tool? +4. Execution — do calls still succeed with prior argument patterns? +5. Intent / side effects — did risk or confirmation needs change? + +Status: layers 1-2 are fully implemented and CI-gateable (structural diff, +exit codes 0/1/2). Layers 3-5 exist only as heuristic warnings today — closing +that gap is the main open work (see Current priorities below). + +## Repo layout + +- `src/tool_semantics/scanner.py` — captures manifests into `InterfaceSnapshot` +- `src/tool_semantics/mcp_capture.py` — live MCP capture over stdio +- `src/tool_semantics/diff.py` — structural comparison engine +- `src/tool_semantics/models.py` — snapshot/report data models +- `src/tool_semantics/probes.py` — offline behavioral probe harness (`Probe`, `ProbeKind`, `evaluate_probes`) +- `src/tool_semantics/adapters.py` — migration adapters (tool alias/arg translation) +- `src/tool_semantics/policy.py` — release-policy enforcement knobs +- `src/tool_semantics/redact.py` — secret/unstable-field redaction +- `src/tool_semantics/report.py` — Markdown/JSON report rendering +- `src/tool_semantics/cli.py` — Typer CLI entrypoint +- `tests/` — pytest suite, one file per module area +- `examples/` — demo MCP-style manifests (GitHub server v1/v2, weather) +- `docs/` — architecture, config, change-codes, github-action, publishing, adapters + +## Dev commands + +```bash +python -m venv .venv +source .venv/bin/activate +pip install -e ".[dev]" + +pytest +ruff check . + +tool-semantics capture examples/github_server_v1.json -o .tool-semantics/v1.json +tool-semantics compare .tool-semantics/v1.json .tool-semantics/v2.json --markdown-output report.md +``` + +## Current priorities (condensed — see docs/PLAN.md for full sequencing) + +1. **P0 — Ship a real release** ([#31](https://github.com/askmy-stack/tool-semantics/issues/31)): no GitHub Release or working `pip install tool-semantics` exists yet, despite the publish workflow being built. This blocks everything downstream. +2. **P1 — Provider-neutral model runner** ([#45](https://github.com/askmy-stack/tool-semantics/issues/45)): unblocks the model-backed probe work (#44, #46, #47) that closes the layers-3-5 gap. +3. **Ongoing**: dependency/CI hygiene is already automated via Dependabot — no action needed unless a PR fails. + +Full phased plan, dependencies between issues, and required agent roles per +phase: [docs/PLAN.md](docs/PLAN.md). diff --git a/docs/PLAN.md b/docs/PLAN.md new file mode 100644 index 0000000..962b6d0 --- /dev/null +++ b/docs/PLAN.md @@ -0,0 +1,112 @@ +# Course of action: tool-semantics roadmap to Milestone 4 + +This is the full, sequenced plan for closing out the remaining tool-semantics +roadmap, based on the current open-issue set (6 open issues, 0 open PRs as of +this writing) and [ROADMAP.md](../ROADMAP.md). For a short summary, see +[CLAUDE.md](../CLAUDE.md). + +Milestones 0, 1, 2, 5, and 6 are shipped. Milestone 3 is mostly done (offline +probes, side-effect expectations); Milestone 4 (model matrix) is entirely +open. The plan below sequences the 6 open issues into four phases, each with +the agent role(s) needed to execute it. + +## Phase 0 — Release unblock + +**Issue:** [#31](https://github.com/askmy-stack/tool-semantics/issues/31) (p0) + +**Why first:** `publish.yml` and trusted-publishing docs already exist +([docs/publishing.md](publishing.md)), but no GitHub Release has ever been +cut and README still says `# later: pip install tool-semantics`. Nothing +downstream — Action adoption, library users, even this plan's later phases — +matters if the package isn't installable. + +**Scope:** +- One-time PyPI trusted publisher + `pypi` GitHub Environment (steps already + documented in `docs/publishing.md`) +- Reconcile `version` in `pyproject.toml` and `src/tool_semantics/__init__.py` +- Move CHANGELOG.md Unreleased → dated `0.2.0` (or next) section +- Tag + GitHub Release `vX.Y.Z`, verify the publish workflow succeeds +- Update README install section to drop "later:" wording +- Verify `pip install tool-semantics` in a clean venv + +**Required agent:** a single **implementation agent** (general-purpose). +This is a checklist execution task with an existing runbook +(`docs/publishing.md`) — no design ambiguity, so no Explore/Plan agent needed. + +## Phase 1 — Remote MCP transport + +**Issue:** [#43](https://github.com/askmy-stack/tool-semantics/issues/43) + +**Scope:** SSE/remote MCP capture support, extending the existing local +stdio transport in `src/tool_semantics/mcp_capture.py`. Must produce the +same deterministic `InterfaceSnapshot` format as local capture, document the +supported transport(s) and auth boundary, reuse `src/tool_semantics/redact.py` +for secret-like metadata, and keep existing stdio behavior unchanged. + +**Required agents:** +1. **Explore agent** — map the current stdio implementation in + `mcp_capture.py` to find the right extension seam (transport + abstraction vs. new function). +2. **Plan agent** — design the transport interface; this is the one phase + with genuine design uncertainty (which remote transport(s) to support, + how auth is passed without being treated as trusted metadata). +3. **Implementation agent** — build it against the Plan agent's design, + with tests for success / invalid-endpoint / auth-error paths. + +## Phase 2 — Provider-neutral model runner + +**Issue:** [#45](https://github.com/askmy-stack/tool-semantics/issues/45) + +**Why this order:** #45 is the foundation #44, #46, and #47 all depend on — +building any of them first would mean redoing them once the runner interface +lands. + +**Scope:** a runner interface that separates provider transport from probe +evaluation, at least one provider adapter behind it, model/provider/version +and run-config metadata recorded in results, configurable timeouts/retries/ +cost limits, and — critically — no required provider SDK dependency for +users who only want the deterministic offline checks. Extend, don't replace, +`src/tool_semantics/probes.py` (`Probe`, `ProbeKind`, `evaluate_probes`). + +**Required agents:** +1. **Plan agent** — the interface design is the crux of this phase (getting + the transport/evaluation separation and the "no forced SDK" constraint + right up front avoids rework in Phase 3). +2. **Implementation agent** — builds the interface + one adapter, with + fakes-based unit tests (no live model calls per the issue's acceptance + criteria). + +## Phase 3 — Model-backed probes, metrics, and stability + +**Issues:** [#44](https://github.com/askmy-stack/tool-semantics/issues/44), +[#46](https://github.com/askmy-stack/tool-semantics/issues/46), +[#47](https://github.com/askmy-stack/tool-semantics/issues/47) + +All three depend on the Phase 2 runner interface and are labeled +`research` — treat their acceptance criteria as a starting point, not a +fixed spec; re-scope after Phase 2 lands if the interface shape changes +assumptions. + +- **#44** — human-reviewed probe format, opt-in model-backed execution + layered onto existing offline probes, results recording selected + tool/arguments/outcome/errors, offline behavior stays backward compatible. +- **#46** — tool-selection accuracy and argument-validity reporting, in both + JSON and Markdown via the existing `src/tool_semantics/report.py` + rendering rather than a new output path. +- **#47** — configurable repeated trials, per-trial + aggregate stability + scoring, reports distinguishing unstable probes from deterministic + failures. + +**Required agents:** three **implementation agents**, one per issue, run +sequentially in the order above (#44 → #46 → #47), since #46's metrics and +#47's stability scoring both consume #44's probe execution results. + +## Cross-cutting rules for every phase + +- Run `pytest`, `ruff check .`, and the existing pre-commit hooks + (`.pre-commit-config.yaml`) before considering a phase done — don't add + new lint config. +- Every phase lands via a PR against `main`, never a direct push. +- Keep new code inside existing module boundaries listed in CLAUDE.md; + reuse `probes.py`, `report.py`, and `redact.py` rather than duplicating + their responsibilities in new files.