From 52dec779f9ad2a6fb7fea97c70ed498731e0c0c3 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:28:21 -0400 Subject: [PATCH 01/17] add snapshot storage design --- .../2026-08-13-snapshot-storage-design.md | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-13-snapshot-storage-design.md diff --git a/docs/superpowers/specs/2026-08-13-snapshot-storage-design.md b/docs/superpowers/specs/2026-08-13-snapshot-storage-design.md new file mode 100644 index 0000000..aa276c7 --- /dev/null +++ b/docs/superpowers/specs/2026-08-13-snapshot-storage-design.md @@ -0,0 +1,106 @@ +# Snapshot Storage and Provenance Design + +## Goal + +Make Tool-Semantics snapshots easy to retain, inspect, and trust across local +development, pull requests, and releases without introducing a database or +storing secrets. + +## Scope + +The first implementation adds three complementary storage paths: + +1. **Git baselines:** users keep approved JSON snapshots in their repository. +2. **CI artifacts:** the composite GitHub Action can upload the candidate + snapshot and compatibility reports from a workflow run. +3. **Provenance sidecar:** capture can optionally write a separate metadata + document describing where and when the snapshot was captured and its + SHA-256 content digest. + +Object storage and a centralized snapshot registry are explicitly out of scope. +They require credentials, retention policy, access control, and an API that this +library does not yet own. + +## User experience + +### Trusted baseline in Git + +Users capture and review a stable JSON baseline just as they do today: + +```bash +tool-semantics capture manifests/server.json \ + -o .tool-semantics/baselines/production.json +``` + +The baseline is an ordinary JSON file that can be reviewed in pull requests, +versioned with Git tags, and used by the existing `compare` command. + +### Provenance sidecar + +`capture` and `capture-mcp` receive an optional `--provenance-output PATH` +option. It writes a sibling or user-selected JSON file such as: + +```json +{ + "snapshot_path": ".tool-semantics/baselines/production.json", + "snapshot_sha256": "…", + "captured_at": "2026-08-13T00:00:00Z", + "source": { + "kind": "manifest", + "location": "manifests/server.json" + } +} +``` + +For live capture, `source.kind` is `mcp-stdio` or the future remote transport. +The source location contains a command or endpoint only after secret-like values +are redacted. Environment variables, authorization headers, and raw tokens are +never written. + +The sidecar is deliberately separate from `InterfaceSnapshot`. Timestamp and +provenance values change between captures and must not create compatibility +diffs. + +### CI artifacts + +The composite compare Action receives optional inputs for a candidate snapshot +and a boolean `upload-artifacts` input, defaulting to `false`. When enabled, it +uploads the candidate snapshot, JSON report, and Markdown report as a GitHub +Actions artifact. It never uploads the baseline unless the workflow author +explicitly supplies it as the candidate path. + +Artifacts are diagnostic outputs, not a source of truth. Git-tracked baselines +remain the official compatibility contract. + +## Data and integrity + +The SHA-256 value is computed from the exact bytes written to the snapshot file. +The provenance schema includes its own version field so it can evolve without +changing the snapshot schema. JSON is deterministic: keys are emitted in a +stable order and each file ends in one newline. + +## Error handling + +- A provenance write failure is treated like snapshot output failure: capture + exits with code 2 and reports the path. +- Artifact upload is optional. If GitHub cannot upload it, the Action fails so + the workflow author does not mistake a missing diagnostic for a saved one. +- Secrets in source metadata are redacted before serialization. Users retain + `--no-redact` only for snapshot capture; provenance never writes supplied + credentials. + +## Testing + +- Unit-test the provenance document for deterministic digest, source redaction, + and stable serialization. +- CLI-test `--provenance-output` for manifest and stdio capture paths. +- Add an Action fixture or static validation for the artifact upload inputs and + paths. +- Retain all existing snapshot and compare tests unchanged. + +## Rollout + +1. Ship provenance sidecars and optional CI artifacts in a minor release. +2. Document Git baselines as the recommended default. +3. Revisit object storage or a registry only after users need cross-repository + retention and search. From 951b7d3396d72dc2bf337d5ca231fb71f25a3d16 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:30:26 -0400 Subject: [PATCH 02/17] add snapshot storage implementation plan --- .../plans/2026-08-13-snapshot-storage.md | 312 ++++++++++++++++++ 1 file changed, 312 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-13-snapshot-storage.md diff --git a/docs/superpowers/plans/2026-08-13-snapshot-storage.md b/docs/superpowers/plans/2026-08-13-snapshot-storage.md new file mode 100644 index 0000000..33c3a65 --- /dev/null +++ b/docs/superpowers/plans/2026-08-13-snapshot-storage.md @@ -0,0 +1,312 @@ +# Snapshot Storage Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add optional provenance sidecars for captured snapshots and optional GitHub Actions artifacts for compare outputs. + +**Architecture:** Keep `InterfaceSnapshot` unchanged so compatibility comparison remains deterministic. A new provenance module hashes the exact snapshot bytes and writes a separate, redacted JSON sidecar. The composite Action conditionally uploads candidate and compare-report files as a diagnostic artifact; Git-tracked snapshot JSON remains the approved baseline. + +**Tech Stack:** Python 3.11+, Pydantic, Typer, pytest, GitHub composite Actions, `actions/upload-artifact@v4`. + +**Spec:** `docs/superpowers/specs/2026-08-13-snapshot-storage-design.md` + +## Global Constraints + +- Do not add a database, object-storage client, or new runtime dependency. +- Do not change the `InterfaceSnapshot` schema or include changing provenance fields in snapshot comparisons. +- Provenance must never serialize authorization headers, environment variables, or unredacted secret-like source values. +- Git baselines remain normal user-managed JSON files; artifacts are optional diagnostics. +- Preserve existing stdio and manifest capture behavior when provenance is not requested. + +--- + +## File Structure + +- Create `src/tool_semantics/provenance.py`: pure provenance creation and JSON writing. +- Create `tests/test_provenance.py`: deterministic digest and redaction behavior. +- Modify `src/tool_semantics/cli.py`: add `--provenance-output` to both capture commands. +- Modify `tests/test_cli.py`: verify both CLI capture paths write usable sidecars. +- Modify `.github/actions/compare/action.yml`: add optional artifact upload. +- Modify `docs/github-action.md` and `README.md`: document baseline, provenance, and artifacts. + +### Task 1: Provenance sidecar module + +**Files:** +- Create: `src/tool_semantics/provenance.py` +- Test: `tests/test_provenance.py` + +**Interfaces:** +- Produces: `write_provenance(snapshot_path: Path, output_path: Path, source: dict[str, Any], captured_at: datetime | None = None) -> None` +- Produces: `snapshot_sha256(snapshot_path: Path) -> str` +- Consumes: `tool_semantics.redact.redact_mapping` + +- [ ] **Step 1: Write the failing digest and redaction tests** + +```python +from datetime import UTC, datetime +import json + +from tool_semantics.provenance import write_provenance + + +def test_write_provenance_hashes_snapshot_and_redacts_source(tmp_path: Path) -> None: + snapshot = tmp_path / "snapshot.json" + snapshot.write_text('{"server_name":"demo"}\n', encoding="utf-8") + output = tmp_path / "provenance.json" + + write_provenance( + snapshot, + output, + {"kind": "mcp-stdio", "command": ["server", "--token=secret"]}, + captured_at=datetime(2026, 8, 13, tzinfo=UTC), + ) + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["provenance_version"] == "0.1" + assert payload["snapshot_sha256"] == hashlib.sha256(snapshot.read_bytes()).hexdigest() + assert payload["source"]["command"][1] == "***REDACTED***" + assert payload["captured_at"] == "2026-08-13T00:00:00+00:00" +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `pytest tests/test_provenance.py::test_write_provenance_hashes_snapshot_and_redacts_source -v` + +Expected: FAIL because `tool_semantics.provenance` does not exist. + +- [ ] **Step 3: Implement the smallest provenance API** + +```python +def snapshot_sha256(snapshot_path: Path) -> str: + return hashlib.sha256(snapshot_path.read_bytes()).hexdigest() + + +def write_provenance( + snapshot_path: Path, + output_path: Path, + source: dict[str, Any], + captured_at: datetime | None = None, +) -> None: + timestamp = captured_at or datetime.now(UTC) + payload = { + "provenance_version": "0.1", + "snapshot_path": str(snapshot_path), + "snapshot_sha256": snapshot_sha256(snapshot_path), + "captured_at": timestamp.isoformat(), + "source": redact_mapping(source), + } + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") +``` + +Use a redaction helper that also detects token-bearing command arguments such as +`--token=secret`, not only mapping keys. + +- [ ] **Step 4: Run the provenance tests to verify they pass** + +Run: `pytest tests/test_provenance.py -v` + +Expected: PASS. + +- [ ] **Step 5: Commit the provenance module** + +```bash +git add src/tool_semantics/provenance.py tests/test_provenance.py +git commit -m "add snapshot provenance sidecars" +``` + +### Task 2: Capture CLI provenance output + +**Files:** +- Modify: `src/tool_semantics/cli.py:50-160` +- Modify: `tests/test_cli.py` + +**Interfaces:** +- Consumes: `write_provenance(snapshot_path, output_path, source)` from Task 1. +- Produces: `capture --provenance-output PATH` and `capture-mcp --provenance-output PATH`. + +- [ ] **Step 1: Write failing CLI tests** + +```python +def test_capture_writes_requested_provenance(tmp_path: Path) -> None: + snapshot = tmp_path / "snapshot.json" + provenance = tmp_path / "snapshot.provenance.json" + result = runner.invoke( + app, + [ + "capture", "examples/github_server_v1.json", "-o", str(snapshot), + "--provenance-output", str(provenance), + ], + ) + assert result.exit_code == 0 + payload = json.loads(provenance.read_text(encoding="utf-8")) + assert payload["source"] == {"kind": "manifest", "location": "examples/github_server_v1.json"} +``` + +Add a second test that invokes `capture-mcp` with the existing fake server and +asserts its source kind is `mcp-stdio` and command values are redacted. + +- [ ] **Step 2: Run the new tests to verify they fail** + +Run: `pytest tests/test_cli.py -v` + +Expected: FAIL because `--provenance-output` is not recognized. + +- [ ] **Step 3: Add the CLI option and write the sidecar after each snapshot write** + +```python +provenance_output: Annotated[ + Path | None, + typer.Option("--provenance-output", help="Write capture provenance JSON separately."), +] = None, +``` + +After `write_snapshot(snapshot, output)`, call `write_provenance` only when the +option is supplied. For `capture`, pass `{"kind": "manifest", "location": str(manifest)}`. +For stdio capture, pass `{"kind": "mcp-stdio", "command": command}`. Route +provenance exceptions through the existing capture error handler as exit code 2. + +- [ ] **Step 4: Run CLI tests to verify they pass** + +Run: `pytest tests/test_cli.py -v` + +Expected: PASS. + +- [ ] **Step 5: Commit CLI support** + +```bash +git add src/tool_semantics/cli.py tests/test_cli.py +git commit -m "add capture provenance output" +``` + +### Task 3: Optional compare artifacts + +**Files:** +- Modify: `.github/actions/compare/action.yml` +- Test: `.github/actions/compare/action.yml` static assertions in a new `tests/test_github_action.py` + +**Interfaces:** +- Produces: Action input `upload-artifacts`, default `"false"`. +- Produces: Artifact named `tool-semantics-report` containing candidate snapshot, + `report.md`, and `report.json` when enabled. + +- [ ] **Step 1: Write a failing Action contract test** + +```python +def test_compare_action_can_upload_candidate_and_reports() -> None: + action = yaml.safe_load(Path(".github/actions/compare/action.yml").read_text()) + assert action["inputs"]["upload-artifacts"]["default"] == "false" + upload = next(step for step in action["runs"]["steps"] if step["name"] == "Upload report artifact") + assert "inputs.upload-artifacts == 'true'" in upload["if"] + assert "${{ inputs.candidate }}" in upload["with"]["path"] + assert "${{ steps.compare.outputs.report-path }}" in upload["with"]["path"] +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `pytest tests/test_github_action.py::test_compare_action_can_upload_candidate_and_reports -v` + +Expected: FAIL because the action has no artifact input or upload step. + +- [ ] **Step 3: Add the optional artifact input and upload step** + +```yaml + upload-artifacts: + description: Upload candidate snapshot and generated reports as a workflow artifact. + required: false + default: "false" +``` + +Add a final `actions/upload-artifact@v4` step named `Upload report artifact`: + +```yaml + - name: Upload report artifact + if: ${{ inputs.upload-artifacts == 'true' }} + uses: actions/upload-artifact@v4 + with: + name: tool-semantics-report + if-no-files-found: error + path: | + ${{ inputs.candidate }} + ${{ steps.compare.outputs.report-path }} + ${{ runner.temp }}/tool-semantics/report.json +``` + +- [ ] **Step 4: Run the Action contract test to verify it passes** + +Run: `pytest tests/test_github_action.py -v` + +Expected: PASS. + +- [ ] **Step 5: Commit Action artifact support** + +```bash +git add .github/actions/compare/action.yml tests/test_github_action.py +git commit -m "add optional compare artifacts" +``` + +### Task 4: Documentation and full verification + +**Files:** +- Modify: `README.md:127-151` +- Modify: `docs/github-action.md:35-82` +- Modify: `docs/architecture.md:34-62` + +**Interfaces:** +- Documents: Git-tracked baselines as the source of truth, optional provenance + sidecars, and diagnostic Action artifacts. + +- [ ] **Step 1: Document the baseline and provenance command** + +Add this README example: + +```bash +tool-semantics capture examples/github_server_v1.json \ + -o .tool-semantics/baselines/github.json \ + --provenance-output .tool-semantics/baselines/github.provenance.json +``` + +Explain that the snapshot is committed as the approved contract, while the +separate provenance file records capture context and digest without affecting +compatibility comparisons. + +- [ ] **Step 2: Document Action artifacts** + +Add `upload-artifacts: "true"` to the GitHub Action example and state that the +artifact includes the candidate snapshot plus Markdown and JSON reports. State +that artifacts are diagnostic and do not replace Git baselines. + +- [ ] **Step 3: Update architecture output labels** + +Add provenance sidecars and optional CI artifacts to the architecture output +description, while retaining JSON snapshots as the canonical compare input. + +- [ ] **Step 4: Run full verification** + +Run: + +```bash +python -m pytest --cov=tool_semantics --cov-report=term-missing +python -m ruff check . +python -m ruff format --check . +python -m mypy src +python -m build +``` + +Expected: all commands succeed. + +- [ ] **Step 5: Commit documentation and verification changes** + +```bash +git add README.md docs/github-action.md docs/architecture.md +git commit -m "document snapshot storage workflow" +``` + +## Plan Self-Review + +- Spec coverage: Git baselines are documented in Task 4; provenance sidecars and + secret handling are implemented in Tasks 1-2; optional CI artifacts are added + in Task 3; no database or object storage is introduced. +- Placeholder scan: no deferred implementation markers or undefined tasks remain. +- Type consistency: Task 1 defines `write_provenance`; Task 2 consumes that exact + function signature. Artifact input and output paths are defined before use. From 63054337d100b22561eab23ad33958d166aa929a Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:37:35 -0400 Subject: [PATCH 03/17] refine snapshot storage plan test --- .../plans/2026-08-13-snapshot-storage.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-snapshot-storage.md b/docs/superpowers/plans/2026-08-13-snapshot-storage.md index 33c3a65..fa30b93 100644 --- a/docs/superpowers/plans/2026-08-13-snapshot-storage.md +++ b/docs/superpowers/plans/2026-08-13-snapshot-storage.md @@ -194,12 +194,13 @@ git commit -m "add capture provenance output" ```python def test_compare_action_can_upload_candidate_and_reports() -> None: - action = yaml.safe_load(Path(".github/actions/compare/action.yml").read_text()) - assert action["inputs"]["upload-artifacts"]["default"] == "false" - upload = next(step for step in action["runs"]["steps"] if step["name"] == "Upload report artifact") - assert "inputs.upload-artifacts == 'true'" in upload["if"] - assert "${{ inputs.candidate }}" in upload["with"]["path"] - assert "${{ steps.compare.outputs.report-path }}" in upload["with"]["path"] + action = Path(".github/actions/compare/action.yml").read_text(encoding="utf-8") + assert 'upload-artifacts:' in action + assert 'default: "false"' in action + assert 'name: Upload report artifact' in action + assert "inputs.upload-artifacts == 'true'" in action + assert "${{ inputs.candidate }}" in action + assert "${{ steps.compare.outputs.report-path }}" in action ``` - [ ] **Step 2: Run the test to verify it fails** From 1574d540384ae35d264fa35651d6e219815b4ba0 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:40:22 -0400 Subject: [PATCH 04/17] add snapshot provenance sidecars --- src/tool_semantics/provenance.py | 62 ++++++++++++++++++++++++++++++++ tests/test_provenance.py | 44 +++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 src/tool_semantics/provenance.py create mode 100644 tests/test_provenance.py diff --git a/src/tool_semantics/provenance.py b/src/tool_semantics/provenance.py new file mode 100644 index 0000000..9d2bff9 --- /dev/null +++ b/src/tool_semantics/provenance.py @@ -0,0 +1,62 @@ +from __future__ import annotations + +import hashlib +import json +import re +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from tool_semantics.redact import redact_mapping + +_COMMAND_SECRET_OPTION_PATTERN = re.compile( + r"^--?(?:secret|token|password|api[_-]?key|authorization|credential|cookie)(?:=|$)", + re.IGNORECASE, +) +_ENVIRONMENT_KEY_PATTERN = re.compile(r"(?:^|_)(?:env|environment)(?:_|$)", re.IGNORECASE) +_REDACTED = "***REDACTED***" + + +def snapshot_sha256(snapshot_path: Path) -> str: + """Return the SHA-256 digest of a snapshot's exact on-disk bytes.""" + return hashlib.sha256(snapshot_path.read_bytes()).hexdigest() + + +def _redact_source(source: dict[str, Any]) -> dict[str, Any]: + redacted = redact_mapping(source) + for key, value in redacted.items(): + if _ENVIRONMENT_KEY_PATTERN.search(key): + redacted[key] = _REDACTED + elif key == "command" and isinstance(value, list): + redacted[key] = _redact_command_arguments(value) + return redacted + + +def _redact_command_arguments(command: list[Any]) -> list[Any]: + redacted = list(command) + for index, argument in enumerate(command): + if isinstance(argument, str) and _COMMAND_SECRET_OPTION_PATTERN.match(argument): + if "=" in argument: + redacted[index] = _REDACTED + elif index + 1 < len(redacted): + redacted[index + 1] = _REDACTED + return redacted + + +def write_provenance( + snapshot_path: Path, + output_path: Path, + source: dict[str, Any], + captured_at: datetime | None = None, +) -> None: + """Write deterministic, redacted provenance for an existing snapshot.""" + timestamp = captured_at or datetime.now(UTC) + payload = { + "provenance_version": "0.1", + "snapshot_path": str(snapshot_path), + "snapshot_sha256": snapshot_sha256(snapshot_path), + "captured_at": timestamp.isoformat(), + "source": _redact_source(source), + } + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") diff --git a/tests/test_provenance.py b/tests/test_provenance.py new file mode 100644 index 0000000..3d59652 --- /dev/null +++ b/tests/test_provenance.py @@ -0,0 +1,44 @@ +import hashlib +import json +from datetime import UTC, datetime +from pathlib import Path + +from tool_semantics.provenance import write_provenance + + +def test_write_provenance_hashes_snapshot_and_redacts_source(tmp_path: Path) -> None: + """Catch a sidecar that leaks source secrets or hashes transformed bytes.""" + snapshot = tmp_path / "snapshot.json" + snapshot.write_text('{"server_name":"demo"}\n', encoding="utf-8") + output = tmp_path / "provenance.json" + + write_provenance( + snapshot, + output, + { + "kind": "mcp-stdio", + "authorization": "Bearer secret", + "command": ["server", "--token=secret", "--api-key", "another-secret"], + }, + captured_at=datetime(2026, 8, 13, tzinfo=UTC), + ) + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["provenance_version"] == "0.1" + assert payload["snapshot_sha256"] == hashlib.sha256(snapshot.read_bytes()).hexdigest() + assert payload["source"]["authorization"] == "***REDACTED***" + assert payload["source"]["command"][1] == "***REDACTED***" + assert payload["source"]["command"][3] == "***REDACTED***" + assert payload["captured_at"] == "2026-08-13T00:00:00+00:00" + + +def test_write_provenance_never_serializes_environment_values(tmp_path: Path) -> None: + """Catch a provenance sidecar that preserves environment variable values.""" + snapshot = tmp_path / "snapshot.json" + snapshot.write_text("{}\n", encoding="utf-8") + output = tmp_path / "nested" / "provenance.json" + + write_provenance(snapshot, output, {"environment": {"API_TOKEN": "secret"}}) + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["source"]["environment"] == "***REDACTED***" From 11c75f3c64ef938865a7ad06d5da8011fe8ba446 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:43:26 -0400 Subject: [PATCH 05/17] harden provenance redaction --- src/tool_semantics/provenance.py | 31 +++++++++++++++++------ tests/test_provenance.py | 43 ++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 7 deletions(-) diff --git a/src/tool_semantics/provenance.py b/src/tool_semantics/provenance.py index 9d2bff9..19fe90b 100644 --- a/src/tool_semantics/provenance.py +++ b/src/tool_semantics/provenance.py @@ -13,6 +13,7 @@ r"^--?(?:secret|token|password|api[_-]?key|authorization|credential|cookie)(?:=|$)", re.IGNORECASE, ) +_COMMAND_HEADER_OPTION_PATTERN = re.compile(r"^(?:-H|--header)(?:=|$)", re.IGNORECASE) _ENVIRONMENT_KEY_PATTERN = re.compile(r"(?:^|_)(?:env|environment)(?:_|$)", re.IGNORECASE) _REDACTED = "***REDACTED***" @@ -24,18 +25,34 @@ def snapshot_sha256(snapshot_path: Path) -> str: def _redact_source(source: dict[str, Any]) -> dict[str, Any]: redacted = redact_mapping(source) - for key, value in redacted.items(): - if _ENVIRONMENT_KEY_PATTERN.search(key): - redacted[key] = _REDACTED - elif key == "command" and isinstance(value, list): - redacted[key] = _redact_command_arguments(value) - return redacted + return { + key: _redact_source_value(key, value) + for key, value in redacted.items() + } + + +def _redact_source_value(key: str, value: Any) -> Any: + if _ENVIRONMENT_KEY_PATTERN.search(key): + return _REDACTED + if key == "command" and isinstance(value, list): + return _redact_command_arguments(value) + if isinstance(value, dict): + return { + child_key: _redact_source_value(child_key, child) + for child_key, child in value.items() + } + if isinstance(value, list): + return [_redact_source_value(key, item) for item in value] + return value def _redact_command_arguments(command: list[Any]) -> list[Any]: redacted = list(command) for index, argument in enumerate(command): - if isinstance(argument, str) and _COMMAND_SECRET_OPTION_PATTERN.match(argument): + if isinstance(argument, str) and ( + _COMMAND_SECRET_OPTION_PATTERN.match(argument) + or _COMMAND_HEADER_OPTION_PATTERN.match(argument) + ): if "=" in argument: redacted[index] = _REDACTED elif index + 1 < len(redacted): diff --git a/tests/test_provenance.py b/tests/test_provenance.py index 3d59652..adfad1d 100644 --- a/tests/test_provenance.py +++ b/tests/test_provenance.py @@ -42,3 +42,46 @@ def test_write_provenance_never_serializes_environment_values(tmp_path: Path) -> payload = json.loads(output.read_text(encoding="utf-8")) assert payload["source"]["environment"] == "***REDACTED***" + + +def test_write_provenance_never_serializes_nested_environment_values(tmp_path: Path) -> None: + """Catch nested source configuration that exposes environment names or values.""" + snapshot = tmp_path / "snapshot.json" + snapshot.write_text("{}\n", encoding="utf-8") + output = tmp_path / "provenance.json" + + write_provenance( + snapshot, + output, + {"connection": {"environment": {"DATABASE_URL": "postgres://user:secret@host/db"}}}, + ) + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["source"]["connection"]["environment"] == "***REDACTED***" + + +def test_write_provenance_redacts_command_header_values(tmp_path: Path) -> None: + """Catch command headers that would serialize authorization credentials.""" + snapshot = tmp_path / "snapshot.json" + snapshot.write_text("{}\n", encoding="utf-8") + output = tmp_path / "provenance.json" + + write_provenance( + snapshot, + output, + { + "command": [ + "server", + "-H", + "Authorization: Bearer first-secret", + "--header=Authorization: Bearer second-secret", + "--header", + "Authorization: Bearer third-secret", + ] + }, + ) + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["source"]["command"][2] == "***REDACTED***" + assert payload["source"]["command"][3] == "***REDACTED***" + assert payload["source"]["command"][5] == "***REDACTED***" From 6bc7a1992455f66dd59c5bd8ec5d981537b78978 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:46:33 -0400 Subject: [PATCH 06/17] add capture provenance output --- src/tool_semantics/cli.py | 25 ++++++++++++++++++-- tests/test_cli.py | 50 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/src/tool_semantics/cli.py b/src/tool_semantics/cli.py index ae9d95c..12700f1 100644 --- a/src/tool_semantics/cli.py +++ b/src/tool_semantics/cli.py @@ -13,6 +13,7 @@ from tool_semantics.diff import compare_snapshots from tool_semantics.mcp_capture import McpCaptureError, capture_mcp_sse, capture_mcp_stdio from tool_semantics.policy import policy_from_name +from tool_semantics.provenance import write_provenance from tool_semantics.report import render_markdown, severity_style from tool_semantics.scanner import ManifestError, capture_manifest, read_snapshot, write_snapshot @@ -65,6 +66,10 @@ def capture( help="Where to write the normalized snapshot JSON.", ), ] = Path(".tool-semantics/snapshot.json"), + provenance_output: Annotated[ + Path | None, + typer.Option("--provenance-output", help="Write capture provenance JSON separately."), + ] = None, verbose: Annotated[ bool, typer.Option( @@ -79,7 +84,13 @@ def capture( try: snapshot = capture_manifest(manifest) write_snapshot(snapshot, output) - except ManifestError as exc: + if provenance_output: + write_provenance( + output, + provenance_output, + {"kind": "manifest", "location": str(manifest)}, + ) + except (ManifestError, OSError) as exc: console.print(f"[red]Capture failed:[/red] {exc}") raise typer.Exit(code=2) from exc _log_verbose( @@ -109,6 +120,10 @@ def capture_mcp( help="Where to write the normalized snapshot JSON.", ), ] = Path(".tool-semantics/snapshot.json"), + provenance_output: Annotated[ + Path | None, + typer.Option("--provenance-output", help="Write capture provenance JSON separately."), + ] = None, sse_url: Annotated[ str | None, typer.Option("--sse", help="SSE MCP endpoint URL (not implemented yet)."), @@ -144,7 +159,13 @@ def capture_mcp( redact=not no_redact, ) write_snapshot(snapshot, output) - except (McpCaptureError, ManifestError) as exc: + if provenance_output and command: + write_provenance( + output, + provenance_output, + {"kind": "mcp-stdio", "command": command}, + ) + except (McpCaptureError, ManifestError, OSError) as exc: console.print(f"[red]MCP capture failed:[/red] {exc}") raise typer.Exit(code=2) from exc _log_verbose( diff --git a/tests/test_cli.py b/tests/test_cli.py index 16eacde..f926c16 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,3 +1,4 @@ +import json from pathlib import Path from typer.testing import CliRunner @@ -5,6 +6,7 @@ from tool_semantics.cli import app runner = CliRunner() +FIXTURE = Path(__file__).parent / "fixtures" / "fake_mcp_server.py" def test_capture_verbose_writes_stderr(tmp_path: Path) -> None: @@ -24,6 +26,54 @@ def test_capture_verbose_writes_stderr(tmp_path: Path) -> None: assert output.is_file() +def test_capture_writes_requested_provenance(tmp_path: Path) -> None: + snapshot = tmp_path / "snapshot.json" + provenance = tmp_path / "snapshot.provenance.json" + result = runner.invoke( + app, + [ + "capture", + "examples/github_server_v1.json", + "-o", + str(snapshot), + "--provenance-output", + str(provenance), + ], + ) + assert result.exit_code == 0 + payload = json.loads(provenance.read_text(encoding="utf-8")) + assert payload["source"] == { + "kind": "manifest", + "location": "examples/github_server_v1.json", + } + + +def test_capture_mcp_writes_redacted_requested_provenance(tmp_path: Path) -> None: + snapshot = tmp_path / "snapshot.json" + provenance = tmp_path / "snapshot.provenance.json" + result = runner.invoke( + app, + [ + "capture-mcp", + "-o", + str(snapshot), + "--provenance-output", + str(provenance), + "--", + "python", + str(FIXTURE), + "--token", + "command-secret", + ], + ) + assert result.exit_code == 0 + payload = json.loads(provenance.read_text(encoding="utf-8")) + assert payload["source"] == { + "kind": "mcp-stdio", + "command": ["python", str(FIXTURE), "--token", "***REDACTED***"], + } + + def test_compare_verbose_and_config_ignore(tmp_path: Path) -> None: baseline = tmp_path / "v1.json" candidate = tmp_path / "v2.json" From f1ab288879d9266d7e70c2e062f4c0ab0fc825ba Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:49:03 -0400 Subject: [PATCH 07/17] preserve snapshot write errors --- src/tool_semantics/cli.py | 24 ++++++++++++++++-------- tests/test_cli.py | 10 ++++++++++ 2 files changed, 26 insertions(+), 8 deletions(-) diff --git a/src/tool_semantics/cli.py b/src/tool_semantics/cli.py index 12700f1..b929810 100644 --- a/src/tool_semantics/cli.py +++ b/src/tool_semantics/cli.py @@ -84,15 +84,19 @@ def capture( try: snapshot = capture_manifest(manifest) write_snapshot(snapshot, output) - if provenance_output: + except ManifestError as exc: + console.print(f"[red]Capture failed:[/red] {exc}") + raise typer.Exit(code=2) from exc + if provenance_output: + try: write_provenance( output, provenance_output, {"kind": "manifest", "location": str(manifest)}, ) - except (ManifestError, OSError) as exc: - console.print(f"[red]Capture failed:[/red] {exc}") - raise typer.Exit(code=2) from exc + except OSError as exc: + console.print(f"[red]Capture failed:[/red] {exc}") + raise typer.Exit(code=2) from exc _log_verbose( verbose, f"Wrote snapshot {output.resolve()} with {len(snapshot.tools)} tools " @@ -159,15 +163,19 @@ def capture_mcp( redact=not no_redact, ) write_snapshot(snapshot, output) - if provenance_output and command: + except (McpCaptureError, ManifestError) as exc: + console.print(f"[red]MCP capture failed:[/red] {exc}") + raise typer.Exit(code=2) from exc + if provenance_output and command: + try: write_provenance( output, provenance_output, {"kind": "mcp-stdio", "command": command}, ) - except (McpCaptureError, ManifestError, OSError) as exc: - console.print(f"[red]MCP capture failed:[/red] {exc}") - raise typer.Exit(code=2) from exc + except OSError as exc: + console.print(f"[red]MCP capture failed:[/red] {exc}") + raise typer.Exit(code=2) from exc _log_verbose( verbose, ( diff --git a/tests/test_cli.py b/tests/test_cli.py index f926c16..1a18007 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -48,6 +48,16 @@ def test_capture_writes_requested_provenance(tmp_path: Path) -> None: } +def test_capture_preserves_snapshot_write_os_error_without_provenance(tmp_path: Path) -> None: + result = runner.invoke( + app, + ["capture", "examples/github_server_v1.json", "-o", str(tmp_path)], + ) + assert result.exit_code == 1 + assert isinstance(result.exception, IsADirectoryError) + assert "Capture failed:" not in result.stdout + + def test_capture_mcp_writes_redacted_requested_provenance(tmp_path: Path) -> None: snapshot = tmp_path / "snapshot.json" provenance = tmp_path / "snapshot.provenance.json" From 38a99cf2662939e30cb11c4edf32d62260e8af88 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:52:15 -0400 Subject: [PATCH 08/17] add optional compare artifacts --- .github/actions/compare/action.yml | 15 +++++++++++++++ tests/test_github_action.py | 12 ++++++++++++ 2 files changed, 27 insertions(+) create mode 100644 tests/test_github_action.py diff --git a/.github/actions/compare/action.yml b/.github/actions/compare/action.yml index c9b7289..93ab8db 100644 --- a/.github/actions/compare/action.yml +++ b/.github/actions/compare/action.yml @@ -21,6 +21,10 @@ inputs: description: If true, post or update a PR comment with the Markdown report. required: false default: "true" + upload-artifacts: + description: Upload candidate snapshot and generated reports as a workflow artifact. + required: false + default: "false" fail-on-breaking: description: Deprecated alias for policy=compatible when true. required: false @@ -155,3 +159,14 @@ runs: body, }); } + + - name: Upload report artifact + if: ${{ inputs.upload-artifacts == 'true' }} + uses: actions/upload-artifact@v4 + with: + name: tool-semantics-report + if-no-files-found: error + path: | + ${{ inputs.candidate }} + ${{ steps.compare.outputs.report-path }} + ${{ runner.temp }}/tool-semantics/report.json diff --git a/tests/test_github_action.py b/tests/test_github_action.py new file mode 100644 index 0000000..c390690 --- /dev/null +++ b/tests/test_github_action.py @@ -0,0 +1,12 @@ +from pathlib import Path + + +def test_compare_action_can_upload_candidate_and_reports() -> None: + action = Path(".github/actions/compare/action.yml").read_text(encoding="utf-8") + + assert "upload-artifacts:" in action + assert 'default: "false"' in action + assert "name: Upload report artifact" in action + assert "inputs.upload-artifacts == 'true'" in action + assert "${{ inputs.candidate }}" in action + assert "${{ steps.compare.outputs.report-path }}" in action From 098761c17c91ee36887226790ea0395da279b1eb Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:55:16 -0400 Subject: [PATCH 09/17] fix compare artifact upload paths --- .github/actions/compare/action.yml | 21 +++++++++++++++++++-- tests/test_github_action.py | 13 +++++++++++-- 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/.github/actions/compare/action.yml b/.github/actions/compare/action.yml index 93ab8db..53d3a0c 100644 --- a/.github/actions/compare/action.yml +++ b/.github/actions/compare/action.yml @@ -160,13 +160,30 @@ runs: }); } + - name: Resolve artifact paths + id: artifact-paths + if: ${{ always() && inputs.upload-artifacts == 'true' }} + shell: bash + working-directory: ${{ inputs.working-directory }} + env: + CANDIDATE: ${{ inputs.candidate }} + run: | + python - <<'PY' + import os + from pathlib import Path + + candidate = Path(os.environ["CANDIDATE"]).resolve() + with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output: + print(f"candidate-path={candidate}", file=output) + PY + - name: Upload report artifact - if: ${{ inputs.upload-artifacts == 'true' }} + if: ${{ always() && inputs.upload-artifacts == 'true' }} uses: actions/upload-artifact@v4 with: name: tool-semantics-report if-no-files-found: error path: | - ${{ inputs.candidate }} + ${{ steps.artifact-paths.outputs.candidate-path }} ${{ steps.compare.outputs.report-path }} ${{ runner.temp }}/tool-semantics/report.json diff --git a/tests/test_github_action.py b/tests/test_github_action.py index c390690..6cfbdda 100644 --- a/tests/test_github_action.py +++ b/tests/test_github_action.py @@ -7,6 +7,15 @@ def test_compare_action_can_upload_candidate_and_reports() -> None: assert "upload-artifacts:" in action assert 'default: "false"' in action assert "name: Upload report artifact" in action - assert "inputs.upload-artifacts == 'true'" in action - assert "${{ inputs.candidate }}" in action + assert "if: ${{ always() && inputs.upload-artifacts == 'true' }}" in action + assert "${{ steps.artifact-paths.outputs.candidate-path }}" in action assert "${{ steps.compare.outputs.report-path }}" in action + + +def test_compare_action_resolves_artifact_candidate_from_working_directory() -> None: + action = Path(".github/actions/compare/action.yml").read_text(encoding="utf-8") + + assert "name: Resolve artifact paths" in action + assert "id: artifact-paths" in action + assert "working-directory: ${{ inputs.working-directory }}" in action + assert 'Path(os.environ["CANDIDATE"]).resolve()' in action From c9f66c6ad7fefbfeb528acfdbcfaed915934eadc Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:58:06 -0400 Subject: [PATCH 10/17] document snapshot storage workflow --- README.md | 22 ++++++++++++++++++++-- docs/architecture.md | 17 +++++++++++++++-- docs/github-action.md | 10 ++++++++++ 3 files changed, 45 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index dc6d6e8..12d9774 100644 --- a/README.md +++ b/README.md @@ -134,8 +134,10 @@ print("compatible:", report.is_compatible) ```bash tool-semantics --version -tool-semantics capture [-o .tool-semantics/snapshot.json] [-v] -tool-semantics capture-mcp -o snap.json -- python my_mcp_server.py +tool-semantics capture [-o .tool-semantics/snapshot.json] \ + [--provenance-output snapshot.provenance.json] [-v] +tool-semantics capture-mcp -o snap.json \ + [--provenance-output snap.provenance.json] -- python my_mcp_server.py tool-semantics compare \ [--json-output report.json] \ [--markdown-output report.md] \ @@ -147,6 +149,22 @@ tool-semantics compare \ - `--config` loads ignore rules; if omitted, `.tool-semantics.toml` in the cwd is used when present. - `capture-mcp` speaks MCP JSON-RPC over stdio; secrets-like keys are redacted by default. +### Approved baselines and provenance + +Capture the approved interface into a Git-tracked baseline. The snapshot is the +contract that `compare` uses; review and commit it when an interface change is +intentional. + +```bash +tool-semantics capture examples/github_server_v1.json \ + -o .tool-semantics/baselines/github.json \ + --provenance-output .tool-semantics/baselines/github.provenance.json +``` + +The optional provenance sidecar records capture context and a digest of the +snapshot. It is separate from the snapshot and never affects compatibility +comparisons. + JSON reports include `changes`, `is_compatible`, and `counts` by severity. Change-code catalog: [docs/change-codes.md](docs/change-codes.md). Ignore-config schema: [docs/config.md](docs/config.md). diff --git a/docs/architecture.md b/docs/architecture.md index 10c26b3..7ec83d1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -22,8 +22,11 @@ flowchart TB subgraph outputs CLI[Rich CLI table] MD[Markdown report] - JSON[JSON artifact] + JSON[JSON compatibility report] + SNAP[Snapshot JSON — canonical compare input] + PROV[Optional provenance sidecar] CI[Exit codes / GitHub Action] + ART[Optional CI diagnostic artifact] end M --> S L --> S @@ -37,6 +40,9 @@ flowchart TB R --> MD R --> JSON R --> CI + N --> SNAP + SNAP -.-> PROV + CI -.-> ART ``` ## Components @@ -53,7 +59,14 @@ flowchart TB | **Migration adapters** (`adapters.py`) | Tool aliases, argument/enum maps, output wrappers | | **Report** (`report.py`) | Human-readable Markdown / styling helpers | | **CLI** (`cli.py`) | `capture`, `capture-mcp`, `compare`, and related entry points | -| **GitHub Action** (`.github/actions/compare`) | CI compare + optional PR comment | +| **GitHub Action** (`.github/actions/compare`) | CI compare + optional PR comment; can upload candidate and reports as a diagnostic artifact | + +Snapshots are the canonical JSON inputs to compatibility comparisons and are +normally committed to Git as approved baselines. Capture can also write an +optional provenance sidecar with capture context and the snapshot digest; it is +not part of the snapshot schema or compare input. Optional CI artifacts contain +the candidate snapshot and generated reports for diagnosis only, not a +replacement for Git-tracked baselines. ### Still planned diff --git a/docs/github-action.md b/docs/github-action.md index e1076d1..75071ef 100644 --- a/docs/github-action.md +++ b/docs/github-action.md @@ -51,6 +51,7 @@ jobs: config: .tool-semantics.toml policy: strict comment-on-pr: "true" + upload-artifacts: "true" ``` ## Inputs @@ -62,6 +63,7 @@ jobs: | `config` | no | `""` | Optional ignore/policy config path | | `policy` | no | `""` | `compatible` / `strict` / `critical-only` / `permissive` | | `comment-on-pr` | no | `true` | Upsert a PR comment with the report | +| `upload-artifacts` | no | `false` | Upload the candidate snapshot and generated reports as a workflow artifact | | `fail-on-breaking` | no | `true` | Legacy; `false` maps to `permissive` when `policy` unset | | `working-directory` | no | `.` | Directory for install/compare | @@ -73,6 +75,14 @@ jobs: | `policy-failed` | `true` if the selected release policy failed | | `report-path` | Path to the Markdown report artifact | +## Baselines and diagnostic artifacts + +Commit the approved baseline snapshot to Git and treat it as the compatibility +contract. When `upload-artifacts: "true"`, the Action uploads the candidate +snapshot plus the generated Markdown and JSON reports in the +`tool-semantics-report` artifact. These files help diagnose a CI run; they do +not replace the Git-tracked baseline. + ## Permissions When `comment-on-pr` is enabled on `pull_request` events, the workflow needs From 7f8fcb51355009dd5796dc001b82b90c6fb2d1d0 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 21:00:03 -0400 Subject: [PATCH 11/17] document committed action baselines --- docs/github-action.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/github-action.md b/docs/github-action.md index 75071ef..8659998 100644 --- a/docs/github-action.md +++ b/docs/github-action.md @@ -39,16 +39,16 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - name: Capture baseline and candidate + # .tool-semantics/baselines/github.json was captured, reviewed, and + # committed before this pull request. + - name: Capture candidate snapshot run: | pip install "tool-semantics==0.2.0" - tool-semantics capture manifests/baseline.json -o .tool-semantics/baseline.json tool-semantics capture manifests/candidate.json -o .tool-semantics/candidate.json - uses: askmy-stack/tool-semantics/.github/actions/compare@v0.2.0 with: - baseline: .tool-semantics/baseline.json + baseline: .tool-semantics/baselines/github.json candidate: .tool-semantics/candidate.json - config: .tool-semantics.toml policy: strict comment-on-pr: "true" upload-artifacts: "true" @@ -77,11 +77,15 @@ jobs: ## Baselines and diagnostic artifacts -Commit the approved baseline snapshot to Git and treat it as the compatibility -contract. When `upload-artifacts: "true"`, the Action uploads the candidate -snapshot plus the generated Markdown and JSON reports in the -`tool-semantics-report` artifact. These files help diagnose a CI run; they do -not replace the Git-tracked baseline. +Capture, review, and commit the approved baseline snapshot to Git before it is +used in CI; treat that file as the compatibility contract. A pull-request job +should capture only its candidate and compare it to the committed baseline, not +recreate the baseline during the run. To intentionally update the contract, +capture a new baseline, review its diff, and commit that snapshot change. + +When `upload-artifacts: "true"`, the Action uploads the candidate snapshot plus +the generated Markdown and JSON reports in the `tool-semantics-report` artifact. +These files help diagnose a CI run; they do not replace the Git-tracked baseline. ## Permissions From 657f9b44ba901722d30f43a5e6beee865799b6b8 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 21:05:15 -0400 Subject: [PATCH 12/17] fix snapshot storage review findings --- .github/actions/compare/action.yml | 16 +++++++ .../plans/2026-08-13-snapshot-storage.md | 14 +++--- src/tool_semantics/cli.py | 11 +++++ src/tool_semantics/provenance.py | 25 ++++++----- tests/test_cli.py | 45 +++++++++++++++++++ tests/test_github_action.py | 11 +++++ tests/test_provenance.py | 28 ++++++++++++ 7 files changed, 135 insertions(+), 15 deletions(-) diff --git a/.github/actions/compare/action.yml b/.github/actions/compare/action.yml index 53d3a0c..8aab221 100644 --- a/.github/actions/compare/action.yml +++ b/.github/actions/compare/action.yml @@ -177,6 +177,22 @@ runs: print(f"candidate-path={candidate}", file=output) PY + # All three files are required: an incomplete diagnostic artifact is misleading. + # This runs after a compare failure as well when upload was explicitly requested. + - name: Validate artifact paths + if: ${{ always() && inputs.upload-artifacts == 'true' }} + shell: bash + run: | + for path in \ + "${{ steps.artifact-paths.outputs.candidate-path }}" \ + "${{ steps.compare.outputs.report-path }}" \ + "${{ runner.temp }}/tool-semantics/report.json"; do + if [ ! -f "$path" ]; then + echo "Missing required artifact file: $path" >&2 + exit 1 + fi + done + - name: Upload report artifact if: ${{ always() && inputs.upload-artifacts == 'true' }} uses: actions/upload-artifact@v4 diff --git a/docs/superpowers/plans/2026-08-13-snapshot-storage.md b/docs/superpowers/plans/2026-08-13-snapshot-storage.md index fa30b93..5c260f8 100644 --- a/docs/superpowers/plans/2026-08-13-snapshot-storage.md +++ b/docs/superpowers/plans/2026-08-13-snapshot-storage.md @@ -134,8 +134,12 @@ def test_capture_writes_requested_provenance(tmp_path: Path) -> None: result = runner.invoke( app, [ - "capture", "examples/github_server_v1.json", "-o", str(snapshot), - "--provenance-output", str(provenance), + "capture", + "examples/github_server_v1.json", + "-o", + str(snapshot), + "--provenance-output", + str(provenance), ], ) assert result.exit_code == 0 @@ -158,7 +162,7 @@ Expected: FAIL because `--provenance-output` is not recognized. provenance_output: Annotated[ Path | None, typer.Option("--provenance-output", help="Write capture provenance JSON separately."), -] = None, +] = (None,) ``` After `write_snapshot(snapshot, output)`, call `write_provenance` only when the @@ -195,9 +199,9 @@ git commit -m "add capture provenance output" ```python def test_compare_action_can_upload_candidate_and_reports() -> None: action = Path(".github/actions/compare/action.yml").read_text(encoding="utf-8") - assert 'upload-artifacts:' in action + assert "upload-artifacts:" in action assert 'default: "false"' in action - assert 'name: Upload report artifact' in action + assert "name: Upload report artifact" in action assert "inputs.upload-artifacts == 'true'" in action assert "${{ inputs.candidate }}" in action assert "${{ steps.compare.outputs.report-path }}" in action diff --git a/src/tool_semantics/cli.py b/src/tool_semantics/cli.py index b929810..2f3ff36 100644 --- a/src/tool_semantics/cli.py +++ b/src/tool_semantics/cli.py @@ -48,6 +48,15 @@ def _log_verbose(verbose: bool, message: str) -> None: err_console.print(f"[dim]{message}[/dim]") +def _reject_equivalent_output_paths(snapshot_path: Path, provenance_path: Path | None) -> None: + """Prevent a provenance sidecar from replacing the newly captured snapshot.""" + if provenance_path is not None and snapshot_path.resolve() == provenance_path.resolve(): + console.print( + "[red]Capture failed:[/red] Snapshot and provenance output paths must be different." + ) + raise typer.Exit(code=2) + + @app.command() def capture( manifest: Annotated[ @@ -80,6 +89,7 @@ def capture( ] = False, ) -> None: """Normalize a JSON tool manifest into a Tool-Semantics snapshot.""" + _reject_equivalent_output_paths(output, provenance_output) _log_verbose(verbose, f"Reading manifest {manifest.resolve()}") try: snapshot = capture_manifest(manifest) @@ -146,6 +156,7 @@ def capture_mcp( ] = False, ) -> None: """Capture a live MCP server over stdio (or attempt SSE).""" + _reject_equivalent_output_paths(output, provenance_output) try: if sse_url: snapshot = capture_mcp_sse(sse_url) diff --git a/src/tool_semantics/provenance.py b/src/tool_semantics/provenance.py index 19fe90b..9809e71 100644 --- a/src/tool_semantics/provenance.py +++ b/src/tool_semantics/provenance.py @@ -10,11 +10,16 @@ from tool_semantics.redact import redact_mapping _COMMAND_SECRET_OPTION_PATTERN = re.compile( - r"^--?(?:secret|token|password|api[_-]?key|authorization|credential|cookie)(?:=|$)", + r"^--?(?:[a-z0-9_-]*(?:secret|token|password|api[_-]?key|authorization|credential|cookie)|auth|bearer)(?:=|$)", re.IGNORECASE, ) _COMMAND_HEADER_OPTION_PATTERN = re.compile(r"^(?:-H|--header)(?:=|$)", re.IGNORECASE) _ENVIRONMENT_KEY_PATTERN = re.compile(r"(?:^|_)(?:env|environment)(?:_|$)", re.IGNORECASE) +_ENVIRONMENT_ASSIGNMENT_PATTERN = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)=", re.IGNORECASE) +_SECRET_NAME_PATTERN = re.compile( + r"secret|token|password|api[_-]?key|authorization|credential|cookie", + re.IGNORECASE, +) _REDACTED = "***REDACTED***" @@ -25,10 +30,7 @@ def snapshot_sha256(snapshot_path: Path) -> str: def _redact_source(source: dict[str, Any]) -> dict[str, Any]: redacted = redact_mapping(source) - return { - key: _redact_source_value(key, value) - for key, value in redacted.items() - } + return {key: _redact_source_value(key, value) for key, value in redacted.items()} def _redact_source_value(key: str, value: Any) -> Any: @@ -38,8 +40,7 @@ def _redact_source_value(key: str, value: Any) -> Any: return _redact_command_arguments(value) if isinstance(value, dict): return { - child_key: _redact_source_value(child_key, child) - for child_key, child in value.items() + child_key: _redact_source_value(child_key, child) for child_key, child in value.items() } if isinstance(value, list): return [_redact_source_value(key, item) for item in value] @@ -49,9 +50,13 @@ def _redact_source_value(key: str, value: Any) -> Any: def _redact_command_arguments(command: list[Any]) -> list[Any]: redacted = list(command) for index, argument in enumerate(command): - if isinstance(argument, str) and ( - _COMMAND_SECRET_OPTION_PATTERN.match(argument) - or _COMMAND_HEADER_OPTION_PATTERN.match(argument) + if not isinstance(argument, str): + continue + environment_assignment = _ENVIRONMENT_ASSIGNMENT_PATTERN.match(argument) + if environment_assignment and _SECRET_NAME_PATTERN.search(environment_assignment.group(1)): + redacted[index] = _REDACTED + elif _COMMAND_SECRET_OPTION_PATTERN.match(argument) or _COMMAND_HEADER_OPTION_PATTERN.match( + argument ): if "=" in argument: redacted[index] = _REDACTED diff --git a/tests/test_cli.py b/tests/test_cli.py index 1a18007..99c8291 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -48,6 +48,26 @@ def test_capture_writes_requested_provenance(tmp_path: Path) -> None: } +def test_capture_rejects_provenance_path_that_would_overwrite_snapshot(tmp_path: Path) -> None: + snapshot = tmp_path / "snapshot.json" + snapshot.write_text("original snapshot\n", encoding="utf-8") + result = runner.invoke( + app, + [ + "capture", + "examples/github_server_v1.json", + "-o", + str(snapshot), + "--provenance-output", + str(snapshot), + ], + ) + + assert result.exit_code == 2 + assert snapshot.read_text(encoding="utf-8") == "original snapshot\n" + assert "must be different" in result.stdout + + def test_capture_preserves_snapshot_write_os_error_without_provenance(tmp_path: Path) -> None: result = runner.invoke( app, @@ -84,6 +104,31 @@ def test_capture_mcp_writes_redacted_requested_provenance(tmp_path: Path) -> Non } +def test_capture_mcp_rejects_equivalent_provenance_path_before_snapshot_write( + tmp_path: Path, +) -> None: + snapshot = tmp_path / "snapshot.json" + snapshot.write_text("original snapshot\n", encoding="utf-8") + equivalent_path = tmp_path / "." / "snapshot.json" + result = runner.invoke( + app, + [ + "capture-mcp", + "-o", + str(snapshot), + "--provenance-output", + str(equivalent_path), + "--", + "python", + str(FIXTURE), + ], + ) + + assert result.exit_code == 2 + assert snapshot.read_text(encoding="utf-8") == "original snapshot\n" + assert "must be different" in result.stdout + + def test_compare_verbose_and_config_ignore(tmp_path: Path) -> None: baseline = tmp_path / "v1.json" candidate = tmp_path / "v2.json" diff --git a/tests/test_github_action.py b/tests/test_github_action.py index 6cfbdda..3cae0d1 100644 --- a/tests/test_github_action.py +++ b/tests/test_github_action.py @@ -19,3 +19,14 @@ def test_compare_action_resolves_artifact_candidate_from_working_directory() -> assert "id: artifact-paths" in action assert "working-directory: ${{ inputs.working-directory }}" in action assert 'Path(os.environ["CANDIDATE"]).resolve()' in action + + +def test_compare_action_requires_every_requested_artifact_file() -> None: + action = Path(".github/actions/compare/action.yml").read_text(encoding="utf-8") + + assert "name: Validate artifact paths" in action + assert "for path in" in action + assert '"${{ steps.artifact-paths.outputs.candidate-path }}"' in action + assert '"${{ steps.compare.outputs.report-path }}"' in action + assert '"${{ runner.temp }}/tool-semantics/report.json"' in action + assert "Missing required artifact file: $path" in action diff --git a/tests/test_provenance.py b/tests/test_provenance.py index adfad1d..e1aa03d 100644 --- a/tests/test_provenance.py +++ b/tests/test_provenance.py @@ -85,3 +85,31 @@ def test_write_provenance_redacts_command_header_values(tmp_path: Path) -> None: assert payload["source"]["command"][2] == "***REDACTED***" assert payload["source"]["command"][3] == "***REDACTED***" assert payload["source"]["command"][5] == "***REDACTED***" + + +def test_write_provenance_redacts_auth_options_and_environment_assignments(tmp_path: Path) -> None: + """Catch credential-bearing command forms that bypass snapshot redaction.""" + snapshot = tmp_path / "snapshot.json" + snapshot.write_text("{}\n", encoding="utf-8") + output = tmp_path / "provenance.json" + + write_provenance( + snapshot, + output, + { + "command": [ + "API_TOKEN=secret", + "server", + "--auth=secret", + "--bearer-token=secret", + ] + }, + ) + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["source"]["command"] == [ + "***REDACTED***", + "server", + "***REDACTED***", + "***REDACTED***", + ] From 2efc54413d75ac30411b0beaa36c5d65d7bda210 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 21:05:40 -0400 Subject: [PATCH 13/17] add snapshot storage final fix report --- .../final-fix-report.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 .superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md diff --git a/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md b/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md new file mode 100644 index 0000000..28b29a5 --- /dev/null +++ b/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md @@ -0,0 +1,29 @@ +# Snapshot storage final-fix report + +## Changes + +- Provenance command sanitization now redacts secret-like option names including + `--auth=…` and `--bearer-token=…`, plus secret-bearing environment assignments + such as `API_TOKEN=…`. This provenance-specific protection remains active even + when capture uses `--no-redact`. +- Capture and capture-mcp reject equal or equivalent snapshot/provenance paths + before writing, preserving an existing snapshot. +- The compare Action validates that the candidate, Markdown report, and JSON + report all exist before its explicitly opted-in artifact upload. Candidate path + resolution continues to use the requested working directory. +- Applied the exact Ruff formatting changes required in the snapshot-storage + plan, provenance module, and Action regression test. + +## Verification + +- `python -m pytest --cov=tool_semantics --cov-report=term-missing`: 54 passed; + total coverage 88%. +- `ruff check .`: passed. +- `ruff format --check .`: passed (41 files already formatted). +- `mypy src`: passed with no issues in 14 source files. +- `python -m build`: passed; built sdist and wheel. +- `git diff --check`: passed. + +## Implementation commit + +`657f9b44ba901722d30f43a5e6beee865799b6b8` (`fix snapshot storage review findings`) From e4729635e0fd961466fd56523a8e05cff4a5a6fd Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 21:07:46 -0400 Subject: [PATCH 14/17] harden snapshot artifact and provenance checks --- .github/actions/compare/action.yml | 3 ++- src/tool_semantics/provenance.py | 6 +----- tests/test_github_action.py | 5 +++++ tests/test_provenance.py | 4 ++++ 4 files changed, 12 insertions(+), 6 deletions(-) diff --git a/.github/actions/compare/action.yml b/.github/actions/compare/action.yml index 8aab221..d2a9d44 100644 --- a/.github/actions/compare/action.yml +++ b/.github/actions/compare/action.yml @@ -180,6 +180,7 @@ runs: # All three files are required: an incomplete diagnostic artifact is misleading. # This runs after a compare failure as well when upload was explicitly requested. - name: Validate artifact paths + id: validate-artifact-paths if: ${{ always() && inputs.upload-artifacts == 'true' }} shell: bash run: | @@ -194,7 +195,7 @@ runs: done - name: Upload report artifact - if: ${{ always() && inputs.upload-artifacts == 'true' }} + if: ${{ always() && inputs.upload-artifacts == 'true' && steps.validate-artifact-paths.outcome == 'success' }} uses: actions/upload-artifact@v4 with: name: tool-semantics-report diff --git a/src/tool_semantics/provenance.py b/src/tool_semantics/provenance.py index 9809e71..b664a56 100644 --- a/src/tool_semantics/provenance.py +++ b/src/tool_semantics/provenance.py @@ -16,10 +16,6 @@ _COMMAND_HEADER_OPTION_PATTERN = re.compile(r"^(?:-H|--header)(?:=|$)", re.IGNORECASE) _ENVIRONMENT_KEY_PATTERN = re.compile(r"(?:^|_)(?:env|environment)(?:_|$)", re.IGNORECASE) _ENVIRONMENT_ASSIGNMENT_PATTERN = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)=", re.IGNORECASE) -_SECRET_NAME_PATTERN = re.compile( - r"secret|token|password|api[_-]?key|authorization|credential|cookie", - re.IGNORECASE, -) _REDACTED = "***REDACTED***" @@ -53,7 +49,7 @@ def _redact_command_arguments(command: list[Any]) -> list[Any]: if not isinstance(argument, str): continue environment_assignment = _ENVIRONMENT_ASSIGNMENT_PATTERN.match(argument) - if environment_assignment and _SECRET_NAME_PATTERN.search(environment_assignment.group(1)): + if environment_assignment: redacted[index] = _REDACTED elif _COMMAND_SECRET_OPTION_PATTERN.match(argument) or _COMMAND_HEADER_OPTION_PATTERN.match( argument diff --git a/tests/test_github_action.py b/tests/test_github_action.py index 3cae0d1..b6a508e 100644 --- a/tests/test_github_action.py +++ b/tests/test_github_action.py @@ -30,3 +30,8 @@ def test_compare_action_requires_every_requested_artifact_file() -> None: assert '"${{ steps.compare.outputs.report-path }}"' in action assert '"${{ runner.temp }}/tool-semantics/report.json"' in action assert "Missing required artifact file: $path" in action + assert "id: validate-artifact-paths" in action + assert ( + "if: ${{ always() && inputs.upload-artifacts == 'true' && " + "steps.validate-artifact-paths.outcome == 'success' }}" + ) in action diff --git a/tests/test_provenance.py b/tests/test_provenance.py index e1aa03d..912ae77 100644 --- a/tests/test_provenance.py +++ b/tests/test_provenance.py @@ -99,6 +99,8 @@ def test_write_provenance_redacts_auth_options_and_environment_assignments(tmp_p { "command": [ "API_TOKEN=secret", + "GITHUB_PAT=secret", + "AWS_ACCESS_KEY_ID=secret", "server", "--auth=secret", "--bearer-token=secret", @@ -108,6 +110,8 @@ def test_write_provenance_redacts_auth_options_and_environment_assignments(tmp_p payload = json.loads(output.read_text(encoding="utf-8")) assert payload["source"]["command"] == [ + "***REDACTED***", + "***REDACTED***", "***REDACTED***", "server", "***REDACTED***", From 2451c6d82bbcb7dfdead58d5508c7541504df6c3 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 21:07:59 -0400 Subject: [PATCH 15/17] update snapshot storage final fix report --- .../sdd/2026-08-13-snapshot-storage/final-fix-report.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md b/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md index 28b29a5..5ec57d9 100644 --- a/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md +++ b/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md @@ -13,6 +13,10 @@ resolution continues to use the requested working directory. - Applied the exact Ruff formatting changes required in the snapshot-storage plan, provenance module, and Action regression test. +- P1 re-review follow-up: all command-form `NAME=value` environment assignments + are redacted, including names such as `GITHUB_PAT` and `AWS_ACCESS_KEY_ID`. + The Action upload step now requires the successful outcome of the complete + artifact-path validation step while retaining `always()` for failed compares. ## Verification @@ -27,3 +31,7 @@ ## Implementation commit `657f9b44ba901722d30f43a5e6beee865799b6b8` (`fix snapshot storage review findings`) + +## P1 re-review follow-up commit + +`e4729635e0fd961466fd56523a8e05cff4a5a6fd` (`harden snapshot artifact and provenance checks`) From fe53dc84bf234f9e78a4cbd5278aef98c76668a2 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 21:09:32 -0400 Subject: [PATCH 16/17] redact command environment option values --- src/tool_semantics/provenance.py | 6 ++++++ tests/test_provenance.py | 14 ++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/src/tool_semantics/provenance.py b/src/tool_semantics/provenance.py index b664a56..5cb9662 100644 --- a/src/tool_semantics/provenance.py +++ b/src/tool_semantics/provenance.py @@ -14,6 +14,7 @@ re.IGNORECASE, ) _COMMAND_HEADER_OPTION_PATTERN = re.compile(r"^(?:-H|--header)(?:=|$)", re.IGNORECASE) +_COMMAND_ENVIRONMENT_OPTION_PATTERN = re.compile(r"^(?:--env|-e)(?:=|$)", re.IGNORECASE) _ENVIRONMENT_KEY_PATTERN = re.compile(r"(?:^|_)(?:env|environment)(?:_|$)", re.IGNORECASE) _ENVIRONMENT_ASSIGNMENT_PATTERN = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)=", re.IGNORECASE) _REDACTED = "***REDACTED***" @@ -51,6 +52,11 @@ def _redact_command_arguments(command: list[Any]) -> list[Any]: environment_assignment = _ENVIRONMENT_ASSIGNMENT_PATTERN.match(argument) if environment_assignment: redacted[index] = _REDACTED + elif _COMMAND_ENVIRONMENT_OPTION_PATTERN.match(argument): + if "=" in argument: + redacted[index] = f"{argument.split('=', maxsplit=1)[0]}={_REDACTED}" + elif index + 1 < len(redacted): + redacted[index + 1] = _REDACTED elif _COMMAND_SECRET_OPTION_PATTERN.match(argument) or _COMMAND_HEADER_OPTION_PATTERN.match( argument ): diff --git a/tests/test_provenance.py b/tests/test_provenance.py index 912ae77..fd0eec2 100644 --- a/tests/test_provenance.py +++ b/tests/test_provenance.py @@ -101,6 +101,13 @@ def test_write_provenance_redacts_auth_options_and_environment_assignments(tmp_p "API_TOKEN=secret", "GITHUB_PAT=secret", "AWS_ACCESS_KEY_ID=secret", + "export", + "API_TOKEN=export-secret", + "--env", + "API_TOKEN=environment-secret", + "--env=API_TOKEN=inline-secret", + "-e", + "API_TOKEN=short-option-secret", "server", "--auth=secret", "--bearer-token=secret", @@ -113,6 +120,13 @@ def test_write_provenance_redacts_auth_options_and_environment_assignments(tmp_p "***REDACTED***", "***REDACTED***", "***REDACTED***", + "export", + "***REDACTED***", + "--env", + "***REDACTED***", + "--env=***REDACTED***", + "-e", + "***REDACTED***", "server", "***REDACTED***", "***REDACTED***", From a3b7a9e13135a23010af49ef40b56337130b4e50 Mon Sep 17 00:00:00 2001 From: Abhinaysai Kamineni <66816045+askmy-stack@users.noreply.github.com> Date: Wed, 12 Aug 2026 21:09:43 -0400 Subject: [PATCH 17/17] record command environment redaction fix --- .../sdd/2026-08-13-snapshot-storage/final-fix-report.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md b/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md index 5ec57d9..db753e4 100644 --- a/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md +++ b/.superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md @@ -17,6 +17,10 @@ are redacted, including names such as `GITHUB_PAT` and `AWS_ACCESS_KEY_ID`. The Action upload step now requires the successful outcome of the complete artifact-path validation step while retaining `always()` for failed compares. +- Final P1 follow-up: command environment forms `export NAME=value`, + `--env NAME=value`, `--env=NAME=value`, and `-e NAME=value` cannot serialize + secret values. Option syntax is retained where useful while the supplied + environment setting is redacted. ## Verification @@ -35,3 +39,7 @@ ## P1 re-review follow-up commit `e4729635e0fd961466fd56523a8e05cff4a5a6fd` (`harden snapshot artifact and provenance checks`) + +## Final P1 follow-up commit + +`fe53dc84bf234f9e78a4cbd5278aef98c76668a2` (`redact command environment option values`)