Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/actions/compare/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ inputs:
description: If true, post or update a PR comment with the Markdown report.
required: false
default: "true"
upload-artifacts:
description: Upload candidate snapshot and generated reports as a workflow artifact.
required: false
default: "false"
fail-on-breaking:
description: Deprecated alias for policy=compatible when true.
required: false
Expand Down Expand Up @@ -155,3 +159,48 @@ runs:
body,
});
}

- name: Resolve artifact paths
id: artifact-paths
if: ${{ always() && inputs.upload-artifacts == 'true' }}
shell: bash
working-directory: ${{ inputs.working-directory }}
env:
CANDIDATE: ${{ inputs.candidate }}
run: |
python - <<'PY'
import os
from pathlib import Path

candidate = Path(os.environ["CANDIDATE"]).resolve()
with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output:
print(f"candidate-path={candidate}", file=output)
PY

# All three files are required: an incomplete diagnostic artifact is misleading.
# This runs after a compare failure as well when upload was explicitly requested.
- name: Validate artifact paths
id: validate-artifact-paths
if: ${{ always() && inputs.upload-artifacts == 'true' }}
shell: bash
run: |
for path in \
"${{ steps.artifact-paths.outputs.candidate-path }}" \
"${{ steps.compare.outputs.report-path }}" \
"${{ runner.temp }}/tool-semantics/report.json"; do
if [ ! -f "$path" ]; then
echo "Missing required artifact file: $path" >&2
exit 1
fi
done

- name: Upload report artifact
if: ${{ always() && inputs.upload-artifacts == 'true' && steps.validate-artifact-paths.outcome == 'success' }}
uses: actions/upload-artifact@v4
with:
name: tool-semantics-report
if-no-files-found: error
path: |
${{ steps.artifact-paths.outputs.candidate-path }}
${{ steps.compare.outputs.report-path }}
${{ runner.temp }}/tool-semantics/report.json
45 changes: 45 additions & 0 deletions .superpowers/sdd/2026-08-13-snapshot-storage/final-fix-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Snapshot storage final-fix report

## Changes

- Provenance command sanitization now redacts secret-like option names including
`--auth=…` and `--bearer-token=…`, plus secret-bearing environment assignments
such as `API_TOKEN=…`. This provenance-specific protection remains active even
when capture uses `--no-redact`.
- Capture and capture-mcp reject equal or equivalent snapshot/provenance paths
before writing, preserving an existing snapshot.
- The compare Action validates that the candidate, Markdown report, and JSON
report all exist before its explicitly opted-in artifact upload. Candidate path
resolution continues to use the requested working directory.
- Applied the exact Ruff formatting changes required in the snapshot-storage
plan, provenance module, and Action regression test.
- P1 re-review follow-up: all command-form `NAME=value` environment assignments
are redacted, including names such as `GITHUB_PAT` and `AWS_ACCESS_KEY_ID`.
The Action upload step now requires the successful outcome of the complete
artifact-path validation step while retaining `always()` for failed compares.
- Final P1 follow-up: command environment forms `export NAME=value`,
`--env NAME=value`, `--env=NAME=value`, and `-e NAME=value` cannot serialize
secret values. Option syntax is retained where useful while the supplied
environment setting is redacted.

## Verification

- `python -m pytest --cov=tool_semantics --cov-report=term-missing`: 54 passed;
total coverage 88%.
- `ruff check .`: passed.
- `ruff format --check .`: passed (41 files already formatted).
- `mypy src`: passed with no issues in 14 source files.
- `python -m build`: passed; built sdist and wheel.
- `git diff --check`: passed.

## Implementation commit

`657f9b44ba901722d30f43a5e6beee865799b6b8` (`fix snapshot storage review findings`)

## P1 re-review follow-up commit

`e4729635e0fd961466fd56523a8e05cff4a5a6fd` (`harden snapshot artifact and provenance checks`)

## Final P1 follow-up commit

`fe53dc84bf234f9e78a4cbd5278aef98c76668a2` (`redact command environment option values`)
22 changes: 20 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,8 +134,10 @@ print("compatible:", report.is_compatible)

```bash
tool-semantics --version
tool-semantics capture <manifest.json> [-o .tool-semantics/snapshot.json] [-v]
tool-semantics capture-mcp -o snap.json -- python my_mcp_server.py
tool-semantics capture <manifest.json> [-o .tool-semantics/snapshot.json] \
[--provenance-output snapshot.provenance.json] [-v]
tool-semantics capture-mcp -o snap.json \
[--provenance-output snap.provenance.json] -- python my_mcp_server.py
tool-semantics compare <baseline.json> <candidate.json> \
[--json-output report.json] \
[--markdown-output report.md] \
Expand All @@ -147,6 +149,22 @@ tool-semantics compare <baseline.json> <candidate.json> \
- `--config` loads ignore rules; if omitted, `.tool-semantics.toml` in the cwd is used when present.
- `capture-mcp` speaks MCP JSON-RPC over stdio; secrets-like keys are redacted by default.

### Approved baselines and provenance

Capture the approved interface into a Git-tracked baseline. The snapshot is the
contract that `compare` uses; review and commit it when an interface change is
intentional.

```bash
tool-semantics capture examples/github_server_v1.json \
-o .tool-semantics/baselines/github.json \
--provenance-output .tool-semantics/baselines/github.provenance.json
```

The optional provenance sidecar records capture context and a digest of the
snapshot. It is separate from the snapshot and never affects compatibility
comparisons.

JSON reports include `changes`, `is_compatible`, and `counts` by severity.
Change-code catalog: [docs/change-codes.md](docs/change-codes.md).
Ignore-config schema: [docs/config.md](docs/config.md).
Expand Down
17 changes: 15 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,11 @@ flowchart TB
subgraph outputs
CLI[Rich CLI table]
MD[Markdown report]
JSON[JSON artifact]
JSON[JSON compatibility report]
SNAP[Snapshot JSON — canonical compare input]
PROV[Optional provenance sidecar]
CI[Exit codes / GitHub Action]
ART[Optional CI diagnostic artifact]
end
M --> S
L --> S
Expand All @@ -37,6 +40,9 @@ flowchart TB
R --> MD
R --> JSON
R --> CI
N --> SNAP
SNAP -.-> PROV
CI -.-> ART
```

## Components
Expand All @@ -53,7 +59,14 @@ flowchart TB
| **Migration adapters** (`adapters.py`) | Tool aliases, argument/enum maps, output wrappers |
| **Report** (`report.py`) | Human-readable Markdown / styling helpers |
| **CLI** (`cli.py`) | `capture`, `capture-mcp`, `compare`, and related entry points |
| **GitHub Action** (`.github/actions/compare`) | CI compare + optional PR comment |
| **GitHub Action** (`.github/actions/compare`) | CI compare + optional PR comment; can upload candidate and reports as a diagnostic artifact |

Snapshots are the canonical JSON inputs to compatibility comparisons and are
normally committed to Git as approved baselines. Capture can also write an
optional provenance sidecar with capture context and the snapshot digest; it is
not part of the snapshot schema or compare input. Optional CI artifacts contain
the candidate snapshot and generated reports for diagnosis only, not a
replacement for Git-tracked baselines.

### Still planned

Expand Down
22 changes: 18 additions & 4 deletions docs/github-action.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,18 +39,19 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Capture baseline and candidate
# .tool-semantics/baselines/github.json was captured, reviewed, and
# committed before this pull request.
- name: Capture candidate snapshot
run: |
pip install "tool-semantics==0.2.0"
tool-semantics capture manifests/baseline.json -o .tool-semantics/baseline.json
tool-semantics capture manifests/candidate.json -o .tool-semantics/candidate.json
- uses: askmy-stack/tool-semantics/.github/actions/compare@v0.2.0
with:
baseline: .tool-semantics/baseline.json
baseline: .tool-semantics/baselines/github.json
candidate: .tool-semantics/candidate.json
config: .tool-semantics.toml
policy: strict
comment-on-pr: "true"
upload-artifacts: "true"
```

## Inputs
Expand All @@ -62,6 +63,7 @@ jobs:
| `config` | no | `""` | Optional ignore/policy config path |
| `policy` | no | `""` | `compatible` / `strict` / `critical-only` / `permissive` |
| `comment-on-pr` | no | `true` | Upsert a PR comment with the report |
| `upload-artifacts` | no | `false` | Upload the candidate snapshot and generated reports as a workflow artifact |
| `fail-on-breaking` | no | `true` | Legacy; `false` maps to `permissive` when `policy` unset |
| `working-directory` | no | `.` | Directory for install/compare |

Expand All @@ -73,6 +75,18 @@ jobs:
| `policy-failed` | `true` if the selected release policy failed |
| `report-path` | Path to the Markdown report artifact |

## Baselines and diagnostic artifacts

Capture, review, and commit the approved baseline snapshot to Git before it is
used in CI; treat that file as the compatibility contract. A pull-request job
should capture only its candidate and compare it to the committed baseline, not
recreate the baseline during the run. To intentionally update the contract,
capture a new baseline, review its diff, and commit that snapshot change.

When `upload-artifacts: "true"`, the Action uploads the candidate snapshot plus
the generated Markdown and JSON reports in the `tool-semantics-report` artifact.
These files help diagnose a CI run; they do not replace the Git-tracked baseline.

## Permissions

When `comment-on-pr` is enabled on `pull_request` events, the workflow needs
Expand Down
Loading