Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions .claude/skills/ci-push-checklist/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,26 @@ When shipping a code fix or addressing a discrepancy in GitGalaxy, the following
* **CRITICAL CORPUS WARNING:** NEVER clone a fresh, temporary copy of `language-crucible` inside the `gitgalaxy` workspace just to bypass sandbox or path restrictions. A fresh internal clone alters absolute path metadata and Git footprints, which shifts the entire graph topology and generates massive, invalid diffs that fail in CI. Always point `LANGUAGE_CRUCIBLE_PATH` to the existing pristine sibling directory (e.g., `../language-crucible`) and run with bypass sandbox privileges if needed.

## 4. Discrepancy Ledgers & Tri-Comparison
If your fix resolves an open shape from the Tri-Comparison Ledger (`docs/self_scan/tri_comparison_ledger.json`):
* **NEVER hand-edit `still_reproduces`:** `still_reproduces` is automatically recomputed by the regen script. A shape might have multiple contributing causes, so a single fix doesn't guarantee it stops reproducing. Instead, if you've added a human verdict to an unvalidated shape, only update `"status": "validated"`, `"verdict"`, `"investigated_by"`, and `"investigated_at"`.
* **Regenerate the Chart & Report:** Re-run the tri-comparison to automatically update the SVG, recompute `still_reproduces`, and refresh the point of interest report:
* **CI now gates this automatically if you touched `detector.py`/`prism.py`/`language_standards.py`:**
`tri-comparison-audit.yml` fails your PR if GitGalaxy's own *validated* precision
(`func_precision`/`class_precision`, read after ledger verdicts are applied — never a raw
disagreement count) regresses against the committed `tests/tri_comparison_baseline_<lang>.json`.
Run it yourself before pushing to catch this early: `python tests/tools/tri_comparison_chart.py
--all --ci` (needs real `ctags` on PATH and the language-crucible corpus — see the README's
"Reproducing or updating this locally" section). If your fix intentionally improves a language's
precision, lock it in with `python tests/tools/tri_comparison_chart.py --regenerate --languages
<lang>` and commit the updated baseline file.
* **You no longer need to manually regenerate the chart/ledger/report before pushing.** A
push-to-main companion workflow (`tri-comparison-history.yml`) does that automatically once your
PR merges, opening its own auto-merged PR only if the numbers actually moved. Feel free to still
run it locally for a sanity check, but it's no longer a required pre-push step the way it used
to be:
`python tests/tools/tri_comparison_chart.py --all --write`
`python tests/tools/tri_comparison_report.py --write`
* **NEVER hand-edit `still_reproduces`:** it's automatically recomputed by the regen script (a
shape might have multiple contributing causes, so a single fix doesn't guarantee it stops
reproducing). If you've added a human verdict to an unvalidated shape, only update
`"status": "validated"`, `"verdict"`, `"investigated_by"`, and `"investigated_at"` by hand.

## 5. Tree-Sitter AST & Accuracy Baseline
* **Audit AST Accuracy:** Run `python tests/tools/tree_sitter_accuracy_audit.py --ci --all`.
Expand Down
41 changes: 35 additions & 6 deletions .claude/skills/tree-sitter-accuracy-sweep/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,12 @@ them and the Gemini subagent had to improvise (copying `pytest` over from the ma
build the venv complete the first time, since agy's sandbox cannot `pip install` anything itself
(no outbound network at all).

**Verify real `ctags` is on PATH before dispatch, not just at done-bar time:** `ctags --version`
must print `Universal Ctags`, not error or print an Arduino banner (Ubuntu's `arduino-ctags`
package shadows the binary name -- see `docs/self_scan/tri_comparison_README.md`'s PR #2111
writeup). Step 8's `tri_comparison_chart.py --all --ci` needs this and fails loud without it; check
it once here rather than discovering it's missing after a full sweep of dispatches is already done.

## 5. Grant Gemini/agy access to the new worktree(s)

`~/.gemini/antigravity-cli/settings.json` needs, per worktree:
Expand Down Expand Up @@ -200,11 +206,28 @@ export PATH="$PWD/venv/bin:$PATH"
export LANGUAGE_CRUCIBLE_PATH=/home/joe/nyx_projects/language-crucible
python tests/tools/tree_sitter_accuracy_audit.py --lang <lang> --regenerate # bless the baseline + summary table together
python tests/tools/tree_sitter_accuracy_audit.py --all --ci # see below -- not optional
python tests/tools/tri_comparison_chart.py --all --ci # validated-precision gate -- also not optional, same reason
python tests/tools/crucible_check.py --mode both # see the drift first
python tests/tools/crucible_check.py --mode both --update --yes # bless if drift is expected
python tests/tools/crucible_check.py --mode both # re-run, confirm now PASS/PASS
python tests/tools/audit_check.py --ci # ruff+mypy+dead-key+ast-accuracy
```

**`tri_comparison_chart.py --all --ci`** is CI-enforced now (`tri-comparison-audit.yml`) whenever a
fix touches `detector.py`/`prism.py`/`language_standards.py` -- it fails the PR if GitGalaxy's own
*validated* precision (read after any ledger verdict is applied, never a raw disagreement count --
see `docs/self_scan/tri_comparison_README.md`) regresses against a committed
`tests/tri_comparison_baseline_<lang>.json`. It only has a baseline to check for languages someone
has already run `--regenerate` on -- if `<lang>` doesn't have one yet, `--all --ci` simply skips it
(nothing to bless), same as `tree_sitter_accuracy_audit.py`'s baselines. If `<lang>` DOES already
have one and this fix legitimately moved its precision, don't blind-`--regenerate` the way you
would for the tree-sitter baseline: a validated-precision shift can also mean a ledger entry got
(in)validated by something unrelated to your fix, so read the printed diff first, confirm it's
actually attributable to your change, then
`python tests/tools/tri_comparison_chart.py --regenerate --languages <lang>` and commit the
updated baseline file. Never touches `tri_comparison_chart.svg` / `tri_comparison_ledger.json` /
`tri_comparison_points_of_interest.md` -- those regenerate automatically post-merge via
`tri-comparison-history.yml`, not something this skill's PRs need to do themselves.
`crucible_check.py` handles the two-venv (`.crucible_venvs/{full_precision,zero_dependency}`)
dance itself now -- don't hand-build those venvs. Expect real cascading drift in the golden
masters beyond just the target language's own directory (global PageRank/spatial-coordinate
Expand All @@ -231,7 +254,8 @@ green target-language result and a green general test suite are both necessary b
sufficient on its own to rule out cross-language ripple from a shared-function change.

Commit everything the regenerate/update steps touched (the regex file, the test file, the
per-language baseline JSON, both golden master JSONs), push, `gh pr create`.
per-language tree-sitter baseline JSON, both golden master JSONs, and the per-language
`tri_comparison_baseline_<lang>.json` if you regenerated one), push, `gh pr create`.

**Auto-merge is pre-authorized specifically for this skill's PRs** (user, 2026-08-13): once a fix
has been through the full independent-verification + done-bar checklist above (steps 7-8) and its
Expand Down Expand Up @@ -300,11 +324,16 @@ the whole time.
- `LANGUAGE_CRUCIBLE_PATH` must be passed explicitly from a worktree -- sibling-directory
resolution only works from the main checkout.
- Parallel-PR merge conflicts on the machine-generated files (`golden_master*.json`,
`ruff_audit_baseline.json`, `tree_sitter_accuracy_baseline_*.json`) are a real, expected,
survivable cost of running a pool of 5 -- when one lands, `git merge origin/main` on the others,
take origin/main's version of the conflicting generated files wholesale
(`git checkout --theirs -- <path>`), then regenerate everything fresh on the merged code and
re-run the full step-8 checklist again rather than hand-resolving JSON conflicts.
`ruff_audit_baseline.json`, `tree_sitter_accuracy_baseline_*.json`,
`tri_comparison_baseline_*.json`) are a real, expected, survivable cost of running a pool of 5 --
when one lands, `git merge origin/main` on the others, take origin/main's version of the
conflicting generated files wholesale (`git checkout --theirs -- <path>`), then regenerate
everything fresh on the merged code and re-run the full step-8 checklist again rather than
hand-resolving JSON conflicts.
- `ctags --version` must print `Universal Ctags` before `--all --ci` runs, in every venv/worktree
you check it from -- Ubuntu's `arduino-ctags` shadows the binary name and silently degrades every
language to a 2-tool comparison with no error at all (PR #2111). Check it once per worktree
(step 4), not only when step 8 already fails.
- Editing `~/.gemini/antigravity-cli/settings.json` needs a specific, in-the-moment
`AskUserQuestion` approval each time new worktree paths are added -- a general earlier "go
ahead" doesn't carry forward.
85 changes: 85 additions & 0 deletions .github/workflows/tri-comparison-audit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
name: Tri-Comparison Audit

# Baseline-gated regression check on GitGalaxy's own VALIDATED precision (GitGalaxy vs.
# tree-sitter vs. ctags), same shape as tree-sitter-accuracy-audit.yml's 2-tool equivalent --
# see tests/tools/tri_comparison_chart.py's own "--ci / --regenerate" section for why this
# gates precision only, never recall (recall's shared cross-tool denominator is exactly what
# that module's PANELS docstring section already explains is untrustworthy to rank on).
#
# Scoped to the same core-engine paths tree-sitter-accuracy-audit.yml already uses -- both
# audits exist to catch the same class of change, kept in sync deliberately.
#
# This job only ever MEASURES, never writes. It never requires a contributor to regenerate and
# commit docs/self_scan/tri_comparison_chart.svg / tri_comparison_ledger.json /
# tri_comparison_points_of_interest.md themselves -- that's the separate push-to-main companion,
# tri-comparison-history.yml, same split tree-sitter-accuracy-audit.yml/-history.yml already use
# and for the same reason: requiring every PR author to run a full language-crucible corpus scan
# + a real `ctags` binary locally just to unblock their own merge would reintroduce exactly the
# fragility PR #2111 (2026-08-22) already burned this repo on.

on:
pull_request:
branches:
- main
paths:
- "gitgalaxy/core/detector.py"
- "gitgalaxy/core/prism.py"
- "gitgalaxy/standards/language_standards.py"
- "tests/tools/tri_comparison_chart.py"
- "tests/tools/tri_comparison_gatherer.py"
- "tests/tools/tri_comparison_reconcile.py"
- "tests/tools/tri_comparison_ledger.py"
- "tests/tri_comparison_baseline_*.json"
- ".github/workflows/tri-comparison-audit.yml"

permissions:
contents: read

jobs:
tri-comparison-audit:
runs-on: ubuntu-latest
steps:
- name: Checkout GitGalaxy PR
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: gitgalaxy
persist-credentials: false

- name: Setup Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
cache: "pip"

- name: Install GitGalaxy + tree-sitter-language-pack
run: |
cd gitgalaxy
python -m pip install --upgrade pip
pip install -e .
pip install tree-sitter-language-pack

# Hard preflight, not optional: PR #2111 (2026-08-22, see
# docs/self_scan/tri_comparison_README.md) is a real incident where Ubuntu's
# arduino-ctags package shadowed the `ctags` binary name and silently degraded every
# language to a 2-tool comparison with no error anywhere in that run. A CI gate that
# could suffer the same silent degradation would be worse than no gate at all, since it
# would report false confidence instead of an honest failure.
- name: Install universal-ctags and verify it's the real thing
run: |
sudo apt-get update -qq
sudo apt-get install -y universal-ctags
if ! ctags --version | grep -q "Universal Ctags"; then
echo "::error::ctags on PATH is not Universal Ctags -- refusing to run a silently-degraded audit."
ctags --version || true
exit 1
fi

# Same pin golden-crucible.yml / tree-sitter-accuracy-audit.yml use -- this tool reuses
# that corpus rather than a fresh clone.
- name: Fetch Language Crucible corpus (pinned v1.0)
run: git clone --branch v1.0 --depth 1 https://github.com/squid-protocol/language-crucible.git language-crucible

- name: Run baseline-gated precision regression check across every baselined language
run: |
cd gitgalaxy
python tests/tools/tri_comparison_chart.py --all --ci
133 changes: 133 additions & 0 deletions .github/workflows/tri-comparison-history.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
name: Tri-Comparison History

# Push-to-main companion to tri-comparison-audit.yml's PR-time gate, same split
# tree-sitter-accuracy-audit.yml/-history.yml already use for the sibling 2-tool system. That
# gate only proves GitGalaxy's own validated precision (tests/tri_comparison_baseline_<lang>.json)
# didn't get worse; those baselines stay human-reviewed and only change via
# `tri_comparison_chart.py --regenerate`, same discipline as the golden master fixtures and the
# tree-sitter baselines. This workflow never touches them.
#
# What it DOES do, once a relevant change actually lands on main: regenerate
# docs/self_scan/tri_comparison_chart.svg, docs/self_scan/tri_comparison_ledger.json (which also
# refreshes last_seen_count/last_seen_examples on every existing ledger entry, and recomputes
# still_reproduces -- see tri_comparison_ledger.py's ENTRY LIFECYCLE docstring; a validated
# entry's status/verdict are never touched by this), and
# docs/self_scan/tri_comparison_points_of_interest.md, then opens an auto-merged PR with
# whatever actually changed -- same mechanism gitgalaxy.yml uses for the LLM architecture brief.
#
# Adaptive by design, not just by trigger path: the trigger below fires on any push touching
# detector.py/prism.py/language_standards.py, but not every such push moves a language's measured
# numbers or an existing ledger entry's counts. Unlike the tree-sitter history CSV (an
# append-only file that needs its own _batch_matches_measured dedup check to stay a no-op), the
# three files here are overwritten in place -- "no diff" already IS the adaptivity, so
# create-pull-request finds nothing to diff and opens no PR on a push that didn't actually move
# anything, no separate skip-detection needed.

on:
push:
branches: [main]
paths:
- "gitgalaxy/core/detector.py"
- "gitgalaxy/core/prism.py"
- "gitgalaxy/standards/language_standards.py"
- "tests/tools/tri_comparison_chart.py"
- "tests/tools/tri_comparison_gatherer.py"
- "tests/tools/tri_comparison_reconcile.py"
- "tests/tools/tri_comparison_ledger.py"
- "tests/tri_comparison_baseline_*.json"

permissions:
contents: read

jobs:
tri-comparison-history:
# Guards against the same infinite-loop shape tree-sitter-accuracy-history.yml already
# documents: this workflow's own bot commit touches tri_comparison_ledger.json, which is in
# the trigger's `paths` filter by way of docs/self_scan/ not being excluded -- without this
# it would re-trigger itself on every merge of its own PR.
if: |
!startsWith(github.event.head_commit.message, 'docs: auto-update tri-comparison data')
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- name: Checkout GitGalaxy
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: gitgalaxy
fetch-depth: 0
persist-credentials: false

- name: Setup Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
cache: "pip"

- name: Install GitGalaxy + tree-sitter-language-pack
run: |
cd gitgalaxy
python -m pip install --upgrade pip
pip install -e .
pip install tree-sitter-language-pack

# Same hard preflight as tri-comparison-audit.yml, for the same PR #2111 reason -- this
# job WRITES the chart/ledger, so a silently-degraded ctags here would commit bad data to
# main instead of just failing a single PR check.
- name: Install universal-ctags and verify it's the real thing
run: |
sudo apt-get update -qq
sudo apt-get install -y universal-ctags
if ! ctags --version | grep -q "Universal Ctags"; then
echo "::error::ctags on PATH is not Universal Ctags -- refusing to write a silently-degraded chart/ledger."
ctags --version || true
exit 1
fi

- name: Fetch Language Crucible corpus (pinned v1.0)
run: git clone --branch v1.0 --depth 1 https://github.com/squid-protocol/language-crucible.git language-crucible

- name: Regenerate chart + ledger + points-of-interest report
run: |
cd gitgalaxy
python tests/tools/tri_comparison_chart.py --all --write
python tests/tools/tri_comparison_report.py --write

- name: Open PR with updated chart + ledger + points-of-interest report
id: tri-comparison-history-pr
uses: squid-protocol/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # forked from peter-evans/create-pull-request v8.1.1
with:
token: ${{ secrets.AUTOMATION_PAT }}
path: gitgalaxy
commit-message: "docs: auto-update tri-comparison data"
title: "docs: auto-update tri-comparison data"
body: |
Automated update from the latest tri-comparison measurement on `main`.

- Re-renders `docs/self_scan/tri_comparison_chart.svg`.
- Refreshes `docs/self_scan/tri_comparison_ledger.json` (last_seen_count/
last_seen_examples/still_reproduces on existing entries; new unvalidated entries
for any newly-observed discrepancy shape). Never touches a validated entry's
`status`/`verdict`.
- Regenerates `docs/self_scan/tri_comparison_points_of_interest.md`.

Never touches `tests/tri_comparison_baseline_*.json` -- those stay human-reviewed via
`tri_comparison_chart.py --regenerate`.
branch: auto/tri-comparison-history
delete-branch: true
add-paths: |
docs/self_scan/tri_comparison_chart.svg
docs/self_scan/tri_comparison_ledger.json
docs/self_scan/tri_comparison_points_of_interest.md

# Same rationale as tree-sitter-accuracy-history.yml's own merge step -- nothing else
# gates this PR, so without an explicit merge it just sits mergeable-but-unmerged.
- name: Merge the tri-comparison history PR
if: steps.tri-comparison-history-pr.outputs.pull-request-number
env:
GH_TOKEN: ${{ secrets.AUTOMATION_PAT }}
run: |
PR="${{ steps.tri-comparison-history-pr.outputs.pull-request-number }}"
gh pr merge --squash "$PR" --repo ${{ github.repository }} || \
gh pr merge --squash --auto "$PR" --repo ${{ github.repository }}
Loading
Loading