diff --git a/.claude/skills/ci-push-checklist/SKILL.md b/.claude/skills/ci-push-checklist/SKILL.md index 6e30ffe26..ed293cda0 100644 --- a/.claude/skills/ci-push-checklist/SKILL.md +++ b/.claude/skills/ci-push-checklist/SKILL.md @@ -20,11 +20,26 @@ When shipping a code fix or addressing a discrepancy in GitGalaxy, the following * **CRITICAL CORPUS WARNING:** NEVER clone a fresh, temporary copy of `language-crucible` inside the `gitgalaxy` workspace just to bypass sandbox or path restrictions. A fresh internal clone alters absolute path metadata and Git footprints, which shifts the entire graph topology and generates massive, invalid diffs that fail in CI. Always point `LANGUAGE_CRUCIBLE_PATH` to the existing pristine sibling directory (e.g., `../language-crucible`) and run with bypass sandbox privileges if needed. ## 4. Discrepancy Ledgers & Tri-Comparison -If your fix resolves an open shape from the Tri-Comparison Ledger (`docs/self_scan/tri_comparison_ledger.json`): -* **NEVER hand-edit `still_reproduces`:** `still_reproduces` is automatically recomputed by the regen script. A shape might have multiple contributing causes, so a single fix doesn't guarantee it stops reproducing. Instead, if you've added a human verdict to an unvalidated shape, only update `"status": "validated"`, `"verdict"`, `"investigated_by"`, and `"investigated_at"`. -* **Regenerate the Chart & Report:** Re-run the tri-comparison to automatically update the SVG, recompute `still_reproduces`, and refresh the point of interest report: +* **CI now gates this automatically if you touched `detector.py`/`prism.py`/`language_standards.py`:** + `tri-comparison-audit.yml` fails your PR if GitGalaxy's own *validated* precision + (`func_precision`/`class_precision`, read after ledger verdicts are applied — never a raw + disagreement count) regresses against the committed `tests/tri_comparison_baseline_.json`. + Run it yourself before pushing to catch this early: `python tests/tools/tri_comparison_chart.py + --all --ci` (needs real `ctags` on PATH and the language-crucible corpus — see the README's + "Reproducing or updating this locally" section). If your fix intentionally improves a language's + precision, lock it in with `python tests/tools/tri_comparison_chart.py --regenerate --languages + ` and commit the updated baseline file. +* **You no longer need to manually regenerate the chart/ledger/report before pushing.** A + push-to-main companion workflow (`tri-comparison-history.yml`) does that automatically once your + PR merges, opening its own auto-merged PR only if the numbers actually moved. Feel free to still + run it locally for a sanity check, but it's no longer a required pre-push step the way it used + to be: `python tests/tools/tri_comparison_chart.py --all --write` `python tests/tools/tri_comparison_report.py --write` +* **NEVER hand-edit `still_reproduces`:** it's automatically recomputed by the regen script (a + shape might have multiple contributing causes, so a single fix doesn't guarantee it stops + reproducing). If you've added a human verdict to an unvalidated shape, only update + `"status": "validated"`, `"verdict"`, `"investigated_by"`, and `"investigated_at"` by hand. ## 5. Tree-Sitter AST & Accuracy Baseline * **Audit AST Accuracy:** Run `python tests/tools/tree_sitter_accuracy_audit.py --ci --all`. diff --git a/.claude/skills/tree-sitter-accuracy-sweep/SKILL.md b/.claude/skills/tree-sitter-accuracy-sweep/SKILL.md index e6e54d464..c29bc449b 100644 --- a/.claude/skills/tree-sitter-accuracy-sweep/SKILL.md +++ b/.claude/skills/tree-sitter-accuracy-sweep/SKILL.md @@ -101,6 +101,12 @@ them and the Gemini subagent had to improvise (copying `pytest` over from the ma build the venv complete the first time, since agy's sandbox cannot `pip install` anything itself (no outbound network at all). +**Verify real `ctags` is on PATH before dispatch, not just at done-bar time:** `ctags --version` +must print `Universal Ctags`, not error or print an Arduino banner (Ubuntu's `arduino-ctags` +package shadows the binary name -- see `docs/self_scan/tri_comparison_README.md`'s PR #2111 +writeup). Step 8's `tri_comparison_chart.py --all --ci` needs this and fails loud without it; check +it once here rather than discovering it's missing after a full sweep of dispatches is already done. + ## 5. Grant Gemini/agy access to the new worktree(s) `~/.gemini/antigravity-cli/settings.json` needs, per worktree: @@ -200,11 +206,28 @@ export PATH="$PWD/venv/bin:$PATH" export LANGUAGE_CRUCIBLE_PATH=/home/joe/nyx_projects/language-crucible python tests/tools/tree_sitter_accuracy_audit.py --lang --regenerate # bless the baseline + summary table together python tests/tools/tree_sitter_accuracy_audit.py --all --ci # see below -- not optional +python tests/tools/tri_comparison_chart.py --all --ci # validated-precision gate -- also not optional, same reason python tests/tools/crucible_check.py --mode both # see the drift first python tests/tools/crucible_check.py --mode both --update --yes # bless if drift is expected python tests/tools/crucible_check.py --mode both # re-run, confirm now PASS/PASS python tests/tools/audit_check.py --ci # ruff+mypy+dead-key+ast-accuracy ``` + +**`tri_comparison_chart.py --all --ci`** is CI-enforced now (`tri-comparison-audit.yml`) whenever a +fix touches `detector.py`/`prism.py`/`language_standards.py` -- it fails the PR if GitGalaxy's own +*validated* precision (read after any ledger verdict is applied, never a raw disagreement count -- +see `docs/self_scan/tri_comparison_README.md`) regresses against a committed +`tests/tri_comparison_baseline_.json`. It only has a baseline to check for languages someone +has already run `--regenerate` on -- if `` doesn't have one yet, `--all --ci` simply skips it +(nothing to bless), same as `tree_sitter_accuracy_audit.py`'s baselines. If `` DOES already +have one and this fix legitimately moved its precision, don't blind-`--regenerate` the way you +would for the tree-sitter baseline: a validated-precision shift can also mean a ledger entry got +(in)validated by something unrelated to your fix, so read the printed diff first, confirm it's +actually attributable to your change, then +`python tests/tools/tri_comparison_chart.py --regenerate --languages ` and commit the +updated baseline file. Never touches `tri_comparison_chart.svg` / `tri_comparison_ledger.json` / +`tri_comparison_points_of_interest.md` -- those regenerate automatically post-merge via +`tri-comparison-history.yml`, not something this skill's PRs need to do themselves. `crucible_check.py` handles the two-venv (`.crucible_venvs/{full_precision,zero_dependency}`) dance itself now -- don't hand-build those venvs. Expect real cascading drift in the golden masters beyond just the target language's own directory (global PageRank/spatial-coordinate @@ -231,7 +254,8 @@ green target-language result and a green general test suite are both necessary b sufficient on its own to rule out cross-language ripple from a shared-function change. Commit everything the regenerate/update steps touched (the regex file, the test file, the -per-language baseline JSON, both golden master JSONs), push, `gh pr create`. +per-language tree-sitter baseline JSON, both golden master JSONs, and the per-language +`tri_comparison_baseline_.json` if you regenerated one), push, `gh pr create`. **Auto-merge is pre-authorized specifically for this skill's PRs** (user, 2026-08-13): once a fix has been through the full independent-verification + done-bar checklist above (steps 7-8) and its @@ -300,11 +324,16 @@ the whole time. - `LANGUAGE_CRUCIBLE_PATH` must be passed explicitly from a worktree -- sibling-directory resolution only works from the main checkout. - Parallel-PR merge conflicts on the machine-generated files (`golden_master*.json`, - `ruff_audit_baseline.json`, `tree_sitter_accuracy_baseline_*.json`) are a real, expected, - survivable cost of running a pool of 5 -- when one lands, `git merge origin/main` on the others, - take origin/main's version of the conflicting generated files wholesale - (`git checkout --theirs -- `), then regenerate everything fresh on the merged code and - re-run the full step-8 checklist again rather than hand-resolving JSON conflicts. + `ruff_audit_baseline.json`, `tree_sitter_accuracy_baseline_*.json`, + `tri_comparison_baseline_*.json`) are a real, expected, survivable cost of running a pool of 5 -- + when one lands, `git merge origin/main` on the others, take origin/main's version of the + conflicting generated files wholesale (`git checkout --theirs -- `), then regenerate + everything fresh on the merged code and re-run the full step-8 checklist again rather than + hand-resolving JSON conflicts. +- `ctags --version` must print `Universal Ctags` before `--all --ci` runs, in every venv/worktree + you check it from -- Ubuntu's `arduino-ctags` shadows the binary name and silently degrades every + language to a 2-tool comparison with no error at all (PR #2111). Check it once per worktree + (step 4), not only when step 8 already fails. - Editing `~/.gemini/antigravity-cli/settings.json` needs a specific, in-the-moment `AskUserQuestion` approval each time new worktree paths are added -- a general earlier "go ahead" doesn't carry forward. diff --git a/.github/workflows/tri-comparison-audit.yml b/.github/workflows/tri-comparison-audit.yml new file mode 100644 index 000000000..1abc5259b --- /dev/null +++ b/.github/workflows/tri-comparison-audit.yml @@ -0,0 +1,85 @@ +name: Tri-Comparison Audit + +# Baseline-gated regression check on GitGalaxy's own VALIDATED precision (GitGalaxy vs. +# tree-sitter vs. ctags), same shape as tree-sitter-accuracy-audit.yml's 2-tool equivalent -- +# see tests/tools/tri_comparison_chart.py's own "--ci / --regenerate" section for why this +# gates precision only, never recall (recall's shared cross-tool denominator is exactly what +# that module's PANELS docstring section already explains is untrustworthy to rank on). +# +# Scoped to the same core-engine paths tree-sitter-accuracy-audit.yml already uses -- both +# audits exist to catch the same class of change, kept in sync deliberately. +# +# This job only ever MEASURES, never writes. It never requires a contributor to regenerate and +# commit docs/self_scan/tri_comparison_chart.svg / tri_comparison_ledger.json / +# tri_comparison_points_of_interest.md themselves -- that's the separate push-to-main companion, +# tri-comparison-history.yml, same split tree-sitter-accuracy-audit.yml/-history.yml already use +# and for the same reason: requiring every PR author to run a full language-crucible corpus scan +# + a real `ctags` binary locally just to unblock their own merge would reintroduce exactly the +# fragility PR #2111 (2026-08-22) already burned this repo on. + +on: + pull_request: + branches: + - main + paths: + - "gitgalaxy/core/detector.py" + - "gitgalaxy/core/prism.py" + - "gitgalaxy/standards/language_standards.py" + - "tests/tools/tri_comparison_chart.py" + - "tests/tools/tri_comparison_gatherer.py" + - "tests/tools/tri_comparison_reconcile.py" + - "tests/tools/tri_comparison_ledger.py" + - "tests/tri_comparison_baseline_*.json" + - ".github/workflows/tri-comparison-audit.yml" + +permissions: + contents: read + +jobs: + tri-comparison-audit: + runs-on: ubuntu-latest + steps: + - name: Checkout GitGalaxy PR + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + path: gitgalaxy + persist-credentials: false + + - name: Setup Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: "pip" + + - name: Install GitGalaxy + tree-sitter-language-pack + run: | + cd gitgalaxy + python -m pip install --upgrade pip + pip install -e . + pip install tree-sitter-language-pack + + # Hard preflight, not optional: PR #2111 (2026-08-22, see + # docs/self_scan/tri_comparison_README.md) is a real incident where Ubuntu's + # arduino-ctags package shadowed the `ctags` binary name and silently degraded every + # language to a 2-tool comparison with no error anywhere in that run. A CI gate that + # could suffer the same silent degradation would be worse than no gate at all, since it + # would report false confidence instead of an honest failure. + - name: Install universal-ctags and verify it's the real thing + run: | + sudo apt-get update -qq + sudo apt-get install -y universal-ctags + if ! ctags --version | grep -q "Universal Ctags"; then + echo "::error::ctags on PATH is not Universal Ctags -- refusing to run a silently-degraded audit." + ctags --version || true + exit 1 + fi + + # Same pin golden-crucible.yml / tree-sitter-accuracy-audit.yml use -- this tool reuses + # that corpus rather than a fresh clone. + - name: Fetch Language Crucible corpus (pinned v1.0) + run: git clone --branch v1.0 --depth 1 https://github.com/squid-protocol/language-crucible.git language-crucible + + - name: Run baseline-gated precision regression check across every baselined language + run: | + cd gitgalaxy + python tests/tools/tri_comparison_chart.py --all --ci diff --git a/.github/workflows/tri-comparison-history.yml b/.github/workflows/tri-comparison-history.yml new file mode 100644 index 000000000..796210631 --- /dev/null +++ b/.github/workflows/tri-comparison-history.yml @@ -0,0 +1,133 @@ +name: Tri-Comparison History + +# Push-to-main companion to tri-comparison-audit.yml's PR-time gate, same split +# tree-sitter-accuracy-audit.yml/-history.yml already use for the sibling 2-tool system. That +# gate only proves GitGalaxy's own validated precision (tests/tri_comparison_baseline_.json) +# didn't get worse; those baselines stay human-reviewed and only change via +# `tri_comparison_chart.py --regenerate`, same discipline as the golden master fixtures and the +# tree-sitter baselines. This workflow never touches them. +# +# What it DOES do, once a relevant change actually lands on main: regenerate +# docs/self_scan/tri_comparison_chart.svg, docs/self_scan/tri_comparison_ledger.json (which also +# refreshes last_seen_count/last_seen_examples on every existing ledger entry, and recomputes +# still_reproduces -- see tri_comparison_ledger.py's ENTRY LIFECYCLE docstring; a validated +# entry's status/verdict are never touched by this), and +# docs/self_scan/tri_comparison_points_of_interest.md, then opens an auto-merged PR with +# whatever actually changed -- same mechanism gitgalaxy.yml uses for the LLM architecture brief. +# +# Adaptive by design, not just by trigger path: the trigger below fires on any push touching +# detector.py/prism.py/language_standards.py, but not every such push moves a language's measured +# numbers or an existing ledger entry's counts. Unlike the tree-sitter history CSV (an +# append-only file that needs its own _batch_matches_measured dedup check to stay a no-op), the +# three files here are overwritten in place -- "no diff" already IS the adaptivity, so +# create-pull-request finds nothing to diff and opens no PR on a push that didn't actually move +# anything, no separate skip-detection needed. + +on: + push: + branches: [main] + paths: + - "gitgalaxy/core/detector.py" + - "gitgalaxy/core/prism.py" + - "gitgalaxy/standards/language_standards.py" + - "tests/tools/tri_comparison_chart.py" + - "tests/tools/tri_comparison_gatherer.py" + - "tests/tools/tri_comparison_reconcile.py" + - "tests/tools/tri_comparison_ledger.py" + - "tests/tri_comparison_baseline_*.json" + +permissions: + contents: read + +jobs: + tri-comparison-history: + # Guards against the same infinite-loop shape tree-sitter-accuracy-history.yml already + # documents: this workflow's own bot commit touches tri_comparison_ledger.json, which is in + # the trigger's `paths` filter by way of docs/self_scan/ not being excluded -- without this + # it would re-trigger itself on every merge of its own PR. + if: | + !startsWith(github.event.head_commit.message, 'docs: auto-update tri-comparison data') + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + steps: + - name: Checkout GitGalaxy + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + path: gitgalaxy + fetch-depth: 0 + persist-credentials: false + + - name: Setup Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: "pip" + + - name: Install GitGalaxy + tree-sitter-language-pack + run: | + cd gitgalaxy + python -m pip install --upgrade pip + pip install -e . + pip install tree-sitter-language-pack + + # Same hard preflight as tri-comparison-audit.yml, for the same PR #2111 reason -- this + # job WRITES the chart/ledger, so a silently-degraded ctags here would commit bad data to + # main instead of just failing a single PR check. + - name: Install universal-ctags and verify it's the real thing + run: | + sudo apt-get update -qq + sudo apt-get install -y universal-ctags + if ! ctags --version | grep -q "Universal Ctags"; then + echo "::error::ctags on PATH is not Universal Ctags -- refusing to write a silently-degraded chart/ledger." + ctags --version || true + exit 1 + fi + + - name: Fetch Language Crucible corpus (pinned v1.0) + run: git clone --branch v1.0 --depth 1 https://github.com/squid-protocol/language-crucible.git language-crucible + + - name: Regenerate chart + ledger + points-of-interest report + run: | + cd gitgalaxy + python tests/tools/tri_comparison_chart.py --all --write + python tests/tools/tri_comparison_report.py --write + + - name: Open PR with updated chart + ledger + points-of-interest report + id: tri-comparison-history-pr + uses: squid-protocol/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # forked from peter-evans/create-pull-request v8.1.1 + with: + token: ${{ secrets.AUTOMATION_PAT }} + path: gitgalaxy + commit-message: "docs: auto-update tri-comparison data" + title: "docs: auto-update tri-comparison data" + body: | + Automated update from the latest tri-comparison measurement on `main`. + + - Re-renders `docs/self_scan/tri_comparison_chart.svg`. + - Refreshes `docs/self_scan/tri_comparison_ledger.json` (last_seen_count/ + last_seen_examples/still_reproduces on existing entries; new unvalidated entries + for any newly-observed discrepancy shape). Never touches a validated entry's + `status`/`verdict`. + - Regenerates `docs/self_scan/tri_comparison_points_of_interest.md`. + + Never touches `tests/tri_comparison_baseline_*.json` -- those stay human-reviewed via + `tri_comparison_chart.py --regenerate`. + branch: auto/tri-comparison-history + delete-branch: true + add-paths: | + docs/self_scan/tri_comparison_chart.svg + docs/self_scan/tri_comparison_ledger.json + docs/self_scan/tri_comparison_points_of_interest.md + + # Same rationale as tree-sitter-accuracy-history.yml's own merge step -- nothing else + # gates this PR, so without an explicit merge it just sits mergeable-but-unmerged. + - name: Merge the tri-comparison history PR + if: steps.tri-comparison-history-pr.outputs.pull-request-number + env: + GH_TOKEN: ${{ secrets.AUTOMATION_PAT }} + run: | + PR="${{ steps.tri-comparison-history-pr.outputs.pull-request-number }}" + gh pr merge --squash "$PR" --repo ${{ github.repository }} || \ + gh pr merge --squash --auto "$PR" --repo ${{ github.repository }} diff --git a/CLAUDE.md b/CLAUDE.md index a30839d2c..4d644e2bb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -201,6 +201,13 @@ a bug, or an intentional improvement that needs the baseline re-blessed. **Never fixtures.** Regenerate with `python tests/tools/update_golden_master.py` (shows the diff, asks for confirmation) and explain *why* in the PR description — CI flags any PR touching these files. +The same PR paths also run `tri-comparison-audit.yml`, a baseline-gated regression check on +GitGalaxy's own **validated** precision against tree-sitter+ctags (see `docs/self_scan/ +tri_comparison_README.md`'s "CI enforcement" section) — a distinct, blocking check from the +golden-master diff above, alongside `tree-sitter-accuracy-audit.yml`'s 2-tool equivalent. None of +the three requires you to regenerate and commit chart/history artifacts yourself; that's handled +by separate push-to-main companion workflows after merge. + **Verifying locally before pushing:** use `python tests/tools/crucible_check.py` (add `--update` to regenerate) instead of hand-building venvs. It builds/reuses two venvs at `.crucible_venvs/{full_precision,zero_dependency}` *inside the current checkout* and — critically diff --git a/docs/self_scan/tri_comparison_README.md b/docs/self_scan/tri_comparison_README.md index ed1a376d5..a3efbf62d 100644 --- a/docs/self_scan/tri_comparison_README.md +++ b/docs/self_scan/tri_comparison_README.md @@ -178,6 +178,38 @@ not accounted for (see `ctags_reader.py`'s `_QUALIFY_NAME_WITH_SCOPE` mechanism case is handled: re-join name+scope from ctags' own tag data, gated on the qualified text actually appearing in the tag's own verbatim source line). +## CI enforcement + +This system was skill/human-driven only until it wasn't: any PR touching `detector.py` / +`prism.py` / `language_standards.py` (or this system's own tool files) now runs +`.github/workflows/tri-comparison-audit.yml`, a blocking PR-time check that fails if GitGalaxy's +own **validated** precision (`func_precision`/`class_precision`, read *after* +`apply_verified_adjustments()` has applied any ledger verdict — never a raw, unvalidated +disagreement) regresses against a committed baseline +(`tests/tri_comparison_baseline_.json`, one file per language, same convention as +`tests/tree_sitter_accuracy_baseline_.json`). Recall is deliberately not gated — see the +"RECALL AND ARGS-MATCH WERE REMOVED AS RATIOS" section of `tri_comparison_chart.py`'s own module +docstring for why that ratio's cross-tool denominator isn't trustworthy enough to regression-gate +on, the same reason it isn't rendered as a ranked bar on the chart itself. + +That PR-time gate only measures — it never requires a contributor to regenerate and commit the +chart/ledger/report themselves. A separate push-to-main companion, +`.github/workflows/tri-comparison-history.yml`, does that automatically after a relevant change +lands on `main`: re-render `tri_comparison_chart.svg`, refresh `tri_comparison_ledger.json` +(`last_seen_count`/`last_seen_examples`/`still_reproduces` only — never a validated entry's +`status`/`verdict`), and regenerate `tri_comparison_points_of_interest.md`, opening an auto-merged +PR only when something actually changed. This is the same split +`tree-sitter-accuracy-audit.yml`/`tree-sitter-accuracy-history.yml` already use for the 2-tool +system, adopted here rather than requiring every PR author to run a full corpus scan + a real +`ctags` locally just to unblock their own merge. + +Both workflows hard-fail if `ctags --version` doesn't print `Universal Ctags` before running +anything else — see PR #2111 above; a CI gate that could suffer the same silent degradation would +report false confidence instead of an honest failure. `tests/tri_comparison_baseline_.json` +baselines are populated incrementally, per language, via `tri_comparison_chart.py --regenerate +--languages ` — a language with no committed baseline yet is skipped by `--all --ci`, not +failed. + ## Files - **`tri_comparison_ledger.json`** — the persistent, hand-editable record described above. diff --git a/tests/tools/tri_comparison_chart.py b/tests/tools/tri_comparison_chart.py index f778b6e07..8e0dad732 100644 --- a/tests/tools/tri_comparison_chart.py +++ b/tests/tools/tri_comparison_chart.py @@ -822,6 +822,137 @@ def render_chart(data_by_lang: dict[str, LanguageChartData]) -> str: return "\n".join(parts) +# ---------------------------------------------------------------------------- +# --ci / --regenerate: baseline-gated regression check on GitGalaxy's own +# precision, same shape as tree_sitter_accuracy_audit.py's load_baseline/ +# _regressions/run_ci_check, kept intentionally separate (not a shared helper) +# rather than retrofitting that file's internals -- it's large, heavily tested, +# and not otherwise part of this change. +# +# PRECISION ONLY, DELIBERATELY -- not recall/found-count. This module's own +# docstring (PANELS section) already explains why recall was dropped as a +# RANKED ratio on this chart: its denominator is a cross-tool union, which one +# tool's own bug can corrupt without GitGalaxy doing anything wrong (confirmed +# case: ctags and tree-sitter both independently double-tagged Haskell +# multi-clause functions before that was fixed). Precision's denominator is a +# tool's own claim count -- self-referential, not corruptible the same way -- +# which is exactly why the chart still ranks/badges precision but not recall. +# Gating CI on recall would reintroduce the same untrusted-denominator problem +# the chart already designed around. +# +# Gated numbers are read AFTER run_pipeline() calls +# ledger_mod.apply_verified_adjustments() -- i.e. this is GitGalaxy's +# ledger-VALIDATED precision, never a raw unvalidated disagreement count, per +# CLAUDE.md's "Comparative-correctness claims require verification" rule. +# ---------------------------------------------------------------------------- + +_GATED_METRICS = ("func_precision", "class_precision") + + +def _get_baseline_path(lang: str) -> Path: + return Path(__file__).resolve().parent.parent.parent / "tests" / f"tri_comparison_baseline_{lang}.json" + + +def _extract_precision(data: LanguageChartData) -> dict[str, float]: + """GitGalaxy's own rate_pct for each ranked precision panel. A metric is omitted (not stored + as null) when there's no GitGalaxy score or its rate is undefined (0 slots) -- an absent key + means _regressions has nothing to compare for that language/metric on this run, same + convention tree_sitter_accuracy_audit.py's baseline-key-presence check uses.""" + out: dict[str, float] = {} + gg_func = data.func_precision.get("gitgalaxy") + if gg_func is not None and gg_func.rate_pct is not None: + out["func_precision"] = gg_func.rate_pct + gg_class = data.class_precision.get("gitgalaxy") + if gg_class is not None and gg_class.rate_pct is not None: + out["class_precision"] = gg_class.rate_pct + return out + + +def load_baseline(lang: str) -> dict: + path = _get_baseline_path(lang) + if not path.exists(): + return {} + with open(path, encoding="utf-8") as f: + return json.load(f) + + +def _regressions(current: dict, baseline: dict) -> list[str]: + regressions = [] + for key in _GATED_METRICS: + if key not in baseline or key not in current: + continue + cur, base = current[key], baseline[key] + if cur < base: + regressions.append(f"{key}: {base:.2f}% -> {cur:.2f}% (validated precision got worse)") + return regressions + + +def run_ci_check(lang: str, verbose: bool = True) -> int: + data = run_pipeline([lang], verbose=verbose)[lang] + if not data.has_data: + print(f"tri_comparison_chart: {lang} -- no corpus data available, skipping.") + return 0 + + current = _extract_precision(data) + baseline = load_baseline(lang) + if not baseline: + print(f"tri_comparison_chart: no baseline committed for {lang} -- run with --regenerate to create one, failing closed.") + return 1 + + regressions = _regressions(current, baseline) + if regressions: + print(f"tri_comparison_chart: {lang} -- {len(regressions)} regression(s) against the committed baseline:") + for line in regressions: + print(f" {line}") + return 1 + + improved = [k for k in _GATED_METRICS if k in current and k in baseline and current[k] > baseline[k]] + if improved: + print(f"tri_comparison_chart: {lang} -- OK, improved on {', '.join(improved)} (consider --regenerate to lock it in).") + else: + print(f"tri_comparison_chart: {lang} -- OK, matches committed baseline, no regressions.") + return 0 + + +def run_regenerate(lang: str, verbose: bool = True) -> int: + data = run_pipeline([lang], verbose=verbose)[lang] + if not data.has_data: + print(f"tri_comparison_chart: {lang} -- no corpus data available, cannot regenerate baseline.") + return 1 + + current = _extract_precision(data) + if not current: + print(f"tri_comparison_chart: {lang} -- no gated precision metric available (no GitGalaxy score), nothing to write.") + return 1 + + path = _get_baseline_path(lang) + path.write_text(json.dumps(current, indent=2, sort_keys=True) + "\n", encoding="utf-8") + print(f"tri_comparison_chart: wrote {path}") + return 0 + + +def _all_baseline_langs() -> list[str]: + """Every language with a committed tests/tri_comparison_baseline_.json, sorted.""" + prefix = "tri_comparison_baseline_" + root = Path(__file__).resolve().parent.parent.parent / "tests" + return sorted(p.stem[len(prefix) :] for p in root.glob(f"{prefix}*.json")) + + +def run_all_baseline_mode(languages: list[str], mode_fn, verbose: bool = True) -> int: + failed = [] + for lang in languages: + print(f"\n=== {lang} ===") + if mode_fn(lang, verbose=verbose) != 0: + failed.append(lang) + + print(f"\ntri_comparison_chart --ci: {len(languages)} language(s) checked.") + if failed: + print(f"tri_comparison_chart --ci: {len(failed)} FAILED: {', '.join(failed)}") + return 1 + print("tri_comparison_chart --ci: all OK.") + return 0 + + def main() -> int: parser = argparse.ArgumentParser() group = parser.add_mutually_exclusive_group(required=True) @@ -829,8 +960,29 @@ def main() -> int: group.add_argument("--all", action="store_true", help="Every language with a corpus available.") parser.add_argument("--write", action="store_true", help=f"Write SVG to {CHART_PATH}.") parser.add_argument("--quiet", action="store_true") + mode_group = parser.add_mutually_exclusive_group() + mode_group.add_argument( + "--ci", + action="store_true", + help="Baseline-gated regression check on GitGalaxy's own validated precision, instead of rendering the chart.", + ) + mode_group.add_argument( + "--regenerate", + action="store_true", + help="Accept current GitGalaxy precision as the new committed baseline, instead of rendering the chart.", + ) args = parser.parse_args() + if args.ci or args.regenerate: + # --all in baseline mode means "every language with a committed baseline" (glob-based), + # not "every language with a corpus" (all_languages()) -- a language nobody has + # baselined yet is skipped rather than failing the whole --all run closed. An explicit + # --languages request for an un-baselined language still fails closed inside + # run_ci_check itself, since that's a real ask for an answer that doesn't exist yet. + languages = _all_baseline_langs() if args.all else [s.strip() for s in args.languages.split(",")] + mode_fn = run_regenerate if args.regenerate else run_ci_check + return run_all_baseline_mode(languages, mode_fn, verbose=not args.quiet) + languages = all_languages() if args.all else [s.strip() for s in args.languages.split(",")] data = run_pipeline(languages, verbose=not args.quiet) svg = render_chart(data)