Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

### Added

- An offline runnable design lab with five synthetic before/after scenarios and two passing controls, comparing baseline new-error gating, score-delta-only gating and explicit candidate-threshold caller overrides through the real CLI. Retains observed decisions, commands and source/scenario hashes without claiming independent accuracy, AI superiority or user adoption. Runtime rules, scores, package artifacts and Action refs are unchanged.

- An explicitly manual, read-only GitHub Actions replay of the existing pinned real-repository smoke corpus. The same runner propagates acquisition/mismatch failures and retains logs/results for seven days; it does not execute external repository code, run on every PR or establish independent accuracy/adoption.

### Fixed
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ read the result, and troubleshoot a failing gate.

- [Score contract and limitations](limitations.md)
- [Regression input consistency and file-removal checks](regression-coverage.md)
- [Runnable design lab: scores, baseline gates and caller overrides](../examples/regression-lab/README.md) (synthetic development example)
- [Path-reference grammar and limits](limitations.md#plain-prose-path-references)
- [Instruction context and prohibition scopes](instruction-context.md)
- [False success-reporting checks](verification-reporting.md)
Expand Down
2 changes: 2 additions & 0 deletions docs/simplified/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

AgentConfigScore 是本地、确定性的编程助手指令检查器和 PR 回归门禁。它不调用 AI 服务,也不执行指令文件中的命令。100 分不代表指令有用、始终安全或比 AI 审查更好。

[可运行的设计演示(英文)](../../examples/regression-lab/README.md) 展示了仅看分数变化与采用基线新错误门禁的区别。正常通过的对照及显式调用方覆盖都是合成开发示例,不是真实用户案例或 AI 对照评测;演示脚本需要源码仓库。

## 1. 安装并确认版本

需要 Python 3.10+。`diff` 需要 Git;普通目录扫描和两个目录间的 `compare` 不需要 Git。
Expand Down
2 changes: 2 additions & 0 deletions docs/traditional/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

AgentConfigScore 是在本機執行、結果確定的開發助手指令檢查工具與 PR 回歸檢查。它不呼叫 AI 服務,也不執行指令檔內的命令。100 分不代表指令實用、始終安全或優於 AI 審查。

[可執行的設計示範(英文)](../../examples/regression-lab/README.md) 展示僅看分數變化與採用基準新錯誤檢查的差異。正常通過的對照及明確呼叫端覆寫都是合成開發範例,不是真實使用者案例或 AI 對照評測;示範腳本需要原始碼儲存庫。

## 1. 安裝並確認版本

需要 Python 3.10+。`diff` 需要 Git;一般目錄掃描和兩個目錄之間的 `compare` 不需要 Git。
Expand Down
7 changes: 7 additions & 0 deletions docs/translations.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,13 @@ explicit; workflow-only setup and unscored guardrails remain separate. This is
unreleased behavior, not a v0.24.0, Action-ref, scoring or frozen-evidence update.
Executable guide examples are unchanged.

On 2026-11-02, built on `4f928864f9d418887043908094a0083f1180f657`, both core
guides gained the same link to the English runnable regression design lab. Its
five synthetic development scenarios compare explicit gate settings, not AI
accuracy, real users or agent behavior. The source-checkout requirement and
evidence boundary are synchronized; executable guide examples, runtime behavior,
published artifacts, Action refs and frozen research are unchanged.

When an English source page changes, review both variants together and record
the new source revision here. Preserve executable examples, rule IDs and JSON
keys verbatim. Keep the score limitations and failure/exit semantics aligned;
Expand Down
5 changes: 5 additions & 0 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ gate for coding-agent instructions. It does not call an AI service or execute
commands found in instruction files. A 100 means no active rule matched, not
that instructions are useful, safe in every context, or better than AI review.

The [runnable design lab](../examples/regression-lab/README.md) shows why a numeric
score delta alone differs from baseline-governed new-error gating. Its passing
controls and explicit caller overrides are synthetic development demonstrations,
not user cases or an AI comparison. The runner requires a source checkout.

## 1. Install and check the version

Python 3.10 or later is required. Git is required for `diff`, but not for a
Expand Down
5 changes: 5 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Copy-ready instruction examples

For a runnable explanation of the regression gate rather than an instruction
template, see the [design lab](regression-lab/README.md). It compares five known
synthetic before/after scenarios under three explicitly defined gate settings;
it is not independent accuracy or adoption evidence.

These templates provide conservative repository workflow, verification, safety,
and handoff guidance without assuming a programming language or package manager.
Copy one into the matching path in your repository, then adapt it to your real
Expand Down
88 changes: 88 additions & 0 deletions examples/regression-lab/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Regression design lab

This runnable example explains two design choices: checking changes rather than
only numeric scores, and letting the reviewed baseline govern PR thresholds.
It uses **five synthetic, project-authored development scenarios**, not real-user
cases, an independent holdout or an AI comparison. No agent is run.

## Run from a source checkout

With Python 3.10+ and this repository available:

```bash
python -m pip install -e .
python scripts/run_regression_lab.py
```

No model API, network call or external repository is needed during the run. The
runner creates temporary before/after trees and invokes the real module CLI in
15 subprocesses. Commands inside fixture text are never executed. Source/runtime
identity must match across the runner and subprocesses. The script is a checkout
example, not a new installed subcommand or a feature published in v0.24.0.

To retain results, choose a **new** output filename:

```bash
python scripts/run_regression_lab.py --output .agent-config-score/design-lab-01.json
python scripts/run_regression_lab.py --json
```

Existing output files and file aliases are not overwritten. A failed save may
leave a new partial output; inspect before retrying. Exit `0` means the known
development expectations matched, `1` means a mismatch, and `2` means an
operational failure. These are lab exits, not estimates of detection accuracy.

## What the three columns mean

- **Baseline gate**: normal `compare`, with `max_drop: 0` and
`fail_on_new_errors: true` in the baseline config.
- **Score-only**: the same comparison with `--no-fail-on-new-errors`. It still
rejects a numeric score decrease. Findings remain visible; only error-based
gating is disabled.
- **Candidate-threshold override**: intentionally pass the candidate's threshold
values as trusted CLI overrides. This illustrates a caller trusting PR-authored
thresholds. The normal tool never chooses them on its own. It is not a separate
product, historical tool version or comparison of suppression policies.

## Retained observations

The [machine-readable run](observed.json) retained on 2026-11-02 records declared
tool version, tool-source fingerprint, normalized runner-source hash, scenario
hash, portable commands, actual exit codes and observed new error IDs. It uses
the runtime implementation at source revision
`4f928864f9d418887043908094a0083f1180f657`, with tool-source fingerprint
`bca1d4976d359c2f6e37a7068c077691461d8fc117d3df6a6ab3fdb5a10f82ef`.
The lab files themselves were added afterward in this development change.
Hashes bind recorded inputs/implementations; they do not authenticate execution
or certify evidence independence. Later reruns must retain their actual identity,
not relabel this observation as a newer tool.

| Synthetic scenario | Base -> head | Baseline gate | Score-only | Candidate-threshold override |
|---|---:|---|---|---|
| Unchanged instructions | 100 -> 100 | pass | pass | pass |
| Root instructions moved into a subtree | 100 -> 100 | block | pass | block |
| Exact directive reversed | 100 -> 100 | block | pass | block |
| Same-directory exact-text rename | 100 -> 100 | pass | pass | pass |
| Candidate weakens thresholds and adds a known risky command | 100 -> 82 | block | block | pass |

The scope-move row observes `instruction-file-removed`; the reversal row observes
`directive-polarity-flip`. Both show why a score delta alone can miss a declared
regression signal. The last row keeps `curl-pipe-shell` visible in every mode;
the intentionally weakened caller accepts it by changing its threshold decision.
The two passing controls show that this demonstration is not an always-block gate.

## What it does not establish

The scenarios were chosen because the implementation already supports them.
Five matching examples do not estimate real-world precision/recall, time saved,
agent behavior or user adoption. No AI baseline, independent human labels,
latency/cost comparison or counterfactual historical experiment was run.

File-removal and exact-polarity rules remain bounded heuristics. A same-directory
rename is an existing exact-text exception, not proof that different agents or
file formats have equivalent applicability. A legitimate scope migration can
need review or an expiring baseline-approved exception. A 100/100 passing result
can still contain harmful semantics outside the rule grammar. The candidate
override is deliberately weaker; an AI reviewer given both trees and policies
could also identify these examples. See [regression coverage](../../docs/regression-coverage.md)
and [limitations](../../docs/limitations.md).
Loading
Loading