feat: schema-check a Vale rule before Vale ever sees it

.changeset/vale-authoring-polish.md

@@ -38,3 +38,29 @@ The per-check field tables are measured the same way, which corrects three
 published claims: `capitalization` takes `prefix` (singular) and rejects
 `prefixes` and `suffixes`, `capitalization` rejects `ignorecase`, and
 `occurrence` rejects `exceptions` and `vocab`.
+
+`verify` now schema-checks a Vale rule structurally, before Vale is invoked.
+It previously validated `level` and the presence of the rule's `.vale.ini`, so
+`extends: nonsense` and `scope: fenced` both verified clean. It now also checks:
+
+- **`extends`** against the twelve check types, naming the accepted set.
+- **`scope`** as a grammar over measured operands — a bare value, a list, `~`
+  negation, `&` chaining — rather than a flat enum, which would have rejected
+  working rules. It is deliberately stricter than Vale in one place: a negation
+  over an operand Vale does not know (`~fenced`) fires on everything, having
+  silently lost its exclusion, and is rejected.
+- **Per-check fields**, so a field belonging to another check type is caught
+  before Vale reports `E201`. `consistency` and `spelling` are exempt because
+  the binary accepts any key on those two.
+
+The ordering is the point for two of the three: Vale reads one assembled config
+per run, so an unknown `extends` or a foreign field reaching the binary takes
+down **every** Vale rule's findings, not just the offending rule's.
+
+The schema is hand-authored, because Vale publishes no JSON Schema and its
+machine-readable field knowledge is behind a paid hosted MCP. What holds it to
+the binary is a corpus of 82 minimal rules, each with a document it must flag,
+run through both the vendored Vale and the schema, asserting the two agree —
+with guards so that a rule which "did not fire" because its fixture was
+unreachable cannot pass as a measurement. A Vale upgrade that changes the
+vocabulary fails a test that names the value.

.changeset/vale-schema-generation.md

@@ -0,0 +1,48 @@
+---
+"@taskless/cli": patch
+---
+
+Derive the Vale rule schema's vocabulary from the vendored binary instead of
+transcribing it by hand.
+
+`pnpm generate:vale-schema` runs the pinned Vale against rules it writes itself
+and emits `src/generated/vale-vocabulary.ts` — twelve check types, three levels,
+ten per-check field tables, twenty-eight scope operands, two open scope
+families — plus a divergence report. `src/schemas/vale-rule.ts` imports it and
+stays what it was: the zod layer, the scope grammar, and the error messages that
+explain blast radius to an author. Nothing about `verify`'s behavior changes; all
+86 existing corpus rows pass against the generated schema unmodified, and two
+were added to cover ground the generation newly measured.
+
+What a transcription lost was not the answer but the question. Every value in the
+previous schema _was_ measured — by a script that was then discarded, leaving the
+next person to raise `VALE_VERSION` with a failing test and no way to reproduce
+the measurement it was failing against.
+
+The measuring is also where the errors live, so the generator is built around one
+rule: **every verdict comes from the process exit status and the structured JSON
+output, never from matching stdout against an error phrase.** A Go panic contains
+no `has invalid keys` string, so a phrase-grep scores a crash as a clean run —
+which is exactly how a tokenless `sequence` rule once came to look like a check
+that validates nothing. A run's outcome is a closed set of `clean`, `diagnostic`,
+`panic`, and `unrecognized`, and the last one is fatal at every call site.
+
+Two of the four vocabularies are self-enumerating: an unknown `extends` or
+`level` makes the binary name its own accepted set. If either of those lines
+stops matching, generation **fails** rather than emitting a short enum — a
+truncated enum is _stricter_ than the binary, which is the direction that blocks
+rules that would have worked.
+
+The other two are honest about their limit, and the artifact says so. `E201`
+names the key you got wrong and never the ones you could have used, and an
+unrecognized `scope` raises nothing at all — so field tables and scope operands
+are **verified, not discovered**, from a candidate list seeded from four sources
+with its provenance recorded. A scope verdict is three-valued, and a `scope: raw`
+reach probe must fire on every fixture, so an operand cannot be dropped because
+its fixture was never linted.
+
+Where the binary and Vale's documentation disagree, `vale-vocabulary-report.md`
+records it rather than either side being quietly dropped: `meta` and
+`meta.class.<kind>` are documented and never fire; `frontmatter` and
+`frontmatter.<key>` fire and are documented nowhere; `consistency` and `spelling`
+validate no keys at all.

openspec/changes/vale-authoring-polish/design.md → openspec/changes/archive/2026-08-25-vale-authoring-polish/design.md

@@ -170,10 +170,42 @@ case-sensitive too — `level: WARNING` and `extends: Existence` are both
 rejected. The schema therefore compares field names case-insensitively and
 `extends`/`level` values exactly, which is what the binary does.
 
-**`consistency` and `spelling` accept any key at all.** `bananafield: true` on
-either loads without complaint and is ignored — they do not use the strict
-decode the other ten do. The schema cannot be strict about their fields without
-rejecting rules the binary accepts, so it is not.
+**`consistency` and `spelling` accept any key at all — and only those two.**
+`bananafield: true` on either loads without complaint and is ignored; they do
+not use the strict decode the other ten do. The schema cannot be strict about
+their fields without rejecting rules the binary accepts, so it is not.
+
+The split was re-probed across all twelve with a nonsense key rather than a real
+field borrowed from another check, because `sequence` reads as permissive under
+a careless probe and is not:
+
+| Strict (10)                                                                                                                             | Permissive (2)            |
+| --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
+| `capitalization`, `conditional`, `existence`, `metric`, `occurrence`, `readability`, `repetition`, `script`, `sequence`, `substitution` | `consistency`, `spelling` |
+
+**Why `sequence` misleads, and why it matters more than the answer.** Probe it
+with `extends: sequence` + `message` + an unknown key and Vale reports no
+invalid keys — so a probe that greps its output for `has invalid keys` scores it
+permissive. It is not: give the rule its `tokens` and it rejects the unknown key
+like every other strict check. What happens without `tokens` is worse than an
+`E201`:
+
+```
+panic: interface conversion: interface {} is nil, not []interface {}
+```
+
+The process dies. A panic contains no `has invalid keys` string, so the grep
+sees a clean run. **A measurement taken by grepping for an error string cannot
+tell "no error" from "no output".** Every verdict in the corpus is read from the
+exit status instead.
+
+That probe turned up three shapes that end the process rather than reporting
+anything — a `sequence` with no `tokens`, a `sequence` whose `tokens` is not a
+list, and a `metric` with a `formula` and no `condition`. They are a wider blast
+radius than `E201`: an `E201` names a file and a line, a panic names no rule at
+all and produces no findings for anything in the project. `verify` rejects all
+three, in a `checkFatalShapes` pass that runs after the field tables — a shape
+can only be fatal if every key in it was legal to begin with.
 
 ### The failure modes, re-measured
 
@@ -191,7 +223,13 @@ silent case is `scope`, alone — which makes `scope` the highest-value field in
 the schema for exactly the reason the decision above gives, and makes the other
 two a blast-radius argument rather than a silence argument.
 
+## Resolved during implementation
+
+- **Eleven check types or twelve?** Twelve, and the binary settles it rather than a judgement call — see _Measured against Vale 3.18.0_.
+- **How much of the per-check field table to encode.** The full table, as recommended, for ten of the twelve. `consistency` and `spelling` are encoded as **permissive**: measured, they accept any key at all, so there is no `E201` to make unreachable and a strict table there would only reject rules that work. That is the accept-when-unclear rule applied to a case where the measurement was in fact clear and went the other way.
+- **How strictly to treat a value the measurements left unclear.** Nothing was left unclear in the end. `figure.caption` looked ambiguous until the reach guard showed the control was at fault, and `meta` looked ambiguous until `frontmatter` turned out to be the real name — both resolved by measurement rather than by the fallback. The rule stands unused, which is the outcome to prefer.
+- **One place the schema is deliberately stricter than the binary.** `scope: ~fenced` fires on everything, because there is no `fenced` to subtract, so the binary "accepts" it in the only sense an exit code can express. What the author gets is a rule whose exclusion was silently deleted, which is precisely the class of failure this change exists to close, so `verify` rejects it. It is recorded as a `divergence` row in the corpus, asserted as a disagreement rather than skipped, so the exception is countable.
+
 ## Open Questions
 
-- **How much of the per-check field table to encode.** The full table makes `E201` unreachable; a partial one still leaves the engine-wide suppression possible for the fields it omits. Recommend the full table, since `E201` is the failure with the widest blast radius, but the cost is that every check type must be transcribed and measured rather than just the common header.
 - **Whether `verify` should also report each matcher's selected-file count.** Out of scope here, but adjacent: a matcher glob that reaches nothing produces a rule that is well formed, enabled, green, and inert — the one silent failure this change does not close.

openspec/changes/vale-authoring-polish/tasks.md → openspec/changes/archive/2026-08-25-vale-authoring-polish/tasks.md

@@ -21,45 +21,45 @@
 
 ## 3. The schema (unit 2)
 
-- [ ] 3.1 Author the schema from the measurements in group 1, alongside `packages/cli/src/generated/ast-grep-rule-schema.json`. Pin it to `VALE_VERSION`.
-- [ ] 3.2 Model the common header fields: `extends`, `message`, `level`, `scope`, `link`, `limit`, `action`, `description`, `name`. `vocab` is per-check, not common, and belongs in 3.4.
-- [ ] 3.3 Model `scope` as a grammar over an enum of operands, accepting a string, a list, `~`, and `&`.
-- [ ] 3.4 Model the per-check field tables, so a field belonging to another check type is rejected before `E201` can suppress the engine.
-- [ ] 3.5 Decide and record how strictly to treat a value whose status the measurements left unclear. The design's rule is to accept it.
+- [x] 3.1 Author the schema from the measurements in group 1, alongside `packages/cli/src/generated/ast-grep-rule-schema.json`. Pin it to `VALE_VERSION`. **Placed** at `packages/cli/src/schemas/vale-rule.ts` rather than under `generated/`: that directory is for artifacts a script fetches, and this is hand-authored. It sits beside `schemas/ast-grep-rule.ts`, which is where the ast-grep JSON schema is consumed.
+- [x] 3.2 Model the common header fields: `extends`, `message`, `level`, `scope`, `link`, `limit`, `action`, `description`, `name`. `vocab` is per-check, not common, and belongs in 3.4.
+- [x] 3.3 Model `scope` as a grammar over an enum of operands, accepting a string, a list, `~`, and `&`.
+- [x] 3.4 Model the per-check field tables, so a field belonging to another check type is rejected before `E201` can suppress the engine.
+- [x] 3.5 Decide and record how strictly to treat a value whose status the measurements left unclear. The design's rule is to accept it.
 
 ## 4. Wire it into verify
 
-- [ ] 4.1 Add the schema layer to the Vale verify path, reporting through the existing `LayerResult` and `VerifyResult` shapes so both engines fail the same way.
-- [ ] 4.2 Ensure the error names the field and the accepted values, rather than reporting a raw schema path.
-- [ ] 4.3 Confirm the layer runs before Vale is invoked, so a rule that would throw `E201` never reaches the binary.
-- [ ] 4.4 Confirm `test` still runs `verify` first and stops on its failure.
+- [x] 4.1 Add the schema layer to the Vale verify path, reporting through the existing `LayerResult` and `VerifyResult` shapes so both engines fail the same way.
+- [x] 4.2 Ensure the error names the field and the accepted values, rather than reporting a raw schema path.
+- [x] 4.3 Confirm the layer runs before Vale is invoked, so a rule that would throw `E201` never reaches the binary.
+- [x] 4.4 Confirm `test` still runs `verify` first and stops on its failure.
 
 ## 5. The corpus, and the differential test that makes the schema true
 
 Build this before the schema is finalized — the schema is derived from it. This is the largest group and the one that will be under-built if rushed.
 
-- [ ] 5.1 Design the corpus entry shape: a minimal rule YAML, the fixture that acts as its positive control, and the measured verdict. Keep it a declarative table so a Vale upgrade is a re-run rather than a re-authoring.
-- [ ] 5.2 Establish the three-outcome verdict, since an exit code cannot express it: **fired** (accepted), **did not fire** (the binary ignored the construct), **`E201`** (rejected outright). An unrecognized `extends` or `scope` parses clean, so "did not fire" is the signal that the construct is invalid.
-- [ ] 5.3 Write a positive control per entry. It depends on what the rule matches — a `scope: heading` entry needs a heading, `scope: table.cell` needs a table — so it cannot be generated from the check type. This is the bulk of the work.
-- [ ] 5.4 Guard against a vacuous entry: assert each control fires for at least one known-valid variant, so a control that can never fire is a test failure rather than a quiet pass.
-- [ ] 5.5 Cover every check type from 1.1, every scope operand from 1.2, and the per-check field tables from 1.3, including the `~` and `&` forms.
-- [ ] 5.6 Write the differential test: for every entry, assert `schemaAccepts === binaryAccepts`. Report both directions distinctly — schema-too-lax is the gap being closed, schema-too-strict blocks valid work and is the worse failure.
-- [ ] 5.7 Wire the corpus into `packages/cli/test/vale-vendor-contract.test.ts`, or a sibling beside it, following the existing convention of invoking the vendored binary directly rather than through `runVale`.
-- [ ] 5.8 Add the version-bump case: raising `VALE_VERSION` past a change in accepted check types or scopes fails a test that names the field.
-- [ ] 5.9 Verify the suite is non-vacuous by reverting the schema and watching exactly the expected entries fail.
+- [x] 5.1 Design the corpus entry shape: a minimal rule YAML, the fixture that acts as its positive control, and the measured verdict. Keep it a declarative table so a Vale upgrade is a re-run rather than a re-authoring.
+- [x] 5.2 Establish the three-outcome verdict, since an exit code cannot express it: **fired** (accepted), **did not fire** (the binary ignored the construct), **`E201`** (rejected outright). An unrecognized `extends` or `scope` parses clean, so "did not fire" is the signal that the construct is invalid.
+- [x] 5.3 Write a positive control per entry. It depends on what the rule matches — a `scope: heading` entry needs a heading, `scope: table.cell` needs a table — so it cannot be generated from the check type. This is the bulk of the work.
+- [x] 5.4 Guard against a vacuous entry: assert each control fires for at least one known-valid variant, so a control that can never fire is a test failure rather than a quiet pass.
+- [x] 5.5 Cover every check type from 1.1, every scope operand from 1.2, and the per-check field tables from 1.3, including the `~` and `&` forms.
+- [x] 5.6 Write the differential test: for every entry, assert `schemaAccepts === binaryAccepts`. Report both directions distinctly — schema-too-lax is the gap being closed, schema-too-strict blocks valid work and is the worse failure.
+- [x] 5.7 Wire the corpus into `packages/cli/test/vale-vendor-contract.test.ts`, or a sibling beside it, following the existing convention of invoking the vendored binary directly rather than through `runVale`.
+- [x] 5.8 Add the version-bump case: raising `VALE_VERSION` past a change in accepted check types or scopes fails a test that names the field. Two tripwires: the corpus differential names any construct whose verdict changed, and `vale-vendor-contract.test.ts` asks the binary to enumerate its own check types and compares that to `VALE_CHECK_TYPES`.
+- [x] 5.9 Verify the suite is non-vacuous by reverting the schema and watching exactly the expected entries fail. Five mutations run: dropping `figure.caption` and dropping `readability` each failed _too strict_ naming the value; adding `meta` and allowing `tokens` on `occurrence` each failed _too lax_; breaking a control document failed both vacuity guards.
 
 ## 6. Regression coverage
 
-- [ ] 6.1 Test that `extends: nonsense` fails `verify`, naming the field and the accepted values.
-- [ ] 6.2 Test that an unrecognized `scope` fails `verify`, and that `~` and `&` over recognized operands pass.
-- [ ] 6.3 Test that a foreign field for the declared check type fails `verify`.
-- [ ] 6.4 Test that every rule under `.taskless/rules/vale/` still verifies, so no valid rule regressed.
-- [ ] 6.5 Test that a rule with no fixtures still verifies, preserving the existing requirement.
+- [x] 6.1 Test that `extends: nonsense` fails `verify`, naming the field and the accepted values.
+- [x] 6.2 Test that an unrecognized `scope` fails `verify`, and that `~` and `&` over recognized operands pass.
+- [x] 6.3 Test that a foreign field for the declared check type fails `verify`.
+- [x] 6.4 Test that every rule under `.taskless/rules/vale/` still verifies, so no valid rule regressed. **Adapted:** this repository carries no committed Vale rules, so the test would be vacuous. It verifies the nine worked rules from `create-vale-rule` instead, which is the population the requirement is actually about — a rule an author was told to write.
+- [x] 6.5 Test that a rule with no fixtures still verifies, preserving the existing requirement.
 
 ## 7. Land it
 
-- [ ] 7.1 `pnpm typecheck`, `pnpm lint`, `pnpm test` pass.
-- [ ] 7.2 `pnpm build`, then author a deliberately-broken rule and confirm the real CLI reports what the specs require. Record the actual output.
-- [ ] 7.3 Extend the changeset on the bottom branch with unit 2's scope.
-- [ ] 7.4 Open the stack as two PRs merging forward, the recipe unit first.
-- [ ] 7.5 Archive the change on the tip PR.
+- [x] 7.1 `pnpm typecheck`, `pnpm lint`, `pnpm test` pass.
+- [x] 7.2 `pnpm build`, then author a deliberately-broken rule and confirm the real CLI reports what the specs require. Record the actual output.
+- [x] 7.3 Extend the changeset on the bottom branch with unit 2's scope.
+- [x] 7.4 Open the stack as two PRs merging forward, the recipe unit first.
+- [x] 7.5 Archive the change on the tip PR.

openspec/changes/archive/2026-08-25-vale-schema-generation/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-25