diff --git a/.changeset/vale-schema-generation.md b/.changeset/vale-schema-generation.md new file mode 100644 index 00000000..a344fb6e --- /dev/null +++ b/.changeset/vale-schema-generation.md @@ -0,0 +1,48 @@ +--- +"@taskless/cli": patch +--- + +Derive the Vale rule schema's vocabulary from the vendored binary instead of +transcribing it by hand. + +`pnpm generate:vale-schema` runs the pinned Vale against rules it writes itself +and emits `src/generated/vale-vocabulary.ts` — twelve check types, three levels, +ten per-check field tables, twenty-eight scope operands, two open scope +families — plus a divergence report. `src/schemas/vale-rule.ts` imports it and +stays what it was: the zod layer, the scope grammar, and the error messages that +explain blast radius to an author. Nothing about `verify`'s behavior changes; all +86 existing corpus rows pass against the generated schema unmodified, and two +were added to cover ground the generation newly measured. + +What a transcription lost was not the answer but the question. Every value in the +previous schema _was_ measured — by a script that was then discarded, leaving the +next person to raise `VALE_VERSION` with a failing test and no way to reproduce +the measurement it was failing against. + +The measuring is also where the errors live, so the generator is built around one +rule: **every verdict comes from the process exit status and the structured JSON +output, never from matching stdout against an error phrase.** A Go panic contains +no `has invalid keys` string, so a phrase-grep scores a crash as a clean run — +which is exactly how a tokenless `sequence` rule once came to look like a check +that validates nothing. A run's outcome is a closed set of `clean`, `diagnostic`, +`panic`, and `unrecognized`, and the last one is fatal at every call site. + +Two of the four vocabularies are self-enumerating: an unknown `extends` or +`level` makes the binary name its own accepted set. If either of those lines +stops matching, generation **fails** rather than emitting a short enum — a +truncated enum is _stricter_ than the binary, which is the direction that blocks +rules that would have worked. + +The other two are honest about their limit, and the artifact says so. `E201` +names the key you got wrong and never the ones you could have used, and an +unrecognized `scope` raises nothing at all — so field tables and scope operands +are **verified, not discovered**, from a candidate list seeded from four sources +with its provenance recorded. A scope verdict is three-valued, and a `scope: raw` +reach probe must fire on every fixture, so an operand cannot be dropped because +its fixture was never linted. + +Where the binary and Vale's documentation disagree, `vale-vocabulary-report.md` +records it rather than either side being quietly dropped: `meta` and +`meta.class.` are documented and never fire; `frontmatter` and +`frontmatter.` fire and are documented nowhere; `consistency` and `spelling` +validate no keys at all. diff --git a/openspec/changes/archive/2026-08-25-vale-schema-generation/.openspec.yaml b/openspec/changes/archive/2026-08-25-vale-schema-generation/.openspec.yaml new file mode 100644 index 00000000..e685d45e --- /dev/null +++ b/openspec/changes/archive/2026-08-25-vale-schema-generation/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-25 diff --git a/openspec/changes/archive/2026-08-25-vale-schema-generation/design.md b/openspec/changes/archive/2026-08-25-vale-schema-generation/design.md new file mode 100644 index 00000000..8f16709b --- /dev/null +++ b/openspec/changes/archive/2026-08-25-vale-schema-generation/design.md @@ -0,0 +1,110 @@ +# Design + +## The method rule, stated first because it is the whole risk + +**Every verdict comes from the process exit status and the structured JSON output. Never from grepping stdout for an error phrase.** + +This is not style guidance. It has already produced one wrong answer in this project: + +| Probe | Verdict | +| -------------------------------------------------- | --------------------------------------- | +| `sequence`, bare rule, grep for `has invalid keys` | "accepts any key" — permissive | +| `sequence`, bare rule, actual behavior | `panic: interface conversion: … is nil` | + +A Go panic contains no `has invalid keys` string, so a phrase-grep scores a crash as a clean run. Run Vale with `--output=JSON`; on a config error it writes `{Line, Path, Text, Code, Span}` to **stderr**. The generator therefore models the outcome of a run as a closed set with no escape hatch: + +``` +clean status 0, stdout parses as a findings map +diagnostic status ≠ 0, stderr parses as Vale's JSON object, keyed on Code +panic stderr contains a Go panic header +unrecognized everything else +``` + +`unrecognized` is fatal at every call site. There is deliberately no branch that folds an unfamiliar shape into "fine", because that is the failure the corpus caught the first time and cannot be relied on to catch the second. + +## What is derived and what is seeded + +Four vocabularies, and they do not have equal standing. + +| Vocabulary | Oracle | Discovers? | +| ---------------- | -------------------------------------------------- | ---------- | +| Check types (12) | `'extends' key must be one of [...]` | **yes** | +| Levels (3) | `'level' must be one of [...]` | **yes** | +| Per-check fields | `has invalid keys: ''` — names the _bad_ key | no | +| Scope operands | none; an unknown scope is silent | no | + +The top two are self-enumerating: give the binary a sentinel and it names its own accepted set. A Vale release that adds a check type is picked up rather than merely failing a test. + +The bottom two are **verified, not discovered**, and the difference is load-bearing. `E201` names the key you got wrong and never the ones you could have used, so a field table can only be built by proposing a candidate and asking. A real field nobody proposes is absent from the artifact, and the schema then rejects a rule Vale accepts — the too-strict direction, which the design of the schema itself ranks as the worse failure. The candidate list is therefore seeded from four independent sources and its provenance is documented in the generator: + +1. Vale's published key documentation, **including names the binary rejects** — `prefixes`, `suffixes`, and `ignorecase` on `capitalization` stay in the list precisely so their rejection is a recorded finding rather than an omission. +2. The hand transcription this replaces, so the generator is a superset of what a human already established. +3. Vale's shared keys: `link`, `limit`, `action`, `scope`, `description`, `name`, `comment`, `vocab`. +4. Every check's fields offered to every other check. That cross-probe is what makes the per-check tables a measured _partition_ rather than twelve unrelated lists. + +## Membership, and the two inferences + +The probe sets a candidate key to an arbitrary value (`true`) and reads the outcome as evidence about the **key**, not the value: + +| Outcome | Verdict | Why | +| ----------------------------------------- | ---------------- | --------------------------------------------------------- | +| clean | member | the key decoded | +| `has invalid keys` naming this field | **not** a member | the only negative oracle Vale offers | +| any other `E201` | member | Vale knew the key and objected to the value | +| panic | member | an unknown key is collected as unused, never dereferenced | +| invalid-key list naming a _different_ key | fatal | the base rule is contaminated | + +The last two rows of the positive column are inferences rather than readings, so both are collected and written into the divergence report. On Vale 3.18.0 the type-complaint case fires ten times, all `action: expected a map, got 'bool'`, and the panic case fires zero times. An inference nobody can audit is a grep with better manners; naming them is what keeps the distinction real. + +Two probes are excluded by construction rather than measured: + +- **The header keys.** `extends`, `message` and `level` are in every base rule and were verified running clean. Probing them is also unsound: `extends: true` reaches an `interface{}` → `string` conversion and panics. +- **The sentinel.** The one probe where a panic is _not_ read as evidence. It decides whether a check validates keys at all, and a permissive verdict taken from a crash would make the schema loose for a check that is strict — which is precisely the `sequence` error, re-run. + +Each check needs a **base rule**: a minimal working rule of that check to add the candidate to. That is hand-seeded and has to be, since several checks are not valid empty. Every base is verified clean before a single verdict is taken against it. + +**Common fields are derived, not declared.** They are the _intersection_ of the ten strict checks' measured tables. That is a stronger statement than a hand-written list: nobody had to remember that `vocab` is per-check because five checks reject it — the intersection simply does not contain it. The permissive checks are excluded because they accept everything and so constrain nothing. + +## Scope: three-valued, and only a fixture separates two of the values + +`scope` has no oracle at all. An invalid scope does not fail — the rule loads, runs, and matches nothing, which from outside is identical to a valid scope whose construct is absent from the document. So the verdict is three-valued: + +| Verdict | What happened | +| ------------- | ------------------------------------------------- | +| `fires` | the rule flagged the fixture: the operand is real | +| `silent` | the fixture was linted and the rule found nothing | +| `unreachable` | the fixture was never linted at all | + +Only a hand-written fixture separates the middle from the last, and the generator refuses to guess: every fixture contains the word `bogus`, a `scope: raw` **reach probe** must fire on it, and if it does not the run is fatal. That guard is not theoretical — `figure.caption` measures `silent` when its fixture nests the caption inside a `
`, and so does `scope: text` over the same document, because Vale drops everything inside that element. Without the reach guard the operand would be dropped as one Vale ignores, and every rule using it would then fail `verify`. + +The file extension is part of the fixture, not a detail: it decides which parser Vale routes the document to, and `comment.*` only exists in a source tier. + +## Divergences are reported, not resolved + +Where the binary and the documentation disagree, a generator that dropped the finding would be quietly deciding which to believe. The artifact carries `VALE_DIVERGENCES` and the run writes `vale-vocabulary-report.md`. Measured on 3.18.0: + +- **`meta` and `meta.class.` are documented and never fire.** Omitted from the vocabulary, so `verify` rejects them. A rule written from the docs would otherwise be inert forever with no error anywhere. +- **`frontmatter` and `frontmatter.` fire and are documented nowhere.** Included. This is the standing proof that a candidate list verifies rather than discovers: nobody proposing from the documentation alone would have found them. +- **`consistency` and `spelling` validate no keys at all**, measured with a sentinel. The schema stays permissive there; being strict would reject rules the binary runs. +- **Ten field probes drew a type complaint rather than a key rejection**, all on `action`. Recorded as members, and named so the inference is auditable. + +Three claims carried into this change from earlier measurement did **not** reproduce, and the report is where that surfaced: + +- `figure.caption` does fire, on a _bare_ `
` in an `.html` document. What never fires is the nested-in-`
` form, and the earlier note conflated the two. +- `comment.block` fires — in `.js` and in `.ts` alike. It does not "never fire while `comment.line` catches both". +- `comment.*` does not need a `.ts` fixture specifically; `.js` reaches the same tier. Rows for both now exist in the corpus, because "fires in a source file" and "fires in _this_ source file" are different claims and the schema makes the wider one. + +## What stays hand-written, and why each is not a generation failure + +- **The three fatal shapes.** A tokenless `sequence`, a `sequence` whose `tokens` is not a list, a `metric` with a `formula` and no `condition`. Every key in these rules is a legal field of its check — it is the _shape_ that is fatal, so a field table cannot express it. A panic is also a wider blast radius than `E201`: no rule name, no findings for anything, nothing to act on. +- **The scope grammar.** `~` negation and `&` chaining are operators around the operands, not values in the enum. Rejecting `~fenced` is a deliberate business rule, not a transcription, and the corpus carries it as a recorded divergence. +- **The union members.** They are spelled out rather than mapped over the generated table, because that is what carries each `z.literal` into `ValeRule`; a mapped union infers `extends: string`. The cost is a hand-maintained list beside a derived one, so an import-time guard requires every derived check type to appear exactly once and to be classified strict or permissive. The failure is a sentence naming the check. +- **The version pin.** The artifact records the version it was derived from, and `vale-rule.ts` asserts it against `VALE_VERSION` — as a **conditional type**, not an `if`. Both are string literal types, so the compiler settles it and a mismatch is a build failure naming the line, rather than a throw in front of a user on whichever command loads the schema first. + +## Alternatives considered + +**Generate the zod schema itself.** Rejected. The interesting content of `vale-rule.ts` is not the enums; it is the error messages that explain blast radius to an author, the two-stage `.pipe()` that reproduces the binary's own order, and the case-folding transform. Generating that would mean maintaining a code emitter to reproduce prose. Generating only the vocabulary keeps the derived part small and the reviewed part readable. + +**Emit JSON, like `ast-grep-rule-schema.json`.** Rejected. That file is JSON because upstream publishes JSON. Here the generator authors the artifact, and a `.ts` file gets `as const` — which is what makes `VALE_CHECK_TYPES` a literal union rather than `string[]`, and the import-time guard checkable at all. + +**Keep the transcription and add a checked-in probe script.** Rejected: two sources of truth that agree only by discipline. The failure mode is the one already observed — the answer survives, the question does not. diff --git a/openspec/changes/archive/2026-08-25-vale-schema-generation/proposal.md b/openspec/changes/archive/2026-08-25-vale-schema-generation/proposal.md new file mode 100644 index 00000000..54ca7420 --- /dev/null +++ b/openspec/changes/archive/2026-08-25-vale-schema-generation/proposal.md @@ -0,0 +1,48 @@ +## Why + +`vale-authoring-polish` landed a Vale rule schema and called it, in its own module comment, **a transcription**: twelve check types, three levels, ten field tables and twenty-eight scope operands read off a binary by hand. + +Every one of those was measured — by a script that was then thrown away. What survives is the answer, not the question. The next person to raise `VALE_VERSION` gets a test failure naming a field and no way to reproduce the measurement it is failing against. + +The measuring is also where the errors live, and one is on the record. A probe grepping stdout for `has invalid keys` reads a Go panic as a clean run — a panic contains no such string — which is how a tokenless `sequence` rule came to look like a check that validates nothing: + +``` +sequence, bare rule, probe grepping for "has invalid keys" -> read as "accepts any key" +sequence, bare rule, actual behavior -> panic: interface conversion +``` + +A method that can make that mistake is worth writing down once, correctly. + +## What Changes + +- **A generator derives the vocabulary from the vendored binary.** `pnpm generate:vale-schema` writes `packages/cli/src/generated/vale-vocabulary.ts`, a checked-in artifact pinned to `VALE_VERSION`, mirroring how `fetch-ast-grep-schema.ts` produces `ast-grep-rule-schema.json`. `vale-rule.ts` imports it and keeps zod as the validation layer. +- **Verdicts come from the exit status and the structured JSON, never from a phrase match.** Vale's `--output=JSON` emits `{Line, Path, Text, Code, Span}` on stderr for a config error. The generator keys on `Code`, parses `Text` structurally, detects panics explicitly, and treats any outcome it does not recognize as fatal. +- **The generator fails loudly rather than emitting a short enum.** If the `'extends' key must be one of [...]` line stops matching, it errors. A truncated enum is _stricter_ than the binary, which is the direction that blocks working rules. +- **A divergence report ships with the artifact.** Where the binary and Vale's documentation disagree, the disagreement is written to `vale-vocabulary-report.md` rather than silently dropped on one side or the other. +- **The three fatal-shape checks stay hand-written.** A `sequence` with no `tokens`, a `sequence` whose `tokens` is not a list, and a `metric` with a `formula` and no `condition` each panic the binary. That is behavior a field table cannot express, so it stays a `.check()` beside the union. +- **No behavior change.** All 86 existing corpus rows pass unmodified against the generated schema; two rows are added to cover ground the generation newly measured. + +## Capabilities + +### New Capabilities + +None. This changes how an existing capability's data is produced. + +### Modified Capabilities + +- `cli-rule-validation`: the requirement "The Vale rule schema is pinned to the vendored binary" says the schema "SHALL be authored in this repository" and calls it a transcription. It becomes a derivation: the vocabulary SHALL be generated from the vendored binary by a script in the repository, and the requirement gains the two constraints that make a derivation trustworthy — fail loudly on an unrecognized error shape, and report rather than drop a divergence. + +## Impact + +- `packages/cli/scripts/generate-vale-schema.ts` — new. +- `packages/cli/src/generated/vale-vocabulary.ts` and `vale-vocabulary-report.md` — new, checked in. +- `packages/cli/src/schemas/vale-rule.ts` — the enums and field tables are replaced by imports; the zod construction, the scope grammar, and the fatal-shape checks are unchanged. +- `packages/cli/package.json` — one script. +- `packages/cli/test/vale-corpus.ts` — two rows added, none modified. +- Refs #171. + +## Delivery shape + +**Single PR.** The generator, the artifact it produces, and the schema's switch to importing it are one reviewable diff and are only correct together: the artifact is meaningless without the generator that reproduces it, and `vale-rule.ts` does not compile without the artifact. There is no intermediate state that reaches production safely, and no unit that a reviewer would be better off seeing alone. + +It stacks on `openspec/vale-authoring-polish-schema` (#175), whose schema it rewrites, so it is the tip of that stack and archives the change. diff --git a/openspec/changes/archive/2026-08-25-vale-schema-generation/specs/cli-rule-validation/spec.md b/openspec/changes/archive/2026-08-25-vale-schema-generation/specs/cli-rule-validation/spec.md new file mode 100644 index 00000000..bf0de1ba --- /dev/null +++ b/openspec/changes/archive/2026-08-25-vale-schema-generation/specs/cli-rule-validation/spec.md @@ -0,0 +1,45 @@ +## MODIFIED Requirements + +### Requirement: The Vale rule schema is pinned to the vendored binary + +The Vale rule schema's vocabulary SHALL be derived from the vendored binary by a generator in this repository, written to a checked-in artifact pinned to `VALE_VERSION`, and a differential test SHALL hold the schema's claims against that binary. + +Vale publishes no JSON Schema for its check types. The machine-readable field knowledge exists only behind its hosted MCP server, which is a paid product and unavailable to `verify`. The binary, not the documentation, SHALL therefore be the authority for what the schema asserts — and it SHALL be asked by a script that can be re-run, rather than by hand once. What a transcription loses is not the answer but the question: a later reader inherits a value with no way to reproduce the measurement that produced it. + +The generator SHALL take every verdict from the process exit status and the structured JSON output, and SHALL NOT take one by matching output against an error phrase. A configuration error is reported on stderr as an object carrying a `Code`; a crash produces no such object at all. A probe that matches on a phrase scores a crash as a clean run, which has already produced a wrong entry in this schema. + +Where the derivation rests on a proposed candidate rather than on a set the binary enumerates, that SHALL be recorded as a limit of the artifact. Vale names the key an author got wrong and never the ones they could have used, and an unrecognized scope raises nothing at all — so field tables and scope operands are verified rather than discovered, and a value nobody proposes is absent. + +#### Scenario: A Vale upgrade that invalidates the schema fails loudly + +- **WHEN** `VALE_VERSION` is raised to a version whose accepted check types, fields, or scopes differ from the artifact +- **THEN** the differential test SHALL fail +- **AND** the failure SHALL name the construct whose treatment changed + +#### Scenario: The generator refuses to emit a vocabulary it cannot read + +- **WHEN** the binary's enumeration of its own accepted values no longer matches the shape the generator parses +- **THEN** the generator SHALL fail with an error naming what it could not read +- **AND** it SHALL NOT emit a partial enumeration + +A short enum is _stricter_ than the binary, so a silent partial parse would begin rejecting rules Vale accepts — blocking work that would have functioned, which is the worse of the two failure directions. + +#### Scenario: A divergence from Vale's documentation is reported, not dropped + +- **WHEN** the binary's measured behavior disagrees with Vale's published documentation, in either direction +- **THEN** generation SHALL emit that disagreement as part of its output +- **AND** the artifact SHALL record which side the schema followed + +A documented value that never fires is a trap an author walks into with the docs open. An undocumented value that works is evidence that something real may be missing from the candidate list. Neither may be resolved silently. + +#### Scenario: The vocabulary describes the binary the build ships + +- **WHEN** the checked-in artifact records a different Vale version than `VALE_VERSION` +- **THEN** the build SHALL fail +- **AND** the failure SHALL name the two versions + +#### Scenario: Behavior a schema cannot express stays explicit + +- **WHEN** a rule shape crashes the binary although every key in it is a legal field of its check +- **THEN** the schema SHALL reject that shape through a check stated alongside the generated field tables +- **AND** the rejection SHALL NOT be encoded as a field-table fact diff --git a/openspec/changes/archive/2026-08-25-vale-schema-generation/tasks.md b/openspec/changes/archive/2026-08-25-vale-schema-generation/tasks.md new file mode 100644 index 00000000..b99d5f69 --- /dev/null +++ b/openspec/changes/archive/2026-08-25-vale-schema-generation/tasks.md @@ -0,0 +1,58 @@ +## 1. The probe, built to the method rule + +- [x] 1.1 Run Vale with `--output=JSON` in an isolated temp config (`BasedOnStyles =`, `--no-exit`) and model one run's outcome as a closed set: `clean`, `diagnostic`, `panic`, `unrecognized`. No branch folds an unfamiliar shape into "fine". +- [x] 1.2 Parse the config-error object off **stderr** and key on `Code`. Confirmed the shape on 3.18.0: `{Line, Path, Text, Code: "E201", Span}`. +- [x] 1.3 Make `unrecognized` fatal at every call site, and detect panics explicitly rather than by the absence of a phrase. +- [x] 1.4 Resolve the binary through the shipped `findValeBinary`, and refuse to run if its self-reported version differs from `VALE_VERSION`. + +## 2. Derive the self-enumerating vocabularies + +- [x] 2.1 Read the check types out of `'extends' key must be one of [...]`. Twelve on 3.18.0. +- [x] 2.2 Read the levels out of `'level' must be one of [...]`. Three, left in the binary's own order because that order is severity. +- [x] 2.3 Throw rather than emit a short enum if either line stops matching, with a message saying why a truncated enum is the worse failure. Verified by hand against a deliberately broken pattern. + +## 3. Derive the per-check field tables + +- [x] 3.1 Hand-seed a minimal working base rule per check, and verify each runs clean before any verdict is taken against it. Several checks are not valid empty. +- [x] 3.2 Seed the candidate universe from four sources and document the provenance in the generator, including names the binary rejects. +- [x] 3.3 Cross-probe every check's fields against every other check, so the tables are a measured partition. +- [x] 3.4 Read membership from the outcome set: clean → member, `has invalid keys` naming the field → not a member, any other `E201` → member, panic → member. Record the last two as auditable inferences. Ten type complaints on 3.18.0 (all `action`), zero panics. +- [x] 3.5 Exclude the header keys from probing — they are members by construction, and `extends: true` panics. +- [x] 3.6 Detect the permissive checks with a sentinel key, and do **not** read a panic as evidence there. `consistency` and `spelling`. +- [x] 3.7 Derive the common fields as the intersection of the strict tables rather than declaring them. + +## 4. Derive the scope operands + +- [x] 4.1 Write a fixture per candidate carrying its construct, with the extension that routes it to the right parser. +- [x] 4.2 Run a `scope: raw` reach probe on every fixture and make an unreachable fixture fatal. +- [x] 4.3 Record the three-valued verdict per candidate, including negative controls (`fenced`, `banana`, `heading.h7`, `table.row`). +- [x] 4.4 Include an operand only if it fired; record `documented` per candidate so a disagreement can be stated. + +## 5. Emit + +- [x] 5.1 Write `src/generated/vale-vocabulary.ts` — check types, levels, permissive checks, common fields, per-check tables, scope operands, scope prefixes, divergences — with no timestamp, so the artifact is a pure function of the binary. +- [x] 5.2 Write `src/generated/vale-vocabulary-report.md`: what was derived versus seeded, the divergences, and every scope probed. +- [x] 5.3 Format both through the repository's own prettier, so a re-generation is a no-op diff. +- [x] 5.4 Add `pnpm generate:vale-schema`. +- [x] 5.5 Verify idempotence: deleted both artifacts, re-ran from clean, byte-identical. + +## 6. Wire the schema to the artifact + +- [x] 6.1 Replace the hand-maintained enums and field tables in `vale-rule.ts` with imports. Keep zod as the validation layer. +- [x] 6.2 Keep the scope grammar, the case-folding transform, the two-stage `.pipe()`, and the three fatal-shape checks exactly as they were. +- [x] 6.3 Assert the artifact's version against `VALE_VERSION` as a conditional type, so a mismatch is a build failure rather than a runtime throw. Verified by flipping the artifact's version and watching `tsc` fail naming the line. +- [x] 6.4 Guard the spelled-out union against the derived check types at import: every derived type present exactly once, and classified strict or permissive. + +## 7. Prove it against the corpus + +- [x] 7.1 All 86 existing corpus rows pass unmodified. No row edited to fit the output. +- [x] 7.2 Add rows to cover ground the generation newly measured: `comment.line` and `comment.block` in the TypeScript tier. 88 rows. +- [x] 7.3 Record the three earlier claims that did not reproduce — `figure.caption`, `comment.block`, and the `.ts` requirement — in design.md. +- [x] 7.4 `pnpm typecheck`, `pnpm lint`, `pnpm test` green. +- [x] 7.5 `pnpm build`, then verify the real CLI by hand on a broken rule. + +## 8. Land + +- [x] 8.1 One changeset on this branch. +- [x] 8.2 Archive the change — this is the tip of the stack. +- [x] 8.3 Open the PR against `openspec/vale-authoring-polish-schema`, referencing #171. diff --git a/openspec/specs/cli-rule-validation/spec.md b/openspec/specs/cli-rule-validation/spec.md index 91849ef0..fea2ff1d 100644 --- a/openspec/specs/cli-rule-validation/spec.md +++ b/openspec/specs/cli-rule-validation/spec.md @@ -121,17 +121,44 @@ The rule generation loop SHALL run `verify` and then `test` against a newly auth ### Requirement: The Vale rule schema is pinned to the vendored binary -The Vale rule schema SHALL be authored in this repository and pinned to `VALE_VERSION`, and a vendor-contract test SHALL hold its claims against the vendored binary. +The Vale rule schema's vocabulary SHALL be derived from the vendored binary by a generator in this repository, written to a checked-in artifact pinned to `VALE_VERSION`, and a differential test SHALL hold the schema's claims against that binary. -Vale publishes no JSON Schema for its check types. The machine-readable field knowledge exists only behind its hosted MCP server, which is a paid product and unavailable to `verify`. The schema is therefore a transcription, and a transcription drifts — so the binary, not the documentation, SHALL be the authority for what the schema asserts. +Vale publishes no JSON Schema for its check types. The machine-readable field knowledge exists only behind its hosted MCP server, which is a paid product and unavailable to `verify`. The binary, not the documentation, SHALL therefore be the authority for what the schema asserts — and it SHALL be asked by a script that can be re-run, rather than by hand once. What a transcription loses is not the answer but the question: a later reader inherits a value with no way to reproduce the measurement that produced it. + +The generator SHALL take every verdict from the process exit status and the structured JSON output, and SHALL NOT take one by matching output against an error phrase. A configuration error is reported on stderr as an object carrying a `Code`; a crash produces no such object at all. A probe that matches on a phrase scores a crash as a clean run, which has already produced a wrong entry in this schema. + +Where the derivation rests on a proposed candidate rather than on a set the binary enumerates, that SHALL be recorded as a limit of the artifact. Vale names the key an author got wrong and never the ones they could have used, and an unrecognized scope raises nothing at all — so field tables and scope operands are verified rather than discovered, and a value nobody proposes is absent. #### Scenario: A Vale upgrade that invalidates the schema fails loudly -- **WHEN** `VALE_VERSION` is raised to a version whose accepted check types or scopes differ from the schema -- **THEN** the vendor-contract test SHALL fail -- **AND** the failure SHALL name the field whose accepted values changed +- **WHEN** `VALE_VERSION` is raised to a version whose accepted check types, fields, or scopes differ from the artifact +- **THEN** the differential test SHALL fail +- **AND** the failure SHALL name the construct whose treatment changed + +#### Scenario: The generator refuses to emit a vocabulary it cannot read + +- **WHEN** the binary's enumeration of its own accepted values no longer matches the shape the generator parses +- **THEN** the generator SHALL fail with an error naming what it could not read +- **AND** it SHALL NOT emit a partial enumeration + +A short enum is _stricter_ than the binary, so a silent partial parse would begin rejecting rules Vale accepts — blocking work that would have functioned, which is the worse of the two failure directions. + +#### Scenario: A divergence from Vale's documentation is reported, not dropped + +- **WHEN** the binary's measured behavior disagrees with Vale's published documentation, in either direction +- **THEN** generation SHALL emit that disagreement as part of its output +- **AND** the artifact SHALL record which side the schema followed + +A documented value that never fires is a trap an author walks into with the docs open. An undocumented value that works is evidence that something real may be missing from the candidate list. Neither may be resolved silently. + +#### Scenario: The vocabulary describes the binary the build ships + +- **WHEN** the checked-in artifact records a different Vale version than `VALE_VERSION` +- **THEN** the build SHALL fail +- **AND** the failure SHALL name the two versions -#### Scenario: The schema is not derived from documentation alone +#### Scenario: Behavior a schema cannot express stays explicit -- **WHEN** a value is added to the schema's `extends` or `scope` enumerations -- **THEN** it SHALL be one the vendored binary was measured accepting +- **WHEN** a rule shape crashes the binary although every key in it is a legal field of its check +- **THEN** the schema SHALL reject that shape through a check stated alongside the generated field tables +- **AND** the rejection SHALL NOT be encoded as a field-table fact diff --git a/packages/cli/package.json b/packages/cli/package.json index 990d2d27..eaa8ae5d 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -15,6 +15,7 @@ "generate:api": "openapi-typescript https://app.taskless.io/cli/api/__schema -o src/generated/api.d.ts", "generate:ast-grep-schema": "tsx scripts/fetch-ast-grep-schema.ts", "generate:rule-hash-vectors": "tsx scripts/fetch-rule-hash-vectors.ts", + "generate:vale-schema": "tsx scripts/generate-vale-schema.ts", "prebuild": "tsx scripts/fetch-rule-hash-vectors.ts", "test": "vitest run", "typecheck": "tsc --noEmit" diff --git a/packages/cli/scripts/generate-vale-schema.ts b/packages/cli/scripts/generate-vale-schema.ts new file mode 100644 index 00000000..4cd6d3bf --- /dev/null +++ b/packages/cli/scripts/generate-vale-schema.ts @@ -0,0 +1,1338 @@ +/** + * Derive the Vale rule vocabulary from the vendored binary. + * + * Vale publishes no JSON Schema, so `fetch-ast-grep-schema.ts` has nothing to + * fetch here. What it does have is a binary that answers questions, and this + * script asks them: every value in `src/generated/vale-vocabulary.ts` is the + * recorded answer of the pinned Vale to a rule this script wrote and ran. + * + * ## The method rule + * + * **Every verdict comes from the process exit status and the structured JSON + * Vale emits, never from grepping output for an error phrase.** That is not + * style advice. A probe that greps stdout for `has invalid keys` reads a Go + * panic — which contains no such string — as a clean run, and that is exactly + * how a tokenless `sequence` rule once came to look like a check that + * validates nothing. Run with `--output=JSON`; on a config error Vale writes + * `{Line, Path, Text, Code, Span}` to **stderr**. Key on `Code`, parse `Text` + * structurally, and treat every outcome that does not match a known shape as + * fatal rather than folding it into "fine". + * + * ## What is derived and what is seeded + * + * Three of the four vocabularies are self-enumerating — the binary names its + * own accepted set in the error it raises — and one is not: + * + * | Vocabulary | Oracle | Discovers? | + * | ---------------- | ----------------------------------------------------- | ---------- | + * | Check types | `'extends' key must be one of [...]` | yes | + * | Levels | `'level' must be one of [...]` | yes | + * | Per-check fields | `has invalid keys: ''` — names the *bad* key | no | + * | Scope operands | none: an unknown scope is silent | no | + * + * The bottom two rows are the honest limit of this script. `E201` names the key + * you got wrong and never the ones you could have used, so a field table can + * only be built by proposing a candidate and asking. **That verifies; it cannot + * discover.** A real field nobody proposes is omitted from the artifact, and + * the schema then rejects a rule Vale accepts — the too-strict direction, which + * the design calls the worse failure. {@link FIELD_CANDIDATES} is therefore + * seeded generously and carries its provenance. + * + * `scope` is worse still, because it has no oracle at all: an invalid scope is + * silent, and so is a valid scope whose construct is missing from the fixture. + * The verdict is three-valued, and only a hand-written fixture separates the + * last two. See {@link SCOPE_CANDIDATES}. + * + * ## Failing loudly + * + * If the enumeration line stops matching its pattern, this script errors rather + * than emitting a short enum. A truncated enum is *stricter* than the binary, + * which is the failure direction that blocks working rules — so a parse that no + * longer matches must never be allowed to look like a small vocabulary. + */ + +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +import type { ValeConfigError } from "../src/rules/vale/map.js"; + +const scriptDirectory = dirname(fileURLToPath(import.meta.url)); +const PACKAGE_ROOT = resolve(scriptDirectory, ".."); +const OUTPUT_PATH = resolve( + PACKAGE_ROOT, + "src", + "generated", + "vale-vocabulary.ts" +); +const REPORT_PATH = resolve( + PACKAGE_ROOT, + "src", + "generated", + "vale-vocabulary-report.md" +); + +// --- Locating the binary ----------------------------------------------------- + +/** + * The vendored binary, resolved the same way the CLI resolves it at runtime. + * + * `findValeBinary` lives in `src/`, which this script can import directly: it + * runs under `tsx`, not under the bundle. Using the shipped resolver rather + * than a second copy of the platform-package naming means a packaging change + * cannot leave the generator probing a different Vale than the one `verify` + * will hand rules to. + */ +const { findValeBinary } = await import("../src/rules/vale/binary.js"); +const { VALE_VERSION } = await import("../src/rules/capabilities.js"); +const { asValeConfigError } = await import("../src/rules/vale/map.js"); + +const resolution = findValeBinary(); +if (resolution.path === undefined) { + throw new Error( + "no vendored Vale binary on this host, so there is nothing to derive from. " + + "Run `pnpm install` in packages/cli and retry." + ); +} +const VALE = resolution.path; + +{ + const probe = spawnSync(VALE, ["--version"], { encoding: "utf8" }); + const reported = /vale version (\d+\.\d+\.\d+)/.exec(probe.stdout ?? "")?.[1]; + if (reported !== VALE_VERSION) { + throw new Error( + `the vendored binary reports ${reported ?? "an unreadable version"} but ` + + `VALE_VERSION is pinned to ${VALE_VERSION}. Derivation against a ` + + `different binary than the one the schema claims to describe is the ` + + `drift this artifact exists to prevent.` + ); + } +} + +// --- The probe --------------------------------------------------------------- + +/** + * What one run of the binary did, as a closed set of outcomes. + * + * There is deliberately no "other" member that a caller can shrug at. Every + * shape this script does not recognize becomes {@link Unrecognized}, and every + * consumer of a verdict must decide what to do about it — which in practice + * means throwing. + */ +type ProbeOutcome = + | { kind: "clean"; findings: number } + | { kind: "diagnostic"; diagnostic: ValeConfigError } + | { kind: "panic"; trace: string } + | { + kind: "unrecognized"; + status: number | null; + stdout: string; + stderr: string; + }; + +/** + * Run one rule over one document in a config isolated from everything else. + * + * `BasedOnStyles =` is load-bearing. Without it Vale loads its bundled styles + * and a control document can trip one of those, which a finding count cannot + * tell apart from the rule under test firing. + * + * `--no-exit` suppresses the non-zero status Vale returns merely for *finding* + * something, so a non-zero status here means the config itself failed. + * + * **This config is the sibling of `buildIsolatingConfig` in + * `src/rules/vale/verify.ts`, and the two are kept deliberately separate.** + * They agree on three details: `MinAlertLevel = suggestion`, an empty + * `BasedOnStyles =`, and exactly one assignment of the enabled key in exactly + * one matcher, the last two of which that docstring explains are load-bearing. + * They differ on the fourth, which is why this is not a call to that function: + * `buildIsolatingConfig` needs an absolute `StylesPath` because it writes its + * config to a temp directory while the styles stay in the user's + * `.taskless/rules/vale/` tree, whereas this + * probe owns the whole temp directory and points at a `styles/probe/probe.yml` + * beside the config. Reuse would mean staging a fake `.taskless` layout in tmp + * just to satisfy a path convention no probe has. If a future requirement is + * discovered against either recipe (another key Vale needs to isolate a rule), + * apply it to both. + */ +function probe( + rule: string, + document: string, + extension: string +): ProbeOutcome { + const cwd = mkdtempSync(join(tmpdir(), "vale-derive-")); + try { + mkdirSync(join(cwd, "styles", "probe"), { recursive: true }); + writeFileSync(join(cwd, "styles", "probe", "probe.yml"), rule); + writeFileSync( + join(cwd, ".vale.ini"), + "StylesPath = styles\nMinAlertLevel = suggestion\n\n[*]\nBasedOnStyles =\nprobe.probe = YES\n" + ); + writeFileSync(join(cwd, `doc.${extension}`), document); + + const result = spawnSync( + VALE, + [ + "--config", + ".vale.ini", + "--output=JSON", + "--no-exit", + "--", + `doc.${extension}`, + ], + { cwd, encoding: "utf8" } + ); + + const stdout = result.stdout ?? ""; + const stderr = result.stderr ?? ""; + + if (stderr.includes("panic:")) return { kind: "panic", trace: stderr }; + + if (result.status === 0) { + const payload: unknown = JSON.parse(stdout || "{}"); + if ( + Array.isArray(payload) || + typeof payload !== "object" || + payload === null + ) { + return { kind: "unrecognized", status: result.status, stdout, stderr }; + } + const findings = Object.values( + payload as Record + ).flat(); + return { kind: "clean", findings: findings.length }; + } + + const diagnostic = readDiagnostic(stderr); + if (diagnostic !== undefined) return { kind: "diagnostic", diagnostic }; + + return { kind: "unrecognized", status: result.status, stdout, stderr }; + } finally { + rmSync(cwd, { recursive: true, force: true }); + } +} + +/** + * Read Vale's diagnostic object off stderr. + * + * Vale writes one JSON object per config error, pretty-printed and possibly + * several in a row. `JSON.parse` handles the single-object case; anything else + * returns `undefined` and becomes {@link ProbeOutcome} `unrecognized`, which + * every caller treats as fatal. That is the point: an unreadable diagnostic + * must never be mistaken for a clean run. + * + * The shape check itself is {@link asValeConfigError}, the one the CLI already + * runs over the same payload at runtime. Deriving against a second local copy + * of `{Code, Text, ...}` would mean a future change to Vale's diagnostic + * envelope has to be found twice, and nothing would report the half that was + * missed. + */ +function readDiagnostic(stderr: string): ValeConfigError | undefined { + const trimmed = stderr.trim(); + if (trimmed === "") return undefined; + let parsed: unknown; + try { + parsed = JSON.parse(trimmed); + } catch { + return undefined; + } + return asValeConfigError(Array.isArray(parsed) ? parsed[0] : parsed); +} + +/** + * Read a value the generator itself produced, refusing `undefined`. + * + * `noUncheckedIndexedAccess` is right to insist on this rather than being + * silenced with a `!`. A missing entry means a check type was lost between two + * stages of the derivation, and the alternative to throwing is emitting a + * vocabulary with a silently empty field table — which is the too-strict + * failure, dressed up as a successful run. + */ +function must(value: T | undefined, what: string): T { + if (value === undefined) { + throw new Error( + `internal: ${what} is missing, so the derivation is incomplete and the ` + + `artifact would be emitted short. This is a bug in the generator.` + ); + } + return value; +} + +/** A probe outcome this script has no handling for is never survivable. */ +function fatal(context: string, outcome: ProbeOutcome): never { + const detail = + outcome.kind === "panic" + ? `the binary panicked:\n${outcome.trace.split("\n").slice(0, 3).join("\n")}` + : outcome.kind === "unrecognized" + ? `unrecognized outcome (status ${String(outcome.status)}):\n` + + `stdout: ${outcome.stdout.slice(0, 400)}\nstderr: ${outcome.stderr.slice(0, 400)}` + : outcome.kind === "diagnostic" + ? `unexpected ${outcome.diagnostic.Code}: ${outcome.diagnostic.Text}` + : `unexpected clean run (${String(outcome.findings)} findings)`; + throw new Error(`${context}: ${detail}`); +} + +// --- Stage 1: the self-enumerating vocabularies ------------------------------ + +/** + * Ask the binary to name its own accepted set. + * + * Both `extends` and `level` reject an unknown value by enumerating the legal + * ones, so the probe supplies a sentinel and reads the list back out of the + * diagnostic's `Text`. Nothing here is proposed and then confirmed: the set + * arrives whole, which is why a Vale release that adds a check type is picked + * up rather than merely failing a test. + * + * The pattern is anchored on the whole line, and a miss throws. **A truncated + * enum is stricter than the binary**, so "the message changed shape" must never + * degrade into "the vocabulary got smaller". + */ +function enumerateFromBinary( + key: "extends" | "level", + rule: string, + { sorted }: { sorted: boolean } +): string[] { + const outcome = probe(rule, "bogus simply here.\n", "md"); + if (outcome.kind !== "diagnostic") { + fatal( + `enumerating '${key}': the binary did not reject the sentinel`, + outcome + ); + } + const { Code, Text } = outcome.diagnostic; + if (Code !== "E201") { + throw new Error( + `enumerating '${key}': expected diagnostic code E201, got ${Code} ` + + `("${Text}"). The generator refuses to guess at a vocabulary from an ` + + `error shape it does not know.` + ); + } + const match = new RegExp(String.raw`^'${key}'[^[\]]*\[([^\]]+)\]\.?$`).exec( + Text + ); + if (match === null) { + throw new Error( + `enumerating '${key}': Vale ${VALE_VERSION} answered "${Text}", which no ` + + `longer matches the enumeration shape this generator reads. Refusing ` + + `to emit a vocabulary: a short enum is STRICTER than the binary, so a ` + + `silent partial parse would start rejecting rules Vale accepts. Fix ` + + `the pattern against the new message.` + ); + } + const raw = must(match[1], `the '${key}' enumeration group`) + .trim() + .split(/\s+/); + if (raw.length === 0) { + throw new Error(`enumerating '${key}': the enumeration parsed as empty.`); + } + return sorted ? raw.toSorted() : raw; +} + +/** + * Check types are sorted; levels are left in the order the binary gave them. + * + * The difference is that one order carries meaning and the other does not. The + * check types come out of the binary unordered, so sorting makes the list + * `verify` prints to an author predictable. The levels come out in *severity* + * order — suggestion, warning, error — and alphabetising them would throw that + * away for nothing. + */ +const CHECK_TYPES = enumerateFromBinary( + "extends", + 'extends: __taskless_probe__\nmessage: "x"\nlevel: error\ntokens: [bogus]\n', + { sorted: true } +); +const LEVELS = enumerateFromBinary( + "level", + 'extends: existence\nmessage: "x"\nlevel: __taskless_probe__\ntokens: [bogus]\n', + { sorted: false } +); + +console.log( + `Vale ${VALE_VERSION}: ${String(CHECK_TYPES.length)} check types, ${String(LEVELS.length)} levels` +); + +// --- Stage 2: the per-check field tables ------------------------------------- + +/** + * A minimal rule per check type that the binary runs without complaint. + * + * **Hand-seeded, and it has to be.** A field probe adds one candidate key to a + * rule and asks whether the key was rejected — which is only meaningful if + * everything *else* in the rule was already fine. Several checks are not valid + * empty: a `sequence` with no `tokens` panics rather than reporting, and a + * `metric` with a `formula` and no `condition` does the same. So each base is a + * working rule of its check, and stage 2 asserts that before probing anything. + * + * Provenance: each is the minimal form of the corresponding row in + * `test/vale-corpus.ts`, which measured it accepted against this binary. + */ +const CHECK_BASES: Record> = { + existence: { tokens: ["bogus"] }, + substitution: { swap: { utilize: "use" } }, + capitalization: { match: "$title", style: "AP" }, + occurrence: { token: "very", max: 1 }, + repetition: { tokens: [String.raw`[^\s]+`] }, + conditional: { + first: String.raw`\b([A-Z]{3,5})\b`, + second: String.raw`(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)`, + }, + metric: { formula: "(characters / words)", condition: "> 1" }, + readability: { metrics: ["Gunning Fog"], grade: 1 }, + sequence: { tokens: [{ pattern: "a" }] }, + script: { script: "matches := []\n" }, + consistency: { either: { advisor: "adviser" } }, + spelling: {}, +}; + +/** + * The candidate universe, and why each name is in it. + * + * **This list is the generator's one irreducible act of authorship.** `E201` + * names the key you got wrong, never the ones you could have used, so a field + * table can only be verified, never discovered. A real field absent from here + * is absent from the artifact, and the schema then rejects a rule the binary + * runs — the "too strict" failure the design ranks as the worse one. + * + * Seeded generously and from four independent sources, so that a name missing + * from one is likely present in another: + * + * 1. **Vale's published key documentation** (`vale.sh/docs/keys`), including + * names the binary turns out to reject — `prefixes`, `suffixes` and + * `ignorecase` on `capitalization` are documented and measured rejected, and + * keeping them here is what makes that a recorded finding rather than an + * omission. + * 2. **The hand transcription this artifact replaces**, so the generator is + * strictly a superset of what a human already established. + * 3. **Vale's shared/common keys** — `link`, `limit`, `action`, `scope`, + * `description`, `name`, `comment`, `vocab`. + * 4. **Neighbouring checks' fields**, deliberately cross-probed: every field of + * every check is offered to every other check, which is what turns the + * per-check tables into a measured partition rather than twelve unrelated + * lists. + */ +const FIELD_CANDIDATES: readonly string[] = [ + // Header and shared keys (source 3). + "extends", + "message", + "level", + "scope", + "link", + "limit", + "action", + "description", + "name", + "comment", + "vocab", + // Documented per-check keys (sources 1, 2 and 4 — cross-probed). + "tokens", + "token", + "raw", + "ignorecase", + "nonword", + "exceptions", + "append", + "swap", + "capitalize", + "pos", + "match", + "style", + "threshold", + "indicators", + "prefix", + "prefixes", + "suffixes", + "max", + "min", + "alpha", + "first", + "second", + "formula", + "condition", + "metrics", + "grade", + "script", + "either", + "filter", + "filters", + "ignore", + "dicpath", + "dictionaries", + "custom", + "aff", + "dic", + "negate", + "ordered", + "chars", + "pattern", + "tag", +].toSorted(); + +/** + * A key no check has, used to ask whether a check validates its keys at all. + * + * Two of the twelve do not: `consistency` and `spelling` accept any key and + * ignore it. That is measured rather than assumed, because the schema's answer + * differs in the direction that matters — a strict object over a permissive + * check rejects rules the binary runs. + */ +const SENTINEL_FIELD = "taskless_generator_sentinel"; + +/** How the binary answered "is this a key of this check?". */ +type Membership = "member" | "not-a-member"; + +/** + * Parse the invalid-key list out of a diagnostic, structurally. + * + * `has invalid keys: 'a', 'b'` is the only membership oracle Vale offers, and + * it is read as a *list of names* rather than matched as a phrase. A diagnostic + * that names a key we did not probe means the base rule is contaminated, which + * is fatal — the alternative is recording a verdict about the wrong key. + */ +function invalidKeys(text: string): string[] | undefined { + const match = /^has invalid keys: (.+)$/.exec(text); + if (match === null) return undefined; + return [...must(match[1], "the invalid-key list").matchAll(/'([^']+)'/g)].map( + (found) => must(found[1], "an invalid key name").toLowerCase() + ); +} + +function toYaml(rule: Record): string { + return ( + Object.entries(rule) + .map(([key, value]) => `${key}: ${JSON.stringify(value)}`) + .join("\n") + "\n" + ); +} + +/** + * The keys Vale reads before it decodes a check at all. + * + * They are members of every check by construction — each base rule above + * carries all three and was verified running clean — so they are never probed. + * Probing them would also be actively unsound: the probe value is arbitrary, + * and `extends: true` reaches a bare `interface{}` to `string` conversion that + * panics the process rather than reporting anything. + */ +const HEADER_FIELDS = ["extends", "message", "level"]; + +/** + * Ask one question: does `check` accept `field` as a key? + * + * The verdict is a read of a closed outcome set, never a grep. The probe sets + * the field to an arbitrary value (`true`), so every outcome has to be + * interpreted as evidence about the *key* rather than about the value: + * + * - a clean run means the key was decoded, so it is a member; + * - `has invalid keys` naming this field means it is not — this is the only + * negative oracle Vale offers, and it is parsed as a list of names; + * - **any other diagnostic means member.** Vale recognized the key and objected + * to its value or type instead, which is membership evidence; + * - **a panic means member, for the same reason.** A key Vale does not know is + * collected and reported as unused; it is never dereferenced. Reaching a + * type conversion at all proves the key was routed somewhere. + * + * The last two are inferences rather than direct readings, so both are recorded + * and printed with the artifact. An inference nobody can audit is a grep with + * better manners. + * + * An invalid-key list naming a key we did not probe, or a diagnostic this + * script cannot parse, stays fatal: both mean the verdict would be about + * something other than the question asked. + */ +const typeObjections: string[] = []; +const panicObjections: string[] = []; + +function probeField( + check: string, + field: string, + { panicIsEvidence }: { panicIsEvidence: boolean } +): Membership { + const rule: Record = { + extends: check, + message: "x %s", + level: "error", + ...CHECK_BASES[check], + [field]: true, + }; + const outcome = probe(toYaml(rule), "bogus simply here.\n", "md"); + + if (outcome.kind === "clean") return "member"; + if (outcome.kind === "panic" && panicIsEvidence) { + panicObjections.push( + `${check}.${field}: ${(outcome.trace.split("\n")[0] ?? "").trim()}` + ); + return "member"; + } + if (outcome.kind !== "diagnostic") { + fatal(`probing '${field}' on the ${check} check`, outcome); + } + + const { Code, Text } = outcome.diagnostic; + if (Code !== "E201") { + throw new Error( + `probing '${field}' on the ${check} check: diagnostic code ${Code} is ` + + `not one this generator knows how to read ("${Text}").` + ); + } + + const named = invalidKeys(Text); + if (named === undefined) { + typeObjections.push(`${check}.${field}: ${Text}`); + return "member"; + } + if (named.includes(field.toLowerCase())) return "not-a-member"; + + throw new Error( + `probing '${field}' on the ${check} check: Vale rejected ${named + .map((key) => `'${key}'`) + .join(", ")}, which is not the key under test. The base rule for ` + + `${check} is contaminated, so no verdict here is trustworthy.` + ); +} + +/** Verify the base rule before trusting a single verdict taken against it. */ +for (const check of CHECK_TYPES) { + const base = CHECK_BASES[check]; + if (base === undefined) { + throw new Error( + `Vale ${VALE_VERSION} has a '${check}' check with no base rule in ` + + `CHECK_BASES. A field table cannot be probed without a working rule ` + + `of that check to probe against — add one and re-run.` + ); + } + const outcome = probe( + toYaml({ extends: check, message: "x %s", level: "error", ...base }), + "bogus simply here.\n", + "md" + ); + if (outcome.kind !== "clean") { + fatal(`the base rule for the ${check} check is not valid`, outcome); + } +} + +const permissiveChecks: string[] = []; +const acceptedFields: Record = {}; + +for (const check of CHECK_TYPES) { + // The sentinel is the one probe where a panic is *not* read as evidence. It + // decides whether the check validates keys at all, and a permissive verdict + // taken from a crash would make the schema loose for a check that is strict. + if ( + probeField(check, SENTINEL_FIELD, { panicIsEvidence: false }) === "member" + ) { + permissiveChecks.push(check); + continue; + } + const accepted = new Set([ + ...HEADER_FIELDS, + ...Object.keys(must(CHECK_BASES[check], `the base rule for ${check}`)), + ]); + for (const field of FIELD_CANDIDATES) { + if (accepted.has(field)) continue; + if (probeField(check, field, { panicIsEvidence: true }) === "member") { + accepted.add(field); + } + } + acceptedFields[check] = [...accepted].toSorted(); + console.log(` ${check}: ${String(accepted.size)} fields`); +} + +/** + * The fields every strict check takes, derived rather than declared. + * + * This is the intersection of the twelve measured tables, which is a stronger + * statement than a hand-written "common" list: `vocab` is *not* here, because + * five checks reject it, and nothing had to remember that. The permissive + * checks are excluded from the intersection — they accept everything, so they + * constrain nothing. + */ +const strictChecks = CHECK_TYPES.filter( + (check) => !permissiveChecks.includes(check) +); +const tableFor = (check: string): string[] => + must(acceptedFields[check], `the measured field table for ${check}`); + +const commonFields = FIELD_CANDIDATES.filter((field) => + strictChecks.every((check) => tableFor(check).includes(field)) +); + +const checkFields: Record = {}; +for (const check of strictChecks) { + checkFields[check] = tableFor(check).filter( + (field) => !commonFields.includes(field) + ); +} + +// --- Stage 3: the scope operands --------------------------------------------- + +/** + * A scope candidate, its fixture, and where the name came from. + * + * **`scope` has no oracle.** Vale does not reject an unknown scope: the rule + * loads, runs, and matches nothing, which from outside is indistinguishable + * from a valid scope whose construct is absent from the document. So the + * verdict is three-valued — + * + * | Verdict | What happened | + * | --------------- | --------------------------------------------------- | + * | `fires` | the rule flagged the fixture: the operand is real | + * | `silent` | the fixture was linted and the rule found nothing | + * | `unreachable` | the fixture was never linted at all | + * + * — and only a hand-written fixture separates the middle from the last. Every + * fixture contains the word `bogus`, and a reach probe (`scope: raw`) must fire + * on it; if it does not, the fixture is broken and the run is fatal, because + * "did not fire" would then say nothing about the operand. + * + * `documented` records whether Vale's own documentation names the operand. It + * is not evidence — the binary is — but the disagreements are the report this + * script emits, and a disagreement can only be stated if both sides are here. + * + * `ext` matters more than it looks: the file extension decides which parser + * Vale routes the document to, and `comment.*` only exists in a source tier. + */ +interface ScopeCandidate { + operand: string; + fixture: string; + ext: string; + documented: boolean; + /** For a family whose tail is author-supplied, the prefix to record. */ + prefix?: string; + /** Why this candidate is worth probing, when that is not obvious. */ + note?: string; +} + +const MARKDOWN_HEADING = (level: number): string => + `${"#".repeat(level)} Level ${String(level)} bogus heading\n`; + +const PROSE = "Just bogus do it.\n"; +const FRONTMATTER = "---\ntitle: bogus meta\n---\n\nBody text here.\n"; +const JS_COMMENTS = "// bogus line\n/*\n bogus block\n*/\nconst x = 1;\n"; +const TS_COMMENTS = + "// bogus line\n/*\n bogus block\n*/\nconst x: number = 1;\n"; +const TABLE = "| Head |\n| --- |\n| bogus |\n"; + +const SCOPE_CANDIDATES: readonly ScopeCandidate[] = [ + { operand: "text", fixture: PROSE, ext: "md", documented: true }, + { + operand: "code", + fixture: "Run `bogus now` here.\n", + ext: "md", + documented: true, + }, + { + operand: "raw", + fixture: "Prose bogus.\n\n```\nbogus fenced\n```\n", + ext: "md", + documented: true, + }, + { + operand: "heading", + fixture: "# Do bogus things\n", + ext: "md", + documented: true, + }, + { + operand: "heading.h1", + fixture: MARKDOWN_HEADING(1), + ext: "md", + documented: true, + }, + { + operand: "heading.h2", + fixture: MARKDOWN_HEADING(2), + ext: "md", + documented: true, + }, + { + operand: "heading.h3", + fixture: MARKDOWN_HEADING(3), + ext: "md", + documented: true, + }, + { + operand: "heading.h4", + fixture: MARKDOWN_HEADING(4), + ext: "md", + documented: true, + }, + { + operand: "heading.h5", + fixture: MARKDOWN_HEADING(5), + ext: "md", + documented: true, + }, + { + operand: "heading.h6", + fixture: MARKDOWN_HEADING(6), + ext: "md", + documented: true, + }, + { + operand: "heading.h7", + fixture: MARKDOWN_HEADING(1), + ext: "md", + documented: false, + note: "the negative control for the heading family: HTML stops at h6.", + }, + { + operand: "paragraph", + fixture: "A bogus paragraph here.\n", + ext: "md", + documented: true, + }, + { + operand: "sentence", + fixture: "A bogus sentence here.\n", + ext: "md", + documented: true, + }, + { + operand: "list", + fixture: "- bogus one\n- two\n", + ext: "md", + documented: true, + }, + { + operand: "blockquote", + fixture: "> A bogus quote.\n", + ext: "md", + documented: true, + }, + { + operand: "link", + fixture: "See [bogus link](https://example.com).\n", + ext: "md", + documented: true, + }, + { + operand: "alt", + fixture: "![bogus alt text](x.png)\n", + ext: "md", + documented: true, + }, + { + operand: "summary", + fixture: + "
bogus summary\n\nbody\n\n
\n", + ext: "md", + documented: true, + }, + { + operand: "strong", + fixture: "This is **bogus bold** text.\n", + ext: "md", + documented: true, + }, + { + operand: "emphasis", + fixture: "This is *bogus italic* text.\n", + ext: "md", + documented: true, + }, + { operand: "table", fixture: TABLE, ext: "md", documented: true }, + { + operand: "table.header", + fixture: "| bogus |\n| --- |\n| body |\n", + ext: "md", + documented: true, + }, + { operand: "table.cell", fixture: TABLE, ext: "md", documented: true }, + { + operand: "table.caption", + fixture: + "
bogus caption
body
\n", + ext: "html", + documented: true, + }, + { + operand: "table.row", + fixture: TABLE, + ext: "md", + documented: false, + note: "the negative control for the table family.", + }, + { + operand: "figure.caption", + fixture: "
bogus caption
\n", + ext: "html", + documented: true, + note: + "bare, NOT nested in
. Vale drops everything inside a
" + + "element, so the nested form is unreachable — see the divergence report.", + }, + { + operand: "meta", + fixture: FRONTMATTER, + ext: "md", + documented: true, + note: "documented by Vale; measured never firing. See the divergence report.", + }, + { + operand: "meta.class.title", + fixture: FRONTMATTER, + ext: "md", + documented: true, + prefix: "meta.class.", + note: "documented by Vale; measured never firing. See the divergence report.", + }, + { + operand: "frontmatter", + fixture: FRONTMATTER, + ext: "md", + documented: false, + note: "documented nowhere and measured working — the standing proof that this generator verifies rather than discovers.", + }, + { + operand: "frontmatter.title", + fixture: FRONTMATTER, + ext: "md", + documented: false, + prefix: "frontmatter.", + }, + { + operand: "text.class.foo", + fixture: '

bogus here

\n', + ext: "html", + documented: true, + prefix: "text.class.", + }, + { operand: "comment", fixture: JS_COMMENTS, ext: "js", documented: true }, + { + operand: "comment.line", + fixture: "// bogus line\nconst x = 1;\n", + ext: "js", + documented: true, + }, + { + operand: "comment.block", + fixture: "/*\n bogus block\n*/\nconst x = 1;\n", + ext: "js", + documented: true, + }, + { + operand: "comment.line", + fixture: TS_COMMENTS, + ext: "ts", + documented: true, + note: "the same operand in the TypeScript tier, which routes through a different parser.", + }, + { + operand: "comment.block", + fixture: TS_COMMENTS, + ext: "ts", + documented: true, + note: "the same operand in the TypeScript tier.", + }, + { + operand: "fenced", + fixture: "Prose bogus.\n\n```\nbogus fenced\n```\n", + ext: "md", + documented: false, + note: "the negative control an author is most likely to reach for.", + }, + { + operand: "banana", + fixture: PROSE, + ext: "md", + documented: false, + note: "the negative control that cannot possibly be a scope.", + }, +]; + +type ScopeVerdict = "fires" | "silent" | "unreachable"; + +/** The reach probe: proves a fixture was linted at all, independent of scope. */ +const REACH_RULE = + 'extends: existence\nmessage: "x %s"\nlevel: error\nscope: raw\nnonword: true\ntokens:\n - bogus\n'; + +interface ScopeMeasurement { + candidate: ScopeCandidate; + verdict: ScopeVerdict; +} + +const scopeMeasurements: ScopeMeasurement[] = []; + +for (const candidate of SCOPE_CANDIDATES) { + const rule = + 'extends: existence\nmessage: "x %s"\nlevel: error\n' + + `scope: ${JSON.stringify(candidate.operand)}\ntokens:\n - bogus\n`; + + const reach = probe(REACH_RULE, candidate.fixture, candidate.ext); + if (reach.kind !== "clean") { + fatal(`the reach probe on the ${candidate.operand} fixture`, reach); + } + if (reach.findings === 0) { + throw new Error( + `scope '${candidate.operand}': the reach probe did not fire on its ` + + `.${candidate.ext} fixture, so the fixture was never linted — wrong ` + + `extension, unparseable format, or an unmatched glob. Any verdict ` + + `taken here would say nothing about the operand.` + ); + } + + const outcome = probe(rule, candidate.fixture, candidate.ext); + if (outcome.kind !== "clean") { + fatal(`probing scope '${candidate.operand}'`, outcome); + } + const verdict: ScopeVerdict = outcome.findings > 0 ? "fires" : "silent"; + scopeMeasurements.push({ candidate, verdict }); + console.log(` scope ${candidate.operand} (.${candidate.ext}): ${verdict}`); +} + +/** + * The operands the schema will honor: the ones measured firing, nothing else. + * + * An operand that never fired for any fixture is not in the artifact, whatever + * the documentation says about it. That is the whole asymmetry `scope` is + * subject to — a scope Vale does not have costs an author a rule that is inert + * forever, with no error anywhere. + */ +const firing = scopeMeasurements.filter(({ verdict }) => verdict === "fires"); +const scopeOperands = [ + ...new Set( + firing + .filter(({ candidate }) => candidate.prefix === undefined) + .map(({ candidate }) => candidate.operand) + ), +].toSorted(); +const scopePrefixes = [ + ...new Set( + firing + .filter(({ candidate }) => candidate.prefix !== undefined) + .map(({ candidate }) => candidate.prefix as string) + ), +].toSorted(); + +// --- Stage 4: the divergence report ------------------------------------------ + +/** + * Where the binary and the documentation disagree. + * + * These are the findings a generator must never drop silently. A documented + * operand that never fires is a trap an author walks into with the docs open; + * an undocumented one that works is the standing proof that a candidate list + * verifies rather than discovers, and that something real could be missing from + * it. Both go into a checked-in report, not into a console line nobody reads. + */ +interface Divergence { + subject: string; + finding: string; + consequence: string; +} + +const divergences: Divergence[] = []; + +const byOperand = new Map(); +for (const measurement of scopeMeasurements) { + const existing = byOperand.get(measurement.candidate.operand) ?? []; + existing.push(measurement); + byOperand.set(measurement.candidate.operand, existing); +} + +for (const [operand, measurements] of byOperand) { + const documented = measurements.some(({ candidate }) => candidate.documented); + const fires = measurements.some(({ verdict }) => verdict === "fires"); + const negativeControl = measurements.every(({ candidate }) => + (candidate.note ?? "").includes("negative control") + ); + + if (documented && !fires) { + divergences.push({ + subject: `scope: ${operand}`, + finding: + `Vale ${VALE_VERSION} documents this operand and it never fired, on ` + + `any fixture probed (${measurements + .map(({ candidate }) => `.${candidate.ext}`) + .join(", ")}).`, + consequence: + "It is omitted from the vocabulary, so `verify` rejects it. A rule " + + "written from the documentation would otherwise load, run, and match " + + "nothing, with no error reported anywhere.", + }); + } + + if (!documented && fires && !negativeControl) { + divergences.push({ + subject: `scope: ${operand}`, + finding: `This operand fired and Vale ${VALE_VERSION} documents it nowhere.`, + consequence: + "It is included in the vocabulary. It is also the standing " + + "counterexample to trusting the candidate list: a real operand nobody " + + "proposes is simply absent, and the schema then rejects a rule the " + + "binary honors.", + }); + } + + // A partial firing — the same operand alive in one tier and dead in another — + // is a divergence in its own right, and the one most likely to be read as a + // broken fixture rather than as a property of the binary. + const alive = measurements.filter(({ verdict }) => verdict === "fires"); + const dead = measurements.filter(({ verdict }) => verdict !== "fires"); + if (alive.length > 0 && dead.length > 0) { + divergences.push({ + subject: `scope: ${operand}`, + finding: + `Fires in ${alive.map(({ candidate }) => `.${candidate.ext}`).join(", ")} ` + + `and is silent in ${dead.map(({ candidate }) => `.${candidate.ext}`).join(", ")}, ` + + `on fixtures the reach probe confirmed were linted in both cases.`, + consequence: + "The operand stays in the vocabulary — it is real — but it is not " + + "portable across formats, and `verify` cannot tell an author which " + + "tier they are in.", + }); + } +} + +for (const check of permissiveChecks) { + divergences.push({ + subject: `extends: ${check}`, + finding: + `This check accepted '${SENTINEL_FIELD}', a key no check has. It does ` + + "not validate its keys at all.", + consequence: + "The schema is permissive here. Being strict would reject rules the " + + "binary runs, to catch a typo that costs nothing but a field quietly " + + "ignored — the too-strict direction, which is the worse failure.", + }); +} + +if (typeObjections.length > 0) { + divergences.push({ + subject: "field probes: membership inferred from a type complaint", + finding: + `${String(typeObjections.length)} probes drew an E201 that was not an ` + + `invalid-key list: ${typeObjections.join("; ")}.`, + consequence: + "Each is recorded as a member: Vale recognized the key and objected to " + + "the probe's arbitrary value instead, which is membership evidence. " + + "They are listed so the inference is auditable rather than assumed.", + }); +} + +if (panicObjections.length > 0) { + divergences.push({ + subject: "field probes: membership inferred from a crash", + finding: + `${String(panicObjections.length)} probes panicked the binary: ` + + `${panicObjections.join("; ")}.`, + consequence: + "Each is recorded as a member. A key Vale does not know is collected " + + "and reported as unused, never dereferenced — so reaching a type " + + "conversion at all proves the key was routed somewhere. This is the " + + "weakest inference the generator makes, which is why it is named here.", + }); +} + +// --- Emit -------------------------------------------------------------------- + +function literalList(values: readonly string[]): string { + return values.map((value) => ` ${JSON.stringify(value)},`).join("\n"); +} + +const artifact = `/** + * The Vale rule vocabulary, derived from the vendored binary. + * + * GENERATED FILE — DO NOT EDIT. Run \`pnpm generate:vale-schema\` in + * \`packages/cli\` to reproduce it. The generator is + * \`scripts/generate-vale-schema.ts\`, and its header explains what each value + * below was measured with and what it is worth. + * + * Derived against Vale ${VALE_VERSION}. Every value here is the recorded answer + * of that binary to a rule the generator wrote and ran; nothing is transcribed + * from documentation. Where the binary and the documentation disagree, the + * disagreement is in \`vale-vocabulary-report.md\` rather than dropped. + * + * \`src/schemas/vale-rule.ts\` turns these into the zod schema \`verify\` runs. + */ + +/** The binary this vocabulary was derived from. */ +export const VALE_VOCABULARY_VERSION = ${JSON.stringify(VALE_VERSION)}; + +/** + * Vale's check types, self-enumerated: an unknown \`extends\` makes the binary + * name the whole set, so this is discovered rather than proposed. + */ +export const VALE_CHECK_TYPES = [ +${literalList(CHECK_TYPES)} +] as const; + +/** + * Vale's severities, self-enumerated the same way, and left in the binary's + * own order because that order is severity. The value is case-sensitive. + */ +export const VALE_LEVELS = [ +${literalList(LEVELS)} +] as const; + +/** + * The checks that accept any key at all and ignore what they do not know. + * + * Measured by offering each check a sentinel key no check has. + */ +export const VALE_PERMISSIVE_CHECKS = [ +${literalList(permissiveChecks.toSorted())} +] as const; + +/** + * The fields every strict check accepts, as the intersection of their measured + * tables rather than as a declared list. + * + * Note what the intersection excludes: \`vocab\` is per-check, because several + * checks reject it. + */ +export const VALE_COMMON_FIELDS = [ +${literalList(commonFields)} +] as const; + +/** + * Each strict check's own fields — its measured table minus the common ones. + * + * Membership only: \`E201\` names the key you got wrong and never the ones you + * could have used, so every name here was proposed by the generator's candidate + * list and confirmed. A real field nobody proposed is absent. + */ +export const VALE_CHECK_FIELDS = { +${strictChecks + .toSorted() + .map( + (check) => + ` ${check}: [\n${must( + checkFields[check], + `the emitted table for ${check}` + ) + .map((field) => ` ${JSON.stringify(field)},`) + .join("\n")}\n ],` + ) + .join("\n")} +} as const; + +/** + * The \`scope\` operands measured firing on a fixture carrying their construct. + * + * \`scope\` has no oracle — an unknown scope is silent, and so is a valid scope + * with no construct to match — so an operand is here only if a rule using it + * flagged a fixture that a reach probe independently confirmed was linted. + */ +export const VALE_SCOPE_OPERANDS = [ +${literalList(scopeOperands)} +] as const; + +/** + * Scope families whose tail is author-supplied and cannot be enumerated. + * + * \`frontmatter.title\` names a key in the document's own front matter; + * \`text.class.callout\` names an HTML class. Rejecting an unfamiliar tail would + * be the too-strict failure against a value the binary honors. + */ +export const VALE_SCOPE_PREFIXES = [ +${literalList(scopePrefixes)} +] as const; + +/** + * Where Vale ${VALE_VERSION} and its documentation disagree. + * + * Carried in the artifact rather than only in the report, so that a consumer + * can render them and a reviewer cannot miss them in a diff. + */ +export const VALE_DIVERGENCES = [ +${divergences + .map( + (divergence) => + ` {\n subject: ${JSON.stringify(divergence.subject)},\n finding:\n ${JSON.stringify(divergence.finding)},\n consequence:\n ${JSON.stringify(divergence.consequence)},\n },` + ) + .join("\n")} +] as const; +`; + +writeFileSync(OUTPUT_PATH, artifact, "utf8"); + +const report = `# Vale ${VALE_VERSION} vocabulary: divergence report + +GENERATED FILE — DO NOT EDIT. Produced by \`pnpm generate:vale-schema\` +alongside \`vale-vocabulary.ts\`. + +Every value in the vocabulary is the recorded answer of the vendored Vale +${VALE_VERSION} binary. This file is what the binary said that its own +documentation does not, in both directions. A generator that dropped these +would be quietly deciding which of the two to believe. + +## What was derived, and what was seeded + +| Vocabulary | Oracle | Discovers? | +| --- | --- | --- | +| Check types (${String(CHECK_TYPES.length)}) | \`'extends' key must be one of [...]\` | yes | +| Levels (${String(LEVELS.length)}) | \`'level' must be one of [...]\` | yes | +| Per-check fields | \`has invalid keys: ''\` | **no — membership only** | +| Scope operands | none; an unknown scope is silent | **no — fixture probe** | + +The bottom two rows are verified, not discovered. \`E201\` names the key you got +wrong and never the ones you could have used, and an unknown scope produces no +error at all. A real field or operand that the generator's candidate list does +not propose is simply absent from the vocabulary, and the schema then rejects a +rule the binary accepts — the too-strict direction, which the design ranks as +the worse failure. + +## Divergences + +${ + divergences.length === 0 + ? "None. The binary and the documentation agreed everywhere probed." + : divergences + .map( + (divergence) => + `### \`${divergence.subject}\`\n\n${divergence.finding}\n\n**Consequence.** ${divergence.consequence}` + ) + .join("\n\n") +} + +## Every scope probed + +| Operand | Fixture | Documented | Verdict | +| --- | --- | --- | --- | +${scopeMeasurements + .map( + ({ candidate, verdict }) => + `| \`${candidate.operand}\` | \`.${candidate.ext}\` | ${candidate.documented ? "yes" : "no"} | ${verdict} |` + ) + .join("\n")} +`; + +writeFileSync(REPORT_PATH, report, "utf8"); + +// Formatting is delegated to the repository's own prettier rather than hand +// matched here, so the artifact survives `lint-staged` untouched and a +// re-generation is a no-op diff. +const formatted = spawnSync( + process.execPath, + [ + resolve( + PACKAGE_ROOT, + "..", + "..", + "node_modules", + "prettier", + "bin", + "prettier.cjs" + ), + "--write", + OUTPUT_PATH, + REPORT_PATH, + ], + { encoding: "utf8" } +); +if (formatted.status !== 0) { + throw new Error( + `prettier failed on the generated artifact: ${formatted.stderr}` + ); +} + +console.log(`\nWritten: ${OUTPUT_PATH}`); +console.log(`Written: ${REPORT_PATH}`); +console.log(`Divergences recorded: ${String(divergences.length)}`); +for (const divergence of divergences) { + console.log(` - ${divergence.subject}: ${divergence.finding}`); +} diff --git a/packages/cli/src/generated/vale-vocabulary-report.md b/packages/cli/src/generated/vale-vocabulary-report.md new file mode 100644 index 00000000..b42675ce --- /dev/null +++ b/packages/cli/src/generated/vale-vocabulary-report.md @@ -0,0 +1,112 @@ +# Vale 3.18.0 vocabulary: divergence report + +GENERATED FILE — DO NOT EDIT. Produced by `pnpm generate:vale-schema` +alongside `vale-vocabulary.ts`. + +Every value in the vocabulary is the recorded answer of the vendored Vale +3.18.0 binary. This file is what the binary said that its own +documentation does not, in both directions. A generator that dropped these +would be quietly deciding which of the two to believe. + +## What was derived, and what was seeded + +| Vocabulary | Oracle | Discovers? | +| ---------------- | ------------------------------------ | ------------------------ | +| Check types (12) | `'extends' key must be one of [...]` | yes | +| Levels (3) | `'level' must be one of [...]` | yes | +| Per-check fields | `has invalid keys: ''` | **no — membership only** | +| Scope operands | none; an unknown scope is silent | **no — fixture probe** | + +The bottom two rows are verified, not discovered. `E201` names the key you got +wrong and never the ones you could have used, and an unknown scope produces no +error at all. A real field or operand that the generator's candidate list does +not propose is simply absent from the vocabulary, and the schema then rejects a +rule the binary accepts — the too-strict direction, which the design ranks as +the worse failure. + +## Divergences + +### `scope: meta` + +Vale 3.18.0 documents this operand and it never fired, on any fixture probed (.md). + +**Consequence.** It is omitted from the vocabulary, so `verify` rejects it. A rule written from the documentation would otherwise load, run, and match nothing, with no error reported anywhere. + +### `scope: meta.class.title` + +Vale 3.18.0 documents this operand and it never fired, on any fixture probed (.md). + +**Consequence.** It is omitted from the vocabulary, so `verify` rejects it. A rule written from the documentation would otherwise load, run, and match nothing, with no error reported anywhere. + +### `scope: frontmatter` + +This operand fired and Vale 3.18.0 documents it nowhere. + +**Consequence.** It is included in the vocabulary. It is also the standing counterexample to trusting the candidate list: a real operand nobody proposes is simply absent, and the schema then rejects a rule the binary honors. + +### `scope: frontmatter.title` + +This operand fired and Vale 3.18.0 documents it nowhere. + +**Consequence.** It is included in the vocabulary. It is also the standing counterexample to trusting the candidate list: a real operand nobody proposes is simply absent, and the schema then rejects a rule the binary honors. + +### `extends: consistency` + +This check accepted 'taskless_generator_sentinel', a key no check has. It does not validate its keys at all. + +**Consequence.** The schema is permissive here. Being strict would reject rules the binary runs, to catch a typo that costs nothing but a field quietly ignored — the too-strict direction, which is the worse failure. + +### `extends: spelling` + +This check accepted 'taskless_generator_sentinel', a key no check has. It does not validate its keys at all. + +**Consequence.** The schema is permissive here. Being strict would reject rules the binary runs, to catch a typo that costs nothing but a field quietly ignored — the too-strict direction, which is the worse failure. + +### `field probes: membership inferred from a type complaint` + +10 probes drew an E201 that was not an invalid-key list: capitalization.action: expected a map, got 'bool'; conditional.action: expected a map, got 'bool'; existence.action: expected a map, got 'bool'; metric.action: expected a map, got 'bool'; occurrence.action: expected a map, got 'bool'; readability.action: expected a map, got 'bool'; repetition.action: expected a map, got 'bool'; script.action: expected a map, got 'bool'; sequence.action: expected a map, got 'bool'; substitution.action: expected a map, got 'bool'. + +**Consequence.** Each is recorded as a member: Vale recognized the key and objected to the probe's arbitrary value instead, which is membership evidence. They are listed so the inference is auditable rather than assumed. + +## Every scope probed + +| Operand | Fixture | Documented | Verdict | +| ------------------- | ------- | ---------- | ------- | +| `text` | `.md` | yes | fires | +| `code` | `.md` | yes | fires | +| `raw` | `.md` | yes | fires | +| `heading` | `.md` | yes | fires | +| `heading.h1` | `.md` | yes | fires | +| `heading.h2` | `.md` | yes | fires | +| `heading.h3` | `.md` | yes | fires | +| `heading.h4` | `.md` | yes | fires | +| `heading.h5` | `.md` | yes | fires | +| `heading.h6` | `.md` | yes | fires | +| `heading.h7` | `.md` | no | silent | +| `paragraph` | `.md` | yes | fires | +| `sentence` | `.md` | yes | fires | +| `list` | `.md` | yes | fires | +| `blockquote` | `.md` | yes | fires | +| `link` | `.md` | yes | fires | +| `alt` | `.md` | yes | fires | +| `summary` | `.md` | yes | fires | +| `strong` | `.md` | yes | fires | +| `emphasis` | `.md` | yes | fires | +| `table` | `.md` | yes | fires | +| `table.header` | `.md` | yes | fires | +| `table.cell` | `.md` | yes | fires | +| `table.caption` | `.html` | yes | fires | +| `table.row` | `.md` | no | silent | +| `figure.caption` | `.html` | yes | fires | +| `meta` | `.md` | yes | silent | +| `meta.class.title` | `.md` | yes | silent | +| `frontmatter` | `.md` | no | fires | +| `frontmatter.title` | `.md` | no | fires | +| `text.class.foo` | `.html` | yes | fires | +| `comment` | `.js` | yes | fires | +| `comment.line` | `.js` | yes | fires | +| `comment.block` | `.js` | yes | fires | +| `comment.line` | `.ts` | yes | fires | +| `comment.block` | `.ts` | yes | fires | +| `fenced` | `.md` | no | silent | +| `banana` | `.md` | no | silent | diff --git a/packages/cli/src/generated/vale-vocabulary.ts b/packages/cli/src/generated/vale-vocabulary.ts new file mode 100644 index 00000000..6454912a --- /dev/null +++ b/packages/cli/src/generated/vale-vocabulary.ts @@ -0,0 +1,216 @@ +/** + * The Vale rule vocabulary, derived from the vendored binary. + * + * GENERATED FILE — DO NOT EDIT. Run `pnpm generate:vale-schema` in + * `packages/cli` to reproduce it. The generator is + * `scripts/generate-vale-schema.ts`, and its header explains what each value + * below was measured with and what it is worth. + * + * Derived against Vale 3.18.0. Every value here is the recorded answer + * of that binary to a rule the generator wrote and ran; nothing is transcribed + * from documentation. Where the binary and the documentation disagree, the + * disagreement is in `vale-vocabulary-report.md` rather than dropped. + * + * `src/schemas/vale-rule.ts` turns these into the zod schema `verify` runs. + */ + +/** The binary this vocabulary was derived from. */ +export const VALE_VOCABULARY_VERSION = "3.18.0"; + +/** + * Vale's check types, self-enumerated: an unknown `extends` makes the binary + * name the whole set, so this is discovered rather than proposed. + */ +export const VALE_CHECK_TYPES = [ + "capitalization", + "conditional", + "consistency", + "existence", + "metric", + "occurrence", + "readability", + "repetition", + "script", + "sequence", + "spelling", + "substitution", +] as const; + +/** + * Vale's severities, self-enumerated the same way, and left in the binary's + * own order because that order is severity. The value is case-sensitive. + */ +export const VALE_LEVELS = ["suggestion", "warning", "error"] as const; + +/** + * The checks that accept any key at all and ignore what they do not know. + * + * Measured by offering each check a sentinel key no check has. + */ +export const VALE_PERMISSIVE_CHECKS = ["consistency", "spelling"] as const; + +/** + * The fields every strict check accepts, as the intersection of their measured + * tables rather than as a declared list. + * + * Note what the intersection excludes: `vocab` is per-check, because several + * checks reject it. + */ +export const VALE_COMMON_FIELDS = [ + "action", + "description", + "extends", + "level", + "limit", + "link", + "message", + "name", + "scope", +] as const; + +/** + * Each strict check's own fields — its measured table minus the common ones. + * + * Membership only: `E201` names the key you got wrong and never the ones you + * could have used, so every name here was proposed by the generator's candidate + * list and confirmed. A real field nobody proposed is absent. + */ +export const VALE_CHECK_FIELDS = { + capitalization: [ + "exceptions", + "indicators", + "match", + "prefix", + "style", + "threshold", + "vocab", + ], + conditional: ["exceptions", "first", "ignorecase", "second", "vocab"], + existence: [ + "append", + "exceptions", + "ignorecase", + "nonword", + "raw", + "tokens", + "vocab", + ], + metric: ["condition", "formula"], + occurrence: ["ignorecase", "max", "min", "token"], + readability: ["grade", "metrics"], + repetition: ["alpha", "exceptions", "ignorecase", "max", "tokens", "vocab"], + script: ["script"], + sequence: ["ignorecase", "tokens"], + substitution: [ + "capitalize", + "exceptions", + "ignorecase", + "nonword", + "pos", + "swap", + "vocab", + ], +} as const; + +/** + * The `scope` operands measured firing on a fixture carrying their construct. + * + * `scope` has no oracle — an unknown scope is silent, and so is a valid scope + * with no construct to match — so an operand is here only if a rule using it + * flagged a fixture that a reach probe independently confirmed was linted. + */ +export const VALE_SCOPE_OPERANDS = [ + "alt", + "blockquote", + "code", + "comment", + "comment.block", + "comment.line", + "emphasis", + "figure.caption", + "frontmatter", + "heading", + "heading.h1", + "heading.h2", + "heading.h3", + "heading.h4", + "heading.h5", + "heading.h6", + "link", + "list", + "paragraph", + "raw", + "sentence", + "strong", + "summary", + "table", + "table.caption", + "table.cell", + "table.header", + "text", +] as const; + +/** + * Scope families whose tail is author-supplied and cannot be enumerated. + * + * `frontmatter.title` names a key in the document's own front matter; + * `text.class.callout` names an HTML class. Rejecting an unfamiliar tail would + * be the too-strict failure against a value the binary honors. + */ +export const VALE_SCOPE_PREFIXES = ["frontmatter.", "text.class."] as const; + +/** + * Where Vale 3.18.0 and its documentation disagree. + * + * Carried in the artifact rather than only in the report, so that a consumer + * can render them and a reviewer cannot miss them in a diff. + */ +export const VALE_DIVERGENCES = [ + { + subject: "scope: meta", + finding: + "Vale 3.18.0 documents this operand and it never fired, on any fixture probed (.md).", + consequence: + "It is omitted from the vocabulary, so `verify` rejects it. A rule written from the documentation would otherwise load, run, and match nothing, with no error reported anywhere.", + }, + { + subject: "scope: meta.class.title", + finding: + "Vale 3.18.0 documents this operand and it never fired, on any fixture probed (.md).", + consequence: + "It is omitted from the vocabulary, so `verify` rejects it. A rule written from the documentation would otherwise load, run, and match nothing, with no error reported anywhere.", + }, + { + subject: "scope: frontmatter", + finding: "This operand fired and Vale 3.18.0 documents it nowhere.", + consequence: + "It is included in the vocabulary. It is also the standing counterexample to trusting the candidate list: a real operand nobody proposes is simply absent, and the schema then rejects a rule the binary honors.", + }, + { + subject: "scope: frontmatter.title", + finding: "This operand fired and Vale 3.18.0 documents it nowhere.", + consequence: + "It is included in the vocabulary. It is also the standing counterexample to trusting the candidate list: a real operand nobody proposes is simply absent, and the schema then rejects a rule the binary honors.", + }, + { + subject: "extends: consistency", + finding: + "This check accepted 'taskless_generator_sentinel', a key no check has. It does not validate its keys at all.", + consequence: + "The schema is permissive here. Being strict would reject rules the binary runs, to catch a typo that costs nothing but a field quietly ignored — the too-strict direction, which is the worse failure.", + }, + { + subject: "extends: spelling", + finding: + "This check accepted 'taskless_generator_sentinel', a key no check has. It does not validate its keys at all.", + consequence: + "The schema is permissive here. Being strict would reject rules the binary runs, to catch a typo that costs nothing but a field quietly ignored — the too-strict direction, which is the worse failure.", + }, + { + subject: "field probes: membership inferred from a type complaint", + finding: + "10 probes drew an E201 that was not an invalid-key list: capitalization.action: expected a map, got 'bool'; conditional.action: expected a map, got 'bool'; existence.action: expected a map, got 'bool'; metric.action: expected a map, got 'bool'; occurrence.action: expected a map, got 'bool'; readability.action: expected a map, got 'bool'; repetition.action: expected a map, got 'bool'; script.action: expected a map, got 'bool'; sequence.action: expected a map, got 'bool'; substitution.action: expected a map, got 'bool'.", + consequence: + "Each is recorded as a member: Vale recognized the key and objected to the probe's arbitrary value instead, which is membership evidence. They are listed so the inference is auditable rather than assumed.", + }, +] as const; diff --git a/packages/cli/src/rules/vale/verify.ts b/packages/cli/src/rules/vale/verify.ts index d3c1197c..b3d03500 100644 --- a/packages/cli/src/rules/vale/verify.ts +++ b/packages/cli/src/rules/vale/verify.ts @@ -36,6 +36,11 @@ function stylesPath(cwd: string): string { * positional (a later matcher wins; a repeat inside one matcher is * discarded), so a config that assigned it twice would be relying on the * rule that bit the scoping spec. + * + * `scripts/generate-vale-schema.ts` builds a second isolating config for its + * probes. It cannot call this one, because it owns its whole temp directory + * and so has no absolute StylesPath to hand over, but it shares the other two + * details above. A new requirement found here belongs there too. */ export function buildIsolatingConfig(cwd: string, ruleId: string): string { return [ diff --git a/packages/cli/src/schemas/vale-rule.ts b/packages/cli/src/schemas/vale-rule.ts index 281fb040..684794fe 100644 --- a/packages/cli/src/schemas/vale-rule.ts +++ b/packages/cli/src/schemas/vale-rule.ts @@ -5,15 +5,21 @@ * and the machine-readable field knowledge exists only behind the hosted MCP * server at `api.vale.sh/mcp`, which is a paid product and unavailable to * `verify`. So unlike `ast-grep-rule.ts` — which runs `z.fromJSONSchema()` over - * an upstream artifact fetched per tag — everything here is a - * **transcription**, and a transcription drifts. - * - * What makes it trustworthy is not care in the writing. It is - * `test/vale-corpus.ts`: a table of minimal rules, each with a document it must - * flag, run through the vendored binary and through this module, asserting the - * two agree. Every value enumerated below was measured accepted by the binary - * that way. A Vale bump that changes the vocabulary fails that test naming the - * field, rather than leaving this file quietly wrong. + * an upstream artifact fetched per tag — there is nothing here to fetch. + * + * There is, however, a binary that answers questions. Every vocabulary this + * module enumerates is **derived** from it by `scripts/generate-vale-schema.ts` + * and lives in `../generated/vale-vocabulary.ts`: check types and levels read + * out of the error Vale raises when it enumerates its own accepted set, field + * tables probed key by key, scope operands measured firing on a fixture. This + * file is the validation layer over that vocabulary, not a second copy of it. + * + * The distinction matters because it changes what a Vale bump costs. A + * transcription drifts silently and is re-authored by hand; a derivation is + * re-run, and where the answer moved, the artifact's diff says so. What holds + * both to the binary is unchanged: `test/vale-corpus.ts` is a table of minimal + * rules, each with a document it must flag, run through the vendored binary and + * through this module, asserting the two agree. * * Two directions of error, and they are not symmetric: * @@ -23,6 +29,14 @@ * blocks work that would have functioned. Where a measurement is ambiguous, * accept. * + * That asymmetry is also the honest limit of the generator. Field tables and + * scope operands are established by *membership*: Vale's `E201` names the key + * you got wrong and never the ones you could have used, and an unknown scope + * raises nothing at all. So both are verified rather than discovered, and a + * real field nobody proposed is absent — the too-strict direction. The + * generator's candidate list is seeded accordingly, and its provenance is + * documented there. + * * ## Shape * * Two stages, in the order the binary itself works: @@ -43,48 +57,60 @@ import { z } from "zod"; +import { + VALE_CHECK_FIELDS, + VALE_CHECK_TYPES, + VALE_COMMON_FIELDS, + VALE_LEVELS, + VALE_PERMISSIVE_CHECKS, + VALE_SCOPE_OPERANDS as DERIVED_SCOPE_OPERANDS, + VALE_SCOPE_PREFIXES, + VALE_VOCABULARY_VERSION, +} from "../generated/vale-vocabulary"; import { VALE_VERSION } from "../rules/capabilities"; import { schemaLayer, type SchemaLayerResult } from "./layer"; +export { + VALE_CHECK_TYPES, + VALE_LEVELS, + VALE_PERMISSIVE_CHECKS, +} from "../generated/vale-vocabulary"; + /** - * Vale's check types, quoted from the binary rather than the documentation. + * The Vale release both the vocabulary and the vendored binary refer to. * - * Obtained by giving Vale an `extends` it does not know, which is not silently - * ignored — it fails the file, exits 2, and enumerates the set: + * Regenerating the artifact and bumping {@link VALE_VERSION} are two separate + * edits and nothing sequences them. If they ever disagree, every enum in this + * module is a measurement of some *other* Vale — which is precisely the drift + * deriving the vocabulary was meant to end. * - * ``` - * 'extends' key must be one of [capitalization conditional consistency - * existence occurrence repetition substitution readability spelling sequence - * metric script]. - * ``` + * The conditional type is what settles it. Both sides are string literal types, + * so the compiler can decide the question: on a mismatch the alias resolves to + * `never`, the assignment below is rejected, and the build fails naming this + * line. A runtime `if` would have thrown instead — later, on whichever command + * happened to load the schema first, in front of a user rather than an author. * - * That settles the eleven-versus-twelve question the docs and Vale's own MCP - * guide disagree on: **twelve**. The docs' eleven folds `readability` into - * `metric`; the binary treats them as separate checks whose fields are - * disjoint, and each rejects the other's. - * - * Sorted here rather than left in the binary's order, so the value `verify` - * prints to an author reads predictably. + * Every message in this module interpolates this rather than + * {@link VALE_VERSION}, so the assertion cannot be quietly orphaned. */ -export const VALE_CHECK_TYPES = [ - "capitalization", - "conditional", - "consistency", - "existence", - "metric", - "occurrence", - "readability", - "repetition", - "script", - "sequence", - "spelling", - "substitution", -] as const; +type PinnedValeVersion = + typeof VALE_VOCABULARY_VERSION extends typeof VALE_VERSION + ? typeof VALE_VERSION + : never; -export type ValeCheckType = (typeof VALE_CHECK_TYPES)[number]; +const PINNED_VALE_VERSION: PinnedValeVersion = VALE_VERSION; -/** Vale's three severities. The value is case-sensitive: `WARNING` is rejected. */ -export const VALE_LEVELS = ["suggestion", "warning", "error"] as const; +/** + * The name of one of Vale's twelve check types. + * + * The set itself is derived: an unknown `extends` is not silently ignored, so + * the binary fails the file and enumerates its own accepted values, and the + * generator reads them back out. That is what settles the eleven-versus-twelve + * question the docs and Vale's own MCP guide disagree on — the docs' eleven + * folds `readability` into `metric`, and the binary treats them as separate + * checks whose fields are disjoint. + */ +export type ValeCheckType = (typeof VALE_CHECK_TYPES)[number]; /** * The keys Vale reads literally, before the case-insensitive field decode. @@ -105,49 +131,22 @@ const HEADER_KEYS = new Set(["extends", "message", "level"]); /** * The `scope` operands the binary was measured honoring. * - * Each was established by authoring a rule with that scope over a document the - * rule had to flag, and confirming the finding appeared. A scope that never - * fired for any control is not here. + * Each was derived by authoring a rule with that scope over a fixture carrying + * its construct and confirming the finding appeared, with an independent reach + * probe proving the fixture was linted at all. An operand that never fired is + * not here, whatever the documentation says about it. * * This is the highest-value list in the module, because `scope` is the one * field nothing downstream ever validates. An unrecognized `extends` fails the * run loudly; an unrecognized field raises `E201`. An unrecognized **scope** - * loads, runs, and matches nothing — no error, anywhere, ever. - * - * Note what is *not* here: `meta` and `meta.class.`. The v3.18.0 addition - * is `frontmatter` / `frontmatter.`, which is the vocabulary the binary - * carries and the one that fires. + * loads, runs, and matches nothing — no error, anywhere, ever. It is also the + * list with no oracle: `extends` and `level` enumerate themselves, and this + * one has to be proposed and then measured, which is why the generator emits + * `vale-vocabulary-report.md` alongside it. Two entries there are worth + * knowing: `meta` and `meta.class.` are documented and never fire, and + * `frontmatter` / `frontmatter.` fire while being documented nowhere. */ -const SCOPE_OPERANDS = new Set([ - "text", - "code", - "raw", - "heading", - "heading.h1", - "heading.h2", - "heading.h3", - "heading.h4", - "heading.h5", - "heading.h6", - "paragraph", - "sentence", - "list", - "blockquote", - "link", - "alt", - "summary", - "strong", - "emphasis", - "table", - "table.header", - "table.cell", - "table.caption", - "figure.caption", - "frontmatter", - "comment", - "comment.line", - "comment.block", -]); +const SCOPE_OPERANDS = new Set(DERIVED_SCOPE_OPERANDS); /** * Scope families whose tail is author-supplied and cannot be enumerated. @@ -157,13 +156,18 @@ const SCOPE_OPERANDS = new Set([ * neither has a closed set — rejecting an unfamiliar tail would be the * "too strict" failure against a value the binary honors. */ -const SCOPE_PREFIXES = ["frontmatter.", "text.class."]; +const SCOPE_PREFIXES: readonly string[] = VALE_SCOPE_PREFIXES; -/** Operands, for the message `verify` shows an author. */ +/** + * Operands, for the message `verify` shows an author. + * + * The open families are spelled with a placeholder tail rather than omitted: + * an author told only that `frontmatter.title` is wrong, with no mention of + * `frontmatter.` in the accepted set, learns the wrong lesson. + */ export const VALE_SCOPE_OPERANDS: readonly string[] = [ ...SCOPE_OPERANDS, - "frontmatter.", - "text.class.", + ...SCOPE_PREFIXES.map((prefix) => `${prefix}`), ].toSorted(); function isScopeOperand(operand: string): boolean { @@ -207,11 +211,11 @@ function scopeMessages(scope: string): string[] { if (isScopeOperand(operand)) continue; messages.push( negated - ? `scope: "~${operand}" negates a scope Vale ${VALE_VERSION} does ` + + ? `scope: "~${operand}" negates a scope Vale ${PINNED_VALE_VERSION} does ` + `not have, so it subtracts nothing and the rule fires everywhere — ` + `the exclusion you wrote it for is silently gone. Vale does not ` + `report this, so verify does. ${scopeVocabulary}` - : `scope: "${operand}" is not a Vale ${VALE_VERSION} scope. Vale ` + + : `scope: "${operand}" is not a Vale ${PINNED_VALE_VERSION} scope. Vale ` + `does not reject an unknown scope — the rule loads, runs, and ` + `matches nothing. ${scopeVocabulary}` ); @@ -246,7 +250,7 @@ const valeHeaderSchema = z ) { fail( ["extends"], - `extends "${extendsValue}" is not a Vale ${VALE_VERSION} check type. ` + + `extends "${extendsValue}" is not a Vale ${PINNED_VALE_VERSION} check type. ` + `Vale fails the whole run over this, taking every other Vale rule's ` + `findings with it. Accepted: ${VALE_CHECK_TYPES.join(", ")}.` ); @@ -314,23 +318,24 @@ function canonicalKeys(rule: Record): Record { // --- Stage 2: the per-check field tables ------------------------------------- /** - * Fields every check accepts, whatever it extends. + * Fields every strict check accepts, whatever it extends. + * + * Derived as the *intersection* of the twelve measured field tables rather + * than declared, which is a stronger statement than a hand-written list: no + * one had to remember that `vocab` does not belong here because five checks + * reject it — the intersection simply does not contain it. * - * `vocab` is deliberately absent: five of the twelve reject it, so it is listed - * per check instead. Values are `unknown` throughout — the binary decides what - * a field's type means, and guessing here would be the "too strict" failure - * against types nothing measured. + * Values are `unknown` throughout. The binary decides what a field's type + * means, and guessing here would be the "too strict" failure against types + * nothing measured. */ -const commonFields = { +const commonFields: Record = { + ...Object.fromEntries( + VALE_COMMON_FIELDS.map((field) => [field, z.unknown()]) + ), extends: z.string(), message: z.string(), level: z.string().optional(), - scope: z.unknown(), - link: z.unknown(), - limit: z.unknown(), - action: z.unknown(), - description: z.unknown(), - name: z.unknown(), }; /** The field names above, for the message a rejected field gets. */ @@ -368,9 +373,10 @@ function check(name: ValeCheckType, fields: readonly string[]) { /** * The two checks that validate nothing. * - * Measured: `bananafield: true` on a `consistency` or a `spelling` rule loads - * without complaint and is ignored. They do not use the strict decode the other - * ten do, so the binary raises no `E201` for a foreign field there. + * Derived by offering every check a sentinel key no check has: `consistency` + * and `spelling` load without complaint and ignore it. They do not use the + * strict decode the other ten do, so the binary raises no `E201` for a foreign + * field there. * * `z.looseObject` follows the binary rather than the docs. Being strict here * would reject rules that work — the "too strict" direction — to catch a typo @@ -381,60 +387,21 @@ function permissiveCheck(name: ValeCheckType) { return z.looseObject({ ...commonFields, extends: z.literal(name) }); } -export const VALE_PERMISSIVE_CHECKS: readonly ValeCheckType[] = [ - "consistency", - "spelling", -]; - /** - * The per-check field tables, measured by adding each candidate to a minimal - * rule of that check and watching for `E201: has invalid keys`. + * The per-check field tables, derived rather than transcribed. * - * Kept as a table rather than inlined into the union below, because this *is* - * the measurement — the union is only how it gets enforced. Three entries - * contradict Vale's published docs, and the binary wins: `capitalization` takes - * `prefix` **singular** and rejects `prefixes` and `suffixes`; `capitalization` - * rejects `ignorecase`; `occurrence` rejects `exceptions` and `vocab`. + * Each name here was proposed by the generator's candidate list, added to a + * minimal working rule of that check, and confirmed by the binary. Three + * entries contradict Vale's published docs and the binary wins: + * `capitalization` takes `prefix` **singular** and rejects `prefixes` and + * `suffixes`; `capitalization` rejects `ignorecase`; `occurrence` rejects + * `exceptions` and `vocab`. All three names stay in the generator's candidate + * list precisely so that those stay recorded findings rather than omissions. * * `consistency` and `spelling` are absent on purpose — see * {@link permissiveCheck}. */ -const CHECK_FIELDS = { - existence: [ - "tokens", - "raw", - "ignorecase", - "nonword", - "exceptions", - "append", - "vocab", - ], - substitution: [ - "swap", - "ignorecase", - "nonword", - "exceptions", - "capitalize", - "pos", - "vocab", - ], - capitalization: [ - "match", - "style", - "exceptions", - "threshold", - "indicators", - "prefix", - "vocab", - ], - occurrence: ["token", "max", "min", "ignorecase"], - repetition: ["tokens", "alpha", "ignorecase", "exceptions", "max", "vocab"], - conditional: ["first", "second", "exceptions", "ignorecase", "vocab"], - metric: ["formula", "condition"], - readability: ["metrics", "grade"], - sequence: ["tokens", "ignorecase"], - script: ["script"], -} as const; +const CHECK_FIELDS = VALE_CHECK_FIELDS; /** * The `E201` class, as schema shape. @@ -529,23 +496,96 @@ function fatalShapeMessages( return fatal; } +/** + * The union's members, spelled out, and the guard that keeps them honest. + * + * They are spelled out rather than mapped over {@link CHECK_FIELDS} because + * that is what carries each `z.literal` into {@link ValeRule}; a mapped union + * infers `extends: string` and the type stops saying anything. The cost is + * that this list is hand-maintained while the vocabulary it draws from is + * derived, so a Vale release that adds or renames a check type would leave a + * member missing here, and the schema would reject a rule the binary runs. + * + * This block closes that gap at import, and it reads the discriminants off + * *these members* rather than off a second list of names. A name list would + * only prove that the list agrees with the vocabulary: a maintainer could + * satisfy the guard by adding the name and never adding the `check(...)` call, + * leaving the union a member short while the error that prompted the edit went + * away. Asking the members themselves means the only way to satisfy the guard + * is to build the member. Every derived check type must appear here exactly + * once, and must be classified as either strict or permissive. The failure is + * then a sentence naming the check, rather than a rule that mysteriously stops + * verifying six months later. + */ +const UNION_MEMBERS = [ + check("existence", CHECK_FIELDS.existence), + check("substitution", CHECK_FIELDS.substitution), + check("capitalization", CHECK_FIELDS.capitalization), + check("occurrence", CHECK_FIELDS.occurrence), + check("repetition", CHECK_FIELDS.repetition), + check("conditional", CHECK_FIELDS.conditional), + check("metric", CHECK_FIELDS.metric), + check("readability", CHECK_FIELDS.readability), + check("sequence", CHECK_FIELDS.sequence), + check("script", CHECK_FIELDS.script), + permissiveCheck("consistency"), + permissiveCheck("spelling"), +] as const; + +{ + const discriminants = UNION_MEMBERS.map( + (member) => member.shape.extends.value + ); + const spelled = new Set(discriminants); + const derived = new Set(VALE_CHECK_TYPES); + const missing = [...derived].filter((name) => !spelled.has(name)); + const extra = [...spelled].filter((name) => !derived.has(name)); + const permissive = new Set(VALE_PERMISSIVE_CHECKS); + const unclassified = [...derived].filter( + (name) => !permissive.has(name) && !(name in CHECK_FIELDS) + ); + + if (missing.length > 0) { + throw new Error( + `Vale ${PINNED_VALE_VERSION} has a ${missing.join(", ")} check and the schema's ` + + `union does not, so every rule extending it would be rejected. Add ` + + `the member in src/schemas/vale-rule.ts.` + ); + } + if (extra.length > 0) { + throw new Error( + `the schema's union has a ${extra.join(", ")} member and Vale ` + + `${PINNED_VALE_VERSION} has no such check, so those rules would verify clean ` + + `and then fail the whole run. Remove the member in ` + + `src/schemas/vale-rule.ts.` + ); + } + if (unclassified.length > 0) { + throw new Error( + `${unclassified.join(", ")} is in neither the derived field tables nor ` + + `the permissive set, so there is nothing to build a union member from. ` + + `Re-run the generator: pnpm generate:vale-schema.` + ); + } + if (spelled.size !== discriminants.length) { + // "Exactly once" is a real requirement rather than tidiness: zod matches a + // discriminated union by discriminant, so a second member for a check that + // already has one is dead code that no rule ever reaches. + const repeated = [...spelled].filter( + (name) => discriminants.filter((other) => other === name).length > 1 + ); + throw new Error( + `the schema's union has more than one ${repeated.join(", ")} ` + + `member, and only the first is reachable. Remove the duplicate in ` + + `src/schemas/vale-rule.ts.` + ); + } +} + const valeBodySchema = z .discriminatedUnion( "extends", - [ - check("existence", CHECK_FIELDS.existence), - check("substitution", CHECK_FIELDS.substitution), - check("capitalization", CHECK_FIELDS.capitalization), - check("occurrence", CHECK_FIELDS.occurrence), - check("repetition", CHECK_FIELDS.repetition), - check("conditional", CHECK_FIELDS.conditional), - check("metric", CHECK_FIELDS.metric), - check("readability", CHECK_FIELDS.readability), - check("sequence", CHECK_FIELDS.sequence), - check("script", CHECK_FIELDS.script), - permissiveCheck("consistency"), - permissiveCheck("spelling"), - ], + UNION_MEMBERS, { // Unreachable in practice — the header rejects an unknown `extends` first, // with a message naming all twelve. Supplied so that if the two ever drift diff --git a/packages/cli/test/vale-corpus.ts b/packages/cli/test/vale-corpus.ts index 5167e811..ba89fda7 100644 --- a/packages/cli/test/vale-corpus.ts +++ b/packages/cli/test/vale-corpus.ts @@ -1,12 +1,21 @@ /** * The corpus that makes `src/schemas/vale-rule.ts` true. * - * The schema is a hand transcription — Vale publishes no JSON Schema, and the - * machine-readable field knowledge sits behind its paid hosted MCP. What makes - * a transcription trustworthy is not care in the writing; it is this table. - * Every entry is a minimal rule plus a document the rule must flag, run through - * **both** the vendored binary and the schema in - * `vale-schema-contract.test.ts`, asserting the two agree. + * The schema's vocabulary is derived from the vendored binary by + * `scripts/generate-vale-schema.ts` — Vale publishes no JSON Schema, and the + * machine-readable field knowledge sits behind its paid hosted MCP, so the only + * thing left to ask is the binary. Every entry below is a minimal rule plus a + * document the rule must flag, run through **both** the vendored binary and the + * schema in `vale-schema-contract.test.ts`, asserting the two agree. + * + * A derivation needs this table for a different reason than a transcription + * did, and needs it just as much. A transcription could be wrong because + * someone mistyped; a derivation can be wrong because the generator asked the + * wrong question — a membership probe that misreads a crash as an acceptance, + * a scope fixture that never carried the construct it claimed to test. The + * generator cannot catch either, because both are errors in its own method. + * This table can, because it asks the binary a different question: not "is this + * a legal key" but "does this whole rule do what a rule is for". * * ## Why a verdict is not an exit code * @@ -323,6 +332,23 @@ const SCOPES: ValeCorpusEntry[] = [ control: "/*\n simply block\n*/\nconst x = 1;\n", ext: "js", }, + // The same two operands one tier over. `comment.*` is the only family whose + // membership depends on which parser the extension routes to, so a row that + // only ever asks JavaScript is asserting less than it looks like it is — + // "fires in a source file" and "fires in *this* source file" are different + // claims, and the schema makes the wider one. + { + name: "scope/comment.line-ts", + scope: "comment.line", + control: "// simply line\nconst x: number = 1;\n", + ext: "ts", + }, + { + name: "scope/comment.block-ts", + scope: "comment.block", + control: "/*\n simply block\n*/\nconst x: number = 1;\n", + ext: "ts", + }, // The operators. `scope` is a grammar, not an enum: a flat enum would reject // every one of these, which is worse than the gap the schema closes. { name: "scope/negation", scope: "~code", control: MIXED },