feat: schema-check a Vale rule before Vale ever sees it

[thecodedrift]

Stack (root → tip):

• fix(vale): consistent rule generation and verification #174
  • ➡️ feat: schema-check a Vale rule before Vale ever sees it #175 (you are here)
    • feat: derive the Vale rule schema from the vendored binary #177

What this is

Unit 2 of the vale-authoring-polish stack, on top of #174. It adds the Vale
rule schema, wires it into verify, and brings the corpus that holds the schema
to the vendored binary. It also archives the change, since this is the tip.

Review #174 first — it carries the measurements this unit encodes.

The gap

verify checked level and the presence of the rule's .vale.ini. Measured
against the pinned Vale 3.18.0, three defects got through it:

Rule defect	Before	Blast radius
extends: nonsense	ok: true	Vale exits 2, every rule silent
tokens on an occurrence check	ok: true	E201, every rule silent
scope: fenced	ok: true	rule inert forever, no error at all

The first two are not local failures. Vale reads one assembled config per run,
so either one reaching the binary takes down every other Vale rule's findings.
The third is the quiet one: nothing anywhere reports it, ever.

What verify now checks

packages/cli/src/schemas/vale-rule.ts, a zod schema reported through the same
LayerResult-shaped error list the sg path already uses:

• extends against the twelve check types, naming the accepted set.
• scope as a grammar, not an enum. The binary accepts a bare operand, a
  list, ~ negation, and & chaining; a flat enum would reject all but the
  first, which is worse than the gap it closes. The enum applies to the
  operands, the schema accepts the operators around them.
• Per-check field tables, so a foreign field is caught before E201.
• Field names compared case-insensitively, because Vale decodes them that
  way (Tokens: works), while extends/message/level are read literally
  and their values are case-sensitive — matching the split the binary has.

It runs before Vale is invoked, and test still runs verify first and stops,
so a rule that would take the engine down cannot reach it. There is a test
asserting exactly that, via ran: false.

How it is built

Two stages, in the order the binary itself works.

Stage 1 — the header, a z.record(...).check(...). Vale reads extends,
message and level literally and gives up if they are wrong, so these are
checked against the raw keys. scope rides along because it is common to all
twelve checks.

Stage 2 — the check's own fields, a z.discriminatedUnion("extends", …):
ten z.strictObjects and two z.looseObjects. This is the real win — Vale's
E201 class becomes a property of the schema's shape, so a field belonging to
another check type is rejected by the union rather than by a hand-written key
walk. The two loose members are consistency and spelling, which the binary
was measured accepting any key at all on; making them strict would be the
"too strict" failure.

Stage 3 — the shapes that panic the binary, a .check() after the union.
See below; these are not field-table facts, so they do not belong in it.

.pipe() sequences the two, which reproduces the binary: zod runs stage 2 only
if stage 1 produced no issues, so an extends Vale does not recognize is
reported on its own — with no check type there is no field table to check
against.

Between the stages, a transform lowercases every field name Vale would lowercase
and no others. Vale decodes a check's fields case-insensitively (Tokens: and
ignoreCase: work) but reads the three header keys literally, and LEVEL: warning reaches the field decode as level — which the header reader already
removed — so it comes back as E201 has invalid keys: 'level'. The transform
reproduces that split.

Two things zod cannot express on its own, and both stay explicit rather than
being contorted into schema shape:

• scope is a grammar, not an enum. A .check() with a small parser walks
  the ~ and & operators and applies the operand enum inside them.
• The ~unknown rejection is a business rule, not a transcription, so it
  carries its own message saying so rather than borrowing the generic one.

schemas/layer.ts (18 lines of code) turns a zod parse result into
{ valid, errors }, and the ast-grep path now goes through it too. The two
engines differ only in how one issue becomes one line, which is a parameter:
ast-grep's messages come from an upstream JSON Schema and are generic
("expected string"), so the path prefix is what makes them actionable; every
Vale message is already a full sentence naming its own field.

export type ValeRule = z.infer<typeof valeBodySchema> — the union members are
spelled out rather than mapped over the field table so each z.literal survives
into that type.

Strict versus permissive, measured for all twelve

Re-probed with a key no check has (bananafield) rather than a real field
borrowed from another check, and read off the exit status rather than by
grepping for has invalid keys:

	checks
Strict (10)	capitalization, conditional, existence, metric, occurrence, readability, repetition, script, sequence, substitution
Permissive (2)	consistency, spelling

sequence is strict, and it is the one that misleads. Probe it as
extends: sequence + message + an unknown key and Vale reports no invalid
keys — so a probe grepping for that phrase scores it permissive. It is not: give
the rule its tokens and it rejects the unknown key like every other strict
check. What happens without tokens is worse than an E201:

panic: interface conversion: interface {} is nil, not []interface {}

The process dies, and a Go stack trace contains no has invalid keys string, so
the grep sees a clean run. A measurement taken by grepping for an error string
cannot tell "no error" from "no output." Every corpus verdict is read from the
exit status instead.

Three shapes that end the process

That probe turned up three rules where every key is legal and the shape is
fatal:

Rule	Binary
sequence with no tokens	panic
sequence whose tokens is not a list	panic
metric with a formula and no condition	panic

A panic is a wider blast radius than E201: an E201 names a file and a line,
a panic names no rule at all, produces no findings for anything in the project,
and gives an author nothing to act on. verify is the last place they can be
caught, so it rejects all three — in a pass that runs after the field tables,
since a shape can only be fatal if every key in it was legal to begin with.

$ taskless verify .taskless/rules/vale/hedge-seq
✗ vale/hedge-seq
    hedge-seq.yml: a sequence check needs 'tokens'. Without it Vale does not report an error — it panics, ending the run with a Go stack trace and no findings for any rule in the project.

Cost

vale-rule.ts went 358 → 504 lines (201 → 261 non-comment), plus 49 lines
of layer.ts and 8 fewer in verify.ts. So this is not a line saving: zod's
issue construction is more verbose than pushing a string, and that is where the
~60 lines went. What was bought is that the field tables are enforced by shape
rather than by a walk, the two engines share one adapter, and the rule has an
inferred type.

The measured field tables stayed a plain CHECK_FIELDS table beside a thin
union, deliberately — that table is the measurement, and the union is only how
it gets enforced.

The one deliberate divergence

scope: ~fenced fires on everything, because there is no fenced to
subtract. The binary "accepts" it in the only sense an exit code can express,
and what the author gets is a rule whose exclusion was silently deleted — which
is precisely the class of failure this change exists to close. verify rejects
it.

That is the schema being stricter than the binary, which is normally the worse
direction, so it is not a skip in the differential: it is a divergence row
that asserts the disagreement and carries its reason. Exceptions are
countable, not hidden.

The corpus

packages/cli/test/vale-corpus.ts — 82 rows, each a minimal rule plus a
document it must flag, run through both the binary and the schema by
vale-schema-contract.test.ts. Shape:

• 14 check-type rows — all twelve accepted, plus extends: nonsense and
  extends: Existence rejected.
• 32 scope operand rows, including all six heading levels, the table.* and
  comment.* families, frontmatter/frontmatter.<key>, text.class.<name>,
  figure.caption, and the ~/&/list operators.
• 6 invalid-scope rows, including meta and meta.class.title — the operands
  the design's first draft named, which do not exist.
• 23 field rows: a foreign field per strict check, the valid fields an earlier
  draft would have wrongly rejected, and the unknown-key row for each of the
  three permissiveness candidates — sequence (rejects), consistency and
  spelling (accept).
• 3 shape rows for the panics above.
• 5 header rows and the 1 divergence.

A verdict is not an exit code

Vale does not say "that is not a scope." An unrecognized scope parses clean and
matches nothing, which from outside is identical to a valid rule whose pattern
did not fire. So rows record three outcomes — accepted (fired), ignored
(exit 0, nothing), rejected (the run failed) — and two guards keep a row from
passing while asserting nothing:

1. Reach. A scope: raw probe must fire on every control, so every control
   contains the same token. It proves Vale linted the fixture at all.
2. Attribution. Every ignored row carries a proof: the same rule with
   the construct replaced by a known-valid one, which must fire on the same
   control. Only then is "did not fire" attributable to the construct.

Guard 1 is not theoretical. figure.caption first measured ignored — and
so did a scope: text control over the same document, because Vale drops
everything inside a <figure> element. Read off the first result alone, the
operand would have been dropped from the schema, and every rule using it would
then have failed verify: the "too strict" failure, arrived at through a
careless fixture.

Proving the suite is not vacuous

Seven mutations, each failing exactly the expected row and nothing else:

Mutation	Failure
drop figure.caption from the enum	too strict, naming scope/figure.caption
drop readability (the docs' eleven)	too strict, naming extends/readability
add meta to the enum	too lax, naming scope/meta
allow tokens on occurrence	too lax, naming field/occurrence+tokens
blank the token in one control doc	both vacuity guards, naming the affected rows
make sequence a looseObject	too lax, naming both field/sequence+* rows
drop the fatal-shape pass	too lax, naming all three shape/* rows

Version-bump tripwires

Two, per the spec's requirement that a Vale upgrade fails loudly naming the
field:

• The corpus differential names any construct whose verdict changed.
• vale-vendor-contract.test.ts now asks the binary to enumerate its own
  check types — Vale prints the whole set when rejecting an unknown extends
  — and compares that to VALE_CHECK_TYPES.

Verification

pnpm typecheck, pnpm lint, pnpm test pass — 851 tests across 54 files.

test/vale-corpus.ts was byte-identical across the zod rewrite and all 82
rows passed; it has since gained four rows (and only rows) for the sequence
finding above, and all 86 pass. That is the property that made the refactor safe rather than
hopeful: the corpus asserts schemaAccepts === binaryAccepts against the
vendored binary, so a zod schema that came out stricter or laxer than the
hand-rolled walker would have named the row instead of passing quietly.
pnpm openspec validate --strict --all passes, 24 items.

pnpm build, then the real CLI against a deliberately-broken rule:

$ taskless verify .taskless/rules/vale/no-emdash
✗ vale/no-emdash
    no-emdash.yml: extends "existance" is not a Vale 3.18.0 check type. Vale fails the whole run over this, taking every other Vale rule's findings with it. Accepted: capitalization, conditional, consistency, existence, metric, occurrence, readability, repetition, script, sequence, spelling, substitution.
    no-emdash.yml has an invalid level; it must be suggestion, warning, error.
    no-emdash.yml: scope: "fenced" is not a Vale 3.18.0 scope. Vale does not reject an unknown scope — the rule loads, runs, and matches nothing. Accepted: alt, blockquote, code, comment, comment.block, comment.line, emphasis, figure.caption, frontmatter, frontmatter.<key>, heading, heading.h1, …, text, text.class.<name> (each optionally prefixed with "~", chained with "&", or given as a list).

1 of 1 rule(s) failed.        # exit 1

And the foreign-field case, which is the one with the widest blast radius:

$ taskless verify .taskless/rules/vale/too-many-bangs
✗ vale/too-many-bangs
    too-many-bangs.yml: 'tokens' is not a field of the occurrence check. Vale reports this as E201 and reads one config for the whole run, so it suppresses every other Vale rule's findings. occurrence accepts: action, description, extends, ignorecase, level, limit, link, max, message, min, name, scope, token.

1 of 1 rule(s) failed.        # exit 1

Fixed as the recipe says (extends: existence, level: warning,
nonword: true, scope: raw), the same rule verifies clean and then reports a
real finding on a real file:

$ taskless verify .taskless/rules/vale/no-emdash
✓ vale/no-emdash

$ taskless check README.md
  README.md:1:12
  warning[no-emdash] Avoid the em dash: —

On the size of this diff

~1700 lines, against the ~300-line guideline, and it did not want to be split
further. The breakdown: ~550 lines of schema
and adapter, 33 of wiring, and 1300 of corpus and tests. The corpus is not test scaffolding around the schema — per the
design, it is the artifact the schema is derived from, and the schema without
it is an unverified transcription of a vendored binary. Splitting them would put
a slice on main whose central claim ("these are the values Vale accepts")
nothing checks.

The reviewable core is src/schemas/vale-rule.ts, src/schemas/layer.ts, and
the 20-line change to src/rules/inspect.ts. test/vale-corpus.ts is a table; read a few rows and
the header comment rather than all 86.

Decisions a reviewer might want to push back on

• consistency and spelling are exempt from field checking. Measured,
  they accept any key at all — bananafield: true loads and is ignored. Being
  strict there would reject rules the binary runs, to catch a typo Vale itself
  does not catch. The recipe says so in prose, which is the only place it can be
  said.
• The schema lives at src/schemas/vale-rule.ts, not src/generated/. That
  directory is for artifacts a script fetches; this is hand-authored. It sits
  beside schemas/ast-grep-rule.ts.
• The header stage blocks the field stage. A rule missing message is
  reported without its field errors. That mirrors Vale, which abandons the file
  at the header read — measured: Message: (wrong case) reports only
  "Missing the required 'message' key" — but it does mean two iterations for a
  rule with one problem in each stage.
• An unknown extends suppresses the per-check field report. With no known
  check type there is no field table, so the function reports that one error
  rather than inventing complaints from a table it does not have. Visible in the
  first output above.
• Task 6.4 was adapted. It asks that every rule under .taskless/rules/vale/
  still verifies; this repository carries none, so the test would be vacuous. It
  verifies the nine worked rules from create-vale-rule instead — the
  population the requirement is actually about.

Fixes #171
Refs #170

Contains #177

What this is

Unit 3 of the vale-authoring-polish stack, on top of #175. It replaces the
hand-written Vale vocabulary with one derived from the vendored binary by a
checked-in generator, keeps zod as the validation layer, and ships the
divergences the derivation found as a report rather than dropping them.

Review #175 first — this rewrites the schema it introduced.

Why

#175 called its own schema what it was: a transcription. Twelve check types,
three levels, ten field tables and twenty-eight scope operands, read off a
binary by hand because Vale publishes no JSON Schema and its machine-readable
field knowledge sits behind a paid hosted MCP.

Every one of those values was measured. The measuring was done by a script
that was then thrown away. What survives is the answer, not the question —
the next person to raise VALE_VERSION gets a test failure naming a field and
no way to reproduce the measurement it is failing against.

The measuring is also where the errors live, and one is already on the record:

sequence, bare rule, probe grepping for "has invalid keys"  ->  read as "accepts any key"
sequence, bare rule, actual behavior                        ->  panic: interface conversion

A Go panic contains no has invalid keys string, so a phrase-grep scores a
crash as a clean run. A method that can make that mistake is worth writing
down once, correctly.

The method rule

Every verdict comes from the process exit status and the structured JSON
output. Never from grepping stdout for an error phrase. A run's outcome is a
closed set with no escape hatch:

Outcome	How it is recognized
clean	status 0, stdout parses as a findings map
diagnostic	status ≠ 0, stderr parses as Vale's JSON, keyed on Code
panic	stderr carries a Go panic header
unrecognized	everything else

unrecognized is fatal at every call site. There is deliberately no branch
that folds an unfamiliar shape into "fine".

What is derived, and what is honestly not

Vocabulary	Oracle	Discovers?
Check types (12)	'extends' key must be one of [...]	yes
Levels (3)	'level' must be one of [...]	yes
Per-check fields	has invalid keys: '<name>' — names the bad key	no
Scope operands	none; an unknown scope is silent	no

The top two self-enumerate: hand the binary a sentinel and it names its own
accepted set. If either line stops matching, the generator errors rather than
emitting a short enum — a truncated enum is stricter than the binary, which
is the direction that blocks rules that would have worked.

The bottom two are verified, not discovered, and the artifact says so. The
candidate list is seeded from four sources, including names the binary rejects
(prefixes, suffixes, ignorecase on capitalization) so that their
rejection stays a recorded finding rather than an omission, and every check's
fields are offered to every other check so the tables are a measured partition.

Membership is read as evidence about the key, since the probe value is
arbitrary. Two of the positive verdicts are inferences rather than readings —
"Vale objected to the value instead" and "Vale panicked, so it dereferenced
something" — and both are named in the report, because an inference nobody
can audit is a grep with better manners.

scope has no oracle at all, so its verdict is three-valued and a scope: raw
reach probe must fire on every fixture or the run is fatal. That guard is
not theoretical: figure.caption measures silent when its fixture nests the
caption inside a <figure>, and so does scope: text over the same document,
because Vale drops everything in that element. Without the guard the operand
would be dropped as one Vale ignores, and every rule using it would then fail
verify.

The divergence report

src/generated/vale-vocabulary-report.md, emitted by the run:

Subject	Finding	What the schema did
meta, meta.class.<kind>	documented, never fire	omitted — verify rejects them
frontmatter, frontmatter.<key>	fire, documented nowhere	included
consistency, spelling	accept a sentinel key no check has	stay permissive
action on all ten strict checks	expected a map, got 'bool'	member, inference named

frontmatter is the standing proof of the limit above: nobody proposing from
the documentation alone would have found it.

Three claims carried in from earlier measurement did not reproduce, and the
report is where that surfaced:

• figure.caption does fire, on a bare <figcaption>. What never fires is
  the nested-in-<figure> form; the earlier note conflated the two.
• comment.block fires, in .js and .ts alike.
• comment.* does not need a .ts fixture specifically — .js reaches the
  same tier. Rows for both now exist, because "fires in a source file" and
  "fires in this source file" are different claims.

What stays hand-written

• The three fatal shapes. Every key in a tokenless sequence is a legal
  field of its check; it is the shape that panics, which a field table cannot
  express.
• The scope grammar. ~ and & are operators around the operands, and
  rejecting ~fenced is a business rule rather than a transcription.
• The union members, spelled out so each z.literal survives into
  ValeRule. An import-time guard requires every derived check type to appear
  exactly once and to be classified strict or permissive.
• The version pin, asserted as a conditional type rather than an if.
  Both sides are string literal types, so a mismatch is a build failure naming
  the line instead of a throw in front of a user.

How it is checked

The corpus is the acceptance test. All 86 existing rows pass unmodified
against the generated schema — that is what proves generation reproduced the
transcription. No row was edited to fit the output. Two rows were added for
ground the generation newly measured (comment.line / comment.block in the
TypeScript tier), for 88.

The generated tables matched the hand-written ones exactly: the same twelve
check types, the same three levels, the same nine common fields, all ten
per-check tables identical, the same twenty-eight scope operands and two
prefixes. Nothing in the vocabulary changed; only how it is produced.

Deleting both artifacts and re-running produces byte-identical files.

What it looks like

$ taskless verify
✗ vale/no-simply
    no-simply.yml: scope: "fenced" is not a Vale 3.18.0 scope. Vale does not
    reject an unknown scope — the rule loads, runs, and matches nothing.
    Accepted: alt, blockquote, code, comment, comment.block, comment.line,
    emphasis, figure.caption, frontmatter, frontmatter.<name>, heading,
    heading.h1, … text.class.<name> (each optionally prefixed with "~",
    chained with "&", or given as a list).

$ taskless verify   # with the scope fixed, the foreign field surfaces
✗ vale/no-simply
    no-simply.yml: 'tokens' is not a field of the occurrence check. Vale
    reports this as E201 and reads one config for the whole run, so it
    suppresses every other Vale rule's findings. occurrence accepts: action,
    description, extends, ignorecase, level, limit, link, max, message, min,
    name, scope, token.

The second only appears once the first is fixed, which is the .pipe()
reproducing the binary's own order: with no valid header there is no field
table to check anything against.

Refs #171

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[thecodedrift]

@claude /review

[claude]

Claude finished @thecodedrift's task in 7m 32s —— View job

---

Review mode: incremental — read 0 prior review thread(s) before reviewing.

.prior-review.json contains no review threads, review summaries, or top-level comments beyond the @claude /review trigger itself — this is the first review pass on this PR.

• Read .prior-review.json
• Understand the request (code review, no implementation)
• Review src/schemas/vale-rule.ts and src/schemas/layer.ts
• Review wiring (src/rules/verify.ts, src/rules/inspect.ts)
• Review corpus and contract tests
• Post findings

Summary

This is a well-built unit — the corpus/contract-test discipline (86 rows, measured against the real vendored binary rather than the docs) is exactly the right way to hold a hand-authored transcription honest, and the .pipe()-sequenced two-stage schema faithfully mirrors the binary's own header-then-fields evaluation order. I read vale-rule.ts, layer.ts, the verify.ts/inspect.ts wiring, the corpus, and both contract test files. I found one real gap in the newest addition (the fatal-shape guard from the third commit) and a couple of minor cleanup items.

[New] Correctness: metric panic guard misses a condition: key present with a null value

packages/cli/src/schemas/vale-rule.ts:496-508

if (
  rule.extends === "metric" &&
  rule.formula !== undefined &&
  rule.condition === undefined
) {
  fatal.push({ path: ["condition"], message: "...needs a 'condition'..." });
}

This only checks for absence of condition (=== undefined). YAML parses a key written with no value (condition: on its own line, or condition: followed by nothing) to null, not undefined — so a rule with

extends: metric
formula: |
  (characters / words)
condition:

has rule.condition === null, which is !== undefined, so the guard does not fire and the rule verifies clean. If Vale's panic is triggered by the field being unusable (nil/empty) rather than strictly absent from the map — which is plausible given the documented panic is interface conversion: interface {} is nil, not []interface {} — this reopens exactly the "too lax" gap this pass exists to close, for a typo an author is quite likely to make (forgetting the value after the colon).

This is inconsistent with how the neighboring sequence guard handles the same class of problem a few lines up (vale-rule.ts:475-493): it treats "not an array" (which catches null, since Array.isArray(null) is false) as fatal, not just "undefined". The metric/condition branch doesn't have an equivalent structural check — it only asks whether the key was supplied at all, not whether it's a usable value. Since this can't be checked against the vendored binary from here, I'd flag this as worth a quick binary probe (condition: with a blank/null value) rather than asserting it's necessarily wrong — the corpus has no row for it either way.

[New] Cleanup: VALE_PERMISSIVE_CHECKS is exported but never used

packages/cli/src/schemas/vale-rule.ts:384-387

export const VALE_PERMISSIVE_CHECKS: readonly ValeCheckType[] = [
  "consistency",
  "spelling",
];

Grepped the whole package — nothing imports this outside its own file (not even the corpus or contract tests, which independently hardcode consistency/spelling in prose). It's redundant with permissiveCheck()'s two call sites a few lines below. Either wire it in somewhere (e.g., a test asserting CHECK_FIELDS has no entries for these two, which would make the "exempt on purpose" invariant machine-checked instead of just documented in a comment) or drop the export.

Notes, not findings

• packages/cli/test/vale-schema-contract.test.ts:198-231 (is not too lax / is not too strict) calls validateValeRule twice per failing row (once in .filter(), once in .map()) — harmless since it's O(failures), not O(corpus).
• runOne() in the same file spawns the vendored Vale binary synchronously (spawnSync) up to 3× per corpus row (rule, reach probe, optional proof) across 86 rows inside a single beforeAll — not parallelized, but well within the 120s timeout and test-only, so not flagging it as a real problem.
• The layer.ts / verify.ts / inspect.ts changes (moving the ast-grep path onto the shared schemaLayer/pathPrefixed, and folding the old ad-hoc extends/message/level checks in inspect.ts into validateValeRule) are behavior-preserving — I checked for tests asserting the old literal error wording and found none, so no regression there.
• Everything measured against the binary (check types, scope operands/grammar, per-check field tables, the three panic shapes, the ~fenced divergence) is corroborated by a corpus row and I didn't find a mismatch between the schema code and its corresponding corpus entries.

[thecodedrift]

Re: @claude[bot] — "Claude finished @thecodedrift's task in 7m 32s —— [View job]…"
#175 (comment)

metric/condition guard. Probed against the vendored Vale 3.18.0 binary rather than guessed, and you were right: a metric check with a formula panics when condition is absent, present-but-null (condition: with nothing after the colon), or a blank string, all three with panic: interface conversion: interface {} is float64, not bool. Note that is a different panic from the sequence one quoted in the doc comments, and it fires while the rule runs on a document rather than while it compiles; both comments have been corrected. Other condition types (number, bool, list, map) fail the decode cleanly with an E201, so they are left to the ordinary path. The guard is now structural like the neighbouring sequence arm, and two corpus rows (shape/metric-condition-null, shape/metric-condition-blank) record the measurement. Verified they fail "is not too lax" against the old guard.

VALE_PERMISSIVE_CHECKS. Wired in rather than dropped, since PR #177 (feat/generate-vale-schema, stacked on this branch) imports it from the generated vocabulary and re-exports it here. Rather than exporting CHECK_FIELDS to assert on the table, the new the permissive checks stay permissive block asserts the observable invariant: a foreign field is accepted on every check in VALE_PERMISSIVE_CHECKS and rejected with E201 on every other check type. Same guarantee, no new production export, and it needs no binary.

— AI Coding Agent