feat: add mix usage_rules.validate for reference validation by lukegalea · Pull Request #88 · ash-project/usage_rules

lukegalea · 2026-09-20T19:45:30Z

Implements mix usage_rules.validate from #87.

What it does

Validates references in usage-rules-managed markdown so rotting refs break CI instead of quietly mis-teaching agents:

Module + function refs (Module, Module.fun/arity, :erl_mod.fun/1) extracted from inline code spans and elixir/iex fences (fence-state aware, strings/comments/URLs stripped; all-caps doc basenames like README/SKILL not treated as modules). Resolution: compiled beams (project + deps) + static defmodule scan of lib/umbrella/deps sources for uncompiled projects (warnings, not errors). Function checks via module_info(:exports) + __info__(:macros) + behaviour_info(:callbacks) — arity mismatches report available arities.
Mix task refs (mix task.name from spans + shell fences), including project aliases.
Relative links in managed files.
Scope by default = exactly what sync manages: composed :file + *.md under skills whose SKILL.md carries managed-by: usage-rules. --all escapes scope (excludes deps/_build/doc/hidden). --strict, --format human|json, nonzero exit on violations.

Verification

155 tests, 0 failures (127 pre-existing + 28 new; fixture with 9 deliberate breakages + false-positive non-detections).
e2e in a scratch project: sync → planted breakages caught with file+line, exit 1; JSON well-formed; fixed + re-sync → exit 0; --all flags README refs.

Open to reshaping API/scoping to fit project direction (e.g. wiring into sync as a post-step instead of a separate task).

References #87.

Adds a validator that checks usage-rules-managed files for broken references: - Module, Module.function/arity, and :erlang.module/arity references in code spans and elixir/iex code blocks, resolved against compiled beams of the project and its deps with a static lib/ source scan fallback - Function checks cover exports, macros, and behaviour callbacks - mix task.name references, resolved against dep task modules and project aliases - Relative markdown links, resolved against the containing file By default only rules-managed files are checked (the composed :file and skills marked managed-by: usage-rules); --all validates every project markdown file. Supports --format json, --strict, and exits nonzero on violations for CI use.

zachdaniel · 2026-09-20T20:35:45Z

ex_doc actually has code that does this for documentation, would it be possible instead of rolling our own to use their implementation, only when its compiled otherwise fail? This is a great idea 👌. Library authors should definitely use this to keep their usage rules up to date!

@doc

Resolve references with ex_doc's own machinery instead of a custom implementation, per review feedback: - inline code spans are extracted with ex_doc's markdown pipeline (ExDoc.Markdown + EarmarkParser), so span detection matches what a docs build would autolink - candidates (spans, elixir/iex fence tokens, mix task commands) are parsed and resolved with ExDoc.Autolink.url/3 in strict custom-link mode, backed by ExDoc.Refs; ex_doc's own warning messages become violations, reported with file and line - references now validate exactly like docs-build references: only documented API passes, @doc false/@moduledoc false targets are flagged, callbacks require the c: prefix, and m:/t: references are supported - validation requires ex_doc to be compiled and fails with an actionable error otherwise; usage_rules still compiles in projects without ex_doc - relative markdown link checks are unchanged mix task name mentions in spans/fences are still validated including command lines like `mix task --flags` (only the task name), and bare atoms, all-caps document basenames, and bare lowercase function mentions are skipped as before to avoid false positives.

lukegalea · 2026-09-21T05:02:27Z

Done — validation now delegates to ex_doc instead of rolling our own resolution (d53e059).

How it works

Inline code spans are extracted with ex_doc's markdown pipeline (ExDoc.Markdown + EarmarkParser), so span detection matches what a docs build would autolink, including line numbers.
Every candidate — spans, elixir/iex fence tokens, mix task commands — is resolved through ExDoc.Autolink.url/3 in strict custom-link mode, backed by ExDoc.Refs. Refs that resolve return a URL; refs that parse but don't resolve make ex_doc emit its own warnings, which we capture (warnings: :send) and report as violations with file + line.
Because resolution is ex_doc's, semantics now match a docs build exactly: only documented API validates (@doc false/@moduledoc false targets are flagged), callbacks require the c: prefix, m:/t: refs are supported, and arity mismatches are reported with ex_doc's messages.

When ex_doc isn't compiled

ex_doc is loaded via Code.ensure_loaded (plus Application.ensure_all_started(:ex_doc)); usage_rules itself still compiles in projects without ex_doc (no compile-time struct/alias deps on it). When it's missing at validation time, the task fails fast with an actionable error pointing at {:ex_doc, "~> 0.37", only: [:dev, :test]} + mix deps.get && mix deps.compile. Uncompiled modules therefore fail rather than downgrade to warnings, as you suggested.

Kept identical: file scope, flags (--all/--strict/--format json), report shape, exit codes. Relative-link checks are unchanged (filesystem truth — ex_doc's :extras model only applies inside a docs build). Fences are still scanned since ex_doc deliberately skips pre blocks — but their tokens resolve through ex_doc too. To keep false positives away we still skip bare :atoms, all-caps document basenames (README, SKILL), and bare lowercase function mentions, and mix task --flags spans validate only the task name.

Two deliberate behavior narrowings worth noting: references without explicit arity (Mod.fun) no longer validate (ex_doc's ref grammar requires Mod.fun/arity), and messages come from ex_doc so the old "available arities" detail is gone. Full suite green (160 tests), credo strict + formatter clean. Staying in draft — happy to reshape further.

zachdaniel · 2026-09-21T15:09:45Z

+    * `*.md` files under skills managed by usage-rules (skills whose
+      `SKILL.md` contains `managed-by: usage-rules`)
+
+  Use `--all` to validate every markdown file in the project instead


Don't think we really need a --all

zachdaniel · 2026-09-21T15:10:16Z

+    location = Keyword.get(skills_config, :location, ".claude/skills")
+
+    Path.wildcard(Path.join(location, "*/SKILL.md"))
+    |> Enum.filter(&(File.read!(&1) =~ "managed-by: usage-rules"))


Should we do a simpler String.contains? here?

zachdaniel · 2026-09-21T15:11:33Z

+  defp ex_doc_config(context, path) do
+    # Built with struct/2 (not struct syntax) so this module still compiles
+    # in projects that do not have ex_doc compiled.
+    struct(ExDoc.Autolink,


We can make this compile conditionally on ExDoc being available

zachdaniel · 2026-09-21T15:12:48Z

I don't think this really uses ex_doc the way that I had hoped. ExDoc already emits warnings, and I don't see a case where we would need a json output, so ideally in my mind we can say something like ExDoc.parse_file(the_file) and let it output the warnings it outputs today or something like that. As it stands now we're still doing tons of the work on our own 😓

zachdaniel · 2026-09-21T15:13:45Z

There is also a world where maybe this should be an enhancement made to ex_doc actually? Like a validate_additional_docs: [...] that people could use?

Drive each file through ExDoc.Extras.build + ExDoc.Formatter.autolink and surface ex_doc's own warnings (file:line, native format), exiting nonzero via ex_doc's warned flag. This drops our own fence state machine, candidate extraction, violation mapping, and the JSON report (--all, --strict and --format are gone; explicit file paths are accepted instead). Two scoped complements cover what docs builds stay silent about, pushing spans through ex_doc's own resolution and re-emitting its warning text: mix task mentions and bare undefined dotted module mentions in code spans.

lukegalea · 2026-09-21T19:52:53Z

Great call — way simpler like this, but it misses a few checks like bare undefined module spans (easy to special-case, and I've done exactly that here). Proposed the ex_doc home for the general feature here: elixir-lang/ex_doc#2272. Suggest sticking with this thinned prototype in the interim.

Gate everything that touches ex_doc modules behind Code.ensure_loaded?(ExDoc) so usage_rules still compiles cleanly in projects without ex_doc compiled; validate/1 there fails fast with the existing actionable error. Also use String.contains?/2 for the managed-by marker check.

Drop the custom span-complement machinery: mix task mentions and bare undefined module mentions in plain code spans now stay silent, matching ex_doc's behavior inside moduledocs (per elixir-lang/ex_doc#2272). The validator is now thin glue — feed files through ExDoc.Extras.build + ExDoc.Formatter.autolink and surface ex_doc's own warnings — with file scope discovery staying in the mix task. Reword docs mentions of hidden ex_doc functions so mix docs builds without warnings, and document the zero-dependency EXTRA_DOCS docs-extras alternative in the README.

lukegalea · 2026-09-22T03:09:38Z

Pushed a995915 + b894b5b addressing your comments and taking the "lean on ex_doc" direction to its conclusion:

Conditional compilation (a995915): all ExDoc-touching code now lives behind a module-level if Code.ensure_loaded?(ExDoc) — usage_rules compiles with zero warnings in projects without ex_doc, and validate still fails fast with the actionable error when ex_doc is missing at runtime (verified by compiling the module with the conditional forced false).
Marker check is now String.contains?(File.read!(path), "managed-by: usage-rules") as you suggested.
--all — confirmed already absent at the head you reviewed past; no references left in code, tests, or README.
Warnings are now exactly ex_doc's own (b894b5b): deleted the rest of the custom machinery — the bare-module-span special case, mix-task span checks, and the warning re-emission plumbing. The validator is now thin glue: scoped file discovery → ExDoc.Extras.build → ExDoc.Formatter.autolink → surface ex_doc's warnings with file attribution. Bare spans staying silent now matches a HexDocs build exactly, which is also the stance settled upstream in Option to validate additional standalone markdown files through the extras pipeline (e.g. validate_additional_docs) elixir-lang/ex_doc#2272 (José's position there: no ex_doc change needed — env-gated extras covers it).

Relative-filesystem-link checks remain, since ex_doc's extras model doesn't cover those.

Verification: 143 tests 0 failures · mix credo --strict clean · mix compile --force --warnings-as-errors clean · mix docs 0 warnings (the moduledoc refs to hidden ExDoc.* functions are reworded to plain text) · end-to-end mix usage_rules.validate usage-rules.md exits 0.

The README also gained a "zero-dependency alternative" section documenting the EXTRA_DOCS-gated extras snippet from the ex_doc thread, for library authors who'd rather not add any tooling. Staying in draft — happy to reshape further.

lukegalea mentioned this pull request Sep 21, 2026

Proposal: a serialized semantic layer over Spark DSL state — source spans, stable ids, provenance ash-project/spark#293

Closed

lukegalea marked this pull request as ready for review September 21, 2026 05:03

zachdaniel reviewed Sep 21, 2026

View reviewed changes

lukegalea mentioned this pull request Sep 21, 2026

Option to validate additional standalone markdown files through the extras pipeline (e.g. validate_additional_docs) elixir-lang/ex_doc#2272

Closed

lukegalea added 2 commits September 22, 2026 03:05

Navigation Menu

Uh oh!

feat: add mix usage_rules.validate for reference validation - #88

feat: add mix usage_rules.validate for reference validation#88
lukegalea wants to merge 5 commits into
ash-project:mainfrom
lukegalea:validate-references

lukegalea commented Sep 20, 2026

Uh oh!

zachdaniel commented Sep 20, 2026

Uh oh!

lukegalea commented Sep 21, 2026

Uh oh!

zachdaniel Sep 21, 2026

Uh oh!

zachdaniel Sep 21, 2026

Uh oh!

zachdaniel Sep 21, 2026

Uh oh!

zachdaniel commented Sep 21, 2026

Uh oh!

zachdaniel commented Sep 21, 2026

Uh oh!

lukegalea commented Sep 21, 2026

Uh oh!

lukegalea commented Sep 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lukegalea commented Sep 20, 2026

What it does

Verification

Uh oh!

zachdaniel commented Sep 20, 2026

Uh oh!

lukegalea commented Sep 21, 2026

Uh oh!

zachdaniel Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

zachdaniel Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

zachdaniel Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

zachdaniel commented Sep 21, 2026

Uh oh!

zachdaniel commented Sep 21, 2026

Uh oh!

lukegalea commented Sep 21, 2026

Uh oh!

lukegalea commented Sep 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants