Skip to content

feat(cli): add taskless ast <file> to show a file's named-node AST - #683

Merged
theCodeDrift merged 3 commits into
mainfrom
feat/ast-command
Nov 12, 2026
Merged

theCodeDrift merged 3 commits into
mainfrom
feat/ast-command

Conversation

@theCodeDrift

Copy link
Copy Markdown
Member

Rule authors had no way to see a language's node kinds. create-sg-rule said to read them off --debug-query=ast without saying how to run ast-grep, so agents went hunting for the binary (~/.npm/_npx, /usr/bin/sg, which is shadow-utils on Linux) or pasted whole files as patterns.

What changes

  • taskless ast <file> prints the file's named-node AST, parsed by the bundled ast-grep. Because it shows named nodes only, every kind it prints is one kind: accepts. Each node has its field name and line:column range (1-based lines, 0-based byte columns), and leaves show their source text.
    • Language: by default the command asks the binary itself (scan --inspect entity against an empty temp config), so it gets the same resolution check does, with no second extension table. A stray sgconfig.yml with languageGlobs is ignored, since check does not use it either.
    • --lang <Language> overrides the language (any spelling language: accepts). --range <line>[:<line>] keeps only the nodes on those lines plus the nodes that enclose them. --json gives the tree as data, validated against a zod schema.
    • Errors use the standard envelope: INVALID_INPUT (missing, empty, unrecognized or oversized file, bad flag), ENGINE_UNAVAILABLE, SCAN_FAILED.
  • create-sg-rule (topic v15) tells authors to read every kind: off ast and no longer mentions --debug-query. The sg-regex-needs-kind message from validate now names ast.
  • OpenSpec: a new cli-ast capability and one added cli-agent-authoring requirement, archived in this PR (single-PR delivery).

How it works, and the limit

ast-grep has no command that dumps a file's tree. The only one is run --debug-query=ast, which dumps the pattern, so the command passes the file contents as the pattern. Measured at 0.51.1, the result is exact:

  • Positions are not shifted by leading whitespace.
  • PHP $FOO still dumps as variable_name, so pattern preprocessing does not affect it.
  • The tree is printed to stderr before ast-grep rejects the multi-node pattern with exit 8.

The parser for that text format is pinned by a test against the binary for all 29 languages.

The cost: the file goes through argv, so a file past the platform's argument limit (128 KiB per argument on Linux) is refused with a clear INVALID_INPUT. The issue's optional --pattern filter is not included; --range already solves the problem of long dumps.

Tests

test/ast-command.test.ts: parser, range filter and language-probe units; contract tests against the bundled binary (every language dumps, every language's extension resolves, sgconfig.yml ignored, PHP $FOO, ERROR counting); built-CLI tests for text and JSON output, --range, --lang, a YAML file that starts with -, and each refusal. Full suite: 3218 passed, 11 skipped. pnpm typecheck and pnpm lint are clean.

Closes #673

@theCodeDrift

Copy link
Copy Markdown
Member Author

@claude /review

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @theCodeDrift's task in 3m 26s —— View job


Review mode: incremental — read 0 prior review thread(s) before reviewing.

No prior reviews, review threads, or top-level comments exist on this PR beyond the trigger comment itself, so this is the first review — everything below is a fresh assessment of the full diff.

  • Read prior review data (none found)
  • Fetch PR diff and changed files
  • Review command implementation (ast.ts, ast-dump.ts, schemas/ast.ts)
  • Review doc/spec changes (create-sg-rule.md, openspec specs)
  • Review tests
  • Post inline comments for findings
  • Post final summary

Summary

This is a well-scoped, carefully documented command. The approach (driving run --debug-query=ast with the file as pattern, probing language via scan --inspect entity) is unconventional but the tradeoffs are spelled out in comments and backed by contract tests against the real binary across all 29 languages. Error handling follows the standard envelope consistently, and the openspec/docs/changeset updates are coherent with the implementation.

Two [New] findings below, both narrow edge cases rather than blockers — posted as inline comments:

  1. packages/cli/src/rules/ast-dump.ts — reading the source file as a UTF-8 string (readFile(path, "utf8")) and later re-encoding that string back to bytes (attachLeafText, and implicitly when it's passed to spawn as --pattern=${source}) means a file that isn't valid UTF-8 gets silently mangled (invalid byte sequences become U+FFFD) before ast-grep ever sees it. The dump would reflect the corrupted text, with no error raised, rather than the real file content, for any source using a non-UTF-8 encoding.

  2. packages/cli/src/rules/ast-dump.ts (detectAstLanguage) — the probe uses scan --inspect entity against the real target file/cwd, which means ast-grep's own ignore rules (.gitignore, hidden files, etc.) can cause the file to be skipped during the walk rather than resolved. In that case detectAstLanguage returns undefined indistinguishably from "unrecognized extension," so ast.ts reports ast-grep does not recognize the extension of <file> even when the real cause is that the file is ignored (e.g. a fixture under node_modules, or a dotfile) — a misleading diagnostic for an otherwise-correct extension. This is consistent with check's own resolution (intentional per the PR's stated goal), but the error message doesn't account for this cause, and it isn't covered by a test.

Everything else checked out:

  • parseLineRange / filterByRange / lastLine boundary logic (column-0 end-of-line exclusion, range-not-touched failure) matches the documented semantics and is exercised by both unit and CLI tests.
  • Argument-injection risk from passing raw file bytes through spawn as argv is avoided — spawn is used with an array (no shell), and --pattern= with = correctly defends against a file path/content starting with -.
  • E2BIG/ENAMETOOLONG/NUL-byte failures from oversized files are caught and turned into a clean INVALID_INPUT, tested with a real oversized file.
  • The --stdin flag correctly prevents a "lucky" single-node pattern from silently falling back to scanning the whole cwd.
  • verify.ts's new withKindPointer wrapper and the sg-regex-needs-kind message update are minimal and consistent with the rest of the file's existing (non-alphabetized) import ordering.
  • Command registration (names.ts, index.ts) is correctly placed.
  • create-sg-rule.md edits consistently replace every --debug-query reference and are covered by the updated agent-extensions.test.ts assertion.
  • OpenSpec requirements/scenarios in both the archived change and the live cli-ast/cli-agent-authoring specs match the implemented behavior (language resolution, range cutting, JSON envelope, error taxonomy).

Comment thread packages/cli/src/commands/ast.ts Resolved
Comment thread packages/cli/src/rules/ast-dump.ts Resolved
@theCodeDrift

Copy link
Copy Markdown
Member Author

Re: @github-actions — "Claude finished @theCodeDrift's task in 3m 26s…"
#683 (comment)

Both findings handled in 11ae471: non-UTF-8 input is now refused with INVALID_INPUT rather than silently mangled, and the ignored-file case turned out not to occur (ast-grep never filters an explicitly given path), now pinned by a test. Details on each inline thread.

— AI Coding Agent

Rule authors had no way to see a language's node kinds short of locating
the bundled ast-grep binary by hand. `ast` dumps the named-node tree with
the pinned ast-grep, resolving the language the way `check` does (asked of
the binary), with --lang, --range and --json. create-sg-rule and the
sg-regex-needs-kind message now point at it instead of --debug-query.

Closes #673
A lenient utf8 read replaced invalid bytes with U+FFFD, so the dump showed
a tree of text the file does not hold. Decode strictly and fail with
INVALID_INPUT instead. Also pin that the language probe resolves files
ignore rules would skip in a walk, since ast-grep never filters a path it
is given.
…nd issue

#682 moved sg-regex-needs-kind into the ast-grep rule schema, so the
validateRequirements loop that appended the `taskless ast <file>` pointer
is gone. Apply the pointer in validateSchema's formatter instead, to both
the layer error and its attributed violation, and point the recipe's
regex/kind section at `taskless ast` rather than --debug-query=ast.
@theCodeDrift

Copy link
Copy Markdown
Member Author

Rebased onto main after #682 and #691 landed. #682 moved sg-regex-needs-kind out of the validateRequirements loop and into the ast-grep rule schema, which removed the call sites where this PR appended the taskless ast <file> pointer. Resolution, in its own commit (fix(validate): carry the ast pointer onto the schema-layer regex/kind issue):

— AI Coding Agent

@theCodeDrift
theCodeDrift merged commit 1a83e75 into main Nov 12, 2026
4 checks passed
@theCodeDrift
theCodeDrift deleted the feat/ast-command branch November 12, 2026 04:48
theCodeDrift added a commit that referenced this pull request Nov 12, 2026
Rebasing onto main brought in text from #682, #683, #686 and #691 that
pushed three outputs past 8 KiB:

- create-sg-rule outline: the `taskless ast` guidance moves to an
  on-demand Reading kinds section, step 4 points at it, and step 5 drops
  its quirk enumeration (the Quirks table lists them).
- create-sg-rule pattern-quirks: the relational entries (follows/precedes,
  has/inside, not) move to a Relational quirks section; validate-rule
  names it beside the other two.
- route outline: #686's note on recipes that limit questions moves to an
  on-demand Limits on asking section, with a one-line pointer in
  Preconditions.
theCodeDrift added a commit that referenced this pull request Nov 12, 2026
Rebasing onto main brought in text from #682, #683, #686 and #691 that
pushed three outputs past 8 KiB:

- create-sg-rule outline: the `taskless ast` guidance moves to an
  on-demand Reading kinds section, step 4 points at it, and step 5 drops
  its quirk enumeration (the Quirks table lists them).
- create-sg-rule pattern-quirks: the relational entries (follows/precedes,
  has/inside, not) move to a Relational quirks section; validate-rule
  names it beside the other two.
- route outline: #686's note on recipes that limit questions moves to an
  on-demand Limits on asking section, with a one-line pointer in
  Preconditions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add taskless ast <file> to show a file's named-node AST for rule authoring

1 participant