diff --git a/.changeset/ast-command.md b/.changeset/ast-command.md new file mode 100644 index 00000000..4b01b73a --- /dev/null +++ b/.changeset/ast-command.md @@ -0,0 +1,5 @@ +--- +"@taskless/cli": patch +--- + +New `taskless ast ` prints a file's named-node AST, parsed by the bundled ast-grep as the language the file's extension resolves to, so every kind it shows is one a rule's `kind:` accepts. `--range [:]` cuts the tree to the nodes on those lines, `--lang` overrides the language, and `--json` emits the tree as data. `create-sg-rule` now sends authors to it for kinds instead of to ast-grep's `--debug-query`, which left agents hunting for an ast-grep binary. diff --git a/openspec/changes/archive/2026-11-11-ast-command/.openspec.yaml b/openspec/changes/archive/2026-11-11-ast-command/.openspec.yaml new file mode 100644 index 00000000..2a7773d6 --- /dev/null +++ b/openspec/changes/archive/2026-11-11-ast-command/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-11 diff --git a/openspec/changes/archive/2026-11-11-ast-command/proposal.md b/openspec/changes/archive/2026-11-11-ast-command/proposal.md new file mode 100644 index 00000000..0082aaf3 --- /dev/null +++ b/openspec/changes/archive/2026-11-11-ast-command/proposal.md @@ -0,0 +1,27 @@ +## Why + +Writing an ast-grep rule means knowing the node kinds of the target language, and the CLI gave no way to see them. `create-sg-rule` told authors to read kinds off `--debug-query=ast` without saying how to run ast-grep, so agents went hunting for the binary: under `~/.npm/_npx` or `.taskless/.cache`, into `/usr/bin/sg` (shadow-utils on Linux, not ast-grep), or by pasting whole files as patterns. About 7 of 40 onboarding runs in the October to November agent survey reported it, across C#, Dart, Rust and PHP (#673). + +## What Changes + +- New `taskless ast ` prints the file's named-node AST, parsed by the bundled ast-grep as the language the file's extension resolves to. The language is asked of the binary (a `scan --inspect entity` probe against an empty config), so it is the resolution `check` gets, not a second table. +- Named nodes only, so every kind shown is one `kind:` accepts. Each node carries its field name, 1-based line and 0-based byte column range, and leaves carry their source text. +- `--lang ` overrides the language, `--range [:]` cuts the tree to the nodes touching those lines and the nodes enclosing them, `--json` emits the tree as data. Failures under `--json` use the standard envelope: `INVALID_INPUT` for a missing, empty, unrecognized or oversized file and for a bad flag, `ENGINE_UNAVAILABLE` when there is no ast-grep, `SCAN_FAILED` when ast-grep prints no tree. +- `create-sg-rule` (topic v15) tells authors to read every `kind:` off `ast`, and stops naming `--debug-query`. The `sg-regex-needs-kind` message names `ast` as where to find a kind. + +The tree comes from `ast-grep run --debug-query=ast` with the file's contents as the pattern, the only file-tree output ast-grep has. That passes the file through argv, so a file past the platform's argument limit (128 KiB per argument on Linux) is refused with a message saying so. The `--pattern` subtree filter the issue lists as optional is left out: `--range` covers the length problem without a second ast-grep invocation. + +## Capabilities + +### New Capabilities + +- `cli-ast`: the `taskless ast` command. + +### Modified Capabilities + +- `cli-agent-authoring`: the ast-grep authoring recipe directs kind lookups to `taskless ast`. + +## Impact + +- `packages/cli/src/commands/ast.ts`, `src/rules/ast-dump.ts`, `src/schemas/ast.ts`, the command registry, `src/rules/verify.ts`, and the `create-sg-rule` recipe. +- Delivery shape: **single PR**. One read-only command, its recipe pointer, and tests, well under the review size. diff --git a/openspec/changes/archive/2026-11-11-ast-command/specs/cli-agent-authoring/spec.md b/openspec/changes/archive/2026-11-11-ast-command/specs/cli-agent-authoring/spec.md new file mode 100644 index 00000000..e367bb68 --- /dev/null +++ b/openspec/changes/archive/2026-11-11-ast-command/specs/cli-agent-authoring/spec.md @@ -0,0 +1,11 @@ +## ADDED Requirements + +### Requirement: The ast-grep authoring recipe sends kind lookups to the ast command + +`create-sg-rule` SHALL tell the author to read every `kind:` off `taskless ast ` run on a file the rule targets, and SHALL name its `--range`, `--lang` and `--json` options. It SHALL NOT direct the author to run ast-grep's `--debug-query` or to locate an ast-grep binary. The `sg-regex-needs-kind` message from `validate` SHALL name the same command as where to find a kind to add. + +#### Scenario: The recipe names the command + +- **WHEN** an agent runs `taskless agent create-sg-rule` +- **THEN** the recipe SHALL contain an invocation of `ast ` +- **AND** it SHALL NOT contain `--debug-query` diff --git a/openspec/changes/archive/2026-11-11-ast-command/specs/cli-ast/spec.md b/openspec/changes/archive/2026-11-11-ast-command/specs/cli-ast/spec.md new file mode 100644 index 00000000..b07c1dd9 --- /dev/null +++ b/openspec/changes/archive/2026-11-11-ast-command/specs/cli-ast/spec.md @@ -0,0 +1,51 @@ +## ADDED Requirements + +### Requirement: The ast command prints a file's named-node AST + +The CLI SHALL provide `taskless ast `, which prints the named-node syntax tree of one file as the ast-grep bundled with the CLI parses it. It SHALL print named nodes only, so that every kind it shows is one a rule's `kind:` accepts. Each node SHALL carry its kind, the field it fills in its parent when the grammar names one, and its start and end as a 1-based line and a 0-based byte column. A node with no named children SHALL carry its source text. The command SHALL read the file and write nothing, and SHALL NOT require `.taskless/` to exist. It SHALL accept `--dir`/`-d`, against which a relative file path resolves. + +#### Scenario: A file is dumped by its extension + +- **WHEN** `taskless ast src/a.ts` is run on a file holding `const a = 1;` +- **THEN** the output SHALL name the language `TypeScript` +- **AND** it SHALL list `lexical_declaration`, `variable_declarator` and `name: identifier` with their positions +- **AND** it SHALL NOT list the `const` keyword as a node + +#### Scenario: JSON output + +- **WHEN** `taskless ast --json` succeeds +- **THEN** stdout SHALL be one JSON object with `ok: true`, `file`, `language`, `errorNodes` and a nested `root` node +- **AND** `errorNodes` SHALL count the `ERROR` nodes in the whole tree, so a file that did not parse cleanly as that language is visible + +### Requirement: The ast command uses check's language resolution + +Without `--lang`, the command SHALL parse the file as the language the bundled ast-grep assigns its path, asked of the binary rather than of a table in the CLI, and SHALL NOT apply the `languageGlobs` of an `sgconfig.yml` found above the working directory, since `check` does not. `--lang` SHALL take any spelling a rule's `language:` accepts and parse as the language it resolves to. + +#### Scenario: An extension ast-grep does not recognize + +- **WHEN** `taskless ast notes.unknownext --json` is run +- **THEN** stdout SHALL be an `INVALID_INPUT` envelope that says the extension is not recognized and lists the languages `--lang` accepts +- **AND** the exit code SHALL be 1 + +#### Scenario: A language override + +- **WHEN** `taskless ast list.txt --lang yaml` is run +- **THEN** the file SHALL be parsed as `Yaml` + +### Requirement: The ast command can cut the tree to a line range + +`--range ` or `--range :`, 1-based and inclusive, SHALL cut the tree to the nodes whose lines overlap the range, keeping the nodes that enclose them. A node that ends at column 0 of a line SHALL NOT count as on that line. A range that no node touches, and a malformed range, SHALL fail with `INVALID_INPUT`. + +#### Scenario: One line of a file + +- **WHEN** a file's third line is `foo(b);` and `taskless ast --range 3` is run +- **THEN** the root SHALL keep only the `expression_statement` on line 3 among its children + +### Requirement: The ast command reports what it cannot dump + +The command SHALL fail with the standard `--json` error envelope and exit code 1, and SHALL NOT print a partial tree, when the file is missing, empty, a directory or not valid UTF-8, when `--lang` names no ast-grep language, when the file is too large to hand to ast-grep on this platform (each `INVALID_INPUT`), when no ast-grep binary is found (`ENGINE_UNAVAILABLE`), and when ast-grep prints no tree it can parse (`SCAN_FAILED`). + +#### Scenario: A file too large for the command line + +- **WHEN** `taskless ast big.js --json` is run on a file past the platform's argument limit +- **THEN** stdout SHALL be an `INVALID_INPUT` envelope saying the file is too large to dump diff --git a/openspec/changes/archive/2026-11-11-ast-command/tasks.md b/openspec/changes/archive/2026-11-11-ast-command/tasks.md new file mode 100644 index 00000000..891c1c6b --- /dev/null +++ b/openspec/changes/archive/2026-11-11-ast-command/tasks.md @@ -0,0 +1,17 @@ +## 1. Command + +- [x] 1.1 Resolve a file's language by asking the bundled ast-grep (`scan --inspect entity` against an empty config) +- [x] 1.2 Dump the named-node tree through `run --debug-query=ast` and parse it, with fields, positions and leaf text +- [x] 1.3 Add `taskless ast ` with `--lang`, `--range`, `--json`, `-d`, and the standard error envelope +- [x] 1.4 Register `ast` in the subcommand registry + +## 2. Pointers + +- [x] 2.1 `create-sg-rule`: read kinds off `ast`, drop `--debug-query`, bump the topic to v15 +- [x] 2.2 `sg-regex-needs-kind`: name `ast` as where to find a kind + +## 3. Tests + +- [x] 3.1 Parser, range filter and language-probe units +- [x] 3.2 Against the bundled binary: every language dumps in the parsed format, every language's extension resolves, `sgconfig.yml` is ignored, `$FOO` in PHP is dumped as written +- [x] 3.3 The built CLI: text and JSON output, `--range`, `--lang`, and each `INVALID_INPUT` refusal, including an oversized file diff --git a/openspec/specs/cli-agent-authoring/spec.md b/openspec/specs/cli-agent-authoring/spec.md index ab7d89fb..028de1ad 100644 --- a/openspec/specs/cli-agent-authoring/spec.md +++ b/openspec/specs/cli-agent-authoring/spec.md @@ -266,3 +266,13 @@ A per-engine recipe SHALL keep only what is that engine's own, such as Vale's sc - **WHEN** the agent writes fixtures for a rule that declares `files:` or Vale matchers - **THEN** the shared section SHALL direct it to place each case's files at a path the scope accepts + +### Requirement: The ast-grep authoring recipe sends kind lookups to the ast command + +`create-sg-rule` SHALL tell the author to read every `kind:` off `taskless ast ` run on a file the rule targets, and SHALL name its `--range`, `--lang` and `--json` options. It SHALL NOT direct the author to run ast-grep's `--debug-query` or to locate an ast-grep binary. The `sg-regex-needs-kind` message from `validate` SHALL name the same command as where to find a kind to add. + +#### Scenario: The recipe names the command + +- **WHEN** an agent runs `taskless agent create-sg-rule` +- **THEN** the recipe SHALL contain an invocation of `ast ` +- **AND** it SHALL NOT contain `--debug-query` diff --git a/openspec/specs/cli-ast/spec.md b/openspec/specs/cli-ast/spec.md new file mode 100644 index 00000000..f860916f --- /dev/null +++ b/openspec/specs/cli-ast/spec.md @@ -0,0 +1,57 @@ +# cli-ast Specification + +## Purpose + +The `taskless ast` subcommand: a read-only dump of one file's named-node syntax tree, as the bundled ast-grep parses it, so a rule author can read real `kind:` names instead of guessing them. + +## Requirements + +### Requirement: The ast command prints a file's named-node AST + +The CLI SHALL provide `taskless ast `, which prints the named-node syntax tree of one file as the ast-grep bundled with the CLI parses it. It SHALL print named nodes only, so that every kind it shows is one a rule's `kind:` accepts. Each node SHALL carry its kind, the field it fills in its parent when the grammar names one, and its start and end as a 1-based line and a 0-based byte column. A node with no named children SHALL carry its source text. The command SHALL read the file and write nothing, and SHALL NOT require `.taskless/` to exist. It SHALL accept `--dir`/`-d`, against which a relative file path resolves. + +#### Scenario: A file is dumped by its extension + +- **WHEN** `taskless ast src/a.ts` is run on a file holding `const a = 1;` +- **THEN** the output SHALL name the language `TypeScript` +- **AND** it SHALL list `lexical_declaration`, `variable_declarator` and `name: identifier` with their positions +- **AND** it SHALL NOT list the `const` keyword as a node + +#### Scenario: JSON output + +- **WHEN** `taskless ast --json` succeeds +- **THEN** stdout SHALL be one JSON object with `ok: true`, `file`, `language`, `errorNodes` and a nested `root` node +- **AND** `errorNodes` SHALL count the `ERROR` nodes in the whole tree, so a file that did not parse cleanly as that language is visible + +### Requirement: The ast command uses check's language resolution + +Without `--lang`, the command SHALL parse the file as the language the bundled ast-grep assigns its path, asked of the binary rather than of a table in the CLI, and SHALL NOT apply the `languageGlobs` of an `sgconfig.yml` found above the working directory, since `check` does not. `--lang` SHALL take any spelling a rule's `language:` accepts and parse as the language it resolves to. + +#### Scenario: An extension ast-grep does not recognize + +- **WHEN** `taskless ast notes.unknownext --json` is run +- **THEN** stdout SHALL be an `INVALID_INPUT` envelope that says the extension is not recognized and lists the languages `--lang` accepts +- **AND** the exit code SHALL be 1 + +#### Scenario: A language override + +- **WHEN** `taskless ast list.txt --lang yaml` is run +- **THEN** the file SHALL be parsed as `Yaml` + +### Requirement: The ast command can cut the tree to a line range + +`--range ` or `--range :`, 1-based and inclusive, SHALL cut the tree to the nodes whose lines overlap the range, keeping the nodes that enclose them. A node that ends at column 0 of a line SHALL NOT count as on that line. A range that no node touches, and a malformed range, SHALL fail with `INVALID_INPUT`. + +#### Scenario: One line of a file + +- **WHEN** a file's third line is `foo(b);` and `taskless ast --range 3` is run +- **THEN** the root SHALL keep only the `expression_statement` on line 3 among its children + +### Requirement: The ast command reports what it cannot dump + +The command SHALL fail with the standard `--json` error envelope and exit code 1, and SHALL NOT print a partial tree, when the file is missing, empty, a directory or not valid UTF-8, when `--lang` names no ast-grep language, when the file is too large to hand to ast-grep on this platform (each `INVALID_INPUT`), when no ast-grep binary is found (`ENGINE_UNAVAILABLE`), and when ast-grep prints no tree it can parse (`SCAN_FAILED`). + +#### Scenario: A file too large for the command line + +- **WHEN** `taskless ast big.js --json` is run on a file past the platform's argument limit +- **THEN** stdout SHALL be an `INVALID_INPUT` envelope saying the file is too large to dump diff --git a/packages/cli/src/agent/create-sg-rule.md b/packages/cli/src/agent/create-sg-rule.md index a380beb5..7343449b 100644 --- a/packages/cli/src/agent/create-sg-rule.md +++ b/packages/cli/src/agent/create-sg-rule.md @@ -1,4 +1,4 @@ -# Topic: create-sg-rule (CLI v%(CLI_VERSION)s / topic v15) +# Topic: create-sg-rule (CLI v%(CLI_VERSION)s / topic v16) %(ONE_COMMAND_PER_CALL)s @@ -71,6 +71,24 @@ whole rule. Optional but useful: `note` (multi-line guidance, supports markdown), `fix` (auto-fix pattern), `ignores` (file patterns to skip). + **Read every `kind:` off a real file.** A kind is a node name from + the language's grammar, and guessing one is how a rule ends up a + config error or matching nothing. Print the tree of a file the rule + targets: + + ``` + %(TASKLESS_CLI)s ast + %(TASKLESS_CLI)s ast --range 40:60 + ``` + + It parses the file with the ast-grep this CLI ships, as the language + its extension resolves to, which is the parser `check` uses for it. + It prints named nodes only, so every kind it shows is one `kind:` + accepts. `--range ` or `--range :` keeps the nodes on + those lines and the ones enclosing them, `--lang ` parses + as another language, and `--json` gives the same tree as data. There + is no need to locate an ast-grep binary yourself. + **`language` is ast-grep's vocabulary.** The vendored rule schema types the field as a bare string with no enum, so the schema layer accepts anything. `validate` checks it separately: an off-list spelling @@ -103,7 +121,7 @@ whole rule. assumption passes anyway. The table says which ones `validate` does catch. Look again at: - a `$$$` next to a comma (the separator trap), - - any `kind:` you read off a `--debug-query=cst` dump, + - any `kind:` you did not read off `%(TASKLESS_CLI)s ast`, - a rule that keys on a call's callee, if callers write `await f(...)`, - a `has`, `inside`, `follows` or `precedes` without `stopBy`, or @@ -251,7 +269,7 @@ An `all:` with one branch that sets a `kind` counts too. A `kind` inside `not`, `has` or another relation does not, because it describes a different node. -To find the kind, run ast-grep with `--debug-query=ast` on a sample of +To find the kind, run `%(TASKLESS_CLI)s ast ` on a sample of the code and take the named node the regex should match. Under `constraints`, that is the node the metavariable captures, which is not always the obvious one: the method `$M` captures in `$OBJ.$M()` is @@ -330,7 +348,7 @@ behaves identically, so there is no version to wait for. ### A keyword is not a valid `kind` -`--debug-query=cst` prints keywords as nodes. For a constructor +A concrete syntax tree prints keywords as nodes. For a constructor parameter `readonly x: number` in TypeScript: ``` @@ -342,13 +360,13 @@ required_parameter but `kind: readonly` is a config error, ``Kind `readonly` is invalid``. `readonly` is an anonymous token, and `kind:` accepts only -named nodes. The CST dump prints both the same way, so reading it +named nodes. A CST printout shows both the same way, so reading one cannot tell you which names are valid kinds. This holds for every keyword, not only `readonly`. -Workaround: read kinds off `--debug-query=ast` instead, which prints -named nodes only. A name that appears there is a usable kind; one that -appears only in the CST dump is not. To match on the keyword, match +Workaround: read kinds off `%(TASKLESS_CLI)s ast ` instead, which +prints named nodes only. A name that appears there is a usable kind, and +a keyword never does. To match on the keyword, match its named parent and test the parent's text. Cover every shape the keyword appears in: after a decorator or an accessibility modifier (`private readonly x`), and on an optional parameter (`readonly x?`), diff --git a/packages/cli/src/commands/ast.ts b/packages/cli/src/commands/ast.ts new file mode 100644 index 00000000..10c98e16 --- /dev/null +++ b/packages/cli/src/commands/ast.ts @@ -0,0 +1,216 @@ +import { readFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import process from "node:process"; + +import { defineCommand } from "citty"; + +import { + AstDumpError, + attachLeafText, + countErrorNodes, + detectAstLanguage, + dumpAst, + filterByRange, + lastLine, + parseLineRange, + renderAst, + type LineRange, +} from "../rules/ast-dump"; +import { + astGrepLanguageList, + resolveAstGrepLanguage, + type AstGrepLanguage, +} from "../rules/capabilities"; +import { findSgBinary } from "../rules/scan"; +import { outputSchema as astOutputSchema } from "../schemas/ast"; +import { writeJsonError, type CLIErrorCode } from "../types/errors"; + +/** A failure this command reports itself, with the code `--json` carries. */ +class AstCommandError extends Error { + constructor( + message: string, + readonly code: CLIErrorCode + ) { + super(message); + } +} + +/** + * Decoding is strict: a lenient `"utf8"` read would turn every invalid byte + * into U+FFFD, and the dump would then show a tree of text the file does not + * hold, with nothing to say so. `ignoreBOM` keeps a byte-order mark in the + * string, so byte columns still count from the file's first byte. + */ +const strictUtf8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); + +async function readSource(path: string, given: string): Promise { + let bytes: Buffer; + try { + bytes = await readFile(path); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + throw new AstCommandError( + code === "ENOENT" + ? `No such file: ${given}` + : code === "EISDIR" + ? `${given} is a directory. \`ast\` takes one file.` + : `Could not read ${given}: ${(error as Error).message}`, + "INVALID_INPUT" + ); + } + try { + return strictUtf8.decode(bytes); + } catch { + throw new AstCommandError( + `${given} is not valid UTF-8. ast-grep parses UTF-8 text, so re-encode the file (or a copy) as UTF-8 first.`, + "INVALID_INPUT" + ); + } +} + +function resolveRange(value: string | undefined): LineRange | undefined { + if (value === undefined) return undefined; + const range = parseLineRange(value); + if (range === undefined) { + throw new AstCommandError( + `--range takes or :, 1-based with from <= to; got ${JSON.stringify(value)}.`, + "INVALID_INPUT" + ); + } + return range; +} + +function resolveLanguageOverride( + value: string | undefined +): AstGrepLanguage | undefined { + if (value === undefined) return undefined; + const language = resolveAstGrepLanguage(value); + if (language === undefined) { + throw new AstCommandError( + `ast-grep has no language ${JSON.stringify(value)}. Use one of: ${astGrepLanguageList()}.`, + "INVALID_INPUT" + ); + } + return language; +} + +function requireSgBinary(): string { + try { + return findSgBinary(); + } catch (error) { + throw new AstCommandError((error as Error).message, "ENGINE_UNAVAILABLE"); + } +} + +export const astCommand = defineCommand({ + meta: { + name: "ast", + description: + "Print a file's named-node AST as the bundled ast-grep parses it, to find the kinds a rule can name", + }, + args: { + file: { + type: "positional", + description: "The file to parse", + required: true, + }, + lang: { + type: "string", + description: + "Parse as this ast-grep language instead of the one the file's extension resolves to", + }, + range: { + type: "string", + description: + "Show only the nodes on these lines: or :, 1-based and inclusive", + }, + dir: { + type: "string", + alias: "d", + description: "Working directory", + }, + json: { + type: "boolean", + description: "Output as JSON", + default: false, + }, + }, + async run({ args }) { + const cwd = resolve(args.dir ?? process.cwd()); + // ast reads one file and writes nothing, so it emits no bespoke event; the + // per-invocation cli_run the runner emits covers it, as it does detect. + try { + const range = resolveRange(args.range); + const override = resolveLanguageOverride(args.lang); + + const source = await readSource(resolve(cwd, args.file), args.file); + if (source.length === 0) { + throw new AstCommandError( + `${args.file} is empty, so it has no AST to show.`, + "INVALID_INPUT" + ); + } + + const binary = requireSgBinary(); + const language = + override ?? (await detectAstLanguage(binary, cwd, args.file)); + if (language === undefined) { + throw new AstCommandError( + `ast-grep does not recognize the extension of ${args.file}, so \`check\` does not scan it either. Pass --lang with one of: ${astGrepLanguageList()}.`, + "INVALID_INPUT" + ); + } + + const tree = await dumpAst({ binary, cwd, source, language }); + attachLeafText(tree, source); + const errorNodes = countErrorNodes(tree); + const root = range === undefined ? tree : filterByRange(tree, range); + if (root === undefined) { + throw new AstCommandError( + `No node touches --range ${args.range ?? ""}: the tree of ${args.file} spans lines ${String(tree.start.line)} to ${String(lastLine(tree))}.`, + "INVALID_INPUT" + ); + } + + if (args.json) { + console.log( + JSON.stringify( + astOutputSchema.parse({ + ok: true, + file: args.file, + language, + ...(range === undefined ? {} : { range }), + errorNodes, + root, + }) + ) + ); + return; + } + + console.log( + `${args.file} as ${language}. Named nodes only, so every kind below is valid in \`kind:\`.` + ); + console.log( + "Positions are line:column, with 1-based lines and 0-based byte columns.\n" + ); + for (const line of renderAst(root)) console.log(line); + if (errorNodes > 0) { + console.log( + `\n${String(errorNodes)} ERROR node(s): the file does not parse cleanly as ${language}. If that is the wrong parser, pass --lang.` + ); + } + } catch (error) { + if (error instanceof AstCommandError || error instanceof AstDumpError) { + if (args.json) { + writeJsonError(error.code, error.message); + } else { + console.error(`Error: ${error.message}`); + } + process.exitCode = 1; + return; + } + throw error; + } + }, +}); diff --git a/packages/cli/src/commands/names.ts b/packages/cli/src/commands/names.ts index 142fbe94..4ab45be8 100644 --- a/packages/cli/src/commands/names.ts +++ b/packages/cli/src/commands/names.ts @@ -11,6 +11,7 @@ */ export const SUBCOMMAND_NAMES = [ "agent", + "ast", "auth", "check", "detect", diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index db5b6c0e..79da05ed 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -6,6 +6,7 @@ import { defineCommand, runCommand, showUsage } from "citty"; import { z } from "zod"; import { createAgentCommand } from "./commands/agent"; +import { astCommand } from "./commands/ast"; import { authCommand } from "./commands/auth"; import { checkCommand } from "./commands/check"; import { detectCommand } from "./commands/detect"; @@ -56,6 +57,7 @@ const subCommands = { update: updateCommand, info: infoCommand, detect: detectCommand, + ast: astCommand, check: checkCommand, feedback: feedbackCommand, auth: authCommand, diff --git a/packages/cli/src/rules/ast-dump.ts b/packages/cli/src/rules/ast-dump.ts new file mode 100644 index 00000000..a339bd5f --- /dev/null +++ b/packages/cli/src/rules/ast-dump.ts @@ -0,0 +1,379 @@ +import { spawn } from "node:child_process"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import process from "node:process"; +import { StringDecoder } from "node:string_decoder"; + +import { resolveAstGrepLanguage, type AstGrepLanguage } from "./capabilities"; +import { buildPath, stripSgDeprecationBanner } from "./scan"; + +/** + * Dumping a file's named-node AST with the pinned ast-grep, for `taskless ast`. + * + * ast-grep has no command that prints a file's tree. The closest it has is + * `run --debug-query=ast`, which prints the tree of the PATTERN, so the file's + * contents are handed over as the pattern. Measured at 0.51.1, that is exact + * rather than approximate: + * + * - The dump is of the text as given. Positions are not shifted by leading + * blank lines or indentation, and a metavariable-shaped `$FOO` in PHP still + * prints as `variable_name`, so pattern preprocessing does not reach it. + * - A file is almost never a valid single-node pattern, so ast-grep goes on to + * reject it ("Multiple AST nodes are detected") and exits 8. The dump is + * printed to stderr before that, so the exit code is not the signal: the + * `Debug AST:` block is. + * - `ast` prints named nodes only. Every kind it shows is one `kind:` accepts, + * which is the point: the `cst` dump prints keywords the same way, and + * `kind: readonly` is a config error. + * + * The cost of going through argv is size. Linux caps one argument at 128 KiB + * (`E2BIG`) and Windows a whole command line at 32 Ki characters, so a file + * past that cannot be dumped this way, and {@link dumpAst} says so rather + * than failing with a bare errno. + * + * The dump is ast-grep's own text format, parsed here. That is not the + * "parse generated output" the code style guide forbids: the generator is a + * vendored binary with no structured alternative, and the format is pinned by + * `test/ast-command.test.ts` against the binary across languages. + */ + +/** A position in the file: a 1-based line and a 0-based UTF-8 byte column. */ +export interface AstPosition { + line: number; + column: number; +} + +/** One named node of the tree. */ +export interface AstNode { + kind: string; + /** The field this node fills in its parent, when the grammar names one. */ + field?: string; + start: AstPosition; + end: AstPosition; + /** The node's source text, for a node with no named children. */ + text?: string; + children: AstNode[]; +} + +/** An inclusive, 1-based line range. */ +export interface LineRange { + from: number; + to: number; +} + +export class AstDumpError extends Error { + override name = "AstDumpError"; + constructor( + message: string, + readonly code: "INVALID_INPUT" | "SCAN_FAILED" + ) { + super(message); + } +} + +/** + * Parse `--range`: `` or `:`, 1-based and inclusive. + * Returns undefined for anything else. + */ +export function parseLineRange(value: string): LineRange | undefined { + const match = /^(\d+)(?::(\d+))?$/.exec(value.trim()); + if (match === null) return undefined; + const from = Number(match[1]); + const to = match[2] === undefined ? from : Number(match[2]); + if (from < 1 || to < from) return undefined; + return { from, to }; +} + +interface SgRun { + code: number | null; + stderr: string; +} + +/** Run ast-grep with `argv`, collecting stderr. Rejects when it cannot start. */ +function runSg(binary: string, argv: string[], cwd: string): Promise { + return new Promise((resolve, reject) => { + let child; + try { + child = spawn(binary, argv, { + cwd, + stdio: ["ignore", "ignore", "pipe"], + env: { ...process.env, PATH: buildPath() }, + }); + } catch (error) { + // `E2BIG` and an argument holding a NUL are thrown synchronously, before + // any process exists. + reject(error as Error); + return; + } + const decoder = new StringDecoder("utf8"); + const chunks: string[] = []; + child.stderr.on("data", (chunk: Buffer) => { + chunks.push(decoder.write(chunk)); + }); + child.on("error", reject); + child.on("close", (code) => { + chunks.push(decoder.end()); + resolve({ code, stderr: chunks.join("") }); + }); + }); +} + +/** + * The language ast-grep reads `file` as, asked of the binary rather than of a + * table of extensions. + * + * A `scan` with `--inspect entity` names the language it settled on for every + * file it reached, which is the same resolution `check` gets, since `check` + * also leaves extension mapping to ast-grep. The probe rule is Rust and only + * exists because `scan` needs one; the language is reported whether or not any + * rule applies. It runs against an empty config of its own, because ast-grep + * otherwise picks up an `sgconfig.yml` above `cwd`, and its `languageGlobs` + * would be a mapping `check` does not use. + * + * Undefined when ast-grep does not recognize the extension: such a file is not + * reported at all. Ignore rules are not a second cause: ast-grep applies + * `.gitignore`, hidden-file and similar filters only to what it finds while + * walking a directory, never to a path it is given, and the probe gives it one. + */ +export async function detectAstLanguage( + binary: string, + cwd: string, + file: string +): Promise { + const configDirectory = await mkdtemp(join(tmpdir(), "tskl-ast-")); + try { + const config = join(configDirectory, "sgconfig.yml"); + await writeFile(config, "ruleDirs: []\n", "utf8"); + const { stderr } = await runSg( + binary, + [ + "scan", + "--config", + config, + "--inline-rules", + "id: taskless-ast-probe\nlanguage: Rust\nrule: { kind: source_file }\n", + "--inspect", + "entity", + "--", + file, + ], + cwd + ); + return parseInspectedLanguage(stderr); + } finally { + await rm(configDirectory, { recursive: true, force: true }); + } +} + +/** The `language=` of the first file entity in `--inspect entity` stderr. */ +export function parseInspectedLanguage( + stderr: string +): AstGrepLanguage | undefined { + for (const line of stderr.split("\n")) { + if (!line.startsWith("sg: entity|file|")) continue; + const language = /: language=([^,\s]+),/.exec(line)?.[1]; + if (language !== undefined) return resolveAstGrepLanguage(language); + } + return undefined; +} + +const DEBUG_HEADER = "Debug AST:"; +const NODE_LINE = + /^((?: {2})*)(?:([A-Za-z_]\w*): )?(\S+) \((\d+),(\d+)\)-\((\d+),(\d+)\)$/; + +/** + * Parse the block `--debug-query=ast` prints into a tree. + * + * Lines are `: (,)-(,)`, two spaces + * of indent per depth, rows 0-based, ending at the first blank line. Rows are + * shifted to 1-based lines here so they read like an editor's. + * + * Throws when the block is missing or a line does not have that shape, since + * either means the binary is not the one this parser was measured against. + */ +export function parseDebugAst(stderr: string): AstNode { + const lines = stderr.split(/\r?\n/); + const header = lines.indexOf(DEBUG_HEADER); + if (header === -1) { + throw new AstDumpError( + `ast-grep printed no AST${stderr.trim() === "" ? "" : `: ${stderr.trim()}`}`, + "SCAN_FAILED" + ); + } + + const stack: AstNode[] = []; + let root: AstNode | undefined; + for (const line of lines.slice(header + 1)) { + if (line.trim() === "") break; + const match = NODE_LINE.exec(line); + if (match === null) { + throw new AstDumpError( + `Unexpected line in ast-grep's AST dump: ${JSON.stringify(line)}`, + "SCAN_FAILED" + ); + } + const [, indent = "", field, kind = "", sr, sc, er, ec] = match; + const depth = indent.length / 2; + const node: AstNode = { + kind, + ...(field === undefined ? {} : { field }), + start: { line: Number(sr) + 1, column: Number(sc) }, + end: { line: Number(er) + 1, column: Number(ec) }, + children: [], + }; + if (depth === 0) { + if (root !== undefined) { + throw new AstDumpError( + "ast-grep's AST dump has more than one root", + "SCAN_FAILED" + ); + } + root = node; + } else { + const parent = stack[depth - 1]; + if (parent === undefined) { + throw new AstDumpError( + `ast-grep's AST dump skips a level at ${JSON.stringify(line)}`, + "SCAN_FAILED" + ); + } + parent.children.push(node); + } + stack.length = depth; + stack.push(node); + } + if (root === undefined) { + throw new AstDumpError("ast-grep's AST dump is empty", "SCAN_FAILED"); + } + return root; +} + +/** Fill `text` on every node with no named children, from the file's bytes. */ +export function attachLeafText(node: AstNode, source: string): void { + const bytes = Buffer.from(source, "utf8"); + const lineStarts = [0]; + for (const [index, byte] of bytes.entries()) { + if (byte === 0x0a) lineStarts.push(index + 1); + } + const offset = (position: AstPosition): number => + (lineStarts[position.line - 1] ?? bytes.length) + position.column; + + const visit = (current: AstNode): void => { + if (current.children.length === 0) { + current.text = bytes + .subarray(offset(current.start), offset(current.end)) + .toString("utf8"); + return; + } + for (const child of current.children) visit(child); + }; + visit(node); +} + +/** + * The last line a node occupies. A node that ends at column 0 of a later line + * ends with that line's newline, so it does not occupy the line itself. + */ +export function lastLine(node: AstNode): number { + return node.end.column === 0 && node.end.line > node.start.line + ? node.end.line - 1 + : node.end.line; +} + +/** + * The tree cut down to the nodes that touch `range`: every node whose lines + * overlap it, so the enclosing nodes stay as context and everything on those + * lines is shown. Undefined when the root itself does not touch it. + */ +export function filterByRange( + node: AstNode, + range: LineRange +): AstNode | undefined { + if (node.start.line > range.to || lastLine(node) < range.from) { + return undefined; + } + return { + ...node, + children: node.children + .map((child) => filterByRange(child, range)) + .filter((child): child is AstNode => child !== undefined), + }; +} + +/** How many `ERROR` nodes the tree holds: a file that did not parse cleanly. */ +export function countErrorNodes(node: AstNode): number { + return node.children.reduce( + (total, child) => total + countErrorNodes(child), + node.kind === "ERROR" ? 1 : 0 + ); +} + +const TEXT_PREVIEW_LENGTH = 60; + +/** The tree as indented lines, one node per line. */ +export function renderAst(node: AstNode, depth = 0): string[] { + const position = `${String(node.start.line)}:${String(node.start.column)}-${String(node.end.line)}:${String(node.end.column)}`; + const field = node.field === undefined ? "" : `${node.field}: `; + let text = ""; + if (node.text !== undefined) { + const preview = + node.text.length > TEXT_PREVIEW_LENGTH + ? `${node.text.slice(0, TEXT_PREVIEW_LENGTH)}…` + : node.text; + text = ` ${JSON.stringify(preview)}`; + } + return [ + `${" ".repeat(depth)}${field}${node.kind} [${position}]${text}`, + ...node.children.flatMap((child) => renderAst(child, depth + 1)), + ]; +} + +/** + * Dump `source`'s named-node tree as `language` parses it. + * + * `cwd` only sets where ast-grep runs; the source is handed over whole. + */ +export async function dumpAst(options: { + binary: string; + cwd: string; + source: string; + language: AstGrepLanguage; +}): Promise { + const { binary, cwd, source, language } = options; + let result: SgRun; + try { + result = await runSg( + binary, + [ + "run", + "--debug-query=ast", + "--lang", + language, + // `=` joins the value so a file starting with `-` (a YAML list) is not + // read as a flag. + `--pattern=${source}`, + // Nothing to search: without this, a file that happens to be one valid + // pattern would go on to scan the whole of `cwd` for matches. + "--stdin", + ], + cwd + ); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code === "E2BIG" || code === "ENAMETOOLONG") { + throw new AstDumpError( + `The file is too large to dump (${String(Buffer.byteLength(source))} bytes): ast-grep receives it as a command-line argument, which this platform limits. Dump a smaller file, or copy the part you need into one.`, + "INVALID_INPUT" + ); + } + if (code === "ERR_INVALID_ARG_VALUE") { + throw new AstDumpError( + "The file contains a NUL byte, so it is not source text ast-grep can parse.", + "INVALID_INPUT" + ); + } + throw error; + } + return parseDebugAst(stripSgDeprecationBanner(result.stderr)); +} diff --git a/packages/cli/src/rules/verify.ts b/packages/cli/src/rules/verify.ts index c4d74928..64c7b491 100644 --- a/packages/cli/src/rules/verify.ts +++ b/packages/cli/src/rules/verify.ts @@ -8,7 +8,11 @@ import { REGEX_NEEDS_KIND, TASKLESS_REQUIRED_FIELDS, } from "../schemas/ast-grep-rule"; -import { pathPrefixed, schemaLayer } from "../schemas/layer"; +import { + pathPrefixed, + schemaLayer, + type IssueFormatter, +} from "../schemas/layer"; import { casePassed, describeCaseFailures, @@ -34,6 +38,17 @@ import { violate, type RuleViolation } from "./constraints"; import { isValidRuleId } from "./validate-id"; import { parseSgRuleAlone } from "./sg-parse"; import { findSgTraps } from "./sg-traps"; +import { getCliPrefix } from "../util/package-manager"; + +/** + * The `sg-regex-needs-kind` message, with where to find a kind to add. + * + * Appended here rather than in `findRegexWithoutKind`, which lives with the + * rule schema and stays free of the CLI's own invocation. + */ +function withKindPointer(message: string): string { + return `${message} Run \`${getCliPrefix()} ast \` on a file in the rule's language to see the kinds it has.`; +} // --- Types --- @@ -128,19 +143,24 @@ function validateSchema(ruleData: unknown): LayerResult { // ast-grep's messages come from an upstream JSON Schema and are generic // ("expected string"), so the path is what makes them actionable. const result = astGrepRuleSchema.safeParse(ruleData); - const layer = schemaLayer(result, pathPrefixed); // The regex/kind pairing lives in the schema (see `astGrepRuleJsonSchema`), // and its issues say which constraint they are, so they are attributed - // here rather than re-derived from their wording. + // here rather than re-derived from their wording. Their message carries + // the `ast` pointer in `errors` and `violations` alike, so a violation's + // message is always one of the layer's errors. + const isRegexNeedsKind = (issue: Parameters[0]): boolean => + issue.code === "custom" && issue.params?.constraintId === REGEX_NEEDS_KIND; + const format: IssueFormatter = (issue) => + isRegexNeedsKind(issue) + ? withKindPointer(pathPrefixed(issue)) + : pathPrefixed(issue); + const layer = schemaLayer(result, format); const violations: RuleViolation[] = []; for (const issue of result.error?.issues ?? []) { - if ( - issue.code === "custom" && - issue.params?.constraintId === REGEX_NEEDS_KIND - ) { + if (isRegexNeedsKind(issue)) { violations.push({ constraintId: REGEX_NEEDS_KIND, - message: pathPrefixed(issue), + message: format(issue), }); } } diff --git a/packages/cli/src/schemas/ast.ts b/packages/cli/src/schemas/ast.ts new file mode 100644 index 00000000..aef7e421 --- /dev/null +++ b/packages/cli/src/schemas/ast.ts @@ -0,0 +1,56 @@ +import { z } from "zod"; + +import { AST_GREP_LANGUAGES } from "../rules/capabilities"; + +const positionSchema = z.object({ + line: z.number().int().positive().describe("1-based line"), + column: z.number().int().nonnegative().describe("0-based UTF-8 byte column"), +}); + +interface AstNodeOutput { + kind: string; + field?: string; + start: z.infer; + end: z.infer; + text?: string; + children: AstNodeOutput[]; +} + +/** One named node. Every `kind` here is one a rule's `kind:` accepts. */ +const astNodeSchema: z.ZodType = z.object({ + kind: z.string().describe("The node kind, usable as `kind:` in a rule"), + field: z + .string() + .optional() + .describe("The field this node fills in its parent, when named"), + start: positionSchema, + end: positionSchema.describe("Exclusive end"), + text: z + .string() + .optional() + .describe("Source text, present on nodes with no named children"), + get children() { + return z.array(astNodeSchema); + }, +}); + +/** `taskless ast --json` */ +export const outputSchema = z.object({ + ok: z.literal(true), + file: z.string().describe("The file, as given"), + language: z + .enum(AST_GREP_LANGUAGES) + .describe("The ast-grep language the file was parsed as"), + range: z + .object({ from: z.number().int(), to: z.number().int() }) + .optional() + .describe("The --range the tree was cut to, 1-based and inclusive"), + errorNodes: z + .number() + .int() + .nonnegative() + .describe( + "ERROR nodes in the tree; above zero, the file did not parse cleanly as this language" + ), + root: astNodeSchema, +}); diff --git a/packages/cli/test/agent-extensions.test.ts b/packages/cli/test/agent-extensions.test.ts index 21b6cf4b..bceaa350 100644 --- a/packages/cli/test/agent-extensions.test.ts +++ b/packages/cli/test/agent-extensions.test.ts @@ -539,7 +539,9 @@ describe("create-sg-rule names the tree-sitter traps under Quirks", () => { const result = await runCli(["agent", "create-sg-rule", "-d", cwd]); expect(result.exitCode).toBe(0); expect(result.stdout).toContain("### A keyword is not a valid `kind`"); - expect(result.stdout).toContain("read kinds off `--debug-query=ast`"); + expect(result.stdout).toMatch(/read kinds off `.+ ast `/); + // An agent told to use `--debug-query` goes hunting for a binary (#673). + expect(result.stdout).not.toContain("--debug-query"); }); it("says an awaited generic call hides its callee", async () => { diff --git a/packages/cli/test/ast-command.test.ts b/packages/cli/test/ast-command.test.ts new file mode 100644 index 00000000..0038e468 --- /dev/null +++ b/packages/cli/test/ast-command.test.ts @@ -0,0 +1,388 @@ +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { + attachLeafText, + countErrorNodes, + detectAstLanguage, + dumpAst, + filterByRange, + parseDebugAst, + parseInspectedLanguage, + parseLineRange, + renderAst, + type AstNode, +} from "../src/rules/ast-dump"; +import { AST_GREP_LANGUAGES } from "../src/rules/capabilities"; +import { AST_GREP_LANGUAGE_EXTENSIONS } from "../src/rules/fixture-convert"; +import { findSgBinary } from "../src/rules/scan"; +import { builtCli } from "./support/built-cli"; +import { cliRejectionToResult } from "./support/spawn-cli"; + +const execFileAsync = promisify(execFile); +const binPath = builtCli(); + +async function runCli( + args: string[], + cwd: string +): Promise<{ stdout: string; stderr: string; exitCode: number }> { + try { + const { stdout, stderr } = await execFileAsync("node", [binPath, ...args], { + cwd, + maxBuffer: 64 * 1024 * 1024, + env: { + ...process.env, + DO_NOT_TRACK: "1", + TASKLESS_TELEMETRY_DISABLED: "1", + }, + }); + return { stdout, stderr, exitCode: 0 }; + } catch (error) { + return cliRejectionToResult(error, [binPath, ...args]); + } +} + +/** Kinds in document order, indented by depth, for compact assertions. */ +function outline(node: AstNode, depth = 0): string[] { + const field = node.field === undefined ? "" : `${node.field}: `; + return [ + `${" ".repeat(depth)}${field}${node.kind}`, + ...node.children.flatMap((child) => outline(child, depth + 1)), + ]; +} + +const binary = ((): string | undefined => { + try { + return findSgBinary(); + } catch { + return undefined; + } +})(); +const withSg = binary === undefined ? describe.skip : describe; + +let cwd: string; +beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "taskless-ast-")); +}); +afterEach(async () => { + await rm(cwd, { recursive: true, force: true }); +}); + +describe("parseDebugAst", () => { + // Verbatim from `ast-grep run --debug-query=ast` at 0.51.1. + const DUMP = [ + "Debug AST:", + "program (0,0)-(1,0)", + " lexical_declaration (0,0)-(0,12)", + " variable_declarator (0,6)-(0,11)", + " name: identifier (0,6)-(0,7)", + " value: number (0,10)-(0,11)", + "", + "Error: Cannot parse query as a valid pattern.", + ].join("\n"); + + it("builds the tree with fields and 1-based lines", () => { + const tree = parseDebugAst(DUMP); + expect(outline(tree)).toEqual([ + "program", + " lexical_declaration", + " variable_declarator", + " name: identifier", + " value: number", + ]); + expect(tree.start).toEqual({ line: 1, column: 0 }); + expect(tree.children[0]?.children[0]?.children[1]?.end).toEqual({ + line: 1, + column: 11, + }); + }); + + it("fails on a dump it does not recognize rather than guessing", () => { + expect(() => parseDebugAst("Error: something else")).toThrow( + /printed no AST/ + ); + expect(() => + parseDebugAst("Debug AST:\nprogram (0,0)-(0,1)\n ??? what\n") + ).toThrow(/Unexpected line/); + }); + + it("attaches source text to leaves, by byte column", () => { + const source = 'const é = "ü";\n'; + const tree = parseDebugAst( + [ + "Debug AST:", + "program (0,0)-(1,0)", + " lexical_declaration (0,0)-(0,16)", + " variable_declarator (0,6)-(0,15)", + " name: identifier (0,6)-(0,8)", + " value: string (0,11)-(0,15)", + " string_fragment (0,12)-(0,14)", + ].join("\n") + ); + attachLeafText(tree, source); + const declarator = tree.children[0]?.children[0]; + expect(declarator?.children[0]?.text).toBe("é"); + expect(declarator?.children[1]?.text).toBeUndefined(); + expect(declarator?.children[1]?.children[0]?.text).toBe("ü"); + expect(renderAst(tree)).toContain(' name: identifier [1:6-1:8] "é"'); + }); +}); + +describe("parseLineRange", () => { + it("accepts a line or an inclusive span", () => { + expect(parseLineRange("7")).toEqual({ from: 7, to: 7 }); + expect(parseLineRange("3:9")).toEqual({ from: 3, to: 9 }); + }); + + it("rejects anything else", () => { + for (const value of ["", "0", "9:3", "a", "3-9", "3:", ":3", "-1"]) { + expect(parseLineRange(value)).toBeUndefined(); + } + }); +}); + +/** A node from line/column pairs, for range tests. */ +function node( + kind: string, + from: [number, number], + to: [number, number], + children: AstNode[] = [] +): AstNode { + return { + kind, + start: { line: from[0], column: from[1] }, + end: { line: to[0], column: to[1] }, + children, + }; +} + +describe("filterByRange", () => { + const tree = node( + "program", + [1, 0], + [4, 0], + [ + node("a", [1, 0], [2, 0]), + node("b", [2, 0], [2, 5]), + node("c", [3, 0], [3, 4]), + ] + ); + + it("keeps the nodes on the lines and the ones enclosing them", () => { + expect(outline(filterByRange(tree, { from: 2, to: 2 })!)).toEqual([ + "program", + " b", + ]); + }); + + it("does not count a node ending at column 0 as on that line", () => { + expect( + filterByRange(tree, { from: 2, to: 3 })?.children.map((c) => c.kind) + ).toEqual(["b", "c"]); + }); + + it("is undefined past the end of the tree", () => { + expect(filterByRange(tree, { from: 4, to: 9 })).toBeUndefined(); + }); +}); + +describe("parseInspectedLanguage", () => { + it("reads the language of the file entity", () => { + expect( + parseInspectedLanguage( + "sg: summary|project: isProject=false\nsg: entity|file|a.ts: language=TypeScript,appliedRuleCount=0\n" + ) + ).toBe("TypeScript"); + }); + + it("is undefined for a skipped or unreported file", () => { + expect( + parseInspectedLanguage("sg: entity|file|x.dart: skipped=emptyFile\n") + ).toBeUndefined(); + expect(parseInspectedLanguage("")).toBeUndefined(); + }); +}); + +withSg("against the bundled ast-grep", () => { + it.each(AST_GREP_LANGUAGES)( + "dumps a %s file in the format the parser expects", + async (language) => { + const tree = await dumpAst({ + binary: binary!, + cwd, + source: "x\ny\n", + language, + }); + expect(tree.start.line).toBeGreaterThanOrEqual(1); + expect(renderAst(tree).length).toBeGreaterThan(0); + } + ); + + it("resolves each language's own extension to that language", async () => { + for (const language of AST_GREP_LANGUAGES) { + const file = `sample.${AST_GREP_LANGUAGE_EXTENSIONS[language]}`; + await writeFile(join(cwd, file), "x\n"); + expect(await detectAstLanguage(binary!, cwd, file)).toBe(language); + } + }); + + it("resolves extensions beyond the one per language, as check would", async () => { + await writeFile(join(cwd, "a.mjs"), "x\n"); + await writeFile(join(cwd, "a.yaml"), "x: 1\n"); + expect(await detectAstLanguage(binary!, cwd, "a.mjs")).toBe("JavaScript"); + expect(await detectAstLanguage(binary!, cwd, "a.yaml")).toBe("Yaml"); + }); + + it("resolves a file that ignore rules would skip in a directory walk", async () => { + // The probe names the file, and ast-grep filters only what it walks to, so + // a gitignored, hidden or node_modules path still reports its language. + await mkdir(join(cwd, ".git")); + await mkdir(join(cwd, "ignored")); + await mkdir(join(cwd, "node_modules", "pkg"), { recursive: true }); + await writeFile(join(cwd, ".gitignore"), "ignored/\n"); + for (const file of [ + "ignored/a.ts", + ".hidden.ts", + "node_modules/pkg/a.ts", + ]) { + await writeFile(join(cwd, file), "x\n"); + expect(await detectAstLanguage(binary!, cwd, file)).toBe("TypeScript"); + } + }); + + it("ignores an sgconfig.yml's languageGlobs, which check does not use", async () => { + await writeFile( + join(cwd, "sgconfig.yml"), + 'ruleDirs: []\nlanguageGlobs:\n TypeScript: ["*.weird"]\n' + ); + await writeFile(join(cwd, "a.weird"), "x\n"); + expect(await detectAstLanguage(binary!, cwd, "a.weird")).toBeUndefined(); + }); + + it("dumps the text as written, not as a preprocessed pattern", async () => { + // `$FOO` is metavariable-shaped, and PHP is a language whose patterns + // rewrite `$`. The dump must still show the variable the file holds. + const tree = await dumpAst({ + binary: binary!, + cwd, + source: " { + const tree = await dumpAst({ + binary: binary!, + cwd, + source: "const = ;\n", + language: "JavaScript", + }); + expect(countErrorNodes(tree)).toBeGreaterThan(0); + }); +}); + +describe("taskless ast", () => { + it("prints the named-node tree of a file, by its extension", async () => { + await writeFile(join(cwd, "a.ts"), "const a = 1;\nfoo(a);\n"); + const result = await runCli(["ast", "a.ts"], cwd); + expect(result.exitCode).toBe(0); + expect(result.stdout).toContain("a.ts as TypeScript"); + expect(result.stdout).toContain(" lexical_declaration [1:0-1:12]"); + expect(result.stdout).toContain(' name: identifier [1:6-1:7] "a"'); + expect(result.stdout).toContain("call_expression"); + // Named nodes only: the `const` keyword is not a kind. + expect(result.stdout).not.toMatch(/^\s*const\b/m); + }); + + it("emits the tree as JSON", async () => { + await writeFile(join(cwd, "a.py"), "def f():\n return 1\n"); + const result = await runCli(["ast", "a.py", "--json", "-d", cwd], cwd); + expect(result.exitCode).toBe(0); + const output = JSON.parse(result.stdout) as { + ok: boolean; + language: string; + errorNodes: number; + root: AstNode; + }; + expect(output.ok).toBe(true); + expect(output.language).toBe("Python"); + expect(output.errorNodes).toBe(0); + expect(outline(output.root)).toContain(" function_definition"); + }); + + it("cuts the tree to --range", async () => { + await writeFile(join(cwd, "a.js"), "const a = 1;\nconst b = 2;\nfoo(b);\n"); + const result = await runCli(["ast", "a.js", "--range", "3", "--json"], cwd); + expect(result.exitCode).toBe(0); + const output = JSON.parse(result.stdout) as { + range: unknown; + root: AstNode; + }; + expect(output.range).toEqual({ from: 3, to: 3 }); + expect(output.root.children.map((child) => child.kind)).toEqual([ + "expression_statement", + ]); + }); + + it("parses as --lang when given, and dumps a file starting with -", async () => { + await writeFile(join(cwd, "list.txt"), "- a\n- b\n"); + const result = await runCli( + ["ast", "list.txt", "--lang", "yaml", "--json"], + cwd + ); + expect(result.exitCode).toBe(0); + const output = JSON.parse(result.stdout) as { + language: string; + root: AstNode; + }; + expect(output.language).toBe("Yaml"); + expect(outline(output.root).join("\n")).toContain("block_sequence_item"); + }); + + it.each([ + [["ast", "missing.ts"], /No such file/], + [["ast", "notes.unknownext"], /does not recognize the extension/], + [["ast", "a.ts", "--lang", "C#"], /has no language "C#"/], + [["ast", "a.ts", "--range", "5:2"], /--range takes/], + [["ast", "a.ts", "--range", "40"], /No node touches --range 40/], + [["ast", "empty.ts"], /is empty/], + [["ast", "latin1.js"], /is not valid UTF-8/], + ])("refuses %j with INVALID_INPUT", async (args, message) => { + await writeFile(join(cwd, "a.ts"), "let a = 1;\n"); + await writeFile(join(cwd, "notes.unknownext"), "hello\n"); + await writeFile(join(cwd, "empty.ts"), ""); + // "// café" in Latin-1: 0xE9 alone is not a UTF-8 sequence. + await writeFile( + join(cwd, "latin1.js"), + Buffer.from([...Buffer.from("// caf"), 0xe9, 0x0a]) + ); + const result = await runCli([...args, "--json"], cwd); + expect(result.exitCode).toBe(1); + const envelope = JSON.parse(result.stdout) as { + ok: boolean; + code: string; + message: string; + }; + expect(envelope.ok).toBe(false); + expect(envelope.code).toBe("INVALID_INPUT"); + expect(envelope.message).toMatch(message); + }); + + it("says a file too large for the command line is too large", async () => { + await writeFile(join(cwd, "big.js"), "a;\n".repeat(500_000)); + const result = await runCli(["ast", "big.js", "--json"], cwd); + expect(result.exitCode).toBe(1); + const envelope = JSON.parse(result.stdout) as { + code: string; + message: string; + }; + expect(envelope.code).toBe("INVALID_INPUT"); + expect(envelope.message).toMatch(/too large to dump/); + }); +}); diff --git a/packages/cli/test/verify.test.ts b/packages/cli/test/verify.test.ts index 76b61912..0bdf9427 100644 --- a/packages/cli/test/verify.test.ts +++ b/packages/cli/test/verify.test.ts @@ -291,8 +291,12 @@ describe("verifyRule", () => { expect(result.schema.errors).toContainEqual( expect.stringMatching(/^rule: Every "regex", at any depth/) ); - expect(result.schema.violations).toContainEqual( - expect.objectContaining({ constraintId: "sg-regex-needs-kind" }) + const regexViolation = result.schema.violations?.find( + (violation) => violation.constraintId === "sg-regex-needs-kind" + ); + // Points at `taskless ast` for the kind to add. + expect(regexViolation?.message).toMatch( + / ast ` on a file in the rule's language/ ); });