Skip to content

hrw4u: record spans, comments and operand forms in the AST - #13665

Open
masaori335 wants to merge 2 commits into
apache:masterfrom
masaori335:asf-master-hrw4u-ast-span
Open

hrw4u: record spans, comments and operand forms in the AST#13665
masaori335 wants to merge 2 commits into
apache:masterfrom
masaori335:asf-master-hrw4u-ast-span

Conversation

@masaori335

Copy link
Copy Markdown
Contributor

ast_visitor.py is test-only today, and it cannot reproduce the file it parsed. Four things are dropped, each of which changes either the emitted .conf or the location a diagnostic can name.

A span names a file, line and column rather than a line alone. Procedure inlining splices nodes in from a second file, so a line number can no longer say where a diagnostic belongs. Node.line is replaced outright rather than kept as a property over span: nothing outside the tests reads it, and keeping it would let new code go on producing locations that cannot name their own file.

Comments, because five .conf goldens carry # comments at exact positions.

Parentheses, because a parenthesized factor emits cond %{GROUP}.

Set and iprange operands, which until now both became a bare tuple: in [1.2.3.4] and in {1.2.3.4} produced byte-identical ASTs. They are not interchangeable -- visitor.py emits lhs (...) for a set and lhs {...} for an iprange, and only the set path is sandbox-checked. Each keeps the source text the emitter already uses, so SetValue.raw matches visitor.py's set_().getText()[1:-1] and IpRangeValue.raw its iprange().getText().

Target.from_dotted guessed a namespace/field split that no caller wanted; every table lookup matches the whole dotted name.

ast_visitor.py is test-only today, and it cannot reproduce the file it
parsed. Four things are dropped, each of which changes either the
emitted .conf or the location a diagnostic can name.

A span names a file, line and column rather than a line alone.
Procedure inlining splices nodes in from a second file, so a line
number can no longer say where a diagnostic belongs. Node.line is
replaced outright rather than kept as a property over span: nothing
outside the tests reads it, and keeping it would let new code go on
producing locations that cannot name their own file.

Comments, because five .conf goldens carry # comments at exact
positions.

Parentheses, because a parenthesized factor emits cond %{GROUP}.

Set and iprange operands, which until now both became a bare tuple:
`in [1.2.3.4]` and `in {1.2.3.4}` produced byte-identical ASTs. They
are not interchangeable -- visitor.py emits `lhs (...)` for a set and
`lhs {...}` for an iprange, and only the set path is sandbox-checked.
Each keeps the source text the emitter already uses, so SetValue.raw
matches visitor.py's set_().getText()[1:-1] and IpRangeValue.raw its
iprange().getText().

Target.from_dotted guessed a namespace/field split that no caller
wanted; every table lookup matches the whole dotted name.
@masaori335 masaori335 added this to the 11.0.0 milestone Sep 10, 2026
@masaori335 masaori335 self-assigned this Sep 10, 2026
Copilot AI lite review requested due to automatic review settings September 10, 2026 07:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

This PR updates the HRW4U AST and visitor to preserve source fidelity (file/line/column spans, comments, grouping parentheses) and to disambiguate operand forms that previously collapsed into identical structures.

Changes:

  • Replace Node.line with Span(file, line, column) and thread spans through all AST nodes via ASTVisitor.
  • Preserve comments as first-class AST nodes across program/section/block/vars bodies, keeping original ordering and indentation/column.
  • Introduce Group, SetValue, and IpRangeValue to retain parentheses and distinguish in [...] vs in {...} semantics; simplify assignments to store name rather than a split Target.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
tools/hrw4u/tests/test_ast_visitor.py Updates tests to assert spans, comment preservation, grouping behavior, and new operand/assignment representations; adds strict parsing in tests.
tools/hrw4u/tests/test_ast_nodes.py Replaces Target tests with Span immutability/hashability/equality tests.
tools/hrw4u/src/ast_visitor.py Emits spans on all nodes, preserves comments, wraps parenthesized conditions in Group, and distinguishes set vs iprange RHS forms.
tools/hrw4u/src/ast_nodes.py Adds Span, Comment, Group, SetValue, IpRangeValue; removes Target and replaces line with span; updates node/value unions accordingly.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tools/hrw4u/src/ast_nodes.py
Comment thread tools/hrw4u/src/ast_nodes.py
Comment thread tools/hrw4u/src/ast_visitor.py Outdated
Comment thread tools/hrw4u/tests/test_ast_nodes.py
Comment thread tools/hrw4u/tests/test_ast_nodes.py Outdated
The eight "Unhandled ... at line N" raises predate the span and cannot
name the file, which is the case procedure inlining creates. They go
through one _unhandled() helper now.

test_ast_nodes.py asserted AttributeError for assignment to an unknown
Span field, which proved nothing: FrozenInstanceError subclasses
AttributeError, so frozen intercepted the write and slots was never
exercised. Check the layout instead.

ValueExpr admits IpRangeValue but not SetValue because the grammar's
value rule admits iprange and not set_; say so where the alias is
defined.
Copilot AI review requested due to automatic review settings September 10, 2026 07:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread tools/hrw4u/src/ast_visitor.py
@masaori335

Copy link
Copy Markdown
Contributor Author

[approve ci autest 2of4]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants