Python: Add extractor support for Python 3.15 - #22728
Merged
Merged
Conversation
Constructs such as
```python
match x:
case +5:
pass
```
are now allowed in Python 3.15 (just as `case -5` would have
previously). To support this new syntax, we need to extend the
tree-sitter based parser slightly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Supplement the generator's Unicode 16 identifier properties with the exact Unicode 17 additions, independently of the extraction host's Python version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The synthesized unary-plus nodes omit the operator from their source locations because the location override uses an unrecognized attribute name.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Adds Python 3.15 extraction support for unary-plus match patterns and Unicode 17 identifiers.
Changes:
- Extends tree-sitter grammar and lowering.
- Adds parser and extraction coverage.
- Updates extractor version and documentation.
| File | Description |
|---|---|
python/ql/test/3/extractor-tests/unicode17-identifiers/test.ql |
Queries extracted identifiers. |
python/ql/test/3/extractor-tests/unicode17-identifiers/test.py |
Adds Unicode 17 examples. |
python/ql/test/3/extractor-tests/unicode17-identifiers/test.expected |
Records query results. |
python/ql/test/3/extractor-tests/match-unary-plus/test.ql |
Queries unary expressions. |
python/ql/test/3/extractor-tests/match-unary-plus/test.py |
Exercises unary-plus patterns. |
python/ql/test/3/extractor-tests/match-unary-plus/test.expected |
Records unary-expression results. |
python/ql/lib/change-notes/2026-09-22-python315-extractor.md |
Documents Python 3.15 support. |
python/extractor/tsg-python/tsp/src/node-types.json |
Updates generated node metadata. |
python/extractor/tsg-python/tsp/src/grammar.json |
Updates generated grammar metadata. |
python/extractor/tsg-python/tsp/grammar.js |
Adds grammar support for both features. |
python/extractor/tsg-python/python.tsg |
Lowers unary plus to UAdd. |
python/extractor/tests/parser/unicode_identifiers_new.py |
Adds parser fixture. |
python/extractor/tests/parser/unicode_identifiers_new.expected |
Records Unicode parser AST. |
python/extractor/tests/parser/match_new.py |
Expands match parser coverage. |
python/extractor/tests/parser/match_new.expected |
Records match parser AST. |
python/extractor/semmle/util.py |
Bumps extractor version to 7.2.0. |
python/extractor/extending-the-parser.md |
Updates regeneration guidance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Somehow, in the original implementation of support for `match`, we had used `_start_location` instead of `_location_start`, which meant we didn't override the location after all. This was the only occurrence of this mistake in `python.tsg`.
Contributor
Author
That's fixed in fa4ce46. |
Contributor
|
Thanks. I also saw it as fixed so I was slightly confused about the comment. Thanks for resolving. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In addition to the previous work, two more minor changes were needed:
matchpatterns can now start with unary plus (e.g.case +5: ...).Both of these are fixed in the current PR. I suggest reviewing commit-by-commit (and skipping the second and fourth commits, as they only regenerate parser files).