Skip to content

Python: Add extractor support for Python 3.15 #22728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tausbn merged 6 commits into from
Oct 2, 2026
Merged

Python: Add extractor support for Python 3.15 #22728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
aecf7a9
Python: Support unary plus in match patterns
tausbn Sep 22, 2026
fc13096
Python: Regenerate parser for unary-plus patterns
tausbn Sep 22, 2026
c87180b
Python: Support Unicode 17 identifiers
tausbn Sep 22, 2026
b857c8e
Python: Regenerate parser for Unicode 17 identifiers
tausbn Sep 22, 2026
cfb0925
Python: Bump extractor version
tausbn Oct 1, 2026
fa4ce46
Python: Fix locations of `match` unary operations
tausbn Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions python/extractor/extending-the-parser.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,21 +43,28 @@ This grammar can be found in the `grammar.js` file in the `tsg-python/tsp` subdi
Note that whenever changes are made to `grammar.js`, you must regenerate the parser files by running

```sh
tree-sitter generate
tree-sitter generate --abi 14
```

inside the `tsp` directory.
You'll need to install the `tree-sitter` CLI in order to run this command.
One way to install it is to use `cargo`:

```sh
cargo install tree-sitter-cli
cargo install tree-sitter-cli --version 0.25.10 --locked
```

(This presupposes you have `cargo` available, but you'll need this anyway when compiling `tsg-python`.)

Once the parser files have been regenerated, they'll get picked up automatically when `tsg-python` is rebuilt.

The identifier rule supplements the generator's Unicode 16 properties with the
Unicode 17 `XID_Start` and `XID_Continue` additions from the Unicode Character
Database. Keep both sets of additions in sync when updating Unicode support.
Once the generator includes Unicode 17, these additions can be removed.
The `unicode_identifiers_new.py` parser fixture and `unicode17-identifiers` QL test
cover extraction independently of the host Python interpreter's Unicode version.

> Pro-tip: When you're done with your parser changes, and go to commit these to a branch, put the autogenerated files in their own commit.
> This makes it easier to review the changes, and if you need to go back and regenerate the files again, it's easy to modify just that commit.

Expand Down
2 changes: 1 addition & 1 deletion python/extractor/semmle/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

#Semantic version of extractor.
#Update this if any changes are made
VERSION = "7.1.11"
VERSION = "7.2.0"

PY_EXTENSIONS = ".py", ".pyw"

Expand Down
214 changes: 210 additions & 4 deletions python/extractor/tests/parser/match_new.expected
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Module: [1, 0] - [43, 0]
Module: [1, 0] - [61, 0]
body: [
Match: [1, 0] - [3, 19]
subject:
Expand Down Expand Up @@ -85,7 +85,7 @@ Module: [1, 0] - [43, 0]
literal:
BinOp: [8, 9] - [8, 18]
left:
UnaryOp: [8, 10] - [8, 11]
UnaryOp: [8, 9] - [8, 11]
op: USub
operand:
Num: [8, 10] - [8, 11]
Expand All @@ -104,7 +104,7 @@ Module: [1, 0] - [43, 0]
pattern:
MatchLiteralPattern: [10, 9] - [10, 11]
literal:
UnaryOp: [10, 10] - [10, 11]
UnaryOp: [10, 9] - [10, 11]
op: USub
operand:
Num: [10, 10] - [10, 11]
Expand All @@ -131,7 +131,7 @@ Module: [1, 0] - [43, 0]
literal:
BinOp: [14, 9] - [14, 16]
left:
UnaryOp: [14, 10] - [14, 13]
UnaryOp: [14, 9] - [14, 13]
op: USub
operand:
Num: [14, 10] - [14, 13]
Expand Down Expand Up @@ -379,4 +379,210 @@ Module: [1, 0] - [43, 0]
Pass: [42, 8] - [42, 12]
]
]
Match: [44, 0] - [60, 12]
subject:
Name: [44, 6] - [44, 11]
variable: Variable('value', None)
ctx: Load
cases: [
Case: [45, 4] - [46, 12]
pattern:
MatchLiteralPattern: [45, 9] - [45, 11]
literal:
UnaryOp: [45, 9] - [45, 11]
op: UAdd
operand:
Num: [45, 10] - [45, 11]
n: 1
text: '1'
guard: None
body: [
Pass: [46, 8] - [46, 12]
]
Case: [47, 4] - [48, 12]
pattern:
MatchLiteralPattern: [47, 9] - [47, 13]
literal:
UnaryOp: [47, 9] - [47, 13]
op: UAdd
operand:
Num: [47, 10] - [47, 13]
n: 1.5
text: '1.5'
guard: None
body: [
Pass: [48, 8] - [48, 12]
]
Case: [49, 4] - [50, 12]
pattern:
MatchLiteralPattern: [49, 9] - [49, 12]
literal:
UnaryOp: [49, 9] - [49, 12]
op: UAdd
operand:
Num: [49, 10] - [49, 12]
n: 2j
text: '2j'
guard: None
body: [
Pass: [50, 8] - [50, 12]
]
Case: [51, 4] - [52, 12]
pattern:
MatchLiteralPattern: [51, 9] - [51, 14]
literal:
BinOp: [51, 9] - [51, 14]
left:
UnaryOp: [51, 9] - [51, 11]
op: UAdd
operand:
Num: [51, 10] - [51, 11]
n: 1
text: '1'
op: Add
right:
Num: [51, 12] - [51, 14]
n: 2j
text: '2j'
guard: None
body: [
Pass: [52, 8] - [52, 12]
]
Case: [53, 4] - [54, 12]
pattern:
MatchLiteralPattern: [53, 9] - [53, 14]
literal:
BinOp: [53, 9] - [53, 14]
left:
UnaryOp: [53, 9] - [53, 11]
op: UAdd
operand:
Num: [53, 10] - [53, 11]
n: 1
text: '1'
op: Sub
right:
Num: [53, 12] - [53, 14]
n: 2j
text: '2j'
guard: None
body: [
Pass: [54, 8] - [54, 12]
]
Case: [55, 4] - [56, 12]
pattern:
MatchOrPattern: [55, 9] - [55, 30]
patterns: [
MatchLiteralPattern: [55, 9] - [55, 14]
literal:
UnaryOp: [55, 9] - [55, 14]
op: UAdd
operand:
Num: [55, 10] - [55, 14]
n: 16
text: '0x10'
MatchLiteralPattern: [55, 17] - [55, 22]
literal:
UnaryOp: [55, 17] - [55, 22]
op: UAdd
operand:
Num: [55, 18] - [55, 22]
n: 8
text: '0o10'
MatchLiteralPattern: [55, 25] - [55, 30]
literal:
UnaryOp: [55, 25] - [55, 30]
op: UAdd
operand:
Num: [55, 26] - [55, 30]
n: 2
text: '0b10'
]
guard: None
body: [
Pass: [56, 8] - [56, 12]
]
Case: [57, 4] - [58, 12]
pattern:
MatchSequencePattern: [57, 9] - [57, 19]
patterns: [
MatchLiteralPattern: [57, 10] - [57, 12]
literal:
UnaryOp: [57, 10] - [57, 12]
op: UAdd
operand:
Num: [57, 11] - [57, 12]
n: 0
text: '0'
MatchLiteralPattern: [57, 14] - [57, 18]
literal:
UnaryOp: [57, 14] - [57, 18]
op: USub
operand:
Num: [57, 15] - [57, 18]
n: 0.0
text: '0.0'
]
guard: None
body: [
Pass: [58, 8] - [58, 12]
]
Case: [59, 4] - [60, 12]
pattern:
MatchMappingPattern: [59, 9] - [59, 33]
mappings: [
MatchKeyValuePattern: [59, 10] - [59, 15]
key:
MatchLiteralPattern: [59, 10] - [59, 12]
literal:
UnaryOp: [59, 10] - [59, 12]
op: UAdd
operand:
Num: [59, 11] - [59, 12]
n: 1
text: '1'
value:
MatchCapturePattern: [59, 14] - [59, 15]
variable:
Name: [59, 14] - [59, 15]
variable: Variable('x', None)
ctx: Store
MatchKeyValuePattern: [59, 17] - [59, 24]
key:
MatchLiteralPattern: [59, 17] - [59, 21]
literal:
UnaryOp: [59, 17] - [59, 21]
op: UAdd
operand:
Num: [59, 18] - [59, 21]
n: 1.5
text: '1.5'
value:
MatchCapturePattern: [59, 23] - [59, 24]
variable:
Name: [59, 23] - [59, 24]
variable: Variable('y', None)
ctx: Store
MatchKeyValuePattern: [59, 26] - [59, 32]
key:
MatchLiteralPattern: [59, 26] - [59, 29]
literal:
UnaryOp: [59, 26] - [59, 29]
op: UAdd
operand:
Num: [59, 27] - [59, 29]
n: 2j
text: '2j'
value:
MatchCapturePattern: [59, 31] - [59, 32]
variable:
Name: [59, 31] - [59, 32]
variable: Variable('z', None)
ctx: Store
]
guard: None
body: [
Pass: [60, 8] - [60, 12]
]
]
]
18 changes: 18 additions & 0 deletions python/extractor/tests/parser/match_new.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,21 @@ def soft_keywords():
match z:
case w:
pass

match value:
case +1:
pass
case +1.5:
pass
case +2j:
pass
case +1+2j:
pass
case +1-2j:
pass
case +0x10 | +0o10 | +0b10:
pass
case [+0, -0.0]:
pass
case {+1: x, +1.5: y, +2j: z}:
pass
Loading
Loading