[ysfsmet]

Closes #1805.

Industrial-automation repositories built with Beckhoff TwinCAT 3 (and CODESYS-family IDEs) were entirely invisible to the indexer: PLC sources (.TcPOU, .TcDUT, .TcGVL, .TcIO) and project files (.plcproj, .tsproj) matched nothing in the extension table and were silently skipped. This adds first-class IEC 61131-3 Structured Text support, following the patterns the codebase already sanctions.

Everything below was validated against a real 130-file / 66-POU TwinCAT solution, not only fixtures — that is where most of the interesting defects came from.

What lands

Three new languages.

• CBM_LANG_IEC_ST — the grammar language, for plain .st/.iecst sources.
• CBM_LANG_TWINCAT — transform-only container for .TcPOU/.TcDUT/.TcGVL/.TcIO, mirroring CBM_LANG_OBJECTSCRIPT_EXPORT: internal/cbm/twincat_xml.c recomposes the <Declaration>/<Implementation><ST> CDATA into textual ST, newline-padded so definition lines still point into the physical file, and the generated units are re-extracted as CBM_LANG_IEC_ST.
• CBM_LANG_PLCOPEN_XML — transform-only container for CODESYS/PLCopen TC6 exports, detected by a content sniff on the plcopen.org/xml namespace. PLCopen declares variables as structured XML rather than CDATA, so internal/cbm/plcopen_xml.c synthesizes the declaration text from <inputVars>/<outputVars>/<variable>.

A dedicated project-file pass (src/pipeline/pass_tcproj.c): each .plcproj becomes a Package node with DEPENDS_ON edges to its library references (Tc2_Standard, Tc3_Module, …) and CONTAINS_FILE edges to its members; a .tsproj gets DEPENDS_ON to the PLC projects it references and CONFIGURES to their .xti device descriptions. Both are registered as incremental control files.

Graph mapping. FUNCTION_BLOCK/PROGRAM → Class; FUNCTION → Function; METHOD/PROPERTY → Method with DEFINES_METHOD; INTERFACE → Interface; DUT TYPE → Type; GVL entries → Variable; EXTENDS → INHERITS; IMPLEMENTS → IMPLEMENTS.

The grammar is a fork, and that was necessary

The upstream grammar (HeytalePazguato/tree-sitter-iec61131-3-st, MIT, ABI 15) is standard-only, and standard-only does not parse production TwinCAT: 49 of 130 files parsed partially, and every partial parse cost symbols — whole function blocks reached the graph without their methods.

It is forked into tools/tree-sitter-iec-st/ (grammar.js plus generated parser, same layout as tools/tree-sitter-form). Each rule was derived from a real failing construct and verified against the same corpus: modifiers after the POU keyword (FUNCTION_BLOCK PUBLIC FINAL FB_X), wildcard located addresses (AT %I*, spaced or glued, on VAR entries and DUT fields), bit-in-word access (wError.0 := TRUE), a base type after an enumerator list ((Idle, Run) UINT;), REFERENCE TO beside REF_TO, function-block instance argument lists (fb : FB_T(), ByteBuffer(ADR(x), SIZEOF(x))), the terminator TwinCAT omits after a block-shaped TYPE, attribute pragmas above members, __TRY/__CATCH/__FINALLY/__ENDTRY, ;-terminated interface prototypes, and tolerance for a stray ;.

Result on the same solution: 49 partially-parsed files → 8, and 98 more methods reach the graph. ABI stays 15; generation is reproducible (tree-sitter generate, CLI 0.26.x).

Defects found and fixed along the way

Three of these are worth calling out because they were silent — no error, no failing test, just wrong or missing data:

• POU names became PUBLIC. The grammar bound the access modifier to the declaration's name field and dropped the real identifier into an ERROR node: 13 function blocks named PUBLIC on the real solution. Recovered structurally (header line only, so a body-level parse error can never rename its POU), which serves all three mis-bound POU kinds without needing the source buffer.
• Project files minted one Class per XML element. Mapping .plcproj/.tsproj to CBM_LANG_XML so they would get File nodes also ran the XML grammar over their markup: 120 nodes named Project, PropertyGroup, Name, BitSize buried the 64 genuine POU classes. The definition passes now skip grammar extraction for them — in both the sequential and parallel dispatch.
• A stale grammar object linked silently. grammar_*.c is a one-line wrapper that #includes its vendored parser, and make cannot see through an include. Harmless for grammars that never change; for a fork it meant the rebuilt binary kept the old grammar while the source tree showed the new one. Makefile.cbm now names the vendored parser as an explicit prerequisite (prod/test/tsan).

Also fixed: pass_semantic_edges passed NULL to qsort when a project has Class/Method nodes but no Function node — formal UB that UBSan flags, and a TwinCAT solution is exactly that shape.

Testing

Every registry gate is updated: the capability ledger and its partition counts, the call-argument matrix, the call-node manifest, label goldens, vendored-checksums.txt, and the MANIFEST provenance record (fork noted, so a future re-vendor re-applies the dialect rules instead of reverting them).

New tests cover the transcoders (multi-POU, declaration synthesis, graphical-body tolerance, BOM, UTF-16 rejection, entity decoding, prefix near-misses), the access-modifier recovery, the project-markup exclusion, and an end-to-end pipeline_twincat_project_graph asserting the Package/DEPENDS_ON/CONTAINS_FILE shape plus a transcoded POU's DEFINES_METHOD chain.

All suites pass; make -f Makefile.cbm cbm links.

Two notes for reviewers:

1. tests/test_venue_parity_contract.sh hangs on any machine with ImageMagick installed — it probes scripts/ci/generate-sbom.py with bash, so import datetime runs ImageMagick's screen grabber. That is scripts/test.sh hangs indefinitely at the venue-parity contract when ImageMagick is installed (bash runs generate-sbom.py, 'import' becomes ImageMagick) #1384, not this branch; the suites here were run with DISPLAY unset to get past it.
2. On this machine tests/test_cli.c:1748 (cli_uninstall_quiesces_active_cohort_before_removing_binary_and_index) fails identically on pristine main — verified in a worktree at the merge base. Not introduced here.

Known limitations (documented in README)

Cross-file GVL/DUT references resolve by name rather than declared type; a function-block instance call resolves on the instance identifier rather than the block's type; inside a TwinCAT container the POU's own body line numbers can land after its methods (declaration lines are exact); PLCopen array/pointer-typed variables are dropped rather than approximated, and resource-level <globalVars> outside a POU interface are not transcoded. Eight of the 130 files still parse partially — the remaining constructs are listed in the fork's notes.

🤖 Generated with Claude Code

https://claude.ai/code/session_011BDg22SxMEYCXxGQicWgAc

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[DeusData]

Thank you for this — it is a serious piece of work, and validating against a real 130-file / 66-POU TwinCAT solution rather than only fixtures is exactly the evidence that makes a language contribution credible. Several details show you read how this codebase already does things: mirroring CBM_LANG_OBJECTSCRIPT_EXPORT for the transform-only container, newline-padding the recomposed ST so definition lines still point into the physical file, and the explicit Makefile.cbm dependency so a regenerated grammar cannot silently relink a stale object.

The maintainer decision is that this needs to be split: one PR per language, each reviewed on its own merit.

I want to be straight with you about what that does and does not mean: splitting is not a commitment to merge any of them. Each will be judged separately, and any one could still be declined. I would rather say that now than have you do the work on an implied promise.

The split we are asking for

The dependency order makes itself:

1. CBM_LANG_IEC_ST alone — the grammar language for .st / .iecst, the vendored grammar, and the extraction work in extract_defs.c / lang_specs.c. This is the foundation; both containers re-extract as IEC_ST, so nothing else can be judged before it.
2. CBM_LANG_TWINCAT — the .TcPOU / .TcDUT / .TcGVL / .TcIO container and twincat_xml.c.
3. CBM_LANG_PLCOPEN_XML — the CODESYS/PLCopen container and plcopen_xml.c.

pass_tcproj.c and the .plcproj / .tsproj handling can ride with (2) or come as a fourth — your call, whichever reviews more cleanly.

The practical reason is reviewability. At +282,044 lines over 61 files, no reviewer can give the extraction logic, two XML transforms, a new pipeline pass and a 1,253-line grammar the attention each deserves in one pass — and a self-maintained grammar fork is a long-term maintenance commitment that should be decided on its own, not as a by-product of accepting TwinCAT support.

Things to fix in whichever PR vendors the grammar

• THIRD_PARTY.md is not updated. Our vendoring policy requires the LICENSE vendored with the code and THIRD_PARTY.md plus the SBOM updated in the same change — notices ship inside the binary archives, so a missing entry is a real gap rather than paperwork. You did update MANIFEST.md and scripts/vendored-checksums.txt and vendor the LICENSE, so this is the one that got away.
• The published language count is gated. Each new language has to be reflected on every surface the contract test checks, and adding three at once against a moving count is exactly how two branches each write N+1 and merge to a wrong total.
• This currently conflicts with main and has no CI results at all. Smaller PRs will also get through our queue far better — we are heavily backlogged right now, and a 61-file PR is the worst possible shape for that.

On the security review

Nothing to worry about there, so you do not need to pre-empt it: the grammar is MIT, the new tools/tree-sitter-iec-st/package.json has no install hooks and its dependencies are ordinary tree-sitter tooling in an authoring directory our build does not install, and the Makefile.cbm change is a legitimate build-dependency fix. I checked all three.

I know splitting a landed piece of work is unrewarding, and I am sorry to ask for it without a guarantee at the end. But per-language review is the only way this gets a fair hearing rather than a nervous one.

[DeusData]

Maintainer notice: please disregard comments from @adfjadfj16-a11y on this thread

@adfjadfj16-a11y is not a maintainer of this project and does not speak for it. That account has posted replies on 17 threads here written in the project's voice — promising merges, announcing that a case has been "escalated to the development team", asking to close issues, and in some threads replying as though it were the author of someone else's pull request. None of those were maintainer decisions, and none of them carried any weight.

@DeusData is the only account that gives a maintainer response on this repository. If a comment about the fate of your issue or pull request did not come from @DeusData, it is not a decision, however official it reads.

If you were waiting on something because of one of those comments — a promised merge, a review "immediately", a request to close your ticket — I am sorry. That was noise you had no way to identify as noise, and it should not have been on your thread. Your issue or PR is judged on its own merits, and I will answer it here myself.

Nothing in this notice reflects on your contribution. Thank you for your patience, and thank you for the work.

Dismissed by the maintainer: this account is not a member of the maintainer team and its reviews carry no weight here. Only reviews from DeusData are maintainer decisions.

[DeusData]

Heads-up, with thanks: one small defect fix bundled in this PR is too useful to wait for the language review, so it is split out as #2083 with you credited as co-author: the semantic-edges pass called the sort with a null array whenever a project has no functions or methods, which is undefined behaviour that the Linux sanitizer lane reports today. When you next rebase this branch that hunk will drop out on its own; the language addition itself stays under review here as before.

[DeusData]

Status note: the empty-graph function-sort guard that surfaced while reviewing this PR (a qsort over zero functions) landed on main as 8ec859b (#2083) and ships in v0.10.9, so it is off this PR's plate. The language support itself is still under review on its own merits. Thank you for the patience on this one.

[SarletteD]

Hi @DeusData, hi @ysfsmet,

I've been working on the same topic independently, on
SarletteD/codebase-memory-mcp @ feature/twincat-structured-text,
and use it daily on a real TwinCAT 3 code base (about 1,300 object files spread over several
libraries; the project itself is proprietary, so I can only share numbers). I don't want to open a
competing PR. #1810 was here first, and you've already given it a split plan. So I'm writing here
to ask how this work could help instead.

Where the two approaches differ

• One language, CBM_LANG_ST, on the unmodified upstream grammar
  (HeytalePazguato/tree-sitter-iec61131-3-st @ 00e24f50f8de, vendored via
  scripts/vendor-grammar.sh). TwinCAT XML is TRANSFORM_ONLY: twincat_xml.c reassembles each
  POU and normalizes the dialect the grammar excludes. Every pass puts back the newlines it
  removes, so definitions and usages map back to exact XML lines. Across 1,380 real files that
  brings the parse-error rate from 84.9 % down to 4.8 %, with no grammar fork to maintain.
  PLCopen XML and .plcproj project nodes are not covered. feat(lang): TwinCAT / CODESYS (IEC 61131-3 Structured Text) support #1810 has those.

What the fork adds that I couldn't find in #1810's diff (each measured against an XML ground
truth on that code base):

1. Namespace-qualified bases. EXTENDS Lib_A.FB_Base / IMPLEMENTS Lib_A.I_X resolve through
   the .plcproj placeholder chain (source → nearest .plcproj → alias → placeholder title → the
   library that declares it), in both resolution venues. It never guesses by the last segment,
   because the same short name exists in several libraries. Result: 954/954 resolvable base
   references get an edge, 0 point at the wrong library (IMPLEMENTS 475 → 576, INHERITS 295 → 378).
2. DUT internals as nodes. Enum members and struct/union fields become Field nodes under their
   Type (enum_value, return_type, parent_class). E_X.MEMBER, Alias.E_X.MEMBER and struct
   fields reached through a declared variable resolve to member-level USAGE edges: 2,122 fields,
   6,315 member edges on that tree, and spot checks match a text search.
3. line and count on every Structured Text USAGE edge.
4. get_architecture lists Structured Text / TwinCAT in its language table.

All of it is covered by tests in test_extraction.c / test_edge_structural.c. Full
scripts/test.sh: 141 suites, 8061 passed, 0 failed. lint.sh --ci with clang-format 20.1.8 and cppcheck 2.20.0
reports nothing in the touched files, and make security passes.

What I'd propose

Once the IEC_ST foundation from #1810's split is settled, I could port items 1–4 onto it as small,
separate issues + PRs (<500 lines each, DCO-signed). The namespace resolution (1) and the DUT
members (2) are each independent of how the TwinCAT XML is transcoded. If you'd rather take the
single-language approach above as the base instead, I can also split that into reviewable steps.
Either way, I won't open anything until you say which foundation you prefer.

[DeusData]

Thank you for the grammar and normalization follow-up. There are now competing foundation choices for the IEC ST work, and that direction needs a maintainer decision before more porting effort. We need more time to compare them; please do not interpret this as a request to rework the stack against an unchosen foundation.

[ysfsmet]

I’m completely fine with moving forward with @SarletteD ’s approach if that’s the direction the maintainers would prefer for the IEC ST foundation.

I’ve been dealing with a short-term increase in workload recently, so unfortunately I haven’t been able to dedicate enough time to continue development on this PR.

If you’re willing, @SarletteD, I’d really appreciate it if you could split your implementation into the smaller, focused PRs suggested by @DeusData and submit them separately. I’m happy for the work to continue in that direction, and I can contribute again once my availability improves.

Thanks for picking this up and for the additional work on the TwinCAT/ST support.

[SarletteD]

Thanks a lot, @ysfsmet — for the go-ahead and for the groundwork in this PR, much of which
(the transform-only container, line-preserving recomposition) shaped my own approach.

@DeusData, understood from your note on the 19th: I won't open anything until you have chosen
the foundation. To make the comparison easier, this is how I would split the single-language
approach, in dependency order — one issue + one PR each, DCO-signed, rebased on current main,
CI green before asking for review:

1. CBM_LANG_ST + vendored grammar — the unmodified upstream grammar with LICENSE,
   THIRD_PARTY.md, SBOM, checksums and every language-count surface in the same change, plus
   extraction for plain .st files. The generated parser.c is the bulk; the hand-written part
   stays small.
2. TwinCAT XML container — twincat_xml.c, transform-only, line-preserving normalization.
3. Namespace-qualified EXTENDS / IMPLEMENTS through the .plcproj placeholder chain.
4. DUT members and fields as Field nodes with member-level USAGE edges.
5. line / count on Structured Text USAGE edges.
6. Structured Text / TwinCAT in the get_architecture language table.

PLCopen XML and .plcproj project nodes are out of scope for me; #1810 has them, and they could
follow on top of either foundation.

If a draft of step 1 would help you compare, I can put one up; otherwise I'll wait for your call.

[DeusData]

Thank you both. @ysfsmet, for the groundwork here and for handing it on so generously. @SarletteD, for building on it and for laying out the split so clearly.

We are going with your single-language Structured Text approach as the foundation, @SarletteD, in the six PRs you listed and in that order. Please start with PR 1 (CBM_LANG_ST plus the vendored grammar). A few things will make it land smoothly:

• Vendoring: the grammar's license must be permissive (if it is GPL, check with us first). Vendor its LICENSE file with the code, and update THIRD_PARTY.md, the SBOM and internal/cbm/vendored/grammars/MANIFEST.md in the same PR. You already planned for this, thank you.
• Language count: every language-count surface in the same change. The version-metadata contract test will catch any you miss.
• A small real-world proof: node and edge counts, plus index time, on one public TwinCAT or .st project. That is how we judge graph quality before PRs 2 to 6 build on it.

We'll keep #1810 open as the reference until PR 1 is up, then close it with credit to @ysfsmet for the groundwork. Thank you again for keeping this moving.

[DeusData]

@ysfsmet, thank you for this, and for how generously you handed it on. This PR is where Structured Text support in this project really started: it showed that TwinCAT and CODESYS repositories were invisible to the indexer, and it worked out how to change that on a real 130-file solution rather than only on fixtures.

As announced on 2026-09-25, the Structured Text foundation is going ahead as the single-language series by @SarletteD, and its first PR is now up as #2397. As we said then, that is the point at which we close this one, with credit to you for the groundwork. Several things from here carry forward or are already on main:

1. The transform-only container that mirrors the ObjectScript export, and the line-preserving recomposition of the CDATA into Structured Text, shaped the approach the new series builds on; @SarletteD said as much in this thread.
2. The empty-graph sort guard you found landed on main in fix(pipeline): skip the function sort when the graph has no functions #2083, with you as co-author.
3. The stale grammar object you diagnosed, where make cannot see through the one-line grammar wrapper, has since been closed for every vendored grammar on main (6f290e1).

Two parts of this PR are not in the six-PR plan: the PLCopen XML container and the .plcproj / .tsproj project nodes. If you find the time once the Structured Text foundation has landed, those would be very welcome as small, separate PRs on top of it, each reviewed on its own merit as before. #1805 stays open to track the language support until the series lands.

Thank you again for the work, and for keeping the door open for the next step.