feat(lang): TwinCAT / CODESYS (IEC 61131-3 Structured Text) support - #1810
Open
ysfsmet wants to merge 13 commits into
Open
feat(lang): TwinCAT / CODESYS (IEC 61131-3 Structured Text) support#1810ysfsmet wants to merge 13 commits into
ysfsmet wants to merge 13 commits into
Conversation
Industrial automation repositories built with Beckhoff TwinCAT 3 were entirely invisible to the indexer: PLC sources (.TcPOU/.TcDUT/.TcGVL/.TcIO) and project files (.plcproj/.tsproj) matched nothing in the extension table and were silently skipped. Three layers, each following an existing pattern: 1. CBM_LANG_IEC_ST — vendored HeytalePazguato/tree-sitter-iec61131-3-st (MIT, ABI 15, pinned 00e24f50f8de) for bare .st/.iecst sources. FUNCTION_BLOCK/PROGRAM/INTERFACE/TYPE are class-like nodes whose METHOD/ACTION members are direct children, so find_class_body and get_module_parents get IEC_ST branches; VAR_GLOBAL blocks mint one Variable per declared name, and EXTENDS/IMPLEMENTS fill base_classes (the edge type is decided downstream from the target's label). VAR entries are binding sites, not usages, so declaring a variable no longer makes it look used. 2. CBM_LANG_TWINCAT — transform-only XML container mirroring CBM_LANG_OBJECTSCRIPT_EXPORT: twincat_xml.c recomposes the <Declaration>/<Implementation><ST> CDATA of a TcPlcObject into textual ST (members before the body, sniffed END_* terminator, the semicolon TwinCAT omits after END_STRUCT, ACTION as a parameterless METHOD), newline-padded so definition lines still point into the physical file. UTF-8 BOM is stripped; UTF-16 is rejected rather than garbled. 3. pass_tcproj.c — .plcproj becomes a Package that CONTAINS_FILE its Compile members and DEPENDS_ON its library placeholders; .tsproj DEPENDS_ON the PLC projects it references and CONFIGURES their .xti device descriptions. Both are incremental control files. Registry gates updated accordingly (capability ledger, call-argument matrix B, call-node manifest, label goldens, vendored checksums, grammar MANIFEST provenance). Refs: DeusData#1805 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
Add CBM_LANG_PLCOPEN_XML, a transform-only language for CODESYS/PLCopen TC6 XML exports (<project xmlns=".../plcopen.org/xml/tc6_...">), detected by content sniff and transcoded to textual IEC 61131-3 ST before being extracted as CBM_LANG_IEC_ST -- the same lifecycle CBM_LANG_TWINCAT uses. This is the last code phase of TwinCAT/CODESYS support (phases 1-3 at c828659e: the IEC ST grammar, the TwinCAT XML transcoder, and the .plcproj/.tsproj project pass). Unlike TwinCAT's CDATA-wrapped ST, PLCopen declares variables as structured XML (<inputVars>/<outputVars>/.../<variable><type>...), so internal/cbm/plcopen_xml.c synthesizes the ST declaration text from the XML rather than copying it, then appends the <body><ST> text verbatim (entity-decoded). One export file holds many <pou> elements, so the transcoder returns one ST unit per POU (plus one per <dataType> alias/ struct, which "fell out" of the same variable-list scanner). Two ST grammar keywords (ARRAY, POINTER) can't be used as bare identifiers, so a variable typed that way is skipped rather than emitted as text the grammar would reject; a FUNCTION POU with no <returnType> falls back to ANY so its header still parses. Wiring follows the TwinCAT/Studio-Export precedent arm for arm: enum entry, discover.c content sniff (widened from 256 bytes to 4KB so the namespace declaration is reliably in view), LANG_NAMES/LANG_NAME_TABLE rows, cbm_pipeline_extract_plcopen() next to cbm_pipeline_extract_twincat reusing its append helpers, and a third dispatch arm in both the sequential and parallel pipeline passes. No spec row is added -- cbm_lang_spec(CBM_LANG_PLCOPEN_XML) returns NULL, matching the registry repro's transform-only ledger. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
`git clang-format` over the branch's own changed lines only; upstream lines in tests/test_extraction.c (which was already non-conforming before this branch) are deliberately left alone so the diff stays reviewable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
Update README.md's language count (158 -> 159, one new grammar: iec_st) and add a compact subsection covering indexed file types, POU-to-graph-node mapping, the .plcproj/.tsproj project pass, and the honest limitations of name-based cross-file resolution and TwinCAT/PLCopen transcoding. Add a "Transform-Only Container Languages" subsection to CONTRIBUTING.md for the third language-support shape the branch introduced (enum value with no grammar, transcoded to another registered language and re-extracted). Add the missing iec_st row to the MANIFEST.md custom extraction handling table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
The iec_st grammar has no ACTION node type (iec_st_func_types/iec_st_class_types in lang_specs.c carry nothing action-related); ACTION only reaches the graph as a Method through the TwinCAT/PLCopen container transcoders, which synthesize it as a parameterless METHOD before parsing (twincat_xml.c:310-314, plcopen_xml.c:442-457). The README table and MANIFEST.md's iec_st row overstated this as true of the grammar itself. Also fix PROPERTY: it is one Method per property_declaration, not one per Get/Set accessor (no separate accessor func-type in iec_st_func_types). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
Five blocking findings from final code review: - twincat_xml.c: convert tc_elem's element-name locate step from tail recursion to an iterative loop (mirrors px_next_elem in plcopen_xml.c), so a file full of prefix near-miss tokens (e.g. <Declarationz>) cannot blow the stack at -O1, where GCC does not sibling-call-optimize the tail call away. - twincat_xml.c: tb_pad_to now breaks out if tb_app fails to advance the buffer position, instead of looping forever once the output buffer is exhausted. - pass_tcproj.c: resolve_ref_path uses strtok_r with a local saveptr instead of strtok, matching every other tokenizer call site and the .clang-tidy rationale that only readdir() retains global state. - plcopen_xml.c: correct px_emit_pou's doc comment, which claimed a NULL-name contract the code does not implement (it always emits a unit, substituting "PlcopenPOU" for an empty name). - lang_specs.c: fix the IEC ST comment's ACTION overstatement (no grammar row exists; <Action> is synthesized as METHOD text by the container transcoders) and add the missing CBM_LANG_TWINCAT "no spec row" comment alongside its OBJECTSCRIPT_EXPORT/PLCOPEN_XML siblings, per CONTRIBUTING.md. Adds a regression test (twincat_xml_prefix_near_miss_bounded_recursion) that feeds cbm_twincat_to_st a few hundred <Declarationz>/<STx> near-miss tokens to prove the locate loop is now depth-bounded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
scripts/vendored-checksums.txt covers every file under internal/cbm/vendored/, including grammars/MANIFEST.md itself. The manifest was regenerated when the iec_st grammar was vendored, but the later documentation commit added the iec_st row to MANIFEST.md's custom-extraction table without refreshing it, so tests/test_cli.c's vendored-integrity probe failed on a stale digest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
…edes it TwinCAT writes access modifiers on POU headers (`FUNCTION_BLOCK PUBLIC FB_X`, and `PUBLIC FINAL` in combination). The vendored grammar has no rule for them, so it binds the first modifier to the declaration's `name` field and drops the remaining tokens — including the real identifier — into an ERROR node beside it. The graph then shows a function block literally named "PUBLIC": 13 of them on the real 66-POU solution this was found on, with the actual block names absent. Recover the name structurally rather than by text: the ERROR node that starts on the header line after `name` holds the remaining identifiers, and the last of them is the POU name. Restricting the search to the header line keeps a body-level parse error from ever renaming its POU, and needing no source buffer lets the same helper serve cbm_resolve_func_name (FUNCTION) and the class-name path (FUNCTION_BLOCK/PROGRAM), all three of which the grammar mis-binds. The class-name hook deliberately sits after the fallback switch: that switch only runs when no name was found at all, whereas here the field is populated with the wrong token. The parse error itself still surfaces as a partial-parse range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
.plcproj/.tsproj/.xti were mapped to CBM_LANG_XML so they would be discovered and get File nodes for pass_tcproj to attach to. The side effect: the XML grammar also ran over their markup, and every element became a Class node. On the real solution this was found on that produced 120 nodes named "Project", "PropertyGroup", "FileVersion", "Name", "Comment", "BitSize" — burying the 64 genuine POU classes and polluting both search results and the graph UI. pass_tcproj already owns what these files mean (Package nodes, DEPENDS_ON to library references, CONTAINS_FILE to members, CONFIGURES to .xti), so the definition passes now skip grammar extraction for them. File nodes come from pass_structure and are unaffected, so the project pass still finds every one. The skip is added to BOTH the sequential and the parallel dispatch: the chains must stay symmetric, and an omission in the parallel one would show up only on repositories above the parallel threshold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
pass_semantic_edges sorts its collected Function nodes, but func_count == 0 leaves node_ptrs NULL and qsort declares its base pointer non-null, so the call is formal UB that UBSan flags. Reachable from any project whose graph holds Class/Method nodes but no Function node — a TwinCAT PLC solution is exactly that shape, which is how the new pipeline_twincat_project_graph fixture surfaced it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
The vendored upstream grammar is standard-only, and standard-only does not parse production TwinCAT: on a 130-file solution, 49 files parsed partially. Every partial parse costs symbols — whole function blocks reached the graph without their methods. Forked into tools/tree-sitter-iec-st/ (grammar.js plus the generated parser, same layout as tools/tree-sitter-form) rather than waiting on upstream. Each rule below was derived from a failing construct in that corpus and verified against it: - access/inheritance modifiers AFTER the POU keyword, which is where TwinCAT writes them (`FUNCTION_BLOCK PUBLIC FINAL FB_X`) - wildcard located addresses (`AT %I*`), spaced or glued (`AT%I*`), on VAR entries and on DUT fields - bit-in-word member access (`wError.0 := TRUE`) - a base type after an enumerator list (`(Idle, Run) UINT;`) - `REFERENCE TO` beside the standard `REF_TO` - function-block instance argument lists (`fb : FB_T()`, `ByteBuffer(ADR(x), SIZEOF(x))`), restricted to user-defined type names so `s : STRING(255)` stays a string length - an optional terminator after a block-shaped TYPE definition, which TwinCAT omits (`END_STRUCT` straight to `END_TYPE`) - attribute pragmas above POU members - TwinCAT `__TRY/__CATCH/__FINALLY/__ENDTRY` - interface method prototypes terminated by `;` - tolerance for a stray `;` in a VAR block, so one typo cannot cost a POU's whole symbol set Two GLR conflicts are declared deliberately: a pragma is ambiguous between a member prefix and a body statement, and `__CATCH(e)` looks like a call until the keyword is consumed. Result on the same solution: 49 partially-parsed files down to 8, with no regression across the existing fixtures. ABI stays 15 (CLI 0.26.x), and the generated parser is reproducible — regenerate in tools/tree-sitter-iec-st and copy src/parser.c + src/scanner.c into the vendored path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
…nges grammar_*.c is a one-line wrapper that #includes its vendored parser.c and scanner.c, and make cannot see through that include. For every other grammar the vendored bytes never change after vendoring, so the pattern rule is enough. iec_st is different: it is a self-maintained fork under tools/tree-sitter-iec-st, so regenerating it really does change the vendored parser. Without the explicit dependency the stale object is silently relinked. That is exactly what happened during testing — the source tree carried the forked grammar, the rebuilt binary did not, and the indexer kept reporting the old partial-parse count with no error anywhere to explain it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
The grammar is no longer a straight upstream vendor, so the docs had to say so: README now states that `iec_st` is a self-maintained fork, lists the dialect forms it adds, and carries the measurement that justifies the fork (partial parses on a 130-file production solution: 49 files down to 8). MANIFEST's category counts move one grammar from vendored-from-upstream to first-party/self-maintained. CONTRIBUTING gains a "Self-Maintained Grammar Forks" section with the regenerate-and-vendor loop, the ABI-15 ceiling check, and the reason Makefile.cbm names the vendored parser as an explicit prerequisite: make cannot see through the #include in grammar_*.c, so without it a regenerated grammar links a stale object and the binary silently keeps the old one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yusuf S. Demirci <ysf-samet@outlook.com>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1805.
Industrial-automation repositories built with Beckhoff TwinCAT 3 (and CODESYS-family IDEs) were entirely invisible to the indexer: PLC sources (
.TcPOU,.TcDUT,.TcGVL,.TcIO) and project files (.plcproj,.tsproj) matched nothing in the extension table and were silently skipped. This adds first-class IEC 61131-3 Structured Text support, following the patterns the codebase already sanctions.Everything below was validated against a real 130-file / 66-POU TwinCAT solution, not only fixtures — that is where most of the interesting defects came from.
What lands
Three new languages.
CBM_LANG_IEC_ST— the grammar language, for plain.st/.iecstsources.CBM_LANG_TWINCAT— transform-only container for.TcPOU/.TcDUT/.TcGVL/.TcIO, mirroringCBM_LANG_OBJECTSCRIPT_EXPORT:internal/cbm/twincat_xml.crecomposes the<Declaration>/<Implementation><ST>CDATA into textual ST, newline-padded so definition lines still point into the physical file, and the generated units are re-extracted asCBM_LANG_IEC_ST.CBM_LANG_PLCOPEN_XML— transform-only container for CODESYS/PLCopen TC6 exports, detected by a content sniff on theplcopen.org/xmlnamespace. PLCopen declares variables as structured XML rather than CDATA, sointernal/cbm/plcopen_xml.csynthesizes the declaration text from<inputVars>/<outputVars>/<variable>.A dedicated project-file pass (
src/pipeline/pass_tcproj.c): each.plcprojbecomes aPackagenode withDEPENDS_ONedges to its library references (Tc2_Standard,Tc3_Module, …) andCONTAINS_FILEedges to its members; a.tsprojgetsDEPENDS_ONto the PLC projects it references andCONFIGURESto their.xtidevice descriptions. Both are registered as incremental control files.Graph mapping.
FUNCTION_BLOCK/PROGRAM→Class;FUNCTION→Function;METHOD/PROPERTY→MethodwithDEFINES_METHOD;INTERFACE→Interface; DUTTYPE→Type; GVL entries →Variable;EXTENDS→INHERITS;IMPLEMENTS→IMPLEMENTS.The grammar is a fork, and that was necessary
The upstream grammar (HeytalePazguato/tree-sitter-iec61131-3-st, MIT, ABI 15) is standard-only, and standard-only does not parse production TwinCAT: 49 of 130 files parsed partially, and every partial parse cost symbols — whole function blocks reached the graph without their methods.
It is forked into
tools/tree-sitter-iec-st/(grammar.js plus generated parser, same layout astools/tree-sitter-form). Each rule was derived from a real failing construct and verified against the same corpus: modifiers after the POU keyword (FUNCTION_BLOCK PUBLIC FINAL FB_X), wildcard located addresses (AT %I*, spaced or glued, on VAR entries and DUT fields), bit-in-word access (wError.0 := TRUE), a base type after an enumerator list ((Idle, Run) UINT;),REFERENCE TObesideREF_TO, function-block instance argument lists (fb : FB_T(),ByteBuffer(ADR(x), SIZEOF(x))), the terminator TwinCAT omits after a block-shapedTYPE, attribute pragmas above members,__TRY/__CATCH/__FINALLY/__ENDTRY,;-terminated interface prototypes, and tolerance for a stray;.Result on the same solution: 49 partially-parsed files → 8, and 98 more methods reach the graph. ABI stays 15; generation is reproducible (
tree-sitter generate, CLI 0.26.x).Defects found and fixed along the way
Three of these are worth calling out because they were silent — no error, no failing test, just wrong or missing data:
PUBLIC. The grammar bound the access modifier to the declaration'snamefield and dropped the real identifier into an ERROR node: 13 function blocks namedPUBLICon the real solution. Recovered structurally (header line only, so a body-level parse error can never rename its POU), which serves all three mis-bound POU kinds without needing the source buffer.Classper XML element. Mapping.plcproj/.tsprojtoCBM_LANG_XMLso they would get File nodes also ran the XML grammar over their markup: 120 nodes namedProject,PropertyGroup,Name,BitSizeburied the 64 genuine POU classes. The definition passes now skip grammar extraction for them — in both the sequential and parallel dispatch.grammar_*.cis a one-line wrapper that#includes its vendored parser, and make cannot see through an include. Harmless for grammars that never change; for a fork it meant the rebuilt binary kept the old grammar while the source tree showed the new one.Makefile.cbmnow names the vendored parser as an explicit prerequisite (prod/test/tsan).Also fixed:
pass_semantic_edgespassed NULL toqsortwhen a project hasClass/Methodnodes but noFunctionnode — formal UB that UBSan flags, and a TwinCAT solution is exactly that shape.Testing
Every registry gate is updated: the capability ledger and its partition counts, the call-argument matrix, the call-node manifest, label goldens,
vendored-checksums.txt, and the MANIFEST provenance record (fork noted, so a future re-vendor re-applies the dialect rules instead of reverting them).New tests cover the transcoders (multi-POU, declaration synthesis, graphical-body tolerance, BOM, UTF-16 rejection, entity decoding, prefix near-misses), the access-modifier recovery, the project-markup exclusion, and an end-to-end
pipeline_twincat_project_graphasserting the Package/DEPENDS_ON/CONTAINS_FILE shape plus a transcoded POU'sDEFINES_METHODchain.All suites pass;
make -f Makefile.cbm cbmlinks.Two notes for reviewers:
tests/test_venue_parity_contract.shhangs on any machine with ImageMagick installed — it probesscripts/ci/generate-sbom.pywithbash, soimport datetimeruns ImageMagick's screen grabber. That is scripts/test.sh hangs indefinitely at the venue-parity contract when ImageMagick is installed (bash runs generate-sbom.py, 'import' becomes ImageMagick) #1384, not this branch; the suites here were run withDISPLAYunset to get past it.tests/test_cli.c:1748(cli_uninstall_quiesces_active_cohort_before_removing_binary_and_index) fails identically on pristinemain— verified in a worktree at the merge base. Not introduced here.Known limitations (documented in README)
Cross-file GVL/DUT references resolve by name rather than declared type; a function-block instance call resolves on the instance identifier rather than the block's type; inside a TwinCAT container the POU's own body line numbers can land after its methods (declaration lines are exact); PLCopen array/pointer-typed variables are dropped rather than approximated, and resource-level
<globalVars>outside a POU interface are not transcoded. Eight of the 130 files still parse partially — the remaining constructs are listed in the fork's notes.🤖 Generated with Claude Code
https://claude.ai/code/session_011BDg22SxMEYCXxGQicWgAc