diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 430eae1733..521c48d6a8 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -246,6 +246,15 @@ AST cache schema 43 invalidates older template and conditional recovery records. 4. Add the tree-sitter package to `pyproject.toml` dependencies. 5. Add a fixture file to `tests/fixtures/` and tests to `tests/test_languages.py`. +### MATLAB fallback contract + +For `.m` files, explicit `.graphifyrc` remaps take precedence, then the existing Objective-C content detector; a non-Objective-C `.m` falls back to MATLAB. `.mm` remains Objective-C, and `language.inc=.m` keeps its existing Objective-C interpretation. This adds no MATLAB remap alias or force-language option. + +The file node remains the script/file navigation anchor and records the file kind when established; there is no separate synthetic script node. Located classes and functions are definition nodes; use immediate `contains` for file/owner-to-definition relationships and `method` for class-to-method relationships, not both for one pair. Trustworthy definitions remain even when relationship binding is unsafe; declaration-only methods are declarations, not fabricated executable bodies. This is not full MATLAB runtime semantics, and separately stored `@Class` methods are not associated. + +- **Calls:** emit only uniquely resolved, unqualified parenthesized calls in executable function or method bodies to lexically visible same-file functions. A call nested in a member/qualified call's argument is still eligible; the member/qualified callee itself is not. Parentheses/braces used for indexing, parameter/output/assignment bindings, and uncertain scopes do not produce calls. Script top-level, cross-file, qualified/member, command, handle, constructor/dynamic, and class-method target guesses are omitted; there is no generic unresolved-call pass. +- **Bases:** preserve written superclass names and source locations in class metadata, including external and unresolved names. Emit `inherits` only for a unique exact source match: a qualified namespace path, or an eligible unpackaged sibling file with a matching class declaration. This does not claim to model MATLAB runtime path selection and does not invent a target. +- **Provenance and call sites:** each MATLAB file/definition node keeps nested `metadata.language = "matlab"` through persistence. Extraction retains separate located call sites, while graph assembly coalesces repeated same-pair edges; the persisted graph is not a lossless call-site multigraph. ## Search hook scope The search hook omits project hints when a structured Grep path or every literal diff --git a/CHANGELOG.md b/CHANGELOG.md index 15fe794ae6..71b5b61208 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,8 +16,9 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Perf: **extract** the AST worker pool is sized by the amount of uncached work (one worker per ~500 KB of uncached source, capped at CPU count and file count) when no worker count is set, so small incremental builds stop spinning up a full pool; output is invariant to worker count (thanks @rohit-jsfreaky). - Change: **cloud** the OSS surfaces (README, CLI help/install text, the translated docs in all 31 locales, and the agent skill) now point at the free plan with no card at `app.graphify.com/signup`, instead of a "14-day trial" at `/login` (the trial is paid-plans-only and needs a card, so the old copy conflated the two). Links carry `utm_campaign=oss` UTMs tagged per surface, so a signup can be attributed to where it came from. - Change: **cloud** the post-build CLI nudge is contextual. It is TTY-only (never in CI or the agent pipeline), respects `GRAPHIFY_NO_TIPS`/`GRAPHIFY_NO_CTA`, and shows on builds 1, 5 and 20 then stays quiet for good. When a local graph reaches 25k nodes, the free plan's per-repo cut-off, the nudge switches to a scale-specific line (tagged `utm_content=scale`) that sells Graphify Pro and links to pricing with the 14-day trial, instead of the free signup: a repo that size no longer fits the free plan, so pointing it at the free plan would land straight on the limit. +- Feat: **matlab** `.m` files fall back to MATLAB only after explicit remaps and Objective-C content detection; extraction records located definitions, conservative same-file lexical calls, and exact-source inheritance while preserving `.mm` and `.inc=.m` Objective-C routing. -The AST cache schema is bumped to 44; warm caches re-run the affected C++, PHP, Java and module-level-id extractions. +The AST cache schema is bumped to 45; warm caches re-run the affected C++, PHP, Java, module-level-id and MATLAB extractions. ## 0.9.106 (2026-11-10) diff --git a/README.md b/README.md index a313093aaa..8857151d1a 100644 --- a/README.md +++ b/README.md @@ -401,7 +401,9 @@ remain effective. Create a `.graphifyrc` in the scan root for settings that belong to the project rather than to one run. One `key=value` per line; lines beginning with `#` are comments. -**Ambiguous extensions.** Some extensions mean different languages in different projects: `.inc` is PHP on pfSense, Pascal in a Delphi tree, SQL or assembly elsewhere; `.h` is C or C++; `.m` is Objective-C or MATLAB. graphify has to pick one global default, and when it picks wrong the file does not fail: it parses as the wrong language and yields a handful of incidental nodes, so the graph looks populated while the real symbols are missing. Declare what the extension means in your repo: +**Ambiguous extensions.** Some extensions mean different languages in different projects: `.inc` is PHP on pfSense, Pascal in a Delphi tree, SQL or assembly elsewhere; `.h` is C or C++; `.m` is Objective-C or MATLAB. Explicit `.graphifyrc` remaps let projects declare their intent; otherwise, an ambiguous extension can be parsed as the wrong language and yield incidental nodes while missing its real symbols. Declare what the extension means in your repo: + +For `.m` files, explicit `.graphifyrc` remaps take precedence, then the existing Objective-C content detector; a non-Objective-C `.m` falls back to MATLAB. `.mm` remains Objective-C, and `language.inc=.m` keeps its existing Objective-C interpretation. This adds no MATLAB remap alias or force-language option. ``` # .graphifyrc diff --git a/graphify/build.py b/graphify/build.py index 646d4d2a02..350ff787a2 100644 --- a/graphify/build.py +++ b/graphify/build.py @@ -1226,6 +1226,61 @@ def dedupe_edges(edges: list[dict]) -> list[dict]: return out +def _refresh_matlab_inheritance( + nodes: list[dict], edges: list[dict], root: str | Path | None = None, +) -> list[dict]: + """Replace only source-owned MATLAB inheritance facts for the current nodes.""" + from graphify.extractors.resolution import _matlab_inheritance_edges + _root = str(Path(root).resolve()) if root else None + + def rooted(item: dict) -> dict: + if not isinstance(item, dict): + return item + source = item.get("source_file") + normalized = _norm_source_file(source, _root) if isinstance(source, str) else source + return item if normalized == source else {**item, "source_file": normalized} + + # Resolver inputs may combine absolute incoming paths with relative stored + # paths. Canonicalize private views only; unchanged extraction context is read-only. + resolver_nodes = [rooted(node) for node in nodes] + resolver_edges = [rooted(edge) for edge in edges] + + matlab_sources: dict[str, str] = {} + for node in resolver_nodes: + if not isinstance(node, dict): + continue + metadata = node.get("metadata") + node_id = node.get("id") + source_file = node.get("source_file") + if not isinstance(metadata, dict) or metadata.get("language") != "matlab": + continue + if ( + not isinstance(node_id, str) + or not isinstance(source_file, str) + or not source_file + ): + continue + matlab_sources[node_id] = source_file + + def _is_matlab_inheritance(edge: dict) -> bool: + if not isinstance(edge, dict) or edge.get("relation") != "inherits": + return False + source = edge.get("source") + if not isinstance(source, str): + return False + source_file = matlab_sources.get(source) + return ( + source_file is not None + and edge.get("source_file") == source_file + and _is_ast_tier(edge) + ) + + kept = [edge for edge, resolver_edge in zip(edges, resolver_edges) + if not _is_matlab_inheritance(resolver_edge)] + refreshed = _matlab_inheritance_edges(resolver_nodes) + return kept + [{**edge, "_origin": "ast"} for edge in refreshed] + + def _old_file_stems(rel: Path) -> list[str]: """Pre-migration stem forms a semantic fragment may have used for ``rel``. @@ -2111,6 +2166,20 @@ def _resolve_endpoint(raw_id: str, edge_source_file: object) -> str: # Python `import time` must not bind to a `time.ts`, #1749). _edge_rel = attrs.get("relation") if _edge_rel in ("calls", "imports", "imports_from", "references"): + # .m is shared with Objective-C. Keep that provenance boundary for + # native code while letting source-backed docs/configs reach the + # established family gate below. + src_matlab = (G.nodes[src].get("metadata") or {}).get("language") == "matlab" + tgt_matlab = (G.nodes[tgt].get("metadata") or {}).get("language") == "matlab" + if src_matlab != tgt_matlab: + non_matlab = G.nodes[tgt] if src_matlab else G.nodes[src] + non_matlab_ext = Path(non_matlab.get("source_file") or "").suffix.lower() + source_backed_doc_or_config = bool(non_matlab.get("source_file")) and ( + non_matlab.get("file_type") in {"document", "paper", "image"} + or non_matlab_ext not in _EDGE_LANG_FAMILY + ) + if _edge_rel == "calls" or not source_backed_doc_or_config: + continue src_ext = Path(G.nodes[src].get("source_file") or "").suffix.lower() tgt_ext = Path(G.nodes[tgt].get("source_file") or "").suffix.lower() src_fam = _EDGE_LANG_FAMILY.get(src_ext) @@ -2760,6 +2829,9 @@ def _dropped(item: dict) -> bool: new["hyperedges"] = carried_hyper + list(new.get("hyperedges", [])) if unverified_semantic_shrink: new["_unverified_semantic_shrink"] = unverified_semantic_shrink + new["edges"] = _refresh_matlab_inheritance( + dedupe_nodes(new.get("nodes", [])), new.get("edges", []), root=_eff_root, + ) return new @@ -3049,7 +3121,29 @@ def _prune_match(sf: "str | None") -> bool: if isinstance(n, dict) and n.get("id") } if had_graph else None - all_chunks = base + list(new_chunks) + all_chunks = base + [dict(chunk) for chunk in new_chunks] + if all_chunks: + resolver_nodes = [ + node + for chunk in all_chunks + for node in chunk.get("nodes", []) + if isinstance(node, dict) and not _prune_match(node.get("source_file")) + ] + merged_edges = [ + edge for chunk in all_chunks for edge in chunk.get("edges", []) + ] + if dedup: + from graphify.dedup import _namespace_reference_stub_collisions + resolver_nodes, _ = _namespace_reference_stub_collisions( + resolver_nodes, [], None, + Path(_eff_root).resolve() if _eff_root else None, + ) + resolver_nodes = dedupe_nodes(resolver_nodes) + all_chunks[0]["edges"] = _refresh_matlab_inheritance( + resolver_nodes, merged_edges, root=_eff_root, + ) + for chunk in all_chunks[1:]: + chunk["edges"] = [] G = build( all_chunks, directed=directed, diff --git a/graphify/cache.py b/graphify/cache.py index 017a5702ed..0e92f84678 100644 --- a/graphify/cache.py +++ b/graphify/cache.py @@ -38,7 +38,8 @@ _EXTRACTOR_VERSION = "unknown" # Bump when AST cache-key semantics change independently of the package version. -_AST_CACHE_SCHEMA = 44 # A folded C++ parameterless member records its `definition_file` anchored to the source root instead of a cache-root-relative `../`-walk; a PHP route closure takes its path only from the arguments before it, so a string argument after the closure no longer becomes the route's path or leaks into an enclosing group's prefix (#5135); two module-level declarations in one file differing only by a leading/trailing `_` keep distinct ids (the first keeps the plain id, the rest are salted) across JS, TS, C, C++, PHP, Ruby, Kotlin, Swift and Lua (#2810); PHP methods of one class/trait/interface/enum differing only by a `_` keep distinct ids (#5137); and Java members differing only by a `_` or only by case keep distinct ids (#5151); so cached C++, PHP, Java and module-level-id extractions re-run; prior (43): A dotted Python subscript annotation (`mod.MyBox[Foo]`) walks its value once as the type, not again as its own generic argument, so the false `generic_arg` reference no longer replaces the real `parameter_type`/`return_type` edge, a C++ plain class/struct forward folds into its unique exactly-qualified definition when one file reaches the other through unconditional exact quoted includes, and a parameterless member declaration folds into its proven out-of-line definition by matching return type/qualifiers, with per-node `_cpp_member_records`/`_cpp_class_kind` facts persisted (#5117); a C++ member receiver resolves through a scoped type alias (`using Alias = ns::Type;`) and through inherited fields/methods, with forward-declared pointee types completed at the use site via a lexical-scope lookup, fail-closed on ambiguous/virtual/diamond bases (#5118); and a C++ out-of-line template member keeps its proven primary-template owner while calls in a `#if`-guarded initializer or argument list are recovered from byte-preserving branch views, with dependent receivers left unresolved (#5115); so cached Python and C++ extractions re-run; prior (42): A bare, unqualified free-function mention in a Markdown document (`token` in prose, no `()` and no namespace) binds to a same-named function only when the defining file, or another symbol from it, is cited within the heading's section chain, instead of matching any same-named callable anywhere in the corpus; reference-style links (`[text][label]`) resolve at the use site (#4804, #4806); and a C# classic extension-method call (`x.Foo()` where `Foo` is a `static ... Foo(this X x, ...)`) binds to the extension's declaration through a post-nominal additive pass, gated to a visible namespace, a matching `this` type and a unique applicable method, with a BCL deny-list and fail-closed on ambiguity or an instance-method shadow (#4828); so cached Markdown and C# extractions re-run; prior (41): A C++ member-call chain routed through class fields declared in other headers (`client.actions->Execute()`, with value `.`, raw-pointer `->` and single-arg `unique_ptr`/`shared_ptr` receivers) resolves to the field type's member across the include set, gated to a unique type and member and fail-closed otherwise, and class fields are persisted on the node (#5098); and a Markdown code span naming an uppercase `\`-namespace class (`\Foo\Bar`, `App\API`) or a type-qualified method call (`Type::member()`) is kept as a symbol mention instead of being dropped, while a non-call `Type::CONST`/`Type::class` stays dropped (#5093); so cached C++ and Markdown extractions re-run; prior (40): Solidity members, free functions and types whose names differ only by a leading `_` (or by case) in one scope (`_grantRole(bytes32,address)` / `grantRole(bytes32,address)`) get separate node ids: the one public name keeps its id and the others are salted, as in Python and C# (#3302, #4415), so the `_` function no longer vanishes and its calls stop landing on its twin (#5027); a Solidity call on a receiver of a contract or interface type (`IERC20 token; token.transfer()`, `IERC20(a).transfer()`, `token().balanceOf()`) binds by name and arity through that type's linearization to the type's own member instead of producing no edge, gated to a single visible contract/interface and fail-closed on an unknown or ambiguous type (#5060); a Solidity call attached by a `using A for T` directive (`using SafeCast for uint256; x.toUint48()`) binds to the library or free function it attaches, matched by the receiver's type and an arity one greater than the call (the receiver), fail-closed when the type is unknown or two functions fit (#5062); a C# constructor (instance, static or primary) is its own node owned by its type through a `method` edge, its body and `: base(...)`/`: this(...)` calls credited to it, and `new X(...)`/`: base`/`: this` additively link the arity-correct overload (#5054); and a C# call inside a property/indexer accessor, expression body, operator or conversion is credited to that member (an indexer is a `this[]` node, an operator a `.operator +()` node) instead of being lost (#5056), so cached Solidity and C# extractions re-run; prior (39): A Markdown code span written as a backslash namespace (`\Foo\Bar`) or a type-qualified member (`Type::member`) is recognised as a symbol mention and emits a `ns_qualified` raw_call (#4610, #4695), so cached Markdown extractions re-run; prior (38): C# preprocessor directives and `#if` branches that would derail tree-sitter-c-sharp are blanked before parsing (byte-length preserving, so lines and columns do not move), so a directive no longer closes the enclosing class early and drops later members or calls (#5003); a Solidity library call (`Lib.f()`, including an aliased import), and an `emit`/`revert`/`require` naming a declared event or error, link to the member they name instead of producing no edge, resolved only when a single visible type and a unique name/arity member match (#5001, #5005); and a YAML `<<` merge sequence gives the earlier anchor precedence for a duplicated key, matching the merge spec and PyYAML, so the winning key carries its correct source line (#4995); so cached C#, Solidity and YAML extractions re-run; prior (37): Solidity `import "../x/Y.sol"` resolves the `..` to the real file instead of a dangling stub that kept the importer's own folder (#4985); a Python module-level singleton (`g = Service()` then `g.method()`) resolves its member calls through the instance's type, gated by an exact-spelling and lexical-shadow guard (#4972); Kotlin resolves a call to an overloaded method and attributes calls inside an `init {}` block to the class, and a bare call no longer binds to a class the caller cannot reach (#4935, #4956); a Go same-file member call resolves through the receiver's type instead of a declaration-order-dependent same-named method (#4962, #4966); a Go member call on a receiver typed from another package (`pkg.T`) binds only to the `T` declared at the path the file imports `pkg` from (#3368, #4853); and a Go interface method call additively dispatches to the interface's implicit implementers (#4934), so cached Solidity, Python, Kotlin and Go extractions re-run; prior (36): C++ extraction changes: a type-alias reference (`using Foo = Bar;` then `Foo x;`) binds by exact spelling so it no longer loses to a same-normalized function, and a salted alias keeps its own `contains` edge (#4951); a member call `x.m()` binds by the receiver's type, now typed from parameters and class fields and not just locals, with untyped receivers kept bound in-file (#3215, #4946); an out-of-line/split-header definition attaches to the unique owner whose class file is in its include set via a `_cpp_qualified` identity marker, instead of file-containing or cross-binding to an unrelated same-named function (#4916, #4920); and a local callable value (a lambda or `std::function` local) is kept out of global call resolution by byte-range binding lifetimes so it no longer mints a spurious INFERRED call to a same-named global (#4917, #4921), so cached C++ extractions re-run; prior (35): A Rust module-qualified call like `crate::b::helper()` resolves to the function it names via qualifier-verified cross-file resolution, kept inside the caller's crate and out of local names (#2135, #4944); a Rust call written inside an expression macro's arguments (`println!`, `vec!`, ...) is linked, with a parsed call winning the dedupe (#4945); and cache keys keep their written case (file I/O through os_long_path, cache identity via normpath not normcase, and the digest salt no longer lowercased) so README.md and readme.md stop sharing one digest on a case-sensitive filesystem, which also changes the salt digest for any relative path carrying an uppercase char (#4925, #4937), so cached Rust extractions and any uppercase-path cache entries re-run; prior (34): A bare call inside a Python method no longer self-binds to a same-named method, so it resolves to the imported/module function instead (#4933); a Go method value reached through a receiver field is linked as an indirect_call (#4931); and a Go package-level `var` initialiser references the types it names (#4932), so cached Python and Go extractions re-run; prior (33): A Scala `extension (x: T)` method is scoped to its receiver type T (#2044, #4856); and a Terraform node carries its original scope so ids stay portable when the cached project is moved to a new directory (#4858, #4861), so cached Scala and Terraform extractions re-run; prior (32): A type and a function whose names differ only in case get separate node ids (#4904, #4908); and a PHP `Scope::MEMBER` access in a class-level initializer (const value, property/parameter default) emits a references_constant edge (#4648, #4895); R `setClass`/`setRefClass` `contains=` superclasses get inherits edges (#4902); an Elixir bare call resolves to the caller's own module first (#4900); a Dart mixin `on A, B` clause resolves every constraint (#4899); and a Fortran derived-type `extends` emits an inherits edge (#4897), so cached extractions for those languages re-run (#4904); Go credits a call in a package-level `var` initialiser (and its closures) to the file node (#4867); a Go method value passed as an argument is linked as an `indirect_call` (#4868); a Go receiver's field whose struct is declared in a sibling file of the same package is typed so its member calls resolve (#4866); and a Python dispatch table's `module.func` values resolve to the referenced functions as `indirect_call`s (#4773), so cached Go and Python extractions re-run; prior (30): File and per-declaration ids are kept distinct so a top-level declaration sharing the file's name is not dropped (#4854); a SystemVerilog `function new(...)` constructor gets its own node, `method` edge and parameter refs (#4848); a Go `var x = T{}`/`var x = &T{}` types `x` as `T` for receiver resolution like `x := T{}` (#4843); a PowerShell `using module`/`using assembly` name drops its file extension (#4847); a Solidity free-function `using {a, b} for T` set emits `uses` edges (#4846); a CommonLisp forward-referenced superclass binds to its local node instead of a sourceless stub (#4845); and Pascal `inherited`/`inline` form component blocks are recognised so later `object` blocks keep their parent (#4844), so cached extractions across those languages re-run; prior (29): OCaml `and`-chained module/class bindings are all walked (only the first was, so later bindings were silently dropped, #4822), and a SystemVerilog class `task` method is modeled like a `function` one with its own node and parameter refs while its task-locals no longer leak as class fields (#4774), so cached OCaml and Verilog/SystemVerilog extractions re-run; prior (28): Markdown ATX headings parse per CommonMark (closing `#` runs dropped, 0-3 space indent, hash-only lines mint no node, #4790) and fenced code blocks are tracked by CommonMark rules so headings inside fences stop leaking (#4787); YAML keys carry their written spelling and real line and `on`/`off`/`yes`/`no` keys no longer fold to booleans (#4784); rationale labels redact more credential shapes before caching (#4647); Razor `@implements` interface edges are extracted (#4820); aliased Terraform providers get their own node and `provider =` refs resolve (#4819); COBOL paragraphs after a nested `END PROGRAM` reattach to the enclosing program (#4824); Blade `wire:click` binds the bare method name (#4823); a Pascal record is a type with its own methods (#4825); VB.NET operator declarations are extracted as members with their bodies walked (#4821); and a Fortran call resolves to the subroutine it names (#4795), so cached Markdown/YAML/rationale/Razor/Terraform/COBOL/Blade/Pascal/VB.NET/Fortran extractions re-run; prior (27): VB.NET external namespace/type stubs now carry a `source_file` sentinel (#4725); Python member calls resolve through injected fields and method-result types (#4726); a Rust `use` is scoped to the crate it names (#4699), so cached VB.NET/Python/Rust extractions re-run; prior (26): Zig no longer crosses a call between a method and a same-named free function (#4755), and a Dart unnamed `extension on T` is detected (#4741), so cached Zig and Dart extractions re-run; prior (25): Python/Go/C++/Rust explicit type aliases (and Rust `union` items) are extracted as type nodes so references resolve instead of minting sourceless stubs (#4578/#4579/#4580/#4581); each C# method overload gets its own node bound by arity (#4584); a package.json dependency's manifest entry links to its ref node (#4569), so cached extractions re-run; prior (24): Objective-C `NS_CLOSED_ENUM` declarations are extracted as enums like `NS_ENUM` (#4705), and a `CREATE TABLE` whose column default the SQL grammar rejects (MySQL `DEFAULT current_timestamp()`) is recovered from its ERROR node (#4613), so cached ObjC and SQL extractions re-run; prior (23): `.yaml`/`.yml` files are structurally extracted (keys as nodes, config-vs-data gated) and reclassified from document to code (#2214), so cached YAML re-extracts; prior (22): SQL labels drop T-SQL brackets and routine bodies emit `reads_from`/`writes_to` (#4635), and a `GO` batch separator ends a statement (#4651), so cached `.sql` extractions re-run; prior (21): a CommonJS `const X = require('./m')` binding no longer mints a module-scope node that shadows the class the required file exports, so `new X()` resolves to the required declaration (#4621); prior (20): Obsidian `[[wikilink]]` targets that start with the scan root's own folder names now resolve to the real page (#4559), so cached markdown extractions re-run; prior (19): Dart `typedef` declarations are extracted as aliases (not variables of type `typedef`); C++ Metal/CUDA GPU qualifiers no longer mint sourceless god nodes (#4550); a `.h` header containing `extern "C"` routes to the C++ extractor; prior (18): Kotlin `typealias` (#4553), C# `delegate` (#4552), and Swift `macro` declarations (#4551) are now extracted so references stop minting sourceless stubs; Rust `use` rooted at a `mod` declared in the same scope carries `_rust_use_name` (#4564); prior (17): Python pytest-fixture markers (`_py_fixture`/`_python_fixture_param`, #4332) and module-level class-id collision salting (#2810); TS `this.field` typed from any method incl. static (#4473); Solidity callables carry `metadata.arity` + inherited-call/`super`/modifier raw calls (#4379); Go type-decl and Rust file-item case-distinct node ids (#4444); prior (16): Rust `use` edges carry the imported name as written (`_rust_use_name`, #4439); (15): Go generic receiver types normalized (`Box[T]` -> `Box`) so methods attach to the type node; prior (14): C# receiver resolution for nameless/`new X()`/`foreach` receivers (defer + receiver-chain markers) and `_`/case-distinct node ids (underscore-salted); prior (13): Kotlin/Scala/C++/Swift generic & template type params no longer mint sourceless stubs; Objective-C NS_ENUM/NS_OPTIONS extraction and `_callable`/`_callable_class` markers; Go and PHP receiver-typed member calls park cross-repo (`go_type_table`/`php_type_table`, receiver fields); prior (12): Java inherited-supertype (`receiver_unbound`) and chained/new/cast receiver typing (`java_receiver_chain`, `_java_ret`); C/C++ quoted cross-directory include provenance (`include_kind`); JS external imports anchored to the package, not the path leaf; prior (11): Python property-accessor decorators / `Self`, Swift `Self`, Go generic type params, Python forward type-hint file node, PowerShell dot-source, C# verbatim-interpolated string recovery; (10): C# partial types carry `type_path`; (9): Python property-read raw calls and `_py_property` nodes; (8): Rust `use`-group per-binding edges; (7): Python opaque-base/super(args) markers + nested-class enclosing ids. +# Schema 45 invalidates upstream schema-44 `.m` entries from before MATLAB dispatch provenance. +_AST_CACHE_SCHEMA = 45 # A folded C++ parameterless member records its `definition_file` anchored to the source root instead of a cache-root-relative `../`-walk; a PHP route closure takes its path only from the arguments before it, so a string argument after the closure no longer becomes the route's path or leaks into an enclosing group's prefix (#5135); two module-level declarations in one file differing only by a leading/trailing `_` keep distinct ids (the first keeps the plain id, the rest are salted) across JS, TS, C, C++, PHP, Ruby, Kotlin, Swift and Lua (#2810); PHP methods of one class/trait/interface/enum differing only by a `_` keep distinct ids (#5137); and Java members differing only by a `_` or only by case keep distinct ids (#5151); so cached C++, PHP, Java and module-level-id extractions re-run; prior (43): A dotted Python subscript annotation (`mod.MyBox[Foo]`) walks its value once as the type, not again as its own generic argument, so the false `generic_arg` reference no longer replaces the real `parameter_type`/`return_type` edge, a C++ plain class/struct forward folds into its unique exactly-qualified definition when one file reaches the other through unconditional exact quoted includes, and a parameterless member declaration folds into its proven out-of-line definition by matching return type/qualifiers, with per-node `_cpp_member_records`/`_cpp_class_kind` facts persisted (#5117); a C++ member receiver resolves through a scoped type alias (`using Alias = ns::Type;`) and through inherited fields/methods, with forward-declared pointee types completed at the use site via a lexical-scope lookup, fail-closed on ambiguous/virtual/diamond bases (#5118); and a C++ out-of-line template member keeps its proven primary-template owner while calls in a `#if`-guarded initializer or argument list are recovered from byte-preserving branch views, with dependent receivers left unresolved (#5115); so cached Python and C++ extractions re-run; prior (42): A bare, unqualified free-function mention in a Markdown document (`token` in prose, no `()` and no namespace) binds to a same-named function only when the defining file, or another symbol from it, is cited within the heading's section chain, instead of matching any same-named callable anywhere in the corpus; reference-style links (`[text][label]`) resolve at the use site (#4804, #4806); and a C# classic extension-method call (`x.Foo()` where `Foo` is a `static ... Foo(this X x, ...)`) binds to the extension's declaration through a post-nominal additive pass, gated to a visible namespace, a matching `this` type and a unique applicable method, with a BCL deny-list and fail-closed on ambiguity or an instance-method shadow (#4828); so cached Markdown and C# extractions re-run; prior (41): A C++ member-call chain routed through class fields declared in other headers (`client.actions->Execute()`, with value `.`, raw-pointer `->` and single-arg `unique_ptr`/`shared_ptr` receivers) resolves to the field type's member across the include set, gated to a unique type and member and fail-closed otherwise, and class fields are persisted on the node (#5098); and a Markdown code span naming an uppercase `\`-namespace class (`\Foo\Bar`, `App\API`) or a type-qualified method call (`Type::member()`) is kept as a symbol mention instead of being dropped, while a non-call `Type::CONST`/`Type::class` stays dropped (#5093); so cached C++ and Markdown extractions re-run; prior (40): Solidity members, free functions and types whose names differ only by a leading `_` (or by case) in one scope (`_grantRole(bytes32,address)` / `grantRole(bytes32,address)`) get separate node ids: the one public name keeps its id and the others are salted, as in Python and C# (#3302, #4415), so the `_` function no longer vanishes and its calls stop landing on its twin (#5027); a Solidity call on a receiver of a contract or interface type (`IERC20 token; token.transfer()`, `IERC20(a).transfer()`, `token().balanceOf()`) binds by name and arity through that type's linearization to the type's own member instead of producing no edge, gated to a single visible contract/interface and fail-closed on an unknown or ambiguous type (#5060); a Solidity call attached by a `using A for T` directive (`using SafeCast for uint256; x.toUint48()`) binds to the library or free function it attaches, matched by the receiver's type and an arity one greater than the call (the receiver), fail-closed when the type is unknown or two functions fit (#5062); a C# constructor (instance, static or primary) is its own node owned by its type through a `method` edge, its body and `: base(...)`/`: this(...)` calls credited to it, and `new X(...)`/`: base`/`: this` additively link the arity-correct overload (#5054); and a C# call inside a property/indexer accessor, expression body, operator or conversion is credited to that member (an indexer is a `this[]` node, an operator a `.operator +()` node) instead of being lost (#5056), so cached Solidity and C# extractions re-run; prior (39): A Markdown code span written as a backslash namespace (`\Foo\Bar`) or a type-qualified member (`Type::member`) is recognised as a symbol mention and emits a `ns_qualified` raw_call (#4610, #4695), so cached Markdown extractions re-run; prior (38): C# preprocessor directives and `#if` branches that would derail tree-sitter-c-sharp are blanked before parsing (byte-length preserving, so lines and columns do not move), so a directive no longer closes the enclosing class early and drops later members or calls (#5003); a Solidity library call (`Lib.f()`, including an aliased import), and an `emit`/`revert`/`require` naming a declared event or error, link to the member they name instead of producing no edge, resolved only when a single visible type and a unique name/arity member match (#5001, #5005); and a YAML `<<` merge sequence gives the earlier anchor precedence for a duplicated key, matching the merge spec and PyYAML, so the winning key carries its correct source line (#4995); so cached C#, Solidity and YAML extractions re-run; prior (37): Solidity `import "../x/Y.sol"` resolves the `..` to the real file instead of a dangling stub that kept the importer's own folder (#4985); a Python module-level singleton (`g = Service()` then `g.method()`) resolves its member calls through the instance's type, gated by an exact-spelling and lexical-shadow guard (#4972); Kotlin resolves a call to an overloaded method and attributes calls inside an `init {}` block to the class, and a bare call no longer binds to a class the caller cannot reach (#4935, #4956); a Go same-file member call resolves through the receiver's type instead of a declaration-order-dependent same-named method (#4962, #4966); a Go member call on a receiver typed from another package (`pkg.T`) binds only to the `T` declared at the path the file imports `pkg` from (#3368, #4853); and a Go interface method call additively dispatches to the interface's implicit implementers (#4934), so cached Solidity, Python, Kotlin and Go extractions re-run; prior (36): C++ extraction changes: a type-alias reference (`using Foo = Bar;` then `Foo x;`) binds by exact spelling so it no longer loses to a same-normalized function, and a salted alias keeps its own `contains` edge (#4951); a member call `x.m()` binds by the receiver's type, now typed from parameters and class fields and not just locals, with untyped receivers kept bound in-file (#3215, #4946); an out-of-line/split-header definition attaches to the unique owner whose class file is in its include set via a `_cpp_qualified` identity marker, instead of file-containing or cross-binding to an unrelated same-named function (#4916, #4920); and a local callable value (a lambda or `std::function` local) is kept out of global call resolution by byte-range binding lifetimes so it no longer mints a spurious INFERRED call to a same-named global (#4917, #4921), so cached C++ extractions re-run; prior (35): A Rust module-qualified call like `crate::b::helper()` resolves to the function it names via qualifier-verified cross-file resolution, kept inside the caller's crate and out of local names (#2135, #4944); a Rust call written inside an expression macro's arguments (`println!`, `vec!`, ...) is linked, with a parsed call winning the dedupe (#4945); and cache keys keep their written case (file I/O through os_long_path, cache identity via normpath not normcase, and the digest salt no longer lowercased) so README.md and readme.md stop sharing one digest on a case-sensitive filesystem, which also changes the salt digest for any relative path carrying an uppercase char (#4925, #4937), so cached Rust extractions and any uppercase-path cache entries re-run; prior (34): A bare call inside a Python method no longer self-binds to a same-named method, so it resolves to the imported/module function instead (#4933); a Go method value reached through a receiver field is linked as an indirect_call (#4931); and a Go package-level `var` initialiser references the types it names (#4932), so cached Python and Go extractions re-run; prior (33): A Scala `extension (x: T)` method is scoped to its receiver type T (#2044, #4856); and a Terraform node carries its original scope so ids stay portable when the cached project is moved to a new directory (#4858, #4861), so cached Scala and Terraform extractions re-run; prior (32): A type and a function whose names differ only in case get separate node ids (#4904, #4908); and a PHP `Scope::MEMBER` access in a class-level initializer (const value, property/parameter default) emits a references_constant edge (#4648, #4895); R `setClass`/`setRefClass` `contains=` superclasses get inherits edges (#4902); an Elixir bare call resolves to the caller's own module first (#4900); a Dart mixin `on A, B` clause resolves every constraint (#4899); and a Fortran derived-type `extends` emits an inherits edge (#4897), so cached extractions for those languages re-run (#4904); Go credits a call in a package-level `var` initialiser (and its closures) to the file node (#4867); a Go method value passed as an argument is linked as an `indirect_call` (#4868); a Go receiver's field whose struct is declared in a sibling file of the same package is typed so its member calls resolve (#4866); and a Python dispatch table's `module.func` values resolve to the referenced functions as `indirect_call`s (#4773), so cached Go and Python extractions re-run; prior (30): File and per-declaration ids are kept distinct so a top-level declaration sharing the file's name is not dropped (#4854); a SystemVerilog `function new(...)` constructor gets its own node, `method` edge and parameter refs (#4848); a Go `var x = T{}`/`var x = &T{}` types `x` as `T` for receiver resolution like `x := T{}` (#4843); a PowerShell `using module`/`using assembly` name drops its file extension (#4847); a Solidity free-function `using {a, b} for T` set emits `uses` edges (#4846); a CommonLisp forward-referenced superclass binds to its local node instead of a sourceless stub (#4845); and Pascal `inherited`/`inline` form component blocks are recognised so later `object` blocks keep their parent (#4844), so cached extractions across those languages re-run; prior (29): OCaml `and`-chained module/class bindings are all walked (only the first was, so later bindings were silently dropped, #4822), and a SystemVerilog class `task` method is modeled like a `function` one with its own node and parameter refs while its task-locals no longer leak as class fields (#4774), so cached OCaml and Verilog/SystemVerilog extractions re-run; prior (28): Markdown ATX headings parse per CommonMark (closing `#` runs dropped, 0-3 space indent, hash-only lines mint no node, #4790) and fenced code blocks are tracked by CommonMark rules so headings inside fences stop leaking (#4787); YAML keys carry their written spelling and real line and `on`/`off`/`yes`/`no` keys no longer fold to booleans (#4784); rationale labels redact more credential shapes before caching (#4647); Razor `@implements` interface edges are extracted (#4820); aliased Terraform providers get their own node and `provider =` refs resolve (#4819); COBOL paragraphs after a nested `END PROGRAM` reattach to the enclosing program (#4824); Blade `wire:click` binds the bare method name (#4823); a Pascal record is a type with its own methods (#4825); VB.NET operator declarations are extracted as members with their bodies walked (#4821); and a Fortran call resolves to the subroutine it names (#4795), so cached Markdown/YAML/rationale/Razor/Terraform/COBOL/Blade/Pascal/VB.NET/Fortran extractions re-run; prior (27): VB.NET external namespace/type stubs now carry a `source_file` sentinel (#4725); Python member calls resolve through injected fields and method-result types (#4726); a Rust `use` is scoped to the crate it names (#4699), so cached VB.NET/Python/Rust extractions re-run; prior (26): Zig no longer crosses a call between a method and a same-named free function (#4755), and a Dart unnamed `extension on T` is detected (#4741), so cached Zig and Dart extractions re-run; prior (25): Python/Go/C++/Rust explicit type aliases (and Rust `union` items) are extracted as type nodes so references resolve instead of minting sourceless stubs (#4578/#4579/#4580/#4581); each C# method overload gets its own node bound by arity (#4584); a package.json dependency's manifest entry links to its ref node (#4569), so cached extractions re-run; prior (24): Objective-C `NS_CLOSED_ENUM` declarations are extracted as enums like `NS_ENUM` (#4705), and a `CREATE TABLE` whose column default the SQL grammar rejects (MySQL `DEFAULT current_timestamp()`) is recovered from its ERROR node (#4613), so cached ObjC and SQL extractions re-run; prior (23): `.yaml`/`.yml` files are structurally extracted (keys as nodes, config-vs-data gated) and reclassified from document to code (#2214), so cached YAML re-extracts; prior (22): SQL labels drop T-SQL brackets and routine bodies emit `reads_from`/`writes_to` (#4635), and a `GO` batch separator ends a statement (#4651), so cached `.sql` extractions re-run; prior (21): a CommonJS `const X = require('./m')` binding no longer mints a module-scope node that shadows the class the required file exports, so `new X()` resolves to the required declaration (#4621); prior (20): Obsidian `[[wikilink]]` targets that start with the scan root's own folder names now resolve to the real page (#4559), so cached markdown extractions re-run; prior (19): Dart `typedef` declarations are extracted as aliases (not variables of type `typedef`); C++ Metal/CUDA GPU qualifiers no longer mint sourceless god nodes (#4550); a `.h` header containing `extern "C"` routes to the C++ extractor; prior (18): Kotlin `typealias` (#4553), C# `delegate` (#4552), and Swift `macro` declarations (#4551) are now extracted so references stop minting sourceless stubs; Rust `use` rooted at a `mod` declared in the same scope carries `_rust_use_name` (#4564); prior (17): Python pytest-fixture markers (`_py_fixture`/`_python_fixture_param`, #4332) and module-level class-id collision salting (#2810); TS `this.field` typed from any method incl. static (#4473); Solidity callables carry `metadata.arity` + inherited-call/`super`/modifier raw calls (#4379); Go type-decl and Rust file-item case-distinct node ids (#4444); prior (16): Rust `use` edges carry the imported name as written (`_rust_use_name`, #4439); (15): Go generic receiver types normalized (`Box[T]` -> `Box`) so methods attach to the type node; prior (14): C# receiver resolution for nameless/`new X()`/`foreach` receivers (defer + receiver-chain markers) and `_`/case-distinct node ids (underscore-salted); prior (13): Kotlin/Scala/C++/Swift generic & template type params no longer mint sourceless stubs; Objective-C NS_ENUM/NS_OPTIONS extraction and `_callable`/`_callable_class` markers; Go and PHP receiver-typed member calls park cross-repo (`go_type_table`/`php_type_table`, receiver fields); prior (12): Java inherited-supertype (`receiver_unbound`) and chained/new/cast receiver typing (`java_receiver_chain`, `_java_ret`); C/C++ quoted cross-directory include provenance (`include_kind`); JS external imports anchored to the package, not the path leaf; prior (11): Python property-accessor decorators / `Self`, Swift `Self`, Go generic type params, Python forward type-hint file node, PowerShell dot-source, C# verbatim-interpolated string recovery; (10): C# partial types carry `type_path`; (9): Python property-read raw calls and `_py_property` nodes; (8): Rust `use`-group per-binding edges; (7): Python opaque-base/super(args) markers + nested-class enclosing ids. # Per-process memo: distributions() scans site-packages metadata, which is diff --git a/graphify/cli.py b/graphify/cli.py index c08b21f668..5db2c5e42d 100644 --- a/graphify/cli.py +++ b/graphify/cli.py @@ -1073,6 +1073,10 @@ def _prune_graph_json_sources(graph_path: Path, stale_sources: list[str]) -> int data[links_key] = kept_edges if "hyperedges" in data: data["hyperedges"] = kept_hyper + from graphify.build import _refresh_matlab_inheritance, dedupe_nodes + data[links_key] = _refresh_matlab_inheritance( + dedupe_nodes(kept_nodes), kept_edges + ) from graphify.export import backup_if_protected as _backup _backup(graph_path.parent) from graphify.paths import write_json_atomic diff --git a/graphify/cross_repo_calls.py b/graphify/cross_repo_calls.py index 9ce7b9191a..7377427af4 100644 --- a/graphify/cross_repo_calls.py +++ b/graphify/cross_repo_calls.py @@ -113,6 +113,9 @@ def _index_declarations(merged: "nx.Graph") -> tuple[dict[str, list[str]], set[s for node, data in merged.nodes(data=True): if not data.get("_callable_class") or not data.get("source_file"): continue + # MATLAB shares .m with Objective-C, but cannot answer native parked calls. + if (data.get("metadata") or {}).get("language") == "matlab": + continue if not data.get("repo"): continue name = _key(data.get("label")) diff --git a/graphify/cross_repo_types.py b/graphify/cross_repo_types.py index 3cd9ea0124..cf2434242a 100644 --- a/graphify/cross_repo_types.py +++ b/graphify/cross_repo_types.py @@ -41,6 +41,9 @@ def link_shared_type_declarations(merged: "nx.Graph") -> int: for node, data in merged.nodes(data=True): if not data.get("_callable_class") or not data.get("source_file"): continue + # MATLAB package names do not establish shared types across repositories. + if (data.get("metadata") or {}).get("language") == "matlab": + continue namespace = str((data.get("metadata") or {}).get("namespace") or "") label = str(data.get("label") or "") if not namespace or not label or not data.get("repo"): diff --git a/graphify/extract.py b/graphify/extract.py index e17d87cf36..6ffb79ba56 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -76,6 +76,7 @@ from graphify.extractors.yaml_config import extract_yaml # noqa: F401 from graphify.extractors.commonlisp import extract_commonlisp # noqa: F401 from graphify.extractors.markdown import extract_markdown, _MD_LINK_INDEX_CACHE # noqa: F401 +from graphify.extractors.matlab import extract_matlab # noqa: F401 from graphify.extractors.ocaml import extract_ocaml # noqa: F401 from graphify.extractors.pascal_forms import extract_delphi_form, extract_lazarus_form # noqa: F401 from graphify.extractors.powershell import extract_powershell, extract_powershell_manifest # noqa: F401 @@ -140,6 +141,7 @@ _js_top_level_function_bodies, _load_tsconfig_aliases, _load_tsconfig_base_url, + _matlab_inheritance_edges, _load_workspace_packages, _match_tsconfig_alias, _merge_decl_def_classes, @@ -4716,6 +4718,11 @@ def _node_label_key(node: dict, fold: bool = False) -> str: return key.lower() if fold else key +def _is_matlab_node(node: dict) -> bool: + """Use explicit provenance; `.m` also denotes native Objective-C sources.""" + return (node.get("metadata") or {}).get("language") == "matlab" + + def _is_top_level_function_definition(node: dict) -> bool: """A free/top-level function def (label ``name()``), not a method or type. @@ -4834,8 +4841,11 @@ def keep(crates: set[str]) -> list[dict]: real_by_label_ci: dict[str, list[dict]] = {} # case-INSENSITIVE-language reals only func_by_label: dict[str, list[dict]] = {} # top-level function defs (#1781) stubs: list[dict] = [] + matlab_ids = {node.get("id") for node in nodes if _is_matlab_node(node)} for node in nodes: + if node.get("id") in matlab_ids: + continue key = _node_label_key(node) if not key: continue @@ -4864,6 +4874,8 @@ def keep(crates: set[str]) -> list[dict]: supertype_stub_ids: set[str] = set() # stubs used as a base type — never a function _SUPERTYPE_RELATIONS = {"inherits", "implements", "extends"} for edge in edges: + if edge.get("source") in matlab_ids: + continue rel = edge.get("relation") for endpoint in ("source", "target"): nid = edge.get(endpoint) @@ -5001,6 +5013,8 @@ def _scoped_by_a_use(stub: dict) -> bool: } - {edge.get("target") for edge in edges if edge.get("_rust_use_name")} for edge in edges: + if edge.get("source") in matlab_ids: + continue is_csharp_scoped_edge = ( _is_dotnet_source_file(str(edge.get("source_file", ""))) and edge.get("relation") in csharp_scoped_relations @@ -5095,10 +5109,18 @@ def _merge_swift_extensions( # bare-reference shadow nodes (`let x: Foo`) carry a source_file but are NOT # contained, so excluding them keeps a stub from making a real type look # ambiguous — same predicate the Swift member-call resolver uses (#2538). - contained = {e.get("target") for e in all_edges if e.get("relation") == "contains"} + matlab_ids = {n.get("id") for n in all_nodes if _is_matlab_node(n)} + contained = { + e.get("target") for e in all_edges + if e.get("relation") == "contains" + and e.get("source") not in matlab_ids + and e.get("target") not in matlab_ids + } label_to_canonical: dict[str, list[str]] = {} for n in all_nodes: + if _is_matlab_node(n): + continue if n.get("id") in extension_nids: continue label = n.get("label") @@ -5519,13 +5541,21 @@ def _key(label: str) -> str: # Bare type references create a same-label shadow node (via ensure_named_node) # that carries a source_file but is NOT contained; excluding non-contained # nodes keeps that shadow from making a real type name look ambiguous. - contained = {e.get("target") for e in all_edges if e.get("relation") == "contains"} + matlab_ids = {n.get("id") for n in all_nodes if _is_matlab_node(n)} + contained = { + e.get("target") for e in all_edges + if e.get("relation") == "contains" + and e.get("source") not in matlab_ids + and e.get("target") not in matlab_ids + } # Type name -> definition node ids (real, source-backed, type-like defs only). # len != 1 is the god-node guard: an ambiguous type name bails. type_def_nids: dict[str, list[str]] = {} node_by_id: dict[str, dict] = {} for n in all_nodes: + if _is_matlab_node(n): + continue node_by_id[n.get("id")] = n if n.get("source_file") and n.get("id") in contained and _is_type_like_definition(n): type_def_nids.setdefault(_key(n.get("label", "")), []).append(n["id"]) @@ -5533,7 +5563,7 @@ def _key(label: str) -> str: # (type_node_id, method_key) -> method_node_id, from `method` edges. method_index: dict[tuple[str, str], str] = {} for e in all_edges: - if e.get("relation") != "method": + if e.get("relation") != "method" or e.get("source") in matlab_ids: continue src, tgt = e.get("source"), e.get("target") tnode = node_by_id.get(tgt) @@ -5556,9 +5586,13 @@ def _key(label: str) -> str: # method nid -> marked plain-return target nids (must be exactly one). return_targets_by_method: dict[str, set[str]] = {} for e in all_edges: - if (e.get("relation") == "references" - and e.get("context") == "return_type" - and (e.get("metadata") or {}).get("swift_plain_return")): + if ( + e.get("relation") == "references" + and e.get("context") == "return_type" + and (e.get("metadata") or {}).get("swift_plain_return") + and e.get("source") not in matlab_ids + and e.get("target") not in matlab_ids + ): return_targets_by_method.setdefault( e.get("source"), set()).add(e.get("target")) for path, pending in factory_by_file.items(): @@ -6522,13 +6556,19 @@ def _key(label: str) -> str: return re.sub(r"[^a-zA-Z0-9]+", "", str(label)).lower() # A genuine C++ type is the target of a `contains` edge from its file node; - # bare-reference shadow nodes (ensure_named_node stubs) are not contained, so - # excluding non-contained nodes keeps them from making a real type ambiguous. - contained = {e.get("target") for e in all_edges if e.get("relation") == "contains"} + matlab_ids = {n.get("id") for n in all_nodes if _is_matlab_node(n)} + contained = { + e.get("target") for e in all_edges + if e.get("relation") == "contains" + and e.get("source") not in matlab_ids + and e.get("target") not in matlab_ids + } type_def_nids: dict[str, list[str]] = {} node_by_id: dict[str, dict] = {} for n in all_nodes: + if _is_matlab_node(n): + continue node_by_id[n.get("id")] = n if n.get("source_file") and n.get("id") in contained and _is_type_like_definition(n): type_def_nids.setdefault(_key(n.get("label", "")), []).append(n["id"]) @@ -6542,7 +6582,7 @@ def _key(label: str) -> str: enclosing_type: dict[str, str] = {} for rel in ("defines", "method"): for e in all_edges: - if e.get("relation") != rel: + if e.get("relation") != rel or e.get("source") in matlab_ids: continue src, tgt = e.get("source"), e.get("target") tnode = node_by_id.get(tgt) @@ -6564,6 +6604,8 @@ def _key(label: str) -> str: # exactly-one-candidate. qualified_method_nids: dict[str, list[str]] = {} for n in all_nodes: + if _is_matlab_node(n): + continue label = str(n.get("label", "")) if n.get("source_file") and label.endswith("()") and "::" in label: qualified_method_nids.setdefault(label, []).append(n["id"]) @@ -7721,14 +7763,23 @@ def _resolve_java_member_calls( def key(label: str) -> str: return str(label).strip().removeprefix(".").removesuffix("()") - contained = {edge.get("target") for edge in all_edges - if edge.get("relation") == "contains"} - node_by_id = {node.get("id"): node for node in all_nodes} + matlab_ids = {node.get("id") for node in all_nodes if _is_matlab_node(node)} + contained = { + edge.get("target") for edge in all_edges + if edge.get("relation") == "contains" + and edge.get("source") not in matlab_ids + and edge.get("target") not in matlab_ids + } + node_by_id = { + node.get("id"): node for node in all_nodes + if node.get("id") not in matlab_ids + } type_def_nids: dict[str, list[str]] = {} for node in all_nodes: if ( - node.get("source_file") + node.get("id") not in matlab_ids + and node.get("source_file") and node.get("id") in contained and _is_type_like_definition(node) ): @@ -7737,7 +7788,11 @@ def key(label: str) -> str: method_index: dict[tuple[str, str], set[str]] = {} enclosing_type: dict[str, str] = {} for edge in all_edges: - if edge.get("relation") != "method": + if ( + edge.get("relation") != "method" + or edge.get("source") in matlab_ids + or edge.get("target") in matlab_ids + ): continue owner, method = edge.get("source"), edge.get("target") method_node = node_by_id.get(method) @@ -7746,6 +7801,7 @@ def key(label: str) -> str: enclosing_type.setdefault(method, owner) method_index.setdefault((owner, key(method_node.get("label", ""))), set()).add(method) + existing_pairs = {(edge.get("source"), edge.get("target")) for edge in all_edges} # Inherited fields (#3151): a field declared on a superclass - possibly in # another file - types a `this.` receiver in the subclass. Safe @@ -8028,7 +8084,13 @@ def _resolve_objc_member_calls( def _key(label: str) -> str: return re.sub(r"[^a-zA-Z0-9]+", "", str(label)).lower() - contained = {e.get("target") for e in all_edges if e.get("relation") == "contains"} + matlab_ids = {n.get("id") for n in all_nodes if _is_matlab_node(n)} + contained = { + e.get("target") for e in all_edges + if e.get("relation") == "contains" + and e.get("source") not in matlab_ids + and e.get("target") not in matlab_ids + } def _is_protocol_declaration(n: dict) -> bool: """A ``@protocol`` declaration, which the ObjC extractor labels ````. @@ -8043,6 +8105,8 @@ def _is_protocol_declaration(n: dict) -> bool: type_def_nids: dict[str, list[str]] = {} node_by_id: dict[str, dict] = {} for n in all_nodes: + if _is_matlab_node(n): + continue node_by_id[n.get("id")] = n if (n.get("source_file") and n.get("id") in contained and _is_type_like_definition(n) and not _is_protocol_declaration(n)): @@ -8051,12 +8115,12 @@ def _is_protocol_declaration(n: dict) -> bool: method_index: dict[tuple[str, str], str] = {} enclosing_type: dict[str, str] = {} for e in all_edges: - if e.get("relation") != "method": + if e.get("relation") != "method" or e.get("source") in matlab_ids: continue src, tgt = e.get("source"), e.get("target") - enclosing_type.setdefault(tgt, src) tnode = node_by_id.get(tgt) if tnode is not None: + enclosing_type.setdefault(tgt, src) # ObjC method labels carry a +/- sigil (`-doThing`); strip it so the # selector `doThing` keys to the method. method_index[(src, _key(tnode.get("label", "")))] = tgt @@ -8070,7 +8134,9 @@ def _is_protocol_declaration(n: dict) -> bool: # subclass too (#3151): walk the inherits chain, nearest table first. _objc_bases: dict[str, list[str]] = {} for e in all_edges: - if e.get("relation") == "inherits": + if (e.get("relation") == "inherits" + and e.get("source") not in matlab_ids + and e.get("target") not in matlab_ids): _objc_bases.setdefault(e["source"], []).append(e["target"]) def _field_type_up_chain(cls, receiver): @@ -11002,13 +11068,11 @@ def _get_extractor(path: Path, *, sniff_headers: bool = True) -> Any | None: # grammar has no class_specifier). Reroute to extract_cpp (#1547). if _is_cpp_header(path, head): return extract_cpp - # `.m` is Objective-C OR MATLAB. extract_objc unconditionally would force-parse - # MATLAB through the ObjC grammar into garbage (#1702). Route to extract_objc - # only when the file actually looks like Objective-C; otherwise leave it without - # an extractor (surfaced by the no-AST-extractor warning, #1689) rather than - # mis-parsed. `.mm` is unambiguously Objective-C++ and stays on extract_objc. + # `.m` is Objective-C OR MATLAB. Explicit remaps above retain priority, and + # Objective-C `.m` files keep their current extractor; all other `.m` files + # use the MATLAB frontend. `.mm` remains unambiguously Objective-C++. if suffix == ".m" and not _is_objc_source(path): - return None + return extract_matlab # Extensionless files: resolve by shebang, mirroring detect.classify_file. # Without this, detect labels e.g. `#!/usr/bin/env bash` CLIs as code but # extraction returns no extractor and the file silently contributes nothing. @@ -12587,6 +12651,8 @@ def _looks_like_bash(result: object) -> bool: else: resolution_nodes = all_nodes for n in resolution_nodes: + if _is_matlab_node(n): + continue if n.get("file_type") == "rationale" or n.get("type") == "namespace": continue # A C# constructor is named like its type but never called by name. @@ -12610,11 +12676,17 @@ def _looks_like_bash(result: object) -> bool: # `resolution_nodes`, not `all_nodes` (#2438): an unchanged callee's context node # carries the marker persisted in graph.json, so an incremental rebuild keeps # resolving callbacks into unchanged files while data symbols stay excluded. - callable_nids = {n["id"] for n in resolution_nodes if n.get("_callable")} + callable_nids = { + n["id"] for n in resolution_nodes + if n.get("_callable") and not _is_matlab_node(n) + } # Class defs are callable only via their constructor; they are frequently passed # as descriptive values (`select(Model)`, exception tuples), not invoked. Exclude # them from the indirect_call guard below to avoid false edges (#2137). - class_nids = {n["id"] for n in resolution_nodes if n.get("_callable_class")} + class_nids = { + n["id"] for n in resolution_nodes + if n.get("_callable_class") and not _is_matlab_node(n) + } # JVM name resolution: Java and Kotlin share one package/type namespace. # Run before the Kotlin-only resolver and before call evidence is built. @@ -13431,6 +13503,38 @@ def _canon(nid: str) -> str: if _df and "\\" in str(_df): _item["definition_file"] = PurePath(_df).as_posix() + # Resolve MATLAB inheritance only after IDs and source paths are canonical. + # Unchanged context contributes class/file candidates, never output edges. + _matlab_kinds = {"matlab_file", "matlab_class"} + _fresh_matlab_nodes = [ + n for n in all_nodes + if isinstance(n.get("metadata"), dict) + and n["metadata"].get("language") == "matlab" + and n["metadata"].get("kind") in _matlab_kinds + ] + _fresh_matlab_class_ids = { + n["id"] for n in _fresh_matlab_nodes + if n["metadata"].get("kind") == "matlab_class" + and isinstance(n.get("id"), str) + } + if _fresh_matlab_class_ids: + _fresh_matlab_ids = {n.get("id") for n in _fresh_matlab_nodes} + _matlab_context_nodes = [ + n for n in resolution_context_nodes or [] + if isinstance(n.get("metadata"), dict) + and n["metadata"].get("language") == "matlab" + and n["metadata"].get("kind") in _matlab_kinds + and n.get("id") not in _fresh_matlab_ids + ] + _matlab_edges = [ + edge for edge in _matlab_inheritance_edges( + _fresh_matlab_nodes + _matlab_context_nodes + ) + if edge["source"] in _fresh_matlab_class_ids + ] + for edge in _matlab_edges: + edge["_origin"] = "ast" + all_edges.extend(_matlab_edges) _stamp_cpp_condition_provenance(all_nodes, all_edges) return { diff --git a/graphify/extractors/__init__.py b/graphify/extractors/__init__.py index e2c8afc627..1ffd76cd6f 100644 --- a/graphify/extractors/__init__.py +++ b/graphify/extractors/__init__.py @@ -23,6 +23,7 @@ from graphify.extractors.go import extract_go from graphify.extractors.json_config import extract_json from graphify.extractors.julia import extract_julia +from graphify.extractors.matlab import extract_matlab from graphify.extractors.markdown import extract_markdown from graphify.extractors.objc import extract_objc from graphify.extractors.pascal import extract_pascal @@ -57,6 +58,7 @@ "go": extract_go, "json": extract_json, "julia": extract_julia, + "matlab": extract_matlab, "lazarus_form": extract_lazarus_form, "markdown": extract_markdown, "objc": extract_objc, diff --git a/graphify/extractors/matlab.py b/graphify/extractors/matlab.py new file mode 100644 index 0000000000..62cd5528e7 --- /dev/null +++ b/graphify/extractors/matlab.py @@ -0,0 +1,443 @@ +"""Source-backed MATLAB definitions and conservative same-file lexical calls.""" +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +import re +from typing import Any + +from graphify.extractors.base import ( + _file_stem, + _make_id, + _read_source_bytes, + _read_text, + active_scan_root, +) + + +_DECLARATIONS = {"function_definition", "function_signature", "class_definition"} +_WORKSPACE_CHANGES = {"eval", "evalc", "evalin", "assignin", "load", "clear"} +_MATLAB_IDENTIFIER = re.compile(r"[^\W\d_]\w*", re.UNICODE) +_MATLAB_NAME = re.compile(r"[^\W\d_]\w*(?:\.[^\W\d_]\w*)*", re.UNICODE) + + +@dataclass +class _Scope: + syntax: Any + name: str + kind: str + owner: _Scope | None + record: dict | None = None + children: list[_Scope] = field(default_factory=list) + bindings: set[str] = field(default_factory=set) + unsafe: bool = False + primary: bool = False + + +def extract_matlab(path: Path) -> dict: + """Extract declarations and only statically justified same-file calls.""" + try: + import tree_sitter_matlab + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-matlab not installed"} + + try: + source = _read_source_bytes(path) + root = Parser(Language(tree_sitter_matlab.language())).parse(source).root_node + except Exception as exc: + return {"nodes": [], "edges": [], "error": str(exc)} + + scan_root = active_scan_root() + relative = path + if scan_root is not None: + try: + relative = path.resolve().relative_to(scan_root.resolve()) + except ValueError: + pass + stem = _file_stem(relative) + namespace_parts: list[str] = [] + for part in reversed(relative.parent.parts): + if not part.startswith("+") or not _MATLAB_IDENTIFIER.fullmatch(part[1:]): + break + namespace_parts.append(part[1:]) + namespace = ".".join(reversed(namespace_parts)) + nodes: list[dict] = [] + edges: list[dict] = [] + + def text(node) -> str: + return _read_text(node, source) if node is not None else "" + + def add_node(nid: str, name: str, syntax, kind: str, chain: list[str]) -> dict: + metadata = { + "language": "matlab", + "kind": f"matlab_{kind}", + "scope_chain": chain, + "namespace": namespace, + } + record = { + "id": nid, + "label": name, + "file_type": "code", + "source_file": str(path), + "source_location": f"L{syntax.start_point.row + 1}", + "metadata": metadata, + } + nodes.append(record) + return record + + def add_edge(source_id: str, target_id: str, relation: str, syntax) -> None: + edges.append({ + "source": source_id, + "target": target_id, + "relation": relation, + "confidence": "EXTRACTED", + "weight": 1.0, + "source_file": str(path), + "source_location": f"L{syntax.start_point.row + 1}", + }) + + file_scope = _Scope(root, "", "file", None) + file_scope.record = add_node( + _make_id(relative.as_posix()), path.name, root, "file", [] + ) + scopes = [file_scope] + + def exact_superclass_name(value: str) -> bool: + return bool(_MATLAB_NAME.fullmatch(value)) + + def superclass_facts(class_node) -> list[dict[str, str]]: + facts: list[dict[str, str]] = [] + groups = [child for child in class_node.named_children if child.type == "superclasses"] + for group in groups: + bases = [ + base for base in group.named_children + if base.type not in {"comment", "line_continuation"} + ] + if bases: + for base in bases: + spelling = text(base).strip() + parsed = ( + not group.has_error + and not base.has_error + and not base.is_missing + and base.type == "property_name" + and exact_superclass_name(spelling) + ) + facts.append({ + "name" if parsed else "raw": spelling, + "source_location": f"L{base.start_point.row + 1}", + }) + elif group.has_error: + raw = text(group).strip().lstrip("<").strip() + if raw: + facts.append({ + "raw": raw, + "source_location": f"L{group.start_point.row + 1}", + }) + if not groups: + for child in class_node.named_children: + if child.type != "ERROR" or "<" not in text(child): + continue + raw = text(child).split("<", 1)[1].strip() + if raw: + facts.append({ + "raw": raw, + "source_location": f"L{child.start_point.row + 1}", + }) + return facts + + def collect(syntax, owner: _Scope, chain: list[str], in_methods: bool = False) -> None: + for child in syntax.named_children: + if child.type in _DECLARATIONS: + name_node = child.child_by_field_name("name") + name = text(name_node).strip() + kind = "class" if child.type == "class_definition" else ( + "method" if in_methods and owner.kind == "class" else "function" + ) + scope = _Scope(child, name, kind, owner) + owner.children.append(scope) + scopes.append(scope) + + header = [ + part for part in child.named_children + if part.type not in {"block", "methods", "properties", "events", "enumeration", "superclasses"} + ] + trustworthy_name = ( + name_node is not None + and name_node.type == "identifier" + and not name_node.has_error + and not name_node.is_missing + and bool(name) + ) + trustworthy_header = kind == "class" or not any( + part.has_error or part.is_missing for part in header + ) + if not trustworthy_name or not trustworthy_header: + scope.unsafe = True + if owner.kind in {"function", "method"} and not ( + trustworthy_name and trustworthy_header + ): + owner.unsafe = True + next_chain = [*chain, name] if name else chain + if trustworthy_name and trustworthy_header: + nid = _make_id( + stem, + "matlab", + kind, + *chain, + name, + f"L{child.start_point.row + 1}", + f"C{child.start_point.column + 1}", + ) + scope.record = add_node(nid, name, child, kind, chain) + if child.type == "function_signature": + scope.record["metadata"]["matlab_declaration_only"] = True + if kind == "class": + scope.record["metadata"]["matlab_superclasses"] = superclass_facts(child) + if owner.record is not None: + add_edge( + owner.record["id"], + nid, + "method" if kind == "method" else "contains", + child, + ) + + collect(child, scope, next_chain, False) + elif syntax.type in {"source_file", "function_definition"} and child.type == "block": + collect(child, owner, chain) + elif syntax.type == "class_definition" and child.type == "methods": + collect(child, owner, chain, True) + + collect(root, file_scope, []) + excluded_callable_names = { + scope.name for scope in scopes if scope.kind in {"class", "method"} and scope.name + } + + meaningful = [ + child for child in root.named_children + if child.type not in {"comment", "line_continuation"} + ] + if meaningful: + first = meaningful[0] + while first.type == "block": + nested = next( + (child for child in first.named_children + if child.type not in {"comment", "line_continuation"}), + None, + ) + if nested is None: + break + first = nested + primary = next((scope for scope in file_scope.children if scope.syntax == first), None) + if primary is not None and primary.record is not None: + primary.primary = True + file_scope.record["metadata"].update( + matlab_file_kind=primary.kind, + matlab_primary_name=primary.name, + ) + elif first.type not in {"ERROR", *_DECLARATIONS}: + file_scope.record["metadata"]["matlab_file_kind"] = "script" + + def node_key(node) -> tuple[str, int, int]: + return node.type, node.start_byte, node.end_byte + + known_scope_keys = {node_key(scope.syntax) for scope in scopes[1:]} + + def lvalue_roots(node) -> set[str]: + if node is None: + return set() + if node.type == "identifier": + return {text(node)} + if node.type == "multioutput_variable": + return set().union(*(lvalue_roots(child) for child in node.named_children)) + if node.type == "function_call": + return lvalue_roots(node.child_by_field_name("name")) + if node.type == "field_expression": + return lvalue_roots(node.child_by_field_name("object")) + return set() + + def parameter_names(node) -> set[str]: + names = set() + stack = list(node.named_children) + while stack: + child = stack.pop() + if child.type == "identifier": + names.add(text(child)) + else: + stack.extend(child.named_children) + return names + + def is_call_site(node, scope_node) -> bool: + """A direct unqualified call expression, not an index or dynamic callee.""" + if not any(child.type == "(" for child in node.children): + return False + if any(child.type in {"{", "superclass"} for child in node.children): + return False + parent = node.parent + while parent is not None and node_key(parent) != node_key(scope_node): + if parent.type == "field_expression": + field = parent.child_by_field_name("field") + if field is not None and node_key(field) == node_key(node): + return False + if parent.type in {"lambda", "handle_operator", "arguments_statement"}: + return False + if parent.type == "assignment": + left = parent.child_by_field_name("left") + cursor = node + while cursor is not None and node_key(cursor) != node_key(parent): + if left is not None and node_key(cursor) == node_key(left): + return False + cursor = cursor.parent + if parent.type == "function_call": + outer_name = parent.child_by_field_name("name") + if outer_name is not None and node_key(outer_name) == node_key(node): + return False + parent = parent.parent + return True + + def inspect_scope(scope: _Scope) -> list[tuple[str, Any]]: + """Collect whole-scope bindings and uncertainty barriers.""" + calls: list[tuple[str, Any]] = [] + root_key = node_key(scope.syntax) + for child in scope.syntax.named_children: + if child.type == "function_output": + for output in child.named_children: + scope.bindings.update(lvalue_roots(output)) + elif child.type == "function_arguments": + scope.bindings.update(parameter_names(child)) + + def mark_uncertain_subtree(node) -> None: + stack = [node] + while stack: + child = stack.pop() + if child.type == "ERROR" or child.is_missing: + scope.unsafe = True + if child.type in {"command", "import_statement", "import_declaration"}: + scope.unsafe = True + if child.type == "function_call": + callee = child.child_by_field_name("name") + if ( + callee is not None + and callee.type == "identifier" + and text(callee) in _WORKSPACE_CHANGES + ): + scope.unsafe = True + if child.type not in _DECLARATIONS: + stack.extend(child.children) + + def visit(node) -> None: + if node_key(node) != root_key and node.type in _DECLARATIONS: + if node_key(node) not in known_scope_keys: + scope.unsafe = True + return + if node.type == "ERROR" or node.is_missing: + scope.unsafe = True + if node.type in {"command", "import_statement", "import_declaration"}: + scope.unsafe = True + if node.type == "arguments_statement" and node.has_error: + scope.unsafe = True + if node.type == "arguments_statement": + for prop in node.named_children: + if prop.type != "property": + continue + name_node = prop.child_by_field_name("name") + if name_node is not None and name_node.type == "identifier": + scope.bindings.add(text(name_node)) + mark_uncertain_subtree(node) + return + if node.type == "assignment": + scope.bindings.update(lvalue_roots(node.child_by_field_name("left"))) + elif node.type == "iterator": + first = next((child for child in node.named_children), None) + if first is not None and first.type == "identifier": + scope.bindings.add(text(first)) + elif node.type == "catch_clause": + identifier = next( + (child for child in node.named_children if child.type == "identifier"), + None, + ) + if identifier is not None: + scope.bindings.add(text(identifier)) + elif node.type in {"global_operator", "persistent_operator"}: + scope.bindings.update( + text(child) for child in node.named_children if child.type == "identifier" + ) + + if node.type == "lambda": + mark_uncertain_subtree(node) + return + if node.type == "handle_operator": + return + if node.type == "function_call": + name_node = node.child_by_field_name("name") + if name_node is not None and name_node.type == "identifier": + name = text(name_node) + if name in _WORKSPACE_CHANGES: + scope.unsafe = True + if name != "feval" and is_call_site(node, scope.syntax): + calls.append((name, node)) + for child in node.children: + visit(child) + + visit(scope.syntax) + return calls + + calls_by_scope: list[tuple[_Scope, list[tuple[str, Any]]]] = [] + for scope in scopes[1:]: + if scope.kind in {"function", "method"}: + calls_by_scope.append((scope, inspect_scope(scope))) + + def callable_ancestors(scope: _Scope): + current: _Scope | None = scope + while current is not None and current.kind not in {"file", "class"}: + yield current + current = current.owner + + def caller_is_unsafe(scope: _Scope) -> bool: + return any(ancestor.unsafe for ancestor in callable_ancestors(scope)) + + def lexical_tiers(scope: _Scope): + current: _Scope | None = scope + while current is not None: + yield [child for child in current.children if child.kind == "function"] + parent = current.owner + if parent is None or parent.kind in {"file", "class"}: + return + current = parent + + def file_target(name: str) -> _Scope | None: + declarations = [ + child for child in file_scope.children + if child.kind == "function" and child.name == name + ] + if len(declarations) != 1: + return None + target = declarations[0] + if target.record is None or (target.primary and target.name != path.stem): + return None + return target + + for caller, sites in calls_by_scope: + if caller.record is None or caller_is_unsafe(caller): + continue + for name, syntax in sites: + if name in excluded_callable_names: + continue + if any(name in ancestor.bindings for ancestor in callable_ancestors(caller)): + continue + target = None + found_nearest = False + for tier in lexical_tiers(caller): + declarations = [candidate for candidate in tier if candidate.name == name] + if declarations: + found_nearest = True + if len(declarations) == 1 and declarations[0].record is not None: + target = declarations[0] + break + if not found_nearest: + target = file_target(name) + if target is not None and target.record is not None: + add_edge(caller.record["id"], target.record["id"], "calls", syntax) + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/resolution.py b/graphify/extractors/resolution.py index ee8456f356..8c194aa25b 100644 --- a/graphify/extractors/resolution.py +++ b/graphify/extractors/resolution.py @@ -1220,6 +1220,114 @@ def _disambiguate_colliding_node_ids( elif raw_call.get("evidence_owner") in unambiguous_remaps: raw_call["evidence_owner"] = unambiguous_remaps[str(raw_call["evidence_owner"])] +def _matlab_inheritance_edges(nodes: list[dict]) -> list[dict]: + """Match written bases to exact repository source, never MATLAB's runtime path. + + File context must establish a unique primary class. Keep uncertain source + candidates in the index so malformed duplicate package roots cannot make a + different root appear uniquely resolvable. Inputs and reference metadata are + read-only; callers own final-ID remapping and inheritance refreshes. + """ + identifier = re.compile(r"[A-Za-z][A-Za-z0-9_]*\Z") + files: dict[tuple[str | None, str], list[dict]] = {} + classes: dict[tuple[str | None, str], list[dict]] = {} + for node in nodes: + metadata = node.get("metadata") or {} + if not isinstance(metadata, dict) or metadata.get("language") != "matlab": + continue + source = node.get("source_file") + repo = node.get("repo") + if not isinstance(source, str) or not source or not (repo is None or isinstance(repo, str)): + continue + key = (repo, source) + if metadata.get("kind") == "matlab_file": + files.setdefault(key, []).append(node) + elif metadata.get("kind") == "matlab_class": + classes.setdefault(key, []).append(node) + + qualified: dict[tuple[str | None, str], list[tuple[str | None, str]]] = {} + eligible: dict[tuple[str | None, str], dict] = {} + namespaces: dict[tuple[str | None, str], str] = {} + for key in files.keys() | classes.keys(): + repo, source = key + path = Path(source) + if path.suffix != ".m" or not identifier.fullmatch(path.stem): + continue + if any(part == "private" or part.startswith("@") for part in path.parts[:-1]): + continue + parts = [] + for part in reversed(path.parts[:-1]): + if not part.startswith("+"): + break + parts.append(part[1:]) + if any(not identifier.fullmatch(part) for part in parts): + continue + namespace = ".".join(reversed(parts)) + namespaces[key] = namespace + if namespace: + qualified.setdefault((repo, f"{namespace}.{path.stem}"), []).append(key) + file_nodes = files.get(key, []) + class_nodes = classes.get(key, []) + if len(file_nodes) != 1 or len(class_nodes) != 1: + continue + file_node, class_node = file_nodes[0], class_nodes[0] + file_metadata, class_metadata = file_node["metadata"], class_node["metadata"] + if ( + file_metadata.get("kind") != "matlab_file" + or class_metadata.get("kind") != "matlab_class" + or file_node.get("file_type") != "code" + or class_node.get("file_type") != "code" + or file_node.get("external") + or class_node.get("external") + or not isinstance(class_node.get("id"), str) + or not class_node["id"] + or file_metadata.get("matlab_file_kind") != "class" + or file_metadata.get("matlab_primary_name") != path.stem + or class_node.get("label") != path.stem + or class_metadata.get("scope_chain") != [] + or file_metadata.get("namespace", "") != namespace + or class_metadata.get("namespace", "") != namespace + ): + continue + eligible[key] = class_node + + edges = [] + for key, child in eligible.items(): + repo, source = key + bases = child["metadata"].get("matlab_superclasses", []) + if not isinstance(bases, list): + continue + for base in bases: + if not isinstance(base, dict): + continue + name, location = base.get("name"), base.get("source_location") + if not isinstance(name, str) or not all(identifier.fullmatch(part) for part in name.split(".")): + continue + if not isinstance(location, str) or not re.fullmatch(r"L[1-9][0-9]*", location): + continue + if "." in name: + matches = qualified.get((repo, name), []) + elif not namespaces[key]: + matches = [(repo, str(Path(source).with_name(f"{name}.m")))] + else: + continue + if len(matches) != 1: + continue + target = eligible.get(matches[0]) + if target is None or target["id"] == child["id"]: + continue + edges.append({ + "source": child["id"], + "target": target["id"], + "relation": "inherits", + "confidence": "EXTRACTED", + "source_file": source, + "source_location": location, + "weight": 1.0, + }) + return sorted(edges, key=lambda edge: (edge["source"], edge["source_location"], edge["target"])) + + def _is_type_like_definition(node: dict) -> bool: if node.get("type") == "namespace": return False diff --git a/graphify/watch.py b/graphify/watch.py index 0c71011df0..786b593d73 100644 --- a/graphify/watch.py +++ b/graphify/watch.py @@ -2958,6 +2958,11 @@ def _stamp_build_state() -> None: # graph". print(f"error: {exc}", file=sys.stderr) return False + # Refresh after reconciliation so unchanged MATLAB classes are in scope. + from graphify.build import _refresh_matlab_inheritance + result["edges"] = _refresh_matlab_inheritance( + result.get("nodes", []), result.get("edges", []) + ) # The equality skips below compare parsed data, not layout; an existing # multi-line graph.json must still be rewritten compact (#3413). diff --git a/pyproject.toml b/pyproject.toml index 0e59e39161..88aebc316a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,6 +47,7 @@ dependencies = [ "tree-sitter-elixir>=0.3,<0.5", "tree-sitter-objc>=3.0,<4.0", "tree-sitter-julia>=0.23,<0.25", + "tree-sitter-matlab==1.3.1", "tree-sitter-verilog>=1.0,<2.0", "tree-sitter-fortran>=0.6,<0.8", "tree-sitter-bash>=0.23,<0.27", diff --git a/tests/fixtures/matlab/frontend/scopes.m b/tests/fixtures/matlab/frontend/scopes.m new file mode 100644 index 0000000000..d0a807ba52 --- /dev/null +++ b/tests/fixtures/matlab/frontend/scopes.m @@ -0,0 +1,16 @@ +function entry() + local(); + function local() + sibling(); + end + function sibling() + local(); + end +end + +function public() + local(); +end + +function local() +end diff --git a/tests/fixtures/matlab/frontend/structure.m b/tests/fixtures/matlab/frontend/structure.m new file mode 100644 index 0000000000..c9d1df71bc --- /dev/null +++ b/tests/fixtures/matlab/frontend/structure.m @@ -0,0 +1,16 @@ +classdef C < pkg.Base & handle + methods (Abstract) + y = declared(obj, x) + end + methods + function y = run(obj, x) + y = nested(x); + function z = nested(v) + z = v; + end + end + end +end + +function helper() +end diff --git a/tests/fixtures/matlab/jsonlab/LICENSE_BSD.txt b/tests/fixtures/matlab/jsonlab/LICENSE_BSD.txt new file mode 100644 index 0000000000..5db82c4549 --- /dev/null +++ b/tests/fixtures/matlab/jsonlab/LICENSE_BSD.txt @@ -0,0 +1,33 @@ +Copyright (c) 2011-2020 Qianqian Fang +Copyright (c) 2014,2016 Bastian Bechtold +Copyright (c) 2012, Kota Yamaguchi +Copyright (c) 2009, Nedialko Krouchev +Copyright (c) 2009, François Glineur +Copyright (c) 2008, Joel Feenstra + +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/tests/fixtures/matlab/jsonlab/expected.json b/tests/fixtures/matlab/jsonlab/expected.json new file mode 100644 index 0000000000..43dc928730 --- /dev/null +++ b/tests/fixtures/matlab/jsonlab/expected.json @@ -0,0 +1,562 @@ +{ + "schema_version": 1, + "provenance": { + "project": "JSONLab", + "release": "v2.0", + "repository": "https://github.com/NeuroJSON/jsonlab", + "commit": "5a58fafdca239e61be4d7b2270307b2445bb571a", + "license": "BSD-3-Clause", + "attribution": "Qianqian Fang and contributors; original author header and full LICENSE_BSD.txt retained verbatim.", + "files": [ + { + "path": "loadjson.m", + "source_url": "https://github.com/NeuroJSON/jsonlab/blob/5a58fafdca239e61be4d7b2270307b2445bb571a/loadjson.m", + "sha256": "ecb4bca8b9762a1849923536694f418b6cea9e6240959244e9ec629925d366f5", + "size_bytes": 21604 + }, + { + "path": "LICENSE_BSD.txt", + "source_url": "https://github.com/NeuroJSON/jsonlab/blob/5a58fafdca239e61be4d7b2270307b2445bb571a/LICENSE_BSD.txt", + "sha256": "2e5589a9aa67172658bd5f5d79313160d1327ecdb740ca6cf2fb05f6f4efa74e", + "size_bytes": 1707 + } + ] + }, + "contract": { + "source_file": "loadjson.m", + "coordinates": "One-based rows and UTF-8 byte columns; declaration start is function keyword.", + "keys": "Manifest-local logical keys, not Graphify IDs. Resolve actual endpoints using source file, metadata.kind, exact label, metadata.scope_chain and declaration source_location; all definitions here begin at byte column 1.", + "definitions": "Exhaustive trustworthy declarations; file node listed separately.", + "contains": "Exhaustive immediate containment; source_location is the child declaration row. No method or inherits edges exist in this slice.", + "calls": "Exhaustive eligible resolved same-file raw call sites, before persisted pair coalescing. Compare multisets by relationship, endpoint logical keys and source_location. column/source_text independently locate the written identifier; graph source_location uses L. Every call must have confidence EXTRACTED and weight 1.0.", + "metadata": "Every file/definition node has nested metadata.language=matlab. Definition kind is matlab_function, scope_chain is []. File kind is matlab_file with matlab_file_kind=function and matlab_primary_name=loadjson.", + "denominator": "Cross-file/builtin calls, indexing, handles, anonymous bodies, dotted dispatch and whole-scope-barred calls are excluded, not missing expected calls.", + "independence": "Manually reviewed pinned source and accepted issue 4 rules; no MATLAB frontend output, parser output or JSONLab execution used to choose records.", + "specification": "https://github.com/JulioLeonPhD/graphify/issues/4#issuecomment-6420416127" + }, + "file": { + "key": "file", + "path": "loadjson.m", + "kind": "matlab_file", + "matlab_file_kind": "function", + "matlab_primary_name": "loadjson" + }, + "definitions": [ + { + "key": "loadjson@L1:C1", + "name": "loadjson", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L1", + "column": 1, + "end_line": 167, + "source_text": "function data = loadjson(fname,varargin)" + }, + { + "key": "parse_array@L173:C1", + "name": "parse_array", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L173", + "column": 1, + "end_line": 312, + "source_text": "function [object, pos,index_esc] = parse_array(inputstr, pos, esc, index_esc, varargin) % JSON array is written in row-major order" + }, + { + "key": "parse_char@L315:C1", + "name": "parse_char", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L315", + "column": 1, + "end_line": 323, + "source_text": "function pos=parse_char(inputstr, pos, c)" + }, + { + "key": "next_char@L326:C1", + "name": "next_char", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L326", + "column": 1, + "end_line": 333, + "source_text": "function [c, pos] = next_char(inputstr, pos)" + }, + { + "key": "parseStr@L336:C1", + "name": "parseStr", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L336", + "column": 1, + "end_line": 396, + "source_text": "function [str, pos,index_esc] = parseStr(inputstr, pos, esc, index_esc, varargin)" + }, + { + "key": "parse_number@L399:C1", + "name": "parse_number", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L399", + "column": 1, + "end_line": 406, + "source_text": "function [num, pos] = parse_number(inputstr, pos, varargin)" + }, + { + "key": "parse_value@L409:C1", + "name": "parse_value", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L409", + "column": 1, + "end_line": 448, + "source_text": "function [val, pos,index_esc] = parse_value(inputstr, pos, esc, index_esc, varargin)" + }, + { + "key": "parse_object@L451:C1", + "name": "parse_object", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L451", + "column": 1, + "end_line": 481, + "source_text": "function [object, pos, index_esc] = parse_object(inputstr, pos, esc, index_esc, varargin)" + }, + { + "key": "error_pos@L485:C1", + "name": "error_pos", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L485", + "column": 1, + "end_line": 493, + "source_text": "function pos=error_pos(msg, inputstr, pos)" + }, + { + "key": "skip_whitespace@L497:C1", + "name": "skip_whitespace", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L497", + "column": 1, + "end_line": 502, + "source_text": "function newpos=skip_whitespace(pos, inputstr)" + }, + { + "key": "unescapejsonstring@L505:C1", + "name": "unescapejsonstring", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L505", + "column": 1, + "end_line": 521, + "source_text": "function newstr=unescapejsonstring(str)" + }, + { + "key": "sscanf_prep@L524:C1", + "name": "sscanf_prep", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L524", + "column": 1, + "end_line": 532, + "source_text": "function arraystr=sscanf_prep(str)" + }, + { + "key": "parse2darray@L535:C1", + "name": "parse2darray", + "kind": "matlab_function", + "scope_chain": [], + "source_location": "L535", + "column": 1, + "end_line": 552, + "source_text": "function [obj, nextidx,nextdim]=parse2darray(inputstr,startpos,arraystr)" + } + ], + "contains": [ + { + "source": "file", + "target": "loadjson@L1:C1", + "source_location": "L1" + }, + { + "source": "file", + "target": "parse_array@L173:C1", + "source_location": "L173" + }, + { + "source": "file", + "target": "parse_char@L315:C1", + "source_location": "L315" + }, + { + "source": "file", + "target": "next_char@L326:C1", + "source_location": "L326" + }, + { + "source": "file", + "target": "parseStr@L336:C1", + "source_location": "L336" + }, + { + "source": "file", + "target": "parse_number@L399:C1", + "source_location": "L399" + }, + { + "source": "file", + "target": "parse_value@L409:C1", + "source_location": "L409" + }, + { + "source": "file", + "target": "parse_object@L451:C1", + "source_location": "L451" + }, + { + "source": "file", + "target": "error_pos@L485:C1", + "source_location": "L485" + }, + { + "source": "file", + "target": "skip_whitespace@L497:C1", + "source_location": "L497" + }, + { + "source": "file", + "target": "unescapejsonstring@L505:C1", + "source_location": "L505" + }, + { + "source": "file", + "target": "sscanf_prep@L524:C1", + "source_location": "L524" + }, + { + "source": "file", + "target": "parse2darray@L535:C1", + "source_location": "L535" + } + ], + "calls": [ + { + "source": "loadjson@L1:C1", + "target": "error_pos@L485:C1", + "source_location": "L103", + "column": 8, + "source_text": " error_pos('input file does not exist');" + }, + { + "source": "loadjson@L1:C1", + "target": "next_char@L326:C1", + "source_location": "L136", + "column": 18, + "source_text": " [cc,pos]=next_char(inputstr, pos);" + }, + { + "source": "loadjson@L1:C1", + "target": "parse_object@L451:C1", + "source_location": "L139", + "column": 51, + "source_text": " [data{jsoncount},pos,index_esc] = parse_object(inputstr, pos, esc, index_esc,opt);" + }, + { + "source": "loadjson@L1:C1", + "target": "parse_array@L173:C1", + "source_location": "L141", + "column": 51, + "source_text": " [data{jsoncount},pos,index_esc] = parse_array(inputstr, pos, esc, index_esc,opt);" + }, + { + "source": "loadjson@L1:C1", + "target": "error_pos@L485:C1", + "source_location": "L143", + "column": 21, + "source_text": " pos=error_pos('Outer level structure must be an object or an array',inputstr,pos);" + }, + { + "source": "parse_char@L315:C1", + "target": "skip_whitespace@L497:C1", + "source_location": "L316", + "column": 9, + "source_text": " pos=skip_whitespace(pos, inputstr);" + }, + { + "source": "parse_char@L315:C1", + "target": "error_pos@L485:C1", + "source_location": "L318", + "column": 13, + "source_text": " pos=error_pos(sprintf('Expected %c at position %%d', c),inputstr,pos);" + }, + { + "source": "parse_char@L315:C1", + "target": "skip_whitespace@L497:C1", + "source_location": "L321", + "column": 13, + "source_text": " pos=skip_whitespace(pos, inputstr);" + }, + { + "source": "next_char@L326:C1", + "target": "skip_whitespace@L497:C1", + "source_location": "L327", + "column": 9, + "source_text": " pos=skip_whitespace(pos, inputstr);" + }, + { + "source": "parse_number@L399:C1", + "target": "error_pos@L485:C1", + "source_location": "L403", + "column": 13, + "source_text": " pos=error_pos('Error reading number at position %d',inputstr,pos);" + }, + { + "source": "parse_value@L409:C1", + "target": "parseStr@L336:C1", + "source_location": "L417", + "column": 36, + "source_text": " [val, pos,index_esc] = parseStr(inputstr, pos, esc, index_esc,varargin{:});" + }, + { + "source": "parse_value@L409:C1", + "target": "parse_array@L173:C1", + "source_location": "L420", + "column": 36, + "source_text": " [val, pos,index_esc] = parse_array(inputstr, pos, esc, index_esc, varargin{:});" + }, + { + "source": "parse_value@L409:C1", + "target": "parse_object@L451:C1", + "source_location": "L423", + "column": 36, + "source_text": " [val, pos,index_esc] = parse_object(inputstr, pos, esc, index_esc, varargin{:});" + }, + { + "source": "parse_value@L409:C1", + "target": "parse_number@L399:C1", + "source_location": "L426", + "column": 26, + "source_text": " [val, pos] = parse_number(inputstr, pos, varargin{:});" + }, + { + "source": "parse_value@L409:C1", + "target": "error_pos@L485:C1", + "source_location": "L447", + "column": 9, + "source_text": " pos=error_pos('Value expected at position %d',inputstr,pos);" + }, + { + "source": "parse_object@L451:C1", + "target": "parse_char@L315:C1", + "source_location": "L452", + "column": 9, + "source_text": " pos=parse_char(inputstr, pos, '{');" + }, + { + "source": "parse_object@L451:C1", + "target": "next_char@L326:C1", + "source_location": "L459", + "column": 14, + "source_text": " [cc,pos]=next_char(inputstr,pos);" + }, + { + "source": "parse_object@L451:C1", + "target": "parseStr@L336:C1", + "source_location": "L462", + "column": 37, + "source_text": " [str, pos, index_esc] = parseStr(inputstr, pos, esc, index_esc, varargin{:});" + }, + { + "source": "parse_object@L451:C1", + "target": "error_pos@L485:C1", + "source_location": "L464", + "column": 21, + "source_text": " pos=error_pos('Name of value at position %d cannot be empty',inputstr,pos);" + }, + { + "source": "parse_object@L451:C1", + "target": "parse_char@L315:C1", + "source_location": "L466", + "column": 17, + "source_text": " pos=parse_char(inputstr, pos, ':');" + }, + { + "source": "parse_object@L451:C1", + "target": "parse_value@L409:C1", + "source_location": "L467", + "column": 36, + "source_text": " [val, pos,index_esc] = parse_value(inputstr, pos, esc, index_esc, varargin{:});" + }, + { + "source": "parse_object@L451:C1", + "target": "next_char@L326:C1", + "source_location": "L473", + "column": 22, + "source_text": " [cc,pos]=next_char(inputstr,pos);" + }, + { + "source": "parse_object@L451:C1", + "target": "parse_char@L315:C1", + "source_location": "L477", + "column": 17, + "source_text": " pos=parse_char(inputstr, pos, ',');" + }, + { + "source": "parse_object@L451:C1", + "target": "parse_char@L315:C1", + "source_location": "L480", + "column": 9, + "source_text": " pos=parse_char(inputstr, pos, '}');" + }, + { + "source": "parse2darray@L535:C1", + "target": "sscanf_prep@L524:C1", + "source_location": "L537", + "column": 12, + "source_text": " rowstr=sscanf_prep(inputstr(startpos-1:rowend));" + } + ], + "excluded_same_file_sites": [ + { + "source": "parse_array@L173:C1", + "target": "parse_char@L315:C1", + "source_location": "L174", + "column": 9, + "source_text": " pos=parse_char(inputstr, pos, '[');", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "next_char@L326:C1", + "source_location": "L182", + "column": 14, + "source_text": " [cc,pos]=next_char(inputstr,pos);", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "sscanf_prep@L524:C1", + "source_location": "L191", + "column": 30, + "source_text": " arraystr=sscanf_prep(arraystr);", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "parse_char@L315:C1", + "source_location": "L201", + "column": 41, + "source_text": " pos=parse_char(inputstr, pos, ']');", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "parse2darray@L535:C1", + "source_location": "L210", + "column": 48, + "source_text": " [obj, nextidx]=parse2darray(inputstr,pos+rowstart,arraystr);", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "parse_char@L315:C1", + "source_location": "L217", + "column": 41, + "source_text": " pos=parse_char(inputstr, pos, ']');", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "parse_char@L315:C1", + "source_location": "L244", + "column": 45, + "source_text": " pos=parse_char(inputstr, pos, ']');", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "parse_value@L409:C1", + "source_location": "L272", + "column": 40, + "source_text": " [val, pos,index_esc] = parse_value(inputstr, pos, esc, index_esc,varargin{:});", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "next_char@L326:C1", + "source_location": "L274", + "column": 26, + "source_text": " [cc,pos]=next_char(inputstr,pos);", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "parse_char@L315:C1", + "source_location": "L278", + "column": 21, + "source_text": " pos=parse_char(inputstr, pos, ',');", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parse_array@L173:C1", + "target": "parse_char@L315:C1", + "source_location": "L307", + "column": 9, + "source_text": " pos=parse_char(inputstr, pos, ']');", + "reason": "whole-scope eval barrier at L261" + }, + { + "source": "parseStr@L336:C1", + "target": "error_pos@L485:C1", + "source_location": "L338", + "column": 13, + "source_text": " pos=error_pos('String starting with \" expected at position %d',inputstr,pos);", + "reason": "whole-scope bare command/script invocation barrier: keyboard; at L390" + }, + { + "source": "parseStr@L336:C1", + "target": "error_pos@L485:C1", + "source_location": "L371", + "column": 25, + "source_text": " pos=error_pos('End of file reached right after escape character',inputstr,pos);", + "reason": "whole-scope bare command/script invocation barrier: keyboard; at L390" + }, + { + "source": "parseStr@L336:C1", + "target": "error_pos@L485:C1", + "source_location": "L383", + "column": 33, + "source_text": " pos=error_pos('End of file reached in escaped unicode character',inputstr,pos);", + "reason": "whole-scope bare command/script invocation barrier: keyboard; at L390" + }, + { + "source": "parseStr@L336:C1", + "target": "unescapejsonstring@L505:C1", + "source_location": "L394", + "column": 9, + "source_text": " str=unescapejsonstring(str);", + "reason": "whole-scope bare command/script invocation barrier: keyboard; at L390" + }, + { + "source": "parseStr@L336:C1", + "target": "error_pos@L485:C1", + "source_location": "L395", + "column": 9, + "source_text": " pos=error_pos('End of file while expecting end of inputstr',inputstr,pos);", + "reason": "whole-scope bare command/script invocation barrier: keyboard; at L390" + } + ], + "review_notes": [ + "All 13 functions are immediate file children, each explicitly terminated by end. No nested definitions, classes, methods, duplicate names or inheritance occur. Primary function loadjson agrees with loadjson.m.", + "All accepted targets are unique exact-case file-local functions. Inputs, outputs and assignment roots in each caller do not bind any of these target names; no imports or nested lexical ancestors occur.", + "parse_array: eval(arraystr) at L261 vetoes the entire callable scope, including sites before eval and unreachable paths; catch does not restore binding confidence. The handle @unescapejsonstring at L263 is separately excluded.", + "parseStr: bare keyboard; at L390 is an unknown-workspace command invocation, even in a supposedly unreachable otherwise branch. It vetoes all five same-file call sites in that callable. The declaration and incoming calls remain valid.", + "No eligible same-file sites occur in error_pos, skip_whitespace, unescapejsonstring or sscanf_prep. parse2darray calls external match_bracket at L536 (excluded) and local sscanf_prep at L537 (included).", + "Parenthesis expressions on bound inputstr, data, objid, esc, str, object, poShow, arraystr, astr, dims and other assigned roots are indexing, never calls. Field/indexed assignments do not create callable targets.", + "External helpers include varargin2struct, jsonopt, jdatadecode, fast_match_bracket, nestbracket2dim, encodevarname and match_bracket. None has a declaration in this file, so none is in the call denominator.", + "containers.Map at L455 is dotted constructor syntax; anonymous-function bodies at L285 and L509 and replacement-expression string at L520 are excluded. Bare values such as pwd, inf and nargin in expressions are not direct parenthesized call sites or standalone command statements.", + "L103 error_pos has fewer arguments than its signature; static eligible call identity does not assert runtime validity. Corpus source is not repaired or executed.", + "This corpus is intentionally limited: synthetic fixtures, not this manifest, cover classes, namespaces, nested scopes and duplicate names." + ] +} diff --git a/tests/fixtures/matlab/jsonlab/loadjson.m b/tests/fixtures/matlab/jsonlab/loadjson.m new file mode 100644 index 0000000000..69811ed122 --- /dev/null +++ b/tests/fixtures/matlab/jsonlab/loadjson.m @@ -0,0 +1,552 @@ +function data = loadjson(fname,varargin) +% +% data=loadjson(fname,opt) +% or +% data=loadjson(fname,'param1',value1,'param2',value2,...) +% +% parse a JSON (JavaScript Object Notation) file or string +% +% authors:Qianqian Fang (q.fang neu.edu) +% created on 2011/09/09, including previous works from +% +% Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713 +% created on 2009/11/02 +% François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393 +% created on 2009/03/22 +% Joel Feenstra: +% http://www.mathworks.com/matlabcentral/fileexchange/20565 +% created on 2008/07/03 +% +% input: +% fname: input file name; if fname contains "{}" or "[]", fname +% will be interpreted as a JSON string +% opt: (optional) a struct to store parsing options, opt can be replaced by +% a list of ('param',value) pairs - the param string is equivallent +% to a field in opt. opt can have the following +% fields (first in [.|.] is the default) +% +% SimplifyCell [1|0]: if set to 1, loadjson will call cell2mat +% for each element of the JSON data, and group +% arrays based on the cell2mat rules. +% FastArrayParser [1|0 or integer]: if set to 1, use a +% speed-optimized array parser when loading an +% array object. The fast array parser may +% collapse block arrays into a single large +% array similar to rules defined in cell2mat; 0 to +% use a legacy parser; if set to a larger-than-1 +% value, this option will specify the minimum +% dimension to enable the fast array parser. For +% example, if the input is a 3D array, setting +% FastArrayParser to 1 will return a 3D array; +% setting to 2 will return a cell array of 2D +% arrays; setting to 3 will return to a 2D cell +% array of 1D vectors; setting to 4 will return a +% 3D cell array. +% UseMap [0|1]: if set to 1, loadjson uses a containers.Map to +% store map objects; otherwise use a struct object +% ShowProgress [0|1]: if set to 1, loadjson displays a progress bar. +% ParseStringArray [0|1]: if set to 0, loadjson converts "string arrays" +% (introduced in MATLAB R2016b) to char arrays; if set to 1, +% loadjson skips this conversion. +% FormatVersion [2|float]: set the JSONLab format version; since +% v2.0, JSONLab uses JData specification Draft 1 +% for output format, it is incompatible with all +% previous releases; if old output is desired, +% please set FormatVersion to 1.9 or earlier. +% Encoding ['']: json file encoding. Support all encodings of +% fopen() function +% ObjectID [0|interger or list]: if set to a positive number, +% it returns the specified JSON object by index +% in a multi-JSON document; if set to a vector, +% it returns a list of specified objects. +% JDataDecode [1|0]: if set to 1, call jdatadecode to decode +% JData structures defined in the JData +% Specification. +% +% output: +% dat: a cell array, where {...} blocks are converted into cell arrays, +% and [...] are converted to arrays +% +% examples: +% dat=loadjson('{"obj":{"string":"value","array":[1,2,3]}}') +% dat=loadjson(['examples' filesep 'example1.json']) +% dat=loadjson(['examples' filesep 'example1.json'],'SimplifyCell',0) +% +% license: +% BSD or GPL version 3, see LICENSE_{BSD,GPLv3}.txt files for details +% +% -- this function is part of JSONLab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) +% + + opt=varargin2struct(varargin{:}); + + if(regexp(fname,'^\s*(?:\[.*\])|(?:\{.*\})\s*$','once')) + string=fname; + elseif(exist(fname,'file')) + try + encoding = jsonopt('Encoding','',opt); + if(isempty(encoding)) + string = fileread(fname); + else + fid = fopen(fname,'r','n',encoding); + string = fread(fid,'*char')'; + fclose(fid); + end + catch + try + string = urlread(['file://',fname]); + catch + string = urlread(['file://',fullfile(pwd,fname)]); + end + end + else + error_pos('input file does not exist'); + end + + pos = 1; inputlen = length(string); inputstr = string; + arraytokenidx=find(inputstr=='[' | inputstr==']'); + arraytoken=inputstr(arraytokenidx); + + % String delimiters and escape chars identified to improve speed: + esc = find(inputstr=='"' | inputstr=='\' ); % comparable to: regexp(inputstr, '["\\]'); + index_esc = 1; + + opt.arraytoken_=arraytoken; + opt.arraytokenidx_=arraytokenidx; + opt.simplifycell=jsonopt('SimplifyCell',1,opt); + opt.simplifycellarray=jsonopt('SimplifyCellArray',0,opt); + opt.formatversion=jsonopt('FormatVersion',2,opt); + opt.fastarrayparser=jsonopt('FastArrayParser',1,opt); + opt.parsestringarray=jsonopt('ParseStringArray',0,opt); + opt.usemap=jsonopt('UseMap',0,opt); + opt.arraydepth_=1; + + if(jsonopt('ShowProgress',0,opt)==1) + opt.progressbar_=waitbar(0,'loading ...'); + end + + objid=jsonopt('ObjectID',0,opt); + maxobjid=max(objid); + if(maxobjid==0) + maxobjid=inf; + end + + jsoncount=1; + while pos <= inputlen + [cc,pos]=next_char(inputstr, pos); + switch(cc) + case '{' + [data{jsoncount},pos,index_esc] = parse_object(inputstr, pos, esc, index_esc,opt); + case '[' + [data{jsoncount},pos,index_esc] = parse_array(inputstr, pos, esc, index_esc,opt); + otherwise + pos=error_pos('Outer level structure must be an object or an array',inputstr,pos); + end + if(jsoncount>=maxobjid) + break; + end + jsoncount=jsoncount+1; + end % while + + if(length(objid)>1 || min(objid)>1) + data=data(objid(objid<=length(data))); + end + + jsoncount=length(data); + if(jsoncount==1 && iscell(data)) + data=data{1}; + end + + if(jsonopt('JDataDecode',1,varargin{:})==1) + data=jdatadecode(data,'Base64',1,'Recursive',1,varargin{:}); + end + + if(isfield(opt,'progressbar_')) + close(opt.progressbar_); + end +end + +%%------------------------------------------------------------------------- +%% helper functions +%%------------------------------------------------------------------------- + +function [object, pos,index_esc] = parse_array(inputstr, pos, esc, index_esc, varargin) % JSON array is written in row-major order + pos=parse_char(inputstr, pos, '['); + object = cell(0, 1); + arraydepth=varargin{1}.arraydepth_; + pbar=-1; + if(isfield(varargin{1},'progressbar_')) + pbar=varargin{1}.progressbar_; + end + format=varargin{1}.formatversion; + [cc,pos]=next_char(inputstr,pos); + endpos=[]; + + if cc ~= ']' + try + if((varargin{1}.fastarrayparser)>=1 && arraydepth>=varargin{1}.fastarrayparser) + [endpos, maxlevel]=fast_match_bracket(varargin{1}.arraytoken_,varargin{1}.arraytokenidx_,pos); + if(~isempty(endpos)) + arraystr=['[' inputstr(pos:endpos)]; + arraystr=sscanf_prep(arraystr); + if(isempty(find(arraystr=='"', 1))) + % handle 1D array first + if(maxlevel==1) + astr=arraystr(2:end-1); + astr(astr==' ')=''; + [obj, count, errmsg, nextidx]=sscanf(astr,'%f,',[1,inf]); + if(nextidx>=length(astr)-1) + object=obj; + pos=endpos; + pos=parse_char(inputstr, pos, ']'); + return; + end + end + + % next handle 2D array, these are most common ones + if(maxlevel==2 && ~isempty(regexp(arraystr(2:end),'^\s*\[','once'))) + rowstart=find(arraystr(2:end)=='[',1)+1; + if(rowstart) + [obj, nextidx]=parse2darray(inputstr,pos+rowstart,arraystr); + if(nextidx>=length(arraystr)-1) + object=obj; + if(format>1.9) + object=object.'; + end + pos=endpos; + pos=parse_char(inputstr, pos, ']'); + if(pbar>0) + waitbar(pos/length(inStr),pbar,'loading ...'); + end + return; + end + end + end + + % for N-D packed array in a nested array construct, + % in the future can replace 1d and 2d cases + if(maxlevel>2 && ~isempty(regexp(arraystr(2:end),'^\s*\[\s*\[','once'))) + astr=arraystr; + dims=nestbracket2dim(astr); + if(any(dims==0) || all(mod(dims(:),1) == 0)) % all dimensions are integers - this can be problematic + astr=arraystr; + astr(astr=='[')=''; + astr(astr==']')=''; + astr=regexprep(astr,'\s*,',','); + astr=regexprep(astr,'\s*$',''); + [obj, count, errmsg, nextidx]=sscanf(astr,'%f,',inf); + if(nextidx>=length(astr)-1) + object=reshape(obj,dims); + if(format>1.9) + object=permute(object,ndims(object):-1:1); + end + pos=endpos; + pos=parse_char(inputstr, pos, ']'); + if(pbar>0) + waitbar(pos/length(inStr),pbar,'loading ...'); + end + return; + end + end + end + end + end + end + if(isempty(regexp(arraystr,':','once'))) + arraystr=regexprep(arraystr,'\[','{'); + arraystr=regexprep(arraystr,'\]','}'); + if(varargin{1}.parsestringarray==0) + arraystr=regexprep(arraystr,'\"',''''); + end + object=eval(arraystr); + if(iscell(object)) + object=cellfun(@unescapejsonstring,object,'UniformOutput',false); + end + pos=endpos; + end + catch + end + if(isempty(endpos) || pos~=endpos) + while 1 + varargin{1}.arraydepth_=arraydepth+1; + [val, pos,index_esc] = parse_value(inputstr, pos, esc, index_esc,varargin{:}); + object{end+1} = val; + [cc,pos]=next_char(inputstr,pos); + if cc == ']' + break; + end + pos=parse_char(inputstr, pos, ','); + end + end + end + + if(varargin{1}.simplifycell) + if(iscell(object) && ~isempty(object) && isnumeric(object{1})) + if(all(cellfun(@(e) isequal(size(object{1}), size(e)) , object(2:end)))) + try + oldobj=object; + if(iscell(object) && length(object)>1 && ndims(object{1})>=2) + catdim=size(object{1}); + catdim=ndims(object{1})-(catdim(end)==1)+1; + object=cat(catdim,object{:}); + object=permute(object,ndims(object):-1:1); + else + object=cell2mat(object')'; + end + if(iscell(oldobj) && isstruct(object) && numel(object)>1 && varargin{1}.simplifycellarray==0) + object=oldobj; + end + catch + end + end + end + if(~iscell(object) && size(object,1)>1 && ndims(object)==2) + object=object'; + end + end + pos=parse_char(inputstr, pos, ']'); + + if(pbar>0) + waitbar(pos/length(inputstr),pbar,'loading ...'); + end +end +%%------------------------------------------------------------------------- + +function pos=parse_char(inputstr, pos, c) + pos=skip_whitespace(pos, inputstr); + if pos > length(inputstr) || inputstr(pos) ~= c + pos=error_pos(sprintf('Expected %c at position %%d', c),inputstr,pos); + else + pos = pos + 1; + pos=skip_whitespace(pos, inputstr); + end +end +%%------------------------------------------------------------------------- + +function [c, pos] = next_char(inputstr, pos) + pos=skip_whitespace(pos, inputstr); + if pos > length(inputstr) + c = []; + else + c = inputstr(pos); + end +end + +%%------------------------------------------------------------------------- +function [str, pos,index_esc] = parseStr(inputstr, pos, esc, index_esc, varargin) + if inputstr(pos) ~= '"' + pos=error_pos('String starting with " expected at position %d',inputstr,pos); + else + pos = pos + 1; + end + str = ''; + while pos <= length(inputstr) + while index_esc <= length(esc) && esc(index_esc) < pos + index_esc = index_esc + 1; + end + if index_esc > length(esc) + str = [str inputstr(pos:end)]; + pos = length(inputstr) + 1; + break; + else + str = [str inputstr(pos:esc(index_esc)-1)]; + pos = esc(index_esc); + end + nstr = length(str); + switch inputstr(pos) + case '"' + pos = pos + 1; + if(~isempty(str)) + if(strcmp(str,'_Inf_')) + str=Inf; + elseif(strcmp(str,'-_Inf_')) + str=-Inf; + elseif(strcmp(str,'_NaN_')) + str=NaN; + end + end + return; + case '\' + if pos+1 > length(inputstr) + pos=error_pos('End of file reached right after escape character',inputstr,pos); + end + pos = pos + 1; + switch inputstr(pos) + case {'"' '\' '/'} + str(nstr+1) = inputstr(pos); + pos = pos + 1; + case {'b' 'f' 'n' 'r' 't'} + str(nstr+1) = sprintf(['\' inputstr(pos)]); + pos = pos + 1; + case 'u' + if pos+4 > length(inputstr) + pos=error_pos('End of file reached in escaped unicode character',inputstr,pos); + end + str(nstr+(1:6)) = inputstr(pos-1:pos+4); + pos = pos + 5; + end + otherwise % should never happen + str(nstr+1) = inputstr(pos); + keyboard; + pos = pos + 1; + end + end + str=unescapejsonstring(str); + pos=error_pos('End of file while expecting end of inputstr',inputstr,pos); +end +%%------------------------------------------------------------------------- + +function [num, pos] = parse_number(inputstr, pos, varargin) + currstr=inputstr(pos:min(pos+30,end)); + [num, one, err, delta] = sscanf(currstr, '%f', 1); + if ~isempty(err) + pos=error_pos('Error reading number at position %d',inputstr,pos); + end + pos = pos + delta-1; +end +%%------------------------------------------------------------------------- + +function [val, pos,index_esc] = parse_value(inputstr, pos, esc, index_esc, varargin) + len=length(inputstr); + if(isfield(varargin{1},'progressbar_')) + waitbar(pos/len,varargin{1}.progressbar_,'loading ...'); + end + + switch(inputstr(pos)) + case '"' + [val, pos,index_esc] = parseStr(inputstr, pos, esc, index_esc,varargin{:}); + return; + case '[' + [val, pos,index_esc] = parse_array(inputstr, pos, esc, index_esc, varargin{:}); + return; + case '{' + [val, pos,index_esc] = parse_object(inputstr, pos, esc, index_esc, varargin{:}); + return; + case {'-','0','1','2','3','4','5','6','7','8','9'} + [val, pos] = parse_number(inputstr, pos, varargin{:}); + return; + case 't' + if pos+3 <= len && strcmpi(inputstr(pos:pos+3), 'true') + val = true; + pos = pos + 4; + return; + end + case 'f' + if pos+4 <= len && strcmpi(inputstr(pos:pos+4), 'false') + val = false; + pos = pos + 5; + return; + end + case 'n' + if pos+3 <= len && strcmpi(inputstr(pos:pos+3), 'null') + val = []; + pos = pos + 4; + return; + end + end + pos=error_pos('Value expected at position %d',inputstr,pos); +end + +%%------------------------------------------------------------------------- +function [object, pos, index_esc] = parse_object(inputstr, pos, esc, index_esc, varargin) + pos=parse_char(inputstr, pos, '{'); + usemap=varargin{1}.usemap; + if(usemap) + object = containers.Map(); + else + object = []; + end + [cc,pos]=next_char(inputstr,pos); + if cc ~= '}' + while 1 + [str, pos, index_esc] = parseStr(inputstr, pos, esc, index_esc, varargin{:}); + if isempty(str) + pos=error_pos('Name of value at position %d cannot be empty',inputstr,pos); + end + pos=parse_char(inputstr, pos, ':'); + [val, pos,index_esc] = parse_value(inputstr, pos, esc, index_esc, varargin{:}); + if(usemap) + object(str)=val; + else + object.(encodevarname(str,varargin{:}))=val; + end + [cc,pos]=next_char(inputstr,pos); + if cc == '}' + break; + end + pos=parse_char(inputstr, pos, ','); + end + end + pos=parse_char(inputstr, pos, '}'); +end + +%%------------------------------------------------------------------------- + +function pos=error_pos(msg, inputstr, pos) + poShow = max(min([pos-15 pos-1 pos pos+20],length(inputstr)),1); + if poShow(3) == poShow(2) + poShow(3:4) = poShow(2)+[0 -1]; % display nothing after + end + msg = [sprintf(msg, pos) ': ' ... + inputstr(poShow(1):poShow(2)) '' inputstr(poShow(3):poShow(4)) ]; + error( ['JSONLAB:JSON:InvalidFormat: ' msg] ); +end + +%%------------------------------------------------------------------------- + +function newpos=skip_whitespace(pos, inputstr) + newpos=pos; + while newpos <= length(inputstr) && isspace(inputstr(newpos)) + newpos = newpos + 1; + end +end + +%%------------------------------------------------------------------------- +function newstr=unescapejsonstring(str) + newstr=str; + if(iscell(str)) + try + newstr=cell2mat(cellfun(@(x) cell2mat(x),str(:),'un',0)); + catch + end + end + if(~ischar(str)) + return; + end + escapechars={'\\','\"','\/','\a','\b','\f','\n','\r','\t','\v'}; + for i=1:length(escapechars); + newstr=regexprep(newstr,regexprep(escapechars{i},'\\','\\\\'), escapechars{i}); + end + newstr=regexprep(newstr,'\\u([0-9A-Fa-f]{4})', '${char(base2dec($1,16))}'); +end + +%%------------------------------------------------------------------------- +function arraystr=sscanf_prep(str) + arraystr=str; + if(regexp(str,'"','once')) + arraystr=regexprep(arraystr,'"_NaN_"','NaN'); + arraystr=regexprep(arraystr,'"([-+]*)_Inf_"','$1Inf'); + end + arraystr(arraystr==sprintf('\n'))=[]; + arraystr(arraystr==sprintf('\r'))=[]; +end + +%%------------------------------------------------------------------------- +function [obj, nextidx,nextdim]=parse2darray(inputstr,startpos,arraystr) + rowend=match_bracket(inputstr,startpos); + rowstr=sscanf_prep(inputstr(startpos-1:rowend)); + [vec1, nextdim, errmsg, nextidx]=sscanf(rowstr,'%f,',[1 inf]); + if(nargin==2) + obj=nextdim; + return; + end + astr=arraystr; + astr(astr=='[')=''; + astr(astr==']')=''; + astr=regexprep(deblank(astr),'\s+,',','); + [obj, count, errmsg, nextidx]=sscanf(astr,'%f,',inf); + if(nextidx>=length(astr)-1) + obj=reshape(obj,nextdim,numel(obj)/nextdim); + nextidx=length(arraystr)+1; + end +end diff --git a/tests/test_4978_reconcile_cost.py b/tests/test_4978_reconcile_cost.py index e365f28177..50f6a1e22b 100644 --- a/tests/test_4978_reconcile_cost.py +++ b/tests/test_4978_reconcile_cost.py @@ -80,10 +80,10 @@ def test_h7_every_h_dialect_is_ast_owned(tmp_path, name): assert getattr(extractor, "__name__", "").startswith("extract_") -def test_h7_m_is_the_only_suffix_where_the_sniff_decides_none_ness(tmp_path): - """.m can be None (MATLAB), which is why a cheap predicate must keep it.""" +def test_h7_m_dispatch_routes_matlab_and_objc(tmp_path): + """.m falls back to MATLAB; Objective-C and `.mm` stay AST-owned.""" paths = _write_fixtures(tmp_path / "f") - assert ex._get_extractor(paths["matlab.m"]) is None + assert ex._get_extractor(paths["matlab.m"]) is ex.extract_matlab assert ex._get_extractor(paths["objc.m"]) is not None assert ex._get_extractor(paths["objcpp.mm"]) is not None diff --git a/tests/test_cache.py b/tests/test_cache.py index da16ef029a..8646541aa8 100644 --- a/tests/test_cache.py +++ b/tests/test_cache.py @@ -641,6 +641,33 @@ def test_ast_cache_schema_rejects_same_version_legacy_collision( assert not old_dir.exists() + +def test_matlab_dispatch_schema_retires_upstream_ast_cache(tmp_path, monkeypatch): + """A cache from before MATLAB/Objective-C `.m` dispatch must not be served.""" + import graphify.cache as cache_mod + + source = tmp_path / "legacy.m" + source.write_text("function value = legacy()\nvalue = 1;\nend\n") + monkeypatch.setattr(cache_mod, "_EXTRACTOR_VERSION", "same-version") + monkeypatch.setattr(cache_mod, "_cleaned_ast_dirs", set()) + current_schema = cache_mod._AST_CACHE_SCHEMA + assert current_schema > 44 + + monkeypatch.setattr(cache_mod, "_AST_CACHE_SCHEMA", 44) + old_dir = cache_mod.cache_dir(tmp_path, "ast") + save_cached( + source, + {"nodes": [{"id": "stale_dispatch", "label": "old .m dispatch"}], "edges": []}, + root=tmp_path, + kind="ast", + ) + assert load_cached(source, root=tmp_path, kind="ast") is not None + + monkeypatch.setattr(cache_mod, "_AST_CACHE_SCHEMA", current_schema) + assert load_cached(source, root=tmp_path, kind="ast") is None + assert not old_dir.exists() + + def test_python_receiver_shadow_schema_retires_same_version_raw_calls( tmp_path, monkeypatch, ): diff --git a/tests/test_extract.py b/tests/test_extract.py index 21a3fa6cd5..bc15dd2f8a 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -5703,10 +5703,8 @@ def test_extract_reports_ast_cache_hit_rate_over_file_total(tmp_path, capsys): def test_get_extractor_routes_matlab_m_away_from_objc(tmp_path): - # #1702: .m is shared by Objective-C and MATLAB. A real ObjC .m still routes to - # extract_objc, but a MATLAB .m must NOT be force-parsed by the ObjC grammar - # (which produces garbage) — it gets no extractor instead. - from graphify.extract import _get_extractor, extract_objc + # `.m` stays Objective-C when the detector matches, with MATLAB as its fallback. + from graphify.extract import _get_extractor, extract_matlab, extract_objc objc = tmp_path / "Foo.m" objc.write_text('#import "Foo.h"\n@implementation Foo\n- (void)bar {}\n@end\n') @@ -5717,10 +5715,10 @@ def test_get_extractor_routes_matlab_m_away_from_objc(tmp_path): mm = tmp_path / "x.mm" mm.write_text("#import \n@implementation X\n@end\n") - assert _get_extractor(objc) is extract_objc # real ObjC .m -> objc - assert _get_extractor(matlab_fn) is None # MATLAB function -> no garbage - assert _get_extractor(matlab_cls) is None # MATLAB classdef -> no garbage - assert _get_extractor(mm) is extract_objc # .mm is unambiguously ObjC++ + assert _get_extractor(objc) is extract_objc + assert _get_extractor(matlab_fn) is extract_matlab + assert _get_extractor(matlab_cls) is extract_matlab + assert _get_extractor(mm) is extract_objc def test_markdown_dispatch_matches_resolution_suffixes(): @@ -5734,14 +5732,15 @@ def test_markdown_dispatch_matches_resolution_suffixes(): assert dispatched == MARKDOWN_MENTION_SUFFIXES -def test_matlab_m_not_extracted_as_garbage(tmp_path, capsys): - # End to end: a MATLAB .m produces no (garbage) nodes and is surfaced by the - # no-AST-extractor warning (#1702 + #1689), rather than mis-parsed as ObjC. +def test_matlab_m_is_extracted_as_matlab(tmp_path): m = tmp_path / "controller.m" m.write_text("function u = controller(x)\n u = -x;\nend\n") result = extract([m], cache_root=tmp_path) - assert result["nodes"] == [] # no garbage ObjC nodes - assert "no AST extractor" in capsys.readouterr().err # surfaced, not silent + assert any( + n.get("label") == "controller" + and n.get("metadata", {}).get("language") == "matlab" + for n in result["nodes"] + ) def test_rewire_binds_cross_module_function_reference_to_definition(): diff --git a/tests/test_languages.py b/tests/test_languages.py index ded3e25853..ab95c606a3 100644 --- a/tests/test_languages.py +++ b/tests/test_languages.py @@ -1971,6 +1971,45 @@ def test_objc_splits_inherits_and_implements(): assert ("Dog", "Animal") in _edge_labels(r, "inherits") assert ("Animal", "SampleDelegate") in _edge_labels(r, "implements") +def test_matlab_and_native_same_name_types_remain_distinct(tmp_path): + from graphify.extract import extract + + (tmp_path / "Foo.h").write_text("@interface Foo : NSObject\n@end\n") + (tmp_path / "Foo.m").write_text("classdef Foo < Base\nend\n") + (tmp_path / "Base.m").write_text("classdef Base\nend\n") + (tmp_path / "Native.m").write_text( + "@interface NativeBase : NSObject\n@end\n" + "@interface NativeChild : NativeBase\n@end\n" + ) + (tmp_path / "NativeCpp.mm").write_text( + "@interface NativeCpp : NSObject\n@end\n" + "@implementation NativeCpp\n@end\n" + ) + + result = extract(sorted(tmp_path.glob("*")), root=tmp_path, cache_root=tmp_path, parallel=False) + by_id = {node["id"]: node for node in result["nodes"]} + foo_nodes = [node for node in result["nodes"] if node.get("label") == "Foo"] + assert {node.get("source_file") for node in foo_nodes} == {"Foo.h", "Foo.m"} + assert len({node["id"] for node in foo_nodes}) == 2 + matlab_foo = next(node for node in foo_nodes if node.get("source_file") == "Foo.m") + objc_foo = next(node for node in foo_nodes if node.get("source_file") == "Foo.h") + assert matlab_foo["metadata"]["language"] == "matlab" + + inherits = {(edge["source"], edge["target"]) for edge in result["edges"] + if edge.get("relation") == "inherits"} + base = next(node for node in result["nodes"] if node.get("label") == "Base") + native_base = next(node for node in result["nodes"] if node.get("label") == "NativeBase") + native_child = next(node for node in result["nodes"] if node.get("label") == "NativeChild") + native_cpp = next(node for node in result["nodes"] if node.get("label") == "NativeCpp") + assert (matlab_foo["id"], base["id"]) in inherits + assert base["metadata"]["language"] == "matlab" + assert (objc_foo["id"], base["id"]) not in inherits + assert (native_child["id"], native_base["id"]) in inherits + assert not any(source == matlab_foo["id"] and target == objc_foo["id"] + for source, target in inherits) + assert native_cpp.get("source_file") == "NativeCpp.mm" + assert by_id[native_cpp["id"]]["label"] == "NativeCpp" + def test_objc_protocol_adopts_protocol(): """`@protocol Derived ` must emit an implements edge Derived->Base. diff --git a/tests/test_matlab.py b/tests/test_matlab.py new file mode 100644 index 0000000000..72b89446e7 --- /dev/null +++ b/tests/test_matlab.py @@ -0,0 +1,403 @@ +"""Standalone MATLAB extraction contract; normal dispatch is tested separately.""" + +from pathlib import Path + +import pytest + +from graphify.build import build +from graphify.extract import extract +from graphify.extractors.base import set_active_scan_root +from graphify.extractors.matlab import extract_matlab + + +def extract_source(tmp_path, source, name="main.m"): + path = tmp_path / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(source) + previous = set_active_scan_root(tmp_path) + try: + return extract_matlab(path) + finally: + set_active_scan_root(previous) + + +def extract_fixture(tmp_path, filename, name): + fixture = Path(__file__).parent / "fixtures" / "matlab" / "frontend" / filename + return extract_source(tmp_path, fixture.read_text(), name) + + +def definitions(result): + return [node for node in result["nodes"] if node["metadata"]["kind"] != "matlab_file"] + + +def call_sites(result): + labels = {node["id"]: node["label"] for node in result["nodes"]} + return [(labels[edge["source"]], labels[edge["target"]], edge["source_location"]) + for edge in result["edges"] if edge["relation"] == "calls"] + + +def test_structure_owns_methods_and_nested_functions(tmp_path): + result = extract_fixture(tmp_path, "structure.m", "+pkg/C.m") + found = {node["label"]: node for node in definitions(result)} + assert set(found) == {"C", "declared", "run", "nested", "helper"} + assert found["C"]["metadata"]["matlab_superclasses"] == [ + {"name": "pkg.Base", "source_location": "L1"}, + {"name": "handle", "source_location": "L1"}, + ] + assert found["declared"]["metadata"]["matlab_declaration_only"] is True + assert found["nested"]["metadata"]["scope_chain"] == ["C", "run"] + assert found["helper"]["metadata"]["scope_chain"] == [] + assert all(node["metadata"]["language"] == "matlab" for node in result["nodes"]) + assert result["nodes"][0]["metadata"]["matlab_file_kind"] == "class" + assert result["nodes"][0]["metadata"]["matlab_primary_name"] == "C" + assert found["C"]["metadata"]["namespace"] == "pkg" + ownership = {(edge["source"], edge["target"], edge["relation"]) + for edge in result["edges"] if edge["relation"] in {"contains", "method"}} + assert (found["C"]["id"], found["run"]["id"], "method") in ownership + assert (found["C"]["id"], found["run"]["id"], "contains") not in ownership + assert (found["run"]["id"], found["nested"]["id"], "contains") in ownership + assert not result.get("raw_calls") + + +def test_identities_include_owner_and_declaration_location(tmp_path): + source = """function main() + function Foo() + end + function foo() + end + function left() + function same() + end + end + function right() + function same() + end + end +end +""" + first = extract_source(tmp_path / "one", source) + second = extract_source(tmp_path / "two", source) + assert len({node["id"] for node in definitions(first)}) == 7 + assert [node["id"] for node in first["nodes"]] == [node["id"] for node in second["nodes"]] + + +@pytest.mark.parametrize(("name", "namespace"), [ + ("src/+tools/+io/main.m", "tools.io"), + ("+outer/plain/+inner/main.m", "inner"), + ("+outer/plain/main.m", ""), +]) +def test_namespace_is_contiguous_directory_context(tmp_path, name, namespace): + result = extract_source(tmp_path, "function main()\nend\n", name) + assert definitions(result)[0]["metadata"].get("namespace", "") == namespace + + +def test_nearest_lexical_tiers_and_recursion(tmp_path): + result = extract_source(tmp_path, """function main() + helper(); + main(); + function helper() + helper(); + sibling(); + end + function sibling() + function deeper() + helper(); + end + end +end +function helper() + helper(); +end +function independent() + sibling(); + helper(); +end +""") + assert call_sites(result) == [ + ("main", "helper", "L2"), ("main", "main", "L3"), + ("helper", "helper", "L5"), ("helper", "sibling", "L6"), + ("deeper", "helper", "L10"), ("helper", "helper", "L15"), + ("independent", "helper", "L19"), + ] + calls = [edge for edge in result["edges"] if edge["relation"] == "calls"] + assert calls[0]["target"] != calls[-1]["target"] + assert all(edge["confidence"] == "EXTRACTED" and edge["weight"] == 1.0 for edge in calls) + + +def test_duplicate_tier_vetoes_instead_of_falling_back(tmp_path): + result = extract_source(tmp_path, """function main() + helper(); + function helper() + end + function helper() + end +end +function helper() +end +""") + assert len(definitions(result)) == 4 + assert call_sites(result) == [] + + +def test_case_sensitive_lookup_and_primary_identity(tmp_path): + result = extract_source(tmp_path, """function wrong() + wrong(); + helper(); + Helper(); + HELPER(); +end +function helper() +end +function Helper() +end +""") + assert call_sites(result) == [("wrong", "helper", "L3"), ("wrong", "Helper", "L4")] + assert result["nodes"][0]["metadata"]["matlab_file_kind"] == "function" + assert result["nodes"][0]["metadata"]["matlab_primary_name"] == "wrong" + + +def test_methods_call_functions_but_not_methods_or_constructors(tmp_path): + result = extract_source(tmp_path, """classdef C + methods + function run(obj) + other(); C(); helper(); + end + function other(obj) + end + end +end +function helper() +end +function other() +end +""", "C.m") + assert call_sites(result) == [("run", "helper", "L4")] + + +def test_repeated_call_sites_remain_distinct(tmp_path): + result = extract_source(tmp_path, "function main()\n helper();\n helper();\nend\nfunction helper()\nend\n") + assert call_sites(result) == [("main", "helper", "L2"), ("main", "helper", "L3")] + + +def test_fixture_lexical_visibility_and_local_function_scope(tmp_path): + result = extract_fixture(tmp_path, "scopes.m", "scopes.m") + assert call_sites(result) == [ + ("entry", "local", "L2"), + ("local", "sibling", "L4"), + ("sibling", "local", "L7"), + ("public", "local", "L12"), + ] + assert not result.get("raw_calls") + + +@pytest.mark.parametrize(("header", "prefix", "body"), [ + ("function caller(helper)", "", ""), + ("function [helper, output] = caller()", "", ""), + ("function caller()", "arguments\n helper (1, 1) double\nend\n", ""), + ("function caller()", "", "helper = 1;"), + ("function caller()", "", "[helper, other] = deal(1, 2);"), + ("function caller()", "", "helper(1) = 1;"), + ("function caller()", "", "helper{1} = 1;"), + ("function caller()", "", "helper.field = 1;"), + ("function caller()", "", "for helper = 1:2\n end"), + ("function caller()", "", "parfor helper = 1:2\n end"), + ("function caller()", "", "try\n catch helper\n end"), + ("function caller()", "", "global helper;"), + ("function caller()", "", "persistent helper;"), + ("function caller()", "", "if true\n helper = 1;\n end"), +]) +def test_whole_scope_bindings_veto_calls(tmp_path, header, prefix, body): + source = ( + f"{header}\n{prefix} helper();\n{body}\nend\n" + "function helper()\nend\n" + ) + assert call_sites(extract_source(tmp_path, source)) == [] + + +def test_field_name_is_not_a_variable_binding(tmp_path): + result = extract_source(tmp_path, """function caller() + object.helper = 1; + helper(); +end +function helper() +end +""") + assert call_sites(result) == [("caller", "helper", "L3")] + + +def test_ancestor_bindings_apply_but_sibling_bindings_do_not_leak(tmp_path): + result = extract_source(tmp_path, """function main() + helper = 1; + function nested() + helper(); + end + function sibling() + helper(); + end +end +function assigned() + helper(); + helper = 1; +end +function independent() + helper(); +end +function clean() + helper(); +end +function helper() +end +""") + assert call_sites(result) == [ + ("independent", "helper", "L15"), + ("clean", "helper", "L18"), + ] + + +def test_script_workspace_is_not_a_calling_scope(tmp_path): + result = extract_source(tmp_path, """helper(); +helper = 1; +function local() + helper(); +end +function helper() +end +""") + assert result["nodes"][0]["metadata"]["matlab_file_kind"] == "script" + assert call_sites(result) == [("local", "helper", "L4")] + + +@pytest.mark.parametrize("barrier", [ + "import pkg.helper", + "run_script arg", + "eval('x');", + "evalc('x=1;');", + "evalin('base', 'x');", + "assignin('base', 'x', 1);", + "load('data.mat');", + "clear helper", +]) +def test_workspace_barriers_suppress_the_entire_callable_scope(tmp_path, barrier): + result = extract_source(tmp_path, f"""function caller() + helper(); + {barrier} + helper(); +end +function helper() +end +""") + assert call_sites(result) == [] + + +def test_workspace_barrier_flows_to_nested_functions_only(tmp_path): + result = extract_source(tmp_path, """function main() + evalc('helper = [1 2];'); + function child() + helper(); + end +end +function independent() + helper(); +end +function helper() +end +""") + assert call_sites(result) == [("independent", "helper", "L8")] + + +def test_evalc_dynamic_binding_does_not_become_graph_call(tmp_path): + path = tmp_path / "main.m" + path.write_text("""function main() + evalc('helper = [1 2];'); + helper(1); +end +function helper(x) +end +""") + extraction = extract( + [path], root=tmp_path, cache_root=tmp_path / "cache", parallel=False, + ) + graph = build([extraction], dedup=False, root=tmp_path) + file_node = next(node for node in extraction["nodes"] + if node["metadata"]["kind"] == "matlab_file") + main = next(node["id"] for node in extraction["nodes"] if node["label"] == "main") + helper = next(node["id"] for node in extraction["nodes"] if node["label"] == "helper") + + assert graph.edges[file_node["id"], main]["relation"] == "contains" + assert graph.edges[file_node["id"], helper]["relation"] == "contains" + assert not graph.has_edge(main, helper) + + +def test_unqualified_call_argument_inside_member_call_survives_build(tmp_path): + path = tmp_path / "main.m" + path.write_text( + "function main(obj)\nobj.run(helper());\nend\nfunction helper()\nend\n" + ) + extraction = extract( + [path], root=tmp_path, cache_root=tmp_path / "cache", parallel=False, + ) + assert call_sites(extraction) == [("main", "helper", "L2")] + + graph = build([extraction], dedup=False, root=tmp_path) + main = next(node["id"] for node in extraction["nodes"] if node["label"] == "main") + helper = next(node["id"] for node in extraction["nodes"] if node["label"] == "helper") + assert graph.edges[main, helper]["relation"] == "calls" + + +def test_indexing_qualified_dynamic_and_handle_forms_are_not_calls(tmp_path): + result = extract_source(tmp_path, """function caller() + A = [1, 2]; + A(1); + A{1}; + indexed(1) = 2; + pkg.helper(); + obj.helper(); + feval('helper'); + callback = @helper; + anonymous = @(x) helper(x); + helper(); +end +function A() +end +function indexed() +end +function helper() +end +""") + assert call_sites(result) == [("caller", "helper", "L11")] + + +def test_parser_error_is_contained_to_its_callable_scope(tmp_path): + result = extract_source(tmp_path, """function broken() + helper(); + value = ; +end +function clean() + helper(); +end +function helper() +end +""") + assert {node["label"] for node in definitions(result)} == {"broken", "clean", "helper"} + assert call_sites(result) == [("clean", "helper", "L6")] + + +def test_file_paths_keep_same_named_definitions_distinct(tmp_path): + source = "function main()\n helper();\nend\nfunction helper()\nend\n" + left = extract_source(tmp_path, source, "left/main.m") + right = extract_source(tmp_path, source, "right/main.m") + left_ids = {node["id"] for node in definitions(left)} + right_ids = {node["id"] for node in definitions(right)} + assert left_ids.isdisjoint(right_ids) + assert call_sites(left) == [("main", "helper", "L2")] + assert call_sites(right) == [("main", "helper", "L2")] + + +def test_cell_indexing_is_never_a_call_even_without_a_local_binding(tmp_path): + result = extract_source(tmp_path, """function caller() + A{1}; +end +function A() +end +""") + assert call_sites(result) == [] diff --git a/tests/test_matlab_corpus.py b/tests/test_matlab_corpus.py new file mode 100644 index 0000000000..54902df689 --- /dev/null +++ b/tests/test_matlab_corpus.py @@ -0,0 +1,199 @@ +"""Source-reviewed JSONLab evidence for normal MATLAB extraction.""" +from collections import Counter +import hashlib +import json +from pathlib import Path +import shutil + +from graphify.extract import extract +from graphify.ids import make_id + + +CORPUS = Path(__file__).parent / "fixtures" / "matlab" / "jsonlab" +MANIFEST = json.loads((CORPUS / "expected.json").read_text(encoding="utf-8")) + + +def _stage_jsonlab(root: Path) -> Path: + root.mkdir(parents=True, exist_ok=True) + source = root / MANIFEST["contract"]["source_file"] + shutil.copyfile(CORPUS / source.name, source) + return source + + +def _extract(paths: list[Path], root: Path, cache_root: Path) -> dict: + return extract(paths, root=root, cache_root=cache_root, parallel=False) + + +def _assert_manifest_matches(result: dict, manifest: dict, source: bytes) -> None: + definitions = manifest["definitions"] + path = manifest["file"]["path"] + expected_by_signature = { + ( + definition["kind"], + definition["name"], + tuple(definition["scope_chain"]), + definition["source_location"], + ): definition["key"] + for definition in definitions + } + assert len(expected_by_signature) == len(definitions) + + nodes = result["nodes"] + assert len({node["id"] for node in nodes}) == len(nodes) + assert all((node.get("metadata") or {}).get("language") == "matlab" for node in nodes) + assert all(node.get("source_file") == path for node in nodes) + + file_nodes = [ + node for node in nodes + if (node.get("metadata") or {}).get("kind") == manifest["file"]["kind"] + ] + assert len(file_nodes) == 1 + file_node = file_nodes[0] + file_metadata = file_node["metadata"] + assert file_node["label"] == Path(path).name + assert file_metadata["matlab_file_kind"] == manifest["file"]["matlab_file_kind"] + assert file_metadata["matlab_primary_name"] == manifest["file"]["matlab_primary_name"] + + actual_by_key = {"file": file_node} + for node in nodes: + if node is file_node: + continue + metadata = node["metadata"] + signature = ( + metadata.get("kind"), + node.get("label"), + tuple(metadata.get("scope_chain", ())), + node.get("source_location"), + ) + assert signature in expected_by_signature, f"unexpected definition: {node}" + key = expected_by_signature[signature] + assert key not in actual_by_key, f"duplicate definition for manifest key {key}" + actual_by_key[key] = node + assert set(actual_by_key) == {"file", *(definition["key"] for definition in definitions)} + + source_lines = source.splitlines() + for definition in definitions: + line = source_lines[int(definition["source_location"][1:]) - 1] + assert line.decode("utf-8") == definition["source_text"] + assert line[definition["column"] - 1:].startswith(b"function ") + + expected_edges: Counter[tuple] = Counter() + for edge in manifest["contains"]: + expected_edges[("contains", edge["source"], edge["target"], edge["source_location"])] += 1 + + for call in manifest["calls"]: + line = source_lines[int(call["source_location"][1:]) - 1] + assert line.decode("utf-8") == call["source_text"] + target_name = next( + definition["name"] for definition in definitions + if definition["key"] == call["target"] + ) + assert line[call["column"] - 1:].startswith(target_name.encode("utf-8")) + expected_edges[( + "calls", call["source"], call["target"], call["source_location"], + "EXTRACTED", 1.0, + )] += 1 + + id_to_key = {node["id"]: key for key, node in actual_by_key.items()} + actual_edges: Counter[tuple] = Counter() + for edge in result["edges"]: + relation = edge.get("relation") + if relation not in {"contains", "method", "inherits", "calls"}: + continue + assert edge.get("source_file") == path, f"edge lost source provenance: {edge}" + assert edge.get("source") in id_to_key, f"unknown source endpoint: {edge}" + assert edge.get("target") in id_to_key, f"unknown target endpoint: {edge}" + record = ( + relation, + id_to_key[edge["source"]], + id_to_key[edge["target"]], + edge.get("source_location"), + ) + if relation == "calls": + record += (edge.get("confidence"), edge.get("weight")) + actual_edges[record] += 1 + + missing, unexpected = expected_edges - actual_edges, actual_edges - expected_edges + assert not missing and not unexpected, ( + f"manifest relationship mismatch; missing={dict(missing)}, unexpected={dict(unexpected)}" + ) + + +def test_jsonlab_corpus_matches_reviewed_manifest_through_normal_extraction(tmp_path): + source = _stage_jsonlab(tmp_path / "jsonlab") + source_bytes = source.read_bytes() + source_record = next( + record for record in MANIFEST["provenance"]["files"] + if record["path"] == MANIFEST["contract"]["source_file"] + ) + assert len(source_bytes) == source_record["size_bytes"] + assert hashlib.sha256(source_bytes).hexdigest() == source_record["sha256"] + + result = _extract([source], source.parent, tmp_path / "cache") + _assert_manifest_matches(result, MANIFEST, source_bytes) + + repeated = _extract([source], source.parent, tmp_path / "cache") + relocated_source = _stage_jsonlab(tmp_path / "relocated" / "jsonlab") + relocated = _extract( + [relocated_source], relocated_source.parent, tmp_path / "relocated-cache" + ) + assert _canonical_records(repeated) == _canonical_records(result) + assert _canonical_records(relocated) == _canonical_records(result) + + +def _stage_normalization_collisions(root: Path) -> list[Path]: + source = "function entry()\n Foo();\n foo();\nend\nfunction Foo()\nend\nfunction foo()\nend\n" + paths = [root / "pkg-one" / "main-v1.m", root / "pkg_one" / "main_v1.m"] + for path in paths: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(source, encoding="utf-8") + return paths + + +def _canonical_records(result: dict) -> tuple[tuple[str, ...], tuple[str, ...]]: + nodes = tuple(sorted(json.dumps({ + "id": node["id"], + "label": node.get("label"), + "source_file": node.get("source_file"), + "source_location": node.get("source_location"), + "metadata": node.get("metadata"), + }, sort_keys=True) for node in result["nodes"])) + edges = tuple(sorted(json.dumps({ + key: edge.get(key) + for key in ( + "source", "target", "relation", "source_file", "source_location", + "confidence", "weight", + ) + }, sort_keys=True) for edge in result["edges"])) + return nodes, edges + + +def test_normal_extraction_is_deterministic_across_repeats_relocation_and_order(tmp_path): + assert make_id("pkg-one/main-v1") == make_id("pkg_one/main_v1") + assert make_id("Foo") == make_id("foo") + + root = tmp_path / "first-root" / "repo" + paths = _stage_normalization_collisions(root) + forward = _extract(paths, root, tmp_path / "cache-forward") + repeated = _extract(paths, root, tmp_path / "cache-forward") + reverse = _extract(list(reversed(paths)), root, tmp_path / "cache-reverse") + + relocated_root = tmp_path / "elsewhere" / "repo" + relocated_paths = _stage_normalization_collisions(relocated_root) + relocated = _extract(relocated_paths, relocated_root, tmp_path / "cache-relocated") + + expected = _canonical_records(forward) + assert _canonical_records(repeated) == expected + assert _canonical_records(reverse) == expected + assert _canonical_records(relocated) == expected + + assert len(forward["nodes"]) == 8 + assert len({node["id"] for node in forward["nodes"]}) == 8 + calls = [edge for edge in forward["edges"] if edge.get("relation") == "calls"] + assert len(calls) == 4 + assert {node["source_file"] for node in forward["nodes"]} == { + "pkg-one/main-v1.m", "pkg_one/main_v1.m", + } + for path in ("pkg-one/main-v1.m", "pkg_one/main_v1.m"): + assert sum(node.get("source_file") == path for node in forward["nodes"]) == 4 + assert sum(edge.get("source_file") == path for edge in calls) == 2 diff --git a/tests/test_matlab_incremental.py b/tests/test_matlab_incremental.py new file mode 100644 index 0000000000..b21630926c --- /dev/null +++ b/tests/test_matlab_incremental.py @@ -0,0 +1,465 @@ +"""MATLAB inheritance refresh across raw and clustered incremental merges.""" +from __future__ import annotations + +import copy +import json +from pathlib import Path + +import pytest + +from graphify.build import ( + _refresh_matlab_inheritance, + build, + build_merge, + merge_raw_extraction, +) +from graphify.export import to_json +from graphify.extract import extract + + +MERGE_PATHS = ("raw", "clustered") + + +def _class_source(name: str, superclass: str | None = None, revision: int | None = None) -> str: + parent = f" < {superclass}" if superclass else "" + source = f"classdef {name}{parent}\n" + if revision is not None: + source += ( + " methods\n" + " function value = version(obj)\n" + f" value = {revision};\n" + " end\n" + " end\n" + ) + return source + "end\n" + + +def _write(root: Path, relative: str, source: str) -> Path: + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(source, encoding="utf-8") + return path + + +def _extract(root: Path, paths: list[Path], **kwargs) -> dict: + return extract(paths, root=root, cache_root=root.parent / "cache", parallel=False, **kwargs) + + +def _read_graph(path: Path) -> dict: + return json.loads(path.read_text(encoding="utf-8")) + + +def _edges(graph: dict) -> list[dict]: + return graph.get("links", graph.get("edges", [])) + + +def _write_raw(path: Path, graph: dict) -> None: + path.write_text( + json.dumps({key: graph[key] for key in ("nodes", "edges", "hyperedges") if key in graph}), + encoding="utf-8", + ) + + +def _start_graph(mode: str, root: Path, path: Path, files: list[Path]) -> None: + extracted = _extract(root, files) + if mode == "raw": + _write_raw(path, extracted) + else: + graph = build([extracted], dedup=False, root=root) + assert to_json(graph, {}, str(path), force=True) + + +def _merge( + mode: str, + root: Path, + path: Path, + changed: list[Path] | None = None, + deleted: list[Path] | None = None, +) -> dict: + changed_paths = changed or [] + fresh = _extract(root, changed_paths) if changed_paths else {"nodes": [], "edges": []} + prune = [str(item) for item in (deleted or [])] + if mode == "raw": + merged = merge_raw_extraction( + fresh, path, prune_sources=prune, root=root, ast_sources=changed_paths, + ) + _write_raw(path, merged) + else: + graph = build_merge( + [fresh] if changed_paths else [], path, prune_sources=prune, + dedup=False, root=root, ast_sources=changed_paths, + ) + assert to_json(graph, {}, str(path), force=True) + return _read_graph(path) + + +def _matlab_class(graph: dict, name: str, source_file: str) -> dict: + matches = [ + node for node in graph.get("nodes", []) + if node.get("label") == name + and node.get("source_file") == source_file + and node.get("metadata", {}).get("kind") == "matlab_class" + ] + assert len(matches) == 1, (name, source_file, matches) + return matches[0] + + +def _targets(graph: dict, child: dict) -> list[dict]: + by_id = {node["id"]: node for node in graph.get("nodes", [])} + return [ + by_id[edge["target"]] + for edge in _edges(graph) + if edge.get("relation") == "inherits" and edge.get("source") == child["id"] + ] + + +def _signature(graph: dict) -> tuple: + nodes = tuple(sorted(json.dumps(node, sort_keys=True) for node in graph.get("nodes", []))) + edges = tuple(sorted(json.dumps(edge, sort_keys=True) for edge in _edges(graph))) + return nodes, edges + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_changed_child_replaces_only_itself_and_keeps_context_read_only(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base", revision=1)) + child_path = _write(root, "Child.m", _class_source("Child", "Base", revision=1)) + _start_graph(mode, root, graph_path, [base_path, child_path]) + before = _read_graph(graph_path) + base_before = _matlab_class(before, "Base", "Base.m") + child_before = _matlab_class(before, "Child", "Child.m") + assert [node["label"] for node in _targets(before, child_before)] == ["Base"] + + context_nodes = copy.deepcopy([ + node for node in before["nodes"] if node.get("source_file") == "Base.m" + ]) + context_edges = copy.deepcopy([ + edge for edge in _edges(before) if edge.get("source_file") == "Base.m" + ]) + nodes_snapshot, edges_snapshot = copy.deepcopy(context_nodes), copy.deepcopy(context_edges) + child_path.write_text(_class_source("Child", "Base", revision=2), encoding="utf-8") + fresh = _extract( + root, [child_path], resolution_context_nodes=context_nodes, + resolution_context_edges=context_edges, + ) + assert context_nodes == nodes_snapshot + assert context_edges == edges_snapshot + assert {node.get("source_file") for node in fresh["nodes"]} == {"Child.m"} + + if mode == "raw": + merged = merge_raw_extraction( + fresh, graph_path, root=root, ast_sources=[child_path], + ) + _write_raw(graph_path, merged) + else: + graph = build_merge( + [fresh], graph_path, dedup=False, root=root, ast_sources=[child_path], + ) + assert to_json(graph, {}, str(graph_path), force=True) + after = _read_graph(graph_path) + + base_after = _matlab_class(after, "Base", "Base.m") + child_after = _matlab_class(after, "Child", "Child.m") + assert base_after["id"] == base_before["id"] + assert child_after["id"] == child_before["id"] + assert [node["label"] for node in _targets(after, child_after)] == ["Base"] + assert sum(node.get("source_file") == "Base.m" for node in after["nodes"]) == sum( + node.get("source_file") == "Base.m" for node in before["nodes"] + ) + + +def test_matlab_inheritance_refresh_normalizes_mixed_paths_without_mutating_inputs(tmp_path): + root = tmp_path / "src" + root.mkdir() + child_source = str(root / "Child.m") + nodes = [ + { + "id": "child_file", + "label": "Child.m", + "source_file": child_source, + "file_type": "code", + "metadata": { + "language": "matlab", "kind": "matlab_file", "namespace": "", + "matlab_file_kind": "class", "matlab_primary_name": "Child", + }, + }, + { + "id": "child", + "label": "Child", + "source_file": child_source, + "file_type": "code", + "metadata": { + "language": "matlab", "kind": "matlab_class", "namespace": "", + "scope_chain": [], + "matlab_superclasses": [{"name": "Base", "source_location": "L2"}], + }, + }, + { + "id": "base_file", + "label": "Base.m", + "source_file": "Base.m", + "file_type": "code", + "metadata": { + "language": "matlab", "kind": "matlab_file", "namespace": "", + "matlab_file_kind": "class", "matlab_primary_name": "Base", + }, + }, + { + "id": "base", + "label": "Base", + "source_file": "Base.m", + "file_type": "code", + "metadata": { + "language": "matlab", "kind": "matlab_class", "namespace": "", + "scope_chain": [], + }, + }, + ] + edges = [ + {"source": "base_file", "target": "base", "relation": "contains", "source_file": "Base.m"}, + { + "source": "child", "target": "stale", "relation": "inherits", + "source_file": child_source, "_origin": "ast", + }, + ] + before = copy.deepcopy((nodes, edges)) + + refreshed = _refresh_matlab_inheritance(nodes, edges, root=root) + + assert (nodes, edges) == before + assert [ + (edge["source"], edge["target"], edge["source_file"]) + for edge in refreshed if edge.get("relation") == "inherits" + ] == [("child", "base", "Child.m")] + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_absolute_matlab_reextract_resolves_relative_unchanged_base(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base")) + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph(mode, root, graph_path, [base_path, child_path]) + fresh = _extract(root, [child_path]) + for item in fresh["nodes"] + fresh["edges"]: + if item.get("source_file"): + item["source_file"] = str(root / item["source_file"]) + + if mode == "raw": + merged = merge_raw_extraction( + fresh, graph_path, root=root, ast_sources=[child_path], + ) + _write_raw(graph_path, merged) + else: + graph = build_merge( + [fresh], graph_path, dedup=False, root=root, ast_sources=[child_path], + ) + assert to_json(graph, {}, str(graph_path), force=True) + + after = _read_graph(graph_path) + child = next( + node for node in after["nodes"] + if node.get("label") == "Child" + and node.get("metadata", {}).get("kind") == "matlab_class" + ) + assert [node["label"] for node in _targets(after, child)] == ["Base"] + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_unchanged_child_keeps_inheritance_after_base_content_change(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base", revision=1)) + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph(mode, root, graph_path, [base_path, child_path]) + before = _read_graph(graph_path) + child_before = _matlab_class(before, "Child", "Child.m") + + base_path.write_text(_class_source("Base", revision=2), encoding="utf-8") + after = _merge(mode, root, graph_path, [base_path]) + + child_after = _matlab_class(after, "Child", "Child.m") + assert child_after["id"] == child_before["id"] + assert [node["label"] for node in _targets(after, child_after)] == ["Base"] + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_unchanged_child_drops_inheritance_when_base_is_renamed(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base")) + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph(mode, root, graph_path, [base_path, child_path]) + child_before = _matlab_class(_read_graph(graph_path), "Child", "Child.m") + + renamed_path = root / "Renamed.m" + renamed_path.write_text(_class_source("Renamed"), encoding="utf-8") + base_path.unlink() + after = _merge(mode, root, graph_path, [renamed_path], [base_path]) + + child_after = _matlab_class(after, "Child", "Child.m") + assert child_after["id"] == child_before["id"] + assert _targets(after, child_after) == [] + assert not any(node.get("source_file") == "Base.m" for node in after["nodes"]) + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_unchanged_child_resolves_when_base_is_added(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph(mode, root, graph_path, [child_path]) + child_before = _matlab_class(_read_graph(graph_path), "Child", "Child.m") + assert _targets(_read_graph(graph_path), child_before) == [] + + base_path = _write(root, "Base.m", _class_source("Base")) + after = _merge(mode, root, graph_path, [base_path]) + + child_after = _matlab_class(after, "Child", "Child.m") + assert child_after["id"] == child_before["id"] + assert [node["label"] for node in _targets(after, child_after)] == ["Base"] + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_unchanged_child_loses_inheritance_when_base_is_deleted(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base")) + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph(mode, root, graph_path, [base_path, child_path]) + base_path.unlink() + + after = _merge(mode, root, graph_path, deleted=[base_path]) + + child = _matlab_class(after, "Child", "Child.m") + assert _targets(after, child) == [] + assert not any(node.get("source_file") == "Base.m" for node in after["nodes"]) + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_newly_ambiguous_base_removes_inheritance(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "+pkg/Base.m", _class_source("Base")) + child_path = _write(root, "+pkg/Child.m", _class_source("Child", "pkg.Base")) + _start_graph(mode, root, graph_path, [base_path, child_path]) + child_before = _matlab_class(_read_graph(graph_path), "Child", "+pkg/Child.m") + assert [node["label"] for node in _targets(_read_graph(graph_path), child_before)] == ["Base"] + + duplicate = _write(root, "other/+pkg/Base.m", _class_source("Base")) + after = _merge(mode, root, graph_path, [duplicate]) + + child_after = _matlab_class(after, "Child", "+pkg/Child.m") + assert child_after["id"] == child_before["id"] + assert _targets(after, child_after) == [] + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_disambiguating_deleted_base_restores_inheritance(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + first = _write(root, "one/+pkg/Base.m", _class_source("Base")) + second = _write(root, "two/+pkg/Base.m", _class_source("Base")) + child_path = _write(root, "+pkg/Child.m", _class_source("Child", "pkg.Base")) + _start_graph(mode, root, graph_path, [first, second, child_path]) + before = _read_graph(graph_path) + child_before = _matlab_class(before, "Child", "+pkg/Child.m") + assert _targets(before, child_before) == [] + + first.unlink() + after = _merge(mode, root, graph_path, deleted=[first]) + + child_after = _matlab_class(after, "Child", "+pkg/Child.m") + targets = _targets(after, child_after) + assert len(targets) == 1 + assert targets[0].get("source_file") == "two/+pkg/Base.m" + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_deleted_child_removes_its_nodes_and_derived_inheritance(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base")) + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph(mode, root, graph_path, [base_path, child_path]) + child_path.unlink() + + after = _merge(mode, root, graph_path, deleted=[child_path]) + + assert not any(node.get("source_file") == "Child.m" for node in after["nodes"]) + assert not any(edge.get("source_file") == "Child.m" and edge.get("relation") == "inherits" + for edge in _edges(after)) + assert _matlab_class(after, "Base", "Base.m") + + +@pytest.mark.parametrize("mode", MERGE_PATHS) +def test_no_op_update_preserves_matlab_graph(tmp_path, mode): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base")) + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph(mode, root, graph_path, [base_path, child_path]) + before = _read_graph(graph_path) + + after = _merge(mode, root, graph_path) + + assert _signature(after) == _signature(before) + +def test_raw_graph_prune_refreshes_matlab_inheritance(tmp_path): + root = tmp_path / "src" + root.mkdir() + graph_path = tmp_path / "graph.json" + base_path = _write(root, "Base.m", _class_source("Base")) + child_path = _write(root, "Child.m", _class_source("Child", "Base")) + _start_graph("raw", root, graph_path, [base_path, child_path]) + before = _read_graph(graph_path) + child = _matlab_class(before, "Child", "Child.m") + assert [node["label"] for node in _targets(before, child)] == ["Base"] + + from graphify.cli import _prune_graph_json_sources + + assert _prune_graph_json_sources(graph_path, ["Base.m"]) > 0 + after = _read_graph(graph_path) + child = _matlab_class(after, "Child", "Child.m") + assert _targets(after, child) == [] + assert not any(node.get("source_file") == "Base.m" for node in after["nodes"]) + + +@pytest.mark.parametrize("no_cluster", [True, False], ids=["raw-cli", "clustered-cli"]) +def test_update_cli_refreshes_unchanged_matlab_child_after_base_change(tmp_path, no_cluster): + import os + import subprocess + import sys + + root = tmp_path / "project" + root.mkdir() + base = _write(root, "Base.m", _class_source("Base", revision=1)) + _write(root, "Child.m", _class_source("Child", "Base")) + options = ["--code-only"] + (["--no-cluster"] if no_cluster else []) + first = subprocess.run( + [sys.executable, "-m", "graphify", "extract", str(root), *options], + cwd=tmp_path, capture_output=True, text=True, env=os.environ.copy(), + ) + assert first.returncode == 0, first.stderr + base.write_text(_class_source("Base", revision=2), encoding="utf-8") + update_options = ["--no-cluster"] if no_cluster else [] + second = subprocess.run( + [sys.executable, "-m", "graphify", "update", str(root), *update_options], + cwd=tmp_path, capture_output=True, text=True, env=os.environ.copy(), + ) + assert second.returncode == 0, second.stderr + + graph = _read_graph(root / "graphify-out" / "graph.json") + child_node = _matlab_class(graph, "Child", "Child.m") + assert [node["label"] for node in _targets(graph, child_node)] == ["Base"] diff --git a/tests/test_matlab_inheritance.py b/tests/test_matlab_inheritance.py new file mode 100644 index 0000000000..749a842d6c --- /dev/null +++ b/tests/test_matlab_inheritance.py @@ -0,0 +1,229 @@ +"""The pure repository-source MATLAB inheritance contract (issue #4).""" +from copy import deepcopy + +import pytest + +from graphify.extractors.resolution import _matlab_inheritance_edges + + +def _class(source, name, *, namespace="", bases=(), repo=None): + common = {"file_type": "code", "source_file": source} + if repo is not None: + common["repo"] = repo + prefix = f"{repo}::" if repo is not None else "" + return [ + { + **common, + "id": f"{prefix}{source}:file", + "label": source, + "metadata": { + "language": "matlab", + "kind": "matlab_file", + "namespace": namespace, + "matlab_file_kind": "class", + "matlab_primary_name": name, + }, + }, + { + **common, + "id": f"{prefix}{source}:class:{name}", + "label": name, + "source_location": "L1", + "metadata": { + "language": "matlab", + "kind": "matlab_class", + "namespace": namespace, + "scope_chain": [], + "matlab_superclasses": [ + {"name": base, "source_location": f"L{row}"} + for row, base in enumerate(bases, start=2) + ], + }, + }, + ] + + +def test_exact_source_matches_preserve_written_bases_and_input_records(): + nodes = ( + _class("src/Child.m", "Child", bases=("tools.io.Base", "Sibling", "handle")) + + _class("lib/+tools/+io/Base.m", "Base", namespace="tools.io") + + _class("src/Sibling.m", "Sibling") + ) + original = deepcopy(nodes) + + assert _matlab_inheritance_edges(nodes) == [ + { + "source": "src/Child.m:class:Child", + "target": "lib/+tools/+io/Base.m:class:Base", + "relation": "inherits", + "confidence": "EXTRACTED", + "source_file": "src/Child.m", + "source_location": "L2", + "weight": 1.0, + }, + { + "source": "src/Child.m:class:Child", + "target": "src/Sibling.m:class:Sibling", + "relation": "inherits", + "confidence": "EXTRACTED", + "source_file": "src/Child.m", + "source_location": "L3", + "weight": 1.0, + }, + ] + assert nodes == original + + +@pytest.mark.parametrize( + ("child_path", "child_namespace", "base_path", "base_namespace", "written"), + [ + ("Child.m", "", "elsewhere/Base.m", "", "Base"), + ("Child.m", "", "+pkg/Base.m", "pkg", "Base"), + ("+pkg/Child.m", "pkg", "+pkg/Base.m", "pkg", "Base"), + ("Child.m", "", "+pkg/Base.m", "pkg", "pkg.base"), + ("Child.m", "", "+pkg/Base.m", "pkg", "Pkg.Base"), + ("Child.m", "", "base.m", "", "Base"), + ("Child.m", "", "private/+pkg/Base.m", "pkg", "pkg.Base"), + ("Child.m", "", "+pkg/@Base/Base.m", "", "pkg.Base"), + ("@Child/Child.m", "", "+pkg/Base.m", "pkg", "pkg.Base"), + ("private/Child.m", "", "+pkg/Base.m", "pkg", "pkg.Base"), + ("Child.m", "", "+outer/ordinary/+pkg/Base.m", "pkg", "outer.pkg.Base"), + ("Child.m", "", "+bad-name/Base.m", "bad-name", "bad-name.Base"), + ("Child.m", "", "Base.M", "", "Base"), + ], +) +def test_only_exact_namespace_or_unpacked_sibling_matches( + child_path, child_namespace, base_path, base_namespace, written +): + nodes = _class(child_path, "Child", namespace=child_namespace, bases=(written,)) + nodes += _class(base_path, "Base", namespace=base_namespace) + assert _matlab_inheritance_edges(nodes) == [] + + +def test_namespace_chain_stops_at_ordinary_directory(): + nodes = _class("Child.m", "Child", bases=("pkg.Base",)) + nodes += _class("+outer/ordinary/+pkg/Base.m", "Base", namespace="pkg") + edges = _matlab_inheritance_edges(nodes) + assert [(edge["source"], edge["target"]) for edge in edges] == [ + ("Child.m:class:Child", "+outer/ordinary/+pkg/Base.m:class:Base") + ] + + +def test_duplicate_package_roots_and_class_declarations_are_ambiguous(): + nodes = _class("Child.m", "Child", bases=("pkg.Base", "Sibling")) + nodes += _class("one/+pkg/Base.m", "Base", namespace="pkg") + nodes += _class("two/+pkg/Base.m", "Base", namespace="pkg") + nodes += _class("Sibling.m", "Sibling") + duplicate = deepcopy(nodes[-1]) + duplicate["id"] += ":L10" + duplicate["source_location"] = "L10" + nodes.append(duplicate) + assert _matlab_inheritance_edges(nodes) == [] + + +@pytest.mark.parametrize( + ("record", "field", "value"), + [ + (0, "matlab_file_kind", "script"), + (0, "matlab_primary_name", "Other"), + (0, "namespace", "other"), + (1, "namespace", "other"), + (1, "scope_chain", ["Owner"]), + (1, "scope_chain", None), + (1, "language", "objc"), + ], +) +def test_uncertain_primary_class_context_is_not_a_target(record, field, value): + child = _class("Child.m", "Child", bases=("pkg.Base",)) + base = _class("+pkg/Base.m", "Base", namespace="pkg") + base[record]["metadata"][field] = value + assert _matlab_inheritance_edges(child + base) == [] + + +def test_missing_file_context_and_filename_declaration_mismatch_fail_closed(): + child = _class("Child.m", "Child", bases=("Base",)) + base = _class("Base.m", "Base") + assert _matlab_inheritance_edges(child + base[1:]) == [] + assert _matlab_inheritance_edges(child + _class("Base.m", "Other")) == [] + assert _matlab_inheritance_edges(child + _class("Other.m", "Base")) == [] + + +def test_malformed_duplicate_package_root_cannot_make_another_root_unique(): + nodes = _class("Child.m", "Child", bases=("pkg.Base",)) + nodes += _class("one/+pkg/Base.m", "Base", namespace="pkg") + nodes += _class("two/+pkg/Base.m", "Wrong", namespace="pkg") + assert _matlab_inheritance_edges(nodes) == [] + + +def test_self_inheritance_and_unsupported_or_external_references_stay_metadata(): + nodes = _class( + "+pkg/Child.m", "Child", namespace="pkg", + bases=("pkg.Child", "handle", "matlab.mixin.Copyable", "pkg.Base(x)", "pkg..Base", " pkg.Base"), + ) + nodes[-1]["metadata"]["matlab_superclasses"].extend([ + {"raw": "pkg.Base{T}", "source_location": "L9"}, + {"name": "pkg.Base", "source_location": None}, + {"name": "pkg.Base", "source_location": "L0"}, + {"name": None, "source_location": "L12"}, + None, + ]) + nodes += _class("+pkg/Base.m", "Base", namespace="pkg") + original = deepcopy(nodes) + assert _matlab_inheritance_edges(nodes) == [] + assert nodes == original + + +def test_external_spelling_can_match_a_real_eligible_repository_class(): + nodes = _class("Child.m", "Child", bases=("handle", "matlab.mixin.Copyable")) + nodes += _class("handle.m", "handle") + nodes += _class("+matlab/+mixin/Copyable.m", "Copyable", namespace="matlab.mixin") + assert [edge["target"] for edge in _matlab_inheritance_edges(nodes)] == [ + "handle.m:class:handle", "+matlab/+mixin/Copyable.m:class:Copyable", + ] + + +def test_repository_tags_partition_identical_source_paths_and_namespaces(): + nodes = [] + for repo in (None, "one", "two"): + nodes += _class("Child.m", "Child", bases=("pkg.Base", "Base"), repo=repo) + nodes += _class("+pkg/Base.m", "Base", namespace="pkg", repo=repo) + nodes += _class("Base.m", "Base", repo=repo) + edges = _matlab_inheritance_edges(nodes) + assert {(edge["source"], edge["target"]) for edge in edges} == { + ("Child.m:class:Child", "+pkg/Base.m:class:Base"), + ("Child.m:class:Child", "Base.m:class:Base"), + ("one::Child.m:class:Child", "one::+pkg/Base.m:class:Base"), + ("one::Child.m:class:Child", "one::Base.m:class:Base"), + ("two::Child.m:class:Child", "two::+pkg/Base.m:class:Base"), + ("two::Child.m:class:Child", "two::Base.m:class:Base"), + } + assert _matlab_inheritance_edges(list(reversed(nodes))) == edges + assert _matlab_inheritance_edges(nodes[3:] + nodes[:3]) == edges + + +def test_foreign_repository_and_untagged_candidates_cannot_fill_missing_bases(): + nodes = _class("Child.m", "Child", bases=("pkg.Base", "Base"), repo="child") + nodes += _class("+pkg/Base.m", "Base", namespace="pkg", repo="foreign") + nodes += _class("Base.m", "Base") + assert _matlab_inheritance_edges(nodes) == [] + + +def test_nonsource_nodes_cannot_supply_class_targets(): + child = _class("Child.m", "Child", bases=("Base",)) + for field, value in ( + ("external", True), ("file_type", "document"), ("id", None), + ("metadata.kind", "matlab_function"), + ): + base = _class("Base.m", "Base") + if field == "metadata.kind": + base[-1]["metadata"]["kind"] = value + else: + base[-1][field] = value + assert _matlab_inheritance_edges(child + base) == [] + + +def test_non_file_nodes_cannot_establish_source_context(): + child = _class("Child.m", "Child", bases=("Base",)) + base = _class("Base.m", "Base") + base[0]["metadata"]["kind"] = "matlab_class" + assert _matlab_inheritance_edges(child + base) == [] diff --git a/tests/test_matlab_integration.py b/tests/test_matlab_integration.py new file mode 100644 index 0000000000..4cb0dafffa --- /dev/null +++ b/tests/test_matlab_integration.py @@ -0,0 +1,122 @@ +"""Normal MATLAB dispatch, remap, and persisted-build integration coverage.""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from graphify.build import build +from graphify.export import to_json +from graphify.extract import extract + + +def _extract(root: Path, paths: list[Path]) -> dict: + return extract(paths, root=root, cache_root=root, parallel=False) + + +@pytest.mark.parametrize(("config", "name", "source", "expected_label"), [ + ( + "language.m=objc\n", + "Mapped.m", + "@interface Mapped : NSObject\n@end\n", + "Mapped", + ), + ( + "language.m=python\n", + "mapped.m", + "def chosen():\n return 1\n", + "chosen()", + ), + ( + "language.inc=.m\n", + "Mapped.inc", + "@interface Mapped : NSObject\n@end\n", + "Mapped", + ), +]) +def test_language_remaps_keep_matlab_and_objc_dispatch_contract( + tmp_path: Path, config: str, name: str, source: str, expected_label: str, +) -> None: + (tmp_path / ".graphifyrc").write_text(config, encoding="utf-8") + path = tmp_path / name + path.write_text(source, encoding="utf-8") + + result = _extract(tmp_path, [path]) + + assert any(node.get("label") == expected_label for node in result["nodes"]) + assert result["nodes"], "the selected remap must reach a real language extractor" + + +def test_matlab_calls_and_exact_source_inheritance_survive_build(tmp_path: Path) -> None: + base = tmp_path / "Base.m" + child = tmp_path / "Derived.m" + calls = tmp_path / "Calls.m" + base.write_text("classdef Base\nend\n", encoding="utf-8") + child.write_text("classdef Derived < Base\nend\n", encoding="utf-8") + calls.write_text( + "function y = caller(x)\n" + " y = helper(x);\n" + "end\n" + "function y = helper(x)\n" + " y = x + 1;\n" + "end\n", + encoding="utf-8", + ) + + result = _extract(tmp_path, [base, child, calls]) + by_id = {node["id"]: node for node in result["nodes"]} + raw_call = next( + edge for edge in result["edges"] + if edge.get("relation") == "calls" + and by_id[edge["source"]].get("label") == "caller" + and by_id[edge["target"]].get("label") == "helper" + ) + raw_inherits = next( + edge for edge in result["edges"] + if edge.get("relation") == "inherits" + and by_id[edge["source"]].get("label") == "Derived" + and by_id[edge["target"]].get("label") == "Base" + ) + assert raw_call.get("source_file") == "Calls.m" + assert raw_call.get("source_location") == "L2" + assert raw_inherits.get("source_file") == "Derived.m" + + graph = build([result], dedup=False, root=tmp_path) + path = tmp_path / "built.json" + assert to_json(graph, {}, str(path), force=True) + stored = json.loads(path.read_text(encoding="utf-8")) + stored_nodes = {node["id"]: node for node in stored["nodes"]} + stored_edges = stored.get("links", stored.get("edges", [])) + relations = { + (stored_nodes[edge["source"]].get("label"), + stored_nodes[edge["target"]].get("label"), + edge.get("relation"), edge.get("source_file"), edge.get("source_location")) + for edge in stored_edges + } + assert ("caller", "helper", "calls", "Calls.m", "L2") in relations + assert ("Derived", "Base", "inherits", "Derived.m", "L1") in relations + +def test_pre_upgrade_matlab_cache_does_not_mask_current_extraction(tmp_path, monkeypatch): + from graphify import cache + + source = tmp_path / "controller.m" + source.write_text("function value = controller(input)\n value = input;\nend\n", encoding="utf-8") + cache_root = tmp_path / "cache" + current_schema = cache._AST_CACHE_SCHEMA + # The pre-MATLAB AST cache namespace is the persisted schema-25 namespace. + monkeypatch.setattr(cache, "_AST_CACHE_SCHEMA", 25) + cache.save_cached( + source, + {"nodes": [{"id": "stale_cache_node", "label": "stale MATLAB parse", + "source_file": "controller.m"}], "edges": []}, + root=tmp_path, + cache_root=cache_root, + ) + monkeypatch.setattr(cache, "_AST_CACHE_SCHEMA", current_schema) + + result = extract([source], root=tmp_path, cache_root=cache_root, parallel=False) + + labels = {node.get("label") for node in result["nodes"]} + assert "controller" in labels + assert "stale MATLAB parse" not in labels diff --git a/tests/test_matlab_merge_isolation.py b/tests/test_matlab_merge_isolation.py new file mode 100644 index 0000000000..9f11787869 --- /dev/null +++ b/tests/test_matlab_merge_isolation.py @@ -0,0 +1,284 @@ +"""MATLAB provenance isolates persisted relationships without hiding native matches.""" + +import networkx as nx +import pytest + +from graphify.affected import load_graph +from graphify.build import _refresh_matlab_inheritance, build, build_merge, prefix_graph_for_global +from graphify.cross_repo_calls import link_cross_repo_member_calls +from graphify.cross_repo_types import link_shared_type_declarations +from graphify.export import to_json + + +def _node(node_id, label, source, *, matlab=False, kind="function", **metadata): + if matlab: + metadata.update(language="matlab", kind=f"matlab_{kind}") + return { + "id": node_id, "label": label, "source_file": source, + "source_location": "L1", "file_type": "code", "type": "code", + "_callable": kind != "file", "_callable_class": kind == "class", + "metadata": metadata, + } + + +def _edge(source, target, relation, source_file, confidence="EXTRACTED"): + return { + "source": source, "target": target, "relation": relation, + "source_file": source_file, "source_location": "L3", + "confidence": confidence, "weight": 1.0, + } + + +def _persist(tmp_path, name, graph): + path = tmp_path / f"{name}.json" + to_json(graph, {}, str(path), force=True) + return path, load_graph(path).to_undirected() + + +@pytest.mark.parametrize("native_file", ["Foo.h", "Foo.m", "Foo.mm", "Foo.c"]) +@pytest.mark.parametrize("relation", ["calls", "imports", "imports_from", "references"]) +@pytest.mark.parametrize("reverse", [False, True]) +def test_persisted_matlab_native_homonyms_are_not_neighbors( + tmp_path, native_file, relation, reverse, +): + nodes = [ + _node("matlab", "Foo", "Foo.m", matlab=True), + _node("native", "Foo", native_file), + _node("native_target", "bar", "bar.m"), + ] + source, target = ("matlab", "native") if reverse else ("native", "matlab") + graph = build([{"nodes": nodes, "edges": [ + _edge(source, target, relation, nodes[1 if not reverse else 0]["source_file"], "INFERRED"), + _edge("native", "native_target", relation, native_file, "INFERRED"), + ]}], dedup=False) + _, reloaded = _persist(tmp_path, "mixed", graph) + assert set(reloaded.neighbors("matlab")) == set() + assert set(reloaded.neighbors("native")) == {"native_target"} + assert reloaded.edges["native", "native_target"]["relation"] == relation + + +@pytest.mark.parametrize( + ("source_file", "file_type"), + [("guide.md", "document"), ("settings.json", "code")], +) +def test_source_backed_docs_and_configs_reference_matlab(source_file, file_type): + graph = build([{ + "nodes": [ + { + "id": "source", "label": source_file, "source_file": source_file, + "file_type": file_type, "type": file_type, + }, + _node("main", "main", "main.m", matlab=True), + ], + "edges": [_edge("source", "main", "references", source_file)], + }], dedup=False) + + assert graph.edges["source", "main"]["relation"] == "references" + + +def _type_graph(*, matlab=False, source="Greeter.m", method="-greet"): + return build([{"nodes": [ + _node("greeter", "Greeter", source, matlab=matlab, kind="class", namespace="Shared"), + _node("greet", method, source, matlab=matlab, kind="method"), + ], "edges": [_edge("greeter", "greet", "method", source)]}], dedup=False) + + +@pytest.mark.parametrize("native_sources", [[], ["Greeter.m"], ["Greeter.mm"], ["Greeter.m", "Greeter.mm"]]) +def test_persisted_matlab_collision_does_not_answer_or_obscure_native_calls(tmp_path, native_sources): + caller = build([{"nodes": [_node("caller", "run", "app.m", unresolved_calls=[{ + "lang": "objc", "receiver_type": "Greeter", "callee": "greet", "line": "L3", + }])], "edges": []}], dedup=False) + _, caller = _persist(tmp_path, "caller", caller) + _, matlab = _persist(tmp_path, "matlab", _type_graph(matlab=True)) + graphs = [prefix_graph_for_global(caller, "app"), prefix_graph_for_global(matlab, "matlab")] + for index, source in enumerate(native_sources): + _, native = _persist(tmp_path, f"native{index}", _type_graph(source=source)) + graphs.append(prefix_graph_for_global(native, f"native{index}")) + merged = nx.compose_all(graphs) + link_cross_repo_member_calls(merged) + _, merged = _persist(tmp_path, "merged", merged) + expected = {"native0::greet"} if len(native_sources) == 1 else set() + assert set(merged.neighbors("app::caller")) == expected + assert set(merged.neighbors("matlab::greet")) == {"matlab::greeter"} + + +def test_namespace_matlab_classes_do_not_join_native_or_matlab_types(tmp_path): + graphs = [] + for repo, matlab, source in [ + ("matlab1", True, "+Shared/Greeter.m"), + ("matlab2", True, "+Shared/Greeter.m"), + ("native1", False, "Greeter.cs"), + ("native2", False, "Greeter.cs"), + ]: + _, graph = _persist(tmp_path, repo, _type_graph(matlab=matlab, source=source)) + graphs.append(prefix_graph_for_global(graph, repo)) + merged = nx.compose_all(graphs) + link_shared_type_declarations(merged) + _, merged = _persist(tmp_path, "merged", merged) + assert set(merged.neighbors("matlab1::greeter")) == {"matlab1::greet"} + assert set(merged.neighbors("matlab2::greeter")) == {"matlab2::greet"} + assert set(merged.neighbors("native1::greeter")) == {"native1::greet", "native2::greeter"} + assert merged.edges["native1::greeter", "native2::greeter"]["relation"] == "same_type_as" + + + +def test_matlab_inheritance_refresh_preserves_semantic_edges(): + child_source, base_source = "+Shared/Child.m", "+Shared/Base.m" + nodes = [ + _node( + "child_file", "Child.m", child_source, matlab=True, kind="file", + namespace="Shared", matlab_file_kind="class", matlab_primary_name="Child", + ), + _node( + "child", "Child", child_source, matlab=True, kind="class", + namespace="Shared", scope_chain=[], + matlab_superclasses=[{"name": "Shared.Base", "source_location": "L1"}], + ), + _node( + "base_file", "Base.m", base_source, matlab=True, kind="file", + namespace="Shared", matlab_file_kind="class", matlab_primary_name="Base", + ), + _node( + "base", "Base", base_source, matlab=True, kind="class", + namespace="Shared", scope_chain=[], + ), + ] + stale_ast = { + **_edge("child", "stale_base", "inherits", child_source), + "_origin": "ast", + } + semantic = { + **_edge("child", "semantic_base", "inherits", child_source), + "_origin": "semantic", + } + + refreshed = _refresh_matlab_inheritance(nodes, [stale_ast, semantic]) + + assert semantic in refreshed + assert stale_ast not in refreshed + assert [ + (edge["source"], edge["target"], edge["_origin"]) + for edge in refreshed + if edge.get("relation") == "inherits" and edge.get("_origin") == "ast" + ] == [("child", "base", "ast")] + + +def test_incremental_carry_forward_preserves_local_calls_and_source_inheritance(tmp_path): + nodes = [ + _node("child_file", "Child.m", "+Shared/Child.m", matlab=True, kind="file", + namespace="Shared", matlab_file_kind="class", matlab_primary_name="Child"), + _node("child", "Child", "+Shared/Child.m", matlab=True, kind="class", + namespace="Shared", scope_chain=[], + matlab_superclasses=[{"name": "Shared.Base", "source_location": "L1"}]), + _node("base_file", "Base.m", "+Shared/Base.m", matlab=True, kind="file", + namespace="Shared", matlab_file_kind="class", matlab_primary_name="Base"), + _node("base", "Base", "+Shared/Base.m", matlab=True, kind="class", + namespace="Shared", scope_chain=[]), + _node("run", "run", "+Shared/Child.m", matlab=True, kind="method"), + _node("helper", "helper", "+Shared/Child.m", matlab=True), + ] + edges = [ + _edge("child_file", "child", "contains", "+Shared/Child.m"), + _edge("base_file", "base", "contains", "+Shared/Base.m"), + _edge("child", "base", "inherits", "+Shared/Child.m"), + _edge("child", "run", "method", "+Shared/Child.m"), + _edge("run", "helper", "calls", "+Shared/Child.m"), + ] + path, _ = _persist(tmp_path, "base", build([{"nodes": nodes, "edges": edges}], dedup=False)) + updated = build_merge([{ + "nodes": [_node("native", "Child", "Child.h")], + "edges": [_edge("native", "child", "references", "Child.h", "INFERRED")], + }], path, dedup=False) + _, updated = _persist(tmp_path, "updated", updated) + native = build([{"nodes": [ + _node("child", "Child", "Child.cs", kind="class", namespace="Shared"), + ], "edges": []}], dedup=False) + merged = nx.compose(prefix_graph_for_global(updated, "matlab"), + prefix_graph_for_global(native, "native")) + link_cross_repo_member_calls(merged) + link_shared_type_declarations(merged) + _, merged = _persist(tmp_path, "merged", merged) + assert set(merged.neighbors("matlab::native")) == set() + assert set(merged.neighbors("matlab::run")) == {"matlab::child", "matlab::helper"} + assert set(merged.neighbors("matlab::child")) == { + "matlab::child_file", "matlab::run", "matlab::base", + } + assert merged.edges["matlab::run", "matlab::helper"]["confidence"] == "EXTRACTED" + assert merged.edges["matlab::child", "matlab::base"]["relation"] == "inherits" + assert merged.nodes["matlab::child"]["metadata"]["matlab_superclasses"] == [ + {"name": "Shared.Base", "source_location": "L1"}, + ] + assert merged.nodes["matlab::child_file"]["metadata"]["matlab_primary_name"] == "Child" + +def test_incremental_default_dedup_preserves_matlab_inheritance_with_reference_stub_collision( + tmp_path, +): + child_source, base_source = "+Shared/Child.m", "+Shared/Base.m" + nodes = [ + _node("child_file", "Child.m", child_source, matlab=True, kind="file", + namespace="Shared", matlab_file_kind="class", matlab_primary_name="Child"), + _node("child", "Child", child_source, matlab=True, kind="class", + namespace="Shared", scope_chain=[], + matlab_superclasses=[{"name": "Shared.Base", "source_location": "L1"}]), + _node("base_file", "Base.m", base_source, matlab=True, kind="file", + namespace="Shared", matlab_file_kind="class", matlab_primary_name="Base"), + _node("base", "Base", base_source, matlab=True, kind="class", + namespace="Shared", scope_chain=[]), + ] + edges = [ + _edge("child_file", "child", "contains", child_source), + _edge("base_file", "base", "contains", base_source), + _edge("child", "base", "inherits", child_source), + ] + path, _ = _persist( + tmp_path, "base", build([{"nodes": nodes, "edges": edges}], dedup=False), + ) + guide = _node("docs_guide", "Guide.md", "docs/Guide.md", kind="file") + reference_stub = _node( + "base", "Base", "docs/Guide.md", graphify_reference_stub=True, + ) + for node in (guide, reference_stub): + node.update(source_location=None, _origin="semantic") + reference = _edge("docs_guide", "base", "references", "docs/Guide.md") + reference.update(source_location=None, _origin="semantic") + + merged = build_merge([{ + "nodes": [guide, reference_stub], + "edges": [reference], + }], path, root=tmp_path) + + assert merged.nodes["base"]["source_file"] == base_source + assert merged.nodes["base"]["metadata"]["language"] == "matlab" + assert merged.nodes["docs_guide__ref__base"]["metadata"][ + "graphify_reference_stub" + ] is True + assert merged.edges["child", "base"]["relation"] == "inherits" + assert merged.edges["docs_guide", "docs_guide__ref__base"]["relation"] == "references" + + +def test_incremental_default_dedup_keeps_reference_edge_after_extension_id_remap( + tmp_path, +): + path, _ = _persist( + tmp_path, "base", + build([{ + "nodes": [_node("x", "x", "x.py", kind="class")], + "edges": [], + }], dedup=False), + ) + caller = _node("caller", "Caller", "x.html") + reference_stub = _node( + "x", "x page", "x.html", graphify_reference_stub=True, + ) + for node in (caller, reference_stub): + node.update(source_location=None, _origin="semantic") + reference = _edge("caller", "x", "references", "x.html") + reference.update(source_location=None, _origin="semantic") + + merged = build_merge([{ + "nodes": [caller, reference_stub], + "edges": [reference], + }], path, root=tmp_path) + + assert merged.nodes["x_html"]["source_file"] == "x.html" + assert merged.edges["caller", "x_html"]["relation"] == "references" diff --git a/tests/test_matlab_native_isolation.py b/tests/test_matlab_native_isolation.py new file mode 100644 index 0000000000..0ab149994c --- /dev/null +++ b/tests/test_matlab_native_isolation.py @@ -0,0 +1,251 @@ +"""MATLAB provenance never enters shared/native name resolution candidates.""" +from copy import deepcopy + +import pytest + +from graphify.extract import ( + _merge_swift_extensions, + _resolve_cpp_member_calls, + _resolve_objc_member_calls, + _resolve_swift_member_calls, + _rewire_unique_stub_nodes, + extract, +) + + +def _node(nid, label, source_file, *, matlab=False, **flags): + node = { + "id": nid, "label": label, "source_file": source_file, + "file_type": "code", "source_location": "L1", **flags, + } + if matlab: + node["metadata"] = {"language": "matlab"} + return node + + +@pytest.mark.parametrize("label", ["Widget", "helper()"]) +@pytest.mark.parametrize("native_present", [False, True]) +def test_stub_rewiring_excludes_matlab_before_uniqueness(label, native_present): + nodes = [ + _node("caller", "caller()", "caller.cpp"), + _node("stub", label.strip("()"), ""), + _node("matlab", label, "symbol.m", matlab=True), + ] + if native_present: + nodes.append(_node("native", label, "symbol.cpp")) + edges = [{"source": "caller", "target": "stub", "relation": "references", + "source_file": "caller.cpp"}] + + _rewire_unique_stub_nodes(nodes, edges) + + assert edges[0]["target"] == ("native" if native_present else "stub") + assert any(n["id"] == "matlab" for n in nodes) + + +def test_stub_rewiring_preserves_matlab_source_edges(): + nodes = [ + _node("matlab", "caller()", "caller.m", matlab=True), + _node("native", "Widget", "widget.cpp"), + _node("stub", "Widget", ""), + _node("caller", "caller()", "caller.cpp"), + ] + edges = [ + {"source": "matlab", "target": "stub", "relation": "references", + "source_file": "caller.m"}, + {"source": "caller", "target": "stub", "relation": "references", + "source_file": "caller.cpp"}, + ] + matlab_edge = deepcopy(edges[0]) + + _rewire_unique_stub_nodes(nodes, edges) + + assert edges[0] == matlab_edge + assert edges[1]["target"] == "native" + + +@pytest.mark.parametrize("language,resolver,suffix", [ + ("cpp", _resolve_cpp_member_calls, ".cpp"), + ("objc", _resolve_objc_member_calls, ".m"), + ("swift", _resolve_swift_member_calls, ".swift"), +]) +@pytest.mark.parametrize("native_present", [False, True]) +def test_native_member_candidates_exclude_matlab(language, resolver, suffix, native_present): + caller_file = "caller" + suffix + nodes = [ + _node("caller", "caller()", caller_file), + _node("matlab_class", "Widget", "Widget.m", matlab=True, _callable_class=True), + _node("matlab_method", ".work()", "Widget.m", matlab=True, _callable=True), + ] + edges = [ + {"source": "matlab_file", "target": "matlab_class", "relation": "contains"}, + {"source": "matlab_class", "target": "matlab_method", "relation": "method"}, + ] + if native_present: + nodes.extend([ + _node("native_class", "Widget", "native" + suffix, _callable_class=True), + _node("native_method", ".work()", "native" + suffix, _callable=True), + ]) + edges.extend([ + {"source": "native_file", "target": "native_class", "relation": "contains"}, + {"source": "native_class", "target": "native_method", "relation": "method"}, + ]) + original_edges = deepcopy(edges) + per_file = [{ + "raw_calls": [{"caller_nid": "caller", "callee": "work", "receiver": "Widget", + "is_member_call": True, "lang": language, "source_file": caller_file}], + language + "_type_table": {"path": caller_file, "table": {}}, + }] + + resolver(per_file, nodes, edges) + + assert edges[:len(original_edges)] == original_edges + calls = [e for e in edges if e.get("source") == "caller"] + assert [(e["relation"], e["target"]) for e in calls] == ( + [("calls", "native_method")] if native_present else [] + ) + + +@pytest.mark.parametrize("matlab_as_context", [False, True]) +def test_java_member_call_ignores_matlab_type_candidate(tmp_path, matlab_as_context): + services = tmp_path / "Services.java" + services.write_text("class PaymentGateway { static void ping() {} }\n") + checkout = tmp_path / "Checkout.java" + checkout.write_text( + "class Checkout { void run() { PaymentGateway.ping(); } }\n" + ) + matlab = tmp_path / "PaymentGateway.m" + matlab.write_text("classdef PaymentGateway\nend\n") + + context_nodes = [] + context_edges = [] + paths = [services, checkout, matlab] + if matlab_as_context: + matlab_result = extract( + [matlab], root=tmp_path, cache_root=tmp_path, parallel=False + ) + context_nodes = matlab_result["nodes"] + context_edges = matlab_result["edges"] + paths = [services, checkout] + + result = extract( + paths, + root=tmp_path, + cache_root=tmp_path, + parallel=False, + resolution_context_nodes=context_nodes, + resolution_context_edges=context_edges, + ) + nodes = result["nodes"] + context_nodes + + run = next( + node["id"] for node in nodes + if node.get("label") == ".run()" + and str(node.get("source_file", "")).endswith("Checkout.java") + ) + ping = next( + node["id"] for node in nodes + if node.get("label") == ".ping()" + and str(node.get("source_file", "")).endswith("Services.java") + ) + assert any( + node.get("label") == "PaymentGateway" + and (node.get("metadata") or {}).get("language") == "matlab" + for node in nodes + ) + calls = { + (edge.get("source"), edge.get("target")) + for edge in result["edges"] + if edge.get("relation") == "calls" + } + assert (run, ping) in calls + + +@pytest.mark.parametrize("native_present", [False, True]) +def test_swift_extensions_do_not_merge_into_matlab(native_present): + nodes = [ + _node("extension", "Widget", "extension.swift"), + _node("matlab", "Widget", "Widget.m", matlab=True), + ] + edges = [{"source": "matlab_file", "target": "matlab", "relation": "contains"}, + {"source": "extension", "target": "method", "relation": "method"}] + if native_present: + nodes.append(_node("native", "Widget", "Widget.swift")) + edges.append({"source": "native_file", "target": "native", "relation": "contains"}) + per_file = [{"swift_extensions": [{"nid": "extension", "label": "Widget"}]}] + + _merge_swift_extensions(per_file, nodes, edges) + + assert next(e for e in edges if e["relation"] == "method")["source"] == ( + "native" if native_present else "extension" + ) + assert any(n["id"] == "matlab" for n in nodes) + + +@pytest.mark.parametrize("native_location", ["absent", "fresh", "unchanged"]) +@pytest.mark.parametrize("suffix,caller_body,native_body,matlab_label", [ + (".c", "void caller() { helper(); }", "void helper() {}", "helper()"), + (".php", " str: + parent = f" < {superclass}" if superclass else "" + lines = [f"classdef {name}{parent}"] + if revision is not None: + lines.extend([ + " methods", + " function value = version(obj)", + f" value = {revision};", + " end", + " end", + ]) + return "\n".join([*lines, "end", ""]) + + +def _write(root: Path, relative: str, source: str) -> Path: + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(source, encoding="utf-8") + return path + + +def _graph(path: Path) -> dict: + return json.loads(path.read_text(encoding="utf-8")) + + +def _class(graph: dict, name: str, source_file: str) -> dict: + matches = [ + node for node in graph.get("nodes", []) + if node.get("label") == name + and node.get("source_file") == source_file + and node.get("metadata", {}).get("kind") == "matlab_class" + ] + assert len(matches) == 1, (name, source_file, matches) + return matches[0] + + +def _inheritance_targets(graph: dict, child: dict) -> list[dict]: + by_id = {node["id"]: node for node in graph.get("nodes", [])} + return [ + by_id[edge["target"]] + for edge in graph.get("links", graph.get("edges", [])) + if edge.get("relation") == "inherits" and edge.get("source") == child["id"] + ] + + +@pytest.mark.parametrize("no_cluster", [True, False], ids=["raw", "clustered"]) +def test_incremental_watch_refreshes_unchanged_child_for_base_lifecycle(tmp_path, no_cluster): + root = tmp_path / "project" + root.mkdir() + graph_path = root / "graphify-out" / "graph.json" + _write(root, "+pkg/Child.m", _class_source("Child", "pkg.Base")) + + assert _rebuild_code(root, no_cluster=no_cluster, acquire_lock=False) is True + initial = _graph(graph_path) + child = _class(initial, "Child", "+pkg/Child.m") + child_id = child["id"] + assert _inheritance_targets(initial, child) == [] + + base_path = _write(root, "+pkg/Base.m", _class_source("Base")) + assert _rebuild_code( + root, changed_paths=[base_path], no_cluster=no_cluster, acquire_lock=False + ) is True + added = _graph(graph_path) + child = _class(added, "Child", "+pkg/Child.m") + targets = _inheritance_targets(added, child) + assert child["id"] == child_id + assert [(node["label"], node.get("source_file")) for node in targets] == [ + ("Base", "+pkg/Base.m") + ] + + before_noop = graph_path.read_bytes() + assert _rebuild_code( + root, changed_paths=[], no_cluster=no_cluster, acquire_lock=False + ) is True + assert graph_path.read_bytes() == before_noop + + base_path.write_text(_class_source("Base", revision=1), encoding="utf-8") + assert _rebuild_code( + root, changed_paths=[base_path], no_cluster=no_cluster, acquire_lock=False + ) is True + changed = _graph(graph_path) + child = _class(changed, "Child", "+pkg/Child.m") + assert child["id"] == child_id + assert [node["label"] for node in _inheritance_targets(changed, child)] == ["Base"] + + duplicate_path = _write( + root, "other/+pkg/Base.m", _class_source("Base") + ) + assert _rebuild_code( + root, changed_paths=[duplicate_path], no_cluster=no_cluster, acquire_lock=False + ) is True + ambiguous = _graph(graph_path) + child = _class(ambiguous, "Child", "+pkg/Child.m") + assert child["id"] == child_id + assert _inheritance_targets(ambiguous, child) == [] + + duplicate_path.unlink() + assert _rebuild_code( + root, changed_paths=[duplicate_path], no_cluster=no_cluster, acquire_lock=False + ) is True + disambiguated = _graph(graph_path) + child = _class(disambiguated, "Child", "+pkg/Child.m") + targets = _inheritance_targets(disambiguated, child) + assert child["id"] == child_id + assert [(node["label"], node.get("source_file")) for node in targets] == [ + ("Base", "+pkg/Base.m") + ] + + base_path.unlink() + assert _rebuild_code( + root, changed_paths=[base_path], no_cluster=no_cluster, acquire_lock=False + ) is True + removed = _graph(graph_path) + child = _class(removed, "Child", "+pkg/Child.m") + assert child["id"] == child_id + assert _inheritance_targets(removed, child) == [] + assert not any(node.get("source_file") == "+pkg/Base.m" for node in removed["nodes"]) diff --git a/tests/test_skill_no_backtick_substitution.py b/tests/test_skill_no_backtick_substitution.py index ff72d7d511..5e6899be2c 100644 --- a/tests/test_skill_no_backtick_substitution.py +++ b/tests/test_skill_no_backtick_substitution.py @@ -174,7 +174,7 @@ def run_statements( (tmp_path / "graphify-out" / ".graphify_python").write_text(str(stub)) bad_exits = [] for stmt in stmts: - script = f"exec 9>>{trace}\nBASH_XTRACEFD=9\nPS4='+T '\nset -x\n{stmt}\n" + script = f"exec 9>>{trace} 2>&9\nBASH_XTRACEFD=9\nPS4='+T '\nset -x\n{stmt}\n" r = subprocess.run( [BASH, "-c", script], cwd=tmp_path, env={"PATH": str(bin_)}, capture_output=True, text=True, timeout=30, diff --git a/tests/test_watch.py b/tests/test_watch.py index 1b5dc4e8c1..482b4cbb59 100644 --- a/tests/test_watch.py +++ b/tests/test_watch.py @@ -1818,13 +1818,38 @@ def _watchdog_available() -> bool: return False +def _start_watch_thread(monkeypatch, watch_mod, watch_path, *, debounce): + import threading + from types import SimpleNamespace + + stop = threading.Event() + + def sleep(seconds): + if stop.wait(seconds): + raise KeyboardInterrupt + + monkeypatch.setattr( + watch_mod, + "time", + SimpleNamespace(sleep=sleep, monotonic=time.monotonic), + ) + monkeypatch.setenv("GRAPHIFY_REBUILD_TIMEOUT", "0") + thread = threading.Thread( + target=watch_mod.watch, + args=(watch_path,), + kwargs={"debounce": debounce}, + daemon=True, + ) + thread.start() + return thread, stop + + @pytest.mark.skipif(not _watchdog_available(), reason="watchdog not installed") def test_watch_handler_honors_graphifyignore(tmp_path, monkeypatch): """gh-928: the watch Handler must short-circuit paths matching .graphifyignore so busy volumes (node_modules churn, build artefacts, Time Machine writes, …) don't wake the rebuild pipeline. """ - import threading from graphify import watch as watch_mod watch_root = tmp_path / ".hidden-parent" / "corpus" @@ -1838,30 +1863,28 @@ def test_watch_handler_honors_graphifyignore(tmp_path, monkeypatch): monkeypatch.setattr(watch_mod, "_rebuild_code", lambda p, **kw: rebuild_calls.append(p) or True) monkeypatch.setattr(watch_mod, "_notify_only", lambda p: notify_calls.append(p)) - # Run watch() in a thread with a short debounce so we can verify the - # post-debounce dispatch path actually runs on real events. - t = threading.Thread( - target=watch_mod.watch, - args=(watch_root,), - kwargs={"debounce": 0.2}, - daemon=True, + t, stop = _start_watch_thread( + monkeypatch, watch_mod, watch_root, debounce=0.2 ) - t.start() - time.sleep(0.5) # let observer.start() settle - - # Ignored writes — handler must drop these. - (watch_root / "node_modules" / "junk.js").write_text("// noise\n", encoding="utf-8") - (watch_root / "build" / "out.py").write_text("x = 1\n", encoding="utf-8") - time.sleep(1.0) - assert rebuild_calls == [], "ignored writes triggered a rebuild" - assert notify_calls == [], "ignored writes triggered a notify" - - # Non-ignored write — handler must accept and (after debounce) dispatch. - (watch_root / "app.py").write_text("def f():\n return 1\n", encoding="utf-8") - deadline = time.monotonic() + 5.0 - while time.monotonic() < deadline and not rebuild_calls: - time.sleep(0.1) - assert rebuild_calls, "non-ignored .py write should have triggered _rebuild_code" + try: + time.sleep(0.5) # let observer.start() settle + + # Ignored writes — handler must drop these. + (watch_root / "node_modules" / "junk.js").write_text("// noise\n", encoding="utf-8") + (watch_root / "build" / "out.py").write_text("x = 1\n", encoding="utf-8") + time.sleep(1.0) + assert rebuild_calls == [], "ignored writes triggered a rebuild" + assert notify_calls == [], "ignored writes triggered a notify" + + # Non-ignored write — handler must accept and (after debounce) dispatch. + (watch_root / "app.py").write_text("def f():\n return 1\n", encoding="utf-8") + deadline = time.monotonic() + 5.0 + while time.monotonic() < deadline and not rebuild_calls: + time.sleep(0.1) + assert rebuild_calls, "non-ignored .py write should have triggered _rebuild_code" + finally: + stop.set() + t.join() @pytest.mark.skipif(not _watchdog_available(), reason="watchdog not installed") @@ -1870,7 +1893,6 @@ def test_watch_non_linux_handler_accepts_events_without_watched_entry( ): """#5075: macOS and Windows keep the recursive schedule, so their handler must accept file events even though the manual `watched` map is empty.""" - import threading from graphify import watch as watch_mod from watchdog import observers @@ -1892,6 +1914,11 @@ def schedule(self, handler, path, *, recursive=False, event_filter=None): def start(self): pass + def stop(self): + pass + + def join(self): + pass fake = FakeObserver() monkeypatch.setattr(observers, "Observer", lambda: fake) @@ -1902,24 +1929,23 @@ def start(self): ) monkeypatch.setattr(watch_mod, "_notify_only", lambda p: None) - thread = threading.Thread( - target=watch_mod.watch, - args=(watch_root,), - kwargs={"debounce": 0.1}, - daemon=True, + thread, stop = _start_watch_thread( + monkeypatch, watch_mod, watch_root, debounce=0.1 ) - thread.start() - - deadline = time.monotonic() + 5.0 - while time.monotonic() < deadline and "handler" not in captured: - time.sleep(0.01) - assert captured.get("recursive") is True - - captured["handler"].on_any_event(FileCreatedEvent(str(target))) - deadline = time.monotonic() + 5.0 - while time.monotonic() < deadline and not rebuild_calls: - time.sleep(0.05) - assert rebuild_calls, "the recursive handler dropped a real file event" + try: + deadline = time.monotonic() + 5.0 + while time.monotonic() < deadline and "handler" not in captured: + time.sleep(0.01) + assert captured.get("recursive") is True + + captured["handler"].on_any_event(FileCreatedEvent(str(target))) + deadline = time.monotonic() + 5.0 + while time.monotonic() < deadline and not rebuild_calls: + time.sleep(0.05) + assert rebuild_calls, "the recursive handler dropped a real file event" + finally: + stop.set() + thread.join() @pytest.mark.skipif(not _watchdog_available(), reason="watchdog not installed") @@ -1928,7 +1954,6 @@ def test_watch_loads_graphifyignore_once(tmp_path, monkeypatch): not per filesystem event. Otherwise busy volumes re-read the file thousands of times per second. """ - import threading from graphify import watch as watch_mod from graphify import detect as detect_mod @@ -1947,15 +1972,20 @@ def counting_loader(root, **kwargs): monkeypatch.setattr(watch_mod, "_rebuild_code", lambda p, **kw: True) monkeypatch.setattr(watch_mod, "_notify_only", lambda p: None) - t = threading.Thread(target=watch_mod.watch, args=(tmp_path,), kwargs={"debounce": 0.2}, daemon=True) - t.start() - time.sleep(0.5) + t, stop = _start_watch_thread( + monkeypatch, watch_mod, tmp_path, debounce=0.2 + ) + try: + time.sleep(0.5) - # Generate many events; loader must not be called again. - for i in range(50): - (tmp_path / "ignored" / f"f{i}.py").write_text("x\n", encoding="utf-8") - time.sleep(0.7) - assert calls["n"] == 1, f"_load_graphifyignore called {calls['n']} times; expected 1" + # Generate many events; loader must not be called again. + for i in range(50): + (tmp_path / "ignored" / f"f{i}.py").write_text("x\n", encoding="utf-8") + time.sleep(0.7) + assert calls["n"] == 1, f"_load_graphifyignore called {calls['n']} times; expected 1" + finally: + stop.set() + t.join() # --- _check_shrink: silent-corruption guard with explicit-deletion bypass --- diff --git a/uv.lock b/uv.lock index 82928363c5..a374fa6cce 100644 --- a/uv.lock +++ b/uv.lock @@ -1365,7 +1365,7 @@ wheels = [ [[package]] name = "graphifyy" -version = "0.9.106" +version = "0.9.107" source = { editable = "." } dependencies = [ { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, @@ -1392,6 +1392,7 @@ dependencies = [ { name = "tree-sitter-julia" }, { name = "tree-sitter-kotlin" }, { name = "tree-sitter-lua" }, + { name = "tree-sitter-matlab" }, { name = "tree-sitter-objc" }, { name = "tree-sitter-php" }, { name = "tree-sitter-powershell" }, @@ -1661,6 +1662,7 @@ requires-dist = [ { name = "tree-sitter-language-pack", marker = "extra == 'erlang'", specifier = "==0.11.0" }, { name = "tree-sitter-language-pack", marker = "extra == 'r'", specifier = "==0.11.0" }, { name = "tree-sitter-lua", specifier = ">=0.2,<0.6" }, + { name = "tree-sitter-matlab", specifier = "==1.3.1" }, { name = "tree-sitter-objc", specifier = ">=3.0,<4.0" }, { name = "tree-sitter-ocaml", marker = "extra == 'all'" }, { name = "tree-sitter-ocaml", marker = "extra == 'ocaml'" }, @@ -5984,6 +5986,30 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8c/97/3104ecfa3c34320411bcad9b4f2823956487b6e222edcc83689819badc9d/tree_sitter_lua-0.5.0-cp310-abi3-win_arm64.whl", hash = "sha256:8488f3bea40779896f5771bcfcdc26900eb21e94f6658eb68a848fc37dd39221", size = 23506, upload-time = "2026-02-26T17:07:32.775Z" }, ] +[[package]] +name = "tree-sitter-matlab" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c6/04/be954b2edb5b57648d91d4412e225e361e9c37ac22ce6c45d78713a9b744/tree_sitter_matlab-1.3.1.tar.gz", hash = "sha256:91a2ab61a6179b8aec0216f18f1c1eb4e1409b768641b9c661bab434648bb54d", size = 172140, upload-time = "2026-08-26T06:47:41.047Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/ec/d7f27aa3dd0aed3a9bcdfd3f765d28ee4fba1b2473ca8d3e4c3f39eff78e/tree_sitter_matlab-1.3.1-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:dd080613d88e184c389a8026ff176c3b5a30cd164000ae0f121d5086794afa89", size = 86528, upload-time = "2026-08-26T06:47:19.22Z" }, + { url = "https://files.pythonhosted.org/packages/05/ce/5e8f5fb357426ec96762a3f063e053a8fefcfde7981f1f28016068f7f2e4/tree_sitter_matlab-1.3.1-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:78f79557f6be5e98d3ba4aa76f655efaab1d5c21dea4127ccd7fe25e20415cf4", size = 116326, upload-time = "2026-08-26T06:47:20.768Z" }, + { url = "https://files.pythonhosted.org/packages/d7/f6/0b44ab0452eec5a3d21c33009e58a3447e93a16545d1fa6f2ab491f6c163/tree_sitter_matlab-1.3.1-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:52ada1a896da00cc6e3c918e5548e0b22dd5dcc3cf2ecf061135af45de13d74a", size = 123180, upload-time = "2026-08-26T06:47:22.128Z" }, + { url = "https://files.pythonhosted.org/packages/76/8a/35d394daf205c758a9a0cb375638e5945c290b4200f7b80479d27479acc6/tree_sitter_matlab-1.3.1-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:14e4386c054507d55d0f5d7d88e53b990967a0ef29146fcc77af0ed5995b49e4", size = 120807, upload-time = "2026-08-26T06:47:23.46Z" }, + { url = "https://files.pythonhosted.org/packages/7c/ea/bccd684fc87b7d4ae6b5375405c97a71518c3d1ab401cc7f29ca0b8d8cd6/tree_sitter_matlab-1.3.1-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6a789cf1ed450965f28c476a305c407ff2f0692a336d72fd8ea86a9221b6701f", size = 115012, upload-time = "2026-08-26T06:47:24.79Z" }, + { url = "https://files.pythonhosted.org/packages/77/6b/e0fc1b2ecd8e33c11ef82dacaa8e4f48c768bff29b987dce16206166b726/tree_sitter_matlab-1.3.1-cp310-abi3-win32.whl", hash = "sha256:bf2f24efcceb670828d4a8d82c5a551bd77e4752b0f2913d72145fc2b5380de9", size = 81100, upload-time = "2026-08-26T06:47:25.987Z" }, + { url = "https://files.pythonhosted.org/packages/52/5b/f3151a2c407c0776a8c6d22d06bc727c813d755e3dc7cda3819f7f8ae2e9/tree_sitter_matlab-1.3.1-cp310-abi3-win_amd64.whl", hash = "sha256:505d5efdcbd1d76b7134eb7e79727265135c92f2cadbe119930604b5c65fdb46", size = 81914, upload-time = "2026-08-26T06:47:27.409Z" }, + { url = "https://files.pythonhosted.org/packages/b3/94/55f785a562baf807fddffe4d0208138fb73351eee2a20fd0dc3f65a795bc/tree_sitter_matlab-1.3.1-cp310-abi3-win_arm64.whl", hash = "sha256:325b4af0332ed2e13158d866245974f9f61ad817be159d5245692abaff3bb34e", size = 80980, upload-time = "2026-08-26T06:47:28.728Z" }, + { url = "https://files.pythonhosted.org/packages/b3/d3/330d3500e0323d56367375fb07b41182ab86fc1f45be0f086250fca23b59/tree_sitter_matlab-1.3.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a4c06388d5334ef98d2f7cb40f93d578f3eb66cd8316a5305a59757c256d9060", size = 86587, upload-time = "2026-08-26T06:47:30.205Z" }, + { url = "https://files.pythonhosted.org/packages/2f/b4/23fccc18ca1c8640476795223e49fe95cfe557f07bde10fd4b82e3dc043a/tree_sitter_matlab-1.3.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:5bd17ef2ad965867d70af8e6be245e6886e5e8d115f66a2aa155380884a65252", size = 119143, upload-time = "2026-08-26T06:47:31.565Z" }, + { url = "https://files.pythonhosted.org/packages/1b/43/2d5eb367db854e41417c2dfc1ca549e00f0d6de14dbbcd2221b572831136/tree_sitter_matlab-1.3.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bde730be2a387afc3308ae2aa720ba8d11f91c5c8183acadf149bc427c63bcf8", size = 125836, upload-time = "2026-08-26T06:47:32.819Z" }, + { url = "https://files.pythonhosted.org/packages/a6/6a/e7def13d2f86ba6ae735dd3ea76d62ebc726d8d7dd626e3db1691fb322fb/tree_sitter_matlab-1.3.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:955acf4ad3e7b9a4ec84e14dd8d6c42596a45d36d5f6d1a8fd9fdc68b2970dc3", size = 123558, upload-time = "2026-08-26T06:47:34.073Z" }, + { url = "https://files.pythonhosted.org/packages/2b/b3/69bde9098b0fb3b605e975b4cc9144202fe45a8617e704bf9bff45b17e87/tree_sitter_matlab-1.3.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:3173e14a34e82f13304cd4ae066aad7250bc73b0c855e2be96e889673b8e1f5c", size = 117824, upload-time = "2026-08-26T06:47:35.384Z" }, + { url = "https://files.pythonhosted.org/packages/91/df/24297ff094dbf66e4aebd53a877302b793412748c291481c409c616bdd44/tree_sitter_matlab-1.3.1-cp314-cp314t-win32.whl", hash = "sha256:5afe8ee1ecf3314bb7e111b34ae70bd53289b417059866c145c6e11193a594c3", size = 83702, upload-time = "2026-08-26T06:47:36.8Z" }, + { url = "https://files.pythonhosted.org/packages/04/b4/c02832e71e806bff22d872d168f454d95e38da97d9516f284f662c7f6e27/tree_sitter_matlab-1.3.1-cp314-cp314t-win_amd64.whl", hash = "sha256:b678a165ea792dc3a731580d7edeb838d29c5c95b475fe8bc840752afdc2e9b8", size = 84173, upload-time = "2026-08-26T06:47:38.156Z" }, + { url = "https://files.pythonhosted.org/packages/99/91/71d6470bbe5433c1320b0060b66a30e4b100eb64f4c034bed97790966283/tree_sitter_matlab-1.3.1-cp314-cp314t-win_arm64.whl", hash = "sha256:d13490be96978603f03d9decbb3e2216108b97b3ab434a7709c00278a5afacfd", size = 83503, upload-time = "2026-08-26T06:47:39.579Z" }, +] + [[package]] name = "tree-sitter-objc" version = "3.0.2"