Repository navigation
feat(matlab): add isolated AST extraction and source inheritance - #4836
JulioLeonPhd wants to merge 21 commits into
Conversation
Record skill conventions and link the migrated MATLAB decision map. Mark fork-specific tracker guidance for exclusion from the upstream feature PR. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Use normal extractor dispatch and preserve exact-source inheritance lifecycle on schema 26. Add extraction, language, inheritance, merge isolation, incremental, integration, and watch coverage. Refs #10 Co-Authored-By: OpenAI Codex <noreply@openai.com>
Check normal extraction against the independently reviewed JSONLab manifest and provenance, plus repeat, order, and relocation determinism. Document the conservative MATLAB fallback, calls, inheritance, and persistence contract. Refs: #12 Co-authored-by: OpenAI Codex <noreply@openai.com>
Address final specification review: retain lexical calls inside member arguments and documentation/config citations, and normalize inheritance refresh context without mutating inputs. Add consumer-visible regressions. Centrally verified 203 focused tests, Ruff, generated artifacts and distribution build; type diagnostics match the 656-error baseline. Full suite: 8893 passed with one unchanged failure reproduced at 109cef1. Refs: #12 Co-Authored-By: OpenAI Codex <noreply@openai.com>
Keep the centrally verified member-argument call and source-backed documentation/config reference regressions with the review corrections. Final focused gate: 203 passed. Refs: #12 Co-Authored-By: OpenAI Codex <noreply@openai.com>
# Conflicts: # graphify/cache.py # uv.lock
Remove unrelated agent-workflow guidance from the upstream delta; preserve local copies outside the tracked tree. Regenerate the lock with upstream-compatible uv 0.11.19, retaining existing dependency versions and removing resolver-format churn. Trim extra test EOF whitespace while preserving byte-pinned JSONLab sources and license. Verified 1093 focused regressions with one expected xfail, frozen sync, lock check, Ruff, 134 generated artifacts, real corpus extraction/query, and AST-only graph update. Full-suite and Pyright upstream debt remains disclosed in local maintenance evidence. Co-Authored-By: OpenAI Codex <noreply@openai.com>
|
Thanks for the pull request, @JulioLeonPhd. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic. |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; formal verification proved 0 of 14 changed function(s) (5 sampled, not proven, 9 not verified, not every changed file confirmed checked).
Formal verification. PR-changed functions: 5/14 verified (0 proven, 5 may-equivalent, 0 distinguished) · 9 not verified (6 vacuous, 3 unsupported).
Not verified on this run: build\_from\_json (vacuous: never exercised), build\_merge (vacuous: never exercised), merge\_raw\_extraction (unsupported), \_prune\_graph\_json\_sources (unsupported), \_merge\_swift\_extensions (vacuous: never exercised), \_resolve\_cpp\_member\_calls (vacuous: never exercised), \_resolve\_objc\_member\_calls (vacuous: never exercised), \_resolve\_swift\_member\_calls (vacuous: never exercised), \_rebuild\_code (unsupported).
Graphify review — findings
Adds MATLAB extraction for .m files: explicit .graphifyrc remaps win, then the Objective-C content detector, and any .m not detected as Objective-C falls back to MATLAB, while .mm stays Objective-C. Emits file, class, and function nodes with calls edges only for uniquely resolved same-file functions and inherits edges only for unique exact source matches, leaving script top-level, cross-file, and member calls unlinked. In build_from_json, calls, imports, and references edges between MATLAB and non-MATLAB nodes are skipped unless the other end is a source-backed doc or config, merges re-derive MATLAB inherits edges from the current nodes, and the AST cache schema bump forces pre-MATLAB .m results to re-extract.
Worth a look
- MATLAB provenance gate drops same-file
references/importsedges to nodes that lack MATLAB metadata —graphify/build.py:1872· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 6659 functions depend on the 2401 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
extract()— 1066 callers, 64 callees - worse:
_rebuild_code()— 207 callers, 66 callees - worse:
build_merge()— 107 callers, 18 callees - worse:
to_json()— 94 callers, 11 callees - worse:
save_cached()— 57 callers, 10 callees - worse:
build()— 95 callers, 6 callees - worse:
link_cross_repo_member_calls()— 39 callers, 11 callees - worse:
_rewire_unique_stub_nodes()— 17 callers, 17 callees - …and 22 more — each is listed as a finding
Verification — 6659 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 6390 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
477 of 477 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/pi.test.ts— full-run-safetytests/test_affected_cli.py— impact, full-run-safetytests/test_affected_defines.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_affected_overrides.py— impact, full-run-safetytests/test_affected_path_file_node.py— impact, full-run-safetytests/test_agents_platform.py— impact, full-run-safetytests/test_alias_folding.py— impact, full-run-safetytests/test_analyze.py— impact, full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— impact, full-run-safetytests/test_astro_import_ids.py— impact, full-run-safetytests/test_atomic_canvas_export.py— impact, full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— impact, full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_backend_opt_in.py— impact, full-run-safetytests/test_benchmark.py— impact, full-run-safetytests/test_benchmark_raw_graph.py— impact, full-run-safetytests/test_benchmark_real_query.py— impact, full-run-safetytests/test_binary_media_words.py— impact, full-run-safetytests/test_blade_extractor.py— impact, full-run-safetytests/test_build.py— impact, full-run-safetytests/test_build_located_semantic_identity.py— impact, full-run-safetytests/test_build_merge_dedup_scope.py— impact, full-run-safetytests/test_build_merge_hyperedges_and_prune.py— impact, full-run-safetytests/test_build_merge_shrink_guard.py— impact, full-run-safetytests/test_builtin_global_type_refs.py— impact, full-run-safetytests/test_c_include_resolution.py— impact, full-run-safetytests/test_cache.py— impact, full-run-safetytests/test_cache_long_paths.py— impact, full-run-safetytests/test_cache_stale_import_target.py— impact, full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_cargo_missing_manifest.py— full-run-safetytests/test_carried_hyperedge_remap.py— impact, full-run-safetytests/test_case_sensitive_resolution.py— impact, full-run-safetytests/test_charmap_encoding.py— impact, full-run-safetytests/test_chunking.py— impact, full-run-safetytests/test_cjs_module_extension.py— impact, full-run-safetytests/test_claude_cli_backend.py— impact, full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— impact, full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cloud_cta.py— impact, full-run-safety- … and 427 more
non-code file(s) changed (
ARCHITECTURE.md,README.md,pyproject.toml,tests/fixtures/matlab/jsonlab/LICENSE_BSD.txt,tests/fixtures/matlab/jsonlab/expected.json…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
ARCHITECTURE.md,README.md,graphify/extractors/__init__.py,pyproject.toml,tests/fixtures/matlab/frontend/scopes.m…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
.github/PULL_REQUEST_TEMPLATE.md§ How was this tested? (lines 25-32): references changed symbolstext.github/PULL_REQUEST_TEMPLATE.md§ Graphify-specific checklist (lines 33-39): references changed symbolsrunBENCHMARKS.md§ Reproducing (lines 190-203): references changed symbolsrun,sameCHANGELOG.md§ 0.9.97 (2026-10-27) (lines 5-30): references changed symbolslocal,nested,run,textCHANGELOG.md§ 0.9.96 (2026-10-25) (lines 31-61): references changed symbolsrunCHANGELOG.md§ 0.9.95 (2026-10-23) (lines 62-87): references changed symbolsotherCHANGELOG.md§ 0.9.94 (2026-10-22) (lines 88-109): references changed symbolsFoo,fooCHANGELOG.md§ 0.9.93 (2026-10-20) (lines 110-136): references changed symbolssibling,textCHANGELOG.md§ 0.9.90 (2026-10-13) (lines 176-194): references changed symbolsother,runCHANGELOG.md§ 0.9.88 (2026-10-12) (lines 209-233): references changed symbolsclean,foo
…and 10 more.
Formal verification
Could not verify: Could not verify build\_from\_json.
The verifier did not have enough to check build\_from\_json, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 11 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous
Could not verify: Could not verify build\_merge.
The verifier did not have enough to check build\_merge, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 224 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify merge\_raw\_extraction.
The verifier did not have enough to check merge\_raw\_extraction, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: parameter `prune_sources` is annotated `'list[str] | None'` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_prune\_graph\_json\_sources.
The verifier did not have enough to check \_prune\_graph\_json\_sources, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: the input domain has 83 values but only 11 distinct were tested — a small finite domain must be EXHAUSTED, not sampled (an untested input could invert the result)
No difference found (not proven): No behavior difference found in \_index\_declarations (not a proof).
The verifier ran both versions of \_index\_declarations on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
No difference found (not proven): No behavior difference found in link\_shared\_type\_declarations (not a proof).
The verifier ran both versions of link\_shared\_type\_declarations on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
No difference found (not proven): No behavior difference found in \_get\_extractor (not a proof).
The verifier ran both versions of \_get\_extractor on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_merge\_swift\_extensions.
The verifier did not have enough to check \_merge\_swift\_extensions, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: non-vacuity: domain too small (only 1 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
Could not verify: Could not verify \_resolve\_cpp\_member\_calls.
The verifier did not have enough to check \_resolve\_cpp\_member\_calls, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: non-vacuity: domain too small (only 1 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
Could not verify: Could not verify \_resolve\_objc\_member\_calls.
The verifier did not have enough to check \_resolve\_objc\_member\_calls, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: non-vacuity: domain too small (only 1 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
Could not verify: Could not verify \_resolve\_swift\_member\_calls.
The verifier did not have enough to check \_resolve\_swift\_member\_calls, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: non-vacuity: domain too small (only 1 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
No difference found (not proven): No behavior difference found in \_rewire\_unique\_stub\_nodes (not a proof).
The verifier ran both versions of \_rewire\_unique\_stub\_nodes on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
No difference found (not proven): No behavior difference found in extract (not a proof).
The verifier ran both versions of extract on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_rebuild\_code.
The verifier did not have enough to check \_rebuild\_code, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: nondeterministic output (both versions disagreed with themselves on the witness) — not a behaviour change; a function whose output is not a function of its inputs (random / time / uuid / os.urandom, or state carried across calls) cannot be soundly distinguished by execution, so this tier abstains for the whole function
· 19 grounded finding(s) anchored inline below; 11 more finding(s) on lines outside this diff (see the check run).
| return out | ||
|
|
||
|
|
||
| def _refresh_matlab_inheritance( |
There was a problem hiding this comment.
_refresh_matlab_inheritance()
10 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| primary: bool = False | ||
|
|
||
|
|
||
| def extract_matlab(path: Path) -> dict: |
There was a problem hiding this comment.
extract_matlab()
fans out to 15 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| }) | ||
| return facts | ||
|
|
||
| def collect(syntax, owner: _Scope, chain: list[str], in_methods: bool = False) -> None: |
There was a problem hiding this comment.
collect()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| ) | ||
|
|
||
|
|
||
| def _start_graph(mode: str, root: Path, path: Path, files: list[Path]) -> None: |
There was a problem hiding this comment.
_start_graph()
11 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert to_json(graph, {}, str(path), force=True) | ||
|
|
||
|
|
||
| def _merge( |
There was a problem hiding this comment.
_merge()
fans out to 6 callees (efferent coupling); 8 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
|
|
||
| @pytest.mark.parametrize("mode", MERGE_PATHS) | ||
| def test_no_op_update_preserves_matlab_graph(tmp_path, mode): |
There was a problem hiding this comment.
test_no_op_update_preserves_matlab_graph()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
| assert _signature(after) == _signature(before) | ||
|
|
||
| def test_raw_graph_prune_refreshes_matlab_inheritance(tmp_path): |
There was a problem hiding this comment.
test_raw_graph_prune_refreshes_matlab_inheritance()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
|
|
||
| @pytest.mark.parametrize("native_sources", [[], ["Greeter.m"], ["Greeter.mm"], ["Greeter.m", "Greeter.mm"]]) | ||
| def test_persisted_matlab_collision_does_not_answer_or_obscure_native_calls(tmp_path, native_sources): |
There was a problem hiding this comment.
test_persisted_matlab_collision_does_not_answer_or_obscure_native_calls()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| ] == [("child", "base", "ast")] | ||
|
|
||
|
|
||
| def test_incremental_carry_forward_preserves_local_calls_and_source_inheritance(tmp_path): |
There was a problem hiding this comment.
test_incremental_carry_forward_preserves_local_calls_and_source_inheritance()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
|
|
||
| @pytest.mark.parametrize("no_cluster", [True, False], ids=["raw", "clustered"]) | ||
| def test_incremental_watch_refreshes_unchanged_child_for_base_lifecycle(tmp_path, no_cluster): |
There was a problem hiding this comment.
test_incremental_watch_refreshes_unchanged_child_for_base_lifecycle()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
Thanks for the review. The provenance mismatch rejection is intentional because The CI workflow currently shows |
|
Hi! Friendly ping on this PR when you have the chance. The MATLAB AST extraction is complete and CI is green. It addresses the existing requests for MATLAB AST support (#1702, #2935, discussion #1863). This is my first contribution to Graphify, so I'm happy to make any changes if needed to get it review/merge ready. Thanks! |
Merge upstream/v8 at 06aa51b for PR Graphify-Labs#4836, retaining current extraction, cache and lifecycle behavior. Advance the AST cache schema to 45 and regenerate the dependency lock. Preserve MATLAB inheritance across semantic-reference collisions without rewriting native edges early. Exclude MATLAB candidates from Java member resolution and suppress calls in evalc-affected workspaces. Update the dispatch contract and isolate watcher and shell-tracing test harnesses. Verified full pytest suites on Python 3.10, 3.12, 3.13 and 3.14 (11,257 passed each), five generated-skill checks, Pi guards, disposable-home CLI installation, and an external private MATLAB AST build/update scenario. No private sources or scenario artifacts included. Co-Authored-By: OpenAI Codex <noreply@openai.com>
|
Updated this branch by merging the current Local verification passed on Python 3.10, 3.12, 3.13, and 3.14: 11,257 tests passed per version. An external real-source MATLAB scenario also passed extraction, deterministic persistence, and incremental/no-op updates; no private sources were committed. Hosted checks are now running. Thanks for taking another look! |
Summary
Preserves
.mm,.inc=.m, native-language resolution and source-backed document/config references. Full extraction and incremental build/watch refresh inheritance against surviving source context. Adds the pinned MATLAB grammar, independently reviewed JSONLab corpus, regression coverage and documented boundaries; no new graph schema or force-language option.Merged upstream
v8at06aa51bein commit1e214e6, resolving the integration conflicts while retaining upstream extraction and lifecycle behavior. The integration also fixes MATLAB candidates interfering with Java member resolution, suppresses guessed calls inevalc-affected workspaces, and preserves inheritance across semantic-reference collisions without prematurely rewriting native edges. AST cache schema advances to 45 and the dependency lock is regenerated.Evidence
0a8cd4e1, an isolated CLI extraction of the pinned JSONLabloadjson.mreports.mwithout an extractor, produces no nodes and exits 1.After: The original PR smoke at
a24491c2exits 0 with 14 nodes / 31 persisted edges; CLI query retrievesparse_valueat L409 and its call toparse_arrayat L420. This is historical feature evidence; current integration verification follows.1e214e6: Python 3.10, 3.12, 3.13 and 3.14 each passed 11,257 tests, with 133 skipped and one expected failure. The formerly failing shell-security control now supports macOS Bash 3.2 tracing without weakening its assertions. Watcher test threads are cleaned up to prevent cross-test leakage..mfiles, copied into an external disposable workspace, produced 31 definitions and 38 calls. API extraction, deterministic persistence, CLI build, a comment-only incremental update and a subsequent no-op update passed. A separate synthetic two-class fixture verified inheritance. No private sources, paths, scenario harness or artifacts are included in the contribution; MATLAB code was not executed.Full-suite command, run in isolated frozen environments with disposable homes:
Merge Danger
Door: two-way
Revert the contribution and regenerate AST/graph outputs; no destructive data migration. The AST cache schema bump intentionally invalidates older AST entries, causing re-extraction; cache keys also include grammar versions.
Blast Radius: Extraction
Changes non-Objective-C
.mfallback plus shared extraction, build and incremental-resolution boundaries. Native candidate isolation and lifecycle regressions cover those boundaries. MATLAB relationships deliberately omit uncertain bindings, cross-file/member/qualified/script-top-level calls and runtime-path guesses; separate@Classmethod association and MATLAB-wide runtime compatibility are not claimed. Unresolved superclass text remains metadata, not fabricated targets.JSONLab source/license are pinned and unmodified, including original whitespace. No local graph data or fork workflow setup is included. Material AI assistance is disclosed in implementation and cleanup commit trailers.