Skip to content

docs: worked example — Chilean civil-law corpus (code + Spanish legal doctrine, AST-only) - #3662

Open
elpabloultron wants to merge 1 commit into
Graphify-Labs:v8from
elpabloultron:docs/worked-example-civil-law-chile
Open

elpabloultron wants to merge 1 commit into
Graphify-Labs:v8from
elpabloultron:docs/worked-example-civil-law-chile

Conversation

@elpabloultron

Copy link
Copy Markdown

docs: worked example — Chilean civil-law corpus (code + Spanish legal doctrine, AST-only)

A worked example that follows the contribution guide: a real corpus, the output saved under
worked/civil-law-chile/, and an honest review.md covering what the graph got right and wrong.

Corpus: 4 Python modules + 4 Markdown legal documents (a doctrine index and three Chilean
Academia Judicial training guides) — 8 files, ~3,010 words, 52 KB. Half of it is Spanish prose
doctrine with accents, legal citations and emoji-prefixed headings, which is not what most
corpora in here look like.

Run: graphify update . on 0.9.63, AST-only path — no LLM, no API key, 0 tokens.
Result: 72 nodes, 91 edges, 9 communities, 100% EXTRACTED.

What the review reports, including the uncomfortable parts:

  • The Markdown structure parse is more useful than I expected. The four legal documents
    contribute 21 document nodes and 17 rationale nodes with no LLM involved, and three of the
    ten god nodes are legal guides. For a prose-heavy corpus that is real navigation at zero cost.
  • Cross-language retrieval works: the query token savings (English) started the traversal at
    the Spanish heading ⚡ 3. Estándar de Optimización de Tokens.
  • But substance is out of reach on this path. Queries phrased the way a lawyer asks
    (plazo para contestar la demanda en el juicio oral laboral) reach the correct document, never
    the rule. The summary line "100% EXTRACTED" can be read as "the corpus is in the graph" when the
    prose was only taken to heading level.
  • The corpus-size verdict looks inconsistent with a sibling example: this 3,010-word corpus is
    reported as "large enough that graph structure adds value", while worked/mixed-corpus reports
    the opposite warning at ~4,020 words.
  • Smaller notes: emoji prefixes end up verbatim in heading labels, node ids keep accents and the
    full heading text, and "Surprising Connections" surfaced nothing surprising on a corpus this
    size.

The corpus is a slice of Open Legal Chile (Apache-2.0), trimmed to keep the example small — the
full corpus is 58 documents. The evaluation was executed by an AI agent (Hermes Agent) in a live
session; every number in review.md comes from the run whose graph.json and GRAPH_REPORT.md
are included.

Corpus mixto real: 4 modulos Python + 4 documentos legales chilenos en espanol (un indice de
doctrina y tres guias de la Academia Judicial), 8 archivos / ~3.010 palabras.

Corrida con graphify 0.9.63, solo AST (sin LLM, 0 tokens): 72 nodos, 91 aristas, 9 comunidades,
100% EXTRACTED. El review.md reporta tanto lo que funciono (los documentos aportan 21 nodos
'document' sin LLM y una consulta en ingles aterriza en un encabezado en espanol) como lo que no
(las preguntas sustantivas de abogado llegan al documento, no a la regla).
@elpabloultron

Copy link
Copy Markdown
Author

Ran the suite as the workflow asks (uv run pytest tests/ -q) before/while opening this, and
reporting the result since it is not what the guide implies:

15 failed, 5626 passed, 97 skipped in 64.71s

This PR introduces none of them. I checked the baseline on a clean v8 (same clone, my commit
not applied — 0 files under worked/): 15 failed, 5626 passed, 97 skipped, and the failing test
ids are byte-for-byte the same set:

  • tests/test_ollama_retry_cap.py — 4 failures (test_ollama_defaults_to_zero_sdk_retries,
    test_ollama_honors_explicit_max_retries, test_cloud_backend_keeps_default_retries,
    test_api_timeout_is_passed_to_client)
  • tests/test_skillgen.py — 11 failures (test_audit_coverage_passes,
    test_audit_coverage_passes_for_codex_and_windows, test_all_progressive_hosts_check_and_audit_clean,
    test_monolith_roundtrip_passes_for_aider_and_devin, test_monoliths_change_only_sanctioned_lines,
    test_always_on_roundtrip_is_byte_faithful, test_audit_coverage_passes_for_every_split_host,
    test_audit_catches_an_induced_per_host_drop, test_audit_catches_a_dropped_non_allowlisted_heading,
    test_amp_audit_coverage_passes_against_its_own_v8, test_agents_audit_baseline_is_amps_v8_body)

So v8 is currently 15 tests from green on its own — flagging it in case that is unexpected on your
side. Nothing here touches code: the change is one new folder under worked/ plus its review.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.


Graphify review — findings

Adds a generated knowledge-graph artifact for the civil-law-chile corpus: GRAPH_REPORT.md summarizing 72 nodes/91 edges across 9 communities with god nodes, cross-community bridges, and knowledge gaps, plus the underlying graph.json node/edge data. Both files are new AST-extracted outputs covering the corpus's code modules (LegalGraphBuilder, ColdStartInterviewEngine, stats_tracker.py, config.py) and doctrine documents.

Worth a look

  • safe_urlopen permits SSRF to loopback/private hostsworked/civil-law-chile/raw/code/config.py:87 · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • get_suite_adoption_metrics silently substitutes hardcoded 138 institutions on any DB errorworked/civil-law-chile/raw/code/stats_tracker.py:118 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Telemetry defaults to enabled (opt-out) violating stated privacy postureworked/civil-law-chile/raw/code/stats_tracker.py:148 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • check_configuration reports stale credentials after load_env_fileworked/civil-law-chile/raw/code/config.py:80 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Ollama health check ignores configured OLLAMA_HOSTworked/civil-law-chile/raw/code/config.py:55 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 91 functions depend on the 91 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 91 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 91 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

286 of 286 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/test_affected_cli.py — full-run-safety
  • tests/test_affected_member_seed.py — full-run-safety
  • tests/test_agents_platform.py — full-run-safety
  • tests/test_analyze.py — full-run-safety
  • tests/test_anthropic_custom_endpoint.py — full-run-safety
  • tests/test_antigravity_install.py — full-run-safety
  • tests/test_apm_fallback_version.py — full-run-safety
  • tests/test_architecture_doc.py — full-run-safety
  • tests/test_astro_extraction.py — full-run-safety
  • tests/test_astro_import_ids.py — full-run-safety
  • tests/test_atomic_canvas_export.py — full-run-safety
  • tests/test_atomic_version_stamp.py — full-run-safety
  • tests/test_atomic_writes.py — full-run-safety
  • tests/test_backend_env_isolation.py — full-run-safety
  • tests/test_backend_extras.py — full-run-safety
  • tests/test_benchmark.py — full-run-safety
  • tests/test_benchmark_raw_graph.py — full-run-safety
  • tests/test_build.py — full-run-safety
  • tests/test_build_merge_dedup_scope.py — full-run-safety
  • tests/test_build_merge_hyperedges_and_prune.py — full-run-safety
  • tests/test_build_merge_shrink_guard.py — full-run-safety
  • tests/test_builtin_global_type_refs.py — full-run-safety
  • tests/test_cache.py — full-run-safety
  • tests/test_callflow_html.py — full-run-safety
  • tests/test_cargo_introspect.py — full-run-safety
  • tests/test_carried_hyperedge_remap.py — full-run-safety
  • tests/test_case_sensitive_resolution.py — full-run-safety
  • tests/test_charmap_encoding.py — full-run-safety
  • tests/test_chunking.py — full-run-safety
  • tests/test_cjs_module_extension.py — full-run-safety
  • tests/test_claude_cli_backend.py — full-run-safety
  • tests/test_claude_md.py — full-run-safety
  • tests/test_cli_broken_pipe.py — full-run-safety
  • tests/test_cli_export.py — full-run-safety
  • tests/test_cli_help.py — full-run-safety
  • tests/test_cluster.py — full-run-safety
  • tests/test_codebuddy.py — full-run-safety
  • tests/test_community_hub_labels.py — full-run-safety
  • tests/test_community_labels_skill.py — full-run-safety
  • tests/test_confidence.py — full-run-safety
  • tests/test_corrupt_graph_json.py — full-run-safety
  • tests/test_cpp_nested_and_cli.py — full-run-safety
  • tests/test_cpp_objc_cross_file_calls.py — full-run-safety
  • tests/test_cpp_preprocess.py — full-run-safety
  • tests/test_cross_extension_reexport_self_cycle.py — full-run-safety
  • tests/test_cross_language_call_resolution.py — full-run-safety
  • tests/test_cross_repo_external_call_guards.py — full-run-safety
  • tests/test_cross_repo_member_calls.py — full-run-safety
  • tests/test_cross_repo_shared_types.py — full-run-safety
  • tests/test_csharp_call_site_generic_args.py — full-run-safety
  • … and 236 more

non-code file(s) changed (worked/civil-law-chile/GRAPH_REPORT.md, worked/civil-law-chile/graph.json, worked/civil-law-chile/raw/doctrine/README.md, worked/civil-law-chile/raw/doctrine/aj_conciliacion_laboral.md, worked/civil-law-chile/raw/doctrine/aj_etica_judicial.md …) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (worked/civil-law-chile/GRAPH_REPORT.md, worked/civil-law-chile/raw/code/cold_start.py, worked/civil-law-chile/raw/code/config.py, worked/civil-law-chile/raw/code/grafo_vinculos.py, worked/civil-law-chile/raw/code/stats_tracker.py …) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant