Diagram-sourced reaction connectivity (A→product→B) to fill precedingEvent gaps

[adamjohnwright]

Problem

The generator connects two reactions only when they are linked by precedingEvent. But older pathways under-annotate precedingEvent, so reactions where A's product is literally B's substrate are left disconnected.

Measured gap (shared whole-entity product→substrate pairs with no precedingEvent):

• Mitotic G1/S (modern): 13
• Amino-acid metabolism: 118
• Glycolysis: 24 of 43
• Metabolism (large): 591

Fix

Use the Reactome diagram JSON (release 97) as the authoritative curator-drawn connectivity. For each entity glyph shared between reaction A's output and reaction B's input, merge that shared product into a single node → A → product → B (NOT a reaction→reaction edge; the product carries the signal). Reuses the existing Phase-2 entity-merge; diagram pairs are fed in alongside precedingEvent.

Why the diagram (vs raw Neo4j dbId matching): curators draw cofactors (ATP/ADP/H2O) as separate per-reaction glyphs, so shared-glyph connectivity excludes cofactors for free — no hub threshold to tune. Verified on Mit_G1: shared-glyph carriers are all real species (CCNE:CDK2, DREAM complex, RB1, …), zero cofactors.

Diagrams rsync'd to ~/reactome-diagrams/97/ (R-HSA-<id>.json + .graph.json).

Validation

• Confirm the 9 modern benchmark pathways don't regress (their gap is tiny).
• Show recovered flows on an old pathway (Glycolysis R-HSA-70171).
• Note: gain is coverage/faithfulness for the broad catalog, not a 9-pathway score move (no experimental data for old pathways).

Do NOT retry (dead-ends, benchmarked net-negative)

• Component hand-off edges: bridging reactions on a shared component (not whole entity) — naive explodes (77k edges), max-shared false-couples (−7), hub-guarded (−4). You can't get the reconnected paths without equal coupling error. Kept as env-gated dead code (LNG_HANDOFF_EDGES=0).
• All-complex assembly edges: slight regression, reverted.

[adamjohnwright]

Implemented + validated (2026-07-14)

src/diagram_connectivity.py extracts shared-glyph product→substrate pairs from the release-97 diagram JSON and unions them into reaction_connections (Phase-2 then merges the shared product → A → product → B). Env: LNG_DIAGRAM_CONNECTIVITY (default on), LNG_DIAGRAM_DIR (default ~/reactome-diagrams/97). Only the connectivity/merge step sees the extra pairs; the matching layer stays on pure precedingEvent.

Diagram pairs added (product→substrate missing precedingEvent): ERBB2 +19, PIP3 +13, HDR +13, Mit_G1 +6, S_Phase +6, CCC +5, Mit_Prophase +2, TP53 0, WNT 0. Old pathway (Glucose metabolism R-HSA-70326): +53.

9-pathway experimental: 69.28% → 69.55%, no regression. Driven by ERBB2 73.5% → 79.6%: clean recovery (no_path 6→4, propagator_missed 6→4, pass 36→39, false_positive 1→2) — net +3 correct, +1 fp. Everything else neutral. Schema-valid. Cofactor-clean by construction (curators draw ATP/etc. as separate glyphs).

Contrast with the reverted component-hand-off (net-negative, mostly false-positives): whole-entity curator-drawn connectivity is specific, not promiscuous.

Follow-up

Pathways without their own diagram (e.g. Glycolysis R-HSA-70171) inherit an ancestor's diagram — the current code no-ops on them. To cover those, use the nearest diagrammed ancestor and filter to the pathway's reactions. ~1382 human pathways have their own diagram.

[adamjohnwright]

Clean (noise-free) confirmation — 2026-07-14

Isolated the pure diagram effect by regenerating ERBB2 at fixed PYTHONHASHSEED=0, diagram off vs on (only difference = diagram-merged connectivity):

• OFF: 36/49 = 73.5%
• ON: 39/49 = 79.6% (no_path 6→4, propagator_missed 6→4, pass 36→39, false_positive 1→2) — net +3.

The unseeded 9-pathway "flat" result was partly masked by generator non-determinism (#42): the apparent "TP53 −1" was two networks differing by ~1852 edges from hash-seed ordering alone (TP53 has 0 diagram edges). Other pathways are truly neutral — their added connections don't lie on tested gene→readout paths.

Action: run all generation + benchmarking with PYTHONHASHSEED=0 (see #42) so A/B comparisons aren't confounded by ±1–2 cases of regeneration noise.

[adamjohnwright]

Ancestor-diagram support added (follow-up resolved)

We mostly generate top-level pathways (own diagram), but test pathways are often sub-pathways that inherit an ancestor's diagram. diagram_connectivity.py now:

1. Uses the pathway's own diagram if present, else the nearest diagrammed ancestor (walk hasEvent up in Neo4j, pick nearest with a diagram file).
2. Isolates the target pathway within that ancestor diagram by keeping only shared-glyph pairs whose both reactions are in the pathway's own reaction set (hasEvent* → ReactionLikeEvent).

Verified:

• Mit_G1 (own diagram): unchanged, +6 pairs (filter is a no-op).
• Glycolysis R-HSA-70171 (no own diagram): now uses ancestor R-HSA-70326 (Glucose metabolism), +11 product→substrate pairs (of 29 drawn), all endpoints confirmed to be Glycolysis reactions. Schema-valid. (Was 0 before — silently skipped.)