Skip to content

perf(neo4j): fix O(N²) edge inserts via labeled MATCH + UNWIND batching - #2258

Open
kingjin94 wants to merge 1 commit into
Graphify-Labs:v8from
kingjin94:v8
Open

perf(neo4j): fix O(N²) edge inserts via labeled MATCH + UNWIND batching#2258
kingjin94 wants to merge 1 commit into
Graphify-Labs:v8from
kingjin94:v8

Conversation

@kingjin94

Copy link
Copy Markdown

Problem

push_to_neo4j was catastrophically slow on large graphs (5+ hours for ~40k nodes/edges in production). Two compounding issues:

  1. Per-row round-trips: each node/edge was a separate session.run() auto-commit transaction (~77k Bolt round-trips for a medium repo).

  2. Label-less edge MATCH (the critical one): the edge query was:

    MATCH (a {id: row.src}), (b {id: row.tgt})

    With no label, Neo4j ignores all uniqueness constraints and scans the full node store for every row — O(N²) regardless of indexes or UNWIND batching.

Fix

  1. Group nodes by label and edges by (src_label, tgt_label, rel), send as UNWIND batches of 500 via execute_write().
  2. Include labels in the edge MATCH by building an id→label map during the node pass:
    MATCH (a:Code {id: row.src}), (b:Document {id: row.tgt})
    This hits the per-label b-tree index (requires a uniqueness constraint on id per label, which callers should create before pushing).
  3. Log progress every 60s so long pushes aren't silent.

Benchmark

Ephemeral Neo4j 5, no pre-existing data:

Strategy 10k nodes / 20k edges
baseline (per-row session.run) 88s
UNWIND + constraint, label-less MATCH 71s (barely better)
UNWIND + constraint + labeled MATCH 1.5s (50x faster)

Production: 212k nodes + 398k edges pushed in ~24s.

Notes

  • The label-less MATCH fix is the dominant improvement; UNWIND batching alone barely moves the needle because it just reduces transaction overhead while the per-row full-scan cost remains.
  • Callers should CREATE CONSTRAINT label_id_unique IF NOT EXISTS FOR (n:Label) REQUIRE n.id IS UNIQUE for each label before pushing to get the full benefit.

Two compounding issues made push_to_neo4j catastrophically slow on large
graphs (5+ hours for ~40k nodes/edges):

1. Per-row round-trips: each node/edge was a separate session.run()
   auto-commit transaction (~77k Bolt round-trips for a medium repo).
   Fixed by grouping nodes/edges by label/rel-type and sending UNWIND
   batches of 500 via execute_write().

2. Label-less edge MATCH (the critical one): the edge query was
   `MATCH (a {id: row.src}), (b {id: row.tgt})` with no label, so Neo4j
   ignored all uniqueness constraints and scanned the full node store for
   every row — O(N²) regardless of indexes.
   Fixed by building an id→label map from the node pass and grouping edges
   by (src_label, tgt_label, rel), so the MATCH becomes
   `MATCH (a:Code {id: row.src}), (b:Document {id: row.tgt})` which hits
   the per-label b-tree index.

Benchmark (ephemeral Neo4j 5, no pre-existing data):
  10k nodes / 20k edges:
    baseline (per-row):               88s
    UNWIND + constraint, no label:    71s   (barely better)
    UNWIND + constraint + label:       1.5s  (50x faster)

Production result: 212k nodes + 398k edges pushed in ~24s.
@kingjin94
kingjin94 marked this pull request as ready for review July 28, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant