Skip to content

[spike] storage-backed index: skip chain rebuild + drop edges map - #44

Merged
hungpham10 merged 3 commits into
mainfrom
spike/storage-backed-symbols
Oct 7, 2026
Merged

hungpham10 merged 3 commits into
mainfrom
spike/storage-backed-symbols

Conversation

@hungpham10

@hungpham10 hungpham10 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Mục tiêu

Giảm thời gian + RAM của rebuild() bằng cách tận dụng storage/radix/search đã persist, thay vì dựng lại mọi thứ trong RAM.

Đã làm (2 commits)

  1. perf(graph): skip chain-engine rebuild on reopen via version stamp

    • Mirror version-stamp của name engine cho chain engine ([version|count|sample_func] ở slot 0 storage chính).
    • Đo local (reopen 20k symbols, full): rebuild TOTAL 113.27s → 0.234s (~485×).
    • Thêm test chain_engine_stamp_survives_reopen.
  2. refactor(graph): drop in-memory edges map

    • Xoá GraphIndex.edges: HashMap<(u64,u64), EdgeMeta> + rebuild_edges() + resolve_arg_ids() (arg_ids/is_recursive không ai đọc).
    • Xoá EdgeMeta struct khỏi codegraph-core (rt_edges luôn rỗng — chưa từng được ghi ở production).
    • flow() đọc condition/effect trực tiếp từ call records (cùng nguồn, chính xác theo position hơn bản last-wins cũ).
    • rebuild() bỏ fetch + deserialize all_call_records; đếm edge_count từ chains.
    • stats().edges giữ nguyên (u64) nên API/MCP/tests không đổi.
    • Đo local (reopen 20k): rebuild TOTAL 233ms → 123ms, RSS delta 46.7 → 38.7 MiB.

Cần CI kiểm chứng

  • Clippy/test full matrix (đặc biệt Windows — lần trước lộ bug sibling_file).
  • CodSpeed xem có regression nào không.

Chưa làm (tiếp theo nếu spike này ổn)

  • Lazy SymbolRegistry (Phase 3 trong discussion): symbols map vẫn load-all (~45ms + RAM) — cần async-hoá read path, ripple lớn nên để riêng.

Summary by CodeRabbit

  • Improvements
    • Index reopening now checks whether saved chain data is current and refreshes it when needed, helping keep flow and caller results consistent.
    • Memory reports focus on symbol estimates and remaining graph structures; edge metadata is no longer reported as a separate structure.
  • Release
    • Updated the application and package versions to 2.2.8.

Hung Pham added 2 commits October 7, 2026 15:53
rebuild() luôn clear + re-insert toàn bộ func id vào chain trie dù trie đã
persist trong rt_* — chiếm >99% thời gian mở index (đo được: ~11s@5k,
~113s@20k). Mirror version-stamp của name engine cho chain engine: ghi
[version|count|sample_func] vào slot 0 của storage chính, reopen đọc stamp là
bỏ qua rebuild.

Kết quả (reopen 20k symbols, full shape): rebuild TOTAL 113.27s → 0.234s
(~485×). chains_map vẫn decode đủ cho callees/flow; trie giữ nguyên phục vụ
callers/search_flow.

Thêm test hồi quy chain_engine_stamp_survives_reopen.
…ll records

self.edges: HashMap<(u64,u64), EdgeMeta> (51k entries @ repo này) là bản sao
dữ liệu đã persist trong call records — flow() trước giờ vẫn fallback về
record khi miss. Xoá hẳn:

- GraphIndex.edges + rebuild_edges() + resolve_arg_ids() (arg_ids không ai đọc)
- EdgeMeta struct trong codegraph-core (rt_edges luôn rỗng, chưa từng được ghi)
- edges_mem/edge_meta_heap trong memtrack; predicted_edges trong bench mem
- rebuild(): bỏ fetch+deserialize all_call_records (~130ms@20k), đếm
  edge_count thẳng từ chains; flow() lấy condition/effect trực tiếp từ record
- build_edges_from_calls -> build_call_indexes (đếm edges, không materialize)

stats().edges giữ nguyên (u64 count) nên API/MCP/tests không đổi.
@coderabbitai

coderabbitai Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d4fd692d-dcb5-4a05-b248-c1b767d11dca
📥 Commits

Reviewing files that changed from the base of the PR and between 8b7dd87 and f34b8b9.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (12)
  • Cargo.toml
  • crates/codegraph-bench/src/bin/mem.rs
  • crates/codegraph-core/src/lib.rs
  • crates/codegraph-core/src/semgraph.rs
  • crates/codegraph-graph/src/lib.rs
  • crates/codegraph-graph/src/memtrack.rs
  • crates/codegraph-graph/src/shared.rs
  • crates/codegraph-graph/tests/sqlite.rs
  • packaging/aur/codegraph-rs-bin/PKGBUILD
  • packaging/choco/codegraph.nuspec
  • packaging/winget/codegraph.yaml
  • scripts/install.ps1
 _______________________________________________
< Show me the code! And I'll show you the bugs. >
 -----------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov-commenter

codecov-commenter commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.87629% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.95%. Comparing base (8b7dd87) to head (f34b8b9).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
crates/codegraph-graph/src/lib.rs 95.45% 3 Missing ⚠️
crates/codegraph-bench/src/bin/mem.rs 0.00% 1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (95.87%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #44      +/-   ##
==========================================
+ Coverage   77.93%   77.95%   +0.01%     
==========================================
  Files          96       96              
  Lines       23105    23067      -38     
==========================================
- Hits        18007    17981      -26     
+ Misses       5098     5086      -12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed

codspeed Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×5.6

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 36 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ cold_depth1 361.8 ms 33.5 ms ×11
⚡ cold_depth1 60.4 ms 21.2 ms ×2.9

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing spike/storage-backed-symbols (f34b8b9) with main (d569b4c)

Open in CodSpeed

@hungpham10
hungpham10 marked this pull request as ready for review October 7, 2026 10:41
@hungpham10
hungpham10 merged commit 2bedb4e into main Oct 7, 2026
21 of 23 checks passed
hungpham10 added a commit that referenced this pull request Oct 8, 2026
* perf(graph): skip chain-engine rebuild on reopen via version stamp

rebuild() luôn clear + re-insert toàn bộ func id vào chain trie dù trie đã
persist trong rt_* — chiếm >99% thời gian mở index (đo được: ~11s@5k,
~113s@20k). Mirror version-stamp của name engine cho chain engine: ghi
[version|count|sample_func] vào slot 0 của storage chính, reopen đọc stamp là
bỏ qua rebuild.

Kết quả (reopen 20k symbols, full shape): rebuild TOTAL 113.27s → 0.234s
(~485×). chains_map vẫn decode đủ cho callees/flow; trie giữ nguyên phục vụ
callers/search_flow.

Thêm test hồi quy chain_engine_stamp_survives_reopen.

* refactor(graph): drop in-memory edges map, read edge metadata from call records

self.edges: HashMap<(u64,u64), EdgeMeta> (51k entries @ repo này) là bản sao
dữ liệu đã persist trong call records — flow() trước giờ vẫn fallback về
record khi miss. Xoá hẳn:

- GraphIndex.edges + rebuild_edges() + resolve_arg_ids() (arg_ids không ai đọc)
- EdgeMeta struct trong codegraph-core (rt_edges luôn rỗng, chưa từng được ghi)
- edges_mem/edge_meta_heap trong memtrack; predicted_edges trong bench mem
- rebuild(): bỏ fetch+deserialize all_call_records (~130ms@20k), đếm
  edge_count thẳng từ chains; flow() lấy condition/effect trực tiếp từ record
- build_edges_from_calls -> build_call_indexes (đếm edges, không materialize)

stats().edges giữ nguyên (u64 count) nên API/MCP/tests không đổi.

* Bump version to v2.2.8

---------

Co-authored-by: Hung Pham <hungpham@MacBook-Pro-cua-Hung.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants