Skip to content

[server] Add pre-write buffer memory metrics for primary key tables - #4308

Open
Kaixuan-Duan wants to merge 2 commits into
apache:mainfrom
Kaixuan-Duan:pre-write-buffer-memory-metrics
Open

Kaixuan-Duan wants to merge 2 commits into
apache:mainfrom
Kaixuan-Duan:pre-write-buffer-memory-metrics

Conversation

@Kaixuan-Duan

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #4307

Brief change log

Tests

API and Format

Documentation

@zuston

zuston commented Sep 14, 2026

Copy link
Copy Markdown
Member

looks duplicate with #3533

@platinumhamburg

Copy link
Copy Markdown
Contributor

looks duplicate with #3533

I've looked at both #4308 and #3533. IMO, the way #4308 calculates actual memory usage reflects the true memory footprint — it factors in the real overhead of Java objects, which I believe is the correct approach.

@zuston, do you still plan to move forward with #3533? If so, I'd suggest adjusting it to follow the approach taken in this PR.

Additionally, I think the unified memory accounting introduced in #3751 should be incorporated into this PR. To be more specific: we need a truly accurate global memory usage ledger for KvPrewriteBuffer, maintained by atomic variables. Our future backpressure decisions will depend on it, and the current metrics should also be built on the same source of truth — rather than being the sum of non-atomic samples collected from multiple maps by the metrics framework.

@zuston

zuston commented Sep 14, 2026

Copy link
Copy Markdown
Member

I've looked at both #4308 and #3533. IMO, the way #4308 calculates actual memory usage reflects the true memory footprint — it factors in the real overhead of Java objects, which I believe is the correct approach.

make sense, please go ahead @Kaixuan-Duan . And I will close the #3533

@Kaixuan-Duan
Kaixuan-Duan force-pushed the pre-write-buffer-memory-metrics branch from 77b1e04 to c1806c7 Compare September 16, 2026 08:06
@Kaixuan-Duan

Copy link
Copy Markdown
Contributor Author

@platinumhamburg @zuston Thanks for the direction, and for closing #3533
The unified ledger is now part of this PR: KvPreWriteBufferMemoryLedger is a tabletserver-wide singleton with two AtomicLongs (estimated memory usage including per-entry object overhead, and entry count). Every pre-write buffer reports its accounting deltas to the ledger atomically on the write path (add on put, subtract on flush/truncate), and closes release their remaining accounting, so the ledger stays exact across tablet lifecycles. Both gauges now read directly from this single ledger — the previous per-tablet sampling and summation in KvManager is removed. No backpressure behavior is introduced; the watermarks and rejection logic from #3751 can be built on top of this ledger in a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[server] Add pre-write buffer memory metrics for primary key tables

3 participants