[improve](metrics) Export file cache remote index byte metrics#65398
Merged
Conversation
hoshinojyunn
requested review from
Gabriel39,
gavinchou,
liaoxin01 and
morningman
as code owners
July 8, 2026 17:06
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
hoshinojyunn
force-pushed
the
feat/profile
branch
from
July 8, 2026 17:21
f128c74 to
da72c65
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29891 ms |
Contributor
TPC-DS: Total hot run time: 181241 ms |
Contributor
ClickBench: Total hot run time: 25.24 s |
Issue Number: None
Related PR: None
Problem Summary: File cache profile statistics already track `inverted_index_bytes_read_from_remote` and `segment_footer_index_bytes_read_from_remote`, but BE Doris metrics only exported aggregate cache, remote, and peer read bytes. That made these index-specific remote read costs visible in query profile only, not in the BE metrics endpoint. This change registers two new Doris metrics, extends the existing `FileCacheMetrics` aggregation path to accumulate them, and exports them through the normal metrics hook. The update also extends the existing file cache profile unit test helpers so segment footer index statistics are covered and adds a test for the new metric export path.
Expose `inverted_index_bytes_read_from_remote` and `segment_footer_index_bytes_read_from_remote` in BE Doris metrics.
- Test: Manual compile verification
- Started `./run-be-ut.sh --run --filter=FileCacheProfileReporterTest.*`, but it triggered a large `doris_be_test` rebuild and was interrupted after the changed objects compiled successfully
- Recompiled changed objects with `ninja -C be/ut_build_ASAN src/common/CMakeFiles/Common.dir/metrics/doris_metrics.cpp.o src/io/CMakeFiles/IO.dir/cache/block_file_cache_profile.cpp.o test/CMakeFiles/doris_be_test.dir/io/cache/block_file_cache_profile_reporter_test.cpp.o`
- Behavior changed: Yes (adds two new BE metrics for file cache remote index bytes)
- Does this need documentation: No
hoshinojyunn
force-pushed
the
feat/profile
branch
from
July 9, 2026 02:16
da72c65 to
09d5f0f
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29644 ms |
Contributor
TPC-DS: Total hot run time: 180696 ms |
Contributor
ClickBench: Total hot run time: 25.15 s |
hoshinojyunn
changed the title
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
reviewed
Jul 10, 2026
yiguolei
approved these changes
Jul 10, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
yiguolei
approved these changes
Jul 10, 2026
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review summary for PR #65398.
I reviewed the changed BE metrics, file-cache metric aggregation/export path, BE unit test, and cloud docker regression test. I found no additional blocking issue to submit as a new inline comment.
Duplicate suppression: the existing thread on be/src/io/cache/block_file_cache_profile.cpp:74 already raises the set_value vs add/increment concern, so I did not submit a duplicate. I also checked the surrounding exporter path: FileCacheMetrics::update() accumulates deltas into the internal cumulative statistics, and the existing aggregate file-cache metric hook exports those cumulative values with set_value.
Critical checkpoint conclusions:
- Goal and coverage: the PR exposes two file-cache remote index byte metrics and adds BE/regression coverage for the export path.
- Scope: the change is focused on metric registration, file-cache stat accumulation, and tests.
- Concurrency/lifecycle: no new lock-order or lifecycle issue found; the added fields follow the existing atomic
FileCacheMetricspattern. - Config/compatibility/persistence/transactions: no new config, protocol, storage format, edit-log, or transaction behavior is introduced.
- Parallel paths: both inverted-index and segment-footer read types feed the same file-cache statistics update path; no optimizer/rewrite impact found.
- Performance/observability: the hot path adds only the same style of atomic accumulation used by existing file-cache counters, and the new metrics improve observability.
- Testing standards: the regression test drops before use, orders query output where it inspects rows, and checks both exported metrics.
Validation: static review only in this runner. thirdparty/installed, thirdparty/installed/bin/protoc, and be/ut_build_ASAN are absent, so I did not rerun the claimed BE unit or docker regression tests locally.
User focus: no additional user-provided review focus was supplied.
Subagent conclusions: optimizer-rewrite and tests-session-config both reported NO_NEW_VALUABLE_FINDINGS in the final convergence round for the current ledger and empty proposed final comment set. No subagent candidates were accepted, and my only local candidate was dismissed with evidence because BE-UT builds doris_be_test with -fno-access-control.
Gabriel39
approved these changes
Jul 10, 2026
github-actions Bot
pushed a commit
that referenced
this pull request
Jul 10, 2026
Issue Number: None
Related PR: None
Problem Summary:
File cache statistics already track the remote-read bytes for inverted
index and segment footer
index data, but BE Doris metrics only export the aggregate
cache/local/remote/peer byte counters.
As a result, these two index-specific remote read costs are visible in
query profile internals but
not exposed from the BE `/metrics` endpoint, which makes them harder to
monitor and alert on in
cloud/file-cache scenarios.
This PR exposes the following two BE metrics:
- `doris_be_inverted_index_bytes_read_from_remote`
- `doris_be_segment_footer_index_bytes_read_from_remote`
The change keeps the existing file cache metrics aggregation path and
extends it in three places:
1. Register the two counters in `DorisMetrics`
2. Extend `FileCacheMetrics::AtomicStatistics` and the
aggregation/update path to accumulate them
3. Export them from the existing metrics hook and cover the behavior
with a BE unit test
### Release note
Add two new BE metrics for file cache remote index read bytes:
- `doris_be_inverted_index_bytes_read_from_remote`
- `doris_be_segment_footer_index_bytes_read_from_remote`
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `./run-regression-test.sh --run -d cloud_p0 -s
test_file_cache_remote_index_byte_metrics -runMode=cloud`
- [x] Unit Test
- `./run-be-ut.sh --run
--filter=DorisMetricsTest.*:FileCacheProfileReporterTest.*`
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Number: None
Related PR: None
Problem Summary:
File cache statistics already track the remote-read bytes for inverted index and segment footer
index data, but BE Doris metrics only export the aggregate cache/local/remote/peer byte counters.
As a result, these two index-specific remote read costs are visible in query profile internals but
not exposed from the BE
/metricsendpoint, which makes them harder to monitor and alert on incloud/file-cache scenarios.
This PR exposes the following two BE metrics:
doris_be_inverted_index_bytes_read_from_remotedoris_be_segment_footer_index_bytes_read_from_remoteThe change keeps the existing file cache metrics aggregation path and extends it in three places:
DorisMetricsFileCacheMetrics::AtomicStatisticsand the aggregation/update path to accumulate themRelease note
Add two new BE metrics for file cache remote index read bytes:
doris_be_inverted_index_bytes_read_from_remotedoris_be_segment_footer_index_bytes_read_from_remoteCheck List (For Author)
Test
./run-regression-test.sh --run -d cloud_p0 -s test_file_cache_remote_index_byte_metrics -runMode=cloud./run-be-ut.sh --run --filter=DorisMetricsTest.*:FileCacheProfileReporterTest.*Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)