Skip to content

Add cluster remote client so11y metrics - #14245

Merged
wu-sheng merged 1 commit into
apache:masterfrom
songzhendong:feat/cluster-remote-so11y
Nov 19, 2026
Merged

wu-sheng merged 1 commit into
apache:masterfrom
songzhendong:feat/cluster-remote-so11y

Conversation

@songzhendong

@songzhendong songzhendong commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in remote_out_error_count, which were only logged before.

Companion Horizon UI: apache/skywalking-horizon-ui#222

Test plan

Fork CI

@songzhendong

songzhendong commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

UI screenshots (local verification)

Grafana cluster

Grafana cluster Remote panels

Grafana instance

Grafana instance Remote panels

Horizon OAP so11y

Horizon cluster remote widgets

@wu-sheng wu-sheng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Getting the cluster metrics into the so11y MAL rules and dashboards is a useful addition. Three points:

1. Please drop the reason label from remote_out_error_count.

Every failure behind those reasons is already logged with its exception, and the exception carries the gRPC status: onError logs each failed stream at ERROR (RESOURCE_EXHAUSTED at WARN, once per 120s), and the consume catch, the interrupted slot wait and the self client all log at ERROR. Deployments keep OAP's error logs, so the metric only needs to show that pushes to a peer are failing and how often; the logs say why.

Keeping remote_out_error_count{dest, self} as it is today also avoids changing the label set of an existing metric, and avoids pre-creating 8 series per client. Some values also don't fit this link: auth_failed can never fire, because RemoteServiceHandler is registered without an authentication interceptor, and rate_limited / rejected don't describe what RESOURCE_EXHAUSTED / CANCELLED mean.

This touches RemoteClientMetrics (the reason constants, reasonOf, the pre-created counters), RemoteClientMetricsTest, oap.yaml, oap.data.yaml, dashboards-so11y.md and the two Grafana JSON files. The Horizon UI PR (apache/skywalking-horizon-ui#222) already sums by dest only, so it is unaffected.

2. Please keep counting errors in onError.

On master, onError logs async failures but never increments remote_out_error_count, so deadline-exceeded and unavailable peers, the most common real failures, never reach the metric. Counting them there (without a reason) is a real fix.

3. Changelog and PR description wording.

remote_in_count, remote_in_error_count, remote_in_target_not_found_count, remote_in_latency, remote_out_count, remote_out_error_count and cluster_size have been exported on the Prometheus endpoint since #2133, and the Grafana dashboards already chart the remote in/out counts. What this PR adds is getting them into the so11y MAL rules and dashboard, plus the new client metrics. Suggested changelog line:

* Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in `remote_out_error_count`, which were only logged before.

@songzhendong

songzhendong commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Addressed all three points:

  1. Dropped the reason label from remote_out_error_count. It is back to {dest, self} only (metrics, MAL, Grafana, docs, and tests). Failure details stay in the existing ERROR/WARN logs.
  2. Kept counting asynchronous failures in onError via recordError() (no reason).
  3. Updated the changelog and PR description to the suggested wording.

Fork CI is green after the change:
songzhendong#36

Pushed to this PR branch as 159b2cb.

@songzhendong
songzhendong force-pushed the feat/cluster-remote-so11y branch from 159b2cb to 3fc6ea5 Compare November 19, 2026 00:41
@wu-sheng
wu-sheng force-pushed the feat/cluster-remote-so11y branch from 3fc6ea5 to 8c9a271 Compare November 19, 2026 13:07
@wu-sheng

Copy link
Copy Markdown
Member

Thanks for the quick update. I pushed a follow-up directly to this branch and squashed the branch into one commit (you stay the author), so please save any local work before resetting your local branch to the new head.

What changed and why:

  1. Queue gauge off the push path. push() runs for every metric sent to another OAP node, from many aggregation threads, and it sampled remote_out_queue_used_percentage twice per message. Each sample took queueGaugeLock and the send queue's own lock, which producers and the drain also use. The gauge is now sampled only on the consumer side, after each batch and while waiting for a stream slot, like MetricsAggregateWorker does for its queue gauge.
  2. Failures while a removed peer drains are counted and logged again. When a peer leaves the cluster, close() sets closing and the queue's final drain still sends the remaining batches to that peer. While closing, onError skipped both the counter and the log for every status, and the send path skipped the counter, so genuine send failures during peer removal went uncounted and, from onError, unlogged; master logged them. Now both paths count failures until closed, and onError logs them until then too, because close() sets closed just before shutdownNow() fails the streams still in flight. CANCELLED is not counted, matching its existing debug-only log. Drain streams that succeed before closed now record latency too.
  3. Receiving side. PairingWorker (TAL span pairing) also receives remote data, and its in() can throw, so the new catch around nextWorker.in() does change behaviour: one failing message is now discarded and counted in remote_in_error_count, and the rest of the stream is still processed, where before the exception ended the whole stream. The changelog now says so, a new RemoteServiceHandlerWorkerFailureTestCase covers it, and the log line names the worker instead of printing the whole message.
  4. Tests. The names left over from reason are gone (pushCountsUnavailableStreamAsError, interruptWaitingForStreamSlotCountsOneError), and the errorReasons lists, written by a callback thread and read by the test thread without synchronization, are now an AtomicInteger. The concurrent close-versus-publish test never put anything in the queue, so it could not catch a stale write; it is now a deterministic check that a closed client does not publish the gauge. The drain test now checks that the in-process peer received every queued push, not only that sends were attempted. Two new tests cover point 2: a failure during the drain is counted (from onError and from the send itself), a successful drain stream records latency, and a cancelled stream or a stream failed after close is not counted. Each of these new tests, and the receiving-side test, fails against the previous code.
  5. No timing bounds. The e2e checks required average self latency under 50 ms and SelfRemoteClientTest under 100 ms, which can fail under CI load. They now check that latency samples exist.
  6. Docs and dashboards. The so11y doc table now says what one error and one latency sample are, for peers and for the self client. The Grafana error descriptions (peers, self=N) say what one error is: a failed batch stream, a refused enqueue or an interrupted stream-slot wait. The cluster dashboard's in-flight and queue panels label series {{instance}} -> {{dest}}, so different senders no longer share a legend name with three or more nodes.
  7. Style (CLAUDE.md). incOut() and recordError() only forwarded to a counter, so callers now use the counters through Lombok getters. The inline fully-qualified class names in the tests are imports now.

Checked locally on the pushed commit: the remote client and handler unit tests, the full unit-test suite, checkstyle, and javadoc on JDK 11.

@wu-sheng wu-sheng added this to the 11.2.0 milestone Nov 19, 2026
@wu-sheng wu-sheng added backend OAP backend related. so11y Self Observability labels Nov 19, 2026
Expose the cluster remote metrics OAP already exports on its Prometheus
endpoint (remote_in_*, remote_out_count, remote_out_error_count and
cluster_size) through the OAP self-observability MAL rules and the so11y
dashboard docs, and chart them on the Grafana cluster and instance
dashboards, which so far only had the remote in/out count panels.

Add remote client metrics: remote_out_latency (one sample per
successful batch stream for peers until the client is closed, per
successful push for the self client), remote_out_stream_inflight,
remote_out_stream_block_seconds and remote_out_queue_used_percentage.
The queue gauge is sampled on the consumer side, after each batch and
while waiting for a stream slot, not on the push path that every metric
sent to a peer takes.

remote_out_error_count keeps its dest and self labels. It now also
counts refused enqueues, interrupted stream-slot waits and asynchronous
stream failures reported through onError, none of which were counted
before. Cancelled streams and streams failed by close() shutting the
channel are not counted; failures while a removed peer's queue drains
are counted and logged.

On the receiving side, an exception from the target worker now discards
only that message and counts remote_in_error_count, instead of failing
the whole stream. The log line names the worker rather than printing
the message.
@wu-sheng
wu-sheng force-pushed the feat/cluster-remote-so11y branch from 8c9a271 to 881ba33 Compare November 19, 2026 13:16
@songzhendong

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed follow-up and for landing it on this branch (and for keeping me as author after the squash). — I'll reset locally and review the new head

@wu-sheng
wu-sheng merged commit f6167c1 into apache:master Nov 19, 2026
538 of 543 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend OAP backend related. so11y Self Observability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants