Skip to content

feat(storage): add rpc.client.call.duration metric for gRPC calls - #14743

Open
nidhiii-27 wants to merge 16 commits into
feat/storage-grpc-veneer-operation-metricsfrom
feat/storage-rpc-call-duration
Open

nidhiii-27 wants to merge 16 commits into
feat/storage-grpc-veneer-operation-metricsfrom
feat/storage-rpc-call-duration

Conversation

@nidhiii-27

Copy link
Copy Markdown
Contributor

Instruments gRPC storage calls to record rpc.client.call.duration OpenTelemetry histogram metrics.

  • Adds StorageGrpcMetricsInterceptor to measure per-RPC duration and attach rpc.system.name, rpc.method, rpc.response.status_code, network.path, server.address, and error.type attributes.
  • Wires StorageGrpcMetricsInterceptor into GrpcStorageOptions and initializes rpc.client.call.duration in StorageClientMetrics.
  • Adds unit tests in StorageGrpcMetricsInterceptorTest and StorageClientMetricsTest.

Stacked on #14729.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces StorageGrpcMetricsInterceptor, a new gRPC client interceptor designed to record call metrics such as RPC duration, and integrates it into GrpcStorageOptions. It also adds comprehensive unit tests to verify its behavior. The review feedback is highly constructive, focusing on preventing a potential NullPointerException when combining interceptors, simplifying redundant server address resolution, and optimizing performance on the hot path by precomputing the DirectPath endpoint status in the constructor rather than performing string operations on every RPC completion.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces StorageGrpcMetricsInterceptor to record gRPC client call metrics (such as duration, status codes, and network paths) and integrates it into GrpcStorageOptions. It also adds comprehensive unit tests to verify the interceptor's behavior. The review feedback highlights two key improvement opportunities: adding a null check for grpcInterceptorProvider to prevent a potential NullPointerException when combining providers, and simplifying the redundant logic used to resolve the serverAddress.

…Interceptor

- Precompute defaultServerAddress and isDirectPathEndpoint in StorageGrpcMetricsInterceptor constructor
- Guard grpcInterceptorProvider and its getInterceptors() against null in GrpcStorageOptions
- Add defensive null checks and cause-aware error mapping in StorageGrpcMetricsInterceptor.onClose
- Use JSpecify @nullable and expand unit test coverage in StorageGrpcMetricsInterceptorTest

[Generated-by: AI]
…ls.extractServerAddress

Merge updated feat/storage-grpc-veneer-operation-metrics base branch and update StorageGrpcMetricsInterceptor to call OpenTelemetryBootstrappingUtils.extractServerAddress.

[Generated-by: AI]
@nidhiii-27
nidhiii-27 marked this pull request as ready for review November 5, 2026 09:00
@nidhiii-27
nidhiii-27 requested review from a team as code owners November 5, 2026 09:00
…Interceptor

- Use full gRPC method name (`method.getFullMethodName()`) for `rpc.method` and remove `extractShortMethodName`
- Capture fallback `Context.current()` in `interceptCall` on the caller thread
- Remove `isDirectPathEndpoint` and return `NETWORK_PATH_UNKNOWN` when `remoteAddr` is absent
- Restore `/18` and `/42` CIDR byte-mask check in `isDirectPathIp`

[Generated-by: AI]
… mapping

Update recordAttemptError_specificExceptions_mappedToCanonicalTypes to assert STATUS_DEADLINE_EXCEEDED (1) and STATUS_TIMEOUT (1) following the StorageErrorTypeMapper update.

[Generated-by: AI]
@nidhiii-27
nidhiii-27 requested a review from cpriti-os November 9, 2026 10:50

@cpriti-os cpriti-os left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the earlier comments. Mostly looks good, one real issue with the IPv6 mask plus a few cleanups inline.

@nidhiii-27
nidhiii-27 requested a review from cpriti-os November 12, 2026 08:58
…on-metrics' into feat/storage-rpc-call-duration
- Set network.path on StorageMetricsContext in onHeaders so streaming ReadObject attempts capture peer path before onClose
- Remove @nullable from onClose parameters and simplify status.getCode().name()
- Assert network.path=directpath on rpc.client.call.duration point in StorageGrpcMetricsInterceptorTest

[Generated-by: AI]
@nidhiii-27
nidhiii-27 requested a review from cpriti-os November 17, 2026 07:13
metricsCtx.setNetworkPath(networkPath);
}
String statusCode = status.getCode().name();
String errorType = StorageErrorTypeMapper.mapGrpcStatus(status);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What error.type does a watchdog-aborted ReadObject stream get here? It looks like it lands as CANCELLED (gax cancels the call with a CancellationException) while attempts/errors tag the same attempt TIMEOUT. Fine as a follow-up if it needs a feedback hook through StorageMetricsContext.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — GAX's WatchdogStream.cancelIfStale() cancels the underlying ClientCall with a CancellationException (delivering Status.CANCELLED to onClose) before substituting WatchdogTimeoutException above the listener layer. Tracking a StorageMetricsContext feedback hook as a follow-up alongside stall.duration.

Co-authored by AI Agent

}

if (!NoopGrpcInterceptorProvider.INSTANCE.equals(grpcInterceptorProvider)) {
StorageClientMetrics clientMetrics = getStorageClientMetrics();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Noop and empty-list branches end up identical to the addAll path (NoopGrpcInterceptorProvider.getInterceptors() is an empty list), so this can be a single provider lambda that addAll()s the user interceptors and appends ours.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

@Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
if (metrics.getRpcClientCallDuration() == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recordRpcCallDuration already no-ops on a null histogram and this state can't be reached from fromOptions, so this early return and the disabledMetrics test can go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

@Override
public void start(Listener<RespT> responseListener, Metadata headers) {
long startTimeNanos = clock.nanoTime();
AtomicBoolean callRecorded = new AtomicBoolean(false);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onClose is delivered exactly once per call, so the callRecorded flag and the duplicateOnClose test can go; the Math.max can go too since nanoTime deltas can't be negative.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

.put(KEY_GCP_CLIENT_VERSION, StorageOptions.version())
.put(KEY_GCP_CLIENT_ARTIFACT, ARTIFACT_STORAGE)
.build();
Attributes baseAttributes =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fromOptions already puts rpc.system.name and the real server.address on baseAttributes. Could the tests build metrics via StorageOptions.grpc() so this hardcoded host and the extra rpc.system.name put in recordRpcCallDuration can go?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

clock = new FakeClock();
}

@Test

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rpc.method is a straight pass-through and the success test already asserts the full ReadObject name; can this test and the three extra descriptors go?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

.isEqualTo("storage.googleapis.com");
}

@Test

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DNS_FAILURE cause mapping is already covered in StorageErrorTypeMapperTest and the interceptor just delegates, so this one can go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

assertThat(nullCombinedProvider.getInterceptors().get(0))
.isInstanceOf(StorageGrpcMetricsInterceptor.class);

AtomicBoolean customCalled = new AtomicBoolean(false);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

customCalled can never flip since no call is made, so that assert is vacuous; a plain lambda interceptor would do here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

testCall.close(Status.OK, new Metadata());
}

MetricData rpcDurationData =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this metric lookup be a small findMetric(name) helper? It's repeated in every test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent


private static final class TestChannel extends Channel {
private final TestClientCall<?, ?> call;
private final String authority;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interceptor doesn't read authority anymore, so TestChannel can drop the parameter.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

- Simplify GrpcInterceptorProvider wiring in GrpcStorageOptions to a single lambda
- Remove unreachable null/duplicate guards and Math.max clamping in StorageGrpcMetricsInterceptor
- Rely on StorageClientMetrics.fromOptions baseAttributes for rpc.system.name and server.address
- Deduplicate and streamline StorageGrpcMetricsInterceptorTest

[Generated-by: AI]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants