Repository navigation
feat(storage): add rpc.client.call.duration metric for gRPC calls - #14743
nidhiii-27 wants to merge 16 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces StorageGrpcMetricsInterceptor, a new gRPC client interceptor designed to record call metrics such as RPC duration, and integrates it into GrpcStorageOptions. It also adds comprehensive unit tests to verify its behavior. The review feedback is highly constructive, focusing on preventing a potential NullPointerException when combining interceptors, simplifying redundant server address resolution, and optimizing performance on the hot path by precomputing the DirectPath endpoint status in the constructor rather than performing string operations on every RPC completion.
There was a problem hiding this comment.
Code Review
This pull request introduces StorageGrpcMetricsInterceptor to record gRPC client call metrics (such as duration, status codes, and network paths) and integrates it into GrpcStorageOptions. It also adds comprehensive unit tests to verify the interceptor's behavior. The review feedback highlights two key improvement opportunities: adding a null check for grpcInterceptorProvider to prevent a potential NullPointerException when combining providers, and simplifying the redundant logic used to resolve the serverAddress.
…Interceptor - Precompute defaultServerAddress and isDirectPathEndpoint in StorageGrpcMetricsInterceptor constructor - Guard grpcInterceptorProvider and its getInterceptors() against null in GrpcStorageOptions - Add defensive null checks and cause-aware error mapping in StorageGrpcMetricsInterceptor.onClose - Use JSpecify @nullable and expand unit test coverage in StorageGrpcMetricsInterceptorTest [Generated-by: AI]
…ls.extractServerAddress Merge updated feat/storage-grpc-veneer-operation-metrics base branch and update StorageGrpcMetricsInterceptor to call OpenTelemetryBootstrappingUtils.extractServerAddress. [Generated-by: AI]
[Generated-by: AI]
[Generated-by: AI]
[Generated-by: AI]
…rage-rpc-call-duration
…Interceptor - Use full gRPC method name (`method.getFullMethodName()`) for `rpc.method` and remove `extractShortMethodName` - Capture fallback `Context.current()` in `interceptCall` on the caller thread - Remove `isDirectPathEndpoint` and return `NETWORK_PATH_UNKNOWN` when `remoteAddr` is absent - Restore `/18` and `/42` CIDR byte-mask check in `isDirectPathIp` [Generated-by: AI]
… mapping Update recordAttemptError_specificExceptions_mappedToCanonicalTypes to assert STATUS_DEADLINE_EXCEEDED (1) and STATUS_TIMEOUT (1) following the StorageErrorTypeMapper update. [Generated-by: AI]
cpriti-os
left a comment
There was a problem hiding this comment.
Thanks for addressing the earlier comments. Mostly looks good, one real issue with the IPv6 mask plus a few cleanups inline.
… feat/storage-rpc-call-duration [Generated-by: AI]
…torage-rpc-call-duration
[Generated-by: AI]
…torage-rpc-call-duration [Generated-by: AI]
…on-metrics' into feat/storage-rpc-call-duration
- Set network.path on StorageMetricsContext in onHeaders so streaming ReadObject attempts capture peer path before onClose - Remove @nullable from onClose parameters and simplify status.getCode().name() - Assert network.path=directpath on rpc.client.call.duration point in StorageGrpcMetricsInterceptorTest [Generated-by: AI]
| metricsCtx.setNetworkPath(networkPath); | ||
| } | ||
| String statusCode = status.getCode().name(); | ||
| String errorType = StorageErrorTypeMapper.mapGrpcStatus(status); |
There was a problem hiding this comment.
What error.type does a watchdog-aborted ReadObject stream get here? It looks like it lands as CANCELLED (gax cancels the call with a CancellationException) while attempts/errors tag the same attempt TIMEOUT. Fine as a follow-up if it needs a feedback hook through StorageMetricsContext.
There was a problem hiding this comment.
Confirmed — GAX's WatchdogStream.cancelIfStale() cancels the underlying ClientCall with a CancellationException (delivering Status.CANCELLED to onClose) before substituting WatchdogTimeoutException above the listener layer. Tracking a StorageMetricsContext feedback hook as a follow-up alongside stall.duration.
Co-authored by AI Agent
| } | ||
|
|
||
| if (!NoopGrpcInterceptorProvider.INSTANCE.equals(grpcInterceptorProvider)) { | ||
| StorageClientMetrics clientMetrics = getStorageClientMetrics(); |
There was a problem hiding this comment.
The Noop and empty-list branches end up identical to the addAll path (NoopGrpcInterceptorProvider.getInterceptors() is an empty list), so this can be a single provider lambda that addAll()s the user interceptors and appends ours.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| @Override | ||
| public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall( | ||
| MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) { | ||
| if (metrics.getRpcClientCallDuration() == null) { |
There was a problem hiding this comment.
recordRpcCallDuration already no-ops on a null histogram and this state can't be reached from fromOptions, so this early return and the disabledMetrics test can go.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| @Override | ||
| public void start(Listener<RespT> responseListener, Metadata headers) { | ||
| long startTimeNanos = clock.nanoTime(); | ||
| AtomicBoolean callRecorded = new AtomicBoolean(false); |
There was a problem hiding this comment.
onClose is delivered exactly once per call, so the callRecorded flag and the duplicateOnClose test can go; the Math.max can go too since nanoTime deltas can't be negative.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| .put(KEY_GCP_CLIENT_VERSION, StorageOptions.version()) | ||
| .put(KEY_GCP_CLIENT_ARTIFACT, ARTIFACT_STORAGE) | ||
| .build(); | ||
| Attributes baseAttributes = |
There was a problem hiding this comment.
fromOptions already puts rpc.system.name and the real server.address on baseAttributes. Could the tests build metrics via StorageOptions.grpc() so this hardcoded host and the extra rpc.system.name put in recordRpcCallDuration can go?
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| clock = new FakeClock(); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
rpc.method is a straight pass-through and the success test already asserts the full ReadObject name; can this test and the three extra descriptors go?
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| .isEqualTo("storage.googleapis.com"); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
The DNS_FAILURE cause mapping is already covered in StorageErrorTypeMapperTest and the interceptor just delegates, so this one can go.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| assertThat(nullCombinedProvider.getInterceptors().get(0)) | ||
| .isInstanceOf(StorageGrpcMetricsInterceptor.class); | ||
|
|
||
| AtomicBoolean customCalled = new AtomicBoolean(false); |
There was a problem hiding this comment.
customCalled can never flip since no call is made, so that assert is vacuous; a plain lambda interceptor would do here.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| testCall.close(Status.OK, new Metadata()); | ||
| } | ||
|
|
||
| MetricData rpcDurationData = |
There was a problem hiding this comment.
Can this metric lookup be a small findMetric(name) helper? It's repeated in every test.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
|
|
||
| private static final class TestChannel extends Channel { | ||
| private final TestClientCall<?, ?> call; | ||
| private final String authority; |
There was a problem hiding this comment.
The interceptor doesn't read authority anymore, so TestChannel can drop the parameter.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
- Simplify GrpcInterceptorProvider wiring in GrpcStorageOptions to a single lambda - Remove unreachable null/duplicate guards and Math.max clamping in StorageGrpcMetricsInterceptor - Rely on StorageClientMetrics.fromOptions baseAttributes for rpc.system.name and server.address - Deduplicate and streamline StorageGrpcMetricsInterceptorTest [Generated-by: AI]
Instruments gRPC storage calls to record
rpc.client.call.durationOpenTelemetry histogram metrics.StorageGrpcMetricsInterceptorto measure per-RPC duration and attachrpc.system.name,rpc.method,rpc.response.status_code,network.path,server.address, anderror.typeattributes.StorageGrpcMetricsInterceptorintoGrpcStorageOptionsand initializesrpc.client.call.durationinStorageClientMetrics.StorageGrpcMetricsInterceptorTestandStorageClientMetricsTest.Stacked on #14729.