Repository navigation
feat(storage): add gcp.storage.client.response.body.size metric for read operations - #14747
nidhiii-27 wants to merge 9 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements response body size tracking for the 'ReadObject' operation across both gRPC and JSON implementations of Google Cloud Storage, updating BaseStorageReadChannel, GrpcStorageImpl, StorageImpl, and StorageMetricsContext, along with adding corresponding unit tests. The review feedback suggests refactoring a long chain of excluded key checks in StorageMetricsContext into a static Set constant to improve maintainability, and replacing fully qualified class names with imports in the new test cases to enhance readability.
There was a problem hiding this comment.
Code Review
This pull request introduces tracking and recording of response body size metrics for "ReadObject" operations across both GRPC and JSON/HTTP storage implementations. It updates BaseStorageReadChannel, GrpcStorageImpl, and StorageImpl to capture the read or downloaded bytes and set them in the StorageMetricsContext, which now records the response body size metric. Corresponding unit tests are also added to StorageMetricsContextTest. The reviewer suggested avoiding the use of fully qualified class names in the new test cases to improve code readability, recommending importing them instead.
…se body size - Extract BODY_SIZE_EXCLUDED_KEYS and buildBodySizeAttributes(StorageClientMetrics) in StorageMetricsContext to deduplicate request/response body size attribute filtering. - Guard GrpcStorageImpl.readAllBytes response byte recording with error == null and move countBytes helper to the private helper section. - Wire GrpcStorageImpl.blobReadSession and BlobReadSessionAdapter to track delivered response bytes across all ReadProjectionConfig types and record gcp.storage.client.response.body.size on session close. - Track lastReadError in BaseStorageReadChannel so mid-stream read errors propagate to recordOperation on close while preserving partial read byte counts. - Replace inline FQCNs with imports in StorageMetricsContextTest and add unit tests for channel reads, StorageImpl, GrpcStorageImpl, and BlobReadSessionAdapter. [Generated-by: AI]
…rdResponseBodySize
…ge.operation on response.body.size [Generated-by: AI]
cpriti-os
left a comment
There was a problem hiding this comment.
Mostly looks good, left a few comments inline.
…e-response-body-size [Generated-by: AI]
[Generated-by: AI]
…nse-body-size [Generated-by: AI]
| throw e; | ||
| } finally { | ||
| if (metricsContext != null) { | ||
| metricsContext.setResponseBytes(baos.size()); |
There was a problem hiding this comment.
If the stream fails after a chunk has already been copied into baos, this records bytes the caller never gets since the method throws. Can we only set the size when error == null, like the HTTP path?
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| throws Exception { | ||
| byte[] payload = "0123456789abcdefghij01234".getBytes(StandardCharsets.UTF_8); | ||
| ChecksummedTestContent chunk1 = ChecksummedTestContent.of(payload, 0, 10); | ||
| ChecksummedTestContent chunk2 = ChecksummedTestContent.of(payload, 10, 15); |
There was a problem hiding this comment.
chunk1 is 10 B, smaller than the 8 KiB ByteStreams.copy buffer, so nothing reaches baos before the error and the 0.0 assert passes without exercising the partial-copy path. Can we make the failing chunk larger than 8 KiB? A single Storage with cumulative asserts would also do.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| collected, StorageClientMetrics.METRIC_GCP_STORAGE_CLIENT_RESPONSE_BODY_SIZE); | ||
| assertThat(point.getCount()).isEqualTo(1L); | ||
| assertThat(point.getSum()).isEqualTo(25.0 + 15.0 + 10.0 + 12.0); | ||
| } |
There was a problem hiding this comment.
Can we add a case where a projection fails (e.g. asFutureBytes returning a failed future after a successful read) and the session is then closed, so the error.type and partial byte count on close are covered? None of the adapter's error paths run in tests right now.
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| @Nullable private final StorageClientMetrics storageClientMetrics; | ||
| @Nullable private volatile Throwable lastOperationError; | ||
|
|
||
| BlobReadSessionAdapter(ObjectReadSession session) { |
There was a problem hiding this comment.
The 1-arg constructor and wrap(ApiFuture) no longer have any callers across the repo — can we drop them?
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| private SeekableByteChannel wrapSeekableByteChannel(SeekableByteChannel delegate) { | ||
| return new SeekableByteChannel() { | ||
| @Override | ||
| public int read(ByteBuffer dst) throws IOException { |
There was a problem hiding this comment.
These four read overrides repeat the same try/count/capture block. Can we pull that into a small trackRead helper (with a throwing functional interface) so each override is a one-liner?
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| SdkMeterProvider provider = SdkMeterProvider.builder().registerMetricReader(reader).build(); | ||
| StorageClientMetrics metrics = StorageClientMetrics.create(provider, true, false); | ||
|
|
||
| Attributes baseAttributes = |
There was a problem hiding this comment.
The fixture never sets server.port or rpc.method, so those two isNull asserts don't check anything, and attribute filtering is already covered by the request-body tests. Can we trim this to the size and gcp.storage.operation/gcp.client.method asserts?
There was a problem hiding this comment.
Done
Co-authored by AI Agent
| } | ||
| } | ||
|
|
||
| private static final class FakeSeekableByteChannel implements SeekableByteChannel { |
There was a problem hiding this comment.
FileChannel.open over a tempFolder file already gives a SeekableByteChannel — can we use that instead of this 56-line fake?
There was a problem hiding this comment.
Done
Co-authored by AI Agent
…into feat/storage-response-body-size [Generated-by: AI]
- Guard GrpcStorageImpl.readAllBytes responseBytes update with error == null and coalesce mid-stream IOExceptions before recording error.type - Remove unused 1-arg constructor and wrap(ApiFuture) in BlobReadSessionAdapter, consolidate channel read overrides into trackRead, unwrap IOException causes, and replace detached addCallback with inline transform/catchingAsync - Expand StorageMetricsContextTest with >8 KiB mid-stream readAllBytes error assertion, BlobReadSessionAdapter projection failure coverage, FileChannel/StorageByteChannels test channels, and shared class fixture [Generated-by: AI]
Summary
This PR adds the
gcp.storage.client.response.body.sizemetric (LongHistogram, unitBy) forReadObjectoperations.Changes
StorageMetricsContext:gcp.storage.client.response.body.sizeinsiderecordOperation(...)whenoperationNameis"ReadObject"andrespBytes >= 0(including0bytes for empty reads, immediately closed channels, or failedreadAllBytes/blobReadSessionopen calls).bodySizeAttributes(built frombuildAttemptBaseAttributes(metrics)plusgcp.client.method, excluding attempt/status/error keys) shared withgcp.storage.client.request.body.size.BaseStorageReadChannel/BlobReadChannelV2/GrpcBlobReadChannel:BaseStorageReadChannel.read(ByteBuffer)viametricsContext.addResponseBytes(read).lastOperationErroracrossread(ByteBuffer)so mid-stream errors are passed torecordOperation(...)onclose()alongside partial byte counts.StorageImpl:readAllBytesanddownloadTo(usingCountingOutputStream.getCount()across retries without double-counting).GrpcStorageImpl&BlobReadSessionAdapter:GrpcStorageImpl.readAllBytesonly whenerror == null, accumulates written bytes inGrpcStorageImpl.downloadToviacountBytes, and coalesces mid-streamIOException(StorageException)viaStorageException.coalesce(...)before recordingerror.type.GrpcStorageImpl.blobReadSessionandBlobReadSessionAdapterto accumulate delivered bytes across allReadProjectionConfigtypes (asFutureBytes(),asFutureByteString(),asChannel(),asSeekableChannel()), unwrap mid-streamIOExceptioncauses intrackRead, and record the operation onclose()(or on session open failure).StorageMetricsContextTest:recordOperation,BaseStorageReadChannel(full read, partial read + early close, 0-byte close, mid-stream error),StorageImpl(readAllBytes,downloadTowith retries),BlobReadSessionAdapter(all projections, projection/channel failure + partial bytes onclose(), and open failure), andGrpcStorageImpl(readAllBytesincluding >8 KiB partial copy before mid-stream failure,downloadTo,blobReadSession).