Skip to content

fix(storage): hedged read attempts must not own the connection - #16534

Draft
kalragauri wants to merge 1 commit into
googleapis:mainfrom
kalragauri:fix/mock-leak
Draft

kalragauri wants to merge 1 commit into
googleapis:mainfrom
kalragauri:fix/mock-leak

Conversation

@kalragauri

Copy link
Copy Markdown
Contributor

This PR re-enables RetryClientTest.HedgedReadRecordsMetricsOnGlobalMeterProvider (skipped in #16523) and fixes the lifetime bug behind the gmock leak it reported.

Two references allowed a pool worker to hold the last reference to StorageConnectionImpl after Read() had already returned, destroying the connection (and its mock stub and metrics instruments) on a detached worker
thread after main()'s leak check:

  • HedgedObjectReadSource::ReadRaced() captured child_factory_ by shared_ptr in each attempt task, and the factory captures the connection. Because a worker destroys its task only after promise.set_value() unblocks the caller, the winning worker kept the connection alive past the end of the read. Attempts now capture a weak_ptr and lock it only for the open call; an attempt that starts after the source is destroyed retires with kCancelled.
  • A losing attempt owns a RetryObjectReadSource (which holds the connection) until its Close() returns. This is unchanged: the test now issues one more synchronous read on the single read worker after unblocking the loser, which (FIFO) cannot complete until the loser's task has been destroyed, so the test thread holds the last connection reference. Dropping a Client mid-close in production still tears the connection down on a detached worker, same as AutomaticallyCreatedBackgroundThreads; that is pre-existing and out of scope for this PR.

Fixes #16522

@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Oct 5, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors HedgedObjectReadSource to pass a std::weak_ptr of the ChildFactory to racing attempts, preventing them from keeping the factory and its associated connection/thread pools alive after the read source is destroyed. Additionally, it re-enables the previously skipped HedgedReadRecordsMetricsOnGlobalMeterProvider test with updated synchronization logic and adds a new unit test QueuedAttemptAfterDestructionDoesNotOpen to verify the lifetime behavior. I have no feedback to provide as there are no review comments to assess.

@codecov

codecov Bot commented Oct 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.39785% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.36%. Comparing base (54869ea) to head (e725508).

Files with missing lines Patch % Lines
...storage/internal/hedged_object_read_source_test.cc 79.41% 7 Missing ⚠️
...loud/storage/internal/hedged_object_read_source.cc 93.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16534      +/-   ##
==========================================
+ Coverage   92.34%   92.36%   +0.01%     
==========================================
  Files        2262     2262              
  Lines      217173   217309     +136     
==========================================
+ Hits       200549   200708     +159     
+ Misses      16624    16601      -23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch was successfully deployed

1 active deployment
false — e725508a Deployed Oct 5, 2026 by kalragauri via Save PR ref #12263
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage: RetryClientTest.HedgedReadRecordsMetricsOnGlobalMeterProvider leaking mock objects

1 participant