Skip to content

fix(bigtable): fail pendingRpcs on consecutive failures when pool has no established sessions - #15038

Merged
mutianf merged 9 commits into
mainfrom
fix-session-pool-pending-rpcs-bound
Nov 17, 2026
Merged

mutianf merged 9 commits into
mainfrom
fix-session-pool-pending-rpcs-bound

Conversation

@mutianf

@mutianf mutianf commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Part 1 of 2 for b/584671437.

What changed

  • In SessionPoolImpl, fail pendingRpcs on consecutiveFailures >= consecutive_session_failure_threshold if and only if the pool has no established (READY or IN_USE) sessions (readyCount + inUseCount == 0).
  • Enforce this invariant symmetrically in both onSessionClose() (draining queued pendingRpcs when transitioning to 0 established sessions while consecutiveFailures >= threshold, while preserving pendingRpcs when inUseCount > 0) and PendingVRpc.start() (rejecting new calls during the 60s SessionCreationBudget lockout when no sessions are closing).

@mutianf
mutianf requested review from a team as code owners November 16, 2026 16:59
@mutianf
mutianf added this pull request to stack #15040 November 16, 2026 16:59

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces queue bounding and fast-fail mechanisms to the SessionPoolImpl when consecutive session establishment failures occur or when the pending RPC queue exceeds its maximum capacity. Feedback on these changes highlights two potential issues: first, if lastCloseStatus is OK, rejecting the RPC with an OK status could lead to silent failures, so it should be defensively mapped to UNAVAILABLE; second, trailers can occasionally be null in gRPC, so a defensive null check should be added when assigning lastCloseTrailers to prevent null pointer issues.

…secutive failures

# Conflicts:
#	java-bigtable/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/internal/session/PoolSizerTest.java

# Conflicts:
#	java-bigtable/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/internal/session/SessionPoolImplTest.java
@mutianf
mutianf removed this pull request from stack #15042 November 16, 2026 21:05
@mutianf
mutianf force-pushed the fix-session-pool-pending-rpcs-bound branch from c6ec691 to 54bb35c Compare November 16, 2026 21:05
@mutianf
mutianf changed the base branch from fix-session-list-pending-vrpc-cleanup to main November 16, 2026 21:05
@mutianf
mutianf added this pull request to stack #15043 November 16, 2026 21:05
@mutianf mutianf changed the title fix(bigtable): bound SessionPool pendingRpcs and fast-fail during consecutive failures fix(bigtable): fail pendingRpcs on consecutive failures when pool has no established sessions Nov 16, 2026
private int consecutiveFailures = 0;

@GuardedBy("poolLock")
private Status lastCloseStatus = Status.UNAVAILABLE;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be null?

handle.getSession().close(closeReq);
}
} else if (poolState == PoolState.STARTED) {
if (isFailingWithNoEstablishedSessions()) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the right place for this check? this is the listener for client config udpates, is this the right place to pop closable RPCs? Also, client configuration responses arrives every 5 minutes. So it's longer interval then normal vrpcs deadline.

afterRelease(handle);
toBeClosed = afterRelease(handle);
if (!toBeClosed.isEmpty()) {
closeResult = createConsecutiveFailureResult(lastCloseStatus, lastCloseTrailers);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this mean consecutiveFailure?

afterRelease(handle);
toBeClosed = afterRelease(handle);
if (!toBeClosed.isEmpty()) {
closeResult = createConsecutiveFailureResult(lastCloseStatus, lastCloseTrailers);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question here.

Comment on lines +147 to +153
@GuardedBy("poolLock")
@Nullable
private Status lastCloseStatus = null;

@GuardedBy("poolLock")
@Nullable
private Metadata lastCloseTrailers = null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Instead of storing lastCloseStatus and lastCloseTrailers as two separate fields and calling createConsecutiveFailureResult(...) on every rejected RPC in PendingVRpc.start() (which runs String.format(...) and allocates a new Status + VRpcResult while holding poolLock), consider storing a single pre-built @Nullable private VRpcResult lastFailureResult = null; updated in onSessionClose when prevState == SessionState.STARTING. Both onSessionClose and PendingVRpc.start can then reuse lastFailureResult directly.

Comment on lines 608 to 611
consecutiveFailures = 0;
lastCloseStatus = null;
lastCloseTrailers = null;
consecutiveUnimplementedFailures = 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider extracting a @GuardedBy("poolLock") private void resetConsecutiveFailures() helper for resetting these fields and also calling it in configListener when poolSizer.isDisabled() is true.

Otherwise, if a pool hits consecutiveFailures >= threshold, is disabled via ClientConfiguration, and is later re-enabled, isFailingWithNoEstablishedSessions() will still be true while the newly re-enabled pool's initial sessions are in STARTING. Any RPC arriving before the first handshake completes will fast-fail in PendingVRpc.start() with the stale pre-disable error instead of queueing for the starting sessions.

Comment on lines +1472 to +1483
Field consecutiveFailuresField = SessionPoolImpl.class.getDeclaredField("consecutiveFailures");
consecutiveFailuresField.setAccessible(true);
long deadline = System.currentTimeMillis() + 5000;
while (System.currentTimeMillis() < deadline) {
poolLock.lock();
try {
if (sessionPool.sessions.getAllSessions().isEmpty()
&& consecutiveFailuresField.getInt(sessionPool) >= 10) {
break;
}
} finally {
poolLock.unlock();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this 5s polling loop ever times out on a slow runner before all 50 budget slots finish closing (allSessions is not empty yet), the loop exits silently and rpc.start(...) below could pass by being cancelled via a racing onSessionClose() rather than via PendingVRpc.start().

Could we assert after the loop (or inside poolLock) that sessionPool.sessions.getAllSessions().isEmpty() and consecutiveFailuresField.getInt(sessionPool) >= 10 were actually reached?

Comment on lines +1504 to +1542
@Test
@SuppressWarnings("GuardedBy")
void consecutiveFailuresPreservesPendingRpcsWhilePoolHasInUseSessions() throws Exception {
sessionPool.start(OpenFakeSessionRequest.getDefaultInstance(), new Metadata());
waitForOpenRequestCount(5, Duration.ofSeconds(5));
assertPoolServesVRpc(sessionPool, "initial started pool");

ReentrantLock poolLock = extractPoolLock(sessionPool);
Field consecutiveFailuresField = SessionPoolImpl.class.getDeclaredField("consecutiveFailures");
consecutiveFailuresField.setAccessible(true);

UnaryResponseFuture<SessionFakeScriptedResponse> checkedOutFuture = new UnaryResponseFuture<>();
UnaryResponseFuture<SessionFakeScriptedResponse> pendingFuture = new UnaryResponseFuture<>();
VRpc<SessionFakeScriptedRequest, SessionFakeScriptedResponse> checkedOutRpc;
poolLock.lock();
try {
// Mark 4 of the 5 ready sessions closing, then check out the remaining 1 ready session via
// newCall() so readyCount == 0 and inUseCount == 1.
List<SessionHandle> handles = new ArrayList<>(sessionPool.sessions.getAllSessions());
for (int i = 1; i < handles.size(); i++) {
handles.get(i).onSessionClosing();
}
checkedOutRpc = sessionPool.newCall(FakeDescriptor.SCRIPTED);
assertThat(sessionPool.sessions.getStats().getReadyCount()).isEqualTo(0);
assertThat(sessionPool.sessions.getStats().getInUseCount()).isEqualTo(1);

// Simulate consecutive STARTING failures reaching threshold while inUseCount == 1.
Field lastCloseStatusField = SessionPoolImpl.class.getDeclaredField("lastCloseStatus");
lastCloseStatusField.setAccessible(true);
lastCloseStatusField.set(sessionPool, Status.INTERNAL.withDescription("fake internal error"));
consecutiveFailuresField.setInt(sessionPool, 10);

// A new PendingVRpc started while inUseCount > 0 must queue instead of failing fast.
VRpc<SessionFakeScriptedRequest, SessionFakeScriptedResponse> pendingRpc =
sessionPool.newCall(FakeDescriptor.SCRIPTED);
pendingRpc.start(
SessionFakeScriptedRequest.newBuilder().setTag(0).build(),
VRpcCallContext.create(Deadline.after(1, TimeUnit.MINUTES), true, vrpcTracer),
pendingFuture);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of things in this test setup:

  1. waitForOpenRequestCount(5, ...) only checks that the server received 5 OpenSession requests, not that all 5 sessions have completed onSessionReady() on the client (and assertPoolServesVRpc only guarantees 1 ready session). Because getAllSessions() is a HashSet that includes STARTING sessions, if one of the 5 sessions is still STARTING when poolLock.lock() runs and happens to land at handles.get(0), the loop marking handles.get(1..4) as closing will mark all READY sessions closing (readyCount == 0), causing assertThat(getInUseCount()).isEqualTo(1) to fail flakily. Polling until sessionPool.sessions.getStats().getReadyCount() == 5 under poolLock avoids this.
  2. Setting consecutiveFailures = 10 via reflection before calling pendingRpc.start(...) only exercises the !hasEstablishedSession() guard in PendingVRpc.start(), not the one in onSessionClose() when prevState == SessionState.STARTING. If we also invoke onSessionClose for a STARTING session while pendingRpc is queued in pendingRpcs and inUseCount == 1 (and assert pendingFuture.isDone() is still false), this test will cover both PendingVRpc.start() and onSessionClose().

@mutianf
mutianf enabled stacked auto-merge (squash) November 17, 2026 17:57
@mutianf
mutianf merged commit 3d29024 into main Nov 17, 2026
208 checks passed
@mutianf
mutianf deleted the fix-session-pool-pending-rpcs-bound branch November 17, 2026 18:08
mutianf added a commit that referenced this pull request Nov 18, 2026
…elPoolDpImpl (#15039)

Part 2 of 2 for b/584671437 (stacked on #15038).

### What changed
- Only mark a session stream established in `ChannelPoolDpImpl` when
`OpenSessionResponse` arrives (`onMessage`), rather than when initial
`PeerInfo` headers arrive (`onBeforeSessionStart`).
- Count non-OK stream closes that occur before `OpenSessionResponse` as
consecutive failures and evict the failed `routeObservations` entry so
subsequent `OpenSession` attempts do not remain pinned to a failing
AFE/subzone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants