Skip to content

fix(bigtable): bound SessionPool pendingRpcs and fast-fail during consecutive failures - #15031

Closed
mutianf wants to merge 7 commits into
googleapis:mainfrom
mutianf:fix-session-pool-pending-rpcs-bound
Closed

mutianf wants to merge 7 commits into
googleapis:mainfrom
mutianf:fix-session-pool-pending-rpcs-bound

Conversation

@mutianf

@mutianf mutianf commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Bounds SessionPoolImpl.pendingRpcs at max_session_count * new_session_queue_length and fast-fails incoming calls in PendingVRpc.start() when the pool is in a consecutive-failure state with no ready sessions (e.g. during the 60s SessionCreationBudget lockout) or when the pending queue is full.

Stack

  1. fix(bigtable): cap PoolSizer desiredCapacity at maxSessions #15029
  2. fix(bigtable): remove dead SessionList.inUseSessions and clear PendingVRpc references #15030
  3. fix(bigtable): bound SessionPool pendingRpcs and fast-fail during consecutive failures #15031 👈
  4. fix(bigtable): count pre-OpenSessionResponse stream failures in ChannelPoolDpImpl #15032

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the Bigtable session pool by capping desired capacity, removing unused session tracking, and implementing queue bounds and fast-failing mechanisms for pending RPCs. It also refactors pending RPC cancellation to prevent reference leaks. The feedback suggests addressing potential thread-safety and visibility issues in PendingVRpc due to unsynchronized state mutations, defensively handling null trailers to prevent NullPointerExceptions, and ensuring that Status.OK is not misleadingly returned during consecutive failure fast-path rejections.

Comment on lines 1081 to 1099
void cancelWithResult(VRpcResult result) {
if (deadlineTimeout != null) {
deadlineTimeout.cancel();
BigtableTimer.Timeout timeout = deadlineTimeout;
deadlineTimeout = null;
if (timeout != null) {
timeout.cancel();
}
ctx.getExecutor()
.execute(
() -> {
if (isCancelled) return;
isCancelled = true;
listener.onClose(result);
VRpcListener<RespT> pendingListener = listener;
req = null;
listener = null;
if (pendingListener != null) {
pendingListener.onClose(result);
}
});
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Since ctx.getExecutor() is typically a multi-threaded executor (such as the gRPC callback executor or a user-provided thread pool), tasks submitted to it can run concurrently. Because isCancelled, req, and listener are mutated and read across cancel, cancelWithResult, and drainTo without synchronization, this introduces a data race and visibility issues. To ensure thread safety, we should synchronize on PendingVRpc.this when reading or writing these fields inside the executor tasks. Note that before refactoring to move the callback outside of the synchronized block to avoid potential deadlocks, we should verify if a true circular lock dependency exists, as nested lock acquisition is safe if the methods called within the locked block only perform thread-safe operations and do not acquire other locks or trigger callbacks that could reverse the lock acquisition order.

    void cancelWithResult(VRpcResult result) {
      BigtableTimer.Timeout timeout = deadlineTimeout;
      deadlineTimeout = null;
      if (timeout != null) {
        timeout.cancel();
      }
      ctx.getExecutor()
          .execute(
              () -> {
                VRpcListener<RespT> pendingListener;
                synchronized (PendingVRpc.this) {
                  if (isCancelled) return;
                  isCancelled = true;
                  pendingListener = listener;
                  req = null;
                  listener = null;
                }
                if (pendingListener != null) {
                  pendingListener.onClose(result);
                }
              });
    }
References
  1. Before refactoring code to avoid potential deadlocks by moving method calls outside of synchronized blocks, verify if a true circular lock dependency exists. Nested lock acquisition is safe if the methods called within the locked block only perform thread-safe operations (such as updating atomic variables or statistics) and do not acquire other locks or trigger callbacks that could reverse the lock acquisition order.

Comment on lines +738 to +739
lastCloseStatus = status;
lastCloseTrailers = trailers;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In gRPC, the trailers parameter in onClose can sometimes be null (e.g., in certain failure modes or unit tests). To prevent potential NullPointerException issues when referencing lastCloseTrailers later, we should defensively check if trailers is null and default to an empty Metadata instance.

Suggested change
lastCloseStatus = status;
lastCloseTrailers = trailers;
lastCloseStatus = status;
lastCloseTrailers = trailers != null ? trailers : new Metadata();

Comment on lines +976 to +981
Status.fromCode(lastCloseStatus.getCode())
.withDescription(
String.format(
"Session failed with consecutive failures. Most recent server status:"
+ " %s, metadata: %s.",
lastCloseStatus, lastCloseTrailers)));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If lastCloseStatus is somehow Status.OK (which shouldn't normally happen for a session close failure, but is possible in mock tests or edge cases), calling Status.fromCode(lastCloseStatus.getCode()) will return Status.OK. Returning Status.OK for a rejected error is highly misleading to callers. We should defensively ensure that we map Status.Code.OK to an error status like Status.Code.UNAVAILABLE.

                  Status.fromCode(
                          lastCloseStatus.getCode() == Status.Code.OK
                              ? Status.Code.UNAVAILABLE
                              : lastCloseStatus.getCode())
                      .withDescription(
                          String.format(
                              "Session failed with consecutive failures. Most recent server status:"
                                  + " %s, metadata: %s.",
                              lastCloseStatus, lastCloseTrailers)))

@mutianf mutianf closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant