Skip to content

fix(bigtable): count pre-OpenSessionResponse stream failures in ChannelPoolDpImpl - #15032

Closed
mutianf wants to merge 9 commits into
googleapis:mainfrom
mutianf:fix-channel-pool-pre-open-session-failure
Closed

mutianf wants to merge 9 commits into
googleapis:mainfrom
mutianf:fix-channel-pool-pre-open-session-failure

Conversation

@mutianf

@mutianf mutianf commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Ensures stream closes with non-OK status before OpenSessionResponse is received are counted as consecutive session-establishment failures in ChannelPoolDpImpl, even when PeerInfo headers were sent prior to stream rejection.

Stack

  1. fix(bigtable): cap PoolSizer desiredCapacity at maxSessions #15029
  2. fix(bigtable): remove dead SessionList.inUseSessions and clear PendingVRpc references #15030
  3. fix(bigtable): bound SessionPool pendingRpcs and fast-fail during consecutive failures #15031
  4. fix(bigtable): count pre-OpenSessionResponse stream failures in ChannelPoolDpImpl #15032 👈

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the session pool implementation by capping desired capacity, bounding the pending RPC queue, fast-failing requests during budget lockouts, and clearing references upon cancellation to prevent memory leaks. The review feedback identifies several concurrency issues, recommending synchronization within executor tasks in SessionPoolImpl to avoid race conditions, declaring sessionEstablished as volatile in ChannelPoolDpImpl for thread visibility, and using double-checked locking to safely update session establishment state.

Comment on lines 1065 to 1078
() -> {
if (isCancelled) return;
isCancelled = true;
VRpcListener<RespT> pendingListener = listener;
req = null;
listener = null;
if (realCall != null) {
if (!onlyCancelPendingCall) {
realCall.cancel(status.getDescription(), status.getCause());
}
} else {
listener.onClose(VRpcResult.createRejectedError(status));
} else if (pendingListener != null) {
pendingListener.onClose(VRpcResult.createRejectedError(status));
}
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Since ctx.getExecutor() can be a multi-threaded executor, there is a potential check-then-act race condition on isCancelled and a data race on realCall, req, and listener. Synchronizing on PendingVRpc.this inside the executor task ensures thread safety and prevents duplicate calls to onClose or concurrent modification issues.

Suggested change
() -> {
if (isCancelled) return;
isCancelled = true;
VRpcListener<RespT> pendingListener = listener;
req = null;
listener = null;
if (realCall != null) {
if (!onlyCancelPendingCall) {
realCall.cancel(status.getDescription(), status.getCause());
}
} else {
listener.onClose(VRpcResult.createRejectedError(status));
} else if (pendingListener != null) {
pendingListener.onClose(VRpcResult.createRejectedError(status));
}
});
() -> {
VRpcListener<RespT> pendingListener;
VRpc<ReqT, RespT> localRealCall;
synchronized (PendingVRpc.this) {
if (isCancelled) return;
isCancelled = true;
pendingListener = listener;
req = null;
listener = null;
localRealCall = realCall;
}
if (localRealCall != null) {
if (!onlyCancelPendingCall) {
localRealCall.cancel(status.getDescription(), status.getCause());
}
} else if (pendingListener != null) {
pendingListener.onClose(VRpcResult.createRejectedError(status));
}
});

Comment on lines 1087 to 1098
ctx.getExecutor()
.execute(
() -> {
if (isCancelled) return;
isCancelled = true;
listener.onClose(result);
VRpcListener<RespT> pendingListener = listener;
req = null;
listener = null;
if (pendingListener != null) {
pendingListener.onClose(result);
}
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Similarly to the cancel method, cancelWithResult should synchronize on PendingVRpc.this to avoid race conditions on isCancelled, req, and listener when executed on a multi-threaded executor.

      ctx.getExecutor()
          .execute(
              () -> {
                VRpcListener<RespT> pendingListener;
                synchronized (PendingVRpc.this) {
                  if (isCancelled) return;
                  isCancelled = true;
                  pendingListener = listener;
                  req = null;
                  listener = null;
                }
                if (pendingListener != null) {
                  pendingListener.onClose(result);
                }
              });

return new SessionStreamImpl(innerCall) {
// null until onBeforeSessionStart fires
@Nullable AfeId afeId = null;
boolean sessionEstablished = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The sessionEstablished flag is read outside of the synchronized block in onMessage (line 332). To prevent thread visibility issues and ensure memory consistency across threads, this field should be declared volatile.

Suggested change
boolean sessionEstablished = false;
volatile boolean sessionEstablished = false;
References
  1. Ensure that fields used for cross-thread state checks, such as exceptions set by background threads and read by application threads, are declared as volatile to guarantee memory visibility and establish a proper happens-before relationship under the Java Memory Model.

Comment on lines +331 to +340
public void onMessage(SessionResponse msg) {
if (!sessionEstablished && msg.hasOpenSession()) {
synchronized (ChannelPoolDpImpl.this) {
sessionEstablished = true;
fw.consecutiveFailures = 0;
recycleBackoff = INITIAL_RECYCLE_BACKOFF;
}
}
super.onMessage(msg);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To avoid redundant writes and ensure thread safety when resetting consecutive failures, we should use a double-checked locking pattern under the synchronized block after checking sessionEstablished.

              @Override
              public void onMessage(SessionResponse msg) {
                if (!sessionEstablished && msg.hasOpenSession()) {
                  synchronized (ChannelPoolDpImpl.this) {
                    if (!sessionEstablished) {
                      sessionEstablished = true;
                      fw.consecutiveFailures = 0;
                      recycleBackoff = INITIAL_RECYCLE_BACKOFF;
                    }
                  }
                }
                super.onMessage(msg);
              }

@mutianf mutianf closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant