Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request enhances the Bigtable session pool by capping desired capacity, removing unused session tracking, and implementing queue bounds and fast-failing mechanisms for pending RPCs. It also refactors pending RPC cancellation to prevent reference leaks. The feedback suggests addressing potential thread-safety and visibility issues in PendingVRpc due to unsynchronized state mutations, defensively handling null trailers to prevent NullPointerExceptions, and ensuring that Status.OK is not misleadingly returned during consecutive failure fast-path rejections.
| void cancelWithResult(VRpcResult result) { | ||
| if (deadlineTimeout != null) { | ||
| deadlineTimeout.cancel(); | ||
| BigtableTimer.Timeout timeout = deadlineTimeout; | ||
| deadlineTimeout = null; | ||
| if (timeout != null) { | ||
| timeout.cancel(); | ||
| } | ||
| ctx.getExecutor() | ||
| .execute( | ||
| () -> { | ||
| if (isCancelled) return; | ||
| isCancelled = true; | ||
| listener.onClose(result); | ||
| VRpcListener<RespT> pendingListener = listener; | ||
| req = null; | ||
| listener = null; | ||
| if (pendingListener != null) { | ||
| pendingListener.onClose(result); | ||
| } | ||
| }); | ||
| } |
There was a problem hiding this comment.
Since ctx.getExecutor() is typically a multi-threaded executor (such as the gRPC callback executor or a user-provided thread pool), tasks submitted to it can run concurrently. Because isCancelled, req, and listener are mutated and read across cancel, cancelWithResult, and drainTo without synchronization, this introduces a data race and visibility issues. To ensure thread safety, we should synchronize on PendingVRpc.this when reading or writing these fields inside the executor tasks. Note that before refactoring to move the callback outside of the synchronized block to avoid potential deadlocks, we should verify if a true circular lock dependency exists, as nested lock acquisition is safe if the methods called within the locked block only perform thread-safe operations and do not acquire other locks or trigger callbacks that could reverse the lock acquisition order.
void cancelWithResult(VRpcResult result) {
BigtableTimer.Timeout timeout = deadlineTimeout;
deadlineTimeout = null;
if (timeout != null) {
timeout.cancel();
}
ctx.getExecutor()
.execute(
() -> {
VRpcListener<RespT> pendingListener;
synchronized (PendingVRpc.this) {
if (isCancelled) return;
isCancelled = true;
pendingListener = listener;
req = null;
listener = null;
}
if (pendingListener != null) {
pendingListener.onClose(result);
}
});
}References
- Before refactoring code to avoid potential deadlocks by moving method calls outside of synchronized blocks, verify if a true circular lock dependency exists. Nested lock acquisition is safe if the methods called within the locked block only perform thread-safe operations (such as updating atomic variables or statistics) and do not acquire other locks or trigger callbacks that could reverse the lock acquisition order.
| lastCloseStatus = status; | ||
| lastCloseTrailers = trailers; |
There was a problem hiding this comment.
In gRPC, the trailers parameter in onClose can sometimes be null (e.g., in certain failure modes or unit tests). To prevent potential NullPointerException issues when referencing lastCloseTrailers later, we should defensively check if trailers is null and default to an empty Metadata instance.
| lastCloseStatus = status; | |
| lastCloseTrailers = trailers; | |
| lastCloseStatus = status; | |
| lastCloseTrailers = trailers != null ? trailers : new Metadata(); |
| Status.fromCode(lastCloseStatus.getCode()) | ||
| .withDescription( | ||
| String.format( | ||
| "Session failed with consecutive failures. Most recent server status:" | ||
| + " %s, metadata: %s.", | ||
| lastCloseStatus, lastCloseTrailers))); |
There was a problem hiding this comment.
If lastCloseStatus is somehow Status.OK (which shouldn't normally happen for a session close failure, but is possible in mock tests or edge cases), calling Status.fromCode(lastCloseStatus.getCode()) will return Status.OK. Returning Status.OK for a rejected error is highly misleading to callers. We should defensively ensure that we map Status.Code.OK to an error status like Status.Code.UNAVAILABLE.
Status.fromCode(
lastCloseStatus.getCode() == Status.Code.OK
? Status.Code.UNAVAILABLE
: lastCloseStatus.getCode())
.withDescription(
String.format(
"Session failed with consecutive failures. Most recent server status:"
+ " %s, metadata: %s.",
lastCloseStatus, lastCloseTrailers)))…-pool-pending-rpcs-bound
Bounds
SessionPoolImpl.pendingRpcsatmax_session_count * new_session_queue_lengthand fast-fails incoming calls inPendingVRpc.start()when the pool is in a consecutive-failure state with no ready sessions (e.g. during the 60sSessionCreationBudgetlockout) or when the pending queue is full.Stack