Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors session pool management by removing the unused inUseSessions set, clearing references in PendingVRpc after draining or cancellation to prevent memory leaks, and capping the desired pool capacity. The review feedback identifies a logical error in PoolSizer.java where total capacity is incorrectly capped at maxIdleSessions instead of only limiting idle sessions, which also invalidates the newly added test in PoolSizerTest.java. Additionally, a critical concurrency race condition is highlighted in PendingVRpc within SessionPoolImpl.java due to unsynchronized access to non-volatile fields across asynchronous tasks, which should be resolved using synchronized blocks.
| Math.max(Math.min(unboundedDesiredIdleSessions, maxIdleSessions), minIdleSessions); | ||
|
|
||
| int desiredCapacity = sessionsInUse + desiredIdleSessions; | ||
| int desiredCapacity = Math.min(sessionsInUse + desiredIdleSessions, maxIdleSessions); |
There was a problem hiding this comment.
Capping the total desiredCapacity at maxIdleSessions is a logical error. maxIdleSessions is the limit on the number of idle sessions, not the limit on the total sessions (active + idle).
By capping desiredCapacity at maxIdleSessions, the total pool size can never exceed the maximum idle sessions limit, even under heavy load when sessionsInUse is high. This will prevent the pool from scaling up to handle concurrent active requests unless maxIdleSessions is set to an excessively high value, which would then defeat the purpose of keeping the idle pool size small during low-traffic periods.
If the intention was to cap the total capacity, it should be capped by a separate maxSessions limit, not maxIdleSessions.
| int desiredCapacity = Math.min(sessionsInUse + desiredIdleSessions, maxIdleSessions); | |
| int desiredCapacity = sessionsInUse + desiredIdleSessions; |
| () -> { | ||
| if (isCancelled) return; | ||
| isCancelled = true; | ||
| VRpcListener<RespT> pendingListener = listener; | ||
| req = null; | ||
| listener = null; | ||
| if (realCall != null) { | ||
| if (!onlyCancelPendingCall) { | ||
| realCall.cancel(status.getDescription(), status.getCause()); | ||
| } | ||
| } else { | ||
| listener.onClose(VRpcResult.createRejectedError(status)); | ||
| } else if (pendingListener != null) { | ||
| pendingListener.onClose(VRpcResult.createRejectedError(status)); | ||
| } | ||
| }); |
There was a problem hiding this comment.
There is a critical concurrency race condition in PendingVRpc between cancel, cancelWithResult, and drainTo.
The Issue
The tasks submitted to ctx.getExecutor() run asynchronously and potentially concurrently on a multi-threaded executor. Since the fields isCancelled, req, listener, and realCall are not volatile and are accessed/modified without any synchronization inside these tasks, this leads to data races and race conditions where the gRPC/VRpc contract of a single onClose call may be violated.
Suggested Fix
Use a synchronized(PendingVRpc.this) block inside the runnables to ensure atomic state transitions. Note that per repository rules, since we are using a common lock, explicit atomic state transitions (like compareAndSet) are not strictly necessary. Ensure BigtableTimer.Timeout is cancelled using cancel() without arguments as per repository standards.
private void cancel(Status status, boolean onlyCancelPendingCall) {
BigtableTimer.Timeout timeout = deadlineTimeout;
deadlineTimeout = null;
if (timeout != null) {
timeout.cancel();
}
ctx.getExecutor()
.execute(
() -> {
VRpcListener<RespT> pendingListener;
VRpc<ReqT, RespT> call;
boolean shouldCancelCall = false;
synchronized (PendingVRpc.this) {
if (isCancelled) return;
isCancelled = true;
pendingListener = listener;
req = null;
listener = null;
call = realCall;
if (call != null) {
shouldCancelCall = !onlyCancelPendingCall;
}
}
if (call != null) {
if (shouldCancelCall) {
call.cancel(status.getDescription(), status.getCause());
}
} else if (pendingListener != null) {
pendingListener.onClose(VRpcResult.createRejectedError(status));
}
});
}References
- When concurrent operations are protected by a common lock (e.g., synchronized (this)), atomic state transitions are not strictly necessary.
- Use cancel() without arguments when canceling a BigtableTimer.Timeout instance.
| @Test | ||
| void testDesiredCapacityCappedAtMaxIdleSessions() { | ||
| PoolStats stats = new PoolStats(); | ||
| AtomicInteger pendingCount = new AtomicInteger(0); | ||
| PoolSizer sizer = new PoolSizer(stats, pendingCount::get, DEFAULT_CONFIG); | ||
|
|
||
| // MAX_IDLE_SESSIONS in use + 2 ready -> immediateCapacity = MAX_IDLE_SESSIONS + 2. | ||
| // Total desiredCapacity is capped at MAX_IDLE_SESSIONS, so scaleDelta is -2. | ||
| stats.getTestHelper().setInUseCount(MAX_IDLE_SESSIONS); | ||
| stats.getTestHelper().setReadyCount(2); | ||
| stats.getTestHelper().setExpectedCapacity(MAX_IDLE_SESSIONS + 2); | ||
| assertThat(sizer.getScaleDelta()).isEqualTo(-2); | ||
|
|
||
| // When MAX_IDLE_SESSIONS are in use and 50 RPCs are queued, scaleDelta must be 0 | ||
| // rather than scaling above MAX_IDLE_SESSIONS. | ||
| stats.getTestHelper().setReadyCount(0); | ||
| stats.getTestHelper().setExpectedCapacity(MAX_IDLE_SESSIONS); | ||
| pendingCount.set(50); | ||
| assertThat(sizer.getScaleDelta()).isEqualTo(0); | ||
| assertThat(sizer.handleNewCall()).isFalse(); | ||
| } |
There was a problem hiding this comment.
This test asserts that the total pool capacity is capped at maxIdleSessions. As discussed in the feedback for PoolSizer.java, this capping behavior is incorrect because maxIdleSessions should only limit the number of idle sessions, not the total sessions (active + idle). Once the capping logic in PoolSizer.java is corrected, this test should be updated or removed to reflect the correct scaling behavior.
Removes the unused
SessionList.inUseSessionsset and clearsPendingVRpcreferences (req,listener, anddeadlineTimeout) once a pending vRPC is drained to a real call, cancelled, or completed.Stack