Skip to content

fix(bigtable): remove dead SessionList.inUseSessions and clear PendingVRpc references - #15030

Closed
mutianf wants to merge 4 commits into
googleapis:mainfrom
mutianf:fix-session-list-pending-vrpc-cleanup
Closed

mutianf wants to merge 4 commits into
googleapis:mainfrom
mutianf:fix-session-list-pending-vrpc-cleanup

Conversation

@mutianf

@mutianf mutianf commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors session pool management by removing the unused inUseSessions set, clearing references in PendingVRpc after draining or cancellation to prevent memory leaks, and capping the desired pool capacity. The review feedback identifies a logical error in PoolSizer.java where total capacity is incorrectly capped at maxIdleSessions instead of only limiting idle sessions, which also invalidates the newly added test in PoolSizerTest.java. Additionally, a critical concurrency race condition is highlighted in PendingVRpc within SessionPoolImpl.java due to unsynchronized access to non-volatile fields across asynchronous tasks, which should be resolved using synchronized blocks.

Math.max(Math.min(unboundedDesiredIdleSessions, maxIdleSessions), minIdleSessions);

int desiredCapacity = sessionsInUse + desiredIdleSessions;
int desiredCapacity = Math.min(sessionsInUse + desiredIdleSessions, maxIdleSessions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Capping the total desiredCapacity at maxIdleSessions is a logical error. maxIdleSessions is the limit on the number of idle sessions, not the limit on the total sessions (active + idle).

By capping desiredCapacity at maxIdleSessions, the total pool size can never exceed the maximum idle sessions limit, even under heavy load when sessionsInUse is high. This will prevent the pool from scaling up to handle concurrent active requests unless maxIdleSessions is set to an excessively high value, which would then defeat the purpose of keeping the idle pool size small during low-traffic periods.

If the intention was to cap the total capacity, it should be capped by a separate maxSessions limit, not maxIdleSessions.

Suggested change
int desiredCapacity = Math.min(sessionsInUse + desiredIdleSessions, maxIdleSessions);
int desiredCapacity = sessionsInUse + desiredIdleSessions;

Comment on lines 1027 to 1040
() -> {
if (isCancelled) return;
isCancelled = true;
VRpcListener<RespT> pendingListener = listener;
req = null;
listener = null;
if (realCall != null) {
if (!onlyCancelPendingCall) {
realCall.cancel(status.getDescription(), status.getCause());
}
} else {
listener.onClose(VRpcResult.createRejectedError(status));
} else if (pendingListener != null) {
pendingListener.onClose(VRpcResult.createRejectedError(status));
}
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There is a critical concurrency race condition in PendingVRpc between cancel, cancelWithResult, and drainTo.

The Issue

The tasks submitted to ctx.getExecutor() run asynchronously and potentially concurrently on a multi-threaded executor. Since the fields isCancelled, req, listener, and realCall are not volatile and are accessed/modified without any synchronization inside these tasks, this leads to data races and race conditions where the gRPC/VRpc contract of a single onClose call may be violated.

Suggested Fix

Use a synchronized(PendingVRpc.this) block inside the runnables to ensure atomic state transitions. Note that per repository rules, since we are using a common lock, explicit atomic state transitions (like compareAndSet) are not strictly necessary. Ensure BigtableTimer.Timeout is cancelled using cancel() without arguments as per repository standards.

    private void cancel(Status status, boolean onlyCancelPendingCall) {
      BigtableTimer.Timeout timeout = deadlineTimeout;
      deadlineTimeout = null;
      if (timeout != null) {
        timeout.cancel();
      }
      ctx.getExecutor()
          .execute(
              () -> {
                VRpcListener<RespT> pendingListener;
                VRpc<ReqT, RespT> call;
                boolean shouldCancelCall = false;
                synchronized (PendingVRpc.this) {
                  if (isCancelled) return;
                  isCancelled = true;
                  pendingListener = listener;
                  req = null;
                  listener = null;
                  call = realCall;
                  if (call != null) {
                    shouldCancelCall = !onlyCancelPendingCall;
                  }
                }
                if (call != null) {
                  if (shouldCancelCall) {
                    call.cancel(status.getDescription(), status.getCause());
                  }
                } else if (pendingListener != null) {
                  pendingListener.onClose(VRpcResult.createRejectedError(status));
                }
              });
    }
References
  1. When concurrent operations are protected by a common lock (e.g., synchronized (this)), atomic state transitions are not strictly necessary.
  2. Use cancel() without arguments when canceling a BigtableTimer.Timeout instance.

Comment on lines +253 to +273
@Test
void testDesiredCapacityCappedAtMaxIdleSessions() {
PoolStats stats = new PoolStats();
AtomicInteger pendingCount = new AtomicInteger(0);
PoolSizer sizer = new PoolSizer(stats, pendingCount::get, DEFAULT_CONFIG);

// MAX_IDLE_SESSIONS in use + 2 ready -> immediateCapacity = MAX_IDLE_SESSIONS + 2.
// Total desiredCapacity is capped at MAX_IDLE_SESSIONS, so scaleDelta is -2.
stats.getTestHelper().setInUseCount(MAX_IDLE_SESSIONS);
stats.getTestHelper().setReadyCount(2);
stats.getTestHelper().setExpectedCapacity(MAX_IDLE_SESSIONS + 2);
assertThat(sizer.getScaleDelta()).isEqualTo(-2);

// When MAX_IDLE_SESSIONS are in use and 50 RPCs are queued, scaleDelta must be 0
// rather than scaling above MAX_IDLE_SESSIONS.
stats.getTestHelper().setReadyCount(0);
stats.getTestHelper().setExpectedCapacity(MAX_IDLE_SESSIONS);
pendingCount.set(50);
assertThat(sizer.getScaleDelta()).isEqualTo(0);
assertThat(sizer.handleNewCall()).isFalse();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This test asserts that the total pool capacity is capped at maxIdleSessions. As discussed in the feedback for PoolSizer.java, this capping behavior is incorrect because maxIdleSessions should only limit the number of idle sessions, not the total sessions (active + idle). Once the capping logic in PoolSizer.java is corrected, this test should be updated or removed to reflect the correct scaling behavior.

@mutianf mutianf closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant