Repository navigation
fix(bigtable): remove dead SessionList.inUseSessions and clear PendingVRpc references #15030
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
24d6574
86f6d01
c5f3794
5110ab6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,14 +31,14 @@ class PoolSizerTest { | |
|
|
||
| private static final float IDLE_SESSION_HEADROOM_RATIO = 0.5f; | ||
| private static final int MIN_IDLE_SESSIONS = 5; | ||
| private static final int MAX_IDLE_SESSIONS = 400; | ||
| private static final int MAX_SESSIONS = 400; | ||
| private static final int PENDING_RPCS_PER_SESSION = 10; | ||
|
|
||
| private static final SessionClientConfiguration.SessionPoolConfiguration DEFAULT_CONFIG = | ||
| SessionClientConfiguration.SessionPoolConfiguration.newBuilder() | ||
| .setHeadroom(IDLE_SESSION_HEADROOM_RATIO) | ||
| .setMinSessionCount(MIN_IDLE_SESSIONS) | ||
| .setMaxSessionCount(MAX_IDLE_SESSIONS) | ||
| .setMaxSessionCount(MAX_SESSIONS) | ||
| .setNewSessionQueueLength(PENDING_RPCS_PER_SESSION) | ||
| .build(); | ||
|
|
||
|
|
@@ -193,7 +193,7 @@ void testEquilibriumWhenStartingWithLoad() { | |
| int sessionsNeeded = forPending + stats.getInUseCount(); | ||
| int desiredIdleSessions = (int) Math.ceil(sessionsNeeded * IDLE_SESSION_HEADROOM_RATIO); | ||
| desiredIdleSessions = | ||
| Math.max(Math.min(desiredIdleSessions, MAX_IDLE_SESSIONS), MIN_IDLE_SESSIONS); | ||
| Math.max(Math.min(desiredIdleSessions, MAX_SESSIONS), MIN_IDLE_SESSIONS); | ||
| int desiredCapacity = sessionsNeeded + desiredIdleSessions; | ||
|
|
||
| assertThat(sizer.getScaleDelta()) | ||
|
|
@@ -249,4 +249,26 @@ void testEmptyConfig() { | |
| assertThat(sizer.handleGoAway(GoAwayResponse.getDefaultInstance())).isFalse(); | ||
| assertThat(sizer.handleSessionClose(Status.getDefaultInstance())).isFalse(); | ||
| } | ||
|
|
||
| @Test | ||
| void testDesiredCapacityCappedAtMaxSessions() { | ||
| PoolStats stats = new PoolStats(); | ||
| AtomicInteger pendingCount = new AtomicInteger(0); | ||
| PoolSizer sizer = new PoolSizer(stats, pendingCount::get, DEFAULT_CONFIG); | ||
|
|
||
| // MAX_SESSIONS in use + 2 ready -> immediateCapacity = MAX_SESSIONS + 2. | ||
| // Total desiredCapacity is capped at MAX_SESSIONS, so scaleDelta is -2. | ||
| stats.getTestHelper().setInUseCount(MAX_SESSIONS); | ||
| stats.getTestHelper().setReadyCount(2); | ||
| stats.getTestHelper().setExpectedCapacity(MAX_SESSIONS + 2); | ||
| assertThat(sizer.getScaleDelta()).isEqualTo(-2); | ||
|
|
||
| // When MAX_SESSIONS are in use and 50 RPCs are queued, scaleDelta must be 0 | ||
| // rather than scaling above MAX_SESSIONS. | ||
| stats.getTestHelper().setReadyCount(0); | ||
| stats.getTestHelper().setExpectedCapacity(MAX_SESSIONS); | ||
| pendingCount.set(50); | ||
| assertThat(sizer.getScaleDelta()).isEqualTo(0); | ||
| assertThat(sizer.handleNewCall()).isFalse(); | ||
| } | ||
|
Comment on lines
+253
to
+273
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This test asserts that the total pool capacity is capped at |
||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a critical concurrency race condition in
PendingVRpcbetweencancel,cancelWithResult, anddrainTo.The Issue
The tasks submitted to
ctx.getExecutor()run asynchronously and potentially concurrently on a multi-threaded executor. Since the fieldsisCancelled,req,listener, andrealCallare notvolatileand are accessed/modified without any synchronization inside these tasks, this leads to data races and race conditions where the gRPC/VRpc contract of a singleonClosecall may be violated.Suggested Fix
Use a
synchronized(PendingVRpc.this)block inside the runnables to ensure atomic state transitions. Note that per repository rules, since we are using a common lock, explicit atomic state transitions (likecompareAndSet) are not strictly necessary. EnsureBigtableTimer.Timeoutis cancelled usingcancel()without arguments as per repository standards.References