Repository navigation
fix(bigtable): fail pendingRpcs on consecutive failures when pool has no established sessions - #15038
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces queue bounding and fast-fail mechanisms to the SessionPoolImpl when consecutive session establishment failures occur or when the pending RPC queue exceeds its maximum capacity. Feedback on these changes highlights two potential issues: first, if lastCloseStatus is OK, rejecting the RPC with an OK status could lead to silent failures, so it should be defensively mapped to UNAVAILABLE; second, trailers can occasionally be null in gRPC, so a defensive null check should be added when assigning lastCloseTrailers to prevent null pointer issues.
81131c3 to
c6ec691
Compare
…secutive failures # Conflicts: # java-bigtable/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/internal/session/PoolSizerTest.java # Conflicts: # java-bigtable/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/internal/session/SessionPoolImplTest.java
…l has no established sessions
…block in onSessionClose
c6ec691 to
54bb35c
Compare
| private int consecutiveFailures = 0; | ||
|
|
||
| @GuardedBy("poolLock") | ||
| private Status lastCloseStatus = Status.UNAVAILABLE; |
There was a problem hiding this comment.
should this be null?
| handle.getSession().close(closeReq); | ||
| } | ||
| } else if (poolState == PoolState.STARTED) { | ||
| if (isFailingWithNoEstablishedSessions()) { |
There was a problem hiding this comment.
is this the right place for this check? this is the listener for client config udpates, is this the right place to pop closable RPCs? Also, client configuration responses arrives every 5 minutes. So it's longer interval then normal vrpcs deadline.
| afterRelease(handle); | ||
| toBeClosed = afterRelease(handle); | ||
| if (!toBeClosed.isEmpty()) { | ||
| closeResult = createConsecutiveFailureResult(lastCloseStatus, lastCloseTrailers); |
There was a problem hiding this comment.
why does this mean consecutiveFailure?
| afterRelease(handle); | ||
| toBeClosed = afterRelease(handle); | ||
| if (!toBeClosed.isEmpty()) { | ||
| closeResult = createConsecutiveFailureResult(lastCloseStatus, lastCloseTrailers); |
There was a problem hiding this comment.
same question here.
…lose and PendingVRpc.start
… with no established sessions
| @GuardedBy("poolLock") | ||
| @Nullable | ||
| private Status lastCloseStatus = null; | ||
|
|
||
| @GuardedBy("poolLock") | ||
| @Nullable | ||
| private Metadata lastCloseTrailers = null; |
There was a problem hiding this comment.
Nit: Instead of storing lastCloseStatus and lastCloseTrailers as two separate fields and calling createConsecutiveFailureResult(...) on every rejected RPC in PendingVRpc.start() (which runs String.format(...) and allocates a new Status + VRpcResult while holding poolLock), consider storing a single pre-built @Nullable private VRpcResult lastFailureResult = null; updated in onSessionClose when prevState == SessionState.STARTING. Both onSessionClose and PendingVRpc.start can then reuse lastFailureResult directly.
| consecutiveFailures = 0; | ||
| lastCloseStatus = null; | ||
| lastCloseTrailers = null; | ||
| consecutiveUnimplementedFailures = 0; |
There was a problem hiding this comment.
Consider extracting a @GuardedBy("poolLock") private void resetConsecutiveFailures() helper for resetting these fields and also calling it in configListener when poolSizer.isDisabled() is true.
Otherwise, if a pool hits consecutiveFailures >= threshold, is disabled via ClientConfiguration, and is later re-enabled, isFailingWithNoEstablishedSessions() will still be true while the newly re-enabled pool's initial sessions are in STARTING. Any RPC arriving before the first handshake completes will fast-fail in PendingVRpc.start() with the stale pre-disable error instead of queueing for the starting sessions.
| Field consecutiveFailuresField = SessionPoolImpl.class.getDeclaredField("consecutiveFailures"); | ||
| consecutiveFailuresField.setAccessible(true); | ||
| long deadline = System.currentTimeMillis() + 5000; | ||
| while (System.currentTimeMillis() < deadline) { | ||
| poolLock.lock(); | ||
| try { | ||
| if (sessionPool.sessions.getAllSessions().isEmpty() | ||
| && consecutiveFailuresField.getInt(sessionPool) >= 10) { | ||
| break; | ||
| } | ||
| } finally { | ||
| poolLock.unlock(); |
There was a problem hiding this comment.
If this 5s polling loop ever times out on a slow runner before all 50 budget slots finish closing (allSessions is not empty yet), the loop exits silently and rpc.start(...) below could pass by being cancelled via a racing onSessionClose() rather than via PendingVRpc.start().
Could we assert after the loop (or inside poolLock) that sessionPool.sessions.getAllSessions().isEmpty() and consecutiveFailuresField.getInt(sessionPool) >= 10 were actually reached?
| @Test | ||
| @SuppressWarnings("GuardedBy") | ||
| void consecutiveFailuresPreservesPendingRpcsWhilePoolHasInUseSessions() throws Exception { | ||
| sessionPool.start(OpenFakeSessionRequest.getDefaultInstance(), new Metadata()); | ||
| waitForOpenRequestCount(5, Duration.ofSeconds(5)); | ||
| assertPoolServesVRpc(sessionPool, "initial started pool"); | ||
|
|
||
| ReentrantLock poolLock = extractPoolLock(sessionPool); | ||
| Field consecutiveFailuresField = SessionPoolImpl.class.getDeclaredField("consecutiveFailures"); | ||
| consecutiveFailuresField.setAccessible(true); | ||
|
|
||
| UnaryResponseFuture<SessionFakeScriptedResponse> checkedOutFuture = new UnaryResponseFuture<>(); | ||
| UnaryResponseFuture<SessionFakeScriptedResponse> pendingFuture = new UnaryResponseFuture<>(); | ||
| VRpc<SessionFakeScriptedRequest, SessionFakeScriptedResponse> checkedOutRpc; | ||
| poolLock.lock(); | ||
| try { | ||
| // Mark 4 of the 5 ready sessions closing, then check out the remaining 1 ready session via | ||
| // newCall() so readyCount == 0 and inUseCount == 1. | ||
| List<SessionHandle> handles = new ArrayList<>(sessionPool.sessions.getAllSessions()); | ||
| for (int i = 1; i < handles.size(); i++) { | ||
| handles.get(i).onSessionClosing(); | ||
| } | ||
| checkedOutRpc = sessionPool.newCall(FakeDescriptor.SCRIPTED); | ||
| assertThat(sessionPool.sessions.getStats().getReadyCount()).isEqualTo(0); | ||
| assertThat(sessionPool.sessions.getStats().getInUseCount()).isEqualTo(1); | ||
|
|
||
| // Simulate consecutive STARTING failures reaching threshold while inUseCount == 1. | ||
| Field lastCloseStatusField = SessionPoolImpl.class.getDeclaredField("lastCloseStatus"); | ||
| lastCloseStatusField.setAccessible(true); | ||
| lastCloseStatusField.set(sessionPool, Status.INTERNAL.withDescription("fake internal error")); | ||
| consecutiveFailuresField.setInt(sessionPool, 10); | ||
|
|
||
| // A new PendingVRpc started while inUseCount > 0 must queue instead of failing fast. | ||
| VRpc<SessionFakeScriptedRequest, SessionFakeScriptedResponse> pendingRpc = | ||
| sessionPool.newCall(FakeDescriptor.SCRIPTED); | ||
| pendingRpc.start( | ||
| SessionFakeScriptedRequest.newBuilder().setTag(0).build(), | ||
| VRpcCallContext.create(Deadline.after(1, TimeUnit.MINUTES), true, vrpcTracer), | ||
| pendingFuture); |
There was a problem hiding this comment.
A couple of things in this test setup:
waitForOpenRequestCount(5, ...)only checks that the server received 5OpenSessionrequests, not that all 5 sessions have completedonSessionReady()on the client (andassertPoolServesVRpconly guarantees 1 ready session). BecausegetAllSessions()is aHashSetthat includesSTARTINGsessions, if one of the 5 sessions is stillSTARTINGwhenpoolLock.lock()runs and happens to land athandles.get(0), the loop markinghandles.get(1..4)as closing will mark allREADYsessions closing (readyCount == 0), causingassertThat(getInUseCount()).isEqualTo(1)to fail flakily. Polling untilsessionPool.sessions.getStats().getReadyCount() == 5underpoolLockavoids this.- Setting
consecutiveFailures = 10via reflection before callingpendingRpc.start(...)only exercises the!hasEstablishedSession()guard inPendingVRpc.start(), not the one inonSessionClose()whenprevState == SessionState.STARTING. If we also invokeonSessionClosefor aSTARTINGsession whilependingRpcis queued inpendingRpcsandinUseCount == 1(and assertpendingFuture.isDone()is stillfalse), this test will cover bothPendingVRpc.start()andonSessionClose().
… failures on disable
…veFailuresPreservesPendingRpcsWhilePoolHasInUseSessions
…elPoolDpImpl (#15039) Part 2 of 2 for b/584671437 (stacked on #15038). ### What changed - Only mark a session stream established in `ChannelPoolDpImpl` when `OpenSessionResponse` arrives (`onMessage`), rather than when initial `PeerInfo` headers arrive (`onBeforeSessionStart`). - Count non-OK stream closes that occur before `OpenSessionResponse` as consecutive failures and evict the failed `routeObservations` entry so subsequent `OpenSession` attempts do not remain pinned to a failing AFE/subzone.
Part 1 of 2 for b/584671437.
What changed
SessionPoolImpl, failpendingRpcsonconsecutiveFailures >= consecutive_session_failure_thresholdif and only if the pool has no established (READYorIN_USE) sessions (readyCount + inUseCount == 0).onSessionClose()(draining queuedpendingRpcswhen transitioning to 0 established sessions whileconsecutiveFailures >= threshold, while preservingpendingRpcswheninUseCount > 0) andPendingVRpc.start()(rejecting new calls during the 60sSessionCreationBudgetlockout when no sessions are closing).