Repository navigation
Add liveness check for cached AMQP links in ReactorSession - #50505
Sunghyun Jung (sunghyun1999) wants to merge 7 commits into
Conversation
When an AMQP sender or receiver link is silently detached (e.g. by LB/NAT idle timeout on Kubernetes), the dead link remains cached in ReactorSession.openSendLinks/openReceiveLinks. Subsequent calls to createProducer/createConsumer return the stale link, causing all send attempts to hang until the retry budget (245.6s with defaults) is exhausted. The instance becomes permanently stuck until JVM restart. This change adds an isDisposed() check at two points for both send and receive links: 1. Early cache lookup — before returning a cached link, verify it is still alive. If disposed, remove it from the cache and fall through to create a fresh link. 2. ConcurrentHashMap.compute() lambda — same check to handle the race between the early lookup and the compute call. Fixes Azure#50503 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Thank you for your contribution Sunghyun Jung (@sunghyun1999)! We will review the pull request and get back to you soon. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 31 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
Critical receive-link race and cleanup issues remain, and regression-test setup and coverage need correction.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds liveness checks to recreate disposed cached AMQP send and receive links.
Changes:
- Detects disposed links during cache lookup and computation.
- Removes stale links and creates replacements.
- Adds a producer recreation regression test.
File summaries
| File | Summary |
|---|---|
sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSession.java |
Adds send/receive link liveness and replacement logic; receive-path race and cleanup issues remain. |
sdk/core/azure-core-amqp/src/test/java/com/azure/core/amqp/implementation/ReactorSessionTest.java |
Adds producer recreation coverage; test setup requires lifecycle fixes and receiver coverage. |
Review details
Suppressed comments (3)
sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSession.java:536
- This is a TOCTOU race: after the lookup at line 525 observes the disposed
existing, another caller can install a healthy replacement before this key-only removal runs.removeLinkwill then close that healthy link, and this call will create yet another link. Remove conditionally using the observedexistingsubscription and close it only when that conditional removal succeeds.
removeLink(openSendLinks, linkName);
sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSession.java:565
- When the compute lambda replaces a disposed cached link, the old
LinkSubscriptionis not closed or unsubscribed. Its endpoint-state callback can run after the replacement is installed and callremoveLink(openSendLinks, linkName), removing and closing the new link; it also leaves the old subscription attached. Close the disposed subscription before replacing it so its callback cannot affect the new cache entry.
final AmqpSendLink link = existingLink.getLink();
if (link != null && link.isDisposed()) {
logger.atInfo()
.addKeyValue(LINK_NAME_KEY, linkName)
.log("Cached send link is disposed. Creating a new send link.");
sdk/core/azure-core-amqp/src/test/java/com/azure/core/amqp/implementation/ReactorSessionTest.java:340
- The provider normally creates a new one-shot
SendLinkHandlerfor each link, but this stub reusessendLinkHandlerfor both senders. Disposing the first link terminates that handler, so the recreatedReactorSendercan inherit a completed endpoint stream and be immediately disposed; the test may then pass while the replacement is not live. Return a distinct handler for the second link and assert it is not disposed.
when(reactorHandlerProvider.createSendLinkHandler(ID, HOST, linkName, entityPath)).thenReturn(sendLinkHandler);
- Files reviewed: 2/2 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| .addKeyValue(LINK_NAME_KEY, linkName) | ||
| .addKeyValue(ENTITY_PATH_KEY, entityPath) | ||
| .log("Cached receive link is disposed. Removing to allow recreation."); | ||
| removeLink(openReceiveLinks, linkName); |
| final AmqpReceiveLink link = existing.getLink(); | ||
| if (link != null && link.isDisposed()) { | ||
| logger.atInfo() | ||
| .addKeyValue(LINK_NAME_KEY, linkName) | ||
| .log("Cached receive link is disposed. Creating a new receiver link."); |
| assertNotNull(firstLink); | ||
|
|
||
| // Simulate the link being silently detached — dispose it | ||
| firstLink.dispose(); |
| final AmqpReceiveLink link = existing.getLink(); | ||
| if (link != null && link.isDisposed()) { | ||
| logger.atInfo() | ||
| .addKeyValue(LINK_NAME_KEY, linkName) | ||
| .log("Cached receive link is disposed. Creating a new receiver link."); |
1. TOCTOU race in early cache lookup: Instead of calling removeLink()
(which could remove a healthy replacement installed by another
thread between lookup and removal), simply fall through to the
compute() lambda which handles replacement atomically.
2. Old subscription cleanup in compute(): When replacing a disposed
link, call existingLink.closeAsync(null).subscribe() to close
the old subscription before creating the replacement. This prevents
the old subscription's error/complete callback from later calling
removeLink() and accidentally removing the new link.
3. Test fixes:
- Use distinct SendLinkHandler per link creation so disposing
the first handler doesn't terminate the second link's endpoint
state stream.
- Use sendLinkHandler1.close() instead of firstLink.dispose()
to avoid blocking on the tryTimeout.
- Assert secondLink is not disposed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sendLinkHandler.close() completes the endpoint states flux, but ReactorSender's doOnComplete callback (which sets isDisposed=true) runs asynchronously. Use StepVerifier to wait for the endpoint states to complete before asserting isDisposed. Also shorten log messages in compute() lambdas that exceeded the 120-char line limit enforced by checkstyle. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Run spotless:apply to fix Eclipse formatter violations: - ReactorSession.java: merge logger chain into single line - ReactorSessionTest.java: merge when().thenReturn() chain Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…cting completion The StepVerifier was using thenAwait() which does not consume onNext signals. When the handler closes, it emits CLOSED as onNext before onComplete, causing the expectComplete() assertion to fail. Using thenConsumeWhile(state -> true) properly drains all remaining states before asserting completion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
StepVerifier on cached endpoint states Flux is unreliable: - thenAwait doesn't consume onNext signals (CLOSED state) - thenConsumeWhile on .cache(1) Flux may never see onComplete Instead, poll firstLink.isDisposed() with Mono.repeatWhenEmpty, which directly tests what we care about and is timing-safe. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Handler.close() triggers async doOnComplete callback chain that depends on reactor scheduling, making isDisposed() unreliable across platforms (fails on macOS/Windows). Instead, call closeAsync().subscribe() which immediately sets isDisposed = true via AtomicBoolean in the first line of closeAsync(), independent of any async dispatch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Hi team, gentle ping on this PR — it's been open for about 6 weeks now with all CI checks passing. Would appreciate a review when you get a chance. Thanks! |
Summary
isDisposed()liveness check before returning cached send/receive links fromReactorSession.openSendLinks/openReceiveLinksConcurrentHashMap.compute()lambda to handle race conditionsProblem
When an AMQP sender link is silently detached (e.g. by infrastructure idle timeout on Kubernetes), the dead link remains in
ReactorSession.openSendLinkscache.createProducer()returns this stale link on every subsequent call without checking if it's still alive. Combined withRetryUtil.withRetry()re-subscribing to the same Mono chain (which reacquires the same dead link), this causes:sendMessage()call hangs fortryTimeout(60s default)Production evidence: 10/10 failures over 25 hours on one queue, while another queue on the same AMQP connection succeeded at 12–33ms. A fresh pod sent to the same queue in 87.8ms. See #50503 for full details.
Fix
The fix is minimal and surgical — a single
isDisposed()check at the cache lookup boundary:ReactorSession.java—createProducer()(send link cache):Same pattern applied to
createConsumer()(receive link cache) and bothcompute()lambdas.Why this is safe
isDisposed()is a simpleAtomicBoolean.get()— no side effects, no blockingremoveLink()cleans up the cache entry (already existing method)compute()lambda check handles the race between the early lookup and the CASTest plan
createProducerRecreatesDisposedLink— verifies that after disposing a cached link,createProducer()returns a new instance instead of the stale onecreateProducerandcreateProducerAgainAfterExceptiontests continue to pass (CI)Related
🤖 Generated with Claude Code