Skip to content

Add liveness check for cached AMQP links in ReactorSession - #50505

Open
Sunghyun Jung (sunghyun1999) wants to merge 7 commits into
Azure:mainfrom
sunghyun1999:fix/sender-link-liveness-check
Open

Sunghyun Jung (sunghyun1999) wants to merge 7 commits into
Azure:mainfrom
sunghyun1999:fix/sender-link-liveness-check

Conversation

@sunghyun1999

Copy link
Copy Markdown

Summary

  • Add isDisposed() liveness check before returning cached send/receive links from ReactorSession.openSendLinks / openReceiveLinks
  • If a cached link is disposed (e.g. after silent AMQP detach by LB/NAT idle timeout), remove it from the cache and create a fresh link instead of returning the stale one
  • Apply the same check in the ConcurrentHashMap.compute() lambda to handle race conditions

Problem

When an AMQP sender link is silently detached (e.g. by infrastructure idle timeout on Kubernetes), the dead link remains in ReactorSession.openSendLinks cache. createProducer() returns this stale link on every subsequent call without checking if it's still alive. Combined with RetryUtil.withRetry() re-subscribing to the same Mono chain (which reacquires the same dead link), this causes:

  1. Every sendMessage() call hangs for tryTimeout (60s default)
  2. All retries fail identically → 245.6s total timeout
  3. The instance is permanently stuck until JVM restart
  4. Other queues on the same connection continue to work (cache is keyed by entity path)

Production evidence: 10/10 failures over 25 hours on one queue, while another queue on the same AMQP connection succeeded at 12–33ms. A fresh pod sent to the same queue in 87.8ms. See #50503 for full details.

Fix

The fix is minimal and surgical — a single isDisposed() check at the cache lookup boundary:

ReactorSession.java — createProducer() (send link cache):

final AmqpSendLink existingLink = existing.getLink();
if (existingLink != null && existingLink.isDisposed()) {
    logger.atInfo().addKeyValue(LINK_NAME_KEY, linkName)
        .log("Cached send link is disposed. Removing to allow recreation.");
    removeLink(openSendLinks, linkName);
} else {
    return Mono.just(existingLink);  // existing behavior
}

Same pattern applied to createConsumer() (receive link cache) and both compute() lambdas.

Why this is safe

  • isDisposed() is a simple AtomicBoolean.get() — no side effects, no blocking
  • When disposed, removeLink() cleans up the cache entry (already existing method)
  • Falls through to the existing link creation path — no new code paths introduced
  • A non-disposed link behaves exactly as before — zero behavioral change for healthy links
  • The compute() lambda check handles the race between the early lookup and the CAS

Test plan

  • New unit test: createProducerRecreatesDisposedLink — verifies that after disposing a cached link, createProducer() returns a new instance instead of the stale one
  • Existing createProducer and createProducerAgainAfterException tests continue to pass (CI)
  • Manual verification: deploy on AKS with Service Bus, verify link recovery after idle timeout

Related

🤖 Generated with Claude Code

When an AMQP sender or receiver link is silently detached (e.g. by
LB/NAT idle timeout on Kubernetes), the dead link remains cached in
ReactorSession.openSendLinks/openReceiveLinks. Subsequent calls to
createProducer/createConsumer return the stale link, causing all send
attempts to hang until the retry budget (245.6s with defaults) is
exhausted. The instance becomes permanently stuck until JVM restart.

This change adds an isDisposed() check at two points for both send
and receive links:

1. Early cache lookup — before returning a cached link, verify it is
   still alive. If disposed, remove it from the cache and fall through
   to create a fresh link.

2. ConcurrentHashMap.compute() lambda — same check to handle the race
   between the early lookup and the compute call.

Fixes Azure#50503

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution Sunghyun Jung (@sunghyun1999)! We will review the pull request and get back to you soon.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
31 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical receive-link race and cleanup issues remain, and regression-test setup and coverage need correction.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds liveness checks to recreate disposed cached AMQP send and receive links.

Changes:

  • Detects disposed links during cache lookup and computation.
  • Removes stale links and creates replacements.
  • Adds a producer recreation regression test.
File summaries
File Summary
sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSession.java Adds send/receive link liveness and replacement logic; receive-path race and cleanup issues remain.
sdk/core/azure-core-amqp/src/test/java/com/azure/core/amqp/implementation/ReactorSessionTest.java Adds producer recreation coverage; test setup requires lifecycle fixes and receiver coverage.
Review details

Suppressed comments (3)

sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSession.java:536

  • This is a TOCTOU race: after the lookup at line 525 observes the disposed existing, another caller can install a healthy replacement before this key-only removal runs. removeLink will then close that healthy link, and this call will create yet another link. Remove conditionally using the observed existing subscription and close it only when that conditional removal succeeds.
                removeLink(openSendLinks, linkName);

sdk/core/azure-core-amqp/src/main/java/com/azure/core/amqp/implementation/ReactorSession.java:565

  • When the compute lambda replaces a disposed cached link, the old LinkSubscription is not closed or unsubscribed. Its endpoint-state callback can run after the replacement is installed and call removeLink(openSendLinks, linkName), removing and closing the new link; it also leaves the old subscription attached. Close the disposed subscription before replacing it so its callback cannot affect the new cache entry.
                                final AmqpSendLink link = existingLink.getLink();
                                if (link != null && link.isDisposed()) {
                                    logger.atInfo()
                                        .addKeyValue(LINK_NAME_KEY, linkName)
                                        .log("Cached send link is disposed. Creating a new send link.");

sdk/core/azure-core-amqp/src/test/java/com/azure/core/amqp/implementation/ReactorSessionTest.java:340

  • The provider normally creates a new one-shot SendLinkHandler for each link, but this stub reuses sendLinkHandler for both senders. Disposing the first link terminates that handler, so the recreated ReactorSender can inherit a completed endpoint stream and be immediately disposed; the test may then pass while the replacement is not live. Return a distinct handler for the second link and assert it is not disposed.
        when(reactorHandlerProvider.createSendLinkHandler(ID, HOST, linkName, entityPath)).thenReturn(sendLinkHandler);
  • Files reviewed: 2/2 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

.addKeyValue(LINK_NAME_KEY, linkName)
.addKeyValue(ENTITY_PATH_KEY, entityPath)
.log("Cached receive link is disposed. Removing to allow recreation.");
removeLink(openReceiveLinks, linkName);
Comment on lines +445 to +449
final AmqpReceiveLink link = existing.getLink();
if (link != null && link.isDisposed()) {
logger.atInfo()
.addKeyValue(LINK_NAME_KEY, linkName)
.log("Cached receive link is disposed. Creating a new receiver link.");
assertNotNull(firstLink);

// Simulate the link being silently detached — dispose it
firstLink.dispose();
Comment on lines +445 to +449
final AmqpReceiveLink link = existing.getLink();
if (link != null && link.isDisposed()) {
logger.atInfo()
.addKeyValue(LINK_NAME_KEY, linkName)
.log("Cached receive link is disposed. Creating a new receiver link.");
1. TOCTOU race in early cache lookup: Instead of calling removeLink()
   (which could remove a healthy replacement installed by another
   thread between lookup and removal), simply fall through to the
   compute() lambda which handles replacement atomically.

2. Old subscription cleanup in compute(): When replacing a disposed
   link, call existingLink.closeAsync(null).subscribe() to close
   the old subscription before creating the replacement. This prevents
   the old subscription's error/complete callback from later calling
   removeLink() and accidentally removing the new link.

3. Test fixes:
   - Use distinct SendLinkHandler per link creation so disposing
     the first handler doesn't terminate the second link's endpoint
     state stream.
   - Use sendLinkHandler1.close() instead of firstLink.dispose()
     to avoid blocking on the tryTimeout.
   - Assert secondLink is not disposed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sendLinkHandler.close() completes the endpoint states flux, but
ReactorSender's doOnComplete callback (which sets isDisposed=true)
runs asynchronously. Use StepVerifier to wait for the endpoint states
to complete before asserting isDisposed.

Also shorten log messages in compute() lambdas that exceeded the
120-char line limit enforced by checkstyle.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Run spotless:apply to fix Eclipse formatter violations:
- ReactorSession.java: merge logger chain into single line
- ReactorSessionTest.java: merge when().thenReturn() chain

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…cting completion

The StepVerifier was using thenAwait() which does not consume onNext signals.
When the handler closes, it emits CLOSED as onNext before onComplete, causing
the expectComplete() assertion to fail. Using thenConsumeWhile(state -> true)
properly drains all remaining states before asserting completion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
StepVerifier on cached endpoint states Flux is unreliable:
- thenAwait doesn't consume onNext signals (CLOSED state)
- thenConsumeWhile on .cache(1) Flux may never see onComplete

Instead, poll firstLink.isDisposed() with Mono.repeatWhenEmpty,
which directly tests what we care about and is timing-safe.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Handler.close() triggers async doOnComplete callback chain that
depends on reactor scheduling, making isDisposed() unreliable
across platforms (fails on macOS/Windows).

Instead, call closeAsync().subscribe() which immediately sets
isDisposed = true via AtomicBoolean in the first line of
closeAsync(), independent of any async dispatch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@sunghyun1999

Copy link
Copy Markdown
Author

Hi team, gentle ping on this PR — it's been open for about 6 weeks now with all CI checks passing. Would appreciate a review when you get a chance. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Azure.Core.AMQP azure-core-amqp Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ServiceBusSenderClient permanently stuck when a single AMQP sender link is detached — dead link cached in ReactorSession.openSendLinks

2 participants