Skip to content

[Service Bus] Add batch delete and purge APIs - #48845

Open
Eldert Grootenboer (EldertGrootenboer) wants to merge 28 commits into
mainfrom
fix/servicebus-batch-delete
Open

Eldert Grootenboer (EldertGrootenboer) wants to merge 28 commits into
mainfrom
fix/servicebus-batch-delete

Conversation

@EldertGrootenboer

Copy link
Copy Markdown
Member

Description

Azure Service Bus exposes the com.microsoft:batch-delete-messages management operation. This PR adds synchronous and asynchronous receiver-level batch-delete and purge APIs to azure-servicebus for regular, session, and subqueue receivers.

delete_messages(message_count, *, before_enqueued_time=None, timeout=None)
purge_messages(*, before_enqueued_time=None, max_message_count_per_batch=500, timeout=None)

Both methods return result objects containing deleted_message_count. Purge captures one enqueue-time cutoff and one operation-wide timeout, defaults to 500 messages per request, continues after every positive result, and stops only when the service returns zero. Callers can select another positive signed 32-bit request size; Service Bus enforces the current tier limit of 500 for Basic and Standard and 4,000 for Premium.

Management-link setup is retryable, but a destructive request is dispatched only once. Missing, malformed, negative, or excessive response counts fail instead of being interpreted as successful exhaustion. Samples cover default purge, Premium sizing, sessions, enqueue-time thresholds, returned counts, and partial or unknown outcomes.

Companion implementations: JavaScript, .NET, Java, and Go.

Testing: the full package suite completed with 360 passed and 271 skipped. The final worktree-bound batch-delete suite passes 44 tests. api.md and api.metadata.yml were regenerated from the local package.

All SDK Contribution checklist

  • Public API additions are recorded in the generated API artifacts.
  • CHANGELOG is updated.
  • I have read the contribution guidelines.
  • Pull request includes sync and async test coverage.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Management-link setup remains outside the promised retry and timeout guarantees, and Premium-only samples run unconditionally against Standard resources.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds receiver-level batch delete and purge support to Azure Service Bus, including synchronous and asynchronous APIs.

Changes:

  • Adds delete/purge APIs and result models.
  • Implements AMQP request handling, validation, cutoffs, and timeout budgets.
  • Adds samples, tests, changelog, and API artifacts.
File summaries
File Description
tests/unittests/test_batch_delete.py Tests protocol, validation, sessions, timeouts, and purge loops.
samples/sync_samples/sample_code_servicebus.py Adds synchronous usage examples.
samples/async_samples/sample_code_servicebus_async.py Adds asynchronous usage examples.
CHANGELOG.md Documents the new APIs.
azure/servicebus/aio/_servicebus_receiver_async.py Implements asynchronous delete and purge.
azure/servicebus/aio/_base_handler_async.py Adds timeout-aware asynchronous opening.
azure/servicebus/_servicebus_receiver.py Implements synchronous delete and purge.
azure/servicebus/_models.py Defines result objects.
azure/servicebus/_common/utils.py Adds datetime-to-milliseconds conversion.
azure/servicebus/_common/mgmt_handlers.py Parses and validates delete responses.
azure/servicebus/_common/constants.py Adds management operation constants.
azure/servicebus/_base_handler.py Adds timeout-aware synchronous opening.
azure/servicebus/__init__.py Exports result types.
api.metadata.yml Updates the API snapshot hash.
api.md Records the new public API surface.
Review details
  • Files reviewed: 15/15 changed files
  • Comments generated: 6
  • Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Management-link setup cannot reliably recover from attach failures, and uAMQP bypasses the promised setup retry.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

sdk/servicebus/azure-servicebus/azure/servicebus/_pyamqp/aio/_client_async.py:379

  • The async vendored pyamqp change is not mirrored in Event Hubs (sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/aio/_client_async.py:345-392). The two vendored stacks are kept byte-identical; this introduces transport behavior drift and complicates future shared fixes. Apply the same async management-link refactor to the Event Hubs copy.
    sdk/servicebus/azure-servicebus/azure/servicebus/_pyamqp/client.py:466
  • This vendored pyamqp client change is not mirrored in the Event Hubs copy (sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/client.py:432-478). These stacks are maintained as byte-identical copies; leaving only Service Bus updated creates divergent timeout, cleanup, and management-link behavior that makes future shared fixes unsafe. Apply the equivalent change to the Event Hubs vendored client.

sdk/servicebus/azure-servicebus/azure/servicebus/_pyamqp/aio/_client_async.py:406

  • asyncio.CancelledError is not an Exception, so cancellation during either readiness loop skips this cleanup and leaves the management link cached in an indeterminate state. Catch BaseException for this cleanup boundary and immediately re-raise, as the method already does.
        except Exception:
  • Files reviewed: 23/23 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Timeout handling, response validation, and Event Hubs parity issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

sdk/servicebus/azure-servicebus/azure/servicebus/_base_handler.py:617

  • When pyAMQP management-link setup exhausts its budget, open_mgmt_link() raises the transport's built-in TimeoutError. Unlike _mgmt_request_response() (lines 544-546), this setup path does not translate that error before _do_retryable_operation() calls _handle_exception(), so callers receive a generic ServiceBusError instead of OperationTimeoutError. Translate the transport timeout here to preserve the public timeout contract.
    sdk/servicebus/azure-servicebus/azure/servicebus/_common/mgmt_handlers.py:87
  • Returning zero for a 204 treats a response with no message-count as successful exhaustion. This contradicts the PR contract that missing counts fail and purge stops only when the service explicitly returns zero; a bodyless response can therefore terminate purge while eligible messages remain. Reject this response unless it contains a validated count, and update the test that currently asserts 204 == 0.
    sdk/servicebus/azure-servicebus/azure/servicebus/aio/_base_handler_async.py:464
  • When async pyAMQP management-link setup exhausts its budget, open_mgmt_link_async() raises the transport's built-in TimeoutError. Unlike _mgmt_request_response() (lines 391-393), this setup path does not translate that error before _do_retryable_operation() calls _handle_exception(), so callers receive a generic ServiceBusError instead of OperationTimeoutError. Translate the transport timeout here to preserve the public timeout contract.
  • Files reviewed: 23/23 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Response validation and sync/async timeout-budgeting issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

sdk/servicebus/azure-servicebus/azure/servicebus/_common/utils.py:83

  • Add a second blank line between these top-level functions. The same module consistently leaves two blank lines between definitions (for example, lines 97–100), and Black will otherwise reformat this file.
    return (normalized - EPOCH_UTC) // datetime.timedelta(milliseconds=1)
  • Files reviewed: 23/23 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Async timeout handling can leak a coroutine, and the async session sample may wait indefinitely.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

sdk/servicebus/azure-servicebus/samples/async_samples/sample_code_servicebus_async.py:482

  • This default invocation targets SERVICEBUS_SESSION_ID, but the preceding async session example creates only the literal "<your session id>" (lines 333-340), while the test deployment supplies a different generated ID. On a clean namespace, entering this receiver waits indefinitely for a session this script never created. Keep this session-specific example opt-in or seed the configured session before opening it.

sdk/servicebus/azure-servicebus/azure/servicebus/_common/utils.py:83

  • Add a second blank line after this new top-level helper. The surrounding module uses two blank lines between top-level functions (for example, lines 97–100), and Black will reformat this section otherwise.
    return (normalized - EPOCH_UTC) // datetime.timedelta(milliseconds=1)

sdk/servicebus/azure-servicebus/azure/servicebus/_servicebus_receiver.py:860

  • The PR promises support for subqueue receivers, but the new suite exercises only regular and session receivers; there is no sub_queue case. Because dead-letter/transfer-dead-letter receivers use a different entity management target, add sync and async coverage that creates those receiver variants and verifies the batch-delete request is sent to the subqueue target.
    def delete_messages(
        self,
        message_count: int,
        *,
        before_enqueued_time: Optional[datetime.datetime] = None,
  • Files reviewed: 23/23 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Setup timeout conversion, async session sample seeding, and formatting issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

sdk/servicebus/azure-servicebus/azure/servicebus/_common/utils.py:85

  • Only one blank line separates this new top-level helper from utc_from_timestamp, while this file uses two between top-level functions (for example, lines 97–100). This violates the Black/PEP 8 layout and will fail the formatting check; add the second blank line.
    sdk/servicebus/azure-servicebus/azure/servicebus/_transport/_pyamqp_transport.py:1093
  • open_mgmt_link() raises the transport's raw TimeoutError, but this setup call runs directly inside _do_retryable_operation, not _mgmt_request_response (which normally converts transport timeouts). _handle_exception consequently wraps it as a non-retryable generic ServiceBusError, closes the handler, and aborts setup instead of surfacing OperationTimeoutError. Convert the transport timeout here before it reaches the retry helper.
    sdk/servicebus/azure-servicebus/samples/async_samples/sample_code_servicebus_async.py:450
  • This executable sample opens SERVICEBUS_SESSION_ID, but the preceding example_session_ops_async() seeds only the distinct literal "<your session id>". On a clean sample namespace there is therefore no active message/session for this ID, so the exact-session receiver can wait instead of demonstrating purge. Seed this session inside the example (as the sync flow effectively does) so the snippet also works independently.
  • Files reviewed: 23/23 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It combines destructive public APIs with low-level sync/async AMQP timeout, retry, and cleanup changes across two transports.

0 open findings

🧠 Review effort: Balanced

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

- Translate dispatch failures and invalidate receiver/session state once.
- Apply configured try_timeout without retrying destructive requests.
- Cover both transports and sync/async session cleanup and timeout bounds.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9519ad81-9610-4092-b343-887e9cabdda0
@EldertGrootenboer

Copy link
Copy Markdown
Member Author

Resolved the conflict requests in #48845 (comment) and #48845 (comment). Their merges eb05692 and 26df6fb are both ancestors of the current head. The later approved merge 1420b7c retains the 7.16.0b1 preview and main's separate 7.15.1 changelog section. There are no unmerged index entries, and GitHub reports the current head bf18167 as MERGEABLE. This closes those conflict requests, not the remaining CI, review, or human approval gates.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

- Describe optional fractional setup timeouts and string/bytes targets consistently.
- Reuse inherited receiver timeout forwarding with regression coverage.
- Regenerate API docs without duplicate result attributes; retain read-only properties.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9519ad81-9610-4092-b343-887e9cabdda0

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- Default the cutoff only when omitted, never for falsy invalid inputs.
- Validate datetime cutoffs before opening either receiver transport.
- Cover sync and async delete/purge rejection without network dispatch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9519ad81-9610-4092-b343-887e9cabdda0
- Give creator-owned error cleanup a separate one-second grace after setup failure.
- Preserve original errors, cancellation, and identity-safe cache eviction.
- Document the approved bound and prove immediate and stalled cleanup behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9519ad81-9610-4092-b343-887e9cabdda0
@EldertGrootenboer

Copy link
Copy Markdown
Member Author

Addressed the two risks called out in #48845 (review). In c2d6262, only an omitted cutoff defaults to now; invalid values fail before opening or dispatching. In fb565e5, failed async management-link setup gets the approved separate one-second cleanup grace, including after its deadline expires. Creator ownership, identity-safe cache eviction, and the original exception/cancellation are preserved. Regressions cover immediate cleanup after expiry and cancellation of stalled cleanup; the focused suite passes 154 tests, with 4 live tests skipped. Current-head CI and review gates remain pending.

@github-actions

This comment has been minimized.

- Select the first release guarding removed typing exports on Python 3.15+.
- Preserve the existing CI pin on older interpreters and SDK dependencies.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9519ad81-9610-4092-b343-887e9cabdda0
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor
[Pilot] PR Pipeline Failure Analysis

What failed

The Azure DevOps pipeline python - pullrequest (build 7054800), job Build Test Ubuntu2404_315 (Python 3.15), failed during environment/import setup — before any Service Bus test code executed. No test artifacts were produced (failed_pipeline_tests is empty), so this is a pure environment/infrastructure failure, not a test regression.

Relevant pipeline output
  File "/mnt/vss/_work/1/s/venv/lib/python3.15/site-packages/azure/core/pipeline/__init__.py", line 191, in <module>
    from ._base import Pipeline  # pylint: disable=wrong-import-position
  File "/mnt/vss/_work/1/s/venv/lib/python3.15/site-packages/azure/core/pipeline/_base.py", line 44, in <module>
    from azure.core.pipeline.policies import HTTPPolicy, SansIOHTTPPolicy
  File "/mnt/vss/_work/1/s/venv/lib/python3.15/site-packages/azure/core/pipeline/policies/__init__.py", line 28, in <module>
    from ._authentication import (...)
  File "/mnt/vss/_work/1/s/venv/lib/python3.15/site-packages/azure/core/pipeline/policies/_authentication.py", line 11, in <module>
    from azure.core.credentials import (...)
  File "/mnt/vss/_work/1/s/venv/lib/python3.15/site-packages/azure/core/credentials.py", line 7, in <module>
    from typing_extensions import Protocol, runtime_checkable
  File "/mnt/vss/_work/1/s/venv/lib/python3.15/site-packages/typing_extensions.py", line 3641, in <module>
    no_type_check_decorator = typing.no_type_check_decorator
AttributeError: module 'typing' has no attribute 'no_type_check_decorator'
##[error]PowerShell exited with code '1'.

Diagnosis

This is an infrastructure/environment failure, not caused by this PR's code changes. The pipeline's Python 3.15 (pre-release) job installed a typing_extensions version whose import path references typing.no_type_check_decorator, an attribute that CPython 3.15 removed from the typing module. This breaks azure.core's import chain (azure.core.credentials → typing_extensions) before azure-servicebus or its tests ever load, so every test in that job fails identically regardless of what the PR changed. This is a version-compatibility break between a pinned/installed typing_extensions release and the Python 3.15 interpreter used by the CI image, not something this PR's diff can fix.

Recommended next steps

  • Retry the pipeline — if the CI image's typing_extensions/Python 3.15 combination was updated between runs, a retry may pick up a compatible version.
  • Report this as a CI/tooling issue to the Azure SDK engineering system (eng/) team: the Python 3.15 Ubuntu2404 test image needs a typing_extensions version compatible with CPython 3.15 (or a pin/constraint update in the engineering system's test requirements), since this fails before azure-servicebus code is exercised at all.
  • No change to sdk/servicebus/azure-servicebus source, tests, or samples can resolve this failure.
  • See https://aka.ms/ci-fix

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a
fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis Next Steps · copilot · sonnet50 · 34.6 AIC · ⌖ 0.571 AIC · ⊞ 9.1K · ◷

- Retain main's pip 26.2.2 constraint to resolve the adjacent-line conflict.
- Keep interpreter-scoped typing extensions without an upstream-only merge.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9519ad81-9610-4092-b343-887e9cabdda0
- Preserve the conditional typing-extensions repair
- Retain incoming CI bootstrap updates

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9519ad81-9610-4092-b343-887e9cabdda0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants