[wangbill (YunchuWang)]

Summary

What changed?

• Add opt-in GrpcCallTimeout to gRPC and scheduler client options. It sets a deadline on each unary RPC, not the whole operation, preserves earlier interceptor deadlines, and excludes WaitForInstanceStart, WaitForInstanceCompletion, and streams.
• Treat Cancelled as neutral in channel recovery only when the caller's token is cancelled; otherwise retain application-response reset semantics. Keep named scheduler configuration, shared-channel recovery, and payload interceptors intact.
• Document configuration and the ambiguous outcome of mutations whose deadlines expire.
• Ensure failed or cancelled loopback-host startup is awaited and disposed, and assert the actual 500 ms RPC budget instead of only logging latency.

Why is this change needed?

• Management calls otherwise have no SDK-supplied deadline. A stalled call can remain pending, while local caller cancellation previously cleared the recovery failure count. Opt-in deadlines surface DeadlineExceeded to the existing recovery mechanism without imposing a new default timeout.

Issues / work items

• Resolves Support a per-call deadline on DurableTaskClient calls #824
• Related: none.

---

Project checklist

• Release notes are not required for the next release
  • Otherwise: Notes added to release_notes.md
• Backport is not required
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• All required tests have been added/updated (unit tests, E2E tests)
• Breaking change?
  • If yes:
    • Impact: N/A; additive opt-in options, with no wire or orchestration replay changes.
    • Migration guidance: N/A; the default remains null.

---

AI-assisted code disclosure (required)

Was an AI tool used? (select one)

• No
• Yes, AI helped write parts of this PR (e.g., GitHub Copilot)
• Yes, an AI agent generated most of this PR

If AI was used:

• Tool(s): GitHub Copilot.
• AI-assisted areas/files: Client deadline configuration/interception, recovery cancellation handling, related tests, and client README.
• What you changed after AI output: Applied the approved review fixes for caller-cancellation provenance, startup cleanup, a real RPC-budget assertion, the GrpcCallTimeout name, and a neutral shared long-poll classifier. Human verification checkboxes below remain for the reviewer.

AI verification (required if AI was used):

• I understand the code and can explain it
• I verified referenced APIs/types exist and are correct
• I reviewed edge cases/failure paths (timeouts, retries, cancellation, exceptions)
• I reviewed concurrency/async behavior
• I checked for unintended breaking or behavior changes

---

Testing

Automated tests

• Result: Passed locally for the revised source. Before the revision, server-returned Cancelled reset controls failed on both unary paths; genuine caller-token cancellation controls passed. The revised tests distinguish both cases and retain real HTTP/2 recovery coverage.

Command	Result
dotnet test test\Client\Grpc.Tests\Client.Grpc.Tests.csproj --configuration Release --no-restore --verbosity quiet	Passed, 78 tests
dotnet test test\Client\AzureManaged.Tests\Client.AzureManaged.Tests.csproj --configuration Release --no-restore --verbosity quiet	Passed, 182 tests
dotnet test test\Grpc.IntegrationTests\Grpc.IntegrationTests.csproj --configuration Release --no-restore --filter FullyQualifiedName~ClientDeadlineTests --verbosity quiet	Passed, 3 tests: real loopback recovery and startup failure/cancellation cleanup
dotnet build src\Client\Grpc\Client.Grpc.csproj --configuration Release --no-restore --verbosity quiet	Passed, netstandard2.0/net6.0/net8.0/net10.0
dotnet build src\Client\AzureManaged\Client.AzureManaged.csproj --configuration Release --no-restore --verbosity quiet	Passed, net6.0/net8.0/net10.0

Changed-test formatting checks passed:

dotnet format whitespace test\Client\Grpc.Tests\Client.Grpc.Tests.csproj --no-restore --verify-no-changes --include test\Client\Grpc.Tests\GrpcDurableTaskClientChannelRecreationTests.cs test\Client\Grpc.Tests\GrpcDurableTaskClientInterceptorsTests.cs --verbosity quiet
dotnet format whitespace test\Client\AzureManaged.Tests\Client.AzureManaged.Tests.csproj --no-restore --verify-no-changes --include test\Client\AzureManaged.Tests\DurableTaskSchedulerClientExtensionsTests.cs --verbosity quiet
dotnet format whitespace test\Grpc.IntegrationTests\Grpc.IntegrationTests.csproj --no-restore --verify-no-changes --include test\Grpc.IntegrationTests\ClientDeadlineTests.cs --verbosity quiet

git -c core.whitespace=cr-at-eol diff --check origin/main...HEAD passed; the base version of GrpcDurableTaskClient.cs stores CRLF line endings. Builds retain existing nullability, analyzer, and obsolete-API warnings; no diagnostics were reported on the timeout interceptor. Full-solution, emulator, and cloud checks were not run locally; local results do not imply hosted CI passed.

The previous-head hosted build failed when PurgeActivityDeadlineTests.ReportLargePayloadPurgeResults_WhenDeadlineExpires_CancelsRequestAsync timed out after 5 seconds. That test was not changed or rerun as part of this focused revision; no cause or flaky/pre-existing classification is claimed.

Manual validation (only if runtime/behavior changed)

• Environment (OS, .NET version, components): Windows x64, SDK 10.0.403/runtime 10.0.14, ephemeral Kestrel HTTP/2 loopback server; no external service traffic.
• Steps + observed results: Automated runtime validation, not a human manual run.
  1. Accept reads on one HTTP/2 connection without sending a gRPC response. Assert the wire grpc-timeout has millisecond units (m) and a value from 1 to 500; expiration returns DeadlineExceeded, while interleaved caller cancellation remains Cancelled.
  2. Cross the existing failure threshold with concurrent reads. Observe one recreate per client wrapper, then successful reads on a fresh connection without restarting the client.
  3. Recover a peer using the same refreshed channel cache and verify the actual payload interceptor still resolves response payloads.
• Evidence (optional): ClientDeadlineTests.StalledReads_WithInterleavedCancellation_RecreateOnceAndRecoverSharedChannelAsync.

---

Notes for reviewers

• No breaking change against released packages - GrpcCallTimeout is a new opt-in option and defaults to null; existing released signatures, serialization, and orchestration replay are unchanged. Its earlier name existed only in this unmerged PR, so no obsolete alias is retained.
• Caller-token cancellation remains Cancelled and does not erase prior transport failures. A Cancelled response with no cancelled caller token resets the count like other application responses; gRPC cannot prove its network origin when caller/server cancellation race.
• A deadline does not prove that the server stopped a mutation. This adds no retry policy, keepalive configuration, or worker-stream timeout. Recovery benefits subsequent calls and does not resolve a server outage.
• The loopback test demonstrates the SDK failure/recovery path, not the underlying cause of any production connectivity incident.

[Copilot]

🟢 Approval recommended

No blocking issues were identified in the opt-in deadline behavior, compatibility, or recovery integration, and the changes include focused regression coverage.

No open issues 🎉

What changed in this PR

Adds opt-in unary RPC deadlines to the Durable Task SDK’s gRPC and scheduler clients, addressing stalled management calls without changing the default timeout behavior.

Changes:

• Adds DefaultUnaryCallTimeout, preserves earlier deadlines, and excludes long-poll waits and streams.
• Makes cancellation neutral in channel recovery so it no longer clears prior transport failures.
• Adds unit and loopback integration coverage, plus configuration and mutation-timeout guidance.

File	Description
test/​Grpc.IntegrationTests/​ClientDeadlineTests.cs	Tests stalled-call recovery, shared channels, and payload interception.
test/​Client/​Grpc.Tests/​GrpcDurableTaskClientInterceptorsTests.cs	Tests deadline application, exclusions, validation, and option preservation.
test/​Client/​Grpc.Tests/​GrpcDurableTaskClientChannelRecreationTests.cs	Tests cancellation preserving transport-failure counts.
test/​Client/​AzureManaged.Tests/​DurableTaskSchedulerClientExtensionsTests.cs	Tests named timeout settings and channel-cache reuse.
src/​Client/​Grpc/​README.md	Documents timeout configuration and ambiguous mutation outcomes.
src/​Client/​Grpc/​GrpcDurableTaskClientOptions.cs	Exposes the optional gRPC timeout setting.
src/​Client/​Grpc/​GrpcDurableTaskClient.cs	Integrates timeout enforcement into the interceptor chain.
src/​Client/​Grpc/​DefaultUnaryCallTimeoutInterceptor.cs	Applies bounded unary deadlines while preserving earlier deadlines.
src/​Client/​Grpc/​ChannelRecreatingCallInvoker.cs	Makes cancellation neutral and shares long-poll classification.
src/​Client/​AzureManaged/​DurableTaskSchedulerClientOptions.cs	Exposes the optional scheduler timeout setting.
src/​Client/​AzureManaged/​DurableTaskSchedulerClientExtensions.cs	Transfers named scheduler timeout settings to gRPC options.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Shared-channel recovery and cancellation interactions warrant maintainer sign-off, with current-head hosted validation still unconfirmed in the supplied evidence.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Shared-channel recovery behavior warrants human sign-off and current-head hosted validation given the reported previous-head test timeout.

No open issues 🎉

_{Review effort: Balanced}