feat(observability): suppress redundant gRPC auto-instrumentation spans

[chalmerlowe]

Intent & Context

When upstream OpenTelemetry gRPC auto-instrumentation (opentelemetry.instrumentation.grpc.GrpcInstrumentorClient().instrument()) is enabled globally in a customer application alongside Google Cloud SDK tracing, downstream channel invocations create redundant, un-enriched wire attempt spans.

This PR introduces targeted downstream suppression interceptors for both sync (grpc) and async (grpc.aio) transports:

• Wraps downstream channel execution in OpenTelemetry's native suppress_instrumentation context manager.
• Ensures Google Cloud SDK's enriched T4 attempt span is emitted with all Google Cloud semantic convention attributes intact.
• Prevents redundant, bare-bones upstream auto-instrumentation child spans from being created.
• Stacked directly on PR feat(observability): implement universal 4-path OpenTelemetry tracing #18433 (feat/otel-tracing-universal-4path).

[gemini-code-assist]

Code Review

This pull request introduces client interceptors (_SuppressingClientInterceptor and _AsyncSuppressingClientInterceptor) to suppress redundant downstream generic auto-instrumentation spans when OpenTelemetry is enabled, ensuring that only the feature-rich Google Cloud SDK spans are emitted. It also updates the interceptor pipeline logic to prepend OpenTelemetry interceptors in order. The review feedback suggests critical performance optimizations, specifically moving imports out of the hot path in _suppress_instrumentation to avoid overhead on every RPC call, and removing a redundant inline import of grpc in otel_interceptor.

[chalmerlowe]

A good deal of the above is duplicative with test_auto_instrumentation_suppression_async()

It also happens to be duplicative with a number of the other existing tests.

I am putting together a PR with two standardized assertion functions to help reduce the duplication across the broader array of tests, but did not want to clutter up this PR.

[daniel-sanche]

I don't have time to do a full in-depth review today, but I'd really suggest taking a step back and seeing if there's another way to solve this. This would add a lot of bloat, with the three layers of wrapers fighting each other, which I'm sure would show up in benchmarks. And the workarounds added to apply_channel_interceptors to identify and re-order interceptors feel like a big abstraction overreach

I added a comment about a possible approach to strip out unwanted interceptors before applying our custom otel ones. If that doesn't work, maybe spend some time considering other options, before committing to these new wrapper layers

[daniel-sanche]

My unerstanding of this fix is that channel may already be wrapped with an interceptor, and we need to add another wrapper to supress it

Instead, can we unwrap the channel here?

    def otel_interceptor(channel: grpc.Channel) -> grpc.Channel:
        # Peel off any generic upstream auto-instrumentation wrapper once at setup time.
        while (
            isinstance(getattr(channel, "_interceptor", None), OpenTelemetryClientInterceptor)
            and not getattr(channel._interceptor, "_is_otel_interceptor", False)
            and hasattr(channel, "_channel")
        ):
            channel = channel._channel
        return otel_grpc.intercept_channel(channel, interceptor)

(We could do something similar on the async side, pruning existing the channel's interceptor list before adding our own otel one)

[chalmerlowe]

RECOMMENDATION: LEAVE AS IS

We investigated unwrapping/peeling the channel at setup time as you suggested, but found two significant risks that make runtime suppression the safer path:

1. Dropping Intentionally Configured User Interceptors: Upstream opentelemetry-instrumentation-grpc provides a public factory function client_interceptor(request_hook=..., filter_=...) that developers use to manually configure custom request hooks or filters without subclassing. The returned object is a direct instance of OpenTelemetryClientInterceptor. If a user configures a channel with custom hooks and passes it into our client, peeling off OpenTelemetryClientInterceptor at setup time will silently delete the customer's custom hooks. In contrast, suppress_instrumentation() is the upstream contract designed to suppress span creation without mutating/deleting parts of the channel structure.

2. Shared Channel Breakage: If a customer passes an existing channel and shares it between our SDK and their own internal gRPC services, stripping/mutating interceptors on the channel permanently breaks tracing for their internal services. Runtime suppression avoids this because suppress_instrumentation() is thread/task-local and only active during the execution of OUR SDK's call.

[daniel-sanche]

I would really like to avoid this kind of special-case re-ordering if at all possible. Even using a different method for re-ordering, before calling apply_channel_interceptors if you have to

If you need to add it here, it should be called out in the docstring that otel is treated as a special case

[chalmerlowe]

For clarity, the reordering is required to enable the suppression.
We looked for ways to make it simpler and more robust and to help divide the architecture such that the calling function decides whether to prepend OR append the interceptor.

1. In google-api-core (grpc_helpers_async.py): We stripped all _is_otel_interceptor logic and index math out of apply_channel_interceptors. It is now a generic utility that accepts prepend: bool = False. When prepending, it uses Python slice assignment (target_list[:0] = matching) so the batch is inserted at the front in exact relative order without using stateful index counters (also resolving Tony's sequential ordering comment).
2. In gapic-generator (grpc_asyncio.py.j2): The transport caller explicitly declares its ordering intent. Normal user and logging interceptors are applied with prepend=False (appended). Observability interceptors are applied with prepend=True, placing them at the front of the channel's execution pipeline so they wrap outside and suppress ambient auto-instrumentation on pre-existing channels.
   This keeps google-api-core agnostic of OpenTelemetry while giving calling functions control over pipeline positioning.

[chalmerlowe]

Some suggested changes.

[chalmerlowe]

I don't have time to do a full in-depth review today, but I'd really suggest taking a step back and seeing if there's another way to solve this. This would add a lot of bloat, with the three layers of wrapers fighting each other, which I'm sure would show up in benchmarks. And the workarounds added to apply_channel_interceptors to identify and re-order interceptors feel like a big abstraction overreach

I added a comment about a possible approach to strip out unwanted interceptors before applying our custom otel ones. If that doesn't work, maybe spend some time considering other options, before committing to these new wrapper layers

Thanks for the feedback, @daniel-sanche

We recommend moving forward with the approach we recommended in the PR. Additional clarification can be found on specific comments, but broadly:

1. Industry Standard: Wrapping unwanted interceptors that have been automatically applied by auto-instrumentation with a suppress_instrumentation interceptor is the canonical pattern used across the OpenTelemetry ecosystem. In HTTP, opentelemetry-instrumentation-requests wraps calls in with suppress_http_instrumentation() (the REST analog to suppress_instrumentation): to stop downstream urllib3 from creating duplicate HTTP spans. Upstream opentelemetry-instrumentation-grpc has if not is_instrumentation_enabled(): return invoker(...) in its interceptors specifically so callers can suppress them with suppress_instrumentation().
2. Why Wrappers Don't Contend with wrap_method: This suppression operates at a completely non-overlapping architectural layer from wrap_method:
   • wrap_method operates at the GAPIC Method Layer: it handles the user's retry loop, timeout budget, and the high-level Tier 3 span (Echo/Echo).
   • Our channel interceptors live strictly inside the gRPC Transport Pipeline: they only measure and guard the individual physical wire attempt (Tier 4).
     They don't modify or fight each other; wrap_method delegates to the transport, which delegates to the channel.
3. Benchmark Overhead: We ran timing benchmarks using Python’s standard timeit.repeat across 500,000 iterations (5 rounds of 100k). The net overhead of _SuppressingClientInterceptor entering and exiting suppress_instrumentation() is ~4.51 microseconds (0.0045 ms) per RPC. Compared to real-world gRPC network latencies—ranging from ultra-fast intra-zone calls (~150–250 µs) to intra-region calls (~500–1,000 µs) and cross-region calls (20–40 ms)—4.5 µs represents ~1.8% of intra-zone time, <0.5% intra-region, and <0.02% cross-region, making it negligible in production. Furthermore, when tracing is not enabled, the interceptor is never attached, guaranteeing 0.00 µs overhead.

Beyond that:

• We removed the OTel-specific index tracking from apply_channel_interceptors and made it a clean, generic utility with prepend: bool = False.
• We removed the capability constant from _compat.py.j2 so it stays out of generated client packages.

[snippet-bot]

No region tags are edited in this PR.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

• Refresh this comment