Skip to content

[Feature]: Honor OTEL_TRACES_SAMPLER for durable spans #853

Open

Description

@zhongkechen

What would you like?

Make both Java durable OpenTelemetry views honor the configured provider sampler selected through the standard OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG settings in supported ADOT/autoconfiguration setups.

Currently, an explicit propagated X-Ray Sampled=0/1 decision bypasses the configured sampler and is enforced for durable spans. Consequently, OTEL_TRACES_SAMPLER=always_off can drop ADOT handler/client spans while the durable plugin still exports Workflow, Invocation, operation, and attempt spans when the upstream header contains Sampled=1. This is the documented current behavior; this feature requests changing that contract so the selected sampler determines whether to inherit the parent decision.

Support the standard policies with their OpenTelemetry semantics:

Configured sampler Parent sampled (Sampled=1) Parent not sampled (Sampled=0) No parent supplied to sampler
always_on Sample Sample Sample
always_off Drop Drop Drop
traceidratio Decide from trace ID and ratio Decide from trace ID and ratio Decide from trace ID and ratio
parentbased_always_on Sample Drop Sample
parentbased_always_off Sample Drop Drop
parentbased_traceidratio Sample Drop Decide from trace ID and ratio

Use OTEL_TRACES_SAMPLER_ARG for the ratio through the provider's existing configuration mechanism. A valid parent with an unset sampled bit is different from having no parent; retain OpenTelemetry ParentBased semantics for that case.

Possible Implementation

Use the provider's resolved sampler rather than adding a separate environment parser in the durable plugin. Preserve programmatically configured providers and their ownership/lifecycle; document which supported setup resolves environment configuration and how explicit provider configuration takes precedence.

Apply the change through the existing provider-builder and ADOT extension/autoconfiguration paths. Keep the durable wrapper scoped to SDK spans while preserving the configured delegate for ordinary spans. Qualify supported application/agent classloader arrangements and retain the complete sampling result, including RECORD_ONLY, sampler attributes, and trace state where already supported. Existing in-memory caching must not alter the support boundary or imply durable persistence.

Define the stable parent context supplied to the sampler. ParentBased policies must see the applicable propagated parent, while a root policy must receive no parent. Do not always evaluate ParentBased at root, or feed a synthesized sampled ancestor back into a root decision. Resolve the durable sampling decision consistently and apply it to every SDK-created span; ordinary application and ADOT spans retain their provider behavior.

Sampling must remain deterministic for a durable execution: use its stable canonical trace ID with policy/configuration unchanged across retries and reinvocations. Random, time-dependent, load-dependent, or mutable-state-dependent custom decisions remain unsupported. This feature does not request sampling-decision persistence or make in-memory caching an additional supported sampling mode.

Acceptance criteria:

  • Both Execution and Invocation views implement the policy matrix through supported provider/autoconfiguration paths, including OTEL_TRACES_SAMPLER_ARG.
  • Define and document behavior when the environment variables are unset, invalid, or superseded by programmatic provider configuration; reuse the provider's validation/default behavior where applicable.
  • Test explicit upstream sampled/not-sampled contexts, a valid parent with no sampled flag, and no usable parent. Assert direct-sampler versus ParentBased behavior separately.
  • With always_off and upstream Sampled=1, no durable Workflow, Invocation, operation, attempt, or synthetic-root spans are exported in either the initial or resumed invocation.
  • Deterministic ratio tests exercise trace IDs on both sides of the threshold and initial invocation plus resume/retry. Every durable span follows the execution's resolved decision; completed user work is skipped on replay. Do not require exactly half of a small random batch to be sampled.
  • Preserve provider ownership, ordinary application/ADOT span sampling, trace identity, and context cleanup. No persistent sampling-decision store is introduced.
  • Keep a regression for sampled durable spans encountering a known dropped same-trace ambient handler parent. Honoring always_off alone does not establish that every missing-parent case is fixed. Define the appropriate fallback and distinguish exporter evidence from X-Ray backend/console qualification.
  • Update the Java OTel README and shared sampling contract with compatibility/migration guidance, and coordinate JavaScript/Python behavior and applicable language-neutral conformance requirements before implementing inconsistent per-language precedence.

Is this a breaking change?

Yes. This changes documented trace sampling behavior: explicit upstream Sampled=0/1 currently takes precedence regardless of the configured sampler. Describe migration to parentbased_* for applications that want to retain upstream sampling decisions. Define the unset/default behavior explicitly.

Does this require an RFC?

Yes. Settle the cross-language sampling contract, parent-context inputs, defaults, and compatibility approach before implementation.

Additional Context

The evidence above is Java source and existing-test inspection. It does not claim a new Java ADOT deployment, exporter/backend reproduction, or X-Ray console validation. The related Python dropped-ambient-parent observation is a regression scenario to qualify here, not proof that the same Java deployment behavior has been reproduced.

Activity

  1. zhongkechen commented on Nov 2, 2026

    @zhongkechen
    ContributorAuthor

    Severity - medium

    When it triggers. The Java durable sampler is installed through the supported provider/ADOT setup, and an explicit propagated Sampled=0/1 decision conflicts with the configured sampler. The upstream decision currently wins in both OTel views.

    Impact. With upstream Sampled=1, always_off can suppress ordinary ADOT spans while Workflow, Invocation, operation, and attempt spans are still exported. With upstream Sampled=0, always_on cannot restore durable spans. Consequently, an application cannot use the configured sampler alone to control durable trace volume or align its durable and ordinary instrumentation; the first case can add unwanted export volume and overhead.

    What is correct. This is the documented and explicitly tested sampling contract today, not evidence of a newly broken implementation or a regression. The reported discrepancy concerns telemetry selection; no incorrect execution result or replay behavior is established.

    Why this severity. The impact extends across durable spans for the affected executions and defeats a standard operator control, so it is more than a cosmetic configuration difference. Higher severity is not supported: the trigger requires a conflicting upstream decision, and the issue establishes no execution failure or measured production cost incident. Changing the precedence is a compatibility change and should follow the coordinated cross-language contract and migration to parentbased_* where upstream inheritance is intended.

    Assessment is from source and the existing sampler contract, not a new Java ADOT deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions