Repository navigation
Description
What would you like?
Make both Java durable OpenTelemetry views honor the configured provider sampler selected through the standard OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG settings in supported ADOT/autoconfiguration setups.
Currently, an explicit propagated X-Ray Sampled=0/1 decision bypasses the configured sampler and is enforced for durable spans. Consequently, OTEL_TRACES_SAMPLER=always_off can drop ADOT handler/client spans while the durable plugin still exports Workflow, Invocation, operation, and attempt spans when the upstream header contains Sampled=1. This is the documented current behavior; this feature requests changing that contract so the selected sampler determines whether to inherit the parent decision.
Support the standard policies with their OpenTelemetry semantics:
| Configured sampler | Parent sampled (Sampled=1) |
Parent not sampled (Sampled=0) |
No parent supplied to sampler |
|---|---|---|---|
always_on |
Sample | Sample | Sample |
always_off |
Drop | Drop | Drop |
traceidratio |
Decide from trace ID and ratio | Decide from trace ID and ratio | Decide from trace ID and ratio |
parentbased_always_on |
Sample | Drop | Sample |
parentbased_always_off |
Sample | Drop | Drop |
parentbased_traceidratio |
Sample | Drop | Decide from trace ID and ratio |
Use OTEL_TRACES_SAMPLER_ARG for the ratio through the provider's existing configuration mechanism. A valid parent with an unset sampled bit is different from having no parent; retain OpenTelemetry ParentBased semantics for that case.
Possible Implementation
Use the provider's resolved sampler rather than adding a separate environment parser in the durable plugin. Preserve programmatically configured providers and their ownership/lifecycle; document which supported setup resolves environment configuration and how explicit provider configuration takes precedence.
Apply the change through the existing provider-builder and ADOT extension/autoconfiguration paths. Keep the durable wrapper scoped to SDK spans while preserving the configured delegate for ordinary spans. Qualify supported application/agent classloader arrangements and retain the complete sampling result, including RECORD_ONLY, sampler attributes, and trace state where already supported. Existing in-memory caching must not alter the support boundary or imply durable persistence.
Define the stable parent context supplied to the sampler. ParentBased policies must see the applicable propagated parent, while a root policy must receive no parent. Do not always evaluate ParentBased at root, or feed a synthesized sampled ancestor back into a root decision. Resolve the durable sampling decision consistently and apply it to every SDK-created span; ordinary application and ADOT spans retain their provider behavior.
Sampling must remain deterministic for a durable execution: use its stable canonical trace ID with policy/configuration unchanged across retries and reinvocations. Random, time-dependent, load-dependent, or mutable-state-dependent custom decisions remain unsupported. This feature does not request sampling-decision persistence or make in-memory caching an additional supported sampling mode.
Acceptance criteria:
- Both Execution and Invocation views implement the policy matrix through supported provider/autoconfiguration paths, including
OTEL_TRACES_SAMPLER_ARG. - Define and document behavior when the environment variables are unset, invalid, or superseded by programmatic provider configuration; reuse the provider's validation/default behavior where applicable.
- Test explicit upstream sampled/not-sampled contexts, a valid parent with no sampled flag, and no usable parent. Assert direct-sampler versus ParentBased behavior separately.
- With
always_offand upstreamSampled=1, no durable Workflow, Invocation, operation, attempt, or synthetic-root spans are exported in either the initial or resumed invocation. - Deterministic ratio tests exercise trace IDs on both sides of the threshold and initial invocation plus resume/retry. Every durable span follows the execution's resolved decision; completed user work is skipped on replay. Do not require exactly half of a small random batch to be sampled.
- Preserve provider ownership, ordinary application/ADOT span sampling, trace identity, and context cleanup. No persistent sampling-decision store is introduced.
- Keep a regression for sampled durable spans encountering a known dropped same-trace ambient handler parent. Honoring
always_offalone does not establish that every missing-parent case is fixed. Define the appropriate fallback and distinguish exporter evidence from X-Ray backend/console qualification. - Update the Java OTel README and shared sampling contract with compatibility/migration guidance, and coordinate JavaScript/Python behavior and applicable language-neutral conformance requirements before implementing inconsistent per-language precedence.
Is this a breaking change?
Yes. This changes documented trace sampling behavior: explicit upstream Sampled=0/1 currently takes precedence regardless of the configured sampler. Describe migration to parentbased_* for applications that want to retain upstream sampling decisions. Define the unset/default behavior explicitly.
Does this require an RFC?
Yes. Settle the cross-language sampling contract, parent-context inputs, defaults, and compatibility approach before implementation.
Additional Context
- Companion runtime feature: JavaScript #1133 — Honor OTEL_TRACES_SAMPLER for durable spans.
- Companion runtime feature: Python #887 — Honor OTEL_TRACES_SAMPLER for durable spans. Keep the configured-sampler contract, determinism requirement, and compatibility approach aligned across Java, JavaScript, and Python.
- Documentation coordination: aws/aws-durable-execution-docs#295 now tracks the sampler matrix, support boundary, migration guidance, and per-language/version applicability. Do not advertise the new behavior before the relevant runtime release is available.
- Java source inspected at
4b1aa82b0d898fea495cda45760ce3737fbb341e: OtelPluginSupport.resolveSamplingResult returns immediately for explicit upstream sampled/not-sampled decisions before consulting the configured delegate. - DurableSampler applies the carried decision to durable spans while ordinary spans use the delegate. The ADOT auto-configuration customizer wraps the sampler already resolved by OpenTelemetry configuration.
- DurableSamplerTest.explicitSampled_winsOverConfiguredAlwaysOff and the adjacent unsampled/AlwaysOn test explicitly assert the current precedence. Update these expectations as part of the agreed contract change, with coverage for both views and actual ADOT/autoconfiguration setup.
- The current Java sampling documentation documents unconditional upstream precedence, full-result propagation, and provider/classloader behavior. Preserve those implementation guarantees that remain applicable while revising the precedence contract.
The evidence above is Java source and existing-test inspection. It does not claim a new Java ADOT deployment, exporter/backend reproduction, or X-Ray console validation. The related Python dropped-ambient-parent observation is a regression scenario to qualify here, not proof that the same Java deployment behavior has been reproduced.
Activity
zhongkechen commentedon Nov 2, 2026
Severity - medium
When it triggers. The Java durable sampler is installed through the supported provider/ADOT setup, and an explicit propagated
Sampled=0/1decision conflicts with the configured sampler. The upstream decision currently wins in both OTel views.Impact. With upstream
Sampled=1,always_offcan suppress ordinary ADOT spans while Workflow, Invocation, operation, and attempt spans are still exported. With upstreamSampled=0,always_oncannot restore durable spans. Consequently, an application cannot use the configured sampler alone to control durable trace volume or align its durable and ordinary instrumentation; the first case can add unwanted export volume and overhead.What is correct. This is the documented and explicitly tested sampling contract today, not evidence of a newly broken implementation or a regression. The reported discrepancy concerns telemetry selection; no incorrect execution result or replay behavior is established.
Why this severity. The impact extends across durable spans for the affected executions and defeats a standard operator control, so it is more than a cosmetic configuration difference. Higher severity is not supported: the trigger requires a conflicting upstream decision, and the issue establishes no execution failure or measured production cost incident. Changing the precedence is a compatibility change and should follow the coordinated cross-language contract and migration to
parentbased_*where upstream inheritance is intended.Assessment is from source and the existing sampler contract, not a new Java ADOT deployment.