Skip to content

[SDK Sentinel] Fix concurrent OpenTelemetry ID seed isolation - #1758

Open
sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/python/31980903097-1-694363938b1c
Open

[SDK Sentinel] Fix concurrent OpenTelemetry ID seed isolation#1758
sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/python/31980903097-1-694363938b1c

Conversation

@sdk-sentinel-bot

Copy link
Copy Markdown

Caution

This draft contains untrusted AI-generated code. Do not approve or run CI until a maintainer has reviewed the diff. SDK Sentinel verified that GitHub Actions remained approval-gated with zero executable jobs when this PR was opened.

Summary

The OpenAI OTel hierarchy test failed in an [earliest captured macOS job](https://github.com/temporalio/sdk-python/actions/runs/31820269667/job/94831574136) and a [later Windows recurrence](https://github.com/temporalio/sdk-python/actions/runs/31842411677/job/94919382813). An [intervening successful CI run](https://github.com/temporalio/sdk-python/actions/runs/31823119676) used byte-identical relevant files; no preceding equivalent success was captured before the earliest failure.

Root cause

Propagated trace and span seeds were stored in shared LIFO lists on TemporalIdGenerator. Concurrent worker contexts could consume each other's IDs, corrupting parent relationships nondeterministically.

Fix

Store seed stacks in execution-local context variables. Add a deterministic two-thread regression test covering both trace and span IDs, plus a changelog entry.

Validation

The regression test failed before and passed after the fix. The original focused tracing test also passes. Canonical validation passed all Python lint and type checks, then stopped because the local rustup installation had no toolchain; the unchanged baseline passed the full validation. Python 3.10 macOS and Windows CI remain required.

Validation status: validation-incomplete

  • Flake confidence: high
  • Fix confidence: high
  • Value: high
  • Patch scope: standard — 3 changed files; 6.1 KiB
  • Local reproduction: reproduced — The natural CI assertion did not reproduce in two Linux runs, but a deterministic two-thread regression test reproduced the causal trace/span seed exchange before the fix.
  • Regression coverage: fail-before-pass-after — Both span-ID and trace-ID cases returned the other thread's seed before the fix and their own seed afterward.
  • Unchanged baseline (Poe lint, bridge lint, and editable build) — passed
  • Independent candidate (Poe lint, bridge lint, and editable build) — passed

Investigator-run checks

  • poe test -s -k test_otel_tracing_in_runner — passed; 1 attempt(s). Natural Linux reproduction did not occur before the fix.
  • uv run pytest -s tests/contrib/openai_agents/test_openai_tracing.py::test_otel_tracing_in_runner — passed; 1 attempt(s). A second natural Linux reproduction attempt passed before the fix.
  • poe test -s -k temporal_id_generator_seeds_are_context_local — failed; 1 attempt(s). Expected pre-fix failure: both parameterizations deterministically exchanged IDs across threads.
  • poe test -s -k 'temporal_id_generator_seeds_are_context_local or test_otel_tracing_in_runner' — passed; 1 attempt(s). After the fix, both seed-isolation cases and the original tracing test passed.
  • .ci-flake-runtime/input/validate.sh — failed; 1 attempt(s). Python lint and type checks passed; validation then stopped before bridge lint/build because rustup had no installed or default toolchain. The trusted unchanged-checkout baseline passed the full command.

Required target CI

  • Continuous Integration / build-lint-test (3.10, macos-arm)
  • Continuous Integration / build-lint-test (3.10, windows-latest)
  • Canonical repository validation with the Rust toolchain available

Residual risks

  • The exact macOS/Windows hierarchy assertion was not naturally reproduced locally.
  • Candidate bridge lint and editable bridge build were not rerun because this sandbox had no installed Rust toolchain; the unchanged baseline completed them successfully.

Automation provenance

If this finding should not be fixed, apply the sdk-sentinel:false-positive label and close the PR. Sentinel will suppress the finding until its affected source changes.

SDK Sentinel recurrence history

  • Unique failed CI run attempts: 1
  • First occurrence: 2026-08-14 16:51 UTC
  • Latest occurrence: 2026-08-14 16:51 UTC
  • Recent occurrences:

@CLAassistant

CLAassistant commented Aug 17, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@tconley1428
tconley1428 marked this pull request as ready for review August 17, 2026 17:50
@tconley1428
tconley1428 requested a review from a team as a code owner August 17, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants