Skip to content

[otel] Balance context attach and detach across plugin lifecycles #643

Description

@zhongkechen

Summary

Both OTel plugins call opentelemetry.context.attach() without retaining the returned token or calling detach(). This leaves a workflow, invocation, operation, or attempt span active after its intended scope has ended.

The test suites currently reset the OTel context around every test specifically because plugin state otherwise leaks between tests.

Evidence

  • packages/aws-durable-execution-sdk-python-otel/src/aws_durable_execution_sdk_python_otel/execution_plugin.py, attaches around lines 218-221, 458, and 506-508
  • packages/aws-durable-execution-sdk-python-otel/src/aws_durable_execution_sdk_python_otel/invocation_plugin.py, attaches around lines 565 and 620-622
  • The autouse fixtures in tests/test_execution_plugin.py and tests/test_invocation_plugin.py state that the plugins attach spans without detaching and would otherwise leak context between tests

A focused runtime probe completes a non-terminal invocation and observes that trace.get_current_span() still returns the prior workflow span after on_invocation_end.

OpenTelemetry requirement

The Context specification defines attach as returning a token used to restore the previous context and states that every attach should have a corresponding detach:

https://opentelemetry.io/docs/specs/otel/context/#attach-context
https://opentelemetry.io/docs/specs/otel/context/#detach-context

Impact

  • Auto-instrumented work after plugin cleanup can be parented to a stale span.
  • Ended or abandoned spans can remain current across warm Lambda invocations.
  • Log correlation may report stale trace and span identifiers.
  • Nested operations overwrite context rather than managing a balanced scope.

Acceptance criteria

  • Every context.attach() token owned by the plugin has a corresponding context.detach().
  • Invocation cleanup restores the context that was active before plugin invocation start.
  • User-function cleanup restores the exact enclosing context without accumulating stale scopes.
  • Tests no longer require global context resets to hide plugin lifecycle leaks.
  • Tests cover nested child contexts, multiple sequential steps, failures, and warm invocation reuse.

Metadata

Metadata

Assignees

Labels

bugprojectThe issues with an ongoing project

Type

No type

Projects

Status
In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions