Concurrent multi-processors never flush at interpreter exit - #5569
Open
dwin-gharibi wants to merge 2 commits into
Open
Concurrent multi-processors never flush at interpreter exit#5569dwin-gharibi wants to merge 2 commits into
dwin-gharibi wants to merge 2 commits into
Conversation
The providers register shutdown with atexit, but concurrent.futures registers its cleanup through threading._register_atexit and CPython runs threading._shutdown() before the atexit queue, so the pool is already closed when the provider's shutdown runs. Drive both signals in a subprocess, since the defect only appears at real interpreter exit, and assert the telemetry is exported and no RuntimeError is reported. Unit tests close the pool explicitly and assert the work still runs inline, with a control asserting a healthy pool is unaffected. These tests fail against the current implementation.
TracerProvider and LoggerProvider register their shutdown with atexit, but concurrent.futures registers its own cleanup through threading._register_atexit and CPython runs threading._shutdown() before the atexit queue. The thread pool is therefore always closed by the time an atexit-driven provider shutdown reaches it, so submitting raised "RuntimeError: cannot schedule new futures after shutdown" and the underlying batch processor was never shut down. Every buffered span and log record was lost on every clean exit, reported only as an ignored atexit error. Fall back to running the callable inline when the pool refuses work. There is no concurrency worth preserving at that point in the process lifecycle. force_flush gets the same treatment and propagates a False result from an inline call.
Pull request dashboard statusWaiting on reviewers · refreshed 2026-08-23 16:17 UTC Review the latest changes. Status above doesn't look right?
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5568.
Description
TracerProviderandLoggerProviderregister their shutdown withatexit.register().concurrent.futuresregisters its cleanup throughthreading._register_atexit(), and CPython runsthreading._shutdown()before theatexitqueue.So by the time the provider's shutdown runs, the thread pool is already closed.
_submit_and_awaitraisesRuntimeError: cannot schedule new futures after shutdown, and the underlyingBatchSpanProcessor/BatchLogRecordProcessoris never shut down - its buffered telemetry is simply lost.Root cause
opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py:272(_submit_and_await) and:308(force_flush)opentelemetry-sdk/src/opentelemetry/sdk/_logs/_internal/__init__.py:470(_submit_and_wait) and:494(force_flush)Each submits unconditionally to a
ThreadPoolExecutorthat is guaranteed to be closed by the time an atexit-driven shutdown reaches it.Approach
Catch the
RuntimeErrorfromsubmitand run the callable inline on the calling thread. During interpreter shutdown there is no concurrency to preserve anyway - the point of the pool is parallelism across child processors, and correctness matters more than that here.force_flushtakes the same treatment, propagating aFalseresult from an inline call so the return value stays meaningful.Deliberately narrow: nothing changes while the pool is healthy, which the control test pins down.
Files changed
opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.pyopentelemetry-sdk/src/opentelemetry/sdk/_logs/_internal/__init__.pyopentelemetry-sdk/tests/test_concurrent_processor_at_exit.py.changelog/5568.fixedTesting
The defect only appears at genuine interpreter exit, so the primary tests drive a real subprocess for each signal and assert both that the telemetry was exported and that no RuntimeError was reported on stderr.
Unit tests close the executor explicitly and assert
shutdownandforce_flushstill reach the child processor for both the span and log variants. A control test asserts a healthy pool is still used, so the fallback cannot quietly become the only path.Eight of the nine fail before the change.
Result: 864 passed in
opentelemetry-sdk(855 baseline plus 9 new).Risk / compatibility
The inline fallback only triggers once the pool refuses work, which in practice means interpreter shutdown. Behaviour during normal operation is unchanged. Inline execution is sequential rather than parallel, which is the correct trade at that point in the process lifecycle.