Skip to content

[BUG] Wait on a completion state in the Elasticsearch exporter - #4298

Merged
marcalff merged 9 commits into
open-telemetry:mainfrom
thc1006:bugfix/elasticsearch-completion-state
Aug 2, 2026
Merged

[BUG] Wait on a completion state in the Elasticsearch exporter#4298
marcalff merged 9 commits into
open-telemetry:mainfrom
thc1006:bugfix/elasticsearch-completion-state

Conversation

@thc1006

@thc1006 thc1006 commented Jul 24, 2026

Copy link
Copy Markdown
Member

Fixes #4296

The problem

bool waitForResponse()
{
  std::unique_lock<std::mutex> lk(mutex_);
  cv_.wait(lk);
  return response_received_;
}

OnResponse() set response_received_ under the mutex and called notify_all() after releasing it. If that landed before the exporting thread reached cv_.wait(lk), the notification was gone and the thread blocked with nothing left to wake it. The OnEvent() failure cases were worse: they called notify_all() without taking the mutex and without recording anything, so an early failure left the waiter with no state to observe even if it did wake.

A spurious wakeup also returned response_received_, still false, so a request that was still in flight was reported as a failure.

SessionState::Destroyed neither notified nor recorded anything, so a session that ended without a response or an error event left the waiter blocked indefinitely.

Two threads, one recording the outcome and one waiting, with the ordering flipped between runs:

=== notification arrives AFTER the waiter parks ===
  old: response after wait                 returned success
  new: response after wait                 returned success
=== notification arrives BEFORE the waiter parks ===
  new: response before wait                returned success
  old: response before wait                HUNG, no wakeup left to receive

The last run had to be killed by a timeout.

The fix

The handler records an explicit completion state under the mutex on both the success and the failure paths, and the wait is a predicate over that state:

cv_.wait(lk, [this] { return completion_ != CompletionState::Pending; });
return completion_ == CompletionState::Success;

Because the predicate is evaluated under the same mutex that publishes the state, an outcome recorded before the waiter arrives is simply observed rather than missed, and a spurious wakeup goes back to waiting instead of reporting a failure.

Recording is first writer wins. That matters for Destroyed, which now records a failure too: it fires at the end of every session, including successful ones, and must not overwrite a result that has already been recorded. Its purpose here is to release a waiter when a session ends without ever producing a response.

What this does not change

The wait still has no timeout of its own. It is bounded by the request timeout arriving as a TimedOut session event, plus the Destroyed fallback added here. Giving the wait its own deadline would be a behavior change beyond this fix, and I would rather do it separately if you want it.

Verification

  • The comparison above was run, not reasoned about.
  • clang-format 18 clean.
  • I could not run the repo's clang-tidy against this file locally, since it needs the exporter's full include set. The enum carries an explicit std::uint8_t base type so it does not trip performance-enum-size, and <cstdint> is added for it. CI is the authority on warning_limit.

Scope

Synchronization only. The bulk response parsing is #4295 with its own PR, and the internal linkage of ResponseHandler belongs to the #4196 cleanup.

thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Jul 24, 2026
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.30%. Comparing base (feea622) to head (5459227).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4298      +/-   ##
==========================================
+ Coverage   81.24%   81.30%   +0.06%     
==========================================
  Files         446      447       +1     
  Lines       18978    19028      +50     
==========================================
+ Hits        15417    15469      +52     
+ Misses       3561     3559       -2     
Files with missing lines Coverage Δ
...orters/elasticsearch/src/es_log_record_exporter.cc 12.22% <ø> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

thc1006 added 4 commits July 26, 2026 03:51
waitForResponse() waited on a condition variable with no predicate and
then returned a bool that nothing on the error paths ever set.

OnResponse() set the flag under the mutex and notified after releasing
it, so a notification delivered before the exporting thread reached
cv_.wait() was lost and that thread blocked with nothing left to wake
it. The OnEvent failure cases were worse: they notified without taking
the mutex and without recording anything at all. A spurious wakeup also
returned false, reporting a request that was still in flight as failed.

The handler now records an explicit completion state under the mutex on
both the success and the failure paths, first writer wins so a session
destroyed after a good response does not overwrite it, and the wait is
on a predicate over that state. SessionState::Destroyed also records a
failure if nothing was recorded yet, so a session that ends without a
response no longer leaves the waiter blocked forever.

Checked with a two-thread harness: with the notification delivered
before the waiter parks, the old shape blocks until the test is killed
while the new one returns immediately.

Fixes open-telemetry#4296

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
IWYU reports stdint.h as unused once cstdint is there for the enum base
type. The only fixed-width type in the file is uint8_t, which cstdint
provides.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The gcc 14 abiv2 run failed only on ext.http.curl.BasicCurlHttpTests.ElegantQuitQuick,
which this change does not touch, so re-running to confirm it is a flake.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 force-pushed the bugfix/elasticsearch-completion-state branch from 2129088 to fc4ff13 Compare July 25, 2026 19:55
…completion-state

# Conflicts:
#	CHANGELOG.md
@thc1006
thc1006 marked this pull request as ready for review July 25, 2026 21:16
@thc1006
thc1006 requested a review from a team as a code owner July 25, 2026 21:16
Copilot AI review requested due to automatic review settings July 25, 2026 21:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@marcalff marcalff left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@marcalff

marcalff commented Jul 29, 2026

Copy link
Copy Markdown
Member

@lalitb @ThomsonTan

Please review.

@marcalff
marcalff merged commit 11fa0db into open-telemetry:main Aug 2, 2026
72 checks passed
@thc1006
thc1006 deleted the bugfix/elasticsearch-completion-state branch August 2, 2026 16:57
cv_.notify_all();
recordCompletion(CompletionState::Failure);
break;
case http_client::SessionState::ReadError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR fixed the completion-state handling for most terminal events, but I think ReadError and WriteError still leave the state pending and may block Export() indefinitely. Can we both also call recordCompletion(CompletionState::Failure) for both the cases, like the other error states?

@lalitb

lalitb commented Aug 2, 2026

Copy link
Copy Markdown
Member

Follow-up: could we add tests using an injected fake HTTP client? This should cover completion before the wait starts, ReadError, WriteError, Destroyed, and response followed by Destroyed. The current Elasticsearch exporter tests do not exercise this synchronization path.

thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
Resolves the overlap with open-telemetry#4298, which landed after this branch last synced.

Both changes rewrite the same region of the synchronous ResponseHandler.
open-telemetry#4298 owns transport completion: OnResponse records a completion state under
the handler mutex and Export() waits on a predicate, first writer wins. This
branch owns the application outcome: it keeps the HTTP status alongside the
body so Export() can hand both to IsBulkResponseSuccessful().

The two layer rather than compete. recordCompletionLocked(Success) now means
the transport delivered a response, not that the export succeeded. A failure
event that fired first still keeps Failure, so Export() returns kFailure
without consulting the body.

response_received_ is gone, replaced by the completion state.
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
The seven states open-telemetry#4298 made terminal had no test. A future edit that drops
one back to a bare log would only show up as a hung export in the field.

The companion case covers the other direction: progress states must not
decide the result on their own, or a response arriving after them would
never be consulted.
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
The seven states open-telemetry#4298 made terminal had no test. A future edit that drops
one back to a bare log would only show up as a hung export in the field.

The companion case covers the other direction: progress states must not
decide the result on their own, or a response arriving after them would
never be consulted.
thc1006 added a commit to thc1006/opentelemetry-cpp that referenced this pull request Aug 2, 2026
It sat between open-telemetry#4298 and open-telemetry#4292, which breaks the run the rest of the list
follows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Elasticsearch synchronous export can block forever on a lost notification

4 participants