[BUG] Wait on a completion state in the Elasticsearch exporter - #4298
Conversation
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4298 +/- ##
==========================================
+ Coverage 81.24% 81.30% +0.06%
==========================================
Files 446 447 +1
Lines 18978 19028 +50
==========================================
+ Hits 15417 15469 +52
+ Misses 3561 3559 -2
🚀 New features to boost your workflow:
|
waitForResponse() waited on a condition variable with no predicate and then returned a bool that nothing on the error paths ever set. OnResponse() set the flag under the mutex and notified after releasing it, so a notification delivered before the exporting thread reached cv_.wait() was lost and that thread blocked with nothing left to wake it. The OnEvent failure cases were worse: they notified without taking the mutex and without recording anything at all. A spurious wakeup also returned false, reporting a request that was still in flight as failed. The handler now records an explicit completion state under the mutex on both the success and the failure paths, first writer wins so a session destroyed after a good response does not overwrite it, and the wait is on a predicate over that state. SessionState::Destroyed also records a failure if nothing was recorded yet, so a session that ends without a response no longer leaves the waiter blocked forever. Checked with a two-thread harness: with the notification delivered before the waiter parks, the old shape blocks until the test is killed while the new one returns immediately. Fixes open-telemetry#4296 Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
IWYU reports stdint.h as unused once cstdint is there for the enum base type. The only fixed-width type in the file is uint8_t, which cstdint provides. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
2129088 to
fc4ff13
Compare
…completion-state # Conflicts: # CHANGELOG.md
|
Please review. |
| cv_.notify_all(); | ||
| recordCompletion(CompletionState::Failure); | ||
| break; | ||
| case http_client::SessionState::ReadError: |
There was a problem hiding this comment.
This PR fixed the completion-state handling for most terminal events, but I think ReadError and WriteError still leave the state pending and may block Export() indefinitely. Can we both also call recordCompletion(CompletionState::Failure) for both the cases, like the other error states?
|
Follow-up: could we add tests using an injected fake HTTP client? This should cover completion before the wait starts, ReadError, WriteError, Destroyed, and response followed by Destroyed. The current Elasticsearch exporter tests do not exercise this synchronization path. |
Resolves the overlap with open-telemetry#4298, which landed after this branch last synced. Both changes rewrite the same region of the synchronous ResponseHandler. open-telemetry#4298 owns transport completion: OnResponse records a completion state under the handler mutex and Export() waits on a predicate, first writer wins. This branch owns the application outcome: it keeps the HTTP status alongside the body so Export() can hand both to IsBulkResponseSuccessful(). The two layer rather than compete. recordCompletionLocked(Success) now means the transport delivered a response, not that the export succeeded. A failure event that fired first still keeps Failure, so Export() returns kFailure without consulting the body. response_received_ is gone, replaced by the completion state.
The seven states open-telemetry#4298 made terminal had no test. A future edit that drops one back to a bare log would only show up as a hung export in the field. The companion case covers the other direction: progress states must not decide the result on their own, or a response arriving after them would never be consulted.
The seven states open-telemetry#4298 made terminal had no test. A future edit that drops one back to a bare log would only show up as a hung export in the field. The companion case covers the other direction: progress states must not decide the result on their own, or a response arriving after them would never be consulted.
It sat between open-telemetry#4298 and open-telemetry#4292, which breaks the run the rest of the list follows.
Fixes #4296
The problem
OnResponse()setresponse_received_under the mutex and callednotify_all()after releasing it. If that landed before the exporting thread reachedcv_.wait(lk), the notification was gone and the thread blocked with nothing left to wake it. TheOnEvent()failure cases were worse: they callednotify_all()without taking the mutex and without recording anything, so an early failure left the waiter with no state to observe even if it did wake.A spurious wakeup also returned
response_received_, still false, so a request that was still in flight was reported as a failure.SessionState::Destroyedneither notified nor recorded anything, so a session that ended without a response or an error event left the waiter blocked indefinitely.Two threads, one recording the outcome and one waiting, with the ordering flipped between runs:
The last run had to be killed by a timeout.
The fix
The handler records an explicit completion state under the mutex on both the success and the failure paths, and the wait is a predicate over that state:
Because the predicate is evaluated under the same mutex that publishes the state, an outcome recorded before the waiter arrives is simply observed rather than missed, and a spurious wakeup goes back to waiting instead of reporting a failure.
Recording is first writer wins. That matters for
Destroyed, which now records a failure too: it fires at the end of every session, including successful ones, and must not overwrite a result that has already been recorded. Its purpose here is to release a waiter when a session ends without ever producing a response.What this does not change
The wait still has no timeout of its own. It is bounded by the request timeout arriving as a
TimedOutsession event, plus theDestroyedfallback added here. Giving the wait its own deadline would be a behavior change beyond this fix, and I would rather do it separately if you want it.Verification
std::uint8_tbase type so it does not tripperformance-enum-size, and<cstdint>is added for it. CI is the authority onwarning_limit.Scope
Synchronization only. The bulk response parsing is #4295 with its own PR, and the internal linkage of
ResponseHandlerbelongs to the #4196 cleanup.