Repository navigation
fix(cohere): handle cancellation and forward stream attributes - #1160
Chen Yufeiyang (feiiiiii5) wants to merge 4 commits into
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1a4a5aec46
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| def close(self): | ||
| self.closed = True | ||
|
|
||
| with patch.object(BaseCohere, "chat_stream", lambda *a, **kw: _MockSyncStream()): |
There was a problem hiding this comment.
Use real Cohere streams for the cleanup regression coverage
For normal stream attribute/cleanup behavior, this replaces the provider with hand-written stand-ins, contrary to AGENTS.md's instruction to use real library classes and drive them with cassettes. The sync stand-in isn't even an iterator (__next__ is missing), yet the test passes because no chunks are read, so it cannot establish that closing a real Cohere stream preserves iteration and finalizes tracing. Extend the existing cassette-backed streaming tests to cover partial consumption and explicit closure instead.
AGENTS.md reference: AGENTS.md:L34-L35
Useful? React with 👍 / 👎.
| finally: | ||
| self._finish() |
There was a problem hiding this comment.
Record cleanup failures before finalizing stream spans
When the underlying aclose() raises—for example, when the caller is cancelled while awaiting transport cleanup—this finally path calls _finish() without the exception. _finish() permanently marks the tracker finished and emits a span without an error, so even a surrounding __aexit__ cannot record the cancellation afterward. The newly added sync close() methods have the same issue for cleanup errors. Handle cleanup exceptions and cancellation by finalizing with error=error and re-raising; use the error-free finish only on successful cleanup.
Useful? React with 👍 / 👎.
Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
1a4a5ae to
9aabf83
Compare
Drive the checked-in cassette recording instead of a hand-written stand-in: consume a real stream up to its first text event, close it, and assert the span is finished from the chunks read so far. The same test fails on main, where _TracedChatStream has no close().
|
Thanks — both points checked against the current head. P1 (real streams for cleanup coverage). The normal-path coverage now runs on the checked-in recordings: The iterators that remain hand-built only inject error paths a recording cannot provoke — P2 (cleanup failures swallowed). The Verified with |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0f4d2fa8f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| except (Exception, asyncio.CancelledError) as error: | ||
| _log_error_and_end_span(span, error) |
There was a problem hiding this comment.
Preserve a nonempty diagnostic for ordinary task cancellation
When a request is cancelled with Task.cancel() or by asyncio.wait_for(), the resulting CancelledError normally has no message. _log_error_and_end_span passes that exception directly to span.log, whose bt_safe_deep_copy serialization falls back to str(error). An isolated reproduction of this path exports {"error": ""} and ends the span, losing any indication that cancellation caused the termination. The newly caught streaming and cleanup cancellations have the same problem; the added tests miss it because every injected cancellation supplies a message. Log a formatted exception or fall back to its type name, and cover message-less cancellation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 06f991e. _log_error_and_end_span now falls back to type(error).__name__ when str(error) is empty, so a bare CancelledError from Task.cancel() exports error: "CancelledError" instead of "". The change is in the shared helper, so the streaming and cleanup cancellation paths you flagged are covered by the same fix.
test_wrap_cohere_async_chat_message_less_cancellation_records_error raises asyncio.CancelledError with no message and asserts the exported error is "CancelledError"; it fails without the fallback.
Verified: pytest src/braintrust/integrations/test_utils.py src/braintrust/integrations/cohere/test_cohere.py --vcr-record=none → 78 passed.
A CancelledError raised by Task.cancel() has an empty str(), so the span exported error: "" and lost what ended it. Fall back to the type name and cover the message-less cancellation in the cohere stream tests.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 06f991e9bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| ``error`` field. Fall back to the type name so the span still records what | ||
| ended it. | ||
| """ | ||
| span.log(error=error if str(error) else type(error).__name__) |
There was a problem hiding this comment.
Guard exception stringification before probing for an empty message
When a provider or custom Transformers pipeline raises an exception whose __str__ fails, this new probe throws before either span.log() or span.end(). Previously, the boundary-safe serializer contained that failure through _safe_str; callers now receive the rendering error instead of the original exception and leave the span unfinished. Comparing this path with the parent commit reproduces the regression using RuntimeError with an unprintable message object. Guard the conversion itself and fall back to the exception type when conversion fails, not just when it returns an empty string.
AGENTS.md reference: AGENTS.md:L136-L136
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch - fixed in 2b7f7ec.
The probe is now contained: str(error) is wrapped, and when it raises, the empty result takes the same path as a message-less exception, so span.log(error=type(error).__name__) still runs and span.end() is always reached. Passing the exception object through is unchanged for the normal case, so _safe_str in the serializer still sees it.
test_log_error_and_end_span_survives_an_unprintable_error uses an exception whose __str__ raises and asserts the span is logged and ended with the type name; it fails on the parent commit. cohere + utils suites: 79 passed with --vcr-record=none.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b7f7ecb7a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| assert spans[0]["metrics"]["end"] >= spans[0]["metrics"]["start"] | ||
|
|
||
|
|
||
| def test_wrap_cohere_async_chat_message_less_cancellation_records_error(memory_logger): |
There was a problem hiding this comment.
Parameterize the duplicate async cancellation cases
This new case duplicates test_wrap_cohere_async_chat_cancellation_records_error: the client setup, patch, cancellation propagation, span-count check, and end-time check are identical; only the exception message and expected diagnostic differ. AGENTS.md mandates the integration skill's review checks, whose “Consolidate before handoff” rule requires merging input-only variants into a parametrized test. Parameterize the cancellation exception and expected diagnostic, retaining the message-less regression as a parameter, so the shared setup and finalization assertions cannot drift between copies.
AGENTS.md reference: AGENTS.md:L136-L136
Useful? React with 👍 / 👎.
|
|
||
|
|
||
| def test_wrap_cohere_async_chat_cancellation_records_error(memory_logger): | ||
| from unittest.mock import patch |
There was a problem hiding this comment.
lets avoid mocks/fakes! If we need to we should record new vcr cassettes
There was a problem hiding this comment.
Agreed on avoiding hand-written fakes where a cassette can carry the behavior — the two stream-consumption tests already reuse the checked-in VCR cassette (test_wrap_cohere_chat_stream_v1_sync), and the subscriptions test uses a real client, so there is no stand-in client on those paths.
The remaining four cases cover behavior a cassette cannot express: a task cancelled mid-chat()/chat_stream(), and a stream whose close() raises. VCR replays recorded HTTP responses; there is no response to record when the caller raises CancelledError, and a failing close() is local cleanup rather than traffic. I used a narrow patch.object on the Cohere base client so the wrapper's span-finalization code still runs — the instrumentation itself is not replaced.
I am happy to change the approach if you have a preferred pattern, e.g. cancelling a real VCR-backed client task at a specific await point, or using a fault-injecting transport fixture. Which do you prefer?
Summary
This PR fixes two issues in the Cohere integration tracing:
_trace_async_call,_async_chat_stream_wrapper, and_AsyncTracedChatStream.__anext__previously caught onlyexcept Exception as error:. In Python 3.8+,asyncio.CancelledErrorinherits fromBaseExceptionrather thanException. When an async call or stream was cancelled (e.g. timeout or cancelled task), the span was left open without recording the error, leaking active spans and losing telemetry on cancellations._TracedChatStreamand_AsyncTracedChatStream) wrapped the underlying Cohere stream without forwarding custom/underlying attributes (likeresponse,id,citations) and lacked explicitclose()/aclose()methods to cleanly close the underlying iterator and end the span.Changes
(Exception, asyncio.CancelledError)in_trace_async_call,_async_chat_stream_wrapper, and_AsyncTracedChatStream.__anext__, ensuring_log_error_and_end_spanis called on cancellation.__getattr__to_ChatStreamTrackerto forward attribute lookups to the underlying iterator.close()on_TracedChatStreamand_AsyncTracedChatStream, andaclose()on_AsyncTracedChatStream, delegating to the underlying stream's cleanup methods and ensuringself._finish()runs.test_cohere.pycovering async call cancellation, async stream cancellation, attribute forwarding, and sync/async stream closure.Testing
pytest src/braintrust/integrations/cohere/test_cohere.py(22 passed)ruff check src/braintrust/integrations/cohere/(0 errors)ruff format --check src/braintrust/integrations/cohere/(formatted)