Skip to content

fix(cohere): handle cancellation and forward stream attributes - #1160

Open
Chen Yufeiyang (feiiiiii5) wants to merge 4 commits into
braintrustdata:mainfrom
feiiiiii5:fix/cohere-stream-cancellation
Open

Chen Yufeiyang (feiiiiii5) wants to merge 4 commits into
braintrustdata:mainfrom
feiiiiii5:fix/cohere-stream-cancellation

Conversation

@feiiiiii5

Copy link
Copy Markdown
Contributor

Summary

This PR fixes two issues in the Cohere integration tracing:

  1. _trace_async_call, _async_chat_stream_wrapper, and _AsyncTracedChatStream.__anext__ previously caught only except Exception as error:. In Python 3.8+, asyncio.CancelledError inherits from BaseException rather than Exception. When an async call or stream was cancelled (e.g. timeout or cancelled task), the span was left open without recording the error, leaking active spans and losing telemetry on cancellations.
  2. Traced streams (_TracedChatStream and _AsyncTracedChatStream) wrapped the underlying Cohere stream without forwarding custom/underlying attributes (like response, id, citations) and lacked explicit close() / aclose() methods to cleanly close the underlying iterator and end the span.

Changes

  • Catch (Exception, asyncio.CancelledError) in _trace_async_call, _async_chat_stream_wrapper, and _AsyncTracedChatStream.__anext__, ensuring _log_error_and_end_span is called on cancellation.
  • Add __getattr__ to _ChatStreamTracker to forward attribute lookups to the underlying iterator.
  • Implement close() on _TracedChatStream and _AsyncTracedChatStream, and aclose() on _AsyncTracedChatStream, delegating to the underlying stream's cleanup methods and ensuring self._finish() runs.
  • Add unit and regression tests in test_cohere.py covering async call cancellation, async stream cancellation, attribute forwarding, and sync/async stream closure.

Testing

  • Ran pytest src/braintrust/integrations/cohere/test_cohere.py (22 passed)
  • Ran ruff check src/braintrust/integrations/cohere/ (0 errors)
  • Ran ruff format --check src/braintrust/integrations/cohere/ (formatted)

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T12:29:44.081759Z 2b7f7ec New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a4a5aec46

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

def close(self):
self.closed = True

with patch.object(BaseCohere, "chat_stream", lambda *a, **kw: _MockSyncStream()):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use real Cohere streams for the cleanup regression coverage

For normal stream attribute/cleanup behavior, this replaces the provider with hand-written stand-ins, contrary to AGENTS.md's instruction to use real library classes and drive them with cassettes. The sync stand-in isn't even an iterator (__next__ is missing), yet the test passes because no chunks are read, so it cannot establish that closing a real Cohere stream preserves iteration and finalizes tracing. Extend the existing cassette-backed streaming tests to cover partial consumption and explicit closure instead.

AGENTS.md reference: AGENTS.md:L34-L35

Useful? React with 👍 / 👎.

Comment on lines +950 to +951
finally:
self._finish()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Record cleanup failures before finalizing stream spans

When the underlying aclose() raises—for example, when the caller is cancelled while awaiting transport cleanup—this finally path calls _finish() without the exception. _finish() permanently marks the tracker finished and emits a span without an error, so even a surrounding __aexit__ cannot record the cancellation afterward. The newly added sync close() methods have the same issue for cleanup errors. Handle cleanup exceptions and cancellation by finalizing with error=error and re-raising; use the error-free finish only on successful cleanup.

Useful? React with 👍 / 👎.

Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
Drive the checked-in cassette recording instead of a hand-written stand-in:
consume a real stream up to its first text event, close it, and assert the
span is finished from the chunks read so far. The same test fails on main,
where _TracedChatStream has no close().
@feiiiiii5

Copy link
Copy Markdown
Contributor Author

Thanks — both points checked against the current head.

P1 (real streams for cleanup coverage). The normal-path coverage now runs on the checked-in recordings: test_wrap_cohere_chat_stream_v1_sync closes a real stream after exhausting it, the v2 context-manager test stops after the first event, and this push adds test_wrap_cohere_chat_stream_v1_sync_close_finalizes_partially_read_stream, which reads a real stream up to its first text-generation event, calls close(), and asserts the span is finished from the chunks read so far. That test fails on main (_TracedChatStream has no close()).

The iterators that remain hand-built only inject error paths a recording cannot provoke — close() raising and CancelledError raised from aclose()/the request — which AGENTS.md §6 explicitly allows as a narrow monkeypatch. The sync one is a real iterator (__iter__/__next__/close).

P2 (cleanup failures swallowed). The finally path is gone: close()/aclose() now catch the cleanup error, call _finish(error=error) and re-raise, so the span records the failure instead of closing clean. test_wrap_cohere_chat_stream_close_error_records_error and test_wrap_cohere_async_chat_stream_aclose_error_records_error pin this and both fail on main.

Verified with BRAINTRUST_TEST_PACKAGE_VERSION=latest pytest py/src/braintrust/integrations/cohere/test_cohere.py --vcr-record=none: 24 passed. ruff format/ruff check clean.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0f4d2fa8f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment on lines +700 to 701
except (Exception, asyncio.CancelledError) as error:
_log_error_and_end_span(span, error)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve a nonempty diagnostic for ordinary task cancellation

When a request is cancelled with Task.cancel() or by asyncio.wait_for(), the resulting CancelledError normally has no message. _log_error_and_end_span passes that exception directly to span.log, whose bt_safe_deep_copy serialization falls back to str(error). An isolated reproduction of this path exports {"error": ""} and ends the span, losing any indication that cancellation caused the termination. The newly caught streaming and cleanup cancellations have the same problem; the added tests miss it because every injected cancellation supplies a message. Log a formatted exception or fall back to its type name, and cover message-less cancellation.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 06f991e. _log_error_and_end_span now falls back to type(error).__name__ when str(error) is empty, so a bare CancelledError from Task.cancel() exports error: "CancelledError" instead of "". The change is in the shared helper, so the streaming and cleanup cancellation paths you flagged are covered by the same fix.

test_wrap_cohere_async_chat_message_less_cancellation_records_error raises asyncio.CancelledError with no message and asserts the exported error is "CancelledError"; it fails without the fallback.

Verified: pytest src/braintrust/integrations/test_utils.py src/braintrust/integrations/cohere/test_cohere.py --vcr-record=none → 78 passed.

A CancelledError raised by Task.cancel() has an empty str(), so the span
exported error: "" and lost what ended it. Fall back to the type name and
cover the message-less cancellation in the cohere stream tests.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06f991e9bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread py/src/braintrust/integrations/utils.py Outdated
``error`` field. Fall back to the type name so the span still records what
ended it.
"""
span.log(error=error if str(error) else type(error).__name__)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard exception stringification before probing for an empty message

When a provider or custom Transformers pipeline raises an exception whose __str__ fails, this new probe throws before either span.log() or span.end(). Previously, the boundary-safe serializer contained that failure through _safe_str; callers now receive the rendering error instead of the original exception and leave the span unfinished. Comparing this path with the parent commit reproduces the regression using RuntimeError with an unprintable message object. Guard the conversion itself and fall back to the exception type when conversion fails, not just when it returns an empty string.

AGENTS.md reference: AGENTS.md:L136-L136

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - fixed in 2b7f7ec.

The probe is now contained: str(error) is wrapped, and when it raises, the empty result takes the same path as a message-less exception, so span.log(error=type(error).__name__) still runs and span.end() is always reached. Passing the exception object through is unchanged for the normal case, so _safe_str in the serializer still sees it.

test_log_error_and_end_span_survives_an_unprintable_error uses an exception whose __str__ raises and asserts the span is logged and ended with the type name; it fails on the parent commit. cohere + utils suites: 79 passed with --vcr-record=none.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b7f7ecb7a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

assert spans[0]["metrics"]["end"] >= spans[0]["metrics"]["start"]


def test_wrap_cohere_async_chat_message_less_cancellation_records_error(memory_logger):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Parameterize the duplicate async cancellation cases

This new case duplicates test_wrap_cohere_async_chat_cancellation_records_error: the client setup, patch, cancellation propagation, span-count check, and end-time check are identical; only the exception message and expected diagnostic differ. AGENTS.md mandates the integration skill's review checks, whose “Consolidate before handoff” rule requires merging input-only variants into a parametrized test. Parameterize the cancellation exception and expected diagnostic, retaining the message-less regression as a parameter, so the shared setup and finalization assertions cannot drift between copies.

AGENTS.md reference: AGENTS.md:L136-L136

Useful? React with 👍 / 👎.



def test_wrap_cohere_async_chat_cancellation_records_error(memory_logger):
from unittest.mock import patch

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets avoid mocks/fakes! If we need to we should record new vcr cassettes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on avoiding hand-written fakes where a cassette can carry the behavior — the two stream-consumption tests already reuse the checked-in VCR cassette (test_wrap_cohere_chat_stream_v1_sync), and the subscriptions test uses a real client, so there is no stand-in client on those paths.

The remaining four cases cover behavior a cassette cannot express: a task cancelled mid-chat()/chat_stream(), and a stream whose close() raises. VCR replays recorded HTTP responses; there is no response to record when the caller raises CancelledError, and a failing close() is local cleanup rather than traffic. I used a narrow patch.object on the Cohere base client so the wrapper's span-finalization code still runs — the instrumentation itself is not replaced.

I am happy to change the approach if you have a preferred pattern, e.g. cancelling a real VCR-backed client task at a specific await point, or using a fault-injecting transport fixture. Which do you prefer?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants