Skip to content

fix(anthropic): preserve streaming input and cache usage - #1498

Open
pcbeingused333 wants to merge 1 commit into
SciSharp:masterfrom
pcbeingused333:fix/anthropic-stream-usage-c3501
Open

pcbeingused333 wants to merge 1 commit into
SciSharp:masterfrom
pcbeingused333:fix/anthropic-stream-usage-c3501

Conversation

@pcbeingused333

Copy link
Copy Markdown
Contributor

Anthropic streaming replaces the saved usage with each event's usage. The initial message contains input and cache counts, while later message deltas contain cumulative output counts. Replacing the object loses the input and cache counts, and because the final message_stop event carries no usage, streamed Anthropic completions currently report 0 for every token count passed to AfterGenerated. This is a sibling of #1486; there is no linked issue.

Preserve the initial usage object and update its output count when later usage arrives. Events without usage retain the accumulated statistics. Output counts are assigned rather than summed because Anthropic reports cumulative output usage.

AnthropicStreamedUsageTests covers initial input/cache counts, text events without usage, successive output counts in both supported usage locations, the final message-stop event, an output-only stream and a stream with no usage. Run with dotnet test tests/BotSharp.Core.UnitTests/BotSharp.Core.UnitTests.csproj --filter FullyQualifiedName~AnthropicStreamedUsageTests.

This change was prepared with the assistance of an AI coding tool; the author is responsible for the submission.

@qodo-code-review

Copy link
Copy Markdown
Contributor

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant