Python: fix Anthropic streaming double-counting token usage#7162
Open
he-yufeng wants to merge 1 commit into
Open
Python: fix Anthropic streaming double-counting token usage#7162he-yufeng wants to merge 1 commit into
he-yufeng wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes incorrect token accounting for Anthropic streaming responses in the Python Anthropic package by converting Anthropic’s cumulative usage snapshots into incremental usage updates that can be safely summed by ChatResponse.from_updates, aligning streaming totals with the non-streaming path.
Changes:
- Thread a per-stream
UsageDetailsaccumulator through the streaming generator to emit usage increments instead of cumulative snapshots. - Update
_process_stream_eventto optionally accept the per-stream accumulator and apply_incremental_usageto bothmessage_startandmessage_deltausage. - Add regression tests covering both the basic cumulative-output case and the server-tool cumulative-input case.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/packages/anthropic/agent_framework_anthropic/_chat_client.py | Adds per-stream usage reconciliation (_incremental_usage) and threads an accumulator through streaming event processing to prevent double-counting. |
| python/packages/anthropic/tests/test_anthropic_client.py | Adds regression tests ensuring streaming usage totals match Anthropic-reported cumulative totals for both output-only and input+output delta usage. |
Comment on lines
+1188
to
+1195
| emitted_counts = cast("dict[str, int]", emitted) | ||
| delta: dict[str, int] = {} | ||
| for key, value in cast("dict[str, int | None]", cumulative).items(): | ||
| if value is None: | ||
| continue | ||
| delta[key] = value - emitted_counts.get(key, 0) | ||
| emitted_counts[key] = value | ||
| return cast("UsageDetails", delta) |
eavanvalkenburg
left a comment
eavanvalkenburg
left a comment
Member
There was a problem hiding this comment.
overall this is fine, but it is redoing stuff already covered in the core library
| return usage_details | ||
|
|
||
| @staticmethod | ||
| def _incremental_usage(cumulative: UsageDetails, emitted: UsageDetails | None) -> UsageDetails: |
Member
There was a problem hiding this comment.
we have a built-in helper for this: _types.add_usage_details
Contributor
Python Test Coverage Report •
Python Unit Test Overview
|
||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation & Context
On the Anthropic streaming path, usage tokens are double-counted.
_process_stream_eventemits a usageContenton bothmessage_startandmessage_delta, andChatResponse.from_updatessums every usageContentit sees. But Anthropic'smessage_deltausage is a cumulative total for the message, not a per-delta increment (from their streaming docs: "The token counts shown in the usage field of the message_delta event are cumulative"). Summing themessage_startseed onto the cumulativemessage_deltatotal inflatesusage_details:output_token_countlands at 26 when the API reported 25 (the seed isoutput_tokens: 1), and for server-tool turns that also report cumulative input onmessage_delta, the prompt tokens get double-counted too.The streaming total should match what the non-streaming path produces.
Description & Review Guide
from_updatesexpects each update's usage to be an increment it can sum._process_stream_eventnow takes a per-streamemitted_usageaccumulator and emits the increment of each snapshot over what has already been emitted, so the summation reconstructs the final cumulative usage instead of inflating it. The accumulator lives in the per-request_stream()generator rather than on the client, so concurrent streams don't interfere, and when no accumulator is passed (a direct single-event call) the snapshot is returned unchanged, so existing callers and tests are untouched.response.usage_detailsafter a streaming Anthropic call now equals the cumulative totals the API reported, matching the non-streaming path. This fixes both the unconditional output-seed inflation and the server-tool input double-count._incremental_usage, and thatmessage_start's input-side counts are preserved whenmessage_deltacarries onlyoutput_tokens(the basic case).Two regression tests cover the basic case (output 25 / input 10, not 26) and the server-tool case where
message_deltaalso reports cumulative input (output 25 / input 12, not doubled).Related Issue
Fixes #7143
Contribution Checklist