Repository navigation
fix: include ruby_llm cache tokens in total_tokens - #712
Open
pcbeingused333 wants to merge 1 commit into
Open
pcbeingused333 wants to merge 1 commit into
pcbeingused333 wants to merge 1 commit into
Conversation
ruby_llm reports cache reads and writes apart from input. The total was input plus output, so those tokens were missing. Stamp the total after a stream merges its chunks, so a later output-only chunk does not drop cache already counted.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A ruby_llm response undercounts
usage.total_tokenswhen the prompt cache is used. ruby_llm keeps standard input apart from cache reads and writes: OpenAI and Gemini subtract those counts from the prompt total, and Anthropic already reports them outsideinput_tokens. The provider copies that split into usage, and the total then falls back to input + output, so the cached tokens never appear. A reply that read 5 cached tokens and wrote 2 reported 16 instead of 23. OpenAI's own total for a prompt of 12 with 5 cached and 4 output is 16; the provider reported 11.The total is now input + output + cache read + cache write. It is written after a stream merges its chunks, because a later chunk often carries only output and totaling that chunk alone would drop cache already counted.
input_tokens,cached_tokens, andcache_creation_tokensare unchanged.reasoning_tokensstays out of the total: ruby_llm's output already includes thinking when the provider bills it as output. Parsers that taketotal_tokensfrom OpenAI or Anthropic directly are untouched.Testing and validation
The ruby_llm response and streaming tests cover a non-stream OpenAI payload (prompt 12, cached 5, output 4, total 16), an Anthropic payload (input 12, output 4, cache read 5, cache write 2, total 23), and streams that deliver input and cache before output. Those assertions fail against the previous provider (9 failures) and pass with this change (48 runs, 138 assertions). The same files pass on ruby_llm 1.16.0 (44 runs; the 2.x-only cases skip). The rest of
test/providers/ruby_llmpasses on ruby_llm 2.2.0 (147 runs, 366 assertions, 3 skips). No request was made to a provider API.This change was prepared with the help of an AI coding assistant (Grok).