Skip to content

fix: include ruby_llm cache tokens in total_tokens - #712

Open
pcbeingused333 wants to merge 1 commit into
activeagents:mainfrom
pcbeingused333:fix/ruby-llm-cache-total
Open

pcbeingused333 wants to merge 1 commit into
activeagents:mainfrom
pcbeingused333:fix/ruby-llm-cache-total

Conversation

@pcbeingused333

Copy link
Copy Markdown
Contributor

A ruby_llm response undercounts usage.total_tokens when the prompt cache is used. ruby_llm keeps standard input apart from cache reads and writes: OpenAI and Gemini subtract those counts from the prompt total, and Anthropic already reports them outside input_tokens. The provider copies that split into usage, and the total then falls back to input + output, so the cached tokens never appear. A reply that read 5 cached tokens and wrote 2 reported 16 instead of 23. OpenAI's own total for a prompt of 12 with 5 cached and 4 output is 16; the provider reported 11.

The total is now input + output + cache read + cache write. It is written after a stream merges its chunks, because a later chunk often carries only output and totaling that chunk alone would drop cache already counted. input_tokens, cached_tokens, and cache_creation_tokens are unchanged. reasoning_tokens stays out of the total: ruby_llm's output already includes thinking when the provider bills it as output. Parsers that take total_tokens from OpenAI or Anthropic directly are untouched.

Testing and validation

The ruby_llm response and streaming tests cover a non-stream OpenAI payload (prompt 12, cached 5, output 4, total 16), an Anthropic payload (input 12, output 4, cache read 5, cache write 2, total 23), and streams that deliver input and cache before output. Those assertions fail against the previous provider (9 failures) and pass with this change (48 runs, 138 assertions). The same files pass on ruby_llm 1.16.0 (44 runs; the 2.x-only cases skip). The rest of test/providers/ruby_llm passes on ruby_llm 2.2.0 (147 runs, 366 assertions, 3 skips). No request was made to a provider API.

This change was prepared with the help of an AI coding assistant (Grok).

ruby_llm reports cache reads and writes apart from input. The total
was input plus output, so those tokens were missing. Stamp the total
after a stream merges its chunks, so a later output-only chunk does
not drop cache already counted.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant