Repository navigation
feat: --thinking-budget caps reasoning tokens per request - #226
Merged
Merged
Conversation
Adds --thinking-budget N (default off) and a per-request thinking_budget field. After N reasoning tokens the model's own </think> is forced and it answers; 0 closes the block immediately. Built on mlx-swift-lm's ThinkingBudgetProcessor with an immediate transition, since Qwen3.5 declares none of its own. A budget the request cannot honour (max_tokens too small for the budget plus the answer) is an OpenAI-shaped 400 before any prefill. It applies to the plain, hybrid-cache and sliding-window paths; speculative, MTP and DFlash decoding log a warning and run without it. Also fixes the non-streaming response keeping a leading </think> in the content when the reasoning was empty, which a zero budget hits every time. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Member
|
Thanks @CodeAndCanvas728 — this is a clean integration of the existing
Small, non-blocking notes (no action needed for this PR):
One thing I did not verify: an end-to-end run with a real reasoning model (e.g. Qwen3 with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A hard cap on how many reasoning tokens a request may spend.
--thinking-budget Nsets a server default (off unless given); a per-requestthinking_budgetfield overrides it.</think>is forced and it answers.0closes the block immediately.400(invalid_thinking_budget) before any prefill, e.g.max_tokenstoo small to hold the budget plus the answer.It uses
ThinkingBudgetProcessor/GenerationComponents.applyingThinkingBudgetthat mlx-swift-lm already ships. The server never used it. Qwen3.5 declares no validated hard-budget transition, so the server passes.immediate(close with the model's own end delimiter, nothing extra). That makes it an explicit opt-in.Why
Agent clients that run a thinking model sometimes get a runaway reasoning block, which on a memory-bound machine is minutes of wall time. A client-side nudge only works for clients that implement one; this caps it server-side.
Scope
<think>; on a prompt-cache hit that tail is the uncached suffix, which still ends with the assistant turn header.</think>incontentwhen the reasoning was empty (the handler only used the split when reasoning was non-empty). A zero budget hits this every time, so it is fixed here.Testing
ThinkingBudgetTests): when a budget applies and when it does not, too-smallmax_tokens, zero and negative budgets, the 400 body shape, flag and request-field parsing, and the empty-reasoning contract. FullSwiftLMTests: 308 pass on the rebased branch.--stream-experts --no-vision --thinking --prefill-size 2048, greedy, on the commit before rebasing onto currentmain(the server change is identical; the rebase only moved the dependency pins):--thinking-budget 64on a prompt that reasons at length: reasoning stopped at about 48 tokens, then the model answered.thinking_budget: 32: exactly 32 reasoning chunks, then content.thinking_budget: 0: no reasoning, the answer starts immediately, no stray</think>in the content.max_tokens: 50with a budget of 64:400 invalid_thinking_budget("requires maxTokens >= 69 ...").