Skip to content

feat: --thinking-budget caps reasoning tokens per request - #226

Merged
solderzzc merged 1 commit into
SharpAI:mainfrom
CodeAndCanvas728:feat/thinking-budget
Oct 11, 2026
Merged

solderzzc merged 1 commit into
SharpAI:mainfrom
CodeAndCanvas728:feat/thinking-budget

Conversation

@CodeAndCanvas728

Copy link
Copy Markdown
Contributor

What

A hard cap on how many reasoning tokens a request may spend.

  • --thinking-budget N sets a server default (off unless given); a per-request thinking_budget field overrides it.
  • After N tokens inside the thinking block the model's own </think> is forced and it answers. 0 closes the block immediately.
  • Applies only when thinking is on and the model declares a reasoning protocol. Otherwise the request is untouched.
  • A budget the request cannot honour is an OpenAI-shaped 400 (invalid_thinking_budget) before any prefill, e.g. max_tokens too small to hold the budget plus the answer.

It uses ThinkingBudgetProcessor / GenerationComponents.applyingThinkingBudget that mlx-swift-lm already ships. The server never used it. Qwen3.5 declares no validated hard-budget transition, so the server passes .immediate (close with the model's own end delimiter, nothing extra). That makes it an explicit opt-in.

Why

Agent clients that run a thinking model sometimes get a runaway reasoning block, which on a memory-bound machine is minutes of wall time. A client-side nudge only works for clients that implement one; this caps it server-side.

Scope

  • Wired into the plain, hybrid-prompt-cache and sliding-window generate paths. Speculative, MTP and DFlash decoding log a warning and run without the cap.
  • The processor reads the prompt tail to see that the template pre-opened <think>; on a prompt-cache hit that tail is the uncached suffix, which still ends with the assistant turn header.
  • Small related fix: the non-streaming handler kept a leading </think> in content when the reasoning was empty (the handler only used the split when reasoning was non-empty). A zero budget hits this every time, so it is fixed here.

Testing

  • 14 new unit tests (ThinkingBudgetTests): when a budget applies and when it does not, too-small max_tokens, zero and negative budgets, the 400 body shape, flag and request-field parsing, and the empty-reasoning contract. Full SwiftLMTests: 308 pass on the rebased branch.
  • Live, 16 GB M2 Air, 35B-A3B hybrid MoE (mixed 4/5/8-bit affine), --stream-experts --no-vision --thinking --prefill-size 2048, greedy, on the commit before rebasing onto current main (the server change is identical; the rebase only moved the dependency pins):
    • --thinking-budget 64 on a prompt that reasons at length: reasoning stopped at about 48 tokens, then the model answered.
    • Streaming with thinking_budget: 32: exactly 32 reasoning chunks, then content.
    • thinking_budget: 0: no reasoning, the answer starts immediately, no stray </think> in the content.
    • max_tokens: 50 with a budget of 64: 400 invalid_thinking_budget ("requires maxTokens >= 69 ...").

Adds --thinking-budget N (default off) and a per-request thinking_budget field.
After N reasoning tokens the model's own </think> is forced and it answers; 0
closes the block immediately. Built on mlx-swift-lm's ThinkingBudgetProcessor
with an immediate transition, since Qwen3.5 declares none of its own.

A budget the request cannot honour (max_tokens too small for the budget plus
the answer) is an OpenAI-shaped 400 before any prefill. It applies to the plain,
hybrid-cache and sliding-window paths; speculative, MTP and DFlash decoding log
a warning and run without it.

Also fixes the non-streaming response keeping a leading </think> in the content
when the reasoning was empty, which a zero budget hits every time.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@solderzzc

Copy link
Copy Markdown
Member

Thanks @CodeAndCanvas728 — this is a clean integration of the existing ThinkingBudgetProcessor. I read the diff and the processor/validator code in mlx-swift-lm and checked these points:

  • The invalid-budget 400 happens before promptCache.restore, releases the slot and settles stats, so a rejected request leaves no cache or slot state behind.
  • Validation runs against the final tokenLimit, so a budget that doesn't fit max_tokens returns a 400 invalid_thinking_budget instead of being silently truncated.
  • A fresh processor is built per generation, so no state leaks between requests.
  • Negative --thinking-budget is rejected at startup, and the 14 unit tests cover the validation paths.

Small, non-blocking notes (no action needed for this PR):

  1. With a global --thinking-budget, clients that send a small max_tokens will get a 400. That is the right behavior, but it deserves a line in the README.
  2. The non-streaming change from if let extracted to unconditional assignment should be a no-op when there are no think tags; a dedicated test would pin that down.
  3. The "budget not applied" warning for speculative/MTP/DFlash only appears in the server log, so clients can't tell the budget was ignored.

One thing I did not verify: an end-to-end run with a real reasoning model (e.g. Qwen3 with thinking_budget: 0 and 64). The tests here are unit-level only. Merging now.

@solderzzc
solderzzc merged commit 7e219b4 into SharpAI:main Oct 11, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants