Skip to content

fix: accept stop as a string, and answer a request that does not decode with 400 - #223

Merged
solderzzc merged 3 commits into
SharpAI:mainfrom
HamedShams:fix/stop-string-and-400
Oct 10, 2026
Merged

solderzzc merged 3 commits into
SharpAI:mainfrom
HamedShams:fix/stop-string-and-400

Conversation

@HamedShams

Copy link
Copy Markdown
Contributor

Closes #221.

What

Two request-handling gaps, both on /v1/chat/completions and /v1/completions:

  1. stop accepts a bare string. OpenAI documents stop as "string / array / null". Both request types decoded only the array, so "stop": "\n" (what several SDKs and llama.cpp-style clients send) failed to decode. StopSequences now takes either form; the array form and null behave exactly as before.
  2. A body that does not decode is a 400, not a 500. The decode failure was answered as 500 server_error / internal_error. It is now 400 invalid_request_error with the offending param named (stop, messages, messages.0.role, …), the shape OpenAI uses and clients key on.

Only DecodingError from the request-body decode is mapped; everything else on these routes still returns 500 as today.

Why

Before, on main:

POST /v1/chat/completions {"messages":[…],"stop":"three"}
→ 500 {"error":{"type":"server_error","code":"internal_error","message":"DecodingError.typeMismatch: Expected value of type Array<Any>. Path: stop. …"}}

The OpenAI SDKs treat a 400 as final but retry a 5xx (openai-python: two retries by default), so a client sending a string stop paid three round trips to learn nothing, and the error told it the server was at fault.

Verified

  • RequestValidationTests (new): stop as string / array / null / absent on both request types, a non-string value still rejected, and the 400 payload for a missing key, a type mismatch, a nested path and malformed JSON. The string-stop cases are red on main.
  • tests/test-contract.sh gained two assertions: a bare stop string is accepted and honoured, and "stop": 42 answers 400 invalid_request_error with param: "stop". Both red on main.
  • swift test --filter SwiftLMTests green locally (M1 Max, Xcode 27 / Swift 6.4); contract script run against mlx-community/Qwen3-0.6B-4bit.

…code with 400

OpenAI's `stop` is a string, an array, or null; both request types decoded
only the array form, so a bare string failed to decode. StopSequences now
takes either form, and null or absent still means no client stops.

That decode failure was answered as 500 server_error, which the OpenAI SDKs
retry as a transient error. Only a DecodingError from the request-body
decode now maps to 400 invalid_request_error with `param` taken from the
coding path; everything else on these routes still returns 500.

Closes SharpAI#221.
Both SharpAI#220 and this branch added a block before the results footer of
tests/test-contract.sh. The seed assertions keep their numbers (9, 9b);
the request-shape assertions move to 10 and 10b.
@HamedShams

HamedShams commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor Author

The one red job, dflash-speculative-decoding, is unrelated to this change and flaky: on main's own last run (748c4bf) it failed the same way on attempts 1 and 2 and passed on the third; on this PR's first commit it failed once and passed on the retry; on the merge commit all three attempts hit it. The other 14 jobs, including contract, are green.

The uploaded dflash-test-logs artifact shows what happens. After Test 3's answer the DFlash loop keeps generating past <|im_end|> (<|endoftext|><|im_start|>user…), because the server never passes it any stop token IDs and nothing cancels it when the SSE stream ends on the text-level stop. The slot is released at that point, so Test 5's first request starts a second DFlash generation on the same model, the two interleave (Cycle 7… Cycle 1… Cycle 8… Cycle 2… in the log), and MLX aborts with [eval] Attempting to eval an array without a primitive. Whether that overlap happens depends on how much of max_tokens is left when the stop fires, hence the flakiness.

That is a separate bug, written up with the line references in #224.
Could you plz re-run the job here? @solderzzc

@HamedShams

Copy link
Copy Markdown
Contributor Author

Fix for #224 is up as #225. Once that is in, I'll merge main into this branch so the dflash-speculative-decoding job runs here with the fix; a plain re-run of the existing job would reuse the old merge snapshot and stay a coin flip, same as on main.

@solderzzc

Copy link
Copy Markdown
Member

Thank you for this, and for keeping the branch current so quickly. The decode-to-400 mapping and the StopSequences change both look right to us, and the tests cover the cases we care about.

A note on what we did to the branch: the last CI run on it (fedf764) failed in dflash-speculative-decoding, which is the flake from #224 and unrelated to this change. #225 fixes it and is now on main, so we pushed one plain merge commit (c9df438, a normal push, no force, no history rewritten) that merges main into the branch, to get a CI run that includes the fix. Nothing else was changed. Please pull before pushing anything further.

We will merge once that run is green.

@solderzzc
solderzzc merged commit 562babb into SharpAI:main Oct 10, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stop as a plain string is rejected, and malformed requests get a 500 instead of a 400

2 participants