fix(providers): correct max-tokens param and add schema guidance for NVIDIA/Z.ai#5569
Conversation
…NVIDIA/Z.ai - NVIDIA NIM and Z.ai both document max_tokens for output-length control, not OpenAI's newer max_completion_tokens - the latter was silently ignored by both vLLM-served NIM models and Z.ai's GLM models - Z.ai has no json_schema response_format mode (only text/json_object), so structured-output requests now also inject the expected schema into the system prompt as best-effort guidance, since the request param alone can't enforce field names/types
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryMedium Risk Overview For Z.ai structured output (no native
Reviewed by Cursor Bugbot for commit 159a48a. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Greptile SummaryThis PR updates the NVIDIA and Z.ai provider adapters. The main changes are:
Confidence Score: 5/5This looks safe to merge.
Important Files Changed
Reviews (3): Last reviewed commit: "fix(providers): scope Z.ai schema guidan..." | Re-trigger Greptile |
Consolidated the scattered narrative // comments in nvidia/index.ts and zai/index.ts into a single TSDoc block per provider documenting the provider-specific API quirks; removed the rest where they only restated what the code already shows. Also dropped an inline comment on zai's modelPatterns field in models.ts.
|
@cursor review |
…l only - schemaGuidance now falls back to the bare responseFormat object when .schema is absent, matching the schema-or-format fallback used elsewhere in the codebase (was silently injecting nothing for callers that pass a bare JSON schema) - guidance is now only appended to the messages sent alongside an actual response_format (the immediate call, or whichever pass deferResponseFormat applies it to) instead of every turn of an active tool loop, where it wrongly told the model to return final JSON instead of continuing to call tools
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 159a48a. Configure here.
1 participant
Summary
Follow-up to #5560 addressing review findings that surfaced on #5567 (staging→main) after merge:
max_tokensfor output-length control, not OpenAI's newermax_completion_tokens— verified against each provider's own API reference. The latter was silently ignored, so user-configured output caps were never applied. Cursor caught this for NVIDIA; I found the identical bug in Z.ai while fixing it and corrected both.response_formathas nojson_schemamode (onlytext/json_object, re-verified againstdocs.z.ai), so schema/name/strict fields were silently dropped for structured-output requests. Now also injects the expected schema into the system prompt as guidance, in addition to thejson_objecthint — can't be enforced at the API level for this provider, but gives the model a real shot at matching the expected shape instead of nothing.tool_choice: 'auto'(already logged as a warning) — no code change, reasoned pushback on the review thread.Type of Change
Testing
bunx vitest run providers/models.test.ts— 18/18 passingtsc --noEmit— zero new type errorsbiome check— cleanChecklist