ci(sdk-live): judge features by evidence, probe undeclared ones

[caiotheodoro]

Follow-up to #5506 (merged). Draft for review by its author.

Flow

flowchart TD
  Y[model YAML] --> B[baseline call]
  B -->|fails| X[fail: model broken]
  B -->|ok| D[each declared feature]
  B -->|ok| U[each undeclared feature<br/>+ removed params<br/>+ reasoning_effort if absent]
  D --> E{reply shows it worked?}
  E -->|yes| P1[pass]
  E -->|cut at token cap| W1[warn]
  E -->|no| F1[fail]
  U --> Q[probe via catalogue copy<br/>that declares the field]
  Q --> R{400/422 naming the field,<br/>or OpenRouter 404?}
  R -->|yes| P2[pass: correctly undeclared]
  R -->|no, it works| W2[warn: undeclared but working]
  B -.->|429 / 5xx| RT[retry 2s, 6s]

Loading

Why

On #5506 a declared feature passes on any 2xx, and nothing checks what the YAML leaves out. In the last 90 days, humans added features to bot PRs 233 times and removed wrong ones 41 times, and added reasoning_effort 55 times. Example: #5533 declares assistant_prefill and tool_choice on Opus 5.5, while #5524 removes both for Anthropic.

What changes

• Declared features: the reply has to show the feature worked. tool_choice forces get_time on a weather question, json_output must return bare JSON, structured_output must match the schema. A reply cut at the token cap is a warn, not a fail.
• Undeclared features and params: one probe each (removed params, plus reasoning_effort when it isn't declared).
  • The probe runs against a catalogue copy that declares the field, so the SDK forwards it as sent.
  • A 400 or 422 counts as a rejection only when the baseline call worked and the error names the field. OpenRouter's 404 "no endpoints found that support" also counts.
  • A field that works but isn't declared is a warn, never a fail.
• Retries: 429 and 5xx are retried twice (2 s, 6 s).
• Not probed: max_tokens and max_completion_tokens. The SDK rewrites one into the other, so a wrong choice already fails the baseline call. Sampling params the SDK drops for base gpt-5 are skipped too.
• Summary: gains a Warn column and an "Undeclared but working" list.

Testing

• npm run test:unit: 53/53, on simulated responses.
• tsc: only the audioForm BlobPart error that is already on the base branch.
• Not yet run against live providers.

Limits

• Worst case 14 extra calls per chat model: 8 feature probes and up to 6 param probes, capped at 64–256 output tokens (1,024 for reasoning models, 4,096 for the reasoning_effort probe).
• The harness has no credentials for microsoft-foundry, aws-bedrock, google-vertex or azure. Most human fixes are on those four, and feat(microsoft-foundry/claude-opus-5-5-fast-2): add new models [bot] #5533 is one of them.