feat(Decision): use AI SDK decisions, configure decision models in ai.decisionModel (#5742)

* feat(Decision): use AI SDK experimental_decide

Replace hand-written fetch calls to OpenRouter/TypeSafe decision endpoints
with AI SDK experimental_decide via @openrouter/ai-sdk-provider and
@ai-sdk/typesafe-ai. Bumps ai to v7, which experimental_decide requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(Decision): use OpenRouter Decisions API only, document custom decisions

Drop the decision provider option and @ai-sdk/typesafe-ai: all decision
models, including TypeSafe Jev, are served by the OpenRouter Decisions API.
Document how decisions use AI SDK experimental_decide and how to ask
choice/score questions from a custom helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(Decision): accept AI SDK decision models in config

ai.decisionModel.model and visualModel accept an OpenRouter model ID or an
AI SDK decision model, e.g. createOpenRouter().evaluationModel(id).
Document configuring decision models in ai.md; drop custom decisions section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(Decision): configure decision models only via OpenRouter provider

ai.decisionModel.model and visualModel must be AI SDK decision models
created with createOpenRouter().evaluationModel(). Drop model ID strings,
default models and apiKey option. @openrouter/ai-sdk-provider becomes a
dev dependency; users install it like other AI SDK providers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(deps): pin ai to 7.0.132

ai 7.0.133 wraps decision state into SDK parts that
@openrouter/ai-sdk-provider 3.1.0 forwards untranslated, so OpenRouter
receives wrapped state and screenshots lose the image_url format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Author: 3 people

docs/agents.md

@@ -186,6 +186,8 @@ I.decide([
 
 The agent runs the statement on the live page like any other command and keeps it only if it passes. The test then checks it on every run.
 
+To let the agent use decisions, [configure a decision model](/ai#configure-decision-models) and enable the Decision helper.
+
 Decision models like [Jev](https://openrouter.ai/typesafe/jev-1.13) are built for this. They answer in a fraction of a second, cost a fraction of a cent per request, and return a probability instead of free text. That makes them fast and cheap enough to run on every CI build, and predictable enough to keep in a test.
 
 ## Skills bundle

docs/ai.md

@@ -353,7 +353,47 @@ I.decideVisually('sidebar is shown')
 
 It uses a [decision model](https://openrouter.ai/models?output_modalities=decisions) like [Jev](https://openrouter.ai/typesafe/jev-1.13) instead of a chat model. A decision model reads the page and returns the probability that a statement is true. It is fast, costs a fraction of a cent per request, and gives a probability instead of free text, so a step passes or fails on a confidence threshold you set.
 
-Configure it in `ai.decisionModel`. Decisions call the decisions API directly, so they don't need `ai.model` or the `--ai` flag. See [Decision Assertions](/assertions#decision-assertions) for setup and usage.
+Decision models are separate from `ai.model`: decisions don't need `ai.model` or the `--ai` flag. CodeceptJS calls them with AI SDK [`experimental_decide`](https://ai-sdk.dev/docs/ai-sdk-core/decisions).
+
+### Configure Decision Models
+
+Decision models are served by the [OpenRouter Decisions API](https://openrouter.ai/models?output_modalities=decisions). Install the [OpenRouter provider](https://ai-sdk.dev/providers/community-providers/openrouter) `@openrouter/ai-sdk-provider`, set `OPENROUTER_API_KEY`, and create the decision model with `evaluationModel()` in `codecept.conf.js`:
+
+```js
+import { createOpenRouter } from '@openrouter/ai-sdk-provider'
+
+const openrouter = createOpenRouter({
+  apiKey: process.env.OPENROUTER_API_KEY,
+})
+
+export default {
+  ai: {
+    decisionModel: {
+      model: openrouter.evaluationModel('typesafe/jev-1.13'),
+      confidence: 0.8,
+    },
+  },
+  helpers: {
+    Playwright: { url: 'http://localhost' },
+    Decision: {},
+  },
+}
+```
+
+`model` is used by `I.decide`.
+
+### Visual Decision Model
+
+`I.decideVisually` sends a screenshot, so it needs a separate decision model with image input. Set it as `visualModel`:
+
+```js
+decisionModel: {
+  model: openrouter.evaluationModel('typesafe/jev-1.13'),
+  visualModel: openrouter.evaluationModel('cloudflare/clef'),
+},
+```
+
+See [Decision Assertions](/assertions#decision-assertions) for all options and usage.
 
 ## Advanced Configuration
 

docs/assertions.md

@@ -407,15 +407,9 @@ A [decision model](https://openrouter.ai/models?output_modalities=decisions), li
 - **Cost-efficient.** A request costs a fraction of a cent. You can run decision assertions in every CI build.
 - **Reliable.** The answer is a probability, not free text. There is nothing to parse, and you choose how confident the model must be for the step to pass.
 
-Set `OPENROUTER_API_KEY`, configure the decision model in the `ai` section, and enable the helper next to your browser helper:
+Configure the decision model in `ai.decisionModel` as described in [Configure Decision Models](/ai#configure-decision-models), and enable the helper next to your browser helper:
 
 ```js
-ai: {
-  decisionModel: {
-    model: 'typesafe/jev-1.13',
-    confidence: 0.7,
-  },
-},
 helpers: {
   Playwright: { url: 'http://localhost' },
   Decision: {},
@@ -426,15 +420,13 @@ helpers: {
 
 | Option | Default | Description |
 |---|---|---|
-| `provider` | `openrouter` | `openrouter` reads `OPENROUTER_API_KEY`, `typesafe` reads `TYPESAFE_API_KEY` |
-| `apiKey` | | API key, overrides the environment variable |
-| `model` | `typesafe/jev-1.13` | model for `I.decide` |
-| `visualModel` | `cloudflare/clef` | model with image input for `I.decideVisually`, OpenRouter only |
+| `model` | | decision model for `I.decide` |
+| `visualModel` | | decision model with image input for `I.decideVisually` |
 | `confidence` | `0.7` | minimal probability for a statement to pass |
 | `timeout` | `15000` | request timeout in ms |
 | `maxLength` | `12000` | maximal length of ARIA snapshot or HTML sent to the model |
 
-Decisions don't need the `--ai` flag, and `ai.model` is not required.
+Any model from the [OpenRouter decision models](https://openrouter.ai/models?output_modalities=decisions) list can be used with `openrouter.evaluationModel()`. Decisions don't need the `--ai` flag, and `ai.model` is not required.
 
 Then assert statements about the current page:
 
@@ -457,7 +449,7 @@ I.decideVisually('sidebar is shown')
 expected page to satisfy "success message is shown" (12%) with confidence of 70%
 ```
 
-`I.decideVisually` also sends a screenshot, so it needs a model with image input. It uses `visualModel`, which is [Clef](https://openrouter.ai/cloudflare/clef) by default. Visual decisions are experimental.
+`I.decideVisually` also sends a screenshot, so it needs a [visual decision model](/ai#visual-decision-model) with image input, like [Clef](https://openrouter.ai/cloudflare/clef), set as `visualModel`. Visual decisions are experimental.
 
 To turn decisions off without removing the helper, set its `mode`:
 

docs/helpers/Decision.md

@@ -36,31 +36,44 @@ This helper must be enabled together with a web helper (Playwright, Puppeteer, W
 
 ## Configuration
 
-Decision model is configured in the `ai.decisionModel` section of the config:
+Decision model is created with the OpenRouter provider from `@openrouter/ai-sdk-provider`
+and configured in the `ai.decisionModel` section of the config:
 
 ```js
-ai: {
-  decisionModel: {
-    provider: 'openrouter',
-    model: 'typesafe/jev-1.13',
-    visualModel: 'cloudflare/clef',
-    confidence: 0.7,
+import { createOpenRouter } from '@openrouter/ai-sdk-provider'
+
+const openrouter = createOpenRouter({
+  apiKey: process.env.OPENROUTER_API_KEY,
+})
+
+export default {
+  ai: {
+    decisionModel: {
+      model: openrouter.evaluationModel('typesafe/jev-1.13'),
+      confidence: 0.8,
+    },
+  },
+  helpers: {
+    Playwright: { url: 'http://localhost' },
+    Decision: {},
   },
-},
-helpers: {
-  Playwright: { url: 'http://localhost', browser: 'chromium' },
-  Decision: {},
 }
 ```
 
-*   `provider` (default: `openrouter`) - decision API to call: `openrouter` (reads `OPENROUTER_API_KEY`) or `typesafe` (reads `TYPESAFE_API_KEY`).
-*   `apiKey` (optional) - API key, overrides the environment variable.
-*   `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`. Use `jev-latest` with the `typesafe` provider.
-*   `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`. Available on OpenRouter only.
+*   `model` - decision model used by `decide`.
 *   `confidence` (default: `0.7`) - minimal probability, between 0 and 1, for a statement to pass.
 *   `timeout` (default: `15000`) - request timeout in ms.
 *   `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model.
 
+`decideVisually` uses a separate decision model with image input, configured as `visualModel`:
+
+```js
+decisionModel: {
+  model: openrouter.evaluationModel('typesafe/jev-1.13'),
+  visualModel: openrouter.evaluationModel('cloudflare/clef'),
+},
+```
+
 The helper has one option:
 
 *   `mode` (default: `assert`) - how decisions are executed:

lib/ai.js

@@ -3,7 +3,7 @@ const debug = debugModule('codeceptjs:ai')
 import output from './output.js'
 import event from './event.js'
 import { removeNonInteractiveElements, minifyHtml, splitByChunks } from './html.js'
-import { generateText } from 'ai'
+import { generateText, experimental_decide as decide, APICallError } from 'ai'
 import { fileURLToPath } from 'url'
 import path from 'path'
 import { fileExists, resolveImportModulePath } from './utils.js'
@@ -271,15 +271,7 @@ function parseCodeBlocks(response) {
   return modifiedSnippets.filter(snippet => !!snippet)
 }
 
-const DECISION_ENDPOINTS = {
-  openrouter: { url: 'https://openrouter.ai/api/alpha/decisions', keyName: 'OPENROUTER_API_KEY' },
-  typesafe: { url: 'https://api.typesafe.ai/v1/systemone', keyName: 'TYPESAFE_API_KEY' },
-}
-
 const defaultDecisionConfig = {
-  provider: 'openrouter',
-  model: 'typesafe/jev-1.13',
-  visualModel: 'cloudflare/clef',
   confidence: 0.7,
   timeout: 15000,
   maxLength: 12000,
@@ -291,97 +283,68 @@ class DecisionAI {
   constructor(config = {}) {
     this.config = { ...defaultDecisionConfig, ...config }
 
-    const { provider, confidence } = this.config
-    this.endpoint = DECISION_ENDPOINTS[provider]
-    if (!this.endpoint) throw new Error(`Unknown decision provider "${provider}" in ai.decisionModel, use one of: ${Object.keys(DECISION_ENDPOINTS).join(', ')}`)
+    const { confidence } = this.config
     if (!(confidence > 0 && confidence < 1)) throw new Error(`ai.decisionModel.confidence must be between 0 and 1, got ${confidence}`)
-
-    this.fetchImpl = fetch
   }
 
-  checkModel() {
-    if (this.config.apiKey || process.env[this.endpoint.keyName]) return
-
-    const noKeyErrorMessage = `
-     No API key is set for decision model.
+  checkModel(option) {
+    if (this.config[option]) return
 
-     [!] Set ${this.endpoint.keyName} environment variable or apiKey in ai.decisionModel config.
-
-     Example (connect to OpenRouter, default):
-
-     export OPENROUTER_API_KEY=sk-or-...
-
-     ai: {
-       decisionModel: {
-         model: 'typesafe/jev-1.13',
-         confidence: 0.7,
-       }
-     }
+    const noModelErrorMessage = `
+     No decision model is set in ai.decisionModel.${option} config.
 
-     Get a key at https://openrouter.ai/settings/keys
+     [!] Configure decision model with OpenRouter provider:
 
-     Example (connect to TypeSafe):
+     import { createOpenRouter } from '@openrouter/ai-sdk-provider'
 
-     export TYPESAFE_API_KEY=...
+     const openrouter = createOpenRouter({
+       apiKey: process.env.OPENROUTER_API_KEY,
+     })
 
      ai: {
        decisionModel: {
-         provider: 'typesafe',
-         model: 'jev-latest',
+         model: openrouter.evaluationModel('typesafe/jev-1.13'),
+         visualModel: openrouter.evaluationModel('cloudflare/clef'),
        }
      }
 
      See https://openrouter.ai/models?output_modalities=decisions for all decision models.
     `.trim()
 
-    throw new Error(noKeyErrorMessage)
+    throw new Error(noModelErrorMessage)
   }
 
-  async decide(model, state, statements) {
-    this.checkModel()
+  async decide(option, state, statements) {
+    this.checkModel(option)
+    const model = this.config[option]
 
-    const apiKey = this.config.apiKey || process.env[this.endpoint.keyName]
-    const questions = Object.fromEntries(statements.map((statement, i) => [`q${i}`, { type: 'noul', instructions: statement }]))
-    debug('Decision request', model, statements)
+    const questions = Object.fromEntries(statements.map((statement, i) => [`q${i}`, { type: 'boolean', instructions: statement }]))
+    debug('Decision request', model.modelId, statements)
 
     const controller = new AbortController()
     const timer = setTimeout(() => controller.abort(), this.config.timeout)
 
     let result
     try {
-      let response
-      try {
-        response = await this.fetchImpl(this.endpoint.url, {
-          method: 'POST',
-          headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
-          body: JSON.stringify({ model, state, questions }),
-          signal: controller.signal,
-        })
-      } catch (err) {
-        throw new DecisionConnectionError(`Decision model ${model} request failed: ${err.message}`)
-      }
-
-      if (!response.ok) {
-        const body = await response.text().catch(() => '')
-        throw new Error(`Decision model ${model} responded with ${response.status}: ${body}`)
-      }
-
-      result = await response.json()
+      result = await decide({
+        model,
+        state,
+        questions,
+        maxRetries: 0,
+        abortSignal: controller.signal,
+      })
     } catch (err) {
-      if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model} did not respond in ${this.config.timeout}ms`)
+      if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model.modelId} did not respond in ${this.config.timeout}ms`)
+      if (APICallError.isInstance(err) && !err.statusCode) throw new DecisionConnectionError(`Decision model ${model.modelId} request failed: ${err.message}`)
+      if (APICallError.isInstance(err)) throw new Error(`Decision model ${model.modelId} responded with ${err.statusCode}: ${err.responseBody || err.message}`)
       throw err
     } finally {
       clearTimeout(timer)
     }
 
-    debug('Decision response', result?.answers, result?.usage)
+    debug('Decision response', result.answers, result.usage)
 
-    const answers = result?.answers || {}
-    return statements.map((statement, i) => {
-      const probability = answers[`q${i}`]?.noul
-      if (typeof probability !== 'number') throw new Error(`Decision model ${model} returned no answer for "${statement}"`)
-      return probability
-    })
+    return Object.keys(questions).map(id => result.answers[id].probability)
   }
 }