diff --git a/DESIGN.md b/DESIGN.md index 1604e68..58f9a32 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -115,7 +115,9 @@ eval/ harness lives with the code it gates (see EVAL.md) usable input / 1M advertised / 384k output; effort levels off|low|high|xhigh|max; budget math always uses 616k. - Flash-first, pro-escalation: flash is the default executor; `--model` - pins. A session transcript is pinned to ONE model (Round 4 rule); the + pins. (Since V4.1 Flash, 2026-09-10, `deepseek-flash` is the default + and pro is no longer the stronger tier on DeepSeek's own agent + benchmarks; see README "Models". Escalation stays open as a mechanism.) A session transcript is pinned to ONE model (Round 4 rule); the `/pro` escalation mechanism — same-transcript switch vs evidence-packet handoff — is an open question, eval-gated at M4. - Usage accounting from provider `usage` only; cost computed from our own @@ -201,8 +203,8 @@ Small and stable — the schema is part of the cached prefix: - `task` tool: spawn/wait/result/cancel lifecycle (DeepSeek-TUI shape), parallel by default, results return as compressed reports. - Roles as prompt+permission presets: explorer (read-only, flash), - implementer (write, flash), reviewer (read-only, pro), tester (bash, - flash). Verified pattern: bounded sub-contexts beat one giant context. + implementer (write, flash), reviewer (read-only, pro until 2026-09-10, + flash since), tester (bash, flash). Verified pattern: bounded sub-contexts beat one giant context. - Every sub-agent gets a budget envelope: max tokens, max turns, max wall-clock. A looping child dies quietly; the parent gets a partial report. (Capability-token idea from research, minimum viable form.) diff --git a/EVAL.md b/EVAL.md index a44b797..b4e49f4 100644 --- a/EVAL.md +++ b/EVAL.md @@ -86,7 +86,7 @@ lower hit% while improving quality and total cost). ## Workflow -- `bun eval/run.ts --tasks all --adapters dsc --models deepseek-v4-flash +- `bun eval/run.ts --tasks all --adapters dsc --models deepseek-flash --n 3` -> appends runs.jsonl - `bun eval/report.ts` -> matrix like evot's README table. - CI (dsc repo): quick-tag tasks on every engine PR; full matrix diff --git a/README.md b/README.md index ac0bfa8..a571abd 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,8 @@ # deepseek-code (`dsc`) -A DeepSeek-native coding agent for the v4 flash/pro series (pro is GA -as of 2026-08-12, model version DeepSeek-V4-Pro-0813 — same -`deepseek-v4-pro` model ID). TypeScript, Bun, zero runtime -dependencies. +A DeepSeek-native coding agent. Defaults to DeepSeek-V4.1-Flash +(`deepseek-flash`, released 2026-09-10); `deepseek-v4-pro` stays +selectable. TypeScript, Bun, zero runtime dependencies. Not a Claude Code clone and not a general multi-provider harness. Being DeepSeek-only is the design: one protocol state machine (the @@ -83,7 +82,7 @@ printf 'read src/parse.js\nnow fix the regex\n/cost\n' | bun src/cli.ts Useful flags: ``` ---model deepseek-v4-flash|deepseek-v4-pro +--model deepseek-flash|deepseek-v4-pro (default deepseek-flash) --cwd DIR working directory for the run --continue interactive: resume the latest session here --save / --resume ID persist the session to SQLite (~/.dsc/) @@ -94,13 +93,33 @@ Useful flags: --verbose -p only; stream progress to stderr ``` -Both v4 models are 1M context / 384K max output and can be picked -anywhere a model is chosen (`--model`, `/model` in interactive, -`dsc job add --model`). Flash is the default executor; pro is the -stronger tier and the right pick for planning and review passes — the -built-in `reviewer` sub-agent role already runs on it. V4-Pro GA ships -with thinking on by default upstream, and reasoning tokens count -against `max_tokens`, which dsc's budgets already account for. +## Models + +| name | model | off-peak USD / 1M (hit / miss / out) | +| --- | --- | --- | +| `deepseek-flash` (default) | DeepSeek-V4.1-Flash | 0.003 / 0.15 / 0.60 | +| `deepseek-v4-pro` | DeepSeek-V4-Pro-0813 | 0.022 / 0.66 / 1.98 | + +Peak (01:00-04:00 and 06:00-10:00 UTC, Monday to Friday) costs double; +`/cost` and the turn footer price usage on the card and period in force +when they print. Both are 1M context / 384K max output and can be +picked anywhere a model is chosen (`--model`, `/model` in interactive, +`dsc job add --model`). + +Flash is the default and every built-in sub-agent role runs on it, +including `reviewer`, which ran pro until V4.1. The reason is +DeepSeek's own changelog: on the benchmarks both its 2026-09-10 (V4.1 +Flash) and 2026-08-13 (V4 Pro GA) entries report, Flash scores +Terminal Bench 2.1 90.6 vs 87.9, NL2Repo 65.4 vs 61.5, CyberGym 88.1 vs +83.3 and HLE with tools 63.9 vs 60.0; HLE without tools goes the other +way, 36.8 vs 42.7. We have not re-run our own eval on V4.1 yet. + +The retired names `deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` +are still accepted, as the API still accepts them: they are served by +V4.1 Flash and billed at the Flash price. Thinking is on by default +upstream, and reasoning tokens count against `max_tokens` (V4.1 Flash +was measured spending a whole 600-token budget on reasoning), which +dsc's budgets already account for. ## What is actually built @@ -161,7 +180,7 @@ pinned benchmark suite. Numbers decide; taste does not. See the raw rows behind them. ```bash -bun eval/run.ts --tasks all --adapters dsc --models deepseek-v4-flash --n 3 +bun eval/run.ts --tasks all --adapters dsc --models deepseek-flash --n 3 bun eval/report.ts ``` diff --git a/eval/pricing.json b/eval/pricing.json index 021438d..9bd64bd 100644 --- a/eval/pricing.json +++ b/eval/pricing.json @@ -1,9 +1,36 @@ { - "source": "thevibeworks/deepseek-docs content/en/quick_start/pricing.md (fetched 2026-08-02)", - "unit": "USD per 1M tokens, regular (off-peak) rates", - "note": "DeepSeek announced a coming peak-hour policy: 2x all items during 9:00-12:00 and 14:00-18:00 Beijing time. Not yet in effect; runs.jsonl rows record regular-rate cost only. Re-check the pricing doc before trusting absolute dollar figures.", - "models": { - "deepseek-v4-flash": { "input_hit": 0.0028, "input_miss": 0.14, "output": 0.28 }, - "deepseek-v4-pro": { "input_hit": 0.003625, "input_miss": 0.435, "output": 0.87 } - } + "source": "thevibeworks/deepseek-docs content/en/quick_start/pricing.md (re-read 2026-09-18) and content/en/updates.md", + "unit": "USD per 1M tokens. Cards from 2026-08-16 on are OFF-PEAK rates; peak (01:00-04:00 and 06:00-10:00 UTC, Mon-Fri) is 2x. runs.jsonl cost_usd is always computed on the off-peak rate of the card in force when the row was written, so rows from different hours compare.", + "tiers": { + "deepseek-flash": "flash", + "deepseek-v4-pro": "pro", + "deepseek-v4-flash": "flash", + "deepseek-v4-flash-vision-exp": "flash" + }, + "cards": [ + { + "since": "1970-01-01T00:00:00Z", + "label": "flat card published 2026-08-02, no peak", + "rates": { + "flash": { "input_hit": 0.0028, "input_miss": 0.14, "output": 0.28 }, + "pro": { "input_hit": 0.003625, "input_miss": 0.435, "output": 0.87 } + } + }, + { + "since": "2026-08-16T16:00:00Z", + "label": "V4 GA repricing (changelog 2026-08-13)", + "rates": { + "flash": { "input_hit": 0.007, "input_miss": 0.22, "output": 0.66 }, + "pro": { "input_hit": 0.022, "input_miss": 0.66, "output": 1.98 } + } + }, + { + "since": "2026-09-10T11:00:00Z", + "label": "V4.1 Flash (changelog 2026-09-10); instant inferred, see src/provider/catalog.ts", + "rates": { + "flash": { "input_hit": 0.003, "input_miss": 0.15, "output": 0.6 }, + "pro": { "input_hit": 0.022, "input_miss": 0.66, "output": 1.98 } + } + } + ] } diff --git a/eval/run.ts b/eval/run.ts index be39644..3e96fa6 100644 --- a/eval/run.ts +++ b/eval/run.ts @@ -2,7 +2,7 @@ // Matrix runner: (task, adapter, model) x N -> runs.jsonl. // Usage: // bun run.ts --tasks bugfix-slugify,feature-tdd --adapters dsc \ -// --models deepseek-v4-flash --n 3 +// --models deepseek-flash --n 3 // Design: deepseek-code/EVAL.md. One run = fresh fixture copy -> headless // invocation -> dumb bash verifier -> metrics row. Sequential on purpose: // DeepSeek's prefix cache is time-sensitive and parallel runs would blur @@ -51,13 +51,17 @@ const taskNames = .sort() : taskArg.split(","); const adapterNames = arg("adapters", "dsc").split(","); -const models = arg("models", "deepseek-v4-flash").split(","); +const models = arg("models", "deepseek-flash").split(","); const n = parseInt(arg("n", "1"), 10); const pricing = await Bun.file(join(EVAL_DIR, "pricing.json")).json(); +/** Off-peak cost on the card in force now (see pricing.json "unit"). */ function costUsd(model: string, u: Usage | null): number | null { - const p = pricing.models[model]; - if (!p || !u) return null; + const tier = pricing.tiers[model]; + if (!tier || !u) return null; + const now = Date.now(); + const card = pricing.cards.filter((c: { since: string }) => Date.parse(c.since) <= now).at(-1); + const p = card.rates[tier]; return (u.inputFresh * p.input_miss + u.cacheRead * p.input_hit + u.output * p.output) / 1e6; } diff --git a/package.json b/package.json index aa8b988..1bacecc 100644 --- a/package.json +++ b/package.json @@ -3,7 +3,7 @@ "version": "0.7.0", "private": false, "type": "module", - "description": "DeepSeek-native coding agent for the v4 flash/pro series. TypeScript/Bun, zero runtime dependencies.", + "description": "DeepSeek-native coding agent, defaulting to DeepSeek-V4.1-Flash (deepseek-flash). TypeScript/Bun, zero runtime dependencies.", "scripts": { "test": "bun test tests", "eval": "bun eval/run.ts", diff --git a/src/cli.ts b/src/cli.ts index 9f0c19e..e0a4e87 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -25,7 +25,7 @@ // Key resolution: $DEEPSEEK_API_KEY, else ~/.dsc/key. import { resolve } from "node:path"; -import { DEFAULT_BASE_URL, DEFAULT_MODEL, MODELS } from "./provider/catalog"; +import { CURRENT_MODELS, DEFAULT_BASE_URL, DEFAULT_MODEL, MODELS } from "./provider/catalog"; import { resolveApiKey } from "./config"; import { schedulerCli } from "./scheduler/cli"; import { addUsage } from "./provider/types"; @@ -57,7 +57,7 @@ const USAGE = `usage: dsc job|ps|serve scheduled jobs (dsc job for details) dsc skills discovered SKILL.md skills and their sources - --model NAME ${Object.keys(MODELS).join(" | ")} + --model NAME ${CURRENT_MODELS.join(" | ")} (default ${DEFAULT_MODEL}) --cwd DIR working directory for the run --continue interactive: resume the latest session here --resume ID resume a specific session @@ -111,7 +111,7 @@ if (promptIdx >= 0 && prompt === undefined) { const model = argValue("model") ?? process.env.DSC_MODEL ?? DEFAULT_MODEL; if (!MODELS[model]) { - console.error(`dsc: unknown model "${model}" (known: ${Object.keys(MODELS).join(", ")})`); + console.error(`dsc: unknown model "${model}" (known: ${CURRENT_MODELS.join(", ")})`); process.exit(2); } const cwd = resolve(argValue("cwd") ?? process.cwd()); diff --git a/src/engine/compact.ts b/src/engine/compact.ts index c5f9e69..12feac6 100644 --- a/src/engine/compact.ts +++ b/src/engine/compact.ts @@ -146,6 +146,9 @@ export async function summarize( tools: [], messages: [...sanitizeForSummary(messages), ask], maxTokens: opts.maxTokens ?? SUMMARY_MAX_TOKENS, + // The cap bounds the summary; with thinking on, reasoning eats it and + // the call returns no text, silently degrading to the emergency summary. + noThinking: true, signal: opts.signal, maxAttempts: 2, }); diff --git a/src/engine/subagent.ts b/src/engine/subagent.ts index 5017457..f006092 100644 --- a/src/engine/subagent.ts +++ b/src/engine/subagent.ts @@ -7,8 +7,10 @@ // Budget envelopes: max turns / max total tokens / max wall-clock. A // looping child dies quietly; the parent gets a partial report. // Children are model-pinned per role (Round 4 rule #1: one transcript, -// one model) — a pro reviewer is a fresh pro context, never a mid- -// transcript model switch. +// one model) — a role on another model gets a fresh context, never a mid- +// transcript model switch. Every role runs deepseek-flash since V4.1 +// Flash (2026-09-10); the reviewer ran pro before that. The benchmark +// comparison behind the switch is in README.md, "Models". import type { Message, Usage } from "../provider/types"; import { addUsage, zeroUsage } from "../provider/types"; @@ -44,9 +46,9 @@ const READ_ONLY_TOOLS = [readTool, bashTool]; export const ROLES: Record = { explorer: { name: "explorer", - description: "read-only investigation (fast model)", + description: "read-only investigation", tools: READ_ONLY_TOOLS, - model: "deepseek-v4-flash", + model: "deepseek-flash", preamble: "You are an explorer sub-agent: investigate the codebase to answer " + "the task below. Do not modify any files.", @@ -54,9 +56,9 @@ export const ROLES: Record = { }, implementer: { name: "implementer", - description: "makes a code change and verifies it (fast model)", + description: "makes a code change and verifies it", tools: [readTool, bashTool, editTool, writeTool], - model: "deepseek-v4-flash", + model: "deepseek-flash", preamble: "You are an implementer sub-agent: make the change described in the " + "task below, verify it, and report exactly what you changed.", @@ -64,9 +66,9 @@ export const ROLES: Record = { }, reviewer: { name: "reviewer", - description: "read-only code review (strong model)", + description: "read-only code review", tools: READ_ONLY_TOOLS, - model: "deepseek-v4-pro", + model: "deepseek-flash", preamble: "You are a reviewer sub-agent: read the code relevant to the task " + "below and report concrete problems (bugs, missed edge cases, broken " + @@ -75,9 +77,9 @@ export const ROLES: Record = { }, tester: { name: "tester", - description: "runs tests/commands and reports results (fast model)", + description: "runs tests/commands and reports results", tools: READ_ONLY_TOOLS, - model: "deepseek-v4-flash", + model: "deepseek-flash", preamble: "You are a tester sub-agent: run the commands or tests named in the " + "task below and report the results. Do not edit source files.", diff --git a/src/provider/catalog.ts b/src/provider/catalog.ts index e6c5d13..3e3c984 100644 --- a/src/provider/catalog.ts +++ b/src/provider/catalog.ts @@ -1,36 +1,118 @@ // Model catalog with REAL numbers (DESIGN.md: budget math always uses the // measured 616k usable input, never the advertised 1M). -// Pricing: USD per 1M tokens, from deepseek-docs quick_start/pricing.md. -// Re-verified 2026-08-12 at V4-Pro GA (model version DeepSeek-V4-Pro-0813; -// model ID unchanged): rates, context, and max output are unchanged. A -// broad repricing is announced but undated — do NOT apply it here until -// it lands with an effective date. +// Pricing: USD per 1M tokens, from deepseek-docs quick_start/pricing.md, +// re-verified 2026-09-18 against the pricing page, the 2026-09-10 +// changelog, and GET /models (which lists exactly deepseek-flash and +// deepseek-v4-pro). + +export type Tier = "flash" | "pro"; export type ModelInfo = { id: string; + /** Which rate-card row this name bills on. */ + tier: Tier; /** Usable input budget in tokens (measured), for context math. */ inputBudget: number; advertisedContext: number; maxOutput: number; - pricing: { inputMiss: number; inputHit: number; output: number }; + /** A name the API still accepts for a model it has retired. Accepted so + * existing jobs and scripts keep working; never listed as a choice. */ + retired?: boolean; }; +const LIMITS = { inputBudget: 616_000, advertisedContext: 1_000_000, maxOutput: 384_000 }; + export const MODELS: Record = { - "deepseek-v4-flash": { - id: "deepseek-v4-flash", - inputBudget: 616_000, - advertisedContext: 1_000_000, - maxOutput: 384_000, - pricing: { inputMiss: 0.14, inputHit: 0.0028, output: 0.28 }, - }, - "deepseek-v4-pro": { - id: "deepseek-v4-pro", - inputBudget: 616_000, - advertisedContext: 1_000_000, - maxOutput: 384_000, - pricing: { inputMiss: 0.435, inputHit: 0.003625, output: 0.87 }, - }, + // DeepSeek-V4.1-Flash, released 2026-09-10. A floating alias: DeepSeek + // can move it to a newer Flash without a new name. + "deepseek-flash": { id: "deepseek-flash", tier: "flash", ...LIMITS }, + // DeepSeek-V4-Pro-0813. Continues after 2026-09-14 with billing + // unchanged (changelog 2026-09-10). + "deepseek-v4-pro": { id: "deepseek-v4-pro", tier: "pro", ...LIMITS }, + // Retired 2026-09-10: served by V4.1 Flash and billed at the Flash price. + "deepseek-v4-flash": { id: "deepseek-v4-flash", tier: "flash", retired: true, ...LIMITS }, + "deepseek-v4-flash-vision-exp": { id: "deepseek-v4-flash-vision-exp", tier: "flash", retired: true, ...LIMITS }, }; -export const DEFAULT_MODEL = "deepseek-v4-flash"; +/** The names to offer a user, in the order the rate card lists them. */ +export const CURRENT_MODELS = Object.keys(MODELS).filter((m) => MODELS[m].retired !== true); + +export const DEFAULT_MODEL = "deepseek-flash"; export const DEFAULT_BASE_URL = "https://api.deepseek.com/anthropic"; + +export type Rates = { inputMiss: number; inputHit: number; output: number }; + +/** One rate card: what each tier cost from `since` until the next card. + * With `timeOfUse`, the rates are OFF-PEAK and peak costs PEAK_MULTIPLIER + * times them. Superseded cards stay so a cost is always computed on the + * card in force when the call was made. */ +export type Card = { label: string; since: Date; timeOfUse: boolean; rates: Record }; + +const V4_PRO: Rates = { inputHit: 0.022, inputMiss: 0.66, output: 1.98 }; + +export const CARDS: Card[] = [ + { + // Published 2026-08-02; flat, no peak. + label: "flat", + since: new Date(0), + timeOfUse: false, + rates: { + flash: { inputHit: 0.0028, inputMiss: 0.14, output: 0.28 }, + pro: { inputHit: 0.003625, inputMiss: 0.435, output: 0.87 }, + }, + }, + { + // V4 GA repricing, effective 16:00 UTC 2026-08-16 (changelog 2026-08-13). + label: "V4", + since: new Date("2026-08-16T16:00:00Z"), + timeOfUse: true, + rates: { flash: { inputHit: 0.007, inputMiss: 0.22, output: 0.66 }, pro: V4_PRO }, + }, + { + // V4.1 Flash, 2026-09-10: Flash cut on every item, Pro unchanged. + // The instant is INFERRED: our docs mirror read the old card at 04:50 + // UTC and the new one at 11:27 UTC that day. 11:00 is the last whole + // hour between, so a call in the gap can only be overstated. + label: "V4.1", + since: new Date("2026-09-10T11:00:00Z"), + timeOfUse: true, + rates: { flash: { inputHit: 0.003, inputMiss: 0.15, output: 0.6 }, pro: V4_PRO }, + }, +]; + +export const PEAK_MULTIPLIER = 2; + +/** Peak hours, UTC: 01:00-04:00 and 06:00-10:00 (09-12 and 14-18 Beijing). */ +const PEAK_HOURS_UTC: Array<[number, number]> = [ + [1, 4], + [6, 10], +]; + +/** From 16:00 UTC 2026-08-22 (00:00 Beijing, Sunday) weekends bill + * off-peak all day. Every peak hour falls on the same calendar day in UTC + * and Beijing, so the UTC weekday decides. Before it, peak ran daily. */ +const WEEKDAYS_ONLY_SINCE = new Date("2026-08-22T16:00:00Z"); + +export function cardAt(at: Date): Card { + let card = CARDS[0]; + for (const c of CARDS) if (at >= c.since) card = c; + return card; +} + +export function isPeak(at: Date): boolean { + if (!cardAt(at).timeOfUse) return false; + const day = at.getUTCDay(); + if (at >= WEEKDAYS_ONLY_SINCE && (day === 0 || day === 6)) return false; + const h = at.getUTCHours(); + return PEAK_HOURS_UTC.some(([start, end]) => h >= start && h < end); +} + +/** Per-1M rates for a model at an instant, peak applied. Undefined for a + * name the catalog does not know. */ +export function ratesAt(model: string, at: Date): Rates | undefined { + const info = MODELS[model]; + if (info === undefined) return undefined; + const base = cardAt(at).rates[info.tier]; + const k = isPeak(at) ? PEAK_MULTIPLIER : 1; + return { inputMiss: base.inputMiss * k, inputHit: base.inputHit * k, output: base.output * k }; +} diff --git a/src/provider/client.ts b/src/provider/client.ts index 88e753a..7fa8935 100644 --- a/src/provider/client.ts +++ b/src/provider/client.ts @@ -43,6 +43,11 @@ export type StreamOpts = { maxAttempts?: number; /** Tool results to render as reclaimed stubs; set only at run boundaries. */ reclaimIds?: Set; + /** Send `thinking: {type: "disabled"}`. Thinking is on by default upstream + * and its tokens count against max_tokens, so a call whose cap is meant + * to bound the ANSWER must turn it off: deepseek-flash spent a whole + * 512-token cap on reasoning and returned no text (measured 2026-09-18). */ + noThinking?: boolean; }; const DEFAULT_MAX_TOKENS = 65_536; @@ -107,6 +112,7 @@ export function buildPayload(opts: StreamOpts): Record { stream: true, system: opts.system, tools: opts.tools, + ...(opts.noThinking === true ? { thinking: { type: "disabled" } } : {}), messages: toWire(opts.messages, opts.reclaimIds), }; } diff --git a/src/scheduler/ledger.ts b/src/scheduler/ledger.ts index 541711b..1b87ae4 100644 --- a/src/scheduler/ledger.ts +++ b/src/scheduler/ledger.ts @@ -18,7 +18,7 @@ import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync, appendFileSync, statSync } from "node:fs"; import { homedir } from "node:os"; import { isAbsolute, join } from "node:path"; -import { MODELS } from "../provider/catalog"; +import { CURRENT_MODELS, MODELS } from "../provider/catalog"; import { parseCron } from "./cron"; export type ToolPreset = "read" | "write"; @@ -101,7 +101,7 @@ export function validateJobSpec(spec: JobSpec): string[] { if (!isAbsolute(spec.cwd)) problems.push(`cwd "${spec.cwd}" must be absolute`); else if (!existsSync(spec.cwd)) problems.push(`cwd "${spec.cwd}" does not exist`); if (MODELS[spec.model] === undefined) { - problems.push(`unknown model "${spec.model}" (known: ${Object.keys(MODELS).join(", ")})`); + problems.push(`unknown model "${spec.model}" (known: ${CURRENT_MODELS.join(", ")})`); } if (spec.tools !== "read" && spec.tools !== "write") { problems.push(`tools must be "read" or "write", got "${spec.tools}"`); diff --git a/src/ui/render.ts b/src/ui/render.ts index e92f8f7..1deefd7 100644 --- a/src/ui/render.ts +++ b/src/ui/render.ts @@ -11,7 +11,7 @@ import type { AgentEvent } from "../engine/events"; import type { Usage } from "../provider/types"; -import { MODELS } from "../provider/catalog"; +import { ratesAt } from "../provider/catalog"; export type Style = (s: string) => string; @@ -90,10 +90,11 @@ export function formatCount(n: number): string { return `${(n / 1_000_000).toFixed(2)}M`; } -/** Cost in USD from the catalog's per-1M rates. The catalog is the single - * pricing source; eval recomputes independently on purpose. */ -export function costUsd(usage: Usage, model: string): number { - const p = MODELS[model]?.pricing; +/** Cost in USD from the catalog's per-1M rates on the card (and peak or + * off-peak period) in force at `at`. The catalog is the single pricing + * source; eval recomputes independently on purpose. */ +export function costUsd(usage: Usage, model: string, at: Date = new Date()): number { + const p = ratesAt(model, at); if (p === undefined) return 0; return ( (usage.inputFresh * p.inputMiss + usage.cacheRead * p.inputHit + usage.output * p.output) / diff --git a/src/ui/repl.ts b/src/ui/repl.ts index 6cc1f01..31e3c51 100644 --- a/src/ui/repl.ts +++ b/src/ui/repl.ts @@ -26,7 +26,7 @@ import { createInterface, type Interface } from "node:readline"; import { addUsage, zeroUsage, type Message, type Usage } from "../provider/types"; import { compactThreshold } from "../engine/compact"; -import { MODELS } from "../provider/catalog"; +import { CURRENT_MODELS, MODELS } from "../provider/catalog"; import { renderReport, type SubagentManager } from "../engine/subagent"; import type { SkillIndexEntry } from "../engine/prompt"; import type { ToolDefinition } from "../tools/index"; @@ -424,11 +424,11 @@ export class Repl { case "/model": { if (arg === "") { - this.write(` ${cyan(this.model)} ${dim(`(known: ${Object.keys(MODELS).join(", ")})`)}\n\n`); + this.write(` ${cyan(this.model)} ${dim(`(known: ${CURRENT_MODELS.join(", ")})`)}\n\n`); return false; } if (MODELS[arg] === undefined) { - this.write(` ${red(`unknown model "${arg}"`)} ${dim(`(known: ${Object.keys(MODELS).join(", ")})`)}\n\n`); + this.write(` ${red(`unknown model "${arg}"`)} ${dim(`(known: ${CURRENT_MODELS.join(", ")})`)}\n\n`); return false; } this.model = arg; diff --git a/tests/catalog.test.ts b/tests/catalog.test.ts new file mode 100644 index 0000000..b0e9a92 --- /dev/null +++ b/tests/catalog.test.ts @@ -0,0 +1,72 @@ +// The rate cards are dated, so a cost is computed on the card and period +// in force at the instant asked about. Each case pins one boundary the +// schedule actually crossed. + +import { describe, expect, test } from "bun:test"; +import { CARDS, cardAt, CURRENT_MODELS, DEFAULT_MODEL, isPeak, MODELS, ratesAt } from "../src/provider/catalog"; +import { costUsd } from "../src/ui/render"; + +const at = (iso: string) => new Date(iso); +const MILLION = { inputFresh: 1_000_000, cacheRead: 0, output: 0 }; + +describe("model names", () => { + test("the default is V4.1 Flash, and pro is still offered", () => { + expect(DEFAULT_MODEL).toBe("deepseek-flash"); + expect(CURRENT_MODELS).toEqual(["deepseek-flash", "deepseek-v4-pro"]); + }); + + test("retired names are accepted, bill as flash, and are never offered", () => { + for (const name of ["deepseek-v4-flash", "deepseek-v4-flash-vision-exp"]) { + expect(MODELS[name].tier).toBe("flash"); + expect(MODELS[name].retired).toBe(true); + expect(CURRENT_MODELS).not.toContain(name); + } + }); +}); + +describe("dated cards", () => { + test("cards are in time order", () => { + for (let i = 1; i < CARDS.length; i++) expect(CARDS[i].since > CARDS[i - 1].since).toBe(true); + }); + + test("before the V4 repricing: flat card, no peak", () => { + const t = at("2026-08-12T02:00:00Z"); // a weekday, inside a later peak window + expect(cardAt(t).label).toBe("flat"); + expect(isPeak(t)).toBe(false); + expect(ratesAt("deepseek-v4-flash", t)?.inputMiss).toBe(0.14); + }); + + test("V4 card: peak ran daily until the weekend rule", () => { + expect(isPeak(at("2026-08-22T02:00:00Z"))).toBe(true); // Saturday, before 16:00 UTC + expect(isPeak(at("2026-08-29T02:00:00Z"))).toBe(false); // Saturday, after + expect(ratesAt("deepseek-v4-flash", at("2026-09-09T02:00:00Z"))?.inputMiss).toBe(0.44); // Wed peak + }); + + test("V4.1 Flash card from 2026-09-10 11:00 UTC; pro unchanged across it", () => { + const before = at("2026-09-10T10:59:00Z"); + const after = at("2026-09-10T11:00:00Z"); + expect(ratesAt("deepseek-flash", before)).toEqual({ inputHit: 0.007, inputMiss: 0.22, output: 0.66 }); + expect(ratesAt("deepseek-flash", after)).toEqual({ inputHit: 0.003, inputMiss: 0.15, output: 0.6 }); + expect(ratesAt("deepseek-v4-pro", before)).toEqual(ratesAt("deepseek-v4-pro", after)); + expect(ratesAt("deepseek-v4-pro", after)).toEqual({ inputHit: 0.022, inputMiss: 0.66, output: 1.98 }); + }); + + test("peak doubles every item, weekdays only, boundaries end-exclusive", () => { + expect(ratesAt("deepseek-flash", at("2026-09-14T01:00:00Z"))).toEqual({ inputHit: 0.006, inputMiss: 0.3, output: 1.2 }); + expect(isPeak(at("2026-09-14T00:59:00Z"))).toBe(false); + expect(isPeak(at("2026-09-14T04:00:00Z"))).toBe(false); + expect(isPeak(at("2026-09-14T06:00:00Z"))).toBe(true); + expect(isPeak(at("2026-09-14T10:00:00Z"))).toBe(false); + expect(isPeak(at("2026-09-13T02:00:00Z"))).toBe(false); // Sunday + }); + + test("a retired name costs exactly what deepseek-flash costs", () => { + const t = at("2026-09-15T12:00:00Z"); + expect(costUsd(MILLION, "deepseek-v4-flash-vision-exp", t)).toBe(costUsd(MILLION, "deepseek-flash", t)); + expect(costUsd(MILLION, "deepseek-flash", t)).toBeCloseTo(0.15, 10); + }); + + test("an unknown model costs nothing rather than a guess", () => { + expect(costUsd(MILLION, "deepseek-v9", at("2026-09-15T12:00:00Z"))).toBe(0); + }); +}); diff --git a/tests/claims.test.ts b/tests/claims.test.ts new file mode 100644 index 0000000..419f908 --- /dev/null +++ b/tests/claims.test.ts @@ -0,0 +1,78 @@ +// Published claims about models and prices, asserted against the code +// they describe, so a stale README or eval card turns CI red in the run +// that caused the drift. Every occurrence is collected and checked, not +// just the first: a fresh first copy must not hide a stale second one. + +import { describe, expect, test } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { CARDS, CURRENT_MODELS, DEFAULT_MODEL, MODELS } from "../src/provider/catalog"; +import { ROLES } from "../src/engine/subagent"; + +const root = join(import.meta.dir, ".."); +const read = (f: string) => readFileSync(join(root, f), "utf8"); +const README = read("README.md"); +const newest = CARDS[CARDS.length - 1]; + +function all(text: string, re: RegExp): string[] { + return [...text.matchAll(re)].map((m) => m[1]); +} + +describe("default model claims", () => { + test("every 'default' model named in the README is DEFAULT_MODEL", () => { + const named = [ + ...all(README, /`(deepseek-[\w.-]+)` \(default\)/g), + ...all(README, /\(default (deepseek-[\w.-]+)\)/g), + ]; + expect(named.length).toBeGreaterThanOrEqual(2); + for (const m of named) expect(m).toBe(DEFAULT_MODEL); + }); + + test("every eval command in the docs, and the runner's own default, uses DEFAULT_MODEL", () => { + const named = [ + ...all(README, /--models (\S+)/g), + ...all(read("EVAL.md"), /--models (\S+)/g), + ...all(read("eval/run.ts"), /--models (\S+)/g), + ...all(read("eval/run.ts"), /arg\("models", "([^"]+)"\)/g), + ]; + expect(named.length).toBeGreaterThanOrEqual(4); + for (const m of named) expect(m).toBe(DEFAULT_MODEL); + }); + + test("the --model line in the README offers exactly the current models", () => { + const line = all(README, /^--model (\S+)/gm); + expect(line).toEqual([CURRENT_MODELS.join("|")]); + }); + + test("no built-in sub-agent role runs on a retired name", () => { + for (const role of Object.values(ROLES)) { + expect(MODELS[role.model]).toBeDefined(); + expect(MODELS[role.model].retired).toBeUndefined(); + } + }); +}); + +describe("price claims", () => { + test("the README models table matches the newest card, row for row", () => { + const rows = [...README.matchAll(/^\| `(deepseek-[\w.-]+)`[^|]*\|[^|]*\| ([\d.]+) \/ ([\d.]+) \/ ([\d.]+) \|$/gm)]; + expect(rows.map((r) => r[1])).toEqual(CURRENT_MODELS); + for (const [, model, hit, miss, out] of rows) { + const r = newest.rates[MODELS[model].tier]; + expect([Number(hit), Number(miss), Number(out)]).toEqual([r.inputHit, r.inputMiss, r.output]); + } + }); + + test("eval/pricing.json carries the same dated cards as the catalog", () => { + const evalPricing = JSON.parse(read("eval/pricing.json")); + expect(evalPricing.cards.map((c: { since: string }) => Date.parse(c.since))).toEqual( + CARDS.map((c) => c.since.getTime()), + ); + evalPricing.cards.forEach((c: any, i: number) => { + for (const tier of ["flash", "pro"] as const) { + const r = CARDS[i].rates[tier]; + expect(c.rates[tier]).toEqual({ input_hit: r.inputHit, input_miss: r.inputMiss, output: r.output }); + } + }); + for (const [model, info] of Object.entries(MODELS)) expect(evalPricing.tiers[model]).toBe(info.tier); + }); +}); diff --git a/tests/loop-interrupt.test.ts b/tests/loop-interrupt.test.ts index fde8b46..09eb376 100644 --- a/tests/loop-interrupt.test.ts +++ b/tests/loop-interrupt.test.ts @@ -59,7 +59,7 @@ const fastBaseUrl = `http://localhost:${server.port}/fast`; afterAll(() => server.stop(true)); const base = { - model: "deepseek-v4-flash", + model: "deepseek-flash", cwd: process.cwd(), apiKey: "test", baseUrl, diff --git a/tests/prefix-stability.test.ts b/tests/prefix-stability.test.ts index e286718..6a356cf 100644 --- a/tests/prefix-stability.test.ts +++ b/tests/prefix-stability.test.ts @@ -19,7 +19,7 @@ function payloadString(messages: Message[]): string { buildPayload({ apiKey: "k", baseUrl: "http://x", - model: "deepseek-v4-flash", + model: "deepseek-flash", system: buildSystemPrompt(tools, "/work"), tools: toWireTools(tools), messages, diff --git a/tests/repl-status.test.ts b/tests/repl-status.test.ts index a33fabb..c08d5c9 100644 --- a/tests/repl-status.test.ts +++ b/tests/repl-status.test.ts @@ -16,11 +16,11 @@ function makeRepl(): { repl: Repl; output: () => string; store: SessionStore } { const chunks: string[] = []; const repl = new Repl({ store, - session: Session.create(store, "deepseek-v4-flash", "/work"), + session: Session.create(store, "deepseek-flash", "/work"), makeManager: () => new SubagentManager({ apiKey: "k", baseUrl: "http://x", cwd: "/work" }), makeTools: () => [readTool, bashTool], skills: [{ name: "release-notes", description: "Draft release notes." }], - model: "deepseek-v4-flash", + model: "deepseek-flash", cwd: "/work", apiKey: "k", baseUrl: "http://x", @@ -51,7 +51,7 @@ describe("/status", () => { expect(leave).toBe(false); const out = output(); expect(out).toContain("read, bash"); - expect(out).toContain("deepseek-v4-flash"); + expect(out).toContain("deepseek-flash"); expect(out).toContain("context"); expect(out).toContain("release-notes"); }); diff --git a/tests/scheduler.test.ts b/tests/scheduler.test.ts index 0d4889c..61edbb3 100644 --- a/tests/scheduler.test.ts +++ b/tests/scheduler.test.ts @@ -28,7 +28,7 @@ function job(over: Partial = {}): JobSpec { id: "j1", prompt: "do the thing", cwd: dir, - model: "deepseek-v4-flash", + model: "deepseek-flash", tools: "read", trigger: { kind: "cron", expr: "* * * * *" }, budget: { maxTurns: 5, maxTotalTokens: 100_000, maxWallMs: 60_000 }, diff --git a/tests/session-compaction.test.ts b/tests/session-compaction.test.ts index f10f025..28d20c9 100644 --- a/tests/session-compaction.test.ts +++ b/tests/session-compaction.test.ts @@ -2,7 +2,7 @@ // roundtrip/rebuild, resume usage-zeroing (the autocompact-spiral // trap). Live compaction behavior is exercised by the compaction eval. -import { describe, expect, test } from "bun:test"; +import { afterAll, describe, expect, test } from "bun:test"; import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -14,8 +14,10 @@ import { emergencySummary, retainedTail, sanitizeForSummary, + summarize, TASK_PIN_PREFIX, } from "../src/engine/compact"; +import { buildPayload } from "../src/provider/client"; import { SessionStore } from "../src/session/store"; import type { AssistantMessage, Message } from "../src/provider/types"; @@ -123,7 +125,7 @@ describe("SessionStore", () => { test("roundtrip, compaction rebuild, and resume usage-zeroing", () => { const dir = mkdtempSync(join(tmpdir(), "dsc-store-test-")); const store = new SessionStore(join(dir, "s.db")); - const meta = store.create("deepseek-v4-flash", "/work"); + const meta = store.create("deepseek-flash", "/work"); for (const m of transcript) store.appendMessage(meta.id, m); const plain = store.rebuildView(meta.id); @@ -185,3 +187,48 @@ describe("compactedView", () => { expect(s).toContain("(none recorded)"); }); }); + +// Thinking is on by default upstream and bills against max_tokens. With it +// on, deepseek-flash spent a 512-token summary cap entirely on reasoning +// and returned no text (measured 2026-09-18), so compaction fell back to +// the emergency summary without saying so. The summary call must ask for +// no thinking; every other call must not change its bytes. +describe("summary call and thinking", () => { + const bodies: any[] = []; + const reply = [ + { type: "message_start", message: { usage: { input_tokens: 10, output_tokens: 0 } } }, + { type: "content_block_start", index: 0, content_block: { type: "text", text: "" } }, + { type: "content_block_delta", index: 0, delta: { type: "text_delta", text: "the summary" } }, + { type: "content_block_stop", index: 0 }, + { type: "message_delta", delta: { stop_reason: "end_turn" }, usage: { output_tokens: 3 } }, + { type: "message_stop" }, + ] + .map((e) => `event: ${e.type}\ndata: ${JSON.stringify(e)}\n\n`) + .join(""); + const server = Bun.serve({ + port: 0, + fetch: async (req) => { + bodies.push(await req.json()); + return new Response(reply, { headers: { "content-type": "text/event-stream" } }); + }, + }); + afterAll(() => server.stop(true)); + + test("summarize() sends thinking disabled and keeps its cap", async () => { + const out = await summarize(transcript, undefined, { + apiKey: "test", + baseUrl: `http://localhost:${server.port}`, + model: "deepseek-flash", + maxTokens: 512, + }); + expect(out).toEqual({ summary: "the summary", llm: true }); + expect(bodies.at(-1).thinking).toEqual({ type: "disabled" }); + expect(bodies.at(-1).max_tokens).toBe(512); + }); + + test("an ordinary turn carries no thinking key", () => { + const body = buildPayload({ apiKey: "k", baseUrl: "u", model: "deepseek-flash", system: "s", tools: [], messages: [] }); + expect("thinking" in body).toBe(false); + expect(Object.keys(body).at(-1)).toBe("messages"); + }); +}); diff --git a/tests/subagent.test.ts b/tests/subagent.test.ts index 8812560..f90a168 100644 --- a/tests/subagent.test.ts +++ b/tests/subagent.test.ts @@ -72,9 +72,9 @@ describe("SubagentManager", () => { mgr.spawn("implementer", "z"); const [explorer, reviewer, implementer] = stub.calls; expect(explorer.tools.map((t) => t.name)).toEqual(["read", "bash"]); - expect(explorer.model).toBe("deepseek-v4-flash"); + expect(explorer.model).toBe("deepseek-flash"); expect(explorer.maxTurns).toBe(ROLES.explorer.budget.maxTurns); - expect(reviewer.model).toBe("deepseek-v4-pro"); + expect(reviewer.model).toBe("deepseek-flash"); expect(implementer.tools.map((t) => t.name)).toEqual(["read", "bash", "edit", "write"]); // Role preamble + task + report contract ride in the USER prompt. expect(explorer.prompt).toContain("explorer sub-agent");