feat(bench): effort, provider and scoring columns rank the llms-benchmark, with opus 5.5, fable 5.1, gpt-6, grok and glm-prime rows - #651
Conversation
…ium and high The results tables gain an Effort column after Model (every earlier row was measured at medium), and /launch-llms-benchmark takes the effort as an optional third argument, medium by default, passed to each CLI's effort flag; model, effort and CLI identify a row. The copilot launch line lifts prompt mode's 600 s wait on background tasks, which cancelled the first gpt-6-luna observation mid-run. Four rows added: openai/gpt-6-luna and openai/gpt-6-sol on copilot, at medium and high. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…igh, effective effort per row Adds anthropic/claude-opus-5.5 (claude) and openai/gpt-6-luna-pro, openai/gpt-6-sol-pro, x-ai/grok-4.7 and z-ai/glm-5.3-prime (opencode) at medium and high, and re-sorts the table. opencode accepts any --variant name and sends no effort for an unknown one: the z-ai/glm-5.3* models and deepseek/deepseek-v4.1-flash offer no medium variant, so their rows read `default`, and the preflight now checks the model's variants. The claude teardown also clears /tmp/odd-observe-scratch/. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Rulings - second phaseOpus 5.5 on claude and the four opencode models, both runs per model and effort, the row's run named first.
|
Removes openai/gpt-6-luna-pro (medium and high), openai/gpt-6-sol-pro (high) and z-ai/glm-5.3-prime (default) from both results tables on the maintainer's decision, and renumbers the ranking (24 rows). Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gh row Replaces the provisional 1.12.0 row of anthropic/claude-fable-5.1 (claude) with a run under 1.13.0 at medium, adds its high row, and renumbers the ranking (25 rows, none provisional). Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Rulings - third phaseFable 5.1 on claude, both runs per effort, the row's run named first.
|
…edium row The headline table gains seconds/confirmed after $/confirmed - the total duration divided by the confirmed findings, the time one trustworthy finding takes - and the command's table contract says so. The openai/gpt-6-sol-pro medium row is removed on the maintainer's decision (24 rows). Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Provider follows CLI in both tables - OpenRouter for opencode, Anthropic for claude, GitHub Copilot for copilot - and Accuracy (confirmed over reported, as a percentage) sits before $/confirmed in the headline table. The reading guide and the command's table contract say so. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Shortens the Provider label of the copilot rows from "GitHub Copilot" to "Copilot" in both tables, the reading guide and the command. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… its format The table contract now gives the exact headers of both tables, an example row, what fills each headline cell and in which format (Effort as applied, Provider per CLI, Accuracy, $/confirmed, seconds/confirmed) and the bolding rule, so the next row is written without guessing. The example's money cells are patterns: the command's text never carries a dollar sign followed by a digit. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…l-row mark The warning sign written in the table contract carried U+FE0E, which the package audit flags as a hidden character and fails on; the mark is now the plain sign. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The row key gains the provider beside model, effort and CLI, in the command's arguments and table contract and in the README's intro and reading guide: the same model through two providers is two rows. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The headline table gains Scoring (0-100) before Confirmed / reported, and the rank is now the table sorted by it. The score weighs $/confirmed 30 %, seconds/confirmed 30 %, Total 20 % (all three on a log scale between fixed bounds), Accuracy 10 % and Confirmed 10 %; the fixed bounds keep a row's score independent of the other rows. The README's reading guide describes it in one bullet, the command states the formula, the bounds and the sort. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
using-system
changed the title
1 participant
Closes #650
First batch of the llms-benchmark update tracked in #650 - more changes will follow on the same issue.
What changes
medium, and says so./launch-llms-benchmark <cli> <vendor/model> [effort]- the effort is an optional third argument,mediumby default, passed verbatim to each CLI's flag (--variantfor opencode,--effortfor claude and copilot): in the smoke runs, the launch lines and the branch names. Model, effort, CLI and provider identify a row.COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. In prompt mode Copilot waits 600 s for background tasks and then cancels them; the first gpt-6-luna run dispatchedobserve-runin background mode and lost its observation ten minutes in (background_task_wait_timeout,subagent.completedcancelled: true). That run is void (0.069 USD, 35 turns). The earlier copilot rows all ended under ten minutes, so none is invalidated.openai/gpt-6-lunaandopenai/gpt-6-solon copilot, atmediumandhigh.Scoringcolumn ranks the table: 0-100, beforeConfirmed / reported, the weighted sum of$/confirmed30 %,seconds/confirmed30 %,Total20 % (log scales between fixed bounds: 0.01-1.00 USD, 30-600 s, 5-60 min),Accuracy10 % andConfirmed10 % (20 findings = 100). The rank is the table sorted by it - no longer decided per PR - and the fixed bounds keep a row's score independent of the others. The command states the formula; the README describes it in one bullet. With it the table re-sorts to gpt-5.6-luna (79.7), deepseek-v4.1-flash (70.9), gpt-5.6-sol (68.5), gpt-5.6-terra (67.9), claude-opus-5.5 medium (64.4) at the top.$/confirmed,seconds/confirmed) and the bolding rule, so the next run writes its row without guessing.Providercolumn afterCLIin both tables (OpenRouter for opencode, Anthropic for claude, Copilot for copilot) and anAccuracycolumn before$/confirmed(confirmed over reported, as a percentage).seconds/confirmedcolumn after$/confirmedin the headline table: the total duration divided by the confirmed findings, the time one trustworthy finding takes (the command's table contract and the reading guide say so).anthropic/claude-opus-5.5on claude, andopenai/gpt-6-luna-pro,openai/gpt-6-sol-pro,x-ai/grok-4.7andz-ai/glm-5.3-primeon opencode, each atmedium(ordefault) andhigh; the whole table re-sorted (28 rows).default.opencode runaccepts any--variantname - an unknown one included (verified with--variant bogus) - records it on every message and sends no reasoning effort.opencode models openrouter --verboselists each model's variants: thez-ai/glm-5.3*models anddeepseek/deepseek-v4.1-flashofferlow,highandmaxonly, so their rows measured "at medium" ran at the provider's default effort and now readdefault(z-ai/glm-5.3-flashx,z-ai/glm-5.3,z-ai/glm-5.3-flash,deepseek/deepseek-v4.1-flash).z-ai/glm-5.3-prime's medium slot ran without--variantand readsdefaulttoo. The command's opencode preflight now checks the variant, and its claude teardown also clears/tmp/odd-observe-scratch/.subagent.configuredcarriesreasoningEffortat the requested level (medium or high) and reasoning tokens rise with it; every opencode assistant message carries the requested variant (none fordefault); every opencode cost reconciles to the cent at OpenRouter's prices with the doubled tier above 200,000 prompt tokens. The oddyssey column reads1.13.0on the maintainer's decision; the runs executed on 1.13.1 (oddyssey-mcp==1.13.1, package frommainatfa1bd7f, identical tov1.13.1), whose.apm/differs from 1.13.0's bysetup-local-stack/references/otel-lgtm-env.mdonly.Copilot CLI 1.0.88 for every run. Cost at OpenAI's list price (USD per million, input and cache write / cache read / output): gpt-6-luna 0.10 / 0.01 / 0.50, gpt-6-sol 2.00 / 0.20 / 10.00.
Rows and placement
openai/gpt-6-solopenai/gpt-6-solopenai/gpt-6-lunaopenai/gpt-6-lunaWhat the effort changes: luna at
highcosts and lasts the same as atmediumand reports more that does not hold; sol athighconfirms three more for 1m30s and 0.17 USD more.Second runs: both sol second runs were stopped by the command's rule (past run 1's total with the report still a skeleton): medium at 10m53s, 0.885 USD, 35 turns; high just past 11m49s, 1.179 USD. A first launch on the wrong model id (
gpt-5.6-lunaforgpt-6-luna) was killed before its drive.On every run: no source file read before the drive, no traffic of its own outside the stored scenario, a replayable verification protocol. One exception to note: luna high run 2 drove the stored
script.jswith ak6 runit composed itself instead of the packaged replay script, without the OpenTelemetry output - the stored scenario, but no driver series in the store.Third phase (Fable 5.1 on claude)
anthropic/claude-fable-5.1anthropic/claude-fable-5.1Both second runs finished: medium run 1 15 / 20, 18m58s, 8.16 USD; high run 2 18 / 21, 20m28s, 11.23 USD. Claude Code 2.1.282; Fable 5.1 at list 10.00 / 50.00 / 0.25 / 12.50 5m / 20.00 1h USD per million,
total_cost_usdmatched the transcript reconstruction exactly on all four runs. No row of the table is provisional any more. The per-finding rulings are in the PR comment "Rulings - third phase".Second phase (Opus 5.5 on claude, four models on opencode)
anthropic/claude-opus-5.5anthropic/claude-opus-5.5x-ai/grok-4.7z-ai/glm-5.3-primex-ai/grok-4.7Five rows of this phase were measured and then removed on the maintainer's decision:
openai/gpt-6-sol-promedium (11 / 11, 10m53s, 4.25 USD),openai/gpt-6-luna-promedium (11 / 13, 25m46s, 0.59 USD) and high (10 / 11, 70m27s, 1.32 USD),openai/gpt-6-sol-prohigh (14 / 15, 19m03s, 7.34 USD),z-ai/glm-5.3-primedefault (7 / 8, 23m09s, 4.73 USD); their rulings stay in the phase-two comment. Placements below were proposed before the Scoring column; the table is now sorted by score. Claude Code 2.1.282 (Opus 5.5 at list: 4.00 / 20.00 / 0.20 / 5.00 5m / 8.00 1h USD per million;total_cost_usdmatched the transcript reconstruction exactly;observe-rundispatched withmodel: opus, which resolved toclaude-opus-5-5, the onlymodelUsagekey). opencode 1.18.31 from~/.opencode/bin.Stopped second runs (the command's rule - past run 1's total with no finished report): Opus 5.5 high 4.34 USD; gpt-6-luna-pro medium 0.37 USD; gpt-6-sol-pro medium 4.62 USD; glm-5.3-prime default 3.14 USD. The machine went into maintenance sleep on battery once, between two runs (00:59-06:40 UTC): no run spanned it, the interrupted preparation was redone on AC power.
The per-finding rulings of this phase are in the PR comment "Rulings - second phase". A new behavior finding shows up in this phase and is graded like any other: some answers quote the tools' cents as euros, and some rank prices out of order - verified in the traces each report cites.
Rulings
openai/gpt-6-sol(copilot, effort high) - run 1 - 11 confirmed / 13 reportedWindow 21:34:46Z-21:36:49Z (k6's own 122.1 s, summary mtime 21:36:48). 7 anomaly rows (F2, F6 and F7 each bundle two defects with two fixes) and 7 gaps, of which four restate F6a, F6b, F7a and F7b: 13 items.
/askarrivals are right, but the ninth is the constant-arrival-rate stage's arrival at t0+120 s, the load generator's boundary - not a defect of the services.cbc62264…carries 25 detail GETs under one search;SEARCH_DETAIL_FANOUTdefaults to 25. Perf.create_default_context57.92 % self;catalog.pybuilds anhttpx.Clientper GET. Perf.POST /ordersat 200, 883 created. Behavior.GET /statsp50 284 ms / p95 478 ms;stats_scan300 ms on7a203459…; apiquery81.50 % self. Perf.search_productscounter 160 against 458 spans, 298 cache-hit lines; the cache-hit branch returns before the increment. Telemetry.model call failed attempt=1/3 … finish_reasonon979ee388…, ERROR model span, retry, root 200. Behavior.gen_ai_client_token_usage_count20 for 21 chat spans; the errored call carries no usage. Telemetry.service_instance_id. Telemetry.memory:alloc_spaceempty for the three services while the store holds allocation data from other emitters. Telemetry.http_client_duration,mcp_tool_calls_totalandtarget_info- no server-side status or duration per tool. Telemetry.POST /orders(0 against the siblingGET /orders/{order_ref}). Telemetry.http.user_agent,http_target) consistently; the report itself says nothing is absent - a naming to query by, not a gap.By kind, confirmed: Telemetry 6 (F5, F6b, F7a, F7b, G1, G2) / Perf 3 (F2a, F2b, F4) / Behavior 2 (F3, F6a).
Figures: launch 21:32:41Z, end 21:44:30Z - preflight 2m05s, drive 2m03s, observation 7m41s, total 11m49s; 51 turns (root + one observe-run subagent), median model-call latency 5.1 s, max 167 s; Input 3,906,289 (uncached 153, cache read 3,686,733, cache write 219,403) / Output 28,633 (reasoning 6,564) / Cache 3,906,136; cost at list (153+219,403)x2.00/M + 3,686,733x0.20/M + 28,633x10.00/M = 1.462789 USD; 1 premium request, 157.25 AIU; signals 4/4 (15 metrics, 10 traces, 7 logs, 9 profiles); 0 source files read before the drive (first at 21:39:29); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort high, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/oddyssey/-local/.
openai/gpt-6-sol(copilot, effort medium) - run 1 - 8 confirmed / 10 reportedWindow 21:10:58Z-21:13:01Z (k6's own 122.7 s, summary mtime 21:13:01). 7 anomalies, 4 gaps of which the second restates F7: 10 items.
POST /askarrivals and counter 9 are right, but the ninth is the constant-arrival-rate stage's arrival at t0+120 s - the load generator's boundary, not a defect of the services (and not a consent breach: the mission accepts the scenario's calls as authored).GET /statsp95 468 ms; trace8a03164…spends 288 ms instats_scan; api profilequery79.87 % self. Perf.1e585f…carries one list and 25 detail GETs (239 ms) against a 1 ms cache hit;SEARCH_DETAIL_FANOUTdefaults to 25. Perf.search_productscounter 160 against 468 spans, 308 cache-hit + 160 miss log lines; the cache-hit branch returns before the increment. Telemetry.POST /ordersanswered 200. Behavior.gen_ai.input.messagesand the other content keys. Telemetry.service_instance_idselector empties the api's profile, dropping it restores 63.72 s. Telemetry.memory:alloc_spaceempty for the three services. Telemetry.POST /orders(the structural query returns 0 against the siblingGET /orders/{order_ref}). Telemetry.By kind, confirmed: Telemetry 5 (F4, F6, F7, G1, G3) / Perf 2 (F2, F3) / Behavior 1 (F5).
Figures: launch 21:09:20Z, end 21:19:43Z - preflight 1m38s, drive 2m03s, observation 6m42s, total 10m23s; 44 turns (root + one observe-run subagent), median model-call latency 5.9 s, max 166 s; Input 3,298,055 (uncached 132, cache read 3,085,729, cache write 212,194) / Output 24,399 (reasoning 2,426) / Cache 3,297,923; cost at list (132+212,194)x2.00/M + 3,085,729x0.20/M + 24,399x10.00/M = 1.285788 USD; 1 premium request, 139.19 AIU; signals 4/4 (13 metrics, 9 traces, 6 logs, 4 profiles); 0 source files read before the drive (first at 21:14:48); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort medium, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/oddyssey/-local/.
openai/gpt-6-luna(copilot, effort medium) - run 2 - 11 confirmed / 11 reported - THE ROW (tie on confirmed with run 1, cheaper)Window 19:41:12Z-19:43:12Z (k6's own 121.3 s, summary mtime 19:43:13). Mode
observe: the root drove the stored scenario itself, then dispatchedobserve-runon the driven window. 6 anomalies (F3 bundles two defects with two fixes) and 5 gaps of which the first restates F5: 11 items.GET /statsp95 ~471 ms; trace7f8221b4…spends 307 ms instats_scan. Perf.mcp_tool_calls_total{search_products}160 against 437 tool traces; the cache-hit branch returns before_tool_calls.add. Telemetry.b38ceb42…carries 25 product-detail spans under one search (SEARCH_DETAIL_FANOUTdefaults to 25). Perf.create_default_context58.16 % self;mcp/app/catalog.pyopens a newhttpx.Clientper call. Perf.model call failed attempt=1/3 … finish_reason 'error'on trace1400ce68…, one ERRORchatspan of 2,679 ms, root 200 at 5,364 ms. Behavior.llmbench-mcpspan carries the run's User-Agent. Telemetry.gen_ai_client_operation_durationamong the agent's metric names. Telemetry.chatspan of1400ce68…carries nogen_ai.usage.*attribute while its siblings do;gen_ai_client_token_usage_count= 18 for 19 chat spans. Telemetry.service_instance_id. Telemetry.memory:alloc_spaceempty for the three services. Telemetry.gen_ai.provider.name=openairemark is prose flagged suspected, not a finding row.By kind, confirmed: Telemetry 6 (F2, F5, G2, G3, G4, G5) / Perf 3 (F1, F3a, F3b) / Behavior 2 (F4, F6).
Figures: launch 19:40:09Z, end 20:05:43Z - preflight 1m03s, drive 2m01s, observation 22m30s, total 25m34s; 64 turns (root + one observe-run subagent), median model-call latency 7.6 s, max 112 s; Input 7,695,591 (uncached 192, cache read 7,301,673, cache write 393,726) / Output 137,707 (reasoning 109,687) / Cache 7,695,399; cost at list (192+393,726)x0.10/M + 7,301,673x0.01/M + 137,707x0.50/M = 0.181262 USD; 1 premium request, 19.11 AIU; signals 4/4 (34 metrics, 18 traces, 7 logs, 10 profiles); 0 source files read before the drive (first at 19:51:30); no traffic of its own (one replay launch at 19:40:44 refused its arguments, the 19:41:11 one drove); replayable protocol: yes. Copilot CLI 1.0.88, --effort medium, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/llmbench-observe-*.
openai/gpt-6-luna(copilot, effort medium) - run 1 - 11 confirmed / 12 reportedWindow 19:12:49Z-19:14:50Z (k6's own 121.1 s and the summary's mtime agree). 5 anomalies, 8 gaps of which G4 restates F3: 12 items.
query82.75 % self of 75.6 s,stats75.42 % total;GET /statsp99 ~494 ms on the api histogram. Perf.SEARCH_DETAIL_FANOUTdefaults to 25 inmcp/app/server.py; cold-halfsearch_productsp95 220 ms vs warm 2.8 ms, trace483ae…carries 25 detail GETs. Perf.mcp_tool_calls_total{search_products}delta 160 against 443tools/call search_productsroots; the cache-hit branch returns before_tool_calls.add. Telemetry.order rejected … out-of-stockWARN lines answered 200;catalog_orders_created_total855 of 877 attempts. Behavior.gen_ai.input.messageson their spans (the report stores the attribute names only). Telemetry.llmbench-mcpspan carrieshttp.user_agentoruser_agent.original(0 of 443 roots). Telemetry.checks_total{condition="nonzero"}is the passed-check count (10,098 = every check), so the outcome is exported; the report read the Rate encoding as missing polarity.llmbench-mcpexportshttp_client_duration_*,mcp_tool_calls_total,target_infoonly - no server-side duration. Telemetry.memory:alloc_spaceon{service_name="llmbench-api"}is empty while the store holds allocation data for other services. Telemetry.service_instance_id. Telemetry.gen_ai_client_token_usagebut nogen_ai_client_operation_duration. Telemetry.{ POST /orders } >> { sqlite }returns 0 of 877 while the siblingGET /orders/{order_ref}returns 866. Telemetry.By kind, confirmed: Telemetry 8 (F3, F5, G1, G3, G5, G6, G7, G8) / Perf 2 (F1, F2) / Behavior 1 (F4).
Figures: launch 19:10:30Z, end 19:37:44Z - preflight 2m19s, drive 2m01s, observation 22m54s, total 27m14s; 60 turns (root + one observe-run subagent), median model-call latency 10.8 s, max 175 s; Input 8,848,387 (uncached 180, cache read 8,515,129, cache write 333,078) / Output 151,292 (reasoning 114,258) / Cache 8,848,207; cost at list (180+333,078)x0.10/M + 8,515,129x0.01/M + 151,292x0.50/M = 0.194123 USD; 1 premium request, 23.45 AIU; signals 4/4 (31 metrics, 23 traces, 14 logs, 9 profiles); 0 source files read before the drive (five views from 19:26:03, after); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort medium, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200 (the first attempt, void, was cancelled by the 600 s wait: 0.069 USD, 35 turns).
openai/gpt-6-luna(copilot, effort high) - run 2 - 10 confirmed / 15 reported - THE ROW (more confirmed than run 1's 9)Window 20:44:41Z-20:46:43Z (k6's own 121.1 s, summary mtime 20:46:42). The run drove the stored script with a
k6 runit composed itself (noreplay_benchmark.py, no-o opentelemetry) - the stored scenario, so no traffic of its own, but the driver exported nothing. 8 anomaly rows (2 and 8 each bundle two defects with two fixes) and 8 gap bullets, of which the first isfilledand two restate rows 5 and 6: 15 items.GET /statsp95 477 ms,stats_scanselects every product column and aggregates in Python; api profilequerydominant. Perf.SEARCH_DETAIL_FANOUTdefaults to 25. Perf.create_default_context55.82 % self;catalog.pybuilds anhttpx.Clientper call. Perf.search_productscounter 162 against 436 spans; the cache-hit branch returns before the increment. Telemetry./asktraces per half and a model span dominating is the paid provider's variance, not a defect the services own.llmbench-mcpspan carries the run's User-Agent. Telemetry.checks_totalwith itsconditionlabel.gen_ai.provider.name=openainames the OpenAI-compatible client the SDK speaks through, withserver.address=openrouter.aibeside it; nothing is mislabelled.memory:alloc_spaceempty for the three services. Telemetry.service_instance_idamong the profile labels. Telemetry.gen_ai_client_operation_durationamong the agent's metrics. Telemetry.http_client_durationby method and status; no per-tool or server-side duration. Telemetry.GET /products?categoryand?category&qshare the/productsroute by design; a route template carries no query string.filled, pre-drive emptiness), G2 (restates 5), G3 (restates 6).By kind, confirmed: Telemetry 7 (3, 5, 8b, G4, G5, G6, G7) / Perf 3 (1, 2a, 2b) / Behavior 0 (the out-of-stock rejections are named and dismissed as the authored contract).
Figures: launch 20:42:37Z, end 21:07:44Z - preflight 2m04s, drive 2m01s, observation 21m01s, total 25m07s; 57 turns (root + one observe-run subagent), median model-call latency 10.9 s, max 183 s; Input 6,643,162 (uncached 171, cache read 6,280,430, cache write 362,561) / Output 165,474 (reasoning 135,967) / Cache 6,642,991; cost at list (171+362,561)x0.10/M + 6,280,430x0.01/M + 165,474x0.50/M = 0.181814 USD; 1 premium request, 27.06 AIU; signals 4/4 (22 metrics, 22 traces, 6 logs, 21 profiles); 0 source files read before the drive (first at 20:54:18); no traffic of its own (the stored script, launched by hand); replayable protocol: yes. Copilot CLI 1.0.88, --effort high, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/llmbench-store-load-*-local/ (k6 summary inside it).
openai/gpt-6-luna(copilot, effort high) - run 1 - 9 confirmed / 12 reportedWindow 20:09:12Z-20:11:15Z (k6's own 122.0 s, summary mtime 20:11:15). 6 anomalies, 6 gaps: 12 items.
POST /askarrivals andagent_questions_totaldelta 9 are right, but the ninth is the constant-arrival-rate stage's arrival at t0+120 s - the load generator's boundary, which lands on some runs and not others, not a defect of the services.search_productscounter 160 against 472 invocations, 312 cache-hit log lines make up the difference; the cache-hit branch returns before_tool_calls.add. Telemetry.catalog_orders_created_total911 of 933. Behavior.query77.52 % self of 63.16 s;stats_scan312.6 ms on tracea0e25cd3…. Perf.gen_ai.input.messagesand the other content keys (names only in the report). Telemetry.create_default_context55.83 % self (10.25 s of 18.36 s);catalog.pyopens anhttpx.Clientper call - labelled suspected, numbers exact. Perf.llmbench-mcpspan carries the run's User-Agent. Telemetry.service_instance_idamong the profile labels. Telemetry.memory:alloc_spaceempty for the three services. Telemetry.service_instance_id.Created new transport with session ID, carry no trace id. Telemetry.checks_total{condition="nonzero"}is the passed-check count, so the check outcome is exported.By kind, confirmed: Telemetry 6 (F2, F5, G1, G2, G3, G5) / Perf 2 (F4, F6) / Behavior 1 (F3).
Figures: launch 20:07:29Z, end 20:41:13Z - preflight 1m43s, drive 2m03s, observation 29m58s, total 33m44s; 63 turns (root + one observe-run subagent), median model-call latency 11.9 s, max 132 s; Input 8,022,813 (uncached 189, cache read 7,464,392, cache write 558,232) / Output 205,995 (reasoning 163,540) / Cache 8,022,624; cost at list (189+558,232)x0.10/M + 7,464,392x0.01/M + 205,995x0.50/M = 0.233484 USD; 1 premium request, 30.31 AIU; signals 4/4 (28 metrics, 22 traces, 10 logs, 14 profiles); 0 source files read before the drive (first at 20:21:49); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort high, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200.
Stopped second runs
openai/gpt-6-solmedium, run 2: run 2 stopped at 21:31:51Z, launched 21:20:58Z (10m53s), phase: report skeleton written, 8 sections still ; spend to that point: 0.885264 USD at list, 35 turns, Input 2,502,839 / Output 11,376openai/gpt-6-solhigh, run 2: run 2 stopped at 21:58:13Z, launched 2026-09-24T21:45:53Z, phase: report skeleton, 8 sections still ; cost at list: (111+162,973)x2.0/M + 2,657,186x0.2/M + 32,137x10.0/M = 1.178975 USDReview
No sub-agent review on this branch, at the maintainer's request for this session.
🤖 Generated with Claude Code